www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP69 - Implement scope for escape proof references

reply Walter Bright <newshound2 digitalmars.com> writes:
http://wiki.dlang.org/DIP69

Despite its length, this is a fairly simple proposal. It adds the missing 
semantics for the 'scope' storage class in order to make it possible to pass a 
reference to a function without it being possible for it to escape.

This, among other things, makes a ref counting type practical. It also makes it 
more practical to use other storage allocation schemes than garbage collection.

It does not make scope into a type constructor, nor a general type-annotation 
system.

It does not provide an ownership system, though it would complement one.
Dec 04 2014
next sibling parent reply "eles" <eles eles.com> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.
Making it implicit and requiring an explicit "escape" for un-scoped variables?
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:51 AM, eles wrote:
 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the missing
 semantics for the 'scope' storage class in order to make it possible to pass a
 reference to a function without it being possible for it to escape.
Making it implicit and requiring an explicit "escape" for un-scoped variables?
Was afraid that would break too much code.
Dec 04 2014
next sibling parent reply "eles" <eles eles.com> writes:
On Thursday, 4 December 2014 at 10:00:37 UTC, Walter Bright wrote:
 On 12/4/2014 1:51 AM, eles wrote:
 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright 
 wrote:
 http://wiki.dlang.org/DIP69
 Was afraid that would break too much code.
An annotation for functions could make all variables "scope"? Similar to: int foo(int x) scopedvars { //all declarations are implicit "scope"; use "escape" for the others }
Dec 04 2014
parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 04 Dec 2014 10:04:07 +0000
eles via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Thursday, 4 December 2014 at 10:00:37 UTC, Walter Bright wrote:
 On 12/4/2014 1:51 AM, eles wrote:
 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright=20
 wrote:
 http://wiki.dlang.org/DIP69
=20
 Was afraid that would break too much code.
=20 An annotation for functions could make all variables "scope"? =20 Similar to: =20 int foo(int x) scopedvars { //all declarations are implicit "scope"; use "escape" for the=20 others =20 }
please, don't steal this nice identifier! i love variables with name "escape"! stolen "body" still hurts me.
Dec 04 2014
parent "eles" <eles eles.com> writes:
On Thursday, 4 December 2014 at 10:11:25 UTC, ketmar via 
Digitalmars-d wrote:
 On Thu, 04 Dec 2014 10:04:07 +0000
 eles via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Thursday, 4 December 2014 at 10:00:37 UTC, Walter Bright 
 wrote:
 On 12/4/2014 1:51 AM, eles wrote:
 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright 
 wrote:
 please, don't steal this nice identifier! i love variables with 
 name
 "escape"! stolen "body" still hurts me.
It was just an idea, I do not push for it.
Dec 04 2014
prev sibling next sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2014-12-04 10:00:28 +0000, Walter Bright said:

 On 12/4/2014 1:51 AM, eles wrote:
 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the missing
 semantics for the 'scope' storage class in order to make it possible to pass a
 reference to a function without it being possible for it to escape.
Making it implicit and requiring an explicit "escape" for un-scoped variables?
Was afraid that would break too much code.
No, this is super important. Break it all! This kind of change will significantly increase the performance of D code by almost eliminating GC thrashing. Having to type scope EVERYWHERE will make it unused and therefore unusable in other people's libraries. -Shammah.
Dec 10 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 10 Dec 2014 23:06:13 -0800
Shammah Chancellor via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 Was afraid that would break too much code.
No, this is super important. Break it all!
alas, not in this life. each "break our code" just making resistance harder. ;-)
Dec 11 2014
prev sibling parent "zeljkog" <zeljkog home.com> writes:
On Thursday, 4 December 2014 at 10:00:37 UTC, Walter Bright wrote:
 On 12/4/2014 1:51 AM, eles wrote:
 Making it implicit and requiring an explicit "escape" for 
 un-scoped variables?
Was afraid that would break too much code.
Compiler switch: escape=I(gnore)|W(arning)|E(rror). For transition default Ignore.
Dec 11 2014
prev sibling next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 04 Dec 2014 01:24:13 -0800
Walter Bright via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 http://wiki.dlang.org/DIP69
=20
 Despite its length, this is a fairly simple proposal. It adds the missing=
=20
 semantics for the 'scope' storage class in order to make it possible to p=
ass a=20
 reference to a function without it being possible for it to escape.
=20
 This, among other things, makes a ref counting type practical. It also ma=
kes it=20
 more practical to use other storage allocation schemes than garbage colle=
ction.
=20
 It does not make scope into a type constructor, nor a general type-annota=
tion=20
 system.
=20
 It does not provide an ownership system, though it would complement one.
cosmetic issue: some comments are referring to rules by number ("Error, rule 5"), yet the rules aren't explicitly numbered. not a big deal, but still somewhat hard to follow.
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:53 AM, ketmar via Digitalmars-d wrote:
 cosmetic issue: some comments are referring to rules by number ("Error,
 rule 5"), yet the rules aren't explicitly numbered. not a big deal, but
 still somewhat hard to follow.
Yeah, still learning wiki markup!
Dec 04 2014
next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:57:43AM -0800, Walter Bright via Digitalmars-d wrote:
 On 12/4/2014 1:53 AM, ketmar via Digitalmars-d wrote:
cosmetic issue: some comments are referring to rules by number
("Error, rule 5"), yet the rules aren't explicitly numbered. not a
big deal, but still somewhat hard to follow.
Yeah, still learning wiki markup!
I don't understand the line where rule 5 was invoked: scope int* a; ... scope int** f = &a; // Error, rule 5 Why is it an error, since f has a shorter lifetime than a? And what has it gotta do with rule 5, which currently reads: A scope ref variable can be initialized with another scope ref variable - scope ref is idempotent. ? T -- Real men don't take backups. They put their source on a public FTP-server and let the world mirror it. -- Linus Torvalds
Dec 04 2014
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 10:31:07AM -0800, H. S. Teoh via Digitalmars-d wrote:
 On Thu, Dec 04, 2014 at 01:57:43AM -0800, Walter Bright via Digitalmars-d
wrote:
 On 12/4/2014 1:53 AM, ketmar via Digitalmars-d wrote:
cosmetic issue: some comments are referring to rules by number
("Error, rule 5"), yet the rules aren't explicitly numbered. not a
big deal, but still somewhat hard to follow.
Yeah, still learning wiki markup!
I don't understand the line where rule 5 was invoked: scope int* a; ... scope int** f = &a; // Error, rule 5 Why is it an error, since f has a shorter lifetime than a? And what has it gotta do with rule 5, which currently reads: A scope ref variable can be initialized with another scope ref variable - scope ref is idempotent. ?
[...] Ah, nevermind, it should be rule 4, not rule 5. Rule 4 states that the address of scope variables cannot be assigned to another scope variable. Please fix the comment. ;-) T -- I've been around long enough to have seen an endless parade of magic new techniques du jour, most of which purport to remove the necessity of thought about your programming problem. In the end they wind up contributing one or two pieces to the collective wisdom, and fade away in the rearview mirror. -- Walter Bright
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 10:34 AM, H. S. Teoh via Digitalmars-d wrote:
 Please fix the comment. ;-)
done
Dec 04 2014
prev sibling next sibling parent "Robert burner Schadek" <rburners gmail.com> writes:
thank you for pushing on this.

Lifetime last bullet point: ", but lower than any variables in 
higher scopes."
isn't that redundant to the first bullet point? Or am I missing 
something?

Scope affects variables according to these rules:
Could you enumerate the list instead of bullet points, I can't 
count.

Base operation:
That one I don't get. I would assume this is legal as the 
function is not  safe.
e = &c; // Error, lifetime(e's view) is &infin; and is greater 
than lifetime(c)
Dec 04 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.

 This, among other things, makes a ref counting type practical. 
 It also makes it more practical to use other storage allocation 
 schemes than garbage collection.

 It does not make scope into a type constructor, nor a general 
 type-annotation system.

 It does not provide an ownership system, though it would 
 complement one.
Haven't got the time to look at this in detail right now, but if it's as good as it looks on the surface, I'm very excited to use it. Thanks for all the hard work, all involved.
Dec 04 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
 Errors for scope violations are only reported in  safe code.
Why? If I've explicitly designated a reference as scope, why should it be ignored in un- safe code?
Dec 04 2014
next sibling parent "Martin Nowak" <code dawg.eu> writes:
On Thursday, 4 December 2014 at 11:21:27 UTC, Marc Schütz wrote:
 Errors for scope violations are only reported in  safe code.
Why? If I've explicitly designated a reference as scope, why should it be ignored in un- safe code?
Agreed, it should also work for any other code with some function to cast away scope. ref T unscope(scope ref T t) system { auto p = &t; return *p; }
Dec 04 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 3:21 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 Errors for scope violations are only reported in  safe code.
Why? If I've explicitly designated a reference as scope, why should it be ignored in un- safe code?
To interface to code that presents a safe interface, but does things under the hood that can't be verified.
Dec 04 2014
prev sibling next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69
Great stuff. Will this be possible, it's a fairly important use-case. scope ref T setVal(scope ref T t) { t.val = 12; return t; } Another question, how would a reference counted pointer take advantage of scope, i.e. avoid the increment/decrement when being passed to a function? One solution would be to add a function that returns a scoped reference to the underlying value. struct RefCounted(T) { scope ref T borrow() { return *p; } } Will it be possible to deduce, that the lifetime of that scoped value is tied to the smart pointer?
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 4:03 AM, Martin Nowak wrote:
 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69
Great stuff. Will this be possible, it's a fairly important use-case. scope ref T setVal(scope ref T t) { t.val = 12; return t; }
Yes, it would be written: scope ref T setVal(ref T t) { t.val = 12; return t; }
 Another question, how would a reference counted pointer take advantage of
scope,
 i.e. avoid the increment/decrement when being passed to a function?
 One solution would be to add a function that returns a scoped reference to the
 underlying value.
      struct RefCounted(T)
      {
          scope ref T borrow() { return *p; }
      }
 Will it be possible to deduce, that the lifetime of that scoped value is tied
to
 the smart pointer?
struct RefCounted(T) { T t; scope ref T borrow() { return t; } alias this t; } This enables RefCounted!T to be implicitly converted to a T, but with a scoped result. This is a critical feature, one I spent a lot of time thinking about, and hope it's right :-)
Dec 04 2014
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 3:41 PM, Walter Bright wrote:
 On 12/4/2014 4:03 AM, Martin Nowak wrote:
 Will it be possible to deduce, that the lifetime of that scoped value
 is tied to
 the smart pointer?
struct RefCounted(T) { T t; scope ref T borrow() { return t; } alias this t; } This enables RefCounted!T to be implicitly converted to a T, but with a scoped result. This is a critical feature, one I spent a lot of time thinking about, and hope it's right :-)
Hm... did you mean `alias this borrow`? -Steve
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:24 PM, Steven Schveighoffer wrote:
 Hm... did you mean `alias this borrow`?
yes (hangs head in shame)
Dec 04 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 On 12/4/2014 1:24 PM, Steven Schveighoffer wrote:
 Hm... did you mean `alias this borrow`?
yes (hangs head in shame)
No need to be ashamed for similar things, you are doing your best. Bye, bearophile
Dec 04 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 12:41:58PM -0800, Walter Bright via Digitalmars-d wrote:
[...]
   struct RefCounted(T)
   {
     T t;
     scope ref T borrow() { return t; }
     alias this t;
   }
 
 This enables RefCounted!T to be implicitly converted to a T, but with
 a scoped result. This is a critical feature, one I spent a lot of time
 thinking about, and hope it's right :-)
Hold on, how does this convert to T with a scoped result? Or did you mean: ... property scope ref T borrow() { ... } alias borrow this; ? T -- Let's call it an accidental feature. -- Larry Wall
Dec 04 2014
parent Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 04/12/2014 21:23, H. S. Teoh via Digitalmars-d wrote:
 	 property scope ref T borrow() { return t; }
 	alias borrow this;
While this DIP enabling the above to be memory-safe is awesome, a later tweak to AliasThis grammar to allow storage classes could make 'borrow' redundant. Plain alias now allows storage classes: AliasDeclaration: alias StorageClassesopt BasicType Declarator ; So: TweakedAliasThis: alias StorageClassesopt Identifier this ; Given ref and now scope will be storage classes, that would then allow just: alias scope ref t this;
Dec 06 2014
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 12/04/2014 09:41 PM, Walter Bright wrote:
 Yes, it would be written:

    scope ref T setVal(ref T t)
    {
       t.val = 12;
       return t;
    }
But when there is no scope on the argument, I could not call setVal with a local T variable.
Dec 04 2014
next sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 12/04/2014 10:54 PM, Martin Nowak wrote:
 On 12/04/2014 09:41 PM, Walter Bright wrote:
 Yes, it would be written:

    scope ref T setVal(ref T t)
    {
       t.val = 12;
       return t;
    }
But when there is no scope on the argument, I could not call setVal with a local T variable.
Ah, it's inferred. Makes sense now.
Dec 04 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:54 PM, Martin Nowak wrote:
 On 12/04/2014 09:41 PM, Walter Bright wrote:
 Yes, it would be written:

    scope ref T setVal(ref T t)
    {
       t.val = 12;
       return t;
    }
But when there is no scope on the argument, I could not call setVal with a local T variable.
Actually, you can. The difference between 'ref' and 'scope ref' parameters is the latter cannot be returned by ref.
Dec 04 2014
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
I am not expert in such things, but here are few comments and 
questions.

 Errors for scope violations are only reported in  safe code.
This seems acceptable only if the compiler switch "-scope" implies functions to be safe by default and system on request, because currently lot of D programmers don't apply annotations like safe to their D code. Opt-in safety doesn't work well (and in D we still have the problems caused by null pointers and references). - - - - - - - - - - Also consider an annotation like "!scope" or " escape" for the opposite purpose. - - - - - - - - - -
Delegates currently defensively allocate closures with the GC. 
Few actually escape, and with scope only those that actually 
escape need to have the closures allocated.<
So there's no need for extra annotations to make delegates nogc in most cases? - - - - - - - - - - Regarding array literals, some people proposed a syntax for fixed-size arrays to avoid heap-allocations (the "s" after the array literal): void foo(int[2]) {} void bar(int[]) {} void main() nogc { foo([1, 2]s); bar([1, 2]s); } Is DIP69 able to infer those arrays can be both stack-allocated? Is the "s" annotations still useful? - - - - - - - - - - Regarding the benefits, is escape analysis going to be used to allocate _automatically_ some small dynamic arrays and classes/structs on the stack instead of heap? And is escape analysis going to automatically give hints to the GC to deallocate faster GC-allocated memory that doesn't escape? void foo() { // Both dynamic arrays don't escape foo // Automatically stack allocated. auto a = new int[3]; // Heap-allocated but deterministically // deleted at the end of foo scope. auto b = new int[10_000]; } - - - - - - - - - - Bye, bearophile
Dec 04 2014
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
     foo([1, 2]s);
     bar([1, 2]s);
 }
Perhaps better: void foo(int[2]) {} void bar(scope int[]) {} void main() nogc { foo([1, 2]s); bar([1, 2]s); } Bye, bearophile
Dec 04 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 4:55 AM, bearophile wrote:
 I am not expert in such things, but here are few comments and questions.

 Errors for scope violations are only reported in  safe code.
This seems acceptable only if the compiler switch "-scope" implies functions to be safe by default and system on request, because currently lot of D programmers don't apply annotations like safe to their D code. Opt-in safety doesn't work well (and in D we still have the problems caused by null pointers and references).
Safety by default is outside of the sco[pe (!) of this discussion. Also, you can merely put: safe: at the beginning of a module and it is now all safe.
 Also consider an annotation like "!scope" or " escape" for the opposite
purpose.
That's part of a more general issue of negation of attributes.
 Delegates currently defensively allocate closures with the GC. Few actually
 escape, and with scope only those that actually escape need to have the
 closures allocated.<
So there's no need for extra annotations to make delegates nogc in most cases?
Marking delegates as nogc does not affect the closure, as the closure is allocated when the enclosing function is entered.
 Regarding array literals, some people proposed a syntax for fixed-size arrays
to
 avoid heap-allocations (the "s" after the array literal):

 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
      foo([1, 2]s);
      bar([1, 2]s);
 }


 Is DIP69 able to infer those arrays can be both stack-allocated? Is the "s"
 annotations still useful?
It's an idea worth exploring.
 Regarding the benefits, is escape analysis going to be used to allocate
 _automatically_ some small dynamic arrays and classes/structs on the stack
 instead of heap?
This is an optimization opportunity that scope enables.
 And is escape analysis going to automatically give hints to the GC to
deallocate
 faster GC-allocated memory that doesn't escape?

 void foo() {
      // Both dynamic arrays don't escape foo

      // Automatically stack allocated.
      auto a = new int[3];

      // Heap-allocated but deterministically
      // deleted at the end of foo scope.
      auto b = new int[10_000];
 }
Another optimization opportunity.
Dec 04 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 This seems acceptable only if the compiler switch "-scope" 
 implies functions to
 be  safe by default and  system on request, because currently 
 lot of D
 programmers don't apply annotations like  safe to their D 
 code. Opt-in safety
 doesn't work well (and in D we still have the problems caused 
 by null pointers and references).
Safety by default is outside of the sco[pe (!) of this discussion. Also, you can merely put: safe: at the beginning of a module and it is now all safe.
I agree that system code should allow to circumvent the part of the type system that enforces the scoping (this happens in Rust too) (but perhaps this has to happen only on scopes that gets inferred and not the ones specified explicitly). Perhaps you can't leave the " safe on default" issue out of the DIP69. Currently most D code I see around is not tagged with safe. If people don't bother applying that annotation, all the work you will do to implement DIP69 will be wasted or partially wasted. ------------------ From your answer to H. S. Teoh:
 Are there bug reports on this?
I and other people have opened several bug reports on safe, some of them are (but others are present, this is a subset): https://issues.dlang.org/show_bug.cgi?id=13615 https://issues.dlang.org/show_bug.cgi?id=13681 https://issues.dlang.org/show_bug.cgi?id=12948 https://issues.dlang.org/show_bug.cgi?id=13607 https://issues.dlang.org/show_bug.cgi?id=12845 https://issues.dlang.org/show_bug.cgi?id=6646 https://issues.dlang.org/show_bug.cgi?id=11176 https://issues.dlang.org/show_bug.cgi?id=6333 https://issues.dlang.org/show_bug.cgi?id=13054 https://issues.dlang.org/show_bug.cgi?id=13506 https://issues.dlang.org/show_bug.cgi?id=13188 Some examples of the problems (but there are also opposite problems, like in Issue 6646): void main() safe { import std.stdio, std.algorithm, std.bigint, std.typecons, std.array; [1, 2].sort!("a < b", SwapStrategy.stable); auto r = [1, 2].sort().release; writeln; BigInt a; a = a + a; alias Foo = Tuple!int; Foo[] data; data.remove!(x => x == Foo()); int[] b; auto c = b.capacity; b.schwartzSort!(x => x); const r2 = cartesianProduct([1], [1]).array; [Typedef!int(1)].array; } Output, dmd 2.067alpha: test.d(3,11): Error: safe function 'D main' cannot call system function 'std.algorithm.sort!("a < b", cast(SwapStrategy)2, int[]).sort' test.d(4,25): Error: safe function 'D main' cannot call system function 'std.range.SortedRange!(int[], "a < b").SortedRange.release' test.d(5,5): Error: safe function 'D main' cannot call system function 'std.stdio.writeln!().writeln' test.d(7,9): Error: safe function 'D main' cannot call system function 'std.bigint.BigInt.opBinary!("+", BigInt).opBinary' test.d(10,9): Error: safe function 'D main' cannot call system function 'test.main.remove!((x) => x == Foo(), cast(SwapStrategy)2, Tuple!int[]).remove' test.d(12,15): Error: safe function 'D main' cannot call system function 'object.capacity!int.capacity' test.d(13,6): Error: safe function 'D main' cannot call system function 'test.main.schwartzSort!((x) => x, "a < b", cast(SwapStrategy)0, int[]).schwartzSort' test.d(14,42): Error: safe function 'D main' cannot call system function 'std.array.array!(Result).array' test.d(15,21): Error: safe function 'D main' cannot call system function 'std.array.array!(Typedef!(int, 0, null)[]).array' Bye, bearophile
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 2:08 PM, bearophile wrote:
 Perhaps you can't leave the " safe on default" issue out of the DIP69.
Currently
 most D code I see around is not tagged with  safe. If people don't bother
 applying that annotation, all the work you will do to implement DIP69 will be
 wasted or partially wasted.
The attributes don't have much use for small programs. When a program is small enough that one person can keep it all in their head, there is not much use for it. I'd like to keep D usable for quick small program writing. Attributes become more and more useful the larger the program gets, and the more people are working on it. In such cases, I don't think it's near as much of a burden to put in a few annotations, and the payoff is significantly larger.
 I and other people have opened several bug reports on  safe, some of them are
 (but others are present, this is a subset):

 https://issues.dlang.org/show_bug.cgi?id=13615
 https://issues.dlang.org/show_bug.cgi?id=13681
 https://issues.dlang.org/show_bug.cgi?id=12948
 https://issues.dlang.org/show_bug.cgi?id=13607
 https://issues.dlang.org/show_bug.cgi?id=12845
 https://issues.dlang.org/show_bug.cgi?id=6646
 https://issues.dlang.org/show_bug.cgi?id=11176
 https://issues.dlang.org/show_bug.cgi?id=6333
 https://issues.dlang.org/show_bug.cgi?id=13054
 https://issues.dlang.org/show_bug.cgi?id=13506
 https://issues.dlang.org/show_bug.cgi?id=13188
Thanks you.
Dec 04 2014
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 Safety by default is outside of the sco[pe (!) of this 
 discussion. Also, you can merely put:

      safe:

 at the beginning of a module and it is now all safe.
If that topic is outside the scope of this discussion, then I have opened an ER on it: https://issues.dlang.org/show_bug.cgi?id=13838 Bye, bearophile
Dec 08 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 3:34 PM, bearophile wrote:
 If that topic is outside the scope of this discussion, then I have opened an ER
 on it:
 https://issues.dlang.org/show_bug.cgi?id=13838
1. You can always start a new thread here. 2. Thanks for filing the E.R. 3. When filing such, please check the [Hardware] settings. You have it set to [x86][Windows]. I reset them to [All][All].
Dec 09 2014
prev sibling next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 04 Dec 2014 12:55:34 +0000
bearophile via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 This seems acceptable only if the compiler switch "-scope"=20
 implies functions to be  safe by default and  system on request,=20
 because currently lot of D programmers don't apply annotations=20
 like  safe to their D code. Opt-in safety doesn't work well (and=20
 in D we still have the problems caused by null pointers and=20
 references).
actually, for me explicit attributes doesn't work well at all. and D compiler can't infer atributes for functions (ok, it can, but it can't apply that without changing function signature). that's why alot of my code compiles awfully slow with separate compilation: i'm used to write argument-less templates everywhere, so compiler can infer and apply attributes by itself. i even stopped commenting that practice, 'cause it means that much of the code will carry comments like this: // template to allow attribute inference void foo() (A a, int c) { ... } i think that something is very wrong with the function attributes. but i still can't think out how to improve that. but it still feels wrong. that's not about "D designers made a mistake", that's about "i want to invent a better thing!" ;-)
Dec 04 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Dec 05, 2014 at 03:16:36AM +0200, ketmar via Digitalmars-d wrote:
 On Thu, 04 Dec 2014 12:55:34 +0000
 bearophile via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 
 This seems acceptable only if the compiler switch "-scope" 
 implies functions to be  safe by default and  system on request, 
 because currently lot of D programmers don't apply annotations 
 like  safe to their D code. Opt-in safety doesn't work well (and 
 in D we still have the problems caused by null pointers and 
 references).
actually, for me explicit attributes doesn't work well at all. and D compiler can't infer atributes for functions (ok, it can, but it can't apply that without changing function signature). that's why alot of my code compiles awfully slow with separate compilation: i'm used to write argument-less templates everywhere, so compiler can infer and apply attributes by itself. i even stopped commenting that practice, 'cause it means that much of the code will carry comments like this: // template to allow attribute inference void foo() (A a, int c) { ... }
I've been considering to start adopting that practice too.
 i think that something is very wrong with the function attributes. but
 i still can't think out how to improve that. but it still feels wrong.
 
 that's not about "D designers made a mistake", that's about "i want to
 invent a better thing!" ;-)
In an ideal world, programmers would never have to know attributes even existed, the compiler would infer everything and optimize based on that. However, we're not quite there yet. :-) (And I don't know when we'll ever get there. Maybe D3. But we probably won't see that in the next 10 years... probably more.) I've often pondered about the possibility of a language where the compiler will analyze each module and infer any number of attributes and optimization opportunities for each symbol exported by that module, and this information will be saved in the object file (or some other kind of interfacing file). This includes any half-compiled template bodies and whatever else that can't be fully codegen'd until actual use. The attributes will include all sorts of stuff that programmers normally wouldn't want to deal with -- there could be 10+ or 50+ attributes representing various optimization / static checking opportunities. Then every time a module is imported by another module, the compiler never goes to the source code of the imported module anymore, but it will read the object (interface) file, which is fully attributed, and the saved attributes will be used internally for static checking, optimization, and inferring attributes for the current module. This way, the programmer not only gets the benefit of attributes without actually having to explicitly use them, but the language designers can also add many more attributes than you'd ever want to manually type out, or even remove old attributes that are no longer useful. The interface files will also serve as a convenient (and superior) substitute for .di files (uuugly) or .h files (even worse). T -- Right now I'm having amnesia and deja vu at the same time. I think I've forgotten this before.
Dec 04 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"H. S. Teoh via Digitalmars-d"  wrote in message 
news:mailman.2709.1417745546.9932.digitalmars-d puremagic.com...

 I've often pondered about the possibility of a language where the
 compiler will analyze each module and infer any number of attributes and
 optimization opportunities for each symbol exported by that module, and
 this information will be saved in the object file (or some other kind of
 interfacing file). This includes any half-compiled template bodies and
 whatever else that can't be fully codegen'd until actual use.  The
 attributes will include all sorts of stuff that programmers normally
 wouldn't want to deal with -- there could be 10+ or 50+ attributes
 representing various optimization / static checking opportunities.  Then
 every time a module is imported by another module, the compiler never
 goes to the source code of the imported module anymore, but it will read
 the object (interface) file, which is fully attributed, and the saved
 attributes will be used internally for static checking, optimization,
 and inferring attributes for the current module.
This can't be used to infer attributes that can produce errors - those attributes have to be user-visible or the errors don't make any sense. If it's purely for optimization, then that's basically what LTO does.
Dec 04 2014
parent "Tobias Pankrath" <tobias pankrath.net> writes:
On Friday, 5 December 2014 at 02:38:48 UTC, Daniel Murphy wrote:
 "H. S. Teoh via Digitalmars-d"  wrote in message 
 news:mailman.2709.1417745546.9932.digitalmars-d puremagic.com...

 I've often pondered about the possibility of a language where 
 the
 compiler will analyze each module and infer any number of 
 attributes and
 optimization opportunities for each symbol exported by that 
 module, and
 this information will be saved in the object file (or some 
 other kind of
 interfacing file). This includes any half-compiled template 
 bodies and
 whatever else that can't be fully codegen'd until actual use.  
 The
 attributes will include all sorts of stuff that programmers 
 normally
 wouldn't want to deal with -- there could be 10+ or 50+ 
 attributes
 representing various optimization / static checking 
 opportunities.  Then
 every time a module is imported by another module, the 
 compiler never
 goes to the source code of the imported module anymore, but it 
 will read
 the object (interface) file, which is fully attributed, and 
 the saved
 attributes will be used internally for static checking, 
 optimization,
 and inferring attributes for the current module.
This can't be used to infer attributes that can produce errors - those attributes have to be user-visible or the errors don't make any sense. If it's purely for optimization, then that's basically what LTO does.
We could inter attributes if not specified instead of assuming a default, for example nogc and a possible gc. --- int[] foo1(int[] bar) nogc // function is nogc, error if gc is used int[] foo2(int[] bar) gc // function is gc, functions that call foo2 cannot be nogc int[] foo3(int[] bar) // neither nogc nor gc, compiler infers attribute ---
Dec 05 2014
prev sibling next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 4 Dec 2014 18:10:18 -0800
"H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> wrote:

 This way, the programmer not only gets the benefit of attributes without
 actually having to explicitly use them, but the language designers can
 also add many more attributes than you'd ever want to manually type out,
 or even remove old attributes that are no longer useful. The interface
 files will also serve as a convenient (and superior) substitute for .di
 files (uuugly) or .h files (even worse).
and those interface files will ease creation of my pet project: dynamic component framework a-la BlackBox Component Builder. i'm already thinking about exporting semi-processed ASTs as ".mda" files, so i can write tools that bypasses parsing and checking phases. such files can be used as module interfaces, as syntax completion databases and so on. they can even be processed with some bytecode generator to improve CTFE. yet my project plan is to release pre-alpha version not sooner than in 2030, so i'm not really working on it with full speed. ;-)
Dec 04 2014
prev sibling next sibling parent reply "ixid" <nuaccount gmail.com> writes:
 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
     foo([1, 2]s);
     bar([1, 2]s);
 }
That is a rather unfriendly syntax, it is the kind that degenerates into noise with other structures.
Dec 05 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
ixid:

 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
    foo([1, 2]s);
    bar([1, 2]s);
 }
That is a rather unfriendly syntax, it is the kind that degenerates into noise with other structures.
Can you show an example of the noisy code it causes? And are you able to invent something succint that is better? Bye, bearophile
Dec 05 2014
parent "ixid" <nuaccount gmail.com> writes:
On Friday, 5 December 2014 at 14:10:44 UTC, bearophile wrote:
 ixid:

 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
   foo([1, 2]s);
   bar([1, 2]s);
 }
That is a rather unfriendly syntax, it is the kind that degenerates into noise with other structures.
Can you show an example of the noisy code it causes? And are you able to invent something succint that is better? Bye, bearophile
[1,2].stack stack [1,2] stack [1,2] [1,2]stack
Dec 05 2014
prev sibling parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 04/12/2014 12:55, bearophile wrote:
 Regarding array literals, some people proposed a syntax for fixed-size
 arrays to avoid heap-allocations (the "s" after the array literal):

 void foo(int[2]) {}
 void bar(scope int[]) {}
 void main()  nogc {
      foo([1, 2]s);
      bar([1, 2]s);
 }
I think even if the compiler could infer them as static arrays, it may still be useful to be explicit sometimes. We can already use a library template: template staticArray(items...) { import std.traits; alias T = CommonType!items[items.length]; enum T staticArray = [items]; } auto s = staticArray!(1, 2); static assert(is(typeof(s) == int[2])); bar(staticArray!(1, 2)); This might also make the proposed 'int[$] = [...];' syntax unnecessary. I think jmdavis once wrote something similar - although I've used enum here in case it helps avoid function template bloat.
Dec 07 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Nick Treleaven:

 This might also make the proposed 'int[$] = [...];' syntax 
 unnecessary.
Or might not. The [$] proposal is very refined. Bye, bearophile
Dec 07 2014
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69
How inout fits the picture? It has some scope semantics too. Section about expressions suggests you will tack scoping carefully, but then why would you need return by ref tricks, which don't rely on scope tracking and hence look hackable?
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 4:56 AM, Kagamin wrote:
 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69
How inout fits the picture? It has some scope semantics too.
s/inout/ref/
 Section about expressions suggests you will tack scoping carefully, but then
why
 would you need return by ref tricks, which don't rely on scope tracking and
 hence look hackable?
Couldn't think of another way.
Dec 04 2014
prev sibling next sibling parent reply "Daniel N" <ufo orbiting.us> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.
Looks pretty solid, but I have one question based on the following two statements: 1) "Scope is inferred for function parameters if not specified, under the same circumstances as pure, nothrow, nogc and safety are inferred." 2) "Scope is covariant, meaning it can be added to overriding functions." How would this be handled in the current proposal? class C { int bar(ref T); // <-- inferred to be scope } class D : C { override int bar(ref T); // <-- inferred to be NOT scope (but cannot remove scope when overriding) }
Dec 04 2014
parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
 How would this be handled in the current proposal?

 class C
 {
   int bar(ref T); // <-- inferred to be scope
 }

 class D : C
 {
   override int bar(ref T); // <-- inferred to be NOT scope (but 
 cannot remove scope when overriding)
 }
Attribute inference only works for function literals and template functions. So no inference is done for both methods. Template methods can not be overwritten.
Dec 04 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:49:06PM +0000, Tobias Pankrath via Digitalmars-d
wrote:
How would this be handled in the current proposal?

class C
{
  int bar(ref T); // <-- inferred to be scope
}

class D : C
{
  override int bar(ref T); // <-- inferred to be NOT scope (but cannot
remove scope when overriding)
}
Attribute inference only works for function literals and template functions. So no inference is done for both methods. Template methods can not be overwritten.
However, AFAIK, template *classes* trigger attribute inference on its (non-template) member functions, so this would be problematic: class Base(T) { T data; void method(ref T); // inferred to be scope } class Derived : Base!int { override void method(ref T); // oops, cannot override } T -- I think Debian's doing something wrong, `apt-get install pesticide', doesn't seem to remove the bugs on my system! -- Mike Dresser
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 10:45 AM, H. S. Teoh via Digitalmars-d wrote:
 However, AFAIK, template *classes* trigger attribute inference on its
 (non-template) member functions, so this would be problematic:

 	class Base(T) {
 		T data;
 		void method(ref T); // inferred to be scope
 	}

 	class Derived : Base!int {
 		override void method(ref T); // oops, cannot override
 	}
I agree, it's a good point. Scope inference cannot be done for virtual functions. I amended the DIP.
Dec 04 2014
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 5 December 2014 at 07:14, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 12/4/2014 10:45 AM, H. S. Teoh via Digitalmars-d wrote:
 However, AFAIK, template *classes* trigger attribute inference on its
 (non-template) member functions, so this would be problematic:

         class Base(T) {
                 T data;
                 void method(ref T); // inferred to be scope
         }

         class Derived : Base!int {
                 override void method(ref T); // oops, cannot override
         }
I agree, it's a good point. Scope inference cannot be done for virtual functions. I amended the DIP.
No comment...
Dec 05 2014
prev sibling next sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Walter Bright"  wrote in message news:m5p99m$luk$1 digitalmars.com...

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the missing 
 semantics for the 'scope' storage class in order to make it possible to 
 pass a reference to a function without it being possible for it to escape.

 This, among other things, makes a ref counting type practical. It also 
 makes it more practical to use other storage allocation schemes than 
 garbage collection.

 It does not make scope into a type constructor, nor a general 
 type-annotation system.

 It does not provide an ownership system, though it would complement one.
This looks really good. Nice work.
Dec 04 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 4:24 AM, Walter Bright wrote:
 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible for
 it to escape.

 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.

 It does not make scope into a type constructor, nor a general
 type-annotation system.

 It does not provide an ownership system, though it would complement one.
"There can be at most one owner for any piece of data." This doesn't seem right. For GC data, the GC owns the data, that is true. But for Ref-counted data, there is more than one owner, and only when all the owners disown the data can it be destroyed. I think there is a disconnect here, you can't say *nobody* owns the data, and if you say one variable owns the data, which one is it? Continuing to read... -Steve
Dec 04 2014
next sibling parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
 "There can be at most one owner for any piece of data."

 This doesn't seem right. For GC data, the GC owns the data, 
 that is true. But for Ref-counted data, there is more than one 
 owner, and only when all the owners disown the data can it be 
 destroyed.

 I think there is a disconnect here, you can't say *nobody* owns 
 the data, and if you say one variable owns the data, which one 
 is it?

 Continuing to read...

 -Steve
It is in line with this definition
 A variable owns the data it contains if, when the lifetime of 
 the variable is ended, the data can be destroyed.
and the definiton of the lifetime of a variable
 The lifetime of variables is based purely on their lexical 
 scope and order of declaration. The following rules define a 
 hierarchy of lifetimes:
A variable's lifetime starts at the point of its declaration, and ends with the lexical scope it is defined in. An (rvalue) expression's lifetime is temporary; it lives till the end of the statement that it appears in. The lifetime of A is higher than that of B, if A appears in a higher scope than B, or if both appear in the same scope, but A comes lexically before B. This matches the order of destruction of local variables. The lifetime of a function parameter is higher than that of that function's local variables, but lower than any variables in higher scopes. Because the lifetimes of any two variables are different by this definition and the definition of ownership links to variable lifetime, only one variable can own the data. So if you have multiple ref-counted slices to an array, no one owns the array until the ref-count goes down to 1. Question remains, if this is a definition of ownership we want to employ.
 There can be at most one owner for any piece of data.
Dec 04 2014
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 10:06 AM, Tobias Pankrath wrote:

 So if you have multiple ref-counted slices to an array, no one owns the
 array until the ref-count goes down to 1. Question remains, if this is a
 definition of ownership we want to employ.
This is like saying you have multiple owners :) I don't see the reason to avoid that. It's not like you can't define the owners when there are multiple references. Only one of them will end up destroying the variable, but it's not some other entity that will do it, you have a clear definition of who the owners are. -Steve
Dec 04 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 6:58 AM, Steven Schveighoffer wrote:
 This doesn't seem right. For GC data, the GC owns the data, that is true. But
 for Ref-counted data, there is more than one owner, and only when all the
owners
 disown the data can it be destroyed.

 I think there is a disconnect here, you can't say *nobody* owns the data, and
if
 you say one variable owns the data, which one is it?
The owner of the refcounted data is the refcounting wrapper - and the wrapper decides when to destroy the payload. (For GC, the "wrapper" is the GC system, which decides to destroy the payload when there are no longer references to it. Just like a ref counting system.)
Dec 04 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 4 December 2014 at 14:58:47 UTC, Steven 
Schveighoffer wrote:
 "There can be at most one owner for any piece of data."

 This doesn't seem right. For GC data, the GC owns the data, 
 that is true. But for Ref-counted data, there is more than one 
 owner, and only when all the owners disown the data can it be 
 destroyed.
The RC mechanism is the owner. Ownership is loosly defined in this DIp so that it do not close any door for future language evolution.
Dec 04 2014
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 5:48 PM, deadalnix wrote:
 On Thursday, 4 December 2014 at 14:58:47 UTC, Steven Schveighoffer wrote:
 "There can be at most one owner for any piece of data."

 This doesn't seem right. For GC data, the GC owns the data, that is
 true. But for Ref-counted data, there is more than one owner, and only
 when all the owners disown the data can it be destroyed.
The RC mechanism is the owner. Ownership is loosly defined in this DIp so that it do not close any door for future language evolution.
Well, actually the DIP is pretty rigid, it speaks only of ownership in terms of variables -- which variable owns a piece of data. It doesn't allow this kind of ownership via a concept or condition. I would change the DIP to reflect this clarification, if that is what is intended. -Steve
Dec 05 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 4:24 AM, Walter Bright wrote:
 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible for
 it to escape.

 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.

 It does not make scope into a type constructor, nor a general
 type-annotation system.

 It does not provide an ownership system, though it would complement one.
"Scope affects: local variables allocated on the stack" ... "scope int i; // scope is ignored because integers are not references and so are not views" I think I understand what you are trying to say -- the (big S) Scope of a local variable cannot escape, but it serves no purpose to declare a local int as (keyword) scope, since it's not going to be assigned any references. But it reads contradictory. -Steve
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 7:04 AM, Steven Schveighoffer wrote:
 "Scope affects:

      local variables allocated on the stack"

 ...

 "scope int i;       // scope is ignored because integers are not references and
 so are not views"

 I think I understand what you are trying to say -- the (big S) Scope of a local
 variable cannot escape, but it serves no purpose to declare a local int as
 (keyword) scope, since it's not going to be assigned any references.

 But it reads contradictory.
If you think that is contradictory, you should have read the earlier drafts :-) Basically, I had to get straight in my head the difference between the scope of the variable and the scope of its payload. The keyword 'scope' only affects the payload.
Dec 04 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
int* bar(scope int*);
scope int* foo();

bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

I'm trying to understand how foo can be implemented in any case. It has 
no scope ints to return, so where does it get the int from?

I don't see where the proposal defines what exactly can be returned via 
scope.

Another thing I saw early on:

void abc() {
     scope int* a;
     int* b;
     scope ref int* c = a;  // Error, rule 5
     scope ref int* d = b;  // Ok
     int* i = a;            // Ok, scope is inferred for i
     global_ptr = d;        // Error, lifetime(d) < lifetime(global_ptr)
     global_ptr = i;        // Error, lifetime(i) < lifetime(global_ptr)
     int* j;                // Ok, scope is inferred for i
     global_ptr = j;        // Ok, j is not scope
}

Does this mean ref can now be applied to a variable?

I'm not sure what the difference between scope ref and scope is. is d 
defined as a reference to b, or is d defined as a new variable that is 
initialized to what b points at (a la C++ &) ?

-Steve
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:
 int* bar(scope int*);
 scope int* foo();

 bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

 I'm trying to understand how foo can be implemented in any case. It has no
scope
 ints to return, so where does it get the int from?
Could be from a global variable. Or a new'd value.
 I don't see where the proposal defines what exactly can be returned via scope.
The scope return value does not affect what can be returned. It affects how that return value can be used. I.e. the return value cannot be used in such a way that it escapes the lifetime of the expression.
 Another thing I saw early on:

 void abc() {
      scope int* a;
      int* b;
      scope ref int* c = a;  // Error, rule 5
      scope ref int* d = b;  // Ok
      int* i = a;            // Ok, scope is inferred for i
      global_ptr = d;        // Error, lifetime(d) < lifetime(global_ptr)
      global_ptr = i;        // Error, lifetime(i) < lifetime(global_ptr)
      int* j;                // Ok, scope is inferred for i
      global_ptr = j;        // Ok, j is not scope
 }

 Does this mean ref can now be applied to a variable?
scope ref int p = x; is the same meaning as: foo(scope ref int p); foo(x); as far as what scope ref means. I found it convenient to use scope ref local variables to describe semantics. Whether we want to actually enable their usage hinges on if there's a point to it. Technically, it should work.
 I'm not sure what the difference between scope ref and scope is. is d defined
as
 a reference to b, or is d defined as a new variable that is initialized to what
 b points at (a la C++ &) ?
Scope refers to the payload. A scope ref is applying scope to the implied ref pointer. ref works like C++ &.
Dec 04 2014
next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 12:58:56PM -0800, Walter Bright via Digitalmars-d wrote:
 On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:
[...]
I don't see where the proposal defines what exactly can be returned
via scope.
The scope return value does not affect what can be returned. It affects how that return value can be used. I.e. the return value cannot be used in such a way that it escapes the lifetime of the expression.
[...] Ahhh, this certainly clears things up a lot. You should add this explanation to the DIP. T -- Famous last words: I *think* this will work...
Dec 04 2014
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 3:58 PM, Walter Bright wrote:
 On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:
 int* bar(scope int*);
 scope int* foo();

 bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

 I'm trying to understand how foo can be implemented in any case. It
 has no scope
 ints to return, so where does it get the int from?
Could be from a global variable. Or a new'd value.
Well, OK, but why do that?
 I don't see where the proposal defines what exactly can be returned
 via scope.
The scope return value does not affect what can be returned. It affects how that return value can be used. I.e. the return value cannot be used in such a way that it escapes the lifetime of the expression.
I assumed the scope return was so you could do things like: scope int *foo(scope int *x) { return x; } which would be fine, I assume, right?
 Another thing I saw early on:

 void abc() {
      scope int* a;
      int* b;
      scope ref int* c = a;  // Error, rule 5
      scope ref int* d = b;  // Ok
      int* i = a;            // Ok, scope is inferred for i
      global_ptr = d;        // Error, lifetime(d) < lifetime(global_ptr)
      global_ptr = i;        // Error, lifetime(i) < lifetime(global_ptr)
      int* j;                // Ok, scope is inferred for i
      global_ptr = j;        // Ok, j is not scope
 }

 Does this mean ref can now be applied to a variable?
scope ref int p = x; is the same meaning as: foo(scope ref int p); foo(x); as far as what scope ref means. I found it convenient to use scope ref local variables to describe semantics. Whether we want to actually enable their usage hinges on if there's a point to it. Technically, it should work.
My question was about how this kind of allows declaring a ref variable in the middle of a function, which was never allowed before. -Steve
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:32 PM, Steven Schveighoffer wrote:
 On 12/4/14 3:58 PM, Walter Bright wrote:
 On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:
 int* bar(scope int*);
 scope int* foo();

 bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

 I'm trying to understand how foo can be implemented in any case. It
 has no scope
 ints to return, so where does it get the int from?
Could be from a global variable. Or a new'd value.
Well, OK, but why do that?
Why would a programmer do that? I often ask that question! But the language allows it, therefore we must support it.
 I don't see where the proposal defines what exactly can be returned
 via scope.
The scope return value does not affect what can be returned. It affects how that return value can be used. I.e. the return value cannot be used in such a way that it escapes the lifetime of the expression.
I assumed the scope return was so you could do things like: scope int *foo(scope int *x) { return x; } which would be fine, I assume, right?
No. A scope parameter means the value does not escape the function. That means you can't return it.
 My question was about how this kind of allows declaring a ref variable in the
 middle of a function, which was never allowed before.
There's no technical reason it is disallowed - it's just that I didn't see a point to it.
Dec 05 2014
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/5/14 6:09 PM, Walter Bright wrote:
 On 12/4/2014 1:32 PM, Steven Schveighoffer wrote:
 On 12/4/14 3:58 PM, Walter Bright wrote:
 On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:
 int* bar(scope int*);
 scope int* foo();

 bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

 I'm trying to understand how foo can be implemented in any case. It
 has no scope
 ints to return, so where does it get the int from?
Could be from a global variable. Or a new'd value.
Well, OK, but why do that?
Why would a programmer do that? I often ask that question! But the language allows it, therefore we must support it.
But you're the programmer that did it, it's YOUR example! :) However, it's not just that, this is the ONLY example given as to why we support scope returns. The language allows it, therefore we must support it? But it's not allowed now, right? Why add it?
 The scope return value does not affect what can be returned. It affects
 how that return value can be used. I.e. the return value cannot be used
 in such a way that it escapes the lifetime of the expression.
I assumed the scope return was so you could do things like: scope int *foo(scope int *x) { return x; } which would be fine, I assume, right?
No. A scope parameter means the value does not escape the function. That means you can't return it.
I think you should eliminate scope returns then. They are not useful. I can't think of a single reason why a newly allocated via GC or global reference return should have to be restricted to exist only within the statement. Both have infinite lifetimes. -Steve
Dec 08 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 8 December 2014 at 15:12:51 UTC, Steven Schveighoffer 
wrote:
 I think you should eliminate scope returns then. They are not 
 useful. I can't think of a single reason why a newly allocated 
 via GC or global reference return should have to be restricted 
 to exist only within the statement. Both have infinite 
 lifetimes.
It's for references to objects that are owned by the function (or object of which the function is a method). These don't have infinite lifetimes.
Dec 08 2014
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/8/14 10:45 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:
 On Monday, 8 December 2014 at 15:12:51 UTC, Steven Schveighoffer wrote:
 I think you should eliminate scope returns then. They are not useful.
 I can't think of a single reason why a newly allocated via GC or
 global reference return should have to be restricted to exist only
 within the statement. Both have infinite lifetimes.
It's for references to objects that are owned by the function (or object of which the function is a method). These don't have infinite lifetimes.
Why not? An object is allocated on the heap, and has infinite lifetime. -Steve
Dec 08 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 8 December 2014 at 16:04:42 UTC, Steven Schveighoffer 
wrote:
 On 12/8/14 10:45 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:
 On Monday, 8 December 2014 at 15:12:51 UTC, Steven 
 Schveighoffer wrote:
 I think you should eliminate scope returns then. They are not 
 useful.
 I can't think of a single reason why a newly allocated via GC 
 or
 global reference return should have to be restricted to exist 
 only
 within the statement. Both have infinite lifetimes.
It's for references to objects that are owned by the function (or object of which the function is a method). These don't have infinite lifetimes.
Why not? An object is allocated on the heap, and has infinite lifetime.
"object" as in "instance of struct" ;-) And I was referring to objects owned by something with finite lifetime, e.g. a container. struct Array(T) { scope ref T opIndex(size_t); } Array!int container; ref x = container[42]; // no container[42]++; // yes writeln(container[42]); // yes
Dec 08 2014
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/8/14 11:27 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:
 On Monday, 8 December 2014 at 16:04:42 UTC, Steven Schveighoffer wrote:
 On 12/8/14 10:45 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?=
 <schuetzm gmx.net>" wrote:
 On Monday, 8 December 2014 at 15:12:51 UTC, Steven Schveighoffer wrote:
 I think you should eliminate scope returns then. They are not useful.
 I can't think of a single reason why a newly allocated via GC or
 global reference return should have to be restricted to exist only
 within the statement. Both have infinite lifetimes.
It's for references to objects that are owned by the function (or object of which the function is a method). These don't have infinite lifetimes.
Why not? An object is allocated on the heap, and has infinite lifetime.
"object" as in "instance of struct" ;-) And I was referring to objects owned by something with finite lifetime, e.g. a container. struct Array(T) { scope ref T opIndex(size_t); } Array!int container; ref x = container[42]; // no
Why not? x has the same lifetime as container! This is an incorrect arbitrary decision: module foo; Array!int arr; int *x; void bar() { x = &arr[0]; // should be ok. } A struct's member function has no idea of its lifetime, except what it's told. Now, it would make more sense for this: ref T opIndex(size_t); scope ref T opIndex(size_t) scope; But it's sounding to me (from Walter's other comments) like the latter function can't exist, because you can't return scope from scope? And furthermore, there's no way to overload based on scope? I'm really not liking this whole proposal, it either needs to be explained better, or rewritten. -Steve
Dec 08 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:24:13AM -0800, Walter Bright via Digitalmars-d wrote:
 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible
 for it to escape.
Finally! Thanks for the hard work, looking forward for this to be implemented (in some shape of form -- see comments below).
 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.
 
 It does not make scope into a type constructor, nor a general
 type-annotation system.
[...] Can we pretty please use the term "type qualifier" instead of "type constructor"? ;-) Anyway, here are a few comments: 1) Why are scope violations only reported for safe code?? IMO this greatly limits the usefulness of this DIP. If I mark something as scope, I'd expect it should be enforced by the compiler regardless of safe annotations, otherwise what's the point?? Currently, due to the incompleteness of safe, it's difficult to use safe annotations everywhere I'd like to (e.g. I need to use some un- safe Phobos functions that really ought to be safe, but aren't due to various reasons, like compiler limitations, etc.). This greatly limits the usefulness of this DIP, if scope is only enforced in safe code! 2) Is there a way to detect if something is marked as scope? If not, how does this proposal actually enable ref-counted types? (I'm assuming a library ref-counting type here; or are we expecting further compiler enhancements for ref counting?) 3) What does scope mean for delegate parameters? To what does the scope apply, the delegate itself, its body, or its return value, or ...? struct S { void opApply(scope void delegate(ref int) loopBody) { ... // what restrictions (might) apply here w.r.t. // how loopBody can be called? } } And what would be the effect on the caller's side? S s; foreach (i; s) { // what restrictions (might) apply here? } 4) Under "Expressions", how does scope interact with overloaded operators? Do the same rules apply to expressions that use overloaded operators as they apply to built-in operators, or do they apply as though the overloaded operators were written out in function-call syntax? What happens if some overloaded operators take a mix of scope and non-scope parameters? Finally, the following isn't directly related to this DIP, since scope is intended to solve this problem, but I thought I should bring it up. :-) In the section "escaping via return", 5 points are listed as sufficient for detecting the "return func(t);" case. The following section "scope ref" states that these 5 points are "correct" (in implementing escaping reference detection). However, isn't the following a loophole? struct S { int x; } ref int func(ref S s) { return s.x; } ref T foo() { S s; return func(s); // escaping reference } Since S cannot implicitly convert to int, it would appear that this circumvents escaping reference detection. (In fact, dmd happily accepts this code, even if everything is annotated with safe.) Note that the escaping reference to x can be arbitrarily deeply nested inside S, so it's non-trivial to decide whether there's no possibility for references to (parts of) S to leak from func(). T -- If I were two-faced, would I be wearing this one? -- Abraham Lincoln
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:
 Can we pretty please use the term "type qualifier" instead of "type
 constructor"? ;-)
I think "type constructor" is the more pedantically correct term, but you'll have to ask Andrei :-)
 1) Why are scope violations only reported for  safe code?? IMO this
 greatly limits the usefulness of this DIP.  If I mark something as
 scope, I'd expect it should be enforced by the compiler regardless of
  safe annotations, otherwise what's the point??
Because there are inevitably cases where you'll need to wrap unsafe code with a safe interface. By screwing this down too tightly for system code, you'll force people to use really ugly workarounds.
 Currently, due to the incompleteness of  safe, it's difficult to use
  safe annotations everywhere I'd like to (e.g. I need to use some
 un- safe Phobos functions that really ought to be  safe, but aren't due
 to various reasons, like compiler limitations, etc.). This greatly
 limits the usefulness of this DIP, if scope is only enforced in  safe
 code!
Are there bug reports on this?
 2) Is there a way to detect if something is marked as scope?
Using __traits(compiles,...) ought to work.
 If not, how
 does this proposal actually enable ref-counted types? (I'm assuming a
 library ref-counting type here; or are we expecting further compiler
 enhancements for ref counting?)
The ref counting would be done by a wrapper, which implicitly converts to the wrappee by exposing a scope ref.
 3) What does scope mean for delegate parameters? To what does the scope
 apply, the delegate itself, its body, or its return value, or ...?

 	struct S {
 		void opApply(scope void delegate(ref int) loopBody) {
 			...
 			// what restrictions (might) apply here w.r.t.
 			// how loopBody can be called?
 		}
 	}
Hmmm, looks like a problem I didn't think of. darnit!
 And what would be the effect on the caller's side?

 	S s;
 	foreach (i; s) {
 		// what restrictions (might) apply here?
 	}
i would get its scope inferred as necessary, so restrictions would apply as they would to a 'scope i'.
 4) Under "Expressions", how does scope interact with overloaded
 operators? Do the same rules apply to expressions that use overloaded
 operators as they apply to built-in operators, or do they apply as
 though the overloaded operators were written out in function-call
 syntax?  What happens if some overloaded operators take a mix of scope
 and non-scope parameters?
Overloaded operators are treated like calls to the functions that back them.
 Finally, the following isn't directly related to this DIP, since scope
 is intended to solve this problem, but I thought I should bring it up.
 :-) In the section "escaping via return", 5 points are listed as
 sufficient for detecting the "return func(t);" case. The following
 section "scope ref" states that these 5 points are "correct" (in
 implementing escaping reference detection). However, isn't the following
 a loophole?

 	struct S {
 		int x;
 	}
 	ref int func(ref S s) {
 		return s.x;
 	}
 	ref T foo() {
 		S s;
 		return func(s); // escaping reference
 	}

 Since S cannot implicitly convert to int, it would appear that this
 circumvents escaping reference detection.
No, because under this proposal, s.x is treated as if it were just s as far a scope rules are concerned.
 (In fact, dmd happily accepts
 this code, even if everything is annotated with  safe.)
I know, the current escape detection is quite inadequate - hence this proposal.
 Note that the
 escaping reference to x can be arbitrarily deeply nested inside S, so
 it's non-trivial to decide whether there's no possibility for references
 to (parts of) S to leak from func().
This proposal should lock that down.
Dec 04 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:10:31PM -0800, Walter Bright via Digitalmars-d wrote:
 On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:
Can we pretty please use the term "type qualifier" instead of "type
constructor"? ;-)
I think "type constructor" is the more pedantically correct term, but you'll have to ask Andrei :-)
Recently there was a bug filed by Andrei himself, and a bunch of merged PRs, that renamed "type constructor" to "type qualifier". The general sense I got was that we're deprecating "type constructor" in preference for "type qualifier".
1) Why are scope violations only reported for  safe code?? IMO this
greatly limits the usefulness of this DIP.  If I mark something as
scope, I'd expect it should be enforced by the compiler regardless of
 safe annotations, otherwise what's the point??
Because there are inevitably cases where you'll need to wrap unsafe code with a safe interface. By screwing this down too tightly for system code, you'll force people to use really ugly workarounds.
Are there any use cases for bypassing scope semantics in system code? How common are they? Maybe such code *should* look ugly -- to draw attention to the fact that something un- safe is going on. Or perhaps we can make cast() strip away scope? I much rather require explicit annotation for potentially unsafe operations, than to have subtle bugs creep into the code just because I forgot to mark something as safe.
Currently, due to the incompleteness of  safe, it's difficult to use
 safe annotations everywhere I'd like to (e.g. I need to use some
un- safe Phobos functions that really ought to be  safe, but aren't
due to various reasons, like compiler limitations, etc.). This
greatly limits the usefulness of this DIP, if scope is only enforced
in  safe code!
Are there bug reports on this?
There are, though probably incomplete, and there's also an ongoing stream of Phobos PR's for marking things safe that ought not to be system. There's a *lot* of Phobos code that needs to be cleaned up in this way before this can be workable. (Unless we abuse trusted as a temporary workaround -- but that's not an approach I'd like to recommend!)
2) Is there a way to detect if something is marked as scope?
Using __traits(compiles,...) ought to work.
I see.
If not, how does this proposal actually enable ref-counted types?
(I'm assuming a library ref-counting type here; or are we expecting
further compiler enhancements for ref counting?)
The ref counting would be done by a wrapper, which implicitly converts to the wrappee by exposing a scope ref.
Just saw your other post on this topic, makes sense. Still not sure how it would address the RC folks' request for compiler support, but I suppose that's a different topic.
3) What does scope mean for delegate parameters? To what does the
scope apply, the delegate itself, its body, or its return value, or
...?

	struct S {
		void opApply(scope void delegate(ref int) loopBody) {
			...
			// what restrictions (might) apply here w.r.t.
			// how loopBody can be called?
		}
	}
Hmmm, looks like a problem I didn't think of. darnit!
I think it's a fairly important issue, since making this work correctly will also open up optimization opportunities for a good chunk of delegate-based code.
And what would be the effect on the caller's side?

	S s;
	foreach (i; s) {
		// what restrictions (might) apply here?
	}
i would get its scope inferred as necessary, so restrictions would apply as they would to a 'scope i'.
I was thinking more of what implications may hold in the loop body for references to (local) variables outside the loop, since that presents another potential optimization opportunity if we do it right.
4) Under "Expressions", how does scope interact with overloaded
operators? Do the same rules apply to expressions that use overloaded
operators as they apply to built-in operators, or do they apply as
though the overloaded operators were written out in function-call
syntax?  What happens if some overloaded operators take a mix of
scope and non-scope parameters?
Overloaded operators are treated like calls to the functions that back them.
OK. You might want to state this explicitly in the DIP, just so it isn't left up to interpretation. :-)
Finally, the following isn't directly related to this DIP, since
scope is intended to solve this problem, but I thought I should bring
it up.  :-) In the section "escaping via return", 5 points are listed
as sufficient for detecting the "return func(t);" case. The following
section "scope ref" states that these 5 points are "correct" (in
implementing escaping reference detection). However, isn't the
following a loophole?

	struct S {
		int x;
	}
	ref int func(ref S s) {
		return s.x;
	}
	ref T foo() {
		S s;
		return func(s); // escaping reference
	}

Since S cannot implicitly convert to int, it would appear that this
circumvents escaping reference detection.
No, because under this proposal, s.x is treated as if it were just s as far a scope rules are concerned.
[...] I'm not quite sure I see how this could be implemented. When the compiler is compiling func(), say it's in a different module, there's nothing about the function that says it's illegal to return a reference to the member of an incoming argument. First of all, this isn't a template function so scope inference isn't in effect; secondly, even of it were in effect (say we write it as `func()(ref S s)` instead), it wouldn't be able to infer the return value as scope, since it's obviously escaping a reference to a function argument. So, unless I'm missing something obvious, func() will be accepted by the compiler (as it should, since we didn't mark anything as scope here). Now when the compiler compiles foo(), it can no longer see what's inside func(), so there's no way for it to know that the ref int being returned actually comes from the ref S parameter. Furthermore, func()'s signature is not annotated with scope, so how would the compiler know where to apply scoping rules? Unless I'm missing something, this still looks like a loophole. T -- If you compete with slaves, you become a slave. -- Norbert Wiener
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:52 PM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, Dec 04, 2014 at 01:10:31PM -0800, Walter Bright via Digitalmars-d
wrote:
 On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:
 Can we pretty please use the term "type qualifier" instead of "type
 constructor"? ;-)
I think "type constructor" is the more pedantically correct term, but you'll have to ask Andrei :-)
Recently there was a bug filed by Andrei himself, and a bunch of merged PRs, that renamed "type constructor" to "type qualifier". The general sense I got was that we're deprecating "type constructor" in preference for "type qualifier".
Looks like Andrei Hath Spaketh!
 Because there are inevitably cases where you'll need to wrap unsafe
 code with a safe interface. By screwing this down too tightly for
  system code, you'll force people to use really ugly workarounds.
Are there any use cases for bypassing scope semantics in system code?
One is to interface with legacy code that does not use 'scope' properly in its declarations. There are probably more.
 How common are they?
I don't know.
 Maybe such code *should* look ugly -- to draw
 attention to the fact that something un- safe is going on.
system is ugly enough.
 Or perhaps we can make cast() strip away scope? I much rather require
 explicit annotation for potentially unsafe operations, than to have
 subtle bugs creep into the code just because I forgot to mark something
 as  safe.
Start your modules with: safe:
 There are, though probably incomplete, and there's also an ongoing
 stream of Phobos PR's for marking things  safe that ought not to be
  system.  There's a *lot* of Phobos code that needs to be cleaned up in
 this way before this can be workable. (Unless we abuse  trusted as a
 temporary workaround -- but that's not an approach I'd like to
 recommend!)
As long as there's ongoing progress!
 Still not sure how it would address the RC folks' request for compiler
 support, but I suppose that's a different topic.
Yup, different issue.
 I think it's a fairly important issue, since making this work correctly
 will also open up optimization opportunities for a good chunk of
 delegate-based code.
See my other reply with resolution.
 I was thinking more of what implications may hold in the loop body for
 references to (local) variables outside the loop, since that presents
 another potential optimization opportunity if we do it right.
Yes. I think there are a number of optimization properties we can exploit over time.
 Overloaded operators are treated like calls to the functions that back
 them.
OK. You might want to state this explicitly in the DIP, just so it isn't left up to interpretation. :-)
Ok.
 Finally, the following isn't directly related to this DIP, since
 scope is intended to solve this problem, but I thought I should bring
 it up.  :-) In the section "escaping via return", 5 points are listed
 as sufficient for detecting the "return func(t);" case. The following
 section "scope ref" states that these 5 points are "correct" (in
 implementing escaping reference detection). However, isn't the
 following a loophole?

 	struct S {
 		int x;
 	}
 	ref int func(ref S s) {
 		return s.x;
 	}
 	ref T foo() {
 		S s;
 		return func(s); // escaping reference
 	}

 Since S cannot implicitly convert to int, it would appear that this
 circumvents escaping reference detection.
No, because under this proposal, s.x is treated as if it were just s as far a scope rules are concerned.
[...] I'm not quite sure I see how this could be implemented. When the compiler is compiling func(), say it's in a different module, there's nothing about the function that says it's illegal to return a reference to the member of an incoming argument.
That's right, so the compiler assumes it does, unless the parameter is marked as scope.
 First of all, this isn't a
 template function so scope inference isn't in effect; secondly, even of
 it were in effect (say we write it as `func()(ref S s)` instead), it
 wouldn't be able to infer the return value as scope, since it's
 obviously escaping a reference to a function argument.
'scope ref' parameters cannot be returned by 'ref' or by 'scope ref'. The only distinction between 'ref' and 'scope ref' parameters is the latter cannot be returned by 'ref' or 'scope ref'.
 So, unless I'm
 missing something obvious, func() will be accepted by the compiler (as
 it should, since we didn't mark anything as scope here).
The compiler assumes func(s) returns a ref to s, and so it won't allow func(s) to be returned from foo().
 Now when the compiler compiles foo(), it can no longer see what's inside
 func(), so there's no way for it to know that the ref int being returned
 actually comes from the ref S parameter.
It assumes it does, unless 'ref S' is marked 'scope ref S'. It's the whole point of 'scope ref'.
Dec 04 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 03:26:32PM -0800, Walter Bright via Digitalmars-d wrote:
 On 12/4/2014 1:52 PM, H. S. Teoh via Digitalmars-d wrote:
[...]
Are there any use cases for bypassing scope semantics in  system code?
One is to interface with legacy code that does not use 'scope' properly in its declarations. There are probably more.
How common are they?
I don't know.
Maybe such code *should* look ugly -- to draw attention to the fact
that something un- safe is going on.
system is ugly enough.
Not when all code is system by default!
Or perhaps we can make cast() strip away scope? I much rather require
explicit annotation for potentially unsafe operations, than to have
subtle bugs creep into the code just because I forgot to mark
something as  safe.
Start your modules with: safe:
I would, except that Phobos limitations preclude that. Someone else has already posted the list of bugs, so I won't repeat that here. Needless to say, it's still a ways off before putting safe: at the top of your modules will solve the problem. [...]
	struct S {
		int x;
	}
	ref int func(ref S s) {
		return s.x;
	}
	ref T foo() {
		S s;
		return func(s); // escaping reference
	}

Since S cannot implicitly convert to int, it would appear that this
circumvents escaping reference detection.
No, because under this proposal, s.x is treated as if it were just s as far a scope rules are concerned.
[...] I'm not quite sure I see how this could be implemented. When the compiler is compiling func(), say it's in a different module, there's nothing about the function that says it's illegal to return a reference to the member of an incoming argument.
That's right, so the compiler assumes it does, unless the parameter is marked as scope.
First of all, this isn't a template function so scope inference isn't
in effect; secondly, even of it were in effect (say we write it as
`func()(ref S s)` instead), it wouldn't be able to infer the return
value as scope, since it's obviously escaping a reference to a
function argument.
'scope ref' parameters cannot be returned by 'ref' or by 'scope ref'. The only distinction between 'ref' and 'scope ref' parameters is the latter cannot be returned by 'ref' or 'scope ref'.
So, unless I'm missing something obvious, func() will be accepted by
the compiler (as it should, since we didn't mark anything as scope
here).
The compiler assumes func(s) returns a ref to s, and so it won't allow func(s) to be returned from foo().
Ah, I see. So you're saying that under this DIP, `return func(...);` from a ref function will be prohibited unless the arguments to func() were all either scope or non-reference? That sounds good, though it *is* pretty invasive, as any existing ref functions that do this will need to be revised, which potentially spreads 'scope' to many other places, depending on how deeply the chain of tail recursion / tail func calls go.
Now when the compiler compiles foo(), it can no longer see what's
inside func(), so there's no way for it to know that the ref int
being returned actually comes from the ref S parameter.
It assumes it does, unless 'ref S' is marked 'scope ref S'. It's the whole point of 'scope ref'.
Got it, thanks. T -- Shin: (n.) A device for finding furniture in the dark.
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 3:54 PM, H. S. Teoh via Digitalmars-d wrote:
 The compiler assumes func(s) returns a ref to s, and so it won't allow
 func(s) to be returned from foo().
Ah, I see. So you're saying that under this DIP, `return func(...);` from a ref function will be prohibited unless the arguments to func() were all either scope or non-reference?
Right.
 That sounds good, though it *is* pretty invasive, as any existing ref
 functions that do this will need to be revised, which potentially
 spreads 'scope' to many other places, depending on how deeply the chain
 of tail recursion / tail func calls go.
I know: 1. that's why the extensive support for scope inference 2. I couldn't see any other way
Dec 04 2014
parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 04:33:22PM -0800, Walter Bright via Digitalmars-d wrote:
 On 12/4/2014 3:54 PM, H. S. Teoh via Digitalmars-d wrote:
The compiler assumes func(s) returns a ref to s, and so it won't allow
func(s) to be returned from foo().
Ah, I see. So you're saying that under this DIP, `return func(...);` from a ref function will be prohibited unless the arguments to func() were all either scope or non-reference?
Right.
That sounds good, though it *is* pretty invasive, as any existing ref
functions that do this will need to be revised, which potentially
spreads 'scope' to many other places, depending on how deeply the
chain of tail recursion / tail func calls go.
I know: 1. that's why the extensive support for scope inference 2. I couldn't see any other way
I'm not saying it's necessary a bad thing... just that it will be an invasive change that we should be prepared to deal with. T -- English is useful because it is a mess. Since English is a mess, it maps well onto the problem space, which is also a mess, which we call reality. Similarly, Perl was designed to be a mess, though in the nicest of all possible ways. -- Larry Wall
Dec 04 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:52:09PM -0800, H. S. Teoh via Digitalmars-d wrote:
 On Thu, Dec 04, 2014 at 01:10:31PM -0800, Walter Bright via Digitalmars-d
wrote:
 On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:
Can we pretty please use the term "type qualifier" instead of "type
constructor"? ;-)
I think "type constructor" is the more pedantically correct term, but you'll have to ask Andrei :-)
Recently there was a bug filed by Andrei himself, and a bunch of merged PRs, that renamed "type constructor" to "type qualifier". The general sense I got was that we're deprecating "type constructor" in preference for "type qualifier".
[...] FYI: https://issues.dlang.org/show_bug.cgi?id=13671 T -- "A man's wife has more power over him than the state has." -- Ralph Emerson
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:56 PM, H. S. Teoh via Digitalmars-d wrote:
 FYI: https://issues.dlang.org/show_bug.cgi?id=13671
Ok, ok, I fixed the DIP :-)
Dec 04 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:10 PM, Walter Bright wrote:
 On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:
 3) What does scope mean for delegate parameters? To what does the scope
 apply, the delegate itself, its body, or its return value, or ...?

     struct S {
         void opApply(scope void delegate(ref int) loopBody) {
             ...
             // what restrictions (might) apply here w.r.t.
             // how loopBody can be called?
         }
     }
Hmmm, looks like a problem I didn't think of. darnit!
Turns out, 'ref' has exactly the same issue. The resolution is the same: alias ref int delegate() dg_t; // ref applies to return type void foo(dg_t dg) { static int i; ref int bar() { return i; } // ref applies to return type dg = &bar; dg() = 3; } versus: void foo(ref int deletate() dg) { // ref applies to declaration dg int bar() { return 3; } dg = &bar; } Replace 'ref' with 'scope'.
Dec 04 2014
parent reply "Basile Burg" <basile.burg. gmx.com> writes:
 Hmmm, looks like a problem I didn't think of. darnit!
Turns out, 'ref' has exactly the same issue. The resolution is the same: alias ref int delegate() dg_t; // ref applies to return type
Was the goal of the post to represent the new alias syntax ? If so, big fail, Walter,...
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 3:58 PM, Basile Burg wrote:
 Was the goal of the post to represent the new alias syntax ?
??
Dec 04 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.

 This, among other things, makes a ref counting type practical. 
 It also makes it more practical to use other storage allocation 
 schemes than garbage collection.

 It does not make scope into a type constructor, nor a general 
 type-annotation system.

 It does not provide an ownership system, though it would 
 complement one.
So as mentioned, there are various problem with this DIP : - rvalue are defined as having a scope that goes to the end of the statement. That mean they can never be assigned to anything as per spec. - It add more special casing with & (as if it wasn't enough of a mess with property, optional () and the fact the function aren't first class). For instance *e has infinite lifetime when &(*e) is lifetime(e). Also, considering *e has infinite lifetime, (but not e[i] ???) what is the point of scope at all ? If all indirection goes to infinite lifetime (ie GC) then lifetime only apply to local variable and it should not be able to escape these, scope or not. During discussion, I proposed to differentiate lifetime calculation between lvalues and rvalues (which are inherently different beasts with different lifetime) and carry (or not) the scope flag with each expression.
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 3:04 PM, deadalnix wrote:
 So as mentioned, there are various problem with this DIP :
   - rvalue are defined as having a scope that goes to the end of the statement.
 That mean they can never be assigned to anything as per spec.
I don't believe this is correct. Rvalues can be assigned, just like: __gshared int x; { int i; x = i; } i's scope ends at the } but it can still be assigned to x.
   - It add more special casing with & (as if it wasn't enough of a mess with
  property, optional () and the fact the function aren't first class). For
 instance *e has infinite lifetime when &(*e) is lifetime(e).
That's right. I know you're worried about that, but I still don't see it as an actual problem. (The optimizer makes use of this special case all the time.)
 Also, considering *e has infinite lifetime, (but not e[i] ???)
e[i] should be same as *e, unless e is a literal or tuple.
 what is the point of scope at all ? If all indirection goes to infinite
lifetime (ie GC) then
 lifetime only apply to local variable and it should not be able to escape
these,
 scope or not.
I originally had scope only apply to ref, but that made having scoped classes impossible.
 During discussion, I proposed to differentiate lifetime calculation between
 lvalues and rvalues (which are inherently different beasts with different
 lifetime) and carry (or not) the scope flag with each expression.
I'm not sure how that would be different from the DIP as it stands now.
Dec 04 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 04:31:18PM -0800, Walter Bright via Digitalmars-d wrote:
 On 12/4/2014 3:04 PM, deadalnix wrote:
So as mentioned, there are various problem with this DIP :
  - rvalue are defined as having a scope that goes to the end of the statement.
That mean they can never be assigned to anything as per spec.
I don't believe this is correct. Rvalues can be assigned, just like: __gshared int x; { int i; x = i; } i's scope ends at the } but it can still be assigned to x.
[...] Well, the "Expressions" section defines certain lifetimes to be infinity, however, it contradicts the definition of an rvalue's lifetime given under "Definitions" - "Lifetime", which states that an rvalue's lifetime lasts only to the end of the statement. This would imply that the following ought to be rejected: int x = 1 + 2; // lifetime(1 + 2) = end of statement // but lifetime(x) = infinity Sounds like the Definitions section is not precise enough. :-) T -- Curiosity kills the cat. Moral: don't be the cat.
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 4:48 PM, H. S. Teoh via Digitalmars-d wrote:
  Well, the "Expressions" section defines certain lifetimes to be
 infinity, however, it contradicts the definition of an rvalue's lifetime
 given under "Definitions" - "Lifetime", which states that an rvalue's
 lifetime lasts only to the end of the statement. This would imply that
 the following ought to be rejected:

 	int x = 1 + 2; // lifetime(1 + 2) = end of statement
 			// but lifetime(x) = infinity

 Sounds like the Definitions section is not precise enough. :-)
An 'int' is not a view!
Dec 04 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 5 December 2014 at 00:32:32 UTC, Walter Bright wrote:
 On 12/4/2014 3:04 PM, deadalnix wrote:
 So as mentioned, there are various problem with this DIP :
  - rvalue are defined as having a scope that goes to the end 
 of the statement.
 That mean they can never be assigned to anything as per spec.
I don't believe this is correct. Rvalues can be assigned, just like: __gshared int x; { int i; x = i; } i's scope ends at the } but it can still be assigned to x.
It work even better when i has indirections.
  - It add more special casing with & (as if it wasn't enough 
 of a mess with
  property, optional () and the fact the function aren't first 
 class). For
 instance *e has infinite lifetime when &(*e) is lifetime(e).
That's right. I know you're worried about that, but I still don't see it as an actual problem. (The optimizer makes use of this special case all the time.)
Yes, this is the job of the optimizer to do this kind of stunt. Not the semantic analysis.
 Also, considering *e has infinite lifetime, (but not e[i] ???)
e[i] should be same as *e, unless e is a literal or tuple.
Ho I missed the *. Sorry for that.
 what is the point of scope at all ? If all indirection goes to 
 infinite lifetime (ie GC) then
 lifetime only apply to local variable and it should not be 
 able to escape these,
 scope or not.
I originally had scope only apply to ref, but that made having scoped classes impossible.
Promoting scoped class on stack is an ownership problem, and out of scope (!). It make sense to allow it as an optimization. Problem is, lifetime goes to infinite after indirection, so I'm not sure what the guarantee is.
 During discussion, I proposed to differentiate lifetime 
 calculation between
 lvalues and rvalues (which are inherently different beasts 
 with different
 lifetime) and carry (or not) the scope flag with each 
 expression.
I'm not sure how that would be different from the DIP as it stands now.
I cause everything reached through the view to be scope and obliviate the need for things like &(*e) having special meaning.
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 6:56 PM, deadalnix wrote:
 On Friday, 5 December 2014 at 00:32:32 UTC, Walter Bright wrote:
 On 12/4/2014 3:04 PM, deadalnix wrote:
 So as mentioned, there are various problem with this DIP :
  - rvalue are defined as having a scope that goes to the end of the statement.
 That mean they can never be assigned to anything as per spec.
I don't believe this is correct. Rvalues can be assigned, just like: __gshared int x; { int i; x = i; } i's scope ends at the } but it can still be assigned to x.
It work even better when i has indirections.
I understand what you're driving at, but only a scoped rvalue would not be copyable.
  - It add more special casing with & (as if it wasn't enough of a mess with
  property, optional () and the fact the function aren't first class). For
 instance *e has infinite lifetime when &(*e) is lifetime(e).
That's right. I know you're worried about that, but I still don't see it as an actual problem. (The optimizer makes use of this special case all the time.)
Yes, this is the job of the optimizer to do this kind of stunt. Not the semantic analysis.
I don't see any other way, nor do I see the practical problem.
 I originally had scope only apply to ref, but that made having scoped classes
 impossible.
Promoting scoped class on stack is an ownership problem, and out of scope (!). It make sense to allow it as an optimization. Problem is, lifetime goes to infinite after indirection, so I'm not sure what the guarantee is.
The guarantee is there will be no references to the class instance after the scoped class goes out of scope.
 During discussion, I proposed to differentiate lifetime calculation between
 lvalues and rvalues (which are inherently different beasts with different
 lifetime) and carry (or not) the scope flag with each expression.
I'm not sure how that would be different from the DIP as it stands now.
I cause everything reached through the view to be scope and obliviate the need for things like &(*e) having special meaning.
Are you suggesting transitive scope?
Dec 05 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 5 December 2014 at 21:32:53 UTC, Walter Bright wrote:
 I don't believe this is correct. Rvalues can be assigned, 
 just like:

   __gshared int x;
   { int i; x = i; }

 i's scope ends at the } but it can still be assigned to x.
It work even better when i has indirections.
I understand what you're driving at, but only a scoped rvalue would not be copyable.
The DIP say nothing about scoped rvalue having different behavior than non scoped ones.
 I originally had scope only apply to ref, but that made
 having scoped classes
 impossible.
Promoting scoped class on stack is an ownership problem, and out of scope (!). It make sense to allow it as an optimization. Problem is, lifetime goes to infinite after indirection, so I'm not sure what the guarantee is.
The guarantee is there will be no references to the class instance after the scoped class goes out of scope.
Through use of that view. I see it as follow: - When unconsumed owner goes out of scope, it can (must ?) be free automatically. - scope uses do not consume. - When the compiler see a pair of alloc/free, it can promote on stack. That is a much more useful definition as it allow for stack promotion after inlining: class FooBuilder { Foo build() { return new Foo(); } } class Foo {} void bar() { auto f = new FooBuilder().build(); // Use f and do not consume... } This can be reduced in such a way no allocation happen at all.
 I cause everything reached through the view to be scope and
 obliviate the need for things like &(*e) having special 
 meaning.
Are you suggesting transitive scope?
For rvalues, yes. Not for lvalues.
Dec 05 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2014 3:59 PM, deadalnix wrote:
 On Friday, 5 December 2014 at 21:32:53 UTC, Walter Bright wrote:
 I don't believe this is correct. Rvalues can be assigned, just like:

   __gshared int x;
   { int i; x = i; }

 i's scope ends at the } but it can still be assigned to x.
It work even better when i has indirections.
I understand what you're driving at, but only a scoped rvalue would not be copyable.
The DIP say nothing about scoped rvalue having different behavior than non scoped ones.
Can you propose some new wording?
 I originally had scope only apply to ref, but that made
 having scoped classes
 impossible.
Promoting scoped class on stack is an ownership problem, and out of scope (!). It make sense to allow it as an optimization. Problem is, lifetime goes to infinite after indirection, so I'm not sure what the guarantee is.
The guarantee is there will be no references to the class instance after the scoped class goes out of scope.
Through use of that view. I see it as follow: - When unconsumed owner goes out of scope, it can (must ?) be free automatically. - scope uses do not consume. - When the compiler see a pair of alloc/free, it can promote on stack. That is a much more useful definition as it allow for stack promotion after inlining: class FooBuilder { Foo build() { return new Foo(); } } class Foo {} void bar() { auto f = new FooBuilder().build(); // Use f and do not consume... } This can be reduced in such a way no allocation happen at all.
Yes, but I think the proposal allows for that.
 I cause everything reached through the view to be scope and
 obliviate the need for things like &(*e) having special meaning.
Are you suggesting transitive scope?
For rvalues, yes. Not for lvalues.
I don't think that is workable.
Dec 06 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 6 December 2014 at 21:57:03 UTC, Walter Bright wrote:
 On 12/5/2014 3:59 PM, deadalnix wrote:
 The DIP say nothing about scoped rvalue having different 
 behavior
 than non scoped ones.
Can you propose some new wording?
I did: An infinite lifetime is a lifetime greater or equal than any other lifetime. Expression of infinite lifetime are: - literals - GC heap allocated objects - statics and enums. - rvalues of type that do not contain indirections. - non scope rvalues. Dereference share the lifetime of the dereferenced expression (ie infinite lifetime unless the expression is scope). Address of expression shared the lifetime of the base expression, and in addition gain the scope flag.
 Are you suggesting transitive scope?
For rvalues, yes. Not for lvalues.
I don't think that is workable.
That is the only way. For rvalue, you got to take the lower possible lifetime, but for lvalue, you want to consider the highest possible lifetime.
Dec 08 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 4:24 AM, Walter Bright wrote:
 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible for
 it to escape.

 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.

 It does not make scope into a type constructor, nor a general
 type-annotation system.

 It does not provide an ownership system, though it would complement one.
Can we take a step back here? I read many people's comments and I understand only about half of them. Can someone who knows what this new feature is supposed to do give some Ali Çehreli-like description on the feature? Basically, let's strip out the *proof* in the DIP (the how it works and why we have it), and focus on how it is to be used. I still am having a hard time wrapping my head around the benefits and when to use scope, scope ref, why I would use it. I'm worried that we are adding all this complication and it will confuse the shit out of users, to the point where they won't use it. -Steve
Dec 05 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2014 7:27 AM, Steven Schveighoffer wrote:
 Can someone who knows what this new feature is supposed to do give some Ali
 Çehreli-like description on the feature? Basically, let's strip out the
*proof*
 in the DIP (the how it works and why we have it), and focus on how it is to be
 used.

 I still am having a hard time wrapping my head around the benefits and when to
 use scope, scope ref, why I would use it. I'm worried that we are adding all
 this complication and it will confuse the shit out of users, to the point where
 they won't use it.
The tl;dr version is when a declaration is tagged with 'scope', the contents of that variable will not escape the lifetime of that declaration. It means that this code will be safe: void foo(scope int* p); p = malloc(n); foo(p); free(p); The rest is all the nuts and bolts of making that work.
Dec 05 2014
next sibling parent reply "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Friday, 5 December 2014 at 20:55:55 UTC, Walter Bright wrote:
 On 12/5/2014 7:27 AM, Steven Schveighoffer wrote:
 Can someone who knows what this new feature is supposed to do 
 give some Ali
 Çehreli-like description on the feature? Basically, let's 
 strip out the *proof*
 in the DIP (the how it works and why we have it), and focus on 
 how it is to be
 used.

 I still am having a hard time wrapping my head around the 
 benefits and when to
 use scope, scope ref, why I would use it. I'm worried that we 
 are adding all
 this complication and it will confuse the shit out of users, 
 to the point where
 they won't use it.
The tl;dr version is when a declaration is tagged with 'scope', the contents of that variable will not escape the lifetime of that declaration. It means that this code will be safe: void foo(scope int* p); p = malloc(n); foo(p); free(p); The rest is all the nuts and bolts of making that work.
<brainstorm> What about also adding the inverse of scope? Then scope can be inferred. As in: ``` void foo(int* p); void free(P)(consume P* p); p = malloc(n); foo(p); // here foo() gets scope, since free consumes it. free(p); ``` So you do not need to write scope everywhere. And it would prevent this code: ``` { free(p); free(p); // Error: p already consumed } ``` </brainstorm>
Dec 05 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 6 December 2014 at 04:31:48 UTC, Sebastiaan Koppe 
wrote:
 What about also adding the inverse of scope? Then scope can be 
 inferred. As in:

 ```
 void foo(int* p);
 void free(P)(consume P* p);
Yes, this is much better. When I suggested it, it was rejected because D is too concerned about breaking existing code. Which is a not-very-good argument since this breaking change is concervative (you only have to add "consume" or something similar when the compiler complains). The obvious solution is to do as you suggest and in addition do all safe analysis on a high level IR layer using dataflow through and through. Instead D continues down the rather flimsy path of partially addressing these issues in the type system… which will lead to a more complicated and less complete solution where safe basically continues to be a leaky cauldron…
Dec 06 2014
next sibling parent reply "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Saturday, 6 December 2014 at 12:38:24 UTC, Ola Fosheim Grøstad 
wrote:
 On Saturday, 6 December 2014 at 04:31:48 UTC, Sebastiaan Koppe 
 wrote:
 What about also adding the inverse of scope? Then scope can be 
 inferred. As in:

 ```
 void foo(int* p);
 void free(P)(consume P* p);
Yes, this is much better. When I suggested it, it was rejected because D is too concerned about breaking existing code. Which is a not-very-good argument since this breaking change is concervative (you only have to add "consume" or something similar when the compiler complains).
Hmm, I see. Whenever I suggest to break something at my job - which, admittedly is much easier there than in a language - I get faced with some serious reluctance. My argument is always the same: It is going to make you happy, eventually. How would it break anything though? Wouldn't functions eligible for `consume` already have the programmer ensuring the arguments haven't escaped before/after the function call? In case they did a bad job - sure it would break - but it would have been a bug.
 The obvious solution is to do as you suggest and in addition do 
 all  safe analysis on a high level IR layer using dataflow 
 through and through.
I am a big proponent of dataflow analyses, but I got the feeling people think is it pretty hard. Couldn't find much detailed papers on it, so I don't know.
Dec 06 2014
next sibling parent "eles" <eles eles.com> writes:
On Sunday, 7 December 2014 at 06:52:38 UTC, Sebastiaan Koppe 
wrote:
 On Saturday, 6 December 2014 at 12:38:24 UTC, Ola Fosheim 
 Grøstad wrote:
 On Saturday, 6 December 2014 at 04:31:48 UTC, Sebastiaan Koppe 
 wrote:
 I am a big proponent of dataflow analyses, but I got the 
 feeling people think is it pretty hard.
I think D is in a good position to use the information available in assert* and in contracts for that. Other languages have to use dedicated tools for that. In a latter step, D could even formalize this into, as discussed in other threads, VRP sub-language. But, for the time being, the leverage offered by assert and in /out is available there to be used. *yes, I know the war between assert and assume
Dec 07 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 7 December 2014 at 06:52:38 UTC, Sebastiaan Koppe 
wrote:
 How would it break anything though? Wouldn't functions eligible 
 for `consume` already have the programmer ensuring the 
 arguments haven't escaped before/after the function call? In 
 case they did a bad job - sure it would break - but it would 
 have been a bug.
I don't think the breakage is a serious problem in this case, so I obviously agree with you… But, the deep-rooted problem is that you actually need different properties associated with pointers to do this properly: unique, single-threaded shared, multi-threaded shared, gc, non-owning… And this could be resolved with templates and having the compiler recognize pointer resolutions that lead to the same lower level code (to avoid bloat). But the D authors don't want ownership related pointertypes… and I can't see that work.
 I am a big proponent of dataflow analyses, but I got the 
 feeling people think is it pretty hard. Couldn't find much 
 detailed papers on it, so I don't know.
You can do it in a simpler and conservative fashion and accept false positives (or negatives) so that you here-and-there have to inject annotations like this: "trust me I have proven that this property holds here", then let the compiler infer the rest. With a good IDE this should be no biggie: just inject suggestions in the code and let the programmer confirm/refine them. If you want to guarantee memory-safety you should do it with a single uniform mechanism, trying to do it bit-by-bit special casing is not a good idea. So given the current approach, I'm inclined to turn off safe, make everything nogc, write my own libraries with unique_ptr/shared_ptr and call it a day… The current approach does not address situations where I run into memory related bugs in my own code.
Dec 07 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 6 December 2014 at 12:38:24 UTC, Ola Fosheim Grøstad
wrote:
 On Saturday, 6 December 2014 at 04:31:48 UTC, Sebastiaan Koppe 
 wrote:
 What about also adding the inverse of scope? Then scope can be 
 inferred. As in:

 ```
 void foo(int* p);
 void free(P)(consume P* p);
Yes, this is much better. When I suggested it, it was rejected because D is too concerned about breaking existing code. Which is a not-very-good argument since this breaking change is concervative (you only have to add "consume" or something similar when the compiler complains). The obvious solution is to do as you suggest and in addition do all safe analysis on a high level IR layer using dataflow through and through. Instead D continues down the rather flimsy path of partially addressing these issues in the type system… which will lead to a more complicated and less complete solution where safe basically continues to be a leaky cauldron…
This is inherently about ownership. I have a proposal about this. Scope is about using things without ownership. Both are linked but different beast.
Dec 08 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 8 December 2014 at 23:00:05 UTC, deadalnix wrote:
 This is inherently about ownership. I have a proposal about 
 this.
 Scope is about using things without ownership.

 Both are linked but different beast.
Not really different. The activation record (stack frame) is conceptually an object. Scoped objects are owned by the activation record. You have the same problem when handing out references to parts of composite objects. When propagating references down a call chain you can keep track of the associated call-depth, when the call-depth associated with the reference is greater than 0, then it safe to return the reference from a function. Right?
Dec 09 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 9 December 2014 at 19:39:31 UTC, Ola Fosheim Grøstad
wrote:
 On Monday, 8 December 2014 at 23:00:05 UTC, deadalnix wrote:
 This is inherently about ownership. I have a proposal about 
 this.
 Scope is about using things without ownership.

 Both are linked but different beast.
Not really different. The activation record (stack frame) is conceptually an object. Scoped objects are owned by the activation record. You have the same problem when handing out references to parts of composite objects. When propagating references down a call chain you can keep track of the associated call-depth, when the call-depth associated with the reference is greater than 0, then it safe to return the reference from a function. Right?
That why i say they are linked. I don't think your way of stating it contradict what I said. scope allow for manipulation of data without owning them. Whatever the owner is (be it the stack frame or anything else) doesn't really matter here.
Dec 09 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 9 December 2014 at 21:58:48 UTC, deadalnix wrote:
 That why i say they are linked. I don't think your way of 
 stating
 it contradict what I said.

 scope allow for manipulation of data without owning them.
 Whatever the owner is (be it the stack frame or anything else)
 doesn't really matter here.
It does when you return parts of it, like if you pass in a binary tree and return a node. Which you have to be able to do for the concept to make sense. When you solve issues in language design you should address the hard issues first, because the easy issues will then tend to resolve themselves if the design is good. If you start with "the low hanging fruit" you end up with pointless special casing… D is already in that landscape and should try hard not to sink deeper into the muddy waters. I posit that if you have a solution for ownership/retaining references then the solution for scope will come from this as a side-effect.
Dec 10 2014
next sibling parent reply "eles" <eles eles.com> writes:
On Wednesday, 10 December 2014 at 15:15:59 UTC, Ola Fosheim 
Grøstad wrote:
 On Tuesday, 9 December 2014 at 21:58:48 UTC, deadalnix wrote:
 D is already in that landscape and should try hard not to sink 
 deeper into the muddy waters.
Funny thing it is that it started exactly as a reaction to those muddy waters. I believe this is the "Stroustrup curse": " Much of the relative simplicity of Java is - like for most new languages - partly an illusion and partly a function of its incompleteness. As time passes, Java will grow significantly in size and complexity. It will double or triple in size and grow implementation-dependent extensions or libraries. That is the way every commercially successful language has developed. Just look at any language you consider successful on a large scale. I know of no exceptions, and there are good reasons for this phenomenon. [I wrote this before 2000; now (2012), the language part of the Java 7 specification is slightly longer in terms of number of pages than the ISO C++11 language specification.] "
Dec 10 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 10 December 2014 at 16:17:16 UTC, eles wrote:
 I believe this is the "Stroustrup curse":

 "
 Much of the relative simplicity of Java is - like for most new 
 languages - partly an illusion and partly a function of its 
 incompleteness. As time passes, Java will grow significantly in 
 size and complexity. It will double or triple in size and grow 
 implementation-dependent extensions or libraries. That is the 
 way every commercially successful language has developed. Just 
 look at any language you consider successful on a large scale.
I think he is making excuses for himself. Both Java and C++ are rather close to Simula+C as a starting point and have added cruft that should have been predicted in the initial version. Anyway, I think language specs often are too verbose, they probably all hit around 700-1000 pages eventually due to editors cutting it back to that book-like size before it is published (and when the book-sized limit has been reached they feel they have done enough ungrateful and unpaid work so they don't cut more even though they could have).
Dec 10 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 10 December 2014 at 15:15:59 UTC, Ola Fosheim
Grøstad wrote:
 On Tuesday, 9 December 2014 at 21:58:48 UTC, deadalnix wrote:
 That why i say they are linked. I don't think your way of 
 stating
 it contradict what I said.

 scope allow for manipulation of data without owning them.
 Whatever the owner is (be it the stack frame or anything else)
 doesn't really matter here.
It does when you return parts of it, like if you pass in a binary tree and return a node. Which you have to be able to do for the concept to make sense. When you solve issues in language design you should address the hard issues first, because the easy issues will then tend to resolve themselves if the design is good. If you start with "the low hanging fruit" you end up with pointless special casing… D is already in that landscape and should try hard not to sink deeper into the muddy waters. I posit that if you have a solution for ownership/retaining references then the solution for scope will come from this as a side-effect.
That is completely off topic. this is a function parameter, you can return scope, and you don't need to know who own the container for that to work. Granted, the current proposal lack the horsepower to do so.
Dec 10 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 10 December 2014 at 22:05:10 UTC, deadalnix wrote:
 That is completely off topic. this is a function parameter, you
 can return scope, and you don't need to know who own the
 container for that to work.
You have many scopes, if two different scopes pass in "horses" to a function in a recursive chain then you need to know which horse the returned "leg" belongs to… But by all means… go ahead if you think it is off topic. (it isn't)
Dec 10 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 10 December 2014 at 22:20:37 UTC, Ola Fosheim 
Grøstad wrote:
 On Wednesday, 10 December 2014 at 22:05:10 UTC, deadalnix wrote:
 That is completely off topic. this is a function parameter, you
 can return scope, and you don't need to know who own the
 container for that to work.
You have many scopes, if two different scopes pass in "horses" to a function in a recursive chain then you need to know which horse the returned "leg" belongs to… But by all means… go ahead if you think it is off topic. (it isn't)
It is always safe to consider scopeness of the retrun value (if marked scope) as being the intersection of the lifetime of parameters. That should cover most bases, and we can still extends later if this is too limited (I suspect it is ok for most cases).
Dec 10 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
deadalnix:

 That should cover most bases, and we can still extends later if 
 this is too limited (I suspect it is ok for most cases).
What syntax do you suggest to use if you want to extend it that way later? Bye, bearophile
Dec 10 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 00:48:32 UTC, bearophile wrote:
 deadalnix:

 That should cover most bases, and we can still extends later 
 if this is too limited (I suspect it is ok for most cases).
What syntax do you suggest to use if you want to extend it that way later? Bye, bearophile
scope[symbol_name] would get the same lifetime as symbol_name for instance. This is only a proposal, I'm not convinced that this is strictly necessary, and, as long as we know we can add it if needed, we are good.
Dec 10 2014
prev sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 11 December 2014 at 00:35:46 UTC, deadalnix wrote:
 It is always safe to consider scopeness of the retrun value (if 
 marked scope) as being the intersection of the lifetime of 
 parameters.

 That should cover most bases, and we can still extends later if 
 this is too limited (I suspect it is ok for most cases).
Linear typing is already extremely limiting, by limiting it even further you end up with something annoying. You basically get a version of memory safety that does not solve any typical memory unsafe situations. By having pointers that do scope-depth-tracking you do at least get a generic solution that can be optimized away when possible. The D authors have to accept that you need to embed ownership in pointers if you want memory safety and convenience, or that you have to provide means to guide the semantic analysis. You need one or the other, or both, but you cannot pretend that you can do without. Arbitrary constraints are annoying, not convenient. If I as a programmer know that something is safe, then the compiler should accept it, and the language should allow me express it.
Dec 11 2014
next sibling parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 11/12/2014 09:07, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 If I as a programmer know that something is safe, then the compiler
 should accept it, and the language should allow me express it.
It will, use system. Maybe this proposal will be supplemented later, but there are always costs to making the language more complicated. We need to be certain they are worth it before adding them. A cautious approach can be extended later, if the situation demands it. Sometimes innovative workarounds are developed that are difficult to foresee in advance - e.g. in Rust their type system can be restrictive, but they rely on trusted library functions/types to make more things possible in a safe way.
Dec 11 2014
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 11 December 2014 at 13:43:07 UTC, Nick Treleaven 
wrote:
 It will, use  system.
Yes…
 Maybe this proposal will be supplemented later, but there are 
 always costs to making the language more complicated. We need 
 to be certain they are worth it before adding them. A cautious 
 approach can be extended later, if the situation demands it.
Can it be extended later? Without breaking stuff? You need to get it right from the start, otherwise you end up with N different, but incompatible ways of doing the same thing. That means you need to get ownership right first.
Dec 11 2014
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Nick Treleaven:

 Sometimes innovative workarounds are developed that are 
 difficult to foresee in advance - e.g. in Rust their type 
 system can be restrictive, but they rely on trusted library 
 functions/types to make more things possible in a safe way.
Ideally a type system should be flexible enough, in practice giving it a high flexibility has significant costs (in compilation times, amount of code that implements the system, bugs in such implementation, costs for the final programmer in inventing a way to express the semantics, etc), so most languages avoid a too much complex type system (even Haskell does this) and accept reasonable workarounds... Bye, bearophile
Dec 11 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 15:37:27 UTC, bearophile wrote:
 Nick Treleaven:

 Sometimes innovative workarounds are developed that are 
 difficult to foresee in advance - e.g. in Rust their type 
 system can be restrictive, but they rely on trusted library 
 functions/types to make more things possible in a safe way.
Ideally a type system should be flexible enough, in practice giving it a high flexibility has significant costs (in compilation times, amount of code that implements the system, bugs in such implementation, costs for the final programmer in inventing a way to express the semantics, etc), so most languages avoid a too much complex type system (even Haskell does this) and accept reasonable workarounds... Bye, bearophile
What is important is how much expressiveness you get from the complexity. here we get almost none. Considering complexity alone is not going to yield good results.
Dec 11 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 09:07:44 UTC, Ola Fosheim 
Grøstad wrote:
 On Thursday, 11 December 2014 at 00:35:46 UTC, deadalnix wrote:
 It is always safe to consider scopeness of the retrun value 
 (if marked scope) as being the intersection of the lifetime of 
 parameters.

 That should cover most bases, and we can still extends later 
 if this is too limited (I suspect it is ok for most cases).
Linear typing is already extremely limiting, by limiting it even further you end up with something annoying. You basically get a version of memory safety that does not solve any typical memory unsafe situations. By having pointers that do scope-depth-tracking you do at least get a generic solution that can be optimized away when possible. The D authors have to accept that you need to embed ownership in pointers if you want memory safety and convenience, or that you have to provide means to guide the semantic analysis. You need one or the other, or both, but you cannot pretend that you can do without. Arbitrary constraints are annoying, not convenient. If I as a programmer know that something is safe, then the compiler should accept it, and the language should allow me express it.
I have no idea what you are saying. It sounds like randomly generated gibberish.
Dec 11 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 11 December 2014 at 23:26:33 UTC, deadalnix wrote:
 I have no idea what you are saying. It sounds like randomly 
 generated gibberish.
What is your problem? Scope parameters are trying to address what Rust tried to do with borrowing. To get an idea of where you are going you should look at ordered linear and non-linear type systems. Without a dynamic solution OR a proof system OR a way to provide a manual guarantee you will hit the ceiling for what you can do, which will be annoying for the user (programmer).
Dec 11 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/5/14 3:55 PM, Walter Bright wrote:
 On 12/5/2014 7:27 AM, Steven Schveighoffer wrote:
 Can someone who knows what this new feature is supposed to do give
 some Ali
 Çehreli-like description on the feature? Basically, let's strip out
 the *proof*
 in the DIP (the how it works and why we have it), and focus on how it
 is to be
 used.

 I still am having a hard time wrapping my head around the benefits and
 when to
 use scope, scope ref, why I would use it. I'm worried that we are
 adding all
 this complication and it will confuse the shit out of users, to the
 point where
 they won't use it.
The tl;dr version is when a declaration is tagged with 'scope', the contents of that variable will not escape the lifetime of that declaration. It means that this code will be safe: void foo(scope int* p); p = malloc(n); foo(p); free(p); The rest is all the nuts and bolts of making that work.
This is not what I was asking for. What I wanted to know was, when I see scope ref, why is it there? When should I use it? When should I use scope? What nasty things are prevented if I use it? Examples would be preferable. Note, in your example above, marking foo pure solves the problem already. The rules and statements give an inferred benefit. I'd like that benefit to be more fully explained. -Steve
Dec 08 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 7:23 AM, Steven Schveighoffer wrote:
 What I wanted to know was, when I see scope
 ref, why is it there? When should I use it?
It's there to indicate a parameter is NOT returned by ref.
 When should I use scope? What nasty
 things are prevented if I use it? Examples would be preferable.
http://wiki.dlang.org/DIP69#Ref has examples of errors that are prevented.
Dec 14 2014
prev sibling parent zeljkog <zeljkog home.com> writes:
On 05.12.14 21:55, Walter Bright wrote:
 
 It means that this code will be safe:
 
     void foo(scope int* p);
 
     p = malloc(n);
     foo(p);
     free(p);
 
 The rest is all the nuts and bolts of making that work.
 
If it is main goal, opposite looks more natural. void foo(int* p); void foo1(escape int* p); p = malloc(n); p1 = malloc(n); foo(p); free(p); // OK foo1(p1); free(p1); // no
Dec 08 2014
prev sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2014-12-05 15:27:41 +0000, Steven Schveighoffer said:

 On 12/4/14 4:24 AM, Walter Bright wrote:
 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible for
 it to escape.
 
 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.
 
 It does not make scope into a type constructor, nor a general
 type-annotation system.
 
 It does not provide an ownership system, though it would complement one.
Can we take a step back here? I read many people's comments and I understand only about half of them. Can someone who knows what this new feature is supposed to do give some Ali ehreli-like description on the feature? Basically, let's strip out the *proof* in the DIP (the how it works and why we have it), and focus on how it is to be used. I still am having a hard time wrapping my head around the benefits and when to use scope, scope ref, why I would use it. I'm worried that we are adding all this complication and it will confuse the shit out of users, to the point where they won't use it. -Steve
This is exactly why this feature should be default behavior with compiler warnings generated when things escape scope. The basic idea is that most things should be on the stack and go away when the stack goes away. The proposal codifies how the compiler should infer that references to variables which are to be placed on the stack are not escaping their scope. The benefit is that the GC has to do *way* less work from what it has to do now for most of the use cases of D. -Shammah
Dec 10 2014
parent zeljkog <zeljkog home.com> writes:
On 11.12.14 08:25, Shammah Chancellor wrote:
 This is exactly why this feature should be default behavior with
 compiler warnings generated when things escape scope.
It can be compiler switch escape=I(gnore)|W(arning)|E(rror).
Dec 11 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
There are limitations this proposal has in comparison to my 
original one. These limitations might of course be harmless and 
play no role in practice, but on the other hand, they may, so I 
think it's good to list them here.

Additionally I have to agree with Steven Schveighoffer: This DIP 
is very complicated to understand. It's not obvious how the 
various parts play together, and why/to which degree it "works", 
and which are the limitations. I don't think that's only because 
my brain is already locked on my proposal...

1) Escape detection is limited to `ref`.

     T* evil;
     ref T func(scope ref T t, ref T u)  safe {
       return t; // Error: escaping scope ref t
       return u; // ok
       evil = &u; // Error: escaping reference
     }

vs.

     T[] evil;
     T[] func(scope T[] t, T[] u)  safe {
       return t; // Error: cannot return scope
       return u; // ok
       evil = u; // !!! not good
     }

As can be seen, `ref T u` is protected from escaping (apart from 
returning it), while `T[] u` in the second example is not. 
There's no general way to express that `u` can only be returned 
from the function, but will not be retained otherwise by storing 
it in a global variable. Adding `pure` can express this in many 
cases, but is, of course, not always possible.

Another workaround is passing the parameters as `ref`, but this 
would introduce an additional indirection and has different 
semantics (e.g. when the lengths of the slices are modified).

2) `scope ref` return values cannot be stored.

     scope ref int foo();
     void bar(scope ref int a);

     foo().bar();        // allowed
     scope tmp = foo();  // not allowed
     tmp.bar();

Another example:

     struct Container(T) {
         scope ref T opIndex(size_t index);
     }

     void bar(scope ref int a);

     Container c;
     bar(c[42]);            // ok
     scope ref tmp = c[42]; // nope

Both cases should be fine theoretically; the "real" owner lives 
longer than `tmp`. Unfortunately the compiler doesn't know about 
this.

Both restrictions 1) and 2) are because there are no explicit 
lifetime/owner designations (the scope!identifier thingy in my 
proposal).

3) `scope` cannot be used for value types.

I can think of a few use cases for scoped value types (RC and 
file descriptors), but they might only be marginal.

4) No overloading on `scope`.

This is at least partially a consequence of `scope` inference. I 
think overloading can be made to work in the presence of 
inference, but I haven't thought it through.

5) `scope` is a storage class.

Manu complained about `ref` being a storage class. If I 
understand him right, one reason is that we have a large toolkit 
for dealing with type modifiers, but almost nothing for storage 
classes. I have to agree with him there. But I haven't understood 
his point fully, maybe he himself can post more about his 
problems with this?

6) There seem to be problems with chaining.

     scope ref int foo();
     scope ref int bar1(ref int a) {
         return a;
     }
     scope ref int bar2(scope ref int a) {
         return a;
     }
     ref int bar3(ref int a) {
         return a;
     }
     ref int bar4(scope ref int a) {
         return a;
     }
     void baz(scope ref int a);

Which of the following calls would work?

     foo().bar1().baz();
     foo().bar2().baz();
     foo().bar3().baz();
     foo().bar4().baz();

I'm not sure I understand this fully yet, but it could be that 
none of them work...
Dec 05 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 There are limitations this proposal has in comparison to my original one. These
 limitations might of course be harmless and play no role in practice, but on
the
 other hand, they may, so I think it's good to list them here.
Good idea. Certainly, this is less powerful than your proposal. The question, obviously, is what is good enough to get the job done. By "the job", I mean reference counting, migrating many allocations to RAII (meaning the stack), and eliminating a lot of closure GC allocations.
 Additionally I have to agree with Steven Schveighoffer: This DIP is very
 complicated to understand. It's not obvious how the various parts play
together,
 and why/to which degree it "works", and which are the limitations. I don't
think
 that's only because my brain is already locked on my proposal...
I'm still looking for an easier way to explain it. The good news in this, however, is if it is correctly implement the compiler should be a big help in using scope correctly.
 1) Escape detection is limited to `ref`.

      T* evil;
      ref T func(scope ref T t, ref T u)  safe {
        return t; // Error: escaping scope ref t
        return u; // ok
        evil = &u; // Error: escaping reference
      }

 vs.

      T[] evil;
      T[] func(scope T[] t, T[] u)  safe {
        return t; // Error: cannot return scope
        return u; // ok
        evil = u; // !!! not good
right, although: evil = t; // Error: not allowed
      }

 As can be seen, `ref T u` is protected from escaping (apart from returning it),
 while `T[] u` in the second example is not. There's no general way to express
 that `u` can only be returned from the function, but will not be retained
 otherwise by storing it in a global variable. Adding `pure` can express this in
 many cases, but is, of course, not always possible.
As you point out, 'ref' is designed for this.
 Another workaround is passing the parameters as `ref`, but this would introduce
 an additional indirection and has different semantics (e.g. when the lengths of
 the slices are modified).

 2) `scope ref` return values cannot be stored.

      scope ref int foo();
      void bar(scope ref int a);

      foo().bar();        // allowed
      scope tmp = foo();  // not allowed
      tmp.bar();
Right
 Another example:

      struct Container(T) {
          scope ref T opIndex(size_t index);
      }

      void bar(scope ref int a);

      Container c;
      bar(c[42]);            // ok
      scope ref tmp = c[42]; // nope

 Both cases should be fine theoretically; the "real" owner lives longer than
 `tmp`. Unfortunately the compiler doesn't know about this.
Right, though the compiler can optimize to produce the equivalent.
 Both restrictions 1) and 2) are because there are no explicit lifetime/owner
 designations (the scope!identifier thingy in my proposal).

 3) `scope` cannot be used for value types.

 I can think of a few use cases for scoped value types (RC and file
descriptors),
 but they might only be marginal.
I suspect that one can encapsulate such values in a struct where access to them is strictly controlled.
 4) No overloading on `scope`.

 This is at least partially a consequence of `scope` inference. I think
 overloading can be made to work in the presence of inference, but I haven't
 thought it through.
Right. Different overloads can have different semantic implementations, so what should inference do? I also suspect it is bad style to overload on 'scope'.
 5) `scope` is a storage class.

 Manu complained about `ref` being a storage class. If I understand him right,
 one reason is that we have a large toolkit for dealing with type modifiers, but
 almost nothing for storage classes. I have to agree with him there. But I
 haven't understood his point fully, maybe he himself can post more about his
 problems with this?
I didn't fully understand Manu's issue, but it was about 'ref' not being inferred by template type deduction. I didn't understand why 'auto ref' did not work for him. I got the impression that he was trying to program in D the same way he'd do things in C++, and that's where the trouble came in.
 6) There seem to be problems with chaining.

      scope ref int foo();
      scope ref int bar1(ref int a) {
          return a;
      }
      scope ref int bar2(scope ref int a) {
          return a;
      }
      ref int bar3(ref int a) {
          return a;
      }
      ref int bar4(scope ref int a) {
          return a;
      }
      void baz(scope ref int a);

 Which of the following calls would work?

      foo().bar1().baz();
yes
      foo().bar2().baz();
no - cannot return scope ref parameter
      foo().bar3().baz();
yes
      foo().bar4().baz();
no, cannot return scope ref parameter
 I'm not sure I understand this fully yet, but it could be that none of them
work...
Well, you're half right :-)
Dec 05 2014
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 The question, obviously, is what is good enough to get the job 
 done. By "the job", I mean reference counting, migrating many 
 allocations to RAII (meaning the stack), and eliminating a lot 
 of closure GC allocations.
Another essential purpose of a similar proposal is to avoid some bugs caused by using memory from vanished stack frames, etc, that is to have memory safety in D. Bye, bearophile
Dec 05 2014
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 December 2014 at 09:58, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 There are limitations this proposal has in comparison to my original one.
 These
 limitations might of course be harmless and play no role in practice, but
 on the
 other hand, they may, so I think it's good to list them here.
Good idea. Certainly, this is less powerful than your proposal. The question, obviously, is what is good enough to get the job done. By "the job", I mean reference counting, migrating many allocations to RAII (meaning the stack), and eliminating a lot of closure GC allocations.
 Additionally I have to agree with Steven Schveighoffer: This DIP is very
 complicated to understand. It's not obvious how the various parts play
 together,
 and why/to which degree it "works", and which are the limitations. I don't
 think
 that's only because my brain is already locked on my proposal...
I'm still looking for an easier way to explain it. The good news in this, however, is if it is correctly implement the compiler should be a big help in using scope correctly.
 1) Escape detection is limited to `ref`.

      T* evil;
      ref T func(scope ref T t, ref T u)  safe {
        return t; // Error: escaping scope ref t
        return u; // ok
        evil = &u; // Error: escaping reference
      }

 vs.

      T[] evil;
      T[] func(scope T[] t, T[] u)  safe {
        return t; // Error: cannot return scope
        return u; // ok
        evil = u; // !!! not good
right, although: evil = t; // Error: not allowed
      }

 As can be seen, `ref T u` is protected from escaping (apart from returning
 it),
 while `T[] u` in the second example is not. There's no general way to
 express
 that `u` can only be returned from the function, but will not be retained
 otherwise by storing it in a global variable. Adding `pure` can express
 this in
 many cases, but is, of course, not always possible.
As you point out, 'ref' is designed for this.
 Another workaround is passing the parameters as `ref`, but this would
 introduce
 an additional indirection and has different semantics (e.g. when the
 lengths of
 the slices are modified).

 2) `scope ref` return values cannot be stored.

      scope ref int foo();
      void bar(scope ref int a);

      foo().bar();        // allowed
      scope tmp = foo();  // not allowed
      tmp.bar();
Right
 Another example:

      struct Container(T) {
          scope ref T opIndex(size_t index);
      }

      void bar(scope ref int a);

      Container c;
      bar(c[42]);            // ok
      scope ref tmp = c[42]; // nope

 Both cases should be fine theoretically; the "real" owner lives longer
 than
 `tmp`. Unfortunately the compiler doesn't know about this.
Right, though the compiler can optimize to produce the equivalent.
 Both restrictions 1) and 2) are because there are no explicit
 lifetime/owner
 designations (the scope!identifier thingy in my proposal).

 3) `scope` cannot be used for value types.

 I can think of a few use cases for scoped value types (RC and file
 descriptors),
 but they might only be marginal.
I suspect that one can encapsulate such values in a struct where access to them is strictly controlled.
 4) No overloading on `scope`.

 This is at least partially a consequence of `scope` inference. I think
 overloading can be made to work in the presence of inference, but I
 haven't
 thought it through.
Right. Different overloads can have different semantic implementations, so what should inference do? I also suspect it is bad style to overload on 'scope'.
 5) `scope` is a storage class.

 Manu complained about `ref` being a storage class. If I understand him
 right,
 one reason is that we have a large toolkit for dealing with type
 modifiers, but
 almost nothing for storage classes. I have to agree with him there. But I
 haven't understood his point fully, maybe he himself can post more about
 his
 problems with this?
I didn't fully understand Manu's issue, but it was about 'ref' not being inferred by template type deduction. I didn't understand why 'auto ref' did not work for him. I got the impression that he was trying to program in D the same way he'd do things in C++, and that's where the trouble came in.
NO!! I barely program C++ at all! I basically write C code with 'enum' and 'class'. NEVER 'template', and very rarely 'virtual'. Your impression is dead wrong. If I do things in any particular 'way', it is that at all times, I have regard for, and control over the code generation, the ABI, and I also value distribution of code as static libs, which means I must retain tight control over where code is, and isn't, emitted. I expect this from a native language. It's exactly as Marc says, we have the best tools for dealing with types of any language I know, and practically none for dealing with 'storage class'. In my use of meta in D, 'ref' is the single greatest cause of complexity, code bloat, duplication, and text mixins. I've been banging on about this for years! I've been over it so many times. I'll start over if there is actually some possibility I can convince you? Is there? I've lost faith that I am able to have any meaningful impact on issues that matter to me, and I'm fairly sure at this point that my compounded resentment and frustration actually discredit my cause, and certainly, my quality of debate. I'm passionate about these things, but I'm also extremely frustrated. To date, whenever I have engaged in a topic that I *really* care about, it goes the other direction. To participate has proven to be something of a (very time consuming) act of masochism. I __absolutely objected__ to 'auro ref' when it appeared. I argued that it was a massive mistake, and since it's introduction and experience with it in the wild, I am more confident in that conviction than ever. As a programmer, I expect control over whether code is a template, or not. 'ref' doesn't have anything to do with templates. Confusing these 2 concepts was such a big mistake. auto ref makes a template out of something that shouldn't be a template. It's a particularly crude hack to address a prior mistake. ref should have been fixed at that time, not compounded with layers of even more weird and special-case/non-uniform behaviours above it. There has been a couple of instances where the situation has been appropriate that I've tried to make use of auto ref, but in each case, the semantics have never been what I want. auto ref presumes to decide for you when something should be a ref or not. It prescribes a bunch of rules on how that decision is made, but it doesn't know what I'm doing, and it gets it wrong. In my experience, auto ref has proven to be, at best, completely useless. But in some cases I've found where I bump into it in 3rd party code, it's been a nuisance, requiring me to wrap it away. ref is a bad design. C++'s design isn't fantastic, and I appreciate that D made effort to improve on it, but we need to recognise when the experiment was a failure. D's design is terrible; it's basically orthogonal to the rest of the language. It's created way more complicated edge cases for me than C++ references ever have. Anyone who says otherwise obviously hasn't really used it much! Don't double down on that mistake with scope. My intended reply to this post was to ask you to justify making it a storage class, and why the design fails as a type constructor? Can we explore that direction to it's point of failure? As a storage class, it runs the risk of doubling out existing bloat caused by ref. As a type constructor, I see no disadvantages. It even addresses some of the awkward problems right at the face of storage classes, like how/where the attributes actually apply. Type constructors use parens; ie, const(T), and scope(T) would make that matter a lot more clear. Apart from the storage class issue, it looks okay, but it gives me the feeling that it kinda stops short. Marc's proposal addressed more issues. I feel this proposal will result in more edge cases than Marc's proposal. The major edge case that I imagine is that since scope return values can't be assigned to scope local's, that will result in some awkward meta, requiring yet more special cases. I think Mark's proposal may be a lot more relaxed in that way.
Dec 06 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-12-06 10:50, Manu via Digitalmars-d wrote:

 I've been over it so many times.
I suggest you take the time and write down how your vision of "ref" looks like and the issue with the current implementation. A blog post, a DIP or similar. Then you can easily refer to that in cases like this. Then you don't have to repeat yourself so many times. That's what I did when there was a lot of talk about AST macros. I was tired of constantly repeating myself so I created a DIP. It has already saved me more time than it took to write the actual DIP. -- /Jacob Carlborg
Dec 06 2014
parent reply "Piotrek" <pl pl.pl> writes:
On Saturday, 6 December 2014 at 11:06:16 UTC, Jacob Carlborg 
wrote:
 On 2014-12-06 10:50, Manu via Digitalmars-d wrote:

 I've been over it so many times.
I suggest you take the time and write down how your vision of "ref" looks like and the issue with the current implementation. A blog post, a DIP or similar. Then you can easily refer to that in cases like this. Then you don't have to repeat yourself so many times. That's what I did when there was a lot of talk about AST macros. I was tired of constantly repeating myself so I created a DIP. It has already saved me more time than it took to write the actual DIP.
Manu Seconded. Please create even short one. I coulnd'd find any example of use case you are referring to (as you said I don't use D"ref" so often) . I plan to apply D for embedded systems, so full control is a must. But so far Water still has the most accurate taste according to my experience. BTW. I consider game devs to be the most underpaid programmers, so your perspective is very precious to me. Cheers Piotrek
Dec 06 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 6 December 2014 at 23:59:01 UTC, Piotrek wrote:
 On Saturday, 6 December 2014 at 11:06:16 UTC, Jacob Carlborg 
 wrote:
 On 2014-12-06 10:50, Manu via Digitalmars-d wrote:

 I've been over it so many times.
I suggest you take the time and write down how your vision of "ref" looks like and the issue with the current implementation. A blog post, a DIP or similar. Then you can easily refer to that in cases like this. Then you don't have to repeat yourself so many times. That's what I did when there was a lot of talk about AST macros. I was tired of constantly repeating myself so I created a DIP. It has already saved me more time than it took to write the actual DIP.
Manu Seconded. Please create even short one. I coulnd'd find any example of use case you are referring to (as you said I don't use D"ref" so often) . I plan to apply D for embedded systems, so full control is a must. But so far Water still has the most accurate taste according to my experience. BTW. I consider game devs to be the most underpaid programmers, so your perspective is very precious to me. Cheers Piotrek
I'd like to not polute this thread with the ref topic. Long story short: - it is hard to know if something is ref, making it hard to metaprogram. - you sometime want to switch ref on and off (for instance, you may use ref to avoid copies, which is not worthwhile for small values) which is complex. - auto ref do not cut it. That is the extra short version, please start a thread on the subject. Please, please, this one is complicated enough.
Dec 08 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 6 December 2014 at 09:51:15 UTC, Manu via 
Digitalmars-d wrote:
 I've been over it so many times.
 I'll start over if there is actually some possibility I can 
 convince
 you? Is there?

 I've lost faith that I am able to have any meaningful impact on 
 issues
 that matter to me, and I'm fairly sure at this point that my
 compounded resentment and frustration actually discredit my 
 cause, and
 certainly, my quality of debate.
 I'm passionate about these things, but I'm also extremely 
 frustrated.
 To date, whenever I have engaged in a topic that I *really* care
 about, it goes the other direction. To participate has proven 
 to be
 something of a (very time consuming) act of masochism.
That's very sad to hear. I think you have brought up some very important points, especially from a practical point of view.
 I __absolutely objected__ to 'auro ref' when it appeared. I 
 argued
 that it was a massive mistake, and since it's introduction and
 experience with it in the wild, I am more confident in that 
 conviction
 than ever.

 As a programmer, I expect control over whether code is a 
 template, or
 not. 'ref' doesn't have anything to do with templates. 
 Confusing these
 2 concepts was such a big mistake.
 auto ref makes a template out of something that shouldn't be a
 template. It's a particularly crude hack to address a prior 
 mistake.
 ref should have been fixed at that time, not compounded with 
 layers of
 even more weird and special-case/non-uniform behaviours above 
 it.


 There has been a couple of instances where the situation has 
 been
 appropriate that I've tried to make use of auto ref, but in 
 each case,
 the semantics have never been what I want.
 auto ref presumes to decide for you when something should be a 
 ref or
 not. It prescribes a bunch of rules on how that decision is 
 made, but
 it doesn't know what I'm doing, and it gets it wrong.
 In my experience, auto ref has proven to be, at best, completely
 useless. But in some cases I've found where I bump into it in 
 3rd
 party code, it's been a nuisance, requiring me to wrap it away.
I tend to agree with this. Here's an example of the dangers: https://github.com/deadalnix/libd/pull/7 In this case, `auto ref` accepted a class by reference, because it chooses ref-ness based on whether you pass an lvalue or an rvalue. This is not very helpful. I agree that `ref` isn't something that should ever be inferred, because it affects the semantics of the function. Either the function requires `ref` semantics, or not. It doesn't depend on how it is called.
Dec 06 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Marc Schütz:

 I agree that `ref` isn't something that should ever be 
 inferred, because it affects the semantics of the function. 
 Either the function requires `ref` semantics, or not. It 
 doesn't depend on how it is called.
(Perhaps this was already said) I think Ada used to have something like "auto ref" and later it was removed from the language. Bye, bearophile
Dec 06 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/6/2014 1:50 AM, Manu via Digitalmars-d wrote:
 I didn't fully understand Manu's issue, but it was about 'ref' not being
 inferred by template type deduction. I didn't understand why 'auto ref' did
 not work for him. I got the impression that he was trying to program in D
 the same way he'd do things in C++, and that's where the trouble came in.
NO!! I barely program C++ at all! I basically write C code with 'enum' and 'class'. NEVER 'template', and very rarely 'virtual'. Your impression is dead wrong.
I apologize for misunderstanding you.
 It's exactly as Marc says, we have the best tools for dealing with
 types of any language I know, and practically none for dealing with
 'storage class'.
 In my use of meta in D, 'ref' is the single greatest cause of
 complexity, code bloat, duplication, and text mixins. I've been
 banging on about this for years!

 I've been over it so many times.
 I'll start over if there is actually some possibility I can convince
 you? Is there?
I know there's no easy way to derive a storage class from an expression. The difficulty in my understanding is why this is a great cause of problems for you in particular (and by implication not for others). There's something about the way you write code that's different.
 I've lost faith that I am able to have any meaningful impact on issues
 that matter to me, and I'm fairly sure at this point that my
 compounded resentment and frustration actually discredit my cause, and
 certainly, my quality of debate.
This is incorrect, you had enormous influence over the vector type in D! And I wish you had more.
 There has been a couple of instances where the situation has been
 appropriate that I've tried to make use of auto ref, but in each case,
 the semantics have never been what I want.
I don't really know what you want. Well, perhaps a better statement is 'why', not what.
 ref is a bad design. C++'s design isn't fantastic, and I appreciate
 that D made effort to improve on it, but we need to recognise when the
 experiment was a failure. D's design is terrible; it's basically
 orthogonal to the rest of the language. It's created way more
 complicated edge cases for me than C++ references ever have. Anyone
 who says otherwise obviously hasn't really used it much!
 Don't double down on that mistake with scope.
Why does your code need to care so much about to ref or not to ref? That's the central point here, I think.
 My intended reply to this post was to ask you to justify making it a
 storage class, and why the design fails as a type constructor?
 Can we explore that direction to it's point of failure?

 As a storage class, it runs the risk of doubling out existing bloat
 caused by ref. As a type constructor, I see no disadvantages.
 It even addresses some of the awkward problems right at the face of
 storage classes, like how/where the attributes actually apply. Type
 constructors use parens; ie, const(T), and scope(T) would make that
 matter a lot  more clear.

 Apart from the storage class issue, it looks okay, but it gives me the
 feeling that it kinda stops short.
 Marc's proposal addressed more issues. I feel this proposal will
 result in more edge cases than Marc's proposal.
 The major edge case that I imagine is that since scope return values
 can't be assigned to scope local's, that will result in some awkward
 meta, requiring yet more special cases. I think Mark's proposal may be
 a lot more relaxed in that way.
The disadvantages of making it a type qualifier are: 1. far more complexity. Type constructors interact with everything, often in unanticipated ways. We spent *years* working out issues with the 'const' type qualifier, and are still doing so. Kenji just fixed another one. 2. we are never going to get users to use 'scope' qualifiers pervasively. It's been a long struggle to get 'const' used. 3. we added 'inout' as a type qualifier to avoid code duplication engendered by 'const'. It hurts my brain to even think about how that might interact with 'scope' qualifiers. Yes, I agree unequivocably, that 'scope' as a type qualifier is more expressive and more powerful than as a storage class. Multiple inheritance is also more expressive and more powerful than single inheritance. But many times, more power perhaps isn't better than redoing the program design to use something simpler and less complex.
Dec 06 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 December 2014 at 08:15, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 12/6/2014 1:50 AM, Manu via Digitalmars-d wrote:
 I didn't fully understand Manu's issue, but it was about 'ref' not being
 inferred by template type deduction. I didn't understand why 'auto ref'
 did
 not work for him. I got the impression that he was trying to program in D
 the same way he'd do things in C++, and that's where the trouble came in.
NO!! I barely program C++ at all! I basically write C code with 'enum' and 'class'. NEVER 'template', and very rarely 'virtual'. Your impression is dead wrong.
I apologize for misunderstanding you.
 It's exactly as Marc says, we have the best tools for dealing with
 types of any language I know, and practically none for dealing with
 'storage class'.
 In my use of meta in D, 'ref' is the single greatest cause of
 complexity, code bloat, duplication, and text mixins. I've been
 banging on about this for years!

 I've been over it so many times.
 I'll start over if there is actually some possibility I can convince
 you? Is there?
I know there's no easy way to derive a storage class from an expression. The difficulty in my understanding is why this is a great cause of problems for you in particular (and by implication not for others). There's something about the way you write code that's different.
Perhaps it's the tasks I typically perform with meta? I'm a fairly conservative user of templates, but one place where they shine, and I'm always very tempted to use them, is the task of serialisation, and cross-language bindings. Those tasks typically involve mountains of boilerplate, and D is the only language expressive enough to start to really automate that mechanical mess. I've done extensive work bindings to C/C++, Lua, and C#. They're all subtly different tasks, but share a lot in common, and the major characteristic which reveals my problems with things like ref, and auto ref, is that these things aren't open to interpretation. They're not cases where the compiler can make decisions for you (ie, auto ref fails), and the ABI + API are strict (ie, ref must match correctly). There's a lot more more to it, but perhaps I'm among few who have had such extensive need to engage in these tasks?
 I've lost faith that I am able to have any meaningful impact on issues
 that matter to me, and I'm fairly sure at this point that my
 compounded resentment and frustration actually discredit my cause, and
 certainly, my quality of debate.
This is incorrect, you had enormous influence over the vector type in D! And I wish you had more.
I appreciate that. That was uncontroversial though; I didn't need to spend months or years trying to justify my claims on that issue. I feel like that was a known item that was just somewhere slightly down the list, and I was able to bring it forward. I was never in a position where I had to argue against Andrei to sell that one. I have std.simd sitting here, and I really want to finish it, but I still don't have the tools to do so. I need, at least, forceinline to complete it, but that one *is* controversial - we've talked about this for years. GDC and LDC both have a forceinline, so I could theoretically support those compilers, but then I can't practically make use of them without some sort of attribute aliasing system, otherwise I need to triplicate the code for each compiler, just to insert a different (compiler specific) forceinline attribute name. It'd be really great if we agreed on just one. There is also a practical problem with GCC (perhaps it's an incompatbility with my std.simd design), where I need to change the sse-level between functions. There is a GCC attribute to do this ('target'), but it would rely on, at least, a few subtle tweaks. In this case, I need to be able to feed a template argument to a UDA, which doesn't work because UDA declarations seem to be parsed prior to knowledge of functions template args. There may be a further problem with GCC though which I haven't been able to prove yet though. The reason for this is that an important design goal for my work was to be able to semi-automatically generate multiple code paths for different SSE versions, which can be selected at runtime based on available hardware. It's a particularly awkward issue in C/C++, usually requiring you to have multiple modules which are each built with different compile flags, and then do some magic while linking to resolve the runtime selection problem. Anyway, I have been able to use SIMD directly in my own software with the support we have, but I haven't been able to complete the library that I want to produce. I need a little more language support. I've reached out a few times, but there hasn't been too much interest in the past. I really do wanna finish it one of these days though!
 There has been a couple of instances where the situation has been
 appropriate that I've tried to make use of auto ref, but in each case,
 the semantics have never been what I want.
I don't really know what you want. Well, perhaps a better statement is 'why', not what.
Because a function is a function. It's a fundamental and ultimately *simple* element of a language, with simple and reliable in behaviour; you write a function, compiler emits some code with a matching symbol name. There is perhaps nothing more simple or fundamental to the language. A template is not a function. It may *generate* a function, but now we've added many extra details which require careful consideration and handling, like where the code is, or isn't, emitted? How many permutations are there? What about static/dynamic libraries? Calling with various, or arbitrary types, always required careful consideration, more so than an explicit type. Templates are a powerful (often dangerous) **tool**, which should only be applied deliberately and very carefully. Obviously auto-ref gives hard answers to some of those questions, but it is still a template, and that instantly makes it much more complex. Making a function that has nothing to do with templates into a template is not what I want... it couldn't be further from what I ever wanted! The ABI is different; there are 2 functions now (or zero, until one is called). Thich means taking the address of a function with auto-ref is a complex issue. ref-ness isn't a decision I ever want the compiler to make for me. I either want ref, or I don't, and I will state that to the compiler explicitly. This is especially true when interacting with other languages; while D remains in relative infancy, I think that is an overwhelmingly common situation when a project is modestly large. In the situation where templates are involved, it would be nice to be able to make that explicit statement that some type is ref or not at the point of template instantiation, and the resolution should work according to the well-defined rules that we are all familiar with. Within a complex template, if logic should be performed, we have powerful tools to do that already; nobody ever complains about Unqual!, or PointerTarget!... these things are very easy to read and understand. ref should fit right in there with the others, is(x == ref), UnRef!T, etc. Instead, it's practically orthogonal to the language. Wrangling anything involving storage classes is really, really tedious, and almost always results in extensive code duplication, or text mixins when duplication just get too much. This issue really surprised me at the time, because the whole thing was in response to a topic that I consistently raised and pushed. Who was the customer welcoming that solution? I could never work it out; there was nobody else that seemed particularly invested in the problem, except the usual suite of language enthusiasts that took it as an interesting intellectual problem to discuss. The person who cared about that issue the most (afaict) was completely unsatisfied with the solution. I said at the time that I would have preferred that no action was taken, than to compound, and further set in stone, an issue that I was already extremely critical of. Don't double down on this mistake with scope!
 ref is a bad design. C++'s design isn't fantastic, and I appreciate
 that D made effort to improve on it, but we need to recognise when the
 experiment was a failure. D's design is terrible; it's basically
 orthogonal to the rest of the language. It's created way more
 complicated edge cases for me than C++ references ever have. Anyone
 who says otherwise obviously hasn't really used it much!
 Don't double down on that mistake with scope.
Why does your code need to care so much about to ref or not to ref? That's the central point here, I think.
That same logic could ask why I want to have 'short', or 'int' in some places, but not others. There's no absolute answer. As a software engineer, and particularly, a native software engineer, it's a fundamental part of the type system (sorry, not part of the type system!) that I must have control over. Why offer 'ref' if you don't intend people to use it? I put to you, why do you hate 'ref' so much? Remove it from the language if people aren't meant to use it. ref is just a pointer with some semantics removed (pointer re-assignment, indexing). It's use facilitates some forms of generic code which would fail otherwise, and also reduces some possibilities for end-user misuse or mistakes when using pointers (ie, user may intend an assignment, but unsuspectingly re-assign a pointer; result looks the same, but exposes bug somewhere else). It's also used as an optimisation where I require control over the ref-ness of things, but don't want to retrofit the code with a sea of '*' and '&' operators. It's also used when interfacing C++, where it appears in API's extensively. I'm sure there are many more common and useful cases where ref is a good tool.
 My intended reply to this post was to ask you to justify making it a
 storage class, and why the design fails as a type constructor?
 Can we explore that direction to it's point of failure?

 As a storage class, it runs the risk of doubling out existing bloat
 caused by ref. As a type constructor, I see no disadvantages.
 It even addresses some of the awkward problems right at the face of
 storage classes, like how/where the attributes actually apply. Type
 constructors use parens; ie, const(T), and scope(T) would make that
 matter a lot  more clear.

 Apart from the storage class issue, it looks okay, but it gives me the
 feeling that it kinda stops short.
 Marc's proposal addressed more issues. I feel this proposal will
 result in more edge cases than Marc's proposal.
 The major edge case that I imagine is that since scope return values
 can't be assigned to scope local's, that will result in some awkward
 meta, requiring yet more special cases. I think Mark's proposal may be
 a lot more relaxed in that way.
The disadvantages of making it a type qualifier are: 1. far more complexity. Type constructors interact with everything, often in unanticipated ways. We spent *years* working out issues with the 'const' type qualifier, and are still doing so. Kenji just fixed another one.
What you describe is is the opposite of 'complexity'. Uniform behaviour is what people expect from a language. I understand complexity may arise as a result of interaction with other features, but that's a worthwhile issue to explore if you ask me. That's complexity that can actually be addressed, rather than pushed to the side. Edge cases which are orthogonal to the rest of the language (ie, ref), are a terrible idea. You don't buffer against complexity by creating a whole new class of complexity, which we have no effective tools to manage or mitigate.
 2. we are never going to get users to use 'scope' qualifiers pervasively.
 It's been a long struggle to get 'const' used.
I don't think scope will be as important as const. I think we should *try it*, to experimentally see how it plays out. That said, scope should be able to be inferred quite effectively in many most cases. Again, I think we will only be able to measure the effectiveness of this when we try it.
 3. we added 'inout' as a type qualifier to avoid code duplication engendered
 by 'const'. It hurts my brain to even think about how that might interact
 with 'scope' qualifiers.
inout was an interesting idea (with a terrible name!). I'm still not sure if I think it was a good idea or not, but I have found it very useful. What's the issue? I don't quite see how inout and scope overlap (no differently than const or immutable?).
 Yes, I agree unequivocably, that 'scope' as a type qualifier is more
 expressive and more powerful than as a storage class. Multiple inheritance
 is also more expressive and more powerful than single inheritance. But many
 times, more power perhaps isn't better than redoing the program design to
 use something simpler and less complex.
I don't think that's a reasonable comparison. Multiple inheritance is (I think quite well agreed) a wildly unpopular and super-complex disaster. Comparing scope-as-a-storage-class or scope-as-a-type-constructor is nothing like comparing multiple inheritence and interfaces... interfaces aren't orthogonal to the language for a start! And they're well understood, and precedented in other languages. Don't introduce red herrings which incite unrelated, yet strong emotional response. My argument is that scope as storage class is MORE COMPLEX in that it is orthogonal to the language, than scope as type constructor. I think users will also find it extremely unintuitive, when they realise that all the usual tools for interacting with types in the language are unavailable in this special case. I think scope will also prove to be more popular than ref, so, while you don't hear so much about how much of a disaster ref is, you'll start to hear all the exact same problems arise when people are trying to use scope, because it's a far more interesting type qualifier (and should probably be the default). ...I can't wait for 'auto scope'! ;)
Dec 06 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/6/2014 4:49 PM, Manu via Digitalmars-d wrote:
 On 7 December 2014 at 08:15, Walter Bright via Digitalmars-d
 There's a lot more more to it, but perhaps I'm among few who have had
 such extensive need to engage in these tasks?
I don't know, but it's a mystery to me what you're doing that there's no reasonable alternative, or why this is such an extensive issue for you.
 This is incorrect, you had enormous influence over the vector type in D! And
 I wish you had more.
I appreciate that. That was uncontroversial though; I didn't need to spend months or years trying to justify my claims on that issue. I feel like that was a known item that was just somewhere slightly down the list, and I was able to bring it forward. I was never in a position where I had to argue against Andrei to sell that one.
UDA was controversial, and was one of your initiatives.
 I have std.simd sitting here, and I really want to finish it, but I
 still don't have the tools to do so.
 I need, at least, forceinline to complete it, but that one *is*
 controversial - we've talked about this for years.
I proposed a DIP to fix that, but it could not get reasonable consensus. http://wiki.dlang.org/DIP56 You did, after all, convince me that we need an "always inline" and a "never inline" method.
 There is also a practical problem with GCC (perhaps it's an
 incompatbility with my std.simd design), where I need to change the
 sse-level between functions.
 There is a GCC attribute to do this ('target'), but it would rely on,
 at least, a few subtle tweaks. In this case, I need to be able to feed
 a template argument to a UDA, which doesn't work because UDA
 declarations seem to be parsed prior to knowledge of functions
 template args. There may be a further problem with GCC though which I
 haven't been able to prove yet though.
I don't know why this needs to be a deduced template argument.
 The reason for this is that an important design goal for my work was
 to be able to semi-automatically generate multiple code paths for
 different SSE versions, which can be selected at runtime based on
 available hardware.
No argument there.
 I need a little more language support. I've reached out a few times,
 but there hasn't been too much interest in the past. I really do wanna
 finish it one of these days though!
I want it done, too :-)
 I don't really know what you want. Well, perhaps a better statement is
 'why', not what.
Because a function is a function. It's a fundamental and ultimately *simple* element of a language, with simple and reliable in behaviour; you write a function, compiler emits some code with a matching symbol name. There is perhaps nothing more simple or fundamental to the language. A template is not a function. It may *generate* a function, but now we've added many extra details which require careful consideration and handling, like where the code is, or isn't, emitted? How many permutations are there? What about static/dynamic libraries? Calling with various, or arbitrary types, always required careful consideration, more so than an explicit type. Templates are a powerful (often dangerous) **tool**, which should only be applied deliberately and very carefully. Obviously auto-ref gives hard answers to some of those questions, but it is still a template, and that instantly makes it much more complex.
I view a template function as a function that takes both compile-time and run-time arguments, not as a function generator. (The latter is an implementation artifact.)
 Making a function that has nothing to do with templates into a
 template is not what I want... it couldn't be further from what I ever
 wanted!
I don't see any problem with: T func()(runtime args ...) { ... } That idiom has found various uses in practice.
 The ABI is different; there are 2 functions now (or zero, until one is
 called). Thich means taking the address of a function with auto-ref is
 a complex issue.
 ref-ness isn't a decision I ever want the compiler to make for me.
But why?
 I either want ref, or I don't, and I will state that to the compiler
 explicitly.
 This is especially true when interacting with other languages; while D
 remains in relative infancy, I think that is an overwhelmingly common
 situation when a project is modestly large.
The set of other languages that support ref parameters is [C++]. None others I know of that are accessible from D. Is it commonplace in C++ to have these overloads: T foo(S s); T foo(S& s); ? Such a practice would at least raise an eyebrow for me.
 In the situation where templates are involved, it would be nice to be
 able to make that explicit statement that some type is ref or not at
 the point of template instantiation, and the resolution should work
 according to the well-defined rules that we are all familiar with.
 Within a complex template, if logic should be performed, we have
 powerful tools to do that already; nobody ever complains about
 Unqual!, or PointerTarget!... these things are very easy to read and
 understand. ref should fit right in there with the others, is(x ==
 ref), UnRef!T, etc.
http://dlang.org/traits.html#isRef
 Instead, it's practically orthogonal to the language. Wrangling
 anything involving storage classes is really, really tedious, and
 almost always results in extensive code duplication, or text mixins
 when duplication just get too much.

 This issue really surprised me at the time, because the whole thing
 was in response to a topic that I consistently raised and pushed.
 Who was the customer welcoming that solution? I could never work it
 out; there was nobody else that seemed particularly invested in the
 problem, except the usual suite of language enthusiasts that took it
 as an interesting intellectual problem to discuss. The person who
 cared about that issue the most (afaict) was completely unsatisfied
 with the solution. I said at the time that I would have preferred that
 no action was taken, than to compound, and further set in stone, an
 issue that I was already extremely critical of.


 Don't double down on this mistake with scope!
The thing is, I don't understand *why* you want to wrangle storage classes. What is the coding pattern?
 Why does your code need to care so much about to ref or not to ref? That's
 the central point here, I think.
That same logic could ask why I want to have 'short', or 'int' in some places, but not others. There's no absolute answer. As a software engineer, and particularly, a native software engineer, it's a fundamental part of the type system (sorry, not part of the type system!) that I must have control over.
I can give practical technical reasons why for 'short' or 'int'.
 Why offer 'ref' if you don't intend people to use it?
 I put to you, why do you hate 'ref' so much? Remove it from the
 language if people aren't meant to use it.
That's the wrong question. I could ask "why do you hate ref as a storage class"?
 ref is just a pointer with some semantics removed (pointer
 re-assignment, indexing). It's use facilitates some forms of generic
 code which would fail otherwise, and also reduces some possibilities
 for end-user misuse or mistakes when using pointers (ie, user may
 intend an assignment, but unsuspectingly re-assign a pointer; result
 looks the same, but exposes bug somewhere else).

 It's also used as an optimisation where I require control over the
 ref-ness of things, but don't want to retrofit the code with a sea of
 '*' and '&' operators.

 It's also used when interfacing C++, where it appears in API's extensively.

 I'm sure there are many more common and useful cases where ref is a good tool.
And D's ref works for all that.
 The disadvantages of making it a type qualifier are:

 1. far more complexity. Type constructors interact with everything, often in
 unanticipated ways. We spent *years* working out issues with the 'const'
 type qualifier, and are still doing so. Kenji just fixed another one.
What you describe is is the opposite of 'complexity'. Uniform behaviour is what people expect from a language. I understand complexity may arise as a result of interaction with other features, but that's a worthwhile issue to explore if you ask me. That's complexity that can actually be addressed, rather than pushed to the side. Edge cases which are orthogonal to the rest of the language (ie, ref), are a terrible idea. You don't buffer against complexity by creating a whole new class of complexity, which we have no effective tools to manage or mitigate.
Uniform behavior is a nice ideal, and sounds good, but it never happens in practice with programming languages. We can't even get 'int' to behave uniformly! (Quick: what is -int.min ?) You might also consider the "uniformity" of the ref type in C++. It's awful - it's a special case EVERYWHERE in the C++ type system! It just does not fit as a type qualifier.
 2. we are never going to get users to use 'scope' qualifiers pervasively.
 It's been a long struggle to get 'const' used.
I don't think scope will be as important as const. I think we should *try it*, to experimentally see how it plays out. That said, scope should be able to be inferred quite effectively in many most cases. Again, I think we will only be able to measure the effectiveness of this when we try it.
'scope' is so important we may even consider it to be the default.
 3. we added 'inout' as a type qualifier to avoid code duplication engendered
 by 'const'. It hurts my brain to even think about how that might interact
 with 'scope' qualifiers.
inout was an interesting idea (with a terrible name!). I'm still not sure if I think it was a good idea or not, but I have found it very useful. What's the issue? I don't quite see how inout and scope overlap (no differently than const or immutable?).
Every type will have scope and nonscope variants, combinatorialy with the other type qualifiers.
 Yes, I agree unequivocably, that 'scope' as a type qualifier is more
 expressive and more powerful than as a storage class. Multiple inheritance
 is also more expressive and more powerful than single inheritance. But many
 times, more power perhaps isn't better than redoing the program design to
 use something simpler and less complex.
I don't think that's a reasonable comparison. Multiple inheritance is (I think quite well agreed) a wildly unpopular and super-complex disaster. Comparing scope-as-a-storage-class or scope-as-a-type-constructor is nothing like comparing multiple inheritence and interfaces... interfaces aren't orthogonal to the language for a start! And they're well understood, and precedented in other languages. Don't introduce red herrings which incite unrelated, yet strong emotional response.
Nobody has successfully introduced a type qualifier like scope.
 My argument is that scope as storage class is MORE COMPLEX in that it
 is orthogonal to the language, than scope as type constructor. I think
 users will also find it extremely unintuitive, when they realise that
 all the usual tools for interacting with types in the language are
 unavailable in this special case.
 I think scope will also prove to be more popular than ref, so, while
 you don't hear so much about how much of a disaster ref is, you'll
 start to hear all the exact same problems arise when people are trying
 to use scope, because it's a far more interesting type qualifier (and
 should probably be the default).

 ...I can't wait for 'auto scope'! ;)
I guess we'll see! I agree that this design is untried, and we don't really know how it will work out.
Dec 06 2014
next sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Walter Bright"  wrote in message news:m60oa4$kd5$1 digitalmars.com...

 I have std.simd sitting here, and I really want to finish it, but I
 still don't have the tools to do so.
 I need, at least, forceinline to complete it, but that one *is*
 controversial - we've talked about this for years.
I proposed a DIP to fix that, but it could not get reasonable consensus. http://wiki.dlang.org/DIP56 You did, after all, convince me that we need an "always inline" and a "never inline" method.
From the DIP:
 If a pragma specifies always inline, whether or not the target function(s) 
 are actually inlined is
 implementation defined, although the implementation will be expected to 
 inline it if practical.
 Implementations will likely vary in their ability to inline.
I expect a proposal in which 'always' means "inline or give an error" would be much better received.
 The thing is, I don't understand *why* you want to wrangle storage 
 classes. What is the coding pattern?
 That's the wrong question. I could ask "why do you hate ref as a storage 
 class"?
I suspect the answer is that sometimes it is useful to ensure some parameters are passed by reference, not by value. With ref as a type, this would be trivial: template MyParamType(T) { static if (someCriteria!T) alias MyParamType = ref(T); else alias MyParamType = T; } void myFunction(T, U, V)(const MyParamType!T, const MyParamType!U, const MyParamType!V) { ... }
Dec 06 2014
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 7 December 2014 at 05:24:20 UTC, Walter Bright wrote:
 I appreciate that. That was uncontroversial though; I didn't 
 need to
 spend months or years trying to justify my claims on that 
 issue. I
 feel like that was a known item that was just somewhere 
 slightly down
 the list, and I was able to bring it forward.
 I was never in a position where I had to argue against Andrei 
 to sell that one.
UDA was controversial, and was one of your initiatives.
I don't think UDA is controversial, but the way it was done certainly is controverted.
Dec 08 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-12-07 01:49, Manu via Digitalmars-d wrote:

 I have std.simd sitting here, and I really want to finish it, but I
 still don't have the tools to do so.
 I need, at least, forceinline to complete it, but that one *is*
 controversial - we've talked about this for years.

 GDC and LDC both have a forceinline, so I could theoretically support
 those compilers, but then I can't practically make use of them without
 some sort of attribute aliasing system, otherwise I need to triplicate
 the code for each compiler, just to insert a different (compiler
 specific) forceinline attribute name. It'd be really great if we
 agreed on just one.
I don't know about LDC but at least GDC allows you to use UDA's instead of a pragma. Then you can create a dummy attribute for DMD (and LDC): version (GNU) import gcc.attribute else { struct attribute { string attr; } } attribute("forceinline") void foo (); -- /Jacob Carlborg
Dec 07 2014
parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 Dec 2014 10:40, "Jacob Carlborg via Digitalmars-d" <
digitalmars-d puremagic.com> wrote:
 On 2014-12-07 01:49, Manu via Digitalmars-d wrote:

 I have std.simd sitting here, and I really want to finish it, but I
 still don't have the tools to do so.
 I need, at least, forceinline to complete it, but that one *is*
 controversial - we've talked about this for years.

 GDC and LDC both have a forceinline, so I could theoretically support
 those compilers, but then I can't practically make use of them without
 some sort of attribute aliasing system, otherwise I need to triplicate
 the code for each compiler, just to insert a different (compiler
 specific) forceinline attribute name. It'd be really great if we
 agreed on just one.
I don't know about LDC but at least GDC allows you to use UDA's instead
of a pragma. Then you can create a dummy attribute for DMD (and LDC):
 version (GNU)
     import gcc.attribute

 else
 {
     struct attribute
     {
         string attr;
     }
 }

  attribute("forceinline") void foo ();
You can add shorthand aliases for them too. :) forceinline void foo ();
Dec 07 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-12-07 11:50, Iain Buclaw via Digitalmars-d wrote:

 You can add shorthand aliases for them too. :)

  forceinline void foo ();
Good point. -- /Jacob Carlborg
Dec 07 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/6/14 4:49 PM, Manu via Digitalmars-d wrote:
 I need, at least, forceinline to complete it, but that one*is*
 controversial - we've talked about this for years.
I'm still 883 messages behind so here's a drive-by comment - it's time to revisit this, I think the need has become a lot clearer. -- Andrei
Dec 20 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/6/14 4:49 PM, Manu via Digitalmars-d wrote:
 In the situation where templates are involved, it would be nice to be
 able to make that explicit statement that some type is ref or not at
 the point of template instantiation, and the resolution should work
 according to the well-defined rules that we are all familiar with.
Another drive-by comment: I understand the motivation for this and the difficulties involved. There needs to be a clear understanding that adding new type qualifiers is extremely intrusive and expensive. Because of that, I think we should best address binding generation via a set of tactical tools i.e. standard library artifacts that do all that mixin business in an encapsulated and reusable manner. (As an aside forcing a template instantiation to decide ref vs. no ref should be easy but currently can't be done, i.e. this code should work but currently doesn't: T fun(T)(ref T x) { return x + 1; } void main(string[] group) { int function(int) f1 = &fun!int; int function(ref int) f2 = &fun!int; } ) Andrei
Dec 20 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 5 December 2014 at 23:58:41 UTC, Walter Bright wrote:
 On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 1) Escape detection is limited to `ref`.

     T* evil;
     ref T func(scope ref T t, ref T u)  safe {
       return t; // Error: escaping scope ref t
       return u; // ok
       evil = &u; // Error: escaping reference
     }

 vs.

     T[] evil;
     T[] func(scope T[] t, T[] u)  safe {
       return t; // Error: cannot return scope
       return u; // ok
       evil = u; // !!! not good
right, although: evil = t; // Error: not allowed
     }

 As can be seen, `ref T u` is protected from escaping (apart 
 from returning it),
 while `T[] u` in the second example is not. There's no general 
 way to express
 that `u` can only be returned from the function, but will not 
 be retained
 otherwise by storing it in a global variable. Adding `pure` 
 can express this in
 many cases, but is, of course, not always possible.
As you point out, 'ref' is designed for this.
I wouldn't call it "designed", but "repurposed"...
     struct Container(T) {
         scope ref T opIndex(size_t index);
     }

     void bar(scope ref int a);

     Container c;
     bar(c[42]);            // ok
     scope ref tmp = c[42]; // nope

 Both cases should be fine theoretically; the "real" owner 
 lives longer than
 `tmp`. Unfortunately the compiler doesn't know about this.
Right, though the compiler can optimize to produce the equivalent.
??? This is a problem on the semantic level, unrelated to optimization: // contrived example to illustrated the point Container c; scope ref x = c[42]; // not scope ref y = c[44]; // ... scope ref z = c[13]; // allowed foo(x, y, z, x+y, y+z, z+x, x+y+z); // workaround, but error-prone and has different semantics // (opIndex may have side effects, called multiple times) foo(c[42], c[44], c[13], c[42]+c[44], c[44]+c[13], c[13]+c[42], c[42]+c[44]+c[13]); // another workaround, same semantics, but ugly and unreadable (scope ref int x, scope ref int y, scope ref int z) { foo(x, y, z, x+y, y+z, z+x, x+y+z); }(c[42], c[44], c[13]);
 3) `scope` cannot be used for value types.

 I can think of a few use cases for scoped value types (RC and 
 file descriptors),
 but they might only be marginal.
I suspect that one can encapsulate such values in a struct where access to them is strictly controlled.
They can, but again at the cost of an indirection (ref).
 4) No overloading on `scope`.

 This is at least partially a consequence of `scope` inference. 
 I think
 overloading can be made to work in the presence of inference, 
 but I haven't
 thought it through.
Right. Different overloads can have different semantic implementations, so what should inference do? I also suspect it is bad style to overload on 'scope'.
It only makes sense with scope value types (see the RC example). For references, I don't know any useful applications.
 6) There seem to be problems with chaining.

     scope ref int foo();
     scope ref int bar1(ref int a) {
         return a;
     }
     scope ref int bar2(scope ref int a) {
         return a;
     }
     ref int bar3(ref int a) {
         return a;
     }
     ref int bar4(scope ref int a) {
         return a;
     }
     void baz(scope ref int a);

 Which of the following calls would work?

     foo().bar1().baz();
yes
     foo().bar2().baz();
no - cannot return scope ref parameter
     foo().bar3().baz();
yes
     foo().bar4().baz();
no, cannot return scope ref parameter
 I'm not sure I understand this fully yet, but it could be that 
 none of them work...
Well, you're half right :-)
Ok, so let's drop bar2() and bar4(). scope ref int foo(); scope ref int bar1(ref int a) { return a; } ref int bar3(ref int a) { return a; } ref int baz_noscope(/*scope*/ ref int a); foo().bar1().baz_noscope(); foo().bar3().baz_noscope(); And now? In particular, will the return value of `bar3` be treated as if it were `scope ref`?
Dec 06 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/6/2014 5:30 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 On Friday, 5 December 2014 at 23:58:41 UTC, Walter Bright wrote:
 As you point out, 'ref' is designed for this.
I wouldn't call it "designed", but "repurposed"...
Perhaps, but the original reason to even have 'ref' was so it could be a restricted pointer type that could be passed down a call hierarchy, but not up.
     struct Container(T) {
         scope ref T opIndex(size_t index);
     }

     void bar(scope ref int a);

     Container c;
     bar(c[42]);            // ok
     scope ref tmp = c[42]; // nope

 Both cases should be fine theoretically; the "real" owner lives longer than
 `tmp`. Unfortunately the compiler doesn't know about this.
Right, though the compiler can optimize to produce the equivalent.
??? This is a problem on the semantic level, unrelated to optimization: // contrived example to illustrated the point Container c; scope ref x = c[42]; // not scope ref y = c[44]; // ... scope ref z = c[13]; // allowed foo(x, y, z, x+y, y+z, z+x, x+y+z); // workaround, but error-prone and has different semantics // (opIndex may have side effects, called multiple times) foo(c[42], c[44], c[13], c[42]+c[44], c[44]+c[13], c[13]+c[42], c[42]+c[44]+c[13]); // another workaround, same semantics, but ugly and unreadable (scope ref int x, scope ref int y, scope ref int z) { foo(x, y, z, x+y, y+z, z+x, x+y+z); }(c[42], c[44], c[13]);
You are correct, and it remains to be seen if these occur enough to be a problem or not. The workarounds do exist, though. Another workaround: scope ref tmp = c[42]; becomes: auto scope ref tmp() { return c[42]; }
 Ok, so let's drop bar2() and bar4().

      scope ref int foo();
      scope ref int bar1(ref int a) {
          return a;
      }
      ref int bar3(ref int a) {
          return a;
      }
      ref int baz_noscope(/*scope*/ ref int a);

      foo().bar1().baz_noscope();
      foo().bar3().baz_noscope();

 And now? In particular, will the return value of `bar3` be treated as if it
were
 `scope ref`?
Yes.
Dec 06 2014
prev sibling next sibling parent reply "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Friday, 5 December 2014 at 23:58:41 UTC, Walter Bright wrote:
 On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
     scope ref int foo();
     scope ref int bar1(ref int a) {
         return a;
     }
     scope ref int bar2(scope ref int a) {
         return a;
     }
     ref int bar3(ref int a) {
         return a;
     }
     ref int bar4(scope ref int a) {
         return a;
     }
     void baz(scope ref int a);

 Which of the following calls would work?

     foo().bar1().baz();
yes
     foo().bar2().baz();
no - cannot return scope ref parameter
     foo().bar3().baz();
yes
     foo().bar4().baz();
no, cannot return scope ref parameter
I understand that scope will not allow the contents of the variable to escape the lifetime of a declaration. But can you explain why bar1() works, but bar2() doesn't? Isn't the body of bar2() in the line `foo().bar2();` part of the declaration? Besides, what does it mean to return a `scope ref int`? Does it mean that the content of the variable that is returned is not allowed to escape the scope of the calling site? Huh? It seemed so easy when you gave the example. On Friday, 5 December 2014 at 20:55:55 UTC, Walter Bright wrote:
 It means that this code will be safe:

    void foo(scope int* p);

    p = malloc(n);
    foo(p);
    free(p);

 The rest is all the nuts and bolts of making that work.
Dec 07 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/7/2014 2:46 AM, Sebastiaan Koppe wrote:
 On Friday, 5 December 2014 at 23:58:41 UTC, Walter Bright wrote:
 On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
     scope ref int foo();
     scope ref int bar1(ref int a) {
         return a;
     }
     scope ref int bar2(scope ref int a) {
         return a;
     }
     ref int bar3(ref int a) {
         return a;
     }
     ref int bar4(scope ref int a) {
         return a;
     }
     void baz(scope ref int a);

 Which of the following calls would work?

     foo().bar1().baz();
yes
     foo().bar2().baz();
no - cannot return scope ref parameter
     foo().bar3().baz();
yes
     foo().bar4().baz();
no, cannot return scope ref parameter
I understand that scope will not allow the contents of the variable to escape the lifetime of a declaration. But can you explain why bar1() works, but bar2() doesn't?
A 'scope ref' parameter may not be returned as a 'ref' or a 'scope ref'.
 Isn't the body of bar2() in the line `foo().bar2();` part of the
 declaration?

 Besides, what does it mean to return a `scope ref int`? Does it mean that the
 content of the variable that is returned is not allowed to escape the scope of
 the calling site? Huh?
It means the reference itself (the pointer) does not escape.
Dec 08 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 8 December 2014 at 21:16:36 UTC, Walter Bright wrote:
 A 'scope ref' parameter may not be returned as a 'ref' or a 
 'scope ref'.
It can safely be returned if you consider its lifetime as the intersection of the lifetime of the function's parameter.
Dec 08 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 3:21 PM, deadalnix wrote:
 On Monday, 8 December 2014 at 21:16:36 UTC, Walter Bright wrote:
 A 'scope ref' parameter may not be returned as a 'ref' or a 'scope ref'.
It can safely be returned if you consider its lifetime as the intersection of the lifetime of the function's parameter.
The only purpose to a 'scope ref' parameter is to say it isn't being returned. 'ref' itself does not escape in any way other than by returning.
Dec 09 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 10 December 2014 at 05:23:29 UTC, Walter Bright 
wrote:
 On 12/8/2014 3:21 PM, deadalnix wrote:
 On Monday, 8 December 2014 at 21:16:36 UTC, Walter Bright 
 wrote:
 A 'scope ref' parameter may not be returned as a 'ref' or a 
 'scope ref'.
It can safely be returned if you consider its lifetime as the intersection of the lifetime of the function's parameter.
The only purpose to a 'scope ref' parameter is to say it isn't being returned. 'ref' itself does not escape in any way other than by returning.
That is a completely useless feature. Also, you want to have scope return for container like thing.
Dec 09 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/9/2014 10:43 PM, deadalnix wrote:
 On Wednesday, 10 December 2014 at 05:23:29 UTC, Walter Bright wrote:
 On 12/8/2014 3:21 PM, deadalnix wrote:
 On Monday, 8 December 2014 at 21:16:36 UTC, Walter Bright wrote:
 A 'scope ref' parameter may not be returned as a 'ref' or a 'scope ref'.
It can safely be returned if you consider its lifetime as the intersection of the lifetime of the function's parameter.
The only purpose to a 'scope ref' parameter is to say it isn't being returned. 'ref' itself does not escape in any way other than by returning.
That is a completely useless feature. Also, you want to have scope return for container like thing.
I disagree. It's critical for chaining one function to the next.
Dec 10 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright 
wrote:
 I disagree. It's critical for chaining one function to the next.
I one can't return, one can't chain.
Dec 10 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/10/2014 8:56 PM, deadalnix wrote:
 On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright wrote:
 I disagree. It's critical for chaining one function to the next.
I one can't return, one can't chain.
I guess I'm not seeing the problem.
Dec 10 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 07:30:03 UTC, Walter Bright 
wrote:
 On 12/10/2014 8:56 PM, deadalnix wrote:
 On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright 
 wrote:
 I disagree. It's critical for chaining one function to the 
 next.
I one can't return, one can't chain.
I guess I'm not seeing the problem.
a.foo().bar() is the same as bar(foo(a)), that is, using the returned value as first parameter. If you can't return, you can't chain.
Dec 10 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/10/2014 11:55 PM, deadalnix wrote:
 On Thursday, 11 December 2014 at 07:30:03 UTC, Walter Bright wrote:
 On 12/10/2014 8:56 PM, deadalnix wrote:
 On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright wrote:
 I disagree. It's critical for chaining one function to the next.
I one can't return, one can't chain.
I guess I'm not seeing the problem.
a.foo().bar() is the same as bar(foo(a)), that is, using the returned value as first parameter. If you can't return, you can't chain.
Right, and you can chain when the declarations are: scope ref S foo(ref T a); U bar(ref S s); a.foo().bar(); // works
Dec 11 2014
prev sibling parent Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 05/12/2014 23:58, Walter Bright wrote:
 2) `scope ref` return values cannot be stored.

      scope ref int foo();
      void bar(scope ref int a);

      foo().bar();        // allowed
      scope tmp = foo();  // not allowed
      tmp.bar();
Right
From the DIP: "The lifetime of a scope return value is the lifetime of an rvalue. It may not be copied in a way that extends its life." With part of the example: scope int* foo(); ... int* p = foo(); // Error, lifetime(p) is &infin; Maybe the error should be 'scope return value cannot be stored', because otherwise p could be inferred as scope.
Dec 07 2014
prev sibling parent reply "Daniel N" <ufo orbiting.us> writes:
On Friday, 5 December 2014 at 16:48:45 UTC, Marc Schütz wrote:
 There are limitations this proposal has in comparison to my 
 original one. These limitations might of course be harmless and 
 play no role in practice, but on the other hand, they may, so I 
 think it's good to list them here.
One concern I had with your proposal was that it refers to a symbol before it's available. scope!haystack(string) findSubstring(scope(string) haystack, scope(string) needle) C++11 had similar issues and they solved it by introducing trailing return types. ex: "template<class T> auto mul(T a, T b) -> decltype(a*b)" Personally I would prefer not to go down that lane and DIP69 avoids that problem.
Dec 06 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 6 December 2014 at 10:59:24 UTC, Daniel N wrote:
 On Friday, 5 December 2014 at 16:48:45 UTC, Marc Schütz wrote:
 There are limitations this proposal has in comparison to my 
 original one. These limitations might of course be harmless 
 and play no role in practice, but on the other hand, they may, 
 so I think it's good to list them here.
One concern I had with your proposal was that it refers to a symbol before it's available. scope!haystack(string) findSubstring(scope(string) haystack, scope(string) needle) C++11 had similar issues and they solved it by introducing trailing return types. ex: "template<class T> auto mul(T a, T b) -> decltype(a*b)" Personally I would prefer not to go down that lane and DIP69 avoids that problem.
For D, this wouldn't be necessary, because parsing and semantic analysis are strictly separated. The owners would only have to be evaluated very late during the semantic phase. But let's see how DIP69 works out...
Dec 06 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.

 This, among other things, makes a ref counting type practical. 
 It also makes it more practical to use other storage allocation 
 schemes than garbage collection.

 It does not make scope into a type constructor, nor a general 
 type-annotation system.

 It does not provide an ownership system, though it would 
 complement one.
Thanks a lot of trying to move forward with this. Also glad to see DIP36 didn't vanish completely useless :P It will take me some time to provide a detailed response so I'll post a summary of my impression first. I recognize and respect your attempt to go for most simple solution that is still useful in practice. Can't say I am happy about it but it is better to have something working than awesome plans that never get implemented. In this context trying to get most of scope as storage class seems right. But from existing cases it doesn't seem working good enough. For example, not being able to represent idiom of `scope ref int foo(scope ref int x) { return x; }` seems very limiting. There are also issues that pop up because of missing transitivity. Maybe this can be fixed within existing proposal, maybe not. Right now I don't have any strong opinion. I also don't consider `ref` design as a storage class any kind of success at all and generally agree with Manu on this topic. At the same time alternative proposals that make it a qualifier (like Marc did) do impact existing language much more and this no small concern. This won't be easy.
Dec 07 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. For example, not
 being able to represent idiom of `scope ref int foo(scope ref int x) { return
x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
 I also don't consider `ref` design as a storage class any kind of success at
all
 and generally agree with Manu on this topic. At the same time alternative
 proposals that make it a qualifier (like Marc did) do impact existing language
 much more and this no small concern.
My experience with C++ ref as type qualifier is very, very bad. It's a special case EVERYWHERE. Doing type deduction with it is an exercise in a completely baffling set of rules and a different rule for every occasion - Scott Meyers has a great piece on this. There are probably only a handful of people on the planet who actually understand C++ ref. I wished very hard to avoid that with D ref.
Dec 07 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 There are probably only a handful of people on the planet who 
 actually understand C++ ref. I wished very hard to avoid that 
 with D ref.
When C++ programmers say that D-style ranges can't do everything C++ iterators can do, they seem to miss that sometimes it's a good idea to adopt a simpler language feature, that doesn't cover 100% usages, if it covers 80-90% of the cases, and has a simpler syntax, and simpler semantics to understand for the programmer. (The comment above is not about DIP69). Bye, bearophile
Dec 07 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/7/2014 2:58 PM, bearophile wrote:
 When C++ programmers say that D-style ranges can't do everything C++ iterators
 can do, they seem to miss that sometimes it's a good idea to adopt a simpler
 language feature, that doesn't cover 100% usages, if it covers 80-90% of the
 cases, and has a simpler syntax, and simpler semantics to understand for the
 programmer.
I agree, but it's hard to find that sweet spot. I think Java definitely went too far, and Go went too far for my taste.
 (The comment above is not about DIP69).
Yes, it is :-)
Dec 07 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/7/14 4:29 PM, Walter Bright wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. For
 example, not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
So: int x; foo(x) += 1; will compile? I was under the impression this would be disallowed. If you do not connect the scope to the parameter, then the caller has to assume that variable can escape, and you can't use scoped variables as arguments at all.
 My experience with C++ ref as type qualifier is very, very bad. It's a
 special case EVERYWHERE. Doing type deduction with it is an exercise in
 a completely baffling set of rules and a different rule for every
 occasion - Scott Meyers has a great piece on this.

 There are probably only a handful of people on the planet who actually
 understand C++ ref. I wished very hard to avoid that with D ref.
D has so many features that did not exist when ref was created (as inout in D1), that you can ALMOST duplicate ref. The only thing we could not duplicate is the implicit address-taking on construction, which maybe is not such a bad thing. -Steve
Dec 08 2014
parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 08/12/2014 15:53, Steven Schveighoffer wrote:
    scope ref int foo(ref int x);

 will do it.
So: int x; foo(x) += 1; will compile?
Yes, because foo's argument is not scope, it can be returned.
Dec 09 2014
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/9/14 9:23 AM, Nick Treleaven wrote:
 On 08/12/2014 15:53, Steven Schveighoffer wrote:
    scope ref int foo(ref int x);

 will do it.
So: int x; foo(x) += 1; will compile?
Yes, because foo's argument is not scope, it can be returned.
But I thought if you take a reference from the stack, it's inferred as scope? I feel like there's a missing link somewhere if the scope is missing from the parameter. Will ref just automatically bind to any scoped reference? What about this: auto y = &x; foo(*y) += 1; -Steve
Dec 09 2014
next sibling parent Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 09/12/2014 16:25, Steven Schveighoffer wrote:
 But I thought if you take a reference from the stack, it's inferred as
 scope? I feel like there's a missing link somewhere if the scope is
 missing from the parameter.
I think (with this DIP) values on the stack can safely be passed as a ref parameter, which is a bit more flexible than a scope parameter.
 Will ref just automatically bind to any scoped reference?

 What about this:
(for reference:) int x; scope ref int foo(ref int x);
 auto y = &x;

 foo(*y) += 1;
y gets inferred as scope, but *y is not scope, so it should work.
Dec 09 2014
prev sibling parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 09/12/2014 16:25, Steven Schveighoffer wrote:
 Will ref just automatically bind to any scoped reference?
A ref parameter essentially is a scope ref parameter, but it can also be returned: http://forum.dlang.org/post/m64v3g$2bga$1 digitalmars.com
Dec 11 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 14:12:16 UTC, Nick Treleaven 
wrote:
 On 09/12/2014 16:25, Steven Schveighoffer wrote:
 Will ref just automatically bind to any scoped reference?
A ref parameter essentially is a scope ref parameter, but it can also be returned: http://forum.dlang.org/post/m64v3g$2bga$1 digitalmars.com
Introducing a keyword to be able to add arbitrary restriction that do not make the code safer, more optimization or anything is not paying for itself.
Dec 11 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Sunday, 7 December 2014 at 21:29:50 UTC, Walter Bright wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. 
 For example, not
 being able to represent idiom of `scope ref int foo(scope ref 
 int x) { return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
This isn't the same as it does not propagate scope but just restricts return value. Difference is that it cannot be chained. Let's consider practical example based on Phobos: there was an issue with byLine range that it has reused same buffer internally which sometimes caught users off guard when trying to save slice. It is a natural fit for `scope` - make it return `scope string` instead to ensure that no slices get stored. Two issues immediately pop up: 1) scope is not transitive thus it doesn't work at all - you still can store slice of `scope string` as only actual ptr+length struct is protected. 2) even if it worked, existing definition of scope return value makes it impossible to use in typical idiomatic pipeline: `file.byLine.algo1.algo2`. Either algoX is defined to take `scope ref` and thus can't return it or it is defined to take `ref` and can't take another `scope ref` as an argument. At least this is what I get from reading existing examples in DIP69
 I also don't consider `ref` design as a storage class any kind 
 of success at all
 and generally agree with Manu on this topic. At the same time 
 alternative
 proposals that make it a qualifier (like Marc did) do impact 
 existing language
 much more and this no small concern.
My experience with C++ ref as type qualifier is very, very bad. It's a special case EVERYWHERE. Doing type deduction with it is an exercise in a completely baffling set of rules and a different rule for every occasion - Scott Meyers has a great piece on this. There are probably only a handful of people on the planet who actually understand C++ ref. I wished very hard to avoid that with D ref.
While there is no argument that C++ ref is screwed, it is rather hard to say if this is inherent consequence of ref being a type qualifier or just C++ being C++. I mean how many C++ type system features in general are understood my more than a handful of people on the planet? For me `ref` is essentially just a different flavor of `*` - and if the latter can be part of type, I see no reasons why former can't
Dec 08 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 8 December 2014 at 16:25:22 UTC, Dicebot wrote:
 On Sunday, 7 December 2014 at 21:29:50 UTC, Walter Bright wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. 
 For example, not
 being able to represent idiom of `scope ref int foo(scope ref 
 int x) { return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
This isn't the same as it does not propagate scope but just restricts return value. Difference is that it cannot be chained. Let's consider practical example based on Phobos: there was an issue with byLine range that it has reused same buffer internally which sometimes caught users off guard when trying to save slice. It is a natural fit for `scope` - make it return `scope string` instead to ensure that no slices get stored. Two issues immediately pop up: 1) scope is not transitive thus it doesn't work at all - you still can store slice of `scope string` as only actual ptr+length struct is protected. 2) even if it worked, existing definition of scope return value makes it impossible to use in typical idiomatic pipeline: `file.byLine.algo1.algo2`. Either algoX is defined to take `scope ref` and thus can't return it or it is defined to take `ref` and can't take another `scope ref` as an argument.
That's why I asked the question in http://forum.dlang.org/post/xdjsmwocbtxovjnathek forum.dlang.org . It seems Walter wants to allow passing `scope ref` to `ref`, but then automatically treat a normal `ref` of the second function as if it were `scope ref`. But I can't quite see through it :-(
 At least this is what I get from reading existing examples in 
 DIP69

 I also don't consider `ref` design as a storage class any 
 kind of success at all
 and generally agree with Manu on this topic. At the same time 
 alternative
 proposals that make it a qualifier (like Marc did) do impact 
 existing language
 much more and this no small concern.
My experience with C++ ref as type qualifier is very, very bad. It's a special case EVERYWHERE. Doing type deduction with it is an exercise in a completely baffling set of rules and a different rule for every occasion - Scott Meyers has a great piece on this. There are probably only a handful of people on the planet who actually understand C++ ref. I wished very hard to avoid that with D ref.
While there is no argument that C++ ref is screwed, it is rather hard to say if this is inherent consequence of ref being a type qualifier or just C++ being C++. I mean how many C++ type system features in general are understood my more than a handful of people on the planet? For me `ref` is essentially just a different flavor of `*` - and if the latter can be part of type, I see no reasons why former can't
I think most problems would come from type deduction. All the other qualifiers and function attributes, like const and pure, just either work, or they don't, so they can be tested easily, and you want to apply these qualifiers wherever possible. This also applies to scope. But for `ref`, it's presence influences behaviour, therefore I think it would be best if it were never inferred implicitly. Actually that too is just what we do with pointers. We don't change non-pointer types to pointers automatically.
Dec 08 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 16:57:44 UTC, Marc Schütz wrote:
 Two issues immediately pop up:

 1) scope is not transitive thus it doesn't work at all - you 
 still can store slice of `scope string` as only actual 
 ptr+length struct is protected.

 2) even if it worked, existing definition of scope return 
 value makes it impossible to use in typical idiomatic 
 pipeline: `file.byLine.algo1.algo2`. Either algoX is defined 
 to take `scope ref` and thus can't return it or it is defined 
 to take `ref` and can't take another `scope ref` as an 
 argument.
That's why I asked the question in http://forum.dlang.org/post/xdjsmwocbtxovjnathek forum.dlang.org . It seems Walter wants to allow passing `scope ref` to `ref`, but then automatically treat a normal `ref` of the second function as if it were `scope ref`. But I can't quite see through it :-(
Yes I have seen that thread and have no idea what Walter has meant either. His answer only makes sense if return value scope is supposed to be inferred from body - what when bodies are guaranteed to be there everything can be inferred anyway.
Dec 08 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 8:25 AM, Dicebot wrote:
 2) even if it worked, existing definition of scope return value makes it
 impossible to use in typical idiomatic pipeline: `file.byLine.algo1.algo2`.
 Either algoX is defined to take `scope ref` and thus can't return it or it is
 defined to take `ref` and can't take another `scope ref` as an argument.

 At least this is what I get from reading existing examples in DIP69
The difference between 'scope ref' and 'ref' parameters is that the former cannot be returned by reference. The difference between 'scope ref' and 'ref' function returns is that the former cannot be saved by the caller. You can still safely pass the address of a stack variable by 'ref' - it will not escape. Under the current proposal, as now, you cannot store a ref by ref, and cannot take the address of a ref variable.
 There are probably only a handful of people on the planet who actually
 understand C++ ref. I wished very hard to avoid that with D ref.
While there is no argument that C++ ref is screwed, it is rather hard to say if this is inherent consequence of ref being a type qualifier or just C++ being C++.
It's not because C++ designers are idiots. It inevitably follows from what ref is. ref grew, brick by brick, into a monster, each brick being inevitable.
 I mean how many C++ type system features in general are understood my more
 than a handful of people on the planet? For me `ref` is essentially just a
 different flavor of `*` - and if the latter can be part of type, I see no
 reasons why former can't
I agree it's a seductively simple idea. The trouble starts happening when you start when making ref idempotent, when ref can only be at the 'head' of a data structure, when trying to do type deduction of a ref type (do you get the ref, or do you look 'through' the ref?), what happens with overloading, etc., and on and on.
Dec 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 19:44:48 UTC, Walter Bright wrote:
 The difference between 'scope ref' and 'ref' parameters is that 
 the former cannot be returned by reference.

 The difference between 'scope ref' and 'ref' function returns 
 is that the former cannot be saved by the caller.

 You can still safely pass the address of a stack variable by 
 'ref' - it will not escape. Under the current proposal, as now, 
 you cannot store a ref by ref, and cannot take the address of a 
 ref variable.
Easier to go straight with pseudo-code: struct ByLine { scope string front(); // ... } auto byLine(File file) { return ByLine(file); } scope /* ref */ string foo(scope /* ref */ string input) { return input[1..$]; } void main() { auto r = file.byLine.map!foo; string s = r.front; // this should not compile string s = r.front.dup; // this should compile // how foo signature should look like for this to work? }
 I mean how many C++ type system features in general are 
 understood my more
 than a handful of people on the planet? For me `ref` is 
 essentially just a
 different flavor of `*` - and if the latter can be part of 
 type, I see no
 reasons why former can't
I agree it's a seductively simple idea. The trouble starts happening when you start when making ref idempotent, when ref can only be at the 'head' of a data structure, when trying to do type deduction of a ref type (do you get the ref, or do you look 'through' the ref?), what happens with overloading, etc., and on and on.
But was there any reason why those traits (alien to type qualifiers) were pursued? What is the problem with `ref` simply meaning `non-null pointer` and allowing non-idempotent ref(ref(int))?
Dec 08 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 12:54 PM, Dicebot wrote:
 On Monday, 8 December 2014 at 19:44:48 UTC, Walter Bright wrote:
 I agree it's a seductively simple idea. The trouble starts happening when you
 start when making ref idempotent, when ref can only be at the 'head' of a data
 structure, when trying to do type deduction of a ref type (do you get the ref,
 or do you look 'through' the ref?), what happens with overloading, etc., and
 on and on.
But was there any reason why those traits (alien to type qualifiers) were pursued? What is the problem with `ref` simply meaning `non-null pointer` and allowing non-idempotent ref(ref(int))?
Because it isn't just a non-null pointer (and ref's can still be null in C++!), it's an auto-dereferencing pointer.
Dec 08 2014
next sibling parent reply =?UTF-8?Q?Tobias=20M=C3=BCller?= <troplin bluewin.ch> writes:
Walter Bright <newshound2 digitalmars.com> wrote:
[...]
 and ref's can still be null in C++!
AFAIK only if you dereference a NULL pointer, which is UB. So not really. [...] Tobi
Dec 08 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 1:59 PM, Tobias Müller wrote:
 Walter Bright <newshound2 digitalmars.com> wrote:
 [...]
 and ref's can still be null in C++!
AFAIK only if you dereference a NULL pointer, which is UB. So not really.
Saying it's UB doesn't help in the slightest. You can still have null ref's, and no compiler will prevent it.
Dec 08 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 21:07:39 UTC, Walter Bright wrote:
 But was there any reason why those traits (alien to type 
 qualifiers) were
 pursued? What is the problem with `ref` simply meaning 
 `non-null pointer` and
 allowing non-idempotent ref(ref(int))?
Because it isn't just a non-null pointer (and ref's can still be null in C++!), it's an auto-dereferencing pointer.
Don't want to distract from topic that actually matters but would be nice to ask you few more question on topic at some point.
Dec 08 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 12:54 PM, Dicebot wrote:
 struct ByLine
 {
      scope string front();
      // ...
 }

 auto byLine(File file)
 {
      return ByLine(file);
 }

 scope /* ref */ string foo(scope /* ref */ string input)
 {
      return input[1..$];
 }

 void main()
 {
      auto r = file.byLine.map!foo;
      string s = r.front; // this should not compile
      string s = r.front.dup; // this should compile

      // how foo signature should look like for this to work?
 }
front() should return a 'scope ref string'.
Dec 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 21:12:47 UTC, Walter Bright wrote:
 On 12/8/2014 12:54 PM, Dicebot wrote:
 struct ByLine
 {
     scope string front();
     // ...
 }

 auto byLine(File file)
 {
     return ByLine(file);
 }

 scope /* ref */ string foo(scope /* ref */ string input)
 {
     return input[1..$];
 }

 void main()
 {
     auto r = file.byLine.map!foo;
     string s = r.front; // this should not compile
     string s = r.front.dup; // this should compile

     // how foo signature should look like for this to work?
 }
front() should return a 'scope ref string'.
That seems to contradict your other statement:
 A 'scope ref' parameter may not be returned as a 'ref' or a 
 'scope ref'.
Please check `foo()` once more - it needs to accept scope (ref) to be able to accept ByLine.front as an argument. And it also needs to pass it down the call chain - but returning `input` by reference is illegal according to abovementioned rule.
Dec 08 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 3:14 PM, Dicebot wrote:
 On Monday, 8 December 2014 at 21:12:47 UTC, Walter Bright wrote:
 On 12/8/2014 12:54 PM, Dicebot wrote:
 struct ByLine
 {
     scope string front();
     // ...
 }

 auto byLine(File file)
 {
     return ByLine(file);
 }

 scope /* ref */ string foo(scope /* ref */ string input)
 {
     return input[1..$];
 }

 void main()
 {
     auto r = file.byLine.map!foo;
     string s = r.front; // this should not compile
     string s = r.front.dup; // this should compile

     // how foo signature should look like for this to work?
 }
front() should return a 'scope ref string'.
That seems to contradict your other statement:
 A 'scope ref' parameter may not be returned as a 'ref' or a 'scope ref'.
Just make it a 'ref' parameter.
 Please check `foo()` once more - it needs to accept scope (ref) to be able to
 accept ByLine.front as an argument. And it also needs to pass it down the call
 chain - but returning `input` by reference is illegal according to
 abovementioned rule.
It can still be passed down as a 'ref' parameter.
Dec 09 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 December 2014 at 22:24:51 UTC, Walter Bright wrote:
 front() should return a 'scope ref string'.
That seems to contradict your other statement:
 A 'scope ref' parameter may not be returned as a 'ref' or a 
 'scope ref'.
Just make it a 'ref' parameter.
 Please check `foo()` once more - it needs to accept scope 
 (ref) to be able to
 accept ByLine.front as an argument. And it also needs to pass 
 it down the call
 chain - but returning `input` by reference is illegal 
 according to
 abovementioned rule.
It can still be passed down as a 'ref' parameter.
But as far as I understand the spec it will result it this code failing too: auto r = ["aaa", "bbb", "ccc"].map!foo; // should compile but will fail because foo returns scope ref: string s = r.front; What I mean is that in current proposal it is impossible to transfer scope information down the call chain - either function always returns scope ref or never. Which implies the necessity of something like `auto scope`...
Dec 09 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/9/2014 8:34 PM, Dicebot wrote:
 But as far as I understand the spec it will result it this code failing too:

 auto r = ["aaa", "bbb", "ccc"].map!foo;
 // should compile but will fail because foo returns scope  ref:
 string s = r.front;

 What I mean is that in current proposal it is impossible to transfer scope
 information down the call chain - either function always returns scope ref or
 never. Which implies the necessity of something like `auto scope`...
If you want data to 'escape' from r.front, then it shouldn't be marked as scope. Definitely, using scope successfully will require some rethinking of how code is written.
Dec 10 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 11 December 2014 at 03:30:07 UTC, Walter Bright 
wrote:
 If you want data to 'escape' from r.front, then it shouldn't be 
 marked as scope. Definitely, using scope successfully will 
 require some rethinking of how code is written.
Allowing or prohibiting escaping of r.front in that example depends on definition of mapped range. There is nothing to "rethink" about it - case is almost identical to const+inout.
Dec 10 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/10/2014 10:43 PM, Dicebot wrote:
 On Thursday, 11 December 2014 at 03:30:07 UTC, Walter Bright wrote:
 If you want data to 'escape' from r.front, then it shouldn't be marked as
 scope. Definitely, using scope successfully will require some rethinking of
 how code is written.
Allowing or prohibiting escaping of r.front in that example depends on definition of mapped range. There is nothing to "rethink" about it - case is almost identical to const+inout.
One reason why with templates, they'll do scope inference.
Dec 11 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 11 December 2014 at 08:30:55 UTC, Walter Bright 
wrote:
 On 12/10/2014 10:43 PM, Dicebot wrote:
 On Thursday, 11 December 2014 at 03:30:07 UTC, Walter Bright 
 wrote:
 If you want data to 'escape' from r.front, then it shouldn't 
 be marked as
 scope. Definitely, using scope successfully will require some 
 rethinking of
 how code is written.
Allowing or prohibiting escaping of r.front in that example depends on definition of mapped range. There is nothing to "rethink" about it - case is almost identical to const+inout.
One reason why with templates, they'll do scope inference.
So the answer really is "current proposal does not support it"? Well this alone seems limiting enough for me to kills the whole DIP. "Use templates everywhere" is not really an answer, we could as well just prohibit separate compilation without .di files and enjoy full source availability as inherent requirement.
Dec 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 10:04 PM, Dicebot wrote:
 On Thursday, 11 December 2014 at 08:30:55 UTC, Walter Bright wrote:
 On 12/10/2014 10:43 PM, Dicebot wrote:
 On Thursday, 11 December 2014 at 03:30:07 UTC, Walter Bright wrote:
 If you want data to 'escape' from r.front, then it shouldn't be marked as
 scope. Definitely, using scope successfully will require some rethinking of
 how code is written.
Allowing or prohibiting escaping of r.front in that example depends on definition of mapped range. There is nothing to "rethink" about it - case is almost identical to const+inout.
One reason why with templates, they'll do scope inference.
So the answer really is "current proposal does not support it"? Well this alone seems limiting enough for me to kills the whole DIP. "Use templates everywhere" is not really an answer, we could as well just prohibit separate compilation without .di files and enjoy full source availability as inherent requirement.
When you're relying on the definition of an argument, it is inherently a template.
Dec 11 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 8 December 2014 at 20:54:54 UTC, Dicebot wrote:
 But was there any reason why those traits (alien to type 
 qualifiers) were pursued? What is the problem with `ref` simply 
 meaning `non-null pointer` and allowing non-idempotent 
 ref(ref(int))?
Please no. when you do int a; and then use a, you always either refer to a, to memory storage (lvalue) or a, the value stored in that memory (the rvalue). When doing ref int a = xxx; You specify that you don't create a new storage for a, but that you must consider xxx as an lvalue, and bind the name a to that same lvalue. Once you get that, you get why ref(ref(int)) do not make any sense, and is generally undesirable.
Dec 08 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 23:19:26 UTC, deadalnix wrote:
 On Monday, 8 December 2014 at 20:54:54 UTC, Dicebot wrote:
 But was there any reason why those traits (alien to type 
 qualifiers) were pursued? What is the problem with `ref` 
 simply meaning `non-null pointer` and allowing non-idempotent 
 ref(ref(int))?
Please no. when you do int a; and then use a, you always either refer to a, to memory storage (lvalue) or a, the value stored in that memory (the rvalue). When doing ref int a = xxx; You specify that you don't create a new storage for a, but that you must consider xxx as an lvalue, and bind the name a to that same lvalue.
My thoughts have went totally different direction: ref(int) a = &xxx; // ok ref(int) a = xxx; // bad, can't initialize ref from value ref(int) a = intptr; // ok, implicitly inject assert(intptr) ref(int) a = null; // bad, can't initialize ref from literal Now I see how issue may arise because of implicit conversion to matching value type (when passing to as function argument for example) but we do already have structs with `alias this` which may need similar semantics - it needs to be defined anyway. This is still offtopic. I support attempt to make scope a storage class in general because this is an approach of minimal change and still can be useful. Problem with defining it so that it actually is useful.
Dec 08 2014
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 8 December 2014 at 16:25:22 UTC, Dicebot wrote:
 This isn't the same as it does not propagate scope but just 
 restricts return value. Difference is that it cannot be 
 chained. Let's consider practical example based on Phobos:

 there was an issue with byLine range that it has reused same 
 buffer internally which sometimes caught users off guard when 
 trying to save slice. It is a natural fit for `scope` - make it 
 return `scope string` instead to ensure that no slices get 
 stored.
That would only ensure that slice do not outline the range.
 Two issues immediately pop up:

 1) scope is not transitive thus it doesn't work at all - you 
 still can store slice of `scope string` as only actual 
 ptr+length struct is protected.
Yes, that is my whole point whith the scope flag in expression. Without it, the proposal need hacks to work (&(*e)) and is too restrictive to be useful.
 While there is no argument that C++ ref is screwed, it is 
 rather hard to say if this is inherent consequence of ref being 
 a type qualifier or just C++ being C++. I mean how many C++ 
 type system features in general are understood my more than a 
 handful of people on the planet? For me `ref` is essentially 
 just a different flavor of `*` - and if the latter can be part 
 of type, I see no reasons why former can't
No, a type constructor is not the right tool. This about it. scope(int)[] do not make any sense as the slice cannot outlive its content safely. You need to have a flag on expression and declarations to make this work.
Dec 08 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 7 December 2014 at 21:29:50 UTC, Walter Bright wrote:
 My experience with C++ ref as type qualifier is very, very bad. 
 It's a special case EVERYWHERE. Doing type deduction with it is 
 an exercise in a completely baffling set of rules and a 
 different rule for every occasion - Scott Meyers has a great 
 piece on this.

 There are probably only a handful of people on the planet who 
 actually understand C++ ref. I wished very hard to avoid that 
 with D ref.
Type qualifier for scope is the wrong tool. scope(int)[] do not make any sense. The slice cannot outlive its content. scope as a flag on expressions/symbols is very useful.
Dec 08 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 23:05:49 UTC, deadalnix wrote:
 scope(int)[] do not make any sense. The slice cannot outlive its
 content. scope as a flag on expressions/symbols is very useful.
I'd say it strange for directly the opposite reason - pure value type is inherently scope so adding such annotation shouldn't matter in general.
Dec 08 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. For example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
 I also don't consider `ref` design as a storage class any kind of success
 at all
 and generally agree with Manu on this topic. At the same time alternative
 proposals that make it a qualifier (like Marc did) do impact existing
 language
 much more and this no small concern.
My experience with C++ ref as type qualifier is very, very bad. It's a special case EVERYWHERE. Doing type deduction with it is an exercise in a completely baffling set of rules and a different rule for every occasion - Scott Meyers has a great piece on this. There are probably only a handful of people on the planet who actually understand C++ ref. I wished very hard to avoid that with D ref.
I do completely sympathise with you on this point. I understand where you were coming from on 'ref', and I do applaud the effort. Nobody said that ref should be just the same as C++, but I am saying that the D solution is not a success. I'm all about a ref that improves on C++, but I don't think that's what we have. We have something that's just complex and unwieldy in a different (and for my money, even more frustrating) way. I'm just saying, don't repeat that mistake with scope! There must be another way... I'm also quite uneasy with the fact that scope would not be transitive as a storage class. What happens when it's applied to a value type, like a struct, that contains some pointers? An adaptation wrapper for a single pointer is super common; ie, a tiny struct passed by value with scope needs to have it's contained pointer receive the scope-ness of the argument. I don't have the perfect proposal, but I feel very strongly about 2 things: 1. It must not be a storage class; the concept was a disaster with ref, and I struggle with this more frequently than any other 'feature' in D. 2. I feel it's a big mistake to separate it from the type system, which I think most agree, is D's greatest asset by far. Manipulating types is simple and convenient using the type system, and I think manipulating scope will be just as important as any other attribute as soon as even slightly complex use cases begin to arise.
Dec 11 2014
next sibling parent reply "ixid" <nuaccount gmail.com> writes:
Manu have you shown Walter some of the code where ref is 
problematic? He seems to have asked to see examples repeatedly as 
he's not convinced there is a problem.
Dec 11 2014
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 December 2014 at 23:22, ixid via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Manu have you shown Walter some of the code where ref is problematic? He
 seems to have asked to see examples repeatedly as he's not convinced there
 is a problem.
I have discussed issues here in the past, but possibly not as concretely as would be useful to this discussion. I want to take some time to contribute value to this issue, but I have virtually no time at all lately, and I can't even keep up with this posts thread, let alone sit and consider+craft a bunch of demonstration cases. This thread is moving faster than I am able to, so I'm not sure what I can do. There's no TL;DR conversation, and almost all discussion happens outside of my timezone, which wasn't such a problem when I was nocturnal, but I have had to change lifestyle recently :/ I have about 30 posts to try and catch up on and understand right now in 5 minutes before I have to leave >_<
Dec 12 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via 
Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. 
 For example,
 not
 being able to represent idiom of `scope ref int foo(scope ref 
 int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet, and which wasn't obvious for me either, at the beginning: * A `ref` parameter means that it cannot escape the function, _except_ by return. * A `scope ref` parameter means that it cannot escape the function _ever_, not even by return. * A `scope ref` return means that it cannot leave the current statement. Therefore, a `scope ref` return value can be passed on to the next function as a `ref` argument. If that function again returns a reference (even if not explicitly designated as `scope`), the compiler will treat it as if it were `scope ref`.
 I'm also quite uneasy with the fact that scope would not be 
 transitive
 as a storage class. What happens when it's applied to a value 
 type,
 like a struct, that contains some pointers? An adaptation 
 wrapper for
 a single pointer is super common; ie, a tiny struct passed by 
 value
 with scope needs to have it's contained pointer receive the 
 scope-ness
 of the argument.
I agree, this is important. In my proposal, this works without transitivity. The wrapper stores the pointer as a `scope` member, then by copying the wrapper, the pointer gets copied implicitly, to which the normal scope restrictions apply (`scope` on members means "will not outlive the aggregate"). If it stored it as normal non-scope pointer, it couldn't get assigned in the first place. (Additionally, some higher level tricks are possible if we allow scope for value types and overloading on scope.)
Dec 11 2014
next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz wrote:
 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via 
 Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. 
 For example,
 not
 being able to represent idiom of `scope ref int foo(scope 
 ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet, and which wasn't obvious for me either, at the beginning: * A `ref` parameter means that it cannot escape the function, _except_ by return. * A `scope ref` parameter means that it cannot escape the function _ever_, not even by return. * A `scope ref` return means that it cannot leave the current statement. Therefore, a `scope ref` return value can be passed on to the next function as a `ref` argument. If that function again returns a reference (even if not explicitly designated as `scope`), the compiler will treat it as if it were `scope ref`.
OH! I had totally misunderstood that. Cheers for the explanation.
Dec 11 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/11/14 8:55 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:
 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via Digitalmars-d
 wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. For
 example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet, and which wasn't obvious for me either, at the beginning: * A `ref` parameter means that it cannot escape the function, _except_ by return. * A `scope ref` parameter means that it cannot escape the function _ever_, not even by return. * A `scope ref` return means that it cannot leave the current statement. Therefore, a `scope ref` return value can be passed on to the next function as a `ref` argument. If that function again returns a reference (even if not explicitly designated as `scope`), the compiler will treat it as if it were `scope ref`.
Please, put this in the DIP! This is exactly the thing I had been asking for here: http://forum.dlang.org/post/m5sitd$1ki0$1 digitalmars.com Thanks, I'll re-read the proposal with this in mind. -Steve
Dec 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 8:26 AM, Steven Schveighoffer wrote:
 Please, put this in the DIP! This is exactly the thing I had been asking for
 here: http://forum.dlang.org/post/m5sitd$1ki0$1 digitalmars.com

 Thanks, I'll re-read the proposal with this in mind.
Marc is clearly better at explaining things than I am, as I've explained that point in this thread about 5 times now.
Dec 11 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz wrote:
 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via 
 Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. 
 For example,
 not
 being able to represent idiom of `scope ref int foo(scope 
 ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet, and which wasn't obvious for me either, at the beginning: * A `ref` parameter means that it cannot escape the function, _except_ by return. * A `scope ref` parameter means that it cannot escape the function _ever_, not even by return. * A `scope ref` return means that it cannot leave the current statement. Therefore, a `scope ref` return value can be passed on to the next function as a `ref` argument. If that function again returns a reference (even if not explicitly designated as `scope`), the compiler will treat it as if it were `scope ref`.
No, it understood. It is simply not useful.
 I agree, this is important. In my proposal, this works without 
 transitivity. The wrapper stores the pointer as a `scope` 
 member, then by copying the wrapper, the pointer gets copied 
 implicitly, to which the normal scope restrictions apply 
 (`scope` on members means "will not outlive the aggregate"). If 
 it stored it as normal non-scope pointer, it couldn't get 
 assigned in the first place.
Wut ? You cante store anything with grear lifetime, including non scope things (as they'll have infinite lifetime). Meaning the only thing you know, is that thing are possibly scoped. Meaning you have to assume infinite lifetime with every indirection, which make this proposal useless.
Dec 11 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 12 December 2014 at 00:13:10 UTC, deadalnix wrote:
 Therefore, a `scope ref` return value can be passed on to the 
 next function as a `ref` argument. If that function again 
 returns a reference (even if not explicitly designated as 
 `scope`), the compiler will treat it as if it were `scope ref`.
No, it understood. It is simply not useful.
So far I tend to agree, unfortunately. Considering all provided answers and explanations suggested semantics are simply not powerful enough to be useful even for relatively simple idiomatic D code and with no clear way of backward compatible improvements this DIP does not pull own weight. "Those are not scopes you are looking for"
Dec 11 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 10:10 PM, Dicebot wrote:
 So far I tend to agree, unfortunately. Considering all provided answers and
 explanations suggested semantics are simply not powerful enough to be useful
 even for relatively simple idiomatic D code and with no clear way of backward
 compatible improvements this DIP does not pull own weight.
 "Those are not scopes you are looking for"
The proposal provides escape proof passing of arguments. This is necessary in order to make rc safe, for example. What are you looking for?
Dec 12 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-12-13 01:09, Walter Bright wrote:

 The proposal provides escape proof passing of arguments. This is
 necessary in order to make rc safe, for example.

 What are you looking for?
Passing references to stack allocated data to functions safely is one idea that comes to mind. -- /Jacob Carlborg
Dec 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 8:01 AM, Jacob Carlborg wrote:
 On 2014-12-13 01:09, Walter Bright wrote:

 The proposal provides escape proof passing of arguments. This is
 necessary in order to make rc safe, for example.

 What are you looking for?
Passing references to stack allocated data to functions safely is one idea that comes to mind.
'scope ref' does that.
Dec 13 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 13 December 2014 at 00:10:07 UTC, Walter Bright 
wrote:
 The proposal provides escape proof passing of arguments. This 
 is necessary in order to make rc safe, for example.

 What are you looking for?
I am looking for a tool to prevent escaping references to user-defined entities / resources from specific scope. RC case is not interesting at all to me though I recognize the potential benefits of tweaking the scope system to help with it. But there is a big difference between tweak for one specific use case or designing feature around it (so it becomes almost useless in other cases). I have already provided you an example of code I want to be enforced with scope and two major issues with existing proposal that make it lacking. This example is akin to litmus test for any scope proposal - if it doesn't allow to express such design, that proposal is simply not good enough. In case you have forgotten, I am reminding about two critical points that are necessary to make it fly: 1) inout analog for scope to be able to deduce borrowship of return values based on borrowship of input arguments. Current system is conservative and results in template bloat (complicated by the fact that it is storage class thus not actually part of a type) 2) at least optional transitivity to be able to express to protect with scope data referenced by slice or owned linked list referenced from root node. ------ In your tree example I would have never wanted scope protection of one specific node of such tree - but a transitive scope protection of whole tree view available through on of node pointers/references. It doesn't matter who and how owns the data for borrowship implementation - only thing that matters that _it is not me_.
Dec 14 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/14/2014 5:44 PM, Dicebot wrote:
 I am looking for a tool to prevent escaping references to user-defined entities
 / resources from specific scope. RC case is not interesting at all to me though
 I recognize the potential benefits of tweaking the scope system to help with
it.
 But there is a big difference between tweak for one specific use case or
 designing feature around it (so it becomes almost useless in other cases).

 I have already provided you an example of code I want to be enforced with scope
 and two major issues with existing proposal that make it lacking. This example
 is akin to litmus test for any scope proposal - if it doesn't allow to express
 such design, that proposal is simply not good enough.

 In case you have forgotten, I am reminding about two critical points that are
 necessary to make it fly:

 1) inout analog for scope to be able to deduce borrowship of return values
based
 on borrowship of input arguments. Current system is conservative and results in
 template bloat (complicated by the fact that it is storage class thus not
 actually part of a type)
1. All inout actually does is reduce code copy/pasta, it is not critical. 2. This is what scope inference is all about.
 2) at least optional transitivity to be able to express to protect with scope
 data referenced by slice or owned linked list referenced from root node.
1. that won't work unless scope is a type constructor 2. it can be achieved by using wrappers that only allow by-scope references to their data (RC is an example of such a wrapper)
 In your tree example I would have never wanted scope protection of one specific
 node of such tree - but a transitive scope protection of whole tree view
 available through on of node pointers/references. It doesn't matter who and how
 owns the data for borrowship implementation - only thing that matters that _it
 is not me_.
As I explained to Manu, transitive scope makes things like doing a symbol table lookup, then putting a pointer to the found symbol in an AST, not workable.
Dec 14 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 15 December 2014 at 02:45:04 UTC, Walter Bright wrote:
 1. All inout actually does is reduce code copy/pasta, it is not 
 critical.
Being forced to duplicate every single function in two flavors to actually make scope system usable? This is as much critical as it can be.
 2. This is what scope inference is all about.
Which only works with templates and lack of scope on arguments does not affect function body -> templates are not necessary, same as inout.
 2) at least optional transitivity to be able to express to 
 protect with scope
 data referenced by slice or owned linked list referenced from 
 root node.
1. that won't work unless scope is a type constructor
I know and this is why I am leaning toward it being qualifier despite all related issues.
 2. it can be achieved by using wrappers that only allow 
 by-scope references to their data (RC is an example of such a 
 wrapper)
It is viral approach and backwards incompatible one - we can't change signatures of Phobos functions to return Wrapped!(char[]) instead of char[] for example. While theoretically possible I can't call this approach practical - not until see some convincing application example.
 In your tree example I would have never wanted scope 
 protection of one specific
 node of such tree - but a transitive scope protection of whole 
 tree view
 available through on of node pointers/references. It doesn't 
 matter who and how
 owns the data for borrowship implementation - only thing that 
 matters that _it
 is not me_.
As I explained to Manu, transitive scope makes things like doing a symbol table lookup, then putting a pointer to the found symbol in an AST, not workable.
This sounds totally against my understanding of scope. I want scope exactly to prohibit such actions. However it is possible in slightly modified way - where you don't directly insert pointer to AST but use public method of AST control structure that checks if supplied scope pointer belongs to list of nodes it owns and casts away scope after that.
Dec 14 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/14/2014 7:41 PM, Dicebot wrote:
 On Monday, 15 December 2014 at 02:45:04 UTC, Walter Bright wrote:
 1. All inout actually does is reduce code copy/pasta, it is not critical.
Being forced to duplicate every single function in two flavors to actually make scope system usable? This is as much critical as it can be.
C++ seems to do fine without it for const. It's a convenience feature.
 2. This is what scope inference is all about.
Which only works with templates and lack of scope on arguments does not affect function body -> templates are not necessary, same as inout.
It also works with all the lambdas, since source for them is always available. I also wanted to make it (i.e. inference) work with auto functions, but Don Clugston was the primary objector :-) I do view inference as something we need to extend to more functions.
 2) at least optional transitivity to be able to express to protect with scope
 data referenced by slice or owned linked list referenced from root node.
1. that won't work unless scope is a type constructor
I know and this is why I am leaning toward it being qualifier despite all related issues.
That would be a truly massive change to D, and I'm not at all sure it would be worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and idea that had seemed straightforward.
 2. it can be achieved by using wrappers that only allow by-scope references to
 their data (RC is an example of such a wrapper)
It is viral approach and backwards incompatible one - we can't change signatures of Phobos functions to return Wrapped!(char[]) instead of char[] for example. While theoretically possible I can't call this approach practical - not until see some convincing application example.
But you can change signatures to add annotations? You're going to get one or the other.
 This sounds totally against my understanding of scope. I want scope exactly to
 prohibit such actions. However it is possible in slightly modified way - where
 you don't directly insert pointer to AST but use public method of AST control
 structure that checks if supplied scope pointer belongs to list of nodes it
 owns  and casts away scope after that.
Wrappers do that, too. I know that 'wrapper' sounds bad to you, but one of the goals of D's UDT's is to make it easy to adapt the behavior of a type by wrapping it in a struct, rather than build new behavior into the language.
Dec 14 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote:
 C++ seems to do fine without it for const. It's a convenience 
 feature.
C++ const does not really restrict or affect anything "for real", it is non-existent feature. `scope` as proposed would result in inability to store result of predicates if those are ever to accept scope data - unless you defined two versions for each function that may possibly accept scope data with absolutely identical body.
 2. This is what scope inference is all about.
Which only works with templates and lack of scope on arguments does not affect function body -> templates are not necessary, same as inout.
It also works with all the lambdas, since source for them is always available. I also wanted to make it (i.e. inference) work with auto functions, but Don Clugston was the primary objector :-)
This is not really answering my objections but side-stepping. I do support attribute inference. I would actually support full attribute inference in the language (and help convincing Don about it ;)) if it was introduced as foundation of the language and not arbitrary hack (== rethink the way .di interfaces and static libraries are defined)
 I do view inference as something we need to extend to more 
 functions.
That would help with many issues but it need careful design to be well-accepted. How about teaming up to do a DIP about it at some point where this discussion is over? :P
 2) at least optional transitivity to be able to express to 
 protect with scope
 data referenced by slice or owned linked list referenced 
 from root node.
1. that won't work unless scope is a type constructor
I know and this is why I am leaning toward it being qualifier despite all related issues.
That would be a truly massive change to D, and I'm not at all sure it would be worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and idea that had seemed straightforward.
Yes, I know and this why I am leaning towards that path but willing to accept any storage-class based solution - as long as it fits my basic criteria. I wonder if some hybrid approach is possible - for example, keep it storage class but pretend anything transitively accessible though it as if it was of scope storage class on its own. But that quickly gets into "exceptions for special cases" land :(
 2. it can be achieved by using wrappers that only allow 
 by-scope references to
 their data (RC is an example of such a wrapper)
It is viral approach and backwards incompatible one - we can't change signatures of Phobos functions to return Wrapped!(char[]) instead of char[] for example. While theoretically possible I can't call this approach practical - not until see some convincing application example.
But you can change signatures to add annotations?
Such annotation would only prevent from compiling code which was already undefined by documentation. Changing return type (which wasn't Voldemort type before) may break any program that has lines akin to `char[] data = foo()` - perfectly legal working code.
 This sounds totally against my understanding of scope. I want 
 scope exactly to
 prohibit such actions. However it is possible in slightly 
 modified way - where
 you don't directly insert pointer to AST but use public method 
 of AST control
 structure that checks if supplied scope pointer belongs to 
 list of nodes it
 owns  and casts away scope after that.
Wrappers do that, too. I know that 'wrapper' sounds bad to you, but one of the goals of D's UDT's is to make it easy to adapt the behavior of a type by wrapping it in a struct, rather than build new behavior into the language.
Problem is that our UDT are simply not that good enough yet. Wrappers are fine in general but I want to wrap for less common cases and support more common ones natively. I suppose this is your intention too but we disagree on what is actually the common case and what is the special case.
Dec 15 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2014 1:28 AM, Dicebot wrote:
 On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote:
 C++ seems to do fine without it for const. It's a convenience feature.
C++ const does not really restrict or affect anything "for real", it is non-existent feature.
C++ programmers constantly copy/pasta functions for the sole purpose of having one with const and one without.
 `scope` as proposed would result in inability to store
 result of predicates if those are ever to accept scope data - unless you
defined
 two versions for each function that may possibly accept scope data with
 absolutely identical body.
If that is necessary, it is no worse than C++ is with const. It isn't a critical issue, or C++ would not be usable. I'd like to wait and see how this plays out.
 2. This is what scope inference is all about.
Which only works with templates and lack of scope on arguments does not affect function body -> templates are not necessary, same as inout.
It also works with all the lambdas, since source for them is always available. I also wanted to make it (i.e. inference) work with auto functions, but Don Clugston was the primary objector :-)
This is not really answering my objections but side-stepping.
Inference addresses the issue.
 That would help with many issues but it need careful design to be
well-accepted.
 How about teaming up to do a DIP about it at some point where this discussion
is
 over? :P
Ok, sure.
Dec 15 2014
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote:
 On 12/14/2014 7:41 PM, Dicebot wrote:
 Being forced to duplicate every single function in two flavors 
 to actually make
 scope system usable? This is as much critical as it can be.
C++ seems to do fine without it for const. It's a convenience feature.
It's more than that. It has the potential (well, could have) to be used for type erasure, which is a useful and worthwhile concept by itself.
 I know and this is why I am leaning toward it being qualifier 
 despite all
 related issues.
That would be a truly massive change to D, and I'm not at all sure it would be worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and idea that had seemed straightforward.
To be honest, I think one reason for those bugs is the awkward implementation. I had a look at mtype.c to see how difficult it would be to add another type modifier. It's no fun - the current implementation requires a dozen groups of helper functions to be written for every possible combination of type modifiers. Therefore there are helpers like `makeConst`, `makeImmutable`, `makeShared`, `makeSharedConst`, `makeWild` (this is `inout`), `makeWildConst`, `makeSharedWild`, `makeSharedWildConst`, and `makeMutable`. There are also some monster switch statements in the same fashion. No wonder this is error prone. I suspect that with a cleaner implementation most of these problems will go away.
Dec 15 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2014 3:34 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 No wonder this is error prone. I suspect that with a cleaner implementation
most
 of these problems will go away.
Sometimes, it isn't possible to see a cleaner way unless the dirty version is written first. Sometimes, there isn't a cleaner way :-( But there are more issues with inout than are in mtype.c. The template implementation is full of it.
Dec 15 2014
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 15 December 2014 at 01:44:27 UTC, Dicebot wrote:
 On Saturday, 13 December 2014 at 00:10:07 UTC, Walter Bright 
 wrote:
 The proposal provides escape proof passing of arguments. This 
 is necessary in order to make rc safe, for example.

 What are you looking for?
I am looking for a tool to prevent escaping references to user-defined entities / resources from specific scope. RC case is not interesting at all to me though I recognize the potential benefits of tweaking the scope system to help with it. But there is a big difference between tweak for one specific use case or designing feature around it (so it becomes almost useless in other cases). I have already provided you an example of code I want to be enforced with scope and two major issues with existing proposal that make it lacking. This example is akin to litmus test for any scope proposal - if it doesn't allow to express such design, that proposal is simply not good enough. In case you have forgotten, I am reminding about two critical points that are necessary to make it fly: 1) inout analog for scope to be able to deduce borrowship of return values based on borrowship of input arguments. Current system is conservative and results in template bloat (complicated by the fact that it is storage class thus not actually part of a type)
Is it be possible to make `inout` a more general tool for type erasure? I don't think yet another ad-hoc mechanism should be introduced.
 2) at least optional transitivity to be able to express to 
 protect with scope data referenced by slice or owned linked 
 list referenced from root node.

 ------

 In your tree example I would have never wanted scope protection 
 of one specific node of such tree - but a transitive scope 
 protection of whole tree view available through on of node 
 pointers/references. It doesn't matter who and how owns the 
 data for borrowship implementation - only thing that matters 
 that _it is not me_.
I think there are two cases that are relevant. 1) the tree nodes don't own their children, instead they are managed by - let's say - a region allocator, and 2) the tree nodes do own their children. In both cases, they can declare the children as scope members (assuming a proposal that allows that). The only case I can think of where the children should not be scope members is when they are GC managed. But in this case, you don't need to worry about references to them escaping. Therefore, I don't see a use case for transitivity of scope, from a conceptional point of view. If a particular design makes transitivity necessary, this points to a flaw in the design, IMO.
Dec 15 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 15 December 2014 at 11:23:28 UTC, Marc Schütz wrote:
 I think there are two cases that are relevant. 1) the tree 
 nodes don't own their children, instead they are managed by - 
 let's say - a region allocator, and 2) the tree nodes do own 
 their children. In both cases, they can declare the children as 
 scope members (assuming a proposal that allows that).

 The only case I can think of where the children should not be 
 scope members is when they are GC managed. But in this case, 
 you don't need to worry about references to them escaping.

 Therefore, I don't see a use case for transitivity of scope, 
 from a conceptional point of view. If a particular design makes 
 transitivity necessary, this points to a flaw in the design, 
 IMO.
Actually I think in neither case children should be declared as scope members. Instead those should be declared as private members but methods that return pointers/references to children would be declared to return (transitive) scope pointers/references. For me "scopeness" is a property of "view", not object itself - this also makes ownership method of actual data irrelevant. Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available.
Dec 15 2014
next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:
 On Monday, 15 December 2014 at 11:23:28 UTC, Marc Schütz wrote:
 I think there are two cases that are relevant. 1) the tree 
 nodes don't own their children, instead they are managed by - 
 let's say - a region allocator, and 2) the tree nodes do own 
 their children. In both cases, they can declare the children 
 as scope members (assuming a proposal that allows that).

 The only case I can think of where the children should not be 
 scope members is when they are GC managed. But in this case, 
 you don't need to worry about references to them escaping.

 Therefore, I don't see a use case for transitivity of scope, 
 from a conceptional point of view. If a particular design 
 makes transitivity necessary, this points to a flaw in the 
 design, IMO.
Actually I think in neither case children should be declared as scope members. Instead those should be declared as private members but methods that return pointers/references to children would be declared to return (transitive) scope pointers/references.
I see this as the same thing, conceptually (module the "transitive" in in your parentheses). It's just a different mechanism. Scope members are just a short-cut instead of writing an accessor method that returns scope.
 For me "scopeness" is a property of "view", not object itself - 
 this also makes ownership method of actual data irrelevant.
Yes, and this would be true for scope members just the same. (If the aggregate "owns" it's scope members, it will however get in trouble when it tries to free() them, therefore your mechanism is more appropriate in this case.)
 Only difference between GC owned data and stack allocated one 
 is that former can have scoped view optionally but for the 
 latter compiler must force it as the only available.
Agreed.
Dec 15 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:
 For me "scopeness" is a property of "view", not object itself - 
 this also makes ownership method of actual data irrelevant. 
 Only difference between GC owned data and stack allocated one 
 is that former can have scoped view optionally but for the 
 latter compiler must force it as the only available.
Ha finally something start to make sense here.
Dec 15 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2014 5:38 AM, deadalnix wrote:
 On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:
 For me "scopeness" is a property of "view", not object itself - this also
 makes ownership method of actual data irrelevant. Only difference between GC
 owned data and stack allocated one is that former can have scoped view
 optionally but for the latter compiler must force it as the only available.
Ha finally something start to make sense here.
Well, the DIP does defined scope in terms of a 'view' in just this manner. I am obviously terrible at explaining things.
Dec 15 2014
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 16 December 2014 at 01:00:47 UTC, Walter Bright wrote:
 On 12/15/2014 5:38 AM, deadalnix wrote:
 On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:
 For me "scopeness" is a property of "view", not object itself 
 - this also
 makes ownership method of actual data irrelevant. Only 
 difference between GC
 owned data and stack allocated one is that former can have 
 scoped view
 optionally but for the latter compiler must force it as the 
 only available.
Ha finally something start to make sense here.
Well, the DIP does defined scope in terms of a 'view' in just this manner. I am obviously terrible at explaining things.
No, I actually think the concept of view in the proposal is a good one. However, the proposal itself is too limiting.
Dec 16 2014
prev sibling next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 16 December 2014 at 01:00:47 UTC, Walter Bright wrote:
 On 12/15/2014 5:38 AM, deadalnix wrote:
 On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:
 For me "scopeness" is a property of "view", not object itself 
 - this also
 makes ownership method of actual data irrelevant. Only 
 difference between GC
 owned data and stack allocated one is that former can have 
 scoped view
 optionally but for the latter compiler must force it as the 
 only available.
Ha finally something start to make sense here.
Well, the DIP does defined scope in terms of a 'view' in just this manner. I am obviously terrible at explaining things.
Don't you think there is a contradiction between this notion of "view" and being so focused on lifetimes? Does it make sense to explicitly throw away lifetime information and then spend a lot of effort on preventing lifetime-violating assignments to take place? The current proposal is either too limiting or not limiting enough. Too limiting, because programmers expect to gain something for making parameter types explicit. A naive "make no assumptions view" is more suitable as a default, then let the programmer specify the specifics where possible/needed, but you need to figure out the specifics first and let the "make no assumptions view default" be the leftovers… The proposal is not limiting enough, because retaining knowledge about allocation order and aliasing brings important benefits: An ordered "linear" parameter type would have properties such as: - being aliasfree - being on the stack, and cheaply checkable against the stack pointer Sure, ref counting is fairly general for dynamic tracking, but it is also quite expensive. A check against the stack-pointer is much cheaper. I think the uncanny valley metaphor that was brought up was quite good. If you are going to put in constraints on references, then do it in a manner that is easy to deal with and which brings benefits such as aliasfree references. If constraints become "arbitrary" then usability will suffer and it will seem pointless to put in extra work to use them. You have already gone more or less wholesale for templates, so you might as well use it here as well, to bring substantial benefits to stack-allocation, because the current proposal is a very weak in terms of benefits.
Dec 16 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/16/2014 5:37 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 The current proposal is either too limiting or not limiting enough.
I'm afraid I don't understand at all what you wrote.
Dec 16 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 I'm afraid I don't understand at all what you wrote.
Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems Bye, bearophile
Dec 16 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile wrote:
 Walter Bright:

 I'm afraid I don't understand at all what you wrote.
Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems
But note that we want a "relaxed" linear type system. In general, we are fine with multiple aliases, although there are some applications where strict uniqueness is useful. Deadalnix' proposal goes in this direction.
Dec 17 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 17 December 2014 at 11:13:02 UTC, Marc Schütz wrote:
 On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile 
 wrote:
 Walter Bright:

 I'm afraid I don't understand at all what you wrote.
Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems
But note that we want a "relaxed" linear type system. In general, we are fine with multiple aliases, although there are
Pointer aliasing is an optimization killer, which is why C99 added the «restrict» keyword. I think in most cases the use of stack allocated objects tend to be alias-free, so it is important that this knowledge is retained so that the compiler can make use of it. http://en.wikipedia.org/wiki/Pointer_aliasing
Dec 17 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 17 December 2014 at 12:29:21 UTC, Ola Fosheim 
Grøstad wrote:
 On Wednesday, 17 December 2014 at 11:13:02 UTC, Marc Schütz 
 wrote:
 On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile 
 wrote:
 Walter Bright:

 I'm afraid I don't understand at all what you wrote.
Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems
But note that we want a "relaxed" linear type system. In general, we are fine with multiple aliases, although there are
Pointer aliasing is an optimization killer, which is why C99 added the «restrict» keyword. I think in most cases the use of stack allocated objects tend to be alias-free, so it is important that this knowledge is retained so that the compiler can make use of it. http://en.wikipedia.org/wiki/Pointer_aliasing
I know, this is one of those applications. The one I was thinking of is const-borrowing (useful to prevent iterator invalidation and a bunch of other things). I think that enforcing alias-freeness by default (the way a strict linear type system requires) doesn't fit too well into the existing language. Some way of signalling this to the compiler will therefore be required. It might be as simple as C's `restrict` keyword (but it needs to be enforced!), or better yet, an `unrestrict` keyword. Parameters would be alias-free by default unless explicitly specified otherwise. Local aliases (inside one function) are however ok, as long as not more than one is passed to another function at the same time.
Dec 17 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 17 December 2014 at 13:35:53 UTC, Marc Schütz wrote:
 I think that enforcing alias-freeness by default (the way a 
 strict linear type system requires) doesn't fit too well into 
 the existing language. Some way of signalling this to the 
 compiler will therefore be required. It might be as simple as 
 C's `restrict` keyword (but it needs to be enforced!), or 
 better yet, an `unrestrict` keyword. Parameters would be 
 alias-free by default unless explicitly specified otherwise. 
 Local aliases (inside one function) are however ok, as long as 
 not more than one is passed to another function at the same 
 time.
Hmm, I kind of feel that D is having too many parameter keywords/types/qualifiers. Parameter specification is primarily useful where it provides information about constraints that the programmer need to know about when using a library. "restricted scope ref" could be a useful default for non-templated functions, then use templated ownership reference typing or inference to go beyond it. I think that if one improved the templating system so that it was capable of analysing similarities between instantiations and use heuristics to analyse when a new instance is profitable then you could make functions templated by default and propagate knowledge from the call site to the template-instantiation mechanism of the compiler. So if the reference is known to be aliasfree then the compiler can use heuristics to determine if it should instantiate a "restricted" version or use a slower general version of the templated function. Similarily, if refcounters always are on offset 0 and destructors are virtual, then you only need a single container method template instance for refcouting references, even if you have many types using the same method. If refcounters are big enough, you can have a unique_ptr that just set it to a large value and use the same template instance for a unique_ptr and refcounted_ptr behind the scene when referencing objects that provide a refcount field. There are also many other things that could be done to enhance the template system, like querying properties of the block in loops that iterate over a "range", so that you can instantiate different "optimized ranges" depending on whether the loop contains a "break" statement or not. Or like Jonathan Blow's language, where you can issue a "remove element and continue" like statement within a foreach-loop, which might require a different "range iteration" implementation.
Dec 17 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 17 December 2014 at 12:29:21 UTC, Ola Fosheim 
Grøstad wrote:
 On Wednesday, 17 December 2014 at 11:13:02 UTC, Marc Schütz 
 wrote:
 On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile 
 wrote:
 Walter Bright:

 I'm afraid I don't understand at all what you wrote.
Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems
But note that we want a "relaxed" linear type system. In general, we are fine with multiple aliases, although there are
Pointer aliasing is an optimization killer, which is why C99 added the «restrict» keyword. I think in most cases the use of stack allocated objects tend to be alias-free, so it is important that this knowledge is retained so that the compiler can make use of it. http://en.wikipedia.org/wiki/Pointer_aliasing
If you have owner ship, you have free. If you have a pair alloc/free then you can promote on stack. This is a much more powerful way to handle things, as this take advantage of inlining.
Dec 17 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 17 December 2014 at 13:48:24 UTC, deadalnix wrote:
 If you have owner ship, you have free. If you have a pair 
 alloc/free then you can promote on stack.

 This is a much more powerful way to handle things, as this take 
 advantage of inlining.
More powerful than what? Rather than thinking in terms of alloc/free it might be helpful to think about it more generally as resources, use, reuse and side-effects. On the stack you get to reuse memory when the variable is not longer live, meaning you junk writes if the results cannot be read and reuse the memory "prematurely". http://en.wikipedia.org/wiki/Live_variable_analysis If you are saying that turning GC allocations automatically into stack allocations is a good direction for D2, then I agree.
Dec 17 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 17 December 2014 at 05:28:35 UTC, Walter Bright 
wrote:
 On 12/16/2014 5:37 AM, "Ola Fosheim Grøstad" 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 The current proposal is either too limiting or not limiting 
 enough.
I'm afraid I don't understand at all what you wrote.
Well, look at it this way: D1 took 1990s style C++ application programming model and made it more convenient with a GC. Having a GC-memory model with the possibility of stack/heap allocations as optimizations can keep the language simple. D2 is more of a library author's language, but without GC you need to deal with different types of ownership. If you toss out the GC, and want the same level of safety and generality you either: 1. need to track ownership add dynamic runtime checks 2. provide a full blown proof system (not realistic) 3. provide means for the programmer to "help" the semantic analysis What you want is a combination of 1 and 3, but you don't need to keep them separate. Since refcounting is expensive, you don't want that everywhere. For stack allocated objects you have an implicit "refcounting" in the nesting of scopes, you only need to track scope-depth. If the compiler knows that a returned object is stack allocated, then it can check that by comparing against the stack frame address since the stack grows downwards in an ordered fashion. And in most cases it should be able to elide the check. If you also maintain information about order (stack depth) among the stack allocated objects a function receives through reference parameters (for every function call), then you can safely combine the stack allocated objects you receive through parameters in aggregates, like inserting an stack allocated element into a container. Since D is increasingly becoming a library author's language, you probably also would benefit more from strengthening the template support by having a templated ref-type rather than "scope ref". You could define protocols (e.g. predefined UDAs) that tells the compiler what kind of key properties the library type provides, such as ownership. Then use that for a more generic approach to optimization and escape analysis and let the ownership-reference-types be library types. If semantic analysis gets rid of the dynamic aspects of a referencs (such as reference counting) then the compiler should be able to use the same function body, and in other cases just add a inferred wrapper to a generic function body. So the code bloat can be limited. If all references to external resources (like the heap) in the stack allocated object is owned by the object, then you can treat those as a whole. But you need the compiler to understand what ownership is if it is to guarantee memory safety in the kind of situations where memory safety is known to be difficult to achieve. The alternative is to state that safe is not covering non-GC memory and that one should stick to vetted library code when writing safe code that does not use the GC. I actually find that acceptable for D2. In D2 you would be better off spending the effort at finding ways to turn automatically turn GC allocations into stack allocations as optimizations. Then you can find a more general and powerful ownership-oriented approach for non-GC applications for a new major version of D. It is not realistic that people will annotate with "scope ref", and compiler inference could make updates to libraries break application code if you get false negatives. Function signatures are contracts, they ought to be explicit for libraries. So having library functions act as "scope ref" without it being explicit in the signature is not necessarily a good idea.
Dec 17 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 16 December 2014 at 01:00:47 UTC, Walter Bright wrote:
 On 12/15/2014 5:38 AM, deadalnix wrote:
 On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:
 For me "scopeness" is a property of "view", not object itself 
 - this also
 makes ownership method of actual data irrelevant. Only 
 difference between GC
 owned data and stack allocated one is that former can have 
 scoped view
 optionally but for the latter compiler must force it as the 
 only available.
Ha finally something start to make sense here.
Well, the DIP does defined scope in terms of a 'view' in just this manner. I am obviously terrible at explaining things.
Such notion of "view" requires at least some elements of transitivity to be practical in my opinion. Also with my definition in mind your example of tree that stores scope nodes makes absolutely no sense unless whole tree itself is scoped (and nodes are thus scoped transitively). Such view is always assumes worst case about ownership and shouldn't persist in any form (as that would require some serious ownership tracking). I don't think you have explained your case that bad - we simply have a very different use cases in mind as "primary" use case.
Dec 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/19/2014 9:44 PM, Dicebot wrote:
 Such notion of "view" requires at least some elements of transitivity to be
 practical in my opinion.
I have no idea how "some elements of transitivity" can even work. It's either transitive or its not. Please don't think of scope in terms of ownership, ownership is an orthogonal issue.
 Also with my definition in mind your example of tree
 that stores scope nodes makes absolutely no sense unless whole tree itself is
 scoped (and nodes are thus scoped transitively). Such view is always assumes
 worst case about ownership and shouldn't persist in any form (as that would
 require some serious ownership tracking).
This is definitely conflating scope and ownership.
Dec 20 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 December 2014 at 21:39:44 UTC, Walter Bright 
wrote:
 On 12/19/2014 9:44 PM, Dicebot wrote:
 Such notion of "view" requires at least some elements of 
 transitivity to be
 practical in my opinion.
I have no idea how "some elements of transitivity" can even work. It's either transitive or its not. Please don't think of scope in terms of ownership, ownership is an orthogonal issue.
.. and here I was about you to do exactly the same :P What in example I show makes you think of ownership? When I was speaking about "some elements of transitivity" I was thinking in a way of keeping scope storage class but transitively applying same restrictions to all data accessible through it AS IF it had scope storage class on its own - while still making illegal to use scope as a separate type qualifier.
 Also with my definition in mind your example of tree
 that stores scope nodes makes absolutely no sense unless whole 
 tree itself is
 scoped (and nodes are thus scoped transitively). Such view is 
 always assumes
 worst case about ownership and shouldn't persist in any form 
 (as that would
 require some serious ownership tracking).
This is definitely conflating scope and ownership.
No, it is exactly the other way around. The very point of what I am saying is that you DOESN'T CARE about ownership as long as worst case scenario is assumed. I have zero idea why you identify it is conflating as ownership when it is explicitly designed to be distinct.
Dec 21 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/21/2014 2:06 AM, Dicebot wrote:
 No, it is exactly the other way around. The very point of what I am saying is
 that you DOESN'T CARE about ownership as long as worst case scenario is
 assumed.   I have zero idea why you identify it is conflating as ownership when
 it is explicitly designed to be distinct.
The point of transitive scoping would be if the root owned the data reachable through the root.
Dec 21 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 22 December 2014 at 03:07:53 UTC, Walter Bright wrote:
 On 12/21/2014 2:06 AM, Dicebot wrote:
 No, it is exactly the other way around. The very point of what 
 I am saying is
 that you DOESN'T CARE about ownership as long as worst case 
 scenario is
 assumed.   I have zero idea why you identify it is conflating 
 as ownership when
 it is explicitly designed to be distinct.
The point of transitive scoping would be if the root owned the data reachable through the root.
Quoting myself:
 For me "scopeness" is a property of "view", not object itself - 
 this also makes ownership method of actual data irrelevant. Only
 difference between GC owned data and stack allocated one is that
 former can have scoped view optionally but for the latter
 compiler must force it as the only available.
It doesn't matter of root owns the data. We _assume_ that as worst case scenario and allowed actions form a strict subset of allowed actions for any other ownership situation. Such `scope` for stack/GC is same as `const` for mutable/immutable - common denominator. Point of transitive scope is to make easy to expose complex custom data structures without breaking memory safety.
Dec 22 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/22/2014 12:04 AM, Dicebot wrote:
 Point of transitive scope is to make easy to expose complex custom data
 structures without breaking memory safety.
I do understand that. Making it work with the type system is another matter entirely - it's far more complex than just adding a qualifier. 'inout' looks simple but still has ongoing issues. And the thing is, wrappers can be used instead of qualifiers, in the same places in the same way. It's much simpler.
Dec 22 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 22 December 2014 at 20:51:46 UTC, Walter Bright wrote:
 On 12/22/2014 12:04 AM, Dicebot wrote:
 Point of transitive scope is to make easy to expose complex 
 custom data
 structures without breaking memory safety.
I do understand that. Making it work with the type system is another matter entirely - it's far more complex than just adding a qualifier. 'inout' looks simple but still has ongoing issues. And the thing is, wrappers can be used instead of qualifiers, in the same places in the same way. It's much simpler.
Ok, lets consider some examples of code I'd want to work with wrapper approach. Basic tree structure: struct Node { Node* left, right; Payload payload; } struct Tree { Node* root; } Restricting public access to nodes to prevent storing persistent node pointers. Your proposed approach: struct Wrapper(T) { scope T wrapped; alias wrapped this; } struct Node { private: Node* _left, _right; public: auto left() property { return Wrapper!(Node*)(_left); } auto right() property { return Wrapper!(Node*)(_right); } Payload payload; } struct Tree { private: Node* _root; public: auto root() property { return Wrapper!(Node*)(_root); } } Am I correct? If yes, this is exactly the breakage issue I was speaking about : any code that looked like `is(typeof(Tree.root) == Node*)` will stop working. With my latest proposal (pseudo-transitive storage class which is not part of a type) getters would look simply like `scope Node* root() property { return _root; }` and `is(typeof(Tree.root) == Node*)` still remained true. Only impact on existing would be restriction of operations to take address or assign a pointer. I am pretty sure I am missing many complicated parts here but this seems like a big deal in retro-fitting existing libraries to use scope after it was added.
Dec 29 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 12 December 2014 at 00:13:10 UTC, deadalnix wrote:
 On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz 
 wrote:
 This is a point that most people don't seem to understand yet, 
 and which wasn't obvious for me either, at the beginning:

 * A `ref` parameter means that it cannot escape the function, 
 _except_ by return.
 * A `scope ref` parameter means that it cannot escape the 
 function  _ever_, not even by return.
 * A `scope ref` return means that it cannot leave the current 
 statement.

 Therefore, a `scope ref` return value can be passed on to the 
 next function as a `ref` argument. If that function again 
 returns a reference (even if not explicitly designated as 
 `scope`), the compiler will treat it as if it were `scope ref`.
No, it understood.
Steven hadn't, evidently.
 It is simply not useful.
I'm not convinced either.
 I agree, this is important. In my proposal, this works without 
 transitivity. The wrapper stores the pointer as a `scope` 
 member, then by copying the wrapper, the pointer gets copied 
 implicitly, to which the normal scope restrictions apply 
 (`scope` on members means "will not outlive the aggregate"). 
 If it stored it as normal non-scope pointer, it couldn't get 
 assigned in the first place.
Wut ? You cante store anything with grear lifetime, including non scope things (as they'll have infinite lifetime). Meaning the only thing you know, is that thing are possibly scoped. Meaning you have to assume infinite lifetime with every indirection, which make this proposal useless.
My comments above don't refer to this proposal, but my original one: struct Wrapper(T) { scope T payload; // used to be: // scope!this T payload; } scope int a; auto w = Wrapper!(int*)(&a); // ok scope int b; w.payload = &b; // error, w (and therefore w.payload) // lives longer than b Wrapper!(int*) w2 = w; // ok, lifetime(w2) < lifetime(w) w = w2; // error, equivalent to // w.payload = w2.payload Therefore, wrapping is safe without transitivity of scope.
Dec 12 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 4:13 PM, deadalnix wrote:
 On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz wrote:
 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. For example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet, and which wasn't obvious for me either, at the beginning: * A `ref` parameter means that it cannot escape the function, _except_ by return. * A `scope ref` parameter means that it cannot escape the function _ever_, not even by return. * A `scope ref` return means that it cannot leave the current statement. Therefore, a `scope ref` return value can be passed on to the next function as a `ref` argument. If that function again returns a reference (even if not explicitly designated as `scope`), the compiler will treat it as if it were `scope ref`.
No, it understood. It is simply not useful.
The use case is described in the DIP under the 'Scope Ref' heading.
 I agree, this is important. In my proposal, this works without transitivity.
 The wrapper stores the pointer as a `scope` member, then by copying the
 wrapper, the pointer gets copied implicitly, to which the normal scope
 restrictions apply (`scope` on members means "will not outlive the
 aggregate"). If it stored it as normal non-scope pointer, it couldn't get
 assigned in the first place.
Wut ? You cante store anything with grear lifetime, including non scope things (as they'll have infinite lifetime). Meaning the only thing you know, is that thing are possibly scoped. Meaning you have to assume infinite lifetime with every indirection, which make this proposal useless.
It means you'll have to wrap the next level of indirection in a ref counting (or equivalent) manner. The great thing about GC is you can leave everything to the GC. But when explicitly managing memory, you have to decide for EVERY pointer who owns it, when it can be released, etc. Having a transitive scope is NOT going to resolve that for you, it isn't even going to help.
Dec 12 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 December 2014 at 23:55, via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. For
 example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet, and which wasn't obvious for me either, at the beginning: * A `ref` parameter means that it cannot escape the function, _except_ by return. * A `scope ref` parameter means that it cannot escape the function _ever_, not even by return. * A `scope ref` return means that it cannot leave the current statement. Therefore, a `scope ref` return value can be passed on to the next function as a `ref` argument. If that function again returns a reference (even if not explicitly designated as `scope`), the compiler will treat it as if it were `scope ref`.
Ummm. I reckon there's a good reason that people don't seem to understand this... because it would have difficulty being any more unintuitive! It's contrary to the behaviour of basically everything else in D! const, pure, nothrow, etc... they all work a reliable and consistent way. Also, I still don't think I get it... a scope-ref return, which has a restriction as you say, can be subverted by being piped through a function that receives and returns it as ref? What is the point?
 I'm also quite uneasy with the fact that scope would not be transitive
 as a storage class. What happens when it's applied to a value type,
 like a struct, that contains some pointers? An adaptation wrapper for
 a single pointer is super common; ie, a tiny struct passed by value
 with scope needs to have it's contained pointer receive the scope-ness
 of the argument.
I agree, this is important. In my proposal, this works without transitivity. The wrapper stores the pointer as a `scope` member, then by copying the wrapper, the pointer gets copied implicitly, to which the normal scope restrictions apply (`scope` on members means "will not outlive the aggregate"). If it stored it as normal non-scope pointer, it couldn't get assigned in the first place. (Additionally, some higher level tricks are possible if we allow scope for value types and overloading on scope.)
How can a member be marked scope? Apparently it's a storage class, not a type constructor... we can only attribute storage classes to function args/results(?).
Dec 12 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 12 December 2014 at 08:42:22 UTC, Manu via 
Digitalmars-d wrote:
 On 11 December 2014 at 23:55, via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 This is a point that most people don't seem to understand yet, 
 and which
 wasn't obvious for me either, at the beginning:

 * A `ref` parameter means that it cannot escape the function, 
 _except_ by
 return.
 * A `scope ref` parameter means that it cannot escape the 
 function  _ever_,
 not even by return.
 * A `scope ref` return means that it cannot leave the current 
 statement.

 Therefore, a `scope ref` return value can be passed on to the 
 next function
 as a `ref` argument. If that function again returns a 
 reference (even if not
 explicitly designated as `scope`), the compiler will treat it 
 as if it were
 `scope ref`.
Ummm. I reckon there's a good reason that people don't seem to understand this... because it would have difficulty being any more unintuitive! It's contrary to the behaviour of basically everything else in D! const, pure, nothrow, etc... they all work a reliable and consistent way.
Unfortunately I have to agree.
 Also, I still don't think I get it... a scope-ref return, which 
 has a
 restriction as you say, can be subverted by being piped through 
 a
 function that receives and returns it as ref?
 What is the point?
It cannot be subverted. As soon as you pass something into a function G() as `ref` that's been returned from a function F() via `scope ref`, the compiler needs to treat what is returned from G() as `scope ref`, even if it's only declared as `ref`. As you say, it's unintuitive and non-obvious.
 I'm also quite uneasy with the fact that scope would not be 
 transitive
 as a storage class. What happens when it's applied to a value 
 type,
 like a struct, that contains some pointers? An adaptation 
 wrapper for
 a single pointer is super common; ie, a tiny struct passed by 
 value
 with scope needs to have it's contained pointer receive the 
 scope-ness
 of the argument.
I agree, this is important. In my proposal, this works without transitivity. The wrapper stores the pointer as a `scope` member, then by copying the wrapper, the pointer gets copied implicitly, to which the normal scope restrictions apply (`scope` on members means "will not outlive the aggregate"). If it stored it as normal non-scope pointer, it couldn't get assigned in the first place. (Additionally, some higher level tricks are possible if we allow scope for value types and overloading on scope.)
How can a member be marked scope? Apparently it's a storage class, not a type constructor... we can only attribute storage classes to function args/results(?).
I was talking about my proposal here, where it's a type modifier, not a storage class. deadalnix also brought up the problem of transitivity. This, too, would simply go away if it's a type modifier.
Dec 12 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 December 2014 at 22:18, via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Friday, 12 December 2014 at 08:42:22 UTC, Manu via Digitalmars-d wrote:
 How can a member be marked scope? Apparently it's a storage class, not
 a type constructor... we can only attribute storage classes to
 function args/results(?).
I was talking about my proposal here, where it's a type modifier, not a storage class.
Oh okay. Is that still on the table?
 deadalnix also brought up the problem of transitivity. This, too, would
 simply go away if it's a type modifier.
Just for clarity, does type modifier mean the same thing as type constructor?
Dec 12 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 12 December 2014 at 12:34:40 UTC, Manu via 
Digitalmars-d wrote:
 On 12 December 2014 at 22:18, via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 12 December 2014 at 08:42:22 UTC, Manu via 
 Digitalmars-d wrote:
 How can a member be marked scope? Apparently it's a storage 
 class, not
 a type constructor... we can only attribute storage classes to
 function args/results(?).
I was talking about my proposal here, where it's a type modifier, not a storage class.
Oh okay. Is that still on the table?
Dunno, maybe? DIP69 doesn't look very well-received at the moment...
 deadalnix also brought up the problem of transitivity. This, 
 too, would
 simply go away if it's a type modifier.
Just for clarity, does type modifier mean the same thing as type constructor?
Yes. "Type modifier" = const, shared, immutable, etc.; "type constructor" = the syntactical construct by which a type is constructed/modified. Andrei suggested this differentiation some time ago (sorry, don't have a reference at hand).
Dec 12 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 13:38:10 UTC, Marc Schütz wrote:
 Yes. "Type modifier" = const, shared, immutable, etc.; "type 
 constructor" = the syntactical construct by which a type is 
 constructed/modified. Andrei suggested this differentiation 
 some time ago (sorry, don't have a reference at hand).
That won't fly. scope turtle left, type qualifier turtle right.
Dec 12 2014
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 12:34:40 UTC, Manu via
Digitalmars-d wrote:
 deadalnix also brought up the problem of transitivity. This, 
 too, would
 simply go away if it's a type modifier.
Just for clarity, does type modifier mean the same thing as type constructor?
It appears to me that scopeness is a completely different beast than type qualifier, and type qualifier won't work. I think type modifier was just made up here. And why not ? It is more like some metadata (lifetime) being associated with value and symbols, like a type would, but that isn't really per se a type as D understands it right now.
Dec 12 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. For example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
Yes, it can be.
 I don't have the perfect proposal, but I feel very strongly about 2 things:
 1. It must not be a storage class; the concept was a disaster with
 ref, and I struggle with this more frequently than any other 'feature'
 in D.
I simply do not understand why distinguishing beteen ref and not-ref is a cornerstone of everything you do.
 2. I feel it's a big mistake to separate it from the type system,
 which I think most agree, is D's greatest asset by far. Manipulating
 types is simple and convenient using the type system, and I think
 manipulating scope will be just as important as any other attribute as
 soon as even slightly complex use cases begin to arise.
Consider a ref counted type, RC!T. If scope were transitive, then you could not have, say, a tree where the edges were RC!T. I.e., the payload of an RC type should not be forced to be scope.
Dec 11 2014
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-Dec-2014 00:41, Walter Bright пишет:
 On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 I don't have the perfect proposal, but I feel very strongly about 2
 things:
 1. It must not be a storage class; the concept was a disaster with
 ref, and I struggle with this more frequently than any other 'feature'
 in D.
I simply do not understand why distinguishing beteen ref and not-ref is a cornerstone of everything you do.
I would hazard a guess that const T& is something lots of C++ got addicted to as the most common way of parameter passing. There is nothing exactly like that in D, auto ref was quite close but only for templates and it instantiates 2 distinct bodies for ref and non-ref at the moment. It allows easy pass by ref for both r-values and l-values with logical const and is likely what Manu is referring(!) to. Truth be told it's not always the fastest way, in fact Boost C++ has template for optimal parameter passing - call_traits<T>::param_type but it must be too cumbersome for actual use as I never seen it used in the wild. http://www.boost.org/doc/libs/1_57_0/libs/utility/call_traits.htm -- Dmitry Olshansky
Dec 11 2014
prev sibling next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright 
wrote:
 On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. 
 For example,
 not
 being able to represent idiom of `scope ref int foo(scope 
 ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
Yes, it can be.
 I don't have the perfect proposal, but I feel very strongly 
 about 2 things:
 1. It must not be a storage class; the concept was a disaster 
 with
 ref, and I struggle with this more frequently than any other 
 'feature'
 in D.
I simply do not understand why distinguishing beteen ref and not-ref is a cornerstone of everything you do.
Because he requires control over function ABIs for both inter-language communication and performance. In binding D to IDL (Interactive Data, not Interface Description) I found ref often required special casing.
Dec 11 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 December 2014 at 08:53, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright wrote:
 On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. For
 example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
Yes, it can be.
 I don't have the perfect proposal, but I feel very strongly about 2
 things:
 1. It must not be a storage class; the concept was a disaster with
 ref, and I struggle with this more frequently than any other 'feature'
 in D.
I simply do not understand why distinguishing beteen ref and not-ref is a cornerstone of everything you do.
Because he requires control over function ABIs for both inter-language communication and performance. In binding D to IDL (Interactive Data, not Interface Description) I found ref often required special casing.
Thank you. I feel like I'm just repeating myself over and over again.
Dec 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:20 AM, Manu via Digitalmars-d wrote:
 On 12 December 2014 at 08:53, John Colvin via Digitalmars-d
 Because he requires control over function ABIs for both inter-language
 communication and performance.

 In binding D to IDL (Interactive Data, not Interface Description) I found
 ref often required special casing.
Thank you. I feel like I'm just repeating myself over and over again.
Example, please.
Dec 12 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright 
wrote:
 Consider a ref counted type, RC!T. If scope were transitive, 
 then you could not have, say, a tree where the edges were RC!T. 
 I.e., the payload of an RC type should not be forced to be 
 scope.
I don't see how this is related. It would be perfectly ok to declare root of such tree scope if it was transitive (as long as it only controls access and does not attempt early destruction). You may want to explain this in details in DIP if this is motivating case.
Dec 11 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 06:06:40 UTC, Dicebot wrote:
 On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright 
 wrote:
 Consider a ref counted type, RC!T. If scope were transitive, 
 then you could not have, say, a tree where the edges were 
 RC!T. I.e., the payload of an RC type should not be forced to 
 be scope.
I don't see how this is related. It would be perfectly ok to declare root of such tree scope if it was transitive (as long as it only controls access and does not attempt early destruction).
That is my proposal. However, it is not as simply as you think it is without making it a type qualifier (which is not desirable). In effect, that mean that you can see something that has infinite lifetime through a scope reference, so you need to track lifetime rvalue and lvalue differently. I'm still for it. The current proposal is not powerful enough to pull its weight.
Dec 11 2014
parent "Dicebot" <public dicebot.lv> writes:
On Friday, 12 December 2014 at 06:57:54 UTC, deadalnix wrote:
 On Friday, 12 December 2014 at 06:06:40 UTC, Dicebot wrote:
 On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright 
 wrote:
 Consider a ref counted type, RC!T. If scope were transitive, 
 then you could not have, say, a tree where the edges were 
 RC!T. I.e., the payload of an RC type should not be forced to 
 be scope.
I don't see how this is related. It would be perfectly ok to declare root of such tree scope if it was transitive (as long as it only controls access and does not attempt early destruction).
That is my proposal. However, it is not as simply as you think it is without making it a type qualifier (which is not desirable). In effect, that mean that you can see something that has infinite lifetime through a scope reference, so you need to track lifetime rvalue and lvalue differently. I'm still for it. The current proposal is not powerful enough to pull its weight.
I don't hope it will be simple. It is all about making scope as simple as possible to keep it useful for idiomatic D code - but not simpler.
Dec 11 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 10:06 PM, Dicebot wrote:
 On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright wrote:
 Consider a ref counted type, RC!T. If scope were transitive, then you could
 not have, say, a tree where the edges were RC!T. I.e., the payload of an RC
 type should not be forced to be scope.
I don't see how this is related. It would be perfectly ok to declare root of such tree scope if it was transitive (as long as it only controls access and does not attempt early destruction).
Are you suggesting two kinds of scope - transitive and non-transitive? The more I think about it, the more ref counting is the definitive case, as just about everything else can be explained in terms of how ref counting works.
Dec 11 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:
 I don't see how this is related. It would be perfectly ok to 
 declare root of
 such tree scope if it was transitive (as long as it only 
 controls access and
 does not attempt early destruction).
Are you suggesting two kinds of scope - transitive and non-transitive? The more I think about it, the more ref counting is the definitive case, as just about everything else can be explained in terms of how ref counting works.
I don't see applicability of non-transitive scope because I don't understand the problem with the tree or reference counting you have mentioned - that is why I suggested to amend DIP to put explanation there.
Dec 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 11:55 PM, Dicebot wrote:
 On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:
 I don't see how this is related. It would be perfectly ok to declare root of
 such tree scope if it was transitive (as long as it only controls access and
 does not attempt early destruction).
Are you suggesting two kinds of scope - transitive and non-transitive? The more I think about it, the more ref counting is the definitive case, as just about everything else can be explained in terms of how ref counting works.
I don't see applicability of non-transitive scope because I don't understand the problem with the tree or reference counting you have mentioned - that is why I suggested to amend DIP to put explanation there.
Consider every pointer in a tree to be a ref counted pointer. There is no purpose to making scope transitive - traveling from one node to the next goes through the usual ref counting mechanism, and the ref counting wrapper can control that. Also, consider a data structure: A -> B C -> B If scope is transitive, how is C going to simultaneously access B?
Dec 12 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:
 Are you suggesting two kinds of scope - transitive and 
 non-transitive?
Non transitive scope can be added without any language extension. There is no point doing much with the scope keyword if indirection aren't in the loop.
 The more I think about it, the more ref counting is the 
 definitive case, as just about everything else can be explained 
 in terms of how ref counting works.
ref counting is just one form of ownership.
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 12:16 AM, deadalnix wrote:
 On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:
 Are you suggesting two kinds of scope - transitive and non-transitive?
Non transitive scope can be added without any language extension.
In order for it to work, the holes in the language that enabled escapes had to be plugged. That's what this proposal does - plug the holes.
 ref counting is just one form of ownership.
My point is if that works with this proposal, then the other forms can work, too.
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 08:44:56 UTC, Walter Bright wrote:
 On 12/12/2014 12:16 AM, deadalnix wrote:
 On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright 
 wrote:
 Are you suggesting two kinds of scope - transitive and 
 non-transitive?
Non transitive scope can be added without any language extension.
In order for it to work, the holes in the language that enabled escapes had to be plugged. That's what this proposal does - plug the holes.
The holes covered by a non transitive scope are already undefined behavior.
 ref counting is just one form of ownership.
My point is if that works with this proposal, then the other forms can work, too.
No, unless we wrap every single indirection (pointer, slice, classes, delegates) into a wrapper. That is not going to fly very far.
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 2:28 AM, deadalnix wrote:
 On Friday, 12 December 2014 at 08:44:56 UTC, Walter Bright wrote:
 On 12/12/2014 12:16 AM, deadalnix wrote:
 On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:
 Are you suggesting two kinds of scope - transitive and non-transitive?
Non transitive scope can be added without any language extension.
In order for it to work, the holes in the language that enabled escapes had to be plugged. That's what this proposal does - plug the holes.
The holes covered by a non transitive scope are already undefined behavior.
Right, but since the current compiler cannot detect them, it is a hole.
 ref counting is just one form of ownership.
My point is if that works with this proposal, then the other forms can work, too.
No, unless we wrap every single indirection (pointer, slice, classes, delegates) into a wrapper. That is not going to fly very far.
The beauty of GC is that you don't have to do any of this. To have safe other methods of allocation, there has to be annotation everywhere or use a wrapped type in the same places.
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 22:05:21 UTC, Walter Bright wrote:
 forms can work, too.
No, unless we wrap every single indirection (pointer, slice, classes, delegates) into a wrapper. That is not going to fly very far.
The beauty of GC is that you don't have to do any of this. To have safe other methods of allocation, there has to be annotation everywhere or use a wrapped type in the same places.
The whole point of scope is to reduce that cost, both in term of runtime (as copy can be elided) but also on the user.
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 2:09 PM, deadalnix wrote:
 On Friday, 12 December 2014 at 22:05:21 UTC, Walter Bright wrote:
 forms can work, too.
No, unless we wrap every single indirection (pointer, slice, classes, delegates) into a wrapper. That is not going to fly very far.
The beauty of GC is that you don't have to do any of this. To have safe other methods of allocation, there has to be annotation everywhere or use a wrapped type in the same places.
The whole point of scope is to reduce that cost, both in term of runtime (as copy can be elided) but also on the user.
Yes, I do understand that :-) but currently doing a safe refcounted type is impossible in D. This proposal fixes that.
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 22:34:27 UTC, Walter Bright wrote:
 On 12/12/2014 2:09 PM, deadalnix wrote:
 Yes, I do understand that :-) but currently doing a safe 
 refcounted type is impossible in D. This proposal fixes that.
Actually, no it doesn't. It allow to ensure that this is safe, module the RC system is the only owner. This proposal allow for easy to use and fast RC, but certainly do not make it safe.
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 2:48 PM, deadalnix wrote:
 On Friday, 12 December 2014 at 22:34:27 UTC, Walter Bright wrote:
 On 12/12/2014 2:09 PM, deadalnix wrote:
 Yes, I do understand that :-) but currently doing a safe refcounted type is
 impossible in D. This proposal fixes that.
Actually, no it doesn't. It allow to ensure that this is safe, module the RC system is the only owner. This proposal allow for easy to use and fast RC, but certainly do not make it safe.
How not?
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright 
wrote:
 How not?
scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes !
Dec 12 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:07 PM, deadalnix wrote:
 On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote:
 How not?
scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes !
RC needs to take a type as a parameter, not a symbol.
Dec 12 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:07 PM, deadalnix wrote:
 On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote:
 How not?
scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes !
What's the type signature of RC ?
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 13 December 2014 at 00:52:46 UTC, Walter Bright 
wrote:
 On 12/12/2014 4:07 PM, deadalnix wrote:
 On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright 
 wrote:
 How not?
scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes !
What's the type signature of RC ?
RCWrapper!T RC(T)(scope T t) or somethign along these lines.
Dec 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:59 PM, deadalnix wrote:
 On Saturday, 13 December 2014 at 00:52:46 UTC, Walter Bright wrote:
 On 12/12/2014 4:07 PM, deadalnix wrote:
 On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote:
 How not?
scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes !
What's the type signature of RC ?
RCWrapper!T RC(T)(scope T t) or somethign along these lines.
And how does RC assign it to the wrapper? (It will fail to compile, and so won't be a safety hole.)
Dec 12 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 December 2014 at 07:41, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:
 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/7/2014 6:12 AM, Dicebot wrote:
 But from existing cases it doesn't seem working good enough. For
 example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.
scope ref int foo(ref int x); will do it.
Will it? It looks like foo can't be called with scope data?
Yes, it can be.
 I don't have the perfect proposal, but I feel very strongly about 2
 things:
 1. It must not be a storage class; the concept was a disaster with
 ref, and I struggle with this more frequently than any other 'feature'
 in D.
I simply do not understand why distinguishing beteen ref and not-ref is a cornerstone of everything you do.
I've said so many times, it's the single greatest regular frustration I encounter, by far. That is of course a relative measure. It's not the 'cornerstone of everything I do'; I don't start discussions about things that are not broken and otherwise painless. It is the thing that is *the most broken*, and leads to the most edge cases, general bloat, and text mixins. It comes up the most frequently, and by that metric alone, I consider it highest priority on my list. I've also said many times before, it possibly stems from the fact that one of the key cornerstones of almost everything I do in D is interact with other languages. This is a practical reality, I can't write all my code in D, and have mountains of existing code to interact with. Boilerplate is the result. D has powerful systems to automate boilerplate, like a carrot dangling right in front of my face, but it's almost always thwarted by ref, and almost exclusively so. My recent work updating LuaD to support all the features I required wasted about 80% of my time dealing with ref issues. In my past C++ bindings solutions, much code, which was otherwise simple, readable, and elegant, turned into a mess of text mixins, almost exclusively because ref is broken.
 2. I feel it's a big mistake to separate it from the type system,
 which I think most agree, is D's greatest asset by far. Manipulating
 types is simple and convenient using the type system, and I think
 manipulating scope will be just as important as any other attribute as
 soon as even slightly complex use cases begin to arise.
Consider a ref counted type, RC!T. If scope were transitive, then you could not have, say, a tree where the edges were RC!T. I.e., the payload of an RC type should not be forced to be scope.
I'm not sure I quite visualise this correctly... It sounds like you're talking about the same problem where people complain with respect to 'const' and you happily tell them to stop whingeing and deal with it? (FWIW, I think you made the right call on const. It's never caused me any hair loss.) So you're saying that a pointer itself shouldn't be able to escape a call tree, but the thing it points to should be able to escape just fine? It feels like that kinda defeats the purpose... or at least makes the concept about as reliable as 'const' is in C++. I guess you're seeing a situation where 'scope' is almost exclusively useful as a mechanism to tell the RC that it doesn't need to worry about ref-fiddling, and the thing we're passing isn't interested in scope restrictions at any level other than that RC optimisation? I guess I can see your angle... but my reaction is if that's all you are concerned about, maybe scope is the wrong tool for eliding ref fiddling in that case :/ I can see how you arrived at the opinion from your other comment (wherever it was) that you are kinda likening this whole problem to the RC case, if that's how you're looking at it. It sounds like what you want is something like head-scope, in the same way that people often want head-const?
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:19 AM, Manu via Digitalmars-d wrote:
 I simply do not understand why distinguishing beteen ref and not-ref is a
 cornerstone of everything you do.
I've said so many times, it's the single greatest regular frustration I encounter, by far. That is of course a relative measure. It's not the 'cornerstone of everything I do'; I don't start discussions about things that are not broken and otherwise painless. It is the thing that is *the most broken*, and leads to the most edge cases, general bloat, and text mixins. It comes up the most frequently, and by that metric alone, I consider it highest priority on my list. I've also said many times before, it possibly stems from the fact that one of the key cornerstones of almost everything I do in D is interact with other languages. This is a practical reality, I can't write all my code in D, and have mountains of existing code to interact with. Boilerplate is the result. D has powerful systems to automate boilerplate, like a carrot dangling right in front of my face, but it's almost always thwarted by ref, and almost exclusively so. My recent work updating LuaD to support all the features I required wasted about 80% of my time dealing with ref issues. In my past C++ bindings solutions, much code, which was otherwise simple, readable, and elegant, turned into a mess of text mixins, almost exclusively because ref is broken.
You've said this before, many times, but what is lacking is an explanation of WHY. What is the pattern, and why do you need that pattern? You say you don't like auto ref because it doesn't give you exact control, but what are the cases where auto ref is wrong?
 Consider a ref counted type, RC!T. If scope were transitive, then you could
 not have, say, a tree where the edges were RC!T. I.e., the payload of an RC
 type should not be forced to be scope.
I'm not sure I quite visualise this correctly...
struct Tree { RefCount!(Tree*) left; RefCount!(Tree*) right; ... }
 So you're saying that a pointer itself shouldn't be able to escape a
 call tree, but the thing it points to should be able to escape just
 fine?
Yes.
 It feels like that kinda defeats the purpose...
You're arguing that a data structure with only one access point is the only kind of data structure in use. With transitive scope, such a data structure would be the only one possible!
 I guess you're seeing a situation where 'scope' is almost exclusively
 useful as a mechanism to tell the RC that it doesn't need to worry
 about ref-fiddling, and the thing we're passing isn't interested in
 scope restrictions at any level other than that RC optimisation?
 I guess I can see your angle... but my reaction is if that's all you
 are concerned about, maybe scope is the wrong tool for eliding ref
 fiddling in that case :/
'scope' is a way to say that this use of a pointer does not escape this scope. That is incredibly useful. Recall my statements that pervasive refcounting is a terrible performance problem because of all the inc/dec? Knowing that references cannot escape means an inc/dec pair can be elided.
Dec 12 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 December 2014 at 08:14, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 12/12/2014 4:19 AM, Manu via Digitalmars-d wrote:
 I simply do not understand why distinguishing beteen ref and not-ref is a
 cornerstone of everything you do.
I've said so many times, it's the single greatest regular frustration I encounter, by far. That is of course a relative measure. It's not the 'cornerstone of everything I do'; I don't start discussions about things that are not broken and otherwise painless. It is the thing that is *the most broken*, and leads to the most edge cases, general bloat, and text mixins. It comes up the most frequently, and by that metric alone, I consider it highest priority on my list. I've also said many times before, it possibly stems from the fact that one of the key cornerstones of almost everything I do in D is interact with other languages. This is a practical reality, I can't write all my code in D, and have mountains of existing code to interact with. Boilerplate is the result. D has powerful systems to automate boilerplate, like a carrot dangling right in front of my face, but it's almost always thwarted by ref, and almost exclusively so. My recent work updating LuaD to support all the features I required wasted about 80% of my time dealing with ref issues. In my past C++ bindings solutions, much code, which was otherwise simple, readable, and elegant, turned into a mess of text mixins, almost exclusively because ref is broken.
You've said this before, many times, but what is lacking is an explanation of WHY. What is the pattern, and why do you need that pattern? You say you don't like auto ref because it doesn't give you exact control, but what are the cases where auto ref is wrong?
I did just give some examples, I'll repeat; auto ref fails when the function is extern. It is also wrong that when I pass an int or float, it is passed by ref instead of by value... that is never what I want! What do you get when you take a pointer of a function with an auto-ref arg? ...an error, because it's not actually a function! So in a context where I'm dealing with functions and function pointers (very, very frequent), I suddenly have something that's not a function find it's way into my meta and I have to special-case the hell out of it. The solution in this case is to wrap it in a non-auto-ref shim with the ref-ness explicitly stated in the way I expect... which is in itself another problem, because 'ref' is not part of the type! So the annoying de-auto-ref-ing shim must be a text mixin with some very complex, practically unreadable, definitely unmaintainable logic surrounding it. It's insanity on top of insanity! I also run the invisible risk of generating 2^num_args instances of the code for functions containing auto-ref args. When I'm writing a function that's not a template, I intend, and expect, to write a function that's _not a template_. Templates and functions are different things. I think it's a massive mistake to have created a way to write a template that looks nothing like a template. auto-ref is not, and never has been a tool I have wanted. I don't have any use for auto-ref, and it has only proven to make an already severe problem worse. I've tried to use it before in some instances, but it made ref's out of int's and floats, so even in the rare potentially useful cases, I had to reject it. At the time, you introduced auto-ref as a 'solution' to an issue that I was the primary complainant (although the solution was mainly pushed by Andrei iirc?). I absolutely rejected it then, and said it would have the disastrous effect of setting ref in stone, which I was already very critical about. I reject it even more now that I have had some experience with it. Who was the customer that you designed it for? (It wasn't me!) What issue was it designed to address?
 Consider a ref counted type, RC!T. If scope were transitive, then you
 could
 not have, say, a tree where the edges were RC!T. I.e., the payload of an
 RC
 type should not be forced to be scope.
I'm not sure I quite visualise this correctly...
struct Tree { RefCount!(Tree*) left; RefCount!(Tree*) right; ... }
... I don't think I'd ever have a use for this code. I've been using trees for a long time, and I can't imagine a situation where each node in a tree would want to be ref counted. It sounds like a disaster for performance, and I can't imagine any advantage it would offer? Perhaps in some complex graph structure... but I expect that sort of thing would be far more specialised. I can see common cases where nodes may contain a refcounted object as node data, but that's different than the situation you demonstrate here. I'll need to think about how that situation would be affected in this case.
 So you're saying that a pointer itself shouldn't be able to escape a
 call tree, but the thing it points to should be able to escape just
 fine?
Yes.
 It feels like that kinda defeats the purpose...
You're arguing that a data structure with only one access point is the only kind of data structure in use. With transitive scope, such a data structure would be the only one possible!
Surely the possibility remains for such a data structure that is NOT scope... but I think I can see your point. There is a common case, ie, refcounting, where we want to be able to elide rc fiddling, and that may be the sole call for scope in those cases. It seems like that situation calls for something like head-scope. I think transitive scope has a different application, useful in different situations. It seems a problem that a struct that contains a single pointer, which is passed by value, can't have scope applied under your proposal(?). I use this pattern in almost every interaction with other languages that I encounter. Some external API which operates via opaque pointer, wants to be wrapped in such a way that it maintains it's behaviour as a reference type, but also gains convenient member-style access. It's kinda making a pseudo-class out of something that's not a class, and scope needs to support this.
 I guess you're seeing a situation where 'scope' is almost exclusively
 useful as a mechanism to tell the RC that it doesn't need to worry
 about ref-fiddling, and the thing we're passing isn't interested in
 scope restrictions at any level other than that RC optimisation?
 I guess I can see your angle... but my reaction is if that's all you
 are concerned about, maybe scope is the wrong tool for eliding ref
 fiddling in that case :/
'scope' is a way to say that this use of a pointer does not escape this scope. That is incredibly useful. Recall my statements that pervasive refcounting is a terrible performance problem because of all the inc/dec? Knowing that references cannot escape means an inc/dec pair can be elided.
I know this. I'm not arguing against any proposition that we need scope to make RC practical. I've been arguing in favour of that for at least as long as I've been batting for the ARC team, and even then some more, because it would also address rvalue->ref, which is another massive pain point! But there are more things than pointers which need to be protected against escaping their scope, in particular, things that contain pointers... Maybe there's a compromise. If we say scope isn't 'transitive', but it is transitive when it comes to aggregates? Ie, you can apply scope to a by-val struct, and all aggregate members have scope applied. It is not transitive in the sense that it does not follow through pointer members, but any such pointer members may not escape as if it were a pointer argument alone. This is unlike any other behaviour in D, but without that, I think this proposal fails to suit my most common use cases.
Dec 12 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote:
 I did just give some examples, I'll repeat; auto ref fails when the
 function is extern.
Don't make it extern, then.
 It is also wrong that when I pass an int or float, it is passed by ref
 instead of by value... that is never what I want!
If there's source to the function, it'll often be inlined which will remove the indirection.
 What do you get when you take a pointer of a function with an auto-ref
 arg? ...an error, because it's not actually a function!
 So in a context where I'm dealing with functions and function pointers
 (very, very frequent), I suddenly have something that's not a function
 find it's way into my meta and I have to special-case the hell out of
 it.
Why are function pointers and ints going to the same argument of a function? I thought you weren't using templates?
 The solution in this case is to wrap it in a non-auto-ref shim with
 the ref-ness explicitly stated in the way I expect... which is in
 itself another problem, because 'ref' is not part of the type! So the
 annoying de-auto-ref-ing shim must be a text mixin with some very
 complex, practically unreadable, definitely unmaintainable logic
 surrounding it. It's insanity on top of insanity!

 I also run the invisible risk of generating 2^num_args instances of
 the code for functions containing auto-ref args.
I wonder what is the need for the code that you are writing.
 When I'm writing a function that's not a template, I intend, and
 expect, to write a function that's _not a template_.
 Templates and functions are different things. I think it's a massive
 mistake to have created a way to write a template that looks nothing
 like a template.
A function template is a function that takes both compile-time args and run-time args. C++ tried to make them completely different animals, when they are not. Don't think "template", think "compile-time argument to a function". I convinced Andrei to never mention "template" in his D book, as it brings forth all kinds of connotations and expectations that impede understanding what D templates actually are. I think the result was successful.
 auto-ref is not, and never has been a tool I have wanted. I don't have
 any use for auto-ref, and it has only proven to make an already severe
 problem worse. I've tried to use it before in some instances, but it
 made ref's out of int's and floats, so even in the rare potentially
 useful cases, I had to reject it.
If it's a rare useful case, why is it a pervasive problem?
 At the time, you introduced auto-ref as a 'solution' to an issue that
 I was the primary complainant (although the solution was mainly pushed
 by Andrei iirc?). I absolutely rejected it then, and said it would
 have the disastrous effect of setting ref in stone, which I was
 already very critical about. I reject it even more now that I have had
 some experience with it.
 Who was the customer that you designed it for? (It wasn't me!)
 What issue was it designed to address?
I still have no idea what code you are developing that you need to send ints and function pointers to the same argument of a function template, yet you don't use function templates. Nor do I understand what pattern you need that simply must mix up ref and value parameters, and why that pattern appears pervasively in your code.
 struct Tree {
     RefCount!(Tree*) left;
     RefCount!(Tree*) right;
     ...
 }
... I don't think I'd ever have a use for this code.
You have no use for tree structures?
 I've been using trees for a long time, and I can't imagine a situation
 where each node in a tree would want to be ref counted.
You have more than one parent of a node. You never write data structures like that? dmd uses such pervasively (Type and Expression).
 It sounds like a disaster for performance, and I can't imagine any
 advantage it would offer?
How do you propose to manage the memory explicitly otherwise? Use a GC? :-)
 I can see common cases where nodes may contain a refcounted object as
 node data, but that's different than the situation you demonstrate
 here.
That would fail if scope were transitive.
 But there are more things than pointers which need to be protected
 against escaping their scope, in particular, things that contain
 pointers...
Solve it in the general case (this proposal) and RC now works.
 Maybe there's a compromise. If we say scope isn't 'transitive', but it
 is transitive when it comes to aggregates?
One thing I've tried very hard to make work in D is have basic types / aggregages / arrays be interchangeable.
 Ie, you can apply scope to a by-val struct, and all aggregate members
 have scope applied.
 It is not transitive in the sense that it does not follow through
 pointer members, but any such pointer members may not escape as if it
 were a pointer argument alone.

 This is unlike any other behaviour in D, but without that, I think
 this proposal fails to suit my most common use cases.
I have no idea what your use cases are, but when you're explicitly managing memory, every damn pointer needs to be carefully evaluated as to what exactly it's pointing two and who owns it. Scope is not a magic solution to this, neither is shared_ptr, neither are Rust's annotations. The only scheme that absolves the user of having to deal with this is - GC.
Dec 12 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 struct Tree {
    RefCount!(Tree*) left;
    RefCount!(Tree*) right;
    ...
 }
... I don't think I'd ever have a use for this code.
You have no use for tree structures?
Giving a reference counter to every pointer in a binary tree sounds a bit too much. Bye, bearophile
Dec 13 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 2:10 AM, bearophile wrote:
 Walter Bright:

 struct Tree {
    RefCount!(Tree*) left;
    RefCount!(Tree*) right;
    ...
 }
... I don't think I'd ever have a use for this code.
You have no use for tree structures?
Giving a reference counter to every pointer in a binary tree sounds a bit too much.
DMD, for example, uses ASTs where the nodes have multiple parents.
Dec 13 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 13 December 2014 at 20:45:53 UTC, Walter Bright 
wrote:
 On 12/13/2014 2:10 AM, bearophile wrote:
 Walter Bright:

 struct Tree {
   RefCount!(Tree*) left;
   RefCount!(Tree*) right;
   ...
 }
... I don't think I'd ever have a use for this code.
You have no use for tree structures?
Giving a reference counter to every pointer in a binary tree sounds a bit too much.
DMD, for example, uses ASTs where the nodes have multiple parents.
It is a DAG then.
Dec 15 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-12-13 06:11, Walter Bright wrote:

 Don't make it extern, then.
He's said a couple of time he's interfacing with C and/or C++ code.
 Why are function pointers and ints going to the same argument of a
 function? I thought you weren't using templates?
I think he meant taking the address of a function with auto ref. -- /Jacob Carlborg
Dec 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 8:19 AM, Jacob Carlborg wrote:
 On 2014-12-13 06:11, Walter Bright wrote:

 Don't make it extern, then.
He's said a couple of time he's interfacing with C and/or C++ code.
Yes, and C++ code is either ref or not ref, and it is not a problem to insert 'ref' or not, manually.
 Why are function pointers and ints going to the same argument of a
 function? I thought you weren't using templates?
I think he meant taking the address of a function with auto ref.
Right, but why is that even being attempted?
Dec 13 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote:
 I did just give some examples, I'll repeat; auto ref fails when the
 function is extern.
Don't make it extern, then.
Do you think I just interact with other languages for fun or something?
 It is also wrong that when I pass an int or float, it is passed by ref
 instead of by value... that is never what I want!
If there's source to the function, it'll often be inlined which will remove the indirection.
Not if it's extern (applicable to ref, not auto-ref), or wrapped, or if I ever want to capture a function pointer. It's also semantically different; I can change the caller's value.
 What do you get when you take a pointer of a function with an auto-ref
 arg? ...an error, because it's not actually a function!
 So in a context where I'm dealing with functions and function pointers
 (very, very frequent), I suddenly have something that's not a function
 find it's way into my meta and I have to special-case the hell out of
 it.
Why are function pointers and ints going to the same argument of a function? I thought you weren't using templates?
I'm confused. I'm talking about capturing function pointers. Not about function arguments. Capturing pointers of functions with auto-ref args (aka, template functions) is a serious nuisance, and impossible outside the site of instantiation. You need information that's only available at the point of instantiation. That changes the landscape quite a bit.
 The solution in this case is to wrap it in a non-auto-ref shim with
 the ref-ness explicitly stated in the way I expect... which is in
 itself another problem, because 'ref' is not part of the type! So the
 annoying de-auto-ref-ing shim must be a text mixin with some very
 complex, practically unreadable, definitely unmaintainable logic
 surrounding it. It's insanity on top of insanity!

 I also run the invisible risk of generating 2^num_args instances of
 the code for functions containing auto-ref args.
I wonder what is the need for the code that you are writing.
General function adaptation. There are lots of reasons to wrap functions. Adaptation to existing or external API's is the most common in my experience. Functions are what programs are! I really struggle to understand why you have such trouble sympathising with my view here. Languages generate code, which is packaged into blocks we call 'functions'... and they require to adhere to strict ABI requirements. That is programming in a nutshell. Surely it's reasonable to want to retain complete control over this most primitive and basic of tasks. One important aspect of that control is where the code is generated; is it 'here', or 'there'? And that's a critical distinction between functions and templates. Look at LuaD. Lots of awkward cases have emerged there. There are many more to come too; the known bug list are mostly nasty issues of this nature that I've been putting off. I can't offer any insight into my commercial code sadly, and I no longer have access to it :/ Situations like this appear frequently: https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R178 And something like this tends to appear as one aspect of the solution: https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R68 'struct Ref(T)' leads to its own problems though, in that it's a localised concept. No external code anywhere understands it as a 'ref', so if 3rd party code has any special treatment for ref, that code now fails. Then more magic like this: https://github.com/JakobOvrum/LuaD/pull/76/files#diff-ec8c532aeca798240de4d70ee639fc16R90 Since we need to recognise 'ref' and substitute it for our magic 'struct Ref(T)'. I've probably spent more hours wrangling D meta of this sort than most people. This sort of thing always happens. If Ref!T is the tool that resolves the situation, then it's clear demonstration that ref should be part of the type.
 When I'm writing a function that's not a template, I intend, and
 expect, to write a function that's _not a template_.
 Templates and functions are different things. I think it's a massive
 mistake to have created a way to write a template that looks nothing
 like a template.
A function template is a function that takes both compile-time args and run-time args. C++ tried to make them completely different animals, when they are not. Don't think "template", think "compile-time argument to a function". I convinced Andrei to never mention "template" in his D book, as it brings forth all kinds of connotations and expectations that impede understanding what D templates actually are. I think the result was successful.
You can spin it however you like, but it's exactly the same thing. They are completely different animals. A function template is not a function at all until it's instantiated. When it's instantiated, it becomes a function, or even, one of a suite of functions. And it's not known to the author where the function is. (Note: I define 'function' in this context to mean 'some code that is emitted') This one:many relationship between definition and code creates some sorts of problems that aren't present with functions, which relate 1:1 with their codegen. It is possible to take a pointer to a function. It is not possible to take a pointer to a template, unless your code exists at the callsite where the parameters for instantiation are known. This makes certain things more complicated. I reject that 'a template is a function with compile time args'. Template functions have rather different characteristics, which result in special consideration, and some restrictions on usage. It's a cute idea, and something that might sound nice in a book... but it's not the reality. 'functions' are a much simpler, more fundamental concept, and also one that is easily portable between languages.
 auto-ref is not, and never has been a tool I have wanted. I don't have
 any use for auto-ref, and it has only proven to make an already severe
 problem worse. I've tried to use it before in some instances, but it
 made ref's out of int's and floats, so even in the rare potentially
 useful cases, I had to reject it.
If it's a rare useful case, why is it a pervasive problem?
I never argued that auto-ref wasn't rare, but it was *a mistake*. What it did was cement an already very shaky language feature into the foundation. Once there is layers of language built on top of something, it becomes that much harder to refactor at some later time. I'm not saying auto-ref is a pervasive problem, I'm saying (over and over again) that *ref* is a pervasive problem, and auto-ref turns out to be a further (fairly rare) nuisance. It can't be presented as a solution to any problem that I have (how we got onto this tangent), because it's not. I also don't have any idea why it exists! What's it for? You love to make people justify exactly what things are for; you're doing it to me in almost every paragraph. In my case here though, I'm not asking you to invent a language feature like auto-ref, I'm just commenting on the usefulness of an existing feature. Trouble is, my cases are really hard to define and highly context specific. I'm sharing my anecdotal experience that D's ref is an awkward pain in the arse at best, and by my judgement, a failed experiment.
 At the time, you introduced auto-ref as a 'solution' to an issue that
 I was the primary complainant (although the solution was mainly pushed
 by Andrei iirc?). I absolutely rejected it then, and said it would
 have the disastrous effect of setting ref in stone, which I was
 already very critical about. I reject it even more now that I have had
 some experience with it.
 Who was the customer that you designed it for? (It wasn't me!)
 What issue was it designed to address?
I still have no idea what code you are developing that you need to send ints and function pointers to the same argument of a function template, yet you don't use function templates. Nor do I understand what pattern you need that simply must mix up ref and value parameters, and why that pattern appears pervasively in your code.
I was talking about taking function pointers of functions. Not passing function pointers to functions. The auto-ref detail was on a slight tangent; explaining that it's not useful to me to solve the problem of dealing with ref-ness of incoming functions args or results. I think I said that clearly: "What do you get when you take a pointer of a function with an auto-ref arg? ...an error, because it's not actually a function!" WRT to mixing ref and value parameters; I'm wrapping/adapting existing api's. I have no control over the code/api that exists. If that api uses ref, for whatever reason that it makes sense to do so, I need to handle the case. There's no 'pattern', only a mechanical process.
 struct Tree {
     RefCount!(Tree*) left;
     RefCount!(Tree*) right;
     ...
 }
... I don't think I'd ever have a use for this code.
You have no use for tree structures?
I don't have use for a tree where every node is ref counted... tree nodes should only have one reference; their parent.
 I've been using trees for a long time, and I can't imagine a situation
 where each node in a tree would want to be ref counted.
You have more than one parent of a node. You never write data structures like that? dmd uses such pervasively (Type and Expression).
That's not a tree, that's a graph. I made a comment on such graph structures: "Perhaps in some complex graph structure... but I expect that sort of thing would be far more specialised." And I still think that. I'm not going to rule out that an RC graph node might be useful, but I was responding to your example, and that wasn't your example.
 It sounds like a disaster for performance, and I can't imagine any
 advantage it would offer?
How do you propose to manage the memory explicitly otherwise? Use a GC? :-)
Tree elements only have one reference; their parent. Management of trees is very simple. Graph's may be a little more complex, and I can see that an RC node might be useful... but I've been using various graph structures for a long time, and never wanted to RC the nodes. I expect traditional patterns would remain. I don't think an RC is required to know if a graph node is still referenced within the graph; graph nodes typically have bi-directional linkage, so when there are no links to the node, then it is unreferenced, and can easily be destroyed/returned to a pool. I think an RC mechanism is effectively implicit in graph structures I can imagine. The cases where I use RC are for general 'resources'. In my line of work, this is textures, mesh, renderstate, sounds, etc... things that find themselves being shared around arbitrarily. It is important that tree/graph nodes may contain a ref-counted thing, but I have trouble visualising how that affects 'scope' in this case.
 I can see common cases where nodes may contain a refcounted object as
 node data, but that's different than the situation you demonstrate
 here.
That would fail if scope were transitive.
I'm not sure I see why... The situation is: a whole transitive scope graph is received as scope, how does that inhibit my accessing a resource contained in the graph? It inhibits my _escaping_ said resource... as it should. Right? The problem you're addressing is that, if the graph isn't received as scope (ie, I want the ability to escape it's internals), then RC doesn't work efficiently. My reaction is that RC optimisation is perhaps not strongly associated with 'scope'. scope may offer a mechanism by which it can work some of the time, but it's not precisely the same thing. I kinda feel like you're trying to make scope into RC optimisation, rather than RC optimisation into scope...? I was once arguing for: int^ rcInt; I think I'm going back in that direction. That's what other languages do. And scope would create additional opportunity here; it would allow implicit casting of T^ -> T*.
 But there are more things than pointers which need to be protected
 against escaping their scope, in particular, things that contain
 pointers...
Solve it in the general case (this proposal) and RC now works.
Perhaps scope and RC are different things? Solve scope purely for RC, and you've broken scope for other purposes that it's useful; that is, inhibiting escaping of data.
 Maybe there's a compromise. If we say scope isn't 'transitive', but it
 is transitive when it comes to aggregates?
One thing I've tried very hard to make work in D is have basic types / aggregages / arrays be interchangeable.
I'm looking for ways to make scope fit your proposed mould and be useful to me. If I can't do this then my most common use case is unsatisfied: struct Wrap { OpaqueThing *ptr; this() {} ~this() {} // <- constructors that twiddle RC effectively. this(this) {} this() scope {} ~this() scope {} // <- Overloadable for 'scope', in which case, don't twiddle the RC! this(this) scope {} // lots of operations on OpaqueThing wrapped up here... } void f(scope Wrap x) <-- RC twiddling is effectively elided here due to constructor overloads { } If you can propose an alternative solution? This is representative of my 99% case. Sometimes the struct is more than a single pointer though.
 Ie, you can apply scope to a by-val struct, and all aggregate members
 have scope applied.
 It is not transitive in the sense that it does not follow through
 pointer members, but any such pointer members may not escape as if it
 were a pointer argument alone.

 This is unlike any other behaviour in D, but without that, I think
 this proposal fails to suit my most common use cases.
I have no idea what your use cases are, but when you're explicitly managing memory, every damn pointer needs to be carefully evaluated as to what exactly it's pointing two and who owns it. Scope is not a magic solution to this, neither is shared_ptr, neither are Rust's annotations. The only scheme that absolves the user of having to deal with this is - GC.
scope just needs to say "I will not let this memory escape". Then it finally allows us to safely put data on the stack, something we can't do in D today. I don't think scope should aim to do anything more than that. If scope can be useful to RC, that's great! But from this, it's starting to look to me like scope might be partially useful to RC, but scope and RC aren't exactly the same thing. RC seems to require something like head-scope in order to not place awkward restrictions on usage of RC objects. That said, full-scope isn't without use cases; it allows us to safely use the stack, potentially eliminating much GC load.
Dec 13 2014
next sibling parent reply "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Sunday, 14 December 2014 at 00:45:06 UTC, Manu via 
Digitalmars-d wrote:
 On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d
 A function template is a function that takes both compile-time 
 args and
 run-time args. C++ tried to make them completely different 
 animals, when
 they are not. Don't think "template", think "compile-time 
 argument to a
 function". I convinced Andrei to never mention "template" in 
 his D book, as
 it brings forth all kinds of connotations and expectations 
 that impede understanding what D templates actually are. I 
 think the result was successful.
You can spin it however you like, but it's exactly the same thing. They are completely different animals. A function template is not a function at all until it's instantiated. When it's instantiated, it becomes a function, or even, one of a suite of functions. And it's not known to the author where the function is. (Note: I define 'function' in this context to mean 'some code that is emitted')
Philippe Sigaud Excellent "D Templates: A Tutorial"... the MANTRA... "XXX templates are not XXXs, they are templates. With XXX being any of (function, struct, class, interface, union)." --- /Paolo
Dec 14 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sun, 14 Dec 2014 09:04:58 +0000
Paolo Invernizzi via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Philippe Sigaud Excellent "D Templates: A Tutorial"...
 the MANTRA...
=20
 "XXX templates are not XXXs, they are templates. With XXX being=20
 any of (function, struct, class, interface, union)."
and that still doesn't work. ;-) as a somehow related example i recall my first expirience with C. my first "serious" language was Pascal, and it has that nice range-checking feature for arrays. it was very hard for me to understand that C has no arrays at all, not to say range checking. why there is no arrays if i have this nice indexing and all that?! what do you mean by that "buffer overflow" nonsence? if it looks like an array it must be an array!
Dec 14 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 14 December 2014 at 00:45:06 UTC, Manu via 
Digitalmars-d wrote:
 Perhaps scope and RC are different things?
 Solve scope purely for RC, and you've broken scope for other 
 purposes
 that it's useful; that is, inhibiting escaping of data.
Exactly. It shouldn't matter which type this data has - both references and value types are valid. I would define `scope` like this: "A value designated with `scope` may not be assigned to a variable (or parameter, ...) with a lifetime longer than its owner's lifetime."
 Maybe there's a compromise. If we say scope isn't 
 'transitive', but it
 is transitive when it comes to aggregates?
One thing I've tried very hard to make work in D is have basic types / aggregages / arrays be interchangeable.
It should only be applicable to whatever is explicitly marked as `scope`. For references, that's the reference itself, but not what it points to. For aggregates, it's the entire aggregate with all it's members. That way, all types will be treated uniformly. Of course, this can only work with a type modifier, not with a storage class.
 I'm looking for ways to make scope fit your proposed mould and 
 be useful to me.
 If I can't do this then my most common use case is unsatisfied:

 struct Wrap
 {
   OpaqueThing *ptr;

   this() {}
   ~this() {}  // <- constructors that twiddle RC effectively.
   this(this) {}

   this() scope {}
   ~this() scope {}  // <- Overloadable for 'scope', in which 
 case,
 don't twiddle the RC!
   this(this) scope {}

   // lots of operations on OpaqueThing wrapped up here...
 }

 void f(scope Wrap x) <-- RC twiddling is effectively elided 
 here due
 to constructor overloads
 {
 }


 If you can propose an alternative solution?
 This is representative of my 99% case. Sometimes the struct is 
 more
 than a single pointer though.
Unfortunately Walter rejected this when I had proposed it. In the current proposal, we cannot overload on scope. The proposed solution is to not "borrow" the wrapper, but only its payload. But this is restrictive; even in the case of RC, there are use-cases that cannot be served with this technique, namely when the callee can only decide at runtime whether it wants to make a copy of an RC value it received. The correct solution would be to pass the RC wrapper itself as scope, as you show in your example, thereby delegating the decision of whether to adjust the refcount from the caller to the callee.
Dec 14 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 4:44 PM, Manu via Digitalmars-d wrote:
 On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote:
 I did just give some examples, I'll repeat; auto ref fails when the
 function is extern.
Don't make it extern, then.
Do you think I just interact with other languages for fun or something?
I'm not trying to insult you - but why need 'auto ref' for extern functions? The extern functions are ref or not-ref, and declare the interface to match. There's no point to auto ref there.
 If there's source to the function, it'll often be inlined which will remove
 the indirection.
Not if it's extern (applicable to ref, not auto-ref), or wrapped, or if I ever want to capture a function pointer. It's also semantically different; I can change the caller's value.
Again, if it is extern it has a fixed definition of ref or not. The D side doesn't decide if it's ref or not, it just follows whatever the extern declaration is. This is why I do not understand why you are running into this issue everywhere.
 What do you get when you take a pointer of a function with an auto-ref
 arg? ...an error, because it's not actually a function!
 So in a context where I'm dealing with functions and function pointers
 (very, very frequent), I suddenly have something that's not a function
 find it's way into my meta and I have to special-case the hell out of
 it.
Why are function pointers and ints going to the same argument of a function? I thought you weren't using templates?
I'm confused. I'm talking about capturing function pointers. Not about function arguments. Capturing pointers of functions with auto-ref args (aka, template functions) is a serious nuisance, and impossible outside the site of instantiation.
WHY are you doing this? I have no idea what coding pattern would need to pass functions by reference.
 I wonder what is the need for the code that you are writing.
General function adaptation. There are lots of reasons to wrap functions. Adaptation to existing or external API's is the most common in my experience.
Are those API's all templated? My guess is no. If they're not, you don't need templates or auto-ref to interface to them.
 Functions are what programs are! I really struggle to understand why
 you have such trouble sympathising with my view here.
Because you don't provide any examples of why you need this stuff. You say "I need this feature because I'm doing X", but I have no idea what X is or why you need X. I.e. you present and advocate for particular solutions, but not the problems.
 Languages generate code, which is packaged into blocks we call 'functions'...
 and they require to adhere to strict ABI requirements. That is
 programming in a nutshell. Surely it's reasonable to want to retain
 complete control over this most primitive and basic of tasks.
 One important aspect of that control is where the code is generated;
 is it 'here', or 'there'?
Why is that important?
 And that's a critical distinction between functions and templates.
You can control that with function templates, too. The compiler won't instantiate a template locally if the import instantiates it.
 Look at LuaD. Lots of awkward cases have emerged there. There are many
 more to come too; the known bug list are mostly nasty issues of this
 nature that I've been putting off.
 I can't offer any insight into my commercial code sadly, and I no
 longer have access to it :/

 Situations like this appear frequently:
 https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R178

 And something like this tends to appear as one aspect of the solution:
 https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R68
 'struct Ref(T)' leads to its own problems though, in that it's a
 localised concept. No external code anywhere understands it as a
 'ref', so if 3rd party code has any special treatment for ref, that
 code now fails.

 Then more magic like this:
 https://github.com/JakobOvrum/LuaD/pull/76/files#diff-ec8c532aeca798240de4d70ee639fc16R90
 Since we need to recognise 'ref' and substitute it for our magic
 'struct Ref(T)'.


 I've probably spent more hours wrangling D meta of this sort than most
 people. This sort of thing always happens.
 If Ref!T is the tool that resolves the situation, then it's clear
 demonstration that ref should be part of the type.
I'll take a look at your references in a moment.
 When I'm writing a function that's not a template, I intend, and
 expect, to write a function that's _not a template_.
 Templates and functions are different things. I think it's a massive
 mistake to have created a way to write a template that looks nothing
 like a template.
A function template is a function that takes both compile-time args and run-time args. C++ tried to make them completely different animals, when they are not. Don't think "template", think "compile-time argument to a function". I convinced Andrei to never mention "template" in his D book, as it brings forth all kinds of connotations and expectations that impede understanding what D templates actually are. I think the result was successful.
You can spin it however you like, but it's exactly the same thing. They are completely different animals. A function template is not a function at all until it's instantiated. When it's instantiated, it becomes a function, or even, one of a suite of functions. And it's not known to the author where the function is. (Note: I define 'function' in this context to mean 'some code that is emitted')
That's the C++ view of templates, which is outdated and unnecessary.
 This one:many relationship between definition and code creates some
 sorts of problems that aren't present with functions, which relate 1:1
 with their codegen.
 It is possible to take a pointer to a function. It is not possible to
 take a pointer to a template, unless your code exists at the callsite
 where the parameters for instantiation are known.
It's not possible to take a pointer to a function unless that function exists, either.
 This makes certain things more complicated.
I don't see a problem with taking the address of an instantiated template function.
 I reject that 'a template is a function with compile time args'.
 Template functions have rather different characteristics, which result
 in special consideration, and some restrictions on usage.
 It's a cute idea, and something that might sound nice in a book... but
 it's not the reality.
I've found templates to be far more useful when viewed that way.
 'functions' are a much simpler, more fundamental concept, and also one
 that is easily portable between languages.
It's always going to be true that in order to interface D with X, you'll have to use a D subset that X understands. BTW, D can interface to C++ template functions!
 I also don't have any idea why it exists! What's it for?
So you can make a template work with both lvalues and rvalues - and implement perfect forwarding.
 You love to make people justify exactly what things are for;
It's not a question of loving it, it's a question of practicality.
 I think I said that clearly: "What do you get when you take a pointer
 of a function with an auto-ref
 arg? ...an error, because it's not actually a function!"
What's baffling to me is why even declare a function pointer parameter as auto-ref? I don't use floating point to index strings, either, that doesn't mean floating point is a failed concept.
 WRT to mixing ref and value parameters; I'm wrapping/adapting existing
 api's. I have no control over the code/api that exists. If that api
 uses ref, for whatever reason that it makes sense to do so, I need to
 handle the case.
 There's no 'pattern', only a mechanical process.
If the existing code uses ref, use ref on the D side. If it does not, do not use ref on the D side. What's the problem?
 That's not a tree, that's a graph.
Sure. Do you want a scope design that precludes graphs? And even with trees, are you sure you want a design where nobody but the root can point to anything in the tree? How would you do a symbol table? You couldn't ever return a pointer to the found symbol if it was all scoped.
 Tree elements only have one reference; their parent. Management of
 trees is very simple.
Again, what about the contents of the tree?
 That would fail if scope were transitive.
I'm not sure I see why... The situation is: a whole transitive scope graph is received as scope, how does that inhibit my accessing a resource contained in the graph? It inhibits my _escaping_ said resource... as it should. Right?
How would you manage a symbol table lookup, and then store a pointer to the found symbol in the AST?
 I was once arguing for: int^ rcInt;
 I think I'm going back in that direction. That's what other languages
 do. And scope would create additional opportunity here; it would allow
 implicit casting of T^ -> T*.
scope actually does that. (int^ rcInt; isn't transitive, either.)
 Perhaps scope and RC are different things?
 Solve scope purely for RC, and you've broken scope for other purposes
 that it's useful; that is, inhibiting escaping of data.
RC is the canonical use of scope. The beauty is the compiler does not have to recognize RC as special. It becomes just ordinary library code.
 struct Wrap
 {
    OpaqueThing *ptr;

    this() {}
    ~this() {}  // <- constructors that twiddle RC effectively.
    this(this) {}

    this() scope {}
    ~this() scope {}  // <- Overloadable for 'scope', in which case,
 don't twiddle the RC!
    this(this) scope {}

    // lots of operations on OpaqueThing wrapped up here...
 }

 void f(scope Wrap x) <-- RC twiddling is effectively elided here due
 to constructor overloads
 {
 }
First off, get rid of the scope this overloads. Second, add an 'alias this' to implement a default conversion of Wrap to OpaqueThing*. Third, declare f as: void f(scope ref OpaqueThing* x);
 scope just needs to say "I will not let this memory escape". Then it
 finally allows us to safely put data on the stack, something we can't
 do in D today. I don't think scope should aim to do anything more than
 that.
That's EXACTLY what this proposal is!
 If scope can be useful to RC, that's great! But from this, it's
 starting to look to me like scope might be partially useful to RC, but
 scope and RC aren't exactly the same thing.
Wheels are cars aren't the same thing, either, but cars need wheels.
 RC seems to require something like head-scope in order to not place
 awkward restrictions on usage of RC objects. That said, full-scope
 isn't without use cases; it allows us to safely use the stack,
 potentially eliminating much GC load.
This proposal IS head-scope.
Dec 14 2014
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 14 December 2014 at 23:27:15 UTC, Walter Bright wrote:
 I think I said that clearly: "What do you get when you take a 
 pointer
 of a function with an auto-ref
 arg? ...an error, because it's not actually a function!"
What's baffling to me is why even declare a function pointer parameter as auto-ref? I don't use floating point to index strings, either, that doesn't mean floating point is a failed concept.
That's not what he's talking about. My understanding is that Manu is talking about something like this: void foo(auto ref someConcreteType) { ... } auto fooP = &foo; //error, foo is not a function
Dec 15 2014
prev sibling parent reply "Max Samukha" <maxsamukha gmail.com> writes:
On Saturday, 13 December 2014 at 02:55:50 UTC, Manu via 
Digitalmars-d wrote:

 Templates and functions are different things. I think it's a 
 massive
 mistake to have created a way to write a template that looks 
 nothing
 like a template.
That is a misconception spread by C++. Templates are pure functions applied at compile time. Function templates are functions partially applied at compile time.
Dec 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 6:10 AM, Max Samukha wrote:
 On Saturday, 13 December 2014 at 02:55:50 UTC, Manu via Digitalmars-d wrote:

 Templates and functions are different things. I think it's a massive
 mistake to have created a way to write a template that looks nothing
 like a template.
That is a misconception spread by C++. Templates are pure functions applied at compile time. Function templates are functions partially applied at compile time.
It took me an embarrassingly long time to get past the C++ notion that templates are something orthogonal from 'normal' functions and data declarations. Even the C++ notion of a 'primary' template is wholly unnecessary, but is deeply embedded in how it handles the semantics of templates. C++ did get right the partial ordering of templates, and D applied partial ordering to 'normal' functions as well.
Dec 13 2014
prev sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2014-12-04 09:24:13 +0000, Walter Bright said:

 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the 
 missing semantics for the 'scope' storage class in order to make it 
 possible to pass a reference to a function without it being possible 
 for it to escape.
 
 This, among other things, makes a ref counting type practical. It also 
 makes it more practical to use other storage allocation schemes than 
 garbage collection.
 
 It does not make scope into a type constructor, nor a general 
 type-annotation system.
 
 It does not provide an ownership system, though it would complement one.
I like the basics of the proposal and I think it's the right direction. HOWEVER, I strongly believe that function arguments should be scoped by default and `impure` when they take reference which they will keep. -Shammah
Dec 10 2014
parent Shammah Chancellor <anonymous coward.com> writes:
On 2014-12-11 07:03:58 +0000, Shammah Chancellor said:

 On 2014-12-04 09:24:13 +0000, Walter Bright said:
 
 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the 
 missing semantics for the 'scope' storage class in order to make it 
 possible to pass a reference to a function without it being possible 
 for it to escape.
 
 This, among other things, makes a ref counting type practical. It also 
 makes it more practical to use other storage allocation schemes than 
 garbage collection.
 
 It does not make scope into a type constructor, nor a general 
 type-annotation system.
 
 It does not provide an ownership system, though it would complement one.
I like the basics of the proposal and I think it's the right direction. HOWEVER, I strongly believe that function arguments should be scoped by default and `impure` when they take reference which they will keep. -Shammah
Also, more feedback. I would suggest that scope variable are the default inside of functions as well as parameters. Using escape analysis, instead of them being errors, make them warnings and implicitly make them impure. -Shammah
Dec 10 2014