www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - borrowed pointers vs ref

reply Walter Bright <newshound2 digitalmars.com> writes:
It's been brought up more than once that the 'scope' storage class is an 
unimplemented borrowed pointer. But thinking a bit more along those lines, 
actually 'ref' fills the role of a borrowed pointer.

One particularly apropos behavior is that struct member functions pass 'this'
by 
ref, meaning that members can be called without the inc/dec millstone.

ref is still incomplete as far as this goes, but we can go the extra distance 
with it, and then it will be of great help in supporting any ref counting
solution.

What it doesn't work very well with are class references. But Andrei suggested 
that we can focus the use of 'scope' to deal with that in an analogous way.

What do you think?

Anyone want to enumerate a list of the current deficiencies of 'ref' in regards 
to this, so we can think about solving it?
May 12 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
How would you assign a borrowed pointer?
May 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 1:49 PM, Kagamin wrote:
 How would you assign a borrowed pointer?

A ref could only be assigned to another ref.
May 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 2:13 PM, Walter Bright wrote:
 On 5/12/2014 1:49 PM, Kagamin wrote:
 How would you assign a borrowed pointer?

A ref could only be assigned to another ref.

I mean to a ref of the same or smaller scope.
May 12 2014
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 12 May 2014 16:36:12 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 It's been brought up more than once that the 'scope' storage class is an  
 unimplemented borrowed pointer. But thinking a bit more along those  
 lines, actually 'ref' fills the role of a borrowed pointer.

 One particularly apropos behavior is that struct member functions pass  
 'this' by ref, meaning that members can be called without the inc/dec  
 millstone.

 ref is still incomplete as far as this goes, but we can go the extra  
 distance with it, and then it will be of great help in supporting any  
 ref counting solution.

Hm... the one piece that I think would be damaging is to not be able to take an address of the 'this' reference. It's probably OK to just use pointers and static functions in some cases, but member functions do not have that luxury. In other words, operators. Big example would be a doubly linked list used with ~=. -Steve
May 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 2:15 PM, Steven Schveighoffer wrote:
 Hm... the one piece that I think would be damaging is to not be able to take an
 address of the 'this' reference. It's probably OK to just use pointers and
 static functions in some cases, but member functions do not have that luxury.
In
 other words, operators.

 Big example would be a doubly linked list used with ~=.

trusted/ system code will be able to take the address of a ref.
May 12 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
The first thing that comes to my mind is applying this somehow to 
the (T) vs (ref T) function problem. (const ref, scope ref, 
references to r-values, you know the problem.) At the moment I 
just follow this pattern.

void foo(ref const T bar) {
     /* ... */
}

// Second overload to make r-values just work.
void foo(const T bar) {
     foo(bar);
}

auto ref sometimes works, sometimes it's more trouble than its 
worth.
May 12 2014
prev sibling next sibling parent "Daniel N" <ufo orbiting.us> writes:
On Monday, 12 May 2014 at 20:36:10 UTC, Walter Bright wrote:
 It's been brought up more than once that the 'scope' storage 
 class is an unimplemented borrowed pointer. But thinking a bit 
 more along those lines, actually 'ref' fills the role of a 
 borrowed pointer.

 One particularly apropos behavior is that struct member 
 functions pass 'this' by ref, meaning that members can be 
 called without the inc/dec millstone.

 ref is still incomplete as far as this goes, but we can go the 
 extra distance with it, and then it will be of great help in 
 supporting any ref counting solution.

 What it doesn't work very well with are class references. But 
 Andrei suggested that we can focus the use of 'scope' to deal 
 with that in an analogous way.

 What do you think?

 Anyone want to enumerate a list of the current deficiencies of 
 'ref' in regards to this, so we can think about solving it?

I would prefer 'scope ref' that would allow the solution for classes and everything else to be unified, i.e. everything uses scope. When it comes to the implicit 'this' by ref, it could be redefined to pass by scope ref. Another reason is: I know this doesn't(and might never) work in D, but based on the intuitive meaning of 'ref' I fully expected the below example to work when I first started learning the language. struct A { ref int a_m; this(ref int a) { a_m = a; } } Whereas 'scope' on the other hand is self documenting imho.
May 12 2014
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/12/2014 10:36 PM, Walter Bright wrote:
 It's been brought up more than once that the 'scope' storage class is an
unimplemented borrowed pointer. But thinking a bit more along those lines,
actually 'ref' fills the role of a borrowed pointer.

 One particularly apropos behavior is that struct member functions pass 'this'
by ref, meaning that members can be called without the inc/dec millstone.

 ref is still incomplete as far as this goes, but we can go the extra distance
with it, and then it will be of great help in supporting any ref counting
solution.

 What it doesn't work very well with are class references. But Andrei suggested
that we can focus the use of 'scope' to deal with that in an analogous way.

 What do you think?

I think everything should be treated uniformly. But a storage class is not sufficient.
 Anyone want to enumerate a list of the current deficiencies of 'ref' in
 regards to this, so we can think about solving it?

Eg: - Cannot make tail const. / Cannot be reassigned. - Cannot store in data structures. - Cannot borrow slices of memory. - Closures? - (Probably more)
May 12 2014
prev sibling next sibling parent "logicchains" <jonathan.t.barnard gmail.com> writes:
On Monday, 12 May 2014 at 21:15:38 UTC, Steven Schveighoffer 
wrote:
 Hm... the one piece that I think would be damaging is to not be 
 able to take an address of the 'this' reference. It's probably 
 OK to just use pointers and static functions in some cases, but 
 member functions do not have that luxury. In other words, 
 operators.

 Big example would be a doubly linked list used with ~=.

 -Steve

This sounds a bit like an 'issue' of sorts that Rust has with borrowed pointers, where certain types of datastructures cannot be written without resorting to the 'unsafe' parts of the language. The solution they've adopted is having such code written in libraries so that the user doesn't have to mess around with 'unsafe'.
May 12 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 May 2014 06:36, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 It's been brought up more than once that the 'scope' storage class is an
 unimplemented borrowed pointer. But thinking a bit more along those lines,
 actually 'ref' fills the role of a borrowed pointer.

 One particularly apropos behavior is that struct member functions pass
 'this' by ref, meaning that members can be called without the inc/dec
 millstone.

 ref is still incomplete as far as this goes, but we can go the extra
 distance with it, and then it will be of great help in supporting any ref
 counting solution.

 What it doesn't work very well with are class references. But Andrei
 suggested that we can focus the use of 'scope' to deal with that in an
 analogous way.

 What do you think?

 Anyone want to enumerate a list of the current deficiencies of 'ref' in
 regards to this, so we can think about solving it?

I agree, I think finishing scope appears to deserve a priority boost, it would be enabling to a lot of developments in D to have reliable escape analysis. It seems more problematic to repurpose ref than to finish scope though. ref would change meaning quite significantly. ref would probably have to become part of the type (I can imagine needs for overloads arising?). You would need to be able to make ref locals, and ref members of structs so you can do useful work with them. You'd need to be able to create an array of 'ref's I think by-value scope still has some value too. A small struct that's passed by value (like slices) may contain a pointer. You shouldn't need to handle that small struct by reference when you really just wanted to attribute it with scope. I never saw any problems with the scope idea as it stood, and I think ref is still useful in it's existing incarnation; the same way that it's useful in C++, ie, a pointer that must be initialised, hides reassignment and offset/indexing semantics (which can often interfere with generic code). extern(C++) would gain a new problem if ref were repurposed.
May 12 2014
prev sibling next sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2014-05-13 at 04:07 +0000, logicchains via Digitalmars-d wrote:
[…]
 This sounds a bit like an 'issue' of sorts that Rust has with 
 borrowed pointers, where certain types of datastructures cannot 
 be written without resorting to the 'unsafe' parts of the 
 language. The solution they've adopted is having such code 
 written in libraries so that the user doesn't have to mess around 
 with 'unsafe'.

Probably re-finding many of the things people have to use sun.misc.Unsafe for on the JVM. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
May 12 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 13 May 2014 at 04:46:41 UTC, Russel Winder via
Digitalmars-d wrote:
 On Tue, 2014-05-13 at 04:07 +0000, logicchains via 
 Digitalmars-d wrote:
 […]
 This sounds a bit like an 'issue' of sorts that Rust has with 
 borrowed pointers, where certain types of datastructures 
 cannot be written without resorting to the 'unsafe' parts of 
 the language. The solution they've adopted is having such code 
 written in libraries so that the user doesn't have to mess 
 around with 'unsafe'.

Probably re-finding many of the things people have to use sun.misc.Unsafe for on the JVM.

Which is why the Java designers are looking on how to make Unsafe an official package as of Java 9. And did the survey a few months ago, about how Unsafe was being used in major Java projects. -- Paulo
May 12 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 12 May 2014 at 20:36:10 UTC, Walter Bright wrote:
 It's been brought up more than once that the 'scope' storage 
 class is an unimplemented borrowed pointer. But thinking a bit 
 more along those lines, actually 'ref' fills the role of a 
 borrowed pointer.

 One particularly apropos behavior is that struct member 
 functions pass 'this' by ref, meaning that members can be 
 called without the inc/dec millstone.

 ref is still incomplete as far as this goes, but we can go the 
 extra distance with it, and then it will be of great help in 
 supporting any ref counting solution.

 What it doesn't work very well with are class references. But 
 Andrei suggested that we can focus the use of 'scope' to deal 
 with that in an analogous way.

 What do you think?

 Anyone want to enumerate a list of the current deficiencies of 
 'ref' in regards to this, so we can think about solving it?

There are 2 `scope` uses to think about. One is storage class and in that context `scope` is more of owned / unique pointer. Other is parameter qualifier and that one is closer to ref / borrowed pointer. Main problem about making `ref` borrowed pointer is that you will need to prohibit storing it in function transitively. This will need to become invalid code: struct A { int* ptr; } int* gptr; void foo(ref A a) { gptr = a.ptr; // error, can't leak borrowed a.ptr into global context } This feels like too much of a breakage, this is why `scope` (or `scope ref`) feels more appropriate.
May 13 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 13/05/14 15:36, Dicebot wrote:

 There are 2 `scope` uses to think about. One is storage class and in
 that context `scope` is more of owned / unique pointer. Other is
 parameter qualifier and that one is closer to ref / borrowed pointer.

 Main problem about making `ref` borrowed pointer is that you will need
 to prohibit storing it in function transitively. This will need to
 become invalid code:

 struct A
 {
      int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
      gptr = a.ptr; // error, can't leak borrowed a.ptr into global context
 }

 This feels like too much of a breakage, this is why `scope` (or `scope
 ref`) feels more appropriate.

I always though "scope" would behave like that. -- /Jacob Carlborg
May 13 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 6:50 AM, Dicebot wrote:
 Walter's initial post implies that he wanted to re-used `ref` for borrowed
 pointer (which would mean same semantics as `scope` parameter qualifier)

'ref' already is to much extent, for example: safe int foo(ref int x) { auto a = &x; return 3; } dmd foo -c foo.d(4): Error: cannot take address of parameter x in safe function foo
May 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2014 12:29 AM, Idan Arye wrote:
 On Tuesday, 13 May 2014 at 17:44:02 UTC, Walter Bright wrote:
 On 5/13/2014 6:50 AM, Dicebot wrote:
 Walter's initial post implies that he wanted to re-used `ref` for borrowed
 pointer (which would mean same semantics as `scope` parameter qualifier)

'ref' already is to much extent, for example: safe int foo(ref int x) { auto a = &x; return 3; } dmd foo -c foo.d(4): Error: cannot take address of parameter x in safe function foo

This has nothing to do with `ref`. If you remove the `ref` you get the exact same error.

Right, but the thing is, the only address of a local/parameter you can take is by ref, not &.
May 14 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 10:09 AM, Steven Schveighoffer wrote:
 Yes, the difference here is that scope is a storage class, and only affects the
 "head", whereas borrowed would have to be transitive.

Are you sure it would have to be transitive?
May 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 11:21 AM, Steven Schveighoffer wrote:
 I'm not certain I understand the concept correctly, but from Dicebot's code
 example, it is transitive.

I know Dicebot's example assumes a transitivity property, but I question that assumption. In my cursory reading of Rust documentation, it wasn't clear to me if it was transitive or not.
May 13 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 6:36 AM, Dicebot wrote:
 Main problem about making `ref` borrowed pointer is that you will need to
 prohibit storing it in function transitively. This will need to become invalid
 code:

 struct A
 {
      int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
      gptr = a.ptr; // error, can't leak borrowed a.ptr into global context
 }

The lifetime of &a is not at all the same as the lifetime of a.ptr, those are independent pointers. I.e. ref is not transitive (unlike const which is transitive).
May 13 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 10:52 AM, Dicebot wrote:
 It has to be transitive to be useful as borrowed pointer. Consider this
example:

 {
      scope A a; // has some internally managed resources
      foo(a);
 }

 It is not safe to destruct a in the end of the scope here because foo may have
 stored references to a owned resources. But if foo signature is `foo(scope ref
A
 a)` then compiler can statically verify that it is safe which is the very point
 of borrowing guarantees. It must be transitive to guarantee anything of course.

If those internal resources of A are marked as refcounted, then transitivity is not necessary. Consider also that a struct A can completely control any escaping references - transitive borrowing is not necessary.
May 13 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 12:06 PM, Dicebot wrote:
 No, it still can be necessary. `scope` can greatly help not only with resource
 releasing, it is also missing tool to safely cast from shared. Locking shared
 variable can return same variable casted to scope qualifier which will
 guarantee  that no reference has been stored to shared object by the time lock
 is released.

I believe that is the role of `unique`. DIP69 addresses making unique pointers in D, and there have been several PR's implementing aspects of it.
 And "if those are marked as refcounted" as assumption is no better than "if
 those are owned by GC" ;)

I think that an object that wants to completely own its resources must properly encapsulate and restrict unsafe access to them itself.
 Also A can only control escaping of any internal references only by completely
 prohibiting access to it which is not good. You have no means to say "feel free
 to use this reference as long as you don't keep it outside of current scope".
 And you effectively say "make all your array members private to keep borrowing
 guarantees".

You can by only returning ref's.
May 13 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 1:46 PM, Walter Bright wrote:
 I believe that is the role of `unique`. DIP69 addresses making unique pointers
 in D, and there have been several PR's implementing aspects of it.

Argh, DIP29: http://wiki.dlang.org/DIP29
May 13 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2014 6:09 AM, Dicebot wrote:
 On Tuesday, 13 May 2014 at 20:46:19 UTC, Walter Bright wrote:
 On 5/13/2014 12:06 PM, Dicebot wrote:
 No, it still can be necessary. `scope` can greatly help not only with resource
 releasing, it is also missing tool to safely cast from shared. Locking shared
 variable can return same variable casted to scope qualifier which will
 guarantee  that no reference has been stored to shared object by the time lock
 is released.

I believe that is the role of `unique`. DIP69 addresses making unique pointers in D, and there have been several PR's implementing aspects of it.

I don't really understand how you see this working. Sure, unique concept from that DIP can somewhat replace `scope` storage class. But you can't prove that some function does not leak unique variable internals if this is not annotated as such.

I'm not suggesting unique can replace scope, I am suggesting they are separate concepts.
 And "if those are marked as refcounted" as assumption is no better than "if
 those are owned by GC" ;)

I think that an object that wants to completely own its resources must properly encapsulate and restrict unsafe access to them itself.

This statement is not much different from "any programmer who cares about memory should manage it manually" or "instead of const qualifier you can simply use convention". Doing this without compiler help is tedious. Mistakes result in bugs that are insanely hard to debug (storing non-shared reference to shared object by an accident).

My comment applies when the destructor behaves as if it owns resources other than the object instance itself, in which case the class designer is already manually managing them.
 Also A can only control escaping of any internal references only by completely
 prohibiting access to it which is not good. You have no means to say "feel free
 to use this reference as long as you don't keep it outside of current scope".
 And you effectively say "make all your array members private to keep borrowing
 guarantees".

You can by only returning ref's.

Also slices and pointers (or structs with pointers inside).

The idea is that 'ref' are borrowed pointers, so if you're returning pointers, the borrowed semantics do not apply.
May 14 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2014 5:18 AM, Dicebot wrote:
 On Wednesday, 14 May 2014 at 19:03:20 UTC, Walter Bright wrote:
 The idea is that 'ref' are borrowed pointers, so if you're returning pointers,
 the borrowed semantics do not apply.

Somewhat more extended example: struct Buffer { private byte[] data; this() { this.data = (cast(byte*)malloc(42))[0..42]; } ~this() { free(this.data.ptr); } byte[] get(size_t a, size_t b) { return this.data[a..b]; } } void foo(ref Buffer buff) { // here be trouble static int[] slice = buff.get(10, 20); } void foo2() { Buffer buff; foo(buff); // destructor gets called, foo now has pointer to freed memory } Transitivity of borrowing ensures that you can use any object as an argument for function that takes a borrowed pointer and no reference to its internals will persist. Whatever memory management model of object type is. With such borrowing implementation this example code is also totally safe in spirit (assignment to static var will result in compile-time error).

get() is returning a pointer to its internally managed data (in the form of []). You're right that transitivity of borrowing would support this safely, but I am not proposing that for ref. To make slicing Buffer safe, one would have to overload opSlice and then manage access to the slice.
May 15 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2014 9:43 AM, Dicebot wrote:
 Transitive
 borrowing solves certain class of issues that currently rely on convention,
 enabling whole new type of verified safe code (both memory safe and concurrency
 safe). Head-only? Doesn't look so.

I'm concerned that transitive borrowing will *preclude* a number of useful cases.
May 16 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2014 10:33 AM, Dicebot wrote:
 On Friday, 16 May 2014 at 17:22:21 UTC, Walter Bright wrote:
 On 5/16/2014 9:43 AM, Dicebot wrote:
 Transitive
 borrowing solves certain class of issues that currently rely on convention,
 enabling whole new type of verified safe code (both memory safe and concurrency
 safe). Head-only? Doesn't look so.

I'm concerned that transitive borrowing will *preclude* a number of useful cases.

Which is why `ref` itself can't be used for that and usage of `scope` as qualifier is necessary to enable transitive behavior :)

True, but there comes a point where something gets complicated enough that nobody understands it and the implementation gets full of bugs.
May 16 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2014 1:18 PM, Dicebot wrote:
 Also I believe `scope` is one of concepts that are hard to define but
incredibly
 easy to grasp intuitively.

How and where to put the annotations, and the notion of transitivity, are not so incredibly easy to grasp. The mix of ref and scope is also a bit of a wtf.
May 16 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-05-13 19:52, Dicebot wrote:

 It has to be transitive to be useful as borrowed pointer. Consider this
 example:

 {
      scope A a; // has some internally managed resources
      foo(a);
 }

 It is not safe to destruct a in the end of the scope here because foo
 may have stored references to a owned resources. But if foo signature is
 `foo(scope ref A a)` then compiler can statically verify that it is safe
 which is the very point of borrowing guarantees. It must be transitive
 to guarantee anything of course.

What is "scope ref" supposed to do in this example, compared to just "scope"? -- /Jacob Carlborg
May 13 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-05-14 15:00, Dicebot wrote:

 To be a reference ;) But yeah, it is not important in this example,
 plain scope should behave the same if transitive.

I though that "A" was a class in the previous example, but now I see that it was a struct. -- /Jacob Carlborg
May 14 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 13:40:42 UTC, Jacob Carlborg wrote:
 On 13/05/14 15:36, Dicebot wrote:

 There are 2 `scope` uses to think about. One is storage class 
 and in
 that context `scope` is more of owned / unique pointer. Other 
 is
 parameter qualifier and that one is closer to ref / borrowed 
 pointer.

 Main problem about making `ref` borrowed pointer is that you 
 will need
 to prohibit storing it in function transitively. This will 
 need to
 become invalid code:

 struct A
 {
     int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
     gptr = a.ptr; // error, can't leak borrowed a.ptr into 
 global context
 }

 This feels like too much of a breakage, this is why `scope` 
 (or `scope
 ref`) feels more appropriate.

I always though "scope" would behave like that.

Walter's initial post implies that he wanted to re-used `ref` for borrowed pointer (which would mean same semantics as `scope` parameter qualifier)
May 13 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 May 2014 09:50:12 -0400, Dicebot <public dicebot.lv> wrote:

 On Tuesday, 13 May 2014 at 13:40:42 UTC, Jacob Carlborg wrote:
 On 13/05/14 15:36, Dicebot wrote:

 There are 2 `scope` uses to think about. One is storage class and in
 that context `scope` is more of owned / unique pointer. Other is
 parameter qualifier and that one is closer to ref / borrowed pointer.

 Main problem about making `ref` borrowed pointer is that you will need
 to prohibit storing it in function transitively. This will need to
 become invalid code:

 struct A
 {
     int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
     gptr = a.ptr; // error, can't leak borrowed a.ptr into global  
 context
 }

 This feels like too much of a breakage, this is why `scope` (or `scope
 ref`) feels more appropriate.

I always though "scope" would behave like that.

Walter's initial post implies that he wanted to re-used `ref` for borrowed pointer (which would mean same semantics as `scope` parameter qualifier)

Yes, the difference here is that scope is a storage class, and only affects the "head", whereas borrowed would have to be transitive. -Steve
May 13 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 17:09:17 UTC, Steven Schveighoffer 
wrote:
 On Tue, 13 May 2014 09:50:12 -0400, Dicebot <public dicebot.lv> 
 wrote:

 On Tuesday, 13 May 2014 at 13:40:42 UTC, Jacob Carlborg wrote:
 On 13/05/14 15:36, Dicebot wrote:

 There are 2 `scope` uses to think about. One is storage 
 class and in
 that context `scope` is more of owned / unique pointer. 
 Other is
 parameter qualifier and that one is closer to ref / borrowed 
 pointer.

 Main problem about making `ref` borrowed pointer is that you 
 will need
 to prohibit storing it in function transitively. This will 
 need to
 become invalid code:

 struct A
 {
    int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
    gptr = a.ptr; // error, can't leak borrowed a.ptr into 
 global context
 }

 This feels like too much of a breakage, this is why `scope` 
 (or `scope
 ref`) feels more appropriate.

I always though "scope" would behave like that.

Walter's initial post implies that he wanted to re-used `ref` for borrowed pointer (which would mean same semantics as `scope` parameter qualifier)

Yes, the difference here is that scope is a storage class, and only affects the "head", whereas borrowed would have to be transitive. -Steve

`scope` has to be both storage class and qualifier to work
May 13 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 17:41:23 UTC, Walter Bright wrote:
 On 5/13/2014 6:36 AM, Dicebot wrote:
 Main problem about making `ref` borrowed pointer is that you 
 will need to
 prohibit storing it in function transitively. This will need 
 to become invalid
 code:

 struct A
 {
     int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
     gptr = a.ptr; // error, can't leak borrowed a.ptr into 
 global context
 }

The lifetime of &a is not at all the same as the lifetime of a.ptr, those are independent pointers. I.e. ref is not transitive (unlike const which is transitive).

It has to be transitive to be useful as borrowed pointer. Consider this example: { scope A a; // has some internally managed resources foo(a); } It is not safe to destruct a in the end of the scope here because foo may have stored references to a owned resources. But if foo signature is `foo(scope ref A a)` then compiler can statically verify that it is safe which is the very point of borrowing guarantees. It must be transitive to guarantee anything of course.
May 13 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 May 2014 13:51:21 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 5/13/2014 10:09 AM, Steven Schveighoffer wrote:
 Yes, the difference here is that scope is a storage class, and only  
 affects the
 "head", whereas borrowed would have to be transitive.

Are you sure it would have to be transitive?

I'm not certain I understand the concept correctly, but from Dicebot's code example, it is transitive. I defer to the experts. Just trying to help explain the point he was making (and clearly doing a poor job). -Steve
May 13 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 18:48:14 UTC, Walter Bright wrote:
 On 5/13/2014 10:52 AM, Dicebot wrote:
 It has to be transitive to be useful as borrowed pointer. 
 Consider this example:

 {
     scope A a; // has some internally managed resources
     foo(a);
 }

 It is not safe to destruct a in the end of the scope here 
 because foo may have
 stored references to a owned resources. But if foo signature 
 is `foo(scope ref A
 a)` then compiler can statically verify that it is safe which 
 is the very point
 of borrowing guarantees. It must be transitive to guarantee 
 anything of course.

If those internal resources of A are marked as refcounted, then transitivity is not necessary. Consider also that a struct A can completely control any escaping references - transitive borrowing is not necessary.

No, it still can be necessary. `scope` can greatly help not only with resource releasing, it is also missing tool to safely cast from shared. Locking shared variable can return same variable casted to scope qualifier which will guarantee that no reference has been stored to shared object by the time lock is released. And "if those are marked as refcounted" as assumption is no better than "if those are owned by GC" ;) Also A can only control escaping of any internal references only by completely prohibiting access to it which is not good. You have no means to say "feel free to use this reference as long as you don't keep it outside of current scope". And you effectively say "make all your array members private to keep borrowing guarantees". Rust situation is quite different here because all their safe pointers have ownership/lifetime annotation. D doesn't and thus imaginary scope/borrowed rules need to assume worst case scenarios (which is still good enough for many cases).
May 13 2014
parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, May 16, 2014 at 11:57:36AM -0700, Walter Bright via Digitalmars-d wrote:
 On 5/16/2014 10:33 AM, Dicebot wrote:
On Friday, 16 May 2014 at 17:22:21 UTC, Walter Bright wrote:
On 5/16/2014 9:43 AM, Dicebot wrote:
Transitive borrowing solves certain class of issues that currently
rely on convention, enabling whole new type of verified safe code
(both memory safe and concurrency safe). Head-only? Doesn't look
so.

I'm concerned that transitive borrowing will *preclude* a number of useful cases.

Which is why `ref` itself can't be used for that and usage of `scope` as qualifier is necessary to enable transitive behavior :)

True, but there comes a point where something gets complicated enough that nobody understands it and the implementation gets full of bugs.

*snicker* What I am going to tell you about is what we teach our programming students in the third or fourth year of graduate school... It is my task to convince you not to turn away because you don't understand it. You see my programming students don't understand it... That is because I don't understand it. Nobody does. Richard Deeman Source: dlang.org. :-P T -- It is impossible to make anything foolproof because fools are so ingenious. -- Sammy
May 16 2014
prev sibling next sibling parent "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 13 May 2014 at 17:44:02 UTC, Walter Bright wrote:
 On 5/13/2014 6:50 AM, Dicebot wrote:
 Walter's initial post implies that he wanted to re-used `ref` 
 for borrowed
 pointer (which would mean same semantics as `scope` parameter 
 qualifier)

'ref' already is to much extent, for example: safe int foo(ref int x) { auto a = &x; return 3; } dmd foo -c foo.d(4): Error: cannot take address of parameter x in safe function foo

This has nothing to do with `ref`. If you remove the `ref` you get the exact same error.
May 14 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 19:17:16 UTC, Jacob Carlborg wrote:
 What is "scope ref" supposed to do in this example, compared to 
 just "scope"?

To be a reference ;) But yeah, it is not important in this example, plain scope should behave the same if transitive.
May 14 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 20:46:19 UTC, Walter Bright wrote:
 On 5/13/2014 12:06 PM, Dicebot wrote:
 No, it still can be necessary. `scope` can greatly help not 
 only with resource
 releasing, it is also missing tool to safely cast from shared. 
 Locking shared
 variable can return same variable casted to scope qualifier 
 which will
 guarantee  that no reference has been stored to shared object 
 by the time lock
 is released.

I believe that is the role of `unique`. DIP69 addresses making unique pointers in D, and there have been several PR's implementing aspects of it.

I don't really understand how you see this working. Sure, unique concept from that DIP can somewhat replace `scope` storage class. But you can't prove that some function does not leak unique variable internals if this is not annotated as such.
 And "if those are marked as refcounted" as assumption is no 
 better than "if
 those are owned by GC" ;)

I think that an object that wants to completely own its resources must properly encapsulate and restrict unsafe access to them itself.

This statement is not much different from "any programmer who cares about memory should manage it manually" or "instead of const qualifier you can simply use convention". Doing this without compiler help is tedious. Mistakes result in bugs that are insanely hard to debug (storing non-shared reference to shared object by an accident).
 Also A can only control escaping of any internal references 
 only by completely
 prohibiting access to it which is not good. You have no means 
 to say "feel free
 to use this reference as long as you don't keep it outside of 
 current scope".
 And you effectively say "make all your array members private 
 to keep borrowing
 guarantees".

You can by only returning ref's.

Also slices and pointers (or structs with pointers inside).
May 14 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 14 May 2014 at 19:03:20 UTC, Walter Bright wrote:
 Also A can only control escaping of any internal references 
 only by completely
 prohibiting access to it which is not good. You have no 
 means to say "feel free
 to use this reference as long as you don't keep it outside 
 of current scope".
 And you effectively say "make all your array members private 
 to keep borrowing
 guarantees".

You can by only returning ref's.

Also slices and pointers (or structs with pointers inside).

The idea is that 'ref' are borrowed pointers, so if you're returning pointers, the borrowed semantics do not apply.

Somewhat more extended example: struct Buffer { private byte[] data; this() { this.data = (cast(byte*)malloc(42))[0..42]; } ~this() { free(this.data.ptr); } byte[] get(size_t a, size_t b) { return this.data[a..b]; } } void foo(ref Buffer buff) { // here be trouble static int[] slice = buff.get(10, 20); } void foo2() { Buffer buff; foo(buff); // destructor gets called, foo now has pointer to freed memory } Transitivity of borrowing ensures that you can use any object as an argument for function that takes a borrowed pointer and no reference to its internals will persist. Whatever memory management model of object type is. With such borrowing implementation this example code is also totally safe in spirit (assignment to static var will result in compile-time error).
May 15 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 15 May 2014 at 18:08:06 UTC, Walter Bright wrote:
 get() is returning a pointer to its internally managed data (in 
 the form of []). You're right that transitivity of borrowing 
 would support this safely, but I am not proposing that for ref. 
 To make slicing Buffer safe, one would have to overload opSlice 
 and then manage access to the slice.

Sure, I simply question its practical applicability in that case. It is not like I want borrowed pointer semantics only because it looks cool :) Transitive borrowing solves certain class of issues that currently rely on convention, enabling whole new type of verified safe code (both memory safe and concurrency safe). Head-only? Doesn't look so.
May 16 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 16 May 2014 at 17:22:21 UTC, Walter Bright wrote:
 On 5/16/2014 9:43 AM, Dicebot wrote:
 Transitive
 borrowing solves certain class of issues that currently rely 
 on convention,
 enabling whole new type of verified safe code (both memory 
 safe and concurrency
 safe). Head-only? Doesn't look so.

I'm concerned that transitive borrowing will *preclude* a number of useful cases.

Which is why `ref` itself can't be used for that and usage of `scope` as qualifier is necessary to enable transitive behavior :)
May 16 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 16 May 2014 at 18:57:34 UTC, Walter Bright wrote:
 On 5/16/2014 10:33 AM, Dicebot wrote:
 On Friday, 16 May 2014 at 17:22:21 UTC, Walter Bright wrote:
 On 5/16/2014 9:43 AM, Dicebot wrote:
 Transitive
 borrowing solves certain class of issues that currently rely 
 on convention,
 enabling whole new type of verified safe code (both memory 
 safe and concurrency
 safe). Head-only? Doesn't look so.

I'm concerned that transitive borrowing will *preclude* a number of useful cases.

Which is why `ref` itself can't be used for that and usage of `scope` as qualifier is necessary to enable transitive behavior :)

True, but there comes a point where something gets complicated enough that nobody understands it and the implementation gets full of bugs.

Then we are back to square one "`scope` is needed but difficult to implement so lets not touch it" :) Which is understandable but does not warrant adding mostly useless concept simply because it is easier. Also I believe `scope` is one of concepts that are hard to define but incredibly easy to grasp intuitively.
May 16 2014