www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Full closures

reply Frank Benoit <keinfarbton googlemail.com> writes:
Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
i think the current full closure implementation should take a redesign.

My suggestion:
Let D have 2 kind of delegate closures (I don't know how to call this 
correctly)
local delegates and heap delegates

local delegate are that from D1 without modifiction.
They can access every local variable in the surrounding scope, no heap 
allocation if the addess is taken. It is expected the adress is not used 
outside the allowed scope.

heap delegates
- can only access variables from the surrounding scope, if they are 
marked as 'const', 'invariant' or 'final'. Which means, they are not 
expected to changed after initialization.
- with instantiation of the delegate, a heap allocated frame is used to 
store a copy of the reference 'final' varialbles for the delegate.
That means in case of a loop, the delegate get a new heap allocation 
with each iteration.

foreach( element; container ){
   const c = element;
   logLater( new { writefln(c); });
}
Aug 15 2008
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Frank,

 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043 i
 think the current full closure implementation should take a redesign.
 
 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this
 correctly)
 local delegates and heap delegates
 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no heap
 allocation if the addess is taken. It is expected the adress is not
 used
 outside the allowed scope.
 heap delegates
 - can only access variables from the surrounding scope, if they are
 marked as 'const', 'invariant' or 'final'. Which means, they are not
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used
 to
 store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation
 with each iteration.
 foreach( element; container ){
 const c = element;
 logLater( new { writefln(c); });
 }

Sounds reasonable. I like that it makes the semantics of referenced variables explicit. However I would amend it by allowing access to non const (etc.) function arguments (which would be copied on function entry) and variables who's scope is exactly that of the function. void fn(int thisArg) { int andThis; { int butNotThis; } } These would greatly increase the practicality of the design without adding ambiguities. On second thought, the function args might be bit of a trick. I could live without them.
Aug 15 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
BCS wrote:
 Reply to Frank,
 
 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043 i
 think the current full closure implementation should take a redesign.

 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this
 correctly)
 local delegates and heap delegates
 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no heap
 allocation if the addess is taken. It is expected the adress is not
 used
 outside the allowed scope.
 heap delegates
 - can only access variables from the surrounding scope, if they are
 marked as 'const', 'invariant' or 'final'. Which means, they are not
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used
 to
 store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation
 with each iteration.
 foreach( element; container ){
 const c = element;
 logLater( new { writefln(c); });
 }

Sounds reasonable. I like that it makes the semantics of referenced variables explicit. However I would amend it by allowing access to non const (etc.) function arguments (which would be copied on function entry) and variables who's scope is exactly that of the function. void fn(int thisArg) { int andThis; { int butNotThis; } } These would greatly increase the practicality of the design without adding ambiguities. On second thought, the function args might be bit of a trick. I could live without them.

Ewww, -- changing the scope of the variable should never change its behavior, IMO.
Aug 15 2008
parent reply BCS <ao pathlink.com> writes:
Reply to Robert,

 BCS wrote:
 
 [...] allowing access to non const (etc.)
 function arguments (which would be copied on function entry) and
 variables who's scope is exactly that of the function.
 

Ewww, -- changing the scope of the variable should never change its behavior, IMO.

That is a good point, but how else you you get a real closure that has mutable state? int delegate() Seq() { int at = 0; int Next() { return at++; } return &Next; }
Aug 15 2008
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
BCS wrote:
 
 That is a good point, but how else you you get a real closure that has 
 mutable state?
 
 int delegate() Seq()
 {
   int at = 0;
   int Next() { return at++; }
   return &Next;
 }
 
 

Err.... just have a closure like D has now?... one that allows access to any visible variable? -- Bruno Medeiros - Software Developer, MSc. in CS/E graduate http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Aug 18 2008
parent reply BCS <ao pathlink.com> writes:
Reply to Bruno,

 BCS wrote:
 
 That is a good point, but how else you you get a real closure that
 has mutable state?
 
 int delegate() Seq()
 {
 int at = 0;
 int Next() { return at++; }
 return &Next;
 }

to any visible variable?

Franks local delegates won't work because they aren't valid after the other function returns, his heap delegates won't work because the variables they can access can't be altered.
Aug 18 2008
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
BCS wrote:
 Reply to Bruno,
 
 BCS wrote:

 That is a good point, but how else you you get a real closure that
 has mutable state?

 int delegate() Seq()
 {
 int at = 0;
 int Next() { return at++; }
 return &Next;
 }

to any visible variable?

Franks local delegates won't work because they aren't valid after the other function returns, his heap delegates won't work because the variables they can access can't be altered.

I feel there is a misunderstanding here. I wasn't talking about Frank's closures, but the one D currently has. What exactly did you mean with "how else [do] you get a real closure that has mutable state?" ? -- Bruno Medeiros - Software Developer, MSc. in CS/E graduate http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Aug 25 2008
parent BCS <ao pathlink.com> writes:
Reply to Bruno,

 BCS wrote:
 
 Reply to Bruno,
 
 BCS wrote:
 
 That is a good point, but how else you you get a real closure that
 has mutable state?
 
 int delegate() Seq()
 {
 int at = 0;
 int Next() { return at++; }
 return &Next;
 }

access to any visible variable?

other function returns, his heap delegates won't work because the variables they can access can't be altered.

Frank's closures, but the one D currently has. What exactly did you mean with "how else [do] you get a real closure that has mutable state?" ?

The problem with the current ones is that they aren't valid in some cases if the outer function leave the scope the delegate is defined in. void Foo() { int delegate() Bar; for(int m = 3; m; m--) { const int i = m; Ber = {return i;} } { int j = 6; int k = Bar(); //k could be 6 } }
Aug 25 2008
prev sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Frank Benoit wrote:
 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
 i think the current full closure implementation should take a redesign.
 
 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this 
 correctly)
 local delegates and heap delegates
 
 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no heap 
 allocation if the addess is taken. It is expected the adress is not used 
 outside the allowed scope.
 

Refresh my memory on why would that be wanted. Is it because the compiler incorrectly heap allocates some variables in a situation that is statically verifiable that such allocation wasn't necessary? Or is it more of a programmer's help, to avoid him/her making mistakes and writing code that inadvertently causes locals to be heap allocated?
 heap delegates
 - can only access variables from the surrounding scope, if they are 
 marked as 'const', 'invariant' or 'final'. Which means, they are not 
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used to 
 store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation 
 with each iteration.
 
 foreach( element; container ){
   const c = element;
   logLater( new { writefln(c); });
 }
 
 

Why would we want such version of "heap delegates" that are more restrictive in power than D's current full closures? -- Bruno Medeiros - Software Developer, MSc. in CS/E graduate http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Aug 18 2008
parent reply Frank Benoit <keinfarbton googlemail.com> writes:
Bruno Medeiros schrieb:
 Frank Benoit wrote:
 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
 i think the current full closure implementation should take a redesign.

 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this 
 correctly)
 local delegates and heap delegates

 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no heap 
 allocation if the addess is taken. It is expected the adress is not 
 used outside the allowed scope.

Refresh my memory on why would that be wanted. Is it because the compiler incorrectly heap allocates some variables in a situation that is statically verifiable that such allocation wasn't necessary? Or is it more of a programmer's help, to avoid him/her making mistakes and writing code that inadvertently causes locals to be heap allocated?

the D1 nested functions or anonymous classes can access local variable from the surrounding scope. local vars are located on the stack. if the reference to the nested function of anonymous class is escaping the local scope (by storing it somewhere or passing it or returning it) and the surrounding scope if left, the variables are no more valid. Running the nested function or anonymous class accessing those variable will result in crashes. So the D1 feature is very good in performance, but it should be taken care or the livetime of the accessed variables. The D2 "full closure" feature allocated the whole stack frame on the stack, if the compile detects the "take address from nested function".
 
 heap delegates
 - can only access variables from the surrounding scope, if they are 
 marked as 'const', 'invariant' or 'final'. Which means, they are not 
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used 
 to store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation 
 with each iteration.

 foreach( element; container ){
   const c = element;
   logLater( new { writefln(c); });
 }

Why would we want such version of "heap delegates" that are more restrictive in power than D's current full closures?

The current D2 approach of simply allocating the whole stack frame looks good on the first sight, but IMHO it is not sufficient. - it breaks the old semantic - it always do heap allocation, but the old semantic is an important D feature. - it make invariant/const data change value, if used in a loop. See the link to the bug report. This is why i think, for the case of "heap delegates" the access should be restricted to constant data, and that data should be copied to the delegate, instead of heap-ing the surrounding scope stack frame.
Aug 18 2008
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Frank Benoit wrote:
 Bruno Medeiros schrieb:
 Frank Benoit wrote:
 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
 i think the current full closure implementation should take a redesign.

 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this 
 correctly)
 local delegates and heap delegates

 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no 
 heap allocation if the addess is taken. It is expected the adress is 
 not used outside the allowed scope.

Refresh my memory on why would that be wanted. Is it because the compiler incorrectly heap allocates some variables in a situation that is statically verifiable that such allocation wasn't necessary? Or is it more of a programmer's help, to avoid him/her making mistakes and writing code that inadvertently causes locals to be heap allocated?

the D1 nested functions or anonymous classes can access local variable from the surrounding scope. local vars are located on the stack. if the reference to the nested function of anonymous class is escaping the local scope (by storing it somewhere or passing it or returning it) and the surrounding scope if left, the variables are no more valid. Running the nested function or anonymous class accessing those variable will result in crashes. So the D1 feature is very good in performance, but it should be taken care or the livetime of the accessed variables. The D2 "full closure" feature allocated the whole stack frame on the stack, if the compile detects the "take address from nested function".

Agh, I didn't think it would heap allocate that easily. That's the most conservative approach, and it "indeed puts a serious burden on the use of scoped closures". So I agree we either need: * a way to specify scoped closures/delegates. * a smarter compiler that better detects scoped delegates. I'd say it doesn't have to be 100% accurate, but should be better than the simple heuristic "take address from nested function".
 heap delegates
 - can only access variables from the surrounding scope, if they are 
 marked as 'const', 'invariant' or 'final'. Which means, they are not 
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used 
 to store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation 
 with each iteration.

 foreach( element; container ){
   const c = element;
   logLater( new { writefln(c); });
 }

Why would we want such version of "heap delegates" that are more restrictive in power than D's current full closures?

The current D2 approach of simply allocating the whole stack frame looks good on the first sight, but IMHO it is not sufficient. - it breaks the old semantic

bound to D1.0 compatibility (at least not on such early D stage)
 - it always do heap allocation, but the old semantic is an important D 
 feature.

deferred to only when it's strictly needed, such as when the closure delegate is evaluated/created, instead of when the outer variable is declared.
 - it make invariant/const data change value, if used in a loop.
 
 See the link to the bug report.

-- Bruno Medeiros - Software Developer, MSc. in CS/E graduate http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Aug 18 2008
parent BCS <ao pathlink.com> writes:
Reply to Bruno,

 - it make invariant/const data change value, if used in a loop.
 
 See the link to the bug report.

Yeah, but that's a bug, it should go away.

This is a proposal on how to make it go away. It's a design bug not a compiler bug.
Aug 18 2008