www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - confusing (buggy?) closure behaviour

reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:


and delegates, I was stroke by a phenomenon I cannot explain. Here's the code:

module closures01;

import std.stdio;

alias int delegate(int arg) Handler;

Handler incBy(int n)
{
	return delegate(int arg){ return arg + n; };
}

Handler mulBy(int n)
{
	return delegate(int arg){ return arg * n; };
}

void test1()
{
	writefln("\ntest1:\n----------------------------------------");
	int x = 10, y;
	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
}

void test2()
{
	writefln("\ntest2:\n----------------------------------------");
	int x = 10, y;
	Handler times3 = mulBy(3);
	Handler times4 = mulBy(4);
	Handler plus2 = incBy(2);
	y = times3(x); writefln("%d * 3 -> %d", x, y);
	y = times4(x); writefln("%d * 4 -> %d", x, y);
	y = plus2(x); writefln("%d + 2 -> %d", x, y);
}

public void run()
{
	test1();
	test2();
}

/* **************************************** *
 * Compiled with: Digital Mars D Compiler v1.030
 *
 * (Unexplainable) program output:
 
test1:
----------------------------------------
10 * 3 -> 30
10 * 4 -> 40
10 + 2 -> 12

test2:
----------------------------------------
10 * 3 -> 20
10 * 4 -> 42846880
10 + 2 -> 4284698

* **************************************** */

What goes wrong???
Dec 12 2008
next sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski  
<dmd.zoc spamgourmet.com> wrote:




 with D closures and delegates, I was stroke by a phenomenon I cannot  
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???


I'd say that it works as expected and here is why.

First of all, there are two types of closures:  static and dynamic  
closures.
Closures work by having a hidden pointer to function frame where all local  
variables are stored.

When a static closure is created, all the function local variables are  
stored on stack.
It has an advantage that no memory allocation takes place (fast).
It has a disadvantage that once the delegate leaves the scope, it becomes  
invalid since variables were stored on stack and the stack is probably  
overwritten (unsafe).

Dynamic closure allocates memory in a heap and all the local variables are  
placed there.
It has a disadvantage that memory is allocated for dynamic closure (might  
be slow if dynamic closure are created often).
It has an advantage that dynamic closure may leave the scope, i.e. you may  
save it and call whenever you want.

D1 support static closures only! That's why your code doesn't work (in  
test1 stack is still valid, but in test2 stack gets overwritten)
D2 has support for dynamic closures. Just try it - your sample works as is.
Dec 12 2008
next sibling parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Denis Koroskin Wrote:


 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski  
 <dmd.zoc spamgourmet.com> wrote:
 



 with D closures and delegates, I was stroke by a phenomenon I cannot  
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???


 
 I'd say that it works as expected and here is why.
 
 First of all, there are two types of closures:  static and dynamic  
 closures.
 Closures work by having a hidden pointer to function frame where all local  
 variables are stored.
 
 When a static closure is created, all the function local variables are  
 stored on stack.
 It has an advantage that no memory allocation takes place (fast).
 It has a disadvantage that once the delegate leaves the scope, it becomes  
 invalid since variables were stored on stack and the stack is probably  
 overwritten (unsafe).
 
 Dynamic closure allocates memory in a heap and all the local variables are  
 placed there.
 It has a disadvantage that memory is allocated for dynamic closure (might  
 be slow if dynamic closure are created often).
 It has an advantage that dynamic closure may leave the scope, i.e. you may  
 save it and call whenever you want.
 
 D1 support static closures only! That's why your code doesn't work (in  
 test1 stack is still valid, but in test2 stack gets overwritten)
 D2 has support for dynamic closures. Just try it - your sample works as is.


Thx, Denis, but I'm still confused. The stack thing was also my first thought.
But when I tried to actually explain the dynamics that way, I came to the
conclusion that then, test1() shouldn't have worked either.

I assumed the following (schematic) process takes place:

<code>
mulBy(3)(x)
=> push 3; call mullBy;
// upon entry into mulBy: Stack = [ >&mulBy, 3, ... ]
// I assume, the callee cleans up the stack, so...
// upon return from mulBy: Stack = [ &mulBy, 3, >... ]; cpu_register = &delegate
=> push x; call [cpu_register]
// upon entry into delegate: Stack = [ >&delegate, x, ... ]
</code>

(Here, the stack / frame pointer is denoted by ">", and moves from right to
left on push, left to right on pop)

With that mechanism, when the delegate is entered, the memory where previously
the number 3 was stored, should have been overwritten by x.

But it obviously isn't ?!?

How come?
Dec 12 2008
next sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 12 Dec 2008 22:18:28 +0300, Zoran Isailovski  
<dmd.zoc spamgourmet.com> wrote:


 Denis Koroskin Wrote:


 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:






 experimenting

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???


 I'd say that it works as expected and here is why.

 First of all, there are two types of closures:  static and dynamic
 closures.
 Closures work by having a hidden pointer to function frame where all  
 local
 variables are stored.

 When a static closure is created, all the function local variables are
 stored on stack.
 It has an advantage that no memory allocation takes place (fast).
 It has a disadvantage that once the delegate leaves the scope, it  
 becomes
 invalid since variables were stored on stack and the stack is probably
 overwritten (unsafe).

 Dynamic closure allocates memory in a heap and all the local variables  
 are
 placed there.
 It has a disadvantage that memory is allocated for dynamic closure  
 (might
 be slow if dynamic closure are created often).
 It has an advantage that dynamic closure may leave the scope, i.e. you  
 may
 save it and call whenever you want.

 D1 support static closures only! That's why your code doesn't work (in
 test1 stack is still valid, but in test2 stack gets overwritten)
 D2 has support for dynamic closures. Just try it - your sample works as  
 is.


 Thx, Denis, but I'm still confused. The stack thing was also my first  
 thought. But when I tried to actually explain the dynamics that way, I  
 came to the conclusion that then, test1() shouldn't have worked either.

 I assumed the following (schematic) process takes place:

 <code>
 mulBy(3)(x)
 => push 3; call mullBy;
 // upon entry into mulBy: Stack = [ >&mulBy, 3, ... ]
 // I assume, the callee cleans up the stack, so...
 // upon return from mulBy: Stack = [ &mulBy, 3, >... ]; cpu_register =  
 &delegate
 => push x; call [cpu_register]
 // upon entry into delegate: Stack = [ >&delegate, x, ... ]
 </code>

 (Here, the stack / frame pointer is denoted by ">", and moves from right  
 to left on push, left to right on pop)

 With that mechanism, when the delegate is entered, the memory where  
 previously the number 3 was stored, should have been overwritten by x.

 But it obviously isn't ?!?

 How come?


No, you are taking it slightly wrong.
The delegate stores a raw /pointer to stack frame/ so it doesn't depend on  
current stack head (ESP).
Dec 12 2008
parent Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Denis Koroskin Wrote:


 On Fri, 12 Dec 2008 22:18:28 +0300, Zoran Isailovski  
 <dmd.zoc spamgourmet.com> wrote:
 

 Denis Koroskin Wrote:


 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:






 experimenting

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???


 I'd say that it works as expected and here is why.

 First of all, there are two types of closures:  static and dynamic
 closures.
 Closures work by having a hidden pointer to function frame where all  
 local
 variables are stored.

 When a static closure is created, all the function local variables are
 stored on stack.
 It has an advantage that no memory allocation takes place (fast).
 It has a disadvantage that once the delegate leaves the scope, it  
 becomes
 invalid since variables were stored on stack and the stack is probably
 overwritten (unsafe).

 Dynamic closure allocates memory in a heap and all the local variables  
 are
 placed there.
 It has a disadvantage that memory is allocated for dynamic closure  
 (might
 be slow if dynamic closure are created often).
 It has an advantage that dynamic closure may leave the scope, i.e. you  
 may
 save it and call whenever you want.

 D1 support static closures only! That's why your code doesn't work (in
 test1 stack is still valid, but in test2 stack gets overwritten)
 D2 has support for dynamic closures. Just try it - your sample works as  
 is.


 Thx, Denis, but I'm still confused. The stack thing was also my first  
 thought. But when I tried to actually explain the dynamics that way, I  
 came to the conclusion that then, test1() shouldn't have worked either.

 I assumed the following (schematic) process takes place:

 <code>
 mulBy(3)(x)
 => push 3; call mullBy;
 // upon entry into mulBy: Stack = [ >&mulBy, 3, ... ]
 // I assume, the callee cleans up the stack, so...
 // upon return from mulBy: Stack = [ &mulBy, 3, >... ]; cpu_register =  
 &delegate
 => push x; call [cpu_register]
 // upon entry into delegate: Stack = [ >&delegate, x, ... ]
 </code>

 (Here, the stack / frame pointer is denoted by ">", and moves from right  
 to left on push, left to right on pop)

 With that mechanism, when the delegate is entered, the memory where  
 previously the number 3 was stored, should have been overwritten by x.

 But it obviously isn't ?!?

 How come?


 
 No, you are taking it slightly wrong.
 The delegate stores a raw /pointer to stack frame/ so it doesn't depend on  
 current stack head (ESP).


And that's exactly what I was assuming, Denis. But function calls still use the
stack to pass parameters and store the return address, don't they?

So if the way stacks, the stack pointer (ESP) and the frame pointer (EBP) work
hasn't dramatically changed since I last had business with them, then a
"snapshot" of the situation just before returning from mulBy should look like
this:

         lo --- stack memory --- hi
stack: ..., retadr, 3, ...
ESP:       ^
rawptr to n:       ^

"rawptr" is you raw pointer to the stack frame (the memory block on the stack
visible inside mulBy, i.e., basically the value of EBP inside mulBy).

Indeed, it is not bound to ESP or EBP. Upon return from mulBy, the situation
should look like this

         lo --- stack memory --- hi
stack: ..., retadr, 3, ...
ESP:                     ^
rawptr to n:       ^

rawptr has not changed.

Then, upon call to the delegate, x and the return value are pushed onto the
stack. This changes ESP *and* the contents of the stack memory, probably
resulting into somthing like

         lo --- stack memory --- hi
stack: ..., retadr, x, ...
ESP:        ^
rawptr:              ^

rawptr still points to the same memory, but the memory has been changed. It now
contains the value of x, not 3.

This, at least, used to be how things worked at the machine level.

Don't get me wrong. I'm aware that the above "theory" does not explain the
behavior in my sample code. I'd just like to know why, that's all.
Dec 12 2008
prev sibling parent BCS <ao pathlink.com> writes:
Reply to Zoran,


 Thx, Denis, but I'm still confused. The stack thing was also my first
 thought. But when I tried to actually explain the dynamics that way, I
 came to the conclusion that then, test1() shouldn't have worked
 either.
 



what might be happening is that test1 is not using up enough stack space 
to overwrite the arguments.
Dec 12 2008
prev sibling parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Denis Koroskin Wrote:


 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski  
 <dmd.zoc spamgourmet.com> wrote:
 



 with D closures and delegates, I was stroke by a phenomenon I cannot  
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???


 
 I'd say that it works as expected and here is why.
 
 First of all, there are two types of closures:  static and dynamic  
 closures.
 Closures work by having a hidden pointer to function frame where all local  
 variables are stored.
 
 When a static closure is created, all the function local variables are  
 stored on stack.
 It has an advantage that no memory allocation takes place (fast).
 It has a disadvantage that once the delegate leaves the scope, it becomes  
 invalid since variables were stored on stack and the stack is probably  
 overwritten (unsafe).
 
 Dynamic closure allocates memory in a heap and all the local variables are  
 placed there.
 It has a disadvantage that memory is allocated for dynamic closure (might  
 be slow if dynamic closure are created often).
 It has an advantage that dynamic closure may leave the scope, i.e. you may  
 save it and call whenever you want.
 
 D1 support static closures only! That's why your code doesn't work (in  
 test1 stack is still valid, but in test2 stack gets overwritten)
 D2 has support for dynamic closures. Just try it - your sample works as is.


An addition:

Given the complexness of the criteria when a closure works and when not, I
would vote for a compiler error on inappropriate closures usage. (In Java,
closures cannot handle mutable values on the stack, so it's an error for a
method to return a closure that refers to a non-final argument.)
Dec 12 2008
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 12 Dec 2008 22:28:01 +0300, Zoran Isailovski  
<dmd.zoc spamgourmet.com> wrote:


 Denis Koroskin Wrote:


 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:






 experimenting

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???


 I'd say that it works as expected and here is why.

 First of all, there are two types of closures:  static and dynamic
 closures.
 Closures work by having a hidden pointer to function frame where all  
 local
 variables are stored.

 When a static closure is created, all the function local variables are
 stored on stack.
 It has an advantage that no memory allocation takes place (fast).
 It has a disadvantage that once the delegate leaves the scope, it  
 becomes
 invalid since variables were stored on stack and the stack is probably
 overwritten (unsafe).

 Dynamic closure allocates memory in a heap and all the local variables  
 are
 placed there.
 It has a disadvantage that memory is allocated for dynamic closure  
 (might
 be slow if dynamic closure are created often).
 It has an advantage that dynamic closure may leave the scope, i.e. you  
 may
 save it and call whenever you want.

 D1 support static closures only! That's why your code doesn't work (in
 test1 stack is still valid, but in test2 stack gets overwritten)
 D2 has support for dynamic closures. Just try it - your sample works as  
 is.


 An addition:

 Given the complexness of the criteria when a closure works and when not,  
 I would vote for a compiler error on inappropriate closures usage. (In  
 Java, closures cannot handle mutable values on the stack, so it's an  
 error for a method to return a closure that refers to a non-final  
 argument.)


This would restrict their usage so badly that may make then next to  
useless.

No worries, a dynamic closure is created automatically whenever a static  
one it might be unsafe in D2.

As a rule of thumb, you shoudn't don't return local delegate from a  
function in D1, but you may safely pass them down the call stack.
Dec 12 2008
parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Denis Koroskin Wrote:


 On Fri, 12 Dec 2008 22:28:01 +0300, Zoran Isailovski  
 <dmd.zoc spamgourmet.com> wrote:
 

 Denis Koroskin Wrote:


 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:






 experimenting

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???


 I'd say that it works as expected and here is why.

 First of all, there are two types of closures:  static and dynamic
 closures.
 Closures work by having a hidden pointer to function frame where all  
 local
 variables are stored.

 When a static closure is created, all the function local variables are
 stored on stack.
 It has an advantage that no memory allocation takes place (fast).
 It has a disadvantage that once the delegate leaves the scope, it  
 becomes
 invalid since variables were stored on stack and the stack is probably
 overwritten (unsafe).

 Dynamic closure allocates memory in a heap and all the local variables  
 are
 placed there.
 It has a disadvantage that memory is allocated for dynamic closure  
 (might
 be slow if dynamic closure are created often).
 It has an advantage that dynamic closure may leave the scope, i.e. you  
 may
 save it and call whenever you want.

 D1 support static closures only! That's why your code doesn't work (in
 test1 stack is still valid, but in test2 stack gets overwritten)
 D2 has support for dynamic closures. Just try it - your sample works as  
 is.


 An addition:

 Given the complexness of the criteria when a closure works and when not,  
 I would vote for a compiler error on inappropriate closures usage. (In  
 Java, closures cannot handle mutable values on the stack, so it's an  
 error for a method to return a closure that refers to a non-final  
 argument.)


 
 This would restrict their usage so badly that may make then next to  
 useless.
 
 No worries, a dynamic closure is created automatically whenever a static  
 one it might be unsafe in D2.
 
 As a rule of thumb, you shoudn't don't return local delegate from a  
 function in D1, but you may safely pass them down the call stack.


I don't think it is restrictive if the compiler prevented a situation that
would otherwise lead to a run-time error anyway, or worse, weird and confusing
run-time behavior. In my case, if the compiler couldn't SAFELY handle a
reference to the argument n outside the enclosing function, then, IMO,
RETURNING it (but not otherwise using it) should be flagged a compilation
error. Admittedly, detecting this is a bit more involved for the compiler, but
not at all restrictive for the user.
Dec 12 2008
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Zoran,


 I don't think it is restrictive if the compiler prevented a situation
 that would otherwise lead to a run-time error anyway, or worse, weird
 and confusing run-time behavior. In my case, if the compiler couldn't
 SAFELY handle a reference to the argument n outside the enclosing
 function, then, IMO, RETURNING it (but not otherwise using it) should
 be flagged a compilation error. Admittedly, detecting this is a bit
 more involved for the compiler, but not at all restrictive for the
 user.
 


D2.0 handles it all correctly. D1.0 follows the "hear is a gun, there is 
your foot" mentality with regards to this. In general D is *not* a safe
language 
and that is by intent.
Dec 12 2008
parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
BCS Wrote:


 Reply to Zoran,
 

 I don't think it is restrictive if the compiler prevented a situation
 that would otherwise lead to a run-time error anyway, or worse, weird
 and confusing run-time behavior. In my case, if the compiler couldn't
 SAFELY handle a reference to the argument n outside the enclosing
 function, then, IMO, RETURNING it (but not otherwise using it) should
 be flagged a compilation error. Admittedly, detecting this is a bit
 more involved for the compiler, but not at all restrictive for the
 user.
 


 
 D2.0 handles it all correctly. D1.0 follows the "hear is a gun, there is 
 your foot" mentality with regards to this. In general D is *not* a safe
language 
 and that is by intent.
 
 


Oh... I've got the wrong impression from the papers about D. (But then, why
would someone design an *unsafe* language *by intention*??? For that, we've got
C and C++, don't we?)

Anyway, I've been looking for a modern and *safe* language, but without the
overkill of a Java VM or .NET runtime. My hope was with D, but you seem to be
convincing me otherwise...

Does the "D is unsafe by intention" relate to D2.0, too?
Dec 13 2008
next sibling parent reply "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
Zoran Isailovski <dmd.zoc spamgourmet.com> wrote:


 Oh... I've got the wrong impression from the papers about D. (But then,  
 why would someone design an *unsafe* language *by intention*??? For  
 that, we've got C and C++, don't we?)


Because we want D  to be the new C/C++? :p

D is unsafe in that it lets you shoot yourself in the foot with a limited
amount of hassle. It has pointer arithmetics, manual memory management if
you want that, etc. It is however not unsafe in the same way as C/C++
(here's a boot with a gun attached to it, to use it safely, remove the
gun)

Also, there is SafeD, which is not yet implemented, but it's coming.
(http://www.digitalmars.com/d/2.0/safed.html)


 Anyway, I've been looking for a modern and *safe* language, but without  
 the overkill of a Java VM or .NET runtime. My hope was with D, but you  
 seem to be convincing me otherwise...


It may or may not be. As mentioned above, it's still possible to ferk up
with D, but it's a lot harder than with C or C++.


 Does the "D is unsafe by intention" relate to D2.0, too?


D2 still has pointers and optional manual memory management, so yes.
D2 has fixed a lot of the unsafe things from D1, so no.

Clear enough? :p

-- 
Simen
Dec 13 2008
parent Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Simen Kjaeraas Wrote:


 Zoran Isailovski <dmd.zoc spamgourmet.com> wrote:
 

 Oh... I've got the wrong impression from the papers about D. (But then,  
 why would someone design an *unsafe* language *by intention*??? For  
 that, we've got C and C++, don't we?)


 
 Because we want D  to be the new C/C++? :p
 
 D is unsafe in that it lets you shoot yourself in the foot with a limited
 amount of hassle. It has pointer arithmetics, manual memory management if
 you want that, etc. It is however not unsafe in the same way as C/C++
 (here's a boot with a gun attached to it, to use it safely, remove the
 gun)
 
 Also, there is SafeD, which is not yet implemented, but it's coming.
 (http://www.digitalmars.com/d/2.0/safed.html)
 

 Anyway, I've been looking for a modern and *safe* language, but without  
 the overkill of a Java VM or .NET runtime. My hope was with D, but you  
 seem to be convincing me otherwise...


 
 It may or may not be. As mentioned above, it's still possible to ferk up
 with D, but it's a lot harder than with C or C++.
 

 Does the "D is unsafe by intention" relate to D2.0, too?


 
 D2 still has pointers and optional manual memory management, so yes.
 D2 has fixed a lot of the unsafe things from D1, so no.
 
 Clear enough? :p
 
 -- 
 Simen


Yepp! Thanks! :))
Dec 13 2008
prev sibling next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Zoran Isailovski wrote:

 Oh... I've got the wrong impression from the papers about D. (But then, why
would someone design an *unsafe* language *by intention*??? For that, we've got
C and C++, don't we?)
 
 Anyway, I've been looking for a modern and *safe* language, but without the
overkill of a Java VM or .NET runtime. My hope was with D, but you seem to be
convincing me otherwise...
 
 Does the "D is unsafe by intention" relate to D2.0, too?


D tries to make it easy to do the safe thing. It's a systems language, 
so it has to allow you to do unsafe things without too much trouble -- 
but usually with some not-too-pretty syntax to indicate that you're 
doing something unsafe.

In this case, D1 fails. D2 works, though at the cost of additional, 
often unnecessary, heap allocation. Since D is a systems language, this 
is not good and is due to change soon. At least, I think Walter said he 
plans to implement scope delegates in D2.
Dec 13 2008
next sibling parent "Jarrett Billingsley" <jarrett.billingsley gmail.com> writes:
On Sat, Dec 13, 2008 at 9:09 AM, Christopher Wright <dhasenan gmail.com> wrote:

 D tries to make it easy to do the safe thing. It's a systems language, so it
 has to allow you to do unsafe things without too much trouble -- but usually
 with some not-too-pretty syntax to indicate that you're doing something
 unsafe.

 In this case, D1 fails. D2 works, though at the cost of additional, often
 unnecessary, heap allocation. Since D is a systems language, this is not
 good and is due to change soon. At least, I think Walter said he plans to
 implement scope delegates in D2.


He already has, as of the last update.
Dec 13 2008
prev sibling parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Christopher Wright Wrote:


 Zoran Isailovski wrote:

 Oh... I've got the wrong impression from the papers about D. (But then, why
would someone design an *unsafe* language *by intention*??? For that, we've got
C and C++, don't we?)
 
 Anyway, I've been looking for a modern and *safe* language, but without the
overkill of a Java VM or .NET runtime. My hope was with D, but you seem to be
convincing me otherwise...
 
 Does the "D is unsafe by intention" relate to D2.0, too?


 
 D tries to make it easy to do the safe thing. It's a systems language, 
 so it has to allow you to do unsafe things without too much trouble -- 
 but usually with some not-too-pretty syntax to indicate that you're 
 doing something unsafe.


As for the safe/unsafe and system language matter:

I like the approach of Ada, Oberon, and (I think) Modula-3 - all of which have
successfully been used for system programming - in that they allow unsafe
constructs ONLY in modules specifically marked as "unsafe".

I think, providing unsafe features through clumsy syntax makes things worse,
because it reduces readability where readability is most wanted - in sensible,
unsafe places (think "code reviews").

I'd rather stop here, because this is probably not the place to discuss these
things. But perhaps you can direct me to a discussion group or sth.?

Thx for taking the time!


 In this case, D1 fails. D2 works, though at the cost of additional, 
 often unnecessary, heap allocation. Since D is a systems language, this 
 is not good and is due to change soon. At least, I think Walter said he 
 plans to implement scope delegates in D2.
Dec 13 2008
parent BCS <ao pathlink.com> writes:
Reply to Zoran,


 Christopher Wright Wrote:
 
 I'd rather stop here, because this is probably not the place to
 discuss these things. But perhaps you can direct me to a discussion
 group or sth.?
 


Don't worry, This NG has a long standing tradition of epic rambling OT threads 
("[~ot] why is programming so fun?" went on for several days on religion 
of all things)
Dec 13 2008
prev sibling parent BCS <ao pathlink.com> writes:
Reply to Zoran,


 BCS Wrote:
 

 Reply to Zoran,
 

 I don't think it is restrictive if the compiler prevented a
 situation that would otherwise lead to a run-time error anyway, or
 worse, weird and confusing run-time behavior. In my case, if the
 compiler couldn't SAFELY handle a reference to the argument n
 outside the enclosing function, then, IMO, RETURNING it (but not
 otherwise using it) should be flagged a compilation error.
 Admittedly, detecting this is a bit more involved for the compiler,
 but not at all restrictive for the user.
 


 D2.0 handles it all correctly. D1.0 follows the "hear is a gun, there
 is your foot" mentality with regards to this. In general D is *not* a
 safe language and that is by intent.
 


 Oh... I've got the wrong impression from the papers about D. (But
 then, why would someone design an *unsafe* language *by intention*???
 For that, we've got C and C++, don't we?)
 


In that (and only that) regards, D is closer to Ada than C++

http://adrianhoe.com/adrianhoe/2006/08/04/shooting-yourself-in-the-foot-a-humorous-approach-in-comparing-ada-to-other-programming-languages/

you can do it, it's just a bit had to do it on accident.


 
 Does the "D is unsafe by intention" relate to D2.0, too?


D is intentionally not safe (vs. un-safe where the compiler requiter you 
to install a foot shooter mechine and does the shooting for you)
Dec 13 2008
prev sibling parent reply Sergey Gromov <snake.scaly gmail.com> writes:
Fri, 12 Dec 2008 15:24:39 -0500, Zoran Isailovski wrote:


 Denis Koroskin Wrote:
 

 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:






 experimenting

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:






 As a rule of thumb, you shoudn't don't return local delegate from a  
 function in D1, but you may safely pass them down the call stack.


 
 I don't think it is restrictive if the compiler prevented a situation
 that would otherwise lead to a run-time error anyway, or worse, weird
 and confusing run-time behavior. In my case, if the compiler couldn't
 SAFELY handle a reference to the argument n outside the enclosing
 function, then, IMO, RETURNING it (but not otherwise using it) should
 be flagged a compilation error. Admittedly, detecting this is a bit
 more involved for the compiler, but not at all restrictive for the
 user.


There are situations where proving that n escapes the function is hard
and requires extensive flow analysis.  It may be impossible to prove
when one module of a large program is compiled separately.  And it may
be impossible to prove for library code where user code is unknown by
definition.  Therefore the decision is left to the programmer.  This is
indeed unsafe, and D2 tries to solve this by introducing full,
heap-allocated closures.  This in turn brings performance problems in
situations where stack closures would be sufficient.  Now significant
effort is made towards allowing stack closures in D2 where it would be
safe.

That said, the full closure feature won't make its way into D1.  D1
closures will stay as they are.
Dec 12 2008
parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Sergey Gromov Wrote:


 Fri, 12 Dec 2008 15:24:39 -0500, Zoran Isailovski wrote:
 

 Denis Koroskin Wrote:
 

 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:






 experimenting

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:






 As a rule of thumb, you shoudn't don't return local delegate from a  
 function in D1, but you may safely pass them down the call stack.


 
 I don't think it is restrictive if the compiler prevented a situation
 that would otherwise lead to a run-time error anyway, or worse, weird
 and confusing run-time behavior. In my case, if the compiler couldn't
 SAFELY handle a reference to the argument n outside the enclosing
 function, then, IMO, RETURNING it (but not otherwise using it) should
 be flagged a compilation error. Admittedly, detecting this is a bit
 more involved for the compiler, but not at all restrictive for the
 user.


 
 There are situations where proving that n escapes the function is hard
 and requires extensive flow analysis.  It may be impossible to prove
 when one module of a large program is compiled separately.  And it may
 be impossible to prove for library code where user code is unknown by
 definition. Therefore the decision is left to the programmer.  This is
 indeed unsafe, and D2 tries to solve this by introducing full,
 heap-allocated closures.  This in turn brings performance problems in
 situations where stack closures would be sufficient.  Now significant
 effort is made towards allowing stack closures in D2 where it would be
 safe.


Sergey,  thanks for taking the time to respond.

I think you are talking about allowing all usages that may be potentially safe.
I was talking about disallowing all usages  that may be potentially unsafe.
(This was indeed one of the main differences in "philosophy" between languages
in the C family, and languages in the ALGOL family. I truly hope D will be
coming closer to the latter.)

Anyway, following the latter pattern, you don't need global analysis. You can
determine if n is on the stack (it is - it's an argument), you can determine if
it's referenced from within the closure (it is), and you can determine if the
closure is being returned (it is). The compiler should IMO then generate an
error (or warning) about the return statement, perhaps stating something like
"cannot return a delegate that refers to variables in local scope", or
something like that.


 That said, the full closure feature won't make its way into D1.  D1
 closures will stay as they are.


I see. And I'd love to switch to D2, but I just can't get DFL working with it
(on Win32).
Dec 13 2008
parent Sergey Gromov <snake.scaly gmail.com> writes:
Sat, 13 Dec 2008 06:59:51 -0500, Zoran Isailovski wrote:


 Anyway, following the latter pattern, you don't need global analysis.
 You can determine if n is on the stack (it is - it's an argument),
 you can determine if it's referenced from within the closure (it is),
 and you can determine if the closure is being returned (it is). The
 compiler should IMO then generate an error (or warning) about the
 return statement, perhaps stating something like "cannot return a
 delegate that refers to variables in local scope", or something like
 that.


There was a long discussion on this topic.  See
http://www.digitalmars.com/d/archives/digitalmars/D/Escape_analysis_78791.html
for an insight.  To summarize, it's almost always potentially unsafe to
use stack closures.  For instance, if you pass closure to a function,
that function can save that closure to a global variable, so the closure
outlives the container function without being explicitly returned.  If
compiler forbids any potentially unsafe use of closures, it would make
them next to useless and also would break lots of valid code, for
instance a significant part of Tango library.
Dec 13 2008
prev sibling next sibling parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Zoran Isailovski wrote:



and delegates, I was stroke by a phenomenon I cannot explain. Here's the code:
 


[snip]

 
 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }
 


[snip]

 
 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
  


[snip]

 
 What goes wrong???


Dv1.x doesn't automatically allocate closures for escaping delegates, so 
the storage used for parameter "n" is getting overwritten when another 
function is called.
Try the latest v2.x compiler, or explicitly copy the parameter to the heap.
Dec 12 2008
parent Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Frits van Bommel Wrote:


 Zoran Isailovski wrote:



and delegates, I was stroke by a phenomenon I cannot explain. Here's the code:
 


 [snip]

 
 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }
 


 [snip]

 
 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
  


 [snip]

 
 What goes wrong???


 
 Dv1.x doesn't automatically allocate closures for escaping delegates, so 
 the storage used for parameter "n" is getting overwritten when another 
 function is called.
 Try the latest v2.x compiler, or explicitly copy the parameter to the heap.


Thx, Frits, but I'm stll confused. See my response to Denis.
Dec 12 2008
prev sibling next sibling parent =?windows-1252?Q?=22J=E9r=F4me_M=2E_Berger=22?= <jeberger free.fr> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Zoran Isailovski wrote:



and delegates, I was stroke by a phenomenon I cannot explain. Here's the code:
 
 module closures01;
 
 import std.stdio;
 
 alias int delegate(int arg) Handler;
 
 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }
 
 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }
 
 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }
 
 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }
 
 public void run()
 {
 	test1();
 	test2();
 }
 
 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
  
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12
 
 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698
 
 * **************************************** */
 
 What goes wrong???


	In both cases, when your delegate gets called, the corresponding
stack frame is invalid. However, in test1, the stack data is still
intact when the delegate is called, whereas in test2 it has been
overwritten by the calls to mulBy(4), incBy(2) and writefln.

		Jerome
- --
mailto:jeberger free.fr
http://jeberger.free.fr
Jabber: jeberger jabber.fr
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAklC29gACgkQd0kWM4JG3k/w9ACdFsvzgE98G+HBuhQRhwig0ZTg
X+gAn1HVbECBvkiqrUMju3btW2fBGjtx
=6ae5
-----END PGP SIGNATURE-----
Dec 12 2008
prev sibling next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Zoran Isailovski:

 I like the approach of Ada, Oberon, and (I think) Modula-3 - all of which have
successfully been used for system programming - in that they allow unsafe
constructs ONLY in modules specifically marked as "unsafe".<


This is already becoming true in the last version of DMD. But I don't like this
solution. See the main D newsgroup, I'm about to post a note there.



 I think, providing unsafe features through clumsy syntax makes things worse,
because it reduces readability where readability is most wanted - in sensible,
unsafe places (think "code reviews").<


I agree. But the situation isn't that bad.

Bye,
bearophile
Dec 13 2008
prev sibling parent Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Sergey Gromov Wrote:


 Sat, 13 Dec 2008 06:59:51 -0500, Zoran Isailovski wrote:
 

 Anyway, following the latter pattern, you don't need global analysis.
 You can determine if n is on the stack (it is - it's an argument),
 you can determine if it's referenced from within the closure (it is),
 and you can determine if the closure is being returned (it is). The
 compiler should IMO then generate an error (or warning) about the
 return statement, perhaps stating something like "cannot return a
 delegate that refers to variables in local scope", or something like
 that.


 
 There was a long discussion on this topic.  See
 http://www.digitalmars.com/d/archives/digitalmars/D/Escape_analysis_78791.html
 for an insight.  To summarize, it's almost always potentially unsafe to
 use stack closures.  For instance, if you pass closure to a function,
 that function can save that closure to a global variable, so the closure
 outlives the container function without being explicitly returned.  If
 compiler forbids any potentially unsafe use of closures, it would make
 them next to useless and also would break lots of valid code, for
 instance a significant part of Tango library.


Thanks for the hint.
Dec 16 2008