www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - confusing (buggy?) closure behaviour

reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Content-Type: text/plain

I'm an experienced C#, Java and Python programmer, and have employed closures
(and C# delegates) upon numerous occasions. While experimenting with D closures
and delegates, I was stroke by a phenomenon I cannot explain. Here's the code:

module closures01;

import std.stdio;

alias int delegate(int arg) Handler;

Handler incBy(int n)
{
	return delegate(int arg){ return arg + n; };
}

Handler mulBy(int n)
{
	return delegate(int arg){ return arg * n; };
}

void test1()
{
	writefln("\ntest1:\n----------------------------------------");
	int x = 10, y;
	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
}

void test2()
{
	writefln("\ntest2:\n----------------------------------------");
	int x = 10, y;
	Handler times3 = mulBy(3);
	Handler times4 = mulBy(4);
	Handler plus2 = incBy(2);
	y = times3(x); writefln("%d * 3 -> %d", x, y);
	y = times4(x); writefln("%d * 4 -> %d", x, y);
	y = plus2(x); writefln("%d + 2 -> %d", x, y);
}

public void run()
{
	test1();
	test2();
}

/* **************************************** *
 * Compiled with: Digital Mars D Compiler v1.030
 *
 * (Unexplainable) program output:
 
test1:
----------------------------------------
10 * 3 -> 30
10 * 4 -> 40
10 + 2 -> 12

test2:
----------------------------------------
10 * 3 -> 20
10 * 4 -> 42846880
10 + 2 -> 4284698

* **************************************** */

What goes wrong???
Dec 12 2008
next sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski  
<dmd.zoc spamgourmet.com> wrote:

 I'm an experienced C#, Java and Python programmer, and have employed  
 closures (and C# delegates) upon numerous occasions. While experimenting  
 with D closures and delegates, I was stroke by a phenomenon I cannot  
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???

I'd say that it works as expected and here is why. First of all, there are two types of closures: static and dynamic closures. Closures work by having a hidden pointer to function frame where all local variables are stored. When a static closure is created, all the function local variables are stored on stack. It has an advantage that no memory allocation takes place (fast). It has a disadvantage that once the delegate leaves the scope, it becomes invalid since variables were stored on stack and the stack is probably overwritten (unsafe). Dynamic closure allocates memory in a heap and all the local variables are placed there. It has a disadvantage that memory is allocated for dynamic closure (might be slow if dynamic closure are created often). It has an advantage that dynamic closure may leave the scope, i.e. you may save it and call whenever you want. D1 support static closures only! That's why your code doesn't work (in test1 stack is still valid, but in test2 stack gets overwritten) D2 has support for dynamic closures. Just try it - your sample works as is.
Dec 12 2008
next sibling parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Denis Koroskin Wrote:

 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski  
 <dmd.zoc spamgourmet.com> wrote:
 
 I'm an experienced C#, Java and Python programmer, and have employed  
 closures (and C# delegates) upon numerous occasions. While experimenting  
 with D closures and delegates, I was stroke by a phenomenon I cannot  
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???

I'd say that it works as expected and here is why. First of all, there are two types of closures: static and dynamic closures. Closures work by having a hidden pointer to function frame where all local variables are stored. When a static closure is created, all the function local variables are stored on stack. It has an advantage that no memory allocation takes place (fast). It has a disadvantage that once the delegate leaves the scope, it becomes invalid since variables were stored on stack and the stack is probably overwritten (unsafe). Dynamic closure allocates memory in a heap and all the local variables are placed there. It has a disadvantage that memory is allocated for dynamic closure (might be slow if dynamic closure are created often). It has an advantage that dynamic closure may leave the scope, i.e. you may save it and call whenever you want. D1 support static closures only! That's why your code doesn't work (in test1 stack is still valid, but in test2 stack gets overwritten) D2 has support for dynamic closures. Just try it - your sample works as is.

Thx, Denis, but I'm still confused. The stack thing was also my first thought. But when I tried to actually explain the dynamics that way, I came to the conclusion that then, test1() shouldn't have worked either. I assumed the following (schematic) process takes place: <code> mulBy(3)(x) => push 3; call mullBy; // upon entry into mulBy: Stack = [ >&mulBy, 3, ... ] // I assume, the callee cleans up the stack, so... // upon return from mulBy: Stack = [ &mulBy, 3, >... ]; cpu_register = &delegate => push x; call [cpu_register] // upon entry into delegate: Stack = [ >&delegate, x, ... ] </code> (Here, the stack / frame pointer is denoted by ">", and moves from right to left on push, left to right on pop) With that mechanism, when the delegate is entered, the memory where previously the number 3 was stored, should have been overwritten by x. But it obviously isn't ?!? How come?
Dec 12 2008
next sibling parent Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Denis Koroskin Wrote:

 On Fri, 12 Dec 2008 22:18:28 +0300, Zoran Isailovski  
 <dmd.zoc spamgourmet.com> wrote:
 
 Denis Koroskin Wrote:

 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:

 I'm an experienced C#, Java and Python programmer, and have employed
 closures (and C# delegates) upon numerous occasions. While  

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???

I'd say that it works as expected and here is why. First of all, there are two types of closures: static and dynamic closures. Closures work by having a hidden pointer to function frame where all local variables are stored. When a static closure is created, all the function local variables are stored on stack. It has an advantage that no memory allocation takes place (fast). It has a disadvantage that once the delegate leaves the scope, it becomes invalid since variables were stored on stack and the stack is probably overwritten (unsafe). Dynamic closure allocates memory in a heap and all the local variables are placed there. It has a disadvantage that memory is allocated for dynamic closure (might be slow if dynamic closure are created often). It has an advantage that dynamic closure may leave the scope, i.e. you may save it and call whenever you want. D1 support static closures only! That's why your code doesn't work (in test1 stack is still valid, but in test2 stack gets overwritten) D2 has support for dynamic closures. Just try it - your sample works as is.

Thx, Denis, but I'm still confused. The stack thing was also my first thought. But when I tried to actually explain the dynamics that way, I came to the conclusion that then, test1() shouldn't have worked either. I assumed the following (schematic) process takes place: <code> mulBy(3)(x) => push 3; call mullBy; // upon entry into mulBy: Stack = [ >&mulBy, 3, ... ] // I assume, the callee cleans up the stack, so... // upon return from mulBy: Stack = [ &mulBy, 3, >... ]; cpu_register = &delegate => push x; call [cpu_register] // upon entry into delegate: Stack = [ >&delegate, x, ... ] </code> (Here, the stack / frame pointer is denoted by ">", and moves from right to left on push, left to right on pop) With that mechanism, when the delegate is entered, the memory where previously the number 3 was stored, should have been overwritten by x. But it obviously isn't ?!? How come?

No, you are taking it slightly wrong. The delegate stores a raw /pointer to stack frame/ so it doesn't depend on current stack head (ESP).

And that's exactly what I was assuming, Denis. But function calls still use the stack to pass parameters and store the return address, don't they? So if the way stacks, the stack pointer (ESP) and the frame pointer (EBP) work hasn't dramatically changed since I last had business with them, then a "snapshot" of the situation just before returning from mulBy should look like this: lo --- stack memory --- hi stack: ..., retadr, 3, ... ESP: ^ rawptr to n: ^ "rawptr" is you raw pointer to the stack frame (the memory block on the stack visible inside mulBy, i.e., basically the value of EBP inside mulBy). Indeed, it is not bound to ESP or EBP. Upon return from mulBy, the situation should look like this lo --- stack memory --- hi stack: ..., retadr, 3, ... ESP: ^ rawptr to n: ^ rawptr has not changed. Then, upon call to the delegate, x and the return value are pushed onto the stack. This changes ESP *and* the contents of the stack memory, probably resulting into somthing like lo --- stack memory --- hi stack: ..., retadr, x, ... ESP: ^ rawptr: ^ rawptr still points to the same memory, but the memory has been changed. It now contains the value of x, not 3. This, at least, used to be how things worked at the machine level. Don't get me wrong. I'm aware that the above "theory" does not explain the behavior in my sample code. I'd just like to know why, that's all.
Dec 12 2008
prev sibling parent BCS <ao pathlink.com> writes:
Reply to Zoran,

 Thx, Denis, but I'm still confused. The stack thing was also my first
 thought. But when I tried to actually explain the dynamics that way, I
 came to the conclusion that then, test1() shouldn't have worked
 either.
 

what might be happening is that test1 is not using up enough stack space to overwrite the arguments.
Dec 12 2008
prev sibling parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Denis Koroskin Wrote:

 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski  
 <dmd.zoc spamgourmet.com> wrote:
 
 I'm an experienced C#, Java and Python programmer, and have employed  
 closures (and C# delegates) upon numerous occasions. While experimenting  
 with D closures and delegates, I was stroke by a phenomenon I cannot  
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???

I'd say that it works as expected and here is why. First of all, there are two types of closures: static and dynamic closures. Closures work by having a hidden pointer to function frame where all local variables are stored. When a static closure is created, all the function local variables are stored on stack. It has an advantage that no memory allocation takes place (fast). It has a disadvantage that once the delegate leaves the scope, it becomes invalid since variables were stored on stack and the stack is probably overwritten (unsafe). Dynamic closure allocates memory in a heap and all the local variables are placed there. It has a disadvantage that memory is allocated for dynamic closure (might be slow if dynamic closure are created often). It has an advantage that dynamic closure may leave the scope, i.e. you may save it and call whenever you want. D1 support static closures only! That's why your code doesn't work (in test1 stack is still valid, but in test2 stack gets overwritten) D2 has support for dynamic closures. Just try it - your sample works as is.

An addition: Given the complexness of the criteria when a closure works and when not, I would vote for a compiler error on inappropriate closures usage. (In Java, closures cannot handle mutable values on the stack, so it's an error for a method to return a closure that refers to a non-final argument.)
Dec 12 2008
parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Denis Koroskin Wrote:

 On Fri, 12 Dec 2008 22:28:01 +0300, Zoran Isailovski  
 <dmd.zoc spamgourmet.com> wrote:
 
 Denis Koroskin Wrote:

 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:

 I'm an experienced C#, Java and Python programmer, and have employed
 closures (and C# delegates) upon numerous occasions. While  

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???

I'd say that it works as expected and here is why. First of all, there are two types of closures: static and dynamic closures. Closures work by having a hidden pointer to function frame where all local variables are stored. When a static closure is created, all the function local variables are stored on stack. It has an advantage that no memory allocation takes place (fast). It has a disadvantage that once the delegate leaves the scope, it becomes invalid since variables were stored on stack and the stack is probably overwritten (unsafe). Dynamic closure allocates memory in a heap and all the local variables are placed there. It has a disadvantage that memory is allocated for dynamic closure (might be slow if dynamic closure are created often). It has an advantage that dynamic closure may leave the scope, i.e. you may save it and call whenever you want. D1 support static closures only! That's why your code doesn't work (in test1 stack is still valid, but in test2 stack gets overwritten) D2 has support for dynamic closures. Just try it - your sample works as is.

An addition: Given the complexness of the criteria when a closure works and when not, I would vote for a compiler error on inappropriate closures usage. (In Java, closures cannot handle mutable values on the stack, so it's an error for a method to return a closure that refers to a non-final argument.)

This would restrict their usage so badly that may make then next to useless. No worries, a dynamic closure is created automatically whenever a static one it might be unsafe in D2. As a rule of thumb, you shoudn't don't return local delegate from a function in D1, but you may safely pass them down the call stack.

I don't think it is restrictive if the compiler prevented a situation that would otherwise lead to a run-time error anyway, or worse, weird and confusing run-time behavior. In my case, if the compiler couldn't SAFELY handle a reference to the argument n outside the enclosing function, then, IMO, RETURNING it (but not otherwise using it) should be flagged a compilation error. Admittedly, detecting this is a bit more involved for the compiler, but not at all restrictive for the user.
Dec 12 2008
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Zoran,

 I don't think it is restrictive if the compiler prevented a situation
 that would otherwise lead to a run-time error anyway, or worse, weird
 and confusing run-time behavior. In my case, if the compiler couldn't
 SAFELY handle a reference to the argument n outside the enclosing
 function, then, IMO, RETURNING it (but not otherwise using it) should
 be flagged a compilation error. Admittedly, detecting this is a bit
 more involved for the compiler, but not at all restrictive for the
 user.
 

D2.0 handles it all correctly. D1.0 follows the "hear is a gun, there is your foot" mentality with regards to this. In general D is *not* a safe language and that is by intent.
Dec 12 2008
parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
BCS Wrote:

 Reply to Zoran,
 
 I don't think it is restrictive if the compiler prevented a situation
 that would otherwise lead to a run-time error anyway, or worse, weird
 and confusing run-time behavior. In my case, if the compiler couldn't
 SAFELY handle a reference to the argument n outside the enclosing
 function, then, IMO, RETURNING it (but not otherwise using it) should
 be flagged a compilation error. Admittedly, detecting this is a bit
 more involved for the compiler, but not at all restrictive for the
 user.
 

D2.0 handles it all correctly. D1.0 follows the "hear is a gun, there is your foot" mentality with regards to this. In general D is *not* a safe language and that is by intent.

Anyway, I've been looking for a modern and *safe* language, but without the overkill of a Java VM or .NET runtime. My hope was with D, but you seem to be convincing me otherwise... Does the "D is unsafe by intention" relate to D2.0, too?
Dec 13 2008
next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Zoran Isailovski wrote:
 Oh... I've got the wrong impression from the papers about D. (But then, why
would someone design an *unsafe* language *by intention*??? For that, we've got
C and C++, don't we?)
 
 Anyway, I've been looking for a modern and *safe* language, but without the
overkill of a Java VM or .NET runtime. My hope was with D, but you seem to be
convincing me otherwise...
 
 Does the "D is unsafe by intention" relate to D2.0, too?

D tries to make it easy to do the safe thing. It's a systems language, so it has to allow you to do unsafe things without too much trouble -- but usually with some not-too-pretty syntax to indicate that you're doing something unsafe. In this case, D1 fails. D2 works, though at the cost of additional, often unnecessary, heap allocation. Since D is a systems language, this is not good and is due to change soon. At least, I think Walter said he plans to implement scope delegates in D2.
Dec 13 2008
parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Christopher Wright Wrote:

 Zoran Isailovski wrote:
 Oh... I've got the wrong impression from the papers about D. (But then, why
would someone design an *unsafe* language *by intention*??? For that, we've got
C and C++, don't we?)
 
 Anyway, I've been looking for a modern and *safe* language, but without the
overkill of a Java VM or .NET runtime. My hope was with D, but you seem to be
convincing me otherwise...
 
 Does the "D is unsafe by intention" relate to D2.0, too?

D tries to make it easy to do the safe thing. It's a systems language, so it has to allow you to do unsafe things without too much trouble -- but usually with some not-too-pretty syntax to indicate that you're doing something unsafe.

As for the safe/unsafe and system language matter: I like the approach of Ada, Oberon, and (I think) Modula-3 - all of which have successfully been used for system programming - in that they allow unsafe constructs ONLY in modules specifically marked as "unsafe". I think, providing unsafe features through clumsy syntax makes things worse, because it reduces readability where readability is most wanted - in sensible, unsafe places (think "code reviews"). I'd rather stop here, because this is probably not the place to discuss these things. But perhaps you can direct me to a discussion group or sth.? Thx for taking the time!
 In this case, D1 fails. D2 works, though at the cost of additional, 
 often unnecessary, heap allocation. Since D is a systems language, this 
 is not good and is due to change soon. At least, I think Walter said he 
 plans to implement scope delegates in D2.

Dec 13 2008
parent BCS <ao pathlink.com> writes:
Reply to Zoran,

 Christopher Wright Wrote:
 
 I'd rather stop here, because this is probably not the place to
 discuss these things. But perhaps you can direct me to a discussion
 group or sth.?
 

Don't worry, This NG has a long standing tradition of epic rambling OT threads ("[~ot] why is programming so fun?" went on for several days on religion of all things)
Dec 13 2008
prev sibling next sibling parent Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Simen Kjaeraas Wrote:

 Zoran Isailovski <dmd.zoc spamgourmet.com> wrote:
 
 Oh... I've got the wrong impression from the papers about D. (But then,  
 why would someone design an *unsafe* language *by intention*??? For  
 that, we've got C and C++, don't we?)

Because we want D to be the new C/C++? :p D is unsafe in that it lets you shoot yourself in the foot with a limited amount of hassle. It has pointer arithmetics, manual memory management if you want that, etc. It is however not unsafe in the same way as C/C++ (here's a boot with a gun attached to it, to use it safely, remove the gun) Also, there is SafeD, which is not yet implemented, but it's coming. (http://www.digitalmars.com/d/2.0/safed.html)
 Anyway, I've been looking for a modern and *safe* language, but without  
 the overkill of a Java VM or .NET runtime. My hope was with D, but you  
 seem to be convincing me otherwise...

It may or may not be. As mentioned above, it's still possible to ferk up with D, but it's a lot harder than with C or C++.
 Does the "D is unsafe by intention" relate to D2.0, too?

D2 still has pointers and optional manual memory management, so yes. D2 has fixed a lot of the unsafe things from D1, so no. Clear enough? :p -- Simen

Yepp! Thanks! :))
Dec 13 2008
prev sibling parent BCS <ao pathlink.com> writes:
Reply to Zoran,

 BCS Wrote:
 
 Reply to Zoran,
 
 I don't think it is restrictive if the compiler prevented a
 situation that would otherwise lead to a run-time error anyway, or
 worse, weird and confusing run-time behavior. In my case, if the
 compiler couldn't SAFELY handle a reference to the argument n
 outside the enclosing function, then, IMO, RETURNING it (but not
 otherwise using it) should be flagged a compilation error.
 Admittedly, detecting this is a bit more involved for the compiler,
 but not at all restrictive for the user.
 

is your foot" mentality with regards to this. In general D is *not* a safe language and that is by intent.

then, why would someone design an *unsafe* language *by intention*??? For that, we've got C and C++, don't we?)

In that (and only that) regards, D is closer to Ada than C++ http://adrianhoe.com/adrianhoe/2006/08/04/shooting-yourself-in-the-foot-a-humorous-approach-in-comparing-ada-to-other-programming-languages/ you can do it, it's just a bit had to do it on accident.
 
 Does the "D is unsafe by intention" relate to D2.0, too?

D is intentionally not safe (vs. un-safe where the compiler requiter you to install a foot shooter mechine and does the shooting for you)
 

Dec 13 2008
prev sibling next sibling parent reply Sergey Gromov <snake.scaly gmail.com> writes:
Fri, 12 Dec 2008 15:24:39 -0500, Zoran Isailovski wrote:

 Denis Koroskin Wrote:
 
 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:

 I'm an experienced C#, Java and Python programmer, and have employed
 closures (and C# delegates) upon numerous occasions. While  

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:



function in D1, but you may safely pass them down the call stack.

I don't think it is restrictive if the compiler prevented a situation that would otherwise lead to a run-time error anyway, or worse, weird and confusing run-time behavior. In my case, if the compiler couldn't SAFELY handle a reference to the argument n outside the enclosing function, then, IMO, RETURNING it (but not otherwise using it) should be flagged a compilation error. Admittedly, detecting this is a bit more involved for the compiler, but not at all restrictive for the user.

There are situations where proving that n escapes the function is hard and requires extensive flow analysis. It may be impossible to prove when one module of a large program is compiled separately. And it may be impossible to prove for library code where user code is unknown by definition. Therefore the decision is left to the programmer. This is indeed unsafe, and D2 tries to solve this by introducing full, heap-allocated closures. This in turn brings performance problems in situations where stack closures would be sufficient. Now significant effort is made towards allowing stack closures in D2 where it would be safe. That said, the full closure feature won't make its way into D1. D1 closures will stay as they are.
Dec 12 2008
parent reply Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Sergey Gromov Wrote:

 Fri, 12 Dec 2008 15:24:39 -0500, Zoran Isailovski wrote:
 
 Denis Koroskin Wrote:
 
 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:

 I'm an experienced C#, Java and Python programmer, and have employed
 closures (and C# delegates) upon numerous occasions. While  

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:



function in D1, but you may safely pass them down the call stack.

I don't think it is restrictive if the compiler prevented a situation that would otherwise lead to a run-time error anyway, or worse, weird and confusing run-time behavior. In my case, if the compiler couldn't SAFELY handle a reference to the argument n outside the enclosing function, then, IMO, RETURNING it (but not otherwise using it) should be flagged a compilation error. Admittedly, detecting this is a bit more involved for the compiler, but not at all restrictive for the user.

There are situations where proving that n escapes the function is hard and requires extensive flow analysis. It may be impossible to prove when one module of a large program is compiled separately. And it may be impossible to prove for library code where user code is unknown by definition. Therefore the decision is left to the programmer. This is indeed unsafe, and D2 tries to solve this by introducing full, heap-allocated closures. This in turn brings performance problems in situations where stack closures would be sufficient. Now significant effort is made towards allowing stack closures in D2 where it would be safe.

Sergey, thanks for taking the time to respond. I think you are talking about allowing all usages that may be potentially safe. I was talking about disallowing all usages that may be potentially unsafe. (This was indeed one of the main differences in "philosophy" between languages in the C family, and languages in the ALGOL family. I truly hope D will be coming closer to the latter.) Anyway, following the latter pattern, you don't need global analysis. You can determine if n is on the stack (it is - it's an argument), you can determine if it's referenced from within the closure (it is), and you can determine if the closure is being returned (it is). The compiler should IMO then generate an error (or warning) about the return statement, perhaps stating something like "cannot return a delegate that refers to variables in local scope", or something like that.
 That said, the full closure feature won't make its way into D1.  D1
 closures will stay as they are.

I see. And I'd love to switch to D2, but I just can't get DFL working with it (on Win32).
Dec 13 2008
parent Sergey Gromov <snake.scaly gmail.com> writes:
Sat, 13 Dec 2008 06:59:51 -0500, Zoran Isailovski wrote:

 Anyway, following the latter pattern, you don't need global analysis.
 You can determine if n is on the stack (it is - it's an argument),
 you can determine if it's referenced from within the closure (it is),
 and you can determine if the closure is being returned (it is). The
 compiler should IMO then generate an error (or warning) about the
 return statement, perhaps stating something like "cannot return a
 delegate that refers to variables in local scope", or something like
 that.

There was a long discussion on this topic. See http://www.digitalmars.com/d/archives/digitalmars/D/Escape_analysis_78791.html for an insight. To summarize, it's almost always potentially unsafe to use stack closures. For instance, if you pass closure to a function, that function can save that closure to a global variable, so the closure outlives the container function without being explicitly returned. If compiler forbids any potentially unsafe use of closures, it would make them next to useless and also would break lots of valid code, for instance a significant part of Tango library.
Dec 13 2008
prev sibling next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
Zoran Isailovski <dmd.zoc spamgourmet.com> wrote:

 Oh... I've got the wrong impression from the papers about D. (But then,  
 why would someone design an *unsafe* language *by intention*??? For  
 that, we've got C and C++, don't we?)

Because we want D to be the new C/C++? :p D is unsafe in that it lets you shoot yourself in the foot with a limited amount of hassle. It has pointer arithmetics, manual memory management if you want that, etc. It is however not unsafe in the same way as C/C++ (here's a boot with a gun attached to it, to use it safely, remove the gun) Also, there is SafeD, which is not yet implemented, but it's coming. (http://www.digitalmars.com/d/2.0/safed.html)
 Anyway, I've been looking for a modern and *safe* language, but without  
 the overkill of a Java VM or .NET runtime. My hope was with D, but you  
 seem to be convincing me otherwise...

It may or may not be. As mentioned above, it's still possible to ferk up with D, but it's a lot harder than with C or C++.
 Does the "D is unsafe by intention" relate to D2.0, too?

D2 still has pointers and optional manual memory management, so yes. D2 has fixed a lot of the unsafe things from D1, so no. Clear enough? :p -- Simen
Dec 13 2008
prev sibling parent "Jarrett Billingsley" <jarrett.billingsley gmail.com> writes:
On Sat, Dec 13, 2008 at 9:09 AM, Christopher Wright <dhasenan gmail.com> wrote:
 D tries to make it easy to do the safe thing. It's a systems language, so it
 has to allow you to do unsafe things without too much trouble -- but usually
 with some not-too-pretty syntax to indicate that you're doing something
 unsafe.

 In this case, D1 fails. D2 works, though at the cost of additional, often
 unnecessary, heap allocation. Since D is a systems language, this is not
 good and is due to change soon. At least, I think Walter said he plans to
 implement scope delegates in D2.

He already has, as of the last update.
Dec 13 2008
prev sibling next sibling parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Zoran Isailovski wrote:
 I'm an experienced C#, Java and Python programmer, and have employed closures
(and C# delegates) upon numerous occasions. While experimenting with D closures
and delegates, I was stroke by a phenomenon I cannot explain. Here's the code:
 

 
 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }
 

 
 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
  

 
 What goes wrong???

Dv1.x doesn't automatically allocate closures for escaping delegates, so the storage used for parameter "n" is getting overwritten when another function is called. Try the latest v2.x compiler, or explicitly copy the parameter to the heap.
Dec 12 2008
parent Zoran Isailovski <dmd.zoc spamgourmet.com> writes:
Frits van Bommel Wrote:

 Zoran Isailovski wrote:
 I'm an experienced C#, Java and Python programmer, and have employed closures
(and C# delegates) upon numerous occasions. While experimenting with D closures
and delegates, I was stroke by a phenomenon I cannot explain. Here's the code:
 

 
 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }
 

 
 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
  

 
 What goes wrong???

Dv1.x doesn't automatically allocate closures for escaping delegates, so the storage used for parameter "n" is getting overwritten when another function is called. Try the latest v2.x compiler, or explicitly copy the parameter to the heap.

Thx, Frits, but I'm stll confused. See my response to Denis.
Dec 12 2008
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 12 Dec 2008 22:18:28 +0300, Zoran Isailovski  
<dmd.zoc spamgourmet.com> wrote:

 Denis Koroskin Wrote:

 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:

 I'm an experienced C#, Java and Python programmer, and have employed
 closures (and C# delegates) upon numerous occasions. While  

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???

I'd say that it works as expected and here is why. First of all, there are two types of closures: static and dynamic closures. Closures work by having a hidden pointer to function frame where all local variables are stored. When a static closure is created, all the function local variables are stored on stack. It has an advantage that no memory allocation takes place (fast). It has a disadvantage that once the delegate leaves the scope, it becomes invalid since variables were stored on stack and the stack is probably overwritten (unsafe). Dynamic closure allocates memory in a heap and all the local variables are placed there. It has a disadvantage that memory is allocated for dynamic closure (might be slow if dynamic closure are created often). It has an advantage that dynamic closure may leave the scope, i.e. you may save it and call whenever you want. D1 support static closures only! That's why your code doesn't work (in test1 stack is still valid, but in test2 stack gets overwritten) D2 has support for dynamic closures. Just try it - your sample works as is.

Thx, Denis, but I'm still confused. The stack thing was also my first thought. But when I tried to actually explain the dynamics that way, I came to the conclusion that then, test1() shouldn't have worked either. I assumed the following (schematic) process takes place: <code> mulBy(3)(x) => push 3; call mullBy; // upon entry into mulBy: Stack = [ >&mulBy, 3, ... ] // I assume, the callee cleans up the stack, so... // upon return from mulBy: Stack = [ &mulBy, 3, >... ]; cpu_register = &delegate => push x; call [cpu_register] // upon entry into delegate: Stack = [ >&delegate, x, ... ] </code> (Here, the stack / frame pointer is denoted by ">", and moves from right to left on push, left to right on pop) With that mechanism, when the delegate is entered, the memory where previously the number 3 was stored, should have been overwritten by x. But it obviously isn't ?!? How come?

No, you are taking it slightly wrong. The delegate stores a raw /pointer to stack frame/ so it doesn't depend on current stack head (ESP).
Dec 12 2008
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 12 Dec 2008 22:28:01 +0300, Zoran Isailovski  
<dmd.zoc spamgourmet.com> wrote:

 Denis Koroskin Wrote:

 On Fri, 12 Dec 2008 19:32:03 +0300, Zoran Isailovski
 <dmd.zoc spamgourmet.com> wrote:

 I'm an experienced C#, Java and Python programmer, and have employed
 closures (and C# delegates) upon numerous occasions. While  

 with D closures and delegates, I was stroke by a phenomenon I cannot
 explain. Here's the code:

 module closures01;

 import std.stdio;

 alias int delegate(int arg) Handler;

 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }

 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }

 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }

 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }

 public void run()
 {
 	test1();
 	test2();
 }

 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12

 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698

 * **************************************** */

 What goes wrong???

I'd say that it works as expected and here is why. First of all, there are two types of closures: static and dynamic closures. Closures work by having a hidden pointer to function frame where all local variables are stored. When a static closure is created, all the function local variables are stored on stack. It has an advantage that no memory allocation takes place (fast). It has a disadvantage that once the delegate leaves the scope, it becomes invalid since variables were stored on stack and the stack is probably overwritten (unsafe). Dynamic closure allocates memory in a heap and all the local variables are placed there. It has a disadvantage that memory is allocated for dynamic closure (might be slow if dynamic closure are created often). It has an advantage that dynamic closure may leave the scope, i.e. you may save it and call whenever you want. D1 support static closures only! That's why your code doesn't work (in test1 stack is still valid, but in test2 stack gets overwritten) D2 has support for dynamic closures. Just try it - your sample works as is.

An addition: Given the complexness of the criteria when a closure works and when not, I would vote for a compiler error on inappropriate closures usage. (In Java, closures cannot handle mutable values on the stack, so it's an error for a method to return a closure that refers to a non-final argument.)

This would restrict their usage so badly that may make then next to useless. No worries, a dynamic closure is created automatically whenever a static one it might be unsafe in D2. As a rule of thumb, you shoudn't don't return local delegate from a function in D1, but you may safely pass them down the call stack.
Dec 12 2008
prev sibling parent =?windows-1252?Q?=22J=E9r=F4me_M=2E_Berger=22?= <jeberger free.fr> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Zoran Isailovski wrote:
 I'm an experienced C#, Java and Python programmer, and have employed closures
(and C# delegates) upon numerous occasions. While experimenting with D closures
and delegates, I was stroke by a phenomenon I cannot explain. Here's the code:
 
 module closures01;
 
 import std.stdio;
 
 alias int delegate(int arg) Handler;
 
 Handler incBy(int n)
 {
 	return delegate(int arg){ return arg + n; };
 }
 
 Handler mulBy(int n)
 {
 	return delegate(int arg){ return arg * n; };
 }
 
 void test1()
 {
 	writefln("\ntest1:\n----------------------------------------");
 	int x = 10, y;
 	y = mulBy(3)(x); writefln("%d * 3 -> %d", x, y);
 	y = mulBy(4)(x); writefln("%d * 4 -> %d", x, y);
 	y = incBy(2)(x); writefln("%d + 2 -> %d", x, y);
 }
 
 void test2()
 {
 	writefln("\ntest2:\n----------------------------------------");
 	int x = 10, y;
 	Handler times3 = mulBy(3);
 	Handler times4 = mulBy(4);
 	Handler plus2 = incBy(2);
 	y = times3(x); writefln("%d * 3 -> %d", x, y);
 	y = times4(x); writefln("%d * 4 -> %d", x, y);
 	y = plus2(x); writefln("%d + 2 -> %d", x, y);
 }
 
 public void run()
 {
 	test1();
 	test2();
 }
 
 /* **************************************** *
  * Compiled with: Digital Mars D Compiler v1.030
  *
  * (Unexplainable) program output:
  
 test1:
 ----------------------------------------
 10 * 3 -> 30
 10 * 4 -> 40
 10 + 2 -> 12
 
 test2:
 ----------------------------------------
 10 * 3 -> 20
 10 * 4 -> 42846880
 10 + 2 -> 4284698
 
 * **************************************** */
 
 What goes wrong???

In both cases, when your delegate gets called, the corresponding stack frame is invalid. However, in test1, the stack data is still intact when the delegate is called, whereas in test2 it has been overwritten by the calls to mulBy(4), incBy(2) and writefln. Jerome - -- mailto:jeberger free.fr http://jeberger.free.fr Jabber: jeberger jabber.fr -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklC29gACgkQd0kWM4JG3k/w9ACdFsvzgE98G+HBuhQRhwig0ZTg X+gAn1HVbECBvkiqrUMju3btW2fBGjtx =6ae5 -----END PGP SIGNATURE-----
Dec 12 2008