www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Trouble with anon delegates.

reply Pragma <ericanderton yahoo.removeme.com> writes:
All,
   I ran into a small problem regarding the use of anon delegates, and what
their context (.ptr) is set to.  If anyone 
can share any insight as to if this is a bug or not, I would appreciate it.

In short: I'm attempting to create delegates on the fly and use them elsewhere.
 After many strange errors, I narrowed 
it down to the following case.  Note the assertions at the end:

abstract class Foobar{
     Fn getFn();
}
alias void delegate() Fn;

auto foo = new class Foobar{
     int x;
     Fn getFn(){
	return { x = x };
     }
};
auto fn = foo.getFn();
assert((&foo.getFn).ptr is foo); // passes
assert(fn.ptr is foo); // fails (ptr does *not* point to the object)


So by this, anon delegates are not the same as actual object delegates.  Is
this a bug, or is this intentional?

"When comparing with nested functions, the function form is analogous to static
or non-nested functions, and the 
delegate form is analogous to non-static nested functions. In other words, a
delegate literal can access stack variables 
in its enclosing function, a function literal cannot."

As you can see, the docs are as clear as mud.  I suppose a "non-static nested
function" wouldn't know about the 'this' 
pointer, nor care - but then why encumber it with the 'delegate' keyword?


-- 
- EricAnderton at yahoo
Jan 16 2007
next sibling parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Pragma wrote:
 All,
   I ran into a small problem regarding the use of anon delegates, and 
 what their context (.ptr) is set to.  If anyone can share any insight as 
 to if this is a bug or not, I would appreciate it.
 
 In short: I'm attempting to create delegates on the fly and use them 
 elsewhere.  After many strange errors, I narrowed it down to the 
 following case.  Note the assertions at the end:
 
 abstract class Foobar{
     Fn getFn();
 }
 alias void delegate() Fn;
 
 auto foo = new class Foobar{
     int x;
     Fn getFn(){
     return { x = x };
     }
 };

That class has a member function returning a delegate. That returned delegate has the stack frame of said member function as its context pointer and should thus not be called after it returns. Basically, the return value is useless as it can't safely be called.
 auto fn = foo.getFn();

'fn' now contains said delegate, with the context pointer set to an invalidated stackframe.
 assert((&foo.getFn).ptr is foo); // passes

(&foo.getFn) is a delegate with context foo and function pointer Foo.getFn.
 assert(fn.ptr is foo); // fails (ptr does *not* point to the object)

fn.ptr is the invalid stack frame of the call to foo.getFn(), not foo itself.
 So by this, anon delegates are not the same as actual object delegates.  
 Is this a bug, or is this intentional?

Intentional.
 "When comparing with nested functions, the function form is analogous to 
 static or non-nested functions, and the delegate form is analogous to 
 non-static nested functions. In other words, a delegate literal can 
 access stack variables in its enclosing function, a function literal 
 cannot."
 
 As you can see, the docs are as clear as mud.  I suppose a "non-static 
 nested function" wouldn't know about the 'this' pointer, nor care - but 
 then why encumber it with the 'delegate' keyword?

It *does* know about 'this'. However, its context pointer is a stack frame (that happens to store 'this' in your case), not 'this' itself. There are simply several 'types' of delegates. One is a delegate that has a class or struct (or perhaps even a union) instance as its context pointer. These are created by (&instance.memberFunction). Another 'type' has a stack frame as its context pointer. These are created by (&nestedFunction) or { foo(); }. They should not be called after the function call (of the function directly containing their definition) in which they were created returns. Spec reference: http://www.digitalmars.com/d/function.html#closures
Jan 16 2007
prev sibling parent reply BCS <ao pathlink.com> writes:
Reply to Pragma,

 All,
 I ran into a small problem regarding the use of anon delegates, and
 what their context (.ptr) is set to.  If anyone
 can share any insight as to if this is a bug or not, I would
 appreciate it.
 In short: I'm attempting to create delegates on the fly and use them
 elsewhere.  After many strange errors, I narrowed it down to the
 following case.  Note the assertions at the end:
 
 abstract class Foobar{
 Fn getFn();
 }
 alias void delegate() Fn;
 
 auto foo = new class Foobar{
 int x;
 Fn getFn(){
 return { x = x };
 }
 };
 auto fn = foo.getFn();
 assert((&foo.getFn).ptr is foo); // passes
 assert(fn.ptr is foo); // fails (ptr does *not* point to the object)
 So by this, anon delegates are not the same as actual object
 delegates.  Is this a bug, or is this intentional?
 
 "When comparing with nested functions, the function form is analogous
 to static or non-nested functions, and the delegate form is analogous
 to non-static nested functions. In other words, a delegate literal can
 access stack variables in its enclosing function, a function literal
 cannot."
 
 As you can see, the docs are as clear as mud.  I suppose a "non-static
 nested function" wouldn't know about the 'this' pointer, nor care -
 but then why encumber it with the 'delegate' keyword?
 

Really, some BIG RED TEXT needs to be added to the doc in few places. This is one of them. Anon delegates are totally invalid after the function they are declared within returns. Any statement of the form: return {...}; is ALWAYS invalid. This has gotten about a dozen people in the last week or so and, as you said. "the docs are as clear as mud" on this. Anon delegate in any scope use the current function's frame pointer as the context. Once the function returns, this more than likely is junk.
Jan 16 2007
parent reply Pragma <ericanderton yahoo.removeme.com> writes:
BCS wrote:
 Reply to Pragma,
 
 All,
 I ran into a small problem regarding the use of anon delegates, and
 what their context (.ptr) is set to.  If anyone
 can share any insight as to if this is a bug or not, I would
 appreciate it.
 In short: I'm attempting to create delegates on the fly and use them
 elsewhere.  After many strange errors, I narrowed it down to the
 following case.  Note the assertions at the end:

 abstract class Foobar{
 Fn getFn();
 }
 alias void delegate() Fn;

 auto foo = new class Foobar{
 int x;
 Fn getFn(){
 return { x = x };
 }
 };
 auto fn = foo.getFn();
 assert((&foo.getFn).ptr is foo); // passes
 assert(fn.ptr is foo); // fails (ptr does *not* point to the object)
 So by this, anon delegates are not the same as actual object
 delegates.  Is this a bug, or is this intentional?

 "When comparing with nested functions, the function form is analogous
 to static or non-nested functions, and the delegate form is analogous
 to non-static nested functions. In other words, a delegate literal can
 access stack variables in its enclosing function, a function literal
 cannot."

 As you can see, the docs are as clear as mud.  I suppose a "non-static
 nested function" wouldn't know about the 'this' pointer, nor care -
 but then why encumber it with the 'delegate' keyword?

Really, some BIG RED TEXT needs to be added to the doc in few places. This is one of them. Anon delegates are totally invalid after the function they are declared within returns. Any statement of the form: return {...}; is ALWAYS invalid. This has gotten about a dozen people in the last week or so and, as you said. "the docs are as clear as mud" on this. Anon delegate in any scope use the current function's frame pointer as the context. Once the function returns, this more than likely is junk.

Thanks for the clarification. I wanted to be sure that I wasn't out of my mind. This is really disappointing. I was working on a runtime Spirit engine (as a counterpart to your own library) that would use inline delegates as valid parse expressions: auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args parsed"); }; It works fine for this kind of stuff, but once you graduate beyond there and want to bind terminals to class members, you need the ability to perform filtering and aritmetic on them. But once you do that, you find that your stack frame is all out of whack. :( // This construct is used to help create a context for binding and manipulating parse terms. // create() is called within LambdaExpr's constructor to create and cache the Expr to be used // at this stage of the parser. Of course, the inline delegates are invalid once this returns. /** 'factor' rule from the infamous 'infix calculator' BNF grammar. */ auto factor = new class LambdaExpr{ real result,term; Expr create(){ auto posFactor = T('+') >> ws >> term[&term] >> { result += term; }; auto negFactor = T('-') >> ws >> term[&term] >> { result +- term; }; return ws >> term[&result] >> -+(ws >> (posFactor | negFactor)) >> { return result; }; } }; It would be nice if things so succinct would work as I expected. Instead I have to break all the delegates out into proper class methods, which bloats the code and runs contrary to the whole point of having a parser framework like this. The ability to declare *first class* delegates as anonymous and inline like the above is just too tempting to leave alone. I suppose the only way to have your cake and eat it too is to preserve the stack somehow. Maybe this is a job for stack threads? -- - EricAnderton at yahoo
Jan 16 2007
next sibling parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Pragma wrote:
[snip]
 It would be nice if things so succinct would work as I expected.  
 Instead I have to break all the delegates out into proper class methods, 
 which bloats the code and runs contrary to the whole point of having a 
 parser framework like this.
 
 
 The ability to declare *first class* delegates as anonymous and inline 
 like the above is just too tempting to leave alone.  I suppose the only 
 way to have your cake and eat it too is to preserve the stack somehow.  
 Maybe this is a job for stack threads?

Maybe we could get Walter to implement a slight change to such delegates: instead of getting the innermost context pointer, they should get the outermost possible context pointer that still contains all the variables/members used? So then in your case they would get the 'this' pointer, in nested functions they could get the stackframe of the outer function if they didn't reference variables local to the inner function, and in member functions of (non-static) nested classes they could even contain a copy of the 'outer' pointer of the nested class, so only the outer class is referenced if that's all that's needed. I have no idea how difficult this would be to implement, but I think this will go a long way towards making delegates more usable. The only disadvantage would be that the validity of a delegate after a function call returns would depend on the code it contains, which may be a source of tricky bugs... An alternative would be the explicit context syntax I somewhere today (can't find it anymore :( ). Your example delegate would then look something like 'this.{ x = x; }'. Not as pretty, but still much better than having to define a new member function for this operator, don't you think?
Jan 16 2007
prev sibling next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Pragma,

[...]
 Thanks for the clarification.  I wanted to be sure that I wasn't out
 of my mind.
 
 This is really disappointing.  I was working on a runtime Spirit
 engine (as a counterpart to your own library) that would use inline
 delegates as valid parse expressions:

Cool!
 
 auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args
 parsed"); };
 
 It works fine for this kind of stuff, but once you graduate beyond
 there and want to bind terminals to class members, you need the
 ability to perform filtering and aritmetic on them.  But once you do
 that, you find that your stack frame is all out of whack. :(

I tend to use structs for that kind of thing. They entail less overhead than classes. [...]
 It would be nice if things so succinct would work as I expected.
 Instead I have to break all the delegates out into proper class
 methods, which bloats the code and runs contrary to the whole point of
 having a parser framework like this.

I've said it befor, What is needed is to allow delegate leterals to expicetly state there context. auto c = new C; return c.(char[] foo){retrun this.toString() ~ foo;} Another thing would be non static init for anon structs. This and a few others would make for some neat code. int delegate(int,int,int) Bar(int i, int j, int k) { return new struct { int a = i; int b = j; int c = k; }.(int m, int n, int o){ return a*m + b*n + c*o;}; }
 
 The ability to declare *first class* delegates as anonymous and inline
 like the above is just too tempting to leave alone.  I suppose the
 only way to have your cake and eat it too is to preserve the stack
 somehow.

I guess that is more or less what the above does. (Where's the 2.0 wish list? <g> )
Jan 16 2007
parent reply Pragma <ericanderton yahoo.removeme.com> writes:
BCS wrote:
 Reply to Pragma,
 auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args
 parsed"); };

 It works fine for this kind of stuff, but once you graduate beyond
 there and want to bind terminals to class members, you need the
 ability to perform filtering and aritmetic on them.  But once you do
 that, you find that your stack frame is all out of whack. :(

I tend to use structs for that kind of thing. They entail less overhead than classes.

I would have done the same, but I'm making heavy use of polymorphism to get the operators to work as expected. However, that could be refactored into a struct-oriented design with some clever templates and mixins to emulate inheritance.
 
 [...]
 It would be nice if things so succinct would work as I expected.
 Instead I have to break all the delegates out into proper class
 methods, which bloats the code and runs contrary to the whole point of
 having a parser framework like this.

I've said it befor, What is needed is to allow delegate leterals to expicetly state there context. auto c = new C; return c.(char[] foo){retrun this.toString() ~ foo;} Another thing would be non static init for anon structs. This and a few others would make for some neat code. int delegate(int,int,int) Bar(int i, int j, int k) { return new struct { int a = i; int b = j; int c = k; }.(int m, int n, int o){ return a*m + b*n + c*o;}; }

I read the other thread here, in D.learn, on this concept. I like the ability to anonymously declare any construct, but the ability to add literal methods/members to those is where it really shines. I've been using similar techniques in Javascript programming, and it completely transforms the way you approach solving particular problems. As Walter has written a compliant JS engine, you'd think he's be wise to the advantages here? Maybe there's something else blocking this kind of syntax from inclusion into D.
 
 The ability to declare *first class* delegates as anonymous and inline
 like the above is just too tempting to leave alone.  I suppose the
 only way to have your cake and eat it too is to preserve the stack
 somehow.

I guess that is more or less what the above does. (Where's the 2.0 wish list? <g> )

You mean, we dont' have one yet?! -- - EricAnderton at yahoo
Jan 17 2007
parent reply BCS <BCS pathlink.com> writes:
Pragma wrote:
 BCS wrote:
 
 Reply to Pragma,

 auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args
 parsed"); };

 It works fine for this kind of stuff, but once you graduate beyond
 there and want to bind terminals to class members, you need the
 ability to perform filtering and aritmetic on them.  But once you do
 that, you find that your stack frame is all out of whack. :(

I tend to use structs for that kind of thing. They entail less overhead than classes.

I would have done the same, but I'm making heavy use of polymorphism to get the operators to work as expected. However, that could be refactored into a struct-oriented design with some clever templates and mixins to emulate inheritance.

Ah, you must be doing something more complicated than I was thinking of. I often find myself using the pattern of a struct that is used in exactly one place: struct S { int i; int j(){return i} } auto s = new S; s.i=2; return &s.j this can be done with a class, but it has some more overhead. class C { int i; this(int k){i=k;} int j(){return i} } return &(new C(1)).j
 (Where's the 2.0 wish list?  <g> )

You mean, we dont' have one yet?!

Not that I know of :-|
Jan 17 2007
parent Pragma <ericanderton yahoo.removeme.com> writes:
BCS wrote:
 Pragma wrote:
 BCS wrote:

 Reply to Pragma,

 auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args
 parsed"); };

 It works fine for this kind of stuff, but once you graduate beyond
 there and want to bind terminals to class members, you need the
 ability to perform filtering and aritmetic on them.  But once you do
 that, you find that your stack frame is all out of whack. :(

I tend to use structs for that kind of thing. They entail less overhead than classes.

I would have done the same, but I'm making heavy use of polymorphism to get the operators to work as expected. However, that could be refactored into a struct-oriented design with some clever templates and mixins to emulate inheritance.

Ah, you must be doing something more complicated than I was thinking of.

Yea, sorry to talk so much about something that is basically vaporware. I've learned to be cautious of releasing stuff without having completely worked the kinks out of it myself - if it's at all useful, poor quality offerings breed a contemptible support situation for everyone involved (especially for me). But since the cat's out of the bag, I'll explain what I'm talking about. Basically, it's something like this (paraphrased for clarity): abstract class Expr{ Expr opShr(Expr expr){ return new AndGroup(this,expr); } Expr opShr(String value){ return opShr(new Terminal(value)); } Expr opOr(Expr expr){ return new OrGroup(this,expr); } Expr opOr(String value){ return opOr(new Terminal(value)); } Expr opPlus(Expr expr){ return new Multiple(this); } Expr opNeg(Expr expr){ return new Optional(this); } bool Parse(IScanner scanner); } class Terminal: Expr{} class Optional: Expr{} class Multiple: Expr{} class AndGroup: Expr{} class OrGroup: Expr{} ... and so on, plus a few shims to get some expressions to work. So that: Expr ws = /* defined elsewhere */ Expr rule = ws >> 'hello' >> ws >> 'world'; is equivalent to: Expr rule = new AndGroup( ws, new AndGroup( new Terminal('hello'), new AndGroup( ws, new Terminal('world') ) ) ); The various operators are overridden in the sub-classes so optimizations can be made. For instance, AndGroup and OrGroup will maintain a list of terms such that the redundant nesting in the above never happens. Now, to bring it back to where I started, I had some methods on Expr that did this: Expr opShr(void delegate() predicate){ return opShr(new PredicateExpr(predicate)); } Expr opOr(void delegate() predicate){ return opOr(new PredicateExpr(predicate)); } ... but that's dangerous for the reasons we discussed earlier. FWIW, another limitation I ran into is that D offers no static operator overloads in the global scope, so we can't get operators to generate templates in the same way that Spirit can in C++. Granted, that's a small limitation, but a 100% static parser /might/ help somewhat.
 I often find myself using the pattern of a struct that is used in 
 exactly one place:
 
 struct S
 {
     int i;
     int j(){return i}
 }
 
 auto s = new S;
 s.i=2;
 return &s.j
 
 this can be done with a class, but it has some more overhead.
 
 class C
 {
     int i;
     this(int k){i=k;}
     int j(){return i}
 }
 
 return &(new C(1)).j
 
 (Where's the 2.0 wish list?  <g> )

You mean, we dont' have one yet?!

Not that I know of :-|

-- - EricAnderton at yahoo
Jan 17 2007
prev sibling parent BCS <ao pathlink.com> writes:
You might try somthing like this (warning it has been called "evil"), this 
vertion dosn't work but it can actualy be made to by using structs and a 
little refactoring. The basic idea is use structs with delegate as members 
to build operatio trees. In this case I used it for the reduction, you'd 
could using it for parsing as well.

p.s. It realy is evil, you have been warned.


    // a base type for derivation
class Type{}
    // a templated derived type to store things in
class TypeT(T) : Type
{
    public T val;
    static TypeT!(T) opCall(T v)
    {
        auto ret = new TypeT!(T);
        ret.val = v;
        return ret;
    }
}
/*
Doc ::=
     A:a B:b C:c    = new Fig(a,b,c)
    | D:d E:e        = new Fig(d,e)
*/
TypeT!(Fig) delegate() Parse_Doc(inout char[] from)
{
    char[] tmp;

        // copy from
    tmp = from;
    if(auto a = Parse_A(tmp)) // try to parse an "a"
    if(auto b = Parse_B(tmp)) // try to parse an "b"
    if(auto c = Parse_C(tmp)) // try to parse an "c"
    {    // if all's good
            // update what's been parsed
        from = tmp;
        return &(
                // make an array from the args
            [cast(Type delegate())a,b,c]
                // copy it (force off stack)
            .dup
            )
                // form delegate from array
                // and function literal                
            .function TypeT!(Fig)(Type delegate()[] args)
            {
                    // literal calls given code
                auto ret =
                    new Fig(
                        // call args for values
                    cast(a.typeof)args[0]().val,
                    cast(b.typeof)args[1]().val,
                    cast(b.typeof)args[2]().val);

                    // place in object and return
                return TypeT!(ret.typeof)(ret);
            };
    }

        // restore tmp
    tmp = from;

        // blah, blah, blah
    if(auto d = Parse_D(tmp))
    if(auto e = Parse_E(tmp))
    {
        from = tmp;
        return &([cast(Type delegate())d,e].dup)
            .function TypeT!(Fig)(Type delegate()[] args)
            {
                auto ret =
                    new Fig(
                cast(d.typeof)args[0]().val,
                cast(e.typeof)args[1]().val);
                return TypeT!(ret.typeof)(ret);
            };
    }
    return null;
}
/*
A ::=
    "hello":v    = v
*/
TypeT!(char[]) delegate() Parse_A(inout char[] from)
{
    char[] tmp;

    tmp=from;
        // check for a "hello"
    if(from[0.."hello".length] == "hello")
    {
            // update from
        from = from["hello".length..$];
            // return delegate returning "hello" in obj
        return &("hello".dup).function TypeT!(char[])()
            {
                auto ret =
                    from[0.."hello".length];
                return TypeT!(ret.typeof)(ret);
            };
    }

    return null;
}
/// ........
Jan 16 2007