www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Reflection

reply Bruno Medeiros <daiphoenixNO SPAMlycos.com> writes:
Ok, I know many people have asked about reflection before, and surely 
everyone agrees it's a useful feature.
So what I want to know is what are the issues and challenges to design 
and implement such feature? Because now that I think of it, it seems to 
me it would be quite easy to implement runtime reflection, at least for 
field members of class objects. (method members have some problems, 
because of D's limited type information about function types)

So let's see, what do we need for reflection? We need to be able to 
determine the class of an anonymous object (already done with RTTI). And 
we need to be able do determine and access the field members of such 
object. That could be done, I believe, with a list of each field's type 
and offset relative to the object reference (this pointer).
Since we already have a TypeInfo structure, this could be done easily. 
Imagine that the ClassInfo class, has two additional members:
   TypeInfo[] memberids;
an array of the TypeInfo of each field of the class.
   int[] memberoffsets;
a respective array of the offsets relative to the 'this' pointer where 
the member is located. (should actually be size_t[] or something)

So now with this one can use reflection on an object, similarly to how 
one reads argument info from a variadic function with TypeInfo (which is 
where I got my idea). Here's a raw example of usage of such feature, a 
generic object serialization(writing to disk or somewhere else) function:


// Runtime reflection
void serialize(Object obj)
{
   TypeInfo[] memberids = obj.classinfo.memberids;
   int[] memberoffsets = obj.classinfo.memberoffsets;

   writefln(memberids.length, " members");

   foreach(int ix, TypeInfo memberid; memberids)
   {
     if (memberid == typeid(int)) {
       int m = * cast(int*) (cast(byte*)obj + memberoffsets[ix]);
       // Note that the previous could be abbreviate with something like:
       // int m = * obj.getMember!(ix, int);
       writefln("Member ",ix,", type int, value: ", m);
     }
     else if (memberid == typeid(char)) {
       char m = * cast(char*) (cast(byte*)obj + memberoffsets[ix]);
       writefln("Member ",ix,", type char, value: ", m);
     }
     // ... etc. for long, unsigneds and other primitive types

     else if (memberid.isClass()) {
       Object m = cast(Object) (cast(byte*)obj + memberoffsets[ix]);
       serialize(m);
     }
   }
}


Ok, this shows the usefulness of such feature, however there are some 
problems/limitations. It only works with primitive types (there is no 
isClass method of TypeInfo). For this feature to reach the full 
potential, then the TypeInfo class would have to be extended with info 
about the type's type, (i.e., the metatype, like struct, class, 
function, primitive, etc.), so that one could use reflection with 
non-primitive types, especially classes and arrays.

None of this seems to be hard or long to implement, so how about it? (I 
know many details need to be fleshed out, but the base idea seems good)

-- 
Bruno Medeiros - CS/E student
"Certain aspects of D are a pathway to many abilities some consider to 
be... unnatural."
Mar 02 2006
next sibling parent reply pragma <pragma_member pathlink.com> writes:
In article <du6vsh$1cep$1 digitaldaemon.com>, Bruno Medeiros says...
Ok, I know many people have asked about reflection before, and surely 
everyone agrees it's a useful feature.
So what I want to know is what are the issues and challenges to design 
and implement such feature? Because now that I think of it, it seems to 
me it would be quite easy to implement runtime reflection, at least for 
field members of class objects. (method members have some problems, 
because of D's limited type information about function types)

None of this seems to be hard or long to implement, so how about it? (I 
know many details need to be fleshed out, but the base idea seems good)

This is as good as any of the other proposals for Reflection this NG has seen. Given that, I don't think the problem has ever been with the design of any given proposal over the other. :) If I had to guess, the major reason that's holding this kind of thing back from the D specification, is how to justify adding all of the reflection symbol data into the executable/binary/whatever. I've learned that it can take up a very serious amount of space, most of which is usually wasted thanks to stuff that never comes up. For example: you're not likely to attempt to serialze something in the std.stdio namespace. But I also think nobody would argue that runtime reflection is a tradeoff between bloat and a whole range of algorithms that require it. The problem lies more on the side of systems programmers and folks who shouldn't have to contend with such baggage. IMO, the the best way towoard a reflection API would be the addition of a compiler switch that toggles the existance of an internal symbol lookup table*. This way *any* reflection API proposal, Typeinfo extensions included, would simply be built on top of it; plus you get a nice low-level rendition (just the table) to work with too. Should the developer not want that bloat, they simply turn off the lookup generation, which lets the respective D reflection code simply fall flat on its face (symbol searches and lookups will simply return nothing, or throw). This way, you can shift D toward whatever development mode you have in mind, just like we do now with other constructs. (*Alternatively, this could be done via code introspection much like how inlining works - but its just nicer to be in charge of what gets added and what gets tossed out) - EricAnderton at yahoo
Mar 02 2006
parent Bruno Medeiros <daiphoenixNO SPAMlycos.com> writes:
pragma wrote:
 In article <du6vsh$1cep$1 digitaldaemon.com>, Bruno Medeiros says...
 Ok, I know many people have asked about reflection before, and surely 
 everyone agrees it's a useful feature.
 So what I want to know is what are the issues and challenges to design 
 and implement such feature? Because now that I think of it, it seems to 
 me it would be quite easy to implement runtime reflection, at least for 
 field members of class objects. (method members have some problems, 
 because of D's limited type information about function types)

 None of this seems to be hard or long to implement, so how about it? (I 
 know many details need to be fleshed out, but the base idea seems good)

This is as good as any of the other proposals for Reflection this NG has seen. Given that, I don't think the problem has ever been with the design of any given proposal over the other. :) If I had to guess, the major reason that's holding this kind of thing back from the D specification, is how to justify adding all of the reflection symbol data into the executable/binary/whatever. I've learned that it can take up a very serious amount of space, most of which is usually wasted thanks to stuff that never comes up. For example: you're not likely to attempt to serialze something in the std.stdio namespace. But I also think nobody would argue that runtime reflection is a tradeoff between bloat and a whole range of algorithms that require it. The problem lies more on the side of systems programmers and folks who shouldn't have to contend with such baggage. IMO, the the best way towoard a reflection API would be the addition of a compiler switch that toggles the existance of an internal symbol lookup table*. This way *any* reflection API proposal, Typeinfo extensions included, would simply be built on top of it; plus you get a nice low-level rendition (just the table) to work with too. Should the developer not want that bloat, they simply turn off the lookup generation, which lets the respective D reflection code simply fall flat on its face (symbol searches and lookups will simply return nothing, or throw). This way, you can shift D toward whatever development mode you have in mind, just like we do now with other constructs. (*Alternatively, this could be done via code introspection much like how inlining works - but its just nicer to be in charge of what gets added and what gets tossed out) - EricAnderton at yahoo

With the RTTI info we already have, I don't think reflection would be that much more information. (especially if we don't put reflection info about the member's names). But you are right, there should be eventually an option to control whether this info is added or not. If I'm not mistaken, most C++ compilers have options like this, with regards to C++ RTTI. -- Bruno Medeiros - CS/E student "Certain aspects of D are a pathway to many abilities some consider to be... unnatural."
Mar 03 2006
prev sibling next sibling parent reply Bruno Medeiros <daiphoenixNO SPAMlycos.com> writes:
Bruno Medeiros wrote:
 
 Ok, this shows the usefulness of such feature, however there are some 
 problems/limitations. It only works with primitive types (there is no 
 isClass method of TypeInfo). For this feature to reach the full 
 potential, then the TypeInfo class would have to be extended with info 
 about the type's type, (i.e., the metatype, like struct, class, 
 function, primitive, etc.), so that one could use reflection with 
 non-primitive types, especially classes and arrays.
 

object.d where there are children classes of TypeInfo for each of the metatypes. Nice! :D -- Bruno Medeiros - CS/E student "Certain aspects of D are a pathway to many abilities some consider to be... unnatural."
Mar 02 2006
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Bruno Medeiros wrote:
 Bruno Medeiros wrote:
 Ok, this shows the usefulness of such feature, however there are some 
 problems/limitations. It only works with primitive types (there is no 
 isClass method of TypeInfo). For this feature to reach the full 
 potential, then the TypeInfo class would have to be extended with info 
 about the type's type, (i.e., the metatype, like struct, class, 
 function, primitive, etc.), so that one could use reflection with 
 non-primitive types, especially classes and arrays.

object.d where there are children classes of TypeInfo for each of the metatypes. Nice! :D

And there are more in std/typeinfo. The problem is that, as far as I know, the names of these classes are not standardized, nor is the output returned by the name method, so they're of limited direct use. For Ares I actually moved std/typeinfo into the runtime and don't expect the classes to be used directly, but rather for the user to do typeid(int) or whatever to return a reference to the appropriate TypeInfo object. Sean
Mar 02 2006
parent reply Bruno Medeiros <daiphoenixNO SPAMlycos.com> writes:
Sean Kelly wrote:
 And there are more in std/typeinfo.  The problem is that, as far as I 
 know, the names of these classes are not standardized, nor is the output 
 returned by the name method, so they're of limited direct use.  For Ares 
 I actually moved std/typeinfo into the runtime and don't expect the 
 classes to be used directly, but rather for the user to do typeid(int) 
 or whatever to return a reference to the appropriate TypeInfo object.
 
 
 Sean

Uh, I think the idea in D/Phobos is the same, to use typeid(sometype) and not the use the classes directly. -- Bruno Medeiros - CS/E student "Certain aspects of D are a pathway to many abilities some consider to be... unnatural."
Mar 03 2006
parent reply Sean Kelly <sean f4.ca> writes:
Bruno Medeiros wrote:
 Sean Kelly wrote:
 And there are more in std/typeinfo.  The problem is that, as far as I 
 know, the names of these classes are not standardized, nor is the 
 output returned by the name method, so they're of limited direct use.  
 For Ares I actually moved std/typeinfo into the runtime and don't 
 expect the classes to be used directly, but rather for the user to do 
 typeid(int) or whatever to return a reference to the appropriate 
 TypeInfo object.

Uh, I think the idea in D/Phobos is the same, to use typeid(sometype) and not the use the classes directly.

Then they shouldn't be in std. That was my only point. Sean
Mar 03 2006
parent Bruno Medeiros <daiphoenixNO SPAMlycos.com> writes:
Sean Kelly wrote:
 Bruno Medeiros wrote:
 Sean Kelly wrote:
 And there are more in std/typeinfo.  The problem is that, as far as I 
 know, the names of these classes are not standardized, nor is the 
 output returned by the name method, so they're of limited direct 
 use.  For Ares I actually moved std/typeinfo into the runtime and 
 don't expect the classes to be used directly, but rather for the user 
 to do typeid(int) or whatever to return a reference to the 
 appropriate TypeInfo object.

Uh, I think the idea in D/Phobos is the same, to use typeid(sometype) and not the use the classes directly.

Then they shouldn't be in std. That was my only point. Sean

Ah then, got it. And I agree. -- Bruno Medeiros - CS/E student "Certain aspects of D are a pathway to many abilities some consider to be... unnatural."
Mar 04 2006
prev sibling parent Bruno Medeiros <daiphoenixNO SPAMlycos.com> writes:
Bruno Medeiros wrote:
 Bruno Medeiros wrote:
 Ok, this shows the usefulness of such feature, however there are some 
 problems/limitations. It only works with primitive types (there is no 
 isClass method of TypeInfo). For this feature to reach the full 
 potential, then the TypeInfo class would have to be extended with info 
 about the type's type, (i.e., the metatype, like struct, class, 
 function, primitive, etc.), so that one could use reflection with 
 non-primitive types, especially classes and arrays.

object.d where there are children classes of TypeInfo for each of the metatypes. Nice! :D

Hum, perhaps the member reflection could be in the the TypeInfo class itself. This way, not only structs could have reflection too, but maybe functions could also have a "members" like functionality with regards to their parameters! (thus achieving full type info for functions/delegates) -- Bruno Medeiros - CS/E student "Certain aspects of D are a pathway to many abilities some consider to be... unnatural."
Mar 03 2006
prev sibling parent AgentOrange <AgentOrange_member pathlink.com> writes:
Yeah I think this is generally how walter had planned on doing it also. I dont
see it currently, but in an old version of phobos he had a commented out section
inside ClassInfo that said something to the effect // BUG: field typeinfo list
goes here

its a good idea, and we can spit out reflection info all day long, as ive been
experimenting in my dparser project, we just need to come up with a clean way of
doing it.... I think somewhere along the way we are going to want a customized
Object module - ala something like ares....

it would be nice to hear walters ideas, plans, etc. for reflection, just so we
have his take on the matter, but oh well.....




In article <du6vsh$1cep$1 digitaldaemon.com>, Bruno Medeiros says...
Ok, I know many people have asked about reflection before, and surely 
everyone agrees it's a useful feature.
So what I want to know is what are the issues and challenges to design 
and implement such feature? Because now that I think of it, it seems to 
me it would be quite easy to implement runtime reflection, at least for 
field members of class objects. (method members have some problems, 
because of D's limited type information about function types)

So let's see, what do we need for reflection? We need to be able to 
determine the class of an anonymous object (already done with RTTI). And 
we need to be able do determine and access the field members of such 
object. That could be done, I believe, with a list of each field's type 
and offset relative to the object reference (this pointer).
Since we already have a TypeInfo structure, this could be done easily. 
Imagine that the ClassInfo class, has two additional members:
   TypeInfo[] memberids;
an array of the TypeInfo of each field of the class.
   int[] memberoffsets;
a respective array of the offsets relative to the 'this' pointer where 
the member is located. (should actually be size_t[] or something)

So now with this one can use reflection on an object, similarly to how 
one reads argument info from a variadic function with TypeInfo (which is 
where I got my idea). Here's a raw example of usage of such feature, a 
generic object serialization(writing to disk or somewhere else) function:


// Runtime reflection
void serialize(Object obj)
{
   TypeInfo[] memberids = obj.classinfo.memberids;
   int[] memberoffsets = obj.classinfo.memberoffsets;

   writefln(memberids.length, " members");

   foreach(int ix, TypeInfo memberid; memberids)
   {
     if (memberid == typeid(int)) {
       int m = * cast(int*) (cast(byte*)obj + memberoffsets[ix]);
       // Note that the previous could be abbreviate with something like:
       // int m = * obj.getMember!(ix, int);
       writefln("Member ",ix,", type int, value: ", m);
     }
     else if (memberid == typeid(char)) {
       char m = * cast(char*) (cast(byte*)obj + memberoffsets[ix]);
       writefln("Member ",ix,", type char, value: ", m);
     }
     // ... etc. for long, unsigneds and other primitive types

     else if (memberid.isClass()) {
       Object m = cast(Object) (cast(byte*)obj + memberoffsets[ix]);
       serialize(m);
     }
   }
}


Ok, this shows the usefulness of such feature, however there are some 
problems/limitations. It only works with primitive types (there is no 
isClass method of TypeInfo). For this feature to reach the full 
potential, then the TypeInfo class would have to be extended with info 
about the type's type, (i.e., the metatype, like struct, class, 
function, primitive, etc.), so that one could use reflection with 
non-primitive types, especially classes and arrays.

None of this seems to be hard or long to implement, so how about it? (I 
know many details need to be fleshed out, but the base idea seems good)

-- 
Bruno Medeiros - CS/E student
"Certain aspects of D are a pathway to many abilities some consider to 
be... unnatural."

Mar 02 2006