www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Need reflection facilities

reply Dave Woldrich <dave woldrich.com> writes:
Hello all, I read on the realtime type information how-to page that there are
some runtime type information facilities in D.
(http://www.prowiki.org/wiki4d/wiki.cgi?action=browse&id=HowTo/RealtimeTypeInformation)

I am the author of CeeFIT (http://ceefit.woldrich.com), the C++ implementation
of the Framework for Integrated Test (http://fit.c2.com).  It was extremely
painful to implement portable reflection capabilities into C++ to mimic what
goes on in Java.  Basically I used every function pointer, template, macro,
and static linker trick I knew to pull it off (and it's still fugly.)

If I were to implement fit for D (DeeFIT?) it would be extremely helpful to
have Class.forName() functionality out of the box as well as the ability to
get pointers-to-function and references to fields by name as well for a given
object.

Would it be possible for the D compiler to generate some table of symbols or
synthetically generated reflection module and provide that for linkage into
the final binary?  Maybe the compiler could employ the same sorts template
specialization and static initialization techniques I used in CeeFIT to
generate such artifacts.  Clearly such a table could double the size of the
resulting .exe, but not like I care.

Cheers,
Dave Woldrich
Jan 03 2007
next sibling parent Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Dave Woldrich wrote:
 Hello all, I read on the realtime type information how-to page that there are
 some runtime type information facilities in D.
 (http://www.prowiki.org/wiki4d/wiki.cgi?action=browse&id=HowTo/RealtimeTypeInformation)
 
 I am the author of CeeFIT (http://ceefit.woldrich.com), the C++ implementation
 of the Framework for Integrated Test (http://fit.c2.com).  It was extremely
 painful to implement portable reflection capabilities into C++ to mimic what
 goes on in Java.  Basically I used every function pointer, template, macro,
 and static linker trick I knew to pull it off (and it's still fugly.)
 

D's reflection support is certainly better than C++'s. However, it is not quite as powerful as you are wishing it is. :-)
 If I were to implement fit for D (DeeFIT?) it would be extremely helpful to
 have Class.forName() functionality out of the box as well as the ability to
 get pointers-to-function and references to fields by name as well for a given
 object.
 

These are not possible directly. Some other things are: * The reverse operation, getting the name of a class at runtime, is possible. Every Object instance has a "classinfo" property, which in turn has a "name" property. (It also have a "vtbl" property, of type void*[], which might prove useful for this kind of trickery.) Or stop listening to me, and read the docs: :-) http://digitalmars.com/d/phobos/object.html * D is remarkably easy to parse, especially compared to C++. If you have to, parsing user code to generate specialized code can be done in a much more robust way than C++ can dream of. * D's /compile-time/ reflection abilities kick the pants off C++. Search around for Don Clugston's "Nameof" module. It allows you to get the name of just about any symbol at compile time. (The reverse operation of getting a symbol from its name is still not possible, however.)
 Would it be possible for the D compiler to generate some table of symbols or
 synthetically generated reflection module and provide that for linkage into
 the final binary?  Maybe the compiler could employ the same sorts template
 specialization and static initialization techniques I used in CeeFIT to
 generate such artifacts.  Clearly such a table could double the size of the
 resulting .exe, but not like I care.
 

As I said, parsing this information out yourself is almost /easy/ compared to C++. It might also be worth pointing out that strings are allowed as template parameters in D. Also, I once again suggest you look at Don Clugston's Nameof (and Demangle). It's a fine bit of work, which appears to do half of this problem already. -- Kirk McDonald Pyd: Wrapping Python with D http://pyd.dsource.org
Jan 03 2007
prev sibling parent reply Pragma <ericanderton yahoo.removeme.com> writes:
Dave Woldrich wrote:
 Hello all, I read on the realtime type information how-to page that there are
 some runtime type information facilities in D.
 (http://www.prowiki.org/wiki4d/wiki.cgi?action=browse&id=HowTo/RealtimeTypeInformation)
 
 I am the author of CeeFIT (http://ceefit.woldrich.com), the C++ implementation
 of the Framework for Integrated Test (http://fit.c2.com).  It was extremely
 painful to implement portable reflection capabilities into C++ to mimic what
 goes on in Java.  Basically I used every function pointer, template, macro,
 and static linker trick I knew to pull it off (and it's still fugly.)
 
 If I were to implement fit for D (DeeFIT?) it would be extremely helpful to
 have Class.forName() functionality out of the box as well as the ability to
 get pointers-to-function and references to fields by name as well for a given
 object.
 
 Would it be possible for the D compiler to generate some table of symbols or
 synthetically generated reflection module and provide that for linkage into
 the final binary?  Maybe the compiler could employ the same sorts template
 specialization and static initialization techniques I used in CeeFIT to
 generate such artifacts.  Clearly such a table could double the size of the
 resulting .exe, but not like I care.

Howdy Dave. To my knowledge, there are two approaches one can use to put together a reflection interface on top of D, w/o hacking the compiler directly. Neither is very pretty. 1) Use the DMD front-end to compose a utility that pre-processes .d files and generates the needed reflection support code. I think there are other D-ers who are doing this, but I can't recall who at the moment (hasn't been much word about it lately). I honestly don't see why this can't be done today, and made to work everywhere D already does. The facts that the front-end will furnish you with a 100% spec-compliant AST, D supports the #line directive, and the spec allows for custom pragma() statements, only make this approach more appealing. 2) Parse the .map file. D's mangling specification allows for far more useful information than what any C++ compiler spits out. The result is that you can extract entire call-signatures and name-spaces directly from the symbol table. I'm presently using this technique in DDL to bootstrap dynamic linking with object files: http://www.dsource.org/projects/ddl The downside to this approach is that it leaves out just about everything that can resolve to an offset or bitfield: enums, struct members, class members, local vars, etc. Otherwise, everything with a vtable, call signature, typeInfo or moduleInfo is preserved and easily accessed at runtime. Both approaches leave the actual reflection interface (beyond TypeInfo) up for grabs. There have been partial implementations, but nothing as complete as say what .NET or Java offers. My $0.02 on the matter is that if you're first past the post on this, whatever you come up with is likely to become adopted in one way or another. ;) I hope this helps. -- - EricAnderton at yahoo
Jan 03 2007
next sibling parent reply Pragma <ericanderton yahoo.removeme.com> writes:
Pragma wrote:
 Dave Woldrich wrote:
 Hello all, I read on the realtime type information how-to page that 
 there are
 some runtime type information facilities in D.
 (http://www.prowiki.org/wiki4d/wiki.cgi?action=browse&id=HowTo/Real
imeTypeInformation) 


 I am the author of CeeFIT (http://ceefit.woldrich.com), the C++ 
 implementation
 of the Framework for Integrated Test (http://fit.c2.com).  It was 
 extremely
 painful to implement portable reflection capabilities into C++ to 
 mimic what
 goes on in Java.  Basically I used every function pointer, template, 
 macro,
 and static linker trick I knew to pull it off (and it's still fugly.)

 If I were to implement fit for D (DeeFIT?) it would be extremely 
 helpful to
 have Class.forName() functionality out of the box as well as the 
 ability to
 get pointers-to-function and references to fields by name as well for 
 a given
 object.

 Would it be possible for the D compiler to generate some table of 
 symbols or
 synthetically generated reflection module and provide that for linkage 
 into
 the final binary?  Maybe the compiler could employ the same sorts 
 template
 specialization and static initialization techniques I used in CeeFIT to
 generate such artifacts.  Clearly such a table could double the size 
 of the
 resulting .exe, but not like I care.

Howdy Dave. To my knowledge, there are two approaches one can use to put together a reflection interface on top of D, w/o hacking the compiler directly. Neither is very pretty. 1) Use the DMD front-end to compose a utility that pre-processes .d files and generates the needed reflection support code. I think there are other D-ers who are doing this, but I can't recall who at the moment (hasn't been much word about it lately). I honestly don't see why this can't be done today, and made to work everywhere D already does. The facts that the front-end will furnish you with a 100% spec-compliant AST, D supports the #line directive, and the spec allows for custom pragma() statements, only make this approach more appealing. 2) Parse the .map file. D's mangling specification allows for far more useful information than what any C++ compiler spits out. The result is that you can extract entire call-signatures and name-spaces directly from the symbol table. I'm presently using this technique in DDL to bootstrap dynamic linking with object files: http://www.dsource.org/projects/ddl The downside to this approach is that it leaves out just about everything that can resolve to an offset or bitfield: enums, struct members, class members, local vars, etc. Otherwise, everything with a vtable, call signature, typeInfo or moduleInfo is preserved and easily accessed at runtime. Both approaches leave the actual reflection interface (beyond TypeInfo) up for grabs. There have been partial implementations, but nothing as complete as say what .NET or Java offers. My $0.02 on the matter is that if you're first past the post on this, whatever you come up with is likely to become adopted in one way or another. ;) I hope this helps.

Addendum: Kirk is right on the money about the Don's Meta library. You might be able to put something together that works via explicit registration of classes like so: DeeFitRegister!(MyClass); That said, Meta is a work of Template Ninjitsu that is hard to top. Tuples and the rest of D's meta programming capabilities should help you decompose classes and structs from there. But if you need to answer questions like Class.nameof("Foobar"), then you've already out-stripped what can be done at compile-time. :) -- - EricAnderton at yahoo
Jan 03 2007
parent reply Dave Woldrich <dave woldrich.com> writes:
 Addendum:  Kirk is right on the money about the Don's Meta library.  You
 might be able to put something together that works via explicit
 registration of classes like so:
 DeeFitRegister!(MyClass);
 That said, Meta is a work of Template Ninjitsu that is hard to top.
 Tuples and the rest of D's meta programming capabilities should help you
 decompose classes and structs from there.
 But if you need to answer questions like Class.nameof("Foobar"), then
 you've already out-stripped what can be done at compile-time. :)

Yeah, for fit implementations, I need something like Java's "Object Class.forName(String clazz)", "Field Class.getField(String name)", and "Method Class.getMethod(String name, Class[] parameterTypes)" :( So maybe I'm asking for too much, but I hope not since I can mostly get what I want in C++ and that's nearly a full letter less than D. :) Thanks, Dave Woldrich
Jan 03 2007
parent reply Thomas Kuehne <thomas-dloop kuehne.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave Woldrich schrieb am 2007-01-03:
 Addendum:  Kirk is right on the money about the Don's Meta library.  You
 might be able to put something together that works via explicit
 registration of classes like so:
 DeeFitRegister!(MyClass);
 That said, Meta is a work of Template Ninjitsu that is hard to top.
 Tuples and the rest of D's meta programming capabilities should help you
 decompose classes and structs from there.
 But if you need to answer questions like Class.nameof("Foobar"), then
 you've already out-stripped what can be done at compile-time. :)

Yeah, for fit implementations, I need something like Java's "Object Class.forName(String clazz)", "Field Class.getField(String name)", and "Method Class.getMethod(String name, Class[] parameterTypes)" :( So maybe I'm asking for too much, but I hope not since I can mostly get what I want in C++ and that's nearly a full letter less than D. :)

Flectioned should be able to solve your problems, it is however currently rudimental and only available for linux. http://svn.dsource.org/projects/flectioned/downloads/flectioned.zip Class.forName: cast(Class) cn.kuehne.flectioned.types["fully.qualified.name"] Class.getField: that is impossible without template foo Class.getMethod should be available at the end of the week Unpack and have a try: 1) gcc -m32 -c elf.c 2) dmd flectioned.d sample.d elf.o -ofsample 3) ./sample Thomas -----BEGIN PGP SIGNATURE----- iD8DBQFFnE+hLK5blCcjpWoRAvnRAJ9ImMXiLr2waf9/26agv0Dd2WwbgQCfXubs eqow1r9IUc/u3Uxd3ROkeHM= =v2pw -----END PGP SIGNATURE-----
Jan 03 2007
next sibling parent reply Dave Woldrich <dave woldrich.com> writes:
== Quote from Thomas Kuehne (thomas-dloop kuehne.cn)'s article
 Flectioned should be able to solve your problems, it is however currently
 rudimental and only available for linux.
 http://svn.dsource.org/projects/flectioned/downloads/flectioned.zip
 Class.forName:
 	cast(Class) cn.kuehne.flectioned.types["fully.qualified.name"]
 Class.getField:
 	that is impossible without template foo
 Class.getMethod
 	should be available at the end of the week
 Unpack and have a try:
 1) gcc -m32 -c elf.c
 2) dmd flectioned.d sample.d elf.o -ofsample
 3) ./sample
 Thomas

Thomas, Wow, your library is so impressive. You scan the running process for symbols, wicked! I am fascinated by how low level D programs can go, although I see you using a little C there to pull it off. And you made a Class class, yay! The Class.newInstance() method was very impressive. Anyhow, there's many ways to skin a cat, and I see you're an expert cat skinner. I look forward to seeing what you come up with next. I still think if the compiler did this work and generated type metadata code the reflection would execute faster, but your solution is working today, so I guess I shouldn't complain. :) Cheers, Dave Woldrich
Jan 03 2007
parent Thomas Kuehne <thomas-dloop kuehne.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave Woldrich schrieb am 2007-01-04:
== Quote from Thomas Kuehne (thomas-dloop kuehne.cn)'s article

 Wow, your library is so impressive.  You scan the running process for symbols,
 wicked!  I am fascinated by how low level D programs can go, although I see you
 using a little C there to pull it off.

The C part is only there because I reused existing code and haven't bothered to translate it to D. Thomas -----BEGIN PGP SIGNATURE----- iD8DBQFFnKgrLK5blCcjpWoRAk8qAKCDg8ojdx6nAMNtBfsoIXnVic7JBQCbBECy n6txwwyczHFYn1KtqjIAxxA= =0NZA -----END PGP SIGNATURE-----
Jan 03 2007
prev sibling parent janderson <askme me.com> writes:
 
 Flectioned should be able to solve your problems, it is however currently
 rudimental and only available for linux.
 
 http://svn.dsource.org/projects/flectioned/downloads/flectioned.zip
 
 Thomas
 
 

of the other comments made up there too however I'm either too lazy to don't have the time to reformat what was said. http://www.prowiki.org/wiki4d/wiki.cgi?HowTo/RealtimeTypeInformation
Jan 03 2007
prev sibling parent reply Dave Woldrich <dave woldrich.com> writes:
== Quote from Pragma (ericanderton yahoo.removeme.com)'s article

 Neither is very pretty.
 1) Use the DMD front-end to compose a utility that pre-processes .d
 files and generates the needed reflection support code.  I think there
 are other D-ers who are doing this, but I can't recall who at the moment
 (hasn't been much word about it lately).  I honestly don't see why this
 can't be done today, and made to work everywhere D already does.
 The facts that the front-end will furnish you with a 100% spec-compliant
 AST, D supports the #line directive, and the spec allows for custom
 pragma() statements, only make this approach more appealing.
 2) Parse the .map file.  D's mangling specification allows for far more
 useful information than what any C++ compiler spits out.  The result is
 that you can extract entire call-signatures and name-spaces directly
 from the symbol table.  I'm presently using this technique in DDL to
 bootstrap dynamic linking with object files:
 http://www.dsource.org/projects/ddl

Excellent, thanks for this response, I will look into it. I think hacking the compiler wouldn't be a bad deal and would probably produce the fastest executing solution. My thoughts based on what you are saying is that static construction of compiler-generated type metadata classes would be possible and could be enabled with a compiler switch. CeeFIT uses a two tier approach to registering types. Within classes that need reflection, CeeFIT has macros that register fields and functions for reflection for each class instance that is created. Macros place a "FITFIELD<T>" in the class for every field that needs to be found by name where T is the type of the field. They place a "FITTEST<T, U>" for every zero-arg function that needs to be found by name where T is the type of the containing class and U is the type of the return type. There are variants of FITTEST for one-arg functions as well. Each FITFIELD and FITTEST is initialized with a string name and a pointer to the member function or field. Templates take care of the rest. As each FITFIELD and FITTEST member placed by the macros initializes during object construction, they add themselves to a singly-linked list hosted by a base class that facilitates the reflection. The drawback of this is the additional memory and cpu cycles burned whenever a reflectable object is instantiated. I did this out of necessity since the C++ limitations had me hamstrung. Wherever the macros cant or won't work, I have a less efficient manual registration technique that users can use, but that's not really relevant to what I'm discussing here. I figure a D compiler extension that generates these field and function metadata artifacts could generate them to some class outside the classes being described and initialize them only once at startup. Leading to function and field metadata in a hashmap keyed by the string name representation. Slap a simple reflection API on top of that hashmap and you've got it. The second tier of the reflection facilities in CeeFIT is to register the classes themselves by name for construction. To do this, macros place a static REGISTERFIXTURECLASS<T> object after the class definition is closed where T is the class type. The linker ensures that the REGISTERFIXTURECLASS<T> is initialized at program start. REGISTERFIXTURECLASS<T> provides the mapping between class string name and the class's zero-arg default constructor. Upon initialization, the REGISTERFIXTURECLASS<T> adds itself to a global singly-linked list which can be searched at runtime to pull off the Class.forName() functionality I needed. Of course, searching singly-linked lists are going to perform badly for any decent sized application, and a D version of this code could add the statically initialized REGISTERFIXTURECLASS<T> to a hashmap or whatnot instead. I don't know what kind of DLL support there is in D, but dynamically loading and unloading classes and magically keeping this hashmap up to date would be so cherry. I am only just now learning about D, so I don't know what the limitations of the compiler are and whether such extensions would be feasible. As you were saying Eric, the map file could be sufficient to generate all of this type metadata code as a post-processing step, I just figure the compiler has all the information while it is running and therefore would be the best place to put the logic. Plus, to have reflection facilities be a first class feature of the language/compiler/runtime library would be very slick. If I had reflection in a systems-programming language like D, the tools that could be written to work with D would be incredible. I'm sure Eclipse for D would be a dream to use and would benefit from all the refactoring and type information that Java gets. I am already thinking of porting my videogame code to D (since I'm using a replacement GC heap in my C++ code anyhow, not like D is going to perform any slower.) I will probably make some ant tasks for D if I do and there aren't any already. I am just really liking the D language, thank you Walter and whoever else is responsible for this! :) Cheers, Dave Woldrich
Jan 03 2007
parent reply Georg Wrede <georg nospam.org> writes:
Dave Woldrich wrote:
 If I had reflection in a systems-programming language like D, the tools that
could
 be written to work with D would be incredible.

You might want to elaborate a little, maybe Walter is reading. ;-)
 I am already thinking of porting my videogame code to D (since I'm
 using a replacement GC heap in my C++ code anyhow, not like D is going to
perform
 any slower.)

I would love to hear how the porting went, how long it took, and any thoughts that you got doing it. Especially in view of you being pretty new to D while doing it.
 I am just really liking the D language, thank you Walter and whoever else is
 responsible for this!  :)

Join the club!
Jan 07 2007
parent reply Dave Woldrich <dave woldrich.com> writes:
== Quote from Georg Wrede (georg nospam.org)'s article
 Dave Woldrich wrote:
 If I had reflection in a systems-programming language like D, the tools that
could
 be written to work with D would be incredible.


You know, I made that comment sortof as a stream-of-consciousness aside. But your suggestion made me think a little harder about it, because the kind of type information that Eclipse builds for languages like Java would require quite a trick to pull off in D. Basically, Eclipse knows about every symbol and every class as you develop as you type them (in Java). It can do symbol typeahead suggestions, realtime reporting of syntax and missing symbol errors, automatic suggestions for error quick fixes, etc. It has panels that can show you complete type hierarchies of anything in the workspace, and these hierarchies will rearrange themselves in realtime as you make changes to the classes they are displaying. It has refactorings where you can push down and pull up classes, rename symbols globally, etc. Eclipse can do all that because it is constantly compiling java code that you enter and then inspecting the types and/or analyzing the compilation errors it gets. For something like that to be possible with D, we'd need a couple things. One is full reflection, where you can ask the runtime to give you a Class object for a given type's name and look at it's full hierarchy and query that Class's fields, methods, relationships, and dependencies. In terms of tooling for some super-dynamic environment like Eclipse, I think you'd almost have to have the ability to build each and every class as a DLL on the fly and load it. And the DLL loaded would have to init in some sort of "Class metadata only mode" where statics would not init on load - Eclipse would be loading the DLL only to get access to the Class objects themselves, not to actually run any code. Upon loading the class's DLL, the D runtime should be able to dynamically reconfigure it's internal type database so that Class queries will return correct type hierarchy information as the runtime understands that information now. Eclipse could direct the D runtime to unload class DLL's as well - and it would do this constantly as classes the user edits would be constantly recompiled in the background. I feel like the compiler for D would need to be intimately woven into the D runtime to achieve this kind of tooling. The compiler would need to be able to compile a class, while incorporating all symbol information currently present and loaded in the system. The compiler would have to have a runtime interface that could convert a D class file to a DLL and report on the names of the classes that were just built. Eclipse's tooling seems to take this instant compilation a step further: and if there are any bugs in the code that gets compiled, its Java compiler places UnresolvedCompilationError exceptions at the spot where each bug is located. This way, Eclipse's in-editor type system always has a valid class file to load and reflect upon. I think when the D runtime is running in "tool mode" it won't be very fast, but with some creative behind-the-scenes work in the D runtime, you could have a D program operate very dynamically. Class DLL's could be loaded and unloaded and be reflected upon as easily as classes in statically linked libraries are. If D had this kind of robust type reflection and tool support, developing low-level systems programs and games would be truly incredible. I could see plugins and addons for D for Eclipse sprouting up everywhere. The refactorings for D's advanced features would probably change how I write programs. --Dave
Jan 11 2007
parent reply Bradley Smith <digitalmars-com baysmith.com> writes:
Dave Woldrich wrote:
 == Quote from Georg Wrede (georg nospam.org)'s article
 Dave Woldrich wrote:
 If I had reflection in a systems-programming language like D, the tools that
could
 be written to work with D would be incredible.


You know, I made that comment sortof as a stream-of-consciousness aside. But your suggestion made me think a little harder about it, because the kind of type information that Eclipse builds for languages like Java would require quite a trick to pull off in D. Basically, Eclipse knows about every symbol and every class as you develop as you type them (in Java). It can do symbol typeahead suggestions, realtime reporting of syntax and missing symbol errors, automatic suggestions for error quick fixes, etc. It has panels that can show you complete type hierarchies of anything in the workspace, and these hierarchies will rearrange themselves in realtime as you make changes to the classes they are displaying. It has refactorings where you can push down and pull up classes, rename symbols globally, etc.

I could be wrong, but my understanding is that Eclipse implements these features with direct editing of the code symbol tree and automatic incremental compilation, not from reflection. Java reflection is a runtime feature. Regardless, I am in favor of some kind of reflection. Personally, I think reasonable means of accomplishing it would be through optional addition of compile-time information which when coupled with a library would allow for dynamic runtime features. Bradley
Jan 11 2007
parent Lars Ivar Igesund <larsivar igesund.net> writes:
Bradley Smith wrote:

 
 Dave Woldrich wrote:
 == Quote from Georg Wrede (georg nospam.org)'s article
 Dave Woldrich wrote:
 If I had reflection in a systems-programming language like D, the tools
 that could be written to work with D would be incredible.


You know, I made that comment sortof as a stream-of-consciousness aside. But your suggestion made me think a little harder about it, because the kind of type information that Eclipse builds for languages like Java would require quite a trick to pull off in D. Basically, Eclipse knows about every symbol and every class as you develop as you type them (in Java). It can do symbol typeahead suggestions, realtime reporting of syntax and missing symbol errors, automatic suggestions for error quick fixes, etc. It has panels that can show you complete type hierarchies of anything in the workspace, and these hierarchies will rearrange themselves in realtime as you make changes to the classes they are displaying. It has refactorings where you can push down and pull up classes, rename symbols globally, etc.

I could be wrong, but my understanding is that Eclipse implements these features with direct editing of the code symbol tree and automatic incremental compilation, not from reflection. Java reflection is a runtime feature.

Yes, the Descent Eclipse plugin has ported most of the DMD parser to be used in Eclipse, such that it can build it's own code tree necessary for this stuff to work. Reflection is not part of it. -- Lars Ivar Igesund blog at http://larsivi.net DSource & #D: larsivi Dancing the Tango
Jan 12 2007