www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Reflection Idea

reply "Craig Black" <cblack ara.com> writes:
Since DDL provides run-time reflection, and is finally ready to use, I 
thought it would be a good time to share my idea to improve performance when 
accessing metadata.

If we have an instance of a class and we want to it's metadata via a 
reflection database, we should be able to query the reflection database 
using a hash table lookup on the name of the class.

However, we could get the metadata more quickly if it was available via the 
TypeInfo class.  This could be made possible very easily if the TypeInfo 
class contained a reference to an IReflection interface.  This reference 
could be set by the reflection database at program startup.  This would 
provide a direct link to meta data for each class, and would make reflection 
a first class citizen in D.

-Craig 
Sep 14 2006
parent reply Pragma <ericanderton yahoo.removeme.com> writes:
Craig Black wrote:
 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve performance when 
 accessing metadata.

Technically DDL does, although I'm hesitant to call it that since it doesn't have all the typical reflection bells-and-whistles yet.
 
 If we have an instance of a class and we want to it's metadata via a 
 reflection database, we should be able to query the reflection database 
 using a hash table lookup on the name of the class.
 
 However, we could get the metadata more quickly if it was available via the 
 TypeInfo class.  This could be made possible very easily if the TypeInfo 
 class contained a reference to an IReflection interface.  This reference 
 could be set by the reflection database at program startup.  This would 
 provide a direct link to meta data for each class, and would make reflection 
 a first class citizen in D.

The only reason why I'd consider full-on reflection for D a bad move is that it'll bloat program sizes considerably. This is kind of contrary to D's goal as a systems programming language. Taking into consideration the sheer size of your average .map file is a good indicator of just how bad things can get. So I'd caution the addition of adding such a suite to D as something optional if at all possible. Even something minimalisitc would work - like the inclusion of just a hashtable of all the program's symbols (metadata as Craig put it above) that is *optionally* compiled in. Otherwise, we're stuck with .map/.situ support under DDL - i.e. using your .map file as a runtime dependency for the program. While this works well, and is such a dead-simple way of addressing the problem, it's also easy to get out of sync IMO. [PSA] Craig, if you're at all interested in hammering out a reflection interface (for DDL?), the DDL Community wiki there is open for your, and everyone else's abuse. http://www.dsource.org/projects/ddl/wiki/Community I have some reflection concepts, and a few other ideas, outlined under the DevNotes section as well: http://www.dsource.org/projects/ddl/wiki/DevNotes Enjoy! -- - EricAnderton at yahoo
Sep 14 2006
next sibling parent "Craig Black" <cblack ara.com> writes:
"Pragma" <ericanderton yahoo.removeme.com> wrote in message 
news:eeclc4$2206$1 digitaldaemon.com...
 Craig Black wrote:
 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve performance 
 when accessing metadata.

Technically DDL does, although I'm hesitant to call it that since it doesn't have all the typical reflection bells-and-whistles yet.

All in due time I'm sure. At this point, I'd be happy with just the basics.
 The only reason why I'd consider full-on reflection for D a bad move is 
 that it'll bloat program sizes considerably.  This is kind of contrary to 
 D's goal as a systems programming language.  Taking into consideration the 
 sheer size of your average .map file is a good indicator of just how bad 
 things can get.

 So I'd caution the addition of adding such a suite to D as something 
 optional if at all possible.  Even something minimalisitc would work - 
 like the inclusion of just a hashtable of all the program's symbols 
 (metadata as Craig put it above) that is *optionally* compiled in.

Yes, optional of coarse. DDL is a library, not a part of the D compiler. However, an extra reference per TypeInfo instance shouldn't hurt too bad. If not using reflection, the reference would simply be null.
 Otherwise, we're stuck with .map/.situ support under DDL - i.e. using your 
 .map file as a runtime dependency for the program.  While this works well, 
 and is such a dead-simple way of addressing the problem, it's also easy to 
 get out of sync IMO.

 [PSA]

 Craig, if you're at all interested in hammering out a reflection interface 
 (for DDL?), the DDL Community wiki there is open for your, and everyone 
 else's abuse.

I appreciate the offer very much. However, at the moment I am very busy building the next version of our software, which is written in C++. I'm waiting on D to become mature enough so that I can pitch it to my boss as something we cannot live without. The first time I mentioned D to him as a useful technology, he asked if it provided reflection, a feature that would be very useful for our product. I had to admit that it does not provide reflection yet. It's a hard sale for a number of technical reasons. I could elaborate, but probably nobody cares all that much. Anyway, perhaps I will take you up on that offer when I have more time. -Craig
Sep 15 2006
prev sibling next sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Pragma wrote:
 Craig Black wrote:
 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve 
 performance when accessing metadata.

Technically DDL does, although I'm hesitant to call it that since it doesn't have all the typical reflection bells-and-whistles yet.
 If we have an instance of a class and we want to it's metadata via a 
 reflection database, we should be able to query the reflection 
 database using a hash table lookup on the name of the class.

 However, we could get the metadata more quickly if it was available 
 via the TypeInfo class.  This could be made possible very easily if 
 the TypeInfo class contained a reference to an IReflection interface.  
 This reference could be set by the reflection database at program 
 startup.  This would provide a direct link to meta data for each 
 class, and would make reflection a first class citizen in D.

The only reason why I'd consider full-on reflection for D a bad move is that it'll bloat program sizes considerably. This is kind of contrary to D's goal as a systems programming language. Taking into consideration the sheer size of your average .map file is a good indicator of just how bad things can get. So I'd caution the addition of adding such a suite to D as something optional if at all possible. Even something minimalisitc would work - like the inclusion of just a hashtable of all the program's symbols (metadata as Craig put it above) that is *optionally* compiled in. Otherwise, we're stuck with .map/.situ support under DDL - i.e. using your .map file as a runtime dependency for the program. While this works well, and is such a dead-simple way of addressing the problem, it's also easy to get out of sync IMO. [PSA] Craig, if you're at all interested in hammering out a reflection interface (for DDL?), the DDL Community wiki there is open for your, and everyone else's abuse. http://www.dsource.org/projects/ddl/wiki/Community I have some reflection concepts, and a few other ideas, outlined under the DevNotes section as well: http://www.dsource.org/projects/ddl/wiki/DevNotes Enjoy!

Hum, it may be the case for runtime reflection, but as for compile-time reflection, it would have quite a lot of usefulness, and no performance(or other kind of) drawbacks of merely being available. I'm not saying this about compile time reflection in the sense that it would serve as substitute to runtime reflection, it wouldn't. At least not in a complete, for-all-cases sense. I think both kinds have some uses that cannot be covered by the other kind. -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 22 2006
prev sibling parent reply Josh Stern <josh_usenet phadd.net> writes:
On Thu, 14 Sep 2006 18:32:05 -0400, Pragma wrote:

 Craig Black wrote:
 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve performance when 
 accessing metadata.


 However, we could get the metadata more quickly if it was available via the 
 TypeInfo class.  This could be made possible very easily if the TypeInfo 
 class contained a reference to an IReflection interface.  This reference 
 could be set by the reflection database at program startup.  This would 
 provide a direct link to meta data for each class, and would make reflection 
 a first class citizen in D.

The only reason why I'd consider full-on reflection for D a bad move is that it'll bloat program sizes considerably. This is kind of contrary to D's goal as a systems programming language. Taking into consideration the sheer size of your average .map file is a good indicator of just how bad things can get.

Different people undoubtably have contrasting ideas about what they'd like to use reflection for. In my mind, the most important application of Java's reflection interface is the automation of object serialization. Since I/O programming tends to be one of the most time consuming and error-prone aspects of development, any sort of powerful automation in this area is valuable. I'm a newbie to D, but as far as I can tell from the docs, the info to do something comparable isn't available in the binary files or the compile time interface.
 So I'd caution the addition of adding such a suite to D as something 
 optional if at all possible.  Even something minimalisitc would work - 
 like the inclusion of just a hashtable of all the program's symbols 
 (metadata as Craig put it above) that is *optionally* compiled in.

It could be supported without bloat by making the info either something which is either statically accessed or else dynamically loaded from a supplemental database file. Opening up useful interfaces to the language parser and using them in separate tools (i.e. like Open C++) is another possible approach to compile time reflection.
Sep 26 2006
parent reply Pragma <ericanderton yahoo.removeme.com> writes:
Josh Stern wrote:
 On Thu, 14 Sep 2006 18:32:05 -0400, Pragma wrote:
 
 Craig Black wrote:
 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve performance when 
 accessing metadata.


 However, we could get the metadata more quickly if it was available via the 
 TypeInfo class.  This could be made possible very easily if the TypeInfo 
 class contained a reference to an IReflection interface.  This reference 
 could be set by the reflection database at program startup.  This would 
 provide a direct link to meta data for each class, and would make reflection 
 a first class citizen in D.

that it'll bloat program sizes considerably. This is kind of contrary to D's goal as a systems programming language. Taking into consideration the sheer size of your average .map file is a good indicator of just how bad things can get.

Different people undoubtably have contrasting ideas about what they'd like to use reflection for. In my mind, the most important application of Java's reflection interface is the automation of object serialization. Since I/O programming tends to be one of the most time consuming and error-prone aspects of development, any sort of powerful automation in this area is valuable. I'm a newbie to D, but as far as I can tell from the docs, the info to do something comparable isn't available in the binary files or the compile time interface.

I agree with you completely, Josh. D really needs both a runtime and a compile-time reflection facility to suit all modes of use. On the compile-time side, there have been some rather amazing efforts to use templates to provide traits and other info - but there's obviously some stuff missing from TypeInfo.
 
 So I'd caution the addition of adding such a suite to D as something 
 optional if at all possible.  Even something minimalisitc would work - 
 like the inclusion of just a hashtable of all the program's symbols 
 (metadata as Craig put it above) that is *optionally* compiled in.

It could be supported without bloat by making the info either something which is either statically accessed or else dynamically loaded from a supplemental database file. Opening up useful interfaces to the language parser and using them in separate tools (i.e. like Open C++) is another possible approach to compile time reflection.

FWIW, I have the "dynamically loaded from a supplemental database file" part done. DDL just uses the map file: http://www.dsource.org/projects/ddl/wiki/Tutorial/UsingDDL/Reflect //reflect.d module reflect; import ddl.DefaultRegistry; import mango.io.Stdout; void test(){ Stdout.put("Hello DDL World"c).put(CR); } void main(){ auto registry = new DefaultRegistry(); auto inSitu = registry.load("reflect.map"); auto testFn = inSitu.getDExport!(void function(),"reflect.test")(); testFn(); } It's not as pretty as Java's reflection suite, but IMO, it's only a few wrapper classes away from that level of utility. -- - EricAnderton at yahoo
Sep 27 2006
parent reply user domain.invalid writes:
 
 FWIW, I have the "dynamically loaded from a supplemental database file" 
 part done.  DDL just uses the map file:
 
 http://www.dsource.org/projects/ddl/wiki/Tutorial/UsingDDL/Reflect
 
 //reflect.d
 module reflect;
 import ddl.DefaultRegistry;
 import mango.io.Stdout;
 
 void test(){
     Stdout.put("Hello DDL World"c).put(CR);
 }
 
 void main(){
     auto registry = new DefaultRegistry();
     auto inSitu = registry.load("reflect.map");
 
     auto testFn = inSitu.getDExport!(void function(),"reflect.test")();
     testFn();
 }
 
 It's not as pretty as Java's reflection suite, but IMO, it's only a few 
 wrapper classes away from that level of utility.
 

Isn't this dynamic binding rather than reflection? I think of reflection as the ability to find what the fields and methods are for an object. For example, with Java reflection has the following Class c = someObject.getClass() c.getFields(); c.getMethods(); Does DDL have the ability to lookup methods and fields? For example, how would you determine "reflect.test" without knowing it beforehand? Bradley
Sep 28 2006
parent Pragma <ericanderton yahoo.removeme.com> writes:
user domain.invalid wrote:
 FWIW, I have the "dynamically loaded from a supplemental database 
 file" part done.  DDL just uses the map file:

 http://www.dsource.org/projects/ddl/wiki/Tutorial/UsingDDL/Reflect

 //reflect.d
 module reflect;
 import ddl.DefaultRegistry;
 import mango.io.Stdout;

 void test(){
     Stdout.put("Hello DDL World"c).put(CR);
 }

 void main(){
     auto registry = new DefaultRegistry();
     auto inSitu = registry.load("reflect.map");

     auto testFn = inSitu.getDExport!(void function(),"reflect.test")();
     testFn();
 }

 It's not as pretty as Java's reflection suite, but IMO, it's only a 
 few wrapper classes away from that level of utility.

Isn't this dynamic binding rather than reflection? I think of reflection as the ability to find what the fields and methods are for an object. For example, with Java reflection has the following Class c = someObject.getClass() c.getFields(); c.getMethods();

Ahh, I see what you mean. The example I cited was really more along the lines of binding, but it does employ a crude form of reflection to get to the binding step (searches through metadata for a kind of symbol).
 
 
 Does DDL have the ability to lookup methods and fields? 
 For example, how 
 would you determine "reflect.test" without knowing it beforehand?

Yes and no. It does have the ability to lookup methods and fields, but at present, telling the difference from what you do and don't want is still very much DIY: auto inSitu = registry.load("reflect.map"); foreach(module; insitu.getModules){ foreach(symbol; module.getSymbols){ // each symbol.name is a mangled D symbol // that can be inspected for it's type and namespace // symbol.address provides where it's actually located // (ddl.Demangle.d can handle some aspects of this) } } ExporSymbols in the above cover just about everything: ModuleInfo, Class Definitions, ctors, dtors, functions, fields, methods and even vtables, initalizers, asserts and the rest of the zoo of D symbols exposed in .obj files. So you can look for all the metadata you need, without having to bind to a given symbol. Granted, it still feels like grocery shopping out of the "scratched and dented" bin. Having a proper facade over this is a much needed, next step. -- - EricAnderton at yahoo
Sep 28 2006