www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Runtime reflection idea

reply "Adam D. Ruppe" <destructionator gmail.com> writes:
The compiler instantiates a template called RTInfo!T for every 
custom type T that is made available through the TypeInfo class 
(.classinfo, typeid()).

The first idea with this was to do precise GC, but it can do more 
than just that. Currently, it is totally unused in druntime. I 
propose we put it to use to allow some runtime reflection.


First, here's a taste of what it can look like (or whatever else, 
I did name/value strings since that's easy to write for an 
example and access at runtime, even without phobos, but there's 
no reason we couldn't try something else, maybe a class or 
something would be good):


 CustomTypeInfoExtension("name", "value")
 CustomTypeInfoExtension("name2", "value2")
class Test { }

int main(string[] args) {
         auto rtInfo = typeid(Test).rtInfo();

         import std.stdio;
         foreach(ci; rtInfo.customInfo) {
                 writeln(ci.name, " = ", ci.value);
         }
}

// prints:
// name = value
// name2 = value2



Here's some example code I wrote up as a proof of concept in my 
custom druntime:

struct MoreTypeInfo {
     hash_t hash;
     string stringOf;
   /* whatever else we want, this is extensible by druntime */

     // and finally the custom info
     CustomTypeInfoExtension[] customInfo;
}

struct CustomTypeInfoExtension {
    // we can do this however too, I just do name/value pairs
    // as something simple to get us started
         string name;
         string value;
         string meta;
}

// fetch the data from a type using UDA capabilities
CustomTypeInfoExtension[] getCustomInfo(T)() {
        CustomTypeInfoExtension[] ext;
        foreach(attr; __traits(getAttributes, T))
               static if(is(typeof(attr) == 
CustomTypeInfoExtension))
                      ext ~= attr;
         return ext;
}

// and the thing dmd calls:
template RTInfo(T) {
         __gshared static immutable minfo =
             // the first few args are just to show druntime still
             // has special access... and the last one is to fetch
             // the custom stuff
             MoreTypeInfo(typehash!T, T.stringof, getCustomInfo!T);
         enum RTInfo = &minfo;
}






Since the template is already instantiated by the compiler - this 
exists today and works if you wanna modify your object.d in 
druntime - we don't have to do like a mixin registerModule to 
parse this UDA. Of course, you still have that option and it 
rocks, this just gives a simple runtime thing we can use out of 
the box....

and ModuleInfo even has a list of all classes, and all modules in 
a program, so we could go pretty nuts with runtime stuff here.



What do you all think? It is a pretty minor change to druntime to 
enable this kind of thing, it wouldn't get in the way of anything 
else - druntime can still use RTInfo for its own thing when the 
time comes without a problem -  and it seems to me like it could 
be easily useful and convenient for some tasks.
May 30 2013
next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 30 May 2013 10:59:52 -0400, Adam D. Ruppe  
<destructionator gmail.com> wrote:

 The compiler instantiates a template called RTInfo!T for every custom  
 type T that is made available through the TypeInfo class (.classinfo,  
 typeid()).

 The first idea with this was to do precise GC, but it can do more than  
 just that. Currently, it is totally unused in druntime. I propose we put  
 it to use to allow some runtime reflection.

http://forum.dlang.org/post/op.wcvlmg03eav7ka localhost.localdomain
 and ModuleInfo even has a list of all classes, and all modules in a  
 program, so we could go pretty nuts with runtime stuff here.

I also want ModuleInfo to have an equivalent RTInfo member. Walter agreed at the conference that it was a good idea and should be done. See bug report here: http://d.puremagic.com/issues/show_bug.cgi?id=10023
 What do you all think? It is a pretty minor change to druntime to enable  
 this kind of thing, it wouldn't get in the way of anything else -  
 druntime can still use RTInfo for its own thing when the time comes  
 without a problem -  and it seems to me like it could be easily useful  
 and convenient for some tasks.

I would like to see the necessary framework for RTInfo to allow expansion WITHOUT having to modify object.d. I think this can be done, and then it allows anyone to insert whatever they want for RTInfo based on their own needs. This would set D apart from existing runtime information frameworks, like Java's and C#'s, in that it's expandable, and it's selective (enabling runtime information might be as simple as adding UDAs). -Steve
May 30 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 30 May 2013 at 15:12:57 UTC, Steven Schveighoffer 
wrote:
 http://forum.dlang.org/post/op.wcvlmg03eav7ka localhost.localdomain

I probably read that thread originally too....
 I would like to see the necessary framework for RTInfo to allow 
 expansion WITHOUT having to modify object.d.  I think this can 
 be done, and then it allows anyone to insert whatever they want 
 for RTInfo based on their own needs.

What's the difference between what you're thinking and what I outlined here? I used strings, but it could just as well be interfaces (casted back to a custom class at use time), or something to allow custom types to be added and fetched too.
May 30 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 30 May 2013 13:10:35 -0400, Adam D. Ruppe  
<destructionator gmail.com> wrote:

 I would like to see the necessary framework for RTInfo to allow  
 expansion WITHOUT having to modify object.d.  I think this can be done,  
 and then it allows anyone to insert whatever they want for RTInfo based  
 on their own needs.

What's the difference between what you're thinking and what I outlined here? I used strings, but it could just as well be interfaces (casted back to a custom class at use time), or something to allow custom types to be added and fetched too.

If you wanted to add interfaces, would you have to modify object.d? What I'm saying is, make RTInfo extendable without having to modify object.d. Each project could define how the compiler generates the RTInfo for it's specific needs, and everyone can use the same runtime. I think your idea can do this if we just move MoreTypeInfo to be instantiated via a type member, so each type could decide how it's type info is generated. -Steve
May 30 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 30 May 2013 at 17:24:15 UTC, Steven Schveighoffer 
wrote:
 If you wanted to add interfaces, would you have to modify 
 object.d?

Yeah, but that's not something you'd have to do since you can just subclass it and add whatever you want there. class MyThing: Reflectable { blah whatever; } CustomTypeInfoExtension(new MyThing()) class Test {} auto info = typeid(Test).rtInfo(); foreach(item; info.customItems) if(auto c = cast(MyThing) item) { // use c } Or maybe we could implement it like: MyThing c = typeid(Test).rtInfo().getCustomItem!MyThing(); which could perhaps be used with any type. You could also get compile time info on it easily enough: class MyThing(T) : Reflectable {} CustomTypeInfoExtension(new MyThing!Test()) class Test {}
May 30 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 30 May 2013 at 15:12:57 UTC, Steven Schveighoffer 
wrote:
 I also want ModuleInfo to have an equivalent RTInfo member.  
 Walter agreed at the conference that it was a good idea and 
 should be done.  See bug report here:

oh man, I just had a realization: such could be (ab?)used to add new language checks. I'm playing with a bare metal D right now, and there's a few rules I've determined to keep the memory situation sane, including "don't use non-scoped, non-refcounted delegates". An rtInfo type thing run over every module in the program could actually use traits to find any function that breaks the rules and throw a static assert error. It wouldn't just be a documented warning anymore, it'd be like expanding the compiler's own semantic checks. That's kinda awesome.
May 30 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
I got a better thing implemented, what do you think of this?

// usage

 CustomInfo!("hey") // can be anything in there, but don't repeat 
a type
 CustomInfo!(1)
class Test {}

auto info = typeid(Test).rtInfo();

immutable(string)* str = rtInfo.getCustomData!string;
writeln(str is null ? "not there" : *str); // prints "hey"

immutable(int)* i = rtInfo.getCustomData!int;
writeln(i is null ? "not there" : *i); // prints 1.


// implementation

	struct MoreTypeInfo {
                 // the first two fields just show druntime can 
still do its thing
		hash_t hash;
		string stringOf;

                 // and here's the extensible part
		immutable(CustomTypeInfoExtension)[] customInfo;

		immutable(T)* getCustomData(T)() immutable {
			auto hash = typeid(T); // typehash!T;
			foreach(ci; customInfo) {
				if(ci.typeOfData == hash)
					return cast(immutable(T)*) ci.data();
			}
			return null;
		}
	}

	struct CustomTypeInfoExtension {
		TypeInfo typeOfData;
		void* function() data;
	}

	immutable(CustomTypeInfoExtension)[] getCustomInfo(T)() {
		if(__ctfe) {
			//bool[hash_t] seen;
			immutable(CustomTypeInfoExtension)[] ext;
			foreach(attr; __traits(getAttributes, T))
				static if(is(typeof(attr) == CustomTypeInfoExtension)) {
					//auto hash = attr.typeOfData.rtInfo.hash;
					//if(hash in seen)
						//assert(0, "repeated data");
					//seen[hash] = true;
					ext ~= cast(immutable) attr;
				}
			return ext;
		} else return null;
	}


	template CustomInfo(alias T) {
		__gshared static data = T;
		void* getRaw() { return cast(void*) &data; }
		enum CustomInfo = CustomTypeInfoExtension( 
typeid(typeof(data)), &getRaw);
	}

	template RTInfo(T) {
		__gshared static immutable minfo = MoreTypeInfo(typehash!T, 
T.stringof, getCustomInfo!T);
		enum RTInfo = &minfo;
	}
Jun 01 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
also this post:

http://forum.dlang.org/thread/tftjtzmfuauxwcgcolct forum.dlang.org#post-hautysdywtyfigsoasgv:40forum.dlang.org

I'm just playing with my custom druntime and there's so much we 
could add if we wanted, pretty easily too thanks to ctfe.
Jun 01 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
I think this is a very good idea. The only question is how complete this 
RTTI should be. This again highly depends on what it will be used for. 
For some users it might be ok to do a full RTTI, which will increase the 
executable size significantly. Other users might prefer a minimal RTTI 
or even no RTTI at all depending on the use case.

The RTInfo template is very usefull for many different tasks, the 
question is if we shouldn't make the concept more generic so that you 
can have multiple templates which behave like the RTInfo template. I'm 
currently also using it for RTTI info see: http://3d.benjamin-thaut.de/?p=25

-- 
Kind Regards
Benjamin Thaut
Jun 02 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 02.06.2013 11:33, Benjamin Thaut wrote:
 I think this is a very good idea. The only question is how complete this
 RTTI should be. This again highly depends on what it will be used for.
 For some users it might be ok to do a full RTTI, which will increase the
 executable size significantly. Other users might prefer a minimal RTTI
 or even no RTTI at all depending on the use case.

 The RTInfo template is very usefull for many different tasks, the
 question is if we shouldn't make the concept more generic so that you
 can have multiple templates which behave like the RTInfo template. I'm
 currently also using it for RTTI info see:
 http://3d.benjamin-thaut.de/?p=25

As a number of use cases show up for the RTInfo template (the precise GC also uses it), I think a simple approach for not stepping on each other toes would be to declare a struct type RTInfoData, and add each implementation as a member to this struct (not checked whether this actually compiles): struct RTInfoData { immutable(RuntimeReflection)* rr; immutable(thMemberInfo)* th; immutable(PreciseGCData)* gc; } template RTInfo(T) { immutable(RTInfoData) data = RTInfoData(genRuntimeReflection!T, genMemberInfo!T, genGCData!T); enum RTInfo = &data; } and TypeInfo.rtInfo() would then return a pointer to RTInfoData instead of void*. This doesn't make it modifiable from outside object.di, but I have no idea how that could be possible to begin with (without recompiling the runtime library). Unfortunately the compiler sometimes doesn't force the generation of RTInfo, but sets the m_rtInfo to 0 or 1 depending on whether the Type contains pointers or not. I always wanted to figure out why that happens...
Jun 02 2013
next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 02.06.2013 15:28, schrieb Rainer Schuetze:
 On 02.06.2013 11:33, Benjamin Thaut wrote:
 I think this is a very good idea. The only question is how complete this
 RTTI should be. This again highly depends on what it will be used for.
 For some users it might be ok to do a full RTTI, which will increase the
 executable size significantly. Other users might prefer a minimal RTTI
 or even no RTTI at all depending on the use case.

 The RTInfo template is very usefull for many different tasks, the
 question is if we shouldn't make the concept more generic so that you
 can have multiple templates which behave like the RTInfo template. I'm
 currently also using it for RTTI info see:
 http://3d.benjamin-thaut.de/?p=25

As a number of use cases show up for the RTInfo template (the precise GC also uses it), I think a simple approach for not stepping on each other toes would be to declare a struct type RTInfoData, and add each implementation as a member to this struct (not checked whether this actually compiles): struct RTInfoData { immutable(RuntimeReflection)* rr; immutable(thMemberInfo)* th; immutable(PreciseGCData)* gc; } template RTInfo(T) { immutable(RTInfoData) data = RTInfoData(genRuntimeReflection!T, genMemberInfo!T, genGCData!T); enum RTInfo = &data; } and TypeInfo.rtInfo() would then return a pointer to RTInfoData instead of void*. This doesn't make it modifiable from outside object.di, but I have no idea how that could be possible to begin with (without recompiling the runtime library). Unfortunately the compiler sometimes doesn't force the generation of RTInfo, but sets the m_rtInfo to 0 or 1 depending on whether the Type contains pointers or not. I always wanted to figure out why that happens...

That is the obvious solution to the problem. Maybe we should get something like this into drutime ASAP so that the RTInfo template remains extendable even if a percise GC is used. Kind Regards Benjamin Thaut
Jun 02 2013
prev sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 02.06.2013 17:10, Adam D. Ruppe wrote:
 On Sunday, 2 June 2013 at 13:28:28 UTC, Rainer Schuetze wrote:
 and TypeInfo.rtInfo() would then return a pointer to RTInfoData
 instead of void*. This doesn't make it modifiable from outside
 object.di, but I have no idea how that could be possible to begin with
 (without recompiling the runtime library).

I think I have something figured out:

Interesting stuff to implement custom type info declared with the class. But type info as needed for the GC has to be generated for every Type without additional annotation or declaration. Assuming a recompilation of the runtime library, but no modification to the library source, I currently can only imagine having some string import switched to some other file through a different search path. This import would then contain declarations to be mixed into the RTInfoData (or MoreRtInfo in your implementation).
Jun 03 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 2 June 2013 at 09:33:27 UTC, Benjamin Thaut wrote:
 I think this is a very good idea. The only question is how 
 complete this RTTI should be.

For now, I'm going for just a UDA based thing, so it is almost all opt-in. Though one downside of my idea here is it may be slow, since it pulls info from a list (I'm doing a linear array, but it could be a binary search of AA or something without much difficulty, but my guess is most entries will only be a handful of elements long anyway). Actually adding members to the rtinfo struct would avoid this cost, but I can't think of any way to do that without modifying object.d for each one. Perhaps we could have a userRtInfo template that works the same way, but that one is defined in the module that defines the type, instead of in object.d. The module would be responsible for casting the void* in typeinfo (a new method, urtInfo to complement rtInfo) back to the type they defined. I think that would be doable at least. The other thing I'm pondering is changing TypeInfo to be generated at compile time and include all kinds of functions like std.traits offers. This would only be builtin types, user structs and classes have RTInfo instead, so that limits the size somewhat, but it nevertheless does increase exe size and isn't easy to opt out of, since you'd have to recompile at least part of druntime to do it. Expanding typeinfo is arguably less useful since if you want that info, you can do it yourself in a template like std.variant, but one use I can see is the D style variadics. Instead of a list of types with code right there, you could just do typeid().is/getNumeric() or whatever.
 The RTInfo template is very usefull for many different tasks, 
 the question is if we shouldn't make the concept more generic 
 so that you can have multiple templates which behave like the 
 RTInfo template. I'm currently also using it for RTTI info see: 
 http://3d.benjamin-thaut.de/?p=25

very cool.
Jun 02 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 2 June 2013 at 13:28:28 UTC, Rainer Schuetze wrote:
 and TypeInfo.rtInfo() would then return a pointer to RTInfoData 
 instead of void*. This doesn't make it modifiable from outside 
 object.di, but I have no idea how that could be possible to 
 begin with (without recompiling the runtime library).

I think I have something figured out: use side: // i'll use this as a UDA struct InfoStruct { string info; } // and this as the template extension struct MoreRtInfo { string amazing; } CustomInfo!(InfoStruct("info struct data")) // UDA style, goes into an array class RTTest { // template style, goes in a special member static template userRtInfo(This) { static __gshared i = MoreRtInfo("amazingness"); enum userRtInfo = &i; } } Retrieve: auto info = typeid(RTTest).rtInfo(); if(info is null) throw new Exception("wtf"); auto urtInfo = info.userRtInfo; // this is from the template.. if(urtInfo is null) write("urtInfo == null\n"); else { // it comes out as a void*, so we have to cast it back. auto urt = cast(immutable(MoreRtInfo)*) urtInfo; write("cool ",urt.amazing,"\n"); } // and this is fetching the CustomInfo UDA auto cd = info.getCustomInfo!InfoStruct; if(cd is null) // it returns a InfoStruct*, like AA's in operator write("InfoStruct is null\n"); else write(cd.info, "\n"); And the implementation side in object.d: struct MoreTypeInfo { // other stuff druntime is free to declare hash_t hash; string stringOf; // holder for the template thing immutable(void)* userRtInfo; // holder for the UDA thing immutable(CustomTypeInfoExtension)[] customInfo; // helper for the UDA get immutable(T)* getCustomInfo(T)() immutable { auto hash = typeid(T); // typehash!T; foreach(ci; customInfo) { if(ci.typeOfData == hash) return cast(immutable(T)*) ci.data(); } return null; } } // instantiates the T.userRtInfo, if possible template urtInfo(T) { static if (__traits(compiles, { auto a = cast(immutable(void)*) T.userRtInfo!T; })) enum urtInfo = cast(immutable(void)*) T.userRtInfo!T; else enum urtInfo = null; } template RTInfo(T) { __gshared static immutable minfo = MoreTypeInfo( // stuff druntime is free to add typehash!T, T.stringof, // the user defined template urtInfo!T, // getting the UDA stuff getCustomInfoInternal!T); enum RTInfo = &minfo; } // finally, helpers in getting the UDAs, I posted this before too struct CustomTypeInfoExtension { TypeInfo typeOfData; // ctfe complained when I tried to just do &static_data // so instead this little function helper does it for us void* function() data; } immutable(CustomTypeInfoExtension)[] getCustomInfoInternal(T)() { if(__ctfe) { //bool[hash_t] seen; immutable(CustomTypeInfoExtension)[] ext; foreach(attr; __traits(getAttributes, T)) static if(is(typeof(attr) == CustomTypeInfoExtension)) { //auto hash = attr.typeOfData.rtInfo.hash; //if(hash in seen) //assert(0, "repeated data"); //seen[hash] = true; ext ~= cast(immutable) attr; } return ext; } else return null; } // this is the user uses: CustomInfo!(something) template CustomInfo(alias T) { __gshared static data = T; void* getRaw() { return cast(void*) &data; } enum CustomInfo = CustomTypeInfoExtension( typeid(typeof(data))/*typehash!(typeof(data))*/, &getRaw); }
 Unfortunately the compiler sometimes doesn't force the 
 generation of RTInfo, but sets the m_rtInfo to 0 or 1 depending 
 on whether the Type contains pointers or not. I always wanted 
 to figure out why that happens...

weird
Jun 02 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 30 May 2013 at 15:12:57 UTC, Steven Schveighoffer 
wrote:
 I would like to see the necessary framework for RTInfo to allow 
 expansion WITHOUT having to modify object.d.  I think this can 
 be done, and then it allows anyone to insert whatever they want 
 for RTInfo based on their own needs.

Eureka: http://forum.dlang.org/thread/zgjgfprbkeiapjipldwf forum.dlang.org#post-fnxfhvhvmhuweyxfricb:40forum.dlang.org
Jun 21 2013
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 21 Jun 2013 08:43:15 -0400, Adam D. Ruppe  
<destructionator gmail.com> wrote:

 On Thursday, 30 May 2013 at 15:12:57 UTC, Steven Schveighoffer wrote:
 I would like to see the necessary framework for RTInfo to allow  
 expansion WITHOUT having to modify object.d.  I think this can be done,  
 and then it allows anyone to insert whatever they want for RTInfo based  
 on their own needs.

Eureka: http://forum.dlang.org/thread/zgjgfprbkeiapjipldwf forum.dlang.org#post-fnxfhvhvmhuweyxfricb:40forum.dlang.org

Nice :) -Steve
Jun 21 2013