www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.reflection prototype

reply "bitwise" <bitwise.pvt gmail.com> writes:
I came across this post a while back and decided to implement it:
http://forum.dlang.org/thread/juf7sk$16rl$1 digitalmars.com

My implementation:
https://github.com/bitwise-github/D-Reflection

The above conversation seemed to stop abruptly, so I went on to 
assume that no one chose to champion the task.

At the time, I looked around for other conversations or attempts 
at runtime reflection for D, but couldn't really find anything. I 
did find the ModuleInfo/reflection stuff in object.d, but it 
seemed like an effort that may have been abandoned. Also, the 
above conversation seemed to suggest it should be opt-in, which 
also made me wonder if the stuff in object.d was abandoned or for 
a different purpose.

Looking again today, someone seems to have been working on it a 
bit.. For example, MemberInfo_field and MemberInfo_function were 
empty last time I checked.

So what's the current state of things?
Is anybody working on it?

Although MemberInfo_field exists, it doesn't seem to be available 
from TypeInfo_Class... Is that the eventual goal?

Is there anything I can do to help get things moving?


Any comments on my implementation would be welcome as well.
(https://github.com/bitwise-github/D-Reflection)

main.d shows some general use cases, but basically, if the 
reflect(T) template is used on a class,
that class, and any child types will be reflected. Recursive 
reflection only propagates downward, or else it could leak 
sideways and unnecessarily reflect several modules.

Most of the reflection information is available at compile time. 
For example:

enum name = 
reflectModule!(test).findClass("Test").findField("variable").name;
pragma(msg, name); // "variable" will be outputted.

To make a module available for runtime reflection, the following 
can be used:
mixin(runtimeReflection!test);

At this point, the above example can be rewritten as:

string name = 
getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name;
writeln(name);
Mar 29 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 30/03/2015 2:11 p.m., bitwise wrote:
 I came across this post a while back and decided to implement it:
 http://forum.dlang.org/thread/juf7sk$16rl$1 digitalmars.com

 My implementation:
 https://github.com/bitwise-github/D-Reflection

 The above conversation seemed to stop abruptly, so I went on to assume
 that no one chose to champion the task.

 At the time, I looked around for other conversations or attempts at
 runtime reflection for D, but couldn't really find anything. I did find
 the ModuleInfo/reflection stuff in object.d, but it seemed like an
 effort that may have been abandoned. Also, the above conversation seemed
 to suggest it should be opt-in, which also made me wonder if the stuff
 in object.d was abandoned or for a different purpose.

 Looking again today, someone seems to have been working on it a bit..
 For example, MemberInfo_field and MemberInfo_function were empty last
 time I checked.

 So what's the current state of things?
 Is anybody working on it?

 Although MemberInfo_field exists, it doesn't seem to be available from
 TypeInfo_Class... Is that the eventual goal?

 Is there anything I can do to help get things moving?


 Any comments on my implementation would be welcome as well.
 (https://github.com/bitwise-github/D-Reflection)

 main.d shows some general use cases, but basically, if the reflect(T)
 template is used on a class,
 that class, and any child types will be reflected. Recursive reflection
 only propagates downward, or else it could leak sideways and
 unnecessarily reflect several modules.

 Most of the reflection information is available at compile time. For
 example:

 enum name =
 reflectModule!(test).findClass("Test").findField("variable").name;
 pragma(msg, name); // "variable" will be outputted.

 To make a module available for runtime reflection, the following can be
 used:
 mixin(runtimeReflection!test);

 At this point, the above example can be rewritten as:

 string name =
 getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name;

 writeln(name);
I love the idea of it. But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors!
Mar 29 2015
next sibling parent reply "bitwise" <bitwise.pvt gmail.com> writes:
 But first we need to finish off what support we damn well have 
 in druntime.

 m_offTi for TypeInfo_Class currently is not being generated.

 We also need some form of RTInfo for modules. Not just for 
 symbols. This alone will open up quite a few doors!
In the "time for std.reflection" convo, it seemed like Andre was talking about a library solution, and not something compiler generated.. has that idea since been thrown out? Has there been other conversations on this that I missed? Thanks
Mar 29 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 30/03/2015 5:39 p.m., bitwise wrote:
 But first we need to finish off what support we damn well have in
 druntime.

 m_offTi for TypeInfo_Class currently is not being generated.

 We also need some form of RTInfo for modules. Not just for symbols.
 This alone will open up quite a few doors!
In the "time for std.reflection" convo, it seemed like Andre was talking about a library solution, and not something compiler generated.. has that idea since been thrown out?
This definitely should be a library solution. But it seems silly, to not use code in druntime. Either that, or if it is like m_offTi, remove it. After all, it was never implemented.
 Has there been other conversations on this that I missed?
No.
Mar 29 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
 This definitely should be a library solution.
In terms of what Andre was suggesting, I think my implementation is 90% of the way there, so if the will was there to add something like std.reflection to phobos, it wouldn't be much of a leap.
 But it seems silly, to not use code in druntime.
I kind of agree. If I had things my way, all reflection info for all classes would be compiler generated so that it was guaranteed to be there for tools, interop and such. If people were really worried about code bloat, the reflection info could simply be turned off with a -no-reflection flag or something. In terms of playing around with the compiler and the type system, I'm not sure how helpful I could be right now. I'm pretty sure comprehension will not be a problem, but I've got quite a bit of reading to do before I'm up to speed with whats going on with dmd and the type system.
Mar 29 2015
next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 30/03/2015 6:06 p.m., bitwise wrote:
 This definitely should be a library solution.
In terms of what Andre was suggesting, I think my implementation is 90% of the way there, so if the will was there to add something like std.reflection to phobos, it wouldn't be much of a leap.
 But it seems silly, to not use code in druntime.
I kind of agree. If I had things my way, all reflection info for all classes would be compiler generated so that it was guaranteed to be there for tools, interop and such. If people were really worried about code bloat, the reflection info could simply be turned off with a -no-reflection flag or something. In terms of playing around with the compiler and the type system, I'm not sure how helpful I could be right now. I'm pretty sure comprehension will not be a problem, but I've got quite a bit of reading to do before I'm up to speed with whats going on with dmd and the type system.
I have so many views upon all this, thanks to my new (not released as not ready) web service framework. But druntime right now is not ready for proper reflection. There needs to be some modifications to it and dmd. But mostly druntime for RTInfo. For example what shouldn't needed to be done. Here is my install information for my autoconfig system. (Allows for publically importing any module I want AND handling any symbol that the system has used. #Install ## Configure per type RTInfo requires a single modification. [Based upon a forum thread](http://forum.dlang.org/post/majnjuhxdefjuqjlpbmv forum.dlang.org). ```D template RTInfo(T, string moduleName = __MODULE__) { enum RTInfo = cast(void*)0x12345678; static if (__traits(compiles, {import core.config;})) { import core.config; alias checkResult = TypeCheck!(T, moduleName); } } ``` ## Auto imports COPY/core/configimports.d needs to be copied to import/core/configimports.d Also needed is object.di under imports to be modified. Add ``D public import core.configimports;`` anywhere you want.
Mar 29 2015
prev sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Monday, 30 March 2015 at 05:06:14 UTC, bitwise wrote:
 This definitely should be a library solution.
In terms of what Andre was suggesting, I think my implementation is 90% of the way there, so if the will was there to add something like std.reflection to phobos, it wouldn't be much of a leap.
 But it seems silly, to not use code in druntime.
I kind of agree. If I had things my way, all reflection info for all classes would be compiler generated so that it was guaranteed to be there for tools, interop and such. If people were really worried about code bloat, the reflection info could simply be turned off with a -no-reflection flag or something. In terms of playing around with the compiler and the type system, I'm not sure how helpful I could be right now. I'm pretty sure comprehension will not be a problem, but I've got quite a bit of reading to do before I'm up to speed with whats going on with dmd and the type system.
When I tried mine (https://shardsoft.com/stash/projects/SHARD/repos/shardtools/browse/source/Shard ools/Reflection.d), which was admittedly a while back, the template bloat was actually a significant problem. On Linux and OSX, the build for just the unittest library would generate an over 40MB executable. On Windows with Optlink though, it was only 4MB, so perhaps there are optimizations that could be made to reduce the size on OSX / Linux. Either way though, that's way too much bloat for something in druntime, and I don't think it's necessary. While it might be nice to use RTInfo to generate reflection data for your whole program, most people would be perfectly okay with at most a few modules, possibly recursively. Also, from what I can tell you generate a class for every symbol. This could generate a significant amount of bloat as well because it would generate further TypeInfo for each of these classes. I could be wrong here, but if that's the case, perhaps changing to structs would be a better choice if possible?
Mar 29 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
 On Linux and OSX, the build for just the unittest library would 
 generate an over 40MB executable. On Windows with Optlink 
 though, it was only 4MB, so perhaps there are optimizations 
 that could be made to reduce the size on OSX / Linux.
This sounds like a bug.. if it CAN be done with an executable size of 4MB, why would it not be so on OSX/Linux? Also, i was reading through this pull request: https://github.com/D-Programming-Language/druntime/pull/775 and yglukhov mentioned something about needing 5 minutes and 12GB or ram to build phobos with whatever they're talking about. To be honest though, after reading the above pull-comments and this one below, I'm still confused about what they're trying to do. https://github.com/D-Programming-Language/dmd/pull/2271
 Also, from what I can tell you generate a class for every 
 symbol. This could generate a significant amount of bloat as 
 well because it would generate further TypeInfo for each of 
 these classes. I could be wrong here, but if that's the case, 
 perhaps changing to structs would be a better choice if 
 possible?
You're right that I am creating a class per symbol, along with a bunch of other templates. It probably would be a good idea to try and eliminate the templates where possible. I'm not sure why you are suggesting structs instead of classes though.
Mar 30 2015
next sibling parent reply "bitwise" <bitwise.pvt gmail.com> writes:
https://github.com/D-Programming-Language/druntime/pull/775
https://github.com/D-Programming-Language/dmd/pull/2271

I've read over the comments of the above two pull-requests, and 
I'm confused about what is trying to be achieved.

A ModuleInfo for each module already exists and is available at 
runtime. Each ModuleInfo has the method localClasses() to get all 
it's classes. Why not just add an array of MethodInfo and 
FieldInfo to TypeInfo_Class?

It seems like the above two pulls are trying to add some kind of 
custom per-module info. For my purposes, some very basic info for 
suffice.

At the very minimum, if TypeInfo_Class could be completed to 
contain these, I would be happy:

// one per field
class FieldInfo
{
     string name();
     TypeInfo type();
     void *getAddress(void *instance);
}

// one per overload
class MethodInfo
{
     string name();
     TypeInfo returnType();
     TypeInfo[] parameterTypes();
     bool isProperty();
     bool isStatic();
     void *getAddress(void *instance);
}

The above two would be more than enough to do serialization and 
runtime modification of unknown objects.

What am I missing here?
Mar 30 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-03-30 23:31, bitwise wrote:
 https://github.com/D-Programming-Language/druntime/pull/775
 https://github.com/D-Programming-Language/dmd/pull/2271

 I've read over the comments of the above two pull-requests, and I'm
 confused about what is trying to be achieved.
Currently in object.d in druntime there's a template, RTInfo, declared something like this: template RTInfo (T) { enum RTInfo = null; } The compiler will automatically instantiate this template for all user defined types. What the template evaluates to will be placed in the TypeInfo for that given type in the "rtInfo" property. This allows two things: * Add custom type checking on user defined types: template RTInfo (T) { static if (T.stringof == "Foo") static assert(false, "A type named 'Foo' is not allowed"); enum RTInfo = null; } * Add custom type info for a given type which is accessible at runtime: template RTInfo (T) { enum RTInfo = T.stringof; } The original reason why this was added to the language was to extract type info necessary to build a precise garbage collector. The problem with this is to use this feature you need to modify object.d in druntime. The first pull request tries to come up with a solution that doesn't require modifying druntime. The second pull request implemented the same thing as RTInfo but for modules instead of types. So the compiler would automatically instantiate this template with for each module it encounters during compilation. The main reason for implementing RTInfo for modules is to implement reflection. With the template instantiated for each module you could scan the module for class and methods and build up reflection data completely transparently without the user needing to register any types or similar. -- /Jacob Carlborg
Mar 31 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
 Currently in object.d in druntime there's
 a template, RTInfo, declared something
 like this:
 [snip]
Thanks for the breakdown! Those pull requests make a lot more sense when read in the proper context ;)
 The main reason for implementing RTInfo
 for modules is to implement reflection.
 With the template instantiated for each
 module you could scan the module for class and methods and 
 build up reflection
 data completely transparently without
 the user needing to register any types
 or similar.
If I'm understanding correctly, doing it this way is to avoid making changes to the compiler, right? I don't understand this decision because it seems that most of the needed infrastructure is already built into ModuleInfo, and that it just needs to be completed. It would eliminate the problem of template/code bloat from a library like mine, and at the same time, would not require the user to register any types. The downside would be susceptibility to things like .NET Reflector, making binaries very easy to crack/decompile, and of course, the extra space it took for the metadata. But, the generation of the metadata could be made optional via command line switch. Metadata could either be disabled completely, or partially with something like --no-verbose-rtti, which could omit all the human readable portions of the metadata, while leaving behind whatever was needed for precise GC or whatever else may need it. Of course, the inverse would also be an option, to omit all type info and enable it manually via --verbose-rtti. Failing all of the above, I suppose my reflection library does the job well enough for my needs. Some work could be done to optimize the binary size, but feature-wise, I find it more than complete enough. However, I am thinking about making it more versatile in terms of what is generated, because that seems to be a common request. Something like this might make sense: module test; class Test1{} Reflected class Test2{} NotReflected class Test3{} /////////////////// module reflection; enum ReflectionMode { Inclusive, // include all classes unless specified Exclusive // exclude all classes unless specified } template moduleReflection(alias mod, ReflectionMode m) { ... } /////////////////// module test; mixin(moduleReflection!(test, ReflectionMode.Inclusive)); /////////////////// ReflectionMode.Inclusive would generate reflection for Test1 and Test2, ReflectionMode.Exclusive would generate reflection for only Test2. I'm not sure it's possible to make recursiveness of reflection optional without losing compile-time availability of major features though.
 This is needed for sure. For example do you
 count how many subjects on .learn are about
 __traits(allMembers,...  or __traits(getMember,...
Yeah, I agree these are confusing... along with type-tuples, and CTFE. I think all three of these are hard to grasp without knowledge of how a compiler works.
 About the design i would rather see this as
 a user choice, eg the reflexion as a mixin
 template that you can mix in a class or a struct you want to be 
 filled with info.
 This remark is related to the size problem
 pointed in the previous posts.
I would like to make things as flexible/optional as possible without being intrusive to user types. UDAs are about as far as I'm willing to go with per-type boilerplate.
Apr 01 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-02 02:28, bitwise wrote:

 If I'm understanding correctly, doing it this way is to avoid making
 changes to the compiler, right?

 I don't understand this decision because it seems that most of the
 needed infrastructure is already built into ModuleInfo, and that it just
 needs to be completed. It would eliminate the problem of template/code
 bloat from a library like mine, and at the same time, would not require
 the user to register any types.
As I said, the reason for implementing RTInfo for modules was to _not_ have to register anything. There are other good use cases for both RTInfo and RMInfo (runtime module info), they are a more generic solution. Two other threads about unit testing [1], [2] is a good use case. RMInfo can be used to collect all unit test functions and create a custom runner. Here's [3] one example where a unit test runner makes it possible to have CTFE unit tests. This proof of concept only scans the current module, here RMInfo would be really handy to scan all modules. Here's [4] one example where RTInfo is used to check virtual methods. All virtual methods are required to be marked with virtual. There's a lot of missing info in ModuleInfo and TypeInfo. For example MemberInfo_function contains no information about parameters, return types, attributes and so on. [1] http://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com [2] http://forum.dlang.org/thread/mfci6o$13oa$1 digitalmars.com [3] http://forum.dlang.org/thread/ks1brj$1l6c$1 digitalmars.com [4] http://forum.dlang.org/thread/kok86c$126l$1 digitalmars.com -- /Jacob Carlborg
Apr 01 2015
next sibling parent reply "bitwise" <bitwise.pvt gmail.com> writes:
On Thursday, 2 April 2015 at 06:57:25 UTC, Jacob Carlborg wrote:
 On 2015-04-02 02:28, bitwise wrote:

 If I'm understanding correctly, doing it this way is to avoid 
 making
 changes to the compiler, right?

 I don't understand this decision because it seems that most of 
 the
 needed infrastructure is already built into ModuleInfo, and 
 that it just
 needs to be completed. It would eliminate the problem of 
 template/code
 bloat from a library like mine, and at the same time, would 
 not require
 the user to register any types.
As I said, the reason for implementing RTInfo for modules was to _not_ have to register anything. There are other good use cases for both RTInfo and RMInfo (runtime module info), they are a more generic solution. Two other threads about unit testing [1], [2] is a good use case. RMInfo can be used to collect all unit test functions and create a custom runner. Here's [3] one example where a unit test runner makes it possible to have CTFE unit tests. This proof of concept only scans the current module, here RMInfo would be really handy to scan all modules. Here's [4] one example where RTInfo is used to check virtual methods. All virtual methods are required to be marked with virtual. There's a lot of missing info in ModuleInfo and TypeInfo. For example MemberInfo_function contains no information about parameters, return types, attributes and so on. [1] http://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com [2] http://forum.dlang.org/thread/mfci6o$13oa$1 digitalmars.com [3] http://forum.dlang.org/thread/ks1brj$1l6c$1 digitalmars.com [4] http://forum.dlang.org/thread/kok86c$126l$1 digitalmars.com
Ok, I think I understand what you're suggesting now, which is that you want a library to be able to override RTInfo in order to add it's own metadata to all types, which raises the question, what if more than one library wants to add metadata? And I think this question was addressed by the suggestion of the AA for RTInfo where the module's fully qualified name was the key, which won't work because of separate compilation... right? So in my case I could just update my RTInfo to generate a reflection for each type, and make it accessible using a UFC or something. If I understand correctly though, the idea has been dismissed as being impossible due to separate compilation, right? Is there any leads on this at this point?
Apr 04 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-04-04 18:16, bitwise wrote:

 Ok, I think I understand what you're suggesting now, which is that you
 want a library to be able to override RTInfo in order to add it's own
 metadata to all types, which raises the question, what if more than one
 library wants to add metadata? And I think this question was addressed
 by the suggestion of the AA for RTInfo where the module's fully
 qualified name was the key, which won't work because of separate
 compilation... right?
Yes, exactly. I don't know if it works with separate complication or not. But it will make it more complicated.
 So in my case I could just update my RTInfo to generate a reflection for
 each type, and make it accessible using a UFC or something.

 If I understand correctly though, the idea has been dismissed as being
 impossible due to separate compilation, right? Is there any leads on
 this at this point?
I don't think it's impossible, but more it's complicated. I don't think I got a really good elaborated answer form Martin why he didn't like the AA approach, except for being complicated. -- /Jacob Carlborg
Apr 05 2015
prev sibling parent reply "bitwise" <bitwise.pvt gmail.com> writes:
One more question:

Does anyone know why TypeInfo_class.getMembers() was removed? [1]

I found an old post saying that it never worked and returned an 
empty array, so that is most likely the answer, but although 
getMembers was removed, Walter seems to have left behind the 
classes MemberInfo, MemberInfo_field, and MemberInfo_function, so 
I'm guessing there is still hope of having them implemented one 
day. [2]

[1] 
https://github.com/D-Programming-Language/druntime/commit/f957b4ef7222bae5da7f3b4104ab42061dd41319#diff-7eac7eb46e31907f148813e793155274

[2] 
https://github.com/D-Programming-Language/druntime/blob/bd3f4cb2122edd1a7c107f86936c20bba15191b6/src/object.di#L276
Apr 04 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-04-04 21:49, bitwise wrote:
 One more question:

 Does anyone know why TypeInfo_class.getMembers() was removed? [1]

 I found an old post saying that it never worked and returned an empty
 array, so that is most likely the answer
Yes.
 but although getMembers was
 removed, Walter seems to have left behind the classes MemberInfo,
 MemberInfo_field, and MemberInfo_function, so I'm guessing there is
 still hope of having them implemented one day. [2]
I don't know why they were not removed. -- /Jacob Carlborg
Apr 05 2015
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Mon, 30 Mar 2015 21:13:48 +0000, bitwise wrote:

 On Linux and OSX, the build for just the unittest library would
 generate an over 40MB executable. On Windows with Optlink though, it
 was only 4MB, so perhaps there are optimizations that could be made to
 reduce the size on OSX / Linux.
=20 This sounds like a bug.. if it CAN be done with an executable size of 4MB, why would it not be so on OSX/Linux?
'cause DMD on posix doesn't strip binaries by default. it's funny to=20 scare windows people.=
Mar 30 2015
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-03-30 04:09, Rikki Cattermole wrote:

 We also need some form of RTInfo for modules.
I made a pull request for that but unfortunately it hasn't been accepted yet. The reason seems to be that we need to come up with a way to do custom RTInfo without modifying druntime, that can also be used for this template. -- /Jacob Carlborg
Mar 29 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 30/03/2015 7:59 p.m., Jacob Carlborg wrote:
 On 2015-03-30 04:09, Rikki Cattermole wrote:

 We also need some form of RTInfo for modules.
I made a pull request for that but unfortunately it hasn't been accepted yet. The reason seems to be that we need to come up with a way to do custom RTInfo without modifying druntime, that can also be used for this template.
You, me and Walter should have a chat then. I could pretty easily come up with a way to add data into RTInfo.
Mar 30 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-03-30 09:06, Rikki Cattermole wrote:

 You, me and Walter should have a chat then. I could pretty easily come
 up with a way to add data into RTInfo.
I've already come up with a way, any template with the rtInfo UDA is treated the same way as RTInfo is now. The problem is then how to store the data in TypeInfo. Since it would be possible to have multiple data generated for a given type I was thinking it could be stored in an associative array. The keys would be the name of the module which generated the data and the the values would be the data. Something like this: module foo.bar; rtInfo template Foo (T) { enum Foo = "bar"; } assert(typeid(T).rtInfo["foo.bar"] == "bar"); If I recall correctly Martin Nowak didn't like this approach. The associate array would need to be built at load time of the application due to separate compilation. BTW, here [1] is the pull request and the reason why it was closed. [1] https://github.com/D-Programming-Language/dmd/pull/2271#issuecomment-59621060 -- /Jacob Carlborg
Mar 30 2015
prev sibling parent reply "bitwise" <bitwise.pvt gmail.com> writes:
 I love the idea of it.
 But first we need to finish off what support we damn well have 
 in druntime.

 m_offTi for TypeInfo_Class currently is not being generated.

 We also need some form of RTInfo for modules. Not just for 
 symbols. This alone will open up quite a few doors!
Ok, I'm starting to understand this idea a little better. Is anyone currently working on generating m_offTi?
Apr 04 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 5/04/2015 4:41 a.m., bitwise wrote:
 I love the idea of it.
 But first we need to finish off what support we damn well have in
 druntime.

 m_offTi for TypeInfo_Class currently is not being generated.

 We also need some form of RTInfo for modules. Not just for symbols.
 This alone will open up quite a few doors!
Ok, I'm starting to understand this idea a little better. Is anyone currently working on generating m_offTi?
I don't think so. I looked into it, and it looks to be pretty low hanging and easy to do. I'm just not setup *grumbles*.
Apr 04 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
 I don't think so. I looked into it, and it looks to be pretty 
 low hanging and easy to do.

 I'm just not setup *grumbles*.
I'm trying to get it figured out right now. It looks like this will load all the static data for me: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_win32.d#L53 I'm guessing the TypeInfo will be part of the regular data segment. I'm a little confused why the ModuleInfo has it's own special segment.. Would that have to be done for all the MemberInfo too? Anyways, as far as I can tell, druntime would not have to be altered to implement offTi. Looking at the compiler now, I'm trying to figure out where the TypeInfo would be outputted. It seems like this is it: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L1013 I'm pretty sure this function here is converting the typeinfo to a linked list of data that gets outputted to the object file: [code] TypeInfo_toDt(&s->Sdt, tid); [/code] https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L1027 But, inside that call, I find this: [code] TypeInfoDtVisitor v(pdt); d->accept(&v); [/code] https://github.com/D-Programming-Language/dmd/blob/master/src/typinf.c#L602 which finally leads me to... void TypeInfoDtVisitor::visit(TypeInfoClassDeclaration *d) { //printf("TypeInfoClassDeclaration::toDt() %s\n", tinfo->toChars()); assert(0); } which is not helpful. Anyways, I'll keep looking, but help is welcome :)
Apr 05 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
 I'm pretty sure this function here is converting the typeinfo to
 a linked list of data that gets outputted to the object file:
 ......
Scratch that, I think this is what I'm looking for: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L308
Apr 05 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 6/04/2015 6:08 a.m., bitwise wrote:
 I'm pretty sure this function here is converting the typeinfo to
 a linked list of data that gets outputted to the object file:
 ......
Scratch that, I think this is what I'm looking for: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L308
It most certainly is.
Apr 05 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
 It most certainly is.
I've continued to dig and try to reason about the history behind this problem. This code has actually been around since D1, but was never completed: https://github.com/D-Programming-Language/dmd/blob/master/src/dsymbol.c#L1237 I think finishing up getMembers() rather than trying to output OffsetTypeInfo would be a better idea. Thoughts?
Apr 06 2015
parent reply "rcorre" <ryan rcorre.net> writes:
For me, having a solid  reflection library like this is one of 
the most important improvements D can make right now.

After releasing jsonizer (a json serializer) I started talking to 
the authors of PainlessJson (another json serializer that was 
released around the same time) about how there was a substantial 
amount of overlap between our implementations, a large portion of 
which was related to reflection rather than actually handling 
json. While I won't speak for painlessjson, I can say that the 
techniques I used in jsonizer got pretty hacky -- there's a whole 
lot of __traits and one monstrous mixin template.

For the most part, this looks really nice. Thanks for putting it 
together, bitwise!
That being said, there are a few limitiations I see to this 
implementation:

How can you get the user-defined attributes of a given ScopeRefl?
It would be awesome to say:
foreach(field ; reflect!Foo.fields) {
      static if (field.hasAttribute(...)) { ... }
}
  I guess the difficulty here is that a user-defined attribute in 
D can be pretty much anything from a primitive type to a struct 
instance to some arbitrary symbol.

How can you get the static type of a reflected field? For example:
alias T = reflect!Foo.fields[0].field_type
doSomethingWithType!T

This is available in FieldRefImpl, but disappears on FieldRefl so 
it can be stuck into an array with a bunch of other FieldRefl, 
and I'm not sure if there is a way of getting that information 
back (other than resorting back to __traits on the user-end, 
which kind of defeats the purpose).

If I understand correctly, more work needs to be done in druntime 
to better support reflection, but I've read this thread about 3 
times and I'm still having a hard time figuring out whats going 
on :)

If these changes were made to druntime, would they obviate the 
need for a reflection library, or simply make it easier to 
implement a reflection library? Is more help needed? If so, where 
should I start looking? I'm not familiar with druntime at all, 
but wouldn't mind trying to make sense of it.
Apr 13 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Mon, 13 Apr 2015 08:57:26 -0400, rcorre <ryan rcorre.net> wrote:

 For me, having a solid  reflection library like this is one of the most  
 important improvements D can make right now.
At this point, I've kinda hit a wall. Generating a hierarchical/object-oriented representation of the type system at compile-time using templates seems to be an all-or-nothing procedure. Below is an example of the base() method of ClassRefl: const(ClassRefl) base() const { static if(!is(T == Object)) { alias BaseClassesTuple!T base_type; return reflect!(base_type[0]); } } The method returns "reflect!(base_type[0])", which, when instantiated, will create another ClassRefl for the base class. That ClassRefl will also have a base() method, which will instantiate more reflections/templates, and so on... I also had a parent() method at one point, but since a module can be the parent of a class, it was possible that reflecting a single class using reflect!T could build entire reflection hierarchies for several modules. As soon as you reflect anything right now, all dependent reflections must also be instantiated. The amount of bloat(compiled binary size), and the increase in compile time would most likely be a deal breaker for a lot of people. My approach could probably be optimized, but I'm not sure it would be enough to make it usable. I'm out of ideas right now, but open to suggestions if you have a solution to the above problem.
 How can you get the static type of a reflected field? For example:
 alias T = reflect!Foo.fields[0].field_type
 doSomethingWithType!T
I don't believe an answer exists with my current design.
 How can you get the user-defined attributes of a given ScopeRefl?
I don't think this is possible either with the current design. The current design must also work at runtime, and since UDAs can be any symbol, I'm not sure there is a good way to generalize this. I could wrap all attributes in an AttributeRefl class, but there wouldn't be much to tell about the wrapped attribute beside it's type, and maybe a string representation of it(more bloat). I'm sure that more could be done to complement __traits and std.traits, but I don't think that includes using my reflection library in it's current state.
 I can say that the techniques I used in jsonizer got pretty hacky --  
 there's a whole lot of __traits and one monstrous
 mixin template.
You may want to look at "Orange". https://github.com/jacob-carlborg/orange The API is clean, and non-invasive. The only downside for me, is that in order to serialize a class by a base pointer, you have to register it at compile time.
 If I understand correctly, more work needs to be done in druntime to  
 better support reflection, but I've read this thread about 3 times and  
 I'm still having a hard time figuring out whats going on :)
The RTInfo(T) template would, in theory, automatically generate some custom metadata for every type. The compiler automatically instantiates the RTInfo(T) template for each type right now. The idea was suggested that a programmer could create their own RTInfo(T) that the compiler would use that instead. This option was thrown out because it doesn't work with separate compilation. Right now, RTInfo(T) is defined in object.d, which is implicitly imported into every module that you compile, but if you had your own RTInfo defined in your own module, it wouldn't be available to any other modules at compile time, nor would it be available to any third party libraries you may be using. The OffsetTypeInfo thing would be compiler generated. However, the oldest version of DMD I can find on github (2009) says this: // offTi[] dtdword(&dt, 0); dtdword(&dt, 0); // null for now, fix later So I'm not sure why it was never implemented. I may eventually attempt a pull request for the above fix.
 For the most part, this looks really nice. Thanks for putting it  
 together, bitwise!
Thanks ;)
Apr 13 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-14 07:03, bitwise wrote:

 You may want to look at "Orange".
 https://github.com/jacob-carlborg/orange

 The API is clean, and non-invasive. The only downside for me, is that in
 order to serialize a class by a base pointer, you have to register it at
 compile time.
Yeah, this will require runtime reflection to avoid the registering of the type. -- /Jacob Carlborg
Apr 14 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Tue, 14 Apr 2015 04:16:16 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2015-04-14 07:03, bitwise wrote:

 You may want to look at "Orange".
 https://github.com/jacob-carlborg/orange

 The API is clean, and non-invasive. The only downside for me, is that in
 order to serialize a class by a base pointer, you have to register it at
 compile time.
Yeah, this will require runtime reflection to avoid the registering of the type.
This is one of the main reasons I like the OffsetTypeInfo/offTi() option. I think a "name" field would have to be added to OffsetTypeInfo for it to be useful, and I have a feeling that would increase the binary size by quite a bit.. So I'll have to test it and see.
Apr 14 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-14 18:33, bitwise wrote:

 This is one of the main reasons I like the OffsetTypeInfo/offTi()
 option. I think a "name" field would have to be added to OffsetTypeInfo
 for it to be useful, and I have a feeling that would increase the binary
 size by quite a bit.. So I'll have to test it and see.
It needs to be possible to set and get a value of an instance variable based on it's name, through runtime reflection. It also needs to bypass protection, i.e. "private". -- /Jacob Carlborg
Apr 15 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Wed, 15 Apr 2015 05:31:24 -0400, Jacob Carlborg <doob me.com> wrote:

 It needs to be possible to set and get a value of an instance variable  
 based on it's name, through runtime reflection. It also needs to bypass  
 protection, i.e. "private".
Right now, this is the def: /** * Array of pairs giving the offset and type information for each * member in an aggregate. */ struct OffsetTypeInfo { size_t offset; /// Offset of member from start of object TypeInfo ti; /// TypeInfo for this member } If "string name" esd added, and then offTi[] esd actually populated, then I suppose you could do this: class Test { int a = 4; private int b = 5; void print(){ writeln(b); } } void main() { Test test = new Test; // offsetof would instead come from the TypeInfo/OffsetTypeInfo int* b = cast(int*)(cast(void*)test + Test.b.offsetof); *b = 1234; test.print(); } But AFAIK, this is NOT ok in C++ because of the way inheritance works.. is this safe in D?
Apr 15 2015
next sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Wed, 15 Apr 2015 11:26:48 -0400, bitwise <bitwise.pvt gmail.com> wrote:

 But AFAIK, this is NOT ok in C++ because of the way inheritance works..  
 is this safe in D?
To clarify, I'm talking about doing doing things by Object pointer or void pointer, which seems to work fine: class Test { int a = 4; private int b = 5; void print(){ writeln(b); } } struct Test2 { int a = 4; private int b = 5; void print(){ writeln(b); } } void main() { Test test = new Test; Object obj = test; int* b = cast(int*)(cast(void*)obj + Test.b.offsetof); *b = 1234; test.print(); Test2 test2 = Test2(); void *obj = &test2; int* b = cast(int*)(obj + Test2.b.offsetof); *b = 1234; test2.print(); }
Apr 15 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-15 17:26, bitwise wrote:

 Right now, this is the def:

 /**
   * Array of pairs giving the offset and type information for each
   * member in an aggregate.
   */
 struct OffsetTypeInfo
 {
      size_t   offset;    /// Offset of member from start of object
      TypeInfo ti;        /// TypeInfo for this member
 }

 If "string name" esd added, and then offTi[] esd actually populated,
 then I suppose you could do this:

 class Test {
      int a = 4;
      private int b = 5;
      void print(){ writeln(b); }
 }

 void main()
 {
      Test test = new Test;
      // offsetof would instead come from the TypeInfo/OffsetTypeInfo
      int* b = cast(int*)(cast(void*)test + Test.b.offsetof);
      *b = 1234;
      test.print();
 }

 But AFAIK, this is NOT ok in C++ because of the way inheritance works..
 is this safe in D?
I'm not sure, I would assume so. Why is this not safe in C++? -- /Jacob Carlborg
Apr 15 2015
next sibling parent reply bitwise <bitwise.pvt gmail.com> writes:
On Wed, 15 Apr 2015 15:48:28 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2015-04-15 17:26, bitwise wrote:

 Right now, this is the def:

 /**
   * Array of pairs giving the offset and type information for each
   * member in an aggregate.
   */
 struct OffsetTypeInfo
 {
      size_t   offset;    /// Offset of member from start of object
      TypeInfo ti;        /// TypeInfo for this member
 }

 If "string name" esd added, and then offTi[] esd actually populated,
 then I suppose you could do this:

 class Test {
      int a = 4;
      private int b = 5;
      void print(){ writeln(b); }
 }

 void main()
 {
      Test test = new Test;
      // offsetof would instead come from the TypeInfo/OffsetTypeInfo
      int* b = cast(int*)(cast(void*)test + Test.b.offsetof);
      *b = 1234;
      test.print();
 }

 But AFAIK, this is NOT ok in C++ because of the way inheritance works..
 is this safe in D?
I'm not sure, I would assume so. Why is this not safe in C++?
One reason is that casting with multiple inheritance offsets the pointer, and I forget exactly virtual inheritance works, but I'm sure it breaks things too.. #include <iostream> using namespace std; class A { public: int a; }; class B { public: int b; }; class C { public: int c; }; class D : public A, public B, public C { public: int d; }; int main(int argc, const char * argv[]) { D *d = new D; cout << (intptr_t)d << endl; cout << (intptr_t)(C*)d << endl; cout << (intptr_t)(B*)d << endl; cout << (intptr_t)(A*)d << endl; return 0; } possible output: 1098416 1098424 1098420 1098416
Apr 15 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-16 01:32, bitwise wrote:

 One reason is that casting with multiple inheritance offsets the
 pointer, and I forget exactly virtual inheritance works, but I'm sure it
 breaks things too..
In D I would assume this would eventually be possible: class Foo { int a; } class Bar : Foo { int b; } Foo f = new Bar; auto offset = typeid(f).getMemeber("b").offset; auto ptr = cast(int*)(cast(void*)f + offset); *ptr = 4; -- /Jacob Carlborg
Apr 15 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thu, 16 Apr 2015 02:22:25 -0400, Jacob Carlborg <doob me.com> wrote:

 In D I would assume this would eventually be possible:

 class Foo { int a; }
 class Bar : Foo { int b; }

 Foo f = new Bar;

 auto offset = typeid(f).getMemeber("b").offset;
 auto ptr = cast(int*)(cast(void*)f + offset);
 *ptr = 4;
Ok, that sounds right. D has no multiple or virtual inheritance, so I guess things should be fine. C# is the same way(single inheritance, interfaces) which is likely designed to avoid these kinds of issues. I would be modifying the offTi() property though, not getMembers(). getMembers() actually works right now, although it's kinda useless. Basically, if you put any function named "getMembers" at module scope, the address of that function can be retrieved using ModuleInfo.xgetMembers(). /////////////// module mainMod; template isField(alias T) { enum hasInit = is(typeof(typeof(T).init)); enum isManifestConst = __traits(compiles, { enum e = T; }); enum isField = hasInit && !isManifestConst; } MemberInfo[] getMembers() { mixin("alias " ~ .stringof[7..$] ~ " mod;"); MemberInfo[] members; foreach(memberName; __traits(allMembers, mod)) { alias Identity!(__traits(getMember, mod, memberName)) member; static if(isField!member) members ~= new MemberInfo_field(memberName, typeid(member), 0); } return members; } int a = 1; int b = 2; int c = 3; void main() { foreach(mi; ModuleInfo) { if(mi.name == "mainModule") { alias MemberInfo[] function() getMembers_type; getMembers_type getMembers = cast(getMembers_type)mi.xgetMembers(); foreach(m; getMembers()) writeln(m.name); } } } Output: a b c //////////////// offTi() however, even though it's documented, returns an empty array. I'm not sure what the motivation originally was for having an array of TypeInfo and offsets without the field names, but the dead code predates D2. I suppose you could do binary serialization... Anyways, I've put the reflection library to rest for now. Hopefully I'll have some time in the next little while to try and get offTi() working.
Apr 16 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-17 01:25, bitwise wrote:

 Ok, that sounds right. D has no multiple or virtual inheritance, so I
 guess things should be fine. C# is the same way(single inheritance,
 interfaces) which is likely designed to avoid these kinds of issues.

 I would be modifying the offTi() property though, not getMembers().
 getMembers() actually works right now, although it's kinda useless.
 Basically, if you put any function named "getMembers" at module scope,
 the address of that function can be retrieved using
 ModuleInfo.xgetMembers().
We need a "getMembers" on ClassInfo. Because "typeid" will resolve the dynamic type. The problem is this code: class Bar {} class Foo : Bar {} void main () { Bar a = new Foo; writeln(typeid(a)); } Compile time reflection cannot be used to get the members in this case. -- /Jacob Carlborg
Apr 17 2015
parent bitwise <bitwise.pvt gmail.com> writes:
On Fri, 17 Apr 2015 05:30:04 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2015-04-17 01:25, bitwise wrote:

 Ok, that sounds right. D has no multiple or virtual inheritance, so I
 guess things should be fine. C# is the same way(single inheritance,
 interfaces) which is likely designed to avoid these kinds of issues.

 I would be modifying the offTi() property though, not getMembers().
 getMembers() actually works right now, although it's kinda useless.
 Basically, if you put any function named "getMembers" at module scope,
 the address of that function can be retrieved using
 ModuleInfo.xgetMembers().
We need a "getMembers" on ClassInfo. Because "typeid" will resolve the dynamic type. The problem is this code: class Bar {} class Foo : Bar {} void main () { Bar a = new Foo; writeln(typeid(a)); } Compile time reflection cannot be used to get the members in this case.
Sorry, that's what I meant(using this on classes, not modules). xgetMembers/getMembers seems to be designed so that the caller can write their own getMembers() function, whereas offTi() is designed to return an array that was automatically generated. offTi() is actually defined in TypeInfo: https://github.com/D-Programming-Language/druntime/blob/cfcf7480b2faea0af9ab6ddba8e3b0d9f05c4415/src/object_.d#L301 and overridden for TypeInfo_Class: https://github.com/D-Programming-Language/druntime/blob/cfcf7480b2faea0af9ab6ddba8e3b0d9f05c4415/src/object_.d#L831 but TypeInfo_Class.m_offTi is never populated: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L436 So while this code technically works, there will be no output: class Test { int a = 4; int b = 5; int c = 6; } void main() { foreach(const(OffsetTypeInfo) off; typeid(Test).offTi()) writeln(off.offset); }
Apr 17 2015
prev sibling parent bitwise <bitwise.pvt gmail.com> writes:
One more thing.

I had to add a virtual destructor to A to get this to work:

class A {
     public: int a; virtual ~A(){}
};

// ....

A *d2 = (A*)d;
cout << boolalpha << (typeid(*d2) == typeid(D)) << endl;

C *d3 = (C*)d;
cout << boolalpha << (typeid(*d3) == typeid(D)) << endl;


Output is now:

true
false

Without virtual destructor:

false
false

So you can't rely on serialization working properly using only  
TypeInfo/offsets
Apr 15 2015
prev sibling parent "Baz" <bb.temp gmx.com> writes:
On Monday, 30 March 2015 at 01:11:55 UTC, bitwise wrote:
 I came across this post a while back and decided to implement 
 it:
 http://forum.dlang.org/thread/juf7sk$16rl$1 digitalmars.com

 My implementation:
 https://github.com/bitwise-github/D-Reflection

 The above conversation seemed to stop abruptly, so I went on to 
 assume that no one chose to champion the task.

 At the time, I looked around for other conversations or 
 attempts at runtime reflection for D, but couldn't really find 
 anything. I did find the ModuleInfo/reflection stuff in 
 object.d, but it seemed like an effort that may have been 
 abandoned. Also, the above conversation seemed to suggest it 
 should be opt-in, which also made me wonder if the stuff in 
 object.d was abandoned or for a different purpose.

 Looking again today, someone seems to have been working on it a 
 bit.. For example, MemberInfo_field and MemberInfo_function 
 were empty last time I checked.

 So what's the current state of things?
 Is anybody working on it?

 Although MemberInfo_field exists, it doesn't seem to be 
 available from TypeInfo_Class... Is that the eventual goal?

 Is there anything I can do to help get things moving?


 Any comments on my implementation would be welcome as well.
 (https://github.com/bitwise-github/D-Reflection)

 main.d shows some general use cases, but basically, if the 
 reflect(T) template is used on a class,
 that class, and any child types will be reflected. Recursive 
 reflection only propagates downward, or else it could leak 
 sideways and unnecessarily reflect several modules.

 Most of the reflection information is available at compile 
 time. For example:

 enum name = 
 reflectModule!(test).findClass("Test").findField("variable").name;
 pragma(msg, name); // "variable" will be outputted.

 To make a module available for runtime reflection, the 
 following can be used:
 mixin(runtimeReflection!test);

 At this point, the above example can be rewritten as:

 string name = 
 getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name;
 writeln(name);
This is needed for sure. For example do you count how many subjects on .learn are about __traits(allMembers,... or __traits(getMember,... Run-time serialization is an example. About the design i would rather see this as a user choice, eg the reflexion as a mixin template that you can mix in a class or a struct you want to be filled with info. This remark is related to the size problem pointed in the previous posts.
Mar 31 2015