www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Time for std.reflection

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter and I discussed the idea below a long time (years) ago. Most 
likely it's also been discussed in this newsgroup a couple of times. 
Given the state of the compiler back then, back then it seemed like a 
super cool idea that's entirely realizable, it would just take time for 
the compiler to become as capable as needed. Nowadays we're in shape to 
tackle it.

Here "it" is.

Back when runtime reflection was being discussed, my response was "let's 
focus on compile-time reflection, and then we can do run-time reflection 
on demand as a library". Though this might sound sensible, I initially 
didn't have a design. Now here's what we can do.

Currently we have information about symbols as __traits(...) intrinsics 
wrapped in nice but scattered ways. Now that CTFE is good enough to 
manipulate structs and arrays thereof, we have the possibility to 
finally approach things in a nicely unified, structured way.

First, we need to prime std.reflection with a few abstractions that 
characterize entities in a D program.

class ModuleInfo {
 property:
     string name();
     ImportInfo[] imports();
     DataInfo[] data();
     FunctionInfo[] functions();
     ClassInfo[] classes();
     StructInfo[] structs(); // includes unions
     TemplateInfo[] templates();
     EnumInfo[] enums();
     bool hasStaticCtor(), hasStaticDtor(),
       hasSharedCtor(), hasSharedDtor();
}

Probably there are a few more pieces of data, but you get the idea. Then 
for each of the entities mentioned above we have a similar definition. 
For example:

enum Protection { isPublic, isPackage, isProtected, isPrivate }

class ClassInfo {
 property:
     string name();
     string baseName();
     string parentName(); // if applicable, null otherwise
     string[] interfaces();
     bool isShared();
     Protection protection();
     DataMemberInfo[] data();
     MethodInfo[] methods();
     Object defaultConstructor();
     ...
}

Then for an e.g. method declaration we'd have:

class MethodInfo {
 property:
     string name();
     bool isStatic(), isFinal(), isOverride();
     Protection protection();
     string[] parameterTypes();
     string[] parameterNames();
}

Some details may vary, e.g. some may be straight members instead of 
properties etc. (I used properties to allude to use of lazy gathering of 
information).

So so far we have a nice collection of structured data associated with 
the entities in a D program. Note how this structuring differs yet has 
similar power to the primitives in std.traits; std.traits offers 
unstructured bits of information on demand (e.g. ParameterTypeNames) 
etc. but the objects above group information together per entity 
declared. All of the above goes in std.reflection, of course.

===========

On to primitives that return such data.

Given that D can (since relatively recently) create and manipulate class 
objects during compilation too, it follows that the classes above can be 
accessed in two ways - through compile-time API and run-time API. When 
possible, the APIs may even use the same functions; some other times 
they will be necessarily different.

There are two possible approaches to discovering such information. One 
is by fetching the ModuleInfo for the whole module and navigating it. 
Another one is by using search primitives from strings.

So we should have e.g.

// inside std.reflection
ModuleInfo getModuleInfo(string moduleName);

so a CT call would go like:

// client code
static info = getModuleInfo("std.algorithm");

whereas a run-time call would be:

// client code
auto info = getModuleInfo("std.algorithm");

In the latter case, the module needs to save all needed information for 
ri, so it should plant this:

// inside std.algorithm
mixin(makeModuleInfoAvailableDynamically());

The mixin would generate all information needed and would store it for 
later dynamic use.

A search API would go like e.g.

ClassInfo getClassInfo(string className);

In this case the class name could be qualified with module information etc.

===========

With this design we unify compile-time and run-time type manipulation in 
simple ways, by defining structured information about declarations that 
can be queried during compilation or dynamically.

Please chime in with thoughts. Would someone want to pioneer this project?


Andrei
Jul 21 2012
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:
 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.
 
 Please chime in with thoughts.

Are you proposing that we replace std.traits or that std.reflection be built on top of it? Having std.reflection will be awesome for a lot of stuff - particularly the more complicated stuff - and would be a great addition to Phobos, but it seems like overkill for a lot of basic template constraints, so I wouldn't want to see it replace std.traits. - Jonathan M Davis
Jul 21 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 5:51 PM, Jonathan M Davis wrote:
 On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:
 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.

 Please chime in with thoughts.

Are you proposing that we replace std.traits or that std.reflection be built on top of it?

On top. std.reflection would be a complete reflection solution, whereas std.traits gives casual type traits as needed.
 Having std.reflection will be awesome for a lot of stuff - particularly the
more
 complicated stuff - and would be a great addition to Phobos, but it seems like
 overkill for a lot of basic template constraints, so I wouldn't want to see it
 replace std.traits.

My thoughts exactly. Andrei
Jul 21 2012
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Sat, 21 Jul 2012 14:51:30 -0700, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:

 On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:
 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.

 Please chime in with thoughts.

Are you proposing that we replace std.traits or that std.reflection be built on top of it? Having std.reflection will be awesome for a lot of stuff - particularly the more complicated stuff - and would be a great addition to Phobos, but it seems like overkill for a lot of basic template constraints, so I wouldn't want to see it replace std.traits. - Jonathan M Davis

I too would want it in a different module But my read of Andrei's proposal is that it would be. -- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
Jul 21 2012
prev sibling next sibling parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2012-07-21 21:44:51 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 Please chime in with thoughts.

Unifying runtime and compile-time reflection is a good idea, in theory at least. At least it'd make compile-time reflection easier. But when I think about the use case, unification the way you're proposing doesn't seem that great. If you need runtime reflection in one class, you probably need it on all the derived classes too. That's the real problem runtime reflection solves: it allows you to know something about the subclasses you might otherwise not know about (because your code only know the base class). So to make runtime reflection not a hassle, the thing that generates the runtime reflection info should be inherited somehow. The other thing is: you very rarely need runtime reflection of everything. Especially, why would you ever need runtime reflection for a struct, where all the reflected info is available at compile time? And why create info for all fields and methods in a class in a case where all you need to get at is a pair of serialize/unserialize functions? -- Michel Fortin michel.fortin michelf.ca http://michelf.ca/
Jul 21 2012
prev sibling next sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
I agree with most things proposed, however I am not a fan of the 
idea of mixing in runtime reflection info. Many times, you want 
reflection info from a type that is not your own, and thus I 
believe reflection should be generated by specifying a type. More 
importantly, a method should exist for recursively generating 
reflection info.

Also, I'd like to see a hierarchal approach to reflection data. 
The main advantage to std.reflection would be being able to use 
it at run-time, at which point we can't simply rely on templates, 
and instead if we want to store something we must rely on a base 
type. I think the best approach would be a hierarchal system 
where all reflection data derives from MemberInfo. My ideal API 
would like something like: 
https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/
Jul 21 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 8:16 PM, Kapps wrote:
 I agree with most things proposed, however I am not a fan of the idea of
 mixing in runtime reflection info. Many times, you want reflection info
 from a type that is not your own, and thus I believe reflection should
 be generated by specifying a type. More importantly, a method should
 exist for recursively generating reflection info.

I confess I have trouble understanding each of the sentences above.
 Also, I'd like to see a hierarchal approach to reflection data. The main
 advantage to std.reflection would be being able to use it at run-time,
 at which point we can't simply rely on templates, and instead if we want
 to store something we must rely on a base type.

At no place in the proposed approach is the use of templates required or needed.
 I think the best
 approach would be a hierarchal system where all reflection data derives
 from MemberInfo. My ideal API would like something like:
 https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/

I cringed at MemberType -> { Module, Type, Field, Method, Parameter } I was hoping to get away from slapping tagging on types just for the sake of using inheritance. Andrei
Jul 21 2012
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, July 22, 2012 05:19:39 Kapps wrote:
 What I meant was that when something requires reflection, it
 generally requires that everything it contains has reflection
 information as well. If using a mixin directly in the module
 itself, when a module does not include this mixin it will never
 get reflection information. This may or may not be desired.
 However it completely prevents many common uses of reflection,
 including serialization. So long as there is any dependency on
 any type or method that does not have reflection info, the entire
 operation would fail. This means that people will either start
 adding reflection to modules that do not need it in fear that
 someone may try to use a member of their module, or too sparsely
 thus making reflection all but unuseable for many common
 situations.

Runtime reflection _must_ be opt-in. We do _not_ want to make all types pay the cost of having it. That means that however it's done is going to require that every type that has it be marked in one way or another to enable that functionality. That's part of the cost of being a systems language. Whatever solution we come up with must take that into account. - Jonathan M Davis
Jul 21 2012
parent reply deadalnix <deadalnix gmail.com> writes:
On 22/07/2012 05:46, Jonathan M Davis wrote:
 Runtime reflection _must_ be opt-in. We do _not_ want to make all types pay the
 cost of having it. That means that however it's done is going to require that
 every type that has it be marked in one way or another to enable that
 functionality.

I don't think both are that related here. Yes, we are a system language, so this have to be opt-in. But no, this doesn't means that every type have to be marked to be reflected. I'd expect from std.reflection that it is able to reflect recursively from the marked starting point.
Jul 22 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-22 14:04, deadalnix wrote:

 I'd expect from std.reflection that it is able to reflect recursively
 from the marked starting point.

I really hope so. -- /Jacob Carlborg
Jul 22 2012
prev sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Sunday, 22 July 2012 at 03:47:16 UTC, Jonathan M Davis wrote:
 Runtime reflection _must_ be opt-in. We do _not_ want to make 
 all types pay the
 cost of having it. That means that however it's done is going 
 to require that
 every type that has it be marked in one way or another to 
 enable that
 functionality. That's part of the cost of being a systems 
 language. Whatever
 solution we come up with must take that into account.

 - Jonathan M Davis

But the whole point is that not every type has it; only those that are registered. It just happens that you can register types besides yourself, which is an absolute necessity for many common purposes of reflection. For serialization for example, the serializer / deserializer should create reflection information for all types contained in a class.
Jul 21 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-22 02:16, Kapps wrote:
 I agree with most things proposed, however I am not a fan of the idea of
 mixing in runtime reflection info. Many times, you want reflection info
 from a type that is not your own, and thus I believe reflection should
 be generated by specifying a type. More importantly, a method should
 exist for recursively generating reflection info.

I agree. -- /Jacob Carlborg
Jul 22 2012
prev sibling next sibling parent reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 21-07-2012 23:44, Andrei Alexandrescu wrote:
 Walter and I discussed the idea below a long time (years) ago. Most
 likely it's also been discussed in this newsgroup a couple of times.
 Given the state of the compiler back then, back then it seemed like a
 super cool idea that's entirely realizable, it would just take time for
 the compiler to become as capable as needed. Nowadays we're in shape to
 tackle it.

 Here "it" is.

 Back when runtime reflection was being discussed, my response was "let's
 focus on compile-time reflection, and then we can do run-time reflection
 on demand as a library". Though this might sound sensible, I initially
 didn't have a design. Now here's what we can do.

 Currently we have information about symbols as __traits(...) intrinsics
 wrapped in nice but scattered ways. Now that CTFE is good enough to
 manipulate structs and arrays thereof, we have the possibility to
 finally approach things in a nicely unified, structured way.

 First, we need to prime std.reflection with a few abstractions that
 characterize entities in a D program.

 class ModuleInfo {
  property:
      string name();
      ImportInfo[] imports();
      DataInfo[] data();
      FunctionInfo[] functions();
      ClassInfo[] classes();
      StructInfo[] structs(); // includes unions
      TemplateInfo[] templates();
      EnumInfo[] enums();
      bool hasStaticCtor(), hasStaticDtor(),
        hasSharedCtor(), hasSharedDtor();
 }

 Probably there are a few more pieces of data, but you get the idea. Then
 for each of the entities mentioned above we have a similar definition.
 For example:

 enum Protection { isPublic, isPackage, isProtected, isPrivate }

 class ClassInfo {
  property:
      string name();
      string baseName();
      string parentName(); // if applicable, null otherwise
      string[] interfaces();
      bool isShared();
      Protection protection();
      DataMemberInfo[] data();
      MethodInfo[] methods();
      Object defaultConstructor();
      ...
 }

 Then for an e.g. method declaration we'd have:

 class MethodInfo {
  property:
      string name();
      bool isStatic(), isFinal(), isOverride();
      Protection protection();
      string[] parameterTypes();
      string[] parameterNames();
 }

 Some details may vary, e.g. some may be straight members instead of
 properties etc. (I used properties to allude to use of lazy gathering of
 information).

 So so far we have a nice collection of structured data associated with
 the entities in a D program. Note how this structuring differs yet has
 similar power to the primitives in std.traits; std.traits offers
 unstructured bits of information on demand (e.g. ParameterTypeNames)
 etc. but the objects above group information together per entity
 declared. All of the above goes in std.reflection, of course.

 ===========

 On to primitives that return such data.

 Given that D can (since relatively recently) create and manipulate class
 objects during compilation too, it follows that the classes above can be
 accessed in two ways - through compile-time API and run-time API. When
 possible, the APIs may even use the same functions; some other times
 they will be necessarily different.

 There are two possible approaches to discovering such information. One
 is by fetching the ModuleInfo for the whole module and navigating it.
 Another one is by using search primitives from strings.

 So we should have e.g.

 // inside std.reflection
 ModuleInfo getModuleInfo(string moduleName);

 so a CT call would go like:

 // client code
 static info = getModuleInfo("std.algorithm");

 whereas a run-time call would be:

 // client code
 auto info = getModuleInfo("std.algorithm");

 In the latter case, the module needs to save all needed information for
 ri, so it should plant this:

 // inside std.algorithm
 mixin(makeModuleInfoAvailableDynamically());

 The mixin would generate all information needed and would store it for
 later dynamic use.

 A search API would go like e.g.

 ClassInfo getClassInfo(string className);

 In this case the class name could be qualified with module information etc.

 ===========

 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.

 Please chime in with thoughts. Would someone want to pioneer this project?


 Andrei

I'm just curious about one thing: How do you plan to reify templates (they are Turing complete after all)? -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jul 21 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 8:28 PM, Alex Rønne Petersen wrote:
 I'm just curious about one thing: How do you plan to reify templates
 (they are Turing complete after all)?

I don't know. Andrei P.S. Please don't overquote.
Jul 21 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On 21/07/2012 23:44, Andrei Alexandrescu wrote:
 Walter and I discussed the idea below a long time (years) ago. Most
 likely it's also been discussed in this newsgroup a couple of times.
 Given the state of the compiler back then, back then it seemed like a
 super cool idea that's entirely realizable, it would just take time for
 the compiler to become as capable as needed. Nowadays we're in shape to
 tackle it.

 Here "it" is.

 Back when runtime reflection was being discussed, my response was "let's
 focus on compile-time reflection, and then we can do run-time reflection
 on demand as a library". Though this might sound sensible, I initially
 didn't have a design. Now here's what we can do.

 Currently we have information about symbols as __traits(...) intrinsics
 wrapped in nice but scattered ways. Now that CTFE is good enough to
 manipulate structs and arrays thereof, we have the possibility to
 finally approach things in a nicely unified, structured way.

 First, we need to prime std.reflection with a few abstractions that
 characterize entities in a D program.

 class ModuleInfo {
  property:
 string name();
 ImportInfo[] imports();
 DataInfo[] data();
 FunctionInfo[] functions();
 ClassInfo[] classes();
 StructInfo[] structs(); // includes unions
 TemplateInfo[] templates();
 EnumInfo[] enums();
 bool hasStaticCtor(), hasStaticDtor(),
 hasSharedCtor(), hasSharedDtor();
 }

 Probably there are a few more pieces of data, but you get the idea. Then
 for each of the entities mentioned above we have a similar definition.
 For example:

 enum Protection { isPublic, isPackage, isProtected, isPrivate }

 class ClassInfo {
  property:
 string name();
 string baseName();
 string parentName(); // if applicable, null otherwise
 string[] interfaces();
 bool isShared();
 Protection protection();
 DataMemberInfo[] data();
 MethodInfo[] methods();
 Object defaultConstructor();
 ...
 }

 Then for an e.g. method declaration we'd have:

 class MethodInfo {
  property:
 string name();
 bool isStatic(), isFinal(), isOverride();
 Protection protection();
 string[] parameterTypes();
 string[] parameterNames();
 }

 Some details may vary, e.g. some may be straight members instead of
 properties etc. (I used properties to allude to use of lazy gathering of
 information).

 So so far we have a nice collection of structured data associated with
 the entities in a D program. Note how this structuring differs yet has
 similar power to the primitives in std.traits; std.traits offers
 unstructured bits of information on demand (e.g. ParameterTypeNames)
 etc. but the objects above group information together per entity
 declared. All of the above goes in std.reflection, of course.

 ===========

 On to primitives that return such data.

 Given that D can (since relatively recently) create and manipulate class
 objects during compilation too, it follows that the classes above can be
 accessed in two ways - through compile-time API and run-time API. When
 possible, the APIs may even use the same functions; some other times
 they will be necessarily different.

 There are two possible approaches to discovering such information. One
 is by fetching the ModuleInfo for the whole module and navigating it.
 Another one is by using search primitives from strings.

 So we should have e.g.

 // inside std.reflection
 ModuleInfo getModuleInfo(string moduleName);

 so a CT call would go like:

 // client code
 static info = getModuleInfo("std.algorithm");

 whereas a run-time call would be:

 // client code
 auto info = getModuleInfo("std.algorithm");

 In the latter case, the module needs to save all needed information for
 ri, so it should plant this:

 // inside std.algorithm
 mixin(makeModuleInfoAvailableDynamically());

 The mixin would generate all information needed and would store it for
 later dynamic use.

 A search API would go like e.g.

 ClassInfo getClassInfo(string className);

 In this case the class name could be qualified with module information etc.

 ===========

 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.

 Please chime in with thoughts. Would someone want to pioneer this project?


 Andrei

I don't understand. Are theses structures generated by the compiler or some kind of libraries at compile time on a per needed basis ?
Jul 21 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 8:41 PM, deadalnix wrote:
 I don't understand. Are theses structures generated by the compiler or
 some kind of libraries at compile time on a per needed basis ?

These are all library structures. Internally they use std.traits, which in turns uses compiler magic with __traits. Andrei P.S. What this thing wit quoting a long message to make a 1-line point? Is that a thing?
Jul 21 2012
next sibling parent deadalnix <deadalnix gmail.com> writes:
On 22/07/2012 06:48, Andrei Alexandrescu wrote:
 On 7/21/12 8:41 PM, deadalnix wrote:
 I don't understand. Are theses structures generated by the compiler or
 some kind of libraries at compile time on a per needed basis ?

These are all library structures. Internally they use std.traits, which in turns uses compiler magic with __traits. Andrei

Sounds good, but what about derived classes ?
 P.S. What this thing wit quoting a long message to make a 1-line point?
 Is that a thing?

It is what happen when you editor collapse the text :D
Jul 22 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-22 06:48, Andrei Alexandrescu wrote:

 P.S. What this thing wit quoting a long message to make a 1-line point?
 Is that a thing?

It's the new hip thing :) -- /Jacob Carlborg
Jul 22 2012
prev sibling next sibling parent "sclytrack" <sclytrack thailand.com> writes:
 In the latter case, the module needs to save all needed 
 information for
 ri, so it should plant this:

 // inside std.algorithm
 mixin(makeModuleInfoAvailableDynamically());


Is there a way to get the current module at compile time? __traits(allMembers, pkg.modulename)
Nov 19 2012
prev sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Monday, 19 November 2012 at 11:43:36 UTC, sclytrack wrote:

You could do something like this:
string currentModule(alias T = {})() { return __traits(parent, 
T).stringof; }
Nov 19 2012
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/21/12, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 class ModuleInfo {
  property:
      string name();
      ImportInfo[] imports();
      DataInfo[] data();
      FunctionInfo[] functions();
      ClassInfo[] classes();
      StructInfo[] structs(); // includes unions
      TemplateInfo[] templates();
      EnumInfo[] enums();
      bool hasStaticCtor(), hasStaticDtor(),
        hasSharedCtor(), hasSharedDtor();
 }

Are class/struct/function/etc templates going to be stored in the templates field? Then you'd have to tag each template with a type, e.g. "class template" vs "function template" to be able to filter them out. Otherwise classes/functions/etc could have an optional "TemplateTypeInfo[] typeParams" field so you could filter out templated from non-templated types by checking their typeParams field, e.g.: auto tempClasses = filter!(a => !empty(a.typeParams) )(modinfo.classes); I use a similar structure to what you've defined for my code generator and it worked out nicely for me.
Jul 21 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 8:46 PM, Andrej Mitrovic wrote:
 On 7/21/12, Andrei Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:
 class ModuleInfo {
  property:
       string name();
       ImportInfo[] imports();
       DataInfo[] data();
       FunctionInfo[] functions();
       ClassInfo[] classes();
       StructInfo[] structs(); // includes unions
       TemplateInfo[] templates();
       EnumInfo[] enums();
       bool hasStaticCtor(), hasStaticDtor(),
         hasSharedCtor(), hasSharedDtor();
 }

Are class/struct/function/etc templates going to be stored in the templates field? Then you'd have to tag each template with a type, e.g. "class template" vs "function template" to be able to filter them out.

Perhaps we could accommodate parameterized types together with non-parameterized types by having e.g. additional properties that are null for non-parameterized types.
 Otherwise classes/functions/etc could have an optional
 "TemplateTypeInfo[] typeParams" field so you could filter out
 templated from non-templated types by checking their typeParams field,
 e.g.: auto tempClasses = filter!(a =>  !empty(a.typeParams)
 )(modinfo.classes);

 I use a similar structure to what you've defined for my code generator
 and it worked out nicely for me.

That sounds great! Andrei
Jul 21 2012
prev sibling next sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Sunday, 22 July 2012 at 00:21:31 UTC, Andrei Alexandrescu 
wrote:
 On 7/21/12 8:16 PM, Kapps wrote:
 I agree with most things proposed, however I am not a fan of 
 the idea of
 mixing in runtime reflection info. Many times, you want 
 reflection info
 from a type that is not your own, and thus I believe 
 reflection should
 be generated by specifying a type. More importantly, a method 
 should
 exist for recursively generating reflection info.

I confess I have trouble understanding each of the sentences above.

What I meant was that when something requires reflection, it generally requires that everything it contains has reflection information as well. If using a mixin directly in the module itself, when a module does not include this mixin it will never get reflection information. This may or may not be desired. However it completely prevents many common uses of reflection, including serialization. So long as there is any dependency on any type or method that does not have reflection info, the entire operation would fail. This means that people will either start adding reflection to modules that do not need it in fear that someone may try to use a member of their module, or too sparsely thus making reflection all but unuseable for many common situations.
 Also, I'd like to see a hierarchal approach to reflection 
 data. The main
 advantage to std.reflection would be being able to use it at 
 run-time,
 at which point we can't simply rely on templates, and instead 
 if we want
 to store something we must rely on a base type.

At no place in the proposed approach is the use of templates required or needed.

What I mean by this is that people rely on the ability to use templates as a replacement to inheritance or common interfaces. For example, ranges rely on methods existing rather than using an interface for a range. With reflection, this isn't ideal. For example, if you wanted to get all the fields and properties in a module, including nested ones, you'd currently have to get all fields, in a module, add those, then get all types in a module, then all types in that type, then get all fields from those, etc. With a hierarchal approach where MemberInfo has a Children property, you'd simply get all children that are a FieldInfo or PropertyInfo, and recurse for all returned values that have children.
 I think the best
 approach would be a hierarchal system where all reflection 
 data derives
 from MemberInfo. My ideal API would like something like:
 https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/

I cringed at MemberType -> { Module, Type, Field, Method, Parameter } I was hoping to get away from slapping tagging on types just for the sake of using inheritance.

Admittedly, the MemberType wouldn't be necessary and I don't see as having a huge benefit. In almost all cases you know what you want to operate on, inheritance simply makes the API cleaner and gives access to common methods more easily.
Jul 21 2012
prev sibling next sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sat, Jul 21, 2012 at 11:44 PM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:

Nice!

 class ModuleInfo {
  property:
     string name();
     ImportInfo[] imports();
     DataInfo[] data();
     FunctionInfo[] functions();
     ClassInfo[] classes();
     StructInfo[] structs(); // includes unions
     TemplateInfo[] templates();
     EnumInfo[] enums();
     bool hasStaticCtor(), hasStaticDtor(),
       hasSharedCtor(), hasSharedDtor();
 }

A few questions: 1) About templates, would class/struct/function templates be in there? As in: class Foo(T,U) { ... } => class TemplateInfo { property: string name(); FunctionInfo[] functions(); ClassInfo[] classes(); TemplateInfo[] templates(); // Yes, templates in templates are possible, and so on, recursively ... } 2) Why classes, as opposed to structs? Would inheritance/hierarchies play a role there? Or reference semantics? Note that between structs, classes, templates and modules there is a lot of redundancy. A possibility could be to have an AggregateInfo base class. 3) For a class, a (to my eyes) standard request would be to get the parents chain, up to Object. Could that be added in ClassInfo, or as an external free function? (edit: scratch that, you're talking about lazy gathering below). 4) How would that allows queries like "Here is class C, give me all its available subclasses."? Hmm, wait, I get it: extract classes from the module, and recursively from imported modules. From these classes, extract the parent classes and so on, until the search ranged over the whole inheritance tree. I guess inheritance info could be standard enough for std.reflection to provide such a search. 5) The compiler can emit JSON output giving a partial view of the same information. As a long-term goal, I suggest these infos should be compatible somehow. The JSON output should be enriched, and we should ascertain that using std.json to read this kind of automatically-generated information should give std.reflection infos back. 6) Is really all the necessary info available through std.traits and __traits? Imported modules and their subtilities (renamed functions, etc) seem out of reach, no? 7) I know your examples are not complete, but don't forget aliases and symbols, and module-level values. Since these can be of any type, I'm not sure how they are managed in your scheme. I mean, you cannot have IntInfo[], DoubleInfo[], ... 8) (more a remark than a question) To me, it's another example of a tree-like structure. I humbly suggest we get a std.tree somewhere, to manipulate generic n-ary trees: mapping on trees, searching (extracting elements or whole subtrees), folding trees, etc. That is, a module that deals with trees/graphs not so much as containers, but as a way to represent the relationships between elements. I'm not sure I'm so clear here, but ideally std.json, std.xml, std.reflection and probably a few others should use it as a low-level basis. Btw, I have tree / trees algo (including depth-first / breadth-first ranges) / graph / graph algo (strongly connected components, search in graphs, ...) on github, but it's code that predate std.container and is in limbo, pending std.allocator. Philippe
Jul 22 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/22/12 9:39 AM, Philippe Sigaud wrote:
 1) About templates, would class/struct/function templates be in there?
 As in:

 class Foo(T,U) { ... }
 =>
 class TemplateInfo {
  property:
      string name();
      FunctionInfo[] functions();
      ClassInfo[] classes();
      TemplateInfo[] templates(); // Yes, templates in templates are
 possible, and so on, recursively
 ....
 }

Yah, ideally all entities definable in a D module should be available via reflection. But I focused on things that e.g. the user of a dynamically-loaded library would be interested in: functions, classes. std.reflection could become the lynchpin for dynamic library use; once the library is loaded (with dlopen or such), the client needs to call getModuleInfo() (a C function that can be found with dlsym()) and then get access to pointers to functions necessary for doing all other work. (Note that my examples don't yet include pointers to executable code yet.)
 2) Why classes, as opposed to structs? Would inheritance/hierarchies
 play a role there? Or reference semantics?
 Note that between structs, classes, templates and modules there is a
 lot of redundancy. A possibility could be to have an AggregateInfo
 base class.

Initially I used struct, but then I figured reference semantics are more natural for storing cross-entity information, as you indeed took advantage of in your TemplateInfo.
 3) For a class, a (to my eyes) standard request would be to get the
 parents chain, up to Object. Could that be added in ClassInfo, or as
 an external free function? (edit: scratch that, you're talking about
 lazy gathering below).

I used strings because I assumed it's easy to just query the ClassInfo given a string. But yes, we could use references to ClassInfo and InterfaceInfo etc. in a transitive manner.
 4) How would that allows queries like "Here is class C, give me all
 its available subclasses."? Hmm, wait, I get it: extract classes from
 the module, and recursively from imported modules. From these classes,
 extract the parent classes and so on, until the search ranged over the
 whole inheritance tree. I guess inheritance info could be standard
 enough for std.reflection to provide such a search.

Something like that. Note that such a query is not particularly OO-ish, because getting a class' cone (totality of subclasses) works against the modularity that inheritance is meant for. I don't think we should make getting class cones particularly easy.
 5) The compiler can emit JSON output giving a partial view of the same
 information. As a long-term goal, I suggest these infos should be
 compatible somehow. The JSON output should be enriched, and we should
 ascertain that using std.json to read this kind of
 automatically-generated information should give std.reflection infos
 back.

Yes, that's a great connection that Walter and I discussed a bit.
 6) Is really all the necessary info available through std.traits and
 __traits? Imported modules and their subtilities (renamed functions,
 etc) seem out of reach, no?

We'll need indeed to enhance __traits with what's needed. Much of the point of std.reflection is to determine exactly what's there and what's needed. And that starts with the data structures design (the algorithmic aspects are minor).
 7) I know your examples are not complete, but don't forget aliases and
 symbols, and module-level values. Since these can be of any type, I'm
 not sure how they are managed in your scheme. I mean, you cannot have
 IntInfo[], DoubleInfo[], ...

I sort of eschewed part of that by using strings for types.
 8) (more a remark than a question) To me, it's another example of a
 tree-like structure. I humbly suggest we get a std.tree somewhere, to
 manipulate generic n-ary trees: mapping on trees, searching
 (extracting elements or whole subtrees), folding trees, etc. That is,
 a module that deals with trees/graphs not so much as containers, but
 as a way to represent the relationships between elements. I'm not sure
 I'm so clear here, but ideally std.json, std.xml, std.reflection and
 probably a few others should use it as a low-level basis.
 Btw, I have tree / trees algo (including depth-first / breadth-first
 ranges) / graph / graph algo (strongly connected components, search in
 graphs, ...) on github, but it's code that predate std.container and
 is in limbo, pending std.allocator.

Well you're the resident crazy-stuff-during-compilation guy. Did you try your trees during compilation? Andrei
Jul 22 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-22 16:28, Andrei Alexandrescu wrote:

 Something like that. Note that such a query is not particularly OO-ish,
 because getting a class' cone (totality of subclasses) works against the
 modularity that inheritance is meant for. I don't think we should make
 getting class cones particularly easy.

I see no reason to make it difficult on purpose. -- /Jacob Carlborg
Jul 22 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/23/12 2:39 AM, Jacob Carlborg wrote:
 On 2012-07-22 16:28, Andrei Alexandrescu wrote:

 Something like that. Note that such a query is not particularly OO-ish,
 because getting a class' cone (totality of subclasses) works against the
 modularity that inheritance is meant for. I don't think we should make
 getting class cones particularly easy.

I see no reason to make it difficult on purpose.

The reason is that often the cone would be incomplete and therefore confusing. It would also foster non-modular approaches to doing things. Andrei
Jul 22 2012
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 23-Jul-12 10:30, Philippe Sigaud wrote:
 Well you're the resident crazy-stuff-during-compilation guy.

Ah! I wish. I had this wonderful idea of having code be parsed at CT, semantically analyzed, transformed into some machine code at CT and... , oh wait.

I kind of did it... regex pattern ---parse---> bytecode ---generate D code---> profit. BTW you can avoid machine code, just use D code as glorified backend :) (and get optimizations for free, yay!) -- Dmitry Olshansky
Jul 22 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 23-Jul-12 10:47, Philippe Sigaud wrote:
 On Mon, Jul 23, 2012 at 8:41 AM, Dmitry Olshansky <dmitry.olsh gmail.com>
wrote:
 On 23-Jul-12 10:30, Philippe Sigaud wrote:
 Well you're the resident crazy-stuff-during-compilation guy.

Ah! I wish. I had this wonderful idea of having code be parsed at CT, semantically analyzed, transformed into some machine code at CT and... , oh wait.

I kind of did it... regex pattern ---parse---> bytecode ---generate D code---> profit. BTW you can avoid machine code, just use D code as glorified backend :) (and get optimizations for free, yay!)

Hey, that was a joke, about implementing a D compiler in D, to be executed at CT by the real D compiler to basically do its job a second time, only worse :)

Yeah, now I think I got it. Could be a nice benchmark for CTFE though :)
 But yes, your regex work is a great example. I should have a look at
 the code once more.

I'd advice to wait a bit ... it's not really in a good shape (to read it) as there have been tons of improvements in CTFE since the last time I've touched it. I'll have to revisit it and remove some roundabout workarounds. -- Dmitry Olshansky
Jul 22 2012
prev sibling next sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Saturday, 21 July 2012 at 21:44:52 UTC, Andrei Alexandrescu 
wrote:

 Please chime in with thoughts. Would someone want to pioneer 
 this project?

There are some questions. 1. static info = getModuleInfo("std.algorithm"); The language does not allow you to use CTFE parameter values as arguments to __traits/templates. Therefore, to be able to build meta-objects at compile-time, you would have to: static info = getModuleInfo!"std.algorithm"; 2. Then, what is the preferred way of referencing compiler objects - with strings or directly: import std.algorithm; static info = getModuleInfo!(std.algorithm); ? I think using strings while we have direct access to those objects is not an incredibly good idea. Those strings will be mixed-in by getXxxInfo anyway: ModuleInfo getModuleInfo(string name)() { ... mixin("import " ~ name ~ ";"); mixin("alias __traits(getMembers, " ~ name ~ ") members;"); foreach (m; members) ... } 3. auto info = getModuleInfo("std.algorithm"); There is no way to mutate global data structures at compile-time, therefore you would have to build, at run time, a data structure aggregating all meta-objects coming from various modules. That could be achieved with static constructors: module std.reflection; // application-wide module info registry private ModuleInfo[string] moduleInfos; // run-time module info getter string getModuleInfo(string s) { return moduleInfos[s]; } string makeModuleInfoAvailableDynamically() { return q{ shared static this() { shared static mi = getModuleInfo!(__traits(thisModule)); moduleInfos[mi.name] = mi; } }; } Problematic because mixing-in makeModuleInfoAvailableDynamically would mean that circular imports are no longer allowed for that module. Remember the whining babies complaining about this very use case a while ago? What about changing the language so that static constructors marked with system are exempted from circular dependency checking? 3. If you are against inheritance, why classes and not structs? 4. How the whole thing is intended to interact with dynamic link libraries?
Jul 22 2012
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Sunday, 22 July 2012 at 14:28:45 UTC, Andrei Alexandrescu 
wrote:
 Yah, ideally all entities definable in a D module should be 
 available via reflection. But I focused on things that e.g. the 
 user of a dynamically-loaded library would be interested in: 
 functions, classes.

Plugins are usually done by providing factory methods that create needed objects, the methods are usually obtained with dlsym.
Jul 22 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/22/12, Philippe Sigaud <philippe.sigaud gmail.com> wrote:
 2) Why classes, as opposed to structs?

I think either way you'd need reference semantics. For example maybe you're doing code-generation at compile-time but you need to rename a class name in a typeinfo returned by std.reflection before doing any processign. With reference semantics you only have to change one class and all other types which refer to such a class will have access to the new name. You could use structs as well, and actually I use structs in my codegenerator (with a similar layout to what Andrei posted). Each struct (e.g. Class/Function) stores Symbols, which are structs with an ID and a Type. I can look up each symbol in a SymTable which actually holds the structures with data. So a Symbol is like a fake pointer, and the SymTable would be the memory. I originally planned to use classes but some serialization frameworks didn't work with those so I settled using structs and a bit of template mixin magic instead. All of this std.reflection talk is quite exciting actually. If you had AST information about your entire D library you could do some really cool things. You could make a better documentation generator than ddoc, or export the AST into a file for a code-completion plugin, or create a wrapper C library which enables other languages to use your D library.
Jul 22 2012
prev sibling next sibling parent Wouter Verhelst <wouter grep.be> writes:
"Kagamin" <spam here.lot> writes:

 On Sunday, 22 July 2012 at 14:28:45 UTC, Andrei Alexandrescu wrote:
 Yah, ideally all entities definable in a D module should be
 available via reflection. But I focused on things that e.g. the user
 of a dynamically-loaded library would be interested in: functions,
 classes.

Plugins are usually done by providing factory methods that create needed objects, the methods are usually obtained with dlsym.

That's what you'd do in a language that doesn't have something like D's Object.factory(). In D's case, however, you'd have the "factory method" be the ctor. That's all hypothetical, however, since there's no D ABI for shared objects yet... -- The volume of a pizza of thickness a and radius z can be described by the following formula: pi zz a
Jul 22 2012
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Jul 22, 2012 at 4:28 PM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:

 std.reflection could become the lynchpin for dynamic library use; once the
 library is loaded (with dlopen or such), the client needs to call
 getModuleInfo() (a C function that can be found with dlsym()) and then get
 access to pointers to functions necessary for doing all other work. (Note
 that my examples don't yet include pointers to executable code yet.)

I wouldn't know. I have no experience with dlsym().
 2) Why classes, as opposed to structs? Would inheritance/hierarchies
 play a role there? Or reference semantics?
 Note that between structs, classes, templates and modules there is a
 lot of redundancy. A possibility could be to have an AggregateInfo
 base class.

Initially I used struct, but then I figured reference semantics are more natural for storing cross-entity information, as you indeed took advantage of in your TemplateInfo.

I realized a few minutes after posting I answered my own question: because this is a self-referencing structure (a tree, a graph), which are easier to code with classes than structs.
 4) How would that allows queries like "Here is class C, give me all
 its available subclasses."? Hmm, wait, I get it: extract classes from
 the module, and recursively from imported modules. From these classes,
 extract the parent classes and so on, until the search ranged over the
 whole inheritance tree. I guess inheritance info could be standard
 enough for std.reflection to provide such a search.

Something like that. Note that such a query is not particularly OO-ish, because getting a class' cone (totality of subclasses) works against the modularity that inheritance is meant for. I don't think we should make getting class cones particularly easy.

Right. Since people here asked this question, I thought that was a common request in OO. I'm more a structs and mixins guy, myself.
 5) The compiler can emit JSON output giving a partial view of the same
 information. As a long-term goal, I suggest these infos should be
 compatible somehow. The JSON output should be enriched, and we should
 ascertain that using std.json to read this kind of
 automatically-generated information should give std.reflection infos
 back.

Yes, that's a great connection that Walter and I discussed a bit.

Good to know.
 6) Is really all the necessary info available through std.traits and
 __traits? Imported modules and their subtilities (renamed functions,
 etc) seem out of reach, no?

We'll need indeed to enhance __traits with what's needed. Much of the point of std.reflection is to determine exactly what's there and what's needed. And that starts with the data structures design (the algorithmic aspects are minor).

+1 for enhancing __traits locally. - having __traits(allMembers, xxx) work on simple module names, and not only on qualified package.module names - having a way to get imports and a way to know whether they are static / renaming import As for the data structures, other people's designs allured to in this thread seem similar to your proposal. I have two other questions: About functions: should they be subject to reflection also, or not? They have no fields, inner functions are totally interned, etc. All a user need is the 'interface', right? (name, and everything that's in the type: return type, parameters, purity, etc) About imports, what about inner imports, now that they are authorized in almost any scope? My gut feeling right now is that a user does not care if class C internally import std.algorithm in one of its methods, but I could be wrong.
 7) I know your examples are not complete, but don't forget aliases and
 symbols, and module-level values. Since these can be of any type, I'm
 not sure how they are managed in your scheme. I mean, you cannot have
 IntInfo[], DoubleInfo[], ...

I sort of eschewed part of that by using strings for types.

I see. They can be managed like fields in an aggregate (struct / classes), as there are many similarities between D modules and classes / structs. class ModuleInfo { property: ... FieldInfo[] data; // also used in StructInfo and ClassInfo } class FieldInfo { property: string name(); bool isStatic(); Protection protection(); string type(); } As long as this info is available at CT, FieldInfo.type can be mixed-in and used in code. what I'm not sure I get in your design is why some informations are encoded in their own structure, like Protection above (the code is copy-pasted from yours, I'd guess Protection is an enumeration), and then some others are encoded as strings (types). Is that because the values Protection can take are known in advance (and finite)? I wondered whether a design like this could be interesting?: abstract class FieldInfo {} class Field(T) : FieldInfo { property: string name(); bool isStatic(); Protection protection(); alias T Type; } But that doesn't cut it, AFAICT: different fields can be stored in a FieldInfo[] array, but the type information is not easier to get, anyway. So forget it. This kind of manipulation is why I got interested in fully polymorphic trees (tuples of tuples...), able to store any value, while keeping the type information visible. The drastic consequence is to have a tree type depend on its entire content. But I'm coming from Static Typing Land here, whereas this introspection stuff is more dynamic. Anyway, back to gobal values: aliases should be there also. A simple AliasInfo class?
 Well you're the resident crazy-stuff-during-compilation guy.

Ah! I wish. I had this wonderful idea of having code be parsed at CT, semantically analyzed, transformed into some machine code at CT and... , oh wait.
 Did you try your trees during compilation?

Just did. They fail :) Either segmentation fault (core dumped) or an error telling me class literals cannot be returned from CTFE. Hmm, this is old code. I'll have a look since in other projects, I can obtain trees at CT. Philippe
Jul 22 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-21 23:44, Andrei Alexandrescu wrote:
 Walter and I discussed the idea below a long time (years) ago. Most
 likely it's also been discussed in this newsgroup a couple of times.
 Given the state of the compiler back then, back then it seemed like a
 super cool idea that's entirely realizable, it would just take time for
 the compiler to become as capable as needed. Nowadays we're in shape to
 tackle it.

 Here "it" is.

 Back when runtime reflection was being discussed, my response was "let's
 focus on compile-time reflection, and then we can do run-time reflection
 on demand as a library". Though this might sound sensible, I initially
 didn't have a design. Now here's what we can do.

I've been waiting for this :) -- /Jacob Carlborg
Jul 22 2012
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Jul 22, 2012 at 5:10 PM, Max Samukha <maxsamukha gmail.com> wrote:

 The language does not allow you to use CTFE parameter values as arguments to
 __traits/templates. Therefore, to be able to build meta-objects at
 compile-time, you would have to:

 static info = getModuleInfo!"std.algorithm";

Maybe I don't get your comment, but AFAICT, the language does allow you to use CTFE parameters values as arguments to templates: template Twice(double d) { enum Twice = d * 2; } double foo(double d) { return d+1.0; } void main() { enum t = Twice!(foo(1.0)); pragma(msg, t); }
Jul 22 2012
prev sibling next sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Jul 22, 2012 at 6:15 PM, Andrej Mitrovic
<andrej.mitrovich gmail.com> wrote:
 On 7/22/12, Philippe Sigaud <philippe.sigaud gmail.com> wrote:
 2) Why classes, as opposed to structs?

I think either way you'd need reference semantics.

Yeah, I agree, reference semantics look better. And since there is a lot of duplication (struct info is a subtype of class info, itself looking very uch like a module info, I guess a mini-hierarchy is possible). So I'm convinced classes look like a good fit.
 You could use structs as well, and actually I use structs in my
 codegenerator (with a similar layout to what Andrei posted). Each
 struct (e.g. Class/Function) stores Symbols, which are structs with an
 ID and a Type. I can look up each symbol in a SymTable which actually
 holds the structures with data. So a Symbol is like a fake pointer,
 and the SymTable would be the memory. I originally planned to use
 classes but some serialization frameworks didn't work with those so I
 settled using structs and a bit of template mixin magic instead.

Could you describe your design in this thread, and possibly link to some code?
 All of this std.reflection talk is quite exciting actually.

It is!
 If you had
 AST information about your entire D library you could do some really
 cool things. You could make a better documentation generator than
 ddoc, or export the AST into a file for a code-completion plugin, or
 create a wrapper C library which enables other languages to use your D
 library.

You could do macros. 'nuff said.
Jul 22 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-23 08:35, Philippe Sigaud wrote:
 On Sun, Jul 22, 2012 at 6:15 PM, Andrej Mitrovic

 If you had
 AST information about your entire D library you could do some really
 cool things. You could make a better documentation generator than
 ddoc, or export the AST into a file for a code-completion plugin, or
 create a wrapper C library which enables other languages to use your D
 library.

You could do macros. 'nuff said.

If I recall correctly, Scala uses its reflection API as a part off its macros. -- /Jacob Carlborg
Jul 23 2012
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Mon, Jul 23, 2012 at 8:41 AM, Dmitry Olshansky <dmitry.olsh gmail.com> wrote:
 On 23-Jul-12 10:30, Philippe Sigaud wrote:
 Well you're the resident crazy-stuff-during-compilation guy.

Ah! I wish. I had this wonderful idea of having code be parsed at CT, semantically analyzed, transformed into some machine code at CT and... , oh wait.

I kind of did it... regex pattern ---parse---> bytecode ---generate D code---> profit. BTW you can avoid machine code, just use D code as glorified backend :) (and get optimizations for free, yay!)

Hey, that was a joke, about implementing a D compiler in D, to be executed at CT by the real D compiler to basically do its job a second time, only worse :) But yes, your regex work is a great example. I should have a look at the code once more.
Jul 22 2012
prev sibling next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Mon, 23 Jul 2012 08:32:37 +0200, Philippe Sigaud  
<philippe.sigaud gmail.com> wrote:

 Maybe I don't get your comment, but AFAICT, the language does allow
 you to use CTFE parameters values as arguments to templates:

 template Twice(double d)
 {
     enum Twice = d * 2;
 }

 double foo(double d)
 {
     return d+1.0;
 }

 void main()
 {
     enum t = Twice!(foo(1.0));
     pragma(msg, t);
 }

Wrong way around. Try this: template Twice(double d) { enum Twice = d * 2; } double foo(double d) { return Twice!d; } void main() { enum t = foo(1.0); pragma(msg, t); } -- Simen
Jul 23 2012
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Mon, Jul 23, 2012 at 9:46 AM, Simen Kjaeraas <simen.kjaras gmail.com> wrote:

 Wrong way around. Try this:


 template Twice(double d)
 {
     enum Twice = d * 2;
 }

 double foo(double d)
 {
     return Twice!d;
 }

 void main()
 {
     enum  t = foo(1.0);
     pragma(msg, t);
 }

Ah, that, OK. And __ctfe does not help. Would it really limit the proposed scheme for std.reflection?
Jul 23 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Sunday, 22 July 2012 at 13:39:31 UTC, Philippe Sigaud wrote:
 4) How would that allows queries like "Here is class C, give me 
 all
 its available subclasses."? Hmm, wait, I get it: extract 
 classes from
 the module, and recursively from imported modules. From these 
 classes,
 extract the parent classes and so on, until the search ranged 
 over the
 whole inheritance tree. I guess inheritance info could be 
 standard
 enough for std.reflection to provide such a search.

You can't do that without breaking the module system – as long as a class is not final (and then it doesn't make much sense to ask for its subclasses anyway), somebody can always extend it in a module completely separate from the code making the query. To illustrate what I mean, let's assume you define a class Foo in a module A. The user imports A from a second module B, and potentially extends Foo there. Now, if it was possible to get all possible subclasses in A, this would lead to information leaking from B to A, while the import graph only allows the other direction. As a consequence, all kinds of issues related to order dependence, separate compilation, etc. would arise. David
Jul 23 2012
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Mon, Jul 23, 2012 at 2:43 PM, David Nadlinger <see klickverbot.at> wrote=
:
[getting subclasses]
 You can't do that without breaking the module system =E2=80=93 as long as=

 not final (and then it doesn't make much sense to ask for its subclasses
 anyway), somebody can always extend it in a module completely separate fr=

 the code making the query.

 To illustrate what I mean, let's assume you define a class Foo in a modul=

 A. The user imports A from a second module B, and potentially extends Foo
 there. Now, if it was possible to get all possible subclasses in A, this
 would lead to information leaking from B to A, while the import graph onl=

 allows the other direction. As a consequence, all kinds of issues related=

 order dependence, separate compilation, etc. would arise.

Yes, I know. That would be a purely local question: from module A, what subclasses of class Foo do I see? Anyway, maybe I'm warped by CLOS.
Jul 23 2012
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--14dae9399b013d174c04cee2bbb4
Content-Type: text/plain; charset=UTF-8

Huge interest!
I wouldn't want to see static data structures bloat/clutter the exe though
in cases where it's not ever used/queried.
The object factory is already a serious problem.


On 22 July 2012 00:44, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 Walter and I discussed the idea below a long time (years) ago. Most likely
 it's also been discussed in this newsgroup a couple of times. Given the
 state of the compiler back then, back then it seemed like a super cool idea
 that's entirely realizable, it would just take time for the compiler to
 become as capable as needed. Nowadays we're in shape to tackle it.

 Here "it" is.

 Back when runtime reflection was being discussed, my response was "let's
 focus on compile-time reflection, and then we can do run-time reflection on
 demand as a library". Though this might sound sensible, I initially didn't
 have a design. Now here's what we can do.

 Currently we have information about symbols as __traits(...) intrinsics
 wrapped in nice but scattered ways. Now that CTFE is good enough to
 manipulate structs and arrays thereof, we have the possibility to finally
 approach things in a nicely unified, structured way.

 First, we need to prime std.reflection with a few abstractions that
 characterize entities in a D program.

 class ModuleInfo {
  property:
     string name();
     ImportInfo[] imports();
     DataInfo[] data();
     FunctionInfo[] functions();
     ClassInfo[] classes();
     StructInfo[] structs(); // includes unions
     TemplateInfo[] templates();
     EnumInfo[] enums();
     bool hasStaticCtor(), hasStaticDtor(),
       hasSharedCtor(), hasSharedDtor();
 }

 Probably there are a few more pieces of data, but you get the idea. Then
 for each of the entities mentioned above we have a similar definition. For
 example:

 enum Protection { isPublic, isPackage, isProtected, isPrivate }

 class ClassInfo {
  property:
     string name();
     string baseName();
     string parentName(); // if applicable, null otherwise
     string[] interfaces();
     bool isShared();
     Protection protection();
     DataMemberInfo[] data();
     MethodInfo[] methods();
     Object defaultConstructor();
     ...
 }

 Then for an e.g. method declaration we'd have:

 class MethodInfo {
  property:
     string name();
     bool isStatic(), isFinal(), isOverride();
     Protection protection();
     string[] parameterTypes();
     string[] parameterNames();
 }

 Some details may vary, e.g. some may be straight members instead of
 properties etc. (I used properties to allude to use of lazy gathering of
 information).

 So so far we have a nice collection of structured data associated with the
 entities in a D program. Note how this structuring differs yet has similar
 power to the primitives in std.traits; std.traits offers unstructured bits
 of information on demand (e.g. ParameterTypeNames) etc. but the objects
 above group information together per entity declared. All of the above goes
 in std.reflection, of course.

 ===========

 On to primitives that return such data.

 Given that D can (since relatively recently) create and manipulate class
 objects during compilation too, it follows that the classes above can be
 accessed in two ways - through compile-time API and run-time API. When
 possible, the APIs may even use the same functions; some other times they
 will be necessarily different.

 There are two possible approaches to discovering such information. One is
 by fetching the ModuleInfo for the whole module and navigating it. Another
 one is by using search primitives from strings.

 So we should have e.g.

 // inside std.reflection
 ModuleInfo getModuleInfo(string moduleName);

 so a CT call would go like:

 // client code
 static info = getModuleInfo("std.algorithm")**;

 whereas a run-time call would be:

 // client code
 auto info = getModuleInfo("std.algorithm")**;

 In the latter case, the module needs to save all needed information for
 ri, so it should plant this:

 // inside std.algorithm
 mixin(**makeModuleInfoAvailableDynamic**ally());

 The mixin would generate all information needed and would store it for
 later dynamic use.

 A search API would go like e.g.

 ClassInfo getClassInfo(string className);

 In this case the class name could be qualified with module information etc.

 ===========

 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that can
 be queried during compilation or dynamically.

 Please chime in with thoughts. Would someone want to pioneer this project?


 Andrei

--14dae9399b013d174c04cee2bbb4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Huge interest!<div>I wouldn&#39;t want to see static data structures bloat/= clutter the exe though in cases where it&#39;s not ever used/queried.<br><d= iv>The object factory is already a serious problem.</div></div><div class= =3D"gmail_extra"> <br><br><div class=3D"gmail_quote">On 22 July 2012 00:44, Andrei Alexandres= cu <span dir=3D"ltr">&lt;<a href=3D"mailto:SeeWebsiteForEmail erdani.org" t= arget=3D"_blank">SeeWebsiteForEmail erdani.org</a>&gt;</span> wrote:<br><bl= ockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #= ccc solid;padding-left:1ex"> Walter and I discussed the idea below a long time (years) ago. Most likely = it&#39;s also been discussed in this newsgroup a couple of times. Given the= state of the compiler back then, back then it seemed like a super cool ide= a that&#39;s entirely realizable, it would just take time for the compiler = to become as capable as needed. Nowadays we&#39;re in shape to tackle it.<b= r> <br> Here &quot;it&quot; is.<br> <br> Back when runtime reflection was being discussed, my response was &quot;let= &#39;s focus on compile-time reflection, and then we can do run-time reflec= tion on demand as a library&quot;. Though this might sound sensible, I init= ially didn&#39;t have a design. Now here&#39;s what we can do.<br> <br> Currently we have information about symbols as __traits(...) intrinsics wra= pped in nice but scattered ways. Now that CTFE is good enough to manipulate= structs and arrays thereof, we have the possibility to finally approach th= ings in a nicely unified, structured way.<br> <br> First, we need to prime std.reflection with a few abstractions that charact= erize entities in a D program.<br> <br> class ModuleInfo {<br> property:<br> =C2=A0 =C2=A0 string name();<br> =C2=A0 =C2=A0 ImportInfo[] imports();<br> =C2=A0 =C2=A0 DataInfo[] data();<br> =C2=A0 =C2=A0 FunctionInfo[] functions();<br> =C2=A0 =C2=A0 ClassInfo[] classes();<br> =C2=A0 =C2=A0 StructInfo[] structs(); // includes unions<br> =C2=A0 =C2=A0 TemplateInfo[] templates();<br> =C2=A0 =C2=A0 EnumInfo[] enums();<br> =C2=A0 =C2=A0 bool hasStaticCtor(), hasStaticDtor(),<br> =C2=A0 =C2=A0 =C2=A0 hasSharedCtor(), hasSharedDtor();<br> }<br> <br> Probably there are a few more pieces of data, but you get the idea. Then fo= r each of the entities mentioned above we have a similar definition. For ex= ample:<br> <br> enum Protection { isPublic, isPackage, isProtected, isPrivate }<br> <br> class ClassInfo {<br> property:<br> =C2=A0 =C2=A0 string name();<br> =C2=A0 =C2=A0 string baseName();<br> =C2=A0 =C2=A0 string parentName(); // if applicable, null otherwise<br> =C2=A0 =C2=A0 string[] interfaces();<br> =C2=A0 =C2=A0 bool isShared();<br> =C2=A0 =C2=A0 Protection protection();<br> =C2=A0 =C2=A0 DataMemberInfo[] data();<br> =C2=A0 =C2=A0 MethodInfo[] methods();<br> =C2=A0 =C2=A0 Object defaultConstructor();<br> =C2=A0 =C2=A0 ...<br> }<br> <br> Then for an e.g. method declaration we&#39;d have:<br> <br> class MethodInfo {<br> property:<br> =C2=A0 =C2=A0 string name();<br> =C2=A0 =C2=A0 bool isStatic(), isFinal(), isOverride();<br> =C2=A0 =C2=A0 Protection protection();<br> =C2=A0 =C2=A0 string[] parameterTypes();<br> =C2=A0 =C2=A0 string[] parameterNames();<br> }<br> <br> Some details may vary, e.g. some may be straight members instead of propert= ies etc. (I used properties to allude to use of lazy gathering of informati= on).<br> <br> So so far we have a nice collection of structured data associated with the = entities in a D program. Note how this structuring differs yet has similar = power to the primitives in std.traits; std.traits offers unstructured bits = of information on demand (e.g. ParameterTypeNames) etc. but the objects abo= ve group information together per entity declared. All of the above goes in= std.reflection, of course.<br> <br> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> <br> On to primitives that return such data.<br> <br> Given that D can (since relatively recently) create and manipulate class ob= jects during compilation too, it follows that the classes above can be acce= ssed in two ways - through compile-time API and run-time API. When possible= , the APIs may even use the same functions; some other times they will be n= ecessarily different.<br> <br> There are two possible approaches to discovering such information. One is b= y fetching the ModuleInfo for the whole module and navigating it. Another o= ne is by using search primitives from strings.<br> <br> So we should have e.g.<br> <br> // inside std.reflection<br> ModuleInfo getModuleInfo(string moduleName);<br> <br> so a CT call would go like:<br> <br> // client code<br> static info =3D getModuleInfo(&quot;std.algorithm&quot;)<u></u>;<br> <br> whereas a run-time call would be:<br> <br> // client code<br> auto info =3D getModuleInfo(&quot;std.algorithm&quot;)<u></u>;<br> <br> In the latter case, the module needs to save all needed information for ri,= so it should plant this:<br> <br> // inside std.algorithm<br> mixin(<u></u>makeModuleInfoAvailableDynamic<u></u>ally());<br> <br> The mixin would generate all information needed and would store it for late= r dynamic use.<br> <br> A search API would go like e.g.<br> <br> ClassInfo getClassInfo(string className);<br> <br> In this case the class name could be qualified with module information etc.= <br> <br> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> <br> With this design we unify compile-time and run-time type manipulation in si= mple ways, by defining structured information about declarations that can b= e queried during compilation or dynamically.<br> <br> Please chime in with thoughts. Would someone want to pioneer this project?<= span class=3D"HOEnZb"><font color=3D"#888888"><br> <br> <br> Andrei<br> </font></span></blockquote></div><br></div> --14dae9399b013d174c04cee2bbb4--
Nov 19 2012
prev sibling parent "Malte Skarupke" <malteskarupke web.de> writes:
I like it. First a couple details:

 class ClassInfo {
  property:
     string name();
     string baseName();
     string parentName(); // if applicable, null otherwise
     string[] interfaces();
     bool isShared();
     Protection protection();
     DataMemberInfo[] data();
     MethodInfo[] methods();
     Object defaultConstructor();
     ...
 }

Why do you not use recursion here? Why not ClassInfo base(); ClassInfo parent(); and then you get baseName by calling base.name;
 Then for an e.g. method declaration we'd have:

 class MethodInfo {
  property:
     string name();
     bool isStatic(), isFinal(), isOverride();
     Protection protection();
     string[] parameterTypes();
     string[] parameterNames();
 }

Do you think there'll be any way to get the actual types of the arguments and the return type back from the struct? I think that without compiler magic or an uncomfortable amount of templates it probably won't be possible, but I might be wrong. Now to my big complaint: I really dislike making the runtime api accessible through a mixin. I'd much prefer making it a non-member template. Reasons are a) I don't want to store the runtime information in my type. A vtable pointer should be enough to get that information. Maybe use it as an index into a global immutable associative array. For modules and non-class types the name can act as a unique identifier. b) It'd make it possible to get runtime information on types that I get from libraries. c) It would lead to less coupling. d) It would discourage writing types that rely on runtime reflection, but it would not make it impossible.
Nov 19 2012