www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compiler patch for runtime reflection

reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
Hi,

Igor Stepanov has created a patch for DMD and Druntime which adds RTTI  
information for class and struct members.

Example:

import std.stdio;

class Foo
{
	static void PrintHello()
	{
		writeln("Hello");
	}
}
void main()
{
	auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
	assert(info.name == "PrintHello");
	auto print = cast(void function())info.pointer;
	print(); //prints "Hello"
}

While the inclusion of such functionality into the language remains a  
disputed matter, would anyone be interested in an unofficial patch for  
this?

Walter: would it be okay if the compiler changes were published as a  
GitHub fork, or should we stick to patches?

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 21 2011
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 21-10-2011 21:07, Vladimir Panteleev wrote:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

Absolutely awesome!
 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

Yes, very much. I would recommend setting up a fork on GitHub, and then adding it to a branch, e.g. 'reflection', if it's not going to be included in mainline DMD.
 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?

- Alex
Oct 21 2011
next sibling parent =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 22-10-2011 05:36, Robert Jacques wrote:
 On Fri, 21 Oct 2011 17:15:02 -0400, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 On 21-10-2011 21:07, Vladimir Panteleev wrote:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

Absolutely awesome!
 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

Yes, very much. I would recommend setting up a fork on GitHub, and then adding it to a branch, e.g. 'reflection', if it's not going to be included in mainline DMD.

Really? A runtime reflection system has been on the review queue for over six months and I've brought the subject up on the newsgroup. I've received zero feedback. So while I'm sure everyone want to check RTTI off the D features list, I've not seen much real interest in it.

I must have missed that. What should I search for to find your thread? - Alex
Oct 22 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-22 05:36, Robert Jacques wrote:
 On Fri, 21 Oct 2011 17:15:02 -0400, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 On 21-10-2011 21:07, Vladimir Panteleev wrote:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

Absolutely awesome!
 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

Yes, very much. I would recommend setting up a fork on GitHub, and then adding it to a branch, e.g. 'reflection', if it's not going to be included in mainline DMD.

Really? A runtime reflection system has been on the review queue for over six months and I've brought the subject up on the newsgroup. I've received zero feedback. So while I'm sure everyone want to check RTTI off the D features list, I've not seen much real interest in it.

I must have completely missed that. -- /Jacob Carlborg
Oct 22 2011
prev sibling next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 21.10.2011 21:07, schrieb Vladimir Panteleev:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?

I'd love to see proper runtime reflection support in D, including functionality to get information about available methods (their name and parameters) and a way to call them. Something that is close to what Java offers would be great. BTW: I don't really see the problem with providing this information (overhead-wise) - the information needs to be available once per class/struct, but objects of classes just need one pointer to it (other types don't even need that because they're not polymorphic and - like methods of structs - the address of the information is known at compile-time). Cheers, - Daniel
Oct 21 2011
next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 22-10-2011 05:48, Robert Jacques wrote:
 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?

I'd love to see proper runtime reflection support in D, including functionality to get information about available methods (their name and parameters) and a way to call them.

What do you mean by their 'parameters'? What about overloads? Attributes? Arguments? Argument attributes?
 Something that is close to what Java offers would be great.

And what, exactly does JAVA offer? What works? What doesn't work? What's missing?
 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).

1) Unused information is simply bloat: it increases exe size, slows the exe down and increases the runtime memory footprint. 2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets) memory and disk space are both highly constrained resources that you don't want to waste. 3) RTTI provides a back-door into a code-base; one that for many reasons you may want to keep closed.

You could just introduce a -reflection switch to include reflection information. - Alex
Oct 22 2011
parent reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 22-10-2011 15:20, Andrej Mitrovic wrote:
 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net>  wrote:
 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com>  wrote:

 You could just introduce a -reflection switch to include reflection
 information.

As I mentioned in the previous thread on the subject, I think that ideally we should improve compile-time reflection so it's possible to generate runtime reflection information at compilation time. Something like this: import reflection; enum reflectionForStdStdio = generateReflectionForModule("std.stdio"); void main() { reflectionForStdStdio.callFunction("writeln", "Hello, world!"); } -- Best regards, Vladimir mailto:vladimir thecybershadow.net

I think user properties would be a perfect fit for this. :) reflected struct Foo {}

The only downside with either approach is that people need to explicitly mark their types as "reflectable", which can be a slight hindrance in the way of discoverability (and probably also complicates actual reflection - what if you try to get the return type of a reflected method and that type is not available? This would make for an *awful* lot of null checks). In any case, if we need to mark stuff as reflectable in-source, I'd vouch for reflected, though I would really prefer application-global reflection info. - Alex
Oct 22 2011
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 22-10-2011 17:18, Vladimir Panteleev wrote:
 On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 On 22-10-2011 15:20, Andrej Mitrovic wrote:
 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net> wrote:
 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 You could just introduce a -reflection switch to include reflection
 information.

As I mentioned in the previous thread on the subject, I think that ideally we should improve compile-time reflection so it's possible to generate runtime reflection information at compilation time. Something like this: import reflection; enum reflectionForStdStdio = generateReflectionForModule("std.stdio"); void main() { reflectionForStdStdio.callFunction("writeln", "Hello, world!"); } -- Best regards, Vladimir mailto:vladimir thecybershadow.net

I think user properties would be a perfect fit for this. :) reflected struct Foo {}

The only downside with either approach

What do you mean by "either approach"? Which is the other one?

reflected and generateReflectionForModule. - Alex
Oct 22 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/23/2011 12:19 PM, Vladimir Panteleev wrote:
 On Sun, 23 Oct 2011 01:54:52 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 On 22-10-2011 17:18, Vladimir Panteleev wrote:
 On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 On 22-10-2011 15:20, Andrej Mitrovic wrote:
 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net> wrote:
 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 You could just introduce a -reflection switch to include reflection
 information.

As I mentioned in the previous thread on the subject, I think that ideally we should improve compile-time reflection so it's possible to generate runtime reflection information at compilation time. Something like this: import reflection; enum reflectionForStdStdio = generateReflectionForModule("std.stdio"); void main() { reflectionForStdStdio.callFunction("writeln", "Hello, world!"); } -- Best regards, Vladimir mailto:vladimir thecybershadow.net

I think user properties would be a perfect fit for this. :) reflected struct Foo {}

The only downside with either approach

What do you mean by "either approach"? Which is the other one?

reflected and generateReflectionForModule. - Alex

I don't see how my generateReflectionForModule idea requires any modification of the reflected code. The module can generate RTTI for all types used recursively from the starting point. If more precision is needed, it can be specified as generateReflectionForModule/etc. parameters.

The module can generate RTTI for all types recursively from the starting point iff that information is statically available. It does not have to be. A module that comes as .di + static library binary could return a reference to a private class that has a publicly exported base class. How would you generate RTTI for a statically invisible class? Eg: // file A.d module A; class C{int x; ...} private class D: C{int y; ...} C factory(){ return new D; } // file A.di module A; class C{int x; ...} // file B.d import A; // imports A.di file enum reflectionForA = generateReflectionForModule("A"); void main(){ C c = factory(); auto s = reflectionForA.serialize(c); // disaster strikes. }
Oct 23 2011
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 10/23/2011 01:06 PM, Timon Gehr wrote:
 On 10/23/2011 12:19 PM, Vladimir Panteleev wrote:
 On Sun, 23 Oct 2011 01:54:52 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 On 22-10-2011 17:18, Vladimir Panteleev wrote:
 On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 On 22-10-2011 15:20, Andrej Mitrovic wrote:
 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net> wrote:
 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 You could just introduce a -reflection switch to include reflection
 information.

As I mentioned in the previous thread on the subject, I think that ideally we should improve compile-time reflection so it's possible to generate runtime reflection information at compilation time. Something like this: import reflection; enum reflectionForStdStdio = generateReflectionForModule("std.stdio"); void main() { reflectionForStdStdio.callFunction("writeln", "Hello, world!"); } -- Best regards, Vladimir mailto:vladimir thecybershadow.net

I think user properties would be a perfect fit for this. :) reflected struct Foo {}

The only downside with either approach

What do you mean by "either approach"? Which is the other one?

reflected and generateReflectionForModule. - Alex

I don't see how my generateReflectionForModule idea requires any modification of the reflected code. The module can generate RTTI for all types used recursively from the starting point. If more precision is needed, it can be specified as generateReflectionForModule/etc. parameters.

The module can generate RTTI for all types recursively from the starting point iff that information is statically available. It does not have to be. A module that comes as .di + static library binary could return a reference to a private class that has a publicly exported base class. How would you generate RTTI for a statically invisible class? Eg: // file A.d module A; class C{int x; ...} private class D: C{int y; ...} C factory(){ return new D; } // file A.di module A; class C{int x; ...}

C factory();
 // file B.d
 import A; // imports A.di file

 enum reflectionForA = generateReflectionForModule("A");

 void main(){
 C c = factory();
 auto s = reflectionForA.serialize(c); // disaster strikes.
 }

Oct 23 2011
prev sibling next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 23-10-2011 14:51, Vladimir Panteleev wrote:
 On Sun, 23 Oct 2011 14:06:42 +0300, Timon Gehr <timon.gehr gmx.ch> wrote:

 I don't see how my generateReflectionForModule idea requires any
 modification of the reflected code. The module can generate RTTI for all
 types used recursively from the starting point. If more precision is
 needed, it can be specified as generateReflectionForModule/etc.
 parameters.

The module can generate RTTI for all types recursively from the starting point iff that information is statically available. It does not have to be. A module that comes as .di + static library binary could return a reference to a private class that has a publicly exported base class. How would you generate RTTI for a statically invisible class?

The only way to prevent such problems is to have RTTI generated by the compiler and enabled always or by default. However, is such functionality even desirable? It is equivalent to having a back door for accessing private class members (implementation details in general).

Isn't that technically already possible by doing enough cast and pointer arithmetic hacks? I don't think it really makes a difference. If someone really wants to backdoor a type, they will find a way to do it. - Alex
Oct 23 2011
prev sibling next sibling parent =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 23-10-2011 18:03, Robert Jacques wrote:
 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
 [snip]
 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?

You're not supposed to be able to. Runtime reflection should only apply to public data members.

I don't think that's necessarily a good idea. If you're doing something related to e.g. serialization and a type uses properties to hide fields (say, because the properties have in/out contracts), you would be unable to take those into account. - Alex
Oct 23 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-23 18:03, Robert Jacques wrote:
 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
 [snip]
 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?

You're not supposed to be able to. Runtime reflection should only apply to public data members.

It's not enough. Take this for example: class Foo < ActiveRecord::Base before_save :bar private def bar end end The above code is an example from Ruby on Rails. The "before_save" call sets up a callback, "bar", that will called before saving the model. The callback method will be called using reflection. -- /Jacob Carlborg
Oct 23 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-23 20:54, Robert Jacques wrote:
 On Sun, 23 Oct 2011 13:48:03 -0400, Jacob Carlborg <doob me.com> wrote:
 On 2011-10-23 18:03, Robert Jacques wrote:
 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch>
 wrote:
 [snip]
 The module can generate RTTI for all types recursively from the
 starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?

You're not supposed to be able to. Runtime reflection should only apply to public data members.

It's not enough. Take this for example: class Foo < ActiveRecord::Base before_save :bar private def bar end end The above code is an example from Ruby on Rails. The "before_save" call sets up a callback, "bar", that will called before saving the model. The callback method will be called using reflection.

Using a short, cryptic stub from another programming language is not an effective communication tool. Case in point: all I understand is that something is returning a delegate to something, and reflection never enters the picture.

What's happening is that the class method "before_save" is called with the symbol (think string) "bar". Then later ActiveRecord::Base will call the method corresponding to the string "bar" using reflection. In D you would proabably do something like: class Foo : ActiveRecord.Base { private void bar () {} mixin beforeSave!(bar); } Now you still won't be able to call "bar" outside "Foo" if "bar" is private.
 But, for sake of argument, let me assume that ActiveRecord needs to
 reflect Foo for rails to work (i.e. ActiveRecord::Base's ctor reflects
 this). At a very fundamental level, this isn't an argument for or
 against RTTI in D; in D you'd do this with a mixin of some sort. When
 looking a use cases/features from other languages, always remember to
 first ask oneself a) how would I do it in D today? b) Is language X's
 solution 'better, faster, stronger' in some way?

You can think of it like this, just because you haven't come up with a reason for calling privte methods using reflection doesn't mean anyone else won't come up with a good reason. I tried here to give a reason.
 I mean, considering that Ruby is a dynamic language, would bar even be
 considered private from inside the super-class's ctor? Is Ruby's private
 even comparable to D's private? Given that Ruby, by design, requires
 fields to be private and for all external accesses to happen via
 methods, should we consider a way to violate that contract an example of
 mis-feature?

There is no constructor involved here. "before_save" is class method called when the class "Foo" is loaded. You can do basically whatever you want with the code in Ruby. You can replace an existing method/class in the standard library with your own. You can easily add methods to existing classes. It's the programmer that is responsible to not mess up things, just as in D where you can cast away immutable, shared and do unsafe things with pointers. -- /Jacob Carlborg
Oct 24 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-24 17:37, Robert Jacques wrote:
 I'm sorry if I've come across that way. I'm well aware with the reasons
 for wanting access to private methods/fields via reflection and have
 mentioned them in previous posts. What I've tried to point out is very
 eloquently stated by fig. 6.3 on page 204 of TDPL: allowing reflection
 to bypass protection guarantees essentially makes all declarations
 public in nature, if not extern. That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.

If someone is using a library and starts to mess with calling private methods they shouldn't they'll most likely mess things up for them self.
 Back to my actual argument, what I was pointing out was that the
 toolboxes of various languages differ heavily. Just because Ruby has a
 very cool library which uses runtime reflection, one shouldn't simply
 state 'D must have runtime reflection.' One needs to first decompose
 what the library is trying to achieve from the language mechanisms used
 to implement it. Then you can ask yourself 1) Can I do that in D? 2) How
 does D's solution compare to Ruby's?

Of course. I'm just showing that another language/library uses it and therefore there could be a chance that should be doable in D as well.
 My impression of what ActiveRecord, (which my be wrong), is that its
 using runtime-reflection to do various meta-programming activities and
 that under-the-hood, it stretching Ruby to the breaking point in order
 to do so. D, on the other hand, has a ton of meta-programming specific
 features, and doesn't need to rely on reflection to do so. This is
 partly to do with Ruby supporting 'eval' and thus making no distinction
 between compile-time and run-time behavior; that's correct for Ruby, but
 D is a different kettle of fish.

I doubt that ActiveRecord is stretching Ruby to the breaking point. On the other hand it feels like I'm stretching D's metaprogramming capabilities in my serialization library. That's just because D doesn't have proper reflection. In Ruby there is very seldom reason to use "eval". That's because of the excellent support for runtime reflection. There's no clumsy syntax, no hacks, no need to instantiate reflectors and similar. You just call a method like you would call any other method: class Foo def bar end end foo = Foo.new foo.bar() foo.send("bar") In the above code, "bar" is called first using regular method call syntax and then using reflection.
 I mean, considering that Ruby is a dynamic language, would bar even be
 considered private from inside the super-class's ctor? Is Ruby's private
 even comparable to D's private? Given that Ruby, by design, requires
 fields to be private and for all external accesses to happen via
 methods, should we consider a way to violate that contract an example of
 mis-feature?

There is no constructor involved here. "before_save" is class method called when the class "Foo" is loaded.

So it's the static ctor, then.

I guess you could call it that.
 You can do basically whatever you want with the code in Ruby. You can
 replace an existing method/class in the standard library with your own.
 You can easily add methods to existing classes. It's the programmer that
 is responsible to not mess up things, just as in D where you can cast
 away immutable, shared and do unsafe things with pointers.

Well one one hand, none of those things are allowed in SafeD (And you know, that's not a resounding argument for reflection of private members). On the other, you didn't answered my other two questions. In fact, the impression I'm getting is that Ruby's private isn't comparable to D's private.

If we talk about methods it's easiest. If you call a private method in Ruby using the regular method call syntax you get an error. If you do the same in D you get an error as well. The only difference is in Ruby it will be a runtime error and in D it will be a compile time error. In Ruby you can bypass private using reflection with the "send" method. In D you can bypass private using a delegate. In Ruby an instance variable is always private. I don't think you can declare a public instance variable, one uses setters/getters for that. In D you can declare both private and public instance variables. In Ruby you can get/set a value of an instance variable using reflection, instance_variable_set/get. In D you can get/set a value of a private instance variable using compile time reflection, i.e. .tupleof. Now, since we can call private methods and get/set private instance variables using the features we have now, why shouldn't we be able to do the same with runtime reflection? BTW, I don't think it's a mis-feature. -- /Jacob Carlborg
Oct 24 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-25 07:24, Robert Jacques wrote:
 Aren't private methods by definition something people shouldn't be using?

Yes, that's what I mean. But sometimes it CAN be useful.
 :) I don't know myself. I've just heard that the internals of Rails are
 deep, dark and full of non-idiomatic Rudy code.

The parts that I've looked at from Rails are not that special.
 On
 the other hand it feels like I'm stretching D's metaprogramming
 capabilities in my serialization library. That's just because D doesn't
 have proper reflection.

I've written three serialization libraries and two versions of a runtime reflection library; the first one was prior to the string bugs in CTFE being fixed and ran DMD out of memory. I found serialization to be simple and straight forward. And I found compiler bugs to be a greater issue with reflection than any limits in D's capabilities. Indeed, given I wrote the same library using two completely different mechanisms, gives me great hope for D. So, I don't agree with your assessment, but I do understand where you're coming from. A lot of D's metaprogramming capabilities are undocumented, and knowing the right tool for the job is essential to getting nice, clean code.

For example, getting the name of a instance variable: T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $]; I wouldn't consider that simple and straight forward. This works for both D1 and D2, don't know if D2 has a better way. Your serialization library only works on public fields? Doesn't sound very useful.
 (Just checking, but you do know D has static ctors.)

Yes.
 Actually, you're not supposed to be able to; this is bug 1983.
 http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
 https://github.com/D-Programming-Language/dmd/pull/71

I don't see what that has to do with private, it's about const, as far as I can see. I don't see how calling a private method through a delegate can be easily solved. I see three solutions: 1. Make it a compile time error to create a delegate from a private method. This would not allow to call that method via a delegate in the same module, which is possible if you call the method directly. 2. Implement some, most likely, very advanced data flow analysis in the compiler to detect when a delegate pointing to a private method escapes the module. Results in a compile time error. Will most likely not happen. 3. Make it a runtime error when the delegate is actually called. This would probably need some pieces of 2 as well. I don't think the community will be happy about adding runtime checks like these. Oh, BTW the same needs to be done for function pointers as well since you can build a delegate out of a function pointer and an object.
 In Ruby an instance variable is always private. I don't think you can
 declare a public instance variable, one uses setters/getters for that.
 In D you can declare both private and public instance variables.

 In Ruby you can get/set a value of an instance variable using
 reflection, instance_variable_set/get.
 In D you can get/set a value of a private instance variable using
 compile time reflection, i.e. .tupleof.

Yes, but only for source code you control.

You don't need control of the source, you only need the static type. Of course an opaque struct won't work.
 Actually, baring bugs in DMD, .tupleof is the only way to access private
 fields and there is _no way_ to access private members; And .tupleof
 only works if you have access to the source code. Furthermore, .tupleof,
 god-send that it is, isn't part of the D spec; unlike .stringof, it
 didn't make it into TDPL nor into the D documentation. D's official
 method of compile-time reflection is __traits. And __traits doesn't
 bypass protection attributes. So precedence would be against
 runtime-reflection having access to private members.

The compiler has .tupleof and it's mentioned in the spec at d-programming-language.org: http://d-programming-language.org/struct.html -- /Jacob Carlborg
Oct 25 2011
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/25/2011 11:24 AM, Gor Gyolchanyan wrote:
 I said it once, but nobody responded.

 Why not generate full reflection for everything by default and add an
 attribute line  noreflect as an _optimization_ tool in cases, where
 this thing is definitely not gonna be used in reflection?

As I see it, because you almost never need 'full reflection for everything', and idiomatic D code is not supposed to contain clutter like that.
Oct 25 2011
parent Ary Manzana <ary esperanto.org.ar> writes:
On 10/25/11 1:05 PM, Robert Jacques wrote:
 On Tue, 25 Oct 2011 11:31:18 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:
 As i said before, enabling reflection manually is an act of enabling a
 certain range of stuff you can do with the code. Forgetting to enable
 reflection can lead to serious problems, when the reflection is
 required.

And when, precisely, is reflection _required_? Granted, in other languages it is the only tool for the job. But we have a very big toolkit in D, and so far, I have yet to see any example (including Rails) which _requires_ the holistic runtime reflection proposed in these threads.

This is completely true. Something like: class Person < ActiveRecord::Base belongs_to :company end could be done at compile-time. Same as retrieving the schema from mysql: could be done at compile-time, like Nemerle does (or so I remember someone said that). Me too, believes that greater compile-time capabilities will make almost all (but never all) runtime reflection unnecessary.
Oct 25 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-25 16:21, Robert Jacques wrote:
 On Tue, 25 Oct 2011 03:23:40 -0400, Jacob Carlborg <doob me.com> wrote:
 For example, getting the name of a instance variable:

 T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $];

 I wouldn't consider that simple and straight forward. This works for
 both D1 and D2, don't know if D2 has a better way.

Not really, for the advanced user. I do find using "some string".length to be cleaner in these cases where you have to slice .stringof. i.e. static Value from(T)(T value) { //... static if(is(T == struct) || is(T == class)) { Value[string] result; foreach(i,v;value.tupleof) { result[value.tupleof[i].stringof["value.".length..$]] = from(v); } return Value( result ); } //.. } That said, I think anyone can read the above and understand what's happening. And even writing that code would be straightforward is .stringof and .tupleof were better documented.

Yes, I found the snippet I use to get the name of a field by trail and error.
 Also, the first serialization library I wrote used __traits instead. It
 was more awkward, mainly because __traits doesn't an 'allFields' member,
 just an 'allMembers'. If it did then the above becomes:

 foreach(name;__traits(allFields,value) {
 result[name] = from(__traits(getMember, value, name));
 }

 which really is straightforward. Now I originally used 'allMembers' and
 then filtered out everything I didn't want, which was fairly
 straightforward but verbose.

__traits(getMember) is more clean when you the name of all the fields you want.
 Your serialization library only works on public fields? Doesn't sound
 very useful.

In the interest of full disclosure, the library I wrote was quick and dirty and does not handle ploy-morphism, cycle detection, aliasing, etc (mainly because JSON doesn't); I used traits for my first one (so protection was observed) but .tupleof for my second and third and was lazy about it. It's one line of code to add protection to tuple of, but I've never gotten around to it. I figured my code would be replace by orange or std.serialization or something.

Well, things get a bit more complicated when the library needs to support ploy-morphism, cycle detection, aliasing and so on.
 That said, I have never serialized a private variable and have never
 wanted too. The serialized objects are, for the most part, part of the
 external API to my program. I have to do validation checks on the inputs
 and changing anything takes a week to get the other teams to propagate
 the changes on their side. There are also a couple of classes which I
 serialize in a binary format as a poor mans database, but again, there's
 no need for anything in it to be private.

Ok, I see.
 [snip]

 Actually, you're not supposed to be able to; this is bug 1983.
 http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
 https://github.com/D-Programming-Language/dmd/pull/71

I don't see what that has to do with private, it's about const, as far as I can see.

Both are protection attributes. The problem of taking the delegate of a mutable member on a const object, is the exact same problem as taking the delegate of a private member.

I don't agree.
 In short, you can't 'bypass private using a delegate.' That you can
 today, is simple a bug in the compiler.

 I don't see how calling a private method through a
 delegate can be easily solved. I see three solutions:

 1. Make it a compile time error to create a delegate from a private
 method. This would not allow to call that method via a delegate in the
 same module, which is possible if you call the method directly.

 2. Implement some, most likely, very advanced data flow analysis in the
 compiler to detect when a delegate pointing to a private method escapes
 the module. Results in a compile time error. Will most likely not happen.

 3. Make it a runtime error when the delegate is actually called. This
 would probably need some pieces of 2 as well. I don't think the
 community will be happy about adding runtime checks like these.

 Oh, BTW the same needs to be done for function pointers as well since
 you can build a delegate out of a function pointer and an object.

Bypassing the protection type system and delegate escape are very different things. The escape of a delegate is fully controlled by the coder of the class/module; with bypass mechanisms, the library writer has no control.

So how should this be fixed? Make it a compile error to create a delegate from a private method?
 In Ruby an instance variable is always private. I don't think you can
 declare a public instance variable, one uses setters/getters for that.
 In D you can declare both private and public instance variables.

 In Ruby you can get/set a value of an instance variable using
 reflection, instance_variable_set/get.
 In D you can get/set a value of a private instance variable using
 compile time reflection, i.e. .tupleof.

Yes, but only for source code you control.

You don't need control of the source, you only need the static type. Of course an opaque struct won't work.

Sorry, that's what I meant by source you control. i.e. it is available to you to read in some form. Runtime reflection would allow you access to knowledge of opaque types.

Ok, actually I haven't thought of how runtime reflection would be have with an opaque type.
 Actually, baring bugs in DMD, .tupleof is the only way to access private
 fields and there is _no way_ to access private members; And .tupleof
 only works if you have access to the source code. Furthermore, .tupleof,
 god-send that it is, isn't part of the D spec; unlike .stringof, it
 didn't make it into TDPL nor into the D documentation. D's official
 method of compile-time reflection is __traits. And __traits doesn't
 bypass protection attributes. So precedence would be against
 runtime-reflection having access to private members.

The compiler has .tupleof and it's mentioned in the spec at d-programming-language.org: http://d-programming-language.org/struct.html

I stand corrected. Sorry, I just looked at TDPL's index, the page on Properties and googled it. Now that I look for it, it's also on the Class page.

No problem.
 Still, __traits is still the full fledged compile-time reflection
 feature and doesn't bypass protection attributes.

Yeah, but for a serialization library I want to be able to serialize private fields. This is known issue that this breaks encapsulation. -- /Jacob Carlborg
Oct 25 2011
next sibling parent Kapps <Kapps NotValidEmail.com> writes:
The point of runtime reflection is not your own code, that's the thing.
By having opt-in reflection, you are at the mercy that every single 
component, every piece of code, every library, that you did not write, 
the author decided to enable reflection. It is an unreasonable assumption.


On 25/10/2011 11:44 PM, Robert Jacques wrote:
 I understand that at times when serialization has to break
 encapsulation. My opinion though, is that classes should have to opt in,
 before the serializer goes ahead and breaks their carefully designed
 encapsulation. i.e. someone should have to code review the class to make
 sure everything is kosher. For code you are developing, this is as
 simple and straightforward as adding a single mixin statement to the class.

It is the responsibility of the one using your code to determine whether they want take the risk and access private members. Sometimes, it's needed. I have used private methods in production code when programming in .NET, just because the developers of it did not realize that something that's private would be useful to some people. It saved many, many, hours of pain and effort of rolling your own solution. If I upgrade the app to a newer version of .NET and the code breaks, that's the risk that was taken. Presumably the person accessing these things the author did not intend has taken steps to ensure it doesn't break if things change (or doesn't intend to link to a newer version of the library).
Oct 25 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-26 07:44, Robert Jacques wrote:
 On Tue, 25 Oct 2011 12:38:18 -0400, Jacob Carlborg <doob me.com> wrote:
 On 2011-10-25 16:21, Robert Jacques wrote:

 Both are protection attributes. The problem of taking the delegate of a
 mutable member on a const object, is the exact same problem as taking
 the delegate of a private member.

I don't agree.

[snip]
 Bypassing the protection type system and delegate escape are very
 different things. The escape of a delegate is fully controlled by the
 coder of the class/module; with bypass mechanisms, the library writer
 has no control.

So how should this be fixed? Make it a compile error to create a delegate from a private method?

I think we are taking about different things. Today in D, due to a bug, for example foo.bar() won't compile because it's private/not-const/etc. But (&foo.bar)() will compile. That 'feature' is what I thought you were referring to. If the writer of a class wants to let a delegate of a private member function escape, I don't have a fundamental problem with it. But I do have a problem with someone who isn't the writer of the class doing so.

I'm referring to the above but, as I've already written and you haven't replied to is, I don't understand how it can be prevented.
 [snip]

 Still, __traits is still the full fledged compile-time reflection
 feature and doesn't bypass protection attributes.

Yeah, but for a serialization library I want to be able to serialize private fields. This is known issue that this breaks encapsulation.

I understand that at times when serialization has to break encapsulation. My opinion though, is that classes should have to opt in, before the serializer goes ahead and breaks their carefully designed encapsulation. i.e. someone should have to code review the class to make sure everything is kosher. For code you are developing, this is as simple and straightforward as adding a single mixin statement to the class.

It's not straightforward when serializing third party types. One of my goals when I started writing my serialization library was to, somewhere down the road, create a window/interface builder for DWT. The builder would (de)serialize the widgets available in DWT and save it as an XML file. Since I'm the current maintainer of DWT I could go and modify DWT to fit my needs but I don't think that's the right approach and will make merging future versions of SWT more difficult. -- /Jacob Carlborg
Oct 26 2011
prev sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 24.10.2011 17:37, schrieb Robert Jacques:
 That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.
 

I don't think marking a method as private is a good way to hide critical information. If you can actually call methods (like in real calls, not some kind of webservice), you can as well use pointers to read the memory directly. Or he could run it in a debugger. Or (with root privileges) just look at the whole memory of the process. Or ... Cheers, - Daniel
Oct 24 2011
parent reply Kapps <Kapps NotValidEmail.com> writes:
I really like Gor's idea of  noreflect for this. You have a network 
class you don't want as easily reversible, or a security class, or 
private fields such as password. You just mark the module, type, 
methods, or fields, as  noreflect. It could even disable compile-time 
reflection if desired.

It would be important, however, to be able to pass a flag to the 
compiler that marks everything as  noreflect. The idea of opt-in 
reflection is just silly. Standard example is a serialization library. 
Oh hey, you wanna serialize a field of type Vector3? Sorry, the person 
forgot to mark it as  reflected. Oh, you wanna serialize these 
properties of some class? Sorry, not reflected. Opt-out is definitely 
the way to go, as it gives the best of all worlds (and removes all 
overhead if you just pass in something like --version=noreflect to dmd).

On 24/10/2011 10:11 PM, Robert Jacques wrote:
 On Mon, 24 Oct 2011 16:34:11 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 Am 24.10.2011 17:37, schrieb Robert Jacques:
 That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.

I don't think marking a method as private is a good way to hide critical information. If you can actually call methods (like in real calls, not some kind of webservice), you can as well use pointers to read the memory directly. Or he could run it in a debugger. Or (with root privileges) just look at the whole memory of the process. Or ... Cheers, - Daniel

By your argument, because can ever be truly secure there's no point to trying to secure it. I don't buy that. There is a world of difference between obj.reflect("SSN"), and trying to figure out 1) what objects hold SSN numbers 2) what's its field offset is and 3) what its memory representations is. (because remember, you don't have the source code) And that's assuming you can compile D code against the library and the library writer has directly exposed the critical object. More likely, you'll have to repeat steps 1-3 multiple times to find the data you're after. And runtime reflection means access will get exposed across language barriers, e.g. in your application's macro language. Furthermore, security focused languages seem to be centering around the concept of capabilities, of which private, package, protected, public and extern are the most rudimentary form of. Getting back to your original point, I wasn't suggesting that private is a panacea for security concerns, but it is one of the more powerful tools in our tool box.

Oct 25 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-26 07:59, Kapps wrote:
 I really like Gor's idea of  noreflect for this. You have a network
 class you don't want as easily reversible, or a security class, or
 private fields such as password. You just mark the module, type,
 methods, or fields, as  noreflect. It could even disable compile-time
 reflection if desired.

 It would be important, however, to be able to pass a flag to the
 compiler that marks everything as  noreflect. The idea of opt-in
 reflection is just silly. Standard example is a serialization library.
 Oh hey, you wanna serialize a field of type Vector3? Sorry, the person
 forgot to mark it as  reflected. Oh, you wanna serialize these
 properties of some class? Sorry, not reflected. Opt-out is definitely
 the way to go, as it gives the best of all worlds (and removes all
 overhead if you just pass in something like --version=noreflect to dmd).

I agree, than can be SO annoying just because someone forgot to think about reflection/serialization. -- /Jacob Carlborg
Oct 26 2011
prev sibling parent sclytrack <sclytrack fake.com> writes:
Runtime reflection on B, C, D not on A.
No reflection for private, protected.
[properties, methods, events]

class A
{
}

 reflection
class B: A
{
}


class C: B
{
}

class D: C
{
}



 I really like Gor's idea of  noreflect for this. You have a network
 class you don't want as easily reversible, or a security class, or
 private fields such as password. You just mark the module, type,
 methods, or fields, as  noreflect. It could even disable compile-time
 reflection if desired.

Jan 23 2012
prev sibling next sibling parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 22.10.2011 05:48, schrieb Robert Jacques:
 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?

I'd love to see proper runtime reflection support in D, including functionality to get information about available methods (their name and parameters) and a way to call them.

What do you mean by their 'parameters'? What about overloads? Attributes? Arguments? Argument attributes?

Primarily arguments. That should help identifying overloads. But attributes and argument attributes are needed as well.
 Something that is close to what Java offers would be great.

And what, exactly does JAVA offer? What works? What doesn't work? What's missing?

See http://download.oracle.com/javase/6/docs/api/java/lang/Class.html You can access constructors (look for one by it's parameters or get all of them), the same for methods (methods the class declares and all methods, i.e. also inherited ones), implemented interfaces, fields, ...
 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).

1) Unused information is simply bloat: it increases exe size, slows the exe down and increases the runtime memory footprint.

Yeah, but (mostly) only per class, not per object (besides one pointer per object, which shouldn't be that bad and something like this seems is already there for the existing typeinfo)
 2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets)
 memory and disk space are both highly constrained resources that you
 don't want to waste.
 3) RTTI provides a back-door into a code-base; one that for many reasons
 you may want to keep closed.

Yeah, it may make sense to be able to deactivate it. Cheers, - Daniel
Oct 22 2011
prev sibling parent deadalnix <deadalnix gmail.com> writes:
Le 24/10/2011 16:20, Steven Schveighoffer a Ă©crit :
 On Fri, 21 Oct 2011 23:48:51 -0400, Robert Jacques <sandford jhu.edu>
 wrote:

 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?

I'd love to see proper runtime reflection support in D, including functionality to get information about available methods (their name and parameters) and a way to call them.

What do you mean by their 'parameters'? What about overloads? Attributes? Arguments? Argument attributes?
 Something that is close to what Java offers would be great.

And what, exactly does JAVA offer? What works? What doesn't work? What's missing?
 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).

1) Unused information is simply bloat: it increases exe size, slows the exe down and increases the runtime memory footprint. 2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets) memory and disk space are both highly constrained resources that you don't want to waste. 3) RTTI provides a back-door into a code-base; one that for many reasons you may want to keep closed.

I agree to all these points. I think runtime reflection should be generated by the compiler as needed. And the mechanism to determine "as needed" should be an attribute IMO, or an intrinsic function (i.e. if you have all the static information about a type, you should be able to generate the RTTI for it). I do not want programs getting bigger than they already are for the sake of "just in case I want to use reflection"... -Steve

Well, if the compile time reflection is good enough, you can do this without any runtime reflexion support in the compiler, just using a lib. Why not support this option ?
Oct 24 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Fri, 21 Oct 2011 17:15:02 -0400, Alex Rønne Petersen <xtzgzorex gmail.com>
wrote:

 On 21-10-2011 21:07, Vladimir Panteleev wrote:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

Absolutely awesome!
 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

Yes, very much. I would recommend setting up a fork on GitHub, and then adding it to a branch, e.g. 'reflection', if it's not going to be included in mainline DMD.

Really? A runtime reflection system has been on the review queue for over six months and I've brought the subject up on the newsgroup. I've received zero feedback. So while I'm sure everyone want to check RTTI off the D features list, I've not seen much real interest in it.
Oct 21 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson <metalcaedes gmail.com> wrote:
 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?

I'd love to see proper runtime reflection support in D, including functionality to get information about available methods (their name and parameters) and a way to call them.

What do you mean by their 'parameters'? What about overloads? Attributes? Arguments? Argument attributes?
 Something that is close to what Java offers would be great.

And what, exactly does JAVA offer? What works? What doesn't work? What's missing?
 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).

1) Unused information is simply bloat: it increases exe size, slows the exe down and increases the runtime memory footprint. 2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets) memory and disk space are both highly constrained resources that you don't want to waste. 3) RTTI provides a back-door into a code-base; one that for many reasons you may want to keep closed.
Oct 21 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:

 You could just introduce a -reflection switch to include reflection  
 information.

As I mentioned in the previous thread on the subject, I think that ideally we should improve compile-time reflection so it's possible to generate runtime reflection information at compilation time. Something like this: import reflection; enum reflectionForStdStdio = generateReflectionForModule("std.stdio"); void main() { reflectionForStdStdio.callFunction("writeln", "Hello, world!"); } -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Oct 22 2011
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-21 21:07, Vladimir Panteleev wrote:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?

Very cool. -- /Jacob Carlborg
Oct 22 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/22/11, Vladimir Panteleev <vladimir thecybershadow.net> wrote:
 On Sat, 22 Oct 2011 13:50:32 +0300, Alex R=F8nne Petersen
 <xtzgzorex gmail.com> wrote:

 You could just introduce a -reflection switch to include reflection
 information.

As I mentioned in the previous thread on the subject, I think that ideall=

 we should improve compile-time reflection so it's possible to generate
 runtime reflection information at compilation time. Something like this:

 import reflection;

 enum reflectionForStdStdio =3D generateReflectionForModule("std.stdio");

 void main()
 {
      reflectionForStdStdio.callFunction("writeln", "Hello, world!");
 }

 --
 Best regards,
   Vladimir                            mailto:vladimir thecybershadow.net

I think user properties would be a perfect fit for this. :) reflected struct Foo {}
Oct 22 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:

 On 22-10-2011 15:20, Andrej Mitrovic wrote:
 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net>  wrote:
 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com>  wrote:

 You could just introduce a -reflection switch to include reflection
 information.

As I mentioned in the previous thread on the subject, I think that ideally we should improve compile-time reflection so it's possible to generate runtime reflection information at compilation time. Something like this: import reflection; enum reflectionForStdStdio = generateReflectionForModule("std.stdio"); void main() { reflectionForStdStdio.callFunction("writeln", "Hello, world!"); } -- Best regards, Vladimir mailto:vladimir thecybershadow.net

I think user properties would be a perfect fit for this. :) reflected struct Foo {}

The only downside with either approach

What do you mean by "either approach"? Which is the other one? -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Oct 22 2011
prev sibling next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sat, 22 Oct 2011 07:57:17 -0400, Daniel Gibson <metalcaedes gmail.com> wrote:
 Am 22.10.2011 05:48, schrieb Robert Jacques:

 What do you mean by their 'parameters'? What about overloads?
 Attributes? Arguments? Argument attributes?

Primarily arguments. That should help identifying overloads. But attributes and argument attributes are needed as well.

But handling overloads is the responsibility of the dispatch system. What would be the use cases for this information?
 Something that is close to what Java offers would be great.

And what, exactly does JAVA offer? What works? What doesn't work? What's missing?

See http://download.oracle.com/javase/6/docs/api/java/lang/Class.html You can access constructors (look for one by it's parameters or get all of them), the same for methods (methods the class declares and all methods, i.e. also inherited ones), implemented interfaces, fields, ...

Thanks, I'll have a look see.
 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).

1) Unused information is simply bloat: it increases exe size, slows the exe down and increases the runtime memory footprint.

Yeah, but (mostly) only per class, not per object (besides one pointer per object, which shouldn't be that bad and something like this seems is already there for the existing typeinfo)

I was assuming the per object bloat was zero. Oh, and I did forget that RTTI increases compile times.
Oct 22 2011
parent foobar <foo bar.com> writes:
Robert Jacques Wrote:

 On Sat, 22 Oct 2011 07:57:17 -0400, Daniel Gibson <metalcaedes gmail.com>
wrote:
 Am 22.10.2011 05:48, schrieb Robert Jacques:

 What do you mean by their 'parameters'? What about overloads?
 Attributes? Arguments? Argument attributes?

Primarily arguments. That should help identifying overloads. But attributes and argument attributes are needed as well.

But handling overloads is the responsibility of the dispatch system. What would be the use cases for this information?
 Something that is close to what Java offers would be great.

And what, exactly does JAVA offer? What works? What doesn't work? What's missing?

See http://download.oracle.com/javase/6/docs/api/java/lang/Class.html You can access constructors (look for one by it's parameters or get all of them), the same for methods (methods the class declares and all methods, i.e. also inherited ones), implemented interfaces, fields, ...

Thanks, I'll have a look see.
 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).

1) Unused information is simply bloat: it increases exe size, slows the exe down and increases the runtime memory footprint.

Yeah, but (mostly) only per class, not per object (besides one pointer per object, which shouldn't be that bad and something like this seems is already there for the existing typeinfo)

I was assuming the per object bloat was zero. Oh, and I did forget that RTTI increases compile times.

IMHO, RTTI should be a built-in feature of the language. 1) You need one global system agreed upon by all clients. Otherwise, you could end up in a situation where a single class has more than a single set of metadata used in several different scenarios. 2) This information is required anyway in the run-time itself. This is a requirement for an accurate GC. 3) Other tools might use this info as well, debuggers, IDEs, etc. 4) It's with the spirit of D design - make the common/safe case the default. As others mentioned, there should be a compiler switch to turn this off. E.g. when developing for an embedded system where memory is scarce it makes sense to not generate RTTI. It also makes sense to disable the GC and preallocate the memory.
Oct 22 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sat, 22 Oct 2011 06:51:00 -0400, Alex Rønne Petersen <xtzgzorex gmail.com>
wrote:
 On 22-10-2011 05:36, Robert Jacques wrote:
 On Fri, 21 Oct 2011 17:15:02 -0400, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 On 21-10-2011 21:07, Vladimir Panteleev wrote:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

Absolutely awesome!
 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

Yes, very much. I would recommend setting up a fork on GitHub, and then adding it to a branch, e.g. 'reflection', if it's not going to be included in mainline DMD.

Really? A runtime reflection system has been on the review queue for over six months and I've brought the subject up on the newsgroup. I've received zero feedback. So while I'm sure everyone want to check RTTI off the D features list, I've not seen much real interest in it.

I must have missed that. What should I search for to find your thread?

Don't bother; As I said, the thread went nowhere. However, if you'd like to look at the code / docs for my proposed improved variant module: https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht https://jshare.johnshopkins.edu/rjacque2/public_html/variant.d
Oct 22 2011
prev sibling next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sat, 22 Oct 2011 14:38:22 -0400, foobar <foo bar.com> wrote:
[snip]
 IMHO, RTTI should be a built-in feature of the language.
 1) You need one global system agreed upon by all clients. Otherwise, you could
end up in a situation where a single class has more than a single set of
metadata used in several different scenarios.

Or a singleton class/struct. Yes, you should have a single global system, but nothing says that the system has to be the language. Library solution are also valid.
 2) This information is required anyway in the run-time itself. This is a
requirement for an accurate GC.

a) Not every user of D wants to use a GC. In fact, there's currently a push to remove / improve all the GC mandatory features of D to support ref-counting / manual memory usage. b) RTTI is separate for GC BlockInfo. Yes, you could generate the GC_BlockInfo from the RTTI, but doing so would be so inefficient that its not worth contemplating.
 3) Other tools might use this info as well, debuggers, IDEs, etc.

Debuggers and IDEs already have all this information, and a lot more.
 4) It's with the spirit of D design - make the common/safe case the default.

Since when has reflection ever been the common/safe case? .tupleof, despite being a god-send, is one of the most unsafe features of D. My personal impression of reflection is that it's one of those features that you only want 1% of the time, but if you don't have it, you're going to be pulling teeth to work around it.
 As others mentioned, there should be a compiler switch to turn this off.
 E.g. when developing for an embedded system where memory is scarce it makes
sense to not generate RTTI. It also makes sense to disable the GC and
preallocate the memory.

Isn't this statement and statement 4) in conflict with each other? Unlike a GC, turning off RTTI breaks everything that uses it.
Oct 22 2011
parent foobar <foo bar.com> writes:
Robert Jacques Wrote:

 On Sat, 22 Oct 2011 14:38:22 -0400, foobar <foo bar.com> wrote:
 [snip]
 IMHO, RTTI should be a built-in feature of the language.
 1) You need one global system agreed upon by all clients. Otherwise, you could
end up in a situation where a single class has more than a single set of
metadata used in several different scenarios.

Or a singleton class/struct. Yes, you should have a single global system, but nothing says that the system has to be the language. Library solution are also valid.

But couldn't someone implement a different singleton class? How would you make sure the system remains global?
 
 2) This information is required anyway in the run-time itself. This is a
requirement for an accurate GC.

a) Not every user of D wants to use a GC. In fact, there's currently a push to remove / improve all the GC mandatory features of D to support ref-counting / manual memory usage.

Agreed and it is or should be made possible to not use the GC. However the default for D is GC.
 b) RTTI is separate for GC BlockInfo. Yes, you could generate the 
GC_BlockInfo from the RTTI, but doing so would be so inefficient that its not
worth contemplating.

I'm not sure I follow. Are you discussing D's current GC implementation? Generally speaking, an accurate GC AFAIK needs to have full type info. D's current GC does not have full type info and can have false positives hence it's a conservative GC.
 
 3) Other tools might use this info as well, debuggers, IDEs, etc.

Debuggers and IDEs already have all this information, and a lot more.
 4) It's with the spirit of D design - make the common/safe case the default.

Since when has reflection ever been the common/safe case? .tupleof, despite being a god-send, is one of the most unsafe features of D. My personal impression of reflection is that it's one of those features that you only want 1% of the time, but if you don't have it, you're going to be pulling teeth to work around it.

That's why it should be able to turn off RTTI generation if it is not needed but it should be provided by default in order to prevent that pulling teeth sensation. It's safer to provide an off switch rather than an on switch.
 
 As others mentioned, there should be a compiler switch to turn this off.
 E.g. when developing for an embedded system where memory is scarce it makes
sense to not generate RTTI. It also makes sense to disable the GC and
preallocate the memory.

Isn't this statement and statement 4) in conflict with each other? Unlike a GC, turning off RTTI breaks everything that uses it.

where's the conflict? If my application does not use RTTI at all than the 'redundant' info can be removed. In fact, Thinking about it just know, It should be a link-time optimization - only if the linker did not see any usage of RTTI in the program, it'll remove the generated meta data.
Oct 22 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sat, 22 Oct 2011 17:37:40 -0400, foobar <foo bar.com> wrote:

 Robert Jacques Wrote:

 On Sat, 22 Oct 2011 14:38:22 -0400, foobar <foo bar.com> wrote:
 [snip]
 IMHO, RTTI should be a built-in feature of the language.
 1) You need one global system agreed upon by all clients. Otherwise, you could
end up in a situation where a single class has more than a single set of
metadata used in several different scenarios.

Or a singleton class/struct. Yes, you should have a single global system, but nothing says that the system has to be the language. Library solution are also valid.

But couldn't someone implement a different singleton class?

Sure.
 How would you make sure the system remains global?

You put it in Phobos.
 2) This information is required anyway in the run-time itself. This is a
requirement for an accurate GC.

a) Not every user of D wants to use a GC. In fact, there's currently a push to remove / improve all the GC mandatory features of D to support ref-counting / manual memory usage.

Agreed and it is or should be made possible to not use the GC. However the default for D is GC.

But nothing depends on the GC. Things would depend on RTTI.
 b) RTTI is separate for GC BlockInfo. Yes, you could generate the 
GC_BlockInfo from the RTTI, but doing so would be so inefficient that its not
worth contemplating.

I'm not sure I follow. Are you discussing D's current GC implementation? Generally speaking, an accurate GC AFAIK needs to have full type info. D's current GC does not have full type info and can have false positives hence it's a conservative GC.

The type info used for the GC and the type info for reflection are two very different things. Saying that having one implies the other is fallacious. [snip]
 4) It's with the spirit of D design - make the common/safe case the default.

Since when has reflection ever been the common/safe case? .tupleof, despite being a god-send, is one of the most unsafe features of D. My personal impression of reflection is that it's one of those features that you only want 1% of the time, but if you don't have it, you're going to be pulling teeth to work around it.

That's why it should be able to turn off RTTI generation if it is not needed but it should be provided by default in order to prevent that pulling teeth sensation. It's safer to provide an off switch rather than an on switch.

So, let me get this right. If 99% of the time you don't want something, and 1% of the time you do, it's your opinion is that the default should be the 1% case? Because it's safer? By the way, I'm pretty sure we have very different meaning of the word 'safe' here.
 As others mentioned, there should be a compiler switch to turn this off.
 E.g. when developing for an embedded system where memory is scarce it makes
sense to not generate RTTI. It also makes sense to disable the GC and
preallocate the memory.

Isn't this statement and statement 4) in conflict with each other? Unlike a GC, turning off RTTI breaks everything that uses it.

where's the conflict?

Common features are by their very definition, pervasive and thus not disable-able. Hence, your statement 4 and 5 are in conflict with each other. Only one may be true.
 If my application does not use RTTI at all than the 'redundant' info can be
removed.

Generally speaking, no, actually it can't. I guess some form of LTO might be able to do this, but we are getting into the realm of requiring `a sufficiently smart compiler`.
 In fact, Thinking about it just know, It should be a link-time optimization -
only if the linker did not see any usage of RTTI in the program, it'll remove
the generated meta data.

Umm, no. That's not what LTO means. LTO means that instead of having a compiler and a linker, you just have the compiler-front end and the compiler back-end. And then, maybe, it can not generate the meta-data in the first place. Because the compiler never is going to solve the 'was this ever accessed' problem. Besides, D specifically is designed to allow 3rd party object files, obfuscation, etc, which really isn't compatible with LTO.
Oct 22 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 23 Oct 2011 01:54:52 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:

 On 22-10-2011 17:18, Vladimir Panteleev wrote:
 On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 On 22-10-2011 15:20, Andrej Mitrovic wrote:
 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net> wrote:
 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:

 You could just introduce a -reflection switch to include reflection
 information.

As I mentioned in the previous thread on the subject, I think that ideally we should improve compile-time reflection so it's possible to generate runtime reflection information at compilation time. Something like this: import reflection; enum reflectionForStdStdio = generateReflectionForModule("std.stdio"); void main() { reflectionForStdStdio.callFunction("writeln", "Hello, world!"); } -- Best regards, Vladimir mailto:vladimir thecybershadow.net

I think user properties would be a perfect fit for this. :) reflected struct Foo {}

The only downside with either approach

What do you mean by "either approach"? Which is the other one?

reflected and generateReflectionForModule. - Alex

I don't see how my generateReflectionForModule idea requires any modification of the reflected code. The module can generate RTTI for all types used recursively from the starting point. If more precision is needed, it can be specified as generateReflectionForModule/etc. parameters. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Oct 23 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 23 Oct 2011 14:06:42 +0300, Timon Gehr <timon.gehr gmx.ch> wrote:

 I don't see how my generateReflectionForModule idea requires any
 modification of the reflected code. The module can generate RTTI for all
 types used recursively from the starting point. If more precision is
 needed, it can be specified as generateReflectionForModule/etc.  
 parameters.

The module can generate RTTI for all types recursively from the starting point iff that information is statically available. It does not have to be. A module that comes as .di + static library binary could return a reference to a private class that has a publicly exported base class. How would you generate RTTI for a statically invisible class?

The only way to prevent such problems is to have RTTI generated by the compiler and enabled always or by default. However, is such functionality even desirable? It is equivalent to having a back door for accessing private class members (implementation details in general). -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Oct 23 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
[snip]
 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?

You're not supposed to be able to. Runtime reflection should only apply to public data members.
Oct 23 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 23 Oct 2011 12:17:45 -0400, Alex Rønne Petersen <xtzgzorex gmail.com>
wrote:
 On 23-10-2011 18:03, Robert Jacques wrote:
 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
 [snip]
 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?

You're not supposed to be able to. Runtime reflection should only apply to public data members.

I don't think that's necessarily a good idea. If you're doing something related to e.g. serialization and a type uses properties to hide fields (say, because the properties have in/out contracts), you would be unable to take those into account.

A serialized message is part of the public API of your program. It is exposed not only internally, but also to all those who interact with you. This includes malicious hackers; one only has to take a look at the history of JSON parses to see how serialization can be exploited on the web. Exposing private members publicly is an extremely unsafe operation and a violation of programmer intent; it should never be done auto-magically. If a type with private members wants to support serialization, it can add the appropriate custom routines and data verification checks to itself. If fact, it would seriously behoove us, if our serialization library defaulted to requiring that a custom validation check be run by each variable or one of its parents.
Oct 23 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 23 Oct 2011 18:20:32 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:

 On 23-10-2011 14:51, Vladimir Panteleev wrote:
 On Sun, 23 Oct 2011 14:06:42 +0300, Timon Gehr <timon.gehr gmx.ch>  
 wrote:

 I don't see how my generateReflectionForModule idea requires any
 modification of the reflected code. The module can generate RTTI for  
 all
 types used recursively from the starting point. If more precision is
 needed, it can be specified as generateReflectionForModule/etc.
 parameters.

The module can generate RTTI for all types recursively from the starting point iff that information is statically available. It does not have to be. A module that comes as .di + static library binary could return a reference to a private class that has a publicly exported base class. How would you generate RTTI for a statically invisible class?

The only way to prevent such problems is to have RTTI generated by the compiler and enabled always or by default. However, is such functionality even desirable? It is equivalent to having a back door for accessing private class members (implementation details in general).

Isn't that technically already possible by doing enough cast and pointer arithmetic hacks? I don't think it really makes a difference. If someone really wants to backdoor a type, they will find a way to do it.

Yes, but that's neither "safe" nor an actual language feature. Allowing RTTI to do the same would be both: my point was whether there should be a language feature for this (exploring unexposed types). -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Oct 23 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 23 Oct 2011 13:48:03 -0400, Jacob Carlborg <doob me.com> wrote:
 On 2011-10-23 18:03, Robert Jacques wrote:
 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
 [snip]
 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?

You're not supposed to be able to. Runtime reflection should only apply to public data members.

It's not enough. Take this for example: class Foo < ActiveRecord::Base before_save :bar private def bar end end The above code is an example from Ruby on Rails. The "before_save" call sets up a callback, "bar", that will called before saving the model. The callback method will be called using reflection.

Using a short, cryptic stub from another programming language is not an effective communication tool. Case in point: all I understand is that something is returning a delegate to something, and reflection never enters the picture. But, for sake of argument, let me assume that ActiveRecord needs to reflect Foo for rails to work (i.e. ActiveRecord::Base's ctor reflects this). At a very fundamental level, this isn't an argument for or against RTTI in D; in D you'd do this with a mixin of some sort. When looking a use cases/features from other languages, always remember to first ask oneself a) how would I do it in D today? b) Is language X's solution 'better, faster, stronger' in some way? I mean, considering that Ruby is a dynamic language, would bar even be considered private from inside the super-class's ctor? Is Ruby's private even comparable to D's private? Given that Ruby, by design, requires fields to be private and for all external accesses to happen via methods, should we consider a way to violate that contract an example of mis-feature?
Oct 23 2011
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2011 12:07 PM, Vladimir Panteleev wrote:
 Walter: would it be okay if the compiler changes were published as a GitHub
 fork, or should we stick to patches?

I'd prefer as a patch, because forking the language can result in another Phobos v Tango debacle, where everyone loses.
Oct 23 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-10-23 22:14, Walter Bright wrote:
 On 10/21/2011 12:07 PM, Vladimir Panteleev wrote:
 Walter: would it be okay if the compiler changes were published as a
 GitHub
 fork, or should we stick to patches?

I'd prefer as a patch, because forking the language can result in another Phobos v Tango debacle, where everyone loses.

I though we used forks on GitHub with pull requests these days. -- /Jacob Carlborg
Oct 24 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 21 Oct 2011 23:48:51 -0400, Robert Jacques <sandford jhu.edu>  
wrote:

 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson  
 <metalcaedes gmail.com> wrote:
 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?

I'd love to see proper runtime reflection support in D, including functionality to get information about available methods (their name and parameters) and a way to call them.

What do you mean by their 'parameters'? What about overloads? Attributes? Arguments? Argument attributes?
 Something that is close to what Java offers would be great.

And what, exactly does JAVA offer? What works? What doesn't work? What's missing?
 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).

1) Unused information is simply bloat: it increases exe size, slows the exe down and increases the runtime memory footprint. 2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets) memory and disk space are both highly constrained resources that you don't want to waste. 3) RTTI provides a back-door into a code-base; one that for many reasons you may want to keep closed.

I agree to all these points. I think runtime reflection should be generated by the compiler as needed. And the mechanism to determine "as needed" should be an attribute IMO, or an intrinsic function (i.e. if you have all the static information about a type, you should be able to generate the RTTI for it). I do not want programs getting bigger than they already are for the sake of "just in case I want to use reflection"... -Steve
Oct 24 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 24 Oct 2011 03:30:02 -0400, Jacob Carlborg <doob me.com> wrote:
 On 2011-10-23 20:54, Robert Jacques wrote:
 On Sun, 23 Oct 2011 13:48:03 -0400, Jacob Carlborg <doob me.com> wrote:
 On 2011-10-23 18:03, Robert Jacques wrote:
 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch>
 wrote:
 [snip]
 The module can generate RTTI for all types recursively from the
 starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?

You're not supposed to be able to. Runtime reflection should only apply to public data members.

It's not enough. Take this for example: class Foo < ActiveRecord::Base before_save :bar private def bar end end The above code is an example from Ruby on Rails. The "before_save" call sets up a callback, "bar", that will called before saving the model. The callback method will be called using reflection.

Using a short, cryptic stub from another programming language is not an effective communication tool. Case in point: all I understand is that something is returning a delegate to something, and reflection never enters the picture.

What's happening is that the class method "before_save" is called with the symbol (think string) "bar". Then later ActiveRecord::Base will call the method corresponding to the string "bar" using reflection. In D you would proabably do something like: class Foo : ActiveRecord.Base { private void bar () {} mixin beforeSave!(bar); } Now you still won't be able to call "bar" outside "Foo" if "bar" is private.

Thanks. But how does super know to call "bar"? And if you can pass information to super, couldn't you just pass a delegate? And isn't passing information to super simply passing information to yourself?
 But, for sake of argument, let me assume that ActiveRecord needs to
 reflect Foo for rails to work (i.e. ActiveRecord::Base's ctor reflects
 this). At a very fundamental level, this isn't an argument for or
 against RTTI in D; in D you'd do this with a mixin of some sort. When
 looking a use cases/features from other languages, always remember to
 first ask oneself a) how would I do it in D today? b) Is language X's
 solution 'better, faster, stronger' in some way?

You can think of it like this, just because you haven't come up with a reason for calling privte methods using reflection doesn't mean anyone else won't come up with a good reason. I tried here to give a reason.

I'm sorry if I've come across that way. I'm well aware with the reasons for wanting access to private methods/fields via reflection and have mentioned them in previous posts. What I've tried to point out is very eloquently stated by fig. 6.3 on page 204 of TDPL: allowing reflection to bypass protection guarantees essentially makes all declarations public in nature, if not extern. That's a very heavy price to pay, just from a program maintenance perspective. And if you consider someone writing medical or financial software, the privacy concerns of exposing private variable to all become very real. Back to my actual argument, what I was pointing out was that the toolboxes of various languages differ heavily. Just because Ruby has a very cool library which uses runtime reflection, one shouldn't simply state 'D must have runtime reflection.' One needs to first decompose what the library is trying to achieve from the language mechanisms used to implement it. Then you can ask yourself 1) Can I do that in D? 2) How does D's solution compare to Ruby's? My impression of what ActiveRecord, (which my be wrong), is that its using runtime-reflection to do various meta-programming activities and that under-the-hood, it stretching Ruby to the breaking point in order to do so. D, on the other hand, has a ton of meta-programming specific features, and doesn't need to rely on reflection to do so. This is partly to do with Ruby supporting 'eval' and thus making no distinction between compile-time and run-time behavior; that's correct for Ruby, but D is a different kettle of fish.
 I mean, considering that Ruby is a dynamic language, would bar even be
 considered private from inside the super-class's ctor? Is Ruby's private
 even comparable to D's private? Given that Ruby, by design, requires
 fields to be private and for all external accesses to happen via
 methods, should we consider a way to violate that contract an example of
 mis-feature?

There is no constructor involved here. "before_save" is class method called when the class "Foo" is loaded.

So it's the static ctor, then.
 You can do basically whatever you want with the code in Ruby. You can
 replace an existing method/class in the standard library with your own.
 You can easily add methods to existing classes. It's the programmer that
 is responsible to not mess up things, just as in D where you can cast
 away immutable, shared and do unsafe things with pointers.

Well one one hand, none of those things are allowed in SafeD (And you know, that's not a resounding argument for reflection of private members). On the other, you didn't answered my other two questions. In fact, the impression I'm getting is that Ruby's private isn't comparable to D's private.
Oct 24 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 24 Oct 2011 11:18:34 -0400, deadalnix <deadalnix gmail.com> wrote:
[snip]

 Well, if the compile time reflection is good enough, you can do this
 without any runtime reflexion support in the compiler, just using a lib.

 Why not support this option ?

Some of us are: https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht
Oct 24 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 24 Oct 2011 11:18:34 -0400, deadalnix <deadalnix gmail.com> wrot=
e:

 Le 24/10/2011 16:20, Steven Schveighoffer a =C3=A9crit :
 On Fri, 21 Oct 2011 23:48:51 -0400, Robert Jacques <sandford jhu.edu>=


 wrote:

 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:
 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds =





 RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info =3D  =





 cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name =3D=3D "PrintHello");
 auto print =3D cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remain=





 disputed matter, would anyone be interested in an unofficial patch=





 for
 this?

 Walter: would it be okay if the compiler changes were published as=





 GitHub fork, or should we stick to patches?

I'd love to see proper runtime reflection support in D, including functionality to get information about available methods (their nam=




 and
 parameters) and a way to call them.

What do you mean by their 'parameters'? What about overloads? Attributes? Arguments? Argument attributes?
 Something that is close to what Java offers would be great.

And what, exactly does JAVA offer? What works? What doesn't work? What's missing?
 BTW: I don't really see the problem with providing this information=




 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it  =




 (other
 types don't even need that because they're not polymorphic and - li=




 methods of structs - the address of the information is known at
 compile-time).

1) Unused information is simply bloat: it increases exe size, slows the exe down and increases the runtime memory footprint. 2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets) memory and disk space are both highly constrained resources=



 that you don't want to waste.
 3) RTTI provides a back-door into a code-base; one that for many
 reasons you may want to keep closed.

I agree to all these points. I think runtime reflection should be generated by the compiler as needed. And the mechanism to determine "=


 needed" should be an attribute IMO, or an intrinsic function (i.e. if=


 you have all the static information about a type, you should be able =


 generate the RTTI for it). I do not want programs getting bigger than=


 they already are for the sake of "just in case I want to use  =


 reflection"...

 -Steve

Well, if the compile time reflection is good enough, you can do this =

 without any runtime reflexion support in the compiler, just using a li=

 Why not support this option ?

As others have mentioned, there should be a standard way so we don't hav= e = conflicting ways that don't mix. Plus, it seems logical tome since the compiler is generating *some* RTTI= , = it should generate *all* of it (when asked to do so). -Steve
Oct 24 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 24 Oct 2011 16:34:11 -0400, Daniel Gibson <metalcaedes gmail.com> wrote:
 Am 24.10.2011 17:37, schrieb Robert Jacques:
 That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.

I don't think marking a method as private is a good way to hide critical information. If you can actually call methods (like in real calls, not some kind of webservice), you can as well use pointers to read the memory directly. Or he could run it in a debugger. Or (with root privileges) just look at the whole memory of the process. Or ... Cheers, - Daniel

By your argument, because can ever be truly secure there's no point to trying to secure it. I don't buy that. There is a world of difference between obj.reflect("SSN"), and trying to figure out 1) what objects hold SSN numbers 2) what's its field offset is and 3) what its memory representations is. (because remember, you don't have the source code) And that's assuming you can compile D code against the library and the library writer has directly exposed the critical object. More likely, you'll have to repeat steps 1-3 multiple times to find the data you're after. And runtime reflection means access will get exposed across language barriers, e.g. in your application's macro language. Furthermore, security focused languages seem to be centering around the concept of capabilities, of which private, package, protected, public and extern are the most rudimentary form of. Getting back to your original point, I wasn't suggesting that private is a panacea for security concerns, but it is one of the more powerful tools in our tool box.
Oct 24 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 24 Oct 2011 12:38:12 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-24 17:37, Robert Jacques wrote:
 I'm sorry if I've come across that way. I'm well aware with the reasons
 for wanting access to private methods/fields via reflection and have
 mentioned them in previous posts. What I've tried to point out is very
 eloquently stated by fig. 6.3 on page 204 of TDPL: allowing reflection
 to bypass protection guarantees essentially makes all declarations
 public in nature, if not extern. That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.

If someone is using a library and starts to mess with calling private methods they shouldn't they'll most likely mess things up for them self.

Aren't private methods by definition something people shouldn't be using?
 Back to my actual argument, what I was pointing out was that the
 toolboxes of various languages differ heavily. Just because Ruby has a
 very cool library which uses runtime reflection, one shouldn't simply
 state 'D must have runtime reflection.' One needs to first decompose
 what the library is trying to achieve from the language mechanisms used
 to implement it. Then you can ask yourself 1) Can I do that in D? 2) How
 does D's solution compare to Ruby's?

Of course. I'm just showing that another language/library uses it and therefore there could be a chance that should be doable in D as well.

I understand and appreciate your intent. However, do so should be accompanied by a critical dissection of the language/library, both to get those unfamiliar up to speed and to frame the conversation in terms of how best can we improve D.
 My impression of what ActiveRecord, (which my be wrong), is that its
 using runtime-reflection to do various meta-programming activities and
 that under-the-hood, it stretching Ruby to the breaking point in order
 to do so. D, on the other hand, has a ton of meta-programming specific
 features, and doesn't need to rely on reflection to do so. This is
 partly to do with Ruby supporting 'eval' and thus making no distinction
 between compile-time and run-time behavior; that's correct for Ruby, but
 D is a different kettle of fish.

I doubt that ActiveRecord is stretching Ruby to the breaking point.

:) I don't know myself. I've just heard that the internals of Rails are deep, dark and full of non-idiomatic Rudy code.
 On
 the other hand it feels like I'm stretching D's metaprogramming
 capabilities in my serialization library. That's just because D doesn't
 have proper reflection.

I've written three serialization libraries and two versions of a runtime reflection library; the first one was prior to the string bugs in CTFE being fixed and ran DMD out of memory. I found serialization to be simple and straight forward. And I found compiler bugs to be a greater issue with reflection than any limits in D's capabilities. Indeed, given I wrote the same library using two completely different mechanisms, gives me great hope for D. So, I don't agree with your assessment, but I do understand where you're coming from. A lot of D's metaprogramming capabilities are undocumented, and knowing the right tool for the job is essential to getting nice, clean code.
 In Ruby there is very seldom reason to use "eval". That's because of the
 excellent support for runtime reflection. There's no clumsy syntax, no
 hacks, no need to instantiate reflectors and similar. You just call a
 method like you would call any other method:

 class Foo
      def bar
      end
 end

 foo = Foo.new
 foo.bar()
 foo.send("bar")

 In the above code, "bar" is called first using regular method call
 syntax and then using reflection.

 I mean, considering that Ruby is a dynamic language, would bar even be
 considered private from inside the super-class's ctor? Is Ruby's private
 even comparable to D's private? Given that Ruby, by design, requires
 fields to be private and for all external accesses to happen via
 methods, should we consider a way to violate that contract an example of
 mis-feature?

There is no constructor involved here. "before_save" is class method called when the class "Foo" is loaded.

So it's the static ctor, then.

I guess you could call it that.

(Just checking, but you do know D has static ctors.)
 You can do basically whatever you want with the code in Ruby. You can
 replace an existing method/class in the standard library with your own.
 You can easily add methods to existing classes. It's the programmer that
 is responsible to not mess up things, just as in D where you can cast
 away immutable, shared and do unsafe things with pointers.

Well one one hand, none of those things are allowed in SafeD (And you know, that's not a resounding argument for reflection of private members). On the other, you didn't answered my other two questions. In fact, the impression I'm getting is that Ruby's private isn't comparable to D's private.

If we talk about methods it's easiest. If you call a private method in Ruby using the regular method call syntax you get an error. If you do the same in D you get an error as well. The only difference is in Ruby it will be a runtime error and in D it will be a compile time error. In Ruby you can bypass private using reflection with the "send" method. In D you can bypass private using a delegate.

Actually, you're not supposed to be able to; this is bug 1983. http://d.puremagic.com/issues/show_bug.cgi?id=1983 and https://github.com/D-Programming-Language/dmd/pull/71
 In Ruby an instance variable is always private. I don't think you can
 declare a public instance variable, one uses setters/getters for that.
 In D you can declare both private and public instance variables.

 In Ruby you can get/set a value of an instance variable using
 reflection, instance_variable_set/get.
 In D you can get/set a value of a private instance variable using
 compile time reflection, i.e. .tupleof.

Yes, but only for source code you control.
 Now, since we can call private methods and get/set private instance
 variables using the features we have now, why shouldn't we be able to do
 the same with runtime reflection?

 BTW, I don't think it's a mis-feature.

Actually, baring bugs in DMD, .tupleof is the only way to access private fields and there is _no way_ to access private members; And .tupleof only works if you have access to the source code. Furthermore, .tupleof, god-send that it is, isn't part of the D spec; unlike .stringof, it didn't make it into TDPL nor into the D documentation. D's official method of compile-time reflection is __traits. And __traits doesn't bypass protection attributes. So precedence would be against runtime-reflection having access to private members.
Oct 24 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
I said it once, but nobody responded.

Why not generate full reflection for everything by default and add an
attribute line  noreflect as an _optimization_ tool in cases, where
this thing is definitely not gonna be used in reflection?
Oct 25 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
You can always write  noreflect: at the start of your module and it
will work as expected. what's the problem with that?
Also, compiler would be able to automatically add that attribute to
any symbol, which is private and does not escape the module.
Again, not including reflection is purely _optimization_ issue and
should not be interfered with it's functionality.
Reflection is something you rarely have in mind when designing, for
example, classes.
Requiring explicit request for reflection defeats the purpose.

You could just as well say "writing public on every member is a
clutter". It is, but you don't have to do it to achieve the same
effect.

On Tue, Oct 25, 2011 at 2:58 PM, Timon Gehr <timon.gehr gmx.ch> wrote:
 On 10/25/2011 11:24 AM, Gor Gyolchanyan wrote:
 I said it once, but nobody responded.

 Why not generate full reflection for everything by default and add an
 attribute line  noreflect as an _optimization_ tool in cases, where
 this thing is definitely not gonna be used in reflection?

As I see it, because you almost never need 'full reflection for everything', and idiomatic D code is not supposed to contain clutter like that.

Oct 25 2011
prev sibling next sibling parent reply Jonny Dee <jonnyd gmx.net> writes:
Hi,

I would love to see runtime reflection be available in D. I am currently
reading Walter Bright's "The D
Programming Language" and with every page I read I became more and more
fascinated and enthusiastic
about D. But then I started to wonder why I haven't read about runtime
reflection yet, so I did some Internet
research and was a bit disappointed this feature isn't available. D is really
cool, it's not only C done right,
but also Java and C#. Actually, to me, it's one of the best programming
language available yet. It's really a
pitty RTTI is not available. So I hope your approach will somehow influence D's
future :)

Keep going,
Jonny
Oct 25 2011
parent reply Jonny Dee <jonnyd gmx.net> writes:
Am 25.10.11 16:41, schrieb Robert Jacques:
 On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:
 Hi,

 I would love to see runtime reflection be available in D. I am
 currently reading Walter Bright's "The D
 Programming Language" and with every page I read I became more and
 more fascinated and enthusiastic
 about D. But then I started to wonder why I haven't read about runtime
 reflection yet, so I did some Internet
 research and was a bit disappointed this feature isn't available. D is
 really cool, it's not only C done right,
 but also Java and C#. Actually, to me, it's one of the best
 programming language available yet. It's really a
 pitty RTTI is not available. So I hope your approach will somehow
 influence D's future :)

 Keep going,
 Jonny

Hi Jonny, Well, some RTTI is available. How much RTTI is needed, really depends on what you're trying to do. Therefore whether D has or hasn't got RTTI depends heavily on what any particular person means by RTTI. Do you have some use cases in mind? Maybe prototype objects, duck-typing/casting or serialization? Or perhaps something you wrote? Sorry for all the questions, but I feel as if we don't have a resident reflection expert in the dialog, or even the opinions of basic/experienced users. We all know that 'RTTI' is good, but we're a little vague on the how and why. (Well, beyond the use cases above)

Hi Robert, Well, before I tell you what I would like to see I'll cite Wikipedia [1]: " [...] - Discover and modify source code constructions (such as code blocks, classes, methods, protocols, etc.) as a first-class object at runtime. - Convert a string matching the symbolic name of a class or function into a reference to or invocation of that class or function. [...] " Here is what I would dream of for arbitrary objects/classes (not necessarily known at compile-time): - Query an object for its list of methods together with their signatures. Select a method, bind some values to its arguments, call it, and retrieve the return type (if any). - Query an object for its public fields (at least), and provide a way to get/set their values. - Query an object's class for all implemented interfaces and its base class. - Query a module for all type definitions and provide a way to introspect these types in more detail. For instance, it would be really cool if I could find a class with name "Car" in module "cars", get a list of all defined constructors, select one, bind values to the constructor's parameters, and create a corresponding object. Now you might ask where is the use case. I think, a very important one, besides creating GUI designers, is to be able to implement a Dependency Injection container [2,3,4] like, for example, the one provided by the Spring Framework for Java or .NET. In short, they allow you define how object trees should be build up solely by specifying this tree in an XML file. Within the XML file you can create new instances of classes of arbitrary types. You can specify which constructors to use for instantiation, and what values the constructor's arguments (if any) should have. You can even bind such an argument to an object previously created by the XML definition. You can call methods on existing objects, e.g. in order to call setter to further initialize an object with values you also define in that XML file. That XML file can be loaded when your program starts an make the DI container component instantiate all needed objects for you. There is no need for recompilation if the XML file is changed. Just restart your application. Implementing such a DI container heavily depends on reflection, because the DI container component doesn't know anything about the objects to be created during runtime. Qt also extends C++ with a reflection mechanism through the help of its meta object compiler (moc). It analyses the C++ source code, generates meta class definitions [6,7] and weaves them into your Qt class. Hence, in Qt, you can query an object for fields, methods, interfaces, etc. and you can call methods with arbitrary parameters, or you can instantiate a class using an arbitrary constructor. Consequently, somone implemented a DI container for C++ which is based on Qt and works more or less the same way the Spring DI container does. You can build up object trees simply by specifying such trees in an XML file. I don't go into why dependency injection is a very powerful feature. This is Martin Fowler's [3] job ;) But when I program with C++ I miss such a flexible dependency injection mechanism a lot. And I hope this will eventually be available for D. Cheers, Jonny [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29 [2] http://en.wikipedia.org/wiki/Dependency_injection [3] http://martinfowler.com/articles/injection.html [4] http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29 [5] http://qtioccontainer.sourceforge.net/ [6] http://doc.qt.nokia.com/stable/qmetaobject.html [7] http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx
Oct 25 2011
parent reply Jonny Dee <jonnyd gmx.net> writes:
Hello Robert,

Am 26.10.11 07:16, schrieb Robert Jacques:
 On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd gmx.net> wrote:
 Am 25.10.11 16:41, schrieb Robert Jacques:
 On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:
 [...]


Well, before I tell you what I would like to see I'll cite Wikipedia [1]: " [...] - Discover and modify source code constructions (such as code blocks, classes, methods, protocols, etc.) as a first-class object at runtime. - Convert a string matching the symbolic name of a class or function into a reference to or invocation of that class or function. [...] " Here is what I would dream of for arbitrary objects/classes (not necessarily known at compile-time): - Query an object for its list of methods together with their signatures. Select a method, bind some values to its arguments, call it, and retrieve the return type (if any). - Query an object for its public fields (at least), and provide a way to get/set their values. - Query an object's class for all implemented interfaces and its base class. - Query a module for all type definitions and provide a way to introspect these types in more detail. For instance, it would be really cool if I could find a class with name "Car" in module "cars", get a list of all defined constructors, select one, bind values to the constructor's parameters, and create a corresponding object. [...] Implementing such a DI container heavily depends on reflection, because the DI container component doesn't know anything about the objects to be created during runtime. Qt also extends C++ with a reflection mechanism through the help of its meta object compiler (moc). It analyses the C++ source code, generates meta class definitions [6,7] and weaves them into your Qt class. Hence, in Qt, you can query an object for fields, methods, interfaces, etc. and you can call methods with arbitrary parameters, or you can instantiate a class using an arbitrary constructor. Consequently, somone implemented a DI container for C++ which is based on Qt and works more or less the same way the Spring DI container does. You can build up object trees simply by specifying such trees in an XML file. I don't go into why dependency injection is a very powerful feature. This is Martin Fowler's [3] job ;) But when I program with C++ I miss such a flexible dependency injection mechanism a lot. And I hope this will eventually be available for D. Cheers, Jonny [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29 [2] http://en.wikipedia.org/wiki/Dependency_injection [3] http://martinfowler.com/articles/injection.html [4] http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29 [5] http://qtioccontainer.sourceforge.net/ [6] http://doc.qt.nokia.com/stable/qmetaobject.html [7] http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx

Hi Jonny, Thank you for your informative (and well cited) post. It has provided me with a new take on an old design pattern and some enjoyable reading. In return, let me outline my opinion of reflection in D today, and tomorrow, as it pertains to your wish list.

Many thanks to you, too, for your very elaborate answer :)
 Reflection in D today is very different from the host of VM languages
 that have popularized the concept. Being a compiled systems language,
 actual runtime self-modification is too virus like to become at a
 language level feature. However, given the compilation speed of D,
 people have made proof of concept libraries that essentially wrapped the
 compiler and dynamically loaded the result. As LDC uses LLVM, which has
 a jit backend, I'd expect to see something get into and D 'eval' library
 into etc eventually. (phobos uses the BOOST license, which isn't
 compatible with LLVM).

I know, that "runtime self-modification" and runtime code generation is a "dangerous" feature. And there really are rare cases where using such an approach might justify the risc in using it. Although this feature is not on my wish list, it might be good for generating dynamic proxies to arbitrary object instances like they are used by some ORMs. See Hibernate/NHibernate, for example [1,2]. Another example is aspect-oriented programming. But while I can't see the exacty reason for it, such a feature might indeed be a feature which is more appropriate for VM languages.
 Compile-time reflection and generation of code, on the other hand, is
 something D does in spades. It fulfills your dream list, although I
 think module level reflection might only be available in the github
 version. The API design is still in flux and we are actively iterating /
 improving it as find new uses cases and bugs. The current plan is to
 migrate all the traits functions over to a special 'meta' namespace
 (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid
 libraries for each of the concepts I listed, (prototype objects,
 duck-typing/casting or serialization), have been written using the
 compile-time meta-programming features in D. So that's the good.

A absolutely agree! D's compile-time reflection is very good. And it's hard to top this.
 On the other hand, D's runtime capabilities are limited to
 object.factory, the under implemented RTTI and library solutions which
 manually expose information gathered by D's compile-time mechanisms. And
 so far, these tools have been more than enough, from a functionality
 point of view. Most of our desire for better runtime reflection stems
 from a desire for efficiency, composition, cleanliness of user syntax
 and simplification of library code. These are all important issues for
 the widespread use of reflection based libraries, but they're not 'I
 can't implement X' issues.

You are right, there IS certainly always some way to 'implement X'. But, as you know, there is always a consideration of the effort you need to implement X.
 As for the future, I believe that the division in D between compile-time
 and run-time reflection warrants a serious look at the design of the
 run-time half of the system. To that end, I believe that implementing
 reflection in a library would be the best way to experiment and iterate
 an API. To that end, I have a proposal in the review queue to improve
 std.variant which contains dynamic dispatch (i.e. the ability to get/set
 public fields and call methods), duck-typeing/casting and
 prototype-style objects. Notably, it's missing introspection
 capabilities as thats what I'm most unsure about API wise, and simplest
 to add. Designing reflection inside a library keeps Walter & Co's
 bandwidth free for other bugs/features and provides a very good stress
 test of D's meta-programming capabilities. (Both of which I think are
 good things)

I've got no problem with an approach which puts runtime reflection capabiities into a separate library. No matter were you look at, Java, C#, or Qt, all have a library for reflection purpose. I do not see, however, how this might be done without compiler support. As already mentioned, Qt has its own moc compiler, which parses your C++ source code and generates the necessary infra structure. I'm still a beginner with respect to D, so I don't know what is really already possible, and what not. But my current feeling is that a similar approach would also be needed for D. If the D compiler itself, or another post-compiler component should generate the meta information for runtime reflection is another question. I could live with both, although I'd prefer the former built-in one.
 (I am soliciting feedback, if you care to take a look:
 https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)

Thanks for this link, I'll certainly have a look at it.
 And there are many reasons we might want to experiment with D's runtime
 reflection API instead of just copying someone. For example, take
 Dependency Injection. Using Java-style reflection, DI suffers from the
 Reflection injection and Unsafe Reflection security vulnerabilities. To
 do DI safely, you have to validate your inputs and D's compile-time
 reflection provides a perfect way to implement validated DI. Every time
 I hear about some form of injection, be it SQL or JSON or Reflection,
 hit the news, makes me think that 5-10 years from now well look back on
 the use of unvalidated dynamic code constructs the same way we do about
 null terminated arrays today.

Considering security vulnerabilities is of course a very important matter. However, I think security must be assured a by software's design. Enforcing it solely by programming language constructs will not work for all cases. Particularly not, if this programming language allows direct access to a computer's memory, like C, C++, and D does. There is no sandbox out-of-the-box, where the compiled program runs in. So if you have a private field in a class that carries a password, one has to make sure it's only in memory as long as it is required. And if it is not required anymore one should clear it out with zeros, for example. So making a field private is by no means a secure solution. A hacker will not give up just because the official API declares a field to be private. I consider 'unsafe reflection', as you call it, as a tool. Let's compare it to a knife, for instance. It is really a useful tool, I think nobody doubts, but at the same time it can be very dangerous. But inspite of this fact, everybody has not only one at home, I guess. Pointers are a useful tool, too. But they are also dangerous and still available in D, which is a good thing, because you can use this tool where needed. And actually, I don't think runtime reflection must be unsafe. The reflection mechanism provides type information for an object's properties, functions, arguments, etc. So validated DI is even possible with runtime reflection. Let's consider the the XML configuration of object trees once again. If you want to store a string to an int-field, for instance, then the DI container can refuse to do this, because it has access to all information required to enforce correct value types. One more use case for reflection is data binding with GUI components. This approach is heavily used in Windows Presentation Foundation library [3,4]. GUI components can update an object's properties by using reflection. You don't need to register listeners for this purpose anymore. That said, I don't think dynamic code constructs will be old-fashioned in 5-10 years, because you don't have to go an "unvalidated dynamic" way with runtime reflection. BTW, as this thread also discusses an opt-in or opt-out implementation for runtime reflection. I'd prefer an opt-out way, too. Code bloat (I guess the binaries are meant) is not as bad as it might sound with today's memory sizes. And if one wants to avoid it for optimization purpose, one can do it. The use of D as a systems programming language for embedded systems is, as I've read somewhere, not a first citizen anyway, because you'll get a lot of code for the garbage collection mechanism. Cheers, Jonny [1] http://en.wikipedia.org/wiki/Hibernate_%28Java%29 [2] http://en.wikipedia.org/wiki/Nhibernate [3] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation [4] http://blogs.msdn.com/b/wpfsdk/archive/2006/10/19/wpf-basic-data-binding-faq.aspx
Oct 26 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-26 20:20, Jonny Dee wrote:
 One more use case for reflection is data binding with GUI components.
 This approach is heavily used in Windows Presentation Foundation library
 [3,4]. GUI components can update an object's properties by using
 reflection. You don't need to register listeners for this purpose anymore.

Runtime reflection is used quite a lot in GUI programming on Mac OS X (at least the libraries/tools under the hood). -- /Jacob Carlborg
Oct 26 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-26 21:53, Adam Wilson wrote:
 It's also interesting that you bring up WPF because the project I lead,
 The Horizon Project, is an attempt to create a open-source
 multi-platform framework that is as easy to use as the .NET Framework
 and one of it's core pillars is a declarative XML based UI design
 language. I am evaluating D and so far it is blowing C++ out of the
 water, (we'll need to have long conversations about strings though, I
 think .NET has the right answer here) but I have to admit that I am
 seriously missing Reflection. The problem is that there is currently no
 way to do data-binding without member-level reflection because it is a
 purely run-time concept. That said, I am researching ways around this,
 and I do have ideas ... but that's not for this forum.

Are you saying that you consider using D for this Horizon project? I can recommend you take a look at DWT: www.dsource.org/projects/dwt Somewhere down the road I've planed to create an interface/window builder for DWT using XML or something similar. I'm thinking something like how it works on Mac OS X using Interface Builder.
 However, I see no particular reason to reflect on private members by
 default, if you need to get data out, you should be using a public
 property or method to filter the data as needed, that's kind of the
 point of scoping. Private member reflection is NOT a requirement for UX,
 WPF being the case in point. I personally like the solution that
 Reflection in D is opt-out with both command-line options and
  noreflect. I would add that a  deepreflect option be made available for
 those incorrigible souls who absolutely must reflect private members. I
 would also add that reflection should be default disabled by the
 compiler for code the works directly with pointers and direct memory
 access (malloc/free, new/delete, etc.) and the developer can only enable
 it using a compiler option.

I don't understand this one. Should the compiler disable reflection as soon as it sees malloc/free? On what level should it be disabled? I mean, the runtime needs to be able to use these functions to implement the memory manager, i.e. the garbage collector and other things as well. -- /Jacob Carlborg
Oct 27 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-27 20:42, Adam Wilson wrote:
 On Thu, 27 Oct 2011 00:14:35 -0700, Jacob Carlborg <doob me.com> wrote:

 Are you saying that you consider using D for this Horizon project? I
 can recommend you take a look at DWT: www.dsource.org/projects/dwt

 Somewhere down the road I've planed to create an interface/window
 builder for DWT using XML or something similar. I'm thinking something
 like how it works on Mac OS X using Interface Builder.

I am, and I have looked at DWT. My problem with it is a one that is endemic to open-source UI framework. Microsoft recognized a decade ago that UI widgets whose look-and-feel is defined and controlled by the Operating System are going the way of the dodo. Out of that realization WPF was born. UI designers today want the ability to control every pixel of the UI's presentation. And the reasons for this are two-fold. The first is that it turns out that most OS designers are fantastically bad at UI and design in general. It takes epic piles of cash to pull of a decent one and even then there is a still a "programmers were here" look to it (with the notable exception of iOS/OSX where designers rule the roost).

I can agree to that.
 The second is product differentiation. Nobody wants an app that
 looks like every other app because it actually becomes impossible for
 the user to distinguish which app works best for them.

I want that. Because I know how the GUI works and it will be easy for me to learn if it follows the guidelines of the platform. Also I don't need to figure out if I can use the scroll wheel on the mouse on this, what looks like a, scroll bar. If the application have a native look and feel I know how to use the widgets.
 Users ONLY look
 at the UI, and if the app doesn't look good, they wont "buy" it, even if
 it's free. This is non-negotiable. Users, when given two apps that do
 the same thing, even for different prices, will pick the prettier one
 every time, because the prettier one is perceived as being "better".
 It's called the Attractiveness Bias and it is a well-known principle in
 the design world. Who would you rather look at all day, Alessandra
 Ambrosio or Rosie O'Donnell? I rest my case.

That's why I use Mac OS X where the native applications look good :)
 I maintain that this is prime reason that Linux on the desktop has
 failed miserably, and I think Android proves my point. Android's key win
 is that it put a usable UI on top of Linux. People never had a problem
 with the price of Linux, they just couldn't stand to look at it. The
 Linux Desktop LOOKS industrial and it's apps for the most part look the
 same (I know of a few outliers that did a good job, but it isn't the
 norm).

I can agree to that.
 My point is that the day of cookie cutter apps is over. Anyone
 designing for that paradigm is history. Microsoft's latest UI paradigm,
 "Metro", is just a different cut-down version of WPF similar to
 Silverlight. Microsoft has no plans to go back to OS controlled UI
 styles; Metro and WPF are the plan for the next 15 years. (I attended
 the MS BUILD conference, they made this plan very ... nay, EXTREMELY
 clear).

I don't know how Metro or WPF is implemented but how says they can't be the native look and feel of the OS.
 Open-source is chronically behind the big boys with money, precisely
 because FOSS doesn't have the money to sling around for Testing and
 Usability Studies, and most FOSS guys don't want to mess around with
 that stuff anyways. You see FOSS guys tend to be engineers; they can put
 up with, and even like, industrial looking interfaces. But programmers
 also have a giant blind-spot when it comes to users. Most programmers
 view users as a lower species and assume that they will be delighted by
 whatever the programmer deigns to bequeath to them. But if you look at
 the successful people in the tech industry (*ahem* Steve Jobs) you'll
 find an attitude that is the exact opposite. Jobs was so focused on
 delivering what the user wanted that he would publicly berate any
 programmer who thought they knew better than the designer. While I don't
 necessarily agree with Jobs' management style, there is a reason why
 Apple is the second largest company in the world right now, and it has
 nothing to do with how good Apple's engineering is (which I hear is
 average at best). Despite programmers' best efforts, the world of
 technology is no longer controlled by programmers. For better or worse,
 users determine our course now. The open-source community would do well
 to embrace the user.

I have no trouble in letting a designer decide how GUI should look like, in fact, as you say it would be better if they did. But not everyone can have that luxury. Since I'm no designer you do the best I can.
 But without a first-class UI framework, that will never happen. In terms
 of capability and usability, both Apple and Microsoft have beat the best
 FOSS has to offer by a decade at least. I looked, searched, and scoured,
 but the fact of the matter is, even the usable FOSS UI offerings are
 pitiful when compared to the commercial offerings. The Horizon Project
 got it's start because there has been a trend in recent commercial UI
 offerings from towards increasing reliance on the operating system
 itself. Metro XAML just flat won't work on anything other than Windows
 8, Silverlight is a second class citizen at MS now, Cocoa only works on
 Mac etc. My goal with The Horizon Project is to create First-Class UI
 Framework for multiple platforms so that programmers don't have to
 rewrite the UI from scratch for each new platform they want to support
 and then open-source it so that the commercial OS vendors can't pervert
 it for their own purposes. I want to put [some of] the power back into
 the programmers hands.

 Apologies for the length, but this is a topic that is of some interest
 to me. :-)

I's a topic of interest to me as well. But I, on the other hand, prefer a native look and fell of applications. Instead of having yet another application with yet another GUI that doesn't work properly. If I see something that looks like a scroll bar I assume I can use it like a scroll bar. But that's not true for many applications, instead they implement the bare minimum for having the scroll bar "work", i.e. I may not be able to scroll using the wheel on the mouse. A great example of this are scroll bars in games. In my experience of non-native GUI's they always perform worse the native GUI's. Anyway, you can still use DWT to draw your own widgets. Even if you don't use any native widgets you still need to be able to draw something on a surface. DWT has a platform independent way of doing that. IBM Lotus Software and IBM Rational Software are built on SWT (which DWT is a port of) uses a lot a non-native widgets. There's an application called "Azureus" that had its own GUI that didn't look native at all, apparently it's called Vuze these days and it looks native.
 I don't understand this one. Should the compiler disable reflection as
 soon as it sees malloc/free? On what level should it be disabled? I
 mean, the runtime needs to be able to use these functions to implement
 the memory manager, i.e. the garbage collector and other things as well.

The idea is to automatically prevent reflection access to sections of the program where using direct memory manipulation could potentially result in security holes but provide a way out for the programmer if they really wanted it. C/C++ is famous for programmer bugs around memory handling. If the compiler automatically disabled reflection, by sticking noreflect in front of a function that used malloc or new, it could potentially prevent those types of memory manipulation flaws and help keep the Reflection attacks to a minimum. It was an idea that I was throwing out there. But I don't know enough about D yet to know if it's the right way to handle it. And I have to admit I am little confused though as I would hope that Reflection would be disabled on the GC, because I have never personally had a reason to reflect into the GC...

What happens when you use class A from class B and the compiler has added noreflect to class A? Will it add it to B as well? If not, how does the compiler know that? There has been a similar discussion about having the compiler insert "pure" automatic on functions. But what might happens is a function that is called by another function change it implementation making it no longer pure. Which means your function will no longer be pure and you have no idea about it. -- /Jacob Carlborg
Oct 28 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-28 21:36, Adam Wilson wrote:
 Ahh, thats the true power of WPF. Presentation and Implementation are
 cleanly separated. I recent re-skinned every scrollbar in one of our WPF
 apps by writing the 30-line style once, placing it in the global
 resource dictionary with implicit styling enabled, and bam, every
 scrollbar in the app looks different, but functions in the exact same
 way any windows user would expect. There are cases where you can write
 code that calls up into the presentation layer and thus introducing a
 dependency on the presentation layer, but this is actively discouraged,
 and when I found one recently in a 3rd party vendors control, it was
 removed by the next release.
 I can go on and on, but the point is, there are only two frameworks in
 existence that can do that are the WPF family and JavaFX. JavaFX is
 pathetic compared to WPF and is of course tied to an Oracle *shudder*
 controlled langauge. And a few months ago we had thousands of developers
 on the Silverlight forum looking for WPF family alternatives, all that
 they could find was JavaFX.

It's not the look of the, in this example, scroll bar, it's the behavior. It may look like a scroll bar but it may not behave in all the ways as a real, native, scroll bar would.
 Nobody, in fact WPF has three default looks out of the box, one for
 Windows 2000, another for WinXP, and another for Aero (Vista+). WPF
 defaults to the correct one for that system unless you override it. It's
 just that in my experience, the default look is a lowest common
 demoninator, and not a good one.

To me it sounds like WPF is the native GUI. Then why are you complaining about native GUI's?
 Neither am I really, I am a programmer by trade who cut his teeth back
 in the C++ days before C# came along and made me actually productive,
 but design isn't that hard, if one is willing to let go of their
 preferences. More to the point though, my company has no on staff
 designer, so it was either pay large sums of money to a consultant, or I
 as team lead could do it. I lost. To some degree it's a totally
 different way of thinking and I find that after a good design session (a
 week or two), it takes me a couple of days to realign my brain to
 left-brain-progammer-mode. But hey, they pay me, I jump to. *shrug*
 There is no mystical powers that are given to designers, anyone can make
 a good design.

I'm just saying that I'm not a good designer and if a designer is available I would prefer that they did the design.
 Actually we found that by re-skinning our app intelligently we were able
 to load a large list of complex data 4x faster than the default skin.
 The reason for this is that in WPF all drawing is done the same way, it
 doesn't matter if it's a system skin or not, it's just a skin, there are
 no special shortcuts for the system skin to take. At it's heart WPF is
 based on DirectX9. This means that screen complexity is the ONLY
 limiting factor.

If the changing the skin of an application can increase the data load by four times it sounds like something is seriously wrong. On Mac OS X a list only loads the data that is displayed so I don't think the amount of data matters. The same can be done using virtual tables in DWT.
 The other thing that WPF allows is data / presentation separation. I can
 pass a list box a chunk of data, and the data does not have to concern
 itself with how the presentation looks or vice versa. The listbox will
 display the data according to however the style is setup it. And THEN if
 I really want to I can change the presentation style on the fly based on
 user input. Can DWT do that in 2 lines of code? WPF Can. That's the
 power of reflection based data-binding. In fact I spend most of my time
 writing (much more compact) XAML code to do presentation layer work than
 I do write C#, to the point where the C# coding makes up less than half
 of the work done on a screen. It's usually things like load this item
 based on a selection changed event or some such that XAML has no way to
 do (and shouldn't, thats implementation).

No, probably not. It's not as easy to change the theme in SWT/DWT since it's supposed to use the native look and feel. It appears that you can at least set colors and similar properties using CSS: http://jaxenter.com/css-styling-in-eclipse-helios-32465.html
 Anyway, you can still use DWT to draw your own widgets. Even if you
 don't use any native widgets you still need to be able to draw
 something on a surface. DWT has a platform independent way of doing that.

Yea, but how much work is it? Can I create a completely new look for a control in 30 lines or less without modifying the implementation code at all to make the new look work? That's the power I'm after.

No you probably can't. I don't know what you want to achieve with your project but I got the impression that you were looking to build a new GUI library that behaves something like WPF. Regardless of how this GUI library works you need somehow to draw the widgets on the screen. I'm just saying that DWT has a platform independent way of doing this. On the other hand, if you're creating a completely new GUI library you probably would not want to depend on DWT. But instead creating bindings directly to the underlying drawing operations for Cairo, Quartz and DirectX or whatever Windows uses these days. -- /Jacob Carlborg
Oct 29 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-29 22:11, Adam Wilson wrote:
 WPF isn't the native GUI, it just comes with skins that emulate the look
 of the native GUI. The native GUI in Windows is called Win32 GDI
 (Graphices Device Interface). WPF is built on top of DirectX which is
 analagous to OpenGL and is primarily used for games. DirectX completely
 bypasses the GDI and communicates directly with the hardware. It's
 primary claim to fame is that it is fast, but it's not as compatible,
 whereas GDI is compatible with everything that implements VESA (which is
 everything since the early 90's), but it's slow.

We don't have to argue about this but it sounds like WPF is a new native GUI, especially if it bypasses GDI.
 Ok, so I apologize for generalizing there. It turns out that WPF is
 really bad at drawing round shapes, and the default skins (which the
 exception of the Windows 2000 skin) rely heavily on rounded corners. We
 replaced the default skin with one that used square corners and had
 fewer Measure and Arrange stages (where WPF figures out where to place
 the controls) which are equally expensive. WPF has virtual lists as
 well, and we do use it, which also helped contribute to our speed gains,
 but it is not enabled by default. My complaint is that the working with
 WPF requires a non-trivial amount of esoteric experience with it to make
 it work well.

With DWT you only pass an additional flag when creating the table, if I recall correctly.
 I am looking to create an open-source implementation of a GUI framework
 that is 'lookless', i.e. it's underlying implementation has no
 dependencies on how it looks and the look can be changed to however the
 designer wants it to look without having to worry about code. WPF and
 JavaFX are the only two frameworks that are examples of how this might
 work.

 I absolutely agree that relying on something else is probably not the
 best idea, because for better or worse you end up inheriting their
 design choices, which may or may not work for your project. Since I am
 windows guy, the reference implementation is going to be DirectX. First
 I just have to create Direct2D bindings for D...

What's wrong with using the lowest level of the drawing operations available on a given OS. On Mac OS X it would be Quartz which is has been hardware accelerated since a very long time. On Linux it would probably be Cairo and on Windows you could use the lowest level of WPF. Why reinvent the wheel again? If you want to roll your own drawing operations anyway why not have the reference implementation in OpenGL, it's available on basically all platforms including Windows. OpenGL bindings and other things like SDL and FreeType: http://dsource.org/projects/derelict DirectX bindings and bindings in general: http://dsource.org/projects/bindings This is not my strongest side, but wouldn't you need native windows to draw your widgets on?
 And another goal of the framework beyond platform independence is
 language independence. I have to admit that one of the big draws for me
 of D was the C++ ABI and COM compatibility of D while having a MUCH more
 modern capability set. However, it's still lacking in a few key areas
 for my project, and runtime reflection is one of them...

Just for the record, the interface to C++ is very limited. -- /Jacob Carlborg
Oct 30 2011
parent Mehrdad <wfunction hotmail.com> writes:
On 10/30/2011 4:03 AM, Jacob Carlborg wrote:
 On 2011-10-29 22:11, Adam Wilson wrote:
 WPF isn't the native GUI, it just comes with skins that emulate the 
 look of the native GUI. The native GUI in Windows is called Win32 GDI 
 (Graphices Device Interface). WPF is built on top of DirectX which is 
 analagous to OpenGL and is primarily used for games. DirectX 
 completely bypasses the GDI and communicates directly with the 
 hardware. It's primary claim to fame is that it is fast, but it's not 
 as compatible, whereas GDI is compatible with everything that 
 implements VESA (which is everything since the early 90's), but it's 
 slow.

We don't have to argue about this but it sounds like WPF is a new native GUI, especially if it bypasses GDI.

but it's regarding the _windowing_ framework itself (which is separate from, although related to, the graphics framework). _Native_ windows use the _system_ window classes (e.g. SysListView32) to display themselves to the user. WPF (which uses DirectX) and GTK (which uses GDI) and most frameworks, on the other hand, don't -- rather, they try to /look/ like native windows by creating custom windows and intercepting all of the drawing operations. And, of course, they fail. This is indeed hard to notice on XP, but it's blindingly obvious on Windows Vista and later, because, for example, the buttons no longer "feel" native, even though they might "look" native -- e.g. the way they fade or get depressed (the timeout, the fade colors, etc.), the way they get focused, etc. is completely out of sync with the rest of the system's looks. Since native controls use GDI internally, they obviously can't be drawn with DirectX. But the key point is that the inverse is not true -- if something uses GDI (e.g. GTK), that doesn't mean it's native. So no, WPF is not "native" in any way, and neither is GTK -- the only "native" frameworks are those that use the built-in windowing classes, which include MFC, ATL, SWT, and (I /think/) Qt. Most other frameworks don't do this.
Oct 30 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
Hear, hear!

On Tue, Oct 25, 2011 at 5:40 PM, Jonny Dee <jonnyd gmx.net> wrote:
 Hi,

 I would love to see runtime reflection be available in D. I am currently
reading Walter Bright's "The D
 Programming Language" and with every page I read I became more and more
fascinated and enthusiastic
 about D. But then I started to wonder why I haven't read about runtime
reflection yet, so I did some Internet
 research and was a bit disappointed this feature isn't available. D is really
cool, it's not only C done right,
 but also Java and C#. Actually, to me, it's one of the best programming
language available yet. It's really a
 pitty RTTI is not available. So I hope your approach will somehow influence
D's future :)

 Keep going,
 Jonny

Oct 25 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 03:23:40 -0400, Jacob Carlborg <doob me.com> wrote:
 On 2011-10-25 07:24, Robert Jacques wrote:

 On
 the other hand it feels like I'm stretching D's metaprogramming
 capabilities in my serialization library. That's just because D doesn't
 have proper reflection.

I've written three serialization libraries and two versions of a runtime reflection library; the first one was prior to the string bugs in CTFE being fixed and ran DMD out of memory. I found serialization to be simple and straight forward. And I found compiler bugs to be a greater issue with reflection than any limits in D's capabilities. Indeed, given I wrote the same library using two completely different mechanisms, gives me great hope for D. So, I don't agree with your assessment, but I do understand where you're coming from. A lot of D's metaprogramming capabilities are undocumented, and knowing the right tool for the job is essential to getting nice, clean code.

For example, getting the name of a instance variable: T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $]; I wouldn't consider that simple and straight forward. This works for both D1 and D2, don't know if D2 has a better way.

Not really, for the advanced user. I do find using "some string".length to be cleaner in these cases where you have to slice .stringof. i.e. static Value from(T)(T value) { //... static if(is(T == struct) || is(T == class)) { Value[string] result; foreach(i,v;value.tupleof) { result[value.tupleof[i].stringof["value.".length..$]] = from(v); } return Value( result ); } //.. } That said, I think anyone can read the above and understand what's happening. And even writing that code would be straightforward is .stringof and .tupleof were better documented. Also, the first serialization library I wrote used __traits instead. It was more awkward, mainly because __traits doesn't an 'allFields' member, just an 'allMembers'. If it did then the above becomes: foreach(name;__traits(allFields,value) { result[name] = from(__traits(getMember, value, name)); } which really is straightforward. Now I originally used 'allMembers' and then filtered out everything I didn't want, which was fairly straightforward but verbose.
 Your serialization library only works on public fields? Doesn't sound
 very useful.

In the interest of full disclosure, the library I wrote was quick and dirty and does not handle ploy-morphism, cycle detection, aliasing, etc (mainly because JSON doesn't); I used traits for my first one (so protection was observed) but .tupleof for my second and third and was lazy about it. It's one line of code to add protection to tuple of, but I've never gotten around to it. I figured my code would be replace by orange or std.serialization or something. That said, I have never serialized a private variable and have never wanted too. The serialized objects are, for the most part, part of the external API to my program. I have to do validation checks on the inputs and changing anything takes a week to get the other teams to propagate the changes on their side. There are also a couple of classes which I serialize in a binary format as a poor mans database, but again, there's no need for anything in it to be private. [snip]
 Actually, you're not supposed to be able to; this is bug 1983.
 http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
 https://github.com/D-Programming-Language/dmd/pull/71

I don't see what that has to do with private, it's about const, as far as I can see.

Both are protection attributes. The problem of taking the delegate of a mutable member on a const object, is the exact same problem as taking the delegate of a private member. In short, you can't 'bypass private using a delegate.' That you can today, is simple a bug in the compiler.
 I don't see how calling a private method through a
 delegate can be easily solved. I see three solutions:

 1. Make it a compile time error to create a delegate from a private
 method. This would not allow to call that method via a delegate in the
 same module, which is possible if you call the method directly.

 2. Implement some, most likely, very advanced data flow analysis in the
 compiler to detect when a delegate pointing to a private method escapes
 the module. Results in a compile time error. Will most likely not happen.

 3. Make it a runtime error when the delegate is actually called. This
 would probably need some pieces of 2 as well. I don't think the
 community will be happy about adding runtime checks like these.

 Oh, BTW the same needs to be done for function pointers as well since
 you can build a delegate out of a function pointer and an object.

Bypassing the protection type system and delegate escape are very different things. The escape of a delegate is fully controlled by the coder of the class/module; with bypass mechanisms, the library writer has no control.
 In Ruby an instance variable is always private. I don't think you can
 declare a public instance variable, one uses setters/getters for that.
 In D you can declare both private and public instance variables.

 In Ruby you can get/set a value of an instance variable using
 reflection, instance_variable_set/get.
 In D you can get/set a value of a private instance variable using
 compile time reflection, i.e. .tupleof.

Yes, but only for source code you control.

You don't need control of the source, you only need the static type. Of course an opaque struct won't work.

Sorry, that's what I meant by source you control. i.e. it is available to you to read in some form. Runtime reflection would allow you access to knowledge of opaque types.
 Actually, baring bugs in DMD, .tupleof is the only way to access private
 fields and there is _no way_ to access private members; And .tupleof
 only works if you have access to the source code. Furthermore, .tupleof,
 god-send that it is, isn't part of the D spec; unlike .stringof, it
 didn't make it into TDPL nor into the D documentation. D's official
 method of compile-time reflection is __traits. And __traits doesn't
 bypass protection attributes. So precedence would be against
 runtime-reflection having access to private members.

The compiler has .tupleof and it's mentioned in the spec at d-programming-language.org: http://d-programming-language.org/struct.html

I stand corrected. Sorry, I just looked at TDPL's index, the page on Properties and googled it. Now that I look for it, it's also on the Class page. Still, __traits is still the full fledged compile-time reflection feature and doesn't bypass protection attributes.
Oct 25 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:
 Hi,

 I would love to see runtime reflection be available in D. I am currently
reading Walter Bright's "The D
 Programming Language" and with every page I read I became more and more
fascinated and enthusiastic
 about D. But then I started to wonder why I haven't read about runtime
reflection yet, so I did some Internet
 research and was a bit disappointed this feature isn't available. D is really
cool, it's not only C done right,
 but also Java and C#. Actually, to me, it's one of the best programming
language available yet. It's really a
 pitty RTTI is not available. So I hope your approach will somehow influence
D's future :)

 Keep going,
 Jonny

Hi Jonny, Well, some RTTI is available. How much RTTI is needed, really depends on what you're trying to do. Therefore whether D has or hasn't got RTTI depends heavily on what any particular person means by RTTI. Do you have some use cases in mind? Maybe prototype objects, duck-typing/casting or serialization? Or perhaps something you wrote? Sorry for all the questions, but I feel as if we don't have a resident reflection expert in the dialog, or even the opinions of basic/experienced users. We all know that 'RTTI' is good, but we're a little vague on the how and why. (Well, beyond the use cases above)
Oct 25 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 07:30:04 -0400, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:
 You can always write  noreflect: at the start of your module and it
 will work as expected. what's the problem with that?
 Also, compiler would be able to automatically add that attribute to
 any symbol, which is private and does not escape the module.
 Again, not including reflection is purely _optimization_ issue and
 should not be interfered with it's functionality.
 Reflection is something you rarely have in mind when designing, for
 example, classes.
 Requiring explicit request for reflection defeats the purpose.

 You could just as well say "writing public on every member is a
 clutter". It is, but you don't have to do it to achieve the same
 effect.

You can always write reflect: at the start of your module and will work as expected. what's the problem with that? Rhetoric aside, Timon wasn't referring to code clutter on behalf of the programmer. He was referring to code bloat in the compiled program. The two are very different things. Also, just because reflection is something you rarely have in mind when designing, doesn't mean it shouldn't be part of the design API. Reflection is somewhat equivalent to the extern protection modifiers and so should warren careful consideration. Furthermore, bugs with reflection, say due to a change in the API, don't break during compilation; they break at runtime. And practically, there's a decent change your customer, not your test suite, is going to find it.
Oct 25 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
As i said before, enabling reflection manually is an act of enabling a
certain range of stuff you can do with the code. Forgetting to enable
reflection can lead to serious problems, when the reflection is
required.
disabling reflection manually is an act of optimization purely.
Forgetting to disable the reflection can only lead to unnecessary data
being stored.

The second scenario is much less error-prone.
yes, it _may_ lead to code bloat if you forget to disable it where it
is _absolutely positively unneeded_, but forgetting to enable it will
just make your code useless in many situations.

On Tue, Oct 25, 2011 at 7:20 PM, Robert Jacques <sandford jhu.edu> wrote:
 On Tue, 25 Oct 2011 07:30:04 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:
 You can always write  noreflect: at the start of your module and it
 will work as expected. what's the problem with that?
 Also, compiler would be able to automatically add that attribute to
 any symbol, which is private and does not escape the module.
 Again, not including reflection is purely _optimization_ issue and
 should not be interfered with it's functionality.
 Reflection is something you rarely have in mind when designing, for
 example, classes.
 Requiring explicit request for reflection defeats the purpose.

 You could just as well say "writing public on every member is a
 clutter". It is, but you don't have to do it to achieve the same
 effect.

You can always write reflect: at the start of your module and will work as expected. what's the problem with that? Rhetoric aside, Timon wasn't referring to code clutter on behalf of the programmer. He was referring to code bloat in the compiled program. The two are very different things. Also, just because reflection is something you rarely have in mind when designing, doesn't mean it shouldn't be part of the design API. Reflection is somewhat equivalent to the extern protection modifiers and so should warren careful consideration. Furthermore, bugs with reflection, say due to a change in the API, don't break during compilation; they break at runtime. And practically, there's a decent change your customer, not your test suite, is going to find it.

Oct 25 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
Requiring explicit request for reflection contradicts D's "safety
first, performance second" policy.
lack of reflection is a safety issue too, because you won't be able to
do some checks.

On Tue, Oct 25, 2011 at 7:31 PM, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:
 As i said before, enabling reflection manually is an act of enabling a
 certain range of stuff you can do with the code. Forgetting to enable
 reflection can lead to serious problems, when the reflection is
 required.
 disabling reflection manually is an act of optimization purely.
 Forgetting to disable the reflection can only lead to unnecessary data
 being stored.

 The second scenario is much less error-prone.
 yes, it _may_ lead to code bloat if you forget to disable it where it
 is _absolutely positively unneeded_, but forgetting to enable it will
 just make your code useless in many situations.

 On Tue, Oct 25, 2011 at 7:20 PM, Robert Jacques <sandford jhu.edu> wrote:
 On Tue, 25 Oct 2011 07:30:04 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:
 You can always write  noreflect: at the start of your module and it
 will work as expected. what's the problem with that?
 Also, compiler would be able to automatically add that attribute to
 any symbol, which is private and does not escape the module.
 Again, not including reflection is purely _optimization_ issue and
 should not be interfered with it's functionality.
 Reflection is something you rarely have in mind when designing, for
 example, classes.
 Requiring explicit request for reflection defeats the purpose.

 You could just as well say "writing public on every member is a
 clutter". It is, but you don't have to do it to achieve the same
 effect.

You can always write reflect: at the start of your module and will work as expected. what's the problem with that? Rhetoric aside, Timon wasn't referring to code clutter on behalf of the programmer. He was referring to code bloat in the compiled program. The two are very different things. Also, just because reflection is something you rarely have in mind when designing, doesn't mean it shouldn't be part of the design API. Reflection is somewhat equivalent to the extern protection modifiers and so should warren careful consideration. Furthermore, bugs with reflection, say due to a change in the API, don't break during compilation; they break at runtime. And practically, there's a decent change your customer, not your test suite, is going to find it.


Oct 25 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 11:31:18 -0400, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:
 As i said before, enabling reflection manually is an act of enabling a
 certain range of stuff you can do with the code. Forgetting to enable
 reflection can lead to serious problems, when the reflection is
 required.

And when, precisely, is reflection _required_? Granted, in other languages it is the only tool for the job. But we have a very big toolkit in D, and so far, I have yet to see any example (including Rails) which _requires_ the holistic runtime reflection proposed in these threads.
 disabling reflection manually is an act of optimization purely.

No, it isn't. Go read the rest of these threads for counter-arguments. Or the Java Best Practices guide. Or search the Open Web Application Security Project for reflection vulnerabilities.
 Forgetting to disable the reflection can only lead to unnecessary data
 being stored.

Alternatively, sensitive information being exposed.
 The second scenario is much less error-prone.

Incorrect, the second scenario is a security hole waiting to be exploited.
 yes, it _may_ lead to code bloat if you forget to disable it where it
 is _absolutely positively unneeded_,

Having written a (limited) runtime reflection library for D, yes it causes bloat even if you only enable it where _absolutely positively needed_.
 but forgetting to enable it will
 just make your code useless in many situations.

Correction: it will make code designed to _require_ runtime-reflection incompatible with code not designed to support reflection. Given the percentage of code that requires reflection (very small) and the percentage of code which wouldn't be compatible or trivially made to be compatible, (very small) I'm fine with that.
Oct 25 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 11:35:42 -0400, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:
 Requiring explicit request for reflection contradicts D's "safety
 first, performance second" policy.

Umm, it's "safe defaults, performance possible".
 lack of reflection is a safety issue too, because you won't be able to
 do some checks.

What would you use reflection to check?
Oct 25 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
I didn't mean reflection for violating access specifications. In that
case, of course, it's error-prone.
What i meant was a set of dynamic counterparts of static features of D:
* Dynamic interfaces
* Dynamic overloading
* Dynamic modules
* ...
That's what i had in mind.

On Tue, Oct 25, 2011 at 8:08 PM, Robert Jacques <sandford jhu.edu> wrote:
 On Tue, 25 Oct 2011 11:35:42 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:
 Requiring explicit request for reflection contradicts D's "safety
 first, performance second" policy.

Umm, it's "safe defaults, performance possible".
 lack of reflection is a safety issue too, because you won't be able to
 do some checks.

What would you use reflection to check?

Oct 25 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 12:14:58 -0400, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:
 I didn't mean reflection for violating access specifications. In that
 case, of course, it's error-prone.

Thank you, I'm glad you agree.
 What i meant was a set of dynamic counterparts of static features of D:
 * Dynamic interfaces
 * Dynamic overloading
 * Dynamic modules
 * ...
 That's what i had in mind.

Care to explain what would each of those should look like? Would dynamic interfaces be similar to isRange!T? i.e. a function which tests a dynamic-object for certain fields/functions. Would dynamic overload be a subset of prototype-style objects? Would dynamic modules simply container for dynamic functions/objects? Something more?
Oct 25 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
 Thank you, I'm glad you agree.

Of course. Reflection should _never_ expose private stuff.
 Would dynamic interfaces be similar to isRange!T? i.e. a function which tests
a dynamic-object for certain fields/functions.

Hmmm..... Yeap. Exactly. Anything else i would say would be a derivative of it. I can't do it currently. Also, as discussed before, i should never see private stuff this way (any any other way).
 Would dynamic overload be a subset of prototype-style objects?

Huh? Dynamic overloading is basically multi-methods. It chooses the implementation based on run-time type of the parameters. This could be further improved by allowing overloading, based on arbitrary predicates (much like template constraints).
 Would dynamic modules simply container for dynamic functions/objects?

Basically, yes. Dynamic module would contain typemically typed variables, functions, and types.
  Something more?

Cant thing of anything else ATM. Will tell you if i do. On Tue, Oct 25, 2011 at 8:29 PM, Robert Jacques <sandford jhu.edu> wrote:
 On Tue, 25 Oct 2011 12:14:58 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:
 I didn't mean reflection for violating access specifications. In that
 case, of course, it's error-prone.

Thank you, I'm glad you agree.
 What i meant was a set of dynamic counterparts of static features of D:
 * Dynamic interfaces
 * Dynamic overloading
 * Dynamic modules
 * ...
 That's what i had in mind.

Care to explain what would each of those should look like? Would dynamic interfaces be similar to isRange!T? i.e. a function which tests a dynamic-object for certain fields/functions. Would dynamic overload be a subset of prototype-style objects? Would dynamic modules simply container for dynamic functions/objects? Something more?

Oct 25 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd gmx.net> wrote:
 Am 25.10.11 16:41, schrieb Robert Jacques:
 On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:
 Hi,

 I would love to see runtime reflection be available in D. I am
 currently reading Walter Bright's "The D
 Programming Language" and with every page I read I became more and
 more fascinated and enthusiastic
 about D. But then I started to wonder why I haven't read about runtime
 reflection yet, so I did some Internet
 research and was a bit disappointed this feature isn't available. D is
 really cool, it's not only C done right,
 but also Java and C#. Actually, to me, it's one of the best
 programming language available yet. It's really a
 pitty RTTI is not available. So I hope your approach will somehow
 influence D's future :)

 Keep going,
 Jonny

Hi Jonny, Well, some RTTI is available. How much RTTI is needed, really depends on what you're trying to do. Therefore whether D has or hasn't got RTTI depends heavily on what any particular person means by RTTI. Do you have some use cases in mind? Maybe prototype objects, duck-typing/casting or serialization? Or perhaps something you wrote? Sorry for all the questions, but I feel as if we don't have a resident reflection expert in the dialog, or even the opinions of basic/experienced users. We all know that 'RTTI' is good, but we're a little vague on the how and why. (Well, beyond the use cases above)

Hi Robert, Well, before I tell you what I would like to see I'll cite Wikipedia [1]: " [...] - Discover and modify source code constructions (such as code blocks, classes, methods, protocols, etc.) as a first-class object at runtime. - Convert a string matching the symbolic name of a class or function into a reference to or invocation of that class or function. [...] " Here is what I would dream of for arbitrary objects/classes (not necessarily known at compile-time): - Query an object for its list of methods together with their signatures. Select a method, bind some values to its arguments, call it, and retrieve the return type (if any). - Query an object for its public fields (at least), and provide a way to get/set their values. - Query an object's class for all implemented interfaces and its base class. - Query a module for all type definitions and provide a way to introspect these types in more detail. For instance, it would be really cool if I could find a class with name "Car" in module "cars", get a list of all defined constructors, select one, bind values to the constructor's parameters, and create a corresponding object. Now you might ask where is the use case. I think, a very important one, besides creating GUI designers, is to be able to implement a Dependency Injection container [2,3,4] like, for example, the one provided by the Spring Framework for Java or .NET. In short, they allow you define how object trees should be build up solely by specifying this tree in an XML file. Within the XML file you can create new instances of classes of arbitrary types. You can specify which constructors to use for instantiation, and what values the constructor's arguments (if any) should have. You can even bind such an argument to an object previously created by the XML definition. You can call methods on existing objects, e.g. in order to call setter to further initialize an object with values you also define in that XML file. That XML file can be loaded when your program starts an make the DI container component instantiate all needed objects for you. There is no need for recompilation if the XML file is changed. Just restart your application. Implementing such a DI container heavily depends on reflection, because the DI container component doesn't know anything about the objects to be created during runtime. Qt also extends C++ with a reflection mechanism through the help of its meta object compiler (moc). It analyses the C++ source code, generates meta class definitions [6,7] and weaves them into your Qt class. Hence, in Qt, you can query an object for fields, methods, interfaces, etc. and you can call methods with arbitrary parameters, or you can instantiate a class using an arbitrary constructor. Consequently, somone implemented a DI container for C++ which is based on Qt and works more or less the same way the Spring DI container does. You can build up object trees simply by specifying such trees in an XML file. I don't go into why dependency injection is a very powerful feature. This is Martin Fowler's [3] job ;) But when I program with C++ I miss such a flexible dependency injection mechanism a lot. And I hope this will eventually be available for D. Cheers, Jonny [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29 [2] http://en.wikipedia.org/wiki/Dependency_injection [3] http://martinfowler.com/articles/injection.html [4] http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29 [5] http://qtioccontainer.sourceforge.net/ [6] http://doc.qt.nokia.com/stable/qmetaobject.html [7] http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx

Hi Jonny, Thank you for your informative (and well cited) post. It has provided me with a new take on an old design pattern and some enjoyable reading. In return, let me outline my opinion of reflection in D today, and tomorrow, as it pertains to your wish list. Reflection in D today is very different from the host of VM languages that have popularized the concept. Being a compiled systems language, actual runtime self-modification is too virus like to become at a language level feature. However, given the compilation speed of D, people have made proof of concept libraries that essentially wrapped the compiler and dynamically loaded the result. As LDC uses LLVM, which has a jit backend, I'd expect to see something get into and D 'eval' library into etc eventually. (phobos uses the BOOST license, which isn't compatible with LLVM). Compile-time reflection and generation of code, on the other hand, is something D does in spades. It fulfills your dream list, although I think module level reflection might only be available in the github version. The API design is still in flux and we are actively iterating / improving it as find new uses cases and bugs. The current plan is to migrate all the traits functions over to a special 'meta' namespace (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid libraries for each of the concepts I listed, (prototype objects, duck-typing/casting or serialization), have been written using the compile-time meta-programming features in D. So that's the good. On the other hand, D's runtime capabilities are limited to object.factory, the under implemented RTTI and library solutions which manually expose information gathered by D's compile-time mechanisms. And so far, these tools have been more than enough, from a functionality point of view. Most of our desire for better runtime reflection stems from a desire for efficiency, composition, cleanliness of user syntax and simplification of library code. These are all important issues for the widespread use of reflection based libraries, but they're not 'I can't implement X' issues. As for the future, I believe that the division in D between compile-time and run-time reflection warrants a serious look at the design of the run-time half of the system. To that end, I believe that implementing reflection in a library would be the best way to experiment and iterate an API. To that end, I have a proposal in the review queue to improve std.variant which contains dynamic dispatch (i.e. the ability to get/set public fields and call methods), duck-typeing/casting and prototype-style objects. Notably, it's missing introspection capabilities as thats what I'm most unsure about API wise, and simplest to add. Designing reflection inside a library keeps Walter & Co's bandwidth free for other bugs/features and provides a very good stress test of D's meta-programming capabilities. (Both of which I think are good things) (I am soliciting feedback, if you care to take a look: https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht) And there are many reasons we might want to experiment with D's runtime reflection API instead of just copying someone. For example, take Dependency Injection. Using Java-style reflection, DI suffers from the Reflection injection and Unsafe Reflection security vulnerabilities. To do DI safely, you have to validate your inputs and D's compile-time reflection provides a perfect way to implement validated DI. Every time I hear about some form of injection, be it SQL or JSON or Reflection, hit the news, makes me think that 5-10 years from now well look back on the use of unvalidated dynamic code constructs the same way we do about null terminated arrays today.
Oct 25 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 12:38:18 -0400, Jacob Carlborg <doob me.com> wrote:
 On 2011-10-25 16:21, Robert Jacques wrote:

 Both are protection attributes. The problem of taking the delegate of a
 mutable member on a const object, is the exact same problem as taking
 the delegate of a private member.

I don't agree.

[snip]
 Bypassing the protection type system and delegate escape are very
 different things. The escape of a delegate is fully controlled by the
 coder of the class/module; with bypass mechanisms, the library writer
 has no control.

So how should this be fixed? Make it a compile error to create a delegate from a private method?

I think we are taking about different things. Today in D, due to a bug, for example foo.bar() won't compile because it's private/not-const/etc. But (&foo.bar)() will compile. That 'feature' is what I thought you were referring to. If the writer of a class wants to let a delegate of a private member function escape, I don't have a fundamental problem with it. But I do have a problem with someone who isn't the writer of the class doing so. [snip]
 Still, __traits is still the full fledged compile-time reflection
 feature and doesn't bypass protection attributes.

Yeah, but for a serialization library I want to be able to serialize private fields. This is known issue that this breaks encapsulation.

I understand that at times when serialization has to break encapsulation. My opinion though, is that classes should have to opt in, before the serializer goes ahead and breaks their carefully designed encapsulation. i.e. someone should have to code review the class to make sure everything is kosher. For code you are developing, this is as simple and straightforward as adding a single mixin statement to the class.
Oct 25 2011
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 26 Oct 2011 11:20:47 -0700, Jonny Dee <jonnyd gmx.net> wrote:

 Hello Robert,

 Am 26.10.11 07:16, schrieb Robert Jacques:
 On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd gmx.net> wrote:
 Am 25.10.11 16:41, schrieb Robert Jacques:
 On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:
 [...]


Well, before I tell you what I would like to see I'll cite Wikipedia [1]: " [...] - Discover and modify source code constructions (such as code blocks, classes, methods, protocols, etc.) as a first-class object at runtime. - Convert a string matching the symbolic name of a class or function into a reference to or invocation of that class or function. [...] " Here is what I would dream of for arbitrary objects/classes (not necessarily known at compile-time): - Query an object for its list of methods together with their signatures. Select a method, bind some values to its arguments, call it, and retrieve the return type (if any). - Query an object for its public fields (at least), and provide a way to get/set their values. - Query an object's class for all implemented interfaces and its base class. - Query a module for all type definitions and provide a way to introspect these types in more detail. For instance, it would be really cool if I could find a class with name "Car" in module "cars", get a list of all defined constructors, select one, bind values to the constructor's parameters, and create a corresponding object. [...] Implementing such a DI container heavily depends on reflection, because the DI container component doesn't know anything about the objects to be created during runtime. Qt also extends C++ with a reflection mechanism through the help of its meta object compiler (moc). It analyses the C++ source code, generates meta class definitions [6,7] and weaves them into your Qt class. Hence, in Qt, you can query an object for fields, methods, interfaces, etc. and you can call methods with arbitrary parameters, or you can instantiate a class using an arbitrary constructor. Consequently, somone implemented a DI container for C++ which is based on Qt and works more or less the same way the Spring DI container does. You can build up object trees simply by specifying such trees in an XML file. I don't go into why dependency injection is a very powerful feature. This is Martin Fowler's [3] job ;) But when I program with C++ I miss such a flexible dependency injection mechanism a lot. And I hope this will eventually be available for D. Cheers, Jonny [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29 [2] http://en.wikipedia.org/wiki/Dependency_injection [3] http://martinfowler.com/articles/injection.html [4] http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29 [5] http://qtioccontainer.sourceforge.net/ [6] http://doc.qt.nokia.com/stable/qmetaobject.html [7] http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx

Hi Jonny, Thank you for your informative (and well cited) post. It has provided me with a new take on an old design pattern and some enjoyable reading. In return, let me outline my opinion of reflection in D today, and tomorrow, as it pertains to your wish list.

Many thanks to you, too, for your very elaborate answer :)
 Reflection in D today is very different from the host of VM languages
 that have popularized the concept. Being a compiled systems language,
 actual runtime self-modification is too virus like to become at a
 language level feature. However, given the compilation speed of D,
 people have made proof of concept libraries that essentially wrapped the
 compiler and dynamically loaded the result. As LDC uses LLVM, which has
 a jit backend, I'd expect to see something get into and D 'eval' library
 into etc eventually. (phobos uses the BOOST license, which isn't
 compatible with LLVM).

I know, that "runtime self-modification" and runtime code generation is a "dangerous" feature. And there really are rare cases where using such an approach might justify the risc in using it. Although this feature is not on my wish list, it might be good for generating dynamic proxies to arbitrary object instances like they are used by some ORMs. See Hibernate/NHibernate, for example [1,2]. Another example is aspect-oriented programming. But while I can't see the exacty reason for it, such a feature might indeed be a feature which is more appropriate for VM languages.
 Compile-time reflection and generation of code, on the other hand, is
 something D does in spades. It fulfills your dream list, although I
 think module level reflection might only be available in the github
 version. The API design is still in flux and we are actively iterating /
 improving it as find new uses cases and bugs. The current plan is to
 migrate all the traits functions over to a special 'meta' namespace
 (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid
 libraries for each of the concepts I listed, (prototype objects,
 duck-typing/casting or serialization), have been written using the
 compile-time meta-programming features in D. So that's the good.

A absolutely agree! D's compile-time reflection is very good. And it's hard to top this.
 On the other hand, D's runtime capabilities are limited to
 object.factory, the under implemented RTTI and library solutions which
 manually expose information gathered by D's compile-time mechanisms. And
 so far, these tools have been more than enough, from a functionality
 point of view. Most of our desire for better runtime reflection stems
 from a desire for efficiency, composition, cleanliness of user syntax
 and simplification of library code. These are all important issues for
 the widespread use of reflection based libraries, but they're not 'I
 can't implement X' issues.

You are right, there IS certainly always some way to 'implement X'. But, as you know, there is always a consideration of the effort you need to implement X.
 As for the future, I believe that the division in D between compile-time
 and run-time reflection warrants a serious look at the design of the
 run-time half of the system. To that end, I believe that implementing
 reflection in a library would be the best way to experiment and iterate
 an API. To that end, I have a proposal in the review queue to improve
 std.variant which contains dynamic dispatch (i.e. the ability to get/set
 public fields and call methods), duck-typeing/casting and
 prototype-style objects. Notably, it's missing introspection
 capabilities as thats what I'm most unsure about API wise, and simplest
 to add. Designing reflection inside a library keeps Walter & Co's
 bandwidth free for other bugs/features and provides a very good stress
 test of D's meta-programming capabilities. (Both of which I think are
 good things)

I've got no problem with an approach which puts runtime reflection capabiities into a separate library. No matter were you look at, Java, C#, or Qt, all have a library for reflection purpose. I do not see, however, how this might be done without compiler support. As already mentioned, Qt has its own moc compiler, which parses your C++ source code and generates the necessary infra structure. I'm still a beginner with respect to D, so I don't know what is really already possible, and what not. But my current feeling is that a similar approach would also be needed for D. If the D compiler itself, or another post-compiler component should generate the meta information for runtime reflection is another question. I could live with both, although I'd prefer the former built-in one.
 (I am soliciting feedback, if you care to take a look:
 https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)

Thanks for this link, I'll certainly have a look at it.
 And there are many reasons we might want to experiment with D's runtime
 reflection API instead of just copying someone. For example, take
 Dependency Injection. Using Java-style reflection, DI suffers from the
 Reflection injection and Unsafe Reflection security vulnerabilities. To
 do DI safely, you have to validate your inputs and D's compile-time
 reflection provides a perfect way to implement validated DI. Every time
 I hear about some form of injection, be it SQL or JSON or Reflection,
 hit the news, makes me think that 5-10 years from now well look back on
 the use of unvalidated dynamic code constructs the same way we do about
 null terminated arrays today.

Considering security vulnerabilities is of course a very important matter. However, I think security must be assured a by software's design. Enforcing it solely by programming language constructs will not work for all cases. Particularly not, if this programming language allows direct access to a computer's memory, like C, C++, and D does. There is no sandbox out-of-the-box, where the compiled program runs in. So if you have a private field in a class that carries a password, one has to make sure it's only in memory as long as it is required. And if it is not required anymore one should clear it out with zeros, for example. So making a field private is by no means a secure solution. A hacker will not give up just because the official API declares a field to be private. I consider 'unsafe reflection', as you call it, as a tool. Let's compare it to a knife, for instance. It is really a useful tool, I think nobody doubts, but at the same time it can be very dangerous. But inspite of this fact, everybody has not only one at home, I guess. Pointers are a useful tool, too. But they are also dangerous and still available in D, which is a good thing, because you can use this tool where needed. And actually, I don't think runtime reflection must be unsafe. The reflection mechanism provides type information for an object's properties, functions, arguments, etc. So validated DI is even possible with runtime reflection. Let's consider the the XML configuration of object trees once again. If you want to store a string to an int-field, for instance, then the DI container can refuse to do this, because it has access to all information required to enforce correct value types. One more use case for reflection is data binding with GUI components. This approach is heavily used in Windows Presentation Foundation library [3,4]. GUI components can update an object's properties by using reflection. You don't need to register listeners for this purpose anymore. That said, I don't think dynamic code constructs will be old-fashioned in 5-10 years, because you don't have to go an "unvalidated dynamic" way with runtime reflection. BTW, as this thread also discusses an opt-in or opt-out implementation for runtime reflection. I'd prefer an opt-out way, too. Code bloat (I guess the binaries are meant) is not as bad as it might sound with today's memory sizes. And if one wants to avoid it for optimization purpose, one can do it. The use of D as a systems programming language for embedded systems is, as I've read somewhere, not a first citizen anyway, because you'll get a lot of code for the garbage collection mechanism. Cheers, Jonny [1] http://en.wikipedia.org/wiki/Hibernate_%28Java%29 [2] http://en.wikipedia.org/wiki/Nhibernate [3] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation [4] http://blogs.msdn.com/b/wpfsdk/archive/2006/10/19/wpf-basic-data-binding-faq.aspx

I am a first time poster but I've been lurking around here for a while now and this topic is something that is extremely near and dear to my heart. I have been working with .NET since the C# 1.0 Beta 1 days and cut my teeth on WPF while it was still called Avalon. I know you mention that WPF uses Reflection, and on that statement I can only say that you are correct, for better or worse, usually worse. However, thats not the complete story. WPF uses something called a DependencyObject to implement objects with data-binding. And this is important as literally EVERY WPF object inherits this class, you can't build a WPF object without it. And every property you set on a WPF object is really two properties, one of which is a "public static DependencyProperty PropertyName", a DependencyProperty is also being a class that requires a DependencyObject to initialize it. In WPF you absolutely MUST declare this object as public. And while not strictly required Microsoft Best Practices dictate that the actual property is public as well. My point is that even in WPF you don't actually NEED reflection of private members. The reason for this is that back during the design of Silverlight Microsoft knew that SL's reflection would NOT be able to reflect to private members and they needed to make sure that SL XAML would still work with WPF XAML. Remember, SL was originally called WPF/Everywhere. WPF has taken a lot of flack for the amount of reflection that it does precisely because it slows it down, but I think that it has other bigger problems. Personally, I've found the Measure and Arrange calls to be even significantly more expensive, but those can be mitigated with experience, the reflection can't, so it gets blamed. WPF has many other perf issues too, how they draw rounded corners makes grown men cry and any game programmer who wrote code that bad would be fired on the spot, a one thousand polygon circle anyone? It's also interesting that you bring up WPF because the project I lead, The Horizon Project, is an attempt to create a open-source multi-platform framework that is as easy to use as the .NET Framework and one of it's core pillars is a declarative XML based UI design language. I am evaluating D and so far it is blowing C++ out of the water, (we'll need to have long conversations about strings though, I think .NET has the right answer here) but I have to admit that I am seriously missing Reflection. The problem is that there is currently no way to do data-binding without member-level reflection because it is a purely run-time concept. That said, I am researching ways around this, and I do have ideas ... but that's not for this forum. For D, I have two notes. First, data-binding is the way of the future, you can't ignore it, or say that because it doesn't fit with philosophy 'X' that it shouldn't be done. The biggest reason that open-source projects die is that the industry moves in a way that conflicts with the projects philosophy and leaves the project behind. And this is a good thing, we are collective discovering better ways of doing things and part of that process is rejecting old philosophies. Adapt or die. The second is that you can't protect the programmer from all possible mistakes, and in terms of security their are many worse potential exploits that are much easier to reach for than Reflection. And to be honest, I've never actually heard of a successful Reflection exploit via the .NET Framework, although I am sure someone here knows of one... However, I see no particular reason to reflect on private members by default, if you need to get data out, you should be using a public property or method to filter the data as needed, that's kind of the point of scoping. Private member reflection is NOT a requirement for UX, WPF being the case in point. I personally like the solution that Reflection in D is opt-out with both command-line options and noreflect. I would add that a deepreflect option be made available for those incorrigible souls who absolutely must reflect private members. I would also add that reflection should be default disabled by the compiler for code the works directly with pointers and direct memory access (malloc/free, new/delete, etc.) and the developer can only enable it using a compiler option. For most of us, code-bloat is not an issue even in the slightest, todays computers can open multi-megabyte executables running JIT's (think .NET) in less than half a second (we have a working example where I work). In this case, a little extra bloat to make my life significantly easier is a non-choice, I'll take the bloat every time. However, I completely agree that there are cases where bloat is bad, anything system level (kernels, drivers, etc.), mobile or power-limited devices (although more of a toss-up), and others like games. But in the end, I have products to deliver, I need to best tools for the job, if that means using reflection, so be it, ideology should never enter the equation. In the case of reflection, it is my opinion that the compiler should provide a full range of tools and let the developers decide how much they want to use; this would make D superior to even .NET, as .NET is extremely limited in it's reflection control. -- Adam Wilson Project Coordinator The Horizon Project http://www.thehorizonproject.com/
Oct 26 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Wed, 26 Oct 2011 14:20:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:
[snip]
 Am 26.10.11 07:16, schrieb Robert Jacques:

 I know, that "runtime self-modification" and runtime code generation is
 a "dangerous" feature.

As a point of semantics, actual "runtime self-modification" is no longer available outside a VM; the practice is heavily used in viruses, worms, etc and practically nowhere else, so both OS, hardware and anti-virus vendors have ganged up against this practice. :)
Oct 27 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Wed, 26 Oct 2011 14:20:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:
[snip]
 Am 26.10.11 07:16, schrieb Robert Jacques:

 I've got no problem with an approach which puts runtime reflection
 capabiities into a separate library. No matter were you look at, Java,
 C#, or Qt, all have a library for reflection purpose. I do not see,
 however, how this might be done without compiler support.

Long ago, when reflection in was being discussed one of the big reasons we turned to compile-time reflection first, besides its synergy with other meta-programming features, is because compile-time reflection enables runtime-reflection. A decent portion of the community has always assumed that D's runtime reflection support would come from a library exploiting D's compile-time reflection. All it would take is someone willing to put in the effort and time to do so. So far, that someone has been me. :) My new Variant (https://jshare.johnshopkins.edu/rjacque2/publ c_html/variant.mht) makes heavy use of compile-time reflection and provides runtime-reflection capabilities. Currently, I have focused on dispatch, i.e. the calling of member functions, the calling of static members and the setting of fields, as I don't know what the introspection API should look like. I have found that while __traits does provide all the information I need, it's often not in the format I wish to consume it in. So I have been holding back on implementation in order to solicit the feedback of the community. As for ease of use, Variant generates reflection information for every type it is exposed to. That includes all return types, fields and member arguments types for every type assigned to or retrieved from Variant. Most of the time, exposure happens naturally and everything 'Just Works' and no effort is needed. For example: Variant var = student; assert(var.grade.mark == student.grade.mark) Or Variant var = Object.factory("example.Student"); assert(var.grade == (var.get!(Student)).grade); Now it is possible for types, particularly sub-classes, to never be retrieved from or assigned to, in which case only they can only use their interfaces/base-classes type information, without some form of registration. For example, Variant.__register!Student; And with the new module reflection capabilities, the hassle / annoyance of registration can be reduced to that of an import statement. Or even a single statement that would reflect all the imports in a module. (Registration works via static ctors, so their's no worry about using reflection before registration, or duplicate registrations, etc) Now, this probably gets to the heart of why I don't like the concept of noreflect, reflect or any other fixed reflection option: there is no right answer to whether or not a class should support reflection. Trying to find a single choice (or all/none) that fits the embedded, iOS/Driod/Windows 8, Desktop, Server, etc developer is fool hardy. Letting each developer choose how much or how little they need feels right to me. And in D we have the tools to provide the granularity of control to make this declaration of reflection intent as succinct as possible.
Oct 27 2011
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Thu, 27 Oct 2011 00:14:35 -0700, Jacob Carlborg <doob me.com> wrote:

 Are you saying that you consider using D for this Horizon project? I can  
 recommend you take a look at DWT: www.dsource.org/projects/dwt

 Somewhere down the road I've planed to create an interface/window  
 builder for DWT using XML or something similar. I'm thinking something  
 like how it works on Mac OS X using Interface Builder.

I am, and I have looked at DWT. My problem with it is a one that is endemic to open-source UI framework. Microsoft recognized a decade ago that UI widgets whose look-and-feel is defined and controlled by the Operating System are going the way of the dodo. Out of that realization WPF was born. UI designers today want the ability to control every pixel of the UI's presentation. And the reasons for this are two-fold. The first is that it turns out that most OS designers are fantastically bad at UI and design in general. It takes epic piles of cash to pull of a decent one and even then there is a still a "programmers were here" look to it (with the notable exception of iOS/OSX where designers rule the roost). The second is product differentiation. Nobody wants an app that looks like every other app because it actually becomes impossible for the user to distinguish which app works best for them. Users ONLY look at the UI, and if the app doesn't look good, they wont "buy" it, even if it's free. This is non-negotiable. Users, when given two apps that do the same thing, even for different prices, will pick the prettier one every time, because the prettier one is perceived as being "better". It's called the Attractiveness Bias and it is a well-known principle in the design world. Who would you rather look at all day, Alessandra Ambrosio or Rosie O'Donnell? I rest my case. I maintain that this is prime reason that Linux on the desktop has failed miserably, and I think Android proves my point. Android's key win is that it put a usable UI on top of Linux. People never had a problem with the price of Linux, they just couldn't stand to look at it. The Linux Desktop LOOKS industrial and it's apps for the most part look the same (I know of a few outliers that did a good job, but it isn't the norm). My point is that the day of cookie cutter apps is over. Anyone designing for that paradigm is history. Microsoft's latest UI paradigm, "Metro", is just a different cut-down version of WPF similar to Silverlight. Microsoft has no plans to go back to OS controlled UI styles; Metro and WPF are the plan for the next 15 years. (I attended the MS BUILD conference, they made this plan very ... nay, EXTREMELY clear). Open-source is chronically behind the big boys with money, precisely because FOSS doesn't have the money to sling around for Testing and Usability Studies, and most FOSS guys don't want to mess around with that stuff anyways. You see FOSS guys tend to be engineers; they can put up with, and even like, industrial looking interfaces. But programmers also have a giant blind-spot when it comes to users. Most programmers view users as a lower species and assume that they will be delighted by whatever the programmer deigns to bequeath to them. But if you look at the successful people in the tech industry (*ahem* Steve Jobs) you'll find an attitude that is the exact opposite. Jobs was so focused on delivering what the user wanted that he would publicly berate any programmer who thought they knew better than the designer. While I don't necessarily agree with Jobs' management style, there is a reason why Apple is the second largest company in the world right now, and it has nothing to do with how good Apple's engineering is (which I hear is average at best). Despite programmers' best efforts, the world of technology is no longer controlled by programmers. For better or worse, users determine our course now. The open-source community would do well to embrace the user. But without a first-class UI framework, that will never happen. In terms of capability and usability, both Apple and Microsoft have beat the best FOSS has to offer by a decade at least. I looked, searched, and scoured, but the fact of the matter is, even the usable FOSS UI offerings are pitiful when compared to the commercial offerings. The Horizon Project got it's start because there has been a trend in recent commercial UI offerings from towards increasing reliance on the operating system itself. Metro XAML just flat won't work on anything other than Windows 8, Silverlight is a second class citizen at MS now, Cocoa only works on Mac etc. My goal with The Horizon Project is to create First-Class UI Framework for multiple platforms so that programmers don't have to rewrite the UI from scratch for each new platform they want to support and then open-source it so that the commercial OS vendors can't pervert it for their own purposes. I want to put [some of] the power back into the programmers hands. Apologies for the length, but this is a topic that is of some interest to me. :-)
 I don't understand this one. Should the compiler disable reflection as  
 soon as it sees malloc/free? On what level should it be disabled? I  
 mean, the runtime needs to be able to use these functions to implement  
 the memory manager, i.e. the garbage collector and other things as well.

The idea is to automatically prevent reflection access to sections of the program where using direct memory manipulation could potentially result in security holes but provide a way out for the programmer if they really wanted it. C/C++ is famous for programmer bugs around memory handling. If the compiler automatically disabled reflection, by sticking noreflect in front of a function that used malloc or new, it could potentially prevent those types of memory manipulation flaws and help keep the Reflection attacks to a minimum. It was an idea that I was throwing out there. But I don't know enough about D yet to know if it's the right way to handle it. And I have to admit I am little confused though as I would hope that Reflection would be disabled on the GC, because I have never personally had a reason to reflect into the GC... -- Adam Wilson Project Coordinator The Horizon Project http://www.thehorizonproject.com/
Oct 27 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
I think the compile-time reflection should be improved first, because
run-time reflection can be built upon that.
Currently D's compile-time reflection is not fully developed.
Oct 28 2011
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Fri, 28 Oct 2011 00:07:57 -0700, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-27 20:42, Adam Wilson wrote:
 On Thu, 27 Oct 2011 00:14:35 -0700, Jacob Carlborg <doob me.com> wrote:

 Are you saying that you consider using D for this Horizon project? I
 can recommend you take a look at DWT: www.dsource.org/projects/dwt

 Somewhere down the road I've planed to create an interface/window
 builder for DWT using XML or something similar. I'm thinking something
 like how it works on Mac OS X using Interface Builder.

I am, and I have looked at DWT. My problem with it is a one that is endemic to open-source UI framework. Microsoft recognized a decade ago that UI widgets whose look-and-feel is defined and controlled by the Operating System are going the way of the dodo. Out of that realization WPF was born. UI designers today want the ability to control every pixel of the UI's presentation. And the reasons for this are two-fold. The first is that it turns out that most OS designers are fantastically bad at UI and design in general. It takes epic piles of cash to pull of a decent one and even then there is a still a "programmers were here" look to it (with the notable exception of iOS/OSX where designers rule the roost).

I can agree to that.
 The second is product differentiation. Nobody wants an app that
 looks like every other app because it actually becomes impossible for
 the user to distinguish which app works best for them.

I want that. Because I know how the GUI works and it will be easy for me to learn if it follows the guidelines of the platform. Also I don't need to figure out if I can use the scroll wheel on the mouse on this, what looks like a, scroll bar. If the application have a native look and feel I know how to use the widgets.

Ahh, thats the true power of WPF. Presentation and Implementation are cleanly separated. I recent re-skinned every scrollbar in one of our WPF apps by writing the 30-line style once, placing it in the global resource dictionary with implicit styling enabled, and bam, every scrollbar in the app looks different, but functions in the exact same way any windows user would expect. There are cases where you can write code that calls up into the presentation layer and thus introducing a dependency on the presentation layer, but this is actively discouraged, and when I found one recently in a 3rd party vendors control, it was removed by the next release. I can go on and on, but the point is, there are only two frameworks in existence that can do that are the WPF family and JavaFX. JavaFX is pathetic compared to WPF and is of course tied to an Oracle *shudder* controlled langauge. And a few months ago we had thousands of developers on the Silverlight forum looking for WPF family alternatives, all that they could find was JavaFX.
 Users ONLY look
 at the UI, and if the app doesn't look good, they wont "buy" it, even if
 it's free. This is non-negotiable. Users, when given two apps that do
 the same thing, even for different prices, will pick the prettier one
 every time, because the prettier one is perceived as being "better".
 It's called the Attractiveness Bias and it is a well-known principle in
 the design world. Who would you rather look at all day, Alessandra
 Ambrosio or Rosie O'Donnell? I rest my case.

That's why I use Mac OS X where the native applications look good :)
 I maintain that this is prime reason that Linux on the desktop has
 failed miserably, and I think Android proves my point. Android's key win
 is that it put a usable UI on top of Linux. People never had a problem
 with the price of Linux, they just couldn't stand to look at it. The
 Linux Desktop LOOKS industrial and it's apps for the most part look the
 same (I know of a few outliers that did a good job, but it isn't the
 norm).

I can agree to that.
 My point is that the day of cookie cutter apps is over. Anyone
 designing for that paradigm is history. Microsoft's latest UI paradigm,
 "Metro", is just a different cut-down version of WPF similar to
 Silverlight. Microsoft has no plans to go back to OS controlled UI
 styles; Metro and WPF are the plan for the next 15 years. (I attended
 the MS BUILD conference, they made this plan very ... nay, EXTREMELY
 clear).

I don't know how Metro or WPF is implemented but how says they can't be the native look and feel of the OS.

Nobody, in fact WPF has three default looks out of the box, one for Windows 2000, another for WinXP, and another for Aero (Vista+). WPF defaults to the correct one for that system unless you override it. It's just that in my experience, the default look is a lowest common demoninator, and not a good one.
 Open-source is chronically behind the big boys with money, precisely
 because FOSS doesn't have the money to sling around for Testing and
 Usability Studies, and most FOSS guys don't want to mess around with
 that stuff anyways. You see FOSS guys tend to be engineers; they can put
 up with, and even like, industrial looking interfaces. But programmers
 also have a giant blind-spot when it comes to users. Most programmers
 view users as a lower species and assume that they will be delighted by
 whatever the programmer deigns to bequeath to them. But if you look at
 the successful people in the tech industry (*ahem* Steve Jobs) you'll
 find an attitude that is the exact opposite. Jobs was so focused on
 delivering what the user wanted that he would publicly berate any
 programmer who thought they knew better than the designer. While I don't
 necessarily agree with Jobs' management style, there is a reason why
 Apple is the second largest company in the world right now, and it has
 nothing to do with how good Apple's engineering is (which I hear is
 average at best). Despite programmers' best efforts, the world of
 technology is no longer controlled by programmers. For better or worse,
 users determine our course now. The open-source community would do well
 to embrace the user.

I have no trouble in letting a designer decide how GUI should look like, in fact, as you say it would be better if they did. But not everyone can have that luxury. Since I'm no designer you do the best I can.

Neither am I really, I am a programmer by trade who cut his teeth back in the C++ days before C# came along and made me actually productive, but design isn't that hard, if one is willing to let go of their preferences. More to the point though, my company has no on staff designer, so it was either pay large sums of money to a consultant, or I as team lead could do it. I lost. To some degree it's a totally different way of thinking and I find that after a good design session (a week or two), it takes me a couple of days to realign my brain to left-brain-progammer-mode. But hey, they pay me, I jump to. *shrug* There is no mystical powers that are given to designers, anyone can make a good design.
 But without a first-class UI framework, that will never happen. In terms
 of capability and usability, both Apple and Microsoft have beat the best
 FOSS has to offer by a decade at least. I looked, searched, and scoured,
 but the fact of the matter is, even the usable FOSS UI offerings are
 pitiful when compared to the commercial offerings. The Horizon Project
 got it's start because there has been a trend in recent commercial UI
 offerings from towards increasing reliance on the operating system
 itself. Metro XAML just flat won't work on anything other than Windows
 8, Silverlight is a second class citizen at MS now, Cocoa only works on
 Mac etc. My goal with The Horizon Project is to create First-Class UI
 Framework for multiple platforms so that programmers don't have to
 rewrite the UI from scratch for each new platform they want to support
 and then open-source it so that the commercial OS vendors can't pervert
 it for their own purposes. I want to put [some of] the power back into
 the programmers hands.

 Apologies for the length, but this is a topic that is of some interest
 to me. :-)

I's a topic of interest to me as well. But I, on the other hand, prefer a native look and fell of applications. Instead of having yet another application with yet another GUI that doesn't work properly. If I see something that looks like a scroll bar I assume I can use it like a scroll bar. But that's not true for many applications, instead they implement the bare minimum for having the scroll bar "work", i.e. I may not be able to scroll using the wheel on the mouse. A great example of this are scroll bars in games. In my experience of non-native GUI's they always perform worse the native GUI's.

Actually we found that by re-skinning our app intelligently we were able to load a large list of complex data 4x faster than the default skin. The reason for this is that in WPF all drawing is done the same way, it doesn't matter if it's a system skin or not, it's just a skin, there are no special shortcuts for the system skin to take. At it's heart WPF is based on DirectX9. This means that screen complexity is the ONLY limiting factor. The other thing that WPF allows is data / presentation separation. I can pass a list box a chunk of data, and the data does not have to concern itself with how the presentation looks or vice versa. The listbox will display the data according to however the style is setup it. And THEN if I really want to I can change the presentation style on the fly based on user input. Can DWT do that in 2 lines of code? WPF Can. That's the power of reflection based data-binding. In fact I spend most of my time writing (much more compact) XAML code to do presentation layer work than I do write C#, to the point where the C# coding makes up less than half of the work done on a screen. It's usually things like load this item based on a selection changed event or some such that XAML has no way to do (and shouldn't, thats implementation).
 Anyway, you can still use DWT to draw your own widgets. Even if you  
 don't use any native widgets you still need to be able to draw something  
 on a surface. DWT has a platform independent way of doing that.

Yea, but how much work is it? Can I create a completely new look for a control in 30 lines or less without modifying the implementation code at all to make the new look work? That's the power I'm after.
 IBM Lotus Software and IBM Rational Software are built on SWT (which DWT  
 is a port of) uses a lot a non-native widgets. There's an application  
 called "Azureus" that had its own GUI that didn't look native at all,  
 apparently it's called Vuze these days and it looks native.

That's IBM, they have supertanker loads of money, the open-source world does not. I expect IBM to re-skin anything they want to their specifications, they can afford it. WPF type apps are game changers in that they allow very small teams to create powerful, beautiful, apps in something less than a decade, and without requiring multi-million dollar investments. As an example, our flagship app (which we are keeping under tight wraps for the following reason) has the potential to completely wipe out multiple billions of dollars worth of corporate market capitalization and probably sink some long-time industry names in the process, all with a team of less than 10 guys. At least, thats our goal. But that goal is only achievable because we can move orders of magnitude faster than they can. It also doesn't hurt that their interface looks like it was designed in 1999 (seriously) and is definitely a traditional type of framework (the standard Java one IIRC). Thank you WPF, it simply wouldn't be possible without you. So when I say the world is leaving behind traditional UI frameworks, I am not kidding; it just wont happen that quickly as corp IT loathes change, but it's already starting. (See: iOS and Android apps and CxO's demanding their internal apps on those platforms.) Now imagine a world where not only can open-source compete with the big-boys, but take serious chunks out of their consumer mind-share. I want that. But, the only way that is happening is with a framework that allows open-source to compete at the same speed as the big boys can afford while delivering what the customer wants. Any app is only as good as the number of people using it. It's not the big who eat the small, it's the fast who eat the slow. :-)
 I don't understand this one. Should the compiler disable reflection as
 soon as it sees malloc/free? On what level should it be disabled? I
 mean, the runtime needs to be able to use these functions to implement
 the memory manager, i.e. the garbage collector and other things as  
 well.

The idea is to automatically prevent reflection access to sections of the program where using direct memory manipulation could potentially result in security holes but provide a way out for the programmer if they really wanted it. C/C++ is famous for programmer bugs around memory handling. If the compiler automatically disabled reflection, by sticking noreflect in front of a function that used malloc or new, it could potentially prevent those types of memory manipulation flaws and help keep the Reflection attacks to a minimum. It was an idea that I was throwing out there. But I don't know enough about D yet to know if it's the right way to handle it. And I have to admit I am little confused though as I would hope that Reflection would be disabled on the GC, because I have never personally had a reason to reflect into the GC...

What happens when you use class A from class B and the compiler has added noreflect to class A? Will it add it to B as well? If not, how does the compiler know that? There has been a similar discussion about having the compiler insert "pure" automatic on functions. But what might happens is a function that is called by another function change it implementation making it no longer pure. Which means your function will no longer be pure and you have no idea about it.

-- Adam Wilson Project Coordinator The Horizon Project http://www.thehorizonproject.com/
Oct 28 2011
prev sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Sat, 29 Oct 2011 03:59:28 -0700, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-28 21:36, Adam Wilson wrote:
 Ahh, thats the true power of WPF. Presentation and Implementation are
 cleanly separated. I recent re-skinned every scrollbar in one of our WPF
 apps by writing the 30-line style once, placing it in the global
 resource dictionary with implicit styling enabled, and bam, every
 scrollbar in the app looks different, but functions in the exact same
 way any windows user would expect. There are cases where you can write
 code that calls up into the presentation layer and thus introducing a
 dependency on the presentation layer, but this is actively discouraged,
 and when I found one recently in a 3rd party vendors control, it was
 removed by the next release.
 I can go on and on, but the point is, there are only two frameworks in
 existence that can do that are the WPF family and JavaFX. JavaFX is
 pathetic compared to WPF and is of course tied to an Oracle *shudder*
 controlled langauge. And a few months ago we had thousands of developers
 on the Silverlight forum looking for WPF family alternatives, all that
 they could find was JavaFX.

It's not the look of the, in this example, scroll bar, it's the behavior. It may look like a scroll bar but it may not behave in all the ways as a real, native, scroll bar would.
 Nobody, in fact WPF has three default looks out of the box, one for
 Windows 2000, another for WinXP, and another for Aero (Vista+). WPF
 defaults to the correct one for that system unless you override it. It's
 just that in my experience, the default look is a lowest common
 demoninator, and not a good one.

To me it sounds like WPF is the native GUI. Then why are you complaining about native GUI's?

WPF isn't the native GUI, it just comes with skins that emulate the look of the native GUI. The native GUI in Windows is called Win32 GDI (Graphices Device Interface). WPF is built on top of DirectX which is analagous to OpenGL and is primarily used for games. DirectX completely bypasses the GDI and communicates directly with the hardware. It's primary claim to fame is that it is fast, but it's not as compatible, whereas GDI is compatible with everything that implements VESA (which is everything since the early 90's), but it's slow.
 Neither am I really, I am a programmer by trade who cut his teeth back
 in the C++ days before C# came along and made me actually productive,
 but design isn't that hard, if one is willing to let go of their
 preferences. More to the point though, my company has no on staff
 designer, so it was either pay large sums of money to a consultant, or I
 as team lead could do it. I lost. To some degree it's a totally
 different way of thinking and I find that after a good design session (a
 week or two), it takes me a couple of days to realign my brain to
 left-brain-progammer-mode. But hey, they pay me, I jump to. *shrug*
 There is no mystical powers that are given to designers, anyone can make
 a good design.

I'm just saying that I'm not a good designer and if a designer is available I would prefer that they did the design.
 Actually we found that by re-skinning our app intelligently we were able
 to load a large list of complex data 4x faster than the default skin.
 The reason for this is that in WPF all drawing is done the same way, it
 doesn't matter if it's a system skin or not, it's just a skin, there are
 no special shortcuts for the system skin to take. At it's heart WPF is
 based on DirectX9. This means that screen complexity is the ONLY
 limiting factor.

If the changing the skin of an application can increase the data load by four times it sounds like something is seriously wrong. On Mac OS X a list only loads the data that is displayed so I don't think the amount of data matters. The same can be done using virtual tables in DWT.

Ok, so I apologize for generalizing there. It turns out that WPF is really bad at drawing round shapes, and the default skins (which the exception of the Windows 2000 skin) rely heavily on rounded corners. We replaced the default skin with one that used square corners and had fewer Measure and Arrange stages (where WPF figures out where to place the controls) which are equally expensive. WPF has virtual lists as well, and we do use it, which also helped contribute to our speed gains, but it is not enabled by default. My complaint is that the working with WPF requires a non-trivial amount of esoteric experience with it to make it work well.
 The other thing that WPF allows is data / presentation separation. I can
 pass a list box a chunk of data, and the data does not have to concern
 itself with how the presentation looks or vice versa. The listbox will
 display the data according to however the style is setup it. And THEN if
 I really want to I can change the presentation style on the fly based on
 user input. Can DWT do that in 2 lines of code? WPF Can. That's the
 power of reflection based data-binding. In fact I spend most of my time
 writing (much more compact) XAML code to do presentation layer work than
 I do write C#, to the point where the C# coding makes up less than half
 of the work done on a screen. It's usually things like load this item
 based on a selection changed event or some such that XAML has no way to
 do (and shouldn't, thats implementation).

No, probably not. It's not as easy to change the theme in SWT/DWT since it's supposed to use the native look and feel. It appears that you can at least set colors and similar properties using CSS: http://jaxenter.com/css-styling-in-eclipse-helios-32465.html
 Anyway, you can still use DWT to draw your own widgets. Even if you
 don't use any native widgets you still need to be able to draw
 something on a surface. DWT has a platform independent way of doing  
 that.

Yea, but how much work is it? Can I create a completely new look for a control in 30 lines or less without modifying the implementation code at all to make the new look work? That's the power I'm after.

No you probably can't. I don't know what you want to achieve with your project but I got the impression that you were looking to build a new GUI library that behaves something like WPF. Regardless of how this GUI library works you need somehow to draw the widgets on the screen. I'm just saying that DWT has a platform independent way of doing this. On the other hand, if you're creating a completely new GUI library you probably would not want to depend on DWT. But instead creating bindings directly to the underlying drawing operations for Cairo, Quartz and DirectX or whatever Windows uses these days.

I am looking to create an open-source implementation of a GUI framework that is 'lookless', i.e. it's underlying implementation has no dependencies on how it looks and the look can be changed to however the designer wants it to look without having to worry about code. WPF and JavaFX are the only two frameworks that are examples of how this might work. I absolutely agree that relying on something else is probably not the best idea, because for better or worse you end up inheriting their design choices, which may or may not work for your project. Since I am windows guy, the reference implementation is going to be DirectX. First I just have to create Direct2D bindings for D... And another goal of the framework beyond platform independence is language independence. I have to admit that one of the big draws for me of D was the C++ ABI and COM compatibility of D while having a MUCH more modern capability set. However, it's still lacking in a few key areas for my project, and runtime reflection is one of them... -- Adam Wilson Project Coordinator The Horizon Project http://www.thehorizonproject.com/
Oct 29 2011