www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Chances of D getting proper runtime reflection?

reply Jacob Carlborg <doob me.com> writes:
What are the chances of D getting proper runtime reflection? Something 
like this:

class Foo
{
     private int a;

     private void bar (int i)
     {
         a = i;
     }
}

auto foo = new Foo;
Object o = foo;
o.setInstanceVariable("a", 3);
assert(foo.a == 3);

This would be useful as well:

o.send("bar", 5);
assert(foo.a == 5);
assert(foo.a == o.getInstanceVariable("a"));

Currently I have most use of the first example. These example should 
work regardless if the fields/methods are private, protected or public.

As far as I know this wouldn't require any changes to the language, only 
the compiler and runtime would need to be modified. To me, it doesn't 
look that complicated to implement, the runtime just needs to store an 
associative arrays that maps names of instance variables and methods to 
their addresses. This could probably be stored in the ClassInfo. 
Something like this:

Foo.classinfo.setInstanceVariable(foo, "a", 3);

And this could be shorted to:

o.setInstanceVariable("a", 3);

"setInstanceVariable" could just forward the call to the class info.

-- 
/Jacob Carlborg
Aug 21 2011
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Jacob Carlborg:

 class Foo
 {
      private int a;
 
      private void bar (int i)
      {
          a = i;
      }
 }
 
 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

Is this related to the Delphi feature I've recently shown? http://blog.barrkel.com/2010/09/virtual-method-interception.html I presume you will have to write: reflection class Foo { private int a; ... To be sure to not pay for this dictionary when you don't need it. In theory this annotation is unnecessary. Bye, bearophile
Aug 21 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 13:41, bearophile wrote:
 Jacob Carlborg:

 class Foo
 {
       private int a;

       private void bar (int i)
       {
           a = i;
       }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

Is this related to the Delphi feature I've recently shown? http://blog.barrkel.com/2010/09/virtual-method-interception.html

I don't think so, but it might help.
 I presume you will have to write:

  reflection class Foo
 {
      private int a;
 ...

 To be sure to not pay for this dictionary when you don't need it. In theory
this annotation is unnecessary.

 Bye,
 bearophile

That's not an option because it needs to work on third party types as well. BTW, you're already paying for the class info. -- /Jacob Carlborg
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 14:55, Marco Leise wrote:
 Am 21.08.2011, 14:11 Uhr, schrieb Jacob Carlborg <doob me.com>:

 BTW, you're already paying for the class info.

That could be a necessity of D's OOP implementation to fulfill the specification and while it is there that data was exported. Delphi used to have a compiler switch to include RTTI. (If I remember correctly without the compiler switch the pointer to RTTI was simply null.) Do you think that is an option? I figure if a third party library wanted it's classes to be reflected at runtime they would compile their code with that switch on. I don't quite know what your use-case is. For object serialization in Java there is an interface that has to be implemented. This would be similar to the reflected annotation bearophile mentioned, because it is mostly a tag. RTTI for specific classes is not useless at all. I have a module with a lot of classes in a flat hierarchy. They are the typical case of several "commands" inheriting from a base class/interface declaring an "execute" method and such. Since I have the complete list of commands I could annotate them all with reflected to set their properties from some XML definition. The usefulness of an annotation based RTTI depends solely on how many use cases it can cover. It doesn't hurt anyone who doesn't want to use it. The same goes for the compiler switch. Full reflection like in Java ... oh wait. Do you plan on writing a debugger? That's the only use case I can come up with right now? - Marco

It would help for my serialization library. Currently I can only get/set fields based on compile time strings via .tupleof. Currently I'm looping over the tuple and calling the archive, like: "given me the value of field 'foo'". Instead the archive could run through the data and to something this: "ok, now I'm at field 'foo', I'm just setting the value of the field". -- /Jacob Carlborg
Aug 21 2011
prev sibling parent "Marco Leise" <Marco.Leise gmx.de> writes:
Am 21.08.2011, 14:11 Uhr, schrieb Jacob Carlborg <doob me.com>:

 BTW, you're already paying for the class info.

That could be a necessity of D's OOP implementation to fulfill the specification and while it is there that data was exported. Delphi used to have a compiler switch to include RTTI. (If I remember correctly without the compiler switch the pointer to RTTI was simply null.) Do you think that is an option? I figure if a third party library wanted it's classes to be reflected at runtime they would compile their code with that switch on. I don't quite know what your use-case is. For object serialization in Java there is an interface that has to be implemented. This would be similar to the reflected annotation bearophile mentioned, because it is mostly a tag. RTTI for specific classes is not useless at all. I have a module with a lot of classes in a flat hierarchy. They are the typical case of several "commands" inheriting from a base class/interface declaring an "execute" method and such. Since I have the complete list of commands I could annotate them all with reflected to set their properties from some XML definition. The usefulness of an annotation based RTTI depends solely on how many use cases it can cover. It doesn't hurt anyone who doesn't want to use it. The same goes for the compiler switch. Full reflection like in Java ... oh wait. Do you plan on writing a debugger? That's the only use case I can come up with right now? - Marco
Aug 21 2011
prev sibling next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 21/08/2011 11:55, Jacob Carlborg wrote:
 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.

You can actually do all of this now (albeit using a mixin), there's just no standard way of doing it. All you need to do is use compile time reflection to pull out all the fields/methods then store them for use at runtime. It would be good to get this functionality into phobos so there's a standard way of doing it though. The plus side to having to use a mixin is you don't get loads of bloat from classes you never want to reflect at runtime. -- Robert http://octarineparrot.com/
Aug 21 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 13:42, Robert Clipsham wrote:
 On 21/08/2011 11:55, Jacob Carlborg wrote:
 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.

You can actually do all of this now (albeit using a mixin), there's just no standard way of doing it. All you need to do is use compile time reflection to pull out all the fields/methods then store them for use at runtime. It would be good to get this functionality into phobos so there's a standard way of doing it though. The plus side to having to use a mixin is you don't get loads of bloat from classes you never want to reflect at runtime.

Yes, I know. But the point is to be able to do it without the use of mixins. It needs to work with third party types, otherwise there no use. It also needs to work when the static type is Object. You're already paying for the class info, which already contains getMembers. But this methods always returns an empty array and as far as I can see, it's not possible to set/get fields or call methods via these member instances. -- /Jacob Carlborg
Aug 21 2011
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 21-08-2011 15:13, Vladimir Panteleev wrote:
 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:

 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.

I think the best way to do this is to improve compile-time reflection to the point where it'd be possible to enumerate all declarations in a module and create run-time reflection information for them - then you'd just need to specify the modules for which you'd like run-time reflection. One thing to note is that there are use cases where more run-time information about a program's internals may not be desired, e.g. with commercial closed-source applications, so this would need to be optional.

I really don't think this is a valid cause of concern in this day and age. Look at Java and .NET. They're doing just fine even with all metadata being available. Plenty of commercial C++ products ship with RTTI these days, too (AAA games, even). What I mean to say is, I don't think we should let this "concern" hamper runtime reflection in D, because it evidently is not as big of an issue as people have made it out to be in the past. Security through obscurity is bad, as the saying goes. Regards, Alex
Aug 21 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 15:13, Vladimir Panteleev wrote:
 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:

 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.

I think the best way to do this is to improve compile-time reflection to the point where it'd be possible to enumerate all declarations in a module and create run-time reflection information for them - then you'd just need to specify the modules for which you'd like run-time reflection. One thing to note is that there are use cases where more run-time information about a program's internals may not be desired, e.g. with commercial closed-source applications, so this would need to be optional.

This is for a serialization library where third party types need to be serializable. I don't like the idea of annotate everything that should be serializable. -- /Jacob Carlborg
Aug 21 2011
next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that. -- Robert http://octarineparrot.com/
Aug 21 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work. -- /Jacob Carlborg
Aug 21 2011
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/21/11 1:19 PM, Jacob Carlborg wrote:
 On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work.

If we get to the point where adding one line of code inside or (preferably) outside a type to make it serializable, that would be great. Andrei
Aug 21 2011
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:
 On 8/21/11 1:19 PM, Jacob Carlborg wrote:
 On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work.

If we get to the point where adding one line of code inside or (preferably) outside a type to make it serializable, that would be great. Andrei

I've been playing with Jacob's code and we're already at that point. Example (requires Jacob's Orange library to be installed): import std.stdio, orange.serialization._, orange.serialization.archives._, std.exception; class Base { string baseString; } class Derived : Base { string derivedString; } void main() { auto archiver = new XMLArchive!char(); auto serializer = new Serializer(archiver); serializer.register!Derived(); auto derived = new Derived; derived.baseString = "Base"; derived.derivedString = "Derived"; Base baseRef = derived; auto serialized = serializer.serialize(baseRef); auto baseRef2 = serializer.deserialize!(Base)(serialized); enforce(baseRef2.classinfo is Derived.classinfo); auto derived2 = cast(Derived) baseRef; writeln(derived2.baseString, ' ', derived2.derivedString); } The registration is not even necessary if you're serializing a class with a compile time type identical to its runtime type. Of course the situation gets a little more complicated when classes have references to other classes (you need to register everything that could be transitively reachable) but it's still pretty good.
Aug 21 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/21/11 6:03 PM, dsimcha wrote:
 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:
 On 8/21/11 1:19 PM, Jacob Carlborg wrote:
 On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work.

If we get to the point where adding one line of code inside or (preferably) outside a type to make it serializable, that would be great. Andrei

I've been playing with Jacob's code and we're already at that point. Example (requires Jacob's Orange library to be installed): import std.stdio, orange.serialization._, orange.serialization.archives._, std.exception; class Base { string baseString; } class Derived : Base { string derivedString; } void main() { auto archiver = new XMLArchive!char(); auto serializer = new Serializer(archiver); serializer.register!Derived(); auto derived = new Derived; derived.baseString = "Base"; derived.derivedString = "Derived"; Base baseRef = derived; auto serialized = serializer.serialize(baseRef); auto baseRef2 = serializer.deserialize!(Base)(serialized); enforce(baseRef2.classinfo is Derived.classinfo); auto derived2 = cast(Derived) baseRef; writeln(derived2.baseString, ' ', derived2.derivedString); } The registration is not even necessary if you're serializing a class with a compile time type identical to its runtime type. Of course the situation gets a little more complicated when classes have references to other classes (you need to register everything that could be transitively reachable) but it's still pretty good.

That's not how it should work. The line of code needs to be declarative, not imperative. "I want this class to be serializable" as opposed to "I want to serialize this class with this particular archiver". Andrei
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 21:08, Andrei Alexandrescu wrote:
 On 8/21/11 6:03 PM, dsimcha wrote:
 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:
 On 8/21/11 1:19 PM, Jacob Carlborg wrote:
 On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need
 to be
 serializable. I don't like the idea of annotate everything that
 should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work.

If we get to the point where adding one line of code inside or (preferably) outside a type to make it serializable, that would be great. Andrei

I've been playing with Jacob's code and we're already at that point. Example (requires Jacob's Orange library to be installed): import std.stdio, orange.serialization._, orange.serialization.archives._, std.exception; class Base { string baseString; } class Derived : Base { string derivedString; } void main() { auto archiver = new XMLArchive!char(); auto serializer = new Serializer(archiver); serializer.register!Derived(); auto derived = new Derived; derived.baseString = "Base"; derived.derivedString = "Derived"; Base baseRef = derived; auto serialized = serializer.serialize(baseRef); auto baseRef2 = serializer.deserialize!(Base)(serialized); enforce(baseRef2.classinfo is Derived.classinfo); auto derived2 = cast(Derived) baseRef; writeln(derived2.baseString, ' ', derived2.derivedString); } The registration is not even necessary if you're serializing a class with a compile time type identical to its runtime type. Of course the situation gets a little more complicated when classes have references to other classes (you need to register everything that could be transitively reachable) but it's still pretty good.

That's not how it should work. The line of code needs to be declarative, not imperative. "I want this class to be serializable" as opposed to "I want to serialize this class with this particular archiver". Andrei

You mean you want something like this: class Derived : Base { mixin serializable; } Instead of: serializer.register!Derived(); The only way I can think of how this could work is to store a global list of the serializable classes. Any other suggestion how this would work? Note that all this wouldn't be necessary if my original post was implemented. -- /Jacob Carlborg
Aug 22 2011
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08/22/2011 09:19 PM, Jacob Carlborg wrote:
 On 2011-08-22 21:08, Andrei Alexandrescu wrote:
 On 8/21/11 6:03 PM, dsimcha wrote:
 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:
 On 8/21/11 1:19 PM, Jacob Carlborg wrote:
 On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need
 to be
 serializable. I don't like the idea of annotate everything that
 should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work.

If we get to the point where adding one line of code inside or (preferably) outside a type to make it serializable, that would be great. Andrei

I've been playing with Jacob's code and we're already at that point. Example (requires Jacob's Orange library to be installed): import std.stdio, orange.serialization._, orange.serialization.archives._, std.exception; class Base { string baseString; } class Derived : Base { string derivedString; } void main() { auto archiver = new XMLArchive!char(); auto serializer = new Serializer(archiver); serializer.register!Derived(); auto derived = new Derived; derived.baseString = "Base"; derived.derivedString = "Derived"; Base baseRef = derived; auto serialized = serializer.serialize(baseRef); auto baseRef2 = serializer.deserialize!(Base)(serialized); enforce(baseRef2.classinfo is Derived.classinfo); auto derived2 = cast(Derived) baseRef; writeln(derived2.baseString, ' ', derived2.derivedString); } The registration is not even necessary if you're serializing a class with a compile time type identical to its runtime type. Of course the situation gets a little more complicated when classes have references to other classes (you need to register everything that could be transitively reachable) but it's still pretty good.

That's not how it should work. The line of code needs to be declarative, not imperative. "I want this class to be serializable" as opposed to "I want to serialize this class with this particular archiver". Andrei

You mean you want something like this: class Derived : Base { mixin serializable; } Instead of: serializer.register!Derived(); The only way I can think of how this could work is to store a global list of the serializable classes. Any other suggestion how this would work? Note that all this wouldn't be necessary if my original post was implemented.

I think that you maybe register too late in the process?
 auto archiver = new XMLArchive!char();
 auto serializer = new Serializer(archiver);
 serializer.register!Derived();

If it works that way, you have to register every class anew if you want to serialize it with another archiver, right?
Aug 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 22:00, Timon Gehr wrote:
 On 08/22/2011 09:19 PM, Jacob Carlborg wrote:
 On 2011-08-22 21:08, Andrei Alexandrescu wrote:
 On 8/21/11 6:03 PM, dsimcha wrote:
 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:
 On 8/21/11 1:19 PM, Jacob Carlborg wrote:
 On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need
 to be
 serializable. I don't like the idea of annotate everything that
 should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work.

If we get to the point where adding one line of code inside or (preferably) outside a type to make it serializable, that would be great. Andrei

I've been playing with Jacob's code and we're already at that point. Example (requires Jacob's Orange library to be installed): import std.stdio, orange.serialization._, orange.serialization.archives._, std.exception; class Base { string baseString; } class Derived : Base { string derivedString; } void main() { auto archiver = new XMLArchive!char(); auto serializer = new Serializer(archiver); serializer.register!Derived(); auto derived = new Derived; derived.baseString = "Base"; derived.derivedString = "Derived"; Base baseRef = derived; auto serialized = serializer.serialize(baseRef); auto baseRef2 = serializer.deserialize!(Base)(serialized); enforce(baseRef2.classinfo is Derived.classinfo); auto derived2 = cast(Derived) baseRef; writeln(derived2.baseString, ' ', derived2.derivedString); } The registration is not even necessary if you're serializing a class with a compile time type identical to its runtime type. Of course the situation gets a little more complicated when classes have references to other classes (you need to register everything that could be transitively reachable) but it's still pretty good.

That's not how it should work. The line of code needs to be declarative, not imperative. "I want this class to be serializable" as opposed to "I want to serialize this class with this particular archiver". Andrei

You mean you want something like this: class Derived : Base { mixin serializable; } Instead of: serializer.register!Derived(); The only way I can think of how this could work is to store a global list of the serializable classes. Any other suggestion how this would work? Note that all this wouldn't be necessary if my original post was implemented.

I think that you maybe register too late in the process? > auto archiver = new XMLArchive!char(); > auto serializer = new Serializer(archiver); > serializer.register!Derived(); If it works that way, you have to register every class anew if you want to serialize it with another archiver, right?

Yes, that's how it currently works. I can make the "register" method static instead. -- /Jacob Carlborg
Aug 22 2011
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/22/11 2:19 PM, Jacob Carlborg wrote:
 On 2011-08-22 21:08, Andrei Alexandrescu wrote:
 On 8/21/11 6:03 PM, dsimcha wrote:
 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:
 On 8/21/11 1:19 PM, Jacob Carlborg wrote:
 On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need
 to be
 serializable. I don't like the idea of annotate everything that
 should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work.

If we get to the point where adding one line of code inside or (preferably) outside a type to make it serializable, that would be great. Andrei

I've been playing with Jacob's code and we're already at that point. Example (requires Jacob's Orange library to be installed): import std.stdio, orange.serialization._, orange.serialization.archives._, std.exception; class Base { string baseString; } class Derived : Base { string derivedString; } void main() { auto archiver = new XMLArchive!char(); auto serializer = new Serializer(archiver); serializer.register!Derived(); auto derived = new Derived; derived.baseString = "Base"; derived.derivedString = "Derived"; Base baseRef = derived; auto serialized = serializer.serialize(baseRef); auto baseRef2 = serializer.deserialize!(Base)(serialized); enforce(baseRef2.classinfo is Derived.classinfo); auto derived2 = cast(Derived) baseRef; writeln(derived2.baseString, ' ', derived2.derivedString); } The registration is not even necessary if you're serializing a class with a compile time type identical to its runtime type. Of course the situation gets a little more complicated when classes have references to other classes (you need to register everything that could be transitively reachable) but it's still pretty good.

That's not how it should work. The line of code needs to be declarative, not imperative. "I want this class to be serializable" as opposed to "I want to serialize this class with this particular archiver". Andrei

You mean you want something like this: class Derived : Base { mixin serializable; } Instead of: serializer.register!Derived();

Yes. Better yet: class Derived : Base { ... } mixin(serializable!Derived()); i.e. so you can add serialization to classes you can't edit.
 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.

A global structure sounds the way to go. Andrei
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 22:26, Andrei Alexandrescu wrote:
 Yes. Better yet:

 class Derived : Base
 {
 ...
 }

 mixin(serializable!Derived());

 i.e. so you can add serialization to classes you can't edit.

Yeah, of course. So "The line of code needs to be declarative, not imperative" was just a fancy way of saying that you want to move the class registration to global scope :)
 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.

A global structure sounds the way to go. Andrei

Ok, then I just change "register" to a static method. -- /Jacob Carlborg
Aug 22 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 On 2011-08-22 22:26, Andrei Alexandrescu wrote:
 Yes. Better yet:

 class Derived : Base
 {
 ...
 }

 mixin(serializable!Derived());

 i.e. so you can add serialization to classes you can't edit.

Yeah, of course. So "The line of code needs to be declarative, not imperative" was just a fancy way of saying that you want to move the class registration to global scope :)
 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.

A global structure sounds the way to go. Andrei

Ok, then I just change "register" to a static method.

A static method of whom? Andrei
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 On 2011-08-22 22:26, Andrei Alexandrescu wrote:
 Yes. Better yet:

 class Derived : Base
 {
 ...
 }

 mixin(serializable!Derived());

 i.e. so you can add serialization to classes you can't edit.

Yeah, of course. So "The line of code needs to be declarative, not imperative" was just a fancy way of saying that you want to move the class registration to global scope :)
 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.

A global structure sounds the way to go. Andrei

Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer. -- /Jacob Carlborg
Aug 23 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei
Aug 23 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-23 16:38, Andrei Alexandrescu wrote:
 On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei

You don't want to have it in the class and don't want it in the serializer. I mean, it needs to be stored somewhere and I thought that a static method in Serializer would better than a completely global function. Are you thinking about having another serialization library that can use this information as well? I'm not sure if that's good idea, different serialization implementations might need to do very different things with the information. -- /Jacob Carlborg
Aug 23 2011
next sibling parent reply jdrewsen <jdrewsen nospam.com> writes:
Den 23-08-2011 17:03, Jacob Carlborg skrev:
 On 2011-08-23 16:38, Andrei Alexandrescu wrote:
 On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei

You don't want to have it in the class and don't want it in the serializer. I mean, it needs to be stored somewhere and I thought that a static method in Serializer would better than a completely global function. Are you thinking about having another serialization library that can use this information as well? I'm not sure if that's good idea, different serialization implementations might need to do very different things with the information.

It could be used for network transmissions. Correct me if I'm wrong but Orange serializes the entire object. When sending things over the network or when saving something to disk for that matter you most likely are only interested in serializing some of the fields of an object. I really think it would be nice to declaratively mark fields as serializable for certain purposes e.g.: class Foo { int a; // Send over network and saved to file int b; // Saved to file ubyte[] cache; // not to be serialized } mixin(serialize!(Network, Foo, a); mixin(serialize!(File, Foo, a); mixin(serialize!(File, Foo, b); // but still support not specifying each field class Bar { int a; } mixin(serialize!(File, Bar)); Another way would be to just declare a class as serializable and then for each serialization type (ie. Network, File) declare that they should skip a field. Actually I think I better like this approach since it would allow decoupling of serialization type and the declaration of serializability. /Jonas
Aug 23 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-08-23 19:42, jdrewsen wrote:
 Den 23-08-2011 17:03, Jacob Carlborg skrev:
 On 2011-08-23 16:38, Andrei Alexandrescu wrote:
 On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei

You don't want to have it in the class and don't want it in the serializer. I mean, it needs to be stored somewhere and I thought that a static method in Serializer would better than a completely global function. Are you thinking about having another serialization library that can use this information as well? I'm not sure if that's good idea, different serialization implementations might need to do very different things with the information.

It could be used for network transmissions. Correct me if I'm wrong but Orange serializes the entire object. When sending things over the network or when saving something to disk for that matter you most likely are only interested in serializing some of the fields of an object. I really think it would be nice to declaratively mark fields as serializable for certain purposes e.g.:

Yes, that's how it works, but there are several ways to customize the serialization process, see below. If you are interested, please look at the unit tests: https://github.com/jacob-carlborg/orange/tree/master/tests
 class Foo {
 int a; // Send over network and saved to file
 int b; // Saved to file
 ubyte[] cache; // not to be serialized
 }

 mixin(serialize!(Network, Foo, a);

 mixin(serialize!(File, Foo, a);
 mixin(serialize!(File, Foo, b);

 // but still support not specifying each field
 class Bar { int a; }
 mixin(serialize!(File, Bar));


 Another way would be to just declare a class as serializable and then
 for each serialization type (ie. Network, File) declare that they should
 skip a field. Actually I think I better like this approach since it
 would allow decoupling of serialization type and the declaration of
 serializability.

 /Jonas

Orange have several ways to solve this. 1. Mark a field as non-serialized class Foo { int a; mixin NonSeriailzed!(a); } 2. Mark a whole class as non-serialized class Foo { mixin NonSeriailzed; } This will be handy for socket, thread, file and so on. 3. Custom serialization class Foo { int a; void toData (Serializer serializer, Serializer.Data key) { // serialize the class how ever you want } void fromData (Serializer serializer, Serializer.Data key) { // deserialize the class how ever you want } } 4. Custom non-intrusive serialization class Foo { int a; } void serializeFoo (Foo foo, Serializer serializer, Serializer.Data key) { // serialize the class how ever you want } void deserializeFoo (Foo foo, Serializer serializer, Serializer.Data key) { // deserialize the class how ever you want } serializer.registerSerializer(Foo.classinfo.name, &serializeFoo); serializer.registerDeserializer(Foo.classinfo.name, &deserializeFoo); 5. Events class Foo { int a; void onSerializing () { // invoked before serializing of Foo } void onSerialized () { // invoked after serializing of Foo } void onDeserializing () { // invoked before deserializing of Foo } void onDeserialized () { // invoked after deserializing of Foo } mixin OnSerializing!(onSerializing); mixin OnSerialized!(onSerialized); mixin OnDeserializing!(onDeserializing); mixin OnDeserialized!(onDeserialized); } -- /Jacob Carlborg
Aug 23 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-24 08:41, Marco Leise wrote:
 Am 23.08.2011, 19:42 Uhr, schrieb jdrewsen <jdrewsen nospam.com>:

 Den 23-08-2011 17:03, Jacob Carlborg skrev:
 On 2011-08-23 16:38, Andrei Alexandrescu wrote:
 On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei

You don't want to have it in the class and don't want it in the serializer. I mean, it needs to be stored somewhere and I thought that a static method in Serializer would better than a completely global function. Are you thinking about having another serialization library that can use this information as well? I'm not sure if that's good idea, different serialization implementations might need to do very different things with the information.

It could be used for network transmissions. Correct me if I'm wrong but Orange serializes the entire object. When sending things over the network or when saving something to disk for that matter you most likely are only interested in serializing some of the fields of an object. I really think it would be nice to declaratively mark fields as serializable for certain purposes e.g.: class Foo { int a; // Send over network and saved to file int b; // Saved to file ubyte[] cache; // not to be serialized } mixin(serialize!(Network, Foo, a); mixin(serialize!(File, Foo, a); mixin(serialize!(File, Foo, b); // but still support not specifying each field class Bar { int a; } mixin(serialize!(File, Bar)); Another way would be to just declare a class as serializable and then for each serialization type (ie. Network, File) declare that they should skip a field. Actually I think I better like this approach since it would allow decoupling of serialization type and the declaration of serializability. /Jonas

These are good points Jonas. It's a good idea to read up on http://en.wikipedia.org/wiki/Serialization I used Java for a while and I they have their reasons to require a 'tag'/'annotation'/'interface' on serializable classes. Java's choice to not make serializable a default is explained in short like this: - structures like threads or files cannot maintain their semantics when serialized - classes and libraries change over time, this adds extra work for the maintanance of serializable classes - serialization means externalization of private attributes, even passwords .NET adds the option to mark newly added fields as OptionalField, so a previously serialized instance can be loaded without error. NonSerialized marks a field as not serializable which is what Jonas mentioned. Like Java they work with interfaces to declare methods called after deserialization, so the 'cache' in the above example can be reinitialized correctly.

Orange supports NonSerialized fields and classes, (de)serialization events and several other ways to customize the serialization process. It's inspired by how serialization in .NET works. For more information see the Orange unit tests: https://github.com/jacob-carlborg/orange/tree/master/tests -- /Jacob Carlborg
Aug 23 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-24 10:10, Marco Leise wrote:
 Then I'll clearly use Orange when I have a need for serialization. Nice
 work!

Thanks.
 I've taken a closer look at
 http://dsource.org/projects/orange/wiki/Features and I see that
 versioning is still missing. What are your plans there? What happens at
 the moment when I deserialize and object with missing/added fields?

I actually haven't tested that with the new implementation. The idea is to have a callback that is called when a field is missing, or something else goes wrong. Then you can handle it however you want, throw an exception, do nothing or something else. The default callback will throw an exception. You can manually implement versioning by adding a new field to a class, an integer, string or similar, containing a version number. Then you can implement the two methods "toData" and "fromData". When deserializing you being with deserializing the field containing the version and then you can do whatever you want depending on the version number. -- /Jacob Carlborg
Aug 24 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-24 01:34, Martin Nowak wrote:
 On Tue, 23 Aug 2011 17:03:12 +0200, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-23 16:38, Andrei Alexandrescu wrote:
 On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei

You don't want to have it in the class and don't want it in the serializer. I mean, it needs to be stored somewhere and I thought that a static method in Serializer would better than a completely global function. Are you thinking about having another serialization library that can use this information as well? I'm not sure if that's good idea, different serialization implementations might need to do very different things with the information.

I think the idea is to register something that provides the needed information at runtime and separate that from the serializer because it's useful on it's own. Could be as simple as: interface Serializable { string[] allMembers(); Member getMember(string name); } interface Member { final void set(T)(T t) { setRaw(typeid(t), cast(void*)&t); } void setRaw(TypeInfo ti, void* p); final T get(T)() { return *cast(T*)enforce(getRaw(typeid(T))); } void* getRaw(); } It's clearly interesting to make this available if you implement something along that line. But I would not overstress it for there are dozen of different requirements profiles (performance, versioning, pre/post-serialize hooks). So even if I am interested to use something similar for undo/redo or delta creation, I'm not sure one can find a one-for-all solution here (i.e. the above approach seems way to dynamic for my needs). Maybe Variant could help in setting/getting values. martin

As soon as "setRaw" is called all static information is gone. I need to be able to iterate over T.tupleof. -- /Jacob Carlborg
Aug 23 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-08-25 06:30, Martin Nowak wrote:
 It is actually already possible to store meta data in classinfos.

 ---
 import std.stdio;

 class A {
 int a;
 float b;
 }

 mixin template register(T) if(is(T == class)) {
 static this() {
 T.classinfo.m_offTi = [OffsetTypeInfo(0, typeid(int)), OffsetTypeInfo(4,
 typeid(float))];
 }
 }

 mixin register!A;

 void main() {
 foreach(offTi; A.classinfo.offTi())
 writeln(offTi.offset, offTi.ti);
 }
 ---

 The compiler doesn't initialize the m_offTi array. But classinfos are in
 the initialized data section, at least for me, so you can write to them.

 martin

Seems to be very unsafe to rely on m_offTi in ClassInfo. -- /Jacob Carlborg
Aug 25 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-08-25 05:31, Martin Nowak wrote:
 Not sure, I get what you mean.
 Let me try to clarify this:

 static Serializeable[TypeInfo] infos;

 void register(T)() {
 infos[typeid(T)] = new SerializableImpl!T();
 }

 // Serializable binds static reflection to dynamic interface
 class SerializeableImpl!T : Serializable {
 string[] allMembers() { array(__traits(allMembers, T)); }
 Member getMember(string name) {
 if (mi = findMemberInfo(name)) {
 auto res = mi.clone(); // found immutable info, now create wrapper
 res.address = this + mi.offset; // tell the clone where to find
 return res;
 }
 return null;
 }
 }

 // Member binds purely static info to dynamic interface and an actual
 address
 class MemberInfo!T : Member {
 override void setRaw(TypeInfo ti, void* p) {
 if (ti is typeid(T))
 *address = *cast(T*)p;
 }

 override void* getRaw(TypeInfo ti) {
 if (ti is typeid(T))
 return address;
 }

 T* address;
 }

 In object.d there is a lot of this partly implemented. Have a closer
 look at OffsetTypeInfo and MemberInfo_field.
 You can definitely get all the information the compiler could provide
 for RTTI at compile time. If you try hard
 enough you may even be able to store them at compile time. The only
 difference remaining is that you could not
 access it through instance.classinfo but have to use a table lookup.

 It would be great if there was a way to append data to classinfo at
 compile time. Like an extra pointer or an array of ubyte that can be set.
 martin

I have to take a good look at this but I can say right now that currently how "register" is implemented it needs access to instance methods and variables. -- /Jacob Carlborg
Aug 25 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 00:35, Andrei Alexandrescu wrote:
 On 8/21/11 1:19 PM, Jacob Carlborg wrote:
 On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work.

If we get to the point where adding one line of code inside or (preferably) outside a type to make it serializable, that would be great. Andrei

That's already what I have and it's only necessary to register subclasses that are serialized through base class references. If we get runtime reflection as I suggested in my original post, that wouldn't be needed. In addition to that I was hoping it would make it easier to implement a binary archive. Project page: http://www.dsource.org/projects/orange Source code: https://github.com/jacob-carlborg/orange -- /Jacob Carlborg
Aug 21 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/22/11 1:41 AM, Jacob Carlborg wrote:
 On 2011-08-22 00:35, Andrei Alexandrescu wrote:
 On 8/21/11 1:19 PM, Jacob Carlborg wrote:
 On 2011-08-21 17:30, Robert Clipsham wrote:
 On 21/08/2011 16:05, Jacob Carlborg wrote:
 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.

You don't need to annotate it: void serialize(T)(T something); You have access to all the CT reflection stuff, and can generate the necessary runtime reflection with that.

I already have a fully working serialization library. I'm hoping to make it better. The problem is when (de)serializing through a base class reference. Currently you need to register the subclass with the serializer. If T is Base and the actually runtime type is Sub it won't work.

If we get to the point where adding one line of code inside or (preferably) outside a type to make it serializable, that would be great. Andrei

That's already what I have and it's only necessary to register subclasses that are serialized through base class references. If we get runtime reflection as I suggested in my original post, that wouldn't be needed.

FWIW I don't think D should pay the cost of reflection up-front and compulsively for all types. If David's code example is what you have, I don't think Orange got "making a class serializable" properly yet. Andrei
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 21:16, Andrei Alexandrescu wrote:
 FWIW I don't think D should pay the cost of reflection up-front and
 compulsively for all types.

So ClassInfo.getMembers is there because?
 If David's code example is what you have, I don't think Orange got
 "making a class serializable" properly yet.


 Andrei

You do understand that this is ONLY necessary when serializing through a base class reference? Other suggestions are welcome. -- /Jacob Carlborg
Aug 22 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/22/11 2:22 PM, Jacob Carlborg wrote:
 On 2011-08-22 21:16, Andrei Alexandrescu wrote:
 FWIW I don't think D should pay the cost of reflection up-front and
 compulsively for all types.

So ClassInfo.getMembers is there because?

At this point the implementation is unclear on what should and what shouldn't be part of compiler-supplied metadata.
 If David's code example is what you have, I don't think Orange got
 "making a class serializable" properly yet.


 Andrei

You do understand that this is ONLY necessary when serializing through a base class reference? Other suggestions are welcome.

I do understand that point, which is unrelated to mine. Andrei
Aug 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 22:27, Andrei Alexandrescu wrote:
 On 8/22/11 2:22 PM, Jacob Carlborg wrote:
 On 2011-08-22 21:16, Andrei Alexandrescu wrote:
 FWIW I don't think D should pay the cost of reflection up-front and
 compulsively for all types.

So ClassInfo.getMembers is there because?

At this point the implementation is unclear on what should and what shouldn't be part of compiler-supplied metadata.
 If David's code example is what you have, I don't think Orange got
 "making a class serializable" properly yet.


 Andrei

You do understand that this is ONLY necessary when serializing through a base class reference? Other suggestions are welcome.

I do understand that point, which is unrelated to mine. Andrei

Ok, I just want to make sure because I think that the library is working pretty well and turned out quite good with the, somewhat, limit possibilities the language give you, i.e. no runtime reflection. -- /Jacob Carlborg
Aug 22 2011
prev sibling parent reply Kagamin <spam here.lot> writes:
Jacob Carlborg Wrote:

 I already have a fully working serialization library. I'm hoping to make 
 it better. The problem is when (de)serializing through a base class 
 reference. Currently you need to register the subclass with the 
 serializer. If T is Base and the actually runtime type is Sub it won't work.

In .net it won't work either. You have to specify known classes explicitly.
Aug 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 13:27, Kagamin wrote:
 Jacob Carlborg Wrote:

 I already have a fully working serialization library. I'm hoping to make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't work.

In .net it won't work either. You have to specify known classes explicitly.

As far as I know, this is just because they chose to do it this way, not because of limitations in the language/runtime. .Net has runtime reflection, there's no technical need to add an attribute to indicate that a class is serializable. -- /Jacob Carlborg
Aug 22 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 17:30, Vladimir Panteleev wrote:
 On Sun, 21 Aug 2011 18:05:20 +0300, Jacob Carlborg <doob me.com> wrote:

 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.

I think you misunderstood what I meant. I did not imply that one would need to annotate any types - only specify the modules for which run-time reflection should be available.

Would that be possible to do for third party types available in a pre-compiled library? -- /Jacob Carlborg
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 21:43, Vladimir Panteleev wrote:
 On Sun, 21 Aug 2011 21:21:05 +0300, Jacob Carlborg <doob me.com> wrote:

 Would that be possible to do for third party types available in a
 pre-compiled library?

If said types are in the .di files, sure, why not? Otherwise, the answer is obvious... but as someone else mentioned in this thread, if the authors of said libraries choose to hide the source code or internals of the library, they probably won't be very friendly towards reflection either.

To be able to use a library you need parts of the source code, like the methods and fields available. And when that's already available what's wrong with some compiler/runtime changes to make it possible to use these through base class references. -- /Jacob Carlborg
Aug 21 2011
prev sibling parent reply Kagamin <spam here.lot> writes:
Jacob Carlborg Wrote:

 This is for a serialization library where third party types need to be 
 serializable. I don't like the idea of annotate everything that should 
 be serializable.

I believe Adam Ruppe wrote serialization that works automatically.
Aug 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 13:37, Kagamin wrote:
 Jacob Carlborg Wrote:

 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.

I believe Adam Ruppe wrote serialization that works automatically.

My library works automatically for all types, except when serializing through a base class reference. Then the subclass have to be registered with the serializer. -- /Jacob Carlborg
Aug 22 2011
prev sibling next sibling parent Robert Clipsham <robert octarineparrot.com> writes:
On 21/08/2011 14:13, Vladimir Panteleev wrote:
 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:

 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.

I think the best way to do this is to improve compile-time reflection to the point where it'd be possible to enumerate all declarations in a module and create run-time reflection information for them - then you'd just need to specify the modules for which you'd like run-time reflection. One thing to note is that there are use cases where more run-time information about a program's internals may not be desired, e.g. with commercial closed-source applications, so this would need to be optional.

I completely agree. I'd love to get rid of classinfo while we're at it to be honest, that's a heck of lot of bloat that isn't needed/can be done in a far nicer way. -- Robert http://octarineparrot.com/
Aug 21 2011
prev sibling parent reply Jakob Ovrum <jakobovrum+ng gmail.com> writes:
On 2011/08/21 22:13, Vladimir Panteleev wrote:
 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:

 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.

I think the best way to do this is to improve compile-time reflection to the point where it'd be possible to enumerate all declarations in a module and create run-time reflection information for them - then you'd just need to specify the modules for which you'd like run-time reflection. One thing to note is that there are use cases where more run-time information about a program's internals may not be desired, e.g. with commercial closed-source applications, so this would need to be optional.

I definitively think this is the way to go forward. The only situation this method doesn't handle is the case of only having D interface files and binaries for a library you want to serialize. However, if the library is closed-source, one of the reasons for that might be that they don't want this kind of information to be available, and if that's the case and D would generate this runtime information by default, we'd start seeing obfuscators for D as which break the reflection functionality. As for the other way around, where a closed-source library wanted to retain runtime reflection capabilties, they could always include the information by explicitly instantiating the runtime information template or however it would be implemented. Besides, compile-time reflection for entire modules has many uses beyond implementing runtime reflection.
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 18:18, Jakob Ovrum wrote:
 On 2011/08/21 22:13, Vladimir Panteleev wrote:
 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:

 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.

I think the best way to do this is to improve compile-time reflection to the point where it'd be possible to enumerate all declarations in a module and create run-time reflection information for them - then you'd just need to specify the modules for which you'd like run-time reflection. One thing to note is that there are use cases where more run-time information about a program's internals may not be desired, e.g. with commercial closed-source applications, so this would need to be optional.

I definitively think this is the way to go forward. The only situation this method doesn't handle is the case of only having D interface files and binaries for a library you want to serialize. However, if the library is closed-source, one of the reasons for that might be that they don't want this kind of information to be available, and if that's the case and D would generate this runtime information by default, we'd start seeing obfuscators for D as which break the reflection functionality.

The information is already available via .tupleof. This would make it available through base class references and with runtime strings.
 As for the other way around, where a closed-source library wanted to
 retain runtime reflection capabilties, they could always include the
 information by explicitly instantiating the runtime information template
 or however it would be implemented.

 Besides, compile-time reflection for entire modules has many uses beyond
 implementing runtime reflection.

-- /Jacob Carlborg
Aug 21 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:

 Yes, I know. But the point is to be able to do it without the use of  
 mixins. It needs to work with third party types, otherwise there no use.  
 It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains  
 getMembers. But this methods always returns an empty array and as far as  
 I can see, it's not possible to set/get fields or call methods via these  
 member instances.

I think the best way to do this is to improve compile-time reflection to the point where it'd be possible to enumerate all declarations in a module and create run-time reflection information for them - then you'd just need to specify the modules for which you'd like run-time reflection. One thing to note is that there are use cases where more run-time information about a program's internals may not be desired, e.g. with commercial closed-source applications, so this would need to be optional. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Aug 21 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 21 Aug 2011 16:31:14 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:

 I really don't think this is a valid cause of concern in this day and  
 age.

Perhaps not for you.
 Look at Java and .NET. They're doing just fine even with all metadata  
 being available.

And that's why obfuscators exist.
 Plenty of commercial C++ products ship with RTTI these days, too (AAA  
 games, even).

Some people care about this, some don't. Don't forget that many of these AAA games you speak of come bundled with invasive security software, with rootkit/spyware traits. Wouldn't you say that "security through obscurity" is the lesser of two evils?
 What I mean to say is, I don't think we should let this "concern" hamper  
 runtime reflection in D, because it evidently is not as big of an issue  
 as people have made it out to be in the past. Security through obscurity  
 is bad, as the saying goes.

Do you speak for everyone, then? I certainly don't want to give any extra help to crackers, cheaters and griefers. It's stupid to ignore such things, and doing so does a disservice to your users. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Aug 21 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 21 Aug 2011 18:05:20 +0300, Jacob Carlborg <doob me.com> wrote:

 This is for a serialization library where third party types need to be  
 serializable. I don't like the idea of annotate everything that should  
 be serializable.

I think you misunderstood what I meant. I did not imply that one would need to annotate any types - only specify the modules for which run-time reflection should be available. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Aug 21 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 21 Aug 2011 21:21:05 +0300, Jacob Carlborg <doob me.com> wrote:

 Would that be possible to do for third party types available in a  
 pre-compiled library?

If said types are in the .di files, sure, why not? Otherwise, the answer is obvious... but as someone else mentioned in this thread, if the authors of said libraries choose to hide the source code or internals of the library, they probably won't be very friendly towards reflection either. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Aug 21 2011
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Tue, 23 Aug 2011 17:03:12 +0200, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-23 16:38, Andrei Alexandrescu wrote:
 On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei

You don't want to have it in the class and don't want it in the serializer. I mean, it needs to be stored somewhere and I thought that a static method in Serializer would better than a completely global function. Are you thinking about having another serialization library that can use this information as well? I'm not sure if that's good idea, different serialization implementations might need to do very different things with the information.

I think the idea is to register something that provides the needed information at runtime and separate that from the serializer because it's useful on it's own. Could be as simple as: interface Serializable { string[] allMembers(); Member getMember(string name); } interface Member { final void set(T)(T t) { setRaw(typeid(t), cast(void*)&t); } void setRaw(TypeInfo ti, void* p); final T get(T)() { return *cast(T*)enforce(getRaw(typeid(T))); } void* getRaw(); } It's clearly interesting to make this available if you implement something along that line. But I would not overstress it for there are dozen of different requirements profiles (performance, versioning, pre/post-serialize hooks). So even if I am interested to use something similar for undo/redo or delta creation, I'm not sure one can find a one-for-all solution here (i.e. the above approach seems way to dynamic for my needs). Maybe Variant could help in setting/getting values. martin
Aug 23 2011
prev sibling next sibling parent "Marco Leise" <Marco.Leise gmx.de> writes:
Am 23.08.2011, 19:42 Uhr, schrieb jdrewsen <jdrewsen nospam.com>:

 Den 23-08-2011 17:03, Jacob Carlborg skrev:
 On 2011-08-23 16:38, Andrei Alexandrescu wrote:
 On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei

You don't want to have it in the class and don't want it in the serializer. I mean, it needs to be stored somewhere and I thought that a static method in Serializer would better than a completely global function. Are you thinking about having another serialization library that can use this information as well? I'm not sure if that's good idea, different serialization implementations might need to do very different things with the information.

It could be used for network transmissions. Correct me if I'm wrong but Orange serializes the entire object. When sending things over the network or when saving something to disk for that matter you most likely are only interested in serializing some of the fields of an object. I really think it would be nice to declaratively mark fields as serializable for certain purposes e.g.: class Foo { int a; // Send over network and saved to file int b; // Saved to file ubyte[] cache; // not to be serialized } mixin(serialize!(Network, Foo, a); mixin(serialize!(File, Foo, a); mixin(serialize!(File, Foo, b); // but still support not specifying each field class Bar { int a; } mixin(serialize!(File, Bar)); Another way would be to just declare a class as serializable and then for each serialization type (ie. Network, File) declare that they should skip a field. Actually I think I better like this approach since it would allow decoupling of serialization type and the declaration of serializability. /Jonas

These are good points Jonas. It's a good idea to read up on http://en.wikipedia.org/wiki/Serialization I used Java for a while and I they have their reasons to require a 'tag'/'annotation'/'interface' on serializable classes. Java's choice to not make serializable a default is explained in short like this: - structures like threads or files cannot maintain their semantics when serialized - classes and libraries change over time, this adds extra work for the maintanance of serializable classes - serialization means externalization of private attributes, even passwords .NET adds the option to mark newly added fields as OptionalField, so a previously serialized instance can be loaded without error. NonSerialized marks a field as not serializable which is what Jonas mentioned. Like Java they work with interfaces to declare methods called after deserialization, so the 'cache' in the above example can be reinitialized correctly.
Aug 23 2011
prev sibling next sibling parent "Marco Leise" <Marco.Leise gmx.de> writes:
Am 24.08.2011, 08:58 Uhr, schrieb Jacob Carlborg <doob me.com>:

 On 2011-08-24 08:41, Marco Leise wrote:
 Am 23.08.2011, 19:42 Uhr, schrieb jdrewsen <jdrewsen nospam.com>:

 Den 23-08-2011 17:03, Jacob Carlborg skrev:
 On 2011-08-23 16:38, Andrei Alexandrescu wrote:
 On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei

You don't want to have it in the class and don't want it in the serializer. I mean, it needs to be stored somewhere and I thought that a static method in Serializer would better than a completely global function. Are you thinking about having another serialization library that can use this information as well? I'm not sure if that's good idea, different serialization implementations might need to do very different things with the information.

It could be used for network transmissions. Correct me if I'm wrong but Orange serializes the entire object. When sending things over the network or when saving something to disk for that matter you most likely are only interested in serializing some of the fields of an object. I really think it would be nice to declaratively mark fields as serializable for certain purposes e.g.: class Foo { int a; // Send over network and saved to file int b; // Saved to file ubyte[] cache; // not to be serialized } mixin(serialize!(Network, Foo, a); mixin(serialize!(File, Foo, a); mixin(serialize!(File, Foo, b); // but still support not specifying each field class Bar { int a; } mixin(serialize!(File, Bar)); Another way would be to just declare a class as serializable and then for each serialization type (ie. Network, File) declare that they should skip a field. Actually I think I better like this approach since it would allow decoupling of serialization type and the declaration of serializability. /Jonas

These are good points Jonas. It's a good idea to read up on http://en.wikipedia.org/wiki/Serialization I used Java for a while and I they have their reasons to require a 'tag'/'annotation'/'interface' on serializable classes. Java's choice to not make serializable a default is explained in short like this: - structures like threads or files cannot maintain their semantics when serialized - classes and libraries change over time, this adds extra work for the maintanance of serializable classes - serialization means externalization of private attributes, even passwords .NET adds the option to mark newly added fields as OptionalField, so a previously serialized instance can be loaded without error. NonSerialized marks a field as not serializable which is what Jonas mentioned. Like Java they work with interfaces to declare methods called after deserialization, so the 'cache' in the above example can be reinitialized correctly.

Orange supports NonSerialized fields and classes, (de)serialization events and several other ways to customize the serialization process. It's inspired by how serialization in .NET works. For more information see the Orange unit tests: https://github.com/jacob-carlborg/orange/tree/master/tests

Then I'll clearly use Orange when I have a need for serialization. Nice work! I've taken a closer look at http://dsource.org/projects/orange/wiki/Features and I see that versioning is still missing. What are your plans there? What happens at the moment when I deserialize and object with missing/added fields?
Aug 24 2011
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Wed, 24 Aug 2011 08:49:19 +0200, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-24 01:34, Martin Nowak wrote:
 On Tue, 23 Aug 2011 17:03:12 +0200, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-23 16:38, Andrei Alexandrescu wrote:
 On 8/23/11 12:55 AM, Jacob Carlborg wrote:
 On 2011-08-23 08:52, Andrei Alexandrescu wrote:
 On 8/22/11 11:30 PM, Jacob Carlborg wrote:
 Ok, then I just change "register" to a static method.

A static method of whom? Andrei

Well, "register" currently an instance method of Serializer so I would change it to be a static method of Serializer.

I think the ability of a class to be serialized would be independent of the notion of a serializer. To me, "this class is serializable" really means "this class has metadata associated with it that allows interested parties to serialize it". But perhaps this is splitting hairs. Andrei

You don't want to have it in the class and don't want it in the serializer. I mean, it needs to be stored somewhere and I thought that a static method in Serializer would better than a completely global function. Are you thinking about having another serialization library that can use this information as well? I'm not sure if that's good idea, different serialization implementations might need to do very different things with the information.

I think the idea is to register something that provides the needed information at runtime and separate that from the serializer because it's useful on it's own. Could be as simple as: interface Serializable { string[] allMembers(); Member getMember(string name); } interface Member { final void set(T)(T t) { setRaw(typeid(t), cast(void*)&t); } void setRaw(TypeInfo ti, void* p); final T get(T)() { return *cast(T*)enforce(getRaw(typeid(T))); } void* getRaw(); } It's clearly interesting to make this available if you implement something along that line. But I would not overstress it for there are dozen of different requirements profiles (performance, versioning, pre/post-serialize hooks). So even if I am interested to use something similar for undo/redo or delta creation, I'm not sure one can find a one-for-all solution here (i.e. the above approach seems way to dynamic for my needs). Maybe Variant could help in setting/getting values. martin

As soon as "setRaw" is called all static information is gone. I need to be able to iterate over T.tupleof.

Not sure, I get what you mean. Let me try to clarify this: static Serializeable[TypeInfo] infos; void register(T)() { infos[typeid(T)] = new SerializableImpl!T(); } // Serializable binds static reflection to dynamic interface class SerializeableImpl!T : Serializable { string[] allMembers() { array(__traits(allMembers, T)); } Member getMember(string name) { if (mi = findMemberInfo(name)) { auto res = mi.clone(); // found immutable info, now create wrapper res.address = this + mi.offset; // tell the clone where to find return res; } return null; } } // Member binds purely static info to dynamic interface and an actual address class MemberInfo!T : Member { override void setRaw(TypeInfo ti, void* p) { if (ti is typeid(T)) *address = *cast(T*)p; } override void* getRaw(TypeInfo ti) { if (ti is typeid(T)) return address; } T* address; } In object.d there is a lot of this partly implemented. Have a closer look at OffsetTypeInfo and MemberInfo_field. You can definitely get all the information the compiler could provide for RTTI at compile time. If you try hard enough you may even be able to store them at compile time. The only difference remaining is that you could not access it through instance.classinfo but have to use a table lookup. It would be great if there was a way to append data to classinfo at compile time. Like an extra pointer or an array of ubyte that can be set. martin
Aug 24 2011
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
It is actually already possible to store meta data in classinfos.

---
import std.stdio;

class A {
   int a;
   float b;
}

mixin template register(T) if(is(T == class)) {
   static this() {
     T.classinfo.m_offTi = [OffsetTypeInfo(0, typeid(int)),  
OffsetTypeInfo(4, typeid(float))];
   }
}

mixin register!A;

void main() {
   foreach(offTi; A.classinfo.offTi())
     writeln(offTi.offset, offTi.ti);
}
---

The compiler doesn't initialize the m_offTi array. But classinfos are in  
the initialized data section, at least for me, so you can write to them.

martin
Aug 24 2011
prev sibling next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 21 Aug 2011 06:55:52 -0400, Jacob Carlborg <doob me.com> wrote:

 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
      private int a;

      private void bar (int i)
      {
          a = i;
      }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.

I've basically done this entirely in library code: see https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht
Aug 21 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 15:20, Robert Jacques wrote:
 On Sun, 21 Aug 2011 06:55:52 -0400, Jacob Carlborg <doob me.com> wrote:

 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.

I've basically done this entirely in library code: see https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht

The whole point was to have direct support in the language/runtime so these library solution were not needed. Does your solution work when the static type is Object and the runtime type is a subclass? -- /Jacob Carlborg
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 02:43, Robert Jacques wrote:
 On Sun, 21 Aug 2011 11:20:14 -0400, Jacob Carlborg <doob me.com> wrote:
 On 2011-08-21 15:20, Robert Jacques wrote:
 On Sun, 21 Aug 2011 06:55:52 -0400, Jacob Carlborg <doob me.com> wrote:

 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language,
 only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.

I've basically done this entirely in library code: see https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht

The whole point was to have direct support in the language/runtime so these library solution were not needed. Does your solution work when the static type is Object and the runtime type is a subclass?

Yes, so long as at some point the subclass is registered by variant, either manually or auto-magically via reflection on another type. The docs had an example of this: module example; class Grade { real mark; } class Student { Grade grade; } void main(string[] args) { Variant.__register!Student; Variant grade = Object.factory("example.Grade"); grade.mark(96.6); assert(grade.mark == 96.6); } notice how Student had to be manually registered, but Grade was registered auto-magically.

I already have this in my serialization library, I want to avoid that. That was the whole point of my original post. -- /Jacob Carlborg
Aug 21 2011
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 21 Aug 2011 11:20:14 -0400, Jacob Carlborg <doob me.com> wrote:
 On 2011-08-21 15:20, Robert Jacques wrote:
 On Sun, 21 Aug 2011 06:55:52 -0400, Jacob Carlborg <doob me.com> wrote:

 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.

I've basically done this entirely in library code: see https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht

The whole point was to have direct support in the language/runtime so these library solution were not needed. Does your solution work when the static type is Object and the runtime type is a subclass?

Yes, so long as at some point the subclass is registered by variant, either manually or auto-magically via reflection on another type. The docs had an example of this: module example; class Grade { real mark; } class Student { Grade grade; } void main(string[] args) { Variant.__register!Student; Variant grade = Object.factory("example.Grade"); grade.mark(96.6); assert(grade.mark == 96.6); } notice how Student had to be manually registered, but Grade was registered auto-magically.
Aug 21 2011
prev sibling parent reply Mehrdad <wfunction hotmail.com> writes:
On 8/21/2011 3:55 AM, Jacob Carlborg wrote:
 What are the chances of D getting proper runtime reflection? Something 
 like this:

 class Foo
 {
     private int a;

     private void bar (int i)
     {
         a = i;
     }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should 
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, 
 only the compiler and runtime would need to be modified. To me, it 
 doesn't look that complicated to implement, the runtime just needs to 
 store an associative arrays that maps names of instance variables and 
 methods to their addresses. This could probably be stored in the 
 ClassInfo. Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.

It's not /quite/ the same thing as you're asking, but I'm facing a problem that seems to fundamentally prevent dynamic typing to work correctly (which is related, but of course not the same thing as, reflection): http://stackoverflow.com/questions/7102648/using-dynamic-typing-in-d-a-statically-typed-language As much as I'd love to see it happen, I can't see how templates and reflection would actually work at run-time.
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 21:07, Mehrdad wrote:
 On 8/21/2011 3:55 AM, Jacob Carlborg wrote:
 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language,
 only the compiler and runtime would need to be modified. To me, it
 doesn't look that complicated to implement, the runtime just needs to
 store an associative arrays that maps names of instance variables and
 methods to their addresses. This could probably be stored in the
 ClassInfo. Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.

It's not /quite/ the same thing as you're asking, but I'm facing a problem that seems to fundamentally prevent dynamic typing to work correctly (which is related, but of course not the same thing as, reflection): http://stackoverflow.com/questions/7102648/using-dynamic-typing-in-d-a-statically-typed-language As much as I'd love to see it happen, I can't see how templates and reflection would actually work at run-time.

Why don't you just use a variant ? -- /Jacob Carlborg
Aug 21 2011