www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - (De)Serializing interfaces

reply "nims" <thisisnotaworking emailaddre.ess> writes:
I think interfaces are very powerful and I heavily use them. The 
only problem I have with them is that serializing/deserializing 
them to XML or JSON doesn't seem to work. So far I got to try 
Orange and painlessjson. Using Orange all I got was a lot of 
compiler errors. Painlessjson did compile normally but just 
ignores all interface class members.

This is the code I tried (I apologize for not formatting it, I 
have no idea how to do that):
interface MyInterface {
   int GetA();
}

class Foo: MyInterface {
   int a;
   int GetA() { return a; }
}

// maybe add a class Bar later which implements the same interface

class Foobar {
   MyInterface myBar = new Foo();
}

void main()
{
   // serialize it
}
Aug 22 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/23/2015 7:14 AM, nims wrote:

 I think interfaces are very powerful and I heavily use them. The only
 problem I have with them is that serializing/deserializing them to XML
 or JSON doesn't seem to work. So far I got to try Orange and
 painlessjson. Using Orange all I got was a lot of compiler errors.
 Painlessjson did compile normally but just ignores all interface class
 members.

 This is the code I tried (I apologize for not formatting it, I have no
 idea how to do that):
 interface MyInterface {
    int GetA();
 }

 class Foo: MyInterface {
    int a;
    int GetA() { return a; }
 }

 // maybe add a class Bar later which implements the same interface

 class Foobar {
    MyInterface myBar = new Foo();
 }

 void main()
 {
    // serialize it
 }



So let me put this into that context.


serialized.

Most notably is that ISerialize has a method called GetObjectData which 
populates a data table SerializationInfo with enough information to 
perform deserialization.
There is also a special constructor applied to any serializable class so 
it can manually recreated. This is not possible to be called in D 
unfortunately. You will need an empty constructor a separate method to 
emulate this successfully.

Most importantly with D is knowing type sizes, offsets and of course if 
pointer. If it is a pointer is it an array? Again if so, the sizes and 
if pointer ext. ext.

Now if you want to look at how Java does it[1]. It is very similar to 




Simply put, we generally work with the actual type not an interface. So 
libraries like Orange can serialize/deserialize with great certainty 
that it got everything.

However if you need any help with making such a library, please let me know!

[0] 
https://msdn.microsoft.com/en-us/library/system.runtime.serialization.iserializable(v=vs.110).aspx
[1] http://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html
Aug 22 2015
parent reply "nims" <thisisnotaworking emailaddre.ess> writes:
On Sunday, 23 August 2015 at 03:09:03 UTC, Rikki Cattermole wrote:

 Anyway to summise why D doesn't yet have something akin to Java 

 an interface. So libraries like Orange can 
 serialize/deserialize with great certainty that it got 
 everything.

 However if you need any help with making such a library, please 
 let me know!



your explanation was helpful anyway! Thanks!

As I really need a working serialization library I'll try making 
one myself now. However I'm not very experienced with D in 
general and its reflection in particular so I could use some help.

I think in order to keep it simple I'll just have the user to 
write a function called Serialize() which calls the serializer 
which then just writes the values one after the other in a binary 
file (and, of course, for arrays and strings the length too). As 
all classes implement an interface called Serializable (with the 
function Serialize()), finding the right class to serialize won't 
be too hard (at least I think so). However then we have to make 
sure that we deserialize (instantiate) the right class.

Does the runtime have a function which give you something like a 
unique type id and another one which instantiates the type with 
the given id?

Something like:
int id = _magic_runtime_functions.getUniqueTypeId(typeid(T))
Serializable t = _magic_runtime_functions.createTypeOfId(...)

In order to make this clear I wrote some (untested and 
unfinished) code and pushed it into GitHub: 
https://github.com/nims1/interface-serialization-d/blob/master/Serializer.d
I know it still has a lot of problems (hopefully, D has a better 
way of writing binary files into memory...). It's just a quick 
draft.
Aug 23 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/23/2015 8:15 PM, nims wrote:

 On Sunday, 23 August 2015 at 03:09:03 UTC, Rikki Cattermole wrote:


 Simply put, we generally work with the actual type not an interface.
 So libraries like Orange can serialize/deserialize with great
 certainty that it got everything.

 However if you need any help with making such a library, please let me
 know!



 explanation was helpful anyway! Thanks!

 As I really need a working serialization library I'll try making one
 myself now. However I'm not very experienced with D in general and its
 reflection in particular so I could use some help.

 I think in order to keep it simple I'll just have the user to write a
 function called Serialize() which calls the serializer which then just
 writes the values one after the other in a binary file (and, of course,
 for arrays and strings the length too). As all classes implement an
 interface called Serializable (with the function Serialize()), finding
 the right class to serialize won't be too hard (at least I think so).
 However then we have to make sure that we deserialize (instantiate) the
 right class.


What I was thinking was having a serialize method take an output range 
which you will just pass in a value.


 Does the runtime have a function which give you something like a unique
 type id and another one which instantiates the type with the given id?


A hash? Yeah, TypeInfo_Class should. It will take a pointer.
https://github.com/D-Programming-Language/druntime/blob/master/src/object.d#L261


 Something like:
 int id = _magic_runtime_functions.getUniqueTypeId(typeid(T))


This should work

size_t hash = typeid(avalue).getHash(&avalue);

But keep in mind avalue must be an rvalue.


 Serializable t = _magic_runtime_functions.createTypeOfId(...)

 In order to make this clear I wrote some (untested and unfinished) code
 and pushed it into GitHub:
 https://github.com/nims1/interface-serialization-d/blob/master/Serializer.d
 I know it still has a lot of problems (hopefully, D has a better way of
 writing binary files into memory...). It's just a quick draft.


My suggestion would be evaluate the type "down" aka get immutable(char) 
from immutable(char)[] aka string. So don't try to serialize the string 
straight. Grab the length put that to the output range, then try to 
serialize each of the values of the array individually as another 
function call to itself.

Same sort of deal with other classes, check that they have the interface 
and if so, call it's serialize method with yourself.

A little confusing I must admit.
Also small tip, std.traits is awesome. Especially isBasicType ;)
Aug 23 2015
parent reply "nims" <thisisnotaworking emailaddre.ess> writes:
On Sunday, 23 August 2015 at 08:38:14 UTC, Rikki Cattermole wrote:

 What I was thinking was having a serialize method take an 
 output range which you will just pass in a value.


I'm not really sure what you mean. Replacing the operator by a 
range function or serializing everything automatically?


 A hash? Yeah, TypeInfo_Class should. It will take a pointer.
 https://github.com/D-Programming-Language/druntime/blob/master/src/object.d#L261

 size_t hash = typeid(avalue).getHash(&avalue);

 But keep in mind avalue must be an rvalue.


Sounds good! How do I then create an instance having the same 
type?


 My suggestion would be evaluate the type "down" aka get 
 immutable(char) from immutable(char)[] aka string. So don't try 
 to serialize the string straight. Grab the length put that to 
 the output range, then try to serialize each of the values of 
 the array individually as another function call to itself.

 Same sort of deal with other classes, check that they have the 
 interface and if so, call it's serialize method with yourself.

 A little confusing I must admit.


I'm sorry I didn't get that. Would you have a piece of code for 
illustration?
Aug 23 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/23/2015 10:17 PM, nims wrote:

 On Sunday, 23 August 2015 at 08:38:14 UTC, Rikki Cattermole wrote:

 What I was thinking was having a serialize method take an output range
 which you will just pass in a value.


 I'm not really sure what you mean. Replacing the operator by a range
 function or serializing everything automatically?


Oh not quite. Humm, how to explain this.

import std.traits : isBasicType, isArray;

interface Serializable {
	void serialize(OutputRange!ubyte);
	void deserialize(InputRange!ubyte);

	final void putValue(T)(OutputRange!ubyte output, ref T value) {
		static if (isArray!T) {
			putValue(output, value.length);
			foreach(v; value)
				putValue(v);
		} else static if (isBasicType!T) {
			// convert to ubytes ext. ext. and call put on output.
		} else if (Serializable sv = cast(Serializable)value) {
			sv.serialize(output);
		} else {
			static assert(0, "I don't know how to handle this");
		}
	}

	final T getValue(T)(InputRange!ubyte input) {
		ubyte[T.sizeof] ret;
		foreach(i; 0 .. T.sizeof)
			ret[i] = input.moveFront;
		return to!T(ret);
	}
}

class ... : Serializable {
	int x, y;
	float z;

	void serialize(OutputRange!ubyte output) {
		output.putValue(x);
		output.putValue(y);
		output.putValue(z);
     	}

	void deserialize(InputRange!ubyte input) {
		x = input.getValue!int;
		y = input.getValue!int;
		z = input.getValue!float;
	}
}


 A hash? Yeah, TypeInfo_Class should. It will take a pointer.
 https://github.com/D-Programming-Language/druntime/blob/master/src/object.d#L261


 size_t hash = typeid(avalue).getHash(&avalue);

 But keep in mind avalue must be an rvalue.


 Sounds good! How do I then create an instance having the same type?


The hash value won't help you much really. It's meant for comparing 
instances.


 My suggestion would be evaluate the type "down" aka get
 immutable(char) from immutable(char)[] aka string. So don't try to
 serialize the string straight. Grab the length put that to the output
 range, then try to serialize each of the values of the array
 individually as another function call to itself.

 Same sort of deal with other classes, check that they have the
 interface and if so, call it's serialize method with yourself.

 A little confusing I must admit.


 I'm sorry I didn't get that. Would you have a piece of code for
 illustration?


Take a look at the top code snippet.
You will probably want a couple of free functions be the function you 
call to serialize and deserialize. That way it can control embedding 
e.g. the class name as the first set of values. Which is trivial to use 
with the help of e.g. Object.factory or with the help of TypeInfo.init.

On that note I'll see about getting you a snippet of code that may 
interest you here.
If you can please come on[0] to make it slightly easier for me.

[0] https://gitter.im/rikkimax/chatWithMe
Aug 23 2015
parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Sunday, 23 August 2015 at 10:37:11 UTC, Rikki Cattermole wrote:

 On 8/23/2015 10:17 PM, nims wrote:

 On Sunday, 23 August 2015 at 08:38:14 UTC, Rikki Cattermole 
 wrote:

 What I was thinking was having a serialize method take an 
 output range
 which you will just pass in a value.


 I'm not really sure what you mean. Replacing the operator by a 
 range
 function or serializing everything automatically?


 Oh not quite. Humm, how to explain this.

 import std.traits : isBasicType, isArray;

 interface Serializable {
 	void serialize(OutputRange!ubyte);
 	void deserialize(InputRange!ubyte);

 	final void putValue(T)(OutputRange!ubyte output, ref T value) {
 		static if (isArray!T) {
 			putValue(output, value.length);
 			foreach(v; value)
 				putValue(v);
 		} else static if (isBasicType!T) {
 			// convert to ubytes ext. ext. and call put on output.
 		} else if (Serializable sv = cast(Serializable)value) {
 			sv.serialize(output);
 		} else {
 			static assert(0, "I don't know how to handle this");
 		}
 	}

 	final T getValue(T)(InputRange!ubyte input) {
 		ubyte[T.sizeof] ret;
 		foreach(i; 0 .. T.sizeof)
 			ret[i] = input.moveFront;
 		return to!T(ret);
 	}
 }

 class ... : Serializable {
 	int x, y;
 	float z;

 	void serialize(OutputRange!ubyte output) {
 		output.putValue(x);
 		output.putValue(y);
 		output.putValue(z);
     	}

 	void deserialize(InputRange!ubyte input) {
 		x = input.getValue!int;
 		y = input.getValue!int;
 		z = input.getValue!float;
 	}
 }


 A hash? Yeah, TypeInfo_Class should. It will take a pointer.
 https://github.com/D-Programming-Language/druntime/blob/master/src/object.d#L261


 size_t hash = typeid(avalue).getHash(&avalue);

 But keep in mind avalue must be an rvalue.


 Sounds good! How do I then create an instance having the same 
 type?


 The hash value won't help you much really. It's meant for 
 comparing instances.


 My suggestion would be evaluate the type "down" aka get
 immutable(char) from immutable(char)[] aka string. So don't 
 try to
 serialize the string straight. Grab the length put that to 
 the output
 range, then try to serialize each of the values of the array
 individually as another function call to itself.

 Same sort of deal with other classes, check that they have the
 interface and if so, call it's serialize method with yourself.

 A little confusing I must admit.


 I'm sorry I didn't get that. Would you have a piece of code for
 illustration?


 Take a look at the top code snippet.
 You will probably want a couple of free functions be the 
 function you call to serialize and deserialize. That way it can 
 control embedding e.g. the class name as the first set of 
 values. Which is trivial to use with the help of e.g. 
 Object.factory or with the help of TypeInfo.init.

 On that note I'll see about getting you a snippet of code that 
 may interest you here.
 If you can please come on[0] to make it slightly easier for me.

 [0] https://gitter.im/rikkimax/chatWithMe


Short summary of code snippets:

~~~~~~~~~~~~~~~~~~~
module dnetdev.webserver.common.classfinder;

Interface findAndCreateClass(Interface)(string name) if 
(is(Interface == class) || is(Interface == interface)) {
     import std.experimental.allocator;
     auto alloc = theAllocator();

     auto classinfo = TypeInfo_Class.find(name);
     if (classinfo is null)
         return null;

     size_t issize = classinfo.init.length;
     void[] dataallocated = alloc.allocate(issize);

     Object obj;
     dataallocated[] = cast(void[])classinfo.init[];
     if (dataallocated is null)
         return null;

     obj = cast(Object)dataallocated.ptr;

     if (obj !is null) {
         if (classinfo.defaultConstructor !is null)
             (cast(void 
function(Object))classinfo.defaultConstructor)(obj);

         if (Interface obj2 = cast(Interface)obj) {
             return obj2;
         } else {
             alloc.dispose(obj);
             return null;
         }
     } else {
         alloc.dispose(dataallocated);
         return null;
     }
}
~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~
void main() {
     int x;
     foo!x(x);
}

void foo(alias T)(typeof(T) v) {
     pragma(msg, T.stringof);
}
~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~
import std.stdio;

interface Foo {
     final void func(this T)(/* output */) {
         writeln(T.stringof);
         foreach(v; __traits(allMembers, T))
             writeln(v);
     }
}

class Bar : Foo {
     int x;
}

void main() {
     Bar b = new Bar;
     b.func();
}
~~~~~~~~~~~~~~~~~~~~~
Aug 23 2015
prev sibling next sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Saturday, 22 August 2015 at 19:14:16 UTC, nims wrote:

 I think interfaces are very powerful and I heavily use them. 
 The only problem I have with them is that 
 serializing/deserializing them to XML or JSON doesn't seem to 
 work. So far I got to try Orange and painlessjson. Using Orange 
 all I got was a lot of compiler errors. Painlessjson did 
 compile normally but just ignores all interface class members.


I've never used Orange, but one thing you could try is casting 
your object from MyInterface to Object, and registering the type 
Foobar like in 
http://dsource.org/projects/orange/wiki/Tutorials/SerializeBase, 
then serializing/deserializing it as Object rather than 
MyInterface. I'm not sure if this will work, but it's worth a try 
if it doesn't handle interfaces. Interfaces are a bit odd in some 
ways, as they are not necessarily classes (and thus not 
implicitly convertible to Object) in situations like with COM / 
extern(C++).
Aug 22 2015
parent "nims" <thisisnotaworking emailaddre.ess> writes:
On Sunday, 23 August 2015 at 03:25:27 UTC, Kapps wrote:

 I've never used Orange, but one thing you could try is casting 
 your object from MyInterface to Object, and registering the 
 type Foobar like in 
 http://dsource.org/projects/orange/wiki/Tutorials/SerializeBase, then
serializing/deserializing it as Object rather than MyInterface. I'm not sure if
this will work, but it's worth a try if it doesn't handle interfaces.
Interfaces are a bit odd in some ways, as they are not necessarily classes



 (and thus not implicitly convertible to Object)


I suspect that's what Orange is having trouble with.
/usr/local/include/d/orange/serialization/Serializer.di(254): 
Error: function pointer *serializer (Serializer, const(Object), 
Mode) is not callable using argument types (Serializer, 
MyInterface, Mode)
/usr/local/include/d/orange/serialization/Serializer.di(902): 
Error: template 
orange.serialization.Serializer.Serializer.serializeBaseTypes 
cannot deduce function from argument types !()(MyInterface), 
candidates are:
/usr/local/include/d/orange/serialization/Serializer.di(970):     
    
orange.serialization.Serializer.Serializer.serializeBaseTypes(T : 
Object)(inout T value)
/usr/local/include/d/orange/serialization/Serializer.di(259): 
Error: template instance 
orange.serialization.Serializer.Serializer.objectStructSerialize
elper!(MyInterface) error instantiating
/usr/local/include/d/orange/serialization/Serializer.di(157):     
    instantiated from here: serializeObject!(MyInterface)
/usr/local/include/d/orange/serialization/Serializer.di(386):     
    instantiated from here: serializeInternal!(MyInterface)
/usr/local/include/d/orange/serialization/Serializer.di(892):     
    instantiated from here: serializePointer!(MyInterface*)
/usr/local/include/d/orange/serialization/Serializer.di(259):     
    ... (2 instantiations, -v to show) ...
/usr/local/include/d/orange/serialization/Serializer.di(129):     
    instantiated from here: serializeInternal!(Foobar)
test.d(27):        instantiated from here: serialize!(Foobar)

I also can't cast MyInterface to Foo because at compile time I 
have no idea whether Foo or another type implementing MyInterface 
is behind it. Registering doesn't change that.
Aug 23 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-08-22 21:14, nims wrote:


 Using Orange all I got was a lot of compiler errors.


Seems I completely overlooked interfaces. I'll see if I can add support 
for them.

-- 
/Jacob Carlborg
Aug 23 2015
prev sibling next sibling parent reply "Edwin van Leeuwen" <edder tkwsping.nl> writes:
On Saturday, 22 August 2015 at 19:14:16 UTC, nims wrote:

 I think interfaces are very powerful and I heavily use them. 
 The only problem I have with them is that 
 serializing/deserializing them to XML or JSON doesn't seem to 
 work. So far I got to try Orange and painlessjson. Using Orange 
 all I got was a lot of compiler errors. Painlessjson did 
 compile normally but just ignores all interface class members.


Painlessjson indeed does not support interfaces/subclasses at the 
moment. There was some discussion about it here: 
https://github.com/BlackEdder/painlessjson/issues/8 , but we 
haven't really thought of a good way of doing it yet.

There is also:
http://code.dlang.org/packages/jsonizer
which I think should support at least subclasses, not sure about 
intefaces.
Aug 24 2015
parent "Edwin van Leeuwen" <edder tkwsping.nl> writes:
On Monday, 24 August 2015 at 09:26:40 UTC, Edwin van Leeuwen 
wrote:

 On Saturday, 22 August 2015 at 19:14:16 UTC, nims wrote:

 Painlessjson indeed does not support interfaces/subclasses at 
 the moment. There was some discussion about it here: 
 https://github.com/BlackEdder/painlessjson/issues/8 , but we 
 haven't really thought of a good way of doing it yet.


Pull requests are welcome of course :)
Aug 24 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-08-22 21:14, nims wrote:

 I think interfaces are very powerful and I heavily use them. The only
 problem I have with them is that serializing/deserializing them to XML
 or JSON doesn't seem to work. So far I got to try Orange and
 painlessjson. Using Orange all I got was a lot of compiler errors.


I've added support for interfaces to Orange now.

-- 
/Jacob Carlborg
Aug 28 2015