www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Serialization for D. Comments, please!

reply BCS <none anon.com> writes:
I'm planning on taking a crack at a Serialization template library and I'm 
looking for feed back. My thinking so far is up on my blog here:

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html

Please comment! (here or there, doesn't matter, I think I'll see both)
May 19 2009
next sibling parent reply "Tim Matthews" <tim.matthews7 gmail.com> writes:
On Wed, 20 May 2009 15:43:16 +1200, BCS <none anon.com> wrote:

 I'm planning on taking a crack at a Serialization template library and  
 I'm looking for feed back. My thinking so far is up on my blog here:

 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html

 Please comment! (here or there, doesn't matter, I think I'll see both)

Have you thought about making a OODB so have the full oo programming with data and functions but the objects can be saved and retrieved in a transactional way?
May 19 2009
parent BCS <none anon.com> writes:
Hello Tim,

 On Wed, 20 May 2009 15:43:16 +1200, BCS <none anon.com> wrote:
 
 I'm planning on taking a crack at a Serialization template library
 and  I'm looking for feed back. My thinking so far is up on my blog
 here:
 
 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part
 -1-of-n.html
 
 Please comment! (here or there, doesn't matter, I think I'll see
 both)
 

with data and functions but the objects can be saved and retrieved in a transactional way?

The target usage is to stuff data structures into file or network streams. Eventually, I'd like to build a web services lib from this that can serve proxy an interface with very little code: /////common: interface I { ... } interface J { ... } /////Server side class CI : I { ... } class CJ : J { ... } auto server = new Server(); server.Serve!(I)(new CI(), 8080); //proxy that object on port 8080 server.Serve!(J)({return new CJ;}, 8081); // create a objects to proxy on port 8081 server.start(); //// client side I i = new Proxy!(I)("somehost.net", 8080); J j = new Proxy!(J)("somehost.net", 8081);
May 19 2009
prev sibling next sibling parent BCS <none anon.com> writes:
It turned out that the first pass was easy. No unexpected problems. However 
I noticed a few more things I'm going to have to deal with at some point.

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-2-of-n.html

Again, please comment! (here or there, doesn't matter, I think I'll see both)

p.s. Sorry, no code yet (it's still messy) but fixing that is on the list 
as well.
May 23 2009
prev sibling next sibling parent BCS <none anon.com> writes:
Now with Code!

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-3-of-n-with.html
May 25 2009
prev sibling next sibling parent reply Qian Xu <quian.xu stud.tu-ilmenau.de> writes:
BCS wrote:

 I'm planning on taking a crack at a Serialization template library and I'm
 looking for feed back. My thinking so far is up on my blog here:
 

 
 Please comment! (here or there, doesn't matter, I think I'll see both)

A question: Should every object contain "mixin Serializable!()" in its declaration? It is easy to add this in own classes, but not easy to 3rd-party libraries. for instance: tango --Qian
May 26 2009
parent BCS <none anon.com> writes:
Hello Qian,

 BCS wrote:
 
 I'm planning on taking a crack at a Serialization template library
 and I'm looking for feed back. My thinking so far is up on my blog
 here:
 

1-of-n.html
 Please comment! (here or there, doesn't matter, I think I'll see
 both)
 

Should every object contain "mixin Serializable!()" in its declaration? It is easy to add this in own classes, but not easy to 3rd-party libraries. for instance: tango --Qian

I've been mentally dodging that question because it's going to be ugly. If anyone has some ideas on how to make it work, I'm interested. Right now the system has a hard coded list of basic types (byte, real, etc.) that it can work with and failing that, it assume you have added "mixin Serializable!()". The main problem is getting the third case is overload. Overloading doesn't work across mixins or modules last I checked. It would be a hack, but named function called via string mixins (mixin("Marshal"~ typeof(a).stringof ~ "(sink,a);")) might work.
May 26 2009
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 26 May 2009 13:29:39 +0400, Qian Xu <quian.xu stud.tu-ilmenau.de> wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library and  
 I'm
 looking for feed back. My thinking so far is up on my blog here:

 Please comment! (here or there, doesn't matter, I think I'll see both)

A question: Should every object contain "mixin Serializable!()" in its declaration? It is easy to add this in own classes, but not easy to 3rd-party libraries. for instance: tango --Qian

Good serialization library supports external serialization via template specialization (or similar tricks) :)
May 26 2009
prev sibling next sibling parent aarti_pl <aarti interia.pl> writes:
BCS pisze:
 I'm planning on taking a crack at a Serialization template library and 
 I'm looking for feed back. My thinking so far is up on my blog here:
 
 http://arrayboundserror.blogspot.com/2009/05/serialization-fo
-d-part-1-of-n.html 
 
 
 Please comment! (here or there, doesn't matter, I think I'll see both)
 
 

Hello! I would like to mention that serialization library for D already exists: http://www.dsource.org/projects/doost It supports most of common requirements for serialization library: - easy to use API - most built-in types and user defined types (UDT) are already serializable - out of class serialization - versioning of UDT; importing old versions to new versions - thread/exception safe (well... AFAIK :-) ) - large unit test suite - configurable serialization backend (archive): JSon, Binary, SimpleText - serialization of classes with user defined constructors I don't want to discourage you from implementing your own library, but on the other hand you may consider contributing to this, already existing library. It took me few months to get where it is right now, so it is really a LOT of work; Unfortunately I don't have time to develop it further right now... There are still few things which should be implemented: - serialization from base class (difficult, when we don't want to demand additional coding work from user) - Tango support (should be relatively easy) - XML archive support (foundation framework already there) - serialization for complex types - cleanups, improvements, porting to D 2.0 - better documentation If you would be interested just drop me an e-mail: aarti -at- interia -dot- pl I am very open to suggestions and changes :-) Best Regards Marcin Kuszczak (aarti_pl)
May 26 2009
prev sibling next sibling parent reply Bill Baxter <wbaxter gmail.com> writes:
On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com> wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library and
 I'm
 looking for feed back. My thinking so far is up on my blog here:

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html
 Please comment! (here or there, doesn't matter, I think I'll see both)

A question: Should every object contain "mixin Serializable!()" in its declaration? It is easy to add this in own classes, but not easy to 3rd-party libraries. for instance: tango --Qian

Good serialization library supports external serialization via template specialization (or similar tricks) :)

I'll also add that you should be able to properly serialize/deserialize a BaseClass pointer that actually points to a DerivedClass. This is pretty tricky to get working seamlessly when combined with the external serialization requirement. H3r3tic's xpose library has this working IIRC. --bb
May 26 2009
parent BCS <none anon.com> writes:
Hello Bill,

 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.
 --bb

The current version of the code has this working. It was fairly high on my feature list.
May 26 2009
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com> wrote:

 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>  
 wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu  
 <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library and
 I'm
 looking for feed back. My thinking so far is up on my blog here:

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html
 Please comment! (here or there, doesn't matter, I think I'll see both)

A question: Should every object contain "mixin Serializable!()" in its declaration? It is easy to add this in own classes, but not easy to 3rd-party libraries. for instance: tango --Qian

Good serialization library supports external serialization via template specialization (or similar tricks) :)

I'll also add that you should be able to properly serialize/deserialize a BaseClass pointer that actually points to a DerivedClass. This is pretty tricky to get working seamlessly when combined with the external serialization requirement. H3r3tic's xpose library has this working IIRC. --bb

Sure! How about the following test: struct A { mixin Serializable!(); B[] b; } struct B { mixin Serializable!(); A a; int i; } B[] b = new B[1]; b[0].a.b = b; b[0].i = 42; A* a = &b[0].a; data = serialize(a); // should serialize outer struct (B), too, because it is accessible through a a = deserialize(data); assert(a.b.length == 1); assert(a.b[0].i == 42); BTW, thanks for reminding about xpose. I gotta compare it, doost.serialize and mine own one against each other in terms of efficiency and size (for binary output) someday.
May 26 2009
prev sibling next sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Tue, May 26, 2009 at 12:19 PM, Denis Koroskin <2korden gmail.com> wrote:
 On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com> wrote=

 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>
 wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu
 <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library an=





 I'm
 looking for feed back. My thinking so far is up on my blog here:

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-=




 Please comment! (here or there, doesn't matter, I think I'll see both=





 A question:
 Should every object contain "mixin Serializable!()" in its declaration=




 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian

Good serialization library supports external serialization via template specialization (or similar tricks) :)

I'll also add that you should be able to properly serialize/deserialize a BaseClass pointer that actually points to a DerivedClass. This is pretty tricky to get working seamlessly when combined with the external serialization requirement. H3r3tic's xpose library has this working IIRC. --bb

Sure! How about the following test: struct A { =A0 =A0mixin Serializable!(); =A0 =A0B[] b; } struct B { =A0 =A0mixin Serializable!(); =A0 =A0A a; =A0 =A0int i; } B[] b =3D new B[1]; b[0].a.b =3D b; b[0].i =3D 42; A* a =3D &b[0].a; data =3D serialize(a); // should serialize outer struct (B), too, because=

 a =3D deserialize(data);
 assert(a.b.length =3D=3D 1);
 assert(a.b[0].i =3D=3D 42);

 BTW, thanks for reminding about xpose. I gotta compare it, doost.serializ=

binary output) someday. That's not quite what I meant. I mean this: /// 3RD PARTY LIB class BaseClass { string toString() { return "base"; } } class DerivedClass { string toString() { return "derived"; } } /// MY CODE [...some kind of class registration here....] BaseClass b =3D new DerivedClass; data =3D serialize(b); b2 =3D deserialize(data); assert(b2.toString() =3D=3D "derived"); ------------------ The data stream should contain enough info about the instance to reconstruct the exact derived type. --bb
May 26 2009
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 27 May 2009 01:23:58 +0400, Bill Baxter <wbaxter gmail.com> wrote:

 On Tue, May 26, 2009 at 12:19 PM, Denis Koroskin <2korden gmail.com>  
 wrote:
 On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com>  
 wrote:

 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>
 wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu
 <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library  
 and
 I'm
 looking for feed back. My thinking so far is up on my blog here:

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html
 Please comment! (here or there, doesn't matter, I think I'll see  
 both)

A question: Should every object contain "mixin Serializable!()" in its declaration? It is easy to add this in own classes, but not easy to 3rd-party libraries. for instance: tango --Qian

Good serialization library supports external serialization via template specialization (or similar tricks) :)

I'll also add that you should be able to properly serialize/deserialize a BaseClass pointer that actually points to a DerivedClass. This is pretty tricky to get working seamlessly when combined with the external serialization requirement. H3r3tic's xpose library has this working IIRC. --bb

Sure! How about the following test: struct A { mixin Serializable!(); B[] b; } struct B { mixin Serializable!(); A a; int i; } B[] b = new B[1]; b[0].a.b = b; b[0].i = 42; A* a = &b[0].a; data = serialize(a); // should serialize outer struct (B), too, because it is accessible through a a = deserialize(data); assert(a.b.length == 1); assert(a.b[0].i == 42); BTW, thanks for reminding about xpose. I gotta compare it, doost.serialize and mine own one against each other in terms of efficiency and size (for binary output) someday.

That's not quite what I meant. I mean this: /// 3RD PARTY LIB class BaseClass { string toString() { return "base"; } } class DerivedClass { string toString() { return "derived"; } } /// MY CODE [...some kind of class registration here....] BaseClass b = new DerivedClass; data = serialize(b); b2 = deserialize(data); assert(b2.toString() == "derived"); ------------------ The data stream should contain enough info about the instance to reconstruct the exact derived type. --bb

Yes, I understood and showed yet another feature which is quite hard to implement.
May 26 2009
prev sibling next sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Tue, May 26, 2009 at 2:26 PM, Denis Koroskin <2korden gmail.com> wrote:
 On Wed, 27 May 2009 01:23:58 +0400, Bill Baxter <wbaxter gmail.com> wrote=

 On Tue, May 26, 2009 at 12:19 PM, Denis Koroskin <2korden gmail.com>
 wrote:
 On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com>
 wrote:

 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>
 wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu
 <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library
 and
 I'm
 looking for feed back. My thinking so far is up on my blog here:

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-par=






 Please comment! (here or there, doesn't matter, I think I'll see
 both)

A question: Should every object contain "mixin Serializable!()" in its declaration? It is easy to add this in own classes, but not easy to 3rd-party libraries. for instance: tango --Qian

Good serialization library supports external serialization via templa=





 specialization (or similar tricks) :)

I'll also add that you should be able to properly serialize/deserialize a BaseClass pointer that actually points to a DerivedClass. This is pretty tricky to get working seamlessly when combined with the external serialization requirement. H3r3tic's xpose library has this working IIRC. --bb

Sure! How about the following test: struct A { =A0 mixin Serializable!(); =A0 B[] b; } struct B { =A0 mixin Serializable!(); =A0 A a; =A0 int i; } B[] b =3D new B[1]; b[0].a.b =3D b; b[0].i =3D 42; A* a =3D &b[0].a; data =3D serialize(a); // should serialize outer struct (B), too, becau=



 it is accessible through a

 a =3D deserialize(data);
 assert(a.b.length =3D=3D 1);
 assert(a.b[0].i =3D=3D 42);

 BTW, thanks for reminding about xpose. I gotta compare it,
 doost.serialize and mine own one against each other in terms of efficie=



 and size (for binary output) someday.

That's not quite what I meant. I mean this: /// 3RD PARTY LIB class BaseClass { =A0 =A0 string toString() { return "base"; } } class DerivedClass { =A0 =A0 string toString() { return "derived"; } } /// MY CODE [...some kind of class registration here....] BaseClass b =3D new DerivedClass; data =3D serialize(b); b2 =3D deserialize(data); assert(b2.toString() =3D=3D "derived"); ------------------ The data stream should contain enough info about the instance to reconstruct the exact derived type. --bb

Yes, I understood and showed yet another feature which is quite hard to implement.

Ok. I interpreted that as "Sure! How about the following test (of the feature you just mentioned)". Some segue words like "in addition" or "also" probably would have tipped me off that you were changing the subject. :-) --bb
May 26 2009
prev sibling next sibling parent BCS <none anon.com> writes:
And yet more progress. This time I've got "hard-links" working and I'm thinking 
about how to do 3rd party types.

I don't have any useable ideas I really /like/, but I have one I'm sure I 
can make work. I do have one idea down at the bottom that I'd love to use 
but it would need some new language features.

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-5-of-n.html

As always: Comments, Please! (particularly on the "feature request")
May 31 2009
prev sibling parent reply BCS <none anon.com> writes:
the latest and greatest:

http://arrayboundserror.blogspot.com/2009/06/serialization-for-d-part-6-of-n.html

This time I'm hoping for some feedback on how people want to interface with 
3rd party types.
Jun 11 2009
parent reply grauzone <none example.net> writes:
BCS wrote:
 the latest and greatest:
 
 http://arrayboundserror.blogspot.com/2009/06/serialization-fo
-d-part-6-of-n.html 
 
 
 This time I'm hoping for some feedback on how people want to interface 
 with 3rd party types.
 
 

Is there any real reason for all those mixins?
Jun 11 2009
parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,

 BCS wrote:
 
 the latest and greatest:
 
 http://arrayboundserror.blogspot.com/2009/06/serialization-for-d-part
 -6-of-n.html
 
 This time I'm hoping for some feedback on how people want to
 interface with 3rd party types.
 


which ones?
Jun 11 2009
parent reply grauzone <none example.net> writes:
 Is there any real reason for all those mixins?

which ones?

All used by the user. That would be Serializable and SerializableRecuring.
Jun 11 2009
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,

 Is there any real reason for all those mixins?
 


SerializableRecuring.

What else would you use? I guess if I really wanted to I could use the same device as for 3rd party types, but that just pushes the mixin to global scope (the functions have to be defined in the same module as the type or they won't be able to access private members) and introduces unneeded performance issues from using function pointers. If you can spot a better solution, I'm all for it, but I don't see one and don't think this solution is to bad.
Jun 11 2009
parent reply grauzone <none example.net> writes:
BCS wrote:
 Reply to grauzone,
 
 Is there any real reason for all those mixins?


SerializableRecuring.

What else would you use? I guess if I really wanted to I could use the same device as for 3rd party types, but that just pushes the mixin to global scope (the functions have to be defined in the same module as the type or they won't be able to access private members) and introduces unneeded performance issues from using function pointers. If you can spot a better solution, I'm all for it, but I don't see one and don't think this solution is to bad.

tupleof _can_ access private members, even if the type is from a different module than the function that uses the tupleof. This changed somewhere between dmd 1.030 and 1.040, I think.
Jun 11 2009
parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,

 BCS wrote:
 
 Reply to grauzone,
 
 Is there any real reason for all those mixins?
 


SerializableRecuring.

the same device as for 3rd party types, but that just pushes the mixin to global scope (the functions have to be defined in the same module as the type or they won't be able to access private members) and introduces unneeded performance issues from using function pointers. If you can spot a better solution, I'm all for it, but I don't see one and don't think this solution is to bad.

different module than the function that uses the tupleof. This changed somewhere between dmd 1.030 and 1.040, I think.

If it can, I would consider that a bug. And even if Walter sanctions it, I wont use it because I'm very leery of generating a system that automaticly reaches in from the outside and mucks with private members. Whoever makes a type serializeable needs to put at least some thought into how to serialize it. If it is the type's author, I'm comfortable with them saying "just do everything", but if someone else is doing it, that is likely to get feet shot off. If I made a generic "any type" function, it would recur into any referenced types and either spew a big list of type that it's dealing with (that people won't look at) or give no warning that it's walking into new territory and that it may not be doing valid operations.
Jun 11 2009
parent reply grauzone <none example.net> writes:
 tupleof _can_ access private members, even if the type is from a
 different module than the function that uses the tupleof. This changed
 somewhere between dmd 1.030 and 1.040, I think.

If it can, I would consider that a bug.

It would be nice to get an official statement.
 And even if Walter sanctions it, I wont use it because I'm very leery of 
 generating a system that automaticly reaches in from the outside and 
 mucks with private members. Whoever makes a type serializeable needs to 
 put at least some thought into how to serialize it. If it is the type's 

That's why I'd still require types to be marked as serializeable by the programmer. But using mixins and adding unknown members from the serialization machinery isn't that clean either. At leasr for all those structs, it's truly annoying and unneeded. Having to use different mixins for structs/classes sucks even more (wouldn't be required if you wanted). By the way, how are you going to solve the other problems when serializing D types? Here are some troublesome types: - pointers? - function pointers? - delegates? - unions? - deserializing members that are const/immutable? - referential integrity across arrays and slices? Also, it seems I just spotted a bug: floating point numbers loose precision as they are formatted with "%s".
 author, I'm comfortable with them saying "just do everything", but if 
 someone else is doing it, that is likely to get feet shot off. If I made 
 a generic "any type" function, it would recur into any referenced types 
 and either spew a big list of type that it's dealing with (that people 
 won't look at) or give no warning that it's walking into new territory 
 and that it may not be doing valid operations.

I didn't understand the last sentence.
Jun 13 2009
parent reply BCS <none anon.com> writes:
Hello grauzone,


 And even if Walter sanctions it, I wont use it because I'm very leery
 of generating a system that automaticly reaches in from the outside
 and mucks with private members. Whoever makes a type serializeable
 needs to put at least some thought into how to serialize it. If it is
 the type's
 

the programmer.

How would you do that aside from mixins?
 But using mixins and adding unknown members from the
 serialization machinery isn't that clean either.

I'm not going to fix it now, but I have a few ideas how I can only add the public API to the types. That would be one non static serialization function and a static deserialization function.
 At least for all
 those structs, it's truly annoying and unneeded. Having to use
 different mixins for structs/classes sucks even more (wouldn't be
 required if you wanted).

It isn't required, the difference is not struct vs. class but "do you care about repeated references to instances of this type?"
 
 By the way, how are you going to solve the other problems when
 serializing D types? Here are some troublesome types:
 - pointers?

class references are pointers, for struct and what not, I'll just use a similar approach
 - function pointers?
 - delegates?

I won't
 - unions?

for those I'll provide a way for the user to define the serialization themselves.
 - deserializing members that are const/immutable?

D1, when I get to that; the same way you generated them anywhere else? I might even go with "unsafe" casts as I can be sure to get that correct.
 - referential integrity across arrays and slices?

I won't
 Also, it seems I just spotted a bug: floating point numbers loose
 precision as they are formatted with "%s".

I suppose that should be "%A" I'll get to that at some point.
 
 author, I'm comfortable with them saying "just do everything", but if
 someone else is doing it, that is likely to get feet shot off. If I
 made a generic "any type" function, it would recur into any
 referenced types and either spew a big list of type that it's dealing
 with (that people won't look at) or give no warning that it's walking
 into new territory and that it may not be doing valid operations.
 


First off there is not enough information to correctly generate sterilizers for types. So the user has to do some thinking for most types. How do I make the user thinks about how and for what types code is being generated? Option 1) have the template run a paramg(msg,T.stringof) for each type and hope the user don't just ignore the list. Option 2) do nothing and be sure the user won't see anything. Option 3) Make the user write per type code and error when they don't.
Jun 13 2009
parent reply grauzone <none example.net> writes:
BCS wrote:
 That's why I'd still require types to be marked as serializeable by
 the programmer.

How would you do that aside from mixins?

Make the user implement a marker interface, or let him provide a (single) special class member which fulfill the same function, or introduce annotations into the language. As far as marking goes, a mixin would be OK too, but as I said, I don't like adding arbitrary members into the user's scope without the user knowing.
 At least for all
 those structs, it's truly annoying and unneeded. Having to use
 different mixins for structs/classes sucks even more (wouldn't be
 required if you wanted).

It isn't required, the difference is not struct vs. class but "do you care about repeated references to instances of this type?"

IMHO a relatively useless optimization, that will only confuse the user. It will introduce subtle bugs if objects accidentally are "repeated" instead of serialized only once.
 By the way, how are you going to solve the other problems when
 serializing D types? Here are some troublesome types:
 - pointers?

class references are pointers, for struct and what not, I'll just use a similar approach

That's not really the problem here. You have no idea where a pointer points to. Is it a pointer to a new'ed memory block? Does it point into an object? Does it point into an array? Into the data segment? The GC provides no API to find out. You may be able to handle some cases above, but not all.
 - function pointers?
 - delegates?

I won't

Forces the user to use interfaces instead of delegates.
 - unions?

for those I'll provide a way for the user to define the serialization themselves.
 - deserializing members that are const/immutable?

D1, when I get to that; the same way you generated them anywhere else? I might even go with "unsafe" casts as I can be sure to get that correct.

In D1, it comes for free: you simply don't need to handle it. But yes, it looks like you have to use dirty tricks in D2.
 - referential integrity across arrays and slices?

I won't

I wonder if anyone (I mean, especially user of a serialization library) would disagree with that choice. Sure, there are valid D programs that would break with this, but is relying on this really clean?
 author, I'm comfortable with them saying "just do everything", but if
 someone else is doing it, that is likely to get feet shot off. If I
 made a generic "any type" function, it would recur into any
 referenced types and either spew a big list of type that it's dealing
 with (that people won't look at) or give no warning that it's walking
 into new territory and that it may not be doing valid operations.


First off there is not enough information to correctly generate sterilizers for types. So the user has to do some thinking for most types. How do I make the user thinks about how and for what types code is being generated? Option 1) have the template run a paramg(msg,T.stringof) for each type and hope the user don't just ignore the list. Option 2) do nothing and be sure the user won't see anything. Option 3) Make the user write per type code and error when they don't.

Oh, you mean if there are types in the object graphs the serializer can't deal with it? But how would option 2) work?
Jun 14 2009
parent reply BCS <none anon.com> writes:
Hello grauzone,

 BCS wrote:
 
 That's why I'd still require types to be marked as serializeable by
 the programmer.
 



that would work (for classes) but IMHO the side effects of that are more invasive and it still ends up mucking with internal state from the outside
 or let him provide a
 (single) special class member which fulfill the same function,

What's the difference between one and two?
 or
 introduce annotations into the language.

NO, not an option.
 As far as marking goes, a
 mixin would be OK too, but as I said, I don't like adding arbitrary
 members into the user's scope without the user knowing.
 

how about if what the mixins add is documented?
 At least for all
 those structs, it's truly annoying and unneeded. Having to use
 different mixins for structs/classes sucks even more (wouldn't be
 required if you wanted).

care about repeated references to instances of this type?"

user. It will introduce subtle bugs if objects accidentally are "repeated" instead of serialized only once.

Well, I can switch the default but, in my experience, most of the time repetition doesn't matter. I also dissagree on the "relatively useless optimization" bit, it adds some not exactly trivial overhead in about 3 or 4 different places.
 By the way, how are you going to solve the other problems when
 serializing D types? Here are some troublesome types:
 - pointers?

a similar approach

points to. Is it a pointer to a new'ed memory block? Does it point into an object? Does it point into an array? Into the data segment? The GC provides no API to find out. You may be able to handle some cases above, but not all.

What I can't handle will be documented as unsupported.
 - function pointers?
 - delegates?



interfaces are not supported either.
 - referential integrity across arrays and slices?
 


library) would disagree with that choice. Sure, there are valid D programs that would break with this, but is relying on this really clean?

That's what I'm hoping for. D is to low level for setting up serialization to be completely automatic. The programmer will have to be careful how things are set up. My go is to make that as easy as I can.
 First off there is not enough information to correctly generate
 sterilizers for types. So the user has to do some thinking for most
 types. How do I make the user thinks about how and for what types
 code is being generated?
 
 Option 1) have the template run a paramg(msg,T.stringof) for each
 type
 and hope the user don't just ignore the list.
 Option 2) do nothing and be sure the user won't see anything.
 Option 3) Make the user write per type code and error when they
 don't.

Oh, you mean if there are types in the object graphs the serializer can't deal with it?

Almost, I think. I'm looking at the case where a new types is /added/ to the the object graph at some point removed form the use of the sterilizer. Someone needs to be informed that new types are being serialized and that they need to make sure it's working correctly.
 But how would option 2) work?

It dosn't
Jun 14 2009
parent reply grauzone <none example.net> writes:
BCS wrote:
 introduce annotations into the language.

NO, not an option.

What, why? Sure, this is not a realistic option.
 Well, I can switch the default but, in my experience, most of the time 
 repetition doesn't matter. I also dissagree on the "relatively useless 

Oh really?
 optimization" bit, it adds some not exactly trivial overhead in about 3 
 or 4 different places.

Maybe it costs a hash table lookup, but apart from that, you're saving space and time for marshaling additional instances. Of course, this is different with structs. But structs are value types.
 - function pointers?
 - delegates?



interfaces are not supported either.

But supporting interfaces would be very simple.
Jun 14 2009
parent reply BCS <none anon.com> writes:
Hello grauzone,

 BCS wrote:
 
 introduce annotations into the language.
 



D1 is fixed, and D2 will be in the next few months. I'm not going to even think of targeting D3 at this point. I'm writing this to be used, not as a theoretical construct.
 Well, I can switch the default but, in my experience, most of the
 time repetition doesn't matter. I also dissagree on the "relatively
 useless
 


I haven't used a graph data structure in some time. Most of them have been trees. And the cases I can think of, the repeated reference bit has been central the the structure so the chances of getting it wrong (or of missing it under test) are about nil.
 optimization" bit, it adds some not exactly trivial overhead in about
 3 or 4 different places.
 

space and time for marshaling additional instances. Of course, this is different with structs. But structs are value types.

which side are you arguing there? OTOH pointers to struct are not value types...
 interfaces are not supported either.
 


It wouldn't be hard in the current form (you would add a mixin to the interface as well) but the non-mixin, outside in approach would have all sorts of interesting issues like how to get the correct sterilizer function.
Jun 14 2009
parent reply grauzone <none example.net> writes:
BCS wrote:
 Well, I can switch the default but, in my experience, most of the
 time repetition doesn't matter. I also dissagree on the "relatively
 useless


I haven't used a graph data structure in some time. Most of them have been trees. And the cases I can think of, the repeated reference bit has been central the the structure so the chances of getting it wrong (or of missing it under test) are about nil.

IMO, most tree-like structures are still full graphs in memory, because they often contain "parent" pointers, or point back to a parent indirectly (e.g. even if a generic tree data structure is implemented without parent pointers, the data element itself might contain such pointers).
 OTOH pointers to struct are not value types...

Pointers are a whole different thing. A pointer can still point to a "value" type, because that struct might be embedded within an object (a class member that's a struct).
 interfaces are not supported either.


It wouldn't be hard in the current form (you would add a mixin to the interface as well) but the non-mixin, outside in approach would have all sorts of interesting issues like how to get the correct sterilizer function.

Huh? You can simple cast the interface to an object. And then cast the object to the serializeable type. You need to be able to do that anyway, because object references can be of the type "Object", and there's no way you'd add your serialize mixin to Object. Also, is you writing "sterilizer" a typo or not?
Jun 16 2009
parent reply BCS <none anon.com> writes:
Hello grauzone,

 BCS wrote:
 
 Well, I can switch the default but, in my experience, most of the
 time repetition doesn't matter. I also dissagree on the "relatively
 useless
 


been trees. And the cases I can think of, the repeated reference bit has been central the the structure so the chances of getting it wrong (or of missing it under test) are about nil.

because they often contain "parent" pointers, or point back to a parent indirectly (e.g. even if a generic tree data structure is implemented without parent pointers, the data element itself might contain such pointers).

I'm referring to data structure that I could add serialization to, e.i. ones where I would know of they have parent references. I still stand by my assertion.
 
 OTOH pointers to struct are not value types...
 

"value" type, because that struct might be embedded within an object (a class member that's a struct).

pointers to members won't be supported any time soon.
 interfaces are not supported either.
 


interface as well) but the non-mixin, outside in approach would have all sorts of interesting issues like how to get the correct sterilizer function.


That is not safe. not all interface instances are D objects.
 And then cast the
 object to the serializeable type.

Cast only works if you know /at compile time/ what type to cast to so I don't think that's going to work.
 You need to be able to do that
 anyway, because object references can be of the type "Object", and
 there's no way you'd add your serialize mixin to Object.

And that just brought up another issue: how do you serialize a class that only ever shows up as a base class reference? The lib has no way to /find/ the type at compile time so it has no way to generate code to deal with it.
 
 Also, is you writing "sterilizer" a typo or not?

typo (is it in the lib or just this thread?) I'd be even worse without a spellchecker :(
Jun 16 2009
parent reply grauzone <none example.net> writes:
 Huh? You can simple cast the interface to an object.

That is not safe. not all interface instances are D objects.

There are people who care for COM and C++ interfaces? COM is Windows specific, and C++ vtables are... uh, I don't know, platform/architecture/compiler vendor specific? In any case, serializable objects shouldn't contain references to such interfaces in the first place.
 And that just brought up another issue: how do you serialize a class 
 that only ever shows up as a base class reference? The lib has no way to 
 /find/ the type at compile time so it has no way to generate code to 
 deal with it.

But you already handle this. One of your mixins contains a static this ctor (which, btw., makes it impossible to use serializable types in cyclic dependent modules). It seems right now this ctor is only for registering the demarshaller function, but the same can be done with the marshaller function.
Jun 16 2009
next sibling parent reply BCS <none anon.com> writes:
Hello grauzone,

 Huh? You can simple cast the interface to an object.
 


specific, and C++ vtables are... uh, I don't know, platform/architecture/compiler vendor specific? In any case, serializable objects shouldn't contain references to such interfaces in the first place.

I think there are ways to have a D interface implemented by a non D object.
 And that just brought up another issue: how do you serialize a class
 that only ever shows up as a base class reference? The lib has no way
 to /find/ the type at compile time so it has no way to generate code
 to deal with it.
 

ctor (which, btw., makes it impossible to use serializable types in cyclic dependent modules). It seems right now this ctor is only for registering the demarshaller function, but the same can be done with the marshaller function.

The demarshaller function is indexed via a string derived from the original object. What would the marshaller function key on? The best I can think of right now is the typeinfo and as of now, that's broken under DLLs --- The solution I have now works. Unless someone can show an intractable problem with it I'm keeping it. The only significant issue so far voiced is the concern about what all that mixin is adding and I think I can kill that one with some minor refactoring and documentation. p.s. could you please not delete the citation line from the quote, it makes it easier (at least with good NG clients) for people to find replies to there posts.
Jun 16 2009
parent reply grauzone <none example.net> writes:
BCS wrote:

 The demarshaller function is indexed via a string derived from the 
 original object. What would the marshaller function key on? The best I 
 can think of right now is the typeinfo and as of now, that's broken 
 under DLLs

DLLs are broken in general. There are many more problems associated with them, and you won't be happy with them. I write all my code with the assumption in mind, that TypeInfos/ClassInfos for the same type always are the same instance.
Jun 18 2009
parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,

 BCS wrote:
 
 The demarshaller function is indexed via a string derived from the
 original object. What would the marshaller function key on? The best
 I can think of right now is the typeinfo and as of now, that's broken
 under DLLs
 

with them, and you won't be happy with them. I write all my code with the assumption in mind, that TypeInfos/ClassInfos for the same type always are the same instance.

On second pass, even putting DLLs aside, I can't count on typeinfo being the same in both sides because I can't even count on them being in the same process, exe or even under the same compiler, OS or CPU. Can you get the mangled name of an object instance at runtime via typeinfo?
Jun 18 2009
parent reply grauzone <none example.net> writes:
BCS wrote:
 Reply to grauzone,
 
 BCS wrote:

 The demarshaller function is indexed via a string derived from the
 original object. What would the marshaller function key on? The best
 I can think of right now is the typeinfo and as of now, that's broken
 under DLLs

with them, and you won't be happy with them. I write all my code with the assumption in mind, that TypeInfos/ClassInfos for the same type always are the same instance.

On second pass, even putting DLLs aside, I can't count on typeinfo being the same in both sides because I can't even count on them being in the same process, exe or even under the same compiler, OS or CPU.

For that, you had to write the TypeInfo pointer into the serialized data stream, which obviously is not going to work. Obviously, you need some type identification, which is independent from the compiled program binary. But in-memory, using TypeInfo/ClassInfo as unique identifier is no problem. Except if you somehow got 2 or more D runtimes in your process. That's only possible when using the broken DLL support. Use DDL instead.
 Can you get the mangled name of an object instance at runtime via typeinfo?

Not that I know of. IMHO, ClassInfo.name() is good enough. But if you don't like it, just keep using mangleof. You obviously have compile time access to the serializeable type, e.g.: char[][ClassInfo] TypeMangledNames; template SerializeMixin() { static this() { TypeMangledNames[typeof(this).classinfo] = typeof(this).mangleof; } }
Jun 18 2009
parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,

 Can you get the mangled name of an object instance at runtime via
 typeinfo?
 

But if you don't like it, just keep using mangleof. You obviously have compile time access to the serializeable type, e.g.: char[][ClassInfo] TypeMangledNames; template SerializeMixin() { static this() { TypeMangledNames[typeof(this).classinfo] = typeof(this).mangleof;

 }
 

I'm looking at what it would take to get by without a mixin in the types (someone objected to that) without loosing anything I want. with a mixin in the class, I've already got a solution.
Jun 18 2009
parent grauzone <none example.net> writes:
 I'm looking at what it would take to get by without a mixin in the types 
 (someone objected to that) without loosing anything I want. with a mixin 
 in the class, I've already got a solution.

Though I never questioned that you do need compile time access to the serialized code, and that (at least if you don't want to force the user to implement interfaces) some code knowing the static type must be run at initialization.
Jun 18 2009
prev sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Jun 16, 2009 at 8:50 PM, BCS<none anon.com> wrote:
 Hello grauzone,

 Huh? You can simple cast the interface to an object.


specific, and C++ vtables are... uh, I don't know, platform/architecture/compiler vendor specific? In any case, serializable objects shouldn't contain references to such interfaces in the first place.

I think there are ways to have a D interface implemented by a non D object.

He listed the only two possibilities: COM objects and extern(C++) interfaces in D2. typeid(Interface).classinfo.flags & 1 if an interface is COM at least. I don't know if there is any runtime info to indicate whether an interface is C++ or not.
Jun 16 2009
prev sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
grauzone wrote:
 Is there any real reason for all those mixins?

which ones?

All used by the user. That would be Serializable and SerializableRecuring.

Also, what is curing in this context, and why would you need to do it multiple times?
Jun 11 2009
parent BCS <none anon.com> writes:
Hello Christopher,

 SerializableRecuring.

Also, what is curing in this context, and why would you need to do it multiple times?

My spelling sucks: s/Recuring/Recurring/
Jun 11 2009