www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Working on new binary serialization module for phobos (hopefully)

reply "Sean Campbell" <sycam.inc gmail.com> writes:
I've been working on a new serialization module for Phobos and 
its only reliant on 4 Phobos modules
it is available at 
https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

I would like some feedback on it
Jun 01 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-01 14:52, Sean Campbell wrote:
 I've been working on a new serialization module for Phobos and its only
 reliant on 4 Phobos modules
 it is available at
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d


 I would like some feedback on it
I had a quick look at the code. Based on that and how short the code is I'm guessing it lacks some features. I've been working on a serialization package for Phobos for a long time. Basically moving Orange [1] in to Phobos. My serialization library supports: * Fully automatic serialization * Serializing non-public fields * Serializing without registering types (except when serialized through a base class reference) * Serializing third party types * Custom serialization, both intrusive and no-intrusive * Tracking of references, pointers and arrays to only serialize the data once * Properly restores arrays and slices * Separate front end (serializer) from back end (archiver/format) * Deserializing of classes without default constructor [1] https://github.com/jacob-carlborg/orange -- /Jacob Carlborg
Jun 01 2015
parent reply "CraigDillabaugh" <craig.dillabaugh gmail.com> writes:
On Monday, 1 June 2015 at 14:45:08 UTC, Jacob Carlborg wrote:
 On 2015-06-01 14:52, Sean Campbell wrote:
 I've been working on a new serialization module for Phobos and 
 its only
 reliant on 4 Phobos modules
 it is available at
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d


 I would like some feedback on it
I had a quick look at the code. Based on that and how short the code is I'm guessing it lacks some features. I've been working on a serialization package for Phobos for a long time. Basically moving Orange [1] in to Phobos. My serialization library supports: * Fully automatic serialization * Serializing non-public fields * Serializing without registering types (except when serialized through a base class reference) * Serializing third party types * Custom serialization, both intrusive and no-intrusive * Tracking of references, pointers and arrays to only serialize the data once * Properly restores arrays and slices * Separate front end (serializer) from back end (archiver/format) * Deserializing of classes without default constructor [1] https://github.com/jacob-carlborg/orange
I noticed there hasn't been any activity on the Github repo for 8 months. Why is that? Do you consider this a completely finished product, or are you held up by the PHobos review process?
Jun 01 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-01 21:22, CraigDillabaugh wrote:

 I noticed there hasn't been any activity on the Github repo for 8
 months.  Why is that?  Do you consider this a completely finished
 product, or are you held up by the PHobos review process?
I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating. -- /Jacob Carlborg
Jun 01 2015
next sibling parent reply "lobo" <swamplobo gmail.com> writes:
On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:
 On 2015-06-01 21:22, CraigDillabaugh wrote:

 I noticed there hasn't been any activity on the Github repo 
 for 8
 months.  Why is that?  Do you consider this a completely 
 finished
 product, or are you held up by the PHobos review process?
I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating.
+1 for orange. I have used cerealed as well and liked it but with it except orange it's easy to add and switch between custom archiver formats. I couldn't figure out a way to do this with cereraled without hacking it. bye, lobo
Jun 01 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 2 June 2015 at 05:07:29 UTC, lobo wrote:
 On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:
 On 2015-06-01 21:22, CraigDillabaugh wrote:

 I noticed there hasn't been any activity on the Github repo 
 for 8
 months.  Why is that?  Do you consider this a completely 
 finished
 product, or are you held up by the PHobos review process?
I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating.
+1 for orange. I have used cerealed as well and liked it but with it except orange it's easy to add and switch between custom archiver formats. I couldn't figure out a way to do this with cereraled without hacking it.
I only set out to write a binary backend, it didn't occur to me to support anything else. I've considered it but nobody seemed to ask for it, so... Atila
Jun 02 2015
prev sibling next sibling parent "Jonas Drewsen" <nospam4321 hotmail.com > writes:
On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:
 On 2015-06-01 21:22, CraigDillabaugh wrote:

 I noticed there hasn't been any activity on the Github repo 
 for 8
 months.  Why is that?  Do you consider this a completely 
 finished
 product, or are you held up by the PHobos review process?
I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating.
Couldn't be put into std.experimental. That would probably be both motivating and increase testing of the API. /Jonas
Jun 03 2015
prev sibling parent reply "Sean Campbell" <sycam.inc gmail.com> writes:
On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:
 On 2015-06-01 21:22, CraigDillabaugh wrote:

 I noticed there hasn't been any activity on the Github repo 
 for 8
 months.  Why is that?  Do you consider this a completely 
 finished
 product, or are you held up by the PHobos review process?
I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating.
I hope you can get the third review process done because Phobos needs serialization and it would be much better to use tried and tested code like orange rather than something I've pulled together over a couple of weeks sorry for the double-post
Jun 05 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
 I hope you can get the third review process done because Phobos
 needs serialization and it would be much better to use tried
 and tested code like orange rather than something I've pulled
 together over a couple of weeks
Does phobos really _need_ it? It'd be nice, sure, but given there's more than one option available on dub right now I don't think it's a massive deal. Atila
Jun 05 2015
prev sibling next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:
 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it
I'm biased since I wrote this: https://github.com/atilaneves/cerealed. At a glance, I don't like at all that types have to opt-in to be serialised. Why the limitation? You don't need reverseOf, just use std.range.retro. Cerealed has more features than this as well. I'd struggle to write code as short as I did when using it to implement networking protocols. I also only encoded bytes as big-endian since binary serialization is usually followed by sending those bytes over the wire. Given you check the endianess of the system here, how would that work? I liked the union trick, I wonder why I didn't think of that. Well, there's the endianness problem I guess. Atila
Jun 01 2015
next sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Monday, 1 June 2015 at 15:11:54 UTC, Atila Neves wrote:
 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:
 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it
I'm biased since I wrote this: https://github.com/atilaneves/cerealed.
I've used cerealed a bunch, I would be in favor of it being the basis of a phobos library. It's very easy to use. But I have not used Orange, so I can't comment on how it compares.
Jun 01 2015
prev sibling parent reply "Sean Campbell" <sycam.inc gmail.com> writes:
On Monday, 1 June 2015 at 15:11:54 UTC, Atila Neves wrote:
 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:
 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it
I'm biased since I wrote this: https://github.com/atilaneves/cerealed. At a glance, I don't like at all that types have to opt-in to be serialised. Why the limitation?
I changed this earlier but forgot to push changes from my laptop if you check it now it will be fixed
 You don't need reverseOf, just use std.range.retro.
thank you, I have a bad habit of writing already existing algorithms
 Cerealed has more features than this as well. I'd struggle to 
 write code as short as I did when using it to implement 
 networking protocols. I also only encoded bytes as big-endian 
 since binary serialization is usually followed by sending those 
 bytes over the wire. Given you check the endianess of the 
 system here, how would that work?
I don't know what you mean bytes by themselves don't have an endianness, do they? I thought is was only anything that was larger than a byte had endianness
 I liked the union trick, I wonder why I didn't think of that. 
 Well, there's the endianness problem I guess.

 Atila
Jun 01 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
 Cerealed has more features than this as well. I'd struggle to 
 write code as short as I did when using it to implement 
 networking protocols. I also only encoded bytes as big-endian 
 since binary serialization is usually followed by sending 
 those bytes over the wire. Given you check the endianess of 
 the system here, how would that work?
I don't know what you mean bytes by themselves don't have an endianness, do they? I thought is was only anything that was larger than a byte had endianness
Yes. And sometimes mixed endianess depending on the architecture. I might be wrong, but it seems to me that it'd be hard to send the bytes over the wire with this. Atila
Jun 05 2015
parent "Sean Campbell" <sycam.inc gmail.com> writes:
On Friday, 5 June 2015 at 14:55:24 UTC, Atila Neves wrote:
 Cerealed has more features than this as well. I'd struggle to 
 write code as short as I did when using it to implement 
 networking protocols. I also only encoded bytes as big-endian 
 since binary serialization is usually followed by sending 
 those bytes over the wire. Given you check the endianess of 
 the system here, how would that work?
I don't know what you mean bytes by themselves don't have an endianness, do they? I thought is was only anything that was larger than a byte had endianness
Yes. And sometimes mixed endianess depending on the architecture. I might be wrong, but it seems to me that it'd be hard to send the bytes over the wire with this. Atila
if you mean bit numbering then it dosen't matter because its managed by the hardware anyway http://en.wikipedia.org/wiki/Endianness#.22Bit_endianness.22 but just to be sure i'll do some tests
Jun 05 2015
prev sibling parent reply "Basile Burg" <basile.burg gmx.com> writes:
On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:
 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it
We are many on this segment. Your one is "under-featured". One thing i need in serialization is the use of prop getter to serialize and prop setter to deserialize (*) and not only a dump of the fields that __traits() can find. That's why i also done mine: https://github.com/BBasile/iz/blob/master/import/iz/serializer.d based on the concept of property descriptor, UDA anotations: https://github.com/BBasile/iz/blob/master/import/iz/properties.d However still not happy with it... Maybe with every other examples that people gave you, you'll get a better idea of how to design the thing. One plus is that your code is very clean. --- (*): an obvious reason is that when you change a piece in an engine you don't just drop it under the hood and it's done. You have to demount and remount a lot of pieces...that's what the setters do silently.
Jun 02 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-02 12:32, Basile Burg wrote:
 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:
 I've been working on a new serialization module for Phobos and its
 only reliant on 4 Phobos modules
 it is available at
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d


 I would like some feedback on it
We are many on this segment. Your one is "under-featured". One thing i need in serialization is the use of prop getter to serialize and prop setter to deserialize (*) and not only a dump of the fields that __traits() can find.
Orange supports custom serialization [1]. [1] https://github.com/jacob-carlborg/orange/blob/master/tests/Custom.d -- /Jacob Carlborg
Jun 02 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 2 June 2015 at 19:49:39 UTC, Jacob Carlborg wrote:
 On 2015-06-02 12:32, Basile Burg wrote:
 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:
 I've been working on a new serialization module for Phobos 
 and its
 only reliant on 4 Phobos modules
 it is available at
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d


 I would like some feedback on it
We are many on this segment. Your one is "under-featured". One thing i need in serialization is the use of prop getter to serialize and prop setter to deserialize (*) and not only a dump of the fields that __traits() can find.
Orange supports custom serialization [1]. [1] https://github.com/jacob-carlborg/orange/blob/master/tests/Custom.d
So does cerealed, in one of two ways, postBlit being the _much_ better one: https://github.com/atilaneves/cerealed/blob/master/README.md Atila
Jun 02 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-02 22:02, Atila Neves wrote:

 So does cerealed, in one of two ways, postBlit being the _much_ better one:
I forgot to mention that Orange also supports pre and post actions/callbacks [1]. [1] https://github.com/jacob-carlborg/orange/blob/master/tests/Events.d -- /Jacob Carlborg
Jun 03 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 3 June 2015 at 07:00:23 UTC, Jacob Carlborg wrote:
 On 2015-06-02 22:02, Atila Neves wrote:

 So does cerealed, in one of two ways, postBlit being the 
 _much_ better one:
I forgot to mention that Orange also supports pre and post actions/callbacks [1]. [1] https://github.com/jacob-carlborg/orange/blob/master/tests/Events.d
From the unit tests, that doesn't seem to do the same thing, but I could be wrong. Atila
Jun 03 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-03 09:26, Atila Neves wrote:

  From the unit tests, that doesn't seem to do the same thing, but I
 could be wrong.
No, it does not. But it can still be useful. Like if you want to do any post processing after deserialization. -- /Jacob Carlborg
Jun 03 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 3 June 2015 at 12:15:43 UTC, Jacob Carlborg wrote:
 On 2015-06-03 09:26, Atila Neves wrote:

 From the unit tests, that doesn't seem to do the same thing, 
 but I
 could be wrong.
No, it does not. But it can still be useful. Like if you want to do any post processing after deserialization.
Right. In my case, which is totally and completely motivated by writing networking protocols, what I really want is to assemble packets with the least amount of code possible. I'm already getting annoyed at having to write the amount of code that's necessary now; I'm currently working on improving cerealed to be smart enough to know which bytes in the packet correspond to length and do even more automatically. Atila
Jun 03 2015
prev sibling parent reply "Sean Campbell" <sycam.inc gmail.com> writes:
On Tuesday, 2 June 2015 at 10:32:25 UTC, Basile Burg wrote:
 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:
 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it
We are many on this segment. Your one is "under-featured". One thing i need in serialization is the use of prop getter to serialize and prop setter to deserialize (*) and not only a dump of the fields that __traits() can find.
using property setters and getters doesn't seem like the best idea as they may either: they are used to get live data e.g. setter calls hashing function for an object that isn't part of the class/struct property may only have setter or getter not both. but in case you need to store anything that __traits() can find I added support for custom serialized fields and types. just annotate a type with customSerialized and define the static methods serialize[membername] and deserialize[membername] where [membername] is the name of the field or have full custom serialization by defining the static methods serializeThis and deserializeThis.
 That's why i also done mine:

 https://github.com/BBasile/iz/blob/master/import/iz/serializer.d

 based on the concept of property descriptor, UDA anotations:

 https://github.com/BBasile/iz/blob/master/import/iz/properties.d

 However still not happy with it...
 Maybe with every other examples that people gave you, you'll 
 get a better idea of how to design the thing.

 One plus is that your code is very clean.
thank you, it's good to know
 ---

 (*): an obvious reason is that when you change a piece in an 
 engine you don't just drop it under the hood and it's done. You 
 have to demount and remount a lot of pieces...that's what the 
 setters do silently.
Jun 05 2015
parent reply "Basile Burg" <basile.burg gmx.com> writes:
On Friday, 5 June 2015 at 12:21:19 UTC, Sean Campbell wrote:
 On Tuesday, 2 June 2015 at 10:32:25 UTC, Basile Burg wrote:
 [...]
using property setters and getters doesn't seem like the best idea as they may either: they are used to get live data e.g. setter calls hashing function for an object that isn't part of the class/struct property may only have setter or getter not both. but in case you need to store anything that __traits() can find I added support for custom serialized fields and types. just annotate a type with customSerialized and define the static methods serialize[membername] and deserialize[membername] where [membername] is the name of the field or have full custom serialization by defining the static methods serializeThis and deserializeThis.
 [...]
thank you, it's good to know
 [...]
No, you don't get my point with setters: if a during the deserialization you restore, let's say, the _width field and that 12 children controls rely on this field then they won't be aware of the change. But if the deserializer restores using the width(int value) setter, the children can be resized if the the setter contain a sub method like updateChildren()...
Jun 05 2015
next sibling parent reply "Sean Campbell" <sycam.inc gmail.com> writes:
On Friday, 5 June 2015 at 12:53:45 UTC, Basile Burg wrote:
 On Friday, 5 June 2015 at 12:21:19 UTC, Sean Campbell wrote:
 On Tuesday, 2 June 2015 at 10:32:25 UTC, Basile Burg wrote:
 [...]
using property setters and getters doesn't seem like the best idea as they may either: they are used to get live data e.g. setter calls hashing function for an object that isn't part of the class/struct property may only have setter or getter not both. but in case you need to store anything that __traits() can find I added support for custom serialized fields and types. just annotate a type with customSerialized and define the static methods serialize[membername] and deserialize[membername] where [membername] is the name of the field or have full custom serialization by defining the static methods serializeThis and deserializeThis.
 [...]
thank you, it's good to know
 [...]
No, you don't get my point with setters: if a during the deserialization you restore, let's say, the _width field and that 12 children controls rely on this field then they won't be aware of the change. But if the deserializer restores using the width(int value) setter, the children can be resized if the the setter contain a sub method like updateChildren()...
just add customSerialized to _width and make deserialize_width call updateChildren() or whatever
Jun 05 2015
parent "Basile Burg" <basile.burg gmx.com> writes:
On Friday, 5 June 2015 at 13:05:55 UTC, Sean Campbell wrote:
 On Friday, 5 June 2015 at 12:53:45 UTC, Basile Burg wrote:
 On Friday, 5 June 2015 at 12:21:19 UTC, Sean Campbell wrote:
 [...]
No, you don't get my point with setters: if a during the deserialization you restore, let's say, the _width field and that 12 children controls rely on this field then they won't be aware of the change. But if the deserializer restores using the width(int value) setter, the children can be resized if the the setter contain a sub method like updateChildren()...
just add customSerialized to _width and make deserialize_width call updateChildren() or whatever
thank you, it's good to know.
Jun 05 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-06-05 14:53, Basile Burg wrote:

 No, you don't get my point with setters: if a during the deserialization
 you restore, let's say, the _width field and that 12 children controls
 rely on this field then they won't be aware of the change.
 But if the deserializer restores using the width(int value) setter, the
 children can be resized if the the setter contain a sub method like
 updateChildren()...
If the whole code is written in D then that's not necessary. The children will be restored properly as well. But if you have a D class that wraps a native GUI control with a setter that calls some method on the native control then it would be necessary to call the setter when deserializing. -- /Jacob Carlborg
Jun 06 2015