www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Working on new binary serialization module for phobos (hopefully)

reply "Sean Campbell" <sycam.inc gmail.com> writes:
I've been working on a new serialization module for Phobos and 
its only reliant on 4 Phobos modules
it is available at 
https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

I would like some feedback on it
Jun 01 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-01 14:52, Sean Campbell wrote:

 I've been working on a new serialization module for Phobos and its only
 reliant on 4 Phobos modules
 it is available at
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d


 I would like some feedback on it


I had a quick look at the code. Based on that and how short the code is 
I'm guessing it lacks some features.

I've been working on a serialization package for Phobos for a long time. 
Basically moving Orange [1] in to Phobos.

My serialization library supports:

* Fully automatic serialization
* Serializing non-public fields
* Serializing without registering types (except when serialized through 
a base class reference)
* Serializing third party types
* Custom serialization, both intrusive and no-intrusive
* Tracking of references, pointers and arrays to only serialize the data 
once
* Properly restores arrays and slices
* Separate front end (serializer) from back end (archiver/format)
* Deserializing of classes without default constructor

[1] https://github.com/jacob-carlborg/orange

-- 
/Jacob Carlborg
Jun 01 2015
parent reply "CraigDillabaugh" <craig.dillabaugh gmail.com> writes:
On Monday, 1 June 2015 at 14:45:08 UTC, Jacob Carlborg wrote:

 On 2015-06-01 14:52, Sean Campbell wrote:

 I've been working on a new serialization module for Phobos and 
 its only
 reliant on 4 Phobos modules
 it is available at
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d


 I would like some feedback on it


 I had a quick look at the code. Based on that and how short the 
 code is I'm guessing it lacks some features.

 I've been working on a serialization package for Phobos for a 
 long time. Basically moving Orange [1] in to Phobos.

 My serialization library supports:

 * Fully automatic serialization
 * Serializing non-public fields
 * Serializing without registering types (except when serialized 
 through a base class reference)
 * Serializing third party types
 * Custom serialization, both intrusive and no-intrusive
 * Tracking of references, pointers and arrays to only serialize 
 the data once
 * Properly restores arrays and slices
 * Separate front end (serializer) from back end 
 (archiver/format)
 * Deserializing of classes without default constructor

 [1] https://github.com/jacob-carlborg/orange


I noticed there hasn't been any activity on the Github repo for 8 
months.  Why is that?  Do you consider this a completely finished 
product, or are you held up by the PHobos review process?
Jun 01 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-01 21:22, CraigDillabaugh wrote:


 I noticed there hasn't been any activity on the Github repo for 8
 months.  Why is that?  Do you consider this a completely finished
 product, or are you held up by the PHobos review process?


I just haven't worked on it for a while. Working on the , at least, 
third review process isn't really motivating.

-- 
/Jacob Carlborg
Jun 01 2015
next sibling parent reply "lobo" <swamplobo gmail.com> writes:
On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:

 On 2015-06-01 21:22, CraigDillabaugh wrote:


 I noticed there hasn't been any activity on the Github repo 
 for 8
 months.  Why is that?  Do you consider this a completely 
 finished
 product, or are you held up by the PHobos review process?


 I just haven't worked on it for a while. Working on the , at 
 least, third review process isn't really motivating.


+1 for orange.

I have used cerealed as well and liked it but with it except 
orange it's easy to add and switch between custom archiver 
formats. I couldn't figure out a way to do this with cereraled 
without hacking it.

bye,
lobo
Jun 01 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 2 June 2015 at 05:07:29 UTC, lobo wrote:

 On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:

 On 2015-06-01 21:22, CraigDillabaugh wrote:


 I noticed there hasn't been any activity on the Github repo 
 for 8
 months.  Why is that?  Do you consider this a completely 
 finished
 product, or are you held up by the PHobos review process?


 I just haven't worked on it for a while. Working on the , at 
 least, third review process isn't really motivating.


 +1 for orange.

 I have used cerealed as well and liked it but with it except 
 orange it's easy to add and switch between custom archiver 
 formats. I couldn't figure out a way to do this with cereraled 
 without hacking it.


I only set out to write a binary backend, it didn't occur to me 
to support anything else. I've considered it but nobody seemed to 
ask for it, so...

Atila
Jun 02 2015
prev sibling next sibling parent "Jonas Drewsen" <nospam4321 hotmail.com > writes:
On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:

 On 2015-06-01 21:22, CraigDillabaugh wrote:


 I noticed there hasn't been any activity on the Github repo 
 for 8
 months.  Why is that?  Do you consider this a completely 
 finished
 product, or are you held up by the PHobos review process?


 I just haven't worked on it for a while. Working on the , at 
 least, third review process isn't really motivating.


Couldn't be put into std.experimental. That would probably be 
both motivating and increase testing of the API.

/Jonas
Jun 03 2015
prev sibling parent reply "Sean Campbell" <sycam.inc gmail.com> writes:
On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:

 On 2015-06-01 21:22, CraigDillabaugh wrote:


 I noticed there hasn't been any activity on the Github repo 
 for 8
 months.  Why is that?  Do you consider this a completely 
 finished
 product, or are you held up by the PHobos review process?


 I just haven't worked on it for a while. Working on the , at 
 least, third review process isn't really motivating.


I hope you can get the third review process done because Phobos
needs serialization and it would be much better to use tried
and tested code like orange rather than something I've pulled
together over a couple of weeks

sorry for the double-post
Jun 05 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
 I hope you can get the third review process done because Phobos
 needs serialization and it would be much better to use tried
 and tested code like orange rather than something I've pulled
 together over a couple of weeks


Does phobos really _need_ it? It'd be nice, sure, but given 
there's more than one option available on dub right now I don't 
think it's a massive deal.

Atila
Jun 05 2015
prev sibling next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:

 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it


I'm biased since I wrote this: 
https://github.com/atilaneves/cerealed.

At a glance, I don't like at all that types have to opt-in to be 
serialised. Why the limitation?

You don't need reverseOf, just use std.range.retro.

Cerealed has more features than this as well. I'd struggle to 
write code as short as I did when using it to implement 
networking protocols. I also only encoded bytes as big-endian 
since binary serialization is usually followed by sending those 
bytes over the wire. Given you check the endianess of the system 
here, how would that work?

I liked the union trick, I wonder why I didn't think of that. 
Well, there's the endianness problem I guess.

Atila
Jun 01 2015
next sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Monday, 1 June 2015 at 15:11:54 UTC, Atila Neves wrote:

 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:

 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it


 I'm biased since I wrote this: 
 https://github.com/atilaneves/cerealed.


I've used cerealed a bunch, I would be in favor of it being the 
basis of a phobos library. It's very easy to use.

But I have not used Orange, so I can't comment on how it compares.
Jun 01 2015
prev sibling parent reply "Sean Campbell" <sycam.inc gmail.com> writes:
On Monday, 1 June 2015 at 15:11:54 UTC, Atila Neves wrote:

 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:

 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it


 I'm biased since I wrote this: 
 https://github.com/atilaneves/cerealed.

 At a glance, I don't like at all that types have to opt-in to 
 be serialised. Why the limitation?


I changed this earlier but forgot to push changes from my laptop 
if you check it now it will be fixed

 You don't need reverseOf, just use std.range.retro.


thank you, I have a bad habit of writing already existing 
algorithms

 Cerealed has more features than this as well. I'd struggle to 
 write code as short as I did when using it to implement 
 networking protocols. I also only encoded bytes as big-endian 
 since binary serialization is usually followed by sending those 
 bytes over the wire. Given you check the endianess of the 
 system here, how would that work?


I don't know what you mean bytes by themselves don't have an 
endianness, do they?
I thought is was only anything that was larger than a byte had 
endianness

 I liked the union trick, I wonder why I didn't think of that. 
 Well, there's the endianness problem I guess.

 Atila
Jun 01 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
 Cerealed has more features than this as well. I'd struggle to 
 write code as short as I did when using it to implement 
 networking protocols. I also only encoded bytes as big-endian 
 since binary serialization is usually followed by sending 
 those bytes over the wire. Given you check the endianess of 
 the system here, how would that work?


 I don't know what you mean bytes by themselves don't have an 
 endianness, do they?
 I thought is was only anything that was larger than a byte had 
 endianness


Yes. And sometimes mixed endianess depending on the architecture. 
I might be wrong, but it seems to me that it'd be hard to send 
the bytes over the wire with this.

Atila
Jun 05 2015
parent "Sean Campbell" <sycam.inc gmail.com> writes:
On Friday, 5 June 2015 at 14:55:24 UTC, Atila Neves wrote:

 Cerealed has more features than this as well. I'd struggle to 
 write code as short as I did when using it to implement 
 networking protocols. I also only encoded bytes as big-endian 
 since binary serialization is usually followed by sending 
 those bytes over the wire. Given you check the endianess of 
 the system here, how would that work?


 I don't know what you mean bytes by themselves don't have an 
 endianness, do they?
 I thought is was only anything that was larger than a byte had 
 endianness


 Yes. And sometimes mixed endianess depending on the 
 architecture. I might be wrong, but it seems to me that it'd be 
 hard to send the bytes over the wire with this.

 Atila


if you mean bit numbering then it dosen't matter because its 
managed by the
hardware anyway


but just to be sure i'll do some tests
Jun 05 2015
prev sibling parent reply "Basile Burg" <basile.burg gmx.com> writes:
On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:

 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it


We are many on this segment. Your one is "under-featured". One 
thing i need in serialization is the use of prop getter to 
serialize and prop setter to deserialize (*) and not only a dump 
of the fields that __traits() can find.

That's why i also done mine:

https://github.com/BBasile/iz/blob/master/import/iz/serializer.d

based on the concept of property descriptor, UDA anotations:

https://github.com/BBasile/iz/blob/master/import/iz/properties.d

However still not happy with it...
Maybe with every other examples that people gave you, you'll get 
a better idea of how to design the thing.

One plus is that your code is very clean.
---

(*): an obvious reason is that when you change a piece in an 
engine you don't just drop it under the hood and it's done. You 
have to demount and remount a lot of pieces...that's what the 
setters do silently.
Jun 02 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-02 12:32, Basile Burg wrote:

 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:

 I've been working on a new serialization module for Phobos and its
 only reliant on 4 Phobos modules
 it is available at
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d


 I would like some feedback on it


 We are many on this segment. Your one is "under-featured". One thing i
 need in serialization is the use of prop getter to serialize and prop
 setter to deserialize (*) and not only a dump of the fields that
 __traits() can find.


Orange supports custom serialization [1].


[1] https://github.com/jacob-carlborg/orange/blob/master/tests/Custom.d

-- 
/Jacob Carlborg
Jun 02 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 2 June 2015 at 19:49:39 UTC, Jacob Carlborg wrote:

 On 2015-06-02 12:32, Basile Burg wrote:

 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:

 I've been working on a new serialization module for Phobos 
 and its
 only reliant on 4 Phobos modules
 it is available at
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d


 I would like some feedback on it


 We are many on this segment. Your one is "under-featured". One 
 thing i
 need in serialization is the use of prop getter to serialize 
 and prop
 setter to deserialize (*) and not only a dump of the fields 
 that
 __traits() can find.


 Orange supports custom serialization [1].


 [1] 
 https://github.com/jacob-carlborg/orange/blob/master/tests/Custom.d


So does cerealed, in one of two ways, postBlit being the _much_ 
better one:

https://github.com/atilaneves/cerealed/blob/master/README.md

Atila
Jun 02 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-02 22:02, Atila Neves wrote:


 So does cerealed, in one of two ways, postBlit being the _much_ better one:


I forgot to mention that Orange also supports pre and post 
actions/callbacks [1].

[1] https://github.com/jacob-carlborg/orange/blob/master/tests/Events.d

-- 
/Jacob Carlborg
Jun 03 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 3 June 2015 at 07:00:23 UTC, Jacob Carlborg wrote:

 On 2015-06-02 22:02, Atila Neves wrote:


 So does cerealed, in one of two ways, postBlit being the 
 _much_ better one:


 I forgot to mention that Orange also supports pre and post 
 actions/callbacks [1].

 [1] 
 https://github.com/jacob-carlborg/orange/blob/master/tests/Events.d


 From the unit tests, that doesn't seem to do the same thing, but 
I could be wrong.

Atila
Jun 03 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-03 09:26, Atila Neves wrote:


  From the unit tests, that doesn't seem to do the same thing, but I
 could be wrong.


No, it does not. But it can still be useful. Like if you want to do any 
post processing after deserialization.

-- 
/Jacob Carlborg
Jun 03 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 3 June 2015 at 12:15:43 UTC, Jacob Carlborg wrote:

 On 2015-06-03 09:26, Atila Neves wrote:


 From the unit tests, that doesn't seem to do the same thing, 
 but I
 could be wrong.


 No, it does not. But it can still be useful. Like if you want 
 to do any post processing after deserialization.


Right. In my case, which is totally and completely motivated by 
writing networking protocols, what I really want is to assemble 
packets with the least amount of code possible.

I'm already getting annoyed at having to write the amount of code 
that's necessary now; I'm currently working on improving cerealed 
to be smart enough to know which bytes in the packet correspond 
to length and do even more automatically.

Atila
Jun 03 2015
prev sibling parent reply "Sean Campbell" <sycam.inc gmail.com> writes:
On Tuesday, 2 June 2015 at 10:32:25 UTC, Basile Burg wrote:

 On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:

 I've been working on a new serialization module for Phobos and 
 its only reliant on 4 Phobos modules
 it is available at 
 https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d

 I would like some feedback on it


 We are many on this segment. Your one is "under-featured". One 
 thing i need in serialization is the use of prop getter to 
 serialize and prop setter to deserialize (*) and not only a 
 dump of the fields that __traits() can find.


using property setters and getters doesn't seem like the best idea
as they may either:
they are used to get live data e.g. setter calls hashing function
for an object that isn't part of the class/struct

property may only have setter or getter not both.

but in case you need to store anything that __traits() can find
I added support for custom serialized fields and types.
just annotate a type with  customSerialized and define
the static methods serialize[membername] and 
deserialize[membername]
where [membername] is the name of the field or have full custom
serialization by defining the static methods serializeThis
and deserializeThis.


 That's why i also done mine:

 https://github.com/BBasile/iz/blob/master/import/iz/serializer.d

 based on the concept of property descriptor, UDA anotations:

 https://github.com/BBasile/iz/blob/master/import/iz/properties.d

 However still not happy with it...
 Maybe with every other examples that people gave you, you'll 
 get a better idea of how to design the thing.

 One plus is that your code is very clean.


thank you, it's good to know

 ---

 (*): an obvious reason is that when you change a piece in an 
 engine you don't just drop it under the hood and it's done. You 
 have to demount and remount a lot of pieces...that's what the 
 setters do silently.
Jun 05 2015
parent reply "Basile Burg" <basile.burg gmx.com> writes:
On Friday, 5 June 2015 at 12:21:19 UTC, Sean Campbell wrote:

 On Tuesday, 2 June 2015 at 10:32:25 UTC, Basile Burg wrote:

 [...]


 using property setters and getters doesn't seem like the best 
 idea
 as they may either:
 they are used to get live data e.g. setter calls hashing 
 function
 for an object that isn't part of the class/struct

 property may only have setter or getter not both.

 but in case you need to store anything that __traits() can find
 I added support for custom serialized fields and types.
 just annotate a type with  customSerialized and define
 the static methods serialize[membername] and 
 deserialize[membername]
 where [membername] is the name of the field or have full custom
 serialization by defining the static methods serializeThis
 and deserializeThis.


 [...]


 thank you, it's good to know

 [...]




No, you don't get my point with setters: if a during the 
deserialization you restore, let's say, the _width field and that 
12 children controls rely on this field then they won't be aware 
of the change.
But if the deserializer restores using the width(int value) 
setter, the children can be resized if the the setter contain a 
sub method like updateChildren()...
Jun 05 2015
next sibling parent reply "Sean Campbell" <sycam.inc gmail.com> writes:
On Friday, 5 June 2015 at 12:53:45 UTC, Basile Burg wrote:

 On Friday, 5 June 2015 at 12:21:19 UTC, Sean Campbell wrote:

 On Tuesday, 2 June 2015 at 10:32:25 UTC, Basile Burg wrote:

 [...]


 using property setters and getters doesn't seem like the best 
 idea
 as they may either:
 they are used to get live data e.g. setter calls hashing 
 function
 for an object that isn't part of the class/struct

 property may only have setter or getter not both.

 but in case you need to store anything that __traits() can find
 I added support for custom serialized fields and types.
 just annotate a type with  customSerialized and define
 the static methods serialize[membername] and 
 deserialize[membername]
 where [membername] is the name of the field or have full custom
 serialization by defining the static methods serializeThis
 and deserializeThis.


 [...]


 thank you, it's good to know

 [...]




 No, you don't get my point with setters: if a during the 
 deserialization you restore, let's say, the _width field and 
 that 12 children controls rely on this field then they won't be 
 aware of the change.
 But if the deserializer restores using the width(int value) 
 setter, the children can be resized if the the setter contain a 
 sub method like updateChildren()...


just add  customSerialized to _width and make deserialize_width 
call updateChildren() or whatever
Jun 05 2015
parent "Basile Burg" <basile.burg gmx.com> writes:
On Friday, 5 June 2015 at 13:05:55 UTC, Sean Campbell wrote:

 On Friday, 5 June 2015 at 12:53:45 UTC, Basile Burg wrote:

 On Friday, 5 June 2015 at 12:21:19 UTC, Sean Campbell wrote:

 [...]




 No, you don't get my point with setters: if a during the 
 deserialization you restore, let's say, the _width field and 
 that 12 children controls rely on this field then they won't 
 be aware of the change.
 But if the deserializer restores using the width(int value) 
 setter, the children can be resized if the the setter contain 
 a sub method like updateChildren()...


 just add  customSerialized to _width and make deserialize_width 
 call updateChildren() or whatever


thank you, it's good to know.
Jun 05 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-06-05 14:53, Basile Burg wrote:


 No, you don't get my point with setters: if a during the deserialization
 you restore, let's say, the _width field and that 12 children controls
 rely on this field then they won't be aware of the change.
 But if the deserializer restores using the width(int value) setter, the
 children can be resized if the the setter contain a sub method like
 updateChildren()...


If the whole code is written in D then that's not necessary. The 
children will be restored properly as well. But if you have a D class 
that wraps a native GUI control with a setter that calls some method on 
the native control then it would be necessary to call the setter when 
deserializing.

-- 
/Jacob Carlborg
Jun 06 2015