www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Cerealed v0.6.1: even less boilerplate for binary serialization

reply "Atila Neves" <atila.neves gmail.com> writes:
http://code.dlang.org/packages/cerealed

What's new?

* Performance improvements
* New UDAs for networking packets for even less required 
boilerplate

The first new thing is self-explanatory. The second one is 
explained briefly in this blog post:

https://www.reddit.com/r/programming/comments/3flnlt/cerealed_a_d_library_for_declarative_binary/

(also on HN but you know how that goes)

The summary is you can now write this:

     struct UdpPacket {
         static struct Header {
             ushort srcPort;
             ushort dstPort;
             ushort length;
             ushort checksum;
         }

         enum headerSize = unalignedSizeof!Header;
         alias header this;

         Header header;
          LengthInBytes("length - headerSize") ubyte[] data;
     }

Code? Who needs code when the compiler can write it for you?

Atila
Aug 03 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
 The summary is you can now write this:

     struct UdpPacket {
         static struct Header {
             ushort srcPort;
             ushort dstPort;
             ushort length;
             ushort checksum;
         }

         enum headerSize = unalignedSizeof!Header;
         alias header this;

         Header header;
          LengthInBytes("length - headerSize") ubyte[] data;
     }
This deserialization will be identical to casting like this, right? (Not trying to diminish your work, just making sure I get semantics :)) align(1) struct UdpPacket { align(1) static struct Header { ushort srcPort; ushort dstPort; ushort length; ushort checksum; } Header header; ubyte[0] data; } // ... auto packet = cast(UdpPacket*) raw_data.ptr;
Aug 03 2015
next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 3 August 2015 at 09:27:03 UTC, Dicebot wrote:
 On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
 The summary is you can now write this:

     struct UdpPacket {
         static struct Header {
             ushort srcPort;
             ushort dstPort;
             ushort length;
             ushort checksum;
         }

         enum headerSize = unalignedSizeof!Header;
         alias header this;

         Header header;
          LengthInBytes("length - headerSize") ubyte[] data;
     }
This deserialization will be identical to casting like this, right? (Not trying to diminish your work, just making sure I get semantics :)) align(1) struct UdpPacket { align(1) static struct Header { ushort srcPort; ushort dstPort; ushort length; ushort checksum; } Header header; ubyte[0] data; } // ... auto packet = cast(UdpPacket*) raw_data.ptr;
In this case, yes. In the real-life case I was simplifying, it wasn't a ubyte[] array, it was an array of structs with non-trivial serialisation that also depended on a previous deserialised variable. It was more like this: struct Outer { static struct Header { ... } Header header; LengthInBytes("length - headerSize") Inner[] array; } struct Inner { static struct Header { ... } Header header; ArrayLength("length") Unit[] units; //actual length of the array instead of in bytes } struct Unit { ... } So maybe not as useless after all ;) Atila
Aug 03 2015
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 3 August 2015 at 09:40:59 UTC, Atila Neves wrote:
 In this case, yes. In the real-life case I was simplifying, it 
 wasn't a ubyte[] array, it was an array of structs with 
 non-trivial serialisation that also depended on a previous 
 deserialised variable. It was more like this:

 struct Outer {
    static struct Header { ... }
    Header header;
     LengthInBytes("length - headerSize") Inner[] array;
 }

 struct Inner {
     static struct Header { ... }
     Header header;
      ArrayLength("length") Unit[] units; //actual length of the 
 array instead of in bytes
 }

 struct Unit { ... }

 So maybe not as useless after all ;)

 Atila
Thanks. Very nice indeed.
Aug 03 2015
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 03-Aug-2015 12:27, Dicebot wrote:
 On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
 The summary is you can now write this:

     struct UdpPacket {
         static struct Header {
             ushort srcPort;
             ushort dstPort;
             ushort length;
             ushort checksum;
         }

         enum headerSize = unalignedSizeof!Header;
         alias header this;

         Header header;
          LengthInBytes("length - headerSize") ubyte[] data;
     }
This deserialization will be identical to casting like this, right? (Not trying to diminish your work, just making sure I get semantics :)) align(1) struct UdpPacket { align(1) static struct Header { ushort srcPort; ushort dstPort; ushort length; ushort checksum; } Header header; ubyte[0] data; } // ... auto packet = cast(UdpPacket*) raw_data.ptr;
Plus/minus network byte order. -- Dmitry Olshansky
Aug 03 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 3 August 2015 at 10:37:05 UTC, Dmitry Olshansky wrote:
 On 03-Aug-2015 12:27, Dicebot wrote:
 On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
[...]
This deserialization will be identical to casting like this, right? (Not trying to diminish your work, just making sure I get semantics :)) align(1) struct UdpPacket { align(1) static struct Header { ushort srcPort; ushort dstPort; ushort length; ushort checksum; } Header header; ubyte[0] data; } // ... auto packet = cast(UdpPacket*) raw_data.ptr;
Plus/minus network byte order.
Oh yeah, and that. Cerealed always does network order. Atila
Aug 03 2015
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/3/2015 2:21 AM, Atila Neves wrote:
[...]
Please put this as the first comment on the reddit post.
Aug 03 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 3 August 2015 at 11:13:50 UTC, Walter Bright wrote:
 On 8/3/2015 2:21 AM, Atila Neves wrote:
[...]
Please put this as the first comment on the reddit post.
Which one? Atila
Aug 03 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/3/2015 4:46 AM, Atila Neves wrote:
 On Monday, 3 August 2015 at 11:13:50 UTC, Walter Bright wrote:
 On 8/3/2015 2:21 AM, Atila Neves wrote:
 [...]
Please put this as the first comment on the reddit post.
Which one?
The one you started the thread with. On a more meta note, when people look at a discussion on reddit and see zero comments, they move on. Having a comment there saying briefly what the link is about and why the reader might be interested in it is far more effective at getting readers to spend some time reading your article. It also gives you an opportunity to shape the discussion in a direction you'd like. If some troll makes the first post, it sets a trollish tone and can completely ruin all the effort you put into the article. Don't leave things to chance!
Aug 03 2015
prev sibling parent reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
 http://code.dlang.org/packages/cerealed

 What's new?

 * Performance improvements
 * New UDAs for networking packets for even less required 
 boilerplate

 The first new thing is self-explanatory. The second one is 
 explained briefly in this blog post:

 https://www.reddit.com/r/programming/comments/3flnlt/cerealed_a_d_library_for_declarative_binary/

 (also on HN but you know how that goes)

 The summary is you can now write this:

     struct UdpPacket {
         static struct Header {
             ushort srcPort;
             ushort dstPort;
             ushort length;
             ushort checksum;
         }

         enum headerSize = unalignedSizeof!Header;
         alias header this;

         Header header;
          LengthInBytes("length - headerSize") ubyte[] data;
     }

 Code? Who needs code when the compiler can write it for you?

 Atila
1. Are there any convenience functions similar to msgpack's pack() and unpack() providing compactness and elegance as import msgpack; auto x = [1,2]; assert(x.pack.unpack!typeof(x) == x); 2. How does the Cereal performance compare to Msgpack after the recent optimizations?
Aug 03 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 3 August 2015 at 11:43:15 UTC, Per Nordlöw wrote:
 On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
 http://code.dlang.org/packages/cerealed

 What's new?

 * Performance improvements
 * New UDAs for networking packets for even less required 
 boilerplate

 The first new thing is self-explanatory. The second one is 
 explained briefly in this blog post:

 https://www.reddit.com/r/programming/comments/3flnlt/cerealed_a_d_library_for_declarative_binary/

 (also on HN but you know how that goes)

 The summary is you can now write this:

     struct UdpPacket {
         static struct Header {
             ushort srcPort;
             ushort dstPort;
             ushort length;
             ushort checksum;
         }

         enum headerSize = unalignedSizeof!Header;
         alias header this;

         Header header;
          LengthInBytes("length - headerSize") ubyte[] data;
     }

 Code? Who needs code when the compiler can write it for you?

 Atila
1. Are there any convenience functions similar to msgpack's pack() and unpack() providing compactness and elegance as import msgpack; auto x = [1,2]; assert(x.pack.unpack!typeof(x) == x);
Yes. `cerealise` and `decerealise`. The former is slightly weird for performance reasons. It takes a lambda that tells it what to do with the resulting bytes. import cerealed; auto x = MyStruct(); x.cerealise!(bytes => writeln(bytes)); ubyte[] bytes = [...]; auto x = bytes.decerealise!MyStruct;
 2. How does the Cereal performance compare to Msgpack after the 
 recent optimizations?
Close with LDC and DMD, faster with GDC: http://forum.dlang.org/post/nkcelouzpjsgmqtvnonq forum.dlang.org I haven't even tried optimising it myself though, I just merged a contribution from someone else. Atila
Aug 03 2015
parent reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Monday, 3 August 2015 at 11:51:24 UTC, Atila Neves wrote:
 Yes. `cerealise` and `decerealise`. The former is slightly 
 weird for performance reasons. It takes a lambda that tells it 
 what to do with the resulting bytes.

 Close with LDC and DMD, faster with GDC: 
 http://forum.dlang.org/post/nkcelouzpjsgmqtvnonq forum.dlang.org
Nice! Are there any plans to add different backends (for instance msgpack) to Cereal? Then we could have one package to rule them all! I'll try Cereal in favor of msgpack next time! Thanks, Atila!
Aug 03 2015
next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 3 August 2015 at 12:21:05 UTC, Per Nordlöw wrote:
 On Monday, 3 August 2015 at 11:51:24 UTC, Atila Neves wrote:
 Yes. `cerealise` and `decerealise`. The former is slightly 
 weird for performance reasons. It takes a lambda that tells it 
 what to do with the resulting bytes.

 Close with LDC and DMD, faster with GDC: 
 http://forum.dlang.org/post/nkcelouzpjsgmqtvnonq forum.dlang.org
Nice! Are there any plans to add different backends (for instance msgpack) to Cereal? Then we could have one package to rule them all!
I've thought about that (and even JSON or XML) but I'm not 100% convinced that it would work with the existing, and more importantly to me, not high on my list of priorities. I'm definitely not opposed to it.
 I'll try Cereal in favor of msgpack next time!

 Thanks, Atila!
Glad to be useful. :) Atila
Aug 03 2015
prev sibling next sibling parent "TC" <chalucha gmail.com> writes:
On Monday, 3 August 2015 at 12:21:05 UTC, Per Nordlöw wrote:
 On Monday, 3 August 2015 at 11:51:24 UTC, Atila Neves wrote:
 Yes. `cerealise` and `decerealise`. The former is slightly 
 weird for performance reasons. It takes a lambda that tells it 
 what to do with the resulting bytes.

 Close with LDC and DMD, faster with GDC: 
 http://forum.dlang.org/post/nkcelouzpjsgmqtvnonq forum.dlang.org
Nice! Are there any plans to add different backends (for instance msgpack) to Cereal? Then we could have one package to rule them all! I'll try Cereal in favor of msgpack next time! Thanks, Atila!
As I tried the changes on the same test as in [0] It went on my PC like this: DMD debug: Cerealed: 2 secs, 854 ms, 687 μs, and 2 hnsecs MsgPack: 2 secs, 328 ms, 698 μs, and 3 hnsecs DMD release: Cerealed: 1 sec, 619 ms, 570 μs, and 9 hnsecs MsgPack: 1 sec, 554 ms, and 185 μs LDC2 release: Cerealed: 966 ms, 398 μs, and 4 hnsecs MsgPack: 926 ms, 792 μs, and 3 hnsecs GDC release: Cerealed: 1 sec, 97 ms, 892 μs, and 1 hnsec MsgPack: 1 sec, 138 ms, 359 μs, and 2 hnsecs So it's almost the same now. [0] http://forum.dlang.org/post/nkcelouzpjsgmqtvnonq forum.dlang.org
Aug 03 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 03/08/15 14:21, Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> 
wrote:

 Are there any plans to add different backends (for instance msgpack) to
 Cereal? Then we could have one package to rule them all!
That would be Orange [1]. Hopefully I'll get it into Phobos at some point. [1] https://github.com/jacob-carlborg/orange -- /Jacob Carlborg
Aug 03 2015