www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Orange 1.0.0 beta - serialization library

reply Jacob Carlborg <doob me.com> writes:
I've almost finished the rewrite of my serialization library Orange. I'm 
hoping that someone wants to give it a try and see what issues/bugs are 
found.

Project page: http://dsource.org/projects/orange
Source code: https://github.com/jacob-carlborg/orange

There are two usage examples on the project page. For more examples I 
recommend looking at the unit tests in the "tests" directory.

Description:

Orange is a serialization library for D1 and D2, supporting both Tango 
and Phobos. It can serialize most of the available types in D, including 
third party types and can serialize through base class references. It 
supports fully automatic serialization of all supported types and also 
supports several ways to customize the serialization process. Orange has 
a separate front end (the serializer) and back end (the archive) making 
it possible for the user to create new archive types that can be used 
with the existing serializer.

Features:

* Automatically serializes the base classes
* Supports events (before and after (de)serializing)

* Supports non-serialized fields and classes (you can say that some 
fields in a class should not be serialized)

* Licensed under the Boost license
* Std/runtime library independent

* Extendable - possible to create new archive types and use them with 
the existing serializer

* Serializes through base class references
* Serializes third party types

* Customization of the (de)serialization process, both intrusive and 
non-intrusive

* Properly (de)serializes slices and pointers

Known Issues/Limitations:

* Due to limitations in the XML module provided by Phobos the XMLArchive 
will only work with "char" as the template type with D2

* Due to several bugs/limitations in the compiler/runtime even the D2 
version requires you to register the type when serializing through base 
class references

* No built-in support for versioning
* Floating point numbers are not serialized as hexadecimal (D1)

-- 
/Jacob Carlborg
Aug 20 2011
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
This looks really great!  I'll try it out sometime later.  A few questions:

1.  What other archiver formats besides XML might be useful?  (I 
remember binary can't work because of the way keys work, though 
messagepack is probably a better option for lightweight serialization in 
general.  Orange's niche is portability and serializing as many types as 
possible.)

2.  What are the prospects for submitting this for inclusion in Phobos? 
  Serialization is something basic and universally needed enough that it 
should not require a third-party library.

3.  Given that all of the necessary introspection already works, what 
are the prospects for creating a Clone archiver?  A Clone archiver would 
not actually serialize anything and instead would deep clone whatever 
data structure it receives.  If Orange gets into Phobos, cloning could 
be recognized by std.concurrency as a safe, simple way to pass complex 
object graphs between threads.  I'm picturing an API something like this:

import std.stdio, whateverOrangeEndsUpBeingNamed.

struct Cloned(T) {
     T obj;
}

Cloned!T cloned(T)(T obj) { return typeof(return)(obj); }

void spawmMe() {
     receive((int[][]) { writeln("Received an int[][]."); });
}

void main() {
     auto tid = spawn(&spawnMe);
     auto mat = new int[][](42, 42);

     // FAILS:  Implicit sharing.
     tid.send(mat);

     // Works.  mat is automatically deep cloned and the clone is sent
     // to tid.  send recognized the Cloned struct as special and
     // processes the payload accordingly.
     tid.send(cloned(mat));
}



On 8/20/2011 11:13 AM, Jacob Carlborg wrote:
 I've almost finished the rewrite of my serialization library Orange. I'm
 hoping that someone wants to give it a try and see what issues/bugs are
 found.

 Project page: http://dsource.org/projects/orange
 Source code: https://github.com/jacob-carlborg/orange

 There are two usage examples on the project page. For more examples I
 recommend looking at the unit tests in the "tests" directory.

 Description:

 Orange is a serialization library for D1 and D2, supporting both Tango
 and Phobos. It can serialize most of the available types in D, including
 third party types and can serialize through base class references. It
 supports fully automatic serialization of all supported types and also
 supports several ways to customize the serialization process. Orange has
 a separate front end (the serializer) and back end (the archive) making
 it possible for the user to create new archive types that can be used
 with the existing serializer.

 Features:

 * Automatically serializes the base classes
 * Supports events (before and after (de)serializing)

 * Supports non-serialized fields and classes (you can say that some
 fields in a class should not be serialized)

 * Licensed under the Boost license
 * Std/runtime library independent

 * Extendable - possible to create new archive types and use them with
 the existing serializer

 * Serializes through base class references
 * Serializes third party types

 * Customization of the (de)serialization process, both intrusive and
 non-intrusive

 * Properly (de)serializes slices and pointers

 Known Issues/Limitations:

 * Due to limitations in the XML module provided by Phobos the XMLArchive
 will only work with "char" as the template type with D2

 * Due to several bugs/limitations in the compiler/runtime even the D2
 version requires you to register the type when serializing through base
 class references

 * No built-in support for versioning
 * Floating point numbers are not serialized as hexadecimal (D1)

Aug 20 2011
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/

Yeah, this does look quite useful. I tried it a few weeks ago and ran into a couple showstopper bugs. I filed them and he fixed them, but I never got around to trying it again. I actually think that since both are very commonly needed things and satisfy completely different niches, Orange and msgpack4d might both eventually have a rightful place in Phobos. Orange would be recommended when portability and serializing as many types as possible are important, and msgpack4d would be recommended when speed and space efficiency are the primary concerns.
 Is RIFF considered a good format? I've ran into it when porting C
 code,  it seems it's also used by Google and probably other companies.
 http://en.wikipedia.org/wiki/Resource_Interchange_File_Format

From reading the Wikipedia article (I knew nothing about RIFF before this discussion) it sounds like it's in a valley in between two utility peaks. XML provides maximum portability. msgpack provides maximum efficiency. RIFF doesn't really provide either all that well.
Aug 20 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-20 19:45, dsimcha wrote:
 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/

Yeah, this does look quite useful. I tried it a few weeks ago and ran into a couple showstopper bugs. I filed them and he fixed them, but I never got around to trying it again. I actually think that since both are very commonly needed things and satisfy completely different niches, Orange and msgpack4d might both eventually have a rightful place in Phobos. Orange would be recommended when portability and serializing as many types as possible are important, and msgpack4d would be recommended when speed and space efficiency are the primary concerns.

If it's possible to extract a value given a key with messagepack I think it would be possible to use messagepack as an archive for Orange. Since messagepack supports maps every type Orange can serialize should be possible serialize with messagepack, although probably not as efficient as "regular" messagepack. -- /Jacob Carlborg
Aug 20 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 23:16, Masahiro Nakagawa wrote:
 On Sun, 21 Aug 2011 05:28:33 +0900, Jacob Carlborg <doob me.com> wrote:
 
 On 2011-08-20 19:45, dsimcha wrote:
 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/

Yeah, this does look quite useful. I tried it a few weeks ago and ran into a couple showstopper bugs. I filed them and he fixed them, but I never got around to trying it again. I actually think that since both are very commonly needed things and satisfy completely different niches, Orange and msgpack4d might both eventually have a rightful place in Phobos. Orange would be recommended when portability and serializing as many types as possible are important, and msgpack4d would be recommended when speed and space efficiency are the primary concerns.

If it's possible to extract a value given a key with messagepack I think it would be possible to use messagepack as an archive for Orange. Since messagepack supports maps every type Orange can serialize should be possible serialize with messagepack, although probably not as efficient as "regular" messagepack.

I agree. Unfortunately, Orange with MessagePack decreases MessagePack merits...

I suspected that. -- /Jacob Carlborg
Aug 21 2011
prev sibling parent simendsjo <simendsjo gmail.com> writes:
On 20.08.2011 19:45, dsimcha wrote:
 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/

Yeah, this does look quite useful. I tried it a few weeks ago and ran into a couple showstopper bugs. I filed them and he fixed them, but I never got around to trying it again. I actually think that since both are very commonly needed things and satisfy completely different niches, Orange and msgpack4d might both eventually have a rightful place in Phobos. Orange would be recommended when portability and serializing as many types as possible are important, and msgpack4d would be recommended when speed and space efficiency are the primary concerns.
 Is RIFF considered a good format? I've ran into it when porting C
 code, it seems it's also used by Google and probably other companies.
 http://en.wikipedia.org/wiki/Resource_Interchange_File_Format

From reading the Wikipedia article (I knew nothing about RIFF before this discussion) it sounds like it's in a valley in between two utility peaks. XML provides maximum portability. msgpack provides maximum efficiency. RIFF doesn't really provide either all that well.

Agreed. I've worked a lot with RIFF in the past, and it only specifies a high level structure by having sections with a known size. It doesn't give much over a plain binary file.
Aug 21 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-20 18:16, dsimcha wrote:
 This looks really great! I'll try it out sometime later. A few questions:

 1. What other archiver formats besides XML might be useful? (I remember
 binary can't work because of the way keys work, though messagepack is
 probably a better option for lightweight serialization in general.
 Orange's niche is portability and serializing as many types as possible.)

I assume any archive type that can unpack a value given a key will work. I assume JSON and YAML will work as well. I no nothing about binary serialization but I think messagepack can work. Another problem might be pointers and slices. For example, at the first step every array is assumed to be pointing to its own memory, i.e. not a slice. After everything is serialized the serializer post process the serialized data and replaces all arrays that points into another array with a slice.
 2. What are the prospects for submitting this for inclusion in Phobos?
 Serialization is something basic and universally needed enough that it
 should not require a third-party library.

From my part: * Remove all Tango/D1 specific code * Add documentation * There are some few things left in the rewrite process * I've made some modifications to the XML module in Phobos, these need to be applied as well * Probably something else I can't think of right now I'm not particular happy about removing the Tango/D1 specific code if I'm not sure that the library can be included in Phobos.
 3. Given that all of the necessary introspection already works, what are
 the prospects for creating a Clone archiver? A Clone archiver would not
 actually serialize anything and instead would deep clone whatever data
 structure it receives. If Orange gets into Phobos, cloning could be
 recognized by std.concurrency as a safe, simple way to pass complex
 object graphs between threads. I'm picturing an API something like this:

 import std.stdio, whateverOrangeEndsUpBeingNamed.

 struct Cloned(T) {
 T obj;
 }

 Cloned!T cloned(T)(T obj) { return typeof(return)(obj); }

 void spawmMe() {
 receive((int[][]) { writeln("Received an int[][]."); });
 }

 void main() {
 auto tid = spawn(&spawnMe);
 auto mat = new int[][](42, 42);

 // FAILS: Implicit sharing.
 tid.send(mat);

 // Works. mat is automatically deep cloned and the clone is sent
 // to tid. send recognized the Cloned struct as special and
 // processes the payload accordingly.
 tid.send(cloned(mat));
 }

Currently you could serialize the data an pass it send it over to another thread and deserialize on the other side, but I'm guessing this would be not very effective. I'm not exactly sure what's needed to be done to make a clone archive and what can be shared but as you say, the necessary introspection already work.
 On 8/20/2011 11:13 AM, Jacob Carlborg wrote:
 I've almost finished the rewrite of my serialization library Orange. I'm
 hoping that someone wants to give it a try and see what issues/bugs are
 found.

 Project page: http://dsource.org/projects/orange
 Source code: https://github.com/jacob-carlborg/orange

 There are two usage examples on the project page. For more examples I
 recommend looking at the unit tests in the "tests" directory.

 Description:

 Orange is a serialization library for D1 and D2, supporting both Tango
 and Phobos. It can serialize most of the available types in D, including
 third party types and can serialize through base class references. It
 supports fully automatic serialization of all supported types and also
 supports several ways to customize the serialization process. Orange has
 a separate front end (the serializer) and back end (the archive) making
 it possible for the user to create new archive types that can be used
 with the existing serializer.

 Features:

 * Automatically serializes the base classes
 * Supports events (before and after (de)serializing)

 * Supports non-serialized fields and classes (you can say that some
 fields in a class should not be serialized)

 * Licensed under the Boost license
 * Std/runtime library independent

 * Extendable - possible to create new archive types and use them with
 the existing serializer

 * Serializes through base class references
 * Serializes third party types

 * Customization of the (de)serialization process, both intrusive and
 non-intrusive

 * Properly (de)serializes slices and pointers

 Known Issues/Limitations:

 * Due to limitations in the XML module provided by Phobos the XMLArchive
 will only work with "char" as the template type with D2

 * Due to several bugs/limitations in the compiler/runtime even the D2
 version requires you to register the type when serializing through base
 class references

 * No built-in support for versioning
 * Floating point numbers are not serialized as hexadecimal (D1)


-- /Jacob Carlborg
Aug 20 2011
parent reply dsimcha <dsimcha yahoo.com> writes:
On 8/20/2011 4:24 PM, Jacob Carlborg wrote:
 2. What are the prospects for submitting this for inclusion in Phobos?
 Serialization is something basic and universally needed enough that it
 should not require a third-party library.

From my part: * Remove all Tango/D1 specific code * Add documentation * There are some few things left in the rewrite process

Ok, it sounds like this isn't going to be ready that soon. I'll start using it as-is, as a third party library. Then I can make suggestions, file bug reports, etc. and streamline its eventual Phobos review.
 * I've made some modifications to the XML module in Phobos, these need
 to be applied as well

If we're really lucky, by the time your turn in the review queue comes along, Phobos will have a new and much better XML module, since Thomas Sowinki is apparently working on one.
 * Probably something else I can't think of right now

 I'm not particular happy about removing the Tango/D1 specific code if
 I'm not sure that the library can be included in Phobos.

Hmm. We could review the module before you remove it, with the assumption that this code will be removed if accepted.
 Currently you could serialize the data an pass it send it over to
 another thread and deserialize on the other side, but I'm guessing this
 would be not very effective. I'm not exactly sure what's needed to be
 done to make a clone archive and what can be shared but as you say, the
 necessary introspection already work.

Right. The idea would be to bypass serializing and unserializing and leverage the introspection that Orange provides to create an efficient clone() function. This is mostly a long-term prospect, to be considered more seriously if/when Orange gets into Phobos. All we'd need from orange is an archiver that efficiently clones without actually serializing or deserializing anything. Given this, the changes to std.concurrency would be trivial.
Aug 20 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-20 23:29, dsimcha wrote:
 On 8/20/2011 4:24 PM, Jacob Carlborg wrote:
 2. What are the prospects for submitting this for inclusion in Phobos?
 Serialization is something basic and universally needed enough that it
 should not require a third-party library.

From my part: * Remove all Tango/D1 specific code * Add documentation * There are some few things left in the rewrite process

Ok, it sounds like this isn't going to be ready that soon. I'll start using it as-is, as a third party library. Then I can make suggestions, file bug reports, etc. and streamline its eventual Phobos review.

I will add documentation and finish the rewrite regardless so it's only the D1/Tango specific code. Oh, one more thing. I'm using my own kind of unit test "framework", don't know what people think about that. It could easily be removed though.
 * I've made some modifications to the XML module in Phobos, these need
 to be applied as well

If we're really lucky, by the time your turn in the review queue comes along, Phobos will have a new and much better XML module, since Thomas Sowinki is apparently working on one.

Yes, hopefully. Also, some of the changes I've made are because of differences with the Tango and Phobos XML modules. Since it will only support the Phobos module it will be easier and hopefully not require as many changes.
 * Probably something else I can't think of right now

 I'm not particular happy about removing the Tango/D1 specific code if
 I'm not sure that the library can be included in Phobos.

Hmm. We could review the module before you remove it, with the assumption that this code will be removed if accepted.

Exactly, sounds like a good idea.
 Currently you could serialize the data an pass it send it over to
 another thread and deserialize on the other side, but I'm guessing this
 would be not very effective. I'm not exactly sure what's needed to be
 done to make a clone archive and what can be shared but as you say, the
 necessary introspection already work.

Right. The idea would be to bypass serializing and unserializing and leverage the introspection that Orange provides to create an efficient clone() function. This is mostly a long-term prospect, to be considered more seriously if/when Orange gets into Phobos. All we'd need from orange is an archiver that efficiently clones without actually serializing or deserializing anything. Given this, the changes to std.concurrency would be trivial.

Yeah, I wonder if any data needs to be stored in the archive and in that case, how it could be stored. -- /Jacob Carlborg
Aug 21 2011
prev sibling next sibling parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Sun, 21 Aug 2011 02:45:20 +0900, dsimcha <dsimcha yahoo.com> wrote:

 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/

Yeah, this does look quite useful. I tried it a few weeks ago and ran into a couple showstopper bugs. I filed them and he fixed them, but I never got around to trying it again. I actually think that since both are very commonly needed things and satisfy completely different niches, Orange and msgpack4d might both eventually have a rightful place in Phobos.

I agree. In the past, I mentioned this point ;)
 Orange would be recommended when portability and serializing as many  
 types as possible are important, and msgpack4d would be recommended when  
 speed and space efficiency are the primary concerns.

Yes for MessagePack. In addition, communicating other language is important like JSON. P.S. My D libraries moved to github. msgpack4d is here. https://github.com/msgpack/msgpack-d FTTB, bitbucket is available.
Aug 21 2011
prev sibling next sibling parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Sun, 21 Aug 2011 05:28:33 +0900, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-20 19:45, dsimcha wrote:
 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/

Yeah, this does look quite useful. I tried it a few weeks ago and ran into a couple showstopper bugs. I filed them and he fixed them, but I never got around to trying it again. I actually think that since both are very commonly needed things and satisfy completely different niches, Orange and msgpack4d might both eventually have a rightful place in Phobos. Orange would be recommended when portability and serializing as many types as possible are important, and msgpack4d would be recommended when speed and space efficiency are the primary concerns.

If it's possible to extract a value given a key with messagepack I think it would be possible to use messagepack as an archive for Orange. Since messagepack supports maps every type Orange can serialize should be possible serialize with messagepack, although probably not as efficient as "regular" messagepack.

I agree. Unfortunately, Orange with MessagePack decreases MessagePack merits...
Aug 21 2011
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
On Aug 20, 2011, at 9:16 AM, dsimcha wrote:

 This looks really great!  I'll try it out sometime later.  A few =

=20
 1.  What other archiver formats besides XML might be useful?  (I =

messagepack is probably a better option for lightweight serialization in = general. Orange's niche is portability and serializing as many types as = possible.) JSON, Protobuf=85
 2.  What are the prospects for submitting this for inclusion in =

that it should not require a third-party library. Some form of serialization definitely needs to be in Phobos/Druntime. = The other day I was thinking that the basic framework for struct/class = serialization might be nice in Druntime as a replacement for x.toString, = with most of the actual functional bits in Phobos.=
Aug 29 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-29 19:19, Sean Kelly wrote:
 On Aug 20, 2011, at 9:16 AM, dsimcha wrote:

 This looks really great!  I'll try it out sometime later.  A few questions:

 1.  What other archiver formats besides XML might be useful?  (I remember
binary can't work because of the way keys work, though messagepack is probably
a better option for lightweight serialization in general.  Orange's niche is
portability and serializing as many types as possible.)

JSON, Protobuf…

YAML as well.
 2.  What are the prospects for submitting this for inclusion in Phobos? 
Serialization is something basic and universally needed enough that it should
not require a third-party library.

Some form of serialization definitely needs to be in Phobos/Druntime. The other day I was thinking that the basic framework for struct/class serialization might be nice in Druntime as a replacement for x.toString, with most of the actual functional bits in Phobos.

I've been think the same several times before and think it would be really useful for debugging. -- /Jacob Carlborg
Aug 29 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Masahiro Nakagawa seems to be working on msgpack:
https://bitbucket.org/repeatedly/msgpack4d/

Is RIFF considered a good format? I've ran into it when porting C
code,  it seems it's also used by Google and probably other companies.
http://en.wikipedia.org/wiki/Resource_Interchange_File_Format
Aug 20 2011
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:

 I've almost finished the rewrite of my serialization library Orange.

While I've never had the chance to use Orange myself, one of the problems I've often heard being associated with using Orange was large binary file sizes. What are your thoughts about this issue? If this is a real problem, do you think the toolchain could be improved in this regard (perhaps ideas such as merging bitwise-identical functions at link time)? -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 21:50, Vladimir Panteleev wrote:
 On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:

 I've almost finished the rewrite of my serialization library Orange.

While I've never had the chance to use Orange myself, one of the problems I've often heard being associated with using Orange was large binary file sizes. What are your thoughts about this issue? If this is a real problem, do you think the toolchain could be improved in this regard (perhaps ideas such as merging bitwise-identical functions at link time)?

I never heard anything being associated with using Orange. The library takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World app using Orange takes up 1.2 mb, I don't know if that's a problem. I don't have any thoughts, I didn't know it was an issue. Yes I think the tool chain can be improved, at least I hope it can. -- /Jacob Carlborg
Aug 22 2011
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2011-08-23 06:51:57 +0000, Jacob Carlborg <doob me.com> said:

 On 2011-08-22 21:50, Vladimir Panteleev wrote:
 On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:
 
 I've almost finished the rewrite of my serialization library Orange.

While I've never had the chance to use Orange myself, one of the problems I've often heard being associated with using Orange was large binary file sizes. What are your thoughts about this issue? If this is a real problem, do you think the toolchain could be improved in this regard (perhaps ideas such as merging bitwise-identical functions at link time)?

I never heard anything being associated with using Orange. The library takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World app using Orange takes up 1.2 mb, I don't know if that's a problem. I don't have any thoughts, I didn't know it was an issue. Yes I think the tool chain can be improved, at least I hope it can.

If there's any truth in that (which I don't know), it'd probably have something to do with the amount of generated code through templates. When I was still developing the D/Objective-C bridge as a bunch of templates and stub objects, the binary size was growing significantly for each new Objective-C class I added (due to the insane number of stub functions instantiated). And at some point it became totally impractical to use, even though it worked. I doubt this is a problem for Orange because of its more limited scope, but it'd nevertheless be wise to measure how much the binary size grows when you add more types to serialize and unserialize and more code is generated as a result. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Aug 23 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-24 02:53, Michel Fortin wrote:
 On 2011-08-23 06:51:57 +0000, Jacob Carlborg <doob me.com> said:

 On 2011-08-22 21:50, Vladimir Panteleev wrote:
 On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:

 I've almost finished the rewrite of my serialization library Orange.

While I've never had the chance to use Orange myself, one of the problems I've often heard being associated with using Orange was large binary file sizes. What are your thoughts about this issue? If this is a real problem, do you think the toolchain could be improved in this regard (perhaps ideas such as merging bitwise-identical functions at link time)?

I never heard anything being associated with using Orange. The library takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World app using Orange takes up 1.2 mb, I don't know if that's a problem. I don't have any thoughts, I didn't know it was an issue. Yes I think the tool chain can be improved, at least I hope it can.

If there's any truth in that (which I don't know), it'd probably have something to do with the amount of generated code through templates. When I was still developing the D/Objective-C bridge as a bunch of templates and stub objects, the binary size was growing significantly for each new Objective-C class I added (due to the insane number of stub functions instantiated). And at some point it became totally impractical to use, even though it worked.

Yeah, that was insane.
 I doubt this is a problem for Orange because of its more limited scope,
 but it'd nevertheless be wise to measure how much the binary size grows
 when you add more types to serialize and unserialize and more code is
 generated as a result.

I guess I should try serializing a big data structure of some kind. -- /Jacob Carlborg
Aug 23 2011
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Wed, 24 Aug 2011 02:25:19 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-24 02:53, Michel Fortin wrote:
 On 2011-08-23 06:51:57 +0000, Jacob Carlborg <doob me.com> said:

 On 2011-08-22 21:50, Vladimir Panteleev wrote:
 On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:

 I've almost finished the rewrite of my serialization library Orange.

While I've never had the chance to use Orange myself, one of the problems I've often heard being associated with using Orange was large binary file sizes. What are your thoughts about this issue? If this is a real problem, do you think the toolchain could be improved in this regard (perhaps ideas such as merging bitwise-identical functions at link time)?

I never heard anything being associated with using Orange. The library takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World app using Orange takes up 1.2 mb, I don't know if that's a problem. I don't have any thoughts, I didn't know it was an issue. Yes I think the tool chain can be improved, at least I hope it can.

If there's any truth in that (which I don't know), it'd probably have something to do with the amount of generated code through templates. When I was still developing the D/Objective-C bridge as a bunch of templates and stub objects, the binary size was growing significantly for each new Objective-C class I added (due to the insane number of stub functions instantiated). And at some point it became totally impractical to use, even though it worked.

Yeah, that was insane.
 I doubt this is a problem for Orange because of its more limited scope,
 but it'd nevertheless be wise to measure how much the binary size grows
 when you add more types to serialize and unserialize and more code is
 generated as a result.

I guess I should try serializing a big data structure of some kind.

Actually, it's the number of types rather than the amount of memory which causes issues. You can try something like this: template Qual(T) { alias TypeTuple!(T,T[],const(T)[],immutable(T)[],shared(T)[], const T, immutable T, shared T, const T[], immutable T[], shared T[]) Qual; } alias TypeTuple!(bool,ubyte,byte,ushort,short,uint,int,ulong,long,float,double,re l,char,wchar,dchar) ValueTypes; alias staticMap!(Qual,ValueTypes) QualifiedTypes; To generate a large tuple of all the basic D types and then see how much code is generated.
Aug 24 2011