Digital Mars - digitalmars.D.announce - Serialization for D. Comments, please!

BCS <none anon.com> writes:

Hello Tim,

 On Wed, 20 May 2009 15:43:16 +1200, BCS <none anon.com> wrote:
 
 I'm planning on taking a crack at a Serialization template library
 and  I'm looking for feed back. My thinking so far is up on my blog
 here:
 
 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part
 -1-of-n.html
 
 Please comment! (here or there, doesn't matter, I think I'll see
 both)
 


 with  data and functions but the objects can be saved and retrieved in
 a  transactional way?
 


The target usage is to stuff data structures into file or network streams. 
Eventually, I'd like to build a web services lib from this that can serve 
proxy an interface with very little code:

/////common:

interface I { ... }
interface J { ... }

/////Server side

class CI : I { ... }
class CJ : J { ... }

auto server = new Server();

server.Serve!(I)(new CI(), 8080); //proxy that object on port 8080
server.Serve!(J)({return new CJ;}, 8081); // create a objects to proxy on 
port 8081

server.start();

//// client side

I i = new Proxy!(I)("somehost.net", 8080);
J j = new Proxy!(J)("somehost.net", 8081);

May 19 2009

BCS <none anon.com> writes:

Hello Qian,

 BCS wrote:
 
 I'm planning on taking a crack at a Serialization template library
 and I'm looking for feed back. My thinking so far is up on my blog
 here:
 


 1-of-n.html
 
 Please comment! (here or there, doesn't matter, I think I'll see
 both)
 


 Should every object contain "mixin Serializable!()" in its
 declaration? It is easy to add this in own classes, but
 not easy to 3rd-party libraries. for instance: tango
 --Qian
 


I've been mentally dodging that question because it's going to be ugly. If 
anyone has some ideas on how to make it work, I'm interested.

Right now the system has a hard coded list of basic types (byte, real, etc.) 
that it can work with and failing that, it assume you have added "mixin
Serializable!()". 
The main problem is getting the third case is overload. Overloading doesn't 
work across mixins or modules last I checked. It would be a hack, but named 
function called via string mixins (mixin("Marshal"~ typeof(a).stringof ~ 
"(sink,a);")) might work.

May 26 2009

aarti_pl <aarti interia.pl> writes:

BCS pisze:
 I'm planning on taking a crack at a Serialization template library and 
 I'm looking for feed back. My thinking so far is up on my blog here:
 
 http://arrayboundserror.blogspot.com/2009/05/serialization-fo
-d-part-1-of-n.html 
 
 
 Please comment! (here or there, doesn't matter, I think I'll see both)
 
 


Hello!

I would like to mention that serialization library for D already exists:

http://www.dsource.org/projects/doost

It supports most of common requirements for serialization library:
- easy to use API
- most built-in types and user defined types (UDT) are already serializable
- out of class serialization
- versioning of UDT; importing old versions to new versions
- thread/exception safe (well... AFAIK :-) )
- large unit test suite
- configurable serialization backend (archive): JSon, Binary, SimpleText
- serialization of classes with user defined constructors

I don't want to discourage you from implementing your own library, but 
on the other hand you may consider contributing to this, already 
existing library. It took me few months to get where it is right now, so 
it is really a LOT of work; Unfortunately I don't have time to develop 
it further right now...

There are still few things which should be implemented:
- serialization from base class (difficult, when we don't want to demand 
additional coding work from user)
- Tango support (should be relatively easy)
- XML archive support (foundation framework already there)
- serialization for complex types
- cleanups, improvements, porting to D 2.0
- better documentation

If you would be interested just drop me an e-mail:

aarti -at- interia -dot- pl

I am very open to suggestions and changes :-)

Best Regards
Marcin Kuszczak
(aarti_pl)

May 26 2009

Bill Baxter <wbaxter gmail.com> writes:

On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com> wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library and
 I'm
 looking for feed back. My thinking so far is up on my blog here:


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html
 Please comment! (here or there, doesn't matter, I think I'll see both)


 A question:
 Should every object contain "mixin Serializable!()" in its declaration?
 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via template
 specialization (or similar tricks) :)


I'll also add that you should be able to properly
serialize/deserialize a BaseClass pointer that actually points to a
DerivedClass.
This is pretty tricky to get working seamlessly when combined with the
external serialization requirement.
H3r3tic's xpose library has this working IIRC.

--bb

May 26 2009

"Denis Koroskin" <2korden gmail.com> writes:

On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com> wrote:

 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>  
 wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu  
 <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library and
 I'm
 looking for feed back. My thinking so far is up on my blog here:


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html
 Please comment! (here or there, doesn't matter, I think I'll see both)


 A question:
 Should every object contain "mixin Serializable!()" in its declaration?
 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via template
 specialization (or similar tricks) :)


 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.

 --bb


Sure! How about the following test:

struct A
{
    mixin Serializable!();

    B[] b;
}

struct B
{
    mixin Serializable!();

    A a;
    int i;
}

B[] b = new B[1];
b[0].a.b = b;
b[0].i = 42;

A* a = &b[0].a;

data = serialize(a); // should serialize outer struct (B), too, because it is
accessible through a

a = deserialize(data);
assert(a.b.length == 1);
assert(a.b[0].i == 42);

BTW, thanks for reminding about xpose. I gotta compare it, doost.serialize and
mine own one against each other in terms of efficiency and size (for binary
output) someday.

May 26 2009

Bill Baxter <wbaxter gmail.com> writes:

On Tue, May 26, 2009 at 12:19 PM, Denis Koroskin <2korden gmail.com> wrote:
 On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com> wrote=


 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>
 wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu
 <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library an=










 I'm
 looking for feed back. My thinking so far is up on my blog here:


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-=








 Please comment! (here or there, doesn't matter, I think I'll see both=










 A question:
 Should every object contain "mixin Serializable!()" in its declaration=








 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via template
 specialization (or similar tricks) :)


 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.

 --bb


 Sure! How about the following test:

 struct A
 {
 =A0 =A0mixin Serializable!();

 =A0 =A0B[] b;
 }

 struct B
 {
 =A0 =A0mixin Serializable!();

 =A0 =A0A a;
 =A0 =A0int i;
 }

 B[] b =3D new B[1];
 b[0].a.b =3D b;
 b[0].i =3D 42;

 A* a =3D &b[0].a;

 data =3D serialize(a); // should serialize outer struct (B), too, because=


 a =3D deserialize(data);
 assert(a.b.length =3D=3D 1);
 assert(a.b[0].i =3D=3D 42);

 BTW, thanks for reminding about xpose. I gotta compare it, doost.serializ=


binary output) someday.

That's not quite what I meant.

I mean this:

/// 3RD PARTY LIB
class BaseClass
{
     string toString() { return "base"; }
}

class DerivedClass
{
     string toString() { return "derived"; }
}


/// MY CODE

[...some kind of class registration here....]

BaseClass b =3D new DerivedClass;

data =3D serialize(b);
b2 =3D deserialize(data);

assert(b2.toString() =3D=3D "derived");
------------------

The data stream should contain enough info about the instance to
reconstruct the exact derived type.

--bb

May 26 2009

"Denis Koroskin" <2korden gmail.com> writes:

On Wed, 27 May 2009 01:23:58 +0400, Bill Baxter <wbaxter gmail.com> wrote:

 On Tue, May 26, 2009 at 12:19 PM, Denis Koroskin <2korden gmail.com>  
 wrote:
 On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com>  
 wrote:

 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>
 wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu
 <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library  
 and
 I'm
 looking for feed back. My thinking so far is up on my blog here:


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html
 Please comment! (here or there, doesn't matter, I think I'll see  
 both)


 A question:
 Should every object contain "mixin Serializable!()" in its  
 declaration?
 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via  
 template
 specialization (or similar tricks) :)


 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.

 --bb


 Sure! How about the following test:

 struct A
 {
    mixin Serializable!();

    B[] b;
 }

 struct B
 {
    mixin Serializable!();

    A a;
    int i;
 }

 B[] b = new B[1];
 b[0].a.b = b;
 b[0].i = 42;

 A* a = &b[0].a;

 data = serialize(a); // should serialize outer struct (B), too, because  
 it is accessible through a

 a = deserialize(data);
 assert(a.b.length == 1);
 assert(a.b[0].i == 42);

 BTW, thanks for reminding about xpose. I gotta compare it,  
 doost.serialize and mine own one against each other in terms of  
 efficiency and size (for binary output) someday.


 That's not quite what I meant.

 I mean this:

 /// 3RD PARTY LIB
 class BaseClass
 {
      string toString() { return "base"; }
 }

 class DerivedClass
 {
      string toString() { return "derived"; }
 }


 /// MY CODE

 [...some kind of class registration here....]

 BaseClass b = new DerivedClass;

 data = serialize(b);
 b2 = deserialize(data);

 assert(b2.toString() == "derived");
 ------------------

 The data stream should contain enough info about the instance to
 reconstruct the exact derived type.

 --bb


Yes, I understood and showed yet another feature which is quite hard to
implement.

May 26 2009

Bill Baxter <wbaxter gmail.com> writes:

On Tue, May 26, 2009 at 2:26 PM, Denis Koroskin <2korden gmail.com> wrote:
 On Wed, 27 May 2009 01:23:58 +0400, Bill Baxter <wbaxter gmail.com> wrote=


 On Tue, May 26, 2009 at 12:19 PM, Denis Koroskin <2korden gmail.com>
 wrote:
 On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com>
 wrote:

 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>
 wrote:
 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu
 <quian.xu stud.tu-ilmenau.de>
 wrote:

 BCS wrote:

 I'm planning on taking a crack at a Serialization template library
 and
 I'm
 looking for feed back. My thinking so far is up on my blog here:



 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-par=












 Please comment! (here or there, doesn't matter, I think I'll see
 both)


 A question:
 Should every object contain "mixin Serializable!()" in its
 declaration?
 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via templa=










 specialization (or similar tricks) :)


 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.

 --bb


 Sure! How about the following test:

 struct A
 {
 =A0 mixin Serializable!();

 =A0 B[] b;
 }

 struct B
 {
 =A0 mixin Serializable!();

 =A0 A a;
 =A0 int i;
 }

 B[] b =3D new B[1];
 b[0].a.b =3D b;
 b[0].i =3D 42;

 A* a =3D &b[0].a;

 data =3D serialize(a); // should serialize outer struct (B), too, becau=






 it is accessible through a

 a =3D deserialize(data);
 assert(a.b.length =3D=3D 1);
 assert(a.b[0].i =3D=3D 42);

 BTW, thanks for reminding about xpose. I gotta compare it,
 doost.serialize and mine own one against each other in terms of efficie=






 and size (for binary output) someday.


 That's not quite what I meant.

 I mean this:

 /// 3RD PARTY LIB
 class BaseClass
 {
 =A0 =A0 string toString() { return "base"; }
 }

 class DerivedClass
 {
 =A0 =A0 string toString() { return "derived"; }
 }


 /// MY CODE

 [...some kind of class registration here....]

 BaseClass b =3D new DerivedClass;

 data =3D serialize(b);
 b2 =3D deserialize(data);

 assert(b2.toString() =3D=3D "derived");
 ------------------

 The data stream should contain enough info about the instance to
 reconstruct the exact derived type.

 --bb


 Yes, I understood and showed yet another feature which is quite hard to
 implement.


Ok.  I interpreted that as "Sure! How about the following test (of the
feature you just mentioned)".
Some segue words like "in addition" or "also" probably would have
tipped me off that you were changing the subject. :-)

--bb

May 26 2009