www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.jgrandson

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
We need a better json library at Facebook. I'd discussed with Snke the 
possibility of taking vibe.d's json to std but he said it needs some 
more work. So I took std.jgrandson to proof of concept state and hence 
ready for destruction:

http://erdani.com/d/jgrandson.d
http://erdani.com/d/phobos-prerelease/std_jgrandson.html

Here are a few differences compared to vibe.d's library. I think these 
are desirable to have in that library as well:

* Parsing strings is decoupled into tokenization (which is lazy and only 
needs an input range) and parsing proper. Tokenization is lazy, which 
allows users to create their own advanced (e.g. partial/lazy) parsing if 
needed. The parser itself is eager.

* There's no decoding of strings.

* The representation is built on Algebraic, with the advantages that it 
benefits from all of its primitives. Implementation is also very compact 
because Algebraic obviates a bunch of boilerplate. Subsequent 
improvements to Algebraic will also reflect themselves into improvements 
to std.jgrandson.

* The JSON value (called std.jgrandson.Value) has no named member 
variables or methods except for __payload. This is so there's no clash 
between dynamic properties exposed via opDispatch.

Well that's about it. What would it take for this to become a Phobos 
proposal? Destroy.


Andrei
Aug 03 2014
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 03 Aug 2014 00:16:04 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 We need a better json library at Facebook. I'd discussed with S=C3=B6nke
 the possibility of taking vibe.d's json to std but he said it needs
 some more work. So I took std.jgrandson to proof of concept state and
 hence ready for destruction:
=20
 http://erdani.com/d/jgrandson.d
 http://erdani.com/d/phobos-prerelease/std_jgrandson.html
=20
 Here are a few differences compared to vibe.d's library. I think
 these are desirable to have in that library as well:
=20
 * Parsing strings is decoupled into tokenization (which is lazy and
 only needs an input range) and parsing proper. Tokenization is lazy,
 which allows users to create their own advanced (e.g. partial/lazy)
 parsing if needed. The parser itself is eager.
=20
 * There's no decoding of strings.
=20
 * The representation is built on Algebraic, with the advantages that
 it benefits from all of its primitives. Implementation is also very
 compact because Algebraic obviates a bunch of boilerplate. Subsequent=20
 improvements to Algebraic will also reflect themselves into
 improvements to std.jgrandson.
=20
 * The JSON value (called std.jgrandson.Value) has no named member=20
 variables or methods except for __payload. This is so there's no
 clash between dynamic properties exposed via opDispatch.
=20
 Well that's about it. What would it take for this to become a Phobos=20
 proposal? Destroy.
=20
=20
 Andrei
API looks great but I'd like to see some simple serialize/deserialize functions as in vibed: http://vibed.org/api/vibe.data.json/deserializeJson http://vibed.org/api/vibe.data.json/serializeToJson vibe uses UDAs to customize the serialization output. That's actually not json specific and therefore shouldn't be part of this module. But a simple deserializeJson which simply fills in all fields of a struct given a TokenStream is very useful and can be done without allocations (so it's much faster than going through the DOM). Nitpicks: * I'd make Token only store strings, then convert to double/number only when requested. If a user is simply skipping some tokens these conversions are unnecessary overhead. * parseString really shouldn't use appender. Make it somehow possible to supply a buffer to TokenStream and use that. (This way there's no memory allocation. If a user want to keep the string he has to .dup it). A BufferedRange concept might even be better, because you can read in blocks and reuse buffers.
Aug 03 2014
next sibling parent reply "ponce" <contact gam3sfrommars.fr> writes:
 API looks great but I'd like to see some simple 
 serialize/deserialize
 functions as in vibed:
 http://vibed.org/api/vibe.data.json/deserializeJson
 http://vibed.org/api/vibe.data.json/serializeToJson

 vibe uses UDAs to customize the serialization output. That's 
 actually
 not json specific and therefore shouldn't be part of this 
 module. But a
 simple deserializeJson which simply fills in all fields of a 
 struct
 given a TokenStream is very useful and can be done without 
 allocations
 (so it's much faster than going through the DOM).
That's what https://github.com/Orvid/JSONSerialization does. Also msgpack-d https://github.com/msgpack/msgpack-d whose defaults need no UDAs That makes the typical use case very fast to write.
Aug 03 2014
parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 03.08.2014 10:25, schrieb ponce:
 API looks great but I'd like to see some simple serialize/deserialize
 functions as in vibed:
 http://vibed.org/api/vibe.data.json/deserializeJson
 http://vibed.org/api/vibe.data.json/serializeToJson

 vibe uses UDAs to customize the serialization output. That's actually
 not json specific and therefore shouldn't be part of this module. But a
 simple deserializeJson which simply fills in all fields of a struct
 given a TokenStream is very useful and can be done without allocations
 (so it's much faster than going through the DOM).
That's what https://github.com/Orvid/JSONSerialization does. Also msgpack-d https://github.com/msgpack/msgpack-d whose defaults need no UDAs That makes the typical use case very fast to write.
The default mode for vibe.data.serialization also doesn't need any UDAs, but it's still often useful to be able to make customizations.
Aug 03 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 1:02 AM, Johannes Pfau wrote:
 API looks great but I'd like to see some simple serialize/deserialize
 functions as in vibed:
 http://vibed.org/api/vibe.data.json/deserializeJson
 http://vibed.org/api/vibe.data.json/serializeToJson
Agreed.
 vibe uses UDAs to customize the serialization output. That's actually
 not json specific and therefore shouldn't be part of this module. But a
 simple deserializeJson which simply fills in all fields of a struct
 given a TokenStream is very useful and can be done without allocations
 (so it's much faster than going through the DOM).
Nice.
 Nitpicks:

 * I'd make Token only store strings, then convert to double/number only
    when requested. If a user is simply skipping some tokens these
    conversions are unnecessary overhead.
Well... this is tricky. If the input has immutable characters, they can be stored because it can be assumed they'll live forever. If they're mutable or const, that assumption doesn't hold so every number must allocate. At that point it's probably cheaper to just convert to double. One thing is I didn't treat integers specially, but I did notice some json parsers do make that distinction.
 * parseString really shouldn't use appender. Make it somehow possible
    to supply a buffer to TokenStream and use that. (This way there's no
    memory allocation. If a user want to keep the string he has to .dup
    it). A BufferedRange concept might even be better, because you can
    read in blocks and reuse buffers.
Good suggestion, thanks. Andrei
Aug 03 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Sunday, 3 August 2014 at 08:04:40 UTC, Johannes Pfau wrote:
 API looks great but I'd like to see some simple 
 serialize/deserialize
 functions as in vibed:
 http://vibed.org/api/vibe.data.json/deserializeJson
 http://vibed.org/api/vibe.data.json/serializeToJson
Before going this route one needs to have a good vision how it may interact with imaginary std.serialization to avoid later deprecation. At the same time I have recently started to think that dedicated serialization module that decouples aggregate iteration from data storage format is in most cases impractical for performance reasons - different serialization methods imply very different efficient iteration strategies. Probably it is better to define serialization compile-time traits instead and require each `std.data.*` provider to implement those on its own in the most effective fashion.
Aug 03 2014
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 03.08.2014 20:44, schrieb Dicebot:
 On Sunday, 3 August 2014 at 08:04:40 UTC, Johannes Pfau wrote:
 API looks great but I'd like to see some simple serialize/deserialize
 functions as in vibed:
 http://vibed.org/api/vibe.data.json/deserializeJson
 http://vibed.org/api/vibe.data.json/serializeToJson
Before going this route one needs to have a good vision how it may interact with imaginary std.serialization to avoid later deprecation. At the same time I have recently started to think that dedicated serialization module that decouples aggregate iteration from data storage format is in most cases impractical for performance reasons - different serialization methods imply very different efficient iteration strategies. Probably it is better to define serialization compile-time traits instead and require each `std.data.*` provider to implement those on its own in the most effective fashion.
Do you have a specific case in mind where the data format doesn't fit the process used by vibe.data.serialization? The data format iteration part *is* abstracted away there in basically a kind of traits structure (the "Serializer"). When serializing, the data always gets written in the order defined by the input value, while during deserialization the serializer defines how aggregates are iterated. This seems to fit all of the data formats that I had in mind.
Aug 03 2014
parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 3 August 2014 at 19:36:43 UTC, Sönke Ludwig wrote:
 Do you have a specific case in mind where the data format 
 doesn't fit the process used by vibe.data.serialization? The 
 data format iteration part *is* abstracted away there in 
 basically a kind of traits structure (the "Serializer"). When 
 serializing, the data always gets written in the order defined 
 by the input value, while during deserialization the serializer 
 defines how aggregates are iterated. This seems to fit all of 
 the data formats that I had in mind.
For example we use special binary serialization format for structs where serialized content is actually a valid D struct - after updating internal array pointers one can simply do `cast(S*) buffer.ptr` and work with it normally. Doing this efficiently requires breadth-first traversal and keeping track of one upper level to update the pointers. This does not fit very well with classical depth-first recursive traversal usually required by JSON-structure formats.
Aug 03 2014
prev sibling parent reply "Jacob Carlborg" <doob me.com> writes:
On Sunday, 3 August 2014 at 18:44:37 UTC, Dicebot wrote:

 Before going this route one needs to have a good vision how it 
 may interact with imaginary std.serialization to avoid later 
 deprecation.
I suggest only provide functions for serializing primitive types. A separate serialization module/package with a JSON archive type would use this module as its backend.
 At the same time I have recently started to think that 
 dedicated serialization module that decouples aggregate 
 iteration from data storage format is in most cases impractical 
 for performance reasons - different serialization methods imply 
 very different efficient iteration strategies. Probably it is 
 better to define serialization compile-time traits instead and 
 require each `std.data.*` provider to implement those on its 
 own in the most effective fashion.
I'm not sure I agree with that. In my work on std.serialization I have not seen this to be a problem. What problems have you found? -- /Jacob Carlborg
Aug 04 2014
next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Jacob Carlborg"  wrote in message 
news:bjecckhwlmkwkeqegwqa forum.dlang.org...

 I suggest only provide functions for serializing primitive types.
This is exactly what I need in most projects. Basic types, arrays, AAs, and structs are usually enough.
Aug 04 2014
parent reply "Jacob Carlborg" <doob me.com> writes:
On Monday, 4 August 2014 at 09:10:46 UTC, Daniel Murphy wrote:

 This is exactly what I need in most projects.  Basic types, 
 arrays, AAs, and structs are usually enough.
I was more thinking only types that cannot be broken down in to smaller pieces, i.e. integer, floating point, bool and string. The serializer would break down the other types in to smaller pieces. -- /Jacob Carlborg
Aug 04 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Jacob Carlborg"  wrote in message 
news:kvuaxyxjwmpqrorlozrz forum.dlang.org...

 This is exactly what I need in most projects.  Basic types, arrays, AAs, 
 and structs are usually enough.
I was more thinking only types that cannot be broken down in to smaller pieces, i.e. integer, floating point, bool and string. The serializer would break down the other types in to smaller pieces.
I guess I meant types that have an obvious mapping to json types. int/long -> json integer bool -> json bool string -> json string float/real -> json float (close enough) T[] -> json array T[string] -> json object struct -> json object This is usually enough for config and data files. Being able to do this is just awesome: struct AppConfig { string somePath; bool someOption; string[] someList; string[string] someMap; } void main() { auto config = "config.json".readText().parseJSON().fromJson!AppConfig(); } Being able to serialize whole graphs into json is something I need much less often.
Aug 05 2014
next sibling parent reply "Andrea Fontana" <nospam example.com> writes:
On Tuesday, 5 August 2014 at 12:40:25 UTC, Daniel Murphy wrote:
 "Jacob Carlborg"  wrote in message 
 news:kvuaxyxjwmpqrorlozrz forum.dlang.org...

 This is exactly what I need in most projects.  Basic types, 
 arrays, AAs, and structs are usually enough.
I was more thinking only types that cannot be broken down in to smaller pieces, i.e. integer, floating point, bool and string. The serializer would break down the other types in to smaller pieces.
I guess I meant types that have an obvious mapping to json types. int/long -> json integer bool -> json bool string -> json string float/real -> json float (close enough) T[] -> json array T[string] -> json object struct -> json object This is usually enough for config and data files. Being able to do this is just awesome: struct AppConfig { string somePath; bool someOption; string[] someList; string[string] someMap; } void main() { auto config = "config.json".readText().parseJSON().fromJson!AppConfig(); } Being able to serialize whole graphs into json is something I need much less often.
If I'm right, json has just one numeric type. No difference between integers / float and no limits. So probably the mapping is: float/double/real/int/long => number
Aug 05 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrea Fontana"  wrote in message 
news:takluoqmlmmooxlovqya forum.dlang.org...

 If I'm right, json has just one numeric type. No difference between 
 integers / float and no limits.

 So probably the mapping is:

 float/double/real/int/long => number
Maybe, but std.json has three numeric types.
Aug 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/5/14, 8:23 AM, Daniel Murphy wrote:
 "Andrea Fontana"  wrote in message
 news:takluoqmlmmooxlovqya forum.dlang.org...

 If I'm right, json has just one numeric type. No difference between
 integers / float and no limits.

 So probably the mapping is:

 float/double/real/int/long => number
Maybe, but std.json has three numeric types.
I searched around a bit and it seems different libraries have different takes to this numeric matter. A simple reading of the spec suggests that floating point data is the only numeric type. However, many implementations choose to distinguish between floating point and integrals. Andrei
Aug 05 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 5 August 2014 at 17:17:56 UTC, Andrei Alexandrescu 
wrote:
 On 8/5/14, 8:23 AM, Daniel Murphy wrote:
 "Andrea Fontana"  wrote in message
 news:takluoqmlmmooxlovqya forum.dlang.org...

 If I'm right, json has just one numeric type. No difference 
 between
 integers / float and no limits.

 So probably the mapping is:

 float/double/real/int/long => number
Maybe, but std.json has three numeric types.
I searched around a bit and it seems different libraries have different takes to this numeric matter. A simple reading of the spec suggests that floating point data is the only numeric type. However, many implementations choose to distinguish between floating point and integrals.
There is certain benefit in using same primitive types for JSON as ones defined by BSON spec.
Aug 05 2014
prev sibling parent reply "Sean Kelly" <sean invisibleduck.org> writes:
On Tuesday, 5 August 2014 at 17:17:56 UTC, Andrei Alexandrescu
wrote:
 I searched around a bit and it seems different libraries have 
 different takes to this numeric matter. A simple reading of the 
 spec suggests that floating point data is the only numeric 
 type. However, many implementations choose to distinguish 
 between floating point and integrals.
The original point of JSON was that it auto-converts to Javascript data. And since Javascript only has one numeric type, of course JSON does too. But I think it's important that a JSON package for a language maps naturally to the types available in that language. D provides both floating point and integer types, each with their own costs and benefits, and so the JSON package should as well. It ends up being a lot easier to deal with than remembering to round from JSON.number or whatever when assigning to an int. In fact, JSON doesn't even impose any precision restrictions on its numeric type, so one could argue that we should be using BigInt and BigFloat. But this would stink most of the time, so... On an unrelated note, while the default encoding for strings is UTF-8, the RFC absolutely allows for UTF-16 surrogate pairs, and this must be supported. Any strings you get from Internet Explorer will be encoded as UTF-16 surrogate pairs regardless of content, presumably since Windows uses 16 bit wide chars for unicode.
Aug 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/5/14, 10:48 AM, Sean Kelly wrote:
 On Tuesday, 5 August 2014 at 17:17:56 UTC, Andrei Alexandrescu
 wrote:
 I searched around a bit and it seems different libraries have
 different takes to this numeric matter. A simple reading of the spec
 suggests that floating point data is the only numeric type. However,
 many implementations choose to distinguish between floating point and
 integrals.
The original point of JSON was that it auto-converts to Javascript data. And since Javascript only has one numeric type, of course JSON does too. But I think it's important that a JSON package for a language maps naturally to the types available in that language. D provides both floating point and integer types, each with their own costs and benefits, and so the JSON package should as well. It ends up being a lot easier to deal with than remembering to round from JSON.number or whatever when assigning to an int. In fact, JSON doesn't even impose any precision restrictions on its numeric type, so one could argue that we should be using BigInt and BigFloat. But this would stink most of the time, so... On an unrelated note, while the default encoding for strings is UTF-8, the RFC absolutely allows for UTF-16 surrogate pairs, and this must be supported. Any strings you get from Internet Explorer will be encoded as UTF-16 surrogate pairs regardless of content, presumably since Windows uses 16 bit wide chars for unicode.
All good points. Proceed with implementation! :o) -- Andrei
Aug 05 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 5 August 2014 at 17:58:08 UTC, Andrei Alexandrescu 
wrote:
 All good points. Proceed with implementation! :o) -- Andrei
Any news about std.allocator ? ;)
Aug 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/5/14, 10:58 AM, Dicebot wrote:
 On Tuesday, 5 August 2014 at 17:58:08 UTC, Andrei Alexandrescu wrote:
 All good points. Proceed with implementation! :o) -- Andrei
Any news about std.allocator ? ;)
It looks like I need to go all out and write a garbage collector, design and implementation and all. Andrei
Aug 05 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 5 August 2014 at 18:12:54 UTC, Andrei Alexandrescu 
wrote:
 On 8/5/14, 10:58 AM, Dicebot wrote:
 On Tuesday, 5 August 2014 at 17:58:08 UTC, Andrei Alexandrescu 
 wrote:
 All good points. Proceed with implementation! :o) -- Andrei
Any news about std.allocator ? ;)
It looks like I need to go all out and write a garbage collector, design and implementation and all.
A few months ago, you posted a video of a talk where you presented code from a garbage collector (it used templated mark functions to get precise tracing). I remember you said that this code was in use somewhere (I guess at FB?). Can this be used as a basis?
Aug 05 2014
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Aug 05, 2014 at 10:58:08AM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 8/5/14, 10:48 AM, Sean Kelly wrote:
[...]
The original point of JSON was that it auto-converts to
Javascript data.  And since Javascript only has one numeric type,
of course JSON does too.  But I think it's important that a JSON
package for a language maps naturally to the types available in
that language.  D provides both floating point and integer types,
each with their own costs and benefits, and so the JSON package
should as well.  It ends up being a lot easier to deal with than
remembering to round from JSON.number or whatever when assigning
to an int.

In fact, JSON doesn't even impose any precision restrictions on
its numeric type, so one could argue that we should be using
BigInt and BigFloat.  But this would stink most of the time, so...
Would it make sense to wrap a JSON number in an opaque type that implicitly casts to the target built-in type?
On an unrelated note, while the default encoding for strings is
UTF-8, the RFC absolutely allows for UTF-16 surrogate pairs, and
this must be supported.  Any strings you get from Internet
Explorer will be encoded as UTF-16 surrogate pairs regardless of
content, presumably since Windows uses 16 bit wide chars for
unicode.
[...] Wait, I thought surrogate pairs only apply to characters past U+FFFF? Is it even possible to encode BMP characters with surrogate pairs?? Or do you mean UTF-16? T -- Music critic: "That's an imitation fugue!"
Aug 05 2014
parent "Andrea Fontana" <nospam example.com> writes:
On Tuesday, 5 August 2014 at 18:11:21 UTC, H. S. Teoh via 
Digitalmars-d wrote:
 On Tue, Aug 05, 2014 at 10:58:08AM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:
 On 8/5/14, 10:48 AM, Sean Kelly wrote:
[...]
 Would it make sense to wrap a JSON number in an opaque type that
 implicitly casts to the target built-in type?
IMO we should store original json number value as string and then try to convert to what user asks for. As said, it could be a big int, or a big floating point value without any limit.
Aug 06 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-05 14:40, Daniel Murphy wrote:

 I guess I meant types that have an obvious mapping to json types.

 int/long -> json integer
 bool -> json bool
 string -> json string
 float/real -> json float (close enough)
 T[] -> json array
 T[string] -> json object
 struct -> json object

 This is usually enough for config and data files.  Being able to do this
 is just awesome:

 struct AppConfig
 {
     string somePath;
     bool someOption;
     string[] someList;
     string[string] someMap;
 }

 void main()
 {
     auto config =
 "config.json".readText().parseJSON().fromJson!AppConfig();
 }
I'm not saying that is a bad idea or that I don't want to be able to do this. I just prefer this to be handled by a generic serialization module. Which can of course handle the simple cases, like above, as well. -- /Jacob Carlborg
Aug 05 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Jacob Carlborg"  wrote in message news:lrqvfa$2has$1 digitalmars.com...

 I'm not saying that is a bad idea or that I don't want to be able to do 
 this. I just prefer this to be handled by a generic serialization module. 
 Which can of course handle the simple cases, like above, as well.
I know, but I don't really care if it's part of a generic serialization library or not. I just want it there. Chances are tying it to a future generic serialization library is going to make it take longer.
Aug 05 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-05 18:42, Daniel Murphy wrote:

 Chances are tying it to a future
 generic serialization library is going to make it take longer.
Yeah, that's the problem. But where do you draw the line. Should arrays of structs be supported? -- /Jacob Carlborg
Aug 06 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Jacob Carlborg"  wrote in message news:lrsrek$19mf$1 digitalmars.com...

 Chances are tying it to a future
 generic serialization library is going to make it take longer.
Yeah, that's the problem. But where do you draw the line. Should arrays of structs be supported?
Yes. Allow T, where T is any of int, float, long, etc bool struct { T... } T[string] T[] Sure, you _can_ make a struct containing an array that contains itself, but you probably won't.
Aug 06 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-06 13:36, Daniel Murphy wrote:

 Yes.  Allow T, where T is any of

 int, float, long, etc
 bool
 struct { T... }
 T[string]
 T[]
BTW, why not classes? It's basically the same implementation as for structs. -- /Jacob Carlborg
Aug 06 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Jacob Carlborg"  wrote in message news:lrtf8l$22d3$1 digitalmars.com...

 BTW, why not classes? It's basically the same implementation as for 
 structs.
I guess I've just never needed to do it with classes. A lot of the time when I use classes I use inheritance, and this simple translation doesn't work out so will then...
Aug 06 2014
parent "Sean Kelly" <sean invisibleduck.org> writes:
On Wednesday, 6 August 2014 at 15:28:06 UTC, Daniel Murphy wrote:
 "Jacob Carlborg"  wrote in message 
 news:lrtf8l$22d3$1 digitalmars.com...

 BTW, why not classes? It's basically the same implementation 
 as for structs.
I guess I've just never needed to do it with classes. A lot of the time when I use classes I use inheritance, and this simple translation doesn't work out so will then...
We could do something like Jackson. I wouldn't want it as the primary interface for a JSON package, but for serializing classes it's a pretty easy design to work with from a user perspective.
Aug 06 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 4 August 2014 at 07:34:19 UTC, Jacob Carlborg wrote:
 On Sunday, 3 August 2014 at 18:44:37 UTC, Dicebot wrote:

 Before going this route one needs to have a good vision how it 
 may interact with imaginary std.serialization to avoid later 
 deprecation.
I suggest only provide functions for serializing primitive types. A separate serialization module/package with a JSON archive type would use this module as its backend.
Do you consider structs primitive types? This is probably #1 use case for JSON conversion.
 At the same time I have recently started to think that 
 dedicated serialization module that decouples aggregate 
 iteration from data storage format is in most cases 
 impractical for performance reasons - different serialization 
 methods imply very different efficient iteration strategies. 
 Probably it is better to define serialization compile-time 
 traits instead and require each `std.data.*` provider to 
 implement those on its own in the most effective fashion.
I'm not sure I agree with that. In my work on std.serialization I have not seen this to be a problem. What problems have you found?
http://forum.dlang.org/post/mzweposldwqdtmqoltiy forum.dlang.org
Aug 04 2014
parent reply "Jacob Carlborg" <doob me.com> writes:
On Monday, 4 August 2014 at 14:02:22 UTC, Dicebot wrote:

 Do you consider structs primitive types? This is probably #1 
 use case for JSON conversion.
No, only types that cannot be broken down in to smaller pieces, i.e. integral, floating points, bool and strings.
 http://forum.dlang.org/post/mzweposldwqdtmqoltiy forum.dlang.org
I don't understand exactly how that binary serialization works. I think I would need a code example. -- /Jacob Carlborg
Aug 04 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 4 August 2014 at 14:18:41 UTC, Jacob Carlborg wrote:
 On Monday, 4 August 2014 at 14:02:22 UTC, Dicebot wrote:

 Do you consider structs primitive types? This is probably #1 
 use case for JSON conversion.
No, only types that cannot be broken down in to smaller pieces, i.e. integral, floating points, bool and strings.
That is exactly the problem - if `structToJson` won't be provided, complaints are inevitable, it is too basic feature to wait for std.serialization :(
 http://forum.dlang.org/post/mzweposldwqdtmqoltiy forum.dlang.org
I don't understand exactly how that binary serialization works. I think I would need a code example.
Simplified serialization algorithm: 1) write (cast(void*) &struct)[0..struct.sizeof] to target buffer 2) write any of array content to the same buffer after the struct 3.1) if array contains structs, recursion 3.2) go back to buffer[0..struct.sizeof] slice and update array fields to store an index in the same buffer instead of actual ptr Simplified deserialization algorithm: 1) recursively traverse the struct and replace array index offsets with real slices to the buffer (I don't want to bother with getting copyright permissions to publish actual code) I am pretty sure that this is not the only optimized serialization approach out there that does not fit in a content-insensitive primitive-based traversal scheme. And we won't Phobos stuff to be blazingly fast which can lead to situation where new data module will circumvent the std.serialization API to get more performance.
Aug 04 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-04 16:55, Dicebot wrote:

 That is exactly the problem - if `structToJson` won't be provided,
 complaints are inevitable, it is too basic feature to wait for
 std.serialization :(
Hmm, yeah, that's a problem.
 Simplified serialization algorithm:

 1) write (cast(void*) &struct)[0..struct.sizeof] to target buffer
 2) write any of array content to the same buffer after the struct
 3.1) if array contains structs, recursion
 3.2) go back to buffer[0..struct.sizeof] slice and update array fields
 to store an index in the same buffer instead of actual ptr

 Simplified deserialization algorithm:

 1) recursively traverse the struct and replace array index offsets with
 real slices to the buffer
I see. I need to think a bit about this.
 (I don't want to bother with getting copyright permissions to publish
 actual code)
Fair enough. The above was quite descriptive.
 I am pretty sure that this is not the only optimized serialization
 approach out there that does not fit in a content-insensitive
 primitive-based traversal scheme. And we won't Phobos stuff to be
 blazingly fast which can lead to situation where new data module will
 circumvent the std.serialization API to get more performance.
I don't like the idea of having to reimplement serialization for each data type that can be generalized. -- /Jacob Carlborg
Aug 04 2014
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 04.08.2014 20:38, schrieb Jacob Carlborg:
 On 2014-08-04 16:55, Dicebot wrote:

 That is exactly the problem - if `structToJson` won't be provided,
 complaints are inevitable, it is too basic feature to wait for
 std.serialization :(
Hmm, yeah, that's a problem.
On the other hand, a simplistic solution will inevitably result in people needing more. And when at some point a serialization module is in Phobos, there will be duplicate functionality in the library.
 I am pretty sure that this is not the only optimized serialization
 approach out there that does not fit in a content-insensitive
 primitive-based traversal scheme. And we won't Phobos stuff to be
 blazingly fast which can lead to situation where new data module will
 circumvent the std.serialization API to get more performance.
I don't like the idea of having to reimplement serialization for each data type that can be generalized.
I think we could also simply keep the generic default recursive descent behavior, but allow serializers to customize the process using some kind of trait. This could even be added later in a backwards compatible fashion if necessary. BTW, how is the progress for Orange w.r.t. to the conversion to a more template+allocation-less approach, is a new std proposal within the next DMD release cycle realistic? I quite like most of how vibe.data.serialization turned out, but it can't do any alias detection/deduplication (and I have no concrete plans to add support for that), which is why I currently wouldn't consider it as a potential Phobos candidate.
Aug 05 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 5 August 2014 at 09:54:42 UTC, Sönke Ludwig wrote:
 I think we could also simply keep the generic default recursive 
 descent behavior, but allow serializers to customize the 
 process using some kind of trait. This could even be added 
 later in a backwards compatible fashion if necessary.
Simple option is to define required serializer traits and make both std.serialization default and any custom data-specific ones conform it.
Aug 05 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-08-05 11:54, Sönke Ludwig wrote:

 I think we could also simply keep the generic default recursive descent
 behavior, but allow serializers to customize the process using some kind
 of trait. This could even be added later in a backwards compatible
 fashion if necessary.
I have a very flexible trait like system in place. This allows to configure the serializer based on the given archiver and user customizations. To avoid having the serializer do unnecessary work which the archiver cannot handle.
 BTW, how is the progress for Orange w.r.t. to the conversion to a more
 template+allocation-less approach
Slowly. I think the range support in the serializer is basically complete. But the deserializer isn't done yet. I would also like to provide, at least, one additional archiver type besides XML. BTW std.xml doesn't make it any easier to rangify the serializer. I've been focusing on D/Objective-C lately, which I think is in a more complete state than std.serialization. I would really like to get it done and create a pull request so I can get back to std.serialization. But I always get stuck after a merge with something breaking. With the summer and vacations I haven't been able to work that much on D at all. , is a new std proposal within the next
 DMD release cycle realistic?
Probably not.
 I quite like most of how vibe.data.serialization turned out, but it
 can't do any alias detection/deduplication (and I have no concrete plans
 to add support for that), which is why I currently wouldn't consider it
 as a potential Phobos candidate.
I'm quite satisfied with the feature support and flexibility of Orange/std.serialization. With the new trait like system it will be even more flexible. -- /Jacob Carlborg
Aug 05 2014
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 * The representation is built on Algebraic,
Good. But here I'd like a little more readable type: alias Payload = std.variant.VariantN!(16LU, typeof(null), bool, double, string, Value[], Value[string]).VariantN; Like: alias Payload = std.variant.Algebraic!(typeof(null), bool, double, string, Value[], Value[string]); Bye, bearophile
Aug 03 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 1:19 AM, bearophile wrote:
 Andrei Alexandrescu:

 * The representation is built on Algebraic,
Good. But here I'd like a little more readable type: alias Payload = std.variant.VariantN!(16LU, typeof(null), bool, double, string, Value[], Value[string]).VariantN; Like: alias Payload = std.variant.Algebraic!(typeof(null), bool, double, string, Value[], Value[string]);
Yah, the latter is in the code. It's a ddoc problem. -- Andrei
Aug 03 2014
prev sibling next sibling parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
A few thoughts based on my experience with vibe.data.json:

1. No decoding of strings appears to mean that "Value" also always 
contains encoded strings. This seems the be a leaky and also error prone 
leaky abstraction. For the token stream, performance should be top 
priority, so it's okay to not decode there, but "Value" is a high level 
abstraction of a JSON value, so it should really hide all implementation 
details of the storage format.

2. Algebraic is a good choice for its generic handling of operations on 
the contained types (which isn't exposed here, though). However, a 
tagged union type in my experience has quite some advantages for 
usability. Since adding a type tag possibly affects the interface in a 
non-backwards compatible way, this should be evaluated early on.

2.b) I'm currently working on a generic tagged union type that also 
enables operations between values in a natural generic way. This has the 
big advantage of not having to manually define operators like in 
"Value", which is error prone and often limited (I've had to make many 
fixes and additions in this part of the code over time).

3. Use of "opDispatch" for an open set of members has been criticized 
for vibe.data.json before and I agree with that criticism. The only 
advantage is saving a few keystrokes (json.key instead of json["key"]), 
but I came to the conclusion that the right approach to work with JSON 
values in D is to always directly deserialize when/if possible anyway, 
which mostly makes this is a moot point.

This approach has a lot of advantages, e.g. reduction of allocations, 
performance of field access and avoiding typos when accessing fields. 
Especially the last point is interesting, because opDispatch based field 
access gives the false impression that a static field is accessed.

The decision to minimize the number of static fields within "Value" 
reduces the chance of accidentally accessing a static field instead of 
hitting opDispatch, but there are still *some* static fields/methods and 
any later addition of a symbol must now be considered a breaking change.

3.b) Bad interaction of UFCS and opDispatch: Functions like "remove" and 
"assume" certainly look like they could be used with UFCS, but 
opDispatch destroys that possibility.

4. I know the stance on this is often "The D module system has enough 
facilities to disambiguate" (which is not really a valid argument, but 
rather just the lack of a counter argument, IMO), but I highly dislike 
the choice to leave off any mention of "JSON" or "Json" in the global 
symbol names. Using the module either requires to always use a renamed 
import or a manual alias, or the resulting source code will always leave 
the reader wondering what kind of data is actually handled. Handling 
multiple "value" types in a single piece of code, which is not uncommon 
(e.g. JSON + BSON/ini value/...) would always require explicit 
disambiguation. I'd certainly include the "JSON" or "Json" part in the 
names.

5. Whatever happens, *please* let's aim for a module name of 
std.data.json (similar to std.digest.*), so that any data formats added 
later are nicely organized. All existing data format support (XML + CSV) 
doesn't follow contemporary Phobos style, so they will need to be 
deprecated at some point anyway, freeing the way for a clean an 
non-breaking transition to a more organized module hierarchy.

6. (Possibly compile time optional) support for keeping track of 
line/column numbers is often important for better error messages, so 
that would be good to have included as part of the parser and in the 
"Token" type.

Snke
Aug 03 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 2:38 AM, Snke Ludwig wrote:
 A few thoughts based on my experience with vibe.data.json:

 1. No decoding of strings appears to mean that "Value" also always
 contains encoded strings. This seems the be a leaky and also error prone
 leaky abstraction. For the token stream, performance should be top
 priority, so it's okay to not decode there, but "Value" is a high level
 abstraction of a JSON value, so it should really hide all implementation
 details of the storage format.
Nonono. I think there's a confusion. The input strings are not UTF decoded for the simple need there's no need (all tokenization decisions are taken on the basis of ASCII characters/code units). The backslash-prefixed characters are indeed decoded. An optimization I didn't implement yet is to use slices of the input wherever possible (when the input is string, immutable(byte)[], or immutable(ubyte)[]). That will reduce allocations considerably.
 2. Algebraic is a good choice for its generic handling of operations on
 the contained types (which isn't exposed here, though). However, a
 tagged union type in my experience has quite some advantages for
 usability. Since adding a type tag possibly affects the interface in a
 non-backwards compatible way, this should be evaluated early on.
There's a public opCast(Payload) that gives the end user access to the Payload inside a Value. I forgot to add documentation to it. What advantages are to a tagged union? (FWIW: to me Algebraic and Variant are also tagged unions, just that the tags are not 0, 1, ..., n. That can be easily fixed for Algebraic by defining operations to access the index of the currently-stored type.)
 2.b) I'm currently working on a generic tagged union type that also
 enables operations between values in a natural generic way. This has the
 big advantage of not having to manually define operators like in
 "Value", which is error prone and often limited (I've had to make many
 fixes and additions in this part of the code over time).
I did notice that vibe.json has quite a repetitive implementation, so reducing it would be great. The way I see it, good work on tagged unions must be either integrated within std.variant (either by modifying Variant/Algebraic or by adding new types to it). I am very strongly opposed to adding a tagged union type only for JSON purposes, which I'd consider essentially a usability bug in std.variant, the opposite of dogfooding, etc.
 3. Use of "opDispatch" for an open set of members has been criticized
 for vibe.data.json before and I agree with that criticism. The only
 advantage is saving a few keystrokes (json.key instead of json["key"]),
 but I came to the conclusion that the right approach to work with JSON
 values in D is to always directly deserialize when/if possible anyway,
 which mostly makes this is a moot point.
Interesting. Well if experience with opDispatch is negative then it should probably not be used here, or only offered on an opt-in basis.
 This approach has a lot of advantages, e.g. reduction of allocations,
 performance of field access and avoiding typos when accessing fields.
 Especially the last point is interesting, because opDispatch based field
 access gives the false impression that a static field is accessed.
Good point.
 The decision to minimize the number of static fields within "Value"
 reduces the chance of accidentally accessing a static field instead of
 hitting opDispatch, but there are still *some* static fields/methods and
 any later addition of a symbol must now be considered a breaking change.
Right now the idea is that the only named member is __payload. Well then there's opXxxx as well. The idea is/was to add all other functionality as free functions.
 3.b) Bad interaction of UFCS and opDispatch: Functions like "remove" and
 "assume" certainly look like they could be used with UFCS, but
 opDispatch destroys that possibility.
Yah, agreed. The bummer is people coming from Python won't be able to continue using the same style without opDispatch.
 4. I know the stance on this is often "The D module system has enough
 facilities to disambiguate" (which is not really a valid argument, but
 rather just the lack of a counter argument, IMO), but I highly dislike
 the choice to leave off any mention of "JSON" or "Json" in the global
 symbol names. Using the module either requires to always use a renamed
 import or a manual alias, or the resulting source code will always leave
 the reader wondering what kind of data is actually handled. Handling
 multiple "value" types in a single piece of code, which is not uncommon
 (e.g. JSON + BSON/ini value/...) would always require explicit
 disambiguation. I'd certainly include the "JSON" or "Json" part in the
 names.
Good point, I agree.
 5. Whatever happens, *please* let's aim for a module name of
 std.data.json (similar to std.digest.*), so that any data formats added
 later are nicely organized. All existing data format support (XML + CSV)
 doesn't follow contemporary Phobos style, so they will need to be
 deprecated at some point anyway, freeing the way for a clean an
 non-breaking transition to a more organized module hierarchy.
I agree.
 6. (Possibly compile time optional) support for keeping track of
 line/column numbers is often important for better error messages, so
 that would be good to have included as part of the parser and in the
 "Token" type.
Yah, saw that in vibe.d but forgot about it. Thanks, Andrei
Aug 03 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 3 August 2014 at 15:14:43 UTC, Andrei Alexandrescu 
wrote:
 3. Use of "opDispatch" for an open set of members has been 
 criticized
 for vibe.data.json before and I agree with that criticism. The 
 only
 advantage is saving a few keystrokes (json.key instead of 
 json["key"]),
 but I came to the conclusion that the right approach to work 
 with JSON
 values in D is to always directly deserialize when/if possible 
 anyway,
 which mostly makes this is a moot point.
Interesting. Well if experience with opDispatch is negative then it should probably not be used here, or only offered on an opt-in basis.
I support this opinion. opDispatch looks cool with JSON objects when you implement it but it results in many subtle quirks when you consider something like range traits for example - most annoying to encounter and debug. It is not worth the gain.
Aug 03 2014
prev sibling next sibling parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
Am 03.08.2014 17:14, schrieb Andrei Alexandrescu:
 On 8/3/14, 2:38 AM, Snke Ludwig wrote:
 A few thoughts based on my experience with vibe.data.json:

 1. No decoding of strings appears to mean that "Value" also always
 contains encoded strings. This seems the be a leaky and also error prone
 leaky abstraction. For the token stream, performance should be top
 priority, so it's okay to not decode there, but "Value" is a high level
 abstraction of a JSON value, so it should really hide all implementation
 details of the storage format.
Nonono. I think there's a confusion. The input strings are not UTF decoded for the simple need there's no need (all tokenization decisions are taken on the basis of ASCII characters/code units). The backslash-prefixed characters are indeed decoded. An optimization I didn't implement yet is to use slices of the input wherever possible (when the input is string, immutable(byte)[], or immutable(ubyte)[]). That will reduce allocations considerably.
Ah okay, *phew* ;) But in that case I'd actually think about leaving off the backslash decoding in the low level parser, so that slices could be used for immutable inputs in all cases - maybe with a name of "rawString" for the stored data and an additional "string" property that decodes on the fly. This may come in handy when the first comparative benchmarks together with rapidjson and the like are done.
 2. Algebraic is a good choice for its generic handling of operations on
 the contained types (which isn't exposed here, though). However, a
 tagged union type in my experience has quite some advantages for
 usability. Since adding a type tag possibly affects the interface in a
 non-backwards compatible way, this should be evaluated early on.
There's a public opCast(Payload) that gives the end user access to the Payload inside a Value. I forgot to add documentation to it.
I see. Suppose that opDispatch would be dropped, would anything speak against "alias this"ing _payload to avoid the need for the manually defined operators?
 What advantages are to a tagged union? (FWIW: to me Algebraic and
 Variant are also tagged unions, just that the tags are not 0, 1, ..., n.
 That can be easily fixed for Algebraic by defining operations to access
 the index of the currently-stored type.)
The two major points are probably that it's possible to use "final switch" on the type tag if it's an enum, and the type id can be easily stored in both integer and string form (which is not as conveniently possible with a TypeInfo).
 (...)

 The way I see it, good work on tagged unions must be either integrated
 within std.variant (either by modifying Variant/Algebraic or by adding
 new types to it). I am very strongly opposed to adding a tagged union
 type only for JSON purposes, which I'd consider essentially a usability
 bug in std.variant, the opposite of dogfooding, etc.
Definitely agree there. An enum based tagged union design also currently has the unfortunate property that the order of enum values and that of the accepted types must be defined consistently, or bad things will happen. Supporting UDAs on enum values would be a possible direction to fix this: enum JsonType { variantType!string string, variantType!(JsonValue[]) array, variantType!(JsonValue[string]) object } alias JsonValue = TaggedUnion!JsonType; But then there are obviously still issues with cyclic type references. So, anyway, this is something that still requires some thought. It could also be designed in a way that is backwards compatible with a pure "Algebraic", so it shouldn't be a blocker for the current design.
Aug 03 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 11:03 AM, Snke Ludwig wrote:
 Am 03.08.2014 17:14, schrieb Andrei Alexandrescu:
[snip]
 Ah okay, *phew* ;) But in that case I'd actually think about leaving off
 the backslash decoding in the low level parser, so that slices could be
 used for immutable inputs in all cases - maybe with a name of
 "rawString" for the stored data and an additional "string" property that
 decodes on the fly. This may come in handy when the first comparative
 benchmarks together with rapidjson and the like are done.
Yah, that's awesome.
 There's a public opCast(Payload) that gives the end user access to the
 Payload inside a Value. I forgot to add documentation to it.
I see. Suppose that opDispatch would be dropped, would anything speak against "alias this"ing _payload to avoid the need for the manually defined operators?
Correct. In fact the conversion was there but I removed it for the sake of opDispatch.
 What advantages are to a tagged union? (FWIW: to me Algebraic and
 Variant are also tagged unions, just that the tags are not 0, 1, ..., n.
 That can be easily fixed for Algebraic by defining operations to access
 the index of the currently-stored type.)
The two major points are probably that it's possible to use "final switch" on the type tag if it's an enum,
So I just tried this: http://dpaste.dzfl.pl/eeadac68fac0. Sadly, the cast doesn't take. Without the cast the enum does compile, but not the switch. I submitted https://issues.dlang.org/show_bug.cgi?id=13247.
 and the type id can be easily stored in both integer and string form
 (which is not as conveniently possible with a TypeInfo).
I think here pointers to functions "win" because getting a string (or anything else for that matter) is an indirect call away. std.variant has been among the first artifacts I wrote for D. It's a topic I've been dabbling in for a long time in a C++ context (http://goo.gl/zqUwFx), with always almost-satisfactory results. I told myself if I get to implement things in D properly, then this language has good potential. Replacing the integral tag I'd always used with a pointer to function is, I think, net progress. Things turned out fine, save for the switch matter.
 An enum based tagged union design also currently has the unfortunate
 property that the order of enum values and that of the accepted types
 must be defined consistently, or bad things will happen. Supporting UDAs
 on enum values would be a possible direction to fix this:

      enum JsonType {
           variantType!string string,
           variantType!(JsonValue[]) array,
           variantType!(JsonValue[string]) object
      }
      alias JsonValue = TaggedUnion!JsonType;

 But then there are obviously still issues with cyclic type references.
 So, anyway, this is something that still requires some thought. It could
 also be designed in a way that is backwards compatible with a pure
 "Algebraic", so it shouldn't be a blocker for the current design.
I think something can be designed along these lines if necessary. Andrei
Aug 03 2014
prev sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Sunday, 3 August 2014 at 15:14:43 UTC, Andrei Alexandrescu 
wrote:
 On 8/3/14, 2:38 AM, Sönke Ludwig wrote:
 3. Use of "opDispatch" for an open set of members has been
 criticized for vibe.data.json before and I agree with that
 criticism. The only advantage is saving a few keystrokes
 (json.key instead of json["key"]), but I came to the conclusion
 that the right approach to work with JSON values in D is to
 always directly deserialize when/if possible anyway, which
 mostly makes this is a moot point.
Interesting. Well if experience with opDispatch is negative then it should probably not be used here, or only offered on an opt-in basis.
I suspect that depends on the circumstances. I've been using this style (with Adam's jsvar), and I find it quite nice for decomposing my TOML parse trees to Variant-like structures that go several levels deep. It makes reading (and, consequently, reasoning about) them much easier for me. That said, I think the ideal would be that nesting Variant[] should work predictably such that users can just write a one-line opDispatch if they want it to behave that way. -Wyatt
Aug 04 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 2:38 AM, Snke Ludwig wrote:
[snip]

We need to address the matter of std.jgrandson competing with
vibe.data.json. Clearly at a point only one proposal will have to be
accepted so the other would be wasted work.

Following our email exchange I decided to work on this because (a) you
mentioned more work is needed and your schedule was unclear, (b) we need
this at FB sooner rather than later, (c) there were a few things I
thought can be improved in vibe.data.json. I hope that taking
std.jgrandson to proof spurs things into action.

Would you want to merge some of std.jgrandson's deltas into a new
proposal std.data.json based on vibe.data.json? Here's a few things that
I consider necessary:

1. Commit to a schedule. I can't abandon stuff in wait for the perfect 
design that may or may not come someday.

2. Avoid UTF decoding.

3. Offer a lazy token stream as a basis for a non-lazy parser. A lazy 
general parser would be considerably more difficult to write and would 
only serve a small niche. On the other hand, a lazy tokenizer is easy to 
write and make efficient, and serve as a basis for user-defined 
specialized lazy parsers if the user wants so.

4. Avoid string allocation. String allocation can be replaced with 
slices of the input when these two conditions are true: (a) input type 
is string, immutable(byte)[], or immutable(ubyte)[]; (b) there are no 
backslash-encoded sequences in the string, i.e. the input string and the 
actual string are the same.

5. Build on std.variant through and through. Again, anything that 
doesn't work is a usability bug in std.variant, which was designed for 
exactly this kind of stuff. Exposing the representation such that user 
code benefits of the Algebraic's primitives may be desirable.

6. Address w0rp's issue with undefined. In fact std.Algebraic does have 
an uninitialized state :o).

Snke, what do you think?


Andrei
Aug 03 2014
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 03 Aug 2014 08:34:20 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 8/3/14, 2:38 AM, S=C3=B6nke Ludwig wrote:
 [snip]
=20
 We need to address the matter of std.jgrandson competing with
 vibe.data.json. Clearly at a point only one proposal will have to be
 accepted so the other would be wasted work.
=20
 [...]
=20
 4. Avoid string allocation. String allocation can be replaced with=20
 slices of the input when these two conditions are true: (a) input
 type is string, immutable(byte)[], or immutable(ubyte)[]; (b) there
 are no backslash-encoded sequences in the string, i.e. the input
 string and the actual string are the same.
I think for the lowest level interface we could avoid allocation completely: The tokenizer could always return slices to the raw string, even if a string contains backslash-encode sequences or if the token is a number. Simply expose that as token.rawValue. Then add a function, Token.decodeString() and token.decodeNumber() to actually decode the numbers. decodeString could additionally support decoding into a buffer. If the input is not sliceable, read the input into an internal buffer first and slice that buffer. The main usecase for this is if you simply stream lots of data and you only want to parse very little of it and skip over most content. Then you don't need to decode the strings. This is also true if you only write a JSON formatter: No need to decode and encode the strings.
=20
 5. Build on std.variant through and through. Again, anything that=20
 doesn't work is a usability bug in std.variant, which was designed
 for exactly this kind of stuff. Exposing the representation such that
 user code benefits of the Algebraic's primitives may be desirable.
=20
Variant uses TypeInfo internally, right? I think as long as it uses TypeInfo it can't replace all use-cases for a standard tagged union.
Aug 03 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 8:51 AM, Johannes Pfau wrote:
 Am Sun, 03 Aug 2014 08:34:20 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 8/3/14, 2:38 AM, Sönke Ludwig wrote:
 [snip]

 We need to address the matter of std.jgrandson competing with
 vibe.data.json. Clearly at a point only one proposal will have to be
 accepted so the other would be wasted work.

 [...]

 4. Avoid string allocation. String allocation can be replaced with
 slices of the input when these two conditions are true: (a) input
 type is string, immutable(byte)[], or immutable(ubyte)[]; (b) there
 are no backslash-encoded sequences in the string, i.e. the input
 string and the actual string are the same.
I think for the lowest level interface we could avoid allocation completely: The tokenizer could always return slices to the raw string, even if a string contains backslash-encode sequences or if the token is a number. Simply expose that as token.rawValue. Then add a function, Token.decodeString() and token.decodeNumber() to actually decode the numbers. decodeString could additionally support decoding into a buffer.
That works but not e.g. for File.byLine which reuses its internal buffer. But it's a neat idea for arrays of immutable bytes.
 If the input is not sliceable, read the input into an internal buffer
 first and slice that buffer.
At that point the cost of decoding becomes negligible.
 The main usecase for this is if you simply stream lots of data and you
 only want to parse very little of it and skip over most content. Then
 you don't need to decode the strings.
Awesome.
 This is also true if you only
 write a JSON formatter: No need to decode and encode the strings.
But wouldn't that still need to encode \n, \r, \t, \v?
 5. Build on std.variant through and through. Again, anything that
 doesn't work is a usability bug in std.variant, which was designed
 for exactly this kind of stuff. Exposing the representation such that
 user code benefits of the Algebraic's primitives may be desirable.
Variant uses TypeInfo internally, right?
No. Andrei
Aug 03 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 03 Aug 2014 09:17:57 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 8/3/14, 8:51 AM, Johannes Pfau wrote:
 Variant uses TypeInfo internally, right?
No.
https://github.com/D-Programming-Language/phobos/blob/master/std/variant.d#L210 https://github.com/D-Programming-Language/phobos/blob/master/std/variant.d#L371 https://github.com/D-Programming-Language/phobos/blob/master/std/variant.d#L696 Also the handler function concept will always have more overhead than a simple tagged union. It is certainly useful if you want to store any type, but if you only want a limited set of types there are more efficient implementations.
Aug 03 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 11:08 AM, Johannes Pfau wrote:
 Am Sun, 03 Aug 2014 09:17:57 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 8/3/14, 8:51 AM, Johannes Pfau wrote:
 Variant uses TypeInfo internally, right?
No.
https://github.com/D-Programming-Language/phobos/blob/master/std/variant.d#L210
That's a query for the TypeInfo.
 https://github.com/D-Programming-Language/phobos/blob/master/std/variant.d#L371
That could be translated to a comparison of pointers to functions.
 https://github.com/D-Programming-Language/phobos/blob/master/std/variant.d#L696
That, too, could be translated to a comparison of pointers to functions. It's a confision Let me clarify this. What Variant does is to use pointers to functions instead of integers. The space overhead (one word) is generally the same due to alignment issues.
 Also the handler function concept will always have more overhead than a
 simple tagged union. It is certainly useful if you want to store any
 type, but if you only want a limited set of types there are more
 efficient implementations.
I'm not sure at all actually. The way I see it a pointer to a function offers most everything an integer does, plus universal functionality by actually calling the function. What it doesn't offer is ordering of small integers, but that can be easily arranged at a small cost. Andrei
Aug 03 2014
prev sibling parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
Am 03.08.2014 17:34, schrieb Andrei Alexandrescu:
 On 8/3/14, 2:38 AM, Snke Ludwig wrote:
 [snip]

 We need to address the matter of std.jgrandson competing with
 vibe.data.json. Clearly at a point only one proposal will have to be
 accepted so the other would be wasted work.

 Following our email exchange I decided to work on this because (a) you
 mentioned more work is needed and your schedule was unclear, (b) we need
 this at FB sooner rather than later, (c) there were a few things I
 thought can be improved in vibe.data.json. I hope that taking
 std.jgrandson to proof spurs things into action.

 Would you want to merge some of std.jgrandson's deltas into a new
 proposal std.data.json based on vibe.data.json? Here's a few things that
 I consider necessary:

 1. Commit to a schedule. I can't abandon stuff in wait for the perfect
 design that may or may not come someday.
This may be the crux w.r.t. the vibe.data.json implementation. My schedule will be very crowded this month, so I could only really start to work on it beginning of September. But apart from the mentioned points, I think your implementation is already the closest thing to what I have in mind, so I'm all for going the clean slate route (I'll have to do a lot in terms of deprecation work in vibe.d anyway).
 2. Avoid UTF decoding.

 3. Offer a lazy token stream as a basis for a non-lazy parser. A lazy
 general parser would be considerably more difficult to write and would
 only serve a small niche. On the other hand, a lazy tokenizer is easy to
 write and make efficient, and serve as a basis for user-defined
 specialized lazy parsers if the user wants so.

 4. Avoid string allocation. String allocation can be replaced with
 slices of the input when these two conditions are true: (a) input type
 is string, immutable(byte)[], or immutable(ubyte)[]; (b) there are no
 backslash-encoded sequences in the string, i.e. the input string and the
 actual string are the same.

 5. Build on std.variant through and through. Again, anything that
 doesn't work is a usability bug in std.variant, which was designed for
 exactly this kind of stuff. Exposing the representation such that user
 code benefits of the Algebraic's primitives may be desirable.

 6. Address w0rp's issue with undefined. In fact std.Algebraic does have
 an uninitialized state :o).

 Snke, what do you think?
My requirements would be the same, except for 6. The "undefined" state in the vibe.d version was necessary due to early API decisions and it's more or less a prominent part of it (specifically because the API was designed to behave similar to JavaScript). In hindsight, I'd definitely avoid that. However, I don't think its existence (also in the form of Algebraic.init) is an issue per se, as long as such values are properly handled when converting the runtime value back to a JSON string (i.e. skipped or treated as null values).
Aug 03 2014
next sibling parent reply "w0rp" <devw0rp gmail.com> writes:
On Sunday, 3 August 2014 at 18:37:48 UTC, Sönke Ludwig wrote:
 Am 03.08.2014 17:34, schrieb Andrei Alexandrescu:
 6. Address w0rp's issue with undefined. In fact std.Algebraic 
 does have
 an uninitialized state :o).
My requirements would be the same, except for 6. The "undefined" state in the vibe.d version was necessary due to early API decisions and it's more or less a prominent part of it (specifically because the API was designed to behave similar to JavaScript). In hindsight, I'd definitely avoid that. However, I don't think its existence (also in the form of Algebraic.init) is an issue per se, as long as such values are properly handled when converting the runtime value back to a JSON string (i.e. skipped or treated as null values).
My issue with is is that if you ask for a key in an object which doesn't exist, you get an 'undefined' value back, just like JavaScript. I'd rather that be propagated as a RangeError, which is more consistent with associative arrays in the language and probably more correct. A minor issue is being able to create a Json object which isn't a valid Json object by itself. I'd rather the initial value was just 'null', which would match how pointers and class instances behave in the language.
Aug 03 2014
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 03.08.2014 20:57, schrieb w0rp:
 On Sunday, 3 August 2014 at 18:37:48 UTC, Sönke Ludwig wrote:
 The "undefined" state in the vibe.d version was necessary due to early
 API decisions and it's more or less a prominent part of it
 (specifically because the API was designed to behave similar to
 JavaScript). In hindsight, I'd definitely avoid that. However, I don't
 think its existence (also in the form of Algebraic.init) is an issue
 per se, as long as such values are properly handled when converting
 the runtime value back to a JSON string (i.e. skipped or treated as
 null values).
My issue with is is that if you ask for a key in an object which doesn't exist, you get an 'undefined' value back, just like JavaScript. I'd rather that be propagated as a RangeError, which is more consistent with associative arrays in the language and probably more correct.
Yes, this is what I meant with the JavaScript part of API. In addition to opIndex(), there should of course also be a .get(key, default_value) style accessor and the "in" operator.
 A minor
 issue is being able to create a Json object which isn't a valid Json
 object by itself. I'd rather the initial value was just 'null', which
 would match how pointers and class instances behave in the language.
This is what I meant with not being an issue by itself. But having such a special value of course has its pros and cons, and I could personally definitely also live with JSON values being initialized to JSON "null", if somebody hacks Algebraic to support that kind of use case.
Aug 03 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 3 August 2014 at 19:54:12 UTC, Sönke Ludwig wrote:
 Am 03.08.2014 20:57, schrieb w0rp:
 My issue with is is that if you ask for a key in an object 
 which doesn't
 exist, you get an 'undefined' value back, just like 
 JavaScript. I'd
 rather that be propagated as a RangeError, which is more 
 consistent with
 associative arrays in the language and probably more correct.
Yes, this is what I meant with the JavaScript part of API. In addition to opIndex(), there should of course also be a .get(key, default_value) style accessor and the "in" operator.
There is a parallel discussion about the concept of associative ranges: http://forum.dlang.org/thread/jheurakujksdlrjaoncs forum.dlang.org Maybe you could also have a look there, because JSON seems to be a good candidate for an associative range.
Aug 04 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 11:37 AM, Snke Ludwig wrote:
 Am 03.08.2014 17:34, schrieb Andrei Alexandrescu:
 On 8/3/14, 2:38 AM, Snke Ludwig wrote:
 [snip]

 We need to address the matter of std.jgrandson competing with
 vibe.data.json. Clearly at a point only one proposal will have to be
 accepted so the other would be wasted work.

 Following our email exchange I decided to work on this because (a) you
 mentioned more work is needed and your schedule was unclear, (b) we need
 this at FB sooner rather than later, (c) there were a few things I
 thought can be improved in vibe.data.json. I hope that taking
 std.jgrandson to proof spurs things into action.

 Would you want to merge some of std.jgrandson's deltas into a new
 proposal std.data.json based on vibe.data.json? Here's a few things that
 I consider necessary:

 1. Commit to a schedule. I can't abandon stuff in wait for the perfect
 design that may or may not come someday.
This may be the crux w.r.t. the vibe.data.json implementation. My schedule will be very crowded this month, so I could only really start to work on it beginning of September. But apart from the mentioned points, I think your implementation is already the closest thing to what I have in mind, so I'm all for going the clean slate route (I'll have to do a lot in terms of deprecation work in vibe.d anyway).
What would be your estimated time of finishing? Would anyone want to take vibe.data.json and std.jgrandson, put them in a crucible, and have std.data.json emerge from it in a timely manner? My understanding is that everyone involved would be cool with that. Andrei
Aug 03 2014
parent =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
Am 03.08.2014 21:53, schrieb Andrei Alexandrescu:
 What would be your estimated time of finishing?
My rough estimate would be that about two weeks of calender time should suffice for a first candidate, since the functionality and the design is already mostly there. However, it seems that VariantN will need some work, too (currently using opAdd results in an error for an Algebraic defined for JSON usage).
Aug 05 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Sunday, 3 August 2014 at 07:16:05 UTC, Andrei Alexandrescu 
wrote:
 We need a better json library at Facebook. I'd discussed with 
 Sönke the possibility of taking vibe.d's json to std but he 
 said it needs some more work. So I took std.jgrandson to proof 
 of concept state and hence ready for destruction:

 http://erdani.com/d/jgrandson.d
 http://erdani.com/d/phobos-prerelease/std_jgrandson.html

 Here are a few differences compared to vibe.d's library. I 
 think these are desirable to have in that library as well:

 * Parsing strings is decoupled into tokenization (which is lazy 
 and only needs an input range) and parsing proper. Tokenization 
 is lazy, which allows users to create their own advanced (e.g. 
 partial/lazy) parsing if needed. The parser itself is eager.

 * There's no decoding of strings.

 * The representation is built on Algebraic, with the advantages 
 that it benefits from all of its primitives. Implementation is 
 also very compact because Algebraic obviates a bunch of 
 boilerplate. Subsequent improvements to Algebraic will also 
 reflect themselves into improvements to std.jgrandson.

 * The JSON value (called std.jgrandson.Value) has no named 
 member variables or methods except for __payload. This is so 
 there's no clash between dynamic properties exposed via 
 opDispatch.

 Well that's about it. What would it take for this to become a 
 Phobos proposal? Destroy.


 Andrei
I like it. Here's what I think about it. * When I wrote my JSON library, the thing I wanted most was constructors and opAssign functions for creating JSON values easily. JSON x = "some string"; You have this, so it's great. * You didn't include an 'undefined' value like vibe.d, which is a very minor detail, but something I dislike. This is good. * I'd just name Value either 'JSON' or 'JSONValue.' So you can just import the module without using aliases. * opDispatch is kind of "meh" for JSON objects. It works until you hit a name clash with a UFCS function. I don't mind typing the extra three characters. That's all I could think of really.
Aug 03 2014
prev sibling next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 03.08.2014 09:16, schrieb Andrei Alexandrescu:
 We need a better json library at Facebook. I'd discussed with Snke the
 possibility of taking vibe.d's json to std but he said it needs some
 more work. So I took std.jgrandson to proof of concept state and hence
 ready for destruction:

 http://erdani.com/d/jgrandson.d
 http://erdani.com/d/phobos-prerelease/std_jgrandson.html
Is the name supposed to stay or just a working title? "std.j*grandson*" (being the successor of "std.j*son*") is of course a funny play of words, but it's not really obvious on the first sight what it does. i.e. if someone skims the std. modules in the documentation, looking for json, he'd probably not think that this is the new json module. std.json2 or something like that would be more obvious. Cheers, Daniel
Aug 03 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 9:49 AM, Daniel Gibson wrote:
 Am 03.08.2014 09:16, schrieb Andrei Alexandrescu:
 We need a better json library at Facebook. I'd discussed with Snke the
 possibility of taking vibe.d's json to std but he said it needs some
 more work. So I took std.jgrandson to proof of concept state and hence
 ready for destruction:

 http://erdani.com/d/jgrandson.d
 http://erdani.com/d/phobos-prerelease/std_jgrandson.html
Is the name supposed to stay or just a working title?
Just a working title, but of course if it were wildly successful... but then again it's not. -- Andrei
Aug 03 2014
prev sibling next sibling parent reply "Sean Kelly" <sean invisibleduck.org> writes:
I don't want to pay for anything I don't use.  No allocations 
should occur within the parser and it should simply slice up the 
input.  So the lowest layer should allow me to iterate across 
symbols in some way.  When I've done this in the past it was 
SAX-style (ie. a callback per type) but with the range interface 
that shouldn't be necessary.

The parser shouldn't decode or convert anything unless I ask it 
to.  Most of the time I only care about specific values, and 
paying for conversions on everything is wasted process time.

I suggest splitting number into float and integer types.  In a 
language like D where these are distinct internal types, it can 
be valuable to know this up front.

Is there support for output?  I see the makeArray and makeObject 
routines...  Ideally, there should be a way to serialize JSON 
against an OutputRange with optional formatting.
Aug 03 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/3/14, 10:19 AM, Sean Kelly wrote:
 I don't want to pay for anything I don't use.  No allocations should
 occur within the parser and it should simply slice up the input.
What to do about arrays and objects, which would naturally allocate arrays and associative arrays respectively? What about strings with backslash-encoded characters? No allocation works for tokenization, but parsing is a whole different matter.
 So the
 lowest layer should allow me to iterate across symbols in some way.
Yah, that would be the tokenizer.
 When I've done this in the past it was SAX-style (ie. a callback per
 type) but with the range interface that shouldn't be necessary.

 The parser shouldn't decode or convert anything unless I ask it to.
 Most of the time I only care about specific values, and paying for
 conversions on everything is wasted process time.
That's tricky. Once you scan for 2 specific characters you may as well scan for a couple more, the added cost is negligible. In contrast, scanning once for finding termination and then again for decoding purposes will definitely be a lot more expensive.
 I suggest splitting number into float and integer types.  In a language
 like D where these are distinct internal
bfulifbucivrdfvhhjnrunrgultdjbjutypes, it can be valuable to
 know this up front.
Yah, that kept on sticking like a sore thumb throughout.
 Is there support for output?  I see the makeArray and makeObject
 routines...  Ideally, there should be a way to serialize JSON against an
 OutputRange with optional formatting.
Not yet, and yah those should be in. Andrei
Aug 03 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
03-Aug-2014 21:40, Andrei Alexandrescu пишет:
 On 8/3/14, 10:19 AM, Sean Kelly wrote:
 I don't want to pay for anything I don't use.  No allocations should
 occur within the parser and it should simply slice up the input.
What to do about arrays and objects, which would naturally allocate arrays and associative arrays respectively? What about strings with backslash-encoded characters?
SAX-style would imply that array is "parsed" by calling 6 user-defined callbacks inside of a parser: startArray, endArray, startObject, endObject, id and value. A simplified pseudo-code of JSON-parser inner loop is then: if(cur == '[') startArray(); else if(cur == '{'){ startObject(); else if(cur == '}') endObject(); else if(cur == ']') endArray(); else{ if(expectObjectKey){ id(parseAsIdentifier()); } else value(parseAsValue()); } This is as barebones as it can get and is very fast in practice esp. in context of searching/extracting/matching specific sub-tries of JSON documents. -- Dmitry Olshansky
Aug 03 2014
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
03-Aug-2014 23:54, Dmitry Olshansky пишет:
 03-Aug-2014 21:40, Andrei Alexandrescu пишет:
 A simplified pseudo-code of JSON-parser inner loop is then:

 if(cur == '[')
         startArray();
 else if(cur == '{'){
Aw. Stray brace.. -- Dmitry Olshansky
Aug 03 2014
prev sibling parent reply "Sean Kelly" <sean invisibleduck.org> writes:
On Sunday, 3 August 2014 at 17:40:48 UTC, Andrei Alexandrescu 
wrote:
 On 8/3/14, 10:19 AM, Sean Kelly wrote:
 I don't want to pay for anything I don't use.  No allocations 
 should
 occur within the parser and it should simply slice up the 
 input.
What to do about arrays and objects, which would naturally allocate arrays and associative arrays respectively? What about strings with backslash-encoded characters?
This is tricky with a range. With an event-based parser I'd have events for object and array begin / end, but with a range you end up having an element that's a token, which is pretty weird. For encoded characters (and you need to make sure you handle surrogate pairs in your decoder) I'd still provide some means of decoding on demand. If nothing else, decode lazily when the user asks for the string value. That way the user isn't paying to decode strings he isn't interested in.
 No allocation works for tokenization, but parsing is a whole 
 different matter.

 So the
 lowest layer should allow me to iterate across symbols in some 
 way.
Yah, that would be the tokenizer.
But that will halt on comma and colon and such, correct? That's a tad lower than I'd want, though I guess it would be easy enough to build a parser on top of it.
 When I've done this in the past it was SAX-style (ie. a 
 callback per
 type) but with the range interface that shouldn't be necessary.

 The parser shouldn't decode or convert anything unless I ask 
 it to.
 Most of the time I only care about specific values, and paying 
 for
 conversions on everything is wasted process time.
That's tricky. Once you scan for 2 specific characters you may as well scan for a couple more, the added cost is negligible. In contrast, scanning once for finding termination and then again for decoding purposes will definitely be a lot more expensive.
I think I'm getting a bit confused. For the JSON parser I wrote, the parser performs full validation but leaves the content as-is, then provides a routine to decode values from their string representation if the user wishes to. I'm not sure where scanning figures in here.
 Andrei
Aug 03 2014
parent "Jacob Carlborg" <doob me.com> writes:
On Sunday, 3 August 2014 at 20:40:47 UTC, Sean Kelly wrote:

 This is tricky with a range. With an event-based parser I'd 
 have events for object and array begin / end, but with a range 
 you end up having an element that's a token, which is pretty 
 weird.
Have a look at Token.Kind in the top of the module [1]. The enum has objectStart, objectEnd, arrayStart and arrayEnd. By just looking that that, it seems it already works very similar to an event parser, but with a range API. This is exactly like the XML pull parser in Tango. [1] http://erdani.com/d/jgrandson.d -- /Jacob Carlborg
Aug 04 2014
prev sibling parent "Jacob Carlborg" <doob me.com> writes:
On Sunday, 3 August 2014 at 17:19:04 UTC, Sean Kelly wrote:

 Is there support for output?  I see the makeArray and 
 makeObject routines...  Ideally, there should be a way to 
 serialize JSON against an OutputRange with optional formatting.
I think it should only provide very primitive functions to serialize basic data types. Then Phobos should provide a separate module/package for generic serialization where JSON is an archive type using this module as its backend. -- /Jacob Carlborg
Aug 04 2014
prev sibling next sibling parent Orvid King <blah38621 gmail.com> writes:
On 8/3/2014 2:16 AM, Andrei Alexandrescu wrote:
 We need a better json library at Facebook. I'd discussed with Snke the
 possibility of taking vibe.d's json to std but he said it needs some
 more work. So I took std.jgrandson to proof of concept state and hence
 ready for destruction:

 http://erdani.com/d/jgrandson.d
 http://erdani.com/d/phobos-prerelease/std_jgrandson.html

 Here are a few differences compared to vibe.d's library. I think these
 are desirable to have in that library as well:

 * Parsing strings is decoupled into tokenization (which is lazy and only
 needs an input range) and parsing proper. Tokenization is lazy, which
 allows users to create their own advanced (e.g. partial/lazy) parsing if
 needed. The parser itself is eager.

 * There's no decoding of strings.

 * The representation is built on Algebraic, with the advantages that it
 benefits from all of its primitives. Implementation is also very compact
 because Algebraic obviates a bunch of boilerplate. Subsequent
 improvements to Algebraic will also reflect themselves into improvements
 to std.jgrandson.

 * The JSON value (called std.jgrandson.Value) has no named member
 variables or methods except for __payload. This is so there's no clash
 between dynamic properties exposed via opDispatch.

 Well that's about it. What would it take for this to become a Phobos
 proposal? Destroy.


 Andrei
If your looking for serialization from statically known type layouts then I believe my JSON (de)serialization code (https://github.com/Orvid/JSONSerialization) might actually be of interest to you, as it uses no intermediate representation, nor does it allocate when it converts an object to JSON. As far as I know, even when only compiled with DMD, it's among the fastest JSON (de)serialization libraries. Unless it needs to convert a floating point number to a string, in which case I suppose you could certainly use a local buffer to write to, but at the moment it just converts it to a normal string that gets written to the output range. It also supports (de)serializing from, what I called at the time, dynamic types, such as std.variant, which isn't actually supported because that code is only there because I needed it for something else, and wasn't using std.variant at the time.
Aug 03 2014
prev sibling next sibling parent reply "Andrea Fontana" <nospam example.com> writes:
On Sunday, 3 August 2014 at 07:16:05 UTC, Andrei Alexandrescu 
wrote:
 We need a better json library at Facebook. I'd discussed with 
 Sönke the possibility of taking vibe.d's json to std but he 
 said it needs some more work. So I took std.jgrandson to proof 
 of concept state and hence ready for destruction:

 http://erdani.com/d/jgrandson.d
 http://erdani.com/d/phobos-prerelease/std_jgrandson.html

 Here are a few differences compared to vibe.d's library. I 
 think these are desirable to have in that library as well:

 * Parsing strings is decoupled into tokenization (which is lazy 
 and only needs an input range) and parsing proper. Tokenization 
 is lazy, which allows users to create their own advanced (e.g. 
 partial/lazy) parsing if needed. The parser itself is eager.

 * There's no decoding of strings.

 * The representation is built on Algebraic, with the advantages 
 that it benefits from all of its primitives. Implementation is 
 also very compact because Algebraic obviates a bunch of 
 boilerplate. Subsequent improvements to Algebraic will also 
 reflect themselves into improvements to std.jgrandson.

 * The JSON value (called std.jgrandson.Value) has no named 
 member variables or methods except for __payload. This is so 
 there's no clash between dynamic properties exposed via 
 opDispatch.

 Well that's about it. What would it take for this to become a 
 Phobos proposal? Destroy.


 Andrei
On my bson library I found very useful to have some methods to know if a field exists or not, and to get a "defaulted" value. Something like: auto assume(T)(Value v, T default = T.init); Another good method could be something like xpath to get a deep value: Value v = value["/path/to/sub/object"]; Moreover in my library I actually have three different methods to read a value: T get(T)() // Exception if value is not a T or not valid or value doesn't exist T to(T)() // Try to convert value to T using to!string. Exception if doesn't exists or not valid BsonField!T as(T)(lazy T default = T.init) // Always return a value BsonField!T is an "alias this"-ed struct with two fields: T value and bool error(). T value is the aliased field, and error() tells you if value is defaulted (because of an error: field not exists or can't convert to T) So I can write something like this: int myvalue = json["/that/deep/property"].as!int; or auto myvalue = json["/that/deep/property"].as!int(10); if (myvalue.error) writeln("Property doesn't exists, I'm using default value); writeln("Property value: ", myvalue); I hope this can be useful...
Aug 04 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/4/14, 12:47 AM, Andrea Fontana wrote:
 On my bson library I found very useful to have some methods to know if a
 field exists or not, and to get a "defaulted" value. Something like:

 auto assume(T)(Value v, T default = T.init);
Nice. Probably "get" would be better to be in keep with built-in hashtables.
 Another good method could be something like xpath to get a deep value:

 Value v = value["/path/to/sub/object"];
Cool. Is it unlikely that a value contains an actual slash? If so would be value["path"]["to"]["sub"]["object"] more precise?
 Moreover in my library I actually have three different methods to read a
 value:

 T get(T)() // Exception if value is not a T or not valid or value
 doesn't exist
 T to(T)()  // Try to convert value to T using to!string. Exception if
 doesn't exists or not valid

 BsonField!T as(T)(lazy T default = T.init)  // Always return a value

 BsonField!T is an "alias this"-ed struct with two fields: T value and
 bool error(). T value is the aliased field, and error() tells you if
 value is defaulted (because of an error: field not exists or can't
 convert to T)

 So I can write something like this:

 int myvalue = json["/that/deep/property"].as!int;

 or

 auto myvalue = json["/that/deep/property"].as!int(10);

 if (myvalue.error) writeln("Property doesn't exists, I'm using default
 value);

 writeln("Property value: ", myvalue);

 I hope this can be useful...
Sure is, thanks. Listen, would you want to volunteer a std.data.json proposal? Andrei
Aug 04 2014
parent reply "Andrea Fontana" <nospam example.com> writes:
On Monday, 4 August 2014 at 16:58:12 UTC, Andrei Alexandrescu 
wrote:
 On 8/4/14, 12:47 AM, Andrea Fontana wrote:
 On my bson library I found very useful to have some methods to 
 know if a
 field exists or not, and to get a "defaulted" value. Something 
 like:

 auto assume(T)(Value v, T default = T.init);
Nice. Probably "get" would be better to be in keep with built-in hashtables.
I wrote assume just to use proposed syntax :)
 Another good method could be something like xpath to get a 
 deep value:

 Value v = value["/path/to/sub/object"];
Cool. Is it unlikely that a value contains an actual slash? If so would be value["path"]["to"]["sub"]["object"] more precise?
Key with a slash (or dot?) inside is not common at all. Never seen on json data. In many languages there're libraries to bind json to struct or objects so usually people doesn't use strange chars inside key. If needed you can still use old good method to read a single field. value["path"]["to"]["object"] was my first choice but i didn't like it. First: it create a lot of temporary objects. Second: it is easier to implement using a single string (also on assignment) I gave it a try with value["path", "to", "index"] but it's not confortable if you need to generate your path from code.
 Moreover in my library I actually have three different methods 
 to read a
 value:

 T get(T)() // Exception if value is not a T or not valid or 
 value
 doesn't exist
 T to(T)()  // Try to convert value to T using to!string. 
 Exception if
 doesn't exists or not valid

 BsonField!T as(T)(lazy T default = T.init)  // Always return a 
 value

 BsonField!T is an "alias this"-ed struct with two fields: T 
 value and
 bool error(). T value is the aliased field, and error() tells 
 you if
 value is defaulted (because of an error: field not exists or 
 can't
 convert to T)

 So I can write something like this:

 int myvalue = json["/that/deep/property"].as!int;

 or

 auto myvalue = json["/that/deep/property"].as!int(10);

 if (myvalue.error) writeln("Property doesn't exists, I'm using 
 default
 value);

 writeln("Property value: ", myvalue);

 I hope this can be useful...
Sure is, thanks. Listen, would you want to volunteer a std.data.json proposal?
What does it mean? :)
 Andrei
Aug 05 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/5/14, 2:08 AM, Andrea Fontana wrote:
 Sure is, thanks. Listen, would you want to volunteer a std.data.json
 proposal?
What does it mean? :)
One one side enters vibe.data.json with the deltas prompted by std.jgrandson plus your talent and determination, and on the other side comes std.data.json with code and documentation that passes the Phobos review process. -- Andrei
Aug 05 2014
prev sibling parent reply "Jacob Carlborg" <doob me.com> writes:
On Sunday, 3 August 2014 at 07:16:05 UTC, Andrei Alexandrescu 
wrote:
 We need a better json library at Facebook. I'd discussed with 
 Sönke the possibility of taking vibe.d's json to std but he 
 said it needs some more work. So I took std.jgrandson to proof 
 of concept state and hence ready for destruction:

 http://erdani.com/d/jgrandson.d
 http://erdani.com/d/phobos-prerelease/std_jgrandson.html
* Could you please put it on Github to get syntax highlighting and all the other advantages * It doesn't completely follow the Phobos naming conventions * The indentation is off in some places * The unit tests is a bit lacking for the separate parsing functions * There are methods for getting the strings and numbers, what about booleans? * Shouldn't it be called TokenRange? * Shouldn't this be built using the lexer generator you so strongly have been pushing for? * The unit tests for TokenStream is very dense. I would prefer empty newlines for grouping "assert" and calls to "popFront" belonging together -- /Jacob Carlborg
Aug 04 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/4/14, 12:56 AM, Jacob Carlborg wrote:
 On Sunday, 3 August 2014 at 07:16:05 UTC, Andrei Alexandrescu wrote:
 We need a better json library at Facebook. I'd discussed with Sönke
 the possibility of taking vibe.d's json to std but he said it needs
 some more work. So I took std.jgrandson to proof of concept state and
 hence ready for destruction:

 http://erdani.com/d/jgrandson.d
 http://erdani.com/d/phobos-prerelease/std_jgrandson.html
Thanks for your comments! A few responses within:
 * Could you please put it on Github to get syntax highlighting and all
 the other advantages
Quick workaround: http://dpaste.dzfl.pl/65f4dcc36ab8
 * It doesn't completely follow the Phobos naming conventions
What would be the places?
 * The indentation is off in some places
Xamarin/Mono-D is at fault here :o).
 * The unit tests is a bit lacking for the separate parsing functions
Agreed.
 * There are methods for getting the strings and numbers, what about
 booleans?
You mean for Token? Good point. Numbers and strings are somewhat special because they have a payload associated. In contrast Booleans are represented by two distinct tokens. Would be good to add a convenience method.
 * Shouldn't it be called TokenRange?
Yah.
 * Shouldn't this be built using the lexer generator you so strongly have
 been pushing for?
Of course, and in the beginning I've actually pasted some code from it. Then I regressed to minimizing dependencies.
 * The unit tests for TokenStream is very dense. I would prefer empty
 newlines for grouping "assert" and calls to "popFront" belonging together
De gustibus et the coloribus non est disputandum :o). Andrei
Aug 04 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-04 18:55, Andrei Alexandrescu wrote:

 What would be the places?
That's why it's easier with Github ;) I can comment directly on a line. I just had a quick look but "_true", "_false" and "_null" in Token.Kind. If I recall correctly we add an underscore as a suffix for symbols with the same name as keywords.
 You mean for Token? Good point.
Yes, in Token.
 Numbers and strings are somewhat special
 because they have a payload associated. In contrast Booleans are
 represented by two distinct tokens. Would be good to add a convenience
 method.
Right.
 De gustibus et the coloribus non est disputandum :o).
Please avoid these Latin sentences, I have no idea what they mean. This is an international community, please don't make it more complicated than it already is with language barriers. -- /Jacob Carlborg
Aug 04 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/4/14, 11:46 AM, Jacob Carlborg wrote:
 De gustibus et the coloribus non est disputandum :o).
Please avoid these Latin sentences, I have no idea what they mean. This is an international community, please don't make it more complicated than it already is with language barriers.
"Favorite foods and colors are not to be disputed." 51,300 results on google... and please let's end this before it becomes another Epic Debate. -- Andrei
Aug 04 2014