www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Cap'n Proto for D v0.1.2

reply Thomas Brix Larsen <brix brix-verden.dk> writes:
"Cap’n Proto is an insanely fast data interchange format and 
capability-based RPC system. Think JSON, except binary. Or think 
Protocol Buffers, except faster."

This is the initial public release of my optimized port of the 
Java implementation of Cap'n Proto.

State:
* Passes Cap'n Proto testsuite.
* Optimized. Just a little slower than the official C++ 
implementation (see benchmarks on github).
* Missing RPC part of Cap'n Proto.

http://code.dlang.org/packages/capnproto-dlang
https://github.com/ThomasBrixLarsen/capnproto-dlang
Apr 18
next sibling parent reply Swoorup Joshi <swoorupjoshi gmail.com> writes:
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen 
wrote:
 "Cap’n Proto is an insanely fast data interchange format and 
 capability-based RPC system. Think JSON, except binary. Or 
 think Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the 
 Java implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++ 
 implementation (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang
Java?? Yikes
Apr 18
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 4/18/17 9:14 PM, Swoorup Joshi wrote:
 On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote:
 "Cap’n Proto is an insanely fast data interchange format and
 capability-based RPC system. Think JSON, except binary. Or think
 Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the Java
 implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++ implementation
 (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang
Java?? Yikes
Risking a flamewar but what's wrong with Java? --- Dmitry Olshansky
Apr 18
parent Jacob Carlborg <doob me.com> writes:
On 2017-04-18 23:08, Dmitry Olshansky wrote:

 Risking a flamewar but what's wrong with Java?
I don't like any language that force me to write in a particular style or paradigm. Because all problems cannot be solved (or not in a good way) in the same way. That said, my D code is quite heavily influenced by Java. -- /Jacob Carlborg
Apr 19
prev sibling next sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen 
wrote:
 "Cap’n Proto is an insanely fast data interchange format and 
 capability-based RPC system. Think JSON, except binary. Or 
 think Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the 
 Java implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++ 
 implementation (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang
Great Job! I'm following Cap'n Proto, and it's very very interesting... I would love also the RPC part, maybe based on Vibe... have you any plan to implement that? I'm really curious to try it! --- Paolo
Apr 18
parent Thomas Brix Larsen <brix brix-verden.dk> writes:
On Tuesday, 18 April 2017 at 20:46:33 UTC, Paolo Invernizzi wrote:
 On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen 
 wrote:
 "Cap’n Proto is an insanely fast data interchange format and 
 capability-based RPC system. Think JSON, except binary. Or 
 think Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the 
 Java implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++ 
 implementation (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang
Great Job! I'm following Cap'n Proto, and it's very very interesting... I would love also the RPC part, maybe based on Vibe... have you any plan to implement that? I'm really curious to try it! --- Paolo
Well personally I have no need for the RPC part. Though if it gets added to the Java implementation of Cap'n Proto I will probably port it. The D implementation works well. I developed it for a hobby project. Game server in D. Client in Unity. At work we have been using the C++ Cap'n Proto for IPC for a while. Recently we have also started using the D implementation for configuration files.
Apr 18
prev sibling next sibling parent reply Jay Norwood <jayn prismnet.com> writes:
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen 
wrote:
 "Cap’n Proto is an insanely fast data interchange format and 
 capability-based RPC system. Think JSON, except binary. Or 
 think Protocol Buffers, except faster."
The features below, from the capnproto.org description, interest me. However a MappedByteBuffer would be used for the mmap feature in java. https://capnproto.org/ mmap: Read a large Cap’n Proto file by memory-mapping it. The OS won’t even read in the parts that you don’t access. Inter-language communication: Calling C++ code from, say, Java or Python tends to be painful or slow. With Cap’n Proto, the two languages can easily operate on the same in-memory data structure. Inter-process communication: Multiple processes running on the same machine can share a Cap’n Proto message via shared memory. No need to pipe data through the kernel. Calling another process can be just as fast and easy as calling another thread. This info from stackoverflow also seems to imply that MappedByteBuffer would be required for some of the capnproto features. So, could you explain a little more about what are the capabilities of the current d library implementation, with just the ByteBuffer implemented from the java nio code? Thanks, Jay http://stackoverflow.com/questions/29361058/read-proto-partly-instead-of-full-parsing-in-java 'If you are willing to consider using a different protocol framework, Cap'n Proto is extremely similar in design to Protobufs but features in the ability to read only the part of the message you care about. Cap'n Proto incurs no overhead for the fields you don't examine, other than obviously the bandwidth and memory to receive the raw message bytes. If you are reading from a file, and you use memory mapping (MappedByteBuffer in Java), then only the parts of the message you actually use will be read from disk.'
Apr 19
parent reply Thomas Brix Larsen <brix brix-verden.dk> writes:
On Wednesday, 19 April 2017 at 16:34:02 UTC, Jay Norwood wrote:
 [...]

 This info from stackoverflow also seems to imply that 
 MappedByteBuffer would be required for some of the capnproto 
 features.  So, could you explain a little more about what are 
 the capabilities of the current d library implementation, with 
 just the ByteBuffer implemented from the java nio code?  
 Thanks, Jay
 
 [...]
The port of ByteBuffer just wraps a slice when reading in D. Take a look at FileDescriptor[1]. It is a class I've added to support read/write using File from std.stdio. You can create a similar streamer using std.mmfile. I believe that this would be enough for memory mapped reading. [1]: https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/capnproto/FileDescriptor.d
Apr 19
parent reply Jay Norwood <jayn prismnet.com> writes:
On Wednesday, 19 April 2017 at 16:52:14 UTC, Thomas Brix Larsen 
wrote:
 Take a look at FileDescriptor[1]. It is a class I've added to 
 support read/write using File from std.stdio. You can create a 
 similar streamer using std.mmfile. I believe that this would be 
 enough for memory mapped reading.

 [1]: 
 https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/capnproto/FileDescriptor.d
Ok, thanks. I took a look at several capnproto implementations just now, and didn't see any tests for a mmap 'feature'. The roadmap doc below indicates it doesn't exist, and perhaps there are some details yet to be resolved to make it 'friendly' for a mmap. https://capnproto.org/roadmap.html mmap-friendly mutable storage format: Define a standard storage format that is friendly to mmap-based use while allowing modification. (With the current serialization format, mmap is only useful for read-only structures.) Possibly based on the ORM interface, updates only possible at the granularity of a whole ORM entry. In java the MappedByteBuffer can be used with a RandomAccessFile channel, and then the accesses of the loaded map can be random, without requiring sequential accesses of the whole mapped file. So java nio already has some higher level classes in place that would make it easier to develop a first implementation of the mmap features.
Apr 19
parent reply Thomas Brix Larsen <brix brix-verden.dk> writes:
On Wednesday, 19 April 2017 at 18:24:46 UTC, Jay Norwood wrote:
 [...]

 Ok, thanks. I took a look at several capnproto implementations 
 just now, and didn't see any tests for a mmap 'feature'.  The 
 roadmap doc below indicates it doesn't exist, and perhaps there 
 are some details yet to be resolved to make it 'friendly' for a 
 mmap.
But reading using random access *is* a feature of Cap'n Proto. So when reading a memory mapped Cap'n Proto file, getters will be faster if you use it in a non-sequential way. The format of Cap'n Proto doesn't currently support *writing* to a memory mapped file.
Apr 19
parent Thomas Brix Larsen <brix brix-verden.dk> writes:
On Wednesday, 19 April 2017 at 19:30:03 UTC, Thomas Brix Larsen 
wrote:
 On Wednesday, 19 April 2017 at 18:24:46 UTC, Jay Norwood wrote:
 [...]

 Ok, thanks. I took a look at several capnproto implementations 
 just now, and didn't see any tests for a mmap 'feature'.  The 
 roadmap doc below indicates it doesn't exist, and perhaps 
 there are some details yet to be resolved to make it 
 'friendly' for a mmap.
But reading using random access *is* a feature of Cap'n Proto. So when reading a memory mapped Cap'n Proto file, getters will be faster if you use it in a non-sequential way.
I have made some minor changes to make adaptors responsible for allocating buffers. This made it possible to add a new adaptor MemoryMapped and a sample[1] to show it off. I have also verified that no buffer allocations happen while reading using MemoryMapped. Note that writing using memory mapping is still not possible. [1]: https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/samples/mmap.d
Apr 20
prev sibling parent reply Jonathan M Davis via Digitalmars-d-announce writes:
On Tuesday, April 18, 2017 18:09:54 Thomas Brix Larsen via Digitalmars-d-
announce wrote:
 "Cap’n Proto is an insanely fast data interchange format and
 capability-based RPC system. Think JSON, except binary. Or think
 Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the
 Java implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++
 implementation (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang
LOL. Oh, I remember a coworker bringing up this library. It's the one that the site claimed was infinitely faster than protocol buffers (IIRC, because of some work that protocol buffers does that Captain Proto skips entirely). - Jonathan M Davis
Apr 20
parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Thursday, 20 April 2017 at 18:25:03 UTC, Jonathan M Davis 
wrote:
 On Tuesday, April 18, 2017 18:09:54 Thomas Brix Larsen via 
 Digitalmars-d- announce wrote:

 LOL. Oh, I remember a coworker bringing up this library. It's 
 the one that the site claimed was infinitely faster than 
 protocol buffers (IIRC, because of some work that protocol 
 buffers does that Captain Proto skips entirely).
Exactly! Well, "Captain Proto" knows what he does, as he was the man that wrote google protobuf. The same way of working of Cap'n Proto is what Google introduced later in FlatBuffer... but it's missing totally all the good stuff taken from e-lang, that in my opinion is a big plus. Don't take me wrong, I know that a lot is to be implemented, today, but I love man with a clean path towards brilliant ideas [1] [1] https://capnproto.org/rpc.html /Paolo
Apr 20