www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Typed Message Passing between D Processes

reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
Is there an alternative to

http://dlang.org/phobos/std_process.html#.pipe

that can be used to do _typed_ _message_ _passing_ between two D 
processes with the same convenience as `send` and `receive` in

std.concurrency

?

Either in Phobos or in a third party library?
Jun 25 2015
next sibling parent reply "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Thursday, 25 June 2015 at 14:04:23 UTC, Per Nordlöw wrote:
 Is there an alternative to

 http://dlang.org/phobos/std_process.html#.pipe

 that can be used to do _typed_ _message_ _passing_ between two 
 D processes with the same convenience as `send` and `receive` in

 std.concurrency

 ?

 Either in Phobos or in a third party library?
not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ?
Jun 25 2015
parent reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote:
 not sure if this is quite what you are looking for, or if the 
 performance overhead is acceptable but have you looked at 
 msgpack to go on top of a lower level pipe ?
Why is a lower-level pipe needed, when we have pipe in std.process?
Jun 26 2015
parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Friday, 26 June 2015 at 09:06:18 UTC, Per Nordlöw wrote:
 On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote:
 not sure if this is quite what you are looking for, or if the 
 performance overhead is acceptable but have you looked at 
 msgpack to go on top of a lower level pipe ?
Why is a lower-level pipe needed, when we have pipe in std.process?
well, that is the sort of lower level pipe I was referring to.
Jun 26 2015
parent reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Friday, 26 June 2015 at 10:23:26 UTC, Laeeth Isharc wrote:
 On Friday, 26 June 2015 at 09:06:18 UTC, Per Nordlöw wrote:
 On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote:
 not sure if this is quite what you are looking for, or if the 
 performance overhead is acceptable but have you looked at 
 msgpack to go on top of a lower level pipe ?
Why is a lower-level pipe needed, when we have pipe in std.process?
well, that is the sort of lower level pipe I was referring to.
Ok, thanks.
Jun 26 2015
parent "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Friday, 26 June 2015 at 11:13:11 UTC, Per Nordlöw wrote:
 On Friday, 26 June 2015 at 10:23:26 UTC, Laeeth Isharc wrote:
 On Friday, 26 June 2015 at 09:06:18 UTC, Per Nordlöw wrote:
 On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc 
 wrote:
 not sure if this is quite what you are looking for, or if 
 the performance overhead is acceptable but have you looked 
 at msgpack to go on top of a lower level pipe ?
Why is a lower-level pipe needed, when we have pipe in std.process?
well, that is the sort of lower level pipe I was referring to.
Ok, thanks.
It should be easy enough to write so you can switch out the transport protocol. You could use something like nanomsg if you are prepared to use a beta product. I wrote some D bindings, but they are pretty raw and haven't been used much except for tiny applications. Fantastic if you feel like giving them a try and improving them. However if you don't like them you can just call nanomsg C api directly. But nanomsg itself is still early stage. http://nanomsg.org/index.html http://java.dzone.com/articles/look-nanomsg-and-scalability https://github.com/Laeeth/d-nanomsg
Jun 26 2015
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Thursday, 25 June 2015 at 14:04:23 UTC, Per Nordlöw wrote:
 Is there an alternative to

 http://dlang.org/phobos/std_process.html#.pipe

 that can be used to do _typed_ _message_ _passing_ between two 
 D processes with the same convenience as `send` and `receive` in

 std.concurrency

 ?

 Either in Phobos or in a third party library?
You'd have to implement your own IPC for that to work. You can always use TCP and serialise everything. Atila
Jun 26 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
std.concurrency was supposed to be able to handle that by design 
but it is impossible to do without any sort of standard 
serialization utility in Phobos (and, ideally, very fast binary 
serialization utility)
Jun 26 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 26 June 2015 at 12:31:04 UTC, Dicebot wrote:
 std.concurrency was supposed to be able to handle that by 
 design but it is impossible to do without any sort of standard 
 serialization utility in Phobos (and, ideally, very fast binary 
 serialization utility)
I'd have to benchmark it against something, but I'm pretty sure cerealed is fast.
Jun 26 2015
parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote:
 I'd have to benchmark it against something, but I'm pretty sure 
 cerealed is fast.
Faster than msgpack?
Jun 28 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Sunday, 28 June 2015 at 17:02:42 UTC, Nordlöw wrote:
 On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote:
 I'd have to benchmark it against something, but I'm pretty 
 sure cerealed is fast.
Faster than msgpack?
I guess I'm going to have benchmark this now... :) Atila
Jun 29 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 29 June 2015 at 08:45:15 UTC, Atila Neves wrote:
 On Sunday, 28 June 2015 at 17:02:42 UTC, Nordlöw wrote:
 On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote:
 I'd have to benchmark it against something, but I'm pretty 
 sure cerealed is fast.
Faster than msgpack?
I guess I'm going to have benchmark this now... :) Atila
In release builds, _nearly_. Here's the benchmark program: http://dpaste.dzfl.pl/17b0ed9c0204. Results below. msgpack is... fast. From the docs, cerealed seems to have more features that I actually use and need though. Results: DMD debug: Cerealed: 4 secs, 987 ms, 700 μs, and 5 hnsecs MsgPack: 1 sec, 771 ms, 713 μs, and 7 hnsecs DMD release: Cerealed: 2 secs, 556 ms, 714 μs, and 6 hnsecs MsgPack: 1 sec, 89 ms, 561 μs, and 3 hnsecs GDC debug: Cerealed: 4 secs, 863 ms, 501 μs, and 1 hnsec MsgPack: 2 secs, 32 ms, 53 μs, and 1 hnsec GDC release: Cerealed: 1 sec, 740 ms, 726 μs, and 4 hnsecs MsgPack: 1 sec, 20 ms, 287 μs, and 3 hnsecs LDC debug: Cerealed: 7 secs, 711 ms, 154 μs, and 4 hnsecs MsgPack: 3 secs, 694 ms, 566 μs, and 2 hnsecs LDC release: Cerealed: 1 sec, 795 ms, 380 μs, and 7 hnsecs MsgPack: 931 ms, 355 μs, and 5 hnsecs
Jun 29 2015
next sibling parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Monday, 29 June 2015 at 10:22:10 UTC, Atila Neves wrote:
 I guess I'm going to have benchmark this now... :)
What about doing a memory profiling using DMD fresh builtin profiler of http://dpaste.dzfl.pl/17b0ed9c0204 ? I'm guessing the GC might give misguiding results as your testStruct returns a relatively small data structure. I would rather like to see a larger (randomized) structure being tested on. You could make use of my https://github.com/nordlow/justd/blob/master/random_ex.d for random instance generation :)
Jun 29 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 29 June 2015 at 13:59:37 UTC, Nordlöw wrote:
 On Monday, 29 June 2015 at 10:22:10 UTC, Atila Neves wrote:
 I guess I'm going to have benchmark this now... :)
What about doing a memory profiling using DMD fresh builtin profiler of http://dpaste.dzfl.pl/17b0ed9c0204 ? I'm guessing the GC might give misguiding results as your testStruct returns a relatively small data structure. I would rather like to see a larger (randomized) structure being tested on. You could make use of my https://github.com/nordlow/justd/blob/master/random_ex.d for random instance generation :)
I tried your code and added it to the benchmark. It didn't make much of a difference. Atila
Jul 29 2015
prev sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 29 June 2015 at 10:22:10 UTC, Atila Neves wrote:
 On Monday, 29 June 2015 at 08:45:15 UTC, Atila Neves wrote:
 On Sunday, 28 June 2015 at 17:02:42 UTC, Nordlöw wrote:
 On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote:
 I'd have to benchmark it against something, but I'm pretty 
 sure cerealed is fast.
Faster than msgpack?
I guess I'm going to have benchmark this now... :) Atila
In release builds, _nearly_. Here's the benchmark program: http://dpaste.dzfl.pl/17b0ed9c0204. Results below. msgpack is... fast. From the docs, cerealed seems to have more features that I actually use and need though. Results: DMD debug: Cerealed: 4 secs, 987 ms, 700 μs, and 5 hnsecs MsgPack: 1 sec, 771 ms, 713 μs, and 7 hnsecs DMD release: Cerealed: 2 secs, 556 ms, 714 μs, and 6 hnsecs MsgPack: 1 sec, 89 ms, 561 μs, and 3 hnsecs GDC debug: Cerealed: 4 secs, 863 ms, 501 μs, and 1 hnsec MsgPack: 2 secs, 32 ms, 53 μs, and 1 hnsec GDC release: Cerealed: 1 sec, 740 ms, 726 μs, and 4 hnsecs MsgPack: 1 sec, 20 ms, 287 μs, and 3 hnsecs LDC debug: Cerealed: 7 secs, 711 ms, 154 μs, and 4 hnsecs MsgPack: 3 secs, 694 ms, 566 μs, and 2 hnsecs LDC release: Cerealed: 1 sec, 795 ms, 380 μs, and 7 hnsecs MsgPack: 931 ms, 355 μs, and 5 hnsecs
Before I had time to look into it, somebody graciously optimised cerealed for me in this PR: https://github.com/atilaneves/cerealed/pull/6 And now then benchmarks are (dub build --build=release, fastest of 10 runs for each compiler): DMD: Cerealed: 1 sec, 199 ms, 118 μs, and 1 hnsec MsgPack: 1 sec, 85 ms, and 516 μs GDC: Cerealed: 927 ms, 677 μs, and 4 hnsecs MsgPack: 1 sec, 32 ms, 736 μs, and 5 hnsecs LDC: Cerealed: 970 ms, 482 μs, and 6 hnsecs MsgPack: 896 ms, 591 μs, and 2 hnsecs Not too shabby! Atila
Jul 29 2015
parent reply "yawniek" <dlang srtnwz.com> writes:
On Wednesday, 29 July 2015 at 16:36:41 UTC, Atila Neves wrote:
 LDC:
 Cerealed: 970 ms, 482 μs, and 6 hnsecs
 MsgPack:  896 ms, 591 μs, and 2 hnsecs


 Not too shabby!

 Atila
cool. what are the advantages of cereald over msgpack? can you stream in packets with cereald too? cool thing about msgpack is that there exist libraries for many language. so we use it do actually store logs and then process them with other tools too.
Jul 29 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Thursday, 30 July 2015 at 05:53:48 UTC, yawniek wrote:
 On Wednesday, 29 July 2015 at 16:36:41 UTC, Atila Neves wrote:
 LDC:
 Cerealed: 970 ms, 482 μs, and 6 hnsecs
 MsgPack:  896 ms, 591 μs, and 2 hnsecs


 Not too shabby!

 Atila
cool. what are the advantages of cereald over msgpack?
AFAIK, features. The kind of features I need/use to write networking code and reduce the boilerplate to an absolute minimum.
 can you stream in packets with cereald too?
I don't know exactly what you mean. I've only used it to go from network packets structs to bytes and vice-versa.
 cool thing about msgpack is that there exist libraries for many 
 language.
 so we use it do actually store logs and then process them with 
 other tools too.
I wrote cerealed for networking. You _can_ use it to convert whatever else to binary, but its focus is easy networking. Atila
Jul 30 2015