digitalmars.D.learn - Typed Message Passing between D Processes
- "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> (7/7) Jun 25 2015 Is there an alternative to
- Laeeth Isharc (4/11) Jun 25 2015 not sure if this is quite what you are looking for, or if the
- "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> (3/6) Jun 26 2015 Why is a lower-level pipe needed, when we have pipe in
- Laeeth Isharc (2/8) Jun 26 2015 well, that is the sort of lower level pipe I was referring to.
- "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> (2/11) Jun 26 2015 Ok, thanks.
- Laeeth Isharc (11/24) Jun 26 2015 It should be easy enough to write so you can switch out the
- Atila Neves (4/11) Jun 26 2015 You'd have to implement your own IPC for that to work. You can
- Dicebot (4/4) Jun 26 2015 std.concurrency was supposed to be able to handle that by design
- Atila Neves (3/7) Jun 26 2015 I'd have to benchmark it against something, but I'm pretty sure
- =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= (2/4) Jun 28 2015 Faster than msgpack?
- Atila Neves (3/7) Jun 29 2015 I guess I'm going to have benchmark this now... :)
- Atila Neves (24/32) Jun 29 2015 In release builds, _nearly_. Here's the benchmark program:
- =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= (12/13) Jun 29 2015 What about doing a memory profiling using DMD fresh builtin
- Atila Neves (4/17) Jul 29 2015 I tried your code and added it to the benchmark. It didn't make
- Atila Neves (17/51) Jul 29 2015 Before I had time to look into it, somebody graciously optimised
- yawniek (8/13) Jul 29 2015 cool.
- Atila Neves (8/24) Jul 30 2015 AFAIK, features. The kind of features I need/use to write
Is there an alternative to that can be used to do _typed_ _message_ _passing_ between two D processes with the same convenience as `send` and `receive` in std.concurrency ? Either in Phobos or in a third party library?
Jun 25 2015
On Thursday, 25 June 2015 at 14:04:23 UTC, Per Nordlöw wrote:Is there an alternative to that can be used to do _typed_ _message_ _passing_ between two D processes with the same convenience as `send` and `receive` in std.concurrency ? Either in Phobos or in a third party library?not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ?
Jun 25 2015
On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote:not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ?Why is a lower-level pipe needed, when we have pipe in std.process?
Jun 26 2015
On Friday, 26 June 2015 at 09:06:18 UTC, Per Nordlöw wrote:On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote:well, that is the sort of lower level pipe I was referring to.not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ?Why is a lower-level pipe needed, when we have pipe in std.process?
Jun 26 2015
On Friday, 26 June 2015 at 10:23:26 UTC, Laeeth Isharc wrote:On Friday, 26 June 2015 at 09:06:18 UTC, Per Nordlöw wrote:Ok, thanks.On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote:well, that is the sort of lower level pipe I was referring to.not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ?Why is a lower-level pipe needed, when we have pipe in std.process?
Jun 26 2015
On Friday, 26 June 2015 at 11:13:11 UTC, Per Nordlöw wrote:On Friday, 26 June 2015 at 10:23:26 UTC, Laeeth Isharc wrote:It should be easy enough to write so you can switch out the transport protocol. You could use something like nanomsg if you are prepared to use a beta product. I wrote some D bindings, but they are pretty raw and haven't been used much except for tiny applications. Fantastic if you feel like giving them a try and improving them. However if you don't like them you can just call nanomsg C api directly. But nanomsg itself is still early stage. http://nanomsg.org/index.html http://java.dzone.com/articles/look-nanomsg-and-scalability https://github.com/Laeeth/d-nanomsgOn Friday, 26 June 2015 at 09:06:18 UTC, Per Nordlöw wrote:Ok, thanks.On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote:well, that is the sort of lower level pipe I was referring to.not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ?Why is a lower-level pipe needed, when we have pipe in std.process?
Jun 26 2015
On Thursday, 25 June 2015 at 14:04:23 UTC, Per Nordlöw wrote:Is there an alternative to that can be used to do _typed_ _message_ _passing_ between two D processes with the same convenience as `send` and `receive` in std.concurrency ? Either in Phobos or in a third party library?You'd have to implement your own IPC for that to work. You can always use TCP and serialise everything. Atila
Jun 26 2015
std.concurrency was supposed to be able to handle that by design but it is impossible to do without any sort of standard serialization utility in Phobos (and, ideally, very fast binary serialization utility)
Jun 26 2015
On Friday, 26 June 2015 at 12:31:04 UTC, Dicebot wrote:std.concurrency was supposed to be able to handle that by design but it is impossible to do without any sort of standard serialization utility in Phobos (and, ideally, very fast binary serialization utility)I'd have to benchmark it against something, but I'm pretty sure cerealed is fast.
Jun 26 2015
On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote:I'd have to benchmark it against something, but I'm pretty sure cerealed is fast.Faster than msgpack?
Jun 28 2015
On Sunday, 28 June 2015 at 17:02:42 UTC, Nordlöw wrote:On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote:I guess I'm going to have benchmark this now... :) AtilaI'd have to benchmark it against something, but I'm pretty sure cerealed is fast.Faster than msgpack?
Jun 29 2015
On Monday, 29 June 2015 at 08:45:15 UTC, Atila Neves wrote:On Sunday, 28 June 2015 at 17:02:42 UTC, Nordlöw wrote:In release builds, _nearly_. Here's the benchmark program: http://dpaste.dzfl.pl/17b0ed9c0204. Results below. msgpack is... fast. From the docs, cerealed seems to have more features that I actually use and need though. Results: DMD debug: Cerealed: 4 secs, 987 ms, 700 μs, and 5 hnsecs MsgPack: 1 sec, 771 ms, 713 μs, and 7 hnsecs DMD release: Cerealed: 2 secs, 556 ms, 714 μs, and 6 hnsecs MsgPack: 1 sec, 89 ms, 561 μs, and 3 hnsecs GDC debug: Cerealed: 4 secs, 863 ms, 501 μs, and 1 hnsec MsgPack: 2 secs, 32 ms, 53 μs, and 1 hnsec GDC release: Cerealed: 1 sec, 740 ms, 726 μs, and 4 hnsecs MsgPack: 1 sec, 20 ms, 287 μs, and 3 hnsecs LDC debug: Cerealed: 7 secs, 711 ms, 154 μs, and 4 hnsecs MsgPack: 3 secs, 694 ms, 566 μs, and 2 hnsecs LDC release: Cerealed: 1 sec, 795 ms, 380 μs, and 7 hnsecs MsgPack: 931 ms, 355 μs, and 5 hnsecsOn Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote:I guess I'm going to have benchmark this now... :) AtilaI'd have to benchmark it against something, but I'm pretty sure cerealed is fast.Faster than msgpack?
Jun 29 2015
On Monday, 29 June 2015 at 10:22:10 UTC, Atila Neves wrote:What about doing a memory profiling using DMD fresh builtin profiler of http://dpaste.dzfl.pl/17b0ed9c0204 ? I'm guessing the GC might give misguiding results as your testStruct returns a relatively small data structure. I would rather like to see a larger (randomized) structure being tested on. You could make use of my https://github.com/nordlow/justd/blob/master/random_ex.d for random instance generation :)I guess I'm going to have benchmark this now... :)
Jun 29 2015
On Monday, 29 June 2015 at 13:59:37 UTC, Nordlöw wrote:On Monday, 29 June 2015 at 10:22:10 UTC, Atila Neves wrote:I tried your code and added it to the benchmark. It didn't make much of a difference. AtilaWhat about doing a memory profiling using DMD fresh builtin profiler of http://dpaste.dzfl.pl/17b0ed9c0204 ? I'm guessing the GC might give misguiding results as your testStruct returns a relatively small data structure. I would rather like to see a larger (randomized) structure being tested on. You could make use of my https://github.com/nordlow/justd/blob/master/random_ex.d for random instance generation :)I guess I'm going to have benchmark this now... :)
Jul 29 2015
On Monday, 29 June 2015 at 10:22:10 UTC, Atila Neves wrote:On Monday, 29 June 2015 at 08:45:15 UTC, Atila Neves wrote:Before I had time to look into it, somebody graciously optimised cerealed for me in this PR: https://github.com/atilaneves/cerealed/pull/6 And now then benchmarks are (dub build --build=release, fastest of 10 runs for each compiler): DMD: Cerealed: 1 sec, 199 ms, 118 μs, and 1 hnsec MsgPack: 1 sec, 85 ms, and 516 μs GDC: Cerealed: 927 ms, 677 μs, and 4 hnsecs MsgPack: 1 sec, 32 ms, 736 μs, and 5 hnsecs LDC: Cerealed: 970 ms, 482 μs, and 6 hnsecs MsgPack: 896 ms, 591 μs, and 2 hnsecs Not too shabby! AtilaOn Sunday, 28 June 2015 at 17:02:42 UTC, Nordlöw wrote:In release builds, _nearly_. Here's the benchmark program: http://dpaste.dzfl.pl/17b0ed9c0204. Results below. msgpack is... fast. From the docs, cerealed seems to have more features that I actually use and need though. Results: DMD debug: Cerealed: 4 secs, 987 ms, 700 μs, and 5 hnsecs MsgPack: 1 sec, 771 ms, 713 μs, and 7 hnsecs DMD release: Cerealed: 2 secs, 556 ms, 714 μs, and 6 hnsecs MsgPack: 1 sec, 89 ms, 561 μs, and 3 hnsecs GDC debug: Cerealed: 4 secs, 863 ms, 501 μs, and 1 hnsec MsgPack: 2 secs, 32 ms, 53 μs, and 1 hnsec GDC release: Cerealed: 1 sec, 740 ms, 726 μs, and 4 hnsecs MsgPack: 1 sec, 20 ms, 287 μs, and 3 hnsecs LDC debug: Cerealed: 7 secs, 711 ms, 154 μs, and 4 hnsecs MsgPack: 3 secs, 694 ms, 566 μs, and 2 hnsecs LDC release: Cerealed: 1 sec, 795 ms, 380 μs, and 7 hnsecs MsgPack: 931 ms, 355 μs, and 5 hnsecsOn Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote:I guess I'm going to have benchmark this now... :) AtilaI'd have to benchmark it against something, but I'm pretty sure cerealed is fast.Faster than msgpack?
Jul 29 2015
On Wednesday, 29 July 2015 at 16:36:41 UTC, Atila Neves wrote:LDC: Cerealed: 970 ms, 482 μs, and 6 hnsecs MsgPack: 896 ms, 591 μs, and 2 hnsecs Not too shabby! Atilacool. what are the advantages of cereald over msgpack? can you stream in packets with cereald too? cool thing about msgpack is that there exist libraries for many language. so we use it do actually store logs and then process them with other tools too.
Jul 29 2015
On Thursday, 30 July 2015 at 05:53:48 UTC, yawniek wrote:On Wednesday, 29 July 2015 at 16:36:41 UTC, Atila Neves wrote:AFAIK, features. The kind of features I need/use to write networking code and reduce the boilerplate to an absolute minimum.LDC: Cerealed: 970 ms, 482 μs, and 6 hnsecs MsgPack: 896 ms, 591 μs, and 2 hnsecs Not too shabby! Atilacool. what are the advantages of cereald over msgpack?can you stream in packets with cereald too?I don't know exactly what you mean. I've only used it to go from network packets structs to bytes and vice-versa.cool thing about msgpack is that there exist libraries for many language. so we use it do actually store logs and then process them with other tools too.I wrote cerealed for networking. You _can_ use it to convert whatever else to binary, but its focus is easy networking. Atila
Jul 30 2015