digitalmars.D - D vs Go in real life, part 2. Also, Erlang.
- Atila Neves (56/56) Dec 04 2013 It all started here:
- Dicebot (7/12) Dec 04 2013 GC should not impact general performance in such scenario but is
- Atila Neves (10/16) Dec 04 2013 I'd have to dig in and see data to be convinced it's the GC
- Dicebot (12/21) Dec 04 2013 I am also not proficient enough with GC stuff to actually prove
- Atila Neves (6/7) Dec 04 2013 They're not ideal, but they're not unstable with respect to time
- Dicebot (6/14) Dec 04 2013 By "stable" here I mean "reproducible on similar h/w and within
- Jacob Carlborg (4/12) Dec 05 2013 Isn't it most important that all languages were tested in the same way?
- Atila Neves (12/14) Dec 05 2013 Maybe. I actually thought that if there might be a lesson here if
- Dicebot (11/29) Dec 05 2013 Depends on your goals :) If you want just to say "hah, your
- Jeff R. Allen (25/25) Dec 05 2013 I am the buddy who's always going on and on about Go.
- John Colvin (3/6) Dec 05 2013 I bet you're smart enough to do it in D. It's a different world
- Paulo Pinto (13/40) Dec 05 2013 This is why we already did quite a few consultancy projects that
- Walter Bright (10/13) Dec 05 2013 Although C and C++ are capable of extracting every ms out of the hardwar...
- Paulo Pinto (7/19) Dec 05 2013 Fully agree.
- Joseph Rushton Wakeling (6/9) Dec 06 2013 In a way I think it's a disease that one has to go through with those la...
- eles (2/8) Dec 06 2013 add to that the optimization of number of keyboard strokes :D
- Joseph Rushton Wakeling (7/8) Dec 06 2013 Hah, I got burned by that once in a very amusing way. I wrote a blog po...
- Paulo Pinto (2/10) Dec 06 2013 Canonical Go seems to suffer from that as well, in terms of variable nam...
- Max Samukha (4/31) Dec 05 2013 Templates are easy. They are pure functions. You just need to
- Craig Dillabaugh (2/58) Dec 04 2013 So either D is faster or you are a better coder than your buddies!
- Atila Neves (8/10) Dec 04 2013 I'd say we're all the same skill level. One of them sits next to
- Joseph Rushton Wakeling (6/11) Dec 04 2013 What about the relative elegance/maintainability/ease of comprehension o...
- Atila Neves (8/24) Dec 04 2013 I can't read Erlang (yet) so I don't know about that one. The C
- Andrei Alexandrescu (3/26) Dec 04 2013 How do they compare in lines of code?
- Atila Neves (16/36) Dec 04 2013 That's hard to measure. Not only because of the standard fare
- Andrei Alexandrescu (6/14) Dec 04 2013 [snip]
- Atila Neves (7/11) Dec 04 2013 If I convert this post into a blog it's going to take more than a
- Etienne (6/17) Dec 04 2013 It would be interesting to have a GitHub repo with the scripts for each
- Atila Neves (12/18) Dec 04 2013 Oh, I know. I tried it. The threads spent more time sending
- Andrei Alexandrescu (7/18) Dec 04 2013 Yah, I was just trying to lure you into starting on it until it's too
- Atila Neves (2/6) Dec 05 2013 Ask, and you shall receive:
- Dicebot (5/11) Dec 05 2013 _very_ well-written. I can only dream every other benchmarking
- Andrei Alexandrescu (3/16) Dec 05 2013 Fasten your seatbelts, it's gonna be a bumpy ride!
- Andrei Alexandrescu (4/17) Dec 05 2013 https://news.ycombinator.com/item?id=6855278
- Atila Neves (4/6) Dec 05 2013 Yeah, I thought that would happen. :) Thanks!
- qznc (6/10) Dec 06 2013 This might be the first time that I consider the reddit comments
- Justin Whear (3/11) Dec 05 2013 c-in-real-life-mqtt-broker-implementation-shootout/
- Chris Cain (6/12) Dec 05 2013 Very nice article! I'm pretty happy to see D doing so well on
- Atila Neves (5/10) Dec 05 2013 No, it's correct. Instead of measuring the round-trip time I
- Chris Cain (2/14) Dec 05 2013 Oh, okay, I see. Thanks :)
- Chris (24/30) Dec 06 2013 "Go is an opinionated language"
- David Nadlinger (8/10) Dec 04 2013 What are the issues that are blocking you from using it with LDC?
- Dicebot (9/16) Dec 04 2013 Quick search has found these issues:
- Iain Buclaw (6/21) Dec 04 2013 Ditto with GDC too... I did a quick search on those two project and
- Dicebot (5/10) Dec 04 2013 I think main issue is that no one was able to reduce those low
- Etienne (5/14) Dec 04 2013 I tried on linux x64 about 2 weeks ago and it's not vibe.d itself but
- David Nadlinger (3/5) Dec 04 2013 Yes, please!
- qznc (7/63) Dec 05 2013 Nice benchmark! :)
- Atila Neves (10/15) Dec 05 2013 Yeah, the C version was compiled with gcc 4.8.2. I'd've used
It all started here: http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar forum.dlang.org Not wanting to be outdone, my buddy Jeff (the Go guy) went and wrote a new benchmark. My implementation didn't do as well on that one so I went back to the code and optimised it. Also, our other buddy Patrick from work decided to get in on this and use it as an excuse to learn Erlang. So now 4 different implementations in 4 different languages are duking it out. There are two benchmarks, both written in Go. The first is loadtest and it measures throughput. I varied the number of client connections to the server and measured how many messages were sent per second. The second benchmark is pingtest and it measures latency. The quantity I varied here was the number of wildcard subscribers (it'd take a while to explain what they are to anyone who isn't familiar with MQTT). It has a bunch of client connections talk to each other, and instead of trying to send as many messages as computerly possible they wait for the acknowledgement from the partner connection. The results are as follows (I tried making charts but LibreOffice wasn't helping and then I got bored), with the numbers being thousands of messages per second: loadtest Connections: 100 500 750 1k D + vibe.d: 121.7 +/- 1.5 166.9 +/- 1.5 171.1 +/- 3.3 167.9 +/- 1.3 C (mosquitto): 106.1 +/- 0.8 122.4 +/- 0.4 95.2 +/- 1.3 74.7 +/- 0.4 Erlang: 104.1 +/- 2.2 124.2 +/- 5.9 117.6 +/- 4.6 117.7 +/- 3.2 Go: 90.9 +/- 11 100.1 +/- 0.1 99.3 +/- 0.2 98.8 +/- 0.3 pingtest wsubs: 20 200 400 D + vibe.d: 50.9 +/- 0.3 38.3 +/- 0.2 20.1 +/- 0.1 C (mosquitto): 65.4 +/- 4.4 45.2 +/- 0.2 20.0 +/- 0.0 Erlang: 49.1 +/- 0.8 30.9 +/- 0.3 15.6 +/- 0.1 Go: 45.2 +/- 0.2 27.5 +/- 0.1 16.0 +/- 0.1 So, D was faster than the other contenders by far in throughput, 2nd place losing to the C implementation on latency. I'm still not sure why that is. Profiling in this case is tricky. I'm pretty sure the profiler is still ticking away when a fiber yields - the top function is the one that reads from the network, which I can't do much about. GC was pretty much a non-issue, which I'm both surprised at and happy about. I allocated as much memory as I wanted at first, and by the time I tried to optmise memory usage by allocating less all I managed to eke out in performance terms was a paltry extra 8% or so. All in all though, so much for his "nothing is going to be faster than Go because this is what it's good at". If he'd measured it against mosquitto first and realised it was already losing to C I probably wouldn't have written an MQTT broker in D, so I guess that was a good thing. :) Atila P.S. vibe.d is awesome, although I wish it'd compile with ldc or gdc
Dec 04 2013
On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:GC was pretty much a non-issue, which I'm both surprised at and happy about. I allocated as much memory as I wanted at first, and by the time I tried to optmise memory usage by allocating less all I managed to eke out in performance terms was a paltry extra 8% or so.GC should not impact general performance in such scenario but is likely to hinder latency which pretty much matches what you are observing. Can you possibly provide information about network layout, h/w and server code used for testing? There are lot of possible oversights that can make results less legible, would have been nice to verify those ;)
Dec 04 2013
GC should not impact general performance in such scenario but is likely to hinder latency which pretty much matches what you are observing. Can you possibly provide information about network layout, h/w and server code used for testing? There are lot of possible oversights that can make results less legible, would have been nice to verify those ;)I'd have to dig in and see data to be convinced it's the GC making the latency worse than the C implementation, but from what I know you've got more experience with this so who knows :) Doing this on a real network was waaay more work than I was willing to put up with for something a colleague said during lunch break, so it was done on localhost. The h/w is a Core i7 Lenovo laptop (W530) with 8GB of RAM running Arch Linux. My server code can be found at https://github.com/atilaneves/mqtt, the C implementation has its own website: http://mosquitto.org/. Atila
Dec 04 2013
On Wednesday, 4 December 2013 at 13:50:52 UTC, Atila Neves wrote:I'd have to dig in and see data to be convinced it's the GC making the latency worse than the C implementation, but from what I know you've got more experience with this so who knows :)I am also not proficient enough with GC stuff to actually prove that but it fitted observations during my own tests pretty well. But I am not 100% sure.Doing this on a real network was waaay more work than I was willing to put up with for something a colleague said during lunch break, so it was done on localhost. The h/w is a Core i7 Lenovo laptop (W530) with 8GB of RAM running Arch Linux. My server code can be found at https://github.com/atilaneves/mqtt, the C implementation has its own website: http://mosquitto.org/.Ah this is rather sad. It makes tests results somewhat unstable because client load interferes with server load. Ideally client should be separate machine and has more powerful h/w than server. However, that also requires ~gigabit network in between as well as matching network cards to hit the limits of top server performance - this makes such tests rather cumbersome to execute. At the very least you should use process affinity to separate resources between client and server in more predictable manner.
Dec 04 2013
Ah this is rather sad. It makes tests results somewhat unstableThey're not ideal, but they're not unstable with respect to time - I did and redid all those measurements a few times, the error bars are the standard deviation taken to be the systematic error. I got challenged so I decided to beat his implementation, even this level of detail got called going overboard ;) Atila
Dec 04 2013
On Wednesday, 4 December 2013 at 18:02:18 UTC, Atila Neves wrote:By "stable" here I mean "reproducible on similar h/w and within similar (but not same) environment and meaningfully scaling to high load factors". It is simply impossible in general case when single PC is used for testing. That does not make results less interesting though :)Ah this is rather sad. It makes tests results somewhat unstableThey're not ideal, but they're not unstable with respect to time - I did and redid all those measurements a few times, the error bars are the standard deviation taken to be the systematic error. I got challenged so I decided to beat his implementation, even this level of detail got called going overboard ;) Atila
Dec 04 2013
On 2013-12-04 17:20, Dicebot wrote:Ah this is rather sad. It makes tests results somewhat unstable because client load interferes with server load. Ideally client should be separate machine and has more powerful h/w than server. However, that also requires ~gigabit network in between as well as matching network cards to hit the limits of top server performance - this makes such tests rather cumbersome to execute. At the very least you should use process affinity to separate resources between client and server in more predictable manner.Isn't it most important that all languages were tested in the same way? -- /Jacob Carlborg
Dec 05 2013
Isn't it most important that all languages were tested in the same way?Maybe. I actually thought that if there might be a lesson here if it turns out that the quality of the implementation and appropriate use of data structures and algorithms mattered more than the choice of programming language, but in the end competing with each other just made all of the implementations in the same order of magnitude of performance. This was never meant to be scientific, it's my Physics background showing up when I can't make measurements without calculating error bars. This started as a lunch joke. I came here to gloat, then you guys go and ask intelligent questions, what's up with that? :P Atila
Dec 05 2013
On Thursday, 5 December 2013 at 08:21:07 UTC, Jacob Carlborg wrote:On 2013-12-04 17:20, Dicebot wrote:Depends on your goals :) If you want just to say "hah, your language is nothing compared to my language" it is enough. If you want to make some observations/conclusions about specific implementation performance and how it scale for different conditions it becomes important to remove as many side impact as possible. And of course at high load/concurrency levels client competing with server for connections does make notable impact. In other words, it is good enough for comparison but not good enough for actual performance analysis.Ah this is rather sad. It makes tests results somewhat unstable because client load interferes with server load. Ideally client should be separate machine and has more powerful h/w than server. However, that also requires ~gigabit network in between as well as matching network cards to hit the limits of top server performance - this makes such tests rather cumbersome to execute. At the very least you should use process affinity to separate resources between client and server in more predictable manner.Isn't it most important that all languages were tested in the same way?
Dec 05 2013
I am the buddy who's always going on and on about Go. Here's the blog posting I made explaining why/how I made the MQTT server in Go and what I learned: http://blog.nella.org/mqtt-code-golf/ I was a bit surprised and interested to see that MQTT with small transactions is not easy to optimize because kernel/user context switching dominates the work that's done under control of the programmer and which can be optimized via data structures and/or reducing GC overhead. Something that both Atila and I verified in this exercise is that writing a fast network server that scales is so much easier with a very good networking library that takes advantage of lightweight threads, and with a runtime that provides GC. This frees up the programmer to focus on the protocol logic, and leave concurrency and bookkeeping in the capable hands of the machine. And, as has been mentioned on other Go versus D threads, a lot of this is a matter of taste and team dynamics. I've worked on many teams in my career where there was not a critical mass of people who could reason correctly about C++ memory management, and who could use generic programming techniques reliably. And, in line with discoveries from psychological research, it's common that people who are not competent at something do not recognize that fact. Perhaps I'm above average in this regard: I KNOW I'm not smart enough to write correct code using templating techniques. :) -jeff
Dec 05 2013
On Thursday, 5 December 2013 at 14:55:53 UTC, Jeff R. Allen wrote:Perhaps I'm above average in this regard: I KNOW I'm not smart enough to write correct code using templating techniques. :) -jeffI bet you're smart enough to do it in D. It's a different world compared to C++ in this regard.
Dec 05 2013
On Thursday, 5 December 2013 at 14:55:53 UTC, Jeff R. Allen wrote:I am the buddy who's always going on and on about Go. Here's the blog posting I made explaining why/how I made the MQTT server in Go and what I learned: http://blog.nella.org/mqtt-code-golf/ I was a bit surprised and interested to see that MQTT with small transactions is not easy to optimize because kernel/user context switching dominates the work that's done under control of the programmer and which can be optimized via data structures and/or reducing GC overhead. Something that both Atila and I verified in this exercise is that writing a fast network server that scales is so much easier with a very good networking library that takes advantage of lightweight threads, and with a runtime that provides GC. This frees up the programmer to focus on the protocol logic, and leave concurrency and bookkeeping in the capable hands of the machine. And, as has been mentioned on other Go versus D threads, a lot of this is a matter of taste and team dynamics. I've worked on many teams in my career where there was not a critical mass of people who could reason correctly about C++ memory management, and who could use generic programming techniques reliably. And, in line with discoveries from psychological research, it's common that people who are not competent at something do not recognize that fact. Perhaps I'm above average in this regard: I KNOW I'm not smart enough to write correct code using templating techniques. :) -jeffThis is why we already did quite a few consultancy projects that replaced C++ coded servers by JVM/.NET ones with comparable performance. Despite what many anti-GC naysayers say, if you code your data strucutures and algorithms to be GC friendly, coupled with the current state of art JIT compilers, one achieves similar performance coupled with safety. It doesn't need to extract every ms out of the hardware as C and C++ developers always try to do, but to be fast enough to be able to fulfill the task. -- Paulo
Dec 05 2013
On 12/5/2013 7:27 AM, Paulo Pinto wrote:It doesn't need to extract every ms out of the hardware as C and C++ developers always try to do, but to be fast enough to be able to fulfill the task.Although C and C++ are capable of extracting every ms out of the hardware as far as any compiled language goes, the number of programmers who are actually able to get those results are small. Just because someone writes a program in C or C++ doesn't at all mean that they're getting the most out of the machine. Reminds me of when I worked in a metal shop at Caltech fabricating parts. I'd produce crummy, ugly things. The resident machinist, with the same milling machine, would tweak it here and there and produce things of beauty. (That old bridgeport milling machine had more knobs, levers, and wheels than a steam locomotive cockpit, and none of them were labelled.)
Dec 05 2013
On Thursday, 5 December 2013 at 19:58:17 UTC, Walter Bright wrote:On 12/5/2013 7:27 AM, Paulo Pinto wrote:Fully agree. Additionally there seems to be a contiguous disease when using those languages, where one tries to micro-optimize every code line as it gets written. -- PauloIt doesn't need to extract every ms out of the hardware as C and C++ developers always try to do, but to be fast enough to be able to fulfill the task.Although C and C++ are capable of extracting every ms out of the hardware as far as any compiled language goes, the number of programmers who are actually able to get those results are small. Just because someone writes a program in C or C++ doesn't at all mean that they're getting the most out of the machine.
Dec 05 2013
On 06/12/13 08:58, Paulo Pinto wrote:Additionally there seems to be a contiguous disease when using those languages, where one tries to micro-optimize every code line as it gets written.In a way I think it's a disease that one has to go through with those languages, because in the end it usually leads to a few optimization insights (or "don't do this" insights) combined with the realization that it's much more productive to focus on algorithms and to let the compiler get on with its job, which it usually does better than any human.
Dec 06 2013
On Friday, 6 December 2013 at 07:58:46 UTC, Paulo Pinto wrote:On Thursday, 5 December 2013 at 19:58:17 UTC, Walter Bright wrote:On 12/5/2013 7:27 AM, Paulo Pinto wrote:Additionally there seems to be a contiguous disease when using those languages, where one tries to micro-optimize every code line as it gets written.add to that the optimization of number of keyboard strokes :D
Dec 06 2013
On 06/12/13 16:52, eles wrote:add to that the optimization of number of keyboard strokes :DHah, I got burned by that once in a very amusing way. I wrote a blog post comparing an existing C implementation of some data structures and algorithms (not mine) to my new D implementation. One reader on Reddit assumed the terse variable names of the C code extracts (not mine!) must be representative of all the code in the blog post and refused to read past the first paragraph. He was very nice when he realized his mistake, though. :-)
Dec 06 2013
Am 06.12.2013 16:52, schrieb eles:On Friday, 6 December 2013 at 07:58:46 UTC, Paulo Pinto wrote:Canonical Go seems to suffer from that as well, in terms of variable names.On Thursday, 5 December 2013 at 19:58:17 UTC, Walter Bright wrote:On 12/5/2013 7:27 AM, Paulo Pinto wrote:Additionally there seems to be a contiguous disease when using those languages, where one tries to micro-optimize every code line as it gets written.add to that the optimization of number of keyboard strokes :D
Dec 06 2013
On Thursday, 5 December 2013 at 14:55:53 UTC, Jeff R. Allen wrote:I am the buddy who's always going on and on about Go. Here's the blog posting I made explaining why/how I made the MQTT server in Go and what I learned: http://blog.nella.org/mqtt-code-golf/ I was a bit surprised and interested to see that MQTT with small transactions is not easy to optimize because kernel/user context switching dominates the work that's done under control of the programmer and which can be optimized via data structures and/or reducing GC overhead. Something that both Atila and I verified in this exercise is that writing a fast network server that scales is so much easier with a very good networking library that takes advantage of lightweight threads, and with a runtime that provides GC. This frees up the programmer to focus on the protocol logic, and leave concurrency and bookkeeping in the capable hands of the machine. And, as has been mentioned on other Go versus D threads, a lot of this is a matter of taste and team dynamics. I've worked on many teams in my career where there was not a critical mass of people who could reason correctly about C++ memory management, and who could use generic programming techniques reliably. And, in line with discoveries from psychological research, it's common that people who are not competent at something do not recognize that fact. Perhaps I'm above average in this regard: I KNOW I'm not smart enough to write correct code using templating techniques. :) -jeffTemplates are easy. They are pure functions. You just need to overcome idiosyncrasies introduced by people who thought they were something else. ;)
Dec 05 2013
On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:It all started here: http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar forum.dlang.org Not wanting to be outdone, my buddy Jeff (the Go guy) went and wrote a new benchmark. My implementation didn't do as well on that one so I went back to the code and optimised it. Also, our other buddy Patrick from work decided to get in on this and use it as an excuse to learn Erlang. So now 4 different implementations in 4 different languages are duking it out. There are two benchmarks, both written in Go. The first is loadtest and it measures throughput. I varied the number of client connections to the server and measured how many messages were sent per second. The second benchmark is pingtest and it measures latency. The quantity I varied here was the number of wildcard subscribers (it'd take a while to explain what they are to anyone who isn't familiar with MQTT). It has a bunch of client connections talk to each other, and instead of trying to send as many messages as computerly possible they wait for the acknowledgement from the partner connection. The results are as follows (I tried making charts but LibreOffice wasn't helping and then I got bored), with the numbers being thousands of messages per second: loadtest Connections: 100 500 750 1k D + vibe.d: 121.7 +/- 1.5 166.9 +/- 1.5 171.1 +/- 3.3 167.9 +/- 1.3 C (mosquitto): 106.1 +/- 0.8 122.4 +/- 0.4 95.2 +/- 1.3 74.7 +/- 0.4 Erlang: 104.1 +/- 2.2 124.2 +/- 5.9 117.6 +/- 4.6 117.7 +/- 3.2 Go: 90.9 +/- 11 100.1 +/- 0.1 99.3 +/- 0.2 98.8 +/- 0.3 pingtest wsubs: 20 200 400 D + vibe.d: 50.9 +/- 0.3 38.3 +/- 0.2 20.1 +/- 0.1 C (mosquitto): 65.4 +/- 4.4 45.2 +/- 0.2 20.0 +/- 0.0 Erlang: 49.1 +/- 0.8 30.9 +/- 0.3 15.6 +/- 0.1 Go: 45.2 +/- 0.2 27.5 +/- 0.1 16.0 +/- 0.1 So, D was faster than the other contenders by far in throughput, 2nd place losing to the C implementation on latency. I'm still not sure why that is. Profiling in this case is tricky. I'm pretty sure the profiler is still ticking away when a fiber yields - the top function is the one that reads from the network, which I can't do much about. GC was pretty much a non-issue, which I'm both surprised at and happy about. I allocated as much memory as I wanted at first, and by the time I tried to optmise memory usage by allocating less all I managed to eke out in performance terms was a paltry extra 8% or so. All in all though, so much for his "nothing is going to be faster than Go because this is what it's good at". If he'd measured it against mosquitto first and realised it was already losing to C I probably wouldn't have written an MQTT broker in D, so I guess that was a good thing. :) Atila P.S. vibe.d is awesome, although I wish it'd compile with ldc or gdcSo either D is faster or you are a better coder than your buddies!
Dec 04 2013
So either D is faster or you are a better coder than your buddies!I'd say we're all the same skill level. One of them sits next to me and I heard about his Erlang implementation as it was happening. It's probable that certain techniques could be reused from another program but it's hard to tell. The easiest way would be to have another person write a different implementation in the same language and compare the two. It might happen, another colleague has been working on another Erlang implementation as well, but he's taking a while.
Dec 04 2013
On 04/12/13 13:49, Atila Neves wrote:So, D was faster than the other contenders by far in throughput, 2nd place losing to the C implementation on latency. I'm still not sure why that is. Profiling in this case is tricky. I'm pretty sure the profiler is still ticking away when a fiber yields - the top function is the one that reads from the network, which I can't do much about.What about the relative elegance/maintainability/ease of comprehension of the different solutions? Playing devil's advocate for a moment, I can well understand if a preference for one language over another was decided on the basis of its performance being good _enough_ and the code being really easy to work with, rather than simply the best performer.
Dec 04 2013
On Wednesday, 4 December 2013 at 15:02:44 UTC, Joseph Rushton Wakeling wrote:On 04/12/13 13:49, Atila Neves wrote:I can't read Erlang (yet) so I don't know about that one. The C code is... well C code so not great. I have a profound dislike for Go but I'd say the Go and D implementations are just as readable. Then again, I wrote the D code so if I didn't think that was readable... :P AtilaSo, D was faster than the other contenders by far in throughput, 2nd place losing to the C implementation on latency. I'm still not sure why that is. Profiling in this case is tricky. I'm pretty sure the profiler is still ticking away when a fiber yields - the top function is the one that reads from the network, which I can't do much about.What about the relative elegance/maintainability/ease of comprehension of the different solutions? Playing devil's advocate for a moment, I can well understand if a preference for one language over another was decided on the basis of its performance being good _enough_ and the code being really easy to work with, rather than simply the best performer.
Dec 04 2013
On 12/4/13 7:41 AM, Atila Neves wrote:On Wednesday, 4 December 2013 at 15:02:44 UTC, Joseph Rushton Wakeling wrote:How do they compare in lines of code? AndreiOn 04/12/13 13:49, Atila Neves wrote:I can't read Erlang (yet) so I don't know about that one. The C code is... well C code so not great. I have a profound dislike for Go but I'd say the Go and D implementations are just as readable. Then again, I wrote the D code so if I didn't think that was readable... :PSo, D was faster than the other contenders by far in throughput, 2nd place losing to the C implementation on latency. I'm still not sure why that is. Profiling in this case is tricky. I'm pretty sure the profiler is still ticking away when a fiber yields - the top function is the one that reads from the network, which I can't do much about.What about the relative elegance/maintainability/ease of comprehension of the different solutions? Playing devil's advocate for a moment, I can well understand if a preference for one language over another was decided on the basis of its performance being good _enough_ and the code being really easy to work with, rather than simply the best performer.
Dec 04 2013
That's hard to measure. Not only because of the standard fare about not counting empty lines and comments, but because of some other issues as well. The Erlang implementation has the unit tests in the same files so it's all intertwined. The Go implementation makes use of a repository on github, should that be counted? In my own case I used my D serialisation library and also vibe.d. I would say neither should count towards the final tally, but others may beg to differ. All in all the MQTT specific parts in all 3 projects weigh in at around 500 - 700 total lines as measured by wc -l, if memory serves me correctly. I'm on another computer now and pressed for time. I wouldn't be surprised if the Erlang version was the shortest at the end of the day. I know that the Go and D implementations are comparable. AtilaHow do they compare in lines of code?What about the relative elegance/maintainability/ease of comprehension of the different solutions? Playing devil's advocate for a moment, I can well understand if a preference for one language over another was decided on the basis of its performance being good _enough_ and the code being really easy to work with, rather than simply the best performer.I can't read Erlang (yet) so I don't know about that one. The C code is... well C code so not great. I have a profound dislike for Go but I'd say the Go and D implementations are just as readable. Then again, I wrote the D code so if I didn't think that was readable... :P
Dec 04 2013
On 12/4/13 4:49 AM, Atila Neves wrote:It all started here: http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar forum.dlang.org Not wanting to be outdone, my buddy Jeff (the Go guy) went and wrote a new benchmark. My implementation didn't do as well on that one so I went back to the code and optimised it. Also, our other buddy Patrick from work decided to get in on this and use it as an excuse to learn Erlang. So now 4 different implementations in 4 different languages are duking it out.[snip] Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too. Andrei
Dec 04 2013
Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.If I convert this post into a blog it's going to take more than a little adjustment for me to be happy with it. I guess having had to write a PhD thesis has now scarred me for life. I'll see if I get around to it in the next few days and I'll ask the guys about their implementations as well. I guess I should go set up wordpress. Atila
Dec 04 2013
On 2013-12-04 12:59 PM, Atila Neves wrote:It would be interesting to have a GitHub repo with the scripts for each language, vibe.d also has a multithreaded version that is off by default (fibers created from new tcp connections are distributed in a thread pool where communication is done through Isolated!()). So I'd love to be able to fork it and rerun this with some adjustments.Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.If I convert this post into a blog it's going to take more than a little adjustment for me to be happy with it. I guess having had to write a PhD thesis has now scarred me for life. I'll see if I get around to it in the next few days and I'll ask the guys about their implementations as well. I guess I should go set up wordpress. Atila
Dec 04 2013
It would be interesting to have a GitHub repo with the scripts for each language,I might set this up after / because of the blog post. We'll see how it goes.vibe.d also has a multithreaded version that is off by default (fibers created from new tcp connections are distributed in a thread pool where communication is done through Isolated!()).Oh, I know. I tried it. The threads spent more time sending messages to each other than doing actual work. And that was before I changed the data structure for the subscriptions so it did even less work. The problem (at least with those 2 benchmarks) just isn't amenable to be made faster by throwing threads at it. It wasn't just the D/vibe.d version. The Go and Erlang versions were tried with multiple threads as well. They either stayed at the same speed or ran slower. Atila
Dec 04 2013
On 12/4/13 9:59 AM, Atila Neves wrote:Yah, I was just trying to lure you into starting on it until it's too late to change your mind :o).Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.If I convert this post into a blog it's going to take more than a little adjustment for me to be happy with it. I guess having had to write a PhD thesis has now scarred me for life.I'll see if I get around to it in the next few days and I'll ask the guys about their implementations as well. I guess I should go set up wordpress. AtilaThis is terrific. Would make for a sure hit and a lot of interesting discussion on reddit/hackernews. Thanks, Andrei
Dec 04 2013
Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.Ask, and you shall receive: https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/
Dec 05 2013
On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:_very_ well-written. I can only dream every other benchmarking article out there would have been done with same attention to details :) http://www.reddit.com/r/programming/comments/1s5ze3/benchmarking_d_vs_go_vs_erlang_vs_c_for_mqtt/Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.Ask, and you shall receive: https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/
Dec 05 2013
On 12/5/13 7:46 AM, Dicebot wrote:On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:Fasten your seatbelts, it's gonna be a bumpy ride! Andrei_very_ well-written. I can only dream every other benchmarking article out there would have been done with same attention to details :) http://www.reddit.com/r/programming/comments/1s5ze3/benchmarking_d_vs_go_vs_erlang_vs_c_for_mqtt/Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.Ask, and you shall receive: https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/
Dec 05 2013
On 12/5/13 7:46 AM, Dicebot wrote:On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:https://news.ycombinator.com/item?id=6855278 https://twitter.com/D_Programming/status/408634007052492800 Andrei_very_ well-written. I can only dream every other benchmarking article out there would have been done with same attention to details :) http://www.reddit.com/r/programming/comments/1s5ze3/benchmarking_d_vs_go_vs_erlang_vs_c_for_mqtt/Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.Ask, and you shall receive: https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/
Dec 05 2013
https://news.ycombinator.com/item?id=6855278 https://twitter.com/D_Programming/status/408634007052492800Yeah, I thought that would happen. :) Thanks! Anyone care to guess how many comments it will take to get to name calling? Atila
Dec 05 2013
On Thursday, 5 December 2013 at 16:40:06 UTC, Andrei Alexandrescu wrote:On 12/5/13 7:46 AM, Dicebot wrote:This might be the first time that I consider the reddit comments of higher quality than HN. I completely left reddit for HN a few years ago, but in the last months I see myself drifting back. Oh, and thanks for the great article. :)On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves http://www.reddit.com/r/programming/comments/1s5ze3/benchmarking_d_vs_go_vs_erlang_vs_c_for_mqtt/https://news.ycombinator.com/item?id=6855278
Dec 06 2013
On Thu, 05 Dec 2013 16:24:29 +0100, Atila Neves wrote:c-in-real-life-mqtt-broker-implementation-shootout/ Thanks, excellent write-up.Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.Ask, and you shall receive: https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-
Dec 05 2013
On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:Very nice article! I'm pretty happy to see D doing so well on this. "pingtest (latency - bigger is better)" Did you mean "lower is better"? Or is this not measuring time-taken?Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.Ask, and you shall receive: https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/
Dec 05 2013
Very nice article! I'm pretty happy to see D doing so well on this. "pingtest (latency - bigger is better)" Did you mean "lower is better"? Or is this not measuring time-taken?No, it's correct. Instead of measuring the round-trip time I measured how many messages per second were sent. They're equivalent measures, I just preferred this one, which also makes it easy to compare with the other benchmark. Atila
Dec 05 2013
On Thursday, 5 December 2013 at 16:34:11 UTC, Atila Neves wrote:Oh, okay, I see. Thanks :)Very nice article! I'm pretty happy to see D doing so well on this. "pingtest (latency - bigger is better)" Did you mean "lower is better"? Or is this not measuring time-taken?No, it's correct. Instead of measuring the round-trip time I measured how many messages per second were sent. They're equivalent measures, I just preferred this one, which also makes it easy to compare with the other benchmark. Atila
Dec 05 2013
On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:"Go is an opinionated language" It's funny, but I got the same impression after reading about it and looking at the syntax. One of D's advantages is - as I've said before - the freedom you have. When I first started with D, I did Java/Objective-C style things (classes etc.), and it was possible and worked well. As I went along I learned more about structs, templates and everything and I started to "think outside the box", because D is not opinionated. I enjoy coding much more now and it's good when a language gives you freedom instead of telling you what to do. In Java: 1. Write a class 2. Now think ... Objective-C: 1. Think, but write a class anyway. 2. Think how to bypass classes. In D: 1. Think 2. Write a struct, class, template ... whatever is appropriate. "In the end the choice of algorithm and data structures matter more than the programming language so my personal advice is to choose the language that makes you productive." And D actually gives you a choice.Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.Ask, and you shall receive: https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/
Dec 06 2013
On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:P.S. vibe.d is awesome, although I wish it'd compile with ldc or gdcWhat are the issues that are blocking you from using it with LDC? The 2.064 frontend not being officially yet? I couldn't find any mention of vibe.d in the open bugs on our GitHub tracker (besides agree that vibe.d would be a very interesting target for LDC, there is little chance of the situation improving. David
Dec 04 2013
On Wednesday, 4 December 2013 at 18:05:11 UTC, David Nadlinger wrote:What are the issues that are blocking you from using it with LDC? The 2.064 frontend not being officially yet? I couldn't find any mention of vibe.d in the open bugs on our GitHub So even though I agree that vibe.d would be a very interesting target for LDC, there is little chance of the situation improving.Quick search has found these issues: dub: https://github.com/rejectedsoftware/dub/issues/172 vibe.d: https://github.com/rejectedsoftware/vibe.d/issues/351 I remember also someone mentioning crash during fiber switch when using LDC but can't find that place right now.
Dec 04 2013
On 4 December 2013 18:09, Dicebot <public dicebot.lv> wrote:On Wednesday, 4 December 2013 at 18:05:11 UTC, David Nadlinger wrote:Ditto with GDC too... I did a quick search on those two project and found that people had reported supposed bugs in GDC, but again none of which have been raised in GDC itself. Regards Iain.What are the issues that are blocking you from using it with LDC? The 2.064 frontend not being officially yet? I couldn't find any mention of the unstable Win64 port). So even though I agree that vibe.d would be a very interesting target for LDC, there is little chance of the situation improving.Quick search has found these issues: dub: https://github.com/rejectedsoftware/dub/issues/172 vibe.d: https://github.com/rejectedsoftware/vibe.d/issues/351 I remember also someone mentioning crash during fiber switch when using LDC but can't find that place right now.
Dec 04 2013
On Wednesday, 4 December 2013 at 19:05:12 UTC, Iain Buclaw wrote:Ditto with GDC too... I did a quick search on those two project and found that people had reported supposed bugs in GDC, but again none of which have been raised in GDC itself.I think main issue is that no one was able to reduce those low enough for reporting. And yeah, that fiber switch issues is actually one for GDC : https://github.com/rejectedsoftware/vibe.d/issues/101
Dec 04 2013
On 2013-12-04 1:05 PM, David Nadlinger wrote:On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:I tried on linux x64 about 2 weeks ago and it's not vibe.d itself but the subpackages like vibelog, userman, or vibed.org, the issues were with std.range I think. I can start sending bug reports in, it would be great to see this working.P.S. vibe.d is awesome, although I wish it'd compile with ldc or gdcWhat are the issues that are blocking you from using it with LDC? The 2.064 frontend not being officially yet? I couldn't find any mention of concerns the unstable Win64 port). So even though I agree that vibe.d would be a very interesting target for LDC, there is little chance of the situation improving. David
Dec 04 2013
On Wednesday, 4 December 2013 at 18:13:54 UTC, Etienne wrote:I can start sending bug reports in, it would be great to see this working.Yes, please! David
Dec 04 2013
On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:It all started here: http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar forum.dlang.org Not wanting to be outdone, my buddy Jeff (the Go guy) went and wrote a new benchmark. My implementation didn't do as well on that one so I went back to the code and optimised it. Also, our other buddy Patrick from work decided to get in on this and use it as an excuse to learn Erlang. So now 4 different implementations in 4 different languages are duking it out. There are two benchmarks, both written in Go. The first is loadtest and it measures throughput. I varied the number of client connections to the server and measured how many messages were sent per second. The second benchmark is pingtest and it measures latency. The quantity I varied here was the number of wildcard subscribers (it'd take a while to explain what they are to anyone who isn't familiar with MQTT). It has a bunch of client connections talk to each other, and instead of trying to send as many messages as computerly possible they wait for the acknowledgement from the partner connection. The results are as follows (I tried making charts but LibreOffice wasn't helping and then I got bored), with the numbers being thousands of messages per second: loadtest Connections: 100 500 750 1k D + vibe.d: 121.7 +/- 1.5 166.9 +/- 1.5 171.1 +/- 3.3 167.9 +/- 1.3 C (mosquitto): 106.1 +/- 0.8 122.4 +/- 0.4 95.2 +/- 1.3 74.7 +/- 0.4 Erlang: 104.1 +/- 2.2 124.2 +/- 5.9 117.6 +/- 4.6 117.7 +/- 3.2 Go: 90.9 +/- 11 100.1 +/- 0.1 99.3 +/- 0.2 98.8 +/- 0.3 pingtest wsubs: 20 200 400 D + vibe.d: 50.9 +/- 0.3 38.3 +/- 0.2 20.1 +/- 0.1 C (mosquitto): 65.4 +/- 4.4 45.2 +/- 0.2 20.0 +/- 0.0 Erlang: 49.1 +/- 0.8 30.9 +/- 0.3 15.6 +/- 0.1 Go: 45.2 +/- 0.2 27.5 +/- 0.1 16.0 +/- 0.1 So, D was faster than the other contenders by far in throughput, 2nd place losing to the C implementation on latency. I'm still not sure why that is. Profiling in this case is tricky. I'm pretty sure the profiler is still ticking away when a fiber yields - the top function is the one that reads from the network, which I can't do much about. GC was pretty much a non-issue, which I'm both surprised at and happy about. I allocated as much memory as I wanted at first, and by the time I tried to optmise memory usage by allocating less all I managed to eke out in performance terms was a paltry extra 8% or so. All in all though, so much for his "nothing is going to be faster than Go because this is what it's good at". If he'd measured it against mosquitto first and realised it was already losing to C I probably wouldn't have written an MQTT broker in D, so I guess that was a good thing. :) Atila P.S. vibe.d is awesome, although I wish it'd compile with ldc or gdcNice benchmark! :) So, a DMD-compiled D program vs GCC-compiled (?) C program fighting for latency? That might be a reason. Interesting we still win with throughput. The variance/standard deviation of pingtest-C-20 with 4.4 stands out. What happens there?
Dec 05 2013
So, a DMD-compiled D program vs GCC-compiled (?) C program fighting for latency? That might be a reason. Interesting we still win with throughput.Yeah, the C version was compiled with gcc 4.8.2. I'd've used either ldc or gdc with my version but: . vibe.d didn't even compile with ldc. . The gdc in the Arch Linux package couldn't compile anything that included one of the files in std, I can't remember which now, for most of the time I worked on this. . When the gdc package got fixed, I managed to compile my app with vibe.d. It crashed as soon as I tried using it.The variance/standard deviation of pingtest-C-20 with 4.4 stands out. What happens there?Can't remember. Atila
Dec 05 2013