www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Walter's DConf 2014 Talks - Topics in Finance

reply "TJB" <broughtj gmail.com> writes:
Walter,

I see that you will be discussing "High Performance Code Using D" 
at the 2014 DConf. This will be a very welcomed topic for many of 
us.  I am a Finance Professor.  I currently teach and do research 
in computational finance.  Might I suggest that you include some 
finance (say Monte Carlo options pricing) examples?  If you can 
get the finance industry interested in D you might see a massive 
adoption of the language.  Many are desperate for an alternative 
to C++ in that space.

Just a thought.

Best,

TJB
Mar 21 2014
next sibling parent reply "Joakim" <joakim airpost.net> writes:
On Friday, 21 March 2014 at 21:14:15 UTC, TJB wrote:
 Walter,

 I see that you will be discussing "High Performance Code Using 
 D" at the 2014 DConf. This will be a very welcomed topic for 
 many of us.  I am a Finance Professor.  I currently teach and 
 do research in computational finance.  Might I suggest that you 
 include some finance (say Monte Carlo options pricing) 
 examples?  If you can get the finance industry interested in D 
 you might see a massive adoption of the language.  Many are 
 desperate for an alternative to C++ in that space.

Heh, right before I read this, I stumbled across this snippet from Miguel De Icaza's blog from a couple months back, where he regretted using C++ to build Moonlight, their Silverlight implementation: "But this would not be a Saturday blog post without pointing out that Cairo's C-based API is easier and simpler to use than many of those C++ libraries out there. The more sophisticated the use of the C++ language to get some performance benefit, the more unpleasant the API is to use. The incredibly powerful Antigrain sports an insanely fast software renderer and also a quite hostile template-based API. We got to compare Antigrain and Cairo back when we worked on Moonlight. Cairo was the clear winner. We built Moonlight in C++ for all the wrong reasons ("better performance", "memory usage") and was a decision we came to regret. Not only were the reasons wrong, it is not clear we got any performance benefit and it is clear that we did worse with memory usage. But that is a story for another time." http://tirania.org/blog/archive/2014/Jan-04.html
Mar 21 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 21.03.2014 22:39, schrieb w0rp:
 On Friday, 21 March 2014 at 21:30:29 UTC, Joakim wrote:
 Heh, right before I read this, I stumbled across this snippet from
 Miguel De Icaza's blog from a couple months back, where he regretted
 using C++ to build Moonlight, their Silverlight implementation:

 "But this would not be a Saturday blog post without pointing out that
 Cairo's C-based API is easier and simpler to use than many of those
 C++ libraries out there. The more sophisticated the use of the C++
 language to get some performance benefit, the more unpleasant the API
 is to use.

 The incredibly powerful Antigrain sports an insanely fast software
 renderer and also a quite hostile template-based API.

 We got to compare Antigrain and Cairo back when we worked on
 Moonlight. Cairo was the clear winner.

 We built Moonlight in C++ for all the wrong reasons ("better
 performance", "memory usage") and was a decision we came to regret.
 Not only were the reasons wrong, it is not clear we got any
 performance benefit and it is clear that we did worse with memory usage.

 But that is a story for another time."
 http://tirania.org/blog/archive/2014/Jan-04.html

That is a really strange argument. Let's break it down into stages. 1. Use C++ for better performance. 2. Find C++ library with better performance, but it's ugly. 3. Use C library in C++ instead because it's less ugly. 4. Conclude that C++ can't deliver better performance. That is really weak. This is why the industry needs salvation from C++ with D. It would mostly be then, "Oh it has better performance with these template things... and it's not a pain in the ass to use."

I would take Miguel's comments about C++ with a grain of salt. Back when I participated in Gtkmm (early 200x), there were the occasional C++ bashes coming from the Gtk+ guys and Miguel was never found of C++. -- Paulo
Mar 21 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Friday, 21 March 2014 at 21:30:29 UTC, Joakim wrote:
 Heh, right before I read this, I stumbled across this snippet 
 from Miguel De Icaza's blog from a couple months back, where he 
 regretted using C++ to build Moonlight, their Silverlight 
 implementation:

 "But this would not be a Saturday blog post without pointing 
 out that Cairo's C-based API is easier and simpler to use than 
 many of those C++ libraries out there. The more sophisticated 
 the use of the C++ language to get some performance benefit, 
 the more unpleasant the API is to use.

 The incredibly powerful Antigrain sports an insanely fast 
 software renderer and also a quite hostile template-based API.

 We got to compare Antigrain and Cairo back when we worked on 
 Moonlight. Cairo was the clear winner.

 We built Moonlight in C++ for all the wrong reasons ("better 
 performance", "memory usage") and was a decision we came to 
 regret. Not only were the reasons wrong, it is not clear we got 
 any performance benefit and it is clear that we did worse with 
 memory usage.

 But that is a story for another time."
 http://tirania.org/blog/archive/2014/Jan-04.html

That is a really strange argument. Let's break it down into stages. 1. Use C++ for better performance. 2. Find C++ library with better performance, but it's ugly. 3. Use C library in C++ instead because it's less ugly. 4. Conclude that C++ can't deliver better performance. That is really weak. This is why the industry needs salvation from C++ with D. It would mostly be then, "Oh it has better performance with these template things... and it's not a pain in the ass to use."
Mar 21 2014
prev sibling next sibling parent "TJB" <broughtj gmail.com> writes:
On Friday, 21 March 2014 at 21:39:54 UTC, w0rp wrote:
 On Friday, 21 March 2014 at 21:30:29 UTC, Joakim wrote:
 Heh, right before I read this, I stumbled across this snippet 
 from Miguel De Icaza's blog from a couple months back, where 
 he regretted using C++ to build Moonlight, their Silverlight 
 implementation:

 "But this would not be a Saturday blog post without pointing 
 out that Cairo's C-based API is easier and simpler to use than 
 many of those C++ libraries out there. The more sophisticated 
 the use of the C++ language to get some performance benefit, 
 the more unpleasant the API is to use.

 The incredibly powerful Antigrain sports an insanely fast 
 software renderer and also a quite hostile template-based API.

 We got to compare Antigrain and Cairo back when we worked on 
 Moonlight. Cairo was the clear winner.

 We built Moonlight in C++ for all the wrong reasons ("better 
 performance", "memory usage") and was a decision we came to 
 regret. Not only were the reasons wrong, it is not clear we 
 got any performance benefit and it is clear that we did worse 
 with memory usage.

 But that is a story for another time."
 http://tirania.org/blog/archive/2014/Jan-04.html

That is a really strange argument. Let's break it down into stages. 1. Use C++ for better performance. 2. Find C++ library with better performance, but it's ugly. 3. Use C library in C++ instead because it's less ugly. 4. Conclude that C++ can't deliver better performance. That is really weak. This is why the industry needs salvation from C++ with D. It would mostly be then, "Oh it has better performance with these template things... and it's not a pain in the ass to use."

The removal of pain points is indeed, in my mind, the main issue. :-)
Mar 21 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/21/2014 2:14 PM, TJB wrote:
 I see that you will be discussing "High Performance Code Using D" at the 2014
 DConf. This will be a very welcomed topic for many of us.  I am a Finance
 Professor.  I currently teach and do research in computational finance.  Might
I
 suggest that you include some finance (say Monte Carlo options pricing)
 examples?  If you can get the finance industry interested in D you might see a
 massive adoption of the language.  Many are desperate for an alternative to C++
 in that space.

It's a good thought, but I have zero knowledge of how C++ is used for high frequency trading.
Mar 21 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/21/2014 3:33 PM, TJB wrote:
 I would be happy to help you with an option pricing example that
 is commonly used.  Let me know if you are interested.

Sure, please email it to me.
Mar 23 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/23/2014 11:10 AM, TJB wrote:
 Walter, I would be happy to.  Where do I find your email address?  Sorry if
this
 is a dumb question.

My first name followed by digitalmars.com.
Mar 23 2014
prev sibling parent evansl <cppljevans suddenlink.net> writes:
On 03/22/14 06:40, Russel Winder wrote:
[snip]
 What you are alluding to is the use of Monte Carlo approach to solve
 some of the models given boundary conditions. This is a "bog standard"

http://en.wiktionary.org/wiki/bog_standard
 approach to numerical modelling.

 Many of the hedge funds are following in the HEP,

By HEP do you mean "High Energy Physics"? [snip]
 In a number of the major international finance houses a
 Python/Scala/C++ stack has taken hold. However this is not fixed
 forever, since whenever the CTO changes the stack tends to as well.

By CTO do you mean "Chief Technology Officer"? TIA. -regards, Larry
Mar 23 2014
prev sibling next sibling parent "TJB" <broughtj gmail.com> writes:
On Friday, 21 March 2014 at 22:28:36 UTC, Walter Bright wrote:
 On 3/21/2014 2:14 PM, TJB wrote:
 I see that you will be discussing "High Performance Code Using 
 D" at the 2014
 DConf. This will be a very welcomed topic for many of us.  I 
 am a Finance
 Professor.  I currently teach and do research in computational 
 finance.  Might I
 suggest that you include some finance (say Monte Carlo options 
 pricing)
 examples?  If you can get the finance industry interested in D 
 you might see a
 massive adoption of the language.  Many are desperate for an 
 alternative to C++
 in that space.

It's a good thought, but I have zero knowledge of how C++ is used for high frequency trading.

I would be happy to help you with an option pricing example that is commonly used. Let me know if you are interested.
Mar 21 2014
prev sibling next sibling parent "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Friday, 21 March 2014 at 22:28:36 UTC, Walter Bright wrote:
 It's a good thought, but I have zero knowledge of how C++ is 
 used for high frequency trading.

Reading through the Wikipedia article on Computational Finance, it looks like it's basically performing simulations where some data is known but other is not. Random numbers are generated for the unknown data and the simulations are run several times to find the range of possible outcomes given the known values.
Mar 21 2014
prev sibling next sibling parent reply "Daniel Davidson" <nospam spam.com> writes:
On Friday, 21 March 2014 at 21:14:15 UTC, TJB wrote:
 Walter,

 I see that you will be discussing "High Performance Code Using 
 D" at the 2014 DConf. This will be a very welcomed topic for 
 many of us.  I am a Finance Professor.  I currently teach and 
 do research in computational finance.  Might I suggest that you 
 include some finance (say Monte Carlo options pricing) 
 examples?  If you can get the finance industry interested in D 
 you might see a massive adoption of the language.  Many are 
 desperate for an alternative to C++ in that space.

 Just a thought.

 Best,

 TJB

Maybe a good starting point would be to port some of QuantLib and see how the performance compares. In High Frequency Trading I think D would be a tough sell, unfortunately. Thanks Dan
Mar 21 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/21/2014 5:39 PM, bearophile wrote:
 That code must always be hard-real time. So a GC is allowed only during startup
 time (unless it's a quite special GC), hidden heap allocations are forbidden,
 data access patterns need to be carefully chosen, you even have to use most of
 the hot part of the stack, etc.

These are all very doable in D, and I've written apps that do so. The "fear of GC" is completely overblown. However, I do recognize that Phobos can be made a lot more non-GC friendly, and I am working on PR's to do that. For example, https://github.com/D-Programming-Language/phobos/pull/2014
Mar 21 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 22.03.2014 06:58, schrieb deadalnix:
 On Saturday, 22 March 2014 at 01:24:38 UTC, Walter Bright wrote:
 On 3/21/2014 5:39 PM, bearophile wrote:
 That code must always be hard-real time. So a GC is allowed only
 during startup
 time (unless it's a quite special GC), hidden heap allocations are
 forbidden,
 data access patterns need to be carefully chosen, you even have to
 use most of
 the hot part of the stack, etc.

These are all very doable in D, and I've written apps that do so. The "fear of GC" is completely overblown.

Fear of GC is overblown. Fear of D's GC isn't.

Yes, as there are a few high performance trading systems done with JVM/.NET languages. -- Paulo
Mar 22 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 22.03.2014 09:42, schrieb Ziad Hatahet:
 On Sat, Mar 22, 2014 at 12:30 AM, Paulo Pinto <pjmlp progtools.org
 <mailto:pjmlp progtools.org>> wrote:

     Yes, as there are a few  high performance trading systems done with
     JVM/.NET languages.

     --
     Paulo


 What about AAA games? :) Even though I agree with the pro-GC arguments
 you put forth, but I really have a hard time imagining something like
 Battlefield 4 written in Java and running on the JVM (though I would
 love to be proven wrong).

What about ballistic missile tracking radar control systems? http://www.bloomberg.com/apps/news?pid=newsarchive&sid=aHLFBJZrqaoM You don't want those babies out of control. -- Paulo
Mar 22 2014
prev sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 22.03.2014 13:38, schrieb Daniel Davidson:
 On Saturday, 22 March 2014 at 11:46:43 UTC, Russel Winder wrote:

 It is also worth pointing out the LMAX Disruptor which is a lock-free
 ring buffer based framework used to create dealing platforms on the JVM.
 They outperform any other trading platform still.

That is wrong. Trading is competitive and your competitors rarely will inform you they are losing to you or beating you. And when you do not get the fills you want it is not always clear you are being beaten. This article (http://mechanical-sympathy.blogspot.com/2011/08/inter-thread-latency.html) suggests C++ version of the same architecture is faster than the Java. Thanks Dan

Maybe this is a wrong conclusion, but from the comments, I would say the Java version is pretty much on par with the C++ attempts. -- Paulo
Mar 22 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 22.03.2014 14:21, schrieb Daniel Davidson:
 On Saturday, 22 March 2014 at 12:54:11 UTC, Paulo Pinto wrote:
 Am 22.03.2014 13:38, schrieb Daniel Davidson:
 On Saturday, 22 March 2014 at 11:46:43 UTC, Russel Winder wrote:

 It is also worth pointing out the LMAX Disruptor which is a lock-free
 ring buffer based framework used to create dealing platforms on the
 JVM.
 They outperform any other trading platform still.

That is wrong. Trading is competitive and your competitors rarely will inform you they are losing to you or beating you. And when you do not get the fills you want it is not always clear you are being beaten. This article (http://mechanical-sympathy.blogspot.com/2011/08/inter-thread-latency.html) suggests C++ version of the same architecture is faster than the Java. Thanks Dan

Maybe this is a wrong conclusion, but from the comments, I would say the Java version is pretty much on par with the C++ attempts.

I guess it depends where your edge comes from. If it is speed, the following suggests it is worth it to go C++. 10% is huge and reduction in variability is also huge. [[ So what does this all mean for the Disruptor? Basically, the latency of the Disruptor is about as low as we can get from Java. It would be possible to get a ~10% latency improvement by moving to C++. I’d expect a similar improvement in throughput for C++. The main win with C++ would be the control, and therefore, the predictability that comes with it if used correctly. The JVM gives us nice safety features like garbage collection in complex applications but we pay a little for that with the extra instructions it inserts that can be seen if you get Hotspot to dump the assembler instructions it is generating. ]] If your edge is quickly trying out new algorithms and identifying edge by trying many less latency sensitive strategies in the market - then "pretty much on par" is a fine bar. Thanks, Dan

I read a bit more into it, as the post is from 2011, there are some comments of bringing the performance findings into the following Disruptor's version and the discussion about code generation quality of different Java JIT compilers. So I would be curious how the 2014 version of Disruptor with modern JVMs like Hotspot, JRockit, Azul fare against the C++ version in modern C++ optimizing compilers like Intel C++, Visual C++. Going a bit off topic, as although I do like C++, the day job is in JVM/.NET land where we sometimes do replace C++ systems by the later ones, but I don't possess any trade experience besides what I get to read in blogs and articles. Assuming those 10% still happen if the test was done today as suggested, how much are trade companies willing to pay for developers to achieve those 10% in C++ vs having a system although 10% slower, still fast enough for operations while saving salaries for more cheaper developers? Asking, because on our enterprise consulting I see most companies willing to sacrifice performance vs salaries, with direct impact on the technology stack being used. -- Paulo
Mar 22 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/22/2014 7:29 AM, Russel Winder wrote:
 I guess D should be able to do things just as fast as C++, at least
 using LDC or GDC. My little informal microbenchmarks indicate that this
 is the case, but for now this is anecdotal evidence not statistically
 significant.

Having built C++ and D compilers, I know this to be true for a fact. It's what I'll be talking about at Dconf.
Mar 22 2014
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 23.03.2014 08:13, schrieb Russel Winder:
 On Sat, 2014-03-22 at 21:13 +0000, deadalnix wrote:
 […]
 HFT is very latency sensitive. D stop the world GC is a no go.

 D needs a better GC to be viable in these markets.

GC technology was well beyond "stop the world" in Common Lisp in the 1990s. Java learnt this lesson in the 2000s. IBM, Azul, and now OpenJDK have an array of very sophisticated, and indeed some real-time, garbage collectors. Clearly though the Lisp and Java worlds are very different from the C++, D and Go worlds, thus sophisticated algorithms cannot simply be transplanted. As I understand it, Go has a parallel mark and sweep algorithm which is relatively unsophisticated, and a bit "stop the world", but they claim fine for small heaps: generational and compacting collectors (as per Java) are only beneficial for large heaps is the claim (but I haven't seen much data to back this up for Go). There is rumour of a change, but this has been circulating for ages. I guess Herb Sutter's (reported) view that he can write C++ code with no new and so no need for garbage collection, if applied to D would mean that although the collector "stopped the world", the heap would need no work and be small. But for real time you would just have to remove the GC completely to have the needed guarantees.

Well, there is a nice quote from Bjarne as well: "C++ is my favourite garbage collected language because it generates so little garbage" And yet C++11 got to have an optional GC API defined, so even the ANSI/ISO folks do recognize its value in C++. Visual C++ already supports it, http://msdn.microsoft.com/en-us/library/vstudio/hh567368%28v=vs.120%29.aspx -- Paulo
Mar 23 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/23/2014 12:13 AM, Russel Winder wrote:
 But for real time you would just have to remove the
 GC completely to have the needed guarantees.

malloc/free cannot be used in hard real time systems, either. malloc/free do not have latency guarantees.
Mar 23 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/23/2014 11:29 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Sunday, 23 March 2014 at 17:35:37 UTC, Walter Bright wrote:
 malloc/free cannot be used in hard real time systems, either. malloc/free do
 not have latency guarantees.

While that is true you can have a soft real time thread feeding the hard real time thread with new configurations and the associated preallocated buffers using a lock-free queue.

Yes, and you can do that with GC, too.
Mar 23 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/23/2014 12:42 PM, Russel Winder wrote:
 On Sun, 2014-03-23 at 10:35 -0700, Walter Bright wrote:
 On 3/23/2014 12:13 AM, Russel Winder wrote:
 But for real time you would just have to remove the
 GC completely to have the needed guarantees.

malloc/free cannot be used in hard real time systems, either. malloc/free do not have latency guarantees.

By estimating the resource needs and preallocating a freelist, you can get round this issue. C++ is quite good at supporting this sort of stuff.

Yes, and you can do that in D, the same way.
Mar 23 2014
prev sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 3/22/2014 9:47 AM, Paulo Pinto wrote:
 Assuming those 10% still happen if the test was done today as suggested,
 how much are trade companies willing to pay for developers to achieve
 those 10% in C++ vs having a system although 10% slower,
 still fast enough for operations while saving salaries for more cheaper
 developers?

If the domain is high-performance, high-volume, (hard|soft) realtime, then I doubt very much you can get away with significantly cheaper developers, even if it is Java. Unless it just happens to be a sub-par company with a questionable future.
Mar 23 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 23.03.2014 22:04, schrieb Nick Sabalausky:
 On 3/22/2014 9:47 AM, Paulo Pinto wrote:
 Assuming those 10% still happen if the test was done today as suggested,
 how much are trade companies willing to pay for developers to achieve
 those 10% in C++ vs having a system although 10% slower,
 still fast enough for operations while saving salaries for more cheaper
 developers?

If the domain is high-performance, high-volume, (hard|soft) realtime, then I doubt very much you can get away with significantly cheaper developers, even if it is Java. Unless it just happens to be a sub-par company with a questionable future.

My question was precisely because I tend to see that a lot in general, as on my area saving project costs seems to be more valuable than quality. -- Paulo
Mar 23 2014
prev sibling next sibling parent "TJB" <broughtj gmail.com> writes:
On Saturday, 22 March 2014 at 00:14:11 UTC, Daniel Davidson wrote:
 On Friday, 21 March 2014 at 21:14:15 UTC, TJB wrote:
 Walter,

 I see that you will be discussing "High Performance Code Using 
 D" at the 2014 DConf. This will be a very welcomed topic for 
 many of us.  I am a Finance Professor.  I currently teach and 
 do research in computational finance.  Might I suggest that 
 you include some finance (say Monte Carlo options pricing) 
 examples?  If you can get the finance industry interested in D 
 you might see a massive adoption of the language.  Many are 
 desperate for an alternative to C++ in that space.

 Just a thought.

 Best,

 TJB

Maybe a good starting point would be to port some of QuantLib and see how the performance compares. In High Frequency Trading I think D would be a tough sell, unfortunately. Thanks Dan

Dan, Why a tough sell? Please explain. TJB
Mar 21 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
TJB:

 Why a tough sell?  Please explain.

That code must always be hard-real time. So a GC is allowed only during startup time (unless it's a quite special GC), hidden heap allocations are forbidden, data access patterns need to be carefully chosen, you even have to use most of the hot part of the stack, etc. Bye, bearophile
Mar 21 2014
prev sibling next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 22 March 2014 at 00:34:22 UTC, TJB wrote:
 On Saturday, 22 March 2014 at 00:14:11 UTC, Daniel Davidson 
 wrote:
 On Friday, 21 March 2014 at 21:14:15 UTC, TJB wrote:
 Walter,

 I see that you will be discussing "High Performance Code 
 Using D" at the 2014 DConf. This will be a very welcomed 
 topic for many of us.  I am a Finance Professor.  I currently 
 teach and do research in computational finance.  Might I 
 suggest that you include some finance (say Monte Carlo 
 options pricing) examples?  If you can get the finance 
 industry interested in D you might see a massive adoption of 
 the language.  Many are desperate for an alternative to C++ 
 in that space.

 Just a thought.

 Best,

 TJB

Maybe a good starting point would be to port some of QuantLib and see how the performance compares. In High Frequency Trading I think D would be a tough sell, unfortunately. Thanks Dan

Dan, Why a tough sell? Please explain. TJB

Well, I am a fan of D. That said, there is a ton of money that goes into the HFT low latency arms race. I felt they were always willing to spend whatever it takes - but they had to make choices. First, I know for several firms there was a history the C++ vs Java back and forth until it was clear that the lowest latency systems were C/C++. The difference was attributed to many causes, but garbage collection always came up as a big one. I think just that association of C++ beating Java because of garbage collection will be enough to prevent many HFT shops from considering D for a long while, even if you could get rid of the gc entirely. Additionally, many of the advantages firms get come from dealing intelligently with the kernel and lowest level networking techniques which implies expertise in C and that often comes with a well-deserved pride in knowing how to make things run fast in specific ways. Further they are pushing even lower into custom hardware which is expensive. With the money to throw at it and the various alternatives to choose from there is a go big or go home mentality. They are not as worried about how fun it is to code in, how ugly your templates are, or how it might take longer to do something or to build something. I agree if you could get the finance industry interested in D that would be great. These are just my opinions and I left HFT a couple years back now. But, in this field, to expect a firm to get behind something like D you would need demonstrable improvements in performance irrespective of cost to develop. If they can do equivalent or better performance in C++ with more time to develop - then C++ still wins. Now if D can actually beat C++ significantly in quant programming performance things might change. Do you have ideas on how that could be done? If you consider automated, black box trading, most firms roll their own options infrastructure and these days QuantLib would not make the cut. But if you could demonstrate same functionality with improved performance by using D you could make a case. Thanks Dan
Mar 21 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 22 March 2014 at 01:24:38 UTC, Walter Bright wrote:
 On 3/21/2014 5:39 PM, bearophile wrote:
 That code must always be hard-real time. So a GC is allowed 
 only during startup
 time (unless it's a quite special GC), hidden heap allocations 
 are forbidden,
 data access patterns need to be carefully chosen, you even 
 have to use most of
 the hot part of the stack, etc.

These are all very doable in D, and I've written apps that do so. The "fear of GC" is completely overblown.

Fear of GC is overblown. Fear of D's GC isn't.
Mar 21 2014
prev sibling next sibling parent Ziad Hatahet <hatahet gmail.com> writes:
--001a1133d460b0785204f52dfc08
Content-Type: text/plain; charset=ISO-8859-1

On Sat, Mar 22, 2014 at 12:30 AM, Paulo Pinto <pjmlp progtools.org> wrote:

 Yes, as there are a few  high performance trading systems done with
 JVM/.NET languages.

 --
 Paulo

What about AAA games? :) Even though I agree with the pro-GC arguments you put forth, but I really have a hard time imagining something like Battlefield 4 written in Java and running on the JVM (though I would love to be proven wrong). --001a1133d460b0785204f52dfc08 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On S= at, Mar 22, 2014 at 12:30 AM, Paulo Pinto <span dir=3D"ltr">&lt;<a href=3D"= mailto:pjmlp progtools.org" target=3D"_blank">pjmlp progtools.org</a>&gt;</= span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">Yes, as there are a few =A0high performance = trading systems done with JVM/.NET languages.<br> <br> --<br> Paulo<br> </blockquote></div><br></div><div class=3D"gmail_extra">What about AAA game= s? :) Even though I agree with the pro-GC arguments you put forth, but I re= ally have a hard time imagining something like Battlefield 4 written in Jav= a and running on the JVM (though I would love to be proven wrong).<br> </div></div> --001a1133d460b0785204f52dfc08--
Mar 22 2014
prev sibling next sibling parent "zoomba" <null nil.com> writes:
On Saturday, 22 March 2014 at 08:54:46 UTC, Paulo Pinto wrote:
 Am 22.03.2014 09:42, schrieb Ziad Hatahet:
 On Sat, Mar 22, 2014 at 12:30 AM, Paulo Pinto 
 <pjmlp progtools.org
 <mailto:pjmlp progtools.org>> wrote:

    Yes, as there are a few  high performance trading systems 
 done with
    JVM/.NET languages.

    --
    Paulo


 What about AAA games? :) Even though I agree with the pro-GC 
 arguments
 you put forth, but I really have a hard time imagining 
 something like
 Battlefield 4 written in Java and running on the JVM (though I 
 would
 love to be proven wrong).

What about ballistic missile tracking radar control systems? http://www.bloomberg.com/apps/news?pid=newsarchive&sid=aHLFBJZrqaoM You don't want those babies out of control. -- Paulo

Ah Thales... I hope they are better at writing software for missiles than they are at tram management systems.
Mar 22 2014
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
On Fri, 2014-03-21 at 22:44 +0000, Chris Williams wrote:
 On Friday, 21 March 2014 at 22:28:36 UTC, Walter Bright wrote:
 It's a good thought, but I have zero knowledge of how C++ is 
 used for high frequency trading.

Reading through the Wikipedia article on Computational Finance, it looks like it's basically performing simulations where some data is known but other is not. Random numbers are generated for the unknown data and the simulations are run several times to find the range of possible outcomes given the known values.

http://en.wikipedia.org/wiki/Computational_finance Is seriously lacking in actual content but at least it isn't entirely wrong. There are many different things happening in what many would label as computational finance from ultra-high-frequency trading systems to modelling macroeconomics for fund management. In the former case of course the most crucial thing is the length of the cable of you computer to the router :-) In the latter case, and indeed quantitative analysis (quant), the mathematical models can be seriously weird (most likely because they are based on phenomenology rather than science-based modelling). What you are alluding to is the use of Monte Carlo approach to solve some of the models given boundary conditions. This is a "bog standard" approach to numerical modelling. Many quants use Excel, many are C++ folk. Those using Excel really need to stop it. Many of the hedge funds are following in the HEP, bioinformatics direction and using Python (PyPy, NumPy, SciPy, etc.) for writing the initial models and then, if greater speed is needed, using Cython or rewriting in C++. Mathematica, R and Julia are increasingly also players in this game. D could be a very interesting player in this game, but it would have to have some early adopters get on board and show in the domain itself that it can beat Python, C++, Mathematics, R and Julia at their own game. Whilst I am not a finance person myself: I train a lot of finance people, and signal processing people, in Python, Java, Scala, and Groovy. In a number of the major international finance houses a Python/Scala/C++ stack has taken hold. However this is not fixed forever, since whenever the CTO changes the stack tends to as well. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Mar 22 2014
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
On Sat, 2014-03-22 at 00:39 +0000, bearophile wrote:
 TJB:
 
 Why a tough sell?  Please explain.

That code must always be hard-real time. So a GC is allowed only during startup time (unless it's a quite special GC), hidden heap allocations are forbidden, data access patterns need to be carefully chosen, you even have to use most of the hot part of the stack, etc.

Not entirely the case. Yes the ultra-high-frequency traders tend to be C ++ and hard real time with no operating system scheduler active. However there are a number of high frequency trading systems using the JVM with the G1 garbage collector since actually only soft real time is needed for the trading they are doing. It is also worth pointing out the LMAX Disruptor which is a lock-free ring buffer based framework used to create dealing platforms on the JVM. They outperform any other trading platform still. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Mar 22 2014
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
On Sat, 2014-03-22 at 00:14 +0000, Daniel Davidson wrote:
[…]
 Maybe a good starting point would be to port some of QuantLib and 
 see how the performance compares. In High Frequency Trading I 
 think D would be a tough sell, unfortunately.

I would certainly agree that (at least initially) pitching D against the Excel/Python/R/Julia/Mathematica is an easier fight. The question is how to convince someone to take the first step. I suspect a rewrite of QuantLib in D is a bad idea, much better to create an adapter and offer it to the QuantLib folks. The ones they have already tend to be created using SWIG. JQuantLib is an attempt to rewrite QuantLib in pure Java, but I do not know if it is gaining any traction over the Java adapter to QuantLib. The angle here to get D traction would be to have the data visualization capability: the reason for the success of SciPy, R, Julia has been very fast turnaround of changes to the models and the rendering of the results of the computations. Certainly in bioinformatics, and I guess in finance, there is a lot of use of hardware floating point numbers, but also of arbitrary size integers, not just hardware integers. If your languages cannot calculate correctly factorial(40) then there is no hope in these domains, this is why Python, R, Julia get traction they manage integers and the use of hardware and software representations so that the programmer doesn't have to care, it all just works. This is clearly not true of C++ :-) -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Mar 22 2014
prev sibling next sibling parent reply "Saurabh Das" <saurabh.das gmail.com> writes:
You are absolutely correct - the finance industry _wants_ to
switch away fromC++. I work in a fledgeling HFT startup firm and
we are actively pursuing D. We have tested it out in a live
trading environment and the results are very promising.

1. We are measuring better latency numbers in D (As compared to
the older C++ systems). This is good enough reason to switch :)

2. After much testing, we concluded that fear-of-GC-collects is
overblown. Avoid allocating in the main loops as far as possible.

3. Code is _much_ more maintainable and easier to understand.

4. It's fun to code again - and this point cannot be stressed
enough. C++ is a major headache but earlier we had no choice.

I'm quite confident that D is going to make good inroads into the
financial industry in the coming years. Looking forward to
Walter's talk in DConf and indeed all the talks in DConf. Wish I
could attend - but the flight costs too much :(. Maybe next year.

Saurabh

On Friday, 21 March 2014 at 21:14:15 UTC, TJB wrote:
 Walter,

 I see that you will be discussing "High Performance Code Using 
 D" at the 2014 DConf. This will be a very welcomed topic for 
 many of us.  I am a Finance Professor.  I currently teach and 
 do research in computational finance.  Might I suggest that you 
 include some finance (say Monte Carlo options pricing) 
 examples?  If you can get the finance industry interested in D 
 you might see a massive adoption of the language.  Many are 
 desperate for an alternative to C++ in that space.

 Just a thought.

 Best,

 TJB

Mar 22 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/22/2014 5:35 AM, Saurabh Das wrote:
 I'm quite confident that D is going to make good inroads into the
 financial industry in the coming years. Looking forward to
 Walter's talk in DConf and indeed all the talks in DConf. Wish I
 could attend - but the flight costs too much :(. Maybe next year.

C'mon, man, you gotta come. I want to hear more about the HFT stuff!
Mar 22 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/22/2014 5:56 AM, Daniel Davidson wrote:
 I don't yet see where D adds distinction to that game yet - other
 than being a great language.

Isn't that the best kind of distinction?
Mar 22 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/22/2014 7:04 AM, Daniel Davidson wrote:
 But then, with a large amount of time and unlimited funding the techniques
could
 probably be duplicated in C++.

That's correct. It's also true of writing code in C or even assembler. Productivity matters because time-to-deploy matters. I.e. if you can deploy a better system in one month rather than two months, you've got an extra month of making money with it.
Mar 22 2014
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 23.03.2014 18:38, schrieb Sean Kelly:
 On Saturday, 22 March 2014 at 14:04:01 UTC, Daniel Davidson wrote:
 For example, I could see technical reasons why in certain non-quant
 areas like XML parsing where D can be faster than C++.
 (http://dotnot.org/blog/archives/2008/03/12/why-is-dtango-so-fast-at-parsing-xml/)
 But then, with a large amount of time and unlimited funding the
 techniques could probably be duplicated in C++.

Try no funding and a trivial amount of time. The JSON parser I wrote for work in C performs zero allocations and unescaping is performed on demand. D arguably makes this easier by building slicing into the language, but not decoding or copying is a design decision, not a language artifact (at least in the case of C/C++ where aliasing data is allowed). The take-away from that Tango article is that the performance hit for parsing is aggressively decoding data the user may not care about or may not want decoded in the first place. This just happens to be the approach that basically every XML parser on the planet uses for some ridiculous reason.

At least on Java world it is not quite true. If you use XML parsers that return a DOM or SAX, yes quite true. But as far as I can tell, XML streaming parsers (StAX) only parse on demand. Unless I am missing something. -- Paulo
Mar 23 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/23/2014 10:38 AM, Sean Kelly wrote:
 Try no funding and a trivial amount of time.  The JSON parser I wrote for work
 in C performs zero allocations and unescaping is performed on demand.  D
 arguably makes this easier by building slicing into the language, but not
 decoding or copying is a design decision, not a language artifact (at least in
 the case of C/C++ where aliasing data is allowed).  The take-away from that
 Tango article is that the performance hit for parsing is aggressively decoding
 data the user may not care about or may not want decoded in the first place.
 This just happens to be the approach that basically every XML parser on the
 planet uses for some ridiculous reason.

Lazy evaluation FTW. Ranges and algorithms fit right in with that.
Mar 23 2014
prev sibling next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 22 March 2014 at 11:46:43 UTC, Russel Winder wrote:

 It is also worth pointing out the LMAX Disruptor which is a 
 lock-free
 ring buffer based framework used to create dealing platforms on 
 the JVM.
 They outperform any other trading platform still.
 

That is wrong. Trading is competitive and your competitors rarely will inform you they are losing to you or beating you. And when you do not get the fills you want it is not always clear you are being beaten. This article (http://mechanical-sympathy.blogspot.com/2011/08/inter-thread-latency.html) suggests C++ version of the same architecture is faster than the Java. Thanks Dan
Mar 22 2014
prev sibling next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 22 March 2014 at 12:35:50 UTC, Saurabh Das wrote:
 You are absolutely correct - the finance industry _wants_ to
 switch away fromC++. I work in a fledgeling HFT startup firm and
 we are actively pursuing D. We have tested it out in a live
 trading environment and the results are very promising.

Well, the finance industry is pretty big and therefore diverse. By, "the finance industry _wants_ to switch away from C++" I assume you mean on average or maybe those you work with. Glad to hear you are having promising results with D.
 1. We are measuring better latency numbers in D (As compared to
 the older C++ systems). This is good enough reason to switch :)

You should share some of the numbers and analysis - maybe some techniques as well. Where do you suspect that edge comes from and would it be possible to duplicate the improvements by changing/improving the C++ - (not that you would want to)?
 2. After much testing, we concluded that fear-of-GC-collects is
 overblown. Avoid allocating in the main loops as far as 
 possible.

Really? Why is that... because it has no noticeable effect in your system?
 3. Code is _much_ more maintainable and easier to understand.

I would bet so.
 4. It's fun to code again - and this point cannot be stressed
 enough. C++ is a major headache but earlier we had no choice.

Fun is important - especially at an HFT startup. Just imagine the pain endured by new hires at firms with mounds of C++.
 I'm quite confident that D is going to make good inroads into 
 the
 financial industry in the coming years.

My bet is it will make inroads in finance but more on the infrastructure, web, general programming side and less on the HFT, low-latency side. Probably less on the Quant side as well without something compelling. Julia is compelling with its focus on math and clear benefits over current RAD alternatives numpy, R, Matlab, etc. I don't yet see where D adds distinction to that game yet - other than being a great language. We'll see. Thanks Dan
Mar 22 2014
prev sibling next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 22 March 2014 at 12:06:37 UTC, Russel Winder wrote:
 I suspect a rewrite of QuantLib in D is a bad idea, much better 
 to
 create an adapter and offer it to the QuantLib folks. The ones 
 they have
 already tend to be created using SWIG. JQuantLib is an attempt 
 to
 rewrite QuantLib in pure Java, but I do not know if it is 
 gaining any
 traction over the Java adapter to QuantLib.

I guess it depends on the goal. The OP was interested in replacing C++ with D for quant work. If the goal is to use QuantLib functionality in D then you are correct - wrappers are the way to go. But if you want to push D into the quant side of things and show off the benefits there are not many bragging rights to having a great wrapper over C++. I think the exercise of moving some of QuantLib to D would be the education of the benefits/drawbacks of that move and the hope it would be representative of the D vs C++ tradeoffs for quant programming in general.
 The angle here to get D traction would be to have the data 
 visualization
 capability: the reason for the success of SciPy, R, Julia has 
 been very
 fast turnaround of changes to the models and the rendering of 
 the
 results of the computations.

Data storage for high volume would also be nice. A D implementation of HDF5, via wrappers or otherwise, would be a very useful project. Imagine how much more friendly the API could be in D. Python's tables library makes it very simple. You have to choose a language to not only process and visualize data, but store and access it as well. Thanks Dan
Mar 22 2014
prev sibling next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 22 March 2014 at 12:54:11 UTC, Paulo Pinto wrote:
 Am 22.03.2014 13:38, schrieb Daniel Davidson:
 On Saturday, 22 March 2014 at 11:46:43 UTC, Russel Winder 
 wrote:

 It is also worth pointing out the LMAX Disruptor which is a 
 lock-free
 ring buffer based framework used to create dealing platforms 
 on the JVM.
 They outperform any other trading platform still.

That is wrong. Trading is competitive and your competitors rarely will inform you they are losing to you or beating you. And when you do not get the fills you want it is not always clear you are being beaten. This article (http://mechanical-sympathy.blogspot.com/2011/08/inter-thread-latency.html) suggests C++ version of the same architecture is faster than the Java. Thanks Dan

Maybe this is a wrong conclusion, but from the comments, I would say the Java version is pretty much on par with the C++ attempts.

I guess it depends where your edge comes from. If it is speed, the following suggests it is worth it to go C++. 10% is huge and reduction in variability is also huge. [[ So what does this all mean for the Disruptor? Basically, the latency of the Disruptor is about as low as we can get from Java. It would be possible to get a ~10% latency improvement by moving to C++. I’d expect a similar improvement in throughput for C++. The main win with C++ would be the control, and therefore, the predictability that comes with it if used correctly. The JVM gives us nice safety features like garbage collection in complex applications but we pay a little for that with the extra instructions it inserts that can be seen if you get Hotspot to dump the assembler instructions it is generating. ]] If your edge is quickly trying out new algorithms and identifying edge by trying many less latency sensitive strategies in the market - then "pretty much on par" is a fine bar. Thanks, Dan
Mar 22 2014
prev sibling next sibling parent "Saurabh Das" <saurabh.das gmail.com> writes:
Hi Dan,



On Saturday, 22 March 2014 at 12:56:03 UTC, Daniel Davidson wrote:
 On Saturday, 22 March 2014 at 12:35:50 UTC, Saurabh Das wrote:
 You are absolutely correct - the finance industry _wants_ to
 switch away fromC++. I work in a fledgeling HFT startup firm 
 and
 we are actively pursuing D. We have tested it out in a live
 trading environment and the results are very promising.

Well, the finance industry is pretty big and therefore diverse. By, "the finance industry _wants_ to switch away from C++" I assume you mean on average or maybe those you work with. Glad to hear you are having promising results with D.

Yes. The finance industry is very big and diverse. I am in particular referring to the parts of the industry that work heavily in C++ currently. Maybe my sample set is skewed because of the segment I work in - but there is a lot of enthusiasm that I have seen from those I am in contact with.
 1. We are measuring better latency numbers in D (As compared to
 the older C++ systems). This is good enough reason to switch :)

You should share some of the numbers and analysis - maybe some techniques as well. Where do you suspect that edge comes from and would it be possible to duplicate the improvements by changing/improving the C++ - (not that you would want to)?

I don't think I would be allowed to share numbers :( I will check and get back on that. We have been debating about starting a blog where we can share techniques with the wider world. Unfortunately we are just so overloaded currently that getting down to doing this may take some time. The edge for D in our case comes from 3 factors - 1. A lot of statistical data from older C++ systems means better assumptions and decisions in the new D system; and 2. 20% of the system is latency-critical and 80% is not. D allows us to quickly finish 80% and really concentrate on the critical 20%. I must also comment upon how much more productive it is to write a new system in D as compared with C++ - gives us more time to think about the actual problem than try to jump through the C++ hoops. 3. A much better type system - some checks can be moved to compile time. Major benefit.
 2. After much testing, we concluded that fear-of-GC-collects is
 overblown. Avoid allocating in the main loops as far as 
 possible.

Really? Why is that... because it has no noticeable effect in your system?

Yes. I am commenting about our systems - we disable the GC during critical periods and then go back and collect collect collect later when things calm down. And as such there is minimal allocation in critical sections - we try to ensure all allocation is done beforehand. We followed a similar approach in C++ too since malloc/new is so slow.
 3. Code is _much_ more maintainable and easier to understand.

I would bet so.
 4. It's fun to code again - and this point cannot be stressed
 enough. C++ is a major headache but earlier we had no choice.

Fun is important - especially at an HFT startup. Just imagine the pain endured by new hires at firms with mounds of C++.
 I'm quite confident that D is going to make good inroads into 
 the
 financial industry in the coming years.

My bet is it will make inroads in finance but more on the infrastructure, web, general programming side and less on the HFT, low-latency side. Probably less on the Quant side as well without something compelling. Julia is compelling with its focus on math and clear benefits over current RAD alternatives numpy, R, Matlab, etc. I don't yet see where D adds distinction to that game yet - other than being a great language. We'll see.

Yes - R, Matlab et all won't be replaced by D most likely. Let's wait and watch. However I disagree about the HFT/low-latency side. Ofcourse there's no way to say for sure. Let's check again in a year :)
 Thanks
 Dan

Saurabh
Mar 22 2014
prev sibling next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 22 March 2014 at 13:36:01 UTC, Saurabh Das wrote:
 The edge for D in our case comes from 3 factors -

 1. A lot of statistical data from older C++ systems means 
 better assumptions and decisions in the new D system; and

But, clearly that is not necessarily a benefit of D. It is a benefit of prior experience and the learning curve. If you said, we use our data to not only make better assumptions/decisions, but to do things in D that can not be done in C++ - then you make a very strong case.
 2. 20% of the system is latency-critical and 80% is not. D 
 allows us to quickly finish 80% and really concentrate on the 
 critical 20%. I must also comment upon how much more productive 
 it is to write a new system in D as compared with C++ - gives 
 us more time to think about the actual problem than try to jump 
 through the C++ hoops.

Productivity is very important and can mean big $$ for most firms. But if latency is the critical factor in an all-or-nothing game, then it is much less so. Maybe your game is different and you have edge beyond low latency. I hope that is the case.
 3. A much better type system - some checks can be moved to 
 compile time. Major benefit.

What is a simple example of something that could be done with D but not C++ that has nothing to do with building things with less developer time? For example, I could see technical reasons why in certain non-quant areas like XML parsing where D can be faster than C++. (http://dotnot.org/blog/archives/2008/03/12/why-is-dtango-so-f st-at-parsing-xml/) But then, with a large amount of time and unlimited funding the techniques could probably be duplicated in C++. Again, I don't think it is necessary to have any/many cases where D beats C++ hands down in performance for its adoption to widen. But to push D to a wider audience by focusing on areas where the bar is already super high is tough. If I had money to invest in D I would invest it in vibe rather than quant because the relative advantages of D are so much higher.
 Yes - R, Matlab et all won't be replaced by D most likely. 
 Let's wait and watch. However I disagree about the 
 HFT/low-latency side. Ofcourse there's no way to say for sure. 
 Let's check again in a year :)

Sounds good - keep us posted! Thanks Dan
Mar 22 2014
prev sibling next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 22 March 2014 at 13:47:31 UTC, Paulo Pinto wrote:

 Assuming those 10% still happen if the test was done today as 
 suggested, how much are trade companies willing to pay for 
 developers to achieve those 10% in C++ vs having a system 
 although 10% slower,
 still fast enough for operations while saving salaries for more 
 cheaper developers?

Labor is a market like any other. It depends on supply and demand. The demand is obviously high. http://dealbook.nytimes.com/2010/12/10/ex-goldman-programmer-is-convicted/?_php=true&_type=blogs&_r=0 Performance engineers who can eek out that 10% on existing systems do very well. The same engineers who can build it entirely do much better. Thanks Dan
Mar 22 2014
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
On Sat, 2014-03-22 at 12:38 +0000, Daniel Davidson wrote:
 On Saturday, 22 March 2014 at 11:46:43 UTC, Russel Winder wrote:
 
 It is also worth pointing out the LMAX Disruptor which is a 
 lock-free
 ring buffer based framework used to create dealing platforms on 
 the JVM.
 They outperform any other trading platform still.
 

That is wrong. Trading is competitive and your competitors rarely will inform you they are losing to you or beating you. And when you do not get the fills you want it is not always clear you are being beaten. This article (http://mechanical-sympathy.blogspot.com/2011/08/inter-thread-latency.html) suggests C++ version of the same architecture is faster than the Java.

If the data isn't available we'll never really know. It is now over 9 months since the people I know left LMAX for other organizations so I have no current data, or even rumour. It would be interesting to rerun the comparison with JDK8 and C++14. I suspect C++ may well be the winner on latency, whether by 10% I am not sure. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Mar 22 2014
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
On Sat, 2014-03-22 at 14:17 +0000, Daniel Davidson wrote:
[…]
 Performance engineers who can eek out that 10% on existing 
 systems do very well. The same engineers who can build it 
 entirely do much better.

Good C++ programmers appear to be able to get $350k to $400k in NY. Of course the effect of that good programmers can be a profit increase measures in millions. I guess D should be able to do things just as fast as C++, at least using LDC or GDC. My little informal microbenchmarks indicate that this is the case, but for now this is anecdotal evidence not statistically significant. Sadly for a while D programmers won't be able to achieve the same remuneration as the top C++ programmers exactly because there is little demand/supply pressure. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Mar 22 2014
prev sibling next sibling parent "TJB" <broughtj gmail.com> writes:
On Saturday, 22 March 2014 at 13:10:46 UTC, Daniel Davidson wrote:
 Data storage for high volume would also be nice. A D 
 implementation of HDF5, via wrappers or otherwise, would be a 
 very useful project. Imagine how much more friendly the API 
 could be in D. Python's tables library makes it very simple. 
 You have to choose a language to not only process and visualize 
 data, but store and access it as well.

 Thanks
 Dan

Well, I for one, would be hugely interested in such a thing. A nice D API to HDF5 would be a dream for my data problems. Did you use HDF5 in your finance industry days then? Just curious. TJB
Mar 22 2014
prev sibling next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 22 March 2014 at 14:33:02 UTC, TJB wrote:
 Well, I for one, would be hugely interested in such a thing.  A
 nice D API to HDF5 would be a dream for my data problems.

 Did you use HDF5 in your finance industry days then?  Just
 curious.

A bit. You can check out some of my C++ code generation support for hdf5 here: https://github.com/patefacio/codegen/tree/master/cpp/fcs/h5 A description of the code generation infrastructure is available here: https://github.com/patefacio/codegen/blob/master/doc/codegen.pdf And some python usage of tables to parse/store data for HFT analysis of potential benefits of periodic auctions as opposed to continuous markets: https://github.com/patefacio/auction Using hdf5 with tables and simple usage in C++ is very powerful. I did not do a lot with it - simple tables for writing/reading from C++. The API is huge and very old school, even with the C++ wrappers. IMHO an awesome project would be a rewrite in D that abandoned the API but provided file compatibility with cleaner access. D would be great for that.
 TJB

Mar 22 2014
prev sibling next sibling parent "Brian Rogoff" <brogoff gmail.com> writes:
On Friday, 21 March 2014 at 22:33:37 UTC, TJB wrote:
 On Friday, 21 March 2014 at 22:28:36 UTC, Walter Bright wrote:
 It's a good thought, but I have zero knowledge of how C++ is 
 used for high frequency trading.

I would be happy to help you with an option pricing example that is commonly used. Let me know if you are interested.

This is a very interesting thread that you started. Could you flesh it out more with some example C++ that you'd like compared to D? I'm sure quite a few people would assist with a translation. I'm not expert in high frequency trading, but I was inspired by your post to start poking around here http://www.quantstart.com/articles/european-vanilla-option-pricing-with-c-via-monte-carlo-methods and study some of the algorithms. Nothing there that I wouldn't rather see in D than C++. D's GC is problematic, but the hope is that you can avoid allocating from the GC'ed heap and that eventually (soon? please?) it will be replaced by a better precise GC.
Mar 22 2014
prev sibling next sibling parent "TJB" <broughtj gmail.com> writes:
On Saturday, 22 March 2014 at 16:35:07 UTC, Brian Rogoff wrote:

 This is a very interesting thread that you started. Could you 
 flesh it out more with some example C++ that you'd like 
 compared to D? I'm sure quite a few people would assist with a 
 translation.

Well, right away people jumped to high-frequency trading. Although that may be the most visible area in computational finance - it's not the only one. There are areas where performance is crucial, but where trading is done at a lower frequency (where latency is not the main issue).
 I'm not expert in high frequency trading, but I was inspired by 
 your post to start poking around here

 http://www.quantstart.com/articles/european-vanilla-option-pricing-with-c-via-monte-carlo-methods

 and study some of the algorithms. Nothing there that I wouldn't 
 rather see in D than C++.

The example that you link to is exactly what I have in mind. A simple comparison of Monte Carlo routines for pricing options would be a great place to start. The bible on this is the book by Glasserman: http://www.amazon.com/Financial-Engineering-Stochastic-Modelling-Probability/dp/0387004513/ref=sr_1_1?ie=UTF8&qid=1395509317&sr=8-1&keywords=monte+carlo+in+financial+engineering And a great source for approaching this is in C++ is Joshi: http://www.amazon.com/Patterns-Derivatives-Pricing-Mathematics-Finance/dp/0521832357/ref=sr_1_5?s=books&ie=UTF8&qid=1395509376&sr=1-5
 D's GC is problematic, but the hope is that you can avoid 
 allocating from the GC'ed heap and that eventually (soon? 
 please?) it will be replaced by a better precise GC.

Sounds great to me. I would love to see it. Thanks for taking interest.
Mar 22 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 22 March 2014 at 14:30:00 UTC, Russel Winder wrote:
 On Sat, 2014-03-22 at 14:17 +0000, Daniel Davidson wrote:
 […]
 Performance engineers who can eek out that 10% on existing 
 systems do very well. The same engineers who can build it 
 entirely do much better.

Good C++ programmers appear to be able to get $350k to $400k in NY. Of course the effect of that good programmers can be a profit increase measures in millions. I guess D should be able to do things just as fast as C++, at least using LDC or GDC. My little informal microbenchmarks indicate that this is the case, but for now this is anecdotal evidence not statistically significant. Sadly for a while D programmers won't be able to achieve the same remuneration as the top C++ programmers exactly because there is little demand/supply pressure.

HFT is very latency sensitive. D stop the world GC is a no go. D needs a better GC to be viable in these markets.
Mar 22 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
The work Don's company does has very similar requirements to HFT. 
His talks here are totally relevant to the use of D in this area.
Mar 22 2014
prev sibling next sibling parent "TJB" <broughtj gmail.com> writes:
On Saturday, 22 March 2014 at 23:27:18 UTC, Sean Kelly wrote:
 The work Don's company does has very similar requirements to 
 HFT. His talks here are totally relevant to the use of D in 
 this area.

Yes, I'm very much looking forward to that talk as well. His talk last year killed!
Mar 22 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 22 March 2014 at 23:27:18 UTC, Sean Kelly wrote:
 The work Don's company does has very similar requirements to 
 HFT. His talks here are totally relevant to the use of D in 
 this area.

They did significant work to customize th GC in order to meet the requirements.
Mar 22 2014
prev sibling next sibling parent "Saurabh Das" <saurabh.das gmail.com> writes:
On Saturday, 22 March 2014 at 14:04:01 UTC, Daniel Davidson wrote:
 On Saturday, 22 March 2014 at 13:36:01 UTC, Saurabh Das wrote:
 The edge for D in our case comes from 3 factors -

 1. A lot of statistical data from older C++ systems means 
 better assumptions and decisions in the new D system; and

But, clearly that is not necessarily a benefit of D. It is a benefit of prior experience and the learning curve. If you said, we use our data to not only make better assumptions/decisions, but to do things in D that can not be done in C++ - then you make a very strong case.

Yes - I didn't mean this as a point in favour of D, but just to put down the factors that made a difference. However in all microbencharks thus far, D has not done worse than C++.
 2. 20% of the system is latency-critical and 80% is not. D 
 allows us to quickly finish 80% and really concentrate on the 
 critical 20%. I must also comment upon how much more 
 productive it is to write a new system in D as compared with 
 C++ - gives us more time to think about the actual problem 
 than try to jump through the C++ hoops.

Productivity is very important and can mean big $$ for most firms. But if latency is the critical factor in an all-or-nothing game, then it is much less so. Maybe your game is different and you have edge beyond low latency. I hope that is the case.

To clarify - for us latency is critical. The reason that productivity matters is that given resource constraints, I can spend much more time optimizing the 20% in D because the 80% can be written quickly.
 3. A much better type system - some checks can be moved to 
 compile time. Major benefit.

What is a simple example of something that could be done with D but not C++ that has nothing to do with building things with less developer time? For example, I could see technical reasons why in certain non-quant areas like XML parsing where D can be faster than C++. (http://dotnot.org/blog/archives/2008/03/12/why-is-dtango-so-f st-at-parsing-xml/) But then, with a large amount of time and unlimited funding the techniques could probably be duplicated in C++.

None of the type-system stuff CANNOT be duplicated in C++ (given enough resources), but D makes it easy. I'll give you an example which is similar to XML parsing - consider FIX messages. Using templates judiciously, it is possible to write a blazing fast FIX message processing system in D where a lot of the processing is unrolled at compile time. Surely this is possible in C++, but it's going to be a hell of a task.
 Again, I don't think it is necessary to have any/many cases 
 where D beats C++ hands down in performance for its adoption to 
 widen. But to push D to a wider audience by focusing on areas 
 where the bar is already super high is tough. If I had money to 
 invest in D I would invest it in vibe rather than quant because 
 the relative advantages of D are so much higher.

Yes I agree with that.
 Yes - R, Matlab et all won't be replaced by D most likely. 
 Let's wait and watch. However I disagree about the 
 HFT/low-latency side. Ofcourse there's no way to say for sure. 
 Let's check again in a year :)

Sounds good - keep us posted! Thanks Dan

Mar 22 2014
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
On Sat, 2014-03-22 at 21:13 +0000, deadalnix wrote:
[…]
 HFT is very latency sensitive. D stop the world GC is a no go.
 
 D needs a better GC to be viable in these markets.

GC technology was well beyond "stop the world" in Common Lisp in the 1990s. Java learnt this lesson in the 2000s. IBM, Azul, and now OpenJDK have an array of very sophisticated, and indeed some real-time, garbage collectors. Clearly though the Lisp and Java worlds are very different from the C++, D and Go worlds, thus sophisticated algorithms cannot simply be transplanted. As I understand it, Go has a parallel mark and sweep algorithm which is relatively unsophisticated, and a bit "stop the world", but they claim fine for small heaps: generational and compacting collectors (as per Java) are only beneficial for large heaps is the claim (but I haven't seen much data to back this up for Go). There is rumour of a change, but this has been circulating for ages. I guess Herb Sutter's (reported) view that he can write C++ code with no new and so no need for garbage collection, if applied to D would mean that although the collector "stopped the world", the heap would need no work and be small. But for real time you would just have to remove the GC completely to have the needed guarantees. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Mar 23 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Saturday, 22 March 2014 at 14:04:01 UTC, Daniel Davidson wrote:
 For example, I could see technical reasons why in certain 
 non-quant areas like XML parsing where D can be faster than 
 C++. 
 (http://dotnot.org/blog/archives/2008/03/12/why-is-dtango-so-f
st-at-parsing-xml/) 
 But then, with a large amount of time and unlimited funding the 
 techniques could probably be duplicated in C++.

Try no funding and a trivial amount of time. The JSON parser I wrote for work in C performs zero allocations and unescaping is performed on demand. D arguably makes this easier by building slicing into the language, but not decoding or copying is a design decision, not a language artifact (at least in the case of C/C++ where aliasing data is allowed). The take-away from that Tango article is that the performance hit for parsing is aggressively decoding data the user may not care about or may not want decoded in the first place. This just happens to be the approach that basically every XML parser on the planet uses for some ridiculous reason.
Mar 23 2014
prev sibling next sibling parent "TJB" <broughtj gmail.com> writes:
On Sunday, 23 March 2014 at 07:14:06 UTC, Walter Bright wrote:
 On 3/21/2014 3:33 PM, TJB wrote:
 I would be happy to help you with an option pricing example 
 that
 is commonly used.  Let me know if you are interested.

Sure, please email it to me.

Walter, I would be happy to. Where do I find your email address? Sorry if this is a dumb question. TJB
Mar 23 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 23 March 2014 at 17:35:37 UTC, Walter Bright wrote:
 malloc/free cannot be used in hard real time systems, either. 
 malloc/free do not have latency guarantees.

While that is true you can have a soft real time thread feeding the hard real time thread with new configurations and the associated preallocated buffers using a lock-free queue.
Mar 23 2014
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
On Sun, 2014-03-23 at 11:46 -0500, evansl wrote:
 On 03/22/14 06:40, Russel Winder wrote:
 [snip]
 What you are alluding to is the use of Monte Carlo approach to solve
 some of the models given boundary conditions. This is a "bog standard"

http://en.wiktionary.org/wiki/bog_standard

Sort of. This webpage almost, but not quite, totally misses all the British humorous subtleties. But yes. Effectively.
 approach to numerical modelling.

 Many of the hedge funds are following in the HEP,

By HEP do you mean "High Energy Physics"?

Oh yes. But I was only involved in HEP when all the software was in FORTRAN. Recently they have started using Fortran, and even C++. This is almost certainly Stephen Wolfram's fault.
 [snip]
 In a number of the major international finance houses a
 Python/Scala/C++ stack has taken hold. However this is not fixed
 forever, since whenever the CTO changes the stack tends to as well.

By CTO do you mean "Chief Technology Officer"?

Indeed. Many companies outside the US hipster company organization theory may use the term Technical Director. :-)
 TIA.
 
 -regards,
 Larry

Hopefully that is not Page or Ellison. ;-) -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Mar 23 2014
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
On Sun, 2014-03-23 at 19:15 +0100, Paulo Pinto wrote:
[…]
 
 At least on Java world it is not quite true.
 
 If you use XML parsers that return a DOM or SAX, yes quite true.
 
 But as far as I can tell, XML streaming parsers (StAX) only parse on demand.
 
 Unless I am missing something.

This is exactly why Groovy has two distinct XML parsers. British Library consider a small XML document to be about 6GB, you don't use DOM for these ;-) -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Mar 23 2014
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
On Sun, 2014-03-23 at 10:35 -0700, Walter Bright wrote:
 On 3/23/2014 12:13 AM, Russel Winder wrote:
 But for real time you would just have to remove the
 GC completely to have the needed guarantees.

malloc/free cannot be used in hard real time systems, either. malloc/free do not have latency guarantees.

By estimating the resource needs and preallocating a freelist, you can get round this issue. C++ is quite good at supporting this sort of stuff. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Mar 23 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Sunday, 23 March 2014 at 18:15:16 UTC, Paulo Pinto wrote:
 At least on Java world it is not quite true.

And that's why I said a language like C/C++ that allows aliasing.
 If you use XML parsers that return a DOM or SAX, yes quite true.

 But as far as I can tell, XML streaming parsers (StAX) only 
 parse on demand.

It's been a while since I used it, but the Apache SAX parser (Xerces?) converts all string input to wchar_t before passing it to the callback. And since XML input is nearly always in UTF-8, this can mean a ton of transcoding.
Mar 23 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Monday, 24 March 2014 at 05:41:38 UTC, Sean Kelly wrote:
 On Sunday, 23 March 2014 at 18:15:16 UTC, Paulo Pinto wrote:
 At least on Java world it is not quite true.

And that's why I said a language like C/C++ that allows aliasing.
 If you use XML parsers that return a DOM or SAX, yes quite 
 true.

 But as far as I can tell, XML streaming parsers (StAX) only 
 parse on demand.

It's been a while since I used it, but the Apache SAX parser (Xerces?) converts all string input to wchar_t before passing it to the callback. And since XML input is nearly always in UTF-8, this can mean a ton of transcoding.

Ah Xerces! Last time I looked into it was around 2003. I doubt it has any optimizations of modern XML parsers looking how little the web site has changed since then. -- Paulo
Mar 24 2014
prev sibling next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 22 March 2014 at 17:30:45 UTC, TJB wrote:
 On Saturday, 22 March 2014 at 16:35:07 UTC, Brian Rogoff wrote:

 This is a very interesting thread that you started. Could you 
 flesh it out more with some example C++ that you'd like 
 compared to D? I'm sure quite a few people would assist with a 
 translation.

Well, right away people jumped to high-frequency trading. Although that may be the most visible area in computational finance - it's not the only one. There are areas where performance is crucial, but where trading is done at a lower frequency (where latency is not the main issue).

I apologize for that. HFT has been a driver of a lot of business and attention. Of course you are right about areas with less latency sensitivity and D is attractive there. Even in latency sensitive efforts I think could be D attractive for new efforts providing some of the memory management efforts continue to evolve. My main point was selling it would be tough. Thanks Dan
Mar 24 2014
prev sibling next sibling parent "TJB" <broughtj gmail.com> writes:
On Monday, 24 March 2014 at 11:57:14 UTC, Daniel Davidson wrote:
 On Saturday, 22 March 2014 at 17:30:45 UTC, TJB wrote:
 On Saturday, 22 March 2014 at 16:35:07 UTC, Brian Rogoff wrote:

 This is a very interesting thread that you started. Could you 
 flesh it out more with some example C++ that you'd like 
 compared to D? I'm sure quite a few people would assist with 
 a translation.

Well, right away people jumped to high-frequency trading. Although that may be the most visible area in computational finance - it's not the only one. There are areas where performance is crucial, but where trading is done at a lower frequency (where latency is not the main issue).

I apologize for that. HFT has been a driver of a lot of business and attention. Of course you are right about areas with less latency sensitivity and D is attractive there. Even in latency sensitive efforts I think could be D attractive for new efforts providing some of the memory management efforts continue to evolve. My main point was selling it would be tough. Thanks Dan

Dan, Thanks for your thoughtful points. I think your experience is worth listening to - I think it confirms that both: * D is worth pursuing in these areas * It won't be easy to convince others to adopt it These points are worth thinking deeply about as D is developed for the real world. TJB
Mar 24 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 23 March 2014 at 17:38:17 UTC, Sean Kelly wrote:
 On Saturday, 22 March 2014 at 14:04:01 UTC, Daniel Davidson 
 wrote:
 For example, I could see technical reasons why in certain 
 non-quant areas like XML parsing where D can be faster than 
 C++. 
 (http://dotnot.org/blog/archives/2008/03/12/why-is-dtango-so-f
st-at-parsing-xml/) 
 But then, with a large amount of time and unlimited funding 
 the techniques could probably be duplicated in C++.

Try no funding and a trivial amount of time. The JSON parser I wrote for work in C performs zero allocations and unescaping is performed on demand. D arguably makes this easier by building slicing into the language, but not decoding or copying is a design decision, not a language artifact (at least in the case of C/C++ where aliasing data is allowed). The take-away from that Tango article is that the performance hit for parsing is aggressively decoding data the user may not care about or may not want decoded in the first place. This just happens to be the approach that basically every XML parser on the planet uses for some ridiculous reason.

This isn't for ridiculous reasons, this is because in other languages you have no guarantee that what you work on is immutable. So you must aggressively copy anyway. With a separate decoding step, you'll ends up copying twice, which is also wasteful.
Mar 24 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 23 March 2014 at 18:58:20 UTC, Walter Bright wrote:
 On 3/23/2014 11:29 AM, "Ola Fosheim Grøstad"
 While that is true you can have a soft real time thread 
 feeding the hard real time thread

Yes, and you can do that with GC, too.

You can, but the way I view "soft real time" is that you accept jitter, but not prolonged freezing of threads. I don't think the current GC is "soft real time".
Mar 24 2014