www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Netflix opensources its first D library: Vectorflow

reply Walter Bright <newshound2 digitalmars.com> writes:
https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/
Aug 02
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Wednesday, 2 August 2017 at 21:31:19 UTC, Walter Bright wrote:
 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/
No. 2 liked proggit link of the day, should be no. 1 soon: https://www.reddit.com/r/programming/top/?time=day Not doing well on HN though: https://hn.algolia.com/?query=vectorflow
Aug 02
next sibling parent Samuel Lampa <samuel.lampa gmail.com> writes:
On Wednesday, 2 August 2017 at 22:56:32 UTC, Joakim wrote:
 Not doing well on HN though:

 https://hn.algolia.com/?query=vectorflow
HN is very sensitive to time of day when submitting. Did a new try: https://news.ycombinator.com/item?id=14920608
Aug 03
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Wednesday, 2 August 2017 at 22:56:32 UTC, Joakim wrote:
 On Wednesday, 2 August 2017 at 21:31:19 UTC, Walter Bright 
 wrote:
 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/
No. 2 liked proggit link of the day, should be no. 1 soon: https://www.reddit.com/r/programming/top/?time=day Not doing well on HN though: https://hn.algolia.com/?query=vectorflow
Top 3 for the week: https://www.reddit.com/r/programming/top/?sort=top&t=week People seem really enthused by this library.
Aug 05
prev sibling parent reply Matt <sdfssdfd gmail.com> writes:
On Wednesday, 2 August 2017 at 21:31:19 UTC, Walter Bright wrote:
 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/
Speakng of D in data science (where I think it can get traction), is there a standardized linear algebra library in D? Perhaps some hooks to Eigen? I saw a few upstarts LA libraries, but none that I would consider "the one to use in D", like numpy in python or Eigen in C++
Aug 02
parent reply Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:
On Thursday, 3 August 2017 at 03:46:11 UTC, Matt wrote:
 On Wednesday, 2 August 2017 at 21:31:19 UTC, Walter Bright 
 wrote:
 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/
Speakng of D in data science (where I think it can get traction), is there a standardized linear algebra library in D? Perhaps some hooks to Eigen? I saw a few upstarts LA libraries, but none that I would consider "the one to use in D", like numpy in python or Eigen in C++
We're using D in finance. D libraries are far from the maturity of Python, but they are developing quickly. https://github.com/kaleidicassociates/lubeck Library we have open-sourced https://github.com/libmir/mir-algorithm Mir library we sponsor Other mir libraries: https://github.com/libmir/mir-lapack https://github.com/libmir/mir-blas https://github.com/libmir/lapack https://github.com/DlangScience/cblas Performance matters: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html
Aug 02
next sibling parent reply Matt <sdfssdfd gmail.com> writes:
That seems like a good start. I'll probably start contributing 
within a few weeks.

In the long run the goal should probably be to have a "go-to" set 
of LA libraries, with appropriate data structures that have 
friendly bindings to GPU LA computations with DCompute.
Aug 02
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 3 August 2017 at 04:37:30 UTC, Matt wrote:
 That seems like a good start. I'll probably start contributing 
 within a few weeks.
Great, http://gitter.im/libmir is probably the best way to get in contact.
 In the long run the goal should probably be to have a "go-to" 
 set of LA libraries, with appropriate data structures that have 
 friendly bindings to GPU LA computations with DCompute.
That is the idea, https://github.com/libmir/mir-glas already provides a BLAS and I'll eventually get around to getting DCompute integration for it and ndslice.
Aug 02
prev sibling parent reply Matt <sdfssdfd gmail.com> writes:
Also note, one of the main advantages of Eigen is the whole lazy 
evaluation of expressions for compound operations.

I haven't dug in the source, but it's my understanding it's done 
through a lot of compile time C++ template hacking
Aug 02
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Thursday, 3 August 2017 at 04:40:05 UTC, Matt wrote:
 Also note, one of the main advantages of Eigen is the whole 
 lazy evaluation of expressions for compound operations.
Yes, Mir does that too: http://blog.mir.dlang.io/ndslice/algorithm/optimization/2016/12/12/writing-efficient-numerical-code.html
 I haven't dug in the source, but it's my understanding it's 
 done through a lot of compile time C++ template hacking
Meanwhile, the blog post Laeeth gave you shows Mir doing better on matrix multiplication benchmarks than Eigen, significantly better when dealing with complex numbers.
Aug 02
parent reply Matt <sdfssdfd gmail.com> writes:
 Meanwhile, the blog post Laeeth gave you shows Mir doing better 
 on matrix multiplication benchmarks than Eigen, significantly 
 better when dealing with complex numbers.
I mean by now we should all be jaded enough not to simply take toy benchmarks as gospel for which is actually fastest in a non-trivial application. I don't doubt mir is really fast, though.
Aug 03
parent Joakim <dlang joakim.fea.st> writes:
On Thursday, 3 August 2017 at 14:00:31 UTC, Matt wrote:
 Meanwhile, the blog post Laeeth gave you shows Mir doing 
 better on matrix multiplication benchmarks than Eigen, 
 significantly better when dealing with complex numbers.
I mean by now we should all be jaded enough not to simply take toy benchmarks as gospel for which is actually fastest in a non-trivial application.
That's why I didn't make such a general claim and noted that the linked benchmarks only dealt with matrix multiplication. :P
 I don't doubt mir is really fast, though.
Yes, the benchmarks are indicative, but it's up to you come up with a benchmark that characterizes your workload better.
Aug 03
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 3 August 2017 at 04:40:05 UTC, Matt wrote:
 Also note, one of the main advantages of Eigen is the whole 
 lazy evaluation of expressions for compound operations.

 I haven't dug in the source, but it's my understanding it's 
 done through a lot of compile time C++ template hacking
Yep, there's a lot of operator overloading and other C++ black magic. DCompute should be able to do some of the out of the box with the lambda kernels to get some data locality. I'm also in the process of integrating Polly (LLVM's polyhedral optimiser) into LDC to get great loop optimisations.
Aug 02
prev sibling parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Thursday, 3 August 2017 at 04:40:05 UTC, Matt wrote:
 Also note, one of the main advantages of Eigen is the whole 
 lazy evaluation of expressions for compound operations.

 I haven't dug in the source, but it's my understanding it's 
 done through a lot of compile time C++ template hacking
Note that D provides __traits(isRef, Symbol) which can be used to convenient implement lazy-evaluated expressions with free functions. Note that it cannot currently be used to check whether `this` was passed as an l-value or r-value which prevents the C++-expression-template-pattern from being used in operator overloading. For details see: https://issues.dlang.org/show_bug.cgi?id=17734
Aug 08
parent =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Tuesday, 8 August 2017 at 18:40:08 UTC, Nordlöw wrote:
 On Thursday, 3 August 2017 at 04:40:05 UTC, Matt wrote:
 Also note, one of the main advantages of Eigen is the whole 
 lazy evaluation of expressions for compound operations.

 I haven't dug in the source, but it's my understanding it's 
 done through a lot of compile time C++ template hacking
Note that D provides __traits(isRef, Symbol)
To clarify here's an incomplete snippet that should clarify: auto add(A, B)(A a, B b) if (isSomeArithmeticType!A && isSomeArithmeticType!B) { static if (__traits(isRef, a) && __traits(isRef, b)) // both `a` and `b` are l-values { return a + b; // fully eager evaluation } else static if (__traits(isRef, a)) // `b` is an r-value { // `b` can incremented by `a` and returned by move (reused) } else static if (__traits(isRef, b)) // `a` is an r-value { // `a` can incremented by `b` and returned by move (reused) } else // both `a` and `b` are r-values { return Add(a,b); // delay evaluation } }
Aug 08