www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Using D and std.ndslice as a Numpy Replacement

reply Jack Stouffer <jack jackstouffer.com> writes:
http://jackstouffer.com/blog/nd_slice.html

https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
Jan 02
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 01/02/2016 11:49 AM, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
Broken link in "For a more in depth look at ranges, see The official D tutorial's section on ranges". Ali
Jan 02
parent Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 2 January 2016 at 22:15:03 UTC, Ali Çehreli wrote:
 Broken link in "For a more in depth look at ranges, see The 
 official D tutorial's section on ranges".

 Ali
Fixed, thanks!
Jan 02
prev sibling next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
I just wanted to write to you that dip80-ndslice was moved to mir http://code.dlang.org/packages/mir "dependencies": { "dip80-ndslice": "~>0.8.7" }, Ilya
Jan 02
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Saturday, 2 January 2016 at 23:23:38 UTC, Ilya Yaroshenko 
wrote:
 On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer 
 wrote:
 http://jackstouffer.com/blog/nd_slice.html

https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
I just wanted to write to you that dip80-ndslice was moved to mir http://code.dlang.org/packages/mir "dependencies": { "dip80-ndslice": "~>0.8.7" }, Ilya
EDIT: "dependencies": { "mir": "~>0.9.0-beta" }
Jan 02
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/2/16 6:24 PM, Ilya Yaroshenko wrote:
 On Saturday, 2 January 2016 at 23:23:38 UTC, Ilya Yaroshenko wrote:
 On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
I just wanted to write to you that dip80-ndslice was moved to mir http://code.dlang.org/packages/mir "dependencies": { "dip80-ndslice": "~>0.8.7" }, Ilya
EDIT: "dependencies": { "mir": "~>0.9.0-beta" }
What is the relationship between mir and std.experimental.ndslice? -- Andrei
Jan 03
parent reply Ilya <ilyayaroshenko gmail.com> writes:
On Sunday, 3 January 2016 at 23:18:16 UTC, Andrei Alexandrescu 
wrote:
 On 1/2/16 6:24 PM, Ilya Yaroshenko wrote:
 On Saturday, 2 January 2016 at 23:23:38 UTC, Ilya Yaroshenko 
 wrote:
 On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer 
 wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
I just wanted to write to you that dip80-ndslice was moved to mir http://code.dlang.org/packages/mir "dependencies": { "dip80-ndslice": "~>0.8.7" }, Ilya
EDIT: "dependencies": { "mir": "~>0.9.0-beta" }
What is the relationship between mir and std.experimental.ndslice? -- Andrei
1. mir.ndslice is a developer version of std.experimental.ndslice 2. mir can be used with DMD front end >= 2.068, so ndslice can be used with LDC 0.17.0-alpha. It is important for benchmarks. 3. mir is going to be a testing package for the future std.la (generic BLAS implementation) -- Ilya
Jan 03
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/3/16 6:52 PM, Ilya wrote:
 1. mir.ndslice is a developer version of std.experimental.ndslice
 2. mir can be used with DMD front end >= 2.068, so ndslice can be used
 with LDC 0.17.0-alpha. It is important for benchmarks.
 3. mir is going to be a testing package for the future std.la (generic
 BLAS implementation)
The care you show for your users is impressive. We should take a page from your book, hopefully with your own help. Thanks! -- Andrei
Jan 03
prev sibling next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
Jack, Thank you for the article! Link to the benchmark at GitHub https://github.com/DlangScience/examples/blob/master/mean_of_columns.d This benchmark is _not_ lazy, so ndslice faster than Numpy only 3.5 times. -- Ilya
Jan 02
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 2 January 2016 at 23:51:09 UTC, Ilya Yaroshenko 
wrote:
 This benchmark is _not_ lazy, so ndslice faster than Numpy only 
 3.5 times.
I don't know what you mean here, I made sure to call std.array.array to force allocation.
Jan 02
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 3 January 2016 at 00:09:33 UTC, Jack Stouffer wrote:
 On Saturday, 2 January 2016 at 23:51:09 UTC, Ilya Yaroshenko 
 wrote:
 This benchmark is _not_ lazy, so ndslice faster than Numpy 
 only 3.5 times.
I don't know what you mean here, I made sure to call std.array.array to force allocation.
In the article: auto means = 100_000.iota <---- 100_000.iota is lazy range .sliced(100, 1000) .transposed .map!(r => sum(r) / r.length) .array; <---- allocation of the result In GitHub: means = data <---- data is allocated array, it is fair test for real world .sliced(100, 1000) .transposed .map!(r => sum(r, 0L) / cast(double) r.length) .array; <---- allocation of the result -- Ilya
Jan 02
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Sunday, 3 January 2016 at 00:17:23 UTC, Ilya Yaroshenko wrote:
 On Sunday, 3 January 2016 at 00:09:33 UTC, Jack Stouffer wrote:
 On Saturday, 2 January 2016 at 23:51:09 UTC, Ilya Yaroshenko 
 wrote:
 This benchmark is _not_ lazy, so ndslice faster than Numpy 
 only 3.5 times.
I don't know what you mean here, I made sure to call std.array.array to force allocation.
In the article: auto means = 100_000.iota <---- 100_000.iota is lazy range .sliced(100, 1000) .transposed .map!(r => sum(r) / r.length) .array; <---- allocation of the result In GitHub: means = data <---- data is allocated array, it is fair test for real world .sliced(100, 1000) .transposed .map!(r => sum(r, 0L) / cast(double) r.length) .array; <---- allocation of the result -- Ilya
I still have to disagree with you that the example I submitted was fair. Accessing global memory in D is going to be much slower than accessing stack memory, and sense most std.ndslice calculations are going to be on the stack, I believe my benchmark is indicative of normal use.
Jan 03
next sibling parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 3 January 2016 at 18:56:07 UTC, Jack Stouffer wrote:
 On Sunday, 3 January 2016 at 00:17:23 UTC, Ilya Yaroshenko 
 wrote:
  [...]
I still have to disagree with you that the example I submitted was fair. Accessing global memory in D is going to be much slower than accessing stack memory, and sense most std.ndslice calculations are going to be on the stack, I believe my benchmark is indicative of normal use.
No, for real world math calculations, most std.ndslice will be at global memory. Examples: all SciD (if port it to ndslice), future BLAS, future LAPACK. -- Ilya
Jan 03
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Sunday, 3 January 2016 at 18:56:07 UTC, Jack Stouffer wrote:
 I still have to disagree with you that the example I submitted 
 was fair. Accessing global memory in D is going to be much 
 slower than accessing stack memory, […]
What leads you to this belief? (Beyond cache locality considerations, which are not so important if the data is large.)
 and sense most std.ndslice calculations are going to be on the 
 stack, I believe my benchmark is indicative of normal use.
Your iota example does not read the data from memory at all (neither stack nor heap), instead computing it on the fly. — David
Jan 03
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Monday, 4 January 2016 at 00:24:51 UTC, David Nadlinger wrote:
 On Sunday, 3 January 2016 at 18:56:07 UTC, Jack Stouffer wrote:
 I still have to disagree with you that the example I submitted 
 was fair. Accessing global memory in D is going to be much 
 slower than accessing stack memory, […]
What leads you to this belief? (Beyond cache locality considerations, which are not so important if the data is large.)
The example in the article and the example I submitted to DlangScience/examples have very different speeds: https://github.com/DlangScience/examples/blob/master/mean_of_columns.d Article example: 5 µs DlangScience example: 145 µs Both when compiled with LDC
Jan 03
parent reply Ilya <ilyayaroshenko gmail.com> writes:
On Monday, 4 January 2016 at 01:03:49 UTC, Jack Stouffer wrote:
 On Monday, 4 January 2016 at 00:24:51 UTC, David Nadlinger 
 wrote:
 On Sunday, 3 January 2016 at 18:56:07 UTC, Jack Stouffer wrote:
 I still have to disagree with you that the example I 
 submitted was fair. Accessing global memory in D is going to 
 be much slower than accessing stack memory, […]
What leads you to this belief? (Beyond cache locality considerations, which are not so important if the data is large.)
The example in the article and the example I submitted to DlangScience/examples have very different speeds: https://github.com/DlangScience/examples/blob/master/mean_of_columns.d Article example: 5 µs DlangScience example: 145 µs Both when compiled with LDC
To be clear: there is NO data in Article example. Only CPU registers are used. It is not fair. -- Ilya
Jan 03
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Monday, 4 January 2016 at 01:09:30 UTC, Ilya wrote:
 To be clear: there is NO data in Article example. Only CPU 
 registers are used. It is not fair. -- Ilya
Ok, I see were I made the mistake, I apologize. I believed that since I was only testing the np.mean line of code, that the lazy generation of the D in the D version would have no effects on the comparison. But I forgot the fact that the lazy generation would mean that no memory needed to be accessed in the D code and therefore it's not apples to apples with the Numpy code. Thank you for clarifying this, I will update the article.
Jan 03
parent reply Ilya <ilyayaroshenko gmail.com> writes:
On Monday, 4 January 2016 at 01:47:04 UTC, Jack Stouffer wrote:
 On Monday, 4 January 2016 at 01:09:30 UTC, Ilya wrote:
 To be clear: there is NO data in Article example. Only CPU 
 registers are used. It is not fair. -- Ilya
Ok, I see were I made the mistake, I apologize. I believed that since I was only testing the np.mean line of code, that the lazy generation of the D in the D version would have no effects on the comparison. But I forgot the fact that the lazy generation would mean that no memory needed to be accessed in the D code and therefore it's not apples to apples with the Numpy code. Thank you for clarifying this, I will update the article.
Please add links to 1. Mir: http://code.dlang.org/packages/mir 2. Documentation: http://dlang.org/phobos-prerelease/std_experimental_ndslice.html 3. DlangScience example Thanks! -- Ilya
Jan 03
parent Jack Stouffer <jack jackstouffer.com> writes:
On Monday, 4 January 2016 at 02:01:05 UTC, Ilya wrote:
 Please add links to
 1. Mir: http://code.dlang.org/packages/mir
 2. Documentation: 
 http://dlang.org/phobos-prerelease/std_experimental_ndslice.html
 3. DlangScience example

 Thanks! -- Ilya
Updated. I also changed the example slightly from the DlangScience example by increasing the array size because some people on HN were complaining that the size of the data wasn't large enough for the benchmark to be meaningful.
Jan 04
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/2/2016 11:49 AM, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
https://news.ycombinator.com/item?id=10828450 (but access through the front page, not this link)
Jan 02
prev sibling next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
Great piece. And great work done by those who worked on nd_slice.
Jan 02
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
Nicely written, good to see you explain all the code, enjoyed reading it.
Jan 03