www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Finally full multidimensional arrays support in D

reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
Multidimensional arrays indexing and slicing syntax is finally added [1] 
(thanks to Kenji Hara). So it was a good cause to update my 
multidimensional arrays library implementation and add support for the 
new syntax. So here we are: [2].

Also should we add it to the standard library?

[1] https://github.com/D-Programming-Language/dmd/pull/443
[2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html

-- 
Денис В. Шеломовский
Denis V. Shelomovskij
Mar 17 2014
next sibling parent "watcher" <watcher watcher.com> writes:
+100



On Monday, 17 March 2014 at 17:39:41 UTC, Denis Shelomovskij 
wrote:
 Multidimensional arrays indexing and slicing syntax is finally 
 added [1] (thanks to Kenji Hara). So it was a good cause to 
 update my multidimensional arrays library implementation and 
 add support for the new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] 
 http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html

Mar 17 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Mar 17, 2014 at 09:39:40PM +0400, Denis Shelomovskij wrote:
 Multidimensional arrays indexing and slicing syntax is finally added
 [1] (thanks to Kenji Hara). So it was a good cause to update my
 multidimensional arrays library implementation and add support for
 the new syntax. So here we are: [2].
 
 Also should we add it to the standard library?
 
 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html

Let's do it! Put it up for review! I almost gave up hope that this day would come, but this morning I looked at the dmd commit log and saw Kenji's pull was merged. You couldn't believe how happy I was. Finally, sane multidimensional array support!! T -- Almost all proofs have bugs, but almost all theorems are true. -- Paul Pedersen
Mar 17 2014
prev sibling next sibling parent "Jared Miller" <jared economicmodeling.com> writes:
Good news and kudos to Kenji.

And yes, I think that a matrix / linear algebra library, as well 
as NumPy-style ND-Arrays are great candidates for future Phobos 
modules.

We've done a good bit of work with matrices at my workplace 
(EMSI), so I bet someone here could find time to contribute a 
bit. As a start, we've got some basic BLAS and MKL bindings that 
we really should clean up and get into Deimos.

On Monday, 17 March 2014 at 17:39:41 UTC, Denis Shelomovskij 
wrote:
 Multidimensional arrays indexing and slicing syntax is finally 
 added [1] (thanks to Kenji Hara). So it was a good cause to 
 update my multidimensional arrays library implementation and 
 add support for the new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] 
 http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html

Mar 17 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Jared Miller:

 And yes, I think that a matrix / linear algebra library, as 
 well as NumPy-style ND-Arrays are great candidates for future 
 Phobos modules.

I suggest to not put such library in Phobos before few years of usage in the wild. Bye, bearophile
Mar 17 2014
prev sibling next sibling parent reply "Christof Schardt" <csnews schardt.info> writes:
I looked to the doku-page and expected to see, how some basic
matrix-operations could be performed.
Like e.g. (basic syntax) this:

 Dim  a(10,12)
 a(3,4) = 7
 a(i,j) = a(j,i) * 12

Instead I found exotic assignments and calculations.
Did I expect the wrong thing?
Mar 17 2014
parent Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
18.03.2014 1:35, Christof Schardt :
 I looked to the doku-page and expected to see, how some basic
 matrix-operations could be performed.
 Like e.g. (basic syntax) this:

   Dim  a(10,12)
   a(3,4) = 7
   a(i,j) = a(j,i) * 12

 Instead I found exotic assignments and calculations.
 Did I expect the wrong thing?

Sorry for that. Do you mead there is a lack of simple indexing examples or lack of some matrix functionality? The latter case is if you meant "transpose and multiply by 12" with `a(i,j) = a(j,i) * 12`. -- . Denis V. Shelomovskij
Mar 20 2014
prev sibling next sibling parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
17.03.2014 21:39, Denis Shelomovskij пишет:
 Multidimensional arrays indexing and slicing syntax is finally added [1]
 (thanks to Kenji Hara). So it was a good cause to update my
 multidimensional arrays library implementation and add support for the
 new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html

An important note: My proposal is just a sliceable multidimensional array implementation, a try to provide the same effort we have for simple arrays for multidimensional ones. It isn't aimed to be a dedicated matrix / linear algebra module. -- Денис В. Шеломовский Denis V. Shelomovskij
Mar 17 2014
parent "Mason McGill" <mmcgill caltech.edu> writes:
 It will be a good low-level module that will allow a linear 
 algebra
 module to be built on.

Agreed. In fields like graphics and computer vision, large multidimensional arrays aren't always matrices/tensors in the linear algebra sense. And it's always possible to specialize. NumPy got this right.
Mar 17 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Mar 18, 2014 at 01:55:31AM +0400, Denis Shelomovskij wrote:
 17.03.2014 21:39, Denis Shelomovskij пишет:
Multidimensional arrays indexing and slicing syntax is finally added [1]
(thanks to Kenji Hara). So it was a good cause to update my
multidimensional arrays library implementation and add support for the
new syntax. So here we are: [2].

Also should we add it to the standard library?

[1] https://github.com/D-Programming-Language/dmd/pull/443
[2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html

An important note: My proposal is just a sliceable multidimensional array implementation, a try to provide the same effort we have for simple arrays for multidimensional ones. It isn't aimed to be a dedicated matrix / linear algebra module.

It will be a good low-level module that will allow a linear algebra module to be built on. I am against the idea of conflating multidimensional arrays with matrices; the operations and interpretations are not the same, and it leads to poor designs (I've tried it before, and the results were not so good). Better have a general multidimensional array with per-element operations, and then build matrix algebra as a separate module by using multidimensional array as underlying storage implementation. T -- Answer: Because it breaks the logical sequence of discussion. / Question: Why is top posting bad?
Mar 17 2014
prev sibling next sibling parent "Stefan Frijters" <sfrijters gmail.com> writes:
On Monday, 17 March 2014 at 17:39:41 UTC, Denis Shelomovskij 
wrote:
 Multidimensional arrays indexing and slicing syntax is finally 
 added [1] (thanks to Kenji Hara). So it was a good cause to 
 update my multidimensional arrays library implementation and 
 add support for the new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] 
 http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html

First of all, thank you very much for making such nice additions to D available for general use. I finally got around to giving this a spin. I'm using it for a proof-of-context HPC simuation code written in D (currently mostly experimenting with D's features), and as such I'm interfacing with the C MPI library to communicate between processes. The basis of the simulation is a 3D lattice, so I was eagerly awaiting a nice solution in D. So far I've run into two things while using your library. The first is that I need to provide void pointers to the data to the MPI functions, so I currently hacked your code to make the _data storage array publicly accessible and that seems to work. To give an idea, I currently have code like this (just a snippet): arr = multidimArray!T(nxH, nyH, nzH); // [...] fill the array with data // Prepare a buffer to receive a slice from another process. rbuffer = multidimArray!T(haloSize, nyH, nzH); // Prepare a buffer to send a slice to another process. sbuffer = arr[$-2*haloSize-1..$ - haloSize-1, 0..$, 0..$].dup; // Here I now use the pointer of the storage arrays to send the buffer around. MPI_Sendrecv(sbuffer._data.ptr, nyH * nzH, MPI_INT, M.nbx[1], 0, rbuffer._data.ptr, nyH * nzH, mpiType, M.nbx[0], 0, M.comm, &mpiStatus); // Put the buffer in the correct spot in the main array. arr[0..haloSize, 0..$, 0..$] = rbuffer; Am I missing a nicer way to accomplish this? I like the compactness of the code (compared to what I'm currently used to with our F90 simulation code). Secondly, the property that returns the dimensions of the array is called 'dimentions' (with a t), this should be fixed. Regards, Stefan
Apr 16 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Stefan Frijters:

 First of all, thank you very much for making such nice 
 additions to D available for general use. I finally got around 
 to giving this a spin.

Recently I've shown a possible usage example of the multidimensional arrays indexing and slicing syntax: http://forum.dlang.org/thread/cizugfrkaunlkzyjptnp forum.dlang.org Bye, bearophile
Apr 16 2014
prev sibling next sibling parent "CJS" <Prometheus85 hotmail.com> writes:
On Monday, 17 March 2014 at 21:25:34 UTC, bearophile wrote:
 Jared Miller:

 And yes, I think that a matrix / linear algebra library, as 
 well as NumPy-style ND-Arrays are great candidates for future 
 Phobos modules.

I suggest to not put such library in Phobos before few years of usage in the wild.

+1 Good matrix support would be awesome. But getting it wrong would be a catastrophe. I don't really support ever putting it in phobos, but if it is, then it should only be added after lots of experience.
Apr 17 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Apr 17, 2014 at 03:16:20PM +0000, CJS via Digitalmars-d wrote:
 On Monday, 17 March 2014 at 21:25:34 UTC, bearophile wrote:
Jared Miller:

And yes, I think that a matrix / linear algebra library, as well as
NumPy-style ND-Arrays are great candidates for future Phobos
modules.

I suggest to not put such library in Phobos before few years of usage in the wild.

+1 Good matrix support would be awesome. But getting it wrong would be a catastrophe. I don't really support ever putting it in phobos, but if it is, then it should only be added after lots of experience.

I've been longing for a high-quality, flexible, generic linear algebra library in D. I don't have the time / resources to implement it myself, otherwise I would. But I agree that any such candidate library needs to be put in real-life use for a while before being considered for Phobos. I think the first step would be to refine Denis' n-dimensional array library until it's Phobos-quality, then linear algebra specific adaptations can be built on top. I think the two should be separated, even if they are still related. Conflating 2D arrays with matrices at a fundamental level is a mistake IMO. 2D arrays are just one of the possible representations of a matrix, and any linear algebra library should be flexible enough to use other representations (e.g., sparse matrices). T -- He who laughs last thinks slowest.
Apr 17 2014
prev sibling next sibling parent "Stefan Frijters" <sfrijters gmail.com> writes:
I've been using the multidimensional arrays for a while now, but 
recently I've run into a problem w.r.t. optimization:

import std.stdio;
import unstd.multidimarray;

void main() {
   MultidimArray!(double, 3) arr;
   arr = multidimArray!double([1,2,42]);
   writeln(arr.lengths);
}

If I compile with 'dmd multidimtest.d unstd/multidimarray.d', I 
get [1,2,42], as expected, but when I compile in release mode 
'dmd multidimtest.d unstd/multidimarray.d -release' I get 
[0,0,0]. Any ideas what might cause this?

Also, when I compile with -w (which I normally prefer) I get a 
list of errors of the type

unstd/multidimarray.d(101): Warning: calling 
unstd.multidimarray.MultidimArray!(double, 
3LU).MultidimArray.this without side effects discards return 
value of type inout(MultidimArray!(double, 3LU)), prepend a 
cast(void) if intentional

I've acted on these errors in the version I use for my HPC test 
code, and I have not noticed any deleterious effects; would you 
accept a pull request to fix these?

Kind regards,

Stefan Frijters
Jun 10 2014
prev sibling next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 17.03.2014 18:39, schrieb Denis Shelomovskij:
 Multidimensional arrays indexing and slicing syntax is finally added [1]
 (thanks to Kenji Hara). So it was a good cause to update my
 multidimensional arrays library implementation and add support for the
 new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html

Yes please
Jun 10 2014
prev sibling parent "Stefan Frijters" <sfrijters gmail.com> writes:
On Tuesday, 10 June 2014 at 19:04:18 UTC, Stefan Frijters wrote:
 I've been using the multidimensional arrays for a while now, 
 but recently I've run into a problem w.r.t. optimization:

 import std.stdio;
 import unstd.multidimarray;

 void main() {
   MultidimArray!(double, 3) arr;
   arr = multidimArray!double([1,2,42]);
   writeln(arr.lengths);
 }

 If I compile with 'dmd multidimtest.d unstd/multidimarray.d', I 
 get [1,2,42], as expected, but when I compile in release mode 
 'dmd multidimtest.d unstd/multidimarray.d -release' I get 
 [0,0,0]. Any ideas what might cause this?

I've looked into this and it seems there is required code inside a contract 'in', which is omitted with -release. I've gone ahead and opened a pull request to fix this: https://bitbucket.org/denis-sh/unstandard/pull-request/2/bugfixes-for-multidimarrayd Stefan
Jun 11 2014