digitalmars.D - Matrix-type-friendly syntax and more
- bearophile (27/31) Oct 09 2011 Kenji Hara has just copied in GIT a large patch by Don:
- Denis Shelomovskij (8/39) Oct 09 2011 Suddenly, I had a talk with an Fortran guy at previous weekend and have
- kenji hara (25/123) Oct 09 2011 I got an idea for multidimentional indexing and slicing.
- bearophile (8/10) Oct 09 2011 Thank you, you are good.
- kenji hara (6/10) Oct 09 2011 I have no thought about it.
- Don (3/14) Oct 09 2011 Personally, I think that since strided operations are so inefficient,
- bearophile (5/7) Oct 10 2011 In Chapel (that's a high performance language full of nice ideas) the st...
- kennytm (5/22) Oct 10 2011 Slicing an nD static array is just as in/efficient as strides. That said...
- Robert Jacques (2/24) Oct 10 2011 Regarding (1), from a computer vision/graphics perspective, being able t...
- kennytm (2/32) Oct 10 2011 Why not use a 'Color' struct?
- Robert Jacques (2/34) Oct 11 2011 Not all images have the same layout, so having a single Color struct doe...
- Gor Gyolchanyan (6/55) Oct 11 2011 I don't think strides are necessary. You can always opApply your image
- bearophile (6/11) Oct 11 2011 See:
- Norbert Nemec (6/11) Oct 11 2011 Actually, a strided slice to a multidimensional array can easily be
- Robert Jacques (8/13) Oct 11 2011 First, slicing / striding doesn't imply copying.
- bearophile (5/7) Oct 11 2011 OK, a nice striding syntax for user defined types will be a good thing t...
- travert phare.normalesup.org (Christophe) (13/24) Nov 03 2011 : is already used for labels. If slice types are allowed, this can
- Norbert Nemec (10/34) Nov 05 2011 Strided ranges could be a tertiary operator
- Norbert Nemec (5/22) Oct 10 2011 Inefficient?! For multidimensional arrays, all but one dimension have
- Don (5/27) Oct 11 2011 That's true, IF you are happy to have it as a feature which can only be
- bearophile (13/16) Oct 11 2011 This is not wonderful, but for me it's acceptable. Having that syntax su...
- Norbert Nemec (5/17) Oct 11 2011 Sure. For one-dimensional arrays, the overhead would be significant and
- Norbert Nemec (2/14) Oct 09 2011 Nice! This seems like the way to go.
- travert phare.normalesup.org (Christophe) (15/22) Nov 03 2011 Another option, that I must have posted somewhere else in the past:
- kenji hara (31/139) Oct 10 2011 Posted pull request.
- bearophile (7/13) Oct 10 2011 Things are getting interesting. Thank you.
- Denis Shelomovskij (5/12) Oct 13 2011 My usable draft example of multidimensional rectangular arrays (source
- bearophile (5/7) Oct 13 2011 OK, a nice striding syntax for user defined types will be a good thing t...
Kenji Hara has just copied in GIT a large patch by Don: https://github.com/9rnsr/dmd/branches/opDollar It allows user defined collections usable with this syntax: x[$-2, y[$-6, $-9], $-2] See: http://d.puremagic.com/issues/show_bug.cgi?id=3474 But a syntax like this too is quite important for a user-defined matrix type to be used in numerical code: m[i..j, k..w] Similar slicing and dicing of arrays is used all the time in NumPy (array library for Python) code. This is a comment by Don about it:(4) I have NOT implemented $ inside opSlice(), opSliceAssign(). It could be done, but I believe those language features need work. They don't permit multi-dimensional slicing. I think they should be removed, and the functionality folded into opIndex.Isn't it better to improve opSlice() instead of deprecating it? I don't remember if people appreciate the idea of a stride (like the third step argument of std.range.iota), this too is used in scientific array-oriented code: m[i..j:2, k..w:3] --------------- In Python code there is often the need for a good multi-dimensional array type, even in not-scientific code. In the Python standard library there is a array module, but it's almost a joke, it's 1D, and it's not used much: http://docs.python.org/library/array.html Python programmers use the multi-dimensional array of NumPy, it's widely used around the world, it's used by SciPy too (a scientific programming library). The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible array libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ ) and numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html), tells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want. I think that's a good tradeoff between the opposed needs of: - Keeping Phobos of reasonable size (to not increase too much the burden of its management, to not slow down too much its development); - Avoiding the risk of incompatible multi-dimensional array types. Most code out there is able to build on a common foundation. This avoids duplication (like the creation of Python numarray and Numeric), allows a better focusing of efforts and speeds up the development of a language-wide standard for such arrays; - Offer a nD array type to the casual D programmer, even one that doesn't want or can't install other libraries. Even some 30-lines long D programs need multi-dimensional arrays, but they often don't need a complex scientific library too (example of a problem: in the preconditions of my functions that take an array of arrays I always have to test the input is not jagged and it is a rectangular matrix. Such test is not necessary for a multi-dimensional array that is never jagged). Putting the bare bones multi-dimensional array type in Phobos allows people to use them with zero other installs. This multi-dimensional Phobos array type doesn't even need to contain code to invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change of shape, iteration... Everything else is in modules/packages external to Phobos. I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules. Bye, bearophile
Oct 09 2011
09.10.2011 21:29, bearophile ïèøåò:Kenji Hara has just copied in GIT a large patch by Don: https://github.com/9rnsr/dmd/branches/opDollar It allows user defined collections usable with this syntax: x[$-2, y[$-6, $-9], $-2] See: http://d.puremagic.com/issues/show_bug.cgi?id=3474 But a syntax like this too is quite important for a user-defined matrix type to be used in numerical code: m[i..j, k..w] Similar slicing and dicing of arrays is used all the time in NumPy (array library for Python) code. This is a comment by Don about it:Suddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of adding it into Phobos, but just had no time for review of my code. Draft version is in attachment. rarray.d - something like Fortran array main.d - example of using rarray module output.txt - main.d output(4) I have NOT implemented $ inside opSlice(), opSliceAssign(). It could be done, but I believe those language features need work. They don't permit multi-dimensional slicing. I think they should be removed, and the functionality folded into opIndex.Isn't it better to improve opSlice() instead of deprecating it? I don't remember if people appreciate the idea of a stride (like the third step argument of std.range.iota), this too is used in scientific array-oriented code: m[i..j:2, k..w:3] --------------- In Python code there is often the need for a good multi-dimensional array type, even in not-scientific code. In the Python standard library there is a array module, but it's almost a joke, it's 1D, and it's not used much: http://docs.python.org/library/array.html Python programmers use the multi-dimensional array of NumPy, it's widely used around the world, it's used by SciPy too (a scientific programming library). The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible array libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ ) and numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html), tells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want. I think that's a good tradeoff between the opposed needs of: - Keeping Phobos of reasonable size (to not increase too much the burden of its management, to not slow down too much its development); - Avoiding the risk of incompatible multi-dimensional array types. Most code out there is able to build on a common foundation. This avoids duplication (like the creation of Python numarray and Numeric), allows a better focusing of efforts and speeds up the development of a language-wide standard for such arrays; - Offer a nD array type to the casual D programmer, even one that doesn't want or can't install other libraries. Even some 30-lines long D programs need multi-dimensional arrays, but they often don't need a complex scientific library too (example of a problem: in the preconditions of my functions that take an array of arrays I always have to test the input is not jagged and it is a rectangular matrix. Such test is not necessary for a multi-dimensional array that is never jagged). Putting the bare bones multi-dimensional array type in Phobos allows people to use them with zero other installs. This multi-dimensional Phobos array type doesn't even need to contain code to invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change of shape, iteration... Everything else is in modules/packages external to Phobos. I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules. Bye, bearophile
Oct 09 2011
I got an idea for multidimentional indexing and slicing. http://d.puremagic.com/issues/show_bug.cgi?id=3D6798 I think opSlice!n (n is dimension integer typed size_t) is rarely used, therefore using that name for the enhancement is almost safe for backward compatibility. Kenji Hara 2011/10/10 Denis Shelomovskij <verylonglogin.reg gmail.com>:09.10.2011 21:29, bearophile =D0=C9=DB=C5=D4:yKenji Hara has just copied in GIT a large patch by Don: https://github.com/9rnsr/dmd/branches/opDollar It allows user defined collections usable with this syntax: x[$-2, y[$-6, $-9], $-2] See: http://d.puremagic.com/issues/show_bug.cgi?id=3D3474 But a syntax like this too is quite important for a user-defined matrix type to be used in numerical code: m[i..j, k..w] Similar slicing and dicing of arrays is used all the time in NumPy (arra=helibrary for Python) code. This is a comment by Don about it:(4) I have NOT implemented $ inside opSlice(), opSliceAssign(). It could be done, but I believe those language features need work. They don't permit multi-dimensional slicing. I think they should be removed, and t=rdfunctionality folded into opIndex.Isn't it better to improve opSlice() instead of deprecating it? I don't remember if people appreciate the idea of a stride (like the thi=ystep argument of std.range.iota), this too is used in scientific array-oriented code: m[i..j:2, k..w:3] --------------- In Python code there is often the need for a good multi-dimensional arra=atype, even in not-scientific code. In the Python standard library there is a array module, but it's almost =yjoke, it's 1D, and it's not used much: http://docs.python.org/library/array.html Python programmers use the multi-dimensional array of NumPy, it's widely used around the world, it's used by SciPy too (a scientific programming library). The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible arra=andlibraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ )=),numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html=idetells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want. I think that's a good tradeoff between the opposed needs of: - Keeping Phobos of reasonable size (to not increase too much the burden of its management, to not slow down too much its development); - Avoiding the risk of incompatible multi-dimensional array types. Most code out there is able to build on a common foundation. This avoids duplication (like the creation of Python numarray and Numeric), allows a better focusing of efforts and speeds up the development of a language-w=tstandard for such arrays; - Offer a nD array type to the casual D programmer, even one that doesn'=swant or can't install other libraries. Even some 30-lines long D program=sneed multi-dimensional arrays, but they often don't need a complex scientific library too (example of a problem: in the preconditions of my functions that take an array of arrays I always have to test the input i=r anot jagged and it is a rectangular matrix. Such test is not necessary fo=romulti-dimensional array that is never jagged). Putting the bare bones multi-dimensional array type in Phobos allows people to use them with ze=deother installs. This multi-dimensional Phobos array type doesn't even need to contain co=ofto invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change =ngshape, iteration... Everything else is in modules/packages external to Phobos. I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules. Bye, bearophileSuddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of addi=it into Phobos, but just had no time for review of my code. Draft version=isin attachment. rarray.d - something like Fortran array main.d - example of using rarray module output.txt - main.d output
Oct 09 2011
kenji hara:I got an idea for multidimentional indexing and slicing. http://d.puremagic.com/issues/show_bug.cgi?id=6798Thank you, you are good. So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2)) Bye, bearophile
Oct 09 2011
2011/10/10 bearophile <bearophileHUGS lycos.com>:So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Oct 09 2011
On 10.10.2011 04:41, kenji hara wrote:2011/10/10 bearophile<bearophileHUGS lycos.com>:Personally, I think that since strided operations are so inefficient, they should remain ugly.So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Oct 09 2011
Don:Personally, I think that since strided operations are so inefficient, they should remain ugly.In Chapel (that's a high performance language full of nice ideas) the stride is present, see page 16 "Range operators": http://chapel.cray.com/tutorials/CUG2011/CUG11-2-Basics.pdf Bye, bearophile
Oct 10 2011
Don <nospam nospam.com> wrote:On 10.10.2011 04:41, kenji hara wrote:Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].2011/10/10 bearophile<bearophileHUGS lycos.com>:Personally, I think that since strided operations are so inefficient, they should remain ugly.So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Oct 10 2011
On Mon, 10 Oct 2011 08:46:05 -0400, kennytm <kennytm gmail.com> wrote:Don <nospam nospam.com> wrote:Regarding (1), from a computer vision/graphics perspective, being able to only operate on the red channel, for example, is very nice. And I use striding all the time in my CUDA code, but that's not as applicable in D, yet. Striding tends to be most useful for working with packed data of some kind.On 10.10.2011 04:41, kenji hara wrote:Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].2011/10/10 bearophile<bearophileHUGS lycos.com>:Personally, I think that since strided operations are so inefficient, they should remain ugly.So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Oct 10 2011
"Robert Jacques" <sandford jhu.edu> wrote:On Mon, 10 Oct 2011 08:46:05 -0400, kennytm <kennytm gmail.com> wrote:Why not use a 'Color' struct?Don <nospam nospam.com> wrote:Regarding (1), from a computer vision/graphics perspective, being able to only operate on the red channel, for example, is very nice. And I use striding all the time in my CUDA code, but that's not as applicable in D, yet. Striding tends to be most useful for working with packed data of some kind.On 10.10.2011 04:41, kenji hara wrote:Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].2011/10/10 bearophile<bearophileHUGS lycos.com>:Personally, I think that since strided operations are so inefficient, they should remain ugly.So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Oct 10 2011
On Tue, 11 Oct 2011 01:57:06 -0400, kennytm <kennytm gmail.com> wrote:"Robert Jacques" <sandford jhu.edu> wrote:Not all images have the same layout, so having a single Color struct doesn't cut it. (RGB, RGBA, BGR, BGRA, ARGB, ABGR) Although, being able to say: structArr[0..$].red; would be very handy indeed. But all that is / should be is a opDispatch wrapper around a strided operation: byteArr[2..$:3]On Mon, 10 Oct 2011 08:46:05 -0400, kennytm <kennytm gmail.com> wrote:Why not use a 'Color' struct?Don <nospam nospam.com> wrote:Regarding (1), from a computer vision/graphics perspective, being able to only operate on the red channel, for example, is very nice. And I use striding all the time in my CUDA code, but that's not as applicable in D, yet. Striding tends to be most useful for working with packed data of some kind.On 10.10.2011 04:41, kenji hara wrote:Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].2011/10/10 bearophile<bearophileHUGS lycos.com>:Personally, I think that since strided operations are so inefficient, they should remain ugly.So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Oct 11 2011
I don't think strides are necessary. You can always opApply your image in a certain way. Define a private struct, which has an opApply, which iterates over every third channel of each pixel. lazy ranges are generally better, then arrays, because with them, you can avoid copying anything altogether. On Tue, Oct 11, 2011 at 6:36 PM, Robert Jacques <sandford jhu.edu> wrote:On Tue, 11 Oct 2011 01:57:06 -0400, kennytm <kennytm gmail.com> wrote:"Robert Jacques" <sandford jhu.edu> wrote:Not all images have the same layout, so having a single Color struct doesn't cut it. (RGB, RGBA, BGR, BGRA, ARGB, ABGR) Although, being able to say: structArr[0..$].red; would be very handy indeed. But all that is / should be is a opDispatch wrapper around a strided operation: byteArr[2..$:3]On Mon, 10 Oct 2011 08:46:05 -0400, kennytm <kennytm gmail.com> wrote:Why not use a 'Color' struct?Don <nospam nospam.com> wrote:Regarding (1), from a computer vision/graphics perspective, being able to only operate on the red channel, for example, is very nice. And I use striding all the time in my CUDA code, but that's not as applicable in D, yet. Striding tends to be most useful for working with packed data of some kind.On 10.10.2011 04:41, kenji hara wrote:Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].2011/10/10 bearophile<bearophileHUGS lycos.com>:Personally, I think that since strided operations are so inefficient, they should remain ugly.So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Oct 11 2011
Gor Gyolchanyan:I don't think strides are necessary. You can always opApply your image in a certain way. Define a private struct, which has an opApply, which iterates over every third channel of each pixel.See: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=146466lazy ranges are generally better, then arrays, because with them, you can avoid copying anything altogether.Strides is just a syntax. The user defined type is later allowed to perform a lazy striding. Bye, bearophile
Oct 11 2011
On 11.10.2011 17:48, Gor Gyolchanyan wrote:I don't think strides are necessary. You can always opApply your image in a certain way. Define a private struct, which has an opApply, which iterates over every third channel of each pixel. lazy ranges are generally better, then arrays, because with them, you can avoid copying anything altogether.Actually, a strided slice to a multidimensional array can easily be implemented as a view to the original data without any copy action, just like any builtin array slice is a view to the data. Independent of that: of course strides are not "necessary", but they are an extremely convenient syntax for expressing numerical array operations.
Oct 11 2011
On Tue, 11 Oct 2011 11:48:54 -0400, Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> wrote:I don't think strides are necessary. You can always opApply your image in a certain way. Define a private struct, which has an opApply, which iterates over every third channel of each pixel. lazy ranges are generally better, then arrays, because with them, you can avoid copying anything altogether.First, slicing / striding doesn't imply copying. Second, striding allows you to mutate the original data, which many lazy ranges do not. Third, there is a heavy amount of over-head to opApply, as its delegate based. Forth, declaring a custom type is way too much programmer boilerplate, even using something like Tuple!(ubyte,ubyte,ubyte,ubyte) is overly verbose. Fifth, my example was about dynamically supporting multiple data formats using the same code, something not possible with a custom struct. Sixth, we are talking about multi-dimensional arrays, not D's built-in arrays; lazy ranges don't exactly apply. Although, on the subject of Multi-dimensional ranges/iteration, has anyone else taken a look at the FLAME project? (http://z.cs.utexas.edu/wiki/flame.wiki) Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.
Oct 11 2011
Robert Jacques:[... several things] Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.OK, a nice striding syntax for user defined types will be a good thing to have in D2. I have suggested the x..y:y syntax. Is this usable? Is this going to cause problems? Bye, bearophile
Oct 11 2011
bearophile , dans le message (digitalmars.D:146534), a écrit :Robert Jacques:: is already used for labels. If slice types are allowed, this can cause trouble: auto a = 0 .. 10; // a slice!(int, int) type. a : 2; // is it a label or a stride ? Of course, the example here is obvious, but maybe in real code this can cause trouble. stride(x..y, z) works just as good, and does not let people think it is a very efficient langage builtin. The langage just have to allow to define iota-like types with .. Then a[x..y] becomes an opIndex() like Kenji suggested, and slices can be used anywhere, like in : multidimArray.columns(1..5);[... several things] Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.OK, a nice striding syntax for user defined types will be a good thing to have in D2. I have suggested the x..y:y syntax. Is this usable? Is this going to cause problems? Bye, bearophile
Nov 03 2011
Strided ranges could be a tertiary operator similar to x ? a : b one could also parse a .. b : c as one entity. About the efficiency issue, once again: for multi-dimensional arrays, strides have less overhead than for one-dimensional arrays: All but one dimension need run-time defined strides anyway. On 03.11.2011 13:52, Christophe wrote:bearophile , dans le message (digitalmars.D:146534), a écrit :Robert Jacques:: is already used for labels. If slice types are allowed, this can cause trouble: auto a = 0 .. 10; // a slice!(int, int) type. a : 2; // is it a label or a stride ? Of course, the example here is obvious, but maybe in real code this can cause trouble. stride(x..y, z) works just as good, and does not let people think it is a very efficient langage builtin. The langage just have to allow to define iota-like types with .. Then a[x..y] becomes an opIndex() like Kenji suggested, and slices can be used anywhere, like in : multidimArray.columns(1..5);[... several things] Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.OK, a nice striding syntax for user defined types will be a good thing to have in D2. I have suggested the x..y:y syntax. Is this usable? Is this going to cause problems? Bye, bearophile
Nov 05 2011
On 10.10.2011 07:34, Don wrote:On 10.10.2011 04:41, kenji hara wrote:Inefficient?! For multidimensional arrays, all but one dimension have non-trivial, runtime-dependent strides anyway. Anyway, the expressiveness gained by strides is significant and the effort to allow it in the language is minimal.2011/10/10 bearophile<bearophileHUGS lycos.com>:Personally, I think that since strided operations are so inefficient, they should remain ugly.So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Oct 10 2011
On 10.10.2011 20:03, Norbert Nemec wrote:On 10.10.2011 07:34, Don wrote:The problem is with built-in arrays.On 10.10.2011 04:41, kenji hara wrote:Inefficient?! For multidimensional arrays, all but one dimension have non-trivial, runtime-dependent strides anyway.2011/10/10 bearophile<bearophileHUGS lycos.com>:Personally, I think that since strided operations are so inefficient, they should remain ugly.So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji HaraAnyway, the expressiveness gained by strides is significant and the effort to allow it in the language is minimal.That's true, IF you are happy to have it as a feature which can only be used in user-defined types. If you want it apply to built-in arrays as well, the effort is immense.
Oct 11 2011
Don:That's true, IF you are happy to have it as a feature which can only be used in user-defined types. If you want it apply to built-in arrays as well, the effort is immense.This is not wonderful, but for me it's acceptable. Having that syntax support only for user defined types is quite better than having to use a syntax like stride(...) suggested by someone else. ------------- For a simple but nice usage example of strides in Chapel language see pages 14-33 here: http://chapel.cray.com/tutorials/SC10/M10-8-Samples.pdf Strided grid description var cell_centers = [1..7 by 2, 1..5 by 2]; var vertical_edges = [0..8 by 2, 1..5 by 2]; var horizontal_edges = [1..7 by 2, 0..6 by 2]; var vertices = [0..8 by 2, 0..6 by 2]; (See the slides for more info.) Bye, bearophile
Oct 11 2011
On 11.10.2011 09:05, Don wrote:On 10.10.2011 20:03, Norbert Nemec wrote:Sure. For one-dimensional arrays, the overhead would be significant and the use would be limited to exotic cases.Inefficient?! For multidimensional arrays, all but one dimension have non-trivial, runtime-dependent strides anyway.The problem is with built-in arrays.Indeed, I am interested in this feature purely for defining multidimensional arrays for use in numerics.Anyway, the expressiveness gained by strides is significant and the effort to allow it in the language is minimal.That's true, IF you are happy to have it as a feature which can only be used in user-defined types. If you want it apply to built-in arrays as well, the effort is immense.
Oct 11 2011
On 10.10.2011 03:18, bearophile wrote:kenji hara:Nice! This seems like the way to go.I got an idea for multidimentional indexing and slicing. http://d.puremagic.com/issues/show_bug.cgi?id=6798Thank you, you are good. So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
Oct 09 2011
kenji hara , dans le message (digitalmars.D:146395), a écrit :I got an idea for multidimentional indexing and slicing. http://d.puremagic.com/issues/show_bug.cgi?id=6798 I think opSlice!n (n is dimension integer typed size_t) is rarely used, therefore using that name for the enhancement is almost safe for backward compatibility.Another option, that I must have posted somewhere else in the past: T..T is a type similar to a Tuple with two values, and represents a range going from the first element to the second element excluded. Operations are defined for T..T just like they are for arrays: ref T front(size_t..size_t a) { assert(a[0] < a[1]; return a[0]; } void popFront(size_t..size_t a) { assert(a[0] < a[1]_; ++a[0]; } bool empty(size_t..size_t a) { return a[0] == a[1]; } etc... Any template that takes a range can take a slice, since front, popFront and empty are defined. map!"a*2"(1..10); // instead of map!"a*2"(iota(1, 10)); foreach (a, x..y) is a normal iteration over a range. a[1..2, 3] calls A.opIndex(int..int, int). etc...
Nov 03 2011
Posted pull request. https://github.com/D-Programming-Language/dmd/pull/443 This patch does not break existing codes (at least dmd test suite and phobos building). I have not added stride syntax (e.g. a[0..$:2]), but I think that adding it is not so difficult. Kenji Hara 2011/10/10 kenji hara <k.hara.pg gmail.com>:I got an idea for multidimentional indexing and slicing. http://d.puremagic.com/issues/show_bug.cgi?id=3D6798 I think opSlice!n (n is dimension integer typed size_t) is rarely used, therefore using that name for the enhancement is almost safe for backward compatibility. Kenji Hara 2011/10/10 Denis Shelomovskij <verylonglogin.reg gmail.com>:ay09.10.2011 21:29, bearophile =D0=C9=DB=C5=D4:Kenji Hara has just copied in GIT a large patch by Don: https://github.com/9rnsr/dmd/branches/opDollar It allows user defined collections usable with this syntax: x[$-2, y[$-6, $-9], $-2] See: http://d.puremagic.com/issues/show_bug.cgi?id=3D3474 But a syntax like this too is quite important for a user-defined matrix type to be used in numerical code: m[i..j, k..w] Similar slicing and dicing of arrays is used all the time in NumPy (arr=ylibrary for Python) code. This is a comment by Don about it:(4) I have NOT implemented $ inside opSlice(), opSliceAssign(). It could be done, but I believe those language features need work. The=thedon't permit multi-dimensional slicing. I think they should be removed, and =irdfunctionality folded into opIndex.Isn't it better to improve opSlice() instead of deprecating it? I don't remember if people appreciate the idea of a stride (like the th=aystep argument of std.range.iota), this too is used in scientific array-oriented code: m[i..j:2, k..w:3] --------------- In Python code there is often the need for a good multi-dimensional arr=atype, even in not-scientific code. In the Python standard library there is a array module, but it's almost=yjoke, it's 1D, and it's not used much: http://docs.python.org/library/array.html Python programmers use the multi-dimensional array of NumPy, it's widel=ayused around the world, it's used by SciPy too (a scientific programming library). The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible arr=) andlibraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ =l),numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.htm=ntells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want. I think that's a good tradeoff between the opposed needs of: - Keeping Phobos of reasonable size (to not increase too much the burde=aof its management, to not slow down too much its development); - Avoiding the risk of incompatible multi-dimensional array types. Most code out there is able to build on a common foundation. This avoids duplication (like the creation of Python numarray and Numeric), allows =widebetter focusing of efforts and speeds up the development of a language-='tstandard for such arrays; - Offer a nD array type to the casual D programmer, even one that doesn=mswant or can't install other libraries. Even some 30-lines long D progra=yneed multi-dimensional arrays, but they often don't need a complex scientific library too (example of a problem: in the preconditions of m=isfunctions that take an array of arrays I always have to test the input =or anot jagged and it is a rectangular matrix. Such test is not necessary f=eromulti-dimensional array that is never jagged). Putting the bare bones multi-dimensional array type in Phobos allows people to use them with z=odeother installs. This multi-dimensional Phobos array type doesn't even need to contain c=ofto invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change=ingshape, iteration... Everything else is in modules/packages external to Phobos. I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules. Bye, bearophileSuddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of add=n isit into Phobos, but just had no time for review of my code. Draft versio=in attachment. rarray.d - something like Fortran array main.d - example of using rarray module output.txt - main.d output
Oct 10 2011
Kenji Hara:Posted pull request. https://github.com/D-Programming-Language/dmd/pull/443Things are getting interesting. Thank you. Such syntax is the building material for a good enough user-defined dense nD array. I presume the memory representation of such arrays (possibly in Phobos) will be tiled, to help CPU caches.This patch does not break existing codes (at least dmd test suite and phobos building).Given how important multi-dimensional slicing it, a bit of breakage is acceptable if it allows to make the situation simpler & cleaner. People that will implement a nD array years from now will thank you :-)I have not added stride syntax (e.g. a[0..$:2]), but I think that adding it is not so difficult.OK. Bye, bearophile
Oct 10 2011
Suddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of adding it into Phobos, but just had no time for review of my code. Draft version is in attachment. rarray.d - something like Fortran array main.d - example of using rarray module output.txt - main.d outputMy usable draft example of multidimensional rectangular arrays (source looks working stable and usable but still isn't ready): http://deoma-cmd.ru/d/docs/src/my/rarray.html Posted now just for illustrating existed syntax abilities. IMHO, it is not too bad: adding `R[]` isn't a big overhead.
Oct 13 2011
Robert Jacques:[... several things] Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.OK, a nice striding syntax for user defined types will be a good thing to have in D2. I have suggested the x..y:y syntax. Is this usable? Is this going to cause problems? Bye, bearophile
Oct 13 2011