www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - I had a bad time with slice-in-struct array operation

reply Random D user <no email.com> writes:
I wanted to make a 2D array like structure and support D slice 
like operations,
but I had surprisingly bad experience.

I quickly copy pasted the example from the docs: 
https://dlang.org/spec/operatoroverloading.html#array-ops

It's something like this:
struct Array2D(E)
{
     E[] impl;
     int stride;
     int width, height;

     this(int width, int height, E[] initialData = [])
     ref E opIndex(int i, int j)
     Array2D opIndex(int[2] r1, int[2] r2)
     auto opIndex(int[2] r1, int j)
     auto opIndex(int i, int[2] r2)
     int[2] opSlice(size_t dim)(int start, int end)
      property int opDollar(size_t dim : 0)()
      property int opDollar(size_t dim : 1)()
}

So basic indexing works fine:
Array2D!int foo(4, 4);
foo[0, 1] = foo[2, 3];

But array copy and setting/clearing doesn't:
int[] bar = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 
15 ];
foo[] = bar[];

And I get this very cryptic message:
(6): Error: template `example.Array2D!int.Array2D.opSlice` cannot 
deduce function from argument types `!()()`, candidates are:
(51):        `example.Array2D!int.Array2D.opSlice(ulong dim)(int 
start, int end) if (dim >= 0 && (dim < 2))`

1. WTF `!()()` and I haven't even called anything with opSlice 
i.e. `a .. b`?

Anyway, it doesn't overload [] with opIndex(), so fine, I add 
that.
T[] opIndex() { return impl; }

Now I get:
foo[] = bar[]; // or foo[] = bar;
Error: `foo[]` is not an lvalue and cannot be modified

Array copying docs say:
When the slice operator appears as the left-hand side of an 
assignment expression, it means that the contents of the array 
are the target of the assignment rather than a reference to the 
array. Array copying happens when the left-hand side is a slice, 
and the right-hand side is an array of or pointer to the same 
type.

2.WTF I do have slice operator left of assignment.
So I guess [] is just wonky named getter (and not an operator) 
for a slice object and that receives the = so it's trying to 
overwrite/set the slice object itself.

Next I added a ref to the E[] opIndex():
ref E[] opIndex() { return impl; }

Now foo[] = bar[] works as expected, but then I tried
foo[] = 0;
and that fails:
Error: cannot implicitly convert expression `0` of type `int` to 
`int[]`

3. WTF. Didn't I just get reference directly to the slice and 
array copy works, why doesn't array setting?

The ugly foo[][] = 0 does work, but it's so ugly/confusing that 
I'd rather just use a normal function.

So I added:
ref E[] opIndexAssign(E value) { impl[] = value; return impl; }

And now foo[] = 0; works, but foo[0, 1] = foo[2, 3] doesn't.

I get:
Error: function `example.Array2D!int.Array2D.opIndexAssign(int 
f)` is not callable using argument types `(int, int, int)`
expected 1 argument(s), not 3

4. WTF. So basically adding opIndexAssign(E value) disabled ref E 
opIndex(int i, int j). Shouldn't it consider both?

I'm surprised how convoluted this is. Is this really the way it's 
supposed to work or is there a bug or something?


So what is the best/clear/concise/D way to do these for a custom 
type?

I was planning for:
foo[] = bar; // Full copy
foo[] = 0; // Full clear
foo[0 .. 5, 1] = bar[ 0 .. 5]; // Row/Col copy
foo[1, 0 .. 5] = 0; // Row/Col clear
foo[0 .. 5, 2 .. 4] = bar[ 1 .. 6, 0 .. 2 ]; // Box copy
foo[0 .. 5, 2 .. 4] = 0; // Box clear

Anyway, this is not a huge deal breaker for me, I was just 
surprised and felt like I'm missing something.
I suppose I can manually define every case one by one and not 
return/use any references etc.
or use alias this to forward to impl[] (which I don't want to do 
since I don't want to change .length for example)
or just use normal functions and be done with it.

And it's not actually just a regular array I'm making, so that's 
why it will be mostly custom code, except the very basics.
May 04 2019
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 4 May 2019 at 15:18:58 UTC, Random D user wrote:

 I wanted to make a 2D array like structure and support D slice 
 like operations,
 but I had surprisingly bad experience.

 I quickly copy pasted the example from the docs: 
 https://dlang.org/spec/operatoroverloading.html#array-ops

 It's something like this:
 struct Array2D(E)
 {
     E[] impl;
     int stride;
     int width, height;

     this(int width, int height, E[] initialData = [])
     ref E opIndex(int i, int j)
     Array2D opIndex(int[2] r1, int[2] r2)
     auto opIndex(int[2] r1, int j)
     auto opIndex(int i, int[2] r2)
     int[2] opSlice(size_t dim)(int start, int end)
      property int opDollar(size_t dim : 0)()
      property int opDollar(size_t dim : 1)()
 }

 So basic indexing works fine:
 Array2D!int foo(4, 4);
 foo[0, 1] = foo[2, 3];

 But array copy and setting/clearing doesn't:
 int[] bar = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 
 15 ];
 foo[] = bar[];

 And I get this very cryptic message:
 (6): Error: template `example.Array2D!int.Array2D.opSlice` 
 cannot deduce function from argument types `!()()`, candidates 
 are:
 (51):        `example.Array2D!int.Array2D.opSlice(ulong 
 dim)(int start, int end) if (dim >= 0 && (dim < 2))`

 1. WTF `!()()` and I haven't even called anything with opSlice 
 i.e. `a .. b`?

 Anyway, it doesn't overload [] with opIndex(), so fine, I add 
 that.
 T[] opIndex() { return impl; }

 Now I get:
 foo[] = bar[]; // or foo[] = bar;
 Error: `foo[]` is not an lvalue and cannot be modified

 Array copying docs say:
 When the slice operator appears as the left-hand side of an 
 assignment expression, it means that the contents of the array 
 are the target of the assignment rather than a reference to the 
 array. Array copying happens when the left-hand side is a 
 slice, and the right-hand side is an array of or pointer to the 
 same type.

 2.WTF I do have slice operator left of assignment.
 So I guess [] is just wonky named getter (and not an operator) 
 for a slice object and that receives the = so it's trying to 
 overwrite/set the slice object itself.

 Next I added a ref to the E[] opIndex():
 ref E[] opIndex() { return impl; }

 Now foo[] = bar[] works as expected, but then I tried
 foo[] = 0;
 and that fails:
 Error: cannot implicitly convert expression `0` of type `int` 
 to `int[]`

 3. WTF. Didn't I just get reference directly to the slice and 
 array copy works, why doesn't array setting?

 The ugly foo[][] = 0 does work, but it's so ugly/confusing that 
 I'd rather just use a normal function.

 So I added:
 ref E[] opIndexAssign(E value) { impl[] = value; return impl; }

 And now foo[] = 0; works, but foo[0, 1] = foo[2, 3] doesn't.

 I get:
 Error: function `example.Array2D!int.Array2D.opIndexAssign(int 
 f)` is not callable using argument types `(int, int, int)`
 expected 1 argument(s), not 3

 4. WTF. So basically adding opIndexAssign(E value) disabled ref 
 E opIndex(int i, int j). Shouldn't it consider both?

 I'm surprised how convoluted this is. Is this really the way 
 it's supposed to work or is there a bug or something?


 So what is the best/clear/concise/D way to do these for a 
 custom type?

 I was planning for:
 foo[] = bar; // Full copy
 foo[] = 0; // Full clear
 foo[0 .. 5, 1] = bar[ 0 .. 5]; // Row/Col copy
 foo[1, 0 .. 5] = 0; // Row/Col clear
 foo[0 .. 5, 2 .. 4] = bar[ 1 .. 6, 0 .. 2 ]; // Box copy
 foo[0 .. 5, 2 .. 4] = 0; // Box clear

 Anyway, this is not a huge deal breaker for me, I was just 
 surprised and felt like I'm missing something.
 I suppose I can manually define every case one by one and not 
 return/use any references etc.
 or use alias this to forward to impl[] (which I don't want to 
 do since I don't want to change .length for example)
 or just use normal functions and be done with it.

 And it's not actually just a regular array I'm making, so 
 that's why it will be mostly custom code, except the very 
 basics.


The de facto multi dimensional array type in D is mir's ndslice

https://github.com/libmir/mir-algorithm/blob/master/source/mir/ndslice/slice.d#L479
May 04 2019
parent Random D user <no email.com> writes:
On Saturday, 4 May 2019 at 15:36:51 UTC, Nicholas Wilson wrote:

 On Saturday, 4 May 2019 at 15:18:58 UTC, Random D user wrote:

 I wanted to make a 2D array like structure and support D slice 
 like operations,
 but I had surprisingly bad experience.


 The de facto multi dimensional array type in D is mir's ndslice

 https://github.com/libmir/mir-algorithm/blob/master/source/mir/ndslice/slice.d#L479


Thanks. I'll take a look.
May 09 2019
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 4 May 2019 at 15:18:58 UTC, Random D user wrote:

 But array copy and setting/clearing doesn't:
 int[] bar = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 
 15 ];
 foo[] = bar[];

 And I get this very cryptic message:
 (6): Error: template `example.Array2D!int.Array2D.opSlice` 
 cannot deduce function from argument types `!()()`, candidates 
 are:
 (51):        `example.Array2D!int.Array2D.opSlice(ulong 
 dim)(int start, int end) if (dim >= 0 && (dim < 2))`

 1. WTF `!()()` and I haven't even called anything with opSlice




This comes from some old history: arr[] used to call opSlice(), 
and now it is preferred to implement opIndex() instead, but the 
compiler still supports the old zero-arg opSlice() too.

Since opIndex() didn't work, it moved to trying the older 
opSlice(), and that didn't work leading it to give up and issue 
the error.

But yeah, the error should probably mention the newer function 
name instead of the fallback it is failing on...


 Next I added a ref to the E[] opIndex():
 ref E[] opIndex() { return impl; }


I would avoid ref as much as you can, for one because it 
conflates getting and assigning (as you saw), but also because it 
leaks your internal implementation detail to the user; if you 
didn't use an array internally, that api would break.


 So I added:
 ref E[] opIndexAssign(E value) { impl[] = value; return impl; }

 And now foo[] = 0; works, but foo[0, 1] = foo[2, 3] doesn't.
 *snip*
 4. WTF. So basically adding opIndexAssign(E value) disabled ref 
 E opIndex(int i, int j). Shouldn't it consider both?


Once you implement an opIndexAssign - any opIndexAssign - all 
uses of `a[...] = c` will go through it instead of normal opIndex.

Generally speaking, opIndex is for getting, opIndexAssign is for 
setting. Sometimes setting can be done via a getter function 
(like if ref), but they are mostly two different functions and 
you should implement them both if you want read/write access.

This is especially important if your underlying data isn't 
actually in a literal array.


 foo[] = bar; // Full copy


implement:

opIndexAssign(typeof(this) rhs);

In opIndexAssign, generally, the arguments are 
(value_on_right_hand_side, indexes...)

Since there was no index given here, you don't want an index 
argument.


 foo[] = 0; // Full clear


implement:

opIndexAssign(int rhs);

Now it will take any int for the whole thing.


 foo[0 .. 5, 1] = bar[ 0 .. 5]; // Row/Col copy


This is translated to:

this.opIndexAssign(bar.opIndex(bar.opSlice!0(0, 5)), 
this.opSlice!0(0, 5), 1)

Three different functions there. On the left, we see x[...] = y, 
so we know that is opIndexAssign again. The first argument is 
what it is set to, other arguments are the slices given.

Any x .. y is translated to opSlice!dim(low, high). The !0 in 
there is because it was given as the zeroth (first) argument in 
the slice.

In this example, `foo[1 .. 2, 3 .. 4]`, it would call

opIndex( opSlice!0(1, 2), opSlice!1(3, 4) )

With the !0 and !1 indicating which position the slice was in.

The implementation of these functions would depend on just what 
your innards are... but the types might be something like this:

struct SliceHelper {
    size_t start;
    size_t end;
    int stride;
}

SliceHelper opSlice(size_t dimension)(size_t start, size_t end) {
     /* return the helper with the appropriate values */
}

now, we can implement opIndex for getting in terms of that and 
some regular items:

// get a single item at point x, y
int opIndex(size_t x, size_t y) { }

// well not necessarily an array, maybe a range for lazy 
processing, but meh you get the idea
// this gives a slice in the X dimension with a fixed Y coordinate
// e.g. foo[ 0 .. 5, 3]
int[] opIndex(SliceHelper x, size_t y) {}

// foo[0, 4 .. 6]
int[] opIndex(size_t x, SliceHelper y) {}

// and now a 2d section of it
Array2d!int opIndex(SliceHelper x, SliceHelper y) { }

// and the zero-arg version, for foo[]
// here I return this for an example, but by convention,
// this should actually return a range object that is a
// view into this container... which might be `this` but
// might not be, depending on the details of your code like
// if it has internal references or other stuff you don't want
// to leak to the outside.

typeof(this) opIndex() { return this; }


And then a similar combination of arguments for opIndexAssign
(I return void here but it is also common to return this; )

void opIndexAssign(int rhs) // this[] = rhs
void opIndexAssign(int rhs, size_t x, size_t y) // this[x, y] = 
rhs
void opIndexAssign(int rhs, size_t x, SliceHelper y) // this[x, 
y1 .. y2] = rhs


.... you get the idea. The overload for other types of rhs.

This might be dozens of functions! You can minimize that a bit by 
templating them and using internal static if, loops, etc to 
narrow it down.


void opIndexAssign(R, Idx1, Idx2)(R rhs, Idx1 x, Idx2 y) {
     // test types of Idx1+Idx2 for figuring out what to change
     // test type of R to see what to change it to
}



 foo[1, 0 .. 5] = 0; // Row/Col clear


Again, notice the x[...] = y, so we know it is opIndexAssign.


opIndexAssign(0 /* rhs */, 1, opSlice!1(0, 5))


 foo[0 .. 5, 2 .. 4] = bar[ 1 .. 6, 0 .. 2 ]; // Box copy


opIndexAssign( // opIndexAssign as a setter
    bar.opIndex(bar.opSlice!0(1, 6), bar.opSlice!1(0, 2)), // rhs 
value, notice opIndex as a getter
    opSlice!0(0, 5), opSlice!1(2, 4) /* lhs slice args */)


 foo[0 .. 5, 2 .. 4] = 0; // Box clear


foo.opIndexAssign( // setter
    0, // rhs
    foo.opSlice!0(0, 5), foo.opSlice!1(2, 4) // lhs slice args
)



 I suppose I can manually define every case one by one and not 
 return/use any references etc.



Yeah, it can be quite a lot of combinations of arguments. 
Generally, it is n^n * k, where n is the number of dimensions of 
your array and k is the number of different types you want to be 
able to assign to it. Then times two for getters and setters, 
then plus two for the empty arg ones.

For 1d with one type, it is easy: 1^1 * 1 * 2 + 2:

the 1^1 * 1 is simplified: int func(int), and *2 is Index and 
IndexAssign:

opIndex(int) and opIndexAssign(int, int)

then the plus two

opIndex() and opIndexAssign(int) for foo[] and foo[] = n.


OK, maybe actually *3 instead of *2 if you want to do

opOpIndexAssign as well, to enable like

foo[] += 4;

that's separate too.


But for 2d and 3d and more arrays, the number of functions 
explodes really fast. You will probably want to template that 
into something generic and just implement it that way.
May 04 2019
parent Random D user <no email.com> writes:
On Saturday, 4 May 2019 at 16:10:36 UTC, Adam D. Ruppe wrote:

 On Saturday, 4 May 2019 at 15:18:58 UTC, Random D user wrote:

 But array copy and setting/clearing doesn't:
 int[] bar = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 
 14, 15 ];
 foo[] = bar[];


 Generally speaking, opIndex is for getting, opIndexAssign is 
 for setting.


Thanks a lot for a very detailed answer. Sorry about the late 
reply.


 But for 2d and 3d and more arrays, the number of functions 
 explodes really fast.


Yeah, tastes like C++, but I guess I'll bite.
I value debuggability and I only have the 2D case, so I think 
templates are out.
May 09 2019