www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Setting a hard limit on slice size, is this possible?

reply james.p.leblanc <james.p.leblanc gmail.com> writes:
I am aware of the "capacity" concept with slices.

But, I would like to know if it is possible to set a
hard limit on a slice size.

I prefer it to error and crash instead of a doing an
extension or reallocation.

I understand my question screams of "convoluted
thinking".  But, I need to align my slice according to
certain criteria.

(Alternatively, I could use an array ... if I could
align that according to criteria known at compile
time.  Is this possible?).

I have a working solution (ugly trick, maybe is a
better description) to align my slice as desired.

But, the solution would be more robust if I could
guarantee that the slice is never moved in memory.

Any thoughts, hints, etc are welcome!

James
Aug 06 2021
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 6 August 2021 at 10:50:19 UTC, james.p.leblanc wrote:
 I am aware of the "capacity" concept with slices.

 But, I would like to know if it is possible to set a
 hard limit on a slice size.

 I prefer it to error and crash instead of a doing an
 extension or reallocation.
I don't think there's any way to make built-in slices behave like this, but you can pretty easily define your own wrapper type: ```d struct MySlice(T, size_t maxLength) { private T[] payload; invariant(payload.length <= maxLength); this(T[] slice) { payload = slice; } T opIndex(size_t i) { return payload[i]; } // etc. } ```
Aug 06 2021
parent reply james.p.leblanc <james.p.leblanc gmail.com> writes:
On Friday, 6 August 2021 at 11:58:59 UTC, Paul Backus wrote:
 struct MySlice(T, size_t maxLength)
 {
     private T[] payload;
     invariant(payload.length <= maxLength);
     this(T[] slice) { payload = slice; }
     T opIndex(size_t i) { return payload[i]; }
     // etc.
 }
``` Paul, Thanks very much for your reply. I understand only a fraction of the suggested solution. I will need to digest this a bit, and do some more background reading on templates. My initial, naive attempts to use this in a simple main() were unsuccessful. I'll keep plugging at it ... Best Regards, James
Aug 06 2021
parent reply Tejas <notrealemail gmail.com> writes:
On Friday, 6 August 2021 at 16:32:59 UTC, james.p.leblanc wrote:
 On Friday, 6 August 2021 at 11:58:59 UTC, Paul Backus wrote:
 struct MySlice(T, size_t maxLength)
 {
     private T[] payload;
     invariant(payload.length <= maxLength);
     this(T[] slice) { payload = slice; }
     T opIndex(size_t i) { return payload[i]; }
     // etc.
 }
``` Paul, Thanks very much for your reply. I understand only a fraction of the suggested solution. I will need to digest this a bit, and do some more background reading on templates. My initial, naive attempts to use this in a simple main() were unsuccessful. I'll keep plugging at it ... Best Regards, James
If you want something even simpler, maybe this could help: ```d void setCapacity(int/*or whatever type you want*/ a[], ulong capacity){ assert(capacity < 100/*whatever you want to set*/); a.length = capacity; } void main(){ int[] arr; setCapacity(arr, 50);//works :D //setCapacity(arr, 1000);// fails :( } ``` Of course, this won't work if you want to resize the capacity natively, ie, by using ```.length``` directly; then you will have to use Paul's solution. If you still have difficulties, please ping. Also, if you're completely new, please try the book "programming in D", it is a marvellous resource. link: http://ddili.org/ders/d.en/
Aug 06 2021
parent Tejas <notrealemail gmail.com> writes:
On Friday, 6 August 2021 at 17:16:28 UTC, Tejas wrote:
 On Friday, 6 August 2021 at 16:32:59 UTC, james.p.leblanc wrote:
 On Friday, 6 August 2021 at 11:58:59 UTC, Paul Backus wrote:
 [...]
``` Paul, Thanks very much for your reply. I understand only a fraction of the suggested solution. I will need to digest this a bit, and do some more background reading on templates. My initial, naive attempts to use this in a simple main() were unsuccessful. I'll keep plugging at it ... Best Regards, James
If you want something even simpler, maybe this could help: ```d void setCapacity(int[] a)// not int a[]
```
Aug 06 2021
prev sibling parent reply Tejas <notrealemail gmail.com> writes:
On Friday, 6 August 2021 at 10:50:19 UTC, james.p.leblanc wrote:
 I am aware of the "capacity" concept with slices.

 But, I would like to know if it is possible to set a
 hard limit on a slice size.

 I prefer it to error and crash instead of a doing an
 extension or reallocation.

 I understand my question screams of "convoluted
 thinking".  But, I need to align my slice according to
 certain criteria.

 (Alternatively, I could use an array ... if I could
 align that according to criteria known at compile
 time.  Is this possible?).

 I have a working solution (ugly trick, maybe is a
 better description) to align my slice as desired.

 But, the solution would be more robust if I could
 guarantee that the slice is never moved in memory.

 Any thoughts, hints, etc are welcome!

 James
Okay we were overthinking the solution. Just use a static array ```d int[your_max_length]/*or whatever type*/ var; ``` You're good to go! I almost feel stupid now lol
Aug 06 2021
parent reply james.p.leblanc <james.p.leblanc gmail.com> writes:
mes
On Friday, 6 August 2021 at 17:25:24 UTC, Tejas wrote:
 Okay we were overthinking the solution.

 Just use a static array

 ```d
 int[your_max_length]/*or whatever type*/ var;
 ```

 You're good to go!

 I almost feel stupid now lol
Hello Tejas, Kind thanks for your replies ... all are appreciated. However, do NOT feel stupid ... the motivation behind why I cannot use a standard int[your_max_length] (in other words, use a static array), is because I need to do a specified memory alignment (known at compile time) on my slice, or array. I understand that neither a slice or an array is capable to doing an arbitrary memory alignment. (But, perhaps I am wrong about this ...) I believe structs can be aligned, but need to learn more about the speicific of that. In the meantime, I have written a way to get an aligned slice from a static array. Ugly beginner code is below. While I believe this basic idea should work, I would like to guarantee that my slice does not get moved in memory (destroying the alignment). Ugly code here: ---------------------------------------------- import std.stdio; enum MAXLENGTH = 1024; enum ALIGN = 128; void main(){ // static array to allow creation of aligned slice ubyte[MAXLENGTH+ALIGN] u; writeln("u.ptr: ", u.ptr); auto u_i = cast(ulong) u.ptr; auto u_excess = u_i%ALIGN; writeln("u_excess: ", u_excess); ulong mustadd; if(u_excess !=0){ mustadd = ALIGN-u_excess; } writeln("mustadd: ", mustadd); // create aligned pointer for our needed slice auto zp = cast(double*) (u.ptr + mustadd); // create slice double[] z = zp[0 .. MAXLENGTH]; writeln("z.ptr: ", z.ptr); writeln("z.length: ", z.length); auto z_i = cast(ulong) z.ptr; auto z_excess = z_i%ALIGN; writeln("z_excess: ", z_excess); }
Aug 06 2021
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 06, 2021 at 06:02:01PM +0000, james.p.leblanc via
Digitalmars-d-learn wrote:
[...]
 However, do NOT feel stupid ... the motivation behind why
 I cannot use a standard int[your_max_length] (in other words,
 use a static array), is because I need to do a specified
 memory alignment (known at compile time) on my slice, or array.
I've never actually tried this myself, but have you looked into the `align` keyword in the D spec? I'm not 100% sure whether it can be applied to static arrays (or arrays in general) but it might be worth a look.
 I understand that neither a slice or an array is capable to doing
 an arbitrary memory alignment.  (But, perhaps I am wrong about this
 ...)
 
 I believe structs can be aligned, but need to learn more about the
 speicific of that.
One thing that might help you is if you had a struct with the appropriate align(...) declaration that wraps around your array elements. Pretty sure that would guarantee the alignment of array elements, which in turn should guarantee alignment of the array as a whole. T -- It is the quality rather than the quantity that matters. -- Lucius Annaeus Seneca
Aug 06 2021
prev sibling parent reply Tejas <notrealemail gmail.com> writes:
On Friday, 6 August 2021 at 18:02:01 UTC, james.p.leblanc wrote:
 mes
 On Friday, 6 August 2021 at 17:25:24 UTC, Tejas wrote:
 Okay we were overthinking the solution.

 Just use a static array

 ```d
 int[your_max_length]/*or whatever type*/ var;
 ```

 You're good to go!

 I almost feel stupid now lol
Hello Tejas, Kind thanks for your replies ... all are appreciated. However, do NOT feel stupid ... the motivation behind why I cannot use a standard int[your_max_length] (in other words, use a static array), is because I need to do a specified memory alignment (known at compile time) on my slice, or array. I understand that neither a slice or an array is capable to doing an arbitrary memory alignment. (But, perhaps I am wrong about this ...) I believe structs can be aligned, but need to learn more about the speicific of that. In the meantime, I have written a way to get an aligned slice from a static array. Ugly beginner code is below. While I believe this basic idea should work, I would like to guarantee that my slice does not get moved in memory (destroying the alignment). Ugly code here: ---------------------------------------------- import std.stdio; enum MAXLENGTH = 1024; enum ALIGN = 128; void main(){ // static array to allow creation of aligned slice ubyte[MAXLENGTH+ALIGN] u; writeln("u.ptr: ", u.ptr); auto u_i = cast(ulong) u.ptr; auto u_excess = u_i%ALIGN; writeln("u_excess: ", u_excess); ulong mustadd; if(u_excess !=0){ mustadd = ALIGN-u_excess; } writeln("mustadd: ", mustadd); // create aligned pointer for our needed slice auto zp = cast(double*) (u.ptr + mustadd); // create slice double[] z = zp[0 .. MAXLENGTH]; writeln("z.ptr: ", z.ptr); writeln("z.length: ", z.length); auto z_i = cast(ulong) z.ptr; auto z_excess = z_i%ALIGN; writeln("z_excess: ", z_excess); }
If you're being that specialized, don't bother with fundamental data types, the hunt for simplicity will lead you to create very complex things. As HS Teoh said, take a look at ```align```: https://dlang.org/spec/attribute.html#align Stealing Paul's answer now: ```d import std; enum your_max_length = 1000; enum your_align = 256; struct MySlice(T/*, size_t maxLength*/) { private align(your_align)T payload; //invariant(payload.length <= maxLength); //this(T[] slice) { payload = slice; } //T opIndex(size_t i) { return payload[i]; } } void main() { MySlice!(int/*, 1000*/)[your_max_length] slice; writeln(slice.sizeof); } ```
Aug 06 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 6 August 2021 at 19:03:53 UTC, Tejas wrote:
 Stealing Paul's answer now:
 ```d
 import std;

 enum your_max_length = 1000;
 enum your_align = 256;
 struct MySlice(T/*, size_t maxLength*/)
 {
     private align(your_align)T payload;

     //invariant(payload.length <= maxLength);

     //this(T[] slice) { payload = slice; }

     //T opIndex(size_t i) { return payload[i]; }
 }
 void main()
 {
    MySlice!(int/*, 1000*/)[your_max_length] slice;
     writeln(slice.sizeof);
 }
 ```
You can actually pass the alignment as a parameter too: ```d struct Aligned(T, size_t alignment) if (alignment >= T.alignof) { align(alignment) T payload; alias payload this; } void main() { Aligned!(int, 16)[4] x; assert(x.alignof == 16); assert(x.sizeof == 64); Aligned!(int[4], 16) y; assert(y.alignof == 16); assert(y.sizeof == 16); } ```
Aug 06 2021
parent reply Tejas <notrealemail gmail.com> writes:
On Friday, 6 August 2021 at 22:15:00 UTC, Paul Backus wrote:
 On Friday, 6 August 2021 at 19:03:53 UTC, Tejas wrote:
 Stealing Paul's answer now:
 ```d
 import std;

 enum your_max_length = 1000;
 enum your_align = 256;
 struct MySlice(T/*, size_t maxLength*/)
 {
     private align(your_align)T payload;

     //invariant(payload.length <= maxLength);

     //this(T[] slice) { payload = slice; }

     //T opIndex(size_t i) { return payload[i]; }
 }
 void main()
 {
    MySlice!(int/*, 1000*/)[your_max_length] slice;
     writeln(slice.sizeof);
 }
 ```
You can actually pass the alignment as a parameter too: ```d struct Aligned(T, size_t alignment) if (alignment >= T.alignof) { align(alignment) T payload; alias payload this; } void main() { Aligned!(int, 16)[4] x; assert(x.alignof == 16); assert(x.sizeof == 64); Aligned!(int[4], 16) y; assert(y.alignof == 16); assert(y.sizeof == 16); } ```
Is the second example really correct? if the alignment of ```int[4]``` really is 16, then why isn't the size of it 64 as well? It seems that the alignment constraint is not being satisfied _within_ the array, And if it really is correct, then it seems once again that static arrays are the answer after all: ```d align(your_alignment) int[your_length] array; ``` No need for structs \\('_')/
Aug 07 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 7 August 2021 at 07:32:04 UTC, Tejas wrote:
 And if it really is correct, then it seems once again that 
 static arrays are the answer after all:

 ```d
 align(your_alignment) int[your_length] array;
 ```
 No need for structs \\('_')/
The main advantage of the struct is that you can heap-allocate it: ```d auto array = new Aligned!(int[4], 16); ```
Aug 07 2021
parent reply james.p.leblanc <james.p.leblanc gmail.com> writes:
On Saturday, 7 August 2021 at 12:08:00 UTC, Paul Backus wrote:
 On Saturday, 7 August 2021 at 07:32:04 UTC, Tejas wrote:
 And if it really is correct, then it seems once again that 
 static arrays are the answer after all:

 ```d
 align(your_alignment) int[your_length] array;
 ```
 No need for structs \\('_')/
The main advantage of the struct is that you can heap-allocate it: ```d auto array = new Aligned!(int[4], 16); ```
**First, thanks all for helping with this question!** The simple desire to arbitrarily align an array is certainly looking non-trivial. Below is a simple program of both the suggested "struct" and "align array" solutions. Unfortunately, neither is guaranteed to place the array with the desired alignnment. import std.stdio; enum ALIGNMENT=64; enum LENGTH=10; struct Aligned(T, size_t alignment) if (alignment >= T.alignof) { align(alignment) T payload; alias payload this; } void main() { writeln("ALIGNMENT: ", ALIGNMENT, ",\t\tLENGTH: ", LENGTH); int[23] junk; align(ALIGNMENT) int[LENGTH] z; writeln("\nusing 'align(ALIGNMENT) int[LENGTH] z;'"); writeln("\ncast(ulong) z.ptr%ALIGNMENT: ", cast(ulong) z.ptr%ALIGNMENT); writeln("int.alignof: ", int.alignof, ",\tint.sizeof: ", int.sizeof); writeln("&z[0]", &z[0]); writeln("&z[1]", &z[1]); writeln("&z[2]", &z[2]); writeln("\nusing: 'Aligned!(int, ALIGNMENT)[LENGTH] x;'"); Aligned!(int, ALIGNMENT)[LENGTH] x; writeln("x.sizeof: ", x.sizeof, "\t\tx.alignof: ", x.alignof); writeln("x: ", x); writeln("\nx.ptr: ", x.ptr); writeln("&x[0]", &x[0]); writeln("&x[1]", &x[1]); writeln("&x[2]", &x[2]); writeln("\ncast(ulong) x.ptr%ALIGNMENT: ", cast(ulong) x.ptr%ALIGNMENT); } --------------------------------------------------------------- and here is a sample output of a run: ALIGNMENT: 64, LENGTH: 10 using 'align(ALIGNMENT) int[LENGTH] z;' cast(ulong) z.ptr%ALIGNMENT: ***32*** int.alignof: 4, int.sizeof: 4 &z[0]7FFFF8D05B60 &z[1]7FFFF8D05B64 &z[2]7FFFF8D05B68 using: 'Aligned!(int, ALIGNMENT)[LENGTH] x;' x.sizeof: 640 x.alignof: 64 x: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] x.ptr: 7FFFF8D05B90 &x[0]7FFFF8D05B90 &x[1]7FFFF8D05BD0 &x[2]7FFFF8D05C10 cast(ulong) x.ptr%ALIGNMENT: ***16*** --------------------------------------------------------------- Notice, that neither attempt yields an array starting address as zero modulo the alignment value ... Also, a bit weird is that the "align array" solution yields expected spacing between array elements, but the "struct" solution has wider separations ... ------------------------------------------------------ **Again, I am very appreciative of your replies and help with this.** I have learned quite a bit from this discussion. However, I remain a bit stumped by all of this.... Any ideas out there? Best Regards, James
Aug 07 2021
next sibling parent reply Tejas <notrealemail gmail.com> writes:
On Saturday, 7 August 2021 at 13:36:52 UTC, james.p.leblanc wrote:
 On Saturday, 7 August 2021 at 12:08:00 UTC, Paul Backus wrote:
 [...]
**First, thanks all for helping with this question!** The simple desire to arbitrarily align an array is certainly looking non-trivial. Below is a simple program of both the suggested "struct" and "align array" solutions. Unfortunately, neither is guaranteed to place the array with the desired alignnment. [...]
Umm, the ```align array``` solution is flat out wrong, please ignore it. Most likely a bug in the compiler. Also, why will the address of the first element of the array ```modulo``` alignment be 0? The address of the array has absolutely nothing to do with the alignment. You say that the ```align array``` solution has the expected spacing, but it is default aligned, totally ignoring your 64 byte requirement. Don't use ```x.PTR%ALIGNMENT``` Use ```(&x[1]-&[2])%ALIGNMENT``` You will get ALIGNMENT*(2 -1)(since you took difference of 2nd and 1st elem)
Aug 07 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 7 August 2021 at 14:34:49 UTC, Tejas wrote:
 Umm, the ```align array``` solution is flat out wrong,  please 
 ignore it. Most likely a bug in the compiler.

 Also, why will the address of the first element of the array 
 ```modulo``` alignment be 0? The address of the array has 
 absolutely nothing to do with the alignment.

 You say that the ```align array``` solution has the expected 
 spacing,  but it is default aligned, totally ignoring your 64 
 byte requirement.

 Don't use ```x.PTR%ALIGNMENT```
 Use ```(&x[1]-&[2])%ALIGNMENT```

 You will get ALIGNMENT*(2 -1)(since you took difference of 2nd 
 and 1st elem)
For the array as a whole to be aligned, not only must the spacing between the elements respect the alignment, but starting address of the array itself must be a multiple of the alignment. So it is correct to check that `x.ptr%ALIGNMENT == 0`. The issue with `align` attributes being ignored for stack variables is apparently a known bug, first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16098 The issue with `align` attributes being ignored by `new` is also a known bug, and was also first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16508
Aug 07 2021
parent reply Tejas <notrealemail gmail.com> writes:
On Saturday, 7 August 2021 at 15:21:01 UTC, Paul Backus wrote:
 On Saturday, 7 August 2021 at 14:34:49 UTC, Tejas wrote:
 [...]
For the array as a whole to be aligned, not only must the spacing between the elements respect the alignment, but starting address of the array itself must be a multiple of the alignment. So it is correct to check that `x.ptr%ALIGNMENT == 0`. The issue with `align` attributes being ignored for stack variables is apparently a known bug, first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16098 The issue with `align` attributes being ignored by `new` is also a known bug, and was also first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16508
Oh wow, and here I thought I was being smart :( So, how can we work around this without assembly language magic? I'm illiterate at assembly.
Aug 07 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 7 August 2021 at 15:41:24 UTC, Tejas wrote:
 On Saturday, 7 August 2021 at 15:21:01 UTC, Paul Backus wrote:
 The issue with `align` attributes being ignored for stack 
 variables is apparently a known bug, first reported in 2016: 
 https://issues.dlang.org/show_bug.cgi?id=16098

 The issue with `align` attributes being ignored by `new` is 
 also a known bug, and was also first reported in 2016: 
 https://issues.dlang.org/show_bug.cgi?id=16508
Oh wow, and here I thought I was being smart :( So, how can we work around this without assembly language magic? I'm illiterate at assembly.
For stack allocations, you can use the workaround in [Vladimir Panteleev's comment][1] (ignoring the ASM part, which is unrelated to alignment). For heap allocations, I guess the easiest way would be to use [`AlignedMallocator`][2] from `std.experimental.allocator`. [1]: https://issues.dlang.org/show_bug.cgi?id=16098#c3 [2]: https://phobos.dpldocs.info/std.experimental.allocator.mallocator.AlignedMallocator.html
Aug 07 2021
parent reply Tejas <notrealemail gmail.com> writes:
On Saturday, 7 August 2021 at 19:07:04 UTC, Paul Backus wrote:
 On Saturday, 7 August 2021 at 15:41:24 UTC, Tejas wrote:
 On Saturday, 7 August 2021 at 15:21:01 UTC, Paul Backus wrote:
 [...]
Oh wow, and here I thought I was being smart :( So, how can we work around this without assembly language magic? I'm illiterate at assembly.
For stack allocations, you can use the workaround in [Vladimir Panteleev's comment][1] (ignoring the ASM part, which is unrelated to alignment). For heap allocations, I guess the easiest way would be to use [`AlignedMallocator`][2] from `std.experimental.allocator`. [1]: https://issues.dlang.org/show_bug.cgi?id=16098#c3 [2]: https://phobos.dpldocs.info/std.experimental.allocator.mallocator.AlignedMallocator.html
Cool... thanks
Aug 07 2021
parent james.p.leblanc <james.p.leblanc gmail.com> writes:
On Sunday, 8 August 2021 at 02:00:26 UTC, Tejas wrote:
 On Saturday, 7 August 2021 at 19:07:04 UTC, Paul Backus wrote:
 On Saturday, 7 August 2021 at 15:41:24 UTC, Tejas wrote:
 On Saturday, 7 August 2021 at 15:21:01 UTC, Paul Backus wrote:
 [...]
Oh wow, and here I thought I was being smart :( So, how can we work around this without assembly language magic? I'm illiterate at assembly.
For stack allocations, you can use the workaround in [Vladimir Panteleev's comment][1] (ignoring the ASM part, which is unrelated to alignment). For heap allocations, I guess the easiest way would be to use [`AlignedMallocator`][2] from `std.experimental.allocator`. [1]: https://issues.dlang.org/show_bug.cgi?id=16098#c3 [2]: https://phobos.dpldocs.info/std.experimental.allocator.mallocator.AlignedMallocator.html
Cool... thanks
Thanks to everyone for all of the great discussion and hints on this topic! I have learned quite much, and have a gained an understanding on how to use D effectively. Fantastic forum here on a great language. Best Regards, James
Aug 07 2021
prev sibling parent Tejas <notrealemail gmail.com> writes:
On Saturday, 7 August 2021 at 13:36:52 UTC, james.p.leblanc wrote:
 On Saturday, 7 August 2021 at 12:08:00 UTC, Paul Backus wrote:
 [...]
**First, thanks all for helping with this question!** The simple desire to arbitrarily align an array is certainly looking non-trivial. Below is a simple program of both the suggested "struct" and "align array" solutions. Unfortunately, neither is guaranteed to place the array with the desired alignnment. [...]
If you don't believe me just subtract the adresses yourself: x.ptr: 7FFFF8D05B90 &x[0]7FFFF8D05B90 &x[1]7FFFF8D05BD0 &x[2]7FFFF8D05C10 &x[1] 7FFFF8D05BD0 -&x[0] -7FFFF8D05B90 _____________________ 000000000040 0X40 = 64 in base 10 Your alignment requirement was 64 bytes
Aug 07 2021