digitalmars.D.learn - Setting a hard limit on slice size, is this possible?
- james.p.leblanc (17/17) Aug 06 2021 I am aware of the "capacity" concept with slices.
- Paul Backus (13/18) Aug 06 2021 I don't think there's any way to make built-in slices behave like
- james.p.leblanc (11/19) Aug 06 2021 ```
- Tejas (8/25) Aug 06 2021 Okay we were overthinking the solution.
- james.p.leblanc (47/54) Aug 06 2021 Hello Tejas,
- H. S. Teoh (14/24) Aug 06 2021 I've never actually tried this myself, but have you looked into the
- Tejas (24/84) Aug 06 2021 If you're being that specialized, don't bother with fundamental
- Paul Backus (19/37) Aug 06 2021 You can actually pass the alignment as a parameter too:
- Tejas (11/53) Aug 07 2021 Is the second example really correct? if the alignment of
- Paul Backus (5/11) Aug 07 2021 The main advantage of the struct is that you can heap-allocate it:
- james.p.leblanc (77/90) Aug 07 2021 **First, thanks all for helping with this question!**
- Tejas (13/23) Aug 07 2021 Umm, the ```align array``` solution is flat out wrong, please
- Paul Backus (11/23) Aug 07 2021 For the array as a whole to be aligned, not only must the spacing
- Tejas (4/17) Aug 07 2021 Oh wow, and here I thought I was being smart :(
- Paul Backus (9/21) Aug 07 2021 For stack allocations, you can use the workaround in [Vladimir
- Tejas (2/18) Aug 07 2021 Cool... thanks
- james.p.leblanc (8/29) Aug 07 2021 Thanks to everyone for all of the great discussion and hints on
- Tejas (12/22) Aug 07 2021 If you don't believe me just subtract the adresses yourself:
I am aware of the "capacity" concept with slices. But, I would like to know if it is possible to set a hard limit on a slice size. I prefer it to error and crash instead of a doing an extension or reallocation. I understand my question screams of "convoluted thinking". But, I need to align my slice according to certain criteria. (Alternatively, I could use an array ... if I could align that according to criteria known at compile time. Is this possible?). I have a working solution (ugly trick, maybe is a better description) to align my slice as desired. But, the solution would be more robust if I could guarantee that the slice is never moved in memory. Any thoughts, hints, etc are welcome! James
Aug 06 2021
On Friday, 6 August 2021 at 10:50:19 UTC, james.p.leblanc wrote:I am aware of the "capacity" concept with slices. But, I would like to know if it is possible to set a hard limit on a slice size. I prefer it to error and crash instead of a doing an extension or reallocation.I don't think there's any way to make built-in slices behave like this, but you can pretty easily define your own wrapper type: ```d struct MySlice(T, size_t maxLength) { private T[] payload; invariant(payload.length <= maxLength); this(T[] slice) { payload = slice; } T opIndex(size_t i) { return payload[i]; } // etc. } ```
Aug 06 2021
On Friday, 6 August 2021 at 11:58:59 UTC, Paul Backus wrote:struct MySlice(T, size_t maxLength) { private T[] payload; invariant(payload.length <= maxLength); this(T[] slice) { payload = slice; } T opIndex(size_t i) { return payload[i]; } // etc. }``` Paul, Thanks very much for your reply. I understand only a fraction of the suggested solution. I will need to digest this a bit, and do some more background reading on templates. My initial, naive attempts to use this in a simple main() were unsuccessful. I'll keep plugging at it ... Best Regards, James
Aug 06 2021
On Friday, 6 August 2021 at 16:32:59 UTC, james.p.leblanc wrote:On Friday, 6 August 2021 at 11:58:59 UTC, Paul Backus wrote:If you want something even simpler, maybe this could help: ```d void setCapacity(int/*or whatever type you want*/ a[], ulong capacity){ assert(capacity < 100/*whatever you want to set*/); a.length = capacity; } void main(){ int[] arr; setCapacity(arr, 50);//works :D //setCapacity(arr, 1000);// fails :( } ``` Of course, this won't work if you want to resize the capacity natively, ie, by using ```.length``` directly; then you will have to use Paul's solution. If you still have difficulties, please ping. Also, if you're completely new, please try the book "programming in D", it is a marvellous resource. link: http://ddili.org/ders/d.en/struct MySlice(T, size_t maxLength) { private T[] payload; invariant(payload.length <= maxLength); this(T[] slice) { payload = slice; } T opIndex(size_t i) { return payload[i]; } // etc. }``` Paul, Thanks very much for your reply. I understand only a fraction of the suggested solution. I will need to digest this a bit, and do some more background reading on templates. My initial, naive attempts to use this in a simple main() were unsuccessful. I'll keep plugging at it ... Best Regards, James
Aug 06 2021
On Friday, 6 August 2021 at 17:16:28 UTC, Tejas wrote:On Friday, 6 August 2021 at 16:32:59 UTC, james.p.leblanc wrote:```On Friday, 6 August 2021 at 11:58:59 UTC, Paul Backus wrote:If you want something even simpler, maybe this could help: ```d void setCapacity(int[] a)// not int a[][...]``` Paul, Thanks very much for your reply. I understand only a fraction of the suggested solution. I will need to digest this a bit, and do some more background reading on templates. My initial, naive attempts to use this in a simple main() were unsuccessful. I'll keep plugging at it ... Best Regards, James
Aug 06 2021
On Friday, 6 August 2021 at 10:50:19 UTC, james.p.leblanc wrote:I am aware of the "capacity" concept with slices. But, I would like to know if it is possible to set a hard limit on a slice size. I prefer it to error and crash instead of a doing an extension or reallocation. I understand my question screams of "convoluted thinking". But, I need to align my slice according to certain criteria. (Alternatively, I could use an array ... if I could align that according to criteria known at compile time. Is this possible?). I have a working solution (ugly trick, maybe is a better description) to align my slice as desired. But, the solution would be more robust if I could guarantee that the slice is never moved in memory. Any thoughts, hints, etc are welcome! JamesOkay we were overthinking the solution. Just use a static array ```d int[your_max_length]/*or whatever type*/ var; ``` You're good to go! I almost feel stupid now lol
Aug 06 2021
mes On Friday, 6 August 2021 at 17:25:24 UTC, Tejas wrote:Okay we were overthinking the solution. Just use a static array ```d int[your_max_length]/*or whatever type*/ var; ``` You're good to go! I almost feel stupid now lolHello Tejas, Kind thanks for your replies ... all are appreciated. However, do NOT feel stupid ... the motivation behind why I cannot use a standard int[your_max_length] (in other words, use a static array), is because I need to do a specified memory alignment (known at compile time) on my slice, or array. I understand that neither a slice or an array is capable to doing an arbitrary memory alignment. (But, perhaps I am wrong about this ...) I believe structs can be aligned, but need to learn more about the speicific of that. In the meantime, I have written a way to get an aligned slice from a static array. Ugly beginner code is below. While I believe this basic idea should work, I would like to guarantee that my slice does not get moved in memory (destroying the alignment). Ugly code here: ---------------------------------------------- import std.stdio; enum MAXLENGTH = 1024; enum ALIGN = 128; void main(){ // static array to allow creation of aligned slice ubyte[MAXLENGTH+ALIGN] u; writeln("u.ptr: ", u.ptr); auto u_i = cast(ulong) u.ptr; auto u_excess = u_i%ALIGN; writeln("u_excess: ", u_excess); ulong mustadd; if(u_excess !=0){ mustadd = ALIGN-u_excess; } writeln("mustadd: ", mustadd); // create aligned pointer for our needed slice auto zp = cast(double*) (u.ptr + mustadd); // create slice double[] z = zp[0 .. MAXLENGTH]; writeln("z.ptr: ", z.ptr); writeln("z.length: ", z.length); auto z_i = cast(ulong) z.ptr; auto z_excess = z_i%ALIGN; writeln("z_excess: ", z_excess); }
Aug 06 2021
On Fri, Aug 06, 2021 at 06:02:01PM +0000, james.p.leblanc via Digitalmars-d-learn wrote: [...]However, do NOT feel stupid ... the motivation behind why I cannot use a standard int[your_max_length] (in other words, use a static array), is because I need to do a specified memory alignment (known at compile time) on my slice, or array.I've never actually tried this myself, but have you looked into the `align` keyword in the D spec? I'm not 100% sure whether it can be applied to static arrays (or arrays in general) but it might be worth a look.I understand that neither a slice or an array is capable to doing an arbitrary memory alignment. (But, perhaps I am wrong about this ...) I believe structs can be aligned, but need to learn more about the speicific of that.One thing that might help you is if you had a struct with the appropriate align(...) declaration that wraps around your array elements. Pretty sure that would guarantee the alignment of array elements, which in turn should guarantee alignment of the array as a whole. T -- It is the quality rather than the quantity that matters. -- Lucius Annaeus Seneca
Aug 06 2021
On Friday, 6 August 2021 at 18:02:01 UTC, james.p.leblanc wrote:mes On Friday, 6 August 2021 at 17:25:24 UTC, Tejas wrote:If you're being that specialized, don't bother with fundamental data types, the hunt for simplicity will lead you to create very complex things. As HS Teoh said, take a look at ```align```: https://dlang.org/spec/attribute.html#align Stealing Paul's answer now: ```d import std; enum your_max_length = 1000; enum your_align = 256; struct MySlice(T/*, size_t maxLength*/) { private align(your_align)T payload; //invariant(payload.length <= maxLength); //this(T[] slice) { payload = slice; } //T opIndex(size_t i) { return payload[i]; } } void main() { MySlice!(int/*, 1000*/)[your_max_length] slice; writeln(slice.sizeof); } ```Okay we were overthinking the solution. Just use a static array ```d int[your_max_length]/*or whatever type*/ var; ``` You're good to go! I almost feel stupid now lolHello Tejas, Kind thanks for your replies ... all are appreciated. However, do NOT feel stupid ... the motivation behind why I cannot use a standard int[your_max_length] (in other words, use a static array), is because I need to do a specified memory alignment (known at compile time) on my slice, or array. I understand that neither a slice or an array is capable to doing an arbitrary memory alignment. (But, perhaps I am wrong about this ...) I believe structs can be aligned, but need to learn more about the speicific of that. In the meantime, I have written a way to get an aligned slice from a static array. Ugly beginner code is below. While I believe this basic idea should work, I would like to guarantee that my slice does not get moved in memory (destroying the alignment). Ugly code here: ---------------------------------------------- import std.stdio; enum MAXLENGTH = 1024; enum ALIGN = 128; void main(){ // static array to allow creation of aligned slice ubyte[MAXLENGTH+ALIGN] u; writeln("u.ptr: ", u.ptr); auto u_i = cast(ulong) u.ptr; auto u_excess = u_i%ALIGN; writeln("u_excess: ", u_excess); ulong mustadd; if(u_excess !=0){ mustadd = ALIGN-u_excess; } writeln("mustadd: ", mustadd); // create aligned pointer for our needed slice auto zp = cast(double*) (u.ptr + mustadd); // create slice double[] z = zp[0 .. MAXLENGTH]; writeln("z.ptr: ", z.ptr); writeln("z.length: ", z.length); auto z_i = cast(ulong) z.ptr; auto z_excess = z_i%ALIGN; writeln("z_excess: ", z_excess); }
Aug 06 2021
On Friday, 6 August 2021 at 19:03:53 UTC, Tejas wrote:Stealing Paul's answer now: ```d import std; enum your_max_length = 1000; enum your_align = 256; struct MySlice(T/*, size_t maxLength*/) { private align(your_align)T payload; //invariant(payload.length <= maxLength); //this(T[] slice) { payload = slice; } //T opIndex(size_t i) { return payload[i]; } } void main() { MySlice!(int/*, 1000*/)[your_max_length] slice; writeln(slice.sizeof); } ```You can actually pass the alignment as a parameter too: ```d struct Aligned(T, size_t alignment) if (alignment >= T.alignof) { align(alignment) T payload; alias payload this; } void main() { Aligned!(int, 16)[4] x; assert(x.alignof == 16); assert(x.sizeof == 64); Aligned!(int[4], 16) y; assert(y.alignof == 16); assert(y.sizeof == 16); } ```
Aug 06 2021
On Friday, 6 August 2021 at 22:15:00 UTC, Paul Backus wrote:On Friday, 6 August 2021 at 19:03:53 UTC, Tejas wrote:Is the second example really correct? if the alignment of ```int[4]``` really is 16, then why isn't the size of it 64 as well? It seems that the alignment constraint is not being satisfied _within_ the array, And if it really is correct, then it seems once again that static arrays are the answer after all: ```d align(your_alignment) int[your_length] array; ``` No need for structs \\('_')/Stealing Paul's answer now: ```d import std; enum your_max_length = 1000; enum your_align = 256; struct MySlice(T/*, size_t maxLength*/) { private align(your_align)T payload; //invariant(payload.length <= maxLength); //this(T[] slice) { payload = slice; } //T opIndex(size_t i) { return payload[i]; } } void main() { MySlice!(int/*, 1000*/)[your_max_length] slice; writeln(slice.sizeof); } ```You can actually pass the alignment as a parameter too: ```d struct Aligned(T, size_t alignment) if (alignment >= T.alignof) { align(alignment) T payload; alias payload this; } void main() { Aligned!(int, 16)[4] x; assert(x.alignof == 16); assert(x.sizeof == 64); Aligned!(int[4], 16) y; assert(y.alignof == 16); assert(y.sizeof == 16); } ```
Aug 07 2021
On Saturday, 7 August 2021 at 07:32:04 UTC, Tejas wrote:And if it really is correct, then it seems once again that static arrays are the answer after all: ```d align(your_alignment) int[your_length] array; ``` No need for structs \\('_')/The main advantage of the struct is that you can heap-allocate it: ```d auto array = new Aligned!(int[4], 16); ```
Aug 07 2021
On Saturday, 7 August 2021 at 12:08:00 UTC, Paul Backus wrote:On Saturday, 7 August 2021 at 07:32:04 UTC, Tejas wrote:**First, thanks all for helping with this question!** The simple desire to arbitrarily align an array is certainly looking non-trivial. Below is a simple program of both the suggested "struct" and "align array" solutions. Unfortunately, neither is guaranteed to place the array with the desired alignnment. import std.stdio; enum ALIGNMENT=64; enum LENGTH=10; struct Aligned(T, size_t alignment) if (alignment >= T.alignof) { align(alignment) T payload; alias payload this; } void main() { writeln("ALIGNMENT: ", ALIGNMENT, ",\t\tLENGTH: ", LENGTH); int[23] junk; align(ALIGNMENT) int[LENGTH] z; writeln("\nusing 'align(ALIGNMENT) int[LENGTH] z;'"); writeln("\ncast(ulong) z.ptr%ALIGNMENT: ", cast(ulong) z.ptr%ALIGNMENT); writeln("int.alignof: ", int.alignof, ",\tint.sizeof: ", int.sizeof); writeln("&z[0]", &z[0]); writeln("&z[1]", &z[1]); writeln("&z[2]", &z[2]); writeln("\nusing: 'Aligned!(int, ALIGNMENT)[LENGTH] x;'"); Aligned!(int, ALIGNMENT)[LENGTH] x; writeln("x.sizeof: ", x.sizeof, "\t\tx.alignof: ", x.alignof); writeln("x: ", x); writeln("\nx.ptr: ", x.ptr); writeln("&x[0]", &x[0]); writeln("&x[1]", &x[1]); writeln("&x[2]", &x[2]); writeln("\ncast(ulong) x.ptr%ALIGNMENT: ", cast(ulong) x.ptr%ALIGNMENT); } --------------------------------------------------------------- and here is a sample output of a run: ALIGNMENT: 64, LENGTH: 10 using 'align(ALIGNMENT) int[LENGTH] z;' cast(ulong) z.ptr%ALIGNMENT: ***32*** int.alignof: 4, int.sizeof: 4 &z[0]7FFFF8D05B60 &z[1]7FFFF8D05B64 &z[2]7FFFF8D05B68 using: 'Aligned!(int, ALIGNMENT)[LENGTH] x;' x.sizeof: 640 x.alignof: 64 x: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] x.ptr: 7FFFF8D05B90 &x[0]7FFFF8D05B90 &x[1]7FFFF8D05BD0 &x[2]7FFFF8D05C10 cast(ulong) x.ptr%ALIGNMENT: ***16*** --------------------------------------------------------------- Notice, that neither attempt yields an array starting address as zero modulo the alignment value ... Also, a bit weird is that the "align array" solution yields expected spacing between array elements, but the "struct" solution has wider separations ... ------------------------------------------------------ **Again, I am very appreciative of your replies and help with this.** I have learned quite a bit from this discussion. However, I remain a bit stumped by all of this.... Any ideas out there? Best Regards, JamesAnd if it really is correct, then it seems once again that static arrays are the answer after all: ```d align(your_alignment) int[your_length] array; ``` No need for structs \\('_')/The main advantage of the struct is that you can heap-allocate it: ```d auto array = new Aligned!(int[4], 16); ```
Aug 07 2021
On Saturday, 7 August 2021 at 13:36:52 UTC, james.p.leblanc wrote:On Saturday, 7 August 2021 at 12:08:00 UTC, Paul Backus wrote:Umm, the ```align array``` solution is flat out wrong, please ignore it. Most likely a bug in the compiler. Also, why will the address of the first element of the array ```modulo``` alignment be 0? The address of the array has absolutely nothing to do with the alignment. You say that the ```align array``` solution has the expected spacing, but it is default aligned, totally ignoring your 64 byte requirement. Don't use ```x.PTR%ALIGNMENT``` Use ```(&x[1]-&[2])%ALIGNMENT``` You will get ALIGNMENT*(2 -1)(since you took difference of 2nd and 1st elem)[...]**First, thanks all for helping with this question!** The simple desire to arbitrarily align an array is certainly looking non-trivial. Below is a simple program of both the suggested "struct" and "align array" solutions. Unfortunately, neither is guaranteed to place the array with the desired alignnment. [...]
Aug 07 2021
On Saturday, 7 August 2021 at 14:34:49 UTC, Tejas wrote:Umm, the ```align array``` solution is flat out wrong, please ignore it. Most likely a bug in the compiler. Also, why will the address of the first element of the array ```modulo``` alignment be 0? The address of the array has absolutely nothing to do with the alignment. You say that the ```align array``` solution has the expected spacing, but it is default aligned, totally ignoring your 64 byte requirement. Don't use ```x.PTR%ALIGNMENT``` Use ```(&x[1]-&[2])%ALIGNMENT``` You will get ALIGNMENT*(2 -1)(since you took difference of 2nd and 1st elem)For the array as a whole to be aligned, not only must the spacing between the elements respect the alignment, but starting address of the array itself must be a multiple of the alignment. So it is correct to check that `x.ptr%ALIGNMENT == 0`. The issue with `align` attributes being ignored for stack variables is apparently a known bug, first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16098 The issue with `align` attributes being ignored by `new` is also a known bug, and was also first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16508
Aug 07 2021
On Saturday, 7 August 2021 at 15:21:01 UTC, Paul Backus wrote:On Saturday, 7 August 2021 at 14:34:49 UTC, Tejas wrote:Oh wow, and here I thought I was being smart :( So, how can we work around this without assembly language magic? I'm illiterate at assembly.[...]For the array as a whole to be aligned, not only must the spacing between the elements respect the alignment, but starting address of the array itself must be a multiple of the alignment. So it is correct to check that `x.ptr%ALIGNMENT == 0`. The issue with `align` attributes being ignored for stack variables is apparently a known bug, first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16098 The issue with `align` attributes being ignored by `new` is also a known bug, and was also first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16508
Aug 07 2021
On Saturday, 7 August 2021 at 15:41:24 UTC, Tejas wrote:On Saturday, 7 August 2021 at 15:21:01 UTC, Paul Backus wrote:For stack allocations, you can use the workaround in [Vladimir Panteleev's comment][1] (ignoring the ASM part, which is unrelated to alignment). For heap allocations, I guess the easiest way would be to use [`AlignedMallocator`][2] from `std.experimental.allocator`. [1]: https://issues.dlang.org/show_bug.cgi?id=16098#c3 [2]: https://phobos.dpldocs.info/std.experimental.allocator.mallocator.AlignedMallocator.htmlThe issue with `align` attributes being ignored for stack variables is apparently a known bug, first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16098 The issue with `align` attributes being ignored by `new` is also a known bug, and was also first reported in 2016: https://issues.dlang.org/show_bug.cgi?id=16508Oh wow, and here I thought I was being smart :( So, how can we work around this without assembly language magic? I'm illiterate at assembly.
Aug 07 2021
On Saturday, 7 August 2021 at 19:07:04 UTC, Paul Backus wrote:On Saturday, 7 August 2021 at 15:41:24 UTC, Tejas wrote:Cool... thanksOn Saturday, 7 August 2021 at 15:21:01 UTC, Paul Backus wrote:For stack allocations, you can use the workaround in [Vladimir Panteleev's comment][1] (ignoring the ASM part, which is unrelated to alignment). For heap allocations, I guess the easiest way would be to use [`AlignedMallocator`][2] from `std.experimental.allocator`. [1]: https://issues.dlang.org/show_bug.cgi?id=16098#c3 [2]: https://phobos.dpldocs.info/std.experimental.allocator.mallocator.AlignedMallocator.html[...]Oh wow, and here I thought I was being smart :( So, how can we work around this without assembly language magic? I'm illiterate at assembly.
Aug 07 2021
On Sunday, 8 August 2021 at 02:00:26 UTC, Tejas wrote:On Saturday, 7 August 2021 at 19:07:04 UTC, Paul Backus wrote:Thanks to everyone for all of the great discussion and hints on this topic! I have learned quite much, and have a gained an understanding on how to use D effectively. Fantastic forum here on a great language. Best Regards, JamesOn Saturday, 7 August 2021 at 15:41:24 UTC, Tejas wrote:Cool... thanksOn Saturday, 7 August 2021 at 15:21:01 UTC, Paul Backus wrote:For stack allocations, you can use the workaround in [Vladimir Panteleev's comment][1] (ignoring the ASM part, which is unrelated to alignment). For heap allocations, I guess the easiest way would be to use [`AlignedMallocator`][2] from `std.experimental.allocator`. [1]: https://issues.dlang.org/show_bug.cgi?id=16098#c3 [2]: https://phobos.dpldocs.info/std.experimental.allocator.mallocator.AlignedMallocator.html[...]Oh wow, and here I thought I was being smart :( So, how can we work around this without assembly language magic? I'm illiterate at assembly.
Aug 07 2021
On Saturday, 7 August 2021 at 13:36:52 UTC, james.p.leblanc wrote:On Saturday, 7 August 2021 at 12:08:00 UTC, Paul Backus wrote:If you don't believe me just subtract the adresses yourself: x.ptr: 7FFFF8D05B90 &x[0]7FFFF8D05B90 &x[1]7FFFF8D05BD0 &x[2]7FFFF8D05C10 &x[1] 7FFFF8D05BD0 -&x[0] -7FFFF8D05B90 _____________________ 000000000040 0X40 = 64 in base 10 Your alignment requirement was 64 bytes[...]**First, thanks all for helping with this question!** The simple desire to arbitrarily align an array is certainly looking non-trivial. Below is a simple program of both the suggested "struct" and "align array" solutions. Unfortunately, neither is guaranteed to place the array with the desired alignnment. [...]
Aug 07 2021