www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - 24-bit int

reply EntangledQuanta <EQ universe.com> writes:
Is there a way to create a 24-bit int? One that for all practical 
purposes acts as such? This is for 24-bit stuff like audio. It 
would respect endianness, allow for arrays int24[] that work 
properly, etc.
Sep 01
next sibling parent reply Biotronic <simen.kjaras gmail.com> writes:
On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
wrote:
 Is there a way to create a 24-bit int? One that for all 
 practical purposes acts as such? This is for 24-bit stuff like 
 audio. It would respect endianness, allow for arrays int24[] 
 that work properly, etc.
I haven't looked at endianness beyond it working on my computer. If you have special needs in that regard, consider this a starting point: struct int24 { ubyte[3] _payload; this(int x) { value = x; } property int value() { int val = *cast(int*)&_payload & 0xFFFFFF; if (val & 0x800000) { val |= 0xFF000000; } return val; } property int value(int x) { _payload = (cast(ubyte*)&x)[0..3]; return value; } auto opUnary(string op)() { static if (op == "++") { value = value + 1; } else static if (op == "--") { value = value - 1; } else static if (op == "+") { return value; } else static if (op == "-") { return -value; } else static if (op == "~") { return ~value; } else { static assert(false, "Unary operator '"~op~"' is not supported by int24."); } } auto opOpAssign(string op)(int x) { static assert(__traits(compiles, {mixin("value = value "~op~" x;");}), "Binary operator '"~op~"' is not supported by int24."); mixin("value = value "~op~" x;"); return this; } alias value this; } unittest { int24[3] a; assert(a.sizeof == 9); // Max value a[1] = 8388607; assert(a[1] == 8388607); // Test for buffer overflow: assert(a[0] == 0); assert(a[2] == 0); // Overflow a[1] = 8388608; assert(a[1] == -8388608); // Test for buffer overflow: assert(a[0] == 0); assert(a[2] == 0); // negative value a[1] = -1; assert(a[1] == -1); // Test for buffer overflow: assert(a[0] == 0); assert(a[2] == 0); // Unary operators a[1] = 0; assert(~a[1] == -1); a[1]--; assert(a[1] == -1); assert(-a[1] == 1); assert(+a[1] == -1); a[1]++; assert(a[1] == 0); // Binary operators a[1] = 0; a[1] = a[1] + 1; assert(a[1] == 1); a[1] += 1; assert(a[1] == 2); a[1] = a[1] - 1; assert(a[1] == 1); a[1] -= 1; assert(a[1] == 0); a[1] = 3; a[1] = a[1] * 2; assert(a[1] == 6); a[1] = a[1] / 2; assert(a[1] == 3); a[1] *= 2; assert(a[1] == 6); a[1] /= 2; assert(a[1] == 3); a[1] = a[1] << 1; assert(a[1] == 6); a[1] <<= 1; assert(a[1] == 12); a[1] = a[1] >> 1; assert(a[1] == 6); a[1] >>= 1; assert(a[1] == 3); a[1] |= 4; assert(a[1] == 7); a[1] &= 5; assert(a[1] == 5); a[1] = a[1] | 2; assert(a[1] == 7); a[1] = a[1] & 3; assert(a[1] == 3); } -- Biotronic
Sep 01
next sibling parent EntangledQuanta <EQ universe.com> writes:
On Friday, 1 September 2017 at 22:10:43 UTC, Biotronic wrote:
 On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
 wrote:
 [...]
I haven't looked at endianness beyond it working on my computer. If you have special needs in that regard, consider this a starting point: [...]
Thanks, I'll check it out and see.
Sep 01
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Friday, 1 September 2017 at 22:10:43 UTC, Biotronic wrote:
 On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
 wrote:
 Is there a way to create a 24-bit int? One that for all 
 practical purposes acts as such? This is for 24-bit stuff like 
 audio. It would respect endianness, allow for arrays int24[] 
 that work properly, etc.
I haven't looked at endianness beyond it working on my computer. If you have special needs in that regard, consider this a starting point: struct int24 { ubyte[3] _payload; this(int x) { value = x; } ... } -- Biotronic
You may also want to put an align(1) on it so that you dont waste 25% of the allocated memory in an array of int24's
Sep 01
next sibling parent reply EntangledQuanta <EQ universe.com> writes:
On Saturday, 2 September 2017 at 00:43:00 UTC, Nicholas Wilson 
wrote:
 On Friday, 1 September 2017 at 22:10:43 UTC, Biotronic wrote:
 On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
 wrote:
 Is there a way to create a 24-bit int? One that for all 
 practical purposes acts as such? This is for 24-bit stuff 
 like audio. It would respect endianness, allow for arrays 
 int24[] that work properly, etc.
I haven't looked at endianness beyond it working on my computer. If you have special needs in that regard, consider this a starting point: struct int24 { ubyte[3] _payload; this(int x) { value = x; } ... } -- Biotronic
You may also want to put an align(1) on it so that you dont waste 25% of the allocated memory in an array of int24's
The whole point is so that there is no wasted space, so if it requires that then it's not a waste of space but a bug. Audio that is in24 is 3 bytes per sample, not 4. Every 3 bytes are a sample, not every 3 out of 4. Basically a byte[] cast to a int24 array should be 1/3 the size and every 3 bytes are the same as an int24. Thanks for pointing this out if it is necessary.
Sep 01
parent reply Mike Parker <aldacron gmail.com> writes:
On Saturday, 2 September 2017 at 01:19:52 UTC, EntangledQuanta 
wrote:
 The whole point is so that there is no wasted space, so if it 
 requires that then it's not a waste of space but a bug.

 Audio that is in24 is 3 bytes per sample, not 4. Every 3 bytes 
 are a sample, not every 3 out of 4.

 Basically a byte[] cast to a int24 array should be 1/3 the size 
 and every 3 bytes are the same as an int24.

 Thanks for pointing this out if it is necessary.
It's not a bug, but a feature. Data structure alignment is important for efficient reads, so several languages (D, C, C++, Ada, and more) will automatically pad structs so that they can maintain specific byte alignments. On a 32-bit system, 4-byte boundaries are the default. So a struct with 3 ubytes is going to be padded with an extra byte at the end. Telling the compiler to align on a 1-byte boundary (essentially disabling alignment) will save you space, but will will generally cost you cycles in accessing the data.
Sep 01
next sibling parent EntangledQuanta <EQ Universe.com> writes:
On Saturday, 2 September 2017 at 02:37:08 UTC, Mike Parker wrote:
 On Saturday, 2 September 2017 at 01:19:52 UTC, EntangledQuanta 
 wrote:
 The whole point is so that there is no wasted space, so if it 
 requires that then it's not a waste of space but a bug.

 Audio that is in24 is 3 bytes per sample, not 4. Every 3 bytes 
 are a sample, not every 3 out of 4.

 Basically a byte[] cast to a int24 array should be 1/3 the 
 size and every 3 bytes are the same as an int24.

 Thanks for pointing this out if it is necessary.
It's not a bug, but a feature. Data structure alignment is important for efficient reads, so several languages (D, C, C++, Ada, and more) will automatically pad structs so that they can maintain specific byte alignments. On a 32-bit system, 4-byte boundaries are the default. So a struct with 3 ubytes is going to be padded with an extra byte at the end. Telling the compiler to align on a 1-byte boundary (essentially disabling alignment) will save you space, but will will generally cost you cycles in accessing the data.
You fail to read correctly. A bug in his code. If he is treating in24's as int32's and simply ignoring the upper byte then it is not a true int24 and all the work he did would be pointless. I can do that by simply reading an int32 and masking the high bit.
Sep 01
prev sibling parent reply kinke <noone nowhere.com> writes:
On Saturday, 2 September 2017 at 02:37:08 UTC, Mike Parker wrote:
 It's not a bug, but a feature. Data structure alignment is 
 important for efficient reads, so several languages (D, C, C++, 
 Ada, and more) will automatically pad structs so that they can 
 maintain specific byte alignments. On a 32-bit system, 4-byte 
 boundaries are the default. So a struct with 3 ubytes is going 
 to be padded with an extra byte at the end. Telling the 
 compiler to align on a 1-byte boundary (essentially disabling 
 alignment) will save you space, but will will generally cost 
 you cycles in accessing the data.
struct int24 { ubyte[3] _payload; } static assert(int24.sizeof == 3); static assert(int24.alignof == 1); Making absolute sense. ubytes don't need any specific alignment to be read efficiently.
Sep 02
parent Mike Parker <aldacron gmail.com> writes:
On Saturday, 2 September 2017 at 07:20:07 UTC, kinke wrote:

 struct int24 {
     ubyte[3] _payload;
 }

 static assert(int24.sizeof == 3);
 static assert(int24.alignof == 1);

 Making absolute sense. ubytes don't need any specific alignment 
 to be read efficiently.
Yes, that does make sense. It doesn't make sense that I didn't realize it.
Sep 02
prev sibling parent Biotronic <simen.kjaras gmail.com> writes:
On Saturday, 2 September 2017 at 00:43:00 UTC, Nicholas Wilson 
wrote:
 On Friday, 1 September 2017 at 22:10:43 UTC, Biotronic wrote:
 struct int24 {
     ubyte[3] _payload;

     this(int x) {
         value = x;
     }

     ...
 }
You may also want to put an align(1) on it so that you dont waste 25% of the allocated memory in an array of int24's
The very first test in my code checks this: int24[3] a; assert(a.sizeof == 9); On the other hand, using Mir's well-tested code instead of something I hacked together in 10 minutes is probably a good idea. -- Biotronic
Sep 02
prev sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Friday, 1 September 2017 at 22:10:43 UTC, Biotronic wrote:
 On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
 wrote:
 Is there a way to create a 24-bit int? One that for all 
 practical purposes acts as such? This is for 24-bit stuff like 
 audio. It would respect endianness, allow for arrays int24[] 
 that work properly, etc.
I haven't looked at endianness beyond it working on my computer. If you have special needs in that regard, consider this a starting point:
big endian is indeed problematic.
      property
     int value(int x) {
         _payload = (cast(ubyte*)&x)[0..3];
         return value;
     }
will not work on big endian machine. version(BigEndian) _payload = (cast(ubyte*)&x)[1..4];
Sep 03
prev sibling next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
wrote:
 Is there a way to create a 24-bit int? One that for all 
 practical purposes acts as such? This is for 24-bit stuff like 
 audio. It would respect endianness, allow for arrays int24[] 
 that work properly, etc.
Hi, Probably you are looking for bitpack ndslice topology: http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#.bitpack sizediff_t[] data; // creates a packed signed integer slice with max allowed value equal to `2^^24 - 1`. auto packs = data[].sliced.bitpack!24; packs has the same API as D arrays Package is Mir Algorithm http://code.dlang.org/packages/mir-algorithm Best, Ilya
Sep 01
parent reply EntangledQuanta <EQ Universe.com> writes:
On Saturday, 2 September 2017 at 02:49:41 UTC, Ilya Yaroshenko 
wrote:
 On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
 wrote:
 Is there a way to create a 24-bit int? One that for all 
 practical purposes acts as such? This is for 24-bit stuff like 
 audio. It would respect endianness, allow for arrays int24[] 
 that work properly, etc.
Hi, Probably you are looking for bitpack ndslice topology: http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#.bitpack sizediff_t[] data; // creates a packed signed integer slice with max allowed value equal to `2^^24 - 1`. auto packs = data[].sliced.bitpack!24; packs has the same API as D arrays Package is Mir Algorithm http://code.dlang.org/packages/mir-algorithm Best, Ilya
Thanks. Seems useful.
Sep 01
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Saturday, 2 September 2017 at 03:29:20 UTC, EntangledQuanta 
wrote:
 On Saturday, 2 September 2017 at 02:49:41 UTC, Ilya Yaroshenko 
 wrote:
 On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
 wrote:
 Is there a way to create a 24-bit int? One that for all 
 practical purposes acts as such? This is for 24-bit stuff 
 like audio. It would respect endianness, allow for arrays 
 int24[] that work properly, etc.
Hi, Probably you are looking for bitpack ndslice topology: http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#.bitpack sizediff_t[] data; // creates a packed signed integer slice with max allowed value equal to `2^^24 - 1`. auto packs = data[].sliced.bitpack!24; packs has the same API as D arrays Package is Mir Algorithm http://code.dlang.org/packages/mir-algorithm Best, Ilya
Thanks. Seems useful.
Just added `bytegroup` topology. Released in v0.6.12 (will be available in DUB after few minutes.) http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#bytegroup It is faster for your task then `bitpack`. Best regards, Ilya
Sep 02
parent reply EntangledQuanta <EQ universe.com> writes:
On Sunday, 3 September 2017 at 04:01:34 UTC, Ilya Yaroshenko 
wrote:
 On Saturday, 2 September 2017 at 03:29:20 UTC, EntangledQuanta 
 wrote:
 On Saturday, 2 September 2017 at 02:49:41 UTC, Ilya Yaroshenko 
 wrote:
 On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
 wrote:
 Is there a way to create a 24-bit int? One that for all 
 practical purposes acts as such? This is for 24-bit stuff 
 like audio. It would respect endianness, allow for arrays 
 int24[] that work properly, etc.
Hi, Probably you are looking for bitpack ndslice topology: http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#.bitpack sizediff_t[] data; // creates a packed signed integer slice with max allowed value equal to `2^^24 - 1`. auto packs = data[].sliced.bitpack!24; packs has the same API as D arrays Package is Mir Algorithm http://code.dlang.org/packages/mir-algorithm Best, Ilya
Thanks. Seems useful.
Just added `bytegroup` topology. Released in v0.6.12 (will be available in DUB after few minutes.) http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#bytegroup It is faster for your task then `bitpack`. Best regards, Ilya
Thanks! I might end up using this. Is this basically just a logical mapping(cast(int)bytes[i*3]) & 0xFFFFFF) type of stuff or is there more of a performance hit? I could do the mapping myself if that is the case as I do not need much of a general solution. I'll probably be using in a just a few lines of code. It just needs to be nearly as fast as direct access.
Sep 03
parent Ilya <ilyayaroshenko gmail.com> writes:
On Sunday, 3 September 2017 at 23:30:43 UTC, EntangledQuanta 
wrote:
 On Sunday, 3 September 2017 at 04:01:34 UTC, Ilya Yaroshenko 
 wrote:
 On Saturday, 2 September 2017 at 03:29:20 UTC, EntangledQuanta 
 wrote:
 On Saturday, 2 September 2017 at 02:49:41 UTC, Ilya 
 Yaroshenko wrote:
 [...]
Thanks. Seems useful.
Just added `bytegroup` topology. Released in v0.6.12 (will be available in DUB after few minutes.) http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#bytegroup It is faster for your task then `bitpack`. Best regards, Ilya
Thanks! I might end up using this. Is this basically just a logical mapping(cast(int)bytes[i*3]) & 0xFFFFFF) type of stuff or is there more of a performance hit? I could do the mapping myself if that is the case as I do not need much of a general solution. I'll probably be using in a just a few lines of code. It just needs to be nearly as fast as direct access.
The implementation can be found here https://github.com/libmir/mir-algorithm/blob/master/source/mir/ndslice/iterator.d It uses unions and byte loads. The speed should be almost the same as direct access.
Sep 03
prev sibling parent solidstate1991 <laszloszeremi outlook.com> writes:
On Friday, 1 September 2017 at 19:39:14 UTC, EntangledQuanta 
wrote:
 Is there a way to create a 24-bit int? One that for all 
 practical purposes acts as such? This is for 24-bit stuff like 
 audio. It would respect endianness, allow for arrays int24[] 
 that work properly, etc.
If you need to perform math on integer audio, I strongly suggest to use int32 for this purpose to avoid performance degradation, then convert back to int24 when you're finished. Probably going to add such feature into my PCM library.
Sep 02