www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - byte + byte = int: why?

reply Mek101 <oidgemek101 gmail.com> writes:
I have the following line of code:

     temp[fowardI] = temp[fowardI] + temp[backwardI];

Where temp is a byte array (byte[]). When I try to compile it dmd 
gives me this error:

     source/hash.d(11,25): Error: cannot implicitly convert 
expression `cast(int)temp[fowardI] + cast(int)temp[backwardI]` of 
type `int` to `byte`
/usr/bin/dmd failed with exit code 1.

Meaning that the byte type doesn't have a + operator. I know by 

the absence of the operator was justified because the CLI didn't 
support addition between values smaller than an int, and also 
byte wasn't enough "numberish" to be used as such, and I should 
have used int instead.

But why the D language doesn't implement the operator on byte?
Jan 18 2019
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/18/19 12:09 PM, Mek101 wrote:

 I have the following line of code:
 
      temp[fowardI] = temp[fowardI] + temp[backwardI];
 
 Where temp is a byte array (byte[]). When I try to compile it dmd gives 
 me this error:
 
      source/hash.d(11,25): Error: cannot implicitly convert expression 
 `cast(int)temp[fowardI] + cast(int)temp[backwardI]` of type `int` to `byte`
 /usr/bin/dmd failed with exit code 1.
 
 Meaning that the byte type doesn't have a + operator. I know by 

 absence of the operator was justified because the CLI didn't support 
 addition between values smaller than an int, and also byte wasn't enough 
 "numberish" to be used as such, and I should have used int instead.
 
 But why the D language doesn't implement the operator on byte?


What is 127 + 127? Answer: 254. Which if converted to a byte is -127. 
Not what you might expect if you are doing addition.

In fact, D promotes all integral types smaller than int to int to do 
arithmetic. And if the result might not fit into what you are assigning 
it to, it requires a cast or mask.

See documentation here: 
https://dlang.org/spec/type.html#usual-arithmetic-conversions

-Steve
Jan 18 2019
parent reply Mek101 <oidgemek101 gmail.com> writes:
On Friday, 18 January 2019 at 17:15:09 UTC, Steven Schveighoffer 
wrote:

 What is 127 + 127? Answer: 254. Which if converted to a byte is 
 -127. Not what you might expect if you are doing addition.


Quite similar to int.max + int.max


 In fact, D promotes all integral types smaller than int to int 
 to do arithmetic. And if the result might not fit into what you 
 are assigning it to, it requires a cast or mask.


Then why isn't int + int = long? If i did the example above, they 
wouldn't fit in a int for sure, yet the result is of the same 
type. Why should byte be any different?
Jan 18 2019
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 18, 2019 at 05:26:35PM +0000, Mek101 via Digitalmars-d-learn wrote:

 On Friday, 18 January 2019 at 17:15:09 UTC, Steven Schveighoffer wrote:

 What is 127 + 127? Answer: 254. Which if converted to a byte is
 -127.  Not what you might expect if you are doing addition.


 
 Quite similar to int.max + int.max
 

 In fact, D promotes all integral types smaller than int to int to do
 arithmetic. And if the result might not fit into what you are
 assigning it to, it requires a cast or mask.


 
 Then why isn't int + int = long? If i did the example above, they
 wouldn't fit in a int for sure, yet the result is of the same type.
 Why should byte be any different?


It's historical baggage from C's integer promotion rules, that's all
there is to it.  Many people seem to think that's a good thing. I
personally think it sucks, and so I've written a nopromote module to
work around the autopromotion (see my other post).

OTOH if you're concerned about int overflow, just cast to long:

	int x = int.max, y = int.max;
	auto result = cast(long)x + y;

Of course, eventually you run out of options, e.g., if you try to add
ulong.max to ulong.max.  Generally most people won't run into that, but
if you're concerned about that, you could pull out the big guns and
import std.bigint. :-D


T

-- 
Music critic: "That's an imitation fugue!"
Jan 18 2019
prev sibling next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 18 January 2019 at 17:26:35 UTC, Mek101 wrote:

 Then why isn't int + int = long? If i did the example above, 
 they wouldn't fit in a int for sure, yet the result is of the 
 same type. Why should byte be any different?


Because C didn't define it that way. And C didn't define it that 
way because of convenience and efficiency on common hardware 
biased toward int (and nowadays with 64 bit, we kept int at 32 
for compatibility basically, the inertia set in).

We have proposed to change it before, but the main policy D has 
right now is "if it looks like C, it works like C or doesn't 
compile at all", so programmer habits and such don't silently 
bite them.
Jan 18 2019
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/18/19 12:26 PM, Mek101 wrote:

 On Friday, 18 January 2019 at 17:15:09 UTC, Steven Schveighoffer wrote:

 What is 127 + 127? Answer: 254. Which if converted to a byte is -127. 
 Not what you might expect if you are doing addition.


 
 Quite similar to int.max + int.max


Indeed. But this is where C drew the line, so it's where D draws the 
line as well.


 In fact, D promotes all integral types smaller than int to int to do 
 arithmetic. And if the result might not fit into what you are 
 assigning it to, it requires a cast or mask.


 
 Then why isn't int + int = long? If i did the example above, they 
 wouldn't fit in a int for sure, yet the result is of the same type. Why 
 should byte be any different?


There are other inconsistencies too, like += (which works on byte += byte).

As others have said, those are the rules D has for historical reasons, 
you just have to deal with them.

-Steve
Jan 18 2019
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 18, 2019 at 01:49:23PM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:

 On 1/18/19 12:26 PM, Mek101 wrote:


[...]

 Then why isn't int + int = long? If i did the example above, they
 wouldn't fit in a int for sure, yet the result is of the same type.
 Why should byte be any different?


 
 There are other inconsistencies too, like += (which works on byte +=
 byte).
 
 As others have said, those are the rules D has for historical reasons,
 you just have to deal with them.


[...]

Or use my nopromote module. ;-)  </shameless self-promotion>


T

-- 
Nearly all men can stand adversity, but if you want to test a man's character,
give him power. -- Abraham Lincoln
Jan 18 2019
prev sibling parent reply Rekel <paultjeadriaanse gmail.com> writes:
On Friday, 18 January 2019 at 18:49:23 UTC, Steven Schveighoffer 
wrote:

 As others have said, those are the rules D has for historical 
 reasons, you just have to deal with them.


Is that to ensure compatibility with C?
Aug 29 2021
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 8/29/21 3:57 AM, Rekel wrote:

 On Friday, 18 January 2019 at 18:49:23 UTC, Steven Schveighoffer wrote:

 As others have said, those are the rules D has for historical reasons, 
 you just have to deal with them.


 
 Is that to ensure compatibility with C?
 


Yes. It's mentioned multiple times in this zombie-ish thread. :)

Ali
Aug 29 2021
prev sibling next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 18 January 2019 at 17:09:52 UTC, Mek101 wrote:

 Meaning that the byte type doesn't have a + operator.


Well, it DOES have a + operator, it just returns int that can be 
squished to a size if and only if the compiler proves it fits.

A bit of background: the C language was designed on a machine 
that did arithmetic in a fixed size register. It defined `int` to 
be at least the size of this register and for all arithmetic to 
be expanded to at least that same size.

In other words, anything smaller than an `int` is converted to an 
`int` before operations are done to it.


languages) inherited this rule from C. So a + a will be converted 
to int for anything smaller than int, like byte or short.


Now the difference between C and D is that C would let you assign 
that result right back to the small type without complaining, 
whereas D issues the error. Why?

The reason is the carry bit. Consider the case of 127 + 127. The 
answer is too big to fit in a signed byte; one bit will carry 
over and be dropped. D considers this potential data loss and 
wants you to make a conscious choice about it via an explicit 
cast.

If the compiler knows for certain the numbers will fit, it allows 
it. Like 64 + 5 will implicitly cast to a byte because the 
compiler sees the size of the result. But for runtime values, it 
is usually unsure and forces you to do the cast.

  temp[fowardI] = cast(byte) (temp[fowardI] + temp[backwardI]);

that will work.
Jan 18 2019
prev sibling next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 18 January 2019 at 17:09:52 UTC, Mek101 wrote:

 Where temp is a byte array (byte[])



is unsigned.


byte"), range 0-255.
Jan 18 2019
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 18, 2019 at 05:09:52PM +0000, Mek101 via Digitalmars-d-learn wrote:

 I have the following line of code:
 
     temp[fowardI] = temp[fowardI] + temp[backwardI];
 
 Where temp is a byte array (byte[]). When I try to compile it dmd
 gives me this error:
 
     source/hash.d(11,25): Error: cannot implicitly convert expression
 `cast(int)temp[fowardI] + cast(int)temp[backwardI]` of type `int` to `byte`
 /usr/bin/dmd failed with exit code 1.
 
 Meaning that the byte type doesn't have a + operator. I know by

 absence of the operator was justified because the CLI didn't support
 addition between values smaller than an int, and also byte wasn't
 enough "numberish" to be used as such, and I should have used int
 instead.
 
 But why the D language doesn't implement the operator on byte?



various extents) C's integer promotion rules.  Personally, I hate it,
but it's what we have, and changing it now would break too many things,
so we're stuck with it.

Basically, arithmetic on everything smaller than an int will implicitly
promote to int first. Which means the result will be int. But unlike C,
where you can implicitly narrow it back to a byte, in D the narrowing
must be explicit (to avoid accidental overflow when a narrow int type
like byte or short is too small to store the result), resulting in the
annoying situation where byte + byte != byte.

To work around this, you can use my nopromote module (code included
below), that implements a wrapper type that automatically casts back to
the narrow int type:

	ubyte x = 1;
	ubyte y = 2;
	auto z = x.np + y; // np stands for "no promote"
	static assert(is(typeof(x) == ubyte));
	assert(z == 3);

	x = 255;
	z = x.np + y;	// don't promote to int, so this will wrap
	assert(z == 1);	// wrapped around to 1

Check the unittests for more examples.

OT1H, having to write .np everywhere is annoying, but OTOH it does
document intent in the code, that you're expecting wraparound and have
presumably taken the necessary precautions to mitigate any ill-effects
that wrapping may have, so in that sense it's a good thing.


T

-- 
It is widely believed that reinventing the wheel is a waste of time; but I
disagree: without wheel reinventers, we would be still be stuck with wooden
horse-cart wheels.

-----------------------------------snip----------------------------------
/**
 * Truncating wrapper around built-in narrow ints to work around stupid casts.
 */
module nopromote;

enum isNarrowInt(T) = is(T : int) || is(T : uint);

/**
 * A wrapper around a built-in narrow int that truncates the result of
 * arithmetic operations to the narrow type, overriding built-in int promotion
 * rules.
 */
struct Np(T)
    if (isNarrowInt!T)
{
    T impl;
    alias impl this;

    /**
     * Truncating binary operator.
     */
    Np opBinary(string op, U)(U u)
        if (is(typeof((T x, U y) => mixin("x " ~ op ~ " y"))))
    {
        return Np(cast(T) mixin("this.impl " ~ op ~ " u"));
    }

    /**
     * Truncating unary operator.
     */
    Np opUnary(string op)()
        if (is(typeof((T x) => mixin(op ~ "cast(int) x"))))
    {
        return Np(cast(T) mixin(op ~ " cast(int) this.impl"));
    }

    /**
     * Infectiousness: any expression containing Np should automatically use Np
     * operator semantics.
     */
    Np opBinaryRight(string op, U)(U u)
        if (is(typeof((T x, U y) => mixin("x " ~ op ~ " y"))))
    {
        return Np(cast(T) mixin("u " ~ op ~ " this.impl"));
    }
}

/**
 * Returns: A lightweight wrapped type that overrides built-in arithmetic
 * operators to always truncate to the given type without promoting to int or
 * uint.
 */
auto np(T)(T t)
    if (isNarrowInt!T)
{
    return Np!T(t);
}

// Test binary ops
 safe unittest
{
    ubyte x = 1;
    ubyte y = 2;
    auto z = x.np + y;
    static assert(is(typeof(z) : ubyte));
    assert(z == 3);

    byte zz = x.np + y;
    assert(zz == 3);

    x = 255;
    z = x.np + y;
    assert(z == 1);
}

 safe unittest
{
    byte x = 123;
    byte y = 5;
    auto z = x.np + y;
    static assert(is(typeof(z) : byte));
    assert(z == byte.min);

    byte zz = x.np + y;
    assert(zz == byte.min);
}

 safe unittest
{
    import std.random;
    short x = cast(short) uniform(0, 10);
    short y = 10;
    auto z = x.np + y;
    static assert(is(typeof(z) : short));
    assert(z == x + 10);

    short s = x.np + y;
    assert(s == x + 10);
}

// Test unary ops
 safe unittest
{
    byte b = 10;
    auto c = -b.np;
    static assert(is(typeof(c) : byte));
    assert(c == -10);

    ubyte ub = 16;
    auto uc = -ub.np;
    static assert(is(typeof(uc) : ubyte));
    assert(uc == 0xF0);
}

version(unittest)
{
    // These tests are put here as actual module functions, to force optimizer
    // not to discard calls to these functions, so that we can see the actual
    // generated code.
    byte byteNegate(byte b) { return -b.np; }
    ubyte ubyteNegate(ubyte b) { return -b.np; }

    byte byteTest1(int choice, byte a, byte b)
    {
        if (choice == 1)
            return a.np + b;
        if (choice == 2)
            return a.np / b;
        assert(0);
    }

    short shortAdd(short a, short b) { return a.np + b; }

    // Test opBinaryRight
    byte byteRightTest(byte a, byte c)
    {
        auto result = a + c.np;
        static assert(is(typeof(result) : byte));
        return result;
    }

    unittest
    {
        assert(byteRightTest(127, 1) == byte.min);
    }

    short multiTest1(short x, short y)
    {
        return short(2) + 2*(x - y.np);
    }

    unittest
    {
        // Test wraparound semantics.
        assert(multiTest1(32767, 16384) == short.min);
    }

    short multiTest2(short a, short b)
    {
        short x = a;
        short y = b;
        return (2*x + 1) * (y.np/2 - 1);
    }

    unittest
    {
        assert(multiTest2(1, 4) == 3);
    }
}
-----------------------------------snip----------------------------------
Jan 18 2019
prev sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 1/18/19 9:09 AM, Mek101 wrote:


      source/hash.d(11,25): Error: cannot implicitly convert expression
 `cast(int)temp[fowardI] + cast(int)temp[backwardI]` of type `int` to 


`byte`

Others suggested casting as a solution which works but it will hide 
potential errors.

Depending on the situation, you may want to use std.conv.to, which does 
a value range check and throws an exception to prevent an error:

byte foo(byte a, byte b) {
   import std.conv : to;
   return (a + b).to!byte;
}

void main() {
   foo(42, 42);    // Works
   foo(100, 100);  // Throws ConvOverflowException
}

Of course, because of the checks, 'to' is slower than casting blindly.

Ali
Aug 29 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 29 August 2021 at 15:42:18 UTC, Ali Çehreli wrote:

 Depending on the situation, you may want to use std.conv.to, 
 which does a value range check and throws an exception to 
 prevent an error:

     byte foo(byte a, byte b) {
       import std.conv : to;
       return (a + b).to!byte;
     }

     void main() {
       foo(42, 42);    // Works
       foo(100, 100);  // Throws ConvOverflowException
     }


I was going to suggest `std.experimental.checkedint` as an 
alternative here, but it turns out that it does integer promotion 
too--`Checked!byte + Checked!byte == Checked!int`.

This seems obviously wrong to me, but according to run.dlang.io 
it's always worked that way.
Aug 29 2021
parent reply jfondren <julian.fondren gmail.com> writes:
On Sunday, 29 August 2021 at 15:57:18 UTC, Paul Backus wrote:

 On Sunday, 29 August 2021 at 15:42:18 UTC, Ali Çehreli wrote:

 Depending on the situation, you may want to use std.conv.to, 
 which does a value range check and throws an exception to 
 prevent an error:

     byte foo(byte a, byte b) {
       import std.conv : to;
       return (a + b).to!byte;
     }

     void main() {
       foo(42, 42);    // Works
       foo(100, 100);  // Throws ConvOverflowException
     }


 I was going to suggest `std.experimental.checkedint` as an 
 alternative here, but it turns out that it does integer 
 promotion too--`Checked!byte + Checked!byte == Checked!int`.

 This seems obviously wrong to me, but according to run.dlang.io 
 it's always worked that way.


(a.checked + b).get is int, but this works:

```d
import std.experimental.checkedint;

byte foo(byte a, byte b) {
     auto c = a.checked;
     c += b;
     return c.get;
}

unittest {
     import std.exception : assertThrown;
     import core.exception : AssertError;

     foo(42, 42);
     assertThrown!AssertError(foo(100, 100));
}
```

... after Phobos is patched.

```
error: undefined identifier ‘Lhs’, did you mean alias ‘Rhs’?
```
Aug 29 2021
parent Paul Backus <snarwin gmail.com> writes:
On Sunday, 29 August 2021 at 16:21:40 UTC, jfondren wrote:

 ... after Phobos is patched.

 ```
 error: undefined identifier ‘Lhs’, did you mean alias ‘Rhs’?
 ```


Shows how much anyone actually uses this code, I guess--the bug 
was introduced [in 2017][1], and as far as I can tell has never 
even been reported before.

Bugzilla: https://issues.dlang.org/show_bug.cgi?id=22249

[1]: https://github.com/dlang/phobos/commit/1a62d66d93
Aug 29 2021