digitalmars.D - Weird issue with std.range.iota.length
- Meta (30/30) Feb 11 2016 If you try to compile this code, it will currently not work:
- Rikki Cattermole (2/31) Feb 11 2016 So I assume if you compiled as 64bit this will work then?
- Meta (4/5) Feb 11 2016 Yes, because `unsigned(begin - end)` will yield a ulong
- Rikki Cattermole (2/6) Feb 11 2016 So an edge case.
- Meta (4/13) Feb 11 2016 Yes, it's an edge case, although it's a pretty easy one to run
- Jonathan M Davis (22/52) Feb 12 2016 What it comes down to is that length should always be size_t.
- tsbockman (14/17) Feb 12 2016 Many micro-controllers will probably remain 32-bit indefinitely;
- tsbockman (6/8) Feb 12 2016 Just to clarify - the bit about `std.conv.to()` is just an
- Jonathan M Davis (8/16) Feb 12 2016 Yeah. I think that that's the best approach, but it does
- Meta (18/41) Feb 12 2016 What about adding another overload of iota, say, iotaEx or
- Jonathan M Davis (12/56) Feb 12 2016 It would be far better IMHO to just do a check in iota and throw
- Steven Schveighoffer (19/75) Feb 12 2016 Just because you have a range with a ulong length doesn't mean you are
- ixid (4/16) Feb 12 2016 What about a template overload where you can set the length type
- Jonathan M Davis (8/25) Feb 12 2016 No other range has anything but size_t for its length. It's what
- H. S. Teoh via Digitalmars-d (18/42) Feb 12 2016 [...]
- Andrei Alexandrescu (3/8) Feb 12 2016 Tried that for a while, it quickly became a mess of entangled
- H. S. Teoh via Digitalmars-d (6/16) Feb 12 2016 The intestines are still visible in iota(), and anything that uses it in
- Andrei Alexandrescu (2/4) Feb 12 2016 Do we want them everywhere? -- Andrei
- Meta (7/9) Feb 12 2016 What do you think of the solution I proposed above? It's not
- Andrei Alexandrescu (3/11) Feb 12 2016 I'd be more partial to an additional template parameter. Or even change
- Meta (3/19) Feb 12 2016 Okay, I will look into other solutions and post some options soon.
- w0rp (9/9) Feb 14 2016 Maybe I'm missing something, but can't the length just be size_t?
- Jonathan M Davis (10/19) Feb 14 2016 Yeah. It would be easy enough to just make iota size_t and then
- Dominikus Dittes Scherkl (3/22) Feb 14 2016 +1
- Meta (3/33) Feb 21 2016 I made a pull request:
If you try to compile this code, it will currently not work: foreach (n; iota(1UL, 1000).parallel) { //... } This is because of how the length is calculated by iota: auto iota(B, E)(B begin, E end) if (isIntegral!(CommonType!(B, E)) || isPointer!(CommonType!(B, E))) { import std.conv : unsigned; //... //The return type of length. When either begin or end //is ulong, then IndexType == ulong alias IndexType = typeof(unsigned(end - begin)); static struct Result { private Value current, pastLast; //... property IndexType length() const { return unsigned(pastLast - current); } } return Result(begin, end); } And because std.parallelism.TaskPool.defaultWorkUnitSize takes a size_t, which with a 32-bit DMD is uint. What I want to know is, is this considered a bug? If so I will submit a pull request to fix it.
Feb 11 2016
On 12/02/16 6:51 PM, Meta wrote:If you try to compile this code, it will currently not work: foreach (n; iota(1UL, 1000).parallel) { //... } This is because of how the length is calculated by iota: auto iota(B, E)(B begin, E end) if (isIntegral!(CommonType!(B, E)) || isPointer!(CommonType!(B, E))) { import std.conv : unsigned; //... //The return type of length. When either begin or end //is ulong, then IndexType == ulong alias IndexType = typeof(unsigned(end - begin)); static struct Result { private Value current, pastLast; //... property IndexType length() const { return unsigned(pastLast - current); } } return Result(begin, end); } And because std.parallelism.TaskPool.defaultWorkUnitSize takes a size_t, which with a 32-bit DMD is uint. What I want to know is, is this considered a bug? If so I will submit a pull request to fix it.So I assume if you compiled as 64bit this will work then?
Feb 11 2016
On Friday, 12 February 2016 at 05:59:32 UTC, Rikki Cattermole wrote:So I assume if you compiled as 64bit this will work then?Yes, because `unsigned(begin - end)` will yield a ulong regardless, but size_t will then be a ulong as well.
Feb 11 2016
On 12/02/16 7:19 PM, Meta wrote:On Friday, 12 February 2016 at 05:59:32 UTC, Rikki Cattermole wrote:So an edge case.So I assume if you compiled as 64bit this will work then?Yes, because `unsigned(begin - end)` will yield a ulong regardless, but size_t will then be a ulong as well.
Feb 11 2016
On Friday, 12 February 2016 at 06:21:45 UTC, Rikki Cattermole wrote:On 12/02/16 7:19 PM, Meta wrote:Yes, it's an edge case, although it's a pretty easy one to run into.On Friday, 12 February 2016 at 05:59:32 UTC, Rikki Cattermole wrote:So an edge case.So I assume if you compiled as 64bit this will work then?Yes, because `unsigned(begin - end)` will yield a ulong regardless, but size_t will then be a ulong as well.
Feb 11 2016
On Friday, 12 February 2016 at 05:51:34 UTC, Meta wrote:If you try to compile this code, it will currently not work: foreach (n; iota(1UL, 1000).parallel) { //... } This is because of how the length is calculated by iota: auto iota(B, E)(B begin, E end) if (isIntegral!(CommonType!(B, E)) || isPointer!(CommonType!(B, E))) { import std.conv : unsigned; //... //The return type of length. When either begin or end //is ulong, then IndexType == ulong alias IndexType = typeof(unsigned(end - begin)); static struct Result { private Value current, pastLast; //... property IndexType length() const { return unsigned(pastLast - current); } } return Result(begin, end); } And because std.parallelism.TaskPool.defaultWorkUnitSize takes a size_t, which with a 32-bit DMD is uint. What I want to know is, is this considered a bug? If so I will submit a pull request to fix it.What it comes down to is that length should always be size_t. That's what it is for arrays, and that's what most code expects. Allowing other types just causes trouble for generic code. However, in the case of iota with long, if length is size_t, then on 32-bit systems, it's possible to have a range from iota which is longer than size_t can represent (much as it would normally be crazy to have a range that long). So, at some point, someone made it so that iota uses ulong for length instead of size_t when it's a range of longs or ulongs. It's the only thing in Phobos that does, and it causes problems. Changing it back to size_t has been discussed but not agreed upon. But we're between a rock and a hard place with this one. There is no clean solution. Personally, I'd very much like to see iota just always use size_t for length like every other range (the only ranges which would be affected would be ludicrously long anyway, and it would only affect 32-bit programs). But that hasn't happened yet, so iota over longs and ulongs doesn't behave nicely on 32-bit systems. Regardless of which way we go, the problem will _eventually_ go away when 32-bit systems finally die out, but that's likely to take a while. - Jonathan M Davis
Feb 12 2016
On Friday, 12 February 2016 at 08:11:57 UTC, Jonathan M Davis wrote:Regardless of which way we go, the problem will _eventually_ go away when 32-bit systems finally die out, but that's likely to take a while.Many micro-controllers will probably remain 32-bit indefinitely; 64-bit is just a waste of power for most applications. Even if you want to ignore non-PC/smartphone CPUs, we'll still have the same problem whenever someone gets around to implementing 128-bit cent/ucent. Ultimately, the only real solution to this kind of problem is some combination of: 1) Select the right type for the job (don't use `ulong` when you really mean `size_t`), and 2) Bounds checking. If iota's length is not expressible as a `size_t`, convert it to one *safely* using `std.conv.to()`. Anything less is just asking for bugs.
Feb 12 2016
On Friday, 12 February 2016 at 10:34:32 UTC, tsbockman wrote:Bounds checking. If iota's length is not expressible as a `size_t`, convert it to one *safely* using `std.conv.to()`.Just to clarify - the bit about `std.conv.to()` is just an example. A better solution in this specific case would be to check once, while the iota() range is being constructed, whether its length is expressible as a `size_t` or not.
Feb 12 2016
On Friday, 12 February 2016 at 10:37:31 UTC, tsbockman wrote:On Friday, 12 February 2016 at 10:34:32 UTC, tsbockman wrote:Yeah. I think that that's the best approach, but it does unfortunately have the downside that you pretty much can't have a range over all of the longs or ulongs on 32-bit systems (though fortunately that should rarely be an issue, and I don't really see a fix for that that doesn't cause problems with length and other ranges). - Jonathan M DavisBounds checking. If iota's length is not expressible as a `size_t`, convert it to one *safely* using `std.conv.to()`.Just to clarify - the bit about `std.conv.to()` is just an example. A better solution in this specific case would be to check once, while the iota() range is being constructed, whether its length is expressible as a `size_t` or not.
Feb 12 2016
On Friday, 12 February 2016 at 08:11:57 UTC, Jonathan M Davis wrote:What it comes down to is that length should always be size_t. That's what it is for arrays, and that's what most code expects. Allowing other types just causes trouble for generic code. However, in the case of iota with long, if length is size_t, then on 32-bit systems, it's possible to have a range from iota which is longer than size_t can represent (much as it would normally be crazy to have a range that long). So, at some point, someone made it so that iota uses ulong for length instead of size_t when it's a range of longs or ulongs. It's the only thing in Phobos that does, and it causes problems. Changing it back to size_t has been discussed but not agreed upon. But we're between a rock and a hard place with this one. There is no clean solution. Personally, I'd very much like to see iota just always use size_t for length like every other range (the only ranges which would be affected would be ludicrously long anyway, and it would only affect 32-bit programs). But that hasn't happened yet, so iota over longs and ulongs doesn't behave nicely on 32-bit systems. Regardless of which way we go, the problem will _eventually_ go away when 32-bit systems finally die out, but that's likely to take a while. - Jonathan M DavisWhat about adding another overload of iota, say, iotaEx or something along those lines. All it would do is the following: auto iotaEx(B, E)(B begin, E end) { assert(unsigned(end - begin) <= size_t.max); static struct Result { typeof(iota(begin, end)) payload; property size_t length() { return cast(size_t)payload.length; } alias payload this; } return Result(iota(begin, end)); }
Feb 12 2016
On Friday, 12 February 2016 at 14:36:29 UTC, Meta wrote:On Friday, 12 February 2016 at 08:11:57 UTC, Jonathan M Davis wrote:It would be far better IMHO to just do a check in iota and throw a RangeError if the length wouldn't fit in size_t. Having length ever be anything other than size_t is just going to cause problems with other ranges. On 32-bit systems, you lose out on the ability to have a range that covers all values of long or ulong, but that's of very limited usefulness anyway, and as long as the number of elements is no greater than size_t.max, it would be fine - which would cover virtually all use cases. No, it's not perfect, but allowing length to be anything but size_t just causes bugs - especially in generic code. - Jonathan M DavisWhat it comes down to is that length should always be size_t. That's what it is for arrays, and that's what most code expects. Allowing other types just causes trouble for generic code. However, in the case of iota with long, if length is size_t, then on 32-bit systems, it's possible to have a range from iota which is longer than size_t can represent (much as it would normally be crazy to have a range that long). So, at some point, someone made it so that iota uses ulong for length instead of size_t when it's a range of longs or ulongs. It's the only thing in Phobos that does, and it causes problems. Changing it back to size_t has been discussed but not agreed upon. But we're between a rock and a hard place with this one. There is no clean solution. Personally, I'd very much like to see iota just always use size_t for length like every other range (the only ranges which would be affected would be ludicrously long anyway, and it would only affect 32-bit programs). But that hasn't happened yet, so iota over longs and ulongs doesn't behave nicely on 32-bit systems. Regardless of which way we go, the problem will _eventually_ go away when 32-bit systems finally die out, but that's likely to take a while. - Jonathan M DavisWhat about adding another overload of iota, say, iotaEx or something along those lines. All it would do is the following: auto iotaEx(B, E)(B begin, E end) { assert(unsigned(end - begin) <= size_t.max); static struct Result { typeof(iota(begin, end)) payload; property size_t length() { return cast(size_t)payload.length; } alias payload this; } return Result(iota(begin, end)); }
Feb 12 2016
On 2/12/16 10:59 AM, Jonathan M Davis wrote:On Friday, 12 February 2016 at 14:36:29 UTC, Meta wrote:Just because you have a range with a ulong length doesn't mean you are going to iterate the entire thing. What we need is this: standardLength(R)(R r) if (isInputRange!R && hasLength!R) { static if(is(typeof(r.length) == size_t)) return r; else { static struct Result { R wrapped; size_t length() { return to!size_t(wrapped.length); } alias wrapped this; } return Result(r); } } -SteveOn Friday, 12 February 2016 at 08:11:57 UTC, Jonathan M Davis wrote:It would be far better IMHO to just do a check in iota and throw a RangeError if the length wouldn't fit in size_t. Having length ever be anything other than size_t is just going to cause problems with other ranges. On 32-bit systems, you lose out on the ability to have a range that covers all values of long or ulong, but that's of very limited usefulness anyway, and as long as the number of elements is no greater than size_t.max, it would be fine - which would cover virtually all use cases. No, it's not perfect, but allowing length to be anything but size_t just causes bugs - especially in generic code.What it comes down to is that length should always be size_t. That's what it is for arrays, and that's what most code expects. Allowing other types just causes trouble for generic code. However, in the case of iota with long, if length is size_t, then on 32-bit systems, it's possible to have a range from iota which is longer than size_t can represent (much as it would normally be crazy to have a range that long). So, at some point, someone made it so that iota uses ulong for length instead of size_t when it's a range of longs or ulongs. It's the only thing in Phobos that does, and it causes problems. Changing it back to size_t has been discussed but not agreed upon. But we're between a rock and a hard place with this one. There is no clean solution. Personally, I'd very much like to see iota just always use size_t for length like every other range (the only ranges which would be affected would be ludicrously long anyway, and it would only affect 32-bit programs). But that hasn't happened yet, so iota over longs and ulongs doesn't behave nicely on 32-bit systems. Regardless of which way we go, the problem will _eventually_ go away when 32-bit systems finally die out, but that's likely to take a while. - Jonathan M DavisWhat about adding another overload of iota, say, iotaEx or something along those lines. All it would do is the following: auto iotaEx(B, E)(B begin, E end) { assert(unsigned(end - begin) <= size_t.max); static struct Result { typeof(iota(begin, end)) payload; property size_t length() { return cast(size_t)payload.length; } alias payload this; } return Result(iota(begin, end)); }
Feb 12 2016
On Friday, 12 February 2016 at 15:59:09 UTC, Jonathan M Davis wrote:It would be far better IMHO to just do a check in iota and throw a RangeError if the length wouldn't fit in size_t. Having length ever be anything other than size_t is just going to cause problems with other ranges. On 32-bit systems, you lose out on the ability to have a range that covers all values of long or ulong, but that's of very limited usefulness anyway, and as long as the number of elements is no greater than size_t.max, it would be fine - which would cover virtually all use cases. No, it's not perfect, but allowing length to be anything but size_t just causes bugs - especially in generic code. - Jonathan M DavisWhat about a template overload where you can set the length type as separate from the type of the range elements?
Feb 12 2016
On Friday, 12 February 2016 at 16:52:17 UTC, ixid wrote:On Friday, 12 February 2016 at 15:59:09 UTC, Jonathan M Davis wrote:No other range has anything but size_t for its length. It's what arrays use. It's what the container types use. Code in general is going to assume that length is size_t. I think that having any range types with a length type of anything but size_t is a mistake. It interacts badly with everything else. The post that started this thread is just one example of that. - Jonathan M DavisIt would be far better IMHO to just do a check in iota and throw a RangeError if the length wouldn't fit in size_t. Having length ever be anything other than size_t is just going to cause problems with other ranges. On 32-bit systems, you lose out on the ability to have a range that covers all values of long or ulong, but that's of very limited usefulness anyway, and as long as the number of elements is no greater than size_t.max, it would be fine - which would cover virtually all use cases. No, it's not perfect, but allowing length to be anything but size_t just causes bugs - especially in generic code. - Jonathan M DavisWhat about a template overload where you can set the length type as separate from the type of the range elements?
Feb 12 2016
On Fri, Feb 12, 2016 at 05:38:40PM +0000, Jonathan M Davis via Digitalmars-d wrote:On Friday, 12 February 2016 at 16:52:17 UTC, ixid wrote:[...] IMO, fixing a specific type for length is a mistake. Truly generic code should treat the length as an opaque type (possibly satisfying some constraints, like <-comparable, ==-comparable, and maybe supporting basic arithmetic operations), and should use generic templates like CommonType!(...) if it needs to work with multiple length types. Restricting length to size_t or ulong or whatever, is an arbitrary restriction on ranges, because conceptually you can have ranges whose length exceeds the size of all native integral types (e.g., BigInt lengths), but certain length computations may still make sense, and the range is still usable as long as you only ever operate on a small segment of it. Of course, working with an opaque length type makes it harder to write correct code, but such is the price of generic programming. T -- Debian GNU/Linux: Cray on your desktop.On Friday, 12 February 2016 at 15:59:09 UTC, Jonathan M Davis wrote:No other range has anything but size_t for its length. It's what arrays use. It's what the container types use. Code in general is going to assume that length is size_t. I think that having any range types with a length type of anything but size_t is a mistake. It interacts badly with everything else. The post that started this thread is just one example of that.It would be far better IMHO to just do a check in iota and throw a RangeError if the length wouldn't fit in size_t. Having length ever be anything other than size_t is just going to cause problems with other ranges. On 32-bit systems, you lose out on the ability to have a range that covers all values of long or ulong, but that's of very limited usefulness anyway, and as long as the number of elements is no greater than size_t.max, it would be fine - which would cover virtually all use cases. No, it's not perfect, but allowing length to be anything but size_t just causes bugs - especially in generic code. - Jonathan M DavisWhat about a template overload where you can set the length type as separate from the type of the range elements?
Feb 12 2016
On 02/12/2016 12:58 PM, H. S. Teoh via Digitalmars-d wrote:IMO, fixing a specific type for length is a mistake. Truly generic code should treat the length as an opaque type (possibly satisfying some constraints, like <-comparable, ==-comparable, and maybe supporting basic arithmetic operations), and should use generic templates like CommonType!(...) if it needs to work with multiple length types.Tried that for a while, it quickly became a mess of entangled intestines. Not worth it. -- Andrei
Feb 12 2016
On Fri, Feb 12, 2016 at 02:00:29PM -0500, Andrei Alexandrescu via Digitalmars-d wrote:On 02/12/2016 12:58 PM, H. S. Teoh via Digitalmars-d wrote:The intestines are still visible in iota(), and anything that uses it in 32-bit. T -- This is a tpyo.IMO, fixing a specific type for length is a mistake. Truly generic code should treat the length as an opaque type (possibly satisfying some constraints, like <-comparable, ==-comparable, and maybe supporting basic arithmetic operations), and should use generic templates like CommonType!(...) if it needs to work with multiple length types.Tried that for a while, it quickly became a mess of entangled intestines. Not worth it. -- Andrei
Feb 12 2016
On 02/12/2016 02:10 PM, H. S. Teoh via Digitalmars-d wrote:The intestines are still visible in iota(), and anything that uses it in 32-bit.Do we want them everywhere? -- Andrei
Feb 12 2016
On Friday, 12 February 2016 at 19:00:29 UTC, Andrei Alexandrescu wrote:Tried that for a while, it quickly became a mess of entangled intestines. Not worth it. -- AndreiWhat do you think of the solution I proposed above? It's not pretty but there has to be some kind of workaround. Not being able to use `parallel` with an iota-range of ulongs on 32-bit DMD is not good. If you will pre-approve it (comments/suggestions optional) I can submit a PR within the next day or so.
Feb 12 2016
On 02/12/2016 02:18 PM, Meta wrote:On Friday, 12 February 2016 at 19:00:29 UTC, Andrei Alexandrescu wrote:I'd be more partial to an additional template parameter. Or even change the type of length in certain cases. -- AndreiTried that for a while, it quickly became a mess of entangled intestines. Not worth it. -- AndreiWhat do you think of the solution I proposed above? It's not pretty but there has to be some kind of workaround. Not being able to use `parallel` with an iota-range of ulongs on 32-bit DMD is not good. If you will pre-approve it (comments/suggestions optional) I can submit a PR within the next day or so.
Feb 12 2016
On Friday, 12 February 2016 at 19:20:29 UTC, Andrei Alexandrescu wrote:On 02/12/2016 02:18 PM, Meta wrote:Okay, I will look into other solutions and post some options soon.On Friday, 12 February 2016 at 19:00:29 UTC, Andrei Alexandrescu wrote:I'd be more partial to an additional template parameter. Or even change the type of length in certain cases. -- AndreiTried that for a while, it quickly became a mess of entangled intestines. Not worth it. -- AndreiWhat do you think of the solution I proposed above? It's not pretty but there has to be some kind of workaround. Not being able to use `parallel` with an iota-range of ulongs on 32-bit DMD is not good. If you will pre-approve it (comments/suggestions optional) I can submit a PR within the next day or so.
Feb 12 2016
Maybe I'm missing something, but can't the length just be size_t? I doubt there is much you could do with code which generates finite sequences larger than the addressable memory space, aside from very abstract and inefficient mathematical calculations which skip over elements. iota would probably work better with size_t in most cases, and if you really well and truly need something which generates finite sequences of integers larger than the addressable memory space, you can always just write your own version.
Feb 14 2016
On Sunday, 14 February 2016 at 10:59:41 UTC, w0rp wrote:Maybe I'm missing something, but can't the length just be size_t? I doubt there is much you could do with code which generates finite sequences larger than the addressable memory space, aside from very abstract and inefficient mathematical calculations which skip over elements. iota would probably work better with size_t in most cases, and if you really well and truly need something which generates finite sequences of integers larger than the addressable memory space, you can always just write your own version.Yeah. It would be easy enough to just make iota size_t and then have it check the actual length of the range when it's called to make sure that it's not larger than will fit in size_t (and presumably throw a RangeError if the number of elements is too large). Anyone who really wants to have a range with more than size_t.max elements can write their own thing. It's not like it's going to work nicely with other ranges anyway - at least not as long as it defines length. - Jonathan M Davis
Feb 14 2016
On Sunday, 14 February 2016 at 13:16:58 UTC, Jonathan M Davis wrote:On Sunday, 14 February 2016 at 10:59:41 UTC, w0rp wrote:+1Maybe I'm missing something, but can't the length just be size_t? I doubt there is much you could do with code which generates finite sequences larger than the addressable memory space, aside from very abstract and inefficient mathematical calculations which skip over elements. iota would probably work better with size_t in most cases, and if you really well and truly need something which generates finite sequences of integers larger than the addressable memory space, you can always just write your own version.Yeah. It would be easy enough to just make iota size_t and then have it check the actual length of the range when it's called to make sure that it's not larger than will fit in size_t (and presumably throw a RangeError if the number of elements is too large). Anyone who really wants to have a range with more than size_t.max elements can write their own thing. It's not like it's going to work nicely with other ranges anyway - at least not as long as it defines length. - Jonathan M Davis
Feb 14 2016
On Friday, 12 February 2016 at 05:51:34 UTC, Meta wrote:If you try to compile this code, it will currently not work: foreach (n; iota(1UL, 1000).parallel) { //... } This is because of how the length is calculated by iota: auto iota(B, E)(B begin, E end) if (isIntegral!(CommonType!(B, E)) || isPointer!(CommonType!(B, E))) { import std.conv : unsigned; //... //The return type of length. When either begin or end //is ulong, then IndexType == ulong alias IndexType = typeof(unsigned(end - begin)); static struct Result { private Value current, pastLast; //... property IndexType length() const { return unsigned(pastLast - current); } } return Result(begin, end); } And because std.parallelism.TaskPool.defaultWorkUnitSize takes a size_t, which with a 32-bit DMD is uint. What I want to know is, is this considered a bug? If so I will submit a pull request to fix it.I made a pull request: https://github.com/D-Programming-Language/phobos/pull/4013
Feb 21 2016