www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - endsWith() doesn't work with const(wchar[])s

reply %u <wfunction hotmail.com> writes:
Hi,

I've noticed that some functions, such as algorithm.endsWith, don't work with
constant arrays. Is this a bug, or is there a reason behind it? It forces the
user to perform dangerous casts to avoid object creation, and it doesn't seem
like the functions actually need to perform any manipulations.

Thank you!
Jan 13 2011
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday 13 January 2011 21:21:06 %u wrote:
 Hi,
 
 I've noticed that some functions, such as algorithm.endsWith, don't work
 with constant arrays. Is this a bug, or is there a reason behind it? It
 forces the user to perform dangerous casts to avoid object creation, and
 it doesn't seem like the functions actually need to perform any
 manipulations.

Phobos doesn't really deal with const or immutable correctly at this point. A number of things which should be able to handle const or immutable can't. And then there are things which you'd think _should_ be able to but can't because of the transivity of const and immutable. There are a number of outstanding bugs related to const and immutable which makes dealing with them at times a bit of a pain if not outright impossible. It's on the list of things to be focused on after the 64-bit port of dmd is done. As it stands, there are a number of algorithms which just won't work with const or immutable arrays. Regardless, a fully const array is never going to work with a function like endsWith() for the simple reason that such functions have to actually be able to process the range that they're given, and if the range is const, you can't call popFront() or popBack() on it, so it just isn't going to work. Now, if you take a _slice_ of a const array, it should work, because while the elements of the array will remain const, the slice itself won't be, so endsWith() can process it. - Jonathan M Davis
Jan 13 2011
next sibling parent reply %u <wfunction hotmail.com> writes:
 Regardless, a fully const array is never going to work with a function like

process the range that they're given, and if the range is const, you can't call popFront() or popBack() on it, so it just isn't going to work.
 Now, if you take a _slice_ of a const array, it should work, because while the

can process it. I see what's happening, but the problem with slicing is that it doesn't prevent a resize of the array, which, if not allocated with GC memory, could break code. How can you have a non-resizeable slice, while still maintaining the constness of an array? Is that even possible? Thank you!
Jan 13 2011
parent %u <wfunction hotmail.com> writes:
 What you really want is for IFTI to strip down const/immutable on arrays and

discussed on the phobos mailing list. Ah, all right then, nice!
 Since 2.041, this cannot happen.  A resize of a slice where data will be

So I assume the same is true if you try to resize a slice of non-GC-managed memory (even if it won't overwrite anything), correct? That's comforting to know, thank you! :)
Jan 14 2011
prev sibling parent reply Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
Jonathan M Davis wrote:

 On Thursday 13 January 2011 21:21:06 %u wrote:
 Hi,
 
 I've noticed that some functions, such as algorithm.endsWith, don't work
 with constant arrays. Is this a bug, or is there a reason behind it? It
 forces the user to perform dangerous casts to avoid object creation, and
 it doesn't seem like the functions actually need to perform any
 manipulations.

Phobos doesn't really deal with const or immutable correctly at this point. A number of things which should be able to handle const or immutable can't. And then there are things which you'd think _should_ be able to but can't because of the transivity of const and immutable. There are a number of outstanding bugs related to const and immutable which makes dealing with them at times a bit of a pain if not outright impossible. It's on the list of things to be focused on after the 64-bit port of dmd is done. As it stands, there are a number of algorithms which just won't work with const or immutable arrays. Regardless, a fully const array is never going to work with a function like endsWith() for the simple reason that such functions have to actually be able to process the range that they're given, and if the range is const, you can't call popFront() or popBack() on it, so it just isn't going to work. Now, if you take a _slice_ of a const array, it should work, because while the elements of the array will remain const, the slice itself won't be, so endsWith() can process it. - Jonathan M Davis

A slice won't work because it is still const(T[]). It can implicitly convert to const(T)[], but I believe that is a bug currently.
Jan 15 2011
parent Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
Jonathan M Davis wrote:

 On Saturday 15 January 2011 09:02:55 Lutger Blijdestijn wrote:
 Jonathan M Davis wrote:
 On Thursday 13 January 2011 21:21:06 %u wrote:
 Hi,
 
 I've noticed that some functions, such as algorithm.endsWith, don't
 work with constant arrays. Is this a bug, or is there a reason behind
 it? It forces the user to perform dangerous casts to avoid object
 creation, and it doesn't seem like the functions actually need to
 perform any manipulations.

Phobos doesn't really deal with const or immutable correctly at this point. A number of things which should be able to handle const or immutable can't. And then there are things which you'd think _should_ be able to but can't because of the transivity of const and immutable. There are a number of outstanding bugs related to const and immutable which makes dealing with them at times a bit of a pain if not outright impossible. It's on the list of things to be focused on after the 64-bit port of dmd is done. As it stands, there are a number of algorithms which just won't work with const or immutable arrays. Regardless, a fully const array is never going to work with a function like endsWith() for the simple reason that such functions have to actually be able to process the range that they're given, and if the range is const, you can't call popFront() or popBack() on it, so it just isn't going to work. Now, if you take a _slice_ of a const array, it should work, because while the elements of the array will remain const, the slice itself won't be, so endsWith() can process it. - Jonathan M Davis

A slice won't work because it is still const(T[]). It can implicitly convert to const(T)[], but I believe that is a bug currently.

No, I'm pretty sure that that isn't a bug. A slice is a new array. A const(T)[] that points to the same elements of an array that's const(T[]) can't change the elements any more than the const(T[]) can. So, const is not violated for the elements. And since the slice is a new array, the fact that you can alter the slice itself is completely valid. You can resize the slice or set to an entirely different block of memory or whatever, at it won't affect the original array, so it does not violate the const-ness of the const(T[]) array.

Yes, I agree the behavior is correct, it's indeed the inconsistency I was referring to. At this moment I'm not sure how it is gonna work out. I failed searching bugzilla, but relying on this behavior would make me a bit nervous. About the slicing, this works: const(int[]) n = [1,2,3]; const(int)[] a = n[]; assert(endsWith(a, 3)); but this doesn't: auto b = n[]; assert(endsWith(b, 3)); In conclusion, you still have to 'convert' the array to a different type, even though casting is not required for arrays (and arrays only). otoh, treating a const array as a range doesn't make any sense in the first place, since ranges must be mutable by definition. Arrays *are* special in the sense that they act both as a container and 'iterator' over that container.
 So, no, it's not a bug. The bug is that this sort of behavior doesn't work
 for general ranges - just arrays - and that we don't currently have a
 means of making it work with general ranges. It cripples const and
 immutable ranges, but it doesn't cripple const and immutable arrays.
 
 - Jonathan M Davis

Jan 16 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday 13 January 2011 23:21:03 %u wrote:
 Regardless, a fully const array is never going to work with a function
 like

endsWith() for the simple reason that such functions have to actually be able to process the range that they're given, and if the range is const, you can't call popFront() or popBack() on it, so it just isn't going to work.
 Now, if you take a _slice_ of a const array, it should work, because
 while the

elements of the array will remain const, the slice itself won't be, so endsWith() can process it. I see what's happening, but the problem with slicing is that it doesn't prevent a resize of the array, which, if not allocated with GC memory, could break code. How can you have a non-resizeable slice, while still maintaining the constness of an array? Is that even possible?

Well, if you're talking about anything in std.algorithm, it's dealing in ranges, not arrays, so it's not going to resize anything. In fact, very little in Phobos would end up resizing an array. Most stuff deals explicitly with ranges. I doubt that anything in std.string does either. But anything there would either create a straight up slice or create a new string, which would be freshly allocated by the GC. And actually, even if a slice were resized, I'm not sure that it _would_ cause any problems. I'm not sure what would happen if it were to try and resize in place (it probably couldn't since, a non-GC-allocated array likely wouldn't have any extra capacity), but if it couldn't then it would have to allocate a new array, which would then be GC-allocated. The original slice would have just pointed to the original array, so that wouldn't cause a leak. You'd still have to worry about cleaning up the original array, but any new ones which were created would be GC-allocated and then definitely wouldn't be leaking anything. Of course, in the general case, I wouldn't really advise manually allocating arrays unless you really need to. Unless profiling shows that the GC-allocated arrays are a bottleneck, it's just simpler and safer to allow the GC to do its job. The one exception to that would be if the array in question came from a C call or something similar where it couldn't be GC-allocated. But generally, range-based functions shouldn't be allocating new arrays. The closest that I can think of would be save() on Forward Ranges, and I believe that that's a slice for arrays, so nothing would be allocated. In any case, very little range-based anything is going to be able to work on fully const ranges. It just isn't possible. However, very little that's range- based would be doing any kind of reallocation of arrays - _especially_ something like endsWith(). I don't think that even std.string does much in the way of array reallocation - if any at all. Still, if you're really trying to use arrays which aren't GC-allocated, the safest way would be to just use functions that you write yourself. While it's possible to use non-GC-allocated arrays, the language isn't particularly friendly towards them. They're really supposed to be allocated by the GC. Still, I believe that the functions in std.algorithm should work just fine without doing any reallocating, since they all work on ranges, not arrays. - Jonathan M Davis
Jan 14 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Jan 2011 02:21:03 -0500, %u <wfunction hotmail.com> wrote:

 Regardless, a fully const array is never going to work with a function  
 like

able to process the range that they're given, and if the range is const, you can't call popFront() or popBack() on it, so it just isn't going to work.
 Now, if you take a _slice_ of a const array, it should work, because  
 while the

endsWith() can process it.

The issue is that IFTI uses exactly the type used to call it. void foo(T)(T t) { writeln(typeid(T)); } const(int[]) x = [1,2,3]; foo(x); // prints const(int[]) What you really want is for IFTI to strip down const/immutable on arrays and basic types. I think it's a bugzilla bug somewhere, if not it's definitely been discussed on the phobos mailing list.
 I see what's happening, but the problem with slicing is that it doesn't  
 prevent a
 resize of the array, which, if not allocated with GC memory, could break  
 code. How
 can you have a non-resizeable slice, while still maintaining the  
 constness of an
 array? Is that even possible?

Since 2.041, this cannot happen. A resize of a slice where data will be overwritten from the original array always results in a reallocation. -Steve
Jan 14 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Jan 2011 20:54:39 -0500, %u <wfunction hotmail.com> wrote:

 What you really want is for IFTI to strip down const/immutable on  
 arrays and

definitely been discussed on the phobos mailing list. Ah, all right then, nice!
 Since 2.041, this cannot happen.  A resize of a slice where data will be

So I assume the same is true if you try to resize a slice of non-GC-managed memory (even if it won't overwrite anything), correct? That's comforting to know, thank you! :)

A resize of non-GC-managed memory will always reallocate on the GC, because it has no information on it. -Steve
Jan 15 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 15 Jan 2011 12:16:32 -0500, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Fri, 14 Jan 2011 20:54:39 -0500, %u <wfunction hotmail.com> wrote:

 What you really want is for IFTI to strip down const/immutable on  
 arrays and

definitely been discussed on the phobos mailing list. Ah, all right then, nice!
 Since 2.041, this cannot happen.  A resize of a slice where data will  
 be

So I assume the same is true if you try to resize a slice of non-GC-managed memory (even if it won't overwrite anything), correct? That's comforting to know, thank you! :)

A resize of non-GC-managed memory will always reallocate on the GC, because it has no information on it.

Actually, I'll go one further: A resize of non-GC-array memory (memory allocated in any way other than new T[]) will not resize in place, simply because the information to allow resizing doesn't exist. This change is more recent however, I think circa 2.048? -Steve
Jan 15 2011
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 15 January 2011 09:02:55 Lutger Blijdestijn wrote:
 Jonathan M Davis wrote:
 On Thursday 13 January 2011 21:21:06 %u wrote:
 Hi,
 
 I've noticed that some functions, such as algorithm.endsWith, don't work
 with constant arrays. Is this a bug, or is there a reason behind it? It
 forces the user to perform dangerous casts to avoid object creation, and
 it doesn't seem like the functions actually need to perform any
 manipulations.

Phobos doesn't really deal with const or immutable correctly at this point. A number of things which should be able to handle const or immutable can't. And then there are things which you'd think _should_ be able to but can't because of the transivity of const and immutable. There are a number of outstanding bugs related to const and immutable which makes dealing with them at times a bit of a pain if not outright impossible. It's on the list of things to be focused on after the 64-bit port of dmd is done. As it stands, there are a number of algorithms which just won't work with const or immutable arrays. Regardless, a fully const array is never going to work with a function like endsWith() for the simple reason that such functions have to actually be able to process the range that they're given, and if the range is const, you can't call popFront() or popBack() on it, so it just isn't going to work. Now, if you take a _slice_ of a const array, it should work, because while the elements of the array will remain const, the slice itself won't be, so endsWith() can process it. - Jonathan M Davis

A slice won't work because it is still const(T[]). It can implicitly convert to const(T)[], but I believe that is a bug currently.

No, I'm pretty sure that that isn't a bug. A slice is a new array. A const(T)[] that points to the same elements of an array that's const(T[]) can't change the elements any more than the const(T[]) can. So, const is not violated for the elements. And since the slice is a new array, the fact that you can alter the slice itself is completely valid. You can resize the slice or set to an entirely different block of memory or whatever, at it won't affect the original array, so it does not violate the const-ness of the const(T[]) array. So, no, it's not a bug. The bug is that this sort of behavior doesn't work for general ranges - just arrays - and that we don't currently have a means of making it work with general ranges. It cripples const and immutable ranges, but it doesn't cripple const and immutable arrays. - Jonathan M Davis
Jan 15 2011