www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - endsWith() doesn't work with const(wchar[])s

reply %u <wfunction hotmail.com> writes:
Hi,

I've noticed that some functions, such as algorithm.endsWith, don't work with
constant arrays. Is this a bug, or is there a reason behind it? It forces the
user to perform dangerous casts to avoid object creation, and it doesn't seem
like the functions actually need to perform any manipulations.

Thank you!
Jan 13 2011
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday 13 January 2011 21:21:06 %u wrote:

 Hi,
 
 I've noticed that some functions, such as algorithm.endsWith, don't work
 with constant arrays. Is this a bug, or is there a reason behind it? It
 forces the user to perform dangerous casts to avoid object creation, and
 it doesn't seem like the functions actually need to perform any
 manipulations.


Phobos doesn't really deal with const or immutable correctly at this point. A 
number of things which should be able to handle const or immutable can't. And 
then there are things which you'd think _should_ be able to but can't because
of 
the transivity of const and immutable. There are a number of outstanding bugs 
related to const and immutable which makes dealing with them at times a bit of
a 
pain if not outright impossible. It's on the list of things to be focused on 
after the 64-bit port of dmd is done.

As it stands, there are a number of algorithms which just won't work with const 
or immutable arrays.

Regardless, a fully const array is never going to work with a function like 
endsWith() for the simple reason that such functions have to actually be able
to 
process the range that they're given, and if the range is const, you can't call 
popFront() or popBack() on it, so it just isn't going to work.

Now, if you take a _slice_ of a const array, it should work, because while the 
elements of the array will remain const, the slice itself won't be, so 
endsWith() can process it.

- Jonathan M Davis
Jan 13 2011
next sibling parent reply %u <wfunction hotmail.com> writes:
 Regardless, a fully const array is never going to work with a function like


endsWith() for the simple reason that such functions have to actually be able to
process the range that they're given, and if the range is const, you can't call
popFront() or popBack() on it, so it just isn't going to work.

 Now, if you take a _slice_ of a const array, it should work, because while the


elements of the array will remain const, the slice itself won't be, so
endsWith()
can process it.

I see what's happening, but the problem with slicing is that it doesn't prevent
a
resize of the array, which, if not allocated with GC memory, could break code.
How
can you have a non-resizeable slice, while still maintaining the constness of an
array? Is that even possible?

Thank you!
Jan 13 2011
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday 13 January 2011 23:21:03 %u wrote:

 Regardless, a fully const array is never going to work with a function
 like


 
 endsWith() for the simple reason that such functions have to actually be
 able to process the range that they're given, and if the range is const,
 you can't call popFront() or popBack() on it, so it just isn't going to
 work.
 

 Now, if you take a _slice_ of a const array, it should work, because
 while the


 
 elements of the array will remain const, the slice itself won't be, so
 endsWith() can process it.
 
 I see what's happening, but the problem with slicing is that it doesn't
 prevent a resize of the array, which, if not allocated with GC memory,
 could break code. How can you have a non-resizeable slice, while still
 maintaining the constness of an array? Is that even possible?


Well, if you're talking about anything in std.algorithm, it's dealing in
ranges, 
not arrays, so it's not going to resize anything. In fact, very little in
Phobos 
would end up resizing an array. Most stuff deals explicitly with ranges. I
doubt 
that anything in std.string does either. But anything there would either create 
a straight up slice or create a new string, which would be freshly allocated by 
the GC.

And actually, even if a slice were resized, I'm not sure that it _would_ cause 
any problems. I'm not sure what would happen if it were to try and resize in 
place (it probably couldn't since, a non-GC-allocated array likely wouldn't
have 
any extra capacity), but if it couldn't then it would have to allocate a new 
array, which would then be GC-allocated. The original slice would have just 
pointed to the original array, so that wouldn't cause a leak. You'd still have 
to worry about cleaning up the original array, but any new ones which were 
created would be GC-allocated and then definitely wouldn't be leaking anything.

Of course, in the general case, I wouldn't really advise manually allocating 
arrays unless you really need to. Unless profiling shows that the GC-allocated 
arrays are a bottleneck, it's just simpler and safer to allow the GC to do its 
job. The one exception to that would be if the array in question came from a C 
call or something similar where it couldn't be GC-allocated. But generally, 
range-based functions shouldn't be allocating new arrays. The closest that I
can 
think of would be save() on Forward Ranges, and I believe that that's a slice 
for arrays, so nothing would be allocated.

In any case, very little range-based anything is going to be able to work on 
fully const ranges. It just isn't possible. However, very little that's range-
based would be doing any kind of reallocation of arrays - _especially_
something 
like endsWith(). I don't think that even std.string does much in the way of 
array reallocation - if any at all.

Still, if you're really trying to use arrays which aren't GC-allocated, the 
safest way would be to just use functions that you write yourself. While it's 
possible to use non-GC-allocated arrays, the language isn't particularly 
friendly towards them. They're really supposed to be allocated by the GC.
Still, 
I believe that the functions in std.algorithm should work just fine without
doing 
any reallocating, since they all work on ranges, not arrays.

- Jonathan M Davis
Jan 14 2011
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Jan 2011 02:21:03 -0500, %u <wfunction hotmail.com> wrote:


 Regardless, a fully const array is never going to work with a function  
 like


 endsWith() for the simple reason that such functions have to actually be  
 able to
 process the range that they're given, and if the range is const, you  
 can't call
 popFront() or popBack() on it, so it just isn't going to work.

 Now, if you take a _slice_ of a const array, it should work, because  
 while the


 elements of the array will remain const, the slice itself won't be, so  
 endsWith()
 can process it.


The issue is that IFTI uses exactly the type used to call it.

void foo(T)(T t)
{
    writeln(typeid(T));
}

const(int[]) x = [1,2,3];
foo(x); // prints const(int[])

What you really want is for IFTI to strip down const/immutable on arrays  
and basic types.  I think it's a bugzilla bug somewhere, if not it's  
definitely been discussed on the phobos mailing list.


 I see what's happening, but the problem with slicing is that it doesn't  
 prevent a
 resize of the array, which, if not allocated with GC memory, could break  
 code. How
 can you have a non-resizeable slice, while still maintaining the  
 constness of an
 array? Is that even possible?


Since 2.041, this cannot happen.  A resize of a slice where data will be  
overwritten from the original array always results in a reallocation.

-Steve
Jan 14 2011
parent reply %u <wfunction hotmail.com> writes:
 What you really want is for IFTI to strip down const/immutable on arrays and


basic types.  I think it's a bugzilla bug somewhere, if not it's definitely been
discussed on the phobos mailing list.

Ah, all right then, nice!



 Since 2.041, this cannot happen.  A resize of a slice where data will be


overwritten from the original array always results in a reallocation.

So I assume the same is true if you try to resize a slice of non-GC-managed
memory
(even if it won't overwrite anything), correct? That's comforting to know, thank
you! :)
Jan 14 2011
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Jan 2011 20:54:39 -0500, %u <wfunction hotmail.com> wrote:


 What you really want is for IFTI to strip down const/immutable on  
 arrays and


 basic types.  I think it's a bugzilla bug somewhere, if not it's  
 definitely been
 discussed on the phobos mailing list.

 Ah, all right then, nice!



 Since 2.041, this cannot happen.  A resize of a slice where data will be


 overwritten from the original array always results in a reallocation.

 So I assume the same is true if you try to resize a slice of  
 non-GC-managed memory
 (even if it won't overwrite anything), correct? That's comforting to  
 know, thank
 you! :)


A resize of non-GC-managed memory will always reallocate on the GC,  
because it has no information on it.

-Steve
Jan 15 2011
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 15 Jan 2011 12:16:32 -0500, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 On Fri, 14 Jan 2011 20:54:39 -0500, %u <wfunction hotmail.com> wrote:


 What you really want is for IFTI to strip down const/immutable on  
 arrays and


 basic types.  I think it's a bugzilla bug somewhere, if not it's  
 definitely been
 discussed on the phobos mailing list.

 Ah, all right then, nice!



 Since 2.041, this cannot happen.  A resize of a slice where data will  
 be


 overwritten from the original array always results in a reallocation.

 So I assume the same is true if you try to resize a slice of  
 non-GC-managed memory
 (even if it won't overwrite anything), correct? That's comforting to  
 know, thank
 you! :)


 A resize of non-GC-managed memory will always reallocate on the GC,  
 because it has no information on it.


Actually, I'll go one further: A resize of non-GC-array memory (memory  
allocated in any way other than new T[]) will not resize in place, simply  
because the information to allow resizing doesn't exist.  This change is  
more recent however, I think circa 2.048?

-Steve
Jan 15 2011
prev sibling parent reply Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
Jonathan M Davis wrote:


 On Thursday 13 January 2011 21:21:06 %u wrote:

 Hi,
 
 I've noticed that some functions, such as algorithm.endsWith, don't work
 with constant arrays. Is this a bug, or is there a reason behind it? It
 forces the user to perform dangerous casts to avoid object creation, and
 it doesn't seem like the functions actually need to perform any
 manipulations.


 
 Phobos doesn't really deal with const or immutable correctly at this
 point. A number of things which should be able to handle const or
 immutable can't. And then there are things which you'd think _should_ be
 able to but can't because of the transivity of const and immutable. There
 are a number of outstanding bugs related to const and immutable which
 makes dealing with them at times a bit of a pain if not outright
 impossible. It's on the list of things to be focused on after the 64-bit
 port of dmd is done.
 
 As it stands, there are a number of algorithms which just won't work with
 const or immutable arrays.
 
 Regardless, a fully const array is never going to work with a function
 like endsWith() for the simple reason that such functions have to actually
 be able to process the range that they're given, and if the range is
 const, you can't call popFront() or popBack() on it, so it just isn't
 going to work.
 
 Now, if you take a _slice_ of a const array, it should work, because while
 the elements of the array will remain const, the slice itself won't be, so
 endsWith() can process it.
 
 - Jonathan M Davis


A slice won't work because it is still const(T[]). It can implicitly convert 
to const(T)[], but I believe that is a bug currently.
Jan 15 2011
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 15 January 2011 09:02:55 Lutger Blijdestijn wrote:

 Jonathan M Davis wrote:

 On Thursday 13 January 2011 21:21:06 %u wrote:

 Hi,
 
 I've noticed that some functions, such as algorithm.endsWith, don't work
 with constant arrays. Is this a bug, or is there a reason behind it? It
 forces the user to perform dangerous casts to avoid object creation, and
 it doesn't seem like the functions actually need to perform any
 manipulations.


 
 Phobos doesn't really deal with const or immutable correctly at this
 point. A number of things which should be able to handle const or
 immutable can't. And then there are things which you'd think _should_ be
 able to but can't because of the transivity of const and immutable. There
 are a number of outstanding bugs related to const and immutable which
 makes dealing with them at times a bit of a pain if not outright
 impossible. It's on the list of things to be focused on after the 64-bit
 port of dmd is done.
 
 As it stands, there are a number of algorithms which just won't work with
 const or immutable arrays.
 
 Regardless, a fully const array is never going to work with a function
 like endsWith() for the simple reason that such functions have to
 actually be able to process the range that they're given, and if the
 range is const, you can't call popFront() or popBack() on it, so it just
 isn't going to work.
 
 Now, if you take a _slice_ of a const array, it should work, because
 while the elements of the array will remain const, the slice itself
 won't be, so endsWith() can process it.
 
 - Jonathan M Davis


 
 A slice won't work because it is still const(T[]). It can implicitly
 convert to const(T)[], but I believe that is a bug currently.


No, I'm pretty sure that that isn't a bug. A slice is a new array. A const(T)[] 
that points to the same elements of an array that's const(T[]) can't change the 
elements any more than the const(T[]) can. So, const is not violated for the 
elements. And since the slice is a new array, the fact that you can alter the 
slice itself is completely valid. You can resize the slice or set to an
entirely 
different block of memory or whatever, at it won't affect the original array,
so 
it does not violate the const-ness of the const(T[]) array.

So, no, it's not a bug. The bug is that this sort of behavior doesn't work for 
general ranges - just arrays - and that we don't currently have a means of 
making it work with general ranges. It cripples const and immutable ranges, but 
it doesn't cripple const and immutable arrays.

- Jonathan M Davis
Jan 15 2011
parent Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
Jonathan M Davis wrote:


 On Saturday 15 January 2011 09:02:55 Lutger Blijdestijn wrote:

 Jonathan M Davis wrote:

 On Thursday 13 January 2011 21:21:06 %u wrote:

 Hi,
 
 I've noticed that some functions, such as algorithm.endsWith, don't
 work with constant arrays. Is this a bug, or is there a reason behind
 it? It forces the user to perform dangerous casts to avoid object
 creation, and it doesn't seem like the functions actually need to
 perform any manipulations.


 
 Phobos doesn't really deal with const or immutable correctly at this
 point. A number of things which should be able to handle const or
 immutable can't. And then there are things which you'd think _should_
 be able to but can't because of the transivity of const and immutable.
 There are a number of outstanding bugs related to const and immutable
 which makes dealing with them at times a bit of a pain if not outright
 impossible. It's on the list of things to be focused on after the
 64-bit port of dmd is done.
 
 As it stands, there are a number of algorithms which just won't work
 with const or immutable arrays.
 
 Regardless, a fully const array is never going to work with a function
 like endsWith() for the simple reason that such functions have to
 actually be able to process the range that they're given, and if the
 range is const, you can't call popFront() or popBack() on it, so it
 just isn't going to work.
 
 Now, if you take a _slice_ of a const array, it should work, because
 while the elements of the array will remain const, the slice itself
 won't be, so endsWith() can process it.
 
 - Jonathan M Davis


 
 A slice won't work because it is still const(T[]). It can implicitly
 convert to const(T)[], but I believe that is a bug currently.


 
 No, I'm pretty sure that that isn't a bug. A slice is a new array. A
 const(T)[] that points to the same elements of an array that's const(T[])
 can't change the elements any more than the const(T[]) can. So, const is
 not violated for the elements. And since the slice is a new array, the
 fact that you can alter the slice itself is completely valid. You can
 resize the slice or set to an entirely different block of memory or
 whatever, at it won't affect the original array, so it does not violate
 the const-ness of the const(T[]) array.


Yes, I agree the behavior is correct, it's indeed the inconsistency I was 
referring to. At this moment I'm not sure how it is gonna work out. I failed 
searching bugzilla, but relying on this behavior would make me a bit 
nervous. 

About the slicing, this works:

const(int[]) n = [1,2,3];
const(int)[] a = n[];
assert(endsWith(a, 3));

but this doesn't:

auto b = n[]; 
assert(endsWith(b, 3));

In conclusion, you still have to 'convert' the array to a different type, 
even though casting is not required for arrays (and arrays only).

otoh, treating a const array as a range doesn't make any sense in the first 
place, since ranges must be mutable by definition. Arrays *are* special in 
the sense that they act both as a container and 'iterator' over that 
container. 
 

 So, no, it's not a bug. The bug is that this sort of behavior doesn't work
 for general ranges - just arrays - and that we don't currently have a
 means of making it work with general ranges. It cripples const and
 immutable ranges, but it doesn't cripple const and immutable arrays.
 
 - Jonathan M Davis
Jan 16 2011