www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - typeof(string.front) should be char

reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Hello,

For this code:

     auto c = "test"c;
     auto w = "test"w;
     auto d = "test"d;
     pragma(msg, typeof(c.front));
     pragma(msg, typeof(w.front));
     pragma(msg, typeof(d.front));

compiler prints:

dchar
dchar
immutable(dchar)

IMO it should print this:

immutable(char)
immutable(wchar)
immutable(dchar)

Is it a bug?
Mar 02 2012
next sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 03/02/2012 06:30 PM, Piotr Szturmaj wrote:
 Hello,

 For this code:

 auto c = "test"c;
 auto w = "test"w;
 auto d = "test"d;
 pragma(msg, typeof(c.front));
 pragma(msg, typeof(w.front));
 pragma(msg, typeof(d.front));

 compiler prints:

 dchar
 dchar
 immutable(dchar)

 IMO it should print this:

 immutable(char)
 immutable(wchar)
 immutable(dchar)

 Is it a bug?

No, that's by design. When used as InputRange ranges, slices of any character type are exposed as ranges of dchar. Ali
Mar 02 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, March 02, 2012 20:41:35 Ali =C3=87ehreli wrote:
 On 03/02/2012 06:30 PM, Piotr Szturmaj wrote:
  > Hello,
  >=20
  > For this code:
  >=20
  > auto c =3D "test"c;
  > auto w =3D "test"w;
  > auto d =3D "test"d;
  > pragma(msg, typeof(c.front));
  > pragma(msg, typeof(w.front));
  > pragma(msg, typeof(d.front));
  >=20
  > compiler prints:
  >=20
  > dchar
  > dchar
  > immutable(dchar)
  >=20
  > IMO it should print this:
  >=20
  > immutable(char)
  > immutable(wchar)
  > immutable(dchar)
  >=20
  > Is it a bug?
=20
 No, that's by design. When used as InputRange ranges, slices of any
 character type are exposed as ranges of dchar.

Indeed. Strings are always treated as ranges of dchar, because it generally mak= es no=20 sense to operate on individual chars or wchars. A char is a UTF-8 code = unit. A=20 wchar is a UTF-16 code unit. And a dchar is a UTF-32 code unit. The _on= ly_ one=20 of those which is guranteed to be a code point is dchar, since in UTF-3= 2, all=20 code points are a single code unit. If you were to operate on individua= l chars=20 or wchars, you'd be operating on pieces of characters rather than whole= =20 characters, which wreaks havoc with unicode. Now, technically speaking, a code point isn't necessarily a full charac= ter,=20 since you can also combine code points (e.g. adding a subscript to a le= tter),=20 and a full character is what's called a grapheme, and unfortunately, at= the=20 moment, Phobos doesn't have a way to operate on graphemes, but operatin= g on=20 code points is _far_ more correct than operating on code units. It's al= so more=20 efficient. Unfortunately, in order to code completely efficiently with unicode, yo= u have=20 understand quite a bit about it, which most programmers don't, but by=20= operating on ranges of code points, Phobos manages to be correct in the= =20 majority of cases. So, yes. It's very much on purpose that all strings are treated as rang= es of=20 dchar. - Jonathan M Davis
Mar 03 2012
next sibling parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Jonathan M Davis wrote:
 On Friday, March 02, 2012 20:41:35 Ali Çehreli wrote:
 On 03/02/2012 06:30 PM, Piotr Szturmaj wrote:
   >  Hello,
   >
   >  For this code:
   >
   >  auto c = "test"c;
   >  auto w = "test"w;
   >  auto d = "test"d;
   >  pragma(msg, typeof(c.front));
   >  pragma(msg, typeof(w.front));
   >  pragma(msg, typeof(d.front));
   >
   >  compiler prints:
   >
   >  dchar
   >  dchar
   >  immutable(dchar)
   >
   >  IMO it should print this:
   >
   >  immutable(char)
   >  immutable(wchar)
   >  immutable(dchar)
   >
   >  Is it a bug?

 No, that's by design. When used as InputRange ranges, slices of any
 character type are exposed as ranges of dchar.

Indeed. Strings are always treated as ranges of dchar, because it generally makes no sense to operate on individual chars or wchars. A char is a UTF-8 code unit. A wchar is a UTF-16 code unit. And a dchar is a UTF-32 code unit. The _only_ one of those which is guranteed to be a code point is dchar, since in UTF-32, all code points are a single code unit. If you were to operate on individual chars or wchars, you'd be operating on pieces of characters rather than whole characters, which wreaks havoc with unicode. Now, technically speaking, a code point isn't necessarily a full character, since you can also combine code points (e.g. adding a subscript to a letter), and a full character is what's called a grapheme, and unfortunately, at the moment, Phobos doesn't have a way to operate on graphemes, but operating on code points is _far_ more correct than operating on code units. It's also more efficient. Unfortunately, in order to code completely efficiently with unicode, you have understand quite a bit about it, which most programmers don't, but by operating on ranges of code points, Phobos manages to be correct in the majority of cases.

I know about Unicode, code units/points and their encoding.
 So, yes. It's very much on purpose that all strings are treated as ranges of
 dchar.

Foreach gives opportunity to handle any string by char, wchar or dchar, the default dchar is appropriate here, but why for ranges? I was afraid it is on purpose, because it has some bad consequences. It breaks genericity when dealing with ranges. Consider a custom range of char: struct CharRange { property bool empty(); property char front(); void popFront(); } typeof(CharRange.front) and ElementType!CharRange both return _char_ while for string they return _dchar_. This discrepancy pushes the range writer to handle special string cases. I'm currently trying to write ByDchar range: template ByDchar(R) if (isInputRange!R && isSomeChar!(ElementType!R)) { alias ElementType!R E; static if (is(E == dchar)) alias R ByDchar; else static if (is(E == char)) { struct ByDchar { ... } } else static if (is(E == wchar)) { ... } } The problem with that range is when it takes a string type, it aliases this type with itself, because ElementType!R yields dchar. This is why I'm talking about "bad consequences", I just want to iterate string by _char_, not _dchar_.
Mar 03 2012
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 03/03/2012 05:57 AM, Piotr Szturmaj wrote:
 Consider a custom range of
 char:

 struct CharRange
 {
  property bool empty();
  property char front();
 void popFront();
 }

 typeof(CharRange.front) and ElementType!CharRange both return _char_

Yes, and I would expect both to the same type.
 while for string they return _dchar_. This discrepancy pushes the range
 writer to handle special string cases.

Yes, Phobos faces the same issues.
 I'm currently trying to write
 ByDchar range:

 template ByDchar(R)
 if (isInputRange!R && isSomeChar!(ElementType!R))
 {
 alias ElementType!R E;
 static if (is(E == dchar))
 alias R ByDchar;
 else static if (is(E == char))
 {
 struct ByDchar
 {
 ...
 }
 }
 else static if (is(E == wchar))
 {
 ...
 }
 }

 The problem with that range is when it takes a string type, it aliases
 this type with itself, because ElementType!R yields dchar. This is why
 I'm talking about "bad consequences", I just want to iterate string by
 _char_, not _dchar_.

In case you don't know already, there are std.traits.isNarrowString, std.range.ForEachType, etc. which may be useful. Ali
Mar 03 2012
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 03/03/2012 09:40 AM, Jonathan M Davis wrote:
 ...  but operating on
 code points is _far_ more correct than operating on code units. It's also more
 efficient.
 [snip.]

No, it is less efficient.
Mar 03 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 03/03/2012 08:46 PM, Jonathan M Davis wrote:
 On Saturday, March 03, 2012 18:38:44 Timon Gehr wrote:
 On 03/03/2012 09:40 AM, Jonathan M Davis wrote:
 ...  but operating on
 code points is _far_ more correct than operating on code units. It's also
 more efficient.
 [snip.]

No, it is less efficient.

Operating on code points is more efficient than operating on graphemes is what I meant. I can see that I wasn't clear enough on that.

Makes sense.
 It's more correct than operating on code units and less correct than operating
 on graphemes,while it's less efficient than operating on code units and more
 efficient than operating on graphemes.

 - Jonathan M Davis

When the code actually only cares about some characters that have 7-bit ASCII values, most of the time there are no correctness issues when operating on code units directly.
Mar 03 2012
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 03/03/2012 01:42 PM, H. S. Teoh wrote:
 On Sat, Mar 03, 2012 at 12:42:53PM -0800, Jonathan M Davis wrote:
 [...]
 The current solution encourages correct usage (or at least usage which
 is closer to correct, since it still isn't at the grapheme level)
 without disallowing more optimized code.

Speaking of graphemes, is anyone interested in implementing Unicode normalization for D? I looked at the specs briefly, and it seems to be something that is straightforward to implement, albeit somewhat tedious. It would be nice if D string types are normalized (needs slight change to string concatenation). Or at least, if there's a guaranteed normalized string type for those who care about it. T

Denis Spir was working on solving that problem but unfortunately we haven't heard from him for almost a year now. I think this is his site: http://spir.wikidot.com Ali
Mar 03 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-03-03 03:30, Piotr Szturmaj wrote:
 Hello,

 For this code:

 auto c = "test"c;
 auto w = "test"w;
 auto d = "test"d;
 pragma(msg, typeof(c.front));
 pragma(msg, typeof(w.front));
 pragma(msg, typeof(d.front));

 compiler prints:

 dchar
 dchar
 immutable(dchar)

I thought all these would be either "dchar" or "immutable(dchar)". Why are they of different types?
 IMO it should print this:

 immutable(char)
 immutable(wchar)
 immutable(dchar)

 Is it a bug?

-- /Jacob Carlborg
Mar 03 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 03/03/2012 04:36 AM, Jacob Carlborg wrote:
 On 2012-03-03 03:30, Piotr Szturmaj wrote:
 Hello,

 For this code:

 auto c = "test"c;
 auto w = "test"w;
 auto d = "test"d;
 pragma(msg, typeof(c.front));
 pragma(msg, typeof(w.front));
 pragma(msg, typeof(d.front));

 compiler prints:

 dchar
 dchar
 immutable(dchar)

I thought all these would be either "dchar" or "immutable(dchar)". Why are they of different types?

In the case of char and wchar slices, the "elements" are decoded as the iteration happens. In other words, the returned values are not actual elements of the ranges.
 IMO it should print this:

 immutable(char)
 immutable(wchar)
 immutable(dchar)

 Is it a bug?


Ali
Mar 03 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-03-03 15:10, Ali Çehreli wrote:
 On 03/03/2012 04:36 AM, Jacob Carlborg wrote:
 On 2012-03-03 03:30, Piotr Szturmaj wrote:
 Hello,

 For this code:

 auto c = "test"c;
 auto w = "test"w;
 auto d = "test"d;
 pragma(msg, typeof(c.front));
 pragma(msg, typeof(w.front));
 pragma(msg, typeof(d.front));

 compiler prints:

 dchar
 dchar
 immutable(dchar)

I thought all these would be either "dchar" or "immutable(dchar)". Why are they of different types?

In the case of char and wchar slices, the "elements" are decoded as the iteration happens. In other words, the returned values are not actual elements of the ranges.

Ah, I see, thanks. -- /Jacob Carlborg
Mar 03 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, March 03, 2012 18:38:44 Timon Gehr wrote:
 On 03/03/2012 09:40 AM, Jonathan M Davis wrote:
 ...  but operating on
 code points is _far_ more correct than operating on code units. It's also
 more efficient.
 [snip.]

No, it is less efficient.

Operating on code points is more efficient than operating on graphemes is what I meant. I can see that I wasn't clear enough on that. It's more correct than operating on code units and less correct than operating on graphemes,while it's less efficient than operating on code units and more efficient than operating on graphemes. - Jonathan M Davis
Mar 03 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, March 03, 2012 14:57:59 Piotr Szturmaj wrote:
 This discrepancy pushes the range writer to handle special string cases.

Yes it does. And there's _no_ way around that if you want to handle unicode both correctly and efficiently. To handle it correctly, you must operate on code points (or even better, graphemes), but to handle them efficiently, you must take the encoding into account. Phobos has gone with the default of correctness while giving you the tools to special case stuff for efficiency. Phobos itself uses static if all over the place to special case pieces of functions on string type. Stuff like isNarrowString and ElementEncodingType exist specifically for that.
 The problem with that range is when it takes a string type, it aliases
 this type with itself, because ElementType!R yields dchar. This is why
 I'm talking about "bad consequences", I just want to iterate string by
 _char_, not _dchar_.

If you want to iterate by char, then use foreach or use a wrapper range (or cast to ubyte[] and operate on that). Phobos specificically does not to do that, because it breaks unicode. It doesn't stop you from iterating by char or wchar if you really want to, but it operates or ranges of dchar by default, because it's more correct. - Jonathan M Davis
Mar 03 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, March 03, 2012 21:05:40 Timon Gehr wrote:
 On 03/03/2012 08:46 PM, Jonathan M Davis wrote:
 On Saturday, March 03, 2012 18:38:44 Timon Gehr wrote:
 On 03/03/2012 09:40 AM, Jonathan M Davis wrote:
 ...  but operating on
 code points is _far_ more correct than operating on code units. It's
 also
 more efficient.
 [snip.]

No, it is less efficient.

Operating on code points is more efficient than operating on graphemes is what I meant. I can see that I wasn't clear enough on that.

Makes sense.
 It's more correct than operating on code units and less correct than
 operating on graphemes,while it's less efficient than operating on code
 units and more efficient than operating on graphemes.
 
 - Jonathan M Davis

When the code actually only cares about some characters that have 7-bit ASCII values, most of the time there are no correctness issues when operating on code units directly.

True, but writing code without caring about unicode frequently leads to bugs when you actually _do_ have to deal with unicode (the fact that an American programmer runs into unicode less just makes it worse, because they're less likely to catch their bugs), and char is UTF-8 by definition. So, operating specifically on ASCII is an optimization and should be coded for specifically rather than being generally encouraged. And having ranges over strings be code units rather than code points would encourage incorrect usage. The current solution encourages correct usage (or at least usage which is closer to correct, since it still isn't at the grapheme level) without disallowing more optimized code. - Jonathan M Davis
Mar 03 2012
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Mar 03, 2012 at 12:42:53PM -0800, Jonathan M Davis wrote:
[...]
 The current solution encourages correct usage (or at least usage which
 is closer to correct, since it still isn't at the grapheme level)
 without disallowing more optimized code.

Speaking of graphemes, is anyone interested in implementing Unicode normalization for D? I looked at the specs briefly, and it seems to be something that is straightforward to implement, albeit somewhat tedious. It would be nice if D string types are normalized (needs slight change to string concatenation). Or at least, if there's a guaranteed normalized string type for those who care about it. T -- You have to expect the unexpected. -- RL
Mar 03 2012
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Mar 03, 2012 at 11:53:41AM -0800, Jonathan M Davis wrote:
[...]
 If you want to iterate by char, then use foreach or use a wrapper
 range (or cast to ubyte[] and operate on that).

Or use: string str = ...; for (size_t i=0; i < str.length; i++) { /* do something with str[i] */ }
 Phobos specificically does not to do that, because it breaks unicode.
 It doesn't stop you from iterating by char or wchar if you really want
 to, but it operates or ranges of dchar by default, because it's more
 correct.

I think this is the correct approach. Always err on the side of correct and/or safe, but give the programmer the option of getting under the hood if he wants otherwise. T -- A linguistics professor was lecturing to his class one day. "In English," he said, "A double negative forms a positive. In some languages, though, such as Russian, a double negative is still a negative. However, there is no language wherein a double positive can form a negative." A voice from the back of the room piped up, "Yeah, yeah."
Mar 03 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, March 03, 2012 13:46:16 Ali =C3=87ehreli wrote:
 On 03/03/2012 01:42 PM, H. S. Teoh wrote:
 On Sat, Mar 03, 2012 at 12:42:53PM -0800, Jonathan M Davis wrote:
 [...]
=20
 The current solution encourages correct usage (or at least usage w=



 is closer to correct, since it still isn't at the grapheme level)
 without disallowing more optimized code.

[...] =20 Speaking of graphemes, is anyone interested in implementing Unicode=


 normalization for D? I looked at the specs briefly, and it seems to=


 something that is straightforward to implement, albeit somewhat ted=


=20
 It would be nice if D string types are normalized (needs slight cha=


 to string concatenation). Or at least, if there's a guaranteed
 normalized string type for those who care about it.
=20
=20
 T

Denis Spir was working on solving that problem but unfortunately we haven't heard from him for almost a year now. I think this is his sit=

=20
    http://spir.wikidot.com
=20
 Ali

Mar 03 2012
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, March 03, 2012 13:46:16 Ali =C3=87ehreli wrote:
 On 03/03/2012 01:42 PM, H. S. Teoh wrote:
 On Sat, Mar 03, 2012 at 12:42:53PM -0800, Jonathan M Davis wrote:
 [...]
=20
 The current solution encourages correct usage (or at least usage w=



 is closer to correct, since it still isn't at the grapheme level)
 without disallowing more optimized code.

[...] =20 Speaking of graphemes, is anyone interested in implementing Unicode=


 normalization for D? I looked at the specs briefly, and it seems to=


 something that is straightforward to implement, albeit somewhat ted=


=20
 It would be nice if D string types are normalized (needs slight cha=


 to string concatenation). Or at least, if there's a guaranteed
 normalized string type for those who care about it.
=20
=20
 T

Denis Spir was working on solving that problem but unfortunately we haven't heard from him for almost a year now. I think this is his sit=

=20
    http://spir.wikidot.com

There's some stuff in the new std.regex which was done to enhance unico= de=20 support which is currently completely internal to it which may end up b= eing=20 the basis for more, but Dmitry hasn't yet worked on creating a version = of that=20 for more general consumption AFAIK. I'm not quite sure what he did thou= gh,=20 since I'm not familier with std.regex. - Jonathan M Davis
Mar 03 2012