www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - String implementations

reply bearophile <bearophileHUGS lycos.com> writes:
Defining how an ASCII string is best managed by a language is already complex
(ropes or not? Mutable or not? With shared parts or not? Etc), but today ASCII
isn't enough and when you add Unicode matters then string management becomes an
hairy topic, this may be interesting for D developers:

http://weblogs.mozillazine.org/roc/archives/2008/01/string_theory.html

Something curious: sometimes I need mutable strings, but I cope with the
immutable ones when necessary. This author says that even stringAt isn't much
useful! :-)

Bye,
bearophile
Jan 15 2008
next sibling parent Robert Fraser <fraserofthenight gmail.com> writes:
bearophile wrote:
 Defining how an ASCII string is best managed by a language is already complex
(ropes or not? Mutable or not? With shared parts or not? Etc), but today ASCII
isn't enough and when you add Unicode matters then string management becomes an
hairy topic, this may be interesting for D developers:
 
 http://weblogs.mozillazine.org/roc/archives/2008/01/string_theory.html
 
 Something curious: sometimes I need mutable strings, but I cope with the
immutable ones when necessary. This author says that even stringAt isn't much
useful! :-)
 
 Bye,
 bearophile

I agree with pretty much everything in that article, especially the part about charAt not being very useful (occasionally I iterate, but rarely do I need random access, though slicing is a good option for strings I know the format of that don't need regexes). D's "string" (that is, invariant(char)[] ) is a good compromise, although I'd prefer to also have a String class that I can use for *some* strings that can be implicitly used as (but not converted to) a char[] but has interning & hash code caching. But this is impossible within D's type system as it stands today.
Jan 15 2008
prev sibling parent reply Jarrod <qwerty ytre.wq> writes:
On Tue, 15 Jan 2008 21:23:31 -0500, bearophile wrote:

 Defining how an ASCII string is best managed by a language is already
 complex (ropes or not? Mutable or not? With shared parts or not? Etc),
 but today ASCII isn't enough and when you add Unicode matters then
 string management becomes an hairy topic, this may be interesting for D
 developers:
 
 http://weblogs.mozillazine.org/roc/archives/2008/01/string_theory.html
 
 Something curious: sometimes I need mutable strings, but I cope with the
 immutable ones when necessary. This author says that even stringAt isn't
 much useful! :-)
 
 Bye,
 bearophile

This article is pretty correct. While the topic is at hand, I guess I could rant a little; Why does D practically *require* the coder to use different forms of UTF encoding. D can tell if a code unit spans multiple bytes, as evidenced in converting a utf-8 string to utf-32 (D knows where to split the blocks apart), yet we can't index char[] arrays by code units. Instead, D will index char arrays by fixed length bytes, which is almost nonsensical since the D spec asserts that char[] arrays are designed specifically for unicode characters, and that other single byte arrays should instead be made as a byte[]. So if this is the case, then why can't the language itself manage multi- byte characters for us? It would make things a hell of a lot easier and more efficient than having to convert /potentially/ foreign strings to utf-32 for a simple manipulation operation, then converting them back. The only reason I can think of for char arrays being treated as fixed length is for faster indexing, which is hardly useful in most cases since a lot of the time we don't even know if we're dealing with multi-byte characters when handling strings, so we have to convert and traverse the strings anyway. Arg. I know this may probably be a pain to implement, but it would really give D a huge leg-up if it could properly and automatically handle strings for us. Without requiring a bloated string class.
Jan 16 2008
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
"Jarrod" wrote
 On Tue, 15 Jan 2008 21:23:31 -0500, bearophile wrote:

 Defining how an ASCII string is best managed by a language is already
 complex (ropes or not? Mutable or not? With shared parts or not? Etc),
 but today ASCII isn't enough and when you add Unicode matters then
 string management becomes an hairy topic, this may be interesting for D
 developers:

 http://weblogs.mozillazine.org/roc/archives/2008/01/string_theory.html

 Something curious: sometimes I need mutable strings, but I cope with the
 immutable ones when necessary. This author says that even stringAt isn't
 much useful! :-)

 Bye,
 bearophile

This article is pretty correct. While the topic is at hand, I guess I could rant a little; Why does D practically *require* the coder to use different forms of UTF encoding. D can tell if a code unit spans multiple bytes, as evidenced in converting a utf-8 string to utf-32 (D knows where to split the blocks apart), yet we can't index char[] arrays by code units. Instead, D will index char arrays by fixed length bytes, which is almost nonsensical since the D spec asserts that char[] arrays are designed specifically for unicode characters, and that other single byte arrays should instead be made as a byte[]. So if this is the case, then why can't the language itself manage multi- byte characters for us? It would make things a hell of a lot easier and more efficient than having to convert /potentially/ foreign strings to utf-32 for a simple manipulation operation, then converting them back. The only reason I can think of for char arrays being treated as fixed length is for faster indexing, which is hardly useful in most cases since a lot of the time we don't even know if we're dealing with multi-byte characters when handling strings, so we have to convert and traverse the strings anyway.

The algorithmic penalties would be O(n) for a indexed lookup then instead of O(1). I think the way it is now is the best of all worlds. I think the correct method in this case is to convert to utf32 first, then index. Then at least you only take the O(n) penalty once. Or why not just use dchar[] instead of char[] to begin with? -Steve
Jan 16 2008
parent Dan <murpsoft hotmail.com> writes:
Jarrod Wrote:

 On Wed, 16 Jan 2008 10:27:53 -0500, Steven Schveighoffer wrote:
 
 The algorithmic penalties would be O(n) for a indexed lookup then
 instead of O(1).

I understand this, but the compiler could probably optimize this for most situations. Most string access would be sequential and thus positions could be cached on access when need be, and string literals that aren't modified and have all single-byte chars could be optimized into normal indexing. Furthermore, modern processors are incredibly good at sequential iteration and I know from personal experience that they can parse over massive chunks of memory in mere milliseconds (hashing entire executables in memory for potential changes is a common example of this). It shouldn't be noticeable at all to scan over a string. I do believe the author of the article that bearophile linked agrees with me on this regard, in his mention of charAt implementation.
 I think the correct method in this case is to convert to utf32 first,
 then index.  Then at least you only take the O(n) penalty once.  

Well, converting to dchar[] means a full iteration over the entire string to split up the units. Then the program has to allocate space, copy chars over, and add padding. Is it really all that much more efficient? And why should the programmer have to worry about the conversion anyway? Good languages avoid cognitive load on the programmers.
 Or why not just use dchar[] instead of char[] to begin with?

Yes, you could just use dchar[] all the time, but how many people do that? It's very space-inefficient which is the whole reason utf-8 exists. If dchar[]s were meant to be used more often Walter probably would have made them the default string type. Eh, I guess this is just one of those annoying little 'loose ends' I see when I look at D.

Certainly is a whole lot better than the loose ends in other languages; at least we're in UTF and not ASCII (or undefined language). I personally prefer UTF-8. I can write any UTF character in UTF8 if I accept that odd case of a UTF-32 character will be stored as \uXXXX. To be honest, that's acceptable; and gives me the memory savings and O(1) as long as I've got the foresight to predict where the \u's are. I love D's handling of strings, in fact it is my *favorite* feature in D.
Jan 17 2008
prev sibling next sibling parent Jarrod <qwerty ytre.wq> writes:
On Wed, 16 Jan 2008 10:27:53 -0500, Steven Schveighoffer wrote:
 
 The algorithmic penalties would be O(n) for a indexed lookup then
 instead of O(1).

I understand this, but the compiler could probably optimize this for most situations. Most string access would be sequential and thus positions could be cached on access when need be, and string literals that aren't modified and have all single-byte chars could be optimized into normal indexing. Furthermore, modern processors are incredibly good at sequential iteration and I know from personal experience that they can parse over massive chunks of memory in mere milliseconds (hashing entire executables in memory for potential changes is a common example of this). It shouldn't be noticeable at all to scan over a string. I do believe the author of the article that bearophile linked agrees with me on this regard, in his mention of charAt implementation.
 I think the correct method in this case is to convert to utf32 first,
 then index.  Then at least you only take the O(n) penalty once.  

Well, converting to dchar[] means a full iteration over the entire string to split up the units. Then the program has to allocate space, copy chars over, and add padding. Is it really all that much more efficient? And why should the programmer have to worry about the conversion anyway? Good languages avoid cognitive load on the programmers.
 Or why not just use dchar[] instead of char[] to begin with?

Yes, you could just use dchar[] all the time, but how many people do that? It's very space-inefficient which is the whole reason utf-8 exists. If dchar[]s were meant to be used more often Walter probably would have made them the default string type. Eh, I guess this is just one of those annoying little 'loose ends' I see when I look at D.
Jan 16 2008
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Jarrod wrote:
 While the topic is at hand, I guess I could rant a little;
 Why does D practically *require* the coder to use different forms of UTF 
 encoding.

Because I've worked with internationalized code in C/C++ where the encoding isn't specified, and it's very bad.
 D can tell if a code unit spans multiple bytes, as evidenced in 
 converting a utf-8 string to utf-32 (D knows where to split the blocks 
 apart), yet we can't index char[] arrays by code units. Instead, D will 
 index char arrays by fixed length bytes, which is almost nonsensical

It is impractical (i.e. very inefficient) to index arrays otherwise, especially in getting array lengths, doing slicing, etc. In fact, it is rather rare to index by code units. The times you might want to do it are easily handled by foreach(dchar c, string).
 since the D spec asserts that char[] arrays are designed specifically for 
 unicode characters, and that other single byte arrays should instead be 
 made as a byte[].
 So if this is the case, then why can't the language itself manage multi-
 byte characters for us?

It does, see foreach. In general, I don't think it's a good idea for the language to try to completely hide the multibyte nature of UTF. For example, when you're allocating and copying strings around, you need the byte length, not the number of code points.
 It would make things a hell of a lot easier and 
 more efficient than having to convert /potentially/ foreign strings to 
 utf-32 for a simple manipulation operation, then converting them back.
 The only reason I can think of for char arrays being treated as fixed 
 length is for faster indexing, which is hardly useful in most cases since 
 a lot of the time we don't even know if we're dealing with multi-byte 
 characters when handling strings, so we have to convert and traverse the 
 strings anyway.
 Arg.

I was surprised to discover that most indexing work in strings, such as searching, work more efficiently by *not* trying to index by code points. There are standard library functions in std.utf to index by code points, if you do need it.
 I know this may probably be a pain to implement, but it would really give 
 D a huge leg-up if it could properly and automatically handle strings for 
 us. Without requiring a bloated string class.

I believe D already has found the right approach to handling UTF strings. All I can say is try it out for a while.
Jan 17 2008
parent James Dennett <jdennett acm.org> writes:
Janice Caron wrote:
 On 1/20/08, Jarrod <qwerty ytre.wq> wrote:
 But I still don't see what this has got to do with whether or not a[n]
 should identify the (n+1)th character rather than the (n+1)th code unit.

do with the potential input characters given in the file.

You mean the plain text config file of unknown encoding?

Let's stop here. If you don't know the encoding, you can't safely process the file. That's nothing to do with language or library designs. You can't process data whose format you do not know. (Yes, you can employ heuristics to try to guess, but they can be wrong, and in the case of text files there are many files which are valid in numerous encodings but have different meanings.) -- James
Jan 20 2008
prev sibling next sibling parent reply "Janice Caron" <caron800 googlemail.com> writes:
On 1/16/08, Jarrod <qwerty ytre.wq> wrote:
 On Tue, 15 Jan 2008 21:23:31 -0500, bearophile wrote:

 So if this is the case, then why can't the language itself manage multi-
 byte characters for us? It would make things a hell of a lot easier and
 more efficient than having to convert /potentially/ foreign strings to
 utf-32 for a simple manipulation operation, then converting them back.
 The only reason I can think of for char arrays being treated as fixed
 length is for faster indexing, which is hardly useful in most cases since
 a lot of the time we don't even know if we're dealing with multi-byte
 characters when handling strings, so we have to convert and traverse the
 strings anyway.

Because, think about this: char[] a = new char[8]; If a char array were indexed by character instead of codeunit, as you suggest, how many bytes would the compiler need to allocate? It can't know in advance. Also: char[] a = "abcd"; char[] b = "\u20AC"; a[0] = b[0]; would cause big problems. (Would a[1] get overwritten? Would a have to be resized and everything shifted up one byte?) I think D has got it right. Use wchar or dchar when you need character based indexing.
Jan 18 2008
parent reply James Dennett <jdennett acm.org> writes:
Janice Caron wrote:
 On 1/16/08, Jarrod <qwerty ytre.wq> wrote:
 On Tue, 15 Jan 2008 21:23:31 -0500, bearophile wrote:

 So if this is the case, then why can't the language itself manage multi-
 byte characters for us? It would make things a hell of a lot easier and
 more efficient than having to convert /potentially/ foreign strings to
 utf-32 for a simple manipulation operation, then converting them back.
 The only reason I can think of for char arrays being treated as fixed
 length is for faster indexing, which is hardly useful in most cases since
 a lot of the time we don't even know if we're dealing with multi-byte
 characters when handling strings, so we have to convert and traverse the
 strings anyway.

Because, think about this: char[] a = new char[8]; If a char array were indexed by character instead of codeunit, as you suggest, how many bytes would the compiler need to allocate?

8. That's independent of how they are indexed.
 It can't
 know in advance. 

Yup, it can. 8.
 Also:
 
     char[] a = "abcd";
     char[] b = "\u20AC";
     a[0] = b[0];
 
 would cause big problems. (Would a[1] get overwritten? Would a have to
 be resized and everything shifted up one byte?)

It causes big problems with byte-wise addressing, because you can end up with a char[] (which is specified to hold a UTF8 string) which does not contain a UTF8 string.
 I think D has got it right. Use wchar or dchar when you need character
 based indexing.

If you have UTF8, you should not be allowed to access the bytes that make it up, only its characters. If you want a bunch of bytes, use an array of bytes. (On the other hand, D is weak here because it identifies UTF8 strings with arrays of char, but char doesn't hold a UTF8 character. I can't imagine persuading Walter that this is a horrible error is going to work though.) -- James
Jan 18 2008
next sibling parent James Dennett <jdennett acm.org> writes:
Janice Caron wrote:
 On 1/19/08, James Dennett <jdennett acm.org> wrote:
 Because, think about this:

     char[] a = new char[8];

 If a char array were indexed by character instead of codeunit, as you
 suggest, how many bytes would the compiler need to allocate?


My *rhetorical* question was in response to a poster who suggested that char arrays should be indexed by CHARACTER.

Yes, I know. If you don't see that my post answered yours, feel free to ask me to explain more clearly. Indexing by character doesn't automatically tell us anything about how the size specified for new[] works. It would be a little quirky for the size to be in bytes while the indices are in characters, but it's quirky for char[] to pretend to be a UTF8 type without enforcing that.
 I know perfectly well
 that in reality they are indexed by UTF-8 code unit, and therefore
 that the correct answer is eight. I was giving additional reasons
 /why/ indexing by character was not sensible.

I was refuting your claimed reason.
 
 It can't
 know in advance.


Of course. However, /if/ they were indexed by character, /then/ it would be impossible to know in advance how many UTF-8 code units it would take to construct eight characters.

Sure. So it wouldn't be a request for 8 characters.
 Which is a good reason why
 they should /not/ be indexed by character.

I *still* disagree with that.
 As far as I am concerned, D has got it right.

Enjoy ;)
 Also:

     char[] a = "abcd";
     char[] b = "\u20AC";
     a[0] = b[0];

 would cause big problems. (Would a[1] get overwritten? Would a have to
 be resized and everything shifted up one byte?)

can end up with a char[] (which is specified to hold a UTF8 string) which does not contain a UTF8 string.

That's what I said. Thank you for reiterating it.

I think we are *disagreeing* here. I claim that this causes problems _with the current design of D_, which would be resolved if char[] (or however we denote mutable UTF8 strings) string were really a UTF8 type.
 I think D has got it right. Use wchar or dchar when you need character
 based indexing.

that make it up, only its characters.

I absolutely /should/ be (and am) allowed to access the UTF-8 code units stored within an array of UTF-8 code units. This is absolutely as it should be, thank you very much.

But you already illustrated why it should not: you can break the invariant that there is valid UTF8 there, so char[] is lying when it says that it is a UTF8 type.
 
 If you want a bunch of
 bytes, use an array of bytes.

Of course. And if you want an array of UTF-8 code units, use a char[] or a string.

If you're at the level of code units, you can't be sure that they're part of UTF8 characters unless there is a higher-level invariant enforced on them.
 (On the other hand, D is weak here
 because it identifies UTF8 strings with arrays of char, but char
 doesn't hold a UTF8 character.

I believe you are wrong. I would say that D is strong here, because it identifies UTF-8 strings with arrays of char, because char /does/ hold a UTF-8 code unit.

That's the problem. char[] can hold non-UTF8 strings.
 The mistake is assuming that code unit == character.

If you're telling me that I made that mistake, you're sorely mistaken.
 It does not.
 (However, there is overlap in the ASCII range, 0x00 to 0x7F, so the
 assumption is still valid if you are certain that your strings are
 ASCII).
 
 Once you've grokked that char[] == array of UTF-8 code unit,
 everything else falls into place and makes sense.

No, it does not. It's precisely the difference that is why D's char[] is a poor man's UTF8 string. -- James
Jan 19 2008
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
James Dennett wrote:
 (On the other hand, D is weak here
 because it identifies UTF8 strings with arrays of char, but char
 doesn't hold a UTF8 character.  I can't imagine persuading Walter
 that this is a horrible error is going to work though.)

I've actually done considerable work with UTF-8, both in C++ and D. D's method of dealing with it works out very well (and very naturally). This is why you'll have a hard time persuading me otherwise <g>. Note that C++0x is doing things similarly: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2249.html
Jan 20 2008
parent reply James Dennett <jdennett acm.org> writes:
Walter Bright wrote:
 James Dennett wrote:
 (On the other hand, D is weak here
 because it identifies UTF8 strings with arrays of char, but char
 doesn't hold a UTF8 character.  I can't imagine persuading Walter
 that this is a horrible error is going to work though.)

I've actually done considerable work with UTF-8, both in C++ and D.

Yes, by this stage most serious programmers have had to learn in some detail how to work with UTF-8.
 D's 
 method of dealing with it works out very well (and very naturally).

I've given specific problems with it. I've heard no refutation of them. D uses essentially a model of UTF8 which is really just a bunch-of-bytes with smart iteration. C-based projects on which I worked in the 90's did similarly, but with coding conventions that banned direct access to the bytes.
 This is why you'll have a hard time persuading me otherwise <g>.

Because you assert that there's not a problem? ;)
 Note that C++0x is doing things similarly:
 
 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2249.html

Looks very different to me. There's no conflation of char with a code unit of UTF8 (and indeed C++ deliberately supports use of varied encodings for multi-byte characters). Yes, C++ is adding 16- and 32-bit character types which are more akin to D's, but that has little bearing on how differently it handles multi-byte (as opposed to wide-character) strings. -- James
Jan 20 2008
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
James Dennett wrote:
 I've given specific problems with it.  I've heard no refutation
 of them.

It's hard to describe, but after working with UTF-8 for a while, they are just non-problems. Code isn't written that way. If you want, you can create a String class which wraps a char[] and treats it at the level you wish.
 D uses essentially a model of UTF8 which is really just
 a bunch-of-bytes with smart iteration.

That's what UTF-8 is.
 C-based projects on which
 I worked in the 90's did similarly, but with coding conventions
 that banned direct access to the bytes.

Coding conventions are one thing, but banning things in a systems language are quite another. Copying a UTF-8 string by decoding and encoding the characters one-by-one is unacceptably inefficient, for example, compared with just memcpy. Searching a UTF-8 string for a substring is another operation for which treating it like a bag of bytes works best.
 This is why you'll have a hard time persuading me otherwise <g>.


Because I know it works based on experience.
 Note that C++0x is doing things similarly:

 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2249.html

Looks very different to me. There's no conflation of char with a code unit of UTF8 (and indeed C++ deliberately supports use of varied encodings for multi-byte characters). Yes, C++ is adding 16- and 32-bit character types which are more akin to D's, but that has little bearing on how differently it handles multi-byte (as opposed to wide-character) strings.

Since, in the C++ proposal, indexing and length is done by byte/word/dword, not by code point, it's semantically equivalent. I don't see any banning of getting at the underlying representation, nor any attempt to hide it.
Jan 20 2008
parent reply James Dennett <jdennett acm.org> writes:
Walter Bright wrote:
 James Dennett wrote:
 I've given specific problems with it.  I've heard no refutation
 of them.

It's hard to describe, but after working with UTF-8 for a while, they are just non-problems. Code isn't written that way. If you want, you can create a String class which wraps a char[] and treats it at the level you wish.

Indeed, but such a thing should be standard, not reinvented over and over.
 D uses essentially a model of UTF8 which is really just
 a bunch-of-bytes with smart iteration.

That's what UTF-8 is.

That view has lead to many security issues, where different software reacts differently to byte strings which are not valid UTF-8 in places where UTF-8 is expected.
 C-based projects on which
 I worked in the 90's did similarly, but with coding conventions
 that banned direct access to the bytes.

Coding conventions are one thing, but banning things in a systems language are quite another. Copying a UTF-8 string by decoding and encoding the characters one-by-one is unacceptably inefficient, for example, compared with just memcpy. Searching a UTF-8 string for a substring is another operation for which treating it like a bag of bytes works best.

There are alternatives; explicit notation to access the bytes, which *doesn't* look like it's accessing characters, would be better. (char doesn't represent a character in D. Not great naming? But then D almost follows C in this, where char did double duty as a limited character type and a small integral type.)
 This is why you'll have a hard time persuading me otherwise <g>.


Because I know it works based on experience.

And I know, based on experience, of problems with it. So how do we get past this to discuss things more objectively? (Of course, we don't have to. You're the BDFL, and you get to make the call, and try to keep D coherent in the face of a hundred people pushing inconsistent views for how it should evolve. I get the easy job of being just one of those voices.)
 Note that C++0x is doing things similarly:

 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2249.html

Looks very different to me. There's no conflation of char with a code unit of UTF8 (and indeed C++ deliberately supports use of varied encodings for multi-byte characters). Yes, C++ is adding 16- and 32-bit character types which are more akin to D's, but that has little bearing on how differently it handles multi-byte (as opposed to wide-character) strings.

Since, in the C++ proposal, indexing and length is done by byte/word/dword, not by code point, it's semantically equivalent. I don't see any banning of getting at the underlying representation, nor any attempt to hide it.

Whereas D partly attempts to hide it; the mathematician in me hates this kind of fence-sitting. But let's get more concrete: suppose D code finds that an alleged char[] passed to it is, in fact, broken (i.e., violates the UTF8 invariants). What should it do -- abort, throw an exception, offer a policy for handling such bugs, other? -- James
Jan 20 2008
parent reply Matti Niemenmaa <see_signature for.real.address> writes:
Janice Caron wrote:
 The important point that I want to make here is that *D GETS
 IT RIGHT*. D's implementation will throw an exception on all invalid
 UTF-8 sequences, and this will block all such security issues.

Well, mostly right: http://d.puremagic.com/issues/show_bug.cgi?id=978 -- E-mail address: matti.niemenmaa+news, domain is iki (DOT) fi
Jan 21 2008
parent Matti Niemenmaa <see_signature for.real.address> writes:
Janice Caron wrote:
 On Jan 21, 2008 8:47 AM, Matti Niemenmaa <see_signature for.real.address>
wrote:
 Janice Caron wrote:
 The important point that I want to make here is that *D GETS
 IT RIGHT*. D's implementation will throw an exception on all invalid
 UTF-8 sequences, and this will block all such security issues.


Oooh - well spotted! In that case, I amend my statement. D /will/ get it right, once this bug is fixed. One would hope that said bug will be fixed in the next release.

You'll note the bug is a year old. Although that doesn't change the fact that one would, indeed, hope for that. :-) -- E-mail address: matti.niemenmaa+news, domain is iki (DOT) fi
Jan 21 2008
prev sibling parent James Dennett <jdennett acm.org> writes:
Janice Caron wrote:
 On 1/20/08, James Dennett <jdennett acm.org> wrote:
 Looks very different to me.

I thought it looked very similar indeed to D, but there you go. Funny how two different people can read the same document and interpret it in two different ways.

The core issue here, to me, is D's half-hearted attempt to paint char[] as a Unicode string type. C++ has nothing analagous.
 There's no conflation of char with a
 code unit of UTF8

C has no ubyte type. Since time immemorial, C programmers have been using the char type to store every 8-bit wide data type under the sun simply because there's been no alternative (until recently, when int8_t showed up as a typedef for char).

int8_t is necessarily signed, a la "signed char", not a typedef for "char", whose signedness varies (but, unfortunately, is often signed in C and C++).
 That's not a big deal.
 
 
 (and indeed C++ deliberately supports use of
 varied encodings for multi-byte characters).

I must have misread the heading that says "Require UTF", and whose text reads "The C TR makes the encoding of char16_t and char32_t implementation-defined. It also provides macros to indicate whether or not the encoding is UTF. In contrast, this proposal requires UTF encoding." Oh, I see what you're saying - C++ would require UTF for wchar and dchar, but not for char. Well, that's historical legacy for you.

And it's the real world; computer systems need to interface with existing systems which us diverse encodings.
 Yes, C++ is adding
 16- and 32-bit character types which are more akin to D's, but that
 has little bearing on how differently it handles multi-byte (as
 opposed to wide-character) strings.

So it has a bunch of procedural functions instead of foreach. Apart from that, the approach seems the same as D. Where's the difference?

Philosophy: D pushes char[] as if it were a proper UTF8 facility, and goes a small step towards adding language support for that. C++ recognizes diversity in multi-byte character encodings, and doesn't make the language promote one over any other. It admits up-front that you're dealing with code units if you want to work with multi-byte characters. C++ is a long, long way from perfect when it comes to Unicode support. Even C++0x will be. But I'm hoping for more from D, and what I see so far can stand some improvement. -- James
Jan 20 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 1/19/08, James Dennett <jdennett acm.org> wrote:
 Because, think about this:

     char[] a = new char[8];

 If a char array were indexed by character instead of codeunit, as you
 suggest, how many bytes would the compiler need to allocate?

8. That's independent of how they are indexed.

My *rhetorical* question was in response to a poster who suggested that char arrays should be indexed by CHARACTER. I know perfectly well that in reality they are indexed by UTF-8 code unit, and therefore that the correct answer is eight. I was giving additional reasons /why/ indexing by character was not sensible.
 It can't
 know in advance.

Yup, it can. 8.

Of course. However, /if/ they were indexed by character, /then/ it would be impossible to know in advance how many UTF-8 code units it would take to construct eight characters. Which is a good reason why they should /not/ be indexed by character. As far as I am concerned, D has got it right.
 Also:

     char[] a = "abcd";
     char[] b = "\u20AC";
     a[0] = b[0];

 would cause big problems. (Would a[1] get overwritten? Would a have to
 be resized and everything shifted up one byte?)

It causes big problems with byte-wise addressing, because you can end up with a char[] (which is specified to hold a UTF8 string) which does not contain a UTF8 string.

That's what I said. Thank you for reiterating it.
 I think D has got it right. Use wchar or dchar when you need character
 based indexing.

If you have UTF8, you should not be allowed to access the bytes that make it up, only its characters.

I absolutely /should/ be (and am) allowed to access the UTF-8 code units stored within an array of UTF-8 code units. This is absolutely as it should be, thank you very much.
 If you want a bunch of
 bytes, use an array of bytes.

Of course. And if you want an array of UTF-8 code units, use a char[] or a string.
 (On the other hand, D is weak here
 because it identifies UTF8 strings with arrays of char, but char
 doesn't hold a UTF8 character.

I believe you are wrong. I would say that D is strong here, because it identifies UTF-8 strings with arrays of char, because char /does/ hold a UTF-8 code unit. The mistake is assuming that code unit == character. It does not. (However, there is overlap in the ASCII range, 0x00 to 0x7F, so the assumption is still valid if you are certain that your strings are ASCII). Once you've grokked that char[] == array of UTF-8 code unit, everything else falls into place and makes sense.
Jan 19 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 1/19/08, James Dennett <jdennett acm.org> wrote:
     char[] a = "abcd";
     char[] b = "\u20AC";
     a[0] = b[0];



I think we are *disagreeing* here. I claim that this causes problems _with the current design of D_, which would be resolved if char[] (or however we denote mutable UTF8 strings) string were really a UTF8 type.

So you're saying that in your new design, after that assignment, a would equal "\u20ACbcd". The problem is that the compiler would have to allocate extra bytes and then memcpy all the bytes up a bit to make room. That strikes me as kinda slow, which is not something I'd want in a char array.
 That's the problem.  char[] can hold non-UTF8 strings.

Yes, that is possible. But only in buggy code, of course. That really raises the question: is it the compiler's job, or the programmer's, to ensure that the contract is maintained? I don't really have any problem taking responsibility for maintaining UTF-8 correctness. (It's not hard). But if you want to be completely protected from those kinds of errors, I still don't see the problem with using dchar.
 No, it does not.  It's precisely the difference that is why
 D's char[] is a poor man's UTF8 string.

I suppose a library class could be written whose interface behaved like a dchar array, but whose implementation was UTF-8. But when you ever use it?
Jan 19 2008
prev sibling next sibling parent Jarrod <qwerty ytre.wq> writes:
On Sat, 19 Jan 2008 20:21:49 +0000, Janice Caron wrote:

 So you're saying that in your new design, after that assignment, a would
 equal "\u20ACbcd". The problem is that the compiler would have to
 allocate extra bytes and then memcpy all the bytes up a bit to make
 room. That strikes me as kinda slow, which is not something I'd want in
 a char array.

Well how else would you like it to be done? If you were writing something that took a text input much like this very window I'm typing in right now, and the user hit back a few times and input a multi-byte character, how would you deal with it? Allow it to overlap? No. dchars? That's a lot of wasted memory, and it basically makes me wonder why utf-8 even exists if it needs to be dropped for simple text manipulation. May as well stick with utf-32 and ascii. No sir, I don't like it.
Jan 19 2008
prev sibling next sibling parent Jarrod <qwerty ytre.wq> writes:
On Thu, 17 Jan 2008 13:40:12 -0800, Walter Bright wrote:

 Because I've worked with internationalized code in C/C++ where the
 encoding isn't specified, and it's very bad.

I was more referring to the required switching to and from different utf types just to change a few characters around, I wasn't really referring to letting programmer decide what kind of string encoding to use.
 It is impractical (i.e. very inefficient) to index arrays otherwise,
 especially in getting array lengths, doing slicing, etc. In fact, it is
 rather rare to index by code units. The times you might want to do it
 are easily handled by foreach(dchar c, string).

Well yes I'm sure there's a performance hit for changing how it is indexed, but at the same time who would honestly prefer to index by the code point without first getting the unit points? You're practically stabbing in the dark if you try to slice a char[] array without first iterating over it with foreach to find its points.
 It does, see foreach. In general, I don't think it's a good idea for the
 language to try to completely hide the multibyte nature of UTF. For
 example, when you're allocating and copying strings around, you need the
 byte length, not the number of code points.

string str = "etc"; int strlen = str.length; int arrsize = str.sizeof; Seems pretty simple to me. And you don't have to completely hide the multibyte nature. Casting to a byte[] would allow full access to each point, which might sound hackish but at the same time manipulating individual code points in a string sounds like you're more than likely doing something just as hackish.
 I was surprised to discover that most indexing work in strings, such as
 searching, work more efficiently by *not* trying to index by code
 points. There are standard library functions in std.utf to index by code
 points, if you do need it.

Efficiency at the cost of the programmer. :( Perhaps you could design methods to access a string by either unit or index if you see the need to keep index-by-byte behaviour. Something like a toggle method would suit me just fine. str.indexByByte(true);
 I believe D already has found the right approach to handling UTF
 strings. All I can say is try it out for a while.

I am, and it's making working with user-editable config files an annoyance that perl avoids very easily.
Jan 19 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 1/20/08, Jarrod <qwerty ytre.wq> wrote:
 I am, and it's making working with user-editable config files an
 annoyance that perl avoids very easily.

Could you possibly explain that, for the benefit of those of us who don't speak perl? My limited understanding is the perl was invented before Unicode, and probably even before the wheel, so either it deals with Unicode by not dealing with it at all, or else it's a recent edition to the language (or else I've got it completely wrong - like I said, I don't speak perl). Also, isn't perl an interpreted language? You can get away with a lot more in an interpreted language, but you pay the price in speed. Moreover, working with user-editable config files - I would have thought that a job for a text editor, not a programming language. I'm confused.
Jan 19 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 1/20/08, Jarrod <qwerty ytre.wq> wrote:
 If you were writing something that took a text input much like this very
 window I'm typing in right now, and the user hit back a few times and
 input a multi-byte character, how would you deal with it?

I'd write a class, of course. It is simple (though not trivial) to step through the bytes of UTF-8. Bytes in the range 00 to 7F are ASCII; bytes in the range 80 to BF are tail bytes; bytes in the range C0 to F7 are head bytes; and bytes in the range F8 to FF are illegal. Identifying multi-byte sequences is therefore easy. You can make an argument that functions and/or classes to do this sort of thing should perhaps pre-exist in Phobos, but to say it should be built into /the language itself/ ... that's going a bit too far, I feel. Allow it to
 overlap? No. dchars? That's a lot of wasted memory, and it basically
 makes me wonder why utf-8 even exists if it needs to be dropped for
 simple text manipulation. May as well stick with utf-32 and ascii.

 No sir, I don't like it.

Jan 19 2008
prev sibling next sibling parent Jarrod <qwerty ytre.wq> writes:
On Sun, 20 Jan 2008 06:41:29 +0000, Janice Caron wrote:

 On 1/20/08, Jarrod <qwerty ytre.wq> wrote:
 I am, and it's making working with user-editable config files an
 annoyance that perl avoids very easily.

Could you possibly explain that, for the benefit of those of us who don't speak perl? My limited understanding is the perl was invented before Unicode, and probably even before the wheel, so either it deals with Unicode by not dealing with it at all, or else it's a recent edition to the language (or else I've got it completely wrong - like I said, I don't speak perl).

Perl is still being constantly updated although it is indeed quite old. And it works quite well with unicode as you would expect from a language that prides itself on text manipulation.
 Also, isn't perl an interpreted language? You can get away with a lot
 more in an interpreted language, but you pay the price in speed.

Yes, It's interpreted and that does cost it a fair amount of speed, but I see it as a worthwhile trade off for what it can do with strings.
 Moreover, working with user-editable config files - I would have thought
 that a job for a text editor, not a programming language. I'm confused.

Indeed, you are a tad confused. I'm allowing the user to edit config files so that my GUI application can read it in on startup and use it to populate a dialog display as well as fill out numerous options involving how it deals with a web interface. Because I don't know what the user is going to input I have to do a fair amount of converting. Yes, this in indeed the main motivation behind this entire rant.
Jan 19 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 1/20/08, Jarrod <qwerty ytre.wq> wrote:
 Moreover, working with user-editable config files - I would have thought
 that a job for a text editor, not a programming language. I'm confused.

Indeed, you are a tad confused.

Yep. I said so! :-)
 I'm allowing the user to edit config
 files

How? With a GUI interface? With a program written in D? With their favorite text editor of choice? If the latter, then you cannot be sure of the encoding, and that's hardly D's fault!
 so that my GUI application can read it in on startup and use it to
 populate a dialog display as well as fill out numerous options involving
 how it deals with a web interface. Because I don't know what the user is
 going to input I have to do a fair amount of converting.

Right, but converting from one encoding to another is the job of specialised classes. Detecting whether a text file is in ISO-8859-1, or Windows-1252, or MAC-ROMAN, or whatever, is not a trivial task. If your application were going to do that, you'd have to provide the implementation. (Or possibly Tango or some other third party library already provides such converters - I don't know). In any case, it's not a common enough task to warrant built-in language support. But I still don't see what this has got to do with whether or not a[n] should identify the (n+1)th character rather than the (n+1)th code unit.
 Yes, this in indeed the main motivation behind this entire rant.

Cool. So what is the real world use case that necessitates that sequences of UTF-8 code units must be addressable by character index as the default?
Jan 20 2008
prev sibling next sibling parent reply Jarrod <qwerty ytre.wq> writes:
On Sun, 20 Jan 2008 08:04:01 +0000, Janice Caron wrote:

 I'm allowing the user to edit config
 files

How? With a GUI interface? With a program written in D? With their favorite text editor of choice? If the latter, then you cannot be sure of the encoding, and that's hardly D's fault!

It is the latter.
 Right, but converting from one encoding to another is the job of
 specialised classes. Detecting whether a text file is in ISO-8859-1, or
 Windows-1252, or MAC-ROMAN, or whatever, is not a trivial task. If your
 application were going to do that, you'd have to provide the
 implementation. (Or possibly Tango or some other third party library
 already provides such converters - I don't know). In any case, it's not
 a common enough task to warrant built-in language support.
 
 But I still don't see what this has got to do with whether or not a[n]
 should identify the (n+1)th character rather than the (n+1)th code unit.

Because this issue isn't really to do with the input file itself, it's to do with the potential input characters given in the file. As far as I can tell (I'm using a C library to parse the input) it should be ascii or UTF-8 encoding. Anything else would probably cause the C lexer to screw up.
 Cool. So what is the real world use case that necessitates that
 sequences of UTF-8 code units must be addressable by character index as
 the default?

The most important one right now is splicing. I'm allowing both user- defined and program-defined macros in the input data. They can be anywhere within a string, so I need to splice them out and replace them with their correct counterparts. I hear the std lib provided with D is unreliable so I'm unwilling to use it. Plus even if it is fixed up I'd hate to limit string manipulation to regular expressions. I also wish to cut off input at a certain letter count for spacing issues in both the GUI and dealing with the webscript. I'll have to be converting certain characters to their URI equivalent too, that will probably take more splicing as well. The other thing I'm using is single-letter replacement. Simple stuff like capitalising letters and replacing spaces with underscores. I can think of a lot of other situations that would benefit from proper multibyte support too, for instance practically any application that takes frequent user input could benefit. A text editor is a very good example. Any coders who don't natively deal with Latin text would probably benefit greatly too ( think of the poor Japanese coders :< ). I've seen a lot of programs that print a specified number of characters before wrapping around or trailing off, too. The humble gnome console is a good example of that. Very handy to have character indexing in this case. String tokenizing and plain old character counting are two operations I can think of that could probably be done easier too. In the end I think I'm just tired of having to jump through hoops when it comes to string manipulation. I want to be able to say 'this is a character, I don't care what it is. Store it, change it, splice it, print it.' But instead it seems if I don't care what the character type it, it might not fit. Then I have to allocate then store it, find and change it, locate then splice it, convert then print it. Small annoyances build up over time and I'm pretty sure I'm not insured for blood vessels bursting in my eye. I live in the hope that one day in the future I'll see something magical happen, and I'll be able to type char chr = 'Δ'; and chr will be a proper utf-8 character that I can print, insert into an array, and change. What a beautiful day that will be. Welp, I think I'm done ranting for now. Back to screwing around with strings. Or more accurately, procrastinating about screwing around with strings.
Jan 20 2008
parent "Kris" <foo bar.com> writes:
Jarrod: you might find something useful in the way the tango Text class 
operates? It attempts to make the common operation independent of indexing, 
in order to avoid some of these unit/point problems.


"Jarrod" <qwerty ytre.wq> wrote in message 
news:fmv7s7$76h$2 digitalmars.com...
 On Sun, 20 Jan 2008 08:04:01 +0000, Janice Caron wrote:

 I'm allowing the user to edit config
 files

How? With a GUI interface? With a program written in D? With their favorite text editor of choice? If the latter, then you cannot be sure of the encoding, and that's hardly D's fault!

It is the latter.
 Right, but converting from one encoding to another is the job of
 specialised classes. Detecting whether a text file is in ISO-8859-1, or
 Windows-1252, or MAC-ROMAN, or whatever, is not a trivial task. If your
 application were going to do that, you'd have to provide the
 implementation. (Or possibly Tango or some other third party library
 already provides such converters - I don't know). In any case, it's not
 a common enough task to warrant built-in language support.

 But I still don't see what this has got to do with whether or not a[n]
 should identify the (n+1)th character rather than the (n+1)th code unit.

Because this issue isn't really to do with the input file itself, it's to do with the potential input characters given in the file. As far as I can tell (I'm using a C library to parse the input) it should be ascii or UTF-8 encoding. Anything else would probably cause the C lexer to screw up.
 Cool. So what is the real world use case that necessitates that
 sequences of UTF-8 code units must be addressable by character index as
 the default?

The most important one right now is splicing. I'm allowing both user- defined and program-defined macros in the input data. They can be anywhere within a string, so I need to splice them out and replace them with their correct counterparts. I hear the std lib provided with D is unreliable so I'm unwilling to use it. Plus even if it is fixed up I'd hate to limit string manipulation to regular expressions. I also wish to cut off input at a certain letter count for spacing issues in both the GUI and dealing with the webscript. I'll have to be converting certain characters to their URI equivalent too, that will probably take more splicing as well. The other thing I'm using is single-letter replacement. Simple stuff like capitalising letters and replacing spaces with underscores. I can think of a lot of other situations that would benefit from proper multibyte support too, for instance practically any application that takes frequent user input could benefit. A text editor is a very good example. Any coders who don't natively deal with Latin text would probably benefit greatly too ( think of the poor Japanese coders :< ). I've seen a lot of programs that print a specified number of characters before wrapping around or trailing off, too. The humble gnome console is a good example of that. Very handy to have character indexing in this case. String tokenizing and plain old character counting are two operations I can think of that could probably be done easier too. In the end I think I'm just tired of having to jump through hoops when it comes to string manipulation. I want to be able to say 'this is a character, I don't care what it is. Store it, change it, splice it, print it.' But instead it seems if I don't care what the character type it, it might not fit. Then I have to allocate then store it, find and change it, locate then splice it, convert then print it. Small annoyances build up over time and I'm pretty sure I'm not insured for blood vessels bursting in my eye. I live in the hope that one day in the future I'll see something magical happen, and I'll be able to type char chr = '?'; and chr will be a proper utf-8 character that I can print, insert into an array, and change. What a beautiful day that will be. Welp, I think I'm done ranting for now. Back to screwing around with strings. Or more accurately, procrastinating about screwing around with strings.

Jan 20 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 1/20/08, Jarrod <qwerty ytre.wq> wrote:
 But I still don't see what this has got to do with whether or not a[n]
 should identify the (n+1)th character rather than the (n+1)th code unit.

Because this issue isn't really to do with the input file itself, it's to do with the potential input characters given in the file.

You mean the plain text config file of unknown encoding?
 As far as I can
 tell (I'm using a C library to parse the input) it should be ascii or
 UTF-8 encoding.
 Anything else would probably cause the C lexer to screw up.

If it's an unknown encoding, you store it in a ubyte array. Then you identify the encoding, convert it to UTF-8 and store the result in a char array.
 Cool. So what is the real world use case that necessitates that
 sequences of UTF-8 code units must be addressable by character index as
 the default?

The most important one right now is splicing. I'm allowing both user- defined and program-defined macros in the input data. They can be anywhere within a string, so I need to splice them out and replace them with their correct counterparts.

That works right now with ordinary char arrays. Just use find(), rfind(), etc., and slicing.
 I hear the std lib provided with D is
 unreliable

Huh? Please elucidate.
 so I'm unwilling to use it.

That's your loss, but you can hardly expect Walter to consider adding new language features just because you are unwilling to use Phobos.
 I also wish to cut off input at a certain letter count for spacing issues
 in both the GUI and dealing with the webscript.

Well, I hate to spoil things, but even /characters/ are not sufficient to help you figure out spacing issues. For that, you need to be working on the level of /glyphs/. For example, consider the word "caf". (Just in case that didn't render properly, that's c, a, f, followed be e-with-an-acute-accent). You can write this as either caf\u00E9 which consists of five UTF-8 code units, or four characters, or four glyphs; or you can write it as cafe\u0301 which consist of seven UTF-8 code units, or five characters, or four glyphs. In the first case, the e-acute glyph is represented as a single character; in the second case, it is represented as an e character followed by a combining-acute character. In other words, even indexing by character is not sufficient to achieve your goals. You need to index by glyph. At some point, you have to say to yourself: wait a minute - this would be better implemented in a library than in the primitive types of the language.
 The other thing I'm using is single-letter replacement. Simple stuff like
 capitalising letters and replacing spaces with underscores.

I guess what you're getting at here is that uppercasing a character might result in a UTF-8 string longer than that of the original character. And so it might. On the other hand, if you use a foreach loop to do this sort of thing, your problems are solved.
 I can think of a lot of other situations that would benefit from proper
 multibyte support too,

UTF-8 support /is/ proper multibyte support. That's why D has it built in.
 ( think of the poor Japanese coders :< ).

Which is why D uses Unicode. Again, I say, D got it right.
 I've seen a lot of programs that print a specified number of characters
 before wrapping around or trailing off, too. The humble gnome console is
 a good example of that. Very handy to have character indexing in this
 case.

I don't agree. This is a problem in font rendering. If you happen to be using a proportional font, then even character counting won't work. You need to be counting rendered width in pixels - an operation which should be generic enough to work for both fixed-width and proportional fonts.
 In the end I think I'm just tired of having to jump through hoops when it
 comes to string manipulation. I want to be able to say 'this is a
 character, I don't care what it is. Store it, change it, splice it, print
 it.'

dchar.
 happen, and I'll be able to type char chr = ''; and chr will be a proper
 utf-8 character that I can print, insert into an array, and change.
 What a beautiful day that will be.

dchar. Put another way, you want to be insulated from the internal representation. UTF-8 is an implementation detail, wheras what you want is an array of Unicode characters (whose implementation is not necessarily dchar[] but you want to be shielded from it anyway). Again I say, this is a problem for a library class, not a builtin type. And you're probably going to want even higher level abstractions dealing with glyphs too (and then font-rendering tools after that). D allows you to write such libraries. But the builtin types do exactly what is says on the tin. Their behaviour is well-defined, and it's up to the programmer to understand that behaviour.
Jan 20 2008
prev sibling next sibling parent Jarrod <qwerty ytre.wq> writes:
On Sun, 20 Jan 2008 11:45:40 +0000, Janice Caron wrote:

 I hear the std lib provided with D is unreliable

Huh? Please elucidate.

Ah, mistyped my thoughts there. I meant to say std.regexp* At the moment, to me char[] is just byte[], except I guess when it comes to foreach. Pretty watered down when you look at it like that. And yes, I guess a library implementation of string would be fine too. I just figure since strings are one of the most important data types used in any program, I figure D should probably natively support their multi- byte nature more transparently. Perhaps I have been spoiled by scripting languages after leaving C/++ alone for so long, but it would be very nice to see it happen one way or another. I guess I like my language how I like my coffee; filled with sugar.
Jan 20 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 1/20/08, James Dennett <jdennett acm.org> wrote:
 Looks very different to me.

I thought it looked very similar indeed to D, but there you go. Funny how two different people can read the same document and interpret it in two different ways.
 There's no conflation of char with a
 code unit of UTF8

C has no ubyte type. Since time immemorial, C programmers have been using the char type to store every 8-bit wide data type under the sun simply because there's been no alternative (until recently, when int8_t showed up as a typedef for char). That's not a big deal.
 (and indeed C++ deliberately supports use of
 varied encodings for multi-byte characters).

I must have misread the heading that says "Require UTF", and whose text reads "The C TR makes the encoding of char16_t and char32_t implementation-defined. It also provides macros to indicate whether or not the encoding is UTF. In contrast, this proposal requires UTF encoding." Oh, I see what you're saying - C++ would require UTF for wchar and dchar, but not for char. Well, that's historical legacy for you.
 Yes, C++ is adding
 16- and 32-bit character types which are more akin to D's, but that
 has little bearing on how differently it handles multi-byte (as
 opposed to wide-character) strings.

So it has a bunch of procedural functions instead of foreach. Apart from that, the approach seems the same as D. Where's the difference?
Jan 20 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On Jan 20, 2008 11:01 PM, James Dennett <jdennett acm.org> wrote:
 That view has lead to many security issues, where different
 software reacts differently to byte strings which are not
 valid UTF-8 in places where UTF-8 is expected.

Such input should always be rejected. D will throw an exception, which is the right thing to do. If a programmer wants to be more flexible, they can always catch the exception and delete invalid sequences. Security issues have arisen as a result of what are called non-shortest-sequences. For example, the slash character is represented in UTF-8 as 2F. Some hackers have attempted to get past certain filters by representing the slash character as C0 AF. This is not valid UTF-8 (because UTF-8 forbids non-shortest sequences), but a buggy implementation might get that wrong and interpret that as '\u002F'. The important point that I want to make here is that *D GETS IT RIGHT*. D's implementation will throw an exception on all invalid UTF-8 sequences, and this will block all such security issues. The only way they can resurface is if you hand-code your own UTF handling. So long as you stick to the built-in UTF-handling stuff which D provides, you will not encounter these security issues. Other security issues arise as a result of Unicode itself, not UTF-8. This is because Unicode is such a large character set - which makes it really good for phishing attacks. After all, if you spell "amazon" with the greek letter lowercase omicron instead of latin lowercase o, who's going to notice. However, this is not D's problem - it's a problem for browser writers, and one they will encounter regardless of what programming language they use. (Similar issues arise if browsers fail to convert URLs to Normalisation Form C, but again, that would be a browser problem, not a D problem. It would also be a bug).
 (char doesn't represent a character in D.  Not great
 naming?

It's reasonable naming, given that UTF-8 code units in the range 00 to 7F do, in fact, correspond to (ASCII) characters. OK, so it's inappropriately named for holding values 80 to FF, but alternatives such as codeunit, or utf8, would probably not catch on so easily.
 But let's get more concrete:
 suppose D code finds that an alleged char[] passed to it is, in
 fact, broken (i.e., violates the UTF8 invariants).  What should
 it do -- abort, throw an exception, offer a policy for handling
 such bugs, other?

It should, and does, throw an exception. Your program may catch the exception, but it should reject the input.
Jan 21 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On Jan 21, 2008 8:11 AM, Janice Caron <caron800 googlemail.com> wrote:
 But let's get more concrete:
 suppose D code finds that an alleged char[] passed to it is, in
 fact, broken (i.e., violates the UTF8 invariants).  What should
 it do -- abort, throw an exception, offer a policy for handling
 such bugs, other?

It should, and does, throw an exception. Your program may catch the exception, but it should reject the input.

In fact, this goes to the heart of almost all modern securty problems (SQL injection, buffer overruns, etc.). The golden rule is that *ALL* untrusted input must be sanitised. Every time you don't do that, you provide an opportunity for hackers. But at least in the case of UT, it's easy - just let D validate it. If it doesn't validate, throw it out.
Jan 21 2008
prev sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On Jan 21, 2008 8:47 AM, Matti Niemenmaa <see_signature for.real.address> wrote:
 Janice Caron wrote:
 The important point that I want to make here is that *D GETS
 IT RIGHT*. D's implementation will throw an exception on all invalid
 UTF-8 sequences, and this will block all such security issues.

Well, mostly right: http://d.puremagic.com/issues/show_bug.cgi?id=978

Oooh - well spotted! In that case, I amend my statement. D /will/ get it right, once this bug is fixed. One would hope that said bug will be fixed in the next release.
Jan 21 2008