www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Encoding and doFormat

reply Sean Kelly <sean f4.ca> writes:
Is there any instance where we might want to output UTF-16 or UTF-32 encoded
strings from doFormat?  Alternately, is there any instance where we might want
to read UTF-16 or UTF-32 encoded streams?  I've been kicking around how best to
handle this aspect of unFormat and was suddenly struck by the UTF-8 limitation
in the current implementation of doFormat.


Sean
Jul 20 2004
next sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Sean Kelly" <sean f4.ca> wrote in message
news:cdjpll$1pks$1 digitaldaemon.com...
 Is there any instance where we might want to output UTF-16 or UTF-32

 strings from doFormat?  Alternately, is there any instance where we might

 to read UTF-16 or UTF-32 encoded streams?  I've been kicking around how

 handle this aspect of unFormat and was suddenly struck by the UTF-8

 in the current implementation of doFormat.

doFormat() isn't limited by UTF-8. In fact, it's output is in dchar's, which are UTF-32. Look at std.stdio.writef(), it writes its output based on what the stream's format is set to.
Jul 20 2004
parent reply Sean Kelly <sean f4.ca> writes:
In article <cdk2sj$1tvi$1 digitaldaemon.com>, Walter says...
doFormat() isn't limited by UTF-8. In fact, it's output is in dchar's, which
are UTF-32. Look at std.stdio.writef(), it writes its output based on what
the stream's format is set to.

Oops. I misread part of the documentation on std.format where it was talking about printing portions of the format string. So input and output is in dchars? That makes life quite easy. I guess unFormat is pretty much finished then. Sean
Jul 20 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"Sean Kelly" <sean f4.ca> wrote in message
news:cdk425$1uid$1 digitaldaemon.com...
 In article <cdk2sj$1tvi$1 digitaldaemon.com>, Walter says...
doFormat() isn't limited by UTF-8. In fact, it's output is in dchar's,


are UTF-32. Look at std.stdio.writef(), it writes its output based on


the stream's format is set to.

Oops. I misread part of the documentation on std.format where it was

 about printing portions of the format string.  So input and output is in

 That makes life quite easy.  I guess unFormat is pretty much finished

Input is chars, wchars, or dchars.
Jul 20 2004
parent reply Sean Kelly <sean f4.ca> writes:
Walter wrote:
 
 Input is chars, wchars, or dchars.

Right, because all char types can be implicitly cast to dchar, correct? Sean
Jul 20 2004
next sibling parent "Walter" <newshound digitalmars.com> writes:
"Sean Kelly" <sean f4.ca> wrote in message
news:cdkm15$25m7$1 digitaldaemon.com...
 Walter wrote:
 Input is chars, wchars, or dchars.

Right, because all char types can be implicitly cast to dchar, correct?

Not exactly. doFormat() examines the type of each argument, and does any conversions as necessary. Implicit conversions are not necessary.
Jul 20 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cdkm15$25m7$1 digitaldaemon.com>, Sean Kelly says...
Walter wrote:
 
 Input is chars, wchars, or dchars.

Right, because all char types can be implicitly cast to dchar, correct?

They can be implicitly cast, but they cannot be /correctly/ cast. I have mentioned this before (and suggested that it be considered a bug) but Walter was adamant that the runtime overhead involved in checking would be undesirable. The problem can be demonstrated by example. Suppose you cast a char containing UTF-8-fragment 0xC0 (which would ordinarily be the first byte of a two-byte UTF-8 sequence) into a dchar then it will be erroneously converted to U+00C0, instead of (as I would prefer) throwing a UTF conversion exception. In general, char values >0x7F should not be cast to wchars or dchars, because these values are /not characters/. Arcane Jill
Jul 21 2004
next sibling parent reply Derek Parnell <derek psych.ward> writes:
On Wed, 21 Jul 2004 07:24:42 +0000 (UTC), Arcane Jill wrote:

 In article <cdkm15$25m7$1 digitaldaemon.com>, Sean Kelly says...
Walter wrote:
 
 Input is chars, wchars, or dchars.

Right, because all char types can be implicitly cast to dchar, correct?

They can be implicitly cast, but they cannot be /correctly/ cast. I have mentioned this before (and suggested that it be considered a bug) but Walter was adamant that the runtime overhead involved in checking would be undesirable. The problem can be demonstrated by example. Suppose you cast a char containing UTF-8-fragment 0xC0 (which would ordinarily be the first byte of a two-byte UTF-8 sequence) into a dchar then it will be erroneously converted to U+00C0, instead of (as I would prefer) throwing a UTF conversion exception. In general, char values >0x7F should not be cast to wchars or dchars, because these values are /not characters/. Arcane Jill

(Jill, I'm not critizing, disputing, argueing, being ornory, etc... I'm just trying to understand UTF better and I think you'd be one the best sources at the moment) If D char variables are supposed to be UTF-8 characters, then why does D allow a char to contain non-UTF-8 bit patterns (eg. a UTF-8-fragment)? I can see that a byte could, but a char? Or is D a bit simple here? -- Derek Melbourne, Australia 21/Jul/04 5:42:59 PM
Jul 21 2004
parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <cdl74j$2ea7$1 digitaldaemon.com>, Derek Parnell says...

If D char variables are supposed to be UTF-8 characters, then why does D
allow a char to contain non-UTF-8 bit patterns (eg. a UTF-8-fragment)? I
can see that a byte could, but a char? Or is D a bit simple here?

Oh that's easy to answer. Okay, first off, there is no such thing as a "UTF-8 character". UTF-8 is an encoding of Unicode, so there is only a "Unicode character", which may be encoded in UTF-8 as a multi-byte sequence. So a char, in fact, can /only/ contain a UTF-8 fragment. Fortunately, there are some UTF-8 multibyte sequences which are, in fact, exactly one byte long. The Unicode characters represented by such one-byte-sequences are the characters U+0000 to U+007F inclusive - in other words, ASCII. UTF-8 was designed that way on purpose, to maintain compatibility with ASCII. Thus, if a char contains a value in the range 0x00 to 0x7F inclusive then it may be interpretted either as an ASCII character or a "one-byte UTF-8 fragment which happens to represent a complete Unicode character". Both interpretations are equally valid and interchangable. On the other hand, if a char contains a value in the range 0x80 to 0xF8 then it can /only/ be a UTF-8 fragment, since these bytes form part of multibyte sequences which are /at least/ two bytes long, and so cannot be equated with a single character. (Values in the range 0xF9 to 0xFF are completely meaningless. That's one reason why char.init is 0xFF). To answer your question: "Why does D allow a char to contain non-UTF-8 bit patterns" - for the same reason that it allows a dchar to conatain non-UTF-32 bit patterns - it's simply a platform-native integer type. Constraining char to contain values in the range 0x00 to 0xF8 (or constraining dchar to values in the range 0x00000000 to 0x0010FFFF excluding 0x0000D800 to 0x0000DFFF) would add run-time overhead that is simply not necessary. If I have misunderstood your question, and you were actually intending to ask "Why does D allow a char to contain UTF-8 fragments which cannot be interpretted in isolation" then the answer has to be that char exists so that char[] can exist. Only in a /string/ does UTF-8 make any real sense. A string needs an array, and an array has to be an array of /something/. char is that something. Any help? Arcane Jill
Jul 21 2004
prev sibling parent Sean Kelly <sean f4.ca> writes:
Arcane Jill wrote:

 In article <cdkm15$25m7$1 digitaldaemon.com>, Sean Kelly says...
 
Walter wrote:

Input is chars, wchars, or dchars.

Right, because all char types can be implicitly cast to dchar, correct?

They can be implicitly cast, but they cannot be /correctly/ cast. I have mentioned this before (and suggested that it be considered a bug) but Walter was adamant that the runtime overhead involved in checking would be undesirable. The problem can be demonstrated by example. Suppose you cast a char containing UTF-8-fragment 0xC0 (which would ordinarily be the first byte of a two-byte UTF-8 sequence) into a dchar then it will be erroneously converted to U+00C0, instead of (as I would prefer) throwing a UTF conversion exception. In general, char values >0x7F should not be cast to wchars or dchars, because these values are /not characters/.

Oops, right. What unFormat does is read everything into dchars then convert to UTF-8 before writing to a char array or to UTF-16 before writing to a wchar array. The missing piece is converting from UTF-8 or UTF-16 when reading, which should be done in a day or two--I decided to rewrite the utf routines to allow a put/get delegate. Sean
Jul 21 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cdjpll$1pks$1 digitaldaemon.com>, Sean Kelly says...
Is there any instance where we might want to output UTF-16 or UTF-32 encoded
strings from doFormat?  Alternately, is there any instance where we might want
to read UTF-16 or UTF-32 encoded streams?  I've been kicking around how best to
handle this aspect of unFormat and was suddenly struck by the UTF-8 limitation
in the current implementation of doFormat.

Sean

Conventionally, streams are considered to be a sequence of octets (bytes). So, by that reasoning, you would never want to read or write UTF-16 or UTF-32 to/from a stream because the units of those formats are not eight bits wide. However, the formats UTF-16LE, UTF-16BE, UTF-32LE and UTF-32BE are eight-bit-wide standards, merely consisting of UTF-16 and UTF-32 in little-endian and big-endian representation respectively. You would certainly expect to encounter these. Text files can use them, for example. Going beyond streams, there are also "wide" streams, called Readers and Writers in Java, and maybe filters more generically. They exist for such purposes as transcoding, uppercasing, etc., but don't, in general, send their data to consoles, files, sockets, etc (because those devices expect 8-bit wide input). Transcoders of course are capable of transforming a "wide" stream to a normal stream. A UTF-16LE encoder, for example, is utterly trivial. I don't know what doFormat does, so I can't answer your specific question. I can say, though, that if you're processing bytes or ubytes, you don't need to bother with UTF-32 or UTF-16 (or even UTF-8). If you're processing characters, however, you may well be better off keeping everything in dchars throughout. I don't know if that's helpful or not. What does doFormat() do? What's it for? (I should probably know that, but I'm not that on-the-case right now). Arcane Jill
Jul 20 2004
parent reply Sean Kelly <sean f4.ca> writes:
In article <cdk49v$1ump$1 digitaldaemon.com>, Arcane Jill says...
In article <cdjpll$1pks$1 digitaldaemon.com>, Sean Kelly says...
Is there any instance where we might want to output UTF-16 or UTF-32 encoded
strings from doFormat?  Alternately, is there any instance where we might want
to read UTF-16 or UTF-32 encoded streams?  I've been kicking around how best to
handle this aspect of unFormat and was suddenly struck by the UTF-8 limitation
in the current implementation of doFormat.

Sean

Conventionally, streams are considered to be a sequence of octets (bytes). So, by that reasoning, you would never want to read or write UTF-16 or UTF-32 to/from a stream because the units of those formats are not eight bits wide. However, the formats UTF-16LE, UTF-16BE, UTF-32LE and UTF-32BE are eight-bit-wide standards, merely consisting of UTF-16 and UTF-32 in little-endian and big-endian representation respectively. You would certainly expect to encounter these. Text files can use them, for example. Going beyond streams, there are also "wide" streams, called Readers and Writers in Java, and maybe filters more generically. They exist for such purposes as transcoding, uppercasing, etc., but don't, in general, send their data to consoles, files, sockets, etc (because those devices expect 8-bit wide input). Transcoders of course are capable of transforming a "wide" stream to a normal stream. A UTF-16LE encoder, for example, is utterly trivial. I don't know what doFormat does, so I can't answer your specific question. I can say, though, that if you're processing bytes or ubytes, you don't need to bother with UTF-32 or UTF-16 (or even UTF-8). If you're processing characters, however, you may well be better off keeping everything in dchars throughout. I don't know if that's helpful or not. What does doFormat() do? What's it for? (I should probably know that, but I'm not that on-the-case right now).

doFormat handles all the work for writef. And while writef is really for console and file output, doFormat can do in-memory string formatting as well. It turns out I miswrote my original question, as writef uses the wide char output routines but doFormat deals entirely in dchars. So I think it would be safe for me to have readf use wide char input routines and leave unFormat as-is. This puts aside the issue of configurable encoding however, but from what you've said perhaps that isn't much of a problem. Sean
Jul 20 2004
parent Sean Kelly <sean f4.ca> writes:
In article <cdk74m$1vqj$1 digitaldaemon.com>, Sean Kelly says...
It turns out I miswrote my original question, as writef uses the wide char
output routines but doFormat deals entirely in dchars.  So I think it would be
safe for me to have readf use wide char input routines and leave unFormat as-is.

That's what I get for posting hastily. writef actually seems to switch between UTF-8 and (possibly) UTF-16 based on information gleaned from the file pointer. Looks like I'm going to have to play with readf a little more, though I'm not looking forward to handling unget. Stinkin multibyte encoding schemes. Sean
Jul 20 2004