www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Unified String Theory..

reply "Regan Heath" <regan netwin.co.nz> writes:
------------SRJ5HVj2hz31rAqYWOgFlA
Content-Type: text/plain; format=flowed; delsp=yes; charset=iso-8859-15
Content-Transfer-Encoding: 8bit

With the recent Physics slant on some posts here I couldn't resist that  
subject, in actual fact this is an idea for string handling in D which I  
have cooked up recently.

I am going to paste the text here and attach my original document, the  
document may be easier to read than the NG.

I like this idea, it may however be too much of a change for D, I'm hoping  
the advantages outweigh this fact but I'm not going to hold my breath.

It is possible I have missed something obvious and/or am talking out of a  
hole in my head, if that is the case I would appreciate being told so,  
politely ;)

Enough rambling, here is it, be nice!

-----

Proposal: A single unified string type.
Author  : Regan Heath
Version : 1.0a
Date    : 24 Nov 2005 +1300 (New Zealand DST)

[Preamble/Introduction]
After the recent discussion on Unicode, UTF encodings and the current D  
situation it occured to me that many of the issues D has with strings  
could be side-stepped if there was a single string type.

In the past we have assumed that to obtain this we have to choose one of  
the 3 available types and encodings. This wasn't an attractive option  
because each type has different pros/cons and each application may prefer  
one type over another. Another suggested solution was a string class which  
hides the details, this solution suffers from being a class and the  
limitations imposed by that and not being tied directly into the language.

My proposal is a single "string" type built into the language, which can  
represent it's string data in any given UTF encoding. Which will allow  
slicing of "characters" as opposed to what is essentially bytes, shorts,  
and ints. Whose default encoding can be selected at compile time, or  
specified at runtime. Which will implicitly or explicitly transcode where  
required.

There are some requirements for this to be possible, namely knowledge of  
the UTF encodings being built into D, these requirements may prohibit the  
proposal being favourable as it increases the knowledge required to write  
a D compiler. However it occurs to me that DMD and thus D? already  
requires a fair bit of UTF knowledge.


[Key]
First, lets start with some terminology, these are the terms I am going to  
be using and what they mean, if these are incorrect please correct me, but  
take them to have the stated meanings for this document.

code point      := the unicode value for a single and complete character.
code unit       := part of, or a complete character in one of the 3 UTF  
encodings UTF-8,16,32.
code value      := AKA code unit.
transcoding     := the process of converting from one encoding to another.
source          := a file, the keyboard, a tcp socket, a com port, an OS/C  
function call, a 3rd party library.
sink            := a file, the screen, a tcp socket, a com port, an OS/C  
function call, a 3rd party library.
native encoding := application specific "preferred" encoding (more on this  
later)
string          := a sequence of code points.

Anything I am unsure about will be suffixed with (x) where x is a letter  
of the alphabet, and my thoughts will be detailed in the [Questions]  
section.


[Assumptions]
These are what I base my argument/suggestion on, if you disagree with any  
of these you will likely disagree with the proposal. If that is the case  
please post your concerns with any given assumption in it's own post (I  
would like to discuss each issue in it's own thread and avoid mixing  
several issues)

#1: Any given string can be represented in any UTF encoding, it can be  
transcoded to/from any UTF encoding with no loss of data/meaning.

#2: Transcoding has a performance penalty at runtime. This proposal will  
mention the possible runtime penalty wherever appropriate.

#3: There are 2 places where transcoding cannot be avoided; input and  
ouput. Input is the process of obtaining data from a source. Output is the  
process of sending data to a sink. In either case the source or sink will  
have a fixed encoding and is that encoding does not match the native  
encoding the application will need to transcode. (see definitions above  
for what classifies as a source or sink)

#4: String literals can be stored in the binary in any encoding (#1) the  
encoding chosen may have repercusions at runtime (#2 & #3).


[Details]
Many of the details are flexible, i.e. the names of the types etc, the  
important/inflexible details are how it all fits together and achieves  
it's results. I've chosen a bullet point format and tried to make each  
change/point as succint and clear as possible. Feel free to ask for  
clarification on any point or points. Or to ask general questions. Or to  
pose general problems. I will do my best to answer all questions.

* remove char[], wchar[] and dchar[].

* add a new type "string". "string" will store code points in the  
application specific native encoding and be implicitly or explicitly  
transcoded as required (more below).

* the application specific native encoding will default to UTF-8. An  
application can choose another with a compile option or pragma, this  
choice will have no effect on the behaviour of the program (as we only  
have 1 type and all transcoding is handled where required) it will only  
affect performance.

The performance cost cannot be avoided, presuming it is only being done at  
input and output (which is part of what this proposal aims to achieve).  
This cost is application specific and will depend on the tasks and data  
the application is designed to perform and use.

Given that, letting the programmer choose a native encoding will allow  
them to test different encodings for speed and/or provide different builds  
based on the target language, eg an application destined to be used with  
the Japanese language would likely benefit from using UTF-32  
internally/natively.

* keep char, wchar, and dchar but rename them utf8, utf16, utf32. These  
types represent code points (always, not a code units/values) in each  
encoding. Only code points that fit in utf8 will ever be represented by  
utf8, and so on. Thus some code points will always be utf32 values and  
never utf8 or 16. (much like byte/short/int)

* add promotion/comparrison rules for utf8, 16 and 32:

- any given code point represented as utf8 will compare equal to the same  
code point represented as a utf16 or utf32 and vice versa(a)

- any given code point represented as utf8 will be implicitly  
converted/promoted to the same code point represented as utf16 or utf32 as  
required and vice versa(a). If promotion from utf32 to utf16 or 8 causes  
loss in data it should be handled just like int to short or byte.

* add a new type/alias "utf", this would alias utf8, 16 or 32. It  
represents the application specific native encoding. This allows efficient  
code, like:

string s = "test";
foreach(utf c; s) {
}

regardless of the applications selected native encoding.

* slicing string gives another string

* indexing a string gives a utf8, 16, or 32 code point.

* string literals would be of type "string" encoded in the native  
encoding, or if another encoding can be determined at compile time, in  
that encoding (see ASCII example below).

* character literals would default to the native encoding, failing that  
the smallest possible type, and promoted/converted as required.

* there are occasions where you may want to use a specific encoding for a  
part of your application, perhaps you're loading a UTF-16 file and parsing  
it. If all the work is done in a small section of code and it doesn't  
interact with the bulk of your application data which is all in UTF-8 your  
native encoding it likely to be UTF-8.

In this case, for performance reasons, you want to be able to specify the  
encoding to use for your "string" types at runtime, they are exceptions to  
the native encoding. To do this we specify the encoding at  
construction/declaration time, eg.

string s(UTF16);
s.utf16 = ..data read from UTF-16 source..

(or similar, the exact syntax is not important at this stage)

thus...

* the type of encoding used by "string" should be selectable at runtime,  
some sort of encoding type flag must exist for each string at runtime,  
this is starting to head into "implementation details" which I want to  
avoid at this point, however it is important to note the requirement.


[Output]
* the type "char" will still exist, it will now _only_ represent a C  
string, thus when a string is passed as a char it can be implicitly  
transcoded into ASCII(b) with a null terminator, eg.

int strcmp(const char *src, const char *dst);

string test = "this is a test";
if (strcmp(test,"this is a test")==0) { }

the above will implicitly transcode 'test' into ASCII and ensure there is  
a null terminator. The literal "this is a test" will likely be stored in  
the binary as ASCII with a null terminator.

* Native OS functions requiring "char" will use the rule above. eg.

CreateFileA(char *filename...

* Native OS functions requiring unicode will be defined as:

CreateFileW(utf16 *filename...

and "string" will be implicitly transcoded to utf16, with a null  
terminator added..

* When the required encoding is not apparent, eg.

void CreateFile(char *data) { }
void CreateFile(utf16 *data) { }

string test = "this is a test";
CreateFile(test);

an explicit property should be used, eg.

CreateFile(test.char);
CreateFile(test.utf16);

NOTE: this problem still exists! It should however now be relegated to  
interaction with C API's as opposed to happening for native D methods.


[Input]
* Old encodings, Latin-1 etc would be loaded into ubyte[] or byte[] and  
could be cast (painted) to char*, utf8*, 16 or 32 or converted to "string"  
using a routine i.e. string toStringFromXXX(ubyte[] raw).

* A stream class would have a selectable encoding and hide these details  
 from us handling the data and giving a natively encoded "string" instead.  
Meaning, transcoding will naturally occur on input or output where  
required.


[Example application types and the effect of this change]

* the quick and dirty console app which handles ASCII only. It's native  
encoding will be UTF-8, and no transcoding will ever need to occur  
(assuming none of it's input or output is in another encoding)

* an app which loads files in different encodings and needs to process  
them efficiently. In this case the code can select the encoding of  
"string" at runtime and avoid transcoding the data until such time as it  
needs to interface with another part of the application in another  
encoding or it needs to output to a sink, also in another encoding.

* an international app which will handle many languages. this app can be  
custom built with the native string type selected to match each language.


[Advantages]
As I see it, this change would have the following advantages:

* "string" requires no knowledge of UTF encodings (and the associated  
problems) to use making it easy for begginners and for a quick and dirty  
program.

* "string" can be sliced/indexed by character regardless of the encoding  
used for the data.

* overload resolution has only 1 type, not 3 to choose from.

* code written in D would all use the same type "string". no more this  
library uses char[] this one wchar and my app dchar[] problems.


[Disadvantages]
* requirements listed below

* libraries built for a different native type will likely cause  
transcoding. This problem already exists, at least with this suggestion  
the library can be built 3 times, once for each native encoding and the  
correct one linked to your app.

* possibility of implicit and silent transcoding. This can occur between  
libraries built with different native encodings and between "string" and  
char*, utf8*, utf16* and utf32*, the compiler _could_ identify all such  
locations if desired.


[Requirements]
In order to implement all this "string" requires knowledge of all code  
points, how they are encoded in the 3 encodings and how to compare and  
convert between them. So, D and thus any D compiler eg DMD, requires this  
knowledge. I am not entirely sure just how big an "ask" this is. I believe  
DMD and thus D already has much of this capability built in.


[Questions]
(a) Is UTF-8 a subset of UTF-16 and so on? does the codepoint for 'A' have  
the numerical value 65 decimal in UTF-8, UTF-16 _and_ UTF-32, in other  
words is it the same numerical value in all encodings? If so then  
comparing utf8, 16 and 32 is no different to comparing byte, short and int  
and all the same promotion and comparrison rules can apply.

(b) Is this really ASCII or is it system dependant? i.e. Latin-1 or  
similar. Is it ASCII values 127 or less perhaps? To be honest I'm not sure.



------------SRJ5HVj2hz31rAqYWOgFlA
Content-Disposition: attachment; filename="D and UTF.txt"
Content-Type: text/plain; name="D and UTF.txt"
Content-Transfer-Encoding: 8bit

Proposal: A single unified string type.
Author  : Regan Heath
Version : 1.0a
Date    : 24 Nov 2005 +1300 (New Zealand DST)

[Preamble/Introduction]
After the recent discussion on Unicode, UTF encodings and the current D
situation it occured to me that many of the issues D has with strings could be
side-stepped if there was a single string type. 

In the past we have assumed that to obtain this we have to choose one of the 3
available types and encodings. This wasn't an attractive option because each
type has different pros/cons and each application may prefer one type over
another. Another suggested solution was a string class which hides the details,
this solution suffers from being a class and the limitations imposed by that
and not being tied directly into the language.

My proposal is a single "string" type built into the language, which can
represent it's string data in any given UTF encoding. Which will allow slicing
of "characters" as opposed to what is essentially bytes, shorts, and ints.
Whose default encoding can be selected at compile time, or specified at
runtime. Which will implicitly or explicitly transcode where required.

There are some requirements for this to be possible, namely knowledge of the
UTF encodings being built into D, these requirements may prohibit the proposal
being favourable as it increases the knowledge required to write a D compiler.
However it occurs to me that DMD and thus D? already requires a fair bit of UTF
knowledge.


[Key]
First, lets start with some terminology, these are the terms I am going to be
using and what they mean, if these are incorrect please correct me, but take
them to have the stated meanings for this document.

code point      := the unicode value for a single and complete character.
code unit       := part of, or a complete character in one of the 3 UTF
encodings UTF-8,16,32.
code value      := AKA code unit.
transcoding     := the process of converting from one encoding to another.
source          := a file, the keyboard, a tcp socket, a com port, an OS/C
function call, a 3rd party library.
sink            := a file, the screen, a tcp socket, a com port, an OS/C
function call, a 3rd party library.
native encoding := application specific "preferred" encoding (more on this
later)
string          := a sequence of code points.

Anything I am unsure about will be suffixed with (x) where x is a letter of the
alphabet, and my thoughts will be detailed in the [Questions] section.


[Assumptions]
These are what I base my argument/suggestion on, if you disagree with any of
these you will likely disagree with the proposal. If that is the case please
post your concerns with any given assumption in it's own post (I would like to
discuss each issue in it's own thread and avoid mixing several issues)

#1: Any given string can be represented in any UTF encoding, it can be
transcoded to/from any UTF encoding with no loss of data/meaning.

#2: Transcoding has a performance penalty at runtime. This proposal will
mention the possible runtime penalty wherever appropriate.

#3: There are 2 places where transcoding cannot be avoided; input and ouput.
Input is the process of obtaining data from a source. Output is the process of
sending data to a sink. In either case the source or sink will have a fixed
encoding and is that encoding does not match the native encoding the
application will need to transcode. (see definitions above for what classifies
as a source or sink)

#4: String literals can be stored in the binary in any encoding (#1) the
encoding chosen may have repercusions at runtime (#2 & #3).


[Details]
Many of the details are flexible, i.e. the names of the types etc, the
important/inflexible details are how it all fits together and achieves it's
results. I've chosen a bullet point format and tried to make each change/point
as succint and clear as possible. Feel free to ask for clarification on any
point or points. Or to ask general questions. Or to pose general problems. I
will do my best to answer all questions.

* remove char[], wchar[] and dchar[].

* add a new type "string". "string" will store code points in the application
specific native encoding and be implicitly or explicitly transcoded as required
(more below).

* the application specific native encoding will default to UTF-8. An
application can choose another with a compile option or pragma, this choice
will have no effect on the behaviour of the program (as we only have 1 type and
all transcoding is handled where required) it will only affect performance. 

The performance cost cannot be avoided, presuming it is only being done at
input and output (which is part of what this proposal aims to achieve). This
cost is application specific and will depend on the tasks and data the
application is designed to perform and use. 

Given that, letting the programmer choose a native encoding will allow them to
test different encodings for speed and/or provide different builds based on the
target language, eg an application destined to be used with the Japanese
language would likely benefit from using UTF-32 internally/natively.

* keep char, wchar, and dchar but rename them utf8, utf16, utf32. These types
represent code points (always, not a code units/values) in each encoding. Only
code points that fit in utf8 will ever be represented by utf8, and so on. Thus
some code points will always be utf32 values and never utf8 or 16. (much like
byte/short/int)

* add promotion/comparrison rules for utf8, 16 and 32:

- any given code point represented as utf8 will compare equal to the same code
point represented as a utf16 or utf32 and vice versa(a)

- any given code point represented as utf8 will be implicitly
converted/promoted to the same code point represented as utf16 or utf32 as
required and vice versa(a). If promotion from utf32 to utf16 or 8 causes loss
in data it should be handled just like int to short or byte.

* add a new type/alias "utf", this would alias utf8, 16 or 32. It represents
the application specific native encoding. This allows efficient code, like:

string s = "test";
foreach(utf c; s) {
}

regardless of the applications selected native encoding.

* slicing string gives another string

* indexing a string gives a utf8, 16, or 32 code point.

* string literals would be of type "string" encoded in the native encoding, or
if another encoding can be determined at compile time, in that encoding (see
ASCII example below).

* character literals would default to the native encoding, failing that the
smallest possible type, and promoted/converted as required.

* there are occasions where you may want to use a specific encoding for a part
of your application, perhaps you're loading a UTF-16 file and parsing it. If
all the work is done in a small section of code and it doesn't interact with
the bulk of your application data which is all in UTF-8 your native encoding it
likely to be UTF-8. 

In this case, for performance reasons, you want to be able to specify the
encoding to use for your "string" types at runtime, they are exceptions to the
native encoding. To do this we specify the encoding at construction/declaration
time, eg.

string s(UTF16);
s.utf16 = ..data read from UTF-16 source..

(or similar, the exact syntax is not important at this stage)

thus...

* the type of encoding used by "string" should be selectable at runtime, some
sort of encoding type flag must exist for each string at runtime, this is
starting to head into "implementation details" which I want to avoid at this
point, however it is important to note the requirement.


[Output]
* the type "char" will still exist, it will now _only_ represent a C string,
thus when a string is passed as a char it can be implicitly transcoded into
ASCII(b) with a null terminator, eg.

int strcmp(const char *src, const char *dst);

string test = "this is a test";
if (strcmp(test,"this is a test")==0) { }

the above will implicitly transcode 'test' into ASCII and ensure there is a
null terminator. The literal "this is a test" will likely be stored in the
binary as ASCII with a null terminator.

* Native OS functions requiring "char" will use the rule above. eg.

CreateFileA(char *filename...

* Native OS functions requiring unicode will be defined as:

CreateFileW(utf16 *filename...

and "string" will be implicitly transcoded to utf16, with a null terminator
added..

* When the required encoding is not apparent, eg.

void CreateFile(char *data) { }
void CreateFile(utf16 *data) { }

string test = "this is a test";
CreateFile(test);

an explicit property should be used, eg.

CreateFile(test.char);
CreateFile(test.utf16);

NOTE: this problem still exists! It should however now be relegated to
interaction with C API's as opposed to happening for native D methods.


[Input]
* Old encodings, Latin-1 etc would be loaded into ubyte[] or byte[] and could
be cast (painted) to char*, utf8*, 16 or 32 or converted to "string" using a
routine i.e. string toStringFromXXX(ubyte[] raw).

* A stream class would have a selectable encoding and hide these details from
us handling the data and giving a natively encoded "string" instead. Meaning,
transcoding will naturally occur on input or output where required.


[Example application types and the effect of this change]

* the quick and dirty console app which handles ASCII only. It's native
encoding will be UTF-8, and no transcoding will ever need to occur (assuming
none of it's input or output is in another encoding)

* an app which loads files in different encodings and needs to process them
efficiently. In this case the code can select the encoding of "string" at
runtime and avoid transcoding the data until such time as it needs to interface
with another part of the application in another encoding or it needs to output
to a sink, also in another encoding.

* an international app which will handle many languages. this app can be custom
built with the native string type selected to match each language.


[Advantages]
As I see it, this change would have the following advantages:

* "string" requires no knowledge of UTF encodings (and the associated problems)
to use making it easy for begginners and for a quick and dirty program.

* "string" can be sliced/indexed by character regardless of the encoding used
for the data.

* overload resolution has only 1 type, not 3 to choose from.

* code written in D would all use the same type "string". no more this library
uses char[] this one wchar and my app dchar[] problems.


[Disadvantages]
* requirements listed below

* libraries built for a different native type will likely cause transcoding.
This problem already exists, at least with this suggestion the library can be
built 3 times, once for each native encoding and the correct one linked to your
app.

* possibility of implicit and silent transcoding. This can occur between
libraries built with different native encodings and between "string" and char*,
utf8*, utf16* and utf32*, the compiler _could_ identify all such locations if
desired.


[Requirements]
In order to implement all this "string" requires knowledge of all code points,
how they are encoded in the 3 encodings and how to compare and convert between
them. So, D and thus any D compiler eg DMD, requires this knowledge. I am not
entirely sure just how big an "ask" this is. I believe DMD and thus D already
has much of this capability built in.


[Questions]
(a) Is UTF-8 a subset of UTF-16 and so on? does the codepoint for 'A' have the
numerical value 65 decimal in UTF-8, UTF-16 _and_ UTF-32, in other words is it
the same numerical value in all encodings? If so then comparing utf8, 16 and 32
is no different to comparing byte, short and int and all the same promotion and
comparrison rules can apply.

(b) Is this really ASCII or is it system dependant? i.e. Latin-1 or similar. Is
it ASCII values 127 or less perhaps? To be honest I'm not sure.

------------SRJ5HVj2hz31rAqYWOgFlA--
Nov 23 2005
next sibling parent reply Derek Parnell <derek psych.ward> writes:
On Thu, 24 Nov 2005 16:09:13 +1300, Regan Heath wrote:

 With the recent Physics slant on some posts here I couldn't resist that  
 subject

LOL
 Enough rambling, here is it, be nice!

Just some quick thoughts are recorded here. More will come later I suspect. [snip]
 [Key]
 First, lets start with some terminology, these are the terms I am going to  
 be using and what they mean, if these are incorrect please correct me, but  
 take them to have the stated meanings for this document.
 
 code point      := the unicode value for a single and complete character.
 code unit       := part of, or a complete character in one of the 3 UTF  
 encodings UTF-8,16,32.
 code value      := AKA code unit.

The Unicode Consortium defines code value as the smallest (in terms of bits) value that will hold a character in the various encoding formats. Thus for UTF8 it is 1 byte, UTF16 = 2 bytes, and UTF32 = 4 bytes. [snip]
 * remove char[], wchar[] and dchar[].

Do we still have to cater for strings that were formatted in specific encodings outside of our D applications? For example, a C library routine might insist that a pointer to a UTF16 string be supplied, thus we would have to force a specific encoding somehow.
 * add a new type "string". "string" will store code points in the  
 application specific native encoding and be implicitly or explicitly  
 transcoded as required (more below).
 
 * the application specific native encoding will default to UTF-8. An  
 application can choose another with a compile option or pragma, this  
 choice will have no effect on the behaviour of the program (as we only  
 have 1 type and all transcoding is handled where required) it will only  
 affect performance.
 
 The performance cost cannot be avoided, presuming it is only being done at  
 input and output (which is part of what this proposal aims to achieve).  
 This cost is application specific and will depend on the tasks and data  
 the application is designed to perform and use.
 
 Given that, letting the programmer choose a native encoding will allow  
 them to test different encodings for speed and/or provide different builds  
 based on the target language, eg an application destined to be used with  
 the Japanese language would likely benefit from using UTF-32  
 internally/natively.
 
 * keep char, wchar, and dchar but rename them utf8, utf16, utf32. These  
 types represent code points (always, not a code units/values) in each  
 encoding. Only code points that fit in utf8 will ever be represented by  
 utf8, and so on. Thus some code points will always be utf32 values and  
 never utf8 or 16. (much like byte/short/int)

I think you've lost track of your 'code point' definition. A 'code point' is a character. All encodings can hold all characters, every character will fit into UTF8. Sure some might take 1, 2 or 4 'code values', but there are still all code points. There are no exclusive code points in utf32. Every UTF32 code point can also be expressed in UTF8.
 * add promotion/comparrison rules for utf8, 16 and 32:
 
 - any given code point represented as utf8 will compare equal to the same  
 code point represented as a utf16 or utf32 and vice versa(a)
 
 - any given code point represented as utf8 will be implicitly  
 converted/promoted to the same code point represented as utf16 or utf32 as  
 required and vice versa(a). If promotion from utf32 to utf16 or 8 causes  
 loss in data it should be handled just like int to short or byte.

I assume by 'promotion' you really mean 'transcoding'. There is never any data loss when converting between the different encodings. This is your #1 assumption.
 * add a new type/alias "utf", this would alias utf8, 16 or 32. It  
 represents the application specific native encoding. This allows efficient  
 code, like:
 
 string s = "test";
 foreach(utf c; s) {
 }

But utf8, utf16, and utf32 are *strings* not characters, so 'utf' could not be an *alias* for these in your example. I guess you mean it to be a term for a character (code point) in a utf string.
 regardless of the applications selected native encoding.
 
 * slicing string gives another string
 
 * indexing a string gives a utf8, 16, or 32 code point.
 
 * string literals would be of type "string" encoded in the native  
 encoding, or if another encoding can be determined at compile time, in  
 that encoding (see ASCII example below).
 
 * character literals would default to the native encoding, failing that  
 the smallest possible type, and promoted/converted as required.

By 'smallest possible type' do you mean the smallest memory usage?
 * there are occasions where you may want to use a specific encoding for a  
 part of your application, perhaps you're loading a UTF-16 file and parsing  
 it. If all the work is done in a small section of code and it doesn't  
 interact with the bulk of your application data which is all in UTF-8 your  
 native encoding it likely to be UTF-8.
 
 In this case, for performance reasons, you want to be able to specify the  
 encoding to use for your "string" types at runtime, they are exceptions to  
 the native encoding. To do this we specify the encoding at  
 construction/declaration time, eg.
 
 string s(UTF16);
 s.utf16 = ..data read from UTF-16 source..
 
 (or similar, the exact syntax is not important at this stage)

But the idea is that a string has the property of 'utf8', and 'utf16' and 'utf32' encoding at runtime? -- Derek (skype: derek.j.parnell) Melbourne, Australia 24/11/2005 2:34:13 PM
Nov 23 2005
parent "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 24 Nov 2005 15:04:08 +1100, Derek Parnell <derek psych.ward> wrote:
 [Key]
 First, lets start with some terminology, these are the terms I am going  
 to
 be using and what they mean, if these are incorrect please correct me,  
 but
 take them to have the stated meanings for this document.

 code point      := the unicode value for a single and complete  
 character.
 code unit       := part of, or a complete character in one of the 3 UTF
 encodings UTF-8,16,32.
 code value      := AKA code unit.

The Unicode Consortium defines code value as the smallest (in terms of bits) value that will hold a character in the various encoding formats. Thus for UTF8 it is 1 byte, UTF16 = 2 bytes, and UTF32 = 4 bytes.

Thanks for the detailed description. That is what I meant above.
 * remove char[], wchar[] and dchar[].

Do we still have to cater for strings that were formatted in specific encodings outside of our D applications? For example, a C library routine might insist that a pointer to a UTF16 string be supplied, thus we would have to force a specific encoding somehow.

Yes, that is the purpose of char*, utf16*, etc. eg. int strlen(const char *string) {} int CreateFileW(utf16 *filename, ...
 * keep char, wchar, and dchar but rename them utf8, utf16, utf32. These
 types represent code points (always, not a code units/values) in each
 encoding. Only code points that fit in utf8 will ever be represented by
 utf8, and so on. Thus some code points will always be utf32 values and
 never utf8 or 16. (much like byte/short/int)

I think you've lost track of your 'code point' definition.

Not so. I've just failed to explain what I mean here, let me try some more...
 A 'code point' is a character.

Correct.
 All encodings can hold all characters, every character will
 fit into UTF8. Sure some might take 1, 2 or 4 'code values', but there  
 are still all code points. There are no exclusive code points in utf32.  
 Every
 UTF32 code point can also be expressed in UTF8.

I realise all this. It is not what I mean't above. Think of the type "utf8" as being identical to "byte", except that the values it stores are always complete code points, never fragments or code units/values. The type "utf8" will never have part of a complete character in it, it'll either have the whole character or it will be an error. "utf8" can represent the range of code points which are between 0 and 255 (or perhaps it's 127, not sure). perhaps the name "utf8" is missleading, it's not in fact a UTF-8 code unit/value, it is a codepoint, that fits in a byte. The reason it's not called "byte" is because the seperate type is used to trigger transcoding, see my original utf16* example.
 * add promotion/comparrison rules for utf8, 16 and 32:

 - any given code point represented as utf8 will compare equal to the  
 same
 code point represented as a utf16 or utf32 and vice versa(a)

 - any given code point represented as utf8 will be implicitly
 converted/promoted to the same code point represented as utf16 or utf32  
 as
 required and vice versa(a). If promotion from utf32 to utf16 or 8 causes
 loss in data it should be handled just like int to short or byte.

I assume by 'promotion' you really mean 'transcoding'.

No, I think I mean promotion. This is one of the things I am not 100% sure of, bear with me. The character 'A' has ASCII value 65 (decimal). Assuming it's code point is 65 (decimal), then this code point will fit in my "utf8" type. Thus "utf8" can represent the code point 'A'. If you assign that "utf8" to a "utf16", eg. utf8 a = 'A'; utf16 b = a; The utf8 value will be promoted to a utf16 value. The value itself doesn't change (it's not transcoded). It happens in exactly the same way a byte is promoted to a short. Is promoted the right word? That is, provided the value doesn't _need_ to change when going from utf8 to utf16, I am not 100% sure of this. I don't think it does. I believe all the code points that fit in the 1 byte type, have the same numerical value in the 2 byte type (UTF-16), and also the 4 byte type (UTF-32).
 * add a new type/alias "utf", this would alias utf8, 16 or 32. It
 represents the application specific native encoding. This allows  
 efficient
 code, like:

 string s = "test";
 foreach(utf c; s) {
 }

But utf8, utf16, and utf32 are *strings* not characters

No, they're not, not in my proposal. I think I picked bad names.
 , so 'utf' could not be an *alias* for these in your example. I guess  
 you mean it to be a term
 for a character (code point) in a utf string.

utf, utf8, utf16, and utf32 are all types that store complete code points, never code units/values/fragments. Think of them as being identical to byte, short, and int.
 regardless of the applications selected native encoding.

 * slicing string gives another string

 * indexing a string gives a utf8, 16, or 32 code point.

 * string literals would be of type "string" encoded in the native
 encoding, or if another encoding can be determined at compile time, in
 that encoding (see ASCII example below).

 * character literals would default to the native encoding, failing that
 the smallest possible type, and promoted/converted as required.

By 'smallest possible type' do you mean the smallest memory usage?

Yes. utf8 is smaller than utf16 is smaller than utf32.
 * there are occasions where you may want to use a specific encoding for  
 a
 part of your application, perhaps you're loading a UTF-16 file and  
 parsing
 it. If all the work is done in a small section of code and it doesn't
 interact with the bulk of your application data which is all in UTF-8  
 your
 native encoding it likely to be UTF-8.

 In this case, for performance reasons, you want to be able to specify  
 the
 encoding to use for your "string" types at runtime, they are exceptions  
 to
 the native encoding. To do this we specify the encoding at
 construction/declaration time, eg.

 string s(UTF16);
 s.utf16 = ..data read from UTF-16 source..

 (or similar, the exact syntax is not important at this stage)

But the idea is that a string has the property of 'utf8', and 'utf16' and 'utf32' encoding at runtime?

Yes. But you will only need to use these properties when performing input or output (see my definitions of source and sink) and only when the type cannot be inferred by the context, i.e. it's not required here: int CreateFile(utf16* filename) {} string test = "test"; CreateFile(test); Regan
Nov 23 2005
prev sibling next sibling parent reply "Lionello Lunesu" <lio remove.lunesu.com> writes:
Hi Regan,

Two small remarks:

* "wchar" might still be useful for those applications / libraries that 
support 16-bit unicode without aggregates like in Windows NT if I'm correct. 
It's not utf16 since it can't contain a big, >2-byte code point, ie. it's 
ushort.

* I don't see the point of the utf8, utf16 and utf32 types. They can all 
contain any code point, so they should all be just as big? Or do you mean 
that utf8 is like a ubyte[4], utf16 like ushort[2] and utf32 like uint? 
Actually pieces from the respective strings.

L. 
Nov 23 2005
next sibling parent "Lionello Lunesu" <lio remove.lunesu.com> writes:
By the way, I like the proposal! I prefer different compiled libraries to 
many runtime checks or version blocks. It's like the #define UNICODE in 
Windows.

L. 
Nov 23 2005
prev sibling parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 24 Nov 2005 09:56:51 +0200, Lionello Lunesu  
<lio remove.lunesu.com> wrote:
 Two small remarks:

 * "wchar" might still be useful for those applications / libraries that
 support 16-bit unicode without aggregates like in Windows NT if I'm  
 correct.
 It's not utf16 since it can't contain a big, >2-byte code point, ie. it's
 ushort.

 * I don't see the point of the utf8, utf16 and utf32 types. They can all
 contain any code point, so they should all be just as big? Or do you mean
 that utf8 is like a ubyte[4], utf16 like ushort[2] and utf32 like uint?
 Actually pieces from the respective strings.

No. I seem to have done a bad job of explaining it _and_ picked terrible names. The "utf8", "utf16" and "utf32" types I refer to are essentially byte, short and int. They cannot contain any code point, only those that fit (I thought I said that?) We don't need wchar because utf16 replaces it. Perhaps if I had kept the original names... doh! Regan
Nov 24 2005
parent reply "Lionello Lunesu" <lio remove.lunesu.com> writes:
 The "utf8", "utf16" and "utf32" types I refer to are essentially byte, 
 short and int. They cannot contain any code point, only those that fit (I 
 thought I said that?)

In that case I don't like your idea : ) It makes far more sense to have only 1 _character_ type, that holds any UNICODE character. Whether it comes from an utf8, utf16 or utf32 string shouldn't matter: string s="Whatever"; //imagine it with a small circle on the a, comma under the t foreach(uchar u; s) {} Read "uchar" as "unicode char", essentially dchar, could in fact still be named dchar, I just didn't want to mix old/new terminology. The underlying type of "string" would be determined at compile time, but still convertable using properties (that part I liked very much). D's "char" should go back to C's char, signed even. Many decissions in D where made to ease the porting of C code, so why this "char" got overriden beats me. char[] should then behave no differently from byte[] (except maybe the element being signed). L.
Nov 24 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 24 Nov 2005 17:35:24 +0200, Lionello Lunesu  
<lio remove.lunesu.com> wrote:
 The "utf8", "utf16" and "utf32" types I refer to are essentially byte,
 short and int. They cannot contain any code point, only those that fit  
 (I
 thought I said that?)

In that case I don't like your idea : ) It makes far more sense to have only 1 _character_ type, that holds any UNICODE character. Whether it comes from an utf8, utf16 or utf32 string shouldn't matter:

Yeah, I'm starting to think that is the only way it works. The 3 types were an attempt to avoid that for programs which do not need an int-sized type for a char i.e. the quick and dirty ASCII program for example. Interestingly it seems std.stdio and std.format are already involved in a conspiracy to convert all our char[] output to dchar and back again one character at a time before it eventually makes it to the screen.
 string s="Whatever";    //imagine it with a small circle on the a, comma
 under the t
 foreach(uchar u; s) {}

 Read "uchar" as "unicode char", essentially dchar, could in fact still be
 named dchar, I just didn't want to mix old/new terminology. The  
 underlying
 type of "string" would be determined at compile time, but still  
 convertable
 using properties (that part I liked very much).

 D's "char" should go back to C's char, signed even. Many decissions in D
 where made to ease the porting of C code, so why this "char" got  
 overriden
 beats me. char[] should then behave no differently from byte[] (except  
 maybe
 the element being signed).

I like "uchar". I agree "char" should go back to being C's char type. I don't think we need a char[] all the C functions expect a null terminated char*. Regan
Nov 24 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
Regan Heath wrote:
 On Thu, 24 Nov 2005 17:35:24 +0200, Lionello Lunesu 
 
 It makes far more sense to have only 1 _character_ type, that holds
 any UNICODE character. Whether it comes from an utf8, utf16 or
 utf32 string shouldn't matter:


True!
 Yeah, I'm starting to think that is the only way it works. The 3
 types were an attempt to avoid that for programs which do not need an
 int-sized  type for a char i.e. the quick and dirty ASCII program
 for example.
 
 Interestingly it seems std.stdio and std.format are already involved
 in a  conspiracy to convert all our char[] output to dchar and back
 again one  character at a time before it eventually makes it to the
 screen.

Must've been the specters in the night again. :-)
 I like "uchar". I agree "char" should go back to being C's char type.
 I don't think we need a char[] all the C functions expect a null 
 terminated  char*.

That would be nice! What if we even decided that char[] is null-terminated? That'd massively reduce all kinds of bugs when (under pressure) converting code from C(++)!
Nov 24 2005
parent reply Derek Parnell <derek psych.ward> writes:
On Fri, 25 Nov 2005 02:21:52 +0200, Georg Wrede wrote:

 Regan Heath wrote:
 On Thu, 24 Nov 2005 17:35:24 +0200, Lionello Lunesu 
 
 It makes far more sense to have only 1 _character_ type, that holds
 any UNICODE character. Whether it comes from an utf8, utf16 or
 utf32 string shouldn't matter:


True!
 Yeah, I'm starting to think that is the only way it works. The 3
 types were an attempt to avoid that for programs which do not need an
 int-sized  type for a char i.e. the quick and dirty ASCII program
 for example.
 
 Interestingly it seems std.stdio and std.format are already involved
 in a  conspiracy to convert all our char[] output to dchar and back
 again one  character at a time before it eventually makes it to the
 screen.

Must've been the specters in the night again. :-)
 I like "uchar". I agree "char" should go back to being C's char type.
 I don't think we need a char[] all the C functions expect a null 
 terminated  char*.

That would be nice! What if we even decided that char[] is null-terminated? That'd massively reduce all kinds of bugs when (under pressure) converting code from C(++)!

I think that would interfere with the slice concept. char[] a = "some text"; char[] b = a[4 .. 7]; // Making 'b' a reference into 'a'. -- Derek (skype: derek.j.parnell) Melbourne, Australia 25/11/2005 11:37:08 AM
Nov 24 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
Derek Parnell wrote:
 On Fri, 25 Nov 2005 02:21:52 +0200, Georg Wrede wrote:

 I like "uchar". I agree "char" should go back to being C's char
 type. I don't think we need a char[] all the C functions expect a
 null terminated char*.

That would be nice! What if we even decided that char[] is null-terminated? That'd massively reduce all kinds of bugs when (under pressure) converting code from C(++)!

I think that would interfere with the slice concept. char[] a = "some text"; char[] b = a[4 .. 7]; // Making 'b' a reference into 'a'.

Slicing C's char[] implies byte-wide, and non-UTF.
Nov 24 2005
parent reply Derek Parnell <derek psych.ward> writes:
On Fri, 25 Nov 2005 05:16:15 +0200, Georg Wrede wrote:

 Derek Parnell wrote:
 On Fri, 25 Nov 2005 02:21:52 +0200, Georg Wrede wrote:

 I like "uchar". I agree "char" should go back to being C's char
 type. I don't think we need a char[] all the C functions expect a
 null terminated char*.

That would be nice! What if we even decided that char[] is null-terminated? That'd massively reduce all kinds of bugs when (under pressure) converting code from C(++)!

I think that would interfere with the slice concept. char[] a = "some text"; char[] b = a[4 .. 7]; // Making 'b' a reference into 'a'.

Slicing C's char[] implies byte-wide, and non-UTF.

Exactly, and that why I'm worried by the suggestion that char[] be automatically zero-terminated, because slices are usually not zero-terminated. -- Derek (skype: derek.j.parnell) Melbourne, Australia 25/11/2005 3:12:51 PM
Nov 24 2005
parent Georg Wrede <georg.wrede nospam.org> writes:
Derek Parnell wrote:
 On Fri, 25 Nov 2005 05:16:15 +0200, Georg Wrede wrote:
Derek Parnell wrote:
On Fri, 25 Nov 2005 02:21:52 +0200, Georg Wrede wrote:

I like "uchar". I agree "char" should go back to being C's char
type. I don't think we need a char[] all the C functions expect a
null terminated char*.

That would be nice! What if we even decided that char[] is null-terminated? That'd massively reduce all kinds of bugs when (under pressure) converting code from C(++)!

I think that would interfere with the slice concept. char[] a = "some text"; char[] b = a[4 .. 7]; // Making 'b' a reference into 'a'.

Slicing C's char[] implies byte-wide, and non-UTF.

Exactly, and that why I'm worried by the suggestion that char[] be automatically zero-terminated, because slices are usually not zero-terminated.

With what we're doing with the utf, it would be a small additional job to have the "C char" arrays take care of the null byte at the end. So the programmer would not have to think about it. (I admit this takes some further thinking first! So you are right in your concerns!)
Nov 25 2005
prev sibling next sibling parent reply Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Regan Heath wrote:
[snip]
 [Questions]
 (a) Is UTF-8 a subset of UTF-16 and so on? does the codepoint for 'A' have the
numerical value 65 decimal in UTF-8, UTF-16 _and_ UTF-32, in other words is it
the same numerical value in all encodings? If so then comparing utf8, 16 and 32
is no different to comparing byte, short and int and all the same promotion and
comparrison rules can apply.

I think you are making this more complicated than it is by using the name UTF when you actually mean something like: ascii_char (not utf8) (code point < 128) ucs2_char (not utf16) (code point < 65536) unicode_char (not utf32) And yes: ascii is a subset of ucs2 is a subset of unicode.
 (b) Is this really ASCII or is it system dependant? i.e. Latin-1 or similar.
Is it ASCII values 127 or less perhaps? To be honest I'm not sure.

ASCII is equal to the first 128 code points in Unicode. Latin-1 is equal to the first 256 code points in Unicode. Regards, /Oskar
Nov 24 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 24 Nov 2005 09:23:21 +0100, Oskar Linde  
<oskar.lindeREM OVEgmail.com> wrote:
 Regan Heath wrote:
 [snip]
 [Questions]
 (a) Is UTF-8 a subset of UTF-16 and so on? does the codepoint for 'A'  
 have the numerical value 65 decimal in UTF-8, UTF-16 _and_ UTF-32, in  
 other words is it the same numerical value in all encodings? If so then  
 comparing utf8, 16 and 32 is no different to comparing byte, short and  
 int and all the same promotion and comparrison rules can apply.

I think you are making this more complicated than it is by using the name UTF when you actually mean something like: ascii_char (not utf8) (code point < 128) ucs2_char (not utf16) (code point < 65536) unicode_char (not utf32)

I agree, it appears my choice of type names was really confusing. I have posted a change, but perhaps I should repost all over again, perhaps I should have bounced this off one person before posting.
 And yes: ascii is a subset of ucs2 is a subset of unicode.

Excellent. Thanks.
 (b) Is this really ASCII or is it system dependant? i.e. Latin-1 or  
 similar. Is it ASCII values 127 or less perhaps? To be honest I'm not  
 sure.

ASCII is equal to the first 128 code points in Unicode. Latin-1 is equal to the first 256 code points in Unicode.

And which does a C function expect? Or is that defined by the C function? Does strcmp care? Does strlen, strchr, ...? Regan
Nov 24 2005
parent Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Regan Heath wrote:
 On Thu, 24 Nov 2005 09:23:21 +0100, Oskar Linde  
 <oskar.lindeREM OVEgmail.com> wrote:
 
 Regan Heath wrote:

 (b) Is this really ASCII or is it system dependant? i.e. Latin-1 or  
 similar. Is it ASCII values 127 or less perhaps? To be honest I'm 
 not  sure.

ASCII is equal to the first 128 code points in Unicode. Latin-1 is equal to the first 256 code points in Unicode.

And which does a C function expect? Or is that defined by the C function? Does strcmp care? Does strlen, strchr, ...?

This is not defined. strcmp doesn't care. strlen etc only counts bytes until '\0'. You can use latin-1, utf-8 or any 8-bit encoding. This is why UTF-8 is so popular. You can just plug it in and almost everything that used to assume latin-1 or any 8-bit encoding will just work without any changes. Not even the OS cares very much. To the OS, things like a file name, file contents, usernames, etc are just a bunch of bytes. Different file systems may then define different encodings the file names should be interpreted in. This is just how the file name is presented to the user. (Transcoding to/from the terminal) /Oskar
Nov 24 2005
prev sibling next sibling parent reply "Regan Heath" <regan netwin.co.nz> writes:
Ok, it appears I picked some really bad type names in my proposal and it  
is causing some confusion.

The types "utf8" "utf16" and "utf32" do not in fact have anything to do  
with UTF. (Bad Regan).

They are in fact essentially byte, short and int with different names.  
Having different names is important because it triggers the transcoding of  
"string" to the required C, OS, or UTF type.

I could have left them called "char", "wchar" and "dchar", except that I  
wanted a 4th type to represent C's char as well. That type was called  
"char" in the proposal.

So, for the sake of our sanity can we all please assume I have used these  
type names instead:

"utf8"  == "cp1"
"utf16" == "cp2"
"utf32" == "cp4"
"utf"   == "cpn"

(the actual type names are unimportant at this stage, we can pick the best  
possible names later)

The idea behind these types is that they represent code points/characters  
_never_ code units/values/fragments. Which means cp1 can only represent a  
small subset of unicode code points, cp2 slightly more and cp4 all of them  
(IIRC).

It means assigning anything outside their range to them is an error.

It means that you can assign a cp1 to a cp2 and it simply promotes it  
(like it would from byte to short).

"cpn" is simply and alias for the type that is best suited for the chosen  
native encoding. If the native encoding is UTF-8, cpn is an alias for cp1,  
if the native encoding is UTF-16, cpn is an alias for cp2, and so on.

Sorry for all the confusion.

Regan
Nov 24 2005
next sibling parent reply "Regan Heath" <regan netwin.co.nz> writes:
------------HAivqONauagBiEC8USmppj
Content-Type: text/plain; format=flowed; delsp=yes; charset=iso-8859-15
Content-Transfer-Encoding: 8bit

Replying to myself now, in addition to bolloxing the initial proposal up  
with bad type names, I'm on a roll!

Here is version 1.1 of the proposal, with different type names and some  
changes to the other content. Hopefully this one will make more sense,  
fingers crossed.

Regan
------------HAivqONauagBiEC8USmppj
Content-Disposition: attachment; filename="D and UTF.txt"
Content-Type: text/plain; name="D and UTF.txt"
Content-Transfer-Encoding: 8bit

Proposal: A single unified string type.
Author  : Regan Heath
Version : 1.1
Date    : 24 Nov 2005 +1300 (New Zealand DST)

[Preamble/Introduction]
After the recent discussion on Unicode, UTF encodings and the current D
situation it occured to me that many of the issues D has with strings could be
side-stepped if there was a single string type. 

In the past we have assumed that to obtain this we have to choose one of the 3
available types and encodings. This wasn't an attractive option because each
type has different pros/cons and each application may prefer one type over
another. Another suggested solution was a string class which hides the details,
this solution suffers from being a class and the limitations imposed by that
and not being tied directly into the language.

My proposal is a single "string" type built into the language, which can
represent it's string data in any given UTF encoding. Which will allow slicing
of "characters" as opposed to what is essentially bytes, shorts, and ints.
Whose default encoding can be selected at compile time, or specified at
runtime. Which will implicitly or explicitly transcode where required.

There are some requirements for this to be possible, namely knowledge of the
UTF encodings being built into D, these requirements may prohibit the proposal
being favourable as it increases the knowledge required to write a D compiler.
However it occurs to me that DMD and thus D? already requires a fair bit of UTF
knowledge.


[Key]
First, lets start with some terminology, these are the terms I am going to be
using and what they mean, if these are incorrect please correct me, but take
them to have the stated meanings for this document.

code point      := Any value in the Unicode codespace; that is, the range of
integers from 0 to 10FFFF.
code unit       := The minimal bit combination that can represent a unit of
encoded text for processing or interchange. i.e. 8-bit code units in the UTF-8
encoding form, 16-bit code units in the UTF-16 encoding form, and 32-bit code
units in the UTF-32 encoding form.
code value      := AKA code unit.
transcoding     := the process of converting from one encoding to another.
source          := a file, the keyboard, a tcp socket, a com port, an OS/C
function call, a 3rd party library.
sink            := a file, the screen, a tcp socket, a com port, an OS/C
function call, a 3rd party library.
native encoding := application specific "preferred" encoding (more on this
later)

Anything I am unsure about will be suffixed with (x) where x is a letter of the
alphabet, and my thoughts will be detailed in the [Questions] section.


[Assumptions]
These are what I base my argument/suggestion on, if you disagree with any of
these you will likely disagree with the proposal. If that is the case please
post your concerns with any given assumption in it's own post (I would like to
discuss each issue in it's own thread and avoid mixing several issues)

#1: Any given string can be represented in any UTF encoding, it can be
transcoded to/from any UTF encoding with no loss of data/meaning.

#2: Transcoding has a performance penalty at runtime. This proposal will
mention the possible runtime penalty wherever appropriate.

#3: There are 2 places where transcoding cannot be avoided; input and ouput.
Input is the process of obtaining data from a source. Output is the process of
sending data to a sink. In either case the source or sink will have a fixed
encoding and is that encoding does not match the native encoding the
application will need to transcode. (see definitions above for what classifies
as a source or sink)

#4: String literals can be stored in the binary in any encoding (#1) the
encoding chosen may have repercusions at runtime (#2 & #3).


[Details]
Many of the details are flexible, i.e. the names of the types etc, the
important/inflexible details are how it all fits together and achieves it's
results. I've chosen a bullet point format and tried to make each change/point
as succint and clear as possible. Feel free to ask for clarification on any
point or points. Or to ask general questions. Or to pose general problems. I
will do my best to answer all questions.

* remove char[], wchar[] and dchar[].

* add a new type "string". "string" will store code points in the application
specific native encoding and be implicitly or explicitly transcoded as required
(more below).

* the application specific native encoding will default to UTF-8. An
application can choose another with a compile option or pragma, this choice
will have no effect on the behaviour of the program (as we only have 1 type and
all transcoding is handled where required) it will only affect performance. 

The performance cost cannot be avoided, presuming it is only being done at
input and output (which is part of what this proposal aims to achieve). This
cost is application specific and will depend on the tasks and data the
application is designed to perform and use. 

Given that, letting the programmer choose a native encoding will allow them to
test different encodings for speed and/or provide different builds based on the
target language, eg an application destined to be used with the Japanese
language would likely benefit from using UTF-32 internally/natively.

* rename char, wchar, and dchar to cp1, cp2, and cp4. These types represent
code points only, never code units/values. cp1 will only ever represent code
points which fit inside 1 byte. cp2 will only ever represent code points that
fit inside 2 bytes, and cp4 will only ever represent code points that fit
inside 4 bytes. They are essentially byte, short and int with different names.

* use the existing byte, short, int handling rules for cp1, cp2, and cp4, they
are essentially the same thing.

* add a new type/alias "cpn", this alias will be cp1, cp2 or cp4 depending on
the native encoding chosen. This allows efficient code, like:

string s = "test";
foreach(cpn c; s) {
}

* slicing string gives another string

* indexing a string gives a cp1, cp2 or cp4

* string literals would be of type "string" encoded in the native encoding, or
if another encoding can be determined at compile time, in that encoding (see
ASCII example below).

* character literals would default to the native encoding, failing that to the
smallest of cp1, 2 or 4 and are promoted as required.

* there are occasions where you may want to use a specific encoding for a part
of your application, perhaps you're loading a UTF-16 file and parsing it. If
all the work is done in a small section of code and it doesn't interact with
the bulk of your application data which is all in UTF-8 then your native
encoding is likely to be UTF-8 not UTF-16.

In this case, for performance reasons, you want to be able to specify the
encoding to use for your "string" types at runtime, they are exceptions to the
native encoding. To do this we specify the encoding at construction/declaration
time, eg.

string s(UTF16); //construct string which uses UTF-16 to store it's data
//somehow assign the data read from the file to the string, no transcoding
occurs as they are of the same type.
(or similar, the exact syntax is not important at this stage)

thus...

* the type of encoding used by "string" should be selectable at runtime, some
sort of encoding type flag must exist for each string at runtime, this is
starting to head into "implementation details" which I want to avoid at this
point, however it is important to note the requirement.


[Output]
* a new type "char" will exist, it will now _only_ represent a C string, thus
when a string is passed as a char it can be implicitly transcoded into ASCII
with a null terminator, eg.

int strcmp(const char *src, const char *dst);

string test = "this is a test";
if (strcmp(test,"this is a test")==0) { }

the above will implicitly transcode 'test' into ASCII and ensure there is a
null terminator. The literal "this is a test" can be stored in the binary as
ASCII with a null terminator.

* Native OS functions requiring "char" will use the rule above. eg.

CreateFileA(char *filename...

* Native OS functions requiring unicode will be defined as:

CreateFileW(cp2 *filename...

and "string" will be implicitly transcoded to UTF-16, with a null terminator
added..

* When the required encoding is not apparent, eg.

void CreateFile(char *data) { }
void CreateFile(cp2 *data) { }

string test = "this is a test";
CreateFile(test);

an explicit property should be used, eg.

CreateFile(test.char);
CreateFile(test.cp2);


[Input]
* Old encodings, Latin-1 etc would be loaded into ubyte[] or byte[] and could
be cast (painted) to char*, cp1*, cp2* or cp4* or converted to "string" using a
routine i.e. string toStringFromXXX(ubyte[] raw).

* A stream class would have a selectable encoding and hide these details from
us handling the data and giving a natively encoded "string" instead. Meaning,
transcoding will naturally occur on input or output where required.


[Example application types and the effect of this change]

* the quick and dirty console app which handles ASCII only. It's native
encoding will be UTF-8, and no transcoding will ever need to occur (assuming
none of it's input or output is in another encoding)

* an app which loads files in different encodings and needs to process them
efficiently. In this case the code can select the encoding of "string" at
runtime and avoid transcoding the data until such time as it needs to interface
with another part of the application in another encoding or it needs to output
to a sink, also in another encoding.

* an international app which will handle many languages. this app can be custom
built with the native string type selected to match each language.


[Advantages]
As I see it, this change would have the following advantages:

* "string" requires no knowledge of UTF encodings (and the associated problems)
to use making it easy for begginners and for a quick and dirty program.

* "string" can be sliced/indexed by character regardless of the encoding used
for the data.

* overload resolution has only 1 type, not 3 to choose from.

* code written in D would all use the same type "string". no more this library
uses char[] this one wchar and my app dchar[] problems.


[Disadvantages]
* requirements listed below

* libraries built for a different native type will likely cause transcoding.
This problem already exists, at least with this suggestion the library can be
built 3 times, once for each native encoding and the correct one linked to your
app.

* possibility of implicit and silent transcoding. This can occur between
libraries built with different native encodings and between "string" and char*,
cp1*, cp2* and cp4*, the compiler _could_ identify all such locations if
desired.


[Requirements]
In order to implement all this "string" requires knowledge of all code points,
how they are encoded in the 3 encodings and how to compare and convert between
them. So, D and thus any D compiler eg DMD, requires this knowledge. I am not
entirely sure just how big an "ask" this is. I believe DMD and thus D already
has much of this capability built in.

------------HAivqONauagBiEC8USmppj--
Nov 24 2005
next sibling parent reply Georg Wrede <georg.wrede nospam.org> writes:
Congrats, Regan! Great job!

And the thread subject is simply a Killer!



If I understand you correctly, then the following would work:

string st = "aaa\U41bbb\UC4ccc\U0107ddd";   // aaaAbbbÄcccćddd
cp1 s3 = st[3];   // A
cp1 s7 = st[7];   // Ä
cp1 s11 = st[11]; // error, too narrow
cp2 s11 = st[11]; // ć

assert( s3 == 0x41 && s7 == 0xC4 && s11 == 0x107 );

So, s3 would contain "A", which the old system would store as utf8 with 
no problem. s3 is 8 bits.

s7 would contain "Ä", which the old system shouldn't have stored in 
8-bit (char) because it is too big, but with your proposal it would be 
ok, since the _code_point_ (i.e. the "value" of the character in 
Unicode) does fit in 8 bits. And _we_are_storing_ the codepoint, not the 
UTF character here, right?

s11 would error, since even the Unicode value is too big for 8 bits.

The second s11 assignment would be ok, since the Unicode value of ć fits 
in 16 bits.

And, st itself would be "regular" UTF-8 on a Linux, and (probably) 
UTF-16 on Windows.

Yes?
Nov 24 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 24 Nov 2005 13:29:10 +0200, Georg Wrede <georg.wrede nospam.org>  
wrote:
 Congrats, Regan! Great job!
 And the thread subject is simply a Killer!

 If I understand you correctly, then the following would work:

 string st = "aaa\U41bbb\UC4ccc\U0107ddd";   // aaaAbbbÄcccćddd
 cp1 s3 = st[3];   // A
 cp1 s7 = st[7];   // Ä
 cp1 s11 = st[11]; // error, too narrow
 cp2 s11 = st[11]; // ć

 assert( s3 == 0x41 && s7 == 0xC4 && s11 == 0x107 );

 So, s3 would contain "A", which the old system would store as utf8 with  
 no problem. s3 is 8 bits.

 s7 would contain "Ä", which the old system shouldn't have stored in  
 8-bit (char) because it is too big, but with your proposal it would be  
 ok, since the _code_point_ (i.e. the "value" of the character in  
 Unicode) does fit in 8 bits. And _we_are_storing_ the codepoint, not the  
 UTF character here, right?

Yes. That's exactly what I was thinking. However it appears that the idea does hold together to well when it comes to "cpn" the alias, eg: string s = "smörgåsbord"; foreach(cpn c; s) { } "cpn" would need to change size for each character. It would be more than a simple alias. If it cannot change size, then it would need to be the largest size required. If that was also too weird/difficult then it would need to be 32 bits in size all the time. I was trying to avoid this but it seems it may be required?
 s11 would error, since even the Unicode value is too big for 8 bits.

 The second s11 assignment would be ok, since the Unicode value of ć fits  
 in 16 bits.

 And, st itself would be "regular" UTF-8 on a Linux, and (probably)  
 UTF-16 on Windows.

 Yes?

My proposal didn't suggest different encodings based on the system. It was UTF-8 by default (all systems) and application specific otherwise. There is nothing stopping us making the windows default to UTF-16 if that makes sense. Which it seems to. Regan
Nov 24 2005
parent Georg Wrede <georg.wrede nospam.org> writes:
Regan Heath wrote:
 On Thu, 24 Nov 2005 13:29:10 +0200, Georg Wrede 
 <georg.wrede nospam.org>  wrote:

 "cpn" would need to change size for each character. It would be more 
 than  a simple alias.
 
 If it cannot change size, then it would need to be the largest size  
 required.
 
 If that was also too weird/difficult then it would need to be 32 bits 
 in  size all the time. I was trying to avoid this but it seems it may 
 be  required?

Yes. I see no way to avoid "cpn" being 32 bit only.
 My proposal didn't suggest different encodings based on the system. It 
 was  UTF-8 by default (all systems) and application specific otherwise. 
 There  is nothing stopping us making the windows default to UTF-16 if 
 that makes  sense. Which it seems to.

Windows, <sigh>. Looks like it. They seem to have a habit of choosing what seems easiest at the outset, without ever learning to dig into issues first. Had they done it, they'd chosen UTF-8, like everybody else. :-(
Nov 24 2005
prev sibling parent reply Georg Wrede <georg.wrede nospam.org> writes:
Regan Heath wrote:
 
 * add a new type/alias "cpn", this alias will be cp1, cp2 or cp4
 depending on the native encoding chosen. This allows efficient
 code, like:
 
 string s = "test";
 foreach(cpn c; s) {
 }
 
 * slicing string gives another string
 
 * indexing a string gives a cp1, cp2 or cp4

I hope you are not implying that indexing would choose between cp1..4 based on content? And if not, then the cpX would be either some "default", or programmer chosen? Now, that leads to Americans choosing cp1 all over the place, right? (Ah, upon proofreading before posting, I only now noticed the cpn sentence at the top. I'll remark on it at the very end.) --- While we are now intricately submerged in UTF and char width issues, one day, when D is a household word, programmers wouldn't have to even know about UTF and stuff. Just like last summer, when none of us European D folk knew anything about UTF, and just wrote stuff like foo = "Htel California"; and such, without further ado. This also means that (when D is perfect) no normal programmer ever touches string contents directly. They only use library routines for searching and all the other usual string operations. That would make it unusual to use single "characters", althoug not exceptional. If this is true, then we might consider blatantly skipping cp1 and cp2, and only having cp4 (possibly also renaming it utfchar). Then it would be a lot simpler for the programmer, right? He'd have even less need to start researching in this UTF swamp. And everything would "just work". This would make it possible for us to fully automate the extraction and insertion of single "characters" into our new strings. string foo = "gagaga"; utfchar bar = '\UFE9D'; // you don't want to know the name :-) utfchar baf = 'a'; foo ~= bar ~ baf; (I admit the last line doesn't probably work currently, but it should, IMHO.) Anyhow, the point being that if the utfchar type is 32 bits, then it doesn't hurt anybody, and also doesn't lead to gratuituous incompatibility with foreign characters -- which is the D aim all along. For completeness, we could have the painting casts (as opposed to converting casts). They'd be for the (seldom) situations where the programmer _does_ want to do serious tinkering on our strings. ubyte[] myarr1 = cast(ubyte[])foo; ushort[] myarr2 = cast(ushort[]) foo; uint[] myarr3 = cast(uint[]) foo; These give raw arrays, like exact images of the string. The burden of COW would lie on the programmer. To get a sane array of utfchar you'd have to write utfchar[] myarr4 = cast(utfchar[])toUTF32(foo); (This, of course could be nice to have as a library call. :-) ) While this might look tedious (for the library writer and for the library user), I think it is consistent, clear, and both easy to use and understand for the programmer not familiar with this D News Group. --- The cpn remark: I think D programs should be (as much as possible) UTF clean, even if the programmer didn't come to think about it. This has the advantage that his programs won't break embarrassingly when a guy in China suddenly uses them. It would also be quite nice if the programmer didn't have to think about such issues at all. Just code his stuff. Having cpn as something else than 32 bits, will prevent this dream. (Heh, and only having single chars as 32 bits would make writing the libraries so much easier, too, I think.)
Nov 24 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 24 Nov 2005 14:33:53 +0200, Georg Wrede <georg.wrede nospam.org>  
wrote:
 Regan Heath wrote:
  * add a new type/alias "cpn", this alias will be cp1, cp2 or cp4
 depending on the native encoding chosen. This allows efficient
 code, like:
  string s = "test";
 foreach(cpn c; s) {
 }
  * slicing string gives another string
  * indexing a string gives a cp1, cp2 or cp4

I hope you are not implying that indexing would choose between cp1..4 based on content? And if not, then the cpX would be either some "default", or programmer chosen? Now, that leads to Americans choosing cp1 all over the place, right?

I didn't think this part thru enough and Oskar gave me an example which broke my original idea. It seems for this to work cpn would need to be a type which changed size for each character, or always 32 bits large (as you suggest below). I was trying to avoid it being 32 bits large all the time, but it seems to be the only way it works.
 If this is true, then we might consider blatantly skipping cp1 and cp2,  
 and only having cp4 (possibly also renaming it utfchar).

 Then it would be a lot simpler for the programmer, right? He'd have even  
 less need to start researching in this UTF swamp. And everything would  
 "just work".

 This would make it possible for us to fully automate the extraction and  
 insertion of single "characters" into our new strings.

      string foo = "gagaga";
      utfchar bar = '\UFE9D'; // you don't want to know the name :-)
      utfchar baf = 'a';
      foo ~= bar ~ baf;

 (I admit the last line doesn't probably work currently, but it should,  
 IMHO.) Anyhow, the point being that if the utfchar type is 32 bits, then  
 it doesn't hurt anybody, and also doesn't lead to gratuituous  
 incompatibility with foreign characters -- which is the D aim all along.

It seems this may be the best solution. Oskar had a good name for it "uchar". It means quick and dirty ASCII apps will have to use a 32 bit sized char type. I can hear people complain already.. but it's odd that no-one is complaining about writef doing this exact same thing!
 For completeness, we could have the painting casts (as opposed to  
 converting casts). They'd be for the (seldom) situations where the  
 programmer _does_ want to do serious tinkering on our strings.

      ubyte[] myarr1 = cast(ubyte[])foo;
      ushort[] myarr2 = cast(ushort[]) foo;
      uint[] myarr3 = cast(uint[]) foo;

 These give raw arrays, like exact images of the string. The burden of  
 COW would lie on the programmer.

I was thinking of using properties (Sean's idea) to access the data as a certain type, eg. ubyte[] b = foo.utf8; ushort[] s = foo.utf16; uint[] i = foo.utf32; these properties would return the string in the specified encoding using those array types.
 ---

 The cpn remark: I think D programs should be (as much as possible) UTF  
 clean, even if the programmer didn't come to think about it. This has  
 the advantage that his programs won't break embarrassingly when a guy in  
 China suddenly uses them.

 It would also be quite nice if the programmer didn't have to think about  
 such issues at all. Just code his stuff.

 Having cpn as something else than 32 bits, will prevent this dream.
 (Heh, and only having single chars as 32 bits would make writing the  
 libraries so much easier, too, I think.)

Sad but probably true. I was hoping to avoid using 32bits everywhere :( Regan
Nov 24 2005
next sibling parent reply Derek Parnell <derek psych.ward> writes:
On Fri, 25 Nov 2005 09:34:30 +1300, Regan Heath wrote:

 
 Sad but probably true. I was hoping to avoid using 32bits everywhere :(

I also use the Euphoria programming language and this uses 32-bit characters exclusively. You do not notice any performance hit because of that. The only complaint that some people have is that string use too much RAM (but these people also use Windows 95). -- Derek Parnell Melbourne, Australia 25/11/2005 7:39:28 AM
Nov 24 2005
parent "Regan Heath" <regan netwin.co.nz> writes:
On Fri, 25 Nov 2005 07:41:15 +1100, Derek Parnell <derek psych.ward> wrote:
 On Fri, 25 Nov 2005 09:34:30 +1300, Regan Heath wrote:


 Sad but probably true. I was hoping to avoid using 32bits everywhere :(

I also use the Euphoria programming language and this uses 32-bit characters exclusively. You do not notice any performance hit because of that. The only complaint that some people have is that string use too much RAM (but these people also use Windows 95).

Interesting. In that case I think my "string" type has an advantage. The data could actually be stored in either UTF-8, UTF-16 or UTF-32 internally and only convertedd to/from the 32 bit char when required. Regan.
Nov 24 2005
prev sibling parent Georg Wrede <georg.wrede nospam.org> writes:
Regan Heath wrote:
 On Thu, 24 Nov 2005 14:33:53 +0200, Georg Wrede wrote:
 
 I was trying to avoid it being 32 bits large 
 all the  time, but it seems to be the only way it works.

I agree. And I share the feeling. :-)
 If this is true, then we might consider blatantly skipping cp1 and 
 cp2,  and only having cp4 (possibly also renaming it utfchar).

 This would make it possible for us to fully automate the extraction 
 and  insertion of single "characters" into our new strings.

      string foo = "gagaga";
      utfchar bar = '\UFE9D'; // you don't want to know the name :-)
      utfchar baf = 'a';
      foo ~= bar ~ baf;

It seems this may be the best solution. Oskar had a good name for it "uchar". It means quick and dirty ASCII apps will have to use a 32 bit sized char type. I can hear people complain already.. but it's odd that no-one is complaining about writef doing this exact same thing!

Not too many have dissected writef. Or else we'd have heard some complaints already. ;-) I actually thought about "uchar" for a while, but then I remembered that a lot of this utf disaster originates from unfortunate names. And C has a uchar type. So, I'd suggest "utfchar" or "unicode" or something to-the-point and unambiguous that's not in C.
 For completeness, we could have the painting casts (as opposed to  
 converting casts). They'd be for the (seldom) situations where the  
 programmer _does_ want to do serious tinkering on our strings.

      ubyte[] myarr1 = cast(ubyte[])foo;
      ushort[] myarr2 = cast(ushort[]) foo;
      uint[] myarr3 = cast(uint[]) foo;

 These give raw arrays, like exact images of the string.  
 The burden of COW would lie on the programmer.

I was thinking of using properties (Sean's idea) to access the data as a certain type, eg. ubyte[] b = foo.utf8; ushort[] s = foo.utf16; uint[] i = foo.utf32; these properties would return the string in the specified encoding using those array types.

So it'd be the same thing, except your code looks a lot nicer!
Nov 24 2005
prev sibling parent reply Derek Parnell <derek psych.ward> writes:
On Thu, 24 Nov 2005 21:46:50 +1300, Regan Heath wrote:

 Ok, it appears I picked some really bad type names in my proposal and it  
 is causing some confusion.

Regan, the idea stinks. Sorry, but that *is* the nice response. It is far more complicated than it needs to be. Maybe it's the name confusion, but I don't think so. When dealing with strings, almost nobody needs to deal with partial-characters. We really only need to deal with characters except for some obscure functionality (maybe interfacing with an external system?). So we don't need to deal with the individual bytes that make up the characters in the various UTF encodings. Sure, we will need to know how big a character is from time to time. For example, given a string (regardless of encoding format), we might need to know how many bytes the third character uses. The answer will depend on the UTF encoding *and* the code point value. Mostly we won't even need to know the encoding format. We might, if that is an interfacing requirement, and we might in some circumstances to improve performance. But generally, we shouldn't care. So how about we just have a string datatype called 'string'. The default encoding format in RAM is compiler dependant but we can on a declaration basis, define specific internal encoding format for a string. Furthermore, we can access any of the three UTF encoding formats for a string as a property of the string. The compiler would generate the call to transcode if required to. The string could also have array properties such that each element addressed an entire character. If one ever really needed to get down to the byte level of a character they could assign it to a new datatype called a 'unicode' (for example) and that would have properties such as the encoding format and byte size, and the bytes in a unicode could be accessed using array syntax too. string Foo = "Some string"; unicode C; C = Foo[4]; if (C.encoding = unicode.utf8) { foreach (ubyte b; C) { . . . } } We can then dismiss wchar, wchar[], dchar, dchar[] entirely. And make char and char[] array have the C/C++ semantics. If some function absolutely insisted on a utf16 string for example, ... SomeFunc(Foo.utf16); would pass the utf16 version of the string to the function. As for declarations ... utf16 { // force RAM encoding to be utf16 string Foo; string Bar; } string Qwerty; // RAM encoding is compiler choice. -- Derek Parnell Melbourne, Australia 24/11/2005 7:54:01 PM
Nov 24 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
Derek, I must have done a terrible job explaining this, because you've  
completely missunderstood me, in fact your counter proposal is essentially  
what my proposal was intended to be.

More inline...

On Thu, 24 Nov 2005 20:15:56 +1100, Derek Parnell <derek psych.ward> wrote:
 On Thu, 24 Nov 2005 21:46:50 +1300, Regan Heath wrote:

 Ok, it appears I picked some really bad type names in my proposal and it
 is causing some confusion.

Regan, the idea stinks. Sorry, but that *is* the nice response. It is far more complicated than it needs to be. Maybe it's the name confusion, but I don't think so. When dealing with strings, almost nobody needs to deal with partial-characters.

I think you're confused. My proposal removes the need for dealing with partial characters completely, if you think otherwise then I've done a bad job explaining it.
 So we don't need to deal with the individual bytes that make up the
 characters in the various UTF encodings. Sure, we will need to know how  
 big a character is from time to time. For example, given a string  
 (regardless
 of encoding format), we might need to know how many bytes the third
 character uses. The answer will depend on the UTF encoding *and* the code
 point value.

Exactly my point, and the reason for the "cpn" alias.
 Mostly we won't even need to know the encoding format. We might, if that  
 is an interfacing requirement, and we might in some circumstances to  
 improve
 performance. But generally, we shouldn't care.

Yes, exactly.
 So how about we just have a string datatype called 'string'. The default
 encoding format in RAM is compiler dependant but we can on a declaration
 basis, define specific internal encoding format for a string.  
 Furthermore, we can access any of the three UTF encoding formats for a  
 string as a
 property of the string. The compiler would generate the call to transcode
 if required to. The string could also have array properties such that  
 each element addressed an entire character.

That, is exactly what I proposed.
 We can then dismiss wchar, wchar[], dchar, dchar[] entirely. And make  
 char and char[] array have the C/C++ semantics.

I proposed exactly that, except char[] should not exist either. char and char* are all that are required. Regan
Nov 24 2005
parent reply Derek Parnell <derek psych.ward> writes:
On Thu, 24 Nov 2005 22:49:36 +1300, Regan Heath wrote:

 Derek, I must have done a terrible job explaining this, because you've  
 completely missunderstood me, in fact your counter proposal is essentially  
 what my proposal was intended to be.

You seemed to be wanting to have data types that could only hold characters-fragments. I can't see the point of that. If strings must be arrays, then let there be an atomic data type that represents a character and then strings can be arrays of characters. The UTF encoding of the string is just an implementation detail then. All indexing would be done on a character basis regardless of the underlying encoding. In other words, if 'uchar' is the data type that holds a character then alias uchar[] string; could be used. The main 'new thinking' is that uchar could actually have variable size in terms of bytes. It could be 1, 2 or 4 bytes depending on the character it holds and the encoding used at the time. However I still prefer my earlier suggestion.
 On Thu, 24 Nov 2005 20:15:56 +1100, Derek Parnell <derek psych.ward> wrote:
 On Thu, 24 Nov 2005 21:46:50 +1300, Regan Heath wrote:

 Ok, it appears I picked some really bad type names in my proposal and it
 is causing some confusion.

Regan, the idea stinks. Sorry, but that *is* the nice response. It is far more complicated than it needs to be. Maybe it's the name confusion, but I don't think so. When dealing with strings, almost nobody needs to deal with partial-characters.

I think you're confused. My proposal removes the need for dealing with partial characters completely, if you think otherwise then I've done a bad job explaining it.

Apparently, or I'm a bit thicker than suspected ;-)
 So we don't need to deal with the individual bytes that make up the
 characters in the various UTF encodings. Sure, we will need to know how  
 big a character is from time to time. For example, given a string  
 (regardless
 of encoding format), we might need to know how many bytes the third
 character uses. The answer will depend on the UTF encoding *and* the code
 point value.

Exactly my point, and the reason for the "cpn" alias.

But why the need for cp1, cp2, cp4?
 Mostly we won't even need to know the encoding format. We might, if that  
 is an interfacing requirement, and we might in some circumstances to  
 improve
 performance. But generally, we shouldn't care.

Yes, exactly.
 So how about we just have a string datatype called 'string'. The default
 encoding format in RAM is compiler dependant but we can on a declaration
 basis, define specific internal encoding format for a string.  
 Furthermore, we can access any of the three UTF encoding formats for a  
 string as a
 property of the string. The compiler would generate the call to transcode
 if required to. The string could also have array properties such that  
 each element addressed an entire character.

That, is exactly what I proposed.
 We can then dismiss wchar, wchar[], dchar, dchar[] entirely. And make  
 char and char[] array have the C/C++ semantics.

I proposed exactly that, except char[] should not exist either. char and char* are all that are required.

Are you saying that we can have arrays of everything except char? I don't think that'll fly. And char* is a pointer to a single char. -- Derek Parnell Melbourne, Australia 25/11/2005 7:27:14 AM
Nov 24 2005
next sibling parent "Regan Heath" <regan netwin.co.nz> writes:
On Fri, 25 Nov 2005 07:37:18 +1100, Derek Parnell <derek psych.ward> wrote:
 On Thu, 24 Nov 2005 22:49:36 +1300, Regan Heath wrote:

 Derek, I must have done a terrible job explaining this, because you've
 completely missunderstood me, in fact your counter proposal is  
 essentially
 what my proposal was intended to be.

You seemed to be wanting to have data types that could only hold characters-fragments. I can't see the point of that.

No, never fragments, always complete code points. I tried to stress this point. The 8 bit type would hold all the code points with values that fit in 8 bits and never anything else, it's value would always be a code point, not a fragment.
 If strings must be arrays, then let there be an atomic data type that
 represents a character and then strings can be arrays of characters. The
 UTF encoding of the string is just an implementation detail then. All
 indexing would be done on a character basis regardless of the underlying
 encoding.

 In other words, if 'uchar' is the data type that holds a character then

   alias uchar[] string;

 could be used. The main 'new thinking' is that uchar could actually have
 variable size in terms of bytes. It could be 1, 2 or 4 bytes depending on
 the character it holds and the encoding used at the time.

 However I still prefer my earlier suggestion.

I suspect now that all individual characters will have to be represented by a 32 bit type, uchar is a good name for it. If you take my proposal, throw away all the garbage about cp1, cp2, cp4, and cpn then replace them with a new type "uchar" which is 32 bits large, and always use this to represent individual characters then it starts to work, I believe.
 Apparently, or I'm a bit thicker than suspected ;-)

I've just used confusing terms and done a bad job explaining I think.
 But why the need for cp1, cp2, cp4?

This was intended to avoid ASCII programs having to use a 32 bit type for all their characters, and so on.
 I proposed exactly that, except char[] should not exist either.
 char and char* are all that are required.

Are you saying that we can have arrays of everything except char?

Yes. Because we don't need an array of char[]. It's simply there for interfacing to C.
 I don't think that'll fly.  And char* is a pointer to a single char.

Technically true, but when you're talking about a C function it's a pointer to the start of a string which is null terminated. That's all we need it for in D. Regan
Nov 24 2005
prev sibling parent reply Bruno Medeiros <daiphoenixNO SPAMlycos.com> writes:
Derek Parnell wrote:
 On Thu, 24 Nov 2005 22:49:36 +1300, Regan Heath wrote:
 
 
Derek, I must have done a terrible job explaining this, because you've  
completely missunderstood me, in fact your counter proposal is essentially  
what my proposal was intended to be.

You seemed to be wanting to have data types that could only hold characters-fragments. I can't see the point of that. If strings must be arrays, then let there be an atomic data type that represents a character and then strings can be arrays of characters. The UTF encoding of the string is just an implementation detail then. All indexing would be done on a character basis regardless of the underlying encoding. In other words, if 'uchar' is the data type that holds a character then alias uchar[] string; could be used. The main 'new thinking' is that uchar could actually have variable size in terms of bytes. It could be 1, 2 or 4 bytes depending on the character it holds and the encoding used at the time. However I still prefer my earlier suggestion.

primitive type with *variable size* ? It's plain nuts to implement, no wait, it's actually downright impossible. If you have a uchar variable (not an array), how much space do you allocate for it, if it has variable-size? The only way to implement this would be with a fixed-size equal to the max possible size (4 bytes). That would be a dchar then... -- Bruno Medeiros - CS/E student "Certain aspects of D are a pathway to many abilities some consider to be... unnatural."
Nov 25 2005
parent reply Derek Parnell <derek psych.ward> writes:
On Fri, 25 Nov 2005 14:06:34 +0000, Bruno Medeiros wrote:

[snip]

 could be used. The main 'new thinking' is that uchar could actually have
 variable size in terms of bytes. It could be 1, 2 or 4 bytes depending on
 the character it holds and the encoding used at the time.
 
 However I still prefer my earlier suggestion.
  
 

primitive type with *variable size* ? It's plain nuts to implement, no wait, it's actually downright impossible. If you have a uchar variable (not an array), how much space do you allocate for it, if it has variable-size? The only way to implement this would be with a fixed-size equal to the max possible size (4 bytes). That would be a dchar then...

Well not *actually* impossible but certainly something you'd only do if you didn't care about performance. However, I was really talking on a conceptual level rather than an implementation level. As you and others have said, it would most likely be implemented as a 32-bit unsigned integer however certain bits are redundant and are thus (conceptually) not significant. And as I have said earlier, I already work in such a world. The Euphoria programming language only has 32-bit characters. -- Derek Parnell Melbourne, Australia 26/11/2005 8:40:39 AM
Nov 25 2005
parent reply Bruno Medeiros <daiphoenixNO SPAMlycos.com> writes:
Derek Parnell wrote:
 On Fri, 25 Nov 2005 14:06:34 +0000, Bruno Medeiros wrote:
 
 [snip]
 
 
could be used. The main 'new thinking' is that uchar could actually have
variable size in terms of bytes. It could be 1, 2 or 4 bytes depending on
the character it holds and the encoding used at the time.

However I still prefer my earlier suggestion.
 

Whoa, did you ever stop to think on the implications of having a primitive type with *variable size* ? It's plain nuts to implement, no wait, it's actually downright impossible. If you have a uchar variable (not an array), how much space do you allocate for it, if it has variable-size? The only way to implement this would be with a fixed-size equal to the max possible size (4 bytes). That would be a dchar then...

Well not *actually* impossible but certainly something you'd only do if you didn't care about performance.

even more space. I honestly don't see how it would be possible, while using less than 4 bytes and maintaining all other D features/properties (performance not considerated).
 However, I was really talking on a
 conceptual level rather than an implementation level. As you and others
 have said, it would most likely be implemented as a 32-bit unsigned integer
 however certain bits are redundant and are thus (conceptually) not
 significant.
 

be simply be a dchar[] . I've only skimmed trough this discussion but people (Regan & others?) wanted a string type that was space-eficient, allowing itself to be enconded in UTF-8, UTF-16, etc, thus dchar[]/uchar[] would not be acceptable. Unless you wanted this uchar[] to be a basic type by itself, and not an array of basic uchars (which would work, but would be a horrible design) -- In fact, and I'm gonna go a bit on rant mode here (not directed at you in particular, Derek), but I've skimmed through this whole series of threads about Unicode and strings, and I'm getting a bit pissed with all of those meaningless posts based on wrong assumptions, wrong terminology, crazy or unfeasable language changes, and all of this for a problem I've yet failed to grasp why it cannot be fully solved with a dchar array or with a custom made String class (custom-made, that is, *user-coded*, not part of the language). I admit I have no Unicode coding experience, so indeed *I may be* missing something, but on every new thread made all I see is progressively more crazy, ridiculous ideas about a problem I do not see. -- Bruno Medeiros - CS/E student "Certain aspects of D are a pathway to many abilities some consider to be... unnatural."
Nov 26 2005
parent Derek Parnell <derek psych.ward> writes:
On Sat, 26 Nov 2005 13:30:28 +0000, Bruno Medeiros wrote:


[snip]
 Thus one would have the dchar type, and this new "Unified String" would
 be simply be a dchar[] .

 I've skimmed through this whole series of
 threads about Unicode and strings, and 

 I've yet failed to grasp why it cannot be fully solved with a
 dchar array or with a custom made String class (custom-made, that is,
 *user-coded*, not part of the language). I admit I have no Unicode
 coding experience, so indeed *I may be* missing something, but on every
 new thread made all I see is progressively more crazy, ridiculous ideas
 about a problem I do not see.

I agree that it is much better to identify the problem *before* one tries to fix it. I'm sure Water has been having a nice little chuckle at our meandering ways. I see 'the problem' as ... ** We have choice about the representation of strings in D, and thus at times we introduce a degree of ambiguity in our code that the compiler has trouble resolving. ** The 'char' data type is performing multiple roles. In one aspect, it is a fragment of a character in an utf-8 encoded string, and in other aspects it is a byte-sized character for ASCII and C/C++ compatibility purposes. This can be confusing to coders not used to thinking internationally. ** Indexing string that are based on 'char' and 'wchar' can cause bugs because it is possible to access character fragments rather than complete characters. There are some other issues which are not language related, and have to deal with string manipulation that assume ASCII strings only - such as the 'strip()' function which doesn't recognize all the Unicode white-space characters just the ASCII ones. -- Derek Parnell Melbourne, Australia 27/11/2005 7:40:57 AM
Nov 26 2005
prev sibling next sibling parent reply Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Regan Heath wrote:

 * add a new type/alias "utf", this would alias utf8, 16 or 32. It  
 represents the application specific native encoding. This allows 
 efficient  code, like:
 
 string s = "test";
 foreach(utf c; s) {
 }
 
 regardless of the applications selected native encoding.

I will rewrite this with your changed names (cp*):
 * add a new type/alias "cpn", this would alias cp1, cp2 or cp4. It
 represents the application specific native encoding. This allows
 efficient  code, like:

 string s = "test";
 foreach(cpn c; s) {
 }

 regardless of the applications selected native encoding.

Say you instead have: string s = "smrgsbord"; foreach(cpn c; s) { } This code would then work on Win32 (with UTF-16 being the native encoding), but not on Linux (with UTF-8). You have introduced platform dependence where there previously was none. What do you gain by this? As I see it, there are only two views you need on a unicode string: a) The code units b) The unicode characters By your suggestion, there would be a third view: c) The unicode characters that are encoded by a single code unit. Why is this useful? Should the "smrgsbord"-example above throw an error? Isn't what you want instead: assert_only_contains_single_code_unit_characters_in_native_encoding(string) /Oskar
Nov 24 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 24 Nov 2005 10:18:20 +0100, Oskar Linde  
<oskar.lindeREM OVEgmail.com> wrote:
 Regan Heath wrote:

 * add a new type/alias "utf", this would alias utf8, 16 or 32. It   
 represents the application specific native encoding. This allows  
 efficient  code, like:
  string s = "test";
 foreach(utf c; s) {
 }
  regardless of the applications selected native encoding.

I will rewrite this with your changed names (cp*): > * add a new type/alias "cpn", this would alias cp1, cp2 or cp4. It > represents the application specific native encoding. This allows > efficient code, like: > > string s = "test"; > foreach(cpn c; s) { > } > > regardless of the applications selected native encoding. Say you instead have: string s = "smrgsbord"; foreach(cpn c; s) { } This code would then work on Win32 (with UTF-16 being the native encoding), but not on Linux (with UTF-8).

No. "string" would be UTF-8 encoded internally on both platforms. My proposal stated that "cpn" would thus be an alias for "cp1" but clearly that idea isn't going to work in this case as (I'm assuming) it's impossible to represent some of those characters using a single byte. Java uses an int, maybe we should just do the same?
 You have introduced platform dependence where there previously was none.
 What do you gain by this?

No, there is no platform dependance. The choice of encoding is entirely up to the programmer, they choose a default encoding for each program they write, it defaults to UTF-8.
 As I see it, there are only two views you need on a unicode string:
 a) The code units
 b) The unicode characters

(a) is seldom required. (b) is the common and thus goal view IMO.
 By your suggestion, there would be a third view:
 c) The unicode characters that are encoded by a single code unit.

(c) was intended to be equal to (b). It was intended that we have 3 types so that ASCII programs would not be forced to use an int sized variable for single character values. It seems we're stuck doing that.
 Why is this useful?

It's not, it's not what I intended.
 Should the "smrgsbord"-example above throw an error?

No, certainly not.
 Isn't what you want instead:
 assert_only_contains_single_code_unit_characters_in_native_encoding(string)

I have no idea what you mean here. Regan
Nov 24 2005
parent reply Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Regan Heath wrote:
 On Thu, 24 Nov 2005 10:18:20 +0100, Oskar Linde  
 <oskar.lindeREM OVEgmail.com> wrote:
 
 Regan Heath wrote:

 * add a new type/alias "utf", this would alias utf8, 16 or 32. It   
 represents the application specific native encoding. This allows  
 efficient  code, like:
  string s = "test";
 foreach(utf c; s) {
 }
  regardless of the applications selected native encoding.

I will rewrite this with your changed names (cp*): > * add a new type/alias "cpn", this would alias cp1, cp2 or cp4. It > represents the application specific native encoding. This allows > efficient code, like: > > string s = "test"; > foreach(cpn c; s) { > } > > regardless of the applications selected native encoding. Say you instead have: string s = "smrgsbord"; foreach(cpn c; s) { } This code would then work on Win32 (with UTF-16 being the native encoding), but not on Linux (with UTF-8).

No. "string" would be UTF-8 encoded internally on both platforms.

 My proposal stated that "cpn" would thus be an alias for "cp1" but 

Ok. I assumed cpn would be the platform native (preferred) encoding.
 clearly  that idea isn't going to work in this case as (I'm assuming) 
 it's  impossible to represent some of those characters using a single 
 byte. Java  uses an int, maybe we should just do the same?

D uses dchar. Better would maybe be to rename it to char (or maybe character), giving: utf8 (todays char) utf16 (todays wchar) char (todays dchar)
 As I see it, there are only two views you need on a unicode string:
 a) The code units
 b) The unicode characters

(a) is seldom required. (b) is the common and thus goal view IMO.

Actually, I think it is the other way around. (b) is seldom required. You can search, split, trim, parse, etc.. D:s char[], without any regard of encoding. This is the beauty of UTF-8 and the reason D strings all work on code units rather than characters. When would you actually need character based indexing? I believe the answer is less often than you think. /Oskar
Nov 24 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 24 Nov 2005 12:01:04 +0100, Oskar Linde  
<oskar.lindeREM OVEgmail.com> wrote:
 Regan Heath wrote:
 On Thu, 24 Nov 2005 10:18:20 +0100, Oskar Linde   
 <oskar.lindeREM OVEgmail.com> wrote:

 Regan Heath wrote:

 * add a new type/alias "utf", this would alias utf8, 16 or 32. It    
 represents the application specific native encoding. This allows   
 efficient  code, like:
  string s = "test";
 foreach(utf c; s) {
 }
  regardless of the applications selected native encoding.

I will rewrite this with your changed names (cp*): > * add a new type/alias "cpn", this would alias cp1, cp2 or cp4. It > represents the application specific native encoding. This allows > efficient code, like: > > string s = "test"; > foreach(cpn c; s) { > } > > regardless of the applications selected native encoding. Say you instead have: string s = "smrgsbord"; foreach(cpn c; s) { } This code would then work on Win32 (with UTF-16 being the native encoding), but not on Linux (with UTF-8).


 My proposal stated that "cpn" would thus be an alias for "cp1" but

Ok. I assumed cpn would be the platform native (preferred) encoding.

Not platform native, application native. But it's not going to work anyway. It seems an int sized char type is required, I was trying to avoid that.
 clearly  that idea isn't going to work in this case as (I'm assuming)  
 it's  impossible to represent some of those characters using a single  
 byte. Java  uses an int, maybe we should just do the same?

D uses dchar. Better would maybe be to rename it to char (or maybe character), giving: utf8 (todays char) utf16 (todays wchar) char (todays dchar)
 As I see it, there are only two views you need on a unicode string:
 a) The code units
 b) The unicode characters


Actually, I think it is the other way around. (b) is seldom required. You can search, split, trim, parse, etc.. D:s char[], without any regard of encoding.

If you split it without regard for encoding you can get 1/2 a character, which is then an illegal UTF-8 sequence.
 This is the beauty of UTF-8 and the reason D strings all work on code  
 units rather than characters.

But people don't care about code units, they care about characters. When do you want to inspect or modify a single code unit? I would say, just about never. On the other hand you might want to change the 4th character of "smrgsbord" which may be the 4th, 5th, 6th, or 7th index in a char[] array. Ick.
 When would you actually need character based indexing?
 I believe the answer is less often than you think.

Really? How often do you care what UTF-8 fragments are used to represent your characters? The answer _should_ be never, however D forces you to know, you cannot replace the 4th letter of "smrgsbord" without knowing. This is the problem, IMO. Regan
Nov 24 2005
next sibling parent reply =?ISO-8859-15?Q?Jari-Matti_M=E4kel=E4?= <jmjmak invalid_utu.fi> writes:
Regan Heath wrote:
 But people don't care about code units, they care about characters. 
 When  do you want to inspect or modify a single code unit? I would say, 
 just  about never. On the other hand you might want to change the 4th 
 character  of "smrgsbord" which may be the 4th, 5th, 6th, or 7th index 
 in a char[]  array. Ick.

True. BTW, is there a bug in std.string.insert? I tried to do: char[] a = "blaahblaah"; std.string.insert(a, 5, ""); std.stdio.writefln(a); Outputs: blaahblaah
 When would you actually need character based indexing?
 I believe the answer is less often than you think.

Really? How often do you care what UTF-8 fragments are used to represent your characters? The answer _should_ be never, however D forces you to know, you cannot replace the 4th letter of "smrgsbord" without knowing. This is the problem, IMO.

I agree. You don't need it very often, but when you do, there's currently no possibility to do that. I think char[]-slicing and indexing should be a bit better (work in the Unicode character level) since you _never_ want to change code units. (And in case you do, just cast it to void[]) Jari-Matti
Nov 24 2005
parent Derek Parnell <derek psych.ward> writes:
On Fri, 25 Nov 2005 00:31:32 +0200, Jari-Matti Mkel wrote:


 True. BTW, is there a bug in std.string.insert? I tried to do:
 
 char[] a = "blaahblaah";
 std.string.insert(a, 5, "");
 std.stdio.writefln(a);
 
 Outputs:
 
 blaahblaah

No bug. The function is not designed to update the same string passed to the function. It returns an updated string. char[] a = "blaahblaah"; a = std.string.insert(a, 5, ""); std.stdio.writefln(a); -- Derek (skype: derek.j.parnell) Melbourne, Australia 25/11/2005 10:04:41 AM
Nov 24 2005
prev sibling parent reply Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
In article <ops0rhghyl23k2f5 nrage.netwin.co.nz>, Regan Heath says...
On Thu, 24 Nov 2005 12:01:04 +0100, Oskar Linde  
<oskar.lindeREM OVEgmail.com> wrote:
 As I see it, there are only two views you need on a unicode string:
 a) The code units
 b) The unicode characters


Actually, I think it is the other way around. (b) is seldom required. You can search, split, trim, parse, etc.. D:s char[], without any regard of encoding.

If you split it without regard for encoding you can get 1/2 a character, which is then an illegal UTF-8 sequence.

By split, I meant this: char[][] words = "abc def ghi jkl".split(" ");
 This is the beauty of UTF-8 and the reason D strings all work on code  
 units rather than characters.

But people don't care about code units, they care about characters. [...]

Most of the time people care about string contents, neither code units nor characters. Naturally, I'm biased by my own experience. I have written a few applications in D dealing with UTF-8 data, including parsing grammar definition files and communicating with web servers, but not once have I needed character based indexing. One reason may be that all delimeters used are ASCII, and therefore only occupy a single code unit, but I would assume that this is typical for most data. When dealing with UTF-8 streams, you want searching and parsing to work on indices (positions) within this stream, not on the character count up to this position. A code unit index gives you the direct byte position of the stream, whereas a character index would require iterating the entire stream up to the indexed position. The performance difference is hardly negligible.
[...] When  
do you want to inspect or modify a single code unit? I would say, just  
about never. On the other hand you might want to change the 4th character  
of "smrgsbord" which may be the 4th, 5th, 6th, or 7th index in a char[]  
array. Ick.

How often do you need to change the 4th character of a string? I think that scenario is just as unlikely. Of course, there are cases where you need character based access, and that is what dchar[] is ideal for*. If you instead want to sacrifice performance for better memory footprint, use a wrapper class. What I don't agree with is making this sacrifice in performance the default, when its gains are so seldom needed. *) In many cases, such as word processors and similar, you need more efficient data structures than flat arrays. A basic character based string would not be of much help.
 When would you actually need character based indexing?
 I believe the answer is less often than you think.

Really? How often do you care what UTF-8 fragments are used to represent your characters? The answer _should_ be never, however D forces you to know, you cannot replace the 4th letter of "smrgsbord" without knowing. This is the problem, IMO.

I very seldom, if ever, care what UTF-8 framents are used to represent the data as long as I know that ASCII characters (those whose character literals are assignable to a char) are represented by a single code unit. You say that users of char[] need to know some things about UTF-8, and I can't argue with that. Maybe the docs should recommend dchar[] for users that want to remain UTF ignorant. :) Regards, Oskar
Nov 25 2005
parent "Kris" <fu bar.com> writes:
"Oskar Linde" <oskar.lindeREM OVEgmail.com> wrote
 Most of the time people care about string contents, neither code units nor
 characters.
 Naturally, I'm biased by my own experience. I have written a few 
 applications in
 D dealing with UTF-8 data, including parsing grammar definition files and
 communicating with web servers, but not once have I needed character based
 indexing. One reason may be that all delimeters used are ASCII, and 
 therefore
 only occupy a single code unit, but I would assume that this is typical 
 for most
 data.

Absolutely right. This is why, for example, URI classes will remain char[] based. IRI extensions are applied simply by assuming the content is utf8.
[...] When
do you want to inspect or modify a single code unit? I would say, just
about never. On the other hand you might want to change the 4th character
of "smrgsbord" which may be the 4th, 5th, 6th, or 7th index in a char[]
array. Ick.

How often do you need to change the 4th character of a string? I think that scenario is just as unlikely. Of course, there are cases where you need character based access, and that is what dchar[] is ideal for*. If you instead want to sacrifice performance for better memory footprint, use a wrapper class. What I don't agree with is making this sacrifice in performance the default, when its gains are so seldom needed. *) In many cases, such as word processors and similar, you need more efficient data structures than flat arrays. A basic character based string would not be of much help.

Right on. Such things are very much application specific. That, IMO, is where much of the general confusion stems from.
Nov 25 2005
prev sibling next sibling parent reply =?ISO-8859-15?Q?Jari-Matti_M=E4kel=E4?= <jmjmak invalid_utu.fi> writes:
Regan, your proposal is absolutely too complex. I don't get it and I 
really don't like it. D is supposed to be a _simple_ language. Here's an 
alternative proposal:

-Allowed text string types:

char, char[] (we don't need silly aliases nor wchar/dchar)

-Text string implementation:

char - Unicode code unit (UTF-8, it's up to the compiler vendor to 
decide between 1-4xbytes and an int)

char[] - array of char-types, thus a valid Unicode string encoded in 
UTF-8, no BOM is needed because all char[]s are UTF-8.

-Text string operations:

char a = '', b = '';
char[] s = "", t;

t ~= a;		// s == [ '' ]
t ~= b;		// s == [ '', '' ] == ""

s[1..2] == ""

foreach(char c; s) writefln(c);	// outputs:  \n  \n  \n

-I/O:

writef/writefln - does implicit conversion (utf-8 -> terminal encoding)
puts/gets -

File I/O - through UnicodeStream() (handles encoding issues)

-Conversion:

std.utf - two functions needed:

byte[] encode(char[] string, EncodingType et)
char[] decode(byte[] stream, EncodingType et)

-Compatibility:

This new char[] is fully compatible with C-language char*, when 0-127 
ASCII-values and a trailing zero-value are used.

Access to Windows/Unix-API available (std.utf.[en/de]code)
Access to Unicode files available (std.stream.UnicodeStream)

-Advantages:

OS/compiler vendor independent
Easy to use

-Disadvantages:

Hard to implement (or is it, Walter seems to have problems with UTF-8 - 
OTOH this proposal doesn't imply you to implement strings using UTF-8, 
you can also use "fixed-width" UTF16/32)
It's not super high performance (need to convert a lot on Windows&legacy 
systems)
Indexing problem (as UTF-8 streams are variable length, it's hard to 
tell the exact position of a single character. This affects all string 
operations except concatenating.)

---

Please stop whining about the slowness of utf-conversions. If it's 
really so slow, I would certainly want to see some real world benchmarks.
Nov 24 2005
next sibling parent Dawid =?UTF-8?B?Q2nEmcW8YXJraWV3aWN6?= <dawid.ciezarkiewicz gmail.com> writes:
Jari-Matti Mäkelä wrote:

 Regan, your proposal is absolutely too complex. I don't get it and I
 really don't like it. D is supposed to be a _simple_ language. Here's an
 alternative proposal:
 [CUT]

+1
Nov 24 2005
prev sibling parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 24 Nov 2005 13:34:51 +0200, Jari-Matti Mkel  
<jmjmak invalid_utu.fi> wrote:
 Regan, your proposal is absolutely too complex. I don't get it and I  
 really don't like it. D is supposed to be a _simple_ language. Here's an  
 alternative proposal:

<snip> Thanks for your opinion. It appears some parts of my idea were badly thought out. I was trying to end up with something simple, it seems a few of my choices were bad ones and they simply complicated the idea. I was trying to avoid picking any 1 type over the others (as you have suggested here). It appears now that I should replace all my talk about cp1, cp2, cp4 and cpn with "all characters are stored in a 32 bit type called uchar". If anyone has a problem with that, I'd direct them to take a look at std.format.doFormat and std.stdio.writef which convert all char[] data into individual dchar's before converting it back to UTF-8 for output to the screen.
 Please stop whining about the slowness of utf-conversions. If it's  
 really so slow, I would certainly want to see some real world benchmarks.

I mention performance only becase people have been concerned with it in the past. I too have no idea how much time it takes and would like to see a benchmark. The fact that D is already doing it with writef and no-one has complained... Regan.
Nov 24 2005
parent =?ISO-8859-15?Q?Jari-Matti_M=E4kel=E4?= <jmjmak invalid_utu.fi> writes:
Regan Heath wrote:
 On Thu, 24 Nov 2005 13:34:51 +0200, Jari-Matti Mkel  
 <jmjmak invalid_utu.fi> wrote:
 
 Regan, your proposal is absolutely too complex. I don't get it and I  
 really don't like it. D is supposed to be a _simple_ language. Here's 
 an  alternative proposal:


Sorry for being a bit impolite. I just wanted to show that it's completely possible to write Unicode-compliant programs without the need for several string-keywords. I believe a careful design&implementation removes most of the performance drawbacks.
 Thanks for your opinion. It appears some parts of my idea were badly  
 thought out. I was trying to end up with something simple, it seems a 
 few  of my choices were bad ones and they simply complicated the idea.
 

is perfect => designing a modern programming language isn't as easy as it might have seen.
 I was trying to avoid picking any 1 type over the others (as you have  
 suggested here).

Actually I have to change my opinion. I think it would be good, if the compiler were allowed to choose the correct encoding. I don't think there will be any serious problems since nowadays most Win32-things use UTF-16 and *nix-systems UTF-8.
 It appears now that I should replace all my talk about cp1, cp2, cp4 
 and  cpn with "all characters are stored in a 32 bit type called uchar". 
 If  anyone has a problem with that, I'd direct them to take a look at  
 std.format.doFormat and std.stdio.writef which convert all char[] data  
 into individual dchar's before converting it back to UTF-8 for output 
 to  the screen.

That is one solution. Although I might let the compiler decide the encoding.
 Please stop whining about the slowness of utf-conversions. If it's  
 really so slow, I would certainly want to see some real world benchmarks.

I mention performance only becase people have been concerned with it in the past. I too have no idea how much time it takes and would like to see a benchmark. The fact that D is already doing it with writef and no-one has complained...

I can't say anything about the overall complexity class for programs that do Unicode, but at least my simple experiments [1] show that unoptimized use of writefln is 'only' 50% slower than optimized use of printf in C (both using the same gcc-backend). Though I'm not 100% sure this program of mine actually did any transcoding. In addition, I think most 'static' conversions can be precalculated. [1] digitalmars.D.learn/1983 Jari-Matti
Nov 24 2005
prev sibling parent "Regan Heath" <regan netwin.co.nz> writes:
I want to thank everyone for reading and posting opinions on my proposal.

It appears I have done a bad job explaining some of it, and some of it  
simply doesn't work. I have a modified idea in mind which I think might  
work a whole bunch better and should also be much simpler too.

Thanks everyone.

Regan
Nov 24 2005