www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - ICU (International Components for Unicode)

reply Arcane Jill <Arcane_member pathlink.com> writes:
Following Ben's mention of ICU, I've been checking out what it does and doesn't
do. Basically, it does EVERYTHING. There is the work of years there. Not just
Unicode stuff, but a whole swathe of classes for internationalization and
transcoding. It would take me a very long time to duplicate that. It's also
free, open source, and with a license which basically says there's no problem
with our using it.

So I'm thinking seriously about ditching the whole etc.unicode project and
replacing it with a wrapper around ICU.

It's not completely straightforward. ICU is written in C++ (not C), and so we
can't link against it directly. It uses classes, not raw functions. So, I'd have
to write a C wrapper around ICU which gave me a C interface, and /then/ I'd have
to write a D wrapper to call the C wrapper - at which point we could get the
classes back again (and our own choice of architecture, so plugging into std or
mango streams won't suffer).

But the outermost (D) wrapper can, at least, be composed of D classes.

If we want D to be the language of choice for Unicode, we would need all this
functionality. So, if we went the ICU route, we'd need to bundle ICU (currently
a ten megabyte zip file) with DMD, along with whatever wrapper I come up with.
(etc.unicode is not likely to be smaller).

I'd like to see some discussion on this. Read this page to inform yourself:
http://oss.software.ibm.com/icu/userguide/index.html


Finally, back to strings. Ben was right. The ICU says:

"In order to take advantage of Unicode with its large character repertoire and
its well-defined properties, there must be types with consistent definitions and
semantics. The Unicode standard defines a default encoding based on 16-bit code
units. This is supported in ICU by the definition of the UChar to be an unsigned
16-bit integer type. This is the base type for character arrays for strings in
ICU."

Get that? A *16-BIT* type is the basis for ICU strings. ICU defines no special
string class - a string is just an array of wchars. So obviously, if we go the
ICU route, I withdraw my suggestion to ditch the wchar.

What I now recommend is:

(1) Ditch "etc.unicode" in favor of - let's call it "etc.icu" (a D wrapper
around ICU). Eventually I hope for this to change into "std.icu" (as I
originally hoped that "etc.unicode" would turn into "std.unicode").

(2) Ditch the char. 8-bits is really too small for a character these days,
honestly, and all previous arguments still apply. The existence of char only
encourages ASCII and discourages Unicode anyway.

(3) Native D strings shall be arrays of wchars. This means that
Object.toString() must return a wchar[], and string literals in D source must
compile to wchar[]s. UCI's type UChar would map directly to wchar. To reinforce
this, there should probably be an

#    alias wchar[] string;

in object.d.

(4) We retain dchar (so that we can get character properties), but all string
code is based on wchar[]s, not dchar[]s. UCI's type UChar32 would map directly
to dchar.

(5) Transcoding/streams/etc. go ahead as planned, but based around wchar[]s
instead of dchar[]s.

(6) That's pretty much it, although once "char" is gone, we could rename "wchar"
as "char" (a la Java).


Discussion please? And I really do want this talked through because it affects D
work I'm currently involved in.

Input is also requested from Walter - in particular the request that
Object.toString() be re-jigged to return wchar[] instead of char[].

Okay, let's chew this one over.

Jill
Aug 23 2004
next sibling parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgcv4n$2fsf$1 digitaldaemon.com>, Arcane Jill says...

It's not completely straightforward. ICU is written in C++ (not C), and so we
can't link against it directly. It uses classes, not raw functions. So, I'd have
to write a C wrapper around ICU which gave me a C interface, and /then/ I'd have
to write a D wrapper to call the C wrapper - at which point we could get the
classes back again (and our own choice of architecture, so plugging into std or
mango streams won't suffer).

Or I could port it to D! Or... In article <cgd0h9$2ggf$1 digitaldaemon.com>, Juanjo =?ISO-8859-15?Q?=C1lvarez?= says...
AJ, I don't know a shit^D^D^D^D too much about Unicode but you excitement
about ICU is really contagious, only one question, are the C wrappers at
the same level than the C++/Java ones? If so it seems that with a little
easy and boring (compared to writing etc.unicode) wrapping we're going to
have a first-class Unicode lib :) => (i18n version of <g>)

I don't know, as I've only just started looking into it. But either way (port or wrap C) we move the time spend on development from a year or so down to only a few months. It really is worth thinking about, but it /does/ mean that D really should standardize on wchar[] strings, and this has consequences for (a) parsing string literals, and (b) Object.toString() - and probably a few other things too, not to mention all the code it would break, and the future (or not) of the char type. It's all this that I'd be concerned about, and should really be discussed by all of us in the D community, and Walter as its architect - not just those of us interested in Unicode. Arcane Jill
Aug 23 2004
prev sibling next sibling parent reply =?ISO-8859-1?Q?J=F6rg_R=FCppel?= <joerg sharky-x.de> writes:
Arcane Jill wrote:
 
 So I'm thinking seriously about ditching the whole etc.unicode project and
 replacing it with a wrapper around ICU.
 
 It's not completely straightforward. ICU is written in C++ (not C), and so we
 can't link against it directly. It uses classes, not raw functions. So, I'd
have
 to write a C wrapper around ICU which gave me a C interface, and /then/ I'd
have
 to write a D wrapper to call the C wrapper - at which point we could get the
 classes back again (and our own choice of architecture, so plugging into std or
 mango streams won't suffer).

According to the API docs at http://oss.software.ibm.com/icu/apiref/index.html there is a C API. Didn't you see that or is there a reason why that can't be used? Regards, Jörg
Aug 23 2004
parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgd5go$2j1k$1 digitaldaemon.com>, =?ISO-8859-1?Q?J=F6rg_R=FCppel?=
says...

According to the API docs at 
http://oss.software.ibm.com/icu/apiref/index.html there is a C API. 
Didn't you see that or is there a reason why that can't be used?

I just didn't see it, that's all. I think now that the best approach would be something which is part-port and part-wrapper around the C API. I would want the D interface to maintain the classes and so forth which are present in the C++ and Java APIs, so the D API's C wrappers would have to be part-port anyway, even if only to recreate the class heirarchy and put things back into member functions. Jill
Aug 23 2004
prev sibling next sibling parent =?ISO-8859-1?Q?Julio_C=E9sar_Carrascal_Urquijo?= writes:
Most of ICU it's written in C with C++ wrappers.



Arcane Jill wrote:
 Following Ben's mention of ICU, I've been checking out what it does and doesn't
 do. Basically, it does EVERYTHING. There is the work of years there. Not just
 Unicode stuff, but a whole swathe of classes for internationalization and
 transcoding. It would take me a very long time to duplicate that. It's also
 free, open source, and with a license which basically says there's no problem
 with our using it.
 
 So I'm thinking seriously about ditching the whole etc.unicode project and
 replacing it with a wrapper around ICU.
 
 It's not completely straightforward. ICU is written in C++ (not C), and so we
 can't link against it directly. It uses classes, not raw functions. So, I'd
have
 to write a C wrapper around ICU which gave me a C interface, and /then/ I'd
have
 to write a D wrapper to call the C wrapper - at which point we could get the
 classes back again (and our own choice of architecture, so plugging into std or
 mango streams won't suffer).
 
 But the outermost (D) wrapper can, at least, be composed of D classes.
 
 If we want D to be the language of choice for Unicode, we would need all this
 functionality. So, if we went the ICU route, we'd need to bundle ICU (currently
 a ten megabyte zip file) with DMD, along with whatever wrapper I come up with.
 (etc.unicode is not likely to be smaller).
 
 I'd like to see some discussion on this. Read this page to inform yourself:
 http://oss.software.ibm.com/icu/userguide/index.html
 
 
 Finally, back to strings. Ben was right. The ICU says:
 
 "In order to take advantage of Unicode with its large character repertoire and
 its well-defined properties, there must be types with consistent definitions
and
 semantics. The Unicode standard defines a default encoding based on 16-bit code
 units. This is supported in ICU by the definition of the UChar to be an
unsigned
 16-bit integer type. This is the base type for character arrays for strings in
 ICU."
 
 Get that? A *16-BIT* type is the basis for ICU strings. ICU defines no special
 string class - a string is just an array of wchars. So obviously, if we go the
 ICU route, I withdraw my suggestion to ditch the wchar.
 
 What I now recommend is:
 
 (1) Ditch "etc.unicode" in favor of - let's call it "etc.icu" (a D wrapper
 Path: digitalmars.com!drn
 From: Arcane Jill <Arcane_member pathlink.com>
 Newsgroups: digitalmars.D
 Subject: Syntax for pinning
 Date: Sun, 22 Aug 2004 09:18:11 +0000 (UTC)
 Organization: [http://www.pathlink.com]
 Lines: 39
 Sender: usenet www.digitalmars.com
 Message-ID: <cg9ocj$9ce$1 digitaldaemon.com>
 X-Trace: digitaldaemon.com 1093166291 9614 63.105.9.61 (22 Aug 2004 09:18:11
GMT)
 X-Complaints-To: usenet digitalmars.com
 NNTP-Posting-Date: Sun, 22 Aug 2004 09:18:11 +0000 (UTC)
 X-Newsreader: Direct Read News v3.11a
 Xref: digitalmars.com digitalmars.D:9385
 
 In article <cfit27$snc$1 digitaldaemon.com>, Walter says...
 
 
I know how to build a gc that will compact
memory to deal with fragmentation when it does arise, so although this is
not in the current D it is not a technical problem to do it.

That's probably good news, but: Will it be possible to mark specific arrays on the heap as "immovable"?

(Please
say yes. I'm sure it is not a technical problem).

Yes. It's called "pinning". And I can guess why you need this feature <g>.

Walter, hi. Listen - while the prospect of a compacting GC may be a long way away, I'd like to be able to mark specific arrays on the heap as immovable right now, for forward compatibility, so that when the new GC comes along everything will still behave as intended. Could you perhaps reserve a keyword and/or some syntax for pinning? It only has around ICU). Eventually I hope for this to change into "std.icu" (as I originally hoped that "etc.unicode" would turn into "std.unicode"). (2) Ditch the char. 8-bits is really too small for a character these days, honestly, and all previous arguments still apply. The existence of char only encourages ASCII and discourages Unicode anyway. (3) Native D strings shall be arrays of wchars. This means that Object.toString() must return a wchar[], and string literals in D source must compile to wchar[]s. UCI's type UChar would map directly to wchar. To reinforce this, there should probably be an # alias wchar[] string; in object.d. (4) We retain dchar (so that we can get character properties), but all string code is based on wchar[]s, not dchar[]s. UCI's type UChar32 would map directly to dchar. (5) Transcoding/streams/etc. go ahead as planned, but based around wchar[]s instead of dchar[]s. (6) That's pretty much it, although once "char" is gone, we could rename "wchar" as "char" (a la Java). Discussion please? And I really do want this talked through because it affects D work I'm currently involved in. Input is also requested from Walter - in particular the request that Object.toString() be re-jigged to return wchar[] instead of char[]. Okay, let's chew this one over. Jill

Aug 23 2004
prev sibling next sibling parent "antiAlias" <fu bar.com> writes:
This would be a great thing for D to adopt. Just a few things to note:

1) This would be best served as a DLL (given it's size). In fact, the team
apparently like to compile the string-resource files into DLLs (which makes
a lot of sense IMO). If D treated DLLs as first-class citizens, this would
be a no-brainer. Right now, that's not the case.

2) There's a rather nice String class (C++). That's a perfect candidate for
porting directly to D.

3) From what I've seen, the lib is mostly C. Even better, it eschews the
traditional morass of header files. Building a ICU.d import will be much
easier because of this. The project on dsource.org might handle that part
without issue? Wrapping those library functions with D shells would be nice,
if only to take advantage of D arrays.

4) the transcoders deal with arrays of buffered data, so they're efficient.
ICU has transcoders and code-page tables up-the-wazoo.

For those who haven't looked through the lib, it's far more than just
Unicode transcoders (as Jill notes): you get sophisticated and flexible date
& number parsing/formatting; I18N message ordering; BiDi support;
collation/sorting support; text-break algorithms; a text layout engine;
unicode regex; and much more.

It's a first-class suite of libraries, and an awesome resource to leverage.



"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgcv4n$2fsf$1 digitaldaemon.com...
 Following Ben's mention of ICU, I've been checking out what it does and

 do. Basically, it does EVERYTHING. There is the work of years there. Not

 Unicode stuff, but a whole swathe of classes for internationalization and
 transcoding. It would take me a very long time to duplicate that. It's

 free, open source, and with a license which basically says there's no

 with our using it.

 So I'm thinking seriously about ditching the whole etc.unicode project and
 replacing it with a wrapper around ICU.

 It's not completely straightforward. ICU is written in C++ (not C), and so

 can't link against it directly. It uses classes, not raw functions. So,

 to write a C wrapper around ICU which gave me a C interface, and /then/

 to write a D wrapper to call the C wrapper - at which point we could get

 classes back again (and our own choice of architecture, so plugging into

 mango streams won't suffer).

 But the outermost (D) wrapper can, at least, be composed of D classes.

 If we want D to be the language of choice for Unicode, we would need all

 functionality. So, if we went the ICU route, we'd need to bundle ICU

 a ten megabyte zip file) with DMD, along with whatever wrapper I come up

 (etc.unicode is not likely to be smaller).

 I'd like to see some discussion on this. Read this page to inform

 http://oss.software.ibm.com/icu/userguide/index.html


 Finally, back to strings. Ben was right. The ICU says:

 "In order to take advantage of Unicode with its large character repertoire

 its well-defined properties, there must be types with consistent

 semantics. The Unicode standard defines a default encoding based on 16-bit

 units. This is supported in ICU by the definition of the UChar to be an

 16-bit integer type. This is the base type for character arrays for

 ICU."

 Get that? A *16-BIT* type is the basis for ICU strings. ICU defines no

 string class - a string is just an array of wchars. So obviously, if we go

 ICU route, I withdraw my suggestion to ditch the wchar.

 What I now recommend is:

 (1) Ditch "etc.unicode" in favor of - let's call it "etc.icu" (a D wrapper
 around ICU). Eventually I hope for this to change into "std.icu" (as I
 originally hoped that "etc.unicode" would turn into "std.unicode").

 (2) Ditch the char. 8-bits is really too small for a character these days,
 honestly, and all previous arguments still apply. The existence of char

 encourages ASCII and discourages Unicode anyway.

 (3) Native D strings shall be arrays of wchars. This means that
 Object.toString() must return a wchar[], and string literals in D source

 compile to wchar[]s. UCI's type UChar would map directly to wchar. To

 this, there should probably be an

 #    alias wchar[] string;

 in object.d.

 (4) We retain dchar (so that we can get character properties), but all

 code is based on wchar[]s, not dchar[]s. UCI's type UChar32 would map

 to dchar.

 (5) Transcoding/streams/etc. go ahead as planned, but based around

 instead of dchar[]s.

 (6) That's pretty much it, although once "char" is gone, we could rename

 as "char" (a la Java).


 Discussion please? And I really do want this talked through because it

 work I'm currently involved in.

 Input is also requested from Walter - in particular the request that
 Object.toString() be re-jigged to return wchar[] instead of char[].

 Okay, let's chew this one over.

 Jill

Aug 23 2004
prev sibling next sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgcv4n$2fsf$1 digitaldaemon.com...
 Get that? A *16-BIT* type is the basis for ICU strings. ICU defines no

 string class - a string is just an array of wchars. So obviously, if we go

 ICU route, I withdraw my suggestion to ditch the wchar.

Ok, but suppose we ditch char[]. Then, we find some great library we want to bring into D, or build a D interface too, that is in char[].
 Input is also requested from Walter - in particular the request that
 Object.toString() be re-jigged to return wchar[] instead of char[].

My experience with all-wchar is that its performance is not the best. It'll also become a nuisance interfacing with C. I'd rather explore perhaps making implict conversions between the 3 utf types more seamless.
Aug 23 2004
next sibling parent reply Regan Heath <regan netwin.co.nz> writes:
On Mon, 23 Aug 2004 14:06:57 -0700, Walter <newshound digitalmars.com> 
wrote:
 "Arcane Jill" <Arcane_member pathlink.com> wrote in message
 news:cgcv4n$2fsf$1 digitaldaemon.com...
 Get that? A *16-BIT* type is the basis for ICU strings. ICU defines no

 string class - a string is just an array of wchars. So obviously, if we 
 go

 ICU route, I withdraw my suggestion to ditch the wchar.

Ok, but suppose we ditch char[]. Then, we find some great library we want to bring into D, or build a D interface too, that is in char[].
 Input is also requested from Walter - in particular the request that
 Object.toString() be re-jigged to return wchar[] instead of char[].

My experience with all-wchar is that its performance is not the best. It'll also become a nuisance interfacing with C. I'd rather explore perhaps making implict conversions between the 3 utf types more seamless.

YAY! .. Sorry I can't help myself, I think _this_ is the way to go, see my post here: digitalmars.D/9494 for my arguments. I would add one additional argument. I can't imagine the suggested change breaking existing code.. more likely it will fix existing bugs. Regan. -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 23 2004
parent reply Juanjo =?ISO-8859-15?Q?=C1lvarez?= <juanjuxNO SPAMyahoo.es> writes:
Regan Heath wrote:

 Ok, but suppose we ditch char[]. Then, we find some great library we
 want to
 bring into D, or build a D interface too, that is in char[].


Excuse me if I'm saying something stupid but byte[] would not do the job of interfacing with C char[]?
Aug 23 2004
next sibling parent Regan Heath <regan netwin.co.nz> writes:
On Tue, 24 Aug 2004 01:18:09 +0200, Juanjo Álvarez 
<juanjuxNO SPAMyahoo.es> wrote:
 Regan Heath wrote:

I didn't write this! :)
 Ok, but suppose we ditch char[]. Then, we find some great library we
 want to
 bring into D, or build a D interface too, that is in char[].


Excuse me if I'm saying something stupid but byte[] would not do the job of interfacing with C char[]?

I almost made that same comment in reply to 'Walter' (who made the comment above) I think you're right, you could use byte[], in fact it'd be more correct to use byte[] as the C 'char' type is a byte with no specified encoding (whereas D's char[] is utf-8 encoded). If we had no char[] you'd have to transcode the byte[] to dchar[], this is why I disagree with removing char[], I think char[] has a place in D, I just want to see implicit transcoding of char[] to wchar[] to dchar[]. Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 23 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Juanjo Álvarez" <juanjuxNO SPAMyahoo.es> wrote in message
news:cgdu4b$2ed$1 digitaldaemon.com...
 Regan Heath wrote:

 Ok, but suppose we ditch char[]. Then, we find some great library we
 want to
 bring into D, or build a D interface too, that is in char[].


Excuse me if I'm saying something stupid but byte[] would not do the job

 interfacing with C char[]?

That could work, but it just wouldn't look right.
Aug 23 2004
parent reply Regan Heath <regan netwin.co.nz> writes:
On Mon, 23 Aug 2004 16:41:10 -0700, Walter <newshound digitalmars.com> 
wrote:
 "Juanjo Álvarez" <juanjuxNO SPAMyahoo.es> wrote in message
 news:cgdu4b$2ed$1 digitaldaemon.com...
 Regan Heath wrote:

 Ok, but suppose we ditch char[]. Then, we find some great library we
 want to
 bring into D, or build a D interface too, that is in char[].


Excuse me if I'm saying something stupid but byte[] would not do the job

 interfacing with C char[]?

That could work, but it just wouldn't look right.

http://www.digitalmars.com/d/htomodule.html Specifically states that C's 'char' should be represented by a 'byte' in D. So when building an interface to the C lib that uses char[] you'd use byte[]. Regan. -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 23 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"Regan Heath" <regan netwin.co.nz> wrote in message
news:opsc7lqdnp5a2sq9 digitalmars.com...
 http://www.digitalmars.com/d/htomodule.html

 Specifically states that C's 'char' should be represented by a 'byte' in

 So when building an interface to the C lib that uses char[] you'd use
 byte[].

I'm sorry that wasn't clear, but I meant that when 'unsigned char' and 'signed char' in C are used not as text, but as very small integers, the corresponding D types should be ubyte and byte.
Aug 23 2004
parent reply Regan Heath <regan netwin.co.nz> writes:
On Mon, 23 Aug 2004 23:35:07 -0700, Walter <newshound digitalmars.com> 
wrote:
 "Regan Heath" <regan netwin.co.nz> wrote in message
 news:opsc7lqdnp5a2sq9 digitalmars.com...
 http://www.digitalmars.com/d/htomodule.html

 Specifically states that C's 'char' should be represented by a 'byte' in

 So when building an interface to the C lib that uses char[] you'd use
 byte[].

I'm sorry that wasn't clear, but I meant that when 'unsigned char' and 'signed char' in C are used not as text, but as very small integers, the corresponding D types should be ubyte and byte.

However, an old C lib might return latin-1 (or any other encoding) encoded data, in which case you also have to use ubyte then transcode to utf-8 and store in char[] (if that is the desired result). Right? Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
parent "Walter" <newshound digitalmars.com> writes:
"Regan Heath" <regan netwin.co.nz> wrote in message
news:opsc9o90n65a2sq9 digitalmars.com...
 On Mon, 23 Aug 2004 23:35:07 -0700, Walter <newshound digitalmars.com>
 wrote:
 "Regan Heath" <regan netwin.co.nz> wrote in message
 news:opsc7lqdnp5a2sq9 digitalmars.com...
 http://www.digitalmars.com/d/htomodule.html

 Specifically states that C's 'char' should be represented by a 'byte'



 D.
 So when building an interface to the C lib that uses char[] you'd use
 byte[].

I'm sorry that wasn't clear, but I meant that when 'unsigned char' and 'signed char' in C are used not as text, but as very small integers, the corresponding D types should be ubyte and byte.

However, an old C lib might return latin-1 (or any other encoding) encoded data, in which case you also have to use ubyte then transcode to utf-8 and store in char[] (if that is the desired result). Right?

Yup. You'll have to understand what the C code is using the char type for, in order to select the best equivalent D type.
Aug 25 2004
prev sibling next sibling parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgdmj9$2v1a$1 digitaldaemon.com>, Walter says...

My experience with all-wchar is that its performance is not the best. It'll
also become a nuisance interfacing with C. I'd rather explore perhaps making
implict conversions between the 3 utf types more seamless.

Implicit conversions are absolutely fine by me. In fact, that was the first suggestion (then the discussion wandered, as things do, along the lines of "if they're interchangable, why not have just the one type". But sure, I'd be more than happy with implicit conversions between the three UTF types. Since such conversions lose no information in any direction, they are always guaranteed to be harmless. Jill
Aug 24 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgdmj9$2v1a$1 digitaldaemon.com>, Walter says...

My experience with all-wchar is that its performance is not the best.

It's usually regarded as the best by most other sources, however. Converting between wchar[] and dchar[] is /almost/ as fast as doing a memcpy(), because UTF-16 encoding is very, very simple. UTF-16 is as efficient for the codepoint range U+0000 to U+FFFF as UTF-8 is for the codepoint range U+0000 to U+007F. Outside of these ranges, UTF-16 is still very fast, since all remaining characters consist of /precisely/ two wchars, wheras UTF-8 conversion outside of the ASCII range is always going to be inefficient, what with it's variable number of required bytes, variable width bitmasks, and the additional requirements of validation and rejection of non-shortest sequences. If you're arguing on the basis performance, UTF-8 loses hands down.
It'll
also become a nuisance interfacing with C.

Actually, it's D's char (which C doesn't have) which is a nuisance interfacing with C. As others have pointed out, C has no type which enforces UTF-8 encoding, and in fact on the Windows PC on which I am typing right now, every C char is going to be storing characters from Windows code page 1252 unless I take special action to do otherwise. That is /not/ interchangable with D's chars. Beyond U+007F, one C char corresponds to two (or more) D chars. You don't regard that as a nuisance? In fact, I believe that comment of yours which I just quoted above, actually adds further weight to my argument. I argue that the existence of the char type *causes confusion*. People /think/ (erroneously) that it does the same job as C's char, and is interchangable therewith. Now if you, the architect of D, can fall prey to that confusion, I would take that as clear evidence that such confusion exists.
I'd rather explore perhaps making
implict conversions between the 3 utf types more seamless.

Yes. If all three D string types were implicitly convertable, then there would be nothing for me to complain about. (The char confusion would still exist, but that's just education) Arcane Jill
Aug 24 2004
next sibling parent reply =?ISO-8859-1?Q?Julio_C=E9sar_Carrascal_Urquijo?= writes:
Arcane Jill wrote:
 (...)
 adds further weight to my argument. I argue that the existence of the char type
 *causes confusion*. People /think/ (erroneously) that it does the same job as
 C's char, and is interchangable therewith. Now if you, the architect of D, can
 fall prey to that confusion, I would take that as clear evidence that such
 confusion exists.
 (...)

I'm sorry to interrupt. From your reply I would say that you are arguing more about the name of the type been used as the *representation of a character* (which is not). Maybe we should ask Walter to change the names of the types to utf8, utf16 and utf32. I read somewhere that those where the original names in earlier DMD implementations. If that's what you are asking, you have my vote. -- Julio César Carrascal Urquijo
Aug 24 2004
parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgfmdm$v3t$1 digitaldaemon.com>, 

Maybe we should ask Walter to change the names of the types to utf8, 
utf16 and utf32. I read somewhere that those where the original names in 
earlier DMD implementations.

If that's what you are asking, you have my vote.

No, I wasn't asking that, and I don't really care what things are called. I do /try/ to stay on the topic of the thread title, and in this thread the discussion is about how/whether to make use of ICU for our internationalization and Unicode needs. I've been looking more closely at ICU, and I keep being (pleasantly) surprised to discover that it already has zillions of other goodies not previously mentioned in this thread, but which we've been talking about in this forum in the past. For example - Locales, ResourceBundles, everything you need for text internationalization/localization. It's relevant to D's character types because ICU has only two character types - a type equivalent to wchar that is used to make UTF-16 strings, and a type equivalent to dchar that is used to access character properties. The important detail here is that ICU strings are wchar[]s, but D's basic "string" concept is char[]. So, calling lots of ICU routines would result in lots of explicit toUTF8() and toUTF16() calls all over your code, /unless/ either: (1) D adopted wchar[] as the basic string type, or (2) D implicitly auto-converted between its various string types as required I'm trying to suggest that (1) is the best option. That's all. Walter prefers (2), but that's acceptable too. As a corrollary to (1), if we start using wchar[]s as the default native string used by Phobos and the compiler, it would then follow that the char type would be superfluous and could be dropped. Or at least, it seems that way to me. Opinions differ. However, the "should we ditch the char or not?" discussion is over on another thread. Renaming the character types is kind of irrelevant to this, although it is pertainant to Walter's reply. No - I'm not asking that they be renamed (except insofar as, if "char" is ditched, then "wchar" could be renamed "char", but that again is for the other thread). Arcane Jill
Aug 24 2004
prev sibling parent "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgf8d1$os0$1 digitaldaemon.com...
 In article <cgdmj9$2v1a$1 digitaldaemon.com>, Walter says...

My experience with all-wchar is that its performance is not the best.

It's usually regarded as the best by most other sources, however.

 between wchar[] and dchar[] is /almost/ as fast as doing a memcpy(),

 UTF-16 encoding is very, very simple. UTF-16 is as efficient for the

 range U+0000 to U+FFFF as UTF-8 is for the codepoint range U+0000 to

 Outside of these ranges, UTF-16 is still very fast, since all remaining
 characters consist of /precisely/ two wchars, wheras UTF-8 conversion

 the ASCII range is always going to be inefficient, what with it's variable
 number of required bytes, variable width bitmasks, and the additional
 requirements of validation and rejection of non-shortest sequences. If

 arguing on the basis performance, UTF-8 loses hands down.

Converting would be faster, sure, but if the bulk of your app is char[], there is little conversion happening.
It'll
also become a nuisance interfacing with C.


 with C. As others have pointed out, C has no type which enforces UTF-8

 and in fact on the Windows PC on which I am typing right now, every C char

 going to be storing characters from Windows code page 1252 unless I take

 action to do otherwise. That is /not/ interchangable with D's chars.

 U+007F, one C char corresponds to two (or more) D chars. You don't regard

 as a nuisance?

I've been dealing with multibyte charsets in C for decades - it's not just UTF-8 that's multibyte, there are also the Shift-JIS, Korean, and Taiwan code pages. You can also set up your windows machine so UTF-8 *is* the charset used by the "A" APIs. I've written UTF-8 apps in C, and D would map onto them directly. There is no way to avoid, when interfacing with C, dealing with whatever charset it might be in. It can't happen automatically. And that is the source of the nuisance.
 In fact, I believe that comment of yours which I just quoted above,

 adds further weight to my argument. I argue that the existence of the char

 *causes confusion*. People /think/ (erroneously) that it does the same job

 C's char, and is interchangable therewith. Now if you, the architect of D,

 fall prey to that confusion, I would take that as clear evidence that such
 confusion exists.

I'd rather explore perhaps making
implict conversions between the 3 utf types more seamless.

Yes. If all three D string types were implicitly convertable, then there

 be nothing for me to complain about. (The char confusion would still

 that's just education)

Aug 24 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgcv4n$2fsf$1 digitaldaemon.com...
 Following Ben's mention of ICU, I've been checking out what it does and

 do. Basically, it does EVERYTHING. There is the work of years there. Not

 Unicode stuff, but a whole swathe of classes for internationalization and
 transcoding. It would take me a very long time to duplicate that. It's

 free, open source, and with a license which basically says there's no

 with our using it.

 So I'm thinking seriously about ditching the whole etc.unicode project and
 replacing it with a wrapper around ICU.

 It's not completely straightforward. ICU is written in C++ (not C), and so

 can't link against it directly. It uses classes, not raw functions. So,

 to write a C wrapper around ICU which gave me a C interface, and /then/

 to write a D wrapper to call the C wrapper - at which point we could get

 classes back again (and our own choice of architecture, so plugging into

 mango streams won't suffer).

 But the outermost (D) wrapper can, at least, be composed of D classes.

 If we want D to be the language of choice for Unicode, we would need all

 functionality. So, if we went the ICU route, we'd need to bundle ICU

 a ten megabyte zip file) with DMD, along with whatever wrapper I come up

 (etc.unicode is not likely to be smaller).

 I'd like to see some discussion on this. Read this page to inform

 http://oss.software.ibm.com/icu/userguide/index.html

It sounds pretty cool. It being a very large library, is only what you use linked in? Or does using anything tend to pull in the whole shebang? I hope it's the former!
Aug 25 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cghofr$202k$3 digitaldaemon.com>, Walter says...

It sounds pretty cool. It being a very large library, is only what you use
linked in? Or does using anything tend to pull in the whole shebang? I hope
it's the former!

No idea (yet). I worried about that myself, but it's a library not an indivisibile object file, so it must have /some/ granularity. But the functionality of ICU goes way beyond what I was even planning to achieve - there's even stuff for font rendering in there for feck's sake! (Unicode allows you to put any accent over any glyph, ligate any two glyphs into one, etc. Font rendering engines have a fair bit of work to do). So ICU is hard to beat. Therefore, I'm inclining to the view that we'd be better off trying to plumb ICU into D than to reject it because it fails to meet any one single D requirement. So, if it won't split into sufficiently small chunks on linking, I'd say that was as good an argument as any for improving D's DLL support. ICU would clearly be an excellent candidate for a DLL (assuming we can have classes in DLLs). Jill
Aug 25 2004
parent reply "Roald Ribe" <rr.no spam.teikom.no> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cghrn3$21dg$1 digitaldaemon.com...
 In article <cghofr$202k$3 digitaldaemon.com>, Walter says...

It sounds pretty cool. It being a very large library, is only what you


linked in? Or does using anything tend to pull in the whole shebang? I


it's the former!

No idea (yet). I worried about that myself, but it's a library not an indivisibile object file, so it must have /some/ granularity. But the functionality of ICU goes way beyond what I was even planning to

 - there's even stuff for font rendering in there for feck's sake! (Unicode
 allows you to put any accent over any glyph, ligate any two glyphs into

 etc. Font rendering engines have a fair bit of work to do). So ICU is hard

 beat. Therefore, I'm inclining to the view that we'd be better off trying

 plumb ICU into D than to reject it because it fails to meet any one single

 requirement.

 So, if it won't split into sufficiently small chunks on linking, I'd say

 was as good an argument as any for improving D's DLL support. ICU would

 be an excellent candidate for a DLL (assuming we can have classes in

The ICU has a C API, you want to port the C++ API to D, why not compile a DLL with the C API, and a lib with the ported D API? This would also be good for minimum disturbance between the ICU source and the D API. Roald
Aug 25 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgi7f6$25qf$1 digitaldaemon.com>, Roald Ribe says...

The ICU has a C API, you want to port the C++ API to D, why not
compile a DLL with the C API, and a lib with the ported D API?
This would also be good for minimum disturbance between the ICU source
and the D API.

This is possibly the best idea yet, but I'm going to have to study the ICU source before I know how feasible that is. Some fundamentals: (1) ICU's C API requires you to acquire and release (memory) resources. D programmers are accustomed to letting the garbage collector do the releasing. (2) ICU's C++ API requires classes to have a copy constructor and an assignment operator. This isn't possible with D. The above two points mean that the D API is probably going to look more like the Java API than the C++ API. Exactly how much of it can be done via a wrapper around C I'm not sure (yet). But yes - if the C-based core can be put into a DLL, and a D-API built which calls it, I guess that would work a treat. Arcane Jill
Aug 25 2004
parent reply Sean Kelly <sean f4.ca> writes:
In article <cgi8jj$26dr$1 digitaldaemon.com>, Arcane Jill says...
Some fundamentals:
(1) ICU's C API requires you to acquire and release (memory) resources. D
programmers are accustomed to letting the garbage collector do the releasing.
(2) ICU's C++ API requires classes to have a copy constructor and an assignment
operator. This isn't possible with D.

What are those two operators used for? Could the functionality be worked around somehow? Sean
Aug 25 2004
next sibling parent pragma <pragma_member pathlink.com> writes:
In article <cgifct$29kf$1 digitaldaemon.com>, Sean Kelly says...
In article <cgi8jj$26dr$1 digitaldaemon.com>, Arcane Jill says...
Some fundamentals:
(1) ICU's C API requires you to acquire and release (memory) resources. D
programmers are accustomed to letting the garbage collector do the releasing.
(2) ICU's C++ API requires classes to have a copy constructor and an assignment
operator. This isn't possible with D.

What are those two operators used for? Could the functionality be worked around somehow?

I'm sure it can be worked around, as there is a Java interface to ICU already available. ( http://oss.software.ibm.com/icu4j/download/ ) ;) This table ( http://oss.software.ibm.com/icu4j/comparison/index.html ) is about as close an indicator as we're going to get for what D would need to do to compete with what's already out there. -Pragma [[ EricAnderton at (code it, and they will come) yahoo.com ]]
Aug 25 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgifct$29kf$1 digitaldaemon.com>, Sean Kelly says...
In article <cgi8jj$26dr$1 digitaldaemon.com>, Arcane Jill says...
Some fundamentals:
(1) ICU's C API requires you to acquire and release (memory) resources. D
programmers are accustomed to letting the garbage collector do the releasing.
(2) ICU's C++ API requires classes to have a copy constructor and an assignment
operator. This isn't possible with D.

What are those two operators used for?

I don't know. I'm still looking into it all. Fortunately, I don't think it matters.
Could the functionality be worked around somehow?

Obviously yes, since there is a Java API.
Aug 25 2004
parent "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgihb5$2aub$1 digitaldaemon.com...
 In article <cgifct$29kf$1 digitaldaemon.com>, Sean Kelly says...
In article <cgi8jj$26dr$1 digitaldaemon.com>, Arcane Jill says...
Some fundamentals:
(1) ICU's C API requires you to acquire and release (memory) resources.



programmers are accustomed to letting the garbage collector do the



(2) ICU's C++ API requires classes to have a copy constructor and an



operator. This isn't possible with D.

What are those two operators used for?

I don't know. I'm still looking into it all. Fortunately, I don't think it matters.
Could the functionality be worked around somehow?

Obviously yes, since there is a Java API.

The Java API would likely be the best starting point. IBM is no slouch in its support of Java, and the Java interface will have solved the issues with interfacing to a GC language.
Aug 25 2004