www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The case for ditching char and wchar (and renaming "dchar" as "char")

reply Arcane Jill <Arcane_member pathlink.com> writes:
D has come a long way, and much of the original architecture is now redundant.
Template/library based containers are now making built-in associative arrays
redundant, for example. And now a new revolution is on its way - transcoding,
which makes built in support for UTF-8 and friends equally redundant. (It does
not, of course, make Unicode itself redundant!).

D's "char" type is, by definition, a fragment of UTF-8.
But UTF-8 is just an encoding.

D's "wchar" type is, by definition, a fragment of UTF-16.
But UTF-16 is also just an encoding (or two).

D's "dchar" type flits ambiguously between a fragment of UTF-32 and an actual
Unicode codepoint (the two are more or less interchangeable).

<sarcasm>
By extension of this logic, why not:

schar - a fragment of UTF-7
ichar - a fragment of ISO-8859-1
cchar - a fragment of WINDOWS-1252
.. and so on, for every encoding you can think of. Hang on - we're going to run
out of letters!

and of course, Phobos would have to implement all the conversion functions:
toUTF7(), toISO88591(), and so on.
</sarcasm>

Nonsense? Of course it is. But the analogy is intended to show that the current
behavior of D is also nonsense. For N encodings, you need (N squared minus N)
conversion functions, so the number is going to grow quite rapidly as the number
of supported encodings increases. But if you instead use transcoding, then the
number of conversion functions you need is simply N. Not only that, the
mechanism is smoother, neater. Your code is more elegant. You simply don't have
to /worry/ about all that nonsense trying to get the three built-in encodings to
match, because the issue has simply gone away.

And once the issue has gone away, you no longer need a special type to hold
fragments of UTF-8 or UTF-16. Bye bye char. Bye bye wchar.

Kris (antiAlias) has sent me the the transcoding interface which Mango requires.
(Recieved, thanks). I've already written a generic one, but which didn't take
those requirements into account. So today I'm going to merge the two approaches
together and see what Kris thinks. So I'm pretty confident that within a few
days, Kris and I will have got together a transcoding architecture we're both
happy with - and since Kris has expertise in streams/Mango, and I have expertise
in Unicode/internationalization, I'd make a pretty good wager that between us
we're going to get it right. And we'll plumb in the UTF transcoders first. You
can probably expect all that to be done within days rather than weeks.

So why would we then need old-style-char or wchar any more?

For reasons of space-efficiency, one might want to store text in memory in UTF-8
format. Fair enough. But if char were to be ditched, you could still do that.
You'd simply use a ubyte[] for that purpose (just as you are now required to do
if you want to store text in memory in UTF-7). After all - what actually /is/ a
UTF fragment anyway? What meaning does the UTF-8 fragment 0x83 have in
isolation? Answer - none. It has meaning only in the context of the bytes
surrounding it. You don't need a special primitive type just to hold that
fragment. And of course, there is /nothing/ to stop special string classes from
being written to provide implementations of such space-efficient abstractions.

A further argument against char is people coming from C/C++ /will/ try to store
ISO-8859-1 encoded strings in a char[]. And they will get away with it, too, so
long as they don't try calling any toUTFxx() routines on them. Bugs can be
deeply buried in this confusion, failing to surface for a very long time.

Discussion in another thread has focused on the the fact that Object.toString()
returns a char[]. Regan and I have made the suggestion that the three string
types be interchangable. But there's a better way: have just the /one/ string
type. (As they say in Highlander, "There can be only one"). Problem gone away.

With new-style-char redefined, not merely as a UTF-32 code unit (a fragment of
an encoding), but as an actual Unicode character, things become much, much
simpler.

AND it would make life easier for Walter - fewer primitive types; less for the
compiler to understand/do.

Java tried to do it this way. When Java was invented, they had a char type
intended to hold a single Unicode character. They also had a byte type, an array
of which could store ASCII or ISO-8859-1 or UTF-8 encoded text. They also had
transcoding built in, to make it all hang together. Where it went wrong for Java
was that Unicode changed from being 16-bits wide to being 21-bits wide (so
suddenly Java's char was no longer wide enough, and they were forced to redefine
Java strings as being UTF-16 encoded). But please note that Java did /not/
attempt to have separate char types for each encoding. Even /after/ Unicode
exceeded 16-bits, Java was not tempted to introduce a new kind of char. Why not?
Because having more than one char type is an ugly kludge (particularly if you're
using Unicode by definition). It's an ugly kludge in D, too. I thought it was
really good, once upon a time, but now that transcoding is moving out to
libraries, and encompasses many /more/ encodings merely UTF-8/16/32, I no longer
think that. Now is the best time of all for a rethink.

But ...

there's a down-side ... it would break a lot of existing code. Well, so what?
This is a pre-1.0 wart-removing exercise. Like all of those other suggestions
we're voting on in another thread, the time to make this change is now, before
it's too late.

Arcane Jill
Aug 23 2004
next sibling parent Matthias Becker <Matthias_member pathlink.com> writes:
After you proposed these ideas about allowing toString to return any character
type I started thinking about it and finaly I thought: Why do we have more than
one character-type? (Just like you do)
Aug 23 2004
prev sibling next sibling parent Juanjo =?ISO-8859-15?Q?=C1lvarez?= <juanjuxNO SPAMyahoo.es> writes:
Arcane Jill wrote:
 
 ButY there's a better way:
 have just the /one/ string type. (As they say in Highlander, "There can be
 only one"). Problem gone away.

Yes, it makes a lot of sense. You have my (useless) vote.
 AND it would make life easier for Walter - fewer primitive types; less for
 the compiler to understand/do.

I think Walter should like it, if only for this.
 But ...
 
 there's a down-side ... it would break a lot of existing code. Well, so
 what? This is a pre-1.0 wart-removing exercise. Like all of those other
 suggestions we're voting on in another thread, the time to make this
 change is now, before it's too late.

I'll be very happy to change my (little) code now.
Aug 23 2004
prev sibling next sibling parent reply Ben Hinkle <bhinkle4 juno.com> writes:
Arcane Jill wrote:

 
 D has come a long way, and much of the original architecture is now
 redundant. Template/library based containers are now making built-in
 associative arrays redundant, for example. And now a new revolution is on
 its way - transcoding, which makes built in support for UTF-8 and friends
 equally redundant. (It does not, of course, make Unicode itself
 redundant!).
 
 D's "char" type is, by definition, a fragment of UTF-8.
 But UTF-8 is just an encoding.
 
 D's "wchar" type is, by definition, a fragment of UTF-16.
 But UTF-16 is also just an encoding (or two).
 
 D's "dchar" type flits ambiguously between a fragment of UTF-32 and an
 actual Unicode codepoint (the two are more or less interchangeable).
 
 <sarcasm>
 By extension of this logic, why not:
 
 schar - a fragment of UTF-7
 ichar - a fragment of ISO-8859-1
 cchar - a fragment of WINDOWS-1252
 .. and so on, for every encoding you can think of. Hang on - we're going
 to run out of letters!
 
 and of course, Phobos would have to implement all the conversion
 functions: toUTF7(), toISO88591(), and so on.
 </sarcasm>
 
 Nonsense? Of course it is. But the analogy is intended to show that the
 current behavior of D is also nonsense. For N encodings, you need (N
 squared minus N) conversion functions, so the number is going to grow
 quite rapidly as the number of supported encodings increases. But if you
 instead use transcoding, then the number of conversion functions you need
 is simply N. Not only that, the mechanism is smoother, neater. Your code
 is more elegant. You simply don't have to /worry/ about all that nonsense
 trying to get the three built-in encodings to match, because the issue has
 simply gone away.
 
 And once the issue has gone away, you no longer need a special type to
 hold fragments of UTF-8 or UTF-16. Bye bye char. Bye bye wchar.
 
 Kris (antiAlias) has sent me the the transcoding interface which Mango
 requires. (Recieved, thanks). I've already written a generic one, but
 which didn't take those requirements into account. So today I'm going to
 merge the two approaches together and see what Kris thinks. So I'm pretty
 confident that within a few days, Kris and I will have got together a
 transcoding architecture we're both happy with - and since Kris has
 expertise in streams/Mango, and I have expertise in
 Unicode/internationalization, I'd make a pretty good wager that between us
 we're going to get it right. And we'll plumb in the UTF transcoders first.
 You can probably expect all that to be done within days rather than weeks.
 
 So why would we then need old-style-char or wchar any more?
 
 For reasons of space-efficiency, one might want to store text in memory in
 UTF-8 format. Fair enough. But if char were to be ditched, you could still
 do that. You'd simply use a ubyte[] for that purpose (just as you are now
 required to do if you want to store text in memory in UTF-7). After all -
 what actually /is/ a UTF fragment anyway? What meaning does the UTF-8
 fragment 0x83 have in isolation? Answer - none. It has meaning only in the
 context of the bytes surrounding it. You don't need a special primitive
 type just to hold that fragment. And of course, there is /nothing/ to stop
 special string classes from being written to provide implementations of
 such space-efficient abstractions.
 
 A further argument against char is people coming from C/C++ /will/ try to
 store ISO-8859-1 encoded strings in a char[]. And they will get away with
 it, too, so long as they don't try calling any toUTFxx() routines on them.
 Bugs can be deeply buried in this confusion, failing to surface for a very
 long time.
 
 Discussion in another thread has focused on the the fact that
 Object.toString() returns a char[]. Regan and I have made the suggestion
 that the three string types be interchangable. But there's a better way:
 have just the /one/ string type. (As they say in Highlander, "There can be
 only one"). Problem gone away.
 
 With new-style-char redefined, not merely as a UTF-32 code unit (a
 fragment of an encoding), but as an actual Unicode character, things
 become much, much simpler.
 
 AND it would make life easier for Walter - fewer primitive types; less for
 the compiler to understand/do.
 
 Java tried to do it this way. When Java was invented, they had a char type
 intended to hold a single Unicode character. They also had a byte type, an
 array of which could store ASCII or ISO-8859-1 or UTF-8 encoded text. They
 also had transcoding built in, to make it all hang together. Where it went
 wrong for Java was that Unicode changed from being 16-bits wide to being
 21-bits wide (so suddenly Java's char was no longer wide enough, and they
 were forced to redefine Java strings as being UTF-16 encoded). But please
 note that Java did /not/ attempt to have separate char types for each
 encoding. Even /after/ Unicode exceeded 16-bits, Java was not tempted to
 introduce a new kind of char. Why not? Because having more than one char
 type is an ugly kludge (particularly if you're using Unicode by
 definition). It's an ugly kludge in D, too. I thought it was really good,
 once upon a time, but now that transcoding is moving out to libraries, and
 encompasses many /more/ encodings merely UTF-8/16/32, I no longer think
 that. Now is the best time of all for a rethink.
 
 But ...
 
 there's a down-side ... it would break a lot of existing code. Well, so
 what? This is a pre-1.0 wart-removing exercise. Like all of those other
 suggestions we're voting on in another thread, the time to make this
 change is now, before it's too late.
 
 Arcane Jill

There were huge threads about char vs wchar vs dchar a while ago (on the old newsgroup, I think). All kinds of things like what the default should be, what the names should be, what a string class could be etc. For example http://www.digitalmars.com/d/archives/20361.html http://www.digitalmars.com/d/archives/12382.html or actually anything at http://www.digitalmars.com/d/archives/index.html with the word "unicode" in the subject. By the way, why if there are N encodings are there N^2-N converters? Shouldn't there just be ~2*N to convert to/from one standard like dchar[]? IBM's ICU (at http://oss.software.ibm.com/icu/) uses wchar[] as the standard. -Ben
Aug 23 2004
next sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgcoe6$2cq4$1 digitaldaemon.com>, Ben Hinkle says...

There were huge threads about char vs wchar vs dchar a while ago (on the old
newsgroup, I think). All kinds of things like what the default should be,
what the names should be, what a string class could be etc. For example
 http://www.digitalmars.com/d/archives/20361.html
 http://www.digitalmars.com/d/archives/12382.html
or actually anything at
 http://www.digitalmars.com/d/archives/index.html
with the word "unicode" in the subject.

Well spotted. I had a look at some of those old threads, and it does seem that most of the views back there were saying much the same thing as I'm suggesting now, which is good, as I'm happy to count it as more votes for the proposal, AND evidence of ongoing discontent over some years. The difference between now and then is that /now/ we have transcoding classes underway, and we'll have a working architecture very very soon, which will be able to plug into any kind of string or stream class. This is the difference which makes ditching char and wchar an actual practical possibility now. Incidently, there were plenty of views in those archives which basically said that the Unicode functions which now exist in etc.unicode (and which didn't exist at the time) should exist. That's one problem solved.
By the way, why if there are N encodings are there N^2-N converters?
Shouldn't there just be ~2*N to convert to/from one standard like dchar[]?

Well, that's how transcoding will do it, obviously. I was comparing it to the present system, in which N == 3 (UTF-8, UTF-16 and UTF-32), and there are 6 (= 3^2-3) converters in std.utf, these being: *) toUTF8(wchar[]); *) toUTF8(dchar[]); *) toUTF16(char[]); *) toUTF16(dchar[]); *) toUTF32(char[]); *) toUTF32(wchar[]); If the current (std.utf) scheme were to be extended to include, say, UTF-7 and UTF-EBCDIC, how would that scale up?
IBM's ICU (at http://oss.software.ibm.com/icu/)

Bloody hell. I wish someone had pointed me at ICU earlier. That is exceptional. They've even got Unicode Regular Expressions! And transcoding functions. And it's open source, too! Should I just give up on etc.unicode? Maybe we should just put a D wrapper around ICU instead, which would give D full Unicode support right now, and leave me free to do crypto stuff!
IBM's ICU (at http://oss.software.ibm.com/icu/)
uses wchar[] as the standard.

Ah, no it doesn't. I just checked. ICU has the types UChar (platform dependent, but wchar for us) and UChar32 (definitely a dchar). So you see, both wchar[] and dchar[] are "standards" for ICU. (That said, I've only looked at it for a few seconds, so I may have misunderstood). Anwyay, UTF-16 transcoding will easily take care of interfacing with any UTF-16 architecture. The present situation in D is no more compatible than what I'm suggesting. Slightly modified proposal then - ditch char and wchar as before, PLUS, incorporate ICU into D's core and write a D wrapper for it. (And ditch etc.unicode - erk!) The ICU license is at http://oss.software.ibm.com/cvs/icu/~checkout~/icu/license.html. Arcane Jill
Aug 23 2004
parent Juanjo =?ISO-8859-15?Q?=C1lvarez?= <juanjuxNO SPAMyahoo.es> writes:
Arcane Jill wrote:


 Slightly modified proposal then - ditch char and wchar as before, PLUS,
 incorporate ICU into D's core and write a D wrapper for it. (And ditch
 etc.unicode - erk!) The ICU license is at
 http://oss.software.ibm.com/cvs/icu/~checkout~/icu/license.html.

I suppose that if dchar it's what left of this ditching it should renamed to char. AJ, I don't know a shit^D^D^D^D too much about Unicode but you excitement about ICU is really contagious, only one question, are the C wrappers at the same level than the C++/Java ones? If so it seems that with a little easy and boring (compared to writing etc.unicode) wrapping we're going to have a first-class Unicode lib :) => (i18n version of <g>)
Aug 23 2004
prev sibling next sibling parent reply "Roald Ribe" <rr.no spam.teikom.no> writes:
"Ben Hinkle" <bhinkle4 juno.com> wrote in message
news:cgcoe6$2cq4$1 digitaldaemon.com...

[snip]

 There were huge threads about char vs wchar vs dchar a while ago (on the

 newsgroup, I think). All kinds of things like what the default should be,
 what the names should be, what a string class could be etc. For example
  http://www.digitalmars.com/d/archives/20361.html
  http://www.digitalmars.com/d/archives/12382.html
 or actually anything at
  http://www.digitalmars.com/d/archives/index.html
 with the word "unicode" in the subject.

 By the way, why if there are N encodings are there N^2-N converters?
 Shouldn't there just be ~2*N to convert to/from one standard like dchar[]?
 IBM's ICU (at http://oss.software.ibm.com/icu/) uses wchar[] as the
 standard.

Indeed. There were several large discussions about this. Only a few scandinavian/north european readers of this group seemed to be positive at the time. I am happy to see that more people are warming to the idea. wchar (16-bit) is enough. It is even suggested as the best implementation size by some UNICODE coding experts. IBM / Sun / MS can not all be stupid at the same time... I think it would be smart to interoperate with the 16-bit size used internally in ICU, Java and MS-Windows. Only on unix/linux would it make sense to use 32-bits dchar. The 16 bits is enough for 99% of the cases/languages. The last 1% can be handled quite fast by cached indexing techniques in a String object. (this does not make for optimal speed in the 1% case, but it will more than pay for itself speedwise in 99% of all binary i/o operations :) However, that is Walters main issue I think. He wants default 8-bit chars to be default because this will make for the best possible i/o speed with the current state of affairs. That is what I understould from the last discussion at least. I am sure he will comment this thread ;-) and correct me if I am wrong. Regards, Roald
Aug 23 2004
next sibling parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgct0l$2eut$1 digitaldaemon.com>, Roald Ribe says...

However, that is Walters main issue I think. He wants default 8-bit
chars to be default because this will make for the best possible i/o
speed with the current state of affairs. That is what I understould
from the last discussion at least. I am sure he will comment this
thread ;-) and correct me if I am wrong.

Is that true? But UTF-8 /doesn't/ make the best possible I/O speed. To achieve that, you'd need to be using the OS-native encoding internally (ISO-8859-1 on most Linux boxes, WINDOWS-1252 on most Windows boxes). If UTF-8 is not used natively (which most of the time it isn't), you'd still need transcoding. Fact is, transcoding from UTF-16 to ISO-8859-1 or WINDOWS-1252 is going to be much faster than transcoding from UTF-8 to those encodings. And in any case, the time spent transcoding is almost always going to be insignificant compared to time spent doing actual I/O. Think console input; writing to disk; reading from CD-ROM; writing to a socket; .... Transcoding is really not a bottleneck. Jill
Aug 23 2004
prev sibling parent Ben Hinkle <bhinkle4 juno.com> writes:
Roald Ribe wrote:

 
 "Ben Hinkle" <bhinkle4 juno.com> wrote in message
 news:cgcoe6$2cq4$1 digitaldaemon.com...
 
 [snip]
 
 There were huge threads about char vs wchar vs dchar a while ago (on the

 newsgroup, I think). All kinds of things like what the default should be,
 what the names should be, what a string class could be etc. For example
  http://www.digitalmars.com/d/archives/20361.html
  http://www.digitalmars.com/d/archives/12382.html
 or actually anything at
  http://www.digitalmars.com/d/archives/index.html
 with the word "unicode" in the subject.

 By the way, why if there are N encodings are there N^2-N converters?
 Shouldn't there just be ~2*N to convert to/from one standard like
 dchar[]? IBM's ICU (at http://oss.software.ibm.com/icu/) uses wchar[] as
 the standard.

Indeed. There were several large discussions about this. Only a few scandinavian/north european readers of this group seemed to be positive at the time. I am happy to see that more people are warming to the idea. wchar (16-bit) is enough. It is even suggested as the best implementation size by some UNICODE coding experts. IBM / Sun / MS can not all be stupid at the same time... I think it would be smart to interoperate with the 16-bit size used internally in ICU, Java and MS-Windows. Only on unix/linux would it make sense to use 32-bits dchar. The 16 bits is enough for 99% of the cases/languages. The last 1% can be handled quite fast by cached indexing techniques in a String object. (this does not make for optimal speed in the 1% case, but it will more than pay for itself speedwise in 99% of all binary i/o operations :) However, that is Walters main issue I think. He wants default 8-bit chars to be default because this will make for the best possible i/o speed with the current state of affairs. That is what I understould from the last discussion at least. I am sure he will comment this thread ;-) and correct me if I am wrong. Regards, Roald

I didn't mean to suggest D ditch char or wchar or dchar. I'm just saying ICU uses wchar internally as the intermediate representation when converting between encodings. That is different than changing D's concept of strings. I should have added a sentance to my original post saying that I think D should keep its support of the "big three" char, wchar and dchar (with char[] as the standard concept of string) and have the library that handles conversions between unicode and non-unicode (or between non-unicode) encodings use whatever it wants as the intermediate representation. I think for that dchar would probably be fine - but I have no experience with that so that is just a naive guess. Treating the unicode encodings specially seems more practical than saying all non-standard encodings are treated the same. -Ben
Aug 23 2004
prev sibling parent J C Calvarese <jcc7 cox.net> writes:
Ben Hinkle wrote:
...
 There were huge threads about char vs wchar vs dchar a while ago (on the old
 newsgroup, I think). All kinds of things like what the default should be,
 what the names should be, what a string class could be etc. For example
  http://www.digitalmars.com/d/archives/20361.html
  http://www.digitalmars.com/d/archives/12382.html
 or actually anything at
  http://www.digitalmars.com/d/archives/index.html
 with the word "unicode" in the subject.

In case anyone is interested, here's a page with links to many Unicode threads in D newsgroups: http://www.prowiki.org/wiki4d/wiki.cgi?UnicodeIssues -- Justin (a/k/a jcc7) http://jcc_7.tripod.com/d/
Aug 23 2004
prev sibling next sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
The case for retaining wchar has been made, and essentially won. Please see
separate thread about ICU (and maybe move this discussion there).

"char", however, is still up for deletion, since all arguments against it still
apply.

Jill
Aug 23 2004
parent Ben Hinkle <bhinkle4 juno.com> writes:
Arcane Jill wrote:

 
 The case for retaining wchar has been made, and essentially won. Please
 see separate thread about ICU (and maybe move this discussion there).

It's won already? What is this - a Mike Tyson fight? :-) Are you referring to the old threads (on the old newsgroup) or this new thread?
 "char", however, is still up for deletion, since all arguments against it
 still apply.
 
 Jill

Once I argued that D's current concept of char should be called uchar or something to indicate the UTF-8 encoding (as opposed to C's char encoding) and that string literals have type uchar[]. I still think it would be interesting to try but it's a tweak on the current system that isn't *that* important.
Aug 23 2004
prev sibling next sibling parent Andy Friesen <andy ikagames.com> writes:
Arcane Jill wrote:

 For reasons of space-efficiency, one might want to store text in memory in
UTF-8
 format. Fair enough. But if char were to be ditched, you could still do that.
 You'd simply use a ubyte[] for that purpose (just as you are now required to do
 if you want to store text in memory in UTF-7). After all - what actually /is/ a
 UTF fragment anyway? What meaning does the UTF-8 fragment 0x83 have in
 isolation? Answer - none. It has meaning only in the context of the bytes
 surrounding it. You don't need a special primitive type just to hold that
 fragment. And of course, there is /nothing/ to stop special string classes from
 being written to provide implementations of such space-efficient abstractions.

I think it might be worth it for the conceptual clarity. UTF-32 happens to be the character type that's hardest to break. It seems logical that it be the default. The programmer can still take control and use another encoding when the problem domain allows for it, but it's an optimization, not business as usual. -- andy
Aug 23 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
I'm not sure what transcoding means, I assume it's converting from one
string type to another.

I have some experiences with having only one character type - implementing a
C compiler, which is largely string processing using ascii, implementing a
Java compiler and working on the Java vm, which does everything as a wchar,
and implementing a javascript compiler, interpreter, and runtime which does
everything as a dchar.

dchar implementations consume memory at an alarming rate, and if you're
doing server side code, this means you'll soon run into the point where
virtual memory starts thrashing, and performance goes quickly down the
tubes. I had to do a lot of work trying to overcome this problem. dchars are
very convenient to work with, however, and make a great deal of sense as the
temporary common intermediate form of all the conversions. I stress the
temporary, though, as if you keep it around you'll start to notice the
slowdowns it causes.

char implementations can be made really fast and memory efficient. I don't
know if anyone has run any statistics, but the vast bulk of text processing
done by programs is in ASCII.

wchar implementations are, of course, halfway in between. Microsoft went
with wchars back when wchars could handle all of unicode in one word. Now
that it doesn't anymore, that means that all wchar code is going to have to
handle multiword encodings. So it's just as much extra code to write as for
chars.

Java uses wchars, but Java was not designed for high performance (although
years of herculean efforts by a lot of very smart people have brought Java a
long ways in the performance department). My Java compiler written in C++,
which used UTF-8 internally, ran 10x faster than the one written in Java,
which used UTF-16. The speedup wasn't all due to the character encoding, but
it helped.

C# uses wchars, and that maps directly onto the Win32 API, so that makes a
lot of sense for Microsoft. D isn't just a Win32 language, however, and
linux uses UTF-8. Furthermore, a lot of embedded systems stick with ASCII.

In other words, I think there's a strong case for all three character
encodings in D. I agree that those using code pages will run into trouble
now and then with this, but they will anyway, because code pages are just
endless trouble since which code page some data is in is not inherent in the
data. Your idea of having the compiler detect invalid UTF-8 sequences in
string literals is very helpful here in heading a lot of these issues off at
the pass.

I think it makes perfect sense for a transcoding library to standardize on
dchar and dchar[] as its intermediate form. But don't take away char[] for
people like me that want to use it for other purposes!

I also agree that for single characters, using dchar is the best way to go.
Note that I've been redoing Phobos internals this way. For example,
std.format generates dchars to feed to the consumer function. std.ctype
takes dchar's as arguments.
Aug 23 2004
parent reply Regan Heath <regan netwin.co.nz> writes:
Walter,

I pretty much agree with everything you have said here, I don't think we 
should remove any of the char types. That said I think...

The idea that a cast from one char type to another char type (implicit or 
explicit) should perform the correct UTF transcoding(conversion) is a good 
one, my arguments for this are as follows:

  - Q: if you paint a dchar[] as a char[] what do you get?
    A: a possibly (likely?) invalid UTF-8 sequence.

So why do it? I agree we do want to be able to 'paint' one type as 
another, but for a type with a specified encoding I don't think this makes 
any sense, does it? can you think of a reason to do it? Given that you 
could use ubyte, ushort or ulong instead. (types with no specified 
encoding).

  - Doing the transcoding means people writing string handling routines 
need only provide one routine and the result will automatically be 
transcoded to the type they're using.

This is such a great bonus! it will reduce the number of string handling 
routines by 2/3 as any routine will have it's result converted to the 
required type auto-magically.

  - The argument about consistency, that a ubyte cast to a ushort does not 
transcoding so a char to a dchar shouldn't either.

There are two ways of looking at this, on one hand you're saying they 
should all 'paint' as that is consistent. However, on the other I'm saying 
they should all produce a 'valid' result. So my argument here is that when 
you cast you expect a valid result, much like casting a float to an int 
does not just 'paint'.

I am interested to hear your opinions on this idea.

Regan.

On Mon, 23 Aug 2004 12:59:28 -0700, Walter <newshound digitalmars.com> 
wrote:
 I'm not sure what transcoding means, I assume it's converting from one
 string type to another.

 I have some experiences with having only one character type - 
 implementing a
 C compiler, which is largely string processing using ascii, implementing 
 a
 Java compiler and working on the Java vm, which does everything as a 
 wchar,
 and implementing a javascript compiler, interpreter, and runtime which 
 does
 everything as a dchar.

 dchar implementations consume memory at an alarming rate, and if you're
 doing server side code, this means you'll soon run into the point where
 virtual memory starts thrashing, and performance goes quickly down the
 tubes. I had to do a lot of work trying to overcome this problem. dchars 
 are
 very convenient to work with, however, and make a great deal of sense as 
 the
 temporary common intermediate form of all the conversions. I stress the
 temporary, though, as if you keep it around you'll start to notice the
 slowdowns it causes.

 char implementations can be made really fast and memory efficient. I 
 don't
 know if anyone has run any statistics, but the vast bulk of text 
 processing
 done by programs is in ASCII.

 wchar implementations are, of course, halfway in between. Microsoft went
 with wchars back when wchars could handle all of unicode in one word. Now
 that it doesn't anymore, that means that all wchar code is going to have 
 to
 handle multiword encodings. So it's just as much extra code to write as 
 for
 chars.

 Java uses wchars, but Java was not designed for high performance 
 (although
 years of herculean efforts by a lot of very smart people have brought 
 Java a
 long ways in the performance department). My Java compiler written in 
 C++,
 which used UTF-8 internally, ran 10x faster than the one written in Java,
 which used UTF-16. The speedup wasn't all due to the character encoding, 
 but
 it helped.

 C# uses wchars, and that maps directly onto the Win32 API, so that makes 
 a
 lot of sense for Microsoft. D isn't just a Win32 language, however, and
 linux uses UTF-8. Furthermore, a lot of embedded systems stick with 
 ASCII.

 In other words, I think there's a strong case for all three character
 encodings in D. I agree that those using code pages will run into trouble
 now and then with this, but they will anyway, because code pages are just
 endless trouble since which code page some data is in is not inherent in 
 the
 data. Your idea of having the compiler detect invalid UTF-8 sequences in
 string literals is very helpful here in heading a lot of these issues 
 off at
 the pass.

 I think it makes perfect sense for a transcoding library to standardize 
 on
 dchar and dchar[] as its intermediate form. But don't take away char[] 
 for
 people like me that want to use it for other purposes!

 I also agree that for single characters, using dchar is the best way to 
 go.
 Note that I've been redoing Phobos internals this way. For example,
 std.format generates dchars to feed to the consumer function. std.ctype
 takes dchar's as arguments.

-- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 23 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"Regan Heath" <regan netwin.co.nz> wrote in message
news:opsc7hhm0a5a2sq9 digitalmars.com...
 Walter,

 I pretty much agree with everything you have said here, I don't think we
 should remove any of the char types. That said I think...

 The idea that a cast from one char type to another char type (implicit or
 explicit) should perform the correct UTF transcoding(conversion) is a good
 one, my arguments for this are as follows:

   - Q: if you paint a dchar[] as a char[] what do you get?
     A: a possibly (likely?) invalid UTF-8 sequence.

 So why do it? I agree we do want to be able to 'paint' one type as
 another, but for a type with a specified encoding I don't think this makes
 any sense, does it? can you think of a reason to do it? Given that you
 could use ubyte, ushort or ulong instead. (types with no specified
 encoding).

   - Doing the transcoding means people writing string handling routines
 need only provide one routine and the result will automatically be
 transcoded to the type they're using.

 This is such a great bonus! it will reduce the number of string handling
 routines by 2/3 as any routine will have it's result converted to the
 required type auto-magically.

   - The argument about consistency, that a ubyte cast to a ushort does not
 transcoding so a char to a dchar shouldn't either.

 There are two ways of looking at this, on one hand you're saying they
 should all 'paint' as that is consistent. However, on the other I'm saying
 they should all produce a 'valid' result. So my argument here is that when
 you cast you expect a valid result, much like casting a float to an int
 does not just 'paint'.

 I am interested to hear your opinions on this idea.

I think your idea has a lot of merit. I'm certainly leaning that way.
Aug 23 2004
parent reply "antiAlias" <fu bar.com> writes:
"Walter" <newshound digitalmars.com> wrote in message ...
 "Regan" wrote:
 There are two ways of looking at this, on one hand you're saying they
 should all 'paint' as that is consistent. However, on the other I'm


 they should all produce a 'valid' result. So my argument here is that


 you cast you expect a valid result, much like casting a float to an int
 does not just 'paint'.

 I am interested to hear your opinions on this idea.

I think your idea has a lot of merit. I'm certainly leaning that way.

On the one hand, this would be well served by an opCast() and/or opCast_r() on the primitive types; just the kind of thing suggested in a related thread (which talked about overloadable methods for primitive types). On the other hand, we're talking transcoding here. Are you gonna' limit this to UTF-8 only? Then, since the source and destination will typically be of different sizes, do you then force all casts between these types to have the destination be an array reference rather than an instance? One that is always allocated on the fly? Then there's performance. It's entirely possible to write transcoders that are minimally between 5 and 30 times faster than the std.utf ones. Some people actually do care about efficiency. I'm one of them. If you do implement overloadable primitive-methods (like properties) then, will you allow a programmer to override them? So they can make the opCast() do something more specific to their own specific task? That's seems like a lot to build into the core of a language. Personally, I think it's 'borderline' to have so many data types available for similar things. If there were an "alias wchar[] string", and the core Object supported that via "string toString()", and the IUC library were adopted, then I think some of the general confusion would perhaps melt somewhat. In many respects, too many choices is simply a BadThing (TM). Especially when there's precious little solid guidance to help. That guidance might come from a decent library that indicates how the types are used, and uses one obvious type (string?) consistently. Besides, if IUC were adopted, less people would have to worry about the distinction anyway. Believe me when I say that Mango would dearly love to go dchar[] only. Actually, it probably will at the higher levels because it makes life simple for everyone. Oh, and I've been accused many times of being an efficiency fanatic, especially when it comes to servers. But there's always a tradeoff somewhere. Here, the tradeoff is simplicity-of-use versus quantities of RAM. Which one changes dramatically over time? Hmmmm ... let me see now ... 64bit-OS for desktops just around corner? Even on an embedded device I'd probably go "dchar only" regarding I18N. Simply because the quantity of text processed on such devices is very limited. Before anyone shoots me over this one, I regularly write code for devices with just 4KB RAM ~ still use 16bit chars there when dealing with XML input. So what am I saying here? Available RAM will always increase in great leaps. Contemplating that the latter should dictate ease-of-use within D is a serious breach of logic, IMO. Ease of use, and above all, /consistency/ should be paramount; if you have the programmer in mind.
Aug 23 2004
next sibling parent reply Regan Heath <regan netwin.co.nz> writes:
On Mon, 23 Aug 2004 18:29:54 -0700, antiAlias <fu bar.com> wrote:
 "Walter" <newshound digitalmars.com> wrote in message ...
 "Regan" wrote:
 There are two ways of looking at this, on one hand you're saying they
 should all 'paint' as that is consistent. However, on the other I'm


 they should all produce a 'valid' result. So my argument here is that


 you cast you expect a valid result, much like casting a float to an 

 does not just 'paint'.

 I am interested to hear your opinions on this idea.

I think your idea has a lot of merit. I'm certainly leaning that way.

On the one hand, this would be well served by an opCast() and/or opCast_r() on the primitive types; just the kind of thing suggested in a related thread (which talked about overloadable methods for primitive types).

True. However, what else should the opCast for char[] to dchar[] do except transcode it? What about opCast for int to dchar.. it seems to me there is only 1 choice about what to do, anything else would be operator abuse.
 On the other hand, we're talking transcoding here. Are you gonna' limit 
 this
 to UTF-8 only?

I hope not, I am hoping to see: | UTF-8 | UTF-16 | UTF-32 -------------------------------- UTF-8 | - + + UTF-16 | + - + UTF-32 | + + - (+ indicates transcoding occurs)
 Then, since the source and destination will typically be of
 different sizes, do you then force all casts between these types to have 
 the
 destination be an array reference rather than an instance?

I don't think transcoding makes any sense unless you're talking about a 'string' (i.e. char[], wchar[], or dchar[]) as opposed to a UTF-x fragment (i.e. char, wchar) As AJ has frequently pointed out a char or wchar does not equal one "character" in some cases. Given that and assuming 'c' below is "not a whole character", I cannot see how: dchar d; char c = ?; //not a whole character d = c; //implicit d = cast(dchar)c; //explicit would ever be able to transcode. So in this instance this should either be: 1. a compile error 2. a runtime error 3. a simple copy of the value, creating an invalid? (will it be AJ?) utf-32 character. I opt for #1 where possible, otherwise #2, and consider #3 very bad if it has the possibilty to create an invalid utf-x code-point/fragment (I don't know the right term)
 One that is
 always allocated on the fly?

No.. bad idea methinks.
 Then there's performance. It's entirely
 possible to write transcoders that are minimally between 5 and 30 times
 faster than the std.utf ones. Some people actually do care about 
 efficiency.
 I'm one of them.

Same here. I believe we all have the same goal, we just have different ideas about how best to get there.
 If you do implement overloadable primitive-methods (like properties) 
 then,
 will you allow a programmer to override them? So they can make the 
 opCast()
 do something more specific to their own specific task?

Can you think of a task you'd want to put one of these opCast methods to? (and give an example).
 That's seems like a lot to build into the core of a language.

The opCast overloading? or the original idea? The only reservation I had about the original idea was that it seemed like "a lot to build into the core of a language" at first, and then I realised if you cast from one char type to another, the _only_ sensible thing to do is transcode, anything else is a bug/error. In addition this sort of cast (char type to char type) doesn't occur a lot, but when it does you end up calling toUTFxx manually, why not just have it happen.
 Personally, I think it's 'borderline' to have so many data types 
 available for similar things.

Like byte, short, int and long?
 If there were an "alias wchar[] string", and the core
 Object supported that via "string toString()", and the IUC library were
 adopted, then I think some of the general confusion would perhaps melt
 somewhat.

Maybe. However toString for an 'int' (for example) only needs a char[] to represent all it's possible values (with one character equalling one 'char') so why use wchar or dchar for it's toString? Conversly something else might find it more efficient to return it's toString as a dchar[]. You might argue either example is rare, certainly the latter is rarer than the former, but the efficiency gnome in my head won't shut up about this.. If we simply alias wchar[] as 'string' then these examples will need manual conversion to 'string' which involves calling toUTF16 or .. what would you call that conversion function such that it was obvious? toString? :)
 In many respects, too many choices is simply a BadThing (TM).

I think too many very similar but slightly different choices/methods is a bad thing, however, I don't see char, wchar and dchar as that. I think they are more like byte, short and int, different types for different uses, castable to/from each other where required. The choice they give you is being able to choose the right type for the job.
 Especially
 when there's precious little solid guidance to help. That guidance might
 come from a decent library that indicates how the types are used, and 
 uses
 one obvious type (string?) consistently.

I think the confusion comes from being used to only 1 string type, for C/C++ programmers it's typically and 8bit unsigned type usually containing ASCII, for Java a 16bit signed/unsigned? type containing UTF-16? Internationalisation is a new topic for me and many others (I suspect) even for Walter(?). Having 3 types requiring manual transcoding between them _is_ a pain.
 Besides, if IUC were adopted, less
 people would have to worry about the distinction anyway.

The same could be said for the implicit transcoding from one type to the other.
 Believe me when I say that Mango would dearly love to go dchar[] only.
 Actually, it probably will at the higher levels because it makes life 
 simple
 for everyone. Oh, and I've been accused many times of being an efficiency
 fanatic, especially when it comes to servers. But there's always a 
 tradeoff
 somewhere. Here, the tradeoff is simplicity-of-use versus quantities of 
 RAM.
 Which one changes dramatically over time? Hmmmm ... let me see now ...
 64bit-OS for desktops just around corner?

What you really mean is you'd dearly love to not have to worry about the differences between the 3 types, implicit transcoding will give you that. Furthermore it's simplicity without sacraficing RAM. The thing I like about implicit transcoding is that if for example you have a lot of string data stored in memory, you can store it in the most efficient format for that data, which may be char, wchar or dchar. If you then want to call a function which takes a dchar on some/all of your, it data will be implicitly converted to dchar (if not already) for the function. If at a later date you decide to change the format of the stored data, you don't have to find every call to a function and insert/remove toUTFxx calls. To me, this sounds like the most efficient way to handle it.
 Even on an embedded device I'd probably go "dchar only" regarding I18N.
 Simply because the quantity of text processed on such devices is very
 limited. Before anyone shoots me over this one, I regularly write code 
 for
 devices with just 4KB RAM ~ still use 16bit chars there when dealing with
 XML input.

We're you to use implicit transcoding you could store the data in memory in UTF-8, or UTF-16 then only transcode to UTF-32 when required, this would be more efficient.
 So what am I saying here? Available RAM will always increase in great 
 leaps.

Probably.
 Contemplating that the latter should dictate ease-of-use within D is a
 serious breach of logic, IMO.

"the latter"?
 Ease of use, and above all, /consistency/
 should be paramount; if you have the programmer in mind.

This last statement applies to implicit transcoding perfectly. It's easy and consistent. Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 23 2004
next sibling parent reply "antiAlias" <fu bar.com> writes:
"Regan Heath" <regan netwin.co.nz> wrote in message
news:opsc7u56zf5a2sq9 digitalmars.com...
 On Mon, 23 Aug 2004 18:29:54 -0700, antiAlias <fu bar.com> wrote:
 "Walter" <newshound digitalmars.com> wrote in message ...
 "Regan" wrote:
 There are two ways of looking at this, on one hand you're saying they
 should all 'paint' as that is consistent. However, on the other I'm


 they should all produce a 'valid' result. So my argument here is that


 you cast you expect a valid result, much like casting a float to an

 does not just 'paint'.

 I am interested to hear your opinions on this idea.

I think your idea has a lot of merit. I'm certainly leaning that way.

On the one hand, this would be well served by an opCast() and/or opCast_r() on the primitive types; just the kind of thing suggested in a related thread (which talked about overloadable methods for primitive types).

True. However, what else should the opCast for char[] to dchar[] do except transcode it? What about opCast for int to dchar.. it seems to me there is only 1 choice about what to do, anything else would be operator abuse.
 On the other hand, we're talking transcoding here. Are you gonna' limit
 this
 to UTF-8 only?

I hope not, I am hoping to see: | UTF-8 | UTF-16 | UTF-32 -------------------------------- UTF-8 | - + + UTF-16 | + - + UTF-32 | + + - (+ indicates transcoding occurs)

======================== And what happens when just one additional byte-oriented encoding is introduced? Perhaps UTF-7? Perhaps EBCDIC? The basic premise is flawed because there's no flexibility.
 Then, since the source and destination will typically be of
 different sizes, do you then force all casts between these types to have
 the
 destination be an array reference rather than an instance?

I don't think transcoding makes any sense unless you're talking about a 'string' (i.e. char[], wchar[], or dchar[]) as opposed to a UTF-x fragment (i.e. char, wchar)

======================== We /are/ talking about arrays. Perhaps if that sentence had ended "array reference rather than an *array* instance?", it might have been more clear? The point being made is that you would not be able to do anything like this: char[15] dst; dchar[10] src; dst = cast(char[]) src; because there's no ability via a cast() to indicate how many items from src were converted, and how many items in dst were populated. You are forced into this kind of thing: char[] dst; dchar[10] src; dst = cast(char[]) src; You see the distinction? It may be subtle to some, but it's a glaring imbalance to others. The lValue must always be a reference because it's gonna' be allocated dynamically to ensure all of the rValue will fit. In the end, it's just far better to use functions/methods that provide the feedback required so you can actually control what's going on (or such that a library function can). That way, you're not restricted in the same fashion. We don't need more asymmetry in D, and this just reeks of poor design, IMO. To drive this home, consider the decoding version (rather than the encoding above): char[15] src; dchar[] dst; dst = cast(dchar[]) src; What happens when there's a partial character left undecoded at the end of 'src'? There nothing here to tell you that you've got a dangly bit left at the end of the souce-buffer. It's gone. Poof! Any further decoding from the same file/socket/whatever is henceforth trashed, because the ball has been both dropped and buried. End of story.
 Having 3 types requiring manual transcoding between them _is_ a pain.

======================== It certainly is. That's why other languages try to avoid it at all costs. Having it done "generously" by the compiler is also a pain, inflexible, and likely expensive. There are many things a programmer should take responsibility for; transcoding comes under that umbrella because (a) there can be subtle complexity involved and (b) it is relatively expensive to churn through text and convert it; particularly so with the Phobos utf-8 code. What you appear to be suggesting is that this kind of thing should happen silently whilst one nonchalantly passes arguments around between methods. That's insane, so I hope that's not what you're advocating. Java, for example, does that at one specific layer (I/O), but you're apparently suggesting doing it at any old place! And several times over, just in case it wasn't good enough the first time :-) Sorry man. This is inane. D is /not/ a scripting language; instead it's supposed to be a systems language.
 Besides, if IUC were adopted, less
 people would have to worry about the distinction anyway.

The same could be said for the implicit transcoding from one type to the other.

======================== That's pretty short-sighted IMO. You appear to be saying that implicit transcoding would take the place of ICU; terribly misleading. Transcoding is just a very small part of that package. Please try to reread the comment as "most people would be shielded completely by the library functions, therefore there's far fewer scenarios where they'd ever have a need to drop into anything else". This would be a very GoodThing for D users. Far better to have a good library to take case of /all/ this crap than have the D language do some partial-conversions on the fly, and then quit because it doesn't know how to provide any further functionality. This is the classic core-language-versus-library-functionality bitchfest all over again. Building all this into a cast()? Hey! Let's make Walter's Regex class part of the compiler too; and make it do UTF-8 decoding /while/ it's searching, since you'll be able to pass it a dchar[] that will be generously converted to the accepted char[] for you "on-the-fly". Excuse me for jesting, but perhaps the Bidi text algorithms plus date/numeric formatting & parsing will all fit into a single operator also? That's kind of what's being suggested. I believe there's a serious underestimate of the task being discussed.
 Believe me when I say that Mango would dearly love to go dchar[] only.
 Actually, it probably will at the higher levels because it makes life
 simple
 for everyone. Oh, and I've been accused many times of being an


 fanatic, especially when it comes to servers. But there's always a
 tradeoff
 somewhere. Here, the tradeoff is simplicity-of-use versus quantities of
 RAM.
 Which one changes dramatically over time? Hmmmm ... let me see now ...
 64bit-OS for desktops just around corner?

What you really mean is you'd dearly love to not have to worry about the differences between the 3 types, implicit transcoding will give you that. Furthermore it's simplicity without sacraficing RAM.

======================== Ahh Thanks. I didn't realize that's what I "really meant". Wait a minute ...
 Even on an embedded device I'd probably go "dchar only" regarding I18N.
 Simply because the quantity of text processed on such devices is very
 limited. Before anyone shoots me over this one, I regularly write code
 for
 devices with just 4KB RAM ~ still use 16bit chars there when dealing


 XML input.

We're you to use implicit transcoding you could store the data in memory in UTF-8, or UTF-16 then only transcode to UTF-32 when required, this would be more efficient.

======================== That's a rather large assumption, don't you think? More efficient? In which particualr way? Is memory usage or CPU usage more important in /my/ particular applications? Please either refrain, or commit to rewriting all my old code more efficiently for me ... for free <g>
Aug 23 2004
next sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgelmc$dqu$1 digitaldaemon.com>, antiAlias says...

And what happens when just one additional byte-oriented encoding is
introduced? Perhaps UTF-7? Perhaps EBCDIC? The basic premise is flawed
because there's no flexibility.

Yes. But I don't see D introducing a utf7char or a utfEbcdicChar type any time soon. There's no flexibility because there are no plans to extend it. I can live with ONE native string type. I can also live with 3 mutually interoperable native string types. But neither of these schemes precludes anyone else from writing/using other string types.
We /are/ talking about arrays. Perhaps if that sentence had ended "array
reference rather than an *array* instance?", it might have been more clear?
The point being made is that you would not be able to do anything like this:

char[15] dst;
dchar[10] src;

dst = cast(char[]) src;

But that's because you already can't do: # dst = toUTF8(src); and that's not going to change, so that's the end of that. Regan isn't suggesting anything beyond a bit of syntactic sugar here. No-one (so far) has suggested that dynamic arrays be converted to static arrays, or that auto-casting should write into a user-supplied fixed-size buffer. D does not prohibit you from doing the things you have suggested. It's just that you'd have to do them explicitly. /Implicit/ conversion is only being suggested for the three D string types.
because there's no ability via a cast() to indicate how many items from src
were converted, and how many items in dst were populated. You are forced
into this kind of thing:

char[] dst;
dchar[10] src;

dst = cast(char[]) src;

Actually it would be: # dst = src; if I had my way.
You see the distinction? It may be subtle to some, but it's a glaring
imbalance to others. The lValue must always be a reference because it's
gonna' be allocated dynamically to ensure all of the rValue will fit. In the
end, it's just far better to use functions/methods that provide the feedback
required so you can actually control what's going on (or such that a library
function can). That way, you're not restricted in the same fashion. We don't
need more asymmetry in D, and this just reeks of poor design, IMO.

I don't agree with this claim. It is being suggested that: # wchar[] s1; # char s2 = s1; be equivalent to: # wchar[] s1; # char s2 = std.utf.toUTF8(s1); Nobody loses anything by this. All that happens is that things work more smoothly. If you want to call a different function to do your transcoding then there's nothing to stop you. I assume that the use you have in mind is stream-internal transcoding via buffers. You can still do that. The above won't stop you.
To drive this home, consider the decoding version (rather than the encoding
above):

char[15] src;
dchar[] dst;

dst = cast(dchar[]) src;

What happens when there's a partial character left undecoded at the end of
'src'?

*ALL* that is being suggested is that, given the above declarations: # dst = src; would be equivalent to: # dst = std.utf.toUTF32(src); Nothing more. So the answer to your question is, it would throw an exception.
There nothing here to tell you that you've got a dangly bit left at
the end of the souce-buffer. It's gone. Poof! Any further decoding from the
same file/socket/whatever is henceforth trashed, because the ball has been
both dropped and buried. End of story.

This has got nothing to do with either transcoding or streams. This is not being suggested as a general transcoding mechanism, merely as an internal conversion between D's three string types. /General/ transcoding will have to work for all supported encodings, and won't be relying on the std.utf functions. Files and sockets won't use the std.utf functions either because they will employ the general transcoding mechanism. Your transcoding ideas are excellent, but they are not relevant to this.
There are many things a programmer should take
responsibility for; transcoding comes under that umbrella because (a) there
can be subtle complexity involved and (b) it is relatively expensive to
churn through text and convert it; particularly so with the Phobos utf-8
code.

I'm surprised at you. I would have said:
There are some things a programmer should /not/ have to take
responsibility for; transcoding comes under that umbrella because (a) there
can be subtle complexity involved and (b) it is relatively expensive to
churn through text and convert it;

A library is the appropriate place for this stuff. That could be Phobos, Mango, whatever. Here's my take on it: (1) Implicit conversion between native D strings should be handled by the compiler in whatever way it sees fit. If it chooses to invoke a library function then so be it. (And it /should/ be Phobos, because Phobos is shipped with D). Note that this is exactly the same situation that the future "cent" type will incur. If I divide one "cent" by another, I would expect the D compiler to translate this into a function call within Phobos. Why is that such a crime? (2) Fully featured transcoding should be done by ICU, as this is a high performance mature product, and all the code is already written. (3) Some adaptation of the ICU wrapper may be necessary to integrate this more neatly with "the D way". But I'm confident we can do this in a way which is not biased toward Phobos. Phobos UTF-8 code could be faster, I grant you. But perhaps it will be in the next release. We're only talking about a tiny number of functions here, after all.
What you appear to be suggesting is that this kind of thing should happen
silently whilst one nonchalantly passes arguments around between methods.

I would vote for that, yes. But only as a second choice. My /first/ choice would be to have D standardize on one single kind of string, the wchar[].
That's insane, so I hope that's not what you're advocating. Java, for
example, does that at one specific layer (I/O), but you're apparently
suggesting doing it at any old place! And several times over, just in case
it wasn't good enough the first time :-)  Sorry man. This is inane. D is
/not/ a scripting language; instead it's supposed to be a systems language.

How about we all get on the same side here? Like I said, my /first/ choice would be to have D standardize on one single kind of string, the wchar[]. But we don't always get our first choice. Walter doesn't like this idea. I suggest you add your voice to mine and help try to persuade him that ONE kind of string is the way to go. BUT - if we fail in convincing him - the second choice is better than the status quo. Why? Because if we fail to convice Walter that multiple string types is bad, then all that conversion is going to happen ANYWAY. It will happen because Object.toString() will (often) return UTF-8; because string literals will generate UTF-8; and because the functions in ICU will require and return UTF-16. The ONLY difference between this suggestion and the status quo is that won't have to write "cast(char[])" and "cast(wchar[])" all over the place. You're doing a lot of arguing /against/. What are you /for/? Arguing against a suggestion is usually interpreted as a vote for the status quo. Are you really doing that?
That's pretty short-sighted IMO. You appear to be saying that implicit
transcoding would take the place of ICU; terribly misleading.

Of course he's not.
Excuse me for jesting, but perhaps the Bidi text algorithms plus
date/numeric formatting & parsing will all fit into a single operator also?
That's kind of what's being suggested.

Nothing is being suggested except syntactic sugar. My first choice vote still goes to ditching the char. With char gone, it would be natural for wchar[] to become the "standard" D string, which fits in well with ICU. There would be no need for implicit (or even explicit) cast-conversion between wchar[] and dchar[], because dchar[] would be a specialist thing, only used by speed-efficiency fanatics, while wchar[] would be business as usual. But if I can't have my first choice, I'll vote for implicit conversion as my second choice. Arcane Jill
Aug 24 2004
next sibling parent reply "antiAlias" <fu bar.com> writes:
"Arcane Jill" <Arcane_member pathlink.com>
There nothing here to tell you that you've got a dangly bit left at
the end of the souce-buffer. It's gone. Poof! Any further decoding from


same file/socket/whatever is henceforth trashed, because the ball has


both dropped and buried. End of story.

This has got nothing to do with either transcoding or streams. This is not

 suggested as a general transcoding mechanism, merely as an internal

 between D's three string types. /General/ transcoding will have to work

 supported encodings, and won't be relying on the std.utf functions. Files

 sockets won't use the std.utf functions either because they will employ

 general transcoding mechanism.

Oh. I was under the impression the 'solution' being tendered was a jack-of-all trades. If we're simply talking about converting /static/ strings between different representations, then, cool. It's done at compile-time.
 I'm surprised at you. I would have said:

There are some things a programmer should /not/ have to take
responsibility for; transcoding comes under that umbrella because (a)


can be subtle complexity involved and (b) it is relatively expensive to
churn through text and convert it;

A library is the appropriate place for this stuff. That could be Phobos,

 whatever. Here's my take on it:

We agree. My point was that expensive operations such as these should perhaps not be hidden "under the covers"; but explicitly handled by a library call instead. However, I clearly have the wrong impression about the extent of what this implicit-conversion is attempting.
 How about we all get on the same side here?

I really think we are, Jill. My concerns are about trying to build partial versions of ICU functionality into the D language itself, rather than let that extensive and capable library take care of it. But apparently that's not what's happening here. My mistake.
 You're doing a lot of arguing /against/. What are you /for/? Arguing

 suggestion is usually interpreted as a vote for the status quo. Are you

 doing that?

Nope. There appeared to be some concensus-building that transcoding could all be handled via a cast() operator. I felt it worth pointing out where and why that's not a valid approach. The other aspect involved here is that of string-concatenation. D cannot have more that one return type for toString() as you know. It's fixed at char[]. If string concatenation uses the toString() method to retrieve its components (as is being proposed elsewhere), then there will be multiple, redundant, implicit conversions going on where the string really wanted to be dchar[] in the first place. That is: A a; // class instances ... B b; C c; dchar[] message = c ~ b ~ a; Under the proposed "implicit" scheme, if each toString() of A, B, and C wish to return dchar[], then each concatenation causes an implicit conversion/encoding from each dchar[] to char[] (for the toString() return). Then another full conversion/decoding is performed back to the dchar[] assignment once each has been concatenated. This is like the Wintel 'plot' for selling more cpu's :-) Doing this manually, one would forego the toString() altogether: dchar[] message = c.getString() ~ b.getString() ~ a.getString(); ... where getString() is a programmer-specific idiom to return the (natural) dchar[] for these classes, and we carefully avoided all those darned implicit-conversions. However, which approach do you think people will use? My guess is that D may become bogged down in conversion hell over such things. So, to answer your question: What I'm /for/ is not covering up these types of issues with blanket-style implicit conversions. Something more constructive (and with a little more forethought) needs to be done.
Aug 24 2004
next sibling parent reply Regan Heath <regan netwin.co.nz> writes:
On Tue, 24 Aug 2004 11:36:45 -0700, antiAlias <fu bar.com> wrote:

<snip>

 Oh. I was under the impression the 'solution' being tendered was a
 jack-of-all trades. If we're simply talking about converting /static/
 strings between different representations, then, cool. It's done at
 compile-time.

Nope. We are talking about implicit conversion to/from all 3 forms of UTF-x encoded base types where required. Does this make any sense whatsoever currently? char[] c = ?; //some valid utf-8 sequence dchar[] d; d = cast(char[])c; I believe the answer is "no", reasoning: d will now (possibly) contain an invalid utf-32 sequence. The only sensible thing to do is transcode. If you want to 'paint' a char type as a smaller type to get at it's bits/bytes or shorts (snicker) you can and should use ubyte or ushort, _not_ another char type. char types imply a default encoding so painting one to another is illegal, painting something to/from a char type is legal and useful. <snip>
 We agree. My point was that expensive operations such as these should
 perhaps not be hidden "under the covers"; but explicitly handled by a
 library call instead.

On principle I totally agree with this statement. However in this case I am simply suggesting implicit conversion where you would already have to write toUTFxx(), the idea does not _add_ any expense, only convenience. Yes, an un-aware programmer might not realise it's transcoding, they might make some in-efficient choices, but that same programmer will probably also do this: char[] c; dcahr[] d; c = cast(char[])d; and create invalid utf-x sequences (some of the time), and a bug.
 However, I clearly have the wrong impression about the
 extent of what this implicit-conversion is attempting.

No.. I _think_ you understood it fine. I _think_ we just disagree about what is efficient and what is not.
 How about we all get on the same side here?

I really think we are, Jill. My concerns are about trying to build partial versions of ICU functionality into the D language itself, rather than let that extensive and capable library take care of it. But apparently that's not what's happening here. My mistake.

Err... it is, a small part, the part that already exists in std.utf, conversion from utf-x to utf-<another x>.
 You're doing a lot of arguing /against/. What are you /for/? Arguing

 suggestion is usually interpreted as a vote for the status quo. Are you

 doing that?

Nope. There appeared to be some concensus-building that transcoding could all be handled via a cast() operator. I felt it worth pointing out where and why that's not a valid approach.

Actually I want it to transcode implicitly eg. char[] c; dchar[] d; d = c; //transcodes Can you enumerate your reasons why this is 'not a valid approach' (I could search the previous posts and try to do that for you, but I might miss-interpret what you meant).
 The other aspect involved here is that of string-concatenation. D cannot
 have more that one return type for toString() as you know.

True.
 It's fixed at
 char[].

Is it?! I didn't realise that, so this is invalid? class A { dchar[] toString() {} }
 If string concatenation uses the toString() method to retrieve its
 components (as is being proposed elsewhere), then there will be multiple,
 redundant, implicit conversions going on where the string really wanted 
 to
 be dchar[] in the first place. That is:

 A a; // class instances ...
 B b;
 C c;

 dchar[] message = c ~ b ~ a;

 Under the proposed "implicit" scheme, if each toString() of A, B, and C 
 wish
 to return dchar[], then each concatenation causes an implicit
 conversion/encoding from each dchar[] to char[] (for the toString() 
 return).

Assuming toString returned char[] and not dchar[], yes. Assuming that trying to return dchar without transcoding will create invalid UTF-8 sequences. If implicit transcoding were implemented then you should be able to define the return value of your classes toString to be any of char[] wchar[] or dchar[] as it will implicitly transcode to the type it requires. Basically you use the most applicable type, for example AJ's Int class would use char[] (unless AJ has another reason not to) as all the string data required is ASCII and fits best in UTF-8.
 Then another full conversion/decoding is performed back to the dchar[]
 assignment once each has been concatenated. This is like the Wintel 
 'plot' for selling more cpu's :-)

Not if toString can return dchar[] and all 3 classes do that.
 Doing this manually, one would forego the toString() altogether:

 dchar[] message = c.getString() ~ b.getString() ~ a.getString();

 ... where getString() is a programmer-specific idiom to return the 
 (natural)
 dchar[] for these classes, and we carefully avoided all those darned
 implicit-conversions. However, which approach do you think people will 
 use?

Taking into account what I have said above.. the easy one, i.e. implicit transcoding.
 My guess is that D may become bogged down in conversion hell over such
 things.

 So, to answer your question:
 What I'm /for/ is not covering up these types of issues with 
 blanket-style
 implicit conversions. Something more constructive (and with a little more
 forethought) needs to be done.

I believe implicit conversion to be constructive, it stops bugs and makes string handling much easier. What we are doing here _is_ the forethought, after all nothing has been implemented yet. Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
next sibling parent reply "antiAlias" <fu bar.com> writes:
"Regan Heath" <regan netwin.co.nz> wrote
 Is it?!
 I didn't realise that, so this is invalid?

 class A {
    dchar[] toString() {}
 }

Yes. It most certainly is, Regan. I (incorrectly) assumed you understood that. Sorry. There have been a number of posts that note this, and its implications.
Aug 24 2004
parent reply Regan Heath <regan netwin.co.nz> writes:
On Tue, 24 Aug 2004 19:45:07 -0700, antiAlias <fu bar.com> wrote:

 "Regan Heath" <regan netwin.co.nz> wrote
 Is it?!
 I didn't realise that, so this is invalid?

 class A {
    dchar[] toString() {}
 }

Yes. It most certainly is, Regan. I (incorrectly) assumed you understood that.

Either: a. I am overly sensitive/insecure b. You didn't realise c. You're intentionally trying to belittle me because ... "understood" is not the right word "knew" is a better choice.. "understood" implies I knew but didn't understand. That isn't the case. (this time)
 Sorry. There have been a number of posts that note this, and its
 implications.

I must have missed them, or missed the importance of that fact. strange given that I read *everything* in all the D NG's on digitalmars.com. Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
parent "antiAlias" <fu bar.com> writes:
Sorry to have offended your sensibilities, dude. If "knew" is used in your
part of the world then "understood" is used in mine. Too bad for the
misunderstanding.

Here's a link to a post from Matthew. Given that it's a reply, I think you
can safely count at least two posts that you missed <g>

news:cg69c1$120n$1 digitaldaemon.com



"Regan Heath" <regan netwin.co.nz> wrote in message
news:opsc9uihna5a2sq9 digitalmars.com...
 On Tue, 24 Aug 2004 19:45:07 -0700, antiAlias <fu bar.com> wrote:

 "Regan Heath" <regan netwin.co.nz> wrote
 Is it?!
 I didn't realise that, so this is invalid?

 class A {
    dchar[] toString() {}
 }

Yes. It most certainly is, Regan. I (incorrectly) assumed you understood that.

Either: a. I am overly sensitive/insecure b. You didn't realise c. You're intentionally trying to belittle me because ... "understood" is not the right word "knew" is a better choice.. "understood" implies I knew but didn't understand. That isn't the case. (this time)
 Sorry. There have been a number of posts that note this, and its
 implications.

I must have missed them, or missed the importance of that fact. strange given that I read *everything* in all the D NG's on digitalmars.com. Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

Aug 24 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <opsc9m9ccg5a2sq9 digitalmars.com>, Regan Heath says...

Is it?!
I didn't realise that, so this is invalid?

class A {
   dchar[] toString() {}
}

It's not invalid as such, it's just that the return type of an overloaded function has to be "covariant" with the return type of the function it's overloading. So it's a compile error /now/. But if dchar[] and char[] were to be considered mutually covariant then this would magically start to compile. Arcane Jill
Aug 24 2004
next sibling parent reply Regan Heath <regan netwin.co.nz> writes:
On Wed, 25 Aug 2004 05:44:18 +0000 (UTC), Arcane Jill 
<Arcane_member pathlink.com> wrote:
 In article <opsc9m9ccg5a2sq9 digitalmars.com>, Regan Heath says...

 Is it?!
 I didn't realise that, so this is invalid?

 class A {
   dchar[] toString() {}
 }

It's not invalid as such, it's just that the return type of an overloaded function has to be "covariant" with the return type of the function it's overloading. So it's a compile error /now/. But if dchar[] and char[] were to be considered mutually covariant then this would magically start to compile.

Ahh.. excellent, that is what I was hoping to hear. Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <opsc9xu9h35a2sq9 digitalmars.com>, Regan Heath says...

Ahh.. excellent, that is what I was hoping to hear.

It's not /all/ good news however. Consider these two cases: (1) # class A { wchar[] toString(); } # A a = new A(); # wchar[] s = a.toString(); All hunky dory. No conversions happen. /But/ (2) # class A { wchar[] toString(); } # Object a = new A(); # wchar[] s = a.toString(); Now /two/ conversions happen (assuming Object.toString() still returns char[]) - toUTF8(wchar[]) followed by toUTF16(char[]). Still, that's polymorphism for you. It is better than the status quo, but not quite as good (IMO) as having wchar[] be the standard string type. Arcane Jill
Aug 24 2004
parent reply Regan Heath <regan netwin.co.nz> writes:
On Wed, 25 Aug 2004 06:31:16 +0000 (UTC), Arcane Jill 
<Arcane_member pathlink.com> wrote:
 In article <opsc9xu9h35a2sq9 digitalmars.com>, Regan Heath says...

 Ahh.. excellent, that is what I was hoping to hear.

It's not /all/ good news however. Consider these two cases: (1) # class A { wchar[] toString(); } # A a = new A(); # wchar[] s = a.toString(); All hunky dory. No conversions happen. /But/ (2) # class A { wchar[] toString(); } # Object a = new A(); # wchar[] s = a.toString(); Now /two/ conversions happen (assuming Object.toString() still returns char[]) - toUTF8(wchar[]) followed by toUTF16(char[]). Still, that's polymorphism for you.

True, and we can come up with much nastier string concatenation examples too.. I wonder if some cleverness can be thought up to lessen this effect somehow?
 It is better than the status quo, but not quite as good (IMO) as having 
 wchar[]
 be the standard string type.

By 'standard' do you mean that the others do not exist? or that it is the type you are encouraged to use unless you have reason not to? I think the other types have a valid place in the D language, after all each type will be more or less efficient based on the specific circumstances it gets used in. The most generally efficient type (if that's even possible to decide) should be the type we're encouraged to use, if that's wchar so be it. Implicit transcoding will fit nicely with a standard type as when you are using another type the library functions (if all written for wchar for example) will still be available without explicit toUTFxx calls. Regan. -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 25 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <opsdbboxb05a2sq9 digitalmars.com>, Regan Heath says...

 It is better than the status quo, but not quite as good (IMO) as having 
 wchar[]
 be the standard string type.

By 'standard' do you mean that the others do not exist? or that it is the type you are encouraged to use unless you have reason not to?

I guess I mean specifically that: (1) Object.toString() should return wchar[], not char[] (2) String literals such as "hello world" should be interpretted as wchar[], not char[]. (3) Object.d should contain the line: # alias wchar[] string; (4) The text on page http://www.digitalmars.com/d/arrays.html should be changed. Currently it says:
Dynamic arrays in D suggest the obvious solution - a string is just a dynamic
array of characters. String literals become just an easy way to write character
arrays.

    char[] str;
    char[] str1 = "abc";

This should be changed to:
Dynamic arrays in D suggest the obvious solution - a string is just a dynamic
array of characters. String literals become just an easy way to write character
arrays.

    wchar[] str;
    wchar[] str1 = "abc";

(5) There are probably several other things to change. I don't claim this is an exhaustive list. In other words, we could actually have our cake /and/ eat it. The intent is to minimize, as far as possible, the number of calls to toUTFxx(). Ideally, they should occur only at input and output. The way to minimize this is to keep everything in the same type, so conversion is not needed. If the D documentation, the behaviour of the compiler, and the organization of Phobos, were to consistently use the same string type, and others were encouraged to use the same type, conversions would be kept to a minimum. Currently D does that - but it's "string of choice" is the char[], not the wchar[]. Conversion to/from UTF-8 is incredibly slow for non-ASCII characters. (It could be made faster, but it can /never/ be made as fast as UTF-16). So we make wchar[], not char[], the "standard", and hey presto, things get faster (and what's more will interface with ICU without conversion, which is really important for internationalization).
The most generally efficient type (if that's even possible to decide) 
should be the type we're encouraged to use, if that's wchar so be it.

Well, it is usually believed that UTF-8 is the most space-efficient but the least speed-efficient. UTF-32 is the most speed-efficient but the least space-efficient. UTF-16 is the happy medium. However... *) UTF-16 almost as fast as UTF-32, because the UTF-16 encoding is so simple *) UTF-16 is more compact than UTF-8 for codepoints between U+0800 and U+FFFF, each of which require 3 bytes in UTF-8, but only two bytes in UTF-16 *) The characters expressable in UTF-16 in a single wchar include every symbol from every living language, so if you /pretend/ that wchar[] is an array of characters rather than UTF-16 fragments, the effect is relatively harmless (unlike UTF-8).
Implicit transcoding will fit nicely with a standard type as when you are 
using another type the library functions (if all written for wchar for 
example) will still be available without explicit toUTFxx calls.

True. I can't argue with that. But back to the case for ditching char - think of this from another perspective. In Java, would you be prepared to argue the case for the /introduction/ of an 8-bit wide character type to the language? And that this type could only ever be used for UTF-8? There's a reason why that suggestion sounds absurd. It is. Arcane Jill
Aug 26 2004
next sibling parent "antiAlias" <fu bar.com> writes:
Hear! Hear!


"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgk32j$1fj$1 digitaldaemon.com...
 In article <opsdbboxb05a2sq9 digitalmars.com>, Regan Heath says...

 It is better than the status quo, but not quite as good (IMO) as having
 wchar[]
 be the standard string type.

By 'standard' do you mean that the others do not exist? or that it is the type you are encouraged to use unless you have reason not to?

I guess I mean specifically that: (1) Object.toString() should return wchar[], not char[] (2) String literals such as "hello world" should be interpretted as

 char[].

 (3) Object.d should contain the line:
 #    alias wchar[] string;

 (4) The text on page http://www.digitalmars.com/d/arrays.html should be

 Currently it says:

Dynamic arrays in D suggest the obvious solution - a string is just a


array of characters. String literals become just an easy way to write


arrays.

    char[] str;
    char[] str1 = "abc";

This should be changed to:
Dynamic arrays in D suggest the obvious solution - a string is just a


array of characters. String literals become just an easy way to write


arrays.

    wchar[] str;
    wchar[] str1 = "abc";

(5) There are probably several other things to change. I don't claim this

 exhaustive list.



 In other words, we could actually have our cake /and/ eat it. The intent

 minimize, as far as possible, the number of calls to toUTFxx(). Ideally,

 should occur only at input and output. The way to minimize this is to keep
 everything in the same type, so conversion is not needed. If the D
 documentation, the behaviour of the compiler, and the organization of

 were to consistently use the same string type, and others were encouraged

 the same type, conversions would be kept to a minimum.

 Currently D does that - but it's "string of choice" is the char[], not the
 wchar[]. Conversion to/from UTF-8 is incredibly slow for non-ASCII

 (It could be made faster, but it can /never/ be made as fast as UTF-16).

 make wchar[], not char[], the "standard", and hey presto, things get

 what's more will interface with ICU without conversion, which is really
 important for internationalization).




The most generally efficient type (if that's even possible to decide)
should be the type we're encouraged to use, if that's wchar so be it.

Well, it is usually believed that UTF-8 is the most space-efficient but the least speed-efficient. UTF-32 is the most speed-efficient but the least space-efficient. UTF-16 is the happy medium. However... *) UTF-16 almost as fast as UTF-32, because the UTF-16 encoding is so

 *) UTF-16 is more compact than UTF-8 for codepoints between U+0800 and

 each of which require 3 bytes in UTF-8, but only two bytes in UTF-16
 *) The characters expressable in UTF-16 in a single wchar include every

 from every living language, so if you /pretend/ that wchar[] is an array

 characters rather than UTF-16 fragments, the effect is relatively harmless
 (unlike UTF-8).


Implicit transcoding will fit nicely with a standard type as when you are
using another type the library functions (if all written for wchar for
example) will still be available without explicit toUTFxx calls.

True. I can't argue with that. But back to the case for ditching char - think of this from another

 In Java, would you be prepared to argue the case for the /introduction/ of

 8-bit wide character type to the language? And that this type could only

 used for UTF-8?

 There's a reason why that suggestion sounds absurd. It is.

 Arcane Jill

Aug 26 2004
prev sibling next sibling parent reply Regan Heath <regan netwin.co.nz> writes:
On Thu, 26 Aug 2004 07:21:55 +0000 (UTC), Arcane Jill 
<Arcane_member pathlink.com> wrote:
 In article <opsdbboxb05a2sq9 digitalmars.com>, Regan Heath says...

 It is better than the status quo, but not quite as good (IMO) as having
 wchar[]
 be the standard string type.

By 'standard' do you mean that the others do not exist? or that it is the type you are encouraged to use unless you have reason not to?

I guess I mean specifically that: (1) Object.toString() should return wchar[], not char[]

Sure... so long as custom classes can overload and return char[] for situations where the app might be using char[] throughout (for whatever reason). OT: here is an example where we _don't_ want the return value used for method name resolution.
 (2) String literals such as "hello world" should be interpretted as 
 wchar[], not char[].

Currently doesn't it decide the type based on the context? i.e. void foo(dchar[] a); foo("hello world"); would make "hello world" a dchar string literal? I guess what you're saying is the default should be wchar[] where the type is indeterminate i.e. writef("hello world"); but, why not use char[] for the above, it's more efficient in this case. The compiler could do a quick decision based on whether the string contains any code points >= U+0800 if not use char[] otherwise user wchar[], would that be a good soln? After all, I don't mind if my compile is a little slower if it means my app is faster.
 (3) Object.d should contain the line:
 #    alias wchar[] string;

I'm not sure I like this.. will this hide details a programmer should be aware of?
 (4) The text on page http://www.digitalmars.com/d/arrays.html should be 
 changed.
 Currently it says:

 Dynamic arrays in D suggest the obvious solution - a string is just a 
 dynamic
 array of characters. String literals become just an easy way to write 
 character
 arrays.

    char[] str;
    char[] str1 = "abc";

This should be changed to:
 Dynamic arrays in D suggest the obvious solution - a string is just a 
 dynamic
 array of characters. String literals become just an easy way to write 
 character
 arrays.

    wchar[] str;
    wchar[] str1 = "abc";

(5) There are probably several other things to change. I don't claim this is an exhaustive list.

Sure, char[] is probably used and suggested in every example in the manuals except where it's giving an example if the differences in utf-x encodings.
 In other words, we could actually have our cake /and/ eat it. The intent 
 is to
 minimize, as far as possible, the number of calls to toUTFxx().

Agreed.
 Ideally, they should occur only at input and output. The way to minimize 
 this is to keep everything in the same type, so conversion is not 
 needed. If the D
 documentation, the behaviour of the compiler, and the organization of 
 Phobos, were to consistently use the same string type, and others were 
 encouraged to use the same type, conversions would be kept to a minimum.

 Currently D does that - but it's "string of choice" is the char[], not 
 the wchar[]. Conversion to/from UTF-8 is incredibly slow for non-ASCII 
 characters.
 (It could be made faster, but it can /never/ be made as fast as UTF-16). 
 So we make wchar[], not char[], the "standard", and hey presto, things 
 get faster

Qualification: For non ASCII only apps. (and what's more will interface with ICU without conversion, which is really
 important for internationalization).

The fact that ICU has no char type suggests it's a bad choice for D, that is, if we want to assume they knew what they we're doing. Are there any complaints from developers about ICU anywhere, perhaps some digging for dirt would help make an objective decision here?
 The most generally efficient type (if that's even possible to decide)
 should be the type we're encouraged to use, if that's wchar so be it.

Well, it is usually believed that UTF-8 is the most space-efficient but the least speed-efficient. UTF-32 is the most speed-efficient but the least space-efficient. UTF-16 is the happy medium. However... *) UTF-16 almost as fast as UTF-32, because the UTF-16 encoding is so simple *) UTF-16 is more compact than UTF-8 for codepoints between U+0800 and U+FFFF, each of which require 3 bytes in UTF-8, but only two bytes in UTF-16 *) The characters expressable in UTF-16 in a single wchar include every symbol from every living language

.. ? I thought the problem Java had was with Unicode contains characters that are > U+FFFF.. if they're not part of a 'living language' what are they?
 , so if you /pretend/ that wchar[] is an array of
 characters rather than UTF-16 fragments, the effect is relatively 
 harmless
 (unlike UTF-8).

Sure, however if you're only dealing with ASCII doing that with char[] is also fine. Those of us who haven't done any internationalisation are used to dealing only with ASCII. I'd like some more stats and figures, simply: - how many unicode characters are in the range < U+0800? - how many unicode from U+0800 >= x <= U+FFFF? - haw many unicode > U+FFFF? (the answers to the above are probably quite simple, but I want them from someone who 'knows' rather than me who'd be guessing) Then, how commonly is each range used? I imagine this differs depending on exactly what you're doing.. basically when would you use characters in each range and how common is that task? It used to be that ASCII < U+0800 was the most common, it still may be, but I can see that it's not the future, the future is unicode.
 Implicit transcoding will fit nicely with a standard type as when you 
 are
 using another type the library functions (if all written for wchar for
 example) will still be available without explicit toUTFxx calls.

True. I can't argue with that. But back to the case for ditching char - think of this from another perspective. In Java, would you be prepared to argue the case for the /introduction/ of an 8-bit wide character type to the language? And that this type could only ever be used for UTF-8? There's a reason why that suggestion sounds absurd. It is.

Isn't the reason that all the existing Java stuff, of which there is a *lot* uses wchar, so char wouldn't intgrate. That is different to D, where all 3 exist and char is actually the one with the most integration. That said, would the introduction of char to Java give you anything? perhaps.. it would allow you to write an app that only deals with ASCII (chars < U+0800) more space efficiently, correct? Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 26 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <opsdc4sgsi5a2sq9 digitalmars.com>, Regan Heath says...

I guess what you're saying is the default should be wchar[] where the type 
is indeterminate

Yes.
but, why not use char[] for the above, it's more efficient in this case. 
The compiler could do a quick decision based on whether the string 
contains any code points >= U+0800 if not use char[] otherwise user 
wchar[], would that be a good soln?

You suggestion would work. But I'm still thinking along the lines that no conversion is better than some conversion, and no conversion is only achievable by having only one type of string. And even we don't enforce that, we should at least encourage it.
After all, I don't mind if my compile is a little slower if it means my 
app is faster.

UTF-16 is not slower that UTF-8, even for pure ASCII. An app in which all text is ASCII is going to be just as fast in UTF-16 as it is in ASCII. Remember that ASCII is a subset of UTF-16, just as it is a subset of UTF-8. Converting between UTF-8 and UTF-16 won't slow you down much if all your characters are ASCII, of course. Such a conversion is trivial - not much slower than a memcpy. /But/ - you're still making a copy, still allocating stuff off the heap and copying data from one place to another, and that's still overhead which you would have avoided had you used UTF-16 right through.
 (3) Object.d should contain the line:
 #    alias wchar[] string;

I'm not sure I like this.. will this hide details a programmer should be aware of?

It's just a strong hint. If aliases are bad then we shouldn't use them anywhere.
 (It could be made faster, but it can /never/ be made as fast as UTF-16). 
 So we make wchar[], not char[], the "standard", and hey presto, things 
 get faster

Qualification: For non ASCII only apps.

No, for /all/ apps. I don't see any reason why ASCII stored in wchar[]s would be any slower than ASCII stored in char[]s. Can you think of a reason why that would be so? ASCII is a subset of UTF-8 ASCII is a subset of UTF-16 where's the difference? The difference is space, not speed.
The fact that ICU has no char type suggests it's a bad choice for D, that 
is, if we want to assume they knew what they we're doing.

See http://oss.software.ibm.com/icu/userguide/strings.html for UCI's discussion on this.
Are there any 
complaints from developers about ICU anywhere, perhaps some digging for 
dirt would help make an objective decision here?

I don't know. I imagine so. People generally tend to complain about /everything/.
I'd like some more stats and figures, simply:
  - how many unicode characters are in the range < U+0800?

These figures are all from Unicode 4.0. Unicode now stands at 4.0.1, so these figures are out of date - but they'll give you the general idea. 1646
  - how many unicode from U+0800 >= x <= U+FFFF?

54014, of which 6400 are Private Use Characters
  - haw many unicode > U+FFFF?

176012, of which 131068 are Private Use Characters
Then, how commonly is each range used? I imagine this differs depending on 
exactly what you're doing.. basically when would you use characters in 
each range and how common is that task?

The biggest non-BMP chunks are: U+20000 to U+2A6D6 = CJK Compatibility Ideographs U+F0000 to U+FFFFD = Private Use U+100000 to U+10FFFFD = Private Use Compatibility Ideographs are not used in Unicode except for round-trip compatibility with legacy CJK character sets. Every single one of them is nothing but a compatibility alias for another character. The Private Use characters are not defined by Unicode (being reserved for private interchange between consenting parties). The remainder of the non-BMP (> U+FFFF) characters are: *) More CJK compatibility characters *) Old italic variants of ASCII characters *) Gothic letters (no idea what they are) *) The Deseret script (a dead language, so far as I know) *) Musical symbols *) Miscellaneous mathematical symbols *) Mathematical variants of ASCII and Greek letters *) "Tagged" variants of ASCII characters The math characters are used only in math. The tagged characters are used only in /one/ protocol (and in fact, some of those characters are not used at all, even in that protocol). None of these characters are likely to be found in general text, only in specialist applications. Here is the complete list of "blocks": 0000..007F; Basic Latin 0080..00FF; Latin-1 Supplement 0100..017F; Latin Extended-A 0180..024F; Latin Extended-B 0250..02AF; IPA Extensions 02B0..02FF; Spacing Modifier Letters 0300..036F; Combining Diacritical Marks 0370..03FF; Greek 0400..04FF; Cyrillic 0530..058F; Armenian 0590..05FF; Hebrew 0600..06FF; Arabic 0700..074F; Syriac 0780..07BF; Thaana 0900..097F; Devanagari 0980..09FF; Bengali 0A00..0A7F; Gurmukhi 0A80..0AFF; Gujarati 0B00..0B7F; Oriya 0B80..0BFF; Tamil 0C00..0C7F; Telugu 0C80..0CFF; Kannada 0D00..0D7F; Malayalam 0D80..0DFF; Sinhala 0E00..0E7F; Thai 0E80..0EFF; Lao 0F00..0FFF; Tibetan 1000..109F; Myanmar 10A0..10FF; Georgian 1100..11FF; Hangul Jamo 1200..137F; Ethiopic 13A0..13FF; Cherokee 1400..167F; Unified Canadian Aboriginal Syllabics 1680..169F; Ogham 16A0..16FF; Runic 1780..17FF; Khmer 1800..18AF; Mongolian 1E00..1EFF; Latin Extended Additional 1F00..1FFF; Greek Extended 2000..206F; General Punctuation 2070..209F; Superscripts and Subscripts 20A0..20CF; Currency Symbols 20D0..20FF; Combining Marks for Symbols 2100..214F; Letterlike Symbols 2150..218F; Number Forms 2190..21FF; Arrows 2200..22FF; Mathematical Operators 2300..23FF; Miscellaneous Technical 2400..243F; Control Pictures 2440..245F; Optical Character Recognition 2460..24FF; Enclosed Alphanumerics 2500..257F; Box Drawing 2580..259F; Block Elements 25A0..25FF; Geometric Shapes 2600..26FF; Miscellaneous Symbols 2700..27BF; Dingbats 2800..28FF; Braille Patterns 2E80..2EFF; CJK Radicals Supplement 2F00..2FDF; Kangxi Radicals 2FF0..2FFF; Ideographic Description Characters 3000..303F; CJK Symbols and Punctuation 3040..309F; Hiragana 30A0..30FF; Katakana 3100..312F; Bopomofo 3130..318F; Hangul Compatibility Jamo 3190..319F; Kanbun 31A0..31BF; Bopomofo Extended 3200..32FF; Enclosed CJK Letters and Months 3300..33FF; CJK Compatibility 3400..4DB5; CJK Unified Ideographs Extension A 4E00..9FFF; CJK Unified Ideographs A000..A48F; Yi Syllables A490..A4CF; Yi Radicals AC00..D7A3; Hangul Syllables D800..DB7F; High Surrogates DB80..DBFF; High Private Use Surrogates DC00..DFFF; Low Surrogates E000..F8FF; Private Use F900..FAFF; CJK Compatibility Ideographs FB00..FB4F; Alphabetic Presentation Forms FB50..FDFF; Arabic Presentation Forms-A FE20..FE2F; Combining Half Marks FE30..FE4F; CJK Compatibility Forms FE50..FE6F; Small Form Variants FE70..FEFE; Arabic Presentation Forms-B FEFF..FEFF; Specials FF00..FFEF; Halfwidth and Fullwidth Forms FFF0..FFFD; Specials 10300..1032F; Old Italic 10330..1034F; Gothic 10400..1044F; Deseret 1D000..1D0FF; Byzantine Musical Symbols 1D100..1D1FF; Musical Symbols 1D400..1D7FF; Mathematical Alphanumeric Symbols 20000..2A6D6; CJK Unified Ideographs Extension B 2F800..2FA1F; CJK Compatibility Ideographs Supplement E0000..E007F; Tags F0000..FFFFD; Private Use 100000..10FFFD; Private Use
It used to be that ASCII < U+0800 was the most common, it still may be, 
but I can see that it's not the future, the future is unicode.

Most common, perhaps, if you limit your text to Latin, Greek, Hebrew and Russian. But not otherwise.
That said, would the introduction of char to Java give you anything? 
perhaps.. it would allow you to write an app that only deals with ASCII 
(chars < U+0800) more space efficiently, correct?

Only chars < U+0080 (not U+0800) would be more space efficient in UTF-8. Between U+0080 and U+07FFF they both need two bytes. From U+0800 upwards, UTF-8 needs three bytes where UTF-16 needs two. How am I doing on the convincing front? I'd still go for: *) wchar[] for everything in Phobos and everything DMD-generated; *) ditch the char *) lossless implicit conversion between all remaining D string types Arcane Jill
Aug 27 2004
next sibling parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgmr7f$1b90$1 digitaldaemon.com>, Arcane Jill says...

10400..1044F; Deseret
*) The Deseret script (a dead language, so far as I know)

"The Deseret Alphabet was designed as an alternative to the Latin alphabet for writing the English language. It was developed during the 1850s by The Church of Jesus Christ of Latter-day Saints (also known as the "Mormon" or LDS Church) under the guidance of Church President Brigham Young (1801-1877). Brigham Young's secretary, George D. Watt, was among the designers of the Deseret Alphabet." "The LDS Church published four books using the Deseret Alphabet" # ^^^^^^^^^^ See http://www.molossia.org/alphabet.html Jill
Aug 27 2004
prev sibling next sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgmr7f$1b90$1 digitaldaemon.com...
 UTF-16 is not slower that UTF-8, even for pure ASCII. An app in which all

 is ASCII is going to be just as fast in UTF-16 as it is in ASCII.

While the rest of your post has a great deal of merit, this bit here is just not true. A pure ASCII app written using UTF-16 will consume twice as much memory for its data, and there are a lot of operations on that data that will be correspondingly half as fast. Furthermore, you'll start swapping a lot sooner, and then performance takes a dive. It makes sense for Java, Javascript, and for languages where performance is not a top priority to standardize on one character type. But if D does not handle ASCII very efficiently, it will not have a chance at interesting the very performance conscious C/C++ programmers.
Aug 28 2004
next sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cgqlmm$2ui$1 digitaldaemon.com>, Walter says...
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgmr7f$1b90$1 digitaldaemon.com...
 UTF-16 is not slower that UTF-8, even for pure ASCII. An app in which all

 is ASCII is going to be just as fast in UTF-16 as it is in ASCII.

While the rest of your post has a great deal of merit, this bit here is just not true. A pure ASCII app written using UTF-16 will consume twice as much memory for its data, and there are a lot of operations on that data that will be correspondingly half as fast. Furthermore, you'll start swapping a lot sooner, and then performance takes a dive. It makes sense for Java, Javascript, and for languages where performance is not a top priority to standardize on one character type. But if D does not handle ASCII very efficiently, it will not have a chance at interesting the very performance conscious C/C++ programmers.

Agreed. Frankly, I've begun to wonder just what the purpose of this discussion is. I think it's already been agreed that none of the three char types should be removed from D, and it seems clear that there is no "default" char type. Is this a nomenclature issue? ie. that the UTF-8 type is named "char" and thus considered to be somehow more important than the others? Sean
Aug 28 2004
next sibling parent J C Calvarese <jcc7 cox.net> writes:
Sean Kelly wrote:

 In article <cgqlmm$2ui$1 digitaldaemon.com>, Walter says...
 
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgmr7f$1b90$1 digitaldaemon.com...

UTF-16 is not slower that UTF-8, even for pure ASCII. An app in which all

text
is ASCII is going to be just as fast in UTF-16 as it is in ASCII.

While the rest of your post has a great deal of merit, this bit here is just not true. A pure ASCII app written using UTF-16 will consume twice as much memory for its data, and there are a lot of operations on that data that will be correspondingly half as fast. Furthermore, you'll start swapping a lot sooner, and then performance takes a dive. It makes sense for Java, Javascript, and for languages where performance is not a top priority to standardize on one character type. But if D does not handle ASCII very efficiently, it will not have a chance at interesting the very performance conscious C/C++ programmers.

Agreed. Frankly, I've begun to wonder just what the purpose of this discussion is. I think it's already been agreed that none of the three char types should be removed from D, and it seems clear that there is no "default" char type. Is

I think the thread has gone somewhat off topics by this point. Apparently, a lot of people feel oppressed by ASCII. I much be a bad person since 7-bits is all I need most of the time. On a related note, the "performance of char vs wchar" recently degraded into enlightened comments along the lines of "You're a know-it-all American cowboy who discriminates against the all of the Chinese, Japanese, Indians, Russian, British, and members of the European Union in the world. And Mormons, too." Or something like that. Somehow these Unicode-related discussions bring out the best in people. :P
 this a nomenclature issue?  ie. that the UTF-8 type is named "char" and thus
 considered to be somehow more important than the others?
 
 
 Sean

-- Justin (a/k/a jcc7) http://jcc_7.tripod.com/d/
Aug 28 2004
prev sibling parent Matthias Becker <Matthias_member pathlink.com> writes:
Agreed.  Frankly, I've begun to wonder just what the purpose of this discussion
is.

The toString()-method has to return a strig, but in which format? That's what this is all about.
I think it's already been agreed that none of the three char types should
be removed from D, and it seems clear that there is no "default" char type.

Well the language can only use one type as return type for toString(). So there actualy IS a default character type. BTW, isn't the name "char" totaly misleading? char means character, but it can't hold a character it can only hold parts of it. This is confusing.
Is
this a nomenclature issue?  ie. that the UTF-8 type is named "char" and thus
considered to be somehow more important than the others?

Nope. -- Matthias Becker
Aug 29 2004
prev sibling parent reply stonecobra <scott stonecobra.com> writes:
Walter wrote:
 "Arcane Jill" <Arcane_member pathlink.com> wrote in message
 news:cgmr7f$1b90$1 digitaldaemon.com...
 
UTF-16 is not slower that UTF-8, even for pure ASCII. An app in which all

text
is ASCII is going to be just as fast in UTF-16 as it is in ASCII.

While the rest of your post has a great deal of merit, this bit here is just not true. A pure ASCII app written using UTF-16 will consume twice as much memory for its data, and there are a lot of operations on that data that will be correspondingly half as fast. Furthermore, you'll start swapping a lot sooner, and then performance takes a dive. It makes sense for Java, Javascript, and for languages where performance is not a top priority to standardize on one character type. But if D does not handle ASCII very efficiently, it will not have a chance at interesting the very performance conscious C/C++ programmers.

affinciandos, they will never create an array in D, because of the default initialization. :) so, ubyte[] it is <g> Seriously, is performance a concern for D? If it truly is, this should be able to be turned off, if I take ownership of the potential consequences, no? Scott
Aug 29 2004
next sibling parent Juanjo =?ISO-8859-15?Q?=C1lvarez?= <juanjuxNO SPAMyahoo.es> writes:
stonecobra wrote:

 affinciandos

What was that? :-D (I guess what you're trying to say what "aficionado").
Aug 29 2004
prev sibling next sibling parent reply "Ivan Senji" <ivan.senji public.srce.hr> writes:
"stonecobra" <scott stonecobra.com> wrote in message
news:cgu2ia$1vad$1 digitaldaemon.com...
 Walter wrote:
 "Arcane Jill" <Arcane_member pathlink.com> wrote in message
 news:cgmr7f$1b90$1 digitaldaemon.com...

UTF-16 is not slower that UTF-8, even for pure ASCII. An app in which



 text

is ASCII is going to be just as fast in UTF-16 as it is in ASCII.

While the rest of your post has a great deal of merit, this bit here is


 not true. A pure ASCII app written using UTF-16 will consume twice as


 memory for its data, and there are a lot of operations on that data that
 will be correspondingly half as fast. Furthermore, you'll start swapping


 lot sooner, and then performance takes a dive.

 It makes sense for Java, Javascript, and for languages where performance


 not a top priority to standardize on one character type. But if D does


 handle ASCII very efficiently, it will not have a chance at interesting


 very performance conscious C/C++ programmers.

affinciandos, they will never create an array in D, because of the default initialization. :) so, ubyte[] it is <g> Seriously, is performance a concern for D? If it truly is, this should be able to be turned off, if I take ownership of the potential consequences, no?

What about having some standard allocator for arrays so we can: char[] str = new(noinit) char[100];
 Scott

Aug 30 2004
parent Juanjo =?ISO-8859-15?Q?=C1lvarez?= <juanjuxNO SPAMyahoo.es> writes:
Ivan Senji wrote:

 What about having some standard allocator for arrays so we can:
 
 char[] str = new(noinit) char[100];

I like it!
Aug 30 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
"stonecobra" <scott stonecobra.com> wrote in message
news:cgu2ia$1vad$1 digitaldaemon.com...
 They won't worry about it, because if they are true performance
 affinciandos, they will never create an array in D, because of the
 default initialization.   :)  so, ubyte[] it is <g>

 Seriously, is performance a concern for D?  If it truly is, this should
 be able to be turned off, if I take ownership of the potential
 consequences, no?

You can allocate them with std.c.stdlib.malloc() or std.c.stdlib.alloca(), neither of which will do the initialization. Furthermore, std.c.stdlib.alloca(n), where n is a constant, is handled as a special optimization case, and will generate storage as part of the stack frame setup (so it's zero cost).
Aug 30 2004
next sibling parent stonecobra <scott stonecobra.com> writes:
Walter wrote:

 "stonecobra" <scott stonecobra.com> wrote in message
 news:cgu2ia$1vad$1 digitaldaemon.com...
 
They won't worry about it, because if they are true performance
affinciandos, they will never create an array in D, because of the
default initialization.   :)  so, ubyte[] it is <g>

Seriously, is performance a concern for D?  If it truly is, this should
be able to be turned off, if I take ownership of the potential
consequences, no?

You can allocate them with std.c.stdlib.malloc() or std.c.stdlib.alloca(), neither of which will do the initialization. Furthermore, std.c.stdlib.alloca(n), where n is a constant, is handled as a special optimization case, and will generate storage as part of the stack frame setup (so it's zero cost).

they'd do it now (no benefit to moving to d)? Scott
Aug 30 2004
prev sibling parent reply Juanjo =?ISO-8859-15?Q?=C1lvarez?= <juanjuxNO SPAMyahoo.es> writes:
Walter wrote:

 Seriously, is performance a concern for D?  If it truly is, this should
 be able to be turned off, if I take ownership of the potential
 consequences, no?

You can allocate them with std.c.stdlib.malloc() or std.c.stdlib.alloca(), neither of which will do the initialization. Furthermore, std.c.stdlib.alloca(n), where n is a constant, is handled as a special optimization case, and will generate storage as part of the stack frame setup (so it's zero cost).

And will they be garbage-collected? (just asking, I don't know.) Anyway it's an ugly interface to do it, maybe as has been suggested more standard allocators could be included for this case.
Aug 30 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"Juanjo Álvarez" <juanjuxNO SPAMyahoo.es> wrote in message
news:ch0ajb$4d$1 digitaldaemon.com...
 Walter wrote:
 Seriously, is performance a concern for D?  If it truly is, this should
 be able to be turned off, if I take ownership of the potential
 consequences, no?



 neither of which will do the initialization. Furthermore,
 std.c.stdlib.alloca(n), where n is a constant, is handled as a special
 optimization case, and will generate storage as part of the stack frame
 setup (so it's zero cost).


malloc()? No, that will need an explicit call to free(). alloca()? No, that gets deallocated anyway when the function exits
 Anyway it's
 an ugly interface to do it, maybe as has been suggested more standard
 allocators could be included for this case.

You can also overload operators new and delete on a per-class basis, and provide a custom allocator.
Aug 30 2004
next sibling parent reply "Ivan Senji" <ivan.senji public.srce.hr> writes:
"Walter" <newshound digitalmars.com> wrote in message
news:ch0ce4$11t$1 digitaldaemon.com...
 "Juanjo Álvarez" <juanjuxNO SPAMyahoo.es> wrote in message
 news:ch0ajb$4d$1 digitaldaemon.com...
 Walter wrote:
 Seriously, is performance a concern for D?  If it truly is, this




 be able to be turned off, if I take ownership of the potential
 consequences, no?



 neither of which will do the initialization. Furthermore,
 std.c.stdlib.alloca(n), where n is a constant, is handled as a special
 optimization case, and will generate storage as part of the stack



 setup (so it's zero cost).


malloc()? No, that will need an explicit call to free().

But this would be a good way to get back to good old memory leaks from forgetting to free something. This is something that a high level language like D should try to help with. Isn't there a way to extend the syntax to enable us to allocate uninitialised arrays when someone explicitly wants to do that?
 alloca()? No, that gets deallocated anyway when the function exits

 Anyway it's
 an ugly interface to do it, maybe as has been suggested more standard
 allocators could be included for this case.

You can also overload operators new and delete on a per-class basis, and provide a custom allocator.

But not for arrays. This would mean too much wraping.
Aug 31 2004
parent reply Sean Kelly <sean f4.ca> writes:
In article <ch1ulm$ori$1 digitaldaemon.com>, Ivan Senji says...
"Walter" <newshound digitalmars.com> wrote in message
news:ch0ce4$11t$1 digitaldaemon.com...

 malloc()? No, that will need an explicit call to free().

But this would be a good way to get back to good old memory leaks from forgetting to free something. This is something that a high level language like D should try to help with. Isn't there a way to extend the syntax to enable us to allocate uninitialised arrays when someone explicitly wants to do that?

How about a smart pointer? I admit that the syntax wouldn't be quite as nice as in C++, but the basic implementation should be the same. Sean
Aug 31 2004
parent Nick <Nick_member pathlink.com> writes:
In article <ch2a1j$uqi$1 digitaldaemon.com>, Sean Kelly says...

 malloc()? No, that will need an explicit call to free().

But this would be a good way to get back to good old memory leaks from forgetting to free something. This is something that a high level language like D should try to help with.

How about a smart pointer? I admit that the syntax wouldn't be quite as nice as in C++, but the basic implementation should be the same.

How about just using a buffer class that clears up it's mess when collected? Seems to me like a simple solution that would cover most uses for uninitialized buffers. I have written an example here: http://folk.uio.no/mortennk/d/array/membuffer.d and used it here: http://folk.uio.no/mortennk/d/array/uninitarray.d Nick
Aug 31 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <ch0ce4$11t$1 digitaldaemon.com>, Walter says...

 And will they be garbage-collected? (just asking, I don't know.)

malloc()? No, that will need an explicit call to free(). alloca()? No, that gets deallocated anyway when the function exits

..but is forbidden from having a destructor (even if you use alloca() via a custom allocator) because you can't have destructable objects on the stack.
 Anyway it's
 an ugly interface to do it, maybe as has been suggested more standard
 allocators could be included for this case.

You can also overload operators new and delete on a per-class basis, and provide a custom allocator.

..which of course will /also/ not be garbage collected. (I've been down this road before). Arcane Jill
Aug 31 2004
next sibling parent Sean Kelly <sean f4.ca> writes:
In article <ch22nn$qtf$1 digitaldaemon.com>, Arcane Jill says...
In article <ch0ce4$11t$1 digitaldaemon.com>, Walter says...

 And will they be garbage-collected? (just asking, I don't know.)

malloc()? No, that will need an explicit call to free(). alloca()? No, that gets deallocated anyway when the function exits

..but is forbidden from having a destructor (even if you use alloca() via a custom allocator) because you can't have destructable objects on the stack.

You can't? Why not?
 Anyway it's
 an ugly interface to do it, maybe as has been suggested more standard
 allocators could be included for this case.

You can also overload operators new and delete on a per-class basis, and provide a custom allocator.

..which of course will /also/ not be garbage collected. (I've been down this road before).

I'm not sure if it would work, but could you use gc.malloc to allocate heap memory in operator new and then not provide an operator delete? Sean
Aug 31 2004
prev sibling parent "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:ch22nn$qtf$1 digitaldaemon.com...
 In article <ch0ce4$11t$1 digitaldaemon.com>, Walter says...

 And will they be garbage-collected? (just asking, I don't know.)

malloc()? No, that will need an explicit call to free(). alloca()? No, that gets deallocated anyway when the function exits

..but is forbidden from having a destructor (even if you use alloca() via

 custom allocator) because you can't have destructable objects on the

If you make it an auto class, you can.
Aug 31 2004
prev sibling parent Regan Heath <regan netwin.co.nz> writes:
On Fri, 27 Aug 2004 08:26:23 +0000 (UTC), Arcane Jill 
<Arcane_member pathlink.com> wrote:
 In article <opsdc4sgsi5a2sq9 digitalmars.com>, Regan Heath says...
 After all, I don't mind if my compile is a little slower if it means my
 app is faster.

UTF-16 is not slower that UTF-8, even for pure ASCII. An app in which all text is ASCII is going to be just as fast in UTF-16 as it is in ASCII. Remember that ASCII is a subset of UTF-16, just as it is a subset of UTF-8. Converting between UTF-8 and UTF-16 won't slow you down much if all your characters are ASCII, of course. Such a conversion is trivial - not much slower than a memcpy. /But/ - you're still making a copy, still allocating stuff off the heap and copying data from one place to another, and that's still overhead which you would have avoided had you used UTF-16 right through.

You're right.. I would argue however that space == speed when you start to run out, which happens 2 times faster if you use wchar (and ASCII only), right? the overall efficiency of a program is made up of both it's space and cpu requirements, some times you will need to or want to lessen the space requirements.
 (3) Object.d should contain the line:
 #    alias wchar[] string;

I'm not sure I like this.. will this hide details a programmer should be aware of?

It's just a strong hint. If aliases are bad then we shouldn't use them anywhere.

I wasn't suggesting aliases were bad. Aliases that serve to make type declarations clearer are very useful, they make code clearer. This alias just renames a type, so now it has 2 names, this will likely cause some confusion. I think we can suggest a type without it.
 (It could be made faster, but it can /never/ be made as fast as 
 UTF-16).
 So we make wchar[], not char[], the "standard", and hey presto, things
 get faster

Qualification: For non ASCII only apps.

No, for /all/ apps. I don't see any reason why ASCII stored in wchar[]s would be any slower than ASCII stored in char[]s. Can you think of a reason why that would be so? ASCII is a subset of UTF-8 ASCII is a subset of UTF-16 where's the difference? The difference is space, not speed.

Correct, but space == speed (as above).
 The fact that ICU has no char type suggests it's a bad choice for D, 
 that
 is, if we want to assume they knew what they we're doing.

See http://oss.software.ibm.com/icu/userguide/strings.html for UCI's discussion on this.

Thanks.
 Are there any
 complaints from developers about ICU anywhere, perhaps some digging for
 dirt would help make an objective decision here?

I don't know. I imagine so. People generally tend to complain about /everything/.

<g> true, too true...
 I'd like some more stats and figures, simply:
  - how many unicode characters are in the range < U+0800?

These figures are all from Unicode 4.0. Unicode now stands at 4.0.1, so these figures are out of date - but they'll give you the general idea. 1646
  - how many unicode from U+0800 >= x <= U+FFFF?

54014, of which 6400 are Private Use Characters
  - haw many unicode > U+FFFF?

176012, of which 131068 are Private Use Characters
 Then, how commonly is each range used? I imagine this differs depending 
 on
 exactly what you're doing.. basically when would you use characters in
 each range and how common is that task?


..thanks for the lists/figures..
 It used to be that ASCII < U+0800 was the most common, it still may be,
 but I can see that it's not the future, the future is unicode.

Most common, perhaps, if you limit your text to Latin, Greek, Hebrew and Russian. But not otherwise.

So would you say most common worldwide then? It may be due to the fact that I only speak english but I see many more english-only programs than (pick a language)-only programs. Ignoring those applications that come in several languages (as all the big ones do).
 That said, would the introduction of char to Java give you anything?
 perhaps.. it would allow you to write an app that only deals with ASCII
 (chars < U+0800) more space efficiently, correct?

Only chars < U+0080 (not U+0800) would be more space efficient in UTF-8.

Yes, what if they are all you want/need/(are going) to use...
 Between
 U+0080 and U+07FFF they both need two bytes. From U+0800 upwards, UTF-8 
 needs
 three bytes where UTF-16 needs two.

 How am I doing on the convincing front?

You still have work to do <g>
 I'd still go for:
 *) wchar[] for everything in Phobos and everything DMD-generated;

What if you know you're only going to need ASCII(utf-8), what if all your data is going to be in ASCII(utf-8), won't you want your static strings in ASCII(utf-8) also, to cut down on transcoding?
 *) ditch the char

I don't see the point in this. char[] is still useful regardless which type we 'promote' as the best type to use for internationalized strings. If it were removed, then dchar should be removed for the same reason, and both types would have to be implemented in ubyte[] and int[] instead. Coincidentally this is what the ICU have done, I quote... "UTF-8 and UTF-32 are supported with converters (ucnv.h), macros (utf.h), and convenience functions (ustring.h), but not directly as string encoding forms for most APIs." I'm not convinced removing them is more useful than keeping them but having implicit transcoding.
 *) lossless implicit conversion between all remaining D string types

Here we agree, this would make life much easier. Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 29 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgk32j$1fj$1 digitaldaemon.com...
 (2) String literals such as "hello world" should be interpretted as

 char[].

Actually, string literals are already interpreted as char[], wchar[], or dchar[] depending on the context they appear in. The compiler implicitly does a UTF conversion on them as necessary. If you have an overload based on char[] vs wchar[] vs dchar[] and pass a string literal, it should result in an ambiguity error. The only place it would default to char[] would be when it is passed as a ... argument to a variadic function.
Aug 28 2004
parent reply Andy Friesen <andy ikagames.com> writes:
Walter wrote:
 "Arcane Jill" <Arcane_member pathlink.com> wrote in message
 news:cgk32j$1fj$1 digitaldaemon.com...
 
(2) String literals such as "hello world" should be interpretted as

wchar[], not
char[].

Actually, string literals are already interpreted as char[], wchar[], or dchar[] depending on the context they appear in. The compiler implicitly does a UTF conversion on them as necessary. If you have an overload based on char[] vs wchar[] vs dchar[] and pass a string literal, it should result in an ambiguity error.

Is there any chance that this could be adjusted somehow? I know the point is to avoid all the complications that the C++ appoarch entails, but this has a way of throwing a wrench in any interface that wants to handle all three. Presently, we're given a choice: either handle all three char types and therefore demand ugly casts on all string literal arguments, or only handle one and force conversions that aren't necessarily required or desired by either the caller or the callee. If say, in the case of an ambiguity, a string literal were assumed to be of the smallest char type for which a match exists, the code would compile and, in almost all cases, do the right thing. It does complicate the rules some, but it seems preferable to the current dilemma. -- andy
Aug 28 2004
parent reply Sean Kelly <sean f4.ca> writes:
In article <cgqmpo$387$1 digitaldaemon.com>, Andy Friesen says...
I know the point is to avoid all the complications that the C++ appoarch 
entails, but this has a way of throwing a wrench in any interface that 
wants to handle all three.

I'm inclined to agree, though I'm wary of making char types a special case for overload resolution. Perhaps a prefix to indicate type? c"" // utf-8 w"" // utf-16 d"" // utf-32 Still not ideal, but it would require less typing :/ Sean
Aug 28 2004
parent "Walter" <newshound digitalmars.com> writes:
"Sean Kelly" <sean f4.ca> wrote in message
news:cgqnjm$3eq$1 digitaldaemon.com...
 In article <cgqmpo$387$1 digitaldaemon.com>, Andy Friesen says...
I know the point is to avoid all the complications that the C++ appoarch
entails, but this has a way of throwing a wrench in any interface that
wants to handle all three.

I'm inclined to agree, though I'm wary of making char types a special case

 overload resolution.  Perhaps a prefix to indicate type?

 c"" // utf-8
 w"" // utf-16
 d"" // utf-32

 Still not ideal, but it would require less typing :/

I thought of the prefix approach, like C uses, but it just seemed redundant for the odd case where a cast(char[]) will do.
Aug 28 2004
prev sibling parent "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgh8vi$1o7k$1 digitaldaemon.com...
 It's not invalid as such, it's just that the return type of an overloaded
 function has to be "covariant" with the return type of the function it's
 overloading. So it's a compile error /now/. But if dchar[] and char[] were

 considered mutually covariant then this would magically start to compile.

That would be nice, but I don't see how to technically make that work.
Aug 28 2004
prev sibling next sibling parent reply "Carlos Santander B." <carlos8294 msn.com> writes:
"antiAlias" <fu bar.com> escribió en el mensaje
news:cgg1mi$14l2$1 digitaldaemon.com
| A a; // class instances ...
| B b;
| C c;
|
| dchar[] message = c ~ b ~ a;

I have a question regarding this: what if A, B, and C were like this?

//////////////////////////
class A
{
    ... opCat_r (B b) { ... }
    ...
}

class B
{
    ... opCat (A a) { ... }
    ... opCat_r (C c) { ... }
    ...
}

class C
{
    ... opCat (B b) { ... }
    ...
}
//////////////////////////

How would "c ~ b ~ a" work with the proposed automatic call to .toString?

-----------------------
Carlos Santander Bernal
Aug 24 2004
parent reply Regan Heath <regan netwin.co.nz> writes:
On Tue, 24 Aug 2004 21:31:56 -0500, Carlos Santander B. 
<carlos8294 msn.com> wrote:
 "antiAlias" <fu bar.com> escribió en el mensaje
 news:cgg1mi$14l2$1 digitaldaemon.com
 | A a; // class instances ...
 | B b;
 | C c;
 |
 | dchar[] message = c ~ b ~ a;

 I have a question regarding this: what if A, B, and C were like this?

 //////////////////////////
 class A
 {
     ... opCat_r (B b) { ... }
     ...
 }

 class B
 {
     ... opCat (A a) { ... }
     ... opCat_r (C c) { ... }
     ...
 }

 class C
 {
     ... opCat (B b) { ... }
     ...
 }
 //////////////////////////

 How would "c ~ b ~ a" work with the proposed automatic call to .toString?

I assumed opCat's parameter would have to be char[], wchar[] or dchar[], as would it's return value. eg. class B { char[] opCat(char[] rhs){} } given implicit transcoding you could then say. char[] c; wchar[] w; dchar[] d; B b = new B(); char[] p; p = b ~ c; p = b ~ w; p = b ~ d; Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
parent reply "Carlos Santander B." <carlos8294 msn.com> writes:
"Regan Heath" <regan netwin.co.nz> escribió en el mensaje
news:opsc9un10r5a2sq9 digitalmars.com
| I assumed opCat's parameter would have to be char[], wchar[] or dchar[],
| as would it's return value. eg.
|

I don't see why it has to be only that way. ~ is the concatenation operator, so
I could define:

class Set(T)
{
    Set opCat(T newElem) { ... }
}

And expect it to work the way I want it. opCat is not only for strings. And it
shouldn't be.

| class B
| {
|    char[] opCat(char[] rhs){}
| }
|
| given implicit transcoding you could then say.
|
| char[]  c;
| wchar[] w;
| dchar[] d;
|
| B b = new B();
|
| char[] p;
|
| p = b ~ c;
| p = b ~ w;
| p = b ~ d;
|
| Regan
|

-----------------------
Carlos Santander Bernal
Aug 25 2004
parent Arcane Jill <Arcane_member pathlink.com> writes:
This is irrelevant. opCat() does not need to do anything special for D strings
to work, whether we go the wchar[] route or the implicit conversion route. For
wchar[]s, it already works. If we go for implicit converion, then the three
different kinds of D string would be regarded as covariant by the D compiler, so
expressions of the form (wchar[] ~ dchar[]) would be handled by the type
promotion system, not by opCat() - just as (float + int) is handled now.

Jill



In article <cgi5q2$254f$1 digitaldaemon.com>, Carlos Santander B. says...
"Regan Heath" <regan netwin.co.nz> escribió en el mensaje
news:opsc9un10r5a2sq9 digitalmars.com
| I assumed opCat's parameter would have to be char[], wchar[] or dchar[],
| as would it's return value. eg.
|

I don't see why it has to be only that way. ~ is the concatenation operator, so
I could define:

class Set(T)
{
    Set opCat(T newElem) { ... }
}

And expect it to work the way I want it. opCat is not only for strings. And it
shouldn't be.

| class B
| {
|    char[] opCat(char[] rhs){}
| }
|
| given implicit transcoding you could then say.
|
| char[]  c;
| wchar[] w;
| dchar[] d;
|
| B b = new B();
|
| char[] p;
|
| p = b ~ c;
| p = b ~ w;
| p = b ~ d;
|
| Regan
|

-----------------------
Carlos Santander Bernal

Aug 25 2004
prev sibling parent reply Ben Hinkle <bhinkle4 juno.com> writes:
 
 The other aspect involved here is that of string-concatenation. D cannot
 have more that one return type for toString() as you know. It's fixed at
 char[]. If string concatenation uses the toString() method to retrieve its
 components (as is being proposed elsewhere), then there will be multiple,
 redundant, implicit conversions going on where the string really wanted to
 be dchar[] in the first place. That is:
 
 A a; // class instances ...
 B b;
 C c;
 
 dchar[] message = c ~ b ~ a;
 
 Under the proposed "implicit" scheme, if each toString() of A, B, and C
 wish to return dchar[], then each concatenation causes an implicit
 conversion/encoding from each dchar[] to char[] (for the toString()
 return). Then another full conversion/decoding is performed back to the
 dchar[] assignment once each has been concatenated. This is like the
 Wintel 'plot' for selling more cpu's :-)
 
 Doing this manually, one would forego the toString() altogether:
 
 dchar[] message = c.getString() ~ b.getString() ~ a.getString();
 
 ... where getString() is a programmer-specific idiom to return the
 (natural) dchar[] for these classes, and we carefully avoided all those
 darned implicit-conversions. However, which approach do you think people
 will use? My guess is that D may become bogged down in conversion hell
 over such things.

"Conversion hell" will exist any time three standards are in use. It doesn't matter how those standards are wrapped up - implicit, explicit, String class or whatever. That's why we all win by agreeing on one standard and trying to stick to it. In D now life is peachy in char[] land, slightly less peachy in wchar[] and dchar[] land. I don't think there's any way to make life peachy for all three cases.
 So, to answer your question:
 What I'm /for/ is not covering up these types of issues with blanket-style
 implicit conversions. Something more constructive (and with a little more
 forethought) needs to be done.

Aug 24 2004
parent Regan Heath <regan netwin.co.nz> writes:
On Tue, 24 Aug 2004 22:53:47 -0400, Ben Hinkle <bhinkle4 juno.com> wrote:

<snip>

 "Conversion hell" will exist any time three standards are in use. It 
 doesn't
 matter how those standards are wrapped up - implicit, explicit, String
 class or whatever. That's why we all win by agreeing on one standard and
 trying to stick to it. In D now life is peachy in char[] land, slightly
 less peachy in wchar[] and dchar[] land. I don't think there's any way to
 make life peachy for all three cases.

Lets assume implicit transcoding is implemented, why wouldn't that make life peachy in all 3? Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgfen9$rhq$1 digitaldaemon.com...
 Phobos UTF-8 code could be faster, I grant you. But perhaps it will be in

 next release. We're only talking about a tiny number of functions here,

 all.

My first goal with std.utf is to make it work right. Then comes the optimize-the-heck-out-of-it. std.utf is shaping up to be a core dependency for D, so making it run as fast as possible is worthwhile. Any suggestions?
Aug 28 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgqju0$2b4$1 digitaldaemon.com>, Walter says...

My first goal with std.utf is to make it work right. Then comes the
optimize-the-heck-out-of-it. std.utf is shaping up to be a core dependency
for D, so making it run as fast as possible is worthwhile. Any suggestions?

It's only really UTF-8 decoding that's complicated. All the rest are pretty easy, even UTF-8 encoding (as I'm sure you know). The approach I took in the code sample I posted here a while back was to read the first byte (which will be the /only/ byte, in the case of ASCII) and use it as the index into a lookup table. A byte has only 256 possible values - 128 after you've eliminated ASCII chars, and you can look up both the sequence length (or 0 for illegal first-bytes), and the initial value for the (dchar) accumulator. Then you just get six more bits from each of the remaining bytes (after ensuring that the bit pattern is 10xxxxxx). This approach will fail to catch precisely /two/ non-shortest cases, so you have to test for them explicitly. Finally, you make sure that the resulting dchar is not a forbidden value. (From memory, I think that in some cases your current code checks for errors which can never happen, such as checking for a non-shortest 5+ byte sequence /after/ overlong sequences have already been eliminated). You could go further. Kris has mentioned that heap allocation is slow. Presumably, you could start off by allocating a single char buffer of length 3*N (if input=wchars) or 4*N (if input=dchars), decoding into it, and then reducing its length. (Of course, the excess then won't be released). (But never let an invalid input go unnoticed. That would be one optimization too many). In article <cgqju1$2b4$2 digitaldaemon.com>, Walter says...
 It's not invalid as such, it's just that the return type of an overloaded
 function has to be "covariant" with the return type of the function it's
 overloading. So it's a compile error /now/. But if dchar[] and char[] were

 considered mutually covariant then this would magically start to compile.

That would be nice, but I don't see how to technically make that work.

You're right. It wouldn't work. Well, this is the sense in which D does have a "default" string type. It is the case where we see clearly that char[] has special privilege. Currently, Object - and therefore /everything/ - defines a function toString() which returns a char[]. It is not possible to overload this with a function returning wchar[], even if wchar[] would be more appropriate. This affects /all/ objects. What to do about it? Hmmm.... You could change the Object.toString to: # void toString(out char[] s) { /* implementation as before */ } # void toString(out wchar[] s} { char[] t; toString(t); s = t; } # void toString(out dchar[] s} { char[] t; toString(t); s = t; } with the latter two employing implicit conversion for the "s = t;" part. Subclasses of Object which overloaded only toString(out char[]) would get the other three for free. But subclasses of Object which decided to go a bit further could return a wchar[] or a dchar[] directly to cut down on conversions.
Actually, string literals are already interpreted as char[], wchar[], or
dchar[] depending on the context they appear in. The compiler implicitly
does a UTF conversion on them as necessary.

# void main() # { # dchar[] s = "hello"; # dchar[] t = s ~ " world"; # } The error is: incompatible types for ((s) ~ (" world")): 'dchar[]' and 'char[]' But yes - it works /nearly/ always, and that's cool. The above case would be covered by implicit conversion, of course (although that would defer the conversion from compile-time to run-time).
If you have an overload based on
char[] vs wchar[] vs dchar[] and pass a string literal, it should result in
an ambiguity error.

Ah! That's the bit I didn't know. I was wondering how that context thing would work, given that signature matching happens /after/ the evaluation of the function's arguments' types. You could fix this by allowing explicit UTF-8, UTF-16 and UTF-32 literals. Sean suggested c"", w"" and d"" (and similarly for char literals). That would fix it.
 UTF-16 is not slower that UTF-8, even for pure ASCII. An app in which all

 is ASCII is going to be just as fast in UTF-16 as it is in ASCII.

While the rest of your post has a great deal of merit, this bit here is just not true. A pure ASCII app written using UTF-16 will consume twice as much memory for its data, and there are a lot of operations on that data that will be correspondingly half as fast.

That's true. Guess I got a bit carried away there. I was thinking that statements like "c = *p++;" would compile to just one machine code instruction regardless of the data width, and that the byte-wide version wouldn't necessarily be the fastest. But I forgot about all the initializing and copying that you also have to do. Arcane Jill
Aug 28 2004
parent reply J C Calvarese <jcc7 cox.net> writes:
Arcane Jill wrote:
 In article <cgqju0$2b4$1 digitaldaemon.com>, Walter says...

 In article <cgqju1$2b4$2 digitaldaemon.com>, Walter says...
 
It's not invalid as such, it's just that the return type of an overloaded
function has to be "covariant" with the return type of the function it's
overloading. So it's a compile error /now/. But if dchar[] and char[] were

to be
considered mutually covariant then this would magically start to compile.

That would be nice, but I don't see how to technically make that work.

You're right. It wouldn't work. Well, this is the sense in which D does have a "default" string type. It is the case where we see clearly that char[] has special privilege. Currently, Object - and therefore /everything/ - defines a function toString() which returns a char[]. It is not possible to overload this with a function returning wchar[], even if wchar[] would be more appropriate. This affects /all/ objects. What to do about it? Hmmm.... You could change the Object.toString to: # void toString(out char[] s) { /* implementation as before */ } # void toString(out wchar[] s} { char[] t; toString(t); s = t; } # void toString(out dchar[] s} { char[] t; toString(t); s = t; } with the latter two employing implicit conversion for the "s = t;" part. Subclasses of Object which overloaded only toString(out char[]) would get the other three for free. But subclasses of Object which decided to go a bit further could return a wchar[] or a dchar[] directly to cut down on conversions.

Could we ditch toString and replace the functionality with: toUtf8(), toUtf16(), and toUtf32() or toCharStr(), toWCharStr(), and toDCharStr() Usually, the person writing the object could define one and the other two would call conversions. There's probably some reason why this wouldn't work, but it's just such a pleasant idea to me that I was forced to share it. -- Justin (a/k/a jcc7) http://jcc_7.tripod.com/d/
Aug 28 2004
parent reply Ben Hinkle <bhinkle4 juno.com> writes:
J C Calvarese wrote:

 Arcane Jill wrote:
 In article <cgqju0$2b4$1 digitaldaemon.com>, Walter says...

 In article <cgqju1$2b4$2 digitaldaemon.com>, Walter says...
 
It's not invalid as such, it's just that the return type of an
overloaded function has to be "covariant" with the return type of the
function it's overloading. So it's a compile error /now/. But if dchar[]
and char[] were

to be
considered mutually covariant then this would magically start to
compile.

That would be nice, but I don't see how to technically make that work.

You're right. It wouldn't work. Well, this is the sense in which D does have a "default" string type. It is the case where we see clearly that char[] has special privilege. Currently, Object - and therefore /everything/ - defines a function toString() which returns a char[]. It is not possible to overload this with a function returning wchar[], even if wchar[] would be more appropriate. This affects /all/ objects. What to do about it? Hmmm.... You could change the Object.toString to: # void toString(out char[] s) { /* implementation as before */ } # void toString(out wchar[] s} { char[] t; toString(t); s = t; } # void toString(out dchar[] s} { char[] t; toString(t); s = t; } with the latter two employing implicit conversion for the "s = t;" part. Subclasses of Object which overloaded only toString(out char[]) would get the other three for free. But subclasses of Object which decided to go a bit further could return a wchar[] or a dchar[] directly to cut down on conversions.

Could we ditch toString and replace the functionality with: toUtf8(), toUtf16(), and toUtf32() or toCharStr(), toWCharStr(), and toDCharStr() Usually, the person writing the object could define one and the other two would call conversions. There's probably some reason why this wouldn't work, but it's just such a pleasant idea to me that I was forced to share it.

Why is toString such a hot topic anyway? In Java end users hardly ever see the result of a toString. In D I can see things like toString(int) and toString(double) being seen by users but in general Foo.toString should just give a summary of the object - preferably short and easy to transcode. I wouldn't use toString for things like getting user strings out of text fields in a GUI or reading from a file. For those cases I would use another function name like TextBox.getText or File.readStringW. Those other functions can have char[] versions and wchar[] versions as desired. As an example of a "bad" toString see std.stream.Stream.toString. It will usually create a huge string. For classes like AJ's arbitrary sized Int toString should return the whole integer since that is the best summary of the object. So we should still allow toString to return arbitrarily long strings - we just need to be careful how toString is used.
Aug 28 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgra2k$udg$1 digitaldaemon.com>, Ben Hinkle says...

Why is toString such a hot topic anyway?

Ask yourself why toString() exists at all. What does D use it for? If it's an unnecessary, hardly-used function, then it should be removed from Object, because this is OOP, if something doesn't make sense for all Objects then it should not be defined for all Objects. On the other hand, if it /is/ necessary for all objects, it shouldn't be biased one way or the other.
but in general Foo.toString should
just give a summary of the object

Of the object's /value/, yes. So toString() only makes sense for objects which actually /have/ a value. I'm not sure if streams can be said to have a "value" in the sense that Ints do, so maybe it shouldn't be defined at all for streams.
As an
example of a "bad" toString see std.stream.Stream.toString. It will usually
create a huge string.

Yes. Now I'm starting to wonder what toString() is actually for, and whether implementing a three-function interface (Stringizable?) might be better than inheriting from Object.
For classes like AJ's arbitrary sized Int toString should return the whole
integer since that is the best summary of the object. So we should still
allow toString to return arbitrarily long strings - we just need to be
careful how toString is used.

Let's go back to Walter on this one. Walter - why does Object have a toString() function? In what way does D require or rely on it? How badly would D be affected if it didn't exist at all or if it were an interface? Jill
Aug 28 2004
next sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgrsnq$168a$1 digitaldaemon.com...
 Let's go back to Walter on this one. Walter - why does Object have a

 function? In what way does D require or rely on it? How badly would D be
 affected if it didn't exist at all or if it were an interface?

It's so when you pass an object to writef(), there's a way that it can be printed. But Ben is right, I don't see Object.toString() being used to generate very large strings, so any transcoding of it isn't going to be expensive overall.
Aug 29 2004
parent reply Matthias Becker <Matthias_member pathlink.com> writes:
news:cgrsnq$168a$1 digitaldaemon.com...
 Let's go back to Walter on this one. Walter - why does Object have a

 function? In what way does D require or rely on it? How badly would D be
 affected if it didn't exist at all or if it were an interface?

It's so when you pass an object to writef(), there's a way that it can be printed. But Ben is right, I don't see Object.toString() being used to generate very large strings, so any transcoding of it isn't going to be expensive overall.

I don't get it :( Why isn't it possible to use a Stringizeable interface instead? -- Matthias Becker
Aug 29 2004
parent "Walter" <newshound digitalmars.com> writes:
"Matthias Becker" <Matthias_member pathlink.com> wrote in message
news:cgs9b4$1c0h$1 digitaldaemon.com...
news:cgrsnq$168a$1 digitaldaemon.com...
 Let's go back to Walter on this one. Walter - why does Object have a

 function? In what way does D require or rely on it? How badly would D



 affected if it didn't exist at all or if it were an interface?

It's so when you pass an object to writef(), there's a way that it can be printed. But Ben is right, I don't see Object.toString() being used to generate very large strings, so any transcoding of it isn't going to be expensive overall.

I don't get it :( Why isn't it possible to use a Stringizeable interface instead?

It is possible. But I think every Object should have some basic functionality, and one of those is to be able to have itself pretty-printed. This is also nice for a potential D debugger - it can take advantage of toString() to produce a user-friendly representation of the class data.
Aug 30 2004
prev sibling next sibling parent Ben Hinkle <bhinkle4 juno.com> writes:
Arcane Jill wrote:

 In article <cgra2k$udg$1 digitaldaemon.com>, Ben Hinkle says...
 
Why is toString such a hot topic anyway?

Ask yourself why toString() exists at all. What does D use it for?

In Java (and I suppose C#) it is very handy when debugging to print an object out. One drawback with D's Object.toString is that the default implementation doesn't print the object's address (or in Java's case the hash code) so you can't distinguish one object from another. If anything I'd like to see some guidelines for toString so that the output is consistent across D. For example if the object doesn't have an obvious string representation, like a class Foo{int n; double d;} then the result of toString should have the form "[Foo n:0, d:0.0]" - possibly include the address or hash code in there. I think this is basically the format Java uses but I can't remember exactly. In general toString should avoid newlines.
 If it's an unnecessary, hardly-used function, then it should be removed
 from Object, because this is OOP, if something doesn't make sense for all
 Objects then it should not be defined for all Objects.
 
 On the other hand, if it /is/ necessary for all objects, it shouldn't be
 biased one way or the other.

It isn't necessary but it is nice to have around. Does it make sense for all objects? I guess that depends on one's viewpoint of "makes sense". I think printing the class and hash code makes sense. Maybe others don't.
but in general Foo.toString should
just give a summary of the object

Of the object's /value/, yes. So toString() only makes sense for objects which actually /have/ a value. I'm not sure if streams can be said to have a "value" in the sense that Ints do, so maybe it shouldn't be defined at all for streams.

See above comments about making sense and the default toString format.
As an
example of a "bad" toString see std.stream.Stream.toString. It will
usually create a huge string.

Yes. Now I'm starting to wonder what toString() is actually for, and whether implementing a three-function interface (Stringizable?) might be better than inheriting from Object.

That's an option, but it adds more work for users to make something stringizable - two of the stringizable functions will call the "real" one and wrap the result in toUTF8 etc. Is it worth it to do all that just for debugging?
For classes like AJ's arbitrary sized Int toString should return the whole
integer since that is the best summary of the object. So we should still
allow toString to return arbitrarily long strings - we just need to be
careful how toString is used.

Let's go back to Walter on this one. Walter - why does Object have a toString() function? In what way does D require or rely on it? How badly would D be affected if it didn't exist at all or if it were an interface? Jill

Aug 29 2004
prev sibling parent reply Berin Loritsch <bloritsch d-haven.org> writes:
Arcane Jill wrote:

 In article <cgra2k$udg$1 digitaldaemon.com>, Ben Hinkle says...
 
 
Why is toString such a hot topic anyway?

Ask yourself why toString() exists at all. What does D use it for? If it's an unnecessary, hardly-used function, then it should be removed from Object, because this is OOP, if something doesn't make sense for all Objects then it should not be defined for all Objects. On the other hand, if it /is/ necessary for all objects, it shouldn't be biased one way or the other.

There are a couple of uses I use toString() for (in Java apps): 1: debugging. Some useful info is derived when going through a graphical debugger (hovering over a variable will yield its toString() value). 2: easier inclusion in graphical lists. I might have an object encapsulating the ISO-3166 country codes (long name and id), but use the toString() to show the long name. That way in the drop down list I have everything I need to serialize the info to the DB. But that's just Java.... D doesn't have decent IDE integration at this time, nor does it have a standard GUI library that can do the #2 operation.
Aug 30 2004
parent reply Dave <Dave_member pathlink.com> writes:
In article <cgva1p$2gep$1 digitaldaemon.com>, Berin Loritsch says...
Arcane Jill wrote:

 In article <cgra2k$udg$1 digitaldaemon.com>, Ben Hinkle says...
 
 
Why is toString such a hot topic anyway?

Ask yourself why toString() exists at all. What does D use it for? If it's an unnecessary, hardly-used function, then it should be removed from Object, because this is OOP, if something doesn't make sense for all Objects then it should not be defined for all Objects. On the other hand, if it /is/ necessary for all objects, it shouldn't be biased one way or the other.

There are a couple of uses I use toString() for (in Java apps): 1: debugging. Some useful info is derived when going through a graphical debugger (hovering over a variable will yield its toString() value). 2: easier inclusion in graphical lists. I might have an object encapsulating the ISO-3166 country codes (long name and id), but use the toString() to show the long name. That way in the drop down list I have everything I need to serialize the info to the DB. But that's just Java.... D doesn't have decent IDE integration at this time, nor does it have a standard GUI library that can do the #2 operation.

(Also refering to the 'most European chars. are ASCII' post just ahead of this one) And because of their likely small size, even if they are filled with non-ASCII characters, neither of those uses will realistically cause a performance bottleneck if toString() is UTF-8 by 'default', right? No matter how efficient the memory management system is, if the 'default' is UTF-16 rather than UTF-8, most apps. will have to carry that extra de/allocation & initialization burden for no reason other than expediency or ignorance on the programmers part. Often twice the work and 1/2 the available memory for many of the same jobs. UTF-16 used everywhere is probably one reason why heavy-use Java server apps. have the reputation as 'memory-thrashers'. And those runtimes have the benefit of many man-years of development, use-cases and experimentation behind them. Since UTF-8 is the most efficient and adequate for most of D's currently forseeable uses, I say leave as is and put the 'burden' on the software that needs or benefits from other than UTF-8. IMO, D really needs to be a general performance winner in order to get a toe-hold in the current market.
Aug 30 2004
parent "Walter" <newshound digitalmars.com> writes:
"Dave" <Dave_member pathlink.com> wrote in message
news:ch033f$2u7u$1 digitaldaemon.com...
 UTF-16 used everywhere is probably one reason why heavy-use Java server

 have the reputation as 'memory-thrashers'. And those runtimes have the

 of many man-years of development, use-cases and experimentation behind

Many of Java's problems have gradually declined over time due to massive research and development efforts by a lot of very, very smart people. For example, Andy King has just pointed me to some research done by several people that focussed on improving some minor aspects of the Java garbage collector. D doesn't have a billion dollar budget <g>, and simply cannot afford to have problems that need such budgets to find solutions for.
 IMO, D really needs to be a general performance winner in order to get a
 toe-hold in the current market.

Right. And if D acquires an early reputation for being 'slow', it will never have a chance. Currently, D is *faster* than C++ on string benchmarks (see www.digitalmars.com/d/cppstrings.html), and it must be that way, as that seriously blunts criticisms aimed at D for the way it does strings relative to C++.
Aug 30 2004
prev sibling parent reply Regan Heath <regan netwin.co.nz> writes:
On Mon, 23 Aug 2004 23:05:45 -0700, antiAlias <fu bar.com> wrote:
 "Regan Heath" <regan netwin.co.nz> wrote in message
 news:opsc7u56zf5a2sq9 digitalmars.com...
 On Mon, 23 Aug 2004 18:29:54 -0700, antiAlias <fu bar.com> wrote:
 "Walter" <newshound digitalmars.com> wrote in message ...
 "Regan" wrote:
 There are two ways of looking at this, on one hand you're saying 



 should all 'paint' as that is consistent. However, on the other I'm


 they should all produce a 'valid' result. So my argument here is 



 when
 you cast you expect a valid result, much like casting a float to an

 does not just 'paint'.

 I am interested to hear your opinions on this idea.

I think your idea has a lot of merit. I'm certainly leaning that way.

On the one hand, this would be well served by an opCast() and/or opCast_r() on the primitive types; just the kind of thing suggested in a related thread (which talked about overloadable methods for primitive types).

True. However, what else should the opCast for char[] to dchar[] do except transcode it? What about opCast for int to dchar.. it seems to me there is only 1 choice about what to do, anything else would be operator abuse.
 On the other hand, we're talking transcoding here. Are you gonna' 

 this
 to UTF-8 only?

I hope not, I am hoping to see: | UTF-8 | UTF-16 | UTF-32 -------------------------------- UTF-8 | - + + UTF-16 | + - + UTF-32 | + + - (+ indicates transcoding occurs)

======================== And what happens when just one additional byte-oriented encoding is introduced? Perhaps UTF-7? Perhaps EBCDIC? The basic premise is flawed because there's no flexibility.

--------------------------------------- You'll have to check with Walter but I believe he has no plans to add another basic type to hold any specific encoding. Encodings other than UTF-x will be done with library functions and ubyte[], ushort[], uint[], ulong[].
 Then, since the source and destination will typically be of
 different sizes, do you then force all casts between these types to 

 the
 destination be an array reference rather than an instance?

I don't think transcoding makes any sense unless you're talking about a 'string' (i.e. char[], wchar[], or dchar[]) as opposed to a UTF-x fragment (i.e. char, wchar)

======================== We /are/ talking about arrays. Perhaps if that sentence had ended "array reference rather than an *array* instance?", it might have been more clear?

--------------------------------------- Or maybe I just miss-read or missunderstood it :)
 The point being made is that you would not be able to do anything like 
 this:

 char[15] dst;
 dchar[10] src;

 dst = cast(char[]) src;

 because there's no ability via a cast() to indicate how many items from 
 src
 were converted, and how many items in dst were populated. You are forced
 into this kind of thing:

 char[] dst;
 dchar[10] src;

 dst = cast(char[]) src;

--------------------------------------- How is that different to what we have to do now? dst = toUTF8(src); ?
 You see the distinction? It may be subtle to some, but it's a glaring
 imbalance to others. The lValue must always be a reference because it's
 gonna' be allocated dynamically to ensure all of the rValue will fit. In 
 the
 end, it's just far better to use functions/methods that provide the 
 feedback
 required so you can actually control what's going on (or such that a 
 library
 function can). That way, you're not restricted in the same fashion. We 
 don't
 need more asymmetry in D, and this just reeks of poor design, IMO.

--------------------------------------- Correct me if I'm wrong you're suggesting the use of a library function like this... bool toUTF8(char[] dst, dchar[] src) {} or similar, where the caller passes a buffer for the result, and has full control of the size /location/allocation etc of that buffer, correct? 1. The implicit casting idea does not prevent this. 2. The above function doesn't exist currently.. instead we have char[] toUTF8(dchar[] src) {} which is identical to what an implict cast would do.
 To drive this home, consider the decoding version (rather than the 
 encoding
 above):

 char[15] src;
 dchar[] dst;

 dst = cast(dchar[]) src;

 What happens when there's a partial character left undecoded at the end 
 of
 'src'?

--------------------------------------- How is that even possible? Assuming src is a 'valid' UTF-8 sequences, all the characters can and will be encoded into UTF-32 producing a 'valid' UTF-32 sequence. If 'src' _is_ invalid then an exception will be thrown much like the ones we currently have for invalid UTF sequences.
 There nothing here to tell you that you've got a dangly bit left at
 the end of the souce-buffer. It's gone. Poof! Any further decoding from 
 the
 same file/socket/whatever is henceforth trashed, because the ball has 
 been
 both dropped and buried. End of story.


 Having 3 types requiring manual transcoding between them _is_ a pain.

======================== It certainly is. That's why other languages try to avoid it at all costs. Having it done "generously" by the compiler is also a pain, inflexible, and likely expensive.

--------------------------------------- I'd argue that you're wrong about all but the last assertion above. Having it done by the compiler would not be: 1. a 'pain' - you wouldn't notice, and if you did and did not desire the behaviour you can manually convert just like you have to do now. 2. 'inflexible' - this idea does not preclude you doing things another way. It simply provides a default, which IMO is the sensible/correct thing to do. it is however more 'expensive' than the current situation, but, it's no more expensive than doing the conversion manually, which is what you currently have to do..
 There are many things a programmer should take
 responsibility for; transcoding comes under that umbrella because (a) 
 there
 can be subtle complexity involved and (b) it is relatively expensive to
 churn through text and convert it; particularly so with the Phobos utf-8
 code.

 What you appear to be suggesting is that this kind of thing should happen
 silently whilst one nonchalantly passes arguments around between methods.

--------------------------------------- Yes.
 That's insane, so I hope that's not what you're advocating. Java, for
 example, does that at one specific layer (I/O)

--------------------------------------- Which it can do because it only has one string type.
 , but you're apparently
 suggesting doing it at any old place! And several times over, just in 
 case
 it wasn't good enough the first time :-)

--------------------------------------- Yes, as we only want to go to an in-efficient type temporarily eg. Walters comment: "dchars are very convenient to work with, however, and make a great deal of sense as the temporary common intermediate form of all the conversions. I stress the temporary, though, as if you keep it around you'll start to notice the slowdowns it causes." reflects exactly what implicit conversion will give you, the ability to store things in memory the format you think is most efficient and dip in and out of utf-32 _if_ required. utf-32 is not 'required' for anything, it's simply 'convenient' for certain things, as AJ has frequently pointed out you can encode every Unicode character in all 3 formats UTF-8, UTF-16 and UTF-32.
 Sorry man. This is inane. D is
 /not/ a scripting language; instead it's supposed to be a systems 
 language.

--------------------------------------- Which is why having a native UTF-8 type is so useful, and why being able to 'temporarily' implicitly convert to utf-32 for convenience while storing in the most space efficient format is so useful.
 Besides, if IUC were adopted, less
 people would have to worry about the distinction anyway.

The same could be said for the implicit transcoding from one type to the other.

======================== That's pretty short-sighted IMO. You appear to be saying that implicit transcoding would take the place of ICU; terribly misleading.

--------------------------------------- Not at all. I implied this statement: "if <implicit transcoding> were adopted, less people would have to worry about the distinction anyway" or tried to.
 Transcoding is
 just a very small part of that package. Please try to reread the comment 
 as
 "most people would be shielded completely by the library functions,
 therefore there's far fewer scenarios where they'd ever have a need to 
 drop
 into anything else".

 This would be a very GoodThing for D users. Far better to have a good
 library to take case of /all/ this crap than have the D language do some
 partial-conversions on the fly, and then quit because it doesn't know 
 how to
 provide any further functionality. This is the classic
 core-language-versus-library-functionality bitchfest all over again.
 Building all this into a cast()? Hey! Let's make Walter's Regex class 
 part
 of the compiler too; and make it do UTF-8 decoding /while/ it's 
 searching,
 since you'll be able to pass it a dchar[] that will be generously 
 converted
 to the accepted char[] for you "on-the-fly".

 Excuse me for jesting, but perhaps the Bidi text algorithms plus
 date/numeric formatting & parsing will all fit into a single operator 
 also?
 That's kind of what's being suggested. I believe there's a serious
 underestimate of the task being discussed.

--------------------------------------- This is an over-exageration, perhaps you should give Jim Carrey acting lessons <g>
 Believe me when I say that Mango would dearly love to go dchar[] only.
 Actually, it probably will at the higher levels because it makes life
 simple
 for everyone. Oh, and I've been accused many times of being an


 fanatic, especially when it comes to servers. But there's always a
 tradeoff
 somewhere. Here, the tradeoff is simplicity-of-use versus quantities 

 RAM.
 Which one changes dramatically over time? Hmmmm ... let me see now ...
 64bit-OS for desktops just around corner?

What you really mean is you'd dearly love to not have to worry about the differences between the 3 types, implicit transcoding will give you that. Furthermore it's simplicity without sacraficing RAM.

======================== Ahh Thanks. I didn't realize that's what I "really meant". Wait a minute ...

--------------------------------------- Forgive me for sharing my interpretation of what you said.
 Even on an embedded device I'd probably go "dchar only" regarding 

 Simply because the quantity of text processed on such devices is very
 limited. Before anyone shoots me over this one, I regularly write code
 for
 devices with just 4KB RAM ~ still use 16bit chars there when dealing


 XML input.

We're you to use implicit transcoding you could store the data in memory in UTF-8, or UTF-16 then only transcode to UTF-32 when required, this would be more efficient.

======================== That's a rather large assumption, don't you think? More efficient? In which particualr way? Is memory usage or CPU usage more important in /my/ particular applications? Please either refrain, or commit to rewriting all my old code more efficiently for me ... for free <g>

--------------------------------------- As stated earlier this concept does not stop you from optimising your app in any way shape or form, it's simply a sensible default behaviour IMO. Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
parent reply "antiAlias" <fu bar.com> writes:
"Regan Heath" <regan netwin.co.nz> wrote ..
 What happens when there's a partial character left undecoded
 at the end of  'src'?

How is that even possible?

It happens all the time with streamed input. However, as AJ pointed out, neither you nor Walter are apparently suggesting that the cast() approach be used for anything other than trivial conversions. That is, one would not use this approach with respect to IO streaming. I had the (distinctly wrong) impression this implied-conversion was intended to be a jack-of-all-trades. Everything else in the post is therefore cast(void) ~ so let's stop wasting our breath :) If these implicit conversions are put in place, then I respectfully suggest the std.utf functions be replaced with something that avoids fragmenting the heap in the manner they currently do (for non Latin-1); and it's not hard to make them an order-of-magnitude faster, too. Finally; there's still the problems related to string-concatentation and toString(), as described toward the end of this post news:cgg1mi$14l2$1 digitaldaemon.com
Aug 24 2004
next sibling parent Regan Heath <regan netwin.co.nz> writes:
On Tue, 24 Aug 2004 19:29:24 -0700, antiAlias <fu bar.com> wrote:
 "Regan Heath" <regan netwin.co.nz> wrote ..
 What happens when there's a partial character left undecoded
 at the end of  'src'?

How is that even possible?

It happens all the time with streamed input.

Ahhh.. I get it you were referring to not having all the input at one time, with some being left in the 'stream'.. I can see your concern now.
 However, as AJ pointed out,
 neither you nor Walter are apparently suggesting that the cast() 
 approach be
 used for anything other than trivial conversions.

Correct, the cases where the current approach can actually create a bug, a bug that only sometimes happens.
 That is, one would not use
 this approach with respect to IO streaming. I had the (distinctly wrong)
 impression this implied-conversion was intended to be a 
 jack-of-all-trades.

 Everything else in the post is therefore cast(void)  ~  so let's stop
 wasting our breath :)

Yay :)
 If these implicit conversions are put in place, then I respectfully 
 suggest
 the std.utf functions be replaced with something that avoids fragmenting 
 the
 heap in the manner they currently do (for non Latin-1); and it's not 
 hard to
 make them an order-of-magnitude faster, too.

Good idea. If it's done it has to be done as efficiently as possible.
 Finally; there's still the problems related to string-concatentation and
 toString(), as described toward the end of this post

Yep. I think the toString restriction should be lifted, with implicit transcoding any of the string types should be valid. I am still concerned about the number of transcoding operations that might occur in a unsuspecting programmers string concatenation... Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cggtcq$1ird$1 digitaldaemon.com>, antiAlias says...
"Regan Heath" <regan netwin.co.nz> wrote ..
 What happens when there's a partial character left undecoded
 at the end of  'src'?

How is that even possible?

It happens all the time with streamed input. However, as AJ pointed out, neither you nor Walter are apparently suggesting that the cast() approach be used for anything other than trivial conversions. That is, one would not use this approach with respect to IO streaming. I had the (distinctly wrong) impression this implied-conversion was intended to be a jack-of-all-trades.

My modified version of std.utf is meant to address the streaming issue. Basically, I added versions of encode and decode that accept as the source or destination hook. Not perfect perhaps, but it does get around the problem of encode/decode wanting to throw aqn exception of they encounter an invalid sequence.
If these implicit conversions are put in place, then I respectfully suggest
the std.utf functions be replaced with something that avoids fragmenting the
heap in the manner they currently do (for non Latin-1); and it's not hard to
make them an order-of-magnitude faster, too.

Then by all means do so :) Sean
Aug 25 2004
next sibling parent Sean Kelly <sean f4.ca> writes:
In article <cgidgk$28mg$1 digitaldaemon.com>, Sean Kelly says...
Basically, I added versions of encode and decode that accept as the source or
destination hook.

"Accept a delegate." Sean
Aug 25 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cgidgk$28mg$1 digitaldaemon.com>, Sean Kelly says...

If these implicit conversions are put in place, then I respectfully suggest
the std.utf functions be replaced with something that avoids fragmenting the
heap in the manner they currently do (for non Latin-1); and it's not hard to
make them an order-of-magnitude faster, too.

Then by all means do so :) Sean

Some speed-up ideas... I posted a potentially speedier version of UTF-8 decode here a while back. The basic algorithm I used was this: get the first byte; if it's ASCII, return it; else use it as an index into a lookup table to get the sequence length. There's slightly more to it than that, obviously, but that was the basis. Walter wanted to know if there were any standard tests to check whether a UTF-8 function works correctly. I didn't know of any. The big difficulty with UTF-8 is that of being fully Unicode conformant. This is poorly understood, so people are often tempted to make shortcuts. The std.utf functions take no shortcuts and so are conformant. The jist is this, however. You can have two different kinds of UTF-8 decode routine - checked or unchecked. A checked function will ensure that the input contains no invalid sequences (non-shortest sequences are always invalid), and will throw an exception (or otherwise report the error) if that's not the case. Checked decoders can be made fully conformant, but the checking can slow you down. Unchecked decoders, on the other hand, simply /assume/ that the input is valid, and produce garbage if it isn't. Unchecked decoders can be made to go a lot faster, but they are not Unicode conformant ... unless of course you *KNOW* with 100% certainty that the input *IS* valid. (Without this knowledge, your application won't be Unicode conformant, and can actually be a security risk). So, it would be possible to write a fast, unchecked UTF-8 decoder, if you made use of D's Design by Contract. If you validate the string in the function's "in" block, then you can assume valid input in the function body, and thereby go faster (at least in a release build). But watch out for coding errors. The caller *MUST* fulfil that contract, or you have a bug. And you'd still need to have a checked UTF-8 decoder for those cases when you're not sure where the input came from. Being able to distinguish between sequences which have already been validated, and those which have not, can buy you a lot of efficiency. Unfortunately, I don't see how D can take advantage of that. If a D string were a class or a struct, then it could have a class invariant - but D strings are just simple arrays, and constructing invalid UTF-8 arrays is all too easy. Arcane Jill
Aug 25 2004
parent reply "antiAlias" <fu bar.com> writes:
The decoding thing has issues as you point out Jill.

That's not the whole story, however. The Mango utf-8 Encoder is 5 to 15
times faster than the Phobos one, while executing the equivalent algorithm.
The CheckedDecoder is 5 to 15 times faster, whilst the UncheckedDecoder is
10 to 30 times faster (all  variances are due to alternate mixes of char,
wchar, dchar; all timings performed on a P3).

These are rather significant differences. I think it's safe to say that the
Phobos routines were "not written with efficiency in mind". Either that, or
Mango has some secret means of warping time ...



"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cgih09$2ak7$1 digitaldaemon.com...
 In article <cgidgk$28mg$1 digitaldaemon.com>, Sean Kelly says...

If these implicit conversions are put in place, then I respectfully



the std.utf functions be replaced with something that avoids fragmenting



heap in the manner they currently do (for non Latin-1); and it's not



make them an order-of-magnitude faster, too.

Then by all means do so :) Sean

Some speed-up ideas... I posted a potentially speedier version of UTF-8 decode here a while back.

 basic algorithm I used was this: get the first byte; if it's ASCII, return

 else use it as an index into a lookup table to get the sequence length.

 slightly more to it than that, obviously, but that was the basis. Walter

 to know if there were any standard tests to check whether a UTF-8 function

 correctly. I didn't know of any.

 The big difficulty with UTF-8 is that of being fully Unicode conformant.

 poorly understood, so people are often tempted to make shortcuts. The

 functions take no shortcuts and so are conformant.

 The jist is this, however. You can have two different kinds of UTF-8

 routine - checked or unchecked. A checked function will ensure that the

 contains no invalid sequences (non-shortest sequences are always invalid),

 will throw an exception (or otherwise report the error) if that's not the

 Checked decoders can be made fully conformant, but the checking can slow

 down.

 Unchecked decoders, on the other hand, simply /assume/ that the input is

 and produce garbage if it isn't. Unchecked decoders can be made to go a

 faster, but they are not Unicode conformant ... unless of course you

 100% certainty that the input *IS* valid. (Without this knowledge, your
 application won't be Unicode conformant, and can actually be a security

 So, it would be possible to write a fast, unchecked UTF-8 decoder, if you

 use of D's Design by Contract. If you validate the string in the

 block, then you can assume valid input in the function body, and thereby

 faster (at least in a release build). But watch out for coding errors. The
 caller *MUST* fulfil that contract, or you have a bug. And you'd still

 have a checked UTF-8 decoder for those cases when you're not sure where

 input came from.

 Being able to distinguish between sequences which have already been

 and those which have not, can buy you a lot of efficiency. Unfortunately,

 don't see how D can take advantage of that. If a D string were a class or

 struct, then it could have a class invariant - but D strings are just

 arrays, and constructing invalid UTF-8 arrays is all too easy.

 Arcane Jill

Aug 25 2004
parent reply pragma <pragma_member pathlink.com> writes:
In article <cgij3k$2bon$1 digitaldaemon.com>, antiAlias says...
The decoding thing has issues as you point out Jill.

That's not the whole story, however. The Mango utf-8 Encoder is 5 to 15
times faster than the Phobos one, while executing the equivalent algorithm.
The CheckedDecoder is 5 to 15 times faster, whilst the UncheckedDecoder is
10 to 30 times faster (all  variances are due to alternate mixes of char,
wchar, dchar; all timings performed on a P3).

Holy crap, what kind of data are you throwing at it? I don't mean to criticize, but there must be some clever coding on your part or some serious loopholes in the algorithm to get that kind of an improvement. :)
These are rather significant differences. I think it's safe to say that the
Phobos routines were "not written with efficiency in mind". Either that, or
Mango has some secret means of warping time ...

In the case of the latter, may we rename the "Mango" to "Tardis"? -Pragma [[ EricAnderton at (daleks can't use stairs) yahoo.com ]]
Aug 25 2004
parent reply "antiAlias" <fu bar.com> writes:
<g> That's funny.

No loopholes; no clever coding. Just take a look at (for example) what
utf.encode does to the heap. Those order-of-magnitude timings are best-case
for Phobos ~ in a busy server environment they'd be even slower, likely
cause notable heap fragmentation, and persistently lock the heap against
other (much more appropriate) usage by other threads. Imagine multiple
threads doing implicit dchar[] conversions via utf.encode?

Because there's no supported means of resolving such things, one becomes
inclined to simply 'reimplement' instead of pooling ones skills and
resources to fix Phobos.

This is exactly the kind of thing the DSLG should take care of.



"pragma" <pragma_member pathlink.com> wrote in message
news:cgin45$2djh$1 digitaldaemon.com...
 In article <cgij3k$2bon$1 digitaldaemon.com>, antiAlias says...
The decoding thing has issues as you point out Jill.

That's not the whole story, however. The Mango utf-8 Encoder is 5 to 15
times faster than the Phobos one, while executing the equivalent


The CheckedDecoder is 5 to 15 times faster, whilst the UncheckedDecoder


10 to 30 times faster (all  variances are due to alternate mixes of char,
wchar, dchar; all timings performed on a P3).

Holy crap, what kind of data are you throwing at it? I don't mean to

 but there must be some clever coding on your part or some serious

 the algorithm to get that kind of an improvement. :)

These are rather significant differences. I think it's safe to say that


Phobos routines were "not written with efficiency in mind". Either that,


Mango has some secret means of warping time ...

In the case of the latter, may we rename the "Mango" to "Tardis"? -Pragma [[ EricAnderton at (daleks can't use stairs) yahoo.com ]]

Aug 25 2004
next sibling parent Regan Heath <regan netwin.co.nz> writes:
Might I humbly suggest you add these routines to deimos?
Have you seen my post to the DSLG thread, I think my idea has merit..

On Wed, 25 Aug 2004 13:07:57 -0700, antiAlias <fu bar.com> wrote:
 <g> That's funny.

 No loopholes; no clever coding. Just take a look at (for example) what
 utf.encode does to the heap. Those order-of-magnitude timings are 
 best-case
 for Phobos ~ in a busy server environment they'd be even slower, likely
 cause notable heap fragmentation, and persistently lock the heap against
 other (much more appropriate) usage by other threads. Imagine multiple
 threads doing implicit dchar[] conversions via utf.encode?

 Because there's no supported means of resolving such things, one becomes
 inclined to simply 'reimplement' instead of pooling ones skills and
 resources to fix Phobos.

 This is exactly the kind of thing the DSLG should take care of.



 "pragma" <pragma_member pathlink.com> wrote in message
 news:cgin45$2djh$1 digitaldaemon.com...
 In article <cgij3k$2bon$1 digitaldaemon.com>, antiAlias says...
The decoding thing has issues as you point out Jill.

That's not the whole story, however. The Mango utf-8 Encoder is 5 to 15
times faster than the Phobos one, while executing the equivalent


The CheckedDecoder is 5 to 15 times faster, whilst the UncheckedDecoder


10 to 30 times faster (all  variances are due to alternate mixes of 

wchar, dchar; all timings performed on a P3).

Holy crap, what kind of data are you throwing at it? I don't mean to

 but there must be some clever coding on your part or some serious

 the algorithm to get that kind of an improvement. :)

These are rather significant differences. I think it's safe to say that


Phobos routines were "not written with efficiency in mind". Either 


Mango has some secret means of warping time ...

In the case of the latter, may we rename the "Mango" to "Tardis"? -Pragma [[ EricAnderton at (daleks can't use stairs) yahoo.com ]]


-- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 25 2004
prev sibling parent pragma <pragma_member pathlink.com> writes:
In article <cgirdm$2fod$1 digitaldaemon.com>, antiAlias says...
<g> That's funny.

Thanks! That's why I've been doing these goofy sigs lately: if I can make people grin while they type, perhaps tempers won't flare as much in this NG. :)
No loopholes; no clever coding. Just take a look at (for example) what
utf.encode does to the heap. Those order-of-magnitude timings are best-case
for Phobos ~ in a busy server environment they'd be even slower, likely
cause notable heap fragmentation, and persistently lock the heap against
other (much more appropriate) usage by other threads. Imagine multiple
threads doing implicit dchar[] conversions via utf.encode?

Yikes. That's amazing. I only wonder how this might stand up against ICU?
Because there's no supported means of resolving such things, one becomes
inclined to simply 'reimplement' instead of pooling ones skills and
resources to fix Phobos.

A personal favorite of mine: 1st law of engineering: Hit it with a hammer 2nd law of engineering: If law 1 fails, *use a bigger hammer* . or the more classic idiom: use the right tool for the right job. One could liken reimplementation to a "programmer's sin" of sorts only if they doom others to continue that replication. Sadly, everybody's lib is currently "in progress" with no end in sight yet (which should change once D's remaining quirks are stamped out). At least Brad has been moderating product additions on dsource to cut down some of the more coarse-grained duplication. However we're still very much in the Beta years of D's life. ;)
This is exactly the kind of thing the DSLG should take care of.

At first I wasn't too keen on the idea myself, but perhaps it could use some more discussion. (I'll post on the other thread). - Pragma [[ EricAnderton at (its gumby dammit) yahoo.com ]]
Aug 25 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <opsc7u56zf5a2sq9 digitalmars.com>, Regan Heath says...

True. However, what else should the opCast for char[] to dchar[] do except 
transcode it? What about opCast for int to dchar.. it seems to me there is 
only 1 choice about what to do, anything else would be operator abuse.

Correctly me if I'm wrong, but according to the docs, there /is/ no from-to version of opCast(). opCast() remains almost completely useless, despite many suggestions to fix it. But there's no need to opCast() anything. Compiler magic can just call std.toUTFxx() directly (which of course is what you said). If you want to use a different encoder, just do it explicitly. e.g. # dchar[] d = someOtherConverter(c);
I don't think transcoding makes any sense unless you're talking about a 
'string' (i.e. char[], wchar[], or dchar[]) as opposed to a UTF-x fragment 
(i.e. char, wchar)

Correct. It doesn't.
As AJ has frequently pointed out a char or wchar does not equal one 
"character" in some cases. Given that and assuming 'c' below is "not a 
whole character", I cannot see how:

dchar d;
char  c = d; //not a whole character

d = c;            //implicit
d = cast(dchar)c; //explicit

The existing behavior is already flawed, but it's not going to change (unless char is ditched), because these are primitive types, and Walter says so. Here's what's wrong: In the direction dchar -> char: # dchar d = '\u1234'; # char c = cast(char) d; // c = '4'; Well, you'd /expect/ that to go wrong. It's a narrowing conversion. But: # char c = 0xE9; // The first byte of a three-byte UTF-8 sequence which, # // when encoded, will represent a character in the range # // U+9000 to U+9FFF # dchar d = c; // Whoops! d now equals 'é' Casting from char to wchar or dchar will /only/ work if the character is actually ASCII.
3. a simple copy of the value, creating an invalid? (will it be AJ?) 
utf-32 character.

No value in the range U+0000 to U+00FF is invalid UTF32. But you can't convert a UTF-8-fragment to a character (see example above) and expect the result to be meaningful, except for ASCII.
utf-x code-point/fragment (I don't know the right term)

Unicode uses the term "code unit". I have avoided that term as it's not altogether clear to those unfamiliar with the jargon. I usually say "UTF-8 fragment" on this newsgroup.
 Then there's performance. It's entirely
 possible to write transcoders that are minimally between 5 and 30 times
 faster than the std.utf ones. Some people actually do care about 
 efficiency.
 I'm one of them.

Same here. I believe we all have the same goal, we just have different ideas about how best to get there.

The current implementation of the std.utf functions is not really relevant. /Tomorrow/'s implementation could be way faster than today's. I don't see any reason why Walter couldn't some day replace the implementation with the fastest one on the planet. No code would break.
I think the confusion comes from being used to only 1 string type, for 
C/C++ programmers it's typically and 8bit unsigned type usually containing 
ASCII, for Java a 16bit signed/unsigned? type containing UTF-16?

Mebee, but in C++ I can do this: # class JillsString { /*...*/ }; # class RegansString { /*...*/ }; # # JillsString s1 = "hello world"; # RegansString s2 = s1; # std::string s3 = s2; # if (s3 == s1) { /*...*/ } The "confusion" in D arises (IMO) because we don't have implicit conversion. Arcane Jill
Aug 24 2004
parent Regan Heath <regan netwin.co.nz> writes:
On Tue, 24 Aug 2004 12:03:17 +0000 (UTC), Arcane Jill 
<Arcane_member pathlink.com> wrote:

<big snip> Thanks for those explainations.

 The "confusion" in D arises (IMO) because we don't have implicit 
 conversion.

That is my thought also, tho I note you would rather have 1 string type. Lets do a pro's/con's list for implicit conversion and one string type because I am not totally convinced one is better than the other, let me start (trying to be as objective as possible and not favour 'my' idea) [implicit conversion] PROS: P1 - will cause: dchar[] d; char[] c = d; to produce valid utf sequences. P2 - allows you to write 1 of each string returning function (instead of 3) P3 - explicit conversion calls not required. eg toUTFxx(). p1: is vital IMO p2: this means less code replication, and less code in general needs to be written. P3: could be argued to be 'laziness', I've been called lazy in the past. CONS: C1 - transcoding is not FREE and it will happen without obvious indicators that it is happening. C2 - ppl will not learn the difference between char, wchar, and dchar as quickly. c1: I would argue it's is not as big a deal as it first appears, where it happens you would need a toUTFxx call anyway. In string concatenations some extra transcoding will occur and I have no good solution for that, tho, allowing toString to return any of the 3 types would lessen this effect. c2: Might be a 'pro' in disguise, they don't learn the difference because with implicit conversion it doesn't matter. [one string type] PROS: P1 - allows you to write 1 of each string returning function (instead of 3) SAME AS ABOVE CONS: C1 - all your string characters are 16bits wide, space is wasted when support for ASCII or any other 8bit encoding is all that is required. c1: I believe this to be a major 'con' for embedded etc small systems programming. Please everyone add to this list any/all you can think of, correct any you think I have wrong, or miss represented. Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
"antiAlias" <fu bar.com> wrote in message
news:cge5h3$5hv$1 digitaldaemon.com...
 On the one hand, this would be well served by an opCast() and/or

 on the primitive types; just the kind of thing  suggested in a related
 thread (which talked about overloadable methods for primitive types).

 On the other hand, we're talking transcoding here. Are you gonna' limit

 to UTF-8 only?

No, to between char[], wchar[], and dchar[].
 Then, since the source and destination will typically be of
 different sizes, do you then force all casts between these types to have

 destination be an array reference rather than an instance? One that is
 always allocated on the fly?

I see it as implicitly calling the conversion function(s) in std.utf.
 If you do implement overloadable primitive-methods (like properties) then,
 will you allow a programmer to override them? So they can make the

 do something more specific to their own specific task?

 That's seems like a lot to build into the core of a language.

I don't see adding opCast() for builtin types.
 Personally, I think it's 'borderline' to have so many data types available
 for similar things. If there were an "alias wchar[] string", and the core
 Object supported that via "string toString()", and the IUC library were
 adopted, then I think some of the general confusion would perhaps melt
 somewhat.

 In many respects, too many choices is simply a BadThing (TM). Especially
 when there's precious little solid guidance to help. That guidance might
 come from a decent library that indicates how the types are used, and uses
 one obvious type (string?) consistently. Besides, if IUC were adopted,

 people would have to worry about the distinction anyway.

 Believe me when I say that Mango would dearly love to go dchar[] only.
 Actually, it probably will at the higher levels because it makes life

 for everyone. Oh, and I've been accused many times of being an efficiency
 fanatic, especially when it comes to servers. But there's always a

 somewhere. Here, the tradeoff is simplicity-of-use versus quantities of

 Which one changes dramatically over time? Hmmmm ... let me see now ...
 64bit-OS for desktops just around corner?

 Even on an embedded device I'd probably go "dchar only" regarding I18N.
 Simply because the quantity of text processed on such devices is very
 limited. Before anyone shoots me over this one, I regularly write code for
 devices with just 4KB RAM ~ still use 16bit chars there when dealing with
 XML input.

 So what am I saying here? Available RAM will always increase in great

 Contemplating that the latter should dictate ease-of-use within D is a
 serious breach of logic, IMO. Ease of use, and above all, /consistency/
 should be paramount; if you have the programmer in mind.

I thought so, too, until I built a server app that used all dchar[] internally. Server apps tend to be driven to the limit, and reaching that limit 4x sooner means that the customer has to buy 4x more server hardware. Remember, that using 4 bytes per char doesn't just consume more ram, it consumes a LOT more processor cycles with managing the extra memory. (scanning, copying, initializing, gc marking, etc.)
Aug 24 2004
parent reply "antiAlias" <fu bar.com> writes:
"Walter"  wrote in...
 So what am I saying here? Available RAM will always increase in great

 Contemplating that the latter should dictate ease-of-use within D is a
 serious breach of logic, IMO. Ease of use, and above all, /consistency/
 should be paramount; if you have the programmer in mind.

I thought so, too, until I built a server app that used all dchar[] internally. Server apps tend to be driven to the limit, and reaching that limit 4x sooner means that the customer has to buy 4x more server

 Remember, that using 4 bytes per char doesn't just consume more ram, it
 consumes a LOT more processor cycles with managing the extra memory.
 (scanning, copying, initializing, gc marking, etc.)

I disagree with that for a number of reasons <g> a) I was saying that usage of memory should not dictate language ease-of-use. I didn't say that byte[], ubyte[], char[] and wchar[] should all be dumped. If D were dchar[] oriented, rather than char[] oriented, it would arguably make it easier to use for the everyday folks. Those who really care about squeezing bytes can, and should, deal with text encoding and decoding issues. As it is, /everyone/ currently has to deal with those issues at various levels. b) There's an implication that all server apps are text-bound. That's just not the case, but perhaps I'm being pedantic. c) People who write servers have (traditionally) been a little more careful about what they do. There are plenty of ways to avoid allocating memory and thrashing the GC, where that's a concern. I do it all the time. In fact, one of the unwritten goals of writing server software is to avoid regularly using malloc/calloc where possible. d) The predominant modern cpu's all have prefetch built-in, because of the marketing craze for streaming-style application. This is great news for wide chars! It means that a server can stream dchar[] much more effectively than it could just a few years back. It's the conversions that are arguably a problem. e) dchar is the natural width of a 32bit processor, so it's not gonna take more Processor Cycles to process those than 8bit chars. In fact, it's the other way round where UTF-8 is involved. The bottleneck used to be the front-side bus. Not so these days of 1Ghz HyperTransport, 800MHz Intel quad-pumped bus, and prefetch everywhere. So, no. I simply cannot agree that using dchar[] automatically means the customer has to buy 4x more server hardware <g>
Aug 24 2004
next sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cgg5ko$16ja$1 digitaldaemon.com>, antiAlias says...
a) I was saying that usage of memory should not dictate language
ease-of-use. I didn't say that byte[], ubyte[], char[] and wchar[] should
all be dumped.  If D were dchar[] oriented, rather than char[] oriented, it
would arguably make it easier to use for the everyday folks. Those who
really care about squeezing bytes can, and should, deal with text encoding
and decoding issues. As it is, /everyone/ currently has to deal with those
issues at various levels.

Ideally, an i/o library should be able to handle most conversions invisibly, so the user can work in whatever internal format they want without worrying too much about the external format. doFormat already takes char, wchar, and dchar arguments and outputs UTF-8 or UTF-16 as appropriate, and I designed unFormat to do pretty much the same. I will say, however, that multibyte encoding schemes are generally not very easy to deal with, so internal use of dchars still makes a lot of sense.
b) There's an implication that all server apps are text-bound. That's just
not the case, but perhaps I'm being pedantic.

More often than not they are, especially in this age of XML and such. And for the ones that aren't text-bound, who cares how D deals with strings? :)
e) dchar is the natural width of a 32bit processor, so it's not gonna take
more Processor Cycles to process those than 8bit chars. In fact, it's the
other way round where UTF-8 is involved. The bottleneck used to be the
front-side bus. Not so these days of 1Ghz HyperTransport, 800MHz Intel
quad-pumped bus, and prefetch everywhere.

I think that UTF-8 is still more efficient in terms of memory reads and writes, simply because it tends to take up less space than UTF-16 or UTF-32. The tradeoff is in processing time when the data ventures into the multibyte realm, which is becoming increasingly more common. But I'll have to think some more about whether I would always write text-bound servers using dchars. I'd like to since it would simplify handling XML data, but I'm not particularly keen on those 1GB streams suddenly becoming 4GB streams. That's a decent bit of i/o overhead, especially if I know that little to no data in that stream lives beyond the ASCII charset. At the end of the day, I think the programmer should be able to choose the appropriate charset for the job. Implicit conversion between char types is a great idea and should clear up most of the confusion. And the UTF-32 version is called "dchar" which implies to me that it's the native character format for D anyway. Perhaps "char" should be renamed to something else? Sean
Aug 24 2004
parent reply "antiAlias" <fu bar.com> writes:
"Sean Kelly" <sean f4.ca> wrote ...
 I think that UTF-8 is still more efficient in terms of memory reads and


 simply because it tends to take up less space than UTF-16 or UTF-32.  The
 tradeoff is in processing time when the data ventures into the multibyte

 which is becoming increasingly more common.  But I'll have to think some

 about whether I would always write text-bound servers using dchars.

I'm not saying that they should always be dchar[] :-) I'm saying using the additional memory usage of dchar[] as a tradeoff against language ease-of-use is an invalid one. You can always dip down into ubyte[] for those apps that actually care.
 I'd like to
 since it would simplify handling XML data, but I'm not particularly keen

 those 1GB streams suddenly becoming 4GB streams.  That's a decent bit of

 overhead, especially if I know that little to no data in that stream lives
 beyond the ASCII charset.

I'm rather tempted to suggest that a 1GB stream of XML has other problems to content with <g>
Aug 24 2004
parent Sean Kelly <sean f4.ca> writes:
In article <cggca6$19k9$1 digitaldaemon.com>, antiAlias says...
I'm rather tempted to suggest that a 1GB stream of XML has other problems to
content with <g>

Its only problem is that XML is a terrible text format. But for one server I wrote it was not impossible that there may be a 1GB (uncompressed) data stream. This was obviously not going to a web browser :) Sean
Aug 24 2004
prev sibling parent reply Regan Heath <regan netwin.co.nz> writes:
On Tue, 24 Aug 2004 12:44:03 -0700, antiAlias <fu bar.com> wrote:
 "Walter"  wrote in...
 So what am I saying here? Available RAM will always increase in great

 Contemplating that the latter should dictate ease-of-use within D is a
 serious breach of logic, IMO. Ease of use, and above all, 

 should be paramount; if you have the programmer in mind.

I thought so, too, until I built a server app that used all dchar[] internally. Server apps tend to be driven to the limit, and reaching that limit 4x sooner means that the customer has to buy 4x more server

 Remember, that using 4 bytes per char doesn't just consume more ram, it
 consumes a LOT more processor cycles with managing the extra memory.
 (scanning, copying, initializing, gc marking, etc.)

I disagree with that for a number of reasons <g> a) I was saying that usage of memory should not dictate language ease-of-use. I didn't say that byte[], ubyte[], char[] and wchar[] should all be dumped. If D were dchar[] oriented, rather than char[] oriented, it would arguably make it easier to use for the everyday folks. Those who really care about squeezing bytes can, and should, deal with text encoding and decoding issues. As it is, /everyone/ currently has to deal with those issues at various levels.

The exact same thing can be said for implicit transcoding. What I mean is... Implicit transcoding will make it easier to use for everyday folks. Those who really care about squeezing bytes can. The advantage implicit transcoding has is everyday folks will likely be dealing with ASCII and will likely use char[] which has an advantage over dchar[] where you're dealing with ASCII. Furthermore implicit transcoding removes the need to deal with encoding/decoding issue, generally speaking. Those that need to worry about it, can and will optimise where the implicit transcoding causes in-efficient behaviour.
 b) There's an implication that all server apps are text-bound. That's 
 just
 not the case, but perhaps I'm being pedantic.

It depends on the server. The mail server I work on is a candidate to be text-bound, in fact it is, it's disk bound, meaning, we cannot write our email text out to disk as fast as we can receive it (tcp/ip), and process it (transcoding etc).
 c) People who write servers have (traditionally) been a little more 
 careful
 about what they do. There are plenty of ways to avoid allocating memory 
 and
 thrashing the GC, where that's a concern. I do it all the time. In fact, 
 one
 of the unwritten goals of writing server software is to avoid regularly
 using malloc/calloc where possible.

Definately. having a UTF-8 char type which you can implicitly convert to a more convenient format temporarily (dchar[], utf-32) simply makes this easier IMO.
 d) The predominant modern cpu's all have prefetch built-in, because of 
 the
 marketing craze for streaming-style application. This is great news for 
 wide
 chars! It means that a server can stream dchar[] much more effectively 
 than
 it could just a few years back. It's the conversions that are arguably a
 problem.

If we're talking streaming as in streaming to disk or tcp/ip etc, I would argue that the time it takes to transcode is much less than the time it takes to write/send.
 e) dchar is the natural width of a 32bit processor, so it's not gonna 
 take
 more Processor Cycles to process those than 8bit chars. In fact, it's the
 other way round where UTF-8 is involved. The bottleneck used to be the
 front-side bus. Not so these days of 1Ghz HyperTransport, 800MHz Intel
 quad-pumped bus, and prefetch everywhere.

 So, no. I simply cannot agree that using dchar[] automatically means the
 customer has to buy 4x more server hardware <g>

All this arguing about what is more efficient is IMO totally pointless, the types of application vary so much that for one application/situation one method will be best and for another the other method will be. D's goal is not to be specialised for any one style or application, as such 3 char types makes sense, doesn't it? Regardless the only way to settle the performance argument is to benchmark something, therefore... In what situations do you believe using UTF-32 dchar throughout the application will be faster than using all 3 types and implicit transcoding.. Consider: - the input may be in any encoding - the output may be in any encoding - it may need to store large amounts of the input in memory ..can you think of any more? Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004
parent reply "antiAlias" <fu bar.com> writes:
Regan; I appeal to you to try and read things in context. I'm not even
vaguely interested in getting into a pissing contest with you, so please,
try and follow this (and correlate with the text below if you have to):

a) I say available-memory will always increase in great leaps, so using that
as a design guide vis-a-vis a computer language doesn't make sense to me.

b) Walter says he used to think so too, until he build a wide-char-only
server; and points out that wide-chars can force the customer into
purchasing much more hardware due to memory consumption and additional CPU
usage.

c) I disagree with that position, and try to illustrate why I don't think
wide-chars are the demon they might once have been considered. And that
perhaps they get a 'bad rap' for the wrong reasons.

What you added here seems intended to fan some imaginary flames, or to be
argumentative purely for the sake of it, rather than to make any cohesive
point. In fact, four out of the five items you managed to completely
misconstrue. That may be my failing in terms of language use, so I'll accept
the consequences. I will not, however, bite.

Good-day, my friend  :-)


"Regan Heath" <regan netwin.co.nz> wrote in message
news:opsc9o30tt5a2sq9 digitalmars.com...
 On Tue, 24 Aug 2004 12:44:03 -0700, antiAlias <fu bar.com> wrote:
 "Walter"  wrote in...
 So what am I saying here? Available RAM will always increase in great

 Contemplating that the latter should dictate ease-of-use within D is




 serious breach of logic, IMO. Ease of use, and above all,

 should be paramount; if you have the programmer in mind.

I thought so, too, until I built a server app that used all dchar[] internally. Server apps tend to be driven to the limit, and reaching that limit 4x sooner means that the customer has to buy 4x more server

 Remember, that using 4 bytes per char doesn't just consume more ram, it
 consumes a LOT more processor cycles with managing the extra memory.
 (scanning, copying, initializing, gc marking, etc.)

I disagree with that for a number of reasons <g> a) I was saying that usage of memory should not dictate language ease-of-use. I didn't say that byte[], ubyte[], char[] and wchar[]


 all be dumped.  If D were dchar[] oriented, rather than char[] oriented,
 it
 would arguably make it easier to use for the everyday folks. Those who
 really care about squeezing bytes can, and should, deal with text
 encoding
 and decoding issues. As it is, /everyone/ currently has to deal with
 those
 issues at various levels.

The exact same thing can be said for implicit transcoding. What I mean is... Implicit transcoding will make it easier to use for everyday folks. Those who really care about squeezing bytes can. The advantage implicit transcoding has is everyday folks will likely be dealing with ASCII and will likely use char[] which has an advantage over dchar[] where you're dealing with ASCII. Furthermore implicit transcoding removes the need to deal with encoding/decoding issue, generally speaking. Those that need to worry about it, can and will optimise where the implicit transcoding causes in-efficient behaviour.
 b) There's an implication that all server apps are text-bound. That's
 just
 not the case, but perhaps I'm being pedantic.

It depends on the server. The mail server I work on is a candidate to be text-bound, in fact it is, it's disk bound, meaning, we cannot write our email text out to disk as fast as we can receive it (tcp/ip), and process it (transcoding etc).
 c) People who write servers have (traditionally) been a little more
 careful
 about what they do. There are plenty of ways to avoid allocating memory
 and
 thrashing the GC, where that's a concern. I do it all the time. In fact,
 one
 of the unwritten goals of writing server software is to avoid regularly
 using malloc/calloc where possible.

Definately. having a UTF-8 char type which you can implicitly convert to a more convenient format temporarily (dchar[], utf-32) simply makes this easier IMO.
 d) The predominant modern cpu's all have prefetch built-in, because of
 the
 marketing craze for streaming-style application. This is great news for
 wide
 chars! It means that a server can stream dchar[] much more effectively
 than
 it could just a few years back. It's the conversions that are arguably a
 problem.

If we're talking streaming as in streaming to disk or tcp/ip etc, I would argue that the time it takes to transcode is much less than the time it takes to write/send.
 e) dchar is the natural width of a 32bit processor, so it's not gonna
 take
 more Processor Cycles to process those than 8bit chars. In fact, it's


 other way round where UTF-8 is involved. The bottleneck used to be the
 front-side bus. Not so these days of 1Ghz HyperTransport, 800MHz Intel
 quad-pumped bus, and prefetch everywhere.

 So, no. I simply cannot agree that using dchar[] automatically means the
 customer has to buy 4x more server hardware <g>

All this arguing about what is more efficient is IMO totally pointless, the types of application vary so much that for one application/situation one method will be best and for another the other method will be. D's goal is not to be specialised for any one style or application, as such 3 char types makes sense, doesn't it? Regardless the only way to settle the performance argument is to benchmark something, therefore... In what situations do you believe using UTF-32 dchar throughout the application will be faster than using all 3 types and implicit transcoding.. Consider: - the input may be in any encoding - the output may be in any encoding - it may need to store large amounts of the input in memory ..can you think of any more? Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

Aug 24 2004
parent Regan Heath <regan netwin.co.nz> writes:
On Tue, 24 Aug 2004 22:08:39 -0700, antiAlias <fu bar.com> wrote:

<snip>

 What you added here seems intended to fan some imaginary flames, or to be
 argumentative purely for the sake of it, rather than to make any cohesive
 point. In fact, four out of the five items you managed to completely
 misconstrue. That may be my failing in terms of language use, so I'll 
 accept the consequences. I will not, however, bite.

I'm sorry to have come across that way. I was simply trying to add my point of view. If I have miss-understood your comments, sorry, I don't get it right all the time (despite what I might think). Your (miss/understood/guided/ing) friend, Regan -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 24 2004