www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Error: invalid UTF-8 sequence

reply Carotinho <carotinobg yahoo.it> writes:
Hi all!
I'm new here and to D. I wrote a simple program:

import std.stdio;
import std.stream;

int main() {
  char[] stringa;
  stringa = std.stream.stdin.readLine();
  writefln("%s",stringa);
  return 0;
}

If i type normal characters, like a,b,c etc. everything is ok. 
But when I tries to type special characters like Ŕ, ˛, ¨ I get
  Error: invalid UTF-8 sequence
when the program tries to rewrite the string I got.
What is this?

Thanks in advance!

Carotinho
Nov 28 2004
parent reply =?ISO-8859-15?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Carotinho wrote:

 If i type normal characters, like a,b,c etc. everything is ok. 
 But when I tries to type special characters like Ŕ, ˛, ¨ I get
   Error: invalid UTF-8 sequence
 when the program tries to rewrite the string I got.
 What is this?

D only works with Unicode. You need to set your shell to UTF-8. (Or, the tricky version, you can cast(ubyte[]) and convert it ?) --anders
Nov 28 2004
parent reply "Simon Buchan" <currently no.where> writes:
On Mon, 29 Nov 2004 00:39:29 +0100, Anders F Bj├Ârklund <afb algonet.se>  
wrote:

 Carotinho wrote:

 If i type normal characters, like a,b,c etc. everything is ok. But when  
 I tries to type special characters like ├Ę, ├▓, ├╣ I get
   Error: invalid UTF-8 sequence
 when the program tries to rewrite the string I got.
 What is this?

D only works with Unicode. You need to set your shell to UTF-8. (Or, the tricky version, you can cast(ubyte[]) and convert it ?) --anders

I don't think cast works. Unfortunately, the Windows shell can't use UTF. This discussion was referenced somewhere else (maybe digitalmars.D.bugs?) I have a project that tries to write a file with funky punctuation to the screen... the closest I got was to use read/writeString exclusively which gives you rubbish for special characters. There was something mentioned about a Win32 API that converted UTF to codepages and vice-versa... sounded promising, but I don't know if it is currently available to D. Look around, you may get lucky. (and if you do, tell the rest of us :D) -- Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
Nov 28 2004
next sibling parent =?UTF-8?B?QW5kZXJzIEYgQmrDtnJrbHVuZA==?= <afb algonet.se> writes:
Simon Buchan wrote:

 (Or, the tricky version, you can cast(ubyte[]) and convert it ?)

I don't think cast works. Unfortunately, the Windows shell can't use UTF. This discussion was referenced somewhere else (maybe digitalmars.D.bugs?)

It does. The problem is that D just assumes that the shell is UTF-8, and feeds you char[] that are *invalid* (as they are native-encoded) If you translate them yourself, I've found it to work just fine... I don't have a DOS console ( echo off allergies), but it does work with a zsh console set to the ISO-8859-1 encoding (instead of UTF-8) Of course, if the console *is* Unicode - then this doesn't work... Anyway, my test code looked like:
 void main(char[][] args)
 {
 	wchar[256] mapping = iso88591.mapping;
 
 	char[] test = cast(char[]) decode_string(cast(ubyte[]) args[1], mapping);
 	writefln("%s",test);
 	
 	static ubyte[1] z = [ 0 ];
 	printf("%s\n", cast(char*) (encode_string(test, mapping) ~ z) );
 }

Usually when you call old C functions, you want ubyte[] and not char[] since they don't handle UTF-8? The D tradition is to pretend that they have the D definition (char *) anyway, since "it is the same bit size". I use ubyte[] for legacy 8-bit encodings, and char[] for Unicode only.
 There was something mentioned about a Win32 API that converted UTF to
 codepages and vice-versa... sounded promising, but I don't know if
 it is currently available to D. Look around, you may get lucky.
 (and if you do, tell the rest of us :D)

There is a Win32-only API, and some open source libraries (iconv, ICU): http://msdn.microsoft.com/library/en-us/intl/unicode_19mb.asp http://www.gnu.org/software/libiconv/ http://oss.software.ibm.com/icu/ I might share my own little hack later on too, when I've packaged it up. (it just does the 4 main mappings, not the other 200* that the above do, ISO-8859-1 [Latin-1], CP-437 [DOS], CP-1252 [Win], MacRoman [Mac OS 9] ) It's a lot smaller than the real mccoy, and will be under zlib license. http://www.opensource.org/licenses/zlib-license.php (my usual license) If you need the full functionality, look at Mango/ICU or iconv instead? --anders PS. I'm not kidding, it really has hundreds (!) of different encodings: http://oss.software.ibm.com/icu/charset/
Nov 29 2004
prev sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
[snip]

 There was something mentioned about a Win32 API that converted UTF to
 codepages and vice-versa... sounded promising, but I don't know if
 it is currently available to D. Look around, you may get lucky.
 (and if you do, tell the rest of us :D)

The following solution doesn't handle errors well due to some errno confusion I'm trying to figure out, but it is a start. Here's what you can do. Get iconv.dll from the zip file at http://prdownloads.sourceforge.net/gettext/libiconv-1.9.1.bin.woe32.zip?download and put it in the same directory as your executable. The attached libiconv.d will load the three functions you need. The attached iconv_example.d shows how to call iconv to convert utf-8 to utf-16 little endian. I'm looking into the errno issues and will probably have to recompile libiconv with DMC or something. But for typical usage the above instructions should work. Also I'd like to put a small wrapper around the low-level API to make it easier to use for the simple cases when the input is complete. -Ben
Nov 29 2004
next sibling parent reply Carotinho <carotinobg yahoo.it> writes:
I thanks you all, I'll start experiments!
For information, I'm running Linux, and even here I'm quite a newbie :)

Byez!
Nov 29 2004
parent "Ben Hinkle" <ben.hinkle gmail.com> writes:
"Carotinho" <carotinobg yahoo.it> wrote in message 
news:cog796$1rjr$1 digitaldaemon.com...
I thanks you all, I'll start experiments!
 For information, I'm running Linux, and even here I'm quite a newbie :)

 Byez!

oh, even better. you don't need the dll then - just get the .d file that declares the iconv functions and you're all set (well, except for figuring out the API and getting the right encodings).
Nov 29 2004
prev sibling next sibling parent "Simon Buchan" <currently no.where> writes:
On Mon, 29 Nov 2004 14:14:33 -0500, Ben Hinkle <bhinkle mathworks.com>  
wrote:

 [snip]

 There was something mentioned about a Win32 API that converted UTF to
 codepages and vice-versa... sounded promising, but I don't know if
 it is currently available to D. Look around, you may get lucky.
 (and if you do, tell the rest of us :D)

The following solution doesn't handle errors well due to some errno confusion I'm trying to figure out, but it is a start. Here's what you can do. Get iconv.dll from the zip file at http://prdownloads.sourceforge.net/gettext/libiconv-1.9.1.bin.woe32.zip?download and put it in the same directory as your executable. The attached libiconv.d will load the three functions you need. The attached iconv_example.d shows how to call iconv to convert utf-8 to utf-16 little endian. I'm looking into the errno issues and will probably have to recompile libiconv with DMC or something. But for typical usage the above instructions should work. Also I'd like to put a small wrapper around the low-level API to make it easier to use for the simple cases when the input is complete. -Ben

This doesnt let you make UTF-8 into an OEM codepage, though, does it? Linux users should be fine if they set their console to a UTF, but poor Windows users are stuck with weird codepages. (I do have the UTF codepages installed, they have to be, but I don't know how you can tell the console to use them) -- "Unhappy Microsoft customers have a funny way of becoming Linux, Salesforce.com and Oracle customers." - www.microsoft-watch.com: "The Year in Review: Microsoft Opens Up"
Nov 29 2004
prev sibling parent reply =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Ben Hinkle wrote:

 The following solution doesn't handle errors well due to some errno
 confusion I'm trying to figure out, but it is a start. Here's what you can
 do. Get iconv.dll from the zip file at
 http://prdownloads.sourceforge.net/gettext/libiconv-1.9.1.bin.woe32.zip?download
 and put it in the same directory as your executable. The attached libiconv.d
 will load the three functions you need. The attached iconv_example.d shows
 how to call iconv to convert utf-8 to utf-16 little endian.
 I'm looking into the errno issues and will probably have to recompile
 libiconv with DMC or something. But for typical usage the above instructions
 should work. Also I'd like to put a small wrapper around the low-level API
 to make it easier to use for the simple cases when the input is complete.

This code doesn't work everywhere... (POSIX?) At least not without some more modifications.
   // on POSIX systems iconv is built into libc so loading is automatic

It doesn't work on Mac OS X, unfortunately. # gdc -o iconv_example iconv_example.d libiconv.d -liconv
 /usr/bin/ld: Undefined symbols:
 _iconv
 _iconv_close
 _iconv_open
 collect2: ld returned 1 exit status

(It's being loaded from System's /usr/lib/libiconv.dylib) in /usr/include/iconv.h:
 #define iconv_t libiconv_t
 #ifndef LIBICONV_PLUG
 #define iconv_open libiconv_open
 #define iconv libiconv
 #define iconv_close libiconv_close
 #endif

Annoying, isn't it ? So one needs to declare the C functions with the "lib" prefix, and then do wrappers in D for the usual names...
 } else version (darwin) { 
 
   // On Mac OS X, link with -liconv (/usr/lib/libiconv.dylib)
   typedef void *libiconv_t;
 
   // allocate a converter between charsets fromcode and tocode
   extern (C) libiconv_t libiconv_open (char *tocode, char *fromcode);
   iconv_t iconv_open (char *tocode, char *fromcode)
   { return cast(iconv_t) libiconv_open(tocode, fromcode); }
 
   // convert inbuf to outbuf and set inbytesleft to unused input and
   // outbuf to unused output and return number of non-reversable 
   // conversions or -1 on error.
   extern (C) size_t libiconv (libiconv_t cd, void **inbuf,
 			   size_t *inbytesleft,
 			   void **outbuf,
 			   size_t *outbytesleft);
   size_t iconv (iconv_t cd, void **inbuf, size_t *inbytesleft,
 			   void **outbuf, size_t *outbytesleft)
   { return libiconv(cast(libiconv_t) cd, inbuf, inbytesleft, outbuf,
outbytesleft); }
 
   // close converter
   extern (C) int libiconv_close (libiconv_t cd);
   int iconv_close (iconv_t cd)
   { return libiconv_close(cast(libiconv_t) cd); }
 
 } else { 

And the test code assumed that everything is X86:
 version (LittleEndian)
   // convert from utf-8 to utf-16 little endian
   iconv_t cd = iconv_open("UTF-16LE","UTF-8");
 else version (BigEndian)
   // convert from utf-8 to utf-16 big endian
   iconv_t cd = iconv_open("UTF-16BE","UTF-8");

That's actually one of the biggest drawbacks of UTF-16... Besides those little flaws, the code works just fine :-) --anders
Nov 30 2004
parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"Anders F Bj÷rklund" <afb algonet.se> wrote in message
news:coi25c$1ijr$1 digitaldaemon.com...
 Ben Hinkle wrote:

 The following solution doesn't handle errors well due to some errno
 confusion I'm trying to figure out, but it is a start. Here's what you


 do. Get iconv.dll from the zip file at


 and put it in the same directory as your executable. The attached


 will load the three functions you need. The attached iconv_example.d


 how to call iconv to convert utf-8 to utf-16 little endian.
 I'm looking into the errno issues and will probably have to recompile
 libiconv with DMC or something. But for typical usage the above


 should work. Also I'd like to put a small wrapper around the low-level


 to make it easier to use for the simple cases when the input is


 This code doesn't work everywhere... (POSIX?)
 At least not without some more modifications.

   // on POSIX systems iconv is built into libc so loading is automatic

It doesn't work on Mac OS X, unfortunately. # gdc -o iconv_example iconv_example.d libiconv.d -liconv
 /usr/bin/ld: Undefined symbols:
 _iconv
 _iconv_close
 _iconv_open
 collect2: ld returned 1 exit status

(It's being loaded from System's /usr/lib/libiconv.dylib) in /usr/include/iconv.h:
 #define iconv_t libiconv_t
 #ifndef LIBICONV_PLUG
 #define iconv_open libiconv_open
 #define iconv libiconv
 #define iconv_close libiconv_close
 #endif

Annoying, isn't it ? So one needs to declare the C functions with the "lib" prefix, and then do wrappers in D for the usual names...

That is a bummer. Love those #defines!
 } else version (darwin) {

   // On Mac OS X, link with -liconv (/usr/lib/libiconv.dylib)
   typedef void *libiconv_t;

   // allocate a converter between charsets fromcode and tocode
   extern (C) libiconv_t libiconv_open (char *tocode, char *fromcode);
   iconv_t iconv_open (char *tocode, char *fromcode)
   { return cast(iconv_t) libiconv_open(tocode, fromcode); }

   // convert inbuf to outbuf and set inbytesleft to unused input and
   // outbuf to unused output and return number of non-reversable
   // conversions or -1 on error.
   extern (C) size_t libiconv (libiconv_t cd, void **inbuf,
    size_t *inbytesleft,
    void **outbuf,
    size_t *outbytesleft);
   size_t iconv (iconv_t cd, void **inbuf, size_t *inbytesleft,
    void **outbuf, size_t *outbytesleft)
   { return libiconv(cast(libiconv_t) cd, inbuf, inbytesleft, outbuf,


   // close converter
   extern (C) int libiconv_close (libiconv_t cd);
   int iconv_close (iconv_t cd)
   { return libiconv_close(cast(libiconv_t) cd); }

 } else {


Maybe I'll try using std.loader for this case, too, and have iconv be a function pointer. Hmm...
 And the test code assumed that everything is X86:

 version (LittleEndian)
   // convert from utf-8 to utf-16 little endian
   iconv_t cd = iconv_open("UTF-16LE","UTF-8");
 else version (BigEndian)
   // convert from utf-8 to utf-16 big endian
   iconv_t cd = iconv_open("UTF-16BE","UTF-8");

That's actually one of the biggest drawbacks of UTF-16...

That's true. I was being lazy with the example. When I tried just plain-old "UTF-16" I think it used big-endian.
 Besides those little flaws, the code works just fine :-)
 --anders

Thanks for the update. I obviously hadn't tried on the Mac.
Nov 30 2004
next sibling parent =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Ben Hinkle wrote:

 That's true. I was being lazy with the example. When I tried just plain-old
 "UTF-16" I think it used big-endian.

Yes, that is in the Unicode specification. BE is the default order. Unless there is a BOM present to classify it as LE instead, that is... See http://www.unicode.org/faq/utf_bom.html --anders
Nov 30 2004
prev sibling parent "Kris" <fu bar.com> writes:
"Ben Hinkle" <bhinkle mathworks.com> wrote ...

| Maybe I'll try using std.loader for this case, too, and have iconv be a
| function pointer. Hmm...

That won't work with dmd 0.107 ...
Nov 30 2004