www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - UTF-16 endianess

reply Marek Janukowicz <marek janukowicz.net> writes:
I have trouble understanding how endianess works for UTF-16.

For example UTF-16 code for 'ł' character is 0x0142. But this program shows
otherwise:

import std.stdio;

public void main () {
  ubyte[] properOrder = [0x01, 0x42];
	ubyte[] reverseOrder = [0x42, 0x01];
	writefln( "proper: %s, reverse: %s", 
		cast(wchar[])properOrder, 
		cast(wchar[])reverseOrder );
}

output:

proper: 䈁, reverse: ł

Is there anything I should know about UTF endianess?

-- 
Marek Janukowicz
Jan 29
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/29/16 5:36 PM, Marek Janukowicz wrote:
 I have trouble understanding how endianess works for UTF-16.

 For example UTF-16 code for 'ł' character is 0x0142. But this program shows
 otherwise:

 import std.stdio;

 public void main () {
    ubyte[] properOrder = [0x01, 0x42];
 	ubyte[] reverseOrder = [0x42, 0x01];
 	writefln( "proper: %s, reverse: %s",
 		cast(wchar[])properOrder,
 		cast(wchar[])reverseOrder );
 }

 output:

 proper: 䈁, reverse: ł

 Is there anything I should know about UTF endianess?
It's not any different from other endianness. In other words, a UTF16 code unit is expected to be in the endianness of the platform you are running on. If you are on x86 or x86_64 (very likely), then it should be little endian. If your source of data is big-endian (or opposite from your native endianness), it will have to be converted before treating as a wchar[]. Note the version identifiers BigEndian and LittleEndian can be used to compile the correct code. -Steve
Jan 29
parent reply Marek Janukowicz <marek janukowicz.net> writes:
On Fri, 29 Jan 2016 17:43:26 -0500, Steven Schveighoffer wrote:
 Is there anything I should know about UTF endianess?
It's not any different from other endianness. In other words, a UTF16 code unit is expected to be in the endianness of the platform you are running on. If you are on x86 or x86_64 (very likely), then it should be little endian. If your source of data is big-endian (or opposite from your native endianness),
To be precise - my case is IMAP UTF7 folder name encoding and I finally found out it's indeed big endian, which explains my problem (as I'm indeed on x86_64).
 it will have to be converted before treating as a wchar[].
Is there any clever way to do the conversion? Or do I need to swap the bytes manually?
 Note the version identifiers BigEndian and LittleEndian can be used to 
 compile the correct code.
This solution is of no use to me as I don't want to change the endianess in general. -- Marek Janukowicz
Jan 29
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/29/16 6:03 PM, Marek Janukowicz wrote:
 On Fri, 29 Jan 2016 17:43:26 -0500, Steven Schveighoffer wrote:
 Is there anything I should know about UTF endianess?
It's not any different from other endianness. In other words, a UTF16 code unit is expected to be in the endianness of the platform you are running on. If you are on x86 or x86_64 (very likely), then it should be little endian. If your source of data is big-endian (or opposite from your native endianness),
To be precise - my case is IMAP UTF7 folder name encoding and I finally found out it's indeed big endian, which explains my problem (as I'm indeed on x86_64).
 it will have to be converted before treating as a wchar[].
Is there any clever way to do the conversion? Or do I need to swap the bytes manually?
No clever way, just the straightforward way ;) Swapping endianness of 32-bits can be done with core.bitop.bswap. Doing it with 16 bits I believe you have to do bit shifting. Something like: foreach(ref elem; wcharArr) elem = ((elem << 8) & 0xff00) | ((elem >> 8) & 0x00ff); Or you can do it with the bytes directly before casting
 Note the version identifiers BigEndian and LittleEndian can be used to
 compile the correct code.
This solution is of no use to me as I don't want to change the endianess in general.
What I mean is that you can annotate your code with version statements like: version(LittleEndian) { // perform the byteswap ... } so your code is portable to BigEndian systems (where you would not want to byte swap). -Steve
Jan 29
next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 29 Jan 2016 18:58:17 -0500
schrieb Steven Schveighoffer <schveiguy yahoo.com>:

 On 1/29/16 6:03 PM, Marek Janukowicz wrote:
 On Fri, 29 Jan 2016 17:43:26 -0500, Steven Schveighoffer wrote:  
 Is there anything I should know about UTF endianess?  
It's not any different from other endianness. In other words, a UTF16 code unit is expected to be in the endianness of the platform you are running on. If you are on x86 or x86_64 (very likely), then it should be little endian. If your source of data is big-endian (or opposite from your native endianness),
To be precise - my case is IMAP UTF7 folder name encoding and I finally found out it's indeed big endian, which explains my problem (as I'm indeed on x86_64).
 it will have to be converted before treating as a wchar[].  
Is there any clever way to do the conversion? Or do I need to swap the bytes manually?
No clever way, just the straightforward way ;) Swapping endianness of 32-bits can be done with core.bitop.bswap. Doing it with 16 bits I believe you have to do bit shifting. Something like: foreach(ref elem; wcharArr) elem = ((elem << 8) & 0xff00) | ((elem >> 8) & 0x00ff); Or you can do it with the bytes directly before casting
There's also a phobos solution: bigEndianToNative in std.bitmanip.
Jan 29
prev sibling parent Marek Janukowicz <marek janukowicz.net> writes:
On Fri, 29 Jan 2016 18:58:17 -0500, Steven Schveighoffer wrote:
 Note the version identifiers BigEndian and LittleEndian can be used to
 compile the correct code.
This solution is of no use to me as I don't want to change the endianess in general.
What I mean is that you can annotate your code with version statements like: version(LittleEndian) { // perform the byteswap ... } so your code is portable to BigEndian systems (where you would not want to byte swap).
That's a good point, thanks. -- Marek Janukowicz
Jan 30
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 29 January 2016 at 22:36:37 UTC, Marek Janukowicz 
wrote:
 I have trouble understanding how endianess works for UTF-16.
UTF-16 (as well as UTF-32) comes in both little-endian and big-endian variants. A byte-order marker in the file can help you detect which one it is in. See t his t able: http://www.unicode.org/faq/utf_bom.html#gen6
Jan 29