www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Transparent ANSI to UTF-8 conversion

reply Lubos Pintes <lubos.pintes gmail.com> writes:
Hi,
I would like to transparently convert from ANSI to UTF-8 when dealing 
with text files. For example here in Slovakia, virtually every text file 
is in Windows-1250.
If someone opens a text file, he or she expects that it will work 
properly. So I suppose, that it is not feasible to tell someone "if you 
want to use my program, please convert every text to UTF-8".

To obtain the mapping from ANSI to Unicode for particular code page is 
trivial. Maybe even MultibyteToWidechar could help with this.

I however need to know how to do it "D-way". Could I define something 
like TextReader class? Or perhaps some support already exists somewhere?
Thank
Feb 27 2013
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 27 February 2013 at 10:56:16 UTC, Lubos Pintes 
wrote:

 Hi,
 I would like to transparently convert from ANSI to UTF-8 when 
 dealing with text files. For example here in Slovakia, 
 virtually every text file is in Windows-1250.
 If someone opens a text file, he or she expects that it will 
 work properly. So I suppose, that it is not feasible to tell 
 someone "if you want to use my program, please convert every 
 text to UTF-8".

 To obtain the mapping from ANSI to Unicode for particular code 
 page is trivial. Maybe even MultibyteToWidechar could help with 
 this.

 I however need to know how to do it "D-way". Could I define 
 something like TextReader class? Or perhaps some support 
 already exists somewhere?
 Thank


I'd say the D way would be to simply exploit the fact that UTF is 
built into the language, and as such, not worry about encoding, 
and use raw code points.

You get you "Codepage to unicode *codepoint*" table, and then you 
simply map each character to a dchar. From there, D will itself 
convert your raw unicode (aka UTF-32) to UTF8 on the fly, when 
you need it. For example, writing to a file will automatically 
convert input to UTF-8. You can also simply use 
std.conv.to!string to convert any UTF scheme to UTF-8 (or any 
other UTF too for that matter).

This may not be as efficient as a "true" "codepage to UTF8 table" 
but:
1) Given you'll most probably be IO bound anyways, who cares?
2) Scalability. D does everything but the code page to code point 
mapping. Why bother doing any more than that?
Feb 27 2013
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
27-Feb-2013 16:20, monarch_dodra пишет:

 On Wednesday, 27 February 2013 at 10:56:16 UTC, Lubos Pintes wrote:

 Hi,
 I would like to transparently convert from ANSI to UTF-8 when dealing
 with text files. For example here in Slovakia, virtually every text
 file is in Windows-1250.
 If someone opens a text file, he or she expects that it will work
 properly. So I suppose, that it is not feasible to tell someone "if
 you want to use my program, please convert every text to UTF-8".

 To obtain the mapping from ANSI to Unicode for particular code page is
 trivial. Maybe even MultibyteToWidechar could help with this.

 I however need to know how to do it "D-way". Could I define something
 like TextReader class? Or perhaps some support already exists somewhere?
 Thank


 I'd say the D way would be to simply exploit the fact that UTF is built
 into the language, and as such, not worry about encoding, and use raw
 code points.

 You get you "Codepage to unicode *codepoint*" table, and then you simply
 map each character to a dchar. From there, D will itself convert your
 raw unicode (aka UTF-32) to UTF8 on the fly, when you need it. For
 example, writing to a file will automatically convert input to UTF-8.
 You can also simply use std.conv.to!string to convert any UTF scheme to
 UTF-8 (or any other UTF too for that matter).


Making a table that translates ANSI to UTF8 is trivially constructible 
using CTFE from the static one that does ANSI -> dchar.

 This may not be as efficient as a "true" "codepage to UTF8 table" but:
 1) Given you'll most probably be IO bound anyways, who cares?


With in-memory transcoding you won't be. Text editors are typically all 
in-memory or mmap-ed.


 2) Scalability. D does everything but the code page to code point
 mapping. Why bother doing any more than that?



-- 
Dmitry Olshansky
Feb 27 2013
parent reply Lubos Pintes <lubos.pintes gmail.com> writes:
I don't understand the CTFE usage in this context. I thought about 
something like
dchar[] windows_1250=[...];
Isn't this enough?
Thank

Dňa 27. 2. 2013 18:32 Dmitry Olshansky  wrote / napísal(a):

 27-Feb-2013 16:20, monarch_dodra пишет:

 On Wednesday, 27 February 2013 at 10:56:16 UTC, Lubos Pintes wrote:

 Hi,
 I would like to transparently convert from ANSI to UTF-8 when dealing
 with text files. For example here in Slovakia, virtually every text
 file is in Windows-1250.
 If someone opens a text file, he or she expects that it will work
 properly. So I suppose, that it is not feasible to tell someone "if
 you want to use my program, please convert every text to UTF-8".

 To obtain the mapping from ANSI to Unicode for particular code page is
 trivial. Maybe even MultibyteToWidechar could help with this.

 I however need to know how to do it "D-way". Could I define something
 like TextReader class? Or perhaps some support already exists somewhere?
 Thank


 I'd say the D way would be to simply exploit the fact that UTF is built
 into the language, and as such, not worry about encoding, and use raw
 code points.

 You get you "Codepage to unicode *codepoint*" table, and then you simply
 map each character to a dchar. From there, D will itself convert your
 raw unicode (aka UTF-32) to UTF8 on the fly, when you need it. For
 example, writing to a file will automatically convert input to UTF-8.
 You can also simply use std.conv.to!string to convert any UTF scheme to
 UTF-8 (or any other UTF too for that matter).


 Making a table that translates ANSI to UTF8 is trivially constructible
 using CTFE from the static one that does ANSI -> dchar.

 This may not be as efficient as a "true" "codepage to UTF8 table" but:
 1) Given you'll most probably be IO bound anyways, who cares?


 With in-memory transcoding you won't be. Text editors are typically all
 in-memory or mmap-ed.


 2) Scalability. D does everything but the code page to code point
 mapping. Why bother doing any more than that?
Feb 27 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
28-Feb-2013 00:35, Lubos Pintes пишет:

 I don't understand the CTFE usage in this context. I thought about
 something like
 dchar[] windows_1250=[...];
 Isn't this enough?
 Thank


It's fine. What I've meant is if all you want to do is convert ANSI -> 
UTF8 there is no need to convert to dchar and then to UTF-8 chars.

so the table becomes more like:

char[][] windows_1250_to_UTF8 = [...];

Or rather (far better memory footprint):

char[2][] windows_1250_to_UTF8 = [ ... ];

I think 2 UTF-8 chars should be enough for your codepage.

Then CTFE is just a tool create one table from another :
char[2][] windows_1250UTF = createUTF8Table(windows_1250);

The point is that inside of createUTF8Table you create an array it by 
using new and simple loops + std.utf.encode just like in normal code but 
it'll be CTFE-ed.

Same goes for going backwards - you can treat char[2] as ushort and do 
the tables. Though now it may have gaps due to encoding not being linear 
but rather having some stride with certain period.


 Dňa 27. 2. 2013 18:32 Dmitry Olshansky  wrote / napísal(a):

 27-Feb-2013 16:20, monarch_dodra пишет:

 On Wednesday, 27 February 2013 at 10:56:16 UTC, Lubos Pintes wrote:

 Hi,
 I would like to transparently convert from ANSI to UTF-8 when dealing
 with text files. For example here in Slovakia, virtually every text
 file is in Windows-1250.
 If someone opens a text file, he or she expects that it will work
 properly. So I suppose, that it is not feasible to tell someone "if
 you want to use my program, please convert every text to UTF-8".

 To obtain the mapping from ANSI to Unicode for particular code page is
 trivial. Maybe even MultibyteToWidechar could help with this.

 I however need to know how to do it "D-way". Could I define something
 like TextReader class? Or perhaps some support already exists
 somewhere?
 Thank


 I'd say the D way would be to simply exploit the fact that UTF is built
 into the language, and as such, not worry about encoding, and use raw
 code points.

 You get you "Codepage to unicode *codepoint*" table, and then you simply
 map each character to a dchar. From there, D will itself convert your
 raw unicode (aka UTF-32) to UTF8 on the fly, when you need it. For
 example, writing to a file will automatically convert input to UTF-8.
 You can also simply use std.conv.to!string to convert any UTF scheme to
 UTF-8 (or any other UTF too for that matter).


 Making a table that translates ANSI to UTF8 is trivially constructible
 using CTFE from the static one that does ANSI -> dchar.

 This may not be as efficient as a "true" "codepage to UTF8 table" but:
 1) Given you'll most probably be IO bound anyways, who cares?


 With in-memory transcoding you won't be. Text editors are typically all
 in-memory or mmap-ed.


 2) Scalability. D does everything but the code page to code point
 mapping. Why bother doing any more than that?







-- 
Dmitry Olshansky
Feb 27 2013
prev sibling parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Wednesday, 27 February 2013 at 10:56:16 UTC, Lubos Pintes 
wrote:

 Hi,
 I would like to transparently convert from ANSI to UTF-8 when 
 dealing with text files. For example here in Slovakia, 
 virtually every text file is in Windows-1250. If someone opens 
 a text file, he or she expects that it will work properly. So I 
 suppose, that it is not feasible to tell someone "if you want 
 to use my program, please convert every text to UTF-8".


  A while back I wrote a little code that effectively does that, 
mind you it's probably not the right specific encoding, however 
you should be able to find the code points and replace them. I 
think this was for iso-8859-1.

See "Reading ASCII file with some codes above 127 (exten ascii)"

http://forum.dlang.org/thread/lehgyzmwewgvkdgraizv forum.dlang.org
Feb 27 2013
parent Lubos Pintes <lubos.pintes gmail.com> writes:
Thank you all. Now I believe I will be able to solve this.
Dòa 28. 2. 2013 5:25 Era Scarecrow  wrote / napísal(a):

 On Wednesday, 27 February 2013 at 10:56:16 UTC, Lubos Pintes wrote:

 Hi,
 I would like to transparently convert from ANSI to UTF-8 when dealing
 with text files. For example here in Slovakia, virtually every text
 file is in Windows-1250. If someone opens a text file, he or she
 expects that it will work properly. So I suppose, that it is not
 feasible to tell someone "if you want to use my program, please
 convert every text to UTF-8".


   A while back I wrote a little code that effectively does that, mind
 you it's probably not the right specific encoding, however you should be
 able to find the code points and replace them. I think this was for
 iso-8859-1.

 See "Reading ASCII file with some codes above 127 (exten ascii)"

 http://forum.dlang.org/thread/lehgyzmwewgvkdgraizv forum.dlang.org
Feb 28 2013