digitalmars.D.bugs - [Issue 12113] New: A nothrow std.utf.decode with substitution on bad encoding
- d-bugmail puremagic.com (41/41) Feb 08 2014 https://d.puremagic.com/issues/show_bug.cgi?id=12113
- d-bugmail puremagic.com (18/18) Feb 08 2014 https://d.puremagic.com/issues/show_bug.cgi?id=12113
- d-bugmail puremagic.com (23/24) Feb 08 2014 https://d.puremagic.com/issues/show_bug.cgi?id=12113
- d-bugmail puremagic.com (11/11) Mar 21 2014 https://d.puremagic.com/issues/show_bug.cgi?id=12113
- d-bugmail puremagic.com (6/6) Mar 24 2014 https://d.puremagic.com/issues/show_bug.cgi?id=12113
https://d.puremagic.com/issues/show_bug.cgi?id=12113 Summary: A nothrow std.utf.decode with substitution on bad encoding Product: D Version: D2 Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P2 Component: Phobos AssignedTo: nobody puremagic.com ReportedBy: dmitry.olsh gmail.com 14:26:46 PST --- Change the behaviour of decode according to Unicode standard recommendation. As a bonus dealing with partly broken encoding in the text becomes palatable which is hardly possible with the current behaviour. The relevant section of the standard: 5.22 Best Practice for U+FFFD Substitution When converting text from one character encoding to another, a conversion algorithm may encounter unconvertible code units. This is most commonly caused by some sort of corruption of the source data, so that it does not correctly follow the specification for that character encoding. Examples include dropping a byte in a multibyte encoding such as Shift-JIS, improper concatenation of strings, a mismatch between an encoding declaration and actual encoding of text, use of non-shortest form for UTF-8, and so on. ... Whenever an unconvertible offset is reached during conversion of a code unit sequence: 1. The maximal subpart at that offset should be replaced by a single U+FFFD. 2. The conversion should proceed at the offset immediately after the maximal subpart. --- -- Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Feb 08 2014
https://d.puremagic.com/issues/show_bug.cgi?id=12113 bearophile_hugs eml.cc changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bearophile_hugs eml.cc Is it possible to generalize this to layz ranges? Currently this can't be nothrow. Can it become nothrow using your suggestion? void main() nothrow { string txt = "hello"; foreach (dchar c; txt) {} } Currently it gives: test.d(3): Error: '_aApplycd1' is not nothrow test.d(1): Error: function 'D main' is nothrow yet may throw -- Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Feb 08 2014
https://d.puremagic.com/issues/show_bug.cgi?id=12113 Jonathan M Davis <jmdavisProg gmx.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jmdavisProg gmx.com PST ---Is it possible to generalize this to layz ranges?I should point out that that code doesn't use any range-based functions at all. However, if decode and stride are both altered to be nothrow, then any function which was not inferred as nothrow because of them would then be nothrow (assuming that template inference was working correctly, which is not entirely the case right now - it does particularly poorly with Voldemort types). So, there's a good chance that a lot of range-based string stuff would become nothrow. As for your example, that uses pure druntime stuff, as it's a string with foreach. And that code will need to be updated as well as std.utf.decode and std.utf.stride. However, theoretically, it will be in the same boat and be able to become nothrow, which would theoretically make your example nothrow, but I'd have to go digging through the code in druntime to see what all it does in order to be sure. I expect that it could be nothrow, but there might be something unexpected that it's doing that prevents it. -- Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Feb 08 2014
https://d.puremagic.com/issues/show_bug.cgi?id=12113 Walter Bright <bugzilla digitalmars.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bugzilla digitalmars.com 11:28:17 PDT --- The relevant spec: http://www.unicode.org/versions/Unicode6.2.0/UnicodeStandard-6.2.pdf -- Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Mar 21 2014
https://d.puremagic.com/issues/show_bug.cgi?id=12113 11:12:35 PDT --- https://github.com/D-Programming-Language/phobos/pull/2043 -- Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Mar 24 2014