www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Creeping Bloat in Phobos

reply Walter Bright <newshound2 digitalmars.com> writes:
 From time to time, I take a break from bugs and enhancements and just look at 
what some piece of code is actually doing. Sometimes, I'm appalled. Phobos, for 
example, should be a lean and mean fighting machine:

 
http://www.nbcnews.com/id/38545625/ns/technology_and_science-science/t/king-tuts-chariots-were-formula-one-cars/#.VCceNmd0xjs

Instead, we have something more akin to:

 
http://untappedcities.com/2012/10/31/roulez-carrosses-carriages-of-versailles-arrive-in-arras/

More specifically, I looked at std.file.copy():

   https://github.com/D-Programming-Language/phobos/blob/master/std/file.d

Which is 3 lines of code:

   void copy(in char[] from, in char[] to) {
         immutable result = CopyFileW(from.tempCStringW(), to.tempCStringW(), 
false);
         if (!result)
             throw new FileException(to.idup);
   }

Compiling this code for Windows produces the rather awful:

_D3std4file4copyFxAaxAaZv       comdat
         assume  CS:_D3std4file4copyFxAaxAaZv
L0:             push    EBP
                 mov     EBP,ESP
                 mov     EDX,FS:__except_list
                 push    0FFFFFFFFh
                 lea     EAX,-0220h[EBP]
                 push    offset _D3std4file4copyFxAaxAaZv[0106h]
                 push    EDX
                 mov     FS:__except_list,ESP
                 sub     ESP,8
                 sub     ESP,041Ch
                 push    0
                 push    dword ptr 0Ch[EBP]
                 push    dword ptr 8[EBP]
                 call    near ptr 
_D3std8internal7cstring21__T11tempCSÇàÆTuTaZÇìÆFNbNixAaZSÇ┬├3Res
                 mov     dword ptr -4[EBP],0
                 lea     EAX,-0220h[EBP]
                 call    near ptr 
_D3std8internal7cstring21__T11tempCStringTuTaZ11tempCStringFNbNixAaZ3Res3ptrMxFNaNbNdNiNfZPxu
                 push    EAX
                 lea     EAX,-0430h[EBP]
                 push    dword ptr 014h[EBP]
                 push    dword ptr 010h[EBP]
                 call    near ptr 
_D3std8internal7cstring21__T11tempCSÇàÆTuTaZÇìÆFNbNixAaZSÇ┬├3Res
                 mov     dword ptr -4[EBP],1
                 lea     EAX,-0430h[EBP]
                 call    near ptr 
_D3std8internal7cstring21__T11tempCStringTuTaZ11tempCStringFNbNixAaZ3Res3ptrMxFNaNbNdNiNfZPxu
                 push    EAX
                 call    dword ptr __imp__CopyFileW 12
                 mov     -01Ch[EBP],EAX
                 mov     dword ptr -4[EBP],0
                 call    near ptr L83
                 jmp short       L8F
L83:            lea     EAX,-0220h[EBP]
                 call    near ptr 
_D3std8internal7cstring21__T11tempCStringTuTaZ11tempCStringFNbNixAaZ3Res6__dtorMFNbNiZv
                 ret
L8F:            mov     dword ptr -4[EBP],0FFFFFFFFh
                 call    near ptr L9D
                 jmp short       LA9
L9D:            lea     EAX,-0430h[EBP]
                 call    near ptr 
_D3std8internal7cstring21__T11tempCStringTuTaZ11tempCStringFNbNixAaZ3Res6__dtorMFNbNiZv
                 ret
LA9:            cmp     dword ptr -01Ch[EBP],0
                 jne     LF3
                 mov     ECX,offset FLAT:_D3std4file13FileException7__ClassZ
                 push    ECX
                 call    near ptr __d_newclass
                 add     ESP,4
                 push    dword ptr 0Ch[EBP]
                 mov     -018h[EBP],EAX
                 push    dword ptr 8[EBP]
                 call    near ptr _D6object12__T4idupTxaZ4idupFNaNbNdNfAxaZAya
                 push    EDX
                 push    EAX
                 call    dword ptr __imp__GetLastError 0
                 push    EAX
                 push    dword ptr _D3std4file13FileException6__vtblZ[02Ch]
                 push    dword ptr _D3std4file13FileException6__vtblZ[028h]
                 push    095Dh
                 mov     EAX,-018h[EBP]
                 call    near ptr 
_D3std4file13FileException6__ctorMFNfxAakAyakZC3std4file13FileException
                 push    EAX
                 call    near ptr __d_throwc
LF3:            mov     ECX,-0Ch[EBP]
                 mov     FS:__except_list,ECX
                 mov     ESP,EBP
                 pop     EBP
                 ret     010h
                 mov     EAX,offset
FLAT:_D3std4file13FileException6__vtblZ[0310h]
                 jmp     near ptr __d_framehandler

which is TWICE as much generated code as for D1's copy(), which does the same 
thing. No, it is not because D2's compiler sux. It's because it has become 
encrustified with gee-gaws, jewels, decorations, and other crap.

To scrape the barnacles off, I've filed:

https://issues.dlang.org/show_bug.cgi?id=13541
https://issues.dlang.org/show_bug.cgi?id=13542
https://issues.dlang.org/show_bug.cgi?id=13543
https://issues.dlang.org/show_bug.cgi?id=13544

I'm sure there's much more in std.file (and elsewhere) that can be done. Guys, 
when developing Phobos/Druntime code, please look at the assembler once in a 
while and see what is being wrought. You may be appalled, too.
Sep 27 2014
next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 27 September 2014 at 20:57:53 UTC, Walter Bright 
wrote:
 From time to time, I take a break from bugs and enhancements 
 and just look at what some piece of code is actually doing. 
 Sometimes, I'm appalled.
Me too, and yes it can be appalling. It's pretty bad for even simple range chains, e.g. import std.algorithm, std.stdio; int main(string[] args) { return cast(int)args.map!("a.length").reduce!"a+b"(); } Here's what LDC produces (with -O -inline -release -noboundscheck) __Dmain: 0000000100001480 pushq %r15 0000000100001482 pushq %r14 0000000100001484 pushq %rbx 0000000100001485 movq %rsi, %rbx 0000000100001488 movq %rdi, %r14 000000010000148b callq 0x10006df10 ## symbol stub for: __D3std5array14__T5emptyTAyaZ5emptyFNaNbNdNfxAAyaZb 0000000100001490 xorb $0x1, %al 0000000100001492 movzbl %al, %r9d 0000000100001496 leaq _.str12(%rip), %rdx ## literal pool for: "/Users/pja/ldc2-0.14.0-osx-x86_64/bin/../import/std/algorithm.d" 000000010000149d movq 0xcbd2c(%rip), %r8 ## literal pool symbol address: __D3std9algorithm24__T6reduceVAyaa3_612b62Z124__T6reduceTS3std9algorithm85__T9MapResultS633std10functional36__T8unaryFunVAyaa8_612e6c656e677468Z8unaryFunTAAyaZ9MapResultZ6reduceFNaNfS3std9algorithm85__T 00000001000014a4 movl $0x2dd, %edi 00000001000014a9 movl $0x3f, %esi 00000001000014ae xorl %ecx, %ecx 00000001000014b0 callq 0x10006e0a2 ## symbol stub for: __D3std9exception14__T7enforceTbZ7enforceFNaNfbLAxaAyamZb 00000001000014b5 movq (%rbx), %r15 00000001000014b8 leaq 0x10(%rbx), %rsi 00000001000014bc leaq -0x1(%r14), %rdi 00000001000014c0 callq 0x10006df10 ## symbol stub for: __D3std5array14__T5emptyTAyaZ5emptyFNaNbNdNfxAAyaZb 00000001000014c5 testb $0x1, %al 00000001000014c7 jne 0x1000014fa 00000001000014c9 addq $-0x2, %r14 00000001000014cd addq $0x20, %rbx 00000001000014d1 nopw %cs:(%rax,%rax) 00000001000014e0 addq -0x10(%rbx), %r15 00000001000014e4 movq %r14, %rdi 00000001000014e7 movq %rbx, %rsi 00000001000014ea callq 0x10006df10 ## symbol stub for: __D3std5array14__T5emptyTAyaZ5emptyFNaNbNdNfxAAyaZb 00000001000014ef decq %r14 00000001000014f2 addq $0x10, %rbx 00000001000014f6 testb $0x1, %al 00000001000014f8 je 0x1000014e0 00000001000014fa movl %r15d, %eax 00000001000014fd popq %rbx 00000001000014fe popq %r14 0000000100001500 popq %r15 0000000100001502 ret and for: import std.algorithm, std.stdio; int main(string[] args) { int r = 0; foreach (i; 0..args.length) r += args[i].length; return r; } __Dmain: 00000001000015c0 xorl %eax, %eax 00000001000015c2 testq %rdi, %rdi 00000001000015c5 je 0x1000015de 00000001000015c7 nopw (%rax,%rax) 00000001000015d0 movl %eax, %eax 00000001000015d2 addq (%rsi), %rax 00000001000015d5 addq $0x10, %rsi 00000001000015d9 decq %rdi 00000001000015dc jne 0x1000015d0 00000001000015de ret (and sorry, don't even bother looking at what dmd does...) I'm not complaining about LDC here (although I'm surprised array.empty isn't inlined). The way ranges are formulated make them difficult to optimize. I think there's things we can do here in the library. Maybe I'll write up something about that at some point. I think the takeaway here is that people should be aware of (a) what kind of instructions their code is generating, (b) what kind of instructions their code SHOULD be generating, and (c) what is practically possible for present-day compilers. Like you say, it helps to look at the assembled code once in a while to get a feel for this kind of thing. Modern compilers are good, but they aren't magic.
Sep 27 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Sep 27, 2014 at 09:59:17PM +0000, Peter Alexander via Digitalmars-d
wrote:
 On Saturday, 27 September 2014 at 20:57:53 UTC, Walter Bright wrote:
From time to time, I take a break from bugs and enhancements and just
look at what some piece of code is actually doing. Sometimes, I'm
appalled.
Me too, and yes it can be appalling. It's pretty bad for even simple range chains, e.g. import std.algorithm, std.stdio; int main(string[] args) { return cast(int)args.map!("a.length").reduce!"a+b"(); }
I vaguely recall somebody mentioning a while back that range-based code is poorly optimized because compilers weren't designed to recognize such patterns before. I wonder if there are ways for the compiler to recognize range primitives and apply special optimizations to them. I do find, though, that gdc -O3 generally tends to do a pretty good job of reducing range-based code to near-minimal assembly. Sadly, dmd is changing too fast for gdc releases to catch up with the latest and greatest, so I haven't been using gdc very much recently. :-( T -- If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell
Sep 27 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 27 September 2014 at 22:11:39 UTC, H. S. Teoh via 
Digitalmars-d wrote:
 I vaguely recall somebody mentioning a while back that 
 range-based code
 is poorly optimized because compilers weren't designed to 
 recognize
 such patterns before. I wonder if there are ways for the 
 compiler to
 recognize range primitives and apply special optimizations to 
 them.

 I do find, though, that gdc -O3 generally tends to do a pretty 
 good job
 of reducing range-based code to near-minimal assembly. Sadly, 
 dmd is
 changing too fast for gdc releases to catch up with the latest 
 and
 greatest, so I haven't been using gdc very much recently. :-(
That was me, specifically for LLVM (I don't know much about GCC's innards). Hopefully, this is being worked (as it also impact C++'s stdlib).
Sep 27 2014
prev sibling next sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
What we're seeing here is pretty much the same problem that early c++ 
suffered from: abstraction penalty.  It took years of work to help 
overcome it, both from the compiler and the library.  Not having trivial 
functions inlined and optimized down through standard techniques like 
dead store elimination, value range propagation, various loop 
restructurings, etc means that code will look like what Walter and you 
have shown.  Given DMD's relatively weak inliner, I'm not shocked by 
Walter's example.  I am curious why ldc failed to inline those functions.

On 9/27/2014 2:59 PM, Peter Alexander via Digitalmars-d wrote:
 On Saturday, 27 September 2014 at 20:57:53 UTC, Walter Bright wrote:
 From time to time, I take a break from bugs and enhancements and just
 look at what some piece of code is actually doing. Sometimes, I'm
 appalled.
Me too, and yes it can be appalling. It's pretty bad for even simple range chains, e.g. import std.algorithm, std.stdio; int main(string[] args) { return cast(int)args.map!("a.length").reduce!"a+b"(); } Here's what LDC produces (with -O -inline -release -noboundscheck) __Dmain: 0000000100001480 pushq %r15 0000000100001482 pushq %r14 0000000100001484 pushq %rbx 0000000100001485 movq %rsi, %rbx 0000000100001488 movq %rdi, %r14 000000010000148b callq 0x10006df10 ## symbol stub for: __D3std5array14__T5emptyTAyaZ5emptyFNaNbNdNfxAAyaZb 0000000100001490 xorb $0x1, %al 0000000100001492 movzbl %al, %r9d 0000000100001496 leaq _.str12(%rip), %rdx ## literal pool for: "/Users/pja/ldc2-0.14.0-osx-x86_64/bin/../import/std/algorithm.d" 000000010000149d movq 0xcbd2c(%rip), %r8 ## literal pool symbol address: __D3std9algorithm24__T6reduceVAyaa3_612b62Z124__T6reduceTS3std9algorithm85__T9MapResultS633std10functional36__T8unaryFunVAyaa8_612e6c656e677468Z8unaryFunTAAyaZ9MapResultZ6reduceFNaNfS3std9algorithm85__T 00000001000014a4 movl $0x2dd, %edi 00000001000014a9 movl $0x3f, %esi 00000001000014ae xorl %ecx, %ecx 00000001000014b0 callq 0x10006e0a2 ## symbol stub for: __D3std9exception14__T7enforceTbZ7enforceFNaNfbLAxaAyamZb 00000001000014b5 movq (%rbx), %r15 00000001000014b8 leaq 0x10(%rbx), %rsi 00000001000014bc leaq -0x1(%r14), %rdi 00000001000014c0 callq 0x10006df10 ## symbol stub for: __D3std5array14__T5emptyTAyaZ5emptyFNaNbNdNfxAAyaZb 00000001000014c5 testb $0x1, %al 00000001000014c7 jne 0x1000014fa 00000001000014c9 addq $-0x2, %r14 00000001000014cd addq $0x20, %rbx 00000001000014d1 nopw %cs:(%rax,%rax) 00000001000014e0 addq -0x10(%rbx), %r15 00000001000014e4 movq %r14, %rdi 00000001000014e7 movq %rbx, %rsi 00000001000014ea callq 0x10006df10 ## symbol stub for: __D3std5array14__T5emptyTAyaZ5emptyFNaNbNdNfxAAyaZb 00000001000014ef decq %r14 00000001000014f2 addq $0x10, %rbx 00000001000014f6 testb $0x1, %al 00000001000014f8 je 0x1000014e0 00000001000014fa movl %r15d, %eax 00000001000014fd popq %rbx 00000001000014fe popq %r14 0000000100001500 popq %r15 0000000100001502 ret and for: import std.algorithm, std.stdio; int main(string[] args) { int r = 0; foreach (i; 0..args.length) r += args[i].length; return r; } __Dmain: 00000001000015c0 xorl %eax, %eax 00000001000015c2 testq %rdi, %rdi 00000001000015c5 je 0x1000015de 00000001000015c7 nopw (%rax,%rax) 00000001000015d0 movl %eax, %eax 00000001000015d2 addq (%rsi), %rax 00000001000015d5 addq $0x10, %rsi 00000001000015d9 decq %rdi 00000001000015dc jne 0x1000015d0 00000001000015de ret (and sorry, don't even bother looking at what dmd does...) I'm not complaining about LDC here (although I'm surprised array.empty isn't inlined). The way ranges are formulated make them difficult to optimize. I think there's things we can do here in the library. Maybe I'll write up something about that at some point. I think the takeaway here is that people should be aware of (a) what kind of instructions their code is generating, (b) what kind of instructions their code SHOULD be generating, and (c) what is practically possible for present-day compilers. Like you say, it helps to look at the assembled code once in a while to get a feel for this kind of thing. Modern compilers are good, but they aren't magic.
Sep 27 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/27/2014 3:26 PM, Brad Roberts via Digitalmars-d wrote:
 What we're seeing here is pretty much the same problem that early c++ suffered
 from: abstraction penalty.  It took years of work to help overcome it, both
from
 the compiler and the library.  Not having trivial functions inlined and
 optimized down through standard techniques like dead store elimination, value
 range propagation, various loop restructurings, etc means that code will look
 like what Walter and you have shown.  Given DMD's relatively weak inliner, I'm
 not shocked by Walter's example.  I am curious why ldc failed to inline those
 functions.
Again, this accumulation of barnacles is not a failure of the optimizer. It's a failure of adding gee-gaws to the source code without checking their effect.
Sep 27 2014
next sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/27/2014 3:54 PM, Walter Bright via Digitalmars-d wrote:
 On 9/27/2014 3:26 PM, Brad Roberts via Digitalmars-d wrote:
 What we're seeing here is pretty much the same problem that early c++
 suffered
 from: abstraction penalty.  It took years of work to help overcome it,
 both from
 the compiler and the library.  Not having trivial functions inlined and
 optimized down through standard techniques like dead store
 elimination, value
 range propagation, various loop restructurings, etc means that code
 will look
 like what Walter and you have shown.  Given DMD's relatively weak
 inliner, I'm
 not shocked by Walter's example.  I am curious why ldc failed to
 inline those
 functions.
Again, this accumulation of barnacles is not a failure of the optimizer. It's a failure of adding gee-gaws to the source code without checking their effect.
Look at Peter's example, it's better for this, I believe. Why isn't empty being inlined? That's a tiny little function with a lot of impact. Of course there's more than just optimization, but it's a big player in the game too.
Sep 27 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/27/2014 3:59 PM, Brad Roberts via Digitalmars-d wrote:
 Look at Peter's example, it's better for this, I believe.  Why isn't empty
being
 inlined?  That's a tiny little function with a lot of impact.
It's the autodecode'ing front(), which is a fairly complex function.
Sep 27 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/14, 4:02 PM, Walter Bright wrote:
 On 9/27/2014 3:59 PM, Brad Roberts via Digitalmars-d wrote:
 Look at Peter's example, it's better for this, I believe.  Why isn't
 empty being
 inlined?  That's a tiny little function with a lot of impact.
It's the autodecode'ing front(), which is a fairly complex function.
front() should follow a simple pattern that's been very successful in HHVM: small inline function that covers most cases with "if (c < 0x80)" followed by an out-of-line function on the multicharacter case. That approach would make the cost of auto-decoding negligible in the overwhelming majority of cases. Andrei
Sep 27 2014
parent Martin Nowak <code dawg.eu> writes:
On 09/28/2014 02:23 AM, Andrei Alexandrescu wrote:
 front() should follow a simple pattern that's been very successful in
 HHVM: small inline function that covers most cases with "if (c < 0x80)"
 followed by an out-of-line function on the multicharacter case. That
 approach would make the cost of auto-decoding negligible in the
 overwhelming majority of cases.

 Andrei
Well, we're using the same trick for already 3 years now :). https://github.com/D-Programming-Language/phobos/pull/299
Sep 28 2014
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 09/28/2014 01:02 AM, Walter Bright wrote:
 It's the autodecode'ing front(), which is a fairly complex function.
At least for dmd it's caused by a long-standing compiler bug. https://issues.dlang.org/show_bug.cgi?id=7625 https://github.com/D-Programming-Language/phobos/pull/2566
Sep 28 2014
prev sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Saturday, 27 September 2014 at 23:00:20 UTC, Brad Roberts via 
Digitalmars-d wrote:
 Look at Peter's example, it's better for this, I believe.  Why 
 isn't empty being inlined?  That's a tiny little function with 
 a lot of impact.
This is most likely due to an issue with how the new DMD template emission strategy (needsCodegen() et al.) was integrated into LDC: https://github.com/ldc-developers/ldc/issues/674 The issue in question was fixed recently in LDC Git master, but regressed again when 2.066 was merged. David
Sep 27 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/14, 3:54 PM, Walter Bright wrote:
 Again, this accumulation of barnacles is not a failure of the optimizer.
 It's a failure of adding gee-gaws to the source code without checking
 their effect.
The Go project has something nice set up - easy-to-run benchmarks that are part of the acceptance testing. That's good prevention against creeping inefficiencies. -- Andrei
Sep 27 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/27/2014 2:59 PM, Peter Alexander wrote:
 On Saturday, 27 September 2014 at 20:57:53 UTC, Walter Bright wrote:
 From time to time, I take a break from bugs and enhancements and just look at
 what some piece of code is actually doing. Sometimes, I'm appalled.
Me too, and yes it can be appalling. It's pretty bad for even simple range chains, e.g. import std.algorithm, std.stdio; int main(string[] args) { return cast(int)args.map!("a.length").reduce!"a+b"(); } Here's what LDC produces (with -O -inline -release -noboundscheck)
Part of this particular case problem is not a compiler optimizer weakness, but that autodecode problem I've been throwing (!) chairs through windows on.
Sep 27 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Sep 27, 2014 at 03:26:35PM -0700, Walter Bright via Digitalmars-d wrote:
 On 9/27/2014 2:59 PM, Peter Alexander wrote:
On Saturday, 27 September 2014 at 20:57:53 UTC, Walter Bright wrote:
From time to time, I take a break from bugs and enhancements and
just look at what some piece of code is actually doing. Sometimes,
I'm appalled.
Me too, and yes it can be appalling. It's pretty bad for even simple range chains, e.g. import std.algorithm, std.stdio; int main(string[] args) { return cast(int)args.map!("a.length").reduce!"a+b"(); } Here's what LDC produces (with -O -inline -release -noboundscheck)
Part of this particular case problem is not a compiler optimizer weakness, but that autodecode problem I've been throwing (!) chairs through windows on.
If we can get Andrei on board, I'm all for killing off autodecoding. T -- MAS = Mana Ada Sistem?
Sep 27 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
H. S. Teoh:

 If we can get Andrei on board, I'm all for killing off 
 autodecoding.
Killing auto-decoding for std.algorithm functions will break most of my D2 code... perhaps we can do that in a D3 language. Bye, bearophile
Sep 27 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Sep 27, 2014 at 11:00:16PM +0000, bearophile via Digitalmars-d wrote:
 H. S. Teoh:
 
If we can get Andrei on board, I'm all for killing off autodecoding.
Killing auto-decoding for std.algorithm functions will break most of my D2 code... perhaps we can do that in a D3 language.
[...] Well, obviously it's not going to be done in a careless, drastic way! There will be a proper migration path and deprecation cycle. We already have byCodeUnit and byCodePoint, and the first step is probably to migrate towards requiring usage of one or the other for iterating over strings, and only once all code is using them, we will get rid of autodecoding (the job now being done by byCodePoint). Then, the final step would be to allow the direct use of strings in iteration constructs again, but this time without autodecoding by default. Of course, .byCodePoint will still be available for code that needs to use it. T -- Ph.D. = Permanent head Damage
Sep 27 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 27 September 2014 at 23:33:14 UTC, H. S. Teoh via 
Digitalmars-d wrote:
 On Sat, Sep 27, 2014 at 11:00:16PM +0000, bearophile via 
 Digitalmars-d wrote:
 H. S. Teoh:
 
If we can get Andrei on board, I'm all for killing off 
autodecoding.
Killing auto-decoding for std.algorithm functions will break most of my D2 code... perhaps we can do that in a D3 language.
[...] Well, obviously it's not going to be done in a careless, drastic way! There will be a proper migration path and deprecation cycle. We already have byCodeUnit and byCodePoint, and the first step is probably to migrate towards requiring usage of one or the other for iterating over strings, and only once all code is using them, we will get rid of autodecoding (the job now being done by byCodePoint). Then, the final step would be to allow the direct use of strings in iteration constructs again, but this time without autodecoding by default. Of course, .byCodePoint will still be available for code that needs to use it.
The final step would almost inevitably lead to Unicode incorrectness, which was the reason why autodecoding was introduced in the first place. Just require byCodePoint/byCodeUnit, always. It might be a bit inconvenient, but that's a consequence of the fact that we're dealing with Unicode strings.
Sep 28 2014
next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 28 Sep 2014 10:04:21 +0000
schrieb "Marc Sch=C3=BCtz" <schuetzm gmx.net>:

 On Saturday, 27 September 2014 at 23:33:14 UTC, H. S. Teoh via=20
 Digitalmars-d wrote:
 On Sat, Sep 27, 2014 at 11:00:16PM +0000, bearophile via=20
 Digitalmars-d wrote:
 H. S. Teoh:
=20
If we can get Andrei on board, I'm all for killing off=20
autodecoding.
=20 Killing auto-decoding for std.algorithm functions will break=20 most of my D2 code... perhaps we can do that in a D3 language.
[...] Well, obviously it's not going to be done in a careless,=20 drastic way! There will be a proper migration path and deprecation cycle. We=20 already have byCodeUnit and byCodePoint, and the first step is probably=20 to migrate towards requiring usage of one or the other for=20 iterating over strings, and only once all code is using them, we will get rid=20 of autodecoding (the job now being done by byCodePoint). Then, the=20 final step would be to allow the direct use of strings in iteration=20 constructs again, but this time without autodecoding by default. Of course, .byCodePoint will still be available for code that needs to use=20 it.
=20 The final step would almost inevitably lead to Unicode=20 incorrectness, which was the reason why autodecoding was=20 introduced in the first place. Just require=20 byCodePoint/byCodeUnit, always. It might be a bit inconvenient,=20 but that's a consequence of the fact that we're dealing with=20 Unicode strings.
And I would go so far to say that you have to make an informed decision between code unit, code point and grapheme. They are all useful. Graphemes being the most generally useful, hiding away normalization and allowing cutting by "user perceived character". --=20 Marco
Sep 28 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/28/14, 3:04 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 The final step would almost inevitably lead to Unicode incorrectness,
 which was the reason why autodecoding was introduced in the first place.
 Just require byCodePoint/byCodeUnit, always. It might be a bit
 inconvenient, but that's a consequence of the fact that we're dealing
 with Unicode strings.
Also let's not forget how well it's worked for C++ to conflate arrays of char with Unicode strings. -- Andrei
Sep 28 2014
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
H. S. Teoh:

 There will be a proper migration path and deprecation cycle.
I get refusals if I propose tiny breaking changes that require changes in a small amount of user code. In comparison the user code changes you are suggesting are very large. Bye, bearophile
Sep 28 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 3:14 AM, bearophile wrote:
 I get refusals if I propose tiny breaking changes that require changes in a
 small amount of user code.  In comparison the user code changes you are
 suggesting are very large.
I'm painfully aware of what a large change removing autodecoding is. That means it'll take a long time to do it. In the meantime, we can stop adding new code to Phobos that does autodecoding. We have taken the first step by adding the .byDchar and .byCodeUnit adapters.
Sep 28 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 I'm painfully aware of what a large change removing 
 autodecoding is. That means it'll take a long time to do it. In 
 the meantime, we can stop adding new code to Phobos that does 
 autodecoding. We have taken the first step by adding the 
 .byDchar and .byCodeUnit adapters.
We have .representation and .assumeUTF, I am using it to avoid most autodecoding problems. Have you tried to use them in your D code? The changes you propose seem able to break almost every D program I have written (most or all code that uses strings with Phobos ranges/algorithms, and I use them everywhere). Compared to this change, disallowing comma operator to implement nice built-in tuples will cause nearly no breakage in my code (I have done a small analysis of the damages caused by disallowing the tuple operator in my code). It sounds like a change fit for a D3 language, even more than the introduction of reference counting. I think this change will cause some people to permanently stop using D. In the end you are the designer and the benevolent dictator of D, I am not qualified to refuse or oppose such changes. But before doing this change I suggest to study how many changes it causes in an average small D program that uses strings and ranges/algorithms. Bye, bearophile
Sep 28 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 11:39 AM, bearophile wrote:
 Walter Bright:

 I'm painfully aware of what a large change removing autodecoding is. That
 means it'll take a long time to do it. In the meantime, we can stop adding new
 code to Phobos that does autodecoding. We have taken the first step by adding
 the .byDchar and .byCodeUnit adapters.
We have .representation and .assumeUTF, I am using it to avoid most autodecoding problems. Have you tried to use them in your D code?
Yes. They don't work. Well, technically they do "work", but your code gets filled with explicit casts, which is awful. The problem is the "representation" of char[] is type char, not type ubyte.
 The changes you propose seem able to break almost every D program I have
written
 (most or all code that uses strings with Phobos ranges/algorithms, and I use
 them everywhere). Compared to this change, disallowing comma operator to
 implement nice built-in tuples will cause nearly no breakage in my code (I have
 done a small analysis of the damages caused by disallowing the tuple operator
in
 my code). It sounds like a change fit for a D3 language, even more than the
 introduction of reference counting. I think this change will cause some people
 to permanently stop using D.
It's quite possible we will be unable to make this change. But the question that started all this would be what would I change if breaking code was allowed. I suggest that in the future write code that is explicit about the intention - by character or by decoded character - by using adapters .byChar or .byDchar.
Sep 28 2014
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 28 Sep 2014 12:38:25 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 I suggest that in the future write code that is explicit about the intention - 
 by character or by decoded character - by using adapters .byChar or .byDchar.
... or by "user perceived character" or by "word" or by "line". I'm always on the fence with code points. Sure they are the code points, but what does it mean in practice? Is it valid to start a Unicode string with just a diacritical mark? Does it make sense to split in the middle of Korean symbols, effectively removing parts of the glyphs and rendering them invalid? Bearophile, what does your code _do_ with the dchar ranges? How is it not rendered into a caricature of its own attempts to support non-ASCII by the above ? -- Marco
Sep 29 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/14, 4:31 PM, H. S. Teoh via Digitalmars-d wrote:
 On Sat, Sep 27, 2014 at 11:00:16PM +0000, bearophile via Digitalmars-d wrote:
 H. S. Teoh:

 If we can get Andrei on board, I'm all for killing off autodecoding.
Killing auto-decoding for std.algorithm functions will break most of my D2 code... perhaps we can do that in a D3 language.
[...] Well, obviously it's not going to be done in a careless, drastic way!
Stuff that's missing: * Reasonable effort to improve performance of auto-decoding; * A study of the matter revealing either new artifacts and idioms, or the insufficiency of such; * An assessment of the impact on compilability of existing code * An assessment of the impact on correctness of existing code (that compiles and runs in both cases) * An assessment of the improvement in speed of eliminating auto-decoding I think there's a very strong need for this stuff, because claims that current alternatives to selectively avoid auto-decoding use the throwing of hands (and occasional chairs out windows) without any real investigation into how library artifacts may help. This approach to justifying risky moves is frustratingly unprincipled. Also I submit that diverting into this is a huge distraction at probably the worst moment in the history of the D programming language. C++ and GC. C++ and GC... Andrei
Sep 28 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 5:09 AM, Andrei Alexandrescu wrote:
 Stuff that's missing:

 * Reasonable effort to improve performance of auto-decoding;

 * A study of the matter revealing either new artifacts and idioms, or the
 insufficiency of such;

 * An assessment of the impact on compilability of existing code

 * An assessment of the impact on correctness of existing code (that compiles
and
 runs in both cases)

 * An assessment of the improvement in speed of eliminating auto-decoding

 I think there's a very strong need for this stuff, because claims that current
 alternatives to selectively avoid auto-decoding use the throwing of hands (and
 occasional chairs out windows) without any real investigation into how library
 artifacts may help. This approach to justifying risky moves is frustratingly
 unprincipled.
I know I have to go a ways further to convince you :-) This is definitely a longer term issue, not a stop-the-world-we-must-fix-it-now thing.
 Also I submit that diverting into this is a huge distraction at probably the
 worst moment in the history of the D programming language.
I don't plan to work on this particular issue for the time being, but do want to stop adding more autodecoding functions like the proposed std.path.withExtension().
 C++ and GC. C++ and GC...
Currently, the autodecoding functions allocate with the GC and throw as well. (They'll GC allocate an exception and throw it if they encounter an invalid UTF sequence. The adapters use the more common method of inserting a substitution character and continuing on.) This makes it harder to make GC-free Phobos code.
Sep 28 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 but do want to stop adding more autodecoding functions like the 
 proposed std.path.withExtension().
I am not sure that can work. Perhaps you need to create a range2 and algorithm2 modules, and keep adding some autodecoding functions to the old modules. Bye, bearophile
Sep 28 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 11:51 AM, bearophile wrote:
 Walter Bright:

 but do want to stop adding more autodecoding functions like the proposed
 std.path.withExtension().
I am not sure that can work. Perhaps you need to create a range2 and algorithm2 modules, and keep adding some autodecoding functions to the old modules.
It can work just fine, and I wrote it. The problem is convincing someone to pull it :-( as the PR was closed and reopened with autodecoding put back in. As I've explained many times, very few string algorithms actually need decoding at all. 'find', for example, does not. Trying to make a separate universe out of autodecoding algorithms is missing the point. Certainly, setExtension() does not need autodecoding, and in fact all the autodecoding in it does is slow it down, allocate memory on errors, make it throwable, and produce dchar output, meaning at some point later you'll need to put it back to char. I.e. there are no operations on paths that require decoding. I know that you care about performance - you post about it often. I would expect that unnecessary and pervasive decoding would be of concern to you.
Sep 28 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 It can work just fine, and I wrote it. The problem is 
 convincing someone to pull it :-( as the PR was closed and 
 reopened with autodecoding put back in.
Perhaps you need a range2 and algorithm2 modules. Introducing your changes in a sneaky way may not produce well working and predictable user code.
 I know that you care about performance - you post about it 
 often. I would expect that unnecessary and pervasive decoding 
 would be of concern to you.
I care first of all about program correctness (that's why I proposed unusual things like optional strong typing for built-in array indexes, or I proposed the "enum preconditions"). Secondly I care for performance in the functions or parts of code where performance is needed. There are plenty of code where performance is not the most important thing. That's why I have tons of range-base code. In such large parts of code having short, correct, nice looking code that looks correct is more important. Please don't assume I am simple minded :-) Bye, bearophile
Sep 28 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 1:38 PM, bearophile wrote:
 Walter Bright:

 It can work just fine, and I wrote it. The problem is convincing someone to
 pull it :-( as the PR was closed and reopened with autodecoding put back in.
Perhaps you need a range2 and algorithm2 modules. Introducing your changes in a sneaky way may not produce well working and predictable user code.
I'm not suggesting sneaky ways. setExt() was a NEW function.
 I know that you care about performance - you post about it often. I would
 expect that unnecessary and pervasive decoding would be of concern to you.
I care first of all about program correctness (that's why I proposed unusual things like optional strong typing for built-in array indexes, or I proposed the "enum preconditions").
Ok, but you implied at one point that you were not aware of which parts of your string code decoded and which did not. That's not consistent with being very careful about correctness. Note that autodecode does not always happen - it doesn't happen for ranges of chars. It's very hard to look at piece of code and tell if autodecode is going to happen or not.
 Secondly I care for performance in the functions or parts
 of code where performance is needed. There are plenty of code where performance
 is not the most important thing. That's why I have tons of range-base code. In
 such large parts of code having short, correct, nice looking code that looks
 correct is more important. Please don't assume I am simple minded :-)
It's very hard to disable the autodecode when it is not needed, though the new .byCodeUnit has made that much easier.
Sep 28 2014
next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Sunday, 28 September 2014 at 23:06:28 UTC, Walter Bright wrote:
 Note that autodecode does not always happen - it doesn't happen 
 for ranges of chars. It's very hard to look at piece of code 
 and tell if autodecode is going to happen or not.
Arguably, this means we need to unify the behavior of strings, and "string-like" objects. Pointing to an inconsistency doesn't mean the design is flawed and void.
Sep 29 2014
prev sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Sunday, 28 September 2014 at 23:06:28 UTC, Walter Bright wrote:
 It's very hard to disable the autodecode when it is not needed, 
 though the new .byCodeUnit has made that much easier.
One issue with this though is that "byCodeUnit" is not actually an array. As such, by using "byCodeUnit", you have just as much chances of improving performance, as you have of *hurting* performance for algorithms that are string optimized. For example, which would be fastest: "hello world".find(' '); //(1) "hello world".byCodeUnit.find(' '); //(2) Currently, (1) is faster :/ This is a good argument though to instead use ubyte[] or std.encoding.AsciiString. What I think we (maybe) need though is std.encoding.UTF8Array, which explicitly means: This is a range containing UTF8 characters. I don't want decoding. It's an array you may memchr or slice operate on.
Sep 29 2014
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sun, Sep 28, 2014 at 12:57:17PM -0700, Walter Bright via Digitalmars-d wrote:
 On 9/28/2014 11:51 AM, bearophile wrote:
Walter Bright:

but do want to stop adding more autodecoding functions like the
proposed std.path.withExtension().
I am not sure that can work. Perhaps you need to create a range2 and algorithm2 modules, and keep adding some autodecoding functions to the old modules.
It can work just fine, and I wrote it. The problem is convincing someone to pull it :-( as the PR was closed and reopened with autodecoding put back in.
The problem with pulling such PRs is that they introduce a dichotomy into Phobos. Some functions autodecode, some don't, and from a user's POV, it's completely arbitrary and random. Which leads to bugs because people can't possibly remember exactly which functions autodecode and which don't.
 As I've explained many times, very few string algorithms actually need
 decoding at all. 'find', for example, does not. Trying to make a
 separate universe out of autodecoding algorithms is missing the point.
[...] Maybe what we need to do, is to change the implementation of std.algorithm so that it internally uses byCodeUnit for narrow strings where appropriate. We're already specialcasing Phobos code for narrow strings anyway, so it wouldn't make things worse by making those special cases not autodecode. This doesn't quite solve the issue of composing ranges, since one composed range returns dchar in .front composed with another range will have autodecoding built into it. For those cases, perhaps one way to hack around the present situation is to use Phobos-private enums in the wrapper ranges (e.g., enum isNarrowStringUnderneath=true; in struct Filter or something), that ranges downstream can test for, and do the appropriate bypasses. (BTW, before you pick on specific algorithms you might want to actually look at the code for things like find(), because I remember there were a couple o' PRs where find() of narrow strings will use (presumably) fast functions like strstr or strchr, bypassing a foreach loop over an autodecoding .front.) T -- I think Debian's doing something wrong, `apt-get install pesticide', doesn't seem to remove the bugs on my system! -- Mike Dresser
Sep 28 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/28/14, 1:39 PM, H. S. Teoh via Digitalmars-d wrote:
 On Sun, Sep 28, 2014 at 12:57:17PM -0700, Walter Bright via Digitalmars-d
wrote:
 On 9/28/2014 11:51 AM, bearophile wrote:
 Walter Bright:

 but do want to stop adding more autodecoding functions like the
 proposed std.path.withExtension().
I am not sure that can work. Perhaps you need to create a range2 and algorithm2 modules, and keep adding some autodecoding functions to the old modules.
It can work just fine, and I wrote it. The problem is convincing someone to pull it :-( as the PR was closed and reopened with autodecoding put back in.
The problem with pulling such PRs is that they introduce a dichotomy into Phobos. Some functions autodecode, some don't, and from a user's POV, it's completely arbitrary and random. Which leads to bugs because people can't possibly remember exactly which functions autodecode and which don't.
I agree. -- Andrei
Sep 28 2014
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
29-Sep-2014 00:39, H. S. Teoh via Digitalmars-d пишет:
 On Sun, Sep 28, 2014 at 12:57:17PM -0700, Walter Bright via Digitalmars-d
wrote:
 On 9/28/2014 11:51 AM, bearophile wrote:
 Walter Bright:

 but do want to stop adding more autodecoding functions like the
 proposed std.path.withExtension().
I am not sure that can work. Perhaps you need to create a range2 and algorithm2 modules, and keep adding some autodecoding functions to the old modules.
It can work just fine, and I wrote it. The problem is convincing someone to pull it :-( as the PR was closed and reopened with autodecoding put back in.
The problem with pulling such PRs is that they introduce a dichotomy into Phobos. Some functions autodecode, some don't, and from a user's POV, it's completely arbitrary and random. Which leads to bugs because people can't possibly remember exactly which functions autodecode and which don't.
Agreed.
 As I've explained many times, very few string algorithms actually need
 decoding at all. 'find', for example, does not. Trying to make a
 separate universe out of autodecoding algorithms is missing the point.
[...] Maybe what we need to do, is to change the implementation of std.algorithm so that it internally uses byCodeUnit for narrow strings where appropriate. We're already specialcasing Phobos code for narrow strings anyway, so it wouldn't make things worse by making those special cases not autodecode. This doesn't quite solve the issue of composing ranges, since one composed range returns dchar in .front composed with another range will have autodecoding built into it. For those cases, perhaps one way to hack around the present situation is to use Phobos-private enums in the wrapper ranges (e.g., enum isNarrowStringUnderneath=true; in struct Filter or something), that ranges downstream can test for, and do the appropriate bypasses.
We need to either generalize the hack we did for char[] and wchar[] or start creating a whole new phobos without auto-decoding. I'm not sure what's best but the latter is more disruptive.
 (BTW, before you pick on specific algorithms you might want to actually
 look at the code for things like find(), because I remember there were a
 couple o' PRs where find() of narrow strings will use (presumably) fast
 functions like strstr or strchr, bypassing a foreach loop over an
 autodecoding .front.)
Yes, it has fast path. -- Dmitry Olshansky
Sep 28 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 1:39 PM, H. S. Teoh via Digitalmars-d wrote:
 It can work just fine, and I wrote it. The problem is convincing
 someone to pull it :-( as the PR was closed and reopened with
 autodecoding put back in.
The problem with pulling such PRs is that they introduce a dichotomy into Phobos. Some functions autodecode, some don't, and from a user's POV, it's completely arbitrary and random. Which leads to bugs because people can't possibly remember exactly which functions autodecode and which don't.
That's ALREADY the case, as I explained to bearophile. The solution is not to have the ranges autodecode, but to have the ALGORITHMS decide to autodecode (if they need it) or not (if they don't).
 As I've explained many times, very few string algorithms actually need
 decoding at all. 'find', for example, does not. Trying to make a
 separate universe out of autodecoding algorithms is missing the point.
[...] Maybe what we need to do, is to change the implementation of std.algorithm so that it internally uses byCodeUnit for narrow strings where appropriate. We're already specialcasing Phobos code for narrow strings anyway, so it wouldn't make things worse by making those special cases not autodecode.
Those special cases wind up going everywhere and impacting everyone who attempts to write generic algorithms.
 This doesn't quite solve the issue of composing ranges, since one
 composed range returns dchar in .front composed with another range will
 have autodecoding built into it. For those cases, perhaps one way to
 hack around the present situation is to use Phobos-private enums in the
 wrapper ranges (e.g., enum isNarrowStringUnderneath=true; in struct
 Filter or something), that ranges downstream can test for, and do the
 appropriate bypasses.
More complexity :-( for what should be simple tasks.
 (BTW, before you pick on specific algorithms you might want to actually
 look at the code for things like find(), because I remember there were a
 couple o' PRs where find() of narrow strings will use (presumably) fast
 functions like strstr or strchr, bypassing a foreach loop over an
 autodecoding .front.)
Oh, I know that many algorithms have such specializations. Doesn't it strike you as sucky to have to special case a whole basket of algorithms when the InputRange does not behave in a reliable manner? It's very simple for an algorithm to decode if it needs to, it just adds in a .byDchar adapter to its input range. Done. No special casing needed. The lines of code written drop in half. And it works with both arrays of chars, arrays of dchars, and input ranges of either. --- The stalling of setExt() has basically halted my attempts to adjust Phobos so that one can write nothrow and nogc algorithms that work on strings.
Sep 28 2014
next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 28 Sep 2014 16:21:14 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 On 9/28/2014 1:39 PM, H. S. Teoh via Digitalmars-d wrote:
 The problem with pulling such PRs is that they introduce a dichotomy
 into Phobos. Some functions autodecode, some don't, and from a user's
 POV, it's completely arbitrary and random. Which leads to bugs because
 people can't possibly remember exactly which functions autodecode and
 which don't.
That's ALREADY the case, as I explained to bearophile. The solution is not to have the ranges autodecode, but to have the ALGORITHMS decide to autodecode (if they need it) or not (if they don't).
Yes, that sounds like the right abstraction! -- Marco
Sep 29 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 28 September 2014 at 23:21:15 UTC, Walter Bright wrote:
 On 9/28/2014 1:39 PM, H. S. Teoh via Digitalmars-d wrote:
 It can work just fine, and I wrote it. The problem is 
 convincing
 someone to pull it :-( as the PR was closed and reopened with
 autodecoding put back in.
The problem with pulling such PRs is that they introduce a dichotomy into Phobos. Some functions autodecode, some don't, and from a user's POV, it's completely arbitrary and random. Which leads to bugs because people can't possibly remember exactly which functions autodecode and which don't.
That's ALREADY the case, as I explained to bearophile. The solution is not to have the ranges autodecode, but to have the ALGORITHMS decide to autodecode (if they need it) or not (if they don't).
No it isn't, despite you pretending otherwise. Right now there is a simple rule - Phobos does auto-decoding everywhere and any failure to do so is considered a bug. Sometimes it is possible to bypass decoding for speed-up while preserving semantical correctness but it is always implementation detail, from the point of view of the API it can't be noticed (for valid unicode string at least). Your proposal would have been a precedent to adding _intetional_ exception. It is unacceptable.
Sep 29 2014
prev sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Sunday, 28 September 2014 at 23:21:15 UTC, Walter Bright wrote:
 It's very simple for an algorithm to decode if it needs to, it 
 just adds in a .byDchar adapter to its input range. Done. No 
 special casing needed. The lines of code written drop in half. 
 And it works with both arrays of chars, arrays of dchars, and 
 input ranges of either.
This just misses the *entire* familly of algorithms that operate on generic types, such as "map". EG: the totality of std.algorithm. Oops.
Sep 29 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 29 September 2014 at 14:25:34 UTC, monarch_dodra wrote:
 This just misses the *entire* familly of algorithms that 
 operate on generic types, such as "map". EG: the totality of 
 std.algorithm. Oops.
But if you know the common operations on strings used in many programs, then you want them to perform. So you need a mix of special cased precomputed lookup-tables and/or special cased SIMD instructions that will outperform a generic map() by a significant margin. I am not arguing against generic apis being desirable, I am questioning of the feasibility of being competitive in the space of utf-8 strings without designing for SIMD. Obviously, doing bitwise and 0x80808080...80 followed by a simd compare with " " will beat anything scalar based if you want to test for a space in a utf-8 string. Intel has made a real effort at making UTF-8 SIMD optimizable with the last few generations of instruction sets. Figuring out how to tap into that potential is a lot more valuable than defining an api a priori. That means writing SIMD code and comparing it to what you have. If SIMD blows what you have out of the water, then it ain't god enough. If you can define the string APIs to fit what SIMD is good at, then you are onto something that could be really good. D is in a position where you can do that. C++ aint.
Sep 29 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/28/14, 11:36 AM, Walter Bright wrote:
 Currently, the autodecoding functions allocate with the GC and throw as
 well. (They'll GC allocate an exception and throw it if they encounter
 an invalid UTF sequence. The adapters use the more common method of
 inserting a substitution character and continuing on.) This makes it
 harder to make GC-free Phobos code.
The right solution here is refcounted exception plus policy-based functions in conjunction with RCString. I can't believe this focus has already been lost and we're back to let's remove autodecoding and ban exceptions. -- Andrei
Sep 28 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
29-Sep-2014 00:33, Andrei Alexandrescu пишет:
 On 9/28/14, 11:36 AM, Walter Bright wrote:
 Currently, the autodecoding functions allocate with the GC and throw as
 well. (They'll GC allocate an exception and throw it if they encounter
 an invalid UTF sequence. The adapters use the more common method of
 inserting a substitution character and continuing on.) This makes it
 harder to make GC-free Phobos code.
 The right solution here is refcounted exception plus policy-based
 functions in conjunction with RCString. I can't believe this focus has
 already been lost and we're back to let's remove autodecoding and ban
 exceptions. -- Andrei
I've already stated my perception of the "no stinking exceptions", and "no destructors 'cause i want it fast" elsewhere. Code must be correct and fast, with correct being a precondition for any performance tuning and speed hacks. Correct usually entails exceptions and automatic cleanup. I also do not believe the "exceptions have to be slow" motto, they are costly but proportion of such costs was largely exaggerated. -- Dmitry Olshansky
Sep 28 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 2:00 PM, Dmitry Olshansky wrote:
 I've already stated my perception of the "no stinking exceptions", and "no
 destructors 'cause i want it fast" elsewhere.

 Code must be correct and fast, with correct being a precondition for any
 performance tuning and speed hacks.
Sure. I'm not arguing for preferring incorrect code.
 Correct usually entails exceptions and automatic cleanup. I also do not believe
 the "exceptions have to be slow" motto, they are costly but proportion of such
 costs was largely exaggerated.
I think it was you that suggested that instead of throwing on invalid UTF, that the replacement character be used instead? Or maybe not, I'm not quite sure. Regardless, the replacement character method is widely used and accepted practice. There's no reason to throw.
Sep 28 2014
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
29-Sep-2014 03:48, Walter Bright пишет:
 On 9/28/2014 2:00 PM, Dmitry Olshansky wrote:
 I've already stated my perception of the "no stinking exceptions", and
 "no
 destructors 'cause i want it fast" elsewhere.

 Code must be correct and fast, with correct being a precondition for any
 performance tuning and speed hacks.
Sure. I'm not arguing for preferring incorrect code.
 Correct usually entails exceptions and automatic cleanup. I also do
 not believe
 the "exceptions have to be slow" motto, they are costly but proportion
 of such
 costs was largely exaggerated.
I think it was you that suggested that instead of throwing on invalid UTF, that the replacement character be used instead? Or maybe not, I'm not quite sure.
Aye that was me. I'd much prefer nothrow decoding. There should be an option to throw on bad input though (and we have it already), for programs that do not expect to work with even partially broken input.
 Regardless, the replacement character method is widely used and accepted
 practice. There's no reason to throw.
-- Dmitry Olshansky
Sep 29 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 28 Sep 2014 16:48:53 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 Regardless, the replacement character method is widely used and accepted 
 practice. There's no reason to throw.
I feel a bit uneasy about this. Could it introduce a silent loss of information? While the replacement character method is widely used, so is the error method. APIs typically provide flags for this. MultiByteToWideChar: The flag MB_ERR_INVALID_CHARS decides whether the API errors out or drops invalid chars. ICU: You set up an "error callback". The default replaces invalid characters with the Unicode substitution character. (We are talking about characters from arbitrary charsets like Amiga to Unicode.) Other prefab error handlers drop the invalid character or error out. iconv: By default it errors out at the location where an incomplete or invalid sequence is detected. With the "//IGNORE" flag, it will silently drop invalid characters. I'm not opposed to the decision, but I expected the reasoning to me more along the line of: `string` is per definition correct UTF-8. Exception or substitution character is of no concern to a correctly written D program, because decoding errors wont happen. Validate and treat all input as ubyte[]. (Especially when coming from a Windows console) or: We may lose information in the conversion, but it's the only practical way to reach the nogc goal. And we are far from having reference-counted Exceptions. instead of: Many people use the substitution character [in unspecified context], so it follows that it can replace Exceptions for Phobos' string-dchar decoding. :) -- Marco
Sep 29 2014
prev sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Sunday, 28 September 2014 at 23:48:54 UTC, Walter Bright wrote:
 I think it was you that suggested that instead of throwing on 
 invalid UTF, that the replacement character be used instead? Or 
 maybe not, I'm not quite sure.

 Regardless, the replacement character method is widely used and 
 accepted practice. There's no reason to throw.
This I'm OK to stand stand behind as acceptable change (should we decide to go with). It will kill the "auto-decode throws and uses the GC" argument.
Sep 29 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 28 September 2014 at 21:00:46 UTC, Dmitry Olshansky 
wrote:
 29-Sep-2014 00:33, Andrei Alexandrescu пишет:
 The right solution here is refcounted exception plus 
 policy-based
 functions in conjunction with RCString. I can't believe this 
 focus has
 already been lost and we're back to let's remove autodecoding 
 and ban
 exceptions. -- Andrei
Consider what end users are going to use first, then design the library to fit the use cases while retaining general usefulness. If UTF-8 decoding cannot be efficiently done as a generic adapter then D's approach to generic programming is a failure: dead on arrival. Generic programming is not supposed to be special-case-everything-programming. If you cannot do generic programming well on strings, then don't. Provide a concrete single dedicated utf-8 string type instead.
 I've already stated my perception of the "no stinking 
 exceptions", and "no destructors 'cause i want it fast" 
 elsewhere.

 Code must be correct and fast, with correct being a 
 precondition for any performance tuning and speed hacks.

 Correct usually entails exceptions and automatic cleanup. I 
 also do not believe the "exceptions have to be slow" motto, 
 they are costly but proportion of such costs was largely 
 exaggerated.
Correctness has nothing to do with exceptions and an exception-specific cleanup model. It has to with having a well specified model of memory management, understanding the model and implementing code to the model with rigour. The alternative is to have isolates only, higher level constructs only, GC everywhere and uniform activision-records for everything (no conceptual stack). Many high level languages work this way, even in the 60s. When it comes to exception efficiency you have many choices: 1. no exceptions, omit frame pointer 2. no extra overhead when not throwing, standard codegen and linking, slow unwind 3. no extra overhead when not throwing, nonstandard codegen, faster unwind 4. extra overhead when not throwing, nonstandard codegen, fast unwind 5. small extra overhead when not throwing, no RAII/single landingpad, omit frame pointer possible, very fast unwind (C-style longjmp) 6. hidden return error value , fixed medium overhead D has selected standard C++ (2), which means low overhead when not throwing and that you can use regular C backend/linker. In a fully GC language I think that (5) is quite acceptable and what you usually want when doing a web service. You just bail out to the root and you never really acquire resources except for transactions that should be terminated in the handler before exiting (and they will time out anyway). But there is no best model. There are trade offs based on what kind application you write. So as usual it comes back to this: what kind of applications are D actually targeting? D is becoming less and less a system level language, and more a more a compiled scripting framework. The more special casing, the less transparent D becomes, and the more of a scripting framework it becomes. A good system level language requires transparency: - easy to visualize memory layout - predictable compilation of code to machine language - no fixed memory model - no arbitrary limits and presumptions about execution model - allows you to get close to the max hardware performance potential I have a hard time picturing D as a system level language these days. And the "hacks" that try to make it GC free are not making it a better system level language (I don't consider nogc to be a hack). It makes D even less transparent. Despite all the flaws of the D1 compiler, D1 was fairly transparent. You really need to decide if D is supposed to be primarily a system level programming or if it is supposed to provide system level programming as an after thought on top of an application level programming language. Currently it is the latter and more so for every iteration.
Sep 29 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 1:33 PM, Andrei Alexandrescu wrote:
 On 9/28/14, 11:36 AM, Walter Bright wrote:
 Currently, the autodecoding functions allocate with the GC and throw as
 well. (They'll GC allocate an exception and throw it if they encounter
 an invalid UTF sequence. The adapters use the more common method of
 inserting a substitution character and continuing on.) This makes it
 harder to make GC-free Phobos code.
The right solution here is refcounted exception plus policy-based functions in conjunction with RCString. I can't believe this focus has already been lost and we're back to let's remove autodecoding and ban exceptions. -- Andrei
Or setExt() can simply insert .byCodeUnit as I suggested in the PR, and it's done and working correctly and doesn't throw and doesn't allocate and goes fast. Not everything in Phobos can be dealt with so easily, of course, but there's quite a bit of low hanging fruit of this nature we can just take care of now.
Sep 28 2014
prev sibling parent "Kagamin" <spam here.lot> writes:
On Sunday, 28 September 2014 at 12:09:50 UTC, Andrei Alexandrescu 
wrote:
 On 9/27/14, 4:31 PM, H. S. Teoh via Digitalmars-d wrote:
 On Sat, Sep 27, 2014 at 11:00:16PM +0000, bearophile via 
 Digitalmars-d wrote:
 H. S. Teoh:

 If we can get Andrei on board, I'm all for killing off 
 autodecoding.
Killing auto-decoding for std.algorithm functions will break most of my D2 code... perhaps we can do that in a D3 language.
[...] Well, obviously it's not going to be done in a careless, drastic way!
Stuff that's missing: * Reasonable effort to improve performance of auto-decoding; * A study of the matter revealing either new artifacts and idioms, or the insufficiency of such; * An assessment of the impact on compilability of existing code * An assessment of the impact on correctness of existing code (that compiles and runs in both cases) * An assessment of the improvement in speed of eliminating auto-decoding I think there's a very strong need for this stuff, because claims that current alternatives to selectively avoid auto-decoding use the throwing of hands (and occasional chairs out windows) without any real investigation into how library artifacts may help. This approach to justifying risky moves is frustratingly unprincipled.
As far as I see, backward compatibility is fairly easy. Extract autodecoding modules into `autodecoding` dub package and clean up phobos modules into non-decoding behavior. The phobos code will be simplified: it will deal with ranges as is without specialization, the `autodecoding` dub package will be simple: just wraps strings into dchar range and invokes non-decoding function from phobos, preserves current module interface to keep legacy D code working. Run dfix on your sources, it will replace `import std.algorithm` with `import autodecoding.algorithm` - then the code should work. What do you think? Worth a DIP?
Oct 01 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/14, 3:40 PM, H. S. Teoh via Digitalmars-d wrote:
 If we can get Andrei on board, I'm all for killing off autodecoding.
That's rather vague; it's unclear what would replace it. -- Andrei
Sep 27 2014
next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Sunday, 28 September 2014 at 00:13:59 UTC, Andrei Alexandrescu 
wrote:
 On 9/27/14, 3:40 PM, H. S. Teoh via Digitalmars-d wrote:
 If we can get Andrei on board, I'm all for killing off 
 autodecoding.
That's rather vague; it's unclear what would replace it. -- Andrei
No autodecoding ;-) Specifically: 1. ref T front(T[] r) always returns r[0] 2. popFront(ref T[] r) always does { ++r.ptr; --r.length; } 3. Narrow string will be hasLength, hasSlicing, isRandomAccessRange (i.e. they are just like any other array). Also: 4. Disallow implicit conversions, comparisons, or any operation among char, wchar, dchar. This makes things like "foo".find('π') compile time errors (or better, errors until we specialize to it to do "foo".find("π"), as it should) 5. Provide byCodePoint for narrow strings (although I suspect this will be rarely used). The argument is as follows: * First, this is a hell of a lot simpler for the implementation. * People rarely ever search for single, non-ASCII characters in strings, and #4 makes it an error if they do (until we specialize to make it work). * Searching, comparison, joining, and splitting functions will be fast and correct by default. One possible counter argument is that this makes it easier to corrupt strings (since you could, e.g. insert a substring into the middle of a multi-byte code point). To that I say that it's unlikely. When inserting into a string, you're either doing it at the front or back (which is safe), or to some point that you've found by some other means (e.g. using find). I can't imagine a scenario where you could find a point in the middle of a string, that is in the middle of a code point. Of course, I'd probably say this change isn't practical right now, but this is how I'd do things if I were to start over.
Sep 28 2014
prev sibling parent reply "Uranuz" <neuranuz gmail.com> writes:
On Sunday, 28 September 2014 at 00:13:59 UTC, Andrei Alexandrescu 
wrote:
 On 9/27/14, 3:40 PM, H. S. Teoh via Digitalmars-d wrote:
 If we can get Andrei on board, I'm all for killing off 
 autodecoding.
That's rather vague; it's unclear what would replace it. -- Andrei
I believe that removing autodeconding will make things even worse. As far as understand if we will remove it from front() function that operates on narrow strings then it will return just byte of char. I believe that proceeding on narrow string by `user perceived chars` (graphemes) is more common use case. Operating on single bytes of multibyte character is uncommon task and you can do that via direct indexing of char[] array. I believe what number of bytes is in *user perceived chars* is internal implementation of UTF-8 encoding and it should not be considered in common tasks such as parsing, searching, replacing text and etc. If you need byte representation of string you should cast it into ubyte[] and work with it using the same range functions without autodecoding. The main problem that I see that unexpirienced in D programmer can be confused where he operates by bytes or by graphemes. Especially it could happen when he migrates from C#, Python where string is not considered as array of it's bytes. Because *char* in D is not char it's a part of char, but not entire char. It's main inconsistence. Possible solution is to include class or struct implementation of string and hide internal implementation of narrow string for those users who don't need to operate on single bytes of UTF-8 characters. I believe it's the best way to kill all the rabbits)) We could provide this class String with method returning ubyte[] (better way) or char[] that will expose internal implementation for those who need it. A question: can you list some languages that represent UTF-8 narrow strings as array of single bytes?
Sep 28 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sun, Sep 28, 2014 at 12:06:16PM +0000, Uranuz via Digitalmars-d wrote:
 On Sunday, 28 September 2014 at 00:13:59 UTC, Andrei Alexandrescu wrote:
On 9/27/14, 3:40 PM, H. S. Teoh via Digitalmars-d wrote:
If we can get Andrei on board, I'm all for killing off autodecoding.
That's rather vague; it's unclear what would replace it. -- Andrei
I believe that removing autodeconding will make things even worse. As far as understand if we will remove it from front() function that operates on narrow strings then it will return just byte of char. I believe that proceeding on narrow string by `user perceived chars` (graphemes) is more common use case.
[...] Unfortunately this is not what autodecoding does today. Today's autodecoding only segments strings into code *points*, which are not the same thing as graphemes. For example, combining diacritics are normally not considered separate characters from the user's POV, but they *are* separate codepoints from their base character. The only reason today's autodecoding is even remotely considered "correct" from an intuitive POV is because most Western character sets happen to use only precomposed characters rather than combining diacritic sequences. If you were processing, say, Korean text, the present autodecoding .front would *not* give you what you might imagine is a "single character"; it would only be halves of Korean graphemes. Which, from a user's POV, would suffer from the same issues as dealing with individual bytes in a UTF-8 stream -- any mistake on the program's part in handling these half-units will cause "corruption" of the text (not corruption in the same sense as an improperly segmented UTF-8 byte stream, but in the sense that the wrong glyphs will be displayed on the screen -- from the user's POV these two are basically the same thing). You might then be tempted to say, well let's make .front return graphemes instead. That will solve the "single intuitive character" issue, but the performance will be FAR worse than what it is today. So basically, what we have today is neither efficient nor complete, but a halfway solution that mostly works for Western character sets but is incomplete for others. We're paying efficiency for only a partial benefit. Is it worth the cost? I think the correct solution is not for Phobos to decide for the application at what level of abstraction a string ought to be processed. Rather, let the user decide. If they're just dealing with opaque blocks of text, decoding or segmenting by grapheme is completely unnecessary -- they should just operate on byte ranges as opaque data. They should use byCodeUnit. If they need to work with Unicode codepoints, let them use byCodePoint. If they need to work with individual user-perceived characters (i.e., graphemes), let them use byGrapheme. This is why I proposed the deprecation path of making it illegal to pass raw strings to Phobos algorithms -- the caller should specify what level of abstraction they want to work with -- byCodeUnit, byCodePoint, or byGrapheme. The standard library's job is to empower the D programmer by giving him the choice, not to shove a predetermined solution down his throat. T -- Life is unfair. Ask too much from it, and it may decide you don't deserve what you have now either.
Sep 28 2014
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 28 September 2014 at 14:38:57 UTC, H. S. Teoh via 
Digitalmars-d wrote:
 On Sun, Sep 28, 2014 at 12:06:16PM +0000, Uranuz via 
 Digitalmars-d wrote:
 On Sunday, 28 September 2014 at 00:13:59 UTC, Andrei 
 Alexandrescu wrote:
On 9/27/14, 3:40 PM, H. S. Teoh via Digitalmars-d wrote:
If we can get Andrei on board, I'm all for killing off 
autodecoding.
That's rather vague; it's unclear what would replace it. -- Andrei
I believe that removing autodeconding will make things even worse. As far as understand if we will remove it from front() function that operates on narrow strings then it will return just byte of char. I believe that proceeding on narrow string by `user perceived chars` (graphemes) is more common use case.
[...] Unfortunately this is not what autodecoding does today. Today's autodecoding only segments strings into code *points*, which are not the same thing as graphemes. For example, combining diacritics are normally not considered separate characters from the user's POV, but they *are* separate codepoints from their base character. The only reason today's autodecoding is even remotely considered "correct" from an intuitive POV is because most Western character sets happen to use only precomposed characters rather than combining diacritic sequences. If you were processing, say, Korean text, the present autodecoding .front would *not* give you what you might imagine is a "single character"; it would only be halves of Korean graphemes. Which, from a user's POV, would suffer from the same issues as dealing with individual bytes in a UTF-8 stream -- any mistake on the program's part in handling these half-units will cause "corruption" of the text (not corruption in the same sense as an improperly segmented UTF-8 byte stream, but in the sense that the wrong glyphs will be displayed on the screen -- from the user's POV these two are basically the same thing). You might then be tempted to say, well let's make .front return graphemes instead. That will solve the "single intuitive character" issue, but the performance will be FAR worse than what it is today. So basically, what we have today is neither efficient nor complete, but a halfway solution that mostly works for Western character sets but is incomplete for others. We're paying efficiency for only a partial benefit. Is it worth the cost? I think the correct solution is not for Phobos to decide for the application at what level of abstraction a string ought to be processed. Rather, let the user decide. If they're just dealing with opaque blocks of text, decoding or segmenting by grapheme is completely unnecessary -- they should just operate on byte ranges as opaque data. They should use byCodeUnit. If they need to work with Unicode codepoints, let them use byCodePoint. If they need to work with individual user-perceived characters (i.e., graphemes), let them use byGrapheme. This is why I proposed the deprecation path of making it illegal to pass raw strings to Phobos algorithms -- the caller should specify what level of abstraction they want to work with -- byCodeUnit, byCodePoint, or byGrapheme. The standard library's job is to empower the D programmer by giving him the choice, not to shove a predetermined solution down his throat. T
I totally agree with all of that. It's one of those cases where correct by default is far too slow (that would have to be graphemes) but fast by default is far too broken. Better to force an explicit choice. There is no magic bullet for unicode in a systems language such as D. The programmer must be aware of it and make choices about how to treat it.
Sep 28 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 10:03 AM, John Colvin wrote:
 There is no magic bullet for unicode in a systems language such as D. The
 programmer must be aware of it and make choices about how to treat it.
That's really the bottom line. The trouble with autodecode is it is done at the lowest level, meaning it is very hard to bypass. By moving the decision up a level (by using .byDchar or .byCodeUnit adapters) the caller makes the decision.
Sep 28 2014
prev sibling parent reply "Uranuz" <neuranuz gmail.com> writes:
 I totally agree with all of that.

 It's one of those cases where correct by default is far too 
 slow (that would have to be graphemes) but fast by default is 
 far too broken. Better to force an explicit choice.

 There is no magic bullet for unicode in a systems language such 
 as D. The programmer must be aware of it and make choices about 
 how to treat it.
I see didn't know about difference between byCodeUnit and byGrapheme, because I speak Russian and it is close to English, because it doesn't have diacritics. As far as I remember German, that I learned at school have diacritics. So you opened my eyes in this question. My position as usual programmer is that I speaking language which graphemes coded by 2 bytes and I alwas need to do decoding otherwise my programme will be broken. Other possibility is to use wstring or dstring, but it is less memory efficient. Also UTF-8 is more commonly used in the Internet so I don't want to do some conversions to UTF-32, for example. Where I could read about byGrapheme? Isn't this approach overcomplicated? I don't want to write Dostoevskiy's book "War and Peace" in order to write some parser for simple DSL.
Sep 28 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
28-Sep-2014 23:44, Uranuz пишет:
 I totally agree with all of that.

 It's one of those cases where correct by default is far too slow (that
 would have to be graphemes) but fast by default is far too broken.
 Better to force an explicit choice.

 There is no magic bullet for unicode in a systems language such as D.
 The programmer must be aware of it and make choices about how to treat
 it.
I see didn't know about difference between byCodeUnit and byGrapheme, because I speak Russian and it is close to English, because it doesn't have diacritics. As far as I remember German, that I learned at school have diacritics. So you opened my eyes in this question. My position as usual programmer is that I speaking language which graphemes coded by 2 bytes
In UTF-16 and UTF-8.
 and I alwas
 need to do decoding otherwise my programme will be broken. Other
 possibility is to use wstring or dstring, but it is less memory
 efficient. Also UTF-8 is more commonly used in the Internet so I
 don't want to do some conversions to UTF-32, for example.

 Where I could read about byGrapheme?
std.uni docs: http://dlang.org/phobos/std_uni.html#.byGrapheme
 Isn't this approach
 overcomplicated? I don't want to write Dostoevskiy's book "War
 and Peace" in order to write some parser for simple DSL.
It's Tolstoy actually: http://en.wikipedia.org/wiki/War_and_Peace You don't need byGrapheme for simple DSL. In fact as long as DSL is simple enough (ASCII only) you may safely avoid decoding. If it's in Russian you might want to decode. Even in this case there are ways to avoid decoding, it may involve a bit of writing in as for typical short novel ;) In fact I did a couple of such literature exercises in std library. For codepoint lookups on non-decoded strings: http://dlang.org/phobos/std_uni.html#.utfMatcher And to create sets of codepoints to detect with matcher: http://dlang.org/phobos/std_uni.html#.CodepointSet -- Dmitry Olshansky
Sep 28 2014
parent reply "Uranuz" <neuranuz gmail.com> writes:
 It's Tolstoy actually:
 http://en.wikipedia.org/wiki/War_and_Peace

 You don't need byGrapheme for simple DSL. In fact as long as 
 DSL is simple enough (ASCII only) you may safely avoid 
 decoding. If it's in Russian you might want to decode. Even in 
 this case there are ways to avoid decoding, it may involve a 
 bit of writing in as for typical short novel ;)
Yes, my mistake ;) I was thinking about *Crime and Punishment* but writen *War and Peace*. Don't know why. May be because it is longer. Thanks for useful links. As far as we are talking about standard library I think that some stanard aproach should be provided to solve often tasks: searching, sorting, parsing, splitting strings. I see that currently we have a lot of ways of doing similar things with strings. I think this is a problem of documentation at some part. When I parsing text I can't understand why I need to use all of these range interfaces instead of just manipulating on raw narrow string. We have several modules about working on strings: std.range, std.algorithm, std.string, std.array, std.utf and I can't see how they help me to solve my problems. In opposite they just creating me new problem to think of them in order to find *right* way. So most of my time I spend on thinking about it but not solving my task. It is hard for me to accept that we don't need to decode to do some operations. What is annoying is that I always need to think of codelength that I should show to user and byte length that is used to slice char array. It's very easy to be confused with them and do something wrong. I see that all is complicated we have 3 types of character and more than 5 modules for trivial manipulations on strings with 10ths of functions. It all goes into hell. But I don't even started to do my job. And we don't have *standard* way to deal with it in std lib. At least this way in not documented enough.
Sep 28 2014
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
29-Sep-2014 00:44, Uranuz пишет:
 It's Tolstoy actually:
 http://en.wikipedia.org/wiki/War_and_Peace

 You don't need byGrapheme for simple DSL. In fact as long as DSL is
 simple enough (ASCII only) you may safely avoid decoding. If it's in
 Russian you might want to decode. Even in this case there are ways to
 avoid decoding, it may involve a bit of writing in as for typical
 short novel ;)
Yes, my mistake ;) I was thinking about *Crime and Punishment* but writen *War and Peace*. Don't know why. May be because it is longer.
Admittedly both are way too long for my taste :)
 Thanks for useful links. As far as we are talking about standard library
 I think that some stanard aproach should be provided to solve often
 tasks: searching, sorting, parsing, splitting strings. I see that
 currently we have a lot of ways of doing similar things with strings. I
 think this is a problem of documentation at some part.
Some of this is historical, in particular std.string is way older then std.algorithm.
 When I parsing
 text I can't understand why I need to use all of these range interfaces
 instead of just manipulating on raw narrow string. We have several
 modules about working on strings: std.range, std.algorithm, std.string,
 std.array,
std.range publicly imports std.array thus I really do not see why we still have std.array as standalone module. std.utf and I can't see how they help me to solve my
 problems. In opposite they just creating me new problem to think of them
 in order to find *right* way.
There is no *right* way, every level of abstraction has its uses. Also there is a bit of trade-off on performance vs easy/obvious/nice code.
 So most of my time I spend on thinking
 about it but not solving my task.
Takes time to get accustomed with a standard library. See also std.conv and std.format. String processing is indeed shotgun-ed across entire phobos.
 It is hard for me to accept that we don't need to decode to do some
 operations. What is annoying is that I always need to think of
 codelength that I should show to user and byte length that is used to
 slice char array. It's very easy to be confused with them and do
 something wrong.
As long as you use decoding primitives you keep getting back proper indices automatically. That must be what some folks considered correct way to do Unicode until it was apparent to everybody that Unicode is way more then this.
 I see that all is complicated we have 3 types of character and more than
 5 modules for trivial manipulations on strings with 10ths of functions.
 It all goes into hell.
There are many tools, but when I write parsers I actually use almost none of them. Well, nowdays I'm going to use the stuff in std.uni like CodePointSet, utfMatcher etc. std.regex makes some use of these already, but prior to that std.utf.decode was my lone workhorse.
 But I don't even started to do my job. And we
 don't have *standard* way to deal with it in std lib. At least this way
 in not documented enough.
Well on the bright side consider that C has lots of broken functions in stdlib, and even some that are _never_ safe like "gets" ;) -- Dmitry Olshansky
Sep 28 2014
prev sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sun, 28 Sep 2014 19:44:39 +0000
Uranuz via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 I speaking language which graphemes coded by 2 bytes
UCS-4? KOI8? my locale is KOI8, and i HATE D for assuming that everyone one the planet using UTF-8 and happy with it. from my POV, almost all string decoding is broken. string i got from filesystem? good god, lest it not contain anything out of ASCII range! string i got from text file? the same. string i must write to text file or stdout? oh, 'cmon, what do you mean telling me "=D0=BF=C2=A9=D1=8F=E2=94=80=D0=BF=E2=95=A6=D0= =BF=E2=95=A1=D0=BF=E2=95=A3=D1=8F=E2=94=8C"?! i can't read that!
Sep 28 2014
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 29 Sep 2014 04:33:08 +0300
schrieb ketmar via Digitalmars-d <digitalmars-d puremagic.com>:

 On Sun, 28 Sep 2014 19:44:39 +0000
 Uranuz via Digitalmars-d <digitalmars-d puremagic.com> wrote:
=20
 I speaking language which graphemes coded by 2 bytes
UCS-4? KOI8? my locale is KOI8, and i HATE D for assuming that everyone one the planet using UTF-8 and happy with it. from my POV, almost all string decoding is broken. string i got from filesystem? good god, lest it not contain anything out of ASCII range! string i got from text file? the same. string i must write to text file or stdout? oh, 'cmon, what do you mean telling me "=D0=BF=C2=A9=D1=8F=E2=94=80=D0=BF=E2=95=A6=
=D0=BF=E2=95=A1=D0=BF=E2=95=A3=D1=8F=E2=94=8C"?! i can't read that! My friend, we agree here! We must convert the whole world to UTF-8 eventually to end this madness! But for now when we write to a terminal, we have to convert to the system locale, because there are still people who don't use Unicode. (On Windows consoles the wide-char writing functions are good enough for NFC strings.) And a path from the filesystem is actually in no specific encoding on Unix. We only know it is byte based and uses ASCII '/' and '\0' as delimiters. On Windows it is ushort based IIRC. To make matters more messy, Gtk assumes Unicode, while Qt assumes the user's locale for file names. And in reality it is determined by the IO charset at mount time. --=20 Marco
Sep 29 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 5:06 AM, Uranuz wrote:
 A question: can you list some languages that represent UTF-8 narrow strings as
 array of single bytes?
C and C++.
Sep 28 2014
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 28 Sep 2014 11:08:13 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 On 9/28/2014 5:06 AM, Uranuz wrote:
 A question: can you list some languages that represent UTF-8 narrow strings as
 array of single bytes?
C and C++.
Not really, C string are in "C locale", not a specific encoding. I.e. it _cannot_ deal with UTF-8 specifically. Assuming UTF-8 would lead to funny output on consoles and the like when passing D string to C functions. :) -- Marco
Sep 29 2014
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 import std.algorithm, std.stdio;
 int main(string[] args) {
   return cast(int)args.map!("a.length").reduce!"a+b"();
 }

 Here's what LDC produces (with -O -inline -release 
 -noboundscheck)
Part of this particular case problem is not a compiler optimizer weakness, but that autodecode problem I've been throwing (!) chairs through windows on.
There is no char auto decoding in this program, right? Note: in Phobos now we have std.algorithm.sum, that is better than reduce!"a+b"(). Bye, bearophile
Sep 27 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/27/2014 3:52 PM, bearophile wrote:
 There is no char auto decoding in this program, right?
Notice the calls to autodecoding 'front' in the assembler dump.
Sep 27 2014
parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 27 September 2014 at 23:04:00 UTC, Walter Bright 
wrote:
 On 9/27/2014 3:52 PM, bearophile wrote:
 There is no char auto decoding in this program, right?
Notice the calls to autodecoding 'front' in the assembler dump.
I think you're imagining things Walter! There's no auto-decoding my example, it's just adding up the lengths.
Sep 27 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/27/2014 4:06 PM, Peter Alexander wrote:
 On Saturday, 27 September 2014 at 23:04:00 UTC, Walter Bright wrote:
 On 9/27/2014 3:52 PM, bearophile wrote:
 There is no char auto decoding in this program, right?
Notice the calls to autodecoding 'front' in the assembler dump.
I think you're imagining things Walter! There's no auto-decoding my example, it's just adding up the lengths.
oh crap, I misread empty as front!
Sep 27 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/14, 1:57 PM, Walter Bright wrote:
 To scrape the barnacles off, I've filed:

 https://issues.dlang.org/show_bug.cgi?id=13541
 https://issues.dlang.org/show_bug.cgi?id=13542
 https://issues.dlang.org/show_bug.cgi?id=13543
 https://issues.dlang.org/show_bug.cgi?id=13544

 I'm sure there's much more in std.file (and elsewhere) that can be done.
Noice. Should be told, though, that reading the actual file will dominate execution time. -- Andrei
Sep 27 2014
parent reply Martin Nowak <code dawg.eu> writes:
On 09/28/2014 02:12 AM, Andrei Alexandrescu wrote:
 Noice. Should be told, though, that reading the actual file will
 dominate execution time. -- Andrei
Absolutely, and we should have different priorities right now. It would also help to fix compiler bugs that regularly cause performance regressions. https://issues.dlang.org/show_bug.cgi?id=7625 https://issues.dlang.org/buglist.cgi?keywords=performance&resolution=---
Sep 29 2014
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 29 September 2014 at 16:43:17 UTC, Martin Nowak wrote:
 On 09/28/2014 02:12 AM, Andrei Alexandrescu wrote:
 Noice. Should be told, though, that reading the actual file 
 will
 dominate execution time. -- Andrei
Absolutely, and we should have different priorities right now.
That is not obvious. Modern parsing techniques that deals with ambiguity are O(N^3) to O(N^4), add to this the desire for lexer-free parsers and you'll start to see that performance matters.
Sep 29 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdan.org> writes:
Martin Nowak <code dawg.eu> wrote:
 On 09/28/2014 02:12 AM, Andrei Alexandrescu wrote:
 Noice. Should be told, though, that reading the actual file will
 dominate execution time. -- Andrei
Absolutely, and we should have different priorities right now. It would also help to fix compiler bugs that regularly cause performance regressions. https://issues.dlang.org/show_bug.cgi?id=7625 https://issues.dlang.org/buglist.cgi?keywords=performance&resolution=---
Totally, thank you. -- Andrei
Sep 29 2014
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
28-Sep-2014 00:57, Walter Bright пишет:
  From time to time, I take a break from bugs and enhancements and just
 look at what some piece of code is actually doing. Sometimes, I'm
 appalled. Phobos, for example, should be a lean and mean fighting machine:


 http://www.nbcnews.com/id/38545625/ns/technology_and_science-science/t/king-tuts-chariots-were-formula-one-cars/#.VCceNmd0xjs


 Instead, we have something more akin to:


 http://untappedcities.com/2012/10/31/roulez-carrosses-carriages-of-versailles-arrive-in-arras/


 More specifically, I looked at std.file.copy():

    https://github.com/D-Programming-Language/phobos/blob/master/std/file.d

 Which is 3 lines of code:

    void copy(in char[] from, in char[] to) {
          immutable result = CopyFileW(from.tempCStringW(),
 to.tempCStringW(), false);
          if (!result)
              throw new FileException(to.idup);
    }

 Compiling this code for Windows produces the rather awful:
In all honesty - 2 RAII structs w/o inlining + setting up exception frame + creating and allocating an exception + idup-ing a string does account to about this much.
 _D3std4file4copyFxAaxAaZv       comdat
          assume  CS:_D3std4file4copyFxAaxAaZv
 L0:             push    EBP
                  mov     EBP,ESP
                  mov     EDX,FS:__except_list
                  push    0FFFFFFFFh
                  lea     EAX,-0220h[EBP]
                  push    offset _D3std4file4copyFxAaxAaZv[0106h]
                  push    EDX
                  mov     FS:__except_list,ESP
                  sub     ESP,8
                  sub     ESP,041Ch
                  push    0
                  push    dword ptr 0Ch[EBP]
                  push    dword ptr 8[EBP]
                  call    near ptr
 _D3std8internal7cstring21__T11tempCSÇàÆTuTaZÇìÆFNbNixAaZSÇ┬├3Res
                  mov     dword ptr -4[EBP],0
                  lea     EAX,-0220h[EBP]
                  call    near ptr
 _D3std8internal7cstring21__T11tempCStringTuTaZ11tempCStringFNbNixAaZ3Res3ptrMxFNaNbNdNiNfZPxu

                  push    EAX
                  lea     EAX,-0430h[EBP]
                  push    dword ptr 014h[EBP]
                  push    dword ptr 010h[EBP]
                  call    near ptr
 _D3std8internal7cstring21__T11tempCSÇàÆTuTaZÇìÆFNbNixAaZSÇ┬├3Res
                  mov     dword ptr -4[EBP],1
                  lea     EAX,-0430h[EBP]
                  call    near ptr
 _D3std8internal7cstring21__T11tempCStringTuTaZ11tempCStringFNbNixAaZ3Res3ptrMxFNaNbNdNiNfZPxu

                  push    EAX
                  call    dword ptr __imp__CopyFileW 12
                  mov     -01Ch[EBP],EAX
                  mov     dword ptr -4[EBP],0
                  call    near ptr L83
                  jmp short       L8F
 L83:            lea     EAX,-0220h[EBP]
                  call    near ptr
 _D3std8internal7cstring21__T11tempCStringTuTaZ11tempCStringFNbNixAaZ3Res6__dtorMFNbNiZv

                  ret
 L8F:            mov     dword ptr -4[EBP],0FFFFFFFFh
                  call    near ptr L9D
                  jmp short       LA9
 L9D:            lea     EAX,-0430h[EBP]
                  call    near ptr
 _D3std8internal7cstring21__T11tempCStringTuTaZ11tempCStringFNbNixAaZ3Res6__dtorMFNbNiZv

                  ret
 LA9:            cmp     dword ptr -01Ch[EBP],0
                  jne     LF3
                  mov     ECX,offset
 FLAT:_D3std4file13FileException7__ClassZ
                  push    ECX
                  call    near ptr __d_newclass
                  add     ESP,4
                  push    dword ptr 0Ch[EBP]
                  mov     -018h[EBP],EAX
                  push    dword ptr 8[EBP]
                  call    near ptr
 _D6object12__T4idupTxaZ4idupFNaNbNdNfAxaZAya
                  push    EDX
                  push    EAX
                  call    dword ptr __imp__GetLastError 0
                  push    EAX
                  push    dword ptr _D3std4file13FileException6__vtblZ[02Ch]
                  push    dword ptr _D3std4file13FileException6__vtblZ[028h]
                  push    095Dh
                  mov     EAX,-018h[EBP]
                  call    near ptr
 _D3std4file13FileException6__ctorMFNfxAakAyakZC3std4file13FileException
                  push    EAX
                  call    near ptr __d_throwc
 LF3:            mov     ECX,-0Ch[EBP]
                  mov     FS:__except_list,ECX
                  mov     ESP,EBP
                  pop     EBP
                  ret     010h
                  mov     EAX,offset
 FLAT:_D3std4file13FileException6__vtblZ[0310h]
                  jmp     near ptr __d_framehandler

 which is TWICE as much generated code as for D1's copy(), which does the
 same thing. No, it is not because D2's compiler sux. It's because it has
 become encrustified with gee-gaws, jewels, decorations, and other crap.

 To scrape the barnacles off, I've filed:

 https://issues.dlang.org/show_bug.cgi?id=13541
 https://issues.dlang.org/show_bug.cgi?id=13542
 https://issues.dlang.org/show_bug.cgi?id=13543
 https://issues.dlang.org/show_bug.cgi?id=13544

 I'm sure there's much more in std.file (and elsewhere) that can be done.
 Guys, when developing Phobos/Druntime code, please look at the assembler
 once in a while and see what is being wrought. You may be appalled, too.
-- Dmitry Olshansky
Sep 28 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2014 4:46 AM, Dmitry Olshansky wrote:
 In all honesty - 2 RAII structs w/o inlining + setting up exception frame +
 creating and allocating an exception + idup-ing a string does account to about
 this much.
Twice as much generated code as actually necessary, and this is just for 3 lines of source code.
Sep 28 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
I refuse to accept any code gen complaints based on DMD. It's 
optimization facilities are generally crappy compared to gdc / 
ldc and not worth caring about - it is just a reference 
implementation after all. Clean and concise library code is more 
important.

Now if the same inlining failure happens with other two compilers 
- that is something worth talking about (I don't know if it 
happens)
Sep 29 2014
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
29-Sep-2014 16:43, Dicebot пишет:
 I refuse to accept any code gen complaints based on DMD. It's
 optimization facilities are generally crappy compared to gdc / ldc and
 not worth caring about - it is just a reference implementation after
 all. Clean and concise library code is more important.

 Now if the same inlining failure happens with other two compilers - that
 is something worth talking about (I don't know if it happens)
+1 -- Dmitry Olshansky
Sep 30 2014