www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - List of Phobos functions that allocate memory?

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Would anyone be willing to take on the ingrate task of creating a 
comprehensive list with all Phobos functions (and more generally 
artifacts) that allocate memory? That would help a lot with focusing the 
discussion.

Andrei
Feb 06 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 6 February 2014 at 16:28:25 UTC, Andrei Alexandrescu 
wrote:
 Would anyone be willing to take on the ingrate task of creating 
 a comprehensive list with all Phobos functions (and more 
 generally artifacts) that allocate memory? That would help a 
 lot with focusing the discussion.

 Andrei

Merging https://github.com/D-Programming-Language/dmd/pull/1886 and running phobos unit tests should make it relatively simple, at least for a first pass.
Feb 06 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 9:21 AM, Andrej Mitrovic wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 https://raw.github.com/AndrejMitrovic/phobos_allocations/master/phobos_allocations.txt

Ah just realized there are duplicates in the report. I guess -vgc is emitting dupes.

Updated to remove duplicate reports.

Thanks. I guess we'd need to cross-reference to function names from there. Andrei
Feb 06 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 10:05 AM, grm wrote:
 On Thursday, 6 February 2014 at 17:57:45 UTC, Andrei Alexandrescu wrote:
 On 2/6/14, 9:21 AM, Andrej Mitrovic wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 https://raw.github.com/AndrejMitrovic/phobos_allocations/master/phobos_allocations.txt

Ah just realized there are duplicates in the report. I guess -vgc is emitting dupes.

Updated to remove duplicate reports.

Thanks. I guess we'd need to cross-reference to function names from there. Andrei

and also new *XY*Exception doesn't indicate a problem necessarily

Good point. Seems to me code inspection would be a simpler way. Andrei
Feb 06 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 2/6/14, 10:15 AM, Andrej Mitrovic wrote:
 On 2/6/14, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 Thanks. I guess we'd need to cross-reference to function names from there.

Updated to include function names.

Noice. One less phobos_allocations.txt | grep 'In function'| sed "s/.*'\\(.*\\)':/\\1/"|sort|uniq >phobos_allocating_functions.txt later, and... Andrei
Feb 06 2014
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
06-Feb-2014 22:15, Andrej Mitrovic пишет:
 On 2/6/14, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 Thanks. I guess we'd need to cross-reference to function names from there.

Updated to include function names.

Somehow diffing this with coverage report may help filter out CTFE. Some bugs are features :) -- Dmitry Olshansky
Feb 06 2014
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Feb-2014 00:15, H. S. Teoh пишет:
 On Thu, Feb 06, 2014 at 11:39:30PM +0400, Dmitry Olshansky wrote:
 06-Feb-2014 22:15, Andrej Mitrovic пишет:
 On 2/6/14, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 Thanks. I guess we'd need to cross-reference to function names from
 there.

Updated to include function names.

Somehow diffing this with coverage report may help filter out CTFE. Some bugs are features :)

I thought *all* bugs are features... unintentional features. :-P

O.T. From a pragmatic point of view any specific property of a system that is useful to the enduser is a feature. Not all bugs are useful ;)
 T

-- Dmitry Olshansky
Feb 06 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 10:05 AM, Johannes Pfau wrote:
 Am Thu, 06 Feb 2014 16:32:08 +0000
 schrieb "Dicebot" <public dicebot.lv>:

 On Thursday, 6 February 2014 at 16:28:25 UTC, Andrei Alexandrescu
 wrote:
 Would anyone be willing to take on the ingrate task of creating
 a comprehensive list with all Phobos functions (and more
 generally artifacts) that allocate memory? That would help a
 lot with focusing the discussion.

 Andrei

Merging https://github.com/D-Programming-Language/dmd/pull/1886 and running phobos unit tests should make it relatively simple, at least for a first pass.

That's only for implicit allocations though. And please, don't merge yet, it'll get another rewrite this weekend ;-)

Please close if you plan to rewrite.
 One interesting point is that module that were written with avoiding
 allocations in mind usually still allocate when throwing exceptions.

Good point, we need to address that as well. Andrei
Feb 06 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 10:52 AM, fra wrote:
 On Thursday, 6 February 2014 at 18:20:56 UTC, Andrei Alexandrescu wrote:
 One interesting point is that module that were written with avoiding
 allocations in mind usually still allocate when throwing exceptions.

Good point, we need to address that as well. Andrei

Hey, wait a second. How do you throw without allocating?

I don't know yet. That's what the "addressing the problem" will take care of! :o) Andrei
Feb 06 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 11:54 AM, Sean Kelly wrote:
 On Thursday, 6 February 2014 at 18:52:21 UTC, fra wrote:
 On Thursday, 6 February 2014 at 18:20:56 UTC, Andrei Alexandrescu wrote:
 One interesting point is that module that were written with avoiding
 allocations in mind usually still allocate when throwing exceptions.

Good point, we need to address that as well.

Hey, wait a second. How do you throw without allocating?

Does this case even matter? Exceptions are not a normal function of execution, and so should happen rarely to never. And it's a time when I'd expect a delay anyway.

I think it's okay to put this on the backburner and revisit it later. Andrei
Feb 06 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2014 11:54 AM, Sean Kelly wrote:
 Does this case even matter?  Exceptions are not a normal function of execution,
 and so should happen rarely to never.  And it's a time when I'd expect a delay
 anyway.

Right. If you're: 1. using throws as control flow logic 2. requiring a throw in a performance critical loop to be performance critical 3. doing so many throws that the garbage collector needs to run to clean them up you're doing it wrong. I'm tempted to say that the throw expression can call 'new' even if the function is marked as nogc.
Feb 06 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 5:23 PM, Walter Bright wrote:
 I'm tempted to say that the throw expression can call 'new' even if the
 function is marked as  nogc.

That's extreme. A better possibility is to allocate exceptions from a different heap and proclaim that the heap is cleaned once all catch blocks are left. (I'm sure we can find something better, but now is not the time to worry about it.) Andrei
Feb 06 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2014 6:19 PM, Andrei Alexandrescu wrote:
 On 2/6/14, 5:23 PM, Walter Bright wrote:
 I'm tempted to say that the throw expression can call 'new' even if the
 function is marked as  nogc.

That's extreme. A better possibility is to allocate exceptions from a different heap and proclaim that the heap is cleaned once all catch blocks are left. (I'm sure we can find something better, but now is not the time to worry about it.)

That doesn't work, as nothing prevents code from squirreling away the caught exception object handle.
Feb 07 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2014 5:31 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 7 February 2014 at 01:23:44 UTC, Walter Bright wrote:
 Right. If you're:

 1. using throws as control flow logic

 you're doing it wrong.

I disagree. REST based web services tend to use throws all the time. It is a an effective and clean way to break all transactions that are in progress throughout the call chain when you cannot carry through a request, or if the request returns nothing.

They're going to be slow when you do it that way.
Feb 06 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 3:42 AM, Dicebot wrote:
 P.S. Throwing exception is not that slow in D, it is allocating new instance
 that makes a huge impact.

Throwing speed can vary greatly from platform to platform. The idea, as in C++, is when there's a speed tradeoff between throw/catch speed and compromising speed to handle the possibility of exceptions, the non-throw case gets priority.
Feb 07 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 7:33 AM, Adam D. Ruppe wrote:
 On Friday, 7 February 2014 at 11:37:16 UTC, Ola Fosheim Grøstad wrote:
 How slow is slow? Is it slower than in Go and Python?

One problem with allocating the exception is the stop-the-world thing. My cgi.d's built in httpd does some allocations in its constructor, which is run once per request. It can answer requests at a rate of about 6000/sec on my computer...

The gc is not the real speed issue with exceptions, after all, one can preallocate the exception: throw new Exception(); v.s. e = new Exception(); ... throw e; It's the unwinding speed. Just have a look at what deh2.d has to do.
Feb 07 2014
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Feb-2014 23:45, Walter Bright пишет:
 On 2/7/2014 7:33 AM, Adam D. Ruppe wrote:
 On Friday, 7 February 2014 at 11:37:16 UTC, Ola Fosheim Grøstad wrote:
 How slow is slow? Is it slower than in Go and Python?

One problem with allocating the exception is the stop-the-world thing. My cgi.d's built in httpd does some allocations in its constructor, which is run once per request. It can answer requests at a rate of about 6000/sec on my computer...

The gc is not the real speed issue with exceptions, after all, one can preallocate the exception: throw new Exception(); v.s. e = new Exception(); ... throw e;

And the standard library basically can't do this for every function.
 It's the unwinding speed. Just have a look at what deh2.d has to do.

It's deh.d or rather deh_win32./ deh_win64_posix.d and it doesn't look like _all_ that lot especially if you have no finally blocks and the only catch is the top-most catch-all. After all error codes would also have to propagate up the same call stack depth. -- Dmitry Olshansky
Feb 07 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 12:51 PM, Dmitry Olshansky wrote:
 It's deh.d or rather deh_win32./ deh_win64_posix.d and it doesn't look like
 _all_ that lot especially if you have no finally blocks and the only catch is
 the top-most catch-all.

It's a heluva lot slower than "jmp".
Feb 08 2014
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
09-Feb-2014 02:17, Walter Bright пишет:
 On 2/7/2014 12:51 PM, Dmitry Olshansky wrote:
 It's deh.d or rather deh_win32./ deh_win64_posix.d and it doesn't look
 like
 _all_ that lot especially if you have no finally blocks and the only
 catch is
 the top-most catch-all.

It's a heluva lot slower than "jmp".

If you can show me how a single unconditional jump propagates error code 4 calls up the stack I'm sold. I do understand it's slow, it's not that slow to make difference in the discussed case. It's all about jumping to the wrong conclusions. To put it in one pitch: it should be possible to throw/catch in excess of 100k exceptions per second no problem at all (assuming a single core of some run of the mill modern CPU). Nobody is asking to optimize it better then the normal flow. -- Dmitry Olshansky
Feb 09 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/9/2014 2:17 AM, Dmitry Olshansky wrote:
 If you can show me how a single unconditional jump propagates error code 4
calls
 up the stack I'm sold.

 I do understand it's slow, it's not that slow to make difference in the
 discussed case. It's all about jumping to the wrong conclusions.

 To put it in one pitch: it should be possible to throw/catch in excess of 100k
 exceptions per second no problem at all (assuming a single core of some run of
 the mill modern CPU).

 Nobody is asking to optimize it better then the normal flow.

It's the table lookup that's inherently slow.
Feb 10 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2014 2:15 PM, Brad Anderson wrote:
 Personally I don't think bad user input qualifies as an exceptional case
because
 it's expected to happen and the program is expected to handle it (and let the
 user know) when it does. That's just a matter of taste though.

It's not a matter of taste. If your input is subject to a DoS attack, don't put exceptions in the control flow.
Feb 06 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2014 7:08 PM, bearophile wrote:
 Walter Bright:

 It's not a matter of taste. If your input is subject to a DoS attack, don't
 put exceptions in the control flow.

Perhaps the world of today malicious attacks on the software you write should be assumed as the default situation, and then the language+library has to offer something less paranoiac on request. That's why some languages have changed their sorting and hashing routines to make them a little slower but safer on default.

DoS attack resistance requires faster code, not slower code.
Feb 07 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 6:50 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 The specific problem was that it was possible to provoke hash collisions by
 sending carefully crafted input, causing the hash-tables to degrade to linked
 lists. The small performance penalty of using collision-resistant hashes is
 certainly worth it in this case.

That has nothing to do with needing exceptions in the control flow path (and the performance penalty for using exceptions in this manner is certainly not small).
Feb 08 2014
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Feb-2014 06:44, Walter Bright пишет:
 On 2/6/2014 2:15 PM, Brad Anderson wrote:
 Personally I don't think bad user input qualifies as an exceptional
 case because
 it's expected to happen and the program is expected to handle it (and
 let the
 user know) when it does. That's just a matter of taste though.

It's not a matter of taste. If your input is subject to a DoS attack, don't put exceptions in the control flow.

Meh. If exceptions are such a liability we'd better make them (much) faster. -- Dmitry Olshansky
Feb 07 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Feb-2014 20:49, Sean Kelly пишет:
 On Friday, 7 February 2014 at 16:41:00 UTC, Dmitry Olshansky wrote:
 Meh. If exceptions are such a liability we'd better make them (much)
 faster.

It's not stack unwinding speed that's an issue here though, but rather that for client-facing services, throwing an exception when an invalid request is received gives malicious clients an opportunity to hurt service performance by flooding it with invalid requests.

Why throwing a single exception is such a big problem? Surely even C's long_jump wasn't that expensive? *Maybe* we shouldn't re-construct full stack trace on every throw?
 Improving the
 exception code specifically doesn't help here because the real issue is
 with GC collections.

Then the problem is that something so temporary as an exception is allocated on the GC heap in the first place? Let's go for something more sane and deprecate the current behavior, it's not like we are forever stuck with it.
I'd say that the real fix is for such services to
 simply not throw in this case.  But the exception could always be
 recycled as well (since in this case you know that throwing will abort
 the transaction and so will always be immediately discarded).

Exceptions are convenient and they make life that much easier combined with ctors/dtors and scoped lifetime. And then we say **ck it - for busy services, just use good ol': ... if (check42(...) == -1){ call_cleanup42(); return -1; } ... And up the callstack we march. The moment code gets non-trivial there come exceptions and RAII to save the day, I don't see how busy REST services are unlike anything else.
 I'm not
 convinced that there's any need for a language change here to support
 scoped exceptions.  That seems a bit like killing the ant with a
 steamroller.

Well I'm not convinced we should accept that exceptions are many times slower then error codes (with checks on every function that may fail + propagating up the stack). -- Dmitry Olshansky
Feb 07 2014
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Feb-2014 22:54, Sean Kelly пишет:
 On Friday, 7 February 2014 at 17:06:36 UTC, Dmitry Olshansky
 wrote:
 It's not stack unwinding speed that's an issue here though, but rather
 that for client-facing services, throwing an exception when an invalid
 request is received gives malicious clients an opportunity to hurt
 service performance by flooding it with invalid requests.

Why throwing a single exception is such a big problem? Surely even C's long_jump wasn't that expensive? *Maybe* we shouldn't re-construct full stack trace on every throw?

That can be turned off at run time by clearing the traceHandler.

Which should be somehow prominently advertised for release builds. Last time I checked not making it null made exceptions ridiculously slow.
 But yeah, it's the allocations that are a problem in this case,
 not the unwinding.  And specifically, that flooding with bad
 requests effectively generates tons of garbage (an allocation for
 the exception plus another for the trace data) thus triggering
 frequent stop-the-world collections.

So again - the problem is allocations on GC heap. Then let's please not worry about tiny gains of avoiding stack unwind, that is well understood. And I see no reason for allocating exceptions on GC (and none presented so far). The main use case of exception is to consume exception on catch or forward it down the line. Storing a reference to an exception elsewhere is rare case. I could see the whole situation with exceptions in D as "we copied this shit from Java, no idea why" Java at least does go to great lengths to make them fast (by caching them behind the scenes and whatnot).
 Exceptions are convenient and they make life that much easier combined
 with ctors/dtors and scoped lifetime. And then we say **ck it - for
 busy services, just use good ol':
 ...
 if (check42(...) == -1){ call_cleanup42(); return -1; }
 ...

 And up the callstack we march. The moment code gets non-trivial there
 come exceptions and RAII to save the day, I don't see how busy REST
 services are unlike anything else.

I'm sure you can see how a service is different from a desktop application, right?

Aye, in fact I haven't written much in the way of desktop apps.
 In the latter case, there's only one user
 and he's interested in having his application perform well.
 Outside of a QA lab you won't find desktop app. users
 deliberately trying to break their app.  Services are exactly the
 opposite.  It's not an exaggeration when I say that the services
 I work on are under attack from botnets 24/7.  This is a use case
 that must be considered as a first order of business or the
 entire service suffers.

I bet some sanity checks on the level of protocol handling is more then enough. Yeah these might be faster then unwinding due to shear volume of bad data, but it's a fraction of code albeit a critical fraction. I was thinking about the service logic on top of that.
 I'm not convinced that there's any need for a language change here to
 support scoped exceptions.  That seems a bit like killing the ant
 with a steamroller.

Well I'm not convinced we should accept that exceptions are many times slower then error codes (with checks on every function that may fail + propagating up the stack).

Exception-oriented code is typically faster for the success case because all that return code checking can be removed. But the tradeoff is that it's slower in the failure case because stack unwinding is simply slower than checking an error code.

Duly noted. Just stating the obvious - in the majority of cases we talk about 1 unwind vs 10s of checks. The difference isn't THAT big anyway, the only advantage of codes checking is being able to fail faster on some _early_ bad condition.
 But
 again, the issue here isn't the cost of stack unwinding, it's
 that thousands of exceptions thrown per second generates a lot of
 garbage, and garbage collection in D is currently fairly slow
 compared to, say, Java.

Let's stop bashing GC here. This part of design of exceptions in D is just backwards (penalizes usual case) - time to fix it? -- Dmitry Olshansky
Feb 07 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 9:06 AM, Dmitry Olshansky wrote:
 Why throwing a single exception is such a big problem?

Because in order to unwind the stack, you need to find the information about the stack layout. This lookup is rather slow. You can make the lookup faster by compromising the function code generation, but this is considered an unacceptable tradeoff.
Feb 08 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
09-Feb-2014 02:03, Walter Bright пишет:
 On 2/7/2014 9:06 AM, Dmitry Olshansky wrote:
 Why throwing a single exception is such a big problem?

Because in order to unwind the stack, you need to find the information about the stack layout. This lookup is rather slow. You can make the lookup faster by compromising the function code generation, but this is considered an unacceptable tradeoff.

A special table lookup can't be slow compared to writing a dummy HTTP 500 response. Just saying. Yes, it's a tad slower then cmp + jz, I do understand that. Again I'm trying to say that framing stack unwinding as the culprit of vibe.d crawling under bad requests is plain wrong, and that was the focal point of the original argument. -- Dmitry Olshansky
Feb 08 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/8/2014 2:11 PM, Dmitry Olshansky wrote:
 09-Feb-2014 02:03, Walter Bright пишет:
 On 2/7/2014 9:06 AM, Dmitry Olshansky wrote:
 Why throwing a single exception is such a big problem?

Because in order to unwind the stack, you need to find the information about the stack layout. This lookup is rather slow. You can make the lookup faster by compromising the function code generation, but this is considered an unacceptable tradeoff.

A special table lookup can't be slow compared to writing a dummy HTTP 500 response. Just saying. Yes, it's a tad slower then cmp + jz, I do understand that. Again I'm trying to say that framing stack unwinding as the culprit of vibe.d crawling under bad requests is plain wrong, and that was the focal point of the original argument.

I don't know how vibe.d works, but my point is using exception handling to implement normal control flow is bad design and it is going to be slow and the reason it is slow is because of the table lookup and unwinding cost, and that is not going to be fixed.
Feb 08 2014
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, February 08, 2014 21:21:40 Walter Bright wrote:
 On 2/8/2014 2:11 PM, Dmitry Olshansky wrote:
 09-Feb-2014 02:03, Walter Bright пишет:
 On 2/7/2014 9:06 AM, Dmitry Olshansky wrote:
 Why throwing a single exception is such a big problem?

Because in order to unwind the stack, you need to find the information about the stack layout. This lookup is rather slow. You can make the lookup faster by compromising the function code generation, but this is considered an unacceptable tradeoff.

A special table lookup can't be slow compared to writing a dummy HTTP 500 response. Just saying. Yes, it's a tad slower then cmp + jz, I do understand that. Again I'm trying to say that framing stack unwinding as the culprit of vibe.d crawling under bad requests is plain wrong, and that was the focal point of the original argument.

I don't know how vibe.d works, but my point is using exception handling to implement normal control flow is bad design and it is going to be slow and the reason it is slow is because of the table lookup and unwinding cost, and that is not going to be fixed.

I wouldn't have considered throwing on an HTTP error to be "flow control." That's normal error handling, and throwing on HTTP errors is exactly what I would have done. It generally makes code a _lot_ cleaner that way, because you don't have to constantly check return codes for errors, and it's using exceptions for exactly what they're there for - reporting and handling errors. You don't want to use exceptions for stuff other than error reporting, and you don't want to use them in situations where the error case is the frequent case, but that shouldn't be the case for HTTP. Exceptions _will_ be slower than other code paths, and you don't want them to be the normal code path. Nothing is going to make exceptions as fast as the normal code paths either. However, D's exceptions are painfully slow - far slower than is reasonable - whether that's because of allocating the exception or unwinding the stack or creating the string for the stack trace or whatever is a matter for investigation, and I'm not about to claim that I know where the bottlenecks are. Fortunately, it looks like Adam Ruppe has found some ways to speed up exceptions: https://github.com/D-Programming-Language/druntime/pull/717 And there may be other improvements that we can implement as well. I agree that there's a limit to how much we can speed up exceptions, but right now, at minimum, we're getting creamed by Java in terms of speed: https://d.puremagic.com/issues/show_bug.cgi?id=9584 - Jonathan M Davis
Feb 08 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 9 February 2014 at 05:57:44 UTC, Jonathan M Davis 
wrote:
 Exceptions _will_ be slower than other code paths, and you 
 don't want them to
 be the normal code path. Nothing is going to make exceptions as 
 fast as the
 normal code paths either. However, D's exceptions are painfully

Just to be pedantic: this is not true. If you have frame based exception meta-info recording then a throw out of recursion (without try-blocks in the recursion) will be faster than normal returns. You unwind down to the try-block with loading a register and a single JMP. All you have to do is to maintain a single linked list of stack frames that can catch. AFAIK the overhead is neglectible if you avoid doing try-blocks in light-weight function calls. You store one pointer per catching stack-frame. That alone is good enough reason to realize that exception handling strategy should be a compiler switch, not a language policy. Because performance depends on what kind of code patterns you have and the architecture. On current gen of x86 CPUs the decode stage of instructions into micro ops and pipelineing ought to be heavy enough that simple BRA instructions "disappear". Thus the offset strategy ought to work well too (injecting data into the code stream near the return point and branch over it if necessary, but usually not).
Feb 09 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
And with profiling you get the call-frequency between functions, 
so a throw could be replaced with:

if (return_address = 0x1234556){...} // 60%
if (return_address = 0x7899324){...} // 30%
slow_unwinding()

That ought to be obvious.
Feb 09 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/8/2014 2:59 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Saturday, 8 February 2014 at 22:03:13 UTC, Walter Bright wrote:
 You can make the lookup faster by compromising the function code generation,
 but this is considered an unacceptable tradeoff.

"Compromising"? You mean they had to modify codegen, which they didn't want to. Clearly, if you know the return address you also could have stack info access close to it (at a fixed offset), at no runtime cost whatsoever.

Ola, I've done it both ways, I actually do know what I'm talking about. I've sometimes been proven wrong here, so you're welcome to do a pull request proving so.
Feb 08 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 10:10 AM, Dicebot wrote:
 As I have already mentioned, they don't necessarily need to be. But that may
 require tweaking language so that pre-allocated exception usage becomes
reliable
 and I don't see tools right now that allow to express neseccary semantics
(can't
 store reference to instance without deep copy)

It is NOT the allocation that's the issue. C++ code has the same issue. It's the exception handling table lookup.
Feb 08 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 10:54 AM, Sean Kelly wrote:
 But yeah, it's the allocations that are a problem in this case,

Code can always pre-allocate the exception that is thrown. There's no reason whatsoever that allocation is required at the throw point, nor is there any reason the thrown exception has to be newly allocated each time. And, as such, this is entirely a coding issue, not a language or runtime one.
Feb 08 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/7/14, 8:40 AM, Dmitry Olshansky wrote:
 07-Feb-2014 06:44, Walter Bright пишет:
 On 2/6/2014 2:15 PM, Brad Anderson wrote:
 Personally I don't think bad user input qualifies as an exceptional
 case because
 it's expected to happen and the program is expected to handle it (and
 let the
 user know) when it does. That's just a matter of taste though.

It's not a matter of taste. If your input is subject to a DoS attack, don't put exceptions in the control flow.

Meh. If exceptions are such a liability we'd better make them (much) faster.

One simple idea is to statically allocate the same exception and rethrow it over and over. After all there's no guarantee a distinct exception is thrown every time, and the approach is still memory safe (though it might surprise the programmer who saves a reference to an old exception). Andrei
Feb 07 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
08-Feb-2014 15:02, Jakob Ovrum пишет:
 On Saturday, 8 February 2014 at 00:49:46 UTC, Andrei Alexandrescu wrote:
 One simple idea is to statically allocate the same exception and
 rethrow it over and over. After all there's no guarantee a distinct
 exception is thrown every time, and the approach is still memory safe
 (though it might surprise the programmer who saves a reference to an
 old exception).

 Andrei

I don't think it's that simple. What happens if an XException causes another XException and they need to be chained together?

If both are thread-local and cached I see no problem whatsoever. The thing is the current "default" of creating exception is AWFUL. And D stands for sane defaults and the simple path being good last time I checked. -- Dmitry Olshansky
Feb 08 2014
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, February 08, 2014 11:17:25 Jakob Ovrum wrote:
 On Saturday, 8 February 2014 at 11:05:38 UTC, Dmitry Olshansky
 wrote:>
 
 If both are thread-local and cached I see no problem whatsoever.
 The thing is the current "default" of creating exception is
 AWFUL.
 And D stands for sane defaults and the simple path being good
 last time I checked.

How is it not a problem? XException's fields (message, location etc) would be overwritten by the latest throw site, and its `next` field would point to itself.

Then we have multiple of them, or we new up another one when a second one is needed. Even if it were only the first exception which avoided the allocation, it would be a big gain, and in most cases, you're only going to get a single exception, or the exceptions will be of different types. - Jonathan M Davis
Feb 08 2014
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 8 February 2014 at 11:27:27 UTC, Jonathan M Davis 
wrote:
 On Saturday, February 08, 2014 11:17:25 Jakob Ovrum wrote:
 On Saturday, 8 February 2014 at 11:05:38 UTC, Dmitry Olshansky
 wrote:>
 
 If both are thread-local and cached I see no problem 
 whatsoever.
 The thing is the current "default" of creating exception is
 AWFUL.
 And D stands for sane defaults and the simple path being good
 last time I checked.

How is it not a problem? XException's fields (message, location etc) would be overwritten by the latest throw site, and its `next` field would point to itself.

Then we have multiple of them, or we new up another one when a second one is needed. Even if it were only the first exception which avoided the allocation, it would be a big gain, and in most cases, you're only going to get a single exception, or the exceptions will be of different types. - Jonathan M Davis

Yes, I'm sure there is a cool solution, I'm just pointing out that it's not as simple as statically allocating. I think it would be a nice exercise to compose such a solution with std.allocator.
Feb 08 2014
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Sunday, 9 February 2014 at 04:38:23 UTC, Marco Leise wrote:
 Yes, it doesn't seem feasible otherwise. Since you can call
 functions recursively you could potentially chain exceptions
 from the same line of code several times.

   catch (Exception e)
   {
       staticException.line = __LINE__;
       staticException.file = __FILE__;
       staticException.next = e;  // e.next is staticException
       throw staticException;
   }

 You'd have to flag staticException as "in use" and spawn a new
 instance every time you need another one of the same type.
 Since there is no way to reset that flag automatically when
 the last user goes out of scope (i.e. ref counting), that's
 not even an option.

 Preallocated exceptions only work if you are confident your
 exception wont be recursively thrown and thereby chained to
 itself. Granted, the majority of code, but really too much
 cognitive load when writing exception handling code.

While writes directly to line and file and such can't be prevented, `next` could be implemented as a property that does the conditional .dup when assigned to itself (or throw an Error).
Feb 09 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/8/14, 3:02 AM, Jakob Ovrum wrote:
 On Saturday, 8 February 2014 at 00:49:46 UTC, Andrei Alexandrescu wrote:
 One simple idea is to statically allocate the same exception and
 rethrow it over and over. After all there's no guarantee a distinct
 exception is thrown every time, and the approach is still memory safe
 (though it might surprise the programmer who saves a reference to an
 old exception).

 Andrei

I don't think it's that simple. What happens if an XException causes another XException and they need to be chained together?

The chaining method detects that and .dup's one of them. Andrei
Feb 08 2014
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, February 07, 2014 16:49:45 Andrei Alexandrescu wrote:
 On 2/7/14, 8:40 AM, Dmitry Olshansky wrote:
 07-Feb-2014 06:44, Walter Bright пишет:
 On 2/6/2014 2:15 PM, Brad Anderson wrote:
 Personally I don't think bad user input qualifies as an exceptional
 case because
 it's expected to happen and the program is expected to handle it (and
 let the
 user know) when it does. That's just a matter of taste though.

It's not a matter of taste. If your input is subject to a DoS attack, don't put exceptions in the control flow.

Meh. If exceptions are such a liability we'd better make them (much) faster.

One simple idea is to statically allocate the same exception and rethrow it over and over. After all there's no guarantee a distinct exception is thrown every time, and the approach is still memory safe (though it might surprise the programmer who saves a reference to an old exception).

As long as exceptions are cloneable, and people are aware of the fact that they tend to be non-unique, then it can be common practice to clone/dup an exception when you need to keep it around. However, the two potential problems with this overall approach are 1. Do we just always allocate one of each exception type per thread (probably in a static constructor for that exception type)? That would result in a fair number of exceptions being allocated up front. The obvious alternative would be to allocate it the first time that it's thrown so that you only end up with exceptions that get used being allocated, but regardless, we need to take close look at the allocation scheme. 2. This sort of thing has a definite impact on enforce and any idioms related to it. We'd need to either adjust enforce, enforceEx, etc. to avoid the allocation, or we'd need to introduce alternatives to them that expect something like a static opCall on the exception type which returns the common exception for that type or some other standard means of getting at the reusable exception. Regardless, we need to agree upon a standard way to define exception types allow with some set of standard idioms for handling them such that we can deal with exceptions generically (particularly with regards to stuff like enforce) rather than it being an ad-hoc per-exception type thing that you can't reasonably rely on. - Jonathan M Davis
Feb 07 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 8:40 AM, Dmitry Olshansky wrote:
 Meh. If exceptions are such a liability we'd better make them (much) faster.

They can be made faster by slowing down non-exception code. This has been debated at length in the C++ community, and the generally accepted answer is that non-exception code performance is preferred and exception performance is thrown under the bus in order to achieve it. I think it's quite a reasonable conclusion.
Feb 08 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/8/2014 9:00 PM, Marco Leise wrote:
 The reasons for slow exceptions in D could be the generation
 of stack trace strings or the garbage collector instead of
 inherent trade offs to keep the successful code path fast.

Sigh, once again, 1. It is not the collector 2. I've implemented it both ways, I know what I'm talking about. You can see the fast exception way in the Win32 code generation, and the slow way in the Linux code generation.
Feb 08 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/8/14, 9:00 PM, Marco Leise wrote:
 Am Sat, 08 Feb 2014 14:01:12 -0800
 schrieb Walter Bright <newshound2 digitalmars.com>:

 On 2/7/2014 8:40 AM, Dmitry Olshansky wrote:
 Meh. If exceptions are such a liability we'd better make them (much) faster.

They can be made faster by slowing down non-exception code. This has been debated at length in the C++ community, and the generally accepted answer is that non-exception code performance is preferred and exception performance is thrown under the bus in order to achieve it. I think it's quite a reasonable conclusion.

https://yourlogicalfallacyis.com/black-or-white The reasons for slow exceptions in D could be the generation of stack trace strings or the garbage collector instead of inherent trade offs to keep the successful code path fast.

This threads is about memory allocation, not exceptions being slow.
 And static allocation isn't an exactly appealing option...

    throw staticException ? staticException : (staticException =
    new SomethingException("Don't do this at home kids!"));

Function calls could do that. Andrei
Feb 09 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 11:53 AM, Jonathan M Davis wrote:
 or to avoid allocating them

Grep for 'throw' in std.datetime shows that every throw is actually: throw new ... and an example: throw new DateTimeException("SYSTEMTIME cannot hold dates prior to the year 1601."); There is no requirement that the new is done there. You can preallocate the DateTimeException statically, and simply keep rethrowing the same exception instance. I.e. the allocation issue is a coding style issue, not a language problem.
Feb 08 2014
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, February 07, 2014 20:40:57 Dmitry Olshansky wrote:
 07-Feb-2014 06:44, Walter Bright пишет:
 On 2/6/2014 2:15 PM, Brad Anderson wrote:
 Personally I don't think bad user input qualifies as an exceptional
 case because
 it's expected to happen and the program is expected to handle it (and
 let the
 user know) when it does. That's just a matter of taste though.

It's not a matter of taste. If your input is subject to a DoS attack, don't put exceptions in the control flow.

Meh. If exceptions are such a liability we'd better make them (much) faster.

Related: http://d.puremagic.com/issues/show_bug.cgi?id=9584 The DOS aspect of exceptions are not something that I've ever thought about or seen discussed before, but one area where I've found the slowness of D's exceptions to be a real pain is in unit tests. I like to test failure cases as well as successful ones, and if you do much of that, your unit tests start taking a long time due to how insanely slow exceptions are in D. So, while in some situations, the solution may be to not use exceptions (or to use them less), I think that we really need to look at doing what's necessary to make exceptions a lot faster - be it to more efficiently deal with stack traces or to avoid allocating them or whatever else we can come up with to make them fast. I think that the approach of assuming that exceptions don't need to be fast, because they're used for error conditions is a bad one. They're not as performance critical as normal code, but their speed still very much matters. - Jonathan M Davis
Feb 07 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 19:54:14 UTC, Jonathan M Davis 
wrote:
 They're not as performance critical as normal code, but their 
 speed still very much matters.

Well, it is at least more difficult to write reliable code when you have to try to avoid them. Still for a webservice you should probably not have to deal with more than 1000 per second on average, assume 1Ghz, then that is like 1.000.000 cycles of running code per stack unwinding. If you sacrifice 10% of that for exception handling that means you have 100.000 cycles to unwind the stack. If the unwound stack is 5 frames deep you have 20.000 cycles per stack frame. If that is not possible something should be done with the Release-version of the runtime. For a webserver you could of course tie the request handler directly to the request object and instantiate different ones for each request type then have all "unwinding" in the object itself. Quirky, but workable.
Feb 07 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 7:27 PM, Adam D. Ruppe wrote:
 On Friday, 7 February 2014 at 03:14:45 UTC, Sean Kelly wrote:
 On Thursday, 6 February 2014 at 22:20:38 UTC, Dicebot wrote:
 UTFException if str is not well-formed.

unbelievable abomination of a function design.

Yeah, that is absurd. It is a bad, bad sign when almost every time you use a function, you write bool ok = true; try validate(s); catch(UTFException) ok = false; if(!ok) {} yet that's how i use validate...

Add a bugzilla and let's define isValid that returns bool! Andrei
Feb 07 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Feb-2014 20:29, Andrej Mitrovic пишет:
 On Friday, 7 February 2014 at 16:27:35 UTC, Andrei Alexandrescu wrote:
 Add a bugzilla and let's define isValid that returns bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code

Much simpler - it returns a special dchar to designate bad encoding. And there is one defined by Unicode spec. -- Dmitry Olshansky
Feb 07 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Feb-2014 21:07, Andrej Mitrovic пишет:
 On 2/7/14, Dmitry Olshansky <dmitry.olsh gmail.com> wrote:
 Much simpler - it returns a special dchar to designate bad encoding. And
 there is one defined by Unicode spec.

A NaN for chars? Sounds great to me! :)

It's called \uFFFD and is specifically for bad encodings. I wonder why nobody had perused the spec when writing std.utf.decode in the first place... 5.22 Best Practice for U+FFFD Substitution When converting text from one character encoding to another, a conversion algorithm may encounter unconvertible code units. This is most commonly caused by some sort of corruption of the source data, so that it does not correctly follow the specification for that character encoding. Examples include dropping a byte in a multibyte encoding such as Shift-JIS, improper concatenation of strings, a mismatch between an encoding declaration and actual encoding of text, use of non-shortest form for UTF-8, and so on. ... Whenever an unconvertible offset is reached during conversion of a code unit sequence: 1. The maximal subpart at that offset should be replaced by a single U+FFFD. 2. The conversion should proceed at the offset immediately after the maximal subpart. --- Fast, simple and according to the standard. Best of all - no stinkin' exceptions! ;) -- Dmitry Olshansky
Feb 07 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/7/2014 12:14 PM, Dmitry Olshansky wrote:
 Fast, simple and according to the standard. Best of all - no stinkin'
 exceptions! ;)

Nice find. Looks good to me.
Feb 08 2014
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
09-Feb-2014 02:16, Walter Bright пишет:
 On 2/7/2014 12:14 PM, Dmitry Olshansky wrote:
 Fast, simple and according to the standard. Best of all - no stinkin'
 exceptions! ;)

Nice find. Looks good to me.

https://d.puremagic.com/issues/show_bug.cgi?id=12113 -- Dmitry Olshansky
Feb 08 2014
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
08-Feb-2014 02:57, Jonathan M Davis пишет:
 On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:
 07-Feb-2014 20:29, Andrej Mitrovic пишет:
 On Friday, 7 February 2014 at 16:27:35 UTC, Andrei Alexandrescu wrote:
 Add a bugzilla and let's define isValid that returns bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code

Much simpler - it returns a special dchar to designate bad encoding. And there is one defined by Unicode spec.

Isn't that actually worse?

No, it's better and more flexible for those who care to repair broken text in case it's broken. We currently have ZERO facilities to work with partly broken UTF and it's not that rare thing to have it.
 Unless you're suggesting that we stop throwing on
 decode errors,

That is exactly what I suggest. then functions like std.array.front will have to check the
 result on every call to see whether it was valid or not and thus whether they
 should throw, which would mean extra overhead over simply having decode throw
 on decode errors.

Why the heck? It will not throw either. In the very end bad encoding is handled by displaying the 'substituted' (typically '?') character in places where it broke not by throwing up hands in the air and spitting "UTF Exception: offset 4302 bad UTF sequence". This is not good enough (in case somebody though that it is). Those who care about throwing add a trivial map!(x => x != '\uFFFD' || die()) over a string, where die function throws an exception.
 validate has no business throwing, and we definitely should
 add isValidUnicode (or isValid or whatever you want to call it) for validation
 purposes. Code can then call that to validate that a string is valid and not
 worry about any UTFExceptions being thrown as long as it doesn't manipulate
 the string in a way that could result in its Unicode becoming invalid.

Yet later down the road decode will triple check that anyway. Just saying. BTW if the string was checked beforehand there is no difference between 2 approaches at all (don't have to check).
 However, I would argue that assuming that everyone is going to validate their
 strings and that pretty much all string-related functions shouldn't ever have
 to worry about invalid Unicode is just begging for subtle bugs all over the
 place IMHO. You're essentially dealing with error codes at that point, and I
 think that experience has shown quite clearly that error codes are generally a
 bad way to go. Almost no one checks them unless they have to. I think that
 having decode throw on invalid Unicode is exactly what it should be doing. The
 problem is that validate shouldn't.

Every single text editor out there seems to disagree with you: they do show you partially substituted text, not a dialog box "My bad, it's broken UTF-8, I'm giving up!". -- Dmitry Olshansky
Feb 08 2014
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
09-Feb-2014 09:35, Marco Leise пишет:
 Am Sat, 08 Feb 2014 15:21:26 +0400
 schrieb Dmitry Olshansky <dmitry.olsh gmail.com>:

 08-Feb-2014 02:57, Jonathan M Davis пишет:
 On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:
 07-Feb-2014 20:29, Andrej Mitrovic пишет:
 On Friday, 7 February 2014 at 16:27:35 UTC, Andrei Alexandrescu wrote:
 Add a bugzilla and let's define isValid that returns bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code

Much simpler - it returns a special dchar to designate bad encoding. And there is one defined by Unicode spec.

Isn't that actually worse?

No, it's better and more flexible for those who care to repair broken text in case it's broken. We currently have ZERO facilities to work with partly broken UTF and it's not that rare thing to have it.

Your argument is unsubstantiated, since we have this already: http://dlang.org/phobos/std_encoding.html#.sanitize

Working with ranges of dchar? Nobody is taking eager validation from your hands anyway.
 Unless you're suggesting that we stop throwing on
 decode errors,

That is exactly what I suggest. then functions like std.array.front will have to check the
 result on every call to see whether it was valid or not and thus whether they
 should throw, which would mean extra overhead over simply having decode throw
 on decode errors.

Why the heck? It will not throw either. In the very end bad encoding is handled by displaying the 'substituted' (typically '?') character in places where it broke not by throwing up hands in the air and spitting "UTF Exception: offset 4302 bad UTF sequence". This is not good enough (in case somebody though that it is). Those who care about throwing add a trivial map!(x => x != '\uFFFD' || die()) over a string, where die function throws an exception.

Thats neither an improvement over calling "validate" nor does that deal with distinguishing between invalid UTF and

Means text is broken but wasn't ever read...
\uFFFD
 in the input.

Hardly makes any difference to the most applications. Normal text doesn't contain \uFFFD. And you can test a string with proper 'validate', it's just that while decoding the default is to substitute.
 validate has no business throwing, and we definitely should
 add isValidUnicode (or isValid or whatever you want to call it) for validation
 purposes. Code can then call that to validate that a string is valid and not
 worry about any UTFExceptions being thrown as long as it doesn't manipulate
 the string in a way that could result in its Unicode becoming invalid.

Yet later down the road decode will triple check that anyway. Just saying. BTW if the string was checked beforehand there is no difference between 2 approaches at all (don't have to check).
 However, I would argue that assuming that everyone is going to validate their
 strings and that pretty much all string-related functions shouldn't ever have
 to worry about invalid Unicode is just begging for subtle bugs all over the
 place IMHO. You're essentially dealing with error codes at that point, and I
 think that experience has shown quite clearly that error codes are generally a
 bad way to go. Almost no one checks them unless they have to. I think that
 having decode throw on invalid Unicode is exactly what it should be doing. The
 problem is that validate shouldn't.

Every single text editor out there seems to disagree with you: they do show you partially substituted text, not a dialog box "My bad, it's broken UTF-8, I'm giving up!".

Editor do different things. They often try to detect the encoding with a fall back to Latin1. If you open a file explicitly as UTF-8 they may display a substitution char or detect the error and use the fall back, as is the case with Geany and

Throwing exception here is not something useful in 90% of cases. Requiring everybody to call sanitize on every string from the outside smells like a wrong default to me.
 gedit does in fact throw an error message at you
 saying "My bad, it's broken UTF-8, I'm giving up!".

I know and it's piece of junk :) Seriously it doesn't even has regular expressions for search and replace! -- Dmitry Olshansky
Feb 09 2014
next sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Dmitry Olshansky"  wrote in message news:ld7dla$pdg$1 digitalmars.com... 

 gedit does in fact throw an error message at you
 saying "My bad, it's broken UTF-8, I'm giving up!".

I know and it's piece of junk :) Seriously it doesn't even has regular expressions for search and replace!

That would be a luxury, gedit doesn't even have auto-indent.
Feb 09 2014
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
17-Feb-2014 06:19, Marco Leise пишет:
 Am Sun, 09 Feb 2014 12:18:41 +0400
 schrieb Dmitry Olshansky <dmitry.olsh gmail.com>:

 09-Feb-2014 09:35, Marco Leise пишет:
 Thats neither an improvement over calling "validate" nor does
 that deal with distinguishing between invalid UTF and

Means text is broken but wasn't ever read...
 \uFFFD
 in the input.

Hardly makes any difference to the most applications. Normal text doesn't contain \uFFFD.

Of course it does. It is a valid symbol and a lot of websites describing the "Specials" Unicode block make use of it, like the one on Wikipedia: http://en.wikipedia.org/wiki/Specials_(Unicode_block) With your definition, pulling such a document from the web and parsing it in D would mean playing on broken strings.

In a sense, \uFFFD means broken encoding. What about lone surrogates? Private use symbols that must not occur in transmission? They all displayed in various Unicode listings. About 'playing on broken strings' - ignoring broken/partially broken strings, I specifically think that it's what most users/use cases want. A more useful and sensible default of decoding is to substitute on broken encoding. And it's a standard procedure. It's particularly better for displaying text. To remind: since it's only a decode you are still in the control of original text - in fact you may re-test what bytes are there IF you want. The way of "throw on bad encoding" could be useful but I hardly see it as what you want for default. I'm wary of breaking code that relies on throwing. For the moment I think the best course of action would be to introduce xdecode or some such that will do substitution on failure, see how it floats and then change ranges/foreach etc to use xdecode.
 [...]
 Every single text editor out there seems to disagree with you: they do
 show you partially substituted text, not a dialog box "My bad, it's
 broken UTF-8, I'm giving up!".



 gedit does in fact throw an error message at you
 saying "My bad, it's broken UTF-8, I'm giving up!".


 I know and it's piece of junk :)
 Seriously it doesn't even has regular expressions for search and replace!

https://yourlogicalfallacyis.com/no-true-scotsman :p

Well, gedit is a nice example of why just throwing exception is not good enough for many apps (editors in particular). The fact that it's piece of junk might be irrelevant ;) -- Dmitry Olshansky
Feb 18 2014
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
08-Feb-2014 03:01, Meta пишет:
 On Friday, 7 February 2014 at 22:57:26 UTC, Jonathan M Davis wrote:
 On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:

lets you access the naked underlying value.

This is ridiculously distracting suggestion and simply has no merits whatsoever. To underline how impractical this suggestion is: currently every code out there expect dchar out of .front not some magic animal called 'Option!char'. -- Dmitry Olshansky
Feb 08 2014
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, February 08, 2014 18:03:54 Meta wrote:
 On Saturday, 8 February 2014 at 11:24:56 UTC, Dmitry Olshansky
 
 wrote:
 08-Feb-2014 03:01, Meta пишет:
 On Friday, 7 February 2014 at 22:57:26 UTC, Jonathan M Davis
 
 wrote:
 On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:

because it lets you access the naked underlying value.

This is ridiculously distracting suggestion and simply has no merits whatsoever. To underline how impractical this suggestion is: currently every code out there expect dchar out of .front not some magic animal called 'Option!char'.

I'm not actually suggesting a replacement. Just wishful thinking on how the function could've been better designed.

I don't see how returning Nullable!dchar would improve decode function at all. Currently, it throws on invalid UTF, so you don't have to check the return value, and your code can avoid caring about decode errors except for the points where you put your catches (which are generally in far fewer places than the number of places that decode gets called - be it directly or indirectly). On the other hand, with Nullable!dchar, you'd have to always check the result or risking hitting an assertion when you don't check the result (or ending up with dchar.init in -release). I don't see how that's better than the current situation at all. It just makes decode harder to use. And Dmitry's suggestion is better than both. We end up returning the Unicode character specifically intended to designate bad encodings (\uFFFD) such that you don't even have to care that there was a decode error. You just decode the string and use it. It will just be one more character in the string that doesn't match what you're looking for for find and the like, and pretty much nothing should choke on it. Anything which then cares about Unicode validity can use isValidUnicode (once we have it) to validate the string instead of relying on decode to throw. It will clean up string processing in the face of invalid Unicode quite nicely. So, I don't see how using Nullable!dchar as you suggest would ever have been a better design. - Jonathan M Davis
Feb 08 2014
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
08-Feb-2014 12:20, Andrej Mitrovic пишет:
 On 2/7/14, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 However, I would argue that assuming that everyone is going to validate
 their
 strings and that pretty much all string-related functions shouldn't ever
 have
 to worry about invalid Unicode is just begging for subtle bugs all over the

 place IMHO.

I suggested we would introduce an overload, not replace the existing function, so this isn't an issue.
 The problem is that you need to check it. This is _slower_ than exceptions in

Do you have any benchmarks for this? I have vague memory about complaining that the exception code is *de-facto* slower, regardless of input. But I'll try to provide some test-cases later and see where we're at.

Just be sure to test on LDC or GDC. DMD results are irrelevant to the performance-minded of our community. Also be sure to copy the whole code involved in a single file not link to Phobos. People tend to thrown figures like ~10% slower with exceptions turned on but you'll never known what exactly they test. -- Dmitry Olshansky
Feb 08 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/7/14, 8:29 AM, Andrej Mitrovic wrote:
 On Friday, 7 February 2014 at 16:27:35 UTC, Andrei Alexandrescu wrote:
 Add a bugzilla and let's define isValid that returns bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code but assigns the return value through another parameter.

.toBugzilla() Andrei
Feb 07 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-02-07 04:19, Sean Kelly wrote:

 I really like vibe.d.  A lot.  But the way HTTP parse errors are handled
 is a disaster.  Do you know what happened when I was testing vibe.d
 recently and I sent it a bad request?  It sent a stack trace as a
 responses.  A stack trace!  To a client!  I was speechless.  Needless to
 say, I don't support the idea of further enabling this design,
 regardless of whether it can be made a pinnacle of elegance.

Ruby on Rails renders a page with a stack trace in development mode and a standard 500 page in production mode. I can't understand how anyone can do web development without that. There's even a plugin that renders a the stack trace as links pointing back to your editor (if supported). It also allows you to navigate the stack trace with a code snippet and simple debugger for each stack frame. Very convenient. -- /Jacob Carlborg
Feb 07 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-02-07 21:56, Sean Kelly wrote:

 I was mostly surprised that the stack trace was written back to
 the client.  I'd expect something like that in a log on the
 server side.  I do see how it would be convenient to have a stack
 trace included in a bug report, but if this feature is disabled
 in release mode then you can't rely on it anyway.  I'd just
 always be checking the logs (where I'd hope the stack trace would
 always be written).

Ruby on Rails always writes the stack trace to the log. In development mode it will also render it to the client. In production mode we use a plugin that sends an email when an exception occurs. The email will contain the full stack trace, environment variables and some other data about the request that failed. BTW, you can do a lot more with HTML than plain text (log files). -- /Jacob Carlborg
Feb 09 2014
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
06-Feb-2014 21:21, Andrej Mitrovic пишет:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 https://raw.github.com/AndrejMitrovic/phobos_allocations/master/phobos_allocations.txt

Ah just realized there are duplicates in the report. I guess -vgc is emitting dupes.

Updated to remove duplicate reports.

Needs to somehow cut down CTFE-only stuff. E.g. std.regex alocates a lot at CTFE (and in debug sections), it's a prominent example of CTFE but there is a _lot_ more in the same theme. -- Dmitry Olshansky
Feb 06 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/6/14, Dicebot <public dicebot.lv> wrote:
 Merging https://github.com/D-Programming-Language/dmd/pull/1886
 and running phobos unit tests should make it relatively simple,
 at least for a first pass.

Running the tests is overkill, all you have to do is iterate over each module and call "-o- -vgc" on it. We have so many allocations in Phobos that I couldn't even upload my text over to a paste site, most sites have a limit of 150Kb! So here it is on github: https://raw.github.com/AndrejMitrovic/phobos_allocations/master/phobos_allocations.txt
Feb 06 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 We have so many allocations in Phobos that I couldn't even upload my
 text over to a paste site, most sites have a limit of 150Kb! So here
 it is on github:

 https://raw.github.com/AndrejMitrovic/phobos_allocations/master/phobos_allocations.txt

Ah just realized there are duplicates in the report. I guess -vgc is emitting dupes.
Feb 06 2014
prev sibling next sibling parent "Martin Cejp" <minexew gmail.com> writes:
On Thursday, 6 February 2014 at 17:18:59 UTC, Andrej Mitrovic 
wrote:
 On 2/6/14, Dicebot <public dicebot.lv> wrote:
 Merging https://github.com/D-Programming-Language/dmd/pull/1886
 and running phobos unit tests should make it relatively simple,
 at least for a first pass.

Running the tests is overkill, all you have to do is iterate over each module and call "-o- -vgc" on it. We have so many allocations in Phobos that I couldn't even upload my text over to a paste site, most sites have a limit of 150Kb! So here it is on github: https://raw.github.com/AndrejMitrovic/phobos_allocations/master/phobos_allocations.txt

Quite a few of those seem to be false positives. E.g. C:\dmd-git\dmd2\src\phobos\std\internal\digest\sha_SSSE3.d(512): Concatenation causes gc allocation "rol "~T2~",5", looks like something that only ever makes sense at compilation time
Feb 06 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 https://raw.github.com/AndrejMitrovic/phobos_allocations/master/phobos_allocations.txt

Ah just realized there are duplicates in the report. I guess -vgc is emitting dupes.

Updated to remove duplicate reports.
Feb 06 2014
prev sibling next sibling parent "grm" <gerhard.mueller gmsoft.at> writes:
On Thursday, 6 February 2014 at 17:57:45 UTC, Andrei Alexandrescu 
wrote:
 On 2/6/14, 9:21 AM, Andrej Mitrovic wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 https://raw.github.com/AndrejMitrovic/phobos_allocations/master/phobos_allocations.txt

Ah just realized there are duplicates in the report. I guess -vgc is emitting dupes.

Updated to remove duplicate reports.

Thanks. I guess we'd need to cross-reference to function names from there. Andrei

lots of them are throws tough
Feb 06 2014
prev sibling next sibling parent "grm" <gerhard.mueller gmsoft.at> writes:
On Thursday, 6 February 2014 at 17:57:45 UTC, Andrei Alexandrescu 
wrote:
 On 2/6/14, 9:21 AM, Andrej Mitrovic wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 On 2/6/14, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 https://raw.github.com/AndrejMitrovic/phobos_allocations/master/phobos_allocations.txt

Ah just realized there are duplicates in the report. I guess -vgc is emitting dupes.

Updated to remove duplicate reports.

Thanks. I guess we'd need to cross-reference to function names from there. Andrei

and also new *XY*Exception doesn't indicate a problem necessarily
Feb 06 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 06 Feb 2014 16:32:08 +0000
schrieb "Dicebot" <public dicebot.lv>:

 On Thursday, 6 February 2014 at 16:28:25 UTC, Andrei Alexandrescu 
 wrote:
 Would anyone be willing to take on the ingrate task of creating 
 a comprehensive list with all Phobos functions (and more 
 generally artifacts) that allocate memory? That would help a 
 lot with focusing the discussion.

 Andrei

Merging https://github.com/D-Programming-Language/dmd/pull/1886 and running phobos unit tests should make it relatively simple, at least for a first pass.

That's only for implicit allocations though. And please, don't merge yet, it'll get another rewrite this weekend ;-) One interesting point is that module that were written with avoiding allocations in mind usually still allocate when throwing exceptions. Here's some example output for std.uuid/digest/path/range/algorithm/curl: http://dpaste.dzfl.pl/96d3725b06e2
Feb 06 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/6/14, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 Thanks. I guess we'd need to cross-reference to function names from there.

Updated to include function names.
Feb 06 2014
prev sibling next sibling parent "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Thursday, 6 February 2014 at 18:25:34 UTC, Andrei Alexandrescu 
wrote:
 Noice. One

 less phobos_allocations.txt | grep 'In function'| sed
 "s/.*'\\(.*\\)':/\\1/"|sort|uniq
phobos_allocating_functions.txt

later, and...

Well I'm just hacking on the -vgc pull to output what I want, but I should read titles better :). Here's the functions: http://codepad.org/3TsPXryX
Feb 06 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/6/14, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 Noice. One

 less phobos_allocations.txt | grep 'In function'| sed
 "s/.*'\\(.*\\)':/\\1/"|sort|uniq >phobos_allocating_functions.txt

 later, and...

Ah you've attached a file, didn't notice it on the left since I usually skim the avatar part: http://forum.dlang.org/thread/ld0d79$2ife$1 digitalmars.com?page=2#post-ld0k2u:242ptu:241:40digitalmars.com
Feb 06 2014
prev sibling next sibling parent "grm" <gerhard.mueller gmsoft.at> writes:
 That's only for implicit allocations though. And please, don't 
 merge
 yet, it'll get another rewrite this weekend ;-)

Please close if you plan to rewrite. Andrei

expecting the requested close, so some OTs (in random order): - bought TDPL shortly after it's been released - was very impressed by the concept - following the NGs since, I guess, 2010 - great community and *very* smart people - had nothing of value to add yet, tough (since I'm stuck with C/C++/Jave and some proprietary stuff) - and today I submitted my first reply, which was incredibly easy. no annoyance! please make this more obvious for guys like me that do not want to register. thx and good luck to you all hope I can contribute my share some day Kind Regards
Feb 06 2014
prev sibling next sibling parent "fra" <a b.it> writes:
On Thursday, 6 February 2014 at 18:20:56 UTC, Andrei Alexandrescu 
wrote:
 One interesting point is that module that were written with 
 avoiding
 allocations in mind usually still allocate when throwing 
 exceptions.

Good point, we need to address that as well. Andrei

Hey, wait a second. How do you throw without allocating?
Feb 06 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 06 Feb 2014 18:52:20 +0000
schrieb "fra" <a b.it>:

 On Thursday, 6 February 2014 at 18:20:56 UTC, Andrei Alexandrescu 
 wrote:
 One interesting point is that module that were written with 
 avoiding
 allocations in mind usually still allocate when throwing 
 exceptions.

Good point, we need to address that as well. Andrei

Hey, wait a second. How do you throw without allocating?

You can store the exception as a global and that's done for the OutOfMemoryError IIRC, but what I meant was 'allocate with the GC'.
Feb 06 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 6 Feb 2014 20:00:50 +0100
schrieb Johannes Pfau <nospam example.com>:

 Am Thu, 06 Feb 2014 18:52:20 +0000
 schrieb "fra" <a b.it>:
 
 On Thursday, 6 February 2014 at 18:20:56 UTC, Andrei Alexandrescu 
 wrote:
 One interesting point is that module that were written with 
 avoiding
 allocations in mind usually still allocate when throwing 
 exceptions.

Good point, we need to address that as well. Andrei

Hey, wait a second. How do you throw without allocating?

You can store the exception as a global and that's done for the OutOfMemoryError IIRC, but what I meant was 'allocate with the GC'.

Oh and in other languages you can throw by value but I think that wouldn't work in D because of exception chaining.
Feb 06 2014
prev sibling next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 6 February 2014 18:05, Johannes Pfau <nospam example.com> wrote:
 Am Thu, 06 Feb 2014 16:32:08 +0000
 schrieb "Dicebot" <public dicebot.lv>:

 On Thursday, 6 February 2014 at 16:28:25 UTC, Andrei Alexandrescu
 wrote:
 Would anyone be willing to take on the ingrate task of creating
 a comprehensive list with all Phobos functions (and more
 generally artifacts) that allocate memory? That would help a
 lot with focusing the discussion.

 Andrei

Merging https://github.com/D-Programming-Language/dmd/pull/1886 and running phobos unit tests should make it relatively simple, at least for a first pass.

That's only for implicit allocations though. And please, don't merge yet, it'll get another rewrite this weekend ;-) One interesting point is that module that were written with avoiding allocations in mind usually still allocate when throwing exceptions. Here's some example output for std.uuid/digest/path/range/algorithm/curl: http://dpaste.dzfl.pl/96d3725b06e2

That message will look much better with vcolumns. ;) Albeit, it also depends on moving fprint(global.stdmsg, ...) => message(...) http://dpaste.dzfl.pl/5b1961918ed6
Feb 06 2014
prev sibling next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 6 February 2014 18:52, fra <a b.it> wrote:
 On Thursday, 6 February 2014 at 18:20:56 UTC, Andrei Alexandrescu wrote:
 One interesting point is that module that were written with avoiding
 allocations in mind usually still allocate when throwing exceptions.

Good point, we need to address that as well. Andrei

Hey, wait a second. How do you throw without allocating?

You can't. :o)
Feb 06 2014
prev sibling next sibling parent "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Thursday, 6 February 2014 at 19:01:33 UTC, Johannes Pfau wrote:
 You can store the exception as a global and that's done for the
 OutOfMemoryError IIRC.

Hmm.. is that even safe? I mean in some case of exception chaining the same object could be overwritten before being thrown again, thereby losing the original exception state. Thinking out loud here..
Feb 06 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Thursday, 6 February 2014 at 19:05:49 UTC, Andrej Mitrovic 
wrote:
 On Thursday, 6 February 2014 at 19:01:33 UTC, Johannes Pfau 
 wrote:
 You can store the exception as a global and that's done for the
 OutOfMemoryError IIRC.

Hmm.. is that even safe? I mean in some case of exception chaining the same object could be overwritten before being thrown again, thereby losing the original exception state. Thinking out loud here..

You could use a circular buffer with appropriate length.
Feb 06 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 6 February 2014 at 18:20:56 UTC, Andrei Alexandrescu 
wrote:
 On 2/6/14, 10:05 AM, Johannes Pfau wrote:
 Am Thu, 06 Feb 2014 16:32:08 +0000
 schrieb "Dicebot" <public dicebot.lv>:

 On Thursday, 6 February 2014 at 16:28:25 UTC, Andrei 
 Alexandrescu
 wrote:
 Would anyone be willing to take on the ingrate task of 
 creating
 a comprehensive list with all Phobos functions (and more
 generally artifacts) that allocate memory? That would help a
 lot with focusing the discussion.

 Andrei

Merging https://github.com/D-Programming-Language/dmd/pull/1886 and running phobos unit tests should make it relatively simple, at least for a first pass.

That's only for implicit allocations though. And please, don't merge yet, it'll get another rewrite this weekend ;-)

Please close if you plan to rewrite.
 One interesting point is that module that were written with 
 avoiding
 allocations in mind usually still allocate when throwing 
 exceptions.

Good point, we need to address that as well.

I'd think fixing that is probably above and beyond what is required to satisfy most people. If you are throwing so many exceptions that GC pauses are a problem you've got more serious problems than the GC. nothrow doesn't concern itself with Error exceptions, I think nogc should just ignore exceptions generally.
 Andrei

Feb 06 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 6 February 2014 at 18:52:21 UTC, fra wrote:
 Hey, wait a second. How do you throw without allocating?

I think exceptions should be ok. You optimize the typical path, and exceptions are (by definition) an exceptional path. If they are also unacceptable, you could restrict yourself to nothrow functions. (Which can still throw Errors... but meh they are even *more* exceptional)
Feb 06 2014
prev sibling next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 6 February 2014 19:03, Iain Buclaw <ibuclaw gdcproject.org> wrote:
 On 6 February 2014 18:05, Johannes Pfau <nospam example.com> wrote:
 Am Thu, 06 Feb 2014 16:32:08 +0000
 schrieb "Dicebot" <public dicebot.lv>:

 On Thursday, 6 February 2014 at 16:28:25 UTC, Andrei Alexandrescu
 wrote:
 Would anyone be willing to take on the ingrate task of creating
 a comprehensive list with all Phobos functions (and more
 generally artifacts) that allocate memory? That would help a
 lot with focusing the discussion.

 Andrei

Merging https://github.com/D-Programming-Language/dmd/pull/1886 and running phobos unit tests should make it relatively simple, at least for a first pass.

That's only for implicit allocations though. And please, don't merge yet, it'll get another rewrite this weekend ;-) One interesting point is that module that were written with avoiding allocations in mind usually still allocate when throwing exceptions. Here's some example output for std.uuid/digest/path/range/algorithm/curl: http://dpaste.dzfl.pl/96d3725b06e2

That message will look much better with vcolumns. ;) Albeit, it also depends on moving fprint(global.stdmsg, ...) => message(...) http://dpaste.dzfl.pl/5b1961918ed6

Saying that, it seems it doesn't show the column number correctly. http://dpaste.dzfl.pl/31c8800e223a
Feb 06 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Feb 06, 2014 at 11:01:18AM -0800, Andrei Alexandrescu wrote:
 On 2/6/14, 10:52 AM, fra wrote:
On Thursday, 6 February 2014 at 18:20:56 UTC, Andrei Alexandrescu wrote:
One interesting point is that module that were written with
avoiding allocations in mind usually still allocate when throwing
exceptions.

Good point, we need to address that as well. Andrei

Hey, wait a second. How do you throw without allocating?

I don't know yet. That's what the "addressing the problem" will take care of! :o)

You can just pre-declare the Exception as a global variable and then throw that. Well, OK, it's cheating because you still have to allocate it then, but the point is that you get to control how it gets allocated at the top-level rather than having the 'new' buried deep down in the function call chain where you can't control whether the code uses 'new' or a custom allocator (it may not know about which allocator to use). Exception prealloc_exc; static this() { prealloc_exc = ... /* use whatever allocation method you want */ } void main() { try { func(); } catch(Exception e) { // you get prealloc_exc here } } void func() { if (error) { // init exception parameters prealloc_exc.msg = ...; /* presumably you preallocate the * message string too, with the * allocator of your choice */ throw prealloc_exc; // N.B. no allocation } } T -- Doubtless it is a good thing to have an open mind, but a truly open mind should be open at both ends, like the food-pipe, with the capacity for excretion as well as absorption. -- Northrop Frye
Feb 06 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 06 Feb 2014 19:08:39 +0000
schrieb "Adam D. Ruppe" <destructionator gmail.com>:

 On Thursday, 6 February 2014 at 18:52:21 UTC, fra wrote:
 Hey, wait a second. How do you throw without allocating?

I think exceptions should be ok. You optimize the typical path, and exceptions are (by definition) an exceptional path. If they are also unacceptable, you could restrict yourself to nothrow functions. (Which can still throw Errors... but meh they are even *more* exceptional)

That depends on your situation. For games and other applications on normal computers it's OK. For games on systems like embedded gaming systems (think like NintendoDS, 4MB ram) you might not have a GC but still want to use exception handling.
Feb 06 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 6 February 2014 at 19:32:11 UTC, Johannes Pfau wrote:
 For games on systems like embedded gaming systems (think like
 NintendoDS, 4MB ram) you might not have a GC but still want to 
 use exception handling.

Yeah, when I toyed with bare metal D, I did exceptions with manual memory management - malloc when throwing (well, I did malloc in _d_newclass so it was transparent to the throwing code), free when catching. But I think a program written for a special environment will have different coding standards from top to bottom, including the need to free in an exception handler and the option to hack druntime.
Feb 06 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Thursday, 6 February 2014 at 18:52:21 UTC, fra wrote:
 On Thursday, 6 February 2014 at 18:20:56 UTC, Andrei 
 Alexandrescu wrote:
 One interesting point is that module that were written with 
 avoiding
 allocations in mind usually still allocate when throwing 
 exceptions.

Good point, we need to address that as well.

Hey, wait a second. How do you throw without allocating?

Does this case even matter? Exceptions are not a normal function of execution, and so should happen rarely to never. And it's a time when I'd expect a delay anyway.
Feb 06 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Feb 06, 2014 at 11:39:30PM +0400, Dmitry Olshansky wrote:
 06-Feb-2014 22:15, Andrej Mitrovic пишет:
On 2/6/14, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
Thanks. I guess we'd need to cross-reference to function names from
there.

Updated to include function names.

Somehow diffing this with coverage report may help filter out CTFE. Some bugs are features :)

I thought *all* bugs are features... unintentional features. :-P T -- Bomb technician: If I'm running, try to keep up.
Feb 06 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Johannes Pfau:

 Here's some example output for
 std.uuid/digest/path/range/algorithm/curl:
 http://dpaste.dzfl.pl/96d3725b06e2

 ./dmd -vgc ~/Dokumente/d/phobos/std/range.d -c -unittest
 /home/jpf/Dokumente/d/phobos/std/range.d(7307): vgc: Array 
 literals cause gc allocation

Since some time in some cases dynamic array literals don't allocate. And there's also this: https://github.com/D-Programming-Language/dmd/pull/2952 the [1, 2]s syntax guarantees no heap allocation. Bye, bearophile
Feb 06 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Thursday, 6 February 2014 at 20:40:28 UTC, bearophile wrote:
 Johannes Pfau:

 Here's some example output for
 std.uuid/digest/path/range/algorithm/curl:
 http://dpaste.dzfl.pl/96d3725b06e2

 ./dmd -vgc ~/Dokumente/d/phobos/std/range.d -c -unittest
 /home/jpf/Dokumente/d/phobos/std/range.d(7307): vgc: Array 
 literals cause gc allocation

Since some time in some cases dynamic array literals don't allocate. And there's also this: https://github.com/D-Programming-Language/dmd/pull/2952 the [1, 2]s syntax guarantees no heap allocation. Bye, bearophile

and auto[$] pull. :/
Feb 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 6 February 2014 at 18:52:21 UTC, fra wrote:
 Hey, wait a second. How do you throw without allocating?

Throw pre-allocated thread-local exception. And make a deep copy for it if it is going to be put into exception chain to avoid modifying one already in chain. I have been told in that PR that some of language features assume exception instances are always unique and rely on it. It sounds like major language design flaw that will block usage of Phobos in memory-caring code even if other issues are taken care of. Probably language spec should be relaxed to fix this.
Feb 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 6 February 2014 at 19:08:40 UTC, Adam D. Ruppe wrote:
 On Thursday, 6 February 2014 at 18:52:21 UTC, fra wrote:
 Hey, wait a second. How do you throw without allocating?

I think exceptions should be ok. You optimize the typical path, and exceptions are (by definition) an exceptional path. If they are also unacceptable, you could restrict yourself to nothrow functions. (Which can still throw Errors... but meh they are even *more* exceptional)

Hardly so. Any exception allocation can trigger GC collection cycle and Phobos does not provide any other way to handle data errors. Any application that operates on some external user input will be subject to DoS attack vector if it uses Phobos directly. It was huge performance killer for vibe.d last time I have checked, for example.
Feb 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 6 February 2014 at 19:54:27 UTC, Sean Kelly wrote:
 Does this case even matter?  Exceptions are not a normal 
 function of execution, and so should happen rarely to never.  
 And it's a time when I'd expect a delay anyway.

Imagine intentionally crafted broken utf as user input in repeated requests. You don't have control over it. Now if Phobos would have only thrown exceptions in really _exceptional_ situations and handled broken input gracefully...
Feb 06 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 6 February 2014 at 21:38:03 UTC, Dicebot wrote:
 On Thursday, 6 February 2014 at 19:08:40 UTC, Adam D. Ruppe 
 wrote:
 On Thursday, 6 February 2014 at 18:52:21 UTC, fra wrote:
 Hey, wait a second. How do you throw without allocating?

I think exceptions should be ok. You optimize the typical path, and exceptions are (by definition) an exceptional path. If they are also unacceptable, you could restrict yourself to nothrow functions. (Which can still throw Errors... but meh they are even *more* exceptional)

Hardly so. Any exception allocation can trigger GC collection cycle and Phobos does not provide any other way to handle data errors. Any application that operates on some external user input will be subject to DoS attack vector if it uses Phobos directly. It was huge performance killer for vibe.d last time I have checked, for example.

Personally I don't think bad user input qualifies as an exceptional case because it's expected to happen and the program is expected to handle it (and let the user know) when it does. That's just a matter of taste though.
Feb 06 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 6 February 2014 at 21:48:13 UTC, Dicebot wrote:
 On Thursday, 6 February 2014 at 19:54:27 UTC, Sean Kelly wrote:
 Does this case even matter?  Exceptions are not a normal 
 function of execution, and so should happen rarely to never.  
 And it's a time when I'd expect a delay anyway.

Imagine intentionally crafted broken utf as user input in repeated requests. You don't have control over it. Now if Phobos would have only thrown exceptions in really _exceptional_ situations and handled broken input gracefully...

You should probably validate utf from all foreign sources. Catch a problem with it as it comes in rather than in some arbitrary part of your program. http://dlang.org/phobos/std_utf.html#.validate
Feb 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 6 February 2014 at 22:15:11 UTC, Brad Anderson wrote:
 Personally I don't think bad user input qualifies as an 
 exceptional case because it's expected to happen and the 
 program is expected to handle it (and let the user know) when 
 it does. That's just a matter of taste though.

I agree. It kills the whole concept of "exceptions are rare so they don't need to be fast when thrown". But it is how quite lot of Phobos is currently designed and, in my opinion, is biggest design mistake of vibe.d too (it uses exceptions to propagate HTTP status codes)
Feb 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 6 February 2014 at 22:18:10 UTC, Brad Anderson wrote:
 You should probably validate utf from all foreign sources. 
 Catch a problem with it as it comes in rather than in some 
 arbitrary part of your program.

 http://dlang.org/phobos/std_utf.html#.validate

pure safe void validate(S)(in S str) if (isSomeString!S); Throws: UTFException if str is not well-formed. ;)
Feb 06 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 6 February 2014 at 22:19:42 UTC, Dicebot wrote:
 On Thursday, 6 February 2014 at 22:15:11 UTC, Brad Anderson 
 wrote:
 Personally I don't think bad user input qualifies as an 
 exceptional case because it's expected to happen and the 
 program is expected to handle it (and let the user know) when 
 it does. That's just a matter of taste though.

I agree. It kills the whole concept of "exceptions are rare so they don't need to be fast when thrown". But it is how quite lot of Phobos is currently designed and, in my opinion, is biggest design mistake of vibe.d too (it uses exceptions to propagate HTTP status codes)

I must admit that I am guilty of sometimes using exceptions for routine control flow too. It's just so convenient compared to validation/consumption. Maybe we should make a list of Phobos functions that throw exceptions and ensure that (for the ones where this makes sense) they non-throwing validators available. If we can stop gc allocating them that'd be even better but I don't think them being gc allocating should hold up nogc.
Feb 06 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 6 February 2014 at 22:20:38 UTC, Dicebot wrote:
 On Thursday, 6 February 2014 at 22:18:10 UTC, Brad Anderson 
 wrote:
 You should probably validate utf from all foreign sources. 
 Catch a problem with it as it comes in rather than in some 
 arbitrary part of your program.

 http://dlang.org/phobos/std_utf.html#.validate

pure safe void validate(S)(in S str) if (isSomeString!S); Throws: UTFException if str is not well-formed. ;)

Heh, well then... let me just wipe this egg off my face. :P
Feb 06 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 6 February 2014 at 21:38:03 UTC, Dicebot wrote:
 Any application that operates on some external user input will 
 be subject to DoS attack vector if it uses Phobos directly.

Hmm, I hadn't considered that. Maybe exceptions could be handled automatically though due to the facts that there are rarely more than one in flight at any time and they typically don't live for long: 1) prohibit escaping of exception objects from catch blocks (we could just say it is undefined behavior in the spec). The data pointed to by the throwable object should be normal though, if you want to keep the exception, you can thus just shallow copy it. 2) Set aside a static (thread local) buffer early on with a size of like 512 bytes. 3) Make "throw new" call a special function which favors the static buffer. It can do a simple bump-the-pointer allocation in the static region or call the regular GC if there isn't enough room (should be extremely rare). throw e; works the same way it does now. You can pre-allocate with some other method if you want. 4) Have the compiler automatically insert a call to _d_free_exception in a scope(success) block inside every catch block. It checks the given reference, if it is in the static buffer, just zero it all out. If all the chain is in there, zeroing it will free it all. If there's any GC chained exceptions, zeroing it will orphan them and they'll be freed on the next sweep. Otherwise ... well do nothing, let the GC clean up after it. Proof of concept: bool isThrowable(const ClassInfo ci) { if(ci is null) return false; if(ci is typeid(Throwable)) return true; return isThrowable(ci.base); } byte[512] exceptionHolder = 0; size_t exceptionHolderPosition = 0; extern(C) Object _d_newclass(const ClassInfo ci) { if(!isThrowable(ci)) return _d_newclass_original(ci); auto size = ci.init.length; if(exceptionHolderPosition + size > exceptionHolder.length) return _d_newclass_original(ci); byte[] slice = exceptionHolder[exceptionHolderPosition .. exceptionHolderPosition + size]; exceptionHolderPosition += size; slice[] = ci.init[]; import core.stdc.stdio; printf("Magic allocation to %d\n", exceptionHolderPosition); return cast(Object) slice.ptr; } extern(C) void _d_freeexception(Throwable t) { auto ptr = cast(void*) t; if(ptr >= exceptionHolder.ptr && ptr < exceptionHolder.ptr + exceptionHolder.length) { exceptionHolder[] = 0; exceptionHolderPosition = 0; import core.stdc.stdio; printf("Freeing\n"); } // else do nothing, the GC will handle it } void main() { import std.stdio; try { writefln("%s"); // orphaned argument } catch(Exception e) { scope(success) _d_freeexception(e); writeln(e); } } // copy/paste from druntime as fallback extern (C) void onOutOfMemoryError(); extern (C) void* gc_malloc( size_t sz, uint ba = 0 ); extern (C) Object _d_newclass_original(const ClassInfo ci) { import core.stdc.stdlib; static import core.memory; alias BlkAttr = core.memory.GC.BlkAttr; void* p; if (ci.m_flags & TypeInfo_Class.ClassFlags.isCOMclass) { p = malloc(ci.init.length); if (!p) onOutOfMemoryError(); } else { // TODO: should this be + 1 to avoid having pointers to the next block? BlkAttr attr = BlkAttr.FINALIZE; // extern(C++) classes don't have a classinfo pointer in their vtable so the GC can't finalize them if (ci.m_flags & TypeInfo_Class.ClassFlags.isCPPclass) attr &= ~BlkAttr.FINALIZE; if (ci.m_flags & TypeInfo_Class.ClassFlags.noPointers) attr |= BlkAttr.NO_SCAN; p = gc_malloc(ci.init.length, attr); } // initialize it (cast(byte*) p)[0 .. ci.init.length] = ci.init[]; debug(PRINTF) printf("initialization done\n"); return cast(Object) p; } === Just compile and run normally, the linker will prefer our d_newclass to the one in phobos.lib automatically. And you'll see the throw from writeln went into our static buffer and was freed at the end. I toyed with a few other things too: void main() { import std.stdio; try { try { writefln("%s"); // orphaned argument } catch(Exception e) { scope(success) _d_freeexception(e); // don't forget these throw new Exception("LOL", e); } } catch(Exception e) { scope(success) _d_freeexception(e); writeln(e); writeln(e.next); } } still works. Am I missing a fatal flaw here? It seems to work and is kinda simple to do... exceptions don't really need a huge amount of dynamic memory.
Feb 06 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 6 February 2014 at 22:56:45 UTC, Adam D. Ruppe wrote:
 Proof of concept:

code in a link so the lines aren't broken http://arsdnet.net/dcode/except.d
Feb 06 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 06 Feb 2014 14:08:39 -0500, Adam D. Ruppe  
<destructionator gmail.com> wrote:

 On Thursday, 6 February 2014 at 18:52:21 UTC, fra wrote:
 Hey, wait a second. How do you throw without allocating?

I think exceptions should be ok. You optimize the typical path, and exceptions are (by definition) an exceptional path. If they are also unacceptable, you could restrict yourself to nothrow functions. (Which can still throw Errors... but meh they are even *more* exceptional)

I think if reference counting is added, exceptions would be a prime candidate for using it. They are basically discarded immediately after being handled. -Steve
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 01:23:44 UTC, Walter Bright wrote:
 Right. If you're:

 1. using throws as control flow logic

 you're doing it wrong.

I disagree. REST based web services tend to use throws all the time. It is a an effective and clean way to break all transactions that are in progress throughout the call chain when you cannot carry through a request, or if the request returns nothing.
Feb 06 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Friday, 7 February 2014 at 01:31:17 UTC, Ola Fosheim Grøstad
wrote:
 On Friday, 7 February 2014 at 01:23:44 UTC, Walter Bright wrote:
 Right. If you're:

 1. using throws as control flow logic

 you're doing it wrong.

I disagree. REST based web services tend to use throws all the time. It is a an effective and clean way to break all transactions that are in progress throughout the call chain when you cannot carry through a request, or if the request returns nothing.

I think in the case of people using exceptions for control flow a GC.free in your exception handler would suffice for preventing the GC heap from growing to the point where collection times become a concern.
Feb 06 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 02:19:42 UTC, Andrei Alexandrescu 
wrote:
 A better possibility is to allocate exceptions from a different 
 heap and proclaim that the heap is cleaned once all catch 
 blocks are left.

I wrote a quick proof of concept of this that can be tested right now: http://arsdnet.net/dcode/except.d It hooks _d_newclass to allocate Throwables on a little static bump-the-pointer array. Each catch block has a scope(success) in it that zeroes the throwables area back out to zero.
Feb 06 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 It's not a matter of taste. If your input is subject to a DoS 
 attack, don't put exceptions in the control flow.

Perhaps the world of today malicious attacks on the software you write should be assumed as the default situation, and then the language+library has to offer something less paranoiac on request. That's why some languages have changed their sorting and hashing routines to make them a little slower but safer on default. Bye, bearophile
Feb 06 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Thursday, 6 February 2014 at 21:48:13 UTC, Dicebot wrote:
 On Thursday, 6 February 2014 at 19:54:27 UTC, Sean Kelly wrote:
 Does this case even matter?  Exceptions are not a normal 
 function of execution, and so should happen rarely to never.  
 And it's a time when I'd expect a delay anyway.

Imagine intentionally crafted broken utf as user input in repeated requests. You don't have control over it. Now if Phobos would have only thrown exceptions in really _exceptional_ situations and handled broken input gracefully...

That's a tough one. Bad input typically shouldn't generate an exception, but sometimes doing so is handy from a flow control perspective (I know I know, exceptions aren't for flow control). In the few instances where I use an exception for flow control though (like core.demangle) I always use a static instance, so no allocation occurs, and it's entirely internal to the routine. I think it's fair to say that _an_API_ shouldn't allocate and throw an exception to indicate an expected error condition. For a parser, invalid input definitely applies. So then if the user wants to throw an exception in that case, they can do so themselves. Then the choice of allocation is left to the user, not imposed on them. It's generally really easy to let the user supply a delegate to execute on error too, so they don't even necessarily have to check a return code.
Feb 06 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Thursday, 6 February 2014 at 22:20:38 UTC, Dicebot wrote:
 On Thursday, 6 February 2014 at 22:18:10 UTC, Brad Anderson 
 wrote:
 You should probably validate utf from all foreign sources. 
 Catch a problem with it as it comes in rather than in some 
 arbitrary part of your program.

 http://dlang.org/phobos/std_utf.html#.validate

pure safe void validate(S)(in S str) if (isSomeString!S); Throws: UTFException if str is not well-formed.

And somewhere in the world, darkness fell forever on a bright and beautiful countryside. The monsters poured forth and devoured everything in sight, given strength by that unbelievable abomination of a function design.
Feb 06 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Thursday, 6 February 2014 at 22:56:45 UTC, Adam D. Ruppe wrote:
 On Thursday, 6 February 2014 at 21:38:03 UTC, Dicebot wrote:
 Any application that operates on some external user input will 
 be subject to DoS attack vector if it uses Phobos directly.

Hmm, I hadn't considered that. Maybe exceptions could be handled automatically though due to the facts that there are rarely more than one in flight at any time and they typically don't live for long: [snipped lengthy example]

I really like vibe.d. A lot. But the way HTTP parse errors are handled is a disaster. Do you know what happened when I was testing vibe.d recently and I sent it a bad request? It sent a stack trace as a responses. A stack trace! To a client! I was speechless. Needless to say, I don't support the idea of further enabling this design, regardless of whether it can be made a pinnacle of elegance.
Feb 06 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 03:19:32 UTC, Sean Kelly wrote:
 It sent a stack trace as a responses.  A stack trace!  To a 
 client!  I was speechless.

lol, my cgi.d will do that too if you compile with -debug.... I find it convenient at times. (It also sends it to stderr but when doing cgi apps, that means digging into the apache log which is a pain compared to just looking at the browser)
Feb 06 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 03:14:45 UTC, Sean Kelly wrote:
 On Thursday, 6 February 2014 at 22:20:38 UTC, Dicebot wrote:
 UTFException if str is not well-formed.

unbelievable abomination of a function design.

Yeah, that is absurd. It is a bad, bad sign when almost every time you use a function, you write bool ok = true; try validate(s); catch(UTFException) ok = false; if(!ok) {} yet that's how i use validate... fun fact, my little toy scripting language supports var a = try foo();; // if foo throws, a == the exception object but it's a toy scripting language, ugly crap is allowed there :)
Feb 06 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Friday, 7 February 2014 at 01:31:17 UTC, Ola Fosheim Grøstad 
wrote:
 On Friday, 7 February 2014 at 01:23:44 UTC, Walter Bright wrote:
 Right. If you're:

 1. using throws as control flow logic

 you're doing it wrong.

I disagree. REST based web services tend to use throws all the time. It is a an effective and clean way to break all transactions that are in progress throughout the call chain when you cannot carry through a request, or if the request returns nothing.

But let this be up to the programmer working on the service, not imposed on them by the API. Then if they run into something like this DoS issue they can fix it. My experience with these services is that performance is critical and bad input is common, because people are always trying to hack your shit. Where I work, people are serious about performance, our daily volume is ridiculous, and our goal is five nine's of uptime across the board. At the same time, really good asynchronous programmers are about as rare as water on the moon. So something like vibe.d, where mid-level programmers could write correct code that still performs well thanks to the underlying event model, would be a godsend. But only if I really can get what I pay for. The thing I think a lot of people don't realize these days is that performance per watt is just about the most important thing there is. Data centers are expensive, slow to build, and rack space is limited. If you can find a way to increase the concurrent load per box by, say, an order of magnitude by choosing a different language or programming model or whatever, there's a real economic motivation to do so. Java gets by by having a really good GC and a low barrier of entry, but its scalability is really pretty poor all things considered. On the other hand, C/C++ scales tremendously but then you're stuck with the burden those languages impose in terms of semantic complexity, bug frequency, and so on. D seems really promising here but can't rely on having a fantastic incremental GC like Java, and so I think it's a mistake to use Java as a model for how to manage memory. And maybe Java just got it wrong anyway. I know some people who had to go to ridiculous lengths to avoid GC collection cycles in Java because a collection in the app took _20_seconds_ to complete. Now maybe the application was poorly designed or they should have been using an aftermarket GC, but even so. Finally, library programming is the one place where premature optimization really is a good idea, because you can never be sure how people will be using your code. That allocation may not be a big deal to you or 98% of your users, but for the one big client who calls that routine in a tight inner loop or operates at volumes you never conceived of it's a deal breaker. I really don't want Phobos to be the deal breaker :-)
Feb 06 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 6 February 2014 at 21:38:03 UTC, Dicebot wrote:
 Hardly so. Any exception allocation can trigger GC collection 
 cycle and Phobos does not provide any other way to handle data 
 errors. Any application that operates on some external user 
 input will be subject to DoS attack vector if it uses Phobos 
 directly.

Thinking about this more it'd probably be a good idea to use the type system to segregate non-validated user input from the rest of your program. UnvalidatedString or something. UnvalidatedString.validate() returns a string you can then use in the regular fashion. That way unvalidated data can't weasel its way into the trusted portion of your program without getting checked first. Anyway, that's just an idea (and getting further and further off topic).
Feb 06 2014
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, February 06, 2014 22:20:37 Dicebot wrote:
 On Thursday, 6 February 2014 at 22:18:10 UTC, Brad Anderson wrote:
 You should probably validate utf from all foreign sources.
 Catch a problem with it as it comes in rather than in some
 arbitrary part of your program.
 
 http://dlang.org/phobos/std_utf.html#.validate

pure safe void validate(S)(in S str) if (isSomeString!S); Throws: UTFException if str is not well-formed. ;)

In general, I think that throwing on malformed Unicode is a good thing, because it results in code that's less error-prone (as the alternative is to not validate Unicode and try and continue somehow regardless of bad input when decoding Unicode, which would be very bad IMHO). That being said, validating strings when they enter the program is a good way to localize any failures - which is where validate would come in - and I have to agree that the fact that validate throws is horrific. It's a classic example of a function that should return a bool rather than throw. You're asking it whether the string is valid, not asking to report errors when your normal control flow encounters an error that prevents it from functioning normally (which is where exceptions should normally be used). As such, I think that it's clear that we need a new function to replace it (e.g. isValidUnicode). I'll have to take a look at it. If I'm lucky, it won't even take all that long to implement. - Jonathan M Davis
Feb 07 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Dicebot:

 Now if Phobos would have only thrown exceptions in really 
 _exceptional_ situations and handled broken input gracefully...

I wrote two small ideas to reduce throwing exceptions in Phobos: http://d.puremagic.com/issues/show_bug.cgi?id=6840 http://d.puremagic.com/issues/show_bug.cgi?id=11913 Bye, bearophile
Feb 07 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 7 February 2014 at 01:31:17 UTC, Ola Fosheim Grøstad 
wrote:
 On Friday, 7 February 2014 at 01:23:44 UTC, Walter Bright wrote:
 Right. If you're:

 1. using throws as control flow logic

 you're doing it wrong.

I disagree. REST based web services tend to use throws all the time. It is a an effective and clean way to break all transactions that are in progress throughout the call chain when you cannot carry through a request, or if the request returns nothing.

And it is horrible. Exceptions were never designed for this. Try benchmarking trivial vibe.d REST service looking up an entry in an array and throwing 404 upon failure. Difference in performanc between "all requests are 200" and "all requests are 404" will be of order of magnitude.
Feb 07 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 7 February 2014 at 03:14:45 UTC, Sean Kelly wrote:
 pure  safe void validate(S)(in S str) if (isSomeString!S);

 Throws:
 UTFException if str is not well-formed.

And somewhere in the world, darkness fell forever on a bright and beautiful countryside. The monsters poured forth and devoured everything in sight, given strength by that unbelievable abomination of a function design.

True words indeed! To sum up this small thread : I am perfectly OK with exceptions not showing in -vgc if we also agree on cleaning up Phobos from control flow exceptions.
Feb 07 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 7 February 2014 at 05:25:26 UTC, Brad Anderson wrote:
 Thinking about this more it'd probably be a good idea to use the
 type system to segregate non-validated user input from the rest
 of your program. UnvalidatedString or something.
 UnvalidatedString.validate() returns a string you can then use 
 in
 the regular fashion. That way unvalidated data can't weasel its
 way into the trusted portion of your program without getting
 checked first. Anyway, that's just an idea (and getting further
 and further off topic).

Yes, I even had some simple proof-of-concept drafts of such approach for vibe.d but have never finished it. User input is not a problem if Phobos will provide more strongly typed nothrow tools.
Feb 07 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 02:42:14 UTC, Walter Bright wrote:
 They're going to be slow when you do it that way.

How slow is slow? Is it slower than in Go and Python? Why would unwinding 8 stack frames be so slow? Is it a language mandated speed issue or just a runtime issue that could be fixed with a compiler switch? Most of the time is spent waiting for async request from memcaches/databases and other types of network traffic so you usually have some free cycles on a decent CPU. With native code and lightweight threads (coroutines) you should be able to handle 100+ concurrent requests per process.
Feb 07 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 11:37:16 UTC, Ola Fosheim Grøstad 
wrote:
 usually have some free cycles on a decent CPU. With native code 
 and lightweight threads (coroutines) you should be able to 
 handle 100+ concurrent requests per process.

When I think of it you could probably just push the RESTException throwing coroutine onto a "delayed request queue" since a timeout on a transaction might be no worse than aborting it (or carry along some kind of context object). That would make DoS less problematic too and you get better latency for good requests and complete the bad requests when you are idle.
Feb 07 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 7 February 2014 at 11:37:16 UTC, Ola Fosheim Grøstad 
wrote:
 Is it a language mandated speed issue?

It is assumed by http://dlang.org/errors.html
Feb 07 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 7 February 2014 at 11:41:43 UTC, Dicebot wrote:
 On Friday, 7 February 2014 at 11:37:16 UTC, Ola Fosheim Grøstad 
 wrote:
 Is it a language mandated speed issue?

It is assumed by http://dlang.org/errors.html

P.S. Throwing exception is not that slow in D, it is allocating new instance that makes a huge impact.
Feb 07 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 6 February 2014 at 22:15:11 UTC, Brad Anderson wrote:
 Personally I don't think bad user input qualifies as an 
 exceptional case because it's expected to happen and the 
 program is expected to handle it (and let the user know) when 
 it does. That's just a matter of taste though.

Hmm... then what _does_ qualify as exceptional in your opinion? A logic error (i.e. a mistake on the programmers side) doesn't, IMO, it should abort instead. On the other hand, there is the class of situations where e.g. a system call returns an error (say, "permission denied" when opening a file, or out of disk space). Or more generally, an external service, like a database or a remote server. However, I can't see how these are fundamentally different from invalid user input, and indeed, there's often not even a clear separation, e.g. when a user asked you to read a file they don't have access to. So, what's left then?
Feb 07 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 7 February 2014 at 14:26:48 UTC, Marc Schütz wrote:
 Hmm... then what _does_ qualify as exceptional in your opinion?

 A logic error (i.e. a mistake on the programmers side) doesn't, 
 IMO, it should abort instead. On the other hand, there is the 
 class of situations where e.g. a system call returns an error 
 (say, "permission denied" when opening a file, or out of disk 
 space). Or more generally, an external service, like a database 
 or a remote server. However, I can't see how these are 
 fundamentally different from invalid user input, and indeed, 
 there's often not even a clear separation, e.g. when a user 
 asked you to read a file they don't have access to.

 So, what's left then?

It is exceptional situation if input is supposed to be valid but surprisingly is not. For example, calling `decodeGrapheme` on external string without making sure it is valid first. Same goes for file - trying open a missing file is exceptional, but checking for file presence is not.
Feb 07 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 7 February 2014 at 08:30:35 UTC, Walter Bright wrote:
 On 2/6/2014 7:08 PM, bearophile wrote:
 That's why some languages have changed their sorting and 
 hashing routines to
 make them a little slower but safer on default.

DoS attack resistance requires faster code, not slower code.

The specific problem was that it was possible to provoke hash collisions by sending carefully crafted input, causing the hash-tables to degrade to linked lists. The small performance penalty of using collision-resistant hashes is certainly worth it in this case.
Feb 07 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 7 February 2014 at 08:30:35 UTC, Walter Bright wrote:
 On 2/6/2014 7:08 PM, bearophile wrote:
 Walter Bright:

 It's not a matter of taste. If your input is subject to a DoS 
 attack, don't
 put exceptions in the control flow.

Perhaps the world of today malicious attacks on the software you write should be assumed as the default situation, and then the language+library has to offer something less paranoiac on request. That's why some languages have changed their sorting and hashing routines to make them a little slower but safer on default.

DoS attack resistance requires faster code, not slower code.

I think bearophile is referring to a practice of avoiding fast average-case, slow worst-case algorithms in favour of faster worst-cases. If an algorithm has best-case O(n*log(n)) and worst case O(n^2), it's often not practical to build for the worst case, but anything less than that can make you vulnerable to malicious input as part of DOS. In comparison, an algorithm with O(n*log^2(n)) average and worst-case might be acceptable in the average case, but will hold up better in the face of attack. I'm not sure how relevant the point is to the general discussion.
Feb 07 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
John Colvin:

 I think bearophile is referring to

Yes, you have explained well my point. Thank you. Bye, bearophile
Feb 07 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 7 February 2014 at 14:42:18 UTC, Dicebot wrote:
 On Friday, 7 February 2014 at 14:26:48 UTC, Marc Schütz wrote:
 Hmm... then what _does_ qualify as exceptional in your opinion?

 A logic error (i.e. a mistake on the programmers side) 
 doesn't, IMO, it should abort instead. On the other hand, 
 there is the class of situations where e.g. a system call 
 returns an error (say, "permission denied" when opening a 
 file, or out of disk space). Or more generally, an external 
 service, like a database or a remote server. However, I can't 
 see how these are fundamentally different from invalid user 
 input, and indeed, there's often not even a clear separation, 
 e.g. when a user asked you to read a file they don't have 
 access to.

 So, what's left then?

It is exceptional situation if input is supposed to be valid but surprisingly is not. For example, calling `decodeGrapheme` on external string without making sure it is valid first.

If the function expects it to be valid but you pass it an invalid value, you're breaking the contract, which is a logic error and thus should be checked for by assert, not by an exception. => Case number one: logic errors, no exceptions should be used here. If however the function doesn't require it to be valid (for `decodeGrapheme` the docs don't say anything, so I assume it doesn't), then it needs to be able to handle invalid input, for example by throwing an exception. => This is an example of case number two: user errors, exceptions are okay here. But Brad Anderson seems to disagree on case two (or maybe case one?). Or is there a third type of situation not covered by these two cases?
 Same goes for file - trying open a missing file is exceptional, 
 but checking for file presence is not.

I agree here, checking for presence is not exceptional.
Feb 07 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 14:26:48 UTC, Marc Schütz wrote:
 or a remote server. However, I can't see how these are 
 fundamentally different from invalid user input, and indeed, 
 there's often not even a clear separation, e.g. when a user 
 asked you to read a file they don't have access to.

I agree. Any situation where it makes sense to say: "Ouch, this is not going to work out, roll back, roll back, let's move out of this module! We need to try a different approach. We are not going to continue with anything productive down this lane, lets go back to the context and get into a new direction." is suitable for exceptions and it makes code reuse, evolution and modification to error reporting easy. - validation and veracity checking - authentication failures - database failures - transactional retries - serious allocation issues - timeouts are all fiiine for exceptions. You get to write a request handler like this: { auto sid = request.authenticate(); auto data = validator(request.getPost('label1','label2','label3')); auto key = model.create_and_put(sid,data); response.writeJson(key); response.status = 201; return; } And you can change the error reporting at the request dispatcher level rather than sifting through 20 different spaghetti-like request handlers trying to figure out if you got it right: { auto sid = request.authenticate(); if (sid<0){ ... return ...; } auto data = request.getPost('label1','label2','label3'); if (data){ data = validate(data); if (data){ auto key = model.create_and_put(sid,data); if (item){ auto ok = response.writeJson(key); if(ok){ response.status = 201; return; } ....; } else { .... ; } } else { .... ; } } else { ... ; } }
Feb 07 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 11:37:16 UTC, Ola Fosheim Grøstad 
wrote:
 How slow is slow? Is it slower than in Go and Python?

One problem with allocating the exception is the stop-the-world thing. My cgi.d's built in httpd does some allocations in its constructor, which is run once per request. It can answer requests at a rate of about 6000/sec on my computer... Until the allocation have gone too much and the GC starts running. Then all the pending requests stop, killing the throughput. (BTW, interestingly, on Linux it uses separate process pools instead of threads. The GC does NOT stop the world since the other processes can keep going. But, if the requests are fairly uniform - as is typically the case with benchmarks - each process hits the GC threshold at about the same time.... ironically, it is the deterministic nature of the GC that leads to the performance killer there.)
Feb 07 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 11:06:47 UTC, Dicebot wrote:
 Yes, I even had some simple proof-of-concept drafts of such 
 approach for vibe.d but have never finished it. User input is 
 not a problem if Phobos will provide more strongly typed 
  nothrow tools.

Yeah, I think using separate types for printing to users is often a good idea too, since then the type system can help with i18n.
Feb 07 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Friday, 7 February 2014 at 15:33:01 UTC, Adam D. Ruppe wrote:
 On Friday, 7 February 2014 at 11:37:16 UTC, Ola Fosheim Grøstad 
 wrote:
 How slow is slow? Is it slower than in Go and Python?

One problem with allocating the exception is the stop-the-world thing. My cgi.d's built in httpd does some allocations in its constructor, which is run once per request. It can answer requests at a rate of about 6000/sec on my computer... Until the allocation have gone too much and the GC starts running. Then all the pending requests stop, killing the throughput. (BTW, interestingly, on Linux it uses separate process pools instead of threads. The GC does NOT stop the world since the other processes can keep going. But, if the requests are fairly uniform - as is typically the case with benchmarks - each process hits the GC threshold at about the same time.... ironically, it is the deterministic nature of the GC that leads to the performance killer there.)

It's obviously not a solution, but you could change that by having each process call GC.reserve() with a different size.
Feb 07 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 08:32:04 UTC, Walter Bright wrote:
 That doesn't work, as nothing prevents code from squirreling 
 away the caught exception object handle.

scope would. I'm just saying. We could also just document it as undefined behavior and leave matters in the user's hands, but this wouldn't jive nicely with safe :(
Feb 07 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Friday, 7 February 2014 at 15:41:59 UTC, Adam D. Ruppe wrote:
 On Friday, 7 February 2014 at 08:32:04 UTC, Walter Bright wrote:
 That doesn't work, as nothing prevents code from squirreling 
 away the caught exception object handle.

scope would. I'm just saying. We could also just document it as undefined behavior and leave matters in the user's hands, but this wouldn't jive nicely with safe :(

Thread stores an uncaught exception reference so it can be rethrown on join(). But I suppose a case could be made that an uncaught exception could either be discarded or abort the app.
Feb 07 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 15:44:08 UTC, Sean Kelly wrote:
 But I suppose a case could be made that an uncaught exception 
 could either be discarded or abort the app.

It could also make a copy at that time on to the regular GC heap and store that (the members of the throwable class are still GC'd so all the store function has to do is a shallow copy, using the RTTI to get the correct size to copy, onto the gc heap). It'd surely be fewer exceptions to get through that than the thrown, caught, and subsequentely discarded typical case.
Feb 07 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 15:48:56 UTC, Adam D. Ruppe wrote:
 It could also make a copy at that time on to the regular GC 
 heap and store that

lol just add in a quick call to .toGC when you want to store it: T toGC(T)(T t) if(is(T==class)) { auto size = typeid(t).init.length; import core.memory; auto ptr = GC.malloc(size); ptr[0 .. size] = (cast(void*) t)[0 .. size]; return cast(T) ptr; }
Feb 07 2014
prev sibling next sibling parent "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Friday, 7 February 2014 at 16:27:35 UTC, Andrei Alexandrescu 
wrote:
 Add a bugzilla and let's define isValid that returns bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code but assigns the return value through another parameter.
Feb 07 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Friday, 7 February 2014 at 16:41:00 UTC, Dmitry Olshansky 
wrote:
 Meh. If exceptions are such a liability we'd better make them 
 (much) faster.

It's not stack unwinding speed that's an issue here though, but rather that for client-facing services, throwing an exception when an invalid request is received gives malicious clients an opportunity to hurt service performance by flooding it with invalid requests. Improving the exception code specifically doesn't help here because the real issue is with GC collections. I'd say that the real fix is for such services to simply not throw in this case. But the exception could always be recycled as well (since in this case you know that throwing will abort the transaction and so will always be immediately discarded). I'm not convinced that there's any need for a language change here to support scoped exceptions. That seems a bit like killing the ant with a steamroller.
Feb 07 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/7/14, Dmitry Olshansky <dmitry.olsh gmail.com> wrote:
 Much simpler - it returns a special dchar to designate bad encoding. And
 there is one defined by Unicode spec.

A NaN for chars? Sounds great to me! :)
Feb 07 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 15:33:01 UTC, Adam D. Ruppe wrote:
 One problem with allocating the exception is the stop-the-world 
 thing.

Ok, well I guess that primarily is an issue for validation errors where you need to return detailed error reporting. "Not Found" etc can be preallocated as immutable, or?
 constructor, which is run once per request. It can answer 
 requests at a rate of about 6000/sec on my computer...

That sounds pretty good, was that as localhost, or over a network?
 (BTW, interestingly, on Linux it uses separate process pools 
 instead of threads. The GC does NOT stop the world since the 
 other processes can keep going. But, if the requests are fairly 
 uniform - as is typically the case with benchmarks - each 
 process hits the GC threshold at about the same time.... 
 ironically, it is the deterministic nature of the GC that leads 
 to the performance killer there.)

You could synchronize them by calling the GC explicitly N seconds after the other process GC or you if you use a load balancer, maybe the GC could be scheduled by the load balancer or notify the load balancer (assuming all requests are short-lived). This won't work for a simulation type server though. (which is what I am most interested in)
Feb 07 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 7 February 2014 at 17:06:36 UTC, Dmitry Olshansky 
wrote:
 I'm not
 convinced that there's any need for a language change here to 
 support
 scoped exceptions.  That seems a bit like killing the ant with 
 a
 steamroller.

Well I'm not convinced we should accept that exceptions are many times slower then error codes (with checks on every function that may fail + propagating up the stack).

As I have already mentioned, they don't necessarily need to be. But that may require tweaking language so that pre-allocated exception usage becomes reliable and I don't see tools right now that allow to express neseccary semantics (can't store reference to instance without deep copy)
Feb 07 2014
prev sibling next sibling parent Jerry <jlquinn optonline.net> writes:
Walter Bright <newshound2 digitalmars.com> writes:

 On 2/6/2014 6:19 PM, Andrei Alexandrescu wrote:
 On 2/6/14, 5:23 PM, Walter Bright wrote:
 I'm tempted to say that the throw expression can call 'new' even if the
 function is marked as  nogc.

That's extreme. A better possibility is to allocate exceptions from a different heap and proclaim that the heap is cleaned once all catch blocks are left. (I'm sure we can find something better, but now is not the time to worry about it.)

That doesn't work, as nothing prevents code from squirreling away the caught exception object handle.

Very naive question (that may have already been answered), but why can't throw use structs instead of classes? Then the exception would propagate by copy rather than passing the object up the stack?
Feb 07 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 18:28:24 UTC, Jerry wrote:
 throw use structs instead of classes?

I think that'd be more costly and would mess up the whole inheritance checks; catch(Exception) wouldn't catch the same children.
Feb 07 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Jerry:

 throw use structs instead of classes?


This thread discusses the (low) performance of D exceptions, and suggests some ideas: https://d.puremagic.com/issues/show_bug.cgi?id=9584 Another thread: https://d.puremagic.com/issues/show_bug.cgi?id=9581 The thread also discusses an old idea from Java: http://www.javaspecialists.eu/archive/Issue187.html Bye, bearophile
Feb 07 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Friday, 7 February 2014 at 17:06:36 UTC, Dmitry Olshansky
wrote:
 07-Feb-2014 20:49, Sean Kelly пишет:
 On Friday, 7 February 2014 at 16:41:00 UTC, Dmitry Olshansky 
 wrote:
 Meh. If exceptions are such a liability we'd better make them 
 (much)
 faster.

It's not stack unwinding speed that's an issue here though, but rather that for client-facing services, throwing an exception when an invalid request is received gives malicious clients an opportunity to hurt service performance by flooding it with invalid requests.

Why throwing a single exception is such a big problem? Surely even C's long_jump wasn't that expensive? *Maybe* we shouldn't re-construct full stack trace on every throw?

That can be turned off at run time by clearing the traceHandler. But yeah, it's the allocations that are a problem in this case, not the unwinding. And specifically, that flooding with bad requests effectively generates tons of garbage (an allocation for the exception plus another for the trace data) thus triggering frequent stop-the-world collections.
 Exceptions are convenient and they make life that much easier 
 combined with ctors/dtors and scoped lifetime. And then we say 
 **ck it - for busy services, just use good ol':
 ...
 if (check42(...) == -1){ call_cleanup42(); return -1; }
 ...

 And up the callstack we march. The moment code gets non-trivial 
 there come exceptions and RAII to save the day, I don't see how 
 busy REST services are unlike anything else.

I'm sure you can see how a service is different from a desktop application, right? In the latter case, there's only one user and he's interested in having his application perform well. Outside of a QA lab you won't find desktop app. users deliberately trying to break their app. Services are exactly the opposite. It's not an exaggeration when I say that the services I work on are under attack from botnets 24/7. This is a use case that must be considered as a first order of business or the entire service suffers.
 I'm not convinced that there's any need for a language change 
 here to support scoped exceptions.  That seems a bit like 
 killing the ant with a steamroller.

Well I'm not convinced we should accept that exceptions are many times slower then error codes (with checks on every function that may fail + propagating up the stack).

Exception-oriented code is typically faster for the success case because all that return code checking can be removed. But the tradeoff is that it's slower in the failure case because stack unwinding is simply slower than checking an error code. But again, the issue here isn't the cost of stack unwinding, it's that thousands of exceptions thrown per second generates a lot of garbage, and garbage collection in D is currently fairly slow compared to, say, Java. If we could get an incremental GC for D I probably wouldn't even care, but I think that's impossible.
Feb 07 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Friday, 7 February 2014 at 18:45:24 UTC, bearophile wrote:
 Jerry:

 throw use structs instead of classes?


This thread discusses the (low) performance of D exceptions, and suggests some ideas: https://d.puremagic.com/issues/show_bug.cgi?id=9584 Another thread: https://d.puremagic.com/issues/show_bug.cgi?id=9581 The thread also discusses an old idea from Java: http://www.javaspecialists.eu/archive/Issue187.html

Okay, I'm going to look into generating traces lazily. I think it should be possible.
Feb 07 2014
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Fri, 07 Feb 2014 10:54:37 -0800, Sean Kelly <sean invisibleduck.org> =
 =

wrote:

 On Friday, 7 February 2014 at 17:06:36 UTC, Dmitry Olshansky
 wrote:
 07-Feb-2014 20:49, Sean Kelly =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
 On Friday, 7 February 2014 at 16:41:00 UTC, Dmitry Olshansky wrote:
 Meh. If exceptions are such a liability we'd better make them (much=




 faster.

It's not stack unwinding speed that's an issue here though, but rath=



 that for client-facing services, throwing an exception when an inval=



 request is received gives malicious clients an opportunity to hurt
 service performance by flooding it with invalid requests.

Why throwing a single exception is such a big problem? Surely even C'=


 long_jump wasn't that expensive? *Maybe* we shouldn't re-construct   =


 full stack trace on every throw?

That can be turned off at run time by clearing the traceHandler. But yeah, it's the allocations that are a problem in this case, not the unwinding. And specifically, that flooding with bad requests effectively generates tons of garbage (an allocation for the exception plus another for the trace data) thus triggering frequent stop-the-world collections.
 Exceptions are convenient and they make life that much easier combine=


 with ctors/dtors and scoped lifetime. And then we say **ck it - for  =


 busy services, just use good ol':
 ...
 if (check42(...) =3D=3D -1){ call_cleanup42(); return -1; }
 ...

 And up the callstack we march. The moment code gets non-trivial there=


 come exceptions and RAII to save the day, I don't see how busy REST  =


 services are unlike anything else.

I'm sure you can see how a service is different from a desktop application, right? In the latter case, there's only one user and he's interested in having his application perform well. Outside of a QA lab you won't find desktop app. users deliberately trying to break their app. Services are exactly the opposite. It's not an exaggeration when I say that the services I work on are under attack from botnets 24/7. This is a use case that must be considered as a first order of business or the entire service suffers.
 I'm not convinced that there's any need for a language change here t=



 support scoped exceptions.  That seems a bit like killing the ant wi=



 a steamroller.

Well I'm not convinced we should accept that exceptions are many time=


 slower then error codes (with checks on every function that may fail =


 propagating up the stack).

Exception-oriented code is typically faster for the success case because all that return code checking can be removed. But the tradeoff is that it's slower in the failure case because stack unwinding is simply slower than checking an error code. But again, the issue here isn't the cost of stack unwinding, it's that thousands of exceptions thrown per second generates a lot of garbage, and garbage collection in D is currently fairly slow compared to, say, Java. If we could get an incremental GC for D I probably wouldn't even care, but I think that's impossible.

Technically, there is no reason that the current GC can't be made = incremental, insofar as incremental means collecting only what is requir= ed = complete the allocation. -- = Adam Wilson GitHub/IRC: LightBender Aurora Project Coordinator
Feb 07 2014
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, February 07, 2014 14:26:47 Marc Schütz" 
<schuetzm gmx.net> puremagic.com wrote:
 On Thursday, 6 February 2014 at 22:15:11 UTC, Brad Anderson wrote:
 Personally I don't think bad user input qualifies as an
 exceptional case because it's expected to happen and the
 program is expected to handle it (and let the user know) when
 it does. That's just a matter of taste though.

Hmm... then what _does_ qualify as exceptional in your opinion?

Honestly, I think that the typical approach of discussing exceptions as being for "exceptional" circumstances is bad. It inevitably leads to confusion and debate over what "exceptional" means. Some programmers would consider that to mean any bad input, whereas others would take it to the extreme that they should only happen when your program is in an invalid state (essentially what we use Errors for). I've found rather that when discussing exceptions it works much better to explain exactly why you'd use them, and I think that that comes primarily down to three types of circumstances. 1. Code which which should succeed most of the time and which would be far cleaner if it's written to throw exceptions - particularly when the alternative would be to check error codes on every function call (which would be incredibly error-prone). A prime example of this would be a parser. It's far cleaner to write a parser which assumes that each step succeeds than it is to constantly check that each one succeeded. It makes it so that only code that could actually encounter an error has to check for it and so that it can easily and cleanly propagate the error to the top. Doing that with error codes would generally be a mess, and unless failure is the norm, efficiency shouldn't be a problem. 2. Code which you can't actually guarantee will ever succeed. There are some cases where you can avoid errors by doing validation before proceeding (e.g. testing strings for Unicode correctness before doing a lot of string processing), but there are others where you either can't validate ahead of time or where you could still end up with an error in spite of your validation. A prime example of this would be operating on files. For, instance, std.file.isDir will tell you whether a particular file is directory or not by returning bool. If that file does not actually exist, then what is isDir supposed to do? All it can do is throw an exception, unless you want to have a separate out parameter to report whether it succeeded or not or change it so that it returns an error code and returns the bool as out parameter, both of which would make it much uglier to use. And isDir can't assert that the file exists, because that's a runtime condition that cannot be fully verified ahead of time. You can (and should) check whether the file exists first if(file.exists) { if(file.isDir) {} else if(file.isFile) {} else {} } but the file system could actually delete that file right out from under you between the call to exists and the call to isDir (or between the calls to isDir and isFile), so validation reduces how often you hit the error case but cannot eliminate it. It should also be rare that isDir will fail (since you should be checking that the file exists first). So, throwing an exception makes perfect sense. You get clean code that's still able to handle error cases rather than them being ignored (as frequently happens with error codes). 3. Code which should succeed most of the time but where doing validation essentially requires doing what you're validating for anyway. Again, parsers are a good example of this. For instance, to validate that "2013-12-22T01:22:27z" is in the valid ISO extended string format for a timestamp, you have to do pretty much exactly the same work that you have to do to parse out all of the values to convert it to something other than a string (e.g. SysTime). So, if you validated it first, you'd be doing the work twice. As such, why validate first? Just have it throw an exception when the parsing fails. And if for some reason, you expect that there's a high chance that the parsing would fail, then you can have a function which returns an error code and passed out the result as an out parameter instead, but that makes the code much uglier and error-prone. So, in most cases, you'd want it to throw an exception on failure. But regardless, you wouldn't want to validate it first as that would just be expensive all the time rather than more expensive in the (hopefully) rare error case. The areas that you want to normally avoid exceptions are when you're validating up front or when the error condition is likely. If you're validating, you're normally asking a question - is this data valid - in which case, returning bool is the correct thing to do, not throwing on failure (though if the result is false, the caller could choose to throw if appropriate). And trying to do something which has a good chance of failing should probably return whether it succeeded or not, because you don't want exceptions to be your normal code path. Also, performance-critical stuff may need to go the error-code path rather than exceptions simply due to it being performance-critical, but in general, error conditions which aren't bugs in your program should be reported via exceptions (not error codes) with validation being used where appropriate to make it so that the error conditions are infrequent. - Jonathan M Davis
Feb 07 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 7 February 2014 at 17:10:15 UTC, Ola Fosheim Grøstad 
wrote:
 Ok, well I guess that primarily is an issue for validation 
 errors where you need to return detailed error reporting. "Not 
 Found" etc can be preallocated as immutable, or?

yeah, preallocating exceptions might be a really good idea.
 That sounds pretty good, was that as localhost, or over a 
 network?

localhost, and it was just hello world, performance of my thing degrades kinda quickly - it never gets /bad/, but it isn't great either once it starts doing more stuff than the basisc (but it is soooo easy to use! for me anyway)
 You could synchronize them by calling the GC explicitly N 
 seconds after the other process GC or you if you use a load 
 balancer, maybe the GC could be scheduled by the load balancer 
 or notify the load balancer (assuming all requests are 
 short-lived).

yeah. I'm not even sure if it would be a big deal in practice because there's often a lull anyway where the gc can get caught up (certainly not a problem for the lower traffic sites I mostly work on)
Feb 07 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Friday, 7 February 2014 at 20:31:00 UTC, Jacob Carlborg wrote:
 On 2014-02-07 04:19, Sean Kelly wrote:

 I really like vibe.d.  A lot.  But the way HTTP parse errors 
 are handled
 is a disaster.  Do you know what happened when I was testing 
 vibe.d
 recently and I sent it a bad request?  It sent a stack trace 
 as a
 responses.  A stack trace!  To a client!  I was speechless.  
 Needless to
 say, I don't support the idea of further enabling this design,
 regardless of whether it can be made a pinnacle of elegance.

Ruby on Rails renders a page with a stack trace in development mode and a standard 500 page in production mode. I can't understand how anyone can do web development without that. There's even a plugin that renders a the stack trace as links pointing back to your editor (if supported). It also allows you to navigate the stack trace with a code snippet and simple debugger for each stack frame. Very convenient.

I was mostly surprised that the stack trace was written back to the client. I'd expect something like that in a log on the server side. I do see how it would be convenient to have a stack trace included in a bug report, but if this feature is disabled in release mode then you can't rely on it anyway. I'd just always be checking the logs (where I'd hope the stack trace would always be written).
Feb 07 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 20:41:01 UTC, Adam D. Ruppe wrote:
 yeah, preallocating exceptions might be a really good idea.

I wonder if it would be possible to get better unwinding speed by only throwing a single type of exception class and only a single catch. Then do pattern matching on an embedded typefield. I.e.: if (e.id & MASK_5xx) {} if (e.id & MASK_409) {} etc. After looking at the code for stack unwinding it seems like keeping the loops short is essential.
Feb 07 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Jonathan M Davis:

 3. Code which should succeed most of the time but where doing 
 validation
 essentially requires doing what you're validating for anyway. 
 Again, parsers
 are a good example of this. For instance, to validate that
 "2013-12-22T01:22:27z" is in the valid ISO extended string 
 format for a
 timestamp, you have to do pretty much exactly the same work 
 that you have to
 do to parse out all of the values to convert it to something 
 other than a
 string (e.g. SysTime). So, if you validated it first, you'd be 
 doing the work
 twice. As such, why validate first? Just have it throw an 
 exception when the
 parsing fails. And if for some reason, you expect that there's 
 a high chance
 that the parsing would fail, then you can have a function which 
 returns an
 error code and passed out the result as an out parameter 
 instead, but that
 makes the code much uglier and error-prone. So, in most cases, 
 you'd want it
 to throw an exception on failure.

Languages with a good type system solve this with Maybe / Nullable / Optional and similar things. It's both safe (and efficient if the result is equivalent to just a wapping struct). Bye, bearophile
Feb 07 2014
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:
 07-Feb-2014 20:29, Andrej Mitrovic пишет:
 On Friday, 7 February 2014 at 16:27:35 UTC, Andrei Alexandrescu wrote:
 Add a bugzilla and let's define isValid that returns bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code

Much simpler - it returns a special dchar to designate bad encoding. And there is one defined by Unicode spec.

Isn't that actually worse? Unless you're suggesting that we stop throwing on decode errors, then functions like std.array.front will have to check the result on every call to see whether it was valid or not and thus whether they should throw, which would mean extra overhead over simply having decode throw on decode errors. validate has no business throwing, and we definitely should add isValidUnicode (or isValid or whatever you want to call it) for validation purposes. Code can then call that to validate that a string is valid and not worry about any UTFExceptions being thrown as long as it doesn't manipulate the string in a way that could result in its Unicode becoming invalid. However, I would argue that assuming that everyone is going to validate their strings and that pretty much all string-related functions shouldn't ever have to worry about invalid Unicode is just begging for subtle bugs all over the place IMHO. You're essentially dealing with error codes at that point, and I think that experience has shown quite clearly that error codes are generally a bad way to go. Almost no one checks them unless they have to. I think that having decode throw on invalid Unicode is exactly what it should be doing. The problem is that validate shouldn't. - Jonathan M Davis
Feb 07 2014
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, February 07, 2014 21:27:04 bearophile wrote:
 Jonathan M Davis:
 3. Code which should succeed most of the time but where doing
 validation
 essentially requires doing what you're validating for anyway.
 Again, parsers
 are a good example of this. For instance, to validate that
 "2013-12-22T01:22:27z" is in the valid ISO extended string
 format for a
 timestamp, you have to do pretty much exactly the same work
 that you have to
 do to parse out all of the values to convert it to something
 other than a
 string (e.g. SysTime). So, if you validated it first, you'd be
 doing the work
 twice. As such, why validate first? Just have it throw an
 exception when the
 parsing fails. And if for some reason, you expect that there's
 a high chance
 that the parsing would fail, then you can have a function which
 returns an
 error code and passed out the result as an out parameter
 instead, but that
 makes the code much uglier and error-prone. So, in most cases,
 you'd want it
 to throw an exception on failure.

Languages with a good type system solve this with Maybe / Nullable / Optional and similar things. It's both safe (and efficient if the result is equivalent to just a wapping struct).

That can be a good solution, but it also then requires checking the result. One of the big advantages of exceptions is that your code can not care except for the relatively few points that catch exceptions and handle them. Where you run into problems is when the failure case is likely. And if that's the case, then something like Maybe or Nullable is definitely better. - Jonathan M Davis
Feb 07 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Friday, 7 February 2014 at 22:57:26 UTC, Jonathan M Davis 
wrote:
 On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:
 07-Feb-2014 20:29, Andrej Mitrovic пишет:
 On Friday, 7 February 2014 at 16:27:35 UTC, Andrei 
 Alexandrescu wrote:
 Add a bugzilla and let's define isValid that returns bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code

Much simpler - it returns a special dchar to designate bad encoding. And there is one defined by Unicode spec.

Isn't that actually worse? Unless you're suggesting that we stop throwing on decode errors, then functions like std.array.front will have to check the result on every call to see whether it was valid or not and thus whether they should throw, which would mean extra overhead over simply having decode throw on decode errors. validate has no business throwing, and we definitely should add isValidUnicode (or isValid or whatever you want to call it) for validation purposes. Code can then call that to validate that a string is valid and not worry about any UTFExceptions being thrown as long as it doesn't manipulate the string in a way that could result in its Unicode becoming invalid. However, I would argue that assuming that everyone is going to validate their strings and that pretty much all string-related functions shouldn't ever have to worry about invalid Unicode is just begging for subtle bugs all over the place IMHO. You're essentially dealing with error codes at that point, and I think that experience has shown quite clearly that error codes are generally a bad way to go. Almost no one checks them unless they have to. I think that having decode throw on invalid Unicode is exactly what it should be doing. The problem is that validate shouldn't. - Jonathan M Davis

You could always return an Option!char. Nullable won't work because it lets you access the naked underlying value.
Feb 07 2014
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, February 07, 2014 23:01:46 Meta wrote:
 On Friday, 7 February 2014 at 22:57:26 UTC, Jonathan M Davis
 
 wrote:
 On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:
 07-Feb-2014 20:29, Andrej Mitrovic пишет:
 On Friday, 7 February 2014 at 16:27:35 UTC, Andrei
 
 Alexandrescu wrote:
 Add a bugzilla and let's define isValid that returns bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code

Much simpler - it returns a special dchar to designate bad encoding. And there is one defined by Unicode spec.

Isn't that actually worse? Unless you're suggesting that we stop throwing on decode errors, then functions like std.array.front will have to check the result on every call to see whether it was valid or not and thus whether they should throw, which would mean extra overhead over simply having decode throw on decode errors. validate has no business throwing, and we definitely should add isValidUnicode (or isValid or whatever you want to call it) for validation purposes. Code can then call that to validate that a string is valid and not worry about any UTFExceptions being thrown as long as it doesn't manipulate the string in a way that could result in its Unicode becoming invalid. However, I would argue that assuming that everyone is going to validate their strings and that pretty much all string-related functions shouldn't ever have to worry about invalid Unicode is just begging for subtle bugs all over the place IMHO. You're essentially dealing with error codes at that point, and I think that experience has shown quite clearly that error codes are generally a bad way to go. Almost no one checks them unless they have to. I think that having decode throw on invalid Unicode is exactly what it should be doing. The problem is that validate shouldn't. - Jonathan M Davis

You could always return an Option!char. Nullable won't work because it lets you access the naked underlying value.

How is that any better than returning an invalid dchar with a specific value? In either case, you have to check the value. With the exception, code doesn't have to care. If the string is invalid, it'll get a UTFException, and it can handle it appropriately, but having to check the return value just adds overhead (albeit minimal) and is error-prone, because it generally won't be checked (and if it is checked, it complicates the calling code, because it has to do the check). Code that doesn't want to risk a UTFException being thrown can validate up front - and that validator function return bool and _not_ throw. But having decode not throw is going to be error-prone. It also doesn't help performance- wise, because it still has to do all of the same validity checks as it decodes. It's just that instead of throwing, it returns an error value. I really think that having decode throw on invalid Unicode is the right decision, and I don't see what we gain by making it not throw. - Jonathan M Davis
Feb 07 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Friday, 7 February 2014 at 23:45:06 UTC, Jonathan M Davis 
wrote:
 On Friday, February 07, 2014 23:01:46 Meta wrote:
 On Friday, 7 February 2014 at 22:57:26 UTC, Jonathan M Davis
 
 wrote:
 On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:
 07-Feb-2014 20:29, Andrej Mitrovic пишет:
 On Friday, 7 February 2014 at 16:27:35 UTC, Andrei
 
 Alexandrescu wrote:
 Add a bugzilla and let's define isValid that returns 
 bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code

Much simpler - it returns a special dchar to designate bad encoding. And there is one defined by Unicode spec.

Isn't that actually worse? Unless you're suggesting that we stop throwing on decode errors, then functions like std.array.front will have to check the result on every call to see whether it was valid or not and thus whether they should throw, which would mean extra overhead over simply having decode throw on decode errors. validate has no business throwing, and we definitely should add isValidUnicode (or isValid or whatever you want to call it) for validation purposes. Code can then call that to validate that a string is valid and not worry about any UTFExceptions being thrown as long as it doesn't manipulate the string in a way that could result in its Unicode becoming invalid. However, I would argue that assuming that everyone is going to validate their strings and that pretty much all string-related functions shouldn't ever have to worry about invalid Unicode is just begging for subtle bugs all over the place IMHO. You're essentially dealing with error codes at that point, and I think that experience has shown quite clearly that error codes are generally a bad way to go. Almost no one checks them unless they have to. I think that having decode throw on invalid Unicode is exactly what it should be doing. The problem is that validate shouldn't. - Jonathan M Davis

You could always return an Option!char. Nullable won't work because it lets you access the naked underlying value.

How is that any better than returning an invalid dchar with a specific value? In either case, you have to check the value. With the exception, code doesn't have to care. If the string is invalid, it'll get a UTFException, and it can handle it appropriately, but having to check the return value just adds overhead (albeit minimal) and is error-prone, because it generally won't be checked (and if it is checked, it complicates the calling code, because it has to do the check).

We have had this discussion at least once before. A hypothetical Option type will not let you do anything with the wrapped value UNTIL you check it, as opposed to returning null, -1, some special Unicode value, etc. Trying to use it before this check is necessarily a compile-time error. This is both faster than exceptions and safer than special "error values" that are only special by convention. I recall that you've worked with Haskell before, so you must know how useful this pattern is.
 Code that doesn't want to risk a UTFException being thrown can 
 validate up
 front - and that validator function return bool and _not_ 
 throw. But having
 decode not throw is going to be error-prone. It also doesn't 
 help performance-
 wise, because it still has to do all of the same validity 
 checks as it
 decodes. It's just that instead of throwing, it returns an 
 error value. I
 really think that having decode throw on invalid Unicode is the 
 right
 decision, and I don't see what we gain by making it not throw.

 - Jonathan M Davis

Feb 07 2014
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Saturday, February 08, 2014 01:26:10 Meta wrote:
 You could always return an Option!char. Nullable won't work
 because it lets you access the naked underlying value.

How is that any better than returning an invalid dchar with a specific value? In either case, you have to check the value. With the exception, code doesn't have to care. If the string is invalid, it'll get a UTFException, and it can handle it appropriately, but having to check the return value just adds overhead (albeit minimal) and is error-prone, because it generally won't be checked (and if it is checked, it complicates the calling code, because it has to do the check).

We have had this discussion at least once before. A hypothetical Option type will not let you do anything with the wrapped value UNTIL you check it, as opposed to returning null, -1, some special Unicode value, etc. Trying to use it before this check is necessarily a compile-time error. This is both faster than exceptions and safer than special "error values" that are only special by convention. I recall that you've worked with Haskell before, so you must know how useful this pattern is.

The problem is that you need to check it. This is _slower_ than exceptions in the normal case, as invalid Unicode should be the rare case. The great thing with exceptions is that you can write your code as if it will always work and don't need to put checks in it everywhere. Instead, you just put try-catch blocks in the (relatively) few places that you want to handle exceptions. Most of your code doesn't care. And if you validate the string before you start doing a bunch of operations on it, then you don't have to worry about a UTFException being thrown. Also, if code fails to validate a string for one reason or another, the error gets reported rather than an invalid return value being ignored. As for returning Optional/Nullable dchar vs an invalid dchar, I don't see much difference. In both cases, you have to check the return value, which is precisely what you don't want to have to do in most cases. And decode has to do the same work to check for valid Unicode whether it throws an exception or returns a value indicating decode-failure, so why have the extra overhead of having to check the result for decode-failure? Just let it throw an exception in that case and handle it in the appropriate part of your code. Returning a Nullable result or a specific bad value that you have to check rather than throwing an exception only makes sense when it's expected that failures are going to be frequent. If failures are infrequent, it's generally far better to use exceptions, because it will lead to much cleaner, less error-prone code. - Jonathan M Davis
Feb 07 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Jonathan M Davis:

 The problem is that you need to check it. This is _slower_ than 
 exceptions in the normal case,

Right, but verifying the correctness of the Unicode encoding of a string probably on average requires much more than time than testing a single conditional. So I think this tiny added time is acceptable. Bye, bearophile
Feb 07 2014
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Saturday, February 08, 2014 02:41:54 bearophile wrote:
 Jonathan M Davis:
 The problem is that you need to check it. This is _slower_ than
 exceptions in the normal case,

Right, but verifying the correctness of the Unicode encoding of a string probably on average requires much more than time than testing a single conditional. So I think this tiny added time is acceptable.

But why even do it in the first place then? The code is cleaner and less error-prone if it uses exceptions. The only argument I can see being made for not using exceptions with decode is efficiency, because it's more cumbersome to use if it's returning error values of some kind rather than just throwing in the rare case that there's a Unicode decoding error. It's also more error- prone than using exceptions, because most code will just skip checking the result. That's one of the big reasons that error codes are generally a bad idea. But since decode has to do the same validity checks whether it returns an invalid dchar or a Nullable!dchar or if it throws, I don't see why not having the exception buys us anything. It just makes the API worse. - Jonathan M Davis
Feb 07 2014
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 02/08/2014 07:44 PM, Brad Anderson wrote:
 On Saturday, 8 February 2014 at 05:04:35 UTC, Jonathan M Davis
 wrote:
 I think that that would call for us to have 3 related but distinct
 functions:

 1. decode, which throws on invalid Unicode. We already have this.

I wonder if it'd be too reckless to just make decode for string nothrow (we want this function to be as fast as possible) and just require that string, by definition, must be valid unicode. to!string and company could validate strings as they come in from foreign sources. This way invalid unicode is caught early and decode gets a speedup. char[] is different because the mutability means it could be made invalid at any time so we can't rely on it staying valid after it's been checked but once a string has been confirmed valid there is no reason to check it for validity ever again.

"☹"[1..$]
Feb 08 2014
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 07 Feb 2014 22:42:00 -0500
schrieb "Jonathan M Davis" <jmdavisProg gmx.com>:

 On Saturday, February 08, 2014 02:41:54 bearophile wrote:
 Jonathan M Davis:
 The problem is that you need to check it. This is _slower_ than
 exceptions in the normal case,

Right, but verifying the correctness of the Unicode encoding of a string probably on average requires much more than time than testing a single conditional. So I think this tiny added time is acceptable.

But why even do it in the first place then? The code is cleaner and less error-prone if it uses exceptions. The only argument I can see being made for not using exceptions with decode is efficiency, because it's more cumbersome to use if it's returning error values of some kind rather than just throwing in the rare case that there's a Unicode decoding error. It's also more error- prone than using exceptions, because most code will just skip checking the result. That's one of the big reasons that error codes are generally a bad idea. But since decode has to do the same validity checks whether it returns an invalid dchar or a Nullable!dchar or if it throws, I don't see why not having the exception buys us anything. It just makes the API worse. - Jonathan M Davis

I agree with both of you. The Unicode standard tells us that it is correct to replace invalid data with that special code point, so it should be used where applicable, e.g. when one sanitizes an invalid string. On the other hand exceptions are clearly superior to error returns. I guess we just have two use cases here. One where invalid encoding is not an error (e.g. for sanitizing purposes) and one where you don't want to lose information and have to enforce correct encoding. Name the first one "decodeSubst" maybe and have decode call that and check for 0xFFFD? -- Marco
Feb 07 2014
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
08-Feb-2014 09:45, Jonathan M Davis пишет:
 On Friday, February 07, 2014 21:04:08 Jonathan M Davis wrote:
 Actually, thinking this through some more, if we can replace invalid Unicode
 with 0xFFFD, and have all algorithms work with that and consider it valid
 Unicode (rather than getting weird bugs due to invalid Unicode), then if
 decode returned that on error rather than throwing, we wouldn't actually need
 to check the return value. It wouldn't matter that the Unicode was invalid.
 So, we wouldn't even need to _care_ that the Unicode was invalid. Anyone who
 _did_ care could call isValidUnicode to validate the Unicode first, and those
 who didn't wouldn't need to worry about UTFException being thrown, because
 everything would still work even if the string was invalid Unicode.

Hm.. yes. I gotta read the whole thread next time :)
 So, if that's indeed what 0xFFFD does, and that's what Dmitry meant by
 proposing that we return that rather than throwing, then I rescind my
 assessment that throwing was the best way to go and have to agree that
 returning 0xFFFD would be better. I was responding under the assumption that
 you had to check for 0xFFFD and respond to it order to avoid having your code
 be buggy, in which case throwing would be far better. But if 0xFFFD is
 considered valid Unicode,

It is.
 then returning that would be a fantastic solution.
 And if that's the case, we only need two functions, not three:

 1. decode, which returns 0xFFFD on decode failure

 2. isValidUnicode, which returns whether the string is valid

Yay.
 And I actually really like the idea that we could just operate on invalid
 Unicode as valid Unicode this way, making it so that most code doesn't need to
 care, and code that _does_ need to care, can validate the strings first. Right
 now, pretty much all string code needs to care in order to avoid processing
 invalid Unicode, which is much messier.

broken text and have some sane results out of it.
 - Jonathan M Davis

-- Dmitry Olshansky
Feb 08 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 8 Feb 2014 05:29:35 +0100
schrieb Marco Leise <Marco.Leise gmx.de>:

 Name the first one "decodeSubst" maybe and have decode call
 that and check for 0xFFFD?

Err... the other way round. 0xFFFD would actually be valid from an encoding point of view, I guess. -- Marco
Feb 07 2014
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, February 08, 2014 05:29:35 Marco Leise wrote:
 I guess we just have two use cases here. One where invalid
 encoding is not an error (e.g. for sanitizing purposes) and
 one where you don't want to lose information and have to
 enforce correct encoding.
 Name the first one "decodeSubst" maybe and have decode call
 that and check for 0xFFFD?

I think that that would call for us to have 3 related but distinct functions: 1. decode, which throws on invalid Unicode. We already have this. 2. isValidUnicode, which returns whether the string is valid Unicode and does not throw. We don't yet have this. Rather, we have validate which does the same job and then throws instead of returning bool. 3. sanitizeUnicode (or whatever would be a good name for it), which replaces invalid Unicode with 0xFFFD (or whatever the appropriate character is) so that it can be operated on without causing decode to throw in spite of the fact that it was invalid Unicode. We don't have anything like this yet. - Jonathan M Davis
Feb 07 2014
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, February 07, 2014 21:04:08 Jonathan M Davis wrote:
 On Saturday, February 08, 2014 05:29:35 Marco Leise wrote:
 I guess we just have two use cases here. One where invalid
 encoding is not an error (e.g. for sanitizing purposes) and
 one where you don't want to lose information and have to
 enforce correct encoding.
 Name the first one "decodeSubst" maybe and have decode call
 that and check for 0xFFFD?

I think that that would call for us to have 3 related but distinct functions: 1. decode, which throws on invalid Unicode. We already have this. 2. isValidUnicode, which returns whether the string is valid Unicode and does not throw. We don't yet have this. Rather, we have validate which does the same job and then throws instead of returning bool. 3. sanitizeUnicode (or whatever would be a good name for it), which replaces invalid Unicode with 0xFFFD (or whatever the appropriate character is) so that it can be operated on without causing decode to throw in spite of the fact that it was invalid Unicode. We don't have anything like this yet.

Actually, thinking this through some more, if we can replace invalid Unicode with 0xFFFD, and have all algorithms work with that and consider it valid Unicode (rather than getting weird bugs due to invalid Unicode), then if decode returned that on error rather than throwing, we wouldn't actually need to check the return value. It wouldn't matter that the Unicode was invalid. So, we wouldn't even need to _care_ that the Unicode was invalid. Anyone who _did_ care could call isValidUnicode to validate the Unicode first, and those who didn't wouldn't need to worry about UTFException being thrown, because everything would still work even if the string was invalid Unicode. So, if that's indeed what 0xFFFD does, and that's what Dmitry meant by proposing that we return that rather than throwing, then I rescind my assessment that throwing was the best way to go and have to agree that returning 0xFFFD would be better. I was responding under the assumption that you had to check for 0xFFFD and respond to it order to avoid having your code be buggy, in which case throwing would be far better. But if 0xFFFD is considered valid Unicode, then returning that would be a fantastic solution. And if that's the case, we only need two functions, not three: 1. decode, which returns 0xFFFD on decode failure 2. isValidUnicode, which returns whether the string is valid And I actually really like the idea that we could just operate on invalid Unicode as valid Unicode this way, making it so that most code doesn't need to care, and code that _does_ need to care, can validate the strings first. Right now, pretty much all string code needs to care in order to avoid processing invalid Unicode, which is much messier. - Jonathan M Davis
Feb 07 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 07 Feb 2014 21:04:08 -0800
schrieb Jonathan M Davis <jmdavisProg gmx.com>:

 On Saturday, February 08, 2014 05:29:35 Marco Leise wrote:
 I guess we just have two use cases here. One where invalid
 encoding is not an error (e.g. for sanitizing purposes) and
 one where you don't want to lose information and have to
 enforce correct encoding.
 Name the first one "decodeSubst" maybe and have decode call
 that and check for 0xFFFD?

I think that that would call for us to have 3 related but distinct functions: 1. decode, which throws on invalid Unicode. We already have this. 2. isValidUnicode, which returns whether the string is valid Unicode and does not throw. We don't yet have this. Rather, we have validate which does the same job and then throws instead of returning bool.

Yes, that's the one that needs to be added.
 3. sanitizeUnicode (or whatever would be a good name for it), which replaces 
 invalid Unicode with 0xFFFD (or whatever the appropriate character is) so that 
 it can be operated on without causing decode to throw in spite of the fact 
 that it was invalid Unicode. We don't have anything like this yet.

And oh wonder, we actually have that already! Problem solved: http://dlang.org/phobos/std_encoding.html#.sanitize (Not that I knew that before hand *cough*) Or does someone have a need to also sanitize code point by code point?
 - Jonathan M Davis

-- Marco
Feb 07 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/7/14, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 However, I would argue that assuming that everyone is going to validate
 their
 strings and that pretty much all string-related functions shouldn't ever
 have
 to worry about invalid Unicode is just begging for subtle bugs all over the

 place IMHO.

I suggested we would introduce an overload, not replace the existing function, so this isn't an issue.
 The problem is that you need to check it. This is _slower_ than exceptions in

Do you have any benchmarks for this? I have vague memory about complaining that the exception code is *de-facto* slower, regardless of input. But I'll try to provide some test-cases later and see where we're at.
Feb 08 2014
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, February 08, 2014 09:20:15 Andrej Mitrovic wrote:
 On 2/7/14, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 However, I would argue that assuming that everyone is going to validate
 their
 strings and that pretty much all string-related functions shouldn't ever
 have
 to worry about invalid Unicode is just begging for subtle bugs all over
 the
 
 place IMHO.

I suggested we would introduce an overload, not replace the existing function, so this isn't an issue.
 The problem is that you need to check it. This is _slower_ than exceptions
 in

Do you have any benchmarks for this? I have vague memory about complaining that the exception code is *de-facto* slower, regardless of input. But I'll try to provide some test-cases later and see where we're at.

The exception version has to all of the same checks that the version which returns an error value would have to do, while the one returning an error value which had to be checked for validity would have an extra check. So, the only ways that the exception version would be slower are if the plumbing for being able to throw an exception from the function makes it slower (assuming that the other would be nothrow) or if the optimizer just does worse with the exception one for some reason. Because the number of operations that the actual D code would be doing in the successful case would be greater for the non-throwing version. Code generation can do entertaining things to efficiency though, so benchmarking would be required to see what would actually happen. However, as I stated in another post, I've reconsidered the situation. I think that I misunderstood what Dmitry was suggesting and that checking the error value is not actually necessary: http://forum.dlang.org/post/mailman.66.1391838333.21734.digitalmars-d puremagic.com And if that's the case, then we can probably move towards having decode not throw and possibly getting rid of UTFException altogether (certainly, most code wouldn't throw it or have to worry about it, since decode and stride are the two main cases where that's a concern, and if they don't throw anymore, then UTFException would have very little use). - Jonathan M Davis
Feb 08 2014
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 8 February 2014 at 00:49:46 UTC, Andrei Alexandrescu 
wrote:
 One simple idea is to statically allocate the same exception 
 and rethrow it over and over. After all there's no guarantee a 
 distinct exception is thrown every time, and the approach is 
 still memory safe (though it might surprise the programmer who 
 saves a reference to an old exception).

 Andrei

I don't think it's that simple. What happens if an XException causes another XException and they need to be chained together?
Feb 08 2014
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 8 February 2014 at 11:05:38 UTC, Dmitry Olshansky 
wrote:>
 If both are thread-local and cached I see no problem whatsoever.
 The thing is the current "default" of creating exception is 
 AWFUL.
 And D stands for sane defaults and the simple path being good 
 last time I checked.

How is it not a problem? XException's fields (message, location etc) would be overwritten by the latest throw site, and its `next` field would point to itself.
Feb 08 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Saturday, 8 February 2014 at 11:24:56 UTC, Dmitry Olshansky 
wrote:
 08-Feb-2014 03:01, Meta пишет:
 On Friday, 7 February 2014 at 22:57:26 UTC, Jonathan M Davis 
 wrote:
 On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:

because it lets you access the naked underlying value.

This is ridiculously distracting suggestion and simply has no merits whatsoever. To underline how impractical this suggestion is: currently every code out there expect dchar out of .front not some magic animal called 'Option!char'.

I'm not actually suggesting a replacement. Just wishful thinking on how the function could've been better designed.
Feb 08 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 8 February 2014 at 16:50:53 UTC, Andrei Alexandrescu 
wrote:
 The chaining method detects that and .dup's one of them.

 Andrei

After some thinking I don't think it actually helps - exception will be modified _before_ throwing in library code so cloning will be to late. But I don't see any reason why basic exception instances in Phobos can't be made immutable.
Feb 08 2014
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 8 February 2014 at 16:50:53 UTC, Andrei Alexandrescu 
wrote:
 On 2/8/14, 3:02 AM, Jakob Ovrum wrote:
 On Saturday, 8 February 2014 at 00:49:46 UTC, Andrei 
 Alexandrescu wrote:
 One simple idea is to statically allocate the same exception 
 and
 rethrow it over and over. After all there's no guarantee a 
 distinct
 exception is thrown every time, and the approach is still 
 memory safe
 (though it might surprise the programmer who saves a 
 reference to an
 old exception).

 Andrei

I don't think it's that simple. What happens if an XException causes another XException and they need to be chained together?

The chaining method detects that and .dup's one of them. Andrei

What if the statically allocated XException is escaped to be inspected later, but before that is thrown again in a separate exception chain? I suppose it would be no different from the current situation, as it's legal to throw exceptions allocated in any fashion, so there is already no guarantee of uniqueness. It's probable that some code out there still takes exception uniqueness for granted, so changing the allocation scheme would be a (typically silent) breaking change, even if the code is arguably broken in the first place. I suppose we could make that breakage a compile error by making exceptions implicitly `scope` at the catch-site, but that would of course be a much more involved change... Personally I still like the idea, but if implemented, I think something should be done about the change in uniqueness at the same time, even if it's just an added note in the language documentation on exceptions.
Feb 08 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Saturday, 8 February 2014 at 05:04:35 UTC, Jonathan M Davis
wrote:
 I think that that would call for us to have 3 related but 
 distinct functions:

 1. decode, which throws on invalid Unicode. We already have 
 this.

I wonder if it'd be too reckless to just make decode for string nothrow (we want this function to be as fast as possible) and just require that string, by definition, must be valid unicode. to!string and company could validate strings as they come in from foreign sources. This way invalid unicode is caught early and decode gets a speedup. char[] is different because the mutability means it could be made invalid at any time so we can't rely on it staying valid after it's been checked but once a string has been confirmed valid there is no reason to check it for validity ever again.
Feb 08 2014
prev sibling next sibling parent "Dominikus Dittes Scherkl" writes:
On Saturday, 8 February 2014 at 18:44:38 UTC, Brad Anderson wrote:
 I wonder if it'd be too reckless to just make decode for string
 nothrow (we want this function to be as fast as possible) and

 just require that string, by definition, must be valid unicode.

Replacement of broken code is defined by unicode - we should use it. Noone prevents you to call isValidUnicode beforehand and handle that sepearately if it returns "false" (I would recomment that only if security is relevant e.g. if you chack a signature or something like that) or search for 0xFFFD in the result string afterwards and throw if you find some (but this is generally no good idea because the replacement characters may have been there even before and were intended). As default relplacing broken characters is very good. And fast.
Feb 08 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 8 February 2014 at 22:11:13 UTC, Dmitry Olshansky 
wrote:
 Again I'm trying to say that framing stack unwinding as the 
 culprit of vibe.d crawling under bad requests is plain wrong, 
 and that was the focal point of the original argument.

Can you see if it is better with this little patch? https://github.com/D-Programming-Language/druntime/pull/717 on a simple test, I got a 20x speedup on most exceptions by lazy generating the stack trace upon request in toString (though if you are printing it anyway you won't see a difference)
Feb 08 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 8 February 2014 at 22:03:13 UTC, Walter Bright wrote:
 You can make the lookup faster by compromising the function 
 code generation, but this is considered an unacceptable 
 tradeoff.

"Compromising"? You mean they had to modify codegen, which they didn't want to. Clearly, if you know the return address you also could have stack info access close to it (at a fixed offset), at no runtime cost whatsoever.
Feb 08 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
But the c++ Dwarf way of doing it was developed for Itanium which 
was targetting HPC, for which you probably don't need exceptions 
all that often. So it made sense in that context.

For regular applications it makes no sense, and with whole 
program analysis (or language level linker) you probably often 
can get a good match at the throw site.
Feb 08 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
AND (this just has to be said) if D is really meant to be a SAFE 
programming language then the language should NOT encourage 
programmers to a coding style where you can fail to test for 
errors. The obvious solution is to ensure that you cannot ignore 
errors unless you are explicit about it. Exceptions ensure that.

Having 3 different ways of returning errors is not a good 
strategy for safe and bug free programming.

Ah, I just had to say it... ;)
Feb 08 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 08 Feb 2014 11:33:51 +0000
schrieb "Jakob Ovrum" <jakobovrum gmail.com>:

 On Saturday, 8 February 2014 at 11:27:27 UTC, Jonathan M Davis 
 wrote:
 On Saturday, February 08, 2014 11:17:25 Jakob Ovrum wrote:
 On Saturday, 8 February 2014 at 11:05:38 UTC, Dmitry Olshansky
 wrote:>
 
 If both are thread-local and cached I see no problem 
 whatsoever.
 The thing is the current "default" of creating exception is
 AWFUL.
 And D stands for sane defaults and the simple path being good
 last time I checked.

How is it not a problem? XException's fields (message, location etc) would be overwritten by the latest throw site, and its `next` field would point to itself.

Then we have multiple of them, or we new up another one when a second one is needed. Even if it were only the first exception which avoided the allocation, it would be a big gain, and in most cases, you're only going to get a single exception, or the exceptions will be of different types. - Jonathan M Davis

Yes, I'm sure there is a cool solution, I'm just pointing out that it's not as simple as statically allocating. I think it would be a nice exercise to compose such a solution with std.allocator.

Yes, it doesn't seem feasible otherwise. Since you can call functions recursively you could potentially chain exceptions from the same line of code several times. catch (Exception e) { staticException.line = __LINE__; staticException.file = __FILE__; staticException.next = e; // e.next is staticException throw staticException; } You'd have to flag staticException as "in use" and spawn a new instance every time you need another one of the same type. Since there is no way to reset that flag automatically when the last user goes out of scope (i.e. ref counting), that's not even an option. Preallocated exceptions only work if you are confident your exception wont be recursively thrown and thereby chained to itself. Granted, the majority of code, but really too much cognitive load when writing exception handling code. -- Marco
Feb 08 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 08 Feb 2014 14:01:12 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:

 On 2/7/2014 8:40 AM, Dmitry Olshansky wrote:
 Meh. If exceptions are such a liability we'd better make them (much) faster.

They can be made faster by slowing down non-exception code. This has been debated at length in the C++ community, and the generally accepted answer is that non-exception code performance is preferred and exception performance is thrown under the bus in order to achieve it. I think it's quite a reasonable conclusion.

https://yourlogicalfallacyis.com/black-or-white The reasons for slow exceptions in D could be the generation of stack trace strings or the garbage collector instead of inherent trade offs to keep the successful code path fast. And static allocation isn't an exactly appealing option... throw staticException ? staticException : (staticException = new SomethingException("Don't do this at home kids!")); and practically out of question when you need to chain exceptions and your call stack could contain this line of code more than once, resulting in infinite loops in exception chains as a new bug type in D, that is fixed by writing: catch (Exception e) { throw (staticException ? (e.linksTo(staticException) ? staticException.dupThenWrap(e) : staticException) : (staticException = new SomethingException("Don't do this at home kids!")); } -- Marco
Feb 08 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 08 Feb 2014 15:21:26 +0400
schrieb Dmitry Olshansky <dmitry.olsh gmail.com>:

 08-Feb-2014 02:57, Jonathan M Davis =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
 On Friday, February 07, 2014 20:43:38 Dmitry Olshansky wrote:
 07-Feb-2014 20:29, Andrej Mitrovic =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
 On Friday, 7 February 2014 at 16:27:35 UTC, Andrei Alexandrescu wrote:
 Add a bugzilla and let's define isValid that returns bool!

Add std.utf.decode() to that as well. IOW, it should have an overload which returns a status code

Much simpler - it returns a special dchar to designate bad encoding. A=



 there is one defined by Unicode spec.

Isn't that actually worse?

No, it's better and more flexible for those who care to repair broken=20 text in case it's broken. We currently have ZERO facilities to work with=

 partly broken UTF and it's not that rare thing to have it.

Your argument is unsubstantiated, since we have this already: http://dlang.org/phobos/std_encoding.html#.sanitize=20
 Unless you're suggesting that we stop throwing on
 decode errors,

That is exactly what I suggest. =20 then functions like std.array.front will have to check the
 result on every call to see whether it was valid or not and thus whethe=


 should throw, which would mean extra overhead over simply having decode=


 on decode errors.

Why the heck? It will not throw either. In the very end bad encoding is=20 handled by displaying the 'substituted' (typically '?') character in=20 places where it broke not by throwing up hands in the air and spitting=20 "UTF Exception: offset 4302 bad UTF sequence". This is not good enough=20 (in case somebody though that it is). =20 Those who care about throwing add a trivial map!(x =3D> x !=3D '\uFFFD' |=

 die()) over a string, where die function throws an exception.

Thats neither an improvement over calling "validate" nor does that deal with distinguishing between invalid UTF and \uFFFD in the input.
 validate has no business throwing, and we definitely should
 add isValidUnicode (or isValid or whatever you want to call it) for val=


 purposes. Code can then call that to validate that a string is valid an=


 worry about any UTFExceptions being thrown as long as it doesn't manipu=


 the string in a way that could result in its Unicode becoming invalid.

Yet later down the road decode will triple check that anyway. Just=20 saying. BTW if the string was checked beforehand there is no difference=20 between 2 approaches at all (don't have to check). =20
 However, I would argue that assuming that everyone is going to validate=


 strings and that pretty much all string-related functions shouldn't eve=


 to worry about invalid Unicode is just begging for subtle bugs all over=


 place IMHO. You're essentially dealing with error codes at that point, =


 think that experience has shown quite clearly that error codes are gene=


 bad way to go. Almost no one checks them unless they have to. I think t=


 having decode throw on invalid Unicode is exactly what it should be doi=


 problem is that validate shouldn't.

Every single text editor out there seems to disagree with you: they do=20 show you partially substituted text, not a dialog box "My bad, it's=20 broken UTF-8, I'm giving up!".

Editor do different things. They often try to detect the encoding with a fall back to Latin1. If you open a file explicitly as UTF-8 they may display a substitution char or detect the error and use the fall back, as is the case with Geany and gedit does in fact throw an error message at you saying "My bad, it's broken UTF-8, I'm giving up!". --=20 Marco
Feb 08 2014
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, February 08, 2014 14:13:04 Walter Bright wrote:
 On 2/7/2014 11:53 AM, Jonathan M Davis wrote:
 or to avoid allocating them

Grep for 'throw' in std.datetime shows that every throw is actually: throw new ... and an example: throw new DateTimeException("SYSTEMTIME cannot hold dates prior to the year 1601."); There is no requirement that the new is done there. You can preallocate the DateTimeException statically, and simply keep rethrowing the same exception instance. I.e. the allocation issue is a coding style issue, not a language problem.

Of course allocation is not a language issue. The question is whether (and how) we can change our approach to allocating exceptions in order to reduce their cost. And that's a change in how we approach them, not a change in the language itself. It might require some changes in druntime to better deal with other allocation schemes (particularly with how that affects exception chaining), but it's not a language issue. And in general, I would expect that any speed-ups that we could attain with regards to actually throwing an exception would be in druntime's implementation rather than anything in the language itself. Any improvements there could then be combined with any improvements we could make to our approach to allocating exceptions (and for better or worse - probably worse - the normal approach at this point is to allocate a new exception when throwing). - Jonathan M Davis
Feb 08 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Am Sat, 08 Feb 2014 14:01:12 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:

 On 2/7/2014 8:40 AM, Dmitry Olshansky wrote:
 Meh. If exceptions are such a liability we'd better make them (much) faster.

They can be made faster by slowing down non-exception code. This has been debated at length in the C++ community, and the generally accepted answer is that non-exception code performance is preferred and exception performance is thrown under the bus in order to achieve it. I think it's quite a reasonable conclusion.

Am Sat, 08 Feb 2014 21:31:53 -0800 schrieb Walter Bright <newshound2 digitalmars.com>:
 On 2/8/2014 9:00 PM, Marco Leise wrote:
 The reasons for slow exceptions in D could be the generation
 of stack trace strings or the garbage collector instead of
 inherent trade offs to keep the successful code path fast.

Sigh, once again, 1. It is not the collector 2. I've implemented it both ways, I know what I'm talking about. You can see the fast exception way in the Win32 code generation, and the slow way in the Linux code generation.

Ok, I'm on Linux which should be inherently slower at throwing exceptions as you say. So I've written a little test and it shows two things: 1. You are right, about the collector. It is not the bottleneck. 2. It doesn't have anything to do with trading speed for the successful code path either. I called two functions recursively until a nesting depth of 1000. The first version allocates a new exception, the second one reuses an existing exception. At the call site I caught the exception. I did this 10_000 times in a loop. [The code is attached.] Even at this nesting depth the second version still outperformed the first one by a factor of ~200(!) and all the CPU time (>98%) was is spent somewhere in libc. Using static exceptions (or similarly in C++: throwing literal strings) is VERY fast in D already and I see no reason to improve that at the moment. So I repeat my point: The reasons for slow exceptions in D could be the generation of stack trace strings or anything else other than some inherent trade offs to keep the successful code path fast. -- Marco
Feb 08 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 08 Feb 2014 21:29:27 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:

 On 2/8/2014 2:59 PM, "Ola Fosheim Gr=C3=B8stad"=20
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Saturday, 8 February 2014 at 22:03:13 UTC, Walter Bright wrote:
 You can make the lookup faster by compromising the function code gener=



 but this is considered an unacceptable tradeoff.

"Compromising"? You mean they had to modify codegen, which they didn't =


 Clearly, if you know the return address you also could have stack info =


 close to it (at a fixed offset), at no runtime cost whatsoever.

Ola, I've done it both ways, I actually do know what I'm talking about. =20 I've sometimes been proven wrong here, so you're welcome to do a pull req=

 proving so.

It is not the function code gen that needs to be improved on Linux, Walter. In fact that would be premature optimization considering that the *construction* of exceptions outweights unwinding costs for functions with no local variables by multiple orders of magnitude. --=20 Marco
Feb 08 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 9 February 2014 at 05:29:25 UTC, Walter Bright wrote:
 Ola, I've done it both ways, I actually do know what I'm 
 talking about.

Please note that "you" and "they" was meant as "one" or "the c++ community" not personal. It was not ad hominem. So no reason to be defensive about it. I am grateful if you can point out where my reasoning fails, then I learn something new. Maybe you could explain why a single occasional Branch Always over the unwind-pointer would be slow. Clearly the offset should be empirically based (so that you usually can avoid the goto), maybe even set to a separate cache line for some CPUs, and you could fill out the gaps with other data you need there. It's not like I have run i7 on Vtune, so I could be wrong, but I don't see why… And I also think that if you have a CPU with sufficient number of callee save registers you can carry along a pointer to the last try-block stack frame with not much penalty. After all you only have to restore it if the function ruined it and before calling new functions that are not inlined and not nothrow, and you could stick it into a thread local global too where it matters. On 32 bit x86 it probably is quite expensive though. In code where I write try blocks they tend to stay in the "main logic function", this cosde is so heavy that adding the stack frame to a linked list (of stack frames) is a neglectible cost One really need to be careful when doing performance tests of exception handling, because it is easy to construct "theoretical" code. Programmers should write exception handlers with the implementation in mind, so using existing programs as a base line is not a good solution either.
 I've sometimes been proven wrong here, so you're welcome to do 
 a pull request proving so.

You know very well that I am not going to rewrite codegen for DMD. Adding this feature will complicate codegen and you need to understand the code generator well to do the modification. Besides, I am not sure if a system level language should have exceptions at all or that I would use them when doing the kind of stuff I like to use D for. :-P ;-) I like to use exception handling in application-level code, but not in code for audio/simulations/buffer-streaming/low-level-stuff.
Feb 09 2014
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Sunday, 9 February 2014 at 05:00:15 UTC, Marco Leise wrote:
 And static allocation isn't an exactly appealing option...

   throw staticException ? staticException : (staticException =
   new SomethingException("Don't do this at home kids!"));

 and practically out of question when you need to chain
 exceptions and your call stack could contain this line of code
 more than once, resulting in infinite loops in exception
 chains as a new bug type in D, that is fixed by writing:

   catch (Exception e) {
       throw (staticException ? (e.linksTo(staticException) ? 
 staticException.dupThenWrap(e) : staticException) : 
 (staticException = new SomethingException("Don't do this at 
 home kids!"));
   }

This doesn't seem like a valid concern. Nothing stops you from using a (standard) function to do that ugly boilerplate.
Feb 09 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 8 February 2014 at 11:17:26 UTC, Jakob Ovrum wrote:
 On Saturday, 8 February 2014 at 11:05:38 UTC, Dmitry Olshansky 
 wrote:>
 If both are thread-local and cached I see no problem 
 whatsoever.
 The thing is the current "default" of creating exception is 
 AWFUL.
 And D stands for sane defaults and the simple path being good 
 last time I checked.

How is it not a problem? XException's fields (message, location etc) would be overwritten by the latest throw site, and its `next` field would point to itself.

It's supposedly one exception instance per place where it can be thrown, not per exception type. Then the problem would be restricted to recursive calls, where in the exception handler for XException, another XException is thrown.
Feb 09 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 8 February 2014 at 21:59:24 UTC, Walter Bright wrote:
 On 2/7/2014 6:50 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 The specific problem was that it was possible to provoke hash 
 collisions by
 sending carefully crafted input, causing the hash-tables to 
 degrade to linked
 lists. The small performance penalty of using 
 collision-resistant hashes is
 certainly worth it in this case.

That has nothing to do with needing exceptions in the control flow path (and the performance penalty for using exceptions in this manner is certainly not small).

Huh? I responded to this discussion: On Friday, 7 February 2014 at 08:30:35 UTC, Walter Bright wrote:
 On 2/6/2014 7:08 PM, bearophile wrote:
 That's why some languages have changed their sorting and 
 hashing routines to
 make them a little slower but safer on default.

DoS attack resistance requires faster code, not slower code.

I was merely clarifying why in this specific case making the average code path slower _did_ help DoS attack resistance.
Feb 09 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
This is a pretty nice description of the i7 pipeline by Hennesey 
and Patterson:

https://www.inkling.com/read/computer-architecture-hennessy-5th/chapter-3/section-3-13#0113e87a6dc141d7abda84b497128d61

Notice the 28 micro ops buffer before execution. I'd expect a 
short predicted branch to not cause a big bubble, but I don't 
know for sure.
Feb 09 2014
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Sunday, 9 February 2014 at 05:00:15 UTC, Marco Leise wrote:
 https://yourlogicalfallacyis.com/black-or-white

Off topic, but that is a fantastic web site. I wish I had known about it before.
Feb 09 2014
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 9 Feb 2014 22:24:21 +1100
schrieb "Daniel Murphy" <yebbliesnospam gmail.com>:

 "Dmitry Olshansky"  wrote in message news:ld7dla$pdg$1 digitalmars.com... 
 
 gedit does in fact throw an error message at you
 saying "My bad, it's broken UTF-8, I'm giving up!".

I know and it's piece of junk :) Seriously it doesn't even has regular expressions for search and replace!

That would be a luxury, gedit doesn't even have auto-indent.

You can talk about missing features in gedit all day, but from my point of view an editor is broken when it doesn't throw an error message at you. By silently replacing incorrect UTF-8 they change the original text. 0xFFFD should probably be used only when error messages are out of question like when displaying/printing text only. -- Marco
Feb 16 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Marco Leise"  wrote in message 
news:20140217030525.67a21dfc org.homedns.org...

 0xFFFD should probably be used only when error messages are
 out of question like when displaying/printing text only.

What do you use for displaying text, if not a text editor?
Feb 17 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 09 Feb 2014 12:18:41 +0400
schrieb Dmitry Olshansky <dmitry.olsh gmail.com>:

 09-Feb-2014 09:35, Marco Leise =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
 Thats neither an improvement over calling "validate" nor does
 that deal with distinguishing between invalid UTF and

Means text is broken but wasn't ever read...
\uFFFD
 in the input.

=20 Hardly makes any difference to the most applications. Normal text doesn't contain \uFFFD.

Of course it does. It is a valid symbol and a lot of websites describing the "Specials" Unicode block make use of it, like the one on Wikipedia: http://en.wikipedia.org/wiki/Specials_(Unicode_block) With your definition, pulling such a document from the web and parsing it in D would mean playing on broken strings.
 [...]
 Every single text editor out there seems to disagree with you: they do
 show you partially substituted text, not a dialog box "My bad, it's
 broken UTF-8, I'm giving up!".



 gedit does in fact throw an error message at you
 saying "My bad, it's broken UTF-8, I'm giving up!".


 I know and it's piece of junk :)
 Seriously it doesn't even has regular expressions for search and replace!

https://yourlogicalfallacyis.com/no-true-scotsman :p --=20 Marco
Feb 16 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 18 Feb 2014 01:01:53 +1100
schrieb "Daniel Murphy" <yebbliesnospam gmail.com>:

 "Marco Leise"  wrote in message 
 news:20140217030525.67a21dfc org.homedns.org...
 
 0xFFFD should probably be used only when error messages are
 out of question like when displaying/printing text only.

What do you use for displaying text, if not a text editor?

That was directed at D development. Or programming with Unicode encodings in general. If you load a text file and replace broken UTF-8 with \0xFFFD or ? as Sublime 3 does, you loose information. I think that smells and asks for a big red message box. gedit is an editor that works this way. What I meant by displaying text is static UI elements, since there is no risk of propagating the error. Everything else that can notify the user of the incorrect encoding or loss of information should do so. -- Marco
Feb 17 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/18/14, Dmitry Olshansky <dmitry.olsh gmail.com> wrote:
 Well, gedit is a nice example of why just throwing exception is not good
 enough for many apps (editors in particular). The fact that it's piece
 of junk might be irrelevant ;)

OT: Considering how many big-budget events (World Cup / Olympics) do such a poor job at displaying any kind of unicode text (e.g. they frequently display č/ć/đ ad c/c/dj), the only thing that could be worse is a big red dialog box, lol!
Feb 18 2014
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 18 Feb 2014 12:14:58 +0400
schrieb Dmitry Olshansky <dmitry.olsh gmail.com>:

 In a sense, \uFFFD means broken encoding.

In a sense yes, in another no. It is a defined code point and it has a symbol: =EF=BF=BD a diamond with a question mark inside.
 What about lone surrogates?

Those are actual broken encoding.
 Private use symbols that must not occur in transmission?

Then that "transmission" seems to exclude private symbols. It may also exclude special characters like \uFFFD. That's part of the particular protocol and should be handled there.
 They all=20
 displayed in various Unicode listings. About 'playing on broken strings'=

 - ignoring broken/partially broken strings, I specifically think that=20
 it's what most users/use cases want.
=20
 A more useful and sensible default of decoding is to substitute on=20
 broken encoding. And it's a standard procedure. It's particularly better=

 for displaying text.

Correct. I just don't agree that displaying text should the the one true use case and instead prefer exceptions instead of silent loss of information as the default.
 To remind: since it's only a decode you are still in the control of=20
 original text - in fact you may re-test what bytes are there IF you want.
=20
 The way of "throw on bad encoding" could be useful but I hardly see it=20
 as what you want for default.
=20
 I'm wary of breaking code that relies on throwing. For the moment I=20
 think the best course of action would be to introduce xdecode or some=20
 such that will do substitution on failure, see how it floats and then=20
 change ranges/foreach etc to use xdecode.

We wont convince each other. Let's just stop here. --=20 Marco
Feb 18 2014