www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A serious security bug... caused by no bounds checking.

reply "w0rp" <devw0rp gmail.com> writes:
http://heartbleed.com/

This bug has been getting around. The bug was caused by missing 
bounds checking.

I'm glad to be using a language with bounds checking.
Apr 07 2014
next sibling parent "Orvid King" <blah38621 gmail.com> writes:
On Mon, 07 Apr 2014 18:28:02 -0500, w0rp <devw0rp gmail.com> wrote:

 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds  
 checking.

 I'm glad to be using a language with bounds checking.

I thought the standard process (especially for such a massive security vulnerability) for these types of issues was to have a significant span of time between when the fix is publish and when the details of the vulnerability are released, yet from what I can see, they've published extensive details on the vulnerability on the exact same day that the fix was released. I really hope this isn't actually the case. (and more so, I hope none of the US news media who have any idea what it means get ahold of it, because it means that almost nobody in the US will not know about the issue, and believe me when I say, there are a LOT of people out there who would do a lot of harm with such a thing) From what I understand, depending on the exact configuration of the sever, namely who's address space OpenSSL was loaded in, it would be possible to rip database passwords from the server's memory. Servers that act merely as a proxy to the internal servers (the configuration that most large websites would have, which offloads the (de/en)cryption to gateway nodes) wouldn't have as big of an issue, but it would still be an issue.
Apr 07 2014
prev sibling next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 4/7/2014 7:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

Whelp, time for that server system upgrade I've been putting off for far too long...
Apr 07 2014
next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 4/8/2014 12:15 AM, H. S. Teoh wrote:
 I learned the hard way to always keep on top of the security upgrades. A
 year or two ago, I put off a pending upgrade for a week, and the day
 before I finally got around to it, my server was hacked via the same
 vulnerability that the upgrade would've fixed. They got root, so I had
 to nuke the system from orbit after backing up my data, and rebuild the
 server from scratch. :-( Ever since then, I've set up the system to
 notify me as soon as an update is available, and now I dare not delay to
 install it ASAP.

Yea, that's a good idea. Is that Arch? How does your querying for security updates work? Just querying for updates on security-related packages, or somehow filtering on whether a package's update is security-realted...or just a general "grab every update for everything"?
Apr 07 2014
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 4/8/2014 8:50 PM, Steven Schveighoffer wrote:
 On Mon, 07 Apr 2014 21:36:28 -0400, Nick Sabalausky
 <SeeWebsiteToContactMe semitwist.com> wrote:
 Whelp, time for that server system upgrade I've been putting off for
 far too long...

In theory, patching openSSL doesn't solve the problem, because someone could have previously used the vulnerability to get your private key. So technically you need to also get a new cert. This is what my password-generation vendor (lastpass.com) is recommending: 1. Generate a new password for your most critical sites. 2. But only after they get a cert dated after today! I don't think many people understand this aspect. Hopefully, this vulnerability was not known by hackers before it was announced. Even if it was, there is quite a window of opportunity for them as the patched sites roll out.

Very good point. Luckily for me (and yet, simultaneously embarrassing), my server's version of openssl turned out not to be affected. Which is nice since I *just* paid for a new cert about one week ago.
Apr 08 2014
prev sibling next sibling parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.
Apr 07 2014
next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 4/7/2014 9:59 PM, Ary Borenszweig wrote:
 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

I think it's potentially useful on a very careful per-module basis for certain modules specifically intended for no compiler-inserted bounds checking (or better yet, for specific blocks of code). But I certainly would never compile a whole program with it. That's just asking for trouble.
Apr 07 2014
parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig+dforum outerproduct.org> writes:
Am 08.04.2014 04:11, schrieb Nick Sabalausky:
 On 4/7/2014 9:59 PM, Ary Borenszweig wrote:
 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

I think it's potentially useful on a very careful per-module basis for certain modules specifically intended for no compiler-inserted bounds checking (or better yet, for specific blocks of code). But I certainly would never compile a whole program with it. That's just asking for trouble.

Sounds like adding a pragma instead would be a good idea. Of course that should make any affected code automatically system.
Apr 08 2014
prev sibling next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 4/7/2014 10:17 PM, Orvid King wrote:
 On Mon, 07 Apr 2014 20:59:50 -0500, Ary Borenszweig
 <ary esperanto.org.ar> wrote:

 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

The bad thing is, I have some code that having bounds checks enabled actually improves the speed of.

Not surprised, but I imagine it's likely only a handful of places where the bounds checking is actually slowing things down noticeably. If you sniffed those out with a profiler and had a good way to get around bounds checking for those specific cases, I'd bet you'd get nearly the same speedup without sacrificing much safety.
Apr 07 2014
parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 4/7/2014 10:50 PM, Nick Sabalausky wrote:
 On 4/7/2014 10:17 PM, Orvid King wrote:
 The bad thing is, I have some code that having bounds checks enabled
 actually improves the speed of.

Not surprised, but I imagine it's likely only a handful of places where the bounds checking is actually slowing things down noticeably.

Nevermind, I clearly misread your post :)
Apr 08 2014
prev sibling next sibling parent Ary Borenszweig <ary esperanto.org.ar> writes:
On 4/7/14, 11:17 PM, Orvid King wrote:
 On Mon, 07 Apr 2014 20:59:50 -0500, Ary Borenszweig
 <ary esperanto.org.ar> wrote:

 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

The bad thing is, I have some code that having bounds checks enabled actually improves the speed of.

Yes, it happened to me too (using another language). I was really surprised.
Apr 08 2014
prev sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 08.04.2014 16:57, schrieb Steven Schveighoffer:
 On Tue, 08 Apr 2014 04:50:29 -0400, Paulo Pinto <pjmlp progtools.org>
 wrote:

 On Tuesday, 8 April 2014 at 01:59:50 UTC, Ary Borenszweig wrote:
 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

Personally I think it should exist, but in a more controlled way, a compiler pragma. For example Turbo Pascal has something like {$R -} code {$R +}.

Note, you can disable bounds checking on an expression basis by replacing this: arr[x] with this: arr.ptr[x] The only tricky part is if you have to slice, and you are using $: arr.ptr[x..$] doesn't work, you have to use: arr.ptr[x..arr.length] -Steve

Is that only allowed in system? -- Paulo
Apr 08 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 08.04.2014 18:20, schrieb Steven Schveighoffer:
 On Tue, 08 Apr 2014 11:55:35 -0400, Paulo Pinto <pjmlp progtools.org>
 wrote:

 Am 08.04.2014 16:57, schrieb Steven Schveighoffer:
 Note, you can disable bounds checking on an expression basis by
 replacing this:

 arr[x]

 with this:

 arr.ptr[x]

 The only tricky part is if you have to slice, and you are using $:

 arr.ptr[x..$]

 doesn't work, you have to use:

 arr.ptr[x..arr.length]

Is that only allowed in system?

Probably. Is that an issue? Disabling bounds-checking in safe code is a bad idea. -Steve

Completely agree. I expressed myself badly. I think it should only be allowed in system code. -- Paulo
Apr 08 2014
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tuesday, 8 April 2014 at 01:36:29 UTC, Nick Sabalausky wrote:
 On 4/7/2014 7:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by 
 missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

Whelp, time for that server system upgrade I've been putting off for far too long...

Indeed. If you were wondering why the wiki/forum were down, that's why...
Apr 07 2014
prev sibling next sibling parent "Orvid King" <blah38621 gmail.com> writes:
On Mon, 07 Apr 2014 20:59:50 -0500, Ary Borenszweig <ary esperanto.org.ar>  
wrote:

 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

The bad thing is, I have some code that having bounds checks enabled actually improves the speed of.
Apr 07 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Apr 07, 2014 at 09:36:28PM -0400, Nick Sabalausky wrote:
 On 4/7/2014 7:28 PM, w0rp wrote:
http://heartbleed.com/

This bug has been getting around. The bug was caused by missing
bounds checking.

I'm glad to be using a language with bounds checking.

Whelp, time for that server system upgrade I've been putting off for far too long...

I learned the hard way to always keep on top of the security upgrades. A year or two ago, I put off a pending upgrade for a week, and the day before I finally got around to it, my server was hacked via the same vulnerability that the upgrade would've fixed. They got root, so I had to nuke the system from orbit after backing up my data, and rebuild the server from scratch. :-( Ever since then, I've set up the system to notify me as soon as an update is available, and now I dare not delay to install it ASAP. T -- It said to install Windows 2000 or better, so I installed Linux instead.
Apr 07 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Apr 08, 2014 at 12:43:15AM -0400, Nick Sabalausky wrote:
 On 4/8/2014 12:15 AM, H. S. Teoh wrote:
I learned the hard way to always keep on top of the security
upgrades. A year or two ago, I put off a pending upgrade for a week,
and the day before I finally got around to it, my server was hacked
via the same vulnerability that the upgrade would've fixed. They got
root, so I had to nuke the system from orbit after backing up my
data, and rebuild the server from scratch. :-( Ever since then, I've
set up the system to notify me as soon as an update is available, and
now I dare not delay to install it ASAP.

Yea, that's a good idea. Is that Arch? How does your querying for security updates work? Just querying for updates on security-related packages, or somehow filtering on whether a package's update is security-realted...or just a general "grab every update for everything"?

Actually, it's Debian/stable (which only gets security upgrades). I just installed cron-apt and set it up to email me about upgrades. In theory, if I were lazy, I'd set it up to just install all updates automatically, but I do like to review exactly what gets installed before installing it, since I did get bitten before by a careless upgrade breaking existing software in a major way. (The worst instance of this was when I unknowingly upgraded libc6 to a version that's incompatible with the VPS kernel, causing the dynamic linker (and thus *all* executables) to break. I had to resort to heavy-handed tactics[1] to fix it.) [1] Heavy-handed, as in: http://eusebeia.dyndns.org/bashcp T -- Жил-был король когда-то, при нём блоха жила.
Apr 07 2014
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Tuesday, 8 April 2014 at 01:59:50 UTC, Ary Borenszweig wrote:
 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by 
 missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

The flag is useful if the program works in a trusted environment like a developer compiling his own code on his own machine.
Apr 07 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Tuesday, 8 April 2014 at 01:59:50 UTC, Ary Borenszweig wrote:
 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by 
 missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

Personally, I would never turn off bounds checking, except for maybe a single-player video game or similar.
Apr 08 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 4/8/14, w0rp <devw0rp gmail.com> wrote:
 http://heartbleed.com/

Is there a link to a specific section of code where this bug is introduced? That page is massive and all I want to do is see the buggy code. :>
Apr 08 2014
prev sibling next sibling parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Monday, 7 April 2014 at 23:28:03 UTC, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing 
 bounds checking.

 I'm glad to be using a language with bounds checking.

I never got the point of not having bounds checking in C and its ilk. All the alternative system programming languages allow disabling them, if one really needs to do so. Which should only be done after profiling, proving it is the cause of the application not delivering the desired performance and only on the exact spot. Me too am glad see bounds checking in D by default. -- Paulo
Apr 08 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2014 1:47 AM, Paulo Pinto wrote:
 I never got the point of not having bounds checking in C and its ilk.

C hardly even has arrays.
Apr 08 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2014 3:55 AM, Paulo Pinto wrote:
 On Tuesday, 8 April 2014 at 09:46:51 UTC, Walter Bright wrote:
 On 4/8/2014 1:47 AM, Paulo Pinto wrote:
 I never got the point of not having bounds checking in C and its ilk.

C hardly even has arrays.

Yes I know, another broken design decision.

Looking at C's decisions from our perspective is a bit unfair. The only really unforgivable one, from the perspective of the times when it was designed, is the one where arrays decay to pointers when passing them to functions. This completely defeats any attempt at detecting array overflows.
Apr 08 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 08.04.2014 19:18, schrieb Walter Bright:
 On 4/8/2014 3:55 AM, Paulo Pinto wrote:
 On Tuesday, 8 April 2014 at 09:46:51 UTC, Walter Bright wrote:
 On 4/8/2014 1:47 AM, Paulo Pinto wrote:
 I never got the point of not having bounds checking in C and its ilk.

C hardly even has arrays.

Yes I know, another broken design decision.

Looking at C's decisions from our perspective is a bit unfair. The only really unforgivable one, from the perspective of the times when it was designed, is the one where arrays decay to pointers when passing them to functions. This completely defeats any attempt at detecting array overflows.

I don't consider unfair, because there were systems languages at the time like PL/I and Mesa that had bounds checking. C designers explicitly decided against it, with the thought that developers would use lint alongside C, which even today very few do. -- Paulo
Apr 08 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2014 10:44 AM, Paulo Pinto wrote:
 C designers explicitly decided against it, with the thought that developers
 would use lint alongside C, which even today very few do.

The trouble is that C cannot be retrofitted with bounds checking because of the array decay problem.
Apr 08 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 08.04.2014 20:28, schrieb Walter Bright:
 On 4/8/2014 10:44 AM, Paulo Pinto wrote:
 C designers explicitly decided against it, with the thought that
 developers
 would use lint alongside C, which even today very few do.

The trouble is that C cannot be retrofitted with bounds checking because of the array decay problem.

Fully agree with you. That is why when I used to code in C at the university and my first job, I made use of warnings as errors, asserts and did a lot of defensive coding. Having read Code Complete made me realize how I could make C more Pascal/Modula-2 like in terms of safety. But you are right. -- Paulo
Apr 08 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2014 12:26 PM, Paulo Pinto wrote:
 But you are right.

Words to live by :-)
Apr 08 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/8/14, 1:47 AM, Paulo Pinto wrote:
 Me too am glad see bounds checking in D by default.

For the record, dmd used to remove bounds checking in -release mode. I've asked Walter to add a new flag for that, independent from -release, thus keeping release builds safer. It was the first and last time when I used "if you don't do this, I can't work on D anymore" card. True story. Andrei
Apr 08 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/8/14, 1:07 PM, Martin Krejcirik wrote:
 On Tuesday, 8 April 2014 at 19:47:02 UTC, Andrei Alexandrescu wrote:
 For the record, dmd used to remove bounds checking in -release mode.
 I've asked Walter to add a new flag for that, independent

It still does (for un- safe functions) and there is no way to turn it on (if you don't want asserts and contracts).

:o| We need to fix that. Andrei
Apr 08 2014
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 04/10/2014 07:37 PM, Steven Schveighoffer wrote:

That's why we have trusted. safe is a special situation, it's not made for optimization, and should be immune to those attempts in deference to safety. -Steve

safe will often be inferred.
Apr 10 2014
prev sibling next sibling parent Martin Krejcirik <mk-junk i-line.cz> writes:
On 10.4.2014 19:12, Steven Schveighoffer wrote:
 void foo(T)(T[] x)  safe
 {
    x[5] = 3;
 }

Is this common practice ? I'd wouldn't call it a safe design. There should be a length check or version check: version(D_NoBoundsChecks) static assert(0, "bounds checking required"); But I get your point, I have always thought of bounds checking like an optional safety net, you think of it like a required feature. -- mk
Apr 10 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/10/14, 10:56 AM, Steven Schveighoffer wrote:
  safe code can be marked as  trusted instead, and nothing changes,
 except  trusted code can have bounds checks removed. How does this not
 work as a solution?

Doesn't work because trusted removes all additional checks by the compiler. We do want to have safe as mechanically verified.
 As Walter often says about logical const, logical  safe is  safe by
 convention, and it loses all of its teeth.

I think you are wrong here. Andrei
Apr 10 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/10/14, 5:52 PM, Steven Schveighoffer wrote:
 If  safe is just a convention, then I don't see the point of having it
 at all.

That I agree with. -- Andrei
Apr 10 2014
prev sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Steven Schveighoffer"  wrote in message 
news:op.xd3vzecweav7ka stevens-macbook-pro.local...

 No, the author of the  safe code expects bounds checking, it's part of the 
 requirements. To compile his code with it off is like having 
   -compilergeneratedhash switch that overrides any toHash functions with a 
 compiler generated one. You are changing the agreement between the 
 compiler and the code. When I say  safe, I mean "I absolutely always want 
 bounds checks."

If you have code that would ever fail a bounds check, that is a program error, similar to code that may fail an assertion. And like assertions, if you would rather the code was as fast as possible instead of as safe as possible you can use a compiler switch to disable bound checks. The usual switch to do stuff like this is '-release', but because safe functions should still have the 'no memory corruption' even in release mode, disabling those bounds checks was moved into another compiler switch.
 If you want to eliminate bounds checks, use  trusted.

No, trusted means "don't check my code" while safe + noboundschecks means (mostly) "only check my code at compile-time".
Apr 11 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Steven Schveighoffer"  wrote in message 
news:op.xd5lomc1eav7ka stevens-macbook-pro.local...

 Here is the horror scenario I envision:

 1. Company has 100kLOC project, which is marked as  safe (I can dream, 
 can't I?)
 2. They find that performance is lacking, maybe compared to a competitor's 
 C++ based code.
 3. They try compiling with -noboundscheck, get a large performance boost. 
 It really only makes a difference in one function (the inner loop one).
 4. They pat themselves on the back, and release with the new flag, 
 destroying all bounds checks, even bounds checks in library template code 
 that they didn't write or scrutinize.
 5. Buffer overflow attacks abound.
 6. D  safe is labeled a "joke"

Trying to prevent developer stupidity is a lost cause. Bounds checks are on by default. They are even on when you ask for 'fast-over-safe' aka -release. They get turned off when you explicitly ask for it.
 But there is a cost, even to labeling the "one inner" function  trusted. 
 Perhaps that function is extremely long and complex. There should be a way 
 to say, "I still want all the  safety checks, except for this one critical 
 array access, I have manually guaranteed the bounds". We don't have 
 anything like that. All other safety checks are really static, this is the 
 only runtime penalty for safety.

Something like (() trusted => arr.ptr[index]) should do the trick.
 The blunt flag approach is scary.  trusted is better, in that you can 
 focus on one function at a time. But I think we need something more 
 precise. Perhaps you should be able to have  trusted scopes, or  trusted 
 expressions.

trusted delegates get you 99.99% of the way there.
Apr 11 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Meta"  wrote in message news:brrbqfpkfkevxbseoxiq forum.dlang.org...

 Hasn't there been a proposal before to allow  system/ trusted/ safe 
 blocks, allowing it to be a bit more granular than at the function level? 
 Maybe:

  trusted
 {
      arr.ptr[index]
 }

 Could be lowered to (()  trusted => arr.ptr[index]).

It's a little prettier, but less powerful and not really something that _should_ be pretty.
Apr 11 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 8 April 2014 at 01:59:50 UTC, Ary Borenszweig wrote:
 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by 
 missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

Personally I think it should exist, but in a more controlled way, a compiler pragma. For example Turbo Pascal has something like {$R -} code {$R +}. -- Paulo
Apr 08 2014
prev sibling next sibling parent reply "Martin Krejcirik" <mk-junk i-line.cz> writes:
 I'm glad to be using a language with bounds checking.

That brings a question about dmd -release flag. I'd like a flag to turn off asserts and contracts, but leave enabled all bounds checking. IMHO that is not possible currently ?
Apr 08 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2014 2:12 AM, Martin Krejcirik wrote:
 That brings a question about dmd -release flag. I'd like a flag
 to turn off asserts and contracts, but leave enabled all bounds
 checking. IMHO that is not possible currently ?

-release leaves bounds checking on. Use -noboundscheck to turn off bounds checking.
Apr 08 2014
prev sibling next sibling parent "Martin Krejcirik" <mk-junk i-line.cz> writes:
 -release leaves bounds checking on. Use -noboundscheck to turn 
 off bounds checking.

But only for safe functions. I want it on for all functions.
Apr 08 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 8 April 2014 at 09:46:51 UTC, Walter Bright wrote:
 On 4/8/2014 1:47 AM, Paulo Pinto wrote:
 I never got the point of not having bounds checking in C and 
 its ilk.

C hardly even has arrays.

Yes I know, another broken design decision. In regards to Turbo Pascal and successors, there is hardly any C design decision I agree with. -- Paulo
Apr 08 2014
prev sibling next sibling parent "marwy" <mariusz.wyrozumski gmail.com> writes:
On Tuesday, 8 April 2014 at 08:22:09 UTC, Andrej Mitrovic wrote:
 On 4/8/14, w0rp <devw0rp gmail.com> wrote:
 http://heartbleed.com/

Is there a link to a specific section of code where this bug is introduced? That page is massive and all I want to do is see the buggy code. :>

http://blog.existentialize.com/diagnosis-of-the-openssl-heartbleed-bug.html Hope that helps :-).
Apr 08 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 08 Apr 2014 04:50:29 -0400, Paulo Pinto <pjmlp progtools.org>  
wrote:

 On Tuesday, 8 April 2014 at 01:59:50 UTC, Ary Borenszweig wrote:
 On 4/7/14, 8:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

http://www.reddit.com/r/programming/comments/21m0bz/warp_a_fast_c_and_c_preprocessor/cged2y6 I think that flag shouldn't exist.

Personally I think it should exist, but in a more controlled way, a compiler pragma. For example Turbo Pascal has something like {$R -} code {$R +}.

Note, you can disable bounds checking on an expression basis by replacing this: arr[x] with this: arr.ptr[x] The only tricky part is if you have to slice, and you are using $: arr.ptr[x..$] doesn't work, you have to use: arr.ptr[x..arr.length] -Steve
Apr 08 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Apr 08, 2014 at 02:46:57AM -0700, Walter Bright wrote:
 On 4/8/2014 1:47 AM, Paulo Pinto wrote:
I never got the point of not having bounds checking in C and its ilk.

C hardly even has arrays.

And just yesterday, I caught yet another long-standing off-by-1 array overrun bug in the C code I was working on at work, that obviously nobody else noticed. Sigh... T -- War doesn't prove who's right, just who's left. -- BSD Games' Fortune
Apr 08 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 08 Apr 2014 11:55:35 -0400, Paulo Pinto <pjmlp progtools.org>  
wrote:

 Am 08.04.2014 16:57, schrieb Steven Schveighoffer:
 Note, you can disable bounds checking on an expression basis by
 replacing this:

 arr[x]

 with this:

 arr.ptr[x]

 The only tricky part is if you have to slice, and you are using $:

 arr.ptr[x..$]

 doesn't work, you have to use:

 arr.ptr[x..arr.length]

Is that only allowed in system?

Probably. Is that an issue? Disabling bounds-checking in safe code is a bad idea. -Steve
Apr 08 2014
prev sibling next sibling parent "Martin Krejcirik" <mk-junk i-line.cz> writes:
On Tuesday, 8 April 2014 at 19:47:02 UTC, Andrei Alexandrescu 
wrote:
 For the record, dmd used to remove bounds checking in -release 
 mode. I've asked Walter to add a new flag for that, independent

It still does (for un- safe functions) and there is no way to turn it on (if you don't want asserts and contracts).
Apr 08 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 8 April 2014 at 19:47:02 UTC, Andrei Alexandrescu 
wrote:
 On 4/8/14, 1:47 AM, Paulo Pinto wrote:
 Me too am glad see bounds checking in D by default.

For the record, dmd used to remove bounds checking in -release mode. I've asked Walter to add a new flag for that, independent from -release, thus keeping release builds safer. It was the first and last time when I used "if you don't do this, I can't work on D anymore" card. True story. Andrei

There is a lot of confusion about this so I looked into it. dmd still removes bound checking in non- safe code when you specify -release. -noboundscheck just causes it to remove bounds checking in safe code too. This is why I think it should be renamed -nosafeboundschecking or something similar. In dmd's source, useArrayBounds is set to 2 by default (which does bounds checking in all code). -release sets it to 1 (which only checks safe code). -noboundscheck sets it to 0 (which causes it to emit no bounds checking).
Apr 08 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 8 April 2014 at 20:07:30 UTC, Martin Krejcirik wrote:
 On Tuesday, 8 April 2014 at 19:47:02 UTC, Andrei Alexandrescu 
 wrote:
 For the record, dmd used to remove bounds checking in -release 
 mode. I've asked Walter to add a new flag for that, independent

It still does (for un- safe functions)

I think -noboundscheck should be renamed -nosafeboundscheck because it's just confusing with the name it has. I just did a pull request yesterday to help clarify what it does in the command line help, at least, because a lot of people are confused about this.
 and there is no way to turn it on (if you don't want asserts and
 contracts).

Good point. I think perhaps a -boundscheck is in order if the -release behavior is going to stay what it is. It's a shame that the flag already means what it does because we can't just have -noboundscheck remove them in non- safe code and a new -nosafeboundcheck do it for safe code now.
Apr 08 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 08 Apr 2014 16:07:53 -0400, Brad Anderson <eco gnuk.net> wrote:

 On Tuesday, 8 April 2014 at 19:47:02 UTC, Andrei Alexandrescu wrote:
 On 4/8/14, 1:47 AM, Paulo Pinto wrote:
 Me too am glad see bounds checking in D by default.

For the record, dmd used to remove bounds checking in -release mode. I've asked Walter to add a new flag for that, independent from -release, thus keeping release builds safer. It was the first and last time when I used "if you don't do this, I can't work on D anymore" card. True story. Andrei

There is a lot of confusion about this so I looked into it. dmd still removes bound checking in non- safe code when you specify -release. -noboundscheck just causes it to remove bounds checking in safe code too. This is why I think it should be renamed -nosafeboundschecking or something similar.

This does not sound correct. In NO case should you be able to remove bounds checking in safe code. -Steve
Apr 08 2014
prev sibling next sibling parent "Martin Krejcirik" <mk-junk i-line.cz> writes:
On Tuesday, 8 April 2014 at 20:20:59 UTC, Brad Anderson wrote:
 Good point. I think perhaps a -boundscheck is in order if the

What about: -boundscheck=<none|safe|all> which defaults to 'all' and to 'safe' in combination with -release -noboundscheck would be the same as -boundscheck=none If compatibility is not an issue and we want to keep it simple, than -release should just turn off asserts and contracts.
Apr 08 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 8 April 2014 at 20:50:35 UTC, Steven Schveighoffer 
wrote:
 On Tue, 08 Apr 2014 16:07:53 -0400, Brad Anderson 
 <eco gnuk.net> wrote:

 On Tuesday, 8 April 2014 at 19:47:02 UTC, Andrei Alexandrescu 
 wrote:
 On 4/8/14, 1:47 AM, Paulo Pinto wrote:
 Me too am glad see bounds checking in D by default.

For the record, dmd used to remove bounds checking in -release mode. I've asked Walter to add a new flag for that, independent from -release, thus keeping release builds safer. It was the first and last time when I used "if you don't do this, I can't work on D anymore" card. True story. Andrei

There is a lot of confusion about this so I looked into it. dmd still removes bound checking in non- safe code when you specify -release. -noboundscheck just causes it to remove bounds checking in safe code too. This is why I think it should be renamed -nosafeboundschecking or something similar.

This does not sound correct. In NO case should you be able to remove bounds checking in safe code. -Steve

Then we have a bug because that's how it works currently. https://github.com/D-Programming-Language/dmd/blob/a3743bc645fc065104470cdecbd64e3f14034fdf/src/irstate.c#L193 Reminder of the magic numbers' meanings: - 2 is the default[1] - 1 if -release is specified[2] - 0 if -noboundscheck is specified[3]. 1. https://github.com/D-Programming-Language/dmd/blob/404bbbd1543b045d32166a4462b4bf1f271fbe7c/src/mars.c#L567 2. https://github.com/D-Programming-Language/dmd/blob/404bbbd1543b045d32166a4462b4bf1f271fbe7c/src/mars.c#L1144 3. https://github.com/D-Programming-Language/dmd/blob/404bbbd1543b045d32166a4462b4bf1f271fbe7c/src/mars.c#L1148
Apr 08 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 8 April 2014 at 21:01:26 UTC, Martin Krejcirik wrote:
 On Tuesday, 8 April 2014 at 20:20:59 UTC, Brad Anderson wrote:
 Good point. I think perhaps a -boundscheck is in order if the

What about: -boundscheck=<none|safe|all> which defaults to 'all' and to 'safe' in combination with -release -noboundscheck would be the same as -boundscheck=none

Asked Andrei in IRC about it and he liked -boundscheck=. https://d.puremagic.com/issues/show_bug.cgi?id=12550 I'll try to make a pull request if someone else doesn't beat me to it.
Apr 08 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 07 Apr 2014 21:36:28 -0400, Nick Sabalausky  
<SeeWebsiteToContactMe semitwist.com> wrote:

 On 4/7/2014 7:28 PM, w0rp wrote:
 http://heartbleed.com/

 This bug has been getting around. The bug was caused by missing bounds
 checking.

 I'm glad to be using a language with bounds checking.

Whelp, time for that server system upgrade I've been putting off for far too long...

In theory, patching openSSL doesn't solve the problem, because someone could have previously used the vulnerability to get your private key. So technically you need to also get a new cert. This is what my password-generation vendor (lastpass.com) is recommending: 1. Generate a new password for your most critical sites. 2. But only after they get a cert dated after today! I don't think many people understand this aspect. Hopefully, this vulnerability was not known by hackers before it was announced. Even if it was, there is quite a window of opportunity for them as the patched sites roll out. -Steve
Apr 08 2014
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 07 Apr 2014 23:28:02 +0000
schrieb "w0rp" <devw0rp gmail.com>:

 http://heartbleed.com/
 
 This bug has been getting around. The bug was caused by missing 
 bounds checking.
 
 I'm glad to be using a language with bounds checking.

Sorry, but wasn't this security risk instead caused by uninitialized memory, and shouldn't you instead have said: "I'm glad to be using a language with default initialization?" (The attacker could request a larger packet size than required for the requested data and malloc() doesn't zero out the rest of the memory block, possibly containing sensitive data.) In an article I read that malloc implementations using mmap are more seriously affected, since that memory can come from anywhere whereas the sbrk version is usually reusing memory from the calling process. I didn't understand that, since mmap on Linux returns zeroed out pages. Process isolation is actually pretty good. And frankly, had the OpenSSH developers replaced the m with c in malloc, they could have avoided a lot of trouble as well. But as it goes with such low level libraries performance is very important to put load off servers for what is today seen as a normal part of the network stack. I don't think any of these calls for "safer languages" will change that the developers want to win in those throughput benchmarks with their implementation and keep using unsafe functions. -- Marco
Apr 09 2014
parent =?UTF-8?Q?Tobias=20M=C3=BCller?= <troplin bluewin.ch> writes:
Marco Leise <Marco.Leise gmx.de> wrote:
 Am Mon, 07 Apr 2014 23:28:02 +0000
 schrieb "w0rp" <devw0rp gmail.com>:
 
 http://heartbleed.com/
 
 This bug has been getting around. The bug was caused by missing 
 bounds checking.
 
 I'm glad to be using a language with bounds checking.

Sorry, but wasn't this security risk instead caused by uninitialized memory, and shouldn't you instead have said: "I'm glad to be using a language with default initialization?" (The attacker could request a larger packet size than required for the requested data and malloc() doesn't zero out the rest of the memory block, possibly containing sensitive data.)

As far as I understand it, you can read up to 64 KB of data, much more than the typical 4 KB block size. That means that you can read adjacent memory blocks that possibly contain perfectly valid data. Tobi
Apr 09 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 8 April 2014 at 21:52:56 UTC, Brad Anderson wrote:
 On Tuesday, 8 April 2014 at 21:01:26 UTC, Martin Krejcirik 
 wrote:
 On Tuesday, 8 April 2014 at 20:20:59 UTC, Brad Anderson wrote:
 Good point. I think perhaps a -boundscheck is in order if the

What about: -boundscheck=<none|safe|all> which defaults to 'all' and to 'safe' in combination with -release -noboundscheck would be the same as -boundscheck=none

Asked Andrei in IRC about it and he liked -boundscheck=. https://d.puremagic.com/issues/show_bug.cgi?id=12550 I'll try to make a pull request if someone else doesn't beat me to it.

https://github.com/D-Programming-Language/dmd/pull/3443
Apr 09 2014
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 8 April 2014 at 20:50:35 UTC, Steven Schveighoffer 
wrote:
 This does not sound correct. In NO case should you be able to 
 remove bounds checking in  safe code.

It is. In fact, that's the very reason why DMD has -noboundscheck in addition to -release. David
Apr 09 2014
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 8 April 2014 at 21:23:35 UTC, Andrei Alexandrescu 
wrote:
 On 4/8/14, 1:07 PM, Martin Krejcirik wrote:
 On Tuesday, 8 April 2014 at 19:47:02 UTC, Andrei Alexandrescu 
 wrote:
 For the record, dmd used to remove bounds checking in 
 -release mode.
 I've asked Walter to add a new flag for that, independent

It still does (for un- safe functions) and there is no way to turn it on (if you don't want asserts and contracts).

:o| We need to fix that.

Just for the record, LDC has -enable-boundscheck and -disable-boundscheck (in line with all the other flags for enabling-disabling certain features). If neither is specified, the DMD default (-release disables checks in non- safe code) is used. David
Apr 09 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Wednesday, 9 April 2014 at 12:36:49 UTC, Marco Leise wrote:
 Sorry, but wasn't this security risk instead caused by
 uninitialized memory, and shouldn't you instead have said:

 "I'm glad to be using a language with default initialization?"

Nope, it was caused by missing bounds checking. https://www.openssl.org/news/secadv_20140407.txt
 A missing bounds check [...]

Apr 09 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 10 Apr 2014 06:51:40 +0000
schrieb "w0rp" <devw0rp gmail.com>:

 On Wednesday, 9 April 2014 at 12:36:49 UTC, Marco Leise wrote:
 Sorry, but wasn't this security risk instead caused by
 uninitialized memory, and shouldn't you instead have said:

 "I'm glad to be using a language with default initialization?"

Nope, it was caused by missing bounds checking. https://www.openssl.org/news/secadv_20140407.txt
 A missing bounds check [...]


Haha, I tried to read that about an hour ago to inform myself, but it still doesn't load for me. -- Marco
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 09 Apr 2014 13:35:43 -0400, David Nadlinger <code klickverbot.at>  
wrote:

 On Tuesday, 8 April 2014 at 20:50:35 UTC, Steven Schveighoffer wrote:
 This does not sound correct. In NO case should you be able to remove  
 bounds checking in  safe code.

It is. In fact, that's the very reason why DMD has -noboundscheck in addition to -release.

I meant correct as in not wrong, not correct as in the current state of the compiler :) Otherwise, safe is just another meaningless convention. Walter? -Steve
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Thursday, 10 April 2014 at 14:13:30 UTC, Steven Schveighoffer 
wrote:
 On Wed, 09 Apr 2014 13:35:43 -0400, David Nadlinger 
 <code klickverbot.at> wrote:

 On Tuesday, 8 April 2014 at 20:50:35 UTC, Steven Schveighoffer 
 wrote:
 This does not sound correct. In NO case should you be able to 
 remove bounds checking in  safe code.

It is. In fact, that's the very reason why DMD has -noboundscheck in addition to -release.

I meant correct as in not wrong, not correct as in the current state of the compiler :) Otherwise, safe is just another meaningless convention. Walter? -Steve

It's funny because just the other day I tried argue on Rust mailing list why -noboundscheck flag should be added to the Rust compiler. My argument didn't go down very well. But my point was that someone at some point might have a genuine need for that flag, and that having the option to compile the code to an unsafe program doesn't make the language itself any less safe. safe guarantees memory-safety given that any trusted code used doesn't break its promise and that you don't use the -noboundscheck flag. That doesn't sound like a convention to me.
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Thursday, 10 April 2014 at 14:13:30 UTC, Steven Schveighoffer
wrote:
 On Wed, 09 Apr 2014 13:35:43 -0400, David Nadlinger 
 <code klickverbot.at> wrote:

 On Tuesday, 8 April 2014 at 20:50:35 UTC, Steven Schveighoffer 
 wrote:
 This does not sound correct. In NO case should you be able to 
 remove bounds checking in  safe code.

It is. In fact, that's the very reason why DMD has -noboundscheck in addition to -release.

I meant correct as in not wrong, not correct as in the current state of the compiler :) Otherwise, safe is just another meaningless convention. Walter? -Steve

It's funny because just the other day I tried argue on Rust mailing list why -noboundscheck flag should be added to the Rust compiler. My argument didn't go down very well. But my point was that someone at some point might have a genuine need for that flag, and that having the option to compile the code to an unsafe program doesn't make the language itself any less safe. safe guarantees memory-safety given that any trusted code used doesn't break its promise and that you don't use the -noboundscheck flag. That doesn't sound like a convention to me.
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 10:55:26 -0400, Tommi <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 14:13:30 UTC, Steven Schveighoffer wrote:
 On Wed, 09 Apr 2014 13:35:43 -0400, David Nadlinger  
 <code klickverbot.at> wrote:

 On Tuesday, 8 April 2014 at 20:50:35 UTC, Steven Schveighoffer wrote:
 This does not sound correct. In NO case should you be able to remove  
 bounds checking in  safe code.

It is. In fact, that's the very reason why DMD has -noboundscheck in addition to -release.

I meant correct as in not wrong, not correct as in the current state of the compiler :) Otherwise, safe is just another meaningless convention. Walter? -Steve

It's funny because just the other day I tried argue on Rust mailing list why -noboundscheck flag should be added to the Rust compiler. My argument didn't go down very well. But my point was that someone at some point might have a genuine need for that flag, and that having the option to compile the code to an unsafe program doesn't make the language itself any less safe. safe guarantees memory-safety given that any trusted code used doesn't break its promise and that you don't use the -noboundscheck flag. That doesn't sound like a convention to me.

No, the author of the safe code expects bounds checking, it's part of the requirements. To compile his code with it off is like having a -compilergeneratedhash switch that overrides any toHash functions with a compiler generated one. You are changing the agreement between the compiler and the code. When I say safe, I mean "I absolutely always want bounds checks." If you want to eliminate bounds checks, use trusted. -Steve
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Thursday, 10 April 2014 at 15:00:34 UTC, Steven Schveighoffer 
wrote:
 On Thu, 10 Apr 2014 10:55:26 -0400, Tommi 
 <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 14:13:30 UTC, Steven 
 Schveighoffer wrote:
 On Wed, 09 Apr 2014 13:35:43 -0400, David Nadlinger 
 <code klickverbot.at> wrote:

 On Tuesday, 8 April 2014 at 20:50:35 UTC, Steven 
 Schveighoffer wrote:
 This does not sound correct. In NO case should you be able 
 to remove bounds checking in  safe code.

It is. In fact, that's the very reason why DMD has -noboundscheck in addition to -release.

I meant correct as in not wrong, not correct as in the current state of the compiler :) Otherwise, safe is just another meaningless convention. Walter? -Steve

It's funny because just the other day I tried argue on Rust mailing list why -noboundscheck flag should be added to the Rust compiler. My argument didn't go down very well. But my point was that someone at some point might have a genuine need for that flag, and that having the option to compile the code to an unsafe program doesn't make the language itself any less safe. safe guarantees memory-safety given that any trusted code used doesn't break its promise and that you don't use the -noboundscheck flag. That doesn't sound like a convention to me.

No, the author of the safe code expects bounds checking, it's part of the requirements. To compile his code with it off is like having a -compilergeneratedhash switch that overrides any toHash functions with a compiler generated one. You are changing the agreement between the compiler and the code.

Obviously if such or any other compiler flags exist, their existence and behaviour has been specified in binding agreement between the compiler and the source code, and thus, no breach of contract has happened if such compiler flags were used.
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 12:49:26 -0400, Tommi <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 15:00:34 UTC, Steven Schveighoffer wrote:

 No, the author of the  safe code expects bounds checking, it's part of  
 the requirements. To compile his code with it off is like having a  
 -compilergeneratedhash switch that overrides any toHash functions with  
 a compiler generated one. You are changing the agreement between the  
 compiler and the code.

Obviously if such or any other compiler flags exist, their existence and behaviour has been specified in binding agreement between the compiler and the source code, and thus, no breach of contract has happened if such compiler flags were used.

A compiler flag is a blunt instrument. It affects all code the compiler touches, which may or may not affect code that you are intending to change. For example: // compiled without -noboundscheck module compiledlib; void foo(T)(T[] x) safe { x[5] = 3; } ... // compiled with -noboundscheck main() safe { foo([1,2,3]); // memory now corrupted, no warning, no runtime error. } -Steve
Apr 10 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Steven Schveighoffer:

 No, the author of the  safe code expects bounds checking, it's 
 part of the requirements.

Take a look ad Ada language. It has bounds checking and its compilers have a switch to disable those checks. If you want the bounds checking don't use the switch that disables the bounds checking. Safety doesn't mean to have no way to work around safety locks. It means have nice handy locks that are active on default. In a system language total safety is an illusion. Better to focus on real world safety and not a illusion of theoretical safety. Bye, bearophile
Apr 10 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Thursday, 10 April 2014 at 17:25:26 UTC, bearophile wrote:
 Steven Schveighoffer:

 No, the author of the  safe code expects bounds checking, it's 
 part of the requirements.

Take a look ad Ada language. It has bounds checking and its compilers have a switch to disable those checks. If you want the bounds checking don't use the switch that disables the bounds checking. Safety doesn't mean to have no way to work around safety locks. It means have nice handy locks that are active on default. In a system language total safety is an illusion. Better to focus on real world safety and not a illusion of theoretical safety. Bye, bearophile

Yeah, it's like how I have matches in my house. I could use the matches to burn my house down, but I don't think that I will. That is, unless I can manage to pull off a really good insurance fraud scam.
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
 A compiler flag is a blunt instrument. It affects all code the 
 compiler touches, which may or may not affect code that you are 
 intending to change.

Yes, such a compiler flag is a blunt and dangerous instrument and everybody should stay away from it. But everybody agrees on those points already. That's _not_ what you need to prove to show that such a flag shouldn't exist. What you need to show is that no-one will ever find them-self in a situation where such a blunt instrument would be useful.
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 13:25:25 -0400, bearophile <bearophileHUGS lycos.com>  
wrote:

 Steven Schveighoffer:

 No, the author of the  safe code expects bounds checking, it's part of  
 the requirements.

Take a look ad Ada language. It has bounds checking and its compilers have a switch to disable those checks. If you want the bounds checking don't use the switch that disables the bounds checking. Safety doesn't mean to have no way to work around safety locks. It means have nice handy locks that are active on default. In a system language total safety is an illusion. Better to focus on real world safety and not a illusion of theoretical safety.

That's why we have trusted. safe is a special situation, it's not made for optimization, and should be immune to those attempts in deference to safety. -Steve
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 13:35:34 -0400, Tommi <tommitissari hotmail.com> wrote:

 A compiler flag is a blunt instrument. It affects all code the compiler  
 touches, which may or may not affect code that you are intending to  
 change.

Yes, such a compiler flag is a blunt and dangerous instrument and everybody should stay away from it. But everybody agrees on those points already. That's _not_ what you need to prove to show that such a flag shouldn't exist. What you need to show is that no-one will ever find them-self in a situation where such a blunt instrument would be useful.

I contend that they won't. trusted exists and should be used for that purpose. Note that I could find useful disabling of const checks, or override checks, or dynamic casts. It doesn't mean I should get a compiler switch. -Steve
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Thursday, 10 April 2014 at 17:37:53 UTC, Steven Schveighoffer 
wrote:
 On Thu, 10 Apr 2014 13:25:25 -0400, bearophile 
 <bearophileHUGS lycos.com> wrote:

 Steven Schveighoffer:

 No, the author of the  safe code expects bounds checking, 
 it's part of the requirements.

Take a look ad Ada language. It has bounds checking and its compilers have a switch to disable those checks. If you want the bounds checking don't use the switch that disables the bounds checking. Safety doesn't mean to have no way to work around safety locks. It means have nice handy locks that are active on default. In a system language total safety is an illusion. Better to focus on real world safety and not a illusion of theoretical safety.

That's why we have trusted.

No. trusted is for code that cannot be guaranteed to be memory-safe by the compiler (either at runtime or at compile-time), but the programmer still wants to promise that the code is memory-safe. Array bounds checking doesn't land under that moniker, it can be checked by the compiler.
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Thursday, 10 April 2014 at 17:43:42 UTC, Steven Schveighoffer 
wrote:
 [..]
 Note that I could find useful disabling of const checks, or 
 override checks, or dynamic casts. It doesn't mean I should get 
 a compiler switch.

There's no point in turning off any safety / correctness checks that can be performed at compile-time. But I could see an argument made for being able to disable any given category of runtime safety checks.
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 13:45:26 -0400, Tommi <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 17:37:53 UTC, Steven Schveighoffer wrote:
 On Thu, 10 Apr 2014 13:25:25 -0400, bearophile  
 <bearophileHUGS lycos.com> wrote:

 Take a look ad Ada language. It has bounds checking and its compilers  
 have a switch to disable those checks. If you want the bounds checking  
 don't use the switch that disables the bounds checking. Safety doesn't  
 mean to have no way to work around safety locks. It means have nice  
 handy locks that are active on default. In a system language total  
 safety is an illusion. Better to focus on real world safety and not a  
 illusion of theoretical safety.

That's why we have trusted.

No. trusted is for code that cannot be guaranteed to be memory-safe by the compiler (either at runtime or at compile-time), but the programmer still wants to promise that the code is memory-safe. Array bounds checking doesn't land under that moniker, it can be checked by the compiler.

safe code can be marked as trusted instead, and nothing changes, except trusted code can have bounds checks removed. How does this not work as a solution? As Walter often says about logical const, logical safe is safe by convention, and it loses all of its teeth. -Steve
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 13:52:54 -0400, Tommi <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 17:43:42 UTC, Steven Schveighoffer wrote:
 [..]
 Note that I could find useful disabling of const checks, or override  
 checks, or dynamic casts. It doesn't mean I should get a compiler  
 switch.

There's no point in turning off any safety / correctness checks that can be performed at compile-time.

Unless they are warranted. See cast for instance.
 But I could see an argument made for being able to disable any given  
 category of runtime safety checks.

As a compiler switch? I don't think it's worth it. -noboundscheck does what it does for historical reasons, not logical. -Steve
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Thursday, 10 April 2014 at 17:56:04 UTC, Steven Schveighoffer 
wrote:
  safe code can be marked as  trusted instead, and nothing 
 changes, except  trusted code can have bounds checks removed. 
 How does this not work as a solution?

A compiler flag for disabling bounds checking is a blunt instrument. But using search & replace to change each safe to trusted is a blunt _and_ inconvenient instrument.
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 14:08:48 -0400, Tommi <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 17:56:04 UTC, Steven Schveighoffer wrote:
  safe code can be marked as  trusted instead, and nothing changes,  
 except  trusted code can have bounds checks removed. How does this not  
 work as a solution?

A compiler flag for disabling bounds checking is a blunt instrument. But using search & replace to change each safe to trusted is a blunt _and_ inconvenient instrument.

So don't use it bluntly. For example, disabling bounds checks on the args array in main will not help your performance. As a general rule, first profile, then optimize. -Steve
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Thursday, 10 April 2014 at 18:13:30 UTC, Steven Schveighoffer 
wrote:
 On Thu, 10 Apr 2014 14:08:48 -0400, Tommi 
 <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 17:56:04 UTC, Steven 
 Schveighoffer wrote:
  safe code can be marked as  trusted instead, and nothing 
 changes, except  trusted code can have bounds checks removed. 
 How does this not work as a solution?

A compiler flag for disabling bounds checking is a blunt instrument. But using search & replace to change each safe to trusted is a blunt _and_ inconvenient instrument.

So don't use it bluntly. For example, disabling bounds checks on the args array in main will not help your performance.

Sometimes you need that blunt instrument. I wasn't complaining about that.
 As a general rule, first profile, then optimize.

Exactly. I profile the difference between running with and without bounds checking. If the difference is deemed negligible for our purposes, we don't spend time and money in carefully optimizing away bound checks that are analyzed to be reasonably safe to remove. You need the compiler flag to potentially save you all the trouble.
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 15:38:37 -0400, Tommi <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 18:13:30 UTC, Steven Schveighoffer wrote:
 As a general rule, first profile, then optimize.

Exactly. I profile the difference between running with and without bounds checking. If the difference is deemed negligible for our purposes, we don't spend time and money in carefully optimizing away bound checks that are analyzed to be reasonably safe to remove. You need the compiler flag to potentially save you all the trouble.

This is a weak argument. If you need to optimize, do it. Bounds checking is one of a thousand different possible explanations for slow code. You have to weigh that remote possibility with the threat of accidentally/inadvertently neutering safe. You also exaggerate the cost of changing a few safe to trusted. The cost of adding the -noboundscheck flag to the build system in the right places may be just as significant. -Steve
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Thursday, 10 April 2014 at 19:48:16 UTC, Steven Schveighoffer 
wrote:
 On Thu, 10 Apr 2014 15:38:37 -0400, Tommi 
 <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 18:13:30 UTC, Steven 
 Schveighoffer wrote:
 As a general rule, first profile, then optimize.

Exactly. I profile the difference between running with and without bounds checking. If the difference is deemed negligible for our purposes, we don't spend time and money in carefully optimizing away bound checks that are analyzed to be reasonably safe to remove. You need the compiler flag to potentially save you all the trouble.

This is a weak argument. If you need to optimize, do it. Bounds checking is one of a thousand different possible explanations for slow code. You have to weigh that remote possibility with the threat of accidentally/inadvertently neutering safe. You also exaggerate the cost of changing a few safe to trusted. The cost of adding the -noboundscheck flag to the build system in the right places may be just as significant. -Steve

Okay, I give up. You win this debate. Let's prevent D programmers from removing bounds checks from their programs with a compiler flag.
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 16:06:28 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:

 On 04/10/2014 07:37 PM, Steven Schveighoffer wrote:

That's why we have trusted. safe is a special situation, it's not made for optimization, and should be immune to those attempts in deference to safety. -Steve

safe will often be inferred.

A good point. But in this case, the compiler is able to compile 2 versions, one for when safe is required, one for when it's not. But this is only in certain circumstances (global bounds checking is not on, the function is not inlined, code which causes bounds checking is present, etc.) -Steve
Apr 10 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 10 April 2014 at 19:48:16 UTC, Steven Schveighoffer 
wrote:
 On Thu, 10 Apr 2014 15:38:37 -0400, Tommi 
 <tommitissari hotmail.com> wrote:

 On Thursday, 10 April 2014 at 18:13:30 UTC, Steven 
 Schveighoffer wrote:
 As a general rule, first profile, then optimize.

Exactly. I profile the difference between running with and without bounds checking. If the difference is deemed negligible for our purposes, we don't spend time and money in carefully optimizing away bound checks that are analyzed to be reasonably safe to remove. You need the compiler flag to potentially save you all the trouble.

This is a weak argument. If you need to optimize, do it. Bounds checking is one of a thousand different possible explanations for slow code. You have to weigh that remote possibility with the threat of accidentally/inadvertently neutering safe. You also exaggerate the cost of changing a few safe to trusted. The cost of adding the -noboundscheck flag to the build system in the right places may be just as significant. -Steve

Changing druntime and phobos is a much bigger deal than a flag. In a perfect world I'd agree with you completely but as a practical matter I think the flag needs to stay. There is also an issue of marketing (as annoying as it is). This problem is reminiscent of the garbage collector argument against D. For far too many people D is a no-go simply because it has a garbage collector. In reality the garbage collector isn't a problem for the overwhelming majority of use cases but many people will never use the language for this reason. If someone wants to be reckless and turn off the compiler adding bounds checking in safe code D should allow them to. It is a systems programming language, after all. Being allowed to be reckless and even stupid is a hallmark of that. That doesn't mean we can't guide people to make smart choices though. -release will always do bounds checking on safe functions. How about a compromise? A stern warning is given whenever someone compiles with -boundscheck=none (-noboundscheck). One that stresses they should benchmark before commit to turning it off and that it should only be used as a last resort.
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 16:13:17 -0400, Brad Anderson <eco gnuk.net> wrote:

 On Thursday, 10 April 2014 at 19:48:16 UTC, Steven Schveighoffer wrote:
 On Thu, 10 Apr 2014 15:38:37 -0400, Tommi <tommitissari hotmail.com>  
 wrote:

 On Thursday, 10 April 2014 at 18:13:30 UTC, Steven Schveighoffer wrote:
 As a general rule, first profile, then optimize.

Exactly. I profile the difference between running with and without bounds checking. If the difference is deemed negligible for our purposes, we don't spend time and money in carefully optimizing away bound checks that are analyzed to be reasonably safe to remove. You need the compiler flag to potentially save you all the trouble.

This is a weak argument. If you need to optimize, do it. Bounds checking is one of a thousand different possible explanations for slow code. You have to weigh that remote possibility with the threat of accidentally/inadvertently neutering safe. You also exaggerate the cost of changing a few safe to trusted. The cost of adding the -noboundscheck flag to the build system in the right places may be just as significant. -Steve

Changing druntime and phobos is a much bigger deal than a flag.

It's not a flag, it's possibly rebuilding druntime/phobos in an unapproved way. -Steve
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 18:54:57 -0400, Martin Krejcirik <mk-junk i-line.cz>  
wrote:

 On 10.4.2014 19:12, Steven Schveighoffer wrote:
 void foo(T)(T[] x)  safe
 {
    x[5] = 3;
 }

Is this common practice ? I'd wouldn't call it a safe design.

No, this isn't common practice. It's a demonstration of how safe code can become un- safe when compiled independent of the original author. In reality, nobody should rely on a bounds check to catch out of bounds conditions (I think it's an Error anyway -- a non-recoverable situation). But memory bugs in safe code should be caught, at compile time if possible, at runtime if not.
 There
 should be a length check or version check:

 version(D_NoBoundsChecks) static assert(0, "bounds checking required");

 But I get your point, I have always thought of bounds checking like an
 optional safety net, you think of it like a required feature.

It is a safety net, but not optional for safe code. The promise of safe is that you CANNOT have a memory corruption error. To break the promise without the knowledge of the code is an incorrect plan, IMO. When you write safe, it means "this code is memory safe, even if it has bugs." Perhaps Andrei and Walter feel differently. I don't know. -Steve
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 19:21:52 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 4/10/14, 10:56 AM, Steven Schveighoffer wrote:
  safe code can be marked as  trusted instead, and nothing changes,
 except  trusted code can have bounds checks removed. How does this not
 work as a solution?

Doesn't work because trusted removes all additional checks by the compiler. We do want to have safe as mechanically verified.

I think you are missing something. This is code that is marked as safe. Then you change it to trusted. It doesn't change that the code is safe, but now it's trusted so the bounds checks can be removed temporarily. The assertion is that -noboundscheck for safe code is a temporary condition to find possible optimizations. Then if this helps, you can find a more permanent mechanism to fix the performance. A similar mechanism is to use trusted to allow bounds checks to go away temporarily, as a test. Any function that is safe can be instead marked as trusted, and nothing else changes.
 As Walter often says about logical const, logical  safe is  safe by
 convention, and it loses all of its teeth.

I think you are wrong here.

Victims with bleeding hearts around the world are feeling the effects of logical safety. If safe is just a convention, then I don't see the point of having it at all. If it can't be a guarantee, then it's pretty much another tech buzzword with no teeth. -Steve
Apr 10 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Friday, 11 April 2014 at 00:52:25 UTC, Steven Schveighoffer 
wrote:
 If  safe is just a convention, then I don't see the point of 
 having it at all. If it can't be a guarantee, then it's pretty 
 much another tech buzzword with no teeth.

In order to have safe be a guarantee of memory-safety, we need to prevent safe code from calling any trusted code.
Apr 10 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 Steven Schveighoffer:
 If  safe is just a convention, then I don't see the point of 
 having it at all.

That I agree with. -- Andrei

I agree. But a safety that can be disabled only by an explicit and visible request is not a convention. Bye, bearophile
Apr 11 2014
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Thursday, 10 April 2014 at 07:14:10 UTC, Marco Leise wrote:
 Am Thu, 10 Apr 2014 06:51:40 +0000
 schrieb "w0rp" <devw0rp gmail.com>:

 On Wednesday, 9 April 2014 at 12:36:49 UTC, Marco Leise wrote:
 Sorry, but wasn't this security risk instead caused by
 uninitialized memory, and shouldn't you instead have said:

 "I'm glad to be using a language with default 
 initialization?"

Nope, it was caused by missing bounds checking. https://www.openssl.org/news/secadv_20140407.txt
 A missing bounds check [...]


Haha, I tried to read that about an hour ago to inform myself, but it still doesn't load for me.

http://blog.existentialize.com/diagnosis-of-the-openssl-heartbleed-bug.html The server copies data received from the client and sends it back, the length is specified (or forged) by the client, everything is initialized just fine.
Apr 11 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 11 Apr 2014 00:01:17 -0400, Tommi <tommitissari hotmail.com> wrote:

 On Friday, 11 April 2014 at 00:52:25 UTC, Steven Schveighoffer wrote:
 If  safe is just a convention, then I don't see the point of having it  
 at all. If it can't be a guarantee, then it's pretty much another tech  
 buzzword with no teeth.

In order to have safe be a guarantee of memory-safety, we need to prevent safe code from calling any trusted code.

Or manually guarantee the safety of trusted code. I should be able to write an application with only safe functions, and trust that phobos has implemented trusted functions properly. -Steve
Apr 11 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Friday, 11 April 2014 at 12:00:32 UTC, Steven Schveighoffer 
wrote:
 On Fri, 11 Apr 2014 00:01:17 -0400, Tommi 
 <tommitissari hotmail.com> wrote:

 On Friday, 11 April 2014 at 00:52:25 UTC, Steven Schveighoffer 
 wrote:
 If  safe is just a convention, then I don't see the point of 
 having it at all. If it can't be a guarantee, then it's 
 pretty much another tech buzzword with no teeth.

In order to have safe be a guarantee of memory-safety, we need to prevent safe code from calling any trusted code.

Or manually guarantee the safety of trusted code. I should be able to write an application with only safe functions, and trust that phobos has implemented trusted functions properly. -Steve

I was talking about safe in general sense, not only as it pertains to phobos.
Apr 11 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 11 Apr 2014 08:37:59 -0400, Tommi <tommitissari hotmail.com> wrote:

 On Friday, 11 April 2014 at 12:00:32 UTC, Steven Schveighoffer wrote:
 On Fri, 11 Apr 2014 00:01:17 -0400, Tommi <tommitissari hotmail.com>  
 wrote:

 On Friday, 11 April 2014 at 00:52:25 UTC, Steven Schveighoffer wrote:
 If  safe is just a convention, then I don't see the point of having  
 it at all. If it can't be a guarantee, then it's pretty much another  
 tech buzzword with no teeth.

In order to have safe be a guarantee of memory-safety, we need to prevent safe code from calling any trusted code.

Or manually guarantee the safety of trusted code. I should be able to write an application with only safe functions, and trust that phobos has implemented trusted functions properly.

I was talking about safe in general sense, not only as it pertains to phobos.

trusted code needs to be severely scrutinized, since you are turning off safety checks, no matter where the code is. My point was that if I want to ONLY write safe code, I should be able to without fear that the trusted functions in Phobos are susceptible to memory bugs. The idea is that the number of functions that are trusted is very small, so we can be extra cautious and focused with them. Then the number of safe functions can be quite large and useful. Kind of like casts -- you should really scrutinize and focus on casts to make sure nothing untoward is happening, since the compiler is deferring to you. One thing that kind of sucks about this, is that safe vs. trusted is a large bag of safety checks. If all you want is to disable array bounds checking, for maybe one or two array indexes in a large function, it's a large price to pay to mark the entire function trusted. Inner functions help with this a bit, but maybe one should be able to mark an expression as trusted inside a safe function. Give the developer a scalpel instead of a saws-all. -Steve
Apr 11 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 11 Apr 2014 08:35:07 -0400, Daniel Murphy  
<yebbliesnospam gmail.com> wrote:

 "Steven Schveighoffer"  wrote in message  
 news:op.xd3vzecweav7ka stevens-macbook-pro.local...

 No, the author of the  safe code expects bounds checking, it's part of  
 the requirements. To compile his code with it off is like having  
   -compilergeneratedhash switch that overrides any toHash functions  
 with a compiler generated one. You are changing the agreement between  
 the compiler and the code. When I say  safe, I mean "I absolutely  
 always want bounds checks."

If you have code that would ever fail a bounds check, that is a program error, similar to code that may fail an assertion. And like assertions, if you would rather the code was as fast as possible instead of as safe as possible you can use a compiler switch to disable bound checks. The usual switch to do stuff like this is '-release', but because safe functions should still have the 'no memory corruption' even in release mode, disabling those bounds checks was moved into another compiler switch.
 If you want to eliminate bounds checks, use  trusted.

No, trusted means "don't check my code" while safe + noboundschecks means (mostly) "only check my code at compile-time".

Here is the horror scenario I envision: 1. Company has 100kLOC project, which is marked as safe (I can dream, can't I?) 2. They find that performance is lacking, maybe compared to a competitor's C++ based code. 3. They try compiling with -noboundscheck, get a large performance boost. It really only makes a difference in one function (the inner loop one). 4. They pat themselves on the back, and release with the new flag, destroying all bounds checks, even bounds checks in library template code that they didn't write or scrutinize. 5. Buffer overflow attacks abound. 6. D safe is labeled a "joke" But there is a cost, even to labeling the "one inner" function trusted. Perhaps that function is extremely long and complex. There should be a way to say, "I still want all the safety checks, except for this one critical array access, I have manually guaranteed the bounds". We don't have anything like that. All other safety checks are really static, this is the only runtime penalty for safety. The blunt flag approach is scary. trusted is better, in that you can focus on one function at a time. But I think we need something more precise. Perhaps you should be able to have trusted scopes, or trusted expressions. -Steve
Apr 11 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Friday, 11 April 2014 at 13:13:22 UTC, Steven Schveighoffer 
wrote:
 On Fri, 11 Apr 2014 08:35:07 -0400, Daniel Murphy 
 <yebbliesnospam gmail.com> wrote:

 "Steven Schveighoffer"  wrote in message 
 news:op.xd3vzecweav7ka stevens-macbook-pro.local...

 No, the author of the  safe code expects bounds checking, 
 it's part of the requirements. To compile his code with it 
 off is like having
  -compilergeneratedhash switch that overrides any toHash 
 functions with a compiler generated one. You are changing the 
 agreement between the compiler and the code. When I say 
  safe, I mean "I absolutely always want bounds checks."

If you have code that would ever fail a bounds check, that is a program error, similar to code that may fail an assertion. And like assertions, if you would rather the code was as fast as possible instead of as safe as possible you can use a compiler switch to disable bound checks. The usual switch to do stuff like this is '-release', but because safe functions should still have the 'no memory corruption' even in release mode, disabling those bounds checks was moved into another compiler switch.
 If you want to eliminate bounds checks, use  trusted.

No, trusted means "don't check my code" while safe + noboundschecks means (mostly) "only check my code at compile-time".

Here is the horror scenario I envision: 1. Company has 100kLOC project, which is marked as safe (I can dream, can't I?) 2. They find that performance is lacking, maybe compared to a competitor's C++ based code. 3. They try compiling with -noboundscheck, get a large performance boost. It really only makes a difference in one function (the inner loop one). 4. They pat themselves on the back, and release with the new flag, destroying all bounds checks, even bounds checks in library template code that they didn't write or scrutinize. 5. Buffer overflow attacks abound. 6. D safe is labeled a "joke"

More likely: 6. This company's programming department is labeled a "joke".
 There should be a way to say, "I still want all the  safety 
 checks, except for this one critical array access, I have 
 manually guaranteed the bounds". We don't have anything like 
 that.

We have array.ptr[idx]
Apr 11 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 11 Apr 2014 09:35:12 -0400, Tommi <tommitissari hotmail.com> wrote:

 On Friday, 11 April 2014 at 13:13:22 UTC, Steven Schveighoffer wrote:
 Here is the horror scenario I envision:

 1. Company has 100kLOC project, which is marked as  safe (I can dream,  
 can't I?)
 2. They find that performance is lacking, maybe compared to a  
 competitor's C++ based code.
 3. They try compiling with -noboundscheck, get a large performance  
 boost. It really only makes a difference in one function (the inner  
 loop one).
 4. They pat themselves on the back, and release with the new flag,  
 destroying all bounds checks, even bounds checks in library template  
 code that they didn't write or scrutinize.
 5. Buffer overflow attacks abound.
 6. D  safe is labeled a "joke"

More likely: 6. This company's programming department is labeled a "joke".

Perhaps, but it doesn't change the idea that safe code had memory bugs. What we are saying with safe is that you CAN'T have memory bugs, no matter how incompetent your programmers are.
 There should be a way to say, "I still want all the  safety checks,  
 except for this one critical array access, I have manually guaranteed  
 the bounds". We don't have anything like that.

We have array.ptr[idx]

Not allowed in safe code. -Steve
Apr 11 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Friday, 11 April 2014 at 13:44:09 UTC, Steven Schveighoffer 
wrote:
 On Fri, 11 Apr 2014 09:35:12 -0400, Tommi 
 <tommitissari hotmail.com> wrote:

 On Friday, 11 April 2014 at 13:13:22 UTC, Steven Schveighoffer 
 wrote:
 [..]
 6. D  safe is labeled a "joke"

More likely: 6. This company's programming department is labeled a "joke".

Perhaps, but it doesn't change the idea that safe code had memory bugs. What we are saying with safe is that you CAN'T have memory bugs, no matter how incompetent your programmers are.

You can't gurantee safe to be memory-safe in the general case without disallowing calls to trusted, because those incompenent programmers can write buggy trusted functions and call them from safe code.
 There should be a way to say, "I still want all the  safety 
 checks, except for this one critical array access, I have 
 manually guaranteed the bounds". We don't have anything like 
 that.

We have array.ptr[idx]

Not allowed in safe code.

trusted ref T unsafeIndex(T)(T[] array, ulong idx) { return array.ptr[idx]; } There you go.
Apr 11 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Friday, 11 April 2014 at 14:06:33 UTC, Daniel Murphy wrote:
 Trying to prevent developer stupidity is a lost cause.

 Bounds checks are on by default.  They are even on when you ask 
 for 'fast-over-safe' aka -release.  They get turned off when 
 you explicitly ask for it.

 But there is a cost, even to labeling the "one inner" function 
  trusted. Perhaps that function is extremely long and complex. 
 There should be a way to say, "I still want all the  safety 
 checks, except for this one critical array access, I have 
 manually guaranteed the bounds". We don't have anything like 
 that. All other safety checks are really static, this is the 
 only runtime penalty for safety.

Something like (() trusted => arr.ptr[index]) should do the trick.
 The blunt flag approach is scary.  trusted is better, in that 
 you can focus on one function at a time. But I think we need 
 something more precise. Perhaps you should be able to have 
  trusted scopes, or  trusted expressions.

trusted delegates get you 99.99% of the way there.

Hasn't there been a proposal before to allow system/ trusted/ safe blocks, allowing it to be a bit more granular than at the function level? Maybe: trusted { arr.ptr[index] } Could be lowered to (() trusted => arr.ptr[index]).
Apr 11 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 11 April 2014 at 15:15:21 UTC, Meta wrote:
 Hasn't there been a proposal before to allow 
  system/ trusted/ safe blocks, allowing it to be a bit more 
 granular than at the function level? Maybe:

  trusted
 {
     arr.ptr[index]
 }

 Could be lowered to (()  trusted => arr.ptr[index]).

I think it was rejected for the very reason that it gives nothing over writing this: () trusted { arr.ptr[index]; }(); It has resulted in some changes by Kenji though that guarantee that immediately-called delegate is always inlined. Hope those were merged.
Apr 11 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Friday, 11 April 2014 at 15:43:16 UTC, Dicebot wrote:
 On Friday, 11 April 2014 at 15:15:21 UTC, Meta wrote:
 Hasn't there been a proposal before to allow 
  system/ trusted/ safe blocks, allowing it to be a bit more 
 granular than at the function level? Maybe:

  trusted
 {
    arr.ptr[index]
 }

 Could be lowered to (()  trusted => arr.ptr[index]).

I think it was rejected for the very reason that it gives nothing over writing this: () trusted { arr.ptr[index]; }(); It has resulted in some changes by Kenji though that guarantee that immediately-called delegate is always inlined. Hope those were merged.

Besides the fact that it's incredibly ugly.
Apr 11 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 11 April 2014 at 15:45:41 UTC, Meta wrote:
 Besides the fact that it's incredibly ugly.

Which is not enough for introduction of any single new feature at current stage.
Apr 11 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 11 Apr 2014 11:43:15 -0400, Dicebot <public dicebot.lv> wrote:

 On Friday, 11 April 2014 at 15:15:21 UTC, Meta wrote:
 Hasn't there been a proposal before to allow  system/ trusted/ safe  
 blocks, allowing it to be a bit more granular than at the function  
 level? Maybe:

  trusted
 {
     arr.ptr[index]
 }

 Could be lowered to (()  trusted => arr.ptr[index]).

I think it was rejected for the very reason that it gives nothing over writing this: () trusted { arr.ptr[index]; }(); It has resulted in some changes by Kenji though that guarantee that immediately-called delegate is always inlined. Hope those were merged.

If this works as seamlessly as a statement, that is a reasonable solution. Re: ugliness, it's not important. These are not common situations. -Steve
Apr 11 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 11 Apr 2014 11:38:54 +0000
schrieb "Kagamin" <spam here.lot>:

 On Thursday, 10 April 2014 at 07:14:10 UTC, Marco Leise wrote:
 Am Thu, 10 Apr 2014 06:51:40 +0000
 schrieb "w0rp" <devw0rp gmail.com>:

 On Wednesday, 9 April 2014 at 12:36:49 UTC, Marco Leise wrote:
 Sorry, but wasn't this security risk instead caused by
 uninitialized memory, and shouldn't you instead have said:

 "I'm glad to be using a language with default 
 initialization?"

Nope, it was caused by missing bounds checking. https://www.openssl.org/news/secadv_20140407.txt
 A missing bounds check [...]


Haha, I tried to read that about an hour ago to inform myself, but it still doesn't load for me.

http://blog.existentialize.com/diagnosis-of-the-openssl-heartbleed-bug.html The server copies data received from the client and sends it back, the length is specified (or forged) by the client, everything is initialized just fine.

Ah, so this is a typical ping packet, where you copy all payload bytes from the client's packet and send them back. Just that in this case the client can write anything into the length header and OpenSSL would try to copy as many bytes from the client provided packet into the answer packet, even if that means reading beyond the end of that packet. This still doesn't touch D's array bounds checking at all, because the array pointer and length come from an unreliable source to begin with! -- Marco
Apr 11 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 11 Apr 2014 12:21:06 -0400, Marco Leise <Marco.Leise gmx.de> wrote:

 Am Fri, 11 Apr 2014 11:38:54 +0000
 schrieb "Kagamin" <spam here.lot>:

 http://blog.existentialize.com/diagnosis-of-the-openssl-heartbleed-bug.html
 The server copies data received from the client and sends it
 back, the length is specified (or forged) by the client,
 everything is initialized just fine.

Ah, so this is a typical ping packet, where you copy all payload bytes from the client's packet and send them back. Just that in this case the client can write anything into the length header and OpenSSL would try to copy as many bytes from the client provided packet into the answer packet, even if that means reading beyond the end of that packet. This still doesn't touch D's array bounds checking at all, because the array pointer and length come from an unreliable source to begin with!

But in a D-created struct, the data would be an array, instead of a ptr+length. -Steve
Apr 11 2014
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Friday, 11 April 2014 at 16:17:49 UTC, Marco Leise wrote:
 This still doesn't touch D's array bounds checking at all,
 because the array pointer and length come from an unreliable
 source to begin with!

In D implementation the client packet would be reliably confined by a slice, so the forged length will be checked against packet boundaries. byte[] packet = recieve(); int length = get_payload_length(packet); dest[0..length] = packet[3..3+length];
Apr 11 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
 But in a D-created struct, the data would be an array, instead of a  
 ptr+length.
 
 -Steve

If I understand you right, you mean a variation of this: struct Packet { ubyte[] payload; } But indirections don't fly with serialized network packets. Am Fri, 11 Apr 2014 17:35:15 +0000 schrieb "Kagamin" <spam here.lot>:
 In D implementation the client packet would be reliably confined 
 by a slice, so the forged length will be checked against packet 
 boundaries.
 
 byte[] packet = recieve();
 int length = get_payload_length(packet);
 dest[0..length] = packet[3..3+length];

I'd argue that at this point you already knew you would mess up the heartbeat code and designed it directly around D's bounds checks instead of using structs and direct access of header fields. It's a clever solution that you propose here. Have you used it on real code before (i.e. does it scale) or did you come up with it just for this case? -- Marco
Apr 11 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 11 Apr 2014 18:01:29 -0400, Marco Leise <Marco.Leise gmx.de> wrote:

 But in a D-created struct, the data would be an array, instead of a
 ptr+length.

 -Steve

If I understand you right, you mean a variation of this: struct Packet { ubyte[] payload; } But indirections don't fly with serialized network packets.

Indirections were in the struct that was the subject of that article. Here it is: typedef struct ssl3_record_st { int type; /* type of record */ unsigned int length; /* How many bytes available */ unsigned int off; /* read/write offset into 'buf' */ unsigned char *data; /* pointer to the record data */ unsigned char *input; /* where the decode bytes are */ unsigned char *comp; /* only used with decompression - malloc()ed */ unsigned long epoch; /* epoch number, needed by DTLS1 */ unsigned char seq_num[8]; /* sequence number, needed by DTLS1 */ } SSL3_RECORD; No way that's a directly serialized network packet. -Steve
Apr 11 2014
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Friday, 11 April 2014 at 21:58:13 UTC, Marco Leise wrote:
 byte[] packet = recieve();
 int length = get_payload_length(packet);
 dest[0..length] = packet[3..3+length];

I'd argue that at this point you already knew you would mess up the heartbeat code and designed it directly around D's bounds checks instead of using structs and direct access of header fields. It's a clever solution that you propose here. Have you used it on real code before (i.e. does it scale) or did you come up with it just for this case?

Well, I didn't write openssl in D, so this code is clearly made up for this case. But I don't circumvent bounds checking in my code. I don't know, how well it scales, I never needed such microoptimizations, the speed is either acceptable or not, and I have never seen bounds checking to promote the code from the former class to the latter. Though, I don't think openssl developer didn't want bounds checking in their code, they used memcpy to copy the payload. There's memcpy_s function with bounds checking, but it's standardized only as an optional extension to C11, so they were just out of luck to realistically require it. This code is not specialized for this particular buffer overrun bug, slices account for all buffer overrun bugs, wherever they are. This is the whole point in language integrated bounds checks: to make their use simple. I didn't invent anything new, it's *the* D way to handle buffers as slices, pointers are usually not used: http://dlang.org/phobos/std_stdio.html#.File.rawRead http://dlang.org/phobos/std_socket.html#.Socket.receive In fact, openssl didn't use structs to parse the client packet, they used raw byte pointer: hbtype = *p++; //read type n2s(p, payload); //read network-order short and increment pointer by 2 pl = p; memcpy(bp, pl, payload); and equivalent D code would use slice.
Apr 12 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 12 Apr 2014 12:21:55 +0000
schrieb "Kagamin" <spam here.lot>:

 On Friday, 11 April 2014 at 21:58:13 UTC, Marco Leise wrote:
 byte[] packet = recieve();
 int length = get_payload_length(packet);
 dest[0..length] = packet[3..3+length];

I'd argue that at this point you already knew you would mess up the heartbeat code and designed it directly around D's bounds checks instead of using structs and direct access of header fields. It's a clever solution that you propose here. Have you used it on real code before (i.e. does it scale) or did you come up with it just for this case?

Well, I didn't write openssl in D, so this code is clearly made up for this case. But I don't circumvent bounds checking in my code. I don't know, how well it scales, I never needed such microoptimizations, the speed is either acceptable or not, and I have never seen bounds checking to promote the code from the former class to the latter.

Ah! Big misunderstanding. I meant to ask if you used this idiom before, that you use just a byte[] for data you load/receive to be able to use bounds checks and not some header struct with pointers where they become ineffective? And with scale I was aiming for the software architecture. Can you keep this code style with global functions working on byte[] instead of structs or will you wish for the style used in OpenSSH, which is unsafe but probably easier to maintain?
 This code is not specialized for this particular buffer overrun 
 bug, slices account for all buffer overrun bugs, wherever they 
 are. This is the whole point in language integrated bounds 
 checks: to make their use simple. I didn't invent anything new, 
 it's *the* D way to handle buffers as slices, pointers are 
 usually not used:
 http://dlang.org/phobos/std_stdio.html#.File.rawRead
 http://dlang.org/phobos/std_socket.html#.Socket.receive

Because rawRead and receive are meant to read N bytes. Perfect match for a slice. In a library like OpenSSH you know what your headers will look like, though.
 In fact, openssl didn't use structs to parse the client packet, 
 they used raw byte pointer:
 
 hbtype = *p++; //read type
 n2s(p, payload); //read network-order short and increment pointer 
 by 2
 pl = p;
 memcpy(bp, pl, payload);
 
 and equivalent D code would use slice.

Ok, that half-way convinced me. :) hbtype = buffer[0]; // read type, ok // but this??? ugh!!! payload = bigEndianToHost(*cast(short*)buffer[1 .. 3].ptr); bp[0 .. payload] = buffer[3 .. 3+payload]; The code becomes pretty messy in my eyes. I would have just used memcpy as well, if that was the alternative. That's why I asked if it will scale or if you just made it up for the sake of argument. -- Marco
Apr 12 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 11 Apr 2014 22:52:23 -0400
schrieb "Steven Schveighoffer" <schveiguy yahoo.com>:

 On Fri, 11 Apr 2014 18:01:29 -0400, Marco Leise <Marco.Leise gmx.de> wrote:
 
 But in a D-created struct, the data would be an array, instead of a
 ptr+length.

 -Steve

If I understand you right, you mean a variation of this: struct Packet { ubyte[] payload; } But indirections don't fly with serialized network packets.

Indirections were in the struct that was the subject of that article. Here it is: typedef struct ssl3_record_st { int type; /* type of record */ unsigned int length; /* How many bytes available */ unsigned int off; /* read/write offset into 'buf' */ unsigned char *data; /* pointer to the record data */ unsigned char *input; /* where the decode bytes are */ unsigned char *comp; /* only used with decompression - malloc()ed */ unsigned long epoch; /* epoch number, needed by DTLS1 */ unsigned char seq_num[8]; /* sequence number, needed by DTLS1 */ } SSL3_RECORD; No way that's a directly serialized network packet. -Steve

If "data" and "length" are the pointer and size of the raw network packet, yes. I.e. it will turn into: ssl3_record_st.data[] = socket.receiveRawPacket(); ... which is safe. But when you said "D created struct" I assumed you referred to the network packet itself as a struct, which means that the length field would come directly from the client and doesn't need to match the physical length of the packet. With Kagamin's reply I now understand it better. -- Marco
Apr 12 2014
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Saturday, 12 April 2014 at 19:03:58 UTC, Marco Leise wrote:
 Ah! Big misunderstanding. I meant to ask if you used this
 idiom before, that you use just a byte[] for data you
 load/receive to be able to use bounds checks and not some
 header struct with pointers where they become ineffective?

 And with scale I was aiming for the software architecture. Can
 you keep this code style with global functions working on
 byte[] instead of structs or will you wish for the style used
 in OpenSSH, which is unsafe but probably easier to maintain?

You can process fixed-length message with structs, but processing variable-length message requires access to the buffer with packet, as we see in the example of ssl heartbeat.
 Because rawRead and receive are meant to read N bytes. Perfect
 match for a slice. In a library like OpenSSH you know what
 your headers will look like, though.

How is it different from memcpy? It's conceptually copying too. with slice: read(buffer[0..length]); with pointer: read(buffer, length);
 hbtype = buffer[0]; // read type, ok
 // but this??? ugh!!!
 payload = bigEndianToHost(*cast(short*)buffer[1 .. 3].ptr);
 bp[0 .. payload] = buffer[3 .. 3+payload];

 The code becomes pretty messy in my eyes. I would have just
 used memcpy as well, if that was the alternative.
 That's why I asked if it will scale or if you just made it up
 for the sake of argument.

I think, memcpy is messier than slice copy. It will be clearly visible, that you circumvent bounds checking, and that will smell, and everyone looking at it will tell you what they think about it.
Apr 14 2014
prev sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Marco Leise:

 hbtype = buffer[0]; // read type, ok
 // but this??? ugh!!!
 payload = bigEndianToHost(*cast(short*)buffer[1 .. 3].ptr);
 bp[0 .. payload] = buffer[3 .. 3+payload];

 The code becomes pretty messy in my eyes.

In most cases D allows you to find a way to write your code in a reasonably nice and reasonably safe way. You just have to be in the right mind frame. Bye, bearophile
Apr 14 2014