www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - system blocks and safer trusted (ST) functions

reply Bruce Carneal <bcarneal gmail.com> writes:
At beerconf I committed to putting forward a DIP regarding a new 
syntactic element to be made available within  trusted functions, 
the  system block.  The presence of one or more  system blocks 
would enable  safe checking elsewhere in the enclosing  trusted 
function.

I also committed at beerconf to starting the formal DIP on this 
in two weeks so I read the DIP docs just now in order to get a 
jump on things.  Boy, that was an eye-opener.  From the 
incessantly cautionary language there I'd say that I dramatically 
underestimated the effort required to bring a DIP over the finish 
line (or to see it clearly rejected).

Long story short, I'll still do the DIP unless the ideas are 
discredited in subsequent discussion but it will have to be quite 
a bit later in the year as I dont have weeks of time to spend on 
it in the near future.  In the mean time I'd invite others to 
comment on the ST (safer  trusted) idea as sketched in the first 
paragraph.  For starters, we might come up with a better name...

A few notes:

1) Since  system blocks would be a new syntactic element I 
believe ST would be a backward compatible addition.

2) The problematic  trusted lambda escapes creeping in to " safe" 
code could be replaced going forward by a more honestly named 
form,  trusted code with  system blocks.  Best practices could 
evolve to the point that  safe actually meant  safe again.

3) I believe quite a bit of  trusted code is  safe but can not 
currently benefit from automated checking.  The proposal would 
allow a transition of such code to an  safer form while reducing 
the amount of code that requires intense manual checking.

4)  safe blocks within  trusted code were considered but left 
things defaulting in a less useful direction (and they're just 
plain ugly).

Questions and improvements are very welcome but, in the dlang 
tradition, I also welcome your help in destroying this proposal.
Jul 24
next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:
 The presence of one or more  system blocks would enable  safe 
 checking elsewhere in the enclosing  trusted function.
This has an unfortunate result: if in maintenance you edit a trusted function to remove its system blocks, it'll quietly no longer be checked as safe. Of the current behavior, Walter's said that he doesn't want trusted blocks because they should be discouraged in safe code. With this change, we'll have exactly what he doesn't want with different names: s/ trusted/ system/, s/ safe/ trusted/, and the exact same behavior: system blocks are just what trusted blocks would've been, and trusted code with system blocks in it is just safe code with a different name. Instead of people accepting that safe "isn't really safe" in the presence of trusted blocks, and that the whole body of the function has to be audited, with this change we'll have s/ safe/ trusted/ blocks that aren't really safe in the presence of system blocks, and that the whole body of the function has to be audited. The "you have to audit this" signifier is the same, an internal lower-protection block, and all that's gained is that the function attribute's spelled differently. Is this really worth it? One way to avoid the "unfortunate result" above is to permit trusted blocks in safe code. Which results in Rust equivalent functionality: safe code is checked as safe, safe code with internal trusted blocks is still checked as safe but people know to audit it, and system code isn't checked as safe. People like Rust's unsafe system, the current trusted-lambda abuse is a simulation of unsafe, Phobos already uses trusted lambdas in exactly the same way as unsafe blocks are used in Rust, and this proposed behavior is 99.9% identical to unsafe except it has this bonus "you can silently, accidentally remove safe checks from your code now" feature. I appreciate that there's a vision also to safe/ trusted/ system, but it doesn't seem to have stuck, with Phobos having more than twice as many trusted lambdas than trusted functions: ``` phobos$ grep -rP '\W\(\) trusted' --include '*.d'|wc -l 238 phobos$ grep -rP '\w\(\) trusted' --include '*.d'|wc -l 111 ``` I don't think that Rust has everything right. And, I don't pay attention to the Rust community at all; maybe they've a lot of gripes about how they're using unsafe blocks and unsafe functions. But, just look at all those trusted lambadas. If you run the first command without the "|wc -l" on the end you'll see they're almost all single statements or expressions. Adding a trusted block to safe code doesn't discard the safe/ trusted/ system vision, it just lets people follow the unsafe vision that they're already following without so many complaints about how ugly the workaround is, when D's good looks are one of its advantages over Rust. This proposal also doesn't immediately discard the safe/ trusted/ system vision, but it introduces a minor footgun because of a subtle conflict with that vision, and as people adopt it they'll also want another --preview switch to deal with the footgun, and that switch will break all current trusted code that's currently assuming no safe checks, and so there will a long deprecation cycle... trusted blocks win this. Or rather--not to be rude--but if you came out and said that this DIP was just some triangulation to get people to accept trusted blocks, I would say: "good job! It got me thinking!" If not, I'm sorry.
Jul 24
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 25 July 2021 at 06:13:41 UTC, jfondren wrote:
 On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:
 The presence of one or more  system blocks would enable  safe 
 checking elsewhere in the enclosing  trusted function.
This has an unfortunate result: if in maintenance you edit a trusted function to remove its system blocks, it'll quietly no longer be checked as safe.
As opposed to never being checked for safe in legacy trusted? Or always requiring manual vigilance in an " safe" routine corrupted by an trusted lambda? More directly, IIUC, someone explicitly downgrades safety by removing an system block encapsulation from within a recently coded (non-legacy) trusted function and we're supposed to second guess them? They don't actually want to revert to legacy behavior they just happened to remove the system block in an trusted function? I think it would be great if we can get to the point where warnings against legacy trusted reversions are welcome but in the mean time I believe the danger of not protecting against explicit safety degradation maintenance errors is modest when compared to the other issues.
 Of the current behavior, Walter's said that he doesn't want 
  trusted blocks because they should be discouraged in  safe 
 code. With this change, we'll have exactly what he doesn't want 
 with different names: s/ trusted/ system/, s/ safe/ trusted/, 
 and the exact same behavior:  system blocks are just what 
  trusted blocks would've been, and  trusted code with  system 
 blocks in it is just  safe code with a different name.
I think you're misreading Walter on this. He was the one who recommended that I pursue this DIP at beerconf (it was just an idea that I'd thrown out up to that point).
 Instead of people accepting that  safe "isn't really  safe" in 
 the presence of  trusted blocks, and that the whole body of the 
 function has to be audited, with this change we'll have 
 s/ safe/ trusted/ blocks that aren't really  safe in the 
 presence of  system blocks, and that the whole body of the 
 function has to be audited. The "you have to audit this" 
 signifier is the same, an internal lower-protection block, and 
 all that's gained is that the function attribute's spelled 
 differently. Is this really worth it?
Definitely. We shouldn't "accept" contradictions, cognitive load, manual auditing, ... unless we have no alternative. The proposal opens the door to consistent nesting and naming and additional automated checking. Less "unsprung weight".
 
 I appreciate that there's a vision also to 
  safe/ trusted/ system, but it doesn't seem to have stuck, with 
 Phobos having more than twice as many  trusted lambdas than 
  trusted functions:

 ```
 phobos$ grep -rP '\W\(\)  trusted' --include '*.d'|wc -l
 238
 phobos$ grep -rP '\w\(\)  trusted' --include '*.d'|wc -l
 111
 ```
If the lambdas are mostly found within safe code then these stats would support the proposal. Also, the trusted functions could benefit from system block upgrades.
 ...

 Adding a  trusted block to  safe code doesn't discard the 
  safe/ trusted/ system vision, it just lets people follow the 
 unsafe vision that they're already following  without so many 
 complaints about how ugly the workaround is, when D's good 
 looks are one of its advantages over Rust.
I'd like to localize system code and minimize un-automated checking. An trusted lambda in an safe function goes in the other direction. It's not the two or three liner workaround itself that is the main problem, it's the silent expansion of programmer responsibility to the surrounds.
 This proposal also doesn't immediately discard the 
  safe/ trusted/ system vision, but it introduces a minor 
 footgun because of a subtle conflict with that vision, and as 
 people adopt it they'll also want another --preview switch to 
 deal with the footgun, and that switch will break all current 
  trusted code that's currently assuming no  safe checks, and so 
 there will a long deprecation cycle...
I don't see it that way. I see a simple path to safer code. Your "footgun" is, in my opinion and mixing metaphors, small potatoes compared to the additional automated coverage evident in the proposal. As I hope you'll agree, there is a huge gap between "this isn't a big deal, most competent programmers would not mess up here very often..." and "no programmers will ever mess up here because they can't".
  trusted blocks win this. Or rather--not to be rude--but if you 
 came out and said that this DIP was just some triangulation to 
 get people to accept  trusted blocks, I would say: "good job! 
 It got me thinking!" If not, I'm sorry.
No need to be sorry at all. If, broadly, people are happy about the current state of affairs and see no significant benefit to truth-in-naming, consistent nesting, and safe checking within trusted functions then we'll stick with what we've got.
Jul 25
next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Sunday, 25 July 2021 at 08:16:41 UTC, Bruce Carneal wrote:
 Of the current behavior, Walter's said that he doesn't want 
  trusted blocks because they should be discouraged in  safe 
 code. With this change, we'll have exactly what he doesn't 
 want with different names: s/ trusted/ system/, 
 s/ safe/ trusted/, and the exact same behavior:  system blocks 
 are just what  trusted blocks would've been, and  trusted code 
 with  system blocks in it is just  safe code with a different 
 name.
I think you're misreading Walter on this. He was the one who recommended that I pursue this DIP at beerconf (it was just an idea that I'd thrown out up to that point).
I think it's a fair reading of https://www.youtube.com/watch?v=nGX75xNeW_Y&t=379s From "With this change"&c that's all in my voice.
 I'd like to localize system code and minimize un-automated 
 checking.  An  trusted lambda in an  safe function goes in the 
 other direction.
Presently, when you write a function that you want to be mostly safe but that needs to break those restrictions in part, you call it safe and you put the violation in a trusted lambda. With your proposal, when you write a function that you want to be mostly safe but that needs to break those restrictions in part, you call it trusted and you put the violation in a system block. These seem to me to be so identical that I don't see how they can be moving in opposite directions. You need to audit the same amount of code, and you find that code with the same exertion of effort.
 No need to be sorry at all.  If, broadly, people are happy 
 about the current state of affairs and see no significant 
 benefit to truth-in-naming, consistent nesting, and  safe 
 checking within  trusted functions then we'll stick with what 
 we've got.
I feel like you've started with this problem, " trusted functions don't benefit from safe checking at all", and you've found what seems like a good solution to it, but you haven't stepped back from your solution to realize that you've worked your way back to the existing state of affairs. If someone shows you some code with a long trusted function, and you would like safe checks in it, you can change it to a safe function with trusted lambdas. You can do that right now, and that completely solves the problem of trusted functions not getting checked. And this is the 2-to-1 choice of trusted in Phobos. A new option of "you can leave it as trusted and add system blocks" is just the same option that you already have. (Although if you skip back from the timestamp above, Walter points out that trusted blocks might not get inlined in some cases. A deliberate language feature wouldn't have that problem.)
Jul 25
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 25 July 2021 at 09:04:17 UTC, jfondren wrote:
 On Sunday, 25 July 2021 at 08:16:41 UTC, Bruce Carneal wrote:
 Of the current behavior, Walter's said that he doesn't want 
  trusted blocks because they should be discouraged in  safe 
 code. With this change, we'll have exactly what he doesn't 
 want with different names: s/ trusted/ system/, 
 s/ safe/ trusted/, and the exact same behavior:  system 
 blocks are just what  trusted blocks would've been, and 
  trusted code with  system blocks in it is just  safe code 
 with a different name.
I think you're misreading Walter on this. He was the one who recommended that I pursue this DIP at beerconf (it was just an idea that I'd thrown out up to that point).
I think it's a fair reading of https://www.youtube.com/watch?v=nGX75xNeW_Y&t=379s From "With this change"&c that's all in my voice.
It may well have been a fair reading of the past. I reported on the present (yesterday).
 I'd like to localize system code and minimize un-automated 
 checking.  An  trusted lambda in an  safe function goes in the 
 other direction.
Presently, when you write a function that you want to be mostly safe but that needs to break those restrictions in part, you call it safe and you put the violation in a trusted lambda. With your proposal, when you write a function that you want to be mostly safe but that needs to break those restrictions in part, you call it trusted and you put the violation in a system block. These seem to me to be so identical that I don't see how they can be moving in opposite directions. You need to audit the same amount of code, and you find that code with the same exertion of effort.
They are similar in some regards, but if trusted lambdas are the only practical option for this type of code going forward, then safe will require manual checking in perpetuity. I'm not saying mitigating tooling or procedures to cover this can not be devised/employed. I am saying that there is a qualitative difference between "the compiler asserts" and "my super-duper-xtra-linguistic-wonder-procedure asserts". Moving towards the former is a good idea. Moving towards the latter is not.
 No need to be sorry at all.  If, broadly, people are happy 
 about the current state of affairs and see no significant 
 benefit to truth-in-naming, consistent nesting, and  safe 
 checking within  trusted functions then we'll stick with what 
 we've got.
I feel like you've started with this problem, " trusted functions don't benefit from safe checking at all", and you've found what seems like a good solution to it, but you haven't stepped back from your solution to realize that you've worked your way back to the existing state of affairs.
The initial motivation was concern over safe practically transitioning from "compiler checkable" to "not checkable (reported) by the compiler".
 If someone shows you some code with a long  trusted function, 
 and you would like  safe checks in it, you can change it to a 
  safe function with  trusted lambdas. You can do that right 
 now, and that completely solves the problem of  trusted 
 functions not getting checked. And this is the 2-to-1 choice of 
  trusted in Phobos. A new option of "you can leave it as 
  trusted and add  system blocks" is just the same option that 
 you already have.
As hopefully understood from my earlier comments, these are, qualitatively, not the same thing. You will still have to check a conversion to a new style trusted function manually of course, no work savings there, but you'd gain something pretty important: the compiler's assertions regarding your remaining safe code might actually mean something. Also, I'm not sure how choices (2-to-1 or any N-to-1) made prior to the availability of an alternative are to be interpreted. What am I missing?
 ...
Finally, thanks for engaging.
Jul 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 25 July 2021 at 12:56:33 UTC, Bruce Carneal wrote:
 As hopefully understood from my earlier comments, these are, 
 qualitatively, not the same thing.  You will still have to 
 check a conversion to a new style  trusted function manually of 
 course, no work savings there, but you'd gain something pretty 
 important: the compiler's assertions regarding your remaining 
  safe code might actually mean something.
Memory safety is a global property. If even a single line of your new-style ` system`-block (or old-style ` trusted` lambda) causes undefined behavior, it does not matter one bit what the compiler asserts about the ` safe` code in your program: the entire process is corrupted.
Jul 25
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 25 July 2021 at 13:42:52 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 12:56:33 UTC, Bruce Carneal wrote:
 As hopefully understood from my earlier comments, these are, 
 qualitatively, not the same thing.  You will still have to 
 check a conversion to a new style  trusted function manually 
 of course, no work savings there, but you'd gain something 
 pretty important: the compiler's assertions regarding your 
 remaining  safe code might actually mean something.
Memory safety is a global property. If even a single line of your new-style ` system`-block (or old-style ` trusted` lambda) causes undefined behavior, it does not matter one bit what the compiler asserts about the ` safe` code in your program: the entire process is corrupted.
I do not know of any competent programmer who would say otherwise. I also do not know what this has to do with a discussion regarding debasing/improving safe. What am I missing?
Jul 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 25 July 2021 at 14:19:47 UTC, Bruce Carneal wrote:
 On Sunday, 25 July 2021 at 13:42:52 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 12:56:33 UTC, Bruce Carneal wrote:
 As hopefully understood from my earlier comments, these are, 
 qualitatively, not the same thing.  You will still have to 
 check a conversion to a new style  trusted function manually 
 of course, no work savings there, but you'd gain something 
 pretty important: the compiler's assertions regarding your 
 remaining  safe code might actually mean something.
Memory safety is a global property. If even a single line of your new-style ` system`-block (or old-style ` trusted` lambda) causes undefined behavior, it does not matter one bit what the compiler asserts about the ` safe` code in your program: the entire process is corrupted.
I do not know of any competent programmer who would say otherwise. I also do not know what this has to do with a discussion regarding debasing/improving safe. What am I missing?
It is a response to the claim that "the compiler's assertions regarding your remaining safe code might actually mean something." They mean exactly the same thing with your proposal as they do without it: that the ` safe` portion of the program does not violate the language's memory-safety invariants directly.
Jul 25
parent Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 25 July 2021 at 14:36:27 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 14:19:47 UTC, Bruce Carneal wrote:
 On Sunday, 25 July 2021 at 13:42:52 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 12:56:33 UTC, Bruce Carneal wrote:
 As hopefully understood from my earlier comments, these are, 
 qualitatively, not the same thing.  You will still have to 
 check a conversion to a new style  trusted function manually 
 of course, no work savings there, but you'd gain something 
 pretty important: the compiler's assertions regarding your 
 remaining  safe code might actually mean something.
Memory safety is a global property. If even a single line of your new-style ` system`-block (or old-style ` trusted` lambda) causes undefined behavior, it does not matter one bit what the compiler asserts about the ` safe` code in your program: the entire process is corrupted.
I do not know of any competent programmer who would say otherwise. I also do not know what this has to do with a discussion regarding debasing/improving safe. What am I missing?
It is a response to the claim that "the compiler's assertions regarding your remaining safe code might actually mean something." They mean exactly the same thing with your proposal as they do without it: that the ` safe` portion of the program does not violate the language's memory-safety invariants directly.
They don't mean the same thing to me. In one case safe invariants are asserted with a process that limits human error to group-visible forms (compiler errors in the code handling safe). In the other case we add in a more direct form of human error, the failure to correctly review " safe" code that contains trusted lambdas. In one case, your safe code comes in one flavor, machine checkable. In the other case your safe code comes in two flavors, machine checkable and needs-human-intervention since you have no opportunity to segregate to the new trusted form.
Jul 25
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/25/2021 1:16 AM, Bruce Carneal wrote:
 He was the one who recommended that I 
 pursue this DIP at beerconf (it was just an idea that I'd thrown out up to
that 
 point).
That's right. I did not think of the case of removing the system block from a trusted function and then silently losing the safety checking in the rest of the trusted function. Thanks for pointing it out. One solution would be for the compiler to check such trusted functions for safe-ty anyway, and issue an error for trusted functions with no system{} blocks and no safe errors. The user could silence this error by adding an empty system block: system { }
Jul 25
parent reply ShadoLight <ettienne.gilbert gmail.com> writes:
On Sunday, 25 July 2021 at 12:01:01 UTC, Walter Bright wrote:
 One solution would be for the compiler to check such  trusted 
 functions for  safe-ty anyway, and issue an error for  trusted 
 functions with no  system{} blocks and no  safe errors.

 The user could silence this error by adding an empty  system 
 block:

      system { }
...or rather simply marked as a safe function then?
Jul 26
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/26/2021 1:21 AM, ShadoLight wrote:
 On Sunday, 25 July 2021 at 12:01:01 UTC, Walter Bright wrote:
 The user could silence this error by adding an empty  system block:

      system { }
...or rather simply marked as a safe function then?
It's still possible that the programmer wants to keep it trusted because it's doing things not catchable by the safety checks.
Jul 26
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Sunday, 25 July 2021 at 06:13:41 UTC, jfondren wrote:
 On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:
 The presence of one or more  system blocks would enable  safe 
 checking elsewhere in the enclosing  trusted function.
This has an unfortunate result: if in maintenance you edit a trusted function to remove its system blocks, it'll quietly no longer be checked as safe.
Yes, but there is no point in a ` trusted` function without ` system` blocks. The compiler should warn about it so it gets changed (but should not ever be deprecated).
 Of the current behavior, Walter's said that he doesn't want 
  trusted blocks because they should be discouraged in  safe 
 code. With this change, we'll have exactly what he doesn't want 
 with different names: s/ trusted/ system/, s/ safe/ trusted/, 
 and the exact same behavior:  system blocks are just what 
  trusted blocks would've been, and  trusted code with  system 
 blocks in it is just  safe code with a different name.
The name is the important change. A ` safe` function shouldn't require manual verification. A ` trusted` function does. The ` safe` function with ` trusted` escape gives the impression that the ` safe` parts don't need checking, but they do. It's also difficult to use tools to check which parts need review when focusing on memory safety.
 Instead of people accepting that  safe "isn't really  safe" in 
 the presence of  trusted blocks, and that the whole body of the 
 function has to be audited, with this change we'll have 
 s/ safe/ trusted/ blocks that aren't really  safe in the 
 presence of  system blocks, and that the whole body of the 
 function has to be audited. The "you have to audit this" 
 signifier is the same, an internal lower-protection block, and 
 all that's gained is that the function attribute's spelled 
 differently. Is this really worth it?
Yes.
 One way to avoid the "unfortunate result" above is to permit 
  trusted blocks in  safe code. Which results in Rust equivalent 
 functionality:  safe code is checked as  safe,  safe code with 
 internal  trusted blocks is still checked as  safe but people 
 know to audit it, and  system code isn't checked as  safe.
We already have that with ` trusted` lambdas today. This just changes the name to identify which parts are actually trusted and need review.
 I appreciate that there's a vision also to 
  safe/ trusted/ system, but it doesn't seem to have stuck, with 
 Phobos having more than twice as many  trusted lambdas than 
  trusted functions:

 ```
 phobos$ grep -rP '\W\(\)  trusted' --include '*.d'|wc -l
 238
 phobos$ grep -rP '\w\(\)  trusted' --include '*.d'|wc -l
 111
 ```
` trusted` lambdas are required inside templates where you want code to be ` safe` when the type parameters allow it. The resulting function itself needs to be marked ` trusted` with the lambdas replaced with ` system` blocks to achieve the same effect, as memory safety review is still needed. -Steve
Jul 25
next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Sunday, 25 July 2021 at 12:05:10 UTC, Steven Schveighoffer 
wrote:
 On Sunday, 25 July 2021 at 06:13:41 UTC, jfondren wrote:
 Instead of people accepting that  safe "isn't really  safe" in 
 the presence of  trusted blocks, and that the whole body of 
 the function has to be audited, with this change we'll have 
 s/ safe/ trusted/ blocks that aren't really  safe in the 
 presence of  system blocks, and that the whole body of the 
 function has to be audited. The "you have to audit this" 
 signifier is the same, an internal lower-protection block, and 
 all that's gained is that the function attribute's spelled 
 differently. Is this really worth it?
Yes.
OK. I'll argue the opposite position for a bit, then. Here's a trusted function with a non- safe component: ```d ulong getAvailableDiskSpace(scope const(char)[] path) trusted { ULARGE_INTEGER freeBytesAvailable; auto err = GetDiskFreeSpaceExW(path.tempCStringW(), &freeBytesAvailable, null, null); cenforce(err != 0, "Cannot get available disk space"); return freeBytesAvailable.QuadPart; } ``` With this proposal, I imagine: ```d ulong getAvailableDiskSpace(scope const(char)[] path) trusted { ULARGE_INTEGER freeBytesAvailable; auto err = system GetDiskFreeSpaceExW(path.tempCStringW(), &freeBytesAvailable, null, null); // expression usage? system{ auto err = GetDiskFreeSpaceExW(path.tempCStringW(), &freeBytesAvailable, null, null); } // scopeless block? cenforce(err != 0, "Cannot get available disk space"); return freeBytesAvailable.QuadPart; } ``` And in current practice: ```d ulong getAvailableDiskSpace(scope const(char)[] path) safe { ULARGE_INTEGER freeBytesAvailable; auto err = () trusted { return GetDiskFreeSpaceExW(path.tempCStringW(), &freeBytesAvailable, null, null); } (); cenforce(err != 0, "Cannot get available disk space"); return freeBytesAvailable.QuadPart; } ``` So a naive take is "the last two versions are literally the same", but they become distinct when all three versions are framed by how they came to be, respectively: 1. a trusted function as written by someone trying to properly use the language as described by https://dlang.org/spec/memory-safe-d.html 2. a trusted function as written after that document is updated to reflect this DIP. 3. a trusted function in the current " trusted functions are bad because they don't check anything, avoid them" fail state, which is ugly because it's a late adaptation to a fault in the language rather than a result of deliberate design. In the first case, safe/ trusted/ system functions are dutifully written and then bugs are later found in the trusted functions. trusted functions are an attractive nuisance. In the second case, safe/ trusted/ system functions are written and that doesn't happen. trusted functions have been rehabilitated and have their intended role. In the last case, only safe/ system functions are written, and some of the safe functions are secretly, really, trusted functions that have be written in a weird way to work. trusted functions are either a mistake or a shorthand for a safe function whose entire body may as well be in a trusted block. Rather than bless the failure state by giving it a better syntax (and continuing to have trusted as a problem for new programmers), we'd like to fix trusted so that trusted functions are worth writing again. Does that sound about right?
Jul 25
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On Sunday, 25 July 2021 at 13:14:20 UTC, jfondren wrote:
 OK. I'll argue the opposite position for a bit, then.

 Here's a  trusted function with a non- safe component:

 ```d
 ulong getAvailableDiskSpace(scope const(char)[] path)  trusted
 {
     ULARGE_INTEGER freeBytesAvailable;
     auto err = GetDiskFreeSpaceExW(path.tempCStringW(), 
 &freeBytesAvailable, null, null);
     cenforce(err != 0, "Cannot get available disk space");
     return freeBytesAvailable.QuadPart;
 }
 ```
Yep. This function today is overly trusted (meaning that parts that can be at least partly mechanically checked are allowed to be checked.
 With this proposal, I imagine:

 ```d
 ulong getAvailableDiskSpace(scope const(char)[] path)  trusted
 {
     ULARGE_INTEGER freeBytesAvailable;
     auto err =  system GetDiskFreeSpaceExW(path.tempCStringW(), 
 &freeBytesAvailable, null, null);  // expression usage?
      system{ auto err = 
 GetDiskFreeSpaceExW(path.tempCStringW(), &freeBytesAvailable, 
 null, null); }  // scopeless block?
     cenforce(err != 0, "Cannot get available disk space");
     return freeBytesAvailable.QuadPart;
 }
 ```
Yes, that's about right. The exact semantics are TBD (scope or no scope, expressions or statements, etc.). [snip]
 Does that sound about right?
I think all of what you are saying is along the same lines as what I'm thinking (though I look at it more as pragmatic reasoning for how to write such functions rather than some "blessed" way to do things). -Steve
Jul 26
prev sibling next sibling parent Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Sunday, 25 July 2021 at 12:05:10 UTC, Steven Schveighoffer 
wrote:
 The name is the important change. A ` safe` function shouldn't 
 require manual verification. A ` trusted` function does. The 
 ` safe` function with ` trusted` escape gives the impression 
 that the ` safe` parts don't need checking, but they do. It's 
 also difficult to use tools to check which parts need review 
 when focusing on memory safety.
Yes, I agree. First I was skeptical about this DIP, but that I think is the important advantage above trusted blocks. So I would support this DIP.
Jul 26
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Sunday, 25 July 2021 at 12:05:10 UTC, Steven Schveighoffer 
wrote:
 On Sunday, 25 July 2021 at 06:13:41 UTC, jfondren wrote:
 I appreciate that there's a vision also to 
  safe/ trusted/ system, but it doesn't seem to have stuck, 
 with Phobos having more than twice as many  trusted lambdas 
 than  trusted functions:

 ```
 phobos$ grep -rP '\W\(\)  trusted' --include '*.d'|wc -l
 238
 phobos$ grep -rP '\w\(\)  trusted' --include '*.d'|wc -l
 111
 ```
` trusted` lambdas are required inside templates where you want code to be ` safe` when the type parameters allow it. The resulting function itself needs to be marked ` trusted` with the lambdas replaced with ` system` blocks to achieve the same effect, as memory safety review is still needed.
One thing that is incredibly important to get right here is the way to specify what to do with templates. I've been thinking about this a bit, and even this isn't the right way to do things (specify the function as ` trusted` and mark the actual trusted part as ` system`), because you want the compiler to infer ` system` for certain instantiations, and ` trusted` for others. What we need is a way to say "this function should be ` trusted` iff certain sections of code are inferred ` safe`". Perhaps the correct mechanism here is to encapsulate the ` trusted` parts into separate functions, and make sure the API is ` safe` (e.g. static inner functions). I think it would be too confusing to allow ` system` blocks inside inferred templates. -Steve
Jul 26
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Monday, 26 July 2021 at 13:26:56 UTC, Steven Schveighoffer 
wrote:
 On Sunday, 25 July 2021 at 12:05:10 UTC, Steven Schveighoffer 
 wrote:
 On Sunday, 25 July 2021 at 06:13:41 UTC, jfondren wrote:
 I appreciate that there's a vision also to 
  safe/ trusted/ system, but it doesn't seem to have stuck, 
 with Phobos having more than twice as many  trusted lambdas 
 than  trusted functions:

 ```
 phobos$ grep -rP '\W\(\)  trusted' --include '*.d'|wc -l
 238
 phobos$ grep -rP '\w\(\)  trusted' --include '*.d'|wc -l
 111
 ```
` trusted` lambdas are required inside templates where you want code to be ` safe` when the type parameters allow it. The resulting function itself needs to be marked ` trusted` with the lambdas replaced with ` system` blocks to achieve the same effect, as memory safety review is still needed.
One thing that is incredibly important to get right here is the way to specify what to do with templates. I've been thinking about this a bit, and even this isn't the right way to do things (specify the function as ` trusted` and mark the actual trusted part as ` system`), because you want the compiler to infer ` system` for certain instantiations, and ` trusted` for others.
Yes. You have argued, persuasively, that the language should infer these properties and others. IIUC, in a pervasive inference scenario, safe/ trusted/ nogc/... annotations would function as programmer assertions: "if this property doesn't hold, error out". Couple this with Ali's trusted-by-default proposal and you'd be living in a wonderful world (he says before walking off in to the unknown! :-) ).
 What we need is a way to say "this function should be 
 ` trusted` iff certain sections of code are inferred ` safe`".

 Perhaps the correct mechanism here is to encapsulate the 
 ` trusted` parts into separate functions, and make sure the API 
 is ` safe` (e.g. static inner functions).

 I think it would be too confusing to allow ` system` blocks 
 inside inferred templates.
Definitely more to work out, but I now think that we have an opportunity to achieve a much bigger win than that implied by the initial proposal. It feels like there is a simple path forward patiently awaiting discovery.
Jul 26
parent Bruce Carneal <bcarneal gmail.com> writes:
On Monday, 26 July 2021 at 14:18:29 UTC, Bruce Carneal wrote:
 On Monday, 26 July 2021 at 13:26:56 UTC, Steven Schveighoffer 
 wrote:
 On Sunday, 25 July 2021 at 12:05:10 UTC, Steven Schveighoffer 
 wrote:
 ...
Yes. You have argued, persuasively, that the language should infer these properties and others. IIUC, in a pervasive inference scenario, safe/ trusted/ nogc/... annotations would function as programmer assertions: "if this property doesn't hold, error out". Couple this with Ali's trusted-by-default proposal and you'd be living in a wonderful world (he says before walking off in to the unknown! :-) ).
Rather, "if this compiler can't prove that the property holds, error out".
Jul 26
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:
 At beerconf I committed to putting forward a DIP regarding a 
 new syntactic element to be made available within  trusted 
 functions, the  system block.  The presence of one or more 
  system blocks would enable  safe checking elsewhere in the 
 enclosing  trusted function.
Both before and after this proposal, there are 3 kinds of code: 1. Code that is automatically checked for memory safety. 2. Code that is assumed by the compiler to be safe, and must be manually checked for memory safety. 3. Code that is not checked for memory safety, and is assumed to be unsafe. Currently, (1) is marked ` safe`, (2) is marked ` trusted`, and (3) is marked ` system`. Under this proposal, (1) would be marked either ` safe` or ` trusted`, (2) would be marked either ` trusted` or ` system`, and (3) would be marked ` system`. I do not think this is an improvement relative to the status quo.
 The problematic  trusted lambda escapes creeping in to " safe" 
 code could be replaced going forward by a more honestly named 
 form,  trusted code with  system blocks. Best practices could 
 evolve to the point that  safe actually meant  safe again.
What makes ` trusted` lambdas problematic is that they implicitly depend on everything in their enclosing scope, which makes it easy for a change in the ` safe` portion of the code to accidentally violate an assumption that the ` trusted` portion depends on. If we want to address this issue at the language level, the most direct solution is to require that ` trusted` lambdas make their dependencies explicit. This can be done by requiring all ` trusted` nested functions (of which lambdas are a special case) to be `static`. Of course, this is a breaking change, so it would require a deprecation process.
Jul 25
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 25 July 2021 at 13:19:55 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:
 At beerconf I committed to putting forward a DIP regarding a 
 new syntactic element to be made available within  trusted 
 functions, the  system block.  The presence of one or more 
  system blocks would enable  safe checking elsewhere in the 
 enclosing  trusted function.
Both before and after this proposal, there are 3 kinds of code: 1. Code that is automatically checked for memory safety. 2. Code that is assumed by the compiler to be safe, and must be manually checked for memory safety. 3. Code that is not checked for memory safety, and is assumed to be unsafe. Currently, (1) is marked ` safe`, (2) is marked ` trusted`, and (3) is marked ` system`. Under this proposal, (1) would be marked either ` safe` or ` trusted`, (2) would be marked either ` trusted` or ` system`, and (3) would be marked ` system`. I do not think this is an improvement relative to the status quo.
There is no getting away from manually checking the system block(s) so (2) would be , IIUC, just as it is now wrt compiler assumptions. The improvements on the status quo include the ability to easily delimit "should check *very* closely" code and the corresponding ability to engage safety checking on any remainder.
 The problematic  trusted lambda escapes creeping in to " safe" 
 code could be replaced going forward by a more honestly named 
 form,  trusted code with  system blocks. Best practices could 
 evolve to the point that  safe actually meant  safe again.
What makes ` trusted` lambdas problematic is that they implicitly depend on everything in their enclosing scope, which makes it easy for a change in the ` safe` portion of the code to accidentally violate an assumption that the ` trusted` portion depends on.
Yes.
 If we want to address this issue at the language level, the 
 most direct solution is to require that ` trusted` lambdas make 
 their dependencies explicit. This can be done by requiring all 
 ` trusted` nested functions (of which lambdas are a special 
 case) to be `static`.
Yes. As Walter has put this, we'd be forcing args/operands to "come in through the front door" rather than sneaking in the back.
Jul 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 25 July 2021 at 13:55:14 UTC, Bruce Carneal wrote:
 The improvements on the status quo include the ability to 
 easily delimit "should check *very* closely" code and the 
 corresponding ability to engage  safety checking on any 
 remainder.
We already have this ability: simply avoid writing ` trusted` code whose safety depends on out-of-band knowledge about ` safe` code, and enforce this practice via code review. As I've discussed previously [1], there is no way to enforce this at the language level, because the language does not (and cannot possibly) know what knowledge ` trusted` code depends on for its memory safety. [1] https://forum.dlang.org/post/auqcjtbbamviembvcaps forum.dlang.org
Jul 25
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 25 July 2021 at 14:13:45 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 13:55:14 UTC, Bruce Carneal wrote:
 The improvements on the status quo include the ability to 
 easily delimit "should check *very* closely" code and the 
 corresponding ability to engage  safety checking on any 
 remainder.
We already have this ability: simply avoid writing ` trusted` code whose safety depends on out-of-band knowledge about ` safe` code, and enforce this practice via code review. As I've discussed previously [1], there is no way to enforce this at the language level, because the language does not (and cannot possibly) know what knowledge ` trusted` code depends on for its memory safety. [1] https://forum.dlang.org/post/auqcjtbbamviembvcaps forum.dlang.org
I'd like to have assistance from the compiler to the maximum extent possible and then conduct the code review(s). Assuming low (near zero) false positives out of the compiler, I'm not sure why one would prefer manual checking when compiler checking was available, but that option is certainly available in both the current and proposed environments.
Jul 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 25 July 2021 at 14:34:27 UTC, Bruce Carneal wrote:
 I'd like to have assistance from the compiler to the maximum 
 extent possible and then conduct the code review(s).  Assuming 
 low (near zero) false positives out of the compiler, I'm not 
 sure why one would prefer manual checking when compiler 
 checking was available, but that option is certainly available 
 in both the current and proposed environments.
Under your proposal, the proportion of code that must be manually-checked vs. compiler-checked does not change at all.
Jul 25
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 25 July 2021 at 14:38:18 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 14:34:27 UTC, Bruce Carneal wrote:
 I'd like to have assistance from the compiler to the maximum 
 extent possible and then conduct the code review(s).  Assuming 
 low (near zero) false positives out of the compiler, I'm not 
 sure why one would prefer manual checking when compiler 
 checking was available, but that option is certainly available 
 in both the current and proposed environments.
Under your proposal, the proportion of code that must be manually-checked vs. compiler-checked does not change at all.
The direct translation of an embedded lambda safe function to the corresponding trusted/ system leaves you with the exact same amount of work at the point of translation wrt that function, without dispute. Machine advantage comes in other forms. Firstly, we now have a properly segregated code base. safe always means 'machine checkable'. Zero procedural trusted code review errors in that now easily identified class. Another possible benefit (hypothesis only here) is the ease with which existing trusted code can be made safer. External clients already believe that you're trusted, so at any time you can become safer, incrementally (shrink the, initially large, system block(s)). More machine checking available along a reduced friction path. TBD. Another possible benefit (bigger leap of faith here) is that the continuously-variable-safety mechanism will encourage safer coding generally by reducing the friction of moving functions the other way, from system code to safer trusted. More machine checking available along a reduced friction path. Again, TBD. Another possible benefit is that all new trusted code would start with a form that defaults to safe (you opt out). I think that defaults matter and that this one would lead to more machine checkability passively,or "by default". No need to wait for the "right" time to move to safe with trusted escape, just evolve the code. Architecture matters more here than the exact forms most would say, and I'd mostly agree, but I believe the forms are important. I also believe that there are other benefits, primarily in terms of human visuals (more easily discerned and more easily localizable system regions) but the above are what come to mind wrt machine checkability. I'm still thinking about it. It's less work if this is destroyed, so keep those sharp critiques coming! :-)
Jul 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 25 July 2021 at 16:29:38 UTC, Bruce Carneal wrote:
 Machine advantage comes in other forms.  Firstly, we now have a 
 properly segregated code base.   safe always means 'machine 
 checkable'.  Zero procedural  trusted code review errors in 
 that now easily identified class.
I have already demonstrated that this is false. Here's my example from the previous thread on this topic, rewritten slightly to use the new-style ` trusted`/` system` syntax from your proposal: ```d module example; size_t favoriteNumber() safe { return 42; } int favoriteElement(ref int[50] array) trusted { // This is memory safe because we know favoriteNumber returns 42 system { return array.ptr[favoriteNumber()]; } } ``` I make the following claims: 1. This code is memory-safe. No matter how you call `favoriteElement`, it will not result in undefined behavior, or allow undefined behavior to occur in ` safe` code. 2. `favoriteNumber` is 100% machine-checkable ` safe` code. 3. Changes to `favoriteNumber` must be manually reviewed in order to ensure they do not result in memory-safety violations. The only way to ensure that ` safe` code never requires manual review is to enforce coding standards that forbid functions like `favoriteElement` from ever being merged in the first place. One example of a set of coding standards that would reject `favoriteElement` is: 1. Every ` system` block must be accompanied by a comment containing a proof (formal or informal) of memory safety. 2. A memory-safety proof for a ` system` block may not rely on any knowledge about symbols defined outside the block other than * knowledge that is implied by their type signatures. * knowledge that is implied by their documentation and/or any standards they are known to conform to. `favoriteElement` satisfies condition (1), but not condition (2)--there is nothing in `favoriteNumber`'s documentation or type signature that guarantees it will return 42. I believe (though I cannot prove) that any set of coding standards capable of rejecting `favoriteElement` under your proposal is also (mutatis mutandis) capable of rejecting `favoriteElement` in the current language. If this were true, then the proposal would be of no value--we could simply implement those coding standards and reap their benefits right away, with no language changes required. If you can give an example of a set of coding standards that rejects `favoriteElement` under your proposal but fails to reject `favoriteElement` in the D language as it currently exists, I would be very interested in seeing it.
Jul 25
next sibling parent reply claptrap <clap trap.com> writes:
On Sunday, 25 July 2021 at 17:47:40 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 16:29:38 UTC, Bruce Carneal wrote:
 Machine advantage comes in other forms.  Firstly, we now have 
 a properly segregated code base.   safe always means 'machine 
 checkable'.  Zero procedural  trusted code review errors in 
 that now easily identified class.
I have already demonstrated that this is false. Here's my example from the previous thread on this topic, rewritten slightly to use the new-style ` trusted`/` system` syntax from your proposal: ```d module example; size_t favoriteNumber() safe { return 42; } int favoriteElement(ref int[50] array) trusted { // This is memory safe because we know favoriteNumber returns 42 system { return array.ptr[favoriteNumber()]; } } ``` I make the following claims: 1. This code is memory-safe. No matter how you call `favoriteElement`, it will not result in undefined behavior, or allow undefined behavior to occur in ` safe` code. 2. `favoriteNumber` is 100% machine-checkable ` safe` code. 3. Changes to `favoriteNumber` must be manually reviewed in order to ensure they do not result in memory-safety violations.
So... Call a safe function and get an int. Use that int to index into an array with bounds checking turned off. Wheres the memory safety bug? In the function that returns the int or in the system code that bypasses bounds checking? So no that doesn't prove what you say it does, it doesn't mean favouriteNumber needs checking, it means the system block needs checking. favouriteNumber knows nothing about the array length, to assume it does or it should is bad design.
Jul 25
next sibling parent jfondren <julian.fondren gmail.com> writes:
On Sunday, 25 July 2021 at 20:36:09 UTC, claptrap wrote:
 So...

 Call a safe function and get an int.
 Use that int to index into an array with bounds checking turned 
 off.

 Wheres the memory safety bug? In the function that returns the 
 int or in the system code that bypasses bounds checking?
The questioner is what's at issue there, not the question. safe function. This reviewer cannot determine from just the changes, to only safe functions, that they won't cause the patched program to develop an out of bounds access. safe/ trusted/ system is not really helping this questioner. access. This reviewer doesn't have to examine safe functions except when tracing the inputs to a system block/function where they isolated the error. safe/ trusted/ system has potentially saved this questioner a lot of time with getting an answer to "Where's the memory safety bug?"
Jul 25
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 25 July 2021 at 20:36:09 UTC, claptrap wrote:
 So no that doesn't prove what you say it does, it doesn't mean 
 favouriteNumber needs checking, it means the  system block 
 needs checking. favouriteNumber knows nothing about the array 
 length, to assume it does or it should is bad design.
Strictly speaking, you're right; it is the ` system` block that needs checking, not `favoriteNumber`. However, any time you change `favoriteNumber`, you have to *re-check* the ` system` block. From a maintenance perspective, this is no different from `favoriteNumber` itself requiring manual checking--if someone submits a PR that changes `favoriteNumber`, and you accept it without any manual review, you risk introducing a memory-safety bug. The same logic applies to ` trusted` lambdas. Strictly speaking, it is the lambda that requires checking, not the surrounding ` safe` code. However, any changes to the surrounding code require you to *re-check* the lambda, so from a maintenance perspective, you must review changes to the ` safe` part just as carefully as changes to the ` trusted` part. The underlying problem in both cases is that the memory safety of the manually-checked code (` system` block/` trusted` lambda) depends on details of the automatically-checked code that are not robust against change.
Jul 25
next sibling parent reply claptrap <clap trap.com> writes:
On Sunday, 25 July 2021 at 21:32:00 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 20:36:09 UTC, claptrap wrote:
 So no that doesn't prove what you say it does, it doesn't mean 
 favouriteNumber needs checking, it means the  system block 
 needs checking. favouriteNumber knows nothing about the array 
 length, to assume it does or it should is bad design.
Strictly speaking, you're right; it is the ` system` block that needs checking, not `favoriteNumber`. However, any time you change `favoriteNumber`, you have to *re-check* the ` system` block. From a maintenance perspective, this is no different from `favoriteNumber` itself requiring manual checking--if someone submits a PR that changes `favoriteNumber`, and you accept it without any manual review, you risk introducing a memory-safety bug.
Im sorry but it's nonsense. You get an OOB error, it points you at the system block, you add bounds checking, job done. Changing favouriteNumber doesnt introduce a bug, the bug was *already* there in the system block. You cant expect favouriteNumber to be responsible for other code doing stupid things with its result.
Jul 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 25 July 2021 at 22:05:26 UTC, claptrap wrote:
 Im sorry but it's nonsense.

 You get an OOB error, it points you at the system block, you 
 add bounds checking, job done.

 Changing favouriteNumber doesnt introduce a bug, the bug was 
 *already* there in the system block.

 You cant expect favouriteNumber to be responsible for other 
 code doing stupid things with its result.
If the bug is "already there", you should be able to write a program that uses the unmodified versions of `favoriteNumber` and `favoriteElement` to cause undefined behavior in ` safe` code. If you cannot, then you must admit that `favoriteElement` is memory safe as-written.
Jul 25
parent reply claptrap <clap trap.com> writes:
On Sunday, 25 July 2021 at 22:43:26 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 22:05:26 UTC, claptrap wrote:
 Im sorry but it's nonsense.

 You get an OOB error, it points you at the system block, you 
 add bounds checking, job done.

 Changing favouriteNumber doesnt introduce a bug, the bug was 
 *already* there in the system block.

 You cant expect favouriteNumber to be responsible for other 
 code doing stupid things with its result.
If the bug is "already there", you should be able to write a program that uses the unmodified versions of `favoriteNumber` and `favoriteElement` to cause undefined behavior in ` safe` code. If you cannot, then you must admit that `favoriteElement` is memory safe as-written.
Consider this... int foo() { return 42; } void bar() { int[2] what; if (foo() == 24) { what.ptr[2] = 100; // BUG } } Your argument the same as saying that bar() is memory safe as written. True, but it's not bug free. The bug is just waiting for the right set of circumstances to come to life and eat your face :) IE. Memory safe as written != bug free
Jul 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 25 July 2021 at 23:34:35 UTC, claptrap wrote:
 
 Your argument the same as saying that bar() is memory safe as 
 written. True, but it's not bug free. The bug is just waiting 
 for the right set of circumstances to come to life and eat your 
 face :)

 IE. Memory safe as written != bug free
Yes; I agree completely. :) The point of the example is to show that the proposal advanced in this thread does not prevent this type of bug from occurring.
Jul 25
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 26.07.21 01:50, Paul Backus wrote:
 On Sunday, 25 July 2021 at 23:34:35 UTC, claptrap wrote:
 Your argument the same as saying that bar() is memory safe as written. 
 True, but it's not bug free. The bug is just waiting for the right set 
 of circumstances to come to life and eat your face :)

 IE. Memory safe as written != bug free
Yes; I agree completely. :) The point of the example is to show that the proposal advanced in this thread does not prevent this type of bug from occurring.
The original claim was that the new feature is a tool that allows the code base to be properly segregated more easily, not that you can't still write incorrect trusted code. If you have to review safe code to ensure memory safety of your trusted code, your trusted code is incorrect. Note that the trusted lambda idiom is _basically always_ incorrect trusted code. Some people do it anyway, because it's convenient. The new feature allows combined convenience and correctness.
Jul 25
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 26 July 2021 at 03:40:55 UTC, Timon Gehr wrote:
 The original claim was that the new feature is a tool that 
 allows the code base to be properly segregated more easily, not 
 that you can't still write incorrect  trusted code. If you have 
 to review  safe code to ensure memory safety of your  trusted 
 code, your  trusted code is incorrect.
trusted code is correct if and only if it cannot possibly allow undefined behavior to be invoked in safe code. If my example is incorrect as-written, then you should be able to write a program that uses it, without modification, to cause undefined behavior in safe code. Same for any given trusted lambda.
Jul 26
parent reply claptrap <clap trap.com> writes:
On Monday, 26 July 2021 at 07:32:24 UTC, Paul Backus wrote:
 On Monday, 26 July 2021 at 03:40:55 UTC, Timon Gehr wrote:
 The original claim was that the new feature is a tool that 
 allows the code base to be properly segregated more easily, 
 not that you can't still write incorrect  trusted code. If you 
 have to review  safe code to ensure memory safety of your 
  trusted code, your  trusted code is incorrect.
trusted code is correct if and only if it cannot possibly allow undefined behavior to be invoked in safe code.
Your example doesn't invoke undefined behaviour in safe code, it invokes undefined behaviour in system code. The UB is in the system block. The memory corruption happens in the system block. After that all bets are off. There's no way around that, which makes your example moot.
 If my example is incorrect as-written, then you should be able 
 to write a program that uses it, without modification, to cause 
 undefined behavior in  safe code. Same for any given  trusted 
 lambda.
And that proves what? That you can write buggy system code that doesn't cause memory errors in some circumstances?
Jul 26
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 26 July 2021 at 09:39:57 UTC, claptrap wrote:
 On Monday, 26 July 2021 at 07:32:24 UTC, Paul Backus wrote:
  trusted code is correct if and only if it cannot possibly 
 allow undefined behavior to be invoked in  safe code.
Your example doesn't invoke undefined behaviour in safe code, it invokes undefined behaviour in system code. The UB is in the system block. The memory corruption happens in the system block. After that all bets are off.
Well, it is in a ` trusted` function, which is callable from ` safe` code, so any undefined behavior in the ` system` block is also possible undefined behavior in ` safe` code. If you can write a call to `favoriteElement` from ` safe` code that causes UB, that would be sufficient to demonstrate that it is not memory safe. Of course, it only counts as a mistake in my example if you use the version I wrote, not your own modified version. :)
Jul 26
parent reply claptrap <clap trap.com> writes:
On Monday, 26 July 2021 at 13:58:46 UTC, Paul Backus wrote:
 On Monday, 26 July 2021 at 09:39:57 UTC, claptrap wrote:
 On Monday, 26 July 2021 at 07:32:24 UTC, Paul Backus wrote:
  trusted code is correct if and only if it cannot possibly 
 allow undefined behavior to be invoked in  safe code.
Your example doesn't invoke undefined behaviour in safe code, it invokes undefined behaviour in system code. The UB is in the system block. The memory corruption happens in the system block. After that all bets are off.
Well, it is in a ` trusted` function, which is callable from ` safe` code, so any undefined behavior in the ` system` block is also possible undefined behavior in ` safe` code. If you can write a call to `favoriteElement` from ` safe` code that causes UB, that would be sufficient to demonstrate that it is not memory safe. Of course, it only counts as a mistake in my example if you use the version I wrote, not your own modified version. :)
Its a pointless exercise because your example is a red herring, but this breaks it. ```d ref int[50] ohYeah() trusted { return *(cast(int[50]*) 123456); } int redHerring() safe { return favoriteElement(ohYeah()); } ``` See what we've learnt? That trusted / system etc can break safety. Top down, or bottom up. Its a house of cards, you cant escape that, unless you get rid of trusted and system completely. Who knew? :) Another way to think about it is that you are conflating "memory safe by convention" with "memory safe by compiler". Your example is memory safe by convention, IE, it is only so because the programmer has checked favouriteNumber doesnt break favouriteElement. So when you change favouriteNumber to 52 you are violating "safe by convention". You are not violating "safe by compiler". IE when you change favouriteNumber it doesnt change anything in terms of what is checked by the compiler. Or in terms of runtime checks. At least AFAIK. I wonder what it is you expect the compiler to do. And TBH I'm not even sure what your overall point is.
Jul 26
parent reply jfondren <julian.fondren gmail.com> writes:
On Monday, 26 July 2021 at 16:26:53 UTC, claptrap wrote:
 Its a pointless exercise because your example is a red herring, 
 but this breaks it.
...
 And TBH I'm not even sure what your overall point is.
It's a response to overly strong claims about what this DIP will achieve: https://forum.dlang.org/post/fnhvydmbguyagcmaepih forum.dlang.org
 It is a response to the claim that "the compiler's assertions 
 regarding your remaining  safe code might actually mean 
 something." They mean exactly the same thing with your proposal 
 as they do without it: that the  safe portion of the program 
 does not violate the language's memory-safety invariants 
 directly.
And it's again from the perspective of someone reviewing a patch rather than someone troubleshooting a bug. *Once there is a memory error in your program*, then safe helps you by telling you to look elsewhere for the direct violation. If you are reviewing patches with an eye towards not including a memory error in your program, then safe matters a lot less.
Jul 26
parent reply claptrap <clap trap.com> writes:
On Monday, 26 July 2021 at 16:45:07 UTC, jfondren wrote:
 On Monday, 26 July 2021 at 16:26:53 UTC, claptrap wrote:
 Its a pointless exercise because your example is a red 
 herring, but this breaks it.
...
 And TBH I'm not even sure what your overall point is.
It's a response to overly strong claims about what this DIP will achieve: https://forum.dlang.org/post/fnhvydmbguyagcmaepih forum.dlang.org
 It is a response to the claim that "the compiler's assertions 
 regarding your remaining  safe code might actually mean 
 something." They mean exactly the same thing with your 
 proposal as they do without it: that the  safe portion of the 
 program does not violate the language's memory-safety 
 invariants directly.
If you're saying the proposed "system blocks inside trusted functions" provide no advantage over teh current "trusted lambdas inside safe functions" yes thats true. But I think the point is trusted functions get more checking. Even if you say well you can achieve the same by just using a trusted lambda inside a safe function its not the same once you consider what people actually do. If you have just one trusted function in your app, then switching to this new regime would automatically give you more checking. You have to take into account how people will actually behave, even if you can technically achieve the same thing with the current system.
 And it's again from the perspective of someone reviewing a 
 patch rather than someone troubleshooting a bug. *Once there is 
 a memory error in your program*, then  safe helps you by 
 telling you to look elsewhere for the direct violation. If you 
 are reviewing patches with an eye towards not including a 
 memory error in your program, then  safe matters a lot less.
This is still mischaracterizing the problem, if you add safe code and it causes a memory error to occur in trusted or system code, the problem was already there. You're not adding a memory error, just changing the conditions so it is triggered. It would be nice to have a system that would help with problems like that but I think its actually unreasonable, and probably impossible. And it's probably counterproductive to use examples like that to guide the design process. You're designing for constraints that cant be met.
Jul 28
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 28 July 2021 at 08:40:47 UTC, claptrap wrote:
 If you're saying the proposed "system blocks inside trusted 
 functions" provide no advantage over teh current "trusted 
 lambdas inside safe functions" yes thats true. But I think the 
 point is trusted functions get more checking. Even if you say 
 well you can achieve the same by just using a trusted lambda 
 inside a safe function its not the same once you consider what 
 people actually do.

 If you have just one trusted function in your app, then 
 switching to this new regime would automatically give you more 
 checking.

 You have to take into account how people will actually behave, 
 even if you can technically achieve the same thing with the 
 current system.
If I understand correctly, there are two problems being diagnosed in this discussion (overall--not just in your post), and two solutions being proposed: mistakes, and allow too much code to go without automatic checks. as it currently exists. Instead, automatic safety checks will only be disabled in specially-marked blocks (like in Rust). This will encourage programmers to disable automatic checking only for the specific lines of code where it is actually necessary, rather than for the entire function. and maintain because they have implicit dependencies on the surrounding ` safe` code. programmers to put their ` trusted` code into separate functions, which communicate with ` safe` code explicitly via arguments and return values. (See: [Walter's PR][1]). There is logic to both of these proposals, but their solutions conflict: one pushes for less use of function-level ` trusted`, the other for more use of it. [1]: https://github.com/dlang/dlang.org/pull/3077
Jul 28
next sibling parent claptrap <clap trap.com> writes:
On Wednesday, 28 July 2021 at 12:49:28 UTC, Paul Backus wrote:
 On Wednesday, 28 July 2021 at 08:40:47 UTC, claptrap wrote:
 If I understand correctly, there are two problems being 
 diagnosed in this discussion (overall--not just in your post), 
 and two solutions being proposed:


 mistakes, and allow too much code to go without automatic 
 checks.


 ` trusted` as it currently exists. Instead, automatic safety 
 checks will only be disabled in specially-marked blocks (like 
 in Rust). This will encourage programmers to disable automatic 
 checking only for the specific lines of code where it is 
 actually necessary, rather than for the entire function.


 and maintain because they have implicit dependencies on the 
 surrounding ` safe` code.


 programmers to put their ` trusted` code into separate 
 functions, which communicate with ` safe` code explicitly via 
 arguments and return values. (See: [Walter's PR][1]).

 There is logic to both of these proposals, but their solutions 
 conflict: one pushes for less use of function-level ` trusted`, 
 the other for more use of it.

 [1]: https://github.com/dlang/dlang.org/pull/3077
Agree 100%. Maybe you can just ban trusted lambdas from having access to the surrounding scope? Or ban unsafe blocks from having access to the surrounding scope, but then you need to have a way to pass stuff in, maybe like... system (int[] foo, float bar) { ... } unsafe blocks could be lowered to a trusted lambda maybe?
Jul 28
prev sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 28 July 2021 at 12:49:28 UTC, Paul Backus wrote:
 On Wednesday, 28 July 2021 at 08:40:47 UTC, claptrap wrote:
 If you're saying the proposed "system blocks inside trusted 
 functions" provide no advantage over teh current "trusted 
 lambdas inside safe functions" yes thats true. But I think the 
 point is trusted functions get more checking. Even if you say 
 well you can achieve the same by just using a trusted lambda 
 inside a safe function its not the same once you consider what 
 people actually do.

 If you have just one trusted function in your app, then 
 switching to this new regime would automatically give you more 
 checking.

 You have to take into account how people will actually behave, 
 even if you can technically achieve the same thing with the 
 current system.
If I understand correctly, there are two problems being diagnosed in this discussion (overall--not just in your post), and two solutions being proposed: mistakes, and allow too much code to go without automatic checks. ` trusted` as it currently exists. Instead, automatic safety checks will only be disabled in specially-marked blocks (like in Rust). This will encourage programmers to disable automatic checking only for the specific lines of code where it is actually necessary, rather than for the entire function. and maintain because they have implicit dependencies on the surrounding ` safe` code. programmers to put their ` trusted` code into separate functions, which communicate with ` safe` code explicitly via arguments and return values. (See: [Walter's PR][1]). There is logic to both of these proposals, but their solutions conflict: one pushes for less use of function-level ` trusted`, the other for more use of it. [1]: https://github.com/dlang/dlang.org/pull/3077
I see this as two problems with a common solution, rather than a conflict. Problem 1: trusted lambdas within safe functions are very convenient but tricky. Problem 2: we want more practical safety in the evolving dlang code base with as close to zero additional programming load as we can manage. Rather than post some half-baked evolution of the original proposal sketch here, I'll wait til the back and forth with Joe converges. The current candidates there are, in my view, both simpler and more powerful than anticipated in the forum discussion. The biggest open question is how things should evolve to reduce/eliminate meta programming friction (thanks for the caution Steven). Unless we unexpectedly find ourselves with a lot of time on our hands, ETA on the DIP is still the end of the year.
Jul 28
parent reply claptrap <clap trap.com> writes:
On Wednesday, 28 July 2021 at 16:57:41 UTC, Bruce Carneal wrote:
 On Wednesday, 28 July 2021 at 12:49:28 UTC, Paul Backus wrote:
 On Wednesday, 28 July 2021 at 08:40:47 UTC, claptrap wrote:
I see this as two problems with a common solution, rather than a conflict. Problem 1: trusted lambdas within safe functions are very convenient but tricky. Problem 2: we want more practical safety in the evolving dlang code base with as close to zero additional programming load as we can manage. Rather than post some half-baked evolution of the original proposal sketch here, I'll wait til the back and forth with Joe converges. The current candidates there are, in my view, both simpler and more powerful than anticipated in the forum discussion. The biggest open question is how things should evolve to reduce/eliminate meta programming friction (thanks for the caution Steven). Unless we unexpectedly find ourselves with a lot of time on our hands, ETA on the DIP is still the end of the year.
Do you have ideas on how to stop unsafe blocks accessing the variables from the surrounding scope? Is that even a goal for the DIP?
Jul 28
next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 28 July 2021 at 17:25:18 UTC, claptrap wrote:
 On Wednesday, 28 July 2021 at 16:57:41 UTC, Bruce Carneal wrote:
 [...]
Do you have ideas on how to stop unsafe blocks accessing the variables from the surrounding scope? Is that even a goal for the DIP?
Yes. The form and scope of the unsafe block(s) is under discussion, with safety and readability in play.
Jul 28
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 28 July 2021 at 17:25:18 UTC, claptrap wrote:
 Do you have ideas on how to stop unsafe blocks accessing the 
 variables from the surrounding scope? Is that even a goal for 
 the DIP?
I'm not sure it necessarily is. Consider the following example (using the proposed trusted-with- system-blocks syntax): ```D /// Writes something into the provided buffer, e.g. filling the /// buffer with random bytes extern(C) void writeIntoCBuffer (int* ptr, size_t len) system; void writeIntoDBuffer (ref int[] buf) trusted { system { writeIntoCBuffer(buf.ptr, buf.length); } } ``` That seems like a reasonable use-case for a trusted wrapper of an underlying system function, but if the system block was forbidden from accessing variables from the surrounding scope, it wouldn't be possible. Does that make sense, or have I misunderstood what you had in mind?
Jul 29
next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 29 July 2021 at 08:16:08 UTC, Joseph Rushton 
Wakeling wrote:
 On Wednesday, 28 July 2021 at 17:25:18 UTC, claptrap wrote:
 Do you have ideas on how to stop unsafe blocks accessing the 
 variables from the surrounding scope? Is that even a goal for 
 the DIP?
I'm not sure it necessarily is. Consider the following example (using the proposed trusted-with- system-blocks syntax): ```D /// Writes something into the provided buffer, e.g. filling the /// buffer with random bytes extern(C) void writeIntoCBuffer (int* ptr, size_t len) system; void writeIntoDBuffer (ref int[] buf) trusted { system { writeIntoCBuffer(buf.ptr, buf.length); } } ``` That seems like a reasonable use-case for a trusted wrapper of an underlying system function, but if the system block was forbidden from accessing variables from the surrounding scope, it wouldn't be possible. Does that make sense, or have I misunderstood what you had in mind?
Yes, I was a bit sloppy earlier. Full "stopping" is a non-goal. There are, however, various restrictions and syntactic forms to be considered that usefully differ from the full access scopeless variant.
Jul 29
prev sibling parent reply claptrap <clap trap.com> writes:
On Thursday, 29 July 2021 at 08:16:08 UTC, Joseph Rushton 
Wakeling wrote:
 On Wednesday, 28 July 2021 at 17:25:18 UTC, claptrap wrote:
 Do you have ideas on how to stop unsafe blocks accessing the 
 variables from the surrounding scope? Is that even a goal for 
 the DIP?
I'm not sure it necessarily is. Consider the following example (using the proposed trusted-with- system-blocks syntax): ```D /// Writes something into the provided buffer, e.g. filling the /// buffer with random bytes extern(C) void writeIntoCBuffer (int* ptr, size_t len) system; void writeIntoDBuffer (ref int[] buf) trusted { system { writeIntoCBuffer(buf.ptr, buf.length); } } ``` That seems like a reasonable use-case for a trusted wrapper of an underlying system function, but if the system block was forbidden from accessing variables from the surrounding scope, it wouldn't be possible. Does that make sense, or have I misunderstood what you had in mind?
Not exactly, obviously if they cant access variables from the surround scope they'd be useless. But i think the idea (not something i knew about until this thread) is to have a safe api between trusted and system. So there's controlled / restricted access. Otherwise if you have a system block inside a trusted function, the system code could just trash anything it wants from the enclosing scope, which makes any guarantees you have from the code being checked a bit pointless. So it's not just about narrowing down the amount of code that is marked system, but also the amount of state that it can access. IIUC that was the original reason for limiting safe/ trusted/ system to only apply on functions. So it forces you think about API between them. But I guess it didn't work out as expected.
Jul 29
parent Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 29 July 2021 at 13:20:41 UTC, claptrap wrote:
 On Thursday, 29 July 2021 at 08:16:08 UTC, Joseph Rushton 
 Wakeling wrote:
 On Wednesday, 28 July 2021 at 17:25:18 UTC, claptrap wrote:
 Do you have ideas on how to stop unsafe blocks accessing the 
 variables from the surrounding scope? Is that even a goal for 
 the DIP?
I'm not sure it necessarily is. Consider the following example (using the proposed trusted-with- system-blocks syntax): ```D /// Writes something into the provided buffer, e.g. filling the /// buffer with random bytes extern(C) void writeIntoCBuffer (int* ptr, size_t len) system; void writeIntoDBuffer (ref int[] buf) trusted { system { writeIntoCBuffer(buf.ptr, buf.length); } } ``` That seems like a reasonable use-case for a trusted wrapper of an underlying system function, but if the system block was forbidden from accessing variables from the surrounding scope, it wouldn't be possible. Does that make sense, or have I misunderstood what you had in mind?
Not exactly, obviously if they cant access variables from the surround scope they'd be useless. But i think the idea (not something i knew about until this thread) is to have a safe api between trusted and system. So there's controlled / restricted access. Otherwise if you have a system block inside a trusted function, the system code could just trash anything it wants from the enclosing scope, which makes any guarantees you have from the code being checked a bit pointless. So it's not just about narrowing down the amount of code that is marked system, but also the amount of state that it can access. IIUC that was the original reason for limiting safe/ trusted/ system to only apply on functions. So it forces you think about API between them. But I guess it didn't work out as expected.
A design tension that Joseph and I are working with is that between local readability and practical safety. One of the ideas being batted around is to allow parameterless read-only access to the preceding scope from within scoped ncbc (not checked by compiler) blocks. There are other ideas in this area. The DIP ideas are still, quite clearly, in the half-baked category. We (Joseph and I) are considering beerconf updates where we can solicit input on 3/4 baked ideas and open the door to new possibilities in a more productive setting. That said, if lightning strikes and you can't wait til beerconf please feel free to drop me an email. Finally, I've not tightly coordinated with Joseph on this response, so he may have additional/better information to impart.
Jul 29
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/25/2021 8:40 PM, Timon Gehr wrote:
 Note that the  trusted lambda idiom is _basically always_ incorrect  trusted 
 code. Some people do it anyway, because it's convenient.
https://github.com/dlang/dlang.org/pull/3077
Jul 26
prev sibling parent claptrap <clap trap.com> writes:
On Sunday, 25 July 2021 at 23:50:12 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 23:34:35 UTC, claptrap wrote:
 
 Your argument the same as saying that bar() is memory safe as 
 written. True, but it's not bug free. The bug is just waiting 
 for the right set of circumstances to come to life and eat 
 your face :)

 IE. Memory safe as written != bug free
Yes; I agree completely. :) The point of the example is to show that the proposal advanced in this thread does not prevent this type of bug from occurring.
You could probably come up with an example bug that wouldn't be caught by any system you can think of as long as you have an system escape hatch. It doesnt really prove anything imo.
Jul 26
prev sibling parent claptrap <clap trap.com> writes:
On Sunday, 25 July 2021 at 21:32:00 UTC, Paul Backus wrote:
 The underlying problem in both cases is that the memory safety 
 of the manually-checked code (` system` block/` trusted` 
 lambda) depends on details of the automatically-checked code 
 that are not robust against change.
I think the problem is you're conflating memory safety and validity. When you slap safe on favouriteNumber you're not telling the world that it will always return a valid result, you're telling the world that it wont corrupt memory. IE.. safe doesnt mean you can blindly use the result of a function. EG.. if you have a safe version of getchar(), would you blindly use the result of that?
Jul 25
prev sibling next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 25 July 2021 at 17:47:40 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 16:29:38 UTC, Bruce Carneal wrote:
 Machine advantage comes in other forms.  Firstly, we now have 
 a properly segregated code base.   safe always means 'machine 
 checkable'.  Zero procedural  trusted code review errors in 
 that now easily identified class.
I have already demonstrated that this is false. Here's my example from the previous thread on this topic, rewritten slightly to use the new-style ` trusted`/` system` syntax from your proposal: ```d module example; size_t favoriteNumber() safe { return 42; } int favoriteElement(ref int[50] array) trusted { // This is memory safe because we know favoriteNumber returns 42 system { return array.ptr[favoriteNumber()]; } } ``` I make the following claims: 1. This code is memory-safe. No matter how you call `favoriteElement`, it will not result in undefined behavior, or allow undefined behavior to occur in ` safe` code. 2. `favoriteNumber` is 100% machine-checkable ` safe` code. 3. Changes to `favoriteNumber` must be manually reviewed in order to ensure they do not result in memory-safety violations.
OK, I think I see where we're diverging. I never thought that any automated checking could absolve the programmer from actually understanding the code.
 The only way to ensure that ` safe` code never requires manual 
 review is to enforce coding standards that forbid functions 
 like `favoriteElement` from ever being merged in the first 
 place.
There is no way to ensure/prove anything if you are ignorant of the code so per such a framing we can assert, trivially again, that the only way of avoiding "manual review" (reading) is to disallow the code.
 One example of a set of coding standards that would reject 
 `favoriteElement` is:

 ...

 If you can give an example of a set of coding standards that 
 rejects `favoriteElement` under your proposal but fails to 
 reject `favoriteElement` in the D language as it currently 
 exists, I would be very interested in seeing it.
Unless I'm really missing something, nothing interesting is being proven or disproven here beyond the importance of definitions when "proving" things. I'd like to conclude with two points from the beerconf discussion on this proposal, first your excellant observation that trusted might be viewed as the safe <==> system programming analog of "weakly pure": it's where some good degree of localization might occur but where interesting stuff also gets done. That safe actually might be profitably "purified" of the dreaded trusted lambda! :-) Secondly, there was a history-of-the-ideas-embodied-in-the-DIP put together nicely by Joe Wakeling that indicates several possible "origination points" for the ideas. Further posts are welcome but Joe and I are going mostly quiet now, aiming for a concentrated DIP effort very late in the year.
Jul 25
parent reply jfondren <julian.fondren gmail.com> writes:
On Sunday, 25 July 2021 at 21:26:27 UTC, Bruce Carneal wrote:
 Further posts are welcome but Joe and I are going mostly quiet 
 now, aiming for a concentrated DIP effort very late in the year.
Well, thanks, and good luck. I think if your DIP makes strong promises about how it will make the language much safer it is currently, that you'll see these same criticisms again about how potential safety actually remains the same. If you make much more restrained claims, that "The name is the important change.", then any anticipated inconvenience with the rollout will seem to outweigh the benefit. I recommend this justification for the DIP: what you are doing is *rehabilitating trusted functions*, which are currently (for newbies) a bug-filled trap, and (for experts) disused in favor of safe functions containing trusted blocks. The language documentation tells people to use trusted functions but the language in practice doesn't really have them. That's simultaneously a humble change and one that substantially improves the language, vs. alternate proposal like "conform to Rust" that would add to D but leave it still having this safe/ trusted/ system framework that doesn't work as intended. People complain about "half-baked features" and here you would be proposing to bake this feature the rest of the way. That's a lot easier to support, and a more coherent language is something people can think about when they're having to touch a bunch of code as a result of the DIP changing the language.
Jul 25
parent reply ag0aep6g <anonymous example.com> writes:
On Sunday, 25 July 2021 at 23:16:16 UTC, jfondren wrote:
 I recommend this justification for the DIP: what you are doing 
 is *rehabilitating  trusted functions*, which are currently 
 (for newbies) a bug-filled trap, and (for experts) disused in 
 favor of  safe functions containing  trusted blocks. The 
 language documentation tells people to use  trusted functions 
 but the language in practice doesn't really have them.
The language doesn't have trusted blocks. trusted function literals (lambdas) are still functions. Everything the documentation says about trusted functions applies to function literals. People do like to treat trusted function literals as if they were the proposed system blocks. But then they're strictly speaking writing invalid code. They're cheating. And why not? It has significant advantages for the low price of (1) tainting some safe code and (2) having a technically invalid program that works just fine in practice. What the proposal does is turning the common cheat into an official part of the language, a best practice even. And the syntax gets a bit nicer.
Jul 25
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 26.07.21 02:21, ag0aep6g wrote:
 On Sunday, 25 July 2021 at 23:16:16 UTC, jfondren wrote:
 ...
The language doesn't have trusted blocks. trusted function literals (lambdas) are still functions. Everything the documentation says about trusted functions applies to function literals.
The documentation no longer says much about that. https://github.com/dlang/dlang.org/pull/2453#commitcomment-53328593 Probably we should fix that soon.
Jul 25
next sibling parent ag0aep6g <anonymous example.com> writes:
On 26.07.21 05:46, Timon Gehr wrote:
 On 26.07.21 02:21, ag0aep6g wrote:
[...]
 The language doesn't have  trusted blocks.  trusted function literals 
 (lambdas) are still functions. Everything the documentation says about 
  trusted functions applies to function literals.
The documentation no longer says much about that. https://github.com/dlang/dlang.org/pull/2453#commitcomment-53328593 Probably we should fix that soon.
That's just memory-safe-d.dd, which is an odd page that should be merged with function.dd. function.dd still says that trusted functions must have safe interfaces. https://dlang.org/spec/function.html#trusted-functions
Jul 26
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/25/2021 8:46 PM, Timon Gehr wrote:
 The documentation no longer says much about that.
 https://github.com/dlang/dlang.org/pull/2453#commitcomment-53328593
 
 Probably we should fix that soon.
https://github.com/dlang/dlang.org/pull/3076
Jul 26
prev sibling next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Sunday, 25 July 2021 at 17:47:40 UTC, Paul Backus wrote:
 ```d
 module example;

 size_t favoriteNumber()  safe { return 42; }

 int favoriteElement(ref int[50] array)  trusted
 {
     // This is memory safe because we know favoriteNumber 
 returns 42
      system {
         return array.ptr[favoriteNumber()];
     }
 }
 ```
favoriteElement(), all on its own, has an unchecked type error: array is indexed by the int return value of favoriteNumber(), but int has a range outside of the 0..49 type of array's indices. In Ada, array's index type would be specified in the code and you'd have to either perform a checked type conversion to use favoriteNumber() there, or you'd have to change favoriteNumber() to return the index type rather than an int. ```ada with Ada.Text_IO; use Ada.Text_IO; procedure FavElm is subtype Index is Integer range 0 .. 49; type Arg is array (Index) of Integer; function favoriteNumber return Index is (142); function favoriteElement(A : Arg) return Integer is begin return A (favoriteNumber); end favoriteElement; MyArray : Arg := (42 => 5, others => 0); begin Put_Line (Natural'Image (favoriteElement (MyArray))); end FavElm; ``` which compiles with a warning, and fails at runtime as promised: ``` favelm.adb:7:45: warning: value not in range of type "Index" defined at line 4 favelm.adb:7:45: warning: "Constraint_Error" will be raised at run time ``` In a language with dependent types (or the theorem-proving variant of Ada, SPARK) you could get a compile time error, including from functions that return statically unknown values that still have the wrong type, like getchar() as mentioned elsewhere. In such languages the thing you *should* be doing, testing an int's range before using it as an index for an int[50] array, is a compile-time error to not do. You're not forced to check it at every index, but you have to check it at some point. This kind of precision with types isn't so pleasant in D but the class of error is the same and it's something a reviewer could spot when initially checking this code in.
Jul 25
parent reply ag0aep6g <anonymous example.com> writes:
On Monday, 26 July 2021 at 00:08:51 UTC, jfondren wrote:
 On Sunday, 25 July 2021 at 17:47:40 UTC, Paul Backus wrote:
 ```d
 module example;

 size_t favoriteNumber()  safe { return 42; }

 int favoriteElement(ref int[50] array)  trusted
 {
     // This is memory safe because we know favoriteNumber 
 returns 42
      system {
         return array.ptr[favoriteNumber()];
     }
 }
 ```
favoriteElement(), all on its own, has an unchecked type error: array is indexed by the int return value of favoriteNumber(), but int has a range outside of the 0..49 type of array's indices. In Ada, array's index type would be specified in the code and you'd have to either perform a checked type conversion to use favoriteNumber() there, or you'd have to change favoriteNumber() to return the index type rather than an int.
[...]
 This kind of precision with types isn't so pleasant in D but 
 the class of error is the same and it's something a reviewer 
 could spot when initially checking this code in.
What if favoriteNumber originally returns a ubyte, and favoriteElement takes an int[256]? ```d ubyte favoriteNumber() safe { return 42; } int favoriteElement(ref int[256] array) trusted { return array.ptr[favoriteNumber()]; } ``` To your reviewer, there's nothing wrong with favoriteElement, right? But later someone might change the return type of favoriteNumber to size_t and let it return 300. Badaboom: undefined behavior after touching safe code. As far as I can tell, there's no way to truly make it impossible. Maybe disallowing calls to safe functions from trusted and system code would do the trick, but that's impractical.
Jul 25
parent reply jfondren <julian.fondren gmail.com> writes:
On Monday, 26 July 2021 at 00:37:36 UTC, ag0aep6g wrote:
 What if favoriteNumber originally returns a ubyte, and 
 favoriteElement takes an int[256]?

 ```d
 ubyte favoriteNumber()  safe { return 42; }
 int favoriteElement(ref int[256] array)  trusted
 {
     return array.ptr[favoriteNumber()];
 }
 ```

 To your reviewer, there's nothing wrong with favoriteElement, 
 right?

 But later someone might change the return type of 
 favoriteNumber to size_t and let it return 300. Badaboom: 
 undefined behavior after touching  safe code.
That's a much more obviously program-affecting change though, you're changing a function signature. It wouldn't make as compelling an example of someone being surprised that they have to review more than just a safe function when that only that function is changed. If you do name the index type then you can do something like this Nim translation of the Ada: ```nim type Array = array[50, int] Index = range[0..49] var myarray: Array myarray[42] = 5 func favoriteNumber: Index = 42 func favoriteElement(arg: Array): int = let i: Index = favoriteNumber() return arg[i] echo favoriteElement(myarray) ``` (But Nim disappoints here: if you change favoriteNumber to return an int, and then change the number to 142, then Nim doesn't complain at all about this code that assigns an int to a Index variable.)
Jul 25
next sibling parent ag0aep6g <anonymous example.com> writes:
On Monday, 26 July 2021 at 00:50:17 UTC, jfondren wrote:
 That's a much more obviously program-affecting change though, 
 you're changing a function signature. It wouldn't make as 
 compelling an example of someone being surprised that they have 
 to review more than just a  safe function when that only that 
 function is changed.
The point stands: Changes to safe code can compromise memory safety. Bruce claimed we would get "a properly segregated code base", and that safe code would be entirely "machine checkable". But reviewers still have to be on the lookout for safety issues, even when no trusted or system code is touched.
Jul 25
prev sibling parent Araq <rumpf_a web.de> writes:
On Monday, 26 July 2021 at 00:50:17 UTC, jfondren wrote:
 (But Nim disappoints here: if you change favoriteNumber to 
 return an int, and then change the number to 142, then Nim 
 doesn't complain at all about this code that assigns an int to 
 a Index variable.)
Well, it still checks the indexing at runtime, just like Ada, compile-time but then you would complain about the annotation effort and implementation complexity so it would never stop to "disappoint"...
Jul 25
prev sibling next sibling parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Sunday, 25 July 2021 at 17:47:40 UTC, Paul Backus wrote:

 ```d
  system {
    return array.ptr[favoriteNumber()];
 }
 ```

 I make the following claims:

 1. This code is memory-safe.
No, it's not. You use something that is not a literal, so it may change. Even for a constant this should be checked for each new compile. I would do that even if it actually were a literal, but here we have a function call! The definition of the called function may be far away. How would this ever pass a review? The only way to make this memory safe, is to actually test it: ```d system { assert(array.length > favoriteNumber()); return array.ptr[favoriteNumber()]; } ```
Jul 26
parent Paul Backus <snarwin gmail.com> writes:
On Monday, 26 July 2021 at 09:08:05 UTC, Dominikus Dittes Scherkl 
wrote:
 On Sunday, 25 July 2021 at 17:47:40 UTC, Paul Backus wrote:

 ```d
  system {
    return array.ptr[favoriteNumber()];
 }
 ```

 I make the following claims:

 1. This code is memory-safe.
No, it's not. You use something that is not a literal, so it may change.
I agree that *future versions* of the code may not be memory-safe if `favoriteNumber` is changed, but that does not mean the *current version* is unsafe. If you believe that the current version is unsafe, you should be able to demonstrate this unsafety by writing a ` safe` program that uses the current version to cause undefined behavior.
Jul 26
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Sunday, 25 July 2021 at 17:47:40 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 16:29:38 UTC, Bruce Carneal wrote:
 Machine advantage comes in other forms.  Firstly, we now have 
 a properly segregated code base.   safe always means 'machine 
 checkable'.  Zero procedural  trusted code review errors in 
 that now easily identified class.
I have already demonstrated that this is false. Here's my example from the previous thread on this topic, rewritten slightly to use the new-style ` trusted`/` system` syntax from your proposal: ```d module example; size_t favoriteNumber() safe { return 42; } int favoriteElement(ref int[50] array) trusted { // This is memory safe because we know favoriteNumber returns 42 system { return array.ptr[favoriteNumber()]; } } ```
And I have [already demonstrated](https://forum.dlang.org/post/sag7fp$18oj$1 digitalmars.com), `favoriteElement` is invalid. You need to have a definition of what `favoriteNumber` does, and this needs to be reconciled with the definition of `favoriteElement`. In this case, `favoriteElement` is invalid, because `favoriteNumber` has no formal specification. When reviewing `favoriteElement`, one must look at it like a black box: ```d /// Returns: a valid size_t size_t favoriteNumber() safe; int favoriteElement(ref int[50] array) trusted { /* code to review */ } ``` However, with a specification of `favoriteNumber`, `favoriteElement` can be reviewed as correct: ```d /// Returns: a size_t between 0 and 49, inclusive size_t favoriteNumber() safe; ... ``` Now, when reviewing `favoriteElement`, I can prove that it's a valid ` trusted` function given the formal definition of `favoriteNumber`. If reviewing `favoriteNumber`, I can reconcile the implementation changes against the spec for it, and know that as long as I follow the spec, I will not mess up any other code. If the spec changes, now you have to re-review anything that uses it (including other ` safe` code) for correctness. As I said before, code reviews of ` safe` functions are still required for correctness, just not for memory safety. -Steve
Jul 26
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 26 July 2021 at 11:02:48 UTC, Steven Schveighoffer 
wrote:
 However, with a specification of `favoriteNumber`, 
 `favoriteElement` can be reviewed as correct:

 ```d
 /// Returns: a size_t between 0 and 49, inclusive
 size_t favoriteNumber()  safe;

 ...
 ```
If your theory of memory safety leads you to conclude that the presence or absence of a comment can make otherwise-unsafe code memory safe, you have taken a wrong turn somewhere in your reasoning. I agree with you that the version with the comment is better, more maintainable code, and that we should hold our code to such standards in code review. But bad and hard-to-maintain code can still be memory safe (that is: free from possible UB).
Jul 26
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Monday, 26 July 2021 at 13:54:33 UTC, Paul Backus wrote:
 On Monday, 26 July 2021 at 11:02:48 UTC, Steven Schveighoffer 
 wrote:
 However, with a specification of `favoriteNumber`, 
 `favoriteElement` can be reviewed as correct:

 ```d
 /// Returns: a size_t between 0 and 49, inclusive
 size_t favoriteNumber()  safe;

 ...
 ```
If your theory of memory safety leads you to conclude that the presence or absence of a comment can make otherwise-unsafe code memory safe, you have taken a wrong turn somewhere in your reasoning.
It's not a comment, it's a specification. Whereby I can conclude that if in review `favoriteNumber` returns something other than 0 to 49, it is in error. Without specifications, exactly zero trusted lines of code can be written.
 I agree with you that the version with the comment is better, 
 more maintainable code, and that we should hold our code to 
 such standards in code review. But bad and hard-to-maintain 
 code can still be memory safe (that is: free from possible UB).
Consider that the posix function `read` has the specification that it will read data from a file descriptor, put the data into a passed-in buffer, *up to* the amount of bytes indicated in the third parameter. Its prototype is: ```d system extern(C) int read(int fd, void *ptr, size_t nBytes); ``` Without reading the code of `read`, you must conclude from the specification that it does what it says it should do, and not say, ignore `nBytes` and just use the pointed-at data for as many bytes as it wants. Without the specification, just relying on the types, you can conclude nothing, and can never use `read` from ` trusted` code, ever. It's no different from `favoriteNumber`. In order to use it and make the assumption that it always will return 42, the author of that function must agree that that is what it's going to do. Otherwise (as you rightly say), anyone can change it to return something else *legitimately* and that might be beyond the bounds of the array you are using it for an index. So without the specification for `favoriteNumber`, I must conclude that `favoriteElement` is invalid as written. With a specification I can reason about what does and does not constitute validity. -Steve
Jul 26
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 26 July 2021 at 15:54:06 UTC, Steven Schveighoffer 
wrote:
 Consider that the posix function `read` has the specification 
 that it will read data from a file descriptor, put the data 
 into a passed-in buffer, *up to* the amount of bytes indicated 
 in the third parameter. Its prototype is:

 ```d
  system extern(C) int read(int fd, void *ptr, size_t nBytes);
 ```

 Without reading the code of `read`, you must conclude from the 
 specification that it does what it says it should do, and not 
 say, ignore `nBytes` and just use the pointed-at data for as 
 many bytes as it wants. [...]
 It's no different from `favoriteNumber`.
The difference between POSIX `read` and `favoriteNumber` is that you *can* read the source code of `favoriteNumber`. It's literally right there, in the same module. That's the entire reason why you can be certain it returns `42`. If `favoriteNumber` and `favoriteElement` were in different modules, your argument would be correct, because `favoriteElement` could no longer be certain about which version of `favoriteNumber` it was calling.
Jul 26
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Monday, 26 July 2021 at 18:59:45 UTC, Paul Backus wrote:
 The difference between POSIX `read` and `favoriteNumber` is 
 that you *can* read the source code of `favoriteNumber`. It's 
 literally right there, in the same module. That's the entire 
 reason why you can be certain it returns `42`.

 If `favoriteNumber` and `favoriteElement` were in different 
 modules, your argument would be correct, because 
 `favoriteElement` could no longer be certain about which 
 version of `favoriteNumber` it was calling.
If you consider the source to be the spec, then that contradicts your earlier suggestion that `favoriteNumber` can be changed -- its source is the spec, so changing the source to return something other than 42 will violate the spec. If you consider the source to be the spec *and* you think changing the spec at will is OK, then we have different philosophies on what a code review and "good software" means. -Steve
Jul 28
parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 28 July 2021 at 11:12:14 UTC, Steven Schveighoffer 
wrote:
 On Monday, 26 July 2021 at 18:59:45 UTC, Paul Backus wrote:
 The difference between POSIX `read` and `favoriteNumber` is 
 that you *can* read the source code of `favoriteNumber`. It's 
 literally right there, in the same module. That's the entire 
 reason why you can be certain it returns `42`.

 If `favoriteNumber` and `favoriteElement` were in different 
 modules, your argument would be correct, because 
 `favoriteElement` could no longer be certain about which 
 version of `favoriteNumber` it was calling.
If you consider the source to be the spec, then that contradicts your earlier suggestion that `favoriteNumber` can be changed -- its source is the spec, so changing the source to return something other than 42 will violate the spec. If you consider the source to be the spec *and* you think changing the spec at will is OK, then we have different philosophies on what a code review and "good software" means.
Again, I am in 100% in agreement with you that `favoriteElement` is not "good software" and should not pass code review. I have never claimed otherwise. (In fact, I spent the rest of that post talking about how we can reject functions like `favoriteElement` in code review!) That does not change the fact that in the *current version* of my example module, `favoriteElement` is memory safe--meaning, it cannot possibly cause undefined behavior when called with [safe values][1]. It is possible for bad software to be memory safe. It seems to me like we have been talking past each other for most of this discussion, with differences in background assumptions obscured by words like "correct", "valid", "safe", etc. In the future, I will try to be much more careful about defining my terms. [1]: https://dlang.org/spec/function.html#safe-values
Jul 28
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On Sunday, 25 July 2021 at 13:19:55 UTC, Paul Backus wrote:
 On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:
 At beerconf I committed to putting forward a DIP regarding a 
 new syntactic element to be made available within  trusted 
 functions, the  system block.  The presence of one or more 
  system blocks would enable  safe checking elsewhere in the 
 enclosing  trusted function.
Both before and after this proposal, there are 3 kinds of code:
4 kinds.
 1. Code that is automatically checked for memory safety.
Split this into: 1. Code that is automatically checked for memory safety. 1a. Code that has portions that have mechanical checks, but overall still needs manual checking for memory safety.
 2. Code that is assumed by the compiler to be safe, and must be 
 manually checked for memory safety.
 3. Code that is not checked for memory safety, and is assumed 
 to be unsafe.

 Currently, (1) is marked ` safe`, (2) is marked ` trusted`, and 
 (3) is marked ` system`.
(1) and (1a) are marked ` safe`. The latter have ` trusted` lambdas but require manual verification. The difference between 1 and 1a is pretty subtle.
 Under this proposal, (1) would be marked either ` safe` or 
 ` trusted`, (2) would be marked either ` trusted` or ` system`, 
 and (3) would be marked ` system`. I do not think this is an 
 improvement relative to the status quo.
Under this proposal (though I haven't seen exactly the proposal, but I think I've conversed with Bruce enough to have a good understanding), 1 becomes ` safe` *exclusively* (which is the main benefit), 1a becomes ` trusted`, 2 becomes legacy (marked ` trusted` but with no ` system` escapes) and IMO should be warned about to the user, 3 is still ` system`.
 The problematic  trusted lambda escapes creeping in to " safe" 
 code could be replaced going forward by a more honestly named 
 form,  trusted code with  system blocks. Best practices could 
 evolve to the point that  safe actually meant  safe again.
What makes ` trusted` lambdas problematic is that they implicitly depend on everything in their enclosing scope, which makes it easy for a change in the ` safe` portion of the code to accidentally violate an assumption that the ` trusted` portion depends on. If we want to address this issue at the language level, the most direct solution is to require that ` trusted` lambdas make their dependencies explicit. This can be done by requiring all ` trusted` nested functions (of which lambdas are a special case) to be `static`.
This doesn't solve the problem exactly. A ` trusted` lambda is a localized piece of code, and is prone to be abused with the justification "well, it's just for this one time, and I know what this function does". The compiler cannot enforce that the code inside (current) ` trusted` functions actually obeys the ` safe` API.
 Of course, this is a breaking change, so it would require a 
 deprecation process.
No, it shouldn't be. The idea is that ` trusted` code without ` system` escapes is still accepted as legacy code that works as before. I would recommend a message from the compiler though. ` trusted` lambdas should be required to be static, though that still doesn't solve the abuse problem. Perhaps they should just be disallowed? -Steve
Jul 25
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:
 At beerconf I committed to putting forward a DIP regarding a 
 new syntactic element to be made available within  trusted 
 functions, the  system block.  The presence of one or more 
  system blocks would enable  safe checking elsewhere in the 
 enclosing  trusted function.
I'm very happy to hear that. I think this proposal is an important and useful one, and I have been thinking of volunteering to write a DIP myself on the topic. I'll be dropping into tonight's BeerConf in a little while, so hopefully we can touch base then and see if we can find an opportunity for collaboration (maybe two heads and two people's spare time can overcome that burden of effort more readily than one).
Jul 25
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 7/25/21 10:43 AM, Joseph Rushton Wakeling wrote:

 On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:
 At beerconf I committed to putting forward a DIP regarding a new
 syntactic element to be made available within  trusted functions, the
  system block.  The presence of one or more  system blocks would
 enable  safe checking elsewhere in the enclosing  trusted function.
I'm very happy to hear that. I think this proposal is an important and useful one, and I have been thinking of volunteering to write a DIP myself on the topic.
In a recent discussion, I learned from you an idea of Steven Schveighoffer. Add to that the relayed opinion of an academic to the effect of "it is not safe if you can escape out of it." And add to that our failed attempt at "safe by default", the following thought formed in my mind. This thought may be exactly what Steven Schveighoffer or Bruce Carneal are bringing up anyway. If so, sorry for only now understanding it. :) The problems: 1) system by default provides no checking by default 2) trusted is not checked either 3) safe is not safe because you can escape easily How about: 1) Make trusted the default 2) Change trusted's semantics to be safe (i.e. make it the same as today's safe) 3) Allow system inside trusted 4) Strengthen safe: No trusted or system allowed Result: 1) We have safe by default because now trusted is the default and trusted is checked by default 2) safe is actually safe; no embarrassment Existing code: 1) Replace safe keywords with trusted where the compiler complains. (No safety lost because trusted will be behaving exactly like today's safe.) 2) Add system where the compiler complains. This is because all code is trusted by default and trusted is safe. Ali
Jul 25
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Sunday, 25 July 2021 at 21:42:16 UTC, Ali Çehreli wrote:
 [snip]
 Existing code:

 1) Replace  safe keywords with  trusted where the compiler 
 complains. (No safety lost because  trusted will be behaving 
 exactly like today's  safe.)

 2) Add  system where the compiler complains. This is because 
 all code is  trusted by default and  trusted is safe.

 Ali
I'd much rather add new attributes for new behavior than break existing code. If you want safe to not be able to call trusted functions, then introduce a new attribute with that behavior. I think too much is trying to get shoehorned into the safe/ trusted/ system dichotomy. I have favored the whitelist/blacklist approach that had been discussed on the forums previously (ignoring bikeshedding on names). whitelist would only allow operations known to be memory safe (so no trusted escape hatch). blacklist would be the opposite of what is currently safe, in that things that are not allowed in safe would have to be in a blacklist function (or block). This would contrast with system, which basically allows everything in them. Unmarked system functions would be assumed to be blacklist, unless they can be inferred to be whitelist. Whether whitelist is orthogonal to blacklist (such that everything that is not whitelist must be blacklist), I am not 100% sure on, but I would lean to putting things in blacklist only if they are not allowed in safe currently, which may allow for a middle ground. Regardless, a whitelist function can only call whitelist functions. A blacklist function can call any kind of function it wants. safe functions would be able to call whitelist functions, but nothing would stop them from calling trusted blacklist functions. trusted or system functions could call either whitelist or blacklist functions (the use of these might make it easier to narrow down where safety issues are). However, if either safe/ trusted/ system functions call blacklist functions (or include blocks) then they become blacklist (either explicitly or inferred). So this would distinguish an safe function that is whitelist from one that is blacklist. An safe function that is whitelist is verified by the compiler not to only use operations that the compiler verifies are memory safe, while one that is blacklist will have at some point called trusted functions that use blacklist behavior and needs to be manually verified. This approach doesn't break any code and allows for a lot of the flexibility that I think people want with the safe system.
Jul 27
parent reply jfondren <julian.fondren gmail.com> writes:
On Tuesday, 27 July 2021 at 16:36:48 UTC, jmh530 wrote:
 On Sunday, 25 July 2021 at 21:42:16 UTC, Ali Çehreli wrote:
 [snip]
 Existing code:

 1) Replace  safe keywords with  trusted where the compiler 
 complains. (No safety lost because  trusted will be behaving 
 exactly like today's  safe.)

 2) Add  system where the compiler complains. This is because 
 all code is  trusted by default and  trusted is safe.

 Ali
I'd much rather add new attributes for new behavior than break existing code. If you want safe to not be able to call trusted functions, then introduce a new attribute with that behavior. I think too much is trying to get shoehorned into the safe/ trusted/ system dichotomy.
Presently D is shoehorning two different usages into safe/ trusted/ system: 1. the documented usage: safe for checked safe functions, system for unrestricted functions, and trusted for a manually-reviewed interface between the two. (complaint: trusted functions check too little) 2. a usage people have moved to, where the manually-reviewed interfaces are now trusted function literals embedded inside safe functions. (complaint: ' trusted blocks' are too hard to review and their containing safe functions are misleadingly 'safe') So that's already two ways to write SafeD code and both are found wanting. With whitelist/ blacklist there'd be a third way. At that point, why not a fourth? I'm sure some people would like a single unsafe attribute, and maybe it could be justified by a late discovery of a serious complaint with whitelist/ blacklist. And then we could have a function signature containing ` trusted whitelist unsafe`, and then we could have a "The Nightmare of Memory Safety in D" video at cppcon. Opposite the Scylla of "breaking code" isn't safe open water, but the Charbydis of "having to explain SafeD to a skeptical newbie." Why five attributes? Because three wasn't enough. The immediate result of this DIP would also be three SafeD styles: 3. the new way of safe functions without compromise, trusted functions containing system blocks, and system functions farther-off result is just one SafeD again, this time also with the documentation matching what experts actually do.
Jul 27
parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 27 July 2021 at 21:05:52 UTC, jfondren wrote:
 [snip]

 Presently D is shoehorning two different usages into 
  safe/ trusted/ system:

 1. the documented usage:  safe for checked  safe functions, 
  system for unrestricted functions, and  trusted for a 
 manually-reviewed interface between the two. (complaint: 
  trusted functions check too little)

 2. a usage people have moved to, where the manually-reviewed 
 interfaces are now  trusted function literals embedded inside 
  safe functions. (complaint: ' trusted blocks' are too hard to 
 review and their containing  safe functions are misleadingly 
 'safe')
I don't use trusted function literals. I don't have a sense of how popular this usage is. In the context of my suggestion, an trusted literal that is blacklist would result in safe that is blacklist, which would make it a candidate for review.
 So that's already two ways to write SafeD code and both are 
 found wanting. With  whitelist/ blacklist there'd be a third 
 way. At that point, why not a fourth? I'm sure some people 
 would like a single  unsafe attribute, and maybe it could be 
 justified by a late discovery of a serious complaint with 
  whitelist/ blacklist. And then we could have a function 
 signature containing ` trusted  whitelist  unsafe`, and then we 
 could have a "The Nightmare of Memory Safety in D" video at 
 cppcon.
 [snip]
The underlying problem is how to fix design issues with safe/ trusted/ system while minimizing code breakage. Only the bare minimum to address those issues would make sense. Adding an unsafe that overlaps with existing or proposed functionality wouldn't make sense.
Jul 28