www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - trusted considered harmful

reply "David Nadlinger" <see klickverbot.at> writes:
 trusted in its current form needs to go. Its design is badly 
broken, as it leaks implementation details and encourages writing 
unsafe code.

The Problem
———————————

First, there is no point in having  trusted in the function 
signature. Why? From the perspective of the caller of the 
function in question,  safe and  trusted mean exactly the same 
thing. If you are not convinced about that, just consider that 
you can wrap any  trusted function into a  safe function to make 
it  safe, and vice versa.

So the current situation is similar to having two keywords `pure` 
and `pure2` in the language, which are completely equivalent for 
the consumer of an API. This is in itself a problem, since it is 
a pitfall for writing generic code. To stick with the example, 
it's easy to check only for `pure` in a template constraint when 
you really meant to accept both `pure` and `pure2` – and you 
_always_ want to accept both.

But is this alone enough to warrant a change to the language at 
this point? Probably not. But besides that, the current design 
also leads to problems for the implementation side:

One issue is that the distinction unnecessarily restricts the 
implementation in terms of interface stability. Yes,  safe and 
 trusted are equivalent from the caller's perspective, but they 
are mangled differently. This means that changing a function 
signature from one to the other is a breaking change to the ABI, 
and as the mangled name is available in the program (which is 
e.g. what std.traits.FunctionAttributes), also to the API.

Thus, you can't just change  trusted to  safe or vice versa on 
the implementation side if you make changes code which require 
 trusted, resp. cause it to be no longer needed. Sure, you can 
always move the implementation into a new, properly marked 
function, and make the original function just a wrapper around 
it. But this is kludgy at best, and might be unacceptable in the 
case of  safe ->  trusted for performance optimizations, if the 
inliner doesn't kick in. So, the only reasonable choice if you 
want to provide a stable interface in this case is to mark all 
functions which could ever possibly need to access unsafe code as 
 trusted, forgoing all the benefits of automatic safety checking.

But the much bigger problem is that  trusted doesn't play well 
with template attribute inference and makes it much too easy to 
accidentally mark a function as safe to call if it really isn't. 
Both things are a consequence of the fact that it can be applied 
at the function level only; there is no way to apply it 
selectively to only a part of the function.

As an example how this is problematic, consider that you are 
writing a function which takes some generic input data, and needs 
to do (unsafe) low-level buffer handling internally to 
efficiently do its job. You come up with a first implementation, 
maybe only accepting arrays for the sake of getting it working 
quickly, and add  trusted as your dirty buffer magic isn't 
visible from the outside, but does break attribute inference. 
Later, you decide that there is no reason not to take other range 
types as input. Fortunately, the actual implementation doesn't 
require any changes, so you just modify the template constraint 
as needed, and you are good. Well, no – you've just completely 
broken all safety guarantees for every program which calls your 
function, because empty/front/popFront of the passed range might 
be  system.

Now, you might argue that this is a contrived scenario. Yes, the 
mistake could have easily be avoided,  trusted on a template 
declaration should always raise a red flag. But cases like this 
_do_ occur in real-world code, and are easy to miss: The recently 
added std.uuid originally had a similar bug, which went unnoticed 
until during the vote [1] – at that point, a number of people, 
mostly experienced contributors, had reviewed the code. A safety 
system which is easy to break by accident is somewhat of a futile 
exercise.

Can you correctly implement such a template function with today's 
 trusted? Yes, there are workarounds, but it's not quite easy. 
One way is to explicitly detect the  safe-ty of the code accessed 
via template arguments and switch function prototypes using 
static ifs and string mixins to avoid code duplication. For an 
example of this, see Jonathan's new std.range.RefRange [2]. It 
works, but it isn't pretty. The average programmer will, just as 
done in the revised version of std.uuid [3], likely give up and 
accept the fact that the function isn't callable from safe code. 
Which is a pity, as we should really utilize the unique asset we 
got in SafeD to the fullest, but this only works if everything 
that could be  safe is marked as such. The situation won't get 
better as we continue to advocate the use of ranges, either.

To summarize, there are, at least as far as I can see, no 
advantages in distinguishing between  safe and  trusted in 
function signatures, and the function-level granularity of 
 trusted yields to avoidable bugs in real-world code. 
Fortunately, we should be able to resolve both of these issues 
fairly easily, as described below.


A Solution
——————————

Let me make something clear first: I am _not_ intending to remove 
 trusted from the language. As a bridge between the  safe and 
 system worlds, it is an integral part of SafeD. What I'm 
proposing is:

  1) Remove the distinction between  safe and  trusted at the 
interface (ABI, API) level. This implies changing the name 
mangling of  trusted to Nf, and consequently removing the 
distinction in DMD altogether (at least in user-facing parts like 
.stringof and error messages). In theory, this is a breaking 
change, but as any code that doesn't treat them the same is buggy 
anyway, it shouldn't be in practice. As for 
std.traits.FunctionAttribute, we could either make trusted an 
alias for safe, or just remove documentation for the former and 
keep it around for some time (there is no way to deprecate an 
enum member).

  2) The first step is necessary, but mainly of cosmetic nature 
(think `pure`, `pure2`). We still need to address for the 
granularity and attribute inference problem. The obvious solution 
is to add a " trusted" declaration/block, which would allow 
unsafe code in a certain region. Putting  trusted in the function 
header would still be allowed for backwards compatibility (but 
discouraged), and would have the same effect as marking the 
function  safe and wrapping its whole body in a  trusted block. 
It could e.g. look something like this (the   prefix definitely 
looks weird, but I didn't want to introduce a new keyword):

---
  void foo(T)(T t) {
    t.doSomething();
     trusted {
      // Do something dirty.
    }
    t.doSomethingElse();
     trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
  }
---

This is similar to other constructs we have today, for example 
debug {}, which allows impure code. It can be debated whether a 
block »argument« should introduce a new scope or not (like 
static if). The latter currently seems much more attractive to 
me, but I suppose it could be confusing for some.

In any case, while there is probably quite a bit of bikeshedding 
to be done for 2), I don't think there is much controversy about 
1). So, let's try to get this done shortly after the 2.060 
release – as discussed above, it is very unlikely that the 
change will break something, but the odds still increase over 
time. Also, there is currently a Phobos pull request [4] which 
will be influenced by the outcome of this discussion.

David



[1] 
http://forum.dlang.org/thread/jrpni1$rt$1 digitalmars.com?page=2#post-pcaaoymspzelodvmnbvc:40forum.dlang.org
[2] 
https://github.com/D-Programming-Language/phobos/blob/c005f334ec34d3b0295d5dbf48212972e17823d0/std/range.d#L7327
[3] 
https://github.com/D-Programming-Language/phobos/blob/c005f334ec34d3b0295d5dbf48212972e17823d0/std/uuid.d#L1193
[4] https://github.com/D-Programming-Language/phobos/pull/675
Jul 27 2012
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 02:08:28 David Nadlinger wrote:
 This is similar to other constructs we have today, for example
 debug {}, which allows impure code. It can be debated whether a
 block =C2=BBargument=C2=AB should introduce a new scope or not (like
 static if). The latter currently seems much more attractive to
 me, but I suppose it could be confusing for some.

I'd definitely vote for it _not_ introducing a new scope. It will be mu= ch more=20 useful that way.
 In any case, while there is probably quite a bit of bikeshedding
 to be done for 2), I don't think there is much controversy about
 1). So, let's try to get this done shortly after the 2.060
 release =E2=80=93 as discussed above, it is very unlikely that the
 change will break something, but the odds still increase over
 time. Also, there is currently a Phobos pull request [4] which
 will be influenced by the outcome of this discussion.

I'm all for this. Templates and trusted just don't get along very well= , which=20 has a huge impact on ranges. The only real saving grace there is that=20= attribute inferrence solves a lot of the problem if you just never use = safe=20 or trusted on a templated function, but when you have to do system st= uff=20 within such a template, it's a real problem (as RefRange shows). As for the syntax, I think that trusted{ /+ system code+/ } makes perf= ect=20 sense. - Jonathan M Davis
Jul 27 2012
prev sibling next sibling parent =?ISO-8859-1?Q?Jos=E9_Armando_Garc=EDa_Sancio?= <jsancio gmail.com> writes:
On Fri, Jul 27, 2012 at 5:08 PM, David Nadlinger <see klickverbot.at> wrote:
  2) The first step is necessary, but mainly of cosmetic nature (think
 `pure`, `pure2`). We still need to address for the granularity and attribute
 inference problem. The obvious solution is to add a " trusted"
 declaration/block, which would allow unsafe code in a certain region.
 Putting  trusted in the function header would still be allowed for backwards
 compatibility (but discouraged), and would have the same effect as marking
 the function  safe and wrapping its whole body in a  trusted block. It could
 e.g. look something like this (the   prefix definitely looks weird, but I
 didn't want to introduce a new keyword):

Agreed. This is very similar to how Rust works. In Rust all the functions are assumed to be safe. Unsafe code can only be performed in clearly marked blocks. Note: I am not suggesting D should implement Rust's solution as David already pointed out. -Jose
Jul 27 2012
prev sibling next sibling parent "Jesse Phillips" <jessekphillips+D gmail.com> writes:
On Saturday, 28 July 2012 at 00:08:30 UTC, David Nadlinger wrote:

  2) [...] The obvious solution is to add a " trusted" 
 declaration/block, which would allow unsafe code in a certain 
 region. Putting  trusted in the function header would still be 
 allowed for backwards compatibility (but discouraged), and 
 would have the same effect as marking the function  safe and 
 wrapping its whole body in a  trusted block. It could e.g. look 
 something like this (the   prefix definitely looks weird, but I 
 didn't want to introduce a new keyword):

 ---
  void foo(T)(T t) {
    t.doSomething();
     trusted {
      // Do something dirty.
    }
    t.doSomethingElse();
     trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
  }
 ---

I don't see flaw with 1. However 2 doesn't sound right. trusted { // Do something dirty. } You aren't supposed to do dirty things in trusted code. You're supposed to safely wrap a system function to be usable by a safe function. The system function is supposed to be short and getting its hands dirty. Remember this is about memory safety and not lack of bugs safety. The template issue needs fixed, but maybe it is the inference which needs expanded? Maybe a template is only inferred as safe or trusted and require explicitly system? I think I was going to say more, but I'm not versed in the problems for this area, which I'm sure there are many, so this is probably good enough self butchering.
Jul 27 2012
prev sibling next sibling parent "David Piepgrass" <qwertie256 gmail.com> writes:
 I don't see flaw with 1.

 However 2 doesn't sound right.

      trusted {
       // Do something dirty.
     }

 You aren't supposed to do dirty things in  trusted code. You're 
 supposed to  safely wrap a system function to be usable by a 
 safe function. The system function is supposed to be short and 
 getting its hands dirty.

True, but since the proposal is that all functions should be either safe or system, a trusted block is necessary in a safe function in order to call system functions. Perhaps you would suggest that a trusted block should be able to _call_ system code but not actually do anything unsafe directly? That sounds interesting, but it's not how trusted currently works.
Jul 27 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 04:05:14 Jesse Phillips wrote:
 On Saturday, 28 July 2012 at 00:08:30 UTC, David Nadlinger wrote:
  2) [...] The obvious solution is to add a " trusted"
 
 declaration/block, which would allow unsafe code in a certain
 region. Putting  trusted in the function header would still be
 allowed for backwards compatibility (but discouraged), and
 would have the same effect as marking the function  safe and
 wrapping its whole body in a  trusted block. It could e.g. look
 something like this (the   prefix definitely looks weird, but I
 didn't want to introduce a new keyword):
 
 ---
 
  void foo(T)(T t) {
  
    t.doSomething();
     trusted {
    
      // Do something dirty.
    
    }
    t.doSomethingElse();
     trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
  
  }
 
 ---

I don't see flaw with 1. However 2 doesn't sound right. trusted { // Do something dirty. } You aren't supposed to do dirty things in trusted code. You're supposed to safely wrap a system function to be usable by a safe function. The system function is supposed to be short and getting its hands dirty. Remember this is about memory safety and not lack of bugs safety.

The whole point of trusted is to mark system code as being safe. The programmer is certifying that the code is safe, because they know that what it's doing is actually safe in spite of the fact that the compiler can't verify that. It's perfectly acceptable to put "dirty code" in an trusted function. The difference between trusted and system is that with trusted, the programmer is guaranteeing that it's actually safe regardless of what arguments it's given, whereas system is not only doing unsafe things, but whether they're ultimately safe or not depends on arguments and/or other code, so the programmer _can't_ guarantee that it's safe.
 The template issue needs fixed, but maybe it is the inference
 which needs expanded? Maybe a template is only inferred as safe
 or trusted and require explicitly system?
 
 I think I was going to say more, but I'm not versed in the
 problems for this area, which I'm sure there are many, so this is
 probably good enough self butchering.

The main problem is when you have code that is system in a templated function and while you can guarantee that that code is actually safe and therefore mark it as trusted, you can't guarantee that for the rest of the function. With the current situation, you can't mark that function as trusted, because that would be certifying that _everything_ in the function was safe, which isn't necessarily true. Ideally, you would be able to mark the operations that are system but you know are really safe with trusted and let whether the function as a whole is safe or system be inferred by the compiler. But right now, you either have to break up that trusted code into a separate function (which isn't always reasonable) or play games with static if and having two versions of the same function where one is safe and the other is system. For instance, in the new std.range.RefRange, save is defined something like this: private static void _testSave(R)(R* range) { (*range).save; } static if(isSafe!(_testSave!R)) { property auto save() trusted { mixin(_genSave()); } } else { property auto save() { mixin(_genSave()); } } private static string _genSave() safe pure nothrow { return `import std.conv;` ~ `alias typeof((*_range).save) S;` ~ `static assert(isForwardRange!S, S.stringof ~ " is not a forward range.");` ~ `auto mem = new void[S.sizeof];` ~ `emplace!S(mem, cast(S)(*_range).save);` ~ `return RefRange!S(cast(S*)mem.ptr);`; } The problem is that the emplace stuff is system, and I know that it's really safe and want to mark it with trusted, but I _don't_ know that _range's save function is safe, so I can't mark RefRange's save as trusted. I'm stuck either doing nonsense like the code above or giving up on making it possible for save to be safe. With David's suggestion, all of that can be reduced to this: property auto save() { import std.conv; alias typeof((*_range).save) S; static assert(isForwardRange!S, S.stringof ~ " is not a forward range."); trusted { auto mem = new void[S.sizeof];` emplace!S(mem, cast(S)(*_range).save);` return RefRange!S(cast(S*)mem.ptr); } } That is _way_ cleaner. Now, the actual code of in RefRange is actually even _more_ complicated because of the need to handle the possibility of making save const, but if enhancement request# 8407 were implemented, then that would be solved. I believe that there is at least _some_ buy in on Walter with regards to issue# 8407 based on some comments by Andrei, but I have no idea what Walter will think of David's proposal. With both, functions like RefRange's save become reasonably small. Without them, you either give up on const and/or safe, make your templated code require that the functions it uses be const and/or safe (which can be way too restrictive), or you do what I did with RefRange, which works but is downright ugly. - Jonathan M Davis
Jul 27 2012
parent Artur Skawina <art.08.09 gmail.com> writes:
On 07/29/12 07:14, David Piepgrass wrote:
 On Saturday, July 28, 2012 22:08:42 David Nadlinger wrote:
 On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis But unfortunately
wrong – you call S.save in the  trusted block… ;)

Yeah. I screwed that up. I was obviously in too much of a hurry when I wrote it. And actually, in this particular case, since the part that can't be trusted is in the middle of an expression doing system stuff, simply using an trusted block wouldn't do the trick.

Have you guys thought about the possibility that the language could simply not trust any calls that were resolved using a template argument? I'm a bit tired so I may be missing something, but it seems to me that (in a trusted template) if the compiler uses an instantiated template parameter (e.g. actual type Foo standing in for template parameter T) to choose a function to call, the compiler should require that the function be safe, based on the principle that a template cannot vouch for what it can't control. IOW, since a template can't predict what function actually gets called, the compiler should require whatever function gets called to be safe.

This wouldn't be enough, and would just paper over the real issue. While the problem is probably most obvious with templates, it is also present when no generics are involved. If you write a trusted non-templated function like class C { int field; /*...*/ } auto f(C c) trusted { do_something_potentially_unsafe_but_verified_with(c.field); } it will be fine, as it is all obviously safe, right? Well, until somebody else later changes 'C' to class C { property field() { return unsafe_buggy_code(); } /*...*/ } without realizing that 'C.field' is accessed in a safe context. The compiler will silently accept it and run the system code just as if it was marked as safe or trusted. The definition of 'C' and the function 'f' could be in different modules or even libraries. Both will still build, without even giving a hint that something may be wrong. "You can't audit code that isn't available." -- that code may not have been written yet. With the current ' trust' model even an "innocent" access to non-local data can result in all safe checks being bypassed. Keeping all external accesses out of ' trusted' scopes is not really a practical solution. Most accesses will be (perceived as) safe, so programmers will choose to ignore the potential hazards in order to keep the code readable. artur
Jul 29 2012
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2012-07-28 00:08:28 +0000, "David Nadlinger" <see klickverbot.at> said:

  trusted in its current form needs to go. Its design is badly broken, 
 as it leaks implementation details and encourages writing unsafe code.

trusted is a dangerous thing. Since the first time I tried to use it, I always found it troublesome. I agree it needs to be a scope. And for backward compatibility, a trusted function should be exactly the same as a safe function where the whole body is wrapped in a trusted scope. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca/
Jul 27 2012
parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2012-07-28 03:24:58 +0000, Michel Fortin <michel.fortin michelf.ca> said:

 On 2012-07-28 00:08:28 +0000, "David Nadlinger" <see klickverbot.at> said:
 
  trusted in its current form needs to go. Its design is badly broken, 
 as it leaks implementation details and encourages writing unsafe code.

trusted is a dangerous thing. Since the first time I tried to use it, I always found it troublesome. I agree it needs to be a scope. And for backward compatibility, a trusted function should be exactly the same as a safe function where the whole body is wrapped in a trusted scope.

And when I say it "needs to be a scope", what I meat is a block. I don't think it should be a new scope unless someone finds a good reason for it (unsafe struct destructors?) -- Michel Fortin michel.fortin michelf.ca http://michelf.ca/
Jul 27 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, July 27, 2012 23:28:17 Michel Fortin wrote:
 On 2012-07-28 03:24:58 +0000, Michel Fortin <michel.fortin michelf.ca> said:
 On 2012-07-28 00:08:28 +0000, "David Nadlinger" <see klickverbot.at> said:
  trusted in its current form needs to go. Its design is badly broken,
 as it leaks implementation details and encourages writing unsafe code.

trusted is a dangerous thing. Since the first time I tried to use it, I always found it troublesome. I agree it needs to be a scope. And for backward compatibility, a trusted function should be exactly the same as a safe function where the whole body is wrapped in a trusted scope.

And when I say it "needs to be a scope", what I meat is a block. I don't think it should be a new scope unless someone finds a good reason for it (unsafe struct destructors?)

If you need another scope, you can always just use a second set of braces, but if an trusted block introduces another scope, there's no way to make it _not_ introduce one (not without adding more syntax to make it not introduce one anyway). - Jonathan M Davis
Jul 27 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Saturday, 28 July 2012 at 00:53:24 UTC, José Armando García 
Sancio wrote:
 On Fri, Jul 27, 2012 at 5:08 PM, David Nadlinger 
 <see klickverbot.at> wrote:
  2) The first step is necessary, but mainly of cosmetic nature 
 (think
 `pure`, `pure2`). We still need to address for the granularity 
 and attribute
 inference problem. The obvious solution is to add a " trusted"
 declaration/block, which would allow unsafe code in a certain 
 region.
 Putting  trusted in the function header would still be allowed 
 for backwards
 compatibility (but discouraged), and would have the same 
 effect as marking
 the function  safe and wrapping its whole body in a  trusted 
 block. It could
 e.g. look something like this (the   prefix definitely looks 
 weird, but I
 didn't want to introduce a new keyword):

Agreed. This is very similar to how Rust works. In Rust all the functions are assumed to be safe. Unsafe code can only be performed in clearly marked blocks. Note: I am not suggesting D should implement Rust's solution as David already pointed out. -Jose

C#'s approach is also similar with the unsafe keyword.
Jul 27 2012
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 28-Jul-12 04:08, David Nadlinger wrote:
 Let me make something clear first: I am _not_ intending to remove
  trusted from the language. As a bridge between the  safe and  system
 worlds, it is an integral part of SafeD. What I'm proposing is:

   1) Remove the distinction between  safe and  trusted at the interface
 (ABI, API) level. This implies changing the name mangling of  trusted to
 Nf, and consequently removing the distinction in DMD altogether (at
 least in user-facing parts like .stringof and error messages). In
 theory, this is a breaking change, but as any code that doesn't treat
 them the same is buggy anyway, it shouldn't be in practice. As for
 std.traits.FunctionAttribute, we could either make trusted an alias for
 safe, or just remove documentation for the former and keep it around for
 some time (there is no way to deprecate an enum member).

No question here, trusted should be usable in place of safe _transparently_.
   2) The first step is necessary, but mainly of cosmetic nature (think
 `pure`, `pure2`). We still need to address for the granularity and
 attribute inference problem. The obvious solution is to add a " trusted"
 declaration/block, which would allow unsafe code in a certain region.
 Putting  trusted in the function header would still be allowed for
 backwards compatibility (but discouraged), and would have the same
 effect as marking the function  safe and wrapping its whole body in a
  trusted block. It could e.g. look something like this (the   prefix
 definitely looks weird, but I didn't want to introduce a new keyword):

 ---
   void foo(T)(T t) {
     t.doSomething();
      trusted {
       // Do something dirty.
     }
     t.doSomethingElse();
      trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
   }
 ---

 This is similar to other constructs we have today, for example debug {},
 which allows impure code. It can be debated whether a block »argument«
 should introduce a new scope or not (like static if). The latter
 currently seems much more attractive to me, but I suppose it could be
 confusing for some.

Finally proper granularity for trusted! I'd say that SafeD is still unusable mostly because trusted is too blunt (your example from std.uuid). The day writeln works for all safe types (with safe/trusted toString or whatever) I'd call SafeD barely usable. I believe it need not introduce another scope. -- Dmitry Olshansky
Jul 28 2012
prev sibling next sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 07/28/12 02:08, David Nadlinger wrote:
  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.

The problem with trusted is that it is transitive. trusted should allow unsafe operations in the covered scope (right now - the function), but disallow calling unsafe ( system) code. IOW a safe function should only be able to call a safe or trusted one, and the same restriction should apply to trusted functions. This way you can actually audit the code - something which isn't typically possible right now, when you have no control over what the "trusted" code might call. There are two issues w/ such a change 1. Backward compatibility 2. Being able to override the restrictions (eg for debugging) Both can be addressed by having a trust-me-harder mode, which can be implemented as " trusted safe", ie functions marked with both attributes behave just as trusted code does right now. Yes, safe and trusted do not need different name mangling. The ' attribute {}' syntax is something that is needed, but should probably wait for /proper/ attribute handling, which should also allow for a saner 'synchronized' etc. '{}' not introducing a scope is not the most intuitive approach (yes, this includes the currently existing cases too). artur
Jul 28 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/28/12 7:05 AM, Artur Skawina wrote:
 On 07/28/12 02:08, David Nadlinger wrote:
  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.

The problem with trusted is that it is transitive. trusted should allow unsafe operations in the covered scope (right now - the function), but disallow calling unsafe ( system) code.

No. Trusted means "hand-checked, good to go". It can do anything. Andrei
Jul 28 2012
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/28/12 10:34 AM, Artur Skawina wrote:
 On 07/28/12 15:47, Andrei Alexandrescu wrote:
 On 7/28/12 7:05 AM, Artur Skawina wrote:
 On 07/28/12 02:08, David Nadlinger wrote:
  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.

The problem with trusted is that it is transitive. trusted should allow unsafe operations in the covered scope (right now - the function), but disallow calling unsafe ( system) code.

No. Trusted means "hand-checked, good to go". It can do anything.

Exactly, but the only way for it to mean anything is if it really /can/ be hand-checked.

It means someone stares at it until the goat dies.
 A "trusted" function that calls arbitrary, potentially
 unsafe code cannot be trusted.

I think you have it all wrong. Trusted means it's verified by a human, not by a formal method. The compiler allows it to do anything.
 You can't audit code that isn't available.

Correct. If you make e.g. syscalls into a closed-source OS you trust that function to not have bugs. It's a decision made by the human who annotates trusted.
 So the result is bugs (like the ones mentioned in this thread), where  safe
 is bypassed, because the  trusted functions aren't expecting to be used
 with "unsafe" ones.  trusted bypasses *all* safety checks, not just those
 in the hand-checked code. This is something that you will want sometimes,
 but in most cases is neither necessary nor desirable. When dealing with
 safety one has to be conservative. The proposals to limit the scope of
  trusted only address the symptoms, not the cause. If the design is fixed,
 many of the reasons for introducing the finer-grained  trusted disappear,
 and it is the truly unsafe ( trusted that calls  system) code that needs
 the extra annotations -- which is a good thing. Papering over design bugs
 never is.

I don't understand what you suggest here. Is it a sort of a refinement of trusted? Andrei
Jul 28 2012
prev sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 07/28/12 18:16, Andrei Alexandrescu wrote:
 On 7/28/12 10:34 AM, Artur Skawina wrote:
 So the result is bugs (like the ones mentioned in this thread), where  safe
 is bypassed, because the  trusted functions aren't expecting to be used
 with "unsafe" ones.  trusted bypasses *all* safety checks, not just those
 in the hand-checked code. This is something that you will want sometimes,
 but in most cases is neither necessary nor desirable. When dealing with
 safety one has to be conservative. The proposals to limit the scope of
  trusted only address the symptoms, not the cause. If the design is fixed,
 many of the reasons for introducing the finer-grained  trusted disappear,
 and it is the truly unsafe ( trusted that calls  system) code that needs
 the extra annotations -- which is a good thing. Papering over design bugs
 never is.

I don't understand what you suggest here. Is it a sort of a refinement of trusted?

Maybe my other response to David makes things more clear. Yes, there would be a cost to this change (some lib and user code would need additional annotations), but I think it'd be worth it, as right now it is much too easy to 'break' the safe guarantees, often w/o even realizing it. artur
Jul 28 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 07/28/12 15:47, Andrei Alexandrescu wrote:
 On 7/28/12 7:05 AM, Artur Skawina wrote:
 On 07/28/12 02:08, David Nadlinger wrote:
  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.

The problem with trusted is that it is transitive. trusted should allow unsafe operations in the covered scope (right now - the function), but disallow calling unsafe ( system) code.

No. Trusted means "hand-checked, good to go". It can do anything.

Exactly, but the only way for it to mean anything is if it really /can/ be hand-checked. A "trusted" function that calls arbitrary, potentially unsafe code cannot be trusted. You can't audit code that isn't available. So the result is bugs (like the ones mentioned in this thread), where safe is bypassed, because the trusted functions aren't expecting to be used with "unsafe" ones. trusted bypasses *all* safety checks, not just those in the hand-checked code. This is something that you will want sometimes, but in most cases is neither necessary nor desirable. When dealing with safety one has to be conservative. The proposals to limit the scope of trusted only address the symptoms, not the cause. If the design is fixed, many of the reasons for introducing the finer-grained trusted disappear, and it is the truly unsafe ( trusted that calls system) code that needs the extra annotations -- which is a good thing. Papering over design bugs never is. artur
Jul 28 2012
prev sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 07/28/12 17:05, David Nadlinger wrote:
 On Saturday, 28 July 2012 at 11:05:34 UTC, Artur Skawina wrote:
 On 07/28/12 02:08, David Nadlinger wrote:
  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.

The problem with trusted is that it is transitive. trusted should allow unsafe operations in the covered scope (right now - the function), but disallow calling unsafe ( system) code. IOW a safe function should only be able to call a safe or trusted one, and the same restriction should apply to trusted functions. This way you can actually audit the code - something which isn't typically possible right now, when you have no control over what the "trusted" code might call.

Sorry, while I think I know which problem you are referring to, I don't see what your suggestion does to address it. There is no such thing as allowing unsafe code »only in the current scope«. As soon as you are allowed to do unsafe things, you can e.g. just cast a system function to safe and call it. Or jump to it using inline assembly, etc.

Of course inside a trusted scope you will always be able to bypass every restriction. But that does not mean that all restrictions should be disabled by default, so that you are not able to take advantage of them and have to reimplement them by hand, like checking if every called function is 'safe'.
 My suggestion might not be perfect, but it addresses the problem by allowing
you to trust only the specific piece of your code which actually need to
perform unsafe things. Reviewers can concentrate on the smaller sections to
make sure that what they do is indeed  safe (e.g. that they don't call unsafe
external code), and if something unsafe is accidentally done outside, the
compiler will catch it.

Well, what you're proposing is already possible (other than the scope-lessness), your suggestion would result in a nicer syntax. Which would be an improvement, I agree. but it's not really enough. Say somebody writes a function like this one: struct S { size_t m() safe { return 42; } } T* t(T)(T* p, size_t idx) { auto s = p+idx; auto i = s.m(); return p+i; } then later realizes it can't be called from safe code. The easiest solution will be to mark the 't' function is trusted, which will work. T* t(T)(T* p, size_t idx) trusted { auto s = p+idx; auto i = s.m(); return p+i; } Until 't' (much) later gets called with a different struct struct U { size_t m() { return 666; } } Which will succeed and system code will silently run in a safe context; w/o even a warning. What should have happened is that the function, instead of being marked as trusted, should have been rewritten as: T* t(T)(T* p, size_t idx) safe { trusted gets() { return p+idx; } auto s = gets(); auto i = s.m(); trusted getn(size_t i) { return p+i; } return getn(i); } But, realistically, that is not what's going to happen most of the time... Your suggestion is an improvement, in that it makes the above look like: T* t(T)(T* p, size_t idx) safe { trusted { auto s = p+idx; } auto i = s.m(); trusted { return p+i; } } which is much better. But I'm afraid marking the whole function as trusted will still be the 'easier' choice - so the problem won't go away, just become less frequent. Accidentally 'breaking' safeness is a serious problem, at least if safe is to be taken seriously.
 I also think implementing your suggestion, i.e. distinguishing between
»local« and »external« code, would add much value for another reason: In my
experience, the most frequent use case for  trusted is precisely to interface
with external code that is  system, either for legacy reasons of because it
just can't be safe in the general case (e.g. grep the Phobos sources for
 trusted). It seems like the actual low-level pointer/inline asm magic is often
encapsulated into separate ( system) functions anyway.
 
 The ' attribute {}' syntax is something that is needed, but should probabl
 wait for /proper/ attribute handling, which should also allow for a saner
 'synchronized' etc.

What is »proper attribute handling« supposed to be? How would it influence the implementation of trusted{}? How is synchronized related to trusted?

There have been some discussions about attributes in the past on these lists. I have some thoughts on »proper attribute handling«, but that's a different topic. What I mean here is that the /syntax/ could also be used for marking scopes/blocks with other attributes besides trusted; synchronized is an example that would also fit in the ' attr {...}' scheme and is interesting because that also needs an optional declaration block. artur
Jul 28 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/27/12 8:08 PM, David Nadlinger wrote:
 First, there is no point in having  trusted in the function signature.
 Why? From the perspective of the caller of the function in question,
  safe and  trusted mean exactly the same thing. If you are not convinced
 about that, just consider that you can wrap any  trusted function into a
  safe function to make it  safe, and vice versa.

If trusted is not part of the signature, we can't enable e.g. analyzers that verify an entire program or package to be safe. This is not something that's currently used, but I'd hate to look back and say, "heck, I hate that we conflated trusted with safe!"
 One issue is that the distinction unnecessarily restricts the
 implementation in terms of interface stability. Yes,  safe and  trusted
 are equivalent from the caller's perspective, but they are mangled
 differently. This means that changing a function signature from one to
 the other is a breaking change to the ABI, and as the mangled name is
 available in the program (which is e.g. what
 std.traits.FunctionAttributes), also to the API.

I salute this quest for stability, but I don't find it the argument all that compelling. If one makes changes to a library, well, some changes will require clients to relink. Some don't. I don't see why we'd make a thing of the particular change safe <-> trusted. Is this often, fundamental, big,...?
 Thus, you can't just change  trusted to  safe or vice versa on the
 implementation side if you make changes code which require  trusted,
 resp. cause it to be no longer needed.

Can't parse this sentence following "if you make changes code..."
 But the much bigger problem is that  trusted doesn't play well with
 template attribute inference and makes it much too easy to accidentally
 mark a function as safe to call if it really isn't. Both things are a
 consequence of the fact that it can be applied at the function level
 only; there is no way to apply it selectively to only a part of the
 function.

This could be a more serious problem. Could you please write a brief example that shows attribute deduction messing things up? I don't understand how marking a template as trusted is bad.
 The obvious solution is to add a " trusted"
 declaration/block, which would allow unsafe code in a certain region.

This is sensible, but I fail to figure how it adds value over marking functions as trusted. Sure, it's finer-grained, but it's also less structured. Andrei
Jul 28 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 28/07/2012 16:02, Andrei Alexandrescu a écrit :
 This is sensible, but I fail to figure how it adds value over marking
 functions as  trusted. Sure, it's finer-grained, but it's also less
 structured.

Let me explain you the a problem I faced this week which illustrate pefectly the problem. I have a function which is system. The function caller have to ensure the parameters are correct. If it does, then the function is trusted as of current semantic. Let's call that function foo . I have a second function, bar, which is a template function. This bar function uses reflection on an object and also use foo. The usage of foo is checked, so bar is supposed to be trusted if all functions called by reflection are safe or trusted. In pseudo code : void foo(arguments) system; void bar(T)(T t) { // Reflect T and perform operations, sometime calling reflected methods. // call foo with checked arguments. } Now, as multiple reflected method can be called, it is really hard to check if bar is trusted or safe . At the end, I ended up not marking the code trusted which make it system, even if it is safe in most cases. Simply because it is too difficult to know in which case it is trusted. With the proposal, code becomes : void foo(arguments) system; void bar(T)(T t) { // Reflect T and perform operations, sometime calling reflected methods. trusted { // call foo with checked arguments. } } The problem is very real and the solution elegant. Any code analyzer will be able to look for trusted block anyway, as it is certainly a concern to do more code review on trusted blocks.
Jul 29 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 14:02:44 UTC, Andrei Alexandrescu 
wrote:
 If  trusted is not part of the signature, we can't enable e.g. 
 analyzers that verify an entire program or package to be safe. 
 This is not something that's currently used, but I'd hate to 
 look back and say, "heck, I hate that we conflated  trusted 
 with  safe!"

Could you elaborate on that? A safe function is _identical_, from a client point of view, to a trusted one. It can always call a trusted function under the hood without the caller noticing, there is no way around that. Thus, to be able to check that a program consists only of safe code [1], you would need its complete source, i.e. including all the functions it can possibly invoke, to be able to check if trusted code is called in any place. But with all the source available, you can just check the implementation for trusted blocks [2], there is no advantage over having it in the signature. Destroyed? :P David [1] Which is highly unlikely, by the way, as many parts of druntime just can't be safe. [2] Or trusted attributes in the function header – as described in the original post, they won't go away for backwards compatibility.
Jul 28 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 14:02:44 UTC, Andrei Alexandrescu 
wrote:
 But the much bigger problem is that  trusted doesn't play well 
 with
 template attribute inference and makes it much too easy to 
 accidentally
 mark a function as safe to call if it really isn't. Both 
 things are a
 consequence of the fact that it can be applied at the function 
 level
 only; there is no way to apply it selectively to only a part 
 of the
 function.

This could be a more serious problem. Could you please write a brief example that shows attribute deduction messing things up? I don't understand how marking a template as trusted is bad.

See the std.uuid discussion I linked in the original post for a real-world example of this bug. The gist is: You can't ever mark a function which can end up execute code coming from a template parameter, for example a function accepting a range, as trusted, because then you would vouch for all the passed in code as well, which might be system. [1] Templates parameters which just supply data are obviously not a problem. David [1] Unless you explicitly check whether the passed code is safe, that is. If you go down this route, though, you need to duplicate the function declaration, which isn't pretty. See std.range.RefRange.save for an example of this.
Jul 28 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 11:05:34 UTC, Artur Skawina wrote:
 On 07/28/12 02:08, David Nadlinger wrote:
  trusted in its current form needs to go. Its design is badly 
 broken, as it leaks implementation details and encourages 
 writing unsafe code.

The problem with trusted is that it is transitive. trusted should allow unsafe operations in the covered scope (right now - the function), but disallow calling unsafe ( system) code. IOW a safe function should only be able to call a safe or trusted one, and the same restriction should apply to trusted functions. This way you can actually audit the code - something which isn't typically possible right now, when you have no control over what the "trusted" code might call.

Sorry, while I think I know which problem you are referring to, I don't see what your suggestion does to address it. There is no such thing as allowing unsafe code »only in the current scope«. As soon as you are allowed to do unsafe things, you can e.g. just cast a system function to safe and call it. Or jump to it using inline assembly, etc. My suggestion might not be perfect, but it addresses the problem by allowing you to trust only the specific piece of your code which actually need to perform unsafe things. Reviewers can concentrate on the smaller sections to make sure that what they do is indeed safe (e.g. that they don't call unsafe external code), and if something unsafe is accidentally done outside, the compiler will catch it. I also think implementing your suggestion, i.e. distinguishing between »local« and »external« code, would add much value for another reason: In my experience, the most frequent use case for trusted is precisely to interface with external code that is system, either for legacy reasons of because it just can't be safe in the general case (e.g. grep the Phobos sources for trusted). It seems like the actual low-level pointer/inline asm magic is often encapsulated into separate ( system) functions anyway.
 The ' attribute {}' syntax is something that is needed, but 
 should probabl
 wait for /proper/ attribute handling, which should also allow 
 for a saner
 'synchronized' etc.

What is »proper attribute handling« supposed to be? How would it influence the implementation of trusted{}? How is synchronized related to trusted? David
Jul 28 2012
prev sibling next sibling parent "Jesse Phillips" <jessekphillips+D gmail.com> writes:
On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis wrote:
 On Saturday, July 28, 2012 04:05:14 Jesse Phillips wrote:
 You aren't supposed to do dirty things in  trusted code. You're
 supposed to  safely wrap a system function to be usable by a 
 safe
 function. The system function is supposed to be short and 
 getting
 its hands dirty. Remember this is about memory safety and not
 lack of bugs safety.

It's perfectly acceptable to put "dirty code" in an trusted function. The difference between trusted and system is that with trusted, the programmer is guaranteeing that it's actually safe regardless of what arguments it's given, whereas system is not only doing unsafe things, but whether they're ultimately safe or not depends on arguments and/or other code, so the programmer _can't_ guarantee that it's safe.

I realize you can, David Piepgrass was able to read into an idea which I didn't want to specifically state as it isn't as concerning. But why not limit trusted to only do safe and call trusted. But I could see that as an extra layer of annoying, but it would keep the unsafe arithmetic and parameter passing in the system functions. But again, not much thought on it and probably just be a layer of annoying. For your example of the proposal, when does the compiler mark save() as system over trusted? property auto save() { import std.conv; alias typeof((*_range).save) S; static assert(isForwardRange!S, S.stringof ~ " is not a forward range."); trusted { auto mem = new void[S.sizeof];` emplace!S(mem, cast(S)(*_range).save);` return RefRange!S(cast(S*)mem.ptr); } } You make a call to _range.save in the trusted block. So why isn't the function marked trusted? Having taken some more thought on the guarantee provided by safe even when calling trusted code, I do see the issue of templates always being trusted. As it could then be the logic in of the safe which is doing something unsafe. This could be true with trusted but at least the author had the opportunity to make it safe. But at this point I'm not seeing how this proposal is helping. You do want save() to be system when the underlining save() is system, and trusted otherwise? So I'm not seeing that here.
Jul 28 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis wrote:
  property auto save()
 {
     import std.conv;
     alias typeof((*_range).save) S;
     static assert(isForwardRange!S, S.stringof ~ " is not a 
 forward range.");

      trusted
     {
         auto mem = new void[S.sizeof];`
         emplace!S(mem, cast(S)(*_range).save);`
         return RefRange!S(cast(S*)mem.ptr);
     }
 }

 That is _way_ cleaner.

But unfortunately wrong – you call S.save in the trusted block… ;) David
Jul 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 22:08:42 David Nadlinger wrote:
 On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis wrote:
  property auto save()
 {
=20
     import std.conv;
     alias typeof((*_range).save) S;
     static assert(isForwardRange!S, S.stringof ~ " is not a
=20
 forward range.");
=20
      trusted
     {
    =20
         auto mem =3D new void[S.sizeof];`
         emplace!S(mem, cast(S)(*_range).save);`
         return RefRange!S(cast(S*)mem.ptr);
    =20
     }
=20
 }
=20
 That is _way_ cleaner.

But unfortunately wrong =E2=80=93 you call S.save in the trusted block=E2=80=A6 ;)

Yeah. I screwed that up. I was obviously in too much of a hurry when I = wrote=20 it. And actually, in this particular case, since the part that can't be= =20 trusted is in the middle of an expression doing system stuff, simply = using an=20 trusted block wouldn't do the trick. The basic principle is still good= =20 though. Being able to mark specific statements or sections of code tru= sted=20 would greatly simplify templated functions which are making function ca= lls=20 which _can't_ necessarily be trusted. - Jonathan M Davis
Jul 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 21:53:25 Jesse Phillips wrote:
 For your example of the proposal, when does the compiler mark
 save() as  system over  trusted?
 
  property auto save()
 {
     import std.conv;
     alias typeof((*_range).save) S;
     static assert(isForwardRange!S, S.stringof ~ " is not a
 forward range.");
 
      trusted
     {
         auto mem = new void[S.sizeof];`
         emplace!S(mem, cast(S)(*_range).save);`
         return RefRange!S(cast(S*)mem.ptr);
     }
 }
 
 You make a call to _range.save in the  trusted block. So why
 isn't the function marked  trusted?

I screwed up my changes there. I made them in a bit of a hurry. And actually, in this particular case, using an trusted block is still a bit entertaining, since the save call is in middle of an expression which is doing system stuff. But the principle is the same regardless. If you need to do system stuff in a function where you know that the system stuff that you're doing is safe but the function is also calling templated stuff which may or may not be safe, you can't mark the whole function as trusted, or you'll be marking code which is potentially truly unsafe as trusted. With an trusted block, you can mark sections of the code trusted and let the rest be properly inferred based on what the the template was instantiated with. In this particular case, it could still drastically reduce save, because only the line with save would need to be duplicated. So, it could become something more like property auto save() { import std.conv; alias typeof((*_range).save) S; static assert(isForwardRange!S, S.stringof ~ " is not a forward range."); trusted {auto mem = new void[S.sizeof];} static if(isSafelyCallable!((){(*_range).save;})) { trusted { emplace!S(mem, cast(S)(*_range).save); } } else { emplace!S(mem, cast(S)(*_range).save); } trusted {return RefRange!S(cast(S*)mem.ptr);} } It's still a bit ugly, but it's far better than what we can currently do. As it stands, you can't even put the emplace call in a separate function and mark it trusted or system depending on save, because the only way to mark the rest of the function as trusted is to mark the whole thing as trusted. You'd have to specifically put _everything else_ in its own function _and_ emplace in its own function (one trusted, one system) and have the outer function call both. It's a mess. Being able to mark statements as trusted rather than just entire functons would be _huge_. - Jonathan M Davis
Jul 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 10:02:43 Andrei Alexandrescu wrote:
 On 7/27/12 8:08 PM, David Nadlinger wrote:
 First, there is no point in having  trusted in the function signature.
 Why? From the perspective of the caller of the function in question,
  safe and  trusted mean exactly the same thing. If you are not convinced
 about that, just consider that you can wrap any  trusted function into a
  safe function to make it  safe, and vice versa.

If trusted is not part of the signature, we can't enable e.g. analyzers that verify an entire program or package to be safe. This is not something that's currently used, but I'd hate to look back and say, "heck, I hate that we conflated trusted with safe!"

But from the caller's perspective, safe and trusted are identical. And if you want a function to be safe rather than trusted, all you have to do is create an trusted helper function which has the implementation and have the actual function call it, and the the function can be safe. I don't see how it makes any difference at all whether a function is marked safe or trusted except for the fact that with trusted, the implementation can _directly_ use system stuff, whereas with safe, there has to be another layer in between. There's really no difference from the outside and no difference in terms of the actual level of safety. As far as I can tell, having trusted being treated differently in the function signature buys us nothing. All of the difference is in whether the compiler should error out on system stuff being use inside the functions. So, if we had trusted blocks, and trusted on a function meant that it was safe with the entire body being inside of an trusted block, as far as the caller and signature go, the behavior would be identical to now except that the mangling would be different.
 But the much bigger problem is that  trusted doesn't play well with
 template attribute inference and makes it much too easy to accidentally
 mark a function as safe to call if it really isn't. Both things are a
 consequence of the fact that it can be applied at the function level
 only; there is no way to apply it selectively to only a part of the
 function.

This could be a more serious problem. Could you please write a brief example that shows attribute deduction messing things up? I don't understand how marking a template as trusted is bad.

Marking pretty much _any_ templated function - especially range-based functions - as trusted is bad. If I have auto func(R)(R range) if(isForwardRange!R) { while(!r.empty && !condition) { //do system stuff } return range.front; } and func does system stuff which I _know_ is valid based on how it's used (emplace, scoped, pointer arithmetic, etc.), then ideally I'd be able to mark those operations as trusted, but what about front, popFront, empty, save, etc.? I don't know what they do internally. I don't know whether they're safe or system. So, I can't mark func as trusted, or I could be marking system code as trusted when it really isn't safe. Take the new std.range.RefRange for example. The body of Its save function is supposed to look like this: import std.conv; alias typeof((*_range).save) S; static assert(isForwardRange!S, S.stringof ~ " is not a forward range."); auto mem = new void[S.sizeof]; emplace!S(mem, cast(S)(*_range).save); return RefRange!S(cast(S*)mem.ptr); All of the stuff related to emplace can be marked as trusted, but save can't be marked as trusted because of that call to _range's save, which may or may not be safe. So, RefRange uses static ifs and mixins to create multiple versions of the function which are marked as system or trusted based on whether _range's save is system or safe. It's made even worse by the fact that constness isn't inferred (issue# 8407), but the problem with safe remains even if const or inout starts being inferred. In some cases, you can use helper functions which are marked as trusted to segregate the system code that you know is safe, but in others, that just doesn't work (or is really messy if it does), and you need to use static ifs to provide multiple versions of the function. It's exactly the kind of situation for which we introduced attribute inferrence in the first place, except in this case, you _can't_ use inferrence, because you obviously can't infer trusted. By being able to mark blocks of code as trusted instead of whole functions, the required code duplication is significantly reduced, and if the functions of unknown safety being called are separate enough from the code marked as trusted (i.e. not intertwined like they are with the line using save in RefRange's save), then _no_ code duplication is required. But even if the potentially unsafe code is intertwined with the trusted code, at least the code duplication can be restricted to just a few lines. So, with const/inout inferred and trusted blocks, RefRange's save should be able to become something like this; property auto save() { import std.conv; alias typeof((*_range).save) S; static assert(isForwardRange!S, S.stringof ~ " is not a forward range."); auto mem = new void[S.sizeof]; static if(isSafelyCallable!((){(*_range).save;})) trusted { emplace!S(mem, cast(S)(*_range).save); } else emplace!S(mem, cast(S)(*_range).save); trusted { return RefRange!S(cast(S*)mem.ptr); } } which is _far_ shorter than what it is now (around 50 lines of code), since you only need _one_ function declaration instead of four. - Jonathan M Davis
Jul 28 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 20:22:27 UTC, Jonathan M Davis wrote:
  property auto save()
 {
     import std.conv;
     alias typeof((*_range).save) S;
     static assert(isForwardRange!S, S.stringof ~ " is not a 
 forward range.");

      trusted {auto mem = new void[S.sizeof];}

     static if(isSafelyCallable!((){(*_range).save;}))
     {
          trusted { emplace!S(mem, cast(S)(*_range).save); }
     }
     else
     {
         emplace!S(mem, cast(S)(*_range).save);
     }

      trusted {return RefRange!S(cast(S*)mem.ptr);}
 }

 It's still a bit ugly, but it's far better than what we can 
 currently do. As
 it stands, you can't even put the emplace call in a separate 
 function and mark
 it  trusted or  system depending on save, because the only way 
 to mark the
 rest of the function as  trusted is to mark the whole thing as 
  trusted. You'd
 have to specifically put _everything else_ in its own function 
 _and_ emplace in
 its own function (one  trusted, one  system) and have the outer 
 function call
 both. It's a mess. Being able to mark statements as  trusted 
 rather than just
 entire functons would be _huge_.

For the specific case of calling unsafe functions from otherwise safe code, which occurs quite frequently due to parts of Phobos not being properly annotated yet, I've been experimenting with an alternative solution in my code. It allows you to »apply trusted« at a function call level, and is easily implemented in the library with today's D – using it RefRange.save would look like this using it: --- property auto save() { import std.conv; alias typeof((*_range).save) S; static assert(isForwardRange!S, S.stringof ~ " is not a forward range."); auto mem = new void[S.sizeof]; TRUSTED!(emplace!S)(mem, cast(S)(*_range).save); trusted return RefRange!S(cast(S*)mem.ptr); } --- TRUSTED is just a trusted template function which accepts a callable via an alias parameter and invokes it with the given parameters, adding trusted to the function type via a cast (an application of std.traits.SetFunctionAttributes, by the way). It seems to work fine, but there is an obvious catch: Just like trusted, TRUSTED circumvents safety, and all of its uses must be carefully examined. But in contrast to the former, it is not part of the language, so anyone working on the code base, especially reviewers, must be aware of implications. The all-caps name is meant to help drawing attention to . Maybe it would be a good idea to also allow ` trusted(emplace!S)(mem, cast(S)(*_range).save)`, with semantics similar to TRUSTED? Or even applying trusted to arbitrary expressions, similar to `checked` in C#? David
Jul 28 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 20:52:33 UTC, David Nadlinger wrote:
 The all-caps name is meant to help drawing attention to .

Oh, I should proof-read my posts before submitting, but I guess it was clear what I meant – because TRUSTED can only rely on convention, I chose a name which I hoped would immediately catch one's eye when skimming through the code. David
Jul 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 22:52:32 David Nadlinger wrote:
 Maybe it would be a good idea to also allow
 ` trusted(emplace!S)(mem, cast(S)(*_range).save)`, with semantics
 similar to TRUSTED? Or even applying  trusted to arbitrary
 expressions, similar to `checked` in C#?

Even the cast is unsafe. Basically, that entire line is system and needs to be trusted except for (*_range).save. Attribute inferrence needs to be used on that portion. So, aside from inventing yet more syntax like trusted { emplace!S(mem, cast(S) infer{(*_range).save}); } I don't know how you'd avoid the code duplication here. But at least with trusted blocks, it can be reduced to a single line rather than having to duplicate the whole function. - Jonathan M Davis
Jul 28 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 21:03:20 UTC, Jonathan M Davis wrote:
 On Saturday, July 28, 2012 22:52:32 David Nadlinger wrote:
 Maybe it would be a good idea to also allow
 ` trusted(emplace!S)(mem, cast(S)(*_range).save)`, with 
 semantics
 similar to TRUSTED? Or even applying  trusted to arbitrary
 expressions, similar to `checked` in C#?

Even the cast is unsafe. Basically, that entire line is system and needs to be trusted except for (*_range).save.

Aww, snap, missed that. Saving (*_range).save to a temporary would introduce an unnecessary copy, right? In any case, I've found TRUSTED to be convenient when dealing with incorrectly marked Phobos/C library functions, but I'm still unsure it is worth the added maintenance liabilities incurred by it being non-standard. It certainly isn't a replacement for trusted blocks. Maybe allowing to apply trusted at both expression and »block« level would really be an interesting direction: I've had a look at C#'s checked/unchecked keywords [1], which enable/disable integer overflow checking, and they work exactly like that. I must admit that I never actually used them in the little amount of C# code I wrote so far, but C# is generally regarded to be a well-designed language and similar enough to D that this makes me quite confident that implementing trusted like that could be pulled off without feeling overly alien. This still wouldn't solve your save() problem, though, as it acts »the wrong way« round, so I'm not sure if it would be worth the added complexity over allowing it just at the statement level… David [1] http://msdn.microsoft.com/en-us/library/74b4xzyw(v=vs.80).aspx
Jul 28 2012
prev sibling next sibling parent "David Piepgrass" <qwertie256 gmail.com> writes:
 On Saturday, July 28, 2012 22:08:42 David Nadlinger wrote:
 On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis 
 But unfortunately wrong – you call S.save in the  trusted 
 block… ;)

Yeah. I screwed that up. I was obviously in too much of a hurry when I wrote it. And actually, in this particular case, since the part that can't be trusted is in the middle of an expression doing system stuff, simply using an trusted block wouldn't do the trick.

Have you guys thought about the possibility that the language could simply not trust any calls that were resolved using a template argument? I'm a bit tired so I may be missing something, but it seems to me that (in a trusted template) if the compiler uses an instantiated template parameter (e.g. actual type Foo standing in for template parameter T) to choose a function to call, the compiler should require that the function be safe, based on the principle that a template cannot vouch for what it can't control. IOW, since a template can't predict what function actually gets called, the compiler should require whatever function gets called to be safe. If the programmer actually does want his template function to be able to call _unpredictable_ system functions, he should mark his template as system instead of trusted.
Jul 28 2012
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
Le 28/07/2012 02:08, David Nadlinger a écrit :
 As an example how this is problematic, consider that you are writing a
 function which takes some generic input data, and needs to do (unsafe)
 low-level buffer handling internally to efficiently do its job. You come
 up with a first implementation, maybe only accepting arrays for the sake
 of getting it working quickly, and add  trusted as your dirty buffer
 magic isn't visible from the outside, but does break attribute
 inference. Later, you decide that there is no reason not to take other
 range types as input. Fortunately, the actual implementation doesn't
 require any changes, so you just modify the template constraint as
 needed, and you are good. Well, no – you've just completely broken all
 safety guarantees for every program which calls your function, because
 empty/front/popFront of the passed range might be  system.

 Now, you might argue that this is a contrived scenario. Yes, the mistake
 could have easily be avoided,  trusted on a template declaration should
 always raise a red flag.

Run into that exact same problem this week. +1
Jul 29 2012
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 07/28/2012 02:08 AM, David Nadlinger wrote:
   1) Remove the distinction between  safe and  trusted at the interface
 (ABI, API) level. This implies changing the name mangling of  trusted to
 Nf, and consequently removing the distinction in DMD altogether (at
 least in user-facing parts like .stringof and error messages). In
 theory, this is a breaking change, but as any code that doesn't treat
 them the same is buggy anyway, it shouldn't be in practice. As for
 std.traits.FunctionAttribute, we could either make trusted an alias for
 safe, or just remove documentation for the former and keep it around for
 some time (there is no way to deprecate an enum member).

I haven't read through the whole thread, so pardon if that argument already came up. I think you're missing an important point here. SafeD is a subset of the language trusted is not. It's very reasonable to implement a compiler that supports only SafeD. It's probably not feasible to translate system to javascript for example [1]. Having the distinction at the API level makes perfectly sense for this. It's also another use case for attribute overloading ;-) [2]. [1]: http://forum.dlang.org/thread/yfmgvgprfpiquakiyjlk forum.dlang.org#post-yfmgvgprfpiquakiyjlk:40forum.dlang.org [2]: http://d.puremagic.com/issues/show_bug.cgi?id=9511
Feb 14 2013
next sibling parent Martin Nowak <code dawg.eu> writes:
On 02/15/2013 03:49 AM, David Nadlinger wrote:
 On Fri, Feb 15, 2013 at 3:31 AM, Martin Nowak <code dawg.eu> wrote:
 Having the distinction at the API level makes perfectly sense for this.

Sorry, but where is the argument here? David

If you switch the perspective trusted allows you to put a constrain on code. Especially you can chose to NOT trust. The SafeD compiler was not a good example because you have the source code at hand and could inspect it for trusted blocks. But if you think of Chromes Native Client it becomes extremely valuable that one could distinguish a function that requires trusting vs. a function that is safe at the ABI level. One could for example intercept and disallow function calls to trusted functions if a certain plugin is not trusted by the user.
Feb 14 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 15 February 2013 at 04:09:36 UTC, Martin Nowak wrote:
 On 02/15/2013 03:49 AM, David Nadlinger wrote:
 On Fri, Feb 15, 2013 at 3:31 AM, Martin Nowak <code dawg.eu>

 Having the distinction at the API level makes perfectly


 Sorry, but where is the argument here?

 David

If you switch the perspective trusted allows you to put a constrain on code. Especially you can chose to NOT trust. The SafeD compiler was not a good example because you have the source code at hand and could inspect it for trusted blocks. But if you think of Chromes Native Client it becomes extremely valuable that one could distinguish a function that requires trusting vs. a function that is safe at the ABI level. One could for example intercept and disallow function calls to trusted functions if a certain plugin is not trusted by the user.

This is totally irrelevant to the current subject, as trusted code can be called from safe code. You are asking for another functionality here.
Feb 14 2013
prev sibling next sibling parent "Timon Gehr" <timon.gehr hotmail.com> writes:
On Friday, 15 February 2013 at 02:31:08 UTC, Martin Nowak wrote:
 ...
 It's probably not feasible to translate  system to javascript 
 for example [1].
 ...

Yes it's feasible.
Feb 14 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 15 February 2013 at 06:27:55 UTC, Timon Gehr wrote:
 On Friday, 15 February 2013 at 02:31:08 UTC, Martin Nowak wrote:
 ...
 It's probably not feasible to translate  system to javascript 
 for example [1].
 ...

Yes it's feasible.

http://asmjs.org/spec/latest/
Feb 15 2013
prev sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Saturday, 28 July 2012 at 00:08:30 UTC, David Nadlinger wrote:
  trusted in its current form needs to go. Its design is badly 
 broken, as it leaks implementation details and encourages 
 writing unsafe code.

 The Problem
 ———————————

 [...]

 A Solution
 ——————————

 Let me make something clear first: I am _not_ intending to 
 remove  trusted from the language. As a bridge between the 
  safe and  system worlds, it is an integral part of SafeD. What 
 I'm proposing is:

  1) Remove the distinction between  safe and  trusted at the 
 interface (ABI, API) level. This implies changing the name 
 mangling of  trusted to Nf, and consequently removing the 
 distinction in DMD altogether (at least in user-facing parts 
 like .stringof and error messages). In theory, this is a 
 breaking change, but as any code that doesn't treat them the 
 same is buggy anyway, it shouldn't be in practice. As for 
 std.traits.FunctionAttribute, we could either make trusted an 
 alias for safe, or just remove documentation for the former and 
 keep it around for some time (there is no way to deprecate an 
 enum member).

  2) The first step is necessary, but mainly of cosmetic nature 
 (think `pure`, `pure2`). We still need to address for the 
 granularity and attribute inference problem. The obvious 
 solution is to add a " trusted" declaration/block, which would 
 allow unsafe code in a certain region. Putting  trusted in the 
 function header would still be allowed for backwards 
 compatibility (but discouraged), and would have the same effect 
 as marking the function  safe and wrapping its whole body in a 
  trusted block. It could e.g. look something like this (the   
 prefix definitely looks weird, but I didn't want to introduce a 
 new keyword):

 ---
  void foo(T)(T t) {
    t.doSomething();
     trusted {
      // Do something dirty.
    }
    t.doSomethingElse();
     trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
  }
 ---

 This is similar to other constructs we have today, for example 
 debug {}, which allows impure code. It can be debated whether a 
 block »argument« should introduce a new scope or not (like 
 static if). The latter currently seems much more attractive to 
 me, but I suppose it could be confusing for some.

 In any case, while there is probably quite a bit of 
 bikeshedding to be done for 2), I don't think there is much 
 controversy about 1). So, let's try to get this done shortly 
 after the 2.060 release – as discussed above, it is very 
 unlikely that the change will break something, but the odds 
 still increase over time. Also, there is currently a Phobos 
 pull request [4] which will be influenced by the outcome of 
 this discussion.

I completely agree with the above. trusted should be applied to as small blocks of code as possible -- preferably to individual statements -- to make it easier for the programmer to verify and maintain the safety of the code. Lars
Feb 16 2013