www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Non-techincal brain, is safe by default good or not?

reply aberba <karabutaworld gmail.com> writes:
So I'm trying to wrap my head around the DIP 1028 thing here. I'm 
aware  live was introduced to give us rust-like safety. That 
seemed cool as long as it was purely opt-in.

Now I hearing  safe by default which reads like the plan has 
changed and now the direction is going all-in on everything MUST 
BE SAFE. After reading the DIP, I getting a feeling I'll need to 
re-think my programming model to make everything safe. Sound in 
my understanding like most code will break. Communist/Socialist 
kind of coding.

Aside issues with how leadership manage communication and stuff, 
is this move, as I understand it, good or not? (both practical 
everyday use and existing code bases). I'm not getting what 
feedback kicked off this move to  safe by default.

Please help me understand.
May 27
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 27 May 2020 at 12:59:12 UTC, aberba wrote:
 So I'm trying to wrap my head around the DIP 1028 thing here. 
 I'm aware  live was introduced to give us rust-like safety. 
 That seemed cool as long as it was purely opt-in.

 Now I hearing  safe by default which reads like the plan has 
 changed and now the direction is going all-in on everything 
 MUST BE SAFE. After reading the DIP, I getting a feeling I'll 
 need to re-think my programming model to make everything safe. 
 Sound in my understanding like most code will break. 
 Communist/Socialist kind of coding.

 Aside issues with how leadership manage communication and 
 stuff, is this move, as I understand it, good or not? (both 
 practical everyday use and existing code bases). I'm not 
 getting what feedback kicked off this move to  safe by default.

 Please help me understand.
safe by default is good because it encourages code that already *is* safe to be *treated* as safe. Currently, a lot of D code is written that could be safe, but isn't, because the programmer can't be bothered to actually mark it with the safe attribute. When safe is made the default, this won't be a problem. If you write safe code, the compiler will treat it as safe, without you having to put in any extra effort. Almost everybody agrees that this is a good thing. The problem with DIP 1028 is that it also applies safe-by-default to external function declarations, including extern(C) functions like `printf` and `memcpy` that absolutely *shouldn't* be treated as safe. Many, many people, including Andrei Alexandrescu, who used to be one of the Language Maintainers, have been trying to explain to Walter why this is a problem. So far, he remains unconvinced.
May 27
prev sibling next sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Wednesday, 27 May 2020 at 12:59:12 UTC, aberba wrote:
 So I'm trying to wrap my head around the DIP 1028 thing here. 
 I'm aware  live was introduced to give us rust-like safety. 
 That seemed cool as long as it was purely opt-in.

 Now I hearing  safe by default which reads like the plan has 
 changed and now the direction is going all-in on everything 
 MUST BE SAFE. After reading the DIP, I getting a feeling I'll 
 need to re-think my programming model to make everything safe. 
 Sound in my understanding like most code will break. 
 Communist/Socialist kind of coding.

 Aside issues with how leadership manage communication and 
 stuff, is this move, as I understand it, good or not? (both 
 practical everyday use and existing code bases). I'm not 
 getting what feedback kicked off this move to  safe by default.

 Please help me understand.
You will only get opinions, not fact. So here's mine: It depends. It depends on the kind of code you are writing. Could you write probably a good thing for you. Now, are you writing code that needs to be careful about allocation, and needs to have low latency / high performance ? You're likely to have a bunch of un-` safe` tricks all over the place, to bypass bounds checking, make the code easier to optimize, or avoid branches. Walter himself rejected a change that would have made code ` safe` because it would also make it slower (https://github.com/dlang/dmd/pull/10940#issuecomment-600894940). But not everyone is in this craft, either. Maybe you just want to write a library for people to use ? Well I got a bad news for you: Composing attributes doesn't work well with OOP or delegates (contravariance for delegate is not implemented, nor is it possible to tie a function's attributes to its delegate parameter). So you end up having to choose: Does your callback / interface expose a ` safe` / `nothrow` / ` nogc` / `pure` interface, or not ? If it does, you either need to **force** the caller to use ` safe` & co as well, or need to template everything, rendering the use of OOP useless. And the same issues with OOP / delegate affect application code, of course, but add to that the fact that most libraries out there had to choose a set of attributes, and you are locked in a weird place. Next issue: if you want to quickly experiment / prototype, ` safe` by default is likely to get in your way. So you'll end up putting ` system` or ` trusted` everywhere just to satisfy the compiler. Even if you are conscious and **try** to put ` system`, remember that libraries will have to make a choice (and that choice will be ` safe`), so yeah, you're going to greenwash to be able to provide callbacks. And obviously, there's the case of people just copy-pasting their C[++] prototype in their module to get things working and then not wanting to deal with the issue of making things correct. As you can guess, while I said that "It depends", I am personally in the library author / prototyping / low level code category and I think it's a terrible choice. And as someone who has a decent professional and personal investment in the language, I also think it goes against almost everything that makes me, and I believe many other people, choose the language: - ~Great~ Amazing integration with C / C++; - Easy prototyping yet almost production-ready code; - A language that doesn't get in your way, where you pay as you go; All of the above would be compromised by making ` safe` the default, *especially* in the way DIP1028 plans it. If ` safe` didn't get in the way, I would be much more inclined to cheer for it.
May 27
parent reply aberba <karabutaworld gmail.com> writes:
On Wednesday, 27 May 2020 at 14:23:30 UTC, Mathias LANG wrote:
 On Wednesday, 27 May 2020 at 12:59:12 UTC, aberba wrote:
 [...]
You will only get opinions, not fact. So here's mine: It depends. It depends on the kind of code you are writing. Could you write probably a good thing for you. Now, are you writing code that needs to be careful about allocation, and needs to have low latency / high performance ? You're likely to have a bunch of un-` safe` tricks all over the place, to bypass bounds checking, make the code easier to optimize, or avoid branches. Walter himself rejected a change that would have made code ` safe` because it would also make it slower (https://github.com/dlang/dmd/pull/10940#issuecomment-600894940). But not everyone is in this craft, either. Maybe you just want to write a library for people to use ? Well I got a bad news for you: Composing attributes doesn't work well with OOP or delegates (contravariance for delegate is not implemented, nor is it possible to tie a function's attributes to its delegate parameter). So you end up having to choose: Does your callback / interface expose a ` safe` / `nothrow` / ` nogc` / `pure` interface, or not ? If it does, you either need to **force** the caller to use ` safe` & co as well, or need to template everything, rendering the use of OOP useless. And the same issues with OOP / delegate affect application code, of course, but add to that the fact that most libraries out there had to choose a set of attributes, and you are locked in a weird place. Next issue: if you want to quickly experiment / prototype, ` safe` by default is likely to get in your way. So you'll end up putting ` system` or ` trusted` everywhere just to satisfy the compiler. Even if you are conscious and **try** to put ` system`, remember that libraries will have to make a choice (and that choice will be ` safe`), so yeah, you're going to greenwash to be able to provide callbacks. And obviously, there's the case of people just copy-pasting their C[++] prototype in their module to get things working and then not wanting to deal with the issue of making things correct. As you can guess, while I said that "It depends", I am personally in the library author / prototyping / low level code category and I think it's a terrible choice. And as someone who has a decent professional and personal investment in the language, I also think it goes against almost everything that makes me, and I believe many other people, choose the language: - ~Great~ Amazing integration with C / C++; - Easy prototyping yet almost production-ready code; - A language that doesn't get in your way, where you pay as you go; All of the above would be compromised by making ` safe` the default, *especially* in the way DIP1028 plans it. If ` safe` didn't get in the way, I would be much more inclined to cheer for it.
That's what I'm thinking...it seems the interest in taking too from rust seems very concerning. Walter himself has said many times he does not write application/library code. My worst fear is engineering gone bad. Where technical people build a technology with technical judgement only to fail at it because it ends up being too difficult to use for everyday code. And that's how I see things. When its said that MANY people agree that safe by default is THE way to go, I wonder if its based on some few vocal people or its an accurate representation of a general interest in direction.
 You will only get opinions, not fact.
That's very concerning. Are the MANY vocal ones using D in system code or application, production or hobby? This all shapes ones opinion.
May 27
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 27 May 2020 at 15:57:12 UTC, aberba wrote:
 On Wednesday, 27 May 2020 at 14:23:30 UTC, Mathias LANG wrote:
 [...]
That's what I'm thinking...it seems the interest in taking too from rust seems very concerning. Walter himself has said many times he does not write application/library code. My worst fear is engineering gone bad. Where technical people build a technology with technical judgement only to fail at it because it ends up being too difficult to use for everyday code. And that's how I see things. When its said that MANY people agree that safe by default is THE way to go, I wonder if its based on some few vocal people or its an accurate representation of a general interest in direction.
 [...]
That's very concerning. Are the MANY vocal ones using D in system code or application, production or hobby? This all shapes ones opinion.
I think it's simpler. Currently, safe means "machine checked". Post 1028 safe means "machine checked unless you call a C library, or anything you call calls a C library or anything that...". Post 1028 the compiler treats all unmarked extern(C) routines as if they had been machine verified. Since they have not been machine verified, and in most cases can not be machine verified even if the source code were available, the post 1028 compiler is complicit in propagating dangerous code. To be clear, I like safe by default. I just don't like the compiler lying to me. DIP 1028 could be altered to remove the problem but Walter, at least to date, refuses.
May 27
parent Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 27 May 2020 at 16:13:31 UTC, Bruce Carneal wrote:
 On Wednesday, 27 May 2020 at 15:57:12 UTC, aberba wrote:
 [...]
I think it's simpler. Currently, safe means "machine checked". Post 1028 safe means "machine checked unless you call a C library, or anything you call calls a C library or anything that...". Post 1028 the compiler treats all unmarked extern(C) routines as if they had been machine verified. Since they have not been machine verified, and in most cases can not be machine verified even if the source code were available, the post 1028 compiler is complicit in propagating dangerous code. To be clear, I like safe by default. I just don't like the compiler lying to me. DIP 1028 could be altered to remove the problem but Walter, at least to date, refuses.
To answer your question more directly: I believe safe by default could be a wrenching change for current dlang coders but would be especially beneficial to less experienced newcomers in the future. OTOH, safe by default as proposed in DIP 1028 would be a problem for almost everyone in the dlang community.
May 27
prev sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
On Wednesday, 27 May 2020 at 12:59:12 UTC, aberba wrote:

 Now I hearing  safe by default which reads like the plan has 
 changed and now the direction is going all-in on everything 
 MUST BE SAFE. After reading the DIP, I getting a feeling I'll 
 need to re-think my programming model to make everything safe. 
 Sound in my understanding like most code will break. 
 Communist/Socialist kind of coding.
'everyday programming' in D is almost certainly memory safe, except for any c libraries you might use. It's exactly the proposition of DIP 1028 that such code continues to work, except that now the D code is verified for mistakes that might cause memory corruption. As it stands, I believe you don't really have to change much if anything at all, virtually all code will continue to compile and when it doesn't, it's a good thing because then you really are doing something dangerous. You don't really have to change the way you code or re-think your programming model unless you write systems code. But if you do, then you are likely already aware of the issues. It seems most people don't like this, as they feel it's deceiving that the compiler marks such everyday code as memory safe when it frivolously calls into unsafe C code. But that's the controversy - you asked about the DIP itself.
May 28
next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Thursday, 28 May 2020 at 07:30:28 UTC, Lutger wrote:
 On Wednesday, 27 May 2020 at 12:59:12 UTC, aberba wrote:

 [...]
'everyday programming' in D is almost certainly memory safe, except for any c libraries you might use. [...]
It's not so easy ... just try to use big parts of Phobos for common idioms, variants, concurrency, and your code will flourish of casts and trusted function ...
May 28
prev sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Thursday, 28 May 2020 at 07:30:28 UTC, Lutger wrote:
 On Wednesday, 27 May 2020 at 12:59:12 UTC, aberba wrote:

 Now I hearing  safe by default which reads like the plan has 
 changed and now the direction is going all-in on everything 
 MUST BE SAFE. After reading the DIP, I getting a feeling I'll 
 need to re-think my programming model to make everything safe. 
 Sound in my understanding like most code will break. 
 Communist/Socialist kind of coding.
'everyday programming' in D is almost certainly memory safe, except for any c libraries you might use.
I write D on a daily basis. This is most certainly not my experience. I hear this claim over and over, yet I see no evidence of it being true. So could you substantiate your claim ? Let me provide some substance to my own claim. Just right now, I tried to use Phobos' `std.bitmanip.BitArray`. A bit array is most certainly ` safe`, right ? Nope, absolutely not. Okay, that module is old, let's use something that should be prominent: `std.json`: ``` import std.json; struct MyCustomType { public string toString () const system { return null; } alias toString this; } void main () system { JSONValue json; MyCustomType ilovedlang; json = ilovedlang; } ``` Results in: ``` /usr/local/opt/dmd/include/dlang/dmd/std/json.d(459): Error: safe function std.json.JSONValue.assign!(MyCustomType).assign cannot call system function foo.MyCustomType.toString foo.d(5): foo.MyCustomType.toString is declared here /usr/local/opt/dmd/include/dlang/dmd/std/json.d(593): Error: template instance std.json.JSONValue.assign!(MyCustomType) error instantiating foo.d(13): instantiated from here: opAssign!(MyCustomType) ``` But... WHY ? My main is ` system`, why do I get a ` safe` error when I just want to prototype something (in practice, I would not put any attribute on `toString` but wanted to drive the point home). Well because someone decided that `std.json` should be ` safe` and we can't compose attributes easily (as explained before). Well that right here is our future with ` safe` by default. But surely we achieved ` safe`ty, you think ? Well, at what expense ? If you look at the implementation of `std.json` (https://github.com/dlang/phobos/blob/5ebf458b509963725e1143b733fba1b22f22ed3f/s d/json.d#L450-L533) you'll see that it essentially have to duplicate all its argument to ensure safety (otherwise alias this to function would defeat it). Including associative arrays. And yet... No we're not ` safe` yet. Some methods (including `opApply`) are ` system`, which means things like: ``` import std.json; void main () safe { JSONValue json; json = [ "Un", "Deux", "trois", "quatre" ]; foreach (idx, val; json.array) { } } ``` Are not ` safe`. You can see it for a few other functions (e.g. `object`). Pick any module in Phobos, and you will find one of three things: - It is trivial (no user callback, only value types, etc...) - It can only be used by either ` system` or ` safe` users, not both; - It is either overly ` trusted` or everything under the sun is templated; Bonus point: I kinda wanted to find a ` safe`ty failure in Phobos to illustrate things better. `std.json` unfortunately looked fairly solid (because of how limited it is), so I went to the catch-all: `std.algorithm`, more precisely `std.algorithm.mutation`, because, you know, mutation. So here we go: ``` import std.algorithm.mutation; struct S { void opPostMove(const ref S old) system nothrow pure { int* ptr = cast(int*)42; *ptr = 42; } int a; } void main () safe nothrow { S s1; s1.a = 41; S s2 = move(s1); // BOOM assert(s2.a == 42); } ``` This compiles and crash just fine.
May 28
next sibling parent Tove <tove fransson.se> writes:
On Thursday, 28 May 2020 at 10:28:47 UTC, Mathias LANG wrote:
 ```
 import std.json;

 struct MyCustomType
 {
     public string toString () const  system { return null; }
     alias toString this;
 }

 void main ()  system
 {
     JSONValue json;
     MyCustomType ilovedlang;
     json = ilovedlang;
 }
 ```

 Results in:
 ```
 /usr/local/opt/dmd/include/dlang/dmd/std/json.d(459): Error:
Holy insanity... 100% of my D code is system, I was thinking I can avoid this entire DIP by, simply adding " system:" in my files and live happily ever after, so I kept quiet. Now I see why some people was asking for D3. The Top-1 reason why my code is system is because D doesn't allow function local "ref". So I use pointers instead, it's pretty nice since you don't need to use "->". Maybe something like this could work? system import std.json; Or a ton of fixes is needed in phobos and many 3rd-party libraries also...
May 28
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 28.05.20 12:28, Mathias LANG wrote:
 
 Results in:
 ```
 /usr/local/opt/dmd/include/dlang/dmd/std/json.d(459): Error:  safe 
 function std.json.JSONValue.assign!(MyCustomType).assign cannot call 
  system function foo.MyCustomType.toString
 foo.d(5):        foo.MyCustomType.toString is declared here
 /usr/local/opt/dmd/include/dlang/dmd/std/json.d(593): Error: template 
 instance std.json.JSONValue.assign!(MyCustomType) error instantiating
 foo.d(13):        instantiated from here: opAssign!(MyCustomType)
 ```
 
 But... WHY ?
Indeed. Why is it even annotated? It would be inferred anyway...
May 28
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/28/20 6:28 AM, Mathias LANG wrote:
 
 import std.algorithm.mutation;
 struct S
 {
      void opPostMove(const ref S old)  system nothrow pure
      {
          int* ptr = cast(int*)42;
          *ptr = 42;
      }
      int a;
 }
 void main ()  safe nothrow
 {
      S s1;
      s1.a = 41;
      S s2 = move(s1); // BOOM
      assert(s2.a == 42);
 }
 ```
 
 This compiles and crash just fine.
At least this is a legit bug report.
May 28
parent Mathias LANG <geod24 gmail.com> writes:
On Thursday, 28 May 2020 at 10:46:08 UTC, Andrei Alexandrescu 
wrote:
 On 5/28/20 6:28 AM, Mathias LANG wrote:
 
 import std.algorithm.mutation;
 struct S
 {
      void opPostMove(const ref S old)  system nothrow pure
      {
          int* ptr = cast(int*)42;
          *ptr = 42;
      }
      int a;
 }
 void main ()  safe nothrow
 {
      S s1;
      s1.a = 41;
      S s2 = move(s1); // BOOM
      assert(s2.a == 42);
 }
 ```
 
 This compiles and crash just fine.
At least this is a legit bug report.
Well since you like it (filed as https://issues.dlang.org/show_bug.cgi?id=20869) I went ahead and did a quick search on trusted: - https://issues.dlang.org/show_bug.cgi?id=20870 - https://issues.dlang.org/show_bug.cgi?id=20871 - https://issues.dlang.org/show_bug.cgi?id=20872 - https://issues.dlang.org/show_bug.cgi?id=20873 But as fun as it is to find those in Phobos, it's nothing in comparison of the language holes we still have. E.g. today Andrej had to track down a segfault: https://issues.dlang.org/show_bug.cgi?id=20868
May 28
prev sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
Am 28.05.20 um 12:28 schrieb Mathias LANG:
 ```
 import std.json;
 
 struct MyCustomType
 {
     public string toString () const  system { return null; }
     alias toString this;
 }
 
 void main ()  system
 {
     JSONValue json;
     MyCustomType ilovedlang;
     json = ilovedlang;
 }
 ```
 
 Results in:
 ```
 /usr/local/opt/dmd/include/dlang/dmd/std/json.d(459): Error:  safe
 function std.json.JSONValue.assign!(MyCustomType).assign cannot call
  system function foo.MyCustomType.toString
 foo.d(5):        foo.MyCustomType.toString is declared here
 /usr/local/opt/dmd/include/dlang/dmd/std/json.d(593): Error: template
 instance std.json.JSONValue.assign!(MyCustomType) error instantiating
 foo.d(13):        instantiated from here: opAssign!(MyCustomType)
 ```
Please file a bug report. `assign` is a template, it should not be explicitly annotated with safe but instead the attributes should be inferred.
 ```
 import std.algorithm.mutation;
 struct S
 {
     void opPostMove(const ref S old)  system nothrow pure
     {
         int* ptr = cast(int*)42;
         *ptr = 42;
     }
     int a;
 }
 void main ()  safe nothrow
 {
     S s1;
     s1.a = 41;
     S s2 = move(s1); // BOOM
     assert(s2.a == 42);
 }
 ```
 
 This compiles and crash just fine.
Again, please file a bug report. This is either a bug with safe or there is some trusted in `move` (or one of the functions it calls) that calls `opPostMove`. Both of the examples you provide are bugs but I realize that there are real examples of what you are trying to show. One situation where this happens is if libraries have non-template functions that take a callback as parameter. The library author has to make a decision whether to make his function safe, which requires all callbacks passed to it also to be safe, or to make it system, which makes it unusable in safe code, even if it actually is safe because the callback that was passed is safe. trusted allows both kinds of usages but is a really bad idea because it allows unchecked system code to be called from safe code. This is why some people in the community have suggested to allow attribute inference also for non-template functions. That would solve the issue might might be a better default than both system and safe. However, I don't think anybody has thought that through completely yet. Some people have voiced concerns regarding inheritance and .di files (and probably more) but I think it might be possible to do this.
May 28
parent Mathias LANG <geod24 gmail.com> writes:
On Thursday, 28 May 2020 at 11:07:16 UTC, Johannes Loher wrote:
 Again, please file a bug report. This is either a bug with 
  safe or there is some  trusted in `move` (or one of the 
 functions it calls) that calls `opPostMove`.
Oh I know exactly how it happened. I just went over the code quickly, saw ` trusted`, and saw it calling a user-supplied hook, and crafting a test case was trivial. This is because, at the top level, there is a function which checks if the destruction is ` safe` (well, it checks quite a few other things actually, but that's not the point here), and if it is, just blindly trust a lower function, which itself happens to be more correct in its usage of trusted. That's the technical explanation. The higher level explanation is that we had code that was written correctly, audited, then a language feature was added, and support for it was added by different contributors over time. And the *whole* code was not re-audited for ` trusted`.
 Both of the examples you provide are bugs but I realize that 
 there are real examples of what you are trying to show. One 
 situation where this happens is if libraries have non-template 
 functions that take a callback as parameter. The library author 
 has to make a decision whether to make his function  safe, 
 which requires all callbacks passed to it also to be  safe, or 
 to make it  system, which makes it unusable in  safe code, even 
 if it actually is safe because the callback that was passed is 
  safe.  trusted allows both kinds of usages but is a really bad 
 idea because it allows unchecked  system code to be called from 
  safe code.
Yes, that's a point I tried to make over and over, but it seems no one was interested. And as mentioned, it also affects OOP code. Slight correction though: Accepting a ` trusted` delegate will **not** allow you to pass a ` system` delegate. I tried.
 This is why some people in the community have suggested to 
 allow attribute inference also for non-template functions. That 
 would solve the issue might might be a better default than both 
  system and  safe. However, I don't think anybody has thought 
 that through completely yet. Some people have voiced concerns 
 regarding inheritance and .di files (and probably more) but I 
 think it might be possible to do this.
That's a terrible idea: - It will be a massive slowdown for anyone not compiling all-at-once: at the moment the frontend knows about "root module", which are module for which the compiler will do codegen. As such, it doesn't do any semantic analysis on functions that are not in root modules (except for the prototype). Inference for everything means you have to perform full semantic analysis for everything. Compiling by packages means performing full semantic for your whole program multiple times over. It also means whatever function in Phobos, every D program will also need to perform semantic analysis on them if it imports it. The compiler could (should) be smarter and have lazier semantic, but we need to implement it, and at the core, this suggestion does not scale. - You would have to either disable that for OOP code, which would be a weird special case, or find a way to negate it, or ... ? I can't even picture a way to make this work. If you have a class, and your base implementation of `toString` just returns a literal, it has all attributes available, however derived code might want to use `format`. - It doesn't solve the actual problem: You still will write your function prototype as `void requestHTTP(scope void delegate(scope ref HTTPRequest) /* safe */ req)`. You need to express a way to tie `requestHTTP`'s attributes to `req`'s attributes , not just infer `req`'s attributes. And the way you do this needs to be subtle, because you might want to express: "I am ` safe` is this is ` safe`" and "I am `nothrow` no matter what" as well as "I am not ` nogc` even if `req` is". At the moment we can express the second and the third variant but not the first. IMO the real answer is to come up with something like `inout` for attributes. `inout` is a wildcard. It's exactly what you want for delegates. It doesn't solve the issue with OOP though, although it makes it easier to mitigate it (see `Throwable.toString` for how you could have a conditionally ` nogc` / `nothrow` / etc... method).
May 28