www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Direction for safe/-dip1000

reply Florian Weimer <fw deneb.enyo.de> writes:
I've tried to figure out where this is heading. Is the eventual 
goal (irrespective of mechanism) that sticking ` safe` onto the 
`main` function will ensure memory safety for the whole program?

What about data races in updates of multi-word variables? For 
example, the Go language makes avoiding them the responsibility 
of the programmer, but Java ensures that data races cannot 
subvert the type system (although with the demise of the security 
manager, some people argue that this should be changed).
Feb 13
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 13.02.22 12:15, Florian Weimer wrote:
 I've tried to figure out where this is heading. Is the eventual goal 
 (irrespective of mechanism) that sticking ` safe` onto the `main` 
 function will ensure memory safety for the whole program?
Roughly, that's what safe is supposed to be, yes. But it's not quite as simple because `main` isn't the only possible entry point, and because trusted exists.
 What about data races in updates of multi-word variables?
As far as I know, this is D's current approach: All shared data must be marked with `shared`. safe code cannot access `shared` data (only enforced with `-preview=nosharedaccess`). If you want to mess with `shared` data, you need to write system/ trusted code and ensure thread safety yourself. Since you mentioned it in the topic, note that DIP 1000 doesn't make safe safer. It allows some formerly system code to become safe. If you write that kind of code, DIP 1000 is nice to have. If you don't, DIP 1000 doesn't affect you.
Feb 13
parent reply Florian Weimer <fw deneb.enyo.de> writes:
On Sunday, 13 February 2022 at 12:26:52 UTC, ag0aep6g wrote:
 On 13.02.22 12:15, Florian Weimer wrote:
 I've tried to figure out where this is heading. Is the 
 eventual goal (irrespective of mechanism) that sticking 
 ` safe` onto the `main` function will ensure memory safety for 
 the whole program?
Roughly, that's what safe is supposed to be, yes. But it's not quite as simple because `main` isn't the only possible entry point, and because trusted exists.
Right, but it's still a big step forward.
 Since you mentioned it in the topic, note that DIP 1000 doesn't 
 make  safe safer. It allows some formerly  system code to 
 become  safe. If you write that kind of code, DIP 1000 is nice 
 to have. If you don't, DIP 1000 doesn't affect you.
Hmm. This compiles without `-dip1000` (or `-ftransition=dip1000`), but is unsafe: ```d int[] global; safe void f0(int[] val) { global = val; } safe void f1() { int[3] local = [1, 2, 3]; f0(local); } ``` It's rejected in DIP 1000 mode (both by DMD and GDC).
Feb 13
parent ag0aep6g <anonymous example.com> writes:
On 13.02.22 13:37, Florian Weimer wrote:
 Hmm.  This compiles without `-dip1000` (or `-ftransition=dip1000`), but 
 is unsafe:
 
 ```d
 int[] global;
 
  safe
 void f0(int[] val) {
      global = val;
 }
 
  safe
 void f1() {
    int[3] local = [1, 2, 3];
    f0(local);
 }
 
 ```
 
 It's rejected in DIP 1000 mode (both by DMD and GDC).
Yeah, bugs are plenty. And if a bug doesn't manifest with `-preview=dip1000`, then people are even less likely than usual to give a damn about it. Because DIP 1000 is going to become the default eventually, and then the issue will go away anyway. It goes the other way, too: `-preview=dip1000` has safety holes that aren't there without the switch. But those are more likely to get fixed, because some people do care about getting DIP 1000 done.
Feb 13
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/13/2022 3:15 AM, Florian Weimer wrote:
 I've tried to figure out where this is heading. Is the eventual goal 
 (irrespective of mechanism) that sticking ` safe` onto the `main` function
will 
 ensure memory safety for the whole program?
Yes, although safe does not supply complete memory safety. The addition of live fills in much of the rest.
 What about data races in updates of multi-word variables? For example, the Go 
 language makes avoiding them the responsibility of the programmer, but Java 
 ensures that data races cannot subvert the type system (although with the
demise 
 of the security manager, some people argue that this should be changed).
It's up to the programmer to avoid data races in multi-word variables, likely by manually using locks.
Feb 14
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright wrote:
 On 2/13/2022 3:15 AM, Florian Weimer wrote:
 I've tried to figure out where this is heading. Is the 
 eventual goal (irrespective of mechanism) that sticking 
 ` safe` onto the `main` function will ensure memory safety for 
 the whole program?
Yes, although safe does not supply complete memory safety. The addition of live fills in much of the rest.
Huh? My understanding is that modulo compiler bugs and incorrect use of trusted, safe code should be 100% memory safe, even without live. What adding an ownership/borrowing system does (or should do) is, like DIP 1000, make it possible to do things in safe code that previously required system/ trusted--in this case, things like manually freeing memory.
Feb 14
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Feb 14, 2022 at 01:15:26PM +0000, Paul Backus via Digitalmars-d wrote:
 On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright wrote:
[...]
 Yes, although  safe does not supply complete memory safety. The
 addition of  live fills in much of the rest.
Huh? My understanding is that modulo compiler bugs and incorrect use of trusted, safe code should be 100% memory safe, even without live.
The problem with safe as it is implemented today is that it's implemented as a blacklist rather than a whitelist. Cf. points 2 and 3 (as applied to memory safety) in: http://ranum.com/security/computer_security/editorials/dumb/index.html T -- Once the bikeshed is up for painting, the rainbow won't suffice. -- Andrei Alexandrescu
Feb 14
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Monday, 14 February 2022 at 15:50:31 UTC, H. S. Teoh wrote:
 On Mon, Feb 14, 2022 at 01:15:26PM +0000, Paul Backus via 
 Digitalmars-d wrote:
 On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright 
 wrote:
[...]
 Yes, although  safe does not supply complete memory safety. 
 The addition of  live fills in much of the rest.
Huh? My understanding is that modulo compiler bugs and incorrect use of trusted, safe code should be 100% memory safe, even without live.
The problem with safe as it is implemented today is that it's implemented as a blacklist rather than a whitelist.
I did say "should be" and "modulo compiler bugs" for a reason. :) Even with a whitelist implementation, though, we'd still have bugs where something was accidentally whitelisted that shouldn't have been. Several of the recent fixes to -preview=dip1000 are for exactly this type of bug, for example.
Feb 14
prev sibling parent IGotD- <nise nise.com> writes:
On Monday, 14 February 2022 at 15:50:31 UTC, H. S. Teoh wrote:
 The problem with  safe as it is implemented today is that it's 
 implemented as a blacklist rather than a whitelist.

 Cf. points 2 and 3 (as applied to memory safety) in:

 	http://ranum.com/security/computer_security/editorials/dumb/index.html
I think safe has done a fundamental error to allow raw pointers. In safe code the memory management should be completely opaque more simple to work with. Instead D does the opposite and adds stuff which makes memory management more complicated.
Feb 14
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 2/14/22 14:15, Paul Backus wrote:
 On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright wrote:
 On 2/13/2022 3:15 AM, Florian Weimer wrote:
 I've tried to figure out where this is heading. Is the eventual goal 
 (irrespective of mechanism) that sticking ` safe` onto the `main` 
 function will ensure memory safety for the whole program?
Yes, although safe does not supply complete memory safety. The addition of live fills in much of the rest.
Huh? My understanding is that modulo compiler bugs and incorrect use of trusted, safe code should be 100% memory safe, even without live. ...
Exactly, incorrect use of trusted. live is only useful in trusted or system code as a linting tool.
 What adding an ownership/borrowing system does (or should do)
live does not do what an ownership/borrowing system is supposed to do. I have tried to make this point many times, but most people seem to still assume the opposite. I really don't understand why. The proposed design has been public for a long time now and it's actually immediately obvious that it's pretty much useless in safe code, as live is a function annotation.
 is, like DIP 1000, make it possible to do things in  safe code that previously 
 required  system/ trusted--in this case, things like manually freeing 
 memory.
Yes, and that's precisely why live is not extremely useful and comparisons with ownership/borrowing systems from academia are superficial and wildly overblown.
Feb 14
next sibling parent reply Elronnd <elronnd elronnd.net> writes:
On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:
  live is only useful in  trusted or  system code as a linting 
 tool.
Devil's advocate: nogc.
Feb 14
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 14 February 2022 at 21:14:08 UTC, Elronnd wrote:
 On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:
  live is only useful in  trusted or  system code as a linting 
 tool.
Devil's advocate: nogc.
live does not help at all with writing safe nogc code. In order for safe or trusted code to rely on live's ownership invariants (e.g., "a non-scope pointer owns the memory it points to"), it must be impossible for safe code to violate those invariants. Since live's invariants are only enforced in live functions, and safe code is allowed to call non- live functions, it follows that safe code is allowed to violate live's invariants, and therefore that those invariants cannot be relied upon by safe or trusted code. To fix this, you would have to introduce new rules such as * All safe functions must also be live * safe functions cannot call non- live functions Of course, adding rules like this would break literally every safe function in every existing D project, so it is totally infeasible in practice--and that's why the current design for live is a dead-end.
Feb 14
parent deadalnix <deadalnix gmail.com> writes:
On Monday, 14 February 2022 at 22:46:09 UTC, Paul Backus wrote:
 On Monday, 14 February 2022 at 21:14:08 UTC, Elronnd wrote:
 On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:
  live is only useful in  trusted or  system code as a linting 
 tool.
Devil's advocate: nogc.
live does not help at all with writing safe nogc code. In order for safe or trusted code to rely on live's ownership invariants (e.g., "a non-scope pointer owns the memory it points to"), it must be impossible for safe code to violate those invariants. Since live's invariants are only enforced in live functions, and safe code is allowed to call non- live functions, it follows that safe code is allowed to violate live's invariants, and therefore that those invariants cannot be relied upon by safe or trusted code. To fix this, you would have to introduce new rules such as * All safe functions must also be live * safe functions cannot call non- live functions Of course, adding rules like this would break literally every safe function in every existing D project, so it is totally infeasible in practice--and that's why the current design for live is a dead-end.
This is 100% correct. In addition, taking a step back, it is abundantly clear that slapping an attribute on any problem that comes up only create a combinatorial explosion in the language that is fundamentally unsustainable. In the end, because these tend to contradict, the language cannot provide any guarantee one can rely on.
Feb 17
prev sibling parent reply IGotD- <nise nise.com> writes:
On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:
  live does not do what an ownership/borrowing system is 
 supposed to do. I have tried to make this point many times, but 
 most people seem to still assume the opposite. I really don't 
 understand why. The proposed design has been public for a long 
 time now and it's actually immediately obvious that it's pretty 
 much useless in  safe code, as  live is a function annotation.
The borrow/checker system and ownership are two separate things. Borrow/checker ensures that there is only one mutable access at one time or several immutable accesses. Ownership is a completely different topic which connects to memory management. The infamous ownership system in Rust only allows one ownership at one time which is tied to their move semantics. The borrow/checker in D is something I haven't completely understood and what question it is supposed to solve.
Feb 14
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/14/2022 1:43 PM, IGotD- wrote:
 The borrow/checker system and ownership are two separate things.
Borrow/checker 
 ensures that there is only one mutable access at one time or several immutable 
 accesses.
 
 Ownership is a completely different topic which connects to memory management. 
 The infamous ownership system in Rust only allows one ownership at one time 
 which is tied to their move semantics.
 
 The borrow/checker in D is something I haven't completely understood and what 
 question it is supposed to solve.
live only allows one ownership at a time and is tied to move semantics.
Feb 14
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 2/14/22 22:43, IGotD- wrote:
 On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:
  live does not do what an ownership/borrowing system is supposed to 
 do. I have tried to make this point many times, but most people seem 
 to still assume the opposite. I really don't understand why. The 
 proposed design has been public for a long time now and it's actually 
 immediately obvious that it's pretty much useless in  safe code, as 
  live is a function annotation.
The borrow/checker system and ownership are two separate things.
That's why I mentioned both. The stated goal of live is to have some story related to both of those things. I think it's not a particularly good story compared to some of the competition.
 Borrow/checker ensures that there is only one mutable access at one time 
 or several immutable accesses.
 ...
That's just one way to organize borrowing. live has some checks that aim in this direction, but it does not have a consistent explanation in terms of what it is ensuring.
 Ownership is a completely different topic which connects to memory 
 management.
I am not sure what your point is. "Completely different topic" makes little sense, as there is no need for borrowing if you have no ownership. Also, of course, borrowing "connects to memory management" too.
 The infamous ownership system in Rust only allows one 
 ownership at one time which is tied to their move semantics.
 ...
Which of course has no relation at all to why Rust also has an infamous borrow checker. Got it.
 The borrow/checker in D is something I haven't completely understood and 
 what question it is supposed to solve.
I am pretty sure I understand it completely, and that calling it a borrow checker is somewhat misleading.
Feb 16
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/14/2022 5:15 AM, Paul Backus wrote:
 Huh? My understanding is that modulo compiler bugs and incorrect use of 
  trusted,  safe code should be 100% memory safe, even without  live.
 What adding an ownership/borrowing system does (or should do) is, like DIP
1000, 
 make it possible to do things in  safe code that previously required 
  system/ trusted--in this case, things like manually freeing memory.
Without live, one cannot be protected against things like double frees.
Feb 14
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Monday, 14 February 2022 at 22:47:24 UTC, Walter Bright wrote:
 On 2/14/2022 5:15 AM, Paul Backus wrote:
 Huh? My understanding is that modulo compiler bugs and 
 incorrect use of  trusted,  safe code should be 100% memory 
 safe, even without  live.
 What adding an ownership/borrowing system does (or should do) 
 is, like DIP 1000, make it possible to do things in  safe code 
 that previously required  system/ trusted--in this case, 
 things like manually freeing memory.
Without live, one cannot be protected against things like double frees.
If you're writing safe code you're already protected from double frees because you're not allowed to manually free memory at all, never mind doing it twice. :)
Feb 14
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
On Monday, 14 February 2022 at 22:47:24 UTC, Walter Bright wrote:
 On 2/14/2022 5:15 AM, Paul Backus wrote:
 Huh? My understanding is that modulo compiler bugs and 
 incorrect use of  trusted,  safe code should be 100% memory 
 safe, even without  live.
 What adding an ownership/borrowing system does (or should do) 
 is, like DIP 1000, make it possible to do things in  safe code 
 that previously required  system/ trusted--in this case, 
 things like manually freeing memory.
Without live, one cannot be protected against things like double frees.
Free is not safe , so there is nothing to protect against.
Feb 17
prev sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Monday, 14 February 2022 at 22:47:24 UTC, Walter Bright wrote:
 On 2/14/2022 5:15 AM, Paul Backus wrote:
 Huh? My understanding is that modulo compiler bugs and 
 incorrect use of  trusted,  safe code should be 100% memory 
 safe, even without  live.
 What adding an ownership/borrowing system does (or should do) 
 is, like DIP 1000, make it possible to do things in  safe code 
 that previously required  system/ trusted--in this case, 
 things like manually freeing memory.
Without live, one cannot be protected against things like double frees.
How? live is attached to the function and not the type itself, which I never understand why you made that design decision. -Alex
Feb 17
prev sibling parent deadalnix <deadalnix gmail.com> writes:
On Monday, 14 February 2022 at 13:15:26 UTC, Paul Backus wrote:
 On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright 
 wrote:
 On 2/13/2022 3:15 AM, Florian Weimer wrote:
 I've tried to figure out where this is heading. Is the 
 eventual goal (irrespective of mechanism) that sticking 
 ` safe` onto the `main` function will ensure memory safety 
 for the whole program?
Yes, although safe does not supply complete memory safety. The addition of live fills in much of the rest.
Huh? My understanding is that modulo compiler bugs and incorrect use of trusted, safe code should be 100% memory safe, even without live.
Yes, live really isn't necessary for memory safety.
Feb 17