www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Direction for safe/-dip1000

reply Florian Weimer <fw deneb.enyo.de> writes:
I've tried to figure out where this is heading. Is the eventual 
goal (irrespective of mechanism) that sticking ` safe` onto the 
`main` function will ensure memory safety for the whole program?

What about data races in updates of multi-word variables? For 
example, the Go language makes avoiding them the responsibility 
of the programmer, but Java ensures that data races cannot 
subvert the type system (although with the demise of the security 
manager, some people argue that this should be changed).
Feb 13 2022
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 13.02.22 12:15, Florian Weimer wrote:

 I've tried to figure out where this is heading. Is the eventual goal 
 (irrespective of mechanism) that sticking ` safe` onto the `main` 
 function will ensure memory safety for the whole program?


Roughly, that's what  safe is supposed to be, yes. But it's not quite as 
simple because `main` isn't the only possible entry point, and because 
 trusted exists.


 What about data races in updates of multi-word variables?


As far as I know, this is D's current approach:

All shared data must be marked with `shared`.  safe code cannot access 
`shared` data (only enforced with `-preview=nosharedaccess`). If you 
want to mess with `shared` data, you need to write  system/ trusted code 
and ensure thread safety yourself.

Since you mentioned it in the topic, note that DIP 1000 doesn't make 
 safe safer. It allows some formerly  system code to become  safe. If 
you write that kind of code, DIP 1000 is nice to have. If you don't, DIP 
1000 doesn't affect you.
Feb 13 2022
parent reply Florian Weimer <fw deneb.enyo.de> writes:
On Sunday, 13 February 2022 at 12:26:52 UTC, ag0aep6g wrote:

 On 13.02.22 12:15, Florian Weimer wrote:

 I've tried to figure out where this is heading. Is the 
 eventual goal (irrespective of mechanism) that sticking 
 ` safe` onto the `main` function will ensure memory safety for 
 the whole program?


 Roughly, that's what  safe is supposed to be, yes. But it's not 
 quite as simple because `main` isn't the only possible entry 
 point, and because  trusted exists.


Right, but it's still a big step forward.


 Since you mentioned it in the topic, note that DIP 1000 doesn't 
 make  safe safer. It allows some formerly  system code to 
 become  safe. If you write that kind of code, DIP 1000 is nice 
 to have. If you don't, DIP 1000 doesn't affect you.


Hmm.  This compiles without `-dip1000` (or 
`-ftransition=dip1000`), but is unsafe:

```d
int[] global;

 safe
void f0(int[] val) {
     global = val;
}

 safe
void f1() {
   int[3] local = [1, 2, 3];
   f0(local);
}

```

It's rejected in DIP 1000 mode (both by DMD and GDC).
Feb 13 2022
parent ag0aep6g <anonymous example.com> writes:
On 13.02.22 13:37, Florian Weimer wrote:

 Hmm.  This compiles without `-dip1000` (or `-ftransition=dip1000`), but 
 is unsafe:
 
 ```d
 int[] global;
 
  safe
 void f0(int[] val) {
      global = val;
 }
 
  safe
 void f1() {
    int[3] local = [1, 2, 3];
    f0(local);
 }
 
 ```
 
 It's rejected in DIP 1000 mode (both by DMD and GDC).


Yeah, bugs are plenty. And if a bug doesn't manifest with 
`-preview=dip1000`, then people are even less likely than usual to give 
a damn about it. Because DIP 1000 is going to become the default 
eventually, and then the issue will go away anyway.

It goes the other way, too: `-preview=dip1000` has safety holes that 
aren't there without the switch. But those are more likely to get fixed, 
because some people do care about getting DIP 1000 done.
Feb 13 2022
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/13/2022 3:15 AM, Florian Weimer wrote:

 I've tried to figure out where this is heading. Is the eventual goal 
 (irrespective of mechanism) that sticking ` safe` onto the `main` function
will 
 ensure memory safety for the whole program?


Yes, although  safe does not supply complete memory safety. The addition of 
 live fills in much of the rest.



 What about data races in updates of multi-word variables? For example, the Go 
 language makes avoiding them the responsibility of the programmer, but Java 
 ensures that data races cannot subvert the type system (although with the
demise 
 of the security manager, some people argue that this should be changed).


It's up to the programmer to avoid data races in multi-word variables, likely
by 
manually using locks.
Feb 14 2022
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright wrote:

 On 2/13/2022 3:15 AM, Florian Weimer wrote:

 I've tried to figure out where this is heading. Is the 
 eventual goal (irrespective of mechanism) that sticking 
 ` safe` onto the `main` function will ensure memory safety for 
 the whole program?


 Yes, although  safe does not supply complete memory safety. The 
 addition of  live fills in much of the rest.


Huh? My understanding is that modulo compiler bugs and incorrect 
use of  trusted,  safe code should be 100% memory safe, even 
without  live.

What adding an ownership/borrowing system does (or should do) is, 
like DIP 1000, make it possible to do things in  safe code that 
previously required  system/ trusted--in this case, things like 
manually freeing memory.
Feb 14 2022
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Feb 14, 2022 at 01:15:26PM +0000, Paul Backus via Digitalmars-d wrote:

 On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright wrote:


[...]

 Yes, although  safe does not supply complete memory safety. The
 addition of  live fills in much of the rest.


 
 Huh? My understanding is that modulo compiler bugs and incorrect use
 of  trusted,  safe code should be 100% memory safe, even without
  live.


The problem with  safe as it is implemented today is that it's
implemented as a blacklist rather than a whitelist.

Cf. points 2 and 3 (as applied to memory safety) in:

	http://ranum.com/security/computer_security/editorials/dumb/index.html


T

-- 
Once the bikeshed is up for painting, the rainbow won't suffice. -- Andrei
Alexandrescu
Feb 14 2022
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Monday, 14 February 2022 at 15:50:31 UTC, H. S. Teoh wrote:

 On Mon, Feb 14, 2022 at 01:15:26PM +0000, Paul Backus via 
 Digitalmars-d wrote:

 On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright 
 wrote:


 [...]

 Yes, although  safe does not supply complete memory safety. 
 The addition of  live fills in much of the rest.


 
 Huh? My understanding is that modulo compiler bugs and 
 incorrect use of  trusted,  safe code should be 100% memory 
 safe, even without  live.


 The problem with  safe as it is implemented today is that it's 
 implemented as a blacklist rather than a whitelist.


I did say "should be" and "modulo compiler bugs" for a reason. :)

Even with a whitelist implementation, though, we'd still have 
bugs where something was accidentally whitelisted that shouldn't 
have been. Several of the recent fixes to -preview=dip1000 are 
for exactly this type of bug, for example.
Feb 14 2022
prev sibling parent IGotD- <nise nise.com> writes:
On Monday, 14 February 2022 at 15:50:31 UTC, H. S. Teoh wrote:

 The problem with  safe as it is implemented today is that it's 
 implemented as a blacklist rather than a whitelist.

 Cf. points 2 and 3 (as applied to memory safety) in:

 	http://ranum.com/security/computer_security/editorials/dumb/index.html


I think  safe has done a fundamental error to allow raw pointers. 
In  safe code the memory management should be completely opaque 

more simple to work with. Instead D does the opposite and adds 
stuff which makes memory management more complicated.
Feb 14 2022
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 2/14/22 14:15, Paul Backus wrote:

 On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright wrote:

 On 2/13/2022 3:15 AM, Florian Weimer wrote:

 I've tried to figure out where this is heading. Is the eventual goal 
 (irrespective of mechanism) that sticking ` safe` onto the `main` 
 function will ensure memory safety for the whole program?


 Yes, although  safe does not supply complete memory safety. The 
 addition of  live fills in much of the rest.


 
 Huh? My understanding is that modulo compiler bugs and incorrect use of 
  trusted,  safe code should be 100% memory safe, even without  live.
 ...


Exactly, incorrect use of  trusted.  live is only useful in  trusted or 
 system code as a linting tool.


 What adding an ownership/borrowing system does (or should do)


 live does not do what an ownership/borrowing system is supposed to do. 
I have tried to make this point many times, but most people seem to 
still assume the opposite. I really don't understand why. The proposed 
design has been public for a long time now and it's actually immediately 
obvious that it's pretty much useless in  safe code, as  live is a 
function annotation.


 is, like DIP 1000, make it possible to do things in  safe code that previously 
 required  system/ trusted--in this case, things like manually freeing 
 memory.


Yes, and that's precisely why  live is not extremely useful and 
comparisons with ownership/borrowing systems from academia are 
superficial and wildly overblown.
Feb 14 2022
next sibling parent reply Elronnd <elronnd elronnd.net> writes:
On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:

  live is only useful in  trusted or  system code as a linting 
 tool.


Devil's advocate:  nogc.
Feb 14 2022
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 14 February 2022 at 21:14:08 UTC, Elronnd wrote:

 On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:

  live is only useful in  trusted or  system code as a linting 
 tool.


 Devil's advocate:  nogc.


 live does not help at all with writing  safe  nogc code.

In order for  safe or  trusted code to rely on  live's ownership 
invariants (e.g., "a non-scope pointer owns the memory it points 
to"), it must be impossible for  safe code to violate those 
invariants. Since  live's invariants are only enforced in  live 
functions, and  safe code is allowed to call non- live functions, 
it follows that  safe code is allowed to violate  live's 
invariants, and therefore that those invariants cannot be relied 
upon by  safe or  trusted code.

To fix this, you would have to introduce new rules such as

* All  safe functions must also be  live
*  safe functions cannot call non- live functions

Of course, adding rules like this would break literally every 
 safe function in every existing D project, so it is totally 
infeasible in practice--and that's why the current design for 
 live is a dead-end.
Feb 14 2022
parent deadalnix <deadalnix gmail.com> writes:
On Monday, 14 February 2022 at 22:46:09 UTC, Paul Backus wrote:

 On Monday, 14 February 2022 at 21:14:08 UTC, Elronnd wrote:

 On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:

  live is only useful in  trusted or  system code as a linting 
 tool.


 Devil's advocate:  nogc.


  live does not help at all with writing  safe  nogc code.

 In order for  safe or  trusted code to rely on  live's 
 ownership invariants (e.g., "a non-scope pointer owns the 
 memory it points to"), it must be impossible for  safe code to 
 violate those invariants. Since  live's invariants are only 
 enforced in  live functions, and  safe code is allowed to call 
 non- live functions, it follows that  safe code is allowed to 
 violate  live's invariants, and therefore that those invariants 
 cannot be relied upon by  safe or  trusted code.

 To fix this, you would have to introduce new rules such as

 * All  safe functions must also be  live
 *  safe functions cannot call non- live functions

 Of course, adding rules like this would break literally every 
  safe function in every existing D project, so it is totally 
 infeasible in practice--and that's why the current design for 
  live is a dead-end.


This is 100% correct.

In addition, taking a step back, it is abundantly clear that 
slapping an attribute on any problem that comes up only create a 
combinatorial explosion in the language that is fundamentally 
unsustainable.

In the end, because these tend to contradict, the language cannot 
provide any guarantee one can rely on.
Feb 17 2022
prev sibling parent reply IGotD- <nise nise.com> writes:
On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:

  live does not do what an ownership/borrowing system is 
 supposed to do. I have tried to make this point many times, but 
 most people seem to still assume the opposite. I really don't 
 understand why. The proposed design has been public for a long 
 time now and it's actually immediately obvious that it's pretty 
 much useless in  safe code, as  live is a function annotation.


The borrow/checker system and ownership are two separate things. 
Borrow/checker ensures that there is only one mutable access at 
one time or several immutable accesses.

Ownership is a completely different topic which connects to 
memory management. The infamous ownership system in Rust only 
allows one ownership at one time which is tied to their move 
semantics.

The borrow/checker in D is something I haven't completely 
understood and what question it is supposed to solve.
Feb 14 2022
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/14/2022 1:43 PM, IGotD- wrote:

 The borrow/checker system and ownership are two separate things.
Borrow/checker 
 ensures that there is only one mutable access at one time or several immutable 
 accesses.
 
 Ownership is a completely different topic which connects to memory management. 
 The infamous ownership system in Rust only allows one ownership at one time 
 which is tied to their move semantics.
 
 The borrow/checker in D is something I haven't completely understood and what 
 question it is supposed to solve.


 live only allows one ownership at a time and is tied to move semantics.
Feb 14 2022
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 2/14/22 22:43, IGotD- wrote:

 On Monday, 14 February 2022 at 21:02:31 UTC, Timon Gehr wrote:

  live does not do what an ownership/borrowing system is supposed to 
 do. I have tried to make this point many times, but most people seem 
 to still assume the opposite. I really don't understand why. The 
 proposed design has been public for a long time now and it's actually 
 immediately obvious that it's pretty much useless in  safe code, as 
  live is a function annotation.


 
 The borrow/checker system and ownership are two separate things. 


That's why I mentioned both. The stated goal of  live is to have some 
story related to both of those things. I think it's not a particularly 
good story compared to some of the competition.


 Borrow/checker ensures that there is only one mutable access at one time 
 or several immutable accesses.
 ...


That's just one way to organize borrowing.  live has some checks that 
aim in this direction, but it does not have a consistent explanation in 
terms of what it is ensuring.


 Ownership is a completely different topic which connects to memory 
 management.


I am not sure what your point is. "Completely different topic" makes 
little sense, as there is no need for borrowing if you have no 
ownership. Also, of course, borrowing "connects to memory management" too.


 The infamous ownership system in Rust only allows one 
 ownership at one time which is tied to their move semantics.
 ...


Which of course has no relation at all to why Rust also has an infamous 
borrow checker. Got it.


 The borrow/checker in D is something I haven't completely understood and 
 what question it is supposed to solve.


I am pretty sure I understand it completely, and that calling it a 
borrow checker is somewhat misleading.
Feb 16 2022
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/14/2022 5:15 AM, Paul Backus wrote:

 Huh? My understanding is that modulo compiler bugs and incorrect use of 
  trusted,  safe code should be 100% memory safe, even without  live.
 What adding an ownership/borrowing system does (or should do) is, like DIP
1000, 
 make it possible to do things in  safe code that previously required 
  system/ trusted--in this case, things like manually freeing memory.


Without  live, one cannot be protected against things like double frees.
Feb 14 2022
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Monday, 14 February 2022 at 22:47:24 UTC, Walter Bright wrote:

 On 2/14/2022 5:15 AM, Paul Backus wrote:

 Huh? My understanding is that modulo compiler bugs and 
 incorrect use of  trusted,  safe code should be 100% memory 
 safe, even without  live.
 What adding an ownership/borrowing system does (or should do) 
 is, like DIP 1000, make it possible to do things in  safe code 
 that previously required  system/ trusted--in this case, 
 things like manually freeing memory.


 Without  live, one cannot be protected against things like 
 double frees.


If you're writing  safe code you're already protected from double 
frees because you're not allowed to manually free memory at all, 
never mind doing it twice. :)
Feb 14 2022
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
On Monday, 14 February 2022 at 22:47:24 UTC, Walter Bright wrote:

 On 2/14/2022 5:15 AM, Paul Backus wrote:

 Huh? My understanding is that modulo compiler bugs and 
 incorrect use of  trusted,  safe code should be 100% memory 
 safe, even without  live.
 What adding an ownership/borrowing system does (or should do) 
 is, like DIP 1000, make it possible to do things in  safe code 
 that previously required  system/ trusted--in this case, 
 things like manually freeing memory.


 Without  live, one cannot be protected against things like 
 double frees.


Free is not  safe , so there is nothing to protect against.
Feb 17 2022
prev sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Monday, 14 February 2022 at 22:47:24 UTC, Walter Bright wrote:

 On 2/14/2022 5:15 AM, Paul Backus wrote:

 Huh? My understanding is that modulo compiler bugs and 
 incorrect use of  trusted,  safe code should be 100% memory 
 safe, even without  live.
 What adding an ownership/borrowing system does (or should do) 
 is, like DIP 1000, make it possible to do things in  safe code 
 that previously required  system/ trusted--in this case, 
 things like manually freeing memory.


 Without  live, one cannot be protected against things like 
 double frees.


How?  live is attached to the function and not the type itself, 
which I never understand why you made that design decision.


-Alex
Feb 17 2022
prev sibling parent deadalnix <deadalnix gmail.com> writes:
On Monday, 14 February 2022 at 13:15:26 UTC, Paul Backus wrote:

 On Monday, 14 February 2022 at 08:39:58 UTC, Walter Bright 
 wrote:

 On 2/13/2022 3:15 AM, Florian Weimer wrote:

 I've tried to figure out where this is heading. Is the 
 eventual goal (irrespective of mechanism) that sticking 
 ` safe` onto the `main` function will ensure memory safety 
 for the whole program?


 Yes, although  safe does not supply complete memory safety. 
 The addition of  live fills in much of the rest.


 Huh? My understanding is that modulo compiler bugs and 
 incorrect use of  trusted,  safe code should be 100% memory 
 safe, even without  live.


Yes,  live really isn't necessary for memory safety.
Feb 17 2022