www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DIP1000: Memory Safety in a Modern System Programming Language Pt.1

reply Mike Parker <aldacron gmail.com> writes:
Ate Eskola was inspired to write a series of tutorials about 
DIP1000 for the D Blog. The first post in the series is live. If 
you haven't yet dug into DIP1000 much or understood how to use 
it, this should give you enough to get started.

The blog:
https://dlang.org/blog/2022/06/21/dip1000-memory-safety-in-a-modern-system-programming-language-pt-1/

Reddit:
https://www.reddit.com/r/programming/comments/vhfd28/memory_safety_in_a_modern_system_programming/
Jun 21
next sibling parent reply StarCanopy <starcanopy protonmail.com> writes:
On Tuesday, 21 June 2022 at 15:05:46 UTC, Mike Parker wrote:
 [...]
```d int[5] stackData = [-1, -2, -3, -4, -5]; // Lifetime of stackData2 ends // before limitedRef, so this is // disallowed. limitedRef = stackData[]; ``` In the above example, `stackData2` seems to be a typo.
Jun 21
parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 21 June 2022 at 22:55:56 UTC, StarCanopy wrote:
 On Tuesday, 21 June 2022 at 15:05:46 UTC, Mike Parker wrote:
 [...]
```d int[5] stackData = [-1, -2, -3, -4, -5]; // Lifetime of stackData2 ends // before limitedRef, so this is // disallowed. limitedRef = stackData[]; ``` In the above example, `stackData2` seems to be a typo.
Thanks, you're right. Missed that when editing.
Jun 21
parent ezneh <petitv.isat gmail.com> writes:
On Wednesday, 22 June 2022 at 06:48:34 UTC, Dukc wrote:
 On Tuesday, 21 June 2022 at 22:55:56 UTC, StarCanopy wrote:
 On Tuesday, 21 June 2022 at 15:05:46 UTC, Mike Parker wrote:
 [...]
```d int[5] stackData = [-1, -2, -3, -4, -5]; // Lifetime of stackData2 ends // before limitedRef, so this is // disallowed. limitedRef = stackData[]; ``` In the above example, `stackData2` seems to be a typo.
Thanks, you're right. Missed that when editing.
Other typo: ```, as that dcoument is what ```
Jun 22
prev sibling next sibling parent reply zjh <fqbqrr 163.com> writes:
On Tuesday, 21 June 2022 at 15:05:46 UTC, Mike Parker wrote:

Good article!
Jun 22
parent reply zjh <fqbqrr 163.com> writes:
On Wednesday, 22 June 2022 at 07:15:34 UTC, zjh wrote:
 On Tuesday, 21 June 2022 at 15:05:46 UTC, Mike Parker wrote:

 Good article!
[chinese version](https://fqbqrr.blog.csdn.net/article/details/125409915)
Jun 24
parent Dukc <ajieskola gmail.com> writes:
On Saturday, 25 June 2022 at 02:03:01 UTC, zjh wrote:
 On Wednesday, 22 June 2022 at 07:15:34 UTC, zjh wrote:
 On Tuesday, 21 June 2022 at 15:05:46 UTC, Mike Parker wrote:

 Good article!
[chinese version](https://fqbqrr.blog.csdn.net/article/details/125409915)
Wow, thanks!
Jun 25
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 21 June 2022 at 15:05:46 UTC, Mike Parker wrote:
 The blog:
 https://dlang.org/blog/2022/06/21/dip1000-memory-safety-in-a-modern-system-programming-language-pt-1/
Now on 26. place at Hacker News.
Jun 22
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 22 June 2022 at 19:09:28 UTC, Dukc wrote:
 On Tuesday, 21 June 2022 at 15:05:46 UTC, Mike Parker wrote:
 The blog:
 https://dlang.org/blog/2022/06/21/dip1000-memory-safety-in-a-modern-system-programming-language-pt-1/
Now on 26. place at Hacker News.
This was a nice presentation, if there will be a follow up then maybe create examples with a main and a button for «run this» that will show it in run.dlang.org? I suspect some readers will think TLDR when faced with longer blog posts, and just look at the examples (hence the show-don't-tell principle).
Jun 22
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/21/22 11:05 AM, Mike Parker wrote:
 Ate Eskola was inspired to write a series of tutorials about DIP1000 for 
 the D Blog. The first post in the series is live. If you haven't yet dug 
 into DIP1000 much or understood how to use it, this should give you 
 enough to get started.
 
 The blog:
 https://dlang.org/blog/2022/06/21/dip1000-memory-safety-in-a-modern-system-progra
ming-language-pt-1/ 
 
 
 Reddit:
 https://www.reddit.com/r/programming/comments/vhfd28/memory_safety_in_a_modern
system_programming/ 
 
Dip1000's point is starting to seep in. I still think it's going to be a challenge for people new to D (not just us old-timers). But... The part about `scope` being shallow. This is a problem. ```d scope a = "first"; scope b = "second"; string[] arr = [a, b]; // invalid regardless of attributes in safe code ``` Sometimes algorithms require manipulation of structure, such as sorting arrays, or using linked lists, and sometimes it's nice to be able to point at things on the stack, temporarily. This is one of the things I was looking forward to with dip1000, since it does allow pointing at the stack when it can work out the details. Is there any plan to address this other than "just use ` system`"? -Steve
Jun 22
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 22 June 2022 at 20:48:13 UTC, Steven Schveighoffer 
wrote:
 The part about `scope` being shallow. This is a problem.
One thing that will be confusing to most users is that it appears to be using "taint" rather than proper flow analysis on the pointed-to-object? ```d int* test(int arg1, int arg2) { int* p = null; p = &arg1; p = new int(5); return p; // complains about p being scope } ```
Jun 22
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/22/22 5:07 PM, Ola Fosheim Grøstad wrote:
 On Wednesday, 22 June 2022 at 20:48:13 UTC, Steven Schveighoffer wrote:
 The part about `scope` being shallow. This is a problem.
One thing that will be confusing to most users is that it appears to be using "taint" rather than proper flow analysis on the pointed-to-object? ```d int* test(int arg1, int arg2) {     int* p = null;     p = &arg1;     p = new int(5);     return p;  // complains about p being scope } ```
The other option is to complain about the assignment of &arg to p. That might be a better answer. At least it's *understandable*, and not sneaky. Full flow analysis will be defeatable by more complex situations: ```d int *p = null; if(alwaysEvaluateToFalse()) p = &arg; else p = new int(5); return p; ``` That would take a lot of effort just to prove it shouldn't be scope. -Steve
Jun 22
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 22 June 2022 at 21:20:33 UTC, Steven Schveighoffer 
wrote:
 Full flow analysis will be defeatable by more complex 
 situations:

 ```d
 int *p = null;
 if(alwaysEvaluateToFalse()) p = &arg;
 else p = new int(5);
 return p;
 ```

 That would take a lot of effort just to prove it shouldn't be 
 scope.
I guess this is the wrong forum, but two quick points. Some C programmers reuse variables extensively, those programmers will be confused or annoyed. The analysis can be done after an optimization pass, so at least the simple cases go through smoothly.
Jun 22
parent Dom Disc <dominikus scherkl.de> writes:
On Wednesday, 22 June 2022 at 21:58:07 UTC, Ola Fosheim Grøstad 
wrote:
 Some C programmers reuse variables extensively, those 
 programmers will be confused or annoyed.
And rightly so. Misra says since 30 years or longer: don't reuse variables if possible (and it should almost always be possible). If there exists now another way to shoot in your foot with this bad habit, so what?
Jun 23
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 22 June 2022 at 21:07:50 UTC, Ola Fosheim Grøstad 
wrote:
 On Wednesday, 22 June 2022 at 20:48:13 UTC, Steven 
 Schveighoffer wrote:
 The part about `scope` being shallow. This is a problem.
One thing that will be confusing to most users is that it appears to be using "taint" rather than proper flow analysis on the pointed-to-object? ```d int* test(int arg1, int arg2) { int* p = null; p = &arg1; p = new int(5); return p; // complains about p being scope } ```
I'd personally prefer if variable `scope` auto-inference worked only in the declaration, not later assignments. I guess the intention is to break less existing code. Your solution would break even less, but it'd mean the language rules depend on flow analysis. Because the rules are now "official", probably best to leave them as is to avoid confusion.
Jun 22
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/22/22 5:44 PM, Dukc wrote:
 On Wednesday, 22 June 2022 at 21:07:50 UTC, Ola Fosheim Grøstad wrote:
 On Wednesday, 22 June 2022 at 20:48:13 UTC, Steven Schveighoffer wrote:
 The part about `scope` being shallow. This is a problem.
One thing that will be confusing to most users is that it appears to be using "taint" rather than proper flow analysis on the pointed-to-object? ```d int* test(int arg1, int arg2) {     int* p = null;     p = &arg1;     p = new int(5);     return p;  // complains about p being scope } ```
I'd personally prefer if variable `scope` auto-inference worked only in the declaration, not later assignments. I guess the intention is to break less existing code.
I think this is the better option. Either that, or that when it returns `p` that trumps any possible `scope` inference. Imagine you have a function like this: ```d int foo() { int x = 0; x = long.max; x = 2; return x; } ``` Now today, this causes an error on the assignment to `long.max`, because obviously `x` is an int. But what if, instead, the compiler decides to backtrack and say "actually, if I make x a `long`, then it works!", and *now*, at the end, says "Oh, actually, you can't return a long as an int, what were you thinking?!" This is the equivalent here, you declare something *without* scope, assign it to something that is *not* scope, and then because sometime later you assigned it to something that *is* scope, it goes back and rewrites the declaration as if you did make it scope, and then complains to you that the magic trick it tried is not valid. This is going to be one of the most confusing features of DIP1000. -Steve
Jun 22
next sibling parent Johan <j j.nl> writes:
On Thursday, 23 June 2022 at 00:45:09 UTC, Steven Schveighoffer 
wrote:
 On 6/22/22 5:44 PM, Dukc wrote:
 On Wednesday, 22 June 2022 at 21:07:50 UTC, Ola Fosheim 
 Grøstad wrote:
 On Wednesday, 22 June 2022 at 20:48:13 UTC, Steven 
 Schveighoffer wrote:
 The part about `scope` being shallow. This is a problem.
One thing that will be confusing to most users is that it appears to be using "taint" rather than proper flow analysis on the pointed-to-object? ```d int* test(int arg1, int arg2) {     int* p = null;     p = &arg1;     p = new int(5);     return p;  // complains about p being scope } ```
I'd personally prefer if variable `scope` auto-inference worked only in the declaration, not later assignments.
I agree.
 I think this is the better option. Either that, or that when it 
 returns `p` that trumps any possible `scope` inference.

 Imagine you have a function like this:

 ```d
 int foo()
 {
    int x = 0;
    x = long.max;
    x = 2;
    return x;
 }
 ```

 Now today, this causes an error on the assignment to 
 `long.max`, because obviously `x` is an int. But what if, 
 instead, the compiler decides to backtrack and say "actually, 
 if I make x a `long`, then it works!", and *now*, at the end, 
 says "Oh, actually, you can't return a long as an int, what 
 were you thinking?!"

 This is the equivalent here, you declare something *without* 
 scope, assign it to something that is *not* scope, and then 
 because sometime later you assigned it to something that *is* 
 scope, it goes back and rewrites the declaration as if you did 
 make it scope, and then complains to you that the magic trick 
 it tried is not valid.

 This is going to be one of the most confusing features of 
 DIP1000.
Plus that the error message just sounds like a compiler bug. It says that variable `p` is `scope`, but nowhere in the source does it say that. The error message should say that assignment `p = &arg1;` makes the variable `scope`. (An alternative is to give a warning on `p = &arg1;`.) -Johan
Jun 22
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 23 June 2022 at 00:45:09 UTC, Steven Schveighoffer 
wrote:
 I think this is the better option. Either that, or that when it 
 returns `p` that trumps any possible `scope` inference.

 Imagine you have a function like this:

 ```d
 int foo()
 {
    int x = 0;
    x = long.max;
    x = 2;
    return x;
 }
 ```

 Now today, this causes an error on the assignment to 
 `long.max`, because obviously `x` is an int. But what if, 
 instead, the compiler decides to backtrack and say "actually, 
 if I make x a `long`, then it works!", and *now*, at the end, 
 says "Oh, actually, you can't return a long as an int, what 
 were you thinking?!"

 This is the equivalent here, you declare something *without* 
 scope, assign it to something that is *not* scope, and then 
 because sometime later you assigned it to something that *is* 
 scope, it goes back and rewrites the declaration as if you did 
 make it scope, and then complains to you that the magic trick 
 it tried is not valid.

 This is going to be one of the most confusing features of 
 DIP1000.
It is confusing because it introduces flow typing without having flow typing. So this is messing up the user’s mental model of the type system, this is a basic usability flaw. Track the object instead and don’t change the type of the pointer to scope. If D wants to do flow typing, do it properly and make it clear to the user. It would be a good feature to have, but it would become D3.
Jun 22
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 23 June 2022 at 06:36:23 UTC, Ola Fosheim Grøstad 
wrote:
 Track the object instead and don’t change the type of the 
 pointer to scope.
I guess this is flow typing too, but it is less intrusive to say that the object is either of type «scope» or type «heap» and that regular pointers can hold both than to change the concrete pointer type. Specified concrete types should not change.
Jun 22
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 23 June 2022 at 06:52:48 UTC, Ola Fosheim Grøstad 
wrote:
 On Thursday, 23 June 2022 at 06:36:23 UTC, Ola Fosheim Grøstad 
 wrote:
 Track the object instead and don’t change the type of the 
 pointer to scope.
I guess this is flow typing too, but it is less intrusive to say that the object is either of type «scope» or type «heap» and that regular pointers can hold both than to change the concrete pointer type. Specified concrete types should not change.
For people interested in getting more intuition for flow typing: https://www.typescriptlang.org/docs/handbook/2/narrowing.html or chapter 3: https://whiley.org/pdfs/GettingStartedWithWhiley.pdf
Jun 23
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 22 June 2022 at 20:48:13 UTC, Steven Schveighoffer 
wrote:
 On 6/21/22 11:05 AM, Mike Parker wrote:
 The part about `scope` being shallow. This is a problem.

 Is there any plan to address this other than "just use 
 ` system`"?

 -Steve
I think a custom `struct` containing a GC:d slice of `scope` arrays could be done. The struct itself needs ` system` code of course but could be ` safe` from outwards perspective, unless there's some issue I haven't thought of. A relatively quick-and-dirty solution would be to use a static array for this, if you know some upper size the array can't exceed. Kinda cheap but probably better than ` system`.
Jun 22
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/22/22 5:58 PM, Dukc wrote:
 On Wednesday, 22 June 2022 at 20:48:13 UTC, Steven Schveighoffer wrote:
 On 6/21/22 11:05 AM, Mike Parker wrote:
 The part about `scope` being shallow. This is a problem.

 Is there any plan to address this other than "just use ` system`"?
I think a custom `struct` containing a GC:d slice of `scope` arrays could be done. The struct itself needs ` system` code of course but could be ` safe` from outwards perspective, unless there's some issue I haven't thought of. A relatively quick-and-dirty solution would be to use a static array for this, if you know some upper size the array can't exceed. Kinda cheap but probably better than ` system`.
You mean like a system function which removes the scope-ness of an array? Let me introduce you to my other thread: https://forum.dlang.org/thread/t7qd45$1lrb$1 digitalmars.com -Steve
Jun 22
parent reply Dukc <ajieskola gmail.com> writes:
On Thursday, 23 June 2022 at 00:37:24 UTC, Steven Schveighoffer 
wrote:
 You mean like a system function which removes the scope-ness of 
 an array? Let me introduce you to my other thread: 
 https://forum.dlang.org/thread/t7qd45$1lrb$1 digitalmars.com

 -Steve
You are allowed to remove `scope` from an argument in unsafe code. It's only if you escape that argument when you trigger undefined behaviour. Just like you can cast away `const`, if you don't actually mutate the cast variable.
Jun 23
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/23/22 8:01 AM, Dukc wrote:
 On Thursday, 23 June 2022 at 00:37:24 UTC, Steven Schveighoffer wrote:
 You mean like a system function which removes the scope-ness of an 
 array? Let me introduce you to my other thread: 
 https://forum.dlang.org/thread/t7qd45$1lrb$1 digitalmars.com
You are allowed to remove `scope` from an argument in unsafe code. It's only if you escape that argument when you trigger undefined behaviour. Just like you can cast away `const`, if you don't actually mutate the cast variable.
And what do you think a custom struct that circumvents the scopeness is going to do with that parameter? To be clear, I think we're talking about something like: ```d struct ScopeArray(T) { T[] arr; system void opAssign(scope T[] param) { arr = param; // escaping scope } } ``` -Steve
Jun 23
parent Dukc <ajieskola gmail.com> writes:
On Thursday, 23 June 2022 at 14:08:15 UTC, Steven Schveighoffer 
wrote:
 On 6/23/22 8:01 AM, Dukc wrote:
 On Thursday, 23 June 2022 at 00:37:24 UTC, Steven 
 Schveighoffer wrote:
 You mean like a system function which removes the scope-ness 
 of an array? Let me introduce you to my other thread: 
 https://forum.dlang.org/thread/t7qd45$1lrb$1 digitalmars.com
You are allowed to remove `scope` from an argument in unsafe code. It's only if you escape that argument when you trigger undefined behaviour. Just like you can cast away `const`, if you don't actually mutate the cast variable.
And what do you think a custom struct that circumvents the scopeness is going to do with that parameter? To be clear, I think we're talking about something like: ```d struct ScopeArray(T) { T[] arr; system void opAssign(scope T[] param) { arr = param; // escaping scope } } ``` -Steve
By `return scope`. I'm not sure if it'd work with that `opAssign` example since we don't actually return the `this` pointer. I need to recheck what `return scope` means with `void` return type, before I write the next article - There is or at least was some rule regarding that situation. But in any case you could accomplish the same with ```D safe ScopeArray!T(T)(return scope T[] param) { return ScopeArray!T(param); } ``` . Note that in this case you are not even trying to store `scope` variables as elements to an array, so we don't need ` trusted`.
Jun 23
prev sibling parent reply Kagamin <spam here.lot> writes:
On Wednesday, 22 June 2022 at 20:48:13 UTC, Steven Schveighoffer 
wrote:
 Sometimes algorithms require manipulation of structure, such as 
 sorting arrays, or using linked lists, and sometimes it's nice 
 to be able to point at things on the stack, temporarily. This 
 is one of the things I was looking forward to with dip1000, 
 since it does allow pointing at the stack when it can work out 
 the details.
This works: ``` struct S { int[] a; int[] get() return scope safe { return a; } void set(return int[] b) return scope safe { a=b; } } int[] f() safe { int[2] a; scope S t; int[] b=t.get; t.set=a; return b; //no } ```
Jun 23
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/23/22 8:14 AM, Kagamin wrote:
 On Wednesday, 22 June 2022 at 20:48:13 UTC, Steven Schveighoffer wrote:
 Sometimes algorithms require manipulation of structure, such as 
 sorting arrays, or using linked lists, and sometimes it's nice to be 
 able to point at things on the stack, temporarily. This is one of the 
 things I was looking forward to with dip1000, since it does allow 
 pointing at the stack when it can work out the details.
This works: ``` struct S {     int[] a;     int[] get() return scope safe { return a; }     void set(return int[] b) return scope safe     { a=b; } } int[] f() safe {     int[2] a;     scope S t;     int[] b=t.get;     t.set=a;     return b; //no } ```
This is just saying the same thing -- you must allocate your data on the stack in order to have scope elements assignable. This isn't always feasible. ```d scope a = "first"; scope b = "second"; string[2] x = [a, b]; auto arr = x[]; // ok arr = ["first", "second"]; // ok arr = [a, b]; // not ok ``` It's established that arr is attributed such that it's able to point at stack data. Fine. It can also point at allocated data. Fine. But the *allocated data itself* cannot point at scope data. This "somewhat" makes sense, because by the time the GC cleans up this array, the data is likely invalid (not in this case, but that's because we deliberately labeled non-scope data as scope). However, you may need to mix both non-scope and scope data, which means you can't allocate on the stack. But there can be ways to mitigate this: 1. For this simple case, just allocate [a, b] on the stack. It's happening in other places (see my other thread), why not here? 2. The compiler could insert a call at the end of the scope to free the allocated data. It can't escape anyway, and freeing it early is part of the benefit of having scope. Or if it's determined that no destructors are involved, just let the GC clean it up. The more cases where we make this painless for the user, the better. -Steve
Jun 23