www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Discussion Thread: DIP 1035-- system Variables--Community Review Round

reply Mike Parker <aldacron gmail.com> writes:
This is the discussion thread for the second round of Community 
Review of DIP 1035, " system Variables":

https://github.com/dlang/DIPs/blob/c39f6ac62210e0604dcee99b0092c1930839f93a/DIPs/DIP1035.md

The review period will end at 11:59 PM ET on March 11, or when I 
make a post declaring it complete. Discussion in this thread may 
continue beyond that point.

Here in the discussion thread, you are free to discuss anything 
and everything related to the DIP. Express your support or 
opposition, debate alternatives, argue the merits, etc.

However, if you have any specific feedback on how to improve the 
proposal itself, then please post it in the feedback thread. The 
feedback thread will be the source for the review summary that I 
will write at the end of this review round. I will post a link to 
that thread immediately following this post. Just be sure to read 
and understand the Reviewer Guidelines before posting there:

https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

And my blog post on the difference between the Discussion and 
Feedback threads:

https://dlang.org/blog/2020/01/26/dip-reviews-discussion-vs-feedback/

Please stay on topic here. I will delete posts that are 
completely off-topic.
Feb 25
next sibling parent Mike Parker <aldacron gmail.com> writes:
On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker wrote:

 However, if you have any specific feedback on how to improve 
 the proposal itself, then please post it in the feedback 
 thread. The feedback thread will be the source for the review 
 summary that I will write at the end of this review round. I 
 will post a link to that thread immediately following this post.
The Feedback Thread is here: https://forum.dlang.org/post/eivezuohaejnvvlhaeec forum.dlang.org
Feb 25
prev sibling next sibling parent ag0aep6g <anonymous example.com> writes:
On 25.02.21 10:21, Mike Parker wrote:
 This is the discussion thread for the second round of Community Review 
 of DIP 1035, " system Variables":
 
 https://github.com/dlang/DIPs/blob/c39f6ac62210e0604dcee99b0092c1930839
93a/DIPs/DIP1035.md 
Big thanks to Dennis and Paul for championing this! I've only given the DIP a cursory read, but it looks great to me.
Feb 25
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
 Example: User-Defined Slice
Since ptr and length are private, they are only accessible to member functions of IntSlice. I don't think there is any way to prove memory safety of the implementation of this.
  Instead, every function that touches ptr and length, including the  safe 
constructor, must be manually checked. Right. So the idea is to minimize access to .ptr/.length for member functions of IntSlice. This can be done with: struct IntSlice { private static struct Slice { private int* ptr; private size_t length; trusted: this(int[] src) { ptr = src.ptr; length = src.length; } int[] opSlice() { return ptr[0 .. length]; } } // Now, the only access is via trusted functions safe: private Slice slice; this(int[] src) { slice = Slice(src); } ref int opIndex(size_t i) { return slice[][i]; // note no assert(i < length) needed } // ... every function goes here ... } So, the general idea is to restrict access to ` system` fields by encapsulating them separately, and providing a minimized trusted interface to them.
Feb 26
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 26.02.21 10:49, Walter Bright wrote:
  > Example: User-Defined Slice
 
 Since ptr and length are private, they are only accessible to member 
 functions of IntSlice. I don't think there is any way to prove memory 
 safety of the implementation of this.
 
  >  Instead, every function that touches ptr and length, including the 
  safe constructor, must be manually checked.
 
 Right. So the idea is to minimize access to .ptr/.length for member 
 functions of IntSlice. This can be done with:
 
    struct IntSlice
    {
      private static struct Slice
      {
          private int* ptr;
          private size_t length;
 
         trusted:
 
          this(int[] src)
          {
               ptr = src.ptr;
               length = src.length;
          }
 
          int[] opSlice()
          {
               return ptr[0 .. length];
          }
       }
 
       // Now, the only access is via  trusted functions
 
      safe:
 
      private Slice slice;
 
      this(int[] src)
      {
          slice = Slice(src);
      }
 
      ref int opIndex(size_t i)
      {
          return slice[][i]; // note no assert(i < length) needed
      }
 
      // ... every function goes here ...
    }
 
 So, the general idea is to restrict access to ` system` fields by 
 encapsulating them separately, and providing a minimized  trusted 
 interface to them.
Yes, that's the idea, but this is a convention. It's not possible now to make the compiler check your convention. system fields address this limitation.
Feb 26
prev sibling next sibling parent Paul Backus <snarwin gmail.com> writes:
On Friday, 26 February 2021 at 09:49:40 UTC, Walter Bright wrote:
     private static struct Slice
     {
         private int* ptr;
         private size_t length;

        trusted:

         this(int[] src)
         {
              ptr = src.ptr;
              length = src.length;
         }

         int[] opSlice()
         {
              return ptr[0 .. length];
         }
      }
This is equivalent to the example in the DIP, except that you have changed the constructor from safe to trusted by replacing `&src[0]` with `src.ptr`. If you had written it in the same was as the original, you would still have to manually check safe code in order to prove that this interface is memory-safe.
Feb 26
prev sibling parent reply Nick Treleaven <nick geany.org> writes:
On Friday, 26 February 2021 at 09:49:40 UTC, Walter Bright wrote:
 Example: User-Defined Slice
Since ptr and length are private, they are only accessible to member functions of IntSlice.
Er, doesn't private mean module access? This DIP would help verify long modules that might inadvertently access unsafe fields from outside the struct definition, at least in safe code.
Feb 26
parent reply Max Haughton <maxhaton gmail.com> writes:
On Friday, 26 February 2021 at 18:29:26 UTC, Nick Treleaven wrote:
 On Friday, 26 February 2021 at 09:49:40 UTC, Walter Bright 
 wrote:
 Example: User-Defined Slice
Since ptr and length are private, they are only accessible to member functions of IntSlice.
Er, doesn't private mean module access? This DIP would help verify long modules that might inadvertently access unsafe fields from outside the struct definition, at least in safe code.
https://run.dlang.io/is/KalIzj Yes - however code subject to is supposed to be consumed in a different module for the most part, so this could be a nice side effect but it shouldn't (de jure) guide the actual DIP itself.
Feb 26
parent Nick Treleaven <nick geany.org> writes:
On Friday, 26 February 2021 at 22:29:09 UTC, Max Haughton wrote:
 On Friday, 26 February 2021 at 18:29:26 UTC, Nick Treleaven 
 wrote:
 Er, doesn't private mean module access? This DIP would help 
 verify long modules that might inadvertently access unsafe 
 fields from outside the struct definition, at least in  safe 
 code.
https://run.dlang.io/is/KalIzj Yes - however code subject to is supposed to be consumed in a different module for the most part, so this could be a nice side effect but it shouldn't (de jure) guide the actual DIP itself.
It's more than a nice side effect - it's actually necessary to enforce a safe interface for structs that do unsafe things. Otherwise changing safe code can cause memory-safety violations - that's contrary to the design goal of safe. It also would in practice mean a programmer/reviewer would have to read every line of code in a module every time there was a change - that's not a sensible design.
Feb 27
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
 Example: Short String
This example boils down to an invariant not being maintained if the struct is void-initialized. A couple simple solutions come to mind: 1. disallow void initialization in safe code if the struct has a constructor 2. disallow void initialization if the struct has an invariant I suggest that the ShortString struct is not properly written, because it has a hidden invariant that was not made clear with an explicit: invariant { assert(length <= data.length); }
Feb 26
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/26/2021 1:57 AM, Walter Bright wrote:
 2. disallow void initialization if the struct has an invariant
Thinking about this some more. An invariant specifies a subset of values that the fields could have, while a void initialization means any random values are acceptable. Therefore (2) makes sense.
Feb 26
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 26.02.21 11:08, Walter Bright wrote:
 On 2/26/2021 1:57 AM, Walter Bright wrote:
 2. disallow void initialization if the struct has an invariant
Thinking about this some more. An invariant specifies a subset of values that the fields could have, while a void initialization means any random values are acceptable. Therefore (2) makes sense.
An invariant is anything that remains true throughout the execution, and it's not always easy to check. (2) makes sense, but it's not a substitute for system fields. safe code shouldn't be able to mess with unsafe values at all, not only through `void` initialization, and system code should be able to define its own unsafe values.
Feb 26
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 26 February 2021 at 09:57:03 UTC, Walter Bright wrote:
 Example: Short String
This example boils down to an invariant not being maintained if the struct is void-initialized.
Note that it's not just void initialization. Overlap in a union and reinterpreting casts can also create instances with broken invariants; e.g., union U { int n; ShortString s; } safe void main() { U u = { n: 0xDEADBEEF }; writeln(u.s[]); // undefined behavior }
Feb 26
parent reply Walter Bright <newshound2 digitalmars.com> writes:
https://issues.dlang.org/show_bug.cgi?id=21665
Feb 26
parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 27 February 2021 at 05:59:23 UTC, Walter Bright 
wrote:
 https://issues.dlang.org/show_bug.cgi?id=21665
Would that be a compiler error or warning ignoring initialization if you tried to void init a struct having invariant?
Feb 27
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/27/2021 12:38 AM, Imperatorn wrote:
 On Saturday, 27 February 2021 at 05:59:23 UTC, Walter Bright wrote:
 https://issues.dlang.org/show_bug.cgi?id=21665
Would that be a compiler error or warning ignoring initialization if you tried to void init a struct having invariant?
error
Feb 27
prev sibling next sibling parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker wrote:
 This is the discussion thread for the second round of Community 
 Review of DIP 1035, " system Variables":

 [...]
Well done. Everything in the direction of safety if very important to our company. Great work.
Feb 26
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 26.02.21 13:34, Imperatorn wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker wrote:
 This is the discussion thread for the second round of Community Review 
 of DIP 1035, " system Variables":

 [...]
Well done. Everything in the direction of safety if very important to our company. Great work.
Feel free to contribute to what is important to you instead of unproductively trolling discussion threads you do not care about. Thanks. This is my last post in this subthread.
Feb 26
parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Friday, 26 February 2021 at 14:04:45 UTC, Timon Gehr wrote:
 On 26.02.21 13:34, Imperatorn wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker 
 wrote:
 This is the discussion thread for the second round of 
 Community Review of DIP 1035, " system Variables":

 [...]
Well done. Everything in the direction of safety if very important to our company. Great work.
Feel free to contribute to what is important to you instead of unproductively trolling discussion threads you do not care about. Thanks. This is my last post in this subthread.
I don't think you understand the definition of trolling unfortunately. Your behavior is just objectively wierd. You have no idea what I care about. If there are something happening in you life that make you socially underperformed or unable to have good manners in general, direct your last watt of energy towards fixing that instead instead of writing in the forums. Thanks.
Feb 26
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 26.02.21 13:34, Imperatorn wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker wrote:
 This is the discussion thread for the second round of Community Review 
 of DIP 1035, " system Variables":

 [...]
Well done. Everything in the direction of safety if very important to our company. Great work.
On 26.02.21 14:04, Imperatorn wrote:
 
 I don't think you understand the definition of trolling unfortunately.
For some reason I read your original comment as sarcastic. I think I had confused you with another member of the forums who is also posting under a pseudonym. Sorry about that. Mea culpa. On 26.02.21 14:04, Imperatorn wrote:
 If there are something happening in you life that make you socially 
underperformed or unable to have good manners in general, [...] I am sorry for having provoked you into violating basic standards of decency.
Feb 26
parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Friday, 26 February 2021 at 15:14:51 UTC, Timon Gehr wrote:
 On 26.02.21 13:34, Imperatorn wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker 
 wrote:
 This is the discussion thread for the second round of 
 Community Review of DIP 1035, " system Variables":

 [...]
Well done. Everything in the direction of safety if very important to our company. Great work.
 For some reason I read your original comment as sarcastic. I 
 think I had confused you with another member of the forums who 
 is also posting under a pseudonym. Sorry about that. Mea culpa.
I see. No worries. I actually just wanted to say that it was good work and that we care about that. Misunderstandings happens sometimes. Peace
Feb 26
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
A simple workaround is an unsafe wrapper:

struct Unsafe
{
	private T a;
	T get()  system { return a; }
}

struct IntSlice
{
	private Unsafe!(int*) ptr;
	private Unsafe!size_t length;
...
Feb 27
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 February 2021 at 08:23:38 UTC, Kagamin wrote:
 A simple workaround is an unsafe wrapper:

 struct Unsafe
 {
 	private T a;
 	T get()  system { return a; }
 }

 struct IntSlice
 {
 	private Unsafe!(int*) ptr;
 	private Unsafe!size_t length;
 ...
The compiler will still allow you to void-initialize/overlap/reinterpret-cast an Unsafe!size_t in safe code, because it considers size_t to be a safe type. You *can* get the compiler to treat any value as unsafe by overlapping it with a pointer; e.g., struct Unsafe(T) { union Storage { T value; void* dummy; } Storage storage; ref T get() { return Storage.value; } /* ... */ } But this introduces memory overhead for any T smaller than a pointer, which is not ideal.
Feb 27
next sibling parent reply Kagamin <spam here.lot> writes:
On Saturday, 27 February 2021 at 13:32:24 UTC, Paul Backus wrote:
 The compiler will still allow you to 
 void-initialize/overlap/reinterpret-cast an Unsafe!size_t in 
  safe code, because it considers size_t to be a safe type.
The wrapped value still won't be accessible to safe code.
Feb 27
parent Dennis <dkorpel gmail.com> writes:
On Saturday, 27 February 2021 at 15:19:14 UTC, Kagamin wrote:
 On Saturday, 27 February 2021 at 13:32:24 UTC, Paul Backus 
 wrote:
 The compiler will still allow you to 
 void-initialize/overlap/reinterpret-cast an Unsafe!size_t in 
  safe code, because it considers size_t to be a safe type.
The wrapped value still won't be accessible to safe code.
You can access the wrapped value through a union in safe code, and your trusted code can't assume anything about your Unsafe!T, so it does not accomplish much.
Mar 02
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/27/21 8:32 AM, Paul Backus wrote:
 On Saturday, 27 February 2021 at 08:23:38 UTC, Kagamin wrote:
 A simple workaround is an unsafe wrapper:

 struct Unsafe
 {
     private T a;
     T get()  system { return a; }
 }

 struct IntSlice
 {
     private Unsafe!(int*) ptr;
     private Unsafe!size_t length;
 ...
The compiler will still allow you to void-initialize/overlap/reinterpret-cast an Unsafe!size_t in safe code, because it considers size_t to be a safe type. You *can* get the compiler to treat any value as unsafe by overlapping it with a pointer; e.g., struct Unsafe(T) {     union Storage { T value; void* dummy; }     Storage storage;     ref T get() { return Storage.value; }     /* ... */ } But this introduces memory overhead for any T smaller than a pointer, which is not ideal.
Better mark that get as system, or it can be used in safe code. -Steve
Feb 27
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 February 2021 at 15:21:41 UTC, Steven 
Schveighoffer wrote:
 On 2/27/21 8:32 AM, Paul Backus wrote:
 You *can* get the compiler to treat any value as unsafe by 
 overlapping it with a pointer; e.g.,
 
 struct Unsafe(T)
 {
      union Storage { T value; void* dummy; }
      Storage storage;
      ref T get() { return Storage.value; }
      /* ... */
 }
 
 But this introduces memory overhead for any T smaller than a 
 pointer, which is not ideal.
Better mark that get as system, or it can be used in safe code.
It's in a template, so it will be inferred as system.
Feb 27
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/27/21 10:55 AM, Paul Backus wrote:
 On Saturday, 27 February 2021 at 15:21:41 UTC, Steven Schveighoffer wrote:
 On 2/27/21 8:32 AM, Paul Backus wrote:
 You *can* get the compiler to treat any value as unsafe by 
 overlapping it with a pointer; e.g.,

 struct Unsafe(T)
 {
      union Storage { T value; void* dummy; }
      Storage storage;
      ref T get() { return Storage.value; }
      /* ... */
 }

 But this introduces memory overhead for any T smaller than a pointer, 
 which is not ideal.
Better mark that get as system, or it can be used in safe code.
It's in a template, so it will be inferred as system.
D considers that safe. It was a whole section of my Dconf online talk. I filed a bug report on it (in which you seem to think this is OK): https://issues.dlang.org/show_bug.cgi?id=21565 Even without templates: struct DThinksThisIsSafe { union Storage { int value; void *dummy; } Storage storage; safe ref int get() return { return storage.value; } // compiles just fine } -Steve
Feb 27
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 February 2021 at 19:05:01 UTC, Steven 
Schveighoffer wrote:
 D considers that  safe. It was a whole section of my Dconf 
 online talk. I filed a bug report on it (in which you seem to 
 think this is OK): 
 https://issues.dlang.org/show_bug.cgi?id=21565

 Even without templates:

 struct DThinksThisIsSafe
 {
    union Storage { int value; void *dummy; }
    Storage storage;
     safe ref int get() return { return storage.value; } // 
 compiles just fine
 }
Ah, right, because you're only accessing the int, not the pointer. Good catch.
Feb 27
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 February 2021 at 20:07:30 UTC, Paul Backus wrote:
 Ah, right, because you're only accessing the int, not the 
 pointer. Good catch.
...which means the whole approach doesn't actually work to begin with. D is perfectly within its rights to let you void-initialize the union (even though it currently doesn't), because safe code can't access the pointer anyway, so it can never lead to undefined behavior.
Feb 27
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/27/21 3:12 PM, Paul Backus wrote:
 On Saturday, 27 February 2021 at 20:07:30 UTC, Paul Backus wrote:
 Ah, right, because you're only accessing the int, not the pointer. 
 Good catch.
It's telling that you intuitively thought the system should prevent you from doing this (as you should!)
 
 ...which means the whole approach doesn't actually work to begin with. D 
 is perfectly within its rights to let you void-initialize the union 
 (even though it currently doesn't), because  safe code can't access the 
 pointer anyway, so it can never lead to undefined behavior.
D is perfectly within its rights to do whatever it wants for safe code. It could let you write an array length without extending the array, and then only allow you accessing the single element pointed at. It could prevent dereferencing pointers, and still be considered memory-safe. But there is still the question of whether this is useful to programmers or not. -Steve
Feb 27
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 February 2021 at 20:54:34 UTC, Steven 
Schveighoffer wrote:
 D is perfectly within its rights to do whatever it wants for 
  safe code. It could let you write an array length without 
 extending the array, and then only allow you accessing the 
 single element pointed at. It could prevent dereferencing 
 pointers, and still be considered memory-safe. But there is 
 still the question of whether this is useful to programmers or 
 not.
The general trend in the design of safe, as exemplified by DIP 25 and DIP 1000, has been to lift restrictions when the compiler can prove they are not necessary to prevent UB. And I would argue that this is exactly what one wants in a proof system: the strongest result (memory safety) from the fewest axioms (restrictions on safe code). Another way to look at it is that automatically extending an array when its length is changed allows the compiler to prove memory safety for programs that would otherwise require trusted, so it is a useful restriction. On the other hand, forbidding access to safe members of unions does not make any previously- trusted programs safe, so it is a useless restriction. I understand from previous discussions that you have some less-rigorous ideas about what is "useful to programmers" and what is not, but I think this is an occasion where rigor is warranted.
Feb 27
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/27/21 4:13 PM, Paul Backus wrote:
 I understand from previous discussions that you have some less-rigorous 
 ideas about what is "useful to programmers" and what is not, but I think 
 this is an occasion where rigor is warranted.
This is going quite off topic, but I wanted to say this is absolutely not the driver for my point of view. It is not a matter of rigor but a matter of what are the expectations of what safe should imply. My point of view is that if the semantic meaning of a a safe union between an int and a pointer currently is that the pointer isn't usable even in trusted code (even though the compiler doesn't prevent you from doing it), then this is going to violate the expectations of the programmer (why would he write a union, when you can only use one of the members). We should not only focus in our concept of safe in proving the memory safety of the rules we come up with, but in coming up with rules that make sense in the context of utility to the programmer. We can make up whatever rules we want, and based on those rules, we can prove safety, but if the result is "you can do this, but it NEVER is usable", I think we can do better. -Steve
Feb 27
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker wrote:
 This is the discussion thread for the second round of Community 
 Review of DIP 1035, " system Variables":

 [...]
"pointers, arrays, and other reference types are unsafe" This confused me. Pointers aren't unsafe unless you got one from system code. Could you clarify please?
Mar 01
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 1 March 2021 at 20:55:40 UTC, Atila Neves wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker 
 wrote:
 This is the discussion thread for the second round of 
 Community Review of DIP 1035, " system Variables":

 [...]
"pointers, arrays, and other reference types are unsafe" This confused me. Pointers aren't unsafe unless you got one from system code. Could you clarify please?
The language here is sloppy, but the intent is to refer to pointer *types*, not pointer *values*. It should read, "pointer types, dynamic array types, and other reference types are unsafe."
Mar 01
next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 2 March 2021 at 02:46:58 UTC, Paul Backus on the 
feedback thread wrote:
 On Monday, 1 March 2021 at 23:38:28 UTC, Dukc wrote:
 Which leads to, what are the values compiler considers to be 
 safe?
The answer to this question is is already part of the language spec: https://dlang.org/spec/function.html#safe-values
Yes, you can use that. But the dip needs to update that part of the spec, since what is going to be a safe struct instance / union instance / class ref value is changing. For instance, what the compiler will consider safe values for the `ShortString` struct provided in the example?
Mar 02
parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 2 March 2021 at 11:48:09 UTC, Dukc wrote:
 On Tuesday, 2 March 2021 at 02:46:58 UTC, Paul Backus on the 
 feedback thread wrote:
 On Monday, 1 March 2021 at 23:38:28 UTC, Dukc wrote:
 Which leads to, what are the values compiler considers to be 
 safe?
The answer to this question is is already part of the language spec: https://dlang.org/spec/function.html#safe-values
Yes, you can use that. But the dip needs to update that part of the spec, since what is going to be a safe struct instance / union instance / class ref value is changing. For instance, what the compiler will consider safe values for the `ShortString` struct provided in the example?
Good point. For user-defined unsafe types like ShortString, the compiler will have to assume that any value that can be obtained in safe code is a safe value.
Mar 02
parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 2 March 2021 at 14:09:01 UTC, Paul Backus wrote:
 Good point. For user-defined unsafe types like ShortString, the 
 compiler will have to assume that any value that can be 
 obtained in  safe code is a safe value.
But what about determining ` safe`ty of value given by ` system` initializer? Is it just not attempted for user-defined types? Or is the rule "`.init` value is ` safe`, everything else is ` system`"? Or something more complicated?
Mar 02
parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 2 March 2021 at 14:28:43 UTC, Dukc wrote:
 On Tuesday, 2 March 2021 at 14:09:01 UTC, Paul Backus wrote:
 Good point. For user-defined unsafe types like ShortString, 
 the compiler will have to assume that any value that can be 
 obtained in  safe code is a safe value.
But what about determining ` safe`ty of value given by ` system` initializer? Is it just not attempted for user-defined types? Or is the rule "`.init` value is ` safe`, everything else is ` system`"? Or something more complicated?
I think you may be confusing the safety of a *value* with the safety of a *variable*. Consider the following example: safe ShortString makeShortString(string source) { /* ... */ } auto s = makeShortString("hello"); The value of `makeShortString("hello")` is *assumed* to be a safe value, because `makeShortString` is a safe function. Therefore, the variable `s` is *determined* to be a safe variable, because its initializer is a safe value. Regarding `.init`: because default initialization of any type is allowed in safe code, the `.init` value of any user-defined type must be a safe value. This is true today, and it will still be true if DIP 1035 is accepted. The fact that the compiler currently does not enforce this is a bug. [1] [1] https://issues.dlang.org/show_bug.cgi?id=21675
Mar 02
parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 2 March 2021 at 14:57:04 UTC, Paul Backus wrote:
 I think you may be confusing the safety of a *value* with the 
 safety of a *variable*.
No, I was thinking this piece:
 An exception to the above rules is made on unsafe types when
 the compiler knows the resulting value is safe.
 ```D
 int* getNull() pure  system {return null;}
 int* n = getNull(); // despite unsafe type with  system
 initialization expression, inferred as  safe
 ```
So the queastion is: ```D ShortString getSString() pure system {<...>} //under what conditions, if any, is the initialization inferred as safe here? ShortString n = getSString(); ```
Mar 02
parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 2 March 2021 at 15:17:54 UTC, Dukc wrote:
 So the queastion is:

 ```D
 ShortString getSString() pure  system {<...>}
 //under what conditions, if any, is the initialization inferred 
 as  safe here?
 ShortString n = getSString();
 ```
In this example, `n` will always be inferred as a system variable.
Mar 02
parent Dukc <ajieskola gmail.com> writes:
On Tuesday, 2 March 2021 at 16:04:14 UTC, Paul Backus wrote:
 On Tuesday, 2 March 2021 at 15:17:54 UTC, Dukc wrote:
 So the queastion is:

 ```D
 ShortString getSString() pure  system {<...>}
 //under what conditions, if any, is the initialization 
 inferred as  safe here?
 ShortString n = getSString();
 ```
In this example, `n` will always be inferred as a system variable.
Fair enough. Remember to mention that in the DIP - either at the place I just quoted or at the changes you propose to what is considered a safe value.
Mar 02
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 1 March 2021 at 21:26:18 UTC, Paul Backus wrote:
 On Monday, 1 March 2021 at 20:55:40 UTC, Atila Neves wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker 
 wrote:
 This is the discussion thread for the second round of 
 Community Review of DIP 1035, " system Variables":

 [...]
"pointers, arrays, and other reference types are unsafe" This confused me. Pointers aren't unsafe unless you got one from system code. Could you clarify please?
The language here is sloppy, but the intent is to refer to pointer *types*, not pointer *values*. It should read, "pointer types, dynamic array types, and other reference types are unsafe."
Pointer types *can* be unsafe, if the values came from system code. Otherwise they're perfectly safe. Slices (dynamic arrays) are slightly different because of the necessity of bounds checks. But deferencing a pointer is fine in safe code - the possibilities are: * it came from the GC. * is the address of a module-level variable. * is a scoped address on the stack. * is null. Am I missing a case?
Mar 02
next sibling parent ag0aep6g <anonymous example.com> writes:
On 02.03.21 21:46, Atila Neves wrote:
 Pointer types *can* be unsafe, if the values came from  system code. 
Pointer *values* can be unsafe, if they come from system code. A type that can have unsafe values is an unsafe type (so defined in the DIP). So pointer types are unsafe types.
Mar 02
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 2 March 2021 at 20:46:17 UTC, Atila Neves wrote:
 Pointer types *can* be unsafe, if the values came from  system 
 code. Otherwise they're perfectly safe. Slices (dynamic arrays) 
 are slightly different because of the necessity of bounds 
 checks. But deferencing a pointer is fine in  safe code - the 
 possibilities are:

 * it came from the GC.
 * is the address of a module-level variable.
 * is a scoped address on the stack.
 * is null.

 Am I missing a case?
You are conflating types and values. Pointer *values* can be either safe or unsafe, depending on what they point to. Pointer *types* are always unsafe, because they include both safe and unsafe values.
Mar 02
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 2 March 2021 at 21:41:40 UTC, Paul Backus wrote:
 You are conflating types and values. Pointer *values* can be 
 either safe or unsafe, depending on what they point to. Pointer 
 *types* are always unsafe, because they include both safe and 
 unsafe values.
A type-system can be made to distinguish between safe and unsafe pointers. A type is just a tag stuck onto a value.
Mar 02
parent Paulo Pinto <pjmlp progtools.org> writes:
On Tuesday, 2 March 2021 at 22:06:30 UTC, Ola Fosheim Grøstad 
wrote:
 On Tuesday, 2 March 2021 at 21:41:40 UTC, Paul Backus wrote:
 You are conflating types and values. Pointer *values* can be 
 either safe or unsafe, depending on what they point to. 
 Pointer *types* are always unsafe, because they include both 
 safe and unsafe values.
A type-system can be made to distinguish between safe and unsafe pointers. A type is just a tag stuck onto a value.
this pointer distinction. Pointers are always safe, unless explicitly marked otherwise or used in unsafe contexts, converting between both types is also explicit.
Mar 02
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 2 March 2021 at 21:41:40 UTC, Paul Backus wrote:
 On Tuesday, 2 March 2021 at 20:46:17 UTC, Atila Neves wrote:
 Pointer types *can* be unsafe, if the values came from  system 
 code. Otherwise they're perfectly safe. Slices (dynamic 
 arrays) are slightly different because of the necessity of 
 bounds checks. But deferencing a pointer is fine in  safe code 
 - the possibilities are:

 * it came from the GC.
 * is the address of a module-level variable.
 * is a scoped address on the stack.
 * is null.

 Am I missing a case?
You are conflating types and values. Pointer *values* can be either safe or unsafe, depending on what they point to. Pointer *types* are always unsafe, because they include both safe and unsafe values.
I don't think I am, but I think I understand where you're coming from. Let me restate my point and maybe then it will be clearer: if all the code in a program is safe, then pointers are memory safe (with DIP1000). I guess I'd argue that pointer types are safe unless the value was obtained from system code. But throw system code into the mix... Anyway, the wording confused me.
Mar 04
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/4/21 1:23 PM, Atila Neves wrote:
 On Tuesday, 2 March 2021 at 21:41:40 UTC, Paul Backus wrote:
 On Tuesday, 2 March 2021 at 20:46:17 UTC, Atila Neves wrote:
 Pointer types *can* be unsafe, if the values came from  system code. 
 Otherwise they're perfectly safe. Slices (dynamic arrays) are 
 slightly different because of the necessity of bounds checks. But 
 deferencing a pointer is fine in  safe code - the possibilities are:

 * it came from the GC.
 * is the address of a module-level variable.
 * is a scoped address on the stack.
 * is null.

 Am I missing a case?
You are conflating types and values. Pointer *values* can be either safe or unsafe, depending on what they point to. Pointer *types* are always unsafe, because they include both safe and unsafe values.
I don't think I am, but I think I understand where you're coming from. Let me restate my point and maybe then it will be clearer: if all the code in a program is safe, then pointers are memory safe (with DIP1000). I guess I'd argue that pointer types are safe unless the value was obtained from system code. But throw system code into the mix... Anyway, the wording confused me.
At the very top of the DIP: "D's memory safety system distinguishes between safe values, which can be used freely in safe code without causing undefined behavior, and unsafe values, which cannot. A type that has only safe values is a safe type; one that has both safe and unsafe values is an unsafe type." Unsaid here is that it doesn't matter where it comes from ( safe or system). This is how they are defining "safe types" and "unsafe types". Everything follows from that. Given that definition, pointers are unsafe. -Steve
Mar 04
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 4 March 2021 at 18:47:28 UTC, Steven Schveighoffer 
wrote:
 On 3/4/21 1:23 PM, Atila Neves wrote:
 [...]
At the very top of the DIP: "D's memory safety system distinguishes between safe values, which can be used freely in safe code without causing undefined behavior, and unsafe values, which cannot. A type that has only safe values is a safe type; one that has both safe and unsafe values is an unsafe type." Unsaid here is that it doesn't matter where it comes from ( safe or system). This is how they are defining "safe types" and "unsafe types". Everything follows from that. Given that definition, pointers are unsafe. -Steve
That makes sense, but I'm not sure that's how we *should* define it, given that pointers are memory-safe in safe code.
Mar 04
prev sibling parent Paul Backus <snarwin gmail.com> writes:
On Thursday, 4 March 2021 at 18:23:00 UTC, Atila Neves wrote:
 On Tuesday, 2 March 2021 at 21:41:40 UTC, Paul Backus wrote:
 You are conflating types and values. Pointer *values* can be 
 either safe or unsafe, depending on what they point to. 
 Pointer *types* are always unsafe, because they include both 
 safe and unsafe values.
I don't think I am, but I think I understand where you're coming from. Let me restate my point and maybe then it will be clearer: if all the code in a program is safe, then pointers are memory safe (with DIP1000). I guess I'd argue that pointer types are safe unless the value was obtained from system code. But throw system code into the mix... Anyway, the wording confused me.
I agree that the wording is confusing. In particular, the words "safe" and "unsafe" are heavily overloaded. Unfortunately, "safe values" and " safe code" are both official language-spec terms, so there's nothing the DIP can do about those. It would probably still be helpful to replace "safe type" and "unsafe type" with something more distinct. Maybe "unrestricted type" and "restricted type"? E.g., Pointer *values* can be either safe or unsafe, depending on what they point to. Pointer *types* are always restricted types, because they include both safe and unsafe values. A restricted type is any type which has limits placed on what you are allowed to do with it in safe code. Hopefully it is clear that a type must be restricted if and only if it includes unsafe values.
Mar 04
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 2 March 2021 at 20:46:17 UTC, Atila Neves wrote:
 On Monday, 1 March 2021 at 21:26:18 UTC, Paul Backus wrote:
 On Monday, 1 March 2021 at 20:55:40 UTC, Atila Neves wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker 
 wrote:
 This is the discussion thread for the second round of 
 Community Review of DIP 1035, " system Variables":

 [...]
"pointers, arrays, and other reference types are unsafe" This confused me. Pointers aren't unsafe unless you got one from system code. Could you clarify please?
The language here is sloppy, but the intent is to refer to pointer *types*, not pointer *values*. It should read, "pointer types, dynamic array types, and other reference types are unsafe."
Pointer types *can* be unsafe, if the values came from system code. Otherwise they're perfectly safe.
The correct wording to this is: pointer *values* can be unsafe if they come from ` system` code, therefore pointer *types* are unsafe, as defined by this DIP. When the DIP talks about unsafe type, it really means only potentially unsafe type.
 Slices (dynamic arrays) are slightly different because of the 
 necessity of bounds checks.
They are not. `(ubyte*).init[0xBAA..0xBEE]` is just as unsafe as `cast(int*)0xDEADBEEF`, regardless of whether there are bound checks when accessing it. Bounds checked access is safe only if the value is safe, just as with pointer dereference. But deferencing a pointer is fine in safe code - the
 possibilities are:

 * it came from the GC.
 * is the address of a module-level variable.
 * is a scoped address on the stack.
 * is null.

 Am I missing a case?
These are all true. The intention of the DIP is that you could make user-defined types that behave the same way - that ` safe` would guarantee they will not have unsafe values. Think about defining a 16-bit pointer for 32-bit or 64-bit architecture. If you now declare it as `struct ptr16{private short handle;}` you will not be able to make it work in safe code, as it can be void-initialized with wrong values. You'll have to do something like `union ptr16{short handle; void[2] unsafeMarker;}` instead.
Mar 03
parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 3 March 2021 at 12:54:53 UTC, Dukc wrote:
 On Tuesday, 2 March 2021 at 20:46:17 UTC, Atila Neves wrote:
 On Monday, 1 March 2021 at 21:26:18 UTC, Paul Backus wrote:
 On Monday, 1 March 2021 at 20:55:40 UTC, Atila Neves wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker 
 wrote:
 This is the discussion thread for the second round of 
 Community Review of DIP 1035, " system Variables":

 [...]
 Slices (dynamic arrays) are slightly different because of the 
 necessity of bounds checks.
They are not. `(ubyte*).init[0xBAA..0xBEE]` is just as unsafe as `cast(int*)0xDEADBEEF`, regardless of whether there are bound checks when accessing it.
Yes, but not allowed in safe code. My point is that safe can still do: auto arr = new int[7]; auto oops = arr[3_000_000];
Mar 04
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 2 March 2021 at 13:56:28 UTC, Dennis in the feedback 
thread wrote:
 There is no disagreement, the rationale text you mention 
 describes the situation before DIP 1035, while the description 
 describes the situation after DIP 1035.
How is this piece from the rationale section describing the situation before DIP1035? "Since the initialization expression cast(int*) 0xDEADBEEF would not be allowed in a safe function, and since the initial value of y is unknown, the compiler should annotate variables x and y as possibly containing an unsafe value, so they cannot be accessed in a safe function. Only z is known to have a safe initial value in this case, so the compiler could allow access to it in safe code."
Mar 02
parent reply Dennis <dkorpel gmail.com> writes:
On Tuesday, 2 March 2021 at 15:34:31 UTC, Dukc wrote:
 How is this piece from the rationale section describing the 
 situation before DIP1035?

 "Since the initialization expression cast(int*) 0xDEADBEEF 
 would not be allowed in a  safe function, and since the initial 
 value of y is unknown, the compiler should annotate variables x 
 and y as possibly containing an unsafe value, so they cannot be 
 accessed in a  safe function. Only z is known to have a safe 
 initial value in this case, so the compiler could allow access 
 to it in  safe code."
I don't understand the question. Is there a contradiction in that paragraph I'm supposed to see? Note that the description is about what the compiler _should_ do given the current language semantics, not what dmd actually does. The version of the DIP from the previous review round acknowledged existing holes in safe and was critiqued for it, so this version tries to use the language specification as a base rather than the implementation. (Though that's not always easy, since the specification is lacking in certain areas as well)
Mar 03
parent Dukc <ajieskola gmail.com> writes:
On Wednesday, 3 March 2021 at 10:05:34 UTC, Dennis wrote:
 On Tuesday, 2 March 2021 at 15:34:31 UTC, Dukc wrote:
 How is this piece from the rationale section describing the 
 situation before DIP1035?

 [snip]
I don't understand the question.
Don't worry, I think you managed to answer it anyway.
 Is there a contradiction in that paragraph I'm supposed to see? 
 Note that the description is about what the compiler _should_ 
 do given the current language semantics, not what dmd actually 
 does.
That explains it. I thought that since the dip says "should" and dmd currently does not do it (interpreting `extern int*` as ` system`), I thought it was explaining what the DIP will do. My misunderstanding, though changing a few words might be in order to avoid this confusion.
Mar 03
prev sibling next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker wrote:
 This is the discussion thread for the second round of Community 
 Review of DIP 1035, " system Variables":
I don't have time to dig into this right now, and the current system/ safe distinction is in need of being replaced by «something». But, the problem with adding more stuff to this aspect of the type system is that the language is becoming increasingly convoluted. At some point local improvements no longer improve the whole... and a redesign is the only reasonable approach.
Mar 02
parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 2 March 2021 at 16:30:59 UTC, Ola Fosheim Grøstad 
wrote:
 But, the problem with adding more stuff to this aspect of the 
 type system is that the language is becoming increasingly 
 convoluted.

 At some point local improvements no longer improve the whole... 
 and a redesign is the only reasonable approach.
This particular improvement should not add that much complexity cost, as it is rather a completion of a previously added feature ( safe/ trusted/ system) than a new feature. If you know how memory safety of D works right now, you can probably figure out what ` system extern int;` does. And if you have `struct mySlice{int* ptr; system size_t length;}` it is not probably going to surprise you that you can't directly write to either of the slice variables.
Mar 02
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 2 March 2021 at 19:32:38 UTC, Dukc wrote:
 This particular improvement should not add that much complexity 
 cost, as it is rather a completion of a previously added 
 feature ( safe/ trusted/ system) than a new feature.
Maybe, but one could probably say that for every individual feature. E.g. the various " safe" parameter mechanisms, scope, live. It adds up.
Mar 02
prev sibling next sibling parent reply Guillaume Piolat <first.name spam.org> writes:
On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker wrote:
 The review period will end at 11:59 PM ET on March 11, or when 
 I make a post declaring it complete. Discussion in this thread 
 may continue beyond that point.

 Here in the discussion thread, you are free to discuss anything 
 and everything related to the DIP. Express your support or 
 opposition, debate alternatives, argue the merits, etc.
So, in a few words, the merit of the DIP is that after this change, you have to review only trusted functions to ensure a safe function hasn't broken an invariant of the aggregate. Which is strange to me since I was under the impression safe/ trusted was only about memory safety. But in a way it makes sense to use it for "general bugs" since trusted function will be audited. So, no real opinion here. Why not.
Mar 02
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 02.03.21 23:15, Guillaume Piolat wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker wrote:
 The review period will end at 11:59 PM ET on March 11, or when I make 
 a post declaring it complete. Discussion in this thread may continue 
 beyond that point.

 Here in the discussion thread, you are free to discuss anything and 
 everything related to the DIP. Express your support or opposition, 
 debate alternatives, argue the merits, etc.
So, in a few words, the merit of the DIP is that after this change, you have to review only trusted functions to ensure a safe function hasn't broken an invariant of the aggregate. Which is strange to me since I was under the impression safe/ trusted was only about memory safety. But in a way it makes sense to use it for "general bugs" since trusted function will be audited. So, no real opinion here. Why not.
system variables are still about memory safety.
Mar 02
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/2/21 5:15 PM, Guillaume Piolat wrote:
 On Thursday, 25 February 2021 at 09:21:20 UTC, Mike Parker wrote:
 The review period will end at 11:59 PM ET on March 11, or when I make 
 a post declaring it complete. Discussion in this thread may continue 
 beyond that point.

 Here in the discussion thread, you are free to discuss anything and 
 everything related to the DIP. Express your support or opposition, 
 debate alternatives, argue the merits, etc.
So, in a few words, the merit of the DIP is that after this change, you have to review only trusted functions to ensure a safe function hasn't broken an invariant of the aggregate.
I would change this to "you can make it possible" to only have to review trusted functions. Without actually marking the variables system, you would still have to review safe code.
 Which is strange to me since I was under the impression  safe/ trusted 
 was only about memory safety. But in a way it makes sense to use it for 
 "general bugs" since  trusted function will be audited. So, no real 
 opinion here. Why not.
Inside a trusted function, you can assume that any data entering the function is only set to values that obey the safe rules. Currently, that means certain invariants cannot be realized (like that a length field identifies how many bytes are valid past a pointer field). This DIP allows you to restrict the rules further so more assumptions can be made in the trusted function. You can assume that system variables never were modified directly by safe functions, and therefore, it's only necessary to review trusted and system functions to ensure the invariants hold. Given the theory that most code is safe and should be marked or inferred that way, you can therefore vastly reduce the amount of code that needs review for memory safety. -Steve
Mar 04
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/25/21 4:21 AM, Mike Parker wrote:
 This is the discussion thread for the second round of Community Review 
 of DIP 1035, " system Variables":
 
 https://github.com/dlang/DIPs/blob/c39f6ac62210e0604dcee99b0092c1930839
93a/DIPs/DIP1035.md 
 
 
In the example for (2): struct Handle { system int handle; } // struct with system field is an unsafe type safe Handle safeHandle = Handle(1); system Handle systemHandle = Handle(-1); ... void main() safe { Handle h0 = safeHandle; // allowed, safe variable Handle h1 = systemHandle; // error, reading system var of unsafe type ... } I'm concerned about the allowance of just declaring a safe Handle. If Handle(-1) is unsafe, what is stopping me from doing: safe Handle sneakyHandle = Handle(-1); And can I just do this inside main(): Handle h2 = Handle(-1); I guess my biggest problem with this DIP is surrounding the allowance of initialization of system variables without requiring a system call. And/or the weird rules of "you can't do it if it's a system variable, but perfectly fine if you type out the initializer" Or maybe I'm misunderstanding something. Perhaps it would be good to specify how one prevents anything in safe from using Handle(-1). ------ Another note, the ShortString example is unsafe, even with the DIP, as `s[]` will provide access to data that might move elsewhere. Perhaps it would be good to restate the examples with the assumption the DIP is implemented, and show why they are now fully safe. -Steve
Mar 04
parent Paul Backus <snarwin gmail.com> writes:
On Thursday, 4 March 2021 at 20:10:05 UTC, Steven Schveighoffer 
wrote:
 In the example for (2):

 struct Handle {
      system int handle;
 }
[...]
 If Handle(-1) is unsafe, what is stopping me from doing:

  safe Handle sneakyHandle = Handle(-1);
Nothing, because Handle(-1) isn't unsafe. If your next question is "then why use a system variable?", the answer is that this isn't intended to be a realistic example. It's just an illustration of the semantics of system variables. If you actually wanted to maintain an invariant like "Handles can't be negative", you would have to write a constructor: struct Handle { system int handle; this(int n) safe { assert(n >= 0); handle = n; } }
 Another note, the ShortString example is unsafe, even with the 
 DIP, as `s[]` will provide access to data that might move 
 elsewhere.
ShortString doesn't contain internal pointers, and the compiler won't let `s[]` outlive the ShortString's lexical scope, which means that the problem in issue 17448 [1] can't happen here. So it should be fine, unless there's something else I'm missing. It does require DIP 1000, which means that opIndex should be annotated with `return`. [1] https://issues.dlang.org/show_bug.cgi?id=17448
Mar 04