www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - memory safety checks and trust

reply Adam D. Ruppe <destructionator gmail.com> writes:
```
void main() {
         int a;
         b ~= &a;
}

int*[] b;
```

trust.d(3): Error: copying & a into allocated memory escapes a 
reference to local variable a


(Interestingly, `b = [&a]` instead of ~= passes muster. What's 
the difference? Just another bug in this?)


But the inconsistency isn't why I'm posting right now, it is my 
fear that D is losing faith in me. There seems to be no way to 
say "trust me" to the compiler. Note that I'm not even using any 
switches to dmd, and adding  trusted had no effect.

One of the nice things about D is that I very rarely feel like I 
am fighting the language (unless I'm working under constraints 
like -w and  safe pure nothrow  nogc stuff, but that's why I just 
don't use those!). From low-level bit twiddling to high level 
"just make it work", the D language usually works with me, a few 
exceptions excluded - but when those happen, you can cast or 
whatever to tell it to trust me.

But this new thing... can I tell it to trust me? Is that just a 
bug too? Or will I have to trick it with extern(C) or asm or 
something?

(PS I actually changed the code somewhat and used a pointer to a 
static instance which worked for my specific case. But I'm a bit 
concerned about the future.)
Apr 10
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:
 ```
 void main() {
          int a;
          b ~= &a;
 }
 
 int*[] b;
 ```
 
 trust.d(3): Error: copying & a into allocated memory escapes a reference to 
 local variable a
 
 
 (Interestingly, `b = [&a]` instead of ~= passes muster. What's the difference? 
 Just another bug in this?)
You will get the error with -preview=dip1000. Since that will eventually be the default, it's not a bug. You can get it to pass without error with the following: system int* foo(int* p) { return p; } system void test() { int a; b ~= &a; b ~= [foo(&a)]; } int*[] b; The compiler will inline foo(). I highly recommend annotating such code with &system.
 But the inconsistency isn't why I'm posting right now, it is my fear that D is 
 losing faith in me. There seems to be no way to say "trust me" to the
compiler. 
 Note that I'm not even using any switches to dmd, and adding  trusted had no 
 effect.
 
 One of the nice things about D is that I very rarely feel like I am fighting
the 
 language (unless I'm working under constraints like -w and  safe pure nothrow 
  nogc stuff, but that's why I just don't use those!). From low-level bit 
 twiddling to high level "just make it work", the D language usually works with 
 me, a few exceptions excluded - but when those happen, you can cast or
whatever 
 to tell it to trust me.
If system isn't letting you do what you need to do, let me know.
Apr 10
next sibling parent reply Johan <j j.nl> writes:
On Saturday, 11 April 2020 at 02:57:03 UTC, Walter Bright wrote:
 On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:
 ```
 void main() {
          int a;
          b ~= &a;
 }
 
 int*[] b;
 ```
 
 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a
 
 
 (Interestingly, `b = [&a]` instead of ~= passes muster. What's 
 the difference? Just another bug in this?)
You will get the error with -preview=dip1000. Since that will eventually be the default, it's not a bug. You can get it to pass without error with the following: system int* foo(int* p) { return p; } system void test() { int a; b ~= &a; b ~= [foo(&a)]; } int*[] b;
The OP's point was that exactly this does not compile. Trivial to test online: https://d.godbolt.org/z/i8WFcs -Johan
Apr 11
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/11/20 6:01 AM, Johan wrote:
 On Saturday, 11 April 2020 at 02:57:03 UTC, Walter Bright wrote:
 On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:
 ```
 void main() {
          int a;
          b ~= &a;
 }

 int*[] b;
 ```

 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a


 (Interestingly, `b = [&a]` instead of ~= passes muster. What's the 
 difference? Just another bug in this?)
You will get the error with -preview=dip1000. Since that will eventually be the default, it's not a bug. You can get it to pass without error with the following:   system int* foo(int* p) { return p; }   system void test() {         int a;         b ~= &a;         b ~= [foo(&a)];   }   int*[] b;
The OP's point was that exactly this does not compile. Trivial to test online: https://d.godbolt.org/z/i8WFcs
It does if you write it correctly: b ~= foo(&a); That was Walter's point. Once you get out of one expression, the checks stop. You can do this too: auto p = &; b ~= p; Note that a more robust argument for the OP's point is that you can easily make sure the allocation isn't used outside the function. What if you need scratch space to deal with things? e.g.: system void test() { int *[] buf; int a; buf ~= &a; // Same error } How is &a escaping here? -Steve
Apr 11
prev sibling next sibling parent jeckel <jeckel12381236 gmail.com> writes:
On Saturday, 11 April 2020 at 02:57:03 UTC, Walter Bright wrote:
 On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:
 ```
 void main() {
          int a;
          b ~= &a;
 }
 
 int*[] b;
 ```
 
 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a
 
 
 (Interestingly, `b = [&a]` instead of ~= passes muster. What's 
 the difference? Just another bug in this?)
You will get the error with -preview=dip1000. Since that will eventually be the default, it's not a bug. You can get it to pass without error with the following: system int* foo(int* p) { return p; } system void test() { int a; b ~= &a; b ~= [foo(&a)]; } int*[] b; The compiler will inline foo(). I highly recommend annotating such code with &system.
FYI, you don't have to put system. You can save yourself some time as system is the default. This is equivalent: int* foo(int* p) { return p; } void test() { int a; b ~= &a; b ~= [foo(&a)]; } int*[] b;
Apr 11
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11.04.20 04:57, Walter Bright wrote:
 On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:
 ```
 void main() {
          int a;
          b ~= &a;
 }

 int*[] b;
 ```

 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a


 (Interestingly, `b = [&a]` instead of ~= passes muster. What's the 
 difference? Just another bug in this?)
You will get the error with -preview=dip1000. Since that will eventually be the default, it's not a bug. ...
Clearly there is a bug or bad design if the address of a`` escaping in `b ~= &a` and in `b = [&a]` are not treated the same. But like Adam I don't see why there should be such a check in system/ trusted code at all. (I understand that there is a workaround, but that should not be required.) Can we please settle on making safe actually memory safe and system/ trusted actually trust the programmer?
Apr 11
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2020 7:43 PM, Timon Gehr wrote:
 Clearly there is a bug or bad design if the address of a`` escaping in `b ~=
&a` 
 and in `b = [&a]` are not treated the same.
They are treated the same with dip1000.
 But like Adam I don't see why there 
 should be such a check in  system/ trusted code at all. (I understand that
there 
 is a workaround, but that should not be required.)
 
 Can we please settle on making  safe actually memory safe and  system/ trusted 
 actually trust the programmer?
Consider: system int* pumpkin(int i) { return &i); Should that give an error or not? I.e. where does one draw the line?
Apr 13
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 13.04.20 09:50, Walter Bright wrote:
 
 But like Adam I don't see why there should be such a check in 
  system/ trusted code at all. (I understand that there is a 
 workaround, but that should not be required.)

 Can we please settle on making  safe actually memory safe and 
  system/ trusted actually trust the programmer?
Consider:   system int* pumpkin(int i) { return &i; } Should that give an error or not? ...
I don't see why not.
 I.e. where does one draw the line?
I think D has been in a pretty good place for a quite long time. Beyond straightforward parser/type system diagnostics that you can disable using explicit parentheses/type casts, only diagnose cases that have no legitimate use. You might not get this right, so it's probably a good idea to be open to the idea of removing/restricting diagnostics when people complain on the forums that their perfectly valid system/ trusted code was rejected. In system/ trusted code, if you diagnose, the code should usually be wrong and if it is not, the workaround should have the same behavior as if the diagnostic had not been there in the first place (e.g., don't slow down debug builds).
Apr 13
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2020 2:03 AM, Timon Gehr wrote:
 Consider:

     system int* pumpkin(int i) { return &i; }

 Should that give an error or not?
 ...
I don't see why not.
I've written code like that to get the stack pointer value. As to the other case, to my mind putting the address of a stack local into a GC allocated object is highly suspicious: 1. it is very very easy to do it unintentionally 2. GC objects tend to be intended to outlast a function, yet the stack address will not, so why is this a GC allocated object, as opposed to, say, an RAII object or even a stack allocated object? I don't know what Adam's actual use case for this is, but I would reject such code because of (1) and (2) even for system use in any project I have control over. Therefore, whether you agree with where I drew the line or not, this is a subjective decision not an objective one.
Apr 14
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/14/20 5:56 AM, Walter Bright wrote:
 On 4/13/2020 2:03 AM, Timon Gehr wrote:
 Consider:

     system int* pumpkin(int i) { return &i; }

 Should that give an error or not?
 ...
I don't see why not.
I've written code like that to get the stack pointer value.
Can't you use inline assembly?
 
 As to the other case, to my mind putting the address of a stack local 
 into a GC allocated object is highly suspicious:
 
 1. it is very very easy to do it unintentionally
 
 2. GC objects tend to be intended to outlast a function, yet the stack 
 address will not, so why is this a GC allocated object, as opposed to, 
 say, an RAII object or even a stack allocated object?
Because the stack doesn't provide a high limit of data space, and RAII may not be what you want to use. In system code, you should be free to shoot yourself in the foot, and also to correctly write memory safe code knowing where memory will stop being accessed. -Steve
Apr 14
parent aliak <something something.com> writes:
On Tuesday, 14 April 2020 at 12:32:18 UTC, Steven Schveighoffer 
wrote:
 On 4/14/20 5:56 AM, Walter Bright wrote:
 On 4/13/2020 2:03 AM, Timon Gehr wrote:
 Consider:

     system int* pumpkin(int i) { return &i; }

 Should that give an error or not?
 ...
I don't see why not.
I've written code like that to get the stack pointer value.
Can't you use inline assembly?
You can cast at least: ptrdiff_t f(int i) { return cast(ptrdiff_t)&i; } Also, I tried annotating that with safe but it didn't work. What's the danger of casting the address of a variable to an integer type thing? (I know that I can use trusted as a work around)
 
 As to the other case, to my mind putting the address of a 
 stack local into a GC allocated object is highly suspicious:
 
 1. it is very very easy to do it unintentionally
 
 2. GC objects tend to be intended to outlast a function, yet
GC objects are a matter of convenience and safety and are using within function scopes all the time.
 the stack address will not, so why is this a GC allocated 
 object, as opposed to, say, an RAII object or even a stack 
 allocated object?
Because the stack doesn't provide a high limit of data space, and RAII may not be what you want to use. In system code, you should be free to shoot yourself in the foot, and also to correctly write memory safe code knowing where memory will stop being accessed. -Steve
Another reason, you may not know how big your array is and need to rely on the GC (or manual memory management but the GC is just much more easier and safer to use)
Apr 14
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 14 April 2020 at 09:56:39 UTC, Walter Bright wrote:
 As to the other case, to my mind putting the address of a stack 
 local into a GC allocated object is highly suspicious:

 1. it is very very easy to do it unintentionally
Indeed, but isn't that what safe is for? There should be a way do it intentionally, to tell the compiler "trust me". That's my main point with this thread - D is being overbearing now. "Copy the pointer to that array please, D." "I'm afraid I can't do that, Adam." https://www.youtube.com/watch?v=ARJ8cAGm6JE
Apr 14
next sibling parent bachmeier <no spam.net> writes:
On Tuesday, 14 April 2020 at 13:34:00 UTC, Adam D. Ruppe wrote:
 On Tuesday, 14 April 2020 at 09:56:39 UTC, Walter Bright wrote:
 As to the other case, to my mind putting the address of a 
 stack local into a GC allocated object is highly suspicious:

 1. it is very very easy to do it unintentionally
Indeed, but isn't that what safe is for? There should be a way do it intentionally, to tell the compiler "trust me". That's my main point with this thread - D is being overbearing now. "Copy the pointer to that array please, D." "I'm afraid I can't do that, Adam." https://www.youtube.com/watch?v=ARJ8cAGm6JE
+1 Most of us aren't writing web browsers and probably none of us are working on airplanes. Give us enough rope to shoot ourselves in the foot. If I wanted a language to tell me how to program, I'd still be using Go.
Apr 14
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/14/2020 6:34 AM, Adam D. Ruppe wrote:
 Indeed, but isn't that what  safe is for? There should be a way do it 
 intentionally, to tell the compiler "trust me". That's my main point with this 
 thread - D is being overbearing now.
 
 "Copy the pointer to that array please, D."
So, Adam, how do you feel about: system int* pumpkin(int i) { return &i); Timon says that should be an error. What is your opinion?
Apr 14
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 14 April 2020 at 20:10:20 UTC, Walter Bright wrote:
 So, Adam, how do you feel about:

    system int* pumpkin(int i) { return &i);

 Timon says that should be an error. What is your opinion?
I could go either way on it. That does catch a real problem in real world Phobos use and it is very rarely what you actually want: string s() { return hexDigest!MD5("test"); } So I see the value in that error. But the array example is different anyway because the above thing is almost always wrong while the array is far more ambiguous. Is it temporary scratch? Is it being passed to a different thread but you then wait until the other thread is done before returning? (That's what my crazy code that prompted this did btw.) If it is an error 99% of the time, the error seems reasonable. But even then, what if you are one of the 1% of exceptions? Or with the array, I'd guess it is an error 60% of the time. Maybe still fair to have it an error there... but what if you are in the other 40%? What do you do? How do I tell the compiler "I know this looks wrong at first glance, but trust me"? Again, that's why my subject line for this thread was "and trust". I don't mind this error most the time, we can see from experience that this kind of mistake can be made often enough to take note of it. I just want some way to tell the compiler to trust me. OK, I confess, it'd probably still annoy me, like the annoying integral promotion casts kinda drive me nuts. But I can live with it, since at least there's a way to cast it away. But here, trusted didn't work. cast didn't work. So it just stopped me. I changed the code to use a static TLS instance instead of a stack item and got it compiling again, but I'm also concerned that D is going down an ugly path here. What if my workaround hack now gets plugged later? I want something, akin to `cast`, that is formally defined to function as an escape hatch. (Maybe a double cast through void* might work, there seems to be potential in tricking the compiler with that, but those can be even harder to get right than the original lifetime problem, so we do need to make sure the cure isn't worse than the disease in allowing the exceptions.)
Apr 14
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/14/2020 3:58 PM, Adam D. Ruppe wrote:
 but what if you are in the other 40%? What do you do?
I proposed a direct way you can proceed with your method, it works. I also proposed a more meta safe rewrite that produced smaller and faster code than the system example rejected by the compiler. This makes it hard to discern the value in loosening up that particular restriction.
Apr 14
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 15 April 2020 at 02:19:51 UTC, Walter Bright wrote:
 I proposed a direct way you can proceed with your method, it 
 works.
Is that formally defined and guaranteed to continue working in the future? Or is it just exploiting a temporary hole in the system that could be patched at any time without notice? Maybe document it something like "assumeUnique" with "allowEscape" to give that peace of mind and I'll be happy with it. Note that my original problem was already solved before I wrote this opening message. I'm not asking for a solution to this particular case; I have plenty of those (for now at least, who knows until you arbitrarily break them too). I'm asking for confidence in D's future.
Apr 15
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/15/2020 5:01 AM, Adam D. Ruppe wrote:
 I'm asking for confidence in D's future.
If you've got an issue in the future that you need help with, that's what we're here for. We're not going to make D system useless.
Apr 16
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 14.04.20 22:10, Walter Bright wrote:
 On 4/14/2020 6:34 AM, Adam D. Ruppe wrote:
 Indeed, but isn't that what  safe is for? There should be a way do it 
 intentionally, to tell the compiler "trust me". That's my main point 
 with this thread - D is being overbearing now.

 "Copy the pointer to that array please, D."
So, Adam, how do you feel about:   system int* pumpkin(int i) { return &i); Timon says that should be an error.
That's actually not what I said. I said I would not be opposed to it being an error.
Apr 15
prev sibling next sibling parent Claude <claudemr live.fr> writes:
On Monday, 13 April 2020 at 07:50:43 UTC, Walter Bright wrote:
 Consider:

    system int* pumpkin(int i) { return &i);

 Should that give an error or not?

 I.e. where does one draw the line?
I think it should not return an error. It is system code, the programmer should be able to hack around as he wishes. If he chooses safety, safe is here. For me the line is: system code code should behave like plain old C. D just gives some extra expressiveness. As a system-programmer, I wish to do with D what I can already do with C, faster, more elegantly.
Apr 13
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/13/20 3:50 AM, Walter Bright wrote:
 On 4/11/2020 7:43 PM, Timon Gehr wrote:
 Clearly there is a bug or bad design if the address of a`` escaping in 
 `b ~= &a` and in `b = [&a]` are not treated the same.
They are treated the same with dip1000.
 But like Adam I don't see why there should be such a check in 
  system/ trusted code at all. (I understand that there is a 
 workaround, but that should not be required.)

 Can we please settle on making  safe actually memory safe and 
  system/ trusted actually trust the programmer?
Consider:   system int* pumpkin(int i) { return &i); Should that give an error or not?
Yes. This one directly exposes dangling references, and is easily caught by the compiler.
 
 I.e. where does one draw the line?
 
Where it's possible to write valid code that is memory safe even though it cannot be proven. For instance: void foo() { int*[] b; int a; b ~= &a; // use b but don't expose it outside foo } There are no memory safety violations there. This one is also memory safe: int*[] b; void foo() { int a; b ~= &a; // use b, but don't allow it to be exposed in a way that can result in corruption. b = b[0 .. $-1]; } The first case is highly useful, as one often needs scratch space to perform complex calculations or graph algorithms. I see no reason to disallow it. The second case is more questionable, because it's very easy for someone to keep a copy of b at some point, and then you have lost track of who has access to a's memory. But it still is possible for it to be safe. system is supposed to mean "I know what I'm doing". -Steve
Apr 13
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2020 5:25 AM, Steven Schveighoffer wrote:
 For instance:
 
 void foo()
 {
      int*[] b;
      int a;
      b ~= &a;
          // use b but don't expose it outside foo
 }
 
 There are no memory safety violations there.
 The first case is highly useful, as one often needs scratch space to perform 
 complex calculations or graph algorithms. I see no reason to disallow it.
I see a reason: void foo() { int*[1] b = void; int a; b[0] = &a; } It's faster, too. And if it is written as: &safe void foo() { int a; int*[1] b; b[0] = &a; } it's even safe (with -preview=dip1000). I know, I know, this isn't the real use case. But I've done plenty of "use scratch data structures on the stack" programming for speed and I know how to make it work without needing to store addresses in GC objects.
Apr 14
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/14/20 6:08 AM, Walter Bright wrote:
 On 4/13/2020 5:25 AM, Steven Schveighoffer wrote:
 For instance:

 void foo()
 {
      int*[] b;
      int a;
      b ~= &a;
          // use b but don't expose it outside foo
 }

 There are no memory safety violations there.
 The first case is highly useful, as one often needs scratch space to 
 perform complex calculations or graph algorithms. I see no reason to 
 disallow it.
I see a reason:  void foo()  {     int*[1] b = void;     int a;     b[0] = &a;  }
The toy cases aren't real cases, they just show the error and that the usage is safe. We are not trying to solve the toy. Of course, adding one element to an array and doing nothing is not the real case. I'm talking about maybe an array of millions of pointers, some of which are on the stack (consider like a linked list where you store the head on the stack), and ensuring the array doesn't escape the function.
 
 It's faster, too. And if it is written as:
 
   &safe void foo()
   {
      int a;
      int*[1] b;
      b[0] = &a;
   }
 
 it's even  safe (with -preview=dip1000).
 
 I know, I know, this isn't the real use case. But I've done plenty of 
 "use scratch data structures on the stack" programming for speed and I 
 know how to make it work without needing to store addresses in GC objects.
It's not about knowing how to do it other ways, or doing it the fastest way. It's about knowing how to do it correctly *this* way. Maybe I don't want to deal with having to ensure things are freed properly. What if my graph algorithm has cycles, will RAII (e.g. RefCounted) take care of that? If you do it right with the GC, then there are no memory issues, why is the compiler stopping that in system code? system is supposed to be "I know what I am doing". The answer to "why won't system let me write this memory safe code?" shouldn't be "you're doing it wrong". That's factually incorrect, and the compiler shouldn't bug me about it. Especially when it's trivially circumvented. safe is for compiler checks for safety. That being said, I agree with the simple case of returning a pointer from a stack variable directly from a function being disallowed. That also can be easily worked around, which should probably be required, but is never correct anyway. -Steve
Apr 14
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/14/2020 10:16 AM, Steven Schveighoffer wrote:
 That being said, I agree with the simple case of returning a pointer from a 
 stack variable directly from a function being disallowed. That also can be 
 easily worked around, which should probably be required, but is never correct 
 anyway.
Oh, it can be correct, when one wants to examine the stack pointer value. I've used it for that purpose myself. (Examining the stack pointer is valuable when determining whether other pointers are pointing into the stack or not.) As I replied to Timon, you're drawing a subjective (not objective) line at what is acceptable or not.
Apr 14
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/14/20 4:14 PM, Walter Bright wrote:
 On 4/14/2020 10:16 AM, Steven Schveighoffer wrote:
 That being said, I agree with the simple case of returning a pointer 
 from a stack variable directly from a function being disallowed. That 
 also can be easily worked around, which should probably be required, 
 but is never correct anyway.
Oh, it can be correct, when one wants to examine the stack pointer value. I've used it for that purpose myself. (Examining the stack pointer is valuable when determining whether other pointers are pointing into the stack or not.)
Do you need to call a function to do that? Won't just &somevariable work? Also, this compiles just fine, and gives you the stack pointer value: size_t stackPtr() system { int i; return cast(size_t)&i; }
 
 As I replied to Timon, you're drawing a subjective (not objective) line 
 at what is acceptable or not.
 
Yes, it's subjective. But so is common sense. The objective line is that all should be allowed. I'm fine with that too if that means you can write useful memory safe system code. It's an entirely valid and *consistent* proposal to say " system just allows anything, including obvious memory errors. Use safe if you want compiler restrictions." Or, we could say, for the 0.00000001% of the time you want to get a stack pointer value, use a cast, which means the other 99.9999999% of the cases which are actually memory problems are helpfully caught. You must be able to see that there is a difference in common utility between needing to capture the stack pointer value, and wanting to utilize heap space. It doesn't even make sense that array appending is somehow "the one thing" that causes problems, when you can pass stack pointers to functions which you don't control, slice static arrays, create stack pointers at will and *then* do whatever you want, etc. If there is a subjective story for this "feature" it has quite a few holes in it. It's like putting up a fence post instead of a fence to protect a space. It just ends up being more annoying than functional. I can do this too, apparently since it's a TLS array and not a heap array, totes ok! int*[1] arr; void bar() { int i; arr[0] = &i; } -Steve
Apr 14
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/14/20 4:40 PM, Steven Schveighoffer wrote:
 It's an entirely valid and *consistent* proposal to say " system just 
 allows anything, including obvious memory errors. Use  safe if you want 
 compiler restrictions."
Note that this is especially palatable if safe is the default and you have to opt-in to system. -Steve
Apr 14
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/14/2020 1:40 PM, Steven Schveighoffer wrote:
 Also, this compiles just fine, and gives you the stack pointer value:
 
 size_t stackPtr()  system
 {
    int i;
    return cast(size_t)&i;
 }
Yup. I also gave workarounds to Adam's example.
 I can do this too, apparently since it's a TLS array and not a heap array,
totes 
 ok!
 
 int*[1] arr;
 
 void bar()
 {
      int i;
      arr[0] = &i;
 }
Ironically, I argue for consistency with "safe by default" and you argue for special cases, and here the reverse. But be careful, you may talk me into disallowing these cases, too :-/
Apr 14
next sibling parent reply Max Samukha <maxsamukha gmail.com> writes:
On Wednesday, 15 April 2020 at 06:12:22 UTC, Walter Bright wrote:

 Ironically, I argue for consistency with "safe by default" and 
 you argue for special cases, and here the reverse.

 But be careful, you may talk me into disallowing these cases, 
 too :-/
A good reference point would be C. If it is allowed in C, it should be allowed in system D. If I put system on a D function, i tell the compiler not to bug me with incorrect assumptions about my code's safety.
Apr 15
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/15/2020 1:14 AM, Max Samukha wrote:
 A good reference point would be C. If it is allowed in C, it should be allowed 
 in  system D. If I put  system on a D function, i tell the compiler not to bug 
 me with incorrect assumptions about my code's safety.
What matters is having a way to get things done, not any way.
Apr 15
parent Max Samukha <maxsamukha gmail.com> writes:
On Thursday, 16 April 2020 at 04:49:08 UTC, Walter Bright wrote:
 On 4/15/2020 1:14 AM, Max Samukha wrote:
 A good reference point would be C. If it is allowed in C, it 
 should be allowed in  system D. If I put  system on a D 
 function, i tell the compiler not to bug me with incorrect 
 assumptions about my code's safety.
What matters is having a way to get things done, not any way.
Did I propose "any way"? You said that there were no objective criteria for what checks to perform. I proposed C as a reference, which basically means "unsafe enough to rule out complaints such as Adam's". If you are going to make safe the default, it makes sense to reduce the restrictions on system, not the other way round.
Apr 16
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/15/20 2:12 AM, Walter Bright wrote:
 On 4/14/2020 1:40 PM, Steven Schveighoffer wrote:
 Also, this compiles just fine, and gives you the stack pointer value:

 size_t stackPtr()  system
 {
    int i;
    return cast(size_t)&i;
 }
Yup. I also gave workarounds to Adam's example.
Of course. There are workarounds to all these issues. The question then becomes, is the inconsistency for this particular restriction worth the annoyance it causes to work around it. The answer is subjective, as always. IMO, fetching the stack pointer is something that just isn't ever a requirement for code. It's such a specialized need that the annoyance it causes is worth all the memory problems it prevents when people accidentally return stack references. Maybe that's just my code, maybe everyone else needs to fetch the stack pointer in their code. But I didn't think that was the case. On the other hand, using heap data is pretty common in my code, and occasionally I might stick a reference to stack data in there. If I am doing it responsibly, there is no harm. The compiler might make me jump through some annoyance hoops, but it's possible to do. So we have to draw a line, an arbitrary line, that says "everything like this is going to require some workaround, everything else, well it's system code, you got what you signed up for". My line is that returning stack pointers, if provable, is more often than not an error, and I would like to be reminded of it from the compiler. Putting stack data into heap blocks, is already not consistently flagged, so I don't think it's saving much to have this annoyance available. But if the compiler is going to be annoying, be annoying according to a plan, not just random annoyance.
 I can do this too, apparently since it's a TLS array and not a heap 
 array, totes ok!

 int*[1] arr;

 void bar()
 {
      int i;
      arr[0] = &i;
 }
Ironically, I argue for consistency with "safe by default" and you argue for special cases, and here the reverse.
There are different levels of consistency. If you have a reasonable rule, then be consistent in that rule. It might make you have to do some extra workarounds to use the language, but the result is that you should be a better programmer, and you have a rationale to point at for why things are the way they are. I'll take for example the rule that if(cond); is disallowed. It's a wonderful rule, but totally not consistent with the language (I can write empty statements in most other places). BUT it's such a common mistake that it's worth the annoyance. It's caught bugs for me many many times. However, what if while(cond); was allowed? That kind of inconsistency doesn't make sense. It's inconsistent to disallow empty statements for an if block, but allow them for while blocks. Arguing that people would want consistency with while statements but not if statements is hard to justify. In the safe debate, the problem is simply that code that compiles today and is system will compile tomorrow and be safe, without ANY checking or any deprecation. This destroys everything that safe has been built upon. Ironically, what I'm asking for is consistency with existing code, and you are arguing to make an exception that extern(C) system functions compile should silently convert to safe functions without warning. We just cannot change the meaning of all code without warning people or causing an error. If we didn't have existing code that's extern(C) then the debate would be entirely different. It would be a new thing, and it would be a legitimate position to have extern(C) prototypes safe by default. I'll note that I proposed an alteration that has received zero responses that I think would solve all those concerns: https://forum.dlang.org/post/r6kvm4$1vq5$1 digitalmars.com
 
 But be careful, you may talk me into disallowing these cases, too :-/
 
If you did, it would at LEAST be reasonable! At least the rule would be consistent. So I would be fine with it! As Adam says, there is no rhyme or reason for this one rule vs. the allowances for other code, so there's no predictability as to when D might somehow decide what you did was not correct. Ruling by whim is not as understandable as providing clear and consistent rules to base changes on. -Steve
Apr 15
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.20 17:11, Steven Schveighoffer wrote:
 
 If we didn't have existing code that's extern(C) then the debate would 
 be entirely different. It would be a new thing, and it would be a 
 legitimate position to have extern(C) prototypes safe by default.
No.
Apr 15
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.20 17:11, Steven Schveighoffer wrote:
 
 I'll note that I proposed an alteration that has received zero responses 
 that I think would solve all those concerns:
 
 https://forum.dlang.org/post/r6kvm4$1vq5$1 digitalmars.com
I had answered to that with the example of safe functions with inconsistent return values. I think you'd need to do some more mangling to make this acceptable.
Apr 15
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.20 08:12, Walter Bright wrote:

 
 Ironically, I argue for consistency with "safe by default"
You really don't. Allowing memory corruption to be annotated safe is inconsistent with the meaning of safe.
Apr 15
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 14.04.20 22:14, Walter Bright wrote:
 On 4/14/2020 10:16 AM, Steven Schveighoffer wrote:
 That being said, I agree with the simple case of returning a pointer 
 from a stack variable directly from a function being disallowed. That 
 also can be easily worked around, which should probably be required, 
 but is never correct anyway.
Oh, it can be correct, when one wants to examine the stack pointer value. I've used it for that purpose myself. (Examining the stack pointer is valuable when determining whether other pointers are pointing into the stack or not.) As I replied to Timon, you're drawing a subjective (not objective) line at what is acceptable or not.
It depends on the language being consistent. If the language says what you are doing is guaranteed to lead to UB, a diagnostic is fine. If the spec says this is a supported use case that is guaranteed to work across all D compilers, the story is a bit different. The spec is not formal enough to figure out which of those is the case. Does the spec really mandate an implementation to put the program stack at a continuous range of addresses? Are dangling pointers guaranteed to compare correctly to valid pointers? etc.
Apr 15
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Saturday, 11 April 2020 at 01:21:56 UTC, Adam D. Ruppe wrote:
 One of the nice things about D is that I very rarely feel like 
 I am fighting the language (unless I'm working under 
 constraints like -w and  safe pure nothrow  nogc stuff, but 
 that's why I just don't use those!). From low-level bit 
 twiddling to high level "just make it work", the D language 
 usually works with me, a few exceptions excluded - but when 
 those happen, you can cast or whatever to tell it to trust me.
Walter said there's now a third kind of safety for not annotated functions or something like that.
Apr 10
prev sibling parent mipri <mipri minimaltype.com> writes:
On Saturday, 11 April 2020 at 01:21:56 UTC, Adam D. Ruppe wrote:
 ```
 void main() {
         int a;
         b ~= &a;
 }

 int*[] b;
 ```

 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a


 (Interestingly, `b = [&a]` instead of ~= passes muster. What's 
 the difference? Just another bug in this?)


 But the inconsistency isn't why I'm posting right now, it is my 
 fear that D is losing faith in me. There seems to be no way to 
 say "trust me" to the compiler. Note that I'm not even using 
 any switches to dmd, and adding  trusted had no effect.
Meanwhile, in Rust: static mut b: Vec<*mut i32> = Vec::new(); fn bad() { let mut a = 0; unsafe { b.push(&mut a as *mut i32) }; } fn main() { bad(); println!("{:?}", unsafe { &b }); } which outputs (for a particular run): [0x7ffc6e4b23b8] So you have to say "trust me" a couple of times (or more than twice, if you count the muts and the cast), but you *can* say that. You can faithfully implement this bug in Rust. Part of why you can do that, is that this code is using pointers. The same code with only references fails despite the unsafe{} blocks: 5 | unsafe { b.push(&mut a) }; | -------^^^^^^- | | | | | borrowed value does not live long enough | argument requires that `a` is borrowed for `'static` 6 | } | - `a` dropped here while still borrowed Pointers in Rust lack safety and liveness guarantees. That stuff is for references. D has references too... | Rust | D | |----------+---------------| | &i32 | const ref int | | &mut i32 | ref int | | *i32 | const(int)* | | *mut i32 | int* | ... but not really. You can't declare them ("only parameters or foreach declarations can be ref"), and you can't take a reference of a variable, and you can't cast a pointer to a ref. So it's not enough to say "Rust has normal vs. unsafe{} code, and D has safe vs. normal code, and the difference is only a matter of defaults." It's also the case that Rust has unsafe pointers and safe references, but D only has pointers, so any memory protection that D gets must fall on its pointers. With live the protection only applies to pointers within safe code, and with safe-as-default it might be even easier to do things that way, or even to remove protections that formerly applied to system code. So the trend is not necessarily towards it one day being more appropriate to say that Rust has pointers and references, but D only has references which it calls pointers.
Apr 14