www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [OffTopic] A vulnerability postmortem on Network Security Services

reply Paulo Pinto <pjmlp progtools.org> writes:
Google's Project Zero goes through a memory corruption exploit on 
Network Security Services, where despite all static analysers, 
fuzzers and code reviews, it flew under the radar.

https://googleprojectzero.blogspot.com/2021/12/this-shouldnt-have-happened.html

Hence why  safe matters.
Dec 02 2021
next sibling parent reply bauss <jj_1337 live.dk> writes:
On Thursday, 2 December 2021 at 08:09:18 UTC, Paulo Pinto wrote:

 Google's Project Zero goes through a memory corruption exploit 
 on Network Security Services, where despite all static 
 analysers, fuzzers and code reviews, it flew under the radar.

 https://googleprojectzero.blogspot.com/2021/12/this-shouldnt-have-happened.html

 Hence why  safe matters.


Seems like a bounds-checking error and thus would be caught 
without  safe even.
Dec 02 2021
parent user1234 <user1234 12.de> writes:
On Thursday, 2 December 2021 at 10:58:10 UTC, bauss wrote:

 On Thursday, 2 December 2021 at 08:09:18 UTC, Paulo Pinto wrote:

 Google's Project Zero goes through a memory corruption exploit 
 on Network Security Services, where despite all static 
 analysers, fuzzers and code reviews, it flew under the radar.

 https://googleprojectzero.blogspot.com/2021/12/this-shouldnt-have-happened.html

 Hence why  safe matters.


 Seems like a bounds-checking error and thus would be caught 
 without  safe even.


I was thinking about stronger typing of the union members + 
better encapsulation and better way to init the struct (better 
than the memcpy).
Dec 02 2021
prev sibling next sibling parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Thursday, 2 December 2021 at 08:09:18 UTC, Paulo Pinto wrote:

 Google's Project Zero goes through a memory corruption exploit 
 on Network Security Services, where despite all static 
 analysers, fuzzers and code reviews, it flew under the radar.

 https://googleprojectzero.blogspot.com/2021/12/this-shouldnt-have-happened.html

 Hence why  safe matters.


Bottom line:
Use D instead of C 😎
Dec 02 2021
parent reply Johan <j j.nl> writes:
On Thursday, 2 December 2021 at 11:01:07 UTC, Imperatorn wrote:

 On Thursday, 2 December 2021 at 08:09:18 UTC, Paulo Pinto wrote:

 Google's Project Zero goes through a memory corruption exploit 
 on Network Security Services, where despite all static 
 analysers, fuzzers and code reviews, it flew under the radar.

 https://googleprojectzero.blogspot.com/2021/12/this-shouldnt-have-happened.html

 Hence why  safe matters.


 Bottom line:
 Use D instead of C 😎


Sorry to rain on the party here, but D is of course not at all 
immune to this problem.
It was not hard to find out-of-bounds memory access in the D 
compiler, using the fuzz techniques mentioned in the article.

https://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html
https://github.com/dlang/dmd/pull/7050

Note the discussion of bounds checking in the PR...

-Johan
Dec 02 2021
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
On Thursday, 2 December 2021 at 12:15:38 UTC, Johan wrote:

 On Thursday, 2 December 2021 at 11:01:07 UTC, Imperatorn wrote:

 On Thursday, 2 December 2021 at 08:09:18 UTC, Paulo Pinto 
 wrote:

 [...]


 Bottom line:
 Use D instead of C 😎


 Sorry to rain on the party here, but D is of course not at all 
 immune to this problem.
 It was not hard to find out-of-bounds memory access in the D 
 compiler, using the fuzz techniques mentioned in the article.

 https://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html
 https://github.com/dlang/dmd/pull/7050

 Note the discussion of bounds checking in the PR...

 -Johan


Oh well
Dec 02 2021
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 12/2/21 7:15 AM, Johan wrote:

 On Thursday, 2 December 2021 at 11:01:07 UTC, Imperatorn wrote:

 On Thursday, 2 December 2021 at 08:09:18 UTC, Paulo Pinto wrote:

 Google's Project Zero goes through a memory corruption exploit on 
 Network Security Services, where despite all static analysers, 
 fuzzers and code reviews, it flew under the radar.

 https://googleprojectzero.blogspot.com/2021/12/this-shouldn
-have-happened.html 


 Hence why  safe matters.


 Bottom line:
 Use D instead of C 😎


 
 Sorry to rain on the party here, but D is of course not at all immune to 
 this problem.
 It was not hard to find out-of-bounds memory access in the D compiler, 
 using the fuzz techniques mentioned in the article.


The D compiler is not  safe.

Using D isn't enough, you need to use  safe D.

Even that isn't enough, because most of the time people misuse  trusted 
(because it's so easy to misuse).

-Steve
Dec 02 2021
parent reply Tejas <notrealemail gmail.com> writes:
On Thursday, 2 December 2021 at 13:19:09 UTC, Steven 
Schveighoffer wrote:

 On 12/2/21 7:15 AM, Johan wrote:

 On Thursday, 2 December 2021 at 11:01:07 UTC, Imperatorn wrote:

 On Thursday, 2 December 2021 at 08:09:18 UTC, Paulo Pinto 
 wrote:

 [...]


 Bottom line:
 Use D instead of C 😎


 
 Sorry to rain on the party here, but D is of course not at all 
 immune to this problem.
 It was not hard to find out-of-bounds memory access in the D 
 compiler, using the fuzz techniques mentioned in the article.


 The D compiler is not  safe.

 Using D isn't enough, you need to use  safe D.

 Even that isn't enough, because most of the time people misuse 
  trusted (because it's so easy to misuse).

 -Steve


Wish the ` safe` by default DIP had passed :(
Any hope of reviving it and merging into master??
Dec 02 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Thursday, 2 December 2021 at 16:44:42 UTC, Tejas wrote:

 Wish the ` safe` by default DIP had passed :(
 Any hope of reviving it and merging into master??


Only if someone can (a) come up with a better solution for 
handling `extern(C)` functions, and (b) convince Walter to accept 
it.

I think a more likely path forward is to allow the programmer to 
specify default attributes in such a way that they can still be 
overriden by inference. For example:

```d
// Make  safe the default for the rest of this scope
default( safe):

// defaults to  safe
int foo(int n) { return n; }

// error: can't cast integer to pointer in a  safe function
int* bar(int n) { return cast(int*) n; }

// ok: inferred as  system (because of `auto`)
auto baz(int n) { return cast(int*) n; }
```

This is the same idea as [Adam D. Ruppe's attribute proposal][1], 
but with new syntax to avoid potential breakage to existing code.

As Adam explains in his article, we cannot do this by simply 
applying the ` safe` attribute globally, because doing so will 
*override* the compiler's attribute inference, and cause 
compilation of functions like `baz` to fail. (Example: 
https://run.dlang.io/is/s9iuKq)

[1]: 
https://dpldocs.info/this-week-in-d/Blog.Posted_2020_01_13.html
Dec 02 2021
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 2 December 2021 at 17:11:09 UTC, Paul Backus wrote:

 On Thursday, 2 December 2021 at 16:44:42 UTC, Tejas wrote:

 Wish the ` safe` by default DIP had passed :(
 Any hope of reviving it and merging into master??


 Only if someone can (a) come up with a better solution for 
 handling `extern(C)` functions, and (b) convince Walter to 
 accept it.

 I think a more likely path forward is to allow the programmer 
 to specify default attributes in such a way that they can still 
 be overriden by inference. For example:
 [snip]


+1

Will help prevent any breaking changes if if they ever do decide 
to go to  safe by default.
Dec 02 2021
prev sibling next sibling parent reply Tejas <notrealemail gmail.com> writes:
On Thursday, 2 December 2021 at 17:11:09 UTC, Paul Backus wrote:

 On Thursday, 2 December 2021 at 16:44:42 UTC, Tejas wrote:

 Wish the ` safe` by default DIP had passed :(
 Any hope of reviving it and merging into master??


 Only if someone can (a) come up with a better solution for 
 handling `extern(C)` functions, and (b) convince Walter to 
 accept it.

 I think a more likely path forward is to allow the programmer 
 to specify default attributes in such a way that they can still 
 be overriden by inference. For example:

 ```d
 // Make  safe the default for the rest of this scope
 default( safe):

 // defaults to  safe
 int foo(int n) { return n; }

 // error: can't cast integer to pointer in a  safe function
 int* bar(int n) { return cast(int*) n; }

 // ok: inferred as  system (because of `auto`)
 auto baz(int n) { return cast(int*) n; }
 ```

 This is the same idea as [Adam D. Ruppe's attribute 
 proposal][1], but with new syntax to avoid potential breakage 
 to existing code.

 As Adam explains in his article, we cannot do this by simply 
 applying the ` safe` attribute globally, because doing so will 
 *override* the compiler's attribute inference, and cause 
 compilation of functions like `baz` to fail. (Example: 
 https://run.dlang.io/is/s9iuKq)

 [1]: 
 https://dpldocs.info/this-week-in-d/Blog.Posted_2020_01_13.html


Hmm... not a fan of that solution

Still feel marking `extern (C)` stuff as ` trusted` is better.

Introducing a new feature for such a fundamental, yet obvious 
thing seems overkill, IMHO. Forcing not ` safe` stuff to be 
annotated seems better to me.

Maybe we should relent and just let Walter mark the `extern` 
stuff as ` safe` even though it's not... ideal??

Or disallow marking any `extern` function as ` safe`, only 
` trusted`? That's again introducing unnecessary complexity 
though :(

Gah, I say let him have his way, it's not worth adding a new 
feature/syntax for, but not something we should just let sit on 
ice either, else "it will break old code" will rear it's ugly 
head again
Dec 02 2021
next sibling parent reply bachmeier <no spam.net> writes:
On Thursday, 2 December 2021 at 17:21:58 UTC, Tejas wrote:

 On Thursday, 2 December 2021 at 17:11:09 UTC, Paul Backus wrote:

 On Thursday, 2 December 2021 at 16:44:42 UTC, Tejas wrote:

 Wish the ` safe` by default DIP had passed :(
 Any hope of reviving it and merging into master??


 Only if someone can (a) come up with a better solution for 
 handling `extern(C)` functions, and (b) convince Walter to 
 accept it.

 I think a more likely path forward is to allow the programmer 
 to specify default attributes in such a way that they can 
 still be overriden by inference. For example:

 ```d
 // Make  safe the default for the rest of this scope
 default( safe):

 // defaults to  safe
 int foo(int n) { return n; }

 // error: can't cast integer to pointer in a  safe function
 int* bar(int n) { return cast(int*) n; }

 // ok: inferred as  system (because of `auto`)
 auto baz(int n) { return cast(int*) n; }
 ```

 This is the same idea as [Adam D. Ruppe's attribute 
 proposal][1], but with new syntax to avoid potential breakage 
 to existing code.

 As Adam explains in his article, we cannot do this by simply 
 applying the ` safe` attribute globally, because doing so will 
 *override* the compiler's attribute inference, and cause 
 compilation of functions like `baz` to fail. (Example: 
 https://run.dlang.io/is/s9iuKq)

 [1]: 
 https://dpldocs.info/this-week-in-d/Blog.Posted_2020_01_13.html


 Hmm... not a fan of that solution

 Still feel marking `extern (C)` stuff as ` trusted` is better.

 Introducing a new feature for such a fundamental, yet obvious 
 thing seems overkill, IMHO. Forcing not ` safe` stuff to be 
 annotated seems better to me.


Sure, if you don't use extern(C) much and you don't care about 
everyone that uses it extensively in their code moving to other 
languages, go ahead and do that. It's not realistic to break that 
much code written in a language that was designed from the start 
to be closely tied to C. It wouldn't be hard to add a -safe 
compilation flag, but apparently nobody's into that, they'd 
prefer to break code others have written.
Dec 02 2021
parent reply Tejas <notrealemail gmail.com> writes:
On Thursday, 2 December 2021 at 19:35:25 UTC, bachmeier wrote:

 On Thursday, 2 December 2021 at 17:21:58 UTC, Tejas wrote:

 On Thursday, 2 December 2021 at 17:11:09 UTC, Paul Backus 
 wrote:

 [...]


 Hmm... not a fan of that solution

 Still feel marking `extern (C)` stuff as ` trusted` is better.

 Introducing a new feature for such a fundamental, yet obvious 
 thing seems overkill, IMHO. Forcing not ` safe` stuff to be 
 annotated seems better to me.


 Sure, if you don't use extern(C) much and you don't care about 
 everyone that uses it extensively in their code moving to other 
 languages, go ahead and do that. It's not realistic to break 
 that much code written in a language that was designed from the 
 start to be closely tied to C. It wouldn't be hard to add a 
 -safe compilation flag, but apparently nobody's into that, 
 they'd prefer to break code others have written.


I think the problem with adding new compiler flags is that it 
doubles the amount of configs that have to be checked/tested for 
from then on-wards (at least that's what I was told when I asked 
if it was possible to do away with the `short/byte` implicit 
conversion to `int` via a compiler flag).

Rather go with Adam's `default(safe)` than a new compiler flag, 
if that's what it'll come down to.
Dec 02 2021
parent reply bachmeier <no spam.net> writes:
On Friday, 3 December 2021 at 06:57:49 UTC, Tejas wrote:

 On Thursday, 2 December 2021 at 19:35:25 UTC, bachmeier wrote:

 On Thursday, 2 December 2021 at 17:21:58 UTC, Tejas wrote:

 On Thursday, 2 December 2021 at 17:11:09 UTC, Paul Backus 
 wrote:

 [...]


 Hmm... not a fan of that solution

 Still feel marking `extern (C)` stuff as ` trusted` is better.

 Introducing a new feature for such a fundamental, yet obvious 
 thing seems overkill, IMHO. Forcing not ` safe` stuff to be 
 annotated seems better to me.


 Sure, if you don't use extern(C) much and you don't care about 
 everyone that uses it extensively in their code moving to 
 other languages, go ahead and do that. It's not realistic to 
 break that much code written in a language that was designed 
 from the start to be closely tied to C. It wouldn't be hard to 
 add a -safe compilation flag, but apparently nobody's into 
 that, they'd prefer to break code others have written.


 I think the problem with adding new compiler flags is that it 
 doubles the amount of configs that have to be checked/tested 
 for from then on-wards (at least that's what I was told when I 
 asked if it was possible to do away with the `short/byte` 
 implicit conversion to `int` via a compiler flag).


I understand. It was during this conversation that I realized D 
has no strategy that will allow it to evolve (and no strategy to 
develop such a strategy). The cost of making an extreme change 
like safe by default is that you have to accept a compiler flag 
or some other compromise. That seems to be off the table, which 
makes it hard to see D being much different in 2041 than it is 
now.
Dec 03 2021
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 04/12/2021 3:03 PM, bachmeier wrote:

 I understand. It was during this conversation that I realized D has no 
 strategy that will allow it to evolve (and no strategy to develop such a 
 strategy). The cost of making an extreme change like safe by default is 
 that you have to accept a compiler flag or some other compromise. That 
 seems to be off the table, which makes it hard to see D being much 
 different in 2041 than it is now.


In this particular case, I say break my code.

Quite literally one small change, no function body?  system and we would 
have supported the DIP.

Nobody has stepped up to make an amended DIP however.
Dec 03 2021
next sibling parent reply Greg Strong <mageofmaple protonmail.com> writes:
On Saturday, 4 December 2021 at 03:23:07 UTC, rikki cattermole 
wrote:

 Quite literally one small change, no function body?  system and 
 we would have supported the DIP.

 Nobody has stepped up to make an amended DIP however.


I do not entirely understand this reply.  It seems that you are 
saying that this would have happened - with support from Walter - 
had a little more paperwork been done.  Can you confirm?

I don't really know how to submit a DIP but I will figure it out 
if that is really all that is needed.  But I'm skeptical ... I've 
read these forms long enough to know that people generally don't 
put in the effort to work through the red tape because, 
frequently, that is done and the proposal winds up being killed 
anyway.

I'm willing to help out, provided either (A) the proposal cannot 
be unilaterally killed by Walter, or (B) he personally confirms 
that he is on board.
Dec 05 2021
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 06/12/2021 7:09 AM, Greg Strong wrote:

 On Saturday, 4 December 2021 at 03:23:07 UTC, rikki cattermole wrote:

 Quite literally one small change, no function body?  system and we 
 would have supported the DIP.

 Nobody has stepped up to make an amended DIP however.


 
 I do not entirely understand this reply.  It seems that you are saying 
 that this would have happened - with support from Walter - had a little 
 more paperwork been done.  Can you confirm?


The problem was Walter didn't listen to the community. We said pretty 
much from day one, if you can't prove that a function is  safe, it 
simply cannot default to it. As that breaks the guarantee (there is some 
conflating of ABI with safety-ness although if a function is extern(C) 
it shouldn't effect it).

It went through the DIP process without this change.

The accepted thread: 
https://forum.dlang.org/thread/rwjxbgsauknjjrvousti forum.dlang.org
Thread where he announced it was dead: 
https://forum.dlang.org/thread/raq4fg$1ab4$1 digitalmars.com


 I don't really know how to submit a DIP but I will figure it out if that 
 is really all that is needed.  But I'm skeptical ... I've read these 
 forms long enough to know that people generally don't put in the effort 
 to work through the red tape because, frequently, that is done and the 
 proposal winds up being killed anyway.
 
 I'm willing to help out, provided either (A) the proposal cannot be 
 unilaterally killed by Walter, or (B) he personally confirms that he is 
 on board.


This is in a pretty awkward position. Community is ok if it has this one 
change, but we don't know what Walter's position is on an amended DIP 
would be. There are some sour feelings here I think still.
Dec 05 2021
parent reply Greg Strong <mageofmaple protonmail.com> writes:
On Sunday, 5 December 2021 at 18:24:22 UTC, rikki cattermole 
wrote:

 This is in a pretty awkward position. Community is ok if it has 
 this one change, but we don't know what Walter's position is on 
 an amended DIP would be. There are some sour feelings here I 
 think still.


Thanks, Rikki, for your quick and frank reply.  This is what I 
feared.  No one wants to put in significant effort, 
understandably, when there is a good chance it will be 
unilaterally killed even if the entire rest of the community is 
in favor.

D has a real problem.  On the one hand, we have problems moving 
forward.  No need to reiterate those.  But, on the other hand, 
there is no long-term-support (LTS) release, nor so far as I 
know, any plans for an LTS release, so you can't rely on the 
current state either.  This makes D only useful for hobby 
programming.

I know there are plans for a new model of governance.  I hope 
that helps.  But I have so far seen nothing on these forums that 
gives me confidence.

 Mike Parker, here's a clear opportunity for the new governance 
model to sink-or-swim.  It seems the entire community, except for 
Walter, wants this but it has been unilaterally killed.  Can we 
move forward?  If you need me to do the paperwork, I will.  If 
this still cannot go forward under the new model, then the new 
model must be considered a failure and it's time to move on.
Dec 05 2021
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 06/12/2021 7:45 AM, Greg Strong wrote:

 Thanks, Rikki, for your quick and frank reply.  This is what I feared.  
 No one wants to put in significant effort, understandably, when there is 
 a good chance it will be unilaterally killed even if the entire rest of 
 the community is in favor.


Nah this particular change would probably take all of 2 minutes.

I've written a DIP and had it go through the queue myself.

In this particular case its just a matter of waiting while peoples 
emotions calm down some more and hear from Walter that we can proceed.
Dec 05 2021
prev sibling next sibling parent reply SealabJaster <sealabjaster gmail.com> writes:
On Sunday, 5 December 2021 at 18:09:56 UTC, Greg Strong wrote:

 I'm willing to help out, provided either (A) the proposal 
 cannot be unilaterally killed by Walter, or (B) he personally 
 confirms that he is on board.


B is the most realistic option, people are rarely willing to give 
up ultimate power for both good and bad intentions.

No matter how I think about, extern(C) being  safe by default 
makes no sense. Even if it was written in a different language, 
you have no information about *which* language and what safety 
guarantees it (doesn't) provide. I just wish it wasn't the main 
point of contention for  safe by default :(

Honestly I'd go as far as saying no C code should be marked as 
even  trusted by a human (even wrapper funcs). But that 
unfortunately turns into a "D with C" ( safe +  "trusted") vs "D 
with no C"( safe only) kind of argument.

And there's no where near enough libraries in D to make up for 
not being able to interface with C.

While it's really unfortunate we haven't yet gotten  safe by 
default, I'm still weary about it when code is still interfacing 
with possible non- safe languages via extern(C).
Dec 05 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 5 December 2021 at 19:57:18 UTC, SealabJaster wrote:

 Honestly I'd go as far as saying no C code should be marked as 
 even  trusted by a human (even wrapper funcs). But that 
 unfortunately turns into a "D with C" ( safe +  "trusted") vs 
 "D with no C"( safe only) kind of argument.


This is a step too far, I think. There are several functions 
which are guaranteed by the C standard to never invoke undefined 
behavior (e.g., getchar, rand, everything in <math.h>). Allowing 
functions like these to be marked as  trusted is completely 
legitimate.
Dec 05 2021
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 5 December 2021 at 21:53:36 UTC, Paul Backus wrote:

There are several functions
 which are guaranteed by the C standard to never invoke 
 undefined behavior (e.g., getchar, rand, everything in 
 <math.h>). Allowing functions like these to be marked as 
  trusted is completely legitimate.


Most standalone functions and even syscalls can be made safe with 
a thin wrapper. The real challenge is a framework that presumes 
manual memory management. Difficult to deal with, maybe importC 
can enable some static analysis?
Dec 05 2021
prev sibling parent SealabJaster <sealabjaster gmail.com> writes:
On Sunday, 5 December 2021 at 21:53:36 UTC, Paul Backus wrote:

 ...


I should've mentioned that I *do* expect exceptions to that 
statement, especially in regards to things where interfacing to C 
is near enough mandatory (WinAPI, sockets, etc.)

But as the other post said, it's a really hard issue to make 
something fully  safe if it also includes manual memory 
management - a route D could be heading in regards to allocators.
Dec 05 2021
prev sibling parent IGotD- <nise nise.com> writes:
On Sunday, 5 December 2021 at 18:09:56 UTC, Greg Strong wrote:

 I'm willing to help out, provided either (A) the proposal 
 cannot be unilaterally killed by Walter, or (B) he personally 
 confirms that he is on board.


Option C would provide you with that and that would be that D is 
forked and that project is managed better. Say hello to D++, or 
whatever better name you can come up with.
Dec 06 2021
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 12/3/21 10:23 PM, rikki cattermole wrote:

 
 On 04/12/2021 3:03 PM, bachmeier wrote:

 I understand. It was during this conversation that I realized D has no 
 strategy that will allow it to evolve (and no strategy to develop such 
 a strategy). The cost of making an extreme change like safe by default 
 is that you have to accept a compiler flag or some other compromise. 
 That seems to be off the table, which makes it hard to see D being 
 much different in 2041 than it is now.


 
 In this particular case, I say break my code.
 
 Quite literally one small change, no function body?  system and we would 
 have supported the DIP.


I proposed [during the 
discussion](https://forum.dlang.org/post/r6kvm4$1vq5$1 digitalmars.com) 
that you can assume `extern(C)` functions are ` safe` as long as they 
are mangled differently. I think that would both solve the problems 
people had, and allow `extern(C)` to be safe by default.

Walter seemingly ignored it, but silently [hated 
it](https://forum.dlang.org/post/ra7958$2r8v$1 digitalmars.com), so I 
don't see that happening.

-Steve
Dec 08 2021
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 8 December 2021 at 15:46:31 UTC, Steven 
Schveighoffer wrote:

 [snip]

 I proposed [during the 
 discussion](https://forum.dlang.org/post/r6kvm4$1vq5$1 digitalmars.com) that
you can assume `extern(C)` functions are ` safe` as long as they are mangled
differently. I think that would both solve the problems people had, and allow
`extern(C)` to be safe by default.

 Walter seemingly ignored it, but silently [hated 
 it](https://forum.dlang.org/post/ra7958$2r8v$1 digitalmars.com), so I don't
see that happening.

 -Steve


I think Walter (and likely Atila too) might come down on avoiding 
special casing as an important design goal.

To me, the best way to ensure that is to moving  safe from a 
black list to an white list. That is,  safe is currently 
structured so that you cannot do certain things. If instead it is 
structured so that only certain things are allowed, then 
something like having the body of the function available could be 
on that list. There would be no more special casing and it would 
only break code to the extent that the white list and black list 
do not match up (which might be considerable, who knows)
Dec 08 2021
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Dec 08, 2021 at 04:47:20PM +0000, jmh530 via Digitalmars-d wrote:

 On Wednesday, 8 December 2021 at 15:46:31 UTC, Steven Schveighoffer wrote:


[...]

 I proposed [during the
 discussion](https://forum.dlang.org/post/r6kvm4$1vq5$1 digitalmars.com)
 that you can assume `extern(C)` functions are ` safe` as long as
 they are mangled differently. I think that would both solve the
 problems people had, and allow `extern(C)` to be safe by default.
 
 Walter seemingly ignored it, but silently [hated
 it](https://forum.dlang.org/post/ra7958$2r8v$1 digitalmars.com), so
 I don't see that happening.




[...]

 I think Walter (and likely Atila too) might come down on avoiding
 special casing as an important design goal.
 
 To me, the best way to ensure that is to moving  safe from a black
 list to an white list. That is,  safe is currently structured so that
 you cannot do certain things. If instead it is structured so that only
 certain things are allowed, then something like having the body of the
 function available could be on that list. There would be no more
 special casing and it would only break code to the extent that the
 white list and black list do not match up (which might be
 considerable, who knows)


Whether  safe is as a blacklist vs a whitelist is IMO an implementation
detail.  I think what Walter is concerned about is more on the higher
level language perspective: special cases tend to cause unexpected
interactions with other language features, and such interactions tend to
percolate throughout the language, adding complexity.

I'm somewhat sympathetic to this view, but I think in this case it has
unfortunately devolved into the perfect being the enemy of the good. For
pragmatic reasons I would have allowed this, had the decision been up to
me.


T

-- 
I am a consultant. My job is to make your job redundant. -- Mr Tom
Dec 08 2021
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08.12.21 18:26, H. S. Teoh wrote:

 Whether  safe is as a blacklist vs a whitelist is IMO an implementation
 detail.  I think what Walter is concerned about is more on the higher
 level language perspective: special cases tend to cause unexpected
 interactions with other language features, and such interactions tend to
 percolate throughout the language, adding complexity.


It's not a "special case", it's keeping the core promise of  safe...

Framing this as a special case is just not productive. At that point you 
could just as well reason that pointer arithmetic should be allowed in 
 safe code. After all, banning it introduces a weird special case that 
is inconsistent with how other D code works.

This is a pretty obvious safety hole:

```d
void main() safe{
     struct S{
         pragma(mangle,"foo"):
         static extern(C) void foo(int){
             import core.stdc.stdlib;
             free(cast(void*)0xDEADBEEF);
         }
         static extern(C) void foo() safe;
     }
     S.foo();
}
```

There is no  trusted code in that snippet and it frees an invented 
pointer. "Memory safety guarantees". The code is not at fault. It's the 
language. I am all for pragmatism, but it seems awfully out of touch in 
this case.
Dec 08 2021
parent Nick Treleaven <nick geany.org> writes:
On Wednesday, 8 December 2021 at 18:14:36 UTC, Timon Gehr wrote:

 This is a pretty obvious safety hole:

 ```d
 void main() safe{
     struct S{
         pragma(mangle,"foo"):
         static extern(C) void foo(int){
             import core.stdc.stdlib;
             free(cast(void*)0xDEADBEEF);
         }
         static extern(C) void foo() safe;
     }
     S.foo();
 }
 ```

 There is no  trusted code in that snippet and it frees an 
 invented pointer.


It seems pragma(mangle) should require that the function it 
applies to is not  safe. But even without pragma(mangle), foo can 
be made a free function with the same problem.

Forbidding  safe on a function prototype would fix that, 
requiring  trusted instead. Enforcing that is necessary anyway 
for when the function body is not available to be mechanically 
checked for memory safety.
Dec 10 2021
prev sibling parent bauss <jj_1337 live.dk> writes:
On Thursday, 2 December 2021 at 17:21:58 UTC, Tejas wrote:

 Forcing not ` safe` stuff to be annotated seems better to me.


Definitely is better. It's how it is done in literally every 
other langauge.

D is the only one I know that does it opposite, where being 
unsafe is default.



The correct solution in my head for the extern functions is 
simply being able to mark them trusted.
Dec 02 2021
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 02, 2021 at 05:11:09PM +0000, Paul Backus via Digitalmars-d wrote:

 On Thursday, 2 December 2021 at 16:44:42 UTC, Tejas wrote:

 
 Wish the ` safe` by default DIP had passed :(
 Any hope of reviving it and merging into master??


 
 Only if someone can (a) come up with a better solution for handling
 `extern(C)` functions, and (b) convince Walter to accept it.


The solution is very simple: extern(C), extern(C++), and extern anything
except extern(D) really, should be  system, and extern(D) (the default)
should be  safe.

The real issue is convincing Walter to accept this.


T

-- 
Elegant or ugly code as well as fine or rude sentences have something in
common: they don't depend on the language. -- Luca De Vitis
Dec 02 2021
next sibling parent reply Tejas <notrealemail gmail.com> writes:
On Thursday, 2 December 2021 at 17:36:18 UTC, H. S. Teoh wrote:

 On Thu, Dec 02, 2021 at 05:11:09PM +0000, Paul Backus via 
 Digitalmars-d wrote:

 On Thursday, 2 December 2021 at 16:44:42 UTC, Tejas wrote:

 
 Wish the ` safe` by default DIP had passed :(
 Any hope of reviving it and merging into master??


 
 Only if someone can (a) come up with a better solution for 
 handling
 `extern(C)` functions, and (b) convince Walter to accept it.


 The solution is very simple: extern(C), extern(C++), and extern 
 anything
 except extern(D) really, should be  system, and extern(D) (the 
 default)
 should be  safe.

 The real issue is convincing Walter to accept this.


 T


You'll have to atleast create ` trusted` wrappers around each 
such function though, if you want normal D functions to call them 
:(

That's why I feel we should allow ` trusted` to be annotated on 
`extern` functions.
Dec 02 2021
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 02, 2021 at 05:39:11PM +0000, Tejas via Digitalmars-d wrote:

 On Thursday, 2 December 2021 at 17:36:18 UTC, H. S. Teoh wrote:


[...]

 The solution is very simple: extern(C), extern(C++), and extern
 anything except extern(D) really, should be  system, and extern(D)
 (the default) should be  safe.
 
 The real issue is convincing Walter to accept this.




[...]

 You'll have to atleast create ` trusted` wrappers around each such
 function though, if you want normal D functions to call them :(
 
 That's why I feel we should allow ` trusted` to be annotated on
 `extern` functions.


I thought that was obvious?  We're talking about  system **by default**
here.  It does not preclude marking extern(C) declarations as  trusted.

Though TBH, that in itself is suspect; a function like memcpy, for
example, CANNOT be marked  trusted because it does not have a safe API:
the caller can easily pass a wrong argument to memcpy and it will
overrun the buffer.  Marking such a thing as  trusted undermines  safe,
and we're back to square one w.r.t. memory safety.

Unfortunately, this is exactly what most people will do, because they
just want the compile error to go away and stop bothering them. So in
name their code is  safe, but it doesn't mean very much in actuality
because there are badly-tagged  trusted functions inside where who knows
what can happen.


T

-- 
Живёшь только однажды.
Dec 02 2021
prev sibling parent reply Greg Strong <mageofmaple protonmail.com> writes:
On Thursday, 2 December 2021 at 17:36:18 UTC, H. S. Teoh wrote:

 The solution is very simple: extern(C), extern(C++), and extern 
 anything
 except extern(D) really, should be  system, and extern(D) (the 
 default)
 should be  safe.

 The real issue is convincing Walter to accept this.


Yes - this seems like the most reasonable solution to me.  I 
don't see why it would be particularly controversial.
Dec 02 2021
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Thursday, 2 December 2021 at 20:36:46 UTC, Greg Strong wrote:

 On Thursday, 2 December 2021 at 17:36:18 UTC, H. S. Teoh wrote:

 The solution is very simple: extern(C), extern(C++), and 
 extern anything
 except extern(D) really, should be  system, and extern(D) (the 
 default)
 should be  safe.

 The real issue is convincing Walter to accept this.


 Yes - this seems like the most reasonable solution to me.  I 
 don't see why it would be particularly controversial.


The last time this came up, Walter's argument against it was 
(iirc) that having two separate defaults for different kinds of 
functions is much worse for usability than having one universal 
default.

It's worth keeping in mind that extern(C) does not necessarily 
mean "written in C". So applying this rule as written would lead 
to some rather confusing edge cases; for example:

     import std.stdio;

     extern(C) void sayHello() //  system by default
     {
         writeln("Hello!");
     }

     void main() //  safe by default
     {
         sayHello();
         // Error:  safe function `main` cannot call  system 
function `sayHello`
     }
Dec 02 2021
parent Nick Treleaven <nick geany.org> writes:
On Thursday, 2 December 2021 at 20:51:56 UTC, Paul Backus wrote:

     extern(C) void sayHello() //  system by default
     {
         writeln("Hello!");
     }

     void main() //  safe by default
     {
         sayHello();
         // Error:  safe function `main` cannot call  system 
 function `sayHello`
     }


extern(C) with a function body should be  safe by default too, 
because the compiler knows the function is implemented in D. 
extern(C) without a function body should be  system by default, 
because the compiler has no idea what language the function is 
written in. So the above code would not error.
Dec 08 2021
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 02, 2021 at 08:36:46PM +0000, Greg Strong via Digitalmars-d wrote:

 On Thursday, 2 December 2021 at 17:36:18 UTC, H. S. Teoh wrote:

 The solution is very simple: extern(C), extern(C++), and extern
 anything except extern(D) really, should be  system, and extern(D)
 (the default) should be  safe.
 
 The real issue is convincing Walter to accept this.


 
 Yes - this seems like the most reasonable solution to me.  I don't see
 why it would be particularly controversial.


In my mind, it's crystal clear.  But Walter refused to do it for
whatever reason.  IIRC, he was the only person who rejected this.


T

-- 
In a world without fences, who needs Windows and Gates? -- Christian Surchi
Dec 02 2021
prev sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Thursday, 2 December 2021 at 12:15:38 UTC, Johan wrote:

 On Thursday, 2 December 2021 at 11:01:07 UTC, Imperatorn wrote:

 On Thursday, 2 December 2021 at 08:09:18 UTC, Paulo Pinto 
 wrote:

 [...]


 Bottom line:
 Use D instead of C 😎


 Sorry to rain on the party here, but D is of course not at all 
 immune to this problem.
 It was not hard to find out-of-bounds memory access in the D 
 compiler, using the fuzz techniques mentioned in the article.

 https://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html
 https://github.com/dlang/dmd/pull/7050

 Note the discussion of bounds checking in the PR...

 -Johan


Agreed, it was more of a "C sux" comment (I use it at work = have 
seen the horrors).
Anyway, D is still better than C 😊
Dec 02 2021
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 02, 2021 at 08:09:18AM +0000, Paulo Pinto via Digitalmars-d wrote:

 Google's Project Zero goes through a memory corruption exploit on
 Network Security Services, where despite all static analysers, fuzzers
 and code reviews, it flew under the radar.
 
 https://googleprojectzero.blogspot.com/2021/12/this-shouldnt-have-happened.html
 
 Hence why  safe matters.


Another nail in the coffin of C.  Still many more nails to go, but the
inevitable draws ever nearer.


T

-- 
Without geometry, life would be pointless. -- VS
Dec 02 2021
parent reply IGotD- <nise nise.com> writes:
On Thursday, 2 December 2021 at 11:27:01 UTC, H. S. Teoh wrote:

 Another nail in the coffin of C.  Still many more nails to go, 
 but the inevitable draws ever nearer.


 T


Would it be impossible to add bounds checking in C?

It's been over 4 decades and it seems like there is some profound 
resistance to add this.
Dec 02 2021
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
On Thursday, 2 December 2021 at 11:44:28 UTC, IGotD- wrote:

 On Thursday, 2 December 2021 at 11:27:01 UTC, H. S. Teoh wrote:

 Another nail in the coffin of C.  Still many more nails to go, 
 but the inevitable draws ever nearer.


 T


 Would it be impossible to add bounds checking in C?

 It's been over 4 decades and it seems like there is some 
 profound resistance to add this.


Yes, mostly due to culture, all major C compilers have extensions 
and secure libraries.

For example,

https://access.redhat.com/blogs/766093/posts/1976213

https://access.redhat.com/blogs/766093/posts/3606481

https://gcc.gnu.org/onlinedocs/gcc/Object-Size-Checking.html#Object-Size-Checking

Red-Hat and Android make use of FORTIFY_SOURCE among other 
features, for example,

https://source.android.com/devices/tech/debug/native-memory

Oracle has given up almost a decade ago, that is why Solaris on 
SPARC is basically a C Machine, thanks ADI.

https://docs.oracle.com/cd/E37838_01/html/E61059/gqajs.html

While Intel has borked their MPX implementation, ARM also got 
into the C Machine concept, which is being adopted across mobile 
OSes.

https://developer.apple.com/documentation/security/preparing_your_app_to_work_with_pointer_authentication

https://source.android.com/devices/tech/debug/tagged-pointers

So eventually hardware memory tagging will take care of killing 
processes that don't behave, and we will have C Machines with 
memory tagging, because the powers that could fix the language 
don't want to (ISO C, WG 14).
Dec 02 2021
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 02, 2021 at 11:44:28AM +0000, IGotD- via Digitalmars-d wrote:

 On Thursday, 2 December 2021 at 11:27:01 UTC, H. S. Teoh wrote:

 
 Another nail in the coffin of C.  Still many more nails to go, but
 the inevitable draws ever nearer.




[...]

 Would it be impossible to add bounds checking in C?
 
 It's been over 4 decades and it seems like there is some profound
 resistance to add this.


C wouldn't be C anymore once it has bounds checking.  You can bet your
life that the moment somebody proposes such a thing, complaints about
"efficiency" will flare up all over the intarweb.  99% of the C userbase
will revolt, and the proposal will be DOA.

Not to mention, how *do* you add bounds checking to a language where
arrays automatically and implicitly decay to pointers?  No length
information is present for you to do bounds checking on. The only way to
fix this is to reengineer the language from scratch to introduce
bounds-checked arrays.  But that does not fix the problems with the
existing standard C libraries which take bare pointers for arrays.
*Replacing* current arrays with bounds-checked ones essentially turns
it into D, at which point it's not C anymore (see previous paragraph).


T

-- 
Just because you can, doesn't mean you should.
Dec 02 2021
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Thursday, 2 December 2021 at 17:43:44 UTC, H. S. Teoh wrote:

 On Thu, Dec 02, 2021 at 11:44:28AM +0000, IGotD- via 
 Digitalmars-d wrote:

 On Thursday, 2 December 2021 at 11:27:01 UTC, H. S. Teoh wrote:

 
 Another nail in the coffin of C.  Still many more nails to 
 go, but the inevitable draws ever nearer.




 [...]

 Would it be impossible to add bounds checking in C?
 
 It's been over 4 decades and it seems like there is some 
 profound resistance to add this.


 C wouldn't be C anymore once it has bounds checking.  You can 
 bet your life that the moment somebody proposes such a thing, 
 complaints about "efficiency" will flare up all over the 
 intarweb.  99% of the C userbase will revolt, and the proposal 
 will be DOA.

 Not to mention, how *do* you add bounds checking to a language 
 where arrays automatically and implicitly decay to pointers?  
 No length information is present for you to do bounds checking 
 on. The only way to fix this is to reengineer the language from 
 scratch to introduce bounds-checked arrays.  But that does not 
 fix the problems with the existing standard C libraries which 
 take bare pointers for arrays. *Replacing* current arrays with 
 bounds-checked ones essentially turns it into D, at which point 
 it's not C anymore (see previous paragraph).


 T


You add it to the hardware, where every memory reference becomes 
a fat pointer, hence C Machines, so Lisp Machines with memory 
tagging, just designed for C instead.

That is how SPARC ADI, ARM MTE work.

Intel borked their MPX execution, so who knows if they will come 
up with something up, given that iOS and Android are now making 
use of hardware memory tagging.
Dec 02 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 3 December 2021 at 06:56:53 UTC, Paulo Pinto wrote:

 That is how SPARC ADI, ARM MTE work.

 Intel borked their MPX execution, so who knows if they will 
 come up with something up, given that iOS and Android are now 
 making use of hardware memory tagging.


Where did you find info on iOS and Android shipping with memory 
tagging?

According to the ARM MTE paper it adds 4 bits for every 16 bytes 
and is probabilistic in nature, so it does not work with large 
offsets. It also forces 16 byte alignment of objects on the stack.
Dec 03 2021
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Friday, 3 December 2021 at 08:45:06 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 3 December 2021 at 06:56:53 UTC, Paulo Pinto wrote:

 That is how SPARC ADI, ARM MTE work.

 Intel borked their MPX execution, so who knows if they will 
 come up with something up, given that iOS and Android are now 
 making use of hardware memory tagging.


 Where did you find info on iOS and Android shipping with memory 
 tagging?

 According to the ARM MTE paper it adds 4 bits for every 16 
 bytes and is probabilistic in nature, so it does not work with 
 large offsets. It also forces 16 byte alignment of objects on 
 the stack.


Read my previous comment, 
https://forum.dlang.org/post/anyicezeifvbuxurhwkz forum.dlang.org

On Android's case it is quite recent, only the models with 
Armv8.5 have it turned on, for older CPUs there is a lightheight 
version applied to critical components via HWASan.

And on iOS although Project Zero has find some flaws on PAC, they 
have been fixing those issues.
Dec 03 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 3 December 2021 at 10:04:11 UTC, Paulo Pinto wrote:

 Read my previous comment, 
 https://forum.dlang.org/post/anyicezeifvbuxurhwkz forum.dlang.org


Missed that Apple-link, yep, seems like they have it as an option 
for new devices if you use the right compiler switches.


 On Android's case it is quite recent, only the models with 
 Armv8.5 have it turned on, for older CPUs there is a 
 lightheight version applied to critical components via HWASan.

 And on iOS although Project Zero has find some flaws on PAC, 
 they have been fixing those issues.


It is interesting, although it seems in practice to be limited to 
object boundaries and won't work with fields? My take on this is 
that it won't work with arrays embedded in structs if the array 
is followed by another field?
Dec 03 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 3 December 2021 at 10:16:49 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 3 December 2021 at 10:04:11 UTC, Paulo Pinto wrote:

 Read my previous comment, 
 https://forum.dlang.org/post/anyicezeifvbuxurhwkz forum.dlang.org


 Missed that Apple-link, yep, seems like they have it as an 
 option for new devices if you use the right compiler switches.


It is also worth noting that based on the [ARM white 
paper](https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Arm_Memory_Tagging_Exten
ion_Whitepaper.pdf) we can deduce that it has some costs:

1. When initializing memory you have to tag all the memory, so it 
should be combined with clearing the memory. This is a 
performance issue for D as it does not zero-initialize memory by 
default. (NaN vs 0.0 for floats for instance)

2. You should keep the address space small, to avoid tagging 
costs.

3. Avoid allocations/freeing of memory. This is an issue for high 
level programming patterns. You might also run into problems if 
you provide your own allocators! This is an issue for D now that 
there seems to be a move towards using custom allocators.

4. Avoid unused buffer-space on the stack, as unused space has to 
be tagged anyway.

5. Alignment of tagged stack memory of 16 bytes.

If memory tagging is to be enabled for D it seems to have:

- language implications (switch to zero initialization as the 
default).

- library implications (allocators have to do tagging correctly)

- runtime implications (allocators and how to deal with "tagging 
exceptions")

- compiler implications (stack layout and tagging)
Dec 03 2021
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Friday, 3 December 2021 at 10:32:33 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 3 December 2021 at 10:16:49 UTC, Ola Fosheim Grøstad 
 wrote:

 On Friday, 3 December 2021 at 10:04:11 UTC, Paulo Pinto wrote:

 Read my previous comment, 
 https://forum.dlang.org/post/anyicezeifvbuxurhwkz forum.dlang.org


 Missed that Apple-link, yep, seems like they have it as an 
 option for new devices if you use the right compiler switches.


 It is also worth noting that based on the [ARM white 
 paper](https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Arm_Memory_Tagging_Exten
ion_Whitepaper.pdf) we can deduce that it has some costs:

 1. When initializing memory you have to tag all the memory, so 
 it should be combined with clearing the memory. This is a 
 performance issue for D as it does not zero-initialize memory 
 by default. (NaN vs 0.0 for floats for instance)

 2. You should keep the address space small, to avoid tagging 
 costs.

 3. Avoid allocations/freeing of memory. This is an issue for 
 high level programming patterns. You might also run into 
 problems if you provide your own allocators! This is an issue 
 for D now that there seems to be a move towards using custom 
 allocators.

 4. Avoid unused buffer-space on the stack, as unused space has 
 to be tagged anyway.

 5. Alignment of tagged stack memory of 16 bytes.

 If memory tagging is to be enabled for D it seems to have:

 - language implications (switch to zero initialization as the 
 default).

 - library implications (allocators have to do tagging correctly)

 - runtime implications (allocators and how to deal with 
 "tagging exceptions")

 - compiler implications (stack layout and tagging)


Note that on platforms like iOS and Android, going forward, those 
considerations don't matter at the language level, because the 
whole stack is using it.

Other examples would be Solaris SPARC ADI, IBM i TIMI, z/OS 
language environments, ClearPath MCP.

So any third party language needs to plays along if it wants to 
target the platform, no matter what.

These decisions have been taken because the costs outweight the 
security implications of not being able to fix C flaws at the 
language level (or the languages copy-paste compatible with it), 
and finally we have some companies with enough industry power 
that are feeling the money being lost fixing all those CVEs.
Dec 03 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 3 December 2021 at 12:08:59 UTC, Paulo Pinto wrote:

 Note that on platforms like iOS and Android, going forward, 
 those considerations don't matter at the language level, 
 because the whole stack is using it.


So you are saying that this will be required and not an option 
once all CPUs are capable? Right now it seems to be opt-in?


 So any third party language needs to plays along if it wants to 
 target the platform, no matter what.


That is certainly an interesting perspective. If this will be 
required on iOS it probably will be required on MacOS at some 
point too.
Dec 03 2021
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Friday, 3 December 2021 at 12:27:11 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 3 December 2021 at 12:08:59 UTC, Paulo Pinto wrote:

 Note that on platforms like iOS and Android, going forward, 
 those considerations don't matter at the language level, 
 because the whole stack is using it.


 So you are saying that this will be required and not an option 
 once all CPUs are capable? Right now it seems to be opt-in?
 ...


Yes that is the whole point.


 You can use the framework's sxadm command to enable and disable 
 security extensions for selected binaries and to manage their 
 properties.


https://docs.oracle.com/cd/E37838_01/html/E61021/sysauth-secext.html#OSSADsysauth-secext

So on Solaris, the admin gets to say if the OS runs the process 
under hardware memory tagging or not.

On Android,


 Starting in Android 11, for 64-bit processes, all heap 
 allocations have an implementation defined tag set in the top 
 byte of the pointer on devices with kernel support for ARM 
 Top-byte Ignore (TBI). Any application that modifies this tag 
 is terminated when the tag is checked during deallocation. This 
 is necessary for future hardware with ARM Memory Tagging 
 Extension (MTE) support.
....
 TBI requires a compatible kernel that correctly handles tagged 
 pointers passed from userspace. Android Common Kernels from 
 4.14 (Pixel 4) and higher feature the required TBI patches.


https://source.android.com/devices/tech/debug/tagged-pointers

Note the "all heap allocations" on the documentation and it being 
enabled on Pixel 4 and later devices.

You can guess similar documentation for the other links I 
provided earlier.
Dec 03 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 3 December 2021 at 13:33:22 UTC, Paulo Pinto wrote:

 On Friday, 3 December 2021 at 12:27:11 UTC, Ola Fosheim Grøstad

 So you are saying that this will be required and not an option 
 once all CPUs are capable? Right now it seems to be opt-in?
 ...


 Yes that is the whole point.


Why would it be the whole point, though? Processes are isolated 
so bugs do not matter for things like games where speed is more 
important?


 https://source.android.com/devices/tech/debug/tagged-pointers

 Note the "all heap allocations" on the documentation and it 
 being enabled on Pixel 4 and later devices.


Yes, that was pretty clear. MTE will be required in the future on 
Android devices.

Well. Seems like the only sensible initialization strategy is to 
use zero as the default from now on since that is what will be 
provided by the tagging allocator on all ARM devices… as far as I 
can tell.
Dec 03 2021
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 3 December 2021 at 13:44:49 UTC, Ola Fosheim Grøstad 
wrote:

 Yes, that was pretty clear. MTE will be required in the future 
 on Android devices.


That said, that applies to heap allocations and not the stack, I 
think. I don't see how they can enforce it on the stack as the OS 
cannot know what the object boundaries on the stack are.
Dec 03 2021
prev sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Friday, 3 December 2021 at 13:44:49 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 3 December 2021 at 13:33:22 UTC, Paulo Pinto wrote:

 On Friday, 3 December 2021 at 12:27:11 UTC, Ola Fosheim Grøstad

 So you are saying that this will be required and not an 
 option once all CPUs are capable? Right now it seems to be 
 opt-in?
 ...


 Yes that is the whole point.


 Why would it be the whole point, though? Processes are isolated 
 so bugs do not matter for things like games where speed is more 
 important?


Bugs matter for things like games as well, single player games 
without networking capabilities are a tiny market, and then there 
are all those MMO servers, with game items that actually map to 
real money.

I bet a game publisher loosing their MMO servers due to a buffer 
overrun exploited on the game client thanks to some optimizations 
won't be too much happy about it.
Dec 03 2021
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 3 December 2021 at 13:54:34 UTC, Paulo Pinto wrote:

 I bet a game publisher loosing their MMO servers due to a 
 buffer overrun exploited on the game client thanks to some 
 optimizations won't be too much happy about it.


It makes sense for MMOs in general, anything that can make 
cheating a little bit harder also makes sense in that domain.
Dec 03 2021
prev sibling parent Kagamin <spam here.lot> writes:
On Thursday, 2 December 2021 at 08:09:18 UTC, Paulo Pinto wrote:

 Google's Project Zero goes through a memory corruption exploit 
 on Network Security Services, where despite all static 
 analysers, fuzzers and code reviews, it flew under the radar.

 https://googleprojectzero.blogspot.com/2021/12/this-shouldnt-have-happened.html

 Hence why  safe matters.


That's tame, see Apple's complaint against NSO group. Pegasus was 
delivered through a buffer overflow vulnerability too.
Dec 03 2021