www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DIP 1028--Make safe the Default--Formal Assessment

reply Mike Parker <aldacron gmail.com> writes:
DIP 1028, "Make  safe the Default", has been accepted without 
comment.

https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md
May 21 2020
next sibling parent ag0aep6g <anonymous example.com> writes:
On 21.05.20 15:51, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without comment.
just another brick in the wall
May 21 2020
prev sibling next sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md
I guess they be more open to dips that fixes holes in the "safe by default" feature then.
May 21 2020
prev sibling next sibling parent reply Seb <seb wilzba.ch> writes:
On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md
"without comment" - even though there were a lot of unaddressed problems :/ Great! So what's the entire point of this process? To give people the illusion of progress and participation? Why we can't we have a technical board where the community can vote in experts and potentially companies could even buy a seat for $$$ which would mean a lot more for them than the current very vague sponsorship options. I'm aware that Walter doesn't like the idea of giving up ownership, but it makes all the other people question why they should still bother with this process and not simply fork and move to an open, transparent development...
May 21 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community can 
 vote in experts and potentially companies could even buy a seat 
 for $$$ which would mean a lot more for them than the current 
 very vague sponsorship options.
ditto, I think we should have like a seven person elected DIP committee who pass/fail things by majority vote. It is obvious to me that the current process is totally useless.
May 21 2020
next sibling parent reply SashaGreat <s g.com> writes:
On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community can 
 vote in experts and potentially companies could even buy a 
 seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.
ditto, I think we should have like a seven person elected DIP committee who pass/fail things by majority vote. It is obvious to me that the current process is totally useless.
Exactly and I even remember there was even a discussion about this on reddit (/r/programming) sometime ago where someone was pointing the flaws on D, and someone said the DIP process was one of them, and there were some known people around here replying that comment saying it was not the case. The discussion topic about this DIP (https://forum.dlang.org/thread/wkdpnzarkbtqryighzpx forum.dlang.org) had 210 replies, some concerns and in the end was ACCEPT WITHOUT ANY COMMENT. This is what I call a waste of everybody's time. SG.
May 21 2020
parent Les De Ridder <les lesderid.net> writes:
On Thursday, 21 May 2020 at 16:48:22 UTC, SashaGreat wrote:
 On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community 
 can vote in experts and potentially companies could even buy 
 a seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.
ditto, I think we should have like a seven person elected DIP committee who pass/fail things by majority vote. It is obvious to me that the current process is totally useless.
Exactly and I even remember there was even a discussion about this on reddit (/r/programming) sometime ago where someone was pointing the flaws on D, and someone said the DIP process was one of them, and there were some known people around here replying that comment saying it was not the case. The discussion topic about this DIP (https://forum.dlang.org/thread/wkdpnzarkbtqryighzpx forum.dlang.org) had 210 replies, some concerns and in the end was ACCEPT WITHOUT ANY COMMENT. This is what I call a waste of everybody's time.
Completely agree. The way DIPs are being handled is quite disappointing and unproductive. If the authors aren't even intending to address (or even acknowledge) difficult comments, why even write a DIP and have the community spend time on it at all? It also is not very encouraging to potential DIP authors who aren't Walter (or Átila or Andrei).
May 21 2020
prev sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community can 
 vote in experts and potentially companies could even buy a 
 seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.
ditto, I think we should have like a seven person elected DIP committee who pass/fail things by majority vote. It is obvious to me that the current process is totally useless.
As noted earlier, I'm with Steve, Seb, Adam, and "everyone but Walter" as far as I can see. If this stands we'll have gone from safe meaning "the compiler is responsible" to "the compiler can't guarantee anything unless you're pure D all the way down". Atila, what's your take on all this? Is it fork time?
May 21 2020
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 21 May 2020 at 23:49:22 UTC, Bruce Carneal wrote:
 On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 [...]
ditto, I think we should have like a seven person elected DIP committee who pass/fail things by majority vote. It is obvious to me that the current process is totally useless.
As noted earlier, I'm with Steve, Seb, Adam, and "everyone but Walter" as far as I can see. If this stands we'll have gone from safe meaning "the compiler is responsible" to "the compiler can't guarantee anything unless you're pure D all the way down". Atila, what's your take on all this? Is it fork time?
No.
May 22 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/21/20 7:49 PM, Bruce Carneal wrote:
 On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community can vote 
 in experts and potentially companies could even buy a seat for $$$ 
 which would mean a lot more for them than the current very vague 
 sponsorship options.
ditto, I think we should have like a seven person elected DIP committee who pass/fail things by majority vote. It is obvious to me that the current process is totally useless.
As noted earlier, I'm with Steve, Seb, Adam, and "everyone but Walter" as far as I can see.  If this stands we'll have gone from safe meaning "the compiler is responsible" to "the compiler can't guarantee anything unless you're pure D all the way down". Atila, what's your take on all this?  Is it fork time?
A fork does exist. As expected it went nowhere. https://bitbucket.org/larsivi/amber/wiki/Home. There is a paradox about forking the language - anyone good enough to lead a successful fork would also be wise enough to work on the D language instead. A committee would work if the community were at least one order of magnitude larger. It's just big numbers. Walter is world-class, another way of saying there's only a few of comparable strength in the world. A larger community would mean a larger likelihood of there being other people of comparable strength in it. Those could form a committee. As things are, Walter is so much stronger than every one of us, the relationship is highly asymmetric. His strength comes with equally unique ability to explain and debate, which makes many of the discussions in forums very frustrating.
May 23 2020
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 23.05.20 17:07, Andrei Alexandrescu wrote:
 A fork does exist. As expected it went nowhere. 
 https://bitbucket.org/larsivi/amber/wiki/Home. There is a paradox about 
 forking the language - anyone good enough to lead a successful fork 
 would also be wise enough to work on the D language instead.
I can only read that as you calling larsivi not "good enough to lead a successful fork" and not "wise enough to work on the D language instead". Walter says: "Belittling [others] [...] is unprofessional behavior." He was refering to other people on the forum, of course. But I would expand that to authors of other projects as well, particularly ones that are (even) smaller than D. Punching down is not cool.
May 23 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/23/20 6:49 PM, ag0aep6g wrote:
 On 23.05.20 17:07, Andrei Alexandrescu wrote:
 A fork does exist. As expected it went nowhere. 
 https://bitbucket.org/larsivi/amber/wiki/Home. There is a paradox 
 about forking the language - anyone good enough to lead a successful 
 fork would also be wise enough to work on the D language instead.
I can only read that as you calling larsivi not "good enough to lead a successful fork" and not "wise enough to work on the D language instead". Walter says: "Belittling [others] [...] is unprofessional behavior." He was refering to other people on the forum, of course. But I would expand that to authors of other projects as well, particularly ones that are (even) smaller than D. Punching down is not cool.
The point is on the other side - the barrier is very high. Carrying a programming language design and implementation is an extremely difficult task.
May 23 2020
prev sibling parent Arine <arine1283798123 gmail.com> writes:
On Saturday, 23 May 2020 at 15:07:16 UTC, Andrei Alexandrescu 
wrote:
 On 5/21/20 7:49 PM, Bruce Carneal wrote:
 On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community 
 can vote in experts and potentially companies could even buy 
 a seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.
ditto, I think we should have like a seven person elected DIP committee who pass/fail things by majority vote. It is obvious to me that the current process is totally useless.
As noted earlier, I'm with Steve, Seb, Adam, and "everyone but Walter" as far as I can see.  If this stands we'll have gone from safe meaning "the compiler is responsible" to "the compiler can't guarantee anything unless you're pure D all the way down". Atila, what's your take on all this?  Is it fork time?
A fork does exist. As expected it went nowhere. https://bitbucket.org/larsivi/amber/wiki/Home. There is a paradox about forking the language - anyone good enough to lead a successful fork would also be wise enough to work on the D language instead.
Arguably they would then be wise enough to stay away from D (and have).
 His strength comes with equally unique ability to explain and 
 debate, which makes many of the discussions in forums very 
 frustrating.
It is unique that's for sure. That's a good way to put it without being rude.
May 23 2020
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md
"without comment" - even though there were a lot of unaddressed problems :/ Great! So what's the entire point of this process? To give people the illusion of progress and participation? Why we can't we have a technical board where the community can vote in experts and potentially companies could even buy a seat for $$$ which would mean a lot more for them than the current very vague sponsorship options. I'm aware that Walter doesn't like the idea of giving up ownership, but it makes all the other people question why they should still bother with this process and not simply fork and move to an open, transparent development...
I honestly don't know if that would help. We'd be moving from a system where Walter makes decisions based on his mood on a particular day to one where others make decisions based on their moods on a particular day. The only thing worse than letting one person choose what to implement is having a group of people choose what to implement. Everyone has their own view of what is important. In my case, it's beginners and appealing to less technical users. Others view 20-year C++ programmers that specialize in performance optimizations as the only ones that matter. Needless to say, there's not a lot of overlap in the set of changes we think make sense. No matter who is making the decisions, the tradeoff between ease of use and technical awesomeness will continue to exist. The problem as I see it is someone making a decision on his own DIP. That just doesn't make any sense to me, and I've stated that numerous times. Walter has a tendency to throw gas on the fire by ignoring much of the feedback and not spending time to understand the points others are making when he does respond. I really think you should have to convince *someone else* that your proposal is reasonable.
May 21 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Thursday, 21 May 2020 at 17:03:49 UTC, bachmeier wrote:
 The problem as I see it is someone making a decision on his own 
 DIP. That just doesn't make any sense to me, and I've stated 
 that numerous times. Walter has a tendency to throw gas on the 
 fire by ignoring much of the feedback and not spending time to 
 understand the points others are making when he does respond. I 
 really think you should have to convince *someone else* that 
 your proposal is reasonable.
In principle, at least, this is why we have two "language maintainers," Walter and Atila (previously, Walter and Andrei). I think the real problem here is the lack of communication. As it stands, we have no way to tell whether feedback was considered or ignored, or what the ultimate rationale behind this decision was--we can only speculate. Even if we suppose for the sake of argument that the decision is sound on a technical level, this is poor leadership, and bodes ill for the future of the D language and its community.
May 21 2020
next sibling parent bachmeier <no spam.net> writes:
On Thursday, 21 May 2020 at 17:49:27 UTC, Paul Backus wrote:
 On Thursday, 21 May 2020 at 17:03:49 UTC, bachmeier wrote:
 The problem as I see it is someone making a decision on his 
 own DIP. That just doesn't make any sense to me, and I've 
 stated that numerous times. Walter has a tendency to throw gas 
 on the fire by ignoring much of the feedback and not spending 
 time to understand the points others are making when he does 
 respond. I really think you should have to convince *someone 
 else* that your proposal is reasonable.
In principle, at least, this is why we have two "language maintainers," Walter and Atila (previously, Walter and Andrei).
There's a big difference between being part of a three-person committee discussing a proposal by Walter and working together with Walter to make a decision on his own proposal. It's certainly doesn't pass the smell test.
May 21 2020
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 21, 2020 at 05:49:27PM +0000, Paul Backus via
Digitalmars-d-announce wrote:
[...]
 I think the real problem here is the lack of communication. As it
 stands, we have no way to tell whether feedback was considered or
 ignored, or what the ultimate rationale behind this decision was--we
 can only speculate.
I've said pretty much the same thing before, but it seems to have fallen on deaf ears. :-/ What's frustrating to the community isn't primarily the technical aspects (though of course, that's also a factor), but the human aspect of communication. Or rather, the lack thereof. I'd even grant that Walter, being BFDL, can make whatever arbitrary decisions he wants, but at the very least acknowledge the existence of the rest of us. "Accepted without comment" amounts to denial that we even exist, considering how much feedback was given on this DIP. Even a "accepted in spite of community feedback" is better than "without comment". (Maybe our virtual non-existence might ultimately become actual non-existence as this community shrinks to zero. :-P)
 Even if we suppose for the sake of argument that the decision is sound
 on a technical level, this is poor leadership, and bodes ill for the
 future of the D language and its community.
D excels at the technical aspects, but time and time again has shown that it lacks proper management / leadership. Contrary to my own inclinations I'm compelled to suggest hiring a *non-technical* person to take up the management/leadership roles (emphatically non-technical, because let's face it, we techies just don't have the people skillz it takes to do this properly), because the current situation clearly isn't working, and is quite detrimental to D and its future. T -- Computers shouldn't beep through the keyhole.
May 21 2020
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/21/2020 11:36 AM, H. S. Teoh wrote:
 I'd even grant that Walter, being BFDL, can make whatever arbitrary
 decisions he wants, but at the very least acknowledge the existence of
 the rest of us.  "Accepted without comment" amounts to denial that we
 even exist, considering how much feedback was given on this DIP.  Even a
 "accepted in spite of community feedback" is better than "without
 comment".
I've discussed it at length in the n.g. with y'all.
May 21 2020
prev sibling parent reply aberba <karabutaworld gmail.com> writes:
On Thursday, 21 May 2020 at 18:36:51 UTC, H. S. Teoh wrote:
 On Thu, May 21, 2020 at 05:49:27PM +0000, Paul Backus via 
 Digitalmars-d-announce wrote: [...]
 Even if we suppose for the sake of argument that the decision 
 is sound on a technical level, this is poor leadership, and 
 bodes ill for the future of the D language and its community.
D excels at the technical aspects, but time and time again has shown that it lacks proper management / leadership. Contrary to my own inclinations I'm compelled to suggest hiring a *non-technical* person to take up the management/leadership roles (emphatically non-technical, because let's face it, we techies just don't have the people skillz it takes to do this properly), because the current situation clearly isn't working, and is quite detrimental to D and its future. T
Ha ha. I tried suggesting something like this some yrs ago as I saw it happen and deadalnix bashed me like crazy. Didn't articulate it as well as you just did though. This needs to happen.
May 27 2020
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 28/05/2020 12:33 AM, aberba wrote:
 On Thursday, 21 May 2020 at 18:36:51 UTC, H. S. Teoh wrote:
 On Thu, May 21, 2020 at 05:49:27PM +0000, Paul Backus via 
 Digitalmars-d-announce wrote: [...]
 Even if we suppose for the sake of argument that the decision is 
 sound on a technical level, this is poor leadership, and bodes ill 
 for the future of the D language and its community.
D excels at the technical aspects, but time and time again has shown that it lacks proper management / leadership.  Contrary to my own inclinations I'm compelled to suggest hiring a *non-technical* person to take up the management/leadership roles (emphatically non-technical, because let's face it, we techies just don't have the people skillz it takes to do this properly), because the current situation clearly isn't working, and is quite detrimental to D and its future. T
Ha ha. I tried suggesting something like this some yrs ago as I saw it happen and deadalnix bashed me like crazy. Didn't articulate it as well as you just did though. This needs to happen.
I've been saying since 2012 that we need a project manager to help with communication. Mike has taken on parts of this role and has done quite a good job of it.
May 27 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/21/2020 10:03 AM, bachmeier wrote:
 Walter makes decisions based on his mood on a particular day
That's uncalled for.
May 21 2020
parent reply bachmeier <no spam.net> writes:
On Thursday, 21 May 2020 at 20:48:18 UTC, Walter Bright wrote:
 On 5/21/2020 10:03 AM, bachmeier wrote:
 Walter makes decisions based on his mood on a particular day
That's uncalled for.
Regional variation in English? Translation: You make your decisions based on how you feel about the situation at a point in time. "Mood" is used because there's some subjectivity and "gut feel" involved in the decision. It's not intended to be negative, it's simply a description of how any human makes any decision. That's why I said the same about how a committee would make decisions.
May 21 2020
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/21/2020 2:45 PM, bachmeier wrote:
 On Thursday, 21 May 2020 at 20:48:18 UTC, Walter Bright wrote:
 On 5/21/2020 10:03 AM, bachmeier wrote:
 Walter makes decisions based on his mood on a particular day
That's uncalled for.
Regional variation in English? Translation: You make your decisions based on how you feel about the situation at a point in time. "Mood" is used because there's some subjectivity and "gut feel" involved in the decision. It's not intended to be negative, it's simply a description of how any human makes any decision. That's why I said the same about how a committee would make decisions.
As a native English speaker, telling someone they are making decisions based on moods is telling them they are making irrational decisions. I accept that you did not intend it in this manner, but you should be aware that many people would be offended by it.
May 21 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/21/20 12:14 PM, Seb wrote:
 On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md
"without comment" - even though there were a lot of unaddressed problems :/ Great! So what's the entire point of this process? To give people the illusion of progress and participation?
Agree. I will not be participating in the DIP process from now on. It is a complete waste of time. Walter should just make the changes he wants and not bother with the facade of discussion. -Steve
May 21 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/21/2020 10:26 AM, Steven Schveighoffer wrote:
 Agree. I will not be participating in the DIP process from now on. It is a 
 complete waste of time. Walter should just make the changes he wants and not 
 bother with the facade of discussion.
Many replies to you, Steven: https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Final_Review_336354.html I did not ignore you. I just didn't agree.
May 21 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/21/20 4:59 PM, Walter Bright wrote:
 On 5/21/2020 10:26 AM, Steven Schveighoffer wrote:
 Agree. I will not be participating in the DIP process from now on. It 
 is a complete waste of time. Walter should just make the changes he 
 wants and not bother with the facade of discussion.
Many replies to you, Steven: https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Fina _Review_336354.html
Many unsatisfactory replies. Eventually my attempts to explain my position were ignored. It's the equivalent of walking off stage because you lost the debate, only to claim you won because you stopped talking. My points were straightforward, easy to understand, vastly agreed upon (and echoed) by all of the experienced people here (except you), and your rebuttals were extremely dubious, depending on hypothetical non-sequiturs with an imagined developer who doesn't understand how prototypes might work. Short story is, you made poor arguments and considered the matter closed "without comment". This is extremely unsatisfactory.
 I did not ignore you. I just didn't agree.
You didn't agree, and that is your choice. But you also didn't address any of the points I made, instead choosing to focus on confusion from an imaginary person who has no clue why his safe code cannot call unsafe functions even though the attributes are not explicit (which means he should probably spend the 10 seconds to learn and become a better programmer). The unfortunate end result of this change is that safety will be gutted with all C functions being trusted by default, ironically by the person who is claiming that memory safety is of utmost importance, and how C got it wrong. All without delivering a single convincing argument as to why he is right. I even put forth a completely ignored compromise solution that would have solved the problem and allowed extern(C) functions to be considered safe by default: https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Final_Review_336354.html#N336968 This is why I consider the process a waste of time, and why I'm done participating. -Steve
May 21 2020
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Thursday, 21 May 2020 at 21:41:53 UTC, Steven Schveighoffer 
wrote:
 The unfortunate end result of this change is that safety will 
 be gutted with all C functions being trusted by default
I'm really sorry, Walter, but I have to agree with Steve on this point. This was the one aspect of the DIP that I really wanted to get modified, I also proposed a number of options for how to do so, and I'm really worried about the consequences of this decision. I recognize and respect your right to make decisions as you see fit for the language, but I really wish I could persuade you to change your mind about this one :-)
May 21 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/21/2020 2:41 PM, Steven Schveighoffer wrote:
 I even put forth a completely ignored compromise solution that would have
solved 
 the problem and allowed extern(C) functions to be considered  safe by default: 
 https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Final_Review
336354.html#N336968 
I did see it. I can't even get across the notion of what "undefined symbol xxxx" messages from the linker mean to a lot of programmers. I'll never get this one across. The linking process is already as complex as I dare. BTW, two symbols resolving to the same address can be a bit awkward in some object formats and isn't really supported, like with COMDATs. In my experience, unless C or C++ compilers generate certain constructs, linkers (and debuggers) don't work for those constructs, whether they are documented to or not.
 This is why I consider the process a waste of time, and why I'm done
participating.
The level of negativity in that thread was what caused me to stop responding, though I continued reading. Every reply I made produced 10 responses, an exponential explosion, and yet I was just repeating myself. Two sides to every story. FWIW, I am going to try again with another post here, for those who want a convenient summary of the rationale.
May 21 2020
parent reply Faux Amis <faux amis.com> writes:
On 2020-05-22 03:16, Walter Bright wrote:

 
 The level of negativity in that thread was what caused me to stop 
 responding, though I continued reading. Every reply I made produced 10 
 responses, an exponential explosion, and yet I was just repeating 
 myself. Two sides to every story.
 
 FWIW, I am going to try again with another post here, for those who want 
 a convenient summary of the rationale.
Just a suggestion, but sometimes matters are best discussed over audio/video. Would having a public teams/zoom/.. meeting be helpful? I would definitely listen/watch; even if I were muted and could only chat maybe.
May 23 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/23/2020 4:26 AM, Faux Amis wrote:
 Just a suggestion, but sometimes matters are best discussed over audio/video. 
 Would having a public teams/zoom/.. meeting be helpful?
 
 I would definitely listen/watch; even if I were muted and could only chat
maybe.
You're right, and that is the whole purpose behind DConf. It's amazing how our differences melt away when discussing with a beer in hand :-)
May 23 2020
parent Faux Amis <faux amis.com> writes:
On 2020-05-24 00:15, Walter Bright wrote:
 On 5/23/2020 4:26 AM, Faux Amis wrote:
 Just a suggestion, but sometimes matters are best discussed over 
 audio/video. Would having a public teams/zoom/.. meeting be helpful?

 I would definitely listen/watch; even if I were muted and could only 
 chat maybe.
You're right, and that is the whole purpose behind DConf. It's amazing how our differences melt away when discussing with a beer in hand :-)
I meant it could be something being done more often, Dconf is (normally) only once a year. Maybe discussions like these need to be discussed more frequently.. quarterly, or even a quick official Dtalk every month :)
May 24 2020
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Thursday, 21 May 2020 at 20:59:08 UTC, Walter Bright wrote:
 Many replies to you, Steven:

 https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Final_Review_336354.html

 I did not ignore you. I just didn't agree.
One concern here is that these responses are scattered across different parts of a long discussion thread. So it probably would be a good idea for the acceptance to be accompanied by an explanation of what the major objections were to the DIP, and why they were discounted. It's not so different from the way that, at the end of a talk or presentation, it's a good idea to recap the key points that have already been covered.
May 21 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/21/2020 2:44 PM, Joseph Rushton Wakeling wrote:
 One concern here is that these responses are scattered across different parts
of 
 a long discussion thread.  So it probably would be a good idea for the 
 acceptance to be accompanied by an explanation of what the major objections
were 
 to the DIP, and why they were discounted.
 
 It's not so different from the way that, at the end of a talk or presentation, 
 it's a good idea to recap the key points that have already been covered.
Fair enough. I'll do so. Stay tuned.
May 21 2020
parent 12345swordy <alexanderheistermann gmail.com> writes:
On Friday, 22 May 2020 at 00:50:00 UTC, Walter Bright wrote:
 On 5/21/2020 2:44 PM, Joseph Rushton Wakeling wrote:
 One concern here is that these responses are scattered across 
 different parts of a long discussion thread.  So it probably 
 would be a good idea for the acceptance to be accompanied by 
 an explanation of what the major objections were to the DIP, 
 and why they were discounted.
 
 It's not so different from the way that, at the end of a talk 
 or presentation, it's a good idea to recap the key points that 
 have already been covered.
Fair enough. I'll do so. Stay tuned.
Thank you. Having an explanation on why the DIP is accepted despite being highly controversial would be sufficient. -Alex
May 21 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/21/2020 9:14 AM, Seb wrote:
 On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md
"without comment" - even though there were a lot of unaddressed problems :/
I did address, several times over, the problems in the discussion thread.
 Great! So what's the entire point of this process?
 To give people the illusion of progress and participation?
Consider that several of my own DIPs were rejected due to community response.
 Why we can't we have a technical board where the community can vote in experts 
 and potentially companies could even buy a seat for $$$ which would mean a lot 
 more for them than the current very vague sponsorship options.
 I'm aware that Walter doesn't like the idea of giving up ownership, but it
makes 
 all the other people question why they should still bother with this process
and 
 not simply fork and move to an open, transparent development...
I expected flak from this decision. I'm prepared to take the flak because this is the right decision. I did not make it lightly. Please keep in mind that I've made other unpopular decisions that have proven their worth over time. I hope you'll reserve judgement until we all see how this change plays out. If it does turn out badly, it's on me and I'll take my lumps.
May 21 2020
next sibling parent Arine <arine1283798123 gmail.com> writes:
On Thursday, 21 May 2020 at 20:46:09 UTC, Walter Bright wrote:
 On 5/21/2020 9:14 AM, Seb wrote:
 Why we can't we have a technical board where the community can 
 vote in experts and potentially companies could even buy a 
 seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.
 I'm aware that Walter doesn't like the idea of giving up 
 ownership, but it makes all the other people question why they 
 should still bother with this process and not simply fork and 
 move to an open, transparent development...
I expected flak from this decision. I'm prepared to take the flak because this is the right decision. I did not make it lightly.
That's interesting that you are saying it is the "right" decision. Let's analyze that for a second: What this boils down to is Simplicity Vs. Memory Safety Is the **correct** code actually simpler? A beginner that doesn't have the knowledge of D's safety system will ultimately create C bindings that are implicitly safe without realizing the consequences of their actions. This change introduces a risk to memory safety, extern(C) declarations that were system will now be flipped to safe. No, putting system: at the top isn't a solution, extern(C) functions aren't all nicely put in their own modules. Druntime and Phobos both have extern(C) declarations throughout their source; I guarantee you missed them with your PR's trying to illustrate how simple it is. As others will most definitely miss them as well. Reasons for Memory Safety: - Less error prone and simpler for beginners, the default is what the *correct* behavior should be. They organically discover safe writing C bindings. - Less error prone for advanced users, won't accidentally call a C function that isn't explicitly verified as being trusted. - Switch to safe by default won't introduce potential security bugs from extern(C) functions not being annotated system in the switch. - Proven implementation by other safe languages (Rust). Reasons for Simplicity: - simpler logic for the compiler - simpler rules for the user that default to *incorrect* code - Fear of the unknown, unforeseen complexity - trust Walter This is where the divide is, from my perspective. There's this fear of the unknown, that making it slightly more complicated will introduce these unforeseen complications that will far outweigh any benefit that is added from it. I don't see what that can be. There are other languages that have safe by default, and then have C declarations be unsafe by default. They don't have any of these devastation complications you seem to think that there will be. This is what doesn't sit right with me. If you could give an example of a complication that could occur, then I think you could get more people onboard. But it would have to be justifiable against the increased security risk of having extern(C) declarations be safe by default. For a DIP that is trying to increase safety, it is a bit ironic that it will also be reducing safety at the same time. Otherwise right now, as it stands, this basically boils down to Logic Vs. Fear.
 Please keep in mind that I've made other unpopular decisions 
 that have proven their worth over time. I hope you'll reserve 
 judgement until we all see how this change plays out.

 If it does turn out badly, it's on me and I'll take my lumps.
You've also done the opposite, you can't justify it with such a statement unless you've never made a mistake. This whole process is suppose to avoid exactly this; a single person taking a gamble.
May 21 2020
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 21, 2020 at 01:46:09PM -0700, Walter Bright via
Digitalmars-d-announce wrote:
[...]
 I expected flak from this decision. I'm prepared to take the flak
 because this is the right decision. I did not make it lightly.
This makes it sound like you think that those who disagree with you disagree with safe by default. That is not the case. I think in general, (almost?) everyone here agrees that safety by default is the way to go. We all agree that safe by default ought to be in the language, in one form or another. That's not the dispute here. The dispute concerns the specific implementation of safe by default as set out in this DIP. Specifically, the weakening of the promise of safe by implicitly trusting extern(C) declarations to be safe, which is equivalent to programming by convention ("trust the programmer to remember to write system on extern(C) prototypes"). Which, ironically, undermines the whole purpose of this DIP. (That is, unless I misunderstood, and the whole purpose of this DIP is to undermine safety and thereby make it even more of a joke than it currently is right now.)
 Please keep in mind that I've made other unpopular decisions that have
 proven their worth over time. I hope you'll reserve judgement until we
 all see how this change plays out.
[...] Oh I'm waiting to see this one plays out, you bet! What will come of trusting the programmer to do the right thing when it comes to annotating extern(C) prototypes with system? Well, I'm sure it will all work out well in the end, since, after all, it isn't as if we didn't have ~50 years or so of experience with programming by convention, y'know, esp. in C code (that we should implicitly trust when calling from D, btw), which has led to beautiful security holes and exploits caused by memory corruption and other such wonderful things. Or, since you like airplane analogies, there's absolutely nothing wrong with designing your cockpit controls such that the default behaviour, when the pilot does not indicate otherwise, is to nose-dive into the ground. After all, if he didn't mean to do that, he should've remembered to override the default behaviour explicitly, right? That's what he was trained to do, anyway. And besides, it's easier to implement the controls this way, since doing otherwise would add excessive complication to the design. T -- The early bird gets the worm. Moral: ewww...
May 21 2020
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/21/2020 4:45 PM, H. S. Teoh wrote:
 This makes it sound like you think that those who disagree with you
 disagree with  safe by default.  That is not the case.
I'm sure you all know what I'm talking about.
May 21 2020
prev sibling next sibling parent Johannes Loher <johannes.loher fg4f.de> writes:
On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md
As others have mentioned, this really is a farce. I understand that not everybody will be happy with every decision but the fact that there is not even a comment is just disrespectful towards all the people that put in a lot of effort to review the DIP. How can you seriously expect them to continue to do that if you treat them like this? Don't you think their work is valuable? If not, why bother with the DIP process at all? It just seems like a total waste of everybody’s time. I am really disappointed about this...
May 21 2020
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
I have made these points before, but I'll summarize them here
for convenient referral.

----

The proposed amendment to Safe by Default is:

"Extern C and C++ functions without bodies should be  system
by default."

The rationale is that since the compiler cannot check them for  safe,
they should be assumed unsafe.

This is Obviously A Good Idea. Why would I oppose it?

1. I've been hittin' the crack pipe again.
2. I was secretly convinced, but wanted to save face.
3. I make decisions based on consultation with my astrologer.
4. I am evil.

I'm going to play the experience card. Many Obviously Good Ideas
appear again and again. Why they aren't so hot only becomes
clear after (sometimes considerable) experience. Some are:

1. Implicit declaration of variables.
2. AST macros (debated in the n.g. multiple times).
3. Exception specifications.

I'll expound on Exception Specifications, as they are more similar
to this case than the others.

Exception Specifications appeared with Java back in the 90's. With it,
each function must list which exceptions it throws, such as (in D):

     T myFunction() throws Alpha, Gamma;

If myFunction() throws any exception that is not derived from Alpha
or Gamma, the compiler issues a compile-time error.

This is Obviously A Good Idea. Smart, experienced, well-meaning programmers
endorsed it wholeheartedly. (Even C++98 adopted the idea.) One such person,
Bruce Eckel (who incidentally authored the early Zortech C++ manual and
is a friend of mine) authored the well-regarded "Thinking In ..." series of
books, one of which was "Thinking in Java". He bought the idea, and preached
and promoted it heavily.

Then came the hangover. Bruce wrote a brilliant mea culpa (that has since
sadly vanished from the internet) about what went wrong. The troubles were:

1. It was tedious to go through every function called by myFunction() and
figure out what all the possible Exceptions thrown might be.

2. If a low level function started throwing a new type, every single caller
all the way up the call tree had to add it to its exception specification.

This, of course, was incredibly tedious and annoying. So what happened?

1. Functions would just do a try-catch(Throwable) and ignore the exceptions.
Oops.

2. The Exception Specifications became simply `throws Throwable`. Oops.

What was most damning was the Bruce Eckel himself was horrified to find that
this was his behavior when writing his own code. Other advocates of it also
admitted they did that too. Exception specifications became a farce, and
Java was quietly softened to not require them. Exception specifications were
just ignored by the C++ community.

How does this relate to safe by default?

Consider the common (because that's how D started out) case of:

----- clibrary.d --------

     T massage_data(... many parameters ...);
     ... 200 more such declarations ...

----- app.d ----------

     import clibrary;

     void useClibrary( ... parameters ...) {
         massage_data(parameters);
     }

---------------------

This code, today, does not use annotations and it works. It's been working
for a long time. Now, we implement the amendment, and it stops compiling
because useClibrary is  safe and massage_data is  system. The user is faced
with the following alternatives:

1. Go through 200 functions in clibrary.d and determine which are  safe
and which are  system. This is what we want them to do. We try to motivate
this with compiler error messages. Unfortunately, this is both tedious and
thoroughly impractical, as our poor user Will Not Know which are safe and
which are system. We can correctly annotate core.stdc.stdio because I know
those functions intimately. This is not true for other system C APIs, and
even less true for some third party C library we're trying to interface to.

2. Annotate useClibrary() as  trusted or  system. While easier, this causes
all benefits to  safe by default to be lost.

3. Wrap the call to massage_data() with:

     ()  trusted { massage_data(parameters); } ();

If there are a lot of calls to clibrary, this is going to look pretty awful.
Nobody likes writing or reading such ugly code. It's ok here and there, but
not as a general thing.

4. Edit clibrary.d and make the first line:

      safe:

I submit that, just like with Java, Option 4 is what people will reach for,
nearly every time. I've had some private conversations where people admitted
this was what they'd do. People who knew it was wrong to do that.

If it's  safe by default, and then someone chooses to annotate it with  system
here and there, I'd feel a lot more confident about the accuracy of the code
annotations than if it just had  safe: at the top. At least they tried.

What is actually accomplished with this amendment if it was implemented?

1. Adds a funky, special case rule. It's better to have simple, easily
understood rules than ones with special cases offering little improvement.

2. Existing, working code breaks.

3. The most likely code fixes are to just make it compile, absolutely
nothing safety-wise is improved. The added annotations will be a fraud.

D should not encourage "greenwashing" practices like the Java
exception specification engendered. The compiler cannot vet the accuracy
of bodyless C functions, and we'll just have to live with that. The proposed
amendment does not fix that.

And so, I did not incorporate the proposed amendment to the Safe by Default
DIP.
May 21 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.

 ----

 [...]
First, thank you for taking the time to summarize your position. I understand your objection to the proposed amendment much better now than I did after only reviewing the discussion thread. Second, I'd like to present a potential failure mode of DIP 1028 that I think still remains unaddressed. Consider the following scenario: Someone has created bindings for a C library and published them on code.dlang.org. Because they were written prior to DIP 1028, the author assumed that system was the default, and didn't bother to add explicit annotations to system functions. Their code looks like this: --- clibrary.d void monkey_around(...); // assumed system-by-default --- Months or years later, I decide to write a D program that makes use of these bindings. By then, safe-by-default has been fully implemented. I add `clibrary` as a dependency to my Dub project and write the following code: --- app.d import clibrary; void main() // safe-by-default { /* ... code ... */ monkey_around(...); /* ... more code ... */ } --- My program compiles with no errors...and then corrupts memory at run-time, even though every line of code I've written is safe. Oops. This is the nightmare scenario that people are worried about: safety violations being introduced *silently* into existing, correct D code. Of course, the author of clibrary.d *should* update their code to add explicit system annotations, but how will they know there's a problem if the compiler never complains about it? Most likely, by the time they or one of their users finally track down the root of the problem, the damage will already have been done. Something ought to be done to prevent this. It doesn't have to be the exact proposal from the discussion thread, but doing nothing and allowing widespread silent breakage cannot possibly be the best solution.
May 21 2020
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/21/2020 8:36 PM, Paul Backus wrote:
 Something ought to be done to prevent this. It doesn't have to be the exact
 proposal from the discussion thread, but doing nothing and allowing widespread
 silent breakage cannot possibly be the best solution.
I can see that happening. A simple example would be: extern (C) void free(void* p); ... free(p); free(p); The thing is, you are no worse off than now. If free() can be misused by calling it from system code, it can be misused by calling it from safe code. There's no way the compiler can detect this. If you annotate it, you'll just have to annotate it correctly. Forcing an annotation just means slapping safe: at the beginning of the file and moving on - it's not going to help.
May 21 2020
next sibling parent reply Dukc <ajieskola gmail.com> writes:
 I can see that happening. A simple example would be:

     extern (C) void free(void* p);
     ...
     free(p);
     free(p);

 The thing is, you are no worse off than now. If free() can be 
 misused by calling it from system code, it can be misused by 
 calling it from safe code.
Wrong :-(. The scenario is this: ``` safe void foo(int* p) { import customfreefunction.noannotations; p.free; p.free; } ``` Now, this will not compile, because `free` is ` system`. But if `free` is in unannotated module, this will compile after the DIP implementation. Obviously not with the standard library `free` because it'll be annotated ` system` anyway, but with some custom `free` function this is an issue. If the situation in using new features is indeed as dire as you fear, a better alternative would have been to just reject the DIP. Then there would be less ` safe` code, but at least you could trust ` safe` much more, due to the above phenomenon. I do think there would have been a way to have ` safe` by default even with the assumptions you made about it's abuse, but the DIP reviews are over so I don't think it's worth explaining anymore. But at least you gave the reasonable rationale we wanted. Thank you.
May 22 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/22/20 6:13 AM, Dukc wrote:
 Wrong :-(. The scenario is this:
 
 ```
  safe void foo(int* p)
 {   import customfreefunction.noannotations;
      p.free;
      p.free;
 }
 ```
 
 Now, this will not compile, because `free` is ` system`. But if `free` 
 is in unannotated module, this will compile after the DIP 
 implementation. Obviously not with the standard library `free` because 
 it'll be annotated ` system` anyway, but with some custom `free` 
 function this is an issue.
I want to interject slightly to say that this isn't the exact problem. The exact problem is that today `foo` is NOT marked safe (or else it wouldn't compile). But after the DIP it will be IMPLICITLY marked safe, and so will `free`. Thereby silently migrating all system code that should actually be (and was INTENDED to be) system code straight to safe code without any warnings. This in itself isn't a problem, code that compiled yesterday compiles today, and is just as bad. The true problem is that after the DIP is implemented, new users will use foo within safe code they INTEND to be safe, and will be none the wiser that it's not. The scenario I imagine is that unadorned code will exist for years as safe (implicitly) and only after years of being depended on as safe code, is discovered to be completely unsafe, and the only way to fix is to break existing usage. Or worse, it's not discovered and then D's already shaky reputation for safe code is destroyed when a hacker exploits the memory corruption in fully-marked safe code. The actual bug may be extremely subtle. We have code in phobos that does stuff like: static if(is(typeof(() safe { obj.doSomething(); }))) obj.doSomething(); else doSomethingSafely(obj); which will silently switch to using unsafe (but marked safe) mechanisms (for optimization). -Steve
May 22 2020
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Friday, 22 May 2020 at 16:03:00 UTC, Steven Schveighoffer 
wrote:
 [snip]
Fortunately, the above point can be more easily fixed by making `free` system, which will then require annotating every subsequent piece of code that touches it. It's annoying transition, but it's do-able. The problem is when you aren't sure what needs to be marked system. For instance, if you have some function now that is system by default and does pointer arithmetic, then the switch to safe by default will result in that no longer compiling. That's good. The emphasis on prototypes is that the compiler will not be able to notify you that the function should not be compiled. If the `free` function comes from a prototype (and thus cannot be manually verified by the compiler), then that is much worse for safety than a `free` function with the full body available. This comes back to a point I had made on one of the original DIP discussion threads that one issue is that safe is a blacklist of things you can't do rather than a whitelist of allowed things. If safe were designed as a whitelist, then only things that are compiled verified could be marked safe. That means that a function without a body should not be inferred to be safe. That `static if` thing just looks like a nightmare, but I think the point still applies that you will get error messages if the body is available and you try to do something that isn't safe.
May 22 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 22 May 2020 at 16:39:42 UTC, jmh530 wrote:
 Fortunately, the above point can be more easily fixed by making 
 `free`  system
With the o/b system `free` might actually work out OK....
May 22 2020
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 22.05.20 18:43, Adam D. Ruppe wrote:
 On Friday, 22 May 2020 at 16:39:42 UTC, jmh530 wrote:
 Fortunately, the above point can be more easily fixed by making `free` 
  system
With the o/b system `free` might actually work out OK....
free(new int);
May 22 2020
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/22/2020 10:25 AM, Timon Gehr wrote:
 With the o/b system `free` might actually work out OK....
free(new int);
I did mention in the documentation that when using different memory allocators, mixing them up will not be detected. You'd have to declare the allocators using specific pointer types, rather than void*.
May 23 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/22/20 12:39 PM, jmh530 wrote:
 On Friday, 22 May 2020 at 16:03:00 UTC, Steven Schveighoffer wrote:
 [snip]
Fortunately, the above point can be more easily fixed by making `free` system, which will then require annotating every subsequent piece of code that touches it. It's annoying transition, but it's do-able.
You can't, you don't control that code, someone else does (this is important, you can declare extern(C) functions anywhere, even locally). And from the experience of .save for forward ranges, people just aren't going to do this. They are going to do the easiest thing that works without complaint -- i.e. don't bother marking.
 That `static if` thing just looks like a nightmare, but I think the 
 point still applies that you will get error messages if the body is 
 available and you try to do something that isn't  safe.
It essentially is a "best effort" at calling the right thing. The code compiles either way, and be safe either way, but will do the unsafe thing when it really should do the safe thing. It's ACTIVELY trying to ensure safety, but the compiler will thwart it. -Steve
May 22 2020
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Friday, 22 May 2020 at 16:47:34 UTC, Steven Schveighoffer 
wrote:
 [snip]

 You can't, you don't control that code, someone else does (this 
 is important, you can declare extern(C) functions anywhere, 
 even locally).
You can make a separate module with one function that just calls the `free` function and has the system attribute. Then, just change the import for the free function to that new module. Annoying, but potentially fix-able. The bigger issue is that we know that we should check the `free` function, but what about all the other functions out there that might corrupt memory that we don't really know about. It's not scale-able.
 And from the experience of .save for forward ranges, people 
 just aren't going to do this. They are going to do the easiest 
 thing that works without complaint -- i.e. don't bother marking.
Probably true.
May 22 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/22/20 1:52 PM, jmh530 wrote:
 On Friday, 22 May 2020 at 16:47:34 UTC, Steven Schveighoffer wrote:
 [snip]

 You can't, you don't control that code, someone else does (this is 
 important, you can declare extern(C) functions anywhere, even locally).
You can make a separate module with one function that just calls the `free` function and has the system attribute. Then, just change the import for the free function to that new module. Annoying, but potentially fix-able.
You are using library fubar. fubar defines a function foo: void foo(int *p) { import fubar.cfunctions : free; free(p); } After this DIP, it now becomes completely assumed safe. The author of fubar intended it to be system. But now all safe code can now call it, and there's nothing you can do except ask them to fix both modules to be system. You can just stop using the library, sure. But you have no control over whether fubar's code is properly written. That is the point. If the C prototype was instead considered system, then foo stops compiling (for good reason), and you can either motivate the author to fix it, or use a different library. But at least you can't ACCIDENTALLY use it in safe code. Essentially all code written before this DIP is switched on is completely suspect, and needs manual verification -- if you care about safety at all. -Steve
May 22 2020
parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 22 May 2020 at 18:44:09 UTC, Steven Schveighoffer 
wrote:
 [snip]
 You are using library fubar.

 fubar defines a function foo:
I agree with most of what you said. I was trying to make the point that this is a contrived example where we know for sure that one function should be system and is actually safe. For one function where you know it should be system, then you can create a fix. However, it's unreasonable to require that for all of the code from outside modules without attributes (even if it could be possible to write something in D that makes it relatively less painful to do this).
May 22 2020
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 22, 2020 at 04:39:42PM +0000, jmh530 via Digitalmars-d-announce
wrote:
[...]
 This comes back to a point I had made on one of the original DIP
 discussion threads that one issue is that  safe is a blacklist of
 things you can't do rather than a whitelist of allowed things.
[...] Yes, and that's why safe has been on a shaky foundation since the beginning. In theory, a perfect, bug-free blacklist is equivalent to a perfect, bug-free whitelist. But given the rate at which D is adopting new features, and the combinatorial explosion of feature combinations, any of which might lead to violation of safe, my confidence level on the completeness of the blacklist is on the low side. Had it been a whitelist, it would have been much better, because when people ran into a combination of features that ought to be allowed, but isn't, they would file a bug, and then that case could be added to the whitelist after careful vetting. You always err on the safe side. With a blacklist, you're playing catch-a-mole with missed cases, overlooked cases, and you never have complete certainty that you've weeded out all dangerous constructs. T -- "Computer Science is no more about computers than astronomy is about telescopes." -- E.W. Dijkstra
May 22 2020
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 22, 2020 at 12:03:00PM -0400, Steven Schveighoffer via
Digitalmars-d-announce wrote:
[...]
 Or worse, it's not discovered and then D's already shaky reputation
 for  safe code is destroyed when a hacker exploits the memory
 corruption in fully-marked  safe code.
[...] TBH, this whole fiasco just confirms my initial skepticism about safe. The concept of mechanical verifiability is cool, I'm on board with that. But the execution of it in safe up to before this DIP already left a lot to be desired (the beginning of which is the fact that safe is based on a blacklist rather than a whitelist, but let's not go there now). Now with this DIP, it's basically another nail in the safe coffin. Essentially what it means is that safe is a nice tool to have for casual use, but don't depend on it for real-world, mission critical use, because it doesn't have the strong guarantees you might imagine it has. It promises a lot in bold letters, but the fine print contains a lot of disclaimers. Which means, in my case, it's not worth bothering with. The amount of effort required to make code 100% safe is just not worth the leaky "guarantees" it brings. T -- First Rule of History: History doesn't repeat itself -- historians merely repeat each other.
May 22 2020
prev sibling parent Paul Backus <snarwin gmail.com> writes:
On Friday, 22 May 2020 at 05:37:55 UTC, Walter Bright wrote:
 On 5/21/2020 8:36 PM, Paul Backus wrote:
 Something ought to be done to prevent this. It doesn't have to 
 be the exact
 proposal from the discussion thread, but doing nothing and 
 allowing widespread
 silent breakage cannot possibly be the best solution.
I can see that happening. A simple example would be: extern (C) void free(void* p); ... free(p); free(p); The thing is, you are no worse off than now. If free() can be misused by calling it from system code, it can be misused by calling it from safe code. There's no way the compiler can detect this. If you annotate it, you'll just have to annotate it correctly. Forcing an annotation just means slapping safe: at the beginning of the file and moving on - it's not going to help.
Currently, if I want to use safe correctly, all I need to do is annotate my code with safe and fix any compile errors that arise as a result. With -preview=safedefault enabled, if I want to use safe correctly, I am now required to *manually* review every extern function declaration in every module of every dependency of my program to make sure that they are explicitly annotated. If I do not do this, I risk introducing silent safety violations to my program. You are correct that in both cases, I can run into trouble if safe is misused. Nonetheless, I submit to you that I am worse off in the second case than in the first. The compiler cannot be expected to guard against deliberate misuse, but it *can* be expected to give us the necessary tools to use safe correctly. The current implementation of DIP 1028 does not do so.
May 22 2020
prev sibling parent reply bachmeier <no spam.net> writes:
On Friday, 22 May 2020 at 03:36:03 UTC, Paul Backus wrote:
 Someone has created bindings for a C library and published them 
 on
 code.dlang.org. Because they were written prior to DIP 1028, 
 the author assumed
 that  system was the default, and didn't bother to add explicit 
 annotations to
  system functions. Their code looks like this:

 --- clibrary.d

     void monkey_around(...); // assumed  system-by-default

 ---

 Months or years later, I decide to write a D program that makes 
 use of these
 bindings. By then,  safe-by-default has been fully implemented. 
 I add
 `clibrary` as a dependency to my Dub project and write the 
 following code:

 --- app.d

     import clibrary;

     void main() //  safe-by-default
     {
         /* ... code ... */

         monkey_around(...);

         /* ... more code ... */
     }

 ---

 My program compiles with no errors...and then corrupts memory 
 at run-time, even
 though every line of code I've written is  safe. Oops.

 This is the nightmare scenario that people are worried about: 
 safety violations
 being introduced *silently* into existing, correct D code.
Honest question: What is the use case for an absolutely-positively-has-to-be-safe program that calls C code? Why would anyone ever do that? C is not and will never be a safe language. "Someone looked at that blob of horrendous C code and thinks it's safe" does not inspire confidence. Why not rewrite the code in D (or Rust or Haskell or whatever) if safety is that critical?
May 22 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 22 May 2020 at 13:58:14 UTC, bachmeier wrote:
 On Friday, 22 May 2020 at 03:36:03 UTC, Paul Backus wrote:
 This is the nightmare scenario that people are worried about: 
 safety violations
 being introduced *silently* into existing, correct D code.
Honest question: What is the use case for an absolutely-positively-has-to-be-safe program that calls C code? Why would anyone ever do that? C is not and will never be a safe language. "Someone looked at that blob of horrendous C code and thinks it's safe" does not inspire confidence. Why not rewrite the code in D (or Rust or Haskell or whatever) if safety is that critical?
The problem isn't that safety is critical, it's that the D compiler is lying to me about the safety of my code. If the compiler was honest and told me that my code was unsafe, I'd be able to make an informed decision to either (a) accept the lack of safety, or (b) do the additional work needed to make it safe. As-is, DIP 1028 takes that choice away from me, and I am forced to accept a lack of safety whether I want to or not. At that point, why not use C?
May 22 2020
next sibling parent reply bachmeier <no spam.net> writes:
On Friday, 22 May 2020 at 14:13:20 UTC, Paul Backus wrote:
 On Friday, 22 May 2020 at 13:58:14 UTC, bachmeier wrote:
 On Friday, 22 May 2020 at 03:36:03 UTC, Paul Backus wrote:
 This is the nightmare scenario that people are worried about: 
 safety violations
 being introduced *silently* into existing, correct D code.
Honest question: What is the use case for an absolutely-positively-has-to-be-safe program that calls C code? Why would anyone ever do that? C is not and will never be a safe language. "Someone looked at that blob of horrendous C code and thinks it's safe" does not inspire confidence. Why not rewrite the code in D (or Rust or Haskell or whatever) if safety is that critical?
The problem isn't that safety is critical, it's that the D compiler is lying to me about the safety of my code. If the compiler was honest and told me that my code was unsafe, I'd be able to make an informed decision to either (a) accept the lack of safety, or (b) do the additional work needed to make it safe. As-is, DIP 1028 takes that choice away from me, and I am forced to accept a lack of safety whether I want to or not. At that point, why not use C?
If you're compiling a program that calls into C, you know that's what you're doing, so you know you've given up any guarantees of safety. Back in the early days of Rust, someone made a comment along the lines of Rust being a language to rewrite C code in, not to use to call C code. D has, to this point, been a language designed to interoperate easily with C code. I personally don't care about safety guarantees at all. It's not obvious to me that guaranteed safety at all costs is the way to go - especially considering the degree to which that breaks from current behavior.
May 22 2020
parent Paul Backus <snarwin gmail.com> writes:
On Friday, 22 May 2020 at 14:43:27 UTC, bachmeier wrote:
 If you're compiling a program that calls into C, you know 
 that's what you're doing, so you know you've given up any 
 guarantees of safety.
The entire problem is that with DIP 1028, I *don't* know what I'm doing, because the compiler will silently allow me to write safe D code that calls into unsafe C code. Keep in mind that none of this requires *me* to write the call to C directly. It could be buried several layers deep in a dependency, and merely cause a D function I call to be incorrectly inferred as safe.
May 22 2020
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 May 2020 at 14:13:20 UTC, Paul Backus wrote:
 On Friday, 22 May 2020 at 13:58:14 UTC, bachmeier wrote:
 On Friday, 22 May 2020 at 03:36:03 UTC, Paul Backus wrote:
 This is the nightmare scenario that people are worried about: 
 safety violations
 being introduced *silently* into existing, correct D code.
Honest question: What is the use case for an absolutely-positively-has-to-be-safe program that calls C code? Why would anyone ever do that? C is not and will never be a safe language. "Someone looked at that blob of horrendous C code and thinks it's safe" does not inspire confidence. Why not rewrite the code in D (or Rust or Haskell or whatever) if safety is that critical?
The problem isn't that safety is critical, it's that the D compiler is lying to me about the safety of my code.
I understand your argument, especially since I wrote nearly identical words weeks ago. For me, what it boils down to is this: currently the compiler isn't doing what it could because of the system default. I work on a D codebase that can't be called from a safe unittest and we could (probably are) be hiding bugs due to this. Flipping the default will cause more incorrect code to fail to compile. Yes, there's a cost, which is carefully vetting extern(C) and extern(C++) declarations. The decision came down to finding this an acceptable trade-off.
May 22 2020
next sibling parent ag0aep6g <anonymous example.com> writes:
On 22.05.20 19:12, Atila Neves wrote:
 Flipping the default will cause more incorrect code to fail to compile. 
 Yes, there's a cost, which is carefully vetting extern(C) and 
 extern(C++) declarations. The decision came down to finding this an 
 acceptable trade-off.
You can still carefully vet the extern declarations if DMD requires an explicit safety attribute on them. You can do it better even, because DMD will tell you where they are.
May 22 2020
prev sibling next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 22 May 2020 at 17:12:47 UTC, Atila Neves wrote:
 [..]
 Yes, there's a cost, which is carefully vetting extern(C) and 
 extern(C++) declarations. The decision came down to finding 
 this an acceptable trade-off.
How would you feel about a DIP that the only thing it did was assume all non-extern (D) code was implciti trusted? And how about if all such code suddenly became safe without any vetting by developers and not even a compiler switch to revert to the old behavior? As evidenced by the community outrage, no one but Walter and (maybe) you are convinced that safe-by-default on function bodies should imply safe on non-extern(D) function declarations. So how about a compromise? We accept a modified version of DIP1028 in which safe-by-default applies only to function definitions and extern(D) function declarations. And then there's a separate compiler switch that changes non-extern (D) function declarations to be trusted? If you like this "feature" you're free to use it on your personal projects, but please don't force it on everyone who wants safe to mean anything meaningful.
May 23 2020
prev sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 22 May 2020 at 17:12:47 UTC, Atila Neves wrote:
 [..]
 Yes, there's a cost, which is carefully vetting extern(C) and 
 extern(C++) declarations. The decision came down to finding 
 this an acceptable trade-off.
How would you feel about a DIP that the only thing it did was make all non-extern (D) code implicitly trusted? And how about if all such code suddenly became safe without any vetting by developers and not even a compiler switch to revert to the old behavior? Until DIP1028, putting trusted: at the start of a module has always been considered bad practice and rightfully forbidden in libraries with a higher quality bar, such as phobos. But at least doing so is honest: you as an author are admitting that you don't have time to take care of safe-ty for now, but it is likely ok to assume that the rest of your project is safe (well at least the safe functions) *modulo* the trusted parts. That way, later you can come back and search for trusted and address those issues one by one. As evidenced by the community outrage, no one but Walter and (maybe) you are convinced that safe-by-default on function bodies should imply safe on non-extern(D) function declarations. --- So how about a compromise? 1. We accept a modified version of DIP1028 in which safe-by-default applies only to function definitions and extern(D) function declarations. 2. We add a separate compiler switch that changes non-extern (D) function declarations to be trusted. If you enable 1. and 2. you get the current version of DIP1028, but at least corrected, so that extern(C) code trusted, and not safe. If you like this "feature" (2.) you're free to use it on your personal projects, but please don't force it on everyone who wants safe to be something meaningful.
May 23 2020
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 22.05.20 15:58, bachmeier wrote:
 ...
 
 Honest question: What is the use case for an 
 absolutely-positively-has-to-be-safe program that calls C code? Why 
 would anyone ever do that? C is not and will never be a safe language. 
 "Someone looked at that blob of horrendous C code and thinks it's safe" 
 does not inspire confidence. Why not rewrite the code in D (or Rust or 
 Haskell or whatever) if safety is that critical?
Honesty is what's critical. The annotations should mean what they are advertised to mean and making those meanings as simple as possible makes them easier to explain. As things stand, safe can mean that someone accidentally or historically did not annotate an extern(C) prototype and an unsafe API a few calls up was ultimately exposed with the safe attribute because the compiler never complained. How would you feel if you never intended to expose a safe interface, but someone imported your library after determining it contained no trusted annotations (following the advice of articles on safe), relied on the safe annotation and then had weird sporadic memory corruption issues in production that took them months to ultimately trace back to your library? Would you feel responsible or would you rather put the blame on Walter?
May 22 2020
parent reply bachmeier <no spam.net> writes:
On Friday, 22 May 2020 at 14:38:09 UTC, Timon Gehr wrote:
 On 22.05.20 15:58, bachmeier wrote:
 ...
 
 Honest question: What is the use case for an 
 absolutely-positively-has-to-be-safe program that calls C 
 code? Why would anyone ever do that? C is not and will never 
 be a safe language. "Someone looked at that blob of horrendous 
 C code and thinks it's safe" does not inspire confidence. Why 
 not rewrite the code in D (or Rust or Haskell or whatever) if 
 safety is that critical?
Honesty is what's critical. The annotations should mean what they are advertised to mean and making those meanings as simple as possible makes them easier to explain. As things stand, safe can mean that someone accidentally or historically did not annotate an extern(C) prototype and an unsafe API a few calls up was ultimately exposed with the safe attribute because the compiler never complained.
In my opinion, the only advantage of safe is that the compiler has checked the code and determines that it only does things considered safe. I don't see that marking an extern(C) function trusted buys you anything, at least not until you can provide a compiler guarantee for arbitrary C code.
May 22 2020
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 22.05.20 16:49, bachmeier wrote:
 On Friday, 22 May 2020 at 14:38:09 UTC, Timon Gehr wrote:
 On 22.05.20 15:58, bachmeier wrote:
 ...

 Honest question: What is the use case for an 
 absolutely-positively-has-to-be-safe program that calls C code? Why 
 would anyone ever do that? C is not and will never be a safe 
 language. "Someone looked at that blob of horrendous C code and 
 thinks it's safe" does not inspire confidence. Why not rewrite the 
 code in D (or Rust or Haskell or whatever) if safety is that critical?
Honesty is what's critical. The annotations should mean what they are advertised to mean and making those meanings as simple as possible makes them easier to explain. As things stand, safe can mean that someone accidentally or historically did not annotate an extern(C) prototype and an unsafe API a few calls up was ultimately exposed with the safe attribute because the compiler never complained.
In my opinion, the only advantage of safe is that the compiler has checked the code and determines that it only does things considered safe.
Wrong, but let's roll with that.
 I don't see that marking an extern(C) function  trusted buys you 
 anything, at least not until you can provide a compiler guarantee for 
 arbitrary C code.
It buys you the ability to call that function from safe code. Clearly you can't mark it safe because the compiler has not checked it.
May 22 2020
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, May 22, 2020 9:29:03 AM MDT Timon Gehr via Digitalmars-d-announce 
wrote:
 On 22.05.20 16:49, bachmeier wrote:
 I don't see that marking an extern(C) function  trusted buys you
 anything, at least not until you can provide a compiler guarantee for
 arbitrary C code.
It buys you the ability to call that function from safe code. Clearly you can't mark it safe because the compiler has not checked it.
Yes, and trusted in general buys you the ability to segregate and find code that is doing stuff that the compiler was unable to prove was safe. Then when there's a bug related to memory safety, you know where to look. C functions cannot be safe, because they haven't been vetted by the compiler. So, either what they're doing is actually memory safe (and it's fine to mark whem with trusted), or what it's doing is memory safe if used correctly, and the calling code needs to be marked trusted when using it correctly. Either way, the code with potential memory safety issues is segregated, and so when inevitably you do run into a memory safety bug, you know that it's either the trusted code or the system code that it calls which needs to be fixed. Of course, with this DIP, safe no longer truly means that the code was vetted by safety by the compiler but rather that all of the code that the compiler can see has been vetted for safety, meaning that when there is an safety bug, you must look not only for trusted code, but you must look for non-extern(D) declarations which have been implictly treated as trusted by the compiler. safe still has value (and may even provide more value in that it will be used more often), but it provides much weaker guarantees in the process. - Jonathan M Davis
May 27 2020
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 21, 2020 at 06:22:19PM -0700, Walter Bright via
Digitalmars-d-announce wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.
[...] Thank you. This makes your position clear, even if I don't entirely agree with it. On that note, though: this really should have been part of the DIP acceptance announcement, not added as an afterthought. Otherwise people will perceive (correctly or not) the DIP as being accepted without acknowledging any of the issues raised. This summary is exactly what would have defused a lot of the frustrations raised here. For the future may I recommend making it a point to include such a summary with the announcement of each DIP accepted/rejected. T -- Nobody is perfect. I am Nobody. -- pepoluan, GKC forum
May 21 2020
parent Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 May 2020 at 03:57:22 UTC, H. S. Teoh wrote:
 On Thu, May 21, 2020 at 06:22:19PM -0700, Walter Bright via 
 Digitalmars-d-announce wrote:
 I have made these points before, but I'll summarize them here 
 for convenient referral.
[...] Thank you. This makes your position clear, even if I don't entirely agree with it. On that note, though: this really should have been part of the DIP acceptance announcement, not added as an afterthought. Otherwise people will perceive (correctly or not) the DIP as being accepted without acknowledging any of the issues raised. This summary is exactly what would have defused a lot of the frustrations raised here. For the future may I recommend making it a point to include such a summary with the announcement of each DIP accepted/rejected. T
This is a very good point, and I think that in the future Walter and I should attach an explanation of why a DIP got accepted or not for transparency and communication reasons.
May 22 2020
prev sibling next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.

 [...]
Thank's for the reasoning, that should be added to the DIP acceptance since the beginning. Stated that we need to live with that, I'm asking you a simple question: What's the "official" definition of the safe advantages that D offers as a language? Sometime we need to explain that to customers asking for details about the code we wrote for them. I would love to have a good insight detailing the promises that the language is committed to hold, maybe trying to give me a vision on the future ... since from now on I will stop using "mechanical verification" at all as selling argument.
May 21 2020
prev sibling next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.
<big snip> of material indicating, among other things, that even really good programmers can screw up when it comes to language design now and then. Ahem. Continuing...
 How does this relate to safe by default?

 Consider the common (because that's how D started out) case of:

 ----- clibrary.d --------

     T massage_data(... many parameters ...);
     ... 200 more such declarations ...

 ----- app.d ----------

     import clibrary;

     void useClibrary( ... parameters ...) {
         massage_data(parameters);
     }

 ---------------------

 This code, today, does not use annotations and it works. It's 
 been working
 for a long time. Now, we implement the amendment, and it stops 
 compiling
 because useClibrary is  safe and massage_data is  system. The 
 user is faced
 with the following alternatives:

 1. Go through 200 functions in clibrary.d and determine which 
 are  safe
 and which are  system. This is what we want them to do. We try 
 to motivate
 this with compiler error messages. Unfortunately, this is both 
 tedious and
 thoroughly impractical, as our poor user Will Not Know which 
 are safe and
 which are system. We can correctly annotate core.stdc.stdio 
 because I know
 those functions intimately. This is not true for other system C 
 APIs, and
 even less true for some third party C library we're trying to 
 interface to.
Agree completely. Annotating C code correctly can be very difficult. This argues against defaulting such code to safe.
 2. Annotate useClibrary() as  trusted or  system. While easier, 
 this causes
 all benefits to  safe by default to be lost.
No. The benefit of safe is actually having machine checkable safety.
 3. Wrap the call to massage_data() with:

     ()  trusted { massage_data(parameters); } ();

 If there are a lot of calls to clibrary, this is going to look 
 pretty awful.
 Nobody likes writing or reading such ugly code. It's ok here 
 and there, but
 not as a general thing.
Yeah, this is ugly.
 4. Edit clibrary.d and make the first line:

      safe:

 I submit that, just like with Java, Option 4 is what people 
 will reach for,
 nearly every time. I've had some private conversations where 
 people admitted
 this was what they'd do. People who knew it was wrong to do 
 that.
The DIP gives us 4), the worst case, by default. Anywhere and everywhere a C lib shows up it gives us 4), automatically.
 If it's  safe by default, and then someone chooses to annotate 
 it with  system
 here and there, I'd feel a lot more confident about the 
 accuracy of the code
 annotations than if it just had  safe: at the top. At least 
 they tried.
So safe is to be based on a feeling? At least they *tried*? This is a key point of disagreement. Machine checked safety is something. Human hand waving safety is not. It is bizarre that no one has been able to get you to see the contradiction with your earlier writings on safety (big fan there btw). C's biggest blunder? Yeah, well, all the bugs engendered by that blunder and many many more are now to be considered safe by default. What the heck? Finally, Atila, can you shed any light on this? Where do you stand? Why did it seem like a good idea to withhold your veto?
May 22 2020
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 22.05.20 03:22, Walter Bright wrote:
 This is Obviously A Good Idea. Why would I oppose it?
 
 1. I've been hittin' the crack pipe again.
 2. I was secretly convinced, but wanted to save face.
 3. I make decisions based on consultation with my astrologer.
 4. I am evil.
The sarcasm is not appreciated. [...]> Now, we implement the amendment, and it stops compiling
 because useClibrary is  safe and massage_data is  system. The user is faced
 with the following alternatives:
 
 1. Go through 200 functions in clibrary.d and determine which are  safe
 and which are  system. This is what we want them to do. We try to motivate
 this with compiler error messages. Unfortunately, this is both tedious and
 thoroughly impractical, as our poor user Will Not Know which are safe and
 which are system. We can correctly annotate core.stdc.stdio because I know
 those functions intimately. This is not true for other system C APIs, and
 even less true for some third party C library we're trying to interface to.
 
 2. Annotate useClibrary() as  trusted or  system. While easier, this causes
 all benefits to  safe by default to be lost.
In the past you have said repeatedly that the point of safe is to be "100% mechanically checkable, not correct by convention" [1]. In other words, it's supposed to be watertight. Now you're saying that safe can have "benefits" without being watertight. You're even widening an existing hole in safe to make those benefits more accessible. If you have actually given up on a watertight safe, that's disappointing for those of us who were with you on a watertight safe.
 3. Wrap the call to massage_data() with:
 
      ()  trusted { massage_data(parameters); } ();
 
 If there are a lot of calls to clibrary, this is going to look pretty 
 awful.
 Nobody likes writing or reading such ugly code. It's ok here and there, but
 not as a general thing.
Let me get this straight. As a general rule we have: Unsafe stuff is "supposed to be inconvenient" [2]. But unsafe calls to C functions must be convenient, because they're so common. So we add an exception to the rule. And that exception is cool. But making an exception in safe-by-default for `extern (C)` prototypes is not cool. Because exceptions are bad. Makes sense ... not.
 4. Edit clibrary.d and make the first line:
 
       safe:
 
 I submit that, just like with Java, Option 4 is what people will reach for,
 nearly every time. I've had some private conversations where people 
 admitted
 this was what they'd do. People who knew it was wrong to do that.
Unlike the catch-and-ignore of your Java analogy, ` safe:` does no harm. Or rather, it does the same amount of harm that your safe-by-default `extern (C)` declarations do. But with the amendment, the idiot programmer at least has to pull the trigger to shoot their foot off. You're letting the compiler do it for them. That safe even applies to `extern (C)` prototypes is its own issue.
 If it's  safe by default, and then someone chooses to annotate it with 
  system
 here and there, I'd feel a lot more confident about the accuracy of the 
 code
 annotations than if it just had  safe: at the top. At least they tried.
The morale of your Java analogy is that people will find and use the easy way out. No one is going to add system to the declarations if the code already compiles with safe-by-default.
 What is actually accomplished with this amendment if it was implemented?
 
 1. Adds a funky, special case rule. It's better to have simple, easily
 understood rules than ones with special cases offering little improvement.
 
 2. Existing, working code breaks.
You can't forgo breakage when making safe the default. Any template that is inferred as system will throw a wrench in your suddenly- safe `main`.
 3. The most likely code fixes are to just make it compile, absolutely
 nothing safety-wise is improved. The added annotations will be a fraud.
And your solution is to let the compiler add the frauds.
 D should not encourage "greenwashing" practices like the Java
 exception specification engendered. The compiler cannot vet the accuracy
 of bodyless C functions, and we'll just have to live with that. The 
 proposed
 amendment does not fix that.
The proposed amendment at least makes the greenwashing visible. With the DIP as it is, the greenwashing is still there. The compiler just does it for you. Imagine Java doing catch-and-ignore by itself, just so that the idiot programmer doesn't have to do it. It would be better if the greenwashing wasn't even possible. I.e., don't allow safe on `extern (C)` prototypes. If that's what you're going for, great. But you're not, are you? By the way, I don't think you have acknowledged Steven's idea [3] at all, yet. Maybe you've missed it. It's completely different from the "Obviously A Good Idea" amendment you've discussed here. [1] https://issues.dlang.org/show_bug.cgi?id=14125#c26 [2] https://forum.dlang.org/post/r5hnjo$1j40$1 digitalmars.com [3] https://forum.dlang.org/post/r6kvm4$1vq5$1 digitalmars.com
May 22 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/22/20 4:24 AM, ag0aep6g wrote:
 By the way, I don't think you have acknowledged Steven's idea [3] at 
 all, yet.
Yes, he did. With more nitpicking as a rebuttal, but I'm not really interested in having more discussions about this DIP or that proposal. https://forum.dlang.org/post/ra7958$2r8v$1 digitalmars.com -Steve
May 22 2020
prev sibling next sibling parent aliak <something something.com> writes:
First, thank you for the explanation! I have a few observations 
though, mainly that the exception analogy feels like more an 
argument against the DIP than for it. And I also have an 
alternative proposal that might be considered?

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 This, of course, was incredibly tedious and annoying. So what 
 happened?

 1. Functions would just do a try-catch(Throwable) and ignore 
 the exceptions.
 Oops.

 2. The Exception Specifications became simply `throws 
 Throwable`. Oops.
This is what happens in swift these days: func g() /*throws*/ { do { try f() } catch {} } But, it's very rare. And passing something like that in code review is very difficult and, almost always, the reviewer will ask for comments so other people will know why the laziness (if it even really was laziness, sometimes it's the right thing to do). Of course the one big difference between exception specifications and safe is the implied importance; safe is compiler-enforced and has the user's trust, exception specifications are human guess work of what are the important exceptions and how varied they need to be - it's API design - very hard to get right. The exception specification becoming "throws Exception", while highlights the failure of exception specifications, does so with a non-dangerous resolution. I.e. you still need to mark your function as throwing, and you still need to at least "try" to catch the exception. Even a forced "catch(Exception)" is better than nothing and shows where problems can be and encourages scrutiny.
 How does this relate to safe by default?
That people will use the most convenient solution to get their work to compile if the juice is not worth the squeeze?
 1. Go through...

 2. Annotate...safe by default to be lost.

 3. Wrap the call...

 4. Edit clibrary.d and make the first line:

      safe:
5. Edit clibrary.d and make the first line trusted: ? Which brings me to the proposal of making safe illegal on function prototypes. Reason being that safe stays compiler checked. trusted maintains it's path and value in hunting down memory corruption. extern(Anything) functions cannot lie anymore about being compiler-checked.
 I submit that, just like with Java, Option 4 is what people 
 will reach for,
 nearly every time. I've had some private conversations where 
 people admitted
 this was what they'd do. People who knew it was wrong to do 
 that.

 If it's  safe by default, and then someone chooses to annotate 
 it with  system
Which is the rare case in this scenario right? And 1) there's no way to tell if someone made an effort in the case where none are marked system. 2) this is also only for the case of C library bindings (reason being that most of my code, and also others' code I've seen, has extern(C) functions splattered around various D files and not in a dedicated clibrary.d file - is this just me?)
 here and there, I'd feel a lot more confident about the 
 accuracy of the code
 annotations than if it just had  safe: at the top. At least 
 they tried.

 What is actually accomplished with this amendment if it was 
 implemented?

 1. Adds a funky, special case rule. It's better to have simple, 
 easily
 understood rules than ones with special cases offering little 
 improvement.

 2. Existing, working code breaks.

 3. The most likely code fixes are to just make it compile, 
 absolutely
 nothing safety-wise is improved. The added annotations will be 
 a fraud.

 D should not encourage "greenwashing" practices like the Java
 exception specification engendered. The compiler cannot vet the 
 accuracy
 of bodyless C functions, and we'll just have to live with that. 
 The proposed
 amendment does not fix that.

 And so, I did not incorporate the proposed amendment to the 
 Safe by Default
 DIP.
I think there's a failure to understand why explicit greenwashing via a deliberate (possibly code reviewed) addition of safe: at the top is worse than a global implicit greenwashing by the compiler, ignoring the rare "feel-good-but-can't-be-sure" scenario above where some functions are marked as system, and especially given the premise that programmers are lazy. Also ... error on safe prototypes?
May 22 2020
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.
Thanks Walter. I really appreciate you taking the time to do this, as it's obviously no fun to be getting a big tide of negativity in this way.
 This is Obviously A Good Idea. Why would I oppose it?

 1. I've been hittin' the crack pipe again.
 2. I was secretly convinced, but wanted to save face.
 3. I make decisions based on consultation with my astrologer.
 4. I am evil.
... wait, we don't ALL make decision like that? :-) In seriousness: I really, really wish that everyone throwing around personal attacks would take a moment to stop and think about the consequences of pursuing this kind of discourse: it distracts from the technical, practical and organizational issues at hand, it creates a worse mood than exists already, and generally makes it harder to have a constructive conversation around the core disagreements here. It is perfectly possible to critique the decision taken, or the way in which it was taken, or to voice personal frustrations, without casting aspersions on the decision-maker. With the rationale laid out clearly as it is here, I do have some responses in mind. But before sharing them, I'd like to know whether that would be useful right now: I've no wish to just press for a re-hashing of conversations that have already been had.
May 22 2020
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 23/05/2020 12:19 AM, Joseph Rushton Wakeling wrote:
 With the rationale laid out clearly as it is here, I do have some 
 responses in mind.  But before sharing them, I'd like to know whether 
 that would be useful right now: I've no wish to just press for a 
 re-hashing of conversations that have already been had.
No. This wasn't the first and certainly won't be the last time we as a community have been very unhappy with how Walter conducts himself with his DIP's. Although it seems an improvement has been made to how he needs to respond to the DIP assessment. It should also include a statement from Atila as well given his position.
May 22 2020
next sibling parent reply matheus <matheus gmail.com> writes:
On Friday, 22 May 2020 at 12:28:56 UTC, rikki cattermole wrote:
 Although it seems an improvement has been made to how he needs 
 to respond to the DIP assessment. It should also include a 
 statement from Atila as well given his position.
One thing that need to be clear and for I read it was not remotely answered from Walter is why this DIP process and discussion exists if in the end like it or not it will be incorporated. As an end user, I'd like to know if this language will be guided by community or one person, because it seems the "democracy" is very shallow right now. And again why waste time with this process plus 2 rounds of discussion? I mean just do it and tell in this announcement section about the feature. Matheus.
May 22 2020
next sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Friday, 22 May 2020 at 12:47:04 UTC, matheus wrote:
 On Friday, 22 May 2020 at 12:28:56 UTC, rikki cattermole wrote:
 Although it seems an improvement has been made to how he needs 
 to respond to the DIP assessment. It should also include a 
 statement from Atila as well given his position.
One thing that need to be clear and for I read it was not remotely answered from Walter is why this DIP process and discussion exists if in the end like it or not it will be incorporated. As an end user, I'd like to know if this language will be guided by community or one person, because it seems the "democracy" is very shallow right now. And again why waste time with this process plus 2 rounds of discussion? I mean just do it and tell in this announcement section about the feature. Matheus.
There is no democracy. The decisions are simply made by Walter and Atila. From my understanding, the purpose of the DIP process is to improve DIPs and find any potential flaws before a final decision ist made by the two language maintainers. However, how much of the review the maintainers take into account is their own decision. In theory, this results in much higher quality DIPs and the chance of a DIP that actually makes it to the formal assessment stage to be accepted is a lot higher. It also seems to work quite well when the DIP author is not one of the language maintainers. From my experience, the quality of those DIPs really has gone up by a lot. There is also a lively discussion about each DIP and the authors are actively participating. It is in their own interest to do so because getting feedback and addressing it increases the quality of the proposal which in turn increases the chance of the DIP being accepted. If the author is one the language maintainers, this doesn't seem to work that well though. There is simply less motivation for them to do all of that because in the end, they can make the decision by themselves anyways. I am not saying that the language maintainers do this on purpose, it is simply a psychological effect. And the result can be seen very clearly when comparing review threads for DIPs from the language authors with review threads for DIPs from others. In defense of the language authors, it has to be stated that the situation has improved quite a bit over time. Examples of this are how DIP1017 was handled or the fact that Walter now gave a reasoning for his decision regarding DIP1028 (even if only after being asked to do so). But unfortunately, sometimes it still really feels like DIPs from the language maintainers and DIPs from others are handled quite differently by the maintainers.
May 22 2020
parent matheus <matheus gmail.com> writes:
On Friday, 22 May 2020 at 13:16:50 UTC, Johannes Loher wrote:
 ... But unfortunately, sometimes it still really feels like 
 DIPs from the language maintainers and DIPs from others are 
 handled quite differently by the maintainers.
That's was my point about "democracy", maybe not a right word. In fact I really think that in the end someone will need to decide (for good or bad), but it's not the first time I read about this nuances between DIPs created by maintainers and others like you said. I think for such small community, there should be a vote system for those who are actively contributing (I am NOT in this niche by the way) to guide the language. It would be better than a rounds of discussion without any meaning. Matheus.
May 22 2020
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Friday, 22 May 2020 at 12:47:04 UTC, matheus wrote:
 As an end user, I'd like to know if this language will be 
 guided by community or one person, because it seems the 
 "democracy" is very shallow right now.

 And again why waste time with this process plus 2 rounds of 
 discussion?

 I mean just do it and tell in this announcement section about 
 the feature.
The DIP review process is not intended for community approval or rejection of DIPs. It's not a democratic voting process. It's intended to elicit community feedback to enhance the DIP under review (the Feedback Threead) and to allow the airing of opinions (the Discussion Thread). All DIP authors have the freedom to incorporate suggestions into their DIP or not, and Walter and Atila make the decision to accept or reject. If you look at the history of Walter's DIPs, they *do* take the opinions into consideration even when he is the author. Several of his previous DIPs have been withdrawn or rejected. If a popular DIP is rejected, it means neither of them were convinced by opinion to accept it. And, as in the case for this DIP, if an unpopular DIP is accepted, it means they were not persuaded by the arguments against it. From my perspective, the process is working as intended, despite the comments to the contrary in this thread. You either convince a DIP author to modify his DIP, or you don't. You either persuade Walter and Atila to accept or reject it, or you don't.
May 22 2020
next sibling parent reply bachmeier <no spam.net> writes:
On Friday, 22 May 2020 at 13:57:27 UTC, Mike Parker wrote:
 On Friday, 22 May 2020 at 12:47:04 UTC, matheus wrote:
 As an end user, I'd like to know if this language will be 
 guided by community or one person, because it seems the 
 "democracy" is very shallow right now.

 And again why waste time with this process plus 2 rounds of 
 discussion?

 I mean just do it and tell in this announcement section about 
 the feature.
The DIP review process is not intended for community approval or rejection of DIPs. It's not a democratic voting process. It's intended to elicit community feedback to enhance the DIP under review (the Feedback Threead) and to allow the airing of opinions (the Discussion Thread). All DIP authors have the freedom to incorporate suggestions into their DIP or not, and Walter and Atila make the decision to accept or reject. If you look at the history of Walter's DIPs, they *do* take the opinions into consideration even when he is the author. Several of his previous DIPs have been withdrawn or rejected. If a popular DIP is rejected, it means neither of them were convinced by opinion to accept it. And, as in the case for this DIP, if an unpopular DIP is accepted, it means they were not persuaded by the arguments against it. From my perspective, the process is working as intended, despite the comments to the contrary in this thread. You either convince a DIP author to modify his DIP, or you don't. You either persuade Walter and Atila to accept or reject it, or you don't.
I think the source of the problem is that Walter's DIPs require the community to prove that Walter's proposal is so bad that he needs to reject it. Anyone else's proposal has to prove that it's worthy of being added to the language. There's a big perceived gap between those two. As I've said many times, it's odd for someone to judge his own DIPs, and as someone that is an academic administrator and runs a high-profile journal, I can say this type of practice is not the norm in those areas because it doesn't lead to good decisions.
May 22 2020
next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Friday, 22 May 2020 at 14:37:04 UTC, bachmeier wrote:
 I think the source of the problem is that Walter's DIPs require 
 the community to prove that Walter's proposal is so bad that he 
 needs to reject it. Anyone else's proposal has to prove that 
 it's worthy of being added to the language. There's a big 
 perceived gap between those two. As I've said many times, it's 
 odd for someone to judge his own DIPs, ...
When either Walter or Atila author a DIP, their spot in the "convince both or face rejection" duopoly could be taken by another person.
May 22 2020
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 May 2020 at 14:37:04 UTC, bachmeier wrote:
 On Friday, 22 May 2020 at 13:57:27 UTC, Mike Parker wrote:
 [...]
I think the source of the problem is that Walter's DIPs require the community to prove that Walter's proposal is so bad that he needs to reject it. Anyone else's proposal has to prove that it's worthy of being added to the language. There's a big perceived gap between those two. As I've said many times, it's odd for someone to judge his own DIPs, and as someone that is an academic administrator and runs a high-profile journal, I can say this type of practice is not the norm in those areas because it doesn't lead to good decisions.
I could have rejected it, and nearly did.
May 22 2020
prev sibling next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Friday, 22 May 2020 at 13:57:27 UTC, Mike Parker wrote:
 On Friday, 22 May 2020 at 12:47:04 UTC, matheus wrote:
 As an end user, I'd like to know if this language will be 
 guided by community or one person, because it seems the 
 "democracy" is very shallow right now.

 And again why waste time with this process plus 2 rounds of 
 discussion?

 I mean just do it and tell in this announcement section about 
 the feature.
The DIP review process is not intended for community approval or rejection of DIPs. It's not a democratic voting process. It's intended to elicit community feedback to enhance the DIP under review (the Feedback Threead) and to allow the airing of opinions (the Discussion Thread). All DIP authors have the freedom to incorporate suggestions into their DIP or not, and Walter and Atila make the decision to accept or reject. If you look at the history of Walter's DIPs, they *do* take the opinions into consideration even when he is the author. Several of his previous DIPs have been withdrawn or rejected.
No dispute here.
 If a popular DIP is rejected, it means neither of them were 
 convinced by opinion to accept it. And, as in the case for this 
 DIP, if an unpopular DIP is accepted, it means they were not 
 persuaded by the arguments against it.
Not exactly. IIUC logical "and" is required for approval. One of the two might have been persuaded to accept. "not and", a single disapproval, suffices for rejection. I believe this is a useful weighting.
 From my perspective, the process is working as intended, 
 despite the comments to the contrary in this thread. You either 
 convince a DIP author to modify his DIP, or you don't. You 
 either persuade Walter and Atila to accept or reject it, or you 
 don't.
Again, IIUC, you either persuade Walter *or* Atila to reject or you don't. Apropos of which, what does Atila think of this DIP?
May 22 2020
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 22 May 2020 at 13:57:27 UTC, Mike Parker wrote:
 The DIP review process is not intended for community approval 
 or rejection of DIPs. It's not a democratic voting process.
The community needs to unite and fix this. It is an ineffectual process that leads to worse results for the language and further poisons the relationship between us. A small voting review committee solves these problems.
May 22 2020
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 May 2020 at 12:28:56 UTC, rikki cattermole wrote:
 On 23/05/2020 12:19 AM, Joseph Rushton Wakeling wrote:
 With the rationale laid out clearly as it is here, I do have 
 some responses in mind.  But before sharing them, I'd like to 
 know whether that would be useful right now: I've no wish to 
 just press for a re-hashing of conversations that have already 
 been had.
No. This wasn't the first and certainly won't be the last time we as a community have been very unhappy with how Walter conducts himself with his DIP's. Although it seems an improvement has been made to how he needs to respond to the DIP assessment. It should also include a statement from Atila as well given his position.
I'm going through posts in order, so apologies if I'm "ignoring" something that shows up later in the discussion. Personally and initially, I would have preferred it if non-extern(D) declarations were implicitly system. I understood Walter's argument about special cases and how they're bad, but the thought of them being safe made me feel, for lack of a better word, "icky". Then I talked to Walter and he pointed out that if those declarations were system, users would be prevented from calling them from now safe code. A regular user would probably just slap ` safe:` at the top of the bindings module anyway. Then I realised that I did exactly that with my libclang bindings: https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/clang/c/index.d#L254 "Worse", I made all functions `pure` and all parameters `in` as well for good measure. Why? I wanted to call them from my safe pure code with `const` arguments. My reasoning at the time was "I trust libclang". And so I was convinced that everything being safe is actually ok, especially because in real life, most C/C++ APIs aren't going to secretly corrupt your code. Then there's the fact that, today, there's no safety anyway calling anything because everything is system by default. And D source code won't be able to lie. Does that clear it up?
May 22 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 22 May 2020 at 17:07:37 UTC, Atila Neves wrote:
 And so I was convinced that everything being  safe is actually 
 ok, especially because in real life, most C/C++ APIs aren't 
 going to secretly corrupt your code.
I wrote up a thing about this earlier then deleted it because I didn't want to yell into the void, but now I kinda wish I kept it. Short version of my thoughts is yes, most C functions are implicitly trusted, but this depends heavily on using correct arguments and that's what trusted wrappers are supposed to ensure. Consider: char[16] buffer; snprintf(&buffer[0], buffer.length, "%s", data); btw fix this https://issues.dlang.org/show_bug.cgi?id=20853 But it is fair to say that usage is safe/trusted. However, you can just as well write: snprintf(&buffer[12], buffer.length...... or snprintf(&buffer[0], 156, ....); // oops my finger was misaligned on the keyboard and hit 56 without realizing it. or the size changed and this didn't etc This is what a trusted wrapper is supposed to be about - providing an interface that cannot be misused. int snprintf(char[] buf, const char* fmt, ...) { return c.snprintf(buf.ptr, buf.length, fmt, ...); } Well, lol, that's not exactly *correct* thanks to the possibility of format string mismatch, but it bounds checks both sides of the slice and becomes more appropriately trusted due to the parameters being checked too. We now actually have a trusted implementation with a safe interface - which is what trusted is SUPPOSED to mean.
May 22 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 22, 2020 at 05:26:03PM +0000, Adam D. Ruppe via
Digitalmars-d-announce wrote:
[...]
 int snprintf(char[] buf, const char* fmt, ...) {
    return c.snprintf(buf.ptr, buf.length, fmt, ...);
 }
 
 Well, lol, that's not exactly *correct* thanks to the possibility of
 format string mismatch, but it bounds checks both sides of the slice
 and becomes more appropriately trusted due to the parameters being
 checked too.
 
 We now actually have a trusted implementation with a safe interface -
 which is what  trusted is SUPPOSED to mean.
Ironically, it was Walter himself who was a staunch proponent of this. And now we're basically saying, every C API is safe by default unless you take the pains to realize that, oh noes it takes a pointer and a size and therefore passing the wrong size will cause buffer overflow yikes we better mark it system. Seems totally backwards to me. But anyway, thinking about this a little more, it occurs to me that what we're trying to do here is really not 100% watertight safe-ty, rather, what we're trying to do here is to maximize the surface area of compiler mechnical verification. So safe doesn't *really* mean "no memory corruption, guaranteed", it means "compiler will do its best to verify what it can, but what it can't, we just have to take on faith." Or, to put it more nicely, "this code is safe modulo the safety of calls to unverifiable C APIs". Which leads to the idea of having some way of marking a function as safe up to the safety of {a delegate parameter, a C interface, etc.}. A kind of conditional safe-ty. It has been a recurring need that keeps coming up, e.g., in relation to the recent DIP on lazy parameters. For example: // N.B.: hypothetical syntax int myFunc(int delegate(SomeObj obj) cb) conditionalSafe { SomeObj obj = ...; // &obj++; // error: pointer arithmetic not allowed in safe // systemFunc(); // error: cannot call systemFunc from safe return cb(obj); // OK: we are safe modulo safety of cb } void systemFunc() system {...} Then myFunc would become callable from safe code, provided the passed-in argument is also safe. The crucial point here is that while compiling myFunc, the compiler doesn't (need to) know the safe-ty of `cb`, it can just treat it as an opaque object that it assumes the safety of, while it verifies the rest of the function body. This is parallel to C functions being of unverifiable safety, so if extern(C) functions were somehow marked and treated as opaque objects of unknown safety, then the compiler can still verify the rest of the code and produce a certificate of safety (modulo the C APIs used). If we had a way of expressing conditional safety, it could be a way to salvage safe from this current situation. T -- Doubtless it is a good thing to have an open mind, but a truly open mind should be open at both ends, like the food-pipe, with the capacity for excretion as well as absorption. -- Northrop Frye
May 22 2020
parent Q. Schroll <qs.il.paperinik gmail.com> writes:
On Friday, 22 May 2020 at 18:14:12 UTC, H. S. Teoh wrote:
 Then myFunc would become callable from  safe code, provided the 
 passed-in argument is also  safe.

 The crucial point here is that while compiling myFunc, the 
 compiler doesn't (need to) know the  safe-ty of `cb`, it can 
 just treat it as an opaque object that it assumes the safety 
 of, while it verifies the rest of the function body.

 This is parallel to C functions being of unverifiable safety, 
 so if extern(C) functions were somehow marked and treated as 
 opaque objects of unknown safety, then the compiler can still 
 verify the rest of the code and produce a certificate of safety 
 (modulo the C APIs used).

 If we had a way of expressing conditional safety, it could be a 
 way to salvage  safe from this current situation.
I had this thought a million times, tried to explain it at two DConfs but no-one (as it appeared to me) got the point. The main difference is, the function calling `myFunc` "knows" whether the delegate/function pointer it used as an argument is safe or not. It's perfectly decidable. Personally, I'd even go that far and say, that's how all the function attributes (also pure, nothrow, nogc) are intended to work.
May 27 2020
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 23/05/2020 5:07 AM, Atila Neves wrote:
 On Friday, 22 May 2020 at 12:28:56 UTC, rikki cattermole wrote:
 On 23/05/2020 12:19 AM, Joseph Rushton Wakeling wrote:
 With the rationale laid out clearly as it is here, I do have some 
 responses in mind.  But before sharing them, I'd like to know whether 
 that would be useful right now: I've no wish to just press for a 
 re-hashing of conversations that have already been had.
No. This wasn't the first and certainly won't be the last time we as a community have been very unhappy with how Walter conducts himself with his DIP's. Although it seems an improvement has been made to how he needs to respond to the DIP assessment. It should also include a statement from Atila as well given his position.
I'm going through posts in order, so apologies if I'm "ignoring" something that shows up later in the discussion. Personally and initially, I would have preferred it if non-extern(D) declarations were implicitly system. I understood Walter's argument about special cases and how they're bad, but the thought of them being safe made me feel, for lack of a better word, "icky".
This is one of the issues I had a problem with. It is not about the linkage. The problem is solely does the compiler have the source to the function body to verify it?
 Then I talked to Walter and he pointed out that if those declarations 
 were  system, users would be prevented from calling them from now  safe 
 code. A regular user would probably just slap ` safe:` at the top of the 
 bindings module anyway. Then I realised that I did exactly that with my 
 libclang bindings:
 
 https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/
lang/c/index.d#L254 
That is a failure of the language that should be resolved. One of the arguments that has been brought up (although I don't remember if it made its way to the N.G.) is that if you don't have the body, can a function /even/ be safe?
 "Worse", I made all functions `pure` and all parameters `in` as well for 
 good measure. Why? I wanted to call them from my  safe pure code with 
 `const` arguments. My reasoning at the time was "I trust libclang".
You might, but that doesn't give the compiler the right to do so by default. This a decision for a skilled programmer to make.
 And so I was convinced that everything being  safe is actually ok, 
 especially because in real life, most C/C++ APIs aren't going to 
 secretly corrupt your code.
What you did was give up some security in favor of freedom and now you can't get certified to work on top secret projects (analogy).
 Then there's the fact that, today, there's no safety anyway calling 
 anything because everything is  system by default. And D source code 
 won't be able to lie.
 
 Does that clear it up?
No. We simply do not agree, nor do I expect for us to come to terms on it anytime soon. To me at least, this butchers safe/trusted/system into a system that is near useless for guarantees for an entire program. Sure its convenient for a single function, but absolutely useless in trying to solve the actual problem it was trying to solve in the first place.
May 22 2020
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 May 2020 at 17:33:11 UTC, rikki cattermole wrote:
 On 23/05/2020 5:07 AM, Atila Neves wrote:
 [...]
It is not about the linkage. The problem is solely does the compiler have the source to the function body to verify it?
That's what I meant, sorry for not making it clearer.
 [...]
That is a failure of the language that should be resolved.
And how do you suggest we fix it?
 One of the arguments that has been brought up (although I don't 
 remember if it made its way to the N.G.) is that if you don't 
 have the body, can a function /even/ be  safe?
Yes.
 [...]
You might, but that doesn't give the compiler the right to do so by default. This a decision for a skilled programmer to make.
They still can (and should). The DIP is about changing the default.
 [...]
No. We simply do not agree, nor do I expect for us to come to terms on it anytime soon.
I meant "did I explain myself well enough that now you understand where I'm coming from, even though you might not ultimately agree?".
May 22 2020
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 23/05/2020 5:40 AM, Atila Neves wrote:
 On Friday, 22 May 2020 at 17:33:11 UTC, rikki cattermole wrote:
 On 23/05/2020 5:07 AM, Atila Neves wrote:
 [...]
It is not about the linkage. The problem is solely does the compiler have the source to the function body to verify it?
That's what I meant, sorry for not making it clearer.
It kept being swapped about in the discussion thread, so I have been a little on edge over people using non-extern(D). Because linkage doesn't mean anything to anything but to cpu and linker in this discussion.
 [...]
That is a failure of the language that should be resolved.
And how do you suggest we fix it?
I don't know. It is a problem that needs to be explored in more detail. Adam had one idea that I convinced him to write up, but that didn't get very far when commented relating to propagation of attributes. Major changes like this, should have been discussed and RFC'd instead of going to straight to a DIP. That is effectively what happened.
 [...]
No. We simply do not agree, nor do I expect for us to come to terms on it anytime soon.
I meant "did I explain myself well enough that now you understand where I'm coming from, even though you might not ultimately agree?".
Yeah. ------------- I didn't look at the binding, since you described it well previously, but of note is that you used safe. What you should have used is trusted instead. If the programmer wants to slap trusted: at the top of the file, I have no problems with that. You checked it (i.e. its LLVM, so of course its fine!).
May 22 2020
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, May 22, 2020 12:09:16 PM MDT rikki cattermole via Digitalmars-d-
announce wrote:
 It kept being swapped about in the discussion thread, so I have been a
 little on edge over people using non-extern(D). Because linkage doesn't
 mean anything to anything but to cpu and linker in this discussion.
Except that the linker matters a great deal in this discussion with regards to extern(D) functions, because safe and trusted are part of the name mangling for extern(D) functions. That means that if an extern(D) function declaration's attributes do not match its definition, then you'll get a linker error. So, treating non-extern(D) function declarations as safe by default isn't necessarily a problem (though it would certainly work to just treat all function declarations as system by default rather than treating extern(D) function declarations differently). The cases where non-extern(D) function declarations weren't actually safe would be caught during the linking process. Sure, it would be nice if it were caught sooner, but you don't end up with them being invisibly treated safe when they're not like we're going to get with DIP 1028. However, of course, with non-extern(D) declarations, even if the function definition is actually written in D, the fact that the function body was checked is not transmitted via the linking process, because it doesn't end up in the name mangling. So, the linkage used has a huge impact on whether you can rely on the safe attribute on the function declaration actually meaning anything about whether the body was verified by the compiler. - Jonathan M Davis
May 27 2020
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 28 May 2020 at 01:14:43 UTC, Jonathan M Davis wrote:
 On Friday, May 22, 2020 12:09:16 PM MDT rikki cattermole via 
 Digitalmars-d- announce wrote:
 [...]
Except that the linker matters a great deal in this discussion with regards to extern(D) functions, because safe and trusted are part of the name mangling for extern(D) functions. That means that if an extern(D) function declaration's attributes do not match its definition, then you'll get a linker error. So, treating non-extern(D) function declarations as safe by default isn't necessarily a problem (though it would certainly work to just treat all function declarations as system by default rather than treating extern(D) function declarations differently). The cases where non-extern(D) function declarations weren't actually safe would be caught during the linking process. Sure, it would be nice if it were caught sooner, but you don't end up with them being invisibly treated safe when they're not like we're going to get with DIP 1028. [...]
I remember reading a suggestion that additional linker symbols be emitted to carry the attribute and possibly type information while leaving the ABI untouched. Was this found to be impractical?
May 27 2020
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 27, 2020 8:13:52 PM MDT Bruce Carneal via Digitalmars-d-
announce wrote:
 On Thursday, 28 May 2020 at 01:14:43 UTC, Jonathan M Davis wrote:
 On Friday, May 22, 2020 12:09:16 PM MDT rikki cattermole via

 Digitalmars-d- announce wrote:
 [...]
Except that the linker matters a great deal in this discussion with regards to extern(D) functions, because safe and trusted are part of the name mangling for extern(D) functions. That means that if an extern(D) function declaration's attributes do not match its definition, then you'll get a linker error. So, treating non-extern(D) function declarations as safe by default isn't necessarily a problem (though it would certainly work to just treat all function declarations as system by default rather than treating extern(D) function declarations differently). The cases where non-extern(D) function declarations weren't actually safe would be caught during the linking process. Sure, it would be nice if it were caught sooner, but you don't end up with them being invisibly treated safe when they're not like we're going to get with DIP 1028. [...]
I remember reading a suggestion that additional linker symbols be emitted to carry the attribute and possibly type information while leaving the ABI untouched. Was this found to be impractical?
Steven suggested something along those lines. I don't know how practical it would or wouldn't be, but I don't think that Walter even responded to the idea. Regardless, even if it worked perfectly, all such a solution would deal with would be the cases where you have a non-extern(D) function which is actually written in D and had its function body marked as safe (and thus was verified to be safe). Such functions definitely exist, but the main problem is really non-extern(D) functions which are actually written in C or C++ or whatnot. They won't have been checked for safety. Right now, such declarations are treated by the compiler as system by default, because it cannot check them for safety, whereas with DIP 1028, it will then treat them as safe in spite of the fact that it can't check them for safety simply because Walter thinks that it would be too exceptional to treat them differently from the general rule of safe by default and that if they're left as system, too many people will just slap trusted on their code as a whole to get the compiler to shut up. But regardless of whether DIP 1028 is the correct decision, the problem remains that your typical extern(C) function cannot be checked for safety by the compiler, because it was compiled by a C compiler. The question then is just how the D compiler should treat them, and that's the main point of contention. There may be solutions like Steven suggested which would deal with edge cases where the implementation is in D but the linkage isn't, but ultimately, they're just edge cases. - Jonathan M Davis
May 27 2020
parent Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 28 May 2020 at 02:39:40 UTC, Jonathan M Davis wrote:
 On Wednesday, May 27, 2020 8:13:52 PM MDT Bruce Carneal via 
 Digitalmars-d- announce wrote:
 On Thursday, 28 May 2020 at 01:14:43 UTC, Jonathan M Davis 
 wrote:
 [...]
I remember reading a suggestion that additional linker symbols be emitted to carry the attribute and possibly type information while leaving the ABI untouched. Was this found to be impractical?
Steven suggested something along those lines. I don't know how practical it would or wouldn't be, but I don't think that Walter even responded to the idea.
Changing the DIP process to be 2 of 3 LMs required for acceptance could get us past all the "this is useless because Walter xyz..." road blocks. I've a proposal in to Mike for that now. We'll see how it goes. [snip]
 But regardless of whether DIP 1028 is the correct decision, the 
 problem remains that your typical extern(C) function cannot be 
 checked for  safety by the compiler, because it was compiled by 
 a C compiler. The question then is just how the D compiler 
 should treat them, and that's the main point of contention. 
 There may be solutions like Steven suggested which would deal 
 with edge cases where the implementation is in D but the 
 linkage isn't, but ultimately, they're just edge cases.

 - Jonathan M Davis
Yes. I understand the practical limits of machine checking and the 1028 issues. The edge case that I had in mind was piece wise replacement of C libraries with dlang reworkings and LCD FFI libraries written in dlang for Python and other languages. With the symbol additions, ABI could be decoupled from D-specific capabilities. Python and other C FFIs just work. Dlang programs get broader guarantees. Still, as you say, probably not a big concern.
May 27 2020
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 22 May 2020 at 17:40:59 UTC, Atila Neves wrote:
 [snip]
 That is a failure of the language that should be resolved.
And how do you suggest we fix it?
safe as whitelist instead of blacklist. When the compiler cannot verify that it is safe, then it cannot be safe.
May 22 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/22/2020 10:33 AM, rikki cattermole wrote:
 To me at least, this butchers  safe/trusted/system into a system that is near 
 useless for guarantees for an entire program.
It never attempted to guarantee safety in code that was never compiled with a D compiler. It's impossible to do that. No language does that.
May 22 2020
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 22, 2020 at 10:50:02PM -0700, Walter Bright via
Digitalmars-d-announce wrote:
 On 5/22/2020 10:33 AM, rikki cattermole wrote:
 To me at least, this butchers  safe/trusted/system into a system
 that is near useless for guarantees for an entire program.
It never attempted to guarantee safety in code that was never compiled with a D compiler. It's impossible to do that. No language does that.
And therefore what we need is a way of indicating verifiability up to things outside of our control. E.g., some kind of way to express that the safety of a piece of code is keyed upon some external function or delegate, thus enabling safe checks for all code except calls into said external function/delegate. This would work out to be practically where we're at now, except that we don't implicitly pretend external code is safe where there is no verification at all. T -- Designer clothes: how to cover less by paying more.
May 22 2020
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
The more I think of Atila's and Walter's responses, the more they 
are starting to make sense.

When I look my own code that uses the Nuklear GUI library, 
written in C, it's all ` system`. I have not had the time to make 
` trusted` wrappers over the BindBC-nuklear API, so I did what 
tends to occur to us as the next best thing: resign and make the 
whole client code ` system`. Just making ` trusted` wrappers over 
BindBC-nuklear seemed to me as inresponsible use of the 
attribute. And reading this theard, it would seem like most of 
you would agree.

But when I think it, what I have accomplised from avoiding that 
antipattern? The only difference is, that if my D code does 
something ` system`, it'll remain under the radar. So I'm worse 
off than had I submitted to the antipattern!

Now, were I designing a library instead of an application, of 
course I should not pass such client code as ` safe`, regardless 
of which of those two ways I choose. But I think the correct way 
would be to mark the API functions as  system. Then it's be best 
of both cheap tricks: The compiler will verify my D code from 
mistakes, but I won't pretend that my code is truly ` safe`.
May 23 2020
next sibling parent Dukc <ajieskola gmail.com> writes:
On Saturday, 23 May 2020 at 10:55:40 UTC, Dukc wrote:
 Just making ` trusted` wrappers over BindBC-nuklear seemed to 
 me as inresponsible use of the attribute.
Meant: blindly making the wrappers, without thinking whether calling the wrapper would always be ` safe`
May 23 2020
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Saturday, 23 May 2020 at 10:55:40 UTC, Dukc wrote:
 The more I think of Atila's and Walter's responses, the more 
 they are starting to make sense.
 [snip]
In fact this former antipattern means that it'd make sense to have an inverse of ` trusted` attribute, lets say ` suspect`. It would mean that the body of the function is verified just like ` safe`, but calling it is allowed only within ` system` or ` trusted` code.
May 23 2020
prev sibling next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Saturday, 23 May 2020 at 10:55:40 UTC, Dukc wrote:

 When I look my own code that uses the Nuklear GUI library, 
 written in C, it's all ` system`. I have not had the time to 
 make ` trusted` wrappers over the BindBC-nuklear API, so I did 
 what tends to occur to us as the next best thing: resign and 
 make the whole client code ` system`.
I really don't understand, there's no "maybe" memory safety: if there's no time to spend for memory safety in a project, why care?
 Just making ` trusted` wrappers over BindBC-nuklear seemed to 
 me as inresponsible use of the attribute. And reading this 
 theard, it would seem like most of you would agree.
I disagree: if you *really* want to use safe in the rest of the codebase, just mark the binding as trusted, raising your hand towards reviewers (or everybody is interested in checking the memory safety of the project codebase), writing in the comment that "you" have decided to spend no time in a proper wrapping, and the motivations. Let the reviewers and the other guys out there just decide what to do with your codebase.
 But when I think it, what I have accomplised from avoiding that 
 antipattern? The only difference is, that if my D code does 
 something ` system`, it'll remain under the radar. So I'm worse 
 off than had I submitted to the antipattern!
It's not an anti pattern, it clearly show your motivation, that's all about trusted. May also write a big *disclaimer* in the README pointing to the pitfalls
May 23 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/23/20 6:55 AM, Dukc wrote:
 The more I think of Atila's and Walter's responses, the more they are 
 starting to make sense.
 
 When I look my own code that uses the Nuklear GUI library, written in C, 
 it's all ` system`. I have not had the time to make ` trusted` wrappers 
 over the BindBC-nuklear API, so I did what tends to occur to us as the 
 next best thing: resign and make the whole client code ` system`. Just 
 making ` trusted` wrappers over BindBC-nuklear seemed to me as 
 inresponsible use of the attribute. And reading this theard, it would 
 seem like most of you would agree.
This is fine, the code *is* system. There's nothing wrong with system code in D. What is wrong is blanket assumption of safety. In this case, people who actually care about memory safety want to be sure that safe functions are actually safe. If you lie about that, then the whole system breaks down. safe just becomes a convention, and really everything just has to be manually checked.
 
 But when I think it, what I have accomplised from avoiding that 
 antipattern? The only difference is, that if my D code does something 
 ` system`, it'll remain under the radar. So I'm worse off than had I 
 submitted to the antipattern!
I understand what you are saying, you want the safety checking in your code, but do not want to deal with the hassle of making special wrappers. And let's be honest here, if you are OK with it, putting trusted: at the top of your extern(C) functions is fine with me. At least that's not a lie. What is not fine is having the compiler do it for you so nary a trusted marking is in sight. I don't really understand the draw of that. -Steve
May 23 2020
next sibling parent JN <666total wp.pl> writes:
On Saturday, 23 May 2020 at 12:13:45 UTC, Steven Schveighoffer 
wrote:
 What is not fine is having the compiler do it for you so nary a 
  trusted marking is in sight. I don't really understand the 
 draw of that.

 -Steve
I think not only about how safe mechanically verifiable is an advantage, but also trusted blocks are very greppable so it's easy to find "danger zones" in any project. I briefly skimmed through the discussion and I don't really see the issue with trusted blocks other than the slightly awkward syntax. Yes, it's additional effort, but it should be an additional effort, so that people try to avoid unsafe code. blocks whenever interacting with C code, or people build safe wrappers over unsafe APIs, which is exactly what would be expected in D after these changes.
May 23 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/23/2020 5:13 AM, Steven Schveighoffer wrote:
 And let's be honest here, if you are OK with it, putting  trusted: at the top
of 
 your extern(C) functions is fine with me. At least that's not a lie.
trusted says the interface to the function is safe. If the programmer did not actually check the interface, it is greenwashing. For example, memcpy() does not have a safe interface and marking it trusted doesn't improve matters.
May 25 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/25/20 5:17 AM, Walter Bright wrote:
 On 5/23/2020 5:13 AM, Steven Schveighoffer wrote:
 And let's be honest here, if you are OK with it, putting  trusted: at 
 the top of your extern(C) functions is fine with me. At least that's 
 not a lie.
trusted says the interface to the function is safe. If the programmer did not actually check the interface, it is greenwashing. For example, memcpy() does not have a safe interface and marking it trusted doesn't improve matters.
Yes, and the programmer did this on purpose. To quote someone else `My reasoning at the time was "I trust libclang".` In that sense, it's not a lie, or "incorrect," you are just trusting that whomever wrote that C library (maybe it was the author of the prototype) followed strictly the D safety rules. It's possible to write a function that takes 2 pointers and a size_t that is safe, just like memcpy (it won't do anything, but there is nothing inherently wrong with that). If it were possible to determine directly from the types of parameters that a function wasn't safe, we would not be having this discussion. So "trusting" a library regardless of interface is what trusted: does. And while I wouldn't do that, or use a library that does that, at least I can see where the problem is! trusted code is hard to write correctly, and there's going to be greenwashing abound when this change comes about. I can even envision someone *recommending* "Don't import core.stdc.x, just declare the prototype yourself, and you can then call them from safe functions!". Greenwashing by the compiler essentially adding trusted to every C porototype doesn't make it any better. I'd much rather have a clear indication that someone made this conscious decision, correct or not, and here is where you should look to find the problems. If I am interested in ENSURING that all code I use is safe or correctly trusted, then I know how to find those points of contention if the trusted attribute is required where it should be. The biggest problem with the magic greenwashing of this DIP is that there is 10-20 years of existing D code with extern(C) function prototypes that are specifically unmarked BECAUSE the author knew that meant system. I don't know how this DIP can be approved just on that problem alone. I hope something better comes out of this discussion, I'm going to mute it now because I don't have the time to keep arguing against people who don't seem to understand either memory safety or human nature. -Steve
May 26 2020
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, May 23, 2020 at 10:55:40AM +0000, Dukc via Digitalmars-d-announce wrote:
[...]
 When I look my own code that uses the Nuklear GUI library, written in
 C, it's all ` system`. I have not had the time to make ` trusted`
 wrappers over the BindBC-nuklear API, so I did what tends to occur to
 us as the next best thing: resign and make the whole client code
 ` system`. Just making ` trusted` wrappers over BindBC-nuklear seemed
 to me as inresponsible use of the attribute. And reading this theard,
 it would seem like most of you would agree.
 
 But when I think it, what I have accomplised from avoiding that
 antipattern?  The only difference is, that if my D code does something
 ` system`, it'll remain under the radar. So I'm worse off than had I
 submitted to the antipattern!
[...] And this is precisely why I proposed that what we need is a way for the compiler to mechanically check all code *except* certain specified blackboxes that are skipped over. Then you can have your calls to unvetted C functions and still have the mechanical checks enabled for the rest of your code. This is also related to trusted blocks inside a function, the intention of which is to limit the system code to as small a surface area as possible while enabling safe checks for the rest of the function. T -- Gone Chopin. Bach in a minuet.
May 23 2020
parent Paul Backus <snarwin gmail.com> writes:
On Saturday, 23 May 2020 at 15:33:05 UTC, H. S. Teoh wrote:
 On Sat, May 23, 2020 at 10:55:40AM +0000

 And this is precisely why I proposed that what we need is a way 
 for the compiler to mechanically check all code *except* 
 certain specified blackboxes that are skipped over.  Then you 
 can have your calls to unvetted C functions and still have the 
 mechanical checks enabled for the rest of your code.

 This is also related to  trusted blocks inside a function, the 
 intention of which is to limit the  system code to as small a 
 surface area as possible while enabling  safe checks for the 
 rest of the function.


 T
system void foo(...) { () safe { // mechanically checked }(); // not checked }
May 23 2020
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Saturday, 23 May 2020 at 10:55:40 UTC, Dukc wrote:
 The more I think of Atila's and Walter's responses, the more 
 they are starting to make sense.

 [...]
Thank you for the anecdote, especially since it captures the spirit of what I've been trying to convey here.
May 25 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/22/20 1:07 PM, Atila Neves wrote:
 And so I was convinced that everything being  safe is actually ok, 
 especially because in real life, most C/C++ APIs aren't going to 
 secretly corrupt your code.
Yes, it can, but not secretly. Just obviously and easily. Note this function: https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/clang/c/index.d#L861 And so, you are free to pepper your safe code with dangling pointers. Sure, you can claim that the C++ library didn't "corrupt your code", which is the case for ALL libraries if you use them properly. You did it, you created a dangling pointer, not the library. The point of safe is to protect you from corrupting your own code based on the guarantees that safe provides. If we don't care about the guarantees of safe as long as you are using C libraries, why are we bothering at all with any of this? I'd rather see a DIP that just removes safety completely if the language maintainers just trust all C libraries "in real life". BTW, you should fix that invalid attribute, freeing a pointer is never safe unless you can guarantee nobody else has a copy of that pointer (and considering it's passed by value, the CALLER still has that pointer!) Will that break your code? It sure might! It just means YOUR code needs to be marked trusted or system as appropriate. -Steve
May 22 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 May 2020 at 17:41:38 UTC, Steven Schveighoffer 
wrote:
 On 5/22/20 1:07 PM, Atila Neves wrote:
 And so I was convinced that everything being  safe is actually 
 ok, especially because in real life, most C/C++ APIs aren't 
 going to secretly corrupt your code.
Yes, it can, but not secretly. Just obviously and easily. Note this function: https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/clang/c/index.d#L861 And so, you are free to pepper your safe code with dangling pointers. Sure, you can claim that the C++ library didn't "corrupt your code", which is the case for ALL libraries if you use them properly. You did it, you created a dangling pointer, not the library.
Right. And the point I was trying to make wasn't "look at what I did, it's cool". No, what I did was dumb. So dumb it took you no time at all to point out one of my mistakes. My point is that the result of making declarations implicity system instead of safe would make people just slap safe on them without really thinking about it to get their code to compile. Like I did.
 The point of  safe is to protect you from corrupting your own 
 code based on the guarantees that  safe provides.
I agree.
 If we don't care about the guarantees of  safe as long as you 
 are using C libraries, why are we bothering at all with any of 
 this?
We care. Annotations become explicit. Do I think this is ideal? No.
 BTW, you should fix that invalid attribute, freeing a pointer 
 is never  safe unless you can guarantee nobody else has a copy 
 of that pointer (and considering it's passed by value, the 
 CALLER still has that pointer!)
You're completely right.
May 22 2020
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 22.05.20 19:54, Atila Neves wrote:
 Right. And the point I was trying to make wasn't "look at what I did, 
 it's cool". No, what I did was dumb. So dumb it took you no time at all 
 to point out one of my mistakes. My point is that the result of making 
 declarations implicity  system instead of  safe would make people just 
 slap  safe on them without really thinking about it to get their code to 
 compile. Like I did.
Now you have accepted a DIP that does the dumb thing automatically. How is that any better? [...]
 If we don't care about the guarantees of  safe as long as you are 
 using C libraries, why are we bothering at all with any of this?
We care. Annotations become explicit. Do I think this is ideal? No.
"Annotations become explicit." - What now? I probably misunderstand that sentence, but DIP 1028 does not require explicit annotations. That's why everyone is upset.
May 22 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 May 2020 at 18:17:29 UTC, ag0aep6g wrote:
 On 22.05.20 19:54, Atila Neves wrote:
 We care. Annotations become explicit. Do I think this is 
 ideal? No.
"Annotations become explicit." - What now? I probably misunderstand that sentence, but DIP 1028 does not require explicit annotations. That's why everyone is upset.
Sorry, I didn't express myself well. I meant that the user can still mark functions as system, they just have to do it explicitly.
May 22 2020
parent reply Dukc <ajieskola gmail.com> writes:
On Friday, 22 May 2020 at 18:27:42 UTC, Atila Neves wrote:
 Sorry, I didn't express myself well. I meant that the user can 
 still mark functions as  system, they just have to do it 
 explicitly.
Hm, DPP might be of help here. Becuse I quess you are going to make sure it'll mark everything ` system`?
May 22 2020
parent Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 May 2020 at 20:08:37 UTC, Dukc wrote:
 On Friday, 22 May 2020 at 18:27:42 UTC, Atila Neves wrote:
 Sorry, I didn't express myself well. I meant that the user can 
 still mark functions as  system, they just have to do it 
 explicitly.
Hm, DPP might be of help here. Becuse I quess you are going to make sure it'll mark everything ` system`?
My plan is probably to do that, yes. I've been thinking about it and I'll likely need to give users options to opt-in to declaring functions safe, otherwise none of them every would be with this: system { #include "header.h" }
May 25 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/22/20 1:54 PM, Atila Neves wrote:
 My point is that the result of making declarations implicity  system 
 instead of  safe would make people just slap  safe on them without 
 really thinking about it to get their code to compile. Like I did.
So the solution is -- make the compiler be dumb for you? If you are going to incorrectly put safe on it, I want you to have to do it because YOU made a conscious decision to be careless, not have it done for you because you forgot. How about we change the law so if you don't sign a contract, we assumed you signed it anyway. In order to reject the contract, you have to X out every paragraph and write "I don't agree to this" and sign that. Any paragraphs you forget are enforceable. -Steve
May 22 2020
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Friday, 22 May 2020 at 18:32:59 UTC, Steven Schveighoffer 
wrote:
 So the solution is -- make the compiler be dumb for you? If you 
 are going to incorrectly put  safe on it, I want you to have to 
 do it because YOU made a conscious decision to be careless, not 
 have it done for you because you forgot.
Yea, agreed. If a library works around an extern(C) API by slapping ` safe:` on it, this is a visibly problematic thing to do, and people can file bug reports and patches about it. And we can have well documented warnings about this being a bad way to do things. If a library just gets extern(C) APIs as safe by default, then it's not visibly obvious that this is problematic. I would much rather have the problematic thing as something visibly done by the developer, than automatically done by the compiler.
May 22 2020
prev sibling next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Friday, 22 May 2020 at 17:54:26 UTC, Atila Neves wrote:
 On Friday, 22 May 2020 at 17:41:38 UTC, Steven Schveighoffer
 And so, you are free to pepper your  safe code with dangling 
 pointers. Sure, you can claim that the C++ library didn't 
 "corrupt your code", which is the case for ALL libraries if 
 you use them properly. You did it, you created a dangling 
 pointer, not the library.
Right. And the point I was trying to make wasn't "look at what I did, it's cool". No, what I did was dumb. So dumb it took you no time at all to point out one of my mistakes. My point is that the result of making declarations implicity system instead of safe would make people just slap safe on them without really thinking about it to get their code to compile. Like I did.
So force people to slap trusted instead, via compiler complains, not safe, and reviewers will catch the laziness: why this is worst that what you picture?
May 22 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/22/2020 10:54 AM, Atila Neves wrote:
 BTW, you should fix that invalid attribute, freeing a pointer is never  safe 
 unless you can guarantee nobody else has a copy of that pointer (and 
 considering it's passed by value, the CALLER still has that pointer!)
You're completely right.
live is intended to fix that! To get ahead of the curve, annotate pointer parameters with `scope` if you don't intend to transfer ownership to the callee. Better yet, use `ref` instead of pointer parameters where possible.
May 22 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/22/20 8:05 PM, Walter Bright wrote:
 On 5/22/2020 10:54 AM, Atila Neves wrote:
 BTW, you should fix that invalid attribute, freeing a pointer is 
 never  safe unless you can guarantee nobody else has a copy of that 
 pointer (and considering it's passed by value, the CALLER still has 
 that pointer!)
You're completely right.
live is intended to fix that! To get ahead of the curve, annotate pointer parameters with `scope` if you don't intend to transfer ownership to the callee. Better yet, use `ref` instead of pointer parameters where possible.
These are extern(C) functions written in C. How does one apply live to that? You can't change it to ref in the prototype. Besides, a deallocation function shouldn't use live, it's about to destroy the pointer. -Steve
May 23 2020
prev sibling next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Friday, 22 May 2020 at 17:07:37 UTC, Atila Neves wrote:

 And so I was convinced that everything being  safe is actually 
 ok, especially because in real life, most C/C++ APIs aren't 
 going to secretly corrupt your code.
Uh? There's plenty of C/C++ code out there with api that when "used in the wrong way" will corrupt memory. That's the _essence_ of trusted code, the _programmer_ need to _correctly_ handle the usage, because the compiler can't do that job alone.
May 22 2020
prev sibling parent reply ag0aep6g <anonymous example.com> writes:
On 22.05.20 19:07, Atila Neves wrote:
 Personally and initially, I would have preferred it if non-extern(D) 
 declarations were implicitly  system. I understood Walter's argument 
 about special cases and how they're bad, but the thought of them being 
  safe made me feel, for lack of a better word, "icky".
So the DIP itself wasn't good enough to convince you. Should have rejected it then. Walter could have made another, better argued DIP. That's how this is supposed to work, isn't it?
 Then I talked to Walter and he pointed out that if those declarations 
 were  system, users would be prevented from calling them from now  safe 
 code. A regular user would probably just slap ` safe:` at the top of the 
 bindings module anyway. Then I realised that I did exactly that with my 
 libclang bindings:
 
 https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/
lang/c/index.d#L254 
 
 
 "Worse", I made all functions `pure` and all parameters `in` as well for 
 good measure. Why? I wanted to call them from my  safe pure code with 
 `const` arguments. My reasoning at the time was "I trust libclang".
 
 And so I was convinced that everything being  safe is actually ok, 
 especially because in real life, most C/C++ APIs aren't going to 
 secretly corrupt your code.
Your reasoning is fine when you're dealing with a function that has a safe interface. I.e., it can only corrupt your code when it's buggy. Then it makes sense so say "I trust the authors". D users do it with Phobos. Phobos authors do it with the C standard library. This kind of trust is perfectly ok. Your reasoning is completely wrong when you're thinking about a function that doesn't have a safe interface. E.g., C's `memcpy` just isn't safe. It doesn't matter whether you trust the authors or not. The user has to make sure that the call is correct. Otherwise, `memcpy` will corrupt your memory. And `memcpy` is far from the only such function. I'm positive that Walter is aware of the difference. Judging from this post, I'm afraid you might not be.
May 22 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 May 2020 at 18:11:28 UTC, ag0aep6g wrote:
 So the DIP itself wasn't good enough to convince you.
Had that been the case, I would have rejected it.
 Your reasoning is fine when you're dealing with a function that 
 has a safe interface. I.e., it can only corrupt your code when 
 it's buggy. Then it makes sense so say "I trust the authors". D 
 users do it with Phobos. Phobos authors do it with the C 
 standard library. This kind of trust is perfectly ok.
 Your reasoning is completely wrong when you're thinking about a 
 function that doesn't have a safe interface. E.g., C's `memcpy` 
 just isn't safe.
memcpy isn't a good example since it's explicitly system: https://dlang.org/phobos/core_stdc_string.html#.memcpy
 It doesn't matter whether you trust the authors or not. The 
 user has to make sure that the call is correct. Otherwise, 
 `memcpy` will corrupt your memory. And `memcpy` is far from the 
 only such function.
Yes. But most of them aren't like memcpy. Most D code calls other D code, not C. Am I saying nothing bad can happen if we implicitly trust extern(C) declarations? No. I'm saying we'll be no worse off if they're all implicitly system. This compiles with no warnings right *now*: void main() { import core.stdc.stdlib: free; free(cast(void*) 42); free(new int); free(&main); }
May 22 2020
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 22.05.20 20:24, Atila Neves wrote:
 On Friday, 22 May 2020 at 18:11:28 UTC, ag0aep6g wrote:
 So the DIP itself wasn't good enough to convince you.
Had that been the case, I would have rejected it.
You said the DIP as written felt "icky". And only after a chat with Walter did that feeling go away. You're saying you would have accepted a DIP that feels icky? DIPs are supposed to be convincing, not barely tolerable.
 memcpy isn't a good example since it's explicitly  system:
 
 https://dlang.org/phobos/core_stdc_string.html#.memcpy
core.stdc.string.memcpy is a specific binding to C's memcpy. Any other declaration of it will be implicitly safe. "But why would anyone declare `memcpy` themselves, when they can just import the one from `core`?" I hear you ask. And I answer: 1) Shouldn't matter, if we're talking about a principled safety system. But I realize that the leadership isn't interested in that anymore. 2) I have quickly typed out C declarations before, because it was more convenient than looking up where it is exactly in the standard library. And we're all about catering to convenience now, aren't we?
 Yes. But most of them aren't like memcpy. Most D code calls other D 
 code, not C.
Most D code isn't behind an `extern (C)` prototype. Virtually no one is (strongly) against making D functions safe by default.
 Am I saying nothing bad can happen if we implicitly trust extern(C) 
 declarations? No. I'm saying we'll be no worse off if they're all 
 implicitly  system.
"No worse off" should not be good enough for a DIP.
 This compiles with no warnings right *now*:
 
 void main() {
      import core.stdc.stdlib: free;
      free(cast(void*) 42);
      free(new int);
      free(&main);
 }
And this doesn't compile right now, but it will with DIP 1028: ---- extern (C) void free(void*); void main() safe { free(cast(void*) 42); free(new int); free(&main); } ---- Yes, yes, I know. That's soo much less common in the real world. But before the great safe schism that is happening right now, safe wasn't about catching more bugs than not on average. It was about being "100% mechanically checkable" (Walter's words). I for one liked it better when it had that aspiration. And I was contributing towards that goal (latest PR was merged just yesterday, coincidentally [1]). I have no interest in a version of safe that is deliberately fuzzy, where I have to defend safety fixes against convenience arguments. [1] https://github.com/dlang/dlang.org/pull/2773
May 22 2020
parent Paul Backus <snarwin gmail.com> writes:
On Friday, 22 May 2020 at 19:16:26 UTC, ag0aep6g wrote:
 I for one liked it better when it had that aspiration. And I 
 was contributing towards that goal (latest PR was merged just 
 yesterday, coincidentally [1]).

 [1] https://github.com/dlang/dlang.org/pull/2773
Thank you for that PR, by the way. The new version of the spec is *much* clearer, and actually helped me find a subtle safety issue in my own code that I'd previously overlooked. I sincerely hope the D leadership appreciates how valuable this kind of work is.
May 22 2020
prev sibling next sibling parent Mathias LANG <geod24 gmail.com> writes:
On Friday, 22 May 2020 at 18:24:39 UTC, Atila Neves wrote:
 Am I saying nothing bad can happen if we implicitly trust 
 extern(C) declarations? No. I'm saying we'll be no worse off if 
 they're all implicitly  system.

 This compiles with no warnings right *now*:

 void main() {
     import core.stdc.stdlib: free;
     free(cast(void*) 42);
     free(new int);
     free(&main);
 }
And this code won't compile at all when you make ` safe` the default: ``` void main () { int[] myArray = (cast(int*) malloc(42 * int.sizeof))[0 .. 42]; } ``` So, we changed the default. Now those people that want to easily prototype something, or write applications that extensively use system libraries, will be extremely inconvenienced, but we satisfied the people that want more ` safe`ty, which we all known and agree is the future. Except that in that process we also destroyed the meaning of ` safe` by shooting a hole into it, for the convenience of the people that want to easily prototype something and write applications that interact extensively with system libraries, because that's what half of the D users do. Clearly, an improvement.
May 22 2020
prev sibling parent Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Friday, 22 May 2020 at 18:24:39 UTC, Atila Neves wrote:
 memcpy isn't a good example since it's explicitly  system
Yes, it's a good example. Because if you include any C function, you don't know if it uses memcpy internally - except if you have the code. And as memcpy is used heavily within C libraries, it's not unlikely that your "oh so safe" function is using it too. Very bad advice. It should be simply forbidden to slap safe at anything with no code available, because save should be reserved for mechanically verified parts of code. Ok, you can slap trusted on it, and it will compile (the easy way), but anyone using it is at least warned. So it's possible to check - and if no body is availabe (e.g. an object library) I would refrain from trusting it. But I don't want the compiler to greenwash it for me, no thanks. There should be at least someone on the D side to blame, destroying his reputation by slapping trusted at whole modules.
May 23 2020
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 [snip]
Thank you for your reply. How about some time before this DIP is fully in the language, a compiler flag is added that will produce warnings for when extern prototypes without explicit safe/ trusted/ system are used? Or something like that. I have a good sense of when my own code uses extern(C), but not always other people's.
May 22 2020
parent Francesco Mecca <me francescomecca.eu> writes:
On Friday, 22 May 2020 at 12:25:23 UTC, jmh530 wrote:
 On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 [snip]
Thank you for your reply. How about some time before this DIP is fully in the language, a compiler flag is added that will produce warnings for when extern prototypes without explicit safe/ trusted/ system are used? Or something like that. I have a good sense of when my own code uses extern(C), but not always other people's.
There is no reason to make the compiler do that. I think it would be more appropriate for D-scanner to do that or more practically for dub https://github.com/dlang-community/D-Scanner
May 22 2020
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 22.05.20 03:22, Walter Bright wrote:
 
 This is Obviously A Good Idea. Why would I oppose it?
 
 1. I've been hittin' the crack pipe again.
 2. I was secretly convinced, but wanted to save face.
 3. I make decisions based on consultation with my astrologer.
 4. I am evil.
5. You are backwards-rationalizing a wrong intuition that is based on experiences that are not actually analogous. You are ignoring feedback given by many people around you because that worked out well for you in the past. I know that you had many interactions with large groups of ignorant people who thought that you would never be able to pull off a certain thing. This is not one of those cases. I understand the appeal, but the backlash really should not encourage you to soldier on this time.
 
 1. Go through 200 functions in clibrary.d and determine which are  safe
 and which are  system. This is what we want them to do. We try to motivate
 this with compiler error messages. Unfortunately, this is both tedious and
 thoroughly impractical, as our poor user Will Not Know which are safe and
 which are system. We can correctly annotate core.stdc.stdio because I know
 those functions intimately. This is not true for other system C APIs, and
 even less true for some third party C library we're trying to interface to.
 
 2. Annotate useClibrary() as  trusted or  system. While easier,
First do 2, then, over time, do 1. If having the safe tag and no trusted code is important to you, aim to replace the C code with something you can automatically verify, by slowly porting it over to D.
 this causes all benefits to  safe by default to be lost. 
Absolutely not. Greenwashing causes the benefits of certification to be lost. Honesty does not. The value of safe code is what it is because there is code that can't be safe.
 
 4. Edit clibrary.d and make the first line:
 
       safe:
 
 I submit that, just like with Java, Option 4 is what people will reach for,
 nearly every time. I've had some private conversations where people 
 admitted
 this was what they'd do. People who knew it was wrong to do that.
 ...
They should know to put trusted instead of safe, and the compiler should enforce it. Also, why do those people speak for everyone else? They don't speak for me.
 If it's  safe by default, and then someone chooses to annotate it with 
  system
 here and there, I'd feel a lot more confident about the accuracy of the 
 code
 annotations than if it just had  safe: at the top. At least they tried.
 ...
If it has safe:/ trusted: at the top at least you know they were aware what they were doing. Also, what about if it has trusted: at the top and some system annotations here and there? Did they not try?
 What is actually accomplished with this amendment if it was implemented?
 
 1. Adds a funky, special case rule. It's better to have simple, easily
 understood rules than ones with special cases offering little improvement.
 ...
What about the funky special case rule that the compiler is responsible for memory safety of safe code except in this one weird special case?
 2. Existing, working code breaks.
 ...
Making safe the default is bound to break code. It's bad enough that code will break. Avoiding part of that code breakage is no justification for breaking safe.
 3. The most likely code fixes are to just make it compile, absolutely
 nothing safety-wise is improved. The added annotations will be a fraud.
 ...
It's vastly better to have some fraudulent annotations in some projects than a fraudulent compiler compiling all projects. Do you really want to put the responsibility for the memory safety of random C libraries on the compiler developers?
 D should not encourage "greenwashing" practices like the Java
 exception specification engendered.
So your argument is that you don't want D programmers to do have to do the dirty work of greenwashing. Therefore the compiler will implicitly greenwash for them? What about the programmers who actually want to do the right thing and don't want the compiler to implicitly greenwash C libraries for them?
 The compiler cannot vet the accuracy
 of bodyless C functions, and we'll just have to live with that. The 
 proposed
 amendment does not fix that.
 ...
trusted is the fix.
 And so, I did not incorporate the proposed amendment to the Safe by Default
 DIP.
Which (so far) is a harmless mistake with an easy fix.
May 22 2020
prev sibling next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 3. Wrap the call to massage_data() with:

     ()  trusted { massage_data(parameters); } ();
The correct solution is to encapsulate the C functions as-needed with a higher level API - and this is somewhat commonly done already and - as i understand it anyway - the whole purpose of trusted being function level instead of statement level.
 2. Existing, working code breaks.
safe by default is probably the biggest breaking change D has ever seen as of this point. It eclipses anything in the D2 transition. It is D3 in all by name. My preference, as I've said many times, is to fix the propagation of attributes into child scopes and leave the defaults alone, but failing that, I'd actually say embrace this: if we're doing big breakage anyway, go big and do it right.
May 22 2020
prev sibling next sibling parent reply Gregory <g.thompson.1892 gmall.com> writes:
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 Consider the common (because that's how D started out) case of:

 ----- clibrary.d --------

     T massage_data(... many parameters ...);
     ... 200 more such declarations ...

 ----- app.d ----------

     import clibrary;

     void useClibrary( ... parameters ...) {
         massage_data(parameters);
     }

 ---------------------

 This code, today, does not use annotations and it works. It's 
 been working
 for a long time. Now, we implement the amendment, and it stops 
 compiling
 because useClibrary is  safe and massage_data is  system. The 
 user is faced
 with the following alternatives:
Why are you assuming that the only thing wrong with useClibrary() is that it would use a C declaration that is system? All system code I've written is system and wouldn't compile to safe even if extern(C) declarations are wrongly annotated as safe. What if app uses core.stdc? All those functions have already been annotated as system. Any code that uses the C stdlib is going to break right now anyways. What's stopping someone from just as easily slapping trusted: at the top of app.d? Since there are going to be more issues than simply calling extern(C) functions. I find it odd also that you are using one of the rationales against making safe the default (that code will break and lazy solutions will be employed to make them work again), as a means to make safe, less memory safe.
May 22 2020
parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 22 May 2020 at 18:34:32 UTC, Gregory wrote:
 [snip]

 Why are you assuming that the only thing wrong with 
 useClibrary() is that it would use a C declaration that is 
  system? All  system code I've written is  system and wouldn't 
 compile to  safe even if extern(C) declarations are wrongly 
 annotated as  safe.

 What if app uses core.stdc? All those functions have already 
 been annotated as  system. Any code that uses the C stdlib is 
 going to break right now anyways.

 [snip]
The breakage would produce error messages. Annoying to fix, but at least obvious. The worry is more about not receiving any error messages. In other words, some code would compile as system previously, but now would compile as safe with nothing else changing. It's safe, but is it really safe?
May 22 2020
prev sibling next sibling parent reply Johannes T <isrvoid gmail.com> writes:
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.
 [..]
Thank you for the detailed and insightful explanation. Would it be feasible to have a follow-up DIP that enables implicit system extern as an opt-in (e.g. -nosafeextern). The frustration most people seem to have with safe extern is that it lessens the promise of safe without a recourse.
May 23 2020
parent reply Arafel <er.krali gmail.com> writes:
On 23/5/20 20:40, Johannes T wrote:
 On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.
 [..]
Thank you for the detailed and insightful explanation. Would it be feasible to have a follow-up DIP that enables implicit system extern as an opt-in (e.g. -nosafeextern). The frustration most people seem to have with safe extern is that it lessens the promise of safe without a recourse.
Also there will need to be an answer to this probably inthe future not unusual question: My code is 100% safe, there's no trusted block anywhere, why am I getting memory corruption? To me the expectation / promise was that safe code was safe from memory corruption modulo bugs in the implementation. The only source for issues would be then trusted code that could be audited as needed. You could say "well, you always had to check for those", but the difference is that until now the compiler would shout at me if I didn't, so I wouldn't forget about them. Now it has to be clearly explained that you should check for trusted code AND unmarked external C functions.
May 23 2020
parent reply ag0aep6g <anonymous example.com> writes:
On 23.05.20 21:46, Arafel wrote:
 Now it has to be clearly explained that you should check for  trusted 
 code AND unmarked external C functions.
... and system static constructors and `--boundscheck=off` and initializers of globals
May 23 2020
parent reply Arafel <er.krali gmail.com> writes:
On 24/5/20 0:02, ag0aep6g wrote:
 
 ... and  system static constructors and `--boundscheck=off` and 
 initializers of globals
Other than `--boundscheck=off`, that is presumably actively chosen by the user (as is trust), would the others be allowed without ` trusted` in otherwise 100% safe code? I would find concerning that any system code is allowed, but I guess initializers of globals should be ok as long as they are safe themselves?
May 23 2020
parent reply ag0aep6g <anonymous example.com> writes:
On 24.05.20 00:17, Arafel wrote:
 On 24/5/20 0:02, ag0aep6g wrote:
 ... and  system static constructors and `--boundscheck=off` and 
 initializers of globals
Other than `--boundscheck=off`, that is presumably actively chosen by the user (as is trust), would the others be allowed without ` trusted` in otherwise 100% safe code?
Yup. Today they can be unmarked, defaulting to system. With DIP 1028, they can be explicitly marked system. Either way, they don't show up when you only look for " trusted".
 I would find concerning that any  system code is allowed, but I guess 
 initializers of globals should be ok as long as they are  safe themselves?
As long as they're safe, sure. But they can also be system. An example: ---- const int x = 42; const int y = 43; void main() safe { import std.stdio; writeln(x, " ", y); /* Prints "42 43" as expected. */ auto px = &x; auto py = &y; writeln(*px, " ", *py); /* Prints "13 14". Wat? */ } int* p = cast(int*) &x; static this() system { *p = 13; *++p = 14; } ----
May 23 2020
next sibling parent Arafel <er.krali gmail.com> writes:
On 24/5/20 0:38, ag0aep6g wrote:
 On 24.05.20 00:17, Arafel wrote:
 On 24/5/20 0:02, ag0aep6g wrote:
 ... and  system static constructors and `--boundscheck=off` and 
 initializers of globals
Other than `--boundscheck=off`, that is presumably actively chosen by the user (as is trust), would the others be allowed without ` trusted` in otherwise 100% safe code?
Yup. Today they can be unmarked, defaulting to system. With DIP 1028, they can be explicitly marked system. Either way, they don't show up when you only look for " trusted".
 I would find concerning that any  system code is allowed, but I guess 
 initializers of globals should be ok as long as they are  safe 
 themselves?
As long as they're safe, sure. But they can also be system. An example: ---- const int x = 42; const int y = 43; void main() safe {     import std.stdio;     writeln(x, " ", y); /* Prints "42 43" as expected. */     auto px = &x;     auto py = &y;     writeln(*px, " ", *py); /* Prints "13 14". Wat? */ } int* p = cast(int*) &x; static this() system { *p = 13; *++p = 14; } ----
I find this... disturbing... what is worse, it also happens with `immutable`. Then there should most definitely be a list of features to look (in addition to ` trusted`) if you want to make sure your ` safe` code is actually safe.
May 23 2020
prev sibling parent reply Arine <arine1283798123 gmail.com> writes:
On Saturday, 23 May 2020 at 22:38:58 UTC, ag0aep6g wrote:
 On 24.05.20 00:17, Arafel wrote:
 On 24/5/20 0:02, ag0aep6g wrote:
 ... and  system static constructors and `--boundscheck=off` 
 and initializers of globals
Other than `--boundscheck=off`, that is presumably actively chosen by the user (as is trust), would the others be allowed without ` trusted` in otherwise 100% safe code?
Yup. Today they can be unmarked, defaulting to system. With DIP 1028, they can be explicitly marked system. Either way, they don't show up when you only look for " trusted".
 I would find concerning that any  system code is allowed, but 
 I guess initializers of globals should be ok as long as they 
 are  safe themselves?
As long as they're safe, sure. But they can also be system. An example: ---- const int x = 42; const int y = 43; void main() safe { import std.stdio; writeln(x, " ", y); /* Prints "42 43" as expected. */ auto px = &x; auto py = &y; writeln(*px, " ", *py); /* Prints "13 14". Wat? */ } int* p = cast(int*) &x; static this() system { *p = 13; *++p = 14; } ----
That works even if you make the static this() safe, and remove the pointer incrementation. You'd have to make the p initialization safe. safe: int* p = cast(int*) &x; // error But note this doesn't work: safe int* p = cast(int*) &x; // compiles Having the default become safe will help detect this, as I don't imagine that is a whole lot of usage of safe: to begin with.
May 23 2020
parent reply ag0aep6g <anonymous example.com> writes:
On 24.05.20 02:55, Arine wrote:
 That works even if you make the static this()  safe, and remove the 
 pointer incrementation.
Sure. `*p = 13;` is perfectly safe. The static constructor isn't needed for that part. You can just as well do the assignment in `main`. The static constructor is another feature that can smuggle unsafe code (the increment) into your program without the trusted warning label.
 You'd have to make the p initialization  safe.
 
       safe:
          int* p = cast(int*) &x; // error
 
 But note this doesn't work:
 
       safe int* p = cast(int*) &x; // compiles
 
 Having the default become  safe will help detect this, as I don't 
 imagine that is a whole lot of usage of  safe: to begin with.
The example compiles with `-preview=safedefault`. And even if that gets changed, it will probably still compile when marked system. So we still won't find it when looking for " trusted".
May 23 2020
parent reply Arine <arine1283798123 gmail.com> writes:
On Sunday, 24 May 2020 at 01:26:02 UTC, ag0aep6g wrote:
 On 24.05.20 02:55, Arine wrote:
 That works even if you make the static this()  safe, and 
 remove the pointer incrementation.
Sure. `*p = 13;` is perfectly safe. The static constructor isn't needed for that part. You can just as well do the assignment in `main`. The static constructor is another feature that can smuggle unsafe code (the increment) into your program without the trusted warning label.
It'd be no different than passing the pointer into safe code as a parameter from system code. Ultimately the error occurs in system code and directly as a result of system code. It is undefined behavior as well. No amount of safe code can save you from that.
 You'd have to make the p initialization  safe.
 
       safe:
          int* p = cast(int*) &x; // error
 
 But note this doesn't work:
 
       safe int* p = cast(int*) &x; // compiles
 
 Having the default become  safe will help detect this, as I 
 don't imagine that is a whole lot of usage of  safe: to begin 
 with.
The example compiles with `-preview=safedefault`. And even if that gets changed, it will probably still compile when marked system. So we still won't find it when looking for " trusted".
Then that is definitely a bug if that's the case. Someone should probably make a bug report, Walter? If you are still using system with safe, then that would still be somewhere you have to look for not memory safe code. trusted should just mean that someone verified it. system then would mean no one's verified it to be safe, that doesn't mean you don't have to check it.
May 24 2020
parent reply ag0aep6g <anonymous example.com> writes:
On Sunday, 24 May 2020 at 14:39:50 UTC, Arine wrote:
[...]
 It'd be no different than passing the pointer into  safe code 
 as a parameter from  system code. Ultimately the error occurs 
 in  system code and directly as a result of  system code. It is 
 undefined behavior as well. No amount of safe code can save you 
 from that.
I think you're arguing against a point that wasn't made. I'm not saying that there's anything fundamentally unsound about an system static constructor. As you say, it's the same any other system function. I'm just saying that it's another thing you have to check when you want to verify that a program is actually safe. [...]
 Then that is definitely a bug if that's the case. Someone 
 should probably make a bug report, Walter? If you are still 
 using  system with  safe, then that would still be somewhere 
 you have to look for not memory safe code.  trusted should just 
 mean that someone verified it.  system then would mean no one's 
 verified it to be safe, that doesn't mean you don't have to 
 check it.
system does indicate that you don't have to check a function. But its trumped by other indicators: * system entry points (`main`, static constructors, static initializers) - have to check those. * Foreign prototypes (`extern (C)` and friends) - have to check those, whether they're system or safe or trusted. * system functions that are being called by trusted ones - have to check those. But I would say that's part of verifying trusted functions. Other than that (and maybe other special cases that I've missed), you can safely ignore system functions, because your safe program cannot possibly be calling them.
May 24 2020
parent reply Arine <arine1283798123 gmail.com> writes:
On Sunday, 24 May 2020 at 15:42:54 UTC, ag0aep6g wrote:
 On Sunday, 24 May 2020 at 14:39:50 UTC, Arine wrote:
 Then that is definitely a bug if that's the case. Someone 
 should probably make a bug report, Walter? If you are still 
 using  system with  safe, then that would still be somewhere 
 you have to look for not memory safe code.  trusted should 
 just mean that someone verified it.  system then would mean no 
 one's verified it to be safe, that doesn't mean you don't have 
 to check it.
system does indicate that you don't have to check a function. But its trumped by other indicators: * system entry points (`main`, static constructors, static initializers) - have to check those. * Foreign prototypes (`extern (C)` and friends) - have to check those, whether they're system or safe or trusted. * system functions that are being called by trusted ones - have to check those. But I would say that's part of verifying trusted functions. Other than that (and maybe other special cases that I've missed), you can safely ignore system functions, because your safe program cannot possibly be calling them.
You *have* to check system code. That's where you are guarantee'd to have memory safety issues. If you are ignoring system code because you think safe code doesn't interact with it at all, then that's a problem you are creating for yourself. system code can still call safe code, and that system code that is calling the safe code can pass invalid information that causes the safe code to misbehave. You have to check system for memory safety issues. It seems Walter's comments about only have to review trusted are being taken too literally.
May 24 2020
parent ag0aep6g <anonymous example.com> writes:
On 24.05.20 19:44, Arine wrote:
 On Sunday, 24 May 2020 at 15:42:54 UTC, ag0aep6g wrote:
[...]
  system does indicate that you don't have to check a function. But its 
 trumped by other indicators:
[...]
 You *have* to check  system code. That's where you are guarantee'd to 
 have memory safety issues. If you are ignoring  system code because you 
 think  safe code doesn't interact with it at all, then that's a problem 
 you are creating for yourself.  system code can still call  safe code, 
 and that  system code that is calling the  safe code can pass invalid 
 information that causes the  safe code to misbehave. You have to check 
  system for memory safety issues.
You're right; it's not accurate that " system does indicate that you don't have to check a function". That's only true under particular conditions: When your entry points are safe and you have already verified all trusted functions (including their call graphs which might include system functions), then you can ignore any other system functions, because your program doesn't call them anyway. But that's true for any function. If your program doesn't call it, you don't need to check it. So it's not a particularly meaningful thing to say about system, and that's on me.
May 24 2020
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
I'd like to emphasize:

1. It is not possible for the compiler to check any declarations where the 
implementation is not available. Not in D, not in any language. Declaring a 
declaration safe does not make it safe.

2. If un-annotated declarations cause a compile time error, it is highly likely 
the programmer will resort to "greenwashing" - just slapping  safe on it. I've 
greenwashed code. Atila has. Bruce Eckel has. We've all done it. Sometimes even 
for good reasons.

3. Un-annotated declarations are easily detectable in a code review.

4. Greenwashing is not easily detectable in a code review.

5. Greenwashing doesn't fix anything. The code is not safer. It's an illusion, 
not a guarantee.

6. If someone cares to annotate declarations, it means he has at least thought 
about it, because he doesn't need to. Hence it's more likely to be correct than 
when greenwashed.

7. D should *not* make it worthwhile for people to greenwash code.

It is, in a not-at-all obvious way, safer for C declarations to default to
being 
safe.
May 23 2020
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 24.05.20 05:28, Walter Bright wrote:
 I'd like to emphasize:
 
I understand all of those points and most of them are true, and obvious. The issue is that they are not a justification for the decision. You seem to think that greenwashing is not greenwashing when it is done by the compiler without user interaction. Why is that?
 1. It is not possible for the compiler to check any declarations where 
 the implementation is not available. Not in D, not in any language. 
 Declaring a declaration safe does not make it safe.
 ...
Which is exactly why it should not be possible to declare it safe.
 2. If un-annotated declarations cause a compile time error, it is highly 
 likely the programmer will resort to "greenwashing" - just slapping 
  safe on it. I've greenwashed code. Atila has. Bruce Eckel has. We've 
 all done it. Sometimes even for good reasons.
 ...
Slapping safe on it should not even compile. You should slap either system or trusted on it.
 3. Un-annotated declarations are easily detectable in a code review.
 ...
It's easier to find something that is there than something that is not there. Greenwashing is not easier to detect if the compiler did it implicitly.
 4. Greenwashing is not easily detectable in a code review.
 ...
Even though it is easy to miss in a code review, it's easy to detect automatically. Any extern(C) prototype that is annotated safe (explicitly or implicitly) is greenwashed.
 5. Greenwashing doesn't fix anything. The code is not safer.
Actually further down you say that it makes the code safer in a "not-at-all obvious way". Which is it?
 It's an illusion, not a guarantee.
 ...
Yes. On the other hand, trusted is not an illusion, it is a way to clarify responsibilities.
 6. If someone cares to annotate declarations, it means he has at least 
 thought about it, because he doesn't need to.
True, but this is an argument against restrictive defaults in general, in particular safe by default. Also note that if someone cares to annotate declarations, the compiler pointing out missing annotations that would otherwise cause implicit greenwashing is _useful_.
 Hence it's more likely to be correct than when greenwashed.
 ...
This is true whether or not the compiler does the greenwashing implicitly. Annotating with safe is a lie, whether the compiler does it or the programmer. It should be rejected and force system or trusted. You can still quickly see a difference in applied care by checking whether it's a single trusted: or each prototype is annotated individually.
 7. D should *not* make it worthwhile for people to greenwash code.
 ...
Greenwashing automatically is not a solution, it's admitting defeat. Why can't the compiler just reject greenwashing with safe? Slapping trusted on prototypes is not greenwashing, it's saying "I take responsibility for the memory safety of this external C code".
 It is, in a not-at-all obvious way, safer for C declarations to default 
 to being safe.
safe is advertised to give mechanical guarantees, where trusted is a way for programmers to take responsibility for parts of the code. It is not advertised to be an unsound linter with pseudo-pragmatic trade-offs and implicit false negatives.
May 23 2020
next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Sunday, 24 May 2020 at 05:43:45 UTC, Timon Gehr wrote:

  safe is advertised to give mechanical guarantees, where 
  trusted is a way for programmers to take responsibility for 
 parts of the code. It is not advertised to be an unsound linter 
 with pseudo-pragmatic trade-offs and implicit false negatives.
And turns back to my previous question, that Walter (or Atila) never replied: what I need to reply back to customers asking us about safe. safe is for mechanical check or not? An official and public declaration please. /P
May 24 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
I infer your position is the idea that putting  trusted on the declarations 
isn't greenwashing, while  safe is.

I can't see a practical difference between:

 safe extern (C) void whatevs(parameters);
 trusted extern (C) void whatevs(parameters);

Both require that whatevs() provide a safe interface. The difference between 
them is in the implementation of those functions, not the interface. Since the
D 
compiler cannot see those implementations, they are immaterial to the compiler 
and user.
May 24 2020
next sibling parent reply Panke <tobias pankrath.net> writes:
On Sunday, 24 May 2020 at 08:55:32 UTC, Walter Bright wrote:
 I infer your position is the idea that putting  trusted on the 
 declarations isn't greenwashing, while  safe is.

 I can't see a practical difference between:

  safe extern (C) void whatevs(parameters);
  trusted extern (C) void whatevs(parameters);

 Both require that whatevs() provide a safe interface. The 
 difference between them is in the implementation of those 
 functions, not the interface. Since the D compiler cannot see 
 those implementations, they are immaterial to the compiler and 
 user.
I've always understood that the safe, trusted, system machinery provides the following guarantee once all holes are fixed: If I have a memory corruption in my code than I need to only look at the trusted and system parts to find it. Marking whatevs safe violates this, marking it trusted does not.
May 24 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/24/2020 2:29 AM, Panke wrote:
 I've always understood that the  safe, trusted, system machinery provides the 
 following guarantee once all holes are fixed:
 
 If I have a memory corruption in my code than I need to only look at the 
  trusted and  system parts to find it.
 
 Marking whatevs  safe violates this, marking it  trusted does not.
It's a fair point, but without the source code the distinction is meaningless.
May 24 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 24 May 2020 at 09:47:37 UTC, Walter Bright wrote:
 On 5/24/2020 2:29 AM, Panke wrote:
 I've always understood that the  safe, trusted, system 
 machinery provides the following guarantee once all holes are 
 fixed:
 
 If I have a memory corruption in my code than I need to only 
 look at the  trusted and  system parts to find it.
 
 Marking whatevs  safe violates this, marking it  trusted does 
 not.
It's a fair point, but without the source code the distinction is meaningless.
The distinction is that you can find a slapped on trusted with a grep. Rather than valgrind.
May 24 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/24/2020 3:40 AM, Stefan Koch wrote:
 The distinction is that you can find a slapped on trusted with a grep.
It's just as easy to use grep to *not* find trusted.
May 27 2020
parent reply Johannes Loher <johannes.loher fg4f.de> writes:
Am 27.05.20 um 11:25 schrieb Walter Bright:
 On 5/24/2020 3:40 AM, Stefan Koch wrote:
 The distinction is that you can find a slapped on trusted with a grep.
It's just as easy to use grep to *not* find trusted.
But that's not enough. You need a regexp that searches for extern (C(++)) declarations that do not have any of safe, trusted, system. The attributes can also be either before the return type + name + parameters or after it. They can also be mixed with any other attributes. Sure, you can probably write a regex that matches all of this but it is a _lot_ more complicated than simply searching for trusted.
May 27 2020
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 27/05/2020 10:12 PM, Johannes Loher wrote:
 Am 27.05.20 um 11:25 schrieb Walter Bright:
 On 5/24/2020 3:40 AM, Stefan Koch wrote:
 The distinction is that you can find a slapped on trusted with a grep.
It's just as easy to use grep to *not* find trusted.
But that's not enough. You need a regexp that searches for extern (C(++)) declarations that do not have any of safe, trusted, system. The attributes can also be either before the return type + name + parameters or after it. They can also be mixed with any other attributes. Sure, you can probably write a regex that matches all of this but it is a _lot_ more complicated than simply searching for trusted.
extern(Windows) extern(System) COM Few more things there.
May 27 2020
prev sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Sunday, 24 May 2020 at 09:47:37 UTC, Walter Bright wrote:
 It's a fair point, but without the source code the distinction 
 is meaningless.
It's meaningless in terms of what the compiler can check, but it's not meaningless in terms of documenting the assumptions and promises the developer is making.
May 24 2020
prev sibling next sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Sunday, 24 May 2020 at 08:55:32 UTC, Walter Bright wrote:
 I infer your position is the idea that putting  trusted on the 
 declarations isn't greenwashing, while  safe is.

 I can't see a practical difference between:

  safe extern (C) void whatevs(parameters);
  trusted extern (C) void whatevs(parameters);

 Both require that whatevs() provide a safe interface. The 
 difference between them is in the implementation of those 
 functions, not the interface. Since the D compiler cannot see 
 those implementations, they are immaterial to the compiler and 
 user.
From my perspective, there is a clear difference in meaning between them: safe means verified by the compiler (which as you mentioned can never be true for declarations where the code is not available, hence it always is a lie in that case) and trusted means verified by the developer. Whenever trusted is slapped on _anything_, it is clear that extra caution is needed (in codereviews etc.) and as others have mentioned, it is easily searchable. When I put safe on a piece of code, my expectation is that it is actually verified by the compiler. If that is not possible, it should not compile. The same reasoning also applies to the case where the annotations are not added by the developer explicitly and the defaults are used instead: If the default is safe and the code for an unannotated declaration is not available, it should not compile. Making trusted the default is not an option because as mentioned earlier, one of the points of trusted is for it to be actually seen, which is not the case if it is the default, so I will not elaborate on that. If sytem is the default, there is no issue but of course the whole point of this DIP is to move away from that. I understand that today it is possible to slap safe on declarations without code without the compiler complaining but it really _should_ be an error in order to have a consistent meaning of safe, trusted and system ( safe = compiler verified, trusted = user verified, system = not verified) also in the case of declarations where no code is available. Let's assume this is indeed the way to go, then there are 2 possible solutions for this DIP regarding how to handle this: 1. Make declarations without code default to system instead of safe. This is what many here have been arguing for but as you mentioned several times, it has the drawback of adding special cases to the language. As a user, it does not seem that problematic though because the compiler can easily tell you what is going on when there is an issue (i.e. calling an unannotated declaration without code from a safe function could result in compiler error message that explains why this does not work). But I admit that it is still a bit weird to have 2 different defaults. Additionally, explicitly annotating such declarations with safe should be a compile error. 2. Make safe the default for _all_ declarations but still make safe on declarations without code a compile error (because the compiler cannot verify it). This means that annotating function declarations without code with system or trusted is now mandatory. This is what Timon has been arguing for if I understood him correctly. 3. There may be a third option if we introduce something like conditional safety but I do not completely understand that yet. This is what H.S. Theo has been suggesting. Option 1 and 2 both have the „issue“ that people might „greenwash“ things by simply slapping trusted: at the top of a file. But that can always be done anyways and at least it is explicit and searchable. As mentioned, using trusted now has the meaning that it is user verified so it always needs extra caution. Personally I prefer option 2 because it is a lot more consistent: There are no special cases. Until now, I did not discuss whether or not all of this should apply only to extern(C) function declarations without code or all function declarations without code. This is because it is a separate point. Both are possible and have pros and cons: If it applies to extern(C) function declarations without code only, then we can still have safe extern(D) declarations. They are not verified by the compiler but you get linker errors instead because safe is part of the mangling. The benefit is that you get more safe code by default, the drawback is that it relies on linker errors instead of compiler errors and that it introduces a special case. If it applies to all function declarations without code, also extern(D) function declarations without bodies need to be annotated with trusted explicitly if they are safe or trusted. If they are system, we still get linker errors due to name mangling. The benefit is that it is a very consistent solution, no special cases are needed. The drawback is that it requires more manual effort to add the additional annotations (but then, don't we want to encourage people to explicitly annotate things anyways...?). Personally, I prefer the second option because of its simplicity. Also you mentioned somewhere that linker errors are not that helpful to the user and I couldn’t agree more. Also I personally don't use extern(D) function declarations without bodies a lot, so the drawback is negligible for me. Others' experiences might be different though. Steven actually made a proposal regarding creating 2 different manglings for extern(C) functions that are implemented in D. Regardless of which of the solutions is taken, this could provide the same benefits that we have for extern(D) functions (linker errors if safety does not match in the mangling). However, it sounds like a complicated solution (in an answer to him, you already mentioned that there might be technical difficulties regarding some object formats, debugging symbols, etc.) and I am not sure it's worth it. It also makes swapping out the libraries a bit weird: if you use an actual C library, it will always link but if swap to a library implemented in D, it only links if the saftey mangling matches. Coming back to the original point: I think it is really important that we give safe, trusted and sytem their clear meaning (as explained above) also for declarations without bodies. In my opinion, we should choose the second option in both of the questions I presented.
May 24 2020
next sibling parent reply aliak <something something.com> writes:
On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 does not work). But I admit that it is still a bit weird to 
 have 2 different defaults.
Is that any more or less weirder than having functions inferred with different attributes based on context?
May 24 2020
parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Sunday, 24 May 2020 at 11:25:06 UTC, aliak wrote:
 On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 does not work). But I admit that it is still a bit weird to 
 have 2 different defaults.
Is that any more or less weirder than having functions inferred with different attributes based on context?
What exactly are you referring to?
May 24 2020
parent reply aliak <something something.com> writes:
On Sunday, 24 May 2020 at 11:30:53 UTC, Johannes Loher wrote:
 On Sunday, 24 May 2020 at 11:25:06 UTC, aliak wrote:
 On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 does not work). But I admit that it is still a bit weird to 
 have 2 different defaults.
Is that any more or less weirder than having functions inferred with different attributes based on context?
What exactly are you referring to?
Attribute inference by D, specifically template functions. The attributes are inferred based on context (I don't know the exact algorithm). So a function f(T)(T) when called can maybe be pure, maybe safe, maybe not?
May 24 2020
parent Johannes Loher <johannes.loher fg4f.de> writes:
On Sunday, 24 May 2020 at 12:14:13 UTC, aliak wrote:
 On Sunday, 24 May 2020 at 11:30:53 UTC, Johannes Loher wrote:
 On Sunday, 24 May 2020 at 11:25:06 UTC, aliak wrote:
 On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 does not work). But I admit that it is still a bit weird to 
 have 2 different defaults.
Is that any more or less weirder than having functions inferred with different attributes based on context?
What exactly are you referring to?
Attribute inference by D, specifically template functions. The attributes are inferred based on context (I don't know the exact algorithm). So a function f(T)(T) when called can maybe be pure, maybe safe, maybe not?
From what I understand, it does not depend on the context but on the template parameters you pass to the template. I agree that it might be a bit confusing at first, but it makes sense if you realize that templates themselves are not functions but something that can generate functions (and other constructs) from compile time parameters. Why shouldn't the attributes of a generated function be able to depend on the parameters being passed to the template? Basically everything else can depend on them, too. Automatically inferring the attributes is just a very convenient way to do that. But yeah, it's not 100% consistent that they are not also inferred for regular functions (some people have been arguing for that). However, at least for templates, there is a very good reason for the difference (or at least the fact that attributes are inferred for templates): if that was not the case, it would basically be impossible to write generic code that works with all the attribute combinations. However, the very purpose of templates is to enable writing generic code. They wouldn't make that much sense if that capability was strongly limited. On the other hand, having different safety defaults for bodiless function declarations and regular faction declarations does not have such a big benefit, especially when considering the fact, that we can have the same defaults but make safe (default or explicit) bodiless function declarations a compiler error. If we ignore that option for some reason, it would only be dort of a necessity in order to prevent people from shooting them selves in the foot without even noticing. But there is no inherent value in the difference, it doesn't enable anything. That said, I'd still prefer this variant over what DIP1028 does currently. It's just that I think the other option is even better because it is more consistent.
May 24 2020
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/24/20 6:40 AM, Johannes Loher wrote:
 Steven actually made a proposal regarding creating 2 different manglings 
 for extern(C) functions that are implemented in D. Regardless of which 
 of the solutions  is taken, this could provide the same benefits that we 
 have for extern(D) functions (linker errors if  safety does not match in 
 the mangling). However, it sounds like a complicated solution (in an 
 answer to him, you already mentioned that there might be technical 
 difficulties regarding some object formats, debugging symbols, etc.) and 
 I am not sure it's worth it. It also makes swapping out the libraries a 
 bit weird: if you use an actual C library, it will always link but if 
 swap to a library implemented in D, it only links if the  saftey 
 mangling matches.
I don't think the technical difficulties make it impossible. If you can't point at the same address, point at a noop/jmp before the real address. This isn't hard. In terms of actual C libraries, only the unsafe symbol will be defined (naturally), so marking the extern(C) function safe (or assuming it's safe by default) is going to fail to link. Note that "linker errors if safety does not match" is not entirely accurate. This is ONLY the case if the prototype is safe and the implementation is not. In all other cases, the program will link. Next, I'd say that linker errors are reasonable for C functions -- they are not mangled, so the name is obvious (for extern(C) safe functions I would propose a really simple mangling like _d_safe_functionname). Plus, users who declare extern(C) prototypes already have to deal with linker errors because they have to name their prototypes correctly. But I don't anticipate anyone taking up this idea (I lack the skills), it already seems DOA based on Walter's position. -Steve
May 24 2020
prev sibling parent Juraj Mojzis <junk vec4.xyz> writes:
On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 2. Make  safe the default for _all_ declarations but still make 
  safe on declarations without code a compile error (because the 
 compiler cannot verify it). This means that annotating function 
 declarations without code with  system or  trusted is now 
 mandatory. This is what Timon has been arguing for if I 
 understood him correctly.
If this DIP is going to be implemented "as is", this should be added in the future. It could also be a warning, but that is not something that D often does. With this addition, not having a special case for extern(), is a positive.
May 26 2020
prev sibling next sibling parent ag0aep6g <anonymous example.com> writes:
On 24.05.20 10:55, Walter Bright wrote:
 I infer your position is the idea that putting  trusted on the 
 declarations isn't greenwashing, while  safe is.
That's just arguing semantics. Your intended meaning of "greenwashing" is apparently slightly different from Timon's. You can call this greenwashing: trusted extern (C) void whatevs(parameters); And this: safe extern (C) void whatevs(parameters); And then you must also call this greenwashing when compiled with `-preview=safedefault`: extern (C) void whatevs(parameters); They all do the same thing. But the first one is the most preferable form, because it shows up when you search for " trusted" which is our "here be dragons" sign. The second one should be deprecated in favor of the first one. That would move us one baby step closer to the ideal: " safe code is memory safe, unless someone made a mistake with trusted." The third one moves us one step further away from that ideal. That's the wrong direction. It's greenwashing by default. And since greenwashing is bad, greenwashing by default means bad by default.
May 24 2020
prev sibling next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 24 May 2020 at 08:55:32 UTC, Walter Bright wrote:
 I can't see a practical difference between:

  safe extern (C) void whatevs(parameters);
  trusted extern (C) void whatevs(parameters);

 Both require that whatevs() provide a safe interface.
Remember that D has reflection. If we ever do a reflection check on these, trusted stands out better there than safe. Of course it could also just return null in reflection meaning "default applied" and that could be detected as well. Otherwise it is the same, yes, but this reflection thing is a small nice benefit.
May 24 2020
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 24.05.20 10:55, Walter Bright wrote:
 I infer your position is the idea that putting  trusted on the 
 declarations isn't greenwashing, while  safe is.
 ...
It's only greenwashing if it's misleading. Putting safe is a lie, putting trusted is honest.
 I can't see a practical difference between:
 
  safe extern (C) void whatevs(parameters);
  trusted extern (C) void whatevs(parameters);
 
 Both require that whatevs() provide a safe interface. The difference 
 between them is in the implementation of those functions, not the 
 interface. Since the D compiler cannot see those implementations, they 
 are immaterial to the compiler and user.
Sure, that's the point. Your safe by default DIP in practice makes certain declarations trusted by default. safe is a fine default. trusted is a horrible default. That's why your DIP claims it is for safe by default (and not trusted by default). Except in this one weird special case, where it introduces trusted by default.
May 24 2020
next sibling parent reply Johannes T <isrvoid gmail.com> writes:
On Monday, 25 May 2020 at 00:56:04 UTC, Timon Gehr wrote:
 [..]
After thinking about it, Walter ultimately made the right decision, leading to overall higher safety and code quality. We all agree that making extern C safe is incorrect. It's also meaningless. Even if you were to verify the safety of a specific version of your binding, it can't be known what's loaded at runtime. It's not the compiler's concern. safe extern shall be an error. We might additionally make an exception and make all extern C system. It would be correct for the declarations, but inconsistent in regard to default safety. It doesn't affect the outcome. Let's say we'll go with system, it gives us a bit more freedom. We hit compile, our now safe wrappers are errors. We are most likely to do one of two things, depending on the module failing to compile. If it mostly wrappers, we slap trusted: at the top. If there are just a few functions calling C, we only annotate those with trusted. Let's be real, we probably won't begin checking and annotating the trustworthiness of the C functions. An individual programmer might, but not on average. We are formally correct, but trusted can't be trusted at this point. It has lost its meaning. We now have to check all foreign trusted code, which we probably won't. We could have moved the problem one level down and slapped trusted: on top of the system declarations. Now there is a bunch of safe code using them. It doesn't change much. trusted: is easier to grep, but we won't put everything down and begin trustworthily annotate extern C. So yeah, I do believe Walter was right. safe on extern is formally incorrect but leads to safer code because trusted still has its power.
May 25 2020
next sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Monday, 25 May 2020 at 09:25:52 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 00:56:04 UTC, Timon Gehr wrote:
 [..]
After thinking about it, Walter ultimately made the right decision, leading to overall higher safety and code quality. We all agree that making extern C safe is incorrect. It's also meaningless. Even if you were to verify the safety of a specific version of your binding, it can't be known what's loaded at runtime. It's not the compiler's concern. safe extern shall be an error. We might additionally make an exception and make all extern C system. It would be correct for the declarations, but inconsistent in regard to default safety. It doesn't affect the outcome. Let's say we'll go with system, it gives us a bit more freedom. We hit compile, our now safe wrappers are errors. We are most likely to do one of two things, depending on the module failing to compile. If it mostly wrappers, we slap trusted: at the top. If there are just a few functions calling C, we only annotate those with trusted. Let's be real, we probably won't begin checking and annotating the trustworthiness of the C functions. An individual programmer might, but not on average. We are formally correct, but trusted can't be trusted at this point. It has lost its meaning. We now have to check all foreign trusted code, which we probably won't. We could have moved the problem one level down and slapped trusted: on top of the system declarations. Now there is a bunch of safe code using them. It doesn't change much. trusted: is easier to grep, but we won't put everything down and begin trustworthily annotate extern C. So yeah, I do believe Walter was right. safe on extern is formally incorrect but leads to safer code because trusted still has its power.
So basically you are saying we should do it the way it is described in the DIP because otherwise it will lead developers to incorrect usage of trusted (just slapping it on declarations whiteout actually checking the implementation) thus making it loose its „power“ which leads to less safe code. But with the DIP in its current form, we make safe lose its meaning and power, which is much worse in my opinion. It makes the statement „ safe code cannot create memory corruptions except for trusted code“ wrong (it already is wrong now but it really shouldn’t be!). What you are describing is really just an admit of defeat. Then why still bother with safe at all if it doesn't give us any meaningful guarantees? I don't think we are at that point (yet).
May 25 2020
parent reply Johannes T <isrvoid gmail.com> writes:
On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote:
 [..]
 But with the DIP in its current form, we make  safe lose its 
 meaning and power, which is much worse in my opinion.
 [..]
The alternative, not making extern safe, would result in more untrustworthy trusted code we have to worry about. It's a vicious circle. I try to relax my view on extern annotations. They are system. We *should* go ahead and diligently mark with trusted. From experience, it doesn't normally happen. I don't like safe extern, but it seems like the lesser evil. Walter got a lot of flak. I tried to retrace his thoughts and see the merits.
May 25 2020
next sibling parent reply Zoadian <no no.no> writes:
On Monday, 25 May 2020 at 11:40:46 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote:
 [..]
 But with the DIP in its current form, we make  safe lose its 
 meaning and power, which is much worse in my opinion.
 [..]
The alternative, not making extern safe, would result in more untrustworthy trusted code we have to worry about. It's a vicious circle. I try to relax my view on extern annotations. They are system. We *should* go ahead and diligently mark with trusted. From experience, it doesn't normally happen. I don't like safe extern, but it seems like the lesser evil. Walter got a lot of flak. I tried to retrace his thoughts and see the merits.
there is no such thing as a trustworthy trusted. not with how the whole safety system work now. you can break previously verified trusted code by just writing safe code today.
May 25 2020
next sibling parent Panke <tobias pankrath.net> writes:
On Monday, 25 May 2020 at 12:22:25 UTC, Zoadian wrote:
 there is no such thing as a trustworthy  trusted. not with how 
 the whole safety system work now.
 you can break previously verified  trusted code by just writing 
  safe code today.
Do you have an example of code that was correctly labeled trusted in DMD version X, which may not be labeled trusted in DMD version Y because of changes to safe/ trusted / system?
May 25 2020
prev sibling parent Johannes T <isrvoid gmail.com> writes:
On Monday, 25 May 2020 at 12:22:25 UTC, Zoadian wrote:
 [..]
 there is no such thing as a trustworthy  trusted.
 [..]
Sorry, my bad. I meant the quality of trusted annotations would decline.
May 25 2020
prev sibling next sibling parent Johannes Loher <johannes.loher fg4f.de> writes:
On Monday, 25 May 2020 at 11:40:46 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote:
 [..]
 But with the DIP in its current form, we make  safe lose its 
 meaning and power, which is much worse in my opinion.
 [..]
The alternative, not making extern safe, would result in more untrustworthy trusted code we have to worry about. It's a vicious circle. I try to relax my view on extern annotations. They are system. We *should* go ahead and diligently mark with trusted. From experience, it doesn't normally happen. I don't like safe extern, but it seems like the lesser evil. Walter got a lot of flak. I tried to retrace his thoughts and see the merits.
From my perspective it is really simple: Either we have a strict safety system (with exactly one escape hatch) or we don't. At the moment we don't because you can have safe extern(C) declarations but with this DIP it becomes even worse as it is more likely that safe promises are broken without anybody noticing. If you and more importantly Walter and Atila really think this is the lesser evil and that it is worth dropping the strict safety system with exactly one controlled escape hatch for this, then we also need to clarify what safe means after that. We can definitely not claim that it means machine verified except for trusted because extern(C) functions are another escape hatch. The question has been asked in this thread a few times already: What does safe actually mean? How to „sell“ / „advertise“ it?
May 25 2020
prev sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 25 May 2020 at 11:40:46 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote:
 [..]
 But with the DIP in its current form, we make  safe lose its 
 meaning and power, which is much worse in my opinion.
 [..]
The alternative, not making extern safe, would result in more untrustworthy trusted code we have to worry about. It's a vicious circle.
Wrong. The quantity of untrustworthy code remains the same, but with DIP1028 (at least in the current form) the compiler sweeps the previously system code under the rug and makes it harder for those who care about safety to trust safe. safe must mean only one thing: compiler verified. Or otherwise needing less manual review. system and trusted means that code review should be prioritized. safe non-extern (D) marked either by the programmer, or the implicitly by the compiler should be disallowed as it *is* greenwashing.
 I try to relax my view on extern annotations. They are  system. 
 We *should* go ahead and diligently mark with  trusted. From 
 experience, it doesn't normally happen.
It didn't happen, because it didn't need to. Naturally, most things go through the path of least resistance. Most developers are coming from other languages where they have never had the requirement to write safe code [external pressure]. Also previously, safe wasn't the default for D function definitions, so there was less [internal pressure] to do so. With safe being the default of function definitions, it's more difficult to leave code as system (of course, modulo trusted).
 I don't like  safe extern, but it seems like the lesser evil.
No, safe extern is the worst possible option! It basically makes safe meaningless.
 Walter got a lot of flak. I tried to retrace his thoughts and 
 see the merits.
On several occasions (e.g. on Reddit) I have defended Walter from unfair accusations, however, in this case, he's rightfully criticized. He seems to think that he's taking an unpopular decision for the greater good, but that's not the case. safe-by-default on D function definitions could be considered an unpopular decision for the greater good. Implicitly safe non-extern(D) functions is greenwashing, where the responsibility for the action is removed from the developer by a compiler switch. That's basically negating all the benefits of safe-by-default on function definitions.
May 25 2020
prev sibling next sibling parent reply Zoadian <no no.no> writes:
On Monday, 25 May 2020 at 09:25:52 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 00:56:04 UTC, Timon Gehr wrote:
 [..]
After thinking about it, Walter ultimately made the right decision, leading to overall higher safety and code quality. We all agree that making extern C safe is incorrect. It's also meaningless. Even if you were to verify the safety of a specific version of your binding, it can't be known what's loaded at runtime. It's not the compiler's concern. safe extern shall be an error. We might additionally make an exception and make all extern C system. It would be correct for the declarations, but inconsistent in regard to default safety. It doesn't affect the outcome. Let's say we'll go with system, it gives us a bit more freedom. We hit compile, our now safe wrappers are errors. We are most likely to do one of two things, depending on the module failing to compile. If it mostly wrappers, we slap trusted: at the top. If there are just a few functions calling C, we only annotate those with trusted. Let's be real, we probably won't begin checking and annotating the trustworthiness of the C functions. An individual programmer might, but not on average. We are formally correct, but trusted can't be trusted at this point. It has lost its meaning. We now have to check all foreign trusted code, which we probably won't. We could have moved the problem one level down and slapped trusted: on top of the system declarations. Now there is a bunch of safe code using them. It doesn't change much. trusted: is easier to grep, but we won't put everything down and begin trustworthily annotate extern C. So yeah, I do believe Walter was right. safe on extern is formally incorrect but leads to safer code because trusted still has its power.
you complain about trusted losing it's meaning, but safe was ment to mean "mechanically verified memory safety". it should be forbidden to add safe to any function that can not be verified by the compiler. the compiler should do it's best to alert you of any mistakes you might make. if the compiler does silently add safe or trusted to extern(C) functions you are are to miss annotating it. that is a bad thing for everyone that actually cares about safety. you can't prevent programmers from doing stupid things. but you can at least warn them.
May 25 2020
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 25/05/2020 10:29 PM, Zoadian wrote:
 you complain about  trusted losing it's meaning, but  safe was ment to 
 mean "mechanically verified memory safety". it should be forbidden to 
 add  safe to any function that can not be verified by the compiler.
It is meant to mean that at some point it has been mechanically checked by the compiler. Either during current compilation or a prior one. Which means it has to be valid on function declarations without bodies so i.e. .di file generation works correctly which is just a generated D file, nothing special syntax of semantics wise.
May 25 2020
parent reply Zoadian <no no.no> writes:
On Monday, 25 May 2020 at 10:41:43 UTC, rikki cattermole wrote:
 On 25/05/2020 10:29 PM, Zoadian wrote:
 you complain about  trusted losing it's meaning, but  safe was 
 ment to mean "mechanically verified memory safety". it should 
 be forbidden to add  safe to any function that can not be 
 verified by the compiler.
It is meant to mean that at some point it has been mechanically checked by the compiler. Either during current compilation or a prior one. Which means it has to be valid on function declarations without bodies so i.e. .di file generation works correctly which is just a generated D file, nothing special syntax of semantics wise.
.di files _could_ just use trusted instead of safe. but for extern(D) we could at least add it to the name mangling. it's still not 100% safe, but at least you'd have to work hard to get it wrong.
May 25 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 25 May 2020 at 12:30:11 UTC, Zoadian wrote:
 On Monday, 25 May 2020 at 10:41:43 UTC, rikki cattermole wrote:
 It is meant to mean that at some point it has been 
 mechanically checked by the compiler.

 Either during current compilation or a prior one.

 Which means it has to be valid on function declarations 
 without bodies so i.e. .di file generation works correctly 
 which is just a generated D file, nothing special syntax of 
 semantics wise.
.di files _could_ just use trusted instead of safe. but for extern(D) we could at least add it to the name mangling. it's still not 100% safe, but at least you'd have to work hard to get it wrong.
It's been proposed before that safe and trusted should have the same mangling, since there's no difference between them from the calling code's perspective.
May 25 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 25 May 2020 at 12:41:01 UTC, Paul Backus wrote:
 On Monday, 25 May 2020 at 12:30:11 UTC, Zoadian wrote:
 On Monday, 25 May 2020 at 10:41:43 UTC, rikki cattermole wrote:
 It is meant to mean that at some point it has been 
 mechanically checked by the compiler.

 Either during current compilation or a prior one.

 Which means it has to be valid on function declarations 
 without bodies so i.e. .di file generation works correctly 
 which is just a generated D file, nothing special syntax of 
 semantics wise.
.di files _could_ just use trusted instead of safe. but for extern(D) we could at least add it to the name mangling. it's still not 100% safe, but at least you'd have to work hard to get it wrong.
It's been proposed before that safe and trusted should have the same mangling, since there's no difference between them from the calling code's perspective.
It may be true (of course modulo meta-programming) that it doesn't make a difference for the calling code, but I personally want have the guarantees that a function that I'm calling is truly safe (it doesn't contain or call any trusted code, transitively, nor it calls any safe code, which access global variables initialized by system static/module constructors). In my line work (blockchain smart contracts) some of the ways of how this is typically achieved include: * having a very minimal smart contract code size * having either no third-party dependencies or using one or two which are open-source and more importantly verified by multiple teams and having very high reputation * extensive code auditing by third-party teams. Depending on the circumstances, we may end up paying more for the auditing of the code, than the actual development. That said, there is no "strong"- safe today and even if there was, it would account for a tiny subset of all attack vectors that I have to care about (basically all possible logical bugs allowed in type-safe and memory-safe code), but I'm not sure how erasing the difference between safe and trusted on the interface level would help.
May 25 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 25 May 2020 at 13:14:51 UTC, Petar Kirov [ZombineDev] 
wrote:
 It may be true (of course modulo meta-programming) that it 
 doesn't make a difference for the calling code, but I 
 personally want have the guarantees that a function that I'm
doesn't make a difference for the calling code, but personally I want [to] have the guarantees that a function that I'm
 calling is truly  safe (it doesn't contain or call any  trusted 
 code, transitively, nor it calls any  safe code, which access 
 global variables initialized by  system static/module 
 constructors).
This is very far from a rigorous definition of "strong safe-ty" - but I hope it's just enough for the casual reader to understand my intention.
 In my line work (blockchain smart contracts) some of the ways 
 of how this is typically achieved include:
 * having a very minimal smart contract code size
 * having either no third-party dependencies or using one or two 
 which are open-source and more importantly verified by multiple 
 teams and having very high reputation
 * extensive code auditing by third-party teams. Depending on 
 the circumstances, we may end up paying more for the auditing 
 of the code, than the actual development.

 That said, there is no "strong"- safe today and even if there
That said, there is no "strong- safe" [in D] today and even if there
 was, it would account for a tiny subset of all attack vectors 
 that I have to care about (basically all possible logical bugs 
 allowed in type-safe and memory-safe code), but I'm not sure 
 how erasing the difference between  safe and  trusted on the 
 interface level would help.
May 25 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 25 May 2020 at 13:22:36 UTC, Petar Kirov [ZombineDev] 
wrote:
 On Monday, 25 May 2020 at 13:14:51 UTC, Petar Kirov 
 [ZombineDev] wrote:
 It may be true (of course modulo meta-programming) that it 
 doesn't make a difference for the calling code, but I 
 personally want have the guarantees that a function that I'm
doesn't make a difference for the calling code, but personally I want [to] have the guarantees that a function that I'm
 calling is truly  safe (it doesn't contain or call any 
  trusted code, transitively, nor it calls any  safe code, 
 which access global variables initialized by  system 
 static/module constructors).
This is very far from a rigorous definition of "strong safe-ty" - but I hope it's just enough for the casual reader to understand my intention.
I'm sure this is reasonable for your use-case, but I hope you can recognize that this definition of safety is far too narrow to be suitable for a general-purpose programming language (which D purports to be). Most people would like their safe code to be able to do I/O, for example, despite the fact that it necessarily involves calling system code under the hood.
May 25 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 25 May 2020 at 13:43:07 UTC, Paul Backus wrote:
 On Monday, 25 May 2020 at 13:22:36 UTC, Petar Kirov 
 [ZombineDev] wrote:
 On Monday, 25 May 2020 at 13:14:51 UTC, Petar Kirov 
 [ZombineDev] wrote:
 It may be true (of course modulo meta-programming) that it 
 doesn't make a difference for the calling code, but I 
 personally want have the guarantees that a function that I'm
doesn't make a difference for the calling code, but personally I want [to] have the guarantees that a function that I'm
 calling is truly  safe (it doesn't contain or call any 
  trusted code, transitively, nor it calls any  safe code, 
 which access global variables initialized by  system 
 static/module constructors).
This is very far from a rigorous definition of "strong safe-ty" - but I hope it's just enough for the casual reader to understand my intention.
I'm sure this is reasonable for your use-case, but I hope you can recognize that this definition of safety is far too narrow to be suitable for a general-purpose programming language (which D purports to be). Most people would like their safe code to be able to do I/O, for example, despite the fact that it necessarily involves calling system code under the hood.
I don't want to change the definition of safe in D, but would rather like if D supported strongSafe, that interested people like me could opt into. I know that worded like this it may sound like too narrow feature to add to the language (or at least not having favorable complexity/use cases ratio). So instead, I'd like to have transitive UDAs [1], a feature that has been requested by many, for various use cases ;) [1]: Basically I want to be able to implement function attributes like nogc or nothrow in user-space, but that's a long way from now, as first, we need to be able to introspect function bodies.
May 25 2020
parent Paul Backus <snarwin gmail.com> writes:
On Monday, 25 May 2020 at 15:07:19 UTC, Petar Kirov [ZombineDev] 
wrote:
 I don't want to change the definition of  safe in D, but would 
 rather like if D supported  strongSafe, that interested people 
 like me could opt into.
 I know that worded like this it may sound like too narrow 
 feature to add to the language (or at least not having 
 favorable complexity/use cases ratio).
 So instead, I'd like to have transitive UDAs [1], a feature 
 that has been requested by many, for various use cases ;)

 [1]: Basically I want to be able to implement function 
 attributes like  nogc or nothrow in user-space, but that's a 
 long way from now, as first, we need to be able to introspect 
 function bodies.
Sounds to me like a great use-case for the DMD-frontend-as-a-library project.
May 25 2020
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 25.05.20 11:25, Johannes T wrote:
  trusted can't be trusted
That's the point of trusted. ._. The code is trusted by the programmer, not the annotation by the compiler.
May 25 2020
parent reply Johannes T <isrvoid gmail.com> writes:
On Monday, 25 May 2020 at 11:52:27 UTC, Timon Gehr wrote:
 On 25.05.20 11:25, Johannes T wrote:
  trusted can't be trusted
That's the point of trusted. ._. The code is trusted by the programmer, not the annotation by the compiler.
Sorry, I phrased it poorly. I meant trusted would be used more frequently. The focus would spread and lead to less rigorous checks.
May 25 2020
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 25.05.20 14:22, Johannes T wrote:
 On Monday, 25 May 2020 at 11:52:27 UTC, Timon Gehr wrote:
 On 25.05.20 11:25, Johannes T wrote:
  trusted can't be trusted
That's the point of trusted. ._. The code is trusted by the programmer, not the annotation by the compiler.
Sorry, I phrased it poorly. I meant trusted would be used more frequently. The focus would spread and lead to less rigorous checks.
This is just not true. If the compiler forces you to decide to put either trusted or system, then you will not get more usage of trusted than if it implicitly decides for you that what you want is implicit trusted.
May 25 2020
prev sibling parent ag0aep6g <anonymous example.com> writes:
On 25.05.20 14:22, Johannes T wrote:
 Sorry, I phrased it poorly. I meant  trusted would be used more 
 frequently. The focus would spread and lead to less rigorous checks.
The focus is as wide as it's ever been. If you want to verify a program, you have to check those prototypes, whether they're explicitly trusted or implicitly safe. safe-by-default only makes them harder to find.
May 25 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/24/2020 5:56 PM, Timon Gehr wrote:
 It's only greenwashing if it's misleading. Putting  safe is a lie, putting 
  trusted is honest.
It is not honest unless the programmer actually carefully examined the interface and the documentation to determine if it is a safe interface or not. For example, labeling memcpy() with trusted is not honest. Forcing people to add uncheckable annotations is a path to convenience, not honesty.
May 25 2020
next sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Tuesday, 26 May 2020 at 01:16:49 UTC, Walter Bright wrote:
 On 5/24/2020 5:56 PM, Timon Gehr wrote:
 It's only greenwashing if it's misleading. Putting  safe is a 
 lie, putting  trusted is honest.
It is not honest unless the programmer actually carefully examined the interface and the documentation to determine if it is a safe interface or not. For example, labeling memcpy() with trusted is not honest. Forcing people to add uncheckable annotations is a path to convenience, not honesty.
This has already been repeated over and over again but I'll do it once more: If people cannot check the code (or get enough confidence in another way), they really shouldn’t annotate it with trusted but with system instead. If they annotate it with trusted anyways, it is their own fault. And at least in that case they are lying and not the compiler. Also if something bad actually happens, they can at least search for „high risk areas“ ( trusted declarations). Just consider the following situation: A (maybe inexperienced) programmer needs to call an existing C function from D code. Because no bindings exist yet, he creates them himself. He does not add any annotations regarding safety. Now he uses this function in safe code and he uses it in a way that actually creates a memory corruption (the function is not safe, but the developer does not know about that). Now let's compare the two different options: 1. With DIP1028 in its current form, the code will compile and a memory corruption will actually happen. The problem might be extremely difficult to track down for the developer because he has no clues whatsoever where to start looking. 2. With one of the solutions that were presented, the code will not compile as it is. According to your argument of „convenience“, the developer will probably just mark the function incorrectly as trusted which makes the code compile. The memory corruption will happen. However, even if the developer did not think much about potential safety issues when adding trusted to the function, he now still remembers that he did that (it was a conscious decision, even if it was a careless and lazy one). He has a clear point to start looking for the reason of the memory corruption. Do you honestly think option 1 is better?
May 25 2020
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/25/2020 7:04 PM, Johannes Loher wrote:
 Now let's compare the two different options:
 
 1. With DIP1028 in its current form, the code will compile and a memory 
 corruption will actually happen. The problem might be extremely difficult to 
 track down for the developer because he has no clues whatsoever where to start 
 looking.
 
 2. With one of the solutions that were presented, the code will not compile as 
 it is. According to your argument of „convenience“, the developer will
probably 
 just mark the function incorrectly as  trusted which makes the code compile.
The 
 memory corruption will happen. However, even if the developer did not think
much 
 about potential safety issues when adding  trusted to the function, he now
still 
 remembers that he did that (it was a conscious decision, even if it was a 
 careless and lazy one). He has a clear point to start looking for the reason
of 
 the memory corruption.
 
 Do you honestly think option 1 is better?
Yes, for reasons I carefully laid out.
 no clues whatsoever
He can look at unattributed declarations. The whole debate boils down to "is greenwashing better, more honest, more debuggable than leaving things unattributed?" No on all three accounts.
May 25 2020
next sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 On 5/25/2020 7:04 PM, Johannes Loher wrote:
[..]
 
 Do you honestly think option 1 is better?
Yes, for reasons I carefully laid out.
which fails to convince anyone because the reasoning is flawed.
 no clues whatsoever
He can look at unattributed declarations. The whole debate boils down to "is greenwashing better, more honest, more debuggable than leaving things unattributed?" No on all three accounts.
Unattended automatic greenwashing by the compiler is WORSE!
May 25 2020
prev sibling next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 On 5/25/2020 7:04 PM, Johannes Loher wrote:
 Now let's compare the two different options:
 
 1. With DIP1028 in its current form, the code will compile and 
 a memory corruption will actually happen. The problem might be 
 extremely difficult to track down for the developer because he 
 has no clues whatsoever where to start looking.
 
 2. With one of the solutions that were presented, the code 
 will not compile as it is. According to your argument of 
 „convenience“, the developer will probably just mark the 
 function incorrectly as  trusted which makes the code compile. 
 The memory corruption will happen. However, even if the 
 developer did not think much about potential safety issues 
 when adding  trusted to the function, he now still remembers 
 that he did that (it was a conscious decision, even if it was 
 a careless and lazy one). He has a clear point to start 
 looking for the reason of the memory corruption.
 
 Do you honestly think option 1 is better?
Yes, for reasons I carefully laid out.
A direct response to Andre's thorough critique of your reasoning would be appreciated.
 no clues whatsoever
He can look at unattributed declarations.
Again, as many have noted, putting a coverage problem like this on the programmer is problematic.
 The whole debate boils down to "is greenwashing better, more 
 honest, more debuggable than leaving things unattributed?" No 
 on all three accounts.
Greenwashing is bad. En masse greenwashing by the compiler, as mandated by the DIP currently, is really bad.
May 25 2020
prev sibling next sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 He can look at unattributed declarations.
The issue is, you have to know about that beforehand. In the other situation, you explicitly get warned by the compiler (by the compile error) that that something dangerous is going on. Also, this means, there are now 2 different things to look for: trusted function declarations and unannotated extern(C) declarations. The latter is much harder to search for (as Andrei also noted in his reply) and it's just much simpler to only have one escape hatch to look out for. It is also a very weird special case. safe means no memory corruption is possible except for any trusted declarations and any unannotated extern(C) declarations, wrongfully safe extern(C) declarations and wrongfully trusted extern(C) declarations. This just doesn’t sound right.
 The whole debate boils down to "is greenwashing better, more 
 honest, more debuggable than leaving things unattributed?" No 
 on all three accounts.
The question should be rephrased: „Is explicit greenwashing by the developer better, more honest, more debuggable than implicit greenwashing by the compiler?“ Yes for better and more debuggable for the reasons that have already been mentioned so many times. Regarding honesty: In your variant, the compiler is dishonest, in the other variant, the developer is dishonest. Just think about what the developer's reaction would be in the situation I described in my last post, when he actually finds the issue. In your variant, the developer will be questioning why the compiler did not help him at all with realizing that there might be a problem. It is very likely that he blames the issue on the compiler. In the other variant, he will realize that this something he had been warned about by the compiler and that he made a careless wrong decision and that it is his own fault. In the future he will be likely more careful when it comes to annotating stuff with trusted. So this even has an educational effect.
May 26 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2020 12:18 AM, Johannes Loher wrote:
 Just think about what the developer's reaction would be in the situation I 
 described in my last post, when he actually finds the issue.
 
 In your variant, the developer will be questioning why the compiler did not
help 
 him at all with realizing that there might be a problem. It is very likely
that 
 he blames the issue on the compiler.
 
 In the other variant, he will realize that this something he had been warned 
 about by the compiler and that he made a careless wrong decision and that it
is 
 his own fault. In the future he will be likely more careful when it comes to 
 annotating stuff with  trusted. So this even has an educational effect.
It is a fair point. But I am looking a bit farther than that - the team that is responsible for QAing software (sometimes it is a separate team). The QA team cannot tell the difference between correctly annotated trusted code and greenwashed code. But they can tell when code is not annotated (no, it is not harder to do, the annotations are designed to not be hidden - annotations cannot be hidden by the preprocessor nor are they propagated from imports. They have to be there, and if grep doesn't find them, they are not there. I've never had any difficulty finding the annotations belonging to a declaration). Un-annotated C declarations should be a red flag to any competent QA team. Recognizing a false trusted is a whole lot harder. BTW, one good thing that has come out of this issue is people are strongly in favor of what safe does. That bodes well for DIP1000 and the live code, for a long time I seemed to be the only one who cared about it. I also have no problem if the D style checker checked for un-annotated C declarations. That's the kind of thing it's for.
May 27 2020
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 27/05/2020 9:50 PM, Walter Bright wrote:
 BTW, one good thing that has come out of this issue is people are 
 strongly in favor of what  safe does. That bodes well for DIP1000 and 
 the  live code, for a long time I seemed to be the only one who cared 
 about it.
Most of the arguments against safe by default I have seen over the years (including from me) originate in /how/ to make it default, not in it becoming the default. For me at least, I will not ever want to use live, pointers with multiple behaviors depending on how it is used which don't change the syntax? No thanks, I thought we had learned that was a bad idea. But a head const storage class that handles lifetimes with no extra syntax thanks to DIP25/1000, yes please! I will use that.
May 27 2020
prev sibling next sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
Am 27.05.20 um 11:50 schrieb Walter Bright:
 It is a fair point. But I am looking a bit farther than that - the team
 that is responsible for QAing software (sometimes it is a separate
 team). The QA team cannot tell the difference between correctly
 annotated  trusted code and greenwashed code.
 
 But they can tell when code is not annotated (no, it is not harder to
 do, the annotations are designed to not be hidden - annotations cannot
 be hidden by the preprocessor nor are they propagated from imports. They
 have to be there, and if grep doesn't find them, they are not there.
 I've never had any difficulty finding the annotations belonging to a
 declaration).
 
 Un-annotated C declarations should be a red flag to any competent QA
 team. Recognizing a false  trusted is a whole lot harder.
This is a very specific situation. There are a lot of teams / developers that do not work in this manner. I don't know the numbers so I will make no statement about what is more common but my personal experience is a different one. Also what is the difference between your QA department telling you to correctly annotate C declarations and the compiler telling you that? If you expect people to ignore what the compiler is telling them, why do you expect them to listen to the QA department? In my opinion, the compiler actually _is_ one of the best QA departments. Also in my opinion, a competent QA department should carefully look at any trusted code /declarations. Maybe it is not a "red flag" but it is definitely something that needs to be checked with extra care.
May 27 2020
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/27/2020 3:07 AM, Johannes Loher wrote:
 This is a very specific situation. There are a lot of teams / developers
 that do not work in this manner. I don't know the numbers so I will make
 no statement about what is more common but my personal experience is a
 different one.
I've seen larger companies operate this way. Smaller ones cannot afford to.
 Also what is the difference between your QA department telling you to
 correctly annotate C declarations and the compiler telling you that?
 If you expect people to ignore what the compiler is telling them, why do
 you expect them to listen to the QA department?
The QA dept is motivated to not be taken in by greenwashing. (Back in my days at Boeing, I worked in the design department. There was an entirely separate organization call the Stress Group. Their job was to sign off on every nitpicky detail. If they signed off on something that turned out to be wrong, it was bad for their careers. Your designs did not move forward without Stress signoff. If you tried to trick Stress, that was the end of your career at Boeing.) So yeah, there's a huge difference between tricking the compiler and tricking the QA department.
 In my opinion, the
 compiler actually _is_ one of the best QA departments.
Indeed it is, and that's the whole point to safe. My motivation here is make suspicious code stand out. trusted code does not stand out so much, because it is required to exist.
 Also in my opinion, a competent QA department should carefully look at
 any  trusted code /declarations. Maybe it is not a "red flag" but it is
 definitely something that needs to be checked with extra care.
Looking at un-annotated declarations should be the first step. If it is annotated with trusted, at least there's the coffee stain on the drawing indicating that somebody looked at it.
May 27 2020
next sibling parent Johannes Loher <johannes.loher fg4f.de> writes:
Am 27.05.20 um 12:40 schrieb Walter Bright:
 
 Indeed it is, and that's the whole point to  safe. My motivation here is
 make suspicious code stand out.  trusted code does not stand out so
 much, because it is required to exist.
By that logic, wouldn't it also make more sense to implicitly default to trusted instead of safe for any unannotated D code? Following that logic, if safe is the default and somebody writes non- safe code without annotating it, it won't compile they might just slap trusted on it without actually checking it. By that they make it harder for the QA department to spot. If trusted was the default, it would compile, there would be no annotation on it an the QA department would see it is a red flag. What's the difference here?
May 27 2020
prev sibling parent RazvanN <razvan.nitu1305 gmail.com> writes:
On Wednesday, 27 May 2020 at 10:40:18 UTC, Walter Bright wrote:
 On 5/27/2020 3:07 AM, Johannes Loher wrote:
 This is a very specific situation. There are a lot of teams / 
 developers
 that do not work in this manner. I don't know the numbers so I 
 will make
 no statement about what is more common but my personal 
 experience is a
 different one.
I've seen larger companies operate this way. Smaller ones cannot afford to.
 Also what is the difference between your QA department telling 
 you to
 correctly annotate C declarations and the compiler telling you 
 that?
 If you expect people to ignore what the compiler is telling
them, why do
 you expect them to listen to the QA department?
The QA dept is motivated to not be taken in by greenwashing. (Back in my days at Boeing, I worked in the design department. There was an entirely separate organization call the Stress Group. Their job was to sign off on every nitpicky detail. If they signed off on something that turned out to be wrong, it was bad for their careers. Your designs did not move forward without Stress signoff. If you tried to trick Stress, that was the end of your career at Boeing.) So yeah, there's a huge difference between tricking the compiler and tricking the QA department.
 In my opinion, the
 compiler actually _is_ one of the best QA departments.
Indeed it is, and that's the whole point to safe. My motivation here is make suspicious code stand out. trusted code does not stand out so much, because it is required to exist.
I'm utterly confused by this. Trusted code is always suspicious and always stands out because it is not formally verified whereas safe code should be bulletproof. Why would anyone bother to manually check code that is supposed to be safe? Trusted code is the villain here; wherever you see trusted you know that the code might contain vulnerabilities because it was audited by a human and not by a machine. If you have safe code that has memory safety issues (from callinc C functions) then what's the point of safe?
 Also in my opinion, a competent QA department should carefully 
 look at
 any  trusted code /declarations. Maybe it is not a "red flag" 
 but it is
 definitely something that needs to be checked with extra care.
Looking at un-annotated declarations should be the first step. If it is annotated with trusted, at least there's the coffee stain on the drawing indicating that somebody looked at it.
May 27 2020
prev sibling parent WebFreak001 <d.forum webfreak.org> writes:
On Wednesday, 27 May 2020 at 10:07:48 UTC, Johannes Loher wrote:
 Am 27.05.20 um 11:50 schrieb Walter Bright:
 [...]
 
 Un-annotated C declarations should be a red flag to any 
 competent QA team. Recognizing a false  trusted is a whole lot 
 harder.
[...] Also in my opinion, a competent QA department should carefully look at any trusted code /declarations. Maybe it is not a "red flag" but it is definitely something that needs to be checked with extra care.
I think additionally we shouldn't make D code review harder to do than it needs to be. For a reviewer "something correct which is missing" is harder to pick up than "something wrong which is there" ( trusted/ system attributes) When I, as an incompetent but trying QA department, come across a code change introducing a new module wrapping a C library and all tests pass, I would most likely not notice that the author did not type ` system:`. If there was an ` trusted:` at the top of the file it would be picked up by D-Scanner and help in code review. And well if someone slapped ` safe:` at the top of the file... read on: Someone in this thread wrote something like " safe" should be forbidden on functions without method body and instead only use trusted or system. I think it might even be you who brought it up? I love the idea of forbidding safe when the compiler can't actually check the content of a function. safe, trusted and system could then have perfect definitions without edge cases. And you would change the mangling of safe and trusted to be the same which was also brought up in this thread.
May 27 2020
prev sibling next sibling parent Johannes Loher <johannes.loher fg4f.de> writes:
Am 27.05.20 um 11:50 schrieb Walter Bright:
 
 It is a fair point.
By the way, thank you for acknowledging that. I appreciate it.
May 27 2020
prev sibling next sibling parent Arafel <er.krali gmail.com> writes:
On 27/5/20 11:50, Walter Bright wrote:
 
 It is a fair point. But I am looking a bit farther than that - the team 
 that is responsible for QAing software (sometimes it is a separate 
 team). The QA team cannot tell the difference between correctly 
 annotated  trusted code and greenwashed code.
 
This is a most unsubstantiated assertion. If this were the case, I would have to wonder what the QA team is there for, if they can just be replaced by an automatic lint tool. *AND* even if that were so, see below:
 But they can tell when code is not annotated (no, it is not harder to 
 do, the annotations are designed to not be hidden - annotations cannot 
 be hidden by the preprocessor nor are they propagated from imports. They 
 have to be there, and if grep doesn't find them, they are not there. 
 I've never had any difficulty finding the annotations belonging to a 
 declaration).
 
Then the coder just slaps ` safe:`, ` trusted:`, or whatever greenwashing you were trying to avoid, and we're just back to square one since, per your previous assertion, QA can't notice the difference. Heck, in such an environment I might slap ` trusted:` just in case at the beginning of each and every file and live in a land of bliss thereafter!
 Un-annotated C declarations should be a red flag to any competent QA 
 team. Recognizing a false  trusted is a whole lot harder.
 
Exactly, which is why they should be disallowed by the compiler. Considering them ` system` by default is a way to do it. Another (perhaps even better) already proposed one would be to actually consider them ` safe`, but disallow ` safe` declarations where the code can't be analyzed by the compiler (it would be debatable if the mangling should be enough or not).
 BTW, one good thing that has come out of this issue is people are 
 strongly in favor of what  safe does. That bodes well for DIP1000 and 
 the  live code, for a long time I seemed to be the only one who cared 
 about it.
 
Being in favour of what safe purports to do and in favour of what it actually does are two very different things. The whole selling point of safe used to be "you're only responsible for ` trusted` code, now that's no longer true (or at least even less, if we account for the already existing holes). The direction of travel is opposite to the announced destination.
 I also have no problem if the D style checker checked for un-annotated C 
 declarations. That's the kind of thing it's for.
 
So, if you leave them for the style checker that means that you envision cases where this could be the desired behaviour, otherwise you should just let the compiler error. What would these cases look like? Because I can't imagine any.
May 27 2020
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Wednesday, 27 May 2020 at 09:50:50 UTC, Walter Bright wrote:
 Un-annotated C declarations should be a red flag to any 
 competent QA team. Recognizing a false  trusted is a whole lot 
 harder.
Is the actual problem those ` trusted:` declarations at the top of C headers? There could be a simple solution to that: Ban ` trusted:` and ` trusted { }` which apply to multiple symbols. Only allow ` trusted` to apply to a single symbol. For example: --- trusted: extern(C) void memcpy (void*, void*, size_t); extern(C) void write42 (void*); --- Error. --- trusted extern(C) void memcpy (void*, void*, size_t); trusted extern(C) void write42 (void*); --- OK, compiles. The bindings author has clearly added trusted manually to each symbol. Obviously there are escape hatches like mixins, code generation, etc.
May 27 2020
next sibling parent reply Claude <claudemr live.fr> writes:
On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic wrote:
 Is the actual problem those ` trusted:` declarations at the top 
 of C headers?

 There could be a simple solution to that:

 Ban ` trusted:` and ` trusted { }` which apply to multiple 
 symbols. Only allow ` trusted` to apply to a single symbol. For
IMO, it makes things worse. Because the careless programmer will slap trusted to every declaration (maybe with a script or find/replace macro of his editor). So now, we don't know if the annotation is greenwashing or careful examination of the definition. At least with " trusted:" and " trusted { }", the greenwashing is obvious.
May 27 2020
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Wednesday, 27 May 2020 at 13:47:46 UTC, Claude wrote:
 On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic 
 wrote:
 Is the actual problem those ` trusted:` declarations at the 
 top of C headers?

 There could be a simple solution to that:

 Ban ` trusted:` and ` trusted { }` which apply to multiple 
 symbols. Only allow ` trusted` to apply to a single symbol. For
IMO, it makes things worse. Because the careless programmer will slap trusted to every declaration (maybe with a script or find/replace macro of his editor). So now, we don't know if the annotation is greenwashing or careful examination of the definition.
The difference is when adding new symbols. If version control is used, it would be very obvious in a review whether a new trusted symbol was added or not. A diff typically shows several lines of context, and a toplevel ` trusted:` is easy to miss.
May 27 2020
prev sibling parent Zoadian <no no.no> writes:
On Wednesday, 27 May 2020 at 13:47:46 UTC, Claude wrote:
 On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic 
 wrote:
 Is the actual problem those ` trusted:` declarations at the 
 top of C headers?

 There could be a simple solution to that:

 Ban ` trusted:` and ` trusted { }` which apply to multiple 
 symbols. Only allow ` trusted` to apply to a single symbol. For
IMO, it makes things worse. Because the careless programmer will slap trusted to every declaration (maybe with a script or find/replace macro of his editor). So now, we don't know if the annotation is greenwashing or careful examination of the definition. At least with " trusted:" and " trusted { }", the greenwashing is obvious.
the problem here ist: we are talking about careless programmers. for those of us who actually care about safety it will be like: 1. ok lets write an extern(c) function. 2. forget to annotate it by accident. 3. hit compile 4. a) get an compiler error the hints that we missed that annotation or b) no error at all and we now have silently added unsafe code to our codebase. also, you can never trust that an trusted function is actually memory safe. even if it was evaluated in the past. the addition of new code (and even completely safe code) can break any assumptions that trusted fun was relying on. so a competent QA team _has_ to either check all trusted code fragments. or with this dip check all trusted and all extern functions.
May 27 2020
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/27/20 9:42 AM, Andrej Mitrovic wrote:
 On Wednesday, 27 May 2020 at 09:50:50 UTC, Walter Bright wrote:
 Un-annotated C declarations should be a red flag to any competent QA 
 team. Recognizing a false  trusted is a whole lot harder.
Is the actual problem those ` trusted:` declarations at the top of C headers? There could be a simple solution to that: Ban ` trusted:` and ` trusted { }` which apply to multiple symbols. Only allow ` trusted` to apply to a single symbol. For example:
Oh wow what an interesting idea. Thanks.
May 27 2020
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 27, 2020 6:24:25 PM MDT Andrei Alexandrescu via Digitalmars-
d-announce wrote:
 On 5/27/20 9:42 AM, Andrej Mitrovic wrote:
 On Wednesday, 27 May 2020 at 09:50:50 UTC, Walter Bright wrote:
 Un-annotated C declarations should be a red flag to any competent QA
 team. Recognizing a false  trusted is a whole lot harder.
Is the actual problem those ` trusted:` declarations at the top of C headers? There could be a simple solution to that: Ban ` trusted:` and ` trusted { }` which apply to multiple symbols. Only
 allow ` trusted` to apply to a single symbol. For example:
Oh wow what an interesting idea. Thanks.
I've argued for years that mass-applying any attribute is bad practice. It makes it way too easy to accidentally apply an attribute to a function, and experience has shown that it makes it _far_ easier to not realize that a funtion already has a particular attribute (it's definitely happened in Phobos PRs that an attribute has been mass-applied and then someone comes along later and applies it to a specific function, because they didn't think that that function had that attribute). Unfortunately, plenty of people seem to love to mass apply attributes rather than marking each function individually (presumably, because they hate having to use attributes all over the place). And regardless of whether DIP 1028 is a good idea as-is, the sad reality of the matter is that it's not uncommon even in druntime for someone to slap trusted: at the top of a module. It implies that the declarations in question were not actually checked, and it's incredibly error-prone when new declarations are added. Personally, I normally only use : with access-level modifiers, and while I like doing that (especially since I think that public and private functions should normally be segregated anyway), I'm increasingly coming closer to the conclusion that it's actually a bad idea in practice, because it's not uncommon for people to not know which access-level modifier applies - especially when looking at diffs in PRs. I would _love_ to see it become illegal to mass-apply trusted (or even attributes in general), but I have no clue how easy it would be to get such a DIP accepted or how much screaming there would be over it if it were actually accepted. - Jonathan M Davis
May 27 2020
prev sibling parent Q. Schroll <qs.il.paperinik gmail.com> writes:
On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic wrote:
 There could be a simple solution to that:

 Ban ` trusted:` and ` trusted { }` which apply to multiple 
 symbols. Only allow ` trusted` to apply to a single symbol. For 
 example:
That came to my mind immediately. I'm not entirely sure about ` trusted { }`, because it might be handy at times (I don't use ` attribute { }` often). But ` trusted:` is such that it is almost guaranteed that sooner or later it's applied to something it wasn't intended to. I have no idea if that's the case with braces. Since ` trusted` is intended to be special, I'd go the full route and ban it for any non-direct use. Mass-` trusted` makes no sense logically: If the programmer is required to investigate a function to determine it's ` safe`-ty, the time required for the investigation clearly outweighs the time marking the function ` trusted` (if the investigation has that result, of course), even if the judgment is fast. If people go through a whole lot of `extern(C)` declarations and marks them trusted individually because neither ` trusted:` nor ` trusted { }` did the job, it may occur to them it's probably not the right thing. I really hope this makes it through.
May 27 2020
prev sibling next sibling parent Gregory <g.thompson.1892 gmall.com> writes:
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 On 5/25/2020 7:04 PM, Johannes Loher wrote:
 Now let's compare the two different options:
 
 1. With DIP1028 in its current form, the code will compile and 
 a memory corruption will actually happen. The problem might be 
 extremely difficult to track down for the developer because he 
 has no clues whatsoever where to start looking.
 
 2. With one of the solutions that were presented, the code 
 will not compile as it is. According to your argument of 
 „convenience“, the developer will probably just mark the 
 function incorrectly as  trusted which makes the code compile. 
 The memory corruption will happen. However, even if the 
 developer did not think much about potential safety issues 
 when adding  trusted to the function, he now still remembers 
 that he did that (it was a conscious decision, even if it was 
 a careless and lazy one). He has a clear point to start 
 looking for the reason of the memory corruption.
 
 Do you honestly think option 1 is better?
Yes, for reasons I carefully laid out.
 no clues whatsoever
He can look at unattributed declarations. The whole debate boils down to "is greenwashing better, more honest, more debuggable than leaving things unattributed?" No on all three accounts.
I think this is what's the most frustrating part. You aren't actually taking criticism if you cherry-pick what you reply to.
May 26 2020
prev sibling parent reply Johannes T <isrvoid gmail.com> writes:
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]
Thank you very much for your patience with all the negative feedback. I get your decision to not annotate extern C with system by default. The biggest issue with extern system is that trusted would become less useful when dealing with declarations and bindings. trusted would appear more frequently. We wouldn't be able to assume that the author put in effort in assessing the trustworthiness of a declaration. More greenwashing, less safety. Those are severe drawbacks, I agree. However, as Andrei pointed out, PR is a huge problem. We need to be able to sell it with a straight face. I think this might be done by formally redefining the meaning of safety attributes when applied to extern: The safety of extern can't be assessed by the compiler. Implicit " safe" extern within safe code is accepted for the convenience of the average user. Explicit system or trusted can be added for desired behavior. I think this can be explained. It's not an acceptable solution for people who require correctness. ZombineDev can't tell an auditor that he relies on D-Scanner to find the problematic spots. I believe it's unavoidable to provide an option to change the default. It could make extern system or refuse to compile safe code calling safe extern.
May 26 2020
next sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
Am 26.05.20 um 14:20 schrieb Johannes T:
 Thank you very much for your patience with all the negative feedback. I
 get your decision to not annotate extern C with  system by default.
As much as i disagree with the decision, I am still very glad that we are at least having a discussion. I also want to thank Walter for participating here. I understand this can be difficult but it is the right thing to do.
 The biggest issue with extern  system is that  trusted would become less
 useful when dealing with declarations and bindings.  trusted would
 appear more frequently. We wouldn't be able to assume that the author
 put in effort in assessing the trustworthiness of a declaration. More
 greenwashing, less safety. Those are severe drawbacks, I agree.
 However, as Andrei pointed out, PR is a huge problem. We need to be able
 to sell it with a straight face.
 I think this might be done by formally redefining the meaning of safety
 attributes when applied to extern:
 The safety of extern can't be assessed by the compiler. Implicit " safe"
 extern within  safe code is accepted for the convenience of the average
 user. Explicit  system or  trusted can be added for desired behavior.
 I think this can be explained.
Well, the spec actually already mentions that:
 Safe External Functions

 External functions don't have a function body visible to the compiler:

  safe extern (C) void play();

 and so safety cannot be verified automatically.
 Best Practices: Explicitly set an attribute for external functions
rather than relying on default settings. I still don't think this is the correct thing to do. Instead, annotating external (at least extern(C)) functions with safe (implicitly as the default or explicitly) should be a compile error in my opinion.
 It's not an acceptable solution for people who require correctness.
 ZombineDev can't tell an auditor that he relies on D-Scanner to find the
 problematic spots. I believe it's unavoidable to provide an option to
 change the default. It could make extern  system or refuse to compile
 safe code calling  safe extern.
A switch might help but the whole purpose of this DIP is to get the defaults right. In my opinion, such a switch should be enabled by default. There is also the danger of creating 2 versions of the language that are incompatible with each other. Only giving this a quick thought, I believe it could work (you can still link libraries compiled with and without the switch) but I'm not completely sure.
May 26 2020
parent reply Panke <tobias pankrath.net> writes:
On Tuesday, 26 May 2020 at 12:38:54 UTC, Johannes Loher wrote:
 A switch might help but the whole purpose of this DIP is to get 
 the defaults right. In my opinion, such a switch should be 
 enabled by default. There is also the danger of creating 2 
 versions of the language that are incompatible with each other. 
 Only giving this a quick thought, I believe it could work (you 
 can still link libraries compiled with and without the switch) 
 but I'm not completely sure.
The bazel community has lots of such switches. Basically every new behaviour get's introduced with a --preview switch, that will turn into a --revert after some time. For each switch there is a github issue, explaining the change it detail. Why it was necessary, what the changed behaviour is, how to migrate and the timeline for this particular switch.
May 26 2020
parent reply Johannes Loher <johannes.loher fg4f.de> writes:
Am 26.05.20 um 15:10 schrieb Panke:
 
 The bazel community has lots of such switches. Basically every new
 behaviour get's introduced with a --preview switch, that will turn into
 a --revert after some time.
 
 For each switch there is a github issue, explaining the change it
 detail. Why it was necessary, what the changed behaviour is, how to
 migrate and the timeline for this particular switch.
D has the same thing (e.g. -preview=safedefault will enable this DIP). The main difference to Johannes T's suggestion is that these preview and revert switches are only a temporary measure to ease the transition. They do not create different versions of the language permanently.
May 26 2020
parent Panke <tobias pankrath.net> writes:
On Tuesday, 26 May 2020 at 13:21:08 UTC, Johannes Loher wrote:
 Am 26.05.20 um 15:10 schrieb Panke:
 
 The bazel community has lots of such switches. Basically every 
 new behaviour get's introduced with a --preview switch, that 
 will turn into a --revert after some time.
 
 For each switch there is a github issue, explaining the change 
 it detail. Why it was necessary, what the changed behaviour 
 is, how to migrate and the timeline for this particular switch.
D has the same thing (e.g. -preview=safedefault will enable this DIP). The main difference to Johannes T's suggestion is that these preview and revert switches are only a temporary measure to ease the transition. They do not create different versions of the language permanently.
What we don't have is good documentation and a clear timeline for the switches.
May 26 2020
prev sibling next sibling parent reply Panke <tobias pankrath.net> writes:
On Tuesday, 26 May 2020 at 12:20:31 UTC, Johannes T wrote:
 On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]
Thank you very much for your patience with all the negative feedback.
Yes, good think to stop once in a while and appreciate it. "To not complain is sufficient praise", does not always cut it.
May 26 2020
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2020 6:07 AM, Panke wrote:
 On Tuesday, 26 May 2020 at 12:20:31 UTC, Johannes T wrote:
 On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]
Thank you very much for your patience with all the negative feedback.
Yes, good think to stop once in a while and appreciate it. "To not complain is sufficient praise", does not always cut it.
Thank you, it is nice to see this.
May 27 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2020 5:20 AM, Johannes T wrote:
 On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]
Thank you very much for your patience with all the negative feedback. I get your decision to not annotate extern C with system by default. The biggest issue with extern system is that trusted would become less useful when dealing with declarations and bindings. trusted would appear more frequently. We wouldn't be able to assume that the author put in effort in assessing the trustworthiness of a declaration. More greenwashing, less safety. Those are severe drawbacks, I agree. However, as Andrei pointed out, PR is a huge problem. We need to be able to sell it with a straight face.
Frankly, I feel that if I could sit down with you folks, I can get the idea across what I'm trying to accomplish.
 I think this might be done by formally redefining the meaning of safety 
 attributes when applied to extern:
 The safety of extern can't be assessed by the compiler. Implicit " safe"
extern 
 within  safe code is accepted for the convenience of the average user.
Explicit 
  system or  trusted can be added for desired behavior.
 I think this can be explained.
I do, too. But apparently not by typing.
 It's not an acceptable solution for people who require correctness. ZombineDev 
 can't tell an auditor that he relies on D-Scanner to find the problematic
spots. 
 I believe it's unavoidable to provide an option to change the default. It
could 
 make extern  system or refuse to compile safe code calling  safe extern.
People who require correctness are going to have to manually audit each and every C declaration regardless. There's no way around it. A less strict auditor would focus his attention on un-annotated declarations and assume that annotated ones are annotated correctly.
May 27 2020
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 27/05/2020 10:03 PM, Walter Bright wrote:
 Frankly, I feel that if I could sit down with you folks, I can get the 
 idea across what I'm trying to accomplish.
Okay, how is your camera and mic situation? Lets do a Twitch stream, make sure there are some moderators in place as well. Open to everybody and it can be recorded by Twitch.
May 27 2020
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 27 May 2020 at 10:06:41 UTC, rikki cattermole wrote:
 On 27/05/2020 10:03 PM, Walter Bright wrote:
 Frankly, I feel that if I could sit down with you folks, I can 
 get the idea across what I'm trying to accomplish.
Okay, how is your camera and mic situation? Lets do a Twitch stream, make sure there are some moderators in place as well. Open to everybody and it can be recorded by Twitch.
Youtube streams work better in my experience.
May 27 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/27/2020 3:06 AM, rikki cattermole wrote:
 Open to everybody and it can be recorded by Twitch.
I've done the video conferencing many times over the decades. I just don't find it to be productive. Maybe it's some defect in me. They usually go something like this: "Who else is here?" "Hello ... Hello?" "Can anybody hear me? Is my mike working?" "Where's Bob? Is he connected?" "I just got my connection working again. What did I miss?" "Bob? Bob?" "Can someone go find Bob?" "Can you say that again? The audio cut out." "Bob here!" "Hmm. We lost Bob again?" "Fred, can you please turn off your motorcycle?" "Bob here again! Sorry!" "Fred? Now where's Fred? Fred?" What has always worked best for me is a face to face over coffee and/or beer. Which is why I'm missing DConf this year terribly.
May 27 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/27/20 6:22 AM, Walter Bright wrote:
 On 5/27/2020 3:06 AM, rikki cattermole wrote:
 Open to everybody and it can be recorded by Twitch.
I've done the video conferencing many times over the decades. I just don't find it to be productive. Maybe it's some defect in me. They usually go something like this:
[attempt at joke snipped] It's safe to assume things have improved over time.
May 27 2020
parent matheus <matheus gmail.com> writes:
On Wednesday, 27 May 2020 at 11:46:40 UTC, Andrei Alexandrescu 
wrote:
 On 5/27/20 6:22 AM, Walter Bright wrote:
 On 5/27/2020 3:06 AM, rikki cattermole wrote:
 Open to everybody and it can be recorded by Twitch.
I've done the video conferencing many times over the decades. I just don't find it to be productive. Maybe it's some defect in me. They usually go something like this:
[attempt at joke snipped] It's safe to assume things have improved over time.
Currently because COVID-19 we have (At least where I live) Politicians having meetings online and open to the public and it goes without any problems. I've been saying this since last year, why not go for online meetings to fix things instead of only using this Forum which sometimes posts get neglected and ideas misunderstood. Ouch this is about Technology and people don't use it the right way. Matheus.
May 27 2020
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/27/20 6:03 AM, Walter Bright wrote:
 On 5/26/2020 5:20 AM, Johannes T wrote:
 On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]
Thank you very much for your patience with all the negative feedback. I get your decision to not annotate extern C with system by default. The biggest issue with extern system is that trusted would become less useful when dealing with declarations and bindings. trusted would appear more frequently. We wouldn't be able to assume that the author put in effort in assessing the trustworthiness of a declaration. More greenwashing, less safety. Those are severe drawbacks, I agree. However, as Andrei pointed out, PR is a huge problem. We need to be able to sell it with a straight face.
Frankly, I feel that if I could sit down with you folks, I can get the idea across what I'm trying to accomplish.
Didn't work with me.
May 27 2020
prev sibling parent Gregory <g.thompson.1892 gmall.com> writes:
On Wednesday, 27 May 2020 at 10:03:21 UTC, Walter Bright wrote:
 It's not an acceptable solution for people who require 
 correctness. ZombineDev can't tell an auditor that he relies 
 on D-Scanner to find the problematic spots. I believe it's 
 unavoidable to provide an option to change the default. It 
 could make extern  system or refuse to compile safe code 
 calling  safe extern.
People who require correctness are going to have to manually audit each and every C declaration regardless. There's no way around it. A less strict auditor would focus his attention on un-annotated declarations and assume that annotated ones are annotated correctly.
A less strict auditor wouldn't have to look at un-annotated extern(C) functions if they were system by default. It's more work for them to ensure they are actually safe. The majority will be un-annotated if you leave it as safe.
May 27 2020
prev sibling parent JN <666total wp.pl> writes:
On Tuesday, 26 May 2020 at 02:04:02 UTC, Johannes Loher wrote:
 According to your argument of „convenience“, the developer will 
 probably just mark the function incorrectly as  trusted which 
 makes the code compile. The memory corruption will happen. 
 However, even if the developer did not think much about 
 potential safety issues when adding  trusted to the function, 
 he now still remembers that he did that (it was a conscious 
 decision, even if it was a careless and lazy one). He has a 
 clear point to start looking for the reason of the memory 
 corruption.
Or he'll do the right thing, and keep the function system but call it from a trusted block. Then when he's stuck debugging, he can show it to his experienced buddy and his experienced buddy will instantly look in the trusted block because it's suspicious.
May 26 2020
prev sibling parent M.M. <matus email.cz> writes:
On Tuesday, 26 May 2020 at 01:16:49 UTC, Walter Bright wrote:
 On 5/24/2020 5:56 PM, Timon Gehr wrote:
 It's only greenwashing if it's misleading. Putting  safe is a 
 lie, putting  trusted is honest.
It is not honest unless the programmer actually carefully examined the interface and the documentation to determine if it is a safe interface or not. For example, labeling memcpy() with trusted is not honest. Forcing people to add uncheckable annotations is a path to convenience, not honesty.
What is the difference of safe to trusted in that respect? Does the compiler "carefully examines" any interface or documentation? Why not simply introducing new label as a solution, something in the realm extern_safe_dont_know?
May 25 2020
prev sibling next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.
Agree completely. Not in dispute that I've seen. In the same vein, defaulting a declaration to safe doesn't make it safe. For the ultra paranoid, even the name mangling in D libraries is not to be trusted because "the implementation is not available".
 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" - 
 just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.
I don't believe that you or any other competent programmer greenwashes safety critical code. Regardless, the safety conscious must review their dependencies whatever default applies.
 3. Un-annotated declarations are easily detectable in a code 
 review.
Automating this for the transitive closure of defaulted safe functions would help. Maybe that capability is there already and I missed it?
 [snip]
 It is, in a not-at-all obvious way, safer for C declarations to 
 default to being safe.
I agree that it is not-at-all obvious. On a positive note, the DIP discussion/clarification should encourage the safety conscious to rebase code to a machine checkable form whenever feasible.
May 23 2020
next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 24 May 2020 at 06:26:56 UTC, Bruce Carneal wrote:
 On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 [snip]
 3. Un-annotated declarations are easily detectable in a code 
 review.
Automating this for the transitive closure of defaulted safe functions would help. Maybe that capability is there already and I missed it?
This tooling up to try and mitigate the damage caused by the DIP would not be necessary were Timon's model followed. After reading Timon's latest I've regained hope that this could still work out well.
May 23 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/23/2020 11:26 PM, Bruce Carneal wrote:
 I don't believe that you or any other competent programmer greenwashes safety 
 critical code.  Regardless, the safety conscious must review their
dependencies 
 whatever default applies.
That's the theory. But we do, for various reasons. I've seen it a lot over the years, at all levels of programming ability. It particularly happens when someone needs to get the code compiling and running, and the error message is perceived as a nuisance getting in the way. We should be very careful about adding nuisances to the language that make it easier to greenwash than to do the job correctly.
May 24 2020
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 24.05.20 11:10, Walter Bright wrote:
 On 5/23/2020 11:26 PM, Bruce Carneal wrote:
 I don't believe that you or any other competent programmer greenwashes 
 safety critical code.  Regardless, the safety conscious must review 
 their dependencies whatever default applies.
That's the theory. But we do, for various reasons. I've seen it a lot over the years, at all levels of programming ability. It particularly happens when someone needs to get the code compiling and running, and the error message is perceived as a nuisance getting in the way. We should be very careful about adding nuisances to the language that make it easier to greenwash than to do the job correctly.
Implicit greenwashing by the compiler is a nuisance that makes it harder to do the job correctly and easier to do the wrong thing.
May 24 2020
next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Monday, 25 May 2020 at 01:04:24 UTC, Timon Gehr wrote:
 On 24.05.20 11:10, Walter Bright wrote:
 On 5/23/2020 11:26 PM, Bruce Carneal wrote:
 I don't believe that you or any other competent programmer 
 greenwashes safety critical code.  Regardless, the safety 
 conscious must review their dependencies whatever default 
 applies.
That's the theory. But we do, for various reasons. I've seen it a lot over the years, at all levels of programming ability. It particularly happens when someone needs to get the code compiling and running, and the error message is perceived as a nuisance getting in the way. We should be very careful about adding nuisances to the language that make it easier to greenwash than to do the job correctly.
Implicit greenwashing by the compiler is a nuisance that makes it harder to do the job correctly and easier to do the wrong thing.
Yes, it would be a big nuisance. Absent a change in the DIP the safety conscious who want to continue with D will try to back out the compiler lies as best they can: additional tooling, additional code review strictures, selective rewrites, ... Not sure how that unfortunate future would play out exactly but it would not be pretty. Much much better to fix the DIP.
May 24 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/24/2020 6:04 PM, Timon Gehr wrote:
 Implicit greenwashing by the compiler is a nuisance that makes it harder to do 
 the job correctly and easier to do the wrong thing.
You and I are just going to disagree about that.
May 26 2020
parent Aliak <something something.com> writes:
On Wednesday, 27 May 2020 at 02:42:24 UTC, Walter Bright wrote:
 On 5/24/2020 6:04 PM, Timon Gehr wrote:
 Implicit greenwashing by the compiler is a nuisance that makes 
 it harder to do the job correctly and easier to do the wrong 
 thing.
You and I are just going to disagree about that.
Unfortunately science and research Doesn't agree with you. https://en.m.wikipedia.org/wiki/Default_effect
May 26 2020
prev sibling next sibling parent Arafel <er.krali gmail.com> writes:
On 24/5/20 5:28, Walter Bright wrote:
 I'd like to emphasize:
 
 1. It is not possible for the compiler to check any declarations where 
 the implementation is not available. Not in D, not in any language. 
 Declaring a declaration safe does not make it safe.
 
Here, I agree with Timon: only system and trusted should be possible, and in my personal view it should be mandatory to annotate external declarations. If not, at least default to ` system`.
 2. If un-annotated declarations cause a compile time error, it is highly 
 likely the programmer will resort to "greenwashing" - just slapping 
  safe on it. I've greenwashed code. Atila has. Bruce Eckel has. We've 
 all done it. Sometimes even for good reasons.
 
So the compiler helpfully does the "greenwashing" for us. How is that an improvement in any way?
 3. Un-annotated declarations are easily detectable in a code review.
 
No, they are not. If somebody slaps ` trusted:` at the beginning of a file, that's a conscious decision, there will be a commit for that. Fingers can be pointed and questions asked. If the compiler silently does it's highly likely that many people won't notice, and at the very least there will be plausible deniability ("sure, boss, I didn't know you had to slap ` system` to our C declarations, also, I was using safe and the compiler didn't complain").
 4. Greenwashing is not easily detectable in a code review.
 
I would say that greping for ` trusted:` is no that hard. I'm pretty sure tools like d-scanner are more than able to catch the more advanced cases.
 5. Greenwashing doesn't fix anything. The code is not safer. It's an 
 illusion, not a guarantee.
 
Yet the compiler will just do it, breaking all the promises for ` safe` code in the process. Now the whole ` safe` concept becomes the same "illusion".
 6. If someone cares to annotate declarations, it means he has at least 
 thought about it, because he doesn't need to. Hence it's more likely to 
 be correct than when greenwashed.
 
Q.E.D. How come this doesn't apply to the compiler?
 7. D should *not* make it worthwhile for people to greenwash code.
 
Then just disallow blanket greenwashing at all. You're just making the case for forbidding ` trusted:`, and that's something I'd totally support. Or make it so that function declarations need
 It is, in a not-at-all obvious way, safer for C declarations to default 
 to being safe.
It's definitely one of most not-obvious things I've ever seen here.
May 23 2020
prev sibling next sibling parent Johannes T <isrvoid gmail.com> writes:
On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:
 [..]
Thank you. I do see the picture now. When taking safety seriously, you would find unannotated declarations and audit/annotate them. It is not worthy of a compile option. An average user is, from experience, worse off with extern being implicitly system.
May 24 2020
prev sibling next sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.

 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" - 
 just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.

 3. Un-annotated declarations are easily detectable in a code 
 review.

 4. Greenwashing is not easily detectable in a code review.

 5. Greenwashing doesn't fix anything. The code is not safer. 
 It's an illusion, not a guarantee.

 6. If someone cares to annotate declarations, it means he has 
 at least thought about it, because he doesn't need to. Hence 
 it's more likely to be correct than when greenwashed.

 7. D should *not* make it worthwhile for people to greenwash 
 code.

 It is, in a not-at-all obvious way, safer for C declarations to 
 default to being safe.
Apparently, you're of the opinion it's better the compiler does the greenwashing. Got it!
May 24 2020
prev sibling next sibling parent Arine <arine1283798123 gmail.com> writes:
On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.
That's why Rust has gotten rid of declarations all together, except for C, which are implicitly marked unsafe, and they cannot be marked "safe" (there is no such feature).
 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" - 
 just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.
Having extern(C) be safe by default doesn't stop greenwashing. As others have pointed out, you are doing the greenwashing for the programmer. That's not better.
 3. Un-annotated declarations are easily detectable in a code 
 review.
Declarations annotated with trusted are even easier to detect. And arguably declarations that aren't annotated aren't that easy to detect.
 4. Greenwashing is not easily detectable in a code review.

 5. Greenwashing doesn't fix anything. The code is not safer. 
 It's an illusion, not a guarantee.

 6. If someone cares to annotate declarations, it means he has 
 at least thought about it, because he doesn't need to. Hence 
 it's more likely to be correct than when greenwashed.

 7. D should *not* make it worthwhile for people to greenwash 
 code.

 It is, in a not-at-all obvious way, safer for C declarations to 
 default to being safe.
This whole situation just makes it more obvious to me, you shouldn't be able to mark extern(C) declarations as safe or trusted at all. If that truly is your fear, it should just not be possible to mark them as safe/ trusted. 8. D should make incorrect code *more* difficult, not easier. Or are you conveniently ignoring this ideology? Code is going to break no matter what. That's the cost you are going to pay for making safe default. You can't fix that. The difference is what the aftermath will look like.
May 24 2020
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.

 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" - 
 just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.

 3. Un-annotated declarations are easily detectable in a code 
 review.

 [...]
If we were designing a new language from scratch, I would agree 100% with your reasoning. The problem is that there are un-annotated declarations in existing code that have already been reviewed, committed, and published under the assumption of system-by-default. Those declarations need to be flagged for re-review in order to avoid introducing silent safety violations to existing D projects. To address this issue, I'm working on a PR to have dmd emit a diagnostic message when it encounters an un-annotated external function declaration. Feel free to drop by and comment: https://github.com/dlang/dmd/pull/11176
May 24 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 24 May 2020 at 16:44:01 UTC, Paul Backus wrote:
 On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.

 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" 
 - just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.

 3. Un-annotated declarations are easily detectable in a code 
 review.

 [...]
If we were designing a new language from scratch, I would agree 100% with your reasoning. The problem is that there are un-annotated declarations in existing code that have already been reviewed, committed, and published under the assumption of system-by-default. Those declarations need to be flagged for re-review in order to avoid introducing silent safety violations to existing D projects.
I share your concerns on this, but disagree on the likelihood of reviews having gone by under the assumption of system by default. I doubt most people even thought about safe/ trusted/ system, and that's assuming anyone reviewed the code in the first place. A few years ago I submitted several PRs to Phobos to mark all unittests that could with safe explicitly. I'd say that was a good example of nobody reviewing them for their systemness.
May 25 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 On Sunday, 24 May 2020 at 16:44:01 UTC, Paul Backus wrote:
 If we were designing a new language from scratch, I would 
 agree 100% with your reasoning.

 The problem is that there are un-annotated declarations in 
 existing code that have already been reviewed, committed, and 
 published under the assumption of  system-by-default. Those 
 declarations need to be flagged for re-review in order to 
 avoid introducing silent safety violations to existing D 
 projects.
I share your concerns on this, but disagree on the likelihood of reviews having gone by under the assumption of system by default. I doubt most people even thought about safe/ trusted/ system, and that's assuming anyone reviewed the code in the first place. A few years ago I submitted several PRs to Phobos to mark all unittests that could with safe explicitly. I'd say that was a good example of nobody reviewing them for their systemness.
Walter's claim was that "un-annotated declarations are easily detectable in code review." The intent of my response was simply to point out that for existing D code, the opportunity for such review, and therefore for the detection of such declarations, is already in the past. In any case, your response does not change my conclusion: the compiler must warn D programmers that their declarations need to be reviewed for compatibility with safe-by-default. Whether they were reviewed before that or not makes no difference.
May 25 2020
parent reply Clarice <cl ar.ice> writes:
On Monday, 25 May 2020 at 16:47:50 UTC, Paul Backus wrote:
 the compiler must warn D programmers that their declarations 
 need to be reviewed for compatibility with  safe-by-default. 
 Whether they were reviewed before that or not makes no 
 difference.
I'm not an engineer; I'm not learned in CS. So some clarification on this would be nice: if you're pulling in C code, or any code written in an unsafe language, wouldn't the developers doing such a thing already be aware they need to do their due diligence? Foreign code isn't just going to magically appear in your codebase, right? Maybe through dependencies, but one should check those too, eh? (And if the dependency tree is needlessly large, then the language has another, maybe bigger, problem à la the NPM ecosystem.)
May 25 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 25 May 2020 at 17:09:50 UTC, Clarice wrote:
 On Monday, 25 May 2020 at 16:47:50 UTC, Paul Backus wrote:
 the compiler must warn D programmers that their declarations 
 need to be reviewed for compatibility with  safe-by-default. 
 Whether they were reviewed before that or not makes no 
 difference.
I'm not an engineer; I'm not learned in CS. So some clarification on this would be nice: if you're pulling in C code, or any code written in an unsafe language, wouldn't the developers doing such a thing already be aware they need to do their due diligence? Foreign code isn't just going to magically appear in your codebase, right? Maybe through dependencies, but one should check those too, eh? (And if the dependency tree is needlessly large, then the language has another, maybe bigger, problem à la the NPM ecosystem.)
Even if you did review your code *and* all of your dependencies, if you did it before DIP 1028 was accepted (i.e., any time in the last 10 years or so), you may have deliberately left external system functions un-annotated, because system was the default. In fact, I *know* that people have done this, because I've been running my patched version of the compiler [1] against the dmd, druntime, and phobos codebases, and I have found an enormous number of such un-annotated declarations. Here are just a few of them in druntime: https://github.com/dlang/druntime/pull/3117 Lots and lots of projects in the D ecosystem are going to need similar patches to be compatible with DIP 1028. And I don't think it's realistic to expect people to make these changes without some kind of help from the tooling. [1] https://github.com/dlang/dmd/pull/11176
May 25 2020
next sibling parent reply mw <mingwu gmail.com> writes:
On Monday, 25 May 2020 at 17:31:29 UTC, Paul Backus wrote:
 Even if you did review your code *and* all of your 
 dependencies, if you did it before DIP 1028 was accepted (i.e., 
 any time in the last 10 years or so), you may have deliberately 
 left external  system functions un-annotated, because  system 
 was the default.

 In fact, I *know* that people have done this, because I've been 
 running my patched version of the compiler [1] against the dmd, 
 druntime, and phobos codebases, and I have found an enormous 
 number of such un-annotated declarations. Here are just a few 
 of them in druntime:

 https://github.com/dlang/druntime/pull/3117

 Lots and lots of projects in the D ecosystem are going to need 
 similar patches to be compatible with DIP 1028. And I don't 
 think it's realistic to expect people to make these changes 
 without some kind of help from the tooling.

 [1] https://github.com/dlang/dmd/pull/11176
Can we declare the new D compiler version 3.0 for safe-as-default? and keep version 2.x compiler for the old system-as-default? In industries, it is a big-NO-NO to break exiting *working* code base. I've read some past threads talking about the instability of compilers (due to new feature introduced), e.g. in this on_leaving_d thread: https://forum.dlang.org/thread/ncbawciyybdksecurmsc forum.dlang.org?page=1 https://gitlab.com/mihails.strasuns/blog/blob/master/articles/on_leaving_d.md Let me quote what D's *industry users* (whose trademark is on the front page of dlang.org) have said 2 years ago: """ But when it comes to be boring and predictable, D fails terribly. You can't assume that released features will become 100% usable (no bugs, sufficient docs and tests, well-defined integration with other language features) even few years after initial release. You can't assume that next compiler upgrade won't suddenly break your project or any of its transitive dependencies. You can't assume that any compiler version will be maintained for more than few months. You can't assume there is any control over how declared vision documents get executed in practice. You can't trust any promises from language authors because they don't keep any track of those. It is anarchy driven development in all its glory. """ I understand the desire of Walter to continue innovation and I appreciate it, and I also think this change from system to safe-as default is an important step for D's future direction, as I commented in my thread yesterday: https://forum.dlang.org/post/brvexgjuqkbebizdmfjj forum.dlang.org """ -- the recent change safe as default opt-in is good thing: without a nogc phobos, nobody is going to write system level software using D. I think this safe change means the shift of D's focus from system software to application software. And I'm glad to see the the template project that dub generate starts with "app.d", which is the right direction for D. """ But on the other hand, we also need to take care of our existing D users, esp industry users, are they going to be happy about breaking their working code base with a new compiler version? For example, we can define a metric: say, how many percent of the packages on https://code.dlang.org/ will continue work as-it-is *without* to make any change with the new safe-as-default compiler. If > 95% of these packages (the 68–95–99.7 three-sigma rule of thumb https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule) require not any code change, then go ahead, release the new safe-as-default compiler as 2.093. Otherwise, I'd suggest to keep 2 versions of compiler: -- version 2.x as stable compiler for (industry) users to use in production, and will only provide bug fixes, but no new language feature (changes) to the compiler. (And I believe the 2.x version D language has already be *feature rich enough* to beat all those -- version 3.x for Walter to experiment and explore new ideas, and warn all the users that version 3 is unstable compiler feature-wise, use-it-at-your-own-risk, your code base may need major change in the future. Thoughts?
May 25 2020
parent Paul Backus <snarwin gmail.com> writes:
On Monday, 25 May 2020 at 18:39:16 UTC, mw wrote:
 Can we declare the new D compiler version 3.0 for 
  safe-as-default? and keep version 2.x compiler for the old 
  system-as-default?
safe-by-default is currently hidden behind a -preview switch. I think it's safe to assume there will be a long deprecation period before it's made the default, and even then, there will be a -revert switch to turn it off.
May 25 2020
prev sibling parent reply Clarice <cl ar.ice> writes:
On Monday, 25 May 2020 at 17:31:29 UTC, Paul Backus wrote:
 Even if you did review your code *and* all of your 
 dependencies, if you did it before DIP 1028 was accepted (i.e., 
 any time in the last 10 years or so), you may have deliberately 
 left external  system functions un-annotated, because  system 
 was the default.

 In fact, I *know* that people have done this, because I've been 
 running my patched version of the compiler [1] against the dmd, 
 druntime, and phobos codebases, and I have found an enormous 
 number of such un-annotated declarations. Here are just a few 
 of them in druntime:

 https://github.com/dlang/druntime/pull/3117

 Lots and lots of projects in the D ecosystem are going to need 
 similar patches to be compatible with DIP 1028. And I don't 
 think it's realistic to expect people to make these changes 
 without some kind of help from the tooling.

 [1] https://github.com/dlang/dmd/pull/11176
I really appreciate your contributions. (The latter of which will be quite handy.) But isn't the whole point of Walter's implementation of safe-as-default is that FFIs are going to be assumed safe? So they should still compile; that is unless you're referring to the need for re-evaluation.
May 25 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 25 May 2020 at 19:35:04 UTC, Clarice wrote:
 I really appreciate your contributions. (The latter of which 
 will be quite handy.) But isn't the whole point of Walter's 
 implementation of  safe-as-default is that FFIs are going to be 
 assumed  safe? So they should still compile; that is unless 
 you're referring to the need for re-evaluation.
Yes, that's Walter's current plan. I am hoping that he'll change his mind about allowing un-annotated FFIs to be assumed safe, but even if he doesn't, having the compiler print a warning will allow those of us who care about safety to find them and handle them properly.
May 25 2020
parent Clarice <cl ar.ice> writes:
On Monday, 25 May 2020 at 19:44:00 UTC, Paul Backus wrote:
 On Monday, 25 May 2020 at 19:35:04 UTC, Clarice wrote:
 I really appreciate your contributions. (The latter of which 
 will be quite handy.) But isn't the whole point of Walter's 
 implementation of  safe-as-default is that FFIs are going to 
 be assumed  safe? So they should still compile; that is unless 
 you're referring to the need for re-evaluation.
Yes, that's Walter's current plan. I am hoping that he'll change his mind about allowing un-annotated FFIs to be assumed safe, but even if he doesn't, having the compiler print a warning will allow those of us who care about safety to find them and handle them properly.
I'm still unsure which implementation is the right one, but I think this middle-ground you're attempting to get merged is a good compromise. So, again, thank you!
May 25 2020
prev sibling next sibling parent reply Panke <tobias pankrath.net> writes:
On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 A few years ago I submitted several PRs to Phobos to mark all 
 unittests that could with  safe explicitly. I'd say that was a 
 good example of nobody reviewing them for their  systemness.
Ideally you should be able to blindly mark every function definition with safe, because the compiler will catch you if you fall. Only if you type trusted you should need to be careful.
May 25 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 25 May 2020 at 17:01:24 UTC, Panke wrote:
 On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 A few years ago I submitted several PRs to Phobos to mark all 
 unittests that could with  safe explicitly. I'd say that was a 
 good example of nobody reviewing them for their  systemness.
Ideally you should be able to blindly mark every function definition with safe, because the compiler will catch you if you fall. Only if you type trusted you should need to be careful.
Doesn't work for templated functions since their safety might depend on the the particular instantiation (consider std.algorithm.map).
May 26 2020
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 26.05.20 13:09, Atila Neves wrote:
 On Monday, 25 May 2020 at 17:01:24 UTC, Panke wrote:
 On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 A few years ago I submitted several PRs to Phobos to mark all 
 unittests that could with  safe explicitly. I'd say that was a good 
 example of nobody reviewing them for their  systemness.
Ideally you should be able to blindly mark every function definition with safe, because the compiler will catch you if you fall. Only if you type trusted you should need to be careful.
Doesn't work for templated functions since their safety might depend on the the particular instantiation (consider std.algorithm.map).
I think the point was that annotating with safe liberally should not enable memory corruption.
May 26 2020
prev sibling parent Arine <arine1283798123 gmail.com> writes:
On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 I share your concerns on this, but disagree on the likelihood 
 of reviews having gone by under the assumption of  system by 
 default. I doubt most people even thought about 
  safe/ trusted/ system, and that's assuming anyone reviewed the 
 code in the first place.

 A few years ago I submitted several PRs to Phobos to mark all 
 unittests that could with  safe explicitly. I'd say that was a 
 good example of nobody reviewing them for their  systemness.
I don't think a community run project without a lead or team behind it is a very good example. No one is going to spend their free time reviewing code ensuring it is memory safe. I've seen it such that unit tests in phobos/druntime weren't being run on some platforms, and there were tests that failed when run.
May 25 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
The DIP is trying to accomplish (copied from its Rationale):

A. Costs of unsafe code have become ever more apparent and expensive, 
and  safe has grown more capable. Users expect safety to be opt-out, not 
opt-in.

B. Most code should be naturally safe, and system code should be 
relatively rare. It makes sense that the common case - safe code - 
should be the default and not require an annotation.

Any judgment on the regime of extern(C) functions should be judged as 
follows: "To what extent does the chosen regime of C functions push the 
rationale forward?"

The decision fails to support (A) because it keeps safety of C functions 
opt-in. It also fails to support (B) because it keeps code that does not 
require an annotation yet is not safe.

A good process would simply go back to the drawing board once the matter 
is exposed. It is very easy for reasonable people to agree that there is 
no part of rationale that justifies the decision. Either redefine the 
rationale (adding e.g. "keep some applications compilable without 
change" as an important goal), or change the handing of C functions.

The core argument recently detailed is:

A. Define "greenwashing" as a quick workaround to get code to work.
B. Describe in excruciating detail the experience with Java exceptions.
C. Establish a vague analogy with people placing  trusted on C declarations.
D. Assume without stating that the user would do "greenwashing" but the 
compiler doing it surreptitiously is somehow exempt.
E. Conclude that extern(C) code must be trusted.

This argument is already a smorgasbord of fallacies. It iterates a good 
subset of https://en.wikipedia.org/wiki/List_of_fallacies, and I am not 
kidding when I say a few edits to those pages with backreferences to 
this thread would be appropriate.

To start:

False analogy: making extern(C) function system is analogous with the 
Java exeception specification debacle.

https://en.wikipedia.org/wiki/Argument_from_analogy#False_analogy

This is a false analogy belying what seems to be a misunderstanding of 
the problem with Java exceptions: it was a maintenance problem, not an 
introduction problem.

The matter was, during maintenance, routinely code would change the set 
of exceptions they threw. So maintainers would need to touch code that 
didn't belong to them. There was also no "shrinking" of exception 
specification lists because the compiler does not enforce 
larger-than-necessary lists. So these lists would just grow forever 
during maintenance, with no real information attached and no usefulness.

None of these apply to  safe. Functions don't change their safety with 
ease, and when they do you definitely want to be informed. There's no 
growth of attributes for obvious reasons.

To the extent the analogy applies, the burden of proof is entirely on 
the DIP, and in this case the differences are so many and large, it's 
not worth making the argument by analogy in the first place.

The excruciatingly long discussion of Java exception specifications is 
an instance of another fallacy, misleading vividness:

http://nizkor.com/features/fallacies/misleading-vividness.html

It was expected that any of the bad things about exceptions would count 
as yet another good argument for the DIP. This constructs a textbook 
straw man:

https://en.wikipedia.org/wiki/Straw_man

So the argument constructs a straw man (Java exception specifications), 
discusses it at length on a long tangent (misleading vividness), to then 
conclude (by false analogy) that we don't want extern(C) functions to be 
 system.

What's missing is, of course, an explanation on how the decision 
supports the very rationale of the DIP. Which it doesn't - it factually 
works straight against it.

I decided to make this public only once the second boot fell (read on):

On 5/23/20 11:28 PM, Walter Bright wrote:
 I'd like to emphasize:
 
 1. It is not possible for the compiler to check any declarations where 
 the implementation is not available. Not in D, not in any language. 
 Declaring a declaration safe does not make it safe.
Strawman: https://en.wikipedia.org/wiki/Straw_man Nobody asked for that, and listing it as an argument is pointing at a strawman.
 2. If un-annotated declarations cause a compile time error, it is highly 
 likely the programmer will resort to "greenwashing" - just slapping 
  safe on it. I've greenwashed code. Atila has. Bruce Eckel has. We've 
 all done it. Sometimes even for good reasons.
Nirvana fallacy: https://en.wikipedia.org/wiki/Nirvana_fallacy "Electric cars still use polluting materials and processes, so let's continue making gasoline cars." "People will add workarounds to code, so let's accept the dangerous code anyway."
 3. Un-annotated declarations are easily detectable in a code review.
Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion (Um... there's no grep for "find me all un-annnotated declarations")
 4. Greenwashing is not easily detectable in a code review.
Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion (Um... grep ' safe| trusted')
 5. Greenwashing doesn't fix anything. The code is not safer. It's an 
 illusion, not a guarantee.
Shifting the burden of proof from the DIP to the reviewer: https://en.wikipedia.org/wiki/Burden_of_proof_(philosophy) The DIP needs to prove it does not perform greenwashing itself. Which it factually does - it paints all C code as safe with no user intervention. "Quod licet bovis non licet Iovis" much? That smacks of special pleading: https://en.wikipedia.org/wiki/Special_pleading Of course we have the strawman: https://en.wikipedia.org/wiki/Straw_man. Nobody asked for greenwashing.
 6. If someone cares to annotate declarations, it means he has at least 
 thought about it, because he doesn't need to. Hence it's more likely to 
 be correct than when greenwashed.
So... requiring adding annotations to extern(C) is good?
 7. D should *not* make it worthwhile for people to greenwash code.
Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion No accounting for the fact that the very built-in rule does exactly greenwashing as the post seems to define it.
 It is, in a not-at-all obvious way, safer for C declarations to default 
 to being safe.
In ensemble, all this adds up to a good Kettle logic fallacy: https://en.wikipedia.org/wiki/Kettle_logic For the record, I think adoption of this rule works against the very thing that the DIP is trying to accomplish and will also cause a serious PR blow.
May 25 2020
next sibling parent Arine <arine1283798123 gmail.com> writes:
Wowza, wow. The voice of reason returns!
May 25 2020
prev sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 25 May 2020 at 23:39:33 UTC, Andrei Alexandrescu wrote:
 [..]
Thank you, Andrei, you've put this quite eloquently. With more than 200+ replies, unfortunately, this whole discussion looks like an excessively inefficient use of the community's time. One way to resolve this stalemate (which I've proposed in another post) is to split off the point of contention from DIP1028. In other words, amend the newly proposed compiler switch `-preview=safedefault` to not change the meaning of non-extern(D) function declarations and introduce a new compiler switch `-implicitly-{safe,trusted}-extern-fn-decls` (*) that adds the automatic "greenwashing" functionality that Walter desires so much. (*) I think that `-implicitly-safe-extern-fn-decls` would be lie, but `-implicitly-trusted-extern-fn-decls` I can tolerate.
May 25 2020
parent reply NaN <divide by.zero> writes:
On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov [ZombineDev] 
wrote:
 On Monday, 25 May 2020 at 23:39:33 UTC, Andrei Alexandrescu 
 wrote:
 [..]
Thank you, Andrei, you've put this quite eloquently. With more than 200+ replies, unfortunately, this whole discussion looks like an excessively inefficient use of the community's time. One way to resolve this stalemate (which I've proposed in another post) is to split off the point of contention from DIP1028. In other words, amend the newly proposed compiler switch `-preview=safedefault` to not change the meaning of non-extern(D) function declarations and introduce a new compiler switch `-implicitly-{safe,trusted}-extern-fn-decls` (*) that adds the automatic "greenwashing" functionality that Walter desires so much.
If the greenwashing part was separated and delayed it would give time to find out if Walters hypothesis about people just doing it themselves is true.
May 26 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 26 May 2020 at 12:28:06 UTC, NaN wrote:
 On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov 
 [ZombineDev] wrote:
 [...]
If the greenwashing part was separated and delayed it would give time to find out if Walters hypothesis about people just doing it themselves is true.
-preview=safe now -revert=safe "tomorrow"
May 26 2020
next sibling parent NaN <divide by.zero> writes:
On Tuesday, 26 May 2020 at 12:51:59 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 12:28:06 UTC, NaN wrote:
 On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov 
 [ZombineDev] wrote:
 [...]
If the greenwashing part was separated and delayed it would give time to find out if Walters hypothesis about people just doing it themselves is true.
-preview=safe now -revert=safe "tomorrow"
Does that separate "safe by default" from "c functions are assumed safe"?
May 26 2020
prev sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Tuesday, 26 May 2020 at 12:51:59 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 12:28:06 UTC, NaN wrote:
 On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov 
 [ZombineDev] wrote:
 [...]
If the greenwashing part was separated and delayed it would give time to find out if Walters hypothesis about people just doing it themselves is true.
-preview=safe now -revert=safe "tomorrow"
If these are permanent, an ugly fork will have occurred in all but name. If these are temporary, we'll have gained little since the damage caused by assuming human checked routines are safe will remain after the switches are removed.
May 26 2020
prev sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.
[big snip of very long and arguably tangential Java screw-up and other]
 How does this relate to safe by default?

 Consider the common (because that's how D started out) case of:

 ----- clibrary.d --------

     T massage_data(... many parameters ...);
     ... 200 more such declarations ...

 ----- app.d ----------

     import clibrary;

     void useClibrary( ... parameters ...) {
         massage_data(parameters);
     }

 ---------------------

 This code, today, does not use annotations and it works. It's 
 been working
 for a long time. Now, we implement the amendment, and it stops 
 compiling
 because useClibrary is  safe and massage_data is  system. The 
 user is faced
 with the following alternatives:

 1. Go through 200 functions in clibrary.d and determine which 
 are  safe
 and which are  system. This is what we want them to do. We try 
 to motivate
 this with compiler error messages. Unfortunately, this is both 
 tedious and
 thoroughly impractical, as our poor user Will Not Know which 
 are safe and
 which are system. We can correctly annotate core.stdc.stdio 
 because I know
 those functions intimately. This is not true for other system C 
 APIs, and
 even less true for some third party C library we're trying to 
 interface to.
So, to condense and hopefully clarify: we can not use option 1) and system by default because "our poor user Will Not Know which are safe and which are system"? IOW, because someone unable to comment on safety would be inconvenienced we enable them to inject more code into the ecosphere by making safe the default. What am I missing?
 2. Annotate useClibrary() as  trusted or  system. While easier, 
 this causes
 all benefits to  safe by default to be lost.

 3. Wrap the call to massage_data() with:

     ()  trusted { massage_data(parameters); } ();

 If there are a lot of calls to clibrary, this is going to look 
 pretty awful.
 Nobody likes writing or reading such ugly code. It's ok here 
 and there, but
 not as a general thing.

 4. Edit clibrary.d and make the first line:

      safe:
[snip regarding greenwashing and other]
May 27 2020
prev sibling next sibling parent Paul Backus <snarwin gmail.com> writes:
On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md
I've made a draft PR to try and address the potential safety issue discussed in this thread. Any and all constructive feedback is greatly appreciated, so feel free to comment: https://github.com/dlang/dmd/pull/11176
May 22 2020
prev sibling next sibling parent Rivet <ri v.et> writes:
Perhaps not the ideal solution, but would a compiler flag, e.g. 
--strict-safe, that ensures the compiler errors on un- trusted 
non-D function prototypes be appropriate? This way, those who do 
work on mission-critical stuff can feel a tad better.
May 22 2020
prev sibling next sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
I think there is an exit to the current impasse that has been 
overlooked thus far.

The key question is, during a safety audit, how to find the 
unsafe sections of code that need to be vetted thoroughly. If DIP 
1028 is kept as is, these sections can be found by `grep 
 trusted|extern\(C`. If DIP 1028 would be amended as many argue 
for, it would be `grep  trusted`. But isn't grep a rather crude 
tool, and wouldn't the compiler be able to create more useful 
reports either way?

The compiler already has options for listing gc allocations 
(-vgc) and variables that go into thread local storage (-vtls). A 
new option could be implemented to list all  trusted functions 
and all calls into extern(C[++]) functions from within safe 
sections.

It is the calls into extern(C[++]) functions that cannot be 
guaranteed to be safe, irrespective of whether the declarations 
are marked  trusted or  safe. So listing these calls is much more 
valuable than grepping for  trusted, which can be misused as 
" trusted extern(...". If a C library has a proper D wrapper that 
defines a safe interface, only the  trusted functions in the 
wrapper would appear in the list, not the extern(C) calls that 
are made within its  trusted functions. The list could possibly 
be reduced by excluding functions that are never called, making 
it even more valuable.

 trusted is not a seal of approval. It bears no signature, nor a 
hash of the implementation. It is simply a bridge from safe 
territory into unsafe territory, and is a checkpoint that cannot 
be skipped in any certification process.

Considering this, the compiler can actually produce a hash of the 
implementation of trusted functions as part of this list. A 
certification authority can then actually use this list to 
maintain a table of sections, hashes and stamps. If a new release 
needs to renew its certification, the earlier table can be 
compared against the current list, and functions whose hash 
didn't change need not be reviewed again. Extrapolating even 
further, this could be the basis of a repository of approvals 
from various individuals and authorities representing statements 
like "I have looked at this function carefully and I believe it 
can indeed be trusted".

I think this would be a tool that adds real practical value and 
helps to reduce the cost of audits. And not the least, regarding 
the current discussion, it diminishes the importance of whether 
extern(C[++]) declarations are actually  system or  safe.

-- Bastiaan.
May 26 2020
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:
 [snipped an outline of tooling to mitigate 1028 damage]

 I think this would be a tool that adds real practical value and 
 helps to reduce the cost of audits. And not the least, 
 regarding the current discussion, it diminishes the importance 
 of whether extern(C[++]) declarations are actually  system or 
  safe.
Yes. Tooling is good and will be much appreciated if 1028 stands. Reducing the need for tooling is even better. safe: the compiler checks safe post 1028: the compiler checks, sometimes, just not in the scary parts
May 26 2020
parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:
 [snipped an outline of tooling to mitigate 1028 damage]

 I think this would be a tool that adds real practical value 
 and helps to reduce the cost of audits. And not the least, 
 regarding the current discussion, it diminishes the importance 
 of whether extern(C[++]) declarations are actually  system or 
  safe.
Yes. Tooling is good and will be much appreciated if 1028 stands. Reducing the need for tooling is even better. safe: the compiler checks
The compiler does not and cannot check inside trusted. Whether or not one requires extern(C[++]) to be behind or within trusted does not change what the compiler can or cannot check.
  safe post 1028: the compiler checks, sometimes, just not in 
 the scary parts
The amount of code that requires human auditing remains the same. What matters is how to find that code, and how to maintain the validity of the audits. -- Bastiaan.
May 26 2020
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:

  safe: the compiler checks
The compiler does not and cannot check inside trusted. Whether or not one requires extern(C[++]) to be behind or within trusted does not change what the compiler can or cannot check.j
Completely agree but my above says nothing about trusted.
  safe post 1028: the compiler checks, sometimes, just not in 
 the scary parts
The amount of code that requires human auditing remains the same. What matters is how to find that code, and how to maintain the validity of the audits.
Another distinction: pre 1028 your compilation will error out. Post 1028 it will not.
May 26 2020
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:

  safe: the compiler checks
The compiler does not and cannot check inside trusted. Whether or not one requires extern(C[++]) to be behind or within trusted does not change what the compiler can or cannot check.j
Completely agree but my above says nothing about trusted.
  safe post 1028: the compiler checks, sometimes, just not in 
 the scary parts
The amount of code that requires human auditing remains the same. What matters is how to find that code, and how to maintain the validity of the audits.
Another distinction: pre 1028 your compilation will error out. Post 1028 it will not.
Quite the opposite. Most code out there isn't marked as safe/ trusted/ system. If I add a dub dependency and don't bother with safe, I can call anything, in any way. Post 1028... nope. Unmarked system function definitions themselves won't compile.
May 26 2020
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Tuesday, 26 May 2020 at 16:20:23 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 [...]
Completely agree but my above says nothing about trusted.
 [...]
Another distinction: pre 1028 your compilation will error out. Post 1028 it will not.
Quite the opposite. Most code out there isn't marked as safe/ trusted/ system. If I add a dub dependency and don't bother with safe, I can call anything, in any way. Post 1028... nope. Unmarked system function definitions themselves won't compile.
Currently a machine checked safe function calling an unannotated extern C routine will error out during compilation. This is great as the C routine was not machine checked, and generally can not be checked. Post 1028, IIUC, the compilation will go through without complaint. This seems quite clear. What am I missing?
May 26 2020
next sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Tuesday, 26 May 2020 at 16:31:57 UTC, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during compilation. 
 This is great as the C routine was not machine checked, and 
 generally can not be checked.  Post 1028, IIUC, the compilation 
 will go through without complaint.  This seems quite clear.  
 What am I missing?
I agree that being forced to think about a trusted interface to that routine can contribute to safety. Not getting this is the price if DIP 1028 as is. Doing it the other way bears a different price. Still, it does not change the amount of code that must be vetted during an audit, either way. -- Bastiaan.
May 26 2020
parent Bruce Carneal <bcarneal gmail.com> writes:
On Tuesday, 26 May 2020 at 20:38:17 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 16:31:57 UTC, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during 
 compilation. This is great as the C routine was not machine 
 checked, and generally can not be checked.  Post 1028, IIUC, 
 the compilation will go through without complaint.  This seems 
 quite clear.  What am I missing?
I agree that being forced to think about a trusted interface to that routine can contribute to safety. Not getting this is the price if DIP 1028 as is. Doing it the other way bears a different price. Still, it does not change the amount of code that must be vetted during an audit, either way.
If you pick up 100% of the code that the post 1028 compiler lied about, then I agree. Unsafe is unsafe even if your compiler stays mum. If you pick up less than 100%, I disagree. In that situation the compiler will have "helped" you to reduce the amount of code to be audited. And finally, were the 1028 extern!(D) default changed to system, I disagree. The amount of code in non-greenwashed trusted wrappers should be smaller. (note that I discount the one time audit of stable system libraries)
May 26 2020
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/26/20 12:31 PM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated extern 
 C routine will error out during compilation. This is great as the C 
 routine was not machine checked, and generally can not be checked.  Post 
 1028, IIUC, the compilation will go through without complaint.  This 
 seems quite clear.  What am I missing?
If that's the case, it's the death of DIP 1028.
May 26 2020
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 27 May 2020 at 02:58:16 UTC, Andrei Alexandrescu 
wrote:
 On 5/26/20 12:31 PM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during 
 compilation. This is great as the C routine was not machine 
 checked, and generally can not be checked.  Post 1028, IIUC, 
 the compilation will go through without complaint.  This seems 
 quite clear.  What am I missing?
If that's the case, it's the death of DIP 1028.
Walter has confirmed that this is indeed the case. As you can read a few posts up his response to my "What am I missing?" query was "Nothing at all." Yes, it's really that bad. Your help in undoing this mistake would be greatly appreciated by me and by the authors of, literally, hundreds of posts in this thread opposing the DIP.
May 26 2020
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2020-05-27 06:59:28 +0000, Bruce Carneal said:

 Walter has confirmed that this is indeed the case.  As you can read a 
 few posts up his response to my "What am I missing?" query was "Nothing 
 at all."
 
 Yes, it's really that bad.
Will it be possible to see a report of these "greenwashed" call-chains, so that at least there is a chance to detect them? -- Robert M. Mnch http://www.saphirion.com smarter | better | faster
May 28 2020
parent Bruce Carneal <bcarneal gmail.com> writes:
On Friday, 29 May 2020 at 06:55:07 UTC, Robert M. Münch wrote:
 On 2020-05-27 06:59:28 +0000, Bruce Carneal said:

 Walter has confirmed that this is indeed the case.  As you can 
 read a few posts up his response to my "What am I missing?" 
 query was "Nothing at all."
 
 Yes, it's really that bad.
Will it be possible to see a report of these "greenwashed" call-chains, so that at least there is a chance to detect them?
There was indeed activity on the "lets at least make this visible" front until Walter withdrew 1028 about 2 hours ago. The thread title in the Announce forum is: DIP 1028 "Make safe the default" is dead
May 29 2020
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, May 26, 2020 8:58:16 PM MDT Andrei Alexandrescu via Digitalmars-d-
announce wrote:
 On 5/26/20 12:31 PM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated extern
 C routine will error out during compilation. This is great as the C
 routine was not machine checked, and generally can not be checked.  Post
 1028, IIUC, the compilation will go through without complaint.  This
 seems quite clear.  What am I missing?
If that's the case, it's the death of DIP 1028.
Walter has acknowledged the problem and seems to think that because it's the programmer's responsibility to deal with extern(C) functions correctly (since it's not possible for the compiler to do it), it's up to the programmer to go and fix any existing code that should be marked system and isn't and that having the compiler incorrectly mark extern(C) declarations as safe isn't a big problem, because programmers need to be spending the time to check them anyway. He's already created some PRs to try to fix some issues with extern(C) declarations in druntime and explicitly markingthem as system but doesn't seem to think that it's ultimately a big deal. - Jonathan M Davis
May 27 2020
parent Paul Backus <snarwin gmail.com> writes:
On Thursday, 28 May 2020 at 02:47:01 UTC, Jonathan M Davis wrote:
 Walter has acknowledged the problem and seems to think that 
 because it's the programmer's responsibility to deal with 
 extern(C) functions correctly (since it's not possible for the 
 compiler to do it), it's up to the programmer to go and fix any 
 existing code that should be marked  system and isn't and that 
 having the compiler incorrectly mark extern(C) declarations as 
  safe isn't a big problem, because programmers need to be 
 spending the time to check them anyway. He's already created 
 some PRs to try to fix some issues with extern(C) declarations 
 in druntime and explicitly markingthem as  system but doesn't 
 seem to think that it's ultimately a big deal.

 - Jonathan M Davis
I've submitted a PR [1] to fix a whole bunch of these. It's currently blocked on what appear to be irrelevant CI failures (one of which is actually acknowledged as such in the discussion on Walter's PR). No one has given it any attention. Is there something I'm doing wrong? Should I be splitting my PR by file, to make it easier to review? I have a similar PR for Phobos in the works, and it would be nice to have some feedback on this so that one doesn't get ignored too. [1] https://github.com/dlang/druntime/pull/3117
May 28 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated extern C 
 routine will error out during compilation. This is great as the C routine was 
 not machine checked, and generally can not be checked.  Post 1028, IIUC, the 
 compilation will go through without complaint.  This seems quite clear. 
What am 
 I missing?
Nothing at all. But I doubt there is much legacy non-compiling code around.
May 26 2020
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 27 May 2020 at 05:49:49 UTC, Walter Bright wrote:
 On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during 
 compilation. This is great as the C routine was not machine 
 checked, and generally can not be checked.  Post 1028, IIUC, 
 the compilation will go through without complaint.  This seems 
 quite clear.  What am I missing?
Nothing at all. But I doubt there is much legacy non-compiling code around.
The intent of my above was to illustrate the difference between pre 1028 compiler behavior and post 1028 compiler behavior. I'm not at all concerned with legacy non-compiling code of this nature. I am concerned about the change in compiler behavior. As you've confirmed, post 1028 the compiler will accept an safe routine calling in to an un-annotated extern C function without complaint. Andre, I hope this is clear enough for you.
May 26 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of this nature.
Apparently you agree it is not an actual problem.
May 27 2020
parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of 
 this nature.
Apparently you agree it is not an actual problem.
Really? I don't know if you really missed the point being made, or you're being provocative. Both seem unlikely to me. -- Bastiaan.
May 27 2020
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 27.05.20 11:34, Bastiaan Veelo wrote:
 On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of this nature.
Apparently you agree it is not an actual problem.
Really? I don't know if you really missed the point being made, or you're being provocative. Both seem unlikely to me. -- Bastiaan.
It's just selective reading and confirmation bias. Walter did not read past the quoted sentence as it successfully slaughters the straw man he set up in his previous post.
May 27 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/27/2020 2:34 AM, Bastiaan Veelo wrote:
 On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of this nature.
Apparently you agree it is not an actual problem.
Really? I don't know if you really missed the point being made, or you're being provocative. Both seem unlikely to me.
His argument was: "Currently a machine checked safe function calling an unannotated extern C routine will error out during compilation. This is great as the C routine was not machine checked, and generally can not be checked. Post 1028, IIUC, the compilation will go through without complaint. This seems quite clear. What am I missing?" I replied that it was unlikely that such legacy code existed. He replied that he was not concerned about it. I.e. working legacy code is not going break.
May 27 2020
next sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Wednesday, 27 May 2020 at 10:46:11 UTC, Walter Bright wrote:
 On 5/27/2020 2:34 AM, Bastiaan Veelo wrote:
 On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of 
 this nature.
Apparently you agree it is not an actual problem.
Really? I don't know if you really missed the point being made, or you're being provocative. Both seem unlikely to me.
His argument was: "Currently a machine checked safe function calling an unannotated extern C routine will error out during compilation. This is great as the C routine was not machine checked, and generally can not be checked. Post 1028, IIUC, the compilation will go through without complaint. This seems quite clear. What am I missing?" I replied that it was unlikely that such legacy code existed. He replied that he was not concerned about it. I.e. working legacy code is not going break.
The legacy code is not the issue, never was. It always was about unsafe code that will become safe with that DIP. Safe code is safe and DIP doesn't change that. It's all about UNSAFE code becoming magically labelled SAFE by the compiler but that is still UNSAFE in reality.
May 27 2020
prev sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 27 May 2020 at 10:46:11 UTC, Walter Bright wrote:
 On 5/27/2020 2:34 AM, Bastiaan Veelo wrote:
 On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of 
 this nature.
Apparently you agree it is not an actual problem.
Really? I don't know if you really missed the point being made, or you're being provocative. Both seem unlikely to me.
His argument was: "Currently a machine checked safe function calling an unannotated extern C routine will error out during compilation. This is great as the C routine was not machine checked, and generally can not be checked. Post 1028, IIUC, the compilation will go through without complaint. This seems quite clear. What am I missing?" I replied that it was unlikely that such legacy code existed. He replied that he was not concerned about it. I.e. working legacy code is not going break.
You continue to miss the point. Additionally, there never was any "working legacy code". As established, the pre 1080 compiler would have rejected the code. Does-not-compile != working-code.
May 27 2020
parent Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 27 May 2020 at 13:50:25 UTC, Bruce Carneal wrote:
 On Wednesday, 27 May 2020 at 10:46:11 UTC, Walter Bright wrote:
 [...]
You continue to miss the point. Additionally, there never was any "working legacy code". As established, the pre 1080 compiler would have rejected the code. Does-not-compile != working-code.
pre 1028 compiler
May 27 2020
prev sibling next sibling parent John Colvin <john.loughran.colvin gmail.com> writes:
On Wednesday, 27 May 2020 at 05:49:49 UTC, Walter Bright wrote:
 On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during 
 compilation. This is great as the C routine was not machine 
 checked, and generally can not be checked.  Post 1028, IIUC, 
 the compilation will go through without complaint.  This seems 
 quite clear.  What am I missing?
Nothing at all. But I doubt there is much legacy non-compiling code around.
The point isn't that incorrect legacy code that didn't compile now does (although that does matter a bit), it's that newly written code will compile when it shouldn't. Existing code will be full of extern(C) declarations that are implicitly and correctly system now and will become safe with dip1028, which means that when I write new safe code calling those (could be through a deep dependency chain of inferred- safety APIs, multiple dub packages...) I could easily find my new code compiling when it shouldn't. Effectively, by silently safe-ing things you can't infer, you will be changing APIs from system to trusted without any checks. Just in case there's any confusion, here's a timeline: 1. library A is written containing a dangerous but useful extern(C) declaration assuming system by default. 2. application B is written for and compiled with dip1028, safe: at the top of every file. 3. B adds a dependency on A. It continues to compile as safe, calling an unsafe C function. This seems like one of those things where it's either wrong or a showstopper.
May 27 2020
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/27/20 1:49 AM, Walter Bright wrote:
 On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated 
 extern C routine will error out during compilation. This is great as 
 the C routine was not machine checked, and generally can not be 
 checked.  Post 1028, IIUC, the compilation will go through without 
 complaint.  This seems quite clear.  What am I missing?
Nothing at all.
That means safe by default is effectively loosening D's notion of safety. This DIP must go.
May 27 2020
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 27, 2020 3:30:32 AM MDT Andrei Alexandrescu via Digitalmars-
d-announce wrote:
 On 5/27/20 1:49 AM, Walter Bright wrote:
 On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated
 extern C routine will error out during compilation. This is great as
 the C routine was not machine checked, and generally can not be
 checked.  Post 1028, IIUC, the compilation will go through without
 complaint.  This seems quite clear.  What am I missing?
Nothing at all.
That means safe by default is effectively loosening D's notion of safety. This DIP must go.
Which is exactly what most of us have been arguing for weeks (months?). It either needs to go or be ammended so that non-extern(D) declarations continue to be treated as system instead of automatically becoming safe with the DIP. The result of all of that arguing is that Walter accepted the DIP and then started this thread as his more detailed reply when there were a ton of complaints about the DIP's acceptance - and of course, you've already read and replied to his reasoning. As far as I can tell, Walter understands the issues but fundamentally disagrees with pretty much everyone else on the issue. He seems to think that weakening safe is worth doing, because it will ultimately mean that more code will be treated as safe and mechnically checked by the compiler, whereas most everyone else thinks that weakening safe is unacceptable. But since Walter managed to convince Atila, the DIP has been accepted. - Jonathan M Davis
May 27 2020
prev sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Thu, May 28, 2020 at 4:56 AM Jonathan M Davis via
Digitalmars-d-announce <digitalmars-d-announce puremagic.com> wrote:
 As far as I can tell, Walter understands the issues but fundamentally
 disagrees with pretty much everyone else on the issue.
I do not think so, the issue is, that there could be more people who agree with Walter (like me), but because we agree we do not participate.
 He seems to think
 that weakening  safe is worth doing, because it will ultimately mean that
 more code will be treated as  safe and mechnically checked by the compiler,
And I believe he is right.
 whereas most everyone else thinks that weakening  safe is unacceptable.
You make bad assumptions, who is "most everyone else"? I have only see few members of dlang arguing about that. But there is much bigger community around there.
 But since Walter managed to convince Atila, the DIP has been accepted.

 - Jonathan M Davis
So everything is OK right?
May 28 2020
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 28.05.20 10:50, Daniel Kozak wrote:
 He seems to think
 that weakening  safe is worth doing, because it will ultimately mean that
 more code will be treated as  safe and mechnically checked by the compiler,
And I believe he is right.
No, it's a false dichotomy. Weakening safe to allow more code to be safe might have been sensible if there was no trusted annotation. However, as things stand, trusted is sufficient as a tool to introduce potentially wrong assumptions about memory safety, we don't need more, especially not implicit ones. The reason why people are not using safe is partly that it is not the default, but it is mostly that their library dependencies _including Phobos_ are not properly annotated. This needs actual work to fix. If there is significant perceived value in performing safety checks in system code, we can add a new function attribute that causes non-transitive safe checks but otherwise gets treated as system. safe does not have to take this role.
May 28 2020
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 28 May 2020 10:50:44 +0200 schrieb Daniel Kozak:

 On Thu, May 28, 2020 at 4:56 AM Jonathan M Davis via
 Digitalmars-d-announce <digitalmars-d-announce puremagic.com> wrote:
 As far as I can tell, Walter understands the issues but fundamentally
 disagrees with pretty much everyone else on the issue.
I do not think so, the issue is, that there could be more people who agree with Walter (like me), but because we agree we do not participate.
You can not really assume any opinion for people who did not participate, unless you can really prove why there's a bias. I did not participate either and I do not agree with Walter. So now we can say the opinions of those who did not participate in the discussion are split 50:50 ;-) We could assume there's a slight bias of those agreeing with Walter not responding because they don't have to actively convince anyone, as the DIP has been accepted. But given how much negative feedbcak there is, it's also likely people would voice their opinion to support the decision. Really the best thing we can assume is that the opionions of those not participating are split in the same way as the ones of those who are participating. The strawpoll posted recently suggests that as well: https://www.strawpoll.me/20184671/r -- Johannes
May 28 2020
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 27 May 2020 at 05:49:49 UTC, Walter Bright wrote:
 [snip]

 Nothing at all.

 But I doubt there is much legacy non-compiling code around.
I cannot speak for all the code out there, but I know of at least one binding out there [1] that will need to get updated in light of this DIP. I also put together a wrapper [2] that sits on top of that binding (many years ago and no idea if the wrapper still works (I see no reason why the binding wouldn't), so this probably does count as legacy code) and calls the binding functions with templates and no safety annotations. So theoretically, after the DIP is accepted and without any other changes, nlopt.d becomes safe and the binding that calls it also becomes safe and any user code that calls it becomes safe. *************************** On a separate note, the current definition of safe is that you cannot call system functions. This prevents calling unannotated extern(C) prototypes from safe code. Ignoring DIP 1028 for a second, would you consider changing the rules so unannotated extern(C) prototypes are safe instead of system without any other change? Just this change by itself would mean that safe functions can call unannotated extern(C) prototypes. Obviously this by itself could be rejected for special casing. However, I think the underlying question is should they be safe (ignoring safe) or not. An argument made previously is that they should be safe and thus safe. However, I think the overwhelming belief is that they are not safe and thus should not be safe by default. From this perspective, you could think of the DIP as doing two things. 1) it makes unannotated extern(C) prototypes safe instead of system and 2) it makes all other unannotated functions safe instead of system. The big focus is clearly on So, do you think unannotated exten(C) prototypes safe? If you think they are safe, then there is no issue with making them safe by default. However, if you do not think they are safe, then it raises serious concerns to make them safe by default. [1] https://github.com/DlangScience/nlopt/blob/master/source/nlopt.d [2] https://github.com/jmh530/nloptd/blob/master/source/nloptd.d
May 27 2020
parent Gregory <g.thompson.1892 gmall.com> writes:
I'm curious what the distribution looks like between whether 
people agree that extern(C) should be  safe, or  system. People 
that think it should be  trusted, vote  safe, it's pretty much 
the same thing.

  https://www.strawpoll.me/20184671
May 27 2020
prev sibling next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Tuesday, 26 May 2020 at 16:20:23 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 [...]
The compiler does not and cannot check inside trusted. Whether or not one requires extern(C[++]) to be behind or within trusted does not change what the compiler can or cannot check.j
Completely agree but my above says nothing about trusted.
 [...]
The amount of code that requires human auditing remains the same. What matters is how to find that code, and how to maintain the validity of the audits.
Another distinction: pre 1028 your compilation will error out. Post 1028 it will not.
Quite the opposite. Most code out there isn't marked as safe/ trusted/ system. If I add a dub dependency and don't bother with safe, I can call anything, in any way. Post 1028... nope. Unmarked system function definitions themselves won't compile.
I think I see the difficulty. We both support 1028's elimination of the "I can call anything, in any way" default. I really like safe by default for machine checkable code. I don't support safe by default for extern C.
May 26 2020
prev sibling parent reply Gregory <g.thompson.1892 gmall.com> writes:
On Tuesday, 26 May 2020 at 16:20:23 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo 
 wrote:

  safe: the compiler checks
The compiler does not and cannot check inside trusted. Whether or not one requires extern(C[++]) to be behind or within trusted does not change what the compiler can or cannot check.j
Completely agree but my above says nothing about trusted.
  safe post 1028: the compiler checks, sometimes, just not in 
 the scary parts
The amount of code that requires human auditing remains the same. What matters is how to find that code, and how to maintain the validity of the audits.
Another distinction: pre 1028 your compilation will error out. Post 1028 it will not.
Quite the opposite. Most code out there isn't marked as safe/ trusted/ system. If I add a dub dependency and don't bother with safe, I can call anything, in any way. Post 1028... nope. Unmarked system function definitions themselves won't compile.
Which will just lead people to pure trusted: at the top of their code to get it to compile again, with or without extern(C) being safe by default. Then someone that uses it as dependency will mistaken think it is safe. What's to stop this kind of "greenwashing" and why is greenwashing only important to prevent when talking about extern(C) but every other code that will break from this change?
May 26 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 26 May 2020 at 17:50:58 UTC, Gregory wrote:
 Which will just lead people to pure  trusted: at the top of 
 their code to get it to compile again, with or without 
 extern(C) being  safe by default. Then someone that uses it as 
 dependency will mistaken think it is  safe. What's to stop this 
 kind of "greenwashing" and why is greenwashing only important 
 to prevent when talking about extern(C) but every other code 
 that will break from this change?
The reason extern function declarations are particularly problematic is that changing them from system-by-default to safe-by-default can cause *silent* breakage in existing, correct code. By "silent", I mean that there is no message from the compiler warning you that the memory safety of the code has been compromised. It will still compile, and you will only find out that something has gone wrong when you run into memory corruption at run-time. This undermines the entire purpose of safe, which is to automatically catch code with potential memory errors at compile time.
May 26 2020
next sibling parent reply Gregory <g.thompson.1892 gmall.com> writes:
On Tuesday, 26 May 2020 at 20:32:13 UTC, Paul Backus wrote:
 On Tuesday, 26 May 2020 at 17:50:58 UTC, Gregory wrote:
 Which will just lead people to pure  trusted: at the top of 
 their code to get it to compile again, with or without 
 extern(C) being  safe by default. Then someone that uses it as 
 dependency will mistaken think it is  safe. What's to stop 
 this kind of "greenwashing" and why is greenwashing only 
 important to prevent when talking about extern(C) but every 
 other code that will break from this change?
The reason extern function declarations are particularly problematic is that changing them from system-by-default to safe-by-default can cause *silent* breakage in existing, correct code. By "silent", I mean that there is no message from the compiler warning you that the memory safety of the code has been compromised. It will still compile, and you will only find out that something has gone wrong when you run into memory corruption at run-time. This undermines the entire purpose of safe, which is to automatically catch code with potential memory errors at compile time.
Exactly, that's what I think as well. What I'm trying to convey is that there's a lot of greenwashing that is going to happen. Code doesn't work, slapping trusted everywhere til it works is what people are going to do, it doesn't matter if extern(C) is safe or not. Greenwashing occurs no matter what. It's only natural with such a big breaking changing with a feature that you have to "trust". Consider this: Someone is writing code, it's safe cause it is now the default (yay!). They come across unannotated code while searching for an answer. They copy paste it into their source, ops compile error! Can't use X in safe code. Ok, so they annotate it as system. Oh but now their safe code can't call system code. So what do they do? Annotate it as trusted without checking it. If Walter believed greenwashing was actually a problem, then the best solution to prevent it would be to not make safe by default. If it's not that serious of a problem that he will push through safe by default, then greenwashing isn't actually a problem and extern(C) should remain system.
May 26 2020
parent Paul Backus <snarwin gmail.com> writes:
On Tuesday, 26 May 2020 at 22:47:03 UTC, Gregory wrote:
 If Walter believed greenwashing was actually a problem, then 
 the best solution to prevent it would be to not make  safe by 
 default. If it's not that serious of a problem that he will 
 push through  safe by default, then greenwashing isn't actually 
 a problem and extern(C) should remain  system.
I understand your point now, and I agree.
May 26 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2020 1:32 PM, Paul Backus wrote:
 The reason extern function declarations are particularly problematic is that 
 changing them from  system-by-default to  safe-by-default can cause *silent* 
 breakage in existing, correct code.
Can you post an example of currently compiling and correctly working code that will break? Setting aside use of __traits(compiles, ...).
May 26 2020
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 27.05.20 07:54, Walter Bright wrote:
 On 5/26/2020 1:32 PM, Paul Backus wrote:
 The reason extern function declarations are particularly problematic 
 is that changing them from  system-by-default to  safe-by-default can 
 cause *silent* breakage in existing, correct code.
Can you post an example of currently compiling and correctly working code that will break? Setting aside use of __traits(compiles, ...).
What exactly is your standard here? Are you saying we have to produce the following? - Monolithic example, API breakage does not count. - No __traits(compiles, ...) - The code has to compile under both old and new rules. - The code has to corrupt memory in safe code under new rules. This is clearly not possible, exactly because the old safe rules are stronger. But why exactly should API breakage not count?
May 27 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/27/2020 3:01 AM, Timon Gehr wrote:
 This is clearly not possible, exactly because the old  safe rules are
stronger. 
Thank you. We can agree on something.
 But why exactly should API breakage not count?
I've addressed exactly this a dozen times or more, to you and others. Repeating myself has become pointless. It's fine to disagree with me. Argue that point. But don't say I didn't address it.
May 27 2020
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 27.05.20 12:51, Walter Bright wrote:
 On 5/27/2020 3:01 AM, Timon Gehr wrote:
 This is clearly not possible, exactly because the old  safe rules are 
 stronger. 
Thank you. We can agree on something. ...
I am not sure if you noticed that I agree with most of your points, just not about their relevance to the topic at hand.
 But why exactly should API breakage not count?
I've addressed exactly this a dozen times or more, to you
No. You did not. I went over all of your responses to my posts again to make sure. Why are you making this claim? I haven't made API breakage a central point to any of my previous posts and you did not address any of my criticism in any depth. As far as I can tell, the only point you engaged with was that trusted is not greenwashing.
 and others. 
I don't think you did, but I am not going to check.
 Repeating myself has become pointless.
 
 It's fine to disagree with me. Argue that point. But don't say I didn't 
 address it.
As far as I remember, you did not address this specific point, but if I had to extrapolate your response from previous points you made I would expect your opinion to be that implicitly broken APIs should be fixed by universal manual review of not explicitly annotated functions and that this is better than the compiler catching it for you because this way only people who are competent to judge which annotation should be there will notice that it is missing.
May 27 2020
prev sibling next sibling parent Claude <claudemr live.fr> writes:
On Wednesday, 27 May 2020 at 10:51:54 UTC, Walter Bright wrote:
 On 5/27/2020 3:01 AM, Timon Gehr wrote:
 I've addressed exactly this a dozen times or more, to you and 
 others. Repeating myself has become pointless.

 It's fine to disagree with me. Argue that point. But don't say 
 I didn't address it.
I'm trying to understand the logic of " safe by default for extern declaration". So I setup a simplified real-life example that I had in my mind... Let's say we have a project consisting of: - an ASM file containing several function definitions (assembly language scares people away, so it's a good example). - a .di file containing the extern(C) declarations of the former ASM functions. - and a .d file using those functions with safe code. No function is annotated. Before DIP-1028, it compiles and link without annotations. If extern(C) decl are system by default, it will not compile anymore, and a careless programmer will slap a " trusted:" at the top of the ".di" file (greenwashing), and it'll stay there. On the other hand, a careful programmer will need to annotate trusted ONLY on functions that he actually trusts AND make the project compile anyway (because he cannot afford to leave his project broken). So what does he do to make it compile ? Should he slap a " trusted:" at the beginning of the file just like the careless programmer and maybe explicitly annotate individually declarations that are actually trusted ? It's weird... However, if extern(C) decl are safe by default, he does not need to slap a " trusted" at the top of the ".di" file, he just needs to annotate individual declarations with trusted to advertise (to the QA, or his colleague or future self) that he reviewed successfully the function implementation, AND it will still compile. So that example would go in favor of Walter's point. So is it a good example? Does pinpoint the point? Because again, as the others said, it's still controversial, it's hard to sell, it's convoluted... I dunno...
May 27 2020
prev sibling parent Johannes T <isrvoid gmail.com> writes:
On Wednesday, 27 May 2020 at 10:51:54 UTC, Walter Bright wrote:
 [..]
I was reading Bastiaan's inspiring ideas about provable verification of trusted and something occurred to me. If we are serious about safety, we should consider trusted annotation on extern C harmful. It provides little value and might be misleading like a bad comment: // only call once, second call leaks void waitForClose(size_t msTimeout); The implementation was rewritten long ago, no one bothered to remove the remark. The trustworthiness of extern code can be assessed by audit. It can only be set in stone by additional tools. git can nail down the version of a verified function in the CMake list. The compiler can't know what we are linking against or what's loaded at runtime. trusted extern C is meaningless. Pre DIP1028 we had to put it in for safe code to compile. I'm not sure how system fits into this. It gets outdated the same way trusted does.
May 27 2020
prev sibling parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 27 May 2020 at 05:54:32 UTC, Walter Bright wrote:
 On 5/26/2020 1:32 PM, Paul Backus wrote:
 The reason extern function declarations are particularly 
 problematic is that changing them from  system-by-default to 
  safe-by-default can cause *silent* breakage in existing, 
 correct code.
Can you post an example of currently compiling and correctly working code that will break? Setting aside use of __traits(compiles, ...).
I am paraphrasing an earlier post of mine in this thread: https://forum.dlang.org/post/yvcjpcxfyyobqubjrrxv forum.dlang.org The breakage is that functions which are currently correctly considered system will instead be considered safe. As a result, future changes to safe code are at risk of introducing memory corruption. Strictly speaking, the breakage does not occur until those future changes are made, so it is more accurate to say that changing un-annotated extern declarations to safe-by-default will put existing code *at risk* of silent breakage. Given the prevalence of such declarations in existing code (e.g. [1]), I think it is inevitable that this risk will in fact lead to breakage in practice, so I do not consider the distinction particularly important to draw in this context. [1] https://github.com/dlang/druntime/pull/3117
May 27 2020
prev sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:

  safe: the compiler checks
The compiler does not and cannot check inside trusted. Whether or not one requires extern(C[++]) to be behind or within trusted does not change what the compiler can or cannot check.
Completely agree but my above says nothing about trusted.
But it matters. Even if your code is safe, that doesn't mean it is completely safe. Your safe code can call into somebody else's safe code that can call trusted code that calls system code. If you want to guarantee that your code is safe, you'll have to find and audit the trusted bits (as well as everything that is called from there). If extern(C) is implicitly system, you'll have to make calls trusted and they are subject to auditing. If extern(C) is implicitly safe, calls are still subject to auditing. Whatever your preference, I think the compiler can be of greater help in finding the bits that require auditing than grep can, and then the difference isn't all that important anymore. Safe wrappers around C libraries are still written the same way, audited the same way, and unsafe calls into C caught the same way. I agree that making extern(!D) system looks sound, like the way it should be. But it breaks compilation of existing code, and it is too easy to evade by slapping on an explicit trusted or (God forbid) safe. Or we could make extern(!D) to be anything other than system an error, which would be really annoying. I think that what we all strive for is to reduce the strain of coding as well as to reduce the strain of auditing. -- Bastiaan.
May 26 2020
parent Bruce Carneal <bcarneal gmail.com> writes:
On Tuesday, 26 May 2020 at 18:06:00 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo 
 wrote:

  safe: the compiler checks
The compiler does not and cannot check inside trusted. Whether or not one requires extern(C[++]) to be behind or within trusted does not change what the compiler can or cannot check.
Completely agree but my above says nothing about trusted.
But it matters. Even if your code is safe, that doesn't mean it is completely safe. Your safe code can call into somebody else's safe code that can call trusted code that calls system code. If you want to guarantee that your code is safe, you'll have to find and audit the trusted bits (as well as everything that is called from there).
Agree that the transitive closure is difficult, especially for humans.
 If extern(C) is implicitly  system, you'll have to make calls 
  trusted and they are subject to auditing. If extern(C) is 
 implicitly  safe, calls are still subject to auditing. Whatever 
 your preference, I think the compiler can be of greater help in 
 finding the bits that require auditing than grep can, and then 
 the difference isn't all that important anymore. Safe wrappers 
 around C libraries are still written the same way, audited the 
 same way, and unsafe calls into C caught the same way.
Agree. trusted doesn't mean that the code so labelled should be blindly trusted. The aim should be reduction of trusted where practical. Silently letting all extern C functions inside the tent is not a reduction.
 I agree that making extern(!D)  system looks sound, like the 
 way it should be. But it breaks compilation of existing code, 
 and it is too easy to evade by slapping on an explicit  trusted 
 or (God forbid)  safe. Or we could make extern(!D) to be 
 anything other than  system an error, which would be really 
 annoying. I think that what we all strive for is to reduce the 
 strain of coding as well as to reduce the strain of auditing.
I'm all for reducing the strain of writing correct code. As others have implied, we're dealing with, at least, two issues here: 1) any transition from opt-in to opt-out safety will break code. Some programmers will resort to greenwashing in an effort to "fix" it. Will we be better off post transition? And 2) should uncheckable code be considered safe, ever.
May 26 2020
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 28, 2020 2:50:44 AM MDT Daniel Kozak via Digitalmars-d-
announce wrote:
 On Thu, May 28, 2020 at 4:56 AM Jonathan M Davis via

 Digitalmars-d-announce <digitalmars-d-announce puremagic.com> wrote:
 As far as I can tell, Walter understands the issues but fundamentally
 disagrees with pretty much everyone else on the issue.
I do not think so, the issue is, that there could be more people who agree with Walter (like me), but because we agree we do not participate.
There may be some silent people who agree, but in all the discussions on this DIP, almost no one has agreed with Walter on this. It has not been a back and forth discussion with some on Walter's side and some against. It's been pretty much everyone against Walter. He did unfortunately manage to convince Atila, so the DIP has been accepted, but based on the discussions, I think that you may be the only person I've seen say anything positive about the DIP treating extern(C) functions as safe. The fact that safe becomes the default has garned some debate with some in favor and some against (with most seeming to be in favor), but the idea of making extern(C) declarations safe by default has been almost universally considered a bad idea by anyone who has responded on the topic. Most DIPs do not get anywhere close to this level of negative feedback.
 But since Walter managed to convince Atila, the DIP has been accepted.
So everything is OK right?
Everything is okay? Because a bad DIP got accepted? No, most definitely not. Quite the opposite. With the DIP in its current state, safe becomes a lie. The compiler no longer guarantees that safe code is memory safe so long as it doesn't call any trusted code where the programmer incorrectly marked it as trusted. Instead, the compiler blindly treats non-extern(D) declarations as safe and invisibly introduces memory safety bugs into safe code. Nothing about that is "OK." From the way things currently look, we're going to have to deal with that hole in safe in D code in the future, because the DIP has been accepted, but it adds yet another dark corner to the language of the sort that folks here tend to make fun of C++ for. Going forward, we're going to have to be explaining to people why safe code doesn't actually guarantee that code is memory safe (in spite of the fact that that's what it claims to do) and why any and all non-extern(D) declarations have to be treated with extreme caution to avoid invisibly introducing memory safety bugs into your code. Walter is very intelligent and has made many great decisions with D, but IMHO, this is definitely not one of them. - Jonathan M Davis
May 28 2020
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 28 May 2020 at 09:21:09 UTC, Jonathan M Davis wrote:
 He did unfortunately manage to convince Atila, so the DIP has 
 been accepted, but based on the discussions, I think that you 
 may be the only person I've seen say anything positive about 
 the DIP treating extern(C) functions as  safe.

 - Jonathan M Davis
I think Walter had to make a tough call with many tradeoffs. The defining feature of engineering I would say. Is he wrong? Maybe, I don't know. The obvious path is far from always being a winner. If it does come back to haunt him, he can always add a DIP to make extern(!D) system by default. It won't invalidate any work.
May 28 2020
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 28 May 2020 12:28:16 +0000 schrieb Sebastiaan Koppe:

 On Thursday, 28 May 2020 at 09:21:09 UTC, Jonathan M Davis wrote:
 He did unfortunately manage to convince Atila, so the DIP has been
 accepted, but based on the discussions, I think that you may be the
 only person I've seen say anything positive about the DIP treating
 extern(C) functions as  safe.

 - Jonathan M Davis
I think Walter had to make a tough call with many tradeoffs. The defining feature of engineering I would say. Is he wrong? Maybe, I don't know. The obvious path is far from always being a winner. If it does come back to haunt him, he can always add a DIP to make extern(!D) system by default. It won't invalidate any work.
This would be another round of massively breaking user code. And this is going to be exactly the argument that will be used to dismiss any DIP trying to change the defaults later on. -- Johannes
May 28 2020
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 28 May 2020 at 16:01:35 UTC, Johannes Pfau wrote:
 Am Thu, 28 May 2020 12:28:16 +0000 schrieb Sebastiaan Koppe:
 If it does come back to haunt him, he can always add a DIP to 
 make extern(!D)  system by default. It won't invalidate any 
 work.
This would be another round of massively breaking user code.
The breakage will be split in two rounds, but the amount of code needed to be modified would be the same as with the safe-by-default-except-for-extern-non-c. With the possibility of not needing the second round. I know, arguing for a lost cause...
May 29 2020
parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 29 May 2020 at 11:33:01 UTC, Sebastiaan Koppe wrote:
 On Thursday, 28 May 2020 at 16:01:35 UTC, Johannes Pfau wrote:
 [snip]

 This would be another round of massively breaking user code.
The breakage will be split in two rounds, but the amount of code needed to be modified would be the same as with the safe-by-default-except-for-extern-non-c. With the possibility of not needing the second round. I know, arguing for a lost cause...
At the end of the day, I think people would accept "massively breaking user code" if there is a good justification and doesn't drive a hole through safe.
May 29 2020
prev sibling parent Arine <arine1283798123 gmail.com> writes:
On Thursday, 28 May 2020 at 12:28:16 UTC, Sebastiaan Koppe wrote:
 On Thursday, 28 May 2020 at 09:21:09 UTC, Jonathan M Davis 
 wrote:
 He did unfortunately manage to convince Atila, so the DIP has 
 been accepted, but based on the discussions, I think that you 
 may be the only person I've seen say anything positive about 
 the DIP treating extern(C) functions as  safe.

 - Jonathan M Davis
I think Walter had to make a tough call with many tradeoffs. The defining feature of engineering I would say. Is he wrong? Maybe, I don't know. The obvious path is far from always being a winner.
A winner for what. The reasoning given doesn't make sense. Lots of people have pointed that out, and Walter has yet to expand on that.
 If it does come back to haunt him, he can always add a DIP to 
 make extern(!D)  system by default. It won't invalidate any 
 work.
That's worse. It'll cause huge breakage again if you need to convert extern(C) back to system after it was converted to safe. If system by default proves to be a mistake, you can just flip it to safe and it won't break any code. Why take a gamble on something that can't be rationalized?
May 28 2020
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 28, 2020 at 03:21:09AM -0600, Jonathan M Davis via
Digitalmars-d-announce wrote:
[...]
 With the DIP in its current state,  safe becomes a lie.  The compiler
 no longer guarantees that  safe code is memory safe so long as it
 doesn't call any  trusted code where the programmer incorrectly marked
 it as  trusted. Instead, the compiler blindly treats non-extern(D)
 declarations as  safe and invisibly introduces memory safety bugs into
  safe code.  Nothing about that is "OK."
[...] I see it already. The next time someone wants to make a codebase safe but the compiler complains about some violation, just add `extern(C)` to the function and move on. T -- Claiming that your operating system is the best in the world because more people use it is like saying McDonalds makes the best food in the world. -- Carl B. Constantine
May 28 2020