www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Phobos is now compiled with -preview=dip1000

reply Walter Bright <newshound2 digitalmars.com> writes:
https://github.com/dlang/phobos/pull/6931

This is a major milestone in improving the memory safety of D programming. 
Thanks to everyone who helped with this!

Time to start compiling your projects with DIP1000, too!
May 15
next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 15 May 2019 at 07:39:05 UTC, Walter Bright wrote:
 https://github.com/dlang/phobos/pull/6931

 This is a major milestone in improving the memory safety of D 
 programming. Thanks to everyone who helped with this!

 Time to start compiling your projects with DIP1000, too!
For me, the forum claims that your posting time is "from the future". Does that mean that is has somehow leaked a draft and this shouldn't show yet? About -DIP1000, I sure want to use it. But is there currently any practical way to learn it's usage without researching compiler source code?
May 14
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2019 11:49 PM, Dukc wrote:
 Time to start compiling your projects with DIP1000, too!
For me, the forum claims that your posting time is "from the future". Does that mean that is has somehow leaked a draft and this shouldn't show yet?
Maybe the clock is not synchronized somewhere.
 About -DIP1000, I sure want to use it. But is there currently any practical
way 
 to learn it's usage without researching compiler source code?
Simply add the switch -preview=dip1000 to your builds, and follow where it leads.
May 15
next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 15 May 2019 at 07:56:48 UTC, Walter Bright wrote:
 About -DIP1000, I sure want to use it. But is there currently 
 any practical way to learn it's usage without researching 
 compiler source code?
Simply add the switch -preview=dip1000 to your builds, and follow where it leads.
Bound to cause bad practices without nothing to tell why it works how it works. How do I know when I'm supposed to add `scope`? Or how to react when the compiler complains about escaping references? I have some basic image in my head formed from your DIP paper, but I read somewhere that it's outdated. Could be worth a try even without docs, but in the long run we definitely need some explaining.
May 15
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2019 12:21 AM, Dukc wrote:
 Could be worth a try even without docs, but in the long run we definitely need 
 some explaining.
True, but I've tried fairly hard with the error messages. Please post your experiences with them. Also, there shouldn't be any caveats with using it. If it passes the compiler, it should be good to go. (Much like const and pure.) In general, if you find the error messages baffling, try reducing your code to a simpler example. This usually makes the problem clearer. dustmite is a great tool to help with that.
May 15
parent reply Meta <jared771 gmail.com> writes:
On Wednesday, 15 May 2019 at 08:32:09 UTC, Walter Bright wrote:
 On 5/15/2019 12:21 AM, Dukc wrote:
 Could be worth a try even without docs, but in the long run we 
 definitely need some explaining.
True, but I've tried fairly hard with the error messages. Please post your experiences with them.
Walter, can I get you to take a look at this post I made a few months ago, and the contained example? I feel that this is a case that *should* definitely work, but I'm not sure if it can *currently* work - and so far, nobody else seems to be either, save for you. https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org
May 16
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 May 2019 at 04:50:52 UTC, Meta wrote:

 Walter, can I get you to take a look at this post I made a few 
 months ago, and the contained example? I feel that this is a 
 case that *should* definitely work, but I'm not sure if it can 
 *currently* work - and so far, nobody else seems to be either, 
 save for you.

 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org
My assessment (which could be wrong): `scope` and `return` only apply to pointers and `ref`s. If you remove all `scope` and `return` attributes from the function `push`, it works fine. I consider it a bug that the compiler doesn't emit an error when using attributes on types for which they are not intended. Mike
May 16
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 May 2019 at 05:22:30 UTC, Mike Franklin wrote:

 My assessment (which could be wrong):
 `scope` and `return` only apply to pointers and `ref`s.  If you 
 remove all `scope` and `return` attributes from the function 
 `push`, it works fine.

 I consider it a bug that the compiler doesn't emit an error 
 when using attributes on types for which they are not intended.

 Mike
Working example: https://run.dlang.io/is/TCP0td
May 16
parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 05:32:42 UTC, Mike Franklin wrote:
 On Friday, 17 May 2019 at 05:22:30 UTC, Mike Franklin wrote:

 My assessment (which could be wrong):
 `scope` and `return` only apply to pointers and `ref`s.  If 
 you remove all `scope` and `return` attributes from the 
 function `push`, it works fine.

 I consider it a bug that the compiler doesn't emit an error 
 when using attributes on types for which they are not intended.

 Mike
Working example: https://run.dlang.io/is/TCP0td
That does compile, but I don't think that it's working the way I want it to. I believe it only works because a GC-managed string is used for the backing storage. If you change that to a static array on the stack: safe void main() { immutable(char)[16] rawData = "2 6 4 1 0 2 9 4 5"; auto dataRange = makeDataRange(rawData); auto result = dataRange.copyToQueue(); import std.stdio; writeln("The result of data processing is: ", result); } It will refuse to compile with this message: Error: reference to local variable rawData assigned to non-scope parameter input calling makeDataRange `makeDataRange` is defined like this: safe DataRange makeDataRange(string input) { auto range = DataRange(input); return range; } So that static array is getting implicitly sliced, i.e., its address is being taken. It's pretty obvious why `input` is not being inferred as scope - it's being returned from `makeDataRange`. However, when I try to manually annotate it with return or return scope, I run into further errors: DataRange makeDataRange(return scope string input) { ...etc. } Error: scope variable input assigned to non-scope parameter rawData calling DataRange.this Error: scope variable dataRange assigned to non-scope parameter data calling copyToQueue So I continue annotating things with scope or return or return scope whenever the compiler complains about it, going up through the call chain until I arrive back at my original problem mentioned in the post I linked. (My original example with changes made going through this exercise: https://run.dlang.io/is/uQDXG6) This is why I say that I'm not sure that I quite understand dip1000. I *thought* I did, but an example that seems like it should clearly work (at least to me), does not. If you look at `main` above, `rawData` has the same lifetime as the `dataRange` struct returned from `makeDataRange` and the queue returned from `copyToQueue`. True, there is some traditionally unsafe stuff happening in between; however, I thought that the point of adding all these annotations is to tell the compiler how the lifetimes of these objects propagate up and down the call stack, so that it can check that there will be no memory corruption. I'm not doing anything here that will result in a pointer to an expired stack frame, or otherwise cause memory corruption or use after free, or anything like that (*unless* I allow either `dataRange` or `result` to escape from the main function - which dip1000 correctly disallows).
May 17
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 May 2019 at 17:03:51 UTC, Meta wrote:

 If you look at `main` above, `rawData` has the same lifetime as 
 the `dataRange` struct returned from `makeDataRange` and the 
 queue returned from `copyToQueue`. True, there is some 
 traditionally unsafe stuff happening in between; however, I 
 thought that the point of adding all these annotations is to 
 tell the compiler how the lifetimes of these objects propagate 
 up and down the call stack, so that it can check that there 
 will be no memory corruption. I'm not doing anything here that 
 will result in a pointer to an expired stack frame, or 
 otherwise cause memory corruption or use after free, or 
 anything like that (*unless* I allow either `dataRange` or 
 `result` to escape from the main function - which dip1000 
 correctly disallows).
I don't think it does because `Queue!(T).store` has infinite lifetime beyond that of even `main`, at least as far as the compiler is concerned. The compiler doesn't have enough information to know that `store` is tied to the lifetime of `Queue!(T)` (a.k.a `rawData`) and maybe that's a missing language feature. Maybe we should be allowed to declare aggregate fields as `scope` to convey that, but the compiler currently disallows it. loosely related: https://issues.dlang.org/show_bug.cgi?id=18788#c7 Mike
May 17
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 May 2019 at 20:59:43 UTC, Mike Franklin wrote:

 I don't think it does because `Queue!(T).store` has infinite 
 lifetime beyond that of even `main`, at least as far as the 
 compiler is concerned.  The compiler doesn't have enough 
 information to know that `store` is tied to the lifetime of 
 `Queue!(T)` (a.k.a `rawData`) and maybe that's a missing 
 language feature.  Maybe we should be allowed to declare 
 aggregate fields as `scope` to convey that, but the compiler 
 currently disallows it.
Or we build in some way for slices to know their lifetime relative to the source array from which they were created. But I'm not sure how that would work. Mike
May 17
prev sibling parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 20:59:43 UTC, Mike Franklin wrote:
 On Friday, 17 May 2019 at 17:03:51 UTC, Meta wrote:

 If you look at `main` above, `rawData` has the same lifetime 
 as the `dataRange` struct returned from `makeDataRange` and 
 the queue returned from `copyToQueue`. True, there is some 
 traditionally unsafe stuff happening in between; however, I 
 thought that the point of adding all these annotations is to 
 tell the compiler how the lifetimes of these objects propagate 
 up and down the call stack, so that it can check that there 
 will be no memory corruption. I'm not doing anything here that 
 will result in a pointer to an expired stack frame, or 
 otherwise cause memory corruption or use after free, or 
 anything like that (*unless* I allow either `dataRange` or 
 `result` to escape from the main function - which dip1000 
 correctly disallows).
I don't think it does because `Queue!(T).store` has infinite lifetime beyond that of even `main`, at least as far as the compiler is concerned.
I see what you're getting at. The compiler sees a slice type (i.e., Data[]), knows that it's GC-backed and thus has infinite lifetime, and concludes "the data you're trying to put in the store has too long of a lifetime". That makes sense, but slices don't necessarily have to be backed by the GC, so that seems like a faulty heuristic to me and possibly a vector for bugs.
 The compiler doesn't have enough information to know that 
 `store` is tied to the lifetime of `Queue!(T)` (a.k.a 
 `rawData`) and maybe that's a missing language feature.
According to the DIP, "from a lifetime analysis viewpoint, a struct is considered a juxtaposition of its direct members." Who knows if that's still the case, because Walter has considerably changed how it works but has not documented those changes (IIRC, I may be wrong on that). That probably means that a Queue!T has an infinite lifetime, assuming that the compiler sees its T[] member as having an infinite lifetime.
 Maybe we should be allowed to declare aggregate fields as 
 `scope` to convey that, but the compiler currently disallows it.
That might be nice but would also probably cause a dramatic increase in complexity. I haven't thought through the possible ramifications of making a change like that.
May 17
parent Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 21:57:51 UTC, Meta wrote:
 I see what you're getting at. The compiler sees a slice type 
 (i.e., Data[]), knows that it's GC-backed and thus has infinite 
 lifetime, and concludes "the data you're trying to put in the 
 store has too long of a lifetime".
Should be "too _short_".
May 17
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 11:22:30 PM MDT Mike Franklin via Digitalmars-d-
announce wrote:
 I consider it a bug that the compiler doesn't emit an error when
 using attributes on types for which they are not intended.
As in you think that something like auto foo(scope int i) {...} should be illegal, because scope makes no sense on an int? That's nice in theory, but templates make such an approach a serious problem. It needs to work to do something like auto foo(T)(scope T t) {...} without having to have separate overloads for types where scope makes sense and types where it doesn't. Similarly, you don't want to have to use static ifs whenever you declare a variable that you want to be scope in the cases where the template argument is a type where scope does work. In general, D ignores attributes when they don't apply rather than making it an error, because making it an error causes serious problems for generic code. This does unfortunately mean that some people are bound to sometimes end up using an attribute when it doesn't apply, thinking that it does, and that's unfortunate, but overall, it just works better for the compiler not to complain about such cases. - Jonathan M Davis
May 17
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 17.05.19 06:50, Meta wrote:
 Walter, can I get you to take a look at this post I made a few months 
 ago, and the contained example? I feel that this is a case that *should* 
 definitely work, but I'm not sure if it can *currently* work - and so 
 far, nobody else seems to be either, save for you.
 
 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org
You don't like my explanation? https://forum.dlang.org/post/q6r4bf$2hu4$1 digitalmars.com (same thread)
May 16
parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 05:22:31 UTC, ag0aep6g wrote:
 On 17.05.19 06:50, Meta wrote:
 Walter, can I get you to take a look at this post I made a few 
 months ago, and the contained example? I feel that this is a 
 case that *should* definitely work, but I'm not sure if it can 
 *currently* work - and so far, nobody else seems to be either, 
 save for you.
 
 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org
You don't like my explanation? https://forum.dlang.org/post/q6r4bf$2hu4$1 digitalmars.com (same thread)
Your explanation was fine, but I need a good solution, other than wrapping the array assignment in trusted.
May 17
parent reply ag0aep6g <anonymous example.com> writes:
On 17.05.19 14:10, Meta wrote:
 Your explanation was fine, but I need a good solution, other than 
 wrapping the array assignment in  trusted.
I see. As far as I understand DIP 1000, it's not supposed to enable your use case without having to use ` trusted`. DIP 1000 stops at heap allocations. It just assumes infinite lifetime for them. If you want to restrict the lifetime of a heap allocation (in your case: tie it to the lifetime of a struct), you have to do it manually.
May 17
parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 17:05:21 UTC, ag0aep6g wrote:
 On 17.05.19 14:10, Meta wrote:
 Your explanation was fine, but I need a good solution, other 
 than wrapping the array assignment in  trusted.
I see. As far as I understand DIP 1000, it's not supposed to enable your use case without having to use ` trusted`.
If this is true, then I have a big problem with DIP1000. This is an extremely common use case (copying memory from an inner scope with a limited lifetime to some store in an outer scope with a longer or infinite lifetime).
 DIP 1000 stops at heap allocations. It just assumes infinite 
 lifetime for them.
Yes, as per the DIP.
 If you want to restrict the lifetime of a heap allocation (in 
 your case: tie it to the lifetime of a struct), you have to do 
 it manually.
I don't want to *restrict* the lifetime of a heap allocation. I want the compiler to recognize that the lifetime of my original data is the same as the processed output, and thus allow my code to compile.
May 17
next sibling parent ag0aep6g <anonymous example.com> writes:
On 17.05.19 19:25, Meta wrote:
 I don't want to *restrict* the lifetime of a heap allocation. I want the 
 compiler to recognize that the lifetime of my original data is the same 
 as the processed output, and thus allow my code to compile.
You have a heap allocation that references your original data, which might be on the stack (Queue.store[0] points into rawData). The compiler would have to make sure (or recognize) that the pointers on the heap don't outlive the data to which they point. But the lifetime of the heap allocation is infinite, and the lifetime of the stack data is not. So that fails. To make it work you'd need some way to have a shorter-than-infinite lifetime for the heap. Whether you want to restrict the lifetime explicitly or have the compiler figure it out on its own ... it doesn't really matter when we can't do either.
May 17
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, May 17, 2019 11:25:40 AM MDT Meta via Digitalmars-d-announce 
wrote:
 I don't want to *restrict* the lifetime of a heap allocation. I
 want the compiler to recognize that the lifetime of my original
 data is the same as the processed output, and thus allow my code
 to compile.
It is my understanding that DIP 1000 really doesn't track lifetimes at all. It just ensures that no references to the data escape. So, you can't do something like take a scope variable and put any references to it or what it refers to in a container. Honestly, from what I've seen, what you can ultimately do with scope is pretty limited. It definitely helps in simple cases, but it quickly gets to the point that it's unable to be used in more complex cases - at least not without casting and needing to use trusted. So, it's an improvement for some kinds of code, but I suspect that in general, it's just going to be more annoying than it's worth. Time will tell though. - Jonathan M Davis
May 17
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/17/2019 11:45 AM, Jonathan M Davis wrote:
 It is my understanding that DIP 1000 really doesn't track lifetimes at all.
 It just ensures that no references to the data escape. So, you can't do
 something like take a scope variable and put any references to it or what it
 refers to in a container. Honestly, from what I've seen, what you can
 ultimately do with scope is pretty limited. It definitely helps in simple
 cases, but it quickly gets to the point that it's unable to be used in more
 complex cases - at least not without casting and needing to use  trusted.
 So, it's an improvement for some kinds of code, but I suspect that in
 general, it's just going to be more annoying than it's worth. Time will tell
 though.
Dip1000 is key to enable containers to control access to pointers to their innards that they expose.
May 17
parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Friday, 17 May 2019 at 20:04:42 UTC, Walter Bright wrote:
 Dip1000 is key to enable containers to control access to 
 pointers to their innards that they expose.
I haven't looked at the subject for a while, but every time I did the takeaway was the same: dip1000 works great for containers until you need to reallocate or free something, at which point it's back to trusted code with you. I think you said at some point "It's still useful, because it reduces the surface of code that needs to be checked", but even then saying containers can "control access to the data they expose" is a little optimistic. They only control that access as long as you don't need to call resize(). I'd wager that a large fraction of dangling pointer errors made by non-beginner C++ developpers come specifically from this use case.
May 18
parent reply Walter Bright <newshound2 digitalmars.com> writes:
If all access to internals is returned by ref, those lifetimes are restricted
to 
the current expression.
May 18
parent Olivier FAURE <couteaubleu gmail.com> writes:
On Saturday, 18 May 2019 at 19:44:37 UTC, Walter Bright wrote:
 If all access to internals is returned by ref, those lifetimes 
 are restricted to the current expression.
Oh my god, I try my best to be open-minded, but talking about dip1000 design with you is like pulling teeth *at best*. Yes, containers work perfectly if you allocate them on the stack and use their contents during the current stack frame, and then de-allocate them statically. By definition, this represents 0% of the use cases of dynamic containers. Dynamic containers need methods like "push_back", "reserve", "resize", "concatenate" or "clear", which are all impossible to implement with dip1000 without making their implementations trusted, which in turns opens up the program to use-after-free memory corruption. See also: https://forum.dlang.org/post/qbbipvkjqjeweasxknbn forum.dlang.org https://forum.dlang.org/post/rxmwjjphnmkszaxonmje forum.dlang.org Have you talked to Atila Neves at all for the past six months? Why the hell are we having this discussion? This is not a new issue. I have raised it repeatedly in the past (I can even dig up the posts if you're interested; I remember writing a fairly in-depth analysis at some point). Atila's automem and Skoppe's spasm have the same limitation: you can't reallocate memory without writing unsafe code (I'm told spasm gets around that by never deallocating anything). Honestly, the fact that you're the only person with a coherent vision of dip1000, and yet you keep ignoring problems when they're pointed out to you is both worrying and infuriating. Eg:
 So far, the only real shortcoming in the initial design was 
 revealed by the put() semantics, and was fixed with that PR 
 that transmitted scope-ness through the first argument.
Like, yes, I understand that dip1000 is an achievement even if it doesn't allow for resizable containers, and that immutable already allow for functional programming patterns and that's great, but you need to stop acting like everything's going perfect when community members (including highly involved library writers) have complained about the same things over and over again (imprecise semantics, lack of documentation, the resize() use case) and you've kept ignoring them. Seriously, I'm not asking for much. I'm not demanding you take any architecture decision or redesign the language (like some people are prone to demanding here). But it would be nice if you stopped acting like you didn't read a word I wrote, over and over again.
May 18
prev sibling parent Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 18:45:12 UTC, Jonathan M Davis wrote:
 On Friday, May 17, 2019 11:25:40 AM MDT Meta via 
 Digitalmars-d-announce wrote:
 I don't want to *restrict* the lifetime of a heap allocation. 
 I want the compiler to recognize that the lifetime of my 
 original data is the same as the processed output, and thus 
 allow my code to compile.
It is my understanding that DIP 1000 really doesn't track lifetimes at all.
Then why does the DIP, in addition to many of the error messages, use the word lifetime? I feel like I know less about DIP1000 and what it actually does than when I started. Can someone _please_ point me at any up to date documentation on this?
May 17
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2019 9:50 PM, Meta wrote:
 Walter, can I get you to take a look at this post I made a few months ago, and 
 the contained example? I feel that this is a case that *should* definitely
work, 
 but I'm not sure if it can *currently* work - and so far, nobody else seems to 
 be either, save for you.
 
 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org
As always, I recommend drastically reducing the example. It nearly always makes the actual problem emerge from all the noise.
May 16
parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 05:27:02 UTC, Walter Bright wrote:
 On 5/16/2019 9:50 PM, Meta wrote:
 Walter, can I get you to take a look at this post I made a few 
 months ago, and the contained example? I feel that this is a 
 case that *should* definitely work, but I'm not sure if it can 
 *currently* work - and so far, nobody else seems to be either, 
 save for you.
 
 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org
As always, I recommend drastically reducing the example. It nearly always makes the actual problem emerge from all the noise.
I'll try to reduce it further, but this example is already as reduced as I could make it while still having the same structure as my actual code.
May 17
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/17/2019 10:26 AM, Meta wrote:
 I'll try to reduce it further, but this example is already as reduced as I
could 
 make it while still having the same structure as my actual code.
It doesn't need to have the same structure. It just needs to exhibit the problem.
May 17
prev sibling next sibling parent Seb <seb wilzba.ch> writes:
On Wednesday, 15 May 2019 at 07:56:48 UTC, Walter Bright wrote:
 On 5/14/2019 11:49 PM, Dukc wrote:
 Time to start compiling your projects with DIP1000, too!
For me, the forum claims that your posting time is "from the future". Does that mean that is has somehow leaked a draft and this shouldn't show yet?
Maybe the clock is not synchronized somewhere.
 About -DIP1000, I sure want to use it. But is there currently 
 any practical way to learn it's usage without researching 
 compiler source code?
Simply add the switch -preview=dip1000 to your builds, and follow where it leads.
For reference, people need to use dmd-nightly (available tomorrow), build dmd themselves or wait until DMD 2.087 (1st of July). Also, as most projects use Dub, the flag can be added to the dub.sdl too: --- dflags "-preview=dip1000" ----
May 15
prev sibling parent Kagamin <spam here.lot> writes:
On Wednesday, 15 May 2019 at 07:56:48 UTC, Walter Bright wrote:
 Maybe the clock is not synchronized somewhere.
It's off by one hour.
May 15
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 15 May 2019 at 06:49:02 UTC, Dukc wrote:
 For me, the forum claims that your posting time is "from the 
 future". Does that mean that is has somehow leaked a draft and 
 this shouldn't show yet?
No, it is merged. As to why from the future, probably timezones.
May 15
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2019 11:49 PM, Dukc wrote:
 About -DIP1000, I sure want to use it. But is there currently any practical
way 
 to learn it's usage without researching compiler source code?
This is a good start: http://dconf.org/2017/talks/bright.html
May 15
parent Dukc <ajieskola gmail.com> writes:
On Wednesday, 15 May 2019 at 08:26:23 UTC, Walter Bright wrote:
 This is a good start:

 http://dconf.org/2017/talks/bright.html
Ah, at least something. Thanks.
May 15
prev sibling next sibling parent M.M. <matus email.cz> writes:
On Wednesday, 15 May 2019 at 07:39:05 UTC, Walter Bright wrote:
 https://github.com/dlang/phobos/pull/6931

 This is a major milestone in improving the memory safety of D 
 programming. Thanks to everyone who helped with this!

 Time to start compiling your projects with DIP1000, too!
Congratulations to the whole team behind it.
May 15
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 15, 2019 at 12:39:05AM -0700, Walter Bright via
Digitalmars-d-announce wrote:
 https://github.com/dlang/phobos/pull/6931
 
 This is a major milestone in improving the memory safety of D
 programming.  Thanks to everyone who helped with this!
 
 Time to start compiling your projects with DIP1000, too!
My very first attempt to compile my code with -preview=dip1000 led to a regression. :-( Reduced code: ------ import std.container.rbtree; alias Grid = RedBlackTree!(GridPoint); struct GridPoint { private string _srcStr; int opCmp(in GridPoint p) const { return 0; } } ------ Compiler output (with -preview=dip1000): ------ /usr/src/d/phobos/std/container/rbtree.d(1111): Error: ` safe` function `std.container.rbtree.RedBlackTree!(GridPoint, "a < b", false).RedBlackTree.toHash` cannot call ` system` function `core.internal.hash.hashOf!(GridPoint).hashOf` /usr/src/d/druntime/import/core/internal/hash.d(510): `core.internal.hash.hashOf!(GridPoint).hashOf` is declared here numid.d(3): Error: template instance `std.container.rbtree.RedBlackTree!(GridPoint, "a < b", false)` error instantiating ------ The culprit is the 'private' in GridPoint. Removing 'private' gets rid of the problem. *Why* putting 'private' on a field member makes toHash unsafe, is beyond my ability to comprehend. T -- Windows: the ultimate triumph of marketing over technology. -- Adrian von Bidder
May 15
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2019 11:09 AM, H. S. Teoh wrote:
 *Why* putting 'private' on a field member makes toHash unsafe, is beyond
 my ability to comprehend.
That's because the reduced version isn't a reduced version. It imports a vast amount of other code.
May 15
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 15, 2019 at 11:34:44AM -0700, Walter Bright via
Digitalmars-d-announce wrote:
 On 5/15/2019 11:09 AM, H. S. Teoh wrote:
 *Why* putting 'private' on a field member makes toHash unsafe, is
 beyond my ability to comprehend.
That's because the reduced version isn't a reduced version. It imports a vast amount of other code.
Alright, here's a TRULY reduced version: ---- struct S { private int _x; } struct RedBlackTree { size_t toHash() nothrow safe { return .hashOf(S.init); } } void main() { } ---- Compiling with -preview=dip1000 causes a compile error complaining that toHash() is not safe. Removing 'private' makes it go away. Compiling without -preview=dip1000 also makes it go away. Now explain this one. :-D T -- A linguistics professor was lecturing to his class one day. "In English," he said, "A double negative forms a positive. In some languages, though, such as Russian, a double negative is still a negative. However, there is no language wherein a double positive can form a negative." A voice from the back of the room piped up, "Yeah, yeah."
May 15
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 15, 2019 at 05:53:17PM -0700, H. S. Teoh via Digitalmars-d-announce
wrote:
 On Wed, May 15, 2019 at 11:34:44AM -0700, Walter Bright via
Digitalmars-d-announce wrote:
 On 5/15/2019 11:09 AM, H. S. Teoh wrote:
 *Why* putting 'private' on a field member makes toHash unsafe, is
 beyond my ability to comprehend.
That's because the reduced version isn't a reduced version. It imports a vast amount of other code.
Alright, here's a TRULY reduced version:
Gah, so apparently .hashOf is a gigantic overload set of *21* different overloads, so this is not really "truly" reduced. =-O Anybody up for figuring out which overload(s) is/are getting called? Betcha the problem is that -preview=dip1000 causes one of the overloads to fail to compile, thus shuffling to a different overload that isn't safe. I hate SFINAE. T -- Just because you survived after you did it, doesn't mean it wasn't stupid!
May 15
next sibling parent reply evilrat <evilrat666 gmail.com> writes:
On Thursday, 16 May 2019 at 01:05:53 UTC, H. S. Teoh wrote:
  ...
  I hate SFINAE.
But.. But D doesn't have it!11 NOOO!!1!
May 15
parent Dukc <ajieskola gmail.com> writes:
On Thursday, 16 May 2019 at 04:29:10 UTC, evilrat wrote:
 On Thursday, 16 May 2019 at 01:05:53 UTC, H. S. Teoh wrote:
  ...
  I hate SFINAE.
But.. But D doesn't have it!11 NOOO!!1!
Not in the same sense as C++. But if the template constrains rely of is() statements, that is still a kind of explicitly-activated SFINAE. If that's the case here, it's probably the template constraint that's badly designed.
May 16
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 16 May 2019 at 01:05:53 UTC, H. S. Teoh wrote:
 Gah, so apparently .hashOf is a gigantic overload set of *21* 
 different overloads, so this is not really "truly" reduced. =-O

 Anybody up for figuring out which overload(s) is/are getting 
 called?
https://github.com/dlang/druntime/blob/master/src/core/internal/hash.d#L393 static if (hasCallableToHash!(typeof(val))){ ... } // false else { static if (__traits(hasMember, T, "toHash") && is(typeof(T.toHash) == function)) { ... } // false else static if (T.tupleof.length == 0) { ... } // false else static if ((is(T == struct) && !canBitwiseHash!T) || T.tupleof.length == 1)//true { static foreach (i, F; typeof(val.tupleof)) { static if (__traits(isStaticArray, F)) { ... } // false else static if (is(F == struct) || is(F == union)) { ... } // false else { // Nothing special happening. static if (i == 0 && !isChained) size_t h = hashOf(val.tupleof[i]); else h = hashOf(val.tupleof[i], h); } } }
 Betcha the problem is that -preview=dip1000 causes one of the 
 overloads to fail to compile, thus shuffling to a different 
 overload that isn't  safe.  I hate SFINAE.
My money's on access to a private member through .tupleof.
May 15
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 16 May 2019 at 05:14:39 UTC, Nicholas Wilson wrote:
 [...]
Yes that sounds like the culprit. Btw as mentioned on DConf, the dip1000 switch contains a few other breaking changes which will make it even harder to adopt too.
May 15
parent reply Kagamin <spam here.lot> writes:
On Thursday, 16 May 2019 at 05:22:42 UTC, Seb wrote:
 Yes that sounds like the culprit. Btw as mentioned on DConf, 
 the dip1000 switch contains a few other breaking changes which 
 will make it even harder to adopt too.
Well, it's an inherent property of DIP1000 to not compile code that previously compiled. Though safety of tupleof shouldn't depend on DIP1000.
May 16
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 16 May 2019 at 10:03:42 UTC, Kagamin wrote:
 On Thursday, 16 May 2019 at 05:22:42 UTC, Seb wrote:
 Yes that sounds like the culprit. Btw as mentioned on DConf, 
 the dip1000 switch contains a few other breaking changes which 
 will make it even harder to adopt too.
Well, it's an inherent property of DIP1000 to not compile code that previously compiled. Though safety of tupleof shouldn't depend on DIP1000.
Well, here's the full discussion: https://github.com/dlang/dmd/pull/8035
May 16
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 16, 2019 at 10:35:27AM +0000, Seb via Digitalmars-d-announce wrote:
 On Thursday, 16 May 2019 at 10:03:42 UTC, Kagamin wrote:
 On Thursday, 16 May 2019 at 05:22:42 UTC, Seb wrote:
 Yes that sounds like the culprit. Btw as mentioned on DConf, the
 dip1000 switch contains a few other breaking changes which will
 make it even harder to adopt too.
Well, it's an inherent property of DIP1000 to not compile code that previously compiled. Though safety of tupleof shouldn't depend on DIP1000.
Well, here's the full discussion: https://github.com/dlang/dmd/pull/8035
Finally got round to skimming through that discussion. Looks like in this case, what we need is for toHash to be declared trusted when .tupleof includes private members (because toHash is not supposed to modify any private members, and I assume hashing over a private member shouldn't violate safe -- right?). Either that, or RedBlackTree needs to be changed so that safe-ty doesn't depend on random user types having private fields breaking compilation. It's pretty ridiculous, from a user's POV, for a standard container to fail to compile just because the user had the audacity to declare private members in his object! And the fact that this root problem is masked under a totally obscure compile error only adds salt to the wound. T -- IBM = I Blame Microsoft
May 17
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2019 6:05 PM, H. S. Teoh wrote:
 Gah, so apparently .hashOf is a gigantic overload set of *21* different
 overloads, so this is not really "truly" reduced. =-O
I've often thought that Phobos excessively overused overloading. And you're quite right, it's a chore figuring out which one is the culprit. What I do is change the name(s) to .hashOfx so it won't be picked, then one can figure out which one is selected through a process of elimination. Or insert: pragma(msg, __LINE__); statements in each one.
May 16
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 16, 2019 at 10:09:38AM -0700, Walter Bright via
Digitalmars-d-announce wrote:
 On 5/15/2019 6:05 PM, H. S. Teoh wrote:
 Gah, so apparently .hashOf is a gigantic overload set of *21*
 different overloads, so this is not really "truly" reduced. =-O
I've often thought that Phobos excessively overused overloading.
This is in druntime.
 And you're quite right, it's a chore figuring out which one is the
 culprit.
More and more, I'm becoming convinced that this sort of usage of function overloading is an anti-pattern. It should instead be written as something like this: size_t hashOf(T)(T* arg) // N.B.: no sig constraints: because we expect to be // able to hash anything. { static if (is(T == struct)) return hashOfStruct(arg); else static if (is(T == U[], U)) return hashOfArray(arg); ... else static assert(0, "Hash of " ~ T.stringof ~ " not supported"); } The sig constraints, or lack thereof, ought to reflect the *logical* set of acceptable types, not necessarily the actual set supported by the implementation. I.e., hashOf logically *should* support all types, but maybe the current implementation doesn't (yet) support a particular corner case; so it should still accept the type, but emit an error explaining the implementation deficiency in a static assert, rather than just passing the buck back to the compiler which then spews forth a text wall of incomprehensible gibberish of how all 21 overloads failed to match.
 What I do is change the name(s) to .hashOfx so it won't be picked,
 then one can figure out which one is selected through a process of
 elimination. Or insert:
 
     pragma(msg, __LINE__);
 
 statements in each one.
Good idea. But looks like Nicholas has already done the heavy lifting for us. :-D T -- Always remember that you are unique. Just like everybody else. -- despair.com
May 16
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2019 10:24 AM, H. S. Teoh wrote:
 More and more, I'm becoming convinced that this sort of usage of
 function overloading is an anti-pattern.  It should instead be written
 as something like this:
I agree. And there at least you can find all the uses.
 Good idea. But looks like Nicholas has already done the heavy lifting
 for us.  :-D
The problem will come up again.
May 16
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 15, 2019 at 11:09:01AM -0700, H. S. Teoh via Digitalmars-d-announce
wrote:
 On Wed, May 15, 2019 at 12:39:05AM -0700, Walter Bright via
Digitalmars-d-announce wrote:
 https://github.com/dlang/phobos/pull/6931
 
 This is a major milestone in improving the memory safety of D
 programming.  Thanks to everyone who helped with this!
 
 Time to start compiling your projects with DIP1000, too!
My very first attempt to compile my code with -preview=dip1000 led to a regression. :-(
[...] Bugzilla issue: https://issues.dlang.org/show_bug.cgi?id=19877 T -- To err is human; to forgive is not our policy. -- Samuel Adler
May 15