www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Phobos is now compiled with -preview=dip1000

reply Walter Bright <newshound2 digitalmars.com> writes:
https://github.com/dlang/phobos/pull/6931

This is a major milestone in improving the memory safety of D programming. 
Thanks to everyone who helped with this!

Time to start compiling your projects with DIP1000, too!
May 15 2019
next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 15 May 2019 at 07:39:05 UTC, Walter Bright wrote:

 https://github.com/dlang/phobos/pull/6931

 This is a major milestone in improving the memory safety of D 
 programming. Thanks to everyone who helped with this!

 Time to start compiling your projects with DIP1000, too!


For me, the forum claims that your posting time is "from the 
future". Does that mean that is has somehow leaked a draft and 
this shouldn't show yet?

About -DIP1000, I sure want to use it. But is there currently any 
practical way to learn it's usage without researching compiler 
source code?
May 14 2019
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2019 11:49 PM, Dukc wrote:

 Time to start compiling your projects with DIP1000, too!


 For me, the forum claims that your posting time is "from the future". Does
that 
 mean that is has somehow leaked a draft and this shouldn't show yet?


Maybe the clock is not synchronized somewhere.


 About -DIP1000, I sure want to use it. But is there currently any practical
way 
 to learn it's usage without researching compiler source code?


Simply add the switch -preview=dip1000 to your builds, and follow where it
leads.
May 15 2019
next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 15 May 2019 at 07:56:48 UTC, Walter Bright wrote:

 About -DIP1000, I sure want to use it. But is there currently 
 any practical way to learn it's usage without researching 
 compiler source code?


 Simply add the switch -preview=dip1000 to your builds, and 
 follow where it leads.


Bound to cause bad practices without nothing to tell why it works 
how it works. How do I know when I'm supposed to add `scope`? Or 
how to react when the compiler complains about escaping 
references? I have some basic image in my head formed from your 
DIP paper, but I read somewhere that it's outdated.

Could be worth a try even without docs, but in the long run we 
definitely need some explaining.
May 15 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2019 12:21 AM, Dukc wrote:

 Could be worth a try even without docs, but in the long run we definitely need 
 some explaining.


True, but I've tried fairly hard with the error messages. Please post your 
experiences with them.

Also, there shouldn't be any caveats with using it. If it passes the compiler, 
it should be good to go. (Much like const and pure.)

In general, if you find the error messages baffling, try reducing your code to
a 
simpler example. This usually makes the problem clearer. dustmite is a great 
tool to help with that.
May 15 2019
parent reply Meta <jared771 gmail.com> writes:
On Wednesday, 15 May 2019 at 08:32:09 UTC, Walter Bright wrote:

 On 5/15/2019 12:21 AM, Dukc wrote:

 Could be worth a try even without docs, but in the long run we 
 definitely need some explaining.


 True, but I've tried fairly hard with the error messages. 
 Please post your experiences with them.


Walter, can I get you to take a look at this post I made a few 
months ago, and the contained example? I feel that this is a case 
that *should* definitely work, but I'm not sure if it can 
*currently* work - and so far, nobody else seems to be either, 
save for you.

https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org
May 16 2019
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 May 2019 at 04:50:52 UTC, Meta wrote:


 Walter, can I get you to take a look at this post I made a few 
 months ago, and the contained example? I feel that this is a 
 case that *should* definitely work, but I'm not sure if it can 
 *currently* work - and so far, nobody else seems to be either, 
 save for you.

 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org


My assessment (which could be wrong):
`scope` and `return` only apply to pointers and `ref`s.  If you 
remove all `scope` and `return` attributes from the function 
`push`, it works fine.

I consider it a bug that the compiler doesn't emit an error when 
using attributes on types for which they are not intended.

Mike
May 16 2019
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 May 2019 at 05:22:30 UTC, Mike Franklin wrote:


 My assessment (which could be wrong):
 `scope` and `return` only apply to pointers and `ref`s.  If you 
 remove all `scope` and `return` attributes from the function 
 `push`, it works fine.

 I consider it a bug that the compiler doesn't emit an error 
 when using attributes on types for which they are not intended.

 Mike


Working example:  https://run.dlang.io/is/TCP0td
May 16 2019
parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 05:32:42 UTC, Mike Franklin wrote:

 On Friday, 17 May 2019 at 05:22:30 UTC, Mike Franklin wrote:


 My assessment (which could be wrong):
 `scope` and `return` only apply to pointers and `ref`s.  If 
 you remove all `scope` and `return` attributes from the 
 function `push`, it works fine.

 I consider it a bug that the compiler doesn't emit an error 
 when using attributes on types for which they are not intended.

 Mike


 Working example:  https://run.dlang.io/is/TCP0td


That does compile, but I don't think that it's working the way I 
want it to. I believe it only works because a GC-managed string 
is used for the backing storage. If you change that to a static 
array on the stack:

 safe
void main()
{
     immutable(char)[16] rawData = "2 6 4 1 0 2 9 4 5";
     auto dataRange = makeDataRange(rawData);
     auto result = dataRange.copyToQueue();
     import std.stdio;
     writeln("The result of data processing is: ", result);
}

It will refuse to compile with this message:
Error: reference to local variable rawData assigned to non-scope 
parameter input calling makeDataRange

`makeDataRange` is defined like this:
 safe
DataRange makeDataRange(string input)
{
     auto range = DataRange(input);
     return range;
}

So that static array is getting implicitly sliced, i.e., its 
address is being taken. It's pretty obvious why `input` is not 
being inferred as scope - it's being returned from 
`makeDataRange`. However, when I try to manually annotate it with 
return or return scope, I run into further errors:

DataRange makeDataRange(return scope string input)
{ ...etc. }

Error: scope variable input assigned to non-scope parameter 
rawData calling DataRange.this
Error: scope variable dataRange assigned to non-scope parameter 
data calling copyToQueue

So I continue annotating things with scope or return or return 
scope whenever the compiler complains about it, going up through 
the call chain until I arrive back at my original problem 
mentioned in the post I linked.

(My original example with changes made going through this 
exercise: https://run.dlang.io/is/uQDXG6)

This is why I say that I'm not sure that I quite understand 
dip1000. I *thought* I did, but an example that seems like it 
should clearly work (at least to me), does not.

If you look at `main` above, `rawData` has the same lifetime as 
the `dataRange` struct returned from `makeDataRange` and the 
queue returned from `copyToQueue`. True, there is some 
traditionally unsafe stuff happening in between; however, I 
thought that the point of adding all these annotations is to tell 
the compiler how the lifetimes of these objects propagate up and 
down the call stack, so that it can check that there will be no 
memory corruption. I'm not doing anything here that will result 
in a pointer to an expired stack frame, or otherwise cause memory 
corruption or use after free, or anything like that (*unless* I 
allow either `dataRange` or `result` to escape from the main 
function - which dip1000 correctly disallows).
May 17 2019
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 May 2019 at 17:03:51 UTC, Meta wrote:


 If you look at `main` above, `rawData` has the same lifetime as 
 the `dataRange` struct returned from `makeDataRange` and the 
 queue returned from `copyToQueue`. True, there is some 
 traditionally unsafe stuff happening in between; however, I 
 thought that the point of adding all these annotations is to 
 tell the compiler how the lifetimes of these objects propagate 
 up and down the call stack, so that it can check that there 
 will be no memory corruption. I'm not doing anything here that 
 will result in a pointer to an expired stack frame, or 
 otherwise cause memory corruption or use after free, or 
 anything like that (*unless* I allow either `dataRange` or 
 `result` to escape from the main function - which dip1000 
 correctly disallows).


I don't think it does because `Queue!(T).store` has infinite 
lifetime beyond that of even `main`, at least as far as the 
compiler is concerned.  The compiler doesn't have enough 
information to know that `store` is tied to the lifetime of 
`Queue!(T)` (a.k.a `rawData`) and maybe that's a missing language 
feature.  Maybe we should be allowed to declare aggregate fields 
as `scope` to convey that, but the compiler currently disallows 
it.

loosely related:  
https://issues.dlang.org/show_bug.cgi?id=18788#c7

Mike
May 17 2019
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 May 2019 at 20:59:43 UTC, Mike Franklin wrote:


 I don't think it does because `Queue!(T).store` has infinite 
 lifetime beyond that of even `main`, at least as far as the 
 compiler is concerned.  The compiler doesn't have enough 
 information to know that `store` is tied to the lifetime of 
 `Queue!(T)` (a.k.a `rawData`) and maybe that's a missing 
 language feature.  Maybe we should be allowed to declare 
 aggregate fields as `scope` to convey that, but the compiler 
 currently disallows it.


Or we build in some way for slices to know their lifetime 
relative to the source array from which they were created.  But 
I'm not sure how that would work.

Mike
May 17 2019
prev sibling parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 20:59:43 UTC, Mike Franklin wrote:

 On Friday, 17 May 2019 at 17:03:51 UTC, Meta wrote:


 If you look at `main` above, `rawData` has the same lifetime 
 as the `dataRange` struct returned from `makeDataRange` and 
 the queue returned from `copyToQueue`. True, there is some 
 traditionally unsafe stuff happening in between; however, I 
 thought that the point of adding all these annotations is to 
 tell the compiler how the lifetimes of these objects propagate 
 up and down the call stack, so that it can check that there 
 will be no memory corruption. I'm not doing anything here that 
 will result in a pointer to an expired stack frame, or 
 otherwise cause memory corruption or use after free, or 
 anything like that (*unless* I allow either `dataRange` or 
 `result` to escape from the main function - which dip1000 
 correctly disallows).


 I don't think it does because `Queue!(T).store` has infinite 
 lifetime beyond that of even `main`, at least as far as the 
 compiler is concerned.


I see what you're getting at. The compiler sees a slice type 
(i.e., Data[]), knows that it's GC-backed and thus has infinite 
lifetime, and concludes "the data you're trying to put in the 
store has too long of a lifetime". That makes sense, but slices 
don't necessarily have to be backed by the GC, so that seems like 
a faulty heuristic to me and possibly a vector for bugs.


 The compiler doesn't have enough information to know that 
 `store` is tied to the lifetime of `Queue!(T)` (a.k.a 
 `rawData`) and maybe that's a missing language feature.


According to the DIP, "from a lifetime analysis viewpoint, a 
struct is considered a juxtaposition of its direct members." Who 
knows if that's still the case, because Walter has considerably 
changed how it works but has not documented those changes (IIRC, 
I may be wrong on that). That probably means that a Queue!T has 
an infinite lifetime, assuming that the compiler sees its T[] 
member as having an infinite lifetime.


 Maybe we should be allowed to declare aggregate fields as 
 `scope` to convey that, but the compiler currently disallows it.


That might be nice but would also probably cause a dramatic 
increase in complexity. I haven't thought through the possible 
ramifications of making a change like that.
May 17 2019
parent Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 21:57:51 UTC, Meta wrote:

 I see what you're getting at. The compiler sees a slice type 
 (i.e., Data[]), knows that it's GC-backed and thus has infinite 
 lifetime, and concludes "the data you're trying to put in the 
 store has too long of a lifetime".


Should be "too _short_".
May 17 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 11:22:30 PM MDT Mike Franklin via Digitalmars-d-
announce wrote:

 I consider it a bug that the compiler doesn't emit an error when
 using attributes on types for which they are not intended.


As in you think that something like

auto foo(scope int i) {...}

should be illegal, because scope makes no sense on an int? That's nice in
theory, but templates make such an approach a serious problem. It needs to
work to do something like

auto foo(T)(scope T t) {...}

without having to have separate overloads for types where scope makes sense
and types where it doesn't. Similarly, you don't want to have to use static
ifs whenever you declare a variable that you want to be scope in the cases
where the template argument is a type where scope does work.

In general, D ignores attributes when they don't apply rather than making it
an error, because making it an error causes serious problems for generic
code. This does unfortunately mean that some people are bound to sometimes
end up using an attribute when it doesn't apply, thinking that it does, and
that's unfortunate, but overall, it just works better for the compiler not
to complain about such cases.

- Jonathan M Davis
May 17 2019
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 17.05.19 06:50, Meta wrote:

 Walter, can I get you to take a look at this post I made a few months 
 ago, and the contained example? I feel that this is a case that *should* 
 definitely work, but I'm not sure if it can *currently* work - and so 
 far, nobody else seems to be either, save for you.
 
 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org


You don't like my explanation?

https://forum.dlang.org/post/q6r4bf$2hu4$1 digitalmars.com (same thread)
May 16 2019
parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 05:22:31 UTC, ag0aep6g wrote:

 On 17.05.19 06:50, Meta wrote:

 Walter, can I get you to take a look at this post I made a few 
 months ago, and the contained example? I feel that this is a 
 case that *should* definitely work, but I'm not sure if it can 
 *currently* work - and so far, nobody else seems to be either, 
 save for you.
 
 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org


 You don't like my explanation?

 https://forum.dlang.org/post/q6r4bf$2hu4$1 digitalmars.com 
 (same thread)


Your explanation was fine, but I need a good solution, other than 
wrapping the array assignment in  trusted.
May 17 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 17.05.19 14:10, Meta wrote:

 Your explanation was fine, but I need a good solution, other than 
 wrapping the array assignment in  trusted.


I see. As far as I understand DIP 1000, it's not supposed to enable your 
use case without having to use ` trusted`.

DIP 1000 stops at heap allocations. It just assumes infinite lifetime 
for them. If you want to restrict the lifetime of a heap allocation (in 
your case: tie it to the lifetime of a struct), you have to do it manually.
May 17 2019
parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 17:05:21 UTC, ag0aep6g wrote:

 On 17.05.19 14:10, Meta wrote:

 Your explanation was fine, but I need a good solution, other 
 than wrapping the array assignment in  trusted.


 I see. As far as I understand DIP 1000, it's not supposed to 
 enable your use case without having to use ` trusted`.


If this is true, then I have a big problem with DIP1000. This is 
an extremely common use case (copying memory from an inner scope 
with a limited lifetime to some store in an outer scope with a 
longer or infinite lifetime).


 DIP 1000 stops at heap allocations. It just assumes infinite 
 lifetime for them.


Yes, as per the DIP.


 If you want to restrict the lifetime of a heap allocation (in 
 your case: tie it to the lifetime of a struct), you have to do 
 it manually.


I don't want to *restrict* the lifetime of a heap allocation. I 
want the compiler to recognize that the lifetime of my original 
data is the same as the processed output, and thus allow my code 
to compile.
May 17 2019
next sibling parent ag0aep6g <anonymous example.com> writes:
On 17.05.19 19:25, Meta wrote:

 I don't want to *restrict* the lifetime of a heap allocation. I want the 
 compiler to recognize that the lifetime of my original data is the same 
 as the processed output, and thus allow my code to compile.


You have a heap allocation that references your original data, which 
might be on the stack (Queue.store[0] points into rawData). The compiler 
would have to make sure (or recognize) that the pointers on the heap 
don't outlive the data to which they point.

But the lifetime of the heap allocation is infinite, and the lifetime of 
the stack data is not. So that fails. To make it work you'd need some 
way to have a shorter-than-infinite lifetime for the heap.

Whether you want to restrict the lifetime explicitly or have the 
compiler figure it out on its own ... it doesn't really matter when we 
can't do either.
May 17 2019
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, May 17, 2019 11:25:40 AM MDT Meta via Digitalmars-d-announce 
wrote:

 I don't want to *restrict* the lifetime of a heap allocation. I
 want the compiler to recognize that the lifetime of my original
 data is the same as the processed output, and thus allow my code
 to compile.


It is my understanding that DIP 1000 really doesn't track lifetimes at all.
It just ensures that no references to the data escape. So, you can't do
something like take a scope variable and put any references to it or what it
refers to in a container. Honestly, from what I've seen, what you can
ultimately do with scope is pretty limited. It definitely helps in simple
cases, but it quickly gets to the point that it's unable to be used in more
complex cases - at least not without casting and needing to use  trusted.
So, it's an improvement for some kinds of code, but I suspect that in
general, it's just going to be more annoying than it's worth. Time will tell
though.

- Jonathan M Davis
May 17 2019
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/17/2019 11:45 AM, Jonathan M Davis wrote:

 It is my understanding that DIP 1000 really doesn't track lifetimes at all.
 It just ensures that no references to the data escape. So, you can't do
 something like take a scope variable and put any references to it or what it
 refers to in a container. Honestly, from what I've seen, what you can
 ultimately do with scope is pretty limited. It definitely helps in simple
 cases, but it quickly gets to the point that it's unable to be used in more
 complex cases - at least not without casting and needing to use  trusted.
 So, it's an improvement for some kinds of code, but I suspect that in
 general, it's just going to be more annoying than it's worth. Time will tell
 though.


Dip1000 is key to enable containers to control access to pointers to their 
innards that they expose.
May 17 2019
parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Friday, 17 May 2019 at 20:04:42 UTC, Walter Bright wrote:

 Dip1000 is key to enable containers to control access to 
 pointers to their innards that they expose.


I haven't looked at the subject for a while, but every time I did 
the takeaway was the same: dip1000 works great for containers 
until you need to reallocate or free something, at which point 
it's back to  trusted code with you.

I think you said at some point "It's still useful, because it 
reduces the surface of code that needs to be checked", but even 
then saying containers can "control access to the data they 
expose" is a little optimistic.

They only control that access as long as you don't need to call 
resize(). I'd wager that a large fraction of dangling pointer 
errors made by non-beginner C++ developpers come specifically 
from this use case.
May 18 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
If all access to internals is returned by ref, those lifetimes are restricted
to 
the current expression.
May 18 2019
parent Olivier FAURE <couteaubleu gmail.com> writes:
On Saturday, 18 May 2019 at 19:44:37 UTC, Walter Bright wrote:

 If all access to internals is returned by ref, those lifetimes 
 are restricted to the current expression.


Oh my god, I try my best to be open-minded, but talking about 
dip1000 design with you is like pulling teeth *at best*.

Yes, containers work perfectly if you allocate them on the stack 
and use their contents during the current stack frame, and then 
de-allocate them statically. By definition, this represents 0% of 
the use cases of dynamic containers.

Dynamic containers need methods like "push_back", "reserve", 
"resize", "concatenate" or "clear", which are all impossible to 
implement with dip1000 without making their implementations 
trusted, which in turns opens up the program to use-after-free 
memory corruption.

See also:

https://forum.dlang.org/post/qbbipvkjqjeweasxknbn forum.dlang.org

https://forum.dlang.org/post/rxmwjjphnmkszaxonmje forum.dlang.org

Have you talked to Atila Neves at all for the past six months? 
Why the hell are we having this discussion?

This is not a new issue. I have raised it repeatedly in the past 
(I can even dig up the posts if you're interested; I remember 
writing a fairly in-depth analysis at some point). Atila's 
automem and Skoppe's spasm have the same limitation: you can't 
reallocate memory without writing unsafe code (I'm told spasm 
gets around that by never deallocating anything).

Honestly, the fact that you're the only person with a coherent 
vision of dip1000, and yet you keep ignoring problems when 
they're pointed out to you is both worrying and infuriating. Eg:


 So far, the only real shortcoming in the initial design was 
 revealed by the put() semantics, and was fixed with that PR 
 that transmitted scope-ness through the first argument.


Like, yes, I understand that dip1000 is an achievement even if it 
doesn't allow for resizable containers, and that immutable 
already allow for functional programming patterns and that's 
great, but you need to stop acting like everything's going 
perfect when community members (including highly involved library 
writers) have complained about the same things over and over 
again (imprecise semantics, lack of documentation, the resize() 
use case) and you've kept ignoring them.

Seriously, I'm not asking for much. I'm not demanding you take 
any architecture decision or redesign the language (like some 
people are prone to demanding here). But it would be nice if you 
stopped acting like you didn't read a word I wrote, over and over 
again.
May 18 2019
prev sibling parent Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 18:45:12 UTC, Jonathan M Davis wrote:

 On Friday, May 17, 2019 11:25:40 AM MDT Meta via 
 Digitalmars-d-announce wrote:

 I don't want to *restrict* the lifetime of a heap allocation. 
 I want the compiler to recognize that the lifetime of my 
 original data is the same as the processed output, and thus 
 allow my code to compile.


 It is my understanding that DIP 1000 really doesn't track 
 lifetimes at all.


Then why does the DIP, in addition to many of the error messages, 
use the word lifetime? I feel like I know less about DIP1000 and 
what it actually does than when I started. Can someone _please_ 
point me at any up to date documentation on this?
May 17 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2019 9:50 PM, Meta wrote:

 Walter, can I get you to take a look at this post I made a few months ago, and 
 the contained example? I feel that this is a case that *should* definitely
work, 
 but I'm not sure if it can *currently* work - and so far, nobody else seems to 
 be either, save for you.
 
 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org


As always, I recommend drastically reducing the example. It nearly always makes 
the actual problem emerge from all the noise.
May 16 2019
parent reply Meta <jared771 gmail.com> writes:
On Friday, 17 May 2019 at 05:27:02 UTC, Walter Bright wrote:

 On 5/16/2019 9:50 PM, Meta wrote:

 Walter, can I get you to take a look at this post I made a few 
 months ago, and the contained example? I feel that this is a 
 case that *should* definitely work, but I'm not sure if it can 
 *currently* work - and so far, nobody else seems to be either, 
 save for you.
 
 https://forum.dlang.org/post/laqjadtwrsdhdrqokryx forum.dlang.org


 As always, I recommend drastically reducing the example. It 
 nearly always makes the actual problem emerge from all the 
 noise.


I'll try to reduce it further, but this example is already as 
reduced as I could make it while still having the same structure 
as my actual code.
May 17 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/17/2019 10:26 AM, Meta wrote:

 I'll try to reduce it further, but this example is already as reduced as I
could 
 make it while still having the same structure as my actual code.


It doesn't need to have the same structure. It just needs to exhibit the
problem.
May 17 2019
prev sibling next sibling parent Seb <seb wilzba.ch> writes:
On Wednesday, 15 May 2019 at 07:56:48 UTC, Walter Bright wrote:

 On 5/14/2019 11:49 PM, Dukc wrote:

 Time to start compiling your projects with DIP1000, too!


 For me, the forum claims that your posting time is "from the 
 future". Does that mean that is has somehow leaked a draft and 
 this shouldn't show yet?


 Maybe the clock is not synchronized somewhere.


 About -DIP1000, I sure want to use it. But is there currently 
 any practical way to learn it's usage without researching 
 compiler source code?


 Simply add the switch -preview=dip1000 to your builds, and 
 follow where it leads.


For reference, people need to use dmd-nightly (available 
tomorrow), build dmd themselves or wait until DMD 2.087 (1st of 
July).

Also, as most projects use Dub, the flag can be added to the 
dub.sdl too:

---
dflags "-preview=dip1000"
----
May 15 2019
prev sibling parent Kagamin <spam here.lot> writes:
On Wednesday, 15 May 2019 at 07:56:48 UTC, Walter Bright wrote:

 Maybe the clock is not synchronized somewhere.


It's off by one hour.
May 15 2019
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 15 May 2019 at 06:49:02 UTC, Dukc wrote:

 For me, the forum claims that your posting time is "from the 
 future". Does that mean that is has somehow leaked a draft and 
 this shouldn't show yet?


No, it is merged. As to why from the future, probably timezones.
May 15 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2019 11:49 PM, Dukc wrote:

 About -DIP1000, I sure want to use it. But is there currently any practical
way 
 to learn it's usage without researching compiler source code?


This is a good start:

http://dconf.org/2017/talks/bright.html
May 15 2019
parent Dukc <ajieskola gmail.com> writes:
On Wednesday, 15 May 2019 at 08:26:23 UTC, Walter Bright wrote:

 This is a good start:

 http://dconf.org/2017/talks/bright.html


Ah, at least something. Thanks.
May 15 2019
prev sibling next sibling parent M.M. <matus email.cz> writes:
On Wednesday, 15 May 2019 at 07:39:05 UTC, Walter Bright wrote:

 https://github.com/dlang/phobos/pull/6931

 This is a major milestone in improving the memory safety of D 
 programming. Thanks to everyone who helped with this!

 Time to start compiling your projects with DIP1000, too!


Congratulations to the whole team behind it.
May 15 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 15, 2019 at 12:39:05AM -0700, Walter Bright via
Digitalmars-d-announce wrote:

 https://github.com/dlang/phobos/pull/6931
 
 This is a major milestone in improving the memory safety of D
 programming.  Thanks to everyone who helped with this!
 
 Time to start compiling your projects with DIP1000, too!


My very first attempt to compile my code with -preview=dip1000 led to a
regression. :-(

Reduced code:
------
import std.container.rbtree;
alias Grid = RedBlackTree!(GridPoint);
struct GridPoint
{
    private string _srcStr;
    int opCmp(in GridPoint p) const { return 0; }
}
------

Compiler output (with -preview=dip1000):
------
/usr/src/d/phobos/std/container/rbtree.d(1111): Error: ` safe` function
`std.container.rbtree.RedBlackTree!(GridPoint, "a < b",
false).RedBlackTree.toHash` cannot call ` system` function
`core.internal.hash.hashOf!(GridPoint).hashOf`
/usr/src/d/druntime/import/core/internal/hash.d(510):       
`core.internal.hash.hashOf!(GridPoint).hashOf` is declared here
numid.d(3): Error: template instance
`std.container.rbtree.RedBlackTree!(GridPoint, "a < b", false)` error
instantiating
------

The culprit is the 'private' in GridPoint.  Removing 'private' gets rid
of the problem.

*Why* putting 'private' on a field member makes toHash unsafe, is beyond
my ability to comprehend.


T

-- 
Windows: the ultimate triumph of marketing over technology. -- Adrian von Bidder
May 15 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2019 11:09 AM, H. S. Teoh wrote:

 *Why* putting 'private' on a field member makes toHash unsafe, is beyond
 my ability to comprehend.


That's because the reduced version isn't a reduced version. It imports a vast 
amount of other code.
May 15 2019
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 15, 2019 at 11:34:44AM -0700, Walter Bright via
Digitalmars-d-announce wrote:

 On 5/15/2019 11:09 AM, H. S. Teoh wrote:

 *Why* putting 'private' on a field member makes toHash unsafe, is
 beyond my ability to comprehend.


 
 That's because the reduced version isn't a reduced version. It imports
 a vast amount of other code.


Alright, here's a TRULY reduced version:

----
struct S {
	private int _x;
}
struct RedBlackTree
{
    size_t toHash() nothrow  safe
    {
	return .hashOf(S.init);
    }
}
void main() { }
----

Compiling with -preview=dip1000 causes a compile error complaining that
toHash() is not  safe.  Removing 'private' makes it go away. Compiling
without -preview=dip1000 also makes it go away.

Now explain this one. :-D


T

-- 
A linguistics professor was lecturing to his class one day. "In
English," he said, "A double negative forms a positive. In some
languages, though, such as Russian, a double negative is still a
negative. However, there is no language wherein a double positive can
form a negative." A voice from the back of the room piped up, "Yeah,
yeah."
May 15 2019
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 15, 2019 at 05:53:17PM -0700, H. S. Teoh via Digitalmars-d-announce
wrote:

 On Wed, May 15, 2019 at 11:34:44AM -0700, Walter Bright via
Digitalmars-d-announce wrote:

 On 5/15/2019 11:09 AM, H. S. Teoh wrote:

 *Why* putting 'private' on a field member makes toHash unsafe, is
 beyond my ability to comprehend.


 
 That's because the reduced version isn't a reduced version. It
 imports a vast amount of other code.


 
 Alright, here's a TRULY reduced version:


Gah, so apparently .hashOf is a gigantic overload set of *21* different
overloads, so this is not really "truly" reduced. =-O

Anybody up for figuring out which overload(s) is/are getting called?
Betcha the problem is that -preview=dip1000 causes one of the overloads
to fail to compile, thus shuffling to a different overload that isn't
 safe.  I hate SFINAE.


T

-- 
Just because you survived after you did it, doesn't mean it wasn't stupid!
May 15 2019
next sibling parent reply evilrat <evilrat666 gmail.com> writes:
On Thursday, 16 May 2019 at 01:05:53 UTC, H. S. Teoh wrote:

  ...
  I hate SFINAE.


But.. But D doesn't have it!11 NOOO!!1!
May 15 2019
parent Dukc <ajieskola gmail.com> writes:
On Thursday, 16 May 2019 at 04:29:10 UTC, evilrat wrote:

 On Thursday, 16 May 2019 at 01:05:53 UTC, H. S. Teoh wrote:

  ...
  I hate SFINAE.


 But.. But D doesn't have it!11 NOOO!!1!


Not in the same sense as C++. But if the template constrains rely 
of is() statements, that is still a kind of explicitly-activated 
SFINAE.

If that's the case here, it's probably the template constraint 
that's badly designed.
May 16 2019
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 16 May 2019 at 01:05:53 UTC, H. S. Teoh wrote:

 Gah, so apparently .hashOf is a gigantic overload set of *21* 
 different overloads, so this is not really "truly" reduced. =-O

 Anybody up for figuring out which overload(s) is/are getting 
 called?


https://github.com/dlang/druntime/blob/master/src/core/internal/hash.d#L393

static if (hasCallableToHash!(typeof(val))){ ... } // false
else
{
     static if (__traits(hasMember, T, "toHash") && 
is(typeof(T.toHash) == function)) { ... } // false
     else static if (T.tupleof.length == 0) { ... } // false
     else static if ((is(T == struct) && !canBitwiseHash!T) || 
T.tupleof.length == 1)//true
     {
         static foreach (i, F; typeof(val.tupleof))
         {
             static if (__traits(isStaticArray, F)) { ... } // 
false
             else static if (is(F == struct) || is(F == union)) { 
... } // false
             else
             {
                     // Nothing special happening.
                     static if (i == 0 && !isChained)
                         size_t h = hashOf(val.tupleof[i]);
                     else
                         h = hashOf(val.tupleof[i], h);
             }
     }
}


 Betcha the problem is that -preview=dip1000 causes one of the 
 overloads to fail to compile, thus shuffling to a different 
 overload that isn't  safe.  I hate SFINAE.


My money's on access to a private member through .tupleof.
May 15 2019
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 16 May 2019 at 05:14:39 UTC, Nicholas Wilson wrote:

 [...]


Yes that sounds like the culprit. Btw as mentioned on DConf, the 
dip1000 switch contains a few other breaking changes which will 
make it even harder to adopt too.
May 15 2019
parent reply Kagamin <spam here.lot> writes:
On Thursday, 16 May 2019 at 05:22:42 UTC, Seb wrote:

 Yes that sounds like the culprit. Btw as mentioned on DConf, 
 the dip1000 switch contains a few other breaking changes which 
 will make it even harder to adopt too.


Well, it's an inherent property of DIP1000 to not compile code 
that previously compiled. Though safety of tupleof shouldn't 
depend on DIP1000.
May 16 2019
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 16 May 2019 at 10:03:42 UTC, Kagamin wrote:

 On Thursday, 16 May 2019 at 05:22:42 UTC, Seb wrote:

 Yes that sounds like the culprit. Btw as mentioned on DConf, 
 the dip1000 switch contains a few other breaking changes which 
 will make it even harder to adopt too.


 Well, it's an inherent property of DIP1000 to not compile code 
 that previously compiled. Though safety of tupleof shouldn't 
 depend on DIP1000.


Well, here's the full discussion:

https://github.com/dlang/dmd/pull/8035
May 16 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 16, 2019 at 10:35:27AM +0000, Seb via Digitalmars-d-announce wrote:

 On Thursday, 16 May 2019 at 10:03:42 UTC, Kagamin wrote:

 On Thursday, 16 May 2019 at 05:22:42 UTC, Seb wrote:

 Yes that sounds like the culprit. Btw as mentioned on DConf, the
 dip1000 switch contains a few other breaking changes which will
 make it even harder to adopt too.


 
 Well, it's an inherent property of DIP1000 to not compile code that
 previously compiled. Though safety of tupleof shouldn't depend on
 DIP1000.


 
 Well, here's the full discussion:
 
 https://github.com/dlang/dmd/pull/8035


Finally got round to skimming through that discussion.

Looks like in this case, what we need is for toHash to be declared
 trusted when .tupleof includes private members (because toHash is not
supposed to modify any private members, and I assume hashing over a
private member shouldn't violate  safe -- right?).

Either that, or RedBlackTree needs to be changed so that  safe-ty
doesn't depend on random user types having private fields breaking
compilation.  It's pretty ridiculous, from a user's POV, for a standard
container to fail to compile just because the user had the audacity to
declare private members in his object! And the fact that this root
problem is masked under a totally obscure compile error only adds salt
to the wound.


T

-- 
IBM = I Blame Microsoft
May 17 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2019 6:05 PM, H. S. Teoh wrote:

 Gah, so apparently .hashOf is a gigantic overload set of *21* different
 overloads, so this is not really "truly" reduced. =-O


I've often thought that Phobos excessively overused overloading. And you're 
quite right, it's a chore figuring out which one is the culprit.

What I do is change the name(s) to .hashOfx so it won't be picked, then one can 
figure out which one is selected through a process of elimination. Or insert:

     pragma(msg, __LINE__);

statements in each one.
May 16 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 16, 2019 at 10:09:38AM -0700, Walter Bright via
Digitalmars-d-announce wrote:

 On 5/15/2019 6:05 PM, H. S. Teoh wrote:

 Gah, so apparently .hashOf is a gigantic overload set of *21*
 different overloads, so this is not really "truly" reduced. =-O


 
 I've often thought that Phobos excessively overused overloading.


This is in druntime.



 And you're quite right, it's a chore figuring out which one is the
 culprit.


More and more, I'm becoming convinced that this sort of usage of
function overloading is an anti-pattern.  It should instead be written
as something like this:

	size_t hashOf(T)(T* arg)
		// N.B.: no sig constraints: because we expect to be
		// able to hash anything.
	{
		static if (is(T == struct))
			return hashOfStruct(arg);
		else static if (is(T == U[], U))
			return hashOfArray(arg);
		...
		else
			static assert(0, "Hash of " ~ T.stringof ~ " not supported");
	}

The sig constraints, or lack thereof, ought to reflect the *logical* set
of acceptable types, not necessarily the actual set supported by the
implementation. I.e., hashOf logically *should* support all types, but
maybe the current implementation doesn't (yet) support a particular
corner case; so it should still accept the type, but emit an error
explaining the implementation deficiency in a static assert, rather than
just passing the buck back to the compiler which then spews forth a text
wall of incomprehensible gibberish of how all 21 overloads failed to
match.



 What I do is change the name(s) to .hashOfx so it won't be picked,
 then one can figure out which one is selected through a process of
 elimination. Or insert:
 
     pragma(msg, __LINE__);
 
 statements in each one.


Good idea. But looks like Nicholas has already done the heavy lifting
for us.  :-D


T

-- 
Always remember that you are unique. Just like everybody else. -- despair.com
May 16 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2019 10:24 AM, H. S. Teoh wrote:

 More and more, I'm becoming convinced that this sort of usage of
 function overloading is an anti-pattern.  It should instead be written
 as something like this:


I agree. And there at least you can find all the uses.


 Good idea. But looks like Nicholas has already done the heavy lifting
 for us.  :-D


The problem will come up again.
May 16 2019
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 15, 2019 at 11:09:01AM -0700, H. S. Teoh via Digitalmars-d-announce
wrote:

 On Wed, May 15, 2019 at 12:39:05AM -0700, Walter Bright via
Digitalmars-d-announce wrote:

 https://github.com/dlang/phobos/pull/6931
 
 This is a major milestone in improving the memory safety of D
 programming.  Thanks to everyone who helped with this!
 
 Time to start compiling your projects with DIP1000, too!


 
 My very first attempt to compile my code with -preview=dip1000 led to
 a regression. :-(


[...]

Bugzilla issue:

	https://issues.dlang.org/show_bug.cgi?id=19877


T

-- 
To err is human; to forgive is not our policy. -- Samuel Adler
May 15 2019