www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - can we un-deprecate .ptr on arrays in safe code? cf issue 18529

reply Timothee Cour <thelastmammoth gmail.com> writes:
see rationale in https://issues.dlang.org/show_bug.cgi?id=18529
Feb 27 2018
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Tuesday, 27 February 2018 at 08:43:32 UTC, Timothee Cour wrote:

 see rationale in https://issues.dlang.org/show_bug.cgi?id=18529


It looks like the actual deprecation was made with this PR:  
https://github.com/dlang/dmd/pull/5860  Meaning it's been 
deprecated for more than a year and a half.

I think you're going to have to take it up with Walter.

Mike
Feb 27 2018
prev sibling next sibling parent reply bauss <jj_1337 live.dk> writes:
On Tuesday, 27 February 2018 at 08:43:32 UTC, Timothee Cour wrote:

 see rationale in https://issues.dlang.org/show_bug.cgi?id=18529


All I see is a rationale for how it can't be replaced, but not a 
rationale for actually doing it.

I can't imagine a "safe" situation where you want to do it.

Tbh. just put your code in  trusted
Feb 27 2018
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 27 February 2018 at 09:23:19 UTC, bauss wrote:

 On Tuesday, 27 February 2018 at 08:43:32 UTC, Timothee Cour 
 wrote:

 see rationale in https://issues.dlang.org/show_bug.cgi?id=18529


 All I see is a rationale for how it can't be replaced, but not 
 a rationale for actually doing it.

 I can't imagine a "safe" situation where you want to do it.

 Tbh. just put your code in  trusted


Checking if an array is the slice of another.
Feb 27 2018
next sibling parent reply bauss <jj_1337 live.dk> writes:
On Tuesday, 27 February 2018 at 09:47:51 UTC, Stefan Koch wrote:

 On Tuesday, 27 February 2018 at 09:23:19 UTC, bauss wrote:

 On Tuesday, 27 February 2018 at 08:43:32 UTC, Timothee Cour 
 wrote:

 see rationale in 
 https://issues.dlang.org/show_bug.cgi?id=18529


 All I see is a rationale for how it can't be replaced, but not 
 a rationale for actually doing it.

 I can't imagine a "safe" situation where you want to do it.

 Tbh. just put your code in  trusted


 Checking if an array is the slice of another.


Like?

assert(a.ptr != b.ptr);

Which already works in  safe.
Feb 27 2018
parent reply Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On Tuesday, 27 February 2018 at 09:58:00 UTC, bauss wrote:

 On Tuesday, 27 February 2018 at 09:47:51 UTC, Stefan Koch wrote:

 On Tuesday, 27 February 2018 at 09:23:19 UTC, bauss wrote:

 On Tuesday, 27 February 2018 at 08:43:32 UTC, Timothee Cour 
 wrote:

 see rationale in 
 https://issues.dlang.org/show_bug.cgi?id=18529


 All I see is a rationale for how it can't be replaced, but 
 not a rationale for actually doing it.

 I can't imagine a "safe" situation where you want to do it.

 Tbh. just put your code in  trusted


 Checking if an array is the slice of another.


 Like?

 assert(a.ptr != b.ptr);

 Which already works in  safe.


That only checks if the first element is the same. For a full 'is 
slice' check you'd need something like this:

     assert(a.ptr <= b.ptr && b.ptr + b.length <= a.ptr + 
a.length);

// Or:

     auto c = b.ptr - a.ptr;
     assert(c >= 0 && c + b.length <= a.length);

And trust me, the compiler complains about both of these. 
Possibly rightfully in the first example, but the latter never 
does anything scary with the given pointers.

--
   Simen
Feb 27 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, February 27, 2018 11:33:04 Simen Kjærås via Digitalmars-d wrote:

 And trust me, the compiler complains about both of these.
 Possibly rightfully in the first example, but the latter never
 does anything scary with the given pointers.


As I understand it, the way that  safety checks generally work is they check
whether a particular operation is  safe or not. They don't usually care
about what is then done with the result. So, if you do something like take
the address of something, that's immediately  system regardless of what you
do with the result. That changes on some level with DIP 1000 and scope,
because then it uses scope to ensure that the lifetime of stuff like
pointers doesn't exceed the lifetime of what they point to so that it can
then know that taking the address is  safe, but without DIP 1000, it takes
very little for something to become  system. e.g. this is compiles with
-dip1000 but otherwise doesn't:

void main()  safe
{
    int i;
    assert(&i !is null);
}

Now, the compiler does seem to be a bit smarter with dynamic arrays and ptr
given that this compiles without -dip1000

void main()  safe
{
    int[] i;
    assert(i.ptr !is null);
}

However, this doesn't compile with -dip1000:

void main()  safe
{
    int[] i;
    auto j = i.ptr;
    assert(j !is null);
}

and not even this compiles with -dip1000:

void main()  safe
{
    int[] i;
    scope j = i.ptr;
    assert(j !is null);
}

though I'm inclined to think that that's a bug from what I understand of
-dip1000.

In any case,  safety checks tend to be fairly primitive, so once you start
mucking around with pointers, it's not hard to write code that gets treated
as  system because of a single expression in the code that is clearly  safe
within the context of the function, but the compiler can't see it.

And for better or worse, accessing a dynamic array's ptr member is now
 system, because it's not  safe in all circumstances. If the compiler were
smarter, then a number of uses of ptr would probably be  safe, but its
analysis for stuff like that is usually pretty primitive, in part because
making it sophisticated requires stuff like code flow analysis, which the
compiler doesn't do a lot of, precisely because it is complicated and easy
to get wrong. Walter is particularly leery about making it so that stuff is
an error or not based on code flow analysis, and  safe falls into that camp.
Clearly, some of that is going on with DIP 1000, but that seems to be
largely by using the type system to solve the problem rather than doing much
in the way of code flow analysis.

- Jonathan M Davis
Feb 27 2018
prev sibling next sibling parent reply Timothee Cour <thelastmammoth gmail.com> writes:
this would be more bearable if there was a standard  trusted method to
get array `.ptr`, eg:
in `object.d` (so that it's indeed standard)

```
 trusted  nogc pure nothrow
auto pointer(T)(T a){
  return a.ptr;
}
```

again, the deprecation message is misleading because `&a[0]` isn't
equivalent to `a.ptr`
having something like `pointer` (and making deprecation msg use that)
would be a better mitigation



On Tue, Feb 27, 2018 at 3:56 AM, Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, February 27, 2018 11:33:04 Simen Kjærås via Digitalmars-d wrote:

 And trust me, the compiler complains about both of these.
 Possibly rightfully in the first example, but the latter never
 does anything scary with the given pointers.


 As I understand it, the way that  safety checks generally work is they check
 whether a particular operation is  safe or not. They don't usually care
 about what is then done with the result. So, if you do something like take
 the address of something, that's immediately  system regardless of what you
 do with the result. That changes on some level with DIP 1000 and scope,
 because then it uses scope to ensure that the lifetime of stuff like
 pointers doesn't exceed the lifetime of what they point to so that it can
 then know that taking the address is  safe, but without DIP 1000, it takes
 very little for something to become  system. e.g. this is compiles with
 -dip1000 but otherwise doesn't:

 void main()  safe
 {
     int i;
     assert(&i !is null);
 }

 Now, the compiler does seem to be a bit smarter with dynamic arrays and ptr
 given that this compiles without -dip1000

 void main()  safe
 {
     int[] i;
     assert(i.ptr !is null);
 }

 However, this doesn't compile with -dip1000:

 void main()  safe
 {
     int[] i;
     auto j = i.ptr;
     assert(j !is null);
 }

 and not even this compiles with -dip1000:

 void main()  safe
 {
     int[] i;
     scope j = i.ptr;
     assert(j !is null);
 }

 though I'm inclined to think that that's a bug from what I understand of
 -dip1000.

 In any case,  safety checks tend to be fairly primitive, so once you start
 mucking around with pointers, it's not hard to write code that gets treated
 as  system because of a single expression in the code that is clearly  safe
 within the context of the function, but the compiler can't see it.

 And for better or worse, accessing a dynamic array's ptr member is now
  system, because it's not  safe in all circumstances. If the compiler were
 smarter, then a number of uses of ptr would probably be  safe, but its
 analysis for stuff like that is usually pretty primitive, in part because
 making it sophisticated requires stuff like code flow analysis, which the
 compiler doesn't do a lot of, precisely because it is complicated and easy
 to get wrong. Walter is particularly leery about making it so that stuff is
 an error or not based on code flow analysis, and  safe falls into that camp.
 Clearly, some of that is going on with DIP 1000, but that seems to be
 largely by using the type system to solve the problem rather than doing much
 in the way of code flow analysis.

 - Jonathan M Davis
Feb 27 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/27/18 7:20 AM, Timothee Cour wrote:

 this would be more bearable if there was a standard  trusted method to
 get array `.ptr`, eg:
 in `object.d` (so that it's indeed standard)
 
 ```
  trusted  nogc pure nothrow
 auto pointer(T)(T a){
    return a.ptr;
 }
 ```


That completely defeats the purpose of the restriction! Of course we 
aren't going to do that :)

This is the standard way (and it's on you to make sure you don't mess it 
up):

auto x = (()  trusted => a.ptr)();


 again, the deprecation message is misleading because `&a[0]` isn't
 equivalent to `a.ptr`


It isn't equivalent, but it is a  safe mechanism to do it. If you want 
to do unsafe things, then you need to use  trusted escapes, 
druntime/phobos is not going to help you there.

-Steve
Feb 27 2018
prev sibling parent Seb <seb wilzba.ch> writes:
On Tuesday, 27 February 2018 at 11:33:04 UTC, Simen Kjærås wrote:

 That only checks if the first element is the same. For a full 
 'is slice' check you'd need something like this:

     assert(a.ptr <= b.ptr && b.ptr + b.length <= a.ptr + 
 a.length);

 // Or:

     auto c = b.ptr - a.ptr;
     assert(c >= 0 && c + b.length <= a.length);

 And trust me, the compiler complains about both of these. 
 Possibly rightfully in the first example, but the latter never 
 does anything scary with the given pointers.

 --
   Simen


aka isSliceOf -> https://github.com/dlang/phobos/pull/6147
Feb 27 2018
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
I don't think a just iterated array is automatically set to null, 
so taking it's pointer won't hit a memory-proteted area. So 
undeprectating arr.ptr in  safe would break  safety and be a step 
backward.

If this is an issue, one can define a  trusted function which 
takes a starting pointer from array and casts it to size_t before 
returning it so memory corruption cannot happen via it.

On Tuesday, 27 February 2018 at 09:47:51 UTC, Stefan Koch wrote:

 Checking if an array is the slice of another.


For that there is also std.array.overlap().
Feb 28 2018
parent Cym13 <cpicard openmailbox.org> writes:
On Wednesday, 28 February 2018 at 22:34:07 UTC, Dukc wrote:

 I don't think a just iterated array is automatically set to 
 null, so taking it's pointer won't hit a memory-proteted area. 
 So undeprectating arr.ptr in  safe would break  safety and be a 
 step backward.


If it cannot be proven safe by the compiler but is safe anyway it 
doesn't break  safe, it should just be  trusted. That's what 
 trusted is about, not about abusing functions to trick unsafe 
things into compiling  safe-ly.


 If this is an issue, one can define a  trusted function which 
 takes a starting pointer from array and casts it to size_t 
 before returning it so memory corruption cannot happen via it.

 On Tuesday, 27 February 2018 at 09:47:51 UTC, Stefan Koch wrote:

 Checking if an array is the slice of another.


 For that there is also std.array.overlap().
Feb 28 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, February 27, 2018 04:20:38 Timothee Cour via Digitalmars-d 
wrote:

 this would be more bearable if there was a standard  trusted method to
 get array `.ptr`, eg:
 in `object.d` (so that it's indeed standard)

 ```
  trusted  nogc pure nothrow
 auto pointer(T)(T a){
   return a.ptr;
 }
 ```


Except that that's really not how  trusted is supposed to be used. The
programmer needs to verify that the caller is using a.ptr in a manner that
is actually  safe, because the compiler is not smart enough to determine
that for you. Wrapping it in an  trusted function means that the caller
won't get an error and that the programmer won't necessarily know that they
need to verify the calling code. It's the code that's using ptr that needs
to be verified, not the actual accessing of ptr.

Hiding the access to ptr within an  trusted function goes against that
entire idea of  trusted and makes it easy to use ptr without realizing that
you need to be checking the code that's using it, since you just called a
wrapper function to silence the compiler instead of listening the compiler
and studying the code using ptr to verify its  safety.


 again, the deprecation message is misleading because `&a[0]` isn't
 equivalent to `a.ptr`
 having something like `pointer` (and making deprecation msg use that)
 would be a better mitigation


In almost all cases, &a[0] is equivalent to a.ptr except that it does bounds
checking, so it's actually  safe and thus doesn't need to be manually
verified by the programmer, unlike your pointer function suggestion.

If you have a use case where you need a.ptr rather than &a[0], then that
just means that you're going to have to verify that your code is  safe in
spite of using an  system operation and mark it  trusted if it is. Simply
treating a.ptr as  safe when it isn't or wrapping it in an  trusted function
just defeats the purpose of the whole  safety system.

Now, maybe in some simple cases, the compiler can be improved to detect that
what you're doing with a.ptr is actually  safe and not give an error, but as
long as it can't, any use of a.ptr needs to be verified by the programmer.

- Jonathan M Davis
Feb 27 2018
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 27 February 2018 at 12:39:04 UTC, Jonathan M Davis 
wrote:

 On Tuesday, February 27, 2018 04:20:38 Timothee Cour via 
 Digitalmars-d wrote:

 [...]


 Except that that's really not how  trusted is supposed to be 
 used. The programmer needs to verify that the caller is using 
 a.ptr in a manner that is actually  safe, because the compiler 
 is not smart enough to determine that for you. Wrapping it in 
 an  trusted function means that the caller won't get an error 
 and that the programmer won't necessarily know that they need 
 to verify the calling code. It's the code that's using ptr that 
 needs to be verified, not the actual accessing of ptr.

 Hiding the access to ptr within an  trusted function goes 
 against that entire idea of  trusted and makes it easy to use 
 ptr without realizing that you need to be checking the code 
 that's using it, since you just called a wrapper function to 
 silence the compiler instead of listening the compiler and 
 studying the code using ptr to verify its  safety.


 [...]


 In almost all cases, &a[0] is equivalent to a.ptr except that 
 it does bounds checking, so it's actually  safe and thus 
 doesn't need to be manually verified by the programmer, unlike 
 your pointer function suggestion.


There's a common case where it's not equivalent - when the 
pointer is null. Imagine I have a C function I want to call:

extern(C) void fun(int* things);

Imagine also that it's ok to call with null. Well, now I can't 
use a slice to call this and have it be 1)  safe and 2) not throw 
RangeError. I ran into this the other way.

Atila
Feb 27 2018
next sibling parent ag0aep6g <anonymous example.com> writes:
On 02/27/2018 06:32 PM, Atila Neves wrote:

 On Tuesday, 27 February 2018 at 12:39:04 UTC, Jonathan M Davis wrote:


[...]

 In almost all cases, &a[0] is equivalent to a.ptr except that it does 
 bounds checking, so it's actually  safe and thus doesn't need to be 
 manually verified by the programmer, unlike your pointer function 
 suggestion.


 
 There's a common case where it's not equivalent - when the pointer is 
 null. Imagine I have a C function I want to call:
 
 extern(C) void fun(int* things);
 
 Imagine also that it's ok to call with null. Well, now I can't use a 
 slice to call this and have it be 1)  safe and 2) not throw RangeError. 
 I ran into this the other way.


As Jonathan says, you have to manually verify that it's safe, because it 
generally isn't. `arr.ptr` can be invalid but not null, even in  safe code.

Consider:

----
void fun(int* things)  safe
{
     int x = things is null ? 0 : *things;
     import std.stdio;
     writeln(x);
}

void main()
{
     int[] arr;
     fun(arr.ptr); /* Ok, prints "0". */

     arr = [1, 2, 3];
     fun(arr.ptr); /* Ok, prints "1".*/

     arr = arr[$ .. $]; /* This is  safe. */
     fun(arr.ptr); /* Not ok, prints garbage. */
}
----

The first two calls are actually safe and can be  trusted. The third 
call is invalid and must not be possible in  safe code.

Maybe it would be possible require `arr.ptr` to be valid or null in 
 safe code. This would outlaw `arr[$ .. $]` and bounds checking would 
have to catch it. I don't know if that would be any practical than 
banning `arr.ptr`.
Feb 27 2018
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/27/18 12:32 PM, Atila Neves wrote:


 There's a common case where it's not equivalent - when the pointer is 
 null. Imagine I have a C function I want to call:
 
 extern(C) void fun(int* things);
 
 Imagine also that it's ok to call with null. Well, now I can't use a 
 slice to call this and have it be 1)  safe and 2) not throw RangeError. 
 I ran into this the other way.


fun(x.length ? &x[0] : null);

I think even the compiler could elide the bounds check since you already 
did it, but I'm not sure that actually happens.

-Steve
Feb 27 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/27/18 3:00 PM, Steven Schveighoffer wrote:

 On 2/27/18 12:32 PM, Atila Neves wrote:
 

 There's a common case where it's not equivalent - when the pointer is 
 null. Imagine I have a C function I want to call:

 extern(C) void fun(int* things);

 Imagine also that it's ok to call with null. Well, now I can't use a 
 slice to call this and have it be 1)  safe and 2) not throw 
 RangeError. I ran into this the other way.


 
 fun(x.length ? &x[0] : null);


Hm... borrowing from Timothee's suggestion:

 trusted  nogc pure nothrow
T* pointer(T)(T[] a){
    return a.length > 0 ? a.ptr : null;
}

This would be fine and  safe, but may not be useful for all purposes. 
However, it would fix your issue.

-Steve
Feb 27 2018
parent Timothee Cour <thelastmammoth gmail.com> writes:
 Hm... borrowing from Timothee's suggestion:
 This would be fine and  safe, but may not be useful for all purposes. However,
it would fix your issue.



how about this: https://github.com/dlang/phobos/pull/6231

On Tue, Feb 27, 2018 at 12:09 PM, Steven Schveighoffer via
Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On 2/27/18 3:00 PM, Steven Schveighoffer wrote:

 On 2/27/18 12:32 PM, Atila Neves wrote:


 There's a common case where it's not equivalent - when the pointer is
 null. Imagine I have a C function I want to call:

 extern(C) void fun(int* things);

 Imagine also that it's ok to call with null. Well, now I can't use a
 slice to call this and have it be 1)  safe and 2) not throw RangeError. I
 ran into this the other way.



 fun(x.length ? &x[0] : null);



 Hm... borrowing from Timothee's suggestion:

  trusted  nogc pure nothrow
 T* pointer(T)(T[] a){
    return a.length > 0 ? a.ptr : null;
 }

 This would be fine and  safe, but may not be useful for all purposes.
 However, it would fix your issue.

 -Steve
Feb 27 2018
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/27/2018 4:39 AM, Jonathan M Davis wrote:

 Except that that's really not how  trusted is supposed to be used. The
 programmer needs to verify that the caller is using a.ptr in a manner that
 is actually  safe, because the compiler is not smart enough to determine
 that for you. Wrapping it in an  trusted function means that the caller
 won't get an error and that the programmer won't necessarily know that they
 need to verify the calling code. It's the code that's using ptr that needs
 to be verified, not the actual accessing of ptr.
 
 Hiding the access to ptr within an  trusted function goes against that
 entire idea of  trusted and makes it easy to use ptr without realizing that
 you need to be checking the code that's using it, since you just called a
 wrapper function to silence the compiler instead of listening the compiler
 and studying the code using ptr to verify its  safety.


Yes, this bears repeating.
Feb 27 2018