www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - nogc v0.5.0 - DIP1008 works!

reply Atila Neves <atila.neves gmail.com> writes:
I'd been holding off on announcing this until DIP1008 actually 
got implemented, and now it has:

https://code.dlang.org/packages/nogc

This dub package has a  nogc version of `std.conv.text` (which 
probably isn't as good yet) that, instead of returning a `string` 
returns an `automem.vector.Vector` of char. This handles managing 
memory allocation for the exception message itself in 
`NoGcException`, which does what it says on the tin. Confused? 
Here's some code:


 safe  nogc unittest {
     import nogc;
     import std.algorithm: equal;

     int a = 1, b = 2;
     try
         enforce(a == b, a, " was not equal to ", b);
     catch(NoGcException e) {
         assert(equal(e.msg, "1 was not equal to 2"));
     }

     try
         throw new NoGcException(42, " foobar ", 33.3);
     catch(NoGcException e) {
         assert(equal(e.msg, "42 foobar 33.300000"));
         assert(e.file == __FILE__);
         assert(e.line == __LINE__ - 4);
     }
}

It doesn't leak memory either, as proved by ldc's asan.
May 24
next sibling parent Seb <seb wilzba.ch> writes:
On Friday, 24 May 2019 at 11:41:12 UTC, Atila Neves wrote:
 I'd been holding off on announcing this until DIP1008 actually 
 got implemented, and now it has:

 [...]
Awesome!! Now we just need to get to compile Druntime and Phobos with -preview=dip1008, s.t. we can enable it by default :) See e.g. https://github.com/dlang/dmd/pull/8508
May 24
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 24.05.19 13:41, Atila Neves wrote:
 I'd been holding off on announcing this until DIP1008 actually got 
 implemented, and now it has:
 
 https://code.dlang.org/packages/nogc
You've got safety violations: ---- /+ dub.sdl: name "test" dependency "nogc" version="~>0.5.0" +/ import core.stdc.stdio: puts; struct S1 { S2 s2; this(ref const S1 src) const nogc system { this.s2 = src.s2; } } struct S2 { this(ref const S2 src) const nogc system { puts(" system 1"); } } struct Z { char* stringz() const nogc system { puts(" system 2"); return null; } } struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; enum instance = UnsafeAllocator.init; void deallocate(void[] bytes) nogc system { puts(" system 3"); Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { puts(" system 4"); return Mallocator.instance.allocate(sz); } } void main() safe nogc { import nogc: BUFFER_SIZE, text; S1 a; Z* z; auto t = text!(BUFFER_SIZE, UnsafeAllocator)(a, z); } ---- All of the `puts` lines are executed. That should not be possible in safe code. You're applying trusted too liberally.
May 24
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 24 May 2019 at 12:32:45 UTC, ag0aep6g wrote:
 On 24.05.19 13:41, Atila Neves wrote:
 [...]
You've got safety violations: ---- /+ dub.sdl: name "test" dependency "nogc" version="~>0.5.0" +/ import core.stdc.stdio: puts; struct S1 { S2 s2; this(ref const S1 src) const nogc system { this.s2 = src.s2; } } struct S2 { this(ref const S2 src) const nogc system { puts(" system 1"); } } struct Z { char* stringz() const nogc system { puts(" system 2"); return null; } } struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; enum instance = UnsafeAllocator.init; void deallocate(void[] bytes) nogc system { puts(" system 3"); Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { puts(" system 4"); return Mallocator.instance.allocate(sz); } } void main() safe nogc { import nogc: BUFFER_SIZE, text; S1 a; Z* z; auto t = text!(BUFFER_SIZE, UnsafeAllocator)(a, z); } ---- All of the `puts` lines are executed. That should not be possible in safe code. You're applying trusted too liberally.
Thanks for this. I think the only violation is calling `stringz` on `Z`, and that was due to a poorly designed DbI check on being able to call `stringz`. Allocating generally isn't system, and freeing is ok to trust since vector is taking care of it for us. I've pushed a fix.
May 24
parent reply ag0aep6g <anonymous example.com> writes:
On Friday, 24 May 2019 at 13:13:12 UTC, Atila Neves wrote:
 Thanks for this. I think the only violation is calling 
 `stringz` on `Z`, and that was due to a poorly designed DbI 
 check on being able to call `stringz`. Allocating generally 
 isn't  system, and freeing is ok to trust since vector is 
 taking care of it for us. I've pushed a fix.
I think you're missing the point. When your function is marked as safe or trusted, then any execution of a user-provided system function is a safety violation. My `puts`s might not do any harm, but they could just as well be buffer overflows.
May 24
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:
 On Friday, 24 May 2019 at 13:13:12 UTC, Atila Neves wrote:
 Thanks for this. I think the only violation is calling 
 `stringz` on `Z`, and that was due to a poorly designed DbI 
 check on being able to call `stringz`. Allocating generally 
 isn't  system, and freeing is ok to trust since vector is 
 taking care of it for us. I've pushed a fix.
I think you're missing the point. When your function is marked as safe or trusted, then any execution of a user-provided system function is a safety violation. My `puts`s might not do any harm, but they could just as well be buffer overflows.
Could you please give an example of how system allocator code could do that?
May 24
parent reply ag0aep6g <anonymous example.com> writes:
On 24.05.19 18:19, Atila Neves wrote:
 On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:
[...]
 My `puts`s might not do any harm, but they could just as well be 
 buffer overflows.
Could you please give an example of how system allocator code could do that?
Sure. You just write beyond some buffer instead of calling `puts`: ---- char[3] buf; char[3] foo = "foo"; char[3] bar = "bar"; struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; static instance = UnsafeAllocator.init; size_t i; void deallocate(void[] bytes) nogc system { buf.ptr[i .. i + 3] = '!'; Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { buf.ptr[i .. i + 3] = '!'; return Mallocator.instance.allocate(sz); } } void main() safe nogc { { import nogc: BUFFER_SIZE, text; UnsafeAllocator.instance.i = 8; /* greater than buf.length, whoops */ auto t = text!(BUFFER_SIZE, UnsafeAllocator)(42); assert(foo == "foo"); /* fails */ UnsafeAllocator.instance.i = 16; /* also greater than buf.length, whoops again */ } assert(bar == "bar"); /* fails */ } ---- You just can't trust user-provided system code. It doesn't matter if it's allocator code or whatever.
May 24
next sibling parent ag0aep6g <anonymous example.com> writes:
On 24.05.19 18:51, ag0aep6g wrote:
 char[3] buf;
[...]
          buf.ptr[i .. i + 3] = '!';
[...]
          UnsafeAllocator.instance.i = 8;
              /* greater than buf.length, whoops */
Actually, anything greater than zero is a whoops, of course.
May 24
prev sibling next sibling parent Meta <jared771 gmail.com> writes:
On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:
 On 24.05.19 18:19, Atila Neves wrote:
 On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:
[...]
 My `puts`s might not do any harm, but they could just as well 
 be buffer overflows.
Could you please give an example of how system allocator code could do that?
Sure. You just write beyond some buffer instead of calling `puts`: ---- char[3] buf; char[3] foo = "foo"; char[3] bar = "bar"; struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; static instance = UnsafeAllocator.init; size_t i; void deallocate(void[] bytes) nogc system { buf.ptr[i .. i + 3] = '!'; Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { buf.ptr[i .. i + 3] = '!'; return Mallocator.instance.allocate(sz); } } void main() safe nogc { { import nogc: BUFFER_SIZE, text; UnsafeAllocator.instance.i = 8; /* greater than buf.length, whoops */ auto t = text!(BUFFER_SIZE, UnsafeAllocator)(42); assert(foo == "foo"); /* fails */ UnsafeAllocator.instance.i = 16; /* also greater than buf.length, whoops again */ } assert(bar == "bar"); /* fails */ } ---- You just can't trust user-provided system code. It doesn't matter if it's allocator code or whatever.
That's right. If you are wrapping code that is provided by a third party, you should not mark any code as trusted that makes calls to the third party library. By doing this, you are saying "any third party code I call is memory safe (source: dude just trust me)". That may work in the case where this third party code is set in stone and has been hand-audited by either you or the maintainers (ideally both), but you're accepting any implementation through a template argument. Doing this is extremely dangerous, because you're making memory safety promises about every single Allocator implementation in existence, in the present AND for the future. What you have to do is leave the functions that make these calls unmarked (no system, trusted OR safe), and allow the compiler to infer it based on the whether the third party implementation is system/ trusted/ safe. That's the only sane way I can think of to do this.
May 24
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:
 On 24.05.19 18:19, Atila Neves wrote:
 On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:
[...]
 My `puts`s might not do any harm, but they could just as well 
 be buffer overflows.
Could you please give an example of how system allocator code could do that?
Sure. You just write beyond some buffer instead of calling `puts`: ---- char[3] buf; char[3] foo = "foo"; char[3] bar = "bar"; struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; static instance = UnsafeAllocator.init; size_t i; void deallocate(void[] bytes) nogc system { buf.ptr[i .. i + 3] = '!'; Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { buf.ptr[i .. i + 3] = '!'; return Mallocator.instance.allocate(sz); } } void main() safe nogc { { import nogc: BUFFER_SIZE, text; UnsafeAllocator.instance.i = 8; /* greater than buf.length, whoops */ auto t = text!(BUFFER_SIZE, UnsafeAllocator)(42); assert(foo == "foo"); /* fails */ UnsafeAllocator.instance.i = 16; /* also greater than buf.length, whoops again */ } assert(bar == "bar"); /* fails */ } ---- You just can't trust user-provided system code. It doesn't matter if it's allocator code or whatever.
I don't see the problem here. This example would throw RangeError at runtime instead of actually overwriting memory unless bounds checking is turned off. The other issue is that Mallocator has a safe allocate function and a system deallocate function since it's up to the user of the interface to supply a slice that was actually malloc'ed. It's clear that this interface is one that can be used safely (and is by automem.vector.Vector). Likewise, reallocating is system because there might be references to the old pointer, but it makes sense that a trusted block could exist where the reviewer makes sure that there's only ever one reference to the allocated memory. Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with trusted, and unless someone looks at their code nobody will be the wiser.
May 27
next sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 27 May 2019 at 08:54:45 UTC, Atila Neves wrote:
 On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:
 Then there's the fact that if a 3rd party library really does 
 want to corrupt memory they can just tag all their functions 
 with  trusted, and unless someone looks at their code nobody 
 will be the wiser.
... and a safe conscious programmer will not touch that library ever with a 5 five meters pole. I'm still not convinced that trusted code should accept generic system code ... can you elaborate more? Thanks, Paolo
May 27
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 27 May 2019 at 09:07:48 UTC, Paolo Invernizzi wrote:
 On Monday, 27 May 2019 at 08:54:45 UTC, Atila Neves wrote:
 On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:
 Then there's the fact that if a 3rd party library really does 
 want to corrupt memory they can just tag all their functions 
 with  trusted, and unless someone looks at their code nobody 
 will be the wiser.
... and a safe conscious programmer will not touch that library ever with a 5 five meters pole. I'm still not convinced that trusted code should accept generic system code ... can you elaborate more?
I'm not convinced either - I'm having a dialogue to figure out potential issues.
May 27
parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 27 May 2019 at 10:01:15 UTC, Atila Neves wrote:
 On Monday, 27 May 2019 at 09:07:48 UTC, Paolo Invernizzi wrote:
 On Monday, 27 May 2019 at 08:54:45 UTC, Atila Neves wrote:
 On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:
 Then there's the fact that if a 3rd party library really does 
 want to corrupt memory they can just tag all their functions 
 with  trusted, and unless someone looks at their code nobody 
 will be the wiser.
... and a safe conscious programmer will not touch that library ever with a 5 five meters pole. I'm still not convinced that trusted code should accept generic system code ... can you elaborate more?
I'm not convinced either - I'm having a dialogue to figure out potential issues.
:-) My nice-try to reduce the problem is: trusted code block needs to by "manually verified" for safety by humans, so it should be " safe pure", aka, if you can't perform the analysis looking only at the statements in the trusted block, that can't be marked trusted.
May 27
prev sibling parent reply ag0aep6g <anonymous example.com> writes:
On 27.05.19 10:54, Atila Neves wrote:
 I don't see the problem here. This example would throw RangeError at 
 runtime instead of actually overwriting memory unless bounds checking is 
 turned off.
No, it doesn't. It's a complete, runnable example. You can try it at home. It does overwrite `foo` and `bar`. It does not throw a RangeError.
 The other issue is that Mallocator has a  safe allocate function and a 
  system deallocate function since it's up to the user of the interface 
 to supply a slice that was actually malloc'ed. It's clear that this 
 interface is one that can be used  safely (and is by 
 automem.vector.Vector). Likewise, reallocating is  system because there 
 might be references to the old pointer, but it makes sense that a 
  trusted block could exist where the reviewer makes sure that there's 
 only ever one reference to the allocated memory.
Yes, you can use trusted to use Mallocator safely. And your code (probably) does that. But the allocator in my example isn't Mallocator, it's UnsafeAllocator. Your code doesn't use that one safely.
 Then there's the fact that if a 3rd party library really does want to 
 corrupt memory they can just tag all their functions with  trusted, and 
 unless someone looks at their code nobody will be the wiser.
In this thread, you're the author of that 3rd party library. You've got the bad trusted functions that lead to memory corruption. I'm the guy who looked at it, noticed the problem, and tells you.
May 27
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 27 May 2019 at 09:48:27 UTC, ag0aep6g wrote:
 On 27.05.19 10:54, Atila Neves wrote:
 I don't see the problem here. This example would throw 
 RangeError at runtime instead of actually overwriting memory 
 unless bounds checking is turned off.
No, it doesn't. It's a complete, runnable example. You can try it at home. It does overwrite `foo` and `bar`. It does not throw a RangeError.
You're right - I should have run it first.
 Yes, you can use  trusted to use Mallocator safely. And your 
 code (probably) does that. But the allocator in my example 
 isn't Mallocator, it's UnsafeAllocator. Your code doesn't use 
 that one safely.
No, and I guess it can't. I'm trying to figure out what the implications are. Can Vector only be safe for Mallocator? Is it possible to write a safe Vector at all without having to force the allocator to be safe?
 In this thread, you're the author of that 3rd party library. 
 You've got the bad  trusted functions that lead to memory 
 corruption. I'm the guy who looked at it, noticed the problem, 
 and tells you.
Thanks for bringing it up.
May 27
parent reply ag0aep6g <anonymous example.com> writes:
On 27.05.19 12:06, Atila Neves wrote:
 No, and I guess it can't. I'm trying to figure out what the implications 
 are. Can Vector only be  safe for Mallocator? Is it possible to write a 
  safe Vector at all without having to force the allocator to be  safe?
For safe allocators, Vector can be safe. For specific system allocators, like Mallocator, you can make special trusted cases in Vector. For generic system allocators, Vector cannot be safe (or trusted).
May 27
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 27 May 2019 at 10:31:10 UTC, ag0aep6g wrote:
 On 27.05.19 12:06, Atila Neves wrote:
 No, and I guess it can't. I'm trying to figure out what the 
 implications are. Can Vector only be  safe for Mallocator? Is 
 it possible to write a  safe Vector at all without having to 
 force the allocator to be  safe?
For safe allocators, Vector can be safe. For specific system allocators, like Mallocator, you can make special trusted cases in Vector. For generic system allocators, Vector cannot be safe (or trusted).
It's ugly but would work. Right now I don't think I can do any better than to follow your suggestion, but I predict many beard-stroking walks for me along Lake Geneva in the near future. I'd be nice if I could detect at compile-time that it's not just Mallocator but an allocator that's built using it as well (e.g. FallBackAllocator).
May 27
parent reply ag0aep6g <anonymous example.com> writes:
On 27.05.19 15:34, Atila Neves wrote:
 It's ugly but would work. Right now I don't think I can do any better 
 than to follow your suggestion, but I predict many beard-stroking walks 
 for me along Lake Geneva in the near future.
Oh, yeah. Getting trusted right is hard. Getting it right when user-provided types are involved is extra hard, because you can't even trust fundamental operations like assignment or copying. Please note that the allocator stuff is just one of the three violations I had pointed out. You've already pushed a fix for the unsafe .stringz, but you haven't addressed unsafe copy constructors yet. And my list wasn't meant to be complete. There may be other safety holes I didn't notice.
May 27
parent reply Valeriy Fedotov <valeriy.fedotov gmail.com> writes:
On Monday, 27 May 2019 at 14:26:16 UTC, ag0aep6g wrote:
 Oh, yeah. Getting  trusted right is hard. Getting it right when 
 user-provided types are involved is extra hard, because you 
 can't even trust fundamental operations like assignment or 
 copying.
In my point of view trusted means "I use pointer-related operations correctly. Also I am using all system interfaces correctly". The code in question uses allocator interface correctly. User of this code has a contract to supply allocator that conforms to this interface. If a user supplies mallocator that is not correct, there are two possibilities: - Allocator is buggy. Nothing to do with trusted code. - Allocator do not conforms to allocator interface. User has broken the contract. Nothing to do with trusted code. I think we should keep in mind not only technical aspects of trusted and system, but this contract too.
May 29
parent ag0aep6g <anonymous example.com> writes:
On 29.05.19 18:37, Valeriy Fedotov wrote:
 In my point of view  trusted means "I use pointer-related operations 
 correctly. Also I am using all  system interfaces correctly".
trusted means: "This function is as safe as an safe function. But the compiler doesn't verify the implementation." Whether you use anything correctly only matters when it affects safety. And "being safe" means that the function doesn't exhibit undefined behavior (when called with valid inputs), and that it doesn't lead to undefined behavior in safe code that is executed later on (e.g., an trusted function must not return an invalid pointer). [...]
 If a user 
 supplies mallocator that is not correct, there are two possibilities:
 
 - Allocator is buggy. Nothing to do with  trusted code.
 - Allocator do not conforms to allocator interface. User has broken the 
 contract. Nothing to do with  trusted code.
In my example, UnsafeAllocator is correct. It's not buggy, and it conforms to the allocator interface (which doesn't require safe methods). Still the program exhibits memory corruption. So there must be something else that's wrong. The problem is that I'm setting UnsafeAllocator.instance.i to bad values which leads to out-of-bounds accesses. I do that in the safe `main`. You might say: "Don't do that then. It's your own fault if you do that. Your mistake is not the library's fault." But D's safe is supposed catch exactly that kind of mistake. The point of safe is that I can make all the mistakes in the world in safe code (except typing out " trusted") without ever seeing memory corruption.
 I think we should keep in mind not only technical aspects of  trusted 
 and  system, but this contract too.
No. A contract that can be broken in safe code cannot be relied upon in trusted code. This is fundamental to D's safety system.
May 29
prev sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 24 May 2019 at 11:41:12 UTC, Atila Neves wrote:
 I'd been holding off on announcing this until DIP1008 actually 
 got implemented, and now it has:

 https://code.dlang.org/packages/nogc

 This dub package has a  nogc version of `std.conv.text` (which 
 probably isn't as good yet) that, instead of returning a 
 `string` returns an `automem.vector.Vector` of char. This 
 handles managing memory allocation for the exception message 
 itself in `NoGcException`, which does what it says on the tin. 
 Confused? Here's some code:
Ok, so exceptions don't rely on the GC anymore. That's super cool. However, they are still classes. So does that mean they also need RTTI (i.e. TypeInfo)? BetterC builds and some of use trying to use D in a pay-as-you-go fashion intentionally eliminate RTTI from the runtime. Is there any way we can take this a bit further to no longer require RTTI? Do exceptions even necessarily need to be classes? Thanks, Mike
May 24