www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - D Article: Memory Safety

reply Jakob Ovrum <jakobovrum gmail.com> writes:
The article aims to explain how to use  safe,  system and 
importantly,  trusted, including all the hairy details of 
templates.

https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

Any and all feedback appreciated.
Jan 20
next sibling parent default0 <schneider.cedric gmx.de> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:
 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.
Nice article! Feeling like I have a much better grasp on the whole attribute system now that I read it, thanks! :-)
Jan 20
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:
 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.
I like the description of trusted and template inference. Template inference, in particular, was not something that was obvious to me when first reading about D. I'm not sure how clear you make it that you can still mark templates safe and what have you (you seem to just say don't make templates trusted). I wasn't aware of the point that " trusted nested functions in templated functions do not have to have a memory safe interface as long as all calls to the function are memory safe". Interesting.
Jan 20
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Wednesday, 20 January 2016 at 15:28:05 UTC, jmh530 wrote:
 I like the description of  trusted and template inference. 
 Template inference, in particular, was not something that was 
 obvious to me when first reading about D. I'm not sure how 
 clear you make it that you can still mark templates  safe and 
 what have you (you seem to just say don't make templates 
  trusted).
Templated functions can still be explicitly annotated with attributes, which disables inference for those attributes. This is often a good idea even for templated functions when template arguments do not inject code, so that every instantiation has the same, known set of attributes. Attribute inference can handle it, but explicit annotations provide documentation value. I might incorporate this into the article, but I'm wary of it losing focus.
 I wasn't aware of the point that " trusted nested functions in 
 templated functions do not have to have a memory safe interface 
 as long as all calls to the function are memory safe". 
 Interesting.
It is a necessary evil to propagate attributes correctly. Don't use it when you don't have to.
Jan 20
parent "H. S. Teoh via Digitalmars-d-announce" writes:
On Thu, Jan 21, 2016 at 04:38:08AM +0000, Jakob Ovrum via
Digitalmars-d-announce wrote:
 On Wednesday, 20 January 2016 at 15:28:05 UTC, jmh530 wrote:
I like the description of  trusted and template inference. Template
inference, in particular, was not something that was obvious to me
when first reading about D. I'm not sure how clear you make it that
you can still mark templates  safe and what have you (you seem to
just say don't make templates  trusted).
Templated functions can still be explicitly annotated with attributes, which disables inference for those attributes. This is often a good idea even for templated functions when template arguments do not inject code, so that every instantiation has the same, known set of attributes. Attribute inference can handle it, but explicit annotations provide documentation value. I might incorporate this into the article, but I'm wary of it losing focus.
A common idiom used in Phobos related to this is to use an attributed (pure, nothrow, nogc, etc.) unittest that instantiates the template in question, to ensure that it does not accidentally become non-pure, throwing, etc. (the unittest will stop compiling if so). This way, we ensure that the template itself doesn't contain any impure / throwing / system code, even though you can instantiate it with template parameters that may be impure, throwing, system, etc..
I wasn't aware of the point that " trusted nested functions in
templated functions do not have to have a memory safe interface as
long as all calls to the function are memory safe". Interesting.
It is a necessary evil to propagate attributes correctly. Don't use it when you don't have to.
Wouldn't it be better to refactor the code to separate out the part that needs to be trusted into a separate place, with a safe API? Or are there cases for which this is impossible? T -- The early bird gets the worm. Moral: ewww...
Jan 20
prev sibling next sibling parent reply Dicebot <public dicebot.lv> writes:
`auto p = ()  trusted { return &t; } ();`

Huh, I thought Andrei was opposed to this idiom? Is it now 
considered reserved for templates or something has changed?
Jan 20
parent reply "H. S. Teoh via Digitalmars-d-announce" writes:
On Wed, Jan 20, 2016 at 07:25:43PM +0000, Dicebot via Digitalmars-d-announce
wrote:
 `auto p = ()  trusted { return &t; } ();`
 
 Huh, I thought Andrei was opposed to this idiom? Is it now considered
 reserved for templates or something has changed?
Yeah, I thought this was exactly the case where some of us Phobos contributors got lambasted by Andrei and Walter for abusing trusted. The thing is, this is too easy to abuse, and too prone to careless mistakes with nasty consequences. Suppose you have a template function: auto func(T)(T t) { auto p = () trusted { return &t; } (); ... p.bar(); ... } The problem is, the separation between p and p.bar() can be very large in a complicated function, and during maintenance, somebody accidentally introduces unsafe operations on p in the function body without realizing that it came from a function marked trusted. However, the compiler won't catch this, because of the trusted annotation. Any exception to the strict usage of trusted to me smells like a time bomb waiting to explode. It may not be today or tomorrow, but sooner or later somebody is going to slip up and the compiler won't help you. It's bad enough that every single change to a trusted function must be vetted to ensure actual safety; now we have to also vet any modification to any function that contains trusted anonymous functions? In a large template function, it's too easy to miss these trusted sub-functions, because if the code change is far away enough, the trusted annotation won't even show up in the diff. So reviewers may not even realize it's a change that may have broken trusted. I'm pretty sure somebody brought up a case where such a hack was actually necessary... but I'd still tread very, very carefully before recommending such a thing to the general D coder. T -- The best compiler is between your ears. -- Michael Abrash
Jan 20
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Wednesday, 20 January 2016 at 19:55:45 UTC, H. S. Teoh wrote:
 On Wed, Jan 20, 2016 at 07:25:43PM +0000, Dicebot via 
 Digitalmars-d-announce wrote:
 `auto p = ()  trusted { return &t; } ();`
 
 Huh, I thought Andrei was opposed to this idiom? Is it now 
 considered reserved for templates or something has changed?
Yeah, I thought this was exactly the case where some of us Phobos contributors got lambasted by Andrei and Walter for abusing trusted.
That was for non-templated functions where this approach makes no sense. Indeed it is counterproductive, because trusted on the whole function is a better indication of what needs to be reviewed for memory safety (the whole function!).
 Any exception to the strict usage of  trusted to me smells like 
 a time bomb waiting to explode. It may not be today or 
 tomorrow, but sooner or later somebody is going to slip up and 
 the compiler won't help you. It's bad enough that every single 
 change to a  trusted function must be vetted to ensure actual 
 safety; now we have to also vet any modification to any 
 function that contains  trusted anonymous functions? In a large 
 template function, it's too easy to miss these  trusted 
 sub-functions, because if the code change is far away enough, 
 the  trusted annotation won't even show up in the diff. So 
 reviewers may not even realize it's a change that may have 
 broken  trusted.
It is the only way to solve this problem.
Jan 20
parent reply Dicebot <public dicebot.lv> writes:
On Thursday, 21 January 2016 at 04:31:25 UTC, Jakob Ovrum wrote:
 That was for non-templated functions where this approach makes 
 no sense. Indeed it is counterproductive, because  trusted on 
 the whole function is a better indication of what needs to be 
 reviewed for memory safety (the whole function!).
Thanks! I got confused because your used example actually leaves safe hole with this specific usage of trusted : void foo(T)(T t) { auto p = () trusted { return &t; } (); p.bar(); } struct S { int x; } S* global; void bar (S* ptr) safe { global = ptr; } void main () safe { foo(S.init); global.x = 42; // oops, writing to some random stack memory } I'd suggest at the very least to add a comment before "p.bar();" saying "Must not escape 'p' pointer or safe-ty will be compromised".
Jan 21
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 21 January 2016 at 13:39:48 UTC, Dicebot wrote:
 I'd suggest at the very least to add a comment before 
 "p.bar();" saying "Must not escape 'p' pointer or  safe-ty will 
 be compromised".
I thought about this case, but it relies on UFCS which is controlled by the callee. The caller can't inject that call if the callee is careful with its imports. For member functions, the this reference is `ref` and its address cannot be taken in safe code.
Jan 21
parent reply Dicebot <public dicebot.lv> writes:
On Thursday, 21 January 2016 at 13:42:13 UTC, Jakob Ovrum wrote:
 On Thursday, 21 January 2016 at 13:39:48 UTC, Dicebot wrote:
 I'd suggest at the very least to add a comment before 
 "p.bar();" saying "Must not escape 'p' pointer or  safe-ty 
 will be compromised".
I thought about this case, but it relies on UFCS which is controlled by the callee. The caller can't inject that call if the callee is careful with its imports. For member functions, the this reference is `ref` and its address cannot be taken in safe code.
Reasonable, but the UFCS call can result from some other function defined in same module (Phobos modules are not small at all). Even small unlikely violation can completely destroy benefits of safe so in my opinion one can't be overly cautious when documenting stuff that requires verification.
Jan 21
parent Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 21 January 2016 at 13:52:57 UTC, Dicebot wrote:
 Reasonable, but the UFCS call can result from some other 
 function defined in same module (Phobos modules are not small 
 at all). Even small unlikely violation can completely destroy 
 benefits of  safe so in my opinion one can't be overly cautious 
 when documenting stuff that requires verification.
I agree. Do you think it's worth mentioning UFCS functions in the article?
Jan 21
prev sibling next sibling parent reply Jon D <jond noreply.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:
 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.
Nice article. I got a much better understanding reading it. A small thing - I immediately tried adding safe to one of my programs and hit an issue related to the first example in the article: void main() safe { import std.stdio; writeln("hello, world"); } This is passes the safe constraint, but 'stdout.writeln()' and 'stderr.writeln()' do not. (My program uses stderr.) stderr/stdout/stdin are __gshared and can't be referenced by safe code. The module level version of writeln, etc., access a trusted version of stdout to avoid this. I don't have a specific suggestion for addressing this in the article. It's nicely written and delving into this may take away from the key points. But, I wanted to point this out in case others hit it. (Note: This related to a recent thread on the learn forum: http://forum.dlang.org/thread/vkihzrwomhiwdzqelrxa forum.dlang.org)
Jan 20
parent Jakob Ovrum <jakobovrum gmail.com> writes:
On Wednesday, 20 January 2016 at 20:28:03 UTC, Jon D wrote:
 This is passes the  safe constraint, but 'stdout.writeln()' and 
 'stderr.writeln()' do not. (My program uses stderr.) 
 stderr/stdout/stdin are __gshared and can't be referenced by 
 safe code. The module level version of writeln, etc., access a 
 trusted version of stdout to avoid this.
Yeah, the standard library still has a ways to go even with safe. I always imagined that the standard pipes should use shared as opposed to __gshared. I don't think the current implementation is thread-safe, but I don't know how this affects in memory safety in this case.
Jan 20
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:
 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.
Good work. Someone has to re-edit-it if not yet reeddited. Altgough one thing, attributes are not the easy part of D. I've recently encountered a case were in the library attributes were allright, test OK, and then suddently when I've started to use the library in a real life context I had to remove them from the library... safe was unsustainable. Dealing with attributes is the hardest part of D IMO. No one is forced to btw, there are plenty of other cool things in D but to follow the D safety is hard... congrats nice article.
Jan 20
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Thursday, 21 January 2016 at 04:59:01 UTC, Basile B. wrote:
 On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum 
 wrote:
 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.
Good work. Someone has to re-edit-it if not yet reeddited. Altgough one thing, attributes are not the easy part of D. I've recently encountered a case were in the library attributes were allright, test OK, and then suddently when I've started to use the library in a real life context I had to remove them from the library... safe was unsustainable. Dealing with attributes is the hardest part of D IMO. No one is forced to btw, there are plenty of other cool things in D but to follow the D safety is hard... congrats nice article.
I mean ' safe' at too low level is a handicap. It's like 'const'. They are hard to use, mostly because of transitivness. These attributes are never a noop.
Jan 20
parent "H. S. Teoh via Digitalmars-d-announce" writes:
On Thu, Jan 21, 2016 at 05:09:48AM +0000, Basile B. via Digitalmars-d-announce
wrote:
[...]
 I mean ' safe' at too low level is a handicap. It's like 'const'. They
 are hard to use, mostly because of transitivness. These attributes are
 never a noop.
Transitivity also makes const really painful to use in a widespread way. I've tried writing const-correct code before too, but gave up because it quickly became too unwieldy to work with. I started spending more time hunting down missing const attributes than actually writing useful code, so I decided it was time to give up. Generally, though, const is still useful in lower-level code (i.e., near the leaf nodes of your function call tree), to prevent silly mistakes. Knowing how to use const is also helpful in utility functions that need to accept both immutable and mutable, etc.. Just like with (the current state of) safe, though, pervasive use of const is still too onerous currently. Transitivity really makes it painful, especially when important chunks of Phobos still isn't fully const-correct (or at least const-compatible) yet. A lot of progress has been made, but, const being transitive, all it takes is for one small Phobos function to be non-const when it should be const, and your entire call tree can no longer be const. Encounter this a handful of times, and it's hard not to just throw in the towel instead of spending all of your time working around const issues rather than writing useful code. (Recently I'm slowly moving towards writing *all* my code as template functions, and letting the compiler do the tedious work of attributing my code instead of typing them out myself. My secret wish is that one day, the compiler's attribute inference will be good enough that I could just slap one or two const's (or safe, etc.) on top of my modules and everything will Just Work.) T -- Only boring people get bored. -- JM
Jan 20
prev sibling parent "H. S. Teoh via Digitalmars-d-announce" writes:
On Thu, Jan 21, 2016 at 04:59:01AM +0000, Basile B. via Digitalmars-d-announce
wrote:
[...]
 Altgough one thing, attributes are not the easy part of D. I've
 recently encountered a case were in the library attributes were
 allright, test OK, and then suddently when I've started to use the
 library in a real life context I had to remove them from the
 library... safe was unsustainable.
Phobos/druntime still has some ways to go before using it from safe code will be painless. Some pretty fundamental functionality still isn't safe (mainly some stuff in object.di that basically interacts with too many other things that marking one thing as safe will percolate throughout pretty much everything, breaking a whole bunch of stuff at once). I once tried writing a safe program, and it didn't take very long before I threw that idea out the window. Once main() is safe, you're so straitjacketed that you basically can't write anything too much more complex than Hello World. (Well, you *could* just slap trusted on whatever it is that's holding you back, but then that breaks the promise of safe, which defeats the purpose of the entire exercise.) There's also still a good number of safe-related bugs on Bugzilla, several of which involve built-in language constructs that break safe-ty outright. Things have improved a bit since I last checked, but it seems to me that safe is still not quite ready to live up to its promise just yet. Maybe in a few more years' time...
 Dealing with attributes is the hardest part of D IMO.  No one is
 forced to btw, there are plenty of other cool things in D but to
 follow the D safety is hard...
[...] I think Walter has mentioned before that attribute inference is the way to go, and I agree. Once you start writing carefully-attributed code, you'll quickly find that your declarations become painfully verbose, which is never a good sign (it encourages people not to use attributes). However, attribute inference on templates and auto functions (proposed last year, don't know if it's implemented yet) alleviates a lot of the verbosity. Hopefully the scope of attribute inference will increase until it makes attribute use more widespread in your everyday D code. T -- MS Windows: 64-bit rehash of 32-bit extensions and a graphical shell for a 16-bit patch to an 8-bit operating system originally coded for a 4-bit microprocessor, written by a 2-bit company that can't stand 1-bit of competition.
Jan 20
prev sibling next sibling parent reply rsw0x <anonymous anonymous.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:
 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.
my experience with safe: okay, I'll just use safe here... and nothing else in third party libraries/half of phobos is safe friendly so I guess I'll wrap it in trusted oh fuck it
Jan 20
parent Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 21 January 2016 at 06:20:01 UTC, rsw0x wrote:
 okay, I'll just use  safe here... and nothing else in third 
 party libraries/half of phobos is  safe friendly so I guess 
 I'll wrap it in  trusted oh fuck it
Yeah, using trusted like that is counterproductive. Just use system or improve the dependencies.
Jan 20
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:
 The article aims to explain how to use  safe
Um, no, the article doesn't explain how to use safe, it shows patterns that can be used to write safe code. The target audience must already understand safety.
Jan 21
prev sibling next sibling parent Jakob Ovrum <jakobovrum gmail.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:
 snip
Thanks for all the feedback. I've pushed a revision with further changes, most of it based on the feedback in this thread. https://github.com/JakobOvrum/jakobovrum.github.io/commit/07c270567097f6cae5d9b95c88bd4d6c8124498c (I'll try to remember not to force push over this commit and break the link, but if it is broken in the future, sorry, I probably slipped up and forgot.)
Jan 21
prev sibling next sibling parent Joakim <dlang joakim.fea.st> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:
 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.
Nicely written, enjoyed reading it. And the visual presentation is excellent, I guess because it's based on github's code presentation libraries. Really a nice example of how you can find blog posts online that are an order of magnitude better than any book you've ever read, particularly the visuals. Is it on reddit? The dlang.org redesign, Walter's talk, and a couple other D topics did well on there this week, I think this would also.
Jan 21
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/20/2016 09:04 AM, Jakob Ovrum wrote:
 The article aims to explain how to use  safe,  system and importantly,
  trusted, including all the hairy details of templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.
Good work, thanks! Has this been reddited yet? -- Andrei
Jan 21
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 21 January 2016 at 17:39:11 UTC, Andrei Alexandrescu 
wrote:
 Good work, thanks! Has this been reddited yet? -- Andrei
I don't think so. Personally I don't think I have a reddit account, but people are more than welcome to post it wherever they like :)
Jan 21
parent reply Brad Anderson <eco gnuk.net> writes:
On Thursday, 21 January 2016 at 17:42:02 UTC, Jakob Ovrum wrote:
 On Thursday, 21 January 2016 at 17:39:11 UTC, Andrei 
 Alexandrescu wrote:
 Good work, thanks! Has this been reddited yet? -- Andrei
I don't think so. Personally I don't think I have a reddit account, but people are more than welcome to post it wherever they like :)
Someone has submitted it (about a half hour ago): https://www.reddit.com/r/programming/comments/420yhi/memory_safety_in_d/
Jan 21
parent Brad Anderson <eco gnuk.net> writes:
On Thursday, 21 January 2016 at 17:56:19 UTC, Brad Anderson wrote:
 On Thursday, 21 January 2016 at 17:42:02 UTC, Jakob Ovrum wrote:
 On Thursday, 21 January 2016 at 17:39:11 UTC, Andrei 
 Alexandrescu wrote:
 Good work, thanks! Has this been reddited yet? -- Andrei
I don't think so. Personally I don't think I have a reddit account, but people are more than welcome to post it wherever they like :)
Someone has submitted it (about a half hour ago): https://www.reddit.com/r/programming/comments/420yhi/memory_safety_in_d/
I've submitted it to Hacker News.
Jan 21