www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - An interesting consequence of safety requirements

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Something just dawned on me: in safe mode, struct static member 
functions will be preferred to struct non-static member functions.

Why?

Consider:

struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
}

This code can't make it in safe mode because it takes the address of 
this. In general, the compiler must assume that you might have created a 
List object on the stack, e.g.:

List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
}

Now even if there is no ostensible local address taking, the code is in 
error because it has escaped the address of a local.

So prepend() cannot be compiled. The way to make it compile in safe mode is:


struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
}

Now the code compiles and is actually safe because it is impossible to 
pass the address of a local into prepend.

So get ready to use static a lot more ;o).


Andrei
Nov 04 2009
next sibling parent reply Leandro Lucarella <llucax gmail.com> writes:
Andrei Alexandrescu, el  4 de noviembre a las 11:24 me escribiste:
 Something just dawned on me: in safe mode, struct static member
 functions will be preferred to struct non-static member functions.
 
 Why?
 
 Consider:
 
 struct List(T) {
     T payload;
     List * next;
 
     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }
 
 This code can't make it in safe mode because it takes the address of
 this. In general, the compiler must assume that you might have
 created a List object on the stack, e.g.:
 
 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }
 
 Now even if there is no ostensible local address taking, the code is
 in error because it has escaped the address of a local.
 
 So prepend() cannot be compiled. The way to make it compile in safe mode is:
 
 
 struct List(T) {
     T payload;
     List * next;
 
     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }
 
 Now the code compiles and is actually safe because it is impossible
 to pass the address of a local into prepend.
 
 So get ready to use static a lot more ;o).

Or maybe th compiler should rewrite local.prepend(lst); as: auto this_ = &local; f(*_this); Then the compiler can detect when taking the address of local is not legal as it will do if you write the function as static. -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- Sometimes I think the sure sign that life exists elsewhere in the universe Is that that none of them tried to contact us
Nov 04 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Leandro Lucarella wrote:
 Andrei Alexandrescu, el  4 de noviembre a las 11:24 me escribiste:
 Something just dawned on me: in safe mode, struct static member
 functions will be preferred to struct non-static member functions.

 Why?

 Consider:

 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }

 This code can't make it in safe mode because it takes the address of
 this. In general, the compiler must assume that you might have
 created a List object on the stack, e.g.:

 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }

 Now even if there is no ostensible local address taking, the code is
 in error because it has escaped the address of a local.

 So prepend() cannot be compiled. The way to make it compile in safe mode is:


 struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }

 Now the code compiles and is actually safe because it is impossible
 to pass the address of a local into prepend.

 So get ready to use static a lot more ;o).

Or maybe th compiler should rewrite local.prepend(lst); as: auto this_ = &local; f(*_this); Then the compiler can detect when taking the address of local is not legal as it will do if you write the function as static.

Yah, but how would the compiler decide (in a separate compilation approach) that some functions are actually fine? // inside List(T) T getPayload() { return payload; } Andrei
Nov 04 2009
prev sibling next sibling parent Leandro Lucarella <llucax gmail.com> writes:
Leandro Lucarella, el  4 de noviembre a las 16:06 me escribiste:
 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }
 So get ready to use static a lot more ;o).

Or maybe th compiler should rewrite local.prepend(lst); as: auto this_ = &local; f(*_this);

Damn! I mean: auto this_ = &local; List.prepend(*_this, lst); -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- You should've seen her face. It was the exact same look my father gave me when I told him I wanted to be a ventriloquist. -- George Constanza
Nov 04 2009
prev sibling next sibling parent reply grauzone <none example.net> writes:
Andrei Alexandrescu wrote:
 Something just dawned on me: in safe mode, struct static member 
 functions will be preferred to struct non-static member functions.
 
 Why?
 
 Consider:
 
 struct List(T) {
     T payload;
     List * next;
 
     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }
 
 This code can't make it in safe mode because it takes the address of 
 this. In general, the compiler must assume that you might have created a 
 List object on the stack, e.g.:
 
 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }
 
 Now even if there is no ostensible local address taking, the code is in 
 error because it has escaped the address of a local.
 
 So prepend() cannot be compiled. The way to make it compile in safe mode 
 is:
 
 
 struct List(T) {
     T payload;
     List * next;
 
     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }
 
 Now the code compiles and is actually safe because it is impossible to 
 pass the address of a local into prepend.

Maybe I don't get it, but how to you call prepend() from safe code? Also, does anybody really care about SafeD, or would it be better if we had some sort of valgrind for D? Maybe this is one of those features which first sounded nice, but then it turned out it's better to drop them.
 So get ready to use static a lot more ;o).
 
 
 Andrei

Nov 04 2009
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
grauzone wrote:
 Andrei Alexandrescu wrote:
 Something just dawned on me: in safe mode, struct static member 
 functions will be preferred to struct non-static member functions.

 Why?

 Consider:

 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }

 This code can't make it in safe mode because it takes the address of 
 this. In general, the compiler must assume that you might have created 
 a List object on the stack, e.g.:

 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }

 Now even if there is no ostensible local address taking, the code is 
 in error because it has escaped the address of a local.

 So prepend() cannot be compiled. The way to make it compile in safe 
 mode is:


 struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }

 Now the code compiles and is actually safe because it is impossible to 
 pass the address of a local into prepend.

Maybe I don't get it, but how to you call prepend() from safe code?

module(safe) wyda; void main() { auto lst1 = new List; auto lst2 = new List; List.prepend(lst1, lst2) }
 Also, does anybody really care about SafeD, or would it be better if we 
 had some sort of valgrind for D? Maybe this is one of those features 
 which first sounded nice, but then it turned out it's better to drop them.

I'm not sure, but clearly soundness is an important concern in contemporary languages. Andrei
Nov 04 2009
next sibling parent David Gileadi <foo bar.com> writes:
Andrei Alexandrescu wrote:
 grauzone wrote:
 Maybe I don't get it, but how to you call prepend() from safe code?

module(safe) wyda; void main() { auto lst1 = new List; auto lst2 = new List; List.prepend(lst1, lst2) }

This seems to be another case where automatic function rewriting would be nice, a la D's "Functions as Array Properties": void main() { auto lst1 = new List; auto lst2 = new List; // rewritten by compiler to List.prepend(lst1, lst2): lst1.prepend(lst2); }
Nov 04 2009
prev sibling parent reply grauzone <none example.net> writes:
Andrei Alexandrescu wrote:
 grauzone wrote:
 Andrei Alexandrescu wrote:
 Something just dawned on me: in safe mode, struct static member 
 functions will be preferred to struct non-static member functions.

 Why?

 Consider:

 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }

 This code can't make it in safe mode because it takes the address of 
 this. In general, the compiler must assume that you might have 
 created a List object on the stack, e.g.:

 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }

 Now even if there is no ostensible local address taking, the code is 
 in error because it has escaped the address of a local.

 So prepend() cannot be compiled. The way to make it compile in safe 
 mode is:


 struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }

 Now the code compiles and is actually safe because it is impossible 
 to pass the address of a local into prepend.

Maybe I don't get it, but how to you call prepend() from safe code?

module(safe) wyda; void main() { auto lst1 = new List; auto lst2 = new List; List.prepend(lst1, lst2) }

I see... in this case, I'd say the user should be forced to allocate List on the heap by making List a class. Of course, you could say that it should be useable in other, unsafe, contexts too (maybe you want to construct a list of the stack); but then you could use... InSitu!(T), or what was it called?
 Also, does anybody really care about SafeD, or would it be better if 
 we had some sort of valgrind for D? Maybe this is one of those 
 features which first sounded nice, but then it turned out it's better 
 to drop them.

I'm not sure, but clearly soundness is an important concern in contemporary languages. Andrei

Nov 04 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
grauzone wrote:
 Andrei Alexandrescu wrote:
 grauzone wrote:
 Andrei Alexandrescu wrote:
 Something just dawned on me: in safe mode, struct static member 
 functions will be preferred to struct non-static member functions.

 Why?

 Consider:

 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }

 This code can't make it in safe mode because it takes the address of 
 this. In general, the compiler must assume that you might have 
 created a List object on the stack, e.g.:

 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }

 Now even if there is no ostensible local address taking, the code is 
 in error because it has escaped the address of a local.

 So prepend() cannot be compiled. The way to make it compile in safe 
 mode is:


 struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }

 Now the code compiles and is actually safe because it is impossible 
 to pass the address of a local into prepend.

Maybe I don't get it, but how to you call prepend() from safe code?

module(safe) wyda; void main() { auto lst1 = new List; auto lst2 = new List; List.prepend(lst1, lst2) }

I see... in this case, I'd say the user should be forced to allocate List on the heap by making List a class.

Well we'd still like to benefit from structs and pointers in safe mode. The point is to tighten the screws just as much as needed to achieve interesting properties, but not (much) tighter. Andrei
Nov 04 2009
prev sibling next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from grauzone (none example.net)'s article
 Andrei Alexandrescu wrote:
 Also, does anybody really care about SafeD, or would it be better if we
 had some sort of valgrind for D? Maybe this is one of those features
 which first sounded nice, but then it turned out it's better to drop them.

I'm starting to get that feeling. It seems like D was just not meant to be a fully safe language. It was built from the ground up to be a better C++, i.e. a close to the metal language that assumes the programmer knows what he/she is doing, albeit one with enough high level features that it doesn't require attention to the irrelevant in day-to-day programming. Shoe-horning D (originally designed as close-to-the-metal/programmer-knows-best) into a safe Java-like language is going to work about as well as shoe-horning C (originally a very low-level portable assembler) into C++ (a language that tries to be high-level but is too hobbled by C compatibility to actually achieve it). Example (Simple program that has no chance of working in SafeD): import std.getopt; void main(string[] args) { uint foo; getopt(args, "someParam", &foo); // Not allowed in SafeD. } Personally, if an example like that won't even compile, I refuse to ever use SafeD for any project I do because it is simply too rigid. On the other hand, if it does compile, SafeD provides no guarantees and is next to useless.
Nov 04 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
dsimcha wrote:
 == Quote from grauzone (none example.net)'s article
 Andrei Alexandrescu wrote:
 Also, does anybody really care about SafeD, or would it be better if we
 had some sort of valgrind for D? Maybe this is one of those features
 which first sounded nice, but then it turned out it's better to drop them.

I'm starting to get that feeling. It seems like D was just not meant to be a fully safe language. It was built from the ground up to be a better C++, i.e. a close to the metal language that assumes the programmer knows what he/she is doing, albeit one with enough high level features that it doesn't require attention to the irrelevant in day-to-day programming. Shoe-horning D (originally designed as close-to-the-metal/programmer-knows-best) into a safe Java-like language is going to work about as well as shoe-horning C (originally a very low-level portable assembler) into C++ (a language that tries to be high-level but is too hobbled by C compatibility to actually achieve it). Example (Simple program that has no chance of working in SafeD): import std.getopt; void main(string[] args) { uint foo; getopt(args, "someParam", &foo); // Not allowed in SafeD. } Personally, if an example like that won't even compile, I refuse to ever use SafeD for any project I do because it is simply too rigid. On the other hand, if it does compile, SafeD provides no guarantees and is next to useless.

getopt takes pointers just because variadic references aren't available. It's not a matter of principles. I agree that a useful module such as getopt should work in safe mode. Andrei
Nov 04 2009
prev sibling next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
grauzone wrote:
 Also, does anybody really care about SafeD, or would it be better if we 
 had some sort of valgrind for D? Maybe this is one of those features 
 which first sounded nice, but then it turned out it's better to drop them.

valgrind is a runtime thing, so it depends on having a reasonably complete test suite, and cannot say anything about code that is not executed by the test suite. Even if the test suite has 100% coverage of all code paths, that is still no guarantee of not having undefined behavior. SafeD offers a compile time guarantee.
Nov 04 2009
prev sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2009-11-04 14:15:47 -0500, grauzone <none example.net> said:

 Also, does anybody really care about SafeD, or would it be better if we 
 had some sort of valgrind for D? Maybe this is one of those features 
 which first sounded nice, but then it turned out it's better to drop 
 them.

I'm interested in SafeD a lot since it guards against buffer overruns and memory corruption errors, which represents a big slice of the most dangerous security risks. Sure it comes with small performance drawbacks (array bound checks, forced dynamic allocation in some cases). But that shouldn't matter as you can move performance-critical code to unsafe/trusted modules as an optimization (hopefully with more security checkups on these), or just disable SafeD altogether if that really makes a difference. But most of my code isn't performance critical and thus most of my code should be in SafeD. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Nov 04 2009
prev sibling parent reply Chad J <chadjoan __spam.is.bad__gmail.com> writes:
Andrei Alexandrescu wrote:
 Something just dawned on me: in safe mode, struct static member
 functions will be preferred to struct non-static member functions.
 
 Why?
 
 Consider:
 
 struct List(T) {
     T payload;
     List * next;
 
     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }
 
 This code can't make it in safe mode because it takes the address of
 this. In general, the compiler must assume that you might have created a
 List object on the stack, e.g.:
 
 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }
 
 Now even if there is no ostensible local address taking, the code is in
 error because it has escaped the address of a local.
 
 So prepend() cannot be compiled. The way to make it compile in safe mode
 is:
 
 
 struct List(T) {
     T payload;
     List * next;
 
     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }
 
 Now the code compiles and is actually safe because it is impossible to
 pass the address of a local into prepend.
 
 So get ready to use static a lot more ;o).
 
 
 Andrei

This looks to me like it implicates ref parameters in general.
 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }

could be written as struct List(T) { T payload; List * next; static void prepend(ref List zis, List * newNode) { newNode.next = &zis; } } It's not the same as the other static version, since zis is a ref and not a pointer. List * someFun() { List local; List * lst = new List; prepend(local,lst); // <-- dubious; local is trying to hide return lst; // <-- he got away } I think in the general case there might need to be some way to track functions that have ref parameters. Perhaps any ref parameter that has its address taken is marked as being unable to accept local variables as arguments. Additionally, it should also not accept a parent function's ref parameters as arguments, since those could be local to someone else. Then if you rewrite member functions as static versions with ref parameters as the zeroth arg, they will benefit from the analysis as well. Maybe this is too complicated. If not, hope it helps.
Nov 04 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Chad J wrote:
 Andrei Alexandrescu wrote:

 So get ready to use static a lot more ;o).


 Andrei

This looks to me like it implicates ref parameters in general.

Yah, and more precisely: ref parameters that you plan to take the address of. It all holds water: if you take a pointer parameter in SafeD, you know for sure it's dynamically-allocated because the caller could not have taken the address of a stack variable. [snip]
 Perhaps any ref parameter that has its address taken is marked as being
 unable to accept local variables as arguments.  Additionally, it should
 also not accept a parent function's ref parameters as arguments, since
 those could be local to someone else.  Then if you rewrite member
 functions as static versions with ref parameters as the zeroth arg, they
  will benefit from the analysis as well.
 
 Maybe this is too complicated.  If not, hope it helps.

Thanks. It's a reenactment of a discussion that Bartosz, Walter and I have had a few times: should the compiler collect the so-called "function summaries" during compilation and augment the signatures with additional properties, or should we require the user to annotate the signatures themselves? Collecting function summaries is a classic in many program analysis, but is difficult to scale and to combine with separate compilation (usually interesting summaries require collecting info about all functions and then doing a sort of fixed point iteration). So far dmd never relies on collecting a function summary, but that may change in the future. Andrei
Nov 05 2009
parent Brad Roberts <braddr bellevue.puremagic.com> writes:
On Thu, 5 Nov 2009, Andrei Alexandrescu wrote:

 Thanks. It's a reenactment of a discussion that Bartosz, Walter and I have had
 a few times: should the compiler collect the so-called "function summaries"
 during compilation and augment the signatures with additional properties, or
 should we require the user to annotate the signatures themselves?
 
 Collecting function summaries is a classic in many program analysis, but is
 difficult to scale and to combine with separate compilation (usually
 interesting summaries require collecting info about all functions and then
 doing a sort of fixed point iteration).
 
 So far dmd never relies on collecting a function summary, but that may change
 in the future.
 
 
 Andrei

The key difference for me has always been that user specified traits (be they annotations, keywords, whatever) is that it's defining the intended contract. Anything inferred is useful only so far in that it might allow usage where not explicitly intended. For example, imagine a function declared to take a function pointer for which the function must be pure. That explicitly allows any annotated function to be used (that matches the rest of the signature, obviously), but might also allow the use of any detected pure function. The risk being that the detected as pure function might change at some future date -- since it's signature didn't require enforcement of that purity -- and break the call site. From what I can see, there's three alternatives: 1) strict policy -- no inference done at all 2) strict policy -- inference allowed to influence optimization, but not change semantics 3) inference allowed at the semantic layer, potentially allowing future issues. I'm personally most in favor of #2. We're currently in the land of #1. Later, Brad
Nov 05 2009