www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - How mutable is immutable?

reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
So, I'm a function `f`, I have an `immutable(type)[]` argument and I 
want to store it for my friend `g` in an TLS variable `v`:
---
string v;
debug string sure;

void f(string s) { v = s; debug sure = s.idup; }
void g() { assert(v == sure); }
---
I also store a copy of `s` into `sure` for my friend to ensure immutable 
date hasn't been mutated.
Can my friend's assertion ever fail without breaking a type-system?
Sure. Just consider this:
---
void main() {
     auto s = "abba".idup;
     f(s);
     delete s;
     g();
}
---
Is it by-design? Looks like deleting immutable (and const because of 
implicit conversion) data should be prohibited.
OK. Let `delete` be fixed. Can we still fail?
---
void h() {
     immutable(char)[4] s = "abba";
     f(s);
}
void main() {
     h();
     g();
}
---
Damn! So, what can we do with it? Not sure, but I have a proposal.

Fix it in language:
     * disallow `delete` of const/immutable data
     * disallow immutable data on the stack

This makes data really immutable if I don't miss something. Anyway, I 
want `immutable` qualified data to be immutable without breaking a 
type-system (if one do it, its his own responsibility), so some changes 
should be made (IMHO).
Jan 01 2012
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
 So, I'm a function `f`, I have an `immutable(type)[]` argument and I
 want to store it for my friend `g` in an TLS variable `v`:
 ---
 string v;
 debug string sure;

 void f(string s) { v = s; debug sure = s.idup; }
 void g() { assert(v == sure); }
 ---
 I also store a copy of `s` into `sure` for my friend to ensure immutable
 date hasn't been mutated.
 Can my friend's assertion ever fail without breaking a type-system?
 Sure. Just consider this:
 ---
 void main() {
 auto s = "abba".idup;
 f(s);
 delete s;
 g();
 }
 ---
 Is it by-design? Looks like deleting immutable (and const because of
 implicit conversion) data should be prohibited.
 OK. Let `delete` be fixed. Can we still fail?
 ---
 void h() {
 immutable(char)[4] s = "abba";
 f(s);
 }
 void main() {
 h();
 g();
 }
 ---
 Damn! So, what can we do with it? Not sure, but I have a proposal.

 Fix it in language:
 * disallow `delete` of const/immutable data
 * disallow immutable data on the stack

 This makes data really immutable if I don't miss something. Anyway, I
 want `immutable` qualified data to be immutable without breaking a
 type-system (if one do it, its his own responsibility), so some changes
 should be made (IMHO).

You are using unsafe language features to break the type system. That is not the fault of the type system. ' safe:' at the top of the program should stop both examples from working, it is a bug that it does not.
Jan 01 2012
next sibling parent reply Don Clugston <dac nospam.com> writes:
On 01/01/12 13:50, Timon Gehr wrote:
 On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
 So, I'm a function `f`, I have an `immutable(type)[]` argument and I
 want to store it for my friend `g` in an TLS variable `v`:
 ---
 string v;
 debug string sure;

 void f(string s) { v = s; debug sure = s.idup; }
 void g() { assert(v == sure); }
 ---
 I also store a copy of `s` into `sure` for my friend to ensure immutable
 date hasn't been mutated.
 Can my friend's assertion ever fail without breaking a type-system?
 Sure. Just consider this:
 ---
 void main() {
 auto s = "abba".idup;
 f(s);
 delete s;
 g();
 }
 ---
 Is it by-design? Looks like deleting immutable (and const because of
 implicit conversion) data should be prohibited.
 OK. Let `delete` be fixed. Can we still fail?
 ---
 void h() {
 immutable(char)[4] s = "abba";
 f(s);
 }
 void main() {
 h();
 g();
 }
 ---
 Damn! So, what can we do with it? Not sure, but I have a proposal.

 Fix it in language:
 * disallow `delete` of const/immutable data
 * disallow immutable data on the stack

 This makes data really immutable if I don't miss something. Anyway, I
 want `immutable` qualified data to be immutable without breaking a
 type-system (if one do it, its his own responsibility), so some changes
 should be made (IMHO).

You are using unsafe language features to break the type system. That is not the fault of the type system. ' safe:' at the top of the program should stop both examples from working, it is a bug that it does not.

That's the point -- *which* checks are missing from safe? But I'm not sure that you're right, this looks broken to me, even without safe. What does it mean to create immutable data on the stack? The stack is intrinsically mutable! What does it mean to delete immutable data? I think it's reasonable for both of them to require a cast, even in system code.
Oct 17 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/17/2012 01:49 PM, Don Clugston wrote:
 On 01/01/12 13:50, Timon Gehr wrote:
 On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
 So, I'm a function `f`, I have an `immutable(type)[]` argument and I
 want to store it for my friend `g` in an TLS variable `v`:
 ---
 string v;
 debug string sure;

 void f(string s) { v = s; debug sure = s.idup; }
 void g() { assert(v == sure); }
 ---
 I also store a copy of `s` into `sure` for my friend to ensure immutable
 date hasn't been mutated.
 Can my friend's assertion ever fail without breaking a type-system?
 Sure. Just consider this:
 ---
 void main() {
 auto s = "abba".idup;
     f(s);
     delete s;
     g();
 }
 ---
 Is it by-design? Looks like deleting immutable (and const because of
 implicit conversion) data should be prohibited.
 OK. Let `delete` be fixed. Can we still fail?
 ---
 void h() {
     immutable(char)[4] s = "abba";
     f(s);
 }
 void main() {
     h();
     g();
 }
 ---
 Damn! So, what can we do with it? Not sure, but I have a proposal.

 Fix it in language:
 * disallow `delete` of const/immutable data
 * disallow immutable data on the stack

 This makes data really immutable if I don't miss something. Anyway, I
 want `immutable` qualified data to be immutable without breaking a
 type-system (if one do it, its his own responsibility), so some changes
 should be made (IMHO).

You are using unsafe language features to break the type system. That is not the fault of the type system. ' safe:' at the top of the program should stop both examples from working, it is a bug that it does not.

That's the point -- *which* checks are missing from safe?

Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages.
 But I'm not sure that you're right, this looks broken to me, even
 without  safe.

 What does it mean to create immutable data on the stack? The stack is
 intrinsically mutable!

So is the heap. What does it mean to garbage collect immutable data? What does it mean to allocate an 'int' on the stack?
 What does it mean to delete immutable data?

Deallocate the storage for it and make it available for reuse. Accessing it afterwards leads to arbitrary behaviour. This is the same with mutable data. As the program may behave arbitrarily in this case, it is valid behaviour to act as if immutable data changed.
 I think it's reasonable for both of them to require a cast, even in
  system code.

The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping. Rust's borrowed pointers may give some hints on how to extend 'scope' to fields of structs. As to delete, delete is as unsafe when the involved data is immutable as when it is mutable. Why require an additional cast in one case?
Oct 17 2012
parent reply Don Clugston <dac nospam.com> writes:
On 17/10/12 18:02, Timon Gehr wrote:
 On 10/17/2012 01:49 PM, Don Clugston wrote:
 On 01/01/12 13:50, Timon Gehr wrote:
 On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
 So, I'm a function `f`, I have an `immutable(type)[]` argument and I
 want to store it for my friend `g` in an TLS variable `v`:
 ---
 string v;
 debug string sure;

 void f(string s) { v = s; debug sure = s.idup; }
 void g() { assert(v == sure); }
 ---
 I also store a copy of `s` into `sure` for my friend to ensure
 immutable
 date hasn't been mutated.
 Can my friend's assertion ever fail without breaking a type-system?
 Sure. Just consider this:
 ---
 void main() {
 auto s = "abba".idup;
     f(s);
     delete s;
     g();
 }
 ---
 Is it by-design? Looks like deleting immutable (and const because of
 implicit conversion) data should be prohibited.
 OK. Let `delete` be fixed. Can we still fail?
 ---
 void h() {
     immutable(char)[4] s = "abba";
     f(s);
 }
 void main() {
     h();
     g();
 }
 ---
 Damn! So, what can we do with it? Not sure, but I have a proposal.

 Fix it in language:
 * disallow `delete` of const/immutable data
 * disallow immutable data on the stack

 This makes data really immutable if I don't miss something. Anyway, I
 want `immutable` qualified data to be immutable without breaking a
 type-system (if one do it, its his own responsibility), so some changes
 should be made (IMHO).

You are using unsafe language features to break the type system. That is not the fault of the type system. ' safe:' at the top of the program should stop both examples from working, it is a bug that it does not.

That's the point -- *which* checks are missing from safe?

Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages.

HOW do you propose to check for escaping stack data?
 But I'm not sure that you're right, this looks broken to me, even
 without  safe.

 What does it mean to create immutable data on the stack? The stack is
 intrinsically mutable!

So is the heap.

No it is not. Data on the stack *cannot* survive past the end of the function call. Data on the heap can last forever.
 What does it mean to garbage collect immutable data?

From the point of view of the application, it doesn't happen. There are no observable semantics. It's merely an implementation detail.
 What does it mean to allocate an 'int' on the stack?

 What does it mean to delete immutable data?

Deallocate the storage for it and make it available for reuse. Accessing it afterwards leads to arbitrary behaviour. This is the same with mutable data. As the program may behave arbitrarily in this case, it is valid behaviour to act as if immutable data changed.

No, you've broken the type system if you've deleted immutable data. If I have a reference to an immutable variable, I have a guarantee that it will never change. delete will break that guarantee. With a mutable variable, I have no such guarantee. (It's not safe to allocate something different in the deleted location, but it's OK to run the finalizer and then wipe all the memory).
 I think it's reasonable for both of them to require a cast, even in
  system code.

The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping.

No we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that? void foo(bool b, string y) { immutable (char)[4] x = "abba"; string s = b ? x : y; f(s); } Make it safe.
 Rust's borrowed pointers may give some hints on how
 to extend 'scope' to fields of structs.

I think it is more fundamental than that.
 As to delete, delete is as unsafe when the involved data is immutable
 as when it is mutable. Why require an additional cast in one case?

This is not about safety. Modifying immutable data breaks the type system. Deleting mutable data does not. AFAIK it is safe to implement delete as a call to the finalizer, followed by setting the memory to T.init. Only the GC can determine if it is safe to reuse the memory. Deleting immutable data just doesn't make sense.
Oct 18 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/18/2012 10:08 AM, Don Clugston wrote:
 On 17/10/12 18:02, Timon Gehr wrote:
 On 10/17/2012 01:49 PM, Don Clugston wrote:
 ...

 That's the point -- *which* checks are missing from  safe?

Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages.

HOW do you propose to check for escaping stack data?

Static escape analysis. Use the 'scope' qualifier to designate data that is not allowed to be escaped in order to make it modular.
 ...
 The implementation of the 'scope' storage class should be fixed. We
 could then require an unsafe cast(scope) to disable prevention of stack
 address escaping.

No we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that?

f can know that it mustn't escape it, which is enough.
 void foo(bool b, string y)
 {
    immutable (char)[4] x = "abba";
    string s = b ? x : y;
    f(s);
 }

 Make it safe.

It is safe if the parameter to f is marked with 'scope'. (and this in turn obliges f not to escape it.) Analyze scope on the expression level. The analysis would determine that x[] is 'scope'. It would conservatively propagate this fact to (b ? x[] : y). Then the local variable 's' will get the 'scope' storage class. In general, use a fixed-point iteration to determine all local variables that might refer to scope'd data and prevent that they get escaped.
 Rust's borrowed pointers may give some hints on how
 to extend 'scope' to fields of structs.

I think it is more fundamental than that.
 As to delete, delete is as unsafe when the involved data is immutable
 as when it is mutable. Why require an additional cast in one case?

This is not about safety. Modifying immutable data breaks the type system. Deleting mutable data does not. AFAIK it is safe to implement delete as a call to the finalizer, followed by setting the memory to T.init. ...

Now I see where you are coming from. This is indeed a safe approach for references to/arrays of fully mutable value types, but not for delete in general. Make sure to treat void* specially though. struct S{ immutable int x; this(int x){this.x=x;}} void main() safe{ void* s = new S(2); delete s; } Class instance memory does not have a T.init, because it is not assigned a T. And even if it was, how would you know at compile time if the bound instance has any immutable fields? Should that be a runtime exception?
Oct 18 2012
parent Don Clugston <dac nospam.com> writes:
On 18/10/12 19:43, Timon Gehr wrote:
 On 10/18/2012 10:08 AM, Don Clugston wrote:
 On 17/10/12 18:02, Timon Gehr wrote:
 On 10/17/2012 01:49 PM, Don Clugston wrote:
 ...

 That's the point -- *which* checks are missing from  safe?

Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages.

HOW do you propose to check for escaping stack data?

Static escape analysis. Use the 'scope' qualifier to designate data that is not allowed to be escaped in order to make it modular.
 ...
 The implementation of the 'scope' storage class should be fixed. We
 could then require an unsafe cast(scope) to disable prevention of stack
 address escaping.

No we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that?

f can know that it mustn't escape it, which is enough.
 void foo(bool b, string y)
 {
    immutable (char)[4] x = "abba";
    string s = b ? x : y;
    f(s);
 }

 Make it safe.

It is safe if the parameter to f is marked with 'scope'. (and this in turn obliges f not to escape it.)

Well, OK, but that involves changing the semantics of immutable. You could not pass this kind of "local immutable" to _any_ existing code. It would render almost all existing code that uses immutable obsolete. And what do you get in exchange? Practically nothing!
 Analyze scope on the expression level.

 The analysis would determine that x[] is 'scope'. It would
 conservatively propagate this fact to (b ? x[] : y). Then the local
 variable 's' will get the 'scope' storage class.

 In general, use a fixed-point iteration to determine all local
 variables that might refer to scope'd data and prevent that they get
 escaped.

 Rust's borrowed pointers may give some hints on how
 to extend 'scope' to fields of structs.

I think it is more fundamental than that.
 As to delete, delete is as unsafe when the involved data is immutable
 as when it is mutable. Why require an additional cast in one case?

This is not about safety. Modifying immutable data breaks the type system. Deleting mutable data does not. AFAIK it is safe to implement delete as a call to the finalizer, followed by setting the memory to T.init. ...

Now I see where you are coming from. This is indeed a safe approach for references to/arrays of fully mutable value types, but not for delete in general. Make sure to treat void* specially though. struct S{ immutable int x; this(int x){this.x=x;}} void main() safe{ void* s = new S(2); delete s; } Class instance memory does not have a T.init, because it is not assigned a T. And even if it was, how would you know at compile time if the bound instance has any immutable fields? Should that be a runtime exception?

Probably. Yeah, it's a bit hard if you have a base class, you can't statically check if it has immutable members in a derived class. Or you could be conservative and disallow delete of anything where you don't know the exact type at compile time.
Oct 22 2012
prev sibling parent "Malte Skarupke" <malteskarupke web.de> writes:
The issue is that you're thinking as you would in Java.

I guess the rule in D for immutable is this: Immutable data won't 
change as long as it exists.

The last part of that sentence would be a stupid thing to say in 
Java because things don't just cease to exist while you're still 
doing something with them. That is not the case in D.

That being said it's very unlikely that you will ever run into 
this situation. You have to end the lifetime of the object 
manually to run into these issues. And in that case it'll be very 
easy to figure out what's wrong.
Oct 17 2012
prev sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 10/18/12 10:08, Don Clugston wrote:
 On 17/10/12 18:02, Timon Gehr wrote:
 On 10/17/2012 01:49 PM, Don Clugston wrote:
 On 01/01/12 13:50, Timon Gehr wrote:
 On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
 So, I'm a function `f`, I have an `immutable(type)[]` argument and I
 want to store it for my friend `g` in an TLS variable `v`:
 ---
 string v;
 debug string sure;

 void f(string s) { v = s; debug sure = s.idup; }
 void g() { assert(v == sure); }
 ---
 I also store a copy of `s` into `sure` for my friend to ensure
 immutable
 date hasn't been mutated.
 Can my friend's assertion ever fail without breaking a type-system?
 Sure. Just consider this:
 ---
 void main() {
 auto s = "abba".idup;
     f(s);
     delete s;
     g();
 }
 ---
 Is it by-design? Looks like deleting immutable (and const because of
 implicit conversion) data should be prohibited.
 OK. Let `delete` be fixed. Can we still fail?
 ---
 void h() {
     immutable(char)[4] s = "abba";
     f(s);
 }
 void main() {
     h();
     g();
 }
 ---
 Damn! So, what can we do with it? Not sure, but I have a proposal.

 Fix it in language:
 * disallow `delete` of const/immutable data
 * disallow immutable data on the stack

 This makes data really immutable if I don't miss something. Anyway, I
 want `immutable` qualified data to be immutable without breaking a
 type-system (if one do it, its his own responsibility), so some changes
 should be made (IMHO).

You are using unsafe language features to break the type system. That is not the fault of the type system. ' safe:' at the top of the program should stop both examples from working, it is a bug that it does not.

That's the point -- *which* checks are missing from safe?

Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages.

HOW do you propose to check for escaping stack data?

/How/ is not a problem (ignoring implementation costs), the /language definition/ part is trickier - you need a very precise definition of what is allowed and what isn't; otherwise different compilers will make different decisions and every compiler will support only a vendor-specific non-std dialect... (eg storing a scoped-ref into some kind of container, passing that down to other functions could work, but what if you then need to let the container escape and want to do that by removing the scoped-ref? It might be possible for the compiler to prove that it's safe, but it's unlikely that every compiler will act the same)
 But I'm not sure that you're right, this looks broken to me, even
 without  safe.

 What does it mean to create immutable data on the stack? The stack is
 intrinsically mutable!

So is the heap.

No it is not. Data on the stack *cannot* survive past the end of the function call. Data on the heap can last forever.

Lifetime and mutability are different things.
 What does it mean to garbage collect immutable data?

From the point of view of the application, it doesn't happen. There are no observable semantics. It's merely an implementation detail.
 What does it mean to allocate an 'int' on the stack?

 What does it mean to delete immutable data?

Deallocate the storage for it and make it available for reuse. Accessing it afterwards leads to arbitrary behaviour. This is the same with mutable data. As the program may behave arbitrarily in this case, it is valid behaviour to act as if immutable data changed.

No, you've broken the type system if you've deleted immutable data. If I have a reference to an immutable variable, I have a guarantee that it will never change. delete will break that guarantee.

Yes. The alternative (to allow explicit delete on immutable data) would likely be too complicated to be worth implementing in the near future - you need to ensure the data is unique, there are no other refs to it, and forbid accessing it after the 'delete' op. I guess an easy way out would be to ask the GC to run a collect cycle and return back whether an object was successfully collected. But i can't really see a useful application for it, and you'd need a special convention, as the GC would have to given the last ref to the object.
 With a mutable variable, I have no such guarantee. (It's not safe to allocate
something different in the deleted location, but it's OK to run the finalizer
and then wipe all the memory).
 
 I think it's reasonable for both of them to require a cast, even in
  system code.

The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping.

No we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that? void foo(bool b, string y) { immutable (char)[4] x = "abba"; string s = b ? x : y; f(s); } Make it safe.

Trivial. void f(scope string s) { /*v = s; # now illegal */ debug sure = s.idup; } Obviously, it would need a properly working 'scope'. /Then/ making it illegal to pass refs to local (or otherwise) scoped object in a way that would allow them to escape would work. Right now, when 'scope' is just decoration, trying to enforce such restrictions would only create more bugs and confusion, I'm afraid. The other reason why a working 'scope' is important is so that void g(string s) { f(s~"xyz"); } does not cause heap allocation unless necessary. Even when the allocation is necessary the new object can be reused and the given back to the allocator when it's really dead. Reducing the amount of garbage left behind is the best way to optimize the GC... [of course 'scope' should have been the default etc etc, but that's not really D2 material] artur
Oct 18 2012