www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Want reasonable reference counting? Disable automatic sharing of

reply Steven Schveighoffer <schveiguy gmail.com> writes:
One of the prerequisites to doing reference counting is to have a 
mutable piece of data inside an immutable piece of data.

A few years ago, Timon Gehr implemented a concept for `__mutable` that 
looked like it might be the answer. But there was a large problem. As 
the title suggests, it's because immutable is implicitly sharable.

This makes sense in a world of fully transitive immutable, where if you 
have an immutable pointer, nothing in it can ever change. But if you 
have a `__mutable` island, that can change. So the answer at the time 
was, `immutable(__mutable)` -> `shared`.

Unfortunately, this means you have to apply it to `const` as well, 
because `const` could be pointing to `immutable`. I'll note that this 
still might be a valuable concept, just slightly annoying, especially 
with `shared` becoming more tightly controlled.

I thought of an idea -- maybe you make it so if a type `T` had a 
`__mutable` piece, you can't share an `immutable(T)`. This actually 
works quite well, and now you have to explicitly have 
`shared(immutable(T))` in order to share it.

But there is a rather large problem -- classes. Classes might have a 
`__mutable` buried in a derived object that the compiler isn't aware of. 
So that is a no-go.

Another possibility is to introduce new type qualifiers, like 
immutable-with-mutable or const-with-mutable, which are not shared, but 
I feel like the appetite for new qualifiers is pretty low, and the names...

I've come to the conclusion, in order to fix this situation, so 
`__mutable` really means mutable and `shared` is an orthogonal piece, 
you need to remove the implicit sharing of `immutable`. While it can 
make sense, the conflation of the two concepts causes impossible-to-fix 
issues. Not just mutable, things like thread-local garbage collection 
might be easier if you have to explicitly share things.

-Steve
Nov 12 2021
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12.11.21 13:31, Steven Schveighoffer wrote:

 
 I've come to the conclusion, in order to fix this situation, so 
 `__mutable` really means mutable and `shared` is an orthogonal piece, 
 you need to remove the implicit sharing of `immutable`.


I agree, that would be much better.
Nov 12 2021
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-12 8:03, Timon Gehr wrote:

 On 12.11.21 13:31, Steven Schveighoffer wrote:

 I've come to the conclusion, in order to fix this situation, so 
 `__mutable` really means mutable and `shared` is an orthogonal piece, 
 you need to remove the implicit sharing of `immutable`.


 
 I agree, that would be much better.


We discussed this a couple of times. It's interesting. Sadly at this 
point implicit thread sharing of immutable is so baked into the 
language, it would take a lot of care to extricate. It would be very 
difficult even for Timon or Paul.
Nov 12 2021
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/12/21 1:12 PM, Andrei Alexandrescu wrote:

 On 2021-11-12 8:03, Timon Gehr wrote:

 On 12.11.21 13:31, Steven Schveighoffer wrote:

 I've come to the conclusion, in order to fix this situation, so 
 `__mutable` really means mutable and `shared` is an orthogonal piece, 
 you need to remove the implicit sharing of `immutable`.


 I agree, that would be much better.


 
 We discussed this a couple of times. It's interesting. Sadly at this 
 point implicit thread sharing of immutable is so baked into the 
 language, it would take a lot of care to extricate. It would be very 
 difficult even for Timon or Paul.


I'm not going to fight for this, I just wanted to bring it up. But I 
think it's pointless to try for reference counting of immutable data 
(the long RCSlice thread), when it's obviously not immutable data.

The only solution is to make it not mean the same as immutable does now 
(i.e. possibly shared). The fact that `shared(immutable)` would mean the 
same thing as `immutable` does now is a path forward, but I can 
understand if nobody wants to do that. I happen to think most cases 
people are not sharing their immutable data, just like they mostly 
aren't sharing thread-local data.

-Steve
Nov 12 2021
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-12 14:22, Steven Schveighoffer wrote:

 On 11/12/21 1:12 PM, Andrei Alexandrescu wrote:

 On 2021-11-12 8:03, Timon Gehr wrote:

 On 12.11.21 13:31, Steven Schveighoffer wrote:

 I've come to the conclusion, in order to fix this situation, so 
 `__mutable` really means mutable and `shared` is an orthogonal 
 piece, you need to remove the implicit sharing of `immutable`.


 I agree, that would be much better.


 We discussed this a couple of times. It's interesting. Sadly at this 
 point implicit thread sharing of immutable is so baked into the 
 language, it would take a lot of care to extricate. It would be very 
 difficult even for Timon or Paul.


 
 I'm not going to fight for this, I just wanted to bring it up. But I 
 think it's pointless to try for reference counting of immutable data 
 (the long RCSlice thread), when it's obviously not immutable data.


I'm not so sure. I mean no laws of physics or even of typing are broken.

Consider as a baseline the following: there is a global synchronized 
hashtable mapping pointers to objects to pointers to reference counters. 
The table is obviously mutable. Whenever a reference to some piece of 
data is copied around, a function gets called that looks up the pointer 
to the reference count and increments it. Depending on type, the 
increment is interlocked or not.

This is correct code that typechecks and everything. The challenge is to 
make it faster. One common solution is to group together the memory for 
the counter (metadata) together with the memory for the object (data). 
For example, metadata can be placed before the data (negative offsets) 
or just after the data. Then the lookup becomes as cheap as a integral 
to pointer addition. Sure, that all does take some gnarly code that is 
not automatically verifiable, but the result is safe. All languages, no 
matter how safe, have such manipulation in their implementation.

The real challenge is figuring out exactly the minimum amount of change 
to the language specification to allow defining such a scheme.


 The only solution is to make it not mean the same as immutable does now 
 (i.e. possibly shared). The fact that `shared(immutable)` would mean the 
 same thing as `immutable` does now is a path forward, but I can 
 understand if nobody wants to do that. I happen to think most cases 
 people are not sharing their immutable data, just like they mostly 
 aren't sharing thread-local data.


Though that may be a solution, I'm not sure it's the only solution. I 
can tell it's an expensive enough solution (in terms of changes to the 
language and/or broken code) that it warrants looking for another.
Nov 12 2021
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/12/21 5:07 PM, Andrei Alexandrescu wrote:

 On 2021-11-12 14:22, Steven Schveighoffer wrote:

 I'm not going to fight for this, I just wanted to bring it up. But I 
 think it's pointless to try for reference counting of immutable data 
 (the long RCSlice thread), when it's obviously not immutable data.


 
 I'm not so sure. I mean no laws of physics or even of typing are broken.
 
 Consider as a baseline the following: there is a global synchronized 
 hashtable mapping pointers to objects to pointers to reference counters. 
 The table is obviously mutable. Whenever a reference to some piece of 
 data is copied around, a function gets called that looks up the pointer 
 to the reference count and increments it. Depending on type, the 
 increment is interlocked or not.


I'm well aware of that concept. See this post I made from 2008 (!): 
https://forum.dlang.org/post/fsth4a$79l$1 digitalmars.com

The problem is of course that this is not pure. As long as you don't 
need pure reference counting, it's a possible solution.

If you do need pure reference counting, this mechanism *cannot* be used, 
even if it's not technically accessing a global. I don't care what 
tricks you want to pull, the optimizer is going to bite you for it.

I'm skeptical that even normal optimizations outside of pure won't bite 
you on this either.


 The only solution is to make it not mean the same as immutable does 
 now (i.e. possibly shared). The fact that `shared(immutable)` would 
 mean the same thing as `immutable` does now is a path forward, but I 
 can understand if nobody wants to do that. I happen to think most 
 cases people are not sharing their immutable data, just like they 
 mostly aren't sharing thread-local data.


 
 Though that may be a solution, I'm not sure it's the only solution. I 
 can tell it's an expensive enough solution (in terms of changes to the 
 language and/or broken code) that it warrants looking for another.


Understandable, altering a fundamental rule for immutable is definitely 
not the first choice I would have either.

-Steve
Nov 12 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 12 November 2021 at 23:07:02 UTC, Steven Schveighoffer 
wrote:

 The problem is of course that this is not pure. As long as you 
 don't need pure reference counting, it's a possible solution.

 If you do need pure reference counting, this mechanism *cannot* 
 be used, even if it's not technically accessing a global. I 
 don't care what tricks you want to pull, the optimizer is going 
 to bite you for it.


Not if it is built in. Anyway, you can just require all functions 
to take the counter table as a parameter, by rewriting rules, 
that makes it pure even when it isn't builin.


 I'm skeptical that even normal optimizations outside of pure 
 won't bite you on this either.


If it is builtin then optimizations cannot be a concern?
Nov 12 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/12/21 6:32 PM, Ola Fosheim Grøstad wrote:

 On Friday, 12 November 2021 at 23:07:02 UTC, Steven Schveighoffer wrote:

 The problem is of course that this is not pure. As long as you don't 
 need pure reference counting, it's a possible solution.

 If you do need pure reference counting, this mechanism *cannot* be 
 used, even if it's not technically accessing a global. I don't care 
 what tricks you want to pull, the optimizer is going to bite you for it.


 
 Not if it is built in. Anyway, you can just require all functions to 
 take the counter table as a parameter, by rewriting rules, that makes it 
 pure even when it isn't builin.


That means every pure function must be passed the global table as a 
parameter *just in case* there are some reference-counting things to do, 
and that means that no pure function can be strong-pure. Might as well 
not have pure functions in that case.


 I'm skeptical that even normal optimizations outside of pure won't 
 bite you on this either.


 
 If it is builtin then optimizations cannot be a concern?


As I said, I'm skeptical, I'm not sure if some optimization somewhere 
isn't going to make hay with our cleverness. We have smart people 
working on the compilers, so I'll leave the judgment up to them.

-Steve
Nov 12 2021
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Friday, 12 November 2021 at 23:50:15 UTC, Steven Schveighoffer 
wrote:


 That means every pure function must be passed the global table 
 as a parameter *just in case* there are some reference-counting 
 things to do, and that means that no pure function can be 
 strong-pure. Might as well not have pure functions in that case.


Might as well. So long as you're using (current spec of) GC, pure 
functions are a big fat lie. Not even because GC allocates, but 
because it runs finalizers, which can be impure. If a compiler 
makes any assumptions wrt. "pure" on a function that interacts 
with GC in any way, they're gonna be wrong assumptions.
Nov 12 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/12/21 7:27 PM, Stanislav Blinov wrote:

 On Friday, 12 November 2021 at 23:50:15 UTC, Steven Schveighoffer wrote:
 

 That means every pure function must be passed the global table as a 
 parameter *just in case* there are some reference-counting things to 
 do, and that means that no pure function can be strong-pure. Might as 
 well not have pure functions in that case.


 
 Might as well. So long as you're using (current spec of) GC, pure 
 functions are a big fat lie. Not even because GC allocates, but because 
 it runs finalizers, which can be impure. If a compiler makes any 
 assumptions wrt. "pure" on a function that interacts with GC in any way, 
 they're gonna be wrong assumptions.


This is an odd way to look at it. The finalizers are not run directly, 
and maybe not even run on the same thread as the pure function being 
run. They also should not be observable (for the most part), because you 
should not have access to that data any more.

It's like saying an OS context switch that happens in the middle of a 
pure function must somehow be valid pure code.

-Steve
Nov 12 2021
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 13 November 2021 at 03:03:21 UTC, Steven 
Schveighoffer wrote:


 This is an odd way to look at it. The finalizers are not run 
 directly, and maybe not even run on the same thread as the pure 
 function being run. They also should not be observable (for the 
 most part), because you should not have access to that data any 
 more.

 It's like saying an OS context switch that happens in the 
 middle of a pure function must somehow be valid pure code.

 -Steve


Not run directly??? As far as I know, GC, on allocation, may 
hijack the caller to do a collection. Net effect is that it 
executes arbitrary (not pure, not  safe, etc.) code within 
caller. Caller is marked pure. GC may run impure finalizers that 
mutate some global state. Something as stupid as call to "close" 
which may set errno, or, I don't know, freeing half of program 
state, mutating shared globals... There can be no "for the most 
part" here. You've got pure function mutating global state. 
That's something pure functions aren't supposed to be doing.
Unless that changed and GC isn't doing that anymore, that's a bug 
that's been open for some years now.
Nov 12 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/12/21 11:36 PM, Stanislav Blinov wrote:

 On Saturday, 13 November 2021 at 03:03:21 UTC, Steven Schveighoffer wrote:
 

 This is an odd way to look at it. The finalizers are not run directly, 
 and maybe not even run on the same thread as the pure function being 
 run. They also should not be observable (for the most part), because 
 you should not have access to that data any more.

 It's like saying an OS context switch that happens in the middle of a 
 pure function must somehow be valid pure code.


 
 Not run directly??? As far as I know, GC, on allocation, may hijack the 
 caller to do a collection. Net effect is that it executes arbitrary (not 
 pure, not  safe, etc.) code within caller.


Whether it does this or pushes it off to another thread is incidental. 
The running of finalizers is a function of the GC, not the caller.

This is no different than monads in other languages doing impure things 
initiated by pure functions.


 Caller is marked pure. GC may 
 run impure finalizers that mutate some global state. Something as stupid 
 as call to "close" which may set errno, or, I don't know, freeing half 
 of program state, mutating shared globals... There can be no "for the 
 most part" here. You've got pure function mutating global state. That's 
 something pure functions aren't supposed to be doing.


The part you are not getting is that this is not something being called 
by the pure code, it's being run by the GC. Imagine it like a context 
switch to another thread that runs the GC code, and then switches back 
to the pure code. In fact, the GC could do this ALREADY, because it 
could use one of the other threads that it has paused do the collection. 
But it doesn't really make any difference conceptually which thread runs 
it. One of those other threads could be in the middle of a pure function.

It's similar to running some kernel code, or signal code -- it's 
initiated by a separate entity, in this case the GC.


 Unless that changed and GC isn't doing that anymore, that's a bug that's 
 been open for some years now.


It should be closed as invalid. Which bug is that?

-Steve
Nov 13 2021
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 13 November 2021 at 14:34:43 UTC, Steven 
Schveighoffer wrote:


 Whether it does this or pushes it off to another thread is 
 incidental. The running of finalizers is a function of the GC, 
 not the caller.


:)


 The part you are not getting is that this is not something 
 being called by the pure code, it's being run by the GC.


```d
import someLibrary;
// someLibrary defines a module-global
// int threadLocal;

void main()
{
     someLibrary.threadLocal = () pure nothrow {
         return someLibrary.blah(42);
     } ();
     auto old = someLibrary.threadLocal;
     auto someInts = () pure { return new int[1000]; } ();
     assert(someLibrary.threadLocal == old);
}
```

That assert may fail. Or you may even crash before getting to it, 
and not with an `OutOfMemoryError`, but with a `FinalizeError`, 
depending to the value of `threadLocal`. Or it can be totally 
fine if the GC doesn't collect. What am I not getting?..


 Imagine it like a context switch to another thread that runs 
 the GC code, and then switches back to the pure code.


If I imagine that, the assert above should always hold. Because 
there should be no way that imaginary "another thread" would 
access main thread's `threadLocal`. Somehow, reality contradicts 
imagination.


 In fact, the GC could do this ALREADY, because it could use one 
 of the other threads that it has paused do the collection. But 
 it doesn't really make any difference conceptually which thread 
 runs it. One of those other threads could be in the middle of a 
 pure function.


Could be != is.


 It's similar to running some kernel code, or signal code -- 
 it's initiated by a separate entity, in this case the GC.


 Unless that changed and GC isn't doing that anymore, that's a 
 bug that's been open for some years now.


 It should be closed as invalid. Which bug is that?


https://issues.dlang.org/show_bug.cgi?id=19316

Feel free to close it as invalid, if the code above either:
- dies with an OutOfMemoryError
- passes the assert

regardless of value of `threadLocal`.

`someLibrary` can be this for testing:

```d
module someLibrary;

int threadLocal;

class Good
{
     int calc(int input) pure nothrow { return input * 2; }
}

class Bad {

     int calc(int input) pure nothrow { return input + 14; }

     ~this() {
         // I agree with Walter, dtors should always be nothrow,
         // alas current language allows this
         if (threadLocal == 56) throw new Exception("ugh");
         threadLocal = 0;
     }
}

int blah(int input) pure nothrow {
     if (input <= 25)
        return (new Good).calc(input);
     else
        return (new Bad).calc(input);
}
```

Contrived? Maybe. Feel free to substitute `threadLocal` with 
`errno`, and make a syscall in `Bad.~this`.
Nov 13 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/13/21 3:01 PM, Stanislav Blinov wrote:

 On Saturday, 13 November 2021 at 14:34:43 UTC, Steven Schveighoffer wrote:
 

 Whether it does this or pushes it off to another thread is incidental. 
 The running of finalizers is a function of the GC, not the caller.


 
 :)
 

 The part you are not getting is that this is not something being 
 called by the pure code, it's being run by the GC.


 
 ```d
 import someLibrary;
 // someLibrary defines a module-global
 // int threadLocal;
 
 void main()
 {
      someLibrary.threadLocal = () pure nothrow {
          return someLibrary.blah(42);
      } ();
      auto old = someLibrary.threadLocal;
      auto someInts = () pure { return new int[1000]; } ();
      assert(someLibrary.threadLocal == old);
 }
 ```
 
 That assert may fail. Or you may even crash before getting to it, and 
 not with an `OutOfMemoryError`, but with a `FinalizeError`, depending to 
 the value of `threadLocal`. Or it can be totally fine if the GC doesn't 
 collect. What am I not getting?..


You are not getting that the GC collecting has nothing to do with the 
pure function's executation. The GC hijacks the current thread to do its 
business, and then passes back control to the caller.


 

 Imagine it like a context switch to another thread that runs the GC 
 code, and then switches back to the pure code.


 
 If I imagine that, the assert above should always hold. Because there 
 should be no way that imaginary "another thread" would access main 
 thread's `threadLocal`. Somehow, reality contradicts imagination.


Actually, the GC can run finalizers from ANY thread. So accessing thread 
locals in a GC finalizer is risky behavior anyway.


 It's similar to running some kernel code, or signal code -- it's 
 initiated by a separate entity, in this case the GC.


 Unless that changed and GC isn't doing that anymore, that's a bug 
 that's been open for some years now.


 It should be closed as invalid. Which bug is that?


 
 https://issues.dlang.org/show_bug.cgi?id=19316


Thanks! I closed it.


 Feel free to close it as invalid, if the code above either:
 - dies with an OutOfMemoryError
 - passes the assert


The assert is incorrectly written, as the thread local can change at any 
time if the GC happens to run on the current thread, and happens to be 
finalizing a `Bad` object (even if it was allocated via a different 
thread). This is how you set it up.

Honestly, I think accessing thread locals in a GC destructor should be 
in the spec as implementation-defined behavior.

-Steve
Nov 13 2021
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 13 November 2021 at 21:55:21 UTC, Steven 
Schveighoffer wrote:


 You are not getting that the GC collecting has nothing to do 
 with the pure function's executation. The GC hijacks the 
 current thread to do its business, and then passes back control 
 to the caller.


...while introducing side effects, which the caller of the pure 
function was promised WOULD NOT HAPPEN, by the interface of the 
pure function.


 

 Imagine it like a context switch to another thread that runs 
 the GC code, and then switches back to the pure code.


 
 If I imagine that, the assert above should always hold. 
 Because there should be no way that imaginary "another thread" 
 would access main thread's `threadLocal`. Somehow, reality 
 contradicts imagination.


 Actually, the GC can run finalizers from ANY thread. So 
 accessing thread locals in a GC finalizer is risky behavior 
 anyway.


The language allows this, and the runtime does this. There's no 
need for speculation. Risky or not. If a language guarantees 
something can't happen, it better not happen. Pure functions 
cannot mutate global state, except through arguments, in which 
case it's a weakly pure function. That's what the language 
guarantees. Pure functions can only call pure functions. That's 
what the language guarantees.


 It's similar to running some kernel code, or signal code -- 
 it's initiated by a separate entity, in this case the GC.


 Unless that changed and GC isn't doing that anymore, that's 
 a bug that's been open for some years now.


 It should be closed as invalid. Which bug is that?


 
 https://issues.dlang.org/show_bug.cgi?id=19316


 Thanks! I closed it.


Cool. Then it's on you to reopen it back.


 The assert is incorrectly written,


Read `a`, call pure function, read `a`. Where is it possible for 
`a` to mutate, after first read and before second read? Can't be 
another thread, `a` is a thread-local int. Can't be the pure 
function, it's pure and cant mutate `a`. So where is it possible 
for `a` to mutate?


 as the thread local can change at any time if the GC happens to 
 run on the current thread, and happens to be finalizing a `Bad` 
 object (even if it was allocated via a different thread).


...Which means that anything that triggers collection (including 
allocation) cannot be pure. Which is exactly my point. Nor can it 
be  safe.


 This is how you set it up.


I have no words...


 Honestly, I think accessing thread locals in a GC destructor 
 should be in the spec as implementation-defined behavior.


So, then, should be any other side effects. Good luck with that.
Nov 13 2021
next sibling parent reply Elronnd <elronnd elronnd.net> writes:
On Saturday, 13 November 2021 at 23:08:01 UTC, Stanislav Blinov 
wrote:

 Read `a`, call pure function, read `a`. Where is it possible 
 for `a` to mutate, after first read and before second read? 
 Can't be the pure function, it's pure and cant mutate `a`. So
 where is it possible for `a` to mutate?


auto x = new int;
assert(a < 5);
auto y = new int;
assert(a < 5);
*x += *y;

Would you begrudge the compiler the ability to remove the second 
assert?  Because there is the _exact_ same problem there.

Actually, forget allocation.  A signal can occur at _any_ time, 
and its handler can change globals.


 Can't be another thread, `a` is a thread-local int.


You can share pointers to thread-local objects.
Nov 13 2021
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 14 November 2021 at 00:17:29 UTC, Elronnd wrote:

 On Saturday, 13 November 2021 at 23:08:01 UTC, Stanislav Blinov 
 wrote:

 Read `a`, call pure function, read `a`. Where is it possible 
 for `a` to mutate, after first read and before second read? 
 Can't be the pure function, it's pure and cant mutate `a`. So
 where is it possible for `a` to mutate?


 auto x = new int;
 assert(a < 5);
 auto y = new int;
 assert(a < 5);
 *x += *y;

 Would you begrudge the compiler the ability to remove the 
 second assert?  Because there is the _exact_ same problem there.


So long as collection can call any destructors indiscriminately - 
yes, I would. Because with current runtime and GC spec `a` might 
mutate in either call to `new`. Even in both.


 Actually, forget allocation.  A signal can occur at _any_ time, 
 and its handler can change globals.


Indeed, forget allocation, given that I'm talking about 
collection. But anyway, so what? What do signals have to do with 
`new` pretending to be pure when it calls destructors that aren't?


 Can't be another thread, `a` is a thread-local int.


 You can share pointers to thread-local objects.


Indeed you can. How does that apply here?
Nov 13 2021
parent reply Elronnd <elronnd elronnd.net> writes:
On Sunday, 14 November 2021 at 01:15:06 UTC, Stanislav Blinov 
wrote:

 Indeed, forget allocation, given that I'm talking about 
 collection. But anyway, so what? What do signals have to do 
 with `new` pretending to be pure when it calls destructors that 
 aren't?


A signal handler may not be pure, and it may be called at any 
time.  Including during the execution of a pure function.  Hence, 
your argument implies that no function should ever be marked as 
pure, because it may be interrupted by a signal handler which has 
some effects.
Nov 13 2021
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 14 November 2021 at 02:24:42 UTC, Elronnd wrote:


 A signal handler may not be pure, and it may be called at any 
 time.  Including during the execution of a pure function.  
 Hence, your argument implies that no function should ever be 
 marked as pure, because it may be interrupted by a signal 
 handler which has some effects.


Yes, yes, and cosmic radiation may flip a bit in my CPU thus also 
mutate my state. I still don't see what that has to do with the 
language and its runtime.
Nov 13 2021
parent reply Elronnd <elronnd elronnd.net> writes:
On Sunday, 14 November 2021 at 02:54:52 UTC, Stanislav Blinov 
wrote:

 On Sunday, 14 November 2021 at 02:24:42 UTC, Elronnd wrote:


 A signal handler may not be pure, and it may be called at any 
 time.  Including during the execution of a pure function.  
 Hence, your argument implies that no function should ever be 
 marked as pure, because it may be interrupted by a signal 
 handler which has some effects.


 Yes, yes, and cosmic radiation may flip a bit in my CPU thus 
 also mutate my state. I still don't see what that has to do 
 with the language and its runtime.


Signals are part of the language.  Proof: GC must be able to 
hijack threads; so there must be some mechanism for preempting an 
existing thread of execution; and GC is part of the language.

Hence, that a function is 'pure' cannot be a strict indicator 
that, between the time it is called and the time it returns, no 
side effects are performed.

So it seems not at all inconsistent to me that a pure function 
may cause the GC to run, causing a destructor to be run.
Nov 13 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 14 November 2021 at 06:53:24 UTC, Elronnd wrote:

 So it seems not at all inconsistent to me that a pure function 
 may cause the GC to run, causing a destructor to be run.


Thus pure functions are allowed to conduct I/O, albeit a 
primitive one.

Note that a pure function also can prevent destruction merely by 
computing an integer that is aliasing an address of an object 
that is up for destruction. Again, primitive I/O. So clearly a 
side effect.
Nov 14 2021
parent reply Elronnd <elronnd elronnd.net> writes:
On Sunday, 14 November 2021 at 08:29:33 UTC, Ola Fosheim Grøstad 
wrote:

 Thus pure functions are allowed to conduct I/O, albeit a 
 primitive one.

 Note that a pure function also can prevent destruction merely 
 by computing an integer that is aliasing an address of an 
 object that is up for destruction. Again, primitive I/O. So 
 clearly a side effect.


And a pure function can also conduct I/O by constructing a null 
pointer (or being passed one) and dereferencing it; and then 
doing the work in the SIGSEGV handler.

I don't think this makes pure functions less interesting or 
meaningful from a practical standpoint.
Nov 14 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 14 November 2021 at 09:19:24 UTC, Elronnd wrote:

 I don't think this makes pure functions less interesting or 
 meaningful from a practical standpoint.


Just remove the claim that D privides strong purity... Leave it 
at weakly pure and list known caveats.

(It can also never return or terminate the thread by 
overextending the stack, we can just go on and on...)
Nov 14 2021
parent Elronnd <elronnd elronnd.net> writes:
On Sunday, 14 November 2021 at 09:28:14 UTC, Ola Fosheim Grøstad 
wrote:

 On Sunday, 14 November 2021 at 09:19:24 UTC, Elronnd wrote:

 I don't think this makes pure functions less interesting or 
 meaningful from a practical standpoint.


 Just remove the claim that D privides strong purity... Leave it 
 at weakly pure and list known caveats.


Yes!  It is already not strongly pure because it can mutate 
through its parameters.
Nov 14 2021
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Saturday, 13 November 2021 at 23:08:01 UTC, Stanislav Blinov 
wrote:

 On Saturday, 13 November 2021 at 21:55:21 UTC, Steven 
 Schveighoffer wrote:


 You are not getting that the GC collecting has nothing to do 
 with the pure function's executation. The GC hijacks the 
 current thread to do its business, and then passes back 
 control to the caller.


 ...while introducing side effects, which the caller of the pure 
 function was promised WOULD NOT HAPPEN, by the interface of the 
 pure function.


Dude, I don't want to be rude, but you clearly don't have the 
prerequisite t add value to this thread. It's tiring to have to 
swift through pages of this to get to interesting infos.

Arguably, there may be a problem with the way finalizer are 
handled, but this has zero to do with the topic of this thread. 
You could remove the second function call that you assert could 
still fail. It has noting to do with the call, or it being pure 
of any of what you think it has to do with.

Please do not clutter this thread.
Nov 14 2021
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 14 November 2021 at 13:30:28 UTC, deadalnix wrote:


 Dude, I don't want to be rude, but...


If you didn't, you wouldn't have been. Please don't go saying I 
made you.


 Please do not clutter this thread.


Oh don't worry, this is the last of me in this thread. Rejoice 
and break out the drinks.
Nov 14 2021
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 13 November 2021 at 03:03:21 UTC, Steven 
Schveighoffer wrote:

 It's like saying an OS context switch that happens in the 
 middle of a pure function must somehow be valid pure code.


No, by extending the lifetime of a gc object you guarantee that 
the finalizer is not executed. That means you prevent a possible 
side effect from occuring, which in itself is a side effect.

Assume that the finalizer calls exit() or assert(0) or does out 
of bounds indexing.

If you want truly strong purity you can only allow the function 
to extend lifetimes of objects with trivial destruction.

This might be more than you wish for, so just clarify what you 
want to achieve with pure.
Nov 12 2021
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 12 November 2021 at 23:50:15 UTC, Steven Schveighoffer 
wrote:

 That means every pure function must be passed the global table 
 as a parameter *just in case* there are some reference-counting 
 things to do, and that means that no pure function can be 
 strong-pure. Might as well not have pure functions in that case.


Depends on how you define pure.

Is it a tool for compiler optimizations? Not a problem. The 
compiler knows the invariants for reference counting.

Is it a tool to make it easier for programmers to reason about 
programs. Clarfy in the sec what you can assume.

Do you want the compiler to check strong purity? Define it. Make 
it possible to express it and let the compiler check it.

If you require pure to mean that no life times are extended, say 
so in the spec and let the compiler check it.


 I'm skeptical that even normal optimizations outside of pure 
 won't bite you on this either.


 
 If it is builtin then optimizations cannot be a concern?


 As I said, I'm skeptical, I'm not sure if some optimization 
 somewhere isn't going to make hay with our cleverness. We have 
 smart people working on the compilers, so I'll leave the 
 judgment up to them.


An compiler optimization cannot create problems, by definition, 
as it does not change the language. If it creates problems then 
it means that the language specification is in need of fixing. Or 
that that the programmers assume things that the language does 
not guarantee.
Nov 12 2021
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12.11.21 23:07, Andrei Alexandrescu wrote:

 
 The real challenge is figuring out exactly the minimum amount of change 
 to the language specification to allow defining such a scheme.


I think one reason we haven't been making progress on this may be that 
your intuition for the size of the minimum required amount of change to 
the language specification is somewhat too small.

You have stated requirements, some of them are impossible to satisfy, 
most of them will require a sizeable language change. We have to go 
through those problems one by one and address them in order.

You have to factor in that _a lot_ of the work is actually clarifying 
existing semantics and fixing issues there.

My suggestion would be to prioritize satisfying the requirements that do 
not relate to type qualifiers.



 
 - be as close in semantics to T[] as possible, i.e. most code should be able
to simply replace T[] with RCSlice!T and work unchanged


`T[]` is magic. We first need to make it possible to define a template 
in druntime that works as a drop-in replacement for `T[]`, even without 
reference counting.


 - manage its own memory by using reference counting


Easy, except for lifetime bugs in throwing constructors. Some hassle 
with default init.


 - work in pure code just like T[] does


Impossible, due to strong purity. `__mutable` would have fixed this.


 - work with qualifiers like T[] does,


Impossible. `T[]` is magic and `inout` is broken.


 both RCSlice!(qual T)


Easy.


 and qual(RCSlice!T)


Impossible. Would need an escape hatch for the reference count, which 
you have shot down before. `__mutable` would have fixed this.

Adding appropriate subtyping relationships between them also requires a 
new language feature.


 - work in  nogc code just like T[] does


Very easy.


 - work in  safe code just like T[] does 


Impossible. This has spawned DIP1000 and large amounts of follow-up 
work. And all of this work still does not solve the problem it set out 
to address. I think this is the main problem. It's not actually very 
important to support type/function qualifiers beyond `pure  safe  nogc 
nothrow`. The issue is that you cannot borrow out a `ref T`. This could 
be solved with some explicit language support, where the compiler tracks 
the lifetime of the reference and lets the container know once it's dead.
Nov 12 2021
prev sibling next sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Friday, 12 November 2021 at 18:12:14 UTC, Andrei Alexandrescu 
wrote:

 On 2021-11-12 8:03, Timon Gehr wrote:

 On 12.11.21 13:31, Steven Schveighoffer wrote:

 I've come to the conclusion, in order to fix this situation, 
 so `__mutable` really means mutable and `shared` is an 
 orthogonal piece, you need to remove the implicit sharing of 
 `immutable`.


 
 I agree, that would be much better.


 We discussed this a couple of times. It's interesting. Sadly at 
 this point implicit thread sharing of immutable is so baked 
 into the language, it would take a lot of care to extricate. It 
 would be very difficult even for Timon or Paul.


Just track whether an RC target needs atomic RC operations at 
compile time, using the template parameters of RC types. No 
changes to language semantics are necessary.

For example, reject attempts to assign an `RCPtr!(immutable T)` 
to an `RCPtr!(const T)`, while allowing assignment to a 
`RCPtr!(shared const T)`. Borrowed payloads can and should follow 
the normal D qualifier conversion rules.

Or, if more granularity is needed, use a separate template 
parameter:

```D
struct RCPtr(T, bool sharedMeta = is(T == shared))
```
Nov 12 2021
parent Paul Backus <snarwin gmail.com> writes:
On Friday, 12 November 2021 at 19:22:57 UTC, tsbockman wrote:

 Just track whether an RC target needs atomic RC operations at 
 compile time, using the template parameters of RC types. No 
 changes to language semantics are necessary.

 For example, reject attempts to assign an `RCPtr!(immutable T)` 
 to an `RCPtr!(const T)`, while allowing assignment to a 
 `RCPtr!(shared const T)`. Borrowed payloads can and should 
 follow the normal D qualifier conversion rules.

 Or, if more granularity is needed, use a separate template 
 parameter:

 ```D
 struct RCPtr(T, bool sharedMeta = is(T == shared))
 ```


This is essentially what Rust does--except instead of a template 
parameter, they have two separate generic types, [`Rc<T>`][1] and 
[`Arc<T>`][2].

[1]: https://doc.rust-lang.org/std/rc/struct.Rc.html
[2]: https://doc.rust-lang.org/std/sync/struct.Arc.html
Nov 12 2021
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 12 November 2021 at 18:12:14 UTC, Andrei Alexandrescu 
wrote:

 We discussed this a couple of times. It's interesting. Sadly at 
 this point implicit thread sharing of immutable is so baked 
 into the language, it would take a lot of care to extricate. It 
 would be very difficult even for Timon or Paul.


Just define a new keyword for unshared immutable and make 
"immutable" an alias.
Nov 13 2021
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 12 November 2021 at 18:12:14 UTC, Andrei Alexandrescu 
wrote:

 On 2021-11-12 8:03, Timon Gehr wrote:

 On 12.11.21 13:31, Steven Schveighoffer wrote:

 I've come to the conclusion, in order to fix this situation, 
 so `__mutable` really means mutable and `shared` is an 
 orthogonal piece, you need to remove the implicit sharing of 
 `immutable`.


 
 I agree, that would be much better.


 We discussed this a couple of times. It's interesting. Sadly at 
 this point implicit thread sharing of immutable is so baked 
 into the language, it would take a lot of care to extricate. It 
 would be very difficult even for Timon or Paul.


Why? Just introduce ```readonly``` as a qualifier for unshared 
immutable and define ```immutable``` to be a shortcut for 
```shared``` + ```readonly```.
Nov 13 2021
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
I really don't think this makes sense.

Providing people with tool to work on multiple cores (and we are 
aggressively moving toward heterogeneous compute, so it's getting 
worse).

On the other hand, atomic are becoming cheaper and cheaper with 
each generation of pretty much every platforms. This is mostly 
due to the fact most machines now do regular loads/stores 
optimistically and speculate from there, just like they do for 
branch predictions. If it turns out there was contention then 
they rewind and revert back to do it in a more expensive fashion.

So that trade does not seems to make sense.
Nov 12 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/12/21 12:46 PM, deadalnix wrote:

 I really don't think this makes sense.
 
 Providing people with tool to work on multiple cores (and we are 
 aggressively moving toward heterogeneous compute, so it's getting worse).


`shared(immutable)` is fine for such a task. You only need to atomically 
load any `__mutable` pieces inside.

having immutable implicitly shared is stupid when it's obviously not. 
Like an immutable stack variable, why should I have to worry about 
sharability there? Should I even be able to share it with another 
thread, as the stack is easily destroyed?


 On the other hand, atomic are becoming cheaper and cheaper with each 
 generation of pretty much every platforms. This is mostly due to the 
 fact most machines now do regular loads/stores optimistically and 
 speculate from there, just like they do for branch predictions. If it 
 turns out there was contention then they rewind and revert back to do it 
 in a more expensive fashion.


Removing implicit sharing from immutable doesn't change what shared 
immutable means, or affect the performance of it, it just means you need 
to be explicit about sharing.

-Steve
Nov 12 2021
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2021 11:16 AM, Steven Schveighoffer wrote:

 having immutable implicitly shared is stupid when it's obviously not. Like an 
 immutable stack variable, why should I have to worry about sharability there? 
 Should I even be able to share it with another thread, as the stack is easily 
 destroyed?


Mutability and lifetime are orthogonal attributes.
Nov 14 2021
prev sibling next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Friday, 12 November 2021 at 12:31:03 UTC, Steven Schveighoffer 
wrote:

 One of the prerequisites to doing reference counting is to have 
 a mutable piece of data inside an immutable piece of data.

 [...]


Just putting this here for reference (regarding implicitly shared 
immutable)

![Quadrants](https://image.slidesharecdn.com/refactoringtoimmutability-180703134457/95/refactoring-to-imm
tability-24-638.jpg "Immutable-shared")
Nov 12 2021
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
The very idea is strange. If RCSlice is mutable, how can it be 
immutable? Something doesn't add up.
Nov 12 2021
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/12/21 2:38 PM, Kagamin wrote:

 The very idea is strange. If RCSlice is mutable, how can it be 
 immutable? Something doesn't add up.


Look up logical const, that is what this is. The mutable part never is 
immutable or const. We just also don't want it to be shared by default.

See also, C++ mutable keyword.

-Steve
Nov 12 2021
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Friday, 12 November 2021 at 12:31:03 UTC, Steven Schveighoffer 
wrote:

 I've come to the conclusion, in order to fix this situation, so 
 `__mutable` really means mutable and `shared` is an orthogonal 
 piece, you need to remove the implicit sharing of `immutable`. 
 While it can make sense, the conflation of the two concepts 
 causes impossible-to-fix issues. Not just mutable, things like 
 thread-local garbage collection might be easier if you have to 
 explicitly share things.

 -Steve


I'd rather keep `immutable` fully transitive. That's how it's 
designed to work, and does at least some things well, like 
allowing multithearded access. With some mutable gaps it's going 
to have many of the problems of C++ `const`, and also more 
complicated. Plus no changes needed.

I do not think having an immutable counted reference is 
necessary. With present `immutable` we can still have a mutable 
counted reference to immutable payload. It does mean some extra 
complications if the reference is itself stored in immutable 
data, but the cure would be worse than the disease I think.

If we drop the immutability requirement from the reference, we 
can have ` safe pure  nogc` reference counting, or can we?
Nov 13 2021
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to have a mutable
piece 
 of data inside an immutable piece of data.


Or maybe just give up on having immutable ref counted objects. Ref counted 
objects are mutable.
Nov 13 2021
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 14/11/2021 8:16 PM, Walter Bright wrote:

 On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to have a 
 mutable piece of data inside an immutable piece of data.


 
 Or maybe just give up on having immutable ref counted objects. Ref 
 counted objects are mutable.


This isn't entirely true.

If all calls into reference counting have a known delta of zero, all 
calls in are elided therefore nothing changed.

const + scope + ARC has some very interesting possibilities here.
Nov 13 2021
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2021 11:16 PM, Walter Bright wrote:

 Or maybe just give up on having immutable ref counted objects. Ref counted 
 objects are mutable.


Let me expand on that a bit.

This conversation reminds me of the old "logical const" debate, where people 
wanted to have constant objects that were not constant. That just would not
work 
with the notion of transitive const.

Having a __mutable field in an immutable hierarchy is the same abomination. 
Attempting it immediately starts causing the rest of the type axioms in D to 
come apart. As Steven pointed out, sharing stops working.

It's like defining pi to be 3.0, and a right angle to be 89 degrees. Your house 
will not fit together if the corners are 89 degrees.

How does C++ manage it? They don't have immutable types. Const is just a 
documentation attribute.

How do functional languages do it? Reference counted types are not part of the 
type system. They are behind the curtain, just like how the gc works in D is 
behind the curtain, and associative arrays(!).

D has these choices:

1. ref counted objects are mutable

2. ref counted objects are outside of the type system

3. break the type system

I find (3) to be a bit ironic after D has been lambasted for having
inconsistent 
semantic rules.

PS. I suspect that ref counted shared mutable objects are an indicator of
insanity.
Nov 14 2021
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.11.21 09:12, Walter Bright wrote:

 On 11/13/2021 11:16 PM, Walter Bright wrote:

 Or maybe just give up on having immutable ref counted objects. Ref 
 counted objects are mutable.


 
 Let me expand on that a bit.
 
 This conversation reminds me of the old "logical const" debate, where 
 people wanted to have constant objects that were not constant. That just 
 would not work with the notion of transitive const.
 ...


The GC is able to deallocate `immutable` data. I don't think it's 
entirely unreasonable to give that capability to  system user code as 
well, but saying `immutable` means GC or infinite lifetime is of course 
possible.


 Having a __mutable field in an immutable hierarchy is the same 
 abomination. Attempting it immediately starts causing the rest of the 
 type axioms in D to come apart.


Well, the way I imagined it to work is:

- Carefully define what the axioms are in terms of allowed equivalence 
transformations of source code.
-  trusted code that mutates __mutable data has to be prepared for any 
of those equivalence transformations.
- __mutable functions are exempt of most or all of those equivalence 
transformations.

In principle, this could work. (But of course, it would make the axioms 
more complex and may prevent an expansion of rewrites that are based on 
the simpler meaning of attributes in the future.)


 As Steven pointed out, sharing stops working.
 ...


Yes, but implicit sharing has other drawbacks, e.g., it may prevent 
thread-local GC for thread-local immutable data such as strings.


 It's like defining pi to be 3.0, and a right angle to be 89 degrees. 
 Your house will not fit together if the corners are 89 degrees.
 
 How does C++ manage it? They don't have immutable types. Const is just a 
 documentation attribute.
 
 How do functional languages do it? Reference counted types are not part 
 of the type system. They are behind the curtain, just like how the gc 
 works in D is behind the curtain, and associative arrays(!).
 ...


__mutable variables and functions are an attempt to formalize being 
"behind the curtain", so that the runtime ceases to be language magic 
and  system user code can have well-defined access to the other side of 
the curtain. This allows the runtime to be implemented in terms of 
straight lowering to templates without special rules.


 D has these choices:
 
 1. ref counted objects are mutable
 ...


Works, but you may have to extend it to "anything that's not allocated 
with the GC is mutable". Deallocation eventually will change the memory 
you pass into it.


 2. ref counted objects are outside of the type system
 ...


For this to work, there still needs to be a careful definition what 
 trusted functions are allowed to do with data of certain qualification. 
I think adding some type system support that is strictly  system, like 
__mutable variables and functions is better than having no language 
support at all, because otherwise, in order to support reference 
counting, you may end up penalizing code that does not actually use it.


 3. break the type system
 
 I find (3) to be a bit ironic after D has been lambasted for having 
 inconsistent semantic rules.
 ...


Yes, I don't think adding UB to the standard library is an option.


 PS. I suspect that ref counted shared mutable objects are an indicator 
 of insanity.


Possibly, though it's hard to know everyone's use cases in advance.
Nov 14 2021
parent reply Elronnd <elronnd elronnd.net> writes:
On Sunday, 14 November 2021 at 18:24:00 UTC, Timon Gehr wrote:

 1. ref counted objects are mutable


 Works, but you may have to extend it to "anything that's not 
 allocated with the GC is mutable".


Static data can be immutable too.



 2. ref counted objects are outside of the type system


 For this to work, there still needs to be a careful definition 
 what  trusted functions are allowed to do with data of certain 
 qualification. I think adding some type system support that is 
 strictly  system, like __mutable variables and functions is 
 better than having no language support at all, because 
 otherwise, in order to support reference counting, you may end 
 up penalizing code that does not actually use it.


I assumed that 'outside of the type system' meant 'language-level 
support for reference counting'; a a type with opaque 
representation, like a hash table.



 PS. I suspect that ref counted shared mutable objects are an 
 indicator of insanity.


 Possibly, though it's hard to know everyone's use cases in 
 advance.


GC really makes more sense for multithreaded programs.  Travis 
downs mentions at 
https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html.  IMO if
you are sharing mutable data between threads, and you can't stand gc for some
reason, then it's perfectly legitimate to make you manage the lifetime yourself.
Nov 14 2021
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.11.21 22:50, Elronnd wrote:

 On Sunday, 14 November 2021 at 18:24:00 UTC, Timon Gehr wrote:

 1. ref counted objects are mutable


 Works, but you may have to extend it to "anything that's not allocated 
 with the GC is mutable".


 
 Static data can be immutable too.
 ...


True, and you can also just leak. I did mention that in my other post, 
but I forgot to add it here, thanks!


 

 2. ref counted objects are outside of the type system


 For this to work, there still needs to be a careful definition what 
  trusted functions are allowed to do with data of certain 
 qualification. I think adding some type system support that is 
 strictly  system, like __mutable variables and functions is better 
 than having no language support at all, because otherwise, in order to 
 support reference counting, you may end up penalizing code that does 
 not actually use it.


 
 I assumed that 'outside of the type system' meant 'language-level 
 support for reference counting'; a a type with opaque representation, 
 like a hash table.
 ...


Yes, I guess that's another way to go about it, but I think less 
language magic is better. Especially with reference counting, you may 
want to have detailed control over the allocator and/or the strategy for 
storing reference counts.


 

 PS. I suspect that ref counted shared mutable objects are an 
 indicator of insanity.


 Possibly, though it's hard to know everyone's use cases in advance.


 
 GC really makes more sense for multithreaded programs.  Travis downs 
 mentions at 
 https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html.  
 IMO if you are sharing mutable data between threads, and you can't stand 
 gc for some reason, then it's perfectly legitimate to make you manage 
 the lifetime yourself.


Also fair.

However, I think having the language provide some tools to implement 
stuff that's at the same level as the language runtime does make some 
sense for a systems programming language. E.g., see 
std.experimental.allocator. It would be a pity if the language hobbled 
that library by having uncircumventable pitfalls.
Nov 14 2021
parent Elronnd <elronnd elronnd.net> writes:
On Sunday, 14 November 2021 at 21:59:23 UTC, Timon Gehr wrote:

 I assumed that 'outside of the type system' meant 
 'language-level support for reference counting'; a a type with 
 opaque representation, like a hash table.
 ...


 Yes, I guess that's another way to go about it, but I think 
 less language magic is better. Especially with reference 
 counting, you may want to have detailed control over the 
 allocator and/or the strategy for storing reference counts.


Definitely agree!

That said I think there may be a middle ground, involving some 
magic but also some control.  In particular, compared with 
__mutable, language-level support for intrusive rc can be sound 
without leading to surprising results, as long as you're not 
allowed to read from the reference count.  Saying 'build me a 
reference-counted object from this chunk of memory' should be 
pretty easy, and ditto specifying how the memory is disposed of 
once the reference count reaches 0.  Being polymorphic over 
intrusive/extrusive representation is harder, though.
Nov 14 2021
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Sunday, 14 November 2021 at 07:16:43 UTC, Walter Bright wrote:

 On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to 
 have a mutable piece of data inside an immutable piece of data.


 Or maybe just give up on having immutable ref counted objects. 
 Ref counted objects are mutable.


The object can still be immutable, but the reference count and 
the references, save for weak pointers, can't. I think.
Nov 14 2021
prev sibling next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 14 November 2021 at 07:16:43 UTC, Walter Bright wrote:

 On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to 
 have a mutable piece of data inside an immutable piece of data.


 Or maybe just give up on having immutable ref counted objects. 
 Ref counted objects are mutable.


Yes That.
Nov 14 2021
prev sibling next sibling parent Alexandru Ermicioi <alexandru.ermicioi gmail.com> writes:
On Sunday, 14 November 2021 at 07:16:43 UTC, Walter Bright wrote:

 On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to 
 have a mutable piece of data inside an immutable piece of data.


 Or maybe just give up on having immutable ref counted objects. 
 Ref counted objects are mutable.


That's best option imho. The ref counted objects should be 
mutable, while the payload, being of any constness level.

Like why do we even try to have ref counters be immutable? By 
their nature, they imply that some data is going to be modified 
(counter).
We could simply disable the ability to construct const and 
immutable versions of them, and that would remove the problem 
entirely.

For the case when a mutable ref counter becomes const, we could 
just disable access to the payload, forcing user to fix their 
code.
Or the counter could return a wrapper over payload that is not 
copyable or assignable, forcing user, to use wrapper with payload 
as soon as it was returned. Wrapper would have then an opDispatch 
overload, in order to not expose the payload to the user, and 
hence allow borrowing.

Best regards,
Alexandru.
Nov 14 2021
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.11.21 08:16, Walter Bright wrote:

 On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to have a 
 mutable piece of data inside an immutable piece of data.


 
 Or maybe just give up on having immutable ref counted objects. Ref 
 counted objects are mutable.


Yes, this is definitely a reasonable way to resolve this debate.
Nov 14 2021
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2021 10:02 AM, Timon Gehr wrote:

 On 14.11.21 08:16, Walter Bright wrote:

 On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to have a mutable 
 piece of data inside an immutable piece of data.


 Or maybe just give up on having immutable ref counted objects. Ref counted 
 objects are mutable.


 
 Yes, this is definitely a reasonable way to resolve this debate.


Sometimes the obvious way is the best way!

Note that one still can have an immutable *payload* for a ref counted object.
Nov 14 2021
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.11.21 19:08, Walter Bright wrote:

 On 11/14/2021 10:02 AM, Timon Gehr wrote:

 On 14.11.21 08:16, Walter Bright wrote:

 On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to have a 
 mutable piece of data inside an immutable piece of data.


 Or maybe just give up on having immutable ref counted objects. Ref 
 counted objects are mutable.


 Yes, this is definitely a reasonable way to resolve this debate.


 
 Sometimes the obvious way is the best way!
 
 Note that one still can have an immutable *payload* for a ref counted 
 object.


I am not sold on that yet. How do you manually deallocate an immutable 
payload?
Nov 14 2021
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2021 10:25 AM, Timon Gehr wrote:

 How do you manually deallocate an immutable payload?


The same way it is done now. Call free().

Calling free() on an object ends its lifetime. As I mentioned to Steven, 
lifetime and mutability are independent attributes.
Nov 14 2021
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.11.21 20:51, Walter Bright wrote:

 On 11/14/2021 10:25 AM, Timon Gehr wrote:

 How do you manually deallocate an immutable payload?


 
 The same way it is done now. Call free().
 
 Calling free() on an object ends its lifetime. As I mentioned to Steven, 
 lifetime and mutability are independent attributes.
 


I agree that they are independent attributes, but how does the compiler 
know that something is an allocation/deallocation function? (My 
suggestion was to annotate such functions __mutable.)
Nov 14 2021
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2021 12:18 PM, Timon Gehr wrote:

 On 14.11.21 20:51, Walter Bright wrote:

 On 11/14/2021 10:25 AM, Timon Gehr wrote:

 How do you manually deallocate an immutable payload?


 The same way it is done now. Call free().

 Calling free() on an object ends its lifetime. As I mentioned to Steven, 
 lifetime and mutability are independent attributes.


 
 I agree that they are independent attributes, but how does the compiler know 
 that something is an allocation/deallocation function? (My suggestion was to 
 annotate such functions __mutable.)


It's done in  live functions. The functions aren't annotated specially, it's 
just that passing a pointer argument to a non-scope parameter means the pointer 
value is transferred, and is no longer 'live' in the calling function.
Nov 14 2021
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.11.21 03:04, Walter Bright wrote:

 On 11/14/2021 12:18 PM, Timon Gehr wrote:

 On 14.11.21 20:51, Walter Bright wrote:

 On 11/14/2021 10:25 AM, Timon Gehr wrote:

 How do you manually deallocate an immutable payload?


 The same way it is done now. Call free().

 Calling free() on an object ends its lifetime. As I mentioned to 
 Steven, lifetime and mutability are independent attributes.


 I agree that they are independent attributes, but how does the 
 compiler know that something is an allocation/deallocation function? 
 (My suggestion was to annotate such functions __mutable.)


 
 It's done in  live functions. The functions aren't annotated specially, 
 it's just that passing a pointer argument to a non-scope parameter means 
 the pointer value is transferred, and is no longer 'live' in the calling 
 function.


I am aware, but I don't see how it helps. Where is the curtain and how 
do you go behind it?
Nov 14 2021
prev sibling next sibling parent Daniel N <no public.email> writes:
On Sunday, 14 November 2021 at 18:08:29 UTC, Walter Bright wrote:

 Sometimes the obvious way is the best way!

 Note that one still can have an immutable *payload* for a ref 
 counted object.


Maybe, it is reasonable, but I think we didn't fully explore 
mutable, we simply have to take a different route than C++ did. 
As I see it, the problem in D is forced transitivity.

immutable(mutable(int))  = immutable(int) (Unlike C++ !)
immutable(mutable(int)*) = immutable pointer to mutable int

Punching holes in immutable objects like C++ did causes major 
issues as the object might be stored in read-only memory or 
shared, etc.
Nov 14 2021
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Nov 14, 2021 at 10:08:29AM -0800, Walter Bright via Digitalmars-d wrote:

 On 11/14/2021 10:02 AM, Timon Gehr wrote:

 On 14.11.21 08:16, Walter Bright wrote:

 On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to have
 a mutable piece of data inside an immutable piece of data.


 
 Or maybe just give up on having immutable ref counted objects.
 Ref counted objects are mutable.


 
 Yes, this is definitely a reasonable way to resolve this debate.


 
 Sometimes the obvious way is the best way!
 
 Note that one still can have an immutable *payload* for a ref counted
 object.


This seems to be a reasonable way to resolve this issue.

However, it does have far-reaching implications. For example, it implies
that refcounted objects cannot be used inside any type that may get
passed to a const-receiving interface.

As a hypothetical example, say we have this code:

	class SomePayload { ... }

	struct DataContainer {
		string metadata;
		SomePayload payload;
		SomeOtherPayload payload2;
	}

	void updateData(ref DataContainer data) {
		// presumably update data
	}

	void analyzeData(in DataContainer data) {
		// read-only access of data
	}

	void main() {
		DataContainer data = getData(...);
		updateData(data);
		analyzeData(data);
	}

Using `in` for analyzeData is reasonable, since the function does not
wish to modify the payload.

Now, suppose after some development we decide that we want to make
SomePayload a ref-counted resource instead of a GC-allocated class
instance. So we attempt to modify DataContainer thus:

	struct DataContainer {
		string metadata;
		RefCounted!SomePayload payload;
		SomeOtherPayload payload2;
	}

Now we have a problem: we can no longer call analyzeData with the data,
because it requires mutation of the ref count.

I deliberately nested the refcounted object inside another type, to
illustrate my point: if refcounted objects require mutation (which makes
sense since the ref count needs to be updated), that means *no* function
that might possibly receive a refcounted object (as a nested part of a
larger data structure) will be able to take it as a const argument.

IOW, const becomes unusable with any object that could potentially
contain a ref-counted sub-object.

This is probably not a problem in non-template code, but when you're
dealing with generic code, this will exclude ref-counted objects from
any generic function that uses const in any way. Generic code does not
know (nor care) whether some subobject of an argument might possibly be
refcounted, but it seems reasonable (and likely) that a generic function
that does not plan to modify an argument would use `in` or `const` to
qualify its parameters.  Such a function would not be usable with any
argument that might possibly have ref-counted objects nested somewhere
within it.

So, either (1) we cannot use const/in in generic functions, or (2)
ref-counted objects cannot be used in generic code.

Given D's emphasis on generic code, (1) seems like the only viable
option.  Which makes const in D even narrower in scope than ever.

The same applies for wrapper objects: if any part of the object may
contain a ref-counted object, the entire object becomes unusable with
const.

So, const is turtles all the way down, and ref-counting must be mutable
all the way *up*.


T

-- 
Change is inevitable, except from a vending machine.
Nov 14 2021
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.11.21 21:04, H. S. Teoh wrote:

 
 So, either (1) we cannot use const/in in generic functions, or (2)
 ref-counted objects cannot be used in generic code.
 
 Given D's emphasis on generic code, (1) seems like the only viable
 option.  Which makes const in D even narrower in scope than ever.


I don't agree that it changes anything for generic code. `const` already 
prevents useful patterns such as lazy initialization.

The issue is that despite all that, people still think D const is 
similar to C++ const. It's really not. Notions such as "const 
correctness" are misguided in D.
Nov 14 2021
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Nov 14, 2021 at 09:22:39PM +0100, Timon Gehr via Digitalmars-d wrote:

 On 14.11.21 21:04, H. S. Teoh wrote:

 
 So, either (1) we cannot use const/in in generic functions, or (2)
 ref-counted objects cannot be used in generic code.
 
 Given D's emphasis on generic code, (1) seems like the only viable
 option.  Which makes const in D even narrower in scope than ever.


 
 I don't agree that it changes anything for generic code. `const`
 already prevents useful patterns such as lazy initialization.


Yes, which already confines const to a very narrow scope indeed. And if
we now make ref-counting require mutable, then const is basically
relegated to the niche of niches, usable only for very narrow and
specific use cases like const(char)[] or immutable(char)[] aka string.

As I've mentioned before, while const/immutable do provide very strong
guarantees, they are also very narrow in scope, such that IME they are
rarely useful outside of bottom-level types (i.e., leaf nodes in the
graph of type dependencies). Outside of that, they're so restrictive
that they aren't really applicable most of the time.  They are useful
only 1 or 2 levels above PODs, anything else and you start running into
problems.

It seems wasteful to allocate a large part of the type system to
something so limited in applicability in real-life code.



 The issue is that despite all that, people still think D const is
 similar to C++ const. It's really not. Notions such as "const
 correctness" are misguided in D.


C++ const is a joke, I'm not even thinking about it. :-D

The thing is that `in` is touted (in tutorials, documentation, and
sometimes in advice to D newbies) as a good practice for function
parameters when the function is not modifying data. In practice,
however, this is only applicable to PODs and aggregates 1 or 2 levels
above that. Beyond that `in` basically adds no value, since as you
mentioned, it excludes a lot of useful idioms like lazy initialization,
caching, and now ref-counting.

An entire keyword, spent on a feature that can only be used in code of
limited complexity.


T

-- 
Государство делает вид, что платит нам
зарплату, а мы делаем вид, что работаем.
Nov 14 2021
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 14.11.21 21:46, H. S. Teoh wrote:

 On Sun, Nov 14, 2021 at 09:22:39PM +0100, Timon Gehr via Digitalmars-d wrote:

 On 14.11.21 21:04, H. S. Teoh wrote:

 So, either (1) we cannot use const/in in generic functions, or (2)
 ref-counted objects cannot be used in generic code.

 Given D's emphasis on generic code, (1) seems like the only viable
 option.  Which makes const in D even narrower in scope than ever.


 I don't agree that it changes anything for generic code. `const`
 already prevents useful patterns such as lazy initialization.


 
 Yes, which already confines const to a very narrow scope indeed. And if
 we now make ref-counting require mutable, then const is basically
 relegated to the niche of niches, usable only for very narrow and
 specific use cases like const(char)[] or immutable(char)[] aka string.
 ...


Well, my point was, _this is the status quo_. we are not now "making" it 
anything. Of course, it's more likely that those who are unaware will 
notice it if reference counting is popularized as a memory management 
strategy. I guess this was your point.

Interestingly, std.typecons.RefCounted "works" with const objects, but 
that's only because postblit is unsound.
Nov 14 2021
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 14.11.21 21:46, H. S. Teoh wrote:

 The thing is that `in` is touted (in tutorials, documentation, and
 sometimes in advice to D newbies) as a good practice for function
 parameters when the function is not modifying data. In practice,
 however, this is only applicable to PODs and aggregates 1 or 2 levels
 above that. Beyond that `in` basically adds no value, since as you
 mentioned, it excludes a lot of useful idioms like lazy initialization,
 caching, and now ref-counting.
 
 An entire keyword, spent on a feature that can only be used in code of
 limited complexity.


True, that's not optimal.
Nov 14 2021
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 14 November 2021 at 20:46:59 UTC, H. S. Teoh wrote:

 C++ const is a joke, I'm not even thinking about it. :-D


Actually, it isn't a joke. Const encourages the programmer to 
keep things const, and the compiler can establish it by static 
analysis before depending on it. Basically, it means that you 
write more code that is "const" than you otherwise might have 
done.

It is the same as with "pure" (no globals) in D. If D had made 
"pure" the default then programmers would have to think twice 
about accessing globals and written more code that is local and 
easier to maintain and optimize.
Nov 14 2021
prev sibling next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 15/11/2021 9:04 AM, H. S. Teoh wrote:

 IOW, const becomes unusable with any object that could potentially
 contain a ref-counted sub-object.


I've mentioned this before, but this pattern could be rectified with 
scope + ARC.
Nov 14 2021
prev sibling next sibling parent tsbockman <thomas.bockman gmail.com> writes:
On Sunday, 14 November 2021 at 20:04:40 UTC, H. S. Teoh wrote:

 However, it does have far-reaching implications. For example, 
 it implies that refcounted objects cannot be used inside any 
 type that may get passed to a const-receiving interface.


While this is a problem, it is not as extreme as you make it 
sound.

Why? Because borrowing does not involve writing to the reference 
count, and most functions just need access to, not ownership of, 
their arguments. So `const` will often work.

Nevertheless, you still have a strong point. I think the better 
option is to sacrifice `pure` in order to make transitive `const` 
and `immutable` work properly with reference counting.

I can't think of any real problems this would cause; generic code 
usually infers purity, anyway.
Nov 14 2021
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2021 12:04 PM, H. S. Teoh wrote:

 Given D's emphasis on generic code, (1) seems like the only viable
 option.  Which makes const in D even narrower in scope than ever.


C++ const doesn't actually work, which is why C++ ref counted objects can be
const.
Nov 14 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 15 November 2021 at 07:14:12 UTC, Walter Bright wrote:

 On 11/14/2021 12:04 PM, H. S. Teoh wrote:

 Given D's emphasis on generic code, (1) seems like the only 
 viable
 option.  Which makes const in D even narrower in scope than 
 ever.


 C++ const doesn't actually work, which is why C++ ref counted 
 objects can be const.


Uh? C++ shared_ptr stores the count in a separate object.

The reason you can cast away const is so that you can call C-APIs 
that actually have const-behaviour, but not a const signature.

You have to be able to do this in D to.
Nov 15 2021
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 15 November 2021 at 08:38:07 UTC, Ola Fosheim Grøstad 
wrote:

 The reason you can cast away const is so that you can call 
 C-APIs that actually have const-behaviour, but not a const 
 signature.


Also note that objects that are created as const should not be 
modified, quoting 
[cppreference](https://en.cppreference.com/w/cpp/language/cv):

«const object - an object whose type is const-qualified, or a 
non-mutable subobject of a const object. Such object cannot be 
modified: attempt to do so directly is a compile-time error, and 
attempt to do so indirectly (e.g., by modifying the const object 
through a reference or pointer to non-const type) results in 
undefined behavior.»
Nov 15 2021
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
On Monday, 15 November 2021 at 08:38:07 UTC, Ola Fosheim Grøstad 
wrote:

 On Monday, 15 November 2021 at 07:14:12 UTC, Walter Bright 
 wrote:

 On 11/14/2021 12:04 PM, H. S. Teoh wrote:

 Given D's emphasis on generic code, (1) seems like the only 
 viable
 option.  Which makes const in D even narrower in scope than 
 ever.


 C++ const doesn't actually work, which is why C++ ref counted 
 objects can be const.


 Uh? C++ shared_ptr stores the count in a separate object.

 The reason you can cast away const is so that you can call 
 C-APIs that actually have const-behaviour, but not a const 
 signature.

 You have to be able to do this in D to.


I think he meant from optimisation viewpoint. C++ `const` is so 
weak that it can't be used for actual optimisations anyway, at 
least usually. So no performance is lost because of `mutable`. In 
D the compiler can make some assumption based on `const` and 
`immutable`, so we lose some performance potential if we 
implement `mutable`.
Nov 15 2021
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 15 November 2021 at 09:22:16 UTC, Dukc wrote:

 I think he meant from optimisation viewpoint. C++ `const` is so 
 weak that it can't be used for actual optimisations anyway, at 
 least usually.


You cannot assume a lot about the underlying C++ object that is 
accessed through a reference, that is true. But you can infer the 
constness of functions by static analysis.

The most critical const objects are lookup-tables and one should 
be able to establish those as immutable in C++ too, as they are 
const objects that are not accessed through a "remote" pointer.


 So no performance is lost because of `mutable`. In D the 
 compiler can make some assumption based on `const` and 
 `immutable`, so we lose some performance potential if we 
 implement `mutable`.


Maybe, although it is kinda the same as having a mutable object 
with mostly immutable fields.

So, I would think that could be covered by having a mechanism for 
inferring "full immutability"?
Nov 15 2021
parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Monday, 15 November 2021 at 09:42:03 UTC, Ola Fosheim Grøstad 
wrote:

 On Monday, 15 November 2021 at 09:22:16 UTC, Dukc wrote:

 I think he meant from optimisation viewpoint. C++ `const` is 
 so weak that it can't be used for actual optimisations anyway, 
 at least usually.


 You cannot assume a lot about the underlying C++ object that is 
 accessed through a reference, that is true. But you can infer 
 the constness of functions by static analysis.

 The most critical const objects are lookup-tables and one 
 should be able to establish those as immutable in C++ too, as 
 they are const objects that are not accessed through a "remote" 
 pointer.


 So no performance is lost because of `mutable`. In D the 
 compiler can make some assumption based on `const` and 
 `immutable`, so we lose some performance potential if we 
 implement `mutable`.


 Maybe, although it is kinda the same as having a mutable object 
 with mostly immutable fields.

 So, I would think that could be covered by having a mechanism 
 for inferring "full immutability"?


Would a new keyword be required or would it be enough to specify 
it so that optimizations could be done etc? (So that we don't 
loose the "benefits" we have today). Sorry, I'm writing this the 
fly so haven't thought this through.

For me the right solution though seems to be keeping it 
"external".

Why? Don't touch the object, the usage of it is not of it's 
concern.

If we view it as a canonical object like the number 7, we don't 
keep track in the object itself where it's used. When we bring it 
from the platonic to the physical world by "instantiating" it as 
a copy. If you destroy our copy, the canonical (eternal) object 
doesn't change.

Hence, it makes more sense to keep track of it in a separately, 
maybe even in another universe (depending on interpretation).

You can view it as what's happening is something like you get a 
copy of the entire universe with one bit changed, let's say a 
count variable x (I'm only using natural numbers here). The 
canonical values lives outside the omniverse.

In this example a universe can bind a canonical value to a 
variable and observe the value of a variable. I have also thrown 
in the successor function (suc(n) = n+1) and predecessor function 
(pre(n+1) = n) because why not (this is pseudo for the sake of 
discussion)

```d
Universe u0 = Omniverse.getUniverse(0); //You reference a 
universe by a natural number or view it as a new Universe() where 
the omniverse sets the "blueprint", this boils down to the 
interpretation

// Bind 7 to x, it just means that the (canonical value from the 
platonic domain) of 7 is stored within the universe as a 
(imperfect or perfect (hopefully)) representation (data)
u0.bind(x, 7);

// What's happening here is another canonical value is being 
bound and put into x (yes I know)
u0.bind(x, suc(u0.observe(x))); //This is (if the observation is 
not corrupted etc) equivalent with u0.bind(x, 8)

// A new universe would get x = 8 here
Universe u1 = u0; // Create a new universe by copying an old one 
from the omniverse (or not, depending on your interpretation, I 
don't have time to write both atm)

u1.bind(x, pre(u1.observe(x))); // Bind x in this universe to the 
predecessor of x
u1.observe(x); // Would observe 7

// The philosophical discussions begin here...

// Should/can u1 be able to observe something in u0, if so, how?
// What makes the omniverse consistent?
// etc etc, not so interested in this atm :)
```

So the hierarchy would be
Platonic -> Omniverse -> Universe -> Data -> Information 
(interpreted data)

The destruction (removal), mutation (intentional change) or 
corruption (unintentional change) of a variable does not in any 
way make its way up the hierarchy by other than explicit copy 
(using that interpretation), it's contained within the universe 
or can only move horizontally.

How does this relate the question at hand?
In the way that you keep the count separate from the object.

The mapping between the count and the object is just a matter of 
what relation exist in the universe (or even between universes 
depending on the interpretation etc).

One could easily imagine a model where there are multiple 
views/counts of an object in different environments.

So, in essence, if you want to keep a count of something you 
should probably do it separately. I think it has a higher 
probability of being correct, but the con is it will probably 
impact performance a bit (I have no solid stats on it, I'm just 
guessing).

As I see it all boils down to if you view the universe as 
something that mutates itself recursively or that the universe is 
always new with every iteration ie Bohr vs Heisenberg etc.

Wait, is this even the right thread. Nvm, I'll post it here

Now I'm late for lunch and it's your fault! *raises fist*
Nov 15 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 15 November 2021 at 12:58:46 UTC, Imperatorn wrote:

 On Monday, 15 November 2021 at 09:42:03 UTC, Ola Fosheim 
 Grøstad wrote:

 So, I would think that could be covered by having a mechanism 
 for inferring "full immutability"?


 Would a new keyword be required or would it be enough to 
 specify it so that optimizations could be done etc? (So that we 
 don't loose the "benefits" we have today). Sorry, I'm writing 
 this the fly so haven't thought this through.

 For me the right solution though seems to be keeping it 
 "external".

 Why? Don't touch the object, the usage of it is not of it's 
 concern.


I am primarily thinking about potential use cases in 
metaprogramming. For instance: a hashtable that wants to treat 
keys as binary data and requires the key to be "fully immutable" 
(you don't want to hash a refcount). The hashtable does not care 
what the type is, it just want a definitive answer about the 
key's ability to change. You need more information than this, of 
course, you also need to know if they key is in a canonical form.

I'm starting to think that the C-style typing isn't really 
powerful or flexible enough to support meta-programming well. I 
guess such querying also could be done as a library feature, I 
think C++ was onto something in their library when they added 
some information about buffers being contiguous in memory that 
could be queried. Typing in languages are often on a higher 
abstract level, but in system level programming one often want 
machine level knowledge about the concrete representation.

However, doing it on the library level often requires the 
programmer who build types to take these needs into account, 
maybe there are opportunities to create robust language level 
inference that is better? Clearly some things cannot easily be 
inferred (such as whether the key is in canonical form), but 
maybe some things can be inferred and that would be interesting 
to think about.


 Hence, it makes more sense to keep track of it in a separately, 
 maybe even in another universe (depending on interpretation).


Yes, you could say that too. Then you are thinking about it in 
abstract terms.


 Now I'm late for lunch and it's your fault! *raises fist*


\*offers you a cup of tea\*
Nov 15 2021
parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Monday, 15 November 2021 at 14:59:48 UTC, Ola Fosheim Grøstad 
wrote:

 On Monday, 15 November 2021 at 12:58:46 UTC, Imperatorn wrote:

 On Monday, 15 November 2021 at 09:42:03 UTC, Ola Fosheim 
 Grøstad wrote:

 So, I would think that could be covered by having a mechanism 
 for inferring "full immutability"?




 \*offers you a cup of tea\*


Thanks, I needed that ☕
Nov 16 2021
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 15 November 2021 at 09:22:16 UTC, Dukc wrote:

 I think he meant from optimisation viewpoint. C++ `const` is so 
 weak that it can't be used for actual optimisations anyway, at 
 least usually. So no performance is lost because of `mutable`. 
 In D the compiler can make some assumption based on `const` and 
 `immutable`, so we lose some performance potential if we 
 implement `mutable`.


You can only optimize based on `immutable` and even then you need 
to run data-flow analysis to figure out whether someone _could_ 
do any casts which make the optimization invalid.

And const doesn't provide any additional grantees an optimizer 
could use directly.
It's a feature to prevent misuse of an API only.

I have personal experience which show that as soon as multiple 
threads involved const does more harm than good because it only 
allows you to reason about _your access rights_ you cannot infer 
anything about anyone else.
Nov 15 2021
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 15 November 2021 at 11:58:56 UTC, Stefan Koch wrote:

 And const doesn't provide any additional grantees an optimizer 
 could use directly.
 It's a feature to prevent misuse of an API only.

 I have personal experience which show that as soon as multiple 
 threads involved const does more harm than good because it only 
 allows you to reason about _your access rights_ you cannot 
 infer anything about anyone else.


I wonder if it would be interesting to have a const reference 
that guarantees that the underlying data-structure does not 
change as long as the reference or any copies of it are live? I 
guess that could be another reason for adding tracking of 
borrowing.

Use case: you obtain a record from a database thread in a 
transaction, when the transaction is ending the database gets to 
mutate the record again.

You add the constraint that only  trusted functions are allowed 
to create such const-references.
Nov 15 2021
prev sibling parent claptrap <clap trap.com> writes:
On Monday, 15 November 2021 at 09:22:16 UTC, Dukc wrote:

 On Monday, 15 November 2021 at 08:38:07 UTC, Ola Fosheim 
 Grøstad wrote:

 On Monday, 15 November 2021 at 07:14:12 UTC, Walter Bright 
 wrote:

 On 11/14/2021 12:04 PM, H. S. Teoh




 I think he meant from optimisation viewpoint. C++ `const` is so 
 weak that it can't be used for actual optimisations anyway, at 
 least usually. So no performance is lost because of `mutable`. 
 In D the compiler can make some assumption based on `const` and 
 `immutable`, so we lose some performance potential if we 
 implement `mutable`.


Theres a talk on youtube by Chandler Carruth about llvm / clang 
optimisation and he says const is litteraly of no use at all.
Nov 15 2021
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/14/21 2:16 AM, Walter Bright wrote:

 On 11/12/2021 4:31 AM, Steven Schveighoffer wrote:

 One of the prerequisites to doing reference counting is to have a 
 mutable piece of data inside an immutable piece of data.


 
 Or maybe just give up on having immutable ref counted objects. Ref 
 counted objects are mutable.


First, I'll note that `__mutable` on its own is perfectly sound. The 
biggest problem is that `immutable(T)` no longer would mean "fully 
immutable", which removes a lot of optimizations (including strong-pure) 
-- even in the case of a type that doesn't have any `__mutable` in sight.

I'm fine with giving up on it. But... 
https://forum.dlang.org/post/smc5jb$2u8t$1 digitalmars.com

	- work with qualifiers like T[] does, both RCSlice!(qual T) and 
qual(RCSlice!T)

...

	2. Qualifiers compound problems with interlocking: mutable data is 
known to be single-threaded, so no need for interlocking. Immutable data 
may be multi-threaded, meaning reference counting needs atomic 
operations. Const data has unknown origin, which means the information 
of how data originated (mutable or not) must be saved at runtime.

We need a change of requirements if we want to give up on immutable 
reference counting.

-Steve
Nov 14 2021
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2021 1:12 PM, Steven Schveighoffer wrote:

 First, I'll note that `__mutable` on its own is perfectly sound.


I don't think so. It produces two kinds of immutability - one with mutable 
members, one with not. Different rules apply.

But then what do you do with opaque types? The compiler doesn't know which kind 
of immutable they are. So it is forced to go with the worst case - immutables 
are mutable. And it all falls apart.



      2. Qualifiers compound problems with interlocking: mutable data is
known to 
 be single-threaded, so no need for interlocking. Immutable data may be 
 multi-threaded, meaning reference counting needs atomic operations. Const data 
 has unknown origin, which means the information of how data originated
(mutable 
 or not) must be saved at runtime.


Right.


 We need a change of requirements if we want to give up on immutable reference 
 counting.


Yup.
Nov 14 2021
next sibling parent reply Danni Coy <danni.coy gmail.com> writes:
rather than punching a hole in const/immutable

struct S
meta {
int refCount;
....
}
do {
 int payload;
....
}

Immutable S s is now conceptually an immutable object and a mutable
object with metadata behind the scenes
that feels like a single object from a user perspective?
Nov 15 2021
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 15/11/2021 10:28 PM, Danni Coy wrote:

 that feels like a single object from a user perspective?


The most important thing about this is: it'll all be in continuous memory.

Perfect for all of your crashing needs due to read only memory!
Nov 15 2021
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 15.11.21 11:36, rikki cattermole wrote:

 
 On 15/11/2021 10:28 PM, Danni Coy wrote:

 that feels like a single object from a user perspective?


 
 The most important thing about this is: it'll all be in continuous memory.
 
 Perfect for all of your crashing needs due to read only memory!


Whoever allocates the object knows the memory layout and can put it in 
writable memory.
Nov 15 2021
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 15 November 2021 at 12:24:33 UTC, Timon Gehr wrote:

 On 15.11.21 11:36, rikki cattermole wrote:

 
 On 15/11/2021 10:28 PM, Danni Coy wrote:

 that feels like a single object from a user perspective?


 
 The most important thing about this is: it'll all be in 
 continuous memory.
 
 Perfect for all of your crashing needs due to read only memory!


 Whoever allocates the object knows the memory layout and can 
 put it in writable memory.


If the object is allocated, it HAS TO BE in mutable memory.
Nov 15 2021
next sibling parent IGotD- <nise nise.com> writes:
On Monday, 15 November 2021 at 14:27:29 UTC, deadalnix wrote:

 On Monday, 15 November 2021 at 12:24:33 UTC, Timon Gehr wrote:

 On 15.11.21 11:36, rikki cattermole wrote:

 
 On 15/11/2021 10:28 PM, Danni Coy wrote:

 that feels like a single object from a user perspective?


 
 The most important thing about this is: it'll all be in 
 continuous memory.
 
 Perfect for all of your crashing needs due to read only 
 memory!


 Whoever allocates the object knows the memory layout and can 
 put it in writable memory.


 If the object is allocated, it HAS TO BE in mutable memory.


Yes, I thought this discussion was a little bit strange, maybe 
because my lack of knowledge. However, if an object is allocated 
in the heap it has be in mutable memory.

Keep in mind that doesn't need to be the same has the type 
designation, immutable might be in RAM. Anyway, we can cast 
modify constants that are heap allocated. How does the compiler 
know that something is allocated on the heap?

We're still discussing around the main issue which is fixing the 
type system so that managed memory pointers has a distinctive 
type.
Nov 15 2021
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 15.11.21 15:27, deadalnix wrote:

 On Monday, 15 November 2021 at 12:24:33 UTC, Timon Gehr wrote:

 On 15.11.21 11:36, rikki cattermole wrote:

 On 15/11/2021 10:28 PM, Danni Coy wrote:

 that feels like a single object from a user perspective?


 The most important thing about this is: it'll all be in continuous 
 memory.

 Perfect for all of your crashing needs due to read only memory!


 Whoever allocates the object knows the memory layout and can put it in 
 writable memory.


 
 If the object is allocated, it HAS TO BE in mutable memory.


No, you can allocate it on the GC heap in CTFE and then store it in the 
immutable data segment at runtime by assigning to a static immutable 
variable.

In general, you can allocate first and then change the protection 
attributes on the page later.
Nov 15 2021
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 16/11/2021 4:23 AM, Timon Gehr wrote:

 On 15.11.21 15:27, deadalnix wrote:

 On Monday, 15 November 2021 at 12:24:33 UTC, Timon Gehr wrote:

 On 15.11.21 11:36, rikki cattermole wrote:

 On 15/11/2021 10:28 PM, Danni Coy wrote:

 that feels like a single object from a user perspective?


 The most important thing about this is: it'll all be in continuous 
 memory.

 Perfect for all of your crashing needs due to read only memory!


 Whoever allocates the object knows the memory layout and can put it 
 in writable memory.


 If the object is allocated, it HAS TO BE in mutable memory.


 
 No, you can allocate it on the GC heap in CTFE and then store it in the 
 immutable data segment at runtime by assigning to a static immutable 
 variable.
 
 In general, you can allocate first and then change the protection 
 attributes on the page later.


Exactly.

There is nothing stopping the compiler putting immutable values in ROM, 
nor the user who didn't know that something like __mutable is in use 
from making memory they allocated read only.

const and immutables are compiler guarantees, and we shouldn't be 
wanting the compiler to lie to us in our misunderstandings.
Nov 15 2021
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 15 November 2021 at 21:30:29 UTC, rikki cattermole 
wrote:

 const and immutables are compiler guarantees, and we shouldn't 
 be wanting the compiler to lie to us in our misunderstandings.


Hm, ```immutable```, ```const```, and ```pure``` are arbitrary 
symbols that means nothing more than what an updated language 
spec claim. Of course, one might argue that it is bad usability!

Also, you have to be in ``` trusted``` to change the setting of 
memory pages, and in ``` trusted``` you basically take upon 
yourself to uphold the spec (which involves a strong element of 
*hubris* given that few programmers know the full language 
semantics).

So that is basically the key issue. For ``` trusted``` to make 
sense, the type system has to be simple and intuitive.

So, I disagree, but I agree. :)
Nov 15 2021
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 16/11/2021 10:38 AM, Ola Fosheim Grøstad wrote:

 On Monday, 15 November 2021 at 21:30:29 UTC, rikki cattermole wrote:

 const and immutables are compiler guarantees, and we shouldn't be 
 wanting the compiler to lie to us in our misunderstandings.


 
 Hm, ```immutable```, ```const```, and ```pure``` are arbitrary symbols 
 that means nothing more than what an updated language spec claim. Of 
 course, one might argue that it is bad usability!
 
 Also, you have to be in ``` trusted``` to change the setting of memory 
 pages, and in ``` trusted``` you basically take upon yourself to uphold 
 the spec (which involves a strong element of *hubris* given that few 
 programmers know the full language semantics).
 
 So that is basically the key issue. For ``` trusted``` to make sense, 
 the type system has to be simple and intuitive.
 
 So, I disagree, but I agree. :)


Oh yeah, all these guarantees go out the window for  system and  trusted 
code!

But at least you have to cast and do something naughty to do it, which 
is a good thing if it involves a potential crash.

What matters is that const/immutable won't let you do something naughty, 
without you actually proving that you want to do it.

So at least then  safe is still  safe ;)
Nov 15 2021
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 15 November 2021 at 22:00:51 UTC, rikki cattermole 
wrote:

 Oh yeah, all these guarantees go out the window for  system and 
  trusted code!


If you only write  system then the semantics get easier!

That is… up-side-down in a way.
Nov 15 2021
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 15 November 2021 at 21:30:29 UTC, rikki cattermole 
wrote:

 There is nothing stopping the compiler putting immutable values 
 in ROM, nor the user who didn't know that something like 
 __mutable is in use from making memory they allocated read only.


How will the compiler put something that is dynamically allocated 
in ROM?
Nov 15 2021
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 16/11/2021 2:16 PM, deadalnix wrote:

 On Monday, 15 November 2021 at 21:30:29 UTC, rikki cattermole wrote:

 There is nothing stopping the compiler putting immutable values in 
 ROM, nor the user who didn't know that something like __mutable is in 
 use from making memory they allocated read only.


 
 How will the compiler put something that is dynamically allocated in ROM?


It won't (or at least I'm not aware that there is a way to do it outside 
of globals/literals).

That particular situation would be for globals or literals.
Nov 15 2021
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 16.11.21 02:33, rikki cattermole wrote:

 On 16/11/2021 2:16 PM, deadalnix wrote:

 On Monday, 15 November 2021 at 21:30:29 UTC, rikki cattermole wrote:

 There is nothing stopping the compiler putting immutable values in 
 ROM, nor the user who didn't know that something like __mutable is in 
 use from making memory they allocated read only.


 How will the compiler put something that is dynamically allocated in ROM?


 
 It won't (or at least I'm not aware that there is a way to do it outside 
 of globals/literals).
 
 That particular situation would be for globals or literals.


Again, the compiler is aware which memory ranges have __mutable fields 
in them and can just put those in the writable data segment. There is no 
opacity at that point. I think this is just a non-issue, you only pay 
for what you use.
Nov 15 2021
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-15 20:16, deadalnix wrote:

 On Monday, 15 November 2021 at 21:30:29 UTC, rikki cattermole wrote:

 There is nothing stopping the compiler putting immutable values in 
 ROM, nor the user who didn't know that something like __mutable is in 
 use from making memory they allocated read only.


 
 How will the compiler put something that is dynamically allocated in ROM?


At least in theory it could mark pages as read-only. Current compilers 
do that for statically-initialized constant data, but not for anything else.
Nov 16 2021
parent reply Elronnd <elronnd elronnd.net> writes:
On Tuesday, 16 November 2021 at 17:52:10 UTC, Andrei Alexandrescu 
wrote:

 At least in theory it could mark pages as read-only. Current 
 compilers do that for statically-initialized constant data, but 
 not for anything else.


Would require annoying bookkeeping for small allocations.  
Obvious route is on every allocation you unprotect/reprotect; 
that's slow.  Alternately you can make one rw mapping and one ro 
mapping; write the newly created data through rw space, but 
return a pointer to ro space.  But then you are technically not 
'immutable'.
Nov 16 2021
parent IGotD- <nise nise.com> writes:
On Tuesday, 16 November 2021 at 19:39:02 UTC, Elronnd wrote:

 Would require annoying bookkeeping for small allocations.  
 Obvious route is on every allocation you unprotect/reprotect; 
 that's slow.  Alternately you can make one rw mapping and one 
 ro mapping; write the newly created data through rw space, but 
 return a pointer to ro space.  But then you are technically not 
 'immutable'.


I'll give you a hint: Don't do many small allocations. There is 
an overhead regardless of RC or not.
Nov 30 2021
prev sibling parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Monday, 15 November 2021 at 12:24:33 UTC, Timon Gehr wrote:

 On 15.11.21 11:36, rikki cattermole wrote:

 
 On 15/11/2021 10:28 PM, Danni Coy wrote:

 that feels like a single object from a user perspective?


 
 The most important thing about this is: it'll all be in 
 continuous memory.
 
 Perfect for all of your crashing needs due to read only memory!


 Whoever allocates the object knows the memory layout and can 
 put it in writable memory.


Could be implemented by a pointer containing the address of the 
writable sub-struct. The address never changes so the pointer can 
reside in ROM.
Nov 16 2021
parent Daniel N <no public.email> writes:
On Tuesday, 16 November 2021 at 16:26:25 UTC, Patrick Schluter 
wrote:

 On Monday, 15 November 2021 at 12:24:33 UTC, Timon Gehr wrote:

 On 15.11.21 11:36, rikki cattermole wrote:

 
 On 15/11/2021 10:28 PM, Danni Coy wrote:

 that feels like a single object from a user perspective?


 
 The most important thing about this is: it'll all be in 
 continuous memory.
 
 Perfect for all of your crashing needs due to read only 
 memory!


 Whoever allocates the object knows the memory layout and can 
 put it in writable memory.


 Could be implemented by a pointer containing the address of the 
 writable sub-struct. The address never changes so the pointer 
 can reside in ROM.


Exactly, similar to my proposal earlier. We simply have to add a 
way to opt-out of transitive const/immutability.
Nov 16 2021
prev sibling parent reply Elronnd <elronnd elronnd.net> writes:
On Monday, 15 November 2021 at 09:28:41 UTC, Danni Coy wrote:

 rather than punching a hole in const/immutable

 struct S
 meta {
 int refCount;
 ....
 }
 do {
  int payload;
 ....
 }

 Immutable S s is now conceptually an immutable object and a 
 mutable
 object with metadata behind the scenes
 that feels like a single object from a user perspective?


That is functionally the same thing as '__mutable int refCount', 
discussed elsethread.
Nov 15 2021
parent reply Danni Coy <danni.coy gmail.com> writes:
On Mon, Nov 15, 2021 at 9:50 PM Elronnd via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 15 November 2021 at 09:28:41 UTC, Danni Coy wrote:

 rather than punching a hole in const/immutable

 struct S
 meta {
 int refCount;
 ....
 }
 do {
  int payload;
 ....
 }

 Immutable S s is now conceptually an immutable object and a
 mutable
 object with metadata behind the scenes
 that feels like a single object from a user perspective?


 That is functionally the same thing as '__mutable int refCount',
 discussed elsethread.



From an implementation point of view maybe. It has the advantage of


having the linguistic model match what the compiler is doing behind
the scenes (making it easier to reason about / conceptualise).

Would it be possible to limit operation of the writeable components to
a small subset of the language which would allow the necessary
operations but remove a lot of the complications?
Nov 15 2021
parent user1234 <user1234 12.de> writes:
On Monday, 15 November 2021 at 11:40:45 UTC, Danni Coy wrote:

 On Mon, Nov 15, 2021 at 9:50 PM Elronnd via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 On Monday, 15 November 2021 at 09:28:41 UTC, Danni Coy wrote:

 rather than punching a hole in const/immutable

 struct S
 meta {
 int refCount;
 ....
 }
 do {
  int payload;
 ....
 }

 Immutable S s is now conceptually an immutable object and a
 mutable
 object with metadata behind the scenes
 that feels like a single object from a user perspective?


 That is functionally the same thing as '__mutable int 
 refCount', discussed elsethread.



From an implementation point of view maybe. It has the 
advantage of


 having the linguistic model match what the compiler is doing 
 behind the scenes (making it easier to reason about / 
 conceptualise).

 Would it be possible to limit operation of the writeable 
 components to a small subset of the language which would allow 
 the necessary operations but remove a lot of the complications?


Yes using special case in the compiler, special case in the type 
system.
(looks like "meta" is a bit like "__mutable", which's been 
discussed previously).

In the domain of special cases I'd prefer a built-in reference 
counted
"type next", e.g like `[]` `*`, that says rather than an 
attribute created _just to have the library solution to work_.
Nov 15 2021
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.11.21 08:05, Walter Bright wrote:

 On 11/14/2021 1:12 PM, Steven Schveighoffer wrote:

 First, I'll note that `__mutable` on its own is perfectly sound.


 
 I don't think so. It produces two kinds of immutability - one with 
 mutable members, one with not. Different rules apply.
 
 But then what do you do with opaque types? The compiler doesn't know 
 which kind of immutable they are. So it is forced to go with the worst 
 case - immutables are mutable. And it all falls apart.
 


My answer to this was nondeterministic semantics. Just say what all the 
rewrites are that the compiler is allowed to do. Then  trusted __mutable 
functions have to ensure they function correctly _even if_ optimizations 
change the specific __mutable function calls and their arguments.

Another example of nondeterministic semantics is implicit moving. The 
compiler can either move directly or just copy construct and destroy. 
Then, the type itself has to ensure it does not matter for overall 
correctness which specific set of operations is applied even if they do 
not result in exactly the same bit patterns. This works and similar 
nondeterministic semantics can be made to work for __mutable data.

Of course, this is more technically involved than just saying 
"`immutable` means immutable" (a rare feat for D keywords), but you 
still have to address the issue of deallocation anyway.
Nov 15 2021
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/15/21 2:05 AM, Walter Bright wrote:

 On 11/14/2021 1:12 PM, Steven Schveighoffer wrote:

 First, I'll note that `__mutable` on its own is perfectly sound.


 
 I don't think so. It produces two kinds of immutability - one with 
 mutable members, one with not. Different rules apply.


I mean it's sound in that the type checking is valid. You are allowed to 
convert mutable to mutable, so having an always-mutable island is a 
valid. Whether it's *confusing* or not is a different story, and that 
part I agree with you.

Without additional type qualifiers, confusion would always remain.

-Steve
Nov 15 2021
prev sibling parent sclytrack <fake hotmail.com> writes:
On Friday, 12 November 2021 at 12:31:03 UTC, Steven Schveighoffer 
wrote:

 One of the prerequisites to doing reference counting is to have 
 a mutable piece of data inside an immutable piece of data.

 -Steve



I've been thinking and the origin of a reference counted type 
should be mutable or
const and never immutable. Make it illegal to create immutable 
reference counted types.

Then Andrei can apply its cheat codes to const and only const. 
Never to immutable.

So const(RCType) and never immutable(RCType);

I believe there is const and shared const. So immutable remains 
immutable forever
and ever and ever.
Nov 30 2021