www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Rebooting the __metada/__mutable discussion

reply RazvanN <razvan.nitu1305 gmail.com> writes:
Hello everyone,

Lately, I've been looking at the __metadata/__mutable 
discussions. For reference, here is the material I have so far:

1. Forum discussion: 
https://forum.dlang.org/thread/3f1f9263-88d8-fbf7-a8c5-b3a2a5224ce0 erdani.org?page=1

2. Timon's original DIP: 
https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/timon_dip.md

3. My updated DIP which is based on Timon's: 
https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md

We need this to be able to implement generic reference counting. 
So our main
problem is how do we reference count immutable/const objects. 
Timon's original
proposal tried to implement __metadata in a way that does not 
affect purity
based optimizations. I think that this could be done:

1. A strongly pure function (that return types without 
indirections) will return the same result when applied to the 
same immutable arguments.

Fixed by rewritting

```d
auto a = foo(arg)   // foo -> strongly pure
aubo b = foo(arg)
````

to

```d
auto a = foo(arg)
auto b = a
```

This is taken from Timon DIP.

2. The set of references returned from strongly pure functions 
can be safely converted to immutable or shared.

This is no affected by the introduction of __metadata.

3. A strongly pure function whose result is not used may be 
safely elided.

```d
struct S{
     private int __metadata x;
}

void foo(immutable ref S s)pure{
     s.x += 1;
}

void main(){
     immutable S s;
     foo(s); // there is no reason for this call to happen
     assert(s.x==1); // can't rely on this, it might also be 0
}
```

Essentially, if `foo` is strongly pure, then the compiler can 
optimize away
the call to it and your reference count is blown away. If we look 
at it this
way, the problem seems to be unsolvable. However, the idea of 
__metadata is to
be used solely by library developers and even they should take 
extra care.
As such, I would propose that `__metadata` can only be accessed 
from inside
the aggregate that defines it (`private` here means `Java 
private`) and methods
that access __metadata directly need to also private. I think 
that this makes sense since the reference is updated only when 
you call the copy constructor and the assignment operator. These 
methods should be public and they can call the incRef, decRef 
that are mandatory private. This way, it becomes impossible to 
access a `__metadata` field without going through the object 
methods. This makes sense, since the object is the only one that 
should manage the `__metadata`.

Now, if we do it like this, then `foo` will not have any means of 
accessing `x`
apart from assigning s, passing s to a function or copy 
constructing from s.
However, whatever happens to s, once the execution of `foo` is 
over, the reference
count at the call site is going to be the same as when `foo` was 
called. Why?
Because there is no way you can escape any reference to s outside 
of a strongly pure function (other than returning it, but that 
case is taken care of at point 1.).

4. If we have two subsequent pure function invocations 
foo(args1...) and bar(args2...) where data transitively reachable 
from args1 and args2 only overlaps in immutable and const data 
(this includes data accessed through __mutable fields of const 
and immutable objects), the two invocations may safely swap their 
order (of course, this only applies if none of the two functions 
takes arguments)

This should be un-affected.

5. A strongly pure function invocation can always exchange order 
with an adjacent impure function invocation.

This should be un-affected.

What do you think? Am I missing anything? If you think this could 
fly, I could update the DIP and submit it.

Best regards,
RazvanN
Apr 06 2022
next sibling parent reply IGotD- <nise nise.com> writes:
On Wednesday, 6 April 2022 at 09:41:52 UTC, RazvanN wrote:

 We need this to be able to implement generic reference 
 counting. So our main
 problem is how do we reference count immutable/const objects. 
 Timon's original
 proposal tried to implement __metadata in a way that does not 
 affect purity
 based optimizations. I think that this could be done:


Immutable and mutable variables don't mix well at all. With 
immutable there is a high probability that it ends up in read 
only memory so any poking around there will cause the program to 
crash. Also reference counting with immutable objects doesn't 
make any sense.

When it comes changing data on const objects such as reference 
counting, I think that this is low level code that works under 
the hood of the actual language. This means that you will cast 
away whatever you need in order to get the job done.
Apr 06 2022
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 06/04/2022 11:43 PM, IGotD- wrote:

 On Wednesday, 6 April 2022 at 09:41:52 UTC, RazvanN wrote:

 We need this to be able to implement generic reference counting. So 
 our main
 problem is how do we reference count immutable/const objects. Timon's 
 original
 proposal tried to implement __metadata in a way that does not affect 
 purity
 based optimizations. I think that this could be done:


 
 Immutable and mutable variables don't mix well at all. With immutable 
 there is a high probability that it ends up in read only memory so any 
 poking around there will cause the program to crash. Also reference 
 counting with immutable objects doesn't make any sense.
 
 When it comes changing data on const objects such as reference counting, 
 I think that this is low level code that works under the hood of the 
 actual language. This means that you will cast away whatever you need in 
 order to get the job done.


There are alternatives: 
https://forum.dlang.org/post/t2eo0t$mg7$1 digitalmars.com

It is my opinion that a purpose built escape hatch will be better than a 
generic escape hatch. Far less code that you need to review.
Apr 06 2022
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 06.04.22 13:43, IGotD- wrote:

 On Wednesday, 6 April 2022 at 09:41:52 UTC, RazvanN wrote:

 We need this to be able to implement generic reference counting. So 
 our main
 problem is how do we reference count immutable/const objects. Timon's 
 original
 proposal tried to implement __metadata in a way that does not affect 
 purity
 based optimizations. I think that this could be done:


 
 Immutable and mutable variables don't mix well at all. With immutable 
 there is a high probability that it ends up in read only memory so any 
 poking around there will cause the program to crash.


The decision whether or not to place `immutable`-qualified data in 
read-only memory is made at a point where the memory layout of that type 
is known. Just don't do it if there are any `__mutable` fields.


 Also reference counting with immutable objects doesn't make any sense.
 ...


That's not so clear.


 When it comes changing data on const objects such as reference counting, 
 I think that this is low level code that works under the hood of the 
 actual language.


Yes, this is all in the realm of  system code.


 This means that you will cast away whatever you need in 
 order to get the job done.
 


That's disallowed by existing language rules, hence the need to make 
some modifications to the specification.
Apr 06 2022
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 06.04.22 11:41, RazvanN wrote:

 As such, I would propose that `__metadata` can only be accessed from inside
 the aggregate that defines it (`private` here means `Java private`) and methods
 that access __metadata directly need to also private.


Those rules are pretty arbitrary and don't buy much. However, accessing 
`__metadata` should be ` system`.


 
 What do you think? Am I missing anything?


E.g., how to deallocate memory in a pure destructor? (Solved by 
`__mutable` functions in my original draft.)

What if the destructor is both pure and immutable?

More generally, there needs to be a story for how to support custom 
allocators for immutable memory.
Apr 06 2022
parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Wednesday, 6 April 2022 at 18:03:25 UTC, Timon Gehr wrote:

 On 06.04.22 11:41, RazvanN wrote:

 As such, I would propose that `__metadata` can only be 
 accessed from inside
 the aggregate that defines it (`private` here means `Java 
 private`) and methods
 that access __metadata directly need to also private.


 Those rules are pretty arbitrary and don't buy much. However, 
 accessing `__metadata` should be ` system`.


They buy the fact that __metadata fields cannot be accessed from 
outside
of the object that implements the reference count. This offers 
the guarantee
that a strongly pure function will not alter a __metadata field 
and hence it
can be the subject of any purity-based optimization (with the 
exception of
functions that do deallocations).


 
 What do you think? Am I missing anything?


 E.g., how to deallocate memory in a pure destructor? (Solved by 
 `__mutable` functions in my original draft.)

 What if the destructor is both pure and immutable?


Destructors suffer from the same issue as postblits did with 
regards to qualifiers. I would be surprised if you could ever 
call an immutable destructor. Combined with the fact that you 
cannot overload the destructor makes everything worse. So I would 
argue that talking about immutable pure destructors in this 
context is like talking about a broken glass when you're in a 
house on fire.

However, I do get your point and in order to support immutable 
allocators we do need to take into consideration how allocation 
and deallocation is done.

I think that the problem here is that we are conflating a 
high-level concept (purity) with low level operations 
(allocation/deallocation). The latter category is essentially not 
pure because it modifies operating system data structures, so I 
would say that we could view them as system code, not in the 
sense that they are unsafe (although they could be), but because 
they require OS assistance. Now, this sort of functions should be 
at most weakly pure (and this is what mutable functions 
essentially did) even though their signature looks as if they 
were strongly pure. One solution would be to consider 
 system/ trusted pure functions as weakly pure no matter what 
their signature looks like. Constructors/destructors should also 
be considered at most weakly pure. If your function is  safe and 
pure then the signature should be analyzed to decide if it is 
strongly or weakly pure. Also, any function that you are calling 
from another language should be at most trusted. This way, 
 trusted would act as an optimization blocker.



 More generally, there needs to be a story for how to support 
 custom allocators for immutable memory.
Apr 07 2022
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 07/04/2022 8:51 PM, RazvanN wrote:

 I think that the problem here is that we are conflating a high-level 
 concept (purity) with low level operations (allocation/deallocation). 
 The latter category is essentially not pure because it modifies 
 operating system data structures, so I would say that we could view them 
 as system code, not in the sense that they are unsafe (although they 
 could be), but because they require OS assistance.


I've had to argue with myself on if memory mapping should be viewed as 
pure or not. During development of my own allocators.

Ultimately the argument I came up with is that we have already defined 
memory mapping in the form of new as being pure. I disagree that it 
should be defined as such, but it has been.

As such I have classified memory mapping as being a set of "deity" 
functions. Things that modify the execution environment but not the 
logic being executed therefore can be considered pure.
Apr 07 2022
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 07.04.22 19:40, rikki cattermole wrote:

 
 On 07/04/2022 8:51 PM, RazvanN wrote:

 I think that the problem here is that we are conflating a high-level 
 concept (purity) with low level operations (allocation/deallocation). 
 The latter category is essentially not pure because it modifies 
 operating system data structures, so I would say that we could view 
 them as system code, not in the sense that they are unsafe (although 
 they could be), but because they require OS assistance.


 
 I've had to argue with myself on if memory mapping should be viewed as 
 pure or not. During development of my own allocators.
 
 Ultimately the argument I came up with is that we have already defined 
 memory mapping in the form of new as being pure. I disagree that it 
 should be defined as such, but it has been.
 ...


That's just the wrong level of abstraction. Clearly creating new values 
should be pure, you can create arbitrary tree structures in basically 
all purely functional programming languages. It's usually the very core 
of how such code operates... The justification for `pure` in D is to be 
able to somewhat compete with such languages.

Whether allocation/memory mapping itself should be `pure` is debatable. 
It certainly can't be `pure` if allocation failure is not fatal or it 
returns uninitialized memory that can be accessed.

However, `new` (memory mapping + initialization + construction) is not 
setting any precedent for that, it's more high level.


 As such I have classified memory mapping as being a set of "deity" 
 functions. Things that modify the execution environment but not the 
 logic being executed therefore can be considered pure.


There is no issue with allocation plus construction as long as none of 
the allocator state can leak into the observable behavior. (E.g., 
peeking into the bits of pointers should be impure.) Deallocation needs 
to be treated specially in any case though.
Apr 07 2022
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 07.04.22 10:51, RazvanN wrote:

 On Wednesday, 6 April 2022 at 18:03:25 UTC, Timon Gehr wrote:

 On 06.04.22 11:41, RazvanN wrote:

 As such, I would propose that `__metadata` can only be accessed from 
 inside
 the aggregate that defines it (`private` here means `Java private`) 
 and methods
 that access __metadata directly need to also private.


 Those rules are pretty arbitrary and don't buy much. However, 
 accessing `__metadata` should be ` system`.


 They buy the fact that __metadata fields cannot be accessed from outside
 of the object that implements the reference count.


Not really, e.g., just expose a pointer to it. I really don't see why we 
need special-case visibility rules just for this.

Note that reference counts are not the only use case. You can also do 
e.g., lazy initialization.


 This offers the 
 guarantee
 that a strongly pure function will not alter a __metadata field


Not really.


 and hence it
 can be the subject of any purity-based optimization


That's backwards. The purity-based optimizations determine what you are 
allowed to do to `__metadata` fields. Because you shouldn't do arbitrary 
stuff there it should be ` system`, as it's up to the programmer to 
uphold guarantees here, not the compiler. It's a low-level feature.


 (with the exception of functions that do deallocations).
 ...


Note that this means you have to mark functions that are used from 
destructors specially.


 What do you think? Am I missing anything?


 E.g., how to deallocate memory in a pure destructor? (Solved by 
 `__mutable` functions in my original draft.)

 What if the destructor is both pure and immutable?


 
 Destructors suffer from the same issue as postblits did with regards to 
 qualifiers. I would be surprised if you could ever call an immutable 
 destructor.


Well, what I meant is what happens if you destruct an immutable object. 
There need to be some language rules that allow you to have memory that 
is only temporarily immutable, otherwise your reference counting scheme 
will never work for immutable memory anyway.


 Combined with the fact that you cannot overload the 
 destructor makes everything worse. So I would argue that talking about 
 immutable pure destructors in this context is like talking about a 
 broken glass when you're in a house on fire.
 
 However, I do get your point and in order to support immutable 
 allocators we do need to take into consideration how allocation and 
 deallocation is done.
 
 I think that the problem here is that we are conflating a high-level 
 concept (purity) with low level operations (allocation/deallocation). 


It's not really a conflation, it's that you implement the high-level 
concept in terms of low-level operations. This is not problematic, this 
is how it always works.


 The latter category is essentially not pure because it modifies 
 operating system data structures, so I would say that we could view them 
 as system code, not in the sense that they are unsafe (although they 
 could be), but because they require OS assistance.


They are unsafe because you need them to behave in a certain way for it 
to be possible to consider them `pure`.


 Now, this sort of 
 functions should be at most weakly pure (and this is what mutable 
 functions essentially did) even though their signature looks as if they 
 were strongly pure.


I don't think this has much to do with weakly vs strongly pure. It's a 
different category.


 One solution would be to consider  system/ trusted 
 pure functions as weakly pure no matter what their signature looks like. 
 Constructors/destructors should also be considered at most weakly pure. 
 If your function is  safe and pure then the signature should be analyzed 
 to decide if it is strongly or weakly pure. Also, any function that you 
 are calling from another language should be at most trusted. This way, 
  trusted would act as an optimization blocker.
 ...


There is no good reason whatsoever why  trusted or  system should block 
optimizations. Orthogonal language design, please. :(
Apr 07 2022
prev sibling next sibling parent reply Zach Tollen <zach mystic.yeah> writes:
On Wednesday, 6 April 2022 at 09:41:52 UTC, RazvanN wrote:

 What do you think? Am I missing anything? If you think this 
 could fly, I could update the DIP and submit it.


It strikes me that this DIP is eerily similar to the ` system` 
variables described in 
[DIP1035](https://github.com/dlang/DIPs/blob/72f41cffe68ff1f2d4c033b5728ef37e28246
dd/DIPs/DIP1035.md) which is in the final phase of the review process. It seems
perfectly possible to merge the characteristics of `__metadata` as described
here, with the characteristics of ` system` variables in that DIP.

The effect would be that basically, you can do *anything* with 
` system` variables — except access them in ` safe` code, which 
is precisely the point. You can violate the type system. You have 
total freedom. And that's that.

I suppose the downside is that one might not *desire* to have all 
that freedom. But I'm trying to think of any situation where you 
would want a ` system` variable to be immutable. The whole point 
of both ` system` and `__metadata` in the two DIPs, is that 
` system/__meta` data is mutable. (With DIP1035, mutability is 
essential to what makes a given piece of data dangerous enough 
require being marked ` system`. And the whole point of 
`__metadata` is to force mutability bypassing the type system.)

-- Zach
Apr 07 2022
parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 8 April 2022 at 00:37:15 UTC, Zach Tollen wrote:

 (With DIP1035, mutability is essential to what makes a given 
 piece of data dangerous enough require being marked ` system`. 
 And the whole point of `__metadata` is to force mutability 
 bypassing the type system.)


This is not entirely true. DIP 1035 also specifies that variables 
whose initial values would not be allowed in ` safe` code are 
inferred as ` system`, and this applies equally to mutable and 
`immutable` variables.
Apr 07 2022
parent reply Zach Tollen <zach mystic.yeah> writes:
On Friday, 8 April 2022 at 01:24:38 UTC, Paul Backus wrote:

 On Friday, 8 April 2022 at 00:37:15 UTC, Zach Tollen wrote:

 (With DIP1035, mutability is essential to what makes a given 
 piece of data dangerous enough require being marked ` system`. 
 And the whole point of `__metadata` is to force mutability 
 bypassing the type system.)


 This is not entirely true. DIP 1035 also specifies that 
 variables whose initial values would not be allowed in ` safe` 
 code are inferred as ` system`, and this applies equally to 
 mutable and `immutable` variables.


So it's dangerous enough to be pointing to undefined memory, but 
it still needs the protection of immutable because... some 
` system` programmer might want it to point to something else... 
and we can't allow that to happen, because ` system` programmers 
shouldn't be allow to do something like that?

In other words, if it's dangerous enough to start out as 
` system`, how much additional danger are we adding by making it 
mutable?
Apr 07 2022
parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 8 April 2022 at 01:46:35 UTC, Zach Tollen wrote:

 On Friday, 8 April 2022 at 01:24:38 UTC, Paul Backus wrote:

 This is not entirely true. DIP 1035 also specifies that 
 variables whose initial values would not be allowed in ` safe` 
 code are inferred as ` system`, and this applies equally to 
 mutable and `immutable` variables.


 So it's dangerous enough to be pointing to undefined memory, 
 but it still needs the protection of immutable because... some 
 ` system` programmer might want it to point to something 
 else... and we can't allow that to happen, because ` system` 
 programmers shouldn't be allow to do something like that?

 In other words, if it's dangerous enough to start out as 
 ` system`, how much additional danger are we adding by making 
 it mutable?


You're not adding any additional *danger*, I suppose, but it 
would be a non-obvious special case in the type system to have 
the ` system` attribute on variables interact with `immutable` 
like this. As currently proposed by DIP 1035, they are completely 
independent of each other.
Apr 07 2022
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08.04.22 04:32, Paul Backus wrote:

 On Friday, 8 April 2022 at 01:46:35 UTC, Zach Tollen wrote:

 On Friday, 8 April 2022 at 01:24:38 UTC, Paul Backus wrote:

 This is not entirely true. DIP 1035 also specifies that variables 
 whose initial values would not be allowed in ` safe` code are 
 inferred as ` system`, and this applies equally to mutable and 
 `immutable` variables.


 So it's dangerous enough to be pointing to undefined memory, but it 
 still needs the protection of immutable because... some ` system` 
 programmer might want it to point to something else... and we can't 
 allow that to happen, because ` system` programmers shouldn't be allow 
 to do something like that?

 In other words, if it's dangerous enough to start out as ` system`, 
 how much additional danger are we adding by making it mutable?


 
 You're not adding any additional *danger*, I suppose, but it would be a 
 non-obvious special case in the type system to have the ` system` 
 attribute on variables interact with `immutable` like this. As currently 
 proposed by DIP 1035, they are completely independent of each other.


Yes. I do not understand why so many people are so keen to conflate 
entirely different things into the same concepts. That's just bad 
language design.
Apr 08 2022
parent reply Zach Tollen <zach mystic.yeah> writes:
On Friday, 8 April 2022 at 08:23:34 UTC, Timon Gehr wrote:

 Yes. I do not understand why so many people are so keen to 
 conflate entirely different things into the same concepts. 
 That's just bad language design.


No it's not. Bad language design is when you *arbitrarily* 
conflate different things without first examining the pros and 
cons. But it's also bad language design not to be open to the 
possibility of a conflation of different things which ends up 
being harmonious and easy to explain.
Apr 08 2022
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08.04.22 17:07, Zach Tollen wrote:

 On Friday, 8 April 2022 at 08:23:34 UTC, Timon Gehr wrote:

 Yes. I do not understand why so many people are so keen to conflate 
 entirely different things into the same concepts. That's just bad 
 language design.


 
 No it's not.


Of course it is. It hampers expressiveness while adding incidental 
complexity. Lose-lose.


 Bad language design is when you *arbitrarily* conflate 
 different things without first examining the pros and cons.


The language should allow me to state what I mean without adding random 
additional baggage.


 But it's 
 also bad language design not to be open to the possibility of a 
 conflation of different things which ends up being harmonious and easy 
 to explain.


That's just never how it works out. You should only conflate things that 
are actually the same thing.
Apr 08 2022
parent reply Zach Tollen <zach mystic.yeah> writes:
On Friday, 8 April 2022 at 17:28:30 UTC, Timon Gehr wrote:

 On 08.04.22 17:07, Zach Tollen wrote:

 On Friday, 8 April 2022 at 08:23:34 UTC, Timon Gehr wrote:

 Yes. I do not understand why so many people are so keen to 
 conflate entirely different things into the same concepts. 
 That's just bad language design.


 
 No it's not.


 Of course it is. It hampers expressiveness while adding 
 incidental complexity. Lose-lose.


Wrong. Simplicity is fine when it allows you to do what you want 
without forcing you to do what you don't want. If what you say 
were true, then D should have adopted every single built-in 
attribute and every type qualifier which has ever been suggested, 
because the lack of them would "hamper expressiveness."

The incidental complexity of a given multi-purpose feature only 
arises when the one purpose of the feature simultaneously makes 
another purpose of the same feature difficult to achieve. I do 
not consider this an inevitable outcome. I want to examine the 
specific case instead.


 But it's also bad language design not to be open to the 
 possibility of a conflation of different things which ends up 
 being harmonious and easy to explain.


 That's just never how it works out. You should only conflate 
 things that are actually the same thing.


The use cases of DIP1035, for the most part, are a strict subset 
of the uses of `__mutable`/`__metadata`. It is agreed that you 
can only access `__metadata` variables from ` system` code. 
Therefore by implementing `__metadata` you basically get DIP1035 
for free. My proposal is simply to use DIP1035's word ` system` 
two encompass the two ideas.

The purpose of DIP1035 is to be able to force a variable to be 
considered unsafe, because it is already being used in an unsafe 
way. If you're trying to decide whether merging ` system` and 
`__metadata` is a good idea, you would first need to come up with 
a use case proving a possible conflict of purposes as follows:

You would need to show how marking a variable ` system` because 
it is *already* being used dangerously might then lead to it 
being used dangerously *differently*, but now in an unintended 
and undesirable way, solely because before, when it was used 
dangerously, it was immutable.

It seems like a really tall order to me. And if you can't provide 
an example like that, then on what basis can you say that these 
two features can't be combined?

My main point is that mutability, in nearly all cases, is 
precisely what would motivate a variable to require being marked 
` system`. As to global data which is initialized to unsafe 
values and therefore inferred ` system`, I don't even know what 
these would be for. They seem like they would always be bugs to 
me.
Apr 08 2022
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 4/9/22 02:56, Zach Tollen wrote:

 On Friday, 8 April 2022 at 17:28:30 UTC, Timon Gehr wrote:

 On 08.04.22 17:07, Zach Tollen wrote:

 On Friday, 8 April 2022 at 08:23:34 UTC, Timon Gehr wrote:

 Yes. I do not understand why so many people are so keen to conflate 
 entirely different things into the same concepts. That's just bad 
 language design.


 No it's not.


 Of course it is. It hampers expressiveness while adding incidental 
 complexity. Lose-lose.


 
 Wrong. Simplicity is fine


Your proposal is not simpler.


 when it allows you to do what you want without 
 forcing you to do what you don't want.


Your proposal forces me to do what I don't want without allowing me to 
do what I want. I don't want that.

Your proposal fails to meet your own standards.


 If what you say were true, then D 
 should have adopted every single built-in attribute and every type 
 qualifier which has ever been suggested, because the lack of them would 
 "hamper expressiveness."
 ...


I see what you did there. I can do that too: Clearly you think that 
simplicity and expressiveness are both monotone functions of the number 
of attributes that are in the language. If what you say were true, we 
should conflate  safe and pure, as well as  nogc and nothrow because 
that would be "simpler". That's ridiculous!


 The incidental complexity of a given multi-purpose feature only arises 
 when the one purpose of the feature simultaneously makes another purpose 
 of the same feature difficult to achieve. I do not consider this an 
 inevitable outcome. I want to examine the specific case instead.
 

 But it's also bad language design not to be open to the possibility 
 of a conflation of different things which ends up being harmonious 
 and easy to explain.


 That's just never how it works out. You should only conflate things 
 that are actually the same thing.


 
 The use cases of DIP1035, for the most part, are a strict subset of the 
 uses of `__mutable`/`__metadata`.


???

`__metadata` is _much more niche_ and _much more restricted_. Almost no 
D programmer will have to use it directly.


 It is agreed that you can only access 
 `__metadata` variables from ` system` code. Therefore by implementing 
 `__metadata` you basically get DIP1035 for free.


No. I suggest you go read DIP1035 and find all the aspects not covered 
by `__metadata`.


 My proposal is simply to use DIP1035's word ` system` two encompass the two
ideas.
 ...


So your proposal is to take the ` system` and `__metadata` proposal, 
concatenate them and change all occurrences of `__metadata` to 
` system`? That would be even less sensible than what I thought you were 
arguing for, so I guess your proposal is actually not "simply" this. One 
does not simply conflate ` system` and `__metadata`.

In any case, even if your proposal was simple to state formally, it 
would still lead to really bad outcomes.


 The purpose of DIP1035 is to be able to force a variable to be 
 considered unsafe, because it is already being used in an unsafe way. If 
 you're trying to decide whether merging ` system` and `__metadata` is a 
 good idea, you would first need to come up with a use case proving a 
 possible conflict of purposes as follows:
 
 You would need to show


You are making a proposal.


 how marking a variable ` system` because it is 
 *already* being used dangerously might then lead to it being used 
 dangerously *differently*, but now in an unintended and undesirable way, 
 solely because before, when it was used dangerously, it was immutable.
 ...


So you want to argue that everything that's potentially dangerous is 
actually the same thing? Can as well just turn off type checking and 
bounds checks in ` system` code completely at that point. It's such 
fallacious reasoning...


 It seems like a really tall order to me.


To the extent that I can make sense of your paragraph, it's very easy 
and Paul has already done it.

NB: I really dislike it when some people go from "there are no 
counterexamples" to "there is exactly one counterexample and I don't 
care about that particular one" after they have been presented a 
counterexample to their claims. It makes me not want to engage with 
their posts anymore. It's so disrespectful to other people's time. If 
you thought there were no counterexamples and were presented one, 
chances are that you also missed other things.


 And if you can't provide an 
 example like that, then on what basis can you say that these two 
 features can't be combined?
 ...


I am not interested in that particular scenario as I can provide 
multiple examples that immediately kill your proposal. However, it seems 
it's not worth the effort as you will just move the goalposts with 
informal nonsense such as "for the most part" and "in nearly all cases" 
or "used dangerously". The qualifiers have precise formal meanings.


 My main point is that mutability, in nearly all cases, is precisely what 
 would motivate a variable to require being marked ` system`.


Even in cases where mutability is the issue you should not simply be 
able to bypass `const` implicitly/by accident, even in ` system` code...


 As to global data which is initialized to unsafe values and therefore inferred 
 ` system`, I don't even know what these would be for. They seem like 
 they would always be bugs to me.


I agree with the point that it seems that you don't know existing 
language rules and use cases well enough.

Anyway, for anyone who's actually interested, here are some of the 
problems with the proposal (very likely not exhaustive, it's such an 
arbitrary and badly motivated conflation that we'd probably find new and 
horrible consequences for years to come. I have already spent more than 
two hours on this post and don't want to think about this any more, it's 
such a pointless waste of time):

- There is nothing about ` system` that screams "this is mutable", it's 
a very unintuitive language rule. You would literally state ` system 
immutable` and it would be ignored.

- ` system` variables will be a reasonably common occurrence in 
low-level code. Explicit `__mutable` is much more specialized and 
restricted in how it can be applied. It does not make sense to extend 
` system` to encompass this much more narrow use case as most ` system` 
users won't want it.

- ` system` can be inferred from an initializer. It is then _very 
confusing_, if `const`/`immutable` are just stripped away from the 
variable declaration, _especially_ if it does not happen often.

- By conflating `__metadata` and ` system`, the compiler seems to engage 
in implicit, unsafe and unnecessary type-punning. `immutable` would be 
stripped away _transitively_ from an _existing value_ upon 
initialization. There is possibly another design where ` system t = 
foo()` would just not compile because you can't assign the `immutable` 
result of `foo()` to the mutable `t`, but that makes similarly little sense.

- Templated code may have logic that differs based on the mutability of 
the argument (e.g., copy the values of all mutable fields from one 
instance to another).

- Variables inside templates may not always be inferred ` system` the 
same way for different template arguments. Now suddenly you sometimes 
get completely unsafe, completely implicit type-punning, potentially you 
can't even tell from the outside. If you are "lucky", your code might 
not compile because you can't assign a `T v = init();` back to a `T w` 
or something.

- Memory safety may actually depend on a ` system` field keeping its 
value, so it's useful to have `immutable  system` fields.

- There can be mutable and immutable instances of the same type, some of 
them static. If a type has ` system` fields, the proposal would prevent 
immutable instances from being stored in the read-only data segment.

- Other optimizations based on `immutable` similarly still work even 
with ` system` values. (Unless ` system` is arbitrarily conflated with 
`__metadata`.) You should not have to choose between fast code and 
soundness of ` safe` checks.
Apr 09 2022
parent reply Zach Tollen <zach mystic.yeah> writes:
On Saturday, 9 April 2022 at 17:22:38 UTC, Timon Gehr wrote:

 Your proposal is not simpler.


It is simpler. The only question is whether that simplicity comes 
at too high a cost.


 Your proposal fails to meet your own standards.


I don't know if it meets my standards or not. I would need an 
example where a variable which was either explicitly marked, or 
inferred ` system`, and implemented as desired with that label, 
then changed its behavior as a result of also implicitly having 
the privileges of `__mutable`.


 I see what you did there. I can do that too: Clearly you think 
 that simplicity and expressiveness are both monotone functions 
 of the number of attributes that are in the language. If what 
 you say were true, we should conflate  safe and pure, as well 
 as  nogc and nothrow because that would be "simpler". That's 
 ridiculous!


No I'm saying that simplicity and expressiveness are tradeoffs. 
The Go language, for example, traded a lot of expressiveness for 
simplicity. That doesn't make it bad language design. It means 
they weighed the pros and the cons, and they made a decision.


 The use cases of DIP1035, for the most part, are a strict 
 subset of the uses of `__mutable`/`__metadata`.


 ???


All `__metadata` variables qualify as DIP1035 ` system`. If you 
added ` system` to any instance of `__metadata`, it would have no 
effect. Therefore, if only `__metadata` were implemented, users 
could utilize it to achieve the intended goals of DIP1035. (Just 
allow them to mark it wherever they wanted instead of confining 
it to aggregate members, even though most of the use cases are in 
fact aggregate members.) The question is simply whether 
additionally being able to violate the type system would cause 
intractable problems. I don't know. I suspect that almost all use 
cases of ` system` are mutable to begin with. And if they 
weren't, I would have to look the workarounds, to see if issues 
of having undesired `__metadata` functionality caused more pain 
than the simplicity of using only one keyword merited.


 So your proposal is...


My proposal is to take everything DIP1035 marks or infers as 
` system` and extend `__metadata` functionality to it. This may 
sound dangerous. The logic is simply that this is ` system` code, 
and ` system` code is dangerous. But it is also powerful. That's 
the overarching idea. Both dangerous and powerful. (D should also 
be working towards ` safe` by default, of course.)

Again, the only question is whether having `__metadata` 
functionality will cause intractable pain to the use cases of 
` system` which don't require it.


 In any case, even if your proposal was simple to state 
 formally, it would still lead to really bad outcomes.


The difference between us is that you are sure of that, and I am 
not.


 To the extent that I can make sense of your paragraph, it's 
 very easy and Paul has already done it.

 NB: I really dislike it when some people go from "there are no 
 counterexamples" to "there is exactly one counterexample and I 
 don't care about that particular one" after they have been 
 presented a counterexample to their claims. It makes me not 
 want to engage with their posts anymore. It's so disrespectful 
 to other people's time. If you thought there were no 
 counterexamples and were presented one, chances are that you 
 also missed other things.


Give me an example of a global variable which is intended to be 
immutable, and which is also initialized to point to undefined 
memory, which isn't a bug. assuming it's not a bug, since it is 
meant to be immutable, then this is one case where the conflation 
of ` system` and `__metadata` might be risky. The case could be 
ameliorated by the fact that you can't access it in ` safe` code. 
What I'm saying is that I suspect actual cases of this will be 
rare, that they may not outweigh the benefits of having a very 
simple system, with one keyword to encompass all unsafe activity.


 ...


The solution to our debate — insofar as my claims deserve any 
merit at all — is to implement both DIP1035 and the `__metadata` 
DIP, with the additional requirement that all uses of 
`__metadata` be marked ` system` as well. Semantically, it would 
change nothing, and it would actually be easier to understand for 
newcomers.

Then, five years later, to test my theory, add a compiler switch 
`preview=deprecate__metadata`, put the `__metadata` functionality 
into the ` system`, and make `__metadata` a no-op.

If nothing breaks, I was right. If something breaks, and the 
workarounds aren't seamless, then you were right.

Zach
Apr 09 2022
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 10 April 2022 at 00:04:43 UTC, Zach Tollen wrote:

 On Saturday, 9 April 2022 at 17:22:38 UTC, Timon Gehr wrote:


I've read your posts, and the replies from Timon.  Since Timon's 
careful and clear posts did not reach you, I'll summarize my take 
on your posts below rather than attempt a similarly detailed 
correction/illumination of your latest (which would, most 
probably, be a poor use of time for both of us, certainly for me)

Summary response:

I find your posts to be very enthusiastic but also (very) poorly 
reasoned.  I'd suggest working on that with someone you look up 
to.  It's very hard to improve on your own if you don't, 
evidently, understand that you could and should do much better.
Apr 09 2022
parent reply Zach Tollen <zach mystic.yeah> writes:
On Sunday, 10 April 2022 at 03:37:03 UTC, Bruce Carneal wrote:

 On Sunday, 10 April 2022 at 00:04:43 UTC, Zach Tollen wrote:

 On Saturday, 9 April 2022 at 17:22:38 UTC, Timon Gehr wrote:


 I've read your posts, and the replies from Timon.  Since 
 Timon's careful and clear posts did not reach you, I'll 
 summarize my take on your posts below rather than attempt a 
 similarly detailed correction/illumination of your latest 
 (which would, most probably, be a poor use of time for both of 
 us, certainly for me)

 Summary response:

 I find your posts to be very enthusiastic but also (very) 
 poorly reasoned.  I'd suggest working on that with someone you 
 look up to.  It's very hard to improve on your own if you 
 don't, evidently, understand that you could and should do much 
 better.


Well thanks for your interest, at any rate. And your honesty.

Out of curiosity, do you have an opinion of 
[DIP1035](https://github.com/dlang/DIPs/blob/72f41cffe68ff1f2d4c033b5728ef37e282461
d/DIPs/DIP1035.md)? At least intuitively, I feel like the essence of it should
run: "Some variables are dangerous, but the compiler can't figure that out.
Therefore we allow those variables to be marked ` system` so they are treated
as unsafe."

But there are other parts of DIP1035 (initializing globals to 
unsafe values, and also, an entire aggregate is unsafe if a 
single member is unsafe) which seem to be about *inferring* 
certain things to be unsafe as opposed to helping the compiler to 
*determine* what is unsafe. My interest in merging the two DIPS 
came almost entirely from seeing the value in the "help the 
compiler learn what is unsafe" side, and I thought the "inferring 
things unsafe" side could be handled separately. (The 
helping-the-compiler side is eloquently illustrated in the one of 
the DIP's linked [videos](https://youtu.be/O3TO52rXLug) by Steven 
Schveighoffer. That's where I got my understanding of what I 
thought the essence of DIP1035 *should* be about. Of course, 
that's no reason to arbitrarily merge the two features, ` system` 
and `__metadata`. But it might make it easier to explain why I 
thought it was possible.)
Apr 09 2022
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 10 April 2022 at 04:12:08 UTC, Zach Tollen wrote:

 On Sunday, 10 April 2022 at 03:37:03 UTC, Bruce Carneal wrote:

 [...]


 Well thanks for your interest, at any rate. And your honesty.

 Out of curiosity, do you have an opinion of 
 [DIP1035](https://github.com/dlang/DIPs/blob/72f41cffe68ff1f2d4c033b5728ef37e282461
d/DIPs/DIP1035.md)? At least intuitively, I feel like the essence of it should
run: "Some variables are dangerous, but the compiler can't figure that out.
Therefore we allow those variables to be marked ` system` so they are treated
as unsafe."


I view 1035 as a mechanism to extend the reach of  safe, to 
reduce the load on conscientious code reviewers.


 But there are other parts of DIP1035 (initializing globals to 
 unsafe values, and also, an entire aggregate is unsafe if a 
 single member is unsafe) which seem to be about *inferring* 
 certain things to be unsafe as opposed to helping the compiler 
 to *determine* what is unsafe. My interest in merging the two 
 DIPS came almost entirely from seeing the value in the "help 
 the compiler learn what is unsafe" side, and I thought the 
 "inferring things unsafe" side could be handled separately. 
 (The helping-the-compiler side is eloquently illustrated in the 
 one of the DIP's linked [videos](https://youtu.be/O3TO52rXLug) 
 by Steven Schveighoffer. That's where I got my understanding of 
 what I thought the essence of DIP1035 *should* be about. Of 
 course, that's no reason to arbitrarily merge the two features, 
 ` system` and `__metadata`. But it might make it easier to 
 explain why I thought it was possible.)


Thanks for the explanation.  I think Steven is a good one to 
listen to.  He talks sense habitually.

If you're available do join us and others at the next beerconf to 
talk about this and other topics.  Paul, another of the good ones 
and DIP 1035 co-author, sometimes joins in as well.
Apr 09 2022
next sibling parent Zach Tollen <zach mystic.yeah> writes:
On Sunday, 10 April 2022 at 05:41:36 UTC, Bruce Carneal wrote:

 Thanks for the explanation.  I think Steven is a good one to 
 listen to.  He talks sense habitually.

 If you're available do join us and others at the next beerconf 
 to talk about this and other topics.  Paul, another of the good 
 ones and DIP 1035 co-author, sometimes joins in as well.


Wow, thanks. I think I will.

The only time-relevant issue I have is that if adopted as written 
now, DIP1035 may preclude the investigation of what I'm 
proposing, since it uses a single compiler bit for all ` system` 
variables, whereas I realized in my prior 
[post](https://forum.dlang.org/post/sflhxuliqgrtmfgzldrd forum.dlang.org) that
I needed to expand my proposal to two bits — one for ` system`-initialized,
and one for ` system`-variable.

Even if adopted, however, it should be rather effortless to 
expand the underlying implementation of DIP1035 if what I'm 
saying ever proves compelling to actually consider. So it's 
really not that urgent, but I at least wanted to voice my concern 
now to establish precedent. (Again, I realize that my proposal is 
either nuts or misunderstood, depending on who's looking at it.)
Apr 10 2022
prev sibling parent reply Zach Tollen <zach mystic.yeah> writes:
On Sunday, 10 April 2022 at 05:41:36 UTC, Bruce Carneal wrote:

 I view 1035 as a mechanism to extend the reach of  safe, to 
 reduce the load on conscientious code reviewers.


One aspect of DIP1035 I'm confused about in this regard is item 
(1) in the [proposed 
changes](https://github.com/dlang/DIPs/blob/72f41cffe68ff1f2d4c033b5728ef37e282461dd/DIPs/DIP1035.m
#proposed-changes): "An aggregate with at least one  system field is an unsafe
type."

This concerns me because it seems like it would extend the reach 
of * system*, rather than * safe*. It seems like having one 
 system variable would contaminate the whole structure so that 
you always had to use  trusted just to do anything with it.

Maybe I'm thinking too much...
Apr 10 2022
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 10 April 2022 at 13:02:07 UTC, Zach Tollen wrote:

 On Sunday, 10 April 2022 at 05:41:36 UTC, Bruce Carneal wrote:

 I view 1035 as a mechanism to extend the reach of  safe, to 
 reduce the load on conscientious code reviewers.


 One aspect of DIP1035 I'm confused about in this regard is item 
 (1) in the [proposed 
 changes](https://github.com/dlang/DIPs/blob/72f41cffe68ff1f2d4c033b5728ef37e282461dd/DIPs/DIP1035.m
#proposed-changes): "An aggregate with at least one  system field is an unsafe
type."

 This concerns me because it seems like it would extend the 
 reach of * system*, rather than * safe*. It seems like having 
 one  system variable would contaminate the whole structure so 
 that you always had to use  trusted just to do anything with it.


Yes, at what cost do we achieve additional (automated) safety?  I 
have no 1035 improvements to offer, and support it as is, but I 
think that your concern here is on point.
Apr 10 2022
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10.04.22 16:33, Bruce Carneal wrote:

 On Sunday, 10 April 2022 at 13:02:07 UTC, Zach Tollen wrote:

 On Sunday, 10 April 2022 at 05:41:36 UTC, Bruce Carneal wrote:

 I view 1035 as a mechanism to extend the reach of  safe, to reduce 
 the load on conscientious code reviewers.


 One aspect of DIP1035 I'm confused about in this regard is item (1) in 
 the [proposed 
 changes](https://github.com/dlang/DIPs/blob/72f41cffe68ff1f2d4c033b5728ef37e282461dd/DIPs/DIP1035.m
#proposed-changes): 
 "An aggregate with at least one  system field is an unsafe type."

 This concerns me because it seems like it would extend the reach of 
 * system*, rather than * safe*. It seems like having one  system 
 variable would contaminate the whole structure so that you always had 
 to use  trusted just to do anything with it.


 
 Yes, at what cost do we achieve additional (automated) safety?  I have 
 no 1035 improvements to offer, and support it as is, but I think that 
 your concern here is on point.
 
 


Here, "unsafe type" has a very specific meaning that is actually 
elaborated on by the DIP. Even unsafe types can typically be used from 
 safe code just fine. I think this particular concern grew out of a 
reading of the document that was overly cursory. This "contamination" 
concern has nothing in the DIP to back it up.
Apr 10 2022
parent Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 10 April 2022 at 14:52:43 UTC, Timon Gehr wrote:

 On 10.04.22 16:33, Bruce Carneal wrote:

 On Sunday, 10 April 2022 at 13:02:07 UTC, Zach Tollen wrote:

 On Sunday, 10 April 2022 at 05:41:36 UTC, Bruce Carneal wrote:






 Here, "unsafe type" has a very specific meaning that is 
 actually elaborated on by the DIP. Even unsafe types can 
 typically be used from  safe code just fine. I think this 
 particular concern grew out of a reading of the document that 
 was overly cursory. This "contamination" concern has nothing in 
 the DIP to back it up.


Very glad to hear that from a careful reader.



 Apr 10 2022



prev sibling  parent  Paul Backus <snarwin gmail.com>  writes:


On Sunday, 10 April 2022 at 13:02:07 UTC, Zach Tollen wrote:

 On Sunday, 10 April 2022 at 05:41:36 UTC, Bruce Carneal wrote:

 I view 1035 as a mechanism to extend the reach of  safe, to 
 reduce the load on conscientious code reviewers.


 One aspect of DIP1035 I'm confused about in this regard is item 
 (1) in the [proposed 
 changes](https://github.com/dlang/DIPs/blob/72f41cffe68ff1f2d4c033b5728ef37e282461dd/DIPs/DIP1035.m
#proposed-changes): "An aggregate with at least one  system field is an unsafe
type."

 This concerns me because it seems like it would extend the 
 reach of * system*, rather than * safe*. It seems like having 
 one  system variable would contaminate the whole structure so 
 that you always had to use  trusted just to do anything with it.

 Maybe I'm thinking too much...


The point of this is to prevent things like void-initialization 
of types with  system fields in  safe code. You can still access 
non- system fields of the structure without using  trusted.

"Unsafe type" does not mean you can't use it in  safe code. 
Pointer types are unsafe types, for example, and you can use 
pointers in  safe code without any issues. What "unsafe type" 
means is that, in  safe code, your usage of that type is 
restricted to operations that the compiler knows will not cause 
memory corruption.



 Apr 10 2022



prev sibling next sibling parent  Zach Tollen <zach mystic.yeah>  writes:


On Sunday, 10 April 2022 at 00:04:43 UTC, Zach Tollen wrote:

 On Saturday, 9 April 2022 at 17:22:38 UTC, Timon Gehr wrote:

 So your proposal is...


 My proposal is to take everything DIP1035 marks or infers as 
 ` system` and extend `__metadata` functionality to it.


I should amend my proposal to say I don't really think anything 
should be inferred to have `__metadata` functionality. (That's 
going too far even for my bold mind!) You should definitely have 
to say that a specific variable is `__metadata`/enhanced 
` system`. So I'd have to find a different solution as to how 
DIP1035 attempts to resolve global variables which are 
initialized to unsafe data, e.g.
```d
// global scope
 system:
int* x = cast(int*) 0xDEADBEEF;
```
First of all, I don't remember anyone actually complaining about 
this kind of thing. So it's not really bothering me. But this is 
how I'd solve it with my proposal, if I had to:

- Have the compiler make a subtle distinction between a ` system` 
*variable* and a ` system` *initialization*. This is solely to 
avoid having a separate keyword (e.g. ` systemVar`) for empowered 
` system` variables as opposed to just ones which are initialized 
unsafely. It's a little messy, but the result would be
```d
 system:
int* x = cast(int*) 0xDEADBEEF; // detected as initialized 
unsafely
 system int* y = 0xDEADBEEF; // specifically marked  system and 
therefore empowered with `__mutable` too

void main()  safe {
     *x = 10; // error: variable `x` was detected to have been 
initialized
              //   unsafely, so its reference cannot be modified 
in safe code
     *y = 10; // error: ` system` variable `y` cannot be modified 
in safe code
}
```
I'm pretty sure that would solve that problem.

--

DIP1035 also suggests that an "aggregate with at least one 
 system field is an unsafe type." I think this might be going too 
far. If you have to be in ` system` code to touch even the safe 
members of the type, I suspect you will lose the opportunity to 
catch a lot of unsafe bugs. I think it should be "any action 
which read or writes a ` system` variable is a ` system` action." 
But I would seek more clarification from the DIP authors on what 
they really meant by making the whole type unsafe.

--

None of this matters, of course. I don't think I can convince 
anybody that my proposal is worth trying, especially when you can 
just as easily adopt both DIP1035 and `__metadata` separately and 
see how things go. I just wanted to amend my previous claim to 
say that at minimum, you should have to mark empowered 
` system`/`__metadata` specifically and that they shouldn't be 
inferred.

Zach



 Apr 09 2022



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 10.04.22 02:04, Zach Tollen wrote:

 On Saturday, 9 April 2022 at 17:22:38 UTC, Timon Gehr wrote:

 Your proposal is not simpler.


 
 It is simpler.


If ` system` and `__metadata` are separate annotations, you get:

- ` system` rules
- `__metadata` rules

If you conflate them, you get:

- ` system` rules
- `__metadata` rules
- Weird interactions between the rules
- Additional arbitrary rules to be able to pretend the rules for the 
different annotations are compatible anyway

The additional rules and weird interactions are what's causing 
complexity, in addition to the misnamed qualifier and the significantly 
enhanced area of contact between D users and inherent `__metadata` 
weirdness.


 The only question is whether that simplicity comes at too 
 high a cost.
 ...


If "simplicity" is causing costs, you are probably doing it wrong. Don't 
try to make things "simpler" than they are. It's a source of complexity 
and further ad-hoc patches.


 Your proposal fails to meet your own standards.


 
 I don't know if it meets my standards or not. I would need an example 
 where a variable which was either explicitly marked, or inferred 
 ` system`, and implemented as desired with that label, then changed its 
 behavior as a result of also implicitly having the privileges of 
 `__mutable`.
 


It's not a privilege if it's undesirable. Also, you have been given 
multiple examples by now.

(You removed relevant context here.)

 I see what you did there. I can do that too: Clearly you think that 
 simplicity and expressiveness are both monotone functions of the 
 number of attributes that are in the language. If what you say were 
 true, we should conflate  safe and pure, as well as  nogc and nothrow 
 because that would be "simpler". That's ridiculous!


 
 No I'm saying that simplicity and expressiveness are tradeoffs.


I was objecting to your setting up of a straw man that you then 
proceeded to attack. I then simply demonstrated my ability to do the 
same, expecting it might sway you as you seem to enjoy that kind of 
reasoning yourself. Unfortunately this has not worked out.

In any case, there are very simple systems that are very expressive. You 
want to be in that quadrant.


 The Go 
 language, for example, traded a lot of expressiveness for simplicity. 
 That doesn't make it bad language design.


I don't think this is entirely the case. They made some miscalculations 
early on and are now left with a language that is neither as simple nor 
as expressive as it should have been, nor as orthogonal as they claim.


 It means they weighed the pros and the cons, and they made a decision.
 ...


It's probably one level further away from that. They didn't spend a lot 
of effort to make a good language. They just spent vastly more time 
writing the compiler and libraries and my understanding is they did a 
pretty decent job there. Also, perhaps they actively wanted to avoid 
having a cerebral community that cares about abstract things such as PL 
theory. (There was a year-long active hostility towards potentially 
simple things such as generics in the Go community. You can draw your 
own conclusions.)


 The use cases of DIP1035, for the most part, are a strict subset of 
 the uses of `__mutable`/`__metadata`.


 ???


 
 All `__metadata` variables qualify as DIP1035 ` system`. If you added 
 ` system` to any instance of `__metadata`, it would have no effect.


Granted. Now you have shown "var is `__metadata`" => "var is ` system`", 
but that's not some sort of magical relationship between the qualifiers. 
There are multiple reasons why variables might by ` system` as it just 
collects every source of unsafety. `__metadata` is a new such source 
that we are considering adding to the language.

However, your claim above was the exact opposite: "var is  system" => 
"var is __metadata". As you said it is even a _strict subset_, you 
additionally claim that there are use cases where variables should be 
`__metadata` but not ` system`.

You seem to be contradicting yourself. It's either bad logic or bad 
communication.


 Therefore, if only `__metadata` were implemented, users could utilize it 
 to achieve the intended goals of DIP1035.


No, because DIP1035 does not alter variable types in any way. You could 
similarly say: "Just initialize your variables with unsafe values and 
get rid of the need for a ` system` annotation." This is however not a 
simplification. The compiler still needs to track all that and making it 
part of the notation makes it simpler to understand.


 (Just allow them to mark it 
 wherever they wanted instead of confining it to aggregate members, even 
 though most of the use cases are in fact aggregate members.)


This is not a complete specification...


 The question is simply whether additionally being able to violate the type 
 system would cause intractable problems.


No, not at all. Given that you apparently dismissed all the evidence in 
this thread as "tractable problems", your standard is way off in my 
opinion. E.g., I would similarly not run into _intractable_ problems if 
I chopped one of my arms off. That does not make it any less stupid even 
though one arm is clearly simpler than two arms. It would however not 
simplify my life.

You are basically stating that you will not move away from your 
position, no matter what. Digging in like that is not healthy, and this 
proposal is not actually worth spending time on, neither yours nor mine.


 I don't know. I suspect that 
 almost all use cases of ` system` are mutable to begin with. And if they 
 weren't, I would have to look the workarounds, to see if issues of 
 having undesired `__metadata` functionality caused more pain than the 
 simplicity of using only one keyword merited.
 ...


You keep claiming it's simpler. It's not. You are chasing after a prize 
that's not actually worth anything because you are using some useless, 
tunnel-vision notion of simplicity that only considers one (very 
unimportant) aspect of what potentially makes a language simple.

I would not chop off my arm even if someone offered to buy my house for 
one dollar in exchange, even though one dollar is a very simple amount. 
Avoiding that kind of "trade-off" should not be controversial.


 So your proposal is...


 
 My proposal is to take everything DIP1035 marks or infers as ` system` 
 and extend `__metadata` functionality to it.


I see, so if a module-scope variable is inferred ` system`, you will get 
an error because `__metadata` has been applied to a non-member variable. 
(I get that you state at other places you don't want it to work that 
way, but given that you seem to be unable to simply state what your 
proposal is, it's unlikely to actually be all that simple.)


 This may sound dangerous. 


Primarily it sounds utterly pointless. All the reasoning about why it's 
"dangerous" is only required because you refuse to accept that it's 
pointless.


 The logic is simply that this is ` system` code, and ` system` code is 
 dangerous.


Given your own standards of evidence, you will have to do much better 
than that. Please tell me how having separate ` system` and `__metadata` 
will cause intractable pain, suffering and ultimately end civilization 
as we know it. If you can't do that in a way that satisfies me, I will 
dismiss everything you say.


 But it is also powerful. That's the overarching idea. Both 
 dangerous and powerful. (D should also be working towards ` safe` by 
 default, of course.)
 
 Again, the only question is whether having `__metadata` functionality 
 will cause intractable pain to the use cases of ` system` which don't 
 require it.
 ...


I really don't understand why you are willing to suffer everything short 
of "intractable pain" for this issue.


 In any case, even if your proposal was simple to state formally, it 
 would still lead to really bad outcomes.


 
 The difference between us is that you are sure of that, and I am not.
 ...


I wouldn't say that's the whole difference.


 ...
 
 The solution to our debate — insofar as my claims deserve any merit at 
 all — is to implement both DIP1035 and the `__metadata` DIP, with the 
 additional requirement that all uses of `__metadata` be marked ` system` 
 as well.


Not much point in that, it's just more line noise.


 Semantically, it would change nothing, and it would actually be 
 easier to understand for newcomers.
 ...


Newcomers will not have to learn about `__metadata` or ` system` 
variables for a pretty long time. Then they may come across ` system` 
variables at some point, and finally, some further minority of D 
programmers will actually need to significantly concern themselves with 
`__metadata` rules.

If you keep the qualifiers separate, the meaning of ` system` is simple 
and intuitive and nobody who would not otherwise need to will have to 
concern themselves with `__metadata`. That's a much better proposition 
for newcomers than arbitrary conflation.


 Then, five years later, to test my theory, add a compiler switch 
 `preview=deprecate__metadata`, put the `__metadata` functionality into 
 the ` system`, and make `__metadata` a no-op.
 
 If nothing breaks, I was right. If something breaks, and the workarounds 
 aren't seamless, then you were right.
 


...

In other words, you will just move the goalposts until you can claim you 
were right. Maybe you should care less about having been right in the 
past and more about being right in the future. Note that in my very 
first debate on this forum, I admitted that I had been wrong and that 
Andrei was right. For me, this is not about being right, it's about not 
having to go through the hassle of switching programming languages. I 
don't enjoy pointless debate, but weird language rules make it harder to 
justify why I am even using D. I need it to look like things are getting 
better (which generally they have been over the past 10 years), not worse.



 Apr 10 2022



  parent reply Zach Tollen <zach mystic.yeah>  writes:


On Sunday, 10 April 2022 at 14:34:41 UTC, Timon Gehr wrote:

 You seem to be contradicting yourself. It's either bad logic or 
 bad communication.


I think it's the latter. I think the chief source of bad 
communication is actually a design flaw in DIP1035:

https://forum.dlang.org/post/tgbwhgxzkrupdotylkms forum.dlang.org

` system` variables are not a bad idea. DIP1035's mistake is in 
thinking that anyone would ever want a ` system` variable by 
accident.

Once you subtract all of the evidence against me which relied on 
the design flaw in DIP1035, what part of your case still stands?

All of the actual use cases for ` system` variables — I mean real 
` system` variables, not just unsafely initialized ones — that I 
have seen (some are illustrated in the DIP, but also in Steven 
Schveighoffer's really excellent 
[presentation](https://youtu.be/O3TO52rXLug) on tagged unions) 
involve preserving the invariant of a data structure by making 
sure the ` system` variable is only modified simultaneously with 
some other piece of data.

Which means, generally, they are modified along with a normal 
variable. If the ` system` variable had `__mutable` 
characteristics implicitly, the ` system` function in which it 
was modified would also be modifying a non-`__mutable` value. 
Such a function would error anyway.

Maybe there is some other natural use for ` system` variables. 
But none of the use cases I have seen seem particularly 
incompatible with the additional `__mutable` functionality.


 You are basically stating that you will not move away from your 
 position, no matter what.


All I'm doing is trying to make my position clear.

You have to consider everything I've been saying in the context 
of how ` system` variables *should* have been defined from the 
beginning — as always explicitly ` system`. (I didn't realize 
just how devastating DIP1035's alternative, "infer ` system` 
everywhere" design really was until I was forced to examine it 
more closely. I've now made my case about that in the other post.)

This goes for everything I've been saying. Apply the new filter 
of: "A variable is a ` system` variable iff it is explicitly 
marked ` system`."

Me:

 Again, the only question is whether having `__metadata` 
 functionality will cause intractable pain to the use cases of 
 ` system` which don't require it.
 
 The solution to our debate — insofar as my claims deserve any 
 merit at all — is to implement both DIP1035 and the 
 `__metadata` DIP, with the additional requirement that all 
 uses of `__metadata` be marked ` system` as well.




I stand by this statement, assuming modified DIP1035.


 In other words, you will just move the goalposts until you can 
 claim you were right.


I honestly don't think that's what I'm doing. My mistake was 
actually having too strong a sense of the *right* way to 
implement ` system` variables. I only realized too late that 
DIP1035 wasn't quite there.

--

I'm sorry for a lot of the stress I've caused. I'm pretty sure 
that everything I've been saying makes at least a little bit of 
sense, given what I have always meant (to myself at least ^_^) by 
"` system` variable".

Zach



 Apr 13 2022



  parent reply Dennis <dkorpel gmail.com>  writes:


On Wednesday, 13 April 2022 at 12:31:44 UTC, Zach Tollen wrote:

 Maybe there is some other natural use for ` system` variables. 
 But none of the use cases I have seen seem particularly 
 incompatible with the additional `__mutable` functionality.


Without conflating ` system` and `__mutable`:
```D
struct Boolean
{
      system ubyte payload; // must be 0 or 1 or buffer overflows 
could happen
}

immutable Boolean b; // can be placed in read-only data segment
```

When conflating ` system` and `__mutable`, `b` suddenly must be 
placed in mutable memory.



 Apr 13 2022



  parent  Paul Backus <snarwin gmail.com>  writes:


On Wednesday, 13 April 2022 at 12:59:30 UTC, Dennis wrote:

 On Wednesday, 13 April 2022 at 12:31:44 UTC, Zach Tollen wrote:

 Maybe there is some other natural use for ` system` variables. 
 But none of the use cases I have seen seem particularly 
 incompatible with the additional `__mutable` functionality.


 Without conflating ` system` and `__mutable`:
 ```D
 struct Boolean
 {
      system ubyte payload; // must be 0 or 1 or buffer 
 overflows could happen
 }

 immutable Boolean b; // can be placed in read-only data segment
 ```

 When conflating ` system` and `__mutable`, `b` suddenly must be 
 placed in mutable memory.


On top of this: leaving ` system` and `__mutable` separate allows 
`b` to be safely accessed from multiple threads without 
synchronization. If ` system` and `__mutable` were combined, this 
could potentially result in data races, and so would no longer be 
safe.

It's not hard to imagine situations where one might want to 
create an `immutable` copy of a type with ` system` fields in 
order to share it between threads.



 Apr 13 2022



prev sibling  parent  Zach Tollen <zach mystic.yeah>  writes:


On Friday, 8 April 2022 at 02:32:29 UTC, Paul Backus wrote:

 On Friday, 8 April 2022 at 01:46:35 UTC, Zach Tollen wrote:

 On Friday, 8 April 2022 at 01:24:38 UTC, Paul Backus wrote:
 In other words, if it's dangerous enough to start out as 
 ` system`, how much additional danger are we adding by making 
 it mutable?


 You're not adding any additional *danger*, I suppose, but it 
 would be a non-obvious special case in the type system to have 
 the ` system` attribute on variables interact with `immutable` 
 like this. As currently proposed by DIP 1035, they are 
 completely independent of each other.


It's only non-obvious if you define ` system` variables simply as 
"not usable from ` safe` code." If you defined them as "can also 
override the type system," it *would* be obvious.

The only question is whether this is a good language design. It 
is certainly easy to explain:

Q: "What are ` system` variables?"

A: "They are variables you can do anything you want with, as long 
as it's from ` system` code."

Q: "What are they for?"

A: "They're for doing things you can't do with regular variables, 
and also for marking regular variables as dangerous when the 
compiler can't figure that out on its own."



 Apr 08 2022



prev sibling  parent reply Alexandru Ermicioi <alexandru.ermicioi gmail.com>  writes:


On Wednesday, 6 April 2022 at 09:41:52 UTC, RazvanN wrote:

 ...


It may be a dumb and already discussed question, but what's the 
problem of making rc counted structs mutable only?

In this way, you won't have the dilemma of circumventing the 
immutable/const system, to just increment the counter, or any 
metadata.

Note: the payload of rc struct, may be const or immutable if 
needed.

Best regards,
Alexandru.



 Apr 08 2022



 next sibling parent  Alexandru Ermicioi <alexandru.ermicioi gmail.com>  writes:


On Friday, 8 April 2022 at 13:50:01 UTC, Alexandru Ermicioi wrote:

...


Text was misquoted, sorry.



 Apr 08 2022



prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Fri, Apr 08, 2022 at 01:50:01PM +0000, Alexandru Ermicioi via Digitalmars-d
wrote:

 On Wednesday, 6 April 2022 at 09:41:52 UTC, RazvanN wrote:

 ...


 It may be a dumb and already discussed question, but what's the
 problem of making rc counted structs mutable only?
 
 In this way, you won't have the dilemma of circumventing the
 immutable/const system, to just increment the counter, or any
 metadata.
 
 Note: the payload of rc struct, may be const or immutable if needed.


[...]

In theory, this would IMO be the right approach.  However, it does
hamper usability. For example, you could no longer write:

	auto myFunc(in RC!Data data) { ... }

but you'd have to write instead:

	auto myFunc(RC!(const(Data)) data) { ... }

because RC!... has to remain always mutable.

Some of the nice implicit conversions would also no longer work as
desired, e.g., RC!Data would not implicitly convert to RC!(const(Data)),
where with GC'd data, Data* implicit converts to const(Data)*. This
asymmetry makes metaprogramming harder to work with RC.

Things get worse once you have nested structures, e.g.:

	struct SubData {...}
	struct S {
		RC!SubData data;
	}

Now you couldn't pass S to anything that takes const, because
transitivity of const would force RC!SubData to be const(RC!SubData),
which breaks the refcounting mechanism.  Basically, once you have RC!xxx
in any subpart of your data structure, it "reverse-infects" the entire
data structure with mutable, because you can no longer use const on any
containing part of the structure, since it would break RC.

Basically, unless you have a hack like __mutable, you basically have to
throw out const/immutable entirely from your data structures once any
part of it involves RC.


T

-- 
Век живи - век учись. А дураком помрёшь.



 Apr 08 2022



 next sibling parent  Paul Backus <snarwin gmail.com>  writes:


On Friday, 8 April 2022 at 14:23:01 UTC, H. S. Teoh wrote:

 Basically, unless you have a hack like __mutable, you basically 
 have to throw out const/immutable entirely from your data 
 structures once any part of it involves RC.


You can make RC work with const and immutable if you're willing 
to give up on pure:

https://gist.github.com/pbackus/1638523a5b6ea3ce2c0a73358cff4dc6

Personally, I would rather give up on pure (which already has 
loopholes in it) than add a loophole to immutable (which 
currently has none).



 Apr 08 2022



prev sibling  parent  Alexandru Ermicioi <alexandru.ermicioi gmail.com>  writes:


On Friday, 8 April 2022 at 14:23:01 UTC, H. S. Teoh wrote:

 On Fri, Apr 08, 2022 at 01:50:01PM +0000, Alexandru Ermicioi 
 via Digitalmars-d wrote:

 On Wednesday, 6 April 2022 at 09:41:52 UTC, RazvanN wrote:

 ...


 It may be a dumb and already discussed question, but what's 
 the problem of making rc counted structs mutable only?
 
 In this way, you won't have the dilemma of circumventing the 
 immutable/const system, to just increment the counter, or any 
 metadata.
 
 Note: the payload of rc struct, may be const or immutable if 
 needed.


 [...]

 In theory, this would IMO be the right approach.  However, it 
 does hamper usability. For example, you could no longer write:

 	auto myFunc(in RC!Data data) { ... }

 but you'd have to write instead:

 	auto myFunc(RC!(const(Data)) data) { ... }

 because RC!... has to remain always mutable.


Nicer version: RC!(const T)
Though that is true.


 Some of the nice implicit conversions would also no longer work 
 as
 desired, e.g., RC!Data would not implicitly convert to 
 RC!(const(Data)),
 where with GC'd data, Data* implicit converts to const(Data)*. 
 This
 asymmetry makes metaprogramming harder to work with RC.


Can't alias this be employed for this?

I.e. in Rc struct have this:
alias this = asQualified;

Where asQualified returns version with suitable modifier.


 Things get worse once you have nested structures, e.g.:

 	struct SubData {...}
 	struct S {
 		RC!SubData data;
 	}

 Now you couldn't pass S to anything that takes const, because 
 transitivity of const would force RC!SubData to be 
 const(RC!SubData), which breaks the refcounting mechanism.
 Basically, once you have RC!xxx in any subpart of your data 
 structure, it "reverse-infects" the entire data structure with 
 mutable, because you can no longer use const on any containing 
 part of the structure, since it would break RC.


True, it won't be easy to use RC as fields, but could we ignore 
this problem for initial implementation, by prohibiting immutable 
versions?

You can always extend RC structs to support immutable/const 
qualifiers on itself.

I keep reading this forum for a long time, and always see only 
discussions about rc structs and etc, but no viable solutions. 
Perhaps it's better to try implement a restricted version of it 
first? i.e. only mutable rcs?



 Apr 08 2022



prev sibling  parent reply rikki cattermole <rikki cattermole.co.nz>  writes:


Here is my test code that I used to determine what I needed to do to 
make const RC types useless (and hopefully safe).

```d
void main() {
     auto r1 = /+const+/ Reference(2);
     const r2 = r1;
     const r3 = Reference(2);
     const(Reference) r4 = Reference(2);
     //auto r5 = cast(const)r1;
     auto r6 = cast(void*)&r1;
     auto r7 = cast(const)&r1;
}

struct Reference {
     int x;

 safe:

     this(int value) {
         this.x = value;
     }

     this(ref Reference other) {
         this.x = other.x;
     }


     ~this() {
         this.x = 0;
         writeln("destructor");
     }

     void opAssign(ref Reference other) {
         __ctor(other);
     }

     void opAssign(Reference other) {
         __ctor(other);
     }

     // bool isNull() { ... }

      disable this(int value) const;

     // disable this(ref Reference other) const;
      disable this(ref const Reference other) const;
      disable this(this);

     // disable ~this() const;

      disable void opAssign(ref Reference other) const;
      disable void opAssign(Reference other) const;

      disable auto opCast(T)();
}
```



 Apr 08 2022



 next sibling parent reply vit <vit vit.vit>  writes:


On Friday, 8 April 2022 at 18:00:14 UTC, rikki cattermole wrote:

 Here is my test code that I used to determine what I needed to 
 do to make const RC types useless (and hopefully safe).

 ```d
 ...
 ```


I implemented shared_ptr/rc_ptr with this functionality (non 
copyable const rc ptr):
https://github.com/submada/btl/blob/master/source/btl/autoptr/shared_ptr.d
https://github.com/submada/btl/blob/master/source/btl/autoptr/rc_ptr.d

It is in package btl:autoptr 
(https://code.dlang.org/packages/btl).

Example:
```d
SharedPtr!(const int) sp = SharedPtr!int.make(42);

const SharedPtr!(const int) csp1 = sp;  //OK
const SharedPtr!(int) csp2 = sp;  //OK

sp = csp1;   //ERROR
sp = csp2;   //ERROR

//RcPtr is same.

```
Ref counted pointers in D have problem:

RAII in D is bad: move ctor are not implemented, overloading copy 
ctors sometimes crash dmd, copy ctors are not called from GC 
slices, dtors have bad interaction with stack tuples/value 
sequences and opCast, emplace...


Simple ref counted pointers have also this problem:
```d
Rc!(const int) rc1 = Rc!(int).make(42);
Rc!(const int) rc2 = Rc!(immutable int).make(42);

```

Has `Rc!(const int)` atomic counting or not? :)



 Apr 08 2022



 next sibling parent reply Alexandru Ermicioi <alexandru.ermicioi gmail.com>  writes:


On Saturday, 9 April 2022 at 06:25:10 UTC, vit wrote:

 RAII in D is bad: move ctor are not implemented, overloading 
 copy ctors sometimes crash dmd, copy ctors are not called from 
 GC slices, dtors have bad interaction with stack tuples/value 
 sequences and opCast, emplace...


Imho these should be prioritized first before, doing yet another 
language change. If they are fixed, then there might be more 
options available on how to design the rc functionality.



 Apr 10 2022



  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 10.04.22 16:48, Alexandru Ermicioi wrote:

 On Saturday, 9 April 2022 at 06:25:10 UTC, vit wrote:

 RAII in D is bad: move ctor are not implemented, overloading copy 
 ctors sometimes crash dmd, copy ctors are not called from GC slices, 
 dtors have bad interaction with stack tuples/value sequences and 
 opCast, emplace...


 
 Imho these should be prioritized first before, doing yet another 
 language change. ...


I agree that lifetime semantics that actually work should be a much 
higher priority than hiding reference counts in immutable structures. 
(But maybe the people who are willing to fix either thing are not the 
same people.)



 Apr 10 2022



prev sibling  parent  RazvanN <razvan.nitu1305 gmail.com>  writes:


On Saturday, 9 April 2022 at 06:25:10 UTC, vit wrote:


 Ref counted pointers in D have problem:

 RAII in D is bad: move ctor are not implemented,


Move ctor is in the process of making.


 overloading copy ctors sometimes crash dmd,



Overloading cpCtors with rvalue ctors sometimes crash the 
compiler because we don't have all the cases fleshed out. 
However, with https://github.com/dlang/dmd/pull/13976 if you do 
not template your rvalue constructors then you should be fine.



 copy ctors are not called from GC slices


This is also being worked on. We are templating the druntime 
hooks so cpCtors should be called then. Anyway, some cases have 
been fixed by now because and more will come (I know that Teo 
Dutu is almost done with templating the hooks that deal with ~= 
and ~ so cp ctors will be called for those).



 dtors have bad interaction with stack tuples/value sequences 
 and opCast, emplace...


Yeah, that's a big one.



 Apr 12 2022



prev sibling  parent reply Alexandru Ermicioi <alexandru.ermicioi gmail.com>  writes:


On Friday, 8 April 2022 at 18:00:14 UTC, rikki cattermole wrote:

 Here is my test code that I used to determine what I needed to 
 do to make const RC types useless (and hopefully safe).
 ...


Isn't better then to try and find design patterns or rules for 
use with mutable RC structures?

All the talk across the years was basically about trying to eat 
the pancake and also at same time trying to keep it for later, 
and neither is done after all deliberation across the years. 
Perhaps it would be best to just break the existing immutable 
system, and redefine it to allow rc features found in C++ or 
other languages, or just have a limited version of rcs in D 
(compared to C++ or other languages employing them), and 
workaround those restrictions through some well defined rules and 
design patterns.

And when mentioning breaking the immutable system, I really mean 
breaking it, by removing the transitiveness it has, or some other 
major change, like not being really immutable, and therefore safe 
to put in readonly memory.



 Apr 10 2022



 next sibling parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 10.04.22 17:04, Alexandru Ermicioi wrote:

 On Friday, 8 April 2022 at 18:00:14 UTC, rikki cattermole wrote:

 Here is my test code that I used to determine what I needed to do to 
 make const RC types useless (and hopefully safe).
 ...


 
 Isn't better then to try and find design patterns or rules for use with 
 mutable RC structures?
 
 All the talk across the years was basically about trying to eat the 
 pancake and also at same time trying to keep it for later, and neither 
 is done after all deliberation across the years. Perhaps it would be 
 best to just break the existing immutable system, and redefine it to 
 allow rc features found in C++ or other languages, or just have a 
 limited version of rcs in D (compared to C++ or other languages 
 employing them), and workaround those restrictions through some well 
 defined rules and design patterns.
 
 And when mentioning breaking the immutable system, I really mean 
 breaking it, by removing the transitiveness it has, or some other major 
 change, like not being really immutable, and therefore safe to put in 
 readonly memory.
 
 


I think the solution of this particular issue should come down to one of:

- accept that `immutable` means immutable, only use it where it actually 
makes sense to enforce that restriction, derive valid compiler 
optimizations from a specification based on UB. No immutable RC is 
allowed. `immutable` values can always be put in readonly memory, simple 
story for CTFE accessing immutable values.

- accept that there could potentially be mutable parts in `immutable` 
structures exclusively for low-level purposes, separate `immutable` and 
`shared immutable`, explicitly specify valid compiler optimizations 
based on nondeterministic semantics. Immutable RC is a thing. Whether or 
not `immutable` values can be put in readonly memory depends on the 
particular type, more complicated story for CTFE accessing immutable values.

Anyway, this is by far not the only issue with qualifiers applied to 
user-defined types. Built-in slices and pointer types have magical 
interactions with qualifiers that user-defined types cannot reproduce.



 Apr 10 2022



prev sibling  parent reply rikki cattermole <rikki cattermole.co.nz>  writes:


On 11/04/2022 3:04 AM, Alexandru Ermicioi wrote:

 Isn't better then to try and find design patterns or rules for use with 
 mutable RC structures?


Yes.

But if the compiler allows RC to become const and not allow RC to occur, 
that is a bad situation to be in...

That is what my test code above does, it prevents you from using it with 
const as much as possible.

It is a workaround to a much larger problem.



 Apr 10 2022



  parent reply Alexandru Ermicioi <alexandru.ermicioi gmail.com>  writes:


On Sunday, 10 April 2022 at 23:12:03 UTC, rikki cattermole wrote:

 On 11/04/2022 3:04 AM, Alexandru Ermicioi wrote:

 Isn't better then to try and find design patterns or rules for 
 use with mutable RC structures?


 Yes.

 But if the compiler allows RC to become const and not allow RC 
 to occur, that is a bad situation to be in...


This seems like a couple of edge cases and bugs that need to be 
rectified, in order to have proper safe mutable only RC structs. 
You should be able to disallow implicit cast to const of struct 
if it is desired to do so.


 That is what my test code above does, it prevents you from 
 using it with const as much as possible.

 It is a workaround to a much larger problem.


Regarding const rc use, I think there can be a limited access to 
the payload, if you somehow prohibit taking the address of the 
payload or it's sub-elements. Perhaps instead of returning 
payload itself from rc struct, maybe it can be wrapped into a 
wrapper, that prevents any take of address when a field is 
accessed from payload, or payload itself.



 Apr 11 2022



  parent reply Paul Backus <snarwin gmail.com>  writes:


On Monday, 11 April 2022 at 11:46:42 UTC, Alexandru Ermicioi 
wrote:

 On Sunday, 10 April 2022 at 23:12:03 UTC, rikki cattermole 
 wrote:

 On 11/04/2022 3:04 AM, Alexandru Ermicioi wrote:

 Isn't better then to try and find design patterns or rules 
 for use with mutable RC structures?


 Yes.

 But if the compiler allows RC to become const and not allow RC 
 to occur, that is a bad situation to be in...


 This seems like a couple of edge cases and bugs that need to be 
 rectified, in order to have proper safe mutable only RC 
 structs. You should be able to disallow implicit cast to const 
 of struct if it is desired to do so.


There's no issue here. You just write the copy constructor of 
RC!T to require a mutable source and destination object, and the 
compiler will refuse to copy a const(RC!T). (You can still pass 
by `const ref`, of course.)


 Regarding const rc use, I think there can be a limited access 
 to the payload, if you somehow prohibit taking the address of 
 the payload or it's sub-elements. Perhaps instead of returning 
 payload itself from rc struct, maybe it can be wrapped into a 
 wrapper, that prevents any take of address when a field is 
 accessed from payload, or payload itself.


You can limit access by passing a `scope` reference to the 
payload to a callback function, as described in this comment:

https://github.com/dlang/phobos/pull/8368#issuecomment-1024917439



 Apr 11 2022



  parent reply Alexandru Ermicioi <alexandru.ermicioi gmail.com>  writes:


On Monday, 11 April 2022 at 14:44:18 UTC, Paul Backus wrote:

 There's no issue here. You just write the copy constructor of 
 RC!T to require a mutable source and destination object, and 
 the compiler will refuse to copy a const(RC!T). (You can still 
 pass by `const ref`, of course.)


Obviously there are:
```d
import std;

void main()
{
     int j = 2;
     int k = 3;
     const r3 = Counted!int(&j);
     const(Counted!int) r4 = Counted!int(&k);
}

 safe struct Counted(T)
{
     private T* subject;
     private size_t* counter;

     this(T* subject)
     {
         this.counter = new size_t;
         this.subject = subject;
         *this.counter = 1;
     }

     this(ref T subject) const  disable;
     this(T* subject) const  disable;

     this(ref Counted!T counted)
     {
         this.subject = counted.subject;
         this.counter = counted.counter;
         ++(*this.counter);
     }

     this(ref Counted!(const T) counted) const  disable;
     this(ref const Counted!T counted) const  disable;
     this(ref Counted!T counted) const  disable;

     this(this)  disable;

     ~this()
     {
         (*this.counter)--;

         if ((*this.counter) == 0)
         {
             writeln("Freeing the subject: ", *this.subject);

             this.counter = null;
         }
     }

     // ~this() const  disable; Obviously another bug or uncharted 
area.

     void opAssign(ref Counted!T counted)
     {
         --(*this.counter);
         this.subject = counted.subject;
         this.counter = counted.counter;
         ++(*this.counter);
     }

     void opAssign(Counted!T counted)
     {
         this = counted;
     }

     void opAssign(ref const(Counted!T) counted) const  disable;
     void opAssign(const(Counted!T) counted) const  disable;
     void opAssign(Counted!(const T) counted) const  disable;
     void opAssign(ref Counted!(const T) counted) const  disable;
     void opAssign(ref Counted!T counted) const  disable;
     void opAssign(Counted!T counted) const  disable;

     void opCast(T)()  disable;
}
```

Or I missed something that will prevent these two use cases.



 Apr 11 2022



  parent  Paul Backus <snarwin gmail.com>  writes:


On Monday, 11 April 2022 at 16:40:42 UTC, Alexandru Ermicioi 
wrote:

 On Monday, 11 April 2022 at 14:44:18 UTC, Paul Backus wrote:

 There's no issue here. You just write the copy constructor of 
 RC!T to require a mutable source and destination object, and 
 the compiler will refuse to copy a const(RC!T). (You can still 
 pass by `const ref`, of course.)


 Obviously there are:

 [...]


I don't see what the problem here is. Two instances of 
`Counted!int` are created, and both are destroyed. No references 
are leaked or left dangling. If you try to copy-construct an 
additional instance, like

     auto r5 = r4;

...then the compiler correctly rejects the code.

Granted, it only works because the compiler-inserted destructor 
call ignores `const`, which is a little questionable, but I think 
in this case it's basically harmless.



 Apr 11 2022