www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Challenge: write a reference counted slice that works as much as

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
This was prompted by this exchange with Paul Backus that went like this:

Me: "We never got reference counting to work in safe pure nogc code. And 
we don't know how. If we did, we could write a slice-like type that does 
everything a slice does PLUS manages its own memory. This is the real 
problem with containers."

Paul: "I'm pretty sure at least some of us (including Atila) know how."

===

Eduard Staniloiu (at the time a student I was mentoring) tried really 
hard to define a built-in slice RCSlice(T) from the following spec:

- be as close in semantics to T[] as possible, i.e. most code should be 
able to simply replace T[] with RCSlice!T and work unchanged
- manage its own memory by using reference counting
- work in pure code just like T[] does
- work with qualifiers like T[] does, both RCSlice!(qual T) and 
qual(RCSlice!T)
- work in  nogc code just like T[] does
- work in  safe code just like T[] does

We were unable to define that within the limits of D. Here are the major 
issues we encountered:

1. We could not make reference counting work properly in pure functions 
(a pure function does not mutate data, which goes against manipulating 
the reference count).

2. Qualifiers compound problems with interlocking: mutable data is known 
to be single-threaded, so no need for interlocking. Immutable data may 
be multi-threaded, meaning reference counting needs atomic operations. 
Const data has unknown origin, which means the information of how data 
originated (mutable or not) must be saved at runtime.

3. Safety forced the use of trusted blocks all over. Not a showstopper, 
but a complicating factor.

So if there are any takers on getting RCSlice off the ground, it would 
be very interesting.
Nov 08 2021
next sibling parent reply Adam Ruppe <destructionator gmail.com> writes:
On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
wrote:

 - work with qualifiers like T[] does, both RCSlice!(qual T) and 
 qual(RCSlice!T)


I think this is an incorrect requirement. A const(T) is now 
allowed to increase a reference count by definition.

It in theory might still allow borrowing a reference, since 
there'd be no net change in reference count there and thus the 
data can remain immutable, but if isn't statically guaranteed to 
be balanced, you're directly in contradiction with the const rule.
Nov 08 2021
next sibling parent deadalnix <deadalnix gmail.com> writes:
On Monday, 8 November 2021 at 22:07:46 UTC, Adam Ruppe wrote:

 On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
 wrote:

 - work with qualifiers like T[] does, both RCSlice!(qual T) 
 and qual(RCSlice!T)


 I think this is an incorrect requirement. A const(T) is now 
 allowed to increase a reference count by definition.

 It in theory might still allow borrowing a reference, since 
 there'd be no net change in reference count there and thus the 
 data can remain immutable, but if isn't statically guaranteed 
 to be balanced, you're directly in contradiction with the const 
 rule.


To quote Herb Sutter, because it seems to be the fashion of the 
day, const actually means thread safe.

This is not a joke.
Nov 08 2021
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 17:07, Adam Ruppe wrote:

 On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu wrote:

 - work with qualifiers like T[] does, both RCSlice!(qual T) and 
 qual(RCSlice!T)


 
 I think this is an incorrect requirement. A const(T) is now allowed to 
 increase a reference count by definition.


How do you mean "now"?
Nov 08 2021
parent Adam Ruppe <destructionator gmail.com> writes:
On Monday, 8 November 2021 at 22:35:59 UTC, Andrei Alexandrescu 
wrote:

 How do you mean "now"?


errr I meant "not". since it is const, you aren't allowed to 
modify it or *anything* through it. that would include the ref 
count.

(now i know you can look it up in a global AA, using the pointer 
as a key into it, and fake it, so this prohibition isn't really 
that meaningful. but of course pure prohibits even that, so 
there's no winning - if modifying that refcont, either you're 
const and not pure, or you're pure and not const. perhaps the 
definition of const should change though.)
Nov 08 2021
prev sibling next sibling parent Elronnd <elronnd elronnd.net> writes:
On Monday, 8 November 2021 at 22:07:46 UTC, Adam Ruppe wrote:

 On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
 wrote:

 - work with qualifiers like T[] does, both RCSlice!(qual T) 
 and qual(RCSlice!T)


 I think this is an incorrect requirement. A const(T) is not 
 allowed to increase a reference count by definition.

 It in theory might still allow borrowing a reference, since 
 there'd be no net change in reference count there and thus the 
 data can remain immutable, but if isn't statically guaranteed 
 to be balanced, you're directly in contradiction with the const 
 rule.


You can do head-const with delegates.  immutable will be a 
problem, though.
Nov 08 2021
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 8 November 2021 at 22:07:46 UTC, Adam Ruppe wrote:

 On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
 wrote:

 - work with qualifiers like T[] does, both RCSlice!(qual T) 
 and qual(RCSlice!T)


 I think this is an incorrect requirement. A const(T) is now 
 allowed to increase a reference count by definition.


This is technically true, but he wrote "built in", so then you 
don't have to heed the same constraints. You only have to require 
that acquire()/release() match up, so how that is accounted for 
is an implementation detail since you don't provide any access to 
the count.

Same goes for pure functions. You can imagine that all pure 
function calls have a hidden parameter with "ref count storage", 
which it is allowed to modify.

So you might be able do this without breaking the type system, if 
it is builtin, but you have to prove it.

(Basic implementation is to use fat pointers and keep the counter 
in mutable memory, like shared_ptr. Anything beyond that is an 
optimization.)
Nov 09 2021
parent reply deadalnix <deadalnix gmail.com> writes:
On Tuesday, 9 November 2021 at 09:02:32 UTC, Ola Fosheim Grøstad 
wrote:

 On Monday, 8 November 2021 at 22:07:46 UTC, Adam Ruppe wrote:

 On Monday, 8 November 2021 at 21:42:12 UTC, Andrei 
 Alexandrescu wrote:

 - work with qualifiers like T[] does, both RCSlice!(qual T) 
 and qual(RCSlice!T)


 I think this is an incorrect requirement. A const(T) is now 
 allowed to increase a reference count by definition.


 This is technically true, but he wrote "built in", so then you 
 don't have to heed the same constraints. You only have to 
 require that acquire()/release() match up, so how that is 
 accounted for is an implementation detail since you don't 
 provide any access to the count.


Yes. In addition, it is worth mentioning that many functional 
programming languages are using reference counting internally. It 
is so that they can do copy on write instead of copying 
everything all the time for no reason.
Nov 09 2021
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 9 November 2021 at 11:19:38 UTC, deadalnix wrote:

 Yes. In addition, it is worth mentioning that many functional 
 programming languages are using reference counting internally. 
 It is so that they can do copy on write instead of copying 
 everything all the time for no reason.


Functional programming language design and implementation can be 
useful to think about, yes.

The main point is that if you say "built in" then you are allowed 
to do things that seemingly break the rules, but only if you can 
imagine a corresponding simple hypothetical implementation that 
shows that the type system invariants actually are upheld.

One such hypothetical implementation could be to pass in a 
"reference tracker database" as a hidden parameter in all 
function calls and use that for acquire/release/destruction.

This is impractical, but if that is sufficient to uphold all 
invariants, then you only have to prove that an efficient 
implementation is equivalent to the hypothetical implementation.

I think.
Nov 09 2021
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
wrote:

 1. We could not make reference counting work properly in pure 
 functions (a pure function does not mutate data, which goes 
 against manipulating the reference count).


I don't think this one is a real problem, as one can cast a 
function call to pure and go through this. Dirty as hell, but 
done properly it should work even in the face of compiler 
optimization based on purity. GCC or LLVM would have no problem 
optimizing this scafolding away.


 2. Qualifiers compound problems with interlocking: mutable data 
 is known to be single-threaded, so no need for interlocking. 
 Immutable data may be multi-threaded, meaning reference 
 counting needs atomic operations. Const data has unknown 
 origin, which means the information of how data originated 
 (mutable or not) must be saved at runtime.


shared_ptr does atomic operation all the time. The reality is 
that on modern machines, atomic operation are cheap *granted 
there is no contention*. It will certainly limit what the 
optimizer can do, but all in all, it's almost certainly better 
than keeping the info around and doing a runtime check.


 3. Safety forced the use of trusted blocks all over. Not a 
 showstopper, but a complicating factor.

 So if there are any takers on getting RCSlice off the ground, 
 it would be very interesting.


Yes, similar to pure, a bit anoying, but workable.

I think however, you missed several massive problems:
4. Type qualifier transitivity. const RCSlice!T -> const 
RCSlice!(const T) conversion needs to happen transparently.
5. Head mutability. const RCSlice!(const T) -> RCSlice!(const T) 
conversion needs to happen transparently.
6. Safety is impossible to ensure without a runtime check at 
every step, because it is not possible to enforce construction in 
D at the moment, so destrcutors and copy constructor/postblit 
always need to do a runtime check for .init .

I believe 4 and 5 to be impossible in D right now, 6 can be 
solved using a ton of runtime checks.
Nov 08 2021
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 17:15, deadalnix wrote:

 On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu wrote:

 1. We could not make reference counting work properly in pure 
 functions (a pure function does not mutate data, which goes against 
 manipulating the reference count).


 
 I don't think this one is a real problem, as one can cast a function 
 call to pure and go through this. Dirty as hell, but done properly it 
 should work even in the face of compiler optimization based on purity. 
 GCC or LLVM would have no problem optimizing this scafolding away.


If you cast to pure then same compilers will be within their rights to 
optimize away calls - because they're pure.


 2. Qualifiers compound problems with interlocking: mutable data is 
 known to be single-threaded, so no need for interlocking. Immutable 
 data may be multi-threaded, meaning reference counting needs atomic 
 operations. Const data has unknown origin, which means the information 
 of how data originated (mutable or not) must be saved at runtime.


 
 shared_ptr does atomic operation all the time. The reality is that on 
 modern machines, atomic operation are cheap *granted there is no 
 contention*. It will certainly limit what the optimizer can do, but all 
 in all, it's almost certainly better than keeping the info around and 
 doing a runtime check.


In my measurements uncontested atomic increment are 2.5x or more slower 
than the equivalent increment.


 3. Safety forced the use of trusted blocks all over. Not a 
 showstopper, but a complicating factor.

 So if there are any takers on getting RCSlice off the ground, it would 
 be very interesting.


 
 Yes, similar to pure, a bit anoying, but workable.
 
 I think however, you missed several massive problems:
 4. Type qualifier transitivity. const RCSlice!T -> const RCSlice!(const 
 T) conversion needs to happen transparently.
 5. Head mutability. const RCSlice!(const T) -> RCSlice!(const T) 
 conversion needs to happen transparently.
 6. Safety is impossible to ensure without a runtime check at every step, 
 because it is not possible to enforce construction in D at the moment, 
 so destrcutors and copy constructor/postblit always need to do a runtime 
 check for .init .
 
 I believe 4 and 5 to be impossible in D right now, 6 can be solved using 
 a ton of runtime checks.


To which I say, stop posting, start coding.
Nov 08 2021
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 8 November 2021 at 22:38:27 UTC, Andrei Alexandrescu 
wrote:

 I believe 4 and 5 to be impossible in D right now, 6 can be 
 solved using a ton of runtime checks.


 To which I say, stop posting, start coding.


As I said, I believe 4 and 5 to be impossible at the moment.

In fact, I forgot but there is one more unsolvable problem if you 
want the whole thing to be safe: there is no way to prevent 
what's owned the the RCSlice from escaping.

There is a reason why, even though the value is clearly there, 
this endeavor has been on table table for 10 years+ and no real 
progress has been made. You can't just build on quicksand.

If I'm to be honest, I've wasted enough of my time trying to get 
any kind of progress on this front. I've written DIP, I've 
explains how other proposal won't fix the problem, it all proved 
to be a giant waste of time. I'd be thrilled to be able to 
contribute to improve this, but [this message from 
Walter](https://forum.dlang.org/post/slqbqm$2us2$1 digitalmars.com) is a string
indicator that not much has changed and it would be a waste of my time.

At this point, short of some kind of intervention happening, I 
don't really see any significant progress being made on any of 
this.

So yes, I'm coding, every day. But not that, because I know the 
foundation required just aren't there, and I don't have the 
leverage to have anything move on the foundation front.
Nov 08 2021
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 9 November 2021 at 00:44:02 UTC, deadalnix wrote:

 
 So yes, I'm coding, every day. But not that, because I know the 
 foundation required just aren't there, and I don't have the 
 leverage to have anything move on the foundation front.


Do you mean the technical foundation within the compiler to make 
the required analysis possible or something else?
Nov 08 2021
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 19:44, deadalnix wrote:

 On Monday, 8 November 2021 at 22:38:27 UTC, Andrei Alexandrescu wrote:

 I believe 4 and 5 to be impossible in D right now, 6 can be solved 
 using a ton of runtime checks.


 To which I say, stop posting, start coding.


 
 As I said, I believe 4 and 5 to be impossible at the moment.


Do 1, 2, and 3.
Nov 11 2021
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
On Monday, 8 November 2021 at 22:38:27 UTC, Andrei Alexandrescu 
wrote:

 I don't think this one is a real problem, as one can cast a 
 function call to pure and go through this. Dirty as hell, but 
 done properly it should work even in the face of compiler 
 optimization based on purity. GCC or LLVM would have no 
 problem optimizing this scafolding away.


 If you cast to pure then same compilers will be within their 
 rights to optimize away calls - because they're pure.


I think you might be right here, unless one goes through some 
serious stunt by returning some kind of result that you'd use so 
the compiler cannot optimize it away.
Nov 08 2021
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 8 November 2021 at 22:38:27 UTC, Andrei Alexandrescu 
wrote:

 shared_ptr does atomic operation all the time. The reality is 
 that on modern machines, atomic operation are cheap *granted 
 there is no contention*. It will certainly limit what the 
 optimizer can do, but all in all, it's almost certainly better 
 than keeping the info around and doing a runtime check.


 In my measurements uncontested atomic increment are 2.5x or 
 more slower than the equivalent increment.


Do you mind sharing this?

I find that curious, because load/stores on x86 are almost 
sequentially consistent by default, and you don't even need 
sequential consistency to increment the counter to begin with, so 
a good old `inc` instruction is enough.

I'd look to look at what's the compiler is doing here, because 
maybe we are trying to fix the wrong problem.
Nov 08 2021
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 20:12, deadalnix wrote:

 On Monday, 8 November 2021 at 22:38:27 UTC, Andrei Alexandrescu wrote:

 shared_ptr does atomic operation all the time. The reality is that on 
 modern machines, atomic operation are cheap *granted there is no 
 contention*. It will certainly limit what the optimizer can do, but 
 all in all, it's almost certainly better than keeping the info around 
 and doing a runtime check.


 In my measurements uncontested atomic increment are 2.5x or more 
 slower than the equivalent increment.


 
 Do you mind sharing this?


Quick and dirty code that's been long overwritten. Just redo it. Use C++ 
as a baseline.


 I find that curious, because load/stores on x86 are almost sequentially 
 consistent by default, and you don't even need sequential consistency to 
 increment the counter to begin with, so a good old `inc` instruction is 
 enough.
 
 I'd look to look at what's the compiler is doing here, because maybe we 
 are trying to fix the wrong problem.


The overhead comes from the bus "lock" operation which both gcc and 
clang emit: https://godbolt.org/z/zx4cMYE39
Nov 11 2021
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Nov 08, 2021 at 10:15:09PM +0000, deadalnix via Digitalmars-d wrote:
[...]

 I think however, you missed several massive problems:
 4. Type qualifier transitivity. const RCSlice!T -> const
 RCSlice!(const T) conversion needs to happen transparently.


The only way this can happen is via a language change.  The only way
arrays get to enjoy such benefits is because the necessary implicit
conversion rules are baked into the language.  User-defined types do not
have such privileges, and there is currently no combination of language
constructs that can express such a thing.



 5. Head mutability. const RCSlice!(const T) -> RCSlice!(const T)
 conversion needs to happen transparently.


[...]

Ditto.

Basically, for something to be ref-counted, you need:

1) To attach a counter to the type that remains mutable in spite of the
payload being possibly const or immutable.

2) Due to D's mutable/immutable implicit conversion to const, for a
ref-counted type to avoid being cumbersome to use (e.g., pass mutable
RefCounted!T to a function receiving RefCounted!(const T) without
needing to cast / copy / any of various workarounds), a user-defined
type needs to be able to declare implicit conversions of this sort.
Current language rules do not permit this.


T

-- 
Fact is stranger than fiction.
Nov 08 2021
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 09.11.21 01:12, H. S. Teoh wrote:

 On Mon, Nov 08, 2021 at 10:15:09PM +0000, deadalnix via Digitalmars-d wrote:
 [...]

 I think however, you missed several massive problems:
 4. Type qualifier transitivity. const RCSlice!T -> const
 RCSlice!(const T) conversion needs to happen transparently.


 
 The only way this can happen is via a language change.  The only way
 arrays get to enjoy such benefits is because the necessary implicit
 conversion rules are baked into the language.  User-defined types do not
 have such privileges, and there is currently no combination of language
 constructs that can express such a thing.
 
 

 5. Head mutability. const RCSlice!(const T) -> RCSlice!(const T)
 conversion needs to happen transparently.


 [...]
 
 Ditto.
 
 Basically, for something to be ref-counted, you need:
 
 1) To attach a counter to the type that remains mutable in spite of the
 payload being possibly const or immutable.
 ...


And to do that, you have to explicitly specify the semantics of 
qualifiers based on a set of allowed rewrites, and there needs to be a 
way to ensure low-level bookkeeping does not get elided. This was the 
__mutable proposal.
Nov 08 2021
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 19:12, H. S. Teoh wrote:

 On Mon, Nov 08, 2021 at 10:15:09PM +0000, deadalnix via Digitalmars-d wrote:
 [...]

 I think however, you missed several massive problems:
 4. Type qualifier transitivity. const RCSlice!T -> const
 RCSlice!(const T) conversion needs to happen transparently.


 
 The only way this can happen is via a language change.  The only way
 arrays get to enjoy such benefits is because the necessary implicit
 conversion rules are baked into the language.  User-defined types do not
 have such privileges, and there is currently no combination of language
 constructs that can express such a thing.


I keep on thinking of proposing opFunCall() that is called whenever an 
object is passed by value into a function. The lowering for such objects 
would be:

fun(obj);

===>

fun(obj.opFunCall());
Nov 11 2021
parent deadalnix <deadalnix gmail.com> writes:
On Thursday, 11 November 2021 at 21:46:29 UTC, Andrei 
Alexandrescu wrote:

 I keep on thinking of proposing opFunCall() that is called 
 whenever an object is passed by value into a function. The 
 lowering for such objects would be:

 fun(obj);

 ===>

 fun(obj.opFunCall());


This has little to do with function calls. The same problem will 
exist for assignment for instance. Now you could also override 
opAssign, but you see where I'm going with that.

Not only does the head const needs to be dropped, but this needs 
to happen implicitly AND aggressively.

By implicitly, I meant hat the conversion doesn't require extra 
syntax, as in:
```d
const int[] a;
const(int)[] b = a; // OK
```

And by aggressively, I mean that it happens as soon as possible, 
not simply when a conversion is needed. For instance:
```d

void foo(T)(T t) {
     writeln(T.stringof);
}

void main() {
     const int[] a;
     foo(a); // Prints const(int)[]
}
```
Nov 12 2021
prev sibling parent sclytrack <fake hotmail.com> writes:
On Monday, 8 November 2021 at 22:15:09 UTC, deadalnix wrote:

 On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
 wrote:

 1. We could not make reference counting work properly in pure 
 functions (a pure function does not mutate data, which goes 
 against manipulating the reference count).


 I don't think this one is a real problem, as one can cast a 
 function call to pure and go through this. Dirty as hell, but 
 done properly it should work even in the face of compiler 
 optimization based on purity. GCC or LLVM would have no problem 
 optimizing this scafolding away.


 2. Qualifiers compound problems with interlocking: mutable 
 data is known to be single-threaded, so no need for 
 interlocking. Immutable data may be multi-threaded, meaning 
 reference counting needs atomic operations. Const data has 
 unknown origin, which means the information of how data 
 originated (mutable or not) must be saved at runtime.


 shared_ptr does atomic operation all the time. The reality is 
 that on modern machines, atomic operation are cheap *granted 
 there is no contention*. It will certainly limit what the 
 optimizer can do, but all in all, it's almost certainly better 
 than keeping the info around and doing a runtime check.


 3. Safety forced the use of trusted blocks all over. Not a 
 showstopper, but a complicating factor.

 So if there are any takers on getting RCSlice off the ground, 
 it would be very interesting.


 Yes, similar to pure, a bit anoying, but workable.

 I think however, you missed several massive problems:
 4. Type qualifier transitivity. const RCSlice!T -> const 
 RCSlice!(const T) conversion needs to happen transparently.
 5. Head mutability. const RCSlice!(const T) -> RCSlice!(const



It's a copy of the head. ".hidup" or hdup. Head-only duplication.



 T) conversion needs to happen transparently.
 6. Safety is impossible to ensure without a runtime check at 
 every step, because it is not possible to enforce construction 
 in D at the moment, so destrcutors and copy 
 constructor/postblit always need to do a runtime check for 
 .init .

 I believe 4 and 5 to be impossible in D right now, 6 can be 
 solved using a ton of runtime checks.



1) mutable <--> shared immutable
2) inout containers
3) locked unlocked

1---

2---
Container do not to modify the possibly mutable elements.
Same code for mutable and immutable.

container!inout


container!Mutable and container!immutable could have different 
code, different
memory layout, any conversion needs to rearrange that.

3---
//typeof(obj) == "locked TypeName"

synchronize(obj)
{
   //typeof(obj) == "TypeName "
}

//typeof(obj) == "locked TypeName"

?how to keep shared and unshared separate but still reuse the 
same code?
---
Nov 19 2021
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 09/11/2021 10:42 AM, Andrei Alexandrescu wrote:

 1. We could not make reference counting work properly in pure functions 
 (a pure function does not mutate data, which goes against manipulating 
 the reference count).


The concern with pure for me is a bit further down the road, 
deallocation (extending/shrinking too). An allocator has state dependent 
on an external global library (i.e. kernel), there is no way around this.

pure should be  nogc effectively. We are faking it right now and that is 
not ok if you care about pure.


 2. Qualifiers compound problems with interlocking: mutable data is known 
 to be single-threaded, so no need for interlocking. Immutable data may 
 be multi-threaded, meaning reference counting needs atomic operations. 
 Const data has unknown origin, which means the information of how data 
 originated (mutable or not) must be saved at runtime.


This is not true.

All pointers have an unknown origin and transitivity of thread ownership 
unless proven otherwise by the compiler during compilation.

Lastly, as far as I'm concerned a reference counted struct should never 
be const (I've gone to great lengths to make it VERY undesirable to end 
up with a const reference counted struct in my stuff). Deadalnix covered 
this with his point 5.
Nov 08 2021
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 17:26, rikki cattermole wrote:

 a reference counted struct should never be const


So then we're toast. If that's the case we're looking at the most 
important challenge to the D language right now.
Nov 08 2021
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 09/11/2021 11:40 AM, Andrei Alexandrescu wrote:

 On 2021-11-08 17:26, rikki cattermole wrote:

 a reference counted struct should never be const


 
 So then we're toast. If that's the case we're looking at the most 
 important challenge to the D language right now.


I disagree with it as being the most important (empowering scope, value 
no runtime exceptions are at the top).

However I have thought about this quite a bit due to signatures.

In a large number of cases of alias this, its due to wanting a wrapper 
struct around another type that is being managed using i.e. reference 
counting.

So, I think what we are missing is some form of custom reference type.

A type that the compiler understands has some mutable state controlling 
lifetimes (that is optional) and to treat it as another type whose state 
is being stored on the heap some place external.

That is an observation of mine, although it would fit in well with 
const. Throw in ARC, that may solve our problems here.
Nov 08 2021
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 8 November 2021 at 22:40:03 UTC, Andrei Alexandrescu 
wrote:

 On 2021-11-08 17:26, rikki cattermole wrote:

 a reference counted struct should never be const


 So then we're toast. If that's the case we're looking at the 
 most important challenge to the D language right now.


Not necessarily, but this is in fact the same problem as the head 
mutable problem.

If `const RCSlice!(const T)` can convert to `RCSlice!(const T)`, 
which it should to have the same semantic as slice.

The reference counting problem goes away if you can mutate the 
head.
Nov 08 2021
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/8/21 8:04 PM, deadalnix wrote:

 On Monday, 8 November 2021 at 22:40:03 UTC, Andrei Alexandrescu wrote:

 On 2021-11-08 17:26, rikki cattermole wrote:

 a reference counted struct should never be const


 So then we're toast. If that's the case we're looking at the most 
 important challenge to the D language right now.


 
 Not necessarily, but this is in fact the same problem as the head 
 mutable problem.
 
 If `const RCSlice!(const T)` can convert to `RCSlice!(const T)`, which 
 it should to have the same semantic as slice.
 
 The reference counting problem goes away if you can mutate the head.


The reference count cannot be in the head, it has to be in the block. So 
this conversion is actually unsound.

-Steve
Nov 08 2021
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Nov 08, 2021 at 10:38:59PM -0500, Steven Schveighoffer via
Digitalmars-d wrote:

 On 11/8/21 8:04 PM, deadalnix wrote:

 On Monday, 8 November 2021 at 22:40:03 UTC, Andrei Alexandrescu wrote:

 On 2021-11-08 17:26, rikki cattermole wrote:

 a reference counted struct should never be const


 
 So then we're toast. If that's the case we're looking at the most
 important challenge to the D language right now.


 
 Not necessarily, but this is in fact the same problem as the head
 mutable problem.
 
 If `const RCSlice!(const T)` can convert to `RCSlice!(const T)`,
 which it should to have the same semantic as slice.
 
 The reference counting problem goes away if you can mutate the head.


 
 The reference count cannot be in the head, it has to be in the block.
 So this conversion is actually unsound.


[...]

It could work if the block has a mutable counter at the top and the rest
is immutable/const.  But the current type system cannot express such a
thing; if the head is const, then it's const all the way down.


T

-- 
Two wrongs don't make a right; but three rights do make a left...
Nov 08 2021
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-09 0:30, H. S. Teoh wrote:

 On Mon, Nov 08, 2021 at 10:38:59PM -0500, Steven Schveighoffer via
Digitalmars-d wrote:

 On 11/8/21 8:04 PM, deadalnix wrote:

 On Monday, 8 November 2021 at 22:40:03 UTC, Andrei Alexandrescu wrote:

 On 2021-11-08 17:26, rikki cattermole wrote:

 a reference counted struct should never be const


 So then we're toast. If that's the case we're looking at the most
 important challenge to the D language right now.


 Not necessarily, but this is in fact the same problem as the head
 mutable problem.

 If `const RCSlice!(const T)` can convert to `RCSlice!(const T)`,
 which it should to have the same semantic as slice.

 The reference counting problem goes away if you can mutate the head.


 The reference count cannot be in the head, it has to be in the block.
 So this conversion is actually unsound.


 [...]
 
 It could work if the block has a mutable counter at the top and the rest
 is immutable/const.  But the current type system cannot express such a
 thing; if the head is const, then it's const all the way down.


The mutable counter can be at a negative address that is not accessible 
to the "normal" type system. There's nothing wrong with having a piece 
of mutable data next to a piece of immutable data.
Nov 11 2021
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 20:04, deadalnix wrote:

 On Monday, 8 November 2021 at 22:40:03 UTC, Andrei Alexandrescu wrote:

 On 2021-11-08 17:26, rikki cattermole wrote:

 a reference counted struct should never be const


 So then we're toast. If that's the case we're looking at the most 
 important challenge to the D language right now.


 
 Not necessarily, but this is in fact the same problem as the head 
 mutable problem.
 
 If `const RCSlice!(const T)` can convert to `RCSlice!(const T)`, which 
 it should to have the same semantic as slice.
 
 The reference counting problem goes away if you can mutate the head.


I very much wish that were the case. From what I remember working on the 
code, there were multiple other challenges.
Nov 11 2021
prev sibling next sibling parent Elronnd <elronnd elronnd.net> writes:
On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
wrote:

 1. We could not make reference counting work properly in pure 
 functions (a pure function does not mutate data, which goes 
 against manipulating the reference count).


pure functions can mutate data passed to them.  And for a struct 
method (such as a copy constructor) this includes the struct's 
fields.
Nov 08 2021
prev sibling next sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
wrote:

 Eduard Staniloiu (at the time a student I was mentoring) tried 
 really hard to define a built-in slice RCSlice(T) from the 
 following spec:


Your spec is very focused on homogenizing the API for GC and RC 
slices as much as (or rather, more than) possible.

But, it isn't possible to make a truly ` safe`, general-purpose 
RC slice in D *at all* right now, even without all the additional 
constraints that you are placing on the problem.

Borrowing is required for a general-purpose RC type, so that the 
payload can actually be used without a reference to the payload 
escaping outside the lifetime of the counting reference. But, the 
effective lifetime of the counting reference is not visible to 
the `scope` borrow checker, because at any point the reference's 
destructor may be manually called, potentially `free`ing the 
payload while there is still an extant borrowed reference.

With current language semantics, the destructor (and any other 
similar operations, such as reassignment) of the reference type 
must be ` system` to prevent misuse of the destructor in ` safe` 
code.
     https://issues.dlang.org/show_bug.cgi?id=21981

The solution to this problem is to introduce some way of telling 
the compiler, "this destructor is ` safe` to call automatically 
at the end of the object's scope, but ` system` to call early or 
manually."

Also, the DIP1000 implementation is very buggy and incomplete; it 
doesn't work right for several kinds of indirections, including 
slices and `class` objects:
     
https://forum.dlang.org/thread/zsjqqeftthxtfkytrnwp forum.dlang.org?page=1


 - work in pure code just like T[] does
 - work with qualifiers like T[] does, both RCSlice!(qual T) and 
 qual(RCSlice!T)


I wrote a semi-` safe` reference counting system for D recently, 
which includes slices, weak references, shared references with 
atomic counting, etc. It works well enough to be useful to me 
(way better than manual `malloc` and `free`), but not well enough 
to be a candidate for the standard library due to the above 
compiler bugs.

`RCSlice!(qual T)` is no problem; the reference count and the 
payload do not need to use the same qualifiers. Whether the count 
is `shared` or not can be tracked statically as part of the RC 
type, so the "const might be immutable, and therefore have a 
shared reference count, and therefore require expensive atomic 
operations" issue that you raise is easy enough to solve.

On the other hand, `pure` compatibility and a usable 
`immutable(RCSlice!T)` are mutually exclusive, I think:

**Either** the reference count is part of the target of the RC 
type's internal indirection, in which case it can be mutated in 
`pure` code, but is frozen by an outer, transitive `immutable`, 
**or** the reference count is conceptualized as an entry in a 
separate global data structure which can be located by using the 
address of the payload as a key, meaning that incrementing it 
does not mutate the target, but is im`pure`.

I believe the former solution (compatible with `pure`, but not 
outer `immutable`) is preferable since it is the most honest, 
least weird solution, and therefore least likely to trip up 
either the compiler developers or the users somehow.

What you seem to be asking for instead is a way to trick the type 
system into agreeing that mutating a reference count doesn't 
actually mutate anything, which is nonsense. If that's really 
necessary for some reason, it needs to be special cased into the 
language spec, like how `pure` explicitly permits memory 
allocation.
Nov 08 2021
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 09/11/2021 2:14 PM, tsbockman wrote:

 What you seem to be asking for instead is a way to trick the type system 
 into agreeing that mutating a reference count doesn't actually mutate 
 anything, which is nonsense. If that's really necessary for some reason, 
 it needs to be special cased into the language spec, like how `pure` 
 explicitly permits memory allocation.


Imagine saying to someone:

Yes you have made this struct immutable.

Yes you have set this bit of memory containing that immutable struct to 
read only.

Yes you ended up with a crashed program because that immutable struct 
went ahead and tried to write to that read only memory.

And yes, I understand that you couldn't have known that a field that you 
didn't write the implementation of used an escape hatch to write to 
const data.

It doesn't make sense.
Nov 08 2021
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 09, 2021 at 02:22:26PM +1300, rikki cattermole via Digitalmars-d
wrote:

 
 On 09/11/2021 2:14 PM, tsbockman wrote:

 What you seem to be asking for instead is a way to trick the type
 system into agreeing that mutating a reference count doesn't
 actually mutate anything, which is nonsense. If that's really
 necessary for some reason, it needs to be special cased into the
 language spec, like how `pure` explicitly permits memory allocation.


 
 Imagine saying to someone:
 
 Yes you have made this struct immutable.
 
 Yes you have set this bit of memory containing that immutable struct
 to read only.
 
 Yes you ended up with a crashed program because that immutable struct
 went ahead and tried to write to that read only memory.
 
 And yes, I understand that you couldn't have known that a field that
 you didn't write the implementation of used an escape hatch to write
 to const data.
 
 It doesn't make sense.


Yes, subverting the type system with __mutable doesn't make sense. It
just adds all kinds of loopholes and other unwanted interactions to the
language.  A better approach is to have a nice way of converting
const(Wrapper!T) -> Wrapper!(const T), so that we can implement a
refcount in Wrapper without violating the type system.


T

-- 
Freedom of speech: the whole world has no right *not* to hear my spouting off!
Nov 08 2021
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 20:22, rikki cattermole wrote:

 
 On 09/11/2021 2:14 PM, tsbockman wrote:

 What you seem to be asking for instead is a way to trick the type 
 system into agreeing that mutating a reference count doesn't actually 
 mutate anything, which is nonsense. If that's really necessary for 
 some reason, it needs to be special cased into the language spec, like 
 how `pure` explicitly permits memory allocation.


 
 Imagine saying to someone:
 
 Yes you have made this struct immutable.
 
 Yes you have set this bit of memory containing that immutable struct to 
 read only.
 
 Yes you ended up with a crashed program because that immutable struct 
 went ahead and tried to write to that read only memory.
 
 And yes, I understand that you couldn't have known that a field that you 
 didn't write the implementation of used an escape hatch to write to 
 const data.
 
 It doesn't make sense.


It makes perfect sense.

Yes you have made this struct immutable.

Yes you have set this bit of memory containing that immutable struct to 
read only.

Yes the immutable structure has metadata associated with it that is not 
mutable.

Yes that metadata can be in fact lying next to the data itself (for 
optimization purposes) even though it's conceptually global.

No you cannot end up with a crashed program because the mutable metadata 
and the immutable part are carefully handled so as to not interfere.

And yes, I understand that you may find that odd.
Nov 11 2021
parent rikki cattermole <rikki cattermole.co.nz> writes:
I think that we just came to an agreement that the metadata stuff should 
be handled separately potentially using different memory.

I suggested this as an escape hatch with my custom reference type back 
in my second comment!

https://forum.dlang.org/post/smc9r2$7nh$1 digitalmars.com
Nov 11 2021
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 9 November 2021 at 01:14:24 UTC, tsbockman wrote:

 Borrowing is required for a general-purpose RC type, so that 
 the payload can actually be used without a reference to the 
 payload escaping outside the lifetime of the counting 
 reference. But, the effective lifetime of the counting 
 reference is not visible to the `scope` borrow checker, because 
 at any point the reference's destructor may be manually called, 
 potentially `free`ing the payload while there is still an 
 extant borrowed reference.

 With current language semantics, the destructor (and any other 
 similar operations, such as reassignment) of the reference type 
 must be ` system` to prevent misuse of the destructor in 
 ` safe` code.
     https://issues.dlang.org/show_bug.cgi?id=21981

 The solution to this problem is to introduce some way of 
 telling the compiler, "this destructor is ` safe` to call 
 automatically at the end of the object's scope, but ` system` 
 to call early or manually."


I believe it is also possible to make this ` safe` by doing 
borrow checking at runtime, although it would introduce some 
overhead, and make the API less ergonomic.

Maybe the best compromise we can reach in the current language is 
to offer a ` safe` borrow-checked interface alongside an 
unchecked ` system` interface.
Nov 08 2021
next sibling parent tsbockman <thomas.bockman gmail.com> writes:
On Tuesday, 9 November 2021 at 03:43:01 UTC, Paul Backus wrote:

 On Tuesday, 9 November 2021 at 01:14:24 UTC, tsbockman wrote:

 With current language semantics, the destructor (and any other 
 similar operations, such as reassignment) of the reference 
 type must be ` system` to prevent misuse of the destructor in 
 ` safe` code.
     https://issues.dlang.org/show_bug.cgi?id=21981

 The solution to this problem is to introduce some way of 
 telling the compiler, "this destructor is ` safe` to call 
 automatically at the end of the object's scope, but ` system` 
 to call early or manually."


 I believe it is also possible to make this ` safe` by doing 
 borrow checking at runtime, although it would introduce some 
 overhead, and make the API less ergonomic.


How? The general purpose runtime borrow checking schemes I've 
considered all have the same fundamental problems as reference 
counting in general, when examined closely.
Nov 08 2021
prev sibling next sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Tuesday, 9 November 2021 at 03:43:01 UTC, Paul Backus wrote:

 On Tuesday, 9 November 2021 at 01:14:24 UTC, tsbockman wrote:

 With current language semantics, the destructor (and any other 
 similar operations, such as reassignment) of the reference 
 type must be ` system` to prevent misuse of the destructor in 
 ` safe` code.
     https://issues.dlang.org/show_bug.cgi?id=21981

 The solution to this problem is to introduce some way of 
 telling the compiler, "this destructor is ` safe` to call 
 automatically at the end of the object's scope, but ` system` 
 to call early or manually."


 I believe it is also possible to make this ` safe` by doing 
 borrow checking at runtime, although it would introduce some 
 overhead, and make the API less ergonomic.


How? All of the runtime borrow checking schemes that I have 
considered turn out to have the same fundamental problems as 
reference counting, when examined closely.
Nov 08 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 9 November 2021 at 06:32:40 UTC, tsbockman wrote:

 On Tuesday, 9 November 2021 at 03:43:01 UTC, Paul Backus wrote:

 On Tuesday, 9 November 2021 at 01:14:24 UTC, tsbockman wrote:

 With current language semantics, the destructor (and any 
 other similar operations, such as reassignment) of the 
 reference type must be ` system` to prevent misuse of the 
 destructor in ` safe` code.
     https://issues.dlang.org/show_bug.cgi?id=21981

 The solution to this problem is to introduce some way of 
 telling the compiler, "this destructor is ` safe` to call 
 automatically at the end of the object's scope, but ` system` 
 to call early or manually."


 I believe it is also possible to make this ` safe` by doing 
 borrow checking at runtime, although it would introduce some 
 overhead, and make the API less ergonomic.


 How? All of the runtime borrow checking schemes that I have 
 considered turn out to have the same fundamental problems as 
 reference counting, when examined closely.


Here's a basic sketch of the scheme I had in mind: 
https://gist.github.com/run-dlang/d1982a29423b2cb545bc9fa452d94c5e

It's entirely possible I've overlooked something and this is 
actually unsound. As Andrei says: "destroy!"
Nov 09 2021
next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 9 November 2021 at 17:04:29 UTC, Paul Backus wrote:


 It's entirely possible I've overlooked something and this is 
 actually unsound. As Andrei says: "destroy!"


Destruction commenced under the gist.
Nov 09 2021
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Tuesday, 9 November 2021 at 17:04:29 UTC, Paul Backus wrote:

 Here's a basic sketch of the scheme I had in mind: 
 https://gist.github.com/run-dlang/d1982a29423b2cb545bc9fa452d94c5e

 It's entirely possible I've overlooked something and this is 
 actually unsound. As Andrei says: "destroy!"


There are a few problems with the code, indeed.
1/ You need to add runtime check, because in D, destructed 
objects may not have been constructed, so you need to be able to 
destroy .init . in your case, pointers will be null and you'll 
segfault.
2/ For a struct, you cannot guarantee that the struct's method 
don't escape the struct pointer, so none of this can be safe in 
any way.
Nov 10 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 10 November 2021 at 13:48:53 UTC, deadalnix wrote:

 On Tuesday, 9 November 2021 at 17:04:29 UTC, Paul Backus wrote:

 Here's a basic sketch of the scheme I had in mind: 
 https://gist.github.com/run-dlang/d1982a29423b2cb545bc9fa452d94c5e

 It's entirely possible I've overlooked something and this is 
 actually unsound. As Andrei says: "destroy!"


 There are a few problems with the code, indeed.
 1/ You need to add runtime check, because in D, destructed 
 objects may not have been constructed, so you need to be able 
 to destroy .init . in your case, pointers will be null and 
 you'll segfault.


Destroying .init works fine here. `free(null)` is guaranteed by 
the C standard to be a no-op.


 2/ For a struct, you cannot guarantee that the struct's method 
 don't escape the struct pointer, so none of this can be safe in 
 any way.


Good point. I guess you'd need transitive `scope` for this, at 
minimum.
Nov 10 2021
parent deadalnix <deadalnix gmail.com> writes:
On Wednesday, 10 November 2021 at 14:47:34 UTC, Paul Backus wrote:

 Destroying .init works fine here. `free(null)` is guaranteed by 
 the C standard to be a no-op.


Decrementing the reference count is going to segfault if the 
pointer to the owner is null.
Nov 10 2021
prev sibling parent tsbockman <thomas.bockman gmail.com> writes:
On Tuesday, 9 November 2021 at 17:04:29 UTC, Paul Backus wrote:

 On Tuesday, 9 November 2021 at 06:32:40 UTC, tsbockman wrote:

 On Tuesday, 9 November 2021 at 03:43:01 UTC, Paul Backus wrote:
 How? All of the runtime borrow checking schemes that I have 
 considered turn out to have the same fundamental problems as 
 reference counting, when examined closely.


 Here's a basic sketch of the scheme I had in mind: 
 https://gist.github.com/run-dlang/d1982a29423b2cb545bc9fa452d94c5e


Thanks.


 It's entirely possible I've overlooked something and this is 
 actually unsound. As Andrei says: "destroy!"


The general idea is possibly sound, although the current 
implementation definitely is not.

If your runtime borrow checks were combined with the language 
enhancements that I suggested in our previous discussion, I think 
it might actually be possible to write a nice ` safe` reference 
counting API that supports ` safe` 
reassignment/reallocation/resizing.

https://forum.dlang.org/post/zsjqqeftthxtfkytrnwp forum.dlang.org
https://issues.dlang.org/show_bug.cgi?id=21981
Nov 10 2021
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 22:43, Paul Backus wrote:

 On Tuesday, 9 November 2021 at 01:14:24 UTC, tsbockman wrote:

 Borrowing is required for a general-purpose RC type, so that the 
 payload can actually be used without a reference to the payload 
 escaping outside the lifetime of the counting reference. But, the 
 effective lifetime of the counting reference is not visible to the 
 `scope` borrow checker, because at any point the reference's 
 destructor may be manually called, potentially `free`ing the payload 
 while there is still an extant borrowed reference.

 With current language semantics, the destructor (and any other similar 
 operations, such as reassignment) of the reference type must be 
 ` system` to prevent misuse of the destructor in ` safe` code.
     https://issues.dlang.org/show_bug.cgi?id=21981

 The solution to this problem is to introduce some way of telling the 
 compiler, "this destructor is ` safe` to call automatically at the end 
 of the object's scope, but ` system` to call early or manually."


 
 I believe it is also possible to make this ` safe` by doing borrow 
 checking at runtime, although it would introduce some overhead, and make 
 the API less ergonomic.
 
 Maybe the best compromise we can reach in the current language is to 
 offer a ` safe` borrow-checked interface alongside an unchecked 
 ` system` interface.


This is interesting. Maybe you could add a little detail to the bug 
report or to this thread?
Nov 11 2021
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-08 20:14, tsbockman wrote:

 On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu wrote:

 Eduard Staniloiu (at the time a student I was mentoring) tried really 
 hard to define a built-in slice RCSlice(T) from the following spec:


 
 Your spec is very focused on homogenizing the API for GC and RC slices 
 as much as (or rather, more than) possible.
 
 But, it isn't possible to make a truly ` safe`, general-purpose RC slice 
 in D *at all* right now, even without all the additional constraints 
 that you are placing on the problem.
 
 Borrowing is required for a general-purpose RC type, so that the payload 
 can actually be used without a reference to the payload escaping outside 
 the lifetime of the counting reference. But, the effective lifetime of 
 the counting reference is not visible to the `scope` borrow checker, 
 because at any point the reference's destructor may be manually called, 
 potentially `free`ing the payload while there is still an extant 
 borrowed reference.


Walter would argue that his work on scope makes that possible too. The 
short of it is getting an answer to "can it be done" is important in and 
of itself because it gives us hints on what needs to be done.


 With current language semantics, the destructor (and any other similar 
 operations, such as reassignment) of the reference type must be 
 ` system` to prevent misuse of the destructor in ` safe` code.
      https://issues.dlang.org/show_bug.cgi?id=21981
 
 The solution to this problem is to introduce some way of telling the 
 compiler, "this destructor is ` safe` to call automatically at the end 
 of the object's scope, but ` system` to call early or manually."


This I'm not worried about. ` trusted` should be fine in low-level code, 
no? `destroy` is not safe. What am I missing?


 Also, the DIP1000 implementation is very buggy and incomplete; it 
 doesn't work right for several kinds of indirections, including slices 
 and `class` objects:
 https://forum.dlang.org/thread/zsjqqeftthxtfkytrnwp forum.dlang.org?page=1


<nod>


 - work in pure code just like T[] does
 - work with qualifiers like T[] does, both RCSlice!(qual T) and 
 qual(RCSlice!T)


 
 I wrote a semi-` safe` reference counting system for D recently, which 
 includes slices, weak references, shared references with atomic 
 counting, etc. It works well enough to be useful to me (way better than 
 manual `malloc` and `free`), but not well enough to be a candidate for 
 the standard library due to the above compiler bugs.


Interesting, have you published the code?


 `RCSlice!(qual T)` is no problem; the reference count and the payload do 
 not need to use the same qualifiers. Whether the count is `shared` or 
 not can be tracked statically as part of the RC type, so the "const 
 might be immutable, and therefore have a shared reference count, and 
 therefore require expensive atomic operations" issue that you raise is 
 easy enough to solve.
 
 On the other hand, `pure` compatibility and a usable 
 `immutable(RCSlice!T)` are mutually exclusive, I think:
 
 **Either** the reference count is part of the target of the RC type's 
 internal indirection, in which case it can be mutated in `pure` code, 
 but is frozen by an outer, transitive `immutable`, **or** the reference 
 count is conceptualized as an entry in a separate global data structure 
 which can be located by using the address of the payload as a key, 
 meaning that incrementing it does not mutate the target, but is im`pure`.


Yah, my thoughts exactly. My money is on the latter option. The 
reference counter is metadata, not data, and should not be treated as 
part of the data even though implementation-wise it is.

Eduard and I stopped short of being able to formalize this.


 I believe the former solution (compatible with `pure`, but not outer 
 `immutable`) is preferable since it is the most honest, least weird 
 solution, and therefore least likely to trip up either the compiler 
 developers or the users somehow.
 
 What you seem to be asking for instead is a way to trick the type system 
 into agreeing that mutating a reference count doesn't actually mutate 
 anything, which is nonsense. If that's really necessary for some reason, 
 it needs to be special cased into the language spec, like how `pure` 
 explicitly permits memory allocation.


It's not nonsense. Or if it is a lot other things can be categorized as 
nonsense, such as the GC recycling memory of one type and presenting it 
as a different type etc.
Nov 11 2021
next sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 11 November 2021 at 23:17:53 UTC, Andrei 
Alexandrescu wrote:

 On 2021-11-08 20:14, tsbockman wrote:

 With current language semantics, the destructor (and any other 
 similar operations, such as reassignment) of the reference 
 type must be ` system` to prevent misuse of the destructor in 
 ` safe` code.
      https://issues.dlang.org/show_bug.cgi?id=21981
 
 The solution to this problem is to introduce some way of 
 telling the compiler, "this destructor is ` safe` to call 
 automatically at the end of the object's scope, but ` system` 
 to call early or manually."


 This I'm not worried about. ` trusted` should be fine in 
 low-level code, no? `destroy` is not safe. What am I missing?


In the program below, if `Owner.__dtor` is ` system`, then 
` safe` code like `main` can't instantiate `Owner`.

But, if `Owner.__dtor` is ` trusted`, then memory safety can be 
violated with a "use after free".

```D
module app;

struct Owner {
     import core.stdc.stdlib : malloc, free;

     private int* x;

     this(const(int) x)  trusted {
         this.x = cast(int*) malloc(int.sizeof);
         *(this.x) = x;
     }
     inout(int*) borrow() return inout  safe {
         return x; }
     ~this()  trusted {
         free(x);
         x = null;
     }
}

void main()  safe {
     import std.stdio : writeln;

     Owner owner = 1;
     scope x = owner.borrow;
     writeln(*x);
     destroy!false(owner); // owner.__dtor(); // is also  safe
     writeln(*x += 1); // use after free!
}
```
Nov 11 2021
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-11 22:41, tsbockman wrote:

 On Thursday, 11 November 2021 at 23:17:53 UTC, Andrei Alexandrescu wrote:

 On 2021-11-08 20:14, tsbockman wrote:

 With current language semantics, the destructor (and any other 
 similar operations, such as reassignment) of the reference type must 
 be ` system` to prevent misuse of the destructor in ` safe` code.
      https://issues.dlang.org/show_bug.cgi?id=21981

 The solution to this problem is to introduce some way of telling the 
 compiler, "this destructor is ` safe` to call automatically at the 
 end of the object's scope, but ` system` to call early or manually."


 This I'm not worried about. ` trusted` should be fine in low-level 
 code, no? `destroy` is not safe. What am I missing?


 
 In the program below, if `Owner.__dtor` is ` system`, then ` safe` code 
 like `main` can't instantiate `Owner`.
 
 But, if `Owner.__dtor` is ` trusted`, then memory safety can be violated 
 with a "use after free".
 
 ```D
 module app;
 
 struct Owner {
      import core.stdc.stdlib : malloc, free;
 
      private int* x;
 
      this(const(int) x)  trusted {
          this.x = cast(int*) malloc(int.sizeof);
          *(this.x) = x;
      }
      inout(int*) borrow() return inout  safe {
          return x; }
      ~this()  trusted {
          free(x);
          x = null;
      }
 }
 
 void main()  safe {
      import std.stdio : writeln;
 
      Owner owner = 1;
      scope x = owner.borrow;
      writeln(*x);
      destroy!false(owner); // owner.__dtor(); // is also  safe
      writeln(*x += 1); // use after free!
 }
 ```


Interesting, thanks. I submitted 
https://issues.dlang.org/show_bug.cgi?id=22507.
Nov 12 2021
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Nov 11, 2021 at 06:17:53PM -0500, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 2021-11-08 20:14, tsbockman wrote:


[...]

 **Either** the reference count is part of the target of the RC
 type's internal indirection, in which case it can be mutated in
 `pure` code, but is frozen by an outer, transitive `immutable`,
 **or** the reference count is conceptualized as an entry in a
 separate global data structure which can be located by using the
 address of the payload as a key, meaning that incrementing it does
 not mutate the target, but is im`pure`.


 
 Yah, my thoughts exactly. My money is on the latter option. The
 reference counter is metadata, not data, and should not be treated as
 part of the data even though implementation-wise it is.
 
 Eduard and I stopped short of being able to formalize this.


This sounds like it's a good idea.  But how would that interact with
`pure`?  Will it be just part of life that RC types are necessarily
impure?


[...]

 What you seem to be asking for instead is a way to trick the type
 system into agreeing that mutating a reference count doesn't
 actually mutate anything, which is nonsense. If that's really
 necessary for some reason, it needs to be special cased into the
 language spec, like how `pure` explicitly permits memory allocation.


 
 It's not nonsense. Or if it is a lot other things can be categorized
 as nonsense, such as the GC recycling memory of one type and
 presenting it as a different type etc.


As I said elsewhere, it's a different level of abstraction. The GC
operates at a lower level of abstraction, and therefore does "dirty"
things like casting to/from immutable, pointer arithmetic, etc.. But the
interface it presents to user code forms a higher level of abstraction
where these low-level manipulations are hidden away under the hood.


T

-- 
Give me some fresh salted fish, please.
Nov 12 2021
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 12 November 2021 at 17:01:57 UTC, H. S. Teoh wrote:

 As I said elsewhere, it's a different level of abstraction. The 
 GC operates at a lower level of abstraction, and therefore does 
 "dirty" things like casting to/from immutable, pointer 
 arithmetic, etc.. But the interface it presents to user code 
 forms a higher level of abstraction where these low-level 
 manipulations are hidden away under the hood.


 T


I think it is fair to expect any RC system to be able to operate 
at that same level.
Nov 12 2021
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Nov 12, 2021 at 05:23:33PM +0000, deadalnix via Digitalmars-d wrote:

 On Friday, 12 November 2021 at 17:01:57 UTC, H. S. Teoh wrote:

 As I said elsewhere, it's a different level of abstraction. The GC
 operates at a lower level of abstraction, and therefore does "dirty"
 things like casting to/from immutable, pointer arithmetic, etc.. But
 the interface it presents to user code forms a higher level of
 abstraction where these low-level manipulations are hidden away
 under the hood.




[...]

 I think it is fair to expect any RC system to be able to operate at
 that same level.


Yes.  So actually, this *could* be made to work if the RC payload is
only allowed to be allocated from an RC allocator.  The allocator would
allocate n+8 bytes, for example, and return a pointer to offset 8 of the
allocated block, which is cast to whatever type/qualifier(s) needed.
Offset 0 would be the reference count.  The copy ctor would access the
reference count as *(ptr-8), which is technically outside the
const/immutable/whatever payload.  When the ref count reaches 0, the
allocator knows to deallocate the block starting from (ptr-8), which is
the start of the actual allocation.


T

-- 
Tech-savvy: euphemism for nerdy.
Nov 12 2021
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12.11.21 18:44, H. S. Teoh wrote:

 Yes.  So actually, this*could*  be made to work if the RC payload is
 only allowed to be allocated from an RC allocator.  The allocator would
 allocate n+8 bytes, for example, and return a pointer to offset 8 of the
 allocated block, which is cast to whatever type/qualifier(s) needed.
 Offset 0 would be the reference count.  The copy ctor would access the
 reference count as *(ptr-8), which is technically outside the
 const/immutable/whatever payload.  When the ref count reaches 0, the
 allocator knows to deallocate the block starting from (ptr-8), which is
 the start of the actual allocation.


You still need language support. Reaching mutable data through an 
immutable pointer violates transitivity assumptions.
Nov 12 2021
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Nov 12, 2021 at 10:14:28PM +0100, Timon Gehr via Digitalmars-d wrote:

 On 12.11.21 18:44, H. S. Teoh wrote:

 Yes.  So actually, this*could*  be made to work if the RC payload is
 only allowed to be allocated from an RC allocator.  The allocator
 would allocate n+8 bytes, for example, and return a pointer to
 offset 8 of the allocated block, which is cast to whatever
 type/qualifier(s) needed.  Offset 0 would be the reference count.
 The copy ctor would access the reference count as *(ptr-8), which is
 technically outside the const/immutable/whatever payload.  When the
 ref count reaches 0, the allocator knows to deallocate the block
 starting from (ptr-8), which is the start of the actual allocation.


 
 You still need language support. Reaching mutable data through an
 immutable pointer violates transitivity assumptions.


Aside from the technicalities of storing the refcount in the machine
word preceding the allocated block, one could imagine this as looking up
(and updating) state in a global AA using the address of the immutable
ref.

Definitely impure, though, 'cos if we want to remain consistent with the
global AA analogy, we have to consider the refcount as global state, so
we cannot do this kind of refcounting in pure code.


T

-- 
Life begins when you can spend your spare time programming instead of watching
television. -- Cal Keegan
Nov 12 2021
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2021-11-12 16:14, Timon Gehr wrote:

 On 12.11.21 18:44, H. S. Teoh wrote:

 Yes.  So actually, this*could*  be made to work if the RC payload is
 only allowed to be allocated from an RC allocator.  The allocator would
 allocate n+8 bytes, for example, and return a pointer to offset 8 of the
 allocated block, which is cast to whatever type/qualifier(s) needed.
 Offset 0 would be the reference count.  The copy ctor would access the
 reference count as *(ptr-8), which is technically outside the
 const/immutable/whatever payload.  When the ref count reaches 0, the
 allocator knows to deallocate the block starting from (ptr-8), which is
 the start of the actual allocation.


 
 You still need language support. Reaching mutable data through an 
 immutable pointer violates transitivity assumptions.


Indeed, but not indirectly. You can soundly access a pointer to mutable 
data if you use the immutable pointer as a key in a global hashtable.
Nov 12 2021
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 12 November 2021 at 22:09:28 UTC, Andrei Alexandrescu 
wrote:

 On 2021-11-12 16:14, Timon Gehr wrote:

 You still need language support. Reaching mutable data through 
 an immutable pointer violates transitivity assumptions.


 Indeed, but not indirectly. You can soundly access a pointer to 
 mutable data if you use the immutable pointer as a key in a 
 global hashtable.


I guess the question reduces to: what is the simplest thing you 
can do to a pointer that will make the compiler forget its 
provenance?

Adding an offset is clearly not enough. Casting it to an integer, 
taking its hash, and using that hash to look up a value in an 
associative array is probably *more* than enough. Is there 
something in between that's "just right"?

One possibility is casting the pointer to an integer, and then 
immediately casting it back:

     immutable(int)* p;
     size_t n = cast(size_t) (p - 1);
     int* q = cast(int*) n;
     *q = 123;

Assume we know, somehow, that the memory location pointed by q is 
allocated and contains a mutable int. Does the mutation in the 
final line have undefined behavior?

The answer depends on whether integers have provenance or not--a 
question which remains unsettled in the world of C and C++. [1] 
If we decide that integers should *not* have provenance in D, 
then the above code is valid, and we do not need to add any new 
features to the type system to support it.

The downsides of this approach are:

   1) Understanding and maintaining code that relies on obscure 
language rules like this is really difficult.
   2) Existing compiler backends (GCC, LLVM) don't always handle 
code like this correctly, even in C. (See linked article.)

[1] https://www.ralfj.de/blog/2020/12/14/provenance.html
Nov 12 2021
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 13.11.21 01:29, Paul Backus wrote:

 On Friday, 12 November 2021 at 22:09:28 UTC, Andrei Alexandrescu wrote:

 On 2021-11-12 16:14, Timon Gehr wrote:

 You still need language support. Reaching mutable data through an 
 immutable pointer violates transitivity assumptions.


 Indeed, but not indirectly. You can soundly access a pointer to 
 mutable data if you use the immutable pointer as a key in a global 
 hashtable.


 
 I guess the question reduces to: what is the simplest thing you can do 
 to a pointer that will make the compiler forget its provenance?
 
 Adding an offset is clearly not enough. Casting it to an integer, taking 
 its hash, and using that hash to look up a value in an associative array 
 is probably *more* than enough. Is there something in between that's 
 "just right"?
 
 One possibility is casting the pointer to an integer, and then 
 immediately casting it back:
 
      immutable(int)* p;
      size_t n = cast(size_t) (p - 1);
      int* q = cast(int*) n;
      *q = 123;
 
 Assume we know, somehow, that the memory location pointed by q is 
 allocated and contains a mutable int. Does the mutation in the final 
 line have undefined behavior?
 
 The answer depends on whether integers have provenance or not--a 
 question which remains unsettled in the world of C and C++. [1] If we 
 decide that integers should *not* have provenance in D, then the above 
 code is valid, and we do not need to add any new features to the type 
 system to support it.
 ...


It may also depend on whether you are in a pure function or not.
Nov 12 2021
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12.11.21 23:09, Andrei Alexandrescu wrote:

 On 2021-11-12 16:14, Timon Gehr wrote:

 On 12.11.21 18:44, H. S. Teoh wrote:

 Yes.  So actually, this*could*  be made to work if the RC payload is
 only allowed to be allocated from an RC allocator.  The allocator would
 allocate n+8 bytes, for example, and return a pointer to offset 8 of the
 allocated block, which is cast to whatever type/qualifier(s) needed.
 Offset 0 would be the reference count.  The copy ctor would access the
 reference count as *(ptr-8), which is technically outside the
 const/immutable/whatever payload.  When the ref count reaches 0, the
 allocator knows to deallocate the block starting from (ptr-8), which is
 the start of the actual allocation.


 You still need language support. Reaching mutable data through an 
 immutable pointer violates transitivity assumptions.


 
 Indeed, but not indirectly. You can soundly access a pointer to mutable 
 data if you use the immutable pointer as a key in a global hashtable.


Actually, you can't.

On 08.11.21 22:42, Andrei Alexandrescu wrote:

 - work in pure code just like T[] does
Nov 12 2021
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/12/21 10:15 PM, Timon Gehr wrote:

 On 12.11.21 23:09, Andrei Alexandrescu wrote:

 On 2021-11-12 16:14, Timon Gehr wrote:

 On 12.11.21 18:44, H. S. Teoh wrote:

 Yes.  So actually, this*could*  be made to work if the RC payload is
 only allowed to be allocated from an RC allocator.  The allocator would
 allocate n+8 bytes, for example, and return a pointer to offset 8 of 
 the
 allocated block, which is cast to whatever type/qualifier(s) needed.
 Offset 0 would be the reference count.  The copy ctor would access the
 reference count as *(ptr-8), which is technically outside the
 const/immutable/whatever payload.  When the ref count reaches 0, the
 allocator knows to deallocate the block starting from (ptr-8), which is
 the start of the actual allocation.


 You still need language support. Reaching mutable data through an 
 immutable pointer violates transitivity assumptions.


 Indeed, but not indirectly. You can soundly access a pointer to 
 mutable data if you use the immutable pointer as a key in a global 
 hashtable.


 
 Actually, you can't.
 
 On 08.11.21 22:42, Andrei Alexandrescu wrote:

 - work in pure code just like T[] does




Indeed if you add the purity requirement you can't, thanks.
Nov 13 2021
prev sibling parent reply Elronnd <elronnd elronnd.net> writes:
On Friday, 12 November 2021 at 21:14:28 UTC, Timon Gehr wrote:

 You still need language support. Reaching mutable data through 
 an immutable pointer violates transitivity assumptions.


D does not have well-defined provenance semantics.  We need to 
decide:

1) what provenance semantics do we want;

2) how do we implement those provenance semantics; and

3) how do we implement rc given those semantics

Ideally 1 should be decided with an eye towards 2 and 3.  I think 
it would be reasonable to define provenance semantics which 
permit you to derive a mutable pointer from an immutable one 
(though not in  safe code).  I think it would also be reasonable 
to define provenance semantics under which you cannot perform 
such a derivation.

But that's something we have to decide, not take for granted.
Dec 13 2021
parent Elronnd <elronnd elronnd.net> writes:
On Tuesday, 14 December 2021 at 06:27:27 UTC, Elronnd wrote:

 I think it would be reasonable to define provenance semantics 
 which permit you to derive a mutable pointer from an immutable 
 one


Simple example:

struct S { immutable float x; int y; }
S a;
immutable float* x = &a.x;
int y = *cast(int*)(cast(ubyte*)&x + (S.y.offsetof - 
S.x.offsetof))

above should be UB if (cast(ubyte*)&x + (S.y.offsetof - 
S.x.offsetof)) does not actually point to mutable memory.  This 
is subtly different from:

int x;
const int *p = &x;
*cast(int*)p = 5;

but still similar enough to warrant concern; which is why I think 
it would also be reasonable to make the former form illegal.
Dec 13 2021
prev sibling parent deadalnix <deadalnix gmail.com> writes:
On Thursday, 11 November 2021 at 23:17:53 UTC, Andrei 
Alexandrescu wrote:

 Walter would argue that his work on scope makes that possible 
 too. The short of it is getting an answer to "can it be done" 
 is important in and of itself because it gives us hints on what 
 needs to be done.


But it cannot be done. This because scope do not compose, so as 
soon as there is one indirection, all information is gone.
Nov 12 2021
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
wrote:

 This was prompted by this exchange with Paul Backus that went 
 like this:

 Me: "We never got reference counting to work in safe pure nogc 
 code. And we don't know how. If we did, we could write a 
 slice-like type that does everything a slice does PLUS manages 
 its own memory. This is the real problem with containers."

 Paul: "I'm pretty sure at least some of us (including Atila) 
 know how."


I'll hijack this thread with a slide from my DConf Online 2020 
talk, because with the current language it's impossible for a 
library writer to write a useful  safe vector struct:

```d
scope v = vector(1, 2, 3, 4);
scope s = v[];
v ~= 5;  // could reallocate here
s[3] = 42;  // oops
```

I initialy "solved" this with  trusted, but fortunately somebody 
on the forums pointed out I was being an idiot.

It might just be my biases but I think this is a more useful nut 
to crack. And the only way to crack it is to some sort of 
ownership system, since there's no way currently in D to have a 
compile-time guarantee that there's only one alias left.

"We can have  safe vectors, unless you actually want to append" 
isn't a great slogan. And vectors are clearly simpler than ref 
counted arrays.
Nov 09 2021
next sibling parent deadalnix <deadalnix gmail.com> writes:
On Tuesday, 9 November 2021 at 11:56:50 UTC, Atila Neves wrote:

 It might just be my biases but I think this is a more useful 
 nut to crack. And the only way to crack it is to some sort of 
 ownership system, since there's no way currently in D to have a 
 compile-time guarantee that there's only one alias left.


Great mind think alike. i was considering rebooting this 
challenge dropping the RC part altogether. Just reimplement 
slices such as they behave like the builtin ones is enough of a 
challenge already.
Nov 09 2021
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 9 November 2021 at 11:56:50 UTC, Atila Neves wrote:

 I'll hijack this thread with a slide from my DConf Online 2020 
 talk, because with the current language it's impossible for a 
 library writer to write a useful  safe vector struct:

 ```d
 scope v = vector(1, 2, 3, 4);
 scope s = v[];
 v ~= 5;  // could reallocate here
 s[3] = 42;  // oops
 ```


Hm, if the vector had a reference count and would crash if 
appended to when the count is non-zero?

Functions like these could also solve the thing if only called 
from ` live`:

```d
AppendableVector setupAppending(Vector*);
Vector stopAppending(AppendableVector*);
```

Obviously, you could not slice an AppendableVector, and we're not 
proposing that average application code should be ` live` so not 
a very good solution.


 I initialy "solved" this with  trusted, but fortunately 
 somebody on the forums pointed out I was being an idiot.


If that makes anybody idiot, I doubt there are many normal-minded 
programmers. Let alone geniuses.
Nov 09 2021
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 9 November 2021 at 15:26:24 UTC, Dukc wrote:

 On Tuesday, 9 November 2021 at 11:56:50 UTC, Atila Neves wrote:

 I'll hijack this thread with a slide from my DConf Online 2020 
 talk, because with the current language it's impossible for a 
 library writer to write a useful  safe vector struct:

 ```d
 scope v = vector(1, 2, 3, 4);
 scope s = v[];
 v ~= 5;  // could reallocate here
 s[3] = 42;  // oops
 ```


 Hm, if the vector had a reference count


Then it wouldn't be a vector.


 Functions like these could also solve the thing if only called 
 from ` live`:

 ```d
 AppendableVector setupAppending(Vector*);
 Vector stopAppending(AppendableVector*);
 ```

 Obviously, you could not slice an AppendableVector, and we're 
 not proposing that average application code should be ` live` 
 so not a very good solution.


The main point of vector is that one can append to it without 
using the GC. I guess that "this thing has a length only known at 
runtime that won't change and doesn't use the GC" is useful too, 
but not nearly as much as being able to append.
Nov 09 2021
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 9 November 2021 at 15:37:13 UTC, Atila Neves wrote:

 [snip]

 The main point of vector is that one can append to it without 
 using the GC. I guess that "this thing has a length only known 
 at runtime that won't change and doesn't use the GC" is useful 
 too, but not nearly as much as being able to append.


Regardless, the limitations of the language and the solutions 
brought up have been discussed on the forums for years. Getting 
some kind of pathway forward is really what is needed. Breaking 
the problem that Andrei brings up into a smaller one, i.e. 
writing a  safe appendable vector that works as much like a 
built-in one, is a useful start.
Nov 09 2021
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 9 November 2021 at 15:54:01 UTC, jmh530 wrote:

 On Tuesday, 9 November 2021 at 15:37:13 UTC, Atila Neves wrote:

 [snip]

 The main point of vector is that one can append to it without 
 using the GC. I guess that "this thing has a length only known 
 at runtime that won't change and doesn't use the GC" is useful 
 too, but not nearly as much as being able to append.


 Regardless, the limitations of the language and the solutions 
 brought up have been discussed on the forums for years. Getting 
 some kind of pathway forward is really what is needed. Breaking 
 the problem that Andrei brings up into a smaller one, i.e. 
 writing a  safe appendable vector that works as much like a 
 built-in one, is a useful start.


This is already something we're looking into it as part of the 
vision for D. I personally will not rest until such a library 
type can be written and used in  safe code. Nobody* should be 
calling malloc/free, just like nobody should be writing 
new/delete in C++14 and newer.


* Obviously for values of "nobody" that are tiny but not exactly 
equal to 0.
Nov 09 2021
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 9 November 2021 at 17:15:59 UTC, Atila Neves wrote:


 This is already something we're looking into it as part of the 
 vision for D. I personally will not rest until such a library 
 type can be written and used in  safe code. Nobody* should be 
 calling malloc/free, just like nobody should be writing 
 new/delete in C++14 and newer.


 * Obviously for values of "nobody" that are tiny but not 
 exactly equal to 0.


Au contraire. EVERYBODY should be calling malloc/free or any 
other allocators, including the GC. No matter how hard you try, 
you CANNOT abstract the machine and remain a systems language at 
the same time. If you don't want to be worrying about such 
things, there are other, higher level, languages.

It's the calling of such APIs that must be made  safe. Walter is 
attempting just that with  live. Incidentally, if THAT is made 
possible, you will be able to have the cake and eat it too.

BTW, at the moment, "safety" of GC is a lie. Because it can run 
arbitrary unsafe code within a  safe function.
Nov 09 2021
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 09, 2021 at 05:26:32PM +0000, Stanislav Blinov via Digitalmars-d
wrote:

 On Tuesday, 9 November 2021 at 17:15:59 UTC, Atila Neves wrote:
 

 This is already something we're looking into it as part of the
 vision for D. I personally will not rest until such a library type
 can be written and used in  safe code. Nobody* should be calling
 malloc/free, just like nobody should be writing new/delete in C++14
 and newer.
 
 
 * Obviously for values of "nobody" that are tiny but not exactly
 equal to 0.


 
 Au contraire. EVERYBODY should be calling malloc/free or any other
 allocators, including the GC. No matter how hard you try, you CANNOT
 abstract the machine and remain a systems language at the same time.
 If you don't want to be worrying about such things, there are other,
 higher level, languages.


I think Atila's point is that *by default* you should be able to write
high-level D code without needing to call malloc/free, but when
necessary, you can do so *and* still take advantage of the high-level
abstractions of the language (e.g., maintain  safe-ty, etc.).



 It's the calling of such APIs that must be made  safe.


[...]

IMO, that's not possible. In general code that calls malloc/free it's
not possible to prove anything  safe. The only way you can do this in a
waterproof way is to implement the equivalent of GC tracing at
*compile-time* so that every call to free() can be proven to happen
exactly once for every call to malloc(). This is obviously impossible.

In general, it's a mistake to mark malloc/free as  safe or pure, it's
the wrong semantics and will only lead to trouble down the road.


T

-- 
Marketing: the art of convincing people to pay for what they didn't need before
which you fail to deliver after.
Nov 09 2021
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 9 November 2021 at 18:16:03 UTC, H. S. Teoh wrote:


 I think Atila's point is that *by default* you should be able 
 to write high-level D code without needing to call malloc/free, 
 but when necessary, you can do so *and* still take advantage of 
 the high-level abstractions of the language (e.g., maintain 
  safe-ty, etc.).


Oh, I understand the sentiment, I really do. That we need more 
out-of-the-box tried and true robust containers is not in 
question - hence the premise of this very topic by Andrei. It's 
this constant desire to "abstract away" the very things that you 
literally *need* to operate on that puzzles me. Hey, raw pointers 
are bad, don't use! Hey, "new" is bad, don't call! Well if you're 
not using raw pointers and you aren't calling "new", there's 
Python. Or something. You don't get to keep the performance, and 
power, of low level languages, if you desperately abstract 
everything away. "Zero-cost" abstractions aren't a thing, it's a 
buzzword myth.


 It's the calling of such APIs that must be made  safe.


 [...]

 IMO, that's not possible. In general code that calls 
 malloc/free it's not possible to prove anything  safe. The only 
 way you can do this in a waterproof way is to implement the 
 equivalent of GC tracing at *compile-time* so that every call 
 to free() can be proven to happen exactly once for every call 
 to malloc(). This is obviously impossible.


Could be possible for allocators that don't need a free() for 
every malloc(). At least, I think it could be:

while (true)
{
     auto a = alloc.allocate(1);
     auto b = alloc.allocate(100500);
     // ...

     // if here you can prove that nothing escaped, the call is 
safe.
     // which means `allocate` needs to paint the result somehow.
     alloc.freeAll();
}

Yip, that is, effectively, manual GC.


 In general, it's a mistake to mark malloc/free as  safe or 
 pure, it's the wrong semantics and will only lead to trouble 
 down the road.


But if you can express ownership in the language, it should be 
possible to make at least a  safe interface to them.
Nov 09 2021
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 09, 2021 at 07:57:21PM +0000, Stanislav Blinov via Digitalmars-d
wrote:

 On Tuesday, 9 November 2021 at 18:16:03 UTC, H. S. Teoh wrote:


[...]

 It's the calling of such APIs that must be made  safe.


 [...]
 
 IMO, that's not possible. In general code that calls malloc/free
 it's not possible to prove anything  safe. The only way you can do
 this in a waterproof way is to implement the equivalent of GC
 tracing at *compile-time* so that every call to free() can be proven
 to happen exactly once for every call to malloc(). This is obviously
 impossible.


 
 Could be possible for allocators that don't need a free() for every
 malloc().


Yes, but then you're back to malloc/free (well, the C version of it)
being inherently  system, and there is no way you can make them  safe.


[...]

 In general, it's a mistake to mark malloc/free as  safe or pure,
 it's the wrong semantics and will only lead to trouble down the
 road.


 
 But if you can express ownership in the language, it should be
 possible to make at least a  safe interface to them.


To be able to express ownership to the extent that you can do this, is
basically equivalent to importing Rust into D. Is that really what we
want to do?


T

-- 
It is widely believed that reinventing the wheel is a waste of time; but I
disagree: without wheel reinventers, we would be still be stuck with wooden
horse-cart wheels.
Nov 09 2021
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 9 November 2021 at 17:26:32 UTC, Stanislav Blinov 
wrote:

 On Tuesday, 9 November 2021 at 17:15:59 UTC, Atila Neves wrote:


 This is already something we're looking into it as part of the 
 vision for D. I personally will not rest until such a library 
 type can be written and used in  safe code. Nobody* should be 
 calling malloc/free, just like nobody should be writing 
 new/delete in C++14 and newer.


 * Obviously for values of "nobody" that are tiny but not 
 exactly equal to 0.


 Au contraire. EVERYBODY should be calling malloc/free or any 
 other allocators, including the GC. No matter how hard you try, 
 you CANNOT abstract the machine and remain a systems language 
 at the same time. If you don't want to be worrying about such 
 things, there are other, higher level, languages.


Could you please explain why you'd rather do that instead of 
using the equivalent of C++'s std::{vector, unique_ptr, 
shared_ptr} and Rust's std::{vector, unique_ptr, shared_ptr}? I 
cannot myself imagine why anyone would want to.

I think that C++'s greatest gift to the world was the destructor. 
We have those too! Let's use them.


 It's the calling of such APIs that must be made  safe.


I think the focus should be in making it possible to write the 
types I mentioned above in a  safe manner.

My advice is that unless you have a very good reason not to, just 
use the GC and call it a day.
Nov 09 2021
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 On Tuesday, 9 November 2021 at 17:26:32 UTC, Stanislav Blinov 
 wrote:

 On Tuesday, 9 November 2021 at 17:15:59 UTC, Atila Neves wrote:


 This is already something we're looking into it as part of 
 the vision for D. I personally will not rest until such a 
 library type can be written and used in  safe code. Nobody* 
 should be calling malloc/free, just like nobody should be 
 writing new/delete in C++14 and newer.


 * Obviously for values of "nobody" that are tiny but not 
 exactly equal to 0.


 Au contraire. EVERYBODY should be calling malloc/free or any 
 other allocators, including the GC. No matter how hard you 
 try, you CANNOT abstract the machine and remain a systems 
 language at the same time. If you don't want to be worrying 
 about such things, there are other, higher level, languages.


 Could you please explain why you'd rather do that instead of 
 using the equivalent of C++'s std::{vector, unique_ptr, 
 shared_ptr} and Rust's std::{vector, unique_ptr, shared_ptr}? I 
 cannot myself imagine why anyone would want to.


Instead? Not instead. Together with. It's all well and fine to 
rely on proven library solutions. But I'd rather a D or C++ 
programmer, when faced with necessity, be able to write their 
allocations correctly, and not hide under a rug because 
Herb'n'Scott tell 'em that's "bad practice".

We already have at least two (three?) generations of programmers 
who literally have no clue where memory comes from. If we keep 
this up, in a couple decades "nobody" (your definition of nobody) 
would be able to write you a better malloc for your next 
generation of platforms and hardware.

Take a peek in the learn section. Person asks how to translate a 
simple C program into D. Buncha answers that *all* amount to 
"allocate craptons of memory for no reason". At least one from a 
very well known D educator. Only no one even mentions any 
allocations at all. Why even talk about it, right? That array 
just sort of happens out of nowhere...


 I think that C++'s greatest gift to the world was the 
 destructor. We have those too! Let's use them.


Yeah, we have them. And the GC may call them. Any time you 
allocate it may call them. Even if they aren't pure or safe. But 
hey, that's OK between friends.


 It's the calling of such APIs that must be made  safe.


 I think the focus should be in making it possible to write the 
 types I mentioned above in a  safe manner.

 My advice is that unless you have a very good reason not to, 
 just use the GC and call it a day.


Who, using a SYSTEMS language, should not have a reason to care 
about their memory? I can think of literally just one case: 
needing a quick script and being too lazy to switch to something 
other than D just for that.
Nov 09 2021
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 9 November 2021 at 19:39:15 UTC, Stanislav Blinov 
wrote:

 On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 On Tuesday, 9 November 2021 at 17:26:32 UTC, Stanislav Blinov 
 wrote:

 On Tuesday, 9 November 2021 at 17:15:59 UTC, Atila Neves 
 wrote:


 Could you please explain why you'd rather do that instead of 
 using the equivalent of C++'s std::{vector, unique_ptr, 
 shared_ptr} and Rust's std::{vector, unique_ptr, shared_ptr}? 
 I cannot myself imagine why anyone would want to.


 Instead? Not instead. Together with. It's all well and fine to 
 rely on proven library solutions. But I'd rather a D or C++ 
 programmer, when faced with necessity,


I have not yet encountered cases where it would be necessary that 
aren't "I'm implementing the standard library".


 be able to write their allocations correctly, and not hide 
 under a rug because Herb'n'Scott tell 'em that's "bad practice".


I think that decades of experience (and tools like valgrind and 
asan) have shown that programmers aren't able to write their 
allocations correctly.


 We already have at least two (three?) generations of 
 programmers who literally have no clue where memory comes from. 
 If we keep this up, in a couple decades "nobody" (your 
 definition of nobody) would be able to write you a better 
 malloc for your next generation of platforms and hardware.


I don't think this is a problem.


 Take a peek in the learn section. Person asks how to translate 
 a simple C program into D. Buncha answers that *all* amount to 
 "allocate craptons of memory for no reason". At least one from 
 a very well known D educator. Only no one even mentions any 
 allocations at all. Why even talk about it, right?


I wouldn't care about it either.


 My advice is that unless you have a very good reason not to, 
 just use the GC and call it a day.


 Who, using a SYSTEMS language, should not have a reason to care 
 about their memory?


Me, ~99.9% of the time.

My algorithm:

1. Write code
2. Did I notice it being slow or too big? Go to 4.
3. Move on to the next task
4. Profile and optimise, go to 2.

I definitely don't miss having to make things fit into 48k of 
RAM. I wrote code for a microcontroller once with 1k with 
addressable bits. Good times. Sorta.
Nov 11 2021
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves wrote:

 I have not yet encountered cases where it would be necessary 
 that aren't "I'm implementing the standard library".


Not so sure about that. Arena allocators would be done with 
malloc, but you probably should wrap it with RAII.

Allocating directly from the OS (bypassing malloc) can in some 
cases be advantageous. But again, you are usually better off 
creating a RAII wrapper for it.

You also have dynamic stack allocation which can boost 
performance significantly. Sadly, Walter is against having this 
as a feature.



 I definitely don't miss having to make things fit into 48k of 
 RAM. I wrote code for a microcontroller once with 1k with 
 addressable bits. Good times. Sorta.


That's a lot! As a teen I did a project on Motorola 6800. I think 
it was launched with a whopping 128 bytes of static RAM, so you 
had to make almost everything ```immutable``` (EPROM) and avoid 
deep call-trees.
Nov 11 2021
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 11 November 2021 at 10:06:44 UTC, Ola Fosheim 
Grøstad wrote:

 On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves 
 wrote:

 I have not yet encountered cases where it would be necessary 
 that aren't "I'm implementing the standard library".


 Not so sure about that. Arena allocators would be done with 
 malloc, but you probably should wrap it with RAII.

 Allocating directly from the OS (bypassing malloc) can in some 
 cases be advantageous. But again, you are usually better off 
 creating a RAII wrapper for it.

 You also have dynamic stack allocation which can boost 
 performance significantly. Sadly, Walter is against having this 
 as a feature.


So use a different allocator with your vector/smart pointer/what 
have you.
Nov 11 2021
prev sibling parent reply Dom DiSc <dominikus scherkl.de> writes:
On Thursday, 11 November 2021 at 10:06:44 UTC, Ola Fosheim 
Grøstad wrote:

 On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves

 wrote code for a microcontroller once with 1k with 
 addressable bits. Good times. Sorta.


 That's a lot! As a teen I did a project on Motorola 6800. I 
 think it was launched with a whopping 128 bytes of static RAM,


How is 1k bits "a lot" compared to 128 bytes (which is exactly 
the same amount)?!?
Nov 12 2021
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 12 November 2021 at 08:41:41 UTC, Dom DiSc wrote:

 How is 1k bits "a lot" compared to 128 bytes (which is exactly 
 the same amount)?!?


I thought he meant 1KiB with bit level addressing. :-P
Nov 12 2021
prev sibling next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves wrote:


 I have not yet encountered...
 I think that...
 I don't think this is a problem.
 I wouldn't care about it either.
 Me, ~99.9% of the time. I definitely don't miss...


This all amounts to "I don't need it, therefore nobody should 
either". That's not a very good metric.

It's 2021. Your footnote on "nobody" simply does not apply. 
Everybody should be writing code designed to run on multiple 
processors simultaneously. Because that's what the machines are. 
A large portion of that "everybody" should be writing code where 
a large number of those processors are the GPU, or GPU(s), and 
I'm not even talking computer games. In either case, memory 
becomes key. Because, at least today, it's your slow resource 
(CPU) and also your somewhat limited resource (GPU).


 My algorithm:
 1. Write code
 2. Did I notice it being slow or too big? Go to 4.
 3. Move on to the next task.
 4. Profile and optimise, go to 2.


That's an interesting way of doing science. How would you notice 
anything without a baseline, which you haven't even set yet 
because you're not at (4)? You won't. You just wrote a slow piece 
of software and you don't even know it because you didn't look, 
because it didn't appear slow? Eh?

People after you, who use your software, maybe, MAYBE, would 
notice though (because they set the baseline from the start), but 
by that time it is too late, because your software is part of the 
OS or CRT or whatever other dependency the application must rely 
on, but you didn't care. Not because you didn't optimize. Just 
because you pessimized, and never bothered to look.

...and that is exactly why Visual Studio (a text editor) takes 10 
seconds to load a few small text files (whereas the compiler 
itself loads and compiles those same files in less than a 
second), Photoshop (an image editor) greets you with a nice 
splash screen (that contains an image) "almost" immediately, but 
lets you actually work on images only a dozen seconds later, 
Libre- (and any other) Office shows you a progress bar while it's 
"starting up", and so on and so forth... And then we have Andrei 
on stage lamenting on how awesome and lucrative it was to squeeze 
out that meager extra 1%...


 having to make things fit into 48k of RAM


??? You DO have to make things that fit into 32k, or 64k, or 
(insert your cache size here). Over time - modulated by the speed 
of that cache. Which means, you have to take care with how, and 
where, your data is laid out. Which, eventually, brings you back 
to how you allocate. Moreover, you DO have to make things that 
fit into a single cache line, or are bounded by the size of that 
cache line. Which, again, brings you back to how you allocate.

Nobody would do any of that for you. Malloc or new won't (well, 
malloc may try, but it'll inevitably fail). Vector and map won't. 
Unique, shared, and other ptrs won't.
Nov 11 2021
next sibling parent reply Greg Strong <mageofmaple protonmail.com> writes:
On Thursday, 11 November 2021 at 16:09:46 UTC, Stanislav Blinov 
wrote:

 That's an interesting way of doing science.


Most computer programming is not science.  Programming is a 
trade.  There is, of course, such a thing as computer science, 
but it is not what most programmers do most of the time.

On Thursday, 11 November 2021 at 16:09:46 UTC, Stanislav Blinov 
wrote:

 How would you notice anything without a baseline, which you 
 haven't even set yet because you're not at (4)? You won't. You 
 just wrote a slow piece of software and you don't even know it 
 because you didn't look, because it didn't appear slow? Eh?


The simple fact is that for roughly 99% of code, (in terms of 
lines of code, not complete programs), performance is basically 
irrelevant.
Nov 11 2021
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Thursday, 11 November 2021 at 17:03:42 UTC, Greg Strong wrote:


 Most computer programming is not science.  Programming is a 
 trade.  There is, of course, such a thing as computer science, 
 but it is not what most programmers do most of the time.


Call it whatever you like. "I don't think it's slow" is not a 
measurement, therefore drawing any conclusions from that is 
pointless.


 The simple fact is that for roughly 99% of code, (in terms of 
 lines of code, not complete programs), performance is basically 
 irrelevant.


Running fast, and simply not running slow, are two very different 
things. The "simple fact" here is that most people either don't 
care their code is unnecessarily slow, or don't know that it is, 
because they didn't bother to even look.

Man, this is so off-topic... I'm sorry guys.
Nov 11 2021
prev sibling next sibling parent claptrap <clap trap.com> writes:
On Thursday, 11 November 2021 at 16:09:46 UTC, Stanislav Blinov 
wrote:

 On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves

 My algorithm:
 1. Write code
 2. Did I notice it being slow or too big? Go to 4.
 3. Move on to the next task.
 4. Profile and optimise, go to 2.


 That's an interesting way of doing science. How would you 
 notice anything without a baseline, which you haven't even set 
 yet because you're not at (4)?


Maybe it needs to hit a certain FPS, or maybe its "are a lot of 
people complaining that it's slow", or "hmm i though that would 
be a lot faster". You have users, expectations, requirements 
etc.. Could be a whole bunch of reasons to jump to 4, it doesn't 
require an absolute "this is measurably slower than version 1.03"
Nov 11 2021
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 11 November 2021 at 16:09:46 UTC, Stanislav Blinov 
wrote:

 On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves 
 wrote:


 I have not yet encountered...
 I think that...
 I don't think this is a problem.
 I wouldn't care about it either.
 Me, ~99.9% of the time. I definitely don't miss...


 This all amounts to "I don't need it, therefore nobody should 
 either". That's not a very good metric.


I don't think that's an accurate description of what I wrote.


 It's 2021. Your footnote on "nobody" simply does not apply.


Agree to disagree.


 My algorithm:
 1. Write code
 2. Did I notice it being slow or too big? Go to 4.
 3. Move on to the next task.
 4. Profile and optimise, go to 2.


 That's an interesting way of doing science. How would you 
 notice anything without a baseline, which you haven't even set 
 yet because you're not at (4)?


This is my opinion, not fact: if I didn't get to 4, then it 
doesn't matter (again, to me). I don't care if something runs in 
1us or 1ms*. I start caring when it's >20ms, because that's when 
I start noticing it.

* usually
Nov 11 2021
prev sibling parent reply russhy <russhy gmail.com> writes:
On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves wrote:


 Who, using a SYSTEMS language, should not have a reason to 
 care about their memory?


 Me, ~99.9% of the time.



This explains a lot and why D is stuck with a miserable GC
Nov 12 2021
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Nov 12, 2021 at 09:25:23PM +0000, russhy via Digitalmars-d wrote:

 On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves wrote:
 

 Who, using a SYSTEMS language, should not have a reason to care
 about their memory?


 
 Me, ~99.9% of the time.
 


 
 
 This explains a lot and why D is stuck with a miserable GC


It depends on what the code is trying to do.  When I'm using D as a
superior shell scripting language, yeah I couldn't care less about the
intricacies of memory allocation.

If I'm writing performance sensitive code, though, which does happen
quite often, then I *do* care about minimizing allocations and/or
outright eliminating allocation from my inner loops.

It's about taking care of resource usage *where it matters*. There is no
point optimizing the heck out of a hello world program: that's just a
total waste of time. Save your efforts for when you're writing
performance critical code.

This is why I also say, why are people so hung up about the GC?  If your
inner loops have performance problems with the GC, then rewrite it as
 nogc, use malloc/free/whatever it takes to make it faster.  The rest of
the program can still use the GC -- there's no reason why I shouldn't
just allocate away when I'm rendering, say, the splash screen and
loading resource files on program startup, and let the GC clean up for
me. It's much faster to write and I don't waste time micro-optimizing
code that only runs once anyway.  I'd rather save my energy for when the
program starts doing real work and needs to get it done fast -- that's
when I pull out  nogc, GC.disable, malloc/free, etc..  Only the actual
bottlenecks (as measured by an actual profiler) need to do this; the
rest of the code that only runs once in a while don't matter, nobody
will care whether it uses GC or not.  Life is too short to fuss over
code that isn't even relevant to performance in the big picture.


T

-- 
The diminished 7th chord is the most flexible and fear-instilling chord. Use it
often, use it unsparingly, to subdue your listeners into submission!
Nov 12 2021
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Friday, 12 November 2021 at 22:10:22 UTC, H. S. Teoh wrote:


 Life is too short to fuss over code that isn't even relevant to 
 performance in the big picture.


Life is too short to wait on programs that have no reason to be 
slow. If you're "rendering splash screen while loading 
resources", you're most likely doing something wrong. Your 
program should've been on screen and interactive ages ago. 
Granted, there can be exceptions (like slow storage), but those 
are very, very rare. Visual Studio has no business taking 8 
seconds to load just to show its start screen. But it does. In a 
team of 50 people, starting it up twice a day, it wastes 57 
man-hours a year. In other words, VS is a worker that comes in 7 
days a year with full salary and amenities, to do absolutely 
nothing.

"The rest of the code that only runs once in a while don't 
matter" is an anachronism that needs to die, with fire. Whether 
you run once or a hundred times, you bloody well should run as 
fast as you possibly can, because there's always someone (or 
something) waiting on you! Including your own CPU. At the very 
least, it's just irresponsible to run unnecessarily slow.

Scripts for your own self - fine, leave 'em as slow as you're 
willing to suffer through. But please don't inflict your 
sloth-ness on the rest of the world ;)
Nov 12 2021
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Nov 12, 2021 at 10:47:56PM +0000, Stanislav Blinov via Digitalmars-d
wrote:

 On Friday, 12 November 2021 at 22:10:22 UTC, H. S. Teoh wrote:
 

 Life is too short to fuss over code that isn't even relevant to
 performance in the big picture.


 
 Life is too short to wait on programs that have no reason to be slow.
 If you're "rendering splash screen while loading resources", you're
 most likely doing something wrong. Your program should've been on
 screen and interactive ages ago. Granted, there can be exceptions
 (like slow storage), but those are very, very rare. Visual Studio has
 no business taking 8 seconds to load just to show its start screen.


If you're waiting 8 full seconds, that's a sign that something is wrong
with your performance. ;-)  Incidentally, this is why I don't bother
with IDEs; most of them are bloated for no reason and take ages to load
"resources" who knows for what. I could already be typing code in Vim
and could already be compiling the first draft by the time they finish
"starting up".  That's not what I was talking about, though.  I was
talking about programs that load in 1 second or less -- in which case
trying to optimize startup code is a waste of time.  If you're taking 8
seconds to start up, that *is* a performance problem and you should be
optimizing that!



 But it does. In a team of 50 people, starting it up twice a day, it
 wastes 57 man-hours a year. In other words, VS is a worker that comes
 in 7 days a year with full salary and amenities, to do absolutely
 nothing.


I agree, that's why I don't bother with IDEs.  Especially not those that
need to be started up every day.  I mean, who has the time to wait for
that?!  I keep my Vim sessions in GNU Screen for months, if not years,
at a time, and as soon as I sit down at my desk I can continue where I
left off yesterday immediately.



 "The rest of the code that only runs once in a while don't matter" is
 an anachronism that needs to die, with fire. Whether you run once or a
 hundred times, you bloody well should run as fast as you possibly can,
 because there's always someone (or something) waiting on you!
 Including your own CPU. At the very least, it's just irresponsible to
 run unnecessarily slow.


Given that I run my Vim sessions for years at a time, startup time *is*
completely irrelevant to me. :-D  If your program is such that people
have to constantly restart it, something is bogonous with your design
and you *should* consider scrapping it and doing it better.

But you took what I said completely out of context. I was talking about
optimizing the code where the profiler indicates an actual bottleneck;
90% of your code is not part of a bottleneck and therefore "optimizing"
it prematurely is a waste of time, not to mention it makes for
needlessly unreadable and unmaintainable code.  (An 8-second startup
time *is* a bottleneck, BTW. Just so we're on the same page.)  If code
that isn't part of a bottleneck is allocating and waiting for GC
collection, who cares?? It's not the bottleneck, you should be spending
your time fixing actual bottlenecks instead.  That's my point.



 Scripts for your own self - fine, leave 'em as slow as you're willing
 to suffer through. But please don't inflict your sloth-ness on the
 rest of the world ;)


Honestly, just about *anything* written in D runs faster than any shell
script (except perhaps for the most trivial of things like hello world
programs -- actually, even those run faster in D :-P).  Just the very
act of writing it in D has already given me a major speed boost, I'm not
gonna sweat over squeezing microseconds out of it unless I'm also
planning to run it in a tight loop 1,000,000 times per second.

And seriously, "ultra-optimized" code wastes more time than it saves,
because it makes code unreadable and unmaintainable, and the poor fool
who inherits your code will be wasting so much more time trying to
understand just what the heck your code is trying to do than if you had
written it in an obvious way that's slightly slower but doesn't really
matter anyway because it's not even part of any bottleneck.  All those
hours added up is a lot of wasted programmer wages which could have been
spent actually adding business value, like y'know implementing features
and fixing bugs instead of pulling out your hair trying to understand
just what the code is doing.

Do the rest of the world a favor by keeping such unreadable code out of
your codebase except where it's actually *necessary*.


T

-- 
Be in denial for long enough, and one day you'll deny yourself of things you
wish you hadn't.
Nov 12 2021
next sibling parent reply zjh <fqbqrr 163.com> writes:
On Saturday, 13 November 2021 at 01:12:32 UTC, H. S. Teoh wrote:

 On Fri, Nov 12, 2021 at 10:47:56PM

 Life is too short to fuss over code that isn't even relevant 
 to performance in the big picture.


 




Although you spoke very well, you took d into the `big hole`.
If there is always `GC`, I believe that `d` has no future.
Everybody's gone.
Nov 12 2021
parent reply zjh <fqbqrr 163.com> writes:
On Saturday, 13 November 2021 at 02:53:12 UTC, zjh wrote:

 On Saturday, 13 November 2021 at 01:12:32 UTC, H. S. Teoh wrote:



 Everybody's gone.


`Although` `D`'s Metaprogramming is beautiful, with a 'GC',I'd 
rather use a macro language.
Why there is no `GC` in the new language? Doesn't everyone see 
that `d` is biten by the `GC`?
Nov 12 2021
parent zjh <fqbqrr 163.com> writes:
On Saturday, 13 November 2021 at 03:02:30 UTC, zjh wrote:


 Why there is no `GC` in the new language? Doesn't everyone see 
 that `d` is biten by the `GC`?


Which language do you compete with when GC is holding you back?
Nov 12 2021
prev sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 13 November 2021 at 01:12:32 UTC, H. S. Teoh wrote:


 That's not what I was talking about, though.  I was talking 
 about programs that load in 1 second or less -- in which case 
 trying to optimize startup code is a waste of time.


It isn't. One second is four billion cycles. Or eight, sixteen, 
thirty two... if you're actually running multi-core. 60 full 
frames of smooth animation. Or gigabytes and gigabytes of data 
transferred. If you're not up and running in under a dozen 
milliseconds (i.e. one frame), you're slow. And if you are, you 
don't need any silly splash screens.


 If you're taking 8 seconds to start up, that *is* a performance 
 problem and you should be optimizing that!


Yeah, tell that to Microsoft ;) To them, 10s is fine. And they're 
not exactly outliers. Photoshop, Gimp, Krita, Maya, 3D Max, 
literally any video editor... Browsers, don't get me started on 
those.
This very web service takes like 5 full seconds to post a 
message. There was a time when 5 seconds wasted time online was A 
LOT.


 Given that I run my Vim sessions for years at a time, startup 
 time *is* completely irrelevant to me. :-D  If your program is 
 such that people have to constantly restart it, something is 
 bogonous with your design and you *should* consider scrapping 
 it and doing it better.


There may be other reasons for restarting. Such as voluntary or 
mandatory breaks, security regulations. If something is 
irrelevant to you does not necessarily mean it's irrelevant to 
the next guy.
Bottom line is, there's very little software that actually *has* 
to start slow. And the majority of what does start slow today 
isn't that.


 But you took what I said completely out of context.


I don't think I have.


 I was talking about optimizing the code where the profiler 
 indicates an actual bottleneck; 90% of your code is not part of 
 a bottleneck and therefore "optimizing" it prematurely is a 
 waste of time, not to mention it makes for needlessly 
 unreadable and unmaintainable code.


You don't optimize prematurely. It's often enough to just not 
pessimize for no reason. Using a fixed-size array or a 
static_vector instead of dynamic one here. Using a table instead 
of a bunch of math there. Or vice versa, depending. Doing a 
simple cursory assessment for a possibly better algorithm. Making 
one syscall instead of two. Something as trivial as aligning your 
data. Do you really need this ".array" call here, or can you do 
with a ".copy" into existing storage? Maybe layout your array 
differently and let multiple threads at it?..


 (An 8-second startup time *is* a bottleneck, BTW. Just so we're 
 on the same page.)  If code that isn't part of a bottleneck is 
 allocating and waiting for GC collection, who cares??


??? Everybody that comes after you cares! If Atila can make his 
thing run in 1µs, but *chose* to leave it at 1ms, because *to 
him*, anything slower than 20ms doesn't count as slow, he just 
flat out denied 999µs to someone else. Which can be centuries to 
that someone. There is no good reason not to do everything you 
can to stop your code from being slow. To get your data out that 
much sooner. To let the CPU go that much sooner. Either for other 
processes, or so that it can go to sleep and save that power.


 It's not the bottleneck, you should be spending your time 
 fixing actual bottlenecks instead.  That's my point.


Of course you should. But that doesn't mean that everywhere else 
you should be lazy. If it doesn't matter to you, be sure it does 
to someone who executes after you. Or someone who's sitting in 
front of the screen, or on the other end of the wire across the 
world. Or heck, to yourself: whether you'll have enough juice to 
make that important phone call on the weekend.


 Just the very act of writing it in D has already given me a 
 major speed boost, I'm not gonna sweat over squeezing 
 microseconds out of it unless I'm also planning to run it in a 
 tight loop 1,000,000 times per second.


Why not? Why wouldn't you? Ain't gettin' paid for that? I mean, 
it's a reason. Not a very good one, IMO, but a reason. But if not 
that, then what?


 And seriously, "ultra-optimized" code wastes more time than it


I'm not talking about "ultra-optimized" code.


 saves, because it makes code unreadable and unmaintainable, and 
 the poor fool who inherits your code will be wasting so much 
 more time trying to understand just what the heck your code is 
 trying to do than if you had written it in an obvious way 
 that's slightly slower but doesn't really matter anyway because 
 it's not even part of any bottleneck.


It does matter. Why do you keep insisting that it doesn't? It 
does. Just because it's not a bottleneck doesn't mean it's fine 
to just let it be slow. It's wasted cycles. No, you shouldn't 
necessarily "ultra-optimize" it. But neither you should just sign 
off on a sloppy work.


 All those hours added up is a lot of wasted programmer wages 
 which could have been spent actually adding business value, 
 like y'know implementing features and fixing bugs instead of 
 pulling out your hair trying to understand just what the code 
 is doing.

 Do the rest of the world a favor by keeping such unreadable 
 code out of your codebase except where it's actually 
 *necessary*.


No disagreements here.
Nov 12 2021
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 12 November 2021 at 22:10:22 UTC, H. S. Teoh wrote:

 This is why I also say, why are people so hung up about the GC? 
  If your inner loops have performance problems with the GC, 
 then rewrite it as  nogc, use malloc/free/whatever it takes to 
 make it faster.


If you have a large scannable heap then scanning will pollute the 
shared memory caches. Unless your scanning code use load/save 
instructions that bypass the caches. So in essence GC collection 
affects the whole system, also  nogc code. Another downside of 
imprecise scanning is that you need to zero out memory in order 
to make sure memory gets freed.

Now, if you tied a fully precise GC to a smaller unit, like a 
computation, then you could get better performance than other 
schemes since you often could just drop the whole heap without 
scanning when the computation is complete.

But the chosen path for D does not allow this, as of today, which 
does give people reason to wonder about the future direction for 
language evolution.
Nov 12 2021
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 12 November 2021 at 21:25:23 UTC, russhy wrote:

 On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves 
 wrote:


 Who, using a SYSTEMS language, should not have a reason to 
 care about their memory?


 Me, ~99.9% of the time.



 This explains a lot and why D is stuck with a miserable GC


We were waiting on your patches.
Nov 12 2021
parent russhy <russhy gmail.com> writes:
On Friday, 12 November 2021 at 23:05:44 UTC, deadalnix wrote:

 On Friday, 12 November 2021 at 21:25:23 UTC, russhy wrote:

 On Thursday, 11 November 2021 at 09:15:54 UTC, Atila Neves 
 wrote:


 Who, using a SYSTEMS language, should not have a reason to 
 care about their memory?


 Me, ~99.9% of the time.



 This explains a lot and why D is stuck with a miserable GC


 We were waiting on your patches.


You are advocating in favor of the GC, you should be the one to 
come with patches

I'm advocating for allocator driven APIs, but that would involve 
whover who started `std.experimental.allocator` to finish his 
work instead of trying to start bunch of new ones, including 
versioning what's not yet finished
Nov 12 2021
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 I think that C++'s greatest gift to the world was the 
 destructor. We have those too! Let's use them.


Indeed, but I have unfortunate news. We broke the gift. In D, 
object can be destroyed without being constructed first.

Consider: https://godbolt.org/z/EdW75jWGn

This is the first problem we need to fix here, if we don't want 
to have to plaster our code with runtime checks.
Nov 10 2021
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 10 November 2021 at 13:52:26 UTC, deadalnix wrote:

 On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 I think that C++'s greatest gift to the world was the 
 destructor. We have those too! Let's use them.


 Indeed, but I have unfortunate news. We broke the gift. In D, 
 object can be destroyed without being constructed first.

 Consider: https://godbolt.org/z/EdW75jWGn

 This is the first problem we need to fix here, if we don't want 
 to have to plaster our code with runtime checks.


The struct is instantiated [1], it just doesn't run the 
constructor in that case. You can always disable the default 
constructor. Doesn't C++ have an implicit default constructor 
also?


[1] https://dlang.org/spec/struct.html#struct-instantiation
Nov 10 2021
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 10 November 2021 at 14:04:18 UTC, jmh530 wrote:

 The struct is instantiated [1], it just doesn't run the 
 constructor in that case. You can always disable the default 
 constructor. Doesn't C++ have an implicit default constructor 
 also?


D does not allow a default constructor all… that is weird.

In C++ you get implicit constructors only if you provided no 
constructors at all.
Nov 10 2021
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Nov 10, 2021 at 01:52:26PM +0000, deadalnix via Digitalmars-d wrote:

 On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 I think that C++'s greatest gift to the world was the destructor. We
 have those too! Let's use them.
 


 
 Indeed, but I have unfortunate news. We broke the gift. In D, object
 can be destroyed without being constructed first.
 
 Consider: https://godbolt.org/z/EdW75jWGn
 
 This is the first problem we need to fix here, if we don't want to
 have to plaster our code with runtime checks.


Isn't that what  disable this() is for?

And to be pedantic, the object *was* constructed -- by the default ctor.
If you didn't want that, you should  disable this().


T

-- 
"The whole problem with the world is that fools and fanatics are always so
certain of themselves, but wiser people so full of doubts." -- Bertrand
Russell. "How come he didn't put 'I think' at the end of it?" -- Anonymous
Nov 10 2021
parent reply deadalnix <deadalnix gmail.com> writes:
On Wednesday, 10 November 2021 at 17:09:05 UTC, H. S. Teoh wrote:

 Isn't that what  disable this() is for?

 And to be pedantic, the object *was* constructed -- by the 
 default ctor. If you didn't want that, you should  disable 
 this().


 T


No.

Everything that is destructed must have been constructed. It is 
the only way you can enforce invariant within a type.
Nov 10 2021
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Wednesday, 10 November 2021 at 17:24:03 UTC, deadalnix wrote:

 On Wednesday, 10 November 2021 at 17:09:05 UTC, H. S. Teoh 
 wrote:

 Isn't that what  disable this() is for?

 And to be pedantic, the object *was* constructed -- by the 
 default ctor. If you didn't want that, you should  disable 
 this().


 T


 No.

 Everything that is destructed must have been constructed. It is 
 the only way you can enforce invariant within a type.


Invariants must hold also with structures initialised from 
'.init', if you don't want to  disable this()...
Nov 10 2021
parent tsbockman <thomas.bockman gmail.com> writes:
On Wednesday, 10 November 2021 at 18:50:26 UTC, Paolo Invernizzi 
wrote:

 On Wednesday, 10 November 2021 at 17:24:03 UTC, deadalnix wrote:

 On Wednesday, 10 November 2021 at 17:09:05 UTC, H. S. Teoh 
 wrote:

 Isn't that what  disable this() is for?

 And to be pedantic, the object *was* constructed -- by the 
 default ctor. If you didn't want that, you should  disable 
 this().


 T


 No.

 Everything that is destructed must have been constructed. It 
 is the only way you can enforce invariant within a type.


 Invariants must hold also with structures initialised from 
 '.init', if you don't want to  disable this()...


The semantics of `struct` `.init` initialization are (nearly) 
equivalent to mandating that `this()` be `pure`, allocate no 
memory, and work in CTFE. This does significantly limit what 
invariants can be established.
Nov 10 2021
prev sibling next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 10 November 2021 at 13:52:26 UTC, deadalnix wrote:

 On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 I think that C++'s greatest gift to the world was the 
 destructor. We have those too! Let's use them.


 Indeed, but I have unfortunate news. We broke the gift. In D, 
 object can be destroyed without being constructed first.

 Consider: https://godbolt.org/z/EdW75jWGn

 This is the first problem we need to fix here, if we don't want 
 to have to plaster our code with runtime checks.


How is that a problem, or, more to the point, how is D breaking 
anything here, as compared to C++? The latter effectively works 
by convention with its "valid, but unspecified state" for stdlib 
moved-from objects, and it's the best it can afford. We won't 
have that. The state is specified - it's .init. Right now it's 
also a mere convention, but it is effectively going to be set in 
stone with the move semantics DIP. It's a huge win as it allows 
the language to mandate rules for destructor elision, which C++ 
can't and won't do. I'd go so far as to argue that in a future 
version of D, the code presented above won't be calling a 
destructor at all.
Nov 10 2021
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 10 November 2021 at 20:35:57 UTC, Stanislav Blinov 
wrote:

 It's a huge win as it allows the language to mandate rules for 
 destructor elision, which C++ can't and won't do.


How come? In order to know that something has been moved, you 
would also know the state of the object, so I don't see how D get 
any advantage over C++ here. Unless you in the type system 
distinguish between active and inactive objects.
Nov 10 2021
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Wednesday, 10 November 2021 at 20:35:57 UTC, Stanislav Blinov 
wrote:

 How is that a problem, or, more to the point, how is D breaking 
 anything here, as compared to C++?


It break the construction/destruction invariant.


 The latter effectively works by convention with its "valid, but 
 unspecified state" for stdlib moved-from objects, and it's the 
 best it can afford. We won't have that. The state is specified 
 - it's .init. Right now it's also a mere convention, but it is 
 effectively going to be set in stone with the move semantics 
 DIP. It's a huge win as it allows the language to mandate rules 
 for destructor elision, which C++ can't and won't do. I'd go so 
 far as to argue that in a future version of D, the code 
 presented above won't be calling a destructor at all.


copy and destruction elision does not require to break the 
ctor/dtor invariant. I would know, the proposal that is now on 
the table is a variation of a proposal I made almost 10 years 
ago. And the move semantic part of it is broken.
Nov 10 2021
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 11 November 2021 at 00:35:56 UTC, deadalnix wrote:

 copy and destruction elision does not require to break the 
 ctor/dtor invariant. I would know, the proposal that is now on 
 the table is a variation of a proposal I made almost 10 years 
 ago. And the move semantic part of it is broken.


If it requires a move to set it to ```.init``` then it is at 
least highly inefficient. Just think about instances that are 
64KiB in size (like buffer nodes).
Nov 11 2021
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 10 November 2021 at 13:52:26 UTC, deadalnix wrote:

 On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 I think that C++'s greatest gift to the world was the 
 destructor. We have those too! Let's use them.


 Indeed, but I have unfortunate news. We broke the gift. In D, 
 object can be destroyed without being constructed first.

 Consider: https://godbolt.org/z/EdW75jWGn

 This is the first problem we need to fix here, if we don't want 
 to have to plaster our code with runtime checks.


` disable this();`, but you knew that. It's true that requiring a 
programmer to do something to prevent bugs is a terrible idea. 
Sigh.
Nov 11 2021
next sibling parent deadalnix <deadalnix gmail.com> writes:
On Thursday, 11 November 2021 at 09:24:17 UTC, Atila Neves wrote:

 On Wednesday, 10 November 2021 at 13:52:26 UTC, deadalnix wrote:

 On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 I think that C++'s greatest gift to the world was the 
 destructor. We have those too! Let's use them.


 Indeed, but I have unfortunate news. We broke the gift. In D, 
 object can be destroyed without being constructed first.

 Consider: https://godbolt.org/z/EdW75jWGn

 This is the first problem we need to fix here, if we don't 
 want to have to plaster our code with runtime checks.


 ` disable this();`, but you knew that. It's true that requiring 
 a programmer to do something to prevent bugs is a terrible 
 idea. Sigh.


 disable this just make the exemple slightly more convoluted, but 
the fundamental problem doesn't change.
Nov 11 2021
prev sibling parent reply Tejas <notrealemail gmail.com> writes:
On Thursday, 11 November 2021 at 09:24:17 UTC, Atila Neves wrote:

 On Wednesday, 10 November 2021 at 13:52:26 UTC, deadalnix wrote:

 On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 I think that C++'s greatest gift to the world was the 
 destructor. We have those too! Let's use them.


 Indeed, but I have unfortunate news. We broke the gift. In D, 
 object can be destroyed without being constructed first.

 Consider: https://godbolt.org/z/EdW75jWGn

 This is the first problem we need to fix here, if we don't 
 want to have to plaster our code with runtime checks.


 ` disable this();`, but you knew that. It's true that requiring 
 a programmer to do something to prevent bugs is a terrible 
 idea. Sigh.


Sorry for reviving this thread, was just sifting through...
The following code also outputs `dtor!`, unfortunately :(

```d

import std.stdio:writeln;

struct S{
	 disable this();
	~this(){
		writeln("dtor!");
	}
}	

void main(){
	S s = void;
}

```

Compiler :
LDC - the LLVM D compiler (1.25.0):
   based on DMD v2.095.1 and LLVM 11.1.0
Dec 14 2021
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:

 On Thursday, 11 November 2021 at 09:24:17 UTC, Atila Neves 
 wrote:



 ` disable this();`, but you knew that. It's true that 
 requiring a programmer to do something to prevent bugs is a 
 terrible idea. Sigh.


 Sorry for reviving this thread, was just sifting through...
 The following code also outputs `dtor!`, unfortunately :(

 ```d

 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this(){
 		writeln("dtor!");
 	}
 }	

 void main(){
 	S s = void;
 }

 ```



Why is this unfortunate? You void-initialize, it's on you now to 
ensure valid state, i.e. it's exactly the tongue-in-cheek 
"terrible idea" of Atila's ;)

Slap a  safe on that and it won't compile.
Dec 14 2021
parent reply Tejas <notrealemail gmail.com> writes:
On Tuesday, 14 December 2021 at 12:25:30 UTC, Stanislav Blinov 
wrote:

 On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:

 On Thursday, 11 November 2021 at 09:24:17 UTC, Atila Neves 
 wrote:



 ` disable this();`, but you knew that. It's true that 
 requiring a programmer to do something to prevent bugs is a 
 terrible idea. Sigh.


 Sorry for reviving this thread, was just sifting through...
 The following code also outputs `dtor!`, unfortunately :(

 ```d

 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this(){
 		writeln("dtor!");
 	}
 }	

 void main(){
 	S s = void;
 }

 ```



 Why is this unfortunate? You void-initialize, it's on you now 
 to ensure valid state, i.e. it's exactly the tongue-in-cheek 
 "terrible idea" of Atila's ;)

 Slap a  safe on that and it won't compile.


That's what I did after reading your message, with the bonus of 
slapping  safe on `S.__dtor` as well ;)

```d
import std.stdio:writeln;

struct S{
	 disable this();
	~this() safe/*now  safe*/{
		writeln("dtor!");
	}
}	

void main() safe/* now  safe*/{
	S s = void;
}
```

It defeated ` safe` ;_;
Dec 14 2021
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 14 December 2021 at 12:41:23 UTC, Tejas wrote:

 On Tuesday, 14 December 2021 at 12:25:30 UTC, Stanislav Blinov 
 wrote:



 Slap a  safe on that and it won't compile.


 That's what I did after reading your message, with the bonus of 
 slapping  safe on `S.__dtor` as well ;)

 ```d
 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this() safe/*now  safe*/{
 		writeln("dtor!");
 	}
 }	

 void main() safe/* now  safe*/{
 	S s = void;
 }
 ```

 It defeated ` safe` ;_;


Oh, right, S doesn't have any pointers.  safe only cares about 
void-initializing those.
Dec 14 2021
next sibling parent reply Tejas <notrealemail gmail.com> writes:
On Tuesday, 14 December 2021 at 12:51:23 UTC, Stanislav Blinov 
wrote:

 On Tuesday, 14 December 2021 at 12:41:23 UTC, Tejas wrote:

 On Tuesday, 14 December 2021 at 12:25:30 UTC, Stanislav Blinov 
 wrote:



 Slap a  safe on that and it won't compile.


 That's what I did after reading your message, with the bonus 
 of slapping  safe on `S.__dtor` as well ;)

 ```d
 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this() safe/*now  safe*/{
 		writeln("dtor!");
 	}
 }	

 void main() safe/* now  safe*/{
 	S s = void;
 }
 ```

 It defeated ` safe` ;_;


 Oh, right, S doesn't have any pointers.  safe only cares about 
 void-initializing those.


Should it be posted on [bug tracker](issues.dlang.org) ?
Dec 14 2021
next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 14 December 2021 at 12:53:29 UTC, Tejas wrote:

 On Tuesday, 14 December 2021 at 12:51:23 UTC, Stanislav Blinov 
 wrote:

 On Tuesday, 14 December 2021 at 12:41:23 UTC, Tejas wrote:

 On Tuesday, 14 December 2021 at 12:25:30 UTC, Stanislav 
 Blinov wrote:



 Slap a  safe on that and it won't compile.


 That's what I did after reading your message, with the bonus 
 of slapping  safe on `S.__dtor` as well ;)

 ```d
 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this() safe/*now  safe*/{
 		writeln("dtor!");
 	}
 }	

 void main() safe/* now  safe*/{
 	S s = void;
 }
 ```

 It defeated ` safe` ;_;


 Oh, right, S doesn't have any pointers.  safe only cares about 
 void-initializing those.


 Should it be posted on [bug tracker](issues.dlang.org) ?


It could do with an enhancement IMO. Your code specifically isn't 
causing any UB, but it's easy to see how practical code would. 
E.g. plain integers may refer to memory as well as any pointer.
Dec 14 2021
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 14 December 2021 at 12:53:29 UTC, Tejas wrote:

 On Tuesday, 14 December 2021 at 12:51:23 UTC, Stanislav Blinov 
 wrote:

 On Tuesday, 14 December 2021 at 12:41:23 UTC, Tejas wrote:

 On Tuesday, 14 December 2021 at 12:25:30 UTC, Stanislav 
 Blinov wrote:



 Slap a  safe on that and it won't compile.


 That's what I did after reading your message, with the bonus 
 of slapping  safe on `S.__dtor` as well ;)

 ```d
 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this() safe/*now  safe*/{
 		writeln("dtor!");
 	}
 }	

 void main() safe/* now  safe*/{
 	S s = void;
 }
 ```

 It defeated ` safe` ;_;


 Oh, right, S doesn't have any pointers.  safe only cares about 
 void-initializing those.


 Should it be posted on [bug tracker](issues.dlang.org) ?


If you can find a way to actually cause UB in  safe code, you 
should post it on the bug tracker. However, this example does not 
qualify.
Dec 14 2021
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/14/2021 6:05 AM, Paul Backus wrote:

 If you can find a way to actually cause UB in  safe code, you should post it
on 
 the bug tracker. However, this example does not qualify.


UB is not quite right. It's about memory unsafety. Another way to illustrate 
this is:

   import std.stdio;

    safe void main() {
     int x = void;
     writeln(x);
   }

will print garbage, but it is not unsafe.
Dec 15 2021
parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 15 December 2021 at 10:17:49 UTC, Walter Bright 
wrote:

 On 12/14/2021 6:05 AM, Paul Backus wrote:

 If you can find a way to actually cause UB in  safe code, you 
 should post it on the bug tracker. However, this example does 
 not qualify.


 UB is not quite right. It's about memory unsafety. Another way 
 to illustrate this is:

   import std.stdio;

    safe void main() {
     int x = void;
     writeln(x);
   }

 will print garbage, but it is not unsafe.


A violation of memory unsafety is either (a) UB or (b) behavior 
that is not UB itself, but allows UB to occur in  safe code 
(i.e., it violates one of the language's safety invariants). So 
if you've found a memory safety violation, it is always possible 
to use it to create an example of UB in  safe code.

The above example is neither memory-unsafe nor UB. According to 
the spec, it prints an implementation-defined value of type int.
Dec 15 2021
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/14/2021 4:51 AM, Stanislav Blinov wrote:

 That's what I did after reading your message, with the bonus of slapping  safe 
 on `S.__dtor` as well ;)

 ```d
 import std.stdio:writeln;

 struct S{
      disable this();
     ~this() safe/*now  safe*/{
         writeln("dtor!");
     }
 }

 void main() safe/* now  safe*/{
     S s = void;
 }
 ```

 It defeated ` safe` ;_;


 
 Oh, right, S doesn't have any pointers.  safe only cares about
void-initializing 
 those.


 safe is about memory safety. The example code is not memory unsafe.
Dec 15 2021
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:

 On Thursday, 11 November 2021 at 09:24:17 UTC, Atila Neves 
 wrote:

 On Wednesday, 10 November 2021 at 13:52:26 UTC, deadalnix 
 wrote:

 [...]


 ` disable this();`, but you knew that. It's true that 
 requiring a programmer to do something to prevent bugs is a 
 terrible idea. Sigh.


 Sorry for reviving this thread, was just sifting through...
 The following code also outputs `dtor!`, unfortunately :(

 ```d

 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this(){
 		writeln("dtor!");
 	}
 }	

 void main(){
 	S s = void;
 }

 ```

 Compiler :
 LDC - the LLVM D compiler (1.25.0):
   based on DMD v2.095.1 and LLVM 11.1.0


Is there an issue for this?
Dec 14 2021
next sibling parent reply user1234 <user1234 12.de> writes:
On Tuesday, 14 December 2021 at 15:51:58 UTC, Atila Neves wrote:

 On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:

 On Thursday, 11 November 2021 at 09:24:17 UTC, Atila Neves 
 wrote:

 On Wednesday, 10 November 2021 at 13:52:26 UTC, deadalnix 
 wrote:

 [...]


 ` disable this();`, but you knew that. It's true that 
 requiring a programmer to do something to prevent bugs is a 
 terrible idea. Sigh.


 Sorry for reviving this thread, was just sifting through...
 The following code also outputs `dtor!`, unfortunately :(

 ```d

 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this(){
 		writeln("dtor!");
 	}
 }	

 void main(){
 	S s = void;
 }

 ```

 Compiler :
 LDC - the LLVM D compiler (1.25.0):
   based on DMD v2.095.1 and LLVM 11.1.0


 Is there an issue for this?


There no issue there, `this` not used at all. Segfaults will 
start to happen, as expected, when trying to access a member 
variable.
Dec 14 2021
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 14 December 2021 at 16:17:30 UTC, user1234 wrote:

 On Tuesday, 14 December 2021 at 15:51:58 UTC, Atila Neves wrote:

 On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:

 On Thursday, 11 November 2021 at 09:24:17 UTC, Atila Neves 
 wrote:

 On Wednesday, 10 November 2021 at 13:52:26 UTC, deadalnix 
 wrote:

 [...]


 ` disable this();`, but you knew that. It's true that 
 requiring a programmer to do something to prevent bugs is a 
 terrible idea. Sigh.


 Sorry for reviving this thread, was just sifting through...
 The following code also outputs `dtor!`, unfortunately :(

 ```d

 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this(){
 		writeln("dtor!");
 	}
 }	

 void main(){
 	S s = void;
 }

 ```

 Compiler :
 LDC - the LLVM D compiler (1.25.0):
   based on DMD v2.095.1 and LLVM 11.1.0


 Is there an issue for this?


 There no issue there, `this` not used at all. Segfaults will 
 start to happen, as expected, when trying to access a member 
 variable.


No, they wouldn't, per current spec.

https://dlang.org/spec/declaration.html#void_init
https://dlang.org/spec/function.html#safe-functions

Per that, this is  safe:

```d
import core.sys.linux.fcntl;
import core.sys.linux.unistd;

struct MMap
{
     private int fd;
      disable this();
      disable this(this);
     // ...
     ~this()  trusted {
         if (isValid) {
             auto msg = "closed";
             size_t len = msg.length;
             write(fd, &len, len.sizeof);
             write(fd, msg.ptr, len);
             close(fd);
         }
     }

     private bool isValid() const  trusted {
         import core.stdc.errno;
         return fcntl(fd, F_GETFD) != -1 || errno != EBADF;
     }
}

void main()  safe
{
     // ...
     MMap mm = void; // currently allowed because MMap doesn't 
contain indirections
     // ...

} // nothing may happen, or may crash, or may write into someone 
else's memory, or to stdout...
```

Prolly should make an enhancement request for spec of  safe to 
disallow void initialization altogether.
Dec 14 2021
parent Paul Backus <snarwin gmail.com> writes:
On Tuesday, 14 December 2021 at 16:45:20 UTC, Stanislav Blinov 
wrote:

 On Tuesday, 14 December 2021 at 16:17:30 UTC, user1234 wrote:

 There no issue there, `this` not used at all. Segfaults will 
 start to happen, as expected, when trying to access a member 
 variable.


 No, they wouldn't, per current spec.

 https://dlang.org/spec/declaration.html#void_init
 https://dlang.org/spec/function.html#safe-functions

 Per that, this is  safe:

 ```d
 import core.sys.linux.fcntl;
 import core.sys.linux.unistd;

 struct MMap
 {
     private int fd;
      disable this();
      disable this(this);
     // ...
     ~this()  trusted {
         if (isValid) {
             auto msg = "closed";
             size_t len = msg.length;
             write(fd, &len, len.sizeof);
             write(fd, msg.ptr, len);
             close(fd);
         }
     }

     private bool isValid() const  trusted {
         import core.stdc.errno;
         return fcntl(fd, F_GETFD) != -1 || errno != EBADF;
     }
 }

 void main()  safe
 {
     // ...
     MMap mm = void; // currently allowed because MMap doesn't 
 contain indirections
     // ...

 } // nothing may happen, or may crash, or may write into 
 someone else's memory, or to stdout...
 ```

 Prolly should make an enhancement request for spec of  safe to 
 disallow void initialization altogether.


This is one of the problems that [DIP 0135 (` system` 
variables)][1] aims to solve. Specifically, it is the same class 
of problem described in [the `ShortString` example][2], where 
memory safety relies on the integrity of non-pointer data.

[1]: https://github.com/dlang/DIPs/blob/master/DIPs/DIP1035.md
[2]: 
https://github.com/dlang/DIPs/blob/master/DIPs/DIP1035.md#example-short-string
Dec 14 2021
prev sibling next sibling parent Tejas <notrealemail gmail.com> writes:
On Tuesday, 14 December 2021 at 15:51:58 UTC, Atila Neves wrote:

 On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:

 On Thursday, 11 November 2021 at 09:24:17 UTC, Atila Neves 
 wrote:

 On Wednesday, 10 November 2021 at 13:52:26 UTC, deadalnix 
 wrote:

 [...]


 ` disable this();`, but you knew that. It's true that 
 requiring a programmer to do something to prevent bugs is a 
 terrible idea. Sigh.


 Sorry for reviving this thread, was just sifting through...
 The following code also outputs `dtor!`, unfortunately :(

 ```d

 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this(){
 		writeln("dtor!");
 	}
 }	

 void main(){
 	S s = void;
 }

 ```

 Compiler :
 LDC - the LLVM D compiler (1.25.0):
   based on DMD v2.095.1 and LLVM 11.1.0


 Is there an issue for this?


Apparently one will have to develop a  program that will showcase 
UB with this type of code. Only then will a bug report be 
accepted.

And I lack the skill to do that :(
Dec 14 2021
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/14/2021 7:51 AM, Atila Neves wrote:

 On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:

 Sorry for reviving this thread, was just sifting through...
 The following code also outputs `dtor!`, unfortunately :(

 ```d

 import std.stdio:writeln;

 struct S{
      disable this();
     ~this(){
         writeln("dtor!");
     }
 }

 void main(){
     S s = void;
 }

 ```

 Compiler :
 LDC - the LLVM D compiler (1.25.0):
   based on DMD v2.095.1 and LLVM 11.1.0


 
 Is there an issue for this?


Should there be? Using void initialization means it's up to the user to 
initialize it properly before use.
Dec 15 2021
prev sibling parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:


 Sorry for reviving this thread, was just sifting through...
 The following code also outputs `dtor!`, unfortunately :(

 ```d

 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this(){
 		writeln("dtor!");
 	}
 }	

 void main(){
 	S s = void;
 }

 ```

 Compiler :
 LDC - the LLVM D compiler (1.25.0):
   based on DMD v2.095.1 and LLVM 11.1.0


This has come up in the past and typically folks have agreed that 
once you used void initialization it is your problem to make sure 
that the object is valid before destruction. I, personally, do 
not agree with that; since you are bypassing construction, you 
should also bypass destruction, however, an argument can be made 
for both sides.

My perspective is that currently you have an easy way to bypass 
destruction but you need to do the union trick to bypass 
destruction. I would vote for having a dip that would somehow 
make it easy to express the intent that you want to avoid 
destruction for a specific object.
Dec 15 2021
next sibling parent RazvanN <razvan.nitu1305 gmail.com> writes:
On Wednesday, 15 December 2021 at 08:27:25 UTC, RazvanN wrote:

 On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:


   [...]


 This has come up in the past and typically folks have agreed 
 that once you used void initialization it is your problem to 
 make sure that the object is valid before destruction. I, 
 personally, do not agree with that; since you are bypassing 
 construction, you should also bypass destruction, however, an 
 argument can be made for both sides.

 My perspective is that currently you have an easy way to bypass 
 destruction but you need to do the union trick to bypass 
 destruction. I would vote for having a dip that would somehow 
 make it easy to express the intent that you want to avoid 
 destruction for a specific object.


Typo! The first sentence of the last paragraph should be: "My 
perspective is that currently you have an easy way to bypass
construction but you need to do the union trick to bypass 
destruction"
Dec 15 2021
prev sibling next sibling parent Elronnd <elronnd elronnd.net> writes:
On Wednesday, 15 December 2021 at 08:27:25 UTC, RazvanN wrote:

 I would vote for having a dip that would somehow make it easy 
 to express the intent that you want to avoid destruction for a 
 specific object.


What about permitting 'union { S s; }' at function scope?  It is 
a slightly odd syntax; but it it consistent and, well, if you are 
trying to void initialise or bypass deconstruction, you are 
already plumbing dirty parts of the language.
Dec 15 2021
prev sibling next sibling parent Elronnd <elronnd elronnd.net> writes:
On Wednesday, 15 December 2021 at 08:27:25 UTC, RazvanN wrote:

 My perspective is that currently you have an easy way to bypass 
 destruction but you need to do the union trick to bypass 
 destruction. I would vote for having a dip that would somehow 
 make it easy to express the intent that you want to avoid 
 destruction for a specific object.


(In particular, I agree with the other folks you cite that 
void-initialisation should not bypass destruction.  To me, 
void-initialisation is not a way of _bypassing_ construction, but 
is an _optimization_ you can apply when the compiler tried to 
construct some object more than once.

But I agree that it should be possible to bypass destruction.)
Dec 15 2021
prev sibling parent reply Tejas <notrealemail gmail.com> writes:
On Wednesday, 15 December 2021 at 08:27:25 UTC, RazvanN wrote:

 On Tuesday, 14 December 2021 at 12:14:52 UTC, Tejas wrote:


 Sorry for reviving this thread, was just sifting through...
 The following code also outputs `dtor!`, unfortunately :(

 ```d

 import std.stdio:writeln;

 struct S{
 	 disable this();
 	~this(){
 		writeln("dtor!");
 	}
 }	

 void main(){
 	S s = void;
 }

 ```

 Compiler :
 LDC - the LLVM D compiler (1.25.0):
   based on DMD v2.095.1 and LLVM 11.1.0


 This has come up in the past and typically folks have agreed 
 that once you used void initialization it is your problem to 
 make sure that the object is valid before destruction. I, 
 personally, do not agree with that; since you are bypassing 
 construction, you should also bypass destruction, however, an 
 argument can be made for both sides.

 My perspective is that currently you have an easy way to bypass 
 destruction but you need to do the union trick to bypass 
 destruction. I would vote for having a dip that would somehow 
 make it easy to express the intent that you want to avoid 
 destruction for a specific object.


Sorry for the late reply.

What is this trick that bypasses destructors?

Also, I've always heard bad things about Unions, saying how they 
encourage bad practices like type punning. Is using this Union 
trick really recommended?
Dec 20 2021
parent RazvanN <razvan.nitu1305 gmail.com> writes:
On Tuesday, 21 December 2021 at 05:01:54 UTC, Tejas wrote:

 On Wednesday, 15 December 2021 at 08:27:25 UTC, RazvanN wrote:

 [...]


 Sorry for the late reply.

 What is this trick that bypasses destructors?


That was a typo. What I meant was: "There is an easy way to bypass
construction but ...". (What I'm referring to is void 
initialization)

When I realized, I submitted another forum post but you probably 
missed it.

 Also, I've always heard bad things about Unions, saying how 
 they encourage bad practices like type punning. Is using this 
 Union trick really recommended?
Dec 21 2021
prev sibling parent Paulo Pinto <pjmlp progtools.org> writes:
On Tuesday, 9 November 2021 at 18:33:01 UTC, Atila Neves wrote:

 On Tuesday, 9 November 2021 at 17:26:32 UTC, Stanislav Blinov 
 wrote:

 [...]


 Could you please explain why you'd rather do that instead of 
 using the equivalent of C++'s std::{vector, unique_ptr, 
 shared_ptr} and Rust's std::{vector, unique_ptr, shared_ptr}? I 
 cannot myself imagine why anyone would want to.

 I think that C++'s greatest gift to the world was the 
 destructor. We have those too! Let's use them.


Actually that honour belongs to Euclid, C++ made it popular.
Nov 10 2021
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 09.11.21 18:26, Stanislav Blinov wrote:

 
 It's the calling of such APIs that must be made  safe. Walter is 
 attempting just that with  live.


This is not true.


 Incidentally, if THAT is made possible, 
 you will be able to have the cake and eat it too.


But we are on the same page here.
Nov 09 2021
prev sibling parent deadalnix <deadalnix gmail.com> writes:
On Tuesday, 9 November 2021 at 15:37:13 UTC, Atila Neves wrote:

 The main point of vector is that one can append to it without 
 using the GC. I guess that "this thing has a length only known 
 at runtime that won't change and doesn't use the GC" is useful 
 too, but not nearly as much as being able to append.


Pedantic note, but as far as I can tell, this is a point that is 
often missed when discussing D: What you want isn't to avoid the 
GC, it is to avoid leaking.

You absolutely categorically want to allocate using the GC. If 
you don't, then you can't store anything that is managed by the 
GC within the vector, or you risk seeing it collected by the GC 
while it is still live.

What you do not want is to leak. Because leaking is what 
eventually leads to a GC cycle, which is actually what you want 
to avoid.
Nov 10 2021
prev sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 9 November 2021 at 15:26:24 UTC, Dukc wrote:


 Functions like these could also solve the thing if only called 
 from ` live`:


void foo(void*)  safe  nogc pure nothrow  live 
 doYouStillRememberWhatTheFunctionNameIs
{}

How many more attributes do we need before we realize that is not 
the way to write ANY code?
Nov 09 2021
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 09.11.21 17:12, Stanislav Blinov wrote:

 On Tuesday, 9 November 2021 at 15:26:24 UTC, Dukc wrote:
 

 Functions like these could also solve the thing if only called from 
 ` live`:


 
 void foo(void*)  safe  nogc pure nothrow  live 
  doYouStillRememberWhatTheFunctionNameIs
 {}
 
 How many more attributes do we need before we realize that is not the 
 way to write ANY code?


It's uniquely bad in the case of  live, as ownership/borrowing of 
arguments and return values is not actually a property of the function.
Nov 09 2021
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 9 November 2021 at 16:30:41 UTC, Timon Gehr wrote:


 It's uniquely bad in the case of  live, as ownership/borrowing 
 of arguments and return values is not actually a property of 
 the function.


Really? I thought  live wasn't unique. I mean, uh:

```
auto ref inout(T) thing(T)(auto ref return scope inout T 
delegate(ref return scope T) nothrow  nogc pure [1] dg...) return 
scope pure  nogc  safe nothrow;
```

'Tis of course off-topic to this discussion, but jeebuz, this has 
to end somewhere!
Nov 09 2021
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 9 November 2021 at 16:12:25 UTC, Stanislav Blinov 
wrote:

 How many more attributes do we need before we realize that is 
 not the way to write ANY code?


I don't know, but you have them in C++ too unfortunately.
Nov 09 2021
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
I think I have summarized the main points that has been brought up in 
this thread. If I have forgotten something please add.

- ``const``/``immutable`` memory can be in read only memory, attempting 
to modify this will result in the termination of a program.
- All ``extern(D)`` classes today cannot be in read only memory due the 
monitor field.
- Both ``const`` and ``immutable`` today provide guarantees that what 
memory is pointed to is readable. But it may not be writable and 
therefore cannot be written to.
- Any escape hatch must acknowledge that memory lifetimes is not the 
same thing as the memory containing data of a given type.
- For a value reference type, with the outer type being ``const`` is 
equivalent to just the template argument type being ``const``. ``struct 
Type(const T)`` is equivalent to ``const(Type!T)`` and 
``const(Type!(const(T)))``.
- Runtime checks are required for if the metadata or data storage is null.
- Atomic operations are expensive, to lesson this the compiler should 
elide calls to reference counting methods with the help of something 
like ARC and scope.
Nov 11 2021
next sibling parent reply Greg Strong <mageofmaple protonmail.com> writes:
On Friday, 12 November 2021 at 01:34:05 UTC, rikki cattermole 
wrote:

 - Any escape hatch must acknowledge that memory lifetimes is 
 not the same thing as the memory containing data of a given 
 type.


I'm sorry, can you expound on this one a little more?  I don't 
follow.
Nov 11 2021
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 12/11/2021 2:49 PM, Greg Strong wrote:

 On Friday, 12 November 2021 at 01:34:05 UTC, rikki cattermole wrote:

 - Any escape hatch must acknowledge that memory lifetimes is not the 
 same thing as the memory containing data of a given type.


 
 I'm sorry, can you expound on this one a little more?  I don't follow.


So the reference count needs to be heap allocated.
The value of a given type must be heap allocated as well.

These two memory addresses need not be next to each other.

I.e.

struct ReferenceCount {
	shared(int) count;
}

struct Node(Type) {
	Type value;
}

struct Reference(Type) {
	private {
		Node* data;
		ReferenceCount* refCount;
	}
}

Rather than:

struct Reference(Type) {
	private	Storage!type* storage;
}

struct Storage(Type) {
	Type value;
	shared(int) refCount;
}

The idea is that the value may be in read only memory, but the reference 
cannot be due to requirement of mutation.
Nov 11 2021
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12.11.21 02:34, rikki cattermole wrote:

 
 I think I have summarized the main points that has been brought up in 
 this thread. If I have forgotten something please add.
 
 - ``const``/``immutable`` memory can be in read only memory, attempting 
 to modify this will result in the termination of a program.


This is not an issue at all given an explicit escape hatch.


 - All ``extern(D)`` classes today cannot be in read only memory due the 
 monitor field.


The monitor field should be removed.


 - Both ``const`` and ``immutable`` today provide guarantees that what 
 memory is pointed to is readable. But it may not be writable and 
 therefore cannot be written to.
 - Any escape hatch must acknowledge that memory lifetimes is not the 
 same thing as the memory containing data of a given type.
 - For a value reference type, with the outer type being ``const`` is 
 equivalent to just the template argument type being ``const``. ``struct 
 Type(const T)`` is equivalent to ``const(Type!T)`` and 
 ``const(Type!(const(T)))``.
 - Runtime checks are required for if the metadata or data storage is null.
 - Atomic operations are expensive, to lesson this the compiler should 
 elide calls to reference counting methods with the help of something 
 like ARC and scope.


`immutable` data should not be implicitly shared.

- We have to define a complete set of allowed equivalence 
transformations based on type qualifiers. This is necessary so that 
there is a specification of what  trusted code may do. It is a 
precondition for any escape hatch like __mutable/__metadata.
Nov 12 2021
prev sibling next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
wrote:


 So if there are any takers on getting RCSlice off the ground, 
 it would be very interesting.


Well, let's explore what we can about what we break and try to 
figure out how can we improve the language so that we don't have 
to break.
Anyone interested, please submit your comments, fixes, tests, 
etc. etc. etc...

https://github.com/radcapricorn/dlang-rcslice
Dec 13 2021
prev sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Monday, 8 November 2021 at 21:42:12 UTC, Andrei Alexandrescu 
wrote:


 So if there are any takers on getting RCSlice off the ground, 
 it would be very interesting.


So... posting again, is no one up to destroy actual code?

https://github.com/radcapricorn/dlang-rcslice
Jan 14 2022