www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Memory/Allocation attributes for variables

reply Elmar <chrehme gmx.de> writes:
Hello dear D community,

personally I'm interested in security, memory safety and such 
although I'm not an expert.
I would like to know, if D has memory/allocation attributes (I 
will use both terms interchangeably), or if someone knows a 
library for that which statically asserts that only values 
allocated in compatible allocation regions are contained in 
variables of attributed type. Attributes for memory safety can be 
seen as extension to the type system because these memory 
attributes constrain the address domain of values instead of the 
value domain itself and so they become part of the value domain 
of a pointer or reference.

Now there are A LOT of different allocation regions and scopes 
for variables for whatever purpose:

  - static allocation
  - stack frame
  - dynamic allocation w/o GC
  - dynamic allocation with GC
  - fast allocators optimized for specific objects or even 
function-specific allocators
  - peripheral addresses
  - yes even register as allocation region (which allows a value 
to not be stored in RAM and thus being not easily overwritten, 
which is useful for security reasons like storing pointer 
encryption keys, stack canaries or to assign common registers to 
variables)
  - or memory-only allocation (which requires a value to not be 
stored/hold in registers)

Memory safety problems often boil down to that the program 
accidentally stores a pointer value into a variable being 
*semantically* out of bounds or pointing to a too small memory 
area w.r.t. the variable's purpose or scope. (Aliasing of 
variables in languages which better type-safety this probably 
impossible, typical is the case for unbounded data structures 
like C's variadic arguments or attacker-controlled 
variable-length arrays).

I don't know any other language yet which has allocation 
attributes for pointer/reference variables and allocation 
attributes for value-typed variables which restricts the 
allocation region for the data of the variable (some kind of 
contract on variable level). However, value-type variables are 
another story because they are allocated at the same time when 
defined and would only serve as expressive generalization of such 
attributes, generalized to value-types and even control 
structures.

Looking at what attributes D provides I definitly see that memory 
safety related concerns are addressed with existing attributes. 
But I personally find them rather unintuitive to use or difficult 
to learn for beginners and not flexible enough (like two 
different versions of `return ref scope`). As currently defined, 
those attributes don't annotate destinations of data flow but 
sources (like function arguments) of data flow.

What I imagine: More specific scopes (more specifically 
attributed types) or allocation regions correspond to more 
generalized types. Some scopes/allocation regions are contained 
within others (these smaller contained regions are virtually base 
types of bigger regions) and some regions are disjoint (but they 
should not intersect each other incompletely which is against the 
structured-programming paradigm and inheritance analogy in OOP). 
This results in Scope Polymorphy which statically/dynamically 
checks the address type of RHS expressions during assignments and 
memory safety becomes a special case of type safety.

I could annotate return values with attributes to make clear that 
a function returns GC-allocated memory, e.g. using a  gc 
attribute.

```d
 gc string[] stringifyArray(T : U[], U)(T arr) {
     import std.algorithm.iteration : map;
     import std.conv : to;
     return arr.map!(value => value.to!string);
}
 nogc stringtable = stringifyArray([1, 2, 3]);    // error!

// a useless factory example
 new auto makeBoat(double length, double height, Color c) {
     theAllocator = Mallocator;

     auto b = new Boat(length, height, c);

     theAllocator = processAllocator;
     return b;
}

// combining multiple attributes gives a union of both which is 
valid for reference variables
 new  newcpp  gc Boat = makeBoat( ... );
// technically, a union of attributes for value types is possible 
but would
// require inferring the most appropriate attribute from context 
which is difficult
```

Variables with no attributes allow any pointer for assignment and 
will infer the proper attribute from the assignment.

Some of these use cases are already covered by existing 
attributes:

  - `scope` makes sure, that a reference variable is not written 
to an allocation region outside the current function block (which 
corresponds to using "` scope(function)`" with the argument, see 
below) and it would be type-unsafe to assign it to a variable 
type with larger scope. "Scope" basically means, the argument 
belongs to a stack frame in the caller chain. (It corresponds to 
arguments annotated with "` caller`", see below.) It's used to 
tell the function that the referenced value has a limited 
lifetime in a caller stack frame despite being a reference 
variable and the reference could become invalid after the 
function returns so it must not write the value to variables 
outside the function. For arguments this is very useful and I 
would rather prefer the complementary case to be explicit. That's 
where `in` is really useful as a short form.
  - `ref` specifies that the actual allocation region of a 
variable's value is outside of the function scope in which the 
variable is visible (or used). (`out` is similar.)
  - `return ref` specifies that the value (referenced by the 
returned reference) is in the same allocation scope as the 
argument annotated with `return ref` (corresponds to the 
annotation of the return type with "` scope(argName)`", see 
below).
  - `return ref scope`, a combination of two above. The return 
type is seen to have the same allocation region equal to the one 
used by this annotated argument.
  - `__gshared`, `shared`. Variables with these attributes save 
them in a scope accessible across threads. This is the default in 
C so that `__gshared` corresponds to C's volatile values which 
are accepted by ` memory` references.

Here is a (really long) collection of many possible memory 
attributes I am looking for. They define which addresses of 
values are accepted for the pointer/reference:

  -  auto: allocation in any stack frame, which includes 
fixed-size value-type variables passed as arguments or return 
value
  -  stack: dynamic allocation in any stack frame (alloca)
  -  loop: allocation in the region which lives as long as the 
current loop lives
  -  scope(recursion): allocation-scope not yet available in D I 
believe, scope which lives as long as the entire recursion lives, 
equivalent to `loop` in the functional sense. Locals in this 
scope are accessible to all recursive calls of the same function.
  -  scope(function): allocation in the current stack frame 
(`scope`d arguments are a special case of this)
  -  scope(label): allocation in the scope of the labeled control 
structure
  -  scope(identifier): allocation in the same scope as the 
specified variable name, `return ref` can be seen as special case 
for return types.
  -  static: allocation/scope in static memory segment (lifetime 
over entire program runtime), `static` variables and control 
structures are a special case of this attribute
  -  caller: allocation in the caller's stack frame (usuable for 
convenient optimizations like shown below), an "implicit 
argument" when used for value types, corresponds to `ref scope` 
for reference-type variables. Something in between "static" and 
"auto".
  -  gc: allocation region managed by D's garbage collector
  -  nogc: disallows pointer/reference to GC-allocated data
  -  new: allocation region managed by Mallocator
  -  newcpp: allocation region managed by stdcpp allocator thing, 
eases C++ compatibilty
  -  peripheral: target- or even linkerscript-specific memory 
region for peripherals
  -  register: only stored in a register (with compile-time error 
if not possible)
  -  shared: allocation region for values which are synchronized 
between threads
  -  memory: never stored in a register (use case can overlap with 
"` peripheral`", it's used for variables whose content can change 
non-deterministically and must be reloaded from memory each time, 
for example interrupt-handler modified variables, it also 
prevents optimization when unwanted)
  -  make(allocator): allocated by the given allocator (dynamic 
type check required, if "allocator" is a dynamic object)

In the basic version for reference variables these attributes 
statically/dynamically assert that a given pointer value is in 
bounds of that allocation region. Of course, this is a long list 
of personal ideas and some of them could be unpopular in the 
community. But I think, all of them would be a tribute to Systems 
programming.

Why are such attributes useful? At first because type-safe design 
means to restrict value domains as much as possible so that it is 
only as large as required. They restrict the address (pointer 
value) at which a value bounded by a variable can be located and 
provide additional static type checks as well as *allocation 
transparency* (something which I miss in every language I used so 
far). The good thing is, if no attribute is provided, it can be 
inferred from the location where the value-typed variable is 
defined or is inferred from the assigned pointer value for 
reference types.
Maybe also useful: with additional memory safety attributes, it 
could become legitimate to assign to `scope`d reference variables.

For reference-type variables, these attributes are simple value 
domain checks of the pointer variable. A disadvantage of memory 
attributes is (like with polymorphy) that runtime checks might be 
needed in some cases when static analysis isn't sufficient (if 
attributes are casted).

An interesting extension is a generalization to value-type 
variables. It can generalize the `scope` and `return` attribute 
to value-types. While probably not un-controversal it could allow 
fine control over variable allocation and force where a 
value-typed variable is allocated exactly (allocation 
guarantees). You could indirectly define a variable in a nested 
code block which is allocated for another scope. The main 
disadvantage I can think of is only, that it cannot be just 
created as a library add on.

```d
outer: {
     // ...
      scope(inner) uint key = generateKey(seed);  // precomputes 
the RHS
     // and initializes the LHS with the memorized value when 
entering the "inner" block
     seed.destroy();    // do something with seed, modify/destroy 
it, whatever
     // key is not available/accessible here
     // Message cipher;   // <-- implicit but uninitialized
     inner: if (useSecurity) {
         // if not entered, the init-value of the variable is used
          scope(outer) Message cipher = encrypt(key);
         // Implicitly defines "cipher" uninitialized in "outer" 
scope.
         // Generates default init in all other control flow paths 
without  scope(outer) definition
     }
     //else cipher = Messsage.init;   // <-- implicit, actual 
initialization
     decrypt(cipher, key);    // error, key is only available in 
the "inner" scope
}
```

Some would criticize the unconventional visibility of `cipher` 
which doesn't follow common visibility rules. For example if 
`static` variables are defined in functions, they are still only 
visible in the function itself and not in the entire scope in 
which they live. So a likely improvement would be that the 
visibility is not impacted by the attribute, only the point of 
actual creation/destruction. Just looking at the previous 
example, it would seem useless at first, but it's not if loops 
are considered (and variables which have ` loop` scope, that 
means are created on loop entry and only destructed on loop exit).

Also interesting cases can emerge for additional user 
optimization in order to avoid costly recomputation by using a 
larger scope as allocation region:

```d
double permeability(Vec2f direction) {
      caller Vec2f grad = calculateTextureDerivative();
     // "grad" is a common setup shared by all calls to 
"permeability" from the same caller instance
     // It is hidden from the caller because it's an 
implementation detail of this function.
     // All calls of "grad" by the same caller will use the same 
variable.
     // It would be implemented as invisible argument whose 
initialization
     // happens in the caller. The variable is stored on the 
caller's site as
     // invisible variable and is passed with every call.
     return scalprod(direction, grad);
}
```

A main benefit of this feature is readability and in some cases 
optimization because the executed function is not repeated for 
every call, only if the repetition is needed which can be 
computed in the callee instead.
For closures the ` caller` scope is clear but it also works for 
non-closure functions as an invisible argument. Modifications to 
a ` caller ref` variable are remembered for consecutive calls 
from the same caller stack frame whereas ` caller` without ref 
maybe only modifies a local copy.

Or being able to create Arrays easily on the stack which is yet a 
further extension

```d
 auto arr1 = [0, 1, 2, 3];  // asserts fixed-size, okay, but 
variable size would fail
 stack arr2 = dynamicArr.dup;   // create a copy on stack, the 
stack is "scope"d
```

An easy but probably limited implementation would set 
`theAllocator` before the initialization of such an attributed 
value-type variable and resets `theAllocator` afterwards to the 
allocator from before.


Finally, one could even more generally annotate control 
structures with attributes to define in whose scope's entry the 
control structure's arguments are evaluated (e.g. `static if` is 
a special case which represents ` static if` in terms of 
attributes) but this yet another different story and unrelated to 
allocation.


This is it, I'm sorry for the long post. It took me a while to 
write it down and reread.
Regards!
May 29
parent reply Ola Fosheim Grostad <ola.fosheim.grostad gmail.com> writes:
On Sunday, 30 May 2021 at 02:18:38 UTC, Elmar wrote:
 Looking at what attributes D provides I definitly see that 
 memory safety related concerns are addressed with existing 
 attributes. But I personally find them rather unintuitive to 
 use or difficult to learn for beginners and not flexible enough 
 (like two different versions of `return ref scope`). As 
 currently defined, those attributes don't annotate destinations 
 of data flow but sources (like function arguments) of data flow.
I agree that D has jumped down the rabbithole in terms of usability and function signatures are becoming weirder. The reusage of the term "return" is particularly bad. To a large extent this is the aftermath that comes from changing course when it went from D1 to D2. Where simplicity was sacrificed and it was opened for more and more complexity. Once a language becomes complex, it seems difficult to prevent people from adding just-one-more-feature that adds to the complexity. Also, since experienced users influence the process most... There are nobody to stop it. The main issue is however not specifying where it allocates, but keeping track of it when pointers are put into complex datastructures.
May 29
parent reply Elmar <chrehme gmx.de> writes:
On Sunday, 30 May 2021 at 05:13:45 UTC, Ola Fosheim Grostad wrote:
 On Sunday, 30 May 2021 at 02:18:38 UTC, Elmar wrote:
 Looking at what attributes D provides I definitly see that 
 memory safety related concerns are addressed with existing 
 attributes. But I personally find them rather unintuitive to 
 use or difficult to learn for beginners and not flexible 
 enough (like two different versions of `return ref scope`). As 
 currently defined, those attributes don't annotate 
 destinations of data flow but sources (like function 
 arguments) of data flow.
I agree that D has jumped down the rabbithole in terms of usability and function signatures are becoming weirder. The reusage of the term "return" is particularly bad. To a large extent this is the aftermath that comes from changing course when it went from D1 to D2. Where simplicity was sacrificed and it was opened for more and more complexity. Once a language becomes complex, it seems difficult to prevent people from adding just-one-more-feature that adds to the complexity. Also, since experienced users influence the process most... There are nobody to stop it. The main issue is however not specifying where it allocates, but keeping track of it when pointers are put into complex datastructures.
Thank you for your reply. Also sorry for the wordyness, I'm just awkwardly detailed sometimes. In your case it's not what I was thinking. I would count myself to the sophisticated programmers (but not the productive ones, unfortunately). I can cope with all those reused keywords even though I think at this place their design is unintuitive to use. Intuitive would be annotation of the return type because the aliasing is a property of the return type, not the argument. At least I feel like I understood the sparsely explained intension behind the current scope-related attributes but my main point is, I find they can be improved with more expressiveness. It would give programmers a hint of what kind of allocated argument is acceptable for a parameter. And no, this is not trivial. It's the reason for my decision to start this thread: *Functions in phobos accept range iterators of fixed-sized arrays as range argument but even if it fails miserably, it compiles happily and accesses illegal memory without any warning, creating fully non-deterministic results with different compilers. I noticed this when I tried to use "map" with fixed-size arrays. It simply misses any tool to check and signal that fixed-size arrays are illegal as range argument for "map". And sometimes mapping onto fixed-size arrays even works.* Without better memory safety tools, I'd discourage more memory efficient programming techniques in D although I'd really like to see D for embedded and resource constrained systems to replace C. --- I wonder how programming languages don't see the obvious, to consider memory safety as a part of type safety (address/allocation properties to be type properties) and that memory unsafe code only means an incomplete type system. I also don't know whether conventional "type safety" in programming languages suffices to eliminate the possibility of deadly programming bugs (aliased reference variables e.g.). But of course, security and safety is complex and there is no way around complexity to make safe code flexible. The important part is the first one (without generalization to allocation and control structures which I only mentioned as an interesting thought) because I think it's an easy but effective addition. D already has features in that direction which is good, the awareness exists, but it's still weak at some points. My post should be seen as a collection of ideas and a request for comment (because maybe my ideas are totally bad or don't fit D) rather than a request to implement all this. The main point is to consider references/pointers as values with critical type safety which means a way to specify stricter constraints. Memory safety is violated by storing a pointer value in a reference which is out of the intended/reasonable value domain of the pointer (not matching its lifetime). If someone already thought the same like me, there could be a safepointer-like user library which supports additional attributes (which represent restricted pointer domains) by implementing a custom pointer/reference type. (It's not a smart pointer because smart pointers try to fix the problem at the other end and require dynamic allocation which is not that nice.) Due to D's nature, it would support safe pointers and safe references (reference variables) and provides static and dynamic type checks with overloaded operators and memory attributes. Attributes couldn't be inferred automatically I guess but annotation of variables could entirely allow static memory safety checks (which doesn't need to explicitly test whether a pointer value is contained in a set of allowed values) and maybe prevents bugs or unwanted side effects. --- One important aspect which I forgot: aliasing of variables. I know, D allows aliased references as arguments by default. Many memory safety problems derive from aliased variables which were not assumed to be aliased. Aliased variables complicate formal verification of code and confuse people. I would add ` alias(symbol)` to my collection which indicates that a reference explicitly aliases (overlap) another reference in memory or a ` noalias(symbol)`. --- If someone thinks, I heavily missed something, please let me know.
May 31
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 31 May 2021 at 18:21:26 UTC, Elmar wrote:
 I wonder how programming languages don't see the obvious, to 
 consider memory safety as a part of type safety 
 (address/allocation properties to be type properties) and that 
 memory unsafe code only means an incomplete type system.
All high level programming-languages do. Only the low level don't, and that is one of the things what makes their type systems unsound.
 constraints. Memory safety is violated by storing a pointer 
 value in a reference which is out of the intended/reasonable 
 value domain of the pointer (not matching its lifetime).
But how do you keep track of it without requiring that all graphs are acyclic? No back pointers is too constraining. And no, Rust does not solve this. Reference counting does not solve this. How do you prove that a graph remains fully connected when you change one pointer?
 One important aspect which I forgot: aliasing of variables. I 
 know, D allows aliased references as arguments by default. Many 
 memory safety problems derive from aliased variables which were 
 not assumed to be aliased.
So, how do you know that you don't have aliasing when you provide pointers to two graphs? How do you prove that none of the nodes in the graph are shared?
May 31
parent reply Elmar <chrehme gmx.de> writes:
Good questions :-) .

On Monday, 31 May 2021 at 18:56:44 UTC, Ola Fosheim Grøstad wrote:
 On Monday, 31 May 2021 at 18:21:26 UTC, Elmar wrote:
 I wonder how programming languages don't see the obvious, to 
 consider memory safety as a part of type safety 
 (address/allocation properties to be type properties) and that 
 memory unsafe code only means an incomplete type system.
All high level programming-languages do. Only the low level don't, and that is one of the things what makes their type systems unsound.
I suppose you mean the "higher" level languages (because C is by original definition also a high-level language). I neither know any "higher" level language which provides the flexibility of constraining the value domain of a pointer/reference except for restricting `null` (non-nullable pointers are probably the most simple domain constraint for pointers/references). I think, not even Ada nor VHDL have it. The thing I'd like to gain with those attributes is a guarantee, that the referenced value wasn't allocated in a certain address region/scope and lives in a lifetime-compatible scope which can be detected by checking the pointer value against an interval or a range of intervals. For example a returned reference to an integer could have been created with "malloc" or even a C++ allocator or interfacing functions could annotate parameters with such attributes. With guarantees about the scope of arguments function implementations can avoid buggy reference assignments to outside variables. The function could expect compatible references allocated with GC but the caller doesn't know it. Whether any reference variable assignment is legitimate can be checked by comparing the source attributes (the reference value which says where the value is allocated) with the destination attributes (where the reference is stored in memory). Even better are runtime checks of pointer values for a better degree of memory safety but only if the programmers want to use it. A reference assignment is legitimate if the destination scope is compatible with the source's scope, not in any other case. I would suggest a lifetime rating for value addresses as follows: *peripheral > system/kernal > global shared > private global (TLS) > extern global (TLS) > shared GC allocated > shared dynamically allocated > GC allocated (TLS) > dynamically allocated (TLS) <=> RAII/scoped/stack <=> RAII/scoped/stack > register* Heap regions are not always comparable to stack or RAII. So the current practice of not allowing assignment to RAII references (using `scope` attribute) is probably best to continue. Everything other than stack addresses are seen as one single lifetime region with equal lifetime. The comparison between stack addresses assumes that an address deeper in the stack has a higher or equal lifetime. The caller could also provide it's stack frame bounds which allows to consider this interval as one single lifetime. It should constrain the possible value domain of pointers absolutely so that no attack with counterfeited pointers to certain memory addresses is possible. If I would use custom allocators for different types I could expect or delimit what the pointer value can be. On Monday, 31 May 2021 at 18:56:44 UTC, Ola Fosheim Grøstad wrote:
 constraints. Memory safety is violated by storing a pointer 
 value in a reference which is out of the intended/reasonable 
 value domain of the pointer (not matching its lifetime).
But how do you keep track of it without requiring that all graphs are acyclic? No back pointers is too constraining. And no, Rust does not solve this. Reference counting does not solve this. How do you prove that a graph remains fully connected when you change one pointer?
I think, this is GC-related memory management, not type checking. The memory attributes don't solve memory management problems. The problem with reference counting usually is solved by inserting weak pointers into cycles (which also solves the apparent contradiction of a cycle of references). Weak references are used by those objects which are deeper in the graph of data links. Otherwise it's a code smell and one could refactor the links into a joint object and deleted objects will deregister in this joint object. I already thought about other allocation schemes for detecting cycles that could be combined with reference counting. For example tagging structs/classes with the ID of the conntected graph in which they are linked if they aren't leaves. But this ID is difficult to change. It can also analyze at compile time which pointers can only be part of a cycle but more explanation leads to far here. Instead the problem, my idea is intended to solve, is 1. giving hints to programmers (to know which kind of allocated memory works with the implementation, stack addresses apparently won't generally work with `map` for example) 2. having static or dynamic (simple) value domain checks (which checks whether a pointer value is in the allowed interval(s) of the allocation address spaces belonging to the attributes) which ensures that only allowed types of allocation are used. These checks can be used to statically or dynamically dispatch functions. Of course such a check could also be performed manually but it's tedious and requires me to put all different function bodies in one `static if else`. It's more of a lightweight solution and works like an ordinary type check (value-in-range check). Where the feature shines most is function signatures because they separate code and create intransparency which can be countered by memory attributes for return type and argument types. On Monday, 31 May 2021 at 18:56:44 UTC, Ola Fosheim Grøstad wrote:
 One important aspect which I forgot: aliasing of variables. I 
 know, D allows aliased references as arguments by default. 
 Many memory safety problems derive from aliased variables 
 which were not assumed to be aliased.
So, how do you know that you don't have aliasing when you provide pointers to two graphs? How do you prove that none of the nodes in the graph are shared?
Okay, I didn't define aliasing. With "aliasing" I mean that "aliasing references" (or pointers) either point to the exact same address or that the immediately pointed class/struct (pointed to by the reference/pointer) does not overlap. I would consider anything else more complicated than necessary. The definition doesn't care about further indirections. I often only consider the directly pointed struct or class contiguous chunk of memory as "the type". If I code a function, I'm usually only interested in the top level of the type (the "root node" of the type) and further indirections are handled by nested function calls. For example it suffices, if two argument slices are not overlapping. For that I only need to check aliasing as just defined. If you really would like two arguments (graphs) to not share any single pointer value I would suggest using a more appropriate type than a memory attribute, a type which is recursively "unique" (in terms of only using "unique pointers"). Do you think, it sounds like a nice idea to have a data structure attribute `unique` next to `abstract` and `final` which recursively guarantees that any reference or pointer is a unique pointer? If you are interested for a algorithmic answer to your questions, then the best approach (I quickly can think of) is creating an appropriate hash table from all pointers in one graph and testing all pointers in the other graph against it (if I cannot use any properties on the pointers' values, e.g. that certain types and all indirections are allocated in specific pools). But that only works with exactly equal pointer values.
May 31
parent reply Ola Fosheim Grostad <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 1 June 2021 at 00:36:17 UTC, Elmar wrote:
 On Monday, 31 May 2021 at 18:56:44 UTC, Ola Fosheim Grøstad 
 wrote:
 I suppose you mean the "higher" level languages (because C is 
 by original definition also a high-level language).
Yes, I mean system level language vs proper high level languages that abstract away the hardware. "Low level" is not correct, but the usage of the term "system level" tend to lead to debates in these fora as there is a rift between low level and high level programmers...
 The thing I'd like to gain with those attributes is a 
 guarantee, that the referenced value wasn't allocated in a 
 certain address region/scope and lives in a lifetime-compatible 
 scope which can be detected by checking the pointer value 
 against an interval or a range of intervals. For example a 
 returned reference to an integer could have been created with 
 "malloc" or even a C++ allocator or interfacing functions could 
 annotate parameters with such attributes.
Well, I guess you are new, but Walter will refuse having many pointer types. Even the simple distinction between gc and raw pointers will be refused. The reason being that it would lead to a combinatorial explosion of function instances and prevent separate compilation. So for D, this is not a probable solution. That means you cannot do it through the regular type system, so that means you will have to do shape analysis of datastuctures. I personally have in the past argued that it would be an interesting experiment to make all functions templates and template pointer parameter types. That you can do with library pointer types, as a proof of concept, yourself. Then you will see what the effect is.
 lifetime region with equal lifetime. The comparison between 
 stack addresses assumes that an address deeper in the stack has 
 a higher or equal lifetime. The caller could also provide it's 
 stack frame bounds which allows to consider this interval as 
 one single lifetime.
How about coroutines? Now you have multiple stacks.
 I think, this is GC-related memory management, not type 
 checking. The memory attributes don't solve memory management 
 problems. The problem with reference counting usually is solved 
 by inserting weak pointers into cycles (which also solves the 
 apparent contradiction of a cycle of references). Weak 
 references are used by those objects which are deeper in the 
 graph of data links.
No, depth does not work, you could define an acyclic graph of owning pointers and then use weak pointers elsewhere. This restricts modelling and algorithms. So compiler verified restriction of non-weak references might be too restrictive? So basically whenever changing a non-weak reference the compiler has to prove that the graph still is non-weak acyclic. Maybe possible, but does not sound trivial.
  2. having static or dynamic (simple) value domain checks 
 (which checks whether a pointer value is in the allowed 
 interval(s) of the allocation address spaces belonging to the 
 attributes) which ensures that only allowed types of allocation 
 are used. These checks can be used to statically or dynamically 
 dispatch functions. Of course such a check could also be 
 performed manually but it's tedious and requires me to put all 
 different function bodies in one `static if else`.
Dynamic checks are unlikely to be accepted, I suggest you do this as a library.
 Where the feature shines most is function signatures because 
 they separate code and create intransparency which can be 
 countered by memory attributes for return type and argument 
 types.
Unfortunately, this is also why it will be rejected.
 Okay, I didn't define aliasing. With "aliasing" I mean that 
 "aliasing references" (or pointers) either point to the exact 
 same address or that the immediately pointed class/struct 
 (pointed to by the reference/pointer) does not overlap. I would 
 consider anything else more complicated than necessary.
Insufficient for D with library container types and library smart pointers.
 Do you think, it sounds like a nice idea to have a data 
 structure attribute `unique` next to `abstract` and `final` 
 which recursively guarantees that any reference or pointer is a 
 unique pointer?
Yes, some want isolated pointers, but you have to do all this stuff as library smart pointers in D.
May 31
next sibling parent Ola Fosheim Grostad <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 1 June 2021 at 06:12:05 UTC, Ola Fosheim Grostad 
wrote:
 No, depth does not work, you could define an acyclic graph of 
 owning pointers and then use weak pointers elsewhere.
What I meant here is that depth would be too restrictive as it would prevent reasonable insertions nodes.
May 31
prev sibling parent reply Elmar <chrehme gmx.de> writes:
Thank you for answering.

On Tuesday, 1 June 2021 at 06:12:05 UTC, Ola Fosheim Grostad 
wrote:
...
 The thing I'd like to gain with those attributes is a 
 guarantee, that the referenced value wasn't allocated in a 
 certain address region/scope and lives in a 
 lifetime-compatible scope which can be detected by checking 
 the pointer value against an interval or a range of intervals. 
 For example a returned reference to an integer could have been 
 created with "malloc" or even a C++ allocator or interfacing 
 functions could annotate parameters with such attributes.
Well, I guess you are new, but Walter will refuse having many pointer types. Even the simple distinction between gc and raw pointers will be refused. The reason being that it would lead to a combinatorial explosion of function instances and prevent separate compilation.
The separate compilation is a good point. Binary compatibility is a common property considered for security safeguards. But at least static checking with attributes would need no memory addresses at all (also if the compiler can infer the attribute for every value-typed variable automatically from where it is defined). Dynamic checks of pointers accross binary interfaces are difficult. It would work flawlessly with library-internal memory regions but for outside pointer values it can only rely on runtime information (memory regions used by allocators) or cannot perform checks at all (because it doesn't know the address ranges to check against). Or it would work better if binaries would support relocations for application-related memory addresses which are filled at link time. Static checks strike the balance here.
 I personally have in the past argued that it would be an 
 interesting experiment to make all functions templates and 
 template pointer parameter types.

 That you can do with library pointer types, as a proof of 
 concept, yourself. Then you will see what the effect is.
Okay, that's fine. Pointers in D are not debatable, I would not try. I think, any new language should remove the concept of pointers entirely rather than introducing new pointers. Pointers from C should be treated as reference variables, pointers to C as either an unbounded slice (if bounded, there should be another `size_t` argument to the function) or it passes addresses obtained from variables. As a C programmer I'd say that C's pointer concept was never needed as it stands, it just was created to be an unsafe reference variable + a reference + an iterator all-in-one-solution as the simplest generic thing which beats it all (without knowing the use case by looking at the pointer type). Attributes only would check properness of pointer value assignments without code duplication of the function as `auto ref` is doing. (One can still interprete it as part of the type.) On Tuesday, 1 June 2021 at 06:12:05 UTC, Ola Fosheim Grostad wrote:
 lifetime region with equal lifetime. The comparison between 
 stack addresses assumes that an address deeper in the stack 
 has a higher or equal lifetime. The caller could also provide 
 it's stack frame bounds which allows to consider this interval 
 as one single lifetime.
How about coroutines? Now you have multiple stacks.
Thanks, I missed that, at least true coroutines have. Other things also can dissect stack-frame memory (function-specific allocators in the stack-region). But in our case, it's already a question whether such special stack frames still should be allocated in the stack-region, statically (as I implemented it once for C) or in a heap region (like stack frames of continuations). You could at least place coroutine stack frames in some allocator region in static memory. A probably less fragile but more costly solution (when checking stack-addresses) for stack address scope would be storing the stack depth of an address in the upper k-bit portion of a wide pointer value (for a simple check) but this is only a further unrelated idea.
 Dynamic checks are unlikely to be accepted, I suggest you do 
 this as a library.
Right, if nobody tried it so far I'd like myself. Then I can firm my D experience with further practice. I'd compare the nature of static and dynamic attribute checks to the nature of C++ `static_cast` and `dynamic_cast` of class pointers. I was thinking, such a user library could use `__traits` with templated operator overloads.
 Where the feature shines most is function signatures because 
 they separate code and create intransparency which can be 
 countered by memory attributes for return type and argument 
 types.
Unfortunately, this is also why it will be rejected.
So, is that D's tenor that function signatures are thought to create *in*transparency and should continue to do so? Does the community think, allocation and memory transparency is a bad thing or just not needed? IMO, allocation and memory transparency is relevant to being a serious Systems programming language (even Systems Programming :-D ). Isn't the missing memory transparency from outside of functions the reason why global variables are frowned upon by many? Related to referential transparency (side effects), less transparency makes programs harder to debug, decouple and APIs harder to use right. (Just the single `map` issue with fixed-size arrays...)
 Okay, I didn't define aliasing. With "aliasing" I mean that 
 "aliasing references" (or pointers) either point to the exact 
 same address or that the immediately pointed class/struct 
 (pointed to by the reference/pointer) does not overlap. I 
 would consider anything else more complicated than necessary.
Insufficient for D with library container types and library smart pointers.
Yeah. It makes no sense if we consider the pointer layers between the exposed pointer and the actual data (I assume, smart pointers in D are implemented with such a middle layer in between). But if it only means the first payload data layer that represents the actual root node of any graph-like data structure, is it still flawed? At least, if I can annotate all pointer variables in my data structures and if checks are done for every single reference/pointer assignment with any access so that no pointer value range in the entire structure ever becomes violated, isn't it closer to memory safety than without? Of course, I could still pass references to those pointers to a binary which write into it without knowing any type information but that's a deliberate risk which static type checking cannot mitigate, only dynamic value checking of the pointed data after function return. (Probably another useful safety feature for my idea.) Of course attributes are optional, nobody has to annotate anything with the risk of obtaining falsely scoped pointer values. But would you agree, it would be better than not having it? Of course, it doesn't make everything safe, particularly if one can omit it but annotating variables with attributes could help with ownership (I think in a better design than Walter's proposal of yet another function attribute live instead of a variable attribute). With ownership I mean to prevent leakage of (sensible) data out of a function (not just reference values as with `scope`) and could provide some sanity checks and even provide more transparency for API use (because then I can see what kind of allocated memory I can expect for parameters and return value). I think, it could improve interfacing with C++ as well. At the end, I only want certainty about the references and pointers when I look into a function signature. I probably should (try to) implement it myself as a proof of concept. Regards, Elmar
Jun 03
next sibling parent reply sighoya <sighoya gmail.com> writes:
On Thursday, 3 June 2021 at 17:26:03 UTC, Elmar wrote:
```D
 stack arr2 = dynamicArr.dup;   // create a copy on stack, the 
stack is "scope"d
```
An easy but probably limited implementation would set 
theAllocator before the initialization of such an attributed 
value-type variable and resets theAllocator afterwards to the 
allocator from before.
What if dup is creating things on the heap (I don't know by the way). You need to make the allocator dynamically scoped.
```D
 register: only stored in a register (with compile-time error if 
not possible)
```
Only if you have complete control over the backend. Beside the combinatorial explosion in the required logic to check for, what happens if we copy/moving data between different memory annotated variables, e.g. nogc to gc, newcpp to gc. Did we auto copy, cast or throw an error. If we do not throw an error, an annotation might not only restrict access but also change semantics by introducing new references. So annotations become implied actions, that can be ok but is eventually hard to accept for the current uses of annotations.
Does the community think, allocation and memory transparency is 
a bad thing or just not needed? IMO, allocation and memory 
transparency is relevant to being a serious Systems programming 
language (even though C doesn't have it, C++ doesn't have it and 

There is no such thing as memory transparency, strictly speaking, even if you want to allocate things on the stack, what is if your backend doesn't have a stack at all? Or we just rename the heap to stack? In the end, we aren't that better what C or high level languages do, we have some heuristic though that our structures map better to the underlying hardware.
As a C programmer I'd say that C's pointer concept was never 
needed as it stands, it just was created to be an unsafe 
reference variable + a reference + an iterator 
all-in-one-solution as the simplest generic thing which beats it 
all (without knowing the use case by looking at the pointer 
type).
Well, I think having both is problematic/complex. But C has only one of those and C++ has both. It's not quite correct what arrays belong, so that's a mistake.
I think, any new language should remove the concept of pointers 
entirely rather than introducing new pointers.
Why not removing the distinction between values and references/pointers at all? But I think it drifts to hard in a logically high level language and isn't the right way to go in a system level language although very interesting. Annotations seam to be neat, but they parametrize your code: ```D allocator("X"), lifetime("param1", "greather", "param2") void f(Type1 param1, Type2 param2) ``` becomes ```D void f(Allocator X,Lifetime lifetime(param1), Lifetime lifetime(param2))(Type1 param1, Type2 param2) if currentAllocator=X && lifetime(param1)>=lifetime(param2) {...} ``` which literally turns every function allocating something into a template increasing "templatism" unless we get runtime generics as Swift. To summarize, I find these improvements interesting, but - doesn't feel system level - at all possible in a 20 years old language? Some Info: Rust distracts me in the point being a mix of high level with low level. They have values and references for their ownership/borrowing system but then also custom pointer types which doesn't interact well with the former.
Jun 03
parent reply Elmar <chrehme gmx.de> writes:
Thank you for your input.


On Thursday, 3 June 2021 at 19:36:32 UTC, sighoya wrote:
 Beside the combinatorial explosion in the required logic to 
 check for, what happens if we copy/moving data between 
 different memory annotated variables, e.g. nogc to gc, newcpp 
 to gc.
 Did we auto copy, cast or throw an error. If we do not throw an 
 error, an annotation might not only restrict access but also 
 change semantics by introducing new references.
 So annotations become implied actions, that can be ok but is 
 eventually hard to accept for the current uses of annotations.
There is no combinatorial explosion, that would be a bad idea ;-). Annotated references behave like a super class of non-annotated references or, say, a subset of attributes is a super class of a superset of attributes. The best effect description (in the dynamic case) would be viewing memory attributes like a precondition which requires the address value to be in certain interval(s). Currently attributes only have compile-time semantics, you said, so a static check would fit, right? On Thursday, 3 June 2021 at 19:36:32 UTC, sighoya wrote:
 There is no such thing as memory transparency, strictly 
 speaking, even if you want to allocate things on the stack, 
 what is if your backend doesn't have a stack at all? Or we just 
 rename the heap to stack?
Okay, "memory transparency" is a bad name. It could seem that it reveals actual memory addresses. I mean "allocation" or "scope" transparency. Concerning the call stack: Languages which don't provide the abstraction of scoped variables (which is implemented by a (call) stack) basically only have global variables + registers. Currently, I'm not conscious about a high-level language nor processor which wouldn't support that abstraction because it's the most basic abstraction of any high-level language. If you have "functions" then you also have a call stack or let's call it "automatic scope". It doesn't matter, whether automatic scope is allocated in heap area (which can happen with closures and continuations), static memory area (for non-recursive functions) or in its own area at the end of the memory layout, it only matters that it's automatically managed by the function. I also think that CPUs which don't support a call stack cannot be programmed with D at all. If attributes are used with static checks, it will not care about the actual memory address value, only the location in source code where a value was allocated or about the attributes which it gets from the user. The automatic lifetime is the criterion to distinguish it from heap, GC or static memory. For dynamic checks, I indeed made an assumption, that in real programs actual lifetime/scope can be inferred from memory addresses because allocation regions of related scope usually put variables in common memory areas (at least in common memory segments). This would result in pointer types to be value ranges instead of unconstrained 32-bit integers. Ultimately, information from a linker script could be needed for authentic dynamic checks (using relocated address for checking). I could imagine this to be difficult on top. Data from stack frames in the heap would be treated as dynamically allocated and data from static stack frames would be treated as stack. This could lead to unexpected results, false errors, unless more information is passed with the pointer. Dynamic checks would require a separate implementation (separate type) which memorizes in some bits which allocation scope a value was created. Eventually, the dynamic solution is less lightweight in memory but it makes the value check easier. On Thursday, 3 June 2021 at 19:36:32 UTC, sighoya wrote:
 Well, I think having both is problematic/complex. But C has 
 only one of those and C++ has both.
 It's not quite correct what arrays belong, so that's a mistake.
You mean references and pointers right? References (from C++) are immutable pointers (in theory). C++ has pointers for backwards compatibility (and probably because the designer originally didn't understand the problem) but are now discouraged from being used as "raw pointers" (when I wrote "pointer" I mean "raw pointer"). (Raw) Pointers instead are modifiable "reference variables" (like the variables in Java) which additionally provide access to the pointer address and allow modifying it. Reference variables however don't allow casting to non-pointer types. Arrays in C and C++ are actually more like C++ references, i.e. (locally) immutable pointers. On Thursday, 3 June 2021 at 19:36:32 UTC, sighoya wrote:
 Annotations seam to be neat, but they parametrize your code:

 ```D
  allocator("X"),  lifetime("param1", "greather", "param2") void 
 f(Type1 param1, Type2 param2)
 ```

 becomes

 ```D
 void f(Allocator X,Lifetime lifetime(param1), Lifetime 
 lifetime(param2))(Type1 param1, Type2 param2) if 
 currentAllocator=X && lifetime(param1)>=lifetime(param2) {...}
 ```

 which literally turns every function allocating something into 
 a template increasing "templatism" unless we get runtime 
 generics as Swift.
I agree that templatism is bad. Are attributes really lowered to template-arguments by the compiler? I also didn't mean to introduce new syntax with a comma between attributes. With memory attributes I really mean attributes like `scope`, `ref`, `private`, `pure`, ` nogc` ... which are used with reference/pointer types, not functions. You would be right that any assignment operation to an annotated reference needs a templated overload. I can't think of another way how to implement it. In the worst case, it would become something like ```D Ref!(nogc, Flower) tulip; // anything but not allocated by garbage collector Ref!(static, new, Bird) raven; // no automatic allocation ``` I would already be happy with the most important attributes. - "Oh, I see, it returns me GC-allocated memory" - "Oh, the passed argument is allocated automatically, so I can't put the address into a static reference." - "Oh, a slice over a fixed-size array will not work with that function." --- Of course, the amount of safety to get from these attributes depends on the programmer. For example they don't prevent Use-after-free with ` newc` and ` newcpp` in every case because it could be that a referenced value suddenly is deleted by code which interrupts the function execution. The true scope depends not only on the location of allocation but also on the location of associated deallocation. If the deallocation can happen in a code block, which interrupts normal function execution, than I would treat it either like `shared` or ` memory`. The compiler can't know all by itself. The memory safety thus will only work if the proper attributes are used by programmers. But the fact, that D already implements a very small weaker subset of memory (or reference) attributes like `scope` and `return ref` shows, that this idea does fit to D's design. --- PS: On Thursday, 3 June 2021 at 19:36:32 UTC, sighoya wrote:
 Why not removing the distinction between values and 
 references/pointers at all? But I think it drifts to hard in a 
 logically high level language and isn't the right way to go in 
 a system level language although very interesting.
I'm getting offtopic but I totally agree with you! I have a System Programming language idea which treats every variable as a reference variable to get rid of the annoying value categories and value concept by using a unified variable access interface which allows for using different reference implementations for different optimization scenarios (like using registers to store and modify the referenced value). On Thursday, 3 June 2021 at 19:36:32 UTC, sighoya wrote:
 On Thursday, 3 June 2021 at 17:26:03 UTC, Elmar wrote:
```D
 stack arr2 = dynamicArr.dup;   // create a copy on stack, the 
stack is "scope"d
```
 [...]
What if dup is creating things on the heap (I don't know by the way). You need to make the allocator dynamically scoped.
This was supposed to be a side idea unrelated to the main idea. `dup` does allocate memory with GC, so you'd be right when we talk about annotating references, but the snippet here is supposed to inject the annotated allocation of a *value* definition into the RHS, i.e. `dup`s internal implementation. If you like I elaborate more on that idea: Idea: when annotating value declarations it specifies where the return value or expression value of the RHS is allocated and thus, where the variable will be located in memory. This theoretical idea would give more control over the variable's allocation. This idea is not odd because a small subset of such attributes for value variables IS already implemented in D, like static/global variables, automatic local variables (of course) and member scope variables in structs and classes. C also features ` memory` (`volatile`) and to some extend ` register` (C99's `restrict` is only close to it (which keeps pointer dereferenced values in registers for further dereferencing) or with language extensions to map variables to specific registers). I thought, this would be not popular because it seems like D doesn't want to be too much an alternative for C++ Systems Programming and generalizing this concept seems like a bigger change. That's why this idea was only a side note. ```D gc short opal = 3; // eqv. to ref short opal = cast(ref short)GC.make!short(); opal = 5; newc int emerald = 5; // eqv. to ref int emerald = cast(ref int)malloc(int.sizeof); emerald = 5; new float ruby = 8.; // new uses the "new" operator, which is not always dynamic allocation rc int amethyst = 13; // reference counted, basically an abstraction over an underlying shared pointer ... free(&emerald); // needed because newc is not automatically managed ``` A benefit is that these variables still are used like values, i.e. they are passed by value or by reference depending on the function parameter type, although, physically, they are a reference of course (because everything is actually reference which is not stored in a register, variables on the call stack are referenced via the Stack Pointer for example). Goal: The responsibility of allocation is shifted from the service, the callee, (which doesn't know about any concrete client's allocation needs) to the client, the caller, (which knows about it's own allocation needs and actually should know what it gets). GC has been introduced to remove the symptoms of this problem (memory management problems) without solving it (consequence: it gets used way more often than needed and is inefficient). The only way, it would be solved reasonably, is letting the caller side (LHS of assignment) deside what it needs, not the callee side (RHS of assignment), because the caller side has to handle it afterwards. A generic solution would be to use some kind of Dependency Injector which handles the allocation, uses the callee to initialize the value and passes it to the caller. It would turn those attributes into a powerful abstraction. A very easy implementation of the Dependency Injector is overriding `theAllocator` while the RHS is computed.
Jun 05
parent sighoya <sighoya gmail.com> writes:
On Sunday, 6 June 2021 at 01:10:45 UTC, Elmar wrote:

 Goal: The responsibility of allocation is shifted from the 
 service, the callee, (which doesn't know about any concrete 
 client's allocation needs) to the client, the caller, (which 
 knows about it's own allocation needs and actually should know 
 what it gets).
That seems to very much resemble the idea odin strives for: https://odin-lang.org/docs/overview/#implicit-context-system It sounds very interesting and feels indeed system programming like, but I see some issues with that: - the `new` call may not be the same for unique ptr, rc and gc as gc needs more context - rc/arc may need to insert incs and decs to the begin and end of the scope respectively, but this would that means to templatize the callee or to box over a runtime option, both having draws - overwriting allocators is an interesting concept, but how useful is this given that the code can't be re-adapted meaning not only the allocator make execution performant but also the code around it and both are intertwined to each other - you have a small performance hit because of passing function pointer to the callee or by exchanging them when global vars are used for, it's akin to nondetermistic vs deterministic exception handling - even more, how many custom allocators are passed to the callee because the callee as caller has many callees inside them which use allocation and all the custom allocators are propagating up. I recognize that some minority is interested in this, even me to some kind. I'm still skeptical about the gain. However, what is if you try to extend the compiler to see if it is possible, maybe you find other people to realize your idea. Recalling myself, I know that russhy and IGotD are similar interested in custom allocators.
Jun 06
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 3 June 2021 at 17:26:03 UTC, Elmar wrote:
 On Tuesday, 1 June 2021 at 06:12:05 UTC, Ola Fosheim Grostad 
 wrote:
 The separate compilation is a good point. Binary compatibility 
 is a common property considered for security safeguards. But at 
 least static checking with attributes would need no memory 
 addresses at all (also if the compiler can infer the attribute 
 for every value-typed variable automatically from where it is 
 defined).
I don't think separate compilation is a good point. I think a modern language should mix object-file and IR-file linking with caching for speed improvements. Nevertheless, it is being used as an argument for not making D a better language. So, that is what you are up against. D isn't really "modern". It is very much in the C-mold, like C++. It has taken on too many of C++'s flaws. For instance it kept underperforming exceptions instead of making them fast.
 passes addresses obtained from variables. As a C programmer I'd 
 say that C's pointer concept was never needed as it stands, it 
 just was created to be an unsafe reference variable + a 
 reference + an iterator all-in-one-solution as the simplest 
 generic thing which beats it all (without knowing the use case 
 by looking at the pointer type).
C is mostly an abstraction over common machine language instructions. Making a non-optimizing C backend perform reasonably well for handcrafted C-code. C pointers do have a counterpart in C++ STL iterators though. So, one could argue that C-pointers are memory-iterators.
 Right, if nobody tried it so far I'd like myself. Then I can 
 firm my D experience with further practice. I'd compare the 
 nature of static and dynamic attribute checks to the nature of 
 C++ `static_cast` and `dynamic_cast` of class pointers. I was 
 thinking, such a user library could use `__traits` with 
 templated operator overloads.
Sounds like a fun project. (D, as the languages stands, encourages the equivalent of reinterpret_cast, so there is that.)
 So, is that D's tenor that function signatures are thought to 
 create *in*transparency and should continue to do so? Does the 
 community think, allocation and memory transparency is a bad 
 thing or just not needed? IMO, allocation and memory 
 transparency is relevant to being a serious Systems programming 
 language
Let us not confuse community with creators. :). Also, let us not assume that there is a homogeneous community. So, you have the scripty-camp who are not bothered by the current GC and don't really deal with memory allocations much. Then there is the other camp. As one of those in the other camp, I think that the compiler should do the memory management and be free to optimize. So I am not fond of things like "scope". I think they are crutches. I think the language is becoming arcane by patching it up here and there instead of providing a generic solution.
 I probably should (try to) implement it myself as a proof of 
 concept.
The best option is to just introduce a custom pointer-library, like in C++, that tracks what you want it to track. Don't bother with separate compilation issues. Just template all functions. I think LDC will remove duplicates if the bodies of two functions turn into the same machine code? Then you get a feeling for what it would be like.
Jun 04
parent reply Elmar <chrehme gmx.de> writes:
On Friday, 4 June 2021 at 08:29:47 UTC, Ola Fosheim Grøstad wrote:
 I don't think separate compilation is a good point. I think a 
 modern language should mix object-file and IR-file linking with 
 caching for speed improvements.
I know what you mean. Avoiding separate compilation where possible allows for more optimization potential but in general you can't avoid it, particularly when source code is not available. D tries to be linkable with C and C++ which don't know D. That's why working with separate binaries needs to be considered (and compatibility with C/C++ really is important because both languages have large code bases). So with "good point" I meant "an important aspect" in research, not that it's nice to need it :-) .
 C pointers do have a counterpart in C++ STL iterators though. 
 So, one could argue that C-pointers are memory-iterators.
Exactly.
 So, you have the scripty-camp who are not bothered by the 
 current GC and don't really deal with memory allocations much. 
 Then there is the other camp.

 As one of those in the other camp, I think that the compiler 
 should do the memory management and be free to optimize. So I 
 am not fond of things like "scope". I think they are crutches. 
 I think the language is becoming arcane by patching it up here 
 and there instead of providing a generic solution.
`scope` exists for a good reason because guaranteeing compiler optimization is very hard (apparently). The `scope` attribute has two benefits: 1st it makes RAII explicit (which helps to prevent bugs by falsely using it), 2nd it makes the compiler faster and easier because it doesn't need to understand the code in order to search for optimization places. Performing optimization is easier than finding the places that can be optimized and then assuring the correctness of optimization. Finding all optimizations automatically (the ideal case) probably would explode compile-time so therefore attributes are given to programmers for hinting them, it solves the compile-time issue and reduces compiler complexity a lot. The problem with genericity is, it's always a tradeoff with efficiency. It makes coding more complicated and the solution more heavy (because it takes care about more cases), at some point too complicated to be practical. There are cases were the compiler cannot know in any way the correctness of an optimization because it depends on the programmer's intend (or would change behaviour). Then, explicit optimization hints are required.
 Don't bother with separate compilation issues. Just template 
 all functions. I think LDC will remove duplicates if the bodies 
 of two functions turn into the same machine code?
I'm almost certain that duplicate functions will not be removed in all cases, because that's a very difficult problem to solve. It just reminds me of the undecidability of the Liskov Substitution principle according to Wikipedia. This principle requires that contracts/halting still hold with subtype arguments passed to a function. Have a nice day.
Jun 05
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 5 June 2021 at 17:55:26 UTC, Elmar wrote:
 `scope` exists for a good reason because guaranteeing compiler 
 optimization is very hard (apparently). The `scope` attribute 
 has two benefits: 1st it makes RAII explicit (which helps to 
 prevent bugs by falsely using it), 2nd it makes the compiler 
 faster and easier because it doesn't need to understand the 
 code in order to search for optimization places.
But introducing all these special cases just to avoid explicit lifetimes like Rust is making the language more complicated, not less. The intention is to make it less complicated, but that will not be the end result, I think. I don't think one can evolve a solution. It has to be designed as a whole, not one piece at a time like D is doing now.
Jun 05