www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Reimplementing the bulk of std.meta iteratively

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
I'll start with the punchline: at the link below you'll find an 
alpha-quality reimplementation of std.meta in an iterative manner:

https://gist.github.com/andralex/6212ebf2b38c59c96cf66d0009336318

It's the bulk of std.meta - I stopped when it became obvious doing more 
is just more of the same. Compiles and runs with dmd 2.092 and probably 
others.

The implementation uses throughout a reification/dereification approach. 
To reify a type or alias means to convert it into a value. To dereify a 
value means to turn it back into the type or alias it originated from.

The algorithms follow the following pattern consistently:

1. Reify the compile-time parameters into values
2. Carry processing on these values with the usual algorithms
3. If needed, dereify back the results into compile-time parameters

Not all artifacts need step 3. For example, allSatisfy returns a bool, 
not a type.

For the most part implementation has been a healthy gallop that took a 
couple of hours yesterday and a couple of hours today. Its weakest point 
is it uses .stringof for types, which has issues with local types. 
Hopefully https://github.com/dlang/dmd/pull/11797 could help with that.

Most implementations are a few lines long because they get to leverage 
algorithms as implemented in std. Here's a typical one:

alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
}());

To sort an AliasSeq with most derived elements first, create a lambda to 
reify the arguments into values, sort the values, and return the sorted 
result. Then call the lambda and use dereify against its result. And 
that is IT.

Commments and ideas for improvements are welcome.
Sep 26 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Sep 26, 2020 at 12:18:27PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]
 The implementation uses throughout a reification/dereification
 approach. To reify a type or alias means to convert it into a value.
 To dereify a value means to turn it back into the type or alias it
 originated from.
[...]
 Its weakest point is it uses .stringof for types, which has issues
 with local types. Hopefully https://github.com/dlang/dmd/pull/11797
 could help with that.
[...] our current state. It closes the reification/dereification loop, and allows CTFE manipulation of types as strings, which sidesteps the issues of how to manipulate types in imperative code without running into all sorts of problems like the ones that Stefan brought up. In the future, I can imagine CTFE *generation* of these strings, which opens up brand new ways of manipulating types. There could be a standard set of mangle-string manipulation utilities in druntime or phobos, say, that let you perform operations like adding/removing qualifiers, manipulating type lists (in string form!), so you could implement things like Unqual as a CTFE string manipulation function instead of a bunch of templates. Of course, Unqual is simple enough that there's no reason to do this, but the point is, we can do much more complicated manipulations of types without needing to resort to recursive templates, and it leverages existing machinery instead of inventing a slew of new ones. T -- LINUX = Lousy Interface for Nefarious Unix Xenophobes.
Sep 26 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Saturday, 26 September 2020 at 16:40:47 UTC, H. S. Teoh wrote:
 On Sat, Sep 26, 2020 at 12:18:27PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]
 The implementation uses throughout a reification/dereification 
 approach. To reify a type or alias means to convert it into a 
 value. To dereify a value means to turn it back into the type 
 or alias it originated from.
[...]
 Its weakest point is it uses .stringof for types, which has 
 issues with local types. Hopefully 
 https://github.com/dlang/dmd/pull/11797 could help with that.
[...] forward from our current state. It closes the reification/dereification loop, and allows CTFE manipulation of types as strings, which sidesteps the issues of how to manipulate types in imperative code without running into all sorts of problems like the ones that Stefan brought up.
With all the constraints and problems. Iff you can use a string mixin you can use __stringToType. If you can't use a string mixin in the context you can't use stringToType.
Sep 26 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/26/20 1:32 PM, Stefan Koch wrote:

 With all the constraints and problems.
 Iff you can use a string mixin you can use __stringToType.
 If you can't use a string mixin in the context you can't use stringToType.
mixin("+1000"); -Steve
Sep 26 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:
 1. Reify the compile-time parameters into values
 2. Carry processing on these values with the usual algorithms
 3. If needed, dereify back the results into compile-time 
 parameters
That's the right thought, and that's what type functions were designed to do. How do they compare to the vision you have?
 Commments and ideas for improvements are welcome.
The PR referenced does not give, actually give you the ability to dereify types within CTFE. The PR forces a template and template has to have template parameters to take the mangles. I would think that is against the actual goal.
Sep 26 2020
parent Paul Backus <snarwin gmail.com> writes:
On Saturday, 26 September 2020 at 17:30:44 UTC, Stefan Koch wrote:
 The PR referenced does not give, actually give you the ability 
 to dereify types within CTFE.
 The PR forces a template and template has to have template 
 parameters to take the mangles.
 I would think that is against the actual goal.
It's against the goal if your goal is for meta-functions to be 100% monomorphic. But it does accomplish the goal of turning template recursion (O(n) instantiations) into flat iteration (O(1) instantiations). Personally, I don't think 100% monomorphism is necessary or particularly desirable, so I'm ok with a design that eschews it.
Sep 26 2020
prev sibling next sibling parent Paul Backus <snarwin gmail.com> writes:
On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:
 For the most part implementation has been a healthy gallop that 
 took a couple of hours yesterday and a couple of hours today. 
 Its weakest point is it uses .stringof for types, which has 
 issues with local types. Hopefully 
 https://github.com/dlang/dmd/pull/11797 could help with that.
It's amazing to me how obvious this idea seems now that I've seen it. D's had type -> string serialization for ages; of course there should also be a way to do the reverse! And yet somehow, it never even crossed my mind until I read Walter's PR. I think this is a real winner.
Sep 26 2020
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 26.09.20 18:18, Andrei Alexandrescu wrote:
 
 Most implementations are a few lines long because they get to leverage 
 algorithms as implemented in std. Here's a typical one:
 
 alias MostDerived(Args...) = dereify!({
      auto ids = reify!Args;
      sort!((a, b) => is(dereify!a : dereify!b))(ids);
      return ids;
 }());
I am pretty sure this one does not work though.
Sep 26 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 5:16 PM, Timon Gehr wrote:
 On 26.09.20 18:18, Andrei Alexandrescu wrote:
 Most implementations are a few lines long because they get to leverage 
 algorithms as implemented in std. Here's a typical one:

 alias MostDerived(Args...) = dereify!({
      auto ids = reify!Args;
      sort!((a, b) => is(dereify!a : dereify!b))(ids);
      return ids;
 }());
I am pretty sure this one does not work though.
Oops, Steve just told me it doesn't have any unittests. So it may as well not be working. Research goes on.
Sep 26 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/26/20 5:42 PM, Andrei Alexandrescu wrote:
 On 9/26/20 5:16 PM, Timon Gehr wrote:
 On 26.09.20 18:18, Andrei Alexandrescu wrote:
 Most implementations are a few lines long because they get to 
 leverage algorithms as implemented in std. Here's a typical one:

 alias MostDerived(Args...) = dereify!({
      auto ids = reify!Args;
      sort!((a, b) => is(dereify!a : dereify!b))(ids);
      return ids;
 }());
I am pretty sure this one does not work though.
Oops, Steve just told me it doesn't have any unittests. So it may as well not be working. Research goes on.
So just to follow up on this, the reason it doesn't work is because the relation between a and b must determined BEFORE the call to sort. In other words, in order for this to work, you have to establish result of the lambda for all a and b in the array. Or optionally, you can save enough information into the "Id" type that can be able to enable the comparison at runtime. The general concept works, but only if you can create a relation function that doesn't rely on the types themselves, but on the runtime representation of the types. In other words, such a function would need 3 steps: 1. transform the types into runtime data that is comparable given the types. 2. Run the runtime function that does the transformation 3. Convert the result back to the types. Such things are not *impossible*, but something like derivative is a great example that is really difficult to generate a runtime function that can do it. -Steve
Sep 26 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 6:00 PM, Steven Schveighoffer wrote:
 On 9/26/20 5:42 PM, Andrei Alexandrescu wrote:
 On 9/26/20 5:16 PM, Timon Gehr wrote:
 On 26.09.20 18:18, Andrei Alexandrescu wrote:
Such things are not *impossible*, but something like derivative is a great example that is really difficult to generate a runtime function that can do it.
Yah, you'd need i.e. a member function in Id called bool implicitlyConvertible(Id another). The functionality needed is similar to that in Variant, which created the need for implicitConversionTargets (https://dlang.org/library/std/traits/implicit_conversion_targets.html). You'd need to have access to those in the Id space.
Sep 26 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/26/20 6:03 PM, Andrei Alexandrescu wrote:
 On 9/26/20 6:00 PM, Steven Schveighoffer wrote:
 On 9/26/20 5:42 PM, Andrei Alexandrescu wrote:
 On 9/26/20 5:16 PM, Timon Gehr wrote:
 On 26.09.20 18:18, Andrei Alexandrescu wrote:
Such things are not *impossible*, but something like derivative is a great example that is really difficult to generate a runtime function that can do it.
Yah, you'd need i.e. a member function in Id called bool implicitlyConvertible(Id another). The functionality needed is similar to that in Variant, which created the need for implicitConversionTargets (https://dlang.org/library/std/traits/implicit_conversion_targets.html). You'd need to have access to those in the Id space.
Variant works because it can access the parameter type. If you need to access only one type, and the other information is runtime-accessible, then you can do it. If both need to be runtime-accessible, you are in trouble. Like try comparing 2 Variants. If Variant doesn't have any idea about how the types might compare ahead of time, it gives up. -Steve
Sep 26 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 6:15 PM, Steven Schveighoffer wrote:
 On 9/26/20 6:03 PM, Andrei Alexandrescu wrote:
 On 9/26/20 6:00 PM, Steven Schveighoffer wrote:
 On 9/26/20 5:42 PM, Andrei Alexandrescu wrote:
 On 9/26/20 5:16 PM, Timon Gehr wrote:
 On 26.09.20 18:18, Andrei Alexandrescu wrote:
Such things are not *impossible*, but something like derivative is a great example that is really difficult to generate a runtime function that can do it.
Yah, you'd need i.e. a member function in Id called bool implicitlyConvertible(Id another). The functionality needed is similar to that in Variant, which created the need for implicitConversionTargets (https://dlang.org/library/std/traits/implicit_conversion_targets.html). You'd need to have access to those in the Id space.
Variant works because it can access the parameter type. If you need to access only one type, and the other information is runtime-accessible, then you can do it. If both need to be runtime-accessible, you are in trouble.
A possibly angle here is to save that information in runtime format, i.e. reify the inheritance/convertibility relationship. I'm unclear on how to do that efficiently. (The inefficient solution would save the result of ImplicitConversionTargets in a member of Id that has type Id[] and then do searches in it.) In an ideal world it would be nice if id1 <= id2 iff is(reify!id1 : reify!id2).
Sep 26 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/26/20 6:48 PM, Andrei Alexandrescu wrote:
 A possibly angle here is to save that information in runtime format, 
 i.e. reify the inheritance/convertibility relationship.
thinking it through... All basic types have well-defined relationships that can be figured out using a big statically generated switch thing. Enums can be converted to their bases, and whatever they can convert to. Structs can be converted to their alias-this parameter. Classes can also have an alias this, AND a set of base types. Arrays and pointers can convert in certain cases to base types. And, multiply all this by mutability changes (convert to const for things with references, and convert to anything for value types). So for each type, you could generate a list of types that it can convert to. Then on each comparison, search for the target type in the list, and if you find it, return true. I can't imagine this is going to be efficient. We might be able to factor out the const changes. But it still leaves a big list of things. Honestly, I don't know if the compiler does it in a more efficient way than searching through a list. So it's definitely doable, I just don't know if we would want to do it, or if the complication of the resulting code is worth it. is(a : b) is just soooo much easier! And I think this is one aspect that type functions allow. If there were a way to go back to type-land in the middle of runtime-at-ctfe-land, even if you had to define it in a restrictive way, then it might be much more palatable. -Steve
Sep 26 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 10:30 PM, Steven Schveighoffer wrote:
 On 9/26/20 6:48 PM, Andrei Alexandrescu wrote:
 A possibly angle here is to save that information in runtime format, 
 i.e. reify the inheritance/convertibility relationship.
thinking it through... All basic types have well-defined relationships that can be figured out using a big statically generated switch thing. Enums can be converted to their bases, and whatever they can convert to.
*nod*
 Structs can be converted to their alias-this parameter.
*nod* but how is that detectable for a given type T?
 Classes can also have an alias this, AND a set of base types.
yes... https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213 has the base classes part as a proof of concept (no interfaces and no alias this).
 Arrays and pointers can convert in certain cases to base types.
*nod* these would be detectable with relative eash
 And, multiply all this by mutability changes (convert to const for 
 things with references, and convert to anything for value types).
 
 So for each type, you could generate a list of types that it can convert 
 to. Then on each comparison, search for the target type in the list, and 
 if you find it, return true.
 
 I can't imagine this is going to be efficient.
In the words of a guy I heard at a panel: "I'm not sure I understood what the problem is, but it seems the solution is a hashtable". :o)
 We might be able to 
 factor out the const changes. But it still leaves a big list of things. 
 Honestly, I don't know if the compiler does it in a more efficient way 
 than searching through a list.
 
 So it's definitely doable, I just don't know if we would want to do it, 
 or if the complication of the resulting code is worth it. is(a : b) is 
 just soooo much easier! And I think this is one aspect that type 
 functions allow.
It's part of the introspection game: gather the information typically present in the innards of the compiler, in programmatically-available form. I mean you would expect to grab access to implicit conversion targets in a language that has introspection.
 If there were a way to go back to type-land in the middle of 
 runtime-at-ctfe-land, even if you had to define it in a restrictive way, 
 then it might be much more palatable.
I'd be very interested too! (My experience with reification is now measured in hours.) This is the kind of stuff Timon would know.
Sep 26 2020
prev sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Saturday, 26 September 2020 at 21:42:19 UTC, Andrei 
Alexandrescu wrote:
 On 9/26/20 5:16 PM, Timon Gehr wrote:
 On 26.09.20 18:18, Andrei Alexandrescu wrote:
 Most implementations are a few lines long because they get to 
 leverage algorithms as implemented in std. Here's a typical 
 one:

 alias MostDerived(Args...) = dereify!({
      auto ids = reify!Args;
      sort!((a, b) => is(dereify!a : dereify!b))(ids);
      return ids;
 }());
I am pretty sure this one does not work though.
Oops, Steve just told me it doesn't have any unittests. So it may as well not be working. Research goes on.
This example on the other hand works: --- /+ import std.array; import std.range; import std.algorithm; doesn't work yet but only because the template connstraints need to be adjusted. +/ alias alias_array = typeof(((alias T) => __traits(getAttributes, T))(Byte)); auto sort3(alias less, R)(ref /*alias_array*/ R r) { alias tmp; if (r.length == 1) return r; if (r.length == 2) { if (less(r[1], r[0])) { tmp = r[0]; r[0] = r[1]; r[1] = tmp; } } if (r.length == 3) { if (less(r[2], r[1])) { tmp = r[1]; r[1] = r[2]; r[2] = tmp; } if (less(r[1], r[0])) { tmp = r[0]; r[0] = r[1]; r[1] = tmp; } } return r; } auto sortBySize(alias_array types ...) { types.sort3!((alias a, alias b) => a.sizeof < b.sizeof); return types; }; alias Int = int; alias Ushort = ushort; // we need these aliases because the parser won't allow us to call a function with a reserved type identifier. alias Byte = byte; pragma(msg, sortBySize(Ushort, Int, Byte)); //Output: [(byte), (ushort), (int)] ---
Sep 26 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 27 September 2020 at 00:10:33 UTC, Stefan Koch wrote:
 This example on the other hand works:
Here's another potential implementation: --- template sortBySize(T...) { string code() { // step one: gather the data in the template shim static struct Item { size_t idx; size_t size; } Item[] items; foreach(idx, t; T) items ~= Item(idx, t.sizeof); // step two: process the data with normal CTFE import std.algorithm; items.sort!((a, b) => a.size < b.size); // step three: build the code for the template to bring it back // this would be a fine candidate for a helper function string code; foreach(item; items) { if(code.length) code ~= ", "; code ~= "T[" ~ cast(char)(item.idx + '0') ~ "]"; } return "AliasSeq!(" ~ code ~ ")"; } import std.meta; alias sortBySize = mixin(code()); } pragma(msg, sortBySize!(ushort, int, byte)); --- (byte, ushort, int) There's several optimizations here, preallocating, factoring out of the template, not using std.algorithm, you know those tricks already. I'm not in love with it but there is some potential in the concept in today's D.
Sep 26 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 8:27 PM, Adam D. Ruppe wrote:
 On Sunday, 27 September 2020 at 00:10:33 UTC, Stefan Koch wrote:
 This example on the other hand works:
Here's another potential implementation: --- template sortBySize(T...) {
[snip] Sorting by inheritance relationship is more difficult. I got this going, alpha quality: https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213 Now indeed calling std.sort against (reified) types just works. The point is to reify the bases of the type as well so they can be manipulated as objects. Then defining the comparison relation is easy (didn't bother to make it efficient here - just a linear search). More introspection-related matters would be reified the same way (store during construction in the reified object, potentially via pointers if they don't apply to all objects). Instance size would be an obvious candidate, prolly I'll put it in there as a demo. More interestingly we'd have things like parameters/return for functions and data members for classes and structs. Long way ahead. This is really exciting stuff.
Sep 26 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei Alexandrescu 
wrote:
 This is really exciting stuff.
Which I came up with.
Sep 26 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 10:48 PM, Stefan Koch wrote:
 On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei Alexandrescu wrote:
 This is really exciting stuff.
Which I came up with.
Congratulations!
Sep 26 2020
prev sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei Alexandrescu 
wrote:
 On 9/26/20 8:27 PM, Adam D. Ruppe wrote:
 On Sunday, 27 September 2020 at 00:10:33 UTC, Stefan Koch 
 wrote:
 This example on the other hand works:
Here's another potential implementation: --- template sortBySize(T...) {
[snip] Sorting by inheritance relationship is more difficult. I got this going, alpha quality: https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213 Now indeed calling std.sort against (reified) types just works. The point is to reify the bases of the type as well so they can be manipulated as objects. Then defining the comparison relation is easy (didn't bother to make it efficient here - just a linear search). More introspection-related matters would be reified the same way (store during construction in the reified object, potentially via pointers if they don't apply to all objects). Instance size would be an obvious candidate, prolly I'll put it in there as a demo. More interestingly we'd have things like parameters/return for functions and data members for classes and structs. Long way ahead. This is really exciting stuff.
We're in an early stage of exploration of discovered capability, so I understand the excitement. The C++ guys must have been way excited about their meta programming discoveries as well. Just a respectful reminder here: a discovered way to do something is not a designed way to do something. Cleverness can take you a very very long way (Modern C++ anyone? :-) ) but it's unlikely that it will take you to appropriate simplicity. Whatever decisions are made down the road, I'm happy that we're actively looking at extending D's safely accessible meta programming capabilites now. Sounds like a great topic for beerconf-day-2.
Sep 26 2020
parent reply claptrap <clap trap.com> writes:
On Sunday, 27 September 2020 at 03:46:52 UTC, Bruce Carneal wrote:
 On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei 
 Alexandrescu wrote:

 We're in an early stage of exploration of discovered 
 capability, so I understand the excitement.  The C++ guys must 
 have been way excited about their meta programming discoveries 
 as well.

 Just a respectful reminder here: a discovered way to do 
 something is not a designed way to do something.  Cleverness 
 can take you a very very long way (Modern C++ anyone? :-) ) but 
 it's unlikely that it will take you to appropriate simplicity.
D: "Look ma I can peel an apple with a tin opener!" Mom(C++): "That's my boy... the band-aids are in the usual place."
Sep 27 2020
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 27 September 2020 at 16:32:18 UTC, claptrap wrote:
 On Sunday, 27 September 2020 at 03:46:52 UTC, Bruce Carneal 
 wrote:
 On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei 
 Alexandrescu wrote:

 We're in an early stage of exploration of discovered 
 capability, so I understand the excitement.  The C++ guys must 
 have been way excited about their meta programming discoveries 
 as well.

 Just a respectful reminder here: a discovered way to do 
 something is not a designed way to do something.  Cleverness 
 can take you a very very long way (Modern C++ anyone? :-) ) 
 but it's unlikely that it will take you to appropriate 
 simplicity.
D: "Look ma I can peel an apple with a tin opener!" Mom(C++): "That's my boy... the band-aids are in the usual place."
Andrei and others advanced us well beyond the mandated cleverness of C++ meta programming. As Atila has put it, they helped make things "boring" through very good design. I deeply admire that work. I hope and believe that there is another such advance available to us in the form of type functions. The recently discovered alternative, reify/dereify, appears to be equivalent in power but is, comparatively, baroque. If the two approaches are actually equivalent in power, and in their ability to address the current template issues, then the baroque should be preferred if the language is closed to syntactic additions. If the language is not closed, then choosing to displace the already prototyped type function capability with reify/dereify would represent a lost opportunity to administer another complexity smackdown. Disclaimer: Andrei has stated, effectively, that I have little standing to opine on these issues. It is true that while I use templates extensively I confine myself to very simple combinations thereof. I am not experiencing the template pain reported by others. Best wishes to those who continue to work in this area. Here's hoping you lead us to a future that is extremely boring.
Sep 27 2020
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/20 10:58 PM, Bruce Carneal wrote:
 Disclaimer: Andrei has stated, effectively, that I have little standing 
 to opine on these issues.
Sorry! Didn't mean to sound dismissive at all. Meant to just say, don't shun templates, embrace them and thrive.
Sep 27 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Monday, 28 September 2020 at 03:03:01 UTC, Andrei Alexandrescu 
wrote:
 On 9/27/20 10:58 PM, Bruce Carneal wrote:
 Disclaimer: Andrei has stated, effectively, that I have little 
 standing to opine on these issues.
Sorry! Didn't mean to sound dismissive at all. Meant to just say, don't shun templates, embrace them and thrive.
In practice template can quite literally explode though. Which could be a valid reason to shun them. You can embrace them, of course, and thrive. The question is for how long you will thrive.
Sep 28 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 9/28/20 8:46 AM, Stefan Koch wrote:
 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei Alexandrescu wrote:
 On 9/27/20 10:58 PM, Bruce Carneal wrote:
 Disclaimer: Andrei has stated, effectively, that I have little 
 standing to opine on these issues.
Sorry! Didn't mean to sound dismissive at all. Meant to just say, don't shun templates, embrace them and thrive.
In practice template can quite literally explode though. Which could be a valid reason to shun them. You can embrace them, of course, and thrive. The question is for how long you will thrive.
For a very long time judging by the success C++ is enjoying with them.
Sep 28 2020
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Monday, 28 September 2020 at 14:16:20 UTC, Andrei Alexandrescu 
wrote:
 On 9/28/20 8:46 AM, Stefan Koch wrote:
 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei 
 Alexandrescu wrote:
 On 9/27/20 10:58 PM, Bruce Carneal wrote:
 Disclaimer: Andrei has stated, effectively, that I have 
 little standing to opine on these issues.
Sorry! Didn't mean to sound dismissive at all. Meant to just say, don't shun templates, embrace them and thrive.
In practice template can quite literally explode though. Which could be a valid reason to shun them. You can embrace them, of course, and thrive. The question is for how long you will thrive.
For a very long time judging by the success C++ is enjoying with them.
As you note, by employing a variety of "best practices", of extra-language conventions, of one-level-of-indirection wrappers, of "human must certify this correct" mechanisms, the C++ community has indeed "thrived". We've not settled for that meta programming drudgery, that friction, in the past. You know better this better than anyone else on the planet. I hope we don't "settle" going forward.
Sep 28 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 9/28/20 11:55 AM, Bruce Carneal wrote:
 On Monday, 28 September 2020 at 14:16:20 UTC, Andrei Alexandrescu wrote:
 On 9/28/20 8:46 AM, Stefan Koch wrote:
 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei Alexandrescu wrote:
 On 9/27/20 10:58 PM, Bruce Carneal wrote:
 Disclaimer: Andrei has stated, effectively, that I have little 
 standing to opine on these issues.
Sorry! Didn't mean to sound dismissive at all. Meant to just say, don't shun templates, embrace them and thrive.
In practice template can quite literally explode though. Which could be a valid reason to shun them. You can embrace them, of course, and thrive. The question is for how long you will thrive.
For a very long time judging by the success C++ is enjoying with them.
As you note, by employing a variety of "best practices", of extra-language conventions, of one-level-of-indirection wrappers, of "human must certify this correct" mechanisms, the C++ community has indeed "thrived". We've not settled for that meta programming drudgery, that friction, in the past.  You know better this better than anyone else on the planet. I hope we don't "settle" going forward.
(Not getting some of the uses of quotation marks.) That's a bit backhanded because it implies I promote settling for meta programming drudgery. Did you mean to say that? On the contrary, I find type reification interesting exactly because it takes you from said drudgery to familiar land - first-class values that can be manipulated in traditional manner. Should you need to get back to type-land, dereification helps with that. We can now draw from a large body of existing theory and practice. https://en.wikipedia.org/wiki/Reification_(computer_science)
Sep 28 2020
next sibling parent reply claptrap <clap trap.com> writes:
On Monday, 28 September 2020 at 16:57:19 UTC, Andrei Alexandrescu 
wrote:
 On 9/28/20 11:55 AM, Bruce Carneal wrote:
 On Monday, 28 September 2020 at 14:16:20 UTC, Andrei 
 Alexandrescu wrote:
 On 9/28/20 8:46 AM, Stefan Koch wrote:
 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei 
 Alexandrescu wrote:
 On 9/27/20 10:58 PM, Bruce Carneal wrote:
 Disclaimer: Andrei has stated, effectively, that I have 
 little standing to opine on these issues.
Sorry! Didn't mean to sound dismissive at all. Meant to just say, don't shun templates, embrace them and thrive.
In practice template can quite literally explode though. Which could be a valid reason to shun them. You can embrace them, of course, and thrive. The question is for how long you will thrive.
For a very long time judging by the success C++ is enjoying with them.
As you note, by employing a variety of "best practices", of extra-language conventions, of one-level-of-indirection wrappers, of "human must certify this correct" mechanisms, the C++ community has indeed "thrived". We've not settled for that meta programming drudgery, that friction, in the past.  You know better this better than anyone else on the planet. I hope we don't "settle" going forward.
(Not getting some of the uses of quotation marks.) That's a bit backhanded because it implies I promote settling for meta programming drudgery. Did you mean to say that? On the contrary, I find type reification interesting exactly because it takes you from said drudgery to familiar land - first-class values that can be manipulated in traditional manner. Should you need to get back to type-land, dereification helps with that. We can now draw from a large body of existing theory and practice. https://en.wikipedia.org/wiki/Reification_(computer_science)
Who's actually heard of reification outside of compiler writers? Ive been reading a ton of compiler related stuff lately, but not heard of it before now. It's seems like an implementation detail leaking out to me. Instead of first class types lets have whatever this stuff is cause we can get it on the cheap and its intellectually cool. I guess my point is if you want meta-programming that just works as if it was regular programming, the reify / dereify just emphasises that it's two different domains because you have to move from one to the other. It's not unification, it's a bridge yes, but lets be honest the user shouldn't have to think about, it should just work. (Even if it makes the compiler writers job harder)
Sep 28 2020
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 28.09.20 23:08, claptrap wrote:
 Instead of first class types
(Stefan's type functions do not give you first-class types. A first-class type you could put in a runtime variable.)
Sep 28 2020
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Sep 28, 2020 at 11:27:56PM +0200, Timon Gehr via Digitalmars-d wrote:
 On 28.09.20 23:08, claptrap wrote:
 Instead of first class types
(Stefan's type functions do not give you first-class types. A first-class type you could put in a runtime variable.)
(typeid(T) is a first-class type (of sorts), which we already have today. All we really need today is a way to retrieve T from typeid(T) at compile-time.) T -- By understanding a machine-oriented language, the programmer will tend to use a much more efficient method; it is much closer to reality. -- D. Knuth
Sep 28 2020
prev sibling next sibling parent claptrap <clap trap.com> writes:
On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:
 On 28.09.20 23:08, claptrap wrote:
 Instead of first class types
(Stefan's type functions do not give you first-class types. A first-class type you could put in a runtime variable.)
First class in the context of compile time meta programming i guess is more apt. I mean what people want is to manipulate these things at compile time so you want them to be first class in that context. Could just be called reified types to keep Andrei happy ;)
Sep 28 2020
prev sibling next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:
 On 28.09.20 23:08, claptrap wrote:
 Instead of first class types
(Stefan's type functions do not give you first-class types. A first-class type you could put in a runtime variable.)
As I understand it, they allow types to be put in to compile time variables, but not run time variables. It's more than we had before type functions, but less than run-time first class. What name do the type theory folk give to such a capability?
Sep 28 2020
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:
 On 28.09.20 23:08, claptrap wrote:
 Instead of first class types
(Stefan's type functions do not give you first-class types. A first-class type you could put in a runtime variable.)
That's true. At least as it stands right now. I _could_ give the alias a physical form at runtime. (By doing the same thing that the reify template does (or rather will eventually do), that is exporting _all_ the properties of the type into an object) However I am not actually sure what the use of this would be. All usage that I have seen, is in generating code, which is kindof useless if it does not happen at compile time. Perhaps there are others?
Sep 28 2020
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 29/09/2020 12:37 PM, Stefan Koch wrote:
 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or rather will 
 eventually do), that is exporting _all_ the properties of the type into 
 an object)
 
 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is kindof 
 useless if it does not happen at compile time.
 
 Perhaps there are others?
Serialization!
Sep 28 2020
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 29 September 2020 at 00:03:14 UTC, rikki cattermole 
wrote:
 On 29/09/2020 12:37 PM, Stefan Koch wrote:
 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or 
 rather will eventually do), that is exporting _all_ the 
 properties of the type into an object)
 
 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is 
 kindof useless if it does not happen at compile time.
 
 Perhaps there are others?
Serialization!
Well that one you would kindof have to do in a template anyway, no?
Sep 28 2020
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 29/09/2020 1:22 PM, Stefan Koch wrote:
 On Tuesday, 29 September 2020 at 00:03:14 UTC, rikki cattermole wrote:
 On 29/09/2020 12:37 PM, Stefan Koch wrote:
 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or rather will 
 eventually do), that is exporting _all_ the properties of the type 
 into an object)

 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is kindof 
 useless if it does not happen at compile time.

 Perhaps there are others?
Serialization!
Well that one you would kindof have to do in a template anyway, no?
No. Pointer arithmetic alone could do most of the work. But if the end abstraction produced had UDA's and methods available (so full blown reflection), you could do pretty amazing things. This sort of stuff really doesn't need template instances to perform. Same goes for web routers. You really don't need much info beyond some UDA's and to confirm the parameters/return type are correct. The rest is all better off being done at runtime.
Sep 28 2020
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 29.09.20 01:37, Stefan Koch wrote:
 On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:
 On 28.09.20 23:08, claptrap wrote:
 Instead of first class types
(Stefan's type functions do not give you first-class types. A first-class type you could put in a runtime variable.)
That's true. At least as it stands right now. I _could_ give  the alias a physical form at runtime. (By doing the same thing that the reify template does (or rather will eventually do), that is exporting _all_ the properties of the type into an object) However I am not actually sure what the use of this would be. All usage that I have seen, is in generating code, which is kindof useless if it does not happen at compile time. Perhaps there are others?
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
Sep 30 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 01, 2020 at 01:17:27AM +0200, Timon Gehr via Digitalmars-d wrote:
 On 29.09.20 01:37, Stefan Koch wrote:
[...]
 That's true. At least as it stands right now.
 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or rather
 will eventually do), that is exporting _all_ the properties of the
 type into an object)
 
 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is kindof
 useless if it does not happen at compile time.
 
 Perhaps there are others?
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
I think there's a lot of value to be had in making typeid(T) as the reification of T. At compile-time, we treat it specially by augmenting it with the ability to do things that can only be done at compile-time, such as recover T given typeid(T) (pass it into a template parameter, construct new types out of it, etc.). At runtime, it reverts to the current behaviour of typeid(T). Only trouble is, according to Andrei, typeid has been written over so many times that even Walter doesn't understand how it works anymore. Meaning there are probably weird corner cases and quirky behaviours that we may not want to duplicate at compile-time. So I dunno, this seems like a roadblock to further progress. (It really makes one wish for a gradual transition to D3 where we can correct such deeply-rooted flaws that may otherwise be uncorrectible.) T -- To provoke is to call someone stupid; to argue is to call each other stupid.
Sep 30 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/30/20 7:36 PM, H. S. Teoh wrote:
 I think there's a lot of value to be had in making typeid(T) as the
 reification of T.  At compile-time, we treat it specially by augmenting
 it with the ability to do things that can only be done at compile-time,
 such as recover T given typeid(T) (pass it into a template parameter,
 construct new types out of it, etc.). At runtime, it reverts to the
 current behaviour of typeid(T).
 
 Only trouble is, according to Andrei, typeid has been written over so
 many times that even Walter doesn't understand how it works anymore.
 Meaning there are probably weird corner cases and quirky behaviours that
 we may not want to duplicate at compile-time. So I dunno, this seems
 like a roadblock to further progress.
Yah, that would be my favorite by far. It's a one-liner change in the definition of the language. "Since version x.xx, the result of typeid() is usable during compilation." It's also the best /kind/ of change to the language, i.e. lifts a gratuitous restriction that prevents composition of distinct features (here, ctfe and typeid). A promising path according to Walter is to develop a parallel enhanced feature - newtypeid! :o) - and then deprecate typeid.
Sep 30 2020
parent Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 1 October 2020 at 01:49:02 UTC, Andrei Alexandrescu 
wrote:
 On 9/30/20 7:36 PM, H. S. Teoh wrote:
 I think there's a lot of value to be had in making typeid(T) 
 as the
 reification of T.  At compile-time, we treat it specially by 
 augmenting
 it with the ability to do things that can only be done at 
 compile-time,
 such as recover T given typeid(T) (pass it into a template 
 parameter,
 construct new types out of it, etc.). At runtime, it reverts 
 to the
 current behaviour of typeid(T).
 
 Only trouble is, according to Andrei, typeid has been written 
 over so
 many times that even Walter doesn't understand how it works 
 anymore.
 Meaning there are probably weird corner cases and quirky 
 behaviours that
 we may not want to duplicate at compile-time. So I dunno, this 
 seems
 like a roadblock to further progress.
Yah, that would be my favorite by far. It's a one-liner change in the definition of the language. "Since version x.xx, the result of typeid() is usable during compilation." It's also the best /kind/ of change to the language, i.e. lifts a gratuitous restriction that prevents composition of distinct features (here, ctfe and typeid). A promising path according to Walter is to develop a parallel enhanced feature - newtypeid! :o) - and then deprecate typeid.
Glad to see that you're open to changing the language in order to achieve better power/complexity ratio. Right there with you on that sentiment! Two questions: 1) How many additional additions do you estimate you'll need for this? and 2) How do the type function addition(s) compare? Are they also of the "best kind of change" variety?
Sep 30 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr wrote:
 On 29.09.20 01:37, Stefan Koch wrote:
 On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:
 On 28.09.20 23:08, claptrap wrote:
 Instead of first class types
(Stefan's type functions do not give you first-class types. A first-class type you could put in a runtime variable.)
That's true. At least as it stands right now. I _could_ give  the alias a physical form at runtime. (By doing the same thing that the reify template does (or rather will eventually do), that is exporting _all_ the properties of the type into an object) However I am not actually sure what the use of this would be. All usage that I have seen, is in generating code, which is kindof useless if it does not happen at compile time. Perhaps there are others?
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
Yes I do understand that. I was wondering about practical usecases. As far as I an aware, if I made the leap to first class types, that would make all usage of them into static polymorphism. (equivalent to templates) And with that all the issues come back.
Sep 30 2020
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01.10.20 04:49, Stefan Koch wrote:
 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr wrote:
 On 29.09.20 01:37, Stefan Koch wrote:
 On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:
 On 28.09.20 23:08, claptrap wrote:
 Instead of first class types
(Stefan's type functions do not give you first-class types. A first-class type you could put in a runtime variable.)
That's true. At least as it stands right now. I _could_ give  the alias a physical form at runtime. (By doing the same thing that the reify template does (or rather will eventually do), that is exporting _all_ the properties of the type into an object) However I am not actually sure what the use of this would be. All usage that I have seen, is in generating code, which is kindof useless if it does not happen at compile time. Perhaps there are others?
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
Yes I do understand that. I was wondering about practical usecases. As far as I an aware, if I made the leap to first class types, that would make all usage of them into static polymorphism. (equivalent to templates) And with that all the issues come back.
Well, no, the restriction to compile-time-known values makes types not first-class. E.g., this is something you could do with first-class types but not with templates: (type T)=>(T x)=>x; (In terms of compiler implementation, the runtime representation of `T` would contain all information that is necessary to figure out calling conventions of functions that take a T, it would contain sizeof, pointers to destructor/postblit, etc, so basically it's typeid.) Templates are an untyped macro language on top of a simple type system without any form of polymorphism. Types are first-class values only inside that macro language, but the macro language is evaluated only at compile time. I.e., templates are a tool for code generation that is integrated with the type system, but they are not really a part of the type system.
Oct 01 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2020 12:48 AM, Timon Gehr wrote:
 (In terms of compiler implementation, the runtime representation of `T` would 
 contain all information that is necessary to figure out calling conventions of 
 functions that take a T, it would contain sizeof, pointers to 
 destructor/postblit, etc, so basically it's typeid.)
I had looked into this. Unfortunately, many aspects of types are simply too complex to try and represent at run time. For example, is type T1 implicitly convertible to T2? This seemingly simple question is very complex. Yes, `alias this` makes it much worse :-/ I initially had high hopes for typeid back when it was originally designed 20 years ago. It ably fulfills its original purpose of enabling the GC and associative arrays in a language with no templates. I don't see much future for it other than being kept around for legacy compatibility.
Oct 01 2020
next sibling parent Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 08:56:58 UTC, Walter Bright wrote:
 I had looked into this. Unfortunately, many aspects of types 
 are simply too complex to try and represent at run time. For 
 example, is type T1 implicitly convertible to T2? This 
 seemingly simple question is very complex. Yes, `alias this` 
 makes it much worse :-/
The my latest post please! The example on type functions with the conversion matrix. This is all you need: override void visit(IsExp e) { auto targe = e.targ.isTypeExpression(); if(!targe) { e.error("is expressions within type functions may only use type expressions"); result = ErrorExp.get(); return ; } auto targ = ctfeInterpret(targe.exp); auto te = targ.isTypeExp(); result = IntegerExp.createBool(te && te.type && te.type.ty != Terror); auto tspece = e.tspec ? e.tspec.isTypeExpression() : null; auto tspec = tspece ? ctfeInterpret(tspece.exp) : null; auto ts = tspec ? tspec.isTypeExp() : null; // handling of == and && // See IsExp handling in expressionsem.d if (e.tspec && !e.id && !(e.parameters && e.parameters.dim)) { if (e.tok == TOK.colon) { result = IntegerExp.createBool(te.type.implicitConvTo(ts.type) != MATCH.nomatch); } else if(e.tok == TOK.equal) { result = IntegerExp.createBool(te.type.equals(ts.type)); } } else if (e.tok2 != TOK.reserved) { e.error("Complex pattern matching forms of is expressions are not supported in TypeFunctions yet"); result = IntegerExp.createBool(false); } }
Oct 01 2020
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 01.10.20 10:56, Walter Bright wrote:
 On 10/1/2020 12:48 AM, Timon Gehr wrote:
 (In terms of compiler implementation, the runtime representation of 
 `T` would contain all information that is necessary to figure out 
 calling conventions of functions that take a T, it would contain 
 sizeof, pointers to destructor/postblit, etc, so basically it's typeid.)
I had looked into this. Unfortunately, many aspects of types are simply too complex to try and represent at run time. For example, is type T1 implicitly convertible to T2? This seemingly simple question is very complex. Yes, `alias this` makes it much worse :-/ ...
The compiler can figure it out, so you *could* just put that logic into druntime. However, for first-class types, you don't need to check implicit conversions at run time, therefore this would not be necessary.
 I initially had high hopes for typeid back when it was originally 
 designed 20 years ago. It ably fulfills its original purpose of enabling 
 the GC and associative arrays in a language with no templates.
 ...
And a monomorphic type system.
 I don't see much future for it other than being kept around for legacy 
 compatibility.
I was explaining conditions for types to be "first-class". So unless that is a goal, typeid might not see a revival.
Oct 01 2020
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 1 October 2020 at 02:49:43 UTC, Stefan Koch wrote:
 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr 
 wrote:
 On 29.09.20 01:37, Stefan Koch wrote:
 [...]
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
Yes I do understand that. I was wondering about practical usecases.
import std.algorithm: filter, equal; type[] types = [int, uint, long, ulong]; auto size4 = types.filter!(a => a.sizeof == 4); assert(equal(size4, [int, uint])); No more std.meta.Filter.
 As far as I an aware, if I made the leap to first class types, 
 that would make all usage of them into static polymorphism. 
 (equivalent to templates)
 And with that all the issues come back.
I can't see how that's the case.
Oct 08 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 8 October 2020 at 09:44:33 UTC, Atila Neves wrote:
 On Thursday, 1 October 2020 at 02:49:43 UTC, Stefan Koch wrote:
 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr 
 wrote:
 On 29.09.20 01:37, Stefan Koch wrote:
 [...]
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
Yes I do understand that. I was wondering about practical usecases.
import std.algorithm: filter, equal; type[] types = [int, uint, long, ulong]; auto size4 = types.filter!(a => a.sizeof == 4); assert(equal(size4, [int, uint])); No more std.meta.Filter.
 As far as I an aware, if I made the leap to first class types, 
 that would make all usage of them into static polymorphism. 
 (equivalent to templates)
 And with that all the issues come back.
I can't see how that's the case.
What you just described is what type functions do. The point that with first class type you could do also this: // cat type_list.txt: 'int, uint, long, ulong'; auto type_txt = readText("type_list.txt"); type[] types = type_txt.parseTypes() auto size4 = types.filter!(a => a.sizeof == 4); assert(equal(size4, [int, uint])); type[0] myVar1; type[1] myVar2;
Oct 08 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 8 October 2020 at 09:51:28 UTC, Stefan Koch wrote:
 On Thursday, 8 October 2020 at 09:44:33 UTC, Atila Neves wrote:
 On Thursday, 1 October 2020 at 02:49:43 UTC, Stefan Koch wrote:
 [...]
import std.algorithm: filter, equal; type[] types = [int, uint, long, ulong]; auto size4 = types.filter!(a => a.sizeof == 4); assert(equal(size4, [int, uint])); No more std.meta.Filter.
 [...]
I can't see how that's the case.
What you just described is what type functions do. The point that with first class type you could do also this: // cat type_list.txt: 'int, uint, long, ulong'; auto type_txt = readText("type_list.txt"); type[] types = type_txt.parseTypes() auto size4 = types.filter!(a => a.sizeof == 4); assert(equal(size4, [int, uint])); type[0] myVar1; type[1] myVar2;
There's no IO at compile-time.
Oct 08 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 8 October 2020 at 10:05:10 UTC, Atila Neves wrote:
 On Thursday, 8 October 2020 at 09:51:28 UTC, Stefan Koch wrote:
 On Thursday, 8 October 2020 at 09:44:33 UTC, Atila Neves wrote:
 [...]
What you just described is what type functions do. The point that with first class type you could do also this: // cat type_list.txt: 'int, uint, long, ulong'; auto type_txt = readText("type_list.txt"); type[] types = type_txt.parseTypes() auto size4 = types.filter!(a => a.sizeof == 4); assert(equal(size4, [int, uint])); type[0] myVar1; type[1] myVar2;
There's no IO at compile-time.
Exactly! first class type can be determined at runtime!
Oct 08 2020
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 8 October 2020 at 09:44:33 UTC, Atila Neves wrote:
 On Thursday, 1 October 2020 at 02:49:43 UTC, Stefan Koch wrote:
 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr 
 wrote:
 On 29.09.20 01:37, Stefan Koch wrote:
 [...]
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
Yes I do understand that. I was wondering about practical usecases.
import std.algorithm: filter, equal; type[] types = [int, uint, long, ulong]; auto size4 = types.filter!(a => a.sizeof == 4); assert(equal(size4, [int, uint])); No more std.meta.Filter.
 As far as I an aware, if I made the leap to first class types, 
 that would make all usage of them into static polymorphism. 
 (equivalent to templates)
 And with that all the issues come back.
I can't see how that's the case.
This is the correct type function way of doing it: enum type[] types = makeTypeArray(int, uint, long, ulong); enum type[] size4 = types.filter!((type a) => a.sizeof == 4); static assert(equal!((type a, type b) => a is b) (size4, makeTypeArray(int, uint))); Currently type[] fails is isInputRange though and therefore the phobos filter won't instantiate. I am debugging this.
Oct 08 2020
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 8 October 2020 at 10:16:10 UTC, Stefan Koch wrote:
 This is the correct type function way of doing it:
 enum type[] types = makeTypeArray(int, uint, long, ulong);
 enum type[] size4 = types.filter!((type a) => a.sizeof == 4);
 static assert(equal!((type a, type b) => a is b) (size4, 
 makeTypeArray(int, uint)));

 Currently type[] fails is isInputRange though and therefore the 
 phobos filter won't instantiate.
 I am debugging this.
This is plain awesome. The fact that it reuses the filter algorithm is a sign that both filter and type functions are correctly designed. Who could ever imagined that a template would be instantiated with a lambda that takes a type-value as an argument. WAT!? That is some high-order stuff, and perfectly readable. Andrei's de/reify is awesome as well, it has the advantage of working with existing language constructs, but it isn't as simple.
Oct 08 2020
prev sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr wrote:
 On 29.09.20 01:37, Stefan Koch wrote:
 ...
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
Can anything computable using just the source as input be considered a first class type? If so, what do we call type variables that can not be?
Sep 30 2020
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01.10.20 07:37, Bruce Carneal wrote:
 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr wrote:
 On 29.09.20 01:37, Stefan Koch wrote:
 ...
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
Can anything computable using just the source as input be considered a first class type? If so, what do we call type variables that can not be?
Unfortunately I am not sure understand the question. What is an example of a "type variable that cannot be computed using just the source"? (Which source?) Maybe this is related to what you mean? type t = readln().strip()=="int"?int:double; auto f = (t x)=>x;
Oct 01 2020
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2020 12:51 AM, Timon Gehr wrote:
 Maybe this is related to what you mean?
 
 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;
As D is a statically typed language, I don't see how that can ever work.
Oct 01 2020
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01.10.20 10:59, Walter Bright wrote:
 On 10/1/2020 12:51 AM, Timon Gehr wrote:
 Maybe this is related to what you mean?

 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;
As D is a statically typed language, I don't see how that can ever work.
The example is easy to type check statically. In terms of ABI, you just have to make sure that `f` can find the context pointer without already knowing the value of `t`, then it can reconstruct everything it needs to know.
Oct 01 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2020 5:41 AM, Timon Gehr wrote:
 On 01.10.20 10:59, Walter Bright wrote:
 On 10/1/2020 12:51 AM, Timon Gehr wrote:
 Maybe this is related to what you mean?

 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;
As D is a statically typed language, I don't see how that can ever work.
The example is easy to type check statically. In terms of ABI, you just have to make sure that `f` can find the context pointer without already knowing the value of `t`, then it can reconstruct everything it needs to know.
I think you described D's "class" type. To make it work with int and double, you'd need to "box" them with a class. Like Java.
Oct 01 2020
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 02.10.20 06:04, Walter Bright wrote:
 On 10/1/2020 5:41 AM, Timon Gehr wrote:
 On 01.10.20 10:59, Walter Bright wrote:
 On 10/1/2020 12:51 AM, Timon Gehr wrote:
 Maybe this is related to what you mean?

 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;
As D is a statically typed language, I don't see how that can ever work.
The example is easy to type check statically. In terms of ABI, you just have to make sure that `f` can find the context pointer without already knowing the value of `t`, then it can reconstruct everything it needs to know.
I think you described D's "class" type. To make it work with int and double, you'd need to "box" them with a class. Like Java.
No, you'd still pass them by value in appropriate registers or on the stack, but you only figure out at run time where the argument is actually located.
Oct 02 2020
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 1 October 2020 at 08:59:04 UTC, Walter Bright wrote:
 On 10/1/2020 12:51 AM, Timon Gehr wrote:
 Maybe this is related to what you mean?
 
 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;
As D is a statically typed language, I don't see how that can ever work.
Flow analysis. You can deduce that t can only be int|double at the second line and then expand the code into a switch over the type (encoded as enum values). It cannot work with separate compilation though.
Oct 01 2020
prev sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 1 October 2020 at 07:51:28 UTC, Timon Gehr wrote:
 On 01.10.20 07:37, Bruce Carneal wrote:
 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr 
 wrote:
 On 29.09.20 01:37, Stefan Koch wrote:
 ...
It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification: type t = int; auto f = (t x)=>x;
Can anything computable using just the source as input be considered a first class type? If so, what do we call type variables that can not be?
Unfortunately I am not sure understand the question. What is an example of a "type variable that cannot be computed using just the source"? (Which source?) Maybe this is related to what you mean? type t = readln().strip()=="int"?int:double; auto f = (t x)=>x;
Yes. The common understanding of a "statically typed language" is quite a bit more restrictive than "anything which can be determined from a fixed 'source' input" yet that is the bounding condition, it seems, for "static" compilation. I lack the terminology to discuss this succinctly so I asked. Note, my lack of understanding is sufficient that I may not have posed a well formed question. Illumination is requested.
Oct 01 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 9/28/20 5:08 PM, claptrap wrote:
 
 Who's actually heard of reification outside of compiler writers? Ive 
 been reading a ton of compiler related stuff lately, but not heard of it 
 before now. It's seems like an implementation detail leaking out to me. 
 Instead of first class types lets have whatever this stuff is cause we 
 can get it on the cheap and its intellectually cool.
Reification is first class types.
Sep 28 2020
parent reply claptrap <clap trap.com> writes:
On Monday, 28 September 2020 at 21:55:45 UTC, Andrei Alexandrescu 
wrote:
 On 9/28/20 5:08 PM, claptrap wrote:
 
 Who's actually heard of reification outside of compiler 
 writers? Ive been reading a ton of compiler related stuff 
 lately, but not heard of it before now. It's seems like an 
 implementation detail leaking out to me. Instead of first 
 class types lets have whatever this stuff is cause we can get 
 it on the cheap and its intellectually cool.
Reification is first class types.
If you have to litter your code with refiy / dereify then it's not first class. It's not a language feature but a library one. Hence not first class.
Sep 28 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/28/20 6:13 PM, claptrap wrote:
 On Monday, 28 September 2020 at 21:55:45 UTC, Andrei Alexandrescu wrote:
 On 9/28/20 5:08 PM, claptrap wrote:
 Who's actually heard of reification outside of compiler writers? Ive 
 been reading a ton of compiler related stuff lately, but not heard of 
 it before now. It's seems like an implementation detail leaking out 
 to me. Instead of first class types lets have whatever this stuff is 
 cause we can get it on the cheap and its intellectually cool.
Reification is first class types.
If you have to litter your code with refiy / dereify then it's not first class. It's not a language feature but a library one. Hence not first class.
I agree. Reification is a transformation that takes a non-first-class type and produces a first-class value.
Sep 28 2020
prev sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Monday, 28 September 2020 at 16:57:19 UTC, Andrei Alexandrescu 
wrote:
 On 9/28/20 11:55 AM, Bruce Carneal wrote:
 On Monday, 28 September 2020 at 14:16:20 UTC, Andrei 
 Alexandrescu wrote:
 On 9/28/20 8:46 AM, Stefan Koch wrote:
 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei 
 Alexandrescu wrote:
 On 9/27/20 10:58 PM, Bruce Carneal wrote:
 Disclaimer: Andrei has stated, effectively, that I have 
 little standing to opine on these issues.
Sorry! Didn't mean to sound dismissive at all. Meant to just say, don't shun templates, embrace them and thrive.
In practice template can quite literally explode though. Which could be a valid reason to shun them. You can embrace them, of course, and thrive. The question is for how long you will thrive.
For a very long time judging by the success C++ is enjoying with them.
As you note, by employing a variety of "best practices", of extra-language conventions, of one-level-of-indirection wrappers, of "human must certify this correct" mechanisms, the C++ community has indeed "thrived". We've not settled for that meta programming drudgery, that friction, in the past.  You know better this better than anyone else on the planet. I hope we don't "settle" going forward.
(Not getting some of the uses of quotation marks.) That's a bit backhanded because it implies I promote settling for meta programming drudgery. Did you mean to say that?
My gosh, no! My apologies for leaving you with that impression. You were and are one of the major players in the C++ friction/drudgery smackdown that is the D language. I was attempting to say that there's an opportunity for another language level complexity smackdown that I believe covers, en passant, your reify/dereify use case.
 On the contrary, I find type reification interesting exactly 
 because it takes you from said drudgery to familiar land - 
 first-class values that can be manipulated in traditional 
 manner. Should you need to get back to type-land, dereification 
 helps with that. We can now draw from a large body of existing 
 theory and practice.
OK. I'll really try to do better here. Zero disrespect intended. When you leave the type system behind, when you reify, you must assume responsibility for constraints that were previously, and seamlessly, taken care of by the type system. The drudgery, the friction, follows directly from the decision to escape from the type system (reify) rather than remain within it (type functions). I think of it as being similar to CT functions vs templates. Within CT functions you've got the type system on your side. Everybody loves CT functions because everything "just works" as you'd expect. Near zero additional semantic load. Wonderfully boring. Within templates, on the other hand, you'd better get your big-boy britches on because it's pretty much all up to you pardner! (manually inserted constraints, serious tension between generality and debugability, composition difficulties, lazy/latent bugs in the general forms, localization difficulties, ...) If language additions like type functions are off the table, then we're left with LDMs and you've produced what looks like a dandy in reify/dereify. If we have to step outside the language, if language additions are just not in the cards any more, then something like reify/dereify will be the way to go. I hope that we've not hit that wall just yet but even if we have D will remain, in my opinion, head and shoulders above anything else out there. It is a truly wonderful language. I am very grateful for the work you, Walter, and many many others have put in to make it so. (most recent standout, Mathias! I'm a -preview=in fan) Finally, I'd love to hear your comments on type functions vs reify/dereify. It's certainly possible that I've missed something. Maybe it's a type functions+ solution we should be seeking (type functions for everything they can do, and some LDM for anything beyond their capability).
Sep 28 2020
next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Monday, 28 September 2020 at 22:46:55 UTC, Bruce Carneal wrote:
 On Monday, 28 September 2020 at 16:57:19 UTC, Andrei 
 Alexandrescu wrote:
 On 9/28/20 11:55 AM, Bruce Carneal wrote:
...
 If language additions like type functions are off the table, 
 then we're left with LDMs
Language Deficiency Mitigation
 and you've produced what looks like a dandy in reify/dereify.  
 If we have to step outside the language, if language additions 
 are just not in the cards any more, then something like 
 reify/dereify will be the way to go.
Sep 28 2020
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/28/20 6:46 PM, Bruce Carneal wrote:
 
 When you leave the type system behind, when you reify, you must assume 
 responsibility for constraints that were previously, and seamlessly, 
 taken care of by the type system.  The drudgery, the friction, follows 
 directly from the decision to escape from the type system (reify) rather 
 than remain within it (type functions).
 
 I think of it as being similar to CT functions vs templates. Within CT 
 functions you've got the type system on your side. Everybody loves CT 
 functions because everything "just works" as you'd expect.  Near zero 
 additional semantic load.
 Wonderfully boring.
 
 Within templates, on the other hand, you'd better get your big-boy 
 britches on because it's pretty much all up to you pardner!  (manually 
 inserted constraints, serious tension between generality and 
 debugability, composition difficulties, lazy/latent bugs in the general 
 forms, localization difficulties, ...)
 
 If language additions like type functions are off the table, then we're 
 left with LDMs and you've produced what looks like a dandy in 
 reify/dereify.  If we have to step outside the language, if language 
 additions are just not in the cards any more, then something like 
 reify/dereify will be the way to go.
 
 I hope that we've not hit that wall just yet but even if we have D will 
 remain, in my opinion, head and shoulders above anything else out 
 there.  It is a truly wonderful language.  I am very grateful for the 
 work you, Walter, and many many others have put in to make it so.  (most 
 recent standout, Mathias!  I'm a -preview=in fan)
 
 Finally, I'd love to hear your comments on type functions vs 
 reify/dereify.  It's certainly possible that I've missed something.  
 Maybe it's a type functions+ solution we should be seeking (type 
 functions for everything they can do, and some LDM for anything beyond 
 their capability).
Years ago, I was in a panel with Simon Peyton-Jones, the creator of Haskell. Nicest guy around, not to mention an amazing researcher. This question was asked: "What is the most important principle of programming language design?" His answer was so powerful, it made me literally forget everybody else's answer, including my own. (And there were no slouches in that panel, save for me: Martin Odersky, Guido van Rossum, Bertrand Meyer.) He said the following: "The most important principle in a language design is to define a small core of essential primitives. All necessary syntactic sugar lowers to core constructs. Do everything else in libraries." (He didn't use the actual term "lowering", which is familiar to our community, but rather something equivalent such as "reduces".) That kind of killed the panel, in a good way. Because most other questions on programming language design and implementation simply made his point shine quietly. Oh yes, if you had a small core and build on it you could do this easily. And that. And the other. In a wonderfully meta way, all questions got instantly lowered to simpler versions of themselves. I will never forget that experience. D breaks that principle in several places. It has had a cavalier attitude to using magic tricks in the compiler to get things going, at the expense of fuzzy corners and odd limit cases. Look at hashtables. Nobody can create an equivalent user-defined type, and worse, nobody knows exactly why. (I recall vaguely it has something to do, among many other things, with qualified keys that are statically-sized arrays. Hacks in the compiler make those work, but D's own type system rejects the equivalent code. So quite literally D's type system cannot verify its own capabilities.) Or take implicit conversions. They aren't fully documented, and the only way to figure things out is to read most of the 7193 lines of https://github.com/dlang/dmd/blob/master/src/dmd/mtype.d. That's not a small core with a little sugar on top. Or take the foreach statement. Through painful trial and error, somebody figured out all possible shapes of foreach, and defined `each` to support most: https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L904 What should have been a simple forwarding problem took 190 lines that could be characterized as very involved. And mind you, it doesn't capture all cases because per https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L1073: // opApply with >2 parameters. count the delegate args. // only works if it is not templated (otherwise we cannot count the args) I know that stuff (which will probably end on my forehead) because I went and attempted to improve things a bit in https://github.com/dlang/phobos/pull/7638/files#diff-0d6463fc6f41c5fb7 4300832e3135f5R805, which attempts to simplify matters by reducing the foreach cases to seven shapes. To paraphrase Alan Perlis: "If your foreach has seven possible shapes, you probably missed some." Over time, things did get better. We became adept of things such as lowering, and we require precision in DIPs. I very strongly believe that this complexity, if unchecked, will kill the D language. It will sink under its own weight. We need a precise, clear definition of the language, we need a principled approach to defining and extending features. The only right way to extend D at this point is to remove the odd failure modes created by said cavalier approach to doing things. Why can't I access typeid during compilation, when I can access other pointers? Turns out typeid is a palimpsest that has been written over so many times, even Walter cannot understand it. He told me plainly to forget about trying to use typeid and write equivalent code from scratch instead. That code has achieved lifetime employment. And no compiler tricks. I am very, very opposed to Walter's penchant to reach into his bag of tricks whenever a difficult problem comes about. Stop messing with the language! My discussions with him on such matters are like trying to talk a gambler out of the casino. That is a long way to say I am overwhelmingly in favor of in-language solutions as opposed to yet another addition to the language. To me, if I have an in-language solution, it's game, set, and match. No need for language changes, thank you very much. An in-language solution doesn't only mean no addition is needed, but more importantly it means that the language has sufficient power to offer D coders means to solve that and many other problems. I almost skipped a heartbeat when Adam mentioned - mid-sentence! - just yet another little language addition to go with that DIP for interpolated strings. It would make things nicer. You know what? I think I'd rather live without that DIP. So I'm interested in exploring reification mainly - overwhelmingly - because it already works in the existing language. To the extent it has difficulties, it's because of undue limitations in CTFE and introspection. And these are problems we must fix anyway. So it's all a virtuous circle. Bringing myself to write `reify` and `dereify` around my types is a very small price to pay for all that.
Sep 28 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:
 D breaks that principle in several places. It has had a 
 cavalier attitude to using magic tricks in the compiler to get 
 things going, at the expense of fuzzy corners and odd limit 
 cases. Look at hashtables. Nobody can create an equivalent 
 user-defined type, and worse, nobody knows exactly why. (I 
 recall vaguely it has something to do, among many other things, 
 with qualified keys that are statically-sized arrays. Hacks in 
 the compiler make those work, but D's own type system rejects 
 the equivalent code. So quite literally D's type system cannot 
 verify its own capabilities.)
[...]
 I very strongly believe that this complexity, if unchecked, 
 will kill the D language. It will sink under its own weight. We 
 need a precise, clear definition of the language, we need a 
 principled approach to defining and extending features. The 
 only right way to extend D at this point is to remove the odd 
 failure modes created by said cavalier approach to doing 
 things. Why can't I access typeid during compilation, when I 
 can access other pointers? Turns out typeid is a palimpsest 
 that has been written over so many times, even Walter cannot 
 understand it. He told me plainly to forget about trying to use 
 typeid and write equivalent code from scratch instead. That 
 code has achieved lifetime employment.
I think D is already too deep in the tar pit to ever fully extricate itself. All of those special cases and "magic tricks" are there because, as you say, code depends on them to work. Getting read of them would break the world a hundred times over. If you're going to do that, you might as well start a new language from scratch, no? The best we can hope for is to nail down a precise description of all the fiddly corner cases, so that we know what D's semantics *actually* are, rather than what the current spec pretends they are. But even that is likely to be infeasible in practice--who wants to volunteer to be the one to go over, say, funcDeclarationSemantic with a fine-toothed comb, especially after what happened to the last guy? [1] The fact is, there's no incentive for anyone to do this kind of work other than masochism and sheer stubbornness. The amount of effort it would take, and the amount of time that would pass before you saw any benefit, are simply too high. We can talk about principles and good engineering all we want, but we can't expect people to voluntarily act against their own interest. Maybe we can make some improvements at the margins, but my prediction is that the overwhelming majority of D's complexity is here to stay, whether we like it or not. The best we can do is try to offer enough positive value to D programmers to offset its negative effects. And if it doesn't work out, well, there's always Rust. ;) [1] https://github.com/dlang/dmd/pull/8668
Sep 28 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/20 12:56 AM, Paul Backus wrote:
 The best we can hope for is to nail down a precise description of all 
 the fiddly corner cases, so that we know what D's semantics *actually* 
 are, rather than what the current spec pretends they are.
Definitely. I'd go even further and call that "survival". C++ has had tremendous success with meticulously documenting all of the odd things they baked in pre-standardization. And then it was a matter of updating things, and it all got better. Most importantly, there was and is a group of people who understood the vital importance of that.
 But even that 
 is likely to be infeasible in practice--who wants to volunteer to be the 
 one to go over, say, funcDeclarationSemantic with a fine-toothed comb, 
 especially after what happened to the last guy? [1]
Actually Walter made a lot of efforts to do that, too, as he very well understands the importance of correct specification. But nobody cared about his related PRs, which probably are flapping in the wind to this day. He has some work on defining the object model, too, which has also been ignored. A kernel of 3-5 strong souls to commit to this would be very necessary. Walter can't do all of this alone. Now he can't even do it formally because of the "no single person" rule. I should add I've been unfairly harsh about Walter's inclination to do tricks in the compiler. That applies a lot more to the early days than now. As of now our views are aligned to a large proportion on core language vs. library features.
 The fact is, there's no incentive for anyone to do this kind of work 
 other than masochism and sheer stubbornness. The amount of effort it 
 would take, and the amount of time that would pass before you saw any 
 benefit, are simply too high. We can talk about principles and good 
 engineering all we want, but we can't expect people to voluntarily act 
 against their own interest.
 
 Maybe we can make some improvements at the margins, but my prediction is 
 that the overwhelming majority of D's complexity is here to stay, 
 whether we like it or not. The best we can do is try to offer enough 
 positive value to D programmers to offset its negative effects. And if 
 it doesn't work out, well, there's always Rust. ;)
I think things can be managed, but documentation and control are a must. FWIW I don't think Rust fares that well in that area - it's a very complex language, with the complexity mostly in the wrong places leaving mediocre handling of all that's interesting, and its rigorous documentation is greatly outmatched by its marketing brochures.
Sep 29 2020
prev sibling next sibling parent reply claptrap <clap trap.com> writes:
On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:
 On 9/28/20 6:46 PM, Bruce Carneal wrote:
 Years ago, I was in a panel with Simon Peyton-Jones, the 
 creator of Haskell. Nicest guy around, not to mention an 
 amazing researcher. This question was asked: "What is the most 
 important principle of programming language design?"

 His answer was so powerful, it made me literally forget 
 everybody else's answer, including my own. (And there were no 
 slouches in that panel, save for me: Martin Odersky, Guido van 
 Rossum, Bertrand Meyer.) He said the following:

 "The most important principle in a language design is to define 
 a small core of essential primitives. All necessary syntactic 
 sugar lowers to core constructs. Do everything else in 
 libraries."
I agree with pretty much all of what you've just said FWIW. However... How do you decide what is an essential language primitive? Or what is necessary syntactic sugar? I mean syntactic sugar almost by definition isnt necessary, it's just window dressing. It could be argued that compile time first class types is an essential primitive for meta-programming?
Sep 29 2020
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 29 September 2020 at 09:07:39 UTC, claptrap wrote:
 How do you decide what is an essential language primitive? Or 
 what is necessary syntactic sugar? I mean syntactic sugar 
 almost by definition isnt necessary, it's just window dressing.
The for loop and the while loop is syntactic sugar in C-like languages. So clearly syntactic sugar is more than window dressing. The usual way to define a minimal imperative language is to have concepts like "block" and "restart block" (covers both looping and continue) and conditionals. You can essentially do away with most concepts until you have just have a closure with coroutine capabilities + one conditional instruction. Two such tiny imperative languages are Self and Beta. So you basically merge the concept of a block and an object as well.. A constructor/destructor pair is just a single coroutine that has suspended. However the minimal core language tend to be an on-paper construct for non-research languages. Theoretical. The compiler will most certainly have many more concepts than the minimal language to get better compilation times.
Sep 29 2020
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 29 September 2020 at 10:03:12 UTC, Ola Fosheim 
Grøstad wrote:
 conditional instruction. Two such tiny imperative languages are 
 Self and Beta. So you basically merge the concept of a block 
 and an object as well.. A constructor/destructor pair is just a 
 single coroutine that has suspended.
That last sentence is hypothetical. You could construct a language that only consists of concurrent closures. Would it be slow? Only if the optimizer isn't advanced enough, but in reality compiler authors depend on heuristics (common programming patterns) so it is quite difficult to get such languages to perform well in practice. Compilers are sadly not very good at high level optimization... Which is why languages ship with pre-canned constructs, libraries and runtimes...
Sep 29 2020
prev sibling parent reply claptrap <clap trap.com> writes:
On Tuesday, 29 September 2020 at 10:03:12 UTC, Ola Fosheim 
Grøstad wrote:
 On Tuesday, 29 September 2020 at 09:07:39 UTC, claptrap wrote:
 How do you decide what is an essential language primitive? Or 
 what is necessary syntactic sugar? I mean syntactic sugar 
 almost by definition isnt necessary, it's just window dressing.
The for loop and the while loop is syntactic sugar in C-like languages. So clearly syntactic sugar is more than window dressing.
OK yeah bad analogy. My point is that if you say "only what syntactic sugar is necessary" it doesn't really help because what does "necessary" mean? If you take it literally you dont need for and while loops because you have if and goto. But pretty much every language has them, so obviously there's more too it. its a great quote and you can see the wisdom in it but there's a lot lurking under the surface of the words "essential" and "necessary"
Sep 29 2020
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 29 September 2020 at 13:07:35 UTC, claptrap wrote:
 its a great quote and you can see the wisdom in it but there's 
 a lot lurking under the surface of the words "essential" and 
 "necessary"
Indeed. If you only have goto and if, would programmers then write code where common patterns occur or would the compiler writer just optimize for structures that have the properties of noise as programmers might end up writing very different code structures? I strongly suspect that one advantage of having a standard library and high level features is that programmers then tend to write code in similar patterns that compiler writers can take advantage of those patterns.
Sep 29 2020
prev sibling next sibling parent Jackel <jackel894_394 gmail.com> writes:
On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:
 Or take the foreach statement. Through painful trial and error, 
 somebody figured out all possible shapes of foreach, and 
 defined `each` to support most:

 https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L904

 What should have been a simple forwarding problem took 190 
 lines that could be characterized as very involved. And mind 
 you, it doesn't capture all cases because per 
 https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L1073:

 // opApply with >2 parameters. count the delegate args.
 // only works if it is not templated (otherwise we cannot count 
 the args)

 I know that stuff (which will probably end on my forehead) 
 because I went and attempted to improve things a bit in 
 https://github.com/dlang/phobos/pull/7638/files#diff-0d6463fc6f41c5fb7
4300832e3135f5R805, which attempts to simplify matters by reducing the foreach
cases to seven shapes.

 To paraphrase Alan Perlis: "If your foreach has seven possible 
 shapes, you probably missed some."

 Over time, things did get better. We became adept of things 
 such as lowering, and we require precision in DIPs.
I think this is a mischaracterization of a problem. I'm not really a fan of how ranges are implemented, especially how UFCS is used to chain functions together. When they start to become 10+ lines of a chain it is almost impossible to profile such code; with tools. Let alone try to understand what is happening, it would require to know how each function is implemented and how it expands and ends up being executed. Some things are lazy, somethings aren't; eg requiring `.array` after a `.map`. This is where I feel there shouldn't even be a `each` to begin with, just use a for loop. Now that's my opinion, I deal with low level code that needs to be optimized and those UFSC chains are likened to scripting languages that usually don't need to worry about optimization. For short chains I don't think it's that big of a deal, it's usually easy enough to figure out what is going on and not enough is happening for too much to go wrong.
 And mind you, it doesn't capture all cases because per 
 https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L1073:

 // opApply with >2 parameters. count the delegate args.
 // only works if it is not templated (otherwise we cannot count 
 the args)
If you use a template of `opApply()` you lose type inference (as you do with regular templates). It's a limitation of the feature. You'd have the same problem if you have an overload of `opApply()` with delegates that have the same number of arguments; it won't know which one to pick. If you want to support both you'd just need 1 other case, but honestly if you have a template `opApply`, that's just bad practice (due to losing type inference). It doesn't need to be supported. The only reason I've ever wanted a templated opApply() is for auto attribute inference, but that's not worth losing the auto type inference you get with foreach. There was a suggestion for adding a new feature, or rather altering the behavior to allow for both, but I don't think it was well received.
 I know that stuff (which will probably end on my forehead) 
 because I went and attempted to improve things a bit in 
 https://github.com/dlang/phobos/pull/7638/files#diff-0d6463fc6f41c5fb7
4300832e3135f5R805, which attempts to simplify matters by reducing the foreach
cases to seven shapes.
You are going the wrong way about simplifying the code. Why not add an opApply() property to the internal container types (array, assoc array, etc...)? Shouldn't be that difficult. Cases 4, 5, 6, and 7 would collapse into one then. Cases 1, 2, and 3 aren't really different ways of using foreach. They are just conveniences for ranges. That tuple expansion is odd though, doesn't seem to be documented and it's a feature that foreach doesn't have (unlike the second parameter for the index). Doesn't need to be there, but someone probably uses it somewhere now. So really it's just 2 cases with an additional case to support an index counter. That fits the bill, case 1: foreach, case 2: phobos ranges. It's not as bad as you make it out to be, for this case.
Sep 29 2020
prev sibling next sibling parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:
 On 9/28/20 6:46 PM, Bruce Carneal wrote:
[..]
 Years ago, I was in a panel with Simon Peyton-Jones, the 
 creator of Haskell. Nicest guy around, not to mention an 
 amazing researcher. This question was asked: "What is the most 
 important principle of programming language design?"

 His answer was so powerful, it made me literally forget 
 everybody else's answer, including my own. (And there were no 
 slouches in that panel, save for me: Martin Odersky, Guido van 
 Rossum, Bertrand Meyer.) He said the following:

 "The most important principle in a language design is to define 
 a small core of essential primitives. All necessary syntactic 
 sugar lowers to core constructs. Do everything else in 
 libraries."
Sorry to not be completely convinced on the pertinence of that statement. For the language designer it might well be true, for the users of the language I'm not completely sold. There are two languages that imho fit that description very well, they are both very powerfull and extremely elegant, but boy do they suck when you just want to get shit done: Lisp and Forth. And (I know, one should not start a sentence with and) it had been mentioned often in this forum, it is this shear flexibility that are in the way of wider adoption as there are so many ways of doing things, but none of them completely right. Your AA example is a good one. If it wasn't in the language, it would be the same issue with AA as in any other language that defines it in libraries: several different incompatible restricted implementations. Just my uninformed 2 cents.
Sep 29 2020
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 29 September 2020 at 12:02:21 UTC, Patrick Schluter 
wrote:
 Sorry to not be completely convinced on the pertinence of that 
 statement. For the language designer it might well be true, for 
 the users of the language I'm not completely sold. There are 
 two languages that imho fit that description very well, they 
 are both very powerfull and extremely elegant, but boy do they 
 suck when you just want to get shit done: Lisp and Forth.
Most research languages try to fit the description, but they rarely reach production. But in essence you are right. The key problem isn't functional, but syntactical. When I wrote in Beta I could do everything as easily as in most imperative languages, but everything looked the same. So you end up looking a landscape of pebbles in various colours, but that makes the landscape less visible. Compare that to landscape with roads, lakes and mountains. Template programming suffers from the same problem. You can create fancy semantics, but the syntax becomes very hard on the user. It leads to less legible code as code that represent different concepts don't stand out. You can say the same about some C code bases. So template programming is fun, but it will suck until there is a language that finds a good way to provide syntax extensions.
 Your AA example is a good one. If it wasn't in the language, it 
 would be the same issue with AA as in any other language that 
 defines it in libraries: several different incompatible 
 restricted implementations.
There are at least two ways to generalize AA, you can either have a language mechanism for serialization or mandate that all types provide a hash function and identity comparison (equality). I don't see how having a compiler built in is qualitatively different from having a standard library implementation in the general case.
Sep 29 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/29/20 8:02 AM, Patrick Schluter wrote:
 Your AA example is a good one. If it wasn't in the language, it would be 
 the same issue with AA as in any other language that defines it in 
 libraries: several different incompatible restricted implementations.
AA is not that far from being a library type. If you discount the magic casting that AAs can do (which isn't all that useful), there is one (yes ONE) feature that AAs do that a library type cannot. And that is an implicit adding of an element in certain cases. The reason AAs can do it is because the compiler calls a different opIndex (it's not called opIndex, but some extern(C) hook) when it's in this mode, but only for AAs. In fact, AAs are mostly already a library type, thanks to a lot of incremental effort over the years to lower the implementation to templates. Except for this one thing. -Steve
Sep 29 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/20 8:55 AM, Steven Schveighoffer wrote:
 On 9/29/20 8:02 AM, Patrick Schluter wrote:
 Your AA example is a good one. If it wasn't in the language, it would 
 be the same issue with AA as in any other language that defines it in 
 libraries: several different incompatible restricted implementations.
AA is not that far from being a library type. If you discount the magic casting that AAs can do (which isn't all that useful), there is one (yes ONE) feature that AAs do that a library type cannot. And that is an implicit adding of an element in certain cases. The reason AAs can do it is because the compiler calls a different opIndex (it's not called opIndex, but some extern(C) hook) when it's in this mode, but only for AAs. In fact, AAs are mostly already a library type, thanks to a lot of incremental effort over the years to lower the implementation to templates. Except for this one thing.
Thanks, good to know. I now recall I helped a little with that. (Was it the length function? empty? some range-related addition?) This supports the original point - it took much effort over years to disentangle and undo all this magic.
Sep 29 2020
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
On Tuesday, 29 September 2020 at 12:02:21 UTC, Patrick Schluter 
wrote:

 Your AA example is a good one. If it wasn't in the language, it 
 would be the same issue with AA as in any other language that 
 defines it in libraries: several different incompatible 
 restricted implementations.
That's a good example. IMO there's a difference between a language and bits and pieces that can be put together to call random libraries.
Sep 29 2020
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 29 September 2020 at 13:38:25 UTC, bachmeier wrote:
 That's a good example. IMO there's a difference between a 
 language and bits and pieces that can be put together to call 
 random libraries.
Heh, not really. It is not unusual for languages to have a minimal core and default include the most commonly used parts of their standard library. The programmer can often not tell the difference.
Sep 29 2020
parent reply bachmeier <no spam.net> writes:
On Tuesday, 29 September 2020 at 13:44:55 UTC, Ola Fosheim 
Grøstad wrote:
 On Tuesday, 29 September 2020 at 13:38:25 UTC, bachmeier wrote:
 That's a good example. IMO there's a difference between a 
 language and bits and pieces that can be put together to call 
 random libraries.
Heh, not really. It is not unusual for languages to have a minimal core and default include the most commonly used parts of their standard library. The programmer can often not tell the difference.
The statement was this:
 "The most important principle in a language design is to define 
 a small core of essential primitives. All necessary syntactic 
 sugar lowers to core constructs. Do everything else in 
 libraries."
It's possible I'm wrong, but it says to me that the language developers should design the language worrying only about that 'small core of essential primitives'. Maybe 'libraries' refers to the standard library only, but that's not clear to me.
Sep 29 2020
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 29 September 2020 at 18:53:26 UTC, bachmeier wrote:
 On Tuesday, 29 September 2020 at 13:44:55 UTC, Ola Fosheim 
 Grøstad wrote:
 It's possible I'm wrong, but it says to me that the language 
 developers should design the language worrying only about that 
 'small core of essential primitives'. Maybe 'libraries' refers 
 to the standard library only, but that's not clear to me.
Yes, it is not terribly clear what people are talking about in this thread because many different layers of a language is being lumped together. So let me try to paint up my view of these layers: 1. On the one hand you have the type system (I guess you can call that the constraints you can put on constructs) where you probably want a small set of "axioms" from which you cannot deduce contradictions (soundness). A small core set makes it possible to prove that it is consistent. 2. On the other hand you have the conceptual "quasi-theoretical" building blocks from which you can describe everything that is expressible in the language. Keeping this small is also valuable for the same reasons. It is easier to reason about and ensure correctness/soundness and build a language that is easy to grasp. Basically, this often embeds a "modelling idea". (everything should be expressible as an object, a list, a logical proposition or a mix of two core concepts etc). If you look at the Eiffel page they claim that it is a methodology supported by a language. The same was Simula and Beta in a sense, languages supporting modelling with objects. So the minimal core language is often tied to a modelling-idea. 3. Then you have the construct that is actually implemented in the compiler, which reflects the "quasi-theoretical" language, but might be done differently for implementation reasons. Though it should obey the "laws" of the quasi-theoretical language. 4. Then you have the syntax (with syntactical sugar) that allows user to express that. Since it is a massive undertaking to let libraries define new syntax it often is a lot of "syntactical sugar", basically a few words generating many constructs in the underlying "theoretical" language. 5. Then you have the language that programmers use, which basically is what programmers often call "idiomatic" styles of programming for that particular language and that includes frequently used libraries. When newbies write bad code and complain loudly the old timers will point out that they are in essence "abusing the language construct" and "should learn to speak the language before them correctly". So, basically, a programming language design can let the programmer believe that the compiler provides a string concept, although the underlying "theoretical language" only has lists and enumerations. So it strings might look different from lists in the source, but on a "theoretical level" it isn't. What makes this more complicated is that the "true quasi-theoretical" language may emerge over time. As an example: I think the underlying emerging core concept in C++ is vastly different from C, that is, the most important concept in C++ has turned out to be RAII in combination with exceptions. Objects with deterministic constructors and destructors. If one were to do C++ v2, then one probably should start with that and see how small the core language could be made. In practice compilers tend to bolt on poorly thought out "built ins" that do not share the properties of the "theoretical language" and then they stand out as sore thumbs that exhibit behaviour that is perceived as bug-like by programmers. Until recently the "aesthetics" of programming language design has typically been to find the "one true language" that never will have to change. However, languages like typescript pretty much show that is better if programmers can configure the semantics to their specific use case in a config file. Because the context keeps changing. Maybe one can modify a language like D to do the same, so that you could get AA or even data-science libraries imported by default so it feels more like a scripting language with that config. Clearly, embedded programming and data-science programming have different basic needs. Seems reasonable that one could have seemingly builtin AA in one config geared towards scripting-like programming.
Sep 29 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/20 9:38 AM, bachmeier wrote:
 On Tuesday, 29 September 2020 at 12:02:21 UTC, Patrick Schluter wrote:
 
 Your AA example is a good one. If it wasn't in the language, it would 
 be the same issue with AA as in any other language that defines it in 
 libraries: several different incompatible restricted implementations.
That's a good example. IMO there's a difference between a language and bits and pieces that can be put together to call random libraries.
There is no doubt to Walter or myself that the way built-in AAs were done in D is a resounding failure. The way they should have been done was a templated library type AssociativeArray!(K, V) with the V[K] type syntax sugar on top. That would have allowed the implementation to improve along with the rest of the language and its library. (Of course there's historical context to this, i.e. AAs predate templates and then they just continued being there, increasingly standing out like the proverbial sore thumb as everything else evolved around them.) As things are now, AAs don't quite behave like any other things. They are a poor cousin to ranges. No container infrastructure grew around them. They are dramatically less efficient than library hashtables. Worst of all, these problems have been well-known for years and years and the progress on fixing them has moved at a glacial pace.
Sep 29 2020
next sibling parent reply bachmeier <no spam.net> writes:
On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei 
Alexandrescu wrote:
 On 9/29/20 9:38 AM, bachmeier wrote:
 On Tuesday, 29 September 2020 at 12:02:21 UTC, Patrick 
 Schluter wrote:
 
 Your AA example is a good one. If it wasn't in the language, 
 it would be the same issue with AA as in any other language 
 that defines it in libraries: several different incompatible 
 restricted implementations.
That's a good example. IMO there's a difference between a language and bits and pieces that can be put together to call random libraries.
There is no doubt to Walter or myself that the way built-in AAs were done in D is a resounding failure.
If so, it's a pretty good resounding failure, because I like them and they would most likely not be available to me now if they hadn't been built into the language. Long ago when I used Common Lisp, there was the incredible FSet library that provided functional collections kind of like Clojure. Unfortunately few people knew about them or used them, and it doesn't look like anything has been done with them in a long time: https://common-lisp.net/project/fset/ Sometimes a library solution works, but it's in general a risky proposition to rely on libraries for such a fundamental feature. If AAs had been introduced as a library in 2012, the repo would probably have a couple hundred open issues and the last development would have been in 2017.
 The way they should have been done was a templated library type 
 AssociativeArray!(K, V) with the V[K] type syntax sugar on top. 
 That would have allowed the implementation to improve along 
 with the rest of the language and its library.
Does anything prevent the introduction of a new library solution? If it's good enough, it can eventually be added to the language. I'm not going to pretend to be a language designer. My experience as a language user is that a big set of core functionality works better than a small set plus libraries. It's possible I don't understand your proposal.
Sep 29 2020
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Sep 29, 2020 at 06:43:23PM +0000, bachmeier via Digitalmars-d wrote:
 On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:
[...]
 There is no doubt to Walter or myself that the way built-in AAs were
 done in D is a resounding failure.
If so, it's a pretty good resounding failure, because I like them and they would most likely not be available to me now if they hadn't been built into the language. [...]
Same here. Built-in AA's were one of the big reasons that drew me to D. Now granted, the way they are implemented perhaps leaves a lot to be desired, but having built-in support for AA's IMO is a fundamental language feature. Whether it's implemented in the compiler or the standard library is secondary, but it's not something that should be left to an external 3rd party library. It would definitely be a detriment if D had followed the same path as C++ in having hash tables in the standard library (extremely late in the history of C++, I might add -- something I will probably hold forever against C++ :-P), but requiring a lot of manual effort to use: no default hash function for built-in types like structs or ints, need to manually declare all sorts of things just to instantiate a lousy hashtable, having inordinately long derived type names just to name something that in D is as simple as `MyStruct[string]`, and just general needless gratuitous barriers to usage. Some level of language support is necessary to grease all these usability gears: at the very least the language must support nice, easy syntax to declare AA's -- even if the actual implementation is delegated to a library type in the standard library. [...]
 The way they should have been done was a templated library type
 AssociativeArray!(K, V) with the V[K] type syntax sugar on top. That
 would have allowed the implementation to improve along with the rest
 of the language and its library.
Does anything prevent the introduction of a new library solution? If it's good enough, it can eventually be added to the language.
[...] A lot of effort has been put in to move the AA implementation into object.d, and the current implementation has also undergone several rounds of improvement, code quality wise and performance wise. But there remain some magic bits that currently cannot be duplicated in "user space", so to speak, (as opposed to compiler space). One is the handling of const/immutable on keys, which, in spite of being magically handled by the compiler, nevertheless suffers from flaws: to prevent key aliasing problems that cause inconsistencies in the AA, as of several releases ago AA keys are now implicitly `const`. However, this does not actually solve the problem -- because mutable can implicitly convert to const, so the aliasing problem still exists. It would be nice to make it immutable, but that breaks existing code in some places, and in any case switching from const to immutable isn't something that can be done as a gradual deprecation -- it's all or nothing. But regardless of the correctness issue, the handling of const/immutable qualifiers on AA keys depends on compiler magic, and currently either isn't expressible in library code, or else is so cumbersome it's simply impractical. Another is the magic handling of: Data[string][string] aa; aa["abc"]["def"] = Data(...); For this, we need opIndexCreate, which we currently don't have: https://issues.dlang.org/show_bug.cgi?id=7753 There may be one or two other issues that I can't recall off the top of my head, but basically, we've come a long way since the original AA implementation, and now only need to remove a few more magical barriers before a fully-library AA solution is possible in druntime. // Of course, all of this confirms what Andrei is saying about too much magic being boiled into the compiler: it can do stuff library code cannot, and because of that, fixing problems in AAs requires digging deep into compiler innards. Worse yet, some of the rules governing AA magic in the compiler isn't fully compatible with the rules library code is subjected to; and trying to replicate this behaviour in library code is very painful, or outright impossible. T -- Doubt is a self-fulfilling prophecy.
Sep 29 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/29/20 2:43 PM, bachmeier wrote:
 On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:
 There is no doubt to Walter or myself that the way built-in AAs were 
 done in D is a resounding failure.
If so, it's a pretty good resounding failure, because I like them and they would most likely not be available to me now if they hadn't been built into the language. Long ago when I used Common Lisp, there was the incredible FSet library that provided functional collections kind of like Clojure. Unfortunately few people knew about them or used them, and it doesn't look like anything has been done with them in a long time: https://common-lisp.net/project/fset/ Sometimes a library solution works, but it's in general a risky proposition to rely on libraries for such a fundamental feature. If AAs had been introduced as a library in 2012, the repo would probably have a couple hundred open issues and the last development would have been in 2017.
 The way they should have been done was a templated library type 
 AssociativeArray!(K, V) with the V[K] type syntax sugar on top. That 
 would have allowed the implementation to improve along with the rest 
 of the language and its library.
Does anything prevent the introduction of a new library solution? If it's good enough, it can eventually be added to the language. I'm not going to pretend to be a language designer. My experience as a language user is that a big set of core functionality works better than a small set plus libraries. It's possible I don't understand your proposal.
I think there's some disconnect here -- AAs as a language feature are a resounding success. They way they are implemented in the language is not. I wouldn't call it a failure, but it definitely is not as useful as a fully supported library type could be. -Steve
Sep 29 2020
parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/29/2020 12:30 PM, Steven Schveighoffer wrote:
 I think there's some disconnect here -- AAs as a language feature are a 
 resounding success. They way they are implemented in the language is not. I 
 wouldn't call it a failure, but it definitely is not as useful as a fully 
 supported library type could be.
Yes, that is a better statement of my opinion on it. The implementation's problems are, of course, because D's AAs were designed before there were templates and before there was const and immutable.
Sep 30 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei 
Alexandrescu wrote:
 As things are now, AAs don't quite behave like any other things.
The biggest difference I see is how null works, the keyword. void foo(string[string] aa) {} foo(null); // works. But that's outright impossible with a library struct. yes, I'm bringing up my wish for implicit construction again :P
Sep 29 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/29/20 3:32 PM, Adam D. Ruppe wrote:
 On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:
 As things are now, AAs don't quite behave like any other things.
The biggest difference I see is how null works, the keyword. void foo(string[string] aa) {} foo(null); // works. But that's outright impossible with a library struct. yes, I'm bringing up my wish for implicit construction again :P
This is certainly an odd duck. I don't really want to have the language provide all-out implicit construction, but possibly a mechanism to say "I can be implicitly constructed from null" would be useful. null is in its own category of literals, it can change into just about anything (class, pointer, array associative array). Another nice place this would be useful is std.typecons.Nullable. -Steve
Sep 29 2020
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/20 8:02 AM, Patrick Schluter wrote:
 On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei Alexandrescu wrote:
 On 9/28/20 6:46 PM, Bruce Carneal wrote:
[..]
 Years ago, I was in a panel with Simon Peyton-Jones, the creator of 
 Haskell. Nicest guy around, not to mention an amazing researcher. This 
 question was asked: "What is the most important principle of 
 programming language design?"

 His answer was so powerful, it made me literally forget everybody 
 else's answer, including my own. (And there were no slouches in that 
 panel, save for me: Martin Odersky, Guido van Rossum, Bertrand Meyer.) 
 He said the following:

 "The most important principle in a language design is to define a 
 small core of essential primitives. All necessary syntactic sugar 
 lowers to core constructs. Do everything else in libraries."
Sorry to not be completely convinced on the pertinence of that statement. For the language designer it might well be true, for the users of the language I'm not completely sold. There are two languages that imho fit that description very well, they are both very powerfull and extremely elegant, but boy do they suck when you just want to get shit done: Lisp and Forth. And (I know, one should not start a sentence with and) it had been mentioned often in this forum, it is this shear flexibility that are in the way of wider adoption as there are so many ways of doing things, but none of them completely right. Your AA example is a good one. If it wasn't in the language, it would be the same issue with AA as in any other language that defines it in libraries: several different incompatible restricted implementations. Just my uninformed 2 cents.
Completely agree with the sentiment! Also, being informed in language internals paraphernalia is not at stake here. I've studied classical guitar for many years. Got a couple of prizes, too, but never broke out of the amateur ranks. Nevertheless, I got to meet a few professional players and also played to a variety of audiences. One thing that's obvious to a music performer that is not at all clear to listeners is that deficiencies in performance show up as second-order effects. What do I mean by that? I mean that the audience doesn't sit there with the sheet music in hand following the notes and the directions written. They won't put the finger on paper and say, hey, hey, what happened to the mezzoforte in here? No, they just sit back and listen to the music. If they like it, they don't know why. If they don't, they don't know why. Music critics write all about some performer's technique, touché, style, rubato - but listeners are like, yeah, that's beautiful or isn't. An exegete knows that that came out poorly because the performer moved their wrist too much. No bug report will come as "D uses too many tricks in the compiler that are inaccessible to programs, and that's very problematic." No, they'll all come as problems with surprising limitations and inconsistencies, frustration in use, platform dependencies, deficiencies in implementation, lack of orthogonality, very concrete and very many bugs, bugs, and bugs. Deficiencies in programming language design show up as second-order effects. There's also something about limits that needs clarification. Small language kernel being good does not mean the smallest language kernel is the best. (In guitar technique economy of motion is a key ideal, i.e. one should minimize movement of both left and right hand. That of course doesn't mean zero movement is best, because that would produce no sound.) So keeping the core small and reducing sugar to it is an ideal to live into, not a rigid requirement.
Sep 29 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:
I'd rather live without that DIP.
 So I'm interested in exploring reification mainly - 
 overwhelmingly - because it already works in the existing 
 language. To the extent it has difficulties, it's because of 
 undue limitations in CTFE and introspection. And these are 
 problems we must fix anyway. So it's all a virtuous circle. 
 Bringing myself to write `reify` and `dereify` around my types 
 is a very small price to pay for all that.
Alright mate, Let's assume we had the minimal langauge. Very small core. All features are library. You end up with LISP. What's with optimization? Optimization, just in case you don't know, is nothing but a semantic invariant transform, which is based on proving that observed semantics are actually invariant. Let's take: int mul( int x, int y ) { int result = 0; for(int i = 0; i < x; i++) { result += y; } return result; } A sufficiently advanced optimizer should be able to most every calls to `mul` into an imul instruction. If that is the case there is no need to burden the language with a * operator and you could make the parser much simpler. and Indeed ldc with -O3 will generate the following sequence for the function: imull %esi, %edi // y *= x xorl %eax, %eax // result = 0 testl %esi, %esi // if (y) cmovgl %edi, %eax // if (x > 0) result = x; retq // return result; There are a few superfluous checks, for example the loop entrance condition (x > 0) So how does this compare if we write a function that only does * ? int mul( int x, int y ) { return x*y; } The code we get, from the same compiler and the same compiler flags (-O3) is this: movl %edi, %eax // result = x; imull %esi, %eax // result *= y; retq // return result; Which removes one test, and one xor, furthermore one move becomes unconditional. (That reduces code size and the load on the branch predictor.)
Sep 29 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Tuesday, 29 September 2020 at 12:38:42 UTC, Stefan Koch wrote:
 [continuing]
So we had this:
 int mul(
   int x,
   int y
 ) {
   int result = 0;
   for(int i = 0; i < x; i++)
   {
     result += y;
   }
   return result;
 }
And the code generated was imull %esi, %edi // y *= x xorl %eax, %eax // result = 0 testl %esi, %esi // if (y) cmovgl %edi, %eax // if (x > 0) result = x; retq // return result; On first glance these instructions seem superfluous But we have to understand that the compiler had no way of knowing that we actually wanted to generate a mere multilply. For example there is the loop entrance check. Why is it there? We can proof it's necessity quite easily. - let's assume we did not have it. In that case we are allowd to enter loop body even if (x == 0) so we would execute result += y; once. Which would cause mul(0, 32) to result in 32. And that would be a violation of the semantics. So there's no way to get rid of the check. And this code will always perform worse than an intrinsic 'mul' which the compiler knows about.
Sep 29 2020
parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 29 September 2020 at 12:54:53 UTC, Stefan Koch wrote:
 So we had this:

 int mul(
   int x,
   int y
 ) {
   int result = 0;
   for(int i = 0; i < x; i++)
   {
     result += y;
   }
   return result;
 }
And the code generated was imull %esi, %edi // y *= x xorl %eax, %eax // result = 0 testl %esi, %esi // if (y) cmovgl %edi, %eax // if (x > 0) result = x; retq // return result; On first glance these instructions seem superfluous But we have to understand that the compiler had no way of knowing that we actually wanted to generate a mere multilply. For example there is the loop entrance check. Why is it there? We can proof it's necessity quite easily. [snip]
I don't know assembly, but I can see an even simpler reason why the compiler can't generate a multiply instruction. It's `mul(-5, 5)*`.
Sep 29 2020
parent Stefan Koch <uplink.coder gmail.com> writes:
On Tuesday, 29 September 2020 at 14:42:42 UTC, Dukc wrote:
 On Tuesday, 29 September 2020 at 12:54:53 UTC, Stefan Koch 
 wrote:
 So we had this:

 [...]
I don't know assembly, but I can see an even simpler reason why the compiler can't generate a multiply instruction. It's `mul(-5, 5)*`.
Which is another reason why it's harder to state the problem in a lower form. I wrote the example wanting to show two semantically equivalent things and I did not.
Sep 29 2020
prev sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:
 On 9/28/20 6:46 PM, Bruce Carneal wrote:
 [...]
Years ago, I was in a panel with Simon Peyton-Jones, the creator of Haskell. Nicest guy around, not to mention an amazing researcher. This question was asked: "What is the most important principle of programming language design?" His answer was so powerful, it made me literally forget everybody else's answer, including my own. (And there were no slouches in that panel, save for me: Martin Odersky, Guido van Rossum, Bertrand Meyer.) He said the following: "The most important principle in a language design is to define a small core of essential primitives. All necessary syntactic sugar lowers to core constructs. Do everything else in libraries." (He didn't use the actual term "lowering", which is familiar to our community, but rather something equivalent such as "reduces".) That kind of killed the panel, in a good way. Because most other questions on programming language design and implementation simply made his point shine quietly. Oh yes, if you had a small core and build on it you could do this easily. And that. And the other. In a wonderfully meta way, all questions got instantly lowered to simpler versions of themselves. I will never forget that experience. D breaks that principle in several places. It has had a cavalier attitude to using magic tricks in the compiler to get things going, at the expense of fuzzy corners and odd limit cases. Look at hashtables. Nobody can create an equivalent user-defined type, and worse, nobody knows exactly why. (I recall vaguely it has something to do, among many other things, with qualified keys that are statically-sized arrays. Hacks in the compiler make those work, but D's own type system rejects the equivalent code. So quite literally D's type system cannot verify its own capabilities.) Or take implicit conversions. They aren't fully documented, and the only way to figure things out is to read most of the 7193 lines of https://github.com/dlang/dmd/blob/master/src/dmd/mtype.d. That's not a small core with a little sugar on top. Or take the foreach statement. Through painful trial and error, somebody figured out all possible shapes of foreach, and defined `each` to support most: https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L904 What should have been a simple forwarding problem took 190 lines that could be characterized as very involved. And mind you, it doesn't capture all cases because per https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L1073: // opApply with >2 parameters. count the delegate args. // only works if it is not templated (otherwise we cannot count the args) I know that stuff (which will probably end on my forehead) because I went and attempted to improve things a bit in https://github.com/dlang/phobos/pull/7638/files#diff-0d6463fc6f41c5fb7 4300832e3135f5R805, which attempts to simplify matters by reducing the foreach cases to seven shapes. To paraphrase Alan Perlis: "If your foreach has seven possible shapes, you probably missed some." Over time, things did get better. We became adept of things such as lowering, and we require precision in DIPs. I very strongly believe that this complexity, if unchecked, will kill the D language. It will sink under its own weight. We need a precise, clear definition of the language, we need a principled approach to defining and extending features. The only right way to extend D at this point is to remove the odd failure modes created by said cavalier approach to doing things. Why can't I access typeid during compilation, when I can access other pointers? Turns out typeid is a palimpsest that has been written over so many times, even Walter cannot understand it. He told me plainly to forget about trying to use typeid and write equivalent code from scratch instead. That code has achieved lifetime employment. And no compiler tricks. I am very, very opposed to Walter's penchant to reach into his bag of tricks whenever a difficult problem comes about. Stop messing with the language! My discussions with him on such matters are like trying to talk a gambler out of the casino. That is a long way to say I am overwhelmingly in favor of in-language solutions as opposed to yet another addition to the language. To me, if I have an in-language solution, it's game, set, and match. No need for language changes, thank you very much. An in-language solution doesn't only mean no addition is needed, but more importantly it means that the language has sufficient power to offer D coders means to solve that and many other problems. I almost skipped a heartbeat when Adam mentioned - mid-sentence! - just yet another little language addition to go with that DIP for interpolated strings. It would make things nicer. You know what? I think I'd rather live without that DIP. So I'm interested in exploring reification mainly - overwhelmingly - because it already works in the existing language. To the extent it has difficulties, it's because of undue limitations in CTFE and introspection. And these are problems we must fix anyway. So it's all a virtuous circle. Bringing myself to write `reify` and `dereify` around my types is a very small price to pay for all that.
My thanks for the effort apparent in your composition. The main lesson I take from it is that language primitives should be chosen using as much light as we can bring to bear. I believe that language additions can be used to rebase the language on a better set of primitives. I believe that there is more to learn about meta programming and the language features that support it and that the D platform may be the best place to learn. Finally, I believe that a postulated ideal of "a small set of primitives" will always be better than any actual set derived from experience on the frontier. Perhaps because of your strong stand against additions generally, you've not taken a serious look at type functions in particular. I say that you've not taken a serious look because I don't have another way to understand your writings on the reify/dereify design. If you elect to investigate type functions, I would very much like to hear your opinion. I see type functions as an "it just works", "rebasing" addition that lets us displace inappropriate use of templates now, and that may illuminate additional simplification opportunities with use. If you do not elect to investigate type functions, I expect we'll next interact on a different topic. Again, my thanks for the thoughtful composition.
Sep 29 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/28/2020 3:46 PM, Bruce Carneal wrote:
 Finally, I'd love to hear your comments on type functions vs reify/dereify.  
 It's certainly possible that I've missed something.  Maybe it's a type 
 functions+ solution we should be seeking (type functions for everything they
can 
 do, and some LDM for anything beyond their capability).
Andrei and I have been mulling over what to do for a while, and have gone through several ideas. What we're looking for is not a top level solution. We want a fundamental building block, upon which the rest of D's features can exploit naturally and productively. Something that can be explained in 30 seconds. An example of this is the recent PR for __totype(string), which is the complement to .mangleof. It's main shortcoming is it only works with types. I'm currently working on an idea to make that work for any alias. It's so stupidly simple I'm afraid I'm missing something terribly obvious, so I beg your indulgence while exploring it. Stay tuned!
Sep 30 2020
next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 1 October 2020 at 04:08:26 UTC, Walter Bright wrote:
 On 9/28/2020 3:46 PM, Bruce Carneal wrote:
 [...]
Andrei and I have been mulling over what to do for a while, and have gone through several ideas. What we're looking for is not a top level solution. We want a fundamental building block, upon which the rest of D's features can exploit naturally and productively. Something that can be explained in 30 seconds. An example of this is the recent PR for __totype(string), which is the complement to .mangleof. It's main shortcoming is it only works with types. I'm currently working on an idea to make that work for any alias. It's so stupidly simple I'm afraid I'm missing something terribly obvious, so I beg your indulgence while exploring it. Stay tuned!
Sounds exciting! Good luck!
Sep 30 2020
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 1 October 2020 at 04:08:26 UTC, Walter Bright wrote:
 On 9/28/2020 3:46 PM, Bruce Carneal wrote:
 Finally, I'd love to hear your comments on type functions vs 
 reify/dereify.  It's certainly possible that I've missed 
 something.  Maybe it's a type functions+ solution we should be 
 seeking (type functions for everything they can do, and some 
 LDM for anything beyond their capability).
Andrei and I have been mulling over what to do for a while, and have gone through several ideas. What we're looking for is not a top level solution. We want a fundamental building block, upon which the rest of D's features can exploit naturally and productively. Something that can be explained in 30 seconds.
Type functions take less than 30 seconds. They take zero seconds. Do you remember Andrei's non working example?
 alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());
Here is the correct type function which actually works! alias[] MostDerived(alias[] types ...) { sort!((alias a, alias b) => is(a : b))(types); return types; } It is Andrei thought would work. With type functions it just works.
Sep 30 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 10/1/20 2:51 AM, Stefan Koch wrote:
 On Thursday, 1 October 2020 at 04:08:26 UTC, Walter Bright wrote:
 On 9/28/2020 3:46 PM, Bruce Carneal wrote:
 Finally, I'd love to hear your comments on type functions vs 
 reify/dereify.  It's certainly possible that I've missed something.  
 Maybe it's a type functions+ solution we should be seeking (type 
 functions for everything they can do, and some LDM for anything 
 beyond their capability).
Andrei and I have been mulling over what to do for a while, and have gone through several ideas. What we're looking for is not a top level solution. We want a fundamental building block, upon which the rest of D's features can exploit naturally and productively. Something that can be explained in 30 seconds.
Type functions take less than 30 seconds. They take zero seconds. Do you remember Andrei's non working example?
 alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());
Here is the correct type function which actually works! alias[] MostDerived(alias[] types ...) {     sort!((alias a, alias b) => is(a : b))(types);     return types; } It is Andrei thought would work. With type functions it just works.
From https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213: alias DerivedToFront(Args...) = dereify!({ auto ids = reifyArray!Args; ids.sort; return ids; }()); That code defines opCmp such that a < b iff a is a subtype of b, a nice touch reminiscent of the <: operator used in type theory. To implement that, the reified type stores the bases (conversion targets) of the type at construction time. This is easy to generalize to numeric and array types, and some more work to get going for things like covariant functions. One litmus test is to redo Variant "the right way" (btw it should be called Any and moved to druntime). Currently Variant reifies the type into a pointer to function, and uses that pointer to function to dispatch type-dependent work. It is quite messy and incorrect in places. "The right way" would be for the reified type to have enough information to allow things like testing for subtyping/convertibility. Currently Variant is essentially incomplete and incorrect (see https://github.com/dlang/phobos/blob/master/std/variant.d#L285) because it builds on the shaky ground of https://github.com/dlang/phobos/blob/master/std/traits.d#L5027, a sort of a best-effort approach to figuring out the definition of implicit conversions scattered across the language definition (if documented in entirety at all). Having a clear definition of what implicitly converts to what would be a nice perk. One interesting thing about Variant's primitives is that information is half-and-half: one half is present at compile-time, i.e. "Can I read a value of type T?" and the other half is present at runtime, in the reified format stored in the Variant. Given that the latter is dynamic, there's no way around reifying T as well and then using the reified types for testing.
Oct 01 2020
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 1 October 2020 at 15:21:40 UTC, Andrei Alexandrescu 
wrote:
...
 One litmus test is to redo Variant "the right way" (btw it 
 should be called Any and moved to druntime). Currently Variant 
 reifies the type into a pointer to function, and uses that 
 pointer to function to dispatch type-dependent work. It is 
 quite messy and incorrect in places. "The right way" would be 
 for the reified type to have enough information to allow things 
 like testing for subtyping/convertibility. Currently Variant is 
 essentially incomplete and incorrect ...
Litmus test for what? Utility at run-time or as a simple base language addition for use at compile time? My concern here is not with our ability to make something like what you're proposing "work". After all, C++ "works". My concern is that the proposal, by my lights anyway, is already a good deal past "simple" and it's not converging. On a related note, if the reify/dereify code expands as much as I believe it will in order to cover all cases, expands in to a "shadow" front end, you'll want to look at forcing the actual front end to use it so that you can eliminate divergence issues. That could also set us up for an embedded-compiler/jit future which would be another justification for the work. If that's where you're headed, if you want to destroy the compile-time/run-time separation, I'd be interested in hearing more. I'm not a fan of embedding the compiler or exposing the type system for that matter, but it would be worth a listen, no doubt.
Oct 01 2020
parent Mark <smarksc gmail.com> writes:
On Thursday, 1 October 2020 at 18:04:09 UTC, Bruce Carneal wrote:
 If that's where you're headed, if you want to destroy the 
 compile-time/run-time separation, I'd be interested in hearing 
 more.  I'm not a fan of embedding the compiler or exposing the 
 type system for that matter, but it would be worth a listen, no 
 doubt.
worth looking into.
Oct 09 2020
prev sibling parent claptrap <clap trap.com> writes:
On Thursday, 1 October 2020 at 15:21:40 UTC, Andrei Alexandrescu 
wrote:
 On 10/1/20 2:51 AM, Stefan Koch wrote:
 On Thursday, 1 October 2020 at 04:08:26 UTC, Walter Bright 
 wrote:

 Here is the correct type function which actually works!
 
 alias[] MostDerived(alias[] types ...)
 {
      sort!((alias a, alias b) => is(a : b))(types);
      return types;
 }
 
 It is Andrei thought would work.
 With type functions it just works.
From https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213: alias DerivedToFront(Args...) = dereify!({ auto ids = reifyArray!Args; ids.sort; return ids; }());
Try explaining that to a newbie who's never used D meta programming before. It's a function that sorts a list of types from most to least derived. You do it with an alias template. Here we've use the shorthand for an eponymous alias template. So it sort of looks like a regular function call but you need an equals sign after the template parentheses. And the right had side needs to be an expression because its an alias declaration not a function declaration. Oh and you need to wrap the right had side in a call to dereify(), you dont need to know why for now, you just do. And you need to wrap the code you want to execute in an anonymous lamba that is called immediately. And before you can do anything with the args passed to the template you need to call reify() on them. I mean seriously? 30 seconds with a straight face? (Im not even sure ive described it correctly) vs a type function... It's a function that sorts a list of types from most to least derived. You write it just like a regular function but with types.
Oct 01 2020
prev sibling next sibling parent claptrap <clap trap.com> writes:
On Monday, 28 September 2020 at 02:58:13 UTC, Bruce Carneal wrote:
 On Sunday, 27 September 2020 at 16:32:18 UTC, claptrap wrote:

 I hope and believe that there is another such advance available 
 to us in the form of type functions.  The recently discovered 
 alternative, reify/dereify, appears to be equivalent in power 
 but is, comparatively, baroque.

 If the two approaches are actually equivalent in power, and in 
 their ability to address the current template issues, then the 
 baroque should be preferred if the language is closed to 
 syntactic additions.

 If the language is not closed, then choosing to displace the 
 already prototyped type function capability with reify/dereify 
 would represent a lost opportunity to administer another 
 complexity smackdown.
Yeah I think A&W etc.. are more concerned about keeping compiler complexity down than they are about a clean intuitive language interface.
Sep 28 2020
prev sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 28 September 2020 at 02:58:13 UTC, Bruce Carneal wrote:
 I hope and believe that there is another such advance available 
 to us in the form of type functions.  The recently discovered 
 alternative, reify/dereify, appears to be equivalent in power 
 but is, comparatively, baroque.
Having been following Stefan's work on type functions for some time now, and really liking the friendly syntax that it allows, I would really appreciate a more detailed breakdown of the pros and cons when compared to this approach here. As I understand it neither requires very significant compiler changes, and both allow us to rewrite existing Phobos templates in ways that greatly improve performance. Andrei would you be happy to share your thoughts in more detail on all this, and how you feel we should balance out the different concerns and interests?
Sep 30 2020
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/26/2020 9:18 AM, Andrei Alexandrescu wrote:
 [...]
It's too easy. You must have done it all wrong!
Sep 26 2020
prev sibling next sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:
 Commments and ideas for improvements are welcome.
Great idea. 1. I presume we need to benchmark this both in space and time to judge its merits. Have you done any such? How will the number of templates instances change with length of array for each algorithm in std.meta? 2. To reduce (all) template bloat in your new version of std.meta, what about adding a special handling in the compiler for `reify` and `dereify` like was done with `AliasSeq`?
Sep 27 2020
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:
 I'll start with the punchline: at the link below you'll find an 
 alpha-quality reimplementation of std.meta in an iterative 
 manner:

 https://gist.github.com/andralex/6212ebf2b38c59c96cf66d0009336318
This is really cool. It looks like if we add just a tiny bit of compiler support (like Walter's PR) we could make this efficient. And we're just using the power of CTFE to do it. Sweet.
Sep 27 2020
prev sibling next sibling parent Mike <slavo5150 yahoo.com> writes:
On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:

 Commments and ideas for improvements are welcome.
If you're going to go through the trouble of re-implementing std.meta, please move it to druntime so it can be leveraged by the DMD frontend and the druntime implementation, among other benefits. std.meta can then forward to the druntime implementations for backward compatibility.
Sep 28 2020
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:
 I'll start with the punchline: at the link below you'll find an 
 alpha-quality reimplementation of std.meta in an iterative 
 manner:

 https://gist.github.com/andralex/6212ebf2b38c59c96cf66d0009336318

 It's the bulk of std.meta - I stopped when it became obvious 
 doing more is just more of the same. Compiles and runs with dmd 
 2.092 and probably others.

 The implementation uses throughout a reification/dereification 
 approach. To reify a type or alias means to convert it into a 
 value. To dereify a value means to turn it back into the type 
 or alias it originated from.

 The algorithms follow the following pattern consistently:

 1. Reify the compile-time parameters into values
 2. Carry processing on these values with the usual algorithms
 3. If needed, dereify back the results into compile-time 
 parameters

 Not all artifacts need step 3. For example, allSatisfy returns 
 a bool, not a type.

 For the most part implementation has been a healthy gallop that 
 took a couple of hours yesterday and a couple of hours today. 
 Its weakest point is it uses .stringof for types, which has 
 issues with local types. Hopefully 
 https://github.com/dlang/dmd/pull/11797 could help with that.

 Most implementations are a few lines long because they get to 
 leverage algorithms as implemented in std. Here's a typical one:

 alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());

 To sort an AliasSeq with most derived elements first, create a 
 lambda to reify the arguments into values, sort the values, and 
 return the sorted result. Then call the lambda and use dereify 
 against its result. And that is IT.

 Commments and ideas for improvements are welcome.
1. This is awesome. 2. I agree with everything said about keeping the core language simple and lowering to fundamental constructs. I agree with that in general, and it's a lot easier to agree with it in the context of D given that it's a large language already. Lisp got mentioned in this thread as if it's some kind of failure, and for the life for me I have no idea why. Users: dmd has too many bugs! Also users: please add my bugs, err, feature!
Oct 08 2020
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 8 October 2020 at 09:51:15 UTC, Atila Neves wrote:
 1. This is awesome.
 2. I agree with everything said about keeping the core language 
 simple and lowering to fundamental constructs. I agree with 
 that in general, and it's a lot easier to agree with it in the 
 context of D given that it's a large language already.

 Lisp got mentioned in this thread as if it's some kind of 
 failure, and for the life for me I have no idea why.

 Users: dmd has too many bugs!
 Also users: please add my bugs, err, feature!
--- alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());
--- Note that the example I re-posted above doesn't actually work. Because of polymorphism.
Oct 08 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/8/20 7:06 AM, Stefan Koch wrote:
 On Thursday, 8 October 2020 at 09:51:15 UTC, Atila Neves wrote:
 1. This is awesome.
 2. I agree with everything said about keeping the core language simple 
 and lowering to fundamental constructs. I agree with that in general, 
 and it's a lot easier to agree with it in the context of D given that 
 it's a large language already.

 Lisp got mentioned in this thread as if it's some kind of failure, and 
 for the life for me I have no idea why.

 Users: dmd has too many bugs!
 Also users: please add my bugs, err, feature!
--- alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());
--- Note that the example I re-posted above doesn't actually work. Because of polymorphism.
The following works: alias DerivedToFront(Args...) = dereify!({ auto ids = reifyArray!Args; ids.sort; return ids; }()); https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213
Oct 08 2020
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 8 October 2020 at 12:05:22 UTC, Andrei Alexandrescu 
wrote:
 The following works:

 alias DerivedToFront(Args...) = dereify!({
     auto ids = reifyArray!Args;
     ids.sort;
     return ids;
 }());

 https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213
Only with the ID struct that defines the op-compare. Which is significantly more complicated that the custom sort predicate. And also more inflexible.
Oct 08 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/8/20 8:16 AM, Stefan Koch wrote:
 On Thursday, 8 October 2020 at 12:05:22 UTC, Andrei Alexandrescu wrote:
 The following works:

 alias DerivedToFront(Args...) = dereify!({
     auto ids = reifyArray!Args;
     ids.sort;
     return ids;
 }());

 https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213
Only with the ID struct that defines the op-compare. Which is significantly more complicated that the custom sort predicate. And also more inflexible.
Less flexible during compilation, more flexible at runtime. It is quite satisfying that the same information in reified form (strings I think in that gist) is equally usable during compilation, at run-time, or in mixed scenarios (Variant). I wonder what other information would be useful to persist through runtime aside from the trivial (size, alignment) and the more involved (enough information to decide conversions). For functions, I assume that would be the returned type and the parameter types along with all adornments (ref, out, all that stuff). For classes, interfaces, and structs, that would include all public methods - probably on demand by means of an rtti attribute so as to not waste time and space.
Oct 08 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 8 October 2020 at 13:12:22 UTC, Andrei Alexandrescu 
wrote:
 Less flexible during compilation, more flexible at runtime.
With my little lib that uses string lambdas for templates: pragma(msg, reified!q{ // string template lambda a.sizeof == 4 }.run!(types => types.filter!(a => a) // normal std.algorithm! ).over!(string, int, float, Object));
Oct 08 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 8 October 2020 at 13:19:48 UTC, Adam D. Ruppe wrote:
 On Thursday, 8 October 2020 at 13:12:22 UTC, Andrei 
 Alexandrescu wrote:
 Less flexible during compilation, more flexible at runtime.
With my little lib that uses string lambdas for templates: pragma(msg, reified!q{ // string template lambda a.sizeof == 4 }.run!(types => types.filter!(a => a) // normal std.algorithm! ).over!(string, int, float, Object));
please post the source of your lib. Benchmark is incoming.
Oct 08 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 8 October 2020 at 13:27:50 UTC, Stefan Koch wrote:
 please post the source of your lib.
it is over here https://forum.dlang.org/post/zhaziudnbaoybmfdrzxn forum.dlang.org
 Benchmark is incoming.
I don't expect it to win, this is more to demo existing language api style possibilities. This style lost to existing Phobos std.meta when I tried.
Oct 08 2020
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 8 October 2020 at 13:19:48 UTC, Adam D. Ruppe wrote:
         a.sizeof == 4
Of course I probably should have written it: pragma(msg, reified!q{ a.sizeof // just get the size }.run!(types => types.filter!(a => a == 4) // and compare it down here ).over!(string, int, float, Object));
Oct 08 2020