www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - TypeFunction example creatiing a conversion matrix

reply Stefan Koch <uplink.coder gmail.com> writes:
Hi People,

To further show the intuitive type function syntax I just created 
a way to print a conversion matrix.

Here is the code:

string makeConvMatrix(alias[] types ...)
{
     string result;
     foreach(t;types)
     {
         result ~= "\t" ~ t.stringof;
     }
     result ~= "\n";
     foreach(t1;types)
     {
         result ~= t1.stringof;
         foreach(t2;types)
         {
             result ~=  "\t" ~ (is(t1:t2) ? "yes" : "no");
         }
         result ~= "\n";
     }
     return result;
}

alias Byte = byte;
alias Ubyte = ubyte;
alias Short = short;
alias Ushort = ushort;
alias Int = int;
alias Uint = uint;
alias Long = long;
alias Ulong = ulong;

void main()
{
     import std.stdio;
     static immutable convMatrix = makeConvMatrix(Byte, Ubyte, 
Short, Ushort, Int, Uint, Long, Ulong);

     printf("%s\n", convMatrix.ptr);
}

And here is the output:

----
  	byte 	ubyte 	short 	ushort 	int 	uint 	long 	ulong
byte	yes 	yes 	yes 	yes 	yes 	yes 	yes 	yes
ubyte	yes 	yes 	yes 	yes 	yes 	yes 	yes 	yes
short	no 	no 	yes 	yes 	yes 	yes 	yes 	yes
ushort	no 	no 	yes 	yes 	yes 	yes 	yes 	yes
int	no 	no 	no 	no 	yes 	yes 	yes 	yes
uint	no 	no 	no 	no 	yes 	yes 	yes 	yes
long	no 	no 	no 	no 	no 	no 	yes 	yes
ulong	no 	no 	no 	no 	no 	no 	yes 	yes

--
Oct 01 2020
next sibling parent user1234 <user1234 12.de> writes:
On Thursday, 1 October 2020 at 08:21:24 UTC, Stefan Koch wrote:

 Hi People,

 To further show the intuitive type function syntax I just 
 created a way to print a conversion matrix.

 Here is the code:

 string makeConvMatrix(alias[] types ...)
 {
     string result;
     foreach(t;types)
     {
         result ~= "\t" ~ t.stringof;
     }
     result ~= "\n";
     foreach(t1;types)
     {
         result ~= t1.stringof;
         foreach(t2;types)
         {
             result ~=  "\t" ~ (is(t1:t2) ? "yes" : "no");
         }
         result ~= "\n";
     }
     return result;
 }

 alias Byte = byte;
 alias Ubyte = ubyte;
 alias Short = short;
 alias Ushort = ushort;
 alias Int = int;
 alias Uint = uint;
 alias Long = long;
 alias Ulong = ulong;

 void main()
 {
     import std.stdio;
     static immutable convMatrix = makeConvMatrix(Byte, Ubyte, 
 Short, Ushort, Int, Uint, Long, Ulong);

     printf("%s\n", convMatrix.ptr);
 }

 And here is the output:

 ----
  	byte 	ubyte 	short 	ushort 	int 	uint 	long 	ulong
 byte	yes 	yes 	yes 	yes 	yes 	yes 	yes 	yes
 ubyte	yes 	yes 	yes 	yes 	yes 	yes 	yes 	yes
 short	no 	no 	yes 	yes 	yes 	yes 	yes 	yes
 ushort	no 	no 	yes 	yes 	yes 	yes 	yes 	yes
 int	no 	no 	no 	no 	yes 	yes 	yes 	yes
 uint	no 	no 	no 	no 	yes 	yes 	yes 	yes
 long	no 	no 	no 	no 	no 	no 	yes 	yes
 ulong	no 	no 	no 	no 	no 	no 	yes 	yes

 --


nice.
Oct 01 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 08:21:24 UTC, Stefan Koch wrote:


 string makeConvMatrix(alias[] types ...)


Wait .... What the Hell?
Why did this compile?

I must have fixed parsing `alias[]` ?
Oct 01 2020
parent reply user1234 <user1234 12.de> writes:
On Thursday, 1 October 2020 at 08:26:53 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 08:21:24 UTC, Stefan Koch wrote:


 string makeConvMatrix(alias[] types ...)


 Wait .... What the Hell?
 Why did this compile?

 I must have fixed parsing `alias[]` ?


By the way why cant you pass directly the types (i.e lowercase 
builtins) ?
Oct 01 2020
next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 08:38:43 UTC, user1234 wrote:

 On Thursday, 1 October 2020 at 08:26:53 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 08:21:24 UTC, Stefan Koch wrote:


 string makeConvMatrix(alias[] types ...)


 Wait .... What the Hell?
 Why did this compile?

 I must have fixed parsing `alias[]` ?


 By the way why cant you pass directly the types (i.e lowercase 
 builtins) ?


parser limitations.

The grammar won't allow you to do that.
Oct 01 2020
parent user1234 <user1234 12.de> writes:
On Thursday, 1 October 2020 at 08:52:46 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 08:38:43 UTC, user1234 wrote:

 On Thursday, 1 October 2020 at 08:26:53 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 08:21:24 UTC, Stefan Koch 
 wrote:


 string makeConvMatrix(alias[] types ...)


 Wait .... What the Hell?
 Why did this compile?

 I must have fixed parsing `alias[]` ?


 By the way why cant you pass directly the types (i.e lowercase 
 builtins) ?


 parser limitations.

 The grammar won't allow you to do that.


Ah indeed. Eventually you could imagine a syntax that makes the 
parser take a special path but I suppose that for now  this is 
just a detail.
Oct 01 2020
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 1 October 2020 at 08:38:43 UTC, user1234 wrote:

 On Thursday, 1 October 2020 at 08:26:53 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 08:21:24 UTC, Stefan Koch wrote:


 string makeConvMatrix(alias[] types ...)


 Wait .... What the Hell?
 Why did this compile?

 I must have fixed parsing `alias[]` ?


 By the way why cant you pass directly the types (i.e lowercase 
 builtins) ?


Is this something that could be changed theoretically (i.e. with 
a DIP)?
Oct 01 2020
parent Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 11:22:12 UTC, jmh530 wrote:

 On Thursday, 1 October 2020 at 08:38:43 UTC, user1234 wrote:

 On Thursday, 1 October 2020 at 08:26:53 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 08:21:24 UTC, Stefan Koch 
 wrote:


 string makeConvMatrix(alias[] types ...)


 Wait .... What the Hell?
 Why did this compile?

 I must have fixed parsing `alias[]` ?


 By the way why cant you pass directly the types (i.e lowercase 
 builtins) ?


 Is this something that could be changed theoretically (i.e. 
 with a DIP)?


Yes indeed.
Oct 01 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 08:21:24 UTC, Stefan Koch wrote:

 string makeConvMatrix(alias[] types ...)
 {
     string result;
     foreach(t;types)
     {
         result ~= "\t" ~ t.stringof;
     }
     result ~= "\n";
     foreach(t1;types)
     {
         result ~= t1.stringof;
         foreach(t2;types)
         {
             result ~=  "\t" ~ (is(t1:t2) ? "yes" : "no");
         }
         result ~= "\n";
     }
     return result;
 }


Let's compare this now to a template function which does the same 
thing.

---
     string makeConvMatrix(types ...)()
     {
         string result = " ";
         foreach(t;types)
         {
             result ~= "\t" ~ t.stringof;
         }
         result ~= "\n";
         foreach(t1;types)
         {
             result ~= t1.stringof;
             foreach(t2;types)
             {
                 result ~=  "\t" ~ (is(t1:t2) ? "yes" : "no");
             }
             result ~= "\n";
         }
         return result;
     }
---

At those scales there is no difference in compile time.
However there is a difference in file size.

Type function version:

stat  makeConvMatrix.o
   File: 'makeConvMatrix.o'
   Size: 2876

VS template version

stat  makeConvMatrix_tmpl.o
   File: 'makeConvMatrix_tmpl.o'
   Size: 11760
Oct 01 2020
next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 08:57:12 UTC, Stefan Koch wrote:

 At those scales there is no difference in compile time.
 However there is a difference in file size.

 Type function version:

 stat  makeConvMatrix.o
   File: 'makeConvMatrix.o'
   Size: 2876

 VS template version

 stat  makeConvMatrix_tmpl.o
   File: 'makeConvMatrix_tmpl.o'
   Size: 11760


Let me illustrate the user visible diffrence:
type function:
string makeConvMatrix(alias[] types ...)  // 0.245 times the size 
of template object file
Included symbols (extracted with nm)
0000000000000000 R _D14makeConvMatrix12__ModuleInfoZ
                  U _d_dso_registry
                  U __start_minfo
                  U __stop_minfo

template:
string makeConvMatrix(types ...)()        // 4 times the size of 
typefunction object file
included symbols (extracted with nm)
                  U _D12TypeInfo_Aya6__initZ
0000000000000000 R _D14makeConvMatrix12__ModuleInfoZ
0000000000000000 W 
_D14makeConvMatrix__TQtTgThTsTtTiTkTlTmZQBmFNaNbNfZAya
                  U _d_arrayappendT
                  U _d_dso_registry
                  U _GLOBAL_OFFSET_TABLE_
                  U __start_minfo
                  U __stop_minfo
0000000000000000 r _TMP0
0000000000000002 r _TMP1
0000000000000038 r _TMP10
000000000000003d r _TMP11
0000000000000042 r _TMP12
0000000000000048 r _TMP13
000000000000004e r _TMP14
0000000000000052 r _TMP15
0000000000000059 r _TMP16
000000000000005d r _TMP17
0000000000000062 r _TMP18
0000000000000067 r _TMP19
0000000000000008 r _TMP2
000000000000000f r _TMP3
0000000000000016 r _TMP4
000000000000001e r _TMP5
0000000000000023 r _TMP6
0000000000000029 r _TMP7
000000000000002f r _TMP8
0000000000000036 r _TMP9
Oct 01 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 09:28:34 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 08:57:12 UTC, Stefan Koch wrote:

 Let me illustrate the user visible diffrence:
 type function:
 string makeConvMatrix(alias[] types ...)  // 0.245 times the 
 size of template object file
 Included symbols (extracted with nm)
 0000000000000000 R _D14makeConvMatrix12__ModuleInfoZ
                  U _d_dso_registry
                  U __start_minfo
                  U __stop_minfo

 template:
 string makeConvMatrix(types ...)()        // 4 times the size 
 of typefunction object file
 included symbols (extracted with nm)
                  U _D12TypeInfo_Aya6__initZ
 0000000000000000 R _D14makeConvMatrix12__ModuleInfoZ
 0000000000000000 W 
 _D14makeConvMatrix__TQtTgThTsTtTiTkTlTmZQBmFNaNbNfZAya
                  U _d_arrayappendT


And btw the fact that you don't see U _d_arrayappendT in the 
imported symbols of the type function ... means it works with 
-betterC!
Oct 01 2020
parent reply user1234 <user1234 12.de> writes:
On Thursday, 1 October 2020 at 09:33:28 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 09:28:34 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 08:57:12 UTC, Stefan Koch wrote:

 Let me illustrate the user visible diffrence:
 type function:
 string makeConvMatrix(alias[] types ...)  // 0.245 times the 
 size of template object file
 Included symbols (extracted with nm)
 0000000000000000 R _D14makeConvMatrix12__ModuleInfoZ
                  U _d_dso_registry
                  U __start_minfo
                  U __stop_minfo

 template:
 string makeConvMatrix(types ...)()        // 4 times the size 
 of typefunction object file
 included symbols (extracted with nm)
                  U _D12TypeInfo_Aya6__initZ
 0000000000000000 R _D14makeConvMatrix12__ModuleInfoZ
 0000000000000000 W 
 _D14makeConvMatrix__TQtTgThTsTtTiTkTlTmZQBmFNaNbNfZAya
                  U _d_arrayappendT


 And btw the fact that you don't see U _d_arrayappendT in the 
 imported symbols of the type function ... means it works with 
 -betterC!


BTW how is doing the DIP for this ;)
Oct 01 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 09:37:20 UTC, user1234 wrote:

 BTW how is doing the DIP for this ;)


I have a start.
But right now my motivation to continue the DIP is quite low.
It seems Andrei and Walter are immersed in building a cargo-cult 
version of this.
Oct 01 2020
parent reply user1234 <user1234 12.de> writes:
On Thursday, 1 October 2020 at 09:44:42 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 09:37:20 UTC, user1234 wrote:

 BTW how is doing the DIP for this ;)


 I have a start.
 But right now my motivation to continue the DIP is quite low.
 It seems Andrei and Walter are immersed in building a 
 cargo-cult version of this.


That's sad. I start believing to TypeFunction, you see I'm not 
like those who's been immediatly enthusisast, but well, if the 
boss shortcut your work with workarounds (surface changes as you 
said yesterday) having them in D will be a problem, e.g "we can 
do that with... which already exist...bla bla"
Oct 01 2020
next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 09:52:19 UTC, user1234 wrote:

 On Thursday, 1 October 2020 at 09:44:42 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 09:37:20 UTC, user1234 wrote:

 BTW how is doing the DIP for this ;)


 I have a start.
 But right now my motivation to continue the DIP is quite low.
 It seems Andrei and Walter are immersed in building a 
 cargo-cult version of this.


 That's sad. I start believing to TypeFunction, you see I'm not 
 like those who's been immediatly enthusisast, but well, if the 
 boss shortcut your work with workarounds (surface changes as 
 you said yesterday) having them in D will be a problem, e.g "we 
 can do that with... which already exist...bla bla"


I mean on the other hand.
Maybe I should be motivated.
Now I can do a direct comparison to alternative implementations.
Without having to write the alternative myself.
Oct 01 2020
next sibling parent reply user1234 <user1234 12.de> writes:
On Thursday, 1 October 2020 at 09:54:25 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 09:52:19 UTC, user1234 wrote:

 On Thursday, 1 October 2020 at 09:44:42 UTC, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 09:37:20 UTC, user1234 wrote:

 BTW how is doing the DIP for this ;)


 I have a start.
 But right now my motivation to continue the DIP is quite low.
 It seems Andrei and Walter are immersed in building a 
 cargo-cult version of this.


 That's sad. I start believing to TypeFunction, you see I'm not 
 like those who's been immediatly enthusisast, but well, if the 
 boss shortcut your work with workarounds (surface changes as 
 you said yesterday) having them in D will be a problem, e.g 
 "we can do that with... which already exist...bla bla"


 I mean on the other hand.
 Maybe I should be motivated.
 Now I can do a direct comparison to alternative implementations.
 Without having to write the alternative myself.


(note I leave the serious domain)

I'm amused actually by the alternative. It's a FFT that makes you 
travel from the type domain to the string domain and an iFFT that 
makes you travel from the string domain to the type domain if I 
have followed correctly.
Oct 01 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 10:10:47 UTC, user1234 wrote:

 (note I leave the serious domain)

 I'm amused actually by the alternative. It's a FFT that makes 
 you travel from the type domain to the string domain and an 
 iFFT that makes you travel from the string domain to the type 
 domain if I have followed correctly.


Well it's is just kindof what happens.
They transform a type an Object representing the type.
which includes the mangle, which of course identifies the type 
uniquely.
Therefore you can go from TypeObject -> string -> Type
All of which has to be in template land of course.
Type -> TypeObject has to be done in a template there's no way 
around that.
And string -> Type has to be done in a template as well, and that 
string has to be a template parameter.
Therefore you still have to work within templates.
Oct 01 2020
parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Thursday, 1 October 2020 at 10:16:47 UTC, Stefan Koch wrote:

 And string -> Type has to be done in a template as well, and 
 that string has to be a template parameter.
 Therefore you still have to work within templates.


Is it possible to applying the `AliasSeq`-special-handling-hack 
in dmd to make `reify` and `unreify` not be real templates?

Would it be possible to rewrite often-used templates in 
druntime/phobos to use alias functions and highlight savings in 
compile-time/space and object-size for larger use cases?
Oct 01 2020
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 1 October 2020 at 09:54:25 UTC, Stefan Koch wrote:

 [snip]

 I mean on the other hand.
 Maybe I should be motivated.
 Now I can do a direct comparison to alternative implementations.
 Without having to write the alternative myself.


I certainly think it's interesting and seems more straightforward 
than Andrei's reification example, but more concrete results will 
help your case. You might make a few examples of both simple and 
complex use cases of this technique and compare (at least 
compile-time and the size of object files) to the current D 
compiler and Andrei's reification approach. You might have a case 
if you can show quantitative improvements over the alternatives.

One thing that keeps coming to my mind when you are providing 
examples is the zig language's comptime [1]. They have the 
example of

fn max(comptime T: type, a: T, b: T) T {
     return if (a > b) a else b;
}
fn gimmeTheBiggerFloat(a: f32, b: f32) f32 {
     return max(f32, a, b);
}
fn gimmeTheBiggerInteger(a: u64, b: u64) u64 {
     return max(u64, a, b);
}

and obviously this could be done in D with templates. But if type 
functions were in D, then I would imagine you should also be able 
to do it like:

T max(alias T, T a, T b) {
     return (a > b) ? a : b;
}
float gimmeTheBiggerFloat(float a, float b) {
     return max(float, a, b);
}
uint gimmeTheBiggerInteger(uint a, uint b) {
     return max(uint, a, b);
}


[1] 
https://ziglang.org/documentation/master/#Introducing-the-Compile-Time-Concept
Oct 01 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 11:35:10 UTC, jmh530 wrote:

 On Thursday, 1 October 2020 at 09:54:25 UTC, Stefan Koch wrote:

 [snip]


 I would imagine you should also be able to do it like:

 T max(alias T, T a, T b) {
     return (a > b) ? a : b;
 }
 float gimmeTheBiggerFloat(float a, float b) {
     return max(float, a, b);
 }
 uint gimmeTheBiggerInteger(uint a, uint b) {
     return max(uint, a, b);
 }


No you would not be able to write that.
Because type functions are not polymorphic.
I.E. you cannot declare a variable with a type alias.

type functions do a Type -> TypeObject conversion when you pass 
the types in.
inside the type function you get a "type-like" interface.
But it has lost everything that made it a type.
It has sizeof, alignof, stringof, tupleof, you can use __traits 
on it.
But it cannot be used as a type anymore.
Not inside a type-function anyway.

If you return a type or a type tuple from a type function, it 
becomes a regular type/type tuple again.
Oct 01 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 01, 2020 at 12:27:23PM +0000, Stefan Koch via Digitalmars-d wrote:
[...]

 type functions do a Type -> TypeObject conversion when you pass the
 types in.
 inside the type function you get a "type-like" interface.
 But it has lost everything that made it a type.
 It has sizeof, alignof, stringof, tupleof, you can use __traits on it.
 But it cannot be used as a type anymore.
 Not inside a type-function anyway.
 
 If you return a type or a type tuple from a type function, it becomes
 a regular type/type tuple again.


Isn't this just the same thing as Andrei's reification / dereification,
except redressed in terms of aliases?


T

-- 
MACINTOSH: Most Applications Crash, If Not, The Operating System Hangs
Oct 01 2020
next sibling parent Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 13:53:53 UTC, H. S. Teoh wrote:

 On Thu, Oct 01, 2020 at 12:27:23PM +0000, Stefan Koch via 
 Digitalmars-d wrote: [...]

 type functions do a Type -> TypeObject conversion when you 
 pass the
 types in.
 inside the type function you get a "type-like" interface.
 But it has lost everything that made it a type.
 It has sizeof, alignof, stringof, tupleof, you can use 
 __traits on it.
 But it cannot be used as a type anymore.
 Not inside a type-function anyway.
 
 If you return a type or a type tuple from a type function, it 
 becomes a regular type/type tuple again.


 Isn't this just the same thing as Andrei's reification / 
 dereification, except redressed in terms of aliases?


 T


Well you just pose the question the other way around.

Isn't Andrei's thing just a copy of mine, except redressed in 
terms of templates?

Yes. But the object being implicit I can use compiler internals 
to extract the data I need on the fly.
No need to prebuild conversion tables and the like.
No need to define a struct in druntime.
Oct 01 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 9:53 AM, H. S. Teoh wrote:

 On Thu, Oct 01, 2020 at 12:27:23PM +0000, Stefan Koch via Digitalmars-d wrote:
 [...]

 type functions do a Type -> TypeObject conversion when you pass the
 types in.
 inside the type function you get a "type-like" interface.
 But it has lost everything that made it a type.
 It has sizeof, alignof, stringof, tupleof, you can use __traits on it.
 But it cannot be used as a type anymore.
 Not inside a type-function anyway.

 If you return a type or a type tuple from a type function, it becomes
 a regular type/type tuple again.


 
 Isn't this just the same thing as Andrei's reification / dereification,
 except redressed in terms of aliases?


The difference I see is that reification/dereification has to rebuild 
everything that is already in the compiler, but at runtime. And it has 
to make sure the functionality is identical. You are using the compiler 
to build another type introspection system, but just so you can do CTFE 
based things.

It reminds me of "modern" web applications which take over browser 
functionality (page navigation, etc.) and reimplement it in javascript.

If we can't get a system that allows dereifcation DURING function 
execution, then I think type functions are a much better solution. 
There's no reason to reimplement what the compiler already does. Type 
functions are cleaner, clearer, and much more efficient.

-Steve
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 10/1/20 12:12 PM, Steven Schveighoffer wrote:

 
 The difference I see is that reification/dereification has to rebuild 
 everything that is already in the compiler, but at runtime. And it has 
 to make sure the functionality is identical. You are using the compiler 
 to build another type introspection system, but just so you can do CTFE 
 based things.
 
 It reminds me of "modern" web applications which take over browser 
 functionality (page navigation, etc.) and reimplement it in javascript.
 
 If we can't get a system that allows dereifcation DURING function 
 execution, then I think type functions are a much better solution. 
 There's no reason to reimplement what the compiler already does. Type 
 functions are cleaner, clearer, and much more efficient.


Hmmm... interesting. It's also been my accumulated perception that much 
metaprogramming is in fact "doing things in the target language that 
normally only the compiler writer is empowered to do". For example, the 
implementation of switch on strings has been entirely shipped to 
druntime 
(https://github.com/dlang/druntime/blob/master/src/core/internal/switch_.d), 
which does everything a compiler normally does with optimizing a switch: 
looks at the number and content of cases, defines distinct strategies 
depending on them, etc. In smaller form __cmp is the same story 
(optimizes comparison depending on the types involved) and many other 
cases. My take is that that's a good thing, because I don't need to be 
in the compiler do do powerful things.

Of course std.traits would be chock full of this kind of stuff, as no 
doubt most artifacts in there are present in the compiler internals, or 
easily extractable therefrom. In that respect, __traits is an exercise 
in "what is the minimum set of primitives the compiler needs to expose, 
in order to allow regular D code to do everything else that the compiler 
normally does?" It's a bottleneck design (complex entities communicating 
through a narrow interface). Probably not very well done because 
__traits keep on getting a variety of things that are difficult to 
decide upon. I have no good litmus test on a proposed __trait whether 
it's deserving, the right thing, the most enabling, etc. On the other 
hand, std.traits did allow us to do things that would be quite onerous 
to put in the language definition and/or compiler implementation. On the 
whole this bottleneck design has been quite enabling.

The difficulty/challenge of crossing of the barrier from types to values 
and back is good to appreciate when doing things like mixed 
compile-time/run-time uses and integration with type-oriented facilities 
(such as those in std.traits itself).
Oct 01 2020
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 1 October 2020 at 16:49:20 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/20 12:12 PM, Steven Schveighoffer wrote:

 
 The difference I see is that reification/dereification has to 
 rebuild everything that is already in the compiler, but at 
 runtime. And it has to make sure the functionality is 
 identical. You are using the compiler to build another type 
 introspection system, but just so you can do CTFE based things.
 
 It reminds me of "modern" web applications which take over 
 browser functionality (page navigation, etc.) and reimplement 
 it in javascript.
 
 If we can't get a system that allows dereifcation DURING 
 function execution, then I think type functions are a much 
 better solution. There's no reason to reimplement what the 
 compiler already does. Type functions are cleaner, clearer, 
 and much more efficient.


 Hmmm... interesting. It's also been my accumulated perception 
 that much metaprogramming is in fact "doing things in the 
 target language that normally only the compiler writer is 
 empowered to do". For example, the implementation of switch on 
 strings has been entirely shipped to druntime 
 (https://github.com/dlang/druntime/blob/master/src/core/internal/switch_.d),
which does everything a compiler normally does with optimizing a switch: looks
at the number and content of cases, defines distinct strategies depending on
them, etc. In smaller form __cmp is the same story (optimizes comparison
depending on the types involved) and many other cases. My take is that that's a
good thing, because I don't need to be in the compiler do do powerful things.

 Of course std.traits would be chock full of this kind of stuff, 
 as no doubt most artifacts in there are present in the compiler 
 internals, or easily extractable therefrom. In that respect, 
 __traits is an exercise in "what is the minimum set of 
 primitives the compiler needs to expose, in order to allow 
 regular D code to do everything else that the compiler normally 
 does?" It's a bottleneck design (complex entities communicating 
 through a narrow interface). Probably not very well done 
 because __traits keep on getting a variety of things that are 
 difficult to decide upon. I have no good litmus test on a 
 proposed __trait whether it's deserving, the right thing, the 
 most enabling, etc. On the other hand, std.traits did allow us 
 to do things that would be quite onerous to put in the language 
 definition and/or compiler implementation. On the whole this 
 bottleneck design has been quite enabling.

 The difficulty/challenge of crossing of the barrier from types 
 to values and back is good to appreciate when doing things like 
 mixed compile-time/run-time uses and integration with 
 type-oriented facilities (such as those in std.traits itself).


Please do try to express yourself more concisely.
As well as close to the topic.
I cannot make heads or tails of your writing.

This is not about std.traits.
This is more about making std.traits superfluous.
Integrating type based programming so well with the language that 
there is almost no barrier when crossing to type-object land.
The serialized "reified" type is given behavior as close as 
possible to a real type.
Which simplifies the interface, and keeps it intuitive.

All you can do with a type the compiler can already do.
It has to be able to do it.
So at CTFE I merely tap into what the compiler already does.
There's very little code duplication and even the bit there is 
can be improved incrementally making the compiler more modular.
Oct 01 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 12:49 PM, Andrei Alexandrescu wrote:

 On 10/1/20 12:12 PM, Steven Schveighoffer wrote:

 The difference I see is that reification/dereification has to rebuild 
 everything that is already in the compiler, but at runtime. And it has 
 to make sure the functionality is identical. You are using the 
 compiler to build another type introspection system, but just so you 
 can do CTFE based things.

 It reminds me of "modern" web applications which take over browser 
 functionality (page navigation, etc.) and reimplement it in javascript.

 If we can't get a system that allows dereifcation DURING function 
 execution, then I think type functions are a much better solution. 
 There's no reason to reimplement what the compiler already does. Type 
 functions are cleaner, clearer, and much more efficient.


 
 Hmmm... interesting. It's also been my accumulated perception that much 
 metaprogramming is in fact "doing things in the target language that 
 normally only the compiler writer is empowered to do". For example, the 
 implementation of switch on strings has been entirely shipped to 
 druntime 
 (https://github.com/dlang/druntime/blob/master/src/core/internal/switch_.d), 


This is not exactly true -- the compiler still only allows strings in 
switch statements (not arbitrary types), and it also generates case 
labels based on how it expects the runtime to return the correct value. 
The runtime also expects the compiler to pass the strings in a certain 
order.

What *is* true, is that the runtime can make different decisions on what 
a match is. For example, the template could potentially do 
case-insensitive comparison, or use a different mechanism to search 
based on the incoming parameters (what it does now). But I don't know if 
this also is used in CTFE?

However, I will note that before it was this template call, the compiler 
still relied on the runtime to do what it currently does. It just passed 
the strings as a runtime parameter instead of a compile time parameter. 
(https://github.com/dlang/druntime/blob/c9dbcbb8cab8a480b143c1f8a78ba5193d3d2c40/src/rt/switch_.d#L30)

I have nothing against hoisting algorithmic tasks out of the compiler, 
and using the runtime to determine such things. In fact, I prefer it -- 
the runtime is so much more approachable as a developer than the 
compiler. But when the compiler MUST implement is(T : U), and that 
expression is well defined, I don't see why we have to reimplement that 
feature in the runtime as well. At least without a compelling example of 
"the compiler can't do this as well".

-Steve
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 1:26 PM, Steven Schveighoffer wrote:

 But when the compiler MUST implement is(T : U), and that expression is 
 well defined, I don't see why we have to reimplement that feature in the 
 runtime as well. At least without a compelling example of "the compiler 
 can't do this as well".


How do you implement Variant.get(T) without reifying is(T : U)?
Oct 01 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 1 October 2020 at 17:35:51 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/20 1:26 PM, Steven Schveighoffer wrote:

 But when the compiler MUST implement is(T : U), and that 
 expression is well defined, I don't see why we have to 
 reimplement that feature in the runtime as well. At least 
 without a compelling example of "the compiler can't do this as 
 well".


 How do you implement Variant.get(T) without reifying is(T : U)?


What does it do?
is Variant.get returning type T ?
Then I would assume you don't.
You use a template + typeid that's what they are for!

Variant.get is not doing any type computation which you would 
need is (T : U) for?
Where would I use it?
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 1:45 PM, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 17:35:51 UTC, Andrei Alexandrescu wrote:

 On 10/1/20 1:26 PM, Steven Schveighoffer wrote:

 But when the compiler MUST implement is(T : U), and that expression 
 is well defined, I don't see why we have to reimplement that feature 
 in the runtime as well. At least without a compelling example of "the 
 compiler can't do this as well".


 How do you implement Variant.get(T) without reifying is(T : U)?


 
 What does it do?
 is Variant.get returning type T ?


Yes: https://dlang.org/phobos/std_variant.html#.VariantN.get


 Then I would assume you don't.
 You use a template + typeid that's what they are for!
 
 Variant.get is not doing any type computation which you would need is (T 
 : U) for?
 Where would I use it?


Everywhere Variant.get is called. The decision is(T : U) needs to be 
carried, except that one of the types is available only at runtime.
Oct 01 2020
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 1 October 2020 at 17:50:39 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/20 1:45 PM, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 17:35:51 UTC, Andrei 
 Alexandrescu wrote:

 On 10/1/20 1:26 PM, Steven Schveighoffer wrote:

 [...]


 How do you implement Variant.get(T) without reifying is(T : 
 U)?


 
 What does it do?
 is Variant.get returning type T ?


 Yes: https://dlang.org/phobos/std_variant.html#.VariantN.get


 Then I would assume you don't.
 You use a template + typeid that's what they are for!
 
 Variant.get is not doing any type computation which you would 
 need is (T : U) for?
 Where would I use it?


 Everywhere Variant.get is called. The decision is(T : U) needs 
 to be carried, except that one of the types is available only 
 at runtime.


Please show me in the code where is(T : U) is used.
I don't find it in std variant.
Oct 01 2020
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 10/1/20 1:55 PM, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 17:50:39 UTC, Andrei Alexandrescu wrote:

 On 10/1/20 1:45 PM, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 17:35:51 UTC, Andrei Alexandrescu wrote:

 On 10/1/20 1:26 PM, Steven Schveighoffer wrote:

 [...]


 How do you implement Variant.get(T) without reifying is(T : U)?


 What does it do?
 is Variant.get returning type T ?


 Yes: https://dlang.org/phobos/std_variant.html#.VariantN.get


 Then I would assume you don't.
 You use a template + typeid that's what they are for!

 Variant.get is not doing any type computation which you would need is 
 (T : U) for?
 Where would I use it?


 Everywhere Variant.get is called. The decision is(T : U) needs to be 
 carried, except that one of the types is available only at runtime.


 
 Please show me in the code where is(T : U) is used.
 I don't find it in std variant.


It's in the use of ImplicitConversionTargets in std.traits. If I 
remember correctly that template was created exactly to help with 
implementing Variant. It does an incomplete and in places incorrect job 
at figuring out what implicit conversions would work. That information 
is saved as part of the pointer to function stored in the Variant 
object, and used to figure out if the call to get() is valid.
Oct 01 2020
next sibling parent Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 18:10:05 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/20 1:55 PM, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 17:50:39 UTC, Andrei 
 Alexandrescu wrote:

 On 10/1/20 1:45 PM, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 17:35:51 UTC, Andrei 
 Alexandrescu wrote:

 On 10/1/20 1:26 PM, Steven Schveighoffer wrote:

 [...]


 How do you implement Variant.get(T) without reifying is(T : 
 U)?


 What does it do?
 is Variant.get returning type T ?


 Yes: https://dlang.org/phobos/std_variant.html#.VariantN.get


 Then I would assume you don't.
 You use a template + typeid that's what they are for!

 Variant.get is not doing any type computation which you 
 would need is (T : U) for?
 Where would I use it?


 Everywhere Variant.get is called. The decision is(T : U) 
 needs to be carried, except that one of the types is 
 available only at runtime.


 
 Please show me in the code where is(T : U) is used.
 I don't find it in std variant.


 It's in the use of ImplicitConversionTargets in std.traits. If 
 I remember correctly that template was created exactly to help 
 with implementing Variant. It does an incomplete and in places 
 incorrect job at figuring out what implicit conversions would 
 work. That information is saved as part of the pointer to 
 function stored in the Variant object, and used to figure out 
 if the call to get() is valid.


So if your question is could you implement 
'ImplicitConversionTargets' as a type function the answer is most 
likely yes.
I haven't looked at the code too deeply.
But it should work in principle if it doesn't it's an 
implementation bug.
As such this 'ImplicitConversionTargets' is indeed a good 
use-case to look at.
Oct 01 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 2:10 PM, Andrei Alexandrescu wrote:

 On 10/1/20 1:55 PM, Stefan Koch wrote:

 Please show me in the code where is(T : U) is used.
 I don't find it in std variant.


 
 It's in the use of ImplicitConversionTargets in std.traits.


The only use of is(T:U) in that trait is:

is(T : typeof(null))

-Steve
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 10/1/20 2:26 PM, Steven Schveighoffer wrote:

 On 10/1/20 2:10 PM, Andrei Alexandrescu wrote:

 On 10/1/20 1:55 PM, Stefan Koch wrote:

 Please show me in the code where is(T : U) is used.
 I don't find it in std variant.


 It's in the use of ImplicitConversionTargets in std.traits.


 
 The only use of is(T:U) in that trait is:
 
 is(T : typeof(null))


Of course. That's the point - the template is not using "is", because it 
can't: it only has one type available. It "implements" a subset of it. 
See what I'm saying? ImplicitConversionTargets helps Variant implement a 
kinda sorta "is" at runtime.
Oct 01 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 2:32 PM, Andrei Alexandrescu wrote:

 On 10/1/20 2:26 PM, Steven Schveighoffer wrote:

 On 10/1/20 2:10 PM, Andrei Alexandrescu wrote:

 On 10/1/20 1:55 PM, Stefan Koch wrote:

 Please show me in the code where is(T : U) is used.
 I don't find it in std variant.


 It's in the use of ImplicitConversionTargets in std.traits.


 The only use of is(T:U) in that trait is:

 is(T : typeof(null))


 
 Of course. That's the point - the template is not using "is", because it 
 can't: it only has one type available. It "implements" a subset of it. 
 See what I'm saying? ImplicitConversionTargets helps Variant implement a 
 kinda sorta "is" at runtime.
 


Yes, I see. This subthread confused me...

-Steve
Oct 01 2020
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 01, 2020 at 05:55:17PM +0000, Stefan Koch via Digitalmars-d wrote:
[...]

 Please show me in the code where is(T : U) is used.
 I don't find it in std variant.


That's because it doesn't work.

We would like to be able to do this:

	Variant v;
	v = 100; // v now stores an int
	long l = v.get!long; // currently does not work

We cannot do this because the type of 100 is not known by Variant until
runtime, and at runtime, we cannot tell whether the stored type (int) is
implicitly convertible to the requested type (long).  So, we cannot
perform the implicit conversion.

This limitation makes Variant very unfriendly to use: unless you know
precisely what type is stored, you cannot get anything useful out of it.
If somebody stored an int in a Variant and hands it to you, and you want
a long (after all, the language supports implicit conversion from int to
long), you cannot get a long out of it without knowing beforehand it was
an int. In practice, this means lots of code duplication: if v.get!long
throws, try v.get!int; if that still doesn't do it, try v.get!short;
etc.. Basically you end up duplicating implicit conversions that the
compiler should have done for you.

We would like Variant to behave like a top type, but we can't because we
cannot duplicate the implicit conversions of stored types. We can get
most of the way there, but not all the way.


T

-- 
"Maybe" is a strange word.  When mom or dad says it it means "yes", but when my
big brothers say it it means "no"! -- PJ jr.
Oct 01 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 2:19 PM, H. S. Teoh wrote:

 On Thu, Oct 01, 2020 at 05:55:17PM +0000, Stefan Koch via Digitalmars-d wrote:
 [...]

 Please show me in the code where is(T : U) is used.
 I don't find it in std variant.


 
 That's because it doesn't work.
 
 We would like to be able to do this:
 
 	Variant v;
 	v = 100; // v now stores an int
 	long l = v.get!long; // currently does not work


I'm not going to respond to anything else here, but just want to point 
out, this does work.

https://run.dlang.io/is/S3sqDF

-Steve
Oct 01 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 01, 2020 at 02:28:37PM -0400, Steven Schveighoffer via
Digitalmars-d wrote:

 On 10/1/20 2:19 PM, H. S. Teoh wrote:


[...]

 We would like to be able to do this:
 
 	Variant v;
 	v = 100; // v now stores an int
 	long l = v.get!long; // currently does not work


 
 I'm not going to respond to anything else here, but just want to point
 out, this does work.
 
 https://run.dlang.io/is/S3sqDF


OK, I didn't check before I posted, :-/ but the point is that the
implicit conversion is manually hacked with a library template that does
not cover all the cases.  We should not have to reimplement implicit
conversions in library code when the compiler is already perfectly
capable of doing it.

It's not just implicit conversions, though.  There's a host of other
things that we cannot do currently.  Like conversion to string: in
theory, since std.conv.to is able to convert just about any type to
string, we ought to be able to support v.get!string for *any* stored
type.  Yet we cannot because at runtime all we have is typeid(T), so we
cannot dispatch to the correct overload of std.conv.to to create the
string.


T

-- 
Being forced to write comments actually improves code, because it is easier to
fix a crock than to explain it. -- G. Steele
Oct 01 2020
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 10/1/20 2:51 PM, H. S. Teoh wrote:

 Like conversion to string: in
 theory, since std.conv.to is able to convert just about any type to
 string, we ought to be able to support v.get!string for*any*  stored
 type.


Yah, that's quite problematic indeed.
Oct 01 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 2:51 PM, H. S. Teoh wrote:

 On Thu, Oct 01, 2020 at 02:28:37PM -0400, Steven Schveighoffer via
Digitalmars-d wrote:

 On 10/1/20 2:19 PM, H. S. Teoh wrote:


 [...]

 We would like to be able to do this:

 	Variant v;
 	v = 100; // v now stores an int
 	long l = v.get!long; // currently does not work


 I'm not going to respond to anything else here, but just want to point
 out, this does work.

 https://run.dlang.io/is/S3sqDF


 
 OK, I didn't check before I posted, :-/ but the point is that the
 implicit conversion is manually hacked with a library template that does
 not cover all the cases.  We should not have to reimplement implicit
 conversions in library code when the compiler is already perfectly
 capable of doing it.
 
 It's not just implicit conversions, though.  There's a host of other
 things that we cannot do currently.  Like conversion to string: in
 theory, since std.conv.to is able to convert just about any type to
 string, we ought to be able to support v.get!string for *any* stored
 type.  Yet we cannot because at runtime all we have is typeid(T), so we
 cannot dispatch to the correct overload of std.conv.to to create the
 string.


It's not implicit conversions that we are talking about. It's implicit 
conversion *checks*. Actually doing the conversion requires calling a 
function, which so far has not been proposed (and I'm a little scared to 
see how big this beast is going to end up being).

And by the way, Variant.toString works... and calls the correct overload 
of to!string ;) 
https://github.com/dlang/phobos/blob/b0b64c3f41014addc6f5284f9eb20f8e90116ec2/std/variant.d#L438

"solving the Variant problem" needs a problem definition first.

-Steve
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 3:01 PM, Steven Schveighoffer wrote:

 "solving the Variant problem" needs a problem definition first.


Problem definition: Implement Variant.get correctly. :o)

E.g. this produces a type error somewhere in the innards of std.variant: 
https://run.dlang.io/is/joIbeV. I'm not sure how many other cases are 
out there.

BTW thanks Steve for choosing the right angle. This is not a contest, 
and not a search for the proverbial nails for the use of a given hammer. 
The converse approach is best - find what the difficult related problems 
are, and figure how to solve them well.
Oct 01 2020
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 3:15 PM, Andrei Alexandrescu wrote:

 On 10/1/20 3:01 PM, Steven Schveighoffer wrote:

 "solving the Variant problem" needs a problem definition first.


 
 Problem definition: Implement Variant.get correctly. :o)
 
 E.g. this produces a type error somewhere in the innards of std.variant: 
 https://run.dlang.io/is/joIbeV. I'm not sure how many other cases are 
 out there.


There are a lot. And you haven't even looked at contravariance:

alias F1 = void function(A);
alias F2 = void function(const(A));

F1 f1;
F2 f2;

f1 = f2; // ok

Though for some reason function(A) doesn't implicitly convert to 
function(B) (which would open this up completely to all derivative types 
in existence).

And it's not straightforward either. For instance, int can convert to 
long, but int function() cannot convert to long function().

So now you need the following properties:

canBeReturnedInsteadOf(Id)
canBeUsedAsAParameterInsteadOf(Id)

And generate all the possibilities for those (assuming they are bounded 
in one direction)

The code to make this "work" is going to be insane. And generate huge 
amounts of static data based on the type.

So let's give a more concrete definition for "implement Variant.get 
correctly". That is:

Given any two types T and U:

If this compiles:
T t;
U u = t;

Then this should as well:
Variant v = T.init;
U u = v.get!U;

The question now becomes, can we get this to work with reification 
easier or more efficiently than without reification?

I'd pose a further question: Is this important enough to add to Variant? 
I picked a typed language for a reason.


 BTW thanks Steve for choosing the right angle. This is not a contest, 
 and not a search for the proverbial nails for the use of a given hammer. 
 The converse approach is best - find what the difficult related problems 
 are, and figure how to solve them well.


I'm fine with examining multiple solutions to the problem. I'm not 
rooting for a specific solution as much as examining the tradeoffs and 
enabling mechanisms for both.

I also can't help but be biased by having used TypeInfo (which is 
essentially a less powerful Id) and seeing the limitations there.

-Steve
Oct 01 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 1 October 2020 at 19:15:10 UTC, Andrei Alexandrescu 
wrote:

 Problem definition: Implement Variant.get correctly. :o)

 E.g. this produces a type error somewhere in the innards of 
 std.variant: https://run.dlang.io/is/joIbeV. I'm not sure how 
 many other cases are out there.



So here's my proof of concept that this is doable today:


---------------

import std.typecons; // for Rebindable

interface MyTypeInfo {
	const pure nothrow: // yadda yadda yada
	bool implicitlyConvertsTo(const MyTypeInfo t);
	bool isNullable();
	string toiString();
}

interface MyTypeInfoFunction {
	const pure nothrow:

	immutable(MyTypeInfo) returnType();
	immutable(MyTypeInfo)[] paramsTypes();
}

interface MyTypeInfoClass {
	const pure nothrow:

	immutable(MyTypeInfo)[] bases();
}

class MyTypeInfoImpl_FunctionPointer(T) : MyTypeInfo, 
MyTypeInfoFunction {
	const:
	bool isNullable() { return true; }
	string toiString() { return T.stringof; }

	static if(is(T == R function(Params), R, Params...)) {
		immutable(MyTypeInfo) returnType() {
			return mytypeid!R;
		}

		immutable(MyTypeInfo)[] paramsTypes() pure nothrow {
			import std.meta; // I wouldn't normally use this but meh it is 
a demo
			static immutable MyTypeInfo[] result = [staticMap!(mytypeid, 
Params)];

			return result[];
		}
	} else static assert(0, "mistake in " ~ T.stringof);

	bool implicitlyConvertsTo(const MyTypeInfo t) {
		if(this is t)
			return true;

		if(t is mytypeid!(typeof(null)))
			return true;

		auto fp = cast(MyTypeInfoFunction) t;
		if(fp is null)
			return false;

		if(!returnType.implicitlyConvertsTo(fp.returnType))
			return false;

		if(paramsTypes.length != fp.paramsTypes.length)
			return false;

		foreach(idx, arg; fp.paramsTypes)
			if(!arg.implicitlyConvertsTo(paramsTypes[idx]))
				return false;

		return true;
	}
}

class MyTypeInfoImpl_Class(T) : MyTypeInfo, MyTypeInfoClass {
	bool isNullable() const { return true; }
	string toiString() const { return T.stringof; }

	static if(is(T Super == super))
	immutable(MyTypeInfo)[] bases() const {
		import std.meta;
		static immutable MyTypeInfo[] result = [staticMap!(mytypeid, 
Super)];
		return result[];
	}

	bool implicitlyConvertsTo(const MyTypeInfo t) const {
		if(this is t)
			return true;

		if(t is mytypeid!(typeof(null)))
			return true;

		foreach(base; bases)
			if(t is base)
				return true;
		return false;
	}
}

class MyTypeInfoImpl(T) : MyTypeInfo {
	bool isNullable() const {
		return is(typeof(null) : T);
	}

	string toiString() const {
		return T.stringof;
	}

	bool implicitlyConvertsTo(const MyTypeInfo t) const {
		static if(is(T == typeof(null)))
			return t.isNullable();
		return this is t;
	}
}

template mytypeid(T) {
	static if(is(T == return))
		immutable MyTypeInfo mytypeid = new immutable 
MyTypeInfoImpl_FunctionPointer!T;
	else static if(is(T == class))
		immutable MyTypeInfo mytypeid = new immutable 
MyTypeInfoImpl_Class!T;
	else {
		//pragma(msg, "generic " ~ T.stringof);
		immutable MyTypeInfo mytypeid = new immutable MyTypeInfoImpl!T;
	}
}

struct MyVariant {
	this(T)(T t) {
		opAssign(t);
	}

	void opAssign(T)(T t) {
		storedType = mytypeid!T;

		// I know this impl sux but std.variant already solved it so 
not the point here.
		union SD {
			StoredData sd;
			T b;
		}
		SD sd;
		sd.b = t;
		storedData = sd.sd;
	}

	T get(T)() {
		if(storedType is null)
			static if(is(T : typeof(null)))
				return null;
			else
				throw new Exception("null cannot become " ~ T.stringof);
		if(storedType.implicitlyConvertsTo(mytypeid!(T))) {
			union SD {
				StoredData sd;
				T b;
			}
			SD sd;
			sd.sd = storedData;
			return sd.b;
		} else
			throw new Exception(storedType.toiString() ~ " cannot become " 
~ T.stringof);
	}

	Rebindable!(immutable(MyTypeInfo)) storedType;
	// std.variant already has a better impl of this
	struct StoredData {
		void* b1;
		void* b2;
	}
	StoredData storedData;
}


class A {}
class B : A {}
import std.stdio;
A func() { writeln("func A"); return null; }
B func2() { writeln("func2 B"); return null; }

void main() {
	MyVariant v = &func2;
	auto test = &func2;
	typeof(&func) test2 = test;
	A function() a = v.get!(typeof(&func));

	a();
	test2();
}

----------------


I understand that's a decent chunk of code and it is very 
incomplete - it is basically the minimum to achieve the 
challenge. And I think I missed up the contravariant parameter 
implementation there, I always forget the exact rule without 
looking it up. But nevertheless that's an implementation bug, not 
a language barrier.

I hope it shows that this isn't hopeless with today's D. All that 
compile time knowledge we need *is* available to Variant itself, 
it just needs to dig a little deeper to put it together.... and 
then reimplement the compiler's rules, hopefully correctly.

Of course it would be better if we could just reuse dmd's 
implementation! But *that* I don't think is possible because 
Variant crosses the runtime barrier.... even with a type function 
I think it'd need a *pointer* to a typefunction which again hits 
that RT/CT barrier.
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 7:02 PM, Adam D. Ruppe wrote:

 On Thursday, 1 October 2020 at 19:15:10 UTC, Andrei Alexandrescu wrote:

 Problem definition: Implement Variant.get correctly. :o)

 E.g. this produces a type error somewhere in the innards of 
 std.variant: https://run.dlang.io/is/joIbeV. I'm not sure how many 
 other cases are out there.


 
 
 So here's my proof of concept that this is doable today:


[snip]

Is this a visitation/double dispatch?
Oct 01 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 October 2020 at 02:23:25 UTC, Andrei Alexandrescu 
wrote:

 Is this a visitation/double dispatch?


ummmm I'm actually not sure. I thought about doing a proper 
double dispatch but instead the implementation is pretty much 
single... just there is a dynamic cast in the convert methods so 
maybe technically it still falls under the definition.

The basic idea though is it doesn't take magic to do this. All 
the types the Variant needs *are* available to it thanks to 
normal templates, every type it ever sees are sent in (and ones 
it never sees it never needs), just the difficulty is they come 
in two separate calls. Thus it extracts what it needs from them 
into a runtime class to bridge that gap. And then any 
compile-time reflection that requires two types simultaneously 
(like `is(a:b)`) will need to have its logic re-implemented in 
library code instead of leveraging the compiler's built in magic. 
That is a legit hassle but not a fatal barrier.

All the runtime data required can be generated by normal 
templates though normal reflection into normal objects of normal 
data. Nothing special required.
Oct 01 2020
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Oct 02, 2020 at 03:36:32AM +0000, Adam D. Ruppe via Digitalmars-d wrote:
[...]

 The basic idea though is it doesn't take magic to do this. All the
 types the Variant needs *are* available to it thanks to normal
 templates, every type it ever sees are sent in (and ones it never sees
 it never needs), just the difficulty is they come in two separate
 calls. Thus it extracts what it needs from them into a runtime class
 to bridge that gap.


[...]

This is awesome!  I thought about it last night, and realized that this
is the same recipe that I discovered for automatic extraction of
i18n-able strings.  The key ingredient is static ctors, usually wrapped
in a wrapper struct, that register the needed data with a central
collector at program startup (or even when dynamically-loaded libraries
are loaded):

	auto myTemplateFunc(T)(T t) {
		struct RuntimeStartupHook {
			static this() {
				registerType!T( /* some info here */);
			}
		}

		return ... /* function implementation here */;
	}

	static RuntimeInfo db;
	void registerType(T)(...) {
		db.add( ... /* information about T */);
	}

Each time somebody calls myTemplateFunc, information about the type T is
injected into a static ctor that will get called upon program startup,
which registers information about T into the runtime database `db`. This
can be anything from the type name, to the typeid, or whatever traits
introspected from T in RuntimeStartupHook.  By the time main() runs, we
have collected information about *all* types that have been used to
instantiate myTemplateFunc.  This works with separate compilation.

Better yet, this will even work for dynamically-loaded libraries
(provided their static ctors are automatically run upon startup),
because the static ctors will register any new T's that were used in the
library code with the central database. So after loading the library,
`db` will contain this new information, and the implementation of
myTemplateFunc can then make use of it.

In this way, we can collect *all* types used to instantiate a particular
template function, and be able to make decisions over that at runtime.
Such as Variant being able to loop over all types it was instantiated
with.  For example, if we wanted to support, say, invoking some method
.abc on T if it exists, we can do it by having RuntimeStartupHook
introspect T at compile-time to bind any method named .abc and wrap that
in a function pointer or delegate.  Then at runtime, we simply consult
the database to determine whether T has such a method, and if so, invoke
it.  A lot of compile-time information can be transferred to runtime
this way. This is powerful stuff!

About the only thing we can't do is things like implicit conversions
between two instantiations of T, because is(T : U) requires knowledge of
*both* T and U simultaneously at compile-time, but in the scheme above,
T and U are individually registered at runtime, and we cannot know both
simultaneously until after the runtime hooks have run. (E.g., if T was
used in main but U was used in a dynamically-loaded library, then there
is no way to leverage is(T : U) at compile-time.)  So unfortunately, we
still have to re-implement is(T : U) manually in library code.


T

-- 
People walk. Computers run.
Oct 02 2020
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 10/1/20 2:19 PM, H. S. Teoh wrote:

 On Thu, Oct 01, 2020 at 05:55:17PM +0000, Stefan Koch via Digitalmars-d wrote:
 [...]

 Please show me in the code where is(T : U) is used.
 I don't find it in std variant.


 
 That's because it doesn't work.
 
 We would like to be able to do this:
 
 	Variant v;
 	v = 100; // v now stores an int
 	long l = v.get!long; // currently does not work


It actually should. ImplicitConversionTargets does cover numerics pretty 
well.
Oct 01 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 1:35 PM, Andrei Alexandrescu wrote:

 On 10/1/20 1:26 PM, Steven Schveighoffer wrote:

 But when the compiler MUST implement is(T : U), and that expression is 
 well defined, I don't see why we have to reimplement that feature in 
 the runtime as well. At least without a compelling example of "the 
 compiler can't do this as well".


 
 How do you implement Variant.get(T) without reifying is(T : U)?


How do you implement Variant.get(T) *with* reifying is(U : T)?

step 1: determine using reified constructs that U is convertible to T
step 2: ?????
step 3: Return a T from a U!

BTW, Variant already does a *limited* form of this. reification doesn't 
change what needs to happen.

I don't see why Variant's needs have to dictate how you need to reason 
about types at compile time in CTFE.

-Steve
Oct 01 2020
next sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Thursday, 1 October 2020 at 18:08:11 UTC, Steven Schveighoffer 
wrote:

 On 10/1/20 1:35 PM, Andrei Alexandrescu wrote:
 ...

 I don't see why Variant's needs have to dictate how you need to 
 reason about types at compile time in CTFE.


Yes.  If we want to talk about how we might eliminate the 
distinction between run-time and compile-time a separate thread 
would be useful.  I'd sure follow it!
Oct 01 2020
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 01, 2020 at 02:08:11PM -0400, Steven Schveighoffer via
Digitalmars-d wrote:

 On 10/1/20 1:35 PM, Andrei Alexandrescu wrote:


[...]

 How do you implement Variant.get(T) without reifying is(T : U)?


 
 How do you implement Variant.get(T) *with* reifying is(U : T)?
 
 step 1: determine using reified constructs that U is convertible to T
 step 2: ?????
 step 3: Return a T from a U!
 
 BTW, Variant already does a *limited* form of this. reification
 doesn't change what needs to happen.
 
 I don't see why Variant's needs have to dictate how you need to reason
 about types at compile time in CTFE.


[...]

I think what Andrei is getting at is, we want to be able to transfer the
compiler's knowledge and implementation of implicit type conversions to
runtime.  I.e., we'd like to do this:

	Variant v;
	v = 100;	// v now stores an int
	...
	long l = v.get!long; // currently doesn't work

Or maybe, a more motivating example:

	T calculate(T)(Variant number) {
		T val = number.get!T;
		return val*2 + 1;
	}

	Variant v;
	v = 100;
	assert(calculate!long(v) == 201L);
	assert(calculate!float(v) == 201.0f);

I.e., you don't know what type you'd like to get from the Variant until
the caller calls you, but you also don't know what type is stored in
Variant until runtime.  Currently, there's a hack in Variant to handle
some of these implicit conversions, but as Andrei says, it's incomplete
and may have some wrong cases.  This knowledge is already in the
compiler; there should be a way to transfer this to Variant at runtime
without having to re-implement implicit conversions in library code.


T

-- 
A mathematician is a device for turning coffee into theorems. -- P. Erdos
Oct 01 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 2:26 PM, H. S. Teoh wrote:

 On Thu, Oct 01, 2020 at 02:08:11PM -0400, Steven Schveighoffer via
Digitalmars-d wrote:

 On 10/1/20 1:35 PM, Andrei Alexandrescu wrote:


 [...]

 How do you implement Variant.get(T) without reifying is(T : U)?


 How do you implement Variant.get(T) *with* reifying is(U : T)?

 step 1: determine using reified constructs that U is convertible to T
 step 2: ?????
 step 3: Return a T from a U!

 BTW, Variant already does a *limited* form of this. reification
 doesn't change what needs to happen.

 I don't see why Variant's needs have to dictate how you need to reason
 about types at compile time in CTFE.


 [...]
 
 I think what Andrei is getting at is, we want to be able to transfer the
 compiler's knowledge and implementation of implicit type conversions to
 runtime.  I.e., we'd like to do this:
 
 	Variant v;
 	v = 100;	// v now stores an int
 	...
 	long l = v.get!long; // currently doesn't work


Again, this does work, but I don't think this completely destroys your 
point.


 Or maybe, a more motivating example:
 
 	T calculate(T)(Variant number) {
 		T val = number.get!T;
 		return val*2 + 1;
 	}
 
 	Variant v;
 	v = 100;
 	assert(calculate!long(v) == 201L);
 	assert(calculate!float(v) == 201.0f);


This currently works as well.

There *are* cases that don't work. But that's a matter of fixing 
ImplicitlyConversionTargets.


 
 I.e., you don't know what type you'd like to get from the Variant until
 the caller calls you, but you also don't know what type is stored in
 Variant until runtime.  Currently, there's a hack in Variant to handle
 some of these implicit conversions, but as Andrei says, it's incomplete
 and may have some wrong cases.  This knowledge is already in the
 compiler; there should be a way to transfer this to Variant at runtime
 without having to re-implement implicit conversions in library code.


That's all fine. The problem I see is:

1. Variant (a true variant that can hold ANY type) is really the only 
use case for runtime type information. Like, literally the only. 
Algebraic types don't have to use reification *at all*. Any reification 
system is still not going to encompass all you can do with a type. Sure, 
you can probably 100% cover type conversion. It will never 100% cover, 
e.g. comparison or type coersion.
2. Compile time is the main focus here. Variant works, and can be made 
better, and we don't have to introduce a new concept to make it work. 
But in CTFE, pulling back the Oz curtain a bit can make things an insane 
amount more pleasant. We don't need to reimplement what the compiler 
does at runtime for this.

-Steve
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 10/1/20 2:40 PM, Steven Schveighoffer wrote:

 1. Variant (a true variant that can hold ANY type) is really the only 
 use case for runtime type information.Like, literally the only.


I agree. But it's also the keystone: Any (let's call it Any...) is the 
only way to do things like creating objects and invoking functions from 
dynamically-loaded library in a sane manner. This has become clearer to 
me when I was wrestling typeid() - the right compile-time introspection 
is the kind that allows you to create runtime layouts safely.


 Algebraic types don't have to use reification *at all*.


Correct. SumType is awesome btw :o).


 Any reification 
 system is still not going to encompass all you can do with a type. Sure, 
 you can probably 100% cover type conversion. It will never 100% cover, 
 e.g. comparison or type coersion.


Coercion seems to be on the same level of difficulty as implicit 
conversions, so probably having one done right means having the other as 
well. As to what the capabilities are, it comes down to what context 
things can be dereified in (i.e. your opening post).


 2. Compile time is the main focus here. Variant works, and can be made 
 better, and we don't have to introduce a new concept to make it work. 
 But in CTFE, pulling back the Oz curtain a bit can make things an insane 
 amount more pleasant. We don't need to reimplement what the compiler 
 does at runtime for this.


At best of course we'd have everything, and py nothing for it. One good 
question is how the quite baroque introspection facilities in std.traits 
can be helped. Things like, give me the function parameters with types 
and modifiers and attributes and all that. A struct that can be 
manipulated during compilation containing all that information would be 
very valuable.
Oct 01 2020
parent Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 18:55:17 UTC, Andrei Alexandrescu 
wrote:

 A struct that can be manipulated during compilation containing 
 all that information would be very valuable.


Indeed, I considered and implemented that a few months ago.
Type functions can store types in structs. Right now.
I think I did demonstrate that somewhere ....
Structs holding types can never be used outside of CTFE of course.
Because a type is not given an ABI.
Oct 01 2020
prev sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Thursday, 1 October 2020 at 18:26:48 UTC, H. S. Teoh wrote:

 On Thu, Oct 01, 2020 at 02:08:11PM -0400, Steven Schveighoffer 
 via Digitalmars-d wrote:

 On 10/1/20 1:35 PM, Andrei Alexandrescu wrote:


 [...]

 How do you implement Variant.get(T) without reifying is(T : 
 U)?


 
 How do you implement Variant.get(T) *with* reifying is(U : T)?
 
 step 1: determine using reified constructs that U is 
 convertible to T
 step 2: ?????
 step 3: Return a T from a U!
 
 BTW, Variant already does a *limited* form of this. 
 reification doesn't change what needs to happen.
 
 I don't see why Variant's needs have to dictate how you need 
 to reason about types at compile time in CTFE.


 [...]

 I think what Andrei is getting at is, we want to be able to 
 transfer the compiler's knowledge and implementation of 
 implicit type conversions to runtime.  I.e., we'd like to do 
 this:


The compiler's knowledge of implicit conversions is a table that 
hasn't changed since 2010.  Why can't you just have a form of 
this table in Phobos?
Oct 02 2020
parent Stefan Koch <uplink.coder gmail.com> writes:
On Friday, 2 October 2020 at 09:57:18 UTC, Iain Buclaw wrote:

 On Thursday, 1 October 2020 at 18:26:48 UTC, H. S. Teoh wrote:

 On Thu, Oct 01, 2020 at 02:08:11PM -0400, Steven Schveighoffer 
 via Digitalmars-d wrote:

 [...]


 [...]

 [...]


 [...]

 I think what Andrei is getting at is, we want to be able to 
 transfer the compiler's knowledge and implementation of 
 implicit type conversions to runtime.  I.e., we'd like to do 
 this:


 The compiler's knowledge of implicit conversions is a table 
 that hasn't changed since 2010.  Why can't you just have a form 
 of this table in Phobos?


That's true for basic types.
If you want to know it as well for alias this.
it's somewhat more tricky :)
Oct 02 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 10/1/20 2:08 PM, Steven Schveighoffer wrote:

 On 10/1/20 1:35 PM, Andrei Alexandrescu wrote:

 On 10/1/20 1:26 PM, Steven Schveighoffer wrote:

 But when the compiler MUST implement is(T : U), and that expression 
 is well defined, I don't see why we have to reimplement that feature 
 in the runtime as well. At least without a compelling example of "the 
 compiler can't do this as well".


 How do you implement Variant.get(T) without reifying is(T : U)?


 
 How do you implement Variant.get(T) *with* reifying is(U : T)?
 
 step 1: determine using reified constructs that U is convertible to T
 step 2: ?????
 step 3: Return a T from a U!


The mechanics of carrying the conversion are also needed, especially if 
a change in format is necessary (e.g. int to long or float to double). 
It looks like a pointer to function needs to be part of the solution.

Nevertheless, the challenge remains - need to list the possible types a 
given type could convert to. Or in some way or another figure out 
whether T converts to U, when only one of them is available during 
compilation.


 BTW, Variant already does a *limited* form of this. reification doesn't 
 change what needs to happen.
 
 I don't see why Variant's needs have to dictate how you need to reason 
 about types at compile time in CTFE.


The more applicability to difficult problems a mechanism has, the more 
useful it is. Looking good on cherry-picked examples is not enough. 
Difficult related problems that we have are introspection, std.meta, 
std.traits, and std.variant. The latter can be considered a problem of 
having the right primitives in std.traits.
Oct 01 2020
next sibling parent Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 18:30:31 UTC, Andrei Alexandrescu 
wrote:

 std.variant. The latter can be considered a problem of having 
 the right primitives in std.traits.


I don't think that std.variant can ever work cleanly.
Either you are a dynamically typed language, or you aren't.
We can approach, dynamism by emulating it in templates, but 
reaching it ... quite probably impossible.
While is an interesting consideration, it's not at all what type 
functions solve.

An accurate definition of what type functions allow you to do, 
and what they are designed to do is:
   - Replacing templates which could be functions with functions.
   - i.e. If the only reason you are using a template instead of a 
function is that functions can't take types or a list of types. 
Then that's in the problem domain of type functions.
Oct 01 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 2:30 PM, Andrei Alexandrescu wrote:


 The mechanics of carrying the conversion are also needed, especially if 
 a change in format is necessary (e.g. int to long or float to double). 
 It looks like a pointer to function needs to be part of the solution.


Yes, Variant already does this. And it does this by a static unrolled 
loop over the actual types. Not the reified types.

So you'd have to build that mechanism into reification to "replace" 
Variant's usage of ImplicitConversionTargets. In other words, you can't 
just hoist this one piece into reification. You need to hoist ALL OF IT.

At some point, you are going to end up with a complete runtime 
reflection system.


 

 BTW, Variant already does a *limited* form of this. reification 
 doesn't change what needs to happen.

 I don't see why Variant's needs have to dictate how you need to reason 
 about types at compile time in CTFE.


 
 The more applicability to difficult problems a mechanism has, the more 
 useful it is. Looking good on cherry-picked examples is not enough. 
 Difficult related problems that we have are introspection, std.meta, 
 std.traits, and std.variant. The latter can be considered a problem of 
 having the right primitives in std.traits.


Variant already has a solution. And it's tailored to Variant. I do not 
consider *at all* the problems Variant has to be related to a nicer 
compile time usage of types inside CTFE.

FWIW, type functions can replace ImplicitConversionTargets quite easily.

-Steve
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 3:08 PM, Steven Schveighoffer wrote:

 On 10/1/20 2:30 PM, Andrei Alexandrescu wrote:
 

 The mechanics of carrying the conversion are also needed, especially 
 if a change in format is necessary (e.g. int to long or float to 
 double). It looks like a pointer to function needs to be part of the 
 solution.


 
 Yes, Variant already does this. And it does this by a static unrolled 
 loop over the actual types. Not the reified types.


Yah, the difficulty is not execution as much as figuring out the correct 
set of types. That's where the discussion started - why redo "is" during 
compilation, turns out Variant needs kinda half of that, etc.


 So you'd have to build that mechanism into reification to "replace" 
 Variant's usage of ImplicitConversionTargets. In other words, you can't 
 just hoist this one piece into reification. You need to hoist ALL OF IT.
 
 At some point, you are going to end up with a complete runtime 
 reflection system.


And a compile-time reflection system that would allow a runtime 
reflection system would be fantastic. Unclear on where the completeness 
should stop, but this has been a D mantra for a long time now - don't 
build runtime reflection into the language because a good compile-time 
reflection can help you build any runtime reflection you want. We are 
sadly still in the early innings of that.


 BTW, Variant already does a *limited* form of this. reification 
 doesn't change what needs to happen.

 I don't see why Variant's needs have to dictate how you need to 
 reason about types at compile time in CTFE.


 The more applicability to difficult problems a mechanism has, the more 
 useful it is. Looking good on cherry-picked examples is not enough. 
 Difficult related problems that we have are introspection, std.meta, 
 std.traits, and std.variant. The latter can be considered a problem of 
 having the right primitives in std.traits.


 
 Variant already has a solution. And it's tailored to Variant. I do not 
 consider *at all* the problems Variant has to be related to a nicer 
 compile time usage of types inside CTFE.


Hmmm... do we want to eliminate a use case because it doesn't fit the 
narrative?


 FWIW, type functions can replace ImplicitConversionTargets quite easily.


A proof of concept would be helpful.
Oct 01 2020
next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 19:33:09 UTC, Andrei Alexandrescu 
wrote:


 A proof of concept would be helpful.


You mean this one:

template ImplicitConversionTargets(T)
{
     static if (is(T == bool))
         alias ImplicitConversionTargets =
             AliasSeq!(byte, ubyte, short, ushort, int, uint, 
long, ulong, CentTypeList,
                        float, double, real, char, wchar, dchar);
     else static if (is(T == byte))
         alias ImplicitConversionTargets =
             AliasSeq!(short, ushort, int, uint, long, ulong, 
CentTypeList,
                        float, double, real, char, wchar, dchar);
     else static if (is(T == ubyte))
         alias ImplicitConversionTargets =
             AliasSeq!(short, ushort, int, uint, long, ulong, 
CentTypeList,
                        float, double, real, char, wchar, dchar);
     else static if (is(T == short))
         alias ImplicitConversionTargets =
             AliasSeq!(int, uint, long, ulong, CentTypeList, 
float, double, real);
     else static if (is(T == ushort))
         alias ImplicitConversionTargets =
             AliasSeq!(int, uint, long, ulong, CentTypeList, 
float, double, real);
     else static if (is(T == int))
         alias ImplicitConversionTargets =
             AliasSeq!(long, ulong, CentTypeList, float, double, 
real);
     else static if (is(T == uint))
         alias ImplicitConversionTargets =
             AliasSeq!(long, ulong, CentTypeList, float, double, 
real);
     else static if (is(T == long))
         alias ImplicitConversionTargets = AliasSeq!(float, 
double, real);
     else static if (is(T == ulong))
         alias ImplicitConversionTargets = AliasSeq!(float, 
double, real);
     else static if (is(cent) && is(T == cent))
         alias ImplicitConversionTargets = AliasSeq!(float, 
double, real);
     else static if (is(ucent) && is(T == ucent))
         alias ImplicitConversionTargets = AliasSeq!(float, 
double, real);
     else static if (is(T == float))
         alias ImplicitConversionTargets = AliasSeq!(double, real);
     else static if (is(T == double))
         alias ImplicitConversionTargets = AliasSeq!real;
     else static if (is(T == char))
         alias ImplicitConversionTargets =
             AliasSeq!(wchar, dchar, byte, ubyte, short, ushort,
                        int, uint, long, ulong, CentTypeList, 
float, double, real);
     else static if (is(T == wchar))
         alias ImplicitConversionTargets =
             AliasSeq!(dchar, short, ushort, int, uint, long, 
ulong, CentTypeList,
                        float, double, real);
     else static if (is(T == dchar))
         alias ImplicitConversionTargets =
             AliasSeq!(int, uint, long, ulong, CentTypeList, 
float, double, real);
     else static if (is(T : typeof(null)))
         alias ImplicitConversionTargets = AliasSeq!(typeof(null));
     else static if (is(T == class))
         alias ImplicitConversionTargets = 
staticMap!(ApplyLeft!(CopyConstness, T), 
TransitiveBaseTypeTuple!(T));
     else static if (isDynamicArray!T && !is(typeof(T.init[0]) == 
const))
     {
        static if (is(typeof(T.init[0]) == shared))
            alias ImplicitConversionTargets =
            AliasSeq!(const(shared(Unqual!(typeof(T.init[0]))))[]);
        else
            alias ImplicitConversionTargets =
            AliasSeq!(const(Unqual!(typeof(T.init[0])))[]);
     }
     else static if (is(T : void*))
         alias ImplicitConversionTargets = AliasSeq!(void*);
     else
         alias ImplicitConversionTargets = AliasSeq!();
}

 Andrei could you explain the semantics in detail?
The source code is somewhat obtuse.
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 3:37 PM, Stefan Koch wrote:

 
  Andrei could you explain the semantics in detail?
 The source code is somewhat obtuse.


I don't understand the question. The code is there and also publicly 
documented. Are you asking me to read it for you?
Oct 01 2020
next sibling parent Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 19:40:47 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/20 3:37 PM, Stefan Koch wrote:

 
  Andrei could you explain the semantics in detail?
 The source code is somewhat obtuse.


 I don't understand the question. The code is there and also 
 publicly documented. Are you asking me to read it for you?


Ah I was looking at the code only not at the documentation.
Which for properly written D code is usually enough.

`An $(REF AliasSeq,std,meta) with all possible target types of an 
implicit
     conversion `T`.`

is it an implicit conversion from T ?

Yes anything else wouldn't make sense.
Oct 01 2020
prev sibling parent Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 19:40:47 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/20 3:37 PM, Stefan Koch wrote:

 
  Andrei could you explain the semantics in detail?
 The source code is somewhat obtuse.


 I don't understand the question. The code is there and also 
 publicly documented. Are you asking me to read it for you?


It'll need a __trait for unqual but that should be about it.
The reason being that Unqual! is polymorphic.

Should be doable.
Oct 01 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/1/20 3:33 PM, Andrei Alexandrescu wrote:

 On 10/1/20 3:08 PM, Steven Schveighoffer wrote:

 On 10/1/20 2:30 PM, Andrei Alexandrescu wrote:


 The mechanics of carrying the conversion are also needed, especially 
 if a change in format is necessary (e.g. int to long or float to 
 double). It looks like a pointer to function needs to be part of the 
 solution.


 Yes, Variant already does this. And it does this by a static unrolled 
 loop over the actual types. Not the reified types.


 
 Yah, the difficulty is not execution as much as figuring out the correct 
 set of types. That's where the discussion started - why redo "is" during 
 compilation, turns out Variant needs kinda half of that, etc.


My point is that it still needs the list in dereified form. Which you 
can't do at runtime.


 And a compile-time reflection system that would allow a runtime 
 reflection system would be fantastic. Unclear on where the completeness 
 should stop, but this has been a D mantra for a long time now - don't 
 build runtime reflection into the language because a good compile-time 
 reflection can help you build any runtime reflection you want. We are 
 sadly still in the early innings of that.


Yeah, a full runtime reflection system would be really interesting to 
build! I don't know if it's sad that we haven't built one, or if it's 
more indicative that we don't need one.


 The more applicability to difficult problems a mechanism has, the 
 more useful it is. Looking good on cherry-picked examples is not 
 enough. Difficult related problems that we have are introspection, 
 std.meta, std.traits, and std.variant. The latter can be considered a 
 problem of having the right primitives in std.traits.


 Variant already has a solution. And it's tailored to Variant. I do not 
 consider *at all* the problems Variant has to be related to a nicer 
 compile time usage of types inside CTFE.


 
 Hmmm... do we want to eliminate a use case because it doesn't fit the 
 narrative?


It doesn't fit the scope of the problem. The problem is: I want to write 
compile-time type manipulation in a runtime style. Not that I want to do 
everything that I can do at compile time at runtime (in this case at 
*actual* runtime).


 FWIW, type functions can replace ImplicitConversionTargets quite easily.


 
 A proof of concept would be helpful.


I imagine the first section which is comparing just builtin types, you 
can do like:

alias[] types = [int, char, ...];
import std.algorithm : filter;
import std.array : array;
return types.filter!((alias a) => is(t : a)).array;

The other ones would be similar but use a different way to generate the 
types other than filtering a finite list.

-Steve
Oct 01 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 3:57 PM, Steven Schveighoffer wrote:

 I don't know if it's sad that we haven't built one, or if it's more 
 indicative that we don't need one.


We definitely need one, and generally means to generate boilerplate for 
interfacing with other languages. That we don't have a solid one, and 
not for the lack of trying, is indicative that we don't have the right 
ingredients.
Oct 01 2020
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 1 October 2020 at 09:52:19 UTC, user1234 wrote:

 That's sad. I start believing to TypeFunction, you see I'm not 
 like those who's been immediatly enthusisast, but well, if the 
 boss shortcut your work with workarounds (surface changes as


It is interesting to folly what Stefan is trying to achieve, 
introducing type variables is something I have argued for in the 
past.

Although, if you look at all the different discussions that take 
place over time, you might argue that a more comprehensive AST 
interface would solve more issues as language changes are being 
resisted quite heavily at this point.
Oct 01 2020
next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 12:08:40 UTC, Ola Fosheim Grøstad 
wrote:

 On Thursday, 1 October 2020 at 09:52:19 UTC, user1234 wrote:

 That's sad. I start believing to TypeFunction, you see I'm not 
 like those who's been immediatly enthusisast, but well, if the 
 boss shortcut your work with workarounds (surface changes as


 It is interesting to folly what Stefan is trying to achieve, 
 introducing type variables is something I have argued for in 
 the past.

 Although, if you look at all the different discussions that 
 take place over time, you might argue that a more comprehensive 
 AST interface would solve more issues as language changes are 
 being resisted quite heavily at this point.


I kept myself from that.
Because I know that AST macros are frowned upon.
(Although what Walter is recently talking about sounds more and 
more like them)
And because I want to make it easy to migrate from templates.
to static template emulation functions (stef) :)

I do honestly believe that type functions fix a bunch of our 
problems.
And with very little code change, compared to the power they have.
Oct 01 2020
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 1 October 2020 at 12:17:59 UTC, Stefan Koch wrote:

 I do honestly believe that type functions fix a bunch of our 
 problems.
 And with very little code change, compared to the power they 
 have.


Maybe you can do it in such a way that it later can be viewed as 
syntactic sugar for an AST-interfacing mechanism. At least 
something to think about.

Also, when introducing new mechanisms a useful exercise can be to 
try to find existing mechanisms that can be implemented with it. 
If so then one can argue that the core language became simpler 
with the addition (so in the long term the compiler can be 
simplified also).
Oct 01 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 12:32:47 UTC, Ola Fosheim Grøstad 
wrote:

 On Thursday, 1 October 2020 at 12:17:59 UTC, Stefan Koch wrote:

 I do honestly believe that type functions fix a bunch of our 
 problems.
 And with very little code change, compared to the power they 
 have.


 Maybe you can do it in such a way that it later can be viewed 
 as syntactic sugar for an AST-interfacing mechanism. At least 
 something to think about.

 Also, when introducing new mechanisms a useful exercise can be 
 to try to find existing mechanisms that can be implemented with 
 it. If so then one can argue that the core language became 
 simpler with the addition (so in the long term the compiler can 
 be simplified also).


It's an argument.
But I would like to argue my actual use-case first.

This doesn't make the language any simpler internally.
But it does make some code that people write take a fourth of the 
code in the binary.
And we get around all the funky things that happen with template 
emission.
(It's horribly broken right now)

Type functions are a restricted sub language which tries to 
replace common templates, (which should not ever have been 
templates in the first place).
It does not go for internal orthogonality.
It goes for making a very common usage patterns (at least in all 
the big codebases I looked at, it common)
Both simpler, and more performant.

It's placed between templates and functions.
If the you use templates only to reason about types, you can now 
use a function and be sure it won't leave crap in your binary, if 
nothing else.

I don't see the current approaches being able to guarantee that.
But maybe I am wrong?

I also don't see D coming to a slim core. And frankly I don't see 
why it needs to.
D does not cater to purists, it caters to people who value choice 
and creativity.
At least as far as I see it.
At least that is who I am.
Oct 01 2020
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 1 October 2020 at 12:47:08 UTC, Stefan Koch wrote:

 It's an argument.
 But I would like to argue my actual use-case first.


Certainly, I was not discouraging your efforts. Simply trying to 
point out that it might be easier to convince sceptics if you can 
argue that your concept is a cog in the existing machinery or 
that it can be seen as syntactic sugar over desired cogs in a 
future revision of the language. You don't want sceptics to see 
your concept as sitting on a limb.



 It goes for making a very common usage patterns (at least in 
 all the big codebases I looked at, it common)


Yes, this is a good argument. Strengthening existing community 
practice. Making learning/reading easier.



 I also don't see D coming to a slim core. And frankly I don't 
 see why it needs to.
 D does not cater to purists, it caters to people who value 
 choice and creativity.
 At least as far as I see it.
 At least that is who I am.


I think you are right. Powerful features like the compiles-trait 
could make it virtually impossible to streamline language 
concepts in a non-breaking fashion as libraries might start to 
break when corner cases are removed.

Also there are recent additions that support your view. E.g. the 
addition of shared semantics which basically is just 
boxing/unboxing should have been possible as a library construct. 
Yet Manu had found it difficult to do as a pure library construct 
and wanted to see it implemented sooner than later, so D got 
another "limb feature". But there was little resistance to this 
feature because "shared" was already perceived by the community 
as being a language feature (although I would argue that it 
wasn't, it was more of a social construct). If a slim language 
was a goal then it would have been done as a library feature and 
"shared" would just have been syntactical sugar for that library 
feature.

Another vector you might use to argue for your feature is that 
reification/mangling is making future language evolution more 
difficult than your solution. At least, that is my understanding. 
Maybe I am wrong? I could be. But you might find some argument 
you can use there if your concept has a higher abstraction level 
than the alternatives. Which is generally desirable in a language 
as it is better for modelling. And D is marketed as being good 
for modelling, so that would be an argument if favour of your 
idea.

If people argue that your feature could be done using the 
existing language/reification then you should be able to argue 
that your feature is much needed syntactical sugar over such 
constructs. Basically, that you simply are creating an 
abstraction over existing features and practice that improves on 
the usability of the language.

Compiler performance issues should be argued as being 
optimizations. Abstractions often makes optimizations easier to 
implement as less static analysis is needed (the search space is 
smaller). So even though you might see your idea as being a 
compiler addition, you should be able to argue that it isn't 
really, but that it would be beneficial to implement it as such.

Anyway, I am interested to see where you go with this. :-)
Oct 01 2020
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 1 October 2020 at 12:08:40 UTC, Ola Fosheim Grøstad 
wrote:

 On Thursday, 1 October 2020 at 09:52:19 UTC, user1234 wrote:

 That's sad. I start believing to TypeFunction, you see I'm not 
 like those who's been immediatly enthusisast, but well, if the 
 boss shortcut your work with workarounds (surface changes as


 It is interesting to folly what Stefan is trying to achieve,


«follow» (how did that typo happen?)
Oct 01 2020
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 1 October 2020 at 08:57:12 UTC, Stefan Koch wrote:

 At those scales there is no difference in compile time.
 However there is a difference in file size.


Your assert(__ctfe) PR should fix this.
Oct 01 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Thursday, 1 October 2020 at 12:39:45 UTC, Adam D. Ruppe wrote:

 On Thursday, 1 October 2020 at 08:57:12 UTC, Stefan Koch wrote:

 At those scales there is no difference in compile time.
 However there is a difference in file size.


 Your assert(__ctfe) PR should fix this.


Not sure ... not all of it.
0000000000000000 r _TMP0
0000000000000002 r _TMP1
0000000000000038 r _TMP10
000000000000003d r _TMP11
0000000000000042 r _TMP12
0000000000000048 r _TMP13
000000000000004e r _TMP14
0000000000000052 r _TMP15
0000000000000059 r _TMP16
000000000000005d r _TMP17
0000000000000062 r _TMP18
0000000000000067 r _TMP19
0000000000000008 r _TMP2
000000000000000f r _TMP3
0000000000000016 r _TMP4
000000000000001e r _TMP5
0000000000000023 r _TMP6
0000000000000029 r _TMP7
000000000000002f r _TMP8
0000000000000036 r _TMP9

I think those are string literals.
which the static foreach version generates.
Those won't go away, it's hard to proof they are not used within 
the current framework of dmd.
Oct 01 2020
parent Araq <rumpf_a web.de> writes:
On Thursday, 1 October 2020 at 13:41:58 UTC, Stefan Koch wrote:

 I think those are string literals.
 which the static foreach version generates.
 Those won't go away, it's hard to proof they are not used 
 within the current framework of dmd.


So use a linker that can remove unused symbols.
Oct 01 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2020 1:57 AM, Stefan Koch wrote:

 However there is a difference in file size.
 
 Type function version:
 
 stat  makeConvMatrix.o
    File: 'makeConvMatrix.o'
    Size: 2876
 
 VS template version
 
 stat  makeConvMatrix_tmpl.o
    File: 'makeConvMatrix_tmpl.o'
    Size: 11760


Indeed there is, as the compiler generates code for the function and writes the 
code to the object file. The code it generates is placed into a COMDAT section 
which, since it is unreferenced, is supposed to be elided by the linker.

However, many linkers don't seem to do that, and besides, the compiler is 
wasting effort generating unused code. This is a more general inefficiency in 
the design of the compiler internals, and is not necessarily a language design 
defect.

Internally, there is a function called `needsCodegen()` which decides if a 
template instantiation needs code generated for it (duh!). It clearly errs on 
the side of being a bit too conservative.

I suggest an improvement to it where it will return false if the only use of a 
template instantiation is in CTFE. If that is unimplementable (the separate 
compilation model may make it impractical), a fallback position is to have a 
pragma or some other mechanism that says "don't generate code for this
function".

I suspect such a compiler improvement would produce considerable benefit to 
existing template-heavy code.
Oct 01 2020
next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 1 October 2020 at 21:34:54 UTC, Walter Bright wrote:

 I suggest an improvement to it where it will return false if 
 the only use of a template instantiation is in CTFE. If that is 
 unimplementable (the separate compilation model may make it 
 impractical), a fallback position is to have a pragma or some 
 other mechanism that says "don't generate code for this 
 function".


Yes! Stefan actually already wrote this too

https://github.com/dlang/dmd/pull/11007

just a few minor tweaks and it can work to help out. The 
`assert(__ctfe)` pattern is already used in some code in the wild.
Oct 01 2020
prev sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 1 October 2020 at 21:34:54 UTC, Walter Bright wrote:

 On 10/1/2020 1:57 AM, Stefan Koch wrote:

 However there is a difference in file size.
 
 Type function version:
 
 stat  makeConvMatrix.o
    File: 'makeConvMatrix.o'
    Size: 2876
 
 VS template version
 
 stat  makeConvMatrix_tmpl.o
    File: 'makeConvMatrix_tmpl.o'
    Size: 11760


 Indeed there is, as the compiler generates code for the 
 function and writes the code to the object file. The code it 
 generates is placed into a COMDAT section which, since it is 
 unreferenced, is supposed to be elided by the linker.


Linkers are not under our control.
The compiler is.


 However, many linkers don't seem to do that, and besides, the 
 compiler is wasting effort generating unused code. This is a 
 more general inefficiency in the design of the compiler 
 internals, and is not necessarily a language design defect.


So you are saying having a mechanism to replace a macro/template 
with a regular function
does not address a language issue?


 Internally, there is a function called `needsCodegen()` which 
 decides if a template instantiation needs code generated for it 
 (duh!). It clearly errs on the side of being a bit too 
 conservative.


I am intimately familiar with needsCodegen() (- and I still don't 
fully understand it)
It leads to a stack overflow within the compiler when compiling 
one our apps with -allinst.
Atila tries to fix it, and it has taken him a couple of weeks 
already.
Right now the likely solution is to make it much more 
conservative, because that's the only thing we can think of to 
avoid linker errors.


 I suggest an improvement to it where it will return false if 
 the only use of a template instantiation is in CTFE. If that is 
 unimplementable (the separate compilation model may make it 
 impractical), a fallback position is to have a pragma or some 
 other mechanism that says "don't generate code for this 
 function".


Been there. Done that.


 I suspect such a compiler improvement would produce 
 considerable benefit to existing template-heavy code.


It requires the programmer to decide if a function needs to be 
elided which in a template, he may not be able to tell.
Because it may depend on which instance is produced.
Oct 01 2020
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Does the same thing, works today:

string makeConvMatrix(T...)()
{
     string result;
     static foreach(t; T)
     {
         result ~= "\t" ~ t.stringof;
     }
     result ~= "\n";
     static foreach(t1; T)
     {
         result ~= t1.stringof;
         static foreach(t2; T)
         {
             result ~=  "\t" ~ (is(t1:t2) ? "yes" : "no");
         }
         result ~= "\n";
     }
     return result;
}

void main()
{
     import core.stdc.stdio;
     static immutable convMatrix = makeConvMatrix!(byte, ubyte, short, ushort, 
int, uint, long, ulong)();

     printf("%s\n", convMatrix.ptr);
}
Oct 01 2020
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2020 2:04 PM, Walter Bright wrote:

 Does the same thing, works today:


Yes, I see another message posted the same thing.
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 5:22 PM, Walter Bright wrote:

 On 10/1/2020 2:04 PM, Walter Bright wrote:

 Does the same thing, works today:


 
 Yes, I see another message posted the same thing.


Separation of concerns applies to ctfe code as well. The function should 
just fetch the bool matrix. Formatting should not be mixed with that, 
but rather done later either at compile- or run-time:

auto makeConvMatrix(Ts...)() {
     bool[T.length[T.length] result;
     static foreach (i, T : Ts)
         static foreach (j, U : Ts)
             result[i][j] = is(T : U);
     return result;
}
Oct 01 2020
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 2 October 2020 at 02:27:40 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/20 5:22 PM, Walter Bright wrote:

 On 10/1/2020 2:04 PM, Walter Bright wrote:

 Does the same thing, works today:


 
 Yes, I see another message posted the same thing.


 Separation of concerns applies to ctfe code as well. The 
 function should just fetch the bool matrix. Formatting should 
 not be mixed with that, but rather done later either at 
 compile- or run-time:


I reserve the right to post the example that I post.
Thank you very much.

For the code I posted the type function produces a binary which is
  - 4 times smaller than the template,
  - a lot has less symbols
  - and works at betterC.
  - does not relay on static foreach expansion
Oct 01 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2020 7:27 PM, Andrei Alexandrescu wrote:

 auto makeConvMatrix(Ts...)() {
      bool[T.length[T.length] result;
      static foreach (i, T : Ts)
          static foreach (j, U : Ts)
              result[i][j] = is(T : U);
      return result;
 }


This piece of code may be useful in re-implementing std.meta.MostDerived.
Oct 02 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Friday, 2 October 2020 at 09:51:10 UTC, Walter Bright wrote:

 On 10/1/2020 7:27 PM, Andrei Alexandrescu wrote:

 auto makeConvMatrix(Ts...)() {
      bool[T.length[T.length] result;
      static foreach (i, T : Ts)
          static foreach (j, U : Ts)
              result[i][j] = is(T : U);
      return result;
 }


 This piece of code may be useful in re-implementing 
 std.meta.MostDerived.


Doesn't seem like it.
The code results in a parser error.
Oct 02 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 10/2/20 6:51 AM, Stefan Koch wrote:

 On Friday, 2 October 2020 at 09:51:10 UTC, Walter Bright wrote:

 On 10/1/2020 7:27 PM, Andrei Alexandrescu wrote:

 auto makeConvMatrix(Ts...)() {
      bool[T.length[T.length] result;
      static foreach (i, T : Ts)
          static foreach (j, U : Ts)
              result[i][j] = is(T : U);
      return result;
 }


 This piece of code may be useful in re-implementing std.meta.MostDerived.


 
 Doesn't seem like it.
 The code results in a parser error.


So many errors. Deficiency of copypasta strikes again.

auto makeConvMatrix(Ts...)() {
      bool[Ts.length][Ts.length] result;
      static foreach (i, T ; Ts)
          static foreach (j, U ; Ts)
              result[i][j] = is(T : U);
      return result;
}

That won't really help much with MostDerived and friends - as Iain said, 
for known types the conversions are well known.
Oct 02 2020
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 1 October 2020 at 21:04:29 UTC, Walter Bright wrote:

 Does the same thing, works today:


I think it is actually a mistake to focus too much on types.

The "type function" concept might be useful if it provides a way 
to work with compiler tuples in general. Then it might be able to 
do things that are basically impossible right now (well at least 
not without a dose of string mixin) and do it efficiently.

I've been playing with a variety of other implementations and 
they keep ending up slow. A tuple foreach unrolls into a bunch of 
code. Perhaps this can be further optimized with a few tweaks:

1) make a function an any temporaries coming from it as CTFE only 
so it isn't codegen'd, optimized in any way, or emitted to the 
object file.

2) maybe borrow part of Stefan's implementation to avoid 
unrolling certain tuple looks so it runs faster. We already have 
`foreach(alias t; T)` so maybe it can just be optimized.

But then we still have the difficulty of translating the result 
tuple back into something we can use. The only facilities the 
language provides right now are some kind of recursive template 
and/or some kind of string mixin.

These get expensive again (especially for long and/or varied 
argument lists) and may hit other arbitrary limitations like 
runtime/compile time barriers. And it is a bit obnoxious because 
the compiler already knows all this stuff, but instead of saying 
"reuse that please" we have to say "turn that into a string, then 
turn the string into a template, then turn the template into a 
tuple".

So some facility to turn a ctfe array back into a tuple - the 
dereifiy step basically - with compiler assistance I think will 
be practical.


And you know, I'm still a little skeptical of the type functions, 
but I do think they have potential when we start looking at them 
as tuple manipulators more than as template replacements. Tuples 
are this weird thing that are half in the language and half 
not... and bringing them more in would actually be pretty nice.

Would it solve Variant's needs? Well ... I think that's separate, 
we actually can do that in today's D. You can make a list of 
conversion targets in opAssign, make one in get, and compare them 
at runtime. Maybe I'll implement later but she's fussing again 
ttyl
Oct 01 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Thursday, 1 October 2020 at 21:34:50 UTC, Adam D. Ruppe wrote:

 So some facility to turn a ctfe array back into a tuple - the 
 dereifiy step basically - with compiler assistance I think will 
 be practical.


What if we could just mutate tuples locally?

template staticMap(alias F, Args...) {
     static foreach(ref Arg; Args)
         Arg = F!Arg;
     alias staticMap = Args;
}

The obvious objection is "the spec says order of declarations 
doesn't matter", but this is a case where the spec is just 
flat-out wrong (see previous discussion of "compile-time race 
conditions" [1]). So, given that declaration order is in fact 
significant in D, we may as well officially acknowledge it and 
reap the benefits.

[1] 
https://forum.dlang.org/thread/swbmgrhtoiqtqokmqniu forum.dlang.org?page=2#post-rccfbm:24sjv:241:40digitalmars.com
Oct 01 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 7:10 PM, Paul Backus wrote:

 On Thursday, 1 October 2020 at 21:34:50 UTC, Adam D. Ruppe wrote:

 So some facility to turn a ctfe array back into a tuple - the dereifiy 
 step basically - with compiler assistance I think will be practical.


 
 What if we could just mutate tuples locally?
 
 template staticMap(alias F, Args...) {
      static foreach(ref Arg; Args)
          Arg = F!Arg;
      alias staticMap = Args;
 }


Interesting - this is akin to D's relaxed purity whereby a pure function 
can mutate its arguments. I'm trying to wrap my head around it, e.g. is 
Args considered a private copy of the argument much like a static array 
passed by value?


 The obvious objection is "the spec says order of declarations doesn't 
 matter", but this is a case where the spec is just flat-out wrong (see 
 previous discussion of "compile-time race conditions" [1]). So, given 
 that declaration order is in fact significant in D, we may as well 
 officially acknowledge it and reap the benefits.
 
 [1] 
 https://forum.dlang.org/thread/swbmgrhtoiqtqokmqniu forum.dlang.org?page=2#post-rccfbm:24sjv:2
1:40digitalmars.com 


I don't get the interaction. How does mutating the argument affect order 
of declaration?
Oct 01 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 October 2020 at 02:21:03 UTC, Andrei Alexandrescu 
wrote:

 template staticMap(alias F, Args...) {
      static foreach(ref Arg; Args)
          Arg = F!Arg;
      alias staticMap = Args;


 I don't get the interaction. How does mutating the argument 
 affect order of declaration?


Is typeof(Args[0]) in there the original arg or the mapped arg?

In a normal function, there's a distinct before-and-after that, 
in theory, doesn't exist outside functions (you can use stuff 
before they are lexically declared)... but even there, you cannot 
change the type of an already existing variable.

This would mean you can. Would certainly be strange.... but might 
work.
Oct 01 2020
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/1/20 11:38 PM, Adam D. Ruppe wrote:

 On Friday, 2 October 2020 at 02:21:03 UTC, Andrei Alexandrescu wrote:

 template staticMap(alias F, Args...) {
      static foreach(ref Arg; Args)
          Arg = F!Arg;
      alias staticMap = Args;


 I don't get the interaction. How does mutating the argument affect 
 order of declaration?


 
 Is typeof(Args[0]) in there the original arg or the mapped arg?
 
 In a normal function, there's a distinct before-and-after that, in 
 theory, doesn't exist outside functions (you can use stuff before they 
 are lexically declared)... but even there, you cannot change the type of 
 an already existing variable.
 
 This would mean you can. Would certainly be strange.... but might work.


That's a rather large change - ref to aliases and such. Maybe a more 
basic facility such as appending to a tuple is simpler?


template staticMap(alias F, Args...) {
     alias Result = AliasSeq!();
     static foreach(Arg; Args)
         Result ~= F!Arg;
     alias staticMap = Result;
}

Now indeed order of evaluation concerns are rather apparent...

This should work with Filter, Reverse etc. as well. Not with 
DerivedToFront (unless a different algo such as mergesort is used).
Oct 01 2020
parent Stefan Koch <uplink.coder gmail.com> writes:
On Friday, 2 October 2020 at 04:05:00 UTC, Andrei Alexandrescu 
wrote:

 That's a rather large change - ref to aliases and such. Maybe a 
 more basic facility such as appending to a tuple is simpler?


I doubt it.
Compiler Tuples are somewhat hard to model in a polymorphic 
context.
They are essentially monads because they can't be modified.
Also you cannot preallocate a tuple.
And they can only work inside polymorphic contexts.

To be able to append to them you quite likely need strict order 
of declaration rules, as well as changes to D's compilation model 
as a whole.

Type functions or "static tuple emission functions" represent an 
addition, rather than a modification.
Which should make it much more tractable, to reason about them 
than to reason about large scale language changes.

YMMV of course.
Oct 01 2020
prev sibling parent Paul Backus <snarwin gmail.com> writes:
On Friday, 2 October 2020 at 03:38:37 UTC, Adam D. Ruppe wrote:

 On Friday, 2 October 2020 at 02:21:03 UTC, Andrei Alexandrescu 
 wrote:

 template staticMap(alias F, Args...) {
      static foreach(ref Arg; Args)
          Arg = F!Arg;
      alias staticMap = Args;


 I don't get the interaction. How does mutating the argument 
 affect order of declaration?


 Is typeof(Args[0]) in there the original arg or the mapped arg?

 In a normal function, there's a distinct before-and-after that, 
 in theory, doesn't exist outside functions (you can use stuff 
 before they are lexically declared)... but even there, you 
 cannot change the type of an already existing variable.

 This would mean you can. Would certainly be strange.... but 
 might work.


What you get from typeof(Args[0]) would depend on when it's 
evaluated relative to the other declarations in the template 
body. If it's evaluated before Args[0] is mutated (really, 
re-bound), you get the type of the original arg; if it's 
evaluated after, you get the type of the mapped arg.

Obviously the language spec would have to spell out what the 
order of evaluation is. But that's something it ought to do 
anyway, regardless of this proposal.
Oct 01 2020
prev sibling parent Paul Backus <snarwin gmail.com> writes:
On Friday, 2 October 2020 at 02:21:03 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/20 7:10 PM, Paul Backus wrote:

 On Thursday, 1 October 2020 at 21:34:50 UTC, Adam D. Ruppe 
 wrote:

 So some facility to turn a ctfe array back into a tuple - the 
 dereifiy step basically - with compiler assistance I think 
 will be practical.


 
 What if we could just mutate tuples locally?
 
 template staticMap(alias F, Args...) {
      static foreach(ref Arg; Args)
          Arg = F!Arg;
      alias staticMap = Args;
 }


 Interesting - this is akin to D's relaxed purity whereby a pure 
 function can mutate its arguments. I'm trying to wrap my head 
 around it, e.g. is Args considered a private copy of the 
 argument much like a static array passed by value?


Kind of. Args is a private copy, not of the arguments themselves, 
but of a list of *references* to the arguments. (In DMD terms, 
TemplateInstance.tiargs an array of pointers to RootObjects.) So 
mutating it just means re-binding the references to point to new 
things.


 I don't get the interaction. How does mutating the argument 
 affect order of declaration?


You're right; it doesn't matter in this case. It would matter if 
you wanted to write a `staticFold`, though:

template staticFold(alias F, Args...)
     if (Args.length > 0)
{
     static foreach (Arg; Args[1 .. $])
         Args[0] = F!(Init, Arg);
     alias staticFold = Args[0];
}
Oct 01 2020
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 1 October 2020 at 21:04:29 UTC, Walter Bright wrote:

 Does the same thing, works today:

 string makeConvMatrix(T...)()
 {
     string result;
     static foreach(t; T)
     {
         result ~= "\t" ~ t.stringof;
     }
     result ~= "\n";
     static foreach(t1; T)
     {
         result ~= t1.stringof;
         static foreach(t2; T)
         {
             result ~=  "\t" ~ (is(t1:t2) ? "yes" : "no");
         }
         result ~= "\n";
     }
     return result;
 }

 void main()
 {
     import core.stdc.stdio;
     static immutable convMatrix = makeConvMatrix!(byte, ubyte, 
 short, ushort, int, uint, long, ulong)();

     printf("%s\n", convMatrix.ptr);
 }


This code produces a binary which
  - is 4 times biggger
  - has more symobls.
  - doesn't work for -betterC

See: 
https://forum.dlang.org/post/wdaamjqqvjlnwaopamzn forum.dlang.org
Where I address the differences.
Oct 01 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 October 2020 at 03:11:34 UTC, Stefan Koch wrote:

  - doesn't work for -betterC


This is trivially easy to fix. Wrap the function in a template 
and use an eponymous enum to collapse it to a literal:


----
template makeConvMatrix(T...) { // wrapper added
         string helper()
         {
             string result;
             static foreach(t; T)
             {
                 result ~= "\t" ~ t.stringof;
             }
             result ~= "\n";
             static foreach(t1; T)
             {
                 result ~= t1.stringof;
                 static foreach(t2; T)
                 {
                     result ~=  "\t" ~ (is(t1:t2) ? "yes" : "no");
                 }
                 result ~= "\n";
             }
             return result;
         }

         enum makeConvMatrix = helper(); // eponymous call
}

extern(C) // for betterC
void main()
{
     import core.stdc.stdio;
     static immutable convMatrix = makeConvMatrix!(byte, ubyte, 
short, ushort, int, uint, long, ulong); // no more () there

     printf("%s\n", convMatrix.ptr);
}
------

Generates a reasonably small executable too (this pattern btw is 
what my change to dmd a couple months ago is able to recognize):

$ ls -lh bc
-rwxr-xr-x 1 me users 19K Oct  1 23:19 bc
$ ls -lh bc.o
-rw-r--r-- 1 me users 2.4K Oct  1 23:19 bc.o
$ nm bc.o
0000000000000000 t
0000000000000000 D _D2bc4mainUZ10convMatrixyAa
                  U _GLOBAL_OFFSET_TABLE_
0000000000000000 r _TMP0
0000000000000000 W main
                  U printf


Which is identical to if I delete the template from the source 
entirely and just replace it with a string literal. Nothing of it 
is emitted to the object file, so the linker doesn't even have to 
strip it.

Worth noting that not all cases work out this well. But this one 
actually does.
Oct 01 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Friday, 2 October 2020 at 03:26:57 UTC, Adam D. Ruppe wrote:

 On Friday, 2 October 2020 at 03:11:34 UTC, Stefan Koch wrote:

  - doesn't work for -betterC


 This is trivially easy to fix. Wrap the function in a template 
 and use an eponymous enum to collapse it to a literal:


 ----
 template makeConvMatrix(T...) { // wrapper added
         string helper()
         {
             string result;
             static foreach(t; T)
             {
                 result ~= "\t" ~ t.stringof;
             }
             result ~= "\n";
             static foreach(t1; T)
             {
                 result ~= t1.stringof;
                 static foreach(t2; T)
                 {
                     result ~=  "\t" ~ (is(t1:t2) ? "yes" : 
 "no");
                 }
                 result ~= "\n";
             }
             return result;
         }

         enum makeConvMatrix = helper(); // eponymous call
 }

 extern(C) // for betterC
 void main()
 {
     import core.stdc.stdio;
     static immutable convMatrix = makeConvMatrix!(byte, ubyte, 
 short, ushort, int, uint, long, ulong); // no more () there

     printf("%s\n", convMatrix.ptr);
 }
 ------

 Generates a reasonably small executable too (this pattern btw 
 is what my change to dmd a couple months ago is able to 
 recognize):

 $ ls -lh bc
 -rwxr-xr-x 1 me users 19K Oct  1 23:19 bc
 $ ls -lh bc.o
 -rw-r--r-- 1 me users 2.4K Oct  1 23:19 bc.o
 $ nm bc.o
 0000000000000000 t
 0000000000000000 D _D2bc4mainUZ10convMatrixyAa
                  U _GLOBAL_OFFSET_TABLE_
 0000000000000000 r _TMP0
 0000000000000000 W main
                  U printf


 Which is identical to if I delete the template from the source 
 entirely and just replace it with a string literal. Nothing of 
 it is emitted to the object file, so the linker doesn't even 
 have to strip it.

 Worth noting that not all cases work out this well. But this 
 one actually does.


You still create unnecessary symbols.
Just use the template twice with different arguments.

--
     static immutable convMatrix = makeConvMatrix!(byte, ubyte, 
short, ushort, int, uint, long, ulong); // no more () there
     static immutable convMatrix2 = makeConvMatrix!(ubyte, short, 
ushort, int, uint, long, ulong); // no more () there

     printf("%s\n", convMatrix.ptr);
     printf("%s\n", convMatrix2.ptr);
--

0000000000000000 t
0000000000000008 R _D2t412__ModuleInfoZ
0000000000000000 D _D2t44mainUZ10convMatrixyAa
0000000000000010 D _D2t44mainUZ11convMatrix2yAa
                  U _d_dso_registry
                  U _GLOBAL_OFFSET_TABLE_
0000000000000000 W main
                  U printf
                  U __start_minfo
                  U __stop_minfo
0000000000000000 r _TMP0
0000000000000142 r _TMP1

uplink uplink-black:~$ ls -lh t4.o
-rw-r--r-- 1 uplink uplink 4,5K Okt  2 06:51 t4.o
Oct 01 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 October 2020 at 04:53:12 UTC, Stefan Koch wrote:

 You still create unnecessary symbols.


Not true.

extern(C)
void main()
{
     import core.stdc.stdio;
     static immutable convMatrix = "pretend matrix";
     static immutable convMatrix2 = "pretend matrix 2";

     printf("%s\n", convMatrix.ptr);
     printf("%s\n", convMatrix2.ptr);
}


No template in sight.

$ dmd -betterC bc
$ nm bc.o
0000000000000000 t
0000000000000000 D _D2bc4mainUZ10convMatrixyAa
0000000000000010 D _D2bc4mainUZ11convMatrix2yAa
                  U _GLOBAL_OFFSET_TABLE_
0000000000000000 r _TMP0
000000000000000f r _TMP1
0000000000000000 W main
                  U printf


Or without -betterC

$ dmd bc.d
$ nm bc.o
0000000000000000 t
0000000000000008 R _D2bc12__ModuleInfoZ
0000000000000000 D _D2bc4mainUZ10convMatrixyAa
0000000000000010 D _D2bc4mainUZ11convMatrix2yAa
                  U _GLOBAL_OFFSET_TABLE_
0000000000000000 r _TMP0
000000000000000f r _TMP1
                  U __start_minfo
                  U __stop_minfo
                  U _d_dso_registry
0000000000000000 W main
                  U printf


 Just use the template twice with different arguments.

 0000000000000000 t
 0000000000000008 R _D2t412__ModuleInfoZ
 0000000000000000 D _D2t44mainUZ10convMatrixyAa
 0000000000000010 D _D2t44mainUZ11convMatrix2yAa
                  U _d_dso_registry
                  U _GLOBAL_OFFSET_TABLE_
 0000000000000000 W main
                  U printf
                  U __start_minfo
                  U __stop_minfo
 0000000000000000 r _TMP0
 0000000000000142 r _TMP1




Absolutely identical. An eponymous template resolving to an enum 
of a common type is indistinguishable to a literal of that type. 
It does *not* generate anything extra for nested helper functions 
in there.
Oct 02 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Friday, 2 October 2020 at 12:56:35 UTC, Adam D. Ruppe wrote:

 On Friday, 2 October 2020 at 04:53:12 UTC, Stefan Koch wrote:

 You still create unnecessary symbols.


 Not true.

 Absolutely identical. An eponymous template resolving to an 
 enum of a common type is indistinguishable to a literal of that 
 type. It does *not* generate anything extra for nested helper 
 functions in there.


Replace the static immutable with enum and check again.
Oct 02 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 October 2020 at 14:34:54 UTC, Stefan Koch wrote:

 Replace the static immutable with enum and check again.


It is identical again. As expected.


$ dmd -betterC bc
$ nm bc.o
0000000000000000 t
                  U _GLOBAL_OFFSET_TABLE_
0000000000000000 W main
                  U printf



$ dmd -betterC bc
$ nm bc.o
0000000000000000 t
                  U _GLOBAL_OFFSET_TABLE_
0000000000000000 W main
                  U printf
Oct 02 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Friday, 2 October 2020 at 14:42:15 UTC, Adam D. Ruppe wrote:

 On Friday, 2 October 2020 at 14:34:54 UTC, Stefan Koch wrote:

 Replace the static immutable with enum and check again.


 It is identical again. As expected.


 $ dmd -betterC bc
 $ nm bc.o
 0000000000000000 t
                  U _GLOBAL_OFFSET_TABLE_
 0000000000000000 W main
                  U printf



 $ dmd -betterC bc
 $ nm bc.o
 0000000000000000 t
                  U _GLOBAL_OFFSET_TABLE_
 0000000000000000 W main
                  U printf


indeed.
now measure memory consumption during compilation :)
Oct 02 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 October 2020 at 14:45:20 UTC, Stefan Koch wrote:

 now measure memory consumption during compilation :)


I did, it is there but small.

template:
14,096 KB

literal:
13,984 KB

your type function:
14,068 KB


In some individual runs there was enough variation where each one 
would come up on top and at bottom randomly but these are the 
average of 10 runs on my box.

Compile time too small to measure, all averaged 0.02s.

It is hard to say this would even matter if we did 1,000 of 
them.... let's try it.

added up top:

static foreach(i; 0 .. 1000)
mixin("struct i", i, " {}");


and in the function:

     static foreach(i; 0 .. 1000)
     mixin("static immutable convMatrix", i, " = 
makeConvMatrix(Byte, Ubyte, Short, Ushort, Int, Uint, Long, 
Ulong);");

     printf("%s\n", convMatrix0.ptr);



With type function:

0.37s, 71,756 KB

with template pattern:

0.06s, 34,944 KB

with string literal:

0.06s, 33,976 KB


The type function is a clear loser! But this is in part because 
the template is reusing its cached result. Let's break that by 
using those structs I generated

template pattern with 1,000 unique instances:

0.83s, 372,992 KB

type function with 1,000 unique instances:

  Error: function test.makeConvMatrix(alias[] types...) is not 
callable using argument types (i18, byte, ubyte, short, ushort, 
int, uint, long, ulong)
   cannot pass argument i18 of type i18 to parameter alias[] 
types...
[snip etc same thing for all 1000 cases]


Lovely. But I just grabbed your git branch so some in-progress 
work is to be expected to be incomplete...

What about a template?

test.d-mixin-42(42):        cannot pass argument test!18 of type 
test!18 to parameter alias[] types...


Is there a way I can pass it a list of unique types easily?


Probably fair to say it would perform about the same as before 
since the type function shouldn't be caching intermediate results 
(which is what makes the template fast before but bloat memory 
now), but would be nice to prove it.

PS for other readers, the template thing isn't *just* a cache, 
like it can't simply discard entries to it.... yet. With this 
specific pattern, the compiler actually *should* be able to free 
it  the helper function and even discard the whole cache in 
theory, but in practice that can only be done in specialized 
cases and it is very difficult to realize.
Oct 02 2020
next sibling parent Stefan Koch <uplink.coder gmail.com> writes:
On Friday, 2 October 2020 at 15:11:18 UTC, Adam D. Ruppe wrote:

 The type function is a clear loser! But this is in part because 
 the template is reusing its cached result.


Indeed.
You can fix that by wrapping the type function inside a template 
to make use of the instance caching, if you are expecting it to 
be called with the same arguments.

I would assume this to happen less in actual use-cases though.
Also keep in mind that CTFE can be sped up by 10x when switching 
to byte code interpreter.
You can't say the same for templates.
Oct 02 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Friday, 2 October 2020 at 15:11:18 UTC, Adam D. Ruppe wrote:

 type function with 1,000 unique instances:

  Error: function test.makeConvMatrix(alias[] types...) is not 
 callable using argument types (i18, byte, ubyte, short, ushort, 
 int, uint, long, ulong)
   cannot pass argument i18 of type i18 to parameter alias[] 
 types...
 [snip etc same thing for all 1000 cases]


 Lovely. But I just grabbed your git branch so some in-progress 
 work is to be expected to be incomplete...

 What about a template?

 test.d-mixin-42(42):        cannot pass argument test!18 of 
 type test!18 to parameter alias[] types...


Indeed. A am currently fixing this bug.
The problem is that structs and classes are not considered types 
... they are symbols.
Generally dmd doesn't like it if you pass types to functions and 
I have to circumvent the resistance ;)
Oct 02 2020
parent Stefan Koch <uplink.coder gmail.com> writes:
On Friday, 2 October 2020 at 15:36:46 UTC, Stefan Koch wrote:

 On Friday, 2 October 2020 at 15:11:18 UTC, Adam D. Ruppe wrote:

 [...]


 Indeed. A am currently fixing this bug.
 The problem is that structs and classes are not considered 
 types ... they are symbols.
 Generally dmd doesn't like it if you pass types to functions 
 and I have to circumvent the resistance ;)


Should be fixed now.
Oct 02 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 October 2020 at 15:11:18 UTC, Adam D. Ruppe wrote:

 template pattern with 1,000 unique instances:

 0.83s, 372,992 KB

 type function with 1,000 unique instances:


With bug fixed so it builds got:

0.45s, 85,000 KB


So there is a win, and there's a few reasons for it:

1) in the template the function is saved in RAM despite never 
being needed again. The typefunction impl is actually reused. 
This could potentially be optimized in the current code (I tried 
and failed though, kept segfaulting, but someone who knows dmd 
better might be able to pull it off)

2) The template function is larger. All those tuple foreaches are 
unrolled, leading to a large function. The typefunction does not 
do this.

I wonder if the type function's foreach implementation could be 
used in the tuple case too. At least if there's no variable 
declaration based on the iterated variable it seems plausible.

3) This also is kinda the other extreme. All reused args is the 
template's extreme case (one instance reused) and zero reused is 
the TF's extreme case (it runs each time because it legitimately 
must). I suspect most real code will land somewhere in the middle.


But still 2x as fast and 1/4 the memory with today's 
implementations is a real victory in this case.
Oct 02 2020
parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Friday, 2 October 2020 at 17:38:17 UTC, Adam D. Ruppe wrote:

 On Friday, 2 October 2020 at 15:11:18 UTC, Adam D. Ruppe wrote:

 template pattern with 1,000 unique instances:

 0.83s, 372,992 KB

 type function with 1,000 unique instances:


 With bug fixed so it builds got:

 0.45s, 85,000 KB


 So there is a win, and there's a few reasons for it:

 1) in the template the function is saved in RAM despite never 
 being needed again. The typefunction impl is actually reused. 
 This could potentially be optimized in the current code (I 
 tried and failed though, kept segfaulting, but someone who 
 knows dmd better might be able to pull it off)

 2) The template function is larger. All those tuple foreaches 
 are unrolled, leading to a large function. The typefunction 
 does not do this.

 I wonder if the type function's foreach implementation could be 
 used in the tuple case too. At least if there's no variable 
 declaration based on the iterated variable it seems plausible.

 3) This also is kinda the other extreme. All reused args is the 
 template's extreme case (one instance reused) and zero reused 
 is the TF's extreme case (it runs each time because it 
 legitimately must). I suspect most real code will land 
 somewhere in the middle.


 But still 2x as fast and 1/4 the memory with today's 
 implementations is a real victory in this case.


Thanks Adam and Stefan for getting us some concrete performance 
data.

As Adam notes, performance against actual source might differ.

I'm more excited about the ease of learning and better 
debugability of type functions than any compile time performance 
gains but I imagine we'll realize additional speedups if type 
functions are widely adopted.  Iteration will displace recursion 
in many places and we'll see a lot more "left leaning" code 
(early returns).
Oct 02 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 October 2020 at 19:13:43 UTC, Bruce Carneal wrote:

 I'm more excited about the ease of learning and better 
 debugability of type functions than any compile time 
 performance gains but I imagine we'll realize additional 
 speedups if type functions are widely adopted.  Iteration will 
 displace recursion in many places and we'll see a lot more 
 "left leaning" code (early returns).


Eh, I'm still pretty skeptical. Notice that the code in this 
conversion matrix case is *identical* for type function and 
template, except for the header.

template version:

template makeConvMatrix(types...) {
   string helper() {
     /* same stuff */
   }
   enum makeConvMatrix = helper();
}

type function version:

string makeConvMatrix(alias[] types ...) {
    /* same stuff */
}


And then the call of course has the ! in the template version and 
doesn't for the type function version. But the actual function 
implementation is identical, the generated binary is identical. 
So very little actual difference here aside from the compile 
speed+ram mix in different cases.

Using the index pattern too (which actually compiles slower in 
most cases), you can get similar looking code:

template Filter(alias Pred, Args...) {
         private const(char)[] helper() {
                 assert(__ctfe);
                 size_t[Args.length] keep;
                 size_t pos = 0;
                 foreach(idx, alias arg; Args)
                         if(Pred!arg) keep[pos++] = idx;
                 // generic helper function that expands to
                 // AliasSeq!(Args[keep[0]], Args[keep[1]], ...)
                 return makeResult(keep[0 .. pos]);
         }

         alias Filter = mixin(helper());
}


So again a normal loop building a normal array. Nothing too fancy 
here. But in my tests, Phobos' existing implementation is 
actually better in CT speed and memory.

The potential I see in the type functions are:

1) faster builds with less memory. I'm putting this under "maybe" 
since there's some cases I've tried where it wins, some where it 
loses. So I'm skeptical but if it works out it is worth it.

2) making compiler-tuples more natural to work with. Right now 
they are a bizarre beast you kinda have to trick into doing 
anything. The type function (I have been joking to stefan to call 
it Static Tuple Evaluation Function or STEF :P) might make them 
feel just like any other array which would be really cool and can 
make things nicer that right now must be string mixins at the 
usage point.

The current test implementation can't actually do this, but 
there's potential in making it work.

3) maybe just educate us on individual pieces we can borrow in 
other parts of the language. Its not-unrolled tuple foreach I 
think is a win, its CTFE-only nature is a clear win. Paul Backus' 
idea of static foreach declarations actually being ref might be 
an example of this too. These things might be extracted even if 
the overall concept doesn't work out.


We'll see where it goes.
Oct 02 2020
prev sibling parent data pulverizer <data.pulverizer gmail.com> writes:
On Thursday, 1 October 2020 at 08:21:24 UTC, Stefan Koch wrote:

 Hi People,

 To further show the intuitive type function syntax I just 
 created a way to print a conversion matrix [...SNIP...]


Awesome. I like.
Oct 02 2020