www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Can D do some sort of automatic template to "dynamic template"

reply Adam <Adam and.Eve> writes:
Fact: Templates are static!

But templates are still normal functioning code and can actually 
work dynamically. We can see this by including a D compiler 
inside an app and have the app re-compile the templates for 
specific dynamic types. It doesn't need templates to do of 
course, but it shows that "Templates are static" is not quite as 
simple as it sounds.

But could D actually create some sort of internal expression of D 
templates as dynamic objects that can be used in run-time?

If have a template T that only uses 3 types, then one can simple 
do something like(pseudo):

t(object o)
{
// use dynamic reflection or whatever to get the type of the 
object o
if (o.type == 'int')
     return T!(int)(o.value, o.unit);
if (o.type == 'float')
     return T!(float)(o.value, o.unit);
return T!(unknown)(o.value, o.unit);
}

etc...

t then is a "dynamic representation" of the static functionality 
of T.

But my guess is that the compiler could essentially do this for 
us and allow one to use templates in a dynamic way. Most of the 
time it may not be of much use but, say, if one is interacting 
with console and T converts a string to a number. T could convert 
a number with unit such as "3mm" in to some base representation 
such as 0.003(m). float's are handled differently than 
ints(rounding depending on some criteria).

Maybe we use the T internally and its lightning fast. But for the 
console case, we  have write a dynamic version of the template or 
create something similar to the code above.

But the compiler already knows, at least, some of the times that 
will be used and it should be able to instantiate the template 
for each of those types.

e.g., if you use T!(float) in the code, it will know to create 
the "dynamic wrapper" to include that type in the list. If you 
don't every use T!(float) explicitly then you can simply create a 
useless statement like {auto x = T!(float)("3.04mm");} which will 
be optimized but the compiler will then include float as a type 
used in the dynamic version of T. I may not be making much sense 
here, but it's the best I'll do.



The compiler then can do something like assemble the dynamic 
version of the template into a new function which can be used at 
compile time on any object.
Using a simple special character could inform the compiler to use 
the dynamic template version.

object u = userInput();
&T(u); // (instead of using t(u)).


Here the compiler does all the work for us. If u has type float 
then this gets effectively dispatched to T!(float)(cast(float)u). 
If it's an int, then T!(int)(cast(int)u).

What it mainly does for us is simplify having to do the cast our 
self along with the string of if checking for all the types or 
having to code that works dynamically

Or is it impossible for the D compiler to accomplish such a task 
automatically for us?
Sep 18 2015
next sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Friday, 18 September 2015 at 20:55:06 UTC, Adam wrote:
 Fact: Templates are static!

 But templates are still normal functioning code and can 
 actually work dynamically. We can see this by including a D 
 compiler inside an app and have the app re-compile the 
 templates for specific dynamic types. It doesn't need templates 
 to do of course, but it shows that "Templates are static" is 
 not quite as simple as it sounds.

 But could D actually create some sort of internal expression of 
 D templates as dynamic objects that can be used in run-time?

 If have a template T that only uses 3 types, then one can 
 simple do something like(pseudo):

 t(object o)
 {
 // use dynamic reflection or whatever to get the type of the 
 object o
 if (o.type == 'int')
     return T!(int)(o.value, o.unit);
 if (o.type == 'float')
     return T!(float)(o.value, o.unit);
 return T!(unknown)(o.value, o.unit);
 }

 etc...

 t then is a "dynamic representation" of the static 
 functionality of T.

 But my guess is that the compiler could essentially do this for 
 us and allow one to use templates in a dynamic way. Most of the 
 time it may not be of much use but, say, if one is interacting 
 with console and T converts a string to a number. T could 
 convert a number with unit such as "3mm" in to some base 
 representation such as 0.003(m). float's are handled 
 differently than ints(rounding depending on some criteria).

 Maybe we use the T internally and its lightning fast. But for 
 the console case, we  have write a dynamic version of the 
 template or create something similar to the code above.

 But the compiler already knows, at least, some of the times 
 that will be used and it should be able to instantiate the 
 template for each of those types.

 e.g., if you use T!(float) in the code, it will know to create 
 the "dynamic wrapper" to include that type in the list. If you 
 don't every use T!(float) explicitly then you can simply create 
 a useless statement like {auto x = T!(float)("3.04mm");} which 
 will be optimized but the compiler will then include float as a 
 type used in the dynamic version of T. I may not be making much 
 sense here, but it's the best I'll do.



 The compiler then can do something like assemble the dynamic 
 version of the template into a new function which can be used 
 at compile time on any object.
 Using a simple special character could inform the compiler to 
 use the dynamic template version.

 object u = userInput();
 &T(u); // (instead of using t(u)).


 Here the compiler does all the work for us. If u has type float 
 then this gets effectively dispatched to 
 T!(float)(cast(float)u). If it's an int, then 
 T!(int)(cast(int)u).

 What it mainly does for us is simplify having to do the cast 
 our self along with the string of if checking for all the types 
 or having to code that works dynamically

 Or is it impossible for the D compiler to accomplish such a 
 task automatically for us?
I think that what you're asking can only be accomplished in an interpreted or JIT compiled language. With a compiled language, you have to know all the possible types you will support up front. If you're getting the input from a console though, you must be parsing it right? The parser cannot complete it's task without knowing what it's parsing, and the parser should pass along or store any relevant information for future use. For compiled languages, there is the idea of "Type Erasure". Polymorphism is one way to get this done. You will have to create your own base object which specifies all possible functionality. Example: import std.stdio; import std.conv; import std.random; abstract class BaseObj { abstract void print(); } class FloatObj : BaseObj { float value; this(float val) { value = val; } override void print() { writeln("float: " ~ value.to!string); } } class IntObj : BaseObj { int value; this(int val) { value = val; } override void print() { writeln("int: " ~ value.to!string); } } BaseObj userInput() { if(uniform(0, 2) % 2 == 0) return new FloatObj(uniform(0.0f, 100.0f)); else return new IntObj(uniform(0, 100)); } void main(string[] args) { BaseObj[] objs; foreach(i; 0..10) { BaseObj obj = userInput(); obj.print(); } }
Sep 18 2015
prev sibling parent reply Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:
On Friday, 18 September 2015 at 20:55:06 UTC, Adam wrote:
 Fact: Templates are static!

 But templates are still normal functioning code and can 
 actually work dynamically. We can see this by including a D 
 compiler inside an app and have the app re-compile the 
 templates for specific dynamic types. It doesn't need templates 
 to do of course, but it shows that "Templates are static" is 
 not quite as simple as it sounds.

 But could D actually create some sort of internal expression of 
 D templates as dynamic objects that can be used in run-time?

 If have a template T that only uses 3 types, then one can 
 simple do something like(pseudo):

 t(object o)
 {
 // use dynamic reflection or whatever to get the type of the 
 object o
 if (o.type == 'int')
     return T!(int)(o.value, o.unit);
 if (o.type == 'float')
     return T!(float)(o.value, o.unit);
 return T!(unknown)(o.value, o.unit);
 }

 etc...

 t then is a "dynamic representation" of the static 
 functionality of T.

 But my guess is that the compiler could essentially do this for 
 us and allow one to use templates in a dynamic way. Most of the 
 time it may not be of much use but, say, if one is interacting 
 with console and T converts a string to a number. T could 
 convert a number with unit such as "3mm" in to some base 
 representation such as 0.003(m). float's are handled 
 differently than ints(rounding depending on some criteria).

 Maybe we use the T internally and its lightning fast. But for 
 the console case, we  have write a dynamic version of the 
 template or create something similar to the code above.

 But the compiler already knows, at least, some of the times 
 that will be used and it should be able to instantiate the 
 template for each of those types.

 e.g., if you use T!(float) in the code, it will know to create 
 the "dynamic wrapper" to include that type in the list. If you 
 don't every use T!(float) explicitly then you can simply create 
 a useless statement like {auto x = T!(float)("3.04mm");} which 
 will be optimized but the compiler will then include float as a 
 type used in the dynamic version of T. I may not be making much 
 sense here, but it's the best I'll do.



 The compiler then can do something like assemble the dynamic 
 version of the template into a new function which can be used 
 at compile time on any object.
 Using a simple special character could inform the compiler to 
 use the dynamic template version.

 object u = userInput();
 &T(u); // (instead of using t(u)).


 Here the compiler does all the work for us. If u has type float 
 then this gets effectively dispatched to 
 T!(float)(cast(float)u). If it's an int, then 
 T!(int)(cast(int)u).

 What it mainly does for us is simplify having to do the cast 
 our self along with the string of if checking for all the types 
 or having to code that works dynamically

 Or is it impossible for the D compiler to accomplish such a 
 task automatically for us?
I think you could also do this using structs with a type enum inside and a union for the data contents. Then you can use string mixins using traits to create generic runtime functions that dispatch to the appropriate method/free standing function on the underlying type. Take a look at Algebraic and Adam Ruppe's jsvar. Some of the genericity can be handled at compile time. For example, I have a templated time series entry struct. A time series is a slice of such structs. An entry definitively has a date, and it may have either a bid and ask or (a close and optionally open, high, low, adjusted close). It has a volume, and may have an open interest. The date may be a year/month/day or it might have a time too. I want to store things efficiently in memory and on disk. So I use hasMember!(typeof(pricebar.date),"hour") to find out if it's an intraday date or daily for example. See Andrei's allocator talk, his code, and maybe that for cerealed. I have a bunch of templated functions that can do useful things with such a time series. But then I want to pass them around wrapped in an a tagged union and generate versions of these functions that can operate on this untemplated struct. (I will wrap the slices, not the entries). That will make it easier to access my work from a dynamic language like Lua. I haven't written this bit yet, but I have done something similar in another context. A degree of misery in trying to figure out std.traits, but that's a one off cost and it goes quicker if you read others code for inspiration. UDAs help in specifying which functions should have wrapped versions generated etc.
Sep 19 2015
parent reply Adam <Adam and.Eve> writes:
On Saturday, 19 September 2015 at 15:56:23 UTC, Laeeth Isharc 
wrote:
 On Friday, 18 September 2015 at 20:55:06 UTC, Adam wrote:
 Fact: Templates are static!

 But templates are still normal functioning code and can 
 actually work dynamically. We can see this by including a D 
 compiler inside an app and have the app re-compile the 
 templates for specific dynamic types. It doesn't need 
 templates to do of course, but it shows that "Templates are 
 static" is not quite as simple as it sounds.

 But could D actually create some sort of internal expression 
 of D templates as dynamic objects that can be used in run-time?

 If have a template T that only uses 3 types, then one can 
 simple do something like(pseudo):

 t(object o)
 {
 // use dynamic reflection or whatever to get the type of the 
 object o
 if (o.type == 'int')
     return T!(int)(o.value, o.unit);
 if (o.type == 'float')
     return T!(float)(o.value, o.unit);
 return T!(unknown)(o.value, o.unit);
 }

 etc...

 t then is a "dynamic representation" of the static 
 functionality of T.

 But my guess is that the compiler could essentially do this 
 for us and allow one to use templates in a dynamic way. Most 
 of the time it may not be of much use but, say, if one is 
 interacting with console and T converts a string to a number. 
 T could convert a number with unit such as "3mm" in to some 
 base representation such as 0.003(m). float's are handled 
 differently than ints(rounding depending on some criteria).

 Maybe we use the T internally and its lightning fast. But for 
 the console case, we  have write a dynamic version of the 
 template or create something similar to the code above.

 But the compiler already knows, at least, some of the times 
 that will be used and it should be able to instantiate the 
 template for each of those types.

 e.g., if you use T!(float) in the code, it will know to create 
 the "dynamic wrapper" to include that type in the list. If you 
 don't every use T!(float) explicitly then you can simply 
 create a useless statement like {auto x = 
 T!(float)("3.04mm");} which will be optimized but the compiler 
 will then include float as a type used in the dynamic version 
 of T. I may not be making much sense here, but it's the best 
 I'll do.



 The compiler then can do something like assemble the dynamic 
 version of the template into a new function which can be used 
 at compile time on any object.
 Using a simple special character could inform the compiler to 
 use the dynamic template version.

 object u = userInput();
 &T(u); // (instead of using t(u)).


 Here the compiler does all the work for us. If u has type 
 float then this gets effectively dispatched to 
 T!(float)(cast(float)u). If it's an int, then 
 T!(int)(cast(int)u).

 What it mainly does for us is simplify having to do the cast 
 our self along with the string of if checking for all the 
 types or having to code that works dynamically

 Or is it impossible for the D compiler to accomplish such a 
 task automatically for us?
I think you could also do this using structs with a type enum inside and a union for the data contents. Then you can use string mixins using traits to create generic runtime functions that dispatch to the appropriate method/free standing function on the underlying type. Take a look at Algebraic and Adam Ruppe's jsvar. Some of the genericity can be handled at compile time. For example, I have a templated time series entry struct. A time series is a slice of such structs. An entry definitively has a date, and it may have either a bid and ask or (a close and optionally open, high, low, adjusted close). It has a volume, and may have an open interest. The date may be a year/month/day or it might have a time too. I want to store things efficiently in memory and on disk. So I use hasMember!(typeof(pricebar.date),"hour") to find out if it's an intraday date or daily for example. See Andrei's allocator talk, his code, and maybe that for cerealed. I have a bunch of templated functions that can do useful things with such a time series. But then I want to pass them around wrapped in an a tagged union and generate versions of these functions that can operate on this untemplated struct. (I will wrap the slices, not the entries). That will make it easier to access my work from a dynamic language like Lua. I haven't written this bit yet, but I have done something similar in another context. A degree of misery in trying to figure out std.traits, but that's a one off cost and it goes quicker if you read others code for inspiration. UDAs help in specifying which functions should have wrapped versions generated etc.
I'm thinking I've probably been miunderstood here. Let me clarify what I'm talking about and see if it still more sense. Templated functions's types are resolved at compile time, so void foo(T)(T x){} is a templated function that operates on a type T and is a sort of compile time construct(since it depends on the unknown T). I do not mean it only can be used at compile time, but T must be known then. So, foo!T only exists at compile time, right? Yet, for any actual type such as an int, foo!int is a runtime function(it exists in the binary). The compiler figures out all the types used at compile time and creates all the various foo!N's. If K is the set of all the types used at compile type with foo, we could effectively write foo!K as a sort of collection of all the different foo's(but it's not really any different than foo!T except T is any type while K is the actual types used by the programmer). Ok so far? So, if the programmer called foo!int, foo!float in the code, the compiler then constructs foo!K = {foo!int, foo!float} and puts them in the binary. Then these are actual run-time functions and foo!K represents them. foo!K is as real as foo!T is, but restricted to only contains the actual used types and put in the binary. If the compiler could hypothetically put all of foo!T then the binary would be infinite, or, at least very large. So, instead it just keeps foo!K. If you want to expand K at run-time, you are out of luck. Ok so far? we can actually create foo!K easily(assuming foo!T(int)) fooK(string K, object i) { if (K == "int") foo!int(i); else if (K == "float") foo!float(i); else foo!unknown(i); // unknown is special, not needed but it would only add one element to foo!K, so the binary would not be much bigger and we could specialize foo for "unknown" to handle those cases such as throwing an exception. } Ok? now fooK is a function that exists that works at run-time on all types. It is an approximation to foo!T(int). If, say, in your code, you now use foo!string, then on recompile we would have fooK(string K, object i) { if (K == "int") foo!int(i); else if (K == "float") foo!float(i); else if (K == "string") foo!string(i); else foo!unknown(i); } Again, we can do this all by hand. And we can call fooK on objects at run-time rather than known types at compile time. The only reason we can't use foo!T is because the compiler doesn't know all the types we will be using(at run-time), if it did, we could use foo!T at run-time. It simply does the best job it can and gets the compile time uses. It can't do anything about run-time, I guess, unless it dynamically built the functions at run-time, Which is a possibility I suppose but it would require a D compiler inside the code to create the new instance of the template for the new time. fooK though works like any other non-templated function, so it can be used at run time for us. fooK can be generated for us by the compiler if it was "smart enough". It could add type checking and such to remove the string K parameter if it can find out what the type of i is. e.g., use reflection for something like fooK(object i) { string K = typeof(i).stringof; if (K == "int") foo!int(i); else if (K == "float") foo!float(i); else if (K == "string") foo!string(i); else foo!unknown(i); } and, in fact, one could get fancy and use a hash table instead of the if's. Then this would allow one to extend fooK at run-time sort of like oop. (just add your new foo!X to the table and it will be called) The problem for the programmer is that when the number of parameters, the number of if statements grow by #K^n. For 1 parameter it's #K and for 2 it's #K^2 since you'll have to check for all pairs of types. It also makes it hard to maintain the above for new added types. You'll have to do it for every templated function and update all of them every time a new specialization is added. Ok so far? What's cool, if all this makes sense. If fooK is used on "compile time types", it can be optimized out to the static call. So fooK could always be used(we would want to use foo!K for notation, of course) and would function as both a compile time type of templated function and also a run-time function. If foo was implemented with a look up table, then templated functions could be added at run time to extend the behavior. E.g., a plugin could could add a "type handler" to foo!K to handle some new type it's using. Ultimately, what would happen is that templated functions would be "first class citizens" and work not only in compile type but at run-time too. It sort of combines both compile-time reflection(what templates do now) and run-time reflection(what that don't) and the compiler figures it out and keeps track of it all. For the programmer, there would be no difference except for the unknown case. Performance wise, there should be no difference. At compile time, the compiler would do pretty much exactly what it does now, just use the explicitly known type. It would have the same performance as before. But when it can't do this(the types are unknown), it falls back onto the runtime lookup version. Does any of that make sense? ;) The reason one can't implement this as a library function, as far as I know, is because one would have to know all the types the programmer calls foo!T with at "compile-time"(e.g., we would have to have a K, the set of those types). If one could get that information then fooK could be automatically constructed using string mixins. Not sure if optimization would take place, I suppose some static checking could be used to get the compile-time types then a normal lookup table for the others. If such a fooK was known, then programmer would always use that instead and be confident that the compiler would always choose the best situation(if it can use a direct call to the foo!N for some N, then it uses that, if not, it goes through the lookup process). I am making one big assumption here, and that is one can get the type of an object at run-time. I'm not sure if this is possible in D and there maybe other issues with inheritance and such which might cause problems with more complicated cases than I've assumed here. What's more, is with fooK, templated functions can take place as virtual functions! Since fooK encomposes both compile time and dynamic behavior in one) Of course, when used as virtual functions, unless in certain cases when optimization can take place, the run-time behavior will always be used. I doubt anyone would read all so I'll stop now ;)
Sep 19 2015
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 19 September 2015 at 17:46:27 UTC, Adam wrote:
 I doubt anyone would read all so I'll stop now ;)
What you're asking is actually pretty easy. You can just write a function that loops to form the object list yourself or you could make the template add a static module constructor that adds itself to the call list. Then you tag your object with its typeinfo and use that as index into the list.
Sep 19 2015
parent reply Adam <Adam and.Eve> writes:
On Saturday, 19 September 2015 at 18:36:49 UTC, Adam D. Ruppe 
wrote:
 On Saturday, 19 September 2015 at 17:46:27 UTC, Adam wrote:
 I doubt anyone would read all so I'll stop now ;)
What you're asking is actually pretty easy. You can just write a function that loops to form the object list yourself or you could make the template add a static module constructor that adds itself to the call list. Then you tag your object with its typeinfo and use that as index into the list.
It might be, but I don't see how it can be done easily. I don't know enough about D to know if it is implementable as a library solution. Suppose our templated function is auto foo(T)(T x) { return x; } and suppose we specialize it with auto foo(T : object)(T x) { // The static part of the dispatch would look something like static if (is(T == int ) { return foo!int(cast(int)x); .... // This stuff should all be optimized out at compile time if/when the if passes // In fact, it would never be called in this example since int is more special than object. This part is more for when we don't specialize on object and just create a wrapper for the templated function. // If no compile time dispatch, then we need to deal with the unknown type using run-time reflection, we still want to dispatch known types to the foo but no optimization can occur here if (cast(int)x) return foo!int(cast(int)x); // Anymore? else return foo!object(x); // or foo!unknown(x) } foo!object should, in theory, work for all types, run-time or not. But you say constructing foo!object is easy?!?!?! Remember, it has to work in general for all types and number of parameters and should always optimize to a direct template call when the compile time checks are satisfied. Also, the above doesn't allow for dynamic specialization, but that would be easy to add with a lookup table. The main problem I see, which you say is easy, is how to know the list of types to check against? int is the only one used above, but in reality, it depends on context. If the programmer used foo!float, we would have to add that to the dispatch list. We don't want to add it if he doesn't use it because then we are including extra code in the binary that might never be used. It may be used at run time but that's up to the programmer to make sure it is include(by using it at compile time at least ones or through some other method).
Sep 19 2015
parent Kagamin <spam here.lot> writes:
On Saturday, 19 September 2015 at 21:12:52 UTC, Adam wrote:
 Suppose our templated function is

 auto foo(T)(T x) { return x; }
Why not just use http://dlang.org/phobos/std_variant.html#.Algebraic ?
Sep 21 2015
prev sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Saturday, 19 September 2015 at 17:46:27 UTC, Adam wrote:

 Does any of that make sense? ;)
No. Where exactly do these magical dynamic objects come from? How exactly do you expect user input to generate an object which would require an implementation that was not foreseeable at compile time? This sound like an extreme complication for a problem which is trivial to solve using conventional means, like polymorphism, or a struct with an enum in it. How exactly do you plan to improve on these two approaches? Bit
Sep 19 2015