www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Proposal: __traits(code, ...) and/or .codeof

reply "F i L" <witte2008 gmail.com> writes:
So the discussions about Attributes and Aspect Oriented 
Programming (AOP) got me thinking... Basically AOP requires 
injecting code fragments together in a comprehensible way. 
Similarly, Attributes that go beyond  note (such as  GC.NoScan) 
need similar ability.

D already has the ability to mixin arbitrary code fragments at 
compile time, and to process those in useful ways through CTFE. 
Which rocks. What it lacks is the ability to reflect upon the 
actual source code due to IO limitations of CTFE. So creating a 
mixin templates which pieces together a unique object is, to my 
knowledge, currently next to impossible (and slow since you'd 
have to parse and isolate code in .d file multiple times in a 
separate process, then compile again to put it all together).

So, to quote Walter, what compelling features would it bring? 
Here's an example of a simple AOP program from the AOP wiki page 
(probably not the best implementation, but the concept is there):

   struct BankType
   {
     void transfer() { ... }
     void getMoneyBack() { ... }
   }

   struct Logger
   {
     void transfer() {
       log("transferring money...");
     }
     void getMoneyBack() {
       log("User requested money back");
     }
   }

and now some magic...

   string bankCode(T...)(T aspects) {
     auto code = "struct Bank {";
     auto members = [__traits(allMembers, Bank)];
     foreach (m; members) {
       code ~= "void "~m~"() {";
       code ~= __traits(getMember, Bank, m).codeof;
       foreach (a; aspects) {
         if (__traits(hasMember, a, m) {
           code ~= __traits(getMember, a, m).codeof;
         }
       }
       code ~= "}"
     }
     return code ~ "}";
   }

   mixin template Bank(T...)
   {
     mixin(bankCode(T));
   }

   mixin Bank!Logger;

   void main() {
     auto b = Bank();
     b.transfer(); // logs
     b.getMoneyBack(); // ditto
   }

So this would allow us to make "Compilers" within the Compiler 
(Codeception), since we could parse/strip/append any existing 
code fragments together in endless combination. Generic 
"Builders" could probably be built and put into a std.builder lib 
for general use.

One particular use I have in mind is for Behavior Objects (Game 
Scripts). Each behavior would hold Property(T) objects which 
define per-property, per-state "binding" dependencies (eg. 
position.x.bind(other.x, State.Idle)) and execution code. On 
release, the Property(T) object would be stripped away (leaving 
just T) and it's behavior code "compressed" with others into 
optimized functions.

I don't know much about the internals of DMD, so I'm not sure 
this is a realistic request, but I think the idea is compelling. 
Also, for Attributes I'm not sure this technique is really 
applicable. But it's possible that the compiler could exploit 
this internally for certain Attributes like  GC.whatever
Mar 22 2012
next sibling parent reply "F i L" <witte2008 gmail.com> writes:
ps. Mono-C#'s NRefactory, and Microsoft .Net's forthcoming Roslyn 
Project are the only comparable infrastructures I can think of 
with this level of reflection, and they're the foundation to some 
pretty innovative new development tools.

NRefactory: http://wiki.sharpdevelop.net/NRefactory.ashx
Roslyn: http://msdn.microsoft.com/en-us/vstudio/hh500769
Mar 22 2012
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 22.03.2012 18:34, schrieb CTFE-4-the-win:
 On Thursday, 22 March 2012 at 16:55:34 UTC, F i L wrote:
 ps. Mono-C#'s NRefactory, and Microsoft .Net's forthcoming Roslyn
 Project are the only comparable infrastructures I can think of with
 this level of reflection, and they're the foundation to some pretty
 innovative new development tools.

 NRefactory: http://wiki.sharpdevelop.net/NRefactory.ashx
 Roslyn: http://msdn.microsoft.com/en-us/vstudio/hh500769

Even if I'm closer to the opposite of a MS-evangelist... color me impressed, I like your idea. +1.

Same here. Microsoft R&D does lots of cool stuff actually, pity that sometimes they get it so wrong, due to the way the management/marketing works. -- Paulo
Mar 22 2012
prev sibling next sibling parent "CTFE-4-the-win" <CTFE 4the.win> writes:
On Thursday, 22 March 2012 at 16:55:34 UTC, F i L wrote:
 ps. Mono-C#'s NRefactory, and Microsoft .Net's forthcoming 
 Roslyn Project are the only comparable infrastructures I can 
 think of with this level of reflection, and they're the 
 foundation to some pretty innovative new development tools.

 NRefactory: http://wiki.sharpdevelop.net/NRefactory.ashx
 Roslyn: http://msdn.microsoft.com/en-us/vstudio/hh500769

Even if I'm closer to the opposite of a MS-evangelist... color me impressed, I like your idea. +1.
Mar 22 2012
prev sibling next sibling parent reply "Felix Hufnagel" <suicide xited.de> writes:
in addition to .codeof, let's think about .astof returning an abstract  
syntax tree.
Mar 22 2012
next sibling parent reply "F i L" <witte2008 gmail.com> writes:
On Thursday, 22 March 2012 at 18:06:24 UTC, Felix Hufnagel wrote:
 in addition to .codeof, let's think about .astof returning an 
 abstract syntax tree.

I agree, as stated on the IRC, .astof (and therefor .astof.toString()) is a much better concept.
Mar 22 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 03/22/2012 07:08 PM, F i L wrote:
 On Thursday, 22 March 2012 at 18:06:24 UTC, Felix Hufnagel wrote:
 in addition to .codeof, let's think about .astof returning an abstract
 syntax tree.

I agree, as stated on the IRC, .astof (and therefor .astof.toString()) is a much better concept.

I think that would necessitate the addition of AST macros.
Mar 22 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 03/22/2012 08:12 PM, F i L wrote:
 Timon Gehr wrote:
 I think that would necessitate the addition of AST macros.

You mean passing DMD's actual AST tree object? Only a stripped down version?

We have the macro keyword. I envision something like: macro replaceAggregate(macro newAggregate, macro loop : foreach(x; aggr){statements}, macro x, macro aggr, macro statements) { foreach(x; newAggregate){statements} } void main(){ int[] a = [1,2,3]; int[] b = [2,3,4]; replaceAggregate(b, foreach(x;a){writeln(x);}); } (The syntax looks horrible, but you get the idea: AST walking by pattern matching) Ideally, such macros would allow introducing local helper variables inside the macro that are not accessible from outside and introducing variables/enums in the context the macro is used. (prefixing identifiers that should interfere with the enclosing context with $ would be an option). Another idea: macro someFancyDSL(string dsl){ mixin(someFancyDSLtoD(dsl); } void main(){ int n = 5; someFancyDSL { import n; let x = 1 : x let y = product (take n) x export y; } writeln(y); } // this would also enable less fancy but useful stuff: macro until(macro condition, string str){ while(!condition){ mixin(str); // string and mixin not strictly necessary, // maybe enable the feature on macro params too } } void main(){ bool done = false; int x; until(done){ done = foo(x++); } } This is just a very rough sketch though, we would need a much more refined design. I think getting the symbol scoping right is most important.
Mar 22 2012
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 03/23/2012 12:33 AM, F i L wrote:
 Timon Gehr wrote:
 We have the macro keyword. I envision something like:

 macro replaceAggregate(macro newAggregate, macro loop : foreach(x;
 aggr){statements}, macro x, macro aggr, macro statements) {
 foreach(x; newAggregate){statements}
 }

 void main(){
 int[] a = [1,2,3];
 int[] b = [2,3,4];
 replaceAggregate(b, foreach(x;a){writeln(x);});
 }

 (The syntax looks horrible, but you get the idea: AST walking by
 pattern matching)

This looks substantially more complicated than what I had in mind. I think it's a great idea, but something that could be added after the initial functionality was there.
 macro until(macro condition, string str){
 while(!condition){
 mixin(str); // string and mixin not strictly necessary,
 // maybe enable the feature on macro params too
 }
 }

 void main(){
 bool done = false;
 int x;
 until(done){
 done = foo(x++);
 }
 }

 This is just a very rough sketch though, we would need a much more
 refined design. I think getting the symbol scoping right is most
 important.

I'm a bit confused about what's actually going on here, but it certainly looks interesting. What exactly is being passed to "string str" in the macro?

The idea is that if you have something of the form: identifier(arguments) { body } It would get transformed into: identifier(arguments, q{ body });
Mar 23 2012
prev sibling next sibling parent "F i L" <witte2008 gmail.com> writes:
On Thursday, 22 March 2012 at 18:06:24 UTC, Felix Hufnagel wrote:
 in addition to .codeof, let's think about .astof returning an 
 abstract syntax tree.

I haven't really looked at DMD's AST structure, but I assume we could just duplicate that minus any CTFE stuff or codegen stuff.
Mar 22 2012
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 22.03.2012 19:06, schrieb Felix Hufnagel:
 in addition to .codeof, let's think about .astof returning an abstract
 syntax tree.

Would this not require something similar to the expression trees in .NET? http://msdn.microsoft.com/en-us/library/bb397951.aspx -- Paulo
Mar 22 2012
prev sibling next sibling parent "F i L" <witte2008 gmail.com> writes:
Timon Gehr wrote:
 I think that would necessitate the addition of AST macros.

You mean passing DMD's actual AST tree object? Only a stripped down version?
Mar 22 2012
prev sibling next sibling parent "F i L" <witte2008 gmail.com> writes:
Timon Gehr wrote:
 We have the macro keyword. I envision something like:

 macro replaceAggregate(macro newAggregate, macro loop : 
 foreach(x; aggr){statements}, macro x, macro aggr, macro 
 statements) {
     foreach(x; newAggregate){statements}
 }

 void main(){
     int[] a = [1,2,3];
     int[] b = [2,3,4];
     replaceAggregate(b, foreach(x;a){writeln(x);});
 }

 (The syntax looks horrible, but you get the idea: AST walking 
 by pattern matching)

This looks substantially more complicated than what I had in mind. I think it's a great idea, but something that could be added after the initial functionality was there.
 macro until(macro condition, string str){
     while(!condition){
         mixin(str); // string and mixin not strictly necessary,
                     // maybe enable the feature on macro params 
 too
     }
 }

 void main(){
     bool done = false;
     int x;
     until(done){
         done = foo(x++);
     }
 }

 This is just a very rough sketch though, we would need a much 
 more refined design. I think getting the symbol scoping right 
 is most important.

I'm a bit confused about what's actually going on here, but it certainly looks interesting. What exactly is being passed to "string str" in the macro?
Mar 22 2012
prev sibling parent deadalnix <deadalnix gmail.com> writes:
Le 22/03/2012 19:06, Felix Hufnagel a écrit :
 in addition to .codeof, let's think about .astof returning an abstract
 syntax tree.

WAY better !
Mar 24 2012
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
Le 22/03/2012 17:00, F i L a écrit :
 So the discussions about Attributes and Aspect Oriented Programming
 (AOP) got me thinking... Basically AOP requires injecting code fragments
 together in a comprehensible way. Similarly, Attributes that go beyond
  note (such as  GC.NoScan) need similar ability.

 D already has the ability to mixin arbitrary code fragments at compile
 time, and to process those in useful ways through CTFE. Which rocks.
 What it lacks is the ability to reflect upon the actual source code due
 to IO limitations of CTFE. So creating a mixin templates which pieces
 together a unique object is, to my knowledge, currently next to
 impossible (and slow since you'd have to parse and isolate code in .d
 file multiple times in a separate process, then compile again to put it
 all together).

 So, to quote Walter, what compelling features would it bring? Here's an
 example of a simple AOP program from the AOP wiki page (probably not the
 best implementation, but the concept is there):

 struct BankType
 {
 void transfer() { ... }
 void getMoneyBack() { ... }
 }

 struct Logger
 {
 void transfer() {
 log("transferring money...");
 }
 void getMoneyBack() {
 log("User requested money back");
 }
 }

 and now some magic...

 string bankCode(T...)(T aspects) {
 auto code = "struct Bank {";
 auto members = [__traits(allMembers, Bank)];
 foreach (m; members) {
 code ~= "void "~m~"() {";
 code ~= __traits(getMember, Bank, m).codeof;
 foreach (a; aspects) {
 if (__traits(hasMember, a, m) {
 code ~= __traits(getMember, a, m).codeof;
 }
 }
 code ~= "}"
 }
 return code ~ "}";
 }

 mixin template Bank(T...)
 {
 mixin(bankCode(T));
 }

 mixin Bank!Logger;

 void main() {
 auto b = Bank();
 b.transfer(); // logs
 b.getMoneyBack(); // ditto
 }

 So this would allow us to make "Compilers" within the Compiler
 (Codeception), since we could parse/strip/append any existing code
 fragments together in endless combination. Generic "Builders" could
 probably be built and put into a std.builder lib for general use.

 One particular use I have in mind is for Behavior Objects (Game
 Scripts). Each behavior would hold Property(T) objects which define
 per-property, per-state "binding" dependencies (eg.
 position.x.bind(other.x, State.Idle)) and execution code. On release,
 the Property(T) object would be stripped away (leaving just T) and it's
 behavior code "compressed" with others into optimized functions.

 I don't know much about the internals of DMD, so I'm not sure this is a
 realistic request, but I think the idea is compelling. Also, for
 Attributes I'm not sure this technique is really applicable. But it's
 possible that the compiler could exploit this internally for certain
 Attributes like  GC.whatever

As of discussion about properties/attributes, I wanted to do a proposal along that line : ttribute attributeName(AST ast, other template parameters . . .) { // Eponymous trick immutable AST attributeName = process(ast); auto process(AST ast) { // Do some CTFEable ast magic here. return newAst; } } And that would be used that way : attributeName(other template parameters if it make sense) void fun() {} And allow us to do inception within fun. Additionally, I wanted to propose mixin that return the AST of a string. This isn't as easy as you think, it have a lot lot of implication about what you get as string. This is definitively interesting, but require more work for a realistic proposal.
Mar 24 2012
prev sibling next sibling parent "luka8088" <luka8088 owave.net> writes:
Is this at least similar to what you had in mind ?

http://dpaste.dzfl.pl/a5dc2875

module program;

import std.stdio;

mixin template BankAccount () {
   public int amount;
   void deposit (int value) { this.amount += value; }
   void withdraw (int value) { this.amount -= value; }
   auto currentAmount () { return this.amount; }
}

mixin template Logger () {
   void deposit (int value) { writeln("User deposited money"); }
   void withdraw (int value) { writeln("User requested money 
back"); }
}

void main () {

   mixin aspect!("bank", BankAccount, Logger);

   bank b1;
   bank b2;

   b1.deposit(10);
   b1.deposit(20);
   b1.withdraw(5);
   writeln("b1.currentAmount: ", b1.currentAmount);

   b2.deposit(50);
   b2.withdraw(40);
   b2.deposit(100);
   writeln("b2.currentAmount: ", b2.currentAmount);

}


// generic code

mixin template aspect (string name, T...) {
   template aspectDispatch (string name, uint n) {
     import convert = std.conv;
     static if (n >= 1)
       enum aspectDispatch = ""
         ~ "import std.traits;\n"
         ~ "static if (__traits(hasMember, data.aspect_" ~ 
convert.to!string(n) ~ ", `" ~ name ~ "`))\n"
         ~ "  static if (!is(ReturnType!(data.aspect_" ~ 
convert.to!string(n) ~ "." ~ name ~ ") == void))\n"
         ~ "    return data.aspect_" ~ convert.to!string(n) ~ "." 
~ name ~ "(arguments);\n"
         ~ "  else\n"
         ~ "    data.aspect_" ~ convert.to!string(n) ~ "." ~ name 
~ "(arguments);\n"
         ~ aspectDispatch!(name, n - 1)
       ;
     else
       enum aspectDispatch = "";
   }
   auto code () {
     import convert = std.conv;
     string ret = ""
       ~ "struct " ~ name ~ " {\n"
       ~ "  struct aspectData {\n"
     ;
     uint i = 0;
     foreach (a; T)
       ret ~= "    mixin " ~ __traits(identifier, a) ~ " aspect_" 
~ convert.to!string(++i) ~ ";\n";
     ret ~= ""
       ~ "  }\n"
       ~ "  aspectData data;\n"
       ~ "  auto opDispatch (string fn, args...) (args arguments) 
{\n"
       ~ "    mixin(aspectDispatch!(fn, " ~ convert.to!string(i) ~ 
"));\n"
       ~ "  }\n"
       ~ "}\n"
     ;
     return ret;
   }
   mixin(code);
}



On Thursday, 22 March 2012 at 16:00:29 UTC, F i L wrote:
 So the discussions about Attributes and Aspect Oriented 
 Programming (AOP) got me thinking... Basically AOP requires 
 injecting code fragments together in a comprehensible way. 
 Similarly, Attributes that go beyond  note (such as  GC.NoScan) 
 need similar ability.

 D already has the ability to mixin arbitrary code fragments at 
 compile time, and to process those in useful ways through CTFE. 
 Which rocks. What it lacks is the ability to reflect upon the 
 actual source code due to IO limitations of CTFE. So creating a 
 mixin templates which pieces together a unique object is, to my 
 knowledge, currently next to impossible (and slow since you'd 
 have to parse and isolate code in .d file multiple times in a 
 separate process, then compile again to put it all together).

 So, to quote Walter, what compelling features would it bring? 
 Here's an example of a simple AOP program from the AOP wiki 
 page (probably not the best implementation, but the concept is 
 there):

   struct BankType
   {
     void transfer() { ... }
     void getMoneyBack() { ... }
   }

   struct Logger
   {
     void transfer() {
       log("transferring money...");
     }
     void getMoneyBack() {
       log("User requested money back");
     }
   }

 and now some magic...

   string bankCode(T...)(T aspects) {
     auto code = "struct Bank {";
     auto members = [__traits(allMembers, Bank)];
     foreach (m; members) {
       code ~= "void "~m~"() {";
       code ~= __traits(getMember, Bank, m).codeof;
       foreach (a; aspects) {
         if (__traits(hasMember, a, m) {
           code ~= __traits(getMember, a, m).codeof;
         }
       }
       code ~= "}"
     }
     return code ~ "}";
   }

   mixin template Bank(T...)
   {
     mixin(bankCode(T));
   }

   mixin Bank!Logger;

   void main() {
     auto b = Bank();
     b.transfer(); // logs
     b.getMoneyBack(); // ditto
   }

 So this would allow us to make "Compilers" within the Compiler 
 (Codeception), since we could parse/strip/append any existing 
 code fragments together in endless combination. Generic 
 "Builders" could probably be built and put into a std.builder 
 lib for general use.

 One particular use I have in mind is for Behavior Objects (Game 
 Scripts). Each behavior would hold Property(T) objects which 
 define per-property, per-state "binding" dependencies (eg. 
 position.x.bind(other.x, State.Idle)) and execution code. On 
 release, the Property(T) object would be stripped away (leaving 
 just T) and it's behavior code "compressed" with others into 
 optimized functions.

 I don't know much about the internals of DMD, so I'm not sure 
 this is a realistic request, but I think the idea is 
 compelling. Also, for Attributes I'm not sure this technique is 
 really applicable. But it's possible that the compiler could 
 exploit this internally for certain Attributes like  GC.whatever

Oct 09 2012
prev sibling next sibling parent "F i L" <witte2008 gmail.com> writes:
On Tuesday, 9 October 2012 at 13:28:55 UTC, luka8088 wrote:
 Is this at least similar to what you had in mind ?

 [ ..code.. ]

Yes, I realized, a bit after I originally posted that, that my suggestion was already possible if BankType & Logger where mixin-templates instead of struct/classes. Thanks for the code example though. I still think an built-in .codeof/.astof would be nice, but what D really needs to achieve this in a syntactically pleasing and powerful way, is 'macro' templates (like Nimrod has) which work on the AST directly. I doubt this is a major concern ATM however.
Oct 09 2012
prev sibling parent "luka8088" <luka8088 owave.net> writes:
On Tuesday, 9 October 2012 at 19:29:34 UTC, F i L wrote:
 On Tuesday, 9 October 2012 at 13:28:55 UTC, luka8088 wrote:
 Is this at least similar to what you had in mind ?

 [ ..code.. ]

Yes, I realized, a bit after I originally posted that, that my suggestion was already possible if BankType & Logger where mixin-templates instead of struct/classes. Thanks for the code example though. I still think an built-in .codeof/.astof would be nice, but what D really needs to achieve this in a syntactically pleasing and powerful way, is 'macro' templates (like Nimrod has) which work on the AST directly. I doubt this is a major concern ATM however.

My point of making this example was to show that nothing is missing in D itself. You just need to be more creative. If you want to write in a manner more similar to your original example (by that I mean without mixin templates) you can use classes, class methods can be turned to delegates with their context pointer changed before execution, and then you would get the same effect. Also having such syntax could be very confusing because someone could introduce some syntax which is very similar to D but behaves differently and it is embedded in a way that looks just like a D code. Also the idea is to have "// generic code" part in some library and not visible to the *user* so the rest of the code would be syntactically pleasing. If you check current phobos code, you will see that there are some examples of using mixins this way. Please also check the comments on https://github.com/D-Programming-Language/dmd/pull/953 (if you haven't done that already).
Oct 11 2012