www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - AST macros

reply Walter Bright <newshound digitalmars.com> writes:
Marcin Kuszczak wrote:
 Walter Bright wrote:
 It's pretty simple:

 macro foo(args)
 {
 ...syntax that gets inserted...
 }

 and the args and syntax is evaluated in the context of the 


 the macro, not the definition of the macro. You'll be able to do things
 like:

 macro print(arg)
 {
 writefln(__FILE__, __LINE__, arg);
 }

 which get the file and line in the right context. There's no way to do
 that right now.

And what about mixin templates? Couldn't they just be fixed to do

 From docs:
 " Unlike a template instantiation, a template mixin's body is evaluated
 within the scope where the mixin appears, not where the template
 declaration is defined. It is analogous to cutting and pasting the 

 the template into the location of the mixin. It is useful for injecting
 parameterized 'boilerplate' code, as well as for creating templated 

 functions, which is not possible with template instantiations. "

 For me it looks exactly like this what you want to do with macro.

 What would be a difference?

Mixin templates have a fundamental problem with this - the mixin arguments are semantically evaluated before the mixin is instantiated. To manipulate AST's, it's necessary to get at them before the semantic analysis is done.
 For me it seems that the whole mixin thing is rather buggy and 

 the moment - for reference see:
 http://www.digitalmars.com/d/archives/digitalmars/D/learn/3412.html#N3416

They're not buggy, but it's possible they aren't the right design.
Mar 17 2007
next sibling parent Walter Bright <newshound digitalmars.com> writes:
 eao197 wrote:

mixin print(arg) { }

I thought it would be too confusing to have 3 different mixin things.
 What if you want the file in some other context?  It would be nice to have a
complete solution to that, which allows some sort of stack traversal.  Although
I'm not sure its possible, due to not wanting to keep this sort of information
around at release time.
 
 ie:
 
      macro print(arg)
      {
          writefln(__FILE__[stack_level], __LINE__[stack_level], arg);
      }
 
 Or even better:
 
 Stack[0].Line; //Current line
 Stack[1].Line; //Line one level up
 Stack[0].File;
 Stack[0].Module;
 Stack[0].Function;
 Stack[0].NumbArgs; //Some form of reflection
 Stack[0].Arg[N];   //Access to the value of the argument (ie Turple of values)
 Stack[0].ArgIdentifier[N] //String name of the identifier
 Stack[0].FuncType     //The type of function we are in (is it a macro, a
compile time function a member, a regular function)
 ect...

That would be something for much later.
Mar 17 2007
prev sibling next sibling parent reply Walter Bright <newshound digitalmars.com> writes:
 As I can see the content of macro body will be inserted into AST in the 
 place of invocation. But there is not much differences with existing 
 C/C++ macro handling (insertion of right __FILE__, __LINE__ is good 
 thing anyway).
 
 Is there allowed any access to previous parsed entity? For example, can 
 I define macro:
 
 macro some_class_extender(class_name) {
   ...modification of 'class_name' class structure...
 }
 
 class MyCoolClass { ... }
 some_class_extender( MyCoolClass );
 
 and get the modified version of MyCoolClass after some_class_externder 
 invocation?
 
 --Regards,
 Yauheni Akhotnikau

Right now, I see AST macros as manipulating expressions and statements, not declarations.
Mar 17 2007
parent reply eao197 <eao197 intervale.ru> writes:
On Sun, 18 Mar 2007 05:00:34 +0300, Walter Bright  
<newshound digitalmars.com> wrote:

 Right now, I see AST macros as manipulating expressions and statements,  
 not declarations.

If I have understood you correctly the D macros will be used like class_eval/module_eval in Ruby -- for generation new content of the programm. For example, in Ruby I can write: class ProjectDescription # Methods which accept Array as argument. def add_files( files ); ...; end def add_resources( resouces ); ...; end # Helper class instance method for defining singular argument methods # on top on plural form method. def self.define_singular_form_method( method ) class_eval %Q{ # Construct 'method[0...-1]' removes ending 's' from method name. def #{method[0...-1]}(a); #{method}( [ a ] ); end } end ... end The same trick will be available in D via macro: macro defineSingularFormMethod(methodName, paramType) { void methodName[0..$-1]( paramType a ) { methodName( [ a ] ); } } class ProjectDescription { // Methods which accept Array as argument. void addFiles( char[][] files ) { ... } void addResources( char[][] resources ) { ... } // Addition of singular form methods via macros. defineSingularFormMethod( addFiles, char[] ); defineSingularFormMethod( addResources, char[] ); } The main difference between Ruby's class_eval and D's macro is in the time of execution: in Ruby class_eval is executed in run-time, and in D macro is executed in compile-time (at syntax analyzing stage). Am I right? -- Regards, Yauheni Akhotnikau
Mar 18 2007
parent reply Walter Bright <newshound digitalmars.com> writes:
eao197 wrote:
 On Sun, 18 Mar 2007 05:00:34 +0300, Walter Bright 
 <newshound digitalmars.com> wrote:
 
 Right now, I see AST macros as manipulating expressions and 
 statements, not declarations.

If I have understood you correctly the D macros will be used like class_eval/module_eval in Ruby -- for generation new content of the programm. For example, in Ruby I can write:

Macros wouldn't be for adding declarations to classes, for that use template mixins.
Mar 18 2007
parent eao197 <eao197 intervale.ru> writes:
On Sun, 18 Mar 2007 23:30:54 +0300, Walter Bright  =

<newshound digitalmars.com> wrote:

 eao197 wrote:
 On Sun, 18 Mar 2007 05:00:34 +0300, Walter Bright  =


 <newshound digitalmars.com> wrote:

 Right now, I see AST macros as manipulating expressions and  =



 statements, not declarations.



 class_eval/module_eval in Ruby -- for generation new content of the  =


 programm. For example, in Ruby I can write:

Macros wouldn't be for adding declarations to classes, for that use =

 template mixins.

I don't know how use template mixins for such case: // It isn't real D. template ValueAccessor(Type,Name) { Type Name; bool is_<Name>_defined; Type get<Name>() { if( !is_<Name>_defined ) throw new ValueNotDefined("<Name>"); return Name; } void set<Name>( Type value ) { Name =3D value; is_<Name>_defined =3D true; } } class SomeValueHolder { mixin ValueAccessor!( uint, DataCoding ); mixin ValueAccessor!( char[], SourceAddress ); mixin ValueAccessor!( char[], DestinationAddress ); ... // And another bunch of ValueAccessor calls. } auto holder =3D new SomeValueHolder; holder.setDataCoding( 0x01 ); holder.setSourceAddress( "..." ); This can be done via compile-time functions and mixin expression, but I = = don't think it is possible via template mixins. So I think it is better for me to wait while you implement your macro = system in D and see the result. I hope it will be very interesting. Good= = luck! Thank you for your patience. -- = Regards, Yauheni Akhotnikau
Mar 18 2007
prev sibling next sibling parent reply Don Clugston <dac nospam.com.au> writes:
Walter Bright wrote:
 Marcin Kuszczak wrote:
  > Walter Bright wrote:
  >> It's pretty simple:
  >>
  >> macro foo(args)
  >> {
  >> ...syntax that gets inserted...
  >> }
  >>
  >> and the args and syntax is evaluated in the context of the 
 invocation  of
  >> the macro, not the definition of the macro. You'll be able to do things
  >> like:
  >>
  >> macro print(arg)
  >> {
  >> writefln(__FILE__, __LINE__, arg);
  >> }
  >>
  >> which get the file and line in the right context. There's no way to do
  >> that right now.

This looks great already. However, I don't see any specific "abstract syntax tree" syntax in this. Is that still a work-in-progress? An observation: My version of vector operation expression templates is mostly working now, using the existing mixins. It deals with ASTs by constructing a string representing the operations to be performed, and a tuple containing all of the parameters. The operations string is of the form "g+=((a+b)*c)+((d*e)+f)", where a, b, c... correspond to T[0], T[1], T[2]... of the tuple T. A compile-time function converts the string into postfix, and then another CFTE function converts that into optimised x87 asm, which is mixed in. Unexpectedly, I found that it's actually very nice to have flattened tuples. If an AST could recognise that two parameters are the same, it could avoid duplication in the tuple, something that a tree can't easily do: eg somevec+= othervec + somevec*3; could become "T[1]+=T[0]+(T[1]*3)" which allows better code generation in the final stage. It may be useful to separate the tree from the values in this way -- it's not clear to me that (for D) it's desirable to have function calls and parameters mixed together in the way that Lisp does.
Mar 19 2007
parent reply Walter Bright <newshound digitalmars.com> writes:
Don Clugston wrote:
 Walter Bright wrote:
 Marcin Kuszczak wrote:
  > Walter Bright wrote:
  >> It's pretty simple:
  >>
  >> macro foo(args)
  >> {
  >> ...syntax that gets inserted...
  >> }
  >>
  >> and the args and syntax is evaluated in the context of the 
 invocation  of
  >> the macro, not the definition of the macro. You'll be able to do 
 things
  >> like:
  >>
  >> macro print(arg)
  >> {
  >> writefln(__FILE__, __LINE__, arg);
  >> }
  >>
  >> which get the file and line in the right context. There's no way 
 to do
  >> that right now.

This looks great already. However, I don't see any specific "abstract syntax tree" syntax in this. Is that still a work-in-progress?

The beauty of it is that (a+b), before semantic analysis, is an AST! So there's no funky new grammar to learn.
 An observation:
 My version of vector operation expression templates is mostly working 
 now, using the existing mixins. It deals with ASTs by constructing a 
 string representing the operations to be performed, and a tuple 
 containing all of the parameters. The operations string is of the form
 "g+=((a+b)*c)+((d*e)+f)", where a, b, c... correspond to T[0], T[1], 
 T[2]... of the tuple T.
 
 A compile-time function converts the string into postfix, and then 
 another CFTE function converts that into optimised x87 asm, which is 
 mixed in.

Wow! I think the macros can help by obviating the need for the user to do the mixin manually.
 Unexpectedly, I found that it's actually very nice to have flattened 
 tuples. If an AST could recognise that two parameters are the same, it 
 could avoid duplication in the tuple, something that a tree can't easily 
 do:
 eg   somevec+= othervec + somevec*3;
 could become "T[1]+=T[0]+(T[1]*3)" which allows better code generation 
 in the final stage.

There's a way to do this using specialization: macro foo(somevec : somevec+=othervec+somevec, othervec) which will only match for parameters of the form: foo( a += b + a); somevec => a othervec => b It works analogously to how you can match type patterns in templates using specializations.
 It may be useful to separate the tree from the values in this way -- 
 it's not clear to me that (for D) it's desirable to have function calls 
 and parameters mixed together in the way that Lisp does.

Mar 19 2007
parent reply Don Clugston <dac nospam.com.au> writes:
Walter Bright wrote:
 Don Clugston wrote:
 Walter Bright wrote:
 You'll be able to do things like:
  >>
  >> macro print(arg)
  >> {
  >> writefln(__FILE__, __LINE__, arg);
  >> }

This looks great already. However, I don't see any specific "abstract syntax tree" syntax in this. Is that still a work-in-progress?

The beauty of it is that (a+b), before semantic analysis, is an AST! So there's no funky new grammar to learn.

Fantastic! Does this mean that you just use arg.stringof to get the expression? And 'arg' to evaluate it?
 An observation:
 My version of vector operation expression templates is mostly working 
 now, using the existing mixins. It deals with ASTs by constructing a 
 string representing the operations to be performed, and a tuple 
 containing all of the parameters. The operations string is of the form
 "g+=((a+b)*c)+((d*e)+f)", where a, b, c... correspond to T[0], T[1], 
 T[2]... of the tuple T.

 A compile-time function converts the string into postfix, and then 
 another CFTE function converts that into optimised x87 asm, which is 
 mixed in.

Wow! I think the macros can help by obviating the need for the user to do the mixin manually.

Actually usage is OK already. Here's an example: ------ auto p = Vec([1.0, 2, 18]); auto q = Vec([3.5L, 1.1, 3.8]); auto r = Vec([17.0f, 28.1, 1]); q -= ((p+r)*18.0L*314.1L - (p-r))* 35; real d = dot(r, p + 3.7*r); ------ And the only reason for Vec() is because otherwise you can't create array operations. In the first stage, the Vector structs returned by Vec() are removed, and only built-in real[], double[], and float[] vectors are put into the tuple. BTW the generated asm is really good quality; in each case, it generates the best code I can write by hand.
 Unexpectedly, I found that it's actually very nice to have flattened 
 tuples. If an AST could recognise that two parameters are the same, it 
 could avoid duplication in the tuple, something that a tree can't 
 easily do:
 eg   somevec+= othervec + somevec*3;
 could become "T[1]+=T[0]+(T[1]*3)" which allows better code generation 
 in the final stage.

There's a way to do this using specialization: macro foo(somevec : somevec+=othervec+somevec, othervec) which will only match for parameters of the form: foo( a += b + a); somevec => a othervec => b It works analogously to how you can match type patterns in templates using specializations.

Now that could be useful... Are we going to able to write macros for member functions, and operator overloads?
Mar 20 2007
parent Walter Bright <newshound digitalmars.com> writes:
Don Clugston wrote:
 Walter Bright wrote:
 Don Clugston wrote:
 Walter Bright wrote:
 You'll be able to do things like:
  >>
  >> macro print(arg)
  >> {
  >> writefln(__FILE__, __LINE__, arg);
  >> }

This looks great already. However, I don't see any specific "abstract syntax tree" syntax in this. Is that still a work-in-progress?

The beauty of it is that (a+b), before semantic analysis, is an AST! So there's no funky new grammar to learn.

Fantastic! Does this mean that you just use arg.stringof to get the expression? And 'arg' to evaluate it?

I'm not totally sure yet.
 
 An observation:
 My version of vector operation expression templates is mostly working 
 now, using the existing mixins. It deals with ASTs by constructing a 
 string representing the operations to be performed, and a tuple 
 containing all of the parameters. The operations string is of the form
 "g+=((a+b)*c)+((d*e)+f)", where a, b, c... correspond to T[0], T[1], 
 T[2]... of the tuple T.

 A compile-time function converts the string into postfix, and then 
 another CFTE function converts that into optimised x87 asm, which is 
 mixed in.

Wow! I think the macros can help by obviating the need for the user to do the mixin manually.

Actually usage is OK already. Here's an example: ------ auto p = Vec([1.0, 2, 18]); auto q = Vec([3.5L, 1.1, 3.8]); auto r = Vec([17.0f, 28.1, 1]); q -= ((p+r)*18.0L*314.1L - (p-r))* 35; real d = dot(r, p + 3.7*r); ------ And the only reason for Vec() is because otherwise you can't create array operations. In the first stage, the Vector structs returned by Vec() are removed, and only built-in real[], double[], and float[] vectors are put into the tuple. BTW the generated asm is really good quality; in each case, it generates the best code I can write by hand.
 Unexpectedly, I found that it's actually very nice to have flattened 
 tuples. If an AST could recognise that two parameters are the same, 
 it could avoid duplication in the tuple, something that a tree can't 
 easily do:
 eg   somevec+= othervec + somevec*3;
 could become "T[1]+=T[0]+(T[1]*3)" which allows better code 
 generation in the final stage.

There's a way to do this using specialization: macro foo(somevec : somevec+=othervec+somevec, othervec) which will only match for parameters of the form: foo( a += b + a); somevec => a othervec => b It works analogously to how you can match type patterns in templates using specializations.

Now that could be useful... Are we going to able to write macros for member functions, and operator overloads?

Probably not.
Mar 20 2007
prev sibling next sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Walter Bright wrote:
 ...

Question: are AST macros going to use an AST hierarchy exactly like the one in the DMD frontend, or will there be some changes? I say this because I think the DMD frontend AST hierarchy is in some cases quite strange, to say the least. -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Mar 19 2007
parent Walter Bright <newshound digitalmars.com> writes:
Bruno Medeiros wrote:
 Question: are AST macros going to use an AST hierarchy exactly like the 
 one in the DMD frontend, or will there be some changes? I say this 
 because I think the DMD frontend AST hierarchy is in some cases quite 
 strange, to say the least.

You'll just write AST's as normal D code in normal D notation.
Mar 19 2007
prev sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Walter,

I've just been going back over my old (and now fairly crusty) coroutine
implementation, and realised that macros could make these a *major*
feature of D.

Ideally, I'd like to see a syntax like this:

 coro showfiles(char[] msg)(char[], char[]) // <-- input,output types
 {
     char[] path;
     while( (path = yield) != null ) // <-- yield is a protected method
     {
         writefln("%s: %s", msg, path);
     }
 }

Which a macro could then unwind into something ugly that the user doesn't have to see:
 class showfiles : CoroutineT!(char[], char[], char[])
 {
     mixin CoroutineMixin!(char[], char[], char[]);
     void run(char[] msg)
     {
         char[] path;
         while( (path = this.yield) != null )
         {
             writefln("%s: %s", msg, path);
         }
     }
 }

Which is basically how it looks now. Then, using it is nice and clean:
 scope sf = showfiles("Hi there");
 sf("file1");
 sf("file2");

I've tried to come up with other syntaxes using current constructs, and I can't really get any better than having the user define a subclass, mixin a template, and then implement a run method. It just feels... unwieldy. Will the sort of syntax in the first example be possible? If not, do you have any thoughts on how to improve the above kind of construct (either now or with D 2.0)? -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Mar 19 2007
next sibling parent reply "David B. Held" <dheld codelogicconsulting.com> writes:
Daniel Keep wrote:
 Walter,
 
 I've just been going back over my old (and now fairly crusty) coroutine
 implementation, and realised that macros could make these a *major*
 feature of D.
 
 Ideally, I'd like to see a syntax like this:
 
 coro showfiles(char[] msg)(char[], char[]) // <-- input,output types
 {
     char[] path;
     while( (path = yield) != null ) // <-- yield is a protected method
     {
         writefln("%s: %s", msg, path);
     }
 }


I thought Tango had Fibers? Dave
Mar 19 2007
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
David B. Held wrote:
 Daniel Keep wrote:
 Walter,

 I've just been going back over my old (and now fairly crusty) coroutine
 implementation, and realised that macros could make these a *major*
 feature of D.

 Ideally, I'd like to see a syntax like this:

 coro showfiles(char[] msg)(char[], char[]) // <-- input,output types
 {
     char[] path;
     while( (path = yield) != null ) // <-- yield is a protected method
     {
         writefln("%s: %s", msg, path);
     }
 }


I thought Tango had Fibers? Dave

Cooperative multithreading does not a coroutine make. Fibers, IIRC, are just stack-based threads. Coroutines are *entirely different*, although in this particular case they are implemented *using* stack threads. In any case, I have a working library right now... it's just that the syntax to use them is rather ugly and a bit messy. -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Mar 19 2007
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Daniel Keep wrote:
 
 David B. Held wrote:
 Daniel Keep wrote:
 Walter,

 I've just been going back over my old (and now fairly crusty) coroutine
 implementation, and realised that macros could make these a *major*
 feature of D.

 Ideally, I'd like to see a syntax like this:

 coro showfiles(char[] msg)(char[], char[]) // <-- input,output types
 {
     char[] path;
     while( (path = yield) != null ) // <-- yield is a protected method
     {
         writefln("%s: %s", msg, path);
     }
 }


Dave

Cooperative multithreading does not a coroutine make. Fibers, IIRC, are just stack-based threads. Coroutines are *entirely different*, although in this particular case they are implemented *using* stack threads. In any case, I have a working library right now... it's just that the syntax to use them is rather ugly and a bit messy.

For what it's worth, Mikola Lysenko has coroutines as a part of his StackThreads implementation, and it should be pretty easy to adapt them to use Tango's Fibers. The page is here: http://www.assertfalse.com/projects.shtml Sean
Mar 19 2007
parent Sean Kelly <sean f4.ca> writes:
Sean Kelly wrote:
 
 For what it's worth, Mikola Lysenko has coroutines as a part of his 
 StackThreads implementation, and it should be pretty easy to adapt them 
 to use Tango's Fibers.  The page is here: 
 http://www.assertfalse.com/projects.shtml

My fault. It seems that's the one you wrote :-p Sean
Mar 19 2007
prev sibling parent reply "David B. Held" <dheld codelogicconsulting.com> writes:
Daniel Keep wrote:
 
 [...]
 Cooperative multithreading does not a coroutine make.  Fibers, IIRC, are
 just stack-based threads.  Coroutines are *entirely different*, although
 in this particular case they are implemented *using* stack threads.
 [...]

Wikipedia seems to think they are essentially the same thing: http://en.wikipedia.org/wiki/Fiber_%28computer_science%29 Fibers are less than stack-based threads because they are cooperative. Whereas in a coroutine you write "return", in a fiber you write "yield". Otherwise, I believe they are closely isomorphic. Dave
Mar 20 2007
parent Daniel Keep <daniel.keep.lists gmail.com> writes:
David B. Held wrote:
 Daniel Keep wrote:
 [...]
 Cooperative multithreading does not a coroutine make.  Fibers, IIRC, are
 just stack-based threads.  Coroutines are *entirely different*, although
 in this particular case they are implemented *using* stack threads.
 [...]

Wikipedia seems to think they are essentially the same thing: http://en.wikipedia.org/wiki/Fiber_%28computer_science%29 Fibers are less than stack-based threads because they are cooperative. Whereas in a coroutine you write "return", in a fiber you write "yield". Otherwise, I believe they are closely isomorphic. Dave

Hmm... looks that way. I think mine are probably closer to generators or channels than strictly coroutines, the difference being that Fibers just do a transfer of control to the given fiber, whereas mine work more like a function call. Actually, thinking about it, I'm not entirely sure if mine classify as coroutines or not... oh bother :P -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Mar 20 2007
prev sibling parent reply Walter Bright <newshound digitalmars.com> writes:
Daniel Keep wrote:
 Will the sort of syntax in the first example be possible?  If not, do
 you have any thoughts on how to improve the above kind of construct
 (either now or with D 2.0)?

I'm not understanding your example. Where is the macro?
Mar 20 2007
parent Daniel Keep <daniel.keep.lists gmail.com> writes:
Walter Bright wrote:
 Daniel Keep wrote:
 Will the sort of syntax in the first example be possible?  If not, do
 you have any thoughts on how to improve the above kind of construct
 (either now or with D 2.0)?

I'm not understanding your example. Where is the macro?

I want "coro" to be the macro. Basically, I want it to look like a function, except it has a second argument list, where you can stick in the input/output types. You said before that you can do specialisations on syntax, so I suppose it might be something like... macro coro(name : name(typelist)(arglist){block}, typelist, arglist, block) { // blah } At this stage, I'm just trying to describe the kind of syntax I'd like to be able to make, and then you say "you can't do that, but you can do this", and then I say "well what if we changed this little bit here", and so forth until either one of us goes mad or we come to some sort of middle-ground :P Plus, I figure the more potential use-cases we show you, the better the end result will be :) -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Mar 20 2007