www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP 50 - AST macros

reply Jacob Carlborg <doob me.com> writes:
I've been thinking quite long of how AST macros could look like in D. 
I've been posting my vision of AST macros here in the newsgroup a couple 
of times already. I've now been asked to create a DIP out of it, so here 
it is:

http://wiki.dlang.org/DIP50

-- 
/Jacob Carlborg
Nov 10 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Jacob Carlborg:

 http://wiki.dlang.org/DIP50
I suggest to add some more use cases (possibly with their implementation). Bye, bearophile
Nov 10 2013
next sibling parent reply "Rob T" <alanb ucora.com> writes:
On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote:
 Jacob Carlborg:

 http://wiki.dlang.org/DIP50
I suggest to add some more use cases (possibly with their implementation). Bye, bearophile
A scalable and elegant solution to the "inspection" problem may be a AST use case. Discussed here: http://forum.dlang.org/thread/qconpedgdkyeawmdzect forum.dlang.org --rt
Nov 10 2013
next sibling parent reply Timothee Cour <thelastmammoth gmail.com> writes:
<[ .. ]> syntax looks very foreign. How about m{...} (with q{...} as a
precedent)?


On Sun, Nov 10, 2013 at 5:08 PM, Rob T <alanb ucora.com> wrote:

 On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote:

 Jacob Carlborg:

  http://wiki.dlang.org/DIP50

 I suggest to add some more use cases (possibly with their implementation).

 Bye,
 bearophile
A scalable and elegant solution to the "inspection" problem may be a AST use case. Discussed here: http://forum.dlang.org/thread/qconpedgdkyeawmdzect forum.dlang.org --rt
Nov 10 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 02:40, Timothee Cour wrote:
 <[ .. ]> syntax looks very foreign. How about m{...} (with q{...} as a
 precedent)?
As the DIP says, that just one suggestion of the syntax. If we even need a syntax. An alternative, without any new syntax is the "ast" macro: http://wiki.dlang.org/DIP50#The_AST_Macro This is similar to the "refiy" macro in Scala. -- /Jacob Carlborg
Nov 10 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 02:08, Rob T wrote:

 A scalable and elegant solution to the "inspection" problem may be a AST
 use case.

 Discussed here:
 http://forum.dlang.org/thread/qconpedgdkyeawmdzect forum.dlang.org
Please, feel free to add it as an example, if your interested. After all, it is a wiki. -- /Jacob Carlborg
Nov 10 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-10 23:33, bearophile wrote:
 Jacob Carlborg:

 http://wiki.dlang.org/DIP50
I suggest to add some more use cases (possibly with their implementation).
I could try and do that. The problem is I have not given much though on how the reflection API and the API used to build syntax trees should look like. If this get any serious interest and we start to think about the implementation I was hoping that someone we more experience could help out with designing the API. -- /Jacob Carlborg
Nov 10 2013
prev sibling parent reply "simendsjo" <simendsjo gmail.com> writes:
On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote:
 Jacob Carlborg:

 http://wiki.dlang.org/DIP50
I suggest to add some more use cases (possibly with their implementation). Bye, bearophile
I agree examples would help a lot. Trying to define what information actually exists within these types would also help a lot. In the first example, would Ast!(bool) be something like this? opBinary!"==" left = opBinary!"+" left = Literal type = int value = 1 right = Literal type = int value = 2 right = Literal type = int value = 4 Would there be helpers for matching part of the structure? The same applies to the other types used - what information should they have? As for examples, here's a couple of suggestions: * Expression to prefix notation * Expression to SQL * AutoImplement properties (like C#) * Number intervals, like "int i = int[10..20];" where only 10 to 20 are legal values * Discriminated union * Pattern matching
Nov 11 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 09:28, simendsjo wrote:

 I agree examples would help a lot. Trying to define what information
 actually exists within these types would also help a lot.
I can tell you right now that I haven't figured out everything, far from everything.
 In the first example, would Ast!(bool) be something like this?
    opBinary!"=="
      left = opBinary!"+"
               left = Literal
                 type = int
                 value = 1
               right = Literal
                 type = int
                 value = 2
      right = Literal
        type = int
        value = 4

 Would there be helpers for matching part of the structure?
I don't know. As I said above, I haven't figured out everything. I don't see any support for this in the language. But of course, there can be library functions that help with this.
 The same applies to the other types used - what information should they
 have?
Suggestions are welcome. Also, see my reply to bearophile http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com#post-l5q1p5:242bs4:241:40digitalmars.com
 As for examples, here's a couple of suggestions:
 * Expression to prefix notation
 * Expression to SQL
 * AutoImplement properties (like C#)
Would that be something like the attribute macro example? http://wiki.dlang.org/DIP50#Attribute_macros
 * Number intervals, like "int i = int[10..20];" where only 10 to 20 are
 legal values
 * Discriminated union
 * Pattern matching
Pattern matching usually requires new syntax, which this proposal does not support. -- /Jacob Carlborg
Nov 11 2013
prev sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 11.11.2013 09:28, schrieb simendsjo:
 On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote:
 Jacob Carlborg:

 http://wiki.dlang.org/DIP50
I suggest to add some more use cases (possibly with their implementation). Bye, bearophile
I agree examples would help a lot. Trying to define what information actually exists within these types would also help a lot. In the first example, would Ast!(bool) be something like this? opBinary!"==" left = opBinary!"+" left = Literal type = int value = 1 right = Literal type = int value = 2 right = Literal type = int value = 4 Would there be helpers for matching part of the structure? The same applies to the other types used - what information should they have? As for examples, here's a couple of suggestions: * Expression to prefix notation * Expression to SQL * AutoImplement properties (like C#) * Number intervals, like "int i = int[10..20];" where only 10 to 20 are legal values * Discriminated union * Pattern matching
can you add the example to the DIP?
Nov 11 2013
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Jacob Carlborg:

 http://wiki.dlang.org/DIP50
It's also useful to take a look at what F# is doing: http://tomasp.net/blog/2013/computation-zoo-padl/index.html Bye, bearophile
Nov 10 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 02:46, bearophile wrote:

 It's also useful to take a look at what F# is doing:
 http://tomasp.net/blog/2013/computation-zoo-padl/index.html
I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#. -- /Jacob Carlborg
Nov 10 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/10/13 11:46 PM, Jacob Carlborg wrote:
 On 2013-11-11 02:46, bearophile wrote:

 It's also useful to take a look at what F# is doing:
 http://tomasp.net/blog/2013/computation-zoo-padl/index.html
I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#.
There is significant regret about the way macros are defined in Scala. Probably not an example to follow. Andrei
Nov 11 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 12 November 2013 at 02:37:03 UTC, Andrei Alexandrescu 
wrote:
 On 11/10/13 11:46 PM, Jacob Carlborg wrote:
 On 2013-11-11 02:46, bearophile wrote:

 It's also useful to take a look at what F# is doing:
 http://tomasp.net/blog/2013/computation-zoo-padl/index.html
I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#.
There is significant regret about the way macros are defined in Scala. Probably not an example to follow.
Do you have some article/documentation on that ? BTW, scala implemented macros as a plugin called macro paradise, which allowed people to play with it without crippling the core language. We should probably introduce new features that way.
Nov 11 2013
prev sibling next sibling parent reply "Rob T" <alanb ucora.com> writes:
On Tuesday, 12 November 2013 at 02:37:03 UTC, Andrei Alexandrescu 
wrote:
 On 11/10/13 11:46 PM, Jacob Carlborg wrote:
 On 2013-11-11 02:46, bearophile wrote:

 It's also useful to take a look at what F# is doing:
 http://tomasp.net/blog/2013/computation-zoo-padl/index.html
I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#.
There is significant regret about the way macros are defined in Scala. Probably not an example to follow. Andrei
Knowing what went wrong is important so as not to repeat past mistakes, is there more info somewhere to read on what went wrong? --rt
Nov 11 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/13 9:16 PM, Rob T wrote:
 On Tuesday, 12 November 2013 at 02:37:03 UTC, Andrei Alexandrescu wrote:
 On 11/10/13 11:46 PM, Jacob Carlborg wrote:
 On 2013-11-11 02:46, bearophile wrote:

 It's also useful to take a look at what F# is doing:
 http://tomasp.net/blog/2013/computation-zoo-padl/index.html
I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#.
There is significant regret about the way macros are defined in Scala. Probably not an example to follow. Andrei
Knowing what went wrong is important so as not to repeat past mistakes, is there more info somewhere to read on what went wrong?
Not yet. Martin mentioned to me in 2012 he doesn't think they took the right turns with defining macros. I don't know whether more information is available at the moment. Andrei
Nov 11 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 12 November 2013 at 05:59:46 UTC, Andrei Alexandrescu 
wrote:
 Not yet. Martin mentioned to me in 2012 he doesn't think they 
 took the right turns with defining macros. I don't know whether 
 more information is available at the moment.

 Andrei
Loosely related, but worth watching : http://www.infoq.com/presentations/data-types-issues
Nov 11 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 03:37, Andrei Alexandrescu wrote:

 There is significant regret about the way macros are defined in Scala.
 Probably not an example to follow.
Unfortunately it's not easy to do anything about it if you cannot give us any more details to why. So far I haven't found anything. -- /Jacob Carlborg
Nov 11 2013
prev sibling parent "jerro" <a a.com> writes:
 There is significant regret about the way macros are defined in 
 Scala. Probably not an example to follow.

 Andrei
Not an example to follow, but it would definitely be worth looking into how they defined macros and what aspect of that caused problems, so we can avoid making the same mistakes.
Nov 12 2013
prev sibling next sibling parent reply Timothee Cour <thelastmammoth gmail.com> writes:
People have shunned proposals to have  mixin functions because it wouldn't
be obvious at call site that some statement is executed under a mixin
(which could access all variables in scope etc).

The same will happen here; I think it should be clear at call site that a
macro is used.
How about:

macro!myAssert(1 + 2 == 4);
instead of myAssert(1 + 2 == 4);



On Sun, Nov 10, 2013 at 1:20 PM, Jacob Carlborg <doob me.com> wrote:

 I've been thinking quite long of how AST macros could look like in D. I've
 been posting my vision of AST macros here in the newsgroup a couple of
 times already. I've now been asked to create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50

 --
 /Jacob Carlborg
Nov 10 2013
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11/11/2013 02:49 AM, Timothee Cour wrote:
 People have shunned proposals to have  mixin functions because it
 wouldn't be obvious at call site that some statement is executed under a
 mixin (which could access all variables in scope etc).

 The same will happen here; ...
macros are hygienic.
Nov 10 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 02:49, Timothee Cour wrote:
 People have shunned proposals to have  mixin functions because it
 wouldn't be obvious at call site that some statement is executed under a
 mixin (which could access all variables in scope etc).

 The same will happen here; I think it should be clear at call site that
 a macro is used.
 How about:

 macro!myAssert(1 + 2 == 4);
 instead of myAssert(1 + 2 == 4);
I would really prefer if we not had to do that. -- /Jacob Carlborg
Nov 10 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 11 November 2013 at 01:49:45 UTC, Timothee Cour wrote:
 People have shunned proposals to have  mixin functions because 
 it wouldn't
 be obvious at call site that some statement is executed under a 
 mixin
 (which could access all variables in scope etc).

 The same will happen here; I think it should be clear at call 
 site that a
 macro is used.
 How about:

 macro!myAssert(1 + 2 == 4);
 instead of myAssert(1 + 2 == 4);
If macros are supposed to access outer scope, I agree, this is a necessary restriction.
Nov 11 2013
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/11/2013 01:55 PM, Dicebot wrote:
 On Monday, 11 November 2013 at 01:49:45 UTC, Timothee Cour wrote:
 People have shunned proposals to have  mixin functions because it
 wouldn't
 be obvious at call site that some statement is executed under a mixin
 (which could access all variables in scope etc).

 The same will happen here; I think it should be clear at call site that a
 macro is used.
 How about:

 macro!myAssert(1 + 2 == 4);
 instead of myAssert(1 + 2 == 4);
If macros are supposed to access outer scope, I agree, this is a necessary restriction.
I don't agree. (The argument against implicit mixin features was a complete lack of hygiene, not the mere possibility of accessing the caller scope.)
Nov 11 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 11 November 2013 at 16:28:17 UTC, Timon Gehr wrote:
 I don't agree. (The argument against implicit mixin features 
 was a complete lack of hygiene, not the mere possibility of 
 accessing the caller scope.)
How would you expect it to be hygienic in any way if it has implicit outer scope access? Reasoning is pretty much the same as with mixins.
Nov 11 2013
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/11/2013 05:47 PM, Dicebot wrote:
 On Monday, 11 November 2013 at 16:28:17 UTC, Timon Gehr wrote:
 I don't agree. (The argument against implicit mixin features was a
 complete lack of hygiene, not the mere possibility of accessing the
 caller scope.)
How would you expect it to be hygienic in any way if it has implicit outer scope access? Reasoning is pretty much the same as with mixins.
With mixins _every_ identifier is by _default_ inserted/looked up in the so-called 'outer scope', whereas declarations and normal symbol lookups within macros are hygienic by default. Macros are as much about reflection as about code generation. The trade-offs are simly not the same. Implicit outer scope access does not imply it behaves in a counter-intuitive manner. (It could just reflect some debug information after all, or perform some additional checks on the context for more safety, etc.) I will be really unhappy with any proposal that does not allow to interchange function calls with macro calls transparently, even though such a scheme allows more abusive code to be written as well. If in some code base macros should be clearly marked, a naming convention (eg. prefix macro names with 'macro') can be used. (Enforcing such a convention is trivial and amounts to eg. a few lines of additional code in a modified D front end.)
Nov 11 2013
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11/11/2013 10:23 PM, Timon Gehr wrote:
  simly
simply
Nov 11 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 17:28, Timon Gehr wrote:

 I don't agree. (The argument against implicit mixin features was a
 complete lack of hygiene, not the mere possibility of accessing the
 caller scope.)
The macros system need to provide both hygienic and non-hygienic macros. -- /Jacob Carlborg
Nov 11 2013
prev sibling next sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-10 21:20:30 +0000, Jacob Carlborg said:

 I've been thinking quite long of how AST macros could look like in D. 
 I've been posting my vision of AST macros here in the newsgroup a 
 couple of times already. I've now been asked to create a DIP out of it, 
 so here it is:
 
 http://wiki.dlang.org/DIP50
Not a huge fan of the syntax, but I think it's a step in the right direction away from string mixins. R/ Shammah
Nov 10 2013
next sibling parent Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-11 03:04:34 +0000, Shammah Chancellor said:

 On 2013-11-10 21:20:30 +0000, Jacob Carlborg said:
 
 I've been thinking quite long of how AST macros could look like in D. 
 I've been posting my vision of AST macros here in the newsgroup a 
 couple of times already. I've now been asked to create a DIP out of it, 
 so here it is:
 
 http://wiki.dlang.org/DIP50
Not a huge fan of the syntax, but I think it's a step in the right direction away from string mixins. R/ Shammah
I think this would gain a lot more traction if someone wrote an std.expressions which produced workable code from a AST. At that point, it's just for the D-frontend to parse code into said std.expressions. The .NET expressions framework might servce as some insperation. R/ Shammah
Nov 10 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 04:04, Shammah Chancellor wrote:

 Not a huge fan of the syntax, but I think it's a step in the right
 direction away from string mixins.
To you mean the <[ ]> syntax or something else? See my reply to Timothee: http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com?page=2#post-l5q1vh:242c2k:241:40digitalmars.com -- /Jacob Carlborg
Nov 10 2013
prev sibling next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like 
 in D. I've been posting my vision of AST macros here in the 
 newsgroup a couple of times already. I've now been asked to 
 create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50
One of our targets for AST macros should be the ability to replicate roughly linq from c# / .net. An example syntax for use with AST could be: auto data = [5, 7, 9]; int[] data2; query { from value in data where value >= 6 add to data2 } Could be unwrapped to: auto data = [5, 7, 9]; int[] data2; foreach(value; data) { if (value >= 6) data2 ~= value; } This isn't a thought out design but it should at least be a target or a possibility. Also c#'s get set should be rather similar as well. Example: getset { public int id; private bool exit; } Would translate to: private int id; private bool exit; property { void id(int v) {this.id = v;} void exit(bool v) { this.exit = v; } int id() { return this.id; } bool exit() { return this.exit; } } This would definitely open new possibilities up. Disclaimer I don't like c# or .net but I am partial to these features. At current point I think the DIP does have the necessary features to implement this. However it would be nice for safety to be able to get all scope variables of where the macro was initiated from. Being able to check for if a variable exists could provide much needed compile safety and better error messages.
Nov 10 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 08:46, Rikki Cattermole wrote:

 One of our targets for AST macros should be the ability to replicate
 roughly linq from c# / .net.

 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
   from value in data
   where value >= 6
   add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
   if (value >= 6) data2 ~= value;
 }

 This isn't a thought out design but it should at least be a target or a
 possibility.
Absolutely. One of my favorite examples is the database query: auto person = Person.where(e => e.name == "John"); Which translates to the following SQL: select * from person where name = 'John'
 Also c#'s get set should be rather similar as well.
 Example:

 getset {
   public int id;
   private bool exit;
 }

 Would translate to:

 private int id;
 private bool exit;
  property {
   void id(int v) {this.id = v;}
   void exit(bool v) { this.exit = v; }
   int id() { return this.id; }
   bool exit() { return this.exit; }
 }

 This would definitely open new possibilities up.
 Disclaimer I don't like c# or .net but I am partial to these features.
That's quite similar one of the examples, I like to call it "property shortcut": http://wiki.dlang.org/DIP50#Attribute_macros
 At current point I think the DIP does have the necessary features to
 implement this. However it would be nice for safety to be able to get
 all scope variables of where the macro was initiated from. Being able to
 check for if a variable exists could provide much needed compile safety
 and better error messages.
Why not? There's quite a lot that is not specified in this DIP. Mostly because I haven't decided/figured out how it should work exactly. Of course, any help is always appreciated. -- /Jacob Carlborg
Nov 11 2013
next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 08:12:59 UTC, Jacob Carlborg wrote:
 On 2013-11-11 08:46, Rikki Cattermole wrote:

 One of our targets for AST macros should be the ability to 
 replicate
 roughly linq from c# / .net.

 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
  from value in data
  where value >= 6
  add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
  if (value >= 6) data2 ~= value;
 }

 This isn't a thought out design but it should at least be a 
 target or a
 possibility.
Absolutely. One of my favorite examples is the database query: auto person = Person.where(e => e.name == "John"); Which translates to the following SQL: select * from person where name = 'John'
 Also c#'s get set should be rather similar as well.
 Example:

 getset {
  public int id;
  private bool exit;
 }

 Would translate to:

 private int id;
 private bool exit;
  property {
  void id(int v) {this.id = v;}
  void exit(bool v) { this.exit = v; }
  int id() { return this.id; }
  bool exit() { return this.exit; }
 }

 This would definitely open new possibilities up.
 Disclaimer I don't like c# or .net but I am partial to these 
 features.
That's quite similar one of the examples, I like to call it "property shortcut": http://wiki.dlang.org/DIP50#Attribute_macros
 At current point I think the DIP does have the necessary 
 features to
 implement this. However it would be nice for safety to be able 
 to get
 all scope variables of where the macro was initiated from. 
 Being able to
 check for if a variable exists could provide much needed 
 compile safety
 and better error messages.
Why not? There's quite a lot that is not specified in this DIP. Mostly because I haven't decided/figured out how it should work exactly. Of course, any help is always appreciated.
Theres a few other things I think would need to be cleared up. For example take this code: shader { program vertex { #version 150 void main() { } } fragment { #version 150 void main() { } } } At this point the vertex / fragment segments would be treated as strings instead of calling their macros. So perhaps a rule to identify macros inside macros calls? It would save a lot of time for parsing reasons. Also can you alias a macro? alias fragment = tostring; That would make things a lot simpler creating nice structures of them. Perhaps a condition on a macro like the if's we got for templates would be useful in the sense of being able to say: if (lexer.compareLine(0, "[a-zA-Z]{1}[a-zA-Z_0-9]*") && lexer.isSymbol(1, SymbolTypes.Macro, vertex)) This would require us to develop a lexer library. But if done right I don't see why it wouldn't be usable for more than just D macro checks. Preferably also for e.g. c/c++ wink wink for when D's front end is in D.
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 09:31, Rikki Cattermole wrote:

 Theres a few other things I think would need to be cleared up.
 For example take this code:

 shader {
   program
   vertex {
    #version 150
    void main() {
    }
   }
   fragment {
    #version 150
    void main() {
    }
   }
 }

 At this point the vertex / fragment segments would be treated as strings
 instead of calling their macros. So perhaps a rule to identify macros
 inside macros calls? It would save a lot of time for parsing reasons.
They way I see that is the AST of the whole block would be passed to "shader". I have though a bit about macros inside macros, but I haven't come to a conclusion. The easiest way from a design point of view seems to be that "shader" to basically return the whole AST it receives plus any addition it needs to do. The the "vertex" and "fragment" macros are expanded recursively.
 Also can you alias a macro?

 alias fragment = tostring;

 That would make things a lot simpler creating nice structures of them.
I don't see why not.
 Perhaps a condition on a macro like the if's we got for templates would
 be useful in the sense of being able to say:

 if (lexer.compareLine(0, "[a-zA-Z]{1}[a-zA-Z_0-9]*") &&
 lexer.isSymbol(1, SymbolTypes.Macro, vertex))
I don't think I understand how this should be used.
 This would require us to develop a lexer library. But if done right I
 don't see why it wouldn't be usable for more than just D macro checks.
 Preferably also for e.g. c/c++ wink wink for when D's front end is in D.
-- /Jacob Carlborg
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 09:24:31 UTC, Jacob Carlborg wrote:
 On 2013-11-11 09:31, Rikki Cattermole wrote:

 Theres a few other things I think would need to be cleared up.
 For example take this code:

 shader {
  program
  vertex {
   #version 150
   void main() {
   }
  }
  fragment {
   #version 150
   void main() {
   }
  }
 }

 At this point the vertex / fragment segments would be treated 
 as strings
 instead of calling their macros. So perhaps a rule to identify 
 macros
 inside macros calls? It would save a lot of time for parsing 
 reasons.
They way I see that is the AST of the whole block would be passed to "shader". I have though a bit about macros inside macros, but I haven't come to a conclusion. The easiest way from a design point of view seems to be that "shader" to basically return the whole AST it receives plus any addition it needs to do. The the "vertex" and "fragment" macros are expanded recursively.
 Also can you alias a macro?

 alias fragment = tostring;

 That would make things a lot simpler creating nice structures 
 of them.
I don't see why not.
 Perhaps a condition on a macro like the if's we got for 
 templates would
 be useful in the sense of being able to say:

 if (lexer.compareLine(0, "[a-zA-Z]{1}[a-zA-Z_0-9]*") &&
 lexer.isSymbol(1, SymbolTypes.Macro, vertex))
I don't think I understand how this should be used.
 This would require us to develop a lexer library. But if done 
 right I
 don't see why it wouldn't be usable for more than just D macro 
 checks.
 Preferably also for e.g. c/c++ wink wink for when D's front 
 end is in D.
An example of this might be: macro foo (Context context, Ast!(string) str) if (lexer.isSymbol(0, SymbolTypes.Class)) { return str; } class Bar { int i; } foo { Bar(7); } The above code would succeed however the below code will give a compiler error stating that the given macro is not found. struct Haz { int i; } foo { Haz(9); } The above code is based on the assumption of a D lexer in phobos. This is simpler set of code which won't require a lexer. macro foo (Context context, Ast!(string) str) if (str == "working") { return ""; } foo {fails} foo {working} In these cases its a simple string test to determine if the text given is specified value. In the original example I gave, I was using regex to make the point of validation for a given line passed to the macro.
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 10:38, Rikki Cattermole wrote:

 An example of this might be:

 macro foo (Context context, Ast!(string) str)
 if (lexer.isSymbol(0, SymbolTypes.Class))
 {
      return str;
 }

 class Bar {
   int i;
 }

 foo {
   Bar(7);
 }
 The above code would succeed however the below code will give a compiler
 error stating that the given macro is not found.

 struct Haz {
   int i;
 }

 foo {
   Haz(9);
 }

 The above code is based on the assumption of a D lexer in phobos.
 This is simpler set of code which won't require a lexer.

 macro foo (Context context, Ast!(string) str)
 if (str == "working")
 {
      return "";
 }

 foo {fails}
 foo {working}
 In these cases its a simple string test to determine if the text given
 is specified value.

 In the original example I gave, I was using regex to make the point of
 validation for a given line passed to the macro.
My idea of handling errors in macros is more something like triggering a real compile error. That could be done via the context parameter, something like: macro foo (Context context, Ast!(string) str) { if (str.eval() == "bar") context.error("Illegal value 'bar'"); // this will trigger the compile error } -- /Jacob Carlborg
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 10:04:31 UTC, Jacob Carlborg wrote:
 On 2013-11-11 10:38, Rikki Cattermole wrote:

 An example of this might be:

 macro foo (Context context, Ast!(string) str)
 if (lexer.isSymbol(0, SymbolTypes.Class))
 {
     return str;
 }

 class Bar {
  int i;
 }

 foo {
  Bar(7);
 }
 The above code would succeed however the below code will give 
 a compiler
 error stating that the given macro is not found.

 struct Haz {
  int i;
 }

 foo {
  Haz(9);
 }

 The above code is based on the assumption of a D lexer in 
 phobos.
 This is simpler set of code which won't require a lexer.

 macro foo (Context context, Ast!(string) str)
 if (str == "working")
 {
     return "";
 }

 foo {fails}
 foo {working}
 In these cases its a simple string test to determine if the 
 text given
 is specified value.

 In the original example I gave, I was using regex to make the 
 point of
 validation for a given line passed to the macro.
My idea of handling errors in macros is more something like triggering a real compile error. That could be done via the context parameter, something like: macro foo (Context context, Ast!(string) str) { if (str.eval() == "bar") context.error("Illegal value 'bar'"); // this will trigger the compile error }
For errors I was thinking a new pragma. pragma(error, "You did x wrong"); This would fire an error and display the text given. Returning error code as expected. It is after all in the same league as msg. It would also remove our current static asserts in our code. So other words it could be replicated by: pragma(msg, "You did x wrong"); static assert(0); Just without the whole failed assert message. Although what I was showing was the idea of overloading of macros like we have in templates.
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 11:11, Rikki Cattermole wrote:

 For errors I was thinking a new pragma.
 pragma(error, "You did x wrong");
 This would fire an error and display the text given. Returning error
 code as expected. It is after all in the same league as msg.
 It would also remove our current static asserts in our code.
 So other words it could be replicated by:

 pragma(msg, "You did x wrong");
 static assert(0);

 Just without the whole failed assert message.
I like the context parameter better. I'm imagine the context parameter containing a lot of information and functionality to interact with the compiler. A pragma seems not be general enough.
 Although what I was showing was the idea of overloading of macros like
 we have in templates.
Ok, I see. -- /Jacob Carlborg
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 10:19:49 UTC, Jacob Carlborg wrote:
 On 2013-11-11 11:11, Rikki Cattermole wrote:

 For errors I was thinking a new pragma.
 pragma(error, "You did x wrong");
 This would fire an error and display the text given. Returning 
 error
 code as expected. It is after all in the same league as msg.
 It would also remove our current static asserts in our code.
 So other words it could be replicated by:

 pragma(msg, "You did x wrong");
 static assert(0);

 Just without the whole failed assert message.
I like the context parameter better. I'm imagine the context parameter containing a lot of information and functionality to interact with the compiler. A pragma seems not be general enough.
I can understand wanting to have a high level version of it in context. I'm just considering a high reuse language feature that could back it. Really it should have been added a while ago given the static assert trick is used a bit. I do actually like the idea of having the ability to call it on context with a bunch of extra information given to it for you. e.g. line number of initiation ext.
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 11:28, Rikki Cattermole wrote:

 I can understand wanting to have a high level version of it in context.
 I'm just considering a high reuse language feature that could back it.
 Really it should have been added a while ago given the static assert
 trick is used a bit.
 I do actually like the idea of having the ability to call it on context
 with a bunch of extra information given to it for you. e.g. line number
 of initiation ext.
I think we want to have as much as possible of this collected in the same place. I don't think pragmas are a good fit for this. -- /Jacob Carlborg
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 10:47:37 UTC, Jacob Carlborg wrote:
 On 2013-11-11 11:28, Rikki Cattermole wrote:

 I can understand wanting to have a high level version of it in 
 context.
 I'm just considering a high reuse language feature that could 
 back it.
 Really it should have been added a while ago given the static 
 assert
 trick is used a bit.
 I do actually like the idea of having the ability to call it 
 on context
 with a bunch of extra information given to it for you. e.g. 
 line number
 of initiation ext.
I think we want to have as much as possible of this collected in the same place. I don't think pragmas are a good fit for this.
Yes I do agree with that. I'm just more worried about what backs it then pleasantries that we can give the developer. Also if you're not looking towards pragma's to hook into stopping compilation what are your ideas and thoughts around this issue? Because at this point I only know of static assert to do it.
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 11:51, Rikki Cattermole wrote:

 Yes I do agree with that. I'm just more worried about what backs it then
 pleasantries that we can give the developer.
 Also if you're not looking towards pragma's to hook into stopping
 compilation what are your ideas and thoughts around this issue? Because
 at this point I only know of static assert to do it.
A function call on the context parameter as I've already showed. This would just call "error" which is already implemented in the compiler. Although I don't know how to actually do the connection between the compiler internals the user code. I think this is one of the big challenges with this DIP. For this function I guess it could be done like an intrinsic (is that the correct word?). -- /Jacob Carlborg
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 10:59:28 UTC, Jacob Carlborg wrote:
 A function call on the context parameter as I've already 
 showed. This would just call "error" which is already 
 implemented in the compiler. Although I don't know how to 
 actually do the connection between the compiler internals the 
 user code. I think this is one of the big challenges with this 
 DIP. For this function I guess it could be done like an 
 intrinsic (is that the correct word?).
Ok pragmas essentially call functions like error in the compiler [0]. What I am thinking error will do is instead of outputting like msg it'll call error[1]. This is exactly what you want. [0] https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2862 [1] https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2957
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 12:04, Rikki Cattermole wrote:

 Ok pragmas essentially call functions like error in the compiler [0].
 What I am thinking error will do is instead of outputting like msg it'll
 call error[1]. This is exactly what you want.

 [0]
 https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2862

 [1]
 https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2957
Yes, I still don't understand why you would want it as a pragma. Be usable outside of macros? -- /Jacob Carlborg
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 12:30:07 UTC, Jacob Carlborg wrote:
 Yes, I still don't understand why you would want it as a 
 pragma. Be usable outside of macros?
Yes outside of macros would be useful. For example code like this would become redundant: pragma(msg, "Support for x is not implemented on platform y"); static assert(0); Becoming: pragma(error, "Support for x is not implemented on platform y"); Also pragma's core responsibility is to cause the compiler to do something. In this case to say we hit an error during compilation please tell the user/dev and die. It is a hook to the compilers workings. Currently working on getting this implemented. Nearly done with it. Just got some extra spaces that shouldn't be in output.
Nov 11 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 13:36, Rikki Cattermole wrote:

 Yes outside of macros would be useful. For example code like this would
 become redundant:
 pragma(msg, "Support for x is not implemented on platform y");
 static assert(0);

 Becoming:
 pragma(error, "Support for x is not implemented on platform y");

 Also pragma's core responsibility is to cause the compiler to do
 something. In this case to say we hit an error during compilation please
 tell the user/dev and die.

 It is a hook to the compilers workings.

 Currently working on getting this implemented. Nearly done with it. Just
 got some extra spaces that shouldn't be in output.
I just don't want to add a bunch of pragmas. I don't know what's the best solution. -- /Jacob Carlborg
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 13:16:23 UTC, Jacob Carlborg wrote:
 I just don't want to add a bunch of pragmas. I don't know 
 what's the best solution.
I am only suggesting one :) We only need one that'll output any text we want it to and make the compiler consider it an error. On the macro side of things we can build a string which then can be given to the pragma. All of the wrapping giving the goodness would go into the context. Current what I have enables this: void main() { U u; } alias T!("hi", "bye") U; struct T(string a, string b) { pragma(error, a ~ "\n" ~ b); } test.d(8): Error: hi bye test.d(5): Error: template instance test.T!("hi", "bye") error instantiating Not really in pull state but thats the best I can do. Really that final error should be omitted.
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 14:32, Rikki Cattermole wrote:

 I am only suggesting one :)
What about warnings? And no, the current pragma(msg) isn't the same.
 We only need one that'll output any text we want it to and make the
 compiler consider it an error.
 On the macro side of things we can build a string which then can be
 given to the pragma.

 All of the wrapping giving the goodness would go into the context.

 Current what I have enables this:

 void main() {
      U u;
 }

 alias T!("hi", "bye") U;

 struct T(string a, string b) {
      pragma(error, a ~ "\n" ~ b);
 }

 test.d(8): Error: hi
 bye

 test.d(5): Error: template instance test.T!("hi", "bye") error
 instantiating

 Not really in pull state but thats the best I can do. Really that final
 error should be omitted.
I wouldn't say no to this if macros are completely off the table. Perhaps it's a good addition regardless. -- /Jacob Carlborg
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 13:36:14 UTC, Jacob Carlborg wrote:
 What about warnings? And no, the current pragma(msg) isn't the 
 same.
I'm not doing one for warnings. See how the error one goes. The warning one won't stop compilation and pragma msg can be used to fake it.
 I wouldn't say no to this if macros are completely off the 
 table. Perhaps it's a good addition regardless.
Personally I think it was already needed even without macros. Just this has started me off on actually looking into it. We'll see what the compiler devs think though.
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 14:40, Rikki Cattermole wrote:

 I'm not doing one for warnings. See how the error one goes. The warning
 one won't stop compilation and pragma msg can be used to fake it.
Yes, but I want warnings to be available in the context parameter as well. That's what I've been talking about all along. It's not just errors, it's all of the stuff that can be added. Warnings can be turned off, pragma(msg) cannot (at least not as far as I know).
 Personally I think it was already needed even without macros. Just this
 has started me off on actually looking into it. We'll see what the
 compiler devs think though.
I see. I'm not saying that we don't need it. -- /Jacob Carlborg
Nov 11 2013
parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 14:10:32 UTC, Jacob Carlborg wrote:
 On 2013-11-11 14:40, Rikki Cattermole wrote:

 I'm not doing one for warnings. See how the error one goes. 
 The warning
 one won't stop compilation and pragma msg can be used to fake 
 it.
Yes, but I want warnings to be available in the context parameter as well. That's what I've been talking about all along. It's not just errors, it's all of the stuff that can be added. Warnings can be turned off, pragma(msg) cannot (at least not as far as I know).
Ah ok, I'll see what I can drum up in that regard then. I guess I was wrong about being able to fake it with pragma msg.
Nov 11 2013
prev sibling next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 11.11.2013 13:36, schrieb Rikki Cattermole:
 On Monday, 11 November 2013 at 12:30:07 UTC, Jacob Carlborg wrote:
 Yes, I still don't understand why you would want it as a
 pragma. Be usable outside of macros?
Yes outside of macros would be useful. For example code like this would become redundant: pragma(msg, "Support for x is not implemented on platform y"); static assert(0); Becoming: pragma(error, "Support for x is not implemented on platform y"); Also pragma's core responsibility is to cause the compiler to do something. In this case to say we hit an error during compilation please tell the user/dev and die. It is a hook to the compilers workings. Currently working on getting this implemented. Nearly done with it. Just got some extra spaces that shouldn't be in output.
but in macros the context information is much more interesting then anything else - so #pragma(error,...) won't fully fit the needs to context error returning - it could be much more then just a message example: to help the compiler forming better error messages - or maybe recover from deep macro expansion etc...) what you want is just #pragma(error,...) -> break compiliation now Jacob is talking about the feedback for the compiler thing...
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 13:22:34 UTC, dennis luehring 
wrote:
 example: to help the compiler forming better error messages - 
 or maybe recover from deep macro expansion etc...)

 what you want is just #pragma(error,...) -> break compiliation 
 now
 Jacob is talking about the feedback for the compiler thing...
Jacob mentioned originally he wanted to create a compiler error. Or at least thats is how I took it. Hence pragma(error, "text"); We can already message out at both compile and runtime its simply does the compiler recognise it as an error part.
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 14:36, Rikki Cattermole wrote:

 Jacob mentioned originally he wanted to create a compiler error. Or at
 least thats is how I took it. Hence pragma(error, "text");
 We can already message out at both compile and runtime its simply does
 the compiler recognise it as an error part.
"error" would be just a single function that is available in the context parameter. There would be many others. From the DIP: * The arguments used when the compiler was invoked * Functions for emitting messages of various verbosity level, like error, warning and info * Functions for querying various types of settings/options, like which versions are defined, is "debug" or "release" defined and so on * In general providing as much as possible of what the compiler knows about the compile run * The context should have an associative array with references to all scoped variables at initiation point. -- /Jacob Carlborg
Nov 11 2013
next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 14:13:31 UTC, Jacob Carlborg wrote:
 On 2013-11-11 14:36, Rikki Cattermole wrote:

 Jacob mentioned originally he wanted to create a compiler 
 error. Or at
 least thats is how I took it. Hence pragma(error, "text");
 We can already message out at both compile and runtime its 
 simply does
 the compiler recognise it as an error part.
"error" would be just a single function that is available in the context parameter. There would be many others. From the DIP: * The arguments used when the compiler was invoked * Functions for emitting messages of various verbosity level, like error, warning and info * Functions for querying various types of settings/options, like which versions are defined, is "debug" or "release" defined and so on * In general providing as much as possible of what the compiler knows about the compile run * The context should have an associative array with references to all scoped variables at initiation point.
Is there anything specific for info that I should look at? or does pragma msg do everything for you?
Nov 11 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 16:24, Rikki Cattermole wrote:

 Is there anything specific for info that I should look at? or does
 pragma msg do everything for you?
Sorry, I don't understand what you're meaning. -- /Jacob Carlborg
Nov 11 2013
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/11/2013 03:13 PM, Jacob Carlborg wrote:
 "error" would be just a single function that is available in the context
 parameter. There would be many others. From the DIP:
"error" should also be able to specify where the error occurred in the input, so that it is displayed at the call site.
Nov 11 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 17:23, Timon Gehr wrote:

 "error" should also be able to specify where the error occurred in the
 input, so that it is displayed at the call site.
That's a good point. -- /Jacob Carlborg
Nov 11 2013
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/11/2013 01:36 PM, Rikki Cattermole wrote:
 Yes outside of macros would be useful. For example code like this would
 become redundant:
 pragma(msg, "Support for x is not implemented on platform y");
 static assert(0);

 Becoming:
 pragma(error, "Support for x is not implemented on platform y");
static assert(0, "Support for x is not implemented on platform y");
Nov 11 2013
parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 16:22:40 UTC, Timon Gehr wrote:
 On 11/11/2013 01:36 PM, Rikki Cattermole wrote:
 Yes outside of macros would be useful. For example code like 
 this would
 become redundant:
 pragma(msg, "Support for x is not implemented on platform y");
 static assert(0);

 Becoming:
 pragma(error, "Support for x is not implemented on platform 
 y");
static assert(0, "Support for x is not implemented on platform y");
Hmm wasn't aware of that syntax. Needs an example I think on docs. I will still work on pull request as the point of an assert and pragma message is different, atleast I believe.
Nov 11 2013
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Jacob Carlborg:

 One of my favorite examples is the database query:

 auto person = Person.where(e => e.name == "John");

 Which translates to the following SQL:

 select * from person where name = 'John'
Can't you do the same thing with functions similar (same API but different semantics) to std.algorithm ones that generate expression templates? auto person = persons.filter!(e => e.name == "John"); -------- simendsjo:
 * Number intervals, like "int i = int[10..20];" where only 10 
 to 20 are legal values
What's wrong with this syntax that doesn't reqiire macros? It's more uniform with the rest of the language: Ranged!(int, 10, 20) i; Bye, bearophile
Nov 11 2013
next sibling parent reply "simendsjo" <simendsjo gmail.com> writes:
On Monday, 11 November 2013 at 09:02:01 UTC, bearophile wrote:
 Jacob Carlborg:

 One of my favorite examples is the database query:

 auto person = Person.where(e => e.name == "John");

 Which translates to the following SQL:

 select * from person where name = 'John'
Can't you do the same thing with functions similar (same API but different semantics) to std.algorithm ones that generate expression templates? auto person = persons.filter!(e => e.name == "John");
The problem here is that a library need to know that it has to create SELECT * FROM persons WHERE name = 'John'; and not SELECT * FROM persons; and filter it locally. So it needs a way to inspect the body of the delegate and extract "name" "==" and "John".
 --------

 simendsjo:

 * Number intervals, like "int i = int[10..20];" where only 10 
 to 20 are legal values
What's wrong with this syntax that doesn't reqiire macros? It's more uniform with the rest of the language: Ranged!(int, 10, 20) i; Bye, bearophile
Nothing wrong with it, I was just trying to come up with some examples. I don't say they're necessarily good :)
Nov 11 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 10:23, simendsjo wrote:
 On Monday, 11 November 2013 at 09:02:01 UTC, bearophile wrote:
 auto person = persons.filter!(e => e.name == "John");
The problem here is that a library need to know that it has to create SELECT * FROM persons WHERE name = 'John'; and not SELECT * FROM persons; and filter it locally. So it needs a way to inspect the body of the delegate and extract "name" "==" and "John".
This particular example would work. "e" would be a proxy which implements opDispatch which returns another proxy which overloads opEquals. The problem is that != and all the comparison operators won't work. See my reply to bearophile: http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com?page=3#post-l5q892:242kpf:241:40digitalmars.com -- /Jacob Carlborg
Nov 11 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 10:02, bearophile wrote:

 Can't you do the same thing with functions similar (same API but
 different semantics) to std.algorithm ones that generate expression
 templates?

 auto person = persons.filter!(e => e.name == "John");
I have been thinking about that, a solution where "e" is a proxy that implements opDispatch and which returns a new proxy which overloads "==". The problem with this is that you cannot separately overload the equal and comparison operators. In D the all the comparisons operators are implement with the single overload, opCmp. You cannot know from inside of opCmp if it's the "<" or the ">" operator that is being called. Same problem with opEquals. -- /Jacob Carlborg
Nov 11 2013
prev sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
 One of our targets for AST macros should be the ability to
 replicate roughly linq from c# / .net.

 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
    from value in data
    where value >= 6
    add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
    if (value >= 6) data2 ~= value;
 }
could you add the example to the DIP wiki page
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 09:01:31 UTC, dennis luehring 
wrote:
 One of our targets for AST macros should be the ability to
 replicate roughly linq from c# / .net.

 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
   from value in data
   where value >= 6
   add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
   if (value >= 6) data2 ~= value;
 }
could you add the example to the DIP wiki page
Adding use case section with Linq example. Please remove / modify as required as is first time editing wiki.
Nov 11 2013
parent reply Timothee Cour <thelastmammoth gmail.com> writes:
I would really like to use AST macros for the following use case:

myAssert( x < y);
//will print, on failure, a message, along with all values appearing inside
macro myAssert:
x<y failed: x=..., y=...

myAssert( fun(x,y)==z1+z2)
//likewise, but nesting down to all individual variables appearing inside
the macro:
x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=...

This would advantageously replace the plethora of unittest helpers found in
other languages, eg: CHECK_EQ, CHECK_LEQ, etc.
Invaluable for debugging or informative unittests and logs.





On Mon, Nov 11, 2013 at 1:52 AM, Rikki Cattermole
<alphaglosined gmail.com>wrote:

 On Monday, 11 November 2013 at 09:01:31 UTC, dennis luehring wrote:

 One of our targets for AST macros should be the ability to
 replicate roughly linq from c# / .net.

 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
   from value in data
   where value >= 6
   add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
   if (value >= 6) data2 ~= value;
 }
could you add the example to the DIP wiki page
Adding use case section with Linq example. Please remove / modify as required as is first time editing wiki.
Nov 11 2013
next sibling parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 10:10:33 UTC, Timothee Cour wrote:
 I would really like to use AST macros for the following use 
 case:

 myAssert( x < y);
 //will print, on failure, a message, along with all values 
 appearing inside
 macro myAssert:
 x<y failed: x=..., y=...

 myAssert( fun(x,y)==z1+z2)
 //likewise, but nesting down to all individual variables 
 appearing inside
 the macro:
 x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=...

 This would advantageously replace the plethora of unittest 
 helpers found in
 other languages, eg: CHECK_EQ, CHECK_LEQ, etc.
 Invaluable for debugging or informative unittests and logs.





 On Mon, Nov 11, 2013 at 1:52 AM, Rikki Cattermole
 <alphaglosined gmail.com>wrote:

 On Monday, 11 November 2013 at 09:01:31 UTC, dennis luehring 
 wrote:

 One of our targets for AST macros should be the ability to
 replicate roughly linq from c# / .net.

 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
   from value in data
   where value >= 6
   add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
   if (value >= 6) data2 ~= value;
 }
could you add the example to the DIP wiki page
Adding use case section with Linq example. Please remove / modify as required as is first time editing wiki.
Perhaps an alternative would be: asserts { x < y z == z } Where the macro would do something like this once converted: if (!(x < y)) writeln("x < y failed with x=", x, " y= ", y); if (!(z == z)) writeln("z == z failed with z=", z); This would also have the benefit of allowing for multiple assert statements for one block.
Nov 11 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 11:10, Timothee Cour wrote:
 I would really like to use AST macros for the following use case:

 myAssert( x < y);
 //will print, on failure, a message, along with all values appearing
 inside macro myAssert:
 x<y failed: x=..., y=...

 myAssert( fun(x,y)==z1+z2)
 //likewise, but nesting down to all individual variables appearing
 inside the macro:
 x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=...

 This would advantageously replace the plethora of unittest helpers found
 in other languages, eg: CHECK_EQ, CHECK_LEQ, etc.
 Invaluable for debugging or informative unittests and logs.
Agree. That's the first example in the DIP. Although a very simplified version. -- /Jacob Carlborg
Nov 11 2013
parent reply Timothee Cour <thelastmammoth gmail.com> writes:
On Mon, Nov 11, 2013 at 2:21 AM, Jacob Carlborg <doob me.com> wrote:

 On 2013-11-11 11:10, Timothee Cour wrote:

 I would really like to use AST macros for the following use case:

 myAssert( x < y);
 //will print, on failure, a message, along with all values appearing
 inside macro myAssert:
 x<y failed: x=..., y=...

 myAssert( fun(x,y)==z1+z2)
 //likewise, but nesting down to all individual variables appearing
 inside the macro:
 x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=...

 This would advantageously replace the plethora of unittest helpers found
 in other languages, eg: CHECK_EQ, CHECK_LEQ, etc.
 Invaluable for debugging or informative unittests and logs.
Agree. That's the first example in the DIP. Although a very simplified version.
yes, I think your initial example becomes more interesting with this, as in its current form it can already be done in current D. This would make error messages self-documenting: myAssert(!file.exists ); // "!file. exists" failed: dump of AST: file: string ="foobar.d" |_file.exists: bool = false |_!file. exists: bool = false In many cases, this would be so much more useful than an out of date / incomplete string error message, esp w a lot of variables involved.
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 11:39, Timothee Cour wrote:

 yes, I think your initial example becomes more interesting with this, as
 in its current form it can already be done in current D.
The current example prints out the exact expression what was passed to the assert macro. Not the result of evaluating the expression. I don't see how this can currently be done in D
 This would make error messages self-documenting:

 myAssert(!file.exists );
 // "!file. exists" failed: dump of AST:
 file: string ="foobar.d"
 |_file.exists: bool = false
    |_!file. exists: bool = false

 In many cases, this would be so much more useful than an out of date /
 incomplete string error message, esp w a lot of variables involved.
I agree. -- /Jacob Carlborg
Nov 11 2013
next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 10:55:23 UTC, Jacob Carlborg wrote:
 The current example prints out the exact expression what was 
 passed to the assert macro. Not the result of evaluating the 
 expression. I don't see how this can currently be done in D
I made a suggestion regarding a macro able to get scoped variables from initiation point with Linq. And why that would be a good thing. This is another use case for it. Perhaps add it as a use case for having that feature on the DIP?
Nov 11 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 11:58, Rikki Cattermole wrote:

 I made a suggestion regarding a macro able to get scoped variables from
 initiation point with Linq. And why that would be a good thing. This is
 another use case for it. Perhaps add it as a use case for having that
 feature on the DIP?
I already have an example with the assert (not among the use cases but at the top). Or were you referring to something else? -- /Jacob Carlborg
Nov 11 2013
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 11 November 2013 at 11:01:41 UTC, Jacob Carlborg wrote:
 I already have an example with the assert (not among the use 
 cases but at the top). Or were you referring to something else?
I was referring to the use case as to why we should have the ability to get scoped variables from the initiation point. Example: void func(int i) { bool b; macr {} } macro foo (Context context, Ast!(string) str) { writeln(context.scopeVariables!int("i")); writeln(context.scopeVariables!bool("b")); return ""; }
Nov 11 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 12:08, Rikki Cattermole wrote:

 I was referring to the use case as to why we should have the ability to
 get scoped variables from the initiation point.
 Example:

 void func(int i) {
      bool b;
      macr {}
 }

 macro foo (Context context, Ast!(string) str)
 {
      writeln(context.scopeVariables!int("i"));
      writeln(context.scopeVariables!bool("b"));
      return "";
 }
Ok, I see. Please add additional examples to the DIP if you think it's missing. -- /Jacob Carlborg
Nov 11 2013
prev sibling parent reply Timothee Cour <thelastmammoth gmail.com> writes:
On Mon, Nov 11, 2013 at 2:55 AM, Jacob Carlborg <doob me.com> wrote:

 On 2013-11-11 11:39, Timothee Cour wrote:

  yes, I think your initial example becomes more interesting with this, as
 in its current form it can already be done in current D.
The current example prints out the exact expression what was passed to the assert macro. Not the result of evaluating the expression. I don't see how this can currently be done in D
I'm using a modified assert that uses the ugly import(file)[line] trick, so that can be done (but is really ugly and has impact either runtime or compile time). But we're essentially in agreement.
  This would make error messages self-documenting:
 myAssert(!file.exists );
 // "!file. exists" failed: dump of AST:
 file: string ="foobar.d"
 |_file.exists: bool = false
    |_!file. exists: bool = false

 In many cases, this would be so much more useful than an out of date /
 incomplete string error message, esp w a lot of variables involved.
I agree. -- /Jacob Carlborg
Nov 11 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 12:06, Timothee Cour wrote:

 I'm using a modified assert that uses the ugly import(file)[line] trick,
 so that can be done (but is really ugly and has impact either runtime or
 compile time). But we're essentially in agreement.
Right, that ugly trick :) -- /Jacob Carlborg
Nov 11 2013
prev sibling next sibling parent reply luka8088 <luka8088 owave.net> writes:
On 10.11.2013. 22:20, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:
 
 http://wiki.dlang.org/DIP50
 
Thumbs up!
Nov 11 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 09:08, luka8088 wrote:

 Thumbs up!
Thanks. -- /Jacob Carlborg
Nov 11 2013
prev sibling next sibling parent reply "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like 
 in D. I've been posting my vision of AST macros here in the 
 newsgroup a couple of times already. I've now been asked to 
 create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50
I like it, Jacob. With or without <[ ]> it is a good proposal, and I see several places where I would use it. +1
Nov 11 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 10:07, Dejan Lekic wrote:

 I like it, Jacob. With or without <[ ]> it is a good proposal, and I see
 several places where I would use it.

 +1
Thank you. -- /Jacob Carlborg
Nov 11 2013
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11-Nov-2013 01:20, Jacob Carlborg пишет:
 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:

 http://wiki.dlang.org/DIP50
I have to say I like it. I find it nice that it would allow one to finally ensure that inlining is always done by making critical primitives macros instead of functions. This is vastly superior compared to the _only_ current option of generating unrolled code with string mixins. I'd call attribute macros - declaration macros in line with statement macros, then describe how they look - a lot like attributes. Things to note though: - Can't implement scope(exit/success/failure) because that would need to capture the outer scope to pack it into try/catch/finally. - Can't implement foreach if only because of the syntax of the latter. - There is no definition of Ast!(T) construct - I take it's a struct or a class of sorts and the whole thing works with the CTFE engine. Then it also needs specification of Statements, Declarations. - It seems like I can use AST literal in R-T code as well? That would be useful but only if some library actually accepted these AST nodes. -- Dmitry Olshansky
Nov 11 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 12:34, Dmitry Olshansky wrote:

 I have to say I like it.
Cool :)
 I find it nice that it would allow one to finally ensure that inlining
 is always done by making critical primitives macros instead of
 functions. This is vastly superior compared to the _only_ current option
 of generating unrolled code with string mixins.

 I'd call attribute macros - declaration macros in line with statement
 macros, then describe how they look - a lot like attributes.
Ok, I can change that.
 Things to note though:
   - Can't implement scope(exit/success/failure) because that would need
 to capture the outer scope to pack it into try/catch/finally.
Right. scope(exit) could perhaps be implement with a custom struct and calling a block in the destructor.
   - Can't implement foreach if only because of the syntax of the latter.
I would think something like this: macro foreach (T) (Context context, Ast!(Symbol) var, Ast!(T) collection, Statement block) { if (isRange(collection)) { return <[ for (auto __r = $collection; !__r.empty; __r.popFront()) { auto $var = __r.front; block; } ]>; } else if (hasOpApply(collection)) { ... } else ... } Perhaps not all of the existing syntax can fit in a macro.
   - There is no definition of Ast!(T) construct - I take it's a struct
 or a class of sorts and the whole thing works with the CTFE engine. Then
 it also needs specification of Statements, Declarations.
Yes, something like that. I don't know exactly how the Ast type need to look like. See one of my replies to bearophile: http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com
   - It seems like I can use AST literal in R-T code as well? That would
 be useful but only if some library actually accepted these AST nodes.
Actually, I haven't thought about this. I think my original idea was that <[ ]> should only work inside macros. But that it would be possible to pass Ast nodes to other functions. -- /Jacob Carlborg
Nov 11 2013
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 10 Nov 2013 22:20:30 +0100
schrieb Jacob Carlborg <doob me.com>:

 I've been thinking quite long of how AST macros could look like in D. 
 I've been posting my vision of AST macros here in the newsgroup a couple 
 of times already. I've now been asked to create a DIP out of it, so here 
 it is:
 
 http://wiki.dlang.org/DIP50
Wow, some topics really explode recently. Is the D user base growing or are AST macros drawing so much attention? -- Marco
Nov 11 2013
next sibling parent "Chris Cain" <clcain uncg.edu> writes:
On Monday, 11 November 2013 at 12:13:54 UTC, Marco Leise wrote:
 Wow, some topics really explode recently. Is the D user base
 growing or are AST macros drawing so much attention?
I'd like to say both. AST macros are one of those few things that I wouldn't mind seeing breaking changes for and I'm in the camp of "avoid breaking changes at all cost". AST macros are to code what templates are to data. Despite it being possible to go without templates, I'm not at all tempted to switch to Go.
Nov 11 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-11 13:13, Marco Leise wrote:

 Wow, some topics really explode recently. Is the D user base
 growing or are AST macros drawing so much attention?
AST macros are drawing so much attension. For some, it's like the holy grail. -- /Jacob Carlborg
Nov 11 2013
parent "bearophile" <bearophileHUGS lycos.com> writes:
Jacob Carlborg:

 AST macros are drawing so much attension. For some, it's like 
 the holy grail.
That's a bad approach to technology design. You first need to create a good list of things that you want to do, then you have to try to do them without new features, and then if something important is not possible without the new feature (or it becomes too much ugly, too much long, too much bug-prone, etc), you have to see if the complexity of introducing the new feature is paid back. Macros are a significant feature, so they have a significant cost in code complexity, language dis-uniformity, ecc. Bye, bearophile
Nov 11 2013
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like 
 in D. I've been posting my vision of AST macros here in the 
 newsgroup a couple of times already. I've now been asked to 
 create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50
I don't have time to investigate this in details but I'd like to mention that this is one of few possible new features where ROI is really worth it. It removes lot of burden from core language and allows to replace some very arcane template hacks with cleaner code. I'd say it is a bit too early to go for it (really, we need at least to figure out how to make releases without issues first :)) but it is a good future goal even in context of general feature freezing.
Nov 11 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 11 November 2013 at 13:26:37 UTC, Dicebot wrote:
 On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg 
 wrote:
 I've been thinking quite long of how AST macros could look 
 like in D. I've been posting my vision of AST macros here in 
 the newsgroup a couple of times already. I've now been asked 
 to create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50
I don't have time to investigate this in details but I'd like to mention that this is one of few possible new features where ROI is really worth it. It removes lot of burden from core language and allows to replace some very arcane template hacks with cleaner code. I'd say it is a bit too early to go for it (really, we need at least to figure out how to make releases without issues first :)) but it is a good future goal even in context of general feature freezing.
Can you stop reading my mind, please ?
Nov 11 2013
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 11/10/2013 01:20 PM, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:

 http://wiki.dlang.org/DIP50
For macros that generate macros, I think you need a way to escape the splicing and maybe define how splicing works in an inner quasi quote. I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]> Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with the inner quasi quote, you'd have to do something like <[ <[ <[ \\\$exp ]> ]> ]> maybe a way to associate a splice with a quasi quote? <a[ <b[ <c[ $(a, exp) + $(b, exp) + $(c, exp) ]> ]> ]> a's exp is A b's exp is B c's exp is C then splice the quasi quote N times: 1: ast <b[ <c[ A + $(b, exp) + $(c, exp) ]> ]> 2: ast <c[ A + B + $(c, exp) ]> 3: ast A + B + C just dinking around here
Nov 11 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-11 20:23, Ellery Newcomer wrote:

 For macros that generate macros, I think you need a way to escape the
 splicing and maybe define how splicing works in an inner quasi quote.

 I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]>

 Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with
 the inner quasi quote, you'd have to do something like

 <[ <[ <[ \\\$exp ]> ]> ]>

 maybe a way to associate a splice with a quasi quote?

 <a[ <b[ <c[ $(a, exp) + $(b, exp) + $(c, exp)  ]> ]> ]>

 a's exp is A
 b's exp is B
 c's exp is C

 then splice the quasi quote N times:

 1: ast <b[ <c[ A + $(b, exp) + $(c, exp) ]> ]>
 2: ast <c[ A + B + $(c, exp) ]>
 3: ast A + B + C

 just dinking around here
This would require some thought. -- /Jacob Carlborg
Nov 11 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 11 November 2013 at 19:23:21 UTC, Ellery Newcomer 
wrote:
 On 11/10/2013 01:20 PM, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look 
 like in D.
 I've been posting my vision of AST macros here in the 
 newsgroup a couple
 of times already. I've now been asked to create a DIP out of 
 it, so here
 it is:

 http://wiki.dlang.org/DIP50
For macros that generate macros, I think you need a way to escape the splicing and maybe define how splicing works in an inner quasi quote. I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]> Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with the inner quasi quote, you'd have to do something like <[ <[ <[ \\\$exp ]> ]> ]> maybe a way to associate a splice with a quasi quote? <a[ <b[ <c[ $(a, exp) + $(b, exp) + $(c, exp) ]> ]> ]> a's exp is A b's exp is B c's exp is C then splice the quasi quote N times: 1: ast <b[ <c[ A + $(b, exp) + $(c, exp) ]> ]> 2: ast <c[ A + B + $(c, exp) ]> 3: ast A + B + C just dinking around here
$ refers to the enclosing scope. so $$foo should refers to $foo in the enclosing scope. No need for special rule or label.
Nov 11 2013
parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 11/11/2013 12:06 PM, deadalnix wrote:
 On Monday, 11 November 2013 at 19:23:21 UTC, Ellery Newcomer wrote:
 On 11/10/2013 01:20 PM, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:

 http://wiki.dlang.org/DIP50
For macros that generate macros, I think you need a way to escape the splicing and maybe define how splicing works in an inner quasi quote. I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]> Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with the inner quasi quote, you'd have to do something like <[ <[ <[ \\\$exp ]> ]> ]> maybe a way to associate a splice with a quasi quote? <a[ <b[ <c[ $(a, exp) + $(b, exp) + $(c, exp) ]> ]> ]> a's exp is A b's exp is B c's exp is C then splice the quasi quote N times: 1: ast <b[ <c[ A + $(b, exp) + $(c, exp) ]> ]> 2: ast <c[ A + B + $(c, exp) ]> 3: ast A + B + C just dinking around here
$ refers to the enclosing scope. so $$foo should refers to $foo in the enclosing scope. No need for special rule or label.
so if I have to splice my ast N times, then I have to generate M $'s depending on when I want it to expand?
Nov 11 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like 
 in D. I've been posting my vision of AST macros here in the 
 newsgroup a couple of times already. I've now been asked to 
 create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50
Just a quick comment related to the 2007 proposal: http://youtu.be/FRfTk44nuWE?t=1h5m38s Page 45: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf Based on my read though, yours is more powerful. The 2007 proposal is basically the "Quasi-Quoting" portions of your proposal (which is such a small part of your proposal).
Nov 11 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/10/2013 1:20 PM, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like in D. I've been
 posting my vision of AST macros here in the newsgroup a couple of times
already.
 I've now been asked to create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50
I confess I have some serious reservations about AST macros in general: 1. I've seen very heavy use of such macros in macro assemblers. What happens is people use it to invent their own (very baroque) language on top of the existing assembler one. Anyone trying to read the code has to learn this new unique language, and given the limitations of the macro capability, it often comes with unfixable bizarre corner cases and limitations. This got so bad that in some cases I know of, the poor sap who is given the job of making improvements resorted to running the assembler to generate an object file, then disassembling the object file back into source code! Something went very wrong for this to be necessary. I know in my own assembler work I have abandoned all use of macros. (Note that macros in typical assemblers are far more powerful than C's macro language, which stands out for being pathetically underpowered.) 2. Macros are something that should be used very sparingly. However, this is not what happens. I used to be perplexed that even top shelf C/C++ programmers tend to write what I not so humbly consider to be pretty abusive use of macros. Now it just saddens me. 3. Lisp is a language that encourages users to write macros to pretty much invent a custom language for the task at hand. Heavy use of such makes the code unrecognizable as being Lisp code. I believe a language ought to have some "anchors" that the person reading the code can reliably recognize. (GO has taken this idea pretty far.) 4. AST macros, pretty much by definition, manipulate the AST. However, if you examine the semantic routines in DMD, a *lot* more is required to do more significant things. The symbol table must be consulted, etc. I just don't see how one could hope to implement something like function overloading with AST macros. Or template type deduction. Or closure semantics, attribute inference, etc. Sure, some forms of foreach can be rewritten into for statements by manipulating the AST, but not the more interesting forms, such as the ones that deal with opApply(). There are some statements in the DIP about the Context parameter and it'll provide semantic information, but I don't see how this can work without making it an ever-expanding collection of arbitrary methods. 5. We've said "no" in the past to things like user-defined tokens, which are closely related. 6. Note that D has a limited form of AST macros with mixin templates. To sum up, any AST macro proposal has some tough barriers to get over. Within its necessarily severe semantic limitations, it has to deliver a pretty compelling set of solutions, while avoiding the morass of incomprehensibility that far too many macro systems become in practice. For example, the Linq example in the DIP is not compelling, as aesthetically nicer code can be written using D's ranges and algorithms: auto data = arr.filter!(x => x > 5).array; I see no compelling advantage in trying to make D code look like C#; to be blunt it's like the old: #define BEGIN { #define END } macros used in old C code to make it look like Pascal.
Nov 12 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
I forgot to mention that "expression templates" can be used in D in an 
equivalent manner that they are used in C++, and can cover much of the 
functionality used in AST macros.

Eric Anderton who wrote a regex expression template module a few years back:

http://www.dsource.org/projects/ddl/browser/trunk/meta/regex.d
Nov 12 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2013 1:55 AM, Walter Bright wrote:
 I forgot to mention that "expression templates" can be used in D in an
 equivalent manner that they are used in C++, and can cover much of the
 functionality used in AST macros.

 Eric Anderton who wrote a regex expression template module a few years back:

 http://www.dsource.org/projects/ddl/browser/trunk/meta/regex.d
Here's a project to add Linq support to C++: http://www.codeproject.com/Articles/17844/CLinq-LINQ-support-for-the-C-CLI-language There's also: http://pfultz2.github.io/Linq/ https://github.com/hjiang/linqxx/wiki
Nov 12 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-12 11:03, Walter Bright wrote:

 Here's a project to add Linq support to C++:

 http://www.codeproject.com/Articles/17844/CLinq-LINQ-support-for-the-C-CLI-language


 There's also:

 http://pfultz2.github.io/Linq/

 https://github.com/hjiang/linqxx/wiki
Again, operator overloading in D is too limiting to implement something Linq like. -- /Jacob Carlborg
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 12:03 AM, Jacob Carlborg wrote:
 Again, operator overloading in D is too limiting to implement something Linq
like.
Ok, let's set aside the opEquals and opCmp issue for the moment. Can AST macros do anything that expression templates cannot?
Nov 13 2013
next sibling parent reply "Don" <x nospam.com> writes:
On Wednesday, 13 November 2013 at 08:45:13 UTC, Walter Bright 
wrote:
 On 11/13/2013 12:03 AM, Jacob Carlborg wrote:
 Again, operator overloading in D is too limiting to implement 
 something Linq like.
Ok, let's set aside the opEquals and opCmp issue for the moment. Can AST macros do anything that expression templates cannot?
With an expression template you still can't create a statement. It has to revolve to a declaration. But as I said before, it's primarily syntax sugar. Expression templates are just a mass of boilerplate code. I came to the conclusion that the code for expression templates was no less ugly than for string mixins. What is true, though, is everything an AST macro can do, can already be done with a string mixin. The syntax is just ugly. calling syntax power implementation string mixin ugly high ugly expr template ok low ugly AST macro good ? ?
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 1:08 AM, Don wrote:
 On Wednesday, 13 November 2013 at 08:45:13 UTC, Walter Bright wrote:
 On 11/13/2013 12:03 AM, Jacob Carlborg wrote:
 Again, operator overloading in D is too limiting to implement something Linq
 like.
Ok, let's set aside the opEquals and opCmp issue for the moment. Can AST macros do anything that expression templates cannot?
With an expression template you still can't create a statement. It has to revolve to a declaration.
Actually, there is a way to do this. I wrote an article years back on how to write "statement" workalikes using lazy function parameters. I suppose I should dig that up if I can find it.
 But as I said before, it's primarily syntax sugar. Expression templates are
just
 a mass of boilerplate code. I came to the conclusion that the code for
 expression templates was no less ugly than for string mixins.

 What is true, though, is everything an AST macro can do, can already be done
 with a string mixin. The syntax is just ugly.

                   calling syntax power  implementation
 string mixin       ugly         high      ugly
 expr template       ok          low       ugly
 AST macro          good          ?         ?
This implies to me that we ought to explore the limits of expression templates, which already exist, in preference to creating a wholesale new feature.
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 11:24 AM, Walter Bright wrote:
 Actually, there is a way to do this. I wrote an article years back on how to
 write "statement" workalikes using lazy function parameters. I suppose I should
 dig that up if I can find it.
Ah, found the code: void ifthen(bool cond, lazy void dg) { if (cond) dg(); } void ifthen(bool cond, lazy void dgthen, lazy void dgelse) { if (cond) dgthen(); else dgelse(); } void dotimes(int i, lazy int dg) { for (int j = 0; j < i; j++) dg(); } void switcher(bool delegate()[] cases...) { foreach (c; cases) { if (c()) break; } } bool scase(bool b, lazy void dg) { if (b) { dg(); return true; } return false; } bool sdefault(lazy void dg) { dg(); return true; } void whiler(lazy bool cond, lazy void bdy) { while (cond()) bdy(); } void test1() { int x = 3; dotimes(5, printf("%d\n", ++x)); ifthen(true, printf("yes\n")); ifthen(false, printf("no\n")); ifthen(true, printf("yes\n"), printf("no\n")); ifthen(false, printf("yes\n"), printf("no\n")); int v = 2; switcher( scase(v == 1, printf("it is 1\n")), scase(v == 2, printf("it is 2\n")), scase(v == 3, printf("it is 3\n")), sdefault( printf("it is default\n")) ); whiler( x < 100, (printf("%d\n", x), x *= 2) ); }
Nov 13 2013
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/13/2013 08:25 PM, Walter Bright wrote:

 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
      if (cond)
          dg();
 }
int foo(int x){ ifthen(!x, return 2); // uh oh return 3; }
Nov 13 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 12:50 PM, Timon Gehr wrote:
 On 11/13/2013 08:25 PM, Walter Bright wrote:

 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
      if (cond)
          dg();
 }
int foo(int x){ ifthen(!x, return 2); // uh oh return 3; }
This approach definitely hews to the single-entry/single-exit paradigm.
Nov 13 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 13 November 2013 at 19:25:18 UTC, Walter Bright 
wrote:
 On 11/13/2013 11:24 AM, Walter Bright wrote:
 Actually, there is a way to do this. I wrote an article years 
 back on how to
 write "statement" workalikes using lazy function parameters. I 
 suppose I should
 dig that up if I can find it.
Ah, found the code: void ifthen(bool cond, lazy void dg) { if (cond) dg(); } void ifthen(bool cond, lazy void dgthen, lazy void dgelse) { if (cond) dgthen(); else dgelse(); } void dotimes(int i, lazy int dg) { for (int j = 0; j < i; j++) dg(); } void switcher(bool delegate()[] cases...) { foreach (c; cases) { if (c()) break; } } bool scase(bool b, lazy void dg) { if (b) { dg(); return true; } return false; } bool sdefault(lazy void dg) { dg(); return true; } void whiler(lazy bool cond, lazy void bdy) { while (cond()) bdy(); } void test1() { int x = 3; dotimes(5, printf("%d\n", ++x)); ifthen(true, printf("yes\n")); ifthen(false, printf("no\n")); ifthen(true, printf("yes\n"), printf("no\n")); ifthen(false, printf("yes\n"), printf("no\n")); int v = 2; switcher( scase(v == 1, printf("it is 1\n")), scase(v == 2, printf("it is 2\n")), scase(v == 3, printf("it is 3\n")), sdefault( printf("it is default\n")) ); whiler( x < 100, (printf("%d\n", x), x *= 2) ); }
Was is missing is the capability to look into the lazy parameter's ast and act accordingly. Here you can wrap something around code, not generation code depending on passed code.
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 2:18 PM, deadalnix wrote:
 Was is missing is the capability to look into the lazy parameter's ast and act
 accordingly.

 Here you can wrap something around code, not generation code depending on
passed
 code.
That's correct. But you can look into the AST with expression templates. So, expression templates combined with lazy arguments and mixin templates may fill the problem space.
Nov 13 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 13 November 2013 at 22:34:02 UTC, Walter Bright 
wrote:
 On 11/13/2013 2:18 PM, deadalnix wrote:
 Was is missing is the capability to look into the lazy 
 parameter's ast and act
 accordingly.

 Here you can wrap something around code, not generation code 
 depending on passed
 code.
That's correct. But you can look into the AST with expression templates. So, expression templates combined with lazy arguments and mixin templates may fill the problem space.
Expression templates are limited to expression and require to use its own set of types. Still the idea has some merits.
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 3:58 PM, deadalnix wrote:
 On Wednesday, 13 November 2013 at 22:34:02 UTC, Walter Bright wrote:
 On 11/13/2013 2:18 PM, deadalnix wrote:
 Was is missing is the capability to look into the lazy parameter's ast and act
 accordingly.

 Here you can wrap something around code, not generation code depending on
passed
 code.
That's correct. But you can look into the AST with expression templates. So, expression templates combined with lazy arguments and mixin templates may fill the problem space.
Expression templates are limited to expression and require to use its own set of types.
Yes. But that's a good thing. I'd be pretty skeptical of the value of an AST macro that took 3+4 and changed it so it did something other than compute 7.
 Still the idea has some merits.
It has the merit of being implemented and available right now :-) I do understand that expression templates aren't very sexy, and are kinda hard to wrap one's head around. But they're not so bad from the perspective of the user of the expression templates, and I really think that the limits of existing features should be explored before inventing wholesale new ones that may turn out to be redundant.
Nov 13 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 14 November 2013 at 00:16:45 UTC, Walter Bright 
wrote:
 I do understand that expression templates aren't very sexy, and 
 are kinda hard to wrap one's head around. But they're not so 
 bad from the perspective of the user of the expression 
 templates, and I really think that the limits of existing 
 features should be explored before inventing wholesale new ones 
 that may turn out to be redundant.
My very first project in D involved quite a lot of them. So I very aware of what they are. The whole idea is to be able to extends that to statement and declarations. Making what expression templates can do is simply a side effect so to speak.
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 5:14 PM, deadalnix wrote:
 My very first project in D involved quite a lot of them. So I very aware of
what
 they are.

 The whole idea is to be able to extends that to statement and declarations.
My question about that is just what problem is this trying to solve, and why that problem cannot be reformulated in terms of expression templates? I.e. we shouldn't be adding things to D just because they are cool. They need to solve a problem that is fairly commonplace and too painful to solve with existing mechanisms. The larger the proposed language enhancement is, the more commonplace and and painful the problems must be that it purports to solve.
 Making what expression templates can do is simply a side effect so to speak.
Nov 13 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 14 November 2013 at 01:38:29 UTC, Walter Bright 
wrote:
 On 11/13/2013 5:14 PM, deadalnix wrote:
 My very first project in D involved quite a lot of them. So I 
 very aware of what
 they are.

 The whole idea is to be able to extends that to statement and 
 declarations.
My question about that is just what problem is this trying to solve, and why that problem cannot be reformulated in terms of expression templates? I.e. we shouldn't be adding things to D just because they are cool. They need to solve a problem that is fairly commonplace and too painful to solve with existing mechanisms. The larger the proposed language enhancement is, the more commonplace and and painful the problems must be that it purports to solve.
I think the whole point of macro is to NOT add too much feature to the language. See for instance the example I gave before to create generator. This can be added with extra language support (C# async/await is an example). But with macro, the feature can be implemented as a library. It is NOT achievable with expression templates. I can say with certainty that the async/await mechanism is heavily used in some codebases and with great benefit. It is being added to all languages one after another. The idea behind macros is that instead of adding new feature to the language, the language should propose a toolkit to build these features as library. I don't think the time has come for macros in D. As discussed before, it seems like filling the gap in existing feature is more important than adding new ones. This should however be considered very seriously when the gaps has been filled.
Nov 13 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-14 02:56, deadalnix wrote:

 I think the whole point of macro is to NOT add too much feature to the
 language.

 See for instance the example I gave before to create generator. This can
 be added with extra language support (C# async/await is an example). But
 with macro, the feature can be implemented as a library.

 It is NOT achievable with expression templates.

 I can say with certainty that the async/await mechanism is heavily used
 in some codebases and with great benefit. It is being added to all
 languages one after another.

 The idea behind macros is that instead of adding new feature to the
 language, the language should propose a toolkit to build these features
 as library.
Agree, good points.
 I don't think the time has come for macros in D. As discussed before, it
 seems like filling the gap in existing feature is more important than
 adding new ones. This should however be considered very seriously when
 the gaps has been filled.
Unfortunately I don't think we will have a feature freeze from now until those gaps has been filled. Most likely there will be added a couple of small features that could have been solved by AST macros. The problem with that is that each of these features, in themselves, are way to small to add AST macros for. So instead we get new features and even less chance of adding AST macros. Example. There's a bug report about adding support for C++ namespaces. That should be solveable with library code. It's just mangling and we already have pragma(mangle). pragma(mangle, namespace("foo::bar")) extern(C++) void x (); The problem here is that "x" needs to be mangled as well. pragma(mangle, namespace("foo::bar", x)) extern(C++) void x (); The above doesn't work due to forward reference errors. pragma(mangle, namespace("foo::bar", void function (), "x")) extern(C++) void x (); The problem with the above is that we now need to repeat the signature and the name of the function. With AST macros: macro namespace (Context context, Ast!(string) name, Declaration declaration) { ... } namespace("foo::bar") extern (C++) void x (); This works because the macro receives the whole declaration that is "x" and "x" is replaced with whatever the macro returns: pragma(mangle, "mangled_name_of_x") extern (C++) void x (); Since it uses the same syntax as UDA's you it can look like real namespaces in C++: namespace("foo::bar") extern (C++) { void x (); void y (); } -- /Jacob Carlborg
Nov 13 2013
parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 14.11.2013 08:50, schrieb Jacob Carlborg:
 On 2013-11-14 02:56, deadalnix wrote:

 I think the whole point of macro is to NOT add too much feature to the
 language.

 See for instance the example I gave before to create generator. This can
 be added with extra language support (C# async/await is an example). But
 with macro, the feature can be implemented as a library.

 It is NOT achievable with expression templates.

 I can say with certainty that the async/await mechanism is heavily used
 in some codebases and with great benefit. It is being added to all
 languages one after another.

 The idea behind macros is that instead of adding new feature to the
 language, the language should propose a toolkit to build these features
 as library.
Agree, good points.
 I don't think the time has come for macros in D. As discussed before, it
 seems like filling the gap in existing feature is more important than
 adding new ones. This should however be considered very seriously when
 the gaps has been filled.
Unfortunately I don't think we will have a feature freeze from now until those gaps has been filled. Most likely there will be added a couple of small features that could have been solved by AST macros. The problem with that is that each of these features, in themselves, are way to small to add AST macros for. So instead we get new features and even less chance of adding AST macros. Example. There's a bug report about adding support for C++ namespaces. That should be solveable with library code. It's just mangling and we already have pragma(mangle). pragma(mangle, namespace("foo::bar")) extern(C++) void x (); The problem here is that "x" needs to be mangled as well. pragma(mangle, namespace("foo::bar", x)) extern(C++) void x (); The above doesn't work due to forward reference errors. pragma(mangle, namespace("foo::bar", void function (), "x")) extern(C++) void x (); The problem with the above is that we now need to repeat the signature and the name of the function. With AST macros: macro namespace (Context context, Ast!(string) name, Declaration declaration) { ... } namespace("foo::bar") extern (C++) void x (); This works because the macro receives the whole declaration that is "x" and "x" is replaced with whatever the macro returns: pragma(mangle, "mangled_name_of_x") extern (C++) void x (); Since it uses the same syntax as UDA's you it can look like real namespaces in C++: namespace("foo::bar") extern (C++) { void x (); void y (); }
perfect for the DIP example section - more of these please :)
Nov 13 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-14 08:54, dennis luehring wrote:

 perfect for the DIP example section - more of these please :)
Done: http://wiki.dlang.org/DIP50#C.2B.2B_Namespaces_.28issue_7961.29 -- /Jacob Carlborg
Nov 14 2013
parent dennis luehring <dl.soluz gmx.net> writes:
Am 14.11.2013 09:40, schrieb Jacob Carlborg:
 On 2013-11-14 08:54, dennis luehring wrote:

 perfect for the DIP example section - more of these please :)
Done: http://wiki.dlang.org/DIP50#C.2B.2B_Namespaces_.28issue_7961.29
it would be also nice to have (always) an (if possible) string mixin and/or expression-template variant of the example around to show the dirtyness, give can't-work information very fast
Nov 14 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 5:56 PM, deadalnix wrote:
 I think the whole point of macro is to NOT add too much feature to the
language.

 See for instance the example I gave before to create generator. This can be
 added with extra language support (C# async/await is an example). But with
 macro, the feature can be implemented as a library.

 It is NOT achievable with expression templates.

 I can say with certainty that the async/await mechanism is heavily used in some
 codebases and with great benefit. It is being added to all languages one after
 another.
I agree that async/await has to eventually be added to D. I'm not convinced it can or should be done with AST macros.
 The idea behind macros is that instead of adding new feature to the language,
 the language should propose a toolkit to build these features as library.
I do understand this. I've not only extensively used macro systems, I've designed two successful ones and implemented other peoples' designs. I've also seen the sheer awfulness of what happens with these systems over the long term. The fascinating thing about this awfulness is people are well aware of it in everyone's macro libraries but their own.
 I don't think the time has come for macros in D. As discussed before, it seems
 like filling the gap in existing feature is more important than adding new
ones.
 This should however be considered very seriously when the gaps has been filled.
Nov 14 2013
next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 14.11.2013 10:12, schrieb Walter Bright:
 I've also seen the sheer awfulness of what happens with these systems over the
 long term. The fascinating thing about this awfulness is people are well aware
 of it in everyone's macro libraries but their own.
the same was said about templates AND string mixins before - but currently only the wise guys building good/great templates/mixins which are in use by everyone - no terror from the deep
 I don't think the time has come for macros in D.
the DIP needs to grow by many examples and scenarios - some sort of long term dicussion DIP
Nov 14 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2013 1:25 AM, dennis luehring wrote:
 Am 14.11.2013 10:12, schrieb Walter Bright:
 I've also seen the sheer awfulness of what happens with these systems over the
 long term. The fascinating thing about this awfulness is people are well aware
 of it in everyone's macro libraries but their own.
the same was said about templates AND string mixins before - but currently only the wise guys building good/great templates/mixins which are in use by everyone - no terror from the deep
I don't buy that these are analogous. They are very different features.
Nov 14 2013
prev sibling next sibling parent reply "Chris Cain" <clcain uncg.edu> writes:
On Thursday, 14 November 2013 at 09:12:38 UTC, Walter Bright 
wrote:
 I agree that async/await has to eventually be added to D. I'm 
 not convinced it can or should be done with AST macros.
Out of curiousity, how would that be implemented without macros? I could see it being done with macros with a rewrite and using some of the standard library (like std.parallelism), maybe. But with a language feature, you couldn't use the standard library. It seems like some parallelism code would have to leak into the runtime to get it to work as people might expect. I guess I would have thought the exact opposite from you... that this is precisely something that *should* be done with a library-defined macro.
Nov 14 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 14 November 2013 at 10:07:31 UTC, Chris Cain wrote:
 On Thursday, 14 November 2013 at 09:12:38 UTC, Walter Bright 
 wrote:
 I agree that async/await has to eventually be added to D. I'm 
 not convinced it can or should be done with AST macros.
Out of curiousity, how would that be implemented without macros? I could see it being done with macros with a rewrite and using some of the standard library (like std.parallelism), maybe. But with a language feature, you couldn't use the standard library. It seems like some parallelism code would have to leak into the runtime to get it to work as people might expect. I guess I would have thought the exact opposite from you... that this is precisely something that *should* be done with a library-defined macro.
async await do not create parallelism.
Nov 14 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2013 2:07 AM, Chris Cain wrote:
 I guess I would have thought the exact opposite from you... that this is
 precisely something that *should* be done with a library-defined macro.
Again, I reiterate what experience shows happens with macro systems in the long term.
Nov 14 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/14/13 2:17 AM, Walter Bright wrote:
 On 11/14/2013 2:07 AM, Chris Cain wrote:
 I guess I would have thought the exact opposite from you... that this is
 precisely something that *should* be done with a library-defined macro.
Again, I reiterate what experience shows happens with macro systems in the long term.
FWIW Walter talked me back around 2005-2006 into abandoning my own ideas about languages with configurable syntax. There are quite a few failings about comparing programming languages against natural languages, but here's one that I think does have value: fixed syntax is ingrained into people's notion of language, and swapping syntax within an otherwise identical linguistic context is extremely taxing on the brain. Anyone who's read with a C or C++ codebase full of macros and #if-driven code can attest how unbelievably difficult juxtapositions of the normal syntax with preprocessor syntax can quickly become. That's part of why "static if" has been so successful in D - it drives new semantics but within the same syntax. Natural languages are "humans complete" because they are the one vehicle we use to describe and manipulate our understanding of the entire reality. If configurable syntax was something necessary to model the world, it would have inevitably occurred in natural languages one way or another. Instead, all human languages (with no exception I know of) have fixed syntax and prefer to add modeling power in other ways. There must be something about this. Andrei
Nov 14 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-14 19:23, Andrei Alexandrescu wrote:

 FWIW Walter talked me back around 2005-2006 into abandoning my own ideas
 about languages with configurable syntax.

 There are quite a few failings about comparing programming languages
 against natural languages, but here's one that I think does have value:
 fixed syntax is ingrained into people's notion of language, and swapping
 syntax within an otherwise identical linguistic context is extremely
 taxing on the brain. Anyone who's read with a C or C++ codebase full of
 macros and #if-driven code can attest how unbelievably difficult
 juxtapositions of the normal syntax with preprocessor syntax can quickly
 become. That's part of why "static if" has been so successful in D - it
 drives new semantics but within the same syntax.
AST macros cannot change the syntax or introduce new syntax. C preprocessor macros and AST macros are not the same. I'm starting to regret that I called it "macros". -- /Jacob Carlborg
Nov 14 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2013 11:18 AM, Jacob Carlborg wrote:
 AST macros cannot change the syntax or introduce new syntax. C preprocessor
 macros and AST macros are not the same. I'm starting to regret that I called it
 "macros".
If it is powerful enough to do async/await but look like normal D syntax, then is going to suffer from these faults. (A lot of the C++ macro abuse looks like normal syntax to the user, but it behaves in a way that the casual user would find completely unexpected. In the same vein, macro assembler abuse of macros did not introduce new syntax - the syntax for the macros was straightforward. It was what those macros *did* that made them incomprehensible.)
Nov 14 2013
next sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 14.11.2013 21:54, schrieb Walter Bright:
 (A lot of the C++ macro abuse looks like normal syntax to the user, but it
 behaves in a way that the casual user would find completely unexpected.
same can be said about Boost Spirit templates
Nov 14 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-14 21:54, Walter Bright wrote:

 If it is powerful enough to do async/await but look like normal D
 syntax, then is going to suffer from these faults.
I don't think so. The idea is to have it look consistent within the language. Take a look at __traits. It looks just like a function call, but it's far from being a function call. Classes and structs look very similar, only the keyword is different, but they are very different. -- /Jacob Carlborg
Nov 14 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2013 11:38 PM, Jacob Carlborg wrote:
 On 2013-11-14 21:54, Walter Bright wrote:

 If it is powerful enough to do async/await but look like normal D
 syntax, then is going to suffer from these faults.
I don't think so. The idea is to have it look consistent within the language. Take a look at __traits. It looks just like a function call, but it's far from being a function call. Classes and structs look very similar, only the keyword is different, but they are very different.
Think about the desired feature mentioned earlier in this thread of being able to insert a "return;" statement, causing what looks like a function call to return from the caller. That's a major step increase in obfuscation, not just a detail.
Nov 17 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-17 09:58, Walter Bright wrote:

 Think about the desired feature mentioned earlier in this thread of
 being able to insert a "return;" statement, causing what looks like a
 function call to return from the caller.
I'm not entirely sure what you're saying. Could you please post a link to the post mentioning this or show an example.
 That's a major step increase in obfuscation, not just a detail.
-- /Jacob Carlborg
Nov 17 2013
parent reply =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On 2013-11-17 11:37, Jacob Carlborg wrote:
 On 2013-11-17 09:58, Walter Bright wrote:

 Think about the desired feature mentioned earlier in this thread of
 being able to insert a "return;" statement, causing what looks like a
 function call to return from the caller.
I'm not entirely sure what you're saying. Could you please post a link to the post mentioning this or show an example.
I believe it is basically this: int bar() { foo(return 3); return 5; } If that program is allowed to compile, and to return 3.
Nov 17 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 17 November 2013 at 13:36:32 UTC, Simen Kjærås wrote:
 On 2013-11-17 11:37, Jacob Carlborg wrote:
 On 2013-11-17 09:58, Walter Bright wrote:

 Think about the desired feature mentioned earlier in this 
 thread of
 being able to insert a "return;" statement, causing what 
 looks like a
 function call to return from the caller.
I'm not entirely sure what you're saying. Could you please post a link to the post mentioning this or show an example.
I believe it is basically this: int bar() { foo(return 3); return 5; } If that program is allowed to compile, and to return 3.
That never was in the DIP.
Nov 17 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/17/2013 10:47 AM, deadalnix wrote:
 On Sunday, 17 November 2013 at 13:36:32 UTC, Simen Kjærås wrote:
 On 2013-11-17 11:37, Jacob Carlborg wrote:
 On 2013-11-17 09:58, Walter Bright wrote:

 Think about the desired feature mentioned earlier in this thread of
 being able to insert a "return;" statement, causing what looks like a
 function call to return from the caller.
I'm not entirely sure what you're saying. Could you please post a link to the post mentioning this or show an example.
I believe it is basically this: int bar() { foo(return 3); return 5; } If that program is allowed to compile, and to return 3.
That never was in the DIP.
It was this by Timon Gehr essentially posting why lazy parameters weren't good enough:
 On 11/13/2013 08:25 PM, Walter Bright wrote:

 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
      if (cond)
          dg();
 }
int foo(int x){ ifthen(!x, return 2); // uh oh return 3; }
Nov 17 2013
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/17/2013 09:48 PM, Walter Bright wrote:
 It was this by Timon Gehr essentially posting why lazy parameters
 weren't good enough:

 On 11/13/2013 08:25 PM, Walter Bright wrote:

 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
      if (cond)
          dg();
 }
int foo(int x){ ifthen(!x, return 2); // uh oh return 3; }
This code was just supposed to demonstrate that ifthen is not actually a "statement" work-alike as claimed in a previous post.
Nov 17 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/17/2013 1:58 PM, Timon Gehr wrote:
 This code was just supposed to demonstrate that ifthen is not actually a
 "statement" work-alike as claimed in a previous post.
Ok, my apologies.
Nov 17 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 17 November 2013 at 20:48:15 UTC, Walter Bright wrote:
 That never was in the DIP.
It was this by Timon Gehr essentially posting why lazy parameters weren't good enough:
 On 11/13/2013 08:25 PM, Walter Bright wrote:

 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
     if (cond)
         dg();
 }
int foo(int x){ ifthen(!x, return 2); // uh oh return 3; }
As I understand it, Timon choosed that syntax simply to demonstrate the limitation of your proposal using a similar syntax. Not to propose a syntax.
Nov 17 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/17/2013 7:14 PM, deadalnix wrote:
 As I understand it, Timon choosed that syntax simply to demonstrate the
 limitation of your proposal using a similar syntax. Not to propose a syntax.
Ok, then I'm not seeing what AST macros do that lazy parameters / template overloading / mixin templates do not?
Nov 17 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 18 November 2013 at 05:05:02 UTC, Walter Bright wrote:
 On 11/17/2013 7:14 PM, deadalnix wrote:
 As I understand it, Timon choosed that syntax simply to 
 demonstrate the
 limitation of your proposal using a similar syntax. Not to 
 propose a syntax.
Ok, then I'm not seeing what AST macros do that lazy parameters / template overloading / mixin templates do not?
2 things. First, they can act on statement or declaration. Simply not with the proposed syntax. Second they can reflect what is passed as argument and act accordingly, when the lazy expression solution cannot.
Nov 17 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/17/2013 9:10 PM, deadalnix wrote:
 On Monday, 18 November 2013 at 05:05:02 UTC, Walter Bright wrote:
 On 11/17/2013 7:14 PM, deadalnix wrote:
 As I understand it, Timon choosed that syntax simply to demonstrate the
 limitation of your proposal using a similar syntax. Not to propose a syntax.
Ok, then I'm not seeing what AST macros do that lazy parameters / template overloading / mixin templates do not?
2 things. First, they can act on statement or declaration. Simply not with the proposed syntax.
If they can insert a statement or a declaration uplevel, then they are doing what I suggested with the return statement.
 Second they can reflect what is passed as argument and act
 accordingly, when the lazy expression solution cannot.
Expression templates can.
Nov 18 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 18 November 2013 at 09:25:22 UTC, Walter Bright wrote:
 On 11/17/2013 9:10 PM, deadalnix wrote:
 On Monday, 18 November 2013 at 05:05:02 UTC, Walter Bright 
 wrote:
 On 11/17/2013 7:14 PM, deadalnix wrote:
 As I understand it, Timon choosed that syntax simply to 
 demonstrate the
 limitation of your proposal using a similar syntax. Not to 
 propose a syntax.
Ok, then I'm not seeing what AST macros do that lazy parameters / template overloading / mixin templates do not?
2 things. First, they can act on statement or declaration. Simply not with the proposed syntax.
If they can insert a statement or a declaration uplevel, then they are doing what I suggested with the return statement.
 Second they can reflect what is passed as argument and act
 accordingly, when the lazy expression solution cannot.
Expression templates can.
*Expression* templates.
Nov 18 2013
prev sibling parent luka8088 <luka8088 owave.net> writes:
On 18.11.2013. 6:05, Walter Bright wrote:
 On 11/17/2013 7:14 PM, deadalnix wrote:
 
 Ok, then I'm not seeing what AST macros do that lazy parameters /
 template overloading / mixin templates do not?
Well, this is just a small piece of the puzzle but I would like to be able to write (as syntax sugar): query q{ select data; } a compiler to rewrite it to: mixin(query!(q{ select data; })); if the following exists: mixin statement query (string code) { // ... } Maybe I am the only one but I found that string mixins + code generation are used this way even i Phobos and I find the first syntax much more cleaner. Or this is not in the general interest?
Nov 19 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-17 14:36, Simen Kjærås wrote:

 I believe it is basically this:

 int bar() {
      foo(return 3);
      return 5;
 }


 If that program is allowed to compile, and to return 3.
I see, I guess that should work with with the DIP. -- /Jacob Carlborg
Nov 18 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
 Think about the desired feature mentioned earlier in this 
 thread of being able to insert a "return;" statement, causing 
 what looks like a function call to return from the caller.

 That's a major step increase in obfuscation, not just a detail.
You do realize that, as per more strict proposal, it won't change a _single_ thing about semantics of actual D function but allows to generate another one with semantics defined by macro? It has _zero_ added ofbuscation over existing template + traits combo. You have not shown a single example which proves otherwise and keep resorting to personal experience argument instead of doing actual comparison. You have full right to do so but it does not sound like a decision made on behalf of careful reasoning.
Nov 17 2013
prev sibling parent "Rob T" <alanb ucora.com> writes:
Walter,

what do you think about allowing mixins to work with parameter 
list construction?

Currently you cannot generate code unless it is syntactically 
complete on its own, this disallows parameter list construction 
even for syntactically correct parameter lists.

The other failure point of mixins is when stringing together 
function calls with the dot, eg A.mixin(MYFUNC).foo;

A whole lot more could be done with mixins if restrictions were 
relaxed just a bit more in a few key areas. I don't see too much 
obfuscation happening so long as the allowable constructs are 
carefully limited and cannot be manipulated in ways that factor 
into the concerns you mentioned previously.

To make it clear we could have a special parammixin() and 
dotmixin() or whatever to ensure there's no possibility of 
confusion.

--rt
Nov 17 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 14 November 2013 at 18:23:20 UTC, Andrei 
Alexandrescu wrote:
 On 11/14/13 2:17 AM, Walter Bright wrote:
 On 11/14/2013 2:07 AM, Chris Cain wrote:
 I guess I would have thought the exact opposite from you... 
 that this is
 precisely something that *should* be done with a 
 library-defined macro.
Again, I reiterate what experience shows happens with macro systems in the long term.
FWIW Walter talked me back around 2005-2006 into abandoning my own ideas about languages with configurable syntax.
This is not proposing to configure syntax.
 There are quite a few failings about comparing programming 
 languages against natural languages, but here's one that I 
 think does have value: fixed syntax is ingrained into people's 
 notion of language, and swapping syntax within an otherwise 
 identical linguistic context is extremely taxing on the brain. 
 Anyone who's read with a C or C++ codebase full of macros and 
 #if-driven code can attest how unbelievably difficult 
 juxtapositions of the normal syntax with preprocessor syntax 
 can quickly become. That's part of why "static if" has been so 
 successful in D - it drives new semantics but within the same 
 syntax.
C macro can introduce new syntax. This cannot.
 Natural languages are "humans complete" because they are the 
 one vehicle we use to describe and manipulate our understanding 
 of the entire reality. If configurable syntax was something 
 necessary to model the world, it would have inevitably occurred 
 in natural languages one way or another. Instead, all human 
 languages (with no exception I know of) have fixed syntax and 
 prefer to add modeling power in other ways. There must be 
 something about this.


 Andrei
I think this very post pretty much show that you missed the point of the DIP.
Nov 14 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2013 11:40 AM, deadalnix wrote:
 This is not proposing to configure syntax.
Right, it is about inserting arbitrarily different meaning into existing syntax. For example, I have a friend from the middle east who worked at Microsoft. His english was excellent. But sometimes he'd get tripped up - for example, he'd take his idea to his boss, who'd discuss it, and then say: "fine" He took that to mean "my boss approves of my great idea!" Unfortunately, "fine" has undergone a meaning reversal colloquially, and actually means "your idea sux". This wound up getting him into trouble. There were a few other phrases like this with unintentionally hilarious misunderstandings. (Also, a popular way of encrypting communications is to have private meanings for phrases like "my cat ate my homework" meaning "initiate Operation X".) Now, let's take the AST macros, and pass it "4+3". Apparently is a feature that the macro can transform this to produce "hello" as the result. The result is that although the language looks the same, there is no "anchor" a person can rely on to indicate the meaning. The meaning of "fine" is reversed, so to speak. The riposte is, of course, every feature can be abused and good programmers won't do that. Unfortunately, it's IN THE VERY NATURE of macros to be abusive, and even the very best programmers can and will take it to the limit. They will see and decry the abuse in everyone else's code but their own, which they will justify. As I said, I see this over and over again. On the other hand, with string mixins, it is immediately obvious that one is dealing with different syntax, and so expect something different to happen.
Nov 14 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Nov 14, 2013 at 01:12:11PM -0800, Walter Bright wrote:
 On 11/14/2013 11:40 AM, deadalnix wrote:
This is not proposing to configure syntax.
Right, it is about inserting arbitrarily different meaning into existing syntax.
[...]
 Now, let's take the AST macros, and pass it "4+3". Apparently is a
 feature that the macro can transform this to produce "hello" as the
 result. The result is that although the language looks the same,
 there is no "anchor" a person can rely on to indicate the meaning.
 The meaning of "fine" is reversed, so to speak.
[...]
 On the other hand, with string mixins, it is immediately obvious
 that one is dealing with different syntax, and so expect something
 different to happen.
What if macros adopted an overtly different syntax? T -- English has the lovely word "defenestrate", meaning "to execute by throwing someone out a window", or more recently "to remove Windows from a computer and replace it with something useful". :-) -- John Cowan
Nov 14 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2013 1:34 PM, H. S. Teoh wrote:
 What if macros adopted an overtly different syntax?
That would be better, but I don't think enough better.
Nov 17 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-17 10:00, Walter Bright wrote:

 That would be better, but I don't think enough better.
Then why to we have UDA's with the same syntax as language attributes and why do we have operator overloading. You can hide arbitrary code behind operator overloading. -- /Jacob Carlborg
Nov 17 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/17/2013 2:38 AM, Jacob Carlborg wrote:
 On 2013-11-17 10:00, Walter Bright wrote:

 That would be better, but I don't think enough better.
Then why to we have UDA's with the same syntax as language attributes and why do we have operator overloading. You can hide arbitrary code behind operator overloading.
1. I don't believe we can decide on language features by analogy. D is complex enough that one can use analogy to justify anything. 2. You cannot do anything behind a function call - the 'return' discussed earlier, and async/await for another, i.e. operator overloading cannot introduce control flow, cannot introduce variables into the current scope, etc.
Nov 17 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-17 21:52, Walter Bright wrote:

 1. I don't believe we can decide on language features by analogy. D is
 complex enough that one can use analogy to justify anything.

 2. You cannot do anything behind a function call - the 'return'
 discussed earlier, and async/await for another, i.e. operator
 overloading cannot introduce control flow, cannot introduce variables
 into the current scope, etc.
You can turn this: BigInt(3) + 7; Into something that is not 10. You cannot do anything behind a macro. You cannot introduce new syntax. -- /Jacob Carlborg
Nov 18 2013
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/18/2013 09:22 AM, Jacob Carlborg wrote:
 On 2013-11-17 21:52, Walter Bright wrote:

 1. I don't believe we can decide on language features by analogy. D is
 complex enough that one can use analogy to justify anything.

 2. You cannot do anything behind a function call - the 'return'
 discussed earlier, and async/await for another, i.e. operator
 overloading cannot introduce control flow, cannot introduce variables
 into the current scope, etc.
You can turn this: BigInt(3) + 7; Into something that is not 10. You cannot do anything behind a macro. You cannot introduce new syntax.
Well, you could do: screwUp int foo(){ return 3+7; } assert(foo()!=10);
Nov 18 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-18 10:12, Timon Gehr wrote:

 Well, you could do:

  screwUp int foo(){ return 3+7; }

 assert(foo()!=10);
Sure, what I'm saying is that you can do stupid things with other features than macros. -- /Jacob Carlborg
Nov 18 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/18/2013 1:14 AM, Jacob Carlborg wrote:
 Sure, what I'm saying is that you can do stupid things with other features than
 macros.
Yes, you can write terrible code with all features. My point is not that you can do stupid things with macros. It is that macros encourage this to a high degree, and in fact the flexibility that macros have over lazy parameters / expression templates / mixin templates are exactly those features that are abused to the point it is hard to see much good come from them. I am not making this up, this is what happens with macro systems again and again. I know this is a difficult position to argue from and convince people about. D has a deliberate lack of capability in some areas because long experience shows that little good comes from them. Examples include lack of ability to overload && and ||, and lack of expression support for version statements. This is not because these features are hard - they're easy to implement. I've seen attempts to replace version statements with static if in order to get back the ugliness, and the result is ugly and unnecessary and damned hard to figure out what was actually intended and why it didn't work. I fear that implementing these things with the idea that they'll somehow be used with restraint is a pipe dream and that D will degenerate into an ugly snarl that everyone hates, and then some D-Go will be invented that everyone will cheer because it won't allow such code to be written. Case in point - again and again I see utterly wretched use of macros in C++ from the top level of C++ programmers who really ought to know better. Want to see a lovely example? Take a look at the C header files for openssl: https://github.com/D-Programming-Deimos/openssl/tree/master/C This is not unusual. I try to put my money where my mouth is. If you look at older parts of dmd, you'll see macro abuse that I cleverly wrote. The newer bits contain fewer and fewer use of any macros, and I've made an ongoing effort to try and eliminate all of it (yebblies has campaigned to get rid of it, too, even crap I tried to hang on to). This has been done this without compromise on performance and usability and readability of the code - and I hope you'll agree that it looks nice without the usual rat's nest of preprocessor abuse one sees in typical C++ code.
Nov 18 2013
next sibling parent reply "IgorStepanov" <wazar mail.ru> writes:
On Monday, 18 November 2013 at 09:55:42 UTC, Walter Bright wrote:
 On 11/18/2013 1:14 AM, Jacob Carlborg wrote:
 Sure, what I'm saying is that you can do stupid things with 
 other features than
 macros.
Yes, you can write terrible code with all features. My point is not that you can do stupid things with macros. It is that macros encourage this to a high degree, and in fact the flexibility that macros have over lazy parameters / expression templates / mixin templates are exactly those features that are abused to the point it is hard to see much good come from them. I am not making this up, this is what happens with macro systems again and again. I know this is a difficult position to argue from and convince people about. D has a deliberate lack of capability in some areas because long experience shows that little good comes from them. Examples include lack of ability to overload && and ||, and lack of expression support for version statements. This is not because these features are hard - they're easy to implement. I've seen attempts to replace version statements with static if in order to get back the ugliness, and the result is ugly and unnecessary and damned hard to figure out what was actually intended and why it didn't work. I fear that implementing these things with the idea that they'll somehow be used with restraint is a pipe dream and that D will degenerate into an ugly snarl that everyone hates, and then some D-Go will be invented that everyone will cheer because it won't allow such code to be written. Case in point - again and again I see utterly wretched use of macros in C++ from the top level of C++ programmers who really ought to know better. Want to see a lovely example? Take a look at the C header files for openssl: https://github.com/D-Programming-Deimos/openssl/tree/master/C This is not unusual. I try to put my money where my mouth is. If you look at older parts of dmd, you'll see macro abuse that I cleverly wrote. The newer bits contain fewer and fewer use of any macros, and I've made an ongoing effort to try and eliminate all of it (yebblies has campaigned to get rid of it, too, even crap I tried to hang on to). This has been done this without compromise on performance and usability and readability of the code - and I hope you'll agree that it looks nice without the usual rat's nest of preprocessor abuse one sees in typical C++ code.
How macros implemented in C#? I know, that C# have linq which can translate lambda expression to SQL WHERE condition. For example s => s.Number > 2 && s.Number < 20 can be translated to ... WHERE s.Number > 2 AND s.Number < 20 It is interesting function, because it allow to write hard code in preferred language (C#, D) instead of unpreferred (SQL). Also this code hides poorly standardized SQL for the well-standardized language. And С# still not sinking under tons of macro. May be C# has some limitations for ast macros. May be they allowed only for lambda? This is a general macro-feature which I had like to see in D.
Nov 18 2013
parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Monday, 18 November 2013 at 16:03:54 UTC, IgorStepanov wrote:
 How macros implemented in C#?
 I know, that C# have linq which can translate lambda expression 
 to SQL WHERE condition. For example s => s.Number > 2 && 
 s.Number < 20 can be translated to ... WHERE s.Number > 2 AND 
 s.Number < 20

 It is interesting function, because it allow to write hard code 
 in preferred language (C#, D) instead of unpreferred (SQL). 
 Also this code hides poorly standardized SQL for the 
 well-standardized language.

 And С# still not sinking under tons of macro. May be C# has 
 some limitations for ast macros. May be they allowed only for 
 lambda?

 This is a general macro-feature which I had like to see in D.
C# doesn't have macros. Linq to Sql is implemented using expression trees. Essentially, it allows the compiler to generate an expression tree of the lambda expression which your library parses to do some work. It doesn't inject any code, and it's done at runtime. I could be wrong about the exact details, but essentially it comes down to the compiler passing in an expression tree into your method, and then your method parsing it at runtime and doing something with the results; it can't be used to generate new code.
Nov 18 2013
next sibling parent reply "IgorStepanov" <wazar mail.ru> writes:
On Monday, 18 November 2013 at 19:21:35 UTC, Kapps wrote:
 On Monday, 18 November 2013 at 16:03:54 UTC, IgorStepanov wrote:
 How macros implemented in C#?
 I know, that C# have linq which can translate lambda 
 expression to SQL WHERE condition. For example s => s.Number > 
 2 && s.Number < 20 can be translated to ... WHERE s.Number > 2 
 AND s.Number < 20

 It is interesting function, because it allow to write hard 
 code in preferred language (C#, D) instead of unpreferred 
 (SQL). Also this code hides poorly standardized SQL for the 
 well-standardized language.

 And С# still not sinking under tons of macro. May be C# has 
 some limitations for ast macros. May be they allowed only for 
 lambda?

 This is a general macro-feature which I had like to see in D.
C# doesn't have macros. Linq to Sql is implemented using expression trees. Essentially, it allows the compiler to generate an expression tree of the lambda expression which your library parses to do some work. It doesn't inject any code, and it's done at runtime. I could be wrong about the exact details, but essentially it comes down to the compiler passing in an expression tree into your method, and then your method parsing it at runtime and doing something with the results; it can't be used to generate new code.
May be this way is good for D? And CTFE allow to parse this tree at compile-time. Something like: property Collection where(alias cond, Collection)(Collection c) { enum ast = cond.astof; static if(is(Collection : DBResultSet)) { string where_cond = generateCondition!(ast, Collection); return c.doWhere(where_cond); } else { ... } } ... auto result = db.myItems.where!(x => x.Count > 10);
Nov 18 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-18 22:08, IgorStepanov wrote:

 May be this way is good for D? And CTFE allow to parse this tree
 at compile-time.
 Something like:

  property Collection where(alias cond, Collection)(Collection c)
 {
       enum ast = cond.astof;
       static if(is(Collection : DBResultSet))
       {
          string where_cond = generateCondition!(ast, Collection);
          return c.doWhere(where_cond);
       }
       else
       {
          ...
       }
 }

 ...

 auto result = db.myItems.where!(x => x.Count > 10);
The idea is to have something more general. Linq is just one example of what can be implemented. -- /Jacob Carlborg
Nov 18 2013
prev sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 11/18/2013 11:21 AM, Kapps wrote:
 On Monday, 18 November 2013 at 16:03:54 UTC, IgorStepanov wrote:
 it can't be
 used to generate new code.
Sure it can. Generate lambda expression tree; invoke Compile(). You just can't rewrite existing code.
Nov 19 2013
prev sibling parent "Doodoo" <barf barf.com> writes:
 Case in point - again and again I see utterly wretched use of 
 macros in C++ from the top level of C++ programmers who really 
 ought to know better. Want to see a lovely example? Take a look 
 at the C header files for openssl:

 https://github.com/D-Programming-Deimos/openssl/tree/master/C

 This is not unusual.

 I try to put my money where my mouth is. If you look at older 
 parts of dmd, you'll see macro abuse that I cleverly wrote. The 
 newer bits contain fewer and fewer use of any macros, and I've 
 made an ongoing effort to try and eliminate all of it (yebblies 
 has campaigned to get rid of it, too, even crap I tried to hang 
 on to). This has been done this without compromise on 
 performance and usability and readability of the code - and I 
 hope you'll agree that it looks nice without the usual rat's 
 nest of preprocessor abuse one sees in typical C++ code.
Now, now Walter, that is C not C++. In C++ we would use Boost Preprocessor:) Also C macros are just crappy textural replacement, I would assume an AST macro would be along the lines of lisp, which doesn't have much of anything to do with the shitty macro system in C despite using the word "macro". Not that I care so whatevars!
Nov 18 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 18 November 2013 at 09:12:19 UTC, Timon Gehr wrote:
 Well, you could do:

  screwUp int foo(){ return 3+7; }

 assert(foo()!=10);
Anything that allows it routinely should be banned whenever macros are accepted or not. Modifying existing "normal" symbols = no way.
Nov 18 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-18 16:59, Dicebot wrote:

 Anything that allows it routinely should be banned whenever macros are
 accepted or not. Modifying existing "normal" symbols = no way.
The whole point of macros is to modify symbols. Turning this: auto p = Person.where(q => q.name == "John"); To an SQL query. -- /Jacob Carlborg
Nov 18 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 18 November 2013 at 21:09:26 UTC, Jacob Carlborg wrote:
 On 2013-11-18 16:59, Dicebot wrote:

 Anything that allows it routinely should be banned whenever 
 macros are
 accepted or not. Modifying existing "normal" symbols = no way.
The whole point of macros is to modify symbols. Turning this: auto p = Person.where(q => q.name == "John"); To an SQL query.
You don't need to modify actual input lambda for this to become SQL query. I fully support macros as convenient way for AST reflection and related code generation but not a single step further. Changing semantics of existing valid piece of code is never worth it.
Nov 18 2013
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/18/2013 11:09 PM, Dicebot wrote:
 On Monday, 18 November 2013 at 21:09:26 UTC, Jacob Carlborg wrote:
 On 2013-11-18 16:59, Dicebot wrote:

 Anything that allows it routinely should be banned whenever macros are
 accepted or not. Modifying existing "normal" symbols = no way.
The whole point of macros is to modify symbols. Turning this: auto p = Person.where(q => q.name == "John"); To an SQL query.
You don't need to modify actual input lambda for this to become SQL query. I fully support macros as convenient way for AST reflection and related code generation but not a single step further. Changing semantics of existing valid piece of code is never worth it.
So you oppose macro attributes?
Nov 18 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 18 November 2013 at 22:47:59 UTC, Timon Gehr wrote:
 So you oppose macro attributes?
Yes. For me proper "macro attribute" is just an UDA that can be used by macro to tweak code generation. Actually I think I'd oppose a very large part of this specific DIP when it comes to exact syntax and usage models - I am more interested in "AST" than in "macro" part.
Nov 19 2013
parent "Yota" <yotaxp thatGoogleMailThing.com> writes:
On Tuesday, 19 November 2013 at 13:10:44 UTC, Dicebot wrote:
 On Monday, 18 November 2013 at 22:47:59 UTC, Timon Gehr wrote:
 So you oppose macro attributes?
Yes. For me proper "macro attribute" is just an UDA that can be used by macro to tweak code generation. Actually I think I'd oppose a very large part of this specific DIP when it comes to exact syntax and usage models - I am more interested in "AST" than in "macro" part.
This topic really took off. I'm afraid to say that I've only gotten through about half of it so far. I noticed computation expressions were mentioned, but has anyone by chance noted this little-known feature of F#? It sounds exactly like what is being proposed here. http://msdn.microsoft.com/en-us/library/vstudio/dd233212.aspx I don't believe I've ever seen F#'s code quotations being used in real code. But then again, F# doesn't have CTFE or code mixins. The LINQ sample in the DIP kinda rubs me the wrong way. LINQ query syntax (and for that matter, F# computation expressions) does not generate imperative code, but rather a chain of extension methods. More importantly, it does not eagerly chomp through all the data. It runs on IEnumerable, and is therefore lazy. (Like D ranges.) I wouldn't mind, however, if I saw these macros generate state machines specifically made for a given query.
Nov 19 2013
prev sibling next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Monday, 18 November 2013 at 22:09:35 UTC, Dicebot wrote:
 On Monday, 18 November 2013 at 21:09:26 UTC, Jacob Carlborg 
 wrote:
 On 2013-11-18 16:59, Dicebot wrote:

 Anything that allows it routinely should be banned whenever 
 macros are
 accepted or not. Modifying existing "normal" symbols = no way.
The whole point of macros is to modify symbols. Turning this: auto p = Person.where(q => q.name == "John"); To an SQL query.
You don't need to modify actual input lambda for this to become SQL query. I fully support macros as convenient way for AST reflection and related code generation but not a single step further. Changing semantics of existing valid piece of code is never worth it.
And that's exactly the kind of thing Walter doesn't want to see.
Nov 18 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-18 23:09, Dicebot wrote:

 You don't need to modify actual input lambda for this to become SQL
 query. I fully support macros as convenient way for AST reflection and
 related code generation but not a single step further. Changing
 semantics of existing valid piece of code is never worth it.
Well, it depends on how you look at it. You don't need to modify the lambda, but you're changing the meaning of it. -- /Jacob Carlborg
Nov 18 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 19 November 2013 at 07:23:52 UTC, Jacob Carlborg 
wrote:
 Well, it depends on how you look at it. You don't need to 
 modify the lambda, but you're changing the meaning of it.
Not really. You won't say that this snippet changes the meaning of lambda, do you? ``` foo(a => a*2); // ... bool foo(T)(T func) { // does not even call `func` and does something unrelated return is(ReturnType!T == int); } ``` This is pretty much the same. Functions and templates can do arbitrary things unrelated to their input and we do not call it "changing the meaning" of the input. Macros are not much different here. Though I'd still prefer requirement for explicit "macro!" syntax or something similar, just to make clear that code gen is incoming.
Nov 19 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-19 14:16, Dicebot wrote:

 Not really. You won't say that this snippet changes the meaning of
 lambda, do you?

 ```
 foo(a => a*2);
 // ...
 bool foo(T)(T func)
 {
      // does not even call `func` and does something unrelated
      return is(ReturnType!T == int);
 }
 ```
No, I guess not. -- /Jacob Carlborg
Nov 19 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/18/2013 12:22 AM, Jacob Carlborg wrote:
 On 2013-11-17 21:52, Walter Bright wrote:

 1. I don't believe we can decide on language features by analogy. D is
 complex enough that one can use analogy to justify anything.

 2. You cannot do anything behind a function call - the 'return'
 discussed earlier, and async/await for another, i.e. operator
 overloading cannot introduce control flow, cannot introduce variables
 into the current scope, etc.
You can turn this: BigInt(3) + 7; Into something that is not 10.
That is not at all what I was talking about. I was talking about: 3+7 being transformed by AST manipulation into something that is not 10.
 You cannot do anything behind a macro. You cannot introduce new syntax.
Nov 18 2013
prev sibling parent "Doodoo" <barf barf.com> writes:
HS Teoh why u always gotta go and break the forum
Nov 17 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 14 November 2013 at 21:12:11 UTC, Walter Bright 
wrote:
 On the other hand, with string mixins, it is immediately 
 obvious that one is dealing with different syntax, and so 
 expect something different to happen.
This is exactly the reason why I have been asking to make denoting macro usage with explicit keyword mandatory, like it is done with mixins. It does not change much about the concept itself and is not worth fixating upon as a blocker.
Nov 14 2013
prev sibling next sibling parent "Zsombor Barna" <belzurix sch.bme.hu> writes:
On Thursday, 14 November 2013 at 21:12:11 UTC, Walter Bright 
wrote:
 On 11/14/2013 11:40 AM, deadalnix wrote:
 This is not proposing to configure syntax.
Right, it is about inserting arbitrarily different meaning into existing syntax. For example, I have a friend from the middle east who worked at Microsoft. His english was excellent. But sometimes he'd get tripped up - for example, he'd take his idea to his boss, who'd discuss it, and then say: "fine" He took that to mean "my boss approves of my great idea!" Unfortunately, "fine" has undergone a meaning reversal colloquially, and actually means "your idea sux". This wound up getting him into trouble. There were a few other phrases like this with unintentionally hilarious misunderstandings. (Also, a popular way of encrypting communications is to have private meanings for phrases like "my cat ate my homework" meaning "initiate Operation X".) Now, let's take the AST macros, and pass it "4+3". Apparently is a feature that the macro can transform this to produce "hello" as the result. The result is that although the language looks the same, there is no "anchor" a person can rely on to indicate the meaning. The meaning of "fine" is reversed, so to speak. The riposte is, of course, every feature can be abused and good programmers won't do that. Unfortunately, it's IN THE VERY NATURE of macros to be abusive, and even the very best programmers can and will take it to the limit. They will see and decry the abuse in everyone else's code but their own, which they will justify. As I said, I see this over and over again. On the other hand, with string mixins, it is immediately obvious that one is dealing with different syntax, and so expect something different to happen.
You have a point with the example, but just as with templates ( the distinctive binary ! operator), making macros not to look first class language elements when called does great job with readability. #-s or ×-s are just good for this, even if it look very foreign. #loop ( ) { -- code -- } Maybe for macros that return macros it would be ugly: #(#macro1(params1))(params2); But just as in C where the -> operator automatically dereferences pointers, it could be solved. # macro1(params1) =#= (params2); (Just for fun)
Nov 14 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 14 November 2013 at 21:12:11 UTC, Walter Bright 
wrote:
 On 11/14/2013 11:40 AM, deadalnix wrote:
 This is not proposing to configure syntax.
Right, it is about inserting arbitrarily different meaning into existing syntax.
Yes, but EXPLICITELY.
 (Also, a popular way of encrypting communications is to have 
 private meanings for phrases like "my cat ate my homework" 
 meaning "initiate Operation X".)
Note that you do not pretend that sentence with "THIS IS AN ENCRYPTED SENTENCE, HERE TO DECRYPT IT DO XXX".
 Now, let's take the AST macros, and pass it "4+3". Apparently 
 is a feature that the macro can transform this to produce 
 "hello" as the result. The result is that although the language 
 looks the same, there is no "anchor" a person can rely on to 
 indicate the meaning. The meaning of "fine" is reversed, so to 
 speak.
mymacro(4+3); // <= can be "hello" myfunction(4+3); // <= can be "hello as well" Quite frankly, this isn't really convincing.
 The riposte is, of course, every feature can be abused and good 
 programmers won't do that. Unfortunately, it's IN THE VERY 
 NATURE of macros to be abusive, and even the very best 
 programmers can and will take it to the limit. They will see 
 and decry the abuse in everyone else's code but their own, 
 which they will justify. As I said, I see this over and over 
 again.
I can find article on the internet saying the exact same thing for any feature ever invented. You'll find quite easily some about templates or operator overloading, but if you go in Functionnalland, you'll find the same about virtual dispatch and inheritance. If you go in OOPland, you'll find the exact same argument for monads. You are qualifying the nature of macro here, that is not an argument. "Macros are bad, so I conclude macro are bad."
 On the other hand, with string mixins, it is immediately 
 obvious that one is dealing with different syntax, and so 
 expect something different to happen.
string mixins + a CTFE usable lexer+parser+D codegen from AST would achieve something close to the macros proposed here. This would obviously require quite a lot of work and is doomed to be dramatically slow and resources hungry, less hygienic (as identifier are implicitly resolved in the mixed in scope rather than explicitly) but quite close in term of capabilities. You can't argue that string mixin is good while AST macro is bad.
Nov 14 2013
next sibling parent reply "SomeDude" <lovelydear mailmetrash.com> writes:
On Thursday, 14 November 2013 at 22:06:06 UTC, deadalnix wrote:
 mymacro(4+3); // <= can be "hello"
 myfunction(4+3); // <= can be "hello as well"

 Quite frankly, this isn't really convincing.
True but you will hardly use myfunction as an extension to the language. If I understand the issues, I quite understand Walter's reluctance to add such a feature. Once you add it, you add the capability to change the language, or to create an entirely new language. If you let it, no matter how loud you cry, people *will* abuse the feature, and they will do it from day one. The C++ templates experiment shows what happens when you do that. We don't want to repeat the same mistakes. Basically, people want to change the language, but without having to discuss their own extensions in the newsgroups. It's sometimes handy, but you'll end up with crappy features all over the place.
Nov 17 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-17 09:18, SomeDude wrote:

 True but you will hardly use myfunction as an extension to the language.

 If I understand the issues, I quite understand Walter's reluctance to
 add such a feature. Once you add it, you add the capability to change
 the language, or to create an entirely new language. If you let it, no
 matter how loud you cry, people *will* abuse the feature, and they will
 do it from day one. The C++ templates experiment shows what happens when
 you do that. We don't want to repeat the same mistakes.
We already have templates and operator overloading. Perhaps we should remove those, we don't want to take the chance of people abusing them.
 Basically, people want to change the language, but without having to
 discuss their own extensions in the newsgroups. It's sometimes handy,
 but you'll end up with crappy features all over the place.
No, people want a general solution that doesn't require the language to be extended each time they come up with a useful feature. -- /Jacob Carlborg
Nov 17 2013
parent reply "SomeDude" <lovelydear mailmetrash.com> writes:
On Sunday, 17 November 2013 at 10:41:46 UTC, Jacob Carlborg wrote:
 We already have templates and operator overloading. Perhaps we 
 should remove those, we don't want to take the chance of people 
 abusing them.
D templates have been designed with the past experience of C++ templates in mind, so the mistakes have been avoided. With macros, we don't have much experience; Lisp macros can't count as the language is too different.
 Basically, people want to change the language, but without 
 having to
 discuss their own extensions in the newsgroups. It's sometimes 
 handy,
 but you'll end up with crappy features all over the place.
No, people want a general solution that doesn't require the language to be extended each time they come up with a useful feature.
Sorry but I tend to value the feedback of people who have already put some serious effort in expression templates and mixins like Don.
Nov 18 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 18 November 2013 at 21:38:08 UTC, SomeDude wrote:
 No, people want a general solution that doesn't require the 
 language to be extended each time they come up with a useful 
 feature.
Sorry but I tend to value the feedback of people who have already put some serious effort in expression templates and mixins like Don.
The very first code I wrote in D used expression templates to do vector/matrices computation. I'm really well aware of they are and what they can do (and what error messages you get when you do something wrong).
Nov 18 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2013 2:06 PM, deadalnix wrote:
 You can't argue that string mixin is good while AST macro is bad.
True, but you also cannot argue that since D has string mixins it must also have AST macros. What macros do is bless, encourage, and make inevitable a particularly abusive style of programming. D already has super powerful metaprogramming abilities that we are far from exploring the limits of. I don't think we should be adding on even more that experience has shown goes quickly out of control.
Nov 17 2013
parent "Zsombor Barna" <belzurix sch.bme.hu> writes:
On Sunday, 17 November 2013 at 09:13:04 UTC, Walter Bright wrote:
 On 11/14/2013 2:06 PM, deadalnix wrote:
 You can't argue that string mixin is good while AST macro is 
 bad.
True, but you also cannot argue that since D has string mixins it must also have AST macros. What macros do is bless, encourage, and make inevitable a particularly abusive style of programming. D already has super powerful metaprogramming abilities that we are far from exploring the limits of. I don't think we should be adding on even more that experience has shown goes quickly out of control.
Yeah, you are right, but I doubt anybody would use macros in production code without considering much less powerful constructs of the language to solve the problem. In hobby projects, I don't mind if create spaghetti code while experimenting.
Nov 17 2013
prev sibling next sibling parent reply "Zsombor Barna" <belzurix sch.bme.hu> writes:
On Thursday, 14 November 2013 at 18:23:20 UTC, Andrei 
Alexandrescu wrote:
 On 11/14/13 2:17 AM, Walter Bright wrote:
 On 11/14/2013 2:07 AM, Chris Cain wrote:
 I guess I would have thought the exact opposite from you... 
 that this is
 precisely something that *should* be done with a 
 library-defined macro.
Again, I reiterate what experience shows happens with macro systems in the long term.
FWIW Walter talked me back around 2005-2006 into abandoning my own ideas about languages with configurable syntax. There are quite a few failings about comparing programming languages against natural languages, but here's one that I think does have value: fixed syntax is ingrained into people's notion of language, and swapping syntax within an otherwise identical linguistic context is extremely taxing on the brain. Anyone who's read with a C or C++ codebase full of macros and #if-driven code can attest how unbelievably difficult juxtapositions of the normal syntax with preprocessor syntax can quickly become. That's part of why "static if" has been so successful in D - it drives new semantics but within the same syntax. Natural languages are "humans complete" because they are the one vehicle we use to describe and manipulate our understanding of the entire reality. If configurable syntax was something necessary to model the world, it would have inevitably occurred in natural languages one way or another. Instead, all human languages (with no exception I know of) have fixed syntax and prefer to add modeling power in other ways. There must be something about this. Andrei
D's syntax remains the same ( statements, expressions, function calling, numbers etc. ). These AST manipulation tools just define new words or language constructs. Human languages tend to be altered as times passes and the needs change. That's why new words appear. Even the grammar is not the same as the one my grandparents used.
Nov 14 2013
parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Thursday, 14 November 2013 at 19:42:36 UTC, Zsombor Barna 
wrote:
 D's syntax remains the same ( statements, expressions, function 
 calling, numbers etc. ). These AST manipulation tools just 
 define new words or language constructs. Human languages tend 
 to be altered as times passes and the needs change. That's why 
 new words appear. Even the grammar is not the same as the one 
 my grandparents used.
Which shows all the problems raised by Walter: 1. you need an entry in the dictionary that you need to look up for each new word that someone has introduced, 2. with natural language, you don't need to know its *exact* semantics when you use a new word, while in programming it is mandatory, 3. you *don't* want your programming language to arbitrarily change over time. That's probably the biggest issue.
Nov 17 2013
prev sibling parent "Rob T" <alanb ucora.com> writes:
On Thursday, 14 November 2013 at 18:23:20 UTC, Andrei 
Alexandrescu wrote:
[...]
 Natural languages are "humans complete" because they are the 
 one vehicle we use to describe and manipulate our understanding 
 of the entire reality. If configurable syntax was something 
 necessary to model the world, it would have inevitably occurred 
 in natural languages one way or another. Instead, all human 
 languages (with no exception I know of) have fixed syntax and 
 prefer to add modeling power in other ways. There must be 
 something about this.


 Andrei
Not an unreasonable assumption to make, so assuming it's true, then we need a fixed way to deal with a certain set of problems that so far cannot be done reasonably (or at all) in current D. One case, which I also so happen to want to solve, is the async/await type of problem. There's simply no reasonable solution available other than through extensions to the language, which may or may not ever materialize. It's very frustrating when you want to explore possibilities today rather than wait for an extension that may never arrive. Adding in the async/await extensions can help solve only that specific problem, but how many other types of problems that are similar to this one will programmers still want to solve? A more general solution would seem to be far more useful. The other simple problem I encountered, at first seems like a mixin can deal with or perhaps a template, but current D simply cannot do it in a reasonable way that I'm aware of. I want to be able to pass an argument to an "inspect" function, that will log or display the symbol name of the top-level argument and it's corresponding value. Example: int a = 300; inspect( a ); // or inspect!a; The only way I can do this is through double entry of the item to inspect eg inspect( a.stringof, a ); // error prone and irritatingly redundant I can get some success using an alias template parameter, but it does not work in general, only in a few cases will it work, so templates are not a solution without some additional enhancements. May be by adding in something like __ARGS__, or expanding on__traits can solve this type of problem, but there's no doubt many other scenarios that will require similar extensions that users of the language cannot implement. There's no end to this without much better access to reflection. --rt
Nov 14 2013
prev sibling parent reply "Meta" <jared771 gmail.com> writes:
On Thursday, 14 November 2013 at 09:12:38 UTC, Walter Bright 
wrote:
 I do understand this. I've not only extensively used macro 
 systems, I've designed two successful ones and implemented 
 other peoples' designs.

 I've also seen the sheer awfulness of what happens with these 
 systems over the long term. The fascinating thing about this 
 awfulness is people are well aware of it in everyone's macro 
 libraries but their own.
What kind of macro systems are you talking about? C-like macros? Assembly macros? Lisp-like macros? It seems that Jacob's proposed system is similar to none of these.
Nov 14 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2013 9:08 AM, Meta wrote:
 What kind of macro systems are you talking about? C-like macros? Assembly
 macros? Lisp-like macros? It seems that Jacob's proposed system is similar to
 none of these.
See my first comment in this thread - it's an ancestor to this one.
Nov 14 2013
parent reply "Meta" <jared771 gmail.com> writes:
On Thursday, 14 November 2013 at 17:55:57 UTC, Walter Bright 
wrote:
 On 11/14/2013 9:08 AM, Meta wrote:
 What kind of macro systems are you talking about? C-like 
 macros? Assembly
 macros? Lisp-like macros? It seems that Jacob's proposed 
 system is similar to
 none of these.
See my first comment in this thread - it's an ancestor to this one.
I'm using the website forum, so I see posts in a continuous format, not a threaded view. Anyway, it seems like assembly macros are actually quite similar to Lisp macros. Again, it seems like what Jacob proposes is not really much like Lisp macros, nor C macros.
Nov 14 2013
parent reply "simendsjo" <simendsjo gmail.com> writes:
On Thursday, 14 November 2013 at 19:22:08 UTC, Meta wrote:
 On Thursday, 14 November 2013 at 17:55:57 UTC, Walter Bright 
 wrote:
 On 11/14/2013 9:08 AM, Meta wrote:
 What kind of macro systems are you talking about? C-like 
 macros? Assembly
 macros? Lisp-like macros? It seems that Jacob's proposed 
 system is similar to
 none of these.
See my first comment in this thread - it's an ancestor to this one.
I'm using the website forum, so I see posts in a continuous format, not a threaded view. Anyway, it seems like assembly macros are actually quite similar to Lisp macros. Again, it seems like what Jacob proposes is not really much like Lisp macros, nor C macros.
At the top of the page you should have "view mode" links which supports a threaded view.
Nov 14 2013
parent "Meta" <jared771 gmail.com> writes:
On Thursday, 14 November 2013 at 19:27:42 UTC, simendsjo wrote:
 At the top of the page you should have "view mode" links which 
 supports a threaded view.
Neat. I have my browser set up to auto-login, so I rarely, if ever, have to look up there.
Nov 14 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-14 01:16, Walter Bright wrote:

 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.
You can still do stupid things like that with operator overloading. Not on built-in types, but on user defined types. Every language allows you to do stupid things, one way or another. -- /Jacob Carlborg
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 11:37 PM, Jacob Carlborg wrote:
 On 2013-11-14 01:16, Walter Bright wrote:

 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.
You can still do stupid things like that with operator overloading. Not on built-in types, but on user defined types. Every language allows you to do stupid things, one way or another.
Sure, but the issue is that expression templates are not for "int+int", but for "T+int". My question is what value would there be in a rewrite of "int+int" to mean something different.
Nov 14 2013
parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 14.11.2013 09:53, schrieb Walter Bright:
 On 11/13/2013 11:37 PM, Jacob Carlborg wrote:
 On 2013-11-14 01:16, Walter Bright wrote:

 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.
You can still do stupid things like that with operator overloading. Not on built-in types, but on user defined types. Every language allows you to do stupid things, one way or another.
Sure, but the issue is that expression templates are not for "int+int", but for "T+int". My question is what value would there be in a rewrite of "int+int" to mean something different.
"int + int" could be part of an PC<->GPU kombination that generates CUDA- and D-interchange code at compiletime
Nov 14 2013
parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 14.11.2013 10:06, schrieb dennis luehring:
 Am 14.11.2013 09:53, schrieb Walter Bright:
 On 11/13/2013 11:37 PM, Jacob Carlborg wrote:
 On 2013-11-14 01:16, Walter Bright wrote:

 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.
You can still do stupid things like that with operator overloading. Not on built-in types, but on user defined types. Every language allows you to do stupid things, one way or another.
Sure, but the issue is that expression templates are not for "int+int", but for "T+int". My question is what value would there be in a rewrite of "int+int" to mean something different.
"int + int" could be part of an PC<->GPU kombination that generates CUDA- and D-interchange code at compiletime
or like Don Blade Engine with string mixins could produce "high performance x87 asm" code for normal D expressions
Nov 14 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/14/2013 1:08 AM, dennis luehring wrote:
 Am 14.11.2013 10:06, schrieb dennis luehring:
 Am 14.11.2013 09:53, schrieb Walter Bright:
 On 11/13/2013 11:37 PM, Jacob Carlborg wrote:
 On 2013-11-14 01:16, Walter Bright wrote:

 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.
You can still do stupid things like that with operator overloading. Not on built-in types, but on user defined types. Every language allows you to do stupid things, one way or another.
Sure, but the issue is that expression templates are not for "int+int", but for "T+int". My question is what value would there be in a rewrite of "int+int" to mean something different.
"int + int" could be part of an PC<->GPU kombination that generates CUDA- and D-interchange code at compiletime
or like Don Blade Engine with string mixins could produce "high performance x87 asm" code for normal D expressions
Good example, but still quite doable using type T instead.
Nov 14 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 13 November 2013 at 08:45:13 UTC, Walter Bright 
wrote:
 On 11/13/2013 12:03 AM, Jacob Carlborg wrote:
 Again, operator overloading in D is too limiting to implement 
 something Linq like.
Ok, let's set aside the opEquals and opCmp issue for the moment. Can AST macros do anything that expression templates cannot?
Yes, they can reflect the ast provided and act accordingly.
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 1:15 AM, deadalnix wrote:
 On Wednesday, 13 November 2013 at 08:45:13 UTC, Walter Bright wrote:
 On 11/13/2013 12:03 AM, Jacob Carlborg wrote:
 Again, operator overloading in D is too limiting to implement something Linq
 like.
Ok, let's set aside the opEquals and opCmp issue for the moment. Can AST macros do anything that expression templates cannot?
Yes, they can reflect the ast provided and act accordingly.
The reflection ability is not something specific to an AST, it could be added so that any expression node can be reflected. We already do much of that with __traits.
Nov 13 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 13 November 2013 at 19:19:32 UTC, Walter Bright 
wrote:
 The reflection ability is not something specific to an AST, it 
 could be added so that any expression node can be reflected. We 
 already do much of that with __traits.
Yes, that is why Dicebot was mentioning the parallel with compile time reflection.
Nov 13 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-13 20:19, Walter Bright wrote:

 The reflection ability is not something specific to an AST, it could be
 added so that any expression node can be reflected. We already do much
 of that with __traits.
Yes, I been thinking of ways to expand __traits. When there's enough of __traits it's not much left during it into real AST macros. Just the part that passes the AST instead of the evaluated result of the arguments to a function. BTW, is it complicated to return a more structured values, like a struct or a class, than just plain values like strings? -- /Jacob Carlborg
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 11:53 PM, Jacob Carlborg wrote:
 On 2013-11-13 20:19, Walter Bright wrote:

 The reflection ability is not something specific to an AST, it could be
 added so that any expression node can be reflected. We already do much
 of that with __traits.
Yes, I been thinking of ways to expand __traits. When there's enough of __traits it's not much left during it into real AST macros. Just the part that passes the AST instead of the evaluated result of the arguments to a function. BTW, is it complicated to return a more structured values, like a struct or a class, than just plain values like strings?
No, but then you're face with the issue of where those structs are defined.
Nov 14 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-14 10:13, Walter Bright wrote:

 No, but then you're face with the issue of where those structs are defined.
In druntime, just like MouldeInfo, TypeInfo and similar classes. -- /Jacob Carlborg
Nov 14 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-13 09:45, Walter Bright wrote:

 Ok, let's set aside the opEquals and opCmp issue for the moment.

 Can AST macros do anything that expression templates cannot?
I need to read up on what expression templates can do. -- /Jacob Carlborg
Nov 13 2013
parent reply "Rob T" <alanb ucora.com> writes:
On Wednesday, 13 November 2013 at 12:21:22 UTC, Jacob Carlborg 
wrote:

 I need to read up on what expression templates can do.
But where to read? There's no such thing as "expression template" explicitly mentioned in here. http://dlang.org/template.html --rt
Nov 13 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 3:27 PM, Rob T wrote:
 But where to read? There's no such thing as "expression template" explicitly
 mentioned in here.
This may help: http://www.dsource.org/projects/ddl/browser/trunk/meta/regex.d
Nov 13 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-14 00:27, Rob T wrote:
 On Wednesday, 13 November 2013 at 12:21:22 UTC, Jacob Carlborg wrote:

 I need to read up on what expression templates can do.
But where to read? There's no such thing as "expression template" explicitly mentioned in here. http://dlang.org/template.html
I guess everything about C++ expression templates would be a start. http://en.wikipedia.org/wiki/Expression_templates -- /Jacob Carlborg
Nov 13 2013
prev sibling parent reply "Don" <x nospam.com> writes:
On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright wrote:
 I forgot to mention that "expression templates" can be used in 
 D in an equivalent manner that they are used in C++,
They are crippled compared to C++, because you have no control over the return type of opCmp and opEquals, which means that you can't have expression templates involving comparisons. That's extremely limiting. My feeling with AST macros is: (1) AST macros are basically syntax sugar for string mixins. The one and only thing that string mixins have in their favour, is that they have almost no syntax. But that is actually a *huge* plus. Because it's syntax sugar, an AST macro syntax would be most convincing if you took some existing string mixins from various real projects, and show how beautiful they would become with macros. (2) The big functionality we don't have, is comprehensive reflection. I'd like to see more thought in that area. It seems challenging to define reflection in a way that doesn't expose compiler internals. The question I find particularly interesting is, "if we had macros, would reflection look different?"
Nov 12 2013
next sibling parent reply "Rob T" <alanb ucora.com> writes:
On Tuesday, 12 November 2013 at 11:06:17 UTC, Don wrote:
 On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright 
 wrote:
 I forgot to mention that "expression templates" can be used in 
 D in an equivalent manner that they are used in C++,
They are crippled compared to C++, because you have no control over the return type of opCmp and opEquals, which means that you can't have expression templates involving comparisons. That's extremely limiting. My feeling with AST macros is: (1) AST macros are basically syntax sugar for string mixins. The one and only thing that string mixins have in their favour, is that they have almost no syntax. But that is actually a *huge* plus. Because it's syntax sugar, an AST macro syntax would be most convincing if you took some existing string mixins from various real projects, and show how beautiful they would become with macros. (2) The big functionality we don't have, is comprehensive reflection. I'd like to see more thought in that area. It seems challenging to define reflection in a way that doesn't expose compiler internals. The question I find particularly interesting is, "if we had macros, would reflection look different?"
My personal main need for macros at this point, is to make up for a lack of better reflection, but as has been pointed out, it's not clear how macros would help anyway without access to the sort of reflection that I'm currently lacking. What we seem to be discussing, is the need to unify within D the abilities to perform better reflection along with a means to automate code generation during compile time. The current way reflection and mixins are done, are totally different, almost like two different languages with D, so it would be nice to have a more unified system that more closely resembles D itself. BTW, I agree with Walters concerns about AST macros. It would be ugly to work with code that is not comprehensible due to over use of macros that attempt to redefine the language into something else. OTOH there's also significant advantages to be able to add certain things into the language that are currently lacking. For example, I've experimented with a method of implementing co-routines that use switch statements, but it's very ugly to do without macro support to the point of being impractical. --rt
Nov 12 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 19:01, Rob T wrote:

 My personal main need for macros at this point, is to make up for a lack
 of better reflection, but as has been pointed out, it's not clear how
 macros would help anyway without access to the sort of reflection that
 I'm currently lacking.
Macros would need a reflection API to be usable.
 What we seem to be discussing, is the need to unify within D the
 abilities to perform better reflection along with a means to automate
 code generation during compile time. The current way reflection and
 mixins are done, are totally different, almost like two different
 languages with D, so it would be nice to have a more unified system that
 more closely resembles D itself.

 BTW, I agree with Walters concerns about AST macros. It would be ugly to
 work with code that is not comprehensible due to over use of macros that
 attempt to redefine the language into something else. OTOH there's also
 significant advantages to be able to add certain things into the
 language that are currently lacking. For example, I've experimented with
 a method of implementing co-routines that use switch statements, but
 it's very ugly to do without macro support to the point of being
 impractical.

 --rt
-- /Jacob Carlborg
Nov 13 2013
prev sibling next sibling parent reply "Joseph Cassman" <jc7919 outlook.com> writes:
On Tuesday, 12 November 2013 at 11:06:17 UTC, Don wrote:
 On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright 
 wrote:
 I forgot to mention that "expression templates" can be used in 
 D in an equivalent manner that they are used in C++,
They are crippled compared to C++, because you have no control over the return type of opCmp and opEquals, which means that you can't have expression templates involving comparisons. That's extremely limiting. My feeling with AST macros is: (1) AST macros are basically syntax sugar for string mixins. The one and only thing that string mixins have in their favour, is that they have almost no syntax. But that is actually a *huge* plus. Because it's syntax sugar, an AST macro syntax would be most convincing if you took some existing string mixins from various real projects, and show how beautiful they would become with macros. (2) The big functionality we don't have, is comprehensive reflection. I'd like to see more thought in that area. It seems challenging to define reflection in a way that doesn't expose compiler internals. The question I find particularly interesting is, "if we had macros, would reflection look different?"
After using string mixins for a while I have come to feel much like your statement in (1). I like the power and flexibility of string mixins. Text and strings are understandable and straightforward to manipulate. This is a clear win in my opinion. There are also some things I find difficult at the moment. - One is the process of what to do when things do not go as expected. To debug I currently use pragma statements and try to visualize what is going on in my mind. This is do-able. But I feel it could be better. - It would be nice to be able to use some more D language concepts to construct the string mixin inside the template. For example, sometimes I would like to use a loop to construct the result. If these two items could be addressed I would be pretty satisfied with using string mixins as they are. One win that the AST macro concept provides that string mixins does not is manipulation of code syntax trees (either at compile or run time). It would be nice to combine the various code manipulation ideas that have been tossed around before into a single module, say std.meta or std.reflection. In addition to combining the various current code manipulation functionality into a single api (e.g. __traits, std.traits), it could provide functionality to work with code-in-use. For example, it would be pretty cool to be able to do the following (the actual syntax for constructing the code is not that important to me, but the underlying functionality for manipulating code is). sample.d import std.meta; void main(string[] args) { auto e = construct("a + %s", args[1]); writefln("%s", e(5)); auto f = construct(e + "b"); writefln("%s", f(?, 10).toDebugString); writefln("%s", f(5, 10)); } -----
 ./sample 2
 7
 Numerical Expression: a + 2 + 10
 17
Implementing the "expression tree" side of AST macros seems like it could be done as a library, which would be a win for maintainability and cross-compiler portability. There is probably required, compiler-related functionality that I cannot see making this unfeasable or possibly difficult to do. Still, looks good to me on paper at least. Perhaps going this route could gain the good functionality of AST macros while also making use of the already implemented features? Seems to me like some of these ideas tie into previous discussions. Here are two I remember. I am sure there are others. http://forum.dlang.org/thread/juf7sk$16rl$1 digitalmars.com http://forum.dlang.org/thread/mailman.1716.1340388570.24740.digitalmars-d puremagic.com Joseph
Nov 12 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 21:03, Joseph Cassman wrote:

 After using string mixins for a while I have come to feel much like your
 statement in (1). I like the power and flexibility of string mixins.
 Text and strings are understandable and straightforward to manipulate.
 This is a clear win in my opinion.

 There are also some things I find difficult at the moment.

   - One is the process of what to do when things do not go as expected.
 To debug I currently use pragma statements and try to visualize what is
 going on in my mind. This is do-able. But I feel it could be better.
If we got macros we should really have a good way to debug them. At least a flag to shows how the macros are lowered.
   - It would be nice to be able to use some more D language concepts to
 construct the string mixin inside the template. For example, sometimes I
 would like to use a loop to construct the result.

 If these two items could be addressed I would be pretty satisfied with
 using string mixins as they are.

 One win that the AST macro concept provides that string mixins does not
 is manipulation of code syntax trees (either at compile or run time). It
 would be nice to combine the various code manipulation ideas that have
 been tossed around before into a single module, say std.meta or
 std.reflection. In addition to combining the various current code
 manipulation functionality into a single api (e.g. __traits,
 std.traits), it could provide functionality to work with code-in-use.
 For example, it would be pretty cool to be able to do the following (the
 actual syntax for constructing the code is not that important to me, but
 the underlying functionality for manipulating code is).

 sample.d

      import std.meta;
      void main(string[] args) {
          auto e = construct("a + %s", args[1]);
          writefln("%s", e(5));
          auto f = construct(e + "b");
          writefln("%s", f(?, 10).toDebugString);
          writefln("%s", f(5, 10));
      }
Yes, but the problem with string mixins is that you need to reparse a lot of code. With string mixins it becomes: 1. The compiler parses the code and creates an AST (fine so far) 2. Does type checking an other stuff 3. Library code parses a string into an AST 4. Library code manipulates the AST 5. Library code converts the AST back to a string 6. The string is interested with a string mixin at the calling context 7. The compiler parses the code and creates an AST 8. Does type checking an other stuff With AST macaros you can avoid many of these steps: 1. The compiler parses the code and creates an AST 2. Does type checking an other stuff 3. Library code (macro) manipulates the AST 4. Compiler inserts the manipulate AST into the calling context 5. Does type checking an other stuff Instead of eight steps we get five steps. We remove the unnecessary conversions between AST's and strings. -- /Jacob Carlborg
Nov 13 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2013 3:06 AM, Don wrote:
 On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright wrote:
 I forgot to mention that "expression templates" can be used in D in an
 equivalent manner that they are used in C++,
They are crippled compared to C++, because you have no control over the return type of opCmp and opEquals, which means that you can't have expression templates involving comparisons. That's extremely limiting.
It is limiting, but I don't know about extremely limiting. Eric Anderton's regex engine didn't need them. And, if you really do need them, you can always define the templates as regular names: lessThan(a,b) equals(a,b) etc. Not the greatest, but not unworkable.
Nov 12 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-13 00:26, Walter Bright wrote:

 It is limiting, but I don't know about extremely limiting. Eric
 Anderton's regex engine didn't need them. And, if you really do need
 them, you can always define the templates as regular names:

     lessThan(a,b)
     equals(a,b)

 etc. Not the greatest, but not unworkable.
I don't know what's so special about the regex engine. It takes a regex as a string and matches that against another string. Seems to be just like any other regular expression library I've used. -- /Jacob Carlborg
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 12:17 AM, Jacob Carlborg wrote:
 On 2013-11-13 00:26, Walter Bright wrote:

 It is limiting, but I don't know about extremely limiting. Eric
 Anderton's regex engine didn't need them. And, if you really do need
 them, you can always define the templates as regular names:

     lessThan(a,b)
     equals(a,b)

 etc. Not the greatest, but not unworkable.
I don't know what's so special about the regex engine. It takes a regex as a string and matches that against another string. Seems to be just like any other regular expression library I've used.
What's special about it was its use of expression templates. It made for a nice demo of how to do such.
Nov 13 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-13 09:38, Walter Bright wrote:

 What's special about it was its use of expression templates. It made for
 a nice demo of how to do such.
Is there an actual point in doing that, except a nice exercise in expression templates? -- /Jacob Carlborg
Nov 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2013 4:25 AM, Jacob Carlborg wrote:
 On 2013-11-13 09:38, Walter Bright wrote:

 What's special about it was its use of expression templates. It made for
 a nice demo of how to do such.
Is there an actual point in doing that, except a nice exercise in expression templates?
It actually generated a very fast regex engine, though Dmitry's work has since eclipsed it.
Nov 13 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-13 20:16, Walter Bright wrote:

 It actually generated a very fast regex engine, though Dmitry's work has
 since eclipsed it.
Right, a specially tuned engine for that particular regex. -- /Jacob Carlborg
Nov 13 2013
prev sibling next sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Tuesday, 12 November 2013 at 09:20:07 UTC, Walter Bright wrote:
 5. We've said "no" in the past to things like user-defined 
 tokens, which are closely related.
Regarding this point, I get that arbitrary expansion is out, but would you be amenable to adding a small, _strictly-defined_ set of operators that are compile-time errors unless overloaded?
     #define BEGIN {
     #define END }

 macros used in old C code to make it look like Pascal.
A plague o' their houses! -Wyatt
Nov 12 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-12 10:20, Walter Bright wrote:

 For example, the Linq example in the DIP is not compelling, as
 aesthetically nicer code can be written using D's ranges and algorithms:

      auto data = arr.filter!(x => x > 5).array;
I was actually not the one that added the Linq examples. I like the above better. I've used a similar example as well: auto person = Person.where(e => e.name == "John"); Translates to: select * from person where name = 'John' But I don't see how that can currently be done in D. Operator overloading in D isn't flexible enough. For example, when implementing opEquals, you don't know if it's "==" or "!=" that's being called. Same with the comparison operators.
 I see no compelling advantage in trying to make D code look like C#; to
 be blunt it's like the old:

      #define BEGIN {
      #define END }

 macros used in old C code to make it look like Pascal.
I agree with this. -- /Jacob Carlborg
Nov 12 2013
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg 
wrote:
 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'
for those of us entirely unfamiliar with linq, what is this supposed to do? Select people with name "John" from a collection of people, like in sql? It seems trivial to do this using filter, or am I missing something...?
Nov 12 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 12 November 2013 at 14:38:44 UTC, John Colvin wrote:
 for those of us entirely unfamiliar with linq, what is this 
 supposed to do? Select people with name "John" from a 
 collection of people, like in sql? It seems trivial to do this 
 using filter, or am I missing something...?
I don't know much about linq but in macro context I'd expect it to generate an actual SQL query to database (and execute it) based on in-language filter statement (via reflection)
Nov 12 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 16:20, Dicebot wrote:

 I don't know much about linq but in macro context I'd expect it to
 generate an actual SQL query to database (and execute it) based on
 in-language filter statement (via reflection)
Exactly, spot on. The interesting thing there, in the context of macros, is converting the AST of the lambda to an string containing SQL code. -- /Jacob Carlborg
Nov 13 2013
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 11/12/2013 06:38 AM, John Colvin wrote:
 On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg wrote:
 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'
for those of us entirely unfamiliar with linq, what is this supposed to do? Select people with name "John" from a collection of people, like in sql? It seems trivial to do this using filter, or am I missing something...?
linq provides an interface to query any collection, but what is interesting in this case is Person.where(e => e.name == "John") invokes an engine that builds the necessary sql to issue an equivalent query to your sql database and assembles the results into a range of Person. And it can do this for any arbitrary predicate (well, almost. It doesn't handle function calls to arbitrary code too well). the .NET framework can do this because it exposes an api for querying, building, and compiling asts. D cannot do this because it doesn't. (and I have tried to make it work)
Nov 12 2013
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 12 November 2013 at 15:21:16 UTC, Ellery Newcomer 
wrote:
 On 11/12/2013 06:38 AM, John Colvin wrote:
 On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg 
 wrote:
 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'
for those of us entirely unfamiliar with linq, what is this supposed to do? Select people with name "John" from a collection of people, like in sql? It seems trivial to do this using filter, or am I missing something...?
linq provides an interface to query any collection, but what is interesting in this case is Person.where(e => e.name == "John") invokes an engine that builds the necessary sql to issue an equivalent query to your sql database and assembles the results into a range of Person. And it can do this for any arbitrary predicate (well, almost. It doesn't handle function calls to arbitrary code too well). the .NET framework can do this because it exposes an api for querying, building, and compiling asts. D cannot do this because it doesn't. (and I have tried to make it work)
oh, I see. Would AST macros really be enough to make this work in D? "Arbitrary code" is a huge feature space in D, including much that doesn't map well to anything outside of a relatively low-level language, let alone SQL. I can see it quickly becoming a nightmare that would be worse than just issuing the predicate as an sql string or some generic equivalent.
Nov 12 2013
next sibling parent reply "Ellery Newcomer" <ellery-newcomer utulsa.edu> writes:
On Tuesday, 12 November 2013 at 16:14:57 UTC, John Colvin wrote:
 On Tuesday, 12 November 2013 at 15:21:16 UTC, Ellery Newcomer 
 wrote:
 On 11/12/2013 06:38 AM, John Colvin wrote:
 On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg 
 wrote:
 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'
for those of us entirely unfamiliar with linq, what is this supposed to do? Select people with name "John" from a collection of people, like in sql? It seems trivial to do this using filter, or am I missing something...?
linq provides an interface to query any collection, but what is interesting in this case is Person.where(e => e.name == "John") invokes an engine that builds the necessary sql to issue an equivalent query to your sql database and assembles the results into a range of Person. And it can do this for any arbitrary predicate (well, almost. It doesn't handle function calls to arbitrary code too well). the .NET framework can do this because it exposes an api for querying, building, and compiling asts. D cannot do this because it doesn't. (and I have tried to make it work)
oh, I see. Would AST macros really be enough to make this work in D? "Arbitrary code" is a huge feature space in D, including much that doesn't map well to anything outside of a relatively low-level language, let alone SQL. I can see it quickly becoming a nightmare that would be worse than just issuing the predicate as an sql string or some generic equivalent.
Actually, s/arbitrary code/simple to moderately complex expressions that don't call functions that aren't explicitly handled by the translation engine/ and this is a better representation of what linq can do in e.g. entity framework. and you could go the route of django models, e.g. Place.objects.filter(name="Bob's Cafe") as long as you had enough metadata set up. you could probably get this to work in D, but A. it looks like crap B. it in no way resembles sql C. it would look like even worse crap in D (no named parameters!) D. it doesn't readily permit you to make queries with complex expression trees (idk, maybe something like linq's Places.Where(p => p.Elev < 200 && Addresses.Any(a => a.State == p.State)) why am I arguing against this? nobody wants this, nobody asked for this. Also, sql is to databases what assembly is to cpus. It's low level, [moderately] untyped, and it differs on every single platform. provide access to it, sure, but please don't use it to build your abstractions.
Nov 12 2013
parent reply =?ISO-8859-15?Q?Simen_Kj=E6r=E5s?= <simen.kjaras gmail.com> writes:
On 12.11.2013 18:53, Ellery Newcomer wrote:


 Place.objects.filter(name="Bob's Cafe")
[snip]
 C. it would look like even worse crap in D (no named parameters!)
It's perfectly possible in D to make this work: Place.objects.filter(args.name = "Bob", args.state = "unemployed"); Proof of concept: import std.stdio : writeln; struct NamedArg(string name, Args...) { Args value; alias value this; } struct args { property static auto opDispatch(string name, Args...)(Args args) { return NamedArg!(name, Args)(args); } } void foo(NamedArg!("name", string) a) { writeln(a); } void main() { foo(args.name = "Bob"); } And since the name is statically known, you can use this for e.g. lookup on objects. No, it's nowhere near as pretty, and the receiving code is horrible. If you were to use this for some kind of magic behind the scenes, it might be okay. Might. Please don't do it.
 D. it doesn't readily permit you to make queries with complex expression
 trees (idk, maybe something like linq's

 Places.Where(p => p.Elev < 200 && Addresses.Any(a => a.State == p.State))
Sorry, I got nuffin'. :p Waaait... Places.where(args.elev.lessThan(200) & args.state = (state)=>Addresses.any(state)); I *can* make that work. I'm not going to. -- Simen
Nov 13 2013
parent reply "Ellery Newcomer" <ellery-newcomer utulsa.edu> writes:
On Wednesday, 13 November 2013 at 10:51:48 UTC, Simen Kjærås 
wrote:
 On 12.11.2013 18:53, Ellery Newcomer wrote:

 It's perfectly possible in D to make this work:
hey, cool impl *comprehends code* I mean Ewww
 I *can* make that work. I'm not going to.

 --
   Simen
I concur with the second part.
Nov 13 2013
next sibling parent reply =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On 13.11.2013 20:05, Ellery Newcomer wrote:
 On Wednesday, 13 November 2013 at 10:51:48 UTC, Simen Kjærås wrote:
 On 12.11.2013 18:53, Ellery Newcomer wrote:

 It's perfectly possible in D to make this work:
hey, cool impl *comprehends code* I mean Ewww
 I *can* make that work. I'm not going to.

 --
   Simen
I concur with the second part.
I decided to abandon sanity. Luckily, only for the named parameters. I now have this code working: void test(int a, int b = 2, string c = "foo", int d = 14) { } alias test2 = nameify!test; void main() { test2(1, Args.d = 4, Args.c = "bar"); } With reasonable error messages for duplicate and missing parameters, as well as type mismatches. Source is attached. I hope God forgives me. -- Simen
Nov 17 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 17 November 2013 at 19:41:40 UTC, Simen Kjærås wrote:
 On 13.11.2013 20:05, Ellery Newcomer wrote:
 On Wednesday, 13 November 2013 at 10:51:48 UTC, Simen Kjærås 
 wrote:
 On 12.11.2013 18:53, Ellery Newcomer wrote:

 It's perfectly possible in D to make this work:
hey, cool impl *comprehends code* I mean Ewww
 I *can* make that work. I'm not going to.

 --
   Simen
I concur with the second part.
I decided to abandon sanity. Luckily, only for the named parameters. I now have this code working: void test(int a, int b = 2, string c = "foo", int d = 14) { } alias test2 = nameify!test; void main() { test2(1, Args.d = 4, Args.c = "bar"); } With reasonable error messages for duplicate and missing parameters, as well as type mismatches. Source is attached. I hope God forgives me.
That is a really cool idea !
Nov 17 2013
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 11/17/2013 11:41 AM, Simen Kjrs wrote:
 Source is attached. I hope God forgives me.
I suppose you'd have to do something like x.Where(OR( NOT(Args.thing = "thing"), Args.sing = "sing")) for nesting and negation and all that. I'd wait for walter to relax the restrictions on ==, &&, etc operator overloading. Then you probably could do x.Where(X => X.thing != "thing" || X.sing == "sing") let's see here. If X's type is QueryExp!Entity and the return type is SomeRange!Entity then querying looks doable. how about composing expressions? take the result of that lambda EXP, EXP = EXP && X.cling == 1; // should work fine? how about decomposing expressions? EXP = EXP.lhs; // should work fine? EXP.rhs.rhs.value // returns "sing" - should work fine? the lambda should create a closure, so we should be able to get out any value we put in you won't be able to do x.Where(X => upper(X.thing) == "THING") without ast macros but will have to be satisfied with x.Where(X => X.thing.upper() == "THING") will you be able to do x.Where(X => X.thing == "THING" && x.Any(X2 => X2.thing == X.thing && X2.id != X.id)) ? either "Any" would have to be only for template expressions, or X would have to be an outer context like so: x.Where(X => X.it.thing == "THING" && X.x.Any(X2 => X2.it.thing == X.it.thing && X2.it.id != X.it.id)) because x.Any(...) should return bool, but inside the query expression it should return QueryExp. In both cases it would take a param of QueryExp delegate(QueryExp) Anybody else have any other ideas?
Nov 17 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-17 22:15, Ellery Newcomer wrote:
 On 11/17/2013 11:41 AM, Simen Kjrs wrote:
 Source is attached. I hope God forgives me.
I suppose you'd have to do something like x.Where(OR( NOT(Args.thing = "thing"), Args.sing = "sing")) for nesting and negation and all that. I'd wait for walter to relax the restrictions on ==, &&, etc operator overloading. Then you probably could do x.Where(X => X.thing != "thing" || X.sing == "sing") let's see here. If X's type is QueryExp!Entity and the return type is SomeRange!Entity then querying looks doable. how about composing expressions? take the result of that lambda EXP, EXP = EXP && X.cling == 1; // should work fine? how about decomposing expressions? EXP = EXP.lhs; // should work fine? EXP.rhs.rhs.value // returns "sing" - should work fine? the lambda should create a closure, so we should be able to get out any value we put in you won't be able to do x.Where(X => upper(X.thing) == "THING")
That works with the Rails plugin Squeel. Although you need to prefix "upper" with "X.". https://github.com/ernie/squeel -- /Jacob Carlborg
Nov 18 2013
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/17/2013 08:41 PM, Simen Kjrs wrote:
 I decided to abandon sanity. Luckily, only for the named parameters. I
 now have this code working:


 void test(int a, int b = 2, string c = "foo", int d = 14) {
 }

 alias test2 = nameify!test;

 void main() {
      test2(1, Args.d = 4, Args.c = "bar");
 }

 With reasonable error messages for duplicate and missing parameters, as
 well as type mismatches.
 ...
Still missing overload resolution. :o)
Nov 17 2013
parent =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On 17.11.2013 23:21, Timon Gehr wrote:
 On 11/17/2013 08:41 PM, Simen Kjærås wrote:
 I decided to abandon sanity. Luckily, only for the named parameters. I
 now have this code working:


 void test(int a, int b = 2, string c = "foo", int d = 14) {
 }

 alias test2 = nameify!test;

 void main() {
      test2(1, Args.d = 4, Args.c = "bar");
 }

 With reasonable error messages for duplicate and missing parameters, as
 well as type mismatches.
 ...
Still missing overload resolution. :o)
Please don't tempt me. :p -- Simen
Nov 17 2013
prev sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Nov 17, 2013 at 8:41 PM, Simen Kjærås <simen.kjaras gmail.com> wrote:

 Source is attached. I hope God forgives me.
Nice ideas. I particularly like this one: final abstract class Args { property static auto opDispatch(string name, Arg)(Arg arg) { return NamedArg!(name, Arg)(arg); } } Which allows you to have Args.c = 1 (and thus storing the name and the value). Half of your implementation is in fact generic enough to be in Phobos (staticFilter, staticZip). I know I use my own... Btw, with DMD 2.064.2, we can now write: template staticToString(T...) { enum staticToString = T.stringof; } as: alias staticToString(T...) = T.stringof; Which is both easier to understand and easier on the eyes.
Nov 17 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-17 21:04, Philippe Sigaud wrote:

 alias staticToString(T...) = T.stringof;

 Which is both easier to understand and easier on the eyes.
Wouldn't that be enum instead: enum staticToString(T...) = T.stringof; -- /Jacob Carlborg
Nov 18 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-12 17:14, John Colvin wrote:

 oh, I see. Would AST macros really be enough to make this work in D?
 "Arbitrary code" is a huge feature space in D, including much that
 doesn't map well to anything outside of a relatively low-level language,
 let alone SQL.
 I can see it quickly becoming a nightmare that would be worse than just
 issuing the predicate as an sql string or some generic equivalent.
Person.where(e => e.name == "John") I'm thinking that we only need to convert the part that is prefixed with, in this example, "e". Any other code should be executed in the context of the caller. It should be possible to do this as well: auto foo = "John"; auto result = Person.where(e => e.name == foo); Which will result in the same SQL query. I'm using a pluign to Ruby on Rails that does something similar but by overloading operators. The problem with this approach, in Ruby, is that you cannot overload operators like || and &&, so instead they overload | and & resulting in new problems like operator precedence. Example: Person.where{ |e| (e.name == "John") & (e.address == "Main street") } -- /Jacob Carlborg
Nov 13 2013
next sibling parent reply luka8088 <luka8088 owave.net> writes:
On 13.11.2013. 9:26, Jacob Carlborg wrote:
 On 2013-11-12 17:14, John Colvin wrote:
 
 oh, I see. Would AST macros really be enough to make this work in D?
 "Arbitrary code" is a huge feature space in D, including much that
 doesn't map well to anything outside of a relatively low-level language,
 let alone SQL.
 I can see it quickly becoming a nightmare that would be worse than just
 issuing the predicate as an sql string or some generic equivalent.
Person.where(e => e.name == "John") I'm thinking that we only need to convert the part that is prefixed with, in this example, "e". Any other code should be executed in the context of the caller. It should be possible to do this as well: auto foo = "John"; auto result = Person.where(e => e.name == foo); Which will result in the same SQL query. I'm using a pluign to Ruby on Rails that does something similar but by overloading operators. The problem with this approach, in Ruby, is that you cannot overload operators like || and &&, so instead they overload | and & resulting in new problems like operator precedence. Example: Person.where{ |e| (e.name == "John") & (e.address == "Main street") }
What about something like this? class Person { macro where (Context context, Statement statement) { // ... } } auto foo = "John"; auto result = Person.where(e => e.name == foo); // is replaced by auto foo = "John"; auto result = Person.query("select * from person where person.name = " ~ sqlQuote(foo) ~ ";");
Nov 13 2013
next sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 13.11.2013 09:34, schrieb luka8088:
 What about something like this?

 class Person {

    macro where (Context context, Statement statement) {
      // ...
    }
it is not generic - and that is the biggest goal to reach
Nov 13 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-13 09:34, luka8088 wrote:

 What about something like this?

 class Person {

    macro where (Context context, Statement statement) {
      // ...
    }

 }

 auto foo = "John";
 auto result = Person.where(e => e.name == foo);

 // is replaced by
 auto foo = "John";
 auto result = Person.query("select * from person where person.name = " ~
 sqlQuote(foo) ~ ";");
That's basically what would happen. -- /Jacob Carlborg
Nov 13 2013
parent reply luka8088 <luka8088 owave.net> writes:
On 13.11.2013. 13:26, Jacob Carlborg wrote:
 On 2013-11-13 09:34, luka8088 wrote:
 
 What about something like this?

 class Person {

    macro where (Context context, Statement statement) {
      // ...
    }

 }

 auto foo = "John";
 auto result = Person.where(e => e.name == foo);

 // is replaced by
 auto foo = "John";
 auto result = Person.query("select * from person where person.name = " ~
 sqlQuote(foo) ~ ";");
That's basically what would happen.
May/Should I add such example to wiki? -- Luka
Nov 13 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-13 15:47, luka8088 wrote:

 May/Should I add such example to wiki?
Yes, "should". Sure, add it to the DIP. -- /Jacob Carlborg
Nov 13 2013
prev sibling parent reply "Chris Nicholson-Sauls" <ibisbasenji gmail.com> writes:
On Wednesday, 13 November 2013 at 08:26:52 UTC, Jacob Carlborg 
wrote:
 I'm using a pluign to Ruby on Rails that does something similar 
 but by overloading operators. The problem with this approach, 
 in Ruby, is that you cannot overload operators like || and &&, 
 so instead they overload | and & resulting in new problems like 
 operator precedence. Example:

 Person.where{ |e| (e.name == "John") & (e.address == "Main 
 street") }
Is this in any way better than the basic Ruby where( name: 'John', address: 'Main street' ) ? Or was it just something quick and contrived to show the behavior, and not the utility, of the plugin? Just curious.
Nov 13 2013
parent "Jacob Carlborg" <doob me.com> writes:
On Wednesday, 13 November 2013 at 17:13:11 UTC, Chris 
Nicholson-Sauls wrote:

 Is this in any way better than the basic Ruby

   where( name: 'John', address: 'Main street' )

 ?  Or was it just something quick and contrived to show the 
 behavior, and not the utility, of the plugin?  Just curious.
It was mostly to show what happens when the you have limited operator overloading. It has it advantages over the hash-syntax. You can do much more, not equal, less then, greater then, like and so on. -- /Jacob Carlborg
Nov 13 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/12/13 7:21 AM, Ellery Newcomer wrote:
 the .NET framework can do this because it exposes an api for querying,
 building, and compiling asts.

 D cannot do this because it doesn't. (and I have tried to make it work)
Maybe the problem needs to be reformulated for D. I think an SQL mixin that either stays unchanged (for DB engines) or translates to a D expression (for native D data types) would be doable, nontrivial, interesting, and instantly usable for people who already know SQL without any extra learning. In other words... actually better than Linq. Andrei
Nov 12 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 12 November 2013 at 16:39:18 UTC, Andrei Alexandrescu 
wrote:
 Maybe the problem needs to be reformulated for D. I think an 
 SQL mixin that either stays unchanged (for DB engines) or 
 translates to a D expression (for native D data types) would be 
 doable, nontrivial, interesting, and instantly usable for 
 people who already know SQL without any extra learning. In 
 other words... actually better than Linq.
It does not matter that much if it is macro or mixin. Key feature is AST reflection which allows to augment valid D code with additional semantics when appropriate. It is very similar to how UDA's augment declarative approach, but applied to imperative/procedural instead. We can't do that in D right now. Only way is to import own source file and parse it which does not really sound sane.
Nov 12 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/12/13 8:47 AM, Dicebot wrote:
 On Tuesday, 12 November 2013 at 16:39:18 UTC, Andrei Alexandrescu wrote:
 Maybe the problem needs to be reformulated for D. I think an SQL mixin
 that either stays unchanged (for DB engines) or translates to a D
 expression (for native D data types) would be doable, nontrivial,
 interesting, and instantly usable for people who already know SQL
 without any extra learning. In other words... actually better than Linq.
It does not matter that much if it is macro or mixin. Key feature is AST reflection which allows to augment valid D code with additional semantics when appropriate.
I don't see how AST reflection is needed for generating correct D code from an SQL expression. Dmitry did exactly that with ctRegex. Please illuminate. Andrei
Nov 12 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 12 November 2013 at 16:54:19 UTC, Andrei Alexandrescu 
wrote:
 I don't see how AST reflection is needed for generating correct 
 D code from an SQL expression. Dmitry did exactly that with 
 ctRegex. Please illuminate.
Other way around. Generating arbitrary output (including but not limited to SQL expressions) from D code. As I have already mentioned, it is the very same thing we do now with __traits and UDA's but not limited to only declarations.
Nov 12 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/12/13 9:03 AM, Dicebot wrote:
 On Tuesday, 12 November 2013 at 16:54:19 UTC, Andrei Alexandrescu wrote:
 I don't see how AST reflection is needed for generating correct D code
 from an SQL expression. Dmitry did exactly that with ctRegex. Please
 illuminate.
Other way around. Generating arbitrary output (including but not limited to SQL expressions) from D code.
Yah. That I agree with. That's why I said we should reformulate the problem that LINQ solves.
 As I have already mentioned, it is the
 very same thing we do now with __traits and UDA's but not limited to
 only declarations.
Interesting. Andrei
Nov 12 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 12 November 2013 at 17:20:20 UTC, Andrei Alexandrescu 
wrote:
 Other way around. Generating arbitrary output (including but 
 not limited
 to SQL expressions) from D code.
Yah. That I agree with. That's why I said we should reformulate the problem that LINQ solves.
I had an impression it was exactly the context in which linq was originally mentioned, no idea why discussion has moved from that to expressing DSL's (which is not really a problem in D) :)
 As I have already mentioned, it is the
 very same thing we do now with __traits and UDA's but not 
 limited to
 only declarations.
Interesting.
Reason why I am doing this comparison is that some of the cases why UDA's are so useful applies to AST reflection capabilities too. Most importantly, it allows to have the code which is both valid (and used) D code on its own and used to generate some co-product from it in automatic mode. Similar declarative approach has been used to huge success in vibe.d code (rest, forms) and I see no reasons why it can't be so for imperative one.
Nov 12 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 21:25, Dicebot wrote:

 I had an impression it was exactly the context in which linq was
 originally mentioned, no idea why discussion has moved from that to
 expressing DSL's (which is not really a problem in D) :)
My example is a DSL: Person.where(e => e.name == "John"); -- /Jacob Carlborg
Nov 13 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 12 November 2013 at 17:20:20 UTC, Andrei Alexandrescu 
wrote:
 As I have already mentioned, it is the
 very same thing we do now with __traits and UDA's but not 
 limited to
 only declarations.
Interesting.
The thing I'd like to be able to do it to create a async/await/yield like mechanism, purely as library. generator uint foo() { writeln("generating !"); return 0; return 1; writeln("and the last one"); return 2; } => auto foo() { enum States { S0, S1, S2, } struct Generator { State s; uint current; property front() { return current; } void popFront() { final switch(s) with(State) { case S0: current = 1; s = S1; return; case S1: writeln("and the last one"); current = 2; s = S2; return; case S2: assert(0, "Nothing more in here"); } } property empty() { return s == State.S2; } } auto g = Generator(); writeln("generating !"); g.current = 0; g.state = State.S0; return g; }
Nov 12 2013
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 11/12/2013 09:36 PM, deadalnix wrote:
 The thing I'd like to be able to do it to create a async/await/yield
 like mechanism, purely as library.
That's a prime example for AST macros.
Nov 12 2013
parent "Zsombor Barna" <belzurix sch.bme.hu> writes:
On Tuesday, 12 November 2013 at 23:34:38 UTC, Martin Nowak wrote:
 On 11/12/2013 09:36 PM, deadalnix wrote:
 The thing I'd like to be able to do it to create a 
 async/await/yield
 like mechanism, purely as library.
That's a prime example for AST macros.
First of all: I haven't used macros extensively yet. Its merely a theoretical viewpoint, so don't take the arguments well-established. Macros could be used as in ScalaCL ( I mean the latest version, which is experimental : https://github.com/ochafik/ScalaCL ). I like the idea using the host language ( or a subset of it) to program GPUs, not C, like PyOpenCL does. It reduces the complexity of the problem when you switch target of the algorithm from CPU to GPU and back. Macros could also be used in (re)implementing some of the basic syntax - that's what developers of Nemerle did. Somebody can argue whether it was a good idea or not, but it would be interesting to disable syntax by disabling the macro that transforms it by a single line in the file ( not importing the macro ). ( Could be useful for microcontroller platforms to control the runtime's memory usage by disabling certain features - not compiling them into the executable ) In AOT compilation, there isn't any difference between macros and CTFE functions which do something with AST. That would be a way to implement them - creating a compiler API which could only be used by compile-time functions. Yeah, and I know, it's crippling when code is ambigous. But I also think macros are the best way to define interfaces for modules. Using them can obfuscate code very easily when not enough care is taken handling them and it can make the learning curve much steeper, but there always existed ill-designed API-s and we could still use the good old tools to get the API done :) . Maybe it wouldn't be as good as a macro-ed API would be, but still works. If we choose the right tool, it can lower the complexity of the API. Let's have a look at the feature: pointers. A very powerful, be also dangerous weapon that cal blow up the entire program suddenly. With some techniques used to fend off the problems it can cause, it can be much safer. Rust made a trial with their pointer system. I don't think it is a solution, but ignoring the problem isn't either. The second example: the first cannons blew up, killing people. But time showed technology can evolve. The area of macros - in my opinion - is not explored yet. Only the use of them can reveal the ways to make them better. Theoretical scientists cannot create good solutions for everyday use in the first place. It's a trial-and-error process. By allowing usage of macros only in certain cases, language desginer can make the burden of the coders lighter ( although I like statement macros, they can be very ambigous ). Also, if designer make macros visually very distinct from other tokens, users can read the code even easier, but sense is needed when choosing a syntax which fits into the language. ( #-s are very alien from D, in my opinion ). Personally I wouldn't like hacking on the compiler or creating a whole new parser to do the same as we could do with macros, and I suggest a trial. Another option is to wait until somebody comes up with an idea and when time proved if the solution is right, include it - doing this saves a lot of work. That's how cutting-edge technologies work ( even if AST macros are an old idea ).
Nov 12 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 21:36, deadalnix wrote:

 The thing I'd like to be able to do it to create a async/await/yield
 like mechanism, purely as library.
Please add this as an example to the DIP. -- /Jacob Carlborg
Nov 13 2013
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-Nov-2013 21:03, Dicebot пишет:
 On Tuesday, 12 November 2013 at 16:54:19 UTC, Andrei Alexandrescu wrote:
 I don't see how AST reflection is needed for generating correct D code
 from an SQL expression. Dmitry did exactly that with ctRegex. Please
 illuminate.
Other way around. Generating arbitrary output (including but not limited to SQL expressions) from D code. As I have already mentioned, it is the very same thing we do now with __traits and UDA's but not limited to only declarations.
Actually I couldn't shake off the feeling that macros are just CTFE functions on Ast objects. How objects are created and converted back to source code is a separate question. If we just had: //this would invoke compiler's parser at CTFE auto ast = "x = y;".astof and have it work at CTFE to return sensible AST (a big if btw). And then (after some manipulations): ast.toString() //get back a string of D code It may help code generation of D --> DSL. Alternatively one can implement D parser that works at CTFE and watch it crawl :) -- Dmitry Olshansky
Nov 12 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 12 November 2013 at 20:56:34 UTC, Dmitry Olshansky 
wrote:
 If we just had:

 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof

 and have it work at CTFE to return sensible AST (a big if btw).

 And then (after some manipulations):
 ast.toString() //get back a string of D code

 It may help code generation of D --> DSL.
Add there "foo.codeof" in the toolset and it will pretty much give core needed stuff. However, it will destroy compilation times if not coupled with compiler internal parsing/semantic phase. Actual "macro" keyword syntax sugar is hardly important to me.
Nov 12 2013
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
13-Nov-2013 01:16, Dicebot пишет:
 On Tuesday, 12 November 2013 at 20:56:34 UTC, Dmitry Olshansky wrote:
 If we just had:

 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof

 and have it work at CTFE to return sensible AST (a big if btw).

 And then (after some manipulations):
 ast.toString() //get back a string of D code

 It may help code generation of D --> DSL.
Add there "foo.codeof" in the toolset and it will pretty much give core needed stuff. However, it will destroy compilation times if not coupled with compiler internal parsing/semantic phase.
For me it feels a lot like implementing something as a soft-core or emulator in software vs producing a bare-metal chip. It's a good idea to test waters and prototype a soft-thingy before getting dirty. When it comes to manufacture ASICs the idea should be well tested and could be actually sold as is already ;) If we consider that compiler is sort of "hardware" accelerator for getting ast's of strings and functions alike. Ditto with toString of ast. Then if somebody comes up with soft-core implementation of "macros" and shows: a) Cool benefits of manipulating on ASTs compared to cracking some strings b) Limitations of this implementation (unhyginic is one) c) Quantify obvious performance problems d) Prepare a solid file of use cases Then we might get people to manufacture ASICs (pay costs, implement in compiler/spec). Paraphrasing somebody from C++ ISO committee (I think Herb): Boost C++ Lambda is a prime reason we had to add lambdas as a core feature in C++11. If it took these top-notch experts that much of coding to get only this far and it's still brittle, limited and etc. ... we have no choice but to add it to the language.
 Actual "macro" keyword syntax sugar is hardly important to me.
-- Dmitry Olshansky
Nov 13 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On 11/12/2013 10:16 PM, Dicebot wrote:
 Add there "foo.codeof" in the toolset and it will pretty much give core
 needed stuff.
Well, codeof is nice but it lacks the parser.
Nov 13 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-13 23:29, Martin Nowak wrote:

 Well, codeof is nice but it lacks the parser.
Yes, and the you need to mixin it again. Several unnecessary steps, see the bottom of: http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com?page=13#post-l5vcct:242lit:241:40digitalmars.com -- /Jacob Carlborg
Nov 13 2013
prev sibling next sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Tue, Nov 12, 2013 at 9:55 PM, Dmitry Olshansky <dmitry.olsh gmail.com> wrote:

 If we just had:

 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof

 and have it work at CTFE to return sensible AST (a big if btw).

 And then (after some manipulations):
 ast.toString() //get back a string of D code
Which you still have to mixin, btw. Which means any tree manipulation you do must be done at compile-time. Unless, of course, you then put the resulting code into another file, to be compiled and loaded afterwards.
 It may help code generation of D --> DSL.

 Alternatively one can implement D parser that works at CTFE and watch it crawl
:)
*cough cough* But, as other have said, these would be very un-hygienic macros, with no knowledge of the surrounding environment. Unless said CT-compatible parser could also somehow determine the local symbols (you could pass it 'by hand', but who would do that, realistically?)
Nov 12 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 22:55, Philippe Sigaud wrote:

 Which you still have to mixin, btw. Which means any tree manipulation
 you do must be done at compile-time.
 Unless, of course,  you then put the resulting code into another file,
 to be compiled and loaded afterwards.
It would be at compile time with macros anyway. The DIP even says that the macro functions are removed in the code generating phase.
 *cough cough*

 But, as other have said, these would be very un-hygienic macros, with
 no knowledge of the surrounding environment.
 Unless said CT-compatible parser could also somehow determine the
 local symbols (you could pass it 'by hand', but who would do that,
 realistically?)
Exactly. -- /Jacob Carlborg
Nov 13 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 21:55, Dmitry Olshansky wrote:

 Actually I couldn't shake off the feeling that macros are just CTFE
 functions on Ast objects. How objects are created and converted back to
 source code is a separate question.
Yes, and reflection.
 If we just had:

 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof

 and have it work at CTFE to return sensible AST (a big if btw).

 And then (after some manipulations):
 ast.toString() //get back a string of D code

 It may help code generation of D --> DSL.

 Alternatively one can implement D parser that works at CTFE and watch it
 crawl :)
You would still need to do strign mixin. Which results in a some unnecessary conversion between AST's and strings. See the list at the bottom of: http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com?page=13#post-l5vcct:242lit:241:40digitalmars.com It also looks ugly. -- /Jacob Carlborg
Nov 13 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof
 ast.toString() //get back a string of D code
Right, that's the basic functionality we want. What's not clear is how to get the type of y.
Nov 13 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-13 23:29, Martin Nowak wrote:

 Right, that's the basic functionality we want.
 What's not clear is how to get the type of y.
That's what the reflection API is for. I guess many people here know a lot more about building AST's than me. But here's a simple API: auto ast = "x = y".astof; auto type = ast.right.type; auto name = type.name; assert(ast.left.type == type); -- /Jacob Carlborg
Nov 14 2013
prev sibling next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 12.11.2013 17:39, schrieb Andrei Alexandrescu:
 Maybe the problem needs to be reformulated for D. I think an SQL mixin
 that either stays unchanged (for DB engines) or translates to a D
 expression (for native D data types) would be doable, nontrivial,
 interesting, and instantly usable for people who already know SQL
 without any extra learning. In other words... actually better than Linq.
"actually better than Linq" good statement without knowing it deep linq allows construction and querieng of non table-like hierarchical data, so its more an object-hierarchy store/retrival system which can "also" work with the lower just-tables-like world of sql results
Nov 12 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/12/13 8:56 AM, dennis luehring wrote:
 Am 12.11.2013 17:39, schrieb Andrei Alexandrescu:
 Maybe the problem needs to be reformulated for D. I think an SQL mixin
 that either stays unchanged (for DB engines) or translates to a D
 expression (for native D data types) would be doable, nontrivial,
 interesting, and instantly usable for people who already know SQL
 without any extra learning. In other words... actually better than Linq.
"actually better than Linq" good statement without knowing it deep linq allows construction and querieng of non table-like hierarchical data, so its more an object-hierarchy store/retrival system which can "also" work with the lower just-tables-like world of sql results
Yah, it was an exaggeration. Linq is very powerful. Andrei
Nov 12 2013
prev sibling parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Tuesday, 12 November 2013 at 16:39:18 UTC, Andrei Alexandrescu 
wrote:
 On 11/12/13 7:21 AM, Ellery Newcomer wrote:
 the .NET framework can do this because it exposes an api for 
 querying,
 building, and compiling asts.

 D cannot do this because it doesn't. (and I have tried to make 
 it work)
Maybe the problem needs to be reformulated for D. I think an SQL mixin that either stays unchanged (for DB engines) or translates to a D expression (for native D data types) would be doable, nontrivial, interesting, and instantly usable for people who already know SQL without any extra learning. In other words... actually better than Linq. Andrei
I was using linq as an example of what we could do with this DIP or what it should be able to do. It wasn't a suggestion of how we should implement such a library. I also identified with it that having scoped variables from the initiation call is required for safety as well as pragmas for error. Later Jacob alerted me to warning as well. Will submit pull once warning is done. I also suggested a generic lexer with rules to make it well defined. With it also if's like templates have for macro over loading but nobody was keen for this hence not on wiki. Yes this can be done already in theory however it doesnt make it not a valid use case.
Nov 12 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 15:38, John Colvin wrote:

 for those of us entirely unfamiliar with linq, what is this supposed to
 do? Select people with name "John" from a collection of people, like in
 sql? It seems trivial to do this using filter, or am I missing
 something...?
The idea of that example is, as Dicebot said, to convert the AST of the lambda to a string containing SQL code then query the database and return the result. The interesting part here, in the context of macros, is to convert the lambda to an SQL string. The filter is not applied on the result, it's converted to SQL and performed in the database. -- /Jacob Carlborg
Nov 13 2013
prev sibling parent reply "Regan Heath" <regan netmail.co.nz> writes:
On Tue, 12 Nov 2013 14:38:43 -0000, John Colvin  
<john.loughran.colvin gmail.com> wrote:

 On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg wrote:
 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'
for those of us entirely unfamiliar with linq, what is this supposed to do? Select people with name "John" from a collection of people, like in sql? It seems trivial to do this using filter, or am I missing something...?
Linq is often confused with LinqToSQL, the above was a description of what happens in the latter. If Person was an object representing a table from a SQL database, then calling 'where' on it would translate into an IQueryable<Person> object which when iterated upon using foreach would execute the SQL statement shown above and return the resulting rows as <someperson> objects one by one to the foreach body. Pure Linq is a DSL, an alternate syntax, which looks a lot like SQL, which can translate to SQL but is not limited to SQL. It could translate to any alternate database syntax, or simply access a container supporting the required operations. The motivation is the same as for "foreach", it is a general syntax for accessing a wide range of containers including databases with a common syntax. We could, equally, go the other way, use the IMO nicer composition syntax X.where(..).foo(...).etc and translate that to SQL, without the need to the SQL like DSL inside D. For some LINQ examples, see: http://code.msdn.microsoft.com/101-LINQ-Samples-3fb9811b For a specific where example using a simple container, see "Where - simple 1" R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Nov 13 2013
next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 13.11.2013 10:58, schrieb Regan Heath:
 Linq is often confused with LinqToSQL, the above was a description of what
 happens in the latter.  If Person was an object representing a table from
 a SQL database, then calling 'where' on it would translate into an
 IQueryable<Person> object which when iterated upon using foreach would
 execute the SQL statement shown above and return the resulting rows as
 <someperson> objects one by one to the foreach body.

 Pure Linq is a DSL, an alternate syntax, which looks a lot like SQL, which
 can translate to SQL but is not limited to SQL.  It could translate to any
 alternate database syntax, or simply access a container supporting the
 required operations.
linq allows construction AND querying of non table-like hierarchical data, so its more an (basetypes|object)-hierarchy store/retrival system which can "also" work with the lower just-tables-like world of sql results
Nov 13 2013
parent dennis luehring <dl.soluz gmx.net> writes:
Am 13.11.2013 11:41, schrieb dennis luehring:
 Am 13.11.2013 10:58, schrieb Regan Heath:
 Linq is often confused with LinqToSQL, the above was a description of what
 happens in the latter.  If Person was an object representing a table from
 a SQL database, then calling 'where' on it would translate into an
 IQueryable<Person> object which when iterated upon using foreach would
 execute the SQL statement shown above and return the resulting rows as
 <someperson> objects one by one to the foreach body.

 Pure Linq is a DSL, an alternate syntax, which looks a lot like SQL, which
 can translate to SQL but is not limited to SQL.  It could translate to any
 alternate database syntax, or simply access a container supporting the
 required operations.
linq allows construction AND querying of non table-like hierarchical data, so its more an (basetypes|object)-hierarchy store/retrival system which can "also" work with the lower just-tables-like world of sql results
but linq isn't good in graph operations like neo4j http://www.neo4j.org/learn/cypher which woulds be also very nice as an macro feature for in D reference traversing
Nov 13 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-13 10:58, Regan Heath wrote:

 The motivation is the same as for "foreach", it is a general syntax for
 accessing a wide range of containers including databases with a common
 syntax.  We could, equally, go the other way, use the IMO nicer
 composition syntax X.where(..).foo(...).etc and translate that to SQL,
 without the need to the SQL like DSL inside D.
My original example was not Linq. I tried to make it look as much as ranges as possible. Yes, I'm tinking something like what you're suggesting: Post.where(e => e.title == "Foo" && e.comments.title == "Bar").join(e => e.comments); -- /Jacob Carlborg
Nov 13 2013
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On 11/12/2013 10:20 AM, Walter Bright wrote:
 I confess I have some serious reservations about AST macros in general:

 1. I've seen very heavy use of such macros in macro assemblers. What
 happens is people use it to invent their own (very baroque) language on
 top of the existing assembler one. Anyone trying to read the code has to
 learn this new unique language, and given the limitations of the macro
 capability, it often comes with unfixable bizarre corner cases and
 limitations.
Yes, this is a big issue. I have similar concerns about UDA, but it's easier to put the most needed UDAs into phobos making them more standardized.
 4. AST macros, pretty much by definition, manipulate the AST. However,
 if you examine the semantic routines in DMD, a *lot* more is required to
 do more significant things. The symbol table must be consulted, etc. I
 just don't see how one could hope to implement something like function
 overloading with AST macros.  Or template type deduction. Or closure
 semantics, attribute inference, etc. Sure, some forms of foreach can be
 rewritten into for statements by manipulating the AST, but not the more
 interesting forms, such as the ones that deal with opApply().
That's the question we'd have to answer. I think what we should target for is that constructs like async/await or SIMDifying of code can be implemented using AST macros.
 For example, the Linq example in the DIP is not compelling, as
 aesthetically nicer code can be written using D's ranges and algorithms:

      auto data = arr.filter!(x => x > 5).array;

 I see no compelling advantage in trying to make D code look like C#; to
 be blunt it's like the old:

      #define BEGIN {
      #define END }

 macros used in old C code to make it look like Pascal.
True
Nov 12 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-12 10:20, Walter Bright wrote:

 I confess I have some serious reservations about AST macros in general:

 1. I've seen very heavy use of such macros in macro assemblers. What
 happens is people use it to invent their own (very baroque) language on
 top of the existing assembler one. Anyone trying to read the code has to
 learn this new unique language, and given the limitations of the macro
 capability, it often comes with unfixable bizarre corner cases and
 limitations.
There's a word for that, it's called DSL (Domain Specific Language) :)
 This got so bad that in some cases I know of, the poor sap who is given
 the job of making improvements resorted to running the assembler to
 generate an object file, then disassembling the object file back into
 source code! Something went very wrong for this to be necessary. I know
 in my own assembler work I have abandoned all use of macros.
DSL's are so popular so many languages are specifically designed to make it easy to create DLS's.
 (Note that macros in typical assemblers are far more powerful than C's
 macro language, which stands out for being pathetically underpowered.)

 2. Macros are something that should be used very sparingly. However,
 this is not what happens. I used to be perplexed that even top shelf
 C/C++ programmers tend to write what I not so humbly consider to be
 pretty abusive use of macros. Now it just saddens me.
It depends on what kind of macros we're talking about. For C preprocessor macros, absolutely. But just because to different features have the same word "macro", you shouldn't put the in the same box. I guess I shouldn't have used the word "macro" at all in the DIP.
 3. Lisp is a language that encourages users to write macros to pretty
 much invent a custom language for the task at hand. Heavy use of such
 makes the code unrecognizable as being Lisp code. I believe a language
 ought to have some "anchors" that the person reading the code can
 reliably recognize. (GO has taken this idea pretty far.)
Lisp doesn't have any syntax to begin with.
 4. AST macros, pretty much by definition, manipulate the AST. However,
 if you examine the semantic routines in DMD, a *lot* more is required to
 do more significant things. The symbol table must be consulted, etc. I
 just don't see how one could hope to implement something like function
 overloading with AST macros. Or template type deduction. Or closure
 semantics, attribute inference, etc. Sure, some forms of foreach can be
 rewritten into for statements by manipulating the AST, but not the more
 interesting forms, such as the ones that deal with opApply().
Macros are not for implementing function overloading or template type deduction, we already have those features. I don't consider these easy to implement and they're not lowered in the same way as "scope" or "foreach" are.
 There are some statements in the DIP about the Context parameter and
 it'll provide semantic information, but I don't see how this can work
 without making it an ever-expanding collection of arbitrary methods.
We'll just put in what we want and need there. At the worst case you'll get access to the whole compiler. Which is what we want any way, at least the front end, to be usable as a library.
 5. We've said "no" in the past to things like user-defined tokens, which
 are closely related.
No, they're not.
 6. Note that D has a limited form of AST macros with mixin templates.
Yes, I do know. Note that they are too limiting compared to AST macros.
 To sum up, any AST macro proposal has some tough barriers to get over.
 Within its necessarily severe semantic limitations, it has to deliver a
 pretty compelling set of solutions, while avoiding the morass of
 incomprehensibility that far too many macro systems become in practice.
Note that AST macros (according to the DIP) is not anything like C preprocessor macros. Basically the only thing they have in common is the name, "macro". It's not a valid comparison. C macros are textual macros. They can create completely new syntax. AST macros cannot. They work at the semantic level in the compiler, just as templates. They can add new semantic meaning to existing syntax. Just like templates, macros need to lex and parse correctly. Templates do not need to be semantically correct if they're not instantiated. -- /Jacob Carlborg
Nov 13 2013
prev sibling parent "Dejan Lekic" <dejan.lekic gmail.com> writes:
 1. I've seen very heavy use of such macros in macro assemblers. 
 What happens is people use it to invent their own (very 
 baroque) language on top of the existing assembler one. Anyone 
 trying to read the code has to learn this new unique language, 
 and given the limitations of the macro capability, it often 
 comes with unfixable bizarre corner cases and limitations.
Metaprogramming (almost) does the same. At least that is my impression. Sometimes I have to go through hundreds of lines of templated D code to understand what a single line does...
Nov 14 2013
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On 11/10/2013 10:20 PM, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:

 http://wiki.dlang.org/DIP50
Thanks, looks like an insightful discussion.
Nov 12 2013
prev sibling parent reply luka8088 <luka8088 owave.net> writes:
On 10.11.2013. 22:20, Jacob Carlborg wrote:
 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:
 
 http://wiki.dlang.org/DIP50
 
I took a look at it as here is my conclusion for now: Statement and attribute macro examples look great. But I don't like Linq example. I don't think code like the following should be allowed. query { from element in array where element > 2 add element to data } From my point of view this whole idea is great as it makes it easier what is already possible. For example, with current behavior if I wanted to write. foo { writeln("foo"); writeln("foo again"); } I would have to write: mixin(foo!(q{ writeln("foo"); writeln("foo again"); })); So the proposed behavior looks much nicer, and I agree with it as the content of foo block is actually written in D and I think whoever is reading it would be comfortable with it. However, for other, non-D syntax-es I would prefer something like: query q{ from element in array where element > 2 add element to data } Which can be handled by: macro query (Context context, string dsl) { return domainSpecificLanguageToD(dsl); } This in terms is already possible by writing the following, it only allows to be written in a more readable way. And the q{ ... } notation clearly points out that there is something special going on. Also by passing such content as string user can implement custom (or call one of the predefined) tokenizer/lexer/parser. mixin(query!(q{ from element in array where element > 2 add element to data })); I also don't like the <[ ... ]> syntax because: 1. Like others have said, it looks very foreign. 2. I don't think there is a need to add a new syntax. I think that string concatenation is enough (at least for now), and if you want another syntax for templates you can write a macro for that. For example: macro myAssert (Context context, Ast!(bool) val, Ast!(string) str = null) { auto message = str ? "Assertion failure: " ~ str.eval : val.toString(); auto msgExpr = literal(constant(message)); return " if (!" ~ val ~ ") throw new AssertError(" ~ msgExpr ~ "); "; // or return astTemplate q{ if (!$val) throw new AssertError($msgExpr); }; } void main () { myAssert(1 + 2 == 4); } What do you guys think? -- Luka
Nov 13 2013
next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
 From my point of view this whole idea is great as it makes it 
 easier
 what is already possible. For example, with current behavior if 
 I wanted
 to write.

 foo {
   writeln("foo");
   writeln("foo again");
 }
Currently this would be sent as a string. I have suggested previously (given a lexer implementation) a lexer would be included that can work by line and symbol. This is fine but it would mean the lexer would need to parse it before calling the macro itself. Perhaps it should be explored a forced parsing and validation of macro values?
 I would have to write:

 mixin(foo!(q{
   writeln("foo");
   writeln("foo again");
 }));
AST macros currently suggested do not work at the same stage as mixins, as far as I am aware. They work at a much earlier stage. Mixins have limits because of there lateness in lexing process.
 So the proposed behavior looks much nicer, and I agree with it 
 as the
 content of foo block is actually written in D and I think 
 whoever is
 reading it would be comfortable with it.


 However, for other, non-D syntax-es I would prefer something 
 like:

 query q{
   from element in array
   where element > 2
   add element to data
 }
I find it intriguing to add a keyword to distinguish a D code block from a string. As currently they are parsed the same way.
 Which can be handled by:

 macro query (Context context, string dsl) {
     return domainSpecificLanguageToD(dsl);
 }

 This in terms is already possible by writing the following, it 
 only
 allows to be written in a more readable way. And the q{ ... } 
 notation
 clearly points out that there is something special going on. 
 Also by
 passing such content as string user can implement custom (or 
 call one of
 the predefined) tokenizer/lexer/parser.

 mixin(query!(q{
   from element in array
   where element > 2
   add element to data
 }));
Perhaps a different way of identifying such code is required? query "{ from element in array where element > 2 add element to data } Although it would act as a triple quoted string, except only valid for macros maybe? This would signify that the following block of text was to be pushed into a macro. Otherwise it would be assumed as valid D code? Assuming a lexer is available it could check that any code given to a macro is correct. I will be submitting a pull request for pragma error and warning so if it is not valid the macro could fire a compile error.
Nov 13 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-13 12:49, Rikki Cattermole wrote:

 AST macros currently suggested do not work at the same stage as mixins,
 as far as I am aware. They work at a much earlier stage. Mixins have
 limits because of there lateness in lexing process.
I think macros would work on the same stage as mixins.
 Assuming a lexer is available it could check that any code given to a
 macro is correct. I will be submitting a pull request for pragma error
 and warning so if it is not valid the macro could fire a compile error.
All macro invocations are already checked. They're lexed and parsed. Marcos work at the semantic phase. Then when the macro is returned the resulting AST is typed checked as usual. -- /Jacob Carlborg
Nov 13 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-13 09:15, luka8088 wrote:

 I took a look at it as here is my conclusion for now:

 Statement and attribute macro examples look great. But I don't like Linq
 example. I don't think code like the following should be allowed.
I'm not a fan of the Linq example either. Or rather if I put it like this. I think the Linq example is quite a good example what can be done with AST macros. But I don't think it's a good fit in D.
 query {
    from element in array
    where element > 2
    add element to data
 }


  From my point of view this whole idea is great as it makes it easier
 what is already possible. For example, with current behavior if I wanted
 to write.

 foo {
    writeln("foo");
    writeln("foo again");
 }

 I would have to write:

 mixin(foo!(q{
    writeln("foo");
    writeln("foo again");
 }));

 So the proposed behavior looks much nicer, and I agree with it as the
 content of foo block is actually written in D and I think whoever is
 reading it would be comfortable with it.


 However, for other, non-D syntax-es I would prefer something like:
Well, the macros in the DIP only supports the D syntax. The macros work at the semantic level. You cannot create new syntax but you can give existing syntax new meaning. Compared with current D, macros need to lex and parse correctly. But they don't need to be semantically correct. That's the same thing as with templates, they can be semantically invalid as long as they're not instantiated. The Linq example is lexically correct. I'm not sure if the parser will reject it or not.
 query q{
    from element in array
    where element > 2
    add element to data
 }

 Which can be handled by:

 macro query (Context context, string dsl) {
      return domainSpecificLanguageToD(dsl);
 }

 This in terms is already possible by writing the following, it only
 allows to be written in a more readable way. And the q{ ... } notation
 clearly points out that there is something special going on. Also by
 passing such content as string user can implement custom (or call one of
 the predefined) tokenizer/lexer/parser.

 mixin(query!(q{
    from element in array
    where element > 2
    add element to data
 }));


 I also don't like the <[ ... ]> syntax because:
 1. Like others have said, it looks very foreign.
 2. I don't think there is a need to add a new syntax.
As it says in the DIP, this is one suggestion of the syntax, if new syntax is needed at all. It's just easier to show examples like that than building a complete AST with function calls. As you can see I have an alternative that doesn't require new syntax: http://wiki.dlang.org/DIP50#The_AST_Macro Instead of "<[ a + b ]>" you would do "ast(a + b)" or "ast { a + b }".
 I think that string concatenation is enough (at least for now), and if
 you want another syntax for templates you can write a macro for that.
Strings are far from enough. Then you have missed the whole idea. It's not supposed to be syntax sugar for string mixins. -- /Jacob Carlborg
Nov 13 2013
parent reply luka8088 <luka8088 owave.net> writes:
On 13.11.2013. 13:47, Jacob Carlborg wrote:
 On 2013-11-13 09:15, luka8088 wrote:
 I think that string concatenation is enough (at least for now), and if
 you want another syntax for templates you can write a macro for that.
Strings are far from enough. Then you have missed the whole idea. It's not supposed to be syntax sugar for string mixins.
Oh, I see. It seems that I indeed missed the point. It seems to me that this DIP could to be granulated into: AST reflection, AST manipulation and AST template. The reason for this, as far as I see, is that although they overlap in some cases, in other they could be used independently and it could help with understanding. Regarding AST reflection and AST manipulation, I have been thinking and found a few situations that bugs me: In the example of auto person = Person.where(e => e.name == "John"); what would be the response of: auto f = e => e.name == "John"; auto person = Person.where(f); I guess it should be a compiler error because f could be modified at runtime and f's body could be hidden. So basically AST macros are something that look like D but actually are not D. This seems to me like an example that look good as a toy example but fails on the larger scale so I agree with Water in this matter (unless better examples/explanations are provided). Or maybe I am not seeing it clear enough. But! Regarding AST templates, I think something like the following would be a great syntax sugar: int i = 5; Ast a = t{ // t as template int j = $i; int k = $i + $i; }; // ... alter ast here if necessary mixin(a); Where syntax (and semantic) in template must be strictly D. The only difference would be $ sign which would allow reference to symbols outside the template. So if you would want to have conditional template you could write: int i = 5; int j; bool b = false; mixin(t{ $j = $i; static if ($b) { $j++; } });
Nov 19 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-19 19:32, luka8088 wrote:

 Oh, I see. It seems that I indeed missed the point.

 It seems to me that this DIP could to be granulated into: AST
 reflection, AST manipulation and AST template. The reason for this, as
 far as I see, is that although they overlap in some cases, in other they
 could be used independently and it could help with understanding.

 Regarding AST reflection and AST manipulation, I have been thinking and
 found a few situations that bugs me:

 In the example of

 auto person = Person.where(e => e.name == "John");

 what would be the response of:

 auto f = e => e.name == "John";
 auto person = Person.where(f);

 I guess it should be a compiler error because f could be modified at
 runtime and f's body could be hidden.
I haven't thought about that. But AST macros work at compile time
 So basically AST macros are
 something that look like D but actually are not D. This seems to me like
 an example that look good as a toy example but fails on the larger scale
 so I agree with Water in this matter (unless better
 examples/explanations are provided). Or maybe I am not seeing it clear
 enough.

 But! Regarding AST templates, I think something like the following would
 be a great syntax sugar:

 int i = 5;

 Ast a = t{ // t as template
    int j = $i;
    int k = $i + $i;
 };

 // ... alter ast here if necessary

 mixin(a);
So you don't like that it's not a "mixin" there with AST macros?
 Where syntax (and semantic) in template must be strictly D. The only
 difference would be $ sign which would allow reference to symbols
 outside the template.
If the semantics need to be valid this is very limited. Even more limiting than what we have now with string mixins. Note that with AST macros, the input needs to syntactically valid. The resulting AST of the macro needs to both be syntactically and semantically valid D code. -- /Jacob Carlborg
Nov 19 2013
parent reply luka8088 <luka8088 owave.net> writes:
On 19.11.2013. 21:32, Jacob Carlborg wrote:
 On 2013-11-19 19:32, luka8088 wrote:
 
 Oh, I see. It seems that I indeed missed the point.

 It seems to me that this DIP could to be granulated into: AST
 reflection, AST manipulation and AST template. The reason for this, as
 far as I see, is that although they overlap in some cases, in other they
 could be used independently and it could help with understanding.

 Regarding AST reflection and AST manipulation, I have been thinking and
 found a few situations that bugs me:

 In the example of

 auto person = Person.where(e => e.name == "John");

 what would be the response of:

 auto f = e => e.name == "John";
 auto person = Person.where(f);

 I guess it should be a compiler error because f could be modified at
 runtime and f's body could be hidden.
I haven't thought about that. But AST macros work at compile time
Well, do think about that :) auto f = e => e.name == "John"; if (true) f = e => e.name == "Jack"; auto person = Person.where(f); I can think of a many use cases where conditional query generation is required. But I don't see how this could be done using AST macros.
 
 So basically AST macros are
 something that look like D but actually are not D. This seems to me like
 an example that look good as a toy example but fails on the larger scale
 so I agree with Water in this matter (unless better
 examples/explanations are provided). Or maybe I am not seeing it clear
 enough.

 But! Regarding AST templates, I think something like the following would
 be a great syntax sugar:

 int i = 5;

 Ast a = t{ // t as template
    int j = $i;
    int k = $i + $i;
 };

 // ... alter ast here if necessary

 mixin(a);
So you don't like that it's not a "mixin" there with AST macros?
Um, sorry. I don't understand the question. This example (and suggestion) was suppose to show that we could allow AST mixins as well as string mixins. It should behave like string mixins but the main difference is that AST is structured so it much cleaner to manipulate.
 
 Where syntax (and semantic) in template must be strictly D. The only
 difference would be $ sign which would allow reference to symbols
 outside the template.
If the semantics need to be valid this is very limited. Even more limiting than what we have now with string mixins. Note that with AST macros, the input needs to syntactically valid. The resulting AST of the macro needs to both be syntactically and semantically valid D code.
When I first started programing I was introduced to static typing. Then I discovered dynamic typing and dynamic structures (in php mostly). And for me at that time it was much better and easier to use. However, over time I learned that sometimes limitations are good and now like static typing much more :) My point here is that limits are sometimes good, although that don't seem that way. I think that we should think of a more complex (real world) examples and than real issues will reveal themselves.
Nov 19 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-19 21:54, luka8088 wrote:

 Well, do think about that :)

 auto f = e => e.name == "John";
 if (true)
    f = e => e.name == "Jack";

 auto person = Person.where(f);

 I can think of a many use cases where conditional query generation is
 required. But I don't see how this could be done using AST macros.
Using the Rails plugin, which I've got this idea from, I would not make the lambda conditional but the whole statement, translated to D: auto person = Person.scoped(); if (true) person = person.where(e => e.name == "Jack"); else person = person.where(e => e.name == "John");
 Um, sorry. I don't understand the question.

 This example (and suggestion) was suppose to show that we could allow
 AST mixins as well as string mixins. It should behave like string mixins
 but the main difference is that AST is structured so it much cleaner to
 manipulate.
What I was trying to say is, what don't you like about my suggestion. Is it that the "mixin" keyword isn't used. -- /Jacob Carlborg
Nov 20 2013
parent reply luka8088 <luka8088 owave.net> writes:
On 20.11.2013. 9:04, Jacob Carlborg wrote:
 On 2013-11-19 21:54, luka8088 wrote:
 
 Well, do think about that :)

 auto f = e => e.name == "John";
 if (true)
    f = e => e.name == "Jack";

 auto person = Person.where(f);

 I can think of a many use cases where conditional query generation is
 required. But I don't see how this could be done using AST macros.
Using the Rails plugin, which I've got this idea from, I would not make the lambda conditional but the whole statement, translated to D: auto person = Person.scoped(); if (true) person = person.where(e => e.name == "Jack"); else person = person.where(e => e.name == "John");
Ok, I see. Yes, I tried to think of a case where this could not work but I was unable to find one.
 
 Um, sorry. I don't understand the question.

 This example (and suggestion) was suppose to show that we could allow
 AST mixins as well as string mixins. It should behave like string mixins
 but the main difference is that AST is structured so it much cleaner to
 manipulate.
What I was trying to say is, what don't you like about my suggestion. Is it that the "mixin" keyword isn't used.
If I understood you correctly, the issue with current way DSLs are implemented is that code needs to be parsed two times. First time DSL author parses it and creates D code from it, and second time D compiler parses that D code and compiles it. What I would suggest in this case is that instead of intercepting the compiler and "fixing" the semantics before it is verified we allow the user to build D Ast and give it to the compiler. That is why I used the mixin in my example. Ast dCodeAst = dslToD!(q{ if true do() else dont() dont2() whatever() }); // manipulate dCodeAst mixin(dCodeAst); The point of this example is that mixin() accepts Ast instead of string. That way, we parse our DSL to D Ast and give it straight to the compiler and everything is done only once!
Nov 20 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-20 23:25, luka8088 wrote:

 If I understood you correctly, the issue with current way DSLs are
 implemented is that code needs to be parsed two times. First time DSL
 author parses it and creates D code from it, and second time D compiler
 parses that D code and compiles it. What I would suggest in this case is
 that instead of intercepting the compiler and "fixing" the semantics
 before it is verified we allow the user to build D Ast and give it to
 the compiler. That is why I used the mixin in my example.

 Ast dCodeAst = dslToD!(q{
    if true
      do()
    else
      dont()
      dont2()
    whatever()
 });

 // manipulate dCodeAst

 mixin(dCodeAst);

 The point of this example is that mixin() accepts Ast instead of string.
 That way, we parse our DSL to D Ast and give it straight to the compiler
 and everything is done only once!
So how is this different except for my proposal, except the use of q{} and "mixin"? -- /Jacob Carlborg
Nov 20 2013
next sibling parent reply luka8088 <luka8088 owave.net> writes:
On 21.11.2013. 8:28, Jacob Carlborg wrote:
 On 2013-11-20 23:25, luka8088 wrote:
 
 If I understood you correctly, the issue with current way DSLs are
 implemented is that code needs to be parsed two times. First time DSL
 author parses it and creates D code from it, and second time D compiler
 parses that D code and compiles it. What I would suggest in this case is
 that instead of intercepting the compiler and "fixing" the semantics
 before it is verified we allow the user to build D Ast and give it to
 the compiler. That is why I used the mixin in my example.

 Ast dCodeAst = dslToD!(q{
    if true
      do()
    else
      dont()
      dont2()
    whatever()
 });

 // manipulate dCodeAst

 mixin(dCodeAst);

 The point of this example is that mixin() accepts Ast instead of string.
 That way, we parse our DSL to D Ast and give it straight to the compiler
 and everything is done only once!
So how is this different except for my proposal, except the use of q{} and "mixin"?
When using q{} compiler treats the contents as a regular string, and you have to parse it and give it to the compiler using "mixin". So basically you can say to the compiler: this part of code is my DSL and I will parse it and check the semantics instead of you, then when I am done I will give you D AST represantation of it. The interpretation of the content of q{} is up to you, where in other proposals compiler builds AST based on, I would say, guessing. Take a look at this last example. And lets say that it is from python. In python dont2() belong inside else block. So how else would you make sure that compiler behaves accordingly?
Nov 21 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-21 09:01, luka8088 wrote:

 When using q{} compiler treats the contents as a regular string, and you
 have to parse it and give it to the compiler using "mixin". So basically
 you can say to the compiler: this part of code is my DSL and I will
 parse it and check the semantics instead of you, then when I am done I
 will give you D AST represantation of it.

 The interpretation of the content of q{} is up to you, where in other
 proposals compiler builds AST based on, I would say, guessing.

 Take a look at this last example. And lets say that it is from python.
 In python dont2() belong inside else block. So how else would you make
 sure that compiler behaves accordingly?
Sorry, I meant your original suggestion of using t{}. -- /Jacob Carlborg
Nov 21 2013
parent reply luka8088 <luka8088 owave.net> writes:
On 21.11.2013. 9:31, Jacob Carlborg wrote:
 On 2013-11-21 09:01, luka8088 wrote:
 
 When using q{} compiler treats the contents as a regular string, and you
 have to parse it and give it to the compiler using "mixin". So basically
 you can say to the compiler: this part of code is my DSL and I will
 parse it and check the semantics instead of you, then when I am done I
 will give you D AST represantation of it.

 The interpretation of the content of q{} is up to you, where in other
 proposals compiler builds AST based on, I would say, guessing.

 Take a look at this last example. And lets say that it is from python.
 In python dont2() belong inside else block. So how else would you make
 sure that compiler behaves accordingly?
Sorry, I meant your original suggestion of using t{}.
Um, my it's suppose to be the same as <[ ... ]> but I liked t{ ... } syntax better as it looked more consistent with what D already has. But I should have used <[ ... ]> , my mistake sorry.
Nov 22 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-22 10:27, luka8088 wrote:

 Um, my it's suppose to be the same as <[ ... ]> but I liked t{ ... }
 syntax better as it looked more consistent with what D already has. But
 I should have used <[ ... ]> , my mistake sorry.
I thought you argued that the t{ } need to contain semantically valid code? -- /Jacob Carlborg
Nov 22 2013
parent reply luka8088 <luka8088 owave.net> writes:
On 22.11.2013. 11:17, Jacob Carlborg wrote:
 On 2013-11-22 10:27, luka8088 wrote:
 
 Um, my it's suppose to be the same as <[ ... ]> but I liked t{ ... }
 syntax better as it looked more consistent with what D already has. But
 I should have used <[ ... ]> , my mistake sorry.
I thought you argued that the t{ } need to contain semantically valid code?
Yes. I still do. And I think that <[ ... ]> should contain semantically valid code. In my opinion, if you wish to be able to write anything else, the way to go would be: write it using q{ ... }, parse it yourself, build a valid D AST yourself, give D AST to the compiler using mixin(). Currently mixin() only accepts a string that contains a valid D code. So if you have a custom parser for your DSL you need to generate a D code from that DSL and then pass it to the compiler using mixin() in order for the compiler to parse it again. Double parsing could be skipped if mixin() would accept already built D AST and on the other hand parsing DSL and building D AST yourself would allow maximum flexibility.
Nov 22 2013
parent reply "Jacob Carlborg" <doob me.com> writes:
On Friday, 22 November 2013 at 23:43:26 UTC, luka8088 wrote:

 Yes. I still do. And I think that <[ ... ]> should contain 
 semantically
 valid code. In my opinion, if you wish to be able to write 
 anything
 else, the way to go would be: write it using q{ ... }, parse it
 yourself, build a valid D AST yourself, give D AST to the 
 compiler using
 mixin().
Then q{ } or <[ ]> would be very limited. Not even templets need to contain semantically valid code, as long as it's not instantiated. Usually the idea with AST macros is to take code that is not valid (semantically) and create a meaning for it (make it valid). If it's already valid in the first place why use AST macros at all? Just use the code as is. Hmm, it could be useful to only allow semantically valid code and use macros to add new things to the code. But again, very limiting.
 Currently mixin() only accepts a string that contains a valid D 
 code.
The AST returned from a macro need to be valid D code as well.
 So if you have a custom parser for your DSL you need to 
 generate a D code
 from that DSL and then pass it to the compiler using mixin() in 
 order
 for the compiler to parse it again. Double parsing could be 
 skipped if
 mixin() would accept already built D AST and on the other hand 
 parsing
 DSL and building D AST yourself would allow maximum flexibility.
You can already do that today with string mixins, although you would need to convert the AST back to a string first. -- /Jacob Carlborg
Nov 23 2013
parent luka8088 <luka8088 owave.net> writes:
On 23.11.2013. 11:40, Jacob Carlborg wrote:
 On Friday, 22 November 2013 at 23:43:26 UTC, luka8088 wrote:
 
 Then q{ } or <[ ]> would be very limited. Not even templets need to
 contain semantically valid code, as long as it's not instantiated.
 Usually the idea with AST macros is to take code that is not valid
 (semantically) and create a meaning for it (make it valid). If it's
 already valid in the first place why use AST macros at all? Just use the
 code as is.
 
 Hmm, it could be useful to only allow semantically valid code and use
 macros to add new things to the code. But again, very limiting.
 
I don't think so. What I was proposing is to split the problem into sub-problems and then find the best method for each one. Templates are one of them and they solve the problem of not so pretty string concatenation and that is way they should contain only valid D code. I don't think that writing a non valid code and then making in valid in the compilation process should be allowed. Only transforming one valid code into another valid code but hygienically. I don't have a concrete example why that is a bad idea, I can only speak from experience. So this is only my opinion, not an argument.
 Currently mixin() only accepts a string that contains a valid D code.
The AST returned from a macro need to be valid D code as well.
 So if you have a custom parser for your DSL you need to generate a D code
 from that DSL and then pass it to the compiler using mixin() in order
 for the compiler to parse it again. Double parsing could be skipped if
 mixin() would accept already built D AST and on the other hand parsing
 DSL and building D AST yourself would allow maximum flexibility.
You can already do that today with string mixins, although you would need to convert the AST back to a string first.
Yeah. I was referring to http://forum.dlang.org/post/l5vcct$2lit$1 digitalmars.com So for example, the problem from referred post could be addressed by defining as AST struct type that is accessible to user and making mixin() accept both string and AST struct. That is also why I wrote http://forum.dlang.org/post/l6n91e$29hd$1 digitalmars.com It seems to me now that we don't understand each other so well :) Maybe we should put in more examples?
Nov 23 2013
prev sibling parent reply luka8088 <luka8088 owave.net> writes:
On 21.11.2013. 8:28, Jacob Carlborg wrote:
 On 2013-11-20 23:25, luka8088 wrote:
 The point of this example is that mixin() accepts Ast instead of string.
 That way, we parse our DSL to D Ast and give it straight to the compiler
 and everything is done only once!
So how is this different except for my proposal, except the use of q{} and "mixin"?
What we currently have: custom syntax ---> custom parser ---> custom code generator ---> D syntax (as string) D syntax (as string) ---> mixin --- \ D syntax (read by compiler) -----------> D AST (held by the compiler, unreachable) ---> compilation to binary (unreachable) The way I see it we should allow to following: custom syntax ---> custom parser ----- \ D syntax inside <[ ... ]> template -------> D AST (as struct) D AST (as struct) ---> mixin --- \ D syntax (read by compiler) --------> D AST (held by compiler) D AST (held by compiler, reachable) ---> reflection ---> D AST (as struct) D AST (held by compiler) ---> compilation to binary
Nov 22 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-22 10:44, luka8088 wrote:

 What we currently have:
I should have been t{ }. I do understand the difference between t{ } and q{ }. But not between <[ ]> and t{ }. Is it just two different syntaxes for the same? -- /Jacob Carlborg
Nov 22 2013
parent luka8088 <luka8088 owave.net> writes:
On 22.11.2013. 11:19, Jacob Carlborg wrote:
 On 2013-11-22 10:44, luka8088 wrote:
 
 What we currently have:
I should have been t{ }. I do understand the difference between t{ } and q{ }. But not between <[ ]> and t{ }. Is it just two different syntaxes for the same?
Yes. t{ } and <[ ]> are the same.
Nov 22 2013
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
13-Nov-2013 12:15, luka8088 пишет:
 On 10.11.2013. 22:20, Jacob Carlborg wrote:
[snip]
 foo {
    writeln("foo");
    writeln("foo again");
 }

 I would have to write:

 mixin(foo!(q{
    writeln("foo");
    writeln("foo again");
 }));
Actually even now it's can be just foo!q{ ... }; For instance this is compilable: { import std.regex; auto r = ctRegex!"how do you think that works?"; } See ctRegex template in std.regex. -- Dmitry Olshansky
Nov 13 2013