www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - What are AST Macros?

reply Jonathan M Davis <jmdavisprog gmail.com> writes:
Periodically, the term AST macros come up in the discussions here as a possible 
addition to the language - they even get discussed briefly in TDPL as a
possible 
replacement for template mxins. However, I don't think that I've ever seen an 
actual explanation for what exactly is meant by the term AST macro. This raises 
two questions

1. What are AST macros and how do they differ from C-style macros? We obviously 
aren't going to be adding macros like that to D, since that would be dangerous. 
But what would a macro be then if it's not a textual replacement? The best I
can 
think of would be that you'd have to indicate the macros to be replaced by 
surrounding them with macro() or something rather than letting any and all text 
that matches  be replaced. So, I really don't know what AST macros are supposed 
to be other than they're macros of some kind and better than C-style macros.

2. What does AST stand for? The best that I can come up with for what it could 
stand for would be Abstract Syntax Tree, which is a nice, Computer Sciency, 
compiler-related term, but I haven't a clue how that would relate to macros.
So, 
maybe an answer to the first question would answer this one as well, but an 
explanation would be nice.

- Jonathan M Davis
Jul 08 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Jonathan M Davis" <jmdavisprog gmail.com> wrote in message 
news:mailman.301.1278636124.24349.digitalmars-d puremagic.com...
 Periodically, the term AST macros come up in the discussions here as a 
 possible
 addition to the language - they even get discussed briefly in TDPL as a 
 possible
 replacement for template mxins. However, I don't think that I've ever seen 
 an
 actual explanation for what exactly is meant by the term AST macro. This 
 raises
 two questions

 1. What are AST macros and how do they differ from C-style macros? We 
 obviously
 aren't going to be adding macros like that to D, since that would be 
 dangerous.
 But what would a macro be then if it's not a textual replacement? The best 
 I can
 think of would be that you'd have to indicate the macros to be replaced by
 surrounding them with macro() or something rather than letting any and all 
 text
 that matches  be replaced. So, I really don't know what AST macros are 
 supposed
 to be other than they're macros of some kind and better than C-style 
 macros.

 2. What does AST stand for? The best that I can come up with for what it 
 could
 stand for would be Abstract Syntax Tree, which is a nice, Computer 
 Sciency,
 compiler-related term, but I haven't a clue how that would relate to 
 macros. So,
 maybe an answer to the first question would answer this one as well, but 
 an
 explanation would be nice.

 - Jonathan M Davis

Short answer: Reflection/introspection on steroids. Slightly longer answer: You're right, AST is "Abstract Syntax Tree". As for what that actually means: You know the HTML DOM? If you have the DOM for a specific HTML page, that DOM is a (mutable) representation of the HTML page's AST. If you manipulate the DOM (ie, DHTML), then that's basically AST macros. Except for one thing: HTML is merely markup and D is fully turing-complete. So imagine that the browser *also* has a DOM for the JavaScript itself, so the JavaScript can manipulate itself through a DOM...that's what we're talking about (although it'd be compile-time in our case). So yea...clearly powerful stuff. Purely hypothetical example: Suppose you have this code: module foo; void main(string[] args) { int i; i = 7 + 4 * 3; } The AST might look something roughly like this (I'm representing it here in XML, so I don't have to try to draw an ASCII-art graph): <module name="foo"> <function name="main" type="void"> <params> <param name="args" type="string[]" /> </params> <body> <declaration name="i" type="int" /> <assignment> <lvalue> <variable>i</variable> </lvalue> <rvalue> <add> <literal>7</literal> <multiply> <literal>4</literal> <literal>3</literal> </multiply> </add> </rvalue> </assignment> </body> </function> </module> (Or it might look totally different then that, depending on how the AST system is defined, but you get the idea.) Now imagine an API that exposes a mutable version of that tree (or something similar to it) at compile-time. Also, if you know what a parse tree is, an AST is similar to that, but generally a bit more human-understandable and semantics-oriented, whereas a parse tree is purely a literal representation of the formal syntax - no semantics all, and rather awkward to deal with. Another interesting thing: I've never really used LISP, but my understanding is that this sort of thing basically forms the heart of LISP and is the reason LISP is considered so powerful. (AIUI, LISP was invented purely as a hypothetical example of this exact sort of thing, and then somebody went and actually implemented a real interpreter for it.)
Jul 08 2010
parent "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:i15uqd$1917$1 digitalmars.com...
 You're right, AST is "Abstract Syntax Tree". As for what that actually

Also, I'm very close to a new release of my Goldie language-processing project ( http://www.dsource.org/projects/goldie ). There are some tools in it ("Parse" and a customized version of "JsonViewer") that make it *very* easy to inspect the parse tree of a source file (although not D yet, sadly) and get a feel for how parse trees work. But it also has a fairly simple documentation-generating tool (really just an HTML pre-processor) that uses its own language, and when you use it, it can output the AST of the documentation source. Then you can use that customized "JsonViewer" to get a good feel for the AST, and how that works, and how it compares to the parse tree. You you might want to keep an eye out on D.announce for that. I'm hoping it'll be within the next few weeks, although that depends on real-world work...
Jul 08 2010
prev sibling next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Thu, 08 Jul 2010 20:43:39 -0400, Jonathan M Davis  
<jmdavisprog gmail.com> wrote:

 Periodically, the term AST macros come up in the discussions here as a  
 possible
 addition to the language - they even get discussed briefly in TDPL as a  
 possible
 replacement for template mxins. However, I don't think that I've ever  
 seen an
 actual explanation for what exactly is meant by the term AST macro. This  
 raises
 two questions

 1. What are AST macros and how do they differ from C-style macros? We  
 obviously
 aren't going to be adding macros like that to D, since that would be  
 dangerous.
 But what would a macro be then if it's not a textual replacement? The  
 best I can
 think of would be that you'd have to indicate the macros to be replaced  
 by
 surrounding them with macro() or something rather than letting any and  
 all text
 that matches  be replaced. So, I really don't know what AST macros are  
 supposed
 to be other than they're macros of some kind and better than C-style  
 macros.

 2. What does AST stand for? The best that I can come up with for what it  
 could
 stand for would be Abstract Syntax Tree, which is a nice, Computer  
 Sciency,
 compiler-related term, but I haven't a clue how that would relate to  
 macros. So,
 maybe an answer to the first question would answer this one as well, but  
 an
 explanation would be nice.

 - Jonathan M Davis

Check out Walter's slides and/or talk from the D conference. (http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D.announce&article_id=12555) AST does stand for abstract syntax tree and they are much more like Lisp macros as opposed to the C preprocessor.
Jul 08 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Robert Jacques" <sandford jhu.edu> wrote in message 
news:op.vfjy3jlo26stm6 sandford...
 Check out Walter's slides and/or talk from the D conference. 
 (http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D.announce&article_id=12555)
 AST does stand for abstract syntax tree and they are much more like Lisp 
 macros as opposed to the C preprocessor.

Hmm, looks like the D version is much more template-mixin/C-macro-like than what I made it sound like in a previous branch of the thread. Basically like template mixins, but where the arguments are snippets of code, and overloads are resolved by pattern-matching the syntax of their arguments.
Jul 08 2010
prev sibling parent reply Don <nospam nospam.com> writes:
retard wrote:
 Thu, 08 Jul 2010 21:43:57 -0400, Robert Jacques wrote:
 
 Check out Walter's slides and/or talk from the D conference.
 (http://www.digitalmars.com/webnews/newsgroups.php?

 AST does stand for abstract syntax tree and they are much more like Lisp
 macros as opposed to the C preprocessor.

Too bad nothing happened. He promised AST macros, but it will probably still take several years before we have a preliminary implementation and maybe 10-20 years before a written spec & almsot bug-free implementation.

Part of what happened was that at the conference, I showed that the functionality of string mixins was a superset of what was planned for AST macros. Since that time, there has not been any kind of proposal, or even a concept. So it's a bit meaningless to talk about "AST macros" right now as if they were a planned-but-not-yet-implemented feature.
Jul 11 2010
next sibling parent reply Don <nospam nospam.com> writes:
Philippe Sigaud wrote:
 Don wrote:
 
 
     Part of what happened was that at the conference, I showed that the
     functionality of string mixins was a superset of what was planned
     for AST macros. Since that time, there has not been any kind of
     proposal, or even a concept. So it's a bit meaningless to talk about
     "AST macros" right now as if they were a
     planned-but-not-yet-implemented feature.
 
 
 That's interesting. Do you have a link or any text I could read on that? 

Not really, it was just casual discussion on a whiteboard after the conference. I did post some example code on the newsgroup (announce, I think) which did lexing + parsing + basic semantic analysis in 300 lines of code. It gives an idea of what AST macros would need to be able to do.
 String mixins sure are powerful, but I can't get ird of a feeling of 
 'cheating' when using them. Maybe with some kind of string interpolation 
 they could be made more palatable to some?

There's little doubt that it's the part of the language which needs the most attention in the long term. But the way forward is not at all obvious. So it's been deferred.
Jul 11 2010
parent reply pillsy <pillsbury gmail.com> writes:
== Quote from Don (nospam nospam.com)'s article

 Philippe Sigaud wrote:

 String mixins sure are powerful, but I can't get ird of a feeling of
 'cheating' when using them. Maybe with some kind of string interpolation
 they could be made more palatable to some?


 There's little doubt that it's the part of the language which needs the
 most attention in the long term. But the way forward is not at all
 obvious. So it's been deferred.

I'm not clear on how one is supposed to go about making the equivalent of hygienic macros using just string mixins and CTFE. Cheers, Pillsy
Jul 11 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/11/2010 06:57 PM, retard wrote:
 Sun, 11 Jul 2010 22:03:56 +0000, pillsy wrote:

 == Quote from Don (nospam nospam.com)'s article

 Philippe Sigaud wrote:

 String mixins sure are powerful, but I can't get ird of a feeling of
 'cheating' when using them. Maybe with some kind of string
 interpolation they could be made more palatable to some?


 There's little doubt that it's the part of the language which needs the
 most attention in the long term. But the way forward is not at all
 obvious. So it's been deferred.

I'm not clear on how one is supposed to go about making the equivalent of hygienic macros using just string mixins and CTFE.

You can create a trial-and-error gensym function with compile time introspection and CTFE/templates. :o) It's very elegant. CTFE solves all your AST macro use cases.

Wrong. The correct answer is magic.
Jul 11 2010
parent Don <nospam nospam.com> writes:
Ellery Newcomer wrote:
 On 07/11/2010 06:57 PM, retard wrote:
 Sun, 11 Jul 2010 22:03:56 +0000, pillsy wrote:

 == Quote from Don (nospam nospam.com)'s article

 Philippe Sigaud wrote:

 String mixins sure are powerful, but I can't get ird of a feeling of
 'cheating' when using them. Maybe with some kind of string
 interpolation they could be made more palatable to some?


 There's little doubt that it's the part of the language which needs the
 most attention in the long term. But the way forward is not at all
 obvious. So it's been deferred.

I'm not clear on how one is supposed to go about making the equivalent of hygienic macros using just string mixins and CTFE.

You can create a trial-and-error gensym function with compile time introspection and CTFE/templates. :o) It's very elegant. CTFE solves all your AST macro use cases.

Wrong. The correct answer is magic.

Yes, and it has to be that way. The AST macros which people have been talking about are magical.
Jul 12 2010
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-07-11 08:47:26 -0400, "Lars T. Kyllingstad" 
<public kyllingen.NOSPAMnet> said:

 On Sun, 11 Jul 2010 14:26:51 +0200, Philippe Sigaud wrote:
 
 That's interesting. Do you have a link or any text I could read on that?
 String mixins sure are powerful, but I can't get ird of a feeling of
 'cheating' when using them. Maybe with some kind of string interpolation
 they could be made more palatable to some?

I find that using token strings, q{ ... }, rather than ordinary "..." strings, sometimes makes string mixins feel less like a hack. Especially considering that my editor (vim) highlights token strings like ordinary code -- or, more likely, it doesn't recognise them as strings. ;)

Personally, I find it *more* like a hack. q{...} is just a way to disguise a string as not being one, it's like using a second hack to better hide the first hack. But it's too hacks instead of one, and thus it's more obscure. That said, I don't feel like I'm cheating when using string mixins. I find them a quite good substitute to AST macros. And perhaps string mixins are simpler too: you don't have to learn a new macro syntax, you just manipulate strings. Though I'm still waiting for the day we can use string mixins in expressions to do things like this: int num = 1; string result = substitute!"Number: $num"; assert(result == "Number: 1"); -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Jul 11 2010
parent reply Chad J <chadjoan __spam.is.bad__gmail.com> writes:
On 07/11/2010 12:58 PM, Rory McGuire wrote:
 On Sun, 11 Jul 2010 15:29:36 +0200, Michel Fortin
 <michel.fortin michelf.com> wrote:
 
 On 2010-07-11 08:47:26 -0400, "Lars T. Kyllingstad"
 <public kyllingen.NOSPAMnet> said:

 On Sun, 11 Jul 2010 14:26:51 +0200, Philippe Sigaud wrote:

 That's interesting. Do you have a link or any text I could read on
 that?
 String mixins sure are powerful, but I can't get ird of a feeling of
 'cheating' when using them. Maybe with some kind of string
 interpolation
 they could be made more palatable to some?

strings, sometimes makes string mixins feel less like a hack. Especially considering that my editor (vim) highlights token strings like ordinary code -- or, more likely, it doesn't recognise them as strings. ;)

Personally, I find it *more* like a hack. q{...} is just a way to disguise a string as not being one, it's like using a second hack to better hide the first hack. But it's too hacks instead of one, and thus it's more obscure. That said, I don't feel like I'm cheating when using string mixins. I find them a quite good substitute to AST macros. And perhaps string mixins are simpler too: you don't have to learn a new macro syntax, you just manipulate strings. Though I'm still waiting for the day we can use string mixins in expressions to do things like this: int num = 1; string result = substitute!"Number: $num"; assert(result == "Number: 1");

someone already made something like that, I forget where it was. Its old now.

I doubt it. Not unless they did something more like this: int num = 1; string result = mixin(substitute!"Number: $num"); assert(result == "Number: 1"); It'd be really nice to have some sugar for string mixin usage. It could be tied into templates to do some really awesome things. Some prior discussion: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=105781
Jul 11 2010
parent "Nick Sabalausky" <a a.a> writes:
"Chad J" <chadjoan __spam.is.bad__gmail.com> wrote in message 
news:i1d8j4$1347$1 digitalmars.com...
 On 07/11/2010 12:58 PM, Rory McGuire wrote:
 On Sun, 11 Jul 2010 15:29:36 +0200, Michel Fortin
 <michel.fortin michelf.com> wrote:
     int num = 1;
     string result = substitute!"Number: $num";
     assert(result == "Number: 1");

someone already made something like that, I forget where it was. Its old now.

I doubt it. Not unless they did something more like this: int num = 1; string result = mixin(substitute!"Number: $num"); assert(result == "Number: 1"); It'd be really nice to have some sugar for string mixin usage. It could be tied into templates to do some really awesome things. Some prior discussion: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=105781

Yea, I was really disappointed the only thing that came out of that was an improvement in instantiating template mixins, but not string mixins. I use string mixins so much more than template ones.
Jul 11 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Steven Schveighoffer:
 If people agree that AST macros are superseded by mixins,

String mixins are a hack, just a bit better than C preprocessor (because they are scoped), they are not a replacement of clean macros. Bye, bearophile
Jul 12 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/12/2010 07:30 AM, Steven Schveighoffer wrote:
 On Mon, 12 Jul 2010 08:12:31 -0400, bearophile
 <bearophileHUGS lycos.com> wrote:

 Steven Schveighoffer:
 If people agree that AST macros are superseded by mixins,

String mixins are a hack, just a bit better than C preprocessor (because they are scoped), they are not a replacement of clean macros.

Interesting statement, can you back it up? :) What can you do with a macro that you can't do with a mixin? -Steve

I think the big thing about macros is you don't have to worry about lexing and parsing. if <A> is of the form (Assignable, Assignable, ... ), and <B> of the form (AssignExp, AssignExp, ... ) how do you readily rewrite mixin("<A> = <B>;"); to (<A>[0] = <B>[0], <A>[1] = <B>[1], ... ) without a metric shit ton of library support? With the most bare bones of AST macros (maybe a ctfe function that gets passed a AST* and returns a AST* or something equivalent), it's pretty trivial. And you don't need to wrap it in a string literal.
Jul 12 2010
parent reply pillsy <pillsbury gmail.com> writes:
== Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 On Mon, 12 Jul 2010 09:33:34 -0400, Ellery Newcomer
 <ellery-newcomer utulsa.edu> wrote:

 I think the big thing about macros is you don't have to
 worry about lexing and parsing.


 if <A> is of the form (Assignable, Assignable, ... ),
 and <B> of the form (AssignExp, AssignExp, ... )


 how do you readily rewrite


 mixin("<A> = <B>;");


 to


 (<A>[0] = <B>[0], <A>[1] = <B>[1], ... )


 without a metric shit ton of library support?


 With the most bare bones of AST macros (maybe a ctfe
 function that gets passed a AST* and returns a AST*
 or something equivalent), it's pretty trivial. And
 you don't need to wrap it in a string literal.


 But is that common? Is there a common need to parse mixin
 strings and replace with other strings?

In my experience with Lisp macros (which are fundamentally AST macros) doing that sort of thing is fairly common when it's easy.
 Even if it is, can't a string
 mixin system do exactly that?

I think it can, actually. Having library functions pick up the slack may not be a terrible idea at all. Parsing a string into an AST is just a function, transforming that AST into another AST is a function, and transforming the resulting AST back into a string that gets mixed in is a third function. I see two problems with this plan, both which may be addressable without too much trouble. One problem is that string mixins look terrible, what with them being string literals. Having some sort of syntactic support for using macros would be a major improvement. I liked your "macro" keyword suggestion from elsethread. It requires a lot of straight-up hackery. Consider retard's suggested way for creating a unique name. Being able to do this is crucial to writing macros that simultaneously do non-trivial things and are robust, and the inability to create unique names is one reason C preprocessor macros are so lame. It would be far nicer if you could use some sort of __traits form to get a unique name (and then that could be wrapped in a string-mixin macro)!
 Also, to me, it's trivial to understand string manipulation
 where the end result is the language I already know, vs.
 having to learn how the compiler represents syntax.

String manipulation scales very poorly, it's error prone, and it's tedious. Cheers, Pillsy
Jul 12 2010
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 07/12/2010 09:57 AM, pillsy wrote:
 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 On Mon, 12 Jul 2010 09:33:34 -0400, Ellery Newcomer
 <ellery-newcomer utulsa.edu>  wrote:

 I think the big thing about macros is you don't have to
 worry about lexing and parsing.


 if<A>  is of the form (Assignable, Assignable, ... ),
 and<B>  of the form (AssignExp, AssignExp, ... )


 how do you readily rewrite


 mixin("<A>  =<B>;");


 to


 (<A>[0] =<B>[0],<A>[1] =<B>[1], ... )


 without a metric shit ton of library support?


 With the most bare bones of AST macros (maybe a ctfe
 function that gets passed a AST* and returns a AST*
 or something equivalent), it's pretty trivial. And
 you don't need to wrap it in a string literal.


 But is that common? Is there a common need to parse mixin
 strings and replace with other strings?

In my experience with Lisp macros (which are fundamentally AST macros) doing that sort of thing is fairly common when it's easy.
 Even if it is, can't a string
 mixin system do exactly that?

I think it can, actually. Having library functions pick up the slack may not be a terrible idea at all. Parsing a string into an AST is just a function, transforming that AST into another AST is a function, and transforming the resulting AST back into a string that gets mixed in is a third function. I see two problems with this plan, both which may be addressable without too much trouble. One problem is that string mixins look terrible, what with them being string literals. Having some sort of syntactic support for using macros would be a major improvement. I liked your "macro" keyword suggestion from elsethread. It requires a lot of straight-up hackery. Consider retard's suggested way for creating a unique name. Being able to do this is crucial to writing macros that simultaneously do non-trivial things and are robust, and the inability to create unique names is one reason C preprocessor macros are so lame. It would be far nicer if you could use some sort of __traits form to get a unique name (and then that could be wrapped in a string-mixin macro)!
 Also, to me, it's trivial to understand string manipulation
 where the end result is the language I already know, vs.
 having to learn how the compiler represents syntax.

String manipulation scales very poorly, it's error prone, and it's tedious. Cheers, Pillsy

Though I sympathise with all of the above, I should add that I have looked at languages that feature AST macros (Lisp, Scheme, Dylan) and such macros are difficult to create and read as well. We're not looking at day and night improvement there. Andrei
Jul 12 2010
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 07/13/2010 10:43 AM, retard wrote:
 Yes, this would be possible if the dmd compiler didn't crash, leak memory
 etc. The quality is so shitty that I'd say that implementing a compile
 time AST parser for D 2 is almost impossible.

Well it should be said that the quality of the compiler has dramatically improved over the past months. I think it would be worth writing a tokenizer for starters.
 And what's worse is that
 the language is changing constantly so the AST parser meta-library would
 need to be updated once in a while. This is great news for all the fans
 of NIH syndrome.

The rate of language changes has slowed down very visibly since TDPL has come out.
 String manipulation scales very poorly, it's error prone, and it's
 tedious.

Like Andrei said, real AST macros can also be hairy, but they _are_ an improvement over string based macros in many cases. Lisp users have 50+ years of experience with meta-programming. You're mostly talking with amateurs here. Of course they don't see the value of true macros. One fundamental prerequisite when discussing macros is that you've actually used them in at least ONE other language.

Agreed. Reminds me of the Go programmers who don't need generics :o). The Sapir-Whorf theory may or may not apply to natural language, but has a visible presence in programming languages. Andrei
Jul 13 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Thu, 08 Jul 2010 21:43:57 -0400, Robert Jacques wrote:

 Check out Walter's slides and/or talk from the D conference.
 (http://www.digitalmars.com/webnews/newsgroups.php?

 AST does stand for abstract syntax tree and they are much more like Lisp
 macros as opposed to the C preprocessor.

Too bad nothing happened. He promised AST macros, but it will probably still take several years before we have a preliminary implementation and maybe 10-20 years before a written spec & almsot bug-free implementation.
Jul 10 2010
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
--0015175cd450362157048b1bc035
Content-Type: text/plain; charset=ISO-8859-1

Don wrote:

 Part of what happened was that at the conference, I showed that the
 functionality of string mixins was a superset of what was planned for AST
 macros. Since that time, there has not been any kind of proposal, or even a
 concept. So it's a bit meaningless to talk about "AST macros" right now as
 if they were a planned-but-not-yet-implemented feature.

That's interesting. Do you have a link or any text I could read on that? String mixins sure are powerful, but I can't get ird of a feeling of 'cheating' when using them. Maybe with some kind of string interpolation they could be made more palatable to some? Philippe --0015175cd450362157048b1bc035 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">Don wrote:<br><blockquote class=3D"gmail_quote" = style=3D"margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 20= 4); padding-left: 1ex;"><div><div></div><div class=3D"h5"><br></div></div> Part of what happened was that at the conference, I showed that the functio= nality of string mixins was a superset of what was planned for AST macros. = Since that time, there has not been any kind of proposal, or even a concept= . So it&#39;s a bit meaningless to talk about &quot;AST macros&quot; right = now as if they were a planned-but-not-yet-implemented feature.<br> </blockquote><div><br>That&#39;s interesting. Do you have a link or any tex= t I could read on that? String mixins sure are powerful, but I can&#39;t ge= t ird of a feeling of &#39;cheating&#39; when using them. Maybe with some k= ind of string interpolation they could be made more palatable to some?<br> <br><br>Philippe<br><br></div></div><br> --0015175cd450362157048b1bc035--
Jul 11 2010
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
On Sun, 11 Jul 2010 14:26:51 +0200, Philippe Sigaud wrote:

 Don wrote:
 
 
 Part of what happened was that at the conference, I showed that the
 functionality of string mixins was a superset of what was planned for
 AST macros. Since that time, there has not been any kind of proposal,
 or even a concept. So it's a bit meaningless to talk about "AST macros"
 right now as if they were a planned-but-not-yet-implemented feature.

String mixins sure are powerful, but I can't get ird of a feeling of 'cheating' when using them. Maybe with some kind of string interpolation they could be made more palatable to some?

I find that using token strings, q{ ... }, rather than ordinary "..." strings, sometimes makes string mixins feel less like a hack. Especially considering that my editor (vim) highlights token strings like ordinary code -- or, more likely, it doesn't recognise them as strings. ;) -Lars
Jul 11 2010
prev sibling next sibling parent "Rory McGuire" <rmcguire neonova.co.za> writes:
On Sun, 11 Jul 2010 15:29:36 +0200, Michel Fortin  
<michel.fortin michelf.com> wrote:

 On 2010-07-11 08:47:26 -0400, "Lars T. Kyllingstad"  
 <public kyllingen.NOSPAMnet> said:

 On Sun, 11 Jul 2010 14:26:51 +0200, Philippe Sigaud wrote:

 That's interesting. Do you have a link or any text I could read on  
 that?
 String mixins sure are powerful, but I can't get ird of a feeling of
 'cheating' when using them. Maybe with some kind of string  
 interpolation
 they could be made more palatable to some?

strings, sometimes makes string mixins feel less like a hack. Especially considering that my editor (vim) highlights token strings like ordinary code -- or, more likely, it doesn't recognise them as strings. ;)

Personally, I find it *more* like a hack. q{...} is just a way to disguise a string as not being one, it's like using a second hack to better hide the first hack. But it's too hacks instead of one, and thus it's more obscure. That said, I don't feel like I'm cheating when using string mixins. I find them a quite good substitute to AST macros. And perhaps string mixins are simpler too: you don't have to learn a new macro syntax, you just manipulate strings. Though I'm still waiting for the day we can use string mixins in expressions to do things like this: int num = 1; string result = substitute!"Number: $num"; assert(result == "Number: 1");

someone already made something like that, I forget where it was. Its old now.
Jul 11 2010
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
--0016e6dd9905afef3e048b22049f
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Jul 11, 2010 at 18:58, Rory McGuire <rmcguire neonova.co.za> wrote:

 On Sun, 11 Jul 2010 15:29:36 +0200, Michel Fortin <
 michel.fortin michelf.com> wrote:

        int num = 1;
        string result = substitute!"Number: $num";
        assert(result == "Number: 1");

now.

Doesn't Format (in std.metastrings) cover part of this? string result = Format!("Numbers: %s", to!string(num)); I remember finding it cumbersome to use. For one, it could easily take any type for argument, and call to!string on them internally. Why bothering me with this? But maybe I wasn't using it the right way. Hmm, I think what I'd like is some numbered substituting scheme: string result = Substitute!("static if (is (typeof(%1) t == %2!U, U)) { alias U Result; else alias typeof(%1) Result;", a,b); I agree with Rory than someone already did something similar. But I guess now it can be done with CTFE on strings. A CT replace. Philippe --0016e6dd9905afef3e048b22049f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Sun, Jul 11, 2010 at 18:58, Rory McGuire <spa= n dir=3D"ltr">&lt;<a href=3D"mailto:rmcguire neonova.co.za">rmcguire neonov= a.co.za</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D= "margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padd= ing-left: 1ex;"> <div><div class=3D"h5">On Sun, 11 Jul 2010 15:29:36 +0200, Michel Fortin &l= t;<a href=3D"mailto:michel.fortin michelf.com" target=3D"_blank">michel.for= tin michelf.com</a>&gt; wrote:<br><blockquote class=3D"gmail_quote" style= =3D"margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); p= adding-left: 1ex;"> <br> =A0 =A0 =A0 =A0int num =3D 1;<br> =A0 =A0 =A0 =A0string result =3D substitute!&quot;Number: $num&quot;;<br> =A0 =A0 =A0 =A0assert(result =3D=3D &quot;Number: 1&quot;);<br> <br> </blockquote> <br></div></div> someone already made something like that, I forget where it was. Its old no= w.<br> </blockquote></div><br>Doesn&#39;t Format (in std.metastrings) cover part o= f this?<br><br>string result =3D Format!(&quot;Numbers: %s&quot;, to!string= (num));<br><br>I remember finding it cumbersome to use. For one, it could e= asily take any type for argument, and call to!string on them internally. Wh= y bothering me with this?<br> But maybe I wasn&#39;t using it the right way.<br><br>Hmm, I think what I&#= 39;d like is some numbered substituting scheme:<br><br>string result =3D Su= bstitute!(&quot;static if (is (typeof(%1) t =3D=3D %2!U, U)) {<br>=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0 alias U Result;<br> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 else<br>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0 alias typeof(%1) Result;&quot;,=A0 <br>=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 a,b);<br=
<br><br>I agree with Rory than someone already did something similar. But =

<br><br>Philippe<br><br> --0016e6dd9905afef3e048b22049f--
Jul 11 2010
prev sibling next sibling parent "Rory McGuire" <rmcguire neonova.co.za> writes:
------------H3Cn1JYaN20rTAQKL9RAaT
Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sun, 11 Jul 2010 21:55:30 +0200, Philippe Sigaud  
<philippe.sigaud gmail.com> wrote:

 On Sun, Jul 11, 2010 at 18:58, Rory McGuire <rmcguire neonova.co.za>  
 wrote:
 On Sun, 11 Jul 2010 15:29:36 +0200, Michel Fortin  
 <michel.fortin michelf.com> wrote:
        int num = 1;
        string result = substitute!"Number: $num";
        assert(result == "Number: 1");

someone already made something like that, I forget where it was. Its old now.

Doesn't Format (in std.metastrings) cover part of this? string result = Format!("Numbers: %s", to!string(num)); I remember finding it cumbersome to use. For one, it could easily take any type for argument, and call to!string on them internally. Why
bothering me with this?

Hmm, I think what I'd like is some numbered substituting scheme: string result = Substitute!("static if (is (typeof(%1) t == %2!U, U)) { alias U Result; else alias typeof(%1) Result;", a,b); I agree with Rory than someone already did something similar. But I guess now it can be done with CTFE on strings. A CT replace. Philippe

I think I found what I was thinking of (not sure its the same one): http://www.digitalmars.com/d/archives/digitalmars/D/PHP-style_embedded_variables_print_statements_54053.html -Rory ------------H3Cn1JYaN20rTAQKL9RAaT Content-Type: multipart/related; boundary=----------H3Cn1JYaN20rTAFlB7TIoJ ------------H3Cn1JYaN20rTAFlB7TIoJ Content-Type: text/html; charset=utf-8 Content-ID: <op.1278878842523.81bbed546d5da645 10.0.0.88> Content-Transfer-Encoding: Quoted-Printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"> <HTML> <HEAD></HEAD> <BODY style=3D"font-family:'DejaVu Sans Mono'; font-size:12px"><div>On Sun,= 11 Jul 2010 21:55:30 +0200, Philippe Sigaud &lt;philippe.sigaud gmail.com&= gt; wrote:<br><br><blockquote style=3D"margin: 0 0 0.80ex; border-left: #00= 00FF 2px solid; padding-left: 1ex"><div class=3D"gmail_quote">On Sun, Jul 1= 1, 2010 at 18:58, Rory McGuire <span dir=3D"ltr">&lt;<a href=3D"mailto:rmcg= uire neonova.co.za">rmcguire neonova.co.za</a>&gt;</span> wrote:<br><blockq= uote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; border-left:= 1px solid rgb(204, 204, 204); padding-left: 1ex;"> <div><div class=3D"h5">On Sun, 11 Jul 2010 15:29:36 +0200, Michel Fortin &l= t;<a href=3D"mailto:michel.fortin michelf.com" target=3D"_blank">michel.for= tin michelf.com</a>&gt; wrote:<br><blockquote class=3D"gmail_quote" style= =3D"margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); p= adding-left: 1ex;"> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0int num =3D 1;<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0string result =3D substitute!"Number: $num";<br=

<br> </blockquote> <br></div></div> someone already made something like that, I forget where it was. Its old no= w.<br> </blockquote></div><br>Doesn't Format (in std.metastrings) cover part of th= is?<br><br>string result =3D Format!("Numbers: %s", to!string(num));<br><br=
I remember finding it cumbersome to use. For one, it could easily take any=

with this?<br> But maybe I wasn't using it the right way.<br><br>Hmm, I think what I'd lik= e is some numbered substituting scheme:<br><br>string result =3D Substitute= !("static if (is (typeof(%1) t =3D=3D %2!U, U)) {<br>=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 alias U Result;<br> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 else<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 alias typeof(%1) Result;",=C2= =A0 <br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 a,b);<br><br><br>I agree with Rory than someone already did something s= imilar. But I guess now it can be done with CTFE on strings. A CT replace.<= br> <br><br>Philippe<br><br> </blockquote><br><br>I think I found what I was thinking of (not sure its t= he same one):<br></div><div>http://www.digitalmars.com/d/archives/digitalma= rs/D/PHP-style_embedded_variables_print_statements_54053.html</div><div><br=
</div><div>-Rory</div></BODY></HTML>

------------H3Cn1JYaN20rTAQKL9RAaT--
Jul 11 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Sun, 11 Jul 2010 22:03:56 +0000, pillsy wrote:

 == Quote from Don (nospam nospam.com)'s article
 
 Philippe Sigaud wrote:

 String mixins sure are powerful, but I can't get ird of a feeling of
 'cheating' when using them. Maybe with some kind of string
 interpolation they could be made more palatable to some?


 There's little doubt that it's the part of the language which needs the
 most attention in the long term. But the way forward is not at all
 obvious. So it's been deferred.

I'm not clear on how one is supposed to go about making the equivalent of hygienic macros using just string mixins and CTFE.

You can create a trial-and-error gensym function with compile time introspection and CTFE/templates. :o) It's very elegant. CTFE solves all your AST macro use cases.
Jul 11 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sun, 11 Jul 2010 07:01:49 -0400, Don <nospam nospam.com> wrote:

 retard wrote:
 Thu, 08 Jul 2010 21:43:57 -0400, Robert Jacques wrote:

 Check out Walter's slides and/or talk from the D conference.
 (http://www.digitalmars.com/webnews/newsgroups.php?

 AST does stand for abstract syntax tree and they are much more like  
 Lisp
 macros as opposed to the C preprocessor.

still take several years before we have a preliminary implementation and maybe 10-20 years before a written spec & almsot bug-free implementation.

Part of what happened was that at the conference, I showed that the functionality of string mixins was a superset of what was planned for AST macros. Since that time, there has not been any kind of proposal, or even a concept. So it's a bit meaningless to talk about "AST macros" right now as if they were a planned-but-not-yet-implemented feature.

The problem I see with mixins is they are ugly. The difference between mixins and most other parts of the language is the syntax is still super-basic. It would be the equivalent of having to call functions with callfunction(funcname, params). What we need is a way to make mixins look like normal expressions. I proposed at one time to use macro to define easily callable mixins. It was something along the lines of: macro log(level, x) mixin("if(" ~ level ~ " == log.level){log.output(" ~ x ~ ")}"); which makes it possible to have lazy logging without the lazy keyword and without the ugliness of mixins at the call site. If people agree that AST macros are superseded by mixins, then why not take over the macro keyword (which if restricted to AST macros is effectively a dead keyword), and make programmer's lives and generic programming easier? -Steve
Jul 12 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 12 Jul 2010 08:12:31 -0400, bearophile <bearophileHUGS lycos.com>  
wrote:

 Steven Schveighoffer:
 If people agree that AST macros are superseded by mixins,

String mixins are a hack, just a bit better than C preprocessor (because they are scoped), they are not a replacement of clean macros.

Interesting statement, can you back it up? :) What can you do with a macro that you can't do with a mixin? -Steve
Jul 12 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 12 Jul 2010 09:33:34 -0400, Ellery Newcomer  
<ellery-newcomer utulsa.edu> wrote:

 On 07/12/2010 07:30 AM, Steven Schveighoffer wrote:
 On Mon, 12 Jul 2010 08:12:31 -0400, bearophile
 <bearophileHUGS lycos.com> wrote:

 Steven Schveighoffer:
 If people agree that AST macros are superseded by mixins,

String mixins are a hack, just a bit better than C preprocessor (because they are scoped), they are not a replacement of clean macros.

Interesting statement, can you back it up? :) What can you do with a macro that you can't do with a mixin? -Steve

I think the big thing about macros is you don't have to worry about lexing and parsing. if <A> is of the form (Assignable, Assignable, ... ), and <B> of the form (AssignExp, AssignExp, ... ) how do you readily rewrite mixin("<A> = <B>;"); to (<A>[0] = <B>[0], <A>[1] = <B>[1], ... ) without a metric shit ton of library support? With the most bare bones of AST macros (maybe a ctfe function that gets passed a AST* and returns a AST* or something equivalent), it's pretty trivial. And you don't need to wrap it in a string literal.

But is that common? Is there a common need to parse mixin strings and replace with other strings? It seems like the means that AST macros operate, but is that the ultimate goal? Even if it is, can't a string mixin system do exactly that? You say a ton of library support is bad, but why is that? If the end result is the same, what does it matter if you implement it via a library vs. the compiler? Also, to me, it's trivial to understand string manipulation where the end result is the language I already know, vs. having to learn how the compiler represents syntax. I haven't really used AST macros. Does anyone have a good practical example of how it's used in other languages to compare with string mixins? Your example seems trivially solved via simple library functions such as split or using regex. -Steve
Jul 12 2010
prev sibling parent retard <re tard.com.invalid> writes:
Mon, 12 Jul 2010 14:57:35 +0000, pillsy wrote:

 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 On Mon, 12 Jul 2010 09:33:34 -0400, Ellery Newcomer
 <ellery-newcomer utulsa.edu> wrote:

 I think the big thing about macros is you don't have to worry about
 lexing and parsing.


 if <A> is of the form (Assignable, Assignable, ... ), and <B> of the
 form (AssignExp, AssignExp, ... )


 how do you readily rewrite


 mixin("<A> = <B>;");


 to


 (<A>[0] = <B>[0], <A>[1] = <B>[1], ... )


 without a metric shit ton of library support?


 With the most bare bones of AST macros (maybe a ctfe function that
 gets passed a AST* and returns a AST* or something equivalent), it's
 pretty trivial. And you don't need to wrap it in a string literal.


 But is that common? Is there a common need to parse mixin strings and
 replace with other strings?

In my experience with Lisp macros (which are fundamentally AST macros) doing that sort of thing is fairly common when it's easy.
 Even if it is, can't a string
 mixin system do exactly that?

I think it can, actually. Having library functions pick up the slack may not be a terrible idea at all. Parsing a string into an AST is just a function, transforming that AST into another AST is a function, and transforming the resulting AST back into a string that gets mixed in is a third function.

Yes, this would be possible if the dmd compiler didn't crash, leak memory etc. The quality is so shitty that I'd say that implementing a compile time AST parser for D 2 is almost impossible. And what's worse is that the language is changing constantly so the AST parser meta-library would need to be updated once in a while. This is great news for all the fans of NIH syndrome.
 
 I see two problems with this plan, both which may be addressable without
 too much trouble.
 
 One problem is that string mixins look terrible, what with them being
 string literals. Having some sort of syntactic support for using macros
 would be a major improvement. I liked your "macro" keyword suggestion
 from elsethread.
 
 It requires a lot of straight-up hackery. Consider retard's suggested
 way for creating a unique name. Being able to do this is crucial to
 writing macros that simultaneously do non-trivial things and are robust,
 and the inability to create unique names is one reason C preprocessor
 macros are so lame. It would be far nicer if you could use some sort of
 __traits form to get a unique name (and then that could be wrapped in a
 string-mixin macro)!
 
 Also, to me, it's trivial to understand string manipulation where the
 end result is the language I already know, vs. having to learn how the
 compiler represents syntax.

String manipulation scales very poorly, it's error prone, and it's tedious.

Like Andrei said, real AST macros can also be hairy, but they _are_ an improvement over string based macros in many cases. Lisp users have 50+ years of experience with meta-programming. You're mostly talking with amateurs here. Of course they don't see the value of true macros. One fundamental prerequisite when discussing macros is that you've actually used them in at least ONE other language.
Jul 13 2010