www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Mixin template evaluated to string can convert to string mixin

reply kenji hara <k.hara.pg gmail.com> writes:
Issue in bugzilla:
  http://d.puremagic.com/issues/show_bug.cgi?id=6207
Test implementation:
  https://github.com/9rnsr/dmd/commit/52e4491cf6bf9ccf6dccc1fa60581fd6797d39b3

String mixin feature in D is very useful, but its syntax is bit ugly.
My proposal will make it more softly, and introduces more generative
programming.

Any thoughts?

Kenji
Jun 24 2011
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
kenji hara wrote:

 Issue in bugzilla:
   http://d.puremagic.com/issues/show_bug.cgi?id=6207
 Test implementation:
   https://github.com/9rnsr/dmd/commit/52e4491cf6bf9ccf6dccc1fa60581fd6797d39b3

 String mixin feature in D is very useful, but its syntax is bit ugly.
 My proposal will make it more softly, and introduces more generative
 programming.

 Any thoughts?

 Kenji



String mixins themselves are 'a little bit ugly' (but unquestionably very
useful).
I think the syntax should keep reflecting that.

Your proposal is a try to make up for Ds lack of macros. I'd prefer macros.
Also overloading the meaning of 'mixin template' seems to be questionable. As I
understand it, your proposal would make code like this valid?:

mixin template bar(bool b){
    static if(b){
        enum bar="foo(123);"
    }else{
        int foo(int x){
            return 123+x;
        }
    }
}

int main(){
    mixin bar!(0);
    int x=bar!(1);
    assert(x == 246);
}

This does not seem quite right to me.

Cheers,
-Timon
Jun 24 2011
next sibling parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Timon Gehr" <timon.gehr gmx.ch> wrote:

 String mixins themselves are 'a little bit ugly' (but unquestionably very 
 useful).
 I think the syntax should keep reflecting that.

 Your proposal is a try to make up for Ds lack of macros. I'd prefer 
 macros.
 Also overloading the meaning of 'mixin template' seems to be questionable. 
 As I
 understand it, your proposal would make code like this valid?:




With Don's extension of CTFE and Kenji proposal it's possible to fully 
implement macroses in library. I've tried to do that earlier but it was 
impossible with current compiler state. I think it's GREAT idea. Thank Kenji 
for making it. I'm interested in implementing such macro library. Are there 
any instructions how to compile DMD under windows?
Jun 24 2011
next sibling parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"kenji hara" <k.hara.pg gmail.com> wrote:

 My proposal is just a syntactic sugar.
 You can develop such library without it.



yes, but still stable DMD release can't handle pointers in CTFE.
And without such syntactic sugar it's ungly to write something like that
    mixin(sum!q{i : iota(N); mixin(sum!q{j : iota(M), j !=i ; 
(i+j)/(i-j) }) });
this looks much better
    sum!q{i : iota(N); sum!q{j : iota(M), j !=i ; (i+j)/(i-j)}};
Jun 24 2011
parent reply %u <asmasm hotmail.com> writes:
== Quote from kenji hara (k.hara.pg gmail.com)'s article

 Note: A simple way to construct environment for build dmd/druntime/phobos i
 s:
 1. Expand newest release package.
 2. Replace dmd2/src/dmd, dmd2/src/druntime, and dmd2/src/phobos to
 checkouted git repositories.
 3. Copy https://gist.github.com/1045205 into dmd2/src
     I am using this makefile for build every day.
 3. Run make in dmd2/src: make



I have error(archlinux):
cd dmd\src
/bin/sh: line 0: cd: dmdsrc: No such file or directory
make: *** [build-dmd] Error 1
Jun 24 2011
parent "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"%u" <asmasm hotmail.com> wrote:

 == Quote from kenji hara (k.hara.pg gmail.com)'s article

 Note: A simple way to construct environment for build dmd/druntime/phobos 
 i
 s:
 1. Expand newest release package.
 2. Replace dmd2/src/dmd, dmd2/src/druntime, and dmd2/src/phobos to
 checkouted git repositories.
 3. Copy https://gist.github.com/1045205 into dmd2/src
     I am using this makefile for build every day.
 3. Run make in dmd2/src: make



 I have error(archlinux):
 cd dmd\src
 /bin/sh: line 0: cd: dmdsrc: No such file or directory
 make: *** [build-dmd] Error 1



Maybe because Linux uses slash(/) for dir separator and not backslash(\).
Jun 24 2011
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Alex_Dovhal wrote:

 "Timon Gehr" <timon.gehr gmx.ch> wrote:

 String mixins themselves are 'a little bit ugly' (but unquestionably very
 useful).
 I think the syntax should keep reflecting that.

 Your proposal is a try to make up for Ds lack of macros. I'd prefer
 macros.
 Also overloading the meaning of 'mixin template' seems to be questionable.
 As I
 understand it, your proposal would make code like this valid?:




 With Don's extension of CTFE and Kenji proposal it's possible to fully
 implement macroses in library. I've tried to do that earlier but it was
 impossible with current compiler state. I think it's GREAT idea. Thank Kenji
 for making it. I'm interested in implementing such macro library. Are there
 any instructions how to compile DMD under windows?



http://www.digitalmars.com/d/2.0/dmd-windows.html#compiling_dmd

Macros operate on ASTs, not strings. You won't get real macros, just something
that feels somewhat like them. You still could implement the equivalent to C
macros (not real macros either) in a library, but they would have all the
problems
of C macros, as string mixins do now. (You make this fact explicit by typing
"mixin").

For anything but the most trivial code transformations, you have to cheat the
compiler into parsing your code into an AST (further complicated by the fact
that
you cannot overload all operators freely), use clever tricks to get the
information you want and then you have to transform it back into a string so
that
it can then be mixed in (which of course requires reparsing the whole thing.)
This
is just painful and a waste of cycles.

String mixins are useful, but they cannot replace macros.

What would your library be capable of?

Cheers,
-Timon
Jun 24 2011
parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Timon Gehr" <timon.gehr gmx.ch> wrote:

 Macros operate on ASTs, not strings. You won't get real macros, just 
 something
 that feels somewhat like them. You still could implement the equivalent to 
 C
 macros (not real macros either) in a library, but they would have all the 
 problems
 of C macros, as string mixins do now. (You make this fact explicit by 
 typing "mixin").

 For anything but the most trivial code transformations, you have to cheat 
 the
 compiler into parsing your code into an AST (further complicated by the 
 fact that
 you cannot overload all operators freely), use clever tricks to get the
 information you want and then you have to transform it back into a string 
 so that
 it can then be mixed in (which of course requires reparsing the whole 
 thing.) This
 is just painful and a waste of cycles.
 String mixins are useful, but they cannot replace macros.

 What would your library be capable of?



Yes. Library macros can build AST with CTFE :)

About Library interface, one sample:

template macroFunct(string T)
{
    enum macroFunct = macroFunctImpl(T);
}

string macroFunctImpl(string arg)
{
    auto ast = parseAST_1(arg);
    // do something. e.g.
    return ast[0];
}

/*AST, here string[]*/ string[] parseAST_1(string arg)
{
    //do something. e.g.
    return [arg];
}

int main()
{
    assert(mixin(macroFunct!q{123})==123);
    assert(! (mixin(macroFunct!q{321})==322));

    return 0;
}
Jun 24 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Alex_Dovhal wrote:

 "Timon Gehr" <timon.gehr gmx.ch> wrote:

 Macros operate on ASTs, not strings. You won't get real macros, just
 something
 that feels somewhat like them. You still could implement the equivalent to
 C
 macros (not real macros either) in a library, but they would have all the
 problems
 of C macros, as string mixins do now. (You make this fact explicit by
 typing "mixin").

 For anything but the most trivial code transformations, you have to cheat
 the
 compiler into parsing your code into an AST (further complicated by the
 fact that
 you cannot overload all operators freely), use clever tricks to get the
 information you want and then you have to transform it back into a string
 so that
 it can then be mixed in (which of course requires reparsing the whole
 thing.) This
 is just painful and a waste of cycles.
 String mixins are useful, but they cannot replace macros.

 What would your library be capable of?



 Yes. Library macros can build AST with CTFE :)



My question could not possibly be answered with 'yes', actually I was interested
in what functionality your library would provide for the end user.
You answered the question "Would your library be capable of building an AST?".


 About Library interface, one sample:

 template macroFunct(string T)
 {
     enum macroFunct = macroFunctImpl(T);
 }

 string macroFunctImpl(string arg)
 {
     auto ast = parseAST_1(arg);
     // do something. e.g.
     return ast[0];
 }

 /*AST, here string[]*/ string[] parseAST_1(string arg)
 {
     //do something. e.g.
     return [arg];
 }



really hacky, ugly and overlong code.)



 int main()
 {
     assert(mixin(macroFunct!q{123})==123);
     assert(! (mixin(macroFunct!q{321})==322));

     return 0;
 }



You did not fill out the function bodies because that would be a PITA and some
n*1000 lines long, right?
Lexing and parsing code should be the compiler's business.
You want to use a compiler to interpret code that then parses other code into an
AST, that is then mutated and then converted back to a string, just that the
compiler's native parser can parse it again.
The fact that it could be made to work does not mean it is a good idea. :)

Cheers,
-Timon
Jun 24 2011
parent "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Timon Gehr" <timon.gehr gmx.ch> wrote:

 My question could not possibly be answered with 'yes', actually I was 
 interested
 in what functionality your library would provide for the end user.
 You answered the question "Would your library be capable of building an 
 AST?".



Sorry for ambiguity, "yes" was answer for
    > Macros operate on ASTs, not strings.
Yes I plan to build AST, but for every different DSEL/macro type, different 
AST is used.


 /*AST, here string[]*/ string[] parseAST_1(string arg)
 {
     //do something. e.g.
     return [arg];
 }



 want some
 really hacky, ugly and overlong code.)



It's just simplest example which works in CTFE, sure AST builder should be 
class-tree not array of strings.


 int main()
 {
     assert(mixin(macroFunct!q{123})==123);
     assert(! (mixin(macroFunct!q{321})==322));

     return 0;
 }



 You did not fill out the function bodies because that would be a PITA and 
 some
 n*1000 lines long, right?
 Lexing and parsing code should be the compiler's business.



For library yes, but end user must have simple interface. Sorry for not 
giving more concrete info, it's steel in too early stage.


 You want to use a compiler to interpret code that then parses other code 
 into an
 AST, that is then mutated and then converted back to a string, just that 
 the
 compiler's native parser can parse it again.



Compiler can't parse every DSEL AST. Even not everywhere full AST in needed! 
For most task's it's enoungh to parse:
 * comments
 * strings
 * matching bracket blocks (to find one comma separator in  fcn!q{a, (b, 
c)+d}  )
 * comma, colon and semicolon sepatated items (fcn!q{name1: val1, name2: 
val2 ; otherBlock})
to get extended C macros


 The fact that it could be made to work does not mean it is a good idea. :)



But it's good for hobby project to try D.
Jun 24 2011
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
Kenji Hara wrote:

 2011/6/25 Timon Gehr <timon.gehr gmx.ch>:

 String mixins themselves are 'a little bit ugly' (but unquestionably very




 I think the syntax should keep reflecting that.

 Your proposal is a try to make up for Ds lack of macros. I'd prefer macros.
 Also overloading the meaning of 'mixin template' seems to be questionable. As I
 understand it, your proposal would make code like this valid?:

 mixin template bar(bool b){
    static if(b){
        enum bar="foo(123);"
    }else{
        int foo(int x){
            return 123+x;
        }
    }
 }

 int main(){
    mixin bar!(0);
    int x=bar!(1);
    assert(x == 246);
 }

 This does not seem quite right to me.

 Cheers,
 -Timon



 Yes, you can write template like that.
 But it seems to me that is not good even if implicit string mixin is not used.

 Kenji



What I wanted to point out with the example was that your proposal overloads
'mixin template' with two completely orthogonal functionalities:

- Prevent instantiation as a normal template/At instantiation point, require
explicit 'mixin'.
- Allow instantiation as a normal template. At instantiation point, 'mixin(.)'
is
added implicitly.


Timon
Jun 24 2011
prev sibling next sibling parent Paul D. Anderson <paul.d.removethis.anderson comcast.andthis.net> writes:
kenji hara Wrote:


 Issue in bugzilla:
   http://d.puremagic.com/issues/show_bug.cgi?id=6207
 Test implementation:
   https://github.com/9rnsr/dmd/commit/52e4491cf6bf9ccf6dccc1fa60581fd6797d39b3
 
 String mixin feature in D is very useful, but its syntax is bit ugly.
 My proposal will make it more softly, and introduces more generative
 programming.
 
 Any thoughts?
 
 Kenji



Template mixins are considered experimental in TDPL. Andrei warns that this
feature could be replaced in the future. Does this proposal make that less
likely?

Paul
Jun 24 2011
prev sibling next sibling parent kenji hara <k.hara.pg gmail.com> writes:
My proposal is just a syntactic sugar.
You can develop such library without it.

Kenji

2011/6/25 Alex_Dovhal <alex_dovhal yahoo.com>:

 "Timon Gehr" <timon.gehr gmx.ch> wrote:

 String mixins themselves are 'a little bit ugly' (but unquestionably very
 useful).
 I think the syntax should keep reflecting that.

 Your proposal is a try to make up for Ds lack of macros. I'd prefer
 macros.
 Also overloading the meaning of 'mixin template' seems to be questionable.
 As I
 understand it, your proposal would make code like this valid?:




 With Don's extension of CTFE and Kenji proposal it's possible to fully
 implement macroses in library. I've tried to do that earlier but it was
 impossible with current compiler state. I think it's GREAT idea. Thank Kenji
 for making it. I'm interested in implementing such macro library. Are there
 any instructions how to compile DMD under windows?
Jun 24 2011
prev sibling next sibling parent kenji hara <k.hara.pg gmail.com> writes:
2011/6/25 Timon Gehr <timon.gehr gmx.ch>:

 String mixins themselves are 'a little bit ugly' (but unquestionably very=




 I think the syntax should keep reflecting that.

 Your proposal is a try to make up for Ds lack of macros. I'd prefer macro=




 Also overloading the meaning of 'mixin template' seems to be questionable=




 understand it, your proposal would make code like this valid?:

 mixin template bar(bool b){
 =A0 =A0static if(b){
 =A0 =A0 =A0 =A0enum bar=3D"foo(123);"
 =A0 =A0}else{
 =A0 =A0 =A0 =A0int foo(int x){
 =A0 =A0 =A0 =A0 =A0 =A0return 123+x;
 =A0 =A0 =A0 =A0}
 =A0 =A0}
 }

 int main(){
 =A0 =A0mixin bar!(0);
 =A0 =A0int x=3Dbar!(1);
 =A0 =A0assert(x =3D=3D 246);
 }

 This does not seem quite right to me.

 Cheers,
 -Timon



Yes, you can write template like that.
But it seems to me that is not good even if implicit string mixin is not us=
ed.

Kenji
Jun 24 2011
prev sibling next sibling parent kenji hara <k.hara.pg gmail.com> writes:
2011/6/25 Paul D. Anderson <paul.d.removethis.anderson comcast.andthis.net>:

 Template mixins are considered experimental in TDPL. Andrei warns that this
feature could be replaced in the future.



Please tell me the page number that description is written.


 Does this proposal make that less likely?



This is the feature for 'String mixin', and is different from 'Template mixin'.

Template mixins:
  http://d-programming-language.org/template-mixin.html
Mixin Expressions(==String mixins):
  http://d-programming-language.org/expression.html#MixinExpression

Kenji
Jun 24 2011
prev sibling next sibling parent d coder <dlang.coder gmail.com> writes:
--00032555b11a879e7204a6781f08
Content-Type: text/plain; charset=ISO-8859-1


 I also have TDPL, but I have overlooked the description.
 Please tell me the page number that description is written.




--00032555b11a879e7204a6781f08
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div class=3D"gmail_quote"><br><blockquote class=3D"gmail_quote" style=3D"m=
argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div id=3D":=
1nj">I also have TDPL, but I have overlooked the description.<br>
Please tell me the page number that description is written.<br>
<div class=3D"im"><br></div></div></blockquote><div><br></div><div>Third pa=
ragraph of section 7.6 page 282.</div><div>=A0</div></div>

--00032555b11a879e7204a6781f08--
Jun 24 2011
prev sibling next sibling parent reply kenji hara <k.hara.pg gmail.com> writes:
I already checked the test implementation passed dmd build/test and
phobos build/test.
You can checkout 'implicit_mixin' branch from my repository
(https://github.com/9rnsr/dmd/) and use it.

Note: A simple way to construct environment for build dmd/druntime/phobos i=
s:
1. Expand newest release package.
2. Replace dmd2/src/dmd, dmd2/src/druntime, and dmd2/src/phobos to
checkouted git repositories.
3. Copy https://gist.github.com/1045205 into dmd2/src
    I am using this makefile for build every day.
3. Run make in dmd2/src: make

Caution:
In Windows, building compiler needs digitalmars make and MSYS cp command.
And dmd test suite needs cygwin-make.

Kenji

2011/6/25 Alex_Dovhal <alex_dovhal yahoo.com>:

 "kenji hara" <k.hara.pg gmail.com> wrote:

 My proposal is just a syntactic sugar.
 You can develop such library without it.



 yes, but still stable DMD release can't handle pointers in CTFE.
 And without such syntactic sugar it's ungly to write something like that
 =A0 =A0mixin(sum!q{i : iota(N); mixin(sum!q{j : iota(M), j !=3Di ;
 (i+j)/(i-j) }) });
 this looks much better
 =A0 =A0sum!q{i : iota(N); sum!q{j : iota(M), j !=3Di ; (i+j)/(i-j)}};
Jun 24 2011
parent "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"kenji hara" <k.hara.pg gmail.com> wrote:

Note: A simple way to construct environment for build dmd/druntime/phobos 
is:
1. Expand newest release package.
2. Replace dmd2/src/dmd, dmd2/src/druntime, and dmd2/src/phobos to
checkouted git repositories.
3. Copy https://gist.github.com/1045205 into dmd2/src
    I am using this makefile for build every day.
3. Run make in dmd2/src: make



Thanks.
Jun 24 2011
prev sibling next sibling parent kenji hara <k.hara.pg gmail.com> writes:
2011/6/25 d coder <dlang.coder gmail.com>:

 Third paragraph of section 7.6 page 282.



Thank you.

Ah... AST macro. I had heard that is D3 feature, but is it really realize?

Kenji
Jun 24 2011
prev sibling next sibling parent Don <nospam nospam.com> writes:
kenji hara wrote:

 Issue in bugzilla:
   http://d.puremagic.com/issues/show_bug.cgi?id=6207
 Test implementation:
   https://github.com/9rnsr/dmd/commit/52e4491cf6bf9ccf6dccc1fa60581fd6797d39b3
 
 String mixin feature in D is very useful, but its syntax is bit ugly.
 My proposal will make it more softly, and introduces more generative
 programming.
 
 Any thoughts?
 
 Kenji



Yes, it's _less_ ugly, but I just don't think it gets rid of enough of 
the ugliness. You still get that horrid q{ } around the code. I also 
don't see any reason for the feature to be restricted to templates.
The effect is a bit like sugar on syntax anchovies <g>.
Jun 27 2011
prev sibling parent d coder <dlang.coder gmail.com> writes:
--0015175cf88e43e8fe04aaff33af
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 27, 2011 at 2:09 PM, Don <nospam nospam.com> wrote:


 kenji hara wrote:


 Issue in bugzilla:
  http://d.puremagic.com/issues/**show_bug.cgi?id=6207<http://d.puremagic.com/issues/show_bug.cgi?id=6207>
 Test implementation:
  https://github.com/9rnsr/dmd/**commit/**52e4491cf6bf9ccf6dccc1fa60581f**
 d6797d39b3<https://github.com/9rnsr/dmd/commit/52e4491cf6bf9ccf6dccc1fa60581fd6797d39b3>

 String mixin feature in D is very useful, but its syntax is bit ugly.
 My proposal will make it more softly, and introduces more generative
 programming.

 Any thoughts?

 Kenji



 Yes, it's _less_ ugly, but I just don't think it gets rid of enough of the
 ugliness. You still get that horrid q{ } around the code. I also don't see
 any reason for the feature to be restricted to templates.
 The effect is a bit like sugar on syntax anchovies <g>.



Any chance of this feature getting into D language?

I believe that this would be very useful for domain specific embedded
languages.

Regards
- Puneet

--0015175cf88e43e8fe04aaff33af
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<br><br><div class=3D"gmail_quote">On Mon, Jun 27, 2011 at 2:09 PM, Don <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:nospam nospam.com">nospam nospam.com</=
a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0=
 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">

<div class=3D"im">kenji hara wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Issue in bugzilla:<br>
 =A0<a href=3D"http://d.puremagic.com/issues/show_bug.cgi?id=3D6207" target=
=3D"_blank">http://d.puremagic.com/issues/<u></u>show_bug.cgi?id=3D6207</a>=
<br>
Test implementation:<br>
 =A0<a href=3D"https://github.com/9rnsr/dmd/commit/52e4491cf6bf9ccf6dccc1fa=
60581fd6797d39b3" target=3D"_blank">https://github.com/9rnsr/dmd/<u></u>com=
mit/<u></u>52e4491cf6bf9ccf6dccc1fa60581f<u></u>d6797d39b3</a><br>
<br>
String mixin feature in D is very useful, but its syntax is bit ugly.<br>
My proposal will make it more softly, and introduces more generative<br>
programming.<br>
<br>
Any thoughts?<br>
<br>
Kenji<br>
</blockquote>
<br></div>
Yes, it&#39;s _less_ ugly, but I just don&#39;t think it gets rid of enough=
 of the ugliness. You still get that horrid q{ } around the code. I also do=
n&#39;t see any reason for the feature to be restricted to templates.<br>


The effect is a bit like sugar on syntax anchovies &lt;g&gt;.<br>
</blockquote></div><br><div>Any chance of this feature getting into D langu=
age?</div><div><br></div><div>I believe that this would be very useful for =
domain specific embedded languages.</div><div><br></div><div>Regards</div>

<div>- Puneet</div><div><br></div>

--0015175cf88e43e8fe04aaff33af--
Aug 21 2011