www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Thesis on metaprogramming in D

reply Leandro Lucarella <llucarella integratech.com.ar> writes:
Hello. My name is Leandro Lucarella, I'm finishing my grade on Computer 
Engineering in the UBA (University of Buenos Aires, Argentina) and I'm 
planning to do my thesis on meta-programming and I'd like to do it using 
D (because is the language that is closer to my ideal language, if it 
just had Python-like syntax... ;).

The center of my thesis would be adding meta-classes[1] to D, and maybe 
some other meta-programming features D could lack of (maybe some kind of 
access to the AST like XLR[2]).

I'd like to know if you think this is:
1) Doable.
2) Useful.
3) A feature that can be included in the official D specification.

In case it's not clear, I'd like to add support for meta-classes to the 
D language itself, not provide some nasty library to ease the job using 
the existing tools. If there is any D guru that is willing to help me 
out and give me some guidance, that would be great too.

TIA

[1] Like Python meta-classes but, of course, resolved at compile time.
     http://www.python.org/doc/essays/metaclasses/
[2] Extensible Language and Runtime
     http://xlr.sourceforge.net/

-- 
Leandro Lucarella
Integratech S.A.
4571-5252
Nov 27 2006
parent reply Steve Horne <stephenwantshornenospam100 aol.com> writes:
On Mon, 27 Nov 2006 15:23:33 -0300, Leandro Lucarella
<llucarella integratech.com.ar> wrote:

The center of my thesis would be adding meta-classes[1] to D, and maybe 
some other meta-programming features D could lack of (maybe some kind of 
access to the AST like XLR[2]).

I can't really offer any useful support, but if I understand what you are saying, I can offer encouragement. My view of the ideal handling of metaprogramming is that it should have access to all the features that are available at runtime. The only languages that I know that genuinely achieve that are the Lisp-alikes such as Scheme. My ideal is being able to define 'quoted' blocks that are parsed and translated to ASTs, with those ASTs being processed by other code to derive new ASTs which are then submitted for the final back-end optimisation and code generation. Which is not anti-templates as such. I like templates. It just means that, in principle, you could define an alternative template mechanism as a library. The nearest you get to this at the moment (outside Lisp-alikes), at least that I know of, are languages like Python where you can access the AST, including the AST of generated code, at run-time. Thanks for the XLR link, by the way - very interesting, and possibly capable of exactly what I described above, though I've not read it properly yet. -- Remove 'wants' and 'nospam' from e-mail.
Nov 27 2006
next sibling parent reply "Andrey Khropov" <andkhropov_nosp m_mtu-net.ru> writes:
Steve Horne wrote:

 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.
 
 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.

Nemerle does exactly that way! -- AKhropov
Nov 28 2006
next sibling parent reply Johan Granberg <lijat.meREM OVE.gmail.com> writes:
Andrey Khropov wrote:

 Steve Horne wrote:
 
 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.
 
 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.

Nemerle does exactly that way!

Maybe this is a little of topic in a D news group but since nemerle came up, What is the speed of nemerle related to other languages such as C D C++ , Java or python?
Nov 28 2006
parent reply Bill Baxter <wbaxter gmail.com> writes:
Johan Granberg wrote:
 Andrey Khropov wrote:
 

 Maybe this is a little of topic in a D news group but since nemerle came up,
 What is the speed of nemerle related to other languages such as C D C++ ,
 Java or python?

It runs on Microsoft's .NET CLR, so it's probably about the same as C#. --bb
Nov 28 2006
parent "Andrey Khropov" <andkhropov_nosp m_mtu-net.ru> writes:
Bill Baxter wrote:

 It runs on Microsoft's .NET CLR, so it's probably about the same as C#.

Yes, my experiments have shown that this is generally the case. No surprise, because it's a statically typed language unlike Scheme/Lisp. -- AKhropov
Nov 28 2006
prev sibling parent reply Don Clugston <dac nospam.com.au> writes:
Andrey Khropov wrote:
 Steve Horne wrote:
 
 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.

 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.

Nemerle does exactly that way!

When I looked at the Nemerle website, I didn't see much that couldn't be done easily with D templates. It would be interesting to find something it can do, that D can't.
Nov 28 2006
next sibling parent reply Daniel Keep <daniel.keep+lists gmail.com> writes:
Don Clugston wrote:
 Andrey Khropov wrote:
 
 Steve Horne wrote:

 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.

 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.

Nemerle does exactly that way!

When I looked at the Nemerle website, I didn't see much that couldn't be done easily with D templates. It would be interesting to find something it can do, that D can't.

 def sr = System.IO.StreamReader ("SomeFile.txt");

Immutable run-time variables. You can assign anything to them, and then it can't change. AFAIK, D's const doesn't let you do that.
 someFunction(if (cond) expr_1 else expr_2);

No statements--everything is an expression. And yes, it is VERY handy :)
 import some.namespace.SomeClass;

Import contents of static classes as well as modules.
 match (control) {
   | button is Button => ...
   | listv is ListView => ...
   | _ => ... // null case
 }

switch eat your heart out. You can use literal, pattern matching... just about anything.
 def res2 = frobnicate (7, do_qux = true,
                           do_baz = false,
                           do_bar = true);

Named parameters. D doesn't even have hashtable literals, so faking it would be a complete pain.
 ()

That's the void literal. Basically, makes void act like any other type. Try writing generic code where functions can have void return values, and you'll end up writing everything twice since you can't have void variables.
 variant RgbColor {
   | Red
   | Yellow
   | Green
   | Different {
       red : float;
       green : float;
       blue : float;
     }
 }

Different to a union since it remembers what kind of thing is being stored. Fun trick with trees is to use a variant to store either a branch or a leaf.
 macro while_macro (cond, body)
 syntax ("while", "(", cond, ")", body) {
   <[
     def loop () {
       when ($cond) {
         $body;
         loop ()
       }
     }
     loop ()
   ]>
 }

Do THAT with templates :3
 def x = 3;
 System.Console.WriteLine ($"My value of x is $x and I'm happy");

expands to
 def x = 3;
 System.Console.WriteLine ({
   def sb = System.Text.StringBuilder ("My value of x is ");
   sb.Append (x.ToString ());
   sb.Append (" and I'm happy");
   sb.ToString ()
 });


Even if you could fake the one just above that, there's no way you can do that in D. It would be foolish to think that Nemerle isn't an amazingly powerful language. D is good, and its' templates are very powerful, but they're not THAT powerful. There is always more to learn :) -- Daniel
Nov 28 2006
next sibling parent Don Clugston <dac nospam.com.au> writes:
Daniel Keep wrote:
 Don Clugston wrote:
 Andrey Khropov wrote:

 Steve Horne wrote:

 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.

 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.

Nemerle does exactly that way!

When I looked at the Nemerle website, I didn't see much that couldn't be done easily with D templates. It would be interesting to find something it can do, that D can't.

> def sr = System.IO.StreamReader ("SomeFile.txt"); Immutable run-time variables. You can assign anything to them, and then it can't change. AFAIK, D's const doesn't let you do that. > someFunction(if (cond) expr_1 else expr_2); No statements--everything is an expression. And yes, it is VERY handy :) > import some.namespace.SomeClass; Import contents of static classes as well as modules. > match (control) { > | button is Button => ... > | listv is ListView => ... > | _ => ... // null case > } switch eat your heart out. You can use literal, pattern matching... just about anything. > def res2 = frobnicate (7, do_qux = true, > do_baz = false, > do_bar = true); Named parameters. D doesn't even have hashtable literals, so faking it would be a complete pain. > () That's the void literal. Basically, makes void act like any other type. Try writing generic code where functions can have void return values, and you'll end up writing everything twice since you can't have void variables. > variant RgbColor { > | Red > | Yellow > | Green > | Different { > red : float; > green : float; > blue : float; > } > } Different to a union since it remembers what kind of thing is being stored. Fun trick with trees is to use a variant to store either a branch or a leaf. > macro while_macro (cond, body) > syntax ("while", "(", cond, ")", body) { > <[ > def loop () { > when ($cond) { > $body; > loop () > } > } > loop () > ]> > } Do THAT with templates :3

The 'lazy evaluation' page has a similar example. :-)
  >> def x = 3;
  >> System.Console.WriteLine ($"My value of x is $x and I'm happy");
  >
  > expands to
  >
  >> def x = 3;
  >> System.Console.WriteLine ({
  >>   def sb = System.Text.StringBuilder ("My value of x is ");
  >>   sb.Append (x.ToString ());
  >>   sb.Append (" and I'm happy");
  >>   sb.ToString ()
  >> });
 
 Even if you could fake the one just above that, there's no way you can 
 do that in D.

You could if we got the identifier() keyword which I've previously proposed. Admittedly, it still wouldn't be pretty. But there's really because there's no support for embedded variables. But doing the same thing for System.Console.WriteLine("My value of x is ", x, " and I'm happy"); is trivial with the new tuples.
 It would be foolish to think that Nemerle isn't an amazingly powerful 
 language.  D is good, and its' templates are very powerful, but they're 
 not THAT powerful.

I still have the impression that the functionality isn't much different.
 There is always more to learn :)
 
     -- Daniel

Nov 28 2006
prev sibling next sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Daniel Keep wrote:
  > def sr = System.IO.StreamReader ("SomeFile.txt");
 
 Immutable run-time variables.  You can assign anything to them, and then 
 it can't change.  AFAIK, D's const doesn't let you do that.
 

What's the difference from C# and Java's 'final'? Is the variable transitively unchangeable, or just the immediate value? -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Nov 30 2006
parent reply Daniel Keep <daniel.keep+lists gmail.com> writes:
Bruno Medeiros wrote:
 Daniel Keep wrote:
 
  > def sr = System.IO.StreamReader ("SomeFile.txt");

 Immutable run-time variables.  You can assign anything to them, and 
 then it can't change.  AFAIK, D's const doesn't let you do that.

What's the difference from C# and Java's 'final'? Is the variable transitively unchangeable, or just the immediate value?

I'm not really sure what 'transitively unchangeable' means... >_< Firstly, isn't 'final' just used for class members? This is used for local variables (or members, if I remember correctly). As for the second sentence, it's immutable storage. So you can't change what's stored in the variable, but if you've stored a reference, then you can mutate what's being referenced. -- Daniel
Nov 30 2006
parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Daniel Keep wrote:
 Bruno Medeiros wrote:
 Daniel Keep wrote:

  > def sr = System.IO.StreamReader ("SomeFile.txt");

 Immutable run-time variables.  You can assign anything to them, and 
 then it can't change.  AFAIK, D's const doesn't let you do that.

What's the difference from C# and Java's 'final'? Is the variable transitively unchangeable, or just the immediate value?

I'm not really sure what 'transitively unchangeable' means... >_<

case of references or pointers you can't change what is being referenced (and recursively so on). It's a transitive (recursive) read-only.
 Firstly, isn't 'final' just used for class members?  This is used for 
 local variables (or members, if I remember correctly).  As for the 
 second sentence, it's immutable storage.  So you can't change what's 
 stored in the variable, but if you've stored a reference, then you can 
 mutate what's being referenced.
 
     -- Daniel

Then that is just like 'final'. But I've realize now that this 'final' construct is only called 'final' in Java. The C# equivalent is actually called 'readonly' (http://msdn2.microsoft.com/en-us/library/acdd6hb7.aspx), whereas I thought it was 'final' too. Also, indeed the C# 'readonly' can only be used on fields, but the Java 'final' can be used on any variable, such as locals. And like Don said D has that feature too, but hideously hidden behind 'const'. -_-' -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Nov 30 2006
prev sibling parent reply Don Clugston <dac nospam.com.au> writes:
Daniel Keep wrote:
 Don Clugston wrote:
 When I looked at the Nemerle website, I didn't see much that couldn't 
 be done easily with D templates. It would be interesting to find 
 something it can do, that D can't.

> def sr = System.IO.StreamReader ("SomeFile.txt"); Immutable run-time variables. You can assign anything to them, and then it can't change. AFAIK, D's const doesn't let you do that.

It does, but only for class member variables. There are two 'const's in D, one for literal constants, and the other which is once-only assign, and should probably be called 'final' instead. class SomeClass { const StreamReader sr; this(char [] filename) { sr = System.IO.StreamReader(filename);} }; [snip]
  > macro while_macro (cond, body)
  > syntax ("while", "(", cond, ")", body) {
  >   <[
  >     def loop () {
  >       when ($cond) {
  >         $body;
  >         loop ()
  >       }
  >     }
  >     loop ()
  >   ]>
  > }
 
 Do THAT with templates :3

void While(lazy bool cond, void delegate() body) { for (; cond; ) body(); } <g> Or if I can't have for() as a primitive: void While(lazy bool cond, void delegate() body) { loopstart: if(cond) goto done; body(); goto loopstart; done: }
 It would be foolish to think that Nemerle isn't an amazingly powerful 
 language.  D is good, and its' templates are very powerful, but they're 
 not THAT powerful.
 
 There is always more to learn :)

Very true. It does seem, though, that Nemerle and D are exploring a quite similar 'paradigm space' (so there's probably quite a bit each can learn from the other). Yet neither has really caught up with Lisp. Yet.
Nov 30 2006
next sibling parent reply Daniel Keep <daniel.keep+lists gmail.com> writes:
Don Clugston wrote:
 Daniel Keep wrote:
  > def sr = System.IO.StreamReader ("SomeFile.txt");

 Immutable run-time variables.  You can assign anything to them, and 
 then it can't change.  AFAIK, D's const doesn't let you do that.

It does, but only for class member variables. There are two 'const's in D, one for literal constants, and the other which is once-only assign, and should probably be called 'final' instead. class SomeClass { const StreamReader sr; this(char [] filename) { sr = System.IO.StreamReader(filename);} };

The thing that *really* struck me was that when I wrote some stuff in Nemerle, it was amazing how often I didn't actually need mutable variables. I wonder what affect having immutables in D would have optimisation-wise...
 [snip]
 
  > macro while_macro (cond, body)
  > syntax ("while", "(", cond, ")", body) {
  >   <[
  >     def loop () {
  >       when ($cond) {
  >         $body;
  >         loop ()
  >       }
  >     }
  >     loop ()
  >   ]>
  > }

 Do THAT with templates :3

void While(lazy bool cond, void delegate() body) { for (; cond; ) body(); } <g> Or if I can't have for() as a primitive: void While(lazy bool cond, void delegate() body) { loopstart: if(cond) goto done; body(); goto loopstart; done: }

True, but it's not quite the same. The D version has that outlying closing paren which is, frankly, really ugly. Also, the D version is doing a function call, whereas the Nemerle version is actually expanding an AST. Of course, these things could probably be solved with a little more work from the compiler: allowing trailing delegate literals, and inlining delegate literals. *drool*
 It would be foolish to think that Nemerle isn't an amazingly powerful 
 language.  D is good, and its' templates are very powerful, but 
 they're not THAT powerful.

 There is always more to learn :)

Very true. It does seem, though, that Nemerle and D are exploring a quite similar 'paradigm space' (so there's probably quite a bit each can learn from the other). Yet neither has really caught up with Lisp. Yet.

No language will ever catch up to LISP: the only way to do that would be to become LISP, and then you have to deal with all those bloody parentheses :3 -- Daniel
Nov 30 2006
next sibling parent Brad Anderson <brad dsource.org> writes:
Daniel Keep wrote:
 Don Clugston wrote:
 Daniel Keep wrote:
  > def sr = System.IO.StreamReader ("SomeFile.txt");

 Immutable run-time variables.  You can assign anything to them, and
 then it can't change.  AFAIK, D's const doesn't let you do that.

It does, but only for class member variables. There are two 'const's in D, one for literal constants, and the other which is once-only assign, and should probably be called 'final' instead. class SomeClass { const StreamReader sr; this(char [] filename) { sr = System.IO.StreamReader(filename);} };

The thing that *really* struck me was that when I wrote some stuff in Nemerle, it was amazing how often I didn't actually need mutable variables. I wonder what affect having immutables in D would have optimisation-wise...

Not sure, but it would also help get closer to the Erlang model of concurrent programming via message passing. Erlang is all immutables, iirc. CL-MUPROC, a Lisp attempt at this Erlang model, treats immutability as a 'guideline' and let's the developer be grown-up. But it uses OS threads instead of Erlang's lightweight processes. :(
 
 [snip]

  > macro while_macro (cond, body)
  > syntax ("while", "(", cond, ")", body) {
  >   <[
  >     def loop () {
  >       when ($cond) {
  >         $body;
  >         loop ()
  >       }
  >     }
  >     loop ()
  >   ]>
  > }

 Do THAT with templates :3

void While(lazy bool cond, void delegate() body) { for (; cond; ) body(); } <g> Or if I can't have for() as a primitive: void While(lazy bool cond, void delegate() body) { loopstart: if(cond) goto done; body(); goto loopstart; done: }

True, but it's not quite the same. The D version has that outlying closing paren which is, frankly, really ugly. Also, the D version is doing a function call, whereas the Nemerle version is actually expanding an AST. Of course, these things could probably be solved with a little more work from the compiler: allowing trailing delegate literals, and inlining delegate literals. *drool*
 It would be foolish to think that Nemerle isn't an amazingly powerful
 language.  D is good, and its' templates are very powerful, but
 they're not THAT powerful.

 There is always more to learn :)

Very true. It does seem, though, that Nemerle and D are exploring a quite similar 'paradigm space' (so there's probably quite a bit each can learn from the other). Yet neither has really caught up with Lisp. Yet.

No language will ever catch up to LISP: the only way to do that would be to become LISP, and then you have to deal with all those bloody parentheses :3

;)
 
     -- Daniel

Nov 30 2006
prev sibling parent Steve Horne <stephenwantshornenospam100 aol.com> writes:
On Fri, 01 Dec 2006 00:35:36 +0800, Daniel Keep
<daniel.keep+lists gmail.com> wrote:

Don Clugston wrote:

 It does seem, though, that Nemerle and D are exploring a quite similar 
 'paradigm space' (so there's probably quite a bit each can learn from 
 the other). Yet neither has really caught up with Lisp. Yet.

No language will ever catch up to LISP: the only way to do that would be to become LISP, and then you have to deal with all those bloody parentheses :3

Not sure I agree entirely. In my view, the things that define the Lisp language are... 1. The metaprogramming stuff, with a rather sparse core language being extended using library macros. 2. The list based on pair (cons) objects. 3. The explicit AST-structuring using parentheses. Well, Nemerle has metaprogramming that is basically what Lisp does with some extra tweaks. Nemerle has pair-object based lists. And Nemerle has explicit AST-structuring using parentheses (it has a pre-parse stage to handle this). Nemerle is heavily Lisp influenced. It just takes ideas from other places too. Personally, I think it may well have overtaken Lisp - just as powerful (within the limits of the .NET platform), yet more immediately usable. -- Remove 'wants' and 'nospam' from e-mail.
Dec 01 2006
prev sibling parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Don Clugston wrote:
 Daniel Keep wrote:
 Do THAT with templates :3

void While(lazy bool cond, void delegate() body) { for (; cond; ) body(); }

Heh, incorrect answer. I don't see a template :P. </nitpick>
Nov 30 2006
prev sibling parent Steve Horne <stephenwantshornenospam100 aol.com> writes:
On Tue, 28 Nov 2006 13:59:20 +0100, Don Clugston <dac nospam.com.au>
wrote:

Andrey Khropov wrote:
 Steve Horne wrote:
 
 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.

 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.

Nemerle does exactly that way!


Cool!
When I looked at the Nemerle website, I didn't see much that couldn't be 
done easily with D templates. It would be interesting to find something 
it can do, that D can't.

That doesn't worry me much, to be honest. I don't want another Scheme (or at least my perception of Scheme) where you start each journey not by getting into your car, but by re-inventing the wheel. The point in my view is to be able to opt of the standard toolkit when in those special cases where it doesn't quite fit, without having to opt out of the language completely. And so I imagine most generic programming tasks in Nemerle would use the .NET 2 generics. These aren't as flexible as C++ templates, let alone D ones, but they have the advantage that they are closure based and not prone to bloat. Nevertheless, there are useful things that you can't easily do with them. It sounds like Nemerle lets you work around that limitation when you need to. Being a .NET thing, it's presumably more of an apps-level language than systems-level, but maybe it has some transferrable ideas? -- Remove 'wants' and 'nospam' from e-mail.
Nov 28 2006
prev sibling parent Leandro Lucarella <llucarella integratech.com.ar> writes:
Steve Horne escribió:
 On Mon, 27 Nov 2006 15:23:33 -0300, Leandro Lucarella
 <llucarella integratech.com.ar> wrote:
 
 The center of my thesis would be adding meta-classes[1] to D, and maybe 
 some other meta-programming features D could lack of (maybe some kind of 
 access to the AST like XLR[2]).

I can't really offer any useful support, but if I understand what you are saying, I can offer encouragement.

Thanks for your support. Any other thing to say about the original topic? I've noticed a tendency of this list to get rather silly[1]... I mean, off-topic =) [1] I should posted this to a Python mailing list ;) -- Leandro Lucarella Integratech S.A. 4571-5252
Nov 30 2006