www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.dtl - A 'nother way to do this

reply Georg Wrede <georg.wrede nospam.org> writes:
Currently, the DMD template system is making amazing progress!

Being an old warrior, I still haven't entirely put my original ideas 6' 
under. What I'm currently brewing, is a proof-of-concept precompiler. My 
idea is that (or actually was, before these huge leaps Walter and Don 
have made), with a suitable precompiler, one should be able to test 
different ideas about "I want this/that in D", and see for himself 
whether they crumble once put to actual use.

Heh, forcing this thing to everybody who claims to have "this just 
amazing and _essential_ idea to incorporate in D", would reduce some 
garbage bandwidth usage from the newsgroups -- without anybody having to 
shoot 'em down.

There are several different ways one could use this for the advancement 
of D development:

  - a "greyhound race electric rabbit", to "show what the D template 
syntax _could_ be capable of"

  - have the folks with Unbelievable ideas test their ideas themselves 
(before wasting bandwidth here)

  - a general tool to see just _how_ far one can go with a decent meta 
language

  - specifically, to see if we can develop a _clear_ and _obvious_ 
syntax to do hopefully _everything_ that's ever been done in BOOST

  - if folks find ways to do things with this that can't be done with 
the current (or next month's) D template syntax, this might give more 
rigorous food for thought (both for and against) such ideas

  - a theoretical discussion on _what_ should, could, would, be doable 
with a meta language -- the tool is here

Having said all that, this being a preprocessor, makes it possible for 
_any_ language to use _precisely this_ preprocessor to the "up to now, 
in that language, intractable problems". Unfortunately. Now, the aim is 
to incorporate any lessons from this (if there ever will be any), into 
the DMD compiler itself! (Of course, filtered by The Crowd, and whatever 
Walter consideres appropriate.) That is, the existing DMD compiler front 
end.  My dream is that such would give D even more power than anything 
achievable with preprocessor only.

---

The current state is, it's half-way between planning and early alpha. 
Any ideas are welcome. Even if somebody wants to shoot this down, no 
hard feelings!

What I'm currently looking for with this post:

  - hard problems or examples, to challenge if such can be done with 
this approach

  - everyday examples of "ordinary preprocessor things" you've missed 
from the times you used the C(++) preprocessor

  - implicit template instantiation

  - entire classes, and even whole families of classes, wrapped in a 
single template

  - templates parameterized on things other than the function parameter 
types

  - examples of "what can currently be done with C++ and not (yet) with D"

  - what would really be cool, but not even dreamable (until today, be 
it reasonable or not)

  - corporate internal Best Practices examples

  - any (I mean _any at all_) dreams or ideas, or even off-hand 
thoughts, on how the D syntax might be bettered (or worsened :-) ), now 
that we (RSN) have a tool to check those out with.

Such examples might include, ideas from languages currently not thought 
at all implementable in a strongly typed imperative language, weak 
references, adding methods to already compiled classes, the recent C# 
SQL things, a robot language (a bit like Logo), logic programming, 
parser design incorporated right into the language (a la Spirit), etc.

---

The ultimate test would be to

  - reprogram as much as possible from www.boost.org/libs/libraries.htm

  - getting _all_ of that to work would kill the bystanders

  - getting it to work so that an average programmer would understand 
some the metaprogramming involved, would kill the Boost Gurus

  - getting publicity for such, would just kill <your suggestion here>

---

I know, I know, this sounds like the writings of the deranged white 
haired mad scientist. (I'm not gonna argue, heck, even I am not sure of 
this thing. But it does look promising.)

Whatever we do, we should get things done so fast that the C++ 
Design-Committee-Bureaucracy won't be able to follow us until we've 
taken over the world.   :-)

Even if they'll use our precompiler as-is. (Which currently would be no 
problem, unfortunately!)

---

So, let's hear some crap, please!
Nov 29 2005
next sibling parent Don Clugston <dac nospam.com.au> writes:
Georg Wrede wrote:
 Currently, the DMD template system is making amazing progress!
 
 Being an old warrior, I still haven't entirely put my original ideas 6' 
 under. What I'm currently brewing, is a proof-of-concept precompiler. My 
 idea is that (or actually was, before these huge leaps Walter and Don 
 have made), with a suitable precompiler, one should be able to test 
 different ideas about "I want this/that in D", and see for himself 
 whether they crumble once put to actual use.
 
  - specifically, to see if we can develop a _clear_ and _obvious_ syntax 
 to do hopefully _everything_ that's ever been done in BOOST
 
  - if folks find ways to do things with this that can't be done with the 
 current (or next month's) D template syntax, this might give more 
 rigorous food for thought (both for and against) such ideas
 
  - a theoretical discussion on _what_ should, could, would, be doable 
 with a meta language -- the tool is here
 
 Having said all that, this being a preprocessor, makes it possible for 
 _any_ language to use _precisely this_ preprocessor to the "up to now, 
 in that language, intractable problems". Unfortunately. Now, the aim is 
 to incorporate any lessons from this (if there ever will be any), into 
 the DMD compiler itself! (Of course, filtered by The Crowd, and whatever 
 Walter consideres appropriate.) That is, the existing DMD compiler front 
 end.  My dream is that such would give D even more power than anything 
 achievable with preprocessor only.

I wonder how much this is possible without close integration into the language (proper language parsing). I think that the D template system is getting very close to implementing all of the functionality of the C preprocessor. So, it would need to be quite a bit more sophisticated than the C preprocessor.
 What I'm currently looking for with this post:
 
  - hard problems or examples, to challenge if such can be done with this 
 approach
 
  - everyday examples of "ordinary preprocessor things" you've missed 
 from the times you used the C(++) preprocessor

(1) Token pasting. Suppose there was a keyword '__identifier', with syntax: __identifier(StringLiteral) legal anywhere that an identifier is legal. int __identifier("abc"); would be equivalent to: int abc; BUT: the string literal would need to be evaluated before the token pasting occurs, otherwise there's no point. (2) Stringizing. The inverse of the above. int abc; char [] name = __nameof(abc); is equivalent to: char [] name = "abc"; If 'abc' is an alias, it should use the ultimate name.(Ie, it should use the name which is printed out in an error message).
  - implicit template instantiation
  - examples of "what can currently be done with C++ and not (yet) with D"

Using my hack to put delegates into C++ (www.codeproject.com/cpp/fastdelegate.asp). you can retrofit polymorphism, along the lines of Chris Diggin's interface idea. See my comment in http://www.codeproject.com/cpp/retrofitpolymorphism2.asp My method requires implicit template instantiation. (Would be *much* nicer in D). But even better would be to set up proper interfaces, the way he does it with C macros. I think this is something you could do with a preprocessor such as you describe.
  - entire classes, and even whole families of classes, wrapped in a 
 single template
 
  - templates parameterized on things other than the function parameter 
 types

 The ultimate test would be to
 
  - reprogram as much as possible from www.boost.org/libs/libraries.htm
 
  - getting _all_ of that to work would kill the bystanders
 
  - getting it to work so that an average programmer would understand some the
metaprogramming involved, would kill the Boost Gurus

The Boost Metaprogramming Library looks like an easy target, already. It's horribly hacky, and you can't actually do very much with it. Other than implicit template instantiation, I don't think the C++ template system has many advantages left, compared to D.
Nov 30 2005
prev sibling parent reply Ivan Senji <ivan.senji_REMOVE_ _THIS__gmail.com> writes:
Georg Wrede wrote:
 The current state is, it's half-way between planning and early alpha. 
 Any ideas are welcome. Even if somebody wants to shoot this down, no 
 hard feelings!
 

Why would anyone wish to shoot this idea down? It is IMO a good idea, I tryed to make something like this happen too. But as Don Clugston said in his reply to your post a simple preprocesor is hardly going to be enough. If you wish to do anything more complex with syntax you should have a real parser. How is your knowledge on language grammars? I have a working grammar-based parser as a project on dsource and a rather imperfect D-grammar (it doesn't really parse the intire D language). With a fixed grammar and a good lexer hooked up it could easily extend todays D. I have played with it more in the past (too little time now because of the exams), and there are nice things that can be done with it. Just as an example I implemented, a new type of statement in D that looked like this: int[10] numbers; (each[int index] numbers) = index; and it translated to foreach(int index, typeof(numbers[index]) __item; numbers) { __item = index; } And another example was aplply-to-all-like syntax numbers.{writefln(it); it += 2; }; //print each number and add 2 to it. So if you feel this could help you in any way feel free to use it. It is rather hard to start using but it gets easier later. Anyway, good luck with your project!
Nov 30 2005
parent reply Larry Evans <cppljevans cos-internet.com> writes:
On 11/30/2005 02:47 AM, Ivan Senji wrote:
 Georg Wrede wrote:
 
 The current state is, it's half-way between planning and early alpha.


 How is your knowledge on language grammars? I have a working 
 grammar-based parser as a project on dsource and a rather imperfect 
 D-grammar (it doesn't really parse the intire D language). With a fixed 
 grammar and a good lexer hooked up it could easily extend todays D.

 Just as an example I implemented, a new type of statement in D that 
 looked like this:
 
 int[10] numbers;
 
 (each[int index] numbers) = index;
 
 and it translated to
 foreach(int index, typeof(numbers[index]) __item; numbers)
 {
   __item = index;
 }

What I like to see is: (each[int index] int const[10] numbers) = index; IOW, use it to intialize the numbers. This would enable the compiler to know what the values are at each element of numbers; hence, enable better speed optimization. For a concrete example, take a look at little.funvec_.zip in http://tinyurl.com/8bgy5 . In file: little.funvec_constcvec.cpp there's a vector initialized with function pointers. The function pointers have to be entered by the programmer. OTOH, in file: little.funvec_foreach.cpp the function pointers are calculated in a templated CTOR which uses a boost::mpl::for_each to fill the vector. Obviously, funvec_constcvec is faster than funvec_foreach even though they both do *essentially* the same thing, fill a constant vector containing functions pointers (just like a virtual function table ). Could this this be done with your parser? Speed was of prime importance to the original poster of the thread which prompted the little.funvec_* code: http://groups.google.com/group/comp.lang.c++.moderated/msg/23c238cdda9cd878
Dec 02 2005
next sibling parent Ivan Senji <ivan.senji_REMOVE_ _THIS__gmail.com> writes:
Larry Evans wrote:
 What I like to see is:
 
 (each[int index] int const[10] numbers) = index;
 
 IOW, use it to intialize the numbers.  This would
 enable the compiler to know what the values are
 at each element of numbers; hence, enable better
 speed optimization.

Interesting.
 
 For a concrete example, take a look at little.funvec_.zip
 in http://tinyurl.com/8bgy5 .  In file:
 
   little.funvec_constcvec.cpp
 
 there's a vector initialized with function pointers.  The
 function pointers have to be entered by the programmer.
 
 OTOH, in file:
 
   little.funvec_foreach.cpp
 
 the function pointers are calculated in a templated CTOR
 which uses a boost::mpl::for_each to fill the vector.
 
 Obviously, funvec_constcvec is faster than funvec_foreach
 even though they both do *essentially* the same thing,
 fill a constant vector containing functions pointers
 (just like a virtual function table ).  

Didn't really look ad that code because I don't like the way it is structured (unreadable, sorry) :), but...
 Could this
 this be done with your parser?

The question is "is there a way to write this in D (using templates)?" If the answer is yes, then it can be done, the only problem is you have to write code in the grammar definition that describes that translation to normal D. But it takes free time, and that I do not have :(
Dec 02 2005
prev sibling parent reply Don Clugston <dac nospam.com.au> writes:
Larry Evans wrote:
 On 11/30/2005 02:47 AM, Ivan Senji wrote:
 
 Georg Wrede wrote:

 The current state is, it's half-way between planning and early alpha.


[snip]
 How is your knowledge on language grammars? I have a working 
 grammar-based parser as a project on dsource and a rather imperfect 
 D-grammar (it doesn't really parse the intire D language). With a 
 fixed grammar and a good lexer hooked up it could easily extend todays D.

[snip]
 Just as an example I implemented, a new type of statement in D that 
 looked like this:

 int[10] numbers;

 (each[int index] numbers) = index;

 and it translated to
 foreach(int index, typeof(numbers[index]) __item; numbers)
 {
   __item = index;
 }

What I like to see is: (each[int index] int const[10] numbers) = index; IOW, use it to intialize the numbers. This would enable the compiler to know what the values are at each element of numbers; hence, enable better speed optimization.

This sort of thing can already be done with D templates. ------- template square(int n) { const int square = n*n; } const [] numbers = generateArray!(10, square); // const int [] numbers = [0, 1, 4, 9, 16, 25, ...] const [] smallfactorials = generateArray!(20, factorial); // const ulong [] smallfactorials = [0, 1, 2, 6, 20, ... ] ------- const T [] generateArray!(int limits, const T metafunction!(int) f ) is a template I've made, it can accept metafunctions returning any constant type. (For example, f could return generateArray!(5, g) where g returns a string literal). Note that it's also possible for the array length itself to be calculated at compile time. BTW, anyone have a better name for generateArray ? I've also tried calling it "generateLookupTable" and "staticforeach", neither of which seem appropriate.
Dec 05 2005
parent reply Bruno Medeiros <daiphoenixNO SPAMlycos.com> writes:
Don Clugston wrote:
 
 This sort of thing can already be done with D templates.
 -------
 template square(int n)
 {
   const int square = n*n;
 }
 
 const [] numbers = generateArray!(10, square);
 // const int [] numbers = [0, 1, 4, 9, 16, 25, ...]
 
 const [] smallfactorials = generateArray!(20, factorial);
 // const ulong [] smallfactorials = [0, 1, 2, 6, 20, ... ]
 -------
 const T [] generateArray!(int limits, const T metafunction!(int) f )
 is a template I've made, it can accept metafunctions
 returning any constant type. (For example, f could return
 generateArray!(5, g) where g returns a string literal).
 

the code(for generateArray)? -- Bruno Medeiros - CS/E student "Certain aspects of D are a pathway to many abilities some consider to be... unnatural."
Dec 08 2005
parent reply Don Clugston <dac nospam.com.au> writes:
Bruno Medeiros wrote:
 Don Clugston wrote:
 
 This sort of thing can already be done with D templates.
 -------
 template square(int n)
 {
   const int square = n*n;
 }

 const [] numbers = generateArray!(10, square);
 // const int [] numbers = [0, 1, 4, 9, 16, 25, ...]

 const [] smallfactorials = generateArray!(20, factorial);
 // const ulong [] smallfactorials = [0, 1, 2, 6, 20, ... ]
 -------
 const T [] generateArray!(int limits, const T metafunction!(int) f )
 is a template I've made, it can accept metafunctions
 returning any constant type. (For example, f could return
 generateArray!(5, g) where g returns a string literal).

the code(for generateArray)?

It's not pretty, but it's trivial, really. Currently done with a hack (like a C++ preprocessor). --------------- private import hack.hackgenerate; template generateArray(alias generator, int n) { const typeof(generator!(0)) [] generateArray = hackgenerate!(n, generator); } ---------------- I autogenerate a file with the first few entries looking like this: (hackgenerate.d). Has entries for num=1 to 128. I have to use this hack, because D doesn't have proper array literals yet, so arrays can't be concatentated or pulled apart at compile time. As you can imagine, the D code for making the hackgenerate file is very short. -------------- // Autogenerated file - do not edit template hackgenerate(int num, alias gen) { pragma(msg, "Table is too large!"); static assert(0); } template hackgenerate(int num :0, alias gen) { const typeof(gen!(0)) [] hackgenerate = [ gen!(0) ]; } template hackgenerate(int num :1, alias gen) { const typeof(gen!(0)) [] hackgenerate = [ gen!(0), gen!(1) ]; } template hackgenerate(int num :2, alias gen) { const typeof(gen!(0)) [] hackgenerate = [ gen!(0), gen!(1), gen!(2) ]; } --------------
Dec 08 2005
parent reply Larry Evans <cppljevans cos-internet.com> writes:
On 12/09/2005 01:36 AM, Don Clugston wrote:
[snip]
 
 It's not pretty, but it's trivial, really. Currently done with a hack 
 (like a C++ preprocessor).
 ---------------
 private import hack.hackgenerate;
 
 template generateArray(alias generator, int n)
 {
      const typeof(generator!(0)) [] generateArray = hackgenerate!(n, 
 generator);
 }
 ----------------
 I autogenerate a file with the first few entries looking like this: 
 (hackgenerate.d). Has entries for num=1 to 128.
 
 I have to use this hack, because D doesn't have proper array literals 
 yet, so arrays can't be concatentated or pulled apart at compile time.
 As you can imagine, the D code for making the hackgenerate file is very 
 short.
 
 --------------
 // Autogenerated file - do not edit

Don, since I've already done my own hack (in PP_RECFLD.zip of http://boost-consulting.com/vault/index.php?&directory=Template%20Metaprogramming ) I was hoping to avoid this. I'd really like something that can just be done in the language itself without any preprocessing.
Dec 09 2005
parent reply Don Clugston <dac nospam.com.au> writes:
 Don, since I've already done my own hack
 (in PP_RECFLD.zip of
 http://boost-consulting.com/vault/index.php?&directory=Templa
e%20Metaprogramming 
 
 ) I was hoping to avoid this.  I'd really like something
 that can just be done in the language itself without any preprocessing.

I agree entirely. I've done enough of it too. That was just a temporary workaround. The main need for the the hack is the absence of array literals, a feature which Walter has said is coming (but not for a while, because it's tricky to implement). Right now, it is possible to avoid the hack for the special case of strings. But unfortunately, it needs a second hack, because const char [] ~= const char is not evaluated at compile time. You need to convert char into char[]. I'm hoping that this limitation will be removed very soon, ie this year, so that we can at least do one special case perfectly. Once we have that, array literals will become a more fun problem for Walter to work on, so it will probably gain increased priority.
Dec 12 2005
parent Larry Evans <cppljevans cos-internet.com> writes:
On 12/12/2005 02:12 AM, Don Clugston wrote:
 Don, since I've already done my own hack
 (in PP_RECFLD.zip of
 http://boost-consulting.com/vault/index.php?&directory=Templa
e%20Metaprogramming 

 ) I was hoping to avoid this.  I'd really like something
 that can just be done in the language itself without any preprocessing.

I agree entirely. I've done enough of it too. That was just a temporary workaround. The main need for the the hack is the absence of array literals, a feature which Walter has said is coming (but not for a while, because it's tricky to implement).

Thanks very much Don. I've been thinking about moving to D, and this is one more reason to do so (when the new feature is added). However, I'm still wondering if something like mpl::fold is possible, in particular, with placeholders (i.e. mpl::arg). I've needed this for proto_static_disp.zip in: http://boost-consulting.com/vault/ in the 'Template Metaprogramming' directory. I'd also like to see if the policy_ptr can't be somehow incorporated into D as a replacement for the current GC. AFAICT, it would involve adding to each stack frame something like the selected_fields_description_of<FieldsVisitor, RECORD >, defined in: http://cvs.sourceforge.net/viewcvs.py/boost-sandbox/boost-sandbox/boost/fields_visitor/fields_visitor.hpp where RECORD corresponds to the stack frame and FieldsVisitor corresponds to the type of garbage collection needed. But, of course, this is all based on smart_ptr's, which, as pointed out in another post to this newsgroup, D doesn't handle to well because it has no override for operator= :( Oh well. -best regards, Larry
Dec 12 2005