www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The Language Machine - a metalanguage in D with D-to-D translator

reply Peri Hankey <mpah thegreen.co.uk> writes:
Hello

I have just uploaded my language machine, toolkit for language and 
grammar to sourceforge. The  url is http://languagemachine.sourceforge.net.

Here's the summary as posted to freshmeat:

The Language Machine is a free software toolkit for language and 
grammar: a library, main program, and self-hosted rule compilers with 
one frontend. The system is easy to use. It directly implements 
unrestricted rule-based grammars with actions and external interfaces. 
In its present form the language machine consists of a shared library, a 
minimal main program, and a metalanguage (LMN - language machine/meta 
notation). The shared library is written in the Digital Mars D language 
using gdc, the D language front end for the GNU gcc compiler collection. 
The LMN compilers share a common frontend and translate into C and D as 
well as producing C, D, and shell script wrappers for an internal 
format. There is a proof-of-concept interface to Fabrice Bellard's tiny 
C compiler, with a working example. LMN source is compatible with the 
wikipedia text format - the project's web pages are an example of how 
the language machine can be used. Diagnostics include the lm-diagram, a 
diagram which show what happens when unrestricted grammar rules are applied.

The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process. It has also been used to translate DMDscript. The 
translator is a ruleset with a frontend and a backend, and it does a 
fairly thorough analysis, but it may well be inaccurate in its idea of D 
grammar.

C is used only for external interfaces - the whole system is written 
entirely in D with a small metalanguage bootstrap.

Your comments and ideas are welcome.

Peri
Jul 11 2005
next sibling parent reply Nod <Nod_member pathlink.com> writes:
In article <dau3bc$1hj0$1 digitaldaemon.com>, Peri Hankey says...
Hello

I have just uploaded my language machine, toolkit for language and 
grammar to sourceforge. The  url is http://languagemachine.sourceforge.net.

Here's the summary as posted to freshmeat:

The Language Machine is a free software toolkit for language and 
grammar: a library, main program, and self-hosted rule compilers with 
one frontend. The system is easy to use. It directly implements 
unrestricted rule-based grammars with actions and external interfaces. 
In its present form the language machine consists of a shared library, a 
minimal main program, and a metalanguage (LMN - language machine/meta 
notation). The shared library is written in the Digital Mars D language 
using gdc, the D language front end for the GNU gcc compiler collection. 
The LMN compilers share a common frontend and translate into C and D as 
well as producing C, D, and shell script wrappers for an internal 
format. There is a proof-of-concept interface to Fabrice Bellard's tiny 
C compiler, with a working example. LMN source is compatible with the 
wikipedia text format - the project's web pages are an example of how 
the language machine can be used. Diagnostics include the lm-diagram, a 
diagram which show what happens when unrestricted grammar rules are applied.

The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process. It has also been used to translate DMDscript. The 
translator is a ruleset with a frontend and a backend, and it does a 
fairly thorough analysis, but it may well be inaccurate in its idea of D 
grammar.

C is used only for external interfaces - the whole system is written 
entirely in D with a small metalanguage bootstrap.

Your comments and ideas are welcome.

Peri

On first glance, the source looks really clean, and the meta-language well thought out. I'm not too fond of the syntax, but I guess one could change that quite easily, no? The information on the webpage is really good too; I was glued to the guide for quite a while. Me likes meta-languages :) A few questions though: * What exactly does the D-to-D translator do? Have I understood correctly if I say that it lexes D source, verifying syntax correctness, and outputs D source in a standard format? * And what is the work division between the frontend/backend? * Is the language machine meant primarily for interpreted languages, or could it be used as, say, a frontend for GCC? I'll post more thorough impressions when I have had time to play with it a bit. As of currently, I really need to sleep. Good work! -Nod-
Jul 12 2005
parent reply Peri Hankey <mpah thegreen.co.uk> writes:
Nod wrote:
 In article <dau3bc$1hj0$1 digitaldaemon.com>, Peri Hankey says...
 
Hello

I have just uploaded my language machine, toolkit for language and 
grammar to sourceforge. The  url is http://languagemachine.sourceforge.net.

Here's the summary as posted to freshmeat:

The Language Machine is a free software toolkit for language and 
grammar: a library, main program, and self-hosted rule compilers with 
one frontend. The system is easy to use. It directly implements 
unrestricted rule-based grammars with actions and external interfaces. 
In its present form the language machine consists of a shared library, a 
minimal main program, and a metalanguage (LMN - language machine/meta 
notation). The shared library is written in the Digital Mars D language 
using gdc, the D language front end for the GNU gcc compiler collection. 
The LMN compilers share a common frontend and translate into C and D as 
well as producing C, D, and shell script wrappers for an internal 
format. There is a proof-of-concept interface to Fabrice Bellard's tiny 
C compiler, with a working example. LMN source is compatible with the 
wikipedia text format - the project's web pages are an example of how 
the language machine can be used. Diagnostics include the lm-diagram, a 
diagram which show what happens when unrestricted grammar rules are applied.

The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process. It has also been used to translate DMDscript. The 
translator is a ruleset with a frontend and a backend, and it does a 
fairly thorough analysis, but it may well be inaccurate in its idea of D 
grammar.

C is used only for external interfaces - the whole system is written 
entirely in D with a small metalanguage bootstrap.

Your comments and ideas are welcome.

Peri

On first glance, the source looks really clean, and the meta-language well thought out. I'm not too fond of the syntax, but I guess one could change that quite easily, no? The information on the webpage is really good too; I was glued to the guide for quite a while. Me likes meta-languages :)

That's terrific - most of the people I have explained it to fall asleep instantly - a good cure for insomnia. It's easy to change the syntax, but hard to find the right one. I was looking for a minimal syntax to get the job done.
 
 A few questions though:
 * What exactly does the D-to-D translator do? Have I understood correctly if I
 say that it lexes D source, verifying syntax correctness, and outputs D source
 in a standard format?

The D-to-D translator started because I was looking for ways to integrate grammar rules directly as an extension of D (this would probably have been a mistake). The best way to prove to myself that my rules were understanding D source was to produce something could be compiled and tested. Also there was a lot of talk here about experimenting with things that could be done with D, so I thought there might be some interest
 * And what is the work division between the frontend/backend?

Both frontend and backend are just rules: the LMN notation allows you to produce prepackaged analyses such as '?{' whatnot :F expression :X '}' <- thing :{ whatnotkindofthing :{ operator :F :X }}; This means: '?{' then a "whatnot" (with a value we call F) followed by an "expression" (with a value we call X) and by '}' can be treated as a "thing" which produces a bag containing a "whatnotkindofthing" which produces a bag containing an "operator" with the F and the X as its values. A backend rule to handle this would receive a "whatnotkindofthing" in the first place. It can strip off some of the packaging: whatnotkindofthing :Q <- code - 'hoho it\'s a ' Q eoc; Or it can look inside the bag: whatnotkindofthing : thiskindofthing :T :F :X <- code - ..... Here it looks in the bag and forces the "operator" to be treated as a "thiskindofthing". So the frontend rules produce prepackaged stuff that is recognised and translated into some external representation by the backend. There's a hypersimple example of how this is done in the rules for converting mediawiki text to html: http://languagemachine.sourceforge.net/mediawiki.html. The prepackaged stuff is in a sense the symbolic representation of a parse tree that can be treated as input by backend rules. But the same kind of rule is used at the lexical level, for syntax, and in the back end.
 * Is the language machine meant primarily for interpreted languages, or could
it
 be used as, say, a frontend for GCC?

It could be used in either way: in particular, you could use the interface with tcc to compile via C into machine code im memory for immediate execution.
 I'll post more thorough impressions when I have had time to play with it a bit.
 As of currently, I really need to sleep. 
 
 Good work!
 
 -Nod-
 
 

Jul 12 2005
parent reply Nod <Nod_member pathlink.com> writes:
In article <davvoc$8h9$1 digitaldaemon.com>, Peri Hankey says...
 <snip>

Whoa! The Language Machine is way cool, indeed! I have a ton of questions, but I feel it is rather OT for this newsgroup... Would it be possible for you to set up a dedicated communications forum for TLM? A mailing-list on sourceforge would suffice, but might I suggest requesting an account on www.dsource.org? In addition to providing a nice forum, it also gives access to an SVN repository, rather than the ancient CVS used on sf.net. And of course, www.dsource.org is a lot more responsive to feature requests than sf.net :) -Nod-
Jul 14 2005
parent reply Peri Hankey <mpah thegreen.co.uk> writes:
Nod wrote:
 In article <davvoc$8h9$1 digitaldaemon.com>, Peri Hankey says...
 
<snip>

Whoa! The Language Machine is way cool, indeed! I have a ton of questions, but I feel it is rather OT for this newsgroup... Would it be possible for you to set up a dedicated communications forum for TLM? A mailing-list on sourceforge would suffice, but might I suggest requesting an account on www.dsource.org? In addition to providing a nice forum, it also gives access to an SVN repository, rather than the ancient CVS used on sf.net. And of course, www.dsource.org is a lot more responsive to feature requests than sf.net :) -Nod-

Nod - thanks - I agree that this isn't the place. I have emailed dsource.org, and we'll see what happens. In the meantime there are 2 forums at the sourceforge project pages https://sourceforge.net/forum/?group_id=143185 - we can later distill and extract useful stuff to a faq or to wherever the forum finally goes. Peri
Jul 14 2005
parent Brad Anderson <brad dsource.dot.org> writes:
Peri Hankey wrote:
 Nod wrote:
 
 In article <davvoc$8h9$1 digitaldaemon.com>, Peri Hankey says...

 <snip>

Whoa! The Language Machine is way cool, indeed! I have a ton of questions, but I feel it is rather OT for this newsgroup... Would it be possible for you to set up a dedicated communications forum for TLM? A mailing-list on sourceforge would suffice, but might I suggest requesting an account on www.dsource.org? In addition to providing a nice forum, it also gives access to an SVN repository, rather than the ancient CVS used on sf.net. And of course, www.dsource.org is a lot more responsive to feature requests than sf.net :) -Nod-

Nod - thanks - I agree that this isn't the place. I have emailed dsource.org, and we'll see what happens.

Been on vacation but I'm back now. I've sent Peri an email... BA
 
 In the meantime there are 2 forums at the sourceforge project pages
 https://sourceforge.net/forum/?group_id=143185 - we can later distill
 and extract useful stuff to a faq or to wherever the forum finally goes.
 
 Peri
 

Jul 18 2005
prev sibling parent reply B Shropshire <B_member pathlink.com> writes:
In article <dau3bc$1hj0$1 digitaldaemon.com>, Peri Hankey says...

The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process.

How would this thing do for a d-to-HTML converter. Maybe generate a syntax highlighted HTML file from a d file, maybe even parse out the comments into something nicer to look at.
Jul 12 2005
parent Peri Hankey <mpah thegreen.co.uk> writes:
B Shropshire wrote:
 In article <dau3bc$1hj0$1 digitaldaemon.com>, Peri Hankey says...
 
 
The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process.

How would this thing do for a d-to-HTML converter. Maybe generate a syntax highlighted HTML file from a d file, maybe even parse out the comments into something nicer to look at.

present the translator consists of d2xfe.lmn - the frontend ruleset d2dbe.lmn - the d2d backend ruleset But if you look at the metalanguage compilers, these are constructed with (frontend, backend) pairs: lmn2xfe.lmn, lmn2mbe.lmn - compile to own loader format lmn2xfe.lmn, lmn2dbe.lmn - wrap loader format as D lmn2xfe.lmn, lmn2cbe.lmn - wrap loader format as C lmn2xfe.lmn, lmn2Dbe.lmn - compile to D lmn2xfe.lmn, lmn2Cbe.lmn - compile to C The ruleset pairs can either be combined as source or concatenated at runtime. Peri
Jul 12 2005