www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Enki 1.1 Released

reply pragma <pragma_member pathlink.com> writes:
Enki 1.1 is now available.

Documentation (still in progress)
http://www.dsource.org/projects/ddl/wiki/Enki

* Download
http://svn.dsource.org/projects/ddl/downloads/enki.1.1.win32.bin.zip
http://svn.dsource.org/projects/ddl/downloads/enki.sdk.1.1.src.zip

Enki is a frontend parser generator that uses an annotated EBNF syntax to
produce .d sourcefiles.  More details can be found at the link above. 

Enki 1.1 Changes:
- enhancement: semantic analysis of binding types by search is now in
declaration order
- bug: repeated binding declarations in generated parser
- bug: binding names were output as 'bind_x' in the comments for some
expressions
- enhancement: added function-call capability to ' ' operator
- enhancement: added '&' (custom-terminal) operator
- enhancement: added minimal sourcecode for custom parsers to binary
distribution

- EricAnderton at yahoo
Jun 13 2006
next sibling parent reply BCS <BCS pathlink.com> writes:
pragma wrote:
 Enki 1.1 is now available.
 
 Documentation (still in progress)
 http://www.dsource.org/projects/ddl/wiki/Enki
 
 * Download
 http://svn.dsource.org/projects/ddl/downloads/enki.1.1.win32.bin.zip
 http://svn.dsource.org/projects/ddl/downloads/enki.sdk.1.1.src.zip
 
 Enki is a frontend parser generator that uses an annotated EBNF syntax to
 produce .d sourcefiles.  More details can be found at the link above. 
 
 Enki 1.1 Changes:
 - enhancement: semantic analysis of binding types by search is now in
 declaration order
 - bug: repeated binding declarations in generated parser
 - bug: binding names were output as 'bind_x' in the comments for some
 expressions
 - enhancement: added function-call capability to ' ' operator
 - enhancement: added '&' (custom-terminal) operator
 - enhancement: added minimal sourcecode for custom parsers to binary
 distribution
 
 - EricAnderton at yahoo

Cool, tried it out. Didn't get to far (I haven't used a parser generator before, so...). The generated parser still doesn't compile out of the box. I dug through the code and added a few imports to the render() function to get it to work. Here is a patch of what I did. *** old\enki\EnkiBackend.d Wed Jun 14 11:47:30 2006 --- new\sand\EnkiBackend.d Wed Jun 14 11:49:57 2006 *************** *** 184,189 **** --- 184,191 ---- emit("module " ~ moduleName ~ ";"); } emit("debug private import std.stdio;"); + emit("import enki.BaseParser;"); + emit("import enki.types;"); foreach(imp; imports){ emit("private import " ~ imp ~ ";"); }
Jun 14 2006
parent reply pragma <pragma_member pathlink.com> writes:
In article <e6plq9$1upb$1 digitaldaemon.com>, BCS says...
pragma wrote:
 Enki 1.1 is now available.

Cool, tried it out. Didn't get to far (I haven't used a parser generator before, so...). The generated parser still doesn't compile out of the box. I dug through the code and added a few imports to the render() function to get it to work. Here is a patch of what I did. *** old\enki\EnkiBackend.d Wed Jun 14 11:47:30 2006 --- new\sand\EnkiBackend.d Wed Jun 14 11:49:57 2006 *************** *** 184,189 **** --- 184,191 ---- emit("module " ~ moduleName ~ ";"); } emit("debug private import std.stdio;"); + emit("import enki.BaseParser;"); + emit("import enki.types;"); foreach(imp; imports){ emit("private import " ~ imp ~ ";"); }

BCS, thanks for being patient with all this. I hope I haven't lost your interest. ;) What you've done here is hack the code-generator to emit the imports you need - I like the cut of your jib. As the documentation is incomplete, I can't blame you for trying this, but there is an easier way to do this. The .import() directive does this job nicely.
 .import("my.import.here");

A solid example of this and other directives are in Enki's BNF definition: http://svn.dsource.org/projects/ddl/trunk/enki/enki.bnf You may find the .baseclass() .classname() and .code{{{ }}} directives helpful as well. The .start() directive is now depricated, and there's the .include() and .alias() directives that aren't used in that particular file. - EricAnderton at yahoo
Jun 14 2006
parent reply BCS <BCS pathlink.com> writes:
pragma wrote:
 In article <e6plq9$1upb$1 digitaldaemon.com>, BCS says...
Cool, tried it out. Didn't get to far (I haven't used a parser generator 
before, so...). The generated parser still doesn't compile out of the 
box. I dug through the code and added a few imports to the render() 
function to get it to work. Here is a patch of what I did.


 
 
 BCS, thanks for being patient with all this.  I hope I haven't lost your
 interest. ;)

Nope, you have only wet it.
 
 What you've done here is hack the code-generator to emit the imports you need -
 I like the cut of your jib.  As the documentation is incomplete, I can't blame
 you for trying this, but there is an easier way to do this.  The .import()
 directive does this job nicely.
 
 
.import("my.import.here");


Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed.
 
 A solid example of this and other directives are in Enki's BNF definition:
 
 http://svn.dsource.org/projects/ddl/trunk/enki/enki.bnf
 

I'll take closer a look at that soon. ======== And on another topic: another hack It fixes a default case bug. Without it any rule of the form rule ::= (foo bar) always matches even if both foo and bar don't match. The case where the rule has annotations works as advertised. <patch> *** old\enki\Rule.d Wed Jun 14 13:50:06 2006 --- new\enki\Rule.d Wed Jun 14 13:50:59 2006 *************** *** 241,251 **** } public void renderPass(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(true);"); } public void renderFail(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(false);"); } public void semanticPass(Rule thisRule,BaseEnkiParser root){ --- 242,252 ---- } public void renderPass(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(true,true);"); } public void renderFail(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(false,false);"); } public void semanticPass(Rule thisRule,BaseEnkiParser root){ </patch>
Jun 14 2006
parent reply pragma <pragma_member pathlink.com> writes:
In article <e6ptut$2c61$1 digitaldaemon.com>, BCS says...
.import("my.import.here");


Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed.

I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool. I really think that it should try to impose as little as possible, wherever possible. In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser? I'd rather make imports explicit so the developer has this flexibility.
And on another topic: another hack

It fixes a default case bug. Without it any rule of the form

rule ::= (foo bar)

always matches even if both foo and bar don't match. The case where the 
rule has annotations works as advertised.

<patch>
*** old\enki\Rule.d	Wed Jun 14 13:50:06 2006
--- new\enki\Rule.d	Wed Jun 14 13:50:59 2006
***************
*** 241,251 ****
   	}
   	
   	public void renderPass(CodeGenerator generator){
! 		generator.emit("return ResultT!(bool)(true);");
   	}
   	
   	public void renderFail(CodeGenerator generator){
! 		generator.emit("return ResultT!(bool)(false);");
   	}
   	
   	public void semanticPass(Rule thisRule,BaseEnkiParser root){
--- 242,252 ----
   	}
   	
   	public void renderPass(CodeGenerator generator){
! 		generator.emit("return ResultT!(bool)(true,true);");
   	}
   	
   	public void renderFail(CodeGenerator generator){
! 		generator.emit("return ResultT!(bool)(false,false);");
   	}
   	
   	public void semanticPass(Rule thisRule,BaseEnkiParser root){
</patch>

Ah, now that's a serious bug. Thanks for tracking it down. I'm glad that things are written such that you can find this stuff so easily. Thanks for the patch! Another way to tackle this would be tochange the failure rule to this:
   	public void renderFail(CodeGenerator generator){
! 		generator.emit("return ResultT!(bool)();");
   	}

Which is equivalent. When I get around to expanding the docs to detail how to write custom rules, this will make more sense. - EricAnderton at yahoo
Jun 14 2006
next sibling parent reply BCS <BCS pathlink.com> writes:
pragma wrote:
 In article <e6ptut$2c61$1 digitaldaemon.com>, BCS says...
 
.import("my.import.here");


these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed.

I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool. I really think that it should try to impose as little as possible, wherever possible. In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser? I'd rather make imports explicit so the developer has this flexibility.

OK I'll grant that things might move, however in the general case those imports will be needed by most people. Good defaults make for good programs. How about make those imports the default and put in some sort of Override directive.
 Ah, now that's a serious bug.  Thanks for tracking it down.  I'm glad
 that things are written such that you can find this stuff so easily.
 Thanks for the patch!

Find-in-files, grep and kin are wonderful aren't they. <g> Another thought; as it is written, Enki is purely an ASCII text program. It would be nice if the parsed data had its own type that, is only used for the input data. This would allow for the parsing of an array on any type (enums or structs come to mind). Of course this would requirer the introduction of a new terminal matching syntax. Just a sketch of what might be done: <code name="somewher.d"> ... enum bar { i1, i2, i3} struct baz{ bar thisType; } ... </code> <code name="syn.bnf"> ... .parsetype(baz) ... foo ::= .thisType == bar.i1; ... </code>
Jun 14 2006
parent reply pragma <pragma_member pathlink.com> writes:
In article <e6q2pu$2cqo$1 digitaldaemon.com>, BCS says...
pragma wrote:
 In article <e6ptut$2c61$1 digitaldaemon.com>, BCS says...
 
.import("my.import.here");


these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed.

I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool. I really think that it should try to impose as little as possible, wherever possible. In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser? I'd rather make imports explicit so the developer has this flexibility.

OK I'll grant that things might move, however in the general case those imports will be needed by most people. Good defaults make for good programs. How about make those imports the default and put in some sort of Override directive.

That's not a bad idea. Something like a .supportlib() directive or something. I'll consider it. :)
Another thought; as it is written, Enki is purely an ASCII text program. 
   It would be nice if the parsed data had its own type that, is only 
used for the input data. This would allow for the parsing of an array on 
any type (enums or structs come to mind). Of course this would requirer 
the introduction of a new terminal matching syntax.

I'm well ahead of you on this front. I'm presently composing a lexer for D, which will then support a secondary parser for the lexed tokens. Both of these will be Enki based. So far I'm starting to see the limitations of the String/char[] setup, and yearn for multi-byte char support. So I'll be working on a way to provide this. As far as non-ascii parsing is concerned, Enki can already support this to an extent. code{{{ enum MyType: uint{ Foo, Bar, Baz } }}} MyRule ::= &Foo:x &Bar:y &Baz:z; Under the hood, Enki will generate terminal expressions for each & term, which are then generated as "terminal(Foo)". As long as your base parser has a terminal(MyType) declared, you can handle these expressions. It then becomes the responsility of that terminal to determine if the token matches or not. As far as the internal stream of data is concerned, the IParser position() and sliceData() methods abstract the data away to the same extent. When working with a non-text input, sliceData() can be redefined as returning something other than string. The only caveat here is that you must be careful to avoid expressions cases where Enki will evaluate the default type of a binding to String: # Enki has no clue what 'x' should be so it becomes a String Foobar ::= ("hello world"):x;
.parsetype(baz)

A wonderful idea. This way Enki can use this as the default binding type when it encounters expressions like the one above. - EricAnderton at yahoo
Jun 15 2006
parent BCS <BCS pathlink.com> writes:
pragma wrote:
 In article <e6q2pu$2cqo$1 digitaldaemon.com>, BCS says...
 
Another thought; as it is written, Enki is purely an ASCII text program. 
  It would be nice if the parsed data had its own type that, is only 
used for the input data. This would allow for the parsing of an array on 
any type (enums or structs come to mind). Of course this would requirer 
the introduction of a new terminal matching syntax.

I'm well ahead of you on this front. I'm presently composing a lexer for D, which will then support a secondary parser for the lexed tokens. Both of these will be Enki based.

I'm already working on a lexer that so far does just about everything but numeric literals, lexing it down to an array of structs. It is totally native D and is done from the start with the intention that it be easy to change. Adding a static token type (say "~~") involves about three lines of code in the lexer. Adding something like a new type of string only involves adding two of those lines and a function to find the rest. Its still a bit buggy but it works for the most part.
 
 So far I'm starting to see the limitations of the String/char[] setup, and
yearn
 for multi-byte char support.  So I'll be working on a way to provide this.  As
 far as non-ascii parsing is concerned, Enki can already support this to an
 extent.
 

What I did for mine is make a wrapper Stream struct that presents the whole stream as dchars. For convenience it also catches look-ahead and such. I'm not sure what goals your project is perusing, mine is easy modification targeted at alpha testing of features. However I intend to drop my code into the wild at some point so...
 code{{{
 enum MyType: uint{
 Foo, Bar, Baz
 }
 }}}
 
 MyRule ::= &Foo:x &Bar:y &Baz:z;
 
 Under the hood, Enki will generate terminal expressions for each & term, which
 are then generated as "terminal(Foo)".  As long as your base parser has a
 terminal(MyType) declared, you can handle these expressions.  It then becomes
 the responsility of that terminal to determine if the token matches or not.
 
 As far as the internal stream of data is concerned, the IParser position() and
 sliceData() methods abstract the data away to the same extent.  When working
 with a non-text input, sliceData() can be redefined as returning something
other
 than string.  The only caveat here is that you must be careful to avoid
 expressions cases where Enki will evaluate the default type of a binding to
 String:
 
 # Enki has no clue what 'x' should be so it becomes a String
 Foobar ::= ("hello world"):x;
 

I'll check that out.
Jun 15 2006
prev sibling parent Georg Wrede <georg.wrede nospam.org> writes:
pragma wrote:
 In article <e6ptut$2c61$1 digitaldaemon.com>, BCS says...
 
 .import("my.import.here");


from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed.

I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool. I really think that it should try to impose as little as possible, wherever possible. In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser? I'd rather make imports explicit so the developer has this flexibility.

That's what Defaults are for!
Jun 15 2006
prev sibling next sibling parent reply Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
pragma wrote:
 Enki 1.1 is now available.

This subproject of yours couldn't have come at a better time... here I was writing a byte-compiler/vm for a simple scripted object-database language in another project of mine (codename "Lyra"), and you drop this lovely golden egg right in my lap! :) Goodbye messy amateur object hierarchy, hello much easier to work with grammar file and backend library! -- Chris Nicholson-Sauls
Jun 14 2006
parent pragma <pragma_member pathlink.com> writes:
In article <e6pqne$2826$1 digitaldaemon.com>, Chris Nicholson-Sauls says...
pragma wrote:
 Enki 1.1 is now available.

This subproject of yours couldn't have come at a better time... here I was writing a byte-compiler/vm for a simple scripted object-database language in another project of mine (codename "Lyra"), and you drop this lovely golden egg right in my lap! :) Goodbye messy amateur object hierarchy, hello much easier to work with grammar file and backend library! -- Chris Nicholson-Sauls

You're quite welcome! If you find any bugs, or have any suggestions on how to improve things, don't be afraid to check in and make some noise. ;) - EricAnderton at yahoo
Jun 14 2006
prev sibling next sibling parent reply BCS <BCS pathlink.com> writes:
A few patches to make it compile under 0.161

--two do nothing function now cause errors, so I put assert(0)s in them.
--the variable "start" is used in nested scopes a lot, so now it is 
suffixed with its indent level.

e.g.
{
	uint start;
	{
		uint start;
	}
}

becomes

{
	uint start1;
	{
		uint start2;
	}
}



www.uidaho.edu/~shro8822/enki_patch_161.zip
Jun 20 2006
parent pragma <pragma_member pathlink.com> writes:
In article <e79r5q$1ds7$9 digitaldaemon.com>, BCS says...
A few patches to make it compile under 0.161

--two do nothing function now cause errors, so I put assert(0)s in them.
--the variable "start" is used in nested scopes a lot, so now it is 
suffixed with its indent level.

e.g.
{
	uint start;
	{
		uint start;
	}
}

becomes

{
	uint start1;
	{
		uint start2;
	}
}



www.uidaho.edu/~shro8822/enki_patch_161.zip

Thanks! I was just going to get a patch underway for that - I'll take a look at your submission. - EricAnderton at yahoo
Jun 21 2006
prev sibling parent reply BCS <BCS_member pathlink.com> writes:
In article <e6ntg4$270o$1 digitaldaemon.com>, pragma says...
Enki 1.1 is now available.

Documentation (still in progress)
http://www.dsource.org/projects/ddl/wiki/Enki

Yet another patch!!!!! This time to the grammar. Enki won't allow a rule of the form # v-- note second [] Rule = char[][] var ::= ["foo":var Rule:~var] This would logically generate an array of arrays of char. Adding it doesn't seem to brake anything. *** enki.bnf Mon Jun 26 21:24:51 2006 --- enki.bnf Mon Jun 26 18:56:07 2006 *************** *** 94,100 **** ExplicitParam = new Param(bool isArray,String type,String name) ! ::= Identifier:type ws [ "[]":isArray ws] Identifier:name ; # # Expressions # --- 94,103 ---- ExplicitParam = new Param(bool isArray,String type,String name) ! ::= Identifier:type ws [ "[]":isArray Brackets ws] Identifier:name ; ! ! Brackets ::= ["[]" Brackets]; ! # # Expressions #
Jun 26 2006
parent reply BCS <BCS_member pathlink.com> writes:
In article <e7qdln$juq$1 digitaldaemon.com>, BCS says...
In article <e6ntg4$270o$1 digitaldaemon.com>, pragma says...
[...]

Yet another patch!!!!!

Adding it doesn't seem to brake anything.

How to change a program: 1: think 2: try 3: TEST 4: tell I forgot #3, sure, the patched version parses the BNF and even generates code, but the generated code doesn't compile . :op It's making some assumptions that I voided, I'll try to get a working version sooner or later.
Jun 27 2006
parent reply pragma <pragma_member pathlink.com> writes:
In article <e7qngu$10lt$1 digitaldaemon.com>, BCS says...
In article <e7qdln$juq$1 digitaldaemon.com>, BCS says...
In article <e6ntg4$270o$1 digitaldaemon.com>, pragma says...
[...]

Yet another patch!!!!!

Adding it doesn't seem to brake anything.

How to change a program: 1: think 2: try 3: TEST 4: tell

Exactly!
I forgot #3, sure, the patched version parses the BNF and even generates code,
but the generated code doesn't compile . :op It's making some assumptions that I
voided, I'll try to get a working version sooner or later.

No worries. I'm *way* behind schedule on everything these days (being off your ass with a bad cold will do that). I'll get to folding your contributions in as soon as I can get to it. :) Thank you for helping out ! - EricAnderton at yahoo
Jun 27 2006
parent BCS <BCS pathlink.com> writes:
pragma wrote:
[...]

 No worries.  I'm *way* behind schedule on everything these days (being off your
 ass with a bad cold will do that).  I'll get to folding your contributions in
as
 soon as I can get to it. :)
 
 Thank you for helping out !
 
 - EricAnderton at yahoo

Some cases of negation will result in illegal code { ... mismatch53: // nothing to label } this patch should fix the problem. (don't count on it to apply, I had to hand edit it to remove my other changes.) Another solution would be to make an emptyExpression render to "{}" rather than nothing. *** enki/Expression.d Mon Jul 17 18:15:04 2006 --- enki/Expression.d Mon Jul 17 22:55:01 2006 *************** *** 776,792 **** emit("{//Negate"); indent(); render(expr,gotoMatch,gotoMismatch); render(matchLabel); render(fail); render(mismatchLabel); + if(pass.isEmpty) emit("{}"); render(pass); unindent(); emit("}"); } } public String toBNF(){ return "!" ~ expr.toBNF(); --- 776,791 ----
Jul 18 2006