www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - FancyPars

reply "Stefan Koch" <uplink.coder googlemail.com> writes:
Small announcement.

I uploaded my parser-generator onto github.
It is work in progress and unfinished!

I cannot continue working on it anymore.

Because it is quite idiomatic D-code.

I hope that it will be suitable to beginners.

Unfortunately I will not be available to take any questions.

Repo-Location : https://github.com/UplinkCoder/fancypars-lite
Jul 02 2015
next sibling parent reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Thursday, 2 July 2015 at 14:25:09 UTC, Stefan Koch wrote:

 Small announcement.

 I uploaded my parser-generator onto github.
 It is work in progress and unfinished!


How does its design and use differ from Pegged?
Jul 06 2015
next sibling parent "Stefan Koch" <uplink.coder googlemail.com> writes:
On Monday, 6 July 2015 at 09:22:51 UTC, Per Nordlöw wrote:

 On Thursday, 2 July 2015 at 14:25:09 UTC, Stefan Koch wrote:

 Small announcement.

 I uploaded my parser-generator onto github.
 It is work in progress and unfinished!


 How does its design and use differ from Pegged?


The use does not really differ.
The Design however is very diffrent.
instead of templates it generates CTFEable Functions.
Jul 16 2015
prev sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Monday, 6 July 2015 at 09:22:51 UTC, Per Nordlöw wrote:

 How does its design and use differ from Pegged?


FWIW, this is what I learned from my first acquaintance with 
FancyPars (the OP having signalled not to be available for 
questions). My conclusions may be wrong though.

Running dub produces a vibe.d web server demonstrating the 
capabilities of FancyPars. This was a bit confusing at first 
because being a web-app seemed central to the design of 
FancyPars, but I think it is not. Anyway, the first page shows a 
large edit field containing an example grammar, and a button 
"Generate AST". Clicking this button brings up the second page 
containing D code for the lexer and parser for the given grammar, 
type definitions for the nodes of the AST, as well as code for 
printing the AST.

Understanding the source of FancyPars is challenging because the 
core source, example vibe.d application source and supporting 
code, as well as generated lexer/parser code are all contained in 
the same directory and committed in the repository.

The syntax for the grammar definition is different from Pegged, 
and seems to be inspired by D. It supports a hierarchical 
structure. It looks powerful, but is undocumented. The example 
grammar looks like this:

ASTNode {
     Identifier  internal {
         [a-zA-Z_][] identifier
     }

     Group  parent {
         Identifier name, ? " " : Identifier[] annotations : " ", 
"{",
             PatternElement[] elements : "," / Group[] groups,
              "}"
     }

     PatternElement  internal {

         AlternativeElement  noFirst {
             PatternElement[] alternatives : "/"
         }

         LexerElement {

             StringElement {
                 "\"", char[] string_, "\""
             }

             NamedChar {
                 "char", ? "[]" : bool isArray, Identifier name
             }

             CharRange  internal {
                 char rangeBegin,  ? "-" : char RangeEnd
             }

             RangeElement {
                 "[", CharRange[] ranges, "]"
             }

             LookbehindElement {
                 "?lb", "(", StringElement str, ")"
             }

             NotElement {
                 "!", LexerElement ce
             }

         }

         NamedElement {
             Identifier type,  ? "[]" : bool isArray, Identifier 
name,
             ? bool isArray : ? ":" : StringElement lst_sep
         }

         ParenElement {
             "(", PatternElement[] elements : ",", ")"
         }

         FlagElement {
             "bool", Identifier flag_name
         }

         QueryElement {
             "?", "bool", Identifier flag_name, ":", 
PatternElement elem
         }

         OptionalElement {
             "?", LexerElement[] ce : ",", ":", PatternElement elem
         }

     }
}


Its announced support for left-recursion is interesting, and I 
may decide to play a bit further with it. My objective would be 
to see if an Extended Pascal to D translating compiler would be 
feasible.

Cheers,
Bastiaan Veelo.
Sep 14 2015
next sibling parent Rory McGuire via Digitalmars-d-announce writes:
Nice one, thanks for the info.

I just used Pegged to generate an API for a JSON REST service at compile
time so I'm still geeking out about the power of D at compile time, but I'm
always interested in parsers.

On Mon, Sep 14, 2015 at 10:50 AM, Bastiaan Veelo via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:


 On Monday, 6 July 2015 at 09:22:51 UTC, Per Nordl=C3=B6w wrote:


 How does its design and use differ from Pegged?


 FWIW, this is what I learned from my first acquaintance with FancyPars
 (the OP having signalled not to be available for questions). My conclusio=


ns

 may be wrong though.

 Running dub produces a vibe.d web server demonstrating the capabilities o=


f

 FancyPars. This was a bit confusing at first because being a web-app seem=


ed

 central to the design of FancyPars, but I think it is not. Anyway, the
 first page shows a large edit field containing an example grammar, and a
 button "Generate AST". Clicking this button brings up the second page
 containing D code for the lexer and parser for the given grammar, type
 definitions for the nodes of the AST, as well as code for printing the AS=


T.

 Understanding the source of FancyPars is challenging because the core
 source, example vibe.d application source and supporting code, as well as
 generated lexer/parser code are all contained in the same directory and
 committed in the repository.

 The syntax for the grammar definition is different from Pegged, and seems
 to be inspired by D. It supports a hierarchical structure. It looks
 powerful, but is undocumented. The example grammar looks like this:

 ASTNode {
     Identifier  internal {
         [a-zA-Z_][] identifier
     }

     Group  parent {
         Identifier name, ? " " : Identifier[] annotations : " ", "{",
             PatternElement[] elements : "," / Group[] groups,
              "}"
     }

     PatternElement  internal {

         AlternativeElement  noFirst {
             PatternElement[] alternatives : "/"
         }

         LexerElement {

             StringElement {
                 "\"", char[] string_, "\""
             }

             NamedChar {
                 "char", ? "[]" : bool isArray, Identifier name
             }

             CharRange  internal {
                 char rangeBegin,  ? "-" : char RangeEnd
             }

             RangeElement {
                 "[", CharRange[] ranges, "]"
             }

             LookbehindElement {
                 "?lb", "(", StringElement str, ")"
             }

             NotElement {
                 "!", LexerElement ce
             }

         }

         NamedElement {
             Identifier type,  ? "[]" : bool isArray, Identifier name,
             ? bool isArray : ? ":" : StringElement lst_sep
         }

         ParenElement {
             "(", PatternElement[] elements : ",", ")"
         }

         FlagElement {
             "bool", Identifier flag_name
         }

         QueryElement {
             "?", "bool", Identifier flag_name, ":", PatternElement elem
         }

         OptionalElement {
             "?", LexerElement[] ce : ",", ":", PatternElement elem
         }

     }
 }


 Its announced support for left-recursion is interesting, and I may decide
 to play a bit further with it. My objective would be to see if an Extende=


d

 Pascal to D translating compiler would be feasible.

 Cheers,
 Bastiaan Veelo.
Sep 14 2015
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 14 September 2015 at 08:50:48 UTC, Bastiaan Veelo 
wrote:


 Understanding the source of FancyPars is challenging because 
 the core source, example vibe.d application source and 
 supporting code, as well as generated lexer/parser code are all 
 contained in the same directory and committed in the repository.


Sorry about that.
The Files that are really interesting are
1. fancy_grammar.d - contains helpers for working with the 
FancyParsGrammar Definition.
2. fancy_genPars.d - generates a function which parses the output 
form the generated lexer
3. fancy_genLex.d generates the lexer function.
4. fancy_genAST.d - generates the AST


 The syntax for the grammar definition is different from Pegged, 
 and seems to be inspired by D. It supports a hierarchical 
 structure. It looks powerful, but is undocumented. The example 
 grammar looks like this:


The Syntax is inspired by D and Pegged.
It is going to be extend though and you should consider it 
unstable.
I will document it as it becomes more stable.


 Its announced support for left-recursion is interesting, and I 
 may decide to play a bit further with it. My objective would be 
 to see if an Extended Pascal to D translating compiler would be 
 feasible.


TransCompilers are what fancyPars is written for.
please show me a few examples of your pascal gramamr.
might be in pegged or EBNF or something similar.

So I can see if there are idioms I should integrate in fp.

Thanks for the interest.

--
Stefan (Uplink_Coder) Koch
Sep 16 2015
parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Wednesday, 16 September 2015 at 20:17:15 UTC, Stefan Koch 
wrote:

 On Monday, 14 September 2015 at 08:50:48 UTC, Bastiaan Veelo 
 wrote:


 Understanding the source of FancyPars is challenging because 
 the core source, example vibe.d application source and 
 supporting code, as well as generated lexer/parser code are 
 all contained in the same directory and committed in the 
 repository.


 Sorry about that.
 The Files that are really interesting are
 1. fancy_grammar.d - contains helpers for working with the 
 FancyParsGrammar Definition.
 2. fancy_genPars.d - generates a function which parses the 
 output form the generated lexer
 3. fancy_genLex.d generates the lexer function.
 4. fancy_genAST.d - generates the AST


Thanks. At first I thought that 
fancy_[ast|token|lexer|parser|printer].d were generated files 
because their content is so similar to the code produced in the 
vibe application. But on closer look I think it is the other way 
around: that the example grammar in vibe describes its own input 
format, and that the similarity in the produced output to said 
files is an illustration that it works the way it should. Am I 
close?


 The Syntax is inspired by D and Pegged.
 It is going to be extend though and you should consider it 
 unstable.
 I will document it as it becomes more stable.


Wow that is great news! Not so abandoned after all :-)


 TransCompilers are what fancyPars is written for.
 please show me a few examples of your pascal gramamr.
 might be in pegged or EBNF or something similar.

 So I can see if there are idioms I should integrate in fp.


Please have a look in 
https://www.dropbox.com/sh/k0ewq4dkz0q009v/AADq5V6j5lPBfBkmpwIpkMq8a?dl=0

p2d.d is my dance with Pegged, containing a subset of the 
Extended Pascal grammar. It was only after typing close to 400 
lines of grammar that I discovered left-recursions, which are 
commented out. These put a spanner in the works, but before that 
I felt this could go somewhere: even comments are translated.

There are many recursions in the complete grammar, for which you 
find the specification in the PDF. (This is the ANSI/IEEE 
standard; Extended Pascal became an ISO standard a year or so 
later, with identical text.) Note the alphabetical list of EBNF 
starting on page 129; the numbers in front refer to the 
corresponding section.


 Thanks for the interest.


Thanks for finding the time for this.

Bastiaan.
Sep 16 2015
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 16 September 2015 at 21:25:40 UTC, Bastiaan Veelo 
wrote:


 Thanks. At first I thought that 
 fancy_[ast|token|lexer|parser|printer].d were generated files 
 because their content is so similar to the code produced in the 
 vibe application. But on closer look I think it is the other 
 way around: that the example grammar in vibe describes its own 
 input format, and that the similarity in the produced output to 
 said files is an illustration that it works the way it should. 
 Am I close?


fancyPars has gone through a few iterations.
I used fancyPars to generate a parser for itself
and then fixed up the things that fancyPars cannot yet generate.

I eat my own dogfood!

Thanks for the pascal specs and your p2d.d
I will see what I can do about that.
Sep 17 2015
prev sibling next sibling parent reply "Stefan Koch" <uplink.coder googlemail.com> writes:
On Thursday, 2 July 2015 at 14:25:09 UTC, Stefan Koch wrote:


 I cannot continue working on it anymore.


Nontheless an unexpected update that makes FancyPars more 
Feature-complete than than Pegged.

I addded simplistic Left Recursion handling...
but the code is a mess :(
Sep 05 2015
parent reply "Bastiaan Veelo" <Bastiaan Veelo.net> writes:
On Saturday, 5 September 2015 at 19:45:09 UTC, Stefan Koch wrote:

 On Thursday, 2 July 2015 at 14:25:09 UTC, Stefan Koch wrote:

 I addded simplistic Left Recursion handling...


Interesting.

 From the readme:

 it only compiles with dmd 2.0.66.2 because it exploits a bug in 
 the const-ness type-system


However, 2.0.66.2 does not seem to exist [1,2]. Am I overlooking 
something?

Best,
Bastiaan.

[1] http://downloads.dlang.org/releases/2.x/
[2] 
https://github.com/D-Programming-Language/dmd/releases?after=v2.067.1-b1
Sep 06 2015
parent "Stefan Koch" <uplink.coder googlemail.com> writes:
On Sunday, 6 September 2015 at 20:23:40 UTC, Bastiaan Veelo wrote:

 However, 2.0.66.2 does not seem to exist [1,2]. Am I 
 overlooking something?


Ahh yeah it should be 2.066.1
Thanks for catching that
Sep 06 2015
prev sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Thursday, 2 July 2015 at 14:25:09 UTC, Stefan Koch wrote:

 I hope that it will be suitable to beginners.


Sounds like you want to share this, but I can't find a licence. 
In case this turns out to be useful, we would need one :-)

If you want I can prepare a PR for that, just let me know which 
licence to pick.

Best,
Bastiaan.
Sep 16 2015
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 16 September 2015 at 12:16:03 UTC, Bastiaan Veelo 
wrote:

 Sounds like you want to share this, but I can't find a licence. 
 In case this turns out to be useful, we would need one :-)

 If you want I can prepare a PR for that, just let me know which 
 licence to pick.

 Best,
 Bastiaan.


I am not sure.
The source should not be used in any product without my explicit 
permission.
However you may study it to get better in your understanding of 
dlang or parser technology
Sep 17 2015
next sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Thursday, 17 September 2015 at 15:47:42 UTC, Stefan Koch wrote:

 On Wednesday, 16 September 2015 at 12:16:03 UTC, Bastiaan Veelo 
 wrote:

 Sounds like you want to share this, but I can't find a 
 licence. In case this turns out to be useful, we would need 
 one :-)

 If you want I can prepare a PR for that, just let me know 
 which licence to pick.

 Best,
 Bastiaan.


 I am not sure.
 The source should not be used in any product without my 
 explicit permission.
 However you may study it to get better in your understanding of 
 dlang or parser technology


Yikes. Are you sure? Are you familiar with open source licensing?
Sep 17 2015
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 17 September 2015 at 16:02:14 UTC, John Colvin wrote:


 Yikes. Are you sure? Are you familiar with open source 
 licensing?


I would be open to open-source the "base" of fp.
but keeping certin extentions for grammar analysis closed.

What license would you suggest for that.
Sep 17 2015
parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Thursday, 17 September 2015 at 16:33:12 UTC, Stefan Koch wrote:

 On Thursday, 17 September 2015 at 16:02:14 UTC, John Colvin 
 wrote:


 Yikes. Are you sure? Are you familiar with open source 
 licensing?


 I would be open to open-source the "base" of fp.
 but keeping certin extentions for grammar analysis closed.

 What license would you suggest for that.


Assuming you wrote it all, you can license the code in whatever 
way you want. See http://choosealicense.com for more info. You 
can even use multiple licenses, or different licenses for 
different parts of the code.
Sep 17 2015
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 17 September 2015 at 16:55:42 UTC, John Colvin wrote:


 Assuming you wrote it all, you can license the code in whatever 
 way you want. See http://choosealicense.com for more info. You 
 can even use multiple licenses, or different licenses for 
 different parts of the code.


Hmm reading this. No license, is best for now.

 Bastian
   The FancyPars Grammar for pascal will look very very different 
from what you wrote.
   In FancyPars Grammars I worked very hard to avoid repetitions.
   FGPs do not just describe the language grammar. They are 
describing the AST-Structure.
   So just by reading the grammar a person working with the AST 
will know what is what and in which members-variables of the 
AST-Node which information is stored.
   I would recommed you open issues in the FancyPars-repo for 
stuff that is hard to understand.
Sep 17 2015
parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Thursday, 17 September 2015 at 20:32:59 UTC, Stefan Koch wrote:

 On Thursday, 17 September 2015 at 16:55:42 UTC, John Colvin 
 wrote:


 Assuming you wrote it all, you can license the code in 
 whatever way you want. See http://choosealicense.com for more 
 info. You can even use multiple licenses, or different 
 licenses for different parts of the code.


 Hmm reading this. No license, is best for now.


Take your time, but without a license anyone cloning or forking 
your repo is in fact violating your copyright. It is not what 
most people expect on github, and I will have to delete my fork 
and local clone...


  Bastiaan
   The FancyPars Grammar for pascal will look very very 
 different from what you wrote.
   In FancyPars Grammars I worked very hard to avoid repetitions.
   FGPs do not just describe the language grammar. They are 
 describing the AST-Structure.
   So just by reading the grammar a person working with the AST 
 will know what is what and in which members-variables of the 
 AST-Node which information is stored.


I can see the value of that when designing a grammar, or building 
a translator. In my case though the grammar was standardised a 
quarter of a century ago, and available in BNF. Redefining the 
complete language in FPG by hand would be interesting but time 
consuming and error prone -- not sure that would pay in the end. 
Maybe writing a BNF2FPG transcompiler would get me there faster...

But, without a license I am prohibited from experimenting with 
it. Even with permission for educational purposes or the like, 
which I think you have implied, I am not sure that I will be 
allowed to construct a transcompiler intended for the translation 
of proprietary source in the end.

I am afraid I can't afford the time to investigate the 
possibilities of FancyPars until legal uncertainties are resolved.

Maybe you could consider to make the core of FancyPars Open 
Source with one of the mainstream licenses, without further 
restrictions. The parts that you want to keep proprietary I would 
not publish at all. That way, if somebody else decides to write 
an analyser, he will not risk infringing the copyright of your 
closed source, because it is not publicly viewable.

Best,
Bastiaan.
Sep 17 2015
next sibling parent reply Ben Boeckel via Digitalmars-d-announce writes:
On Thu, Sep 17, 2015 at 23:40:49 +0000, Bastiaan Veelo via
Digitalmars-d-announce wrote:

 On Thursday, 17 September 2015 at 20:32:59 UTC, Stefan Koch wrote:

 Hmm reading this. No license, is best for now.


 
 Take your time, but without a license anyone cloning or forking 
 your repo is in fact violating your copyright. It is not what 
 most people expect on github, and I will have to delete my fork 
 and local clone...


By using public repos, you explicitly allow anyone to view and fork your
project. There are no implicit rights of *use* of that clone though.

--Ben
Sep 17 2015
parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Friday, 18 September 2015 at 00:30:25 UTC, Ben Boeckel wrote:

 On Thu, Sep 17, 2015 at 23:40:49 +0000, Bastiaan Veelo via 
 Digitalmars-d-announce wrote:

 On Thursday, 17 September 2015 at 20:32:59 UTC, Stefan Koch 
 wrote:

 Hmm reading this. No license, is best for now.


 
 Take your time, but without a license anyone cloning or 
 forking your repo is in fact violating your copyright. It is 
 not what most people expect on github, and I will have to 
 delete my fork and local clone...


 By using public repos, you explicitly allow anyone to view and 
 fork your project. There are no implicit rights of *use* of 
 that clone though.


You are correct [1], thanks. But I still will have to delete 
mine, because it contains changes.

[1] 
https://help.github.com/articles/open-source-licensing/#what-happens-if-i-dont-choose-a-license
Sep 18 2015
prev sibling parent reply Rory McGuire via Digitalmars-d-announce writes:
On Fri, Sep 18, 2015 at 2:30 AM, Ben Boeckel via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Thu, Sep 17, 2015 at 23:40:49 +0000, Bastiaan Veelo via
 Digitalmars-d-announce wrote:

 On Thursday, 17 September 2015 at 20:32:59 UTC, Stefan Koch wrote:

 Hmm reading this. No license, is best for now.


 Take your time, but without a license anyone cloning or forking
 your repo is in fact violating your copyright. It is not what
 most people expect on github, and I will have to delete my fork
 and local clone...


 By using public repos, you explicitly allow anyone to view and fork your
 project. There are no implicit rights of *use* of that clone though.

 --Ben


There is also no implicit rights if you provide pull requests for said repo.

I for one will have to delete everything I have on FancyPars and avoid,
because I mix work and pleasure all the time, and I have no time in my life
for lawyers, life is too short.
Sep 17 2015
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 18 September 2015 at 06:13:24 UTC, Rory McGuire wrote:


 I for one will have to delete everything I have on FancyPars 
 and avoid, because I mix work and pleasure all the time, and I 
 have no time in my life for lawyers, life is too short.


No worries!
I will not sue anyone! of any copyright infringement or the like.
I will open/source the of FancyPars.

an BNF2FPG translator is possible will not be too useful.
(If it does not do really significant work that is quite hard to 
do without deep-neural networks)
That said a first step to translate existing grammars to fpg cam 
surely be done automatically .
Sep 18 2015
parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Friday, 18 September 2015 at 14:24:09 UTC, Stefan Koch wrote:

 On Friday, 18 September 2015 at 06:13:24 UTC, Rory McGuire 
 wrote:

 I will open/source the of FancyPars.


Great! Looking forward to that.

Bastiaan.
Sep 18 2015
prev sibling parent Ben Boeckel via Digitalmars-d-announce writes:
On Thu, Sep 17, 2015 at 15:47:41 +0000, Stefan Koch via Digitalmars-d-announce
wrote:

 I am not sure.
 The source should not be used in any product without my explicit 
 permission.


FYI, that's not FOSS. Please consider using a LICENSE file which
explicitly states that as the case. Github is already a minefield when
it comes to licensing, please don't help make it worse :) .

--Ben
Sep 17 2015