www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - GSOC 2011

reply Luca Boasso <luke.boasso gmail.com> writes:
Hello,

I am Luca and I am finishing my master in computer engineering. I am currently
an intern at Panasonic in Cupertino.
My great passion has been always programming language design and implementation
and I am studying the D programming language.

I like to play with different languages and in my master thesis I created a
small DSL. I am familiar with parsing theory  and I wrote simple parsers
by hand and with the help of ANTLR.

I am really interested in the following ideas (sorted by interest):
- http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#LexingandParsing
- http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#ANTLRandJavabasedDparserforIDEusage
- http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD

I think that I can give my contribution and I would be more than happy to work
with you. This would be for me a great opportunity to contribute to an open
source project and a way to learn more about D and parsing.

Could you give me more information about the help needed?

Thank you

Luca Boasso
Mar 22 2011
next sibling parent Bernard Helyer <b.helyer gmail.com> writes:
On Tue, 22 Mar 2011 10:47:59 -0700, Luca Boasso wrote:
 I am really interested in the following ideas (sorted by interest): -
 http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#LexingandParsing -
 http://prowiki.org/wiki4d/wiki.cgi?

 - http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD
 
 I think that I can give my contribution and I would be more than happy
 to work with you. This would be for me a great opportunity to contribute
 to an open source project and a way to learn more about D and parsing.
 
 Could you give me more information about the help needed?

Thank you for your interest! Ilya Pupatenko has posted to this NG about their interest to work on the Lexing and Parsing, so you may want to contact them and discuss collaboration. I think if you are familiar with ANTLR, an ANTLR grammar and a Java parser would be very useful. I'm just a Joe Schmo, so that's just one man's opinion -- not an official recommendation, mind.
Mar 22 2011
prev sibling next sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 22/03/2011 17:47, Luca Boasso wrote:
 Hello,

 I am Luca and I am finishing my master in computer engineering. I am currently
 an intern at Panasonic in Cupertino.
 My great passion has been always programming language design and implementation
 and I am studying the D programming language.

 I like to play with different languages and in my master thesis I created a
 small DSL. I am familiar with parsing theory  and I wrote simple parsers
 by hand and with the help of ANTLR.

 I am really interested in the following ideas (sorted by interest):
 - http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#LexingandParsing
 - http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#ANTLRandJavabasedDparserforIDEusage
 - http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD

 I think that I can give my contribution and I would be more than happy to work
 with you. This would be for me a great opportunity to contribute to an open
 source project and a way to learn more about D and parsing.

 Could you give me more information about the help needed?

 Thank you

 Luca Boasso

The student is free to present their own ideas, or to flesh out in more detail a proposal for existing ideas. The first two items (LexingandParsing, ANTLRandJavabasedDparserforIDEusage) are fairly concrete ideas. The last one, D Tools in D is far more general, and can involve a lot of different things. But there are two things that I would like to say regarding that one. First, I think that for GSoC we should focus only on proposals that, if completed successfully, would be usable and useful on their own, or add something of value to another project/tool/library that would be useful. In other words, I don't find it desirable to work on a project that would need more work done on it afterwards to only then make a usable tool or library (for example a D-based D parser). The second, partially related to the above, is that I find that writing certain kinds of D tools in D is a huge waste of time. The prime example is IDE stuff. It is crazy to be reinventing the wheel here, to not use the frameworks and functionality that existing IDE platforms (Eclipse, Netbeans, Visual Studio, etc.) offer. (More like rebuilding a castle than reinventing the wheel) In this IDE case, building a tool in D is only worthwhile if you are able to integrate it with the IDE platform. For some kinds of tools, like build tools, debuggers, this is fairly easy. But for parsing and semantic functionality (for example, code complete), it is much harder. But it is not unheard of. For example the Goclipse plugin (Eclipse-based Go IDE) uses Gocode - an external daemon process - for its core semantic functionality. (I haven't tried it yet to see how well it works). But it's all very like the ideas presented here: http://vimeo.com/16069687 by Steve Yegge regarding new toolchains. My overall point is that people should think of delivering something of value to D developers, and not just an incomplete project. And also make it as sustainable as possible (easier to keep up to date with changes in D or other tools). Of course, all of this is my personal opinion, and doesn't necessarily reflect the opinion of the rest of the community, or the D GSoC umbrella organization. -- Bruno Medeiros - Software Engineer
Mar 23 2011
next sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Bruno Medeiros Wrote:

 Of course, all of this is my personal opinion, and doesn't necessarily 
 reflect the opinion of the rest of the community, or the D GSoC umbrella 
 organization.

If a student is really interested in learning more about language parsing and apply skills they have already learned, D provides a great opportunity. While it would be great to have a complete solution, D does need the basics, though I wouldn't know what would make a good summer long task.
Mar 23 2011
prev sibling next sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 23/03/2011 14:59, spir wrote:
 On 03/23/2011 02:40 PM, Bruno Medeiros wrote:
 The first two items (LexingandParsing,
 ANTLRandJavabasedDparserforIDEusage) are fairly concrete ideas.
 The last one, D Tools in D is far more general, and can involve a lot of
 different things

It is instead more specific: "The aim is to boost the production of fundamental tools for analysis of D source code in D: lexical, syntactic, at best first steps of semantic analysis; producing code representations as D data structures." Its usefulness /in the long term/ can be said more general in that it would enable *further* creation of D tools in D (*), potentially useful to all D programmers. Unlike eg a general-purpose parsing lib in D which is specific to the parsing domain; on the other hand, a general-purpose parsing lib does not parse only D code ;-). (*) See (again): http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD Denis

Yeah, I agree that D tools are (generally speaking) much more useful than a D parsing library. But that's not the point, the point is that with the limited manpower we have available (not just in terms of GSoC, but generally), it's very risky and unwise to embark on projects that then remain incomplete and never see the light of day as part of something useful. It's like comparing a bicycle to a car engine. Yeah, the car engine can allow to build a nice car but it's worthless on it's own and at least the bicycle can get somewhere faster. Even a skate or roller-blades are more useful. How much development has it even been in any D tools in D lately (in the last year or so)? -- Bruno Medeiros - Software Engineer
Mar 23 2011
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/23/11 11:42 AM, Luca Boasso wrote:
 Sorry for the late reply,
 even tough I'm not an ANTLR expert, given my previous experience with the tool
 and having read most of the official book, I could help more on this GSOC
 idea.

 I have looked at http://www.antlr.org/grammar/list and on google and I could
not
 find any recent ANTLR grammar for D.
 In my opinion this project could be a good contribution for the D community
 while being practical and useful in the short term.

There are two aspects: writing an ANTLR grammar that parses D (which you are referring to) and also adding to ANTLR the capability of generating D. The latter would allow D users to create and use parsers for various languages. It would also require close cooperation with the ANTLR people for acceptance etc. (Sticking with the former project for now.) This is a technically interesting project. I'd need to hear more about the high-level motivation. What is the "business" motivation? There is already a parser for D written in C++ (the open-source front-end of dmd), so the project would need to build a good case for the added value. Also there are other D parser projects that people work on (ddmd comes to mind), so I'd need to hear a lot more in the way of justifying the utility of such a project. Thanks, Andrei
Mar 23 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-03-24 05:30, Andrei Alexandrescu wrote:
 On 3/23/11 11:42 AM, Luca Boasso wrote:
 Sorry for the late reply,
 even tough I'm not an ANTLR expert, given my previous experience with
 the tool
 and having read most of the official book, I could help more on this GSOC
 idea.

 I have looked at http://www.antlr.org/grammar/list and on google and I
 could not
 find any recent ANTLR grammar for D.
 In my opinion this project could be a good contribution for the D
 community
 while being practical and useful in the short term.

There are two aspects: writing an ANTLR grammar that parses D (which you are referring to) and also adding to ANTLR the capability of generating D. The latter would allow D users to create and use parsers for various languages. It would also require close cooperation with the ANTLR people for acceptance etc. (Sticking with the former project for now.) This is a technically interesting project. I'd need to hear more about the high-level motivation. What is the "business" motivation? There is already a parser for D written in C++ (the open-source front-end of dmd), so the project would need to build a good case for the added value. Also there are other D parser projects that people work on (ddmd comes to mind), so I'd need to hear a lot more in the way of justifying the utility of such a project. Thanks, Andrei

Isn't it obvious, to be able to create tools that needs to parse/lex D code. If parser/lexer is included in Phobos that is basically a port of the C++ one in DMD and DMD starts to use the D version it will be a huge win. The parser/lexer in Phobos will always automatically be up to date with the one the compiler uses because they are actually the same. As far as I know DDMD cannot be easily used to build other tools that needs to parse/lex D code in it's current state. That is a long term goal but not even the short term goal, of being up to date with DMD and produce the same code as DMD, has been reached yet. -- /Jacob Carlborg
Mar 24 2011
prev sibling next sibling parent spir <denis.spir gmail.com> writes:
On 03/24/2011 10:48 AM, Jacob Carlborg wrote:
 On 2011-03-24 05:30, Andrei Alexandrescu wrote:
 On 3/23/11 11:42 AM, Luca Boasso wrote:
 Sorry for the late reply,
 even tough I'm not an ANTLR expert, given my previous experience with
 the tool
 and having read most of the official book, I could help more on this GSOC
 idea.

 I have looked at http://www.antlr.org/grammar/list and on google and I
 could not
 find any recent ANTLR grammar for D.
 In my opinion this project could be a good contribution for the D
 community
 while being practical and useful in the short term.

There are two aspects: writing an ANTLR grammar that parses D (which you are referring to) and also adding to ANTLR the capability of generating D. The latter would allow D users to create and use parsers for various languages. It would also require close cooperation with the ANTLR people for acceptance etc. (Sticking with the former project for now.) This is a technically interesting project. I'd need to hear more about the high-level motivation. What is the "business" motivation? There is already a parser for D written in C++ (the open-source front-end of dmd), so the project would need to build a good case for the added value. Also there are other D parser projects that people work on (ddmd comes to mind), so I'd need to hear a lot more in the way of justifying the utility of such a project. Thanks, Andrei

Isn't it obvious, to be able to create tools that needs to parse/lex D code. If parser/lexer is included in Phobos that is basically a port of the C++ one in DMD and DMD starts to use the D version it will be a huge win. The parser/lexer in Phobos will always automatically be up to date with the one the compiler uses because they are actually the same. As far as I know DDMD cannot be easily used to build other tools that needs to parse/lex D code in it's current state. That is a long term goal but not even the short term goal, of being up to date with DMD and produce the same code as DMD, has been reached yet.

Along with your motivations, Jacob, see also: http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD Denis -- _________________ vita es estrany spir.wikidot.com
Mar 24 2011
prev sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 24/03/2011 04:30, Andrei Alexandrescu wrote:
 On 3/23/11 11:42 AM, Luca Boasso wrote:
 Sorry for the late reply,
 even tough I'm not an ANTLR expert, given my previous experience with
 the tool
 and having read most of the official book, I could help more on this GSOC
 idea.

 I have looked at http://www.antlr.org/grammar/list and on google and I
 could not
 find any recent ANTLR grammar for D.
 In my opinion this project could be a good contribution for the D
 community
 while being practical and useful in the short term.

There are two aspects: writing an ANTLR grammar that parses D (which you are referring to) and also adding to ANTLR the capability of generating D. The latter would allow D users to create and use parsers for various languages. It would also require close cooperation with the ANTLR people for acceptance etc. (Sticking with the former project for now.) This is a technically interesting project. I'd need to hear more about the high-level motivation. What is the "business" motivation? There is already a parser for D written in C++ (the open-source front-end of dmd), so the project would need to build a good case for the added value. Also there are other D parser projects that people work on (ddmd comes to mind), so I'd need to hear a lot more in the way of justifying the utility of such a project. Thanks, Andrei

There is a clear business motivation: If the ANTLR grammar that parses D is then used to generate a Java-based parser, this can be applied to the DDT IDE project and used instead of the current parser. This issue has been discussed in more detail here: http://www.digitalmars.com/d/archives/digitalmars/D/ide/Future_of_Descent_and_D_Eclipse_IDE_635.html The summary of the situation is: DDT currently uses the same parser as Descent, which is the parser of the Java port of the DMD frontend. Descent is no longer maintained, and neither is the port, so it has fallen behind D latest syntax, and it means certain syntaxes are not recognized properly by either IDE: http://code.google.com/a/eclipselabs.org/p/ddt/issues/detail?id=6 It may also be the case that the parser port has some performance issues (namely with memory usage, a sensitive point with the JVM). This has not been confirmed to be the case with the parser, but according to Ary B. it happens at least with the DMD semantic engine than Descent uses. Whereas DMD is a throwaway process and may not care much about certain memory management, the parser in Eclipse keeps running repeatedly... The alternative solution to this "business" issue is to keep doing things with the DMD port, but I don't think that's the best idea (see that discussion with Ary for more details). But it may as well be on the table for consideration to potential students. -- Bruno Medeiros - Software Engineer
Mar 24 2011
prev sibling parent spir <denis.spir gmail.com> writes:
On 03/24/2011 05:30 AM, Andrei Alexandrescu wrote:
 On 3/23/11 11:42 AM, Luca Boasso wrote:
 Sorry for the late reply,
 even tough I'm not an ANTLR expert, given my previous experience with the tool
 and having read most of the official book, I could help more on this GSOC
 idea.

 I have looked at http://www.antlr.org/grammar/list and on google and I could
not
 find any recent ANTLR grammar for D.
 In my opinion this project could be a good contribution for the D community
 while being practical and useful in the short term.

There are two aspects: writing an ANTLR grammar that parses D (which you are referring to) and also adding to ANTLR the capability of generating D. The latter would allow D users to create and use parsers for various languages. It would also require close cooperation with the ANTLR people for acceptance etc. (Sticking with the former project for now.) This is a technically interesting project. I'd need to hear more about the high-level motivation. What is the "business" motivation? There is already a parser for D written in C++ (the open-source front-end of dmd), so the project would need to build a good case for the added value. Also there are other D parser projects that people work on (ddmd comes to mind), so I'd need to hear a lot more in the way of justifying the utility of such a project.

http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD Denis -- _________________ vita es estrany spir.wikidot.com
Mar 24 2011
prev sibling parent spir <denis.spir gmail.com> writes:
On 03/23/2011 05:44 PM, Bruno Medeiros wrote:
 On 23/03/2011 14:59, spir wrote:
 On 03/23/2011 02:40 PM, Bruno Medeiros wrote:
 The first two items (LexingandParsing,
 ANTLRandJavabasedDparserforIDEusage) are fairly concrete ideas.
 The last one, D Tools in D is far more general, and can involve a lot of
 different things

It is instead more specific: "The aim is to boost the production of fundamental tools for analysis of D source code in D: lexical, syntactic, at best first steps of semantic analysis; producing code representations as D data structures." Its usefulness /in the long term/ can be said more general in that it would enable *further* creation of D tools in D (*), potentially useful to all D programmers. Unlike eg a general-purpose parsing lib in D which is specific to the parsing domain; on the other hand, a general-purpose parsing lib does not parse only D code ;-). (*) See (again): http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD Denis

Yeah, I agree that D tools are (generally speaking) much more useful than a D parsing library. But that's not the point, the point is that with the limited manpower we have available (not just in terms of GSoC, but generally), it's very risky and unwise to embark on projects that then remain incomplete and never see the light of day as part of something useful. It's like comparing a bicycle to a car engine. Yeah, the car engine can allow to build a nice car but it's worthless on it's own and at least the bicycle can get somewhere faster. Even a skate or roller-blades are more useful. How much development has it even been in any D tools in D lately (in the last year or so)?

That's not the point i meant, neither: a parser for D (& in d) is, imo, a far smaller project than a general-purpose parsing lib. Why do you think the opposite? Denis -- _________________ vita es estrany spir.wikidot.com
Mar 23 2011
prev sibling next sibling parent spir <denis.spir gmail.com> writes:
On 03/23/2011 02:40 PM, Bruno Medeiros wrote:
 The first two items (LexingandParsing, ANTLRandJavabasedDparserforIDEusage)
are fairly concrete ideas.
 The last one, D Tools in D is far more general, and can involve a lot of
 different things

It is instead more specific: "The aim is to boost the production of fundamental tools for analysis of D source code in D: lexical, syntactic, at best first steps of semantic analysis; producing code representations as D data structures." Its usefulness /in the long term/ can be said more general in that it would enable *further* creation of D tools in D (*), potentially useful to all D programmers. Unlike eg a general-purpose parsing lib in D which is specific to the parsing domain; on the other hand, a general-purpose parsing lib does not parse only D code ;-). (*) See (again): http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD Denis -- _________________ vita es estrany spir.wikidot.com
Mar 23 2011
prev sibling next sibling parent Luca Boasso <luke.boasso gmail.com> writes:
Sorry for the late reply,
even tough I'm not an ANTLR expert, given my previous experience with the tool
and having read most of the official book, I could help more on this GSOC
idea.

I have looked at http://www.antlr.org/grammar/list and on google and I could not
find any recent ANTLR grammar for D.
In my opinion this project could be a good contribution for the D community
while being practical and useful in the short term.

The current ANTLR v3 is much more powerful than the previous version since it
can handle LL(*) grammars instead of the classic LL(k) one [More on LL(*) at:
http://www.antlr.org/papers/LL-star-PLDI11.pdf].

I had a quick look at the parse.c of dmd compiler. It's a recursive descent
parser but honestly I don't know the grammar enough to understand if
it is LL(k)
but definitively it could be parsed efficiently with the LL(*) parsing strategy.

I think that the essential aspects listed in the idea "ANTLR and Java based D
parser for IDE usage" could be pursued in the given 3 months time-line.

Could somebody tell me something more about the AST that should be generated as
output of the parser?


Luca Boasso


On 3/23/11, Bruno Medeiros <brunodomedeiros+spam com.gmail> wrote:
 On 23/03/2011 14:59, spir wrote:
 On 03/23/2011 02:40 PM, Bruno Medeiros wrote:
 The first two items (LexingandParsing,
 ANTLRandJavabasedDparserforIDEusage) are fairly concrete ideas.
 The last one, D Tools in D is far more general, and can involve a lot of
 different things

It is instead more specific: "The aim is to boost the production of fundamental tools for analysis of D source code in D: lexical, syntactic, at best first steps of semantic analysis; producing code representations as D data structures." Its usefulness /in the long term/ can be said more general in that it would enable *further* creation of D tools in D (*), potentially useful to all D programmers. Unlike eg a general-purpose parsing lib in D which is specific to the parsing domain; on the other hand, a general-purpose parsing lib does not parse only D code ;-). (*) See (again): http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD Denis

Yeah, I agree that D tools are (generally speaking) much more useful than a D parsing library. But that's not the point, the point is that with the limited manpower we have available (not just in terms of GSoC, but generally), it's very risky and unwise to embark on projects that then remain incomplete and never see the light of day as part of something useful. It's like comparing a bicycle to a car engine. Yeah, the car engine can allow to build a nice car but it's worthless on it's own and at least the bicycle can get somewhere faster. Even a skate or roller-blades are more useful. How much development has it even been in any D tools in D lately (in the last year or so)? -- Bruno Medeiros - Software Engineer

Mar 23 2011
prev sibling parent Luca Boasso <luke.boasso gmail.com> writes:
Hello,

I'm new in the D community so I don't have a fully understanding of the short
term needs of the community itself. After reading the posts at
[http://www.digitalmars.com/d/archives/digitalmars/D/ide/Future_of_Descent_and_D_Eclipse_IDE_635.html]
I agree with Bruno that an ANTLR parser could be useful.

With the help of a parser generator it's also possible to experiment with the
syntax of the language since potential future changes of D grammar are (almost)
handled by a tool instead of a manual modification of the hand written parser.
On the other hand the D language v2, as far as I know, is quite stable, so there
is no a strong need to experiment.

I think that adding to ANTLR the capability of generating D would be a complex
task that cannot be accomplished in the time frame of GSOC, considering that I
will not be able to work full time.

Thanks,

Luca Boasso

On 3/24/11, Bruno Medeiros <brunodomedeiros+spam com.gmail> wrote:
 On 24/03/2011 04:30, Andrei Alexandrescu wrote:
 On 3/23/11 11:42 AM, Luca Boasso wrote:
 Sorry for the late reply,
 even tough I'm not an ANTLR expert, given my previous experience with
 the tool
 and having read most of the official book, I could help more on this GSOC
 idea.

 I have looked at http://www.antlr.org/grammar/list and on google and I
 could not
 find any recent ANTLR grammar for D.
 In my opinion this project could be a good contribution for the D
 community
 while being practical and useful in the short term.

There are two aspects: writing an ANTLR grammar that parses D (which you are referring to) and also adding to ANTLR the capability of generating D. The latter would allow D users to create and use parsers for various languages. It would also require close cooperation with the ANTLR people for acceptance etc. (Sticking with the former project for now.) This is a technically interesting project. I'd need to hear more about the high-level motivation. What is the "business" motivation? There is already a parser for D written in C++ (the open-source front-end of dmd), so the project would need to build a good case for the added value. Also there are other D parser projects that people work on (ddmd comes to mind), so I'd need to hear a lot more in the way of justifying the utility of such a project. Thanks, Andrei

There is a clear business motivation: If the ANTLR grammar that parses D is then used to generate a Java-based parser, this can be applied to the DDT IDE project and used instead of the current parser. This issue has been discussed in more detail here: http://www.digitalmars.com/d/archives/digitalmars/D/ide/Future_of_Descent_and_D_Eclipse_IDE_635.html The summary of the situation is: DDT currently uses the same parser as Descent, which is the parser of the Java port of the DMD frontend. Descent is no longer maintained, and neither is the port, so it has fallen behind D latest syntax, and it means certain syntaxes are not recognized properly by either IDE: http://code.google.com/a/eclipselabs.org/p/ddt/issues/detail?id=6 It may also be the case that the parser port has some performance issues (namely with memory usage, a sensitive point with the JVM). This has not been confirmed to be the case with the parser, but according to Ary B. it happens at least with the DMD semantic engine than Descent uses. Whereas DMD is a throwaway process and may not care much about certain memory management, the parser in Eclipse keeps running repeatedly... The alternative solution to this "business" issue is to keep doing things with the DMD port, but I don't think that's the best idea (see that discussion with Ary for more details). But it may as well be on the table for consideration to potential students. -- Bruno Medeiros - Software Engineer

Mar 24 2011