www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - D port of dmd: Lexer, Parser, AND CodeGenerator fully operational

reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
Check it out:
https://github.com/zachthemystic/ddmd-clean/

This program is an adaptation of the work done by the ddmd team:
http://www.dsource.org/projects/ddmd

I described most of it in the README. I hope it runs smoothly for 
you. I only ran it on MAC OSX, and I don't know much about github 
or about how to get things running smoothly for others.

Don't expect miracles. The parser is NOT up to date, e.g. => with 
the lastest lambda syntax.

I'll gladly put a license on it if the leaders of the community 
tell me which one to use ( Artistic, libpng, Boost ).

Onward and upward to IDE functionality!

Zach
Mar 07 2012
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Hi,

which version of the compiler can this be built with?

I get this with 2.058:
dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit of type
bool() overrides but is not covariant with
dmd.expression.Expression.isBit of type int()
dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit does not
override any function
Mar 07 2012
parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in message 
news:cibxxwrywnorlxwthent forum.dlang.org...
 On Wednesday, 7 March 2012 at 20:46:40 UTC, Andrej Mitrovic wrote:
 Hi,

 which version of the compiler can this be built with?

 I get this with 2.058:
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit of type
 bool() overrides but is not covariant with
 dmd.expression.Expression.isBit of type int()
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit does not
 override any function

Well, I was using 2.057 But obviously you've found a bug I can fix rather quickly!

You should check, but I think isBit is dead code anyway.
Mar 07 2012
parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in message 
news:duefgfqidzxwcfvgefac forum.dlang.org...
 On Thursday, 8 March 2012 at 01:38:43 UTC, Daniel Murphy wrote:
 You should check, but I think isBit is dead code anyway.

I think it is. I've left a number of dead codes because it helped me understand the whole system better. There's a lot of isXXX() functions which come in handy, so maybe isBit will too. I have no reason to cut it until I fully understand that it's not going to be useful.

It's a relic from when 'bit' was a basic type in D. I removed a bunch of bit-related stuff about six months ago but missed this one.
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 20:46:40 UTC, Andrej Mitrovic wrote:
 Hi,

 which version of the compiler can this be built with?

 I get this with 2.058:
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit of 
 type
 bool() overrides but is not covariant with
 dmd.expression.Expression.isBit of type int()
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit 
 does not
 override any function

Well, I was using 2.057 But obviously you've found a bug I can fix rather quickly!
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 21:06:25 UTC, Zach the Mystic wrote:
 On Wednesday, 7 March 2012 at 20:46:40 UTC, Andrej Mitrovic 
 wrote:
 Hi,

 which version of the compiler can this be built with?

 I get this with 2.058:
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit 
 of type
 bool() overrides but is not covariant with
 dmd.expression.Expression.isBit of type int()
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit 
 does not
 override any function

Well, I was using 2.057 But obviously you've found a bug I can fix rather quickly!

OK I think it's fixed. Zach
Mar 07 2012
prev sibling next sibling parent Robert Clipsham <robert octarineparrot.com> writes:
On 07/03/2012 20:02, Zach the Mystic wrote:
 Check it out:
 https://github.com/zachthemystic/ddmd-clean/

 This program is an adaptation of the work done by the ddmd team:
 http://www.dsource.org/projects/ddmd

 I described most of it in the README. I hope it runs smoothly for you. I
 only ran it on MAC OSX, and I don't know much about github or about how
 to get things running smoothly for others.

 Don't expect miracles. The parser is NOT up to date, e.g. => with the
 lastest lambda syntax.

 I'll gladly put a license on it if the leaders of the community tell me
 which one to use ( Artistic, libpng, Boost ).

 Onward and upward to IDE functionality!

 Zach

Just because I'm curious - how does it fare against the dmd testsuite/druntime unittests/phobos unittests? -- Robert http://octarineparrot.com/
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 21:26:46 UTC, Robert Clipsham wrote:
 Just because I'm curious - how does it fare against the dmd 
 testsuite/druntime unittests/phobos unittests?

Just because I'm ignorant, I have no idea whatsoever! I'm not even sure how to test these things. You totally gone way past my level!
Mar 07 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
You have a duplicate definition of:

static int isTPL(Parameter[] arguments)

in dmd.parameters.d
Mar 07 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
After fixing that dup definition the project compiles and runs on Win7
x64. Nice! :)
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 21:33:23 UTC, Andrej Mitrovic wrote:
 After fixing that dup definition the project compiles and runs 
 on Win7
 x64. Nice! :)

Thanks :-) And thanks for being the first to try it out, too!
Mar 07 2012
prev sibling next sibling parent Manfred Nowak <svv1999 hotmail.com> writes:
Zach the Mystic wrote:

 Check it out:

Done. Congrats! -manfred
Mar 07 2012
prev sibling next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in message 
news:afqmbmvuvizvgfooefqj forum.dlang.org...

 I'll gladly put a license on it if the leaders of the community tell me 
 which one to use ( Artistic, libpng, Boost ).

 Zach

It will need to be the same license as the frontend (GPL/Artistic). It should be at the top of each c++ source file.
Mar 07 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-03-08 05:54, Jonathan M Davis wrote:
 On Thursday, March 08, 2012 03:12:48 Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:
 "Zach the Mystic"<reachMINUSTHISzachgmail dot.com>  wrote in
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...

 I'll gladly put a license on it if the leaders of the
 community tell me which one to use ( Artistic, libpng, Boost ).

 Zach

It will need to be the same license as the frontend (GPL/Artistic). It should be at the top of each c++ source file.

It looks like the license is going to have to be GPL because it says so strictly in dmd's readme.txt. Somehow that license scares me, though. The "Free Software Foundation" seems like a very Orwellian institution to me. I hope it doesn't scare users away.

If you took it from ddmd, then it's definitely going to have to be GPL. Now, there is interest in having a D parser and lexer in Phobos. I don't know if your version will fit the bill (e.g. it must have a range-based API), but we need one at some point. The original idea was to more or less directly port dmd's lexer and parser with some adjustments to the API as necessary (primarily to make it range-based). But no one has had the time to complete such a project yet (I originally volunteered to do it, but I just haven't had the time). When that project was proposed, Walter agreed to let that port be Boost rather than GPL (since he holds the copyright and the port would be going in Phobos, which uses boost). The problem with what you have (even if the API and implementation were perfect) is that it comes from ddmd, which had other contributors working on it. So, you would have to get permission from not only Walter but all of the relevant ddmd contributors. If you were able to _that_, and it could get passed the review process, then what you've done could be put into Phobos. But that requires that you take the time and effort to take care of getting the appropriate permissions, making sure that the API and implementation are acceptable for Phobos, and putting it through the Phobos review process. It would be great if you could do that though. - Jonathan M Davis

It would be nice if the frontend written in D (which ever it will be) could be used by DMD. Then there wouldn't be any problems of being out of sync. -- /Jacob Carlborg
Mar 08 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-03-08 09:20, Jonathan M Davis wrote:
 On Thursday, March 08, 2012 09:11:03 Jacob Carlborg wrote:
 On 2012-03-08 05:54, Jonathan M Davis wrote:
 On Thursday, March 08, 2012 03:12:48 Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:
 "Zach the Mystic"<reachMINUSTHISzachgmail dot.com>   wrote in
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...

 I'll gladly put a license on it if the leaders of the
 community tell me which one to use ( Artistic, libpng, Boost ).

 Zach

It will need to be the same license as the frontend (GPL/Artistic). It should be at the top of each c++ source file.

It looks like the license is going to have to be GPL because it says so strictly in dmd's readme.txt. Somehow that license scares me, though. The "Free Software Foundation" seems like a very Orwellian institution to me. I hope it doesn't scare users away.

If you took it from ddmd, then it's definitely going to have to be GPL. Now, there is interest in having a D parser and lexer in Phobos. I don't know if your version will fit the bill (e.g. it must have a range-based API), but we need one at some point. The original idea was to more or less directly port dmd's lexer and parser with some adjustments to the API as necessary (primarily to make it range-based). But no one has had the time to complete such a project yet (I originally volunteered to do it, but I just haven't had the time). When that project was proposed, Walter agreed to let that port be Boost rather than GPL (since he holds the copyright and the port would be going in Phobos, which uses boost). The problem with what you have (even if the API and implementation were perfect) is that it comes from ddmd, which had other contributors working on it. So, you would have to get permission from not only Walter but all of the relevant ddmd contributors. If you were able to _that_, and it could get passed the review process, then what you've done could be put into Phobos. But that requires that you take the time and effort to take care of getting the appropriate permissions, making sure that the API and implementation are acceptable for Phobos, and putting it through the Phobos review process. It would be great if you could do that though. - Jonathan M Davis

It would be nice if the frontend written in D (which ever it will be) could be used by DMD. Then there wouldn't be any problems of being out of sync.

Well, having it be easier to keep in sync is one of the reasons that it was originally proposed to simply port the lexer in dmd's frontend to D. But having it _be_ the frontend of dmd would be even better. I don't know how interested Walter is or isn't in that, but having it be a port of the current frontend would likely make convincing him easier than it would be if it were one written from scratch. - Jonathan M davis

Exactly, that is what I'm thinking as well. Maybe it then can gradually be modified to better support an API useful for other things than just the compiler. Or building an higher level API on top of it that the compiler may not need to use. -- /Jacob Carlborg
Mar 08 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-03-08 08:21, Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis wrote:
 If you took it from ddmd, then it's definitely going to have to be GPL.

 Now, there is interest in having a D parser and lexer in Phobos. I
 don't know
 if your version will fit the bill (e.g. it must have a range-based
 API), but we
 need one at some point. The original idea was to more or less directly
 port
 dmd's lexer and parser with some adjustments to the API as necessary
 (primarily to make it range-based). But no one has had the time to
 complete
 such a project yet (I originally volunteered to do it, but I just
 haven't had
 the time).

 When that project was proposed, Walter agreed to let that port be
 Boost rather
 than GPL (since he holds the copyright and the port would be going in
 Phobos,
 which uses boost).

 The problem with what you have (even if the API and implementation were
 perfect) is that it comes from ddmd, which had other contributors
 working on
 it. So, you would have to get permission from not only Walter but all
 of the
 relevant ddmd contributors. If you were able to _that_, and it could get
 passed the review process, then what you've done could be put into
 Phobos. But
 that requires that you take the time and effort to take care of
 getting the
 appropriate permissions, making sure that the API and implementation are
 acceptable for Phobos, and putting it through the Phobos review
 process. It
 would be great if you could do that though.

 - Jonathan M Davis

This is great news. I was really worried that the license was etched in stone. I'll need help finding out who owns the code, plus legal advice if the process is more than just getting a simple confirmation email from each of the original authors.

You have my blessing for the small changes I contributed with. -- /Jacob Carlborg
Mar 08 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 01:38:43 UTC, Daniel Murphy wrote:
 You should check, but I think isBit is dead code anyway.

I think it is. I've left a number of dead codes because it helped me understand the whole system better. There's a lot of isXXX() functions which come in handy, so maybe isBit will too. I have no reason to cut it until I fully understand that it's not going to be useful.
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:
 "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in 
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...

 I'll gladly put a license on it if the leaders of the 
 community tell me which one to use ( Artistic, libpng, Boost ).

 Zach

It will need to be the same license as the frontend (GPL/Artistic). It should be at the top of each c++ source file.

It looks like the license is going to have to be GPL because it says so strictly in dmd's readme.txt. Somehow that license scares me, though. The "Free Software Foundation" seems like a very Orwellian institution to me. I hope it doesn't scare users away. So be it.
Mar 07 2012
prev sibling next sibling parent Ary Manzana <ary esperanto.org.ar> writes:
On 3/7/12 5:02 PM, Zach the Mystic wrote:
 Check it out:
 https://github.com/zachthemystic/ddmd-clean/

 This program is an adaptation of the work done by the ddmd team:
 http://www.dsource.org/projects/ddmd

 I described most of it in the README. I hope it runs smoothly for you. I
 only ran it on MAC OSX, and I don't know much about github or about how
 to get things running smoothly for others.

 Don't expect miracles. The parser is NOT up to date, e.g. => with the
 lastest lambda syntax.

 I'll gladly put a license on it if the leaders of the community tell me
 which one to use ( Artistic, libpng, Boost ).

 Onward and upward to IDE functionality!

 Zach

Very nice! I wonder, how did you port it? And also, how are you going to maintain it synchronized to the lasted dmd version? I ask, because I ported dmd to Java and it was really hard to maintain. I almost had to do it manually. Of course, it will be much easier to diff, copy to the D code and then clean the code a bit. But since the files are different (the names, the extensions, where things are located), I guess it will be a PITA and you'll eventually quit.
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 02:32:11 UTC, Ary Manzana wrote:
 Very nice!

 I wonder, how did you port it?

Manually! It was hard, but I'm new enough to programming in general that it was just as much of a learning process as a tedious grind or something.
 And also, how are you going to maintain it synchronized to the 
 lasted dmd version?

I'm under the impression that the syntax of the language changes much more rarely than the semantic and backend. While I don't have much experience with software maintenance, it will definitely be easier to maintain a mere parser than an entire semantic and backend. I have two more things to say here, actually. One, you know, in a perfect world there would not even _be_ a D compiler written in C++ (i.e. only one written in D!), so I don't see why additions to this compiler have to be precisely like they are in dmd. They just need to be consistent with the language specification. But two, I have a prospective use for this program which I am going to explore further. The idea is that the closer any given part of a compiler is to the human-readable code structure - i.e. the lexer is very close, the parser is a little further away, semantic even further, etc. - the more useful it will be as a development tool. As long as you can reverse any given compilation process - e.g. a parser is reversed by a code generator, a lexer by a "token generator", etc. - you can go backward and show the current state of the program to the user on the screen. So instead of focusing on getting ddmd to work as a full-fledged compiler, I ( we? ) could focus on adding features which are useful for the development process. For example, say we are able to implement templates, which happen pretty soon after parsing, I believe, and also a reverser which allows us to put their current state to screen. Now we could look at the structure which WOULD be generated when compiled. CTFE may also be within reach, but I'm not at all sure because I know very little about it. My immediate goal is to make an editor which doesn't work so much with text files as with fully parsed syntax trees, and merely converts them to code for displaying on the screen, and for writing to file to be sent to the actual D compiler. Continuing with this theme, I've thought about some of the problems I might encounter ahead. But I'm rather excited about the solutions too. One obvious problem is that a parsed program is likely to be much more memory intensive than a text file. The solution I imagine to this problem is to literally convert chunks of a program BACK into text when they go offscreen. In other words, you're storing it as text, but editing it as a program tree. Not only would this text take up less memory, but it would also be easy to search through as text if that were the preferred editing style for the moment. It would also make writing the whole thing to file much faster, which will be extremely necessary because the whole process hinges on being able to quickly convert your tree to file so that it can be picked up by dmd, for example. Also, you could easily use any text editing capabilities you'd already implemented on non-D files. Another thing I enjoy fantasizing about in this regard, is a program which duplicates its already parsed objects, hands them directly to its built-in compiler, and waits for the result without even needing to write to a file. Anyway, the first thing I need is a gui, and a code generator capable of coloring its output appropriately, so I'm working on that, but it's not (even close to) ready for show yet!
Mar 07 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 03:12:48 Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:
 "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...
 
 I'll gladly put a license on it if the leaders of the
 community tell me which one to use ( Artistic, libpng, Boost ).
 
 Zach

It will need to be the same license as the frontend (GPL/Artistic). It should be at the top of each c++ source file.

It looks like the license is going to have to be GPL because it says so strictly in dmd's readme.txt. Somehow that license scares me, though. The "Free Software Foundation" seems like a very Orwellian institution to me. I hope it doesn't scare users away.

If you took it from ddmd, then it's definitely going to have to be GPL. Now, there is interest in having a D parser and lexer in Phobos. I don't know if your version will fit the bill (e.g. it must have a range-based API), but we need one at some point. The original idea was to more or less directly port dmd's lexer and parser with some adjustments to the API as necessary (primarily to make it range-based). But no one has had the time to complete such a project yet (I originally volunteered to do it, but I just haven't had the time). When that project was proposed, Walter agreed to let that port be Boost rather than GPL (since he holds the copyright and the port would be going in Phobos, which uses boost). The problem with what you have (even if the API and implementation were perfect) is that it comes from ddmd, which had other contributors working on it. So, you would have to get permission from not only Walter but all of the relevant ddmd contributors. If you were able to _that_, and it could get passed the review process, then what you've done could be put into Phobos. But that requires that you take the time and effort to take care of getting the appropriate permissions, making sure that the API and implementation are acceptable for Phobos, and putting it through the Phobos review process. It would be great if you could do that though. - Jonathan M Davis
Mar 07 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 08.03.2012 11:48, Jonathan M Davis wrote:
 On Thursday, March 08, 2012 08:21:17 Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis wrote:
 If you took it from ddmd, then it's definitely going to have to
 be GPL.

 Now, there is interest in having a D parser and lexer in
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a
 range-based API), but we
 need one at some point. The original idea was to more or less
 directly port
 dmd's lexer and parser with some adjustments to the API as
 necessary
 (primarily to make it range-based). But no one has had the time
 to complete
 such a project yet (I originally volunteered to do it, but I
 just haven't had
 the time).

 When that project was proposed, Walter agreed to let that port
 be Boost rather
 than GPL (since he holds the copyright and the port would be
 going in Phobos,
 which uses boost).

 The problem with what you have (even if the API and
 implementation were
 perfect) is that it comes from ddmd, which had other
 contributors working on
 it. So, you would have to get permission from not only Walter
 but all of the
 relevant ddmd contributors. If you were able to _that_, and it
 could get
 passed the review process, then what you've done could be put
 into Phobos. But
 that requires that you take the time and effort to take care of
 getting the
 appropriate permissions, making sure that the API and
 implementation are
 acceptable for Phobos, and putting it through the Phobos review
 process. It
 would be great if you could do that though.

 - Jonathan M Davis

This is great news. I was really worried that the license was etched in stone. I'll need help finding out who owns the code, plus legal advice if the process is more than just getting a simple confirmation email from each of the original authors. I have some comments I feel are very interesting regarding the lexer and pointers. There are no pointers in any of the code besides the lexer, so I think that will be very satisfying to you. Now I don't know everything about ranges, but if you simply mean dynamic arrays, then yes, everything except the lexer uses arrays when necessary, although there's simply a lot of code which doesn't need them because most of the objects are really just lists of members, many of which are not arrays. About the lexer, one thing I realized about the Wild-West pointer style as I was porting it is that it must be blazing fast. To my understanding, to call p.popFront() requires two operations, ++p; followed by --p.length; plus possibly array bounds checking, I don't know. ++p is all that the current lexer needs. It used to only check for EOF at each junction, but since I'm parsing little chunks of code instead of whole files now, it checks "if ( p>= endBuf )" at the beginning of each token scan, which gets pretty close to not going out of bounds, since most tokens aren't very long. That lexer is a tribute to very fast programming of an old school which will go away if it changes. Still, I can sense a tidal wave of RANGES coming, and I fear I'll just have to bid the little thing goodbye! :-(

A range is not necessarily a dynamic array, though a dynamic array is a range. The lexer is going to need to take a range of dchar (which may or may not be an array), and it's probably going to need to return a range of tokens. The parser would then take a range of tokens and then output the AST in some form or other - it probably couldn't be range, but I'm not sure. And while the lexer would need to operate on generic ranges of dchar, it would probably have to be special-cased for strings in a number of places in order to make it faster (e.g. checking the first char in a string rather than using front when it's known that the value being checked against is an ASCII character and will therefore fit in a single char - front has to decode the next character, which is less efficient).

Simply put, the decisison on decoding should belong to lexer. Thus strings should be wrapped as input range of char, wchar & dchar respectively.
 So, if you're not familiar with ranges, you probably have a fair bit of
 learning ahead of you, and you're probably going to have to make a number of
 changes to your lexer and parser (though the majority of it will probably be
 able to stay intact). Unfortunately, a proper article and tutorial on them is
 currently lacking in spite of the fact that Phobos uses them heavily.
 Fortunately however, in a book that Ali Çehreli is writing on D, he has a
 chapter on ranges that should help get you started:

 http://ddili.org/ders/d.en/ranges.html

 But I'd suggest that you play around with ranges a fair bit (especially with
 strings) before trying to change what you have to use them. std.algorithm in
 particular makes heavy use of ranges. And it wouldn't surprise me at all if
 some portions of your lexer and parser really should be using some of Phobos'
 functions but isn't currently, because it's originally a port from C++. You
 should also make sure that you understand the basics of Unicode fairly well -
 especially with how they pertain to char, wchar, and dchar - since that will
 affect your ability to correctly translate code to use ranges as well as
 properly optimize them.

 It would probably help if other D developers who are more familiar with ranges
 took a look at what you have and maybe even helped you start adjusting your
 code, but I don't know how many will both have the time and be interested. If
 I have time, I'll probably start poking at it, but I don't know that I'll have
 time any time soon, much as I'd like to.

 Regardless, you need to familiarize yourself with ranges if you want to get
 the lexer and parser ready for inclusion in Phobos. And you really should
 familiarize yourself with them anyway, since they're heavily used in D code in
 general. Not being able to use ranges in D would be like not being able to use
 iterators in C++. You can program in it, but you'd be fairly crippled -
 particularly when dealing with the standard library.

 - Jonathan M Davis

-- Dmitry Olshansky
Mar 08 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 08.03.2012 22:46, Jonathan M Davis wrote:
 On Thursday, March 08, 2012 22:03:12 Dmitry Olshansky wrote:
 On 08.03.2012 11:48, Jonathan M Davis wrote:
 A range is not necessarily a dynamic array, though a dynamic array is a
 range. The lexer is going to need to take a range of dchar (which may or
 may not be an array), and it's probably going to need to return a range
 of tokens. The parser would then take a range of tokens and then output
 the AST in some form or other - it probably couldn't be range, but I'm
 not sure. And while the lexer would need to operate on generic ranges of
 dchar, it would probably have to be special-cased for strings in a number
 of places in order to make it faster (e.g. checking the first char in a
 string rather than using front when it's known that the value being
 checked against is an ASCII character and will therefore fit in a single
 char - front has to decode the next character, which is less efficient).

Simply put, the decisison on decoding should belong to lexer. Thus strings should be wrapped as input range of char, wchar& dchar respectively.

??? The normal way to handle this is to simply special-case certain operations. e.g. static if(Unqual!(isElementEncodingType!R) == char) { ... }

Does isElementEncodingType aware of anything other then string/wstring?
 else
 { ... }

 I'm not sure that wrapping char and wchar arrays in structs that treat them as
 ranges of char or wchar is a good idea. At minimum, I'm not aware of anything
 in Phobos currently working that way (unless you did something like that in
 std.regex?).

Well it's not that I'm insisting of _wrapping_ the arrays or some such in general sense. And yes, I had some hard experience in std.regex with UTF decoding and range design that doesn't exactly fits. What I'm truly against is going overly generic and automatically stomping on your performance. That being said the general design of string ranges has unnessary pessimization already as it's popFront does a UTF length lookup, identical operation is performed when decoding the first codepoint (.front). At any rate building lexer on top of ranges in current setting means using abstraction that _hides_ decoding inside. That's a bad idea, it's loosing a fight from the start. Why? Because in this case range can't know if decoding is needed at this particular state of lexer or not, it is generality that kills this by definition. Yeah, in general auto-magically decoding ranges are OK, as long as the stuff you do has cost much higher then decoding (~10 times would be fine) or things that you strictly can't do otherwise. Lexer doesn't fall into this criteria. Everything either treats them as generic ranges of dchar or
 special cases them. And when you want to be most efficient with string
 processing, I would think that you'd want to treat them exactly as the arrays
 of code units that they are rather than ranges - in which case treating them
 as generic ranges of dchar in most places and then special casing certain
 sections of code which can take advantage of the fact that they're arrays of
 code units seems like the way to go.

Yeah, no arguing that. The thing is that lexer as a whole is precisely one of these special cases. It's good as long as it's fast and that requires more control then "a generic input range of dchar". Now, speaking outside of this specific problem. Basically I would propose formalizing a kind of range that current string/wstring is. And that is a VariableLengthEncoding range (VLE range), a two in one - random access codeunit range and bidirectional 'codepoint' range. I've heard of attempts on this concept before, but now with a use case at hand it could be become more important. The problem is, I think, that current InputRange range is insufficent as it requres to calculate length of first element twice: one time in front and extra one in popFront. -- Dmitry Olshansky
Mar 08 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 09.03.2012 1:12, Jonathan M Davis wrote:
 On Friday, March 09, 2012 00:54:48 Dmitry Olshansky wrote:
 On 08.03.2012 22:46, Jonathan M Davis wrote:
 On Thursday, March 08, 2012 22:03:12 Dmitry Olshansky wrote:
 On 08.03.2012 11:48, Jonathan M Davis wrote:
 A range is not necessarily a dynamic array, though a dynamic array is a
 range. The lexer is going to need to take a range of dchar (which may or
 may not be an array), and it's probably going to need to return a range
 of tokens. The parser would then take a range of tokens and then output
 the AST in some form or other - it probably couldn't be range, but I'm
 not sure. And while the lexer would need to operate on generic ranges of
 dchar, it would probably have to be special-cased for strings in a
 number
 of places in order to make it faster (e.g. checking the first char in a
 string rather than using front when it's known that the value being
 checked against is an ASCII character and will therefore fit in a single
 char - front has to decode the next character, which is less efficient).

Simply put, the decisison on decoding should belong to lexer. Thus strings should be wrapped as input range of char, wchar& dchar respectively.

??? The normal way to handle this is to simply special-case certain operations. e.g. static if(Unqual!(isElementEncodingType!R) == char) { ... }

Does isElementEncodingType aware of anything other then string/wstring?

No. It uses isNarrowString. So, what you'd end up doing is having the lexer accept a generic range of dchar and then have specializations where appropriate for narrow strings. Nothing in Phobos' string handling really supports the idea of dealing with generic char or wchar ranges. It all processes ranges of dchar and specializes on narrow strings where appropriate.

The concept that *only* strings need special casing is broken. I recall somebody already stomped on this: i.e. any range that returns char (a wrapper range, over say memory-mapped file) passes by all the intricate special casing that does decoding of std.algorithm and friends. So to put it simply there is no way to tap into this *decoding by default* infrastructure.
 But is there really a use case for a generic range of char or wchar? I don't
 know. In general, I really don't think that there is.

Memory mapped file wrapper is one, it's just from the top of my head. There could be plenty of them, one needs just to look. 'I don't know' is not a solid argument, sorry. When dealing with ranges
 of characters, they're essentially always either strings or strings which have
 been wrapped in other ranges (generally by calling some other range-based
 function such as take or filter). And those range-based functions pretty much
 inevitably need to treat the strings as ranges of dchar to do what they do
 (potentially with specific optimizations for strings). They aren't designed to
 operate on ranges of char or wchar, and the only reason to make them do so
 would be if there were a use case where you needed a range of char or wchar
 which was not an array. But they're all either arrays or wrapped arrays. So,
 no such use case currently exists with Phobos, and I really question the
 usefulness of trying to optimize on a generic range of char or wchar -
 especially when many of the optimizations for arrays involve random access
 ranges, and if you have a random access range of char or wchar, I _really_
 question that it would ever be anything other than an array.

And that is problem, if you fail to see why we need to stop pretending that all char/wchar ranges are arrays, then I failed somewhere. In the same vane one would argue that there is no other random access range but array, yet there is. For instance if we explore containers there could be Tries, Dequeues and whatnot of char/wchar with their respective ranges.
 So, I'd advise against trying to operate on ranges of char or wchar and just
 stick to operating on ranges of dchar with optimizations for narrow strings
 where appropriate.

Probably the fact that in lexer it's not 'some place for optimizations, it's the whole thing' was missed. That's why I'm looking for more or less generic yet efficient way.
 Now, speaking outside of this specific problem.
 Basically I would propose formalizing a kind of range that current
 string/wstring is. And that is a VariableLengthEncoding range (VLE
 range), a two in one - random access codeunit range and bidirectional
 'codepoint' range. I've heard of attempts on this concept before, but
 now with a use case at hand it could be become more important.

There has been some talk of VLE ranges but really only with regards to the fact that strings are a case of that. Nothing has been done to generalize it. It may be something that should be looked into, but until we've formalized that, I don't think that something like the lexer should be built around the idea. It would be safer to stick with what we've been doing - operating on ranges of dchar and special-casing for narrow strings where appropriate. If need be, it should be possible to adjust it to use VLEs later.
 The problem is, I think, that current InputRange range is insufficent as
 it requres to calculate length of first element twice: one time in front
 and extra one in popFront.

Perhaps an optional function should be added to input ranges which both returns and front and pops it. But VLE ranges such as narrow strings are probably the only ones which would really care, and we can already special case for strings, so it would probably only be of real value for general VLEs.

The goal is to make std.algorithm general when it comes to UTF-x ranges, VLE range seems a best suited abstraction level so far. Other things like base64 encoded stuff could be there, though it needs some thought.
 However, since VLEs would arguably be a new range type, they could just
 implement the new function, and anything which special-cases VLEs can take
 advantage of it, while they still work just fine as input ranges. We could even
 add a free function which both pops and returns front which uses front and
 popFront for most ranges and the new function for VLEs, so you can code with
 that free function in cases where you intend to both take front and pop it off
 without caring about what type of range you're dealing with and without
 forcing ranges in general to implement an extra function.

 There's definitely more work and designing to be done here, but there are
 definitely possibilities.

 Still, in the interim, I'd advise simply special casing on narrow strings in
 the lexer rather than trying to use VLEs initially.

Aye, and write 2 copies of lexer, yikes. I would just ignore *default* array ranges for the moment and do whatever is convenient & fast. Then convince people to provide proper abstraction. The problem of *default* ranges is that they have to do extra work to maintain their invariants, e.g. even array should adjust it's length. Like OP mentioned popFront is --length, ++ptr; whereas ptr++ is all required. So clearly a wrapper of array that doesn't need to adjust length could be faster. A-la: struct Wrapped(E){ private://not visible E[] arr; size_t idx; public: property auto front(){ arr[idx]; } ... } Or even 2 pointer (current+end) version. -- Dmitry Olshansky
Mar 08 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 09.03.2012 11:58, Jonathan M Davis wrote:
 On Friday, March 09, 2012 11:53:51 Dmitry Olshansky wrote:
 The goal is to make std.algorithm general when it comes to UTF-x ranges,
 VLE range seems a best suited abstraction level so far. Other things
 like base64 encoded stuff could be there, though it needs some thought.

My point is that it doesn't make sense to try and design the lexer around an as yet undesigned and unused VLE range.

Interface-wise, yes. Either the lexer should just move
 forward with how things are currently done and then be adjusted later to use
 VLE ranges once they've been sorted out, or it should be postponed until VLE
 ranges are sorted out.

starters) (w|d)string interface. -- Dmitry Olshansky
Mar 09 2012
prev sibling next sibling parent dolive <dolive89 sina.com> writes:
Zach the Mystic Wrote:

 Check it out:
 https://github.com/zachthemystic/ddmd-clean/
 
 This program is an adaptation of the work done by the ddmd team:
 http://www.dsource.org/projects/ddmd
 
 I described most of it in the README. I hope it runs smoothly for 
 you. I only ran it on MAC OSX, and I don't know much about github 
 or about how to get things running smoothly for others.
 
 Don't expect miracles. The parser is NOT up to date, e.g. => with 
 the lastest lambda syntax.
 
 I'll gladly put a license on it if the leaders of the community 
 tell me which one to use ( Artistic, libpng, Boost ).
 
 Onward and upward to IDE functionality!
 
 Zach

Great work ! Ask a few questions : Is there documentation? can it parse dmd c source file ? ( Auto- complete feature of the ide will involve dmd c source file ? ) thank's Dolive
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 05:05:46 UTC, Daniel Murphy wrote:
 "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in 
 message
 news:duefgfqidzxwcfvgefac forum.dlang.org...
 On Thursday, 8 March 2012 at 01:38:43 UTC, Daniel Murphy wrote:
 You should check, but I think isBit is dead code anyway.

I think it is. I've left a number of dead codes because it helped me understand the whole system better. There's a lot of isXXX() functions which come in handy, so maybe isBit will too. I have no reason to cut it until I fully understand that it's not going to be useful.

It's a relic from when 'bit' was a basic type in D. I removed a bunch of bit-related stuff about six months ago but missed this one.

Okay, great. Now I DO fully understand that it's not going to be useful. Thanks for the info I couldn't possibly have known without consulting someone whose been here much longer than I!
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis wrote:
 If you took it from ddmd, then it's definitely going to have to 
 be GPL.

 Now, there is interest in having a D parser and lexer in 
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a 
 range-based API), but we
 need one at some point. The original idea was to more or less 
 directly port
 dmd's lexer and parser with some adjustments to the API as 
 necessary
 (primarily to make it range-based). But no one has had the time 
 to complete
 such a project yet (I originally volunteered to do it, but I 
 just haven't had
 the time).

 When that project was proposed, Walter agreed to let that port 
 be Boost rather
 than GPL (since he holds the copyright and the port would be 
 going in Phobos,
 which uses boost).

 The problem with what you have (even if the API and 
 implementation were
 perfect) is that it comes from ddmd, which had other 
 contributors working on
 it. So, you would have to get permission from not only Walter 
 but all of the
 relevant ddmd contributors. If you were able to _that_, and it 
 could get
 passed the review process, then what you've done could be put 
 into Phobos. But
 that requires that you take the time and effort to take care of 
 getting the
 appropriate permissions, making sure that the API and 
 implementation are
 acceptable for Phobos, and putting it through the Phobos review 
 process. It
 would be great if you could do that though.

 - Jonathan M Davis

This is great news. I was really worried that the license was etched in stone. I'll need help finding out who owns the code, plus legal advice if the process is more than just getting a simple confirmation email from each of the original authors. I have some comments I feel are very interesting regarding the lexer and pointers. There are no pointers in any of the code besides the lexer, so I think that will be very satisfying to you. Now I don't know everything about ranges, but if you simply mean dynamic arrays, then yes, everything except the lexer uses arrays when necessary, although there's simply a lot of code which doesn't need them because most of the objects are really just lists of members, many of which are not arrays. About the lexer, one thing I realized about the Wild-West pointer style as I was porting it is that it must be blazing fast. To my understanding, to call p.popFront() requires two operations, ++p; followed by --p.length; plus possibly array bounds checking, I don't know. ++p is all that the current lexer needs. It used to only check for EOF at each junction, but since I'm parsing little chunks of code instead of whole files now, it checks "if ( p >= endBuf )" at the beginning of each token scan, which gets pretty close to not going out of bounds, since most tokens aren't very long. That lexer is a tribute to very fast programming of an old school which will go away if it changes. Still, I can sense a tidal wave of RANGES coming, and I fear I'll just have to bid the little thing goodbye! :-(
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 05:41:46 UTC, dolive wrote:
 Great work !

 Ask a few questions :
 Is there documentation?  can it parse dmd c source file ? ( 
 Auto- complete feature of the ide will involve dmd c source 
 file ? )

 thank's

 Dolive

No documentation. Even the API is inconsistent between similar parsing calls. The only one called from outside originally was p.parseModule(), and that was after the buffer had been primed with detectByteOrderMarkandConvertToAscii (still not implemented), and #!ignoreFirstlineHashbang. Another downside is that you currently have to prime each buffer you load with p.nextToken() before calling p.parseXXX() to return what you want. Most of the parsing calls are made internally by the parser itself. It's going to take a little more effort to worked out a reasonable API for it, but I suppose that will also be the ideal time to document it. As far as parsing C source files, I think you'll need a C compiler for that!
Mar 07 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 07:21:19 UTC, Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis 
 wrote:
 If you took it from ddmd, then it's definitely going to have 
 to be GPL.

 Now, there is interest in having a D parser and lexer in 
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a 
 range-based API), but we
 need one at some point. The original idea was to more or less 
 directly port
 dmd's lexer and parser with some adjustments to the API as 
 necessary
 (primarily to make it range-based). But no one has had the 
 time to complete
 such a project yet (I originally volunteered to do it, but I 
 just haven't had
 the time).


I have another question. The parser need to parse the tokens into some kind of objects, which I presume will be the current dmd object structure? Many of the class members are only of value to semantic and can be cut. But as much value as a parser is, you need just as much work designing the structures of the objects it creates, which is mostly done, by the way, it's just I'm not sure what exactly you're trying to put into phobos.
Mar 07 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 08:21:17 Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis wrote:
 If you took it from ddmd, then it's definitely going to have to
 be GPL.
=20
 Now, there is interest in having a D parser and lexer in
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a
 range-based API), but we
 need one at some point. The original idea was to more or less
 directly port
 dmd's lexer and parser with some adjustments to the API as
 necessary
 (primarily to make it range-based). But no one has had the time
 to complete
 such a project yet (I originally volunteered to do it, but I
 just haven't had
 the time).
=20
 When that project was proposed, Walter agreed to let that port
 be Boost rather
 than GPL (since he holds the copyright and the port would be
 going in Phobos,
 which uses boost).
=20
 The problem with what you have (even if the API and
 implementation were
 perfect) is that it comes from ddmd, which had other
 contributors working on
 it. So, you would have to get permission from not only Walter
 but all of the
 relevant ddmd contributors. If you were able to _that_, and it
 could get
 passed the review process, then what you've done could be put
 into Phobos. But
 that requires that you take the time and effort to take care of
 getting the
 appropriate permissions, making sure that the API and
 implementation are
 acceptable for Phobos, and putting it through the Phobos review
 process. It
 would be great if you could do that though.
=20
 - Jonathan M Davis

This is great news. I was really worried that the license was etched in stone. I'll need help finding out who owns the code, plus legal advice if the process is more than just getting a simple confirmation email from each of the original authors. =20 I have some comments I feel are very interesting regarding the lexer and pointers. There are no pointers in any of the code besides the lexer, so I think that will be very satisfying to you. Now I don't know everything about ranges, but if you simply mean dynamic arrays, then yes, everything except the lexer uses arrays when necessary, although there's simply a lot of code which doesn't need them because most of the objects are really just lists of members, many of which are not arrays. =20 About the lexer, one thing I realized about the Wild-West pointer style as I was porting it is that it must be blazing fast. To my understanding, to call p.popFront() requires two operations, ++p; followed by --p.length; plus possibly array bounds checking, I don't know. =20 ++p is all that the current lexer needs. It used to only check for EOF at each junction, but since I'm parsing little chunks of code instead of whole files now, it checks "if ( p >=3D endBuf )" at the beginning of each token scan, which gets pretty close to not going out of bounds, since most tokens aren't very long. That lexer is a tribute to very fast programming of an old school which will go away if it changes. Still, I can sense a tidal wave of RANGES coming, and I fear I'll just have to bid the little thing goodbye! :-(

A range is not necessarily a dynamic array, though a dynamic array is a= range.=20 The lexer is going to need to take a range of dchar (which may or may = not be=20 an array), and it's probably going to need to return a range of tokens= . The=20 parser would then take a range of tokens and then output the AST in so= me form=20 or other - it probably couldn't be range, but I'm not sure. And while = the=20 lexer would need to operate on generic ranges of dchar, it would probab= ly have=20 to be special-cased for strings in a number of places in order to make = it=20 faster (e.g. checking the first char in a string rather than using fron= t when=20 it's known that the value being checked against is an ASCII character a= nd will=20 therefore fit in a single char - front has to decode the next character= , which=20 is less efficient). So, if you're not familiar with ranges, you probably have a fair bit of= =20 learning ahead of you, and you're probably going to have to make a numb= er of=20 changes to your lexer and parser (though the majority of it will probab= ly be=20 able to stay intact). Unfortunately, a proper article and tutorial on t= hem is=20 currently lacking in spite of the fact that Phobos uses them heavily.=20= Fortunately however, in a book that Ali =C3=87ehreli is writing on D, h= e has a=20 chapter on ranges that should help get you started: http://ddili.org/ders/d.en/ranges.html But I'd suggest that you play around with ranges a fair bit (especially= with=20 strings) before trying to change what you have to use them. std.algorit= hm in=20 particular makes heavy use of ranges. And it wouldn't surprise me at al= l if=20 some portions of your lexer and parser really should be using some of P= hobos'=20 functions but isn't currently, because it's originally a port from C++.= You=20 should also make sure that you understand the basics of Unicode fairly = well -=20 especially with how they pertain to char, wchar, and dchar - since that= will=20 affect your ability to correctly translate code to use ranges as well a= s=20 properly optimize them. It would probably help if other D developers who are more familiar with= ranges=20 took a look at what you have and maybe even helped you start adjusting = your=20 code, but I don't know how many will both have the time and be interest= ed. If=20 I have time, I'll probably start poking at it, but I don't know that I'= ll have=20 time any time soon, much as I'd like to. Regardless, you need to familiarize yourself with ranges if you want to= get=20 the lexer and parser ready for inclusion in Phobos. And you really shou= ld=20 familiarize yourself with them anyway, since they're heavily used in D = code in=20 general. Not being able to use ranges in D would be like not being able= to use=20 iterators in C++. You can program in it, but you'd be fairly crippled -= =20 particularly when dealing with the standard library. - Jonathan M Davis
Mar 07 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 08:45:13 Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 07:21:19 UTC, Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis
 
 wrote:
 If you took it from ddmd, then it's definitely going to have
 to be GPL.
 
 Now, there is interest in having a D parser and lexer in
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a
 range-based API), but we
 need one at some point. The original idea was to more or less
 directly port
 dmd's lexer and parser with some adjustments to the API as
 necessary
 (primarily to make it range-based). But no one has had the
 time to complete
 such a project yet (I originally volunteered to do it, but I
 just haven't had
 the time).


I have another question. The parser need to parse the tokens into some kind of objects, which I presume will be the current dmd object structure? Many of the class members are only of value to semantic and can be cut. But as much value as a parser is, you need just as much work designing the structures of the objects it creates, which is mostly done, by the way, it's just I'm not sure what exactly you're trying to put into phobos.

Well, the first concern is the lexer. As far as getting stuff into Phobos goes, I'd advise concentrating on getting the lexer ready rather than necessarily trying to do it all at once - _especially_ since the lexer needs to be usable separately from the parser. As for what the parser should output, it's going to need to be the Abstract Syntax Tree. What exactly that looks like in dmd, I don't know. But we're pretty much going to want all of that information outputted by the parser. Ideally, it would be possible to build a compiler on top of the lexer and parser in Phobos. But I can't really give you specifics without studying the code in detail. - Jonathan M Davis
Mar 07 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-03-07 21:02, Zach the Mystic wrote:
 Check it out:
 https://github.com/zachthemystic/ddmd-clean/

 This program is an adaptation of the work done by the ddmd team:
 http://www.dsource.org/projects/ddmd

 I described most of it in the README. I hope it runs smoothly for you. I
 only ran it on MAC OSX, and I don't know much about github or about how
 to get things running smoothly for others.

 Don't expect miracles. The parser is NOT up to date, e.g. => with the
 lastest lambda syntax.

 I'll gladly put a license on it if the leaders of the community tell me
 which one to use ( Artistic, libpng, Boost ).

 Onward and upward to IDE functionality!

 Zach

This looks awesome. -- /Jacob Carlborg
Mar 08 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 09:11:03 Jacob Carlborg wrote:
 On 2012-03-08 05:54, Jonathan M Davis wrote:
 On Thursday, March 08, 2012 03:12:48 Zach the Mystic wrote:
 On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:
 "Zach the Mystic"<reachMINUSTHISzachgmail dot.com>  wrote in
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...
 
 I'll gladly put a license on it if the leaders of the
 community tell me which one to use ( Artistic, libpng, Boost ).
 
 Zach

It will need to be the same license as the frontend (GPL/Artistic). It should be at the top of each c++ source file.

It looks like the license is going to have to be GPL because it says so strictly in dmd's readme.txt. Somehow that license scares me, though. The "Free Software Foundation" seems like a very Orwellian institution to me. I hope it doesn't scare users away.

If you took it from ddmd, then it's definitely going to have to be GPL. Now, there is interest in having a D parser and lexer in Phobos. I don't know if your version will fit the bill (e.g. it must have a range-based API), but we need one at some point. The original idea was to more or less directly port dmd's lexer and parser with some adjustments to the API as necessary (primarily to make it range-based). But no one has had the time to complete such a project yet (I originally volunteered to do it, but I just haven't had the time). When that project was proposed, Walter agreed to let that port be Boost rather than GPL (since he holds the copyright and the port would be going in Phobos, which uses boost). The problem with what you have (even if the API and implementation were perfect) is that it comes from ddmd, which had other contributors working on it. So, you would have to get permission from not only Walter but all of the relevant ddmd contributors. If you were able to _that_, and it could get passed the review process, then what you've done could be put into Phobos. But that requires that you take the time and effort to take care of getting the appropriate permissions, making sure that the API and implementation are acceptable for Phobos, and putting it through the Phobos review process. It would be great if you could do that though. - Jonathan M Davis

It would be nice if the frontend written in D (which ever it will be) could be used by DMD. Then there wouldn't be any problems of being out of sync.

Well, having it be easier to keep in sync is one of the reasons that it was originally proposed to simply port the lexer in dmd's frontend to D. But having it _be_ the frontend of dmd would be even better. I don't know how interested Walter is or isn't in that, but having it be a port of the current frontend would likely make convincing him easier than it would be if it were one written from scratch. - Jonathan M davis
Mar 08 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Wednesday, 7 March 2012 at 20:02:57 UTC, Zach the Mystic wrote:
 https://github.com/zachthemystic/ddmd-clean/

By the way, in compilers, »code generation« is commonly used to refer to the generation of machine code; so using the term to refer to .di file generation/pretty-printing could be misleading for some (it was for me, at least^^). David
Mar 08 2012
prev sibling next sibling parent James Miller <james aatch.net> writes:
On 8 March 2012 22:05, David Nadlinger <see klickverbot.at> wrote:
 On Wednesday, 7 March 2012 at 20:02:57 UTC, Zach the Mystic wrote:
 https://github.com/zachthemystic/ddmd-clean/


I would like to see the parser output an AST for use in other situations. It would be nice to have a tool that can analyse the AST and do thing with it (like autocompletion). Clang has done some pretty cool things in this respect, to the point that it can practically do code completion itself. -- James Miller
Mar 08 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 09:05:05 UTC, David Nadlinger wrote:
 On Wednesday, 7 March 2012 at 20:02:57 UTC, Zach the Mystic 
 wrote:
 https://github.com/zachthemystic/ddmd-clean/

By the way, in compilers, »code generation« is commonly used to refer to the generation of machine code; so using the term to refer to .di file generation/pretty-printing could be misleading for some (it was for me, at least^^). David

I'm sorry. I see your point. I guess I'll call it pretty-printing, although it also occurred to me to call it "disparsing", which then humorously led to the notion "dyslexing". Wait, did I spell that right? :-)
Mar 08 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 04:53:20 UTC, Zach the Mystic wrote:
 Anyway, the first thing I need is a gui, and a code generator 
 capable of coloring its output appropriately, so I'm working on 
 that, but it's not (even close to) ready for show yet!

By "Code Generator" I actually mean pretty-printer, a distinction I was recently made aware of.
Mar 08 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 07:49:57 UTC, Jonathan M Davis wrote:
 The lexer is going to need to take a range of dchar (which may 
 or may not be an array),
 And while the lexer would need to operate on generic ranges of 
 dchar, it would probably have to be special-cased for strings 
 in a number of places

I know what you mean. I actually cut out ddmd's conversion stuff because I had glanced over phobos I saw plenty of functions designed for this! I must have intuited what you are saying. dmd does all conversion to char* prior to sending the buffer to the lexer. I doubt there's a reason to change this procedure, only to put that conversion code directly into module dmd.lexer instead.
 The parser would then take a range of  tokens and then output 
 the AST in some form  or other - it probably couldn't be  
 range, but I'm not sure.

Dmd's AST is pretty idiosyncratic. Example: class FuncDeclaration (function declaration ) has a bunch of named members: { Identifier ident; // the function's name Parameter[] parameters; // its parameters Statement frequire; // the in{} contract, if present Statement fbody; // function body etc. Each one has its own name. I actually was working on how to turn it into a more iterable format, since if you want to edit the AST directly you're going to need to cursor down or up to the element you want. It's actually doable, but it's not a natural range-ish format. That's where I'm confused about the licensing issues, since I'm not sure if the particular object structure which gets parsed is also going to be in phobos or if it must remain GPL, which I'm not sure I want to continue using.
 So, if you're not familiar with ranges, you probably have a 
 fair bit of
 learning ahead of you, and you're probably going to have to 
 make a number of
 changes to your lexer and parser (though the majority of it 
 will probably be
 able to stay intact). Unfortunately, a proper article and 
 tutorial on them is
 currently lacking in spite of the fact that Phobos uses them 
 heavily.
 Fortunately however, in a book that Ali Çehreli is writing on 
 D, he has a
 chapter on ranges that should help get you started:

 http://ddili.org/ders/d.en/ranges.html

 But I'd suggest that you play around with ranges a fair bit 
 (especially with
 strings) before trying to change what you have to use them. 
 std.algorithm in
 particular makes heavy use of ranges. And it wouldn't surprise 
 me at all if
 some portions of your lexer and parser really should be using 
 some of Phobos'
 functions but isn't currently, because it's originally a port 
 from C++. You
 should also make sure that you understand the basics of Unicode 
 fairly well -
 especially with how they pertain to char, wchar, and dchar - 
 since that will
 affect your ability to correctly translate code to use ranges 
 as well as
 properly optimize them.

 It would probably help if other D developers who are more 
 familiar with ranges
 took a look at what you have and maybe even helped you start 
 adjusting your
 code, but I don't know how many will both have the time and be 
 interested. If
 I have time, I'll probably start poking at it, but I don't know 
 that I'll have
 time any time soon, much as I'd like to.

 Regardless, you need to familiarize yourself with ranges if you 
 want to get
 the lexer and parser ready for inclusion in Phobos. And you 
 really should
 familiarize yourself with them anyway, since they're heavily 
 used in D code in
 general. Not being able to use ranges in D would be like not 
 being able to use
 iterators in C++. You can program in it, but you'd be fairly 
 crippled -
 particularly when dealing with the standard library.

 - Jonathan M Davis

Mar 08 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
I hit "send" before I was done writing!

On Thursday, 8 March 2012 at 07:49:57 UTC, Jonathan M Davis wrote:
 Fortunately however, in a book that Ali Çehreli is writing on 
 D, he has a
 chapter on ranges that should help get you started:

 http://ddili.org/ders/d.en/ranges.html

Thanks. This is a really helpful guide. Okay I'm done!
Mar 08 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 22:03:12 Dmitry Olshansky wrote:
 On 08.03.2012 11:48, Jonathan M Davis wrote:
 A range is not necessarily a dynamic array, though a dynamic array is a
 range. The lexer is going to need to take a range of dchar (which may or
 may not be an array), and it's probably going to need to return a range
 of tokens. The parser would then take a range of tokens and then output
 the AST in some form or other - it probably couldn't be range, but I'm
 not sure. And while the lexer would need to operate on generic ranges of
 dchar, it would probably have to be special-cased for strings in a number
 of places in order to make it faster (e.g. checking the first char in a
 string rather than using front when it's known that the value being
 checked against is an ASCII character and will therefore fit in a single
 char - front has to decode the next character, which is less efficient).

Simply put, the decisison on decoding should belong to lexer. Thus strings should be wrapped as input range of char, wchar & dchar respectively.

??? The normal way to handle this is to simply special-case certain operations. e.g. static if(Unqual!(isElementEncodingType!R) == char) { ... } else { ... } I'm not sure that wrapping char and wchar arrays in structs that treat them as ranges of char or wchar is a good idea. At minimum, I'm not aware of anything in Phobos currently working that way (unless you did something like that in std.regex?). Everything either treats them as generic ranges of dchar or special cases them. And when you want to be most efficient with string processing, I would think that you'd want to treat them exactly as the arrays of code units that they are rather than ranges - in which case treating them as generic ranges of dchar in most places and then special casing certain sections of code which can take advantage of the fact that they're arrays of code units seems like the way to go. The lexer is then choosing when something decodes, though the default is to decode, since it requires special-casing to avoid it. - Jonathan M Davis
Mar 08 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, March 09, 2012 00:54:48 Dmitry Olshansky wrote:
 On 08.03.2012 22:46, Jonathan M Davis wrote:
 On Thursday, March 08, 2012 22:03:12 Dmitry Olshansky wrote:
 On 08.03.2012 11:48, Jonathan M Davis wrote:
 A range is not necessarily a dynamic array, though a dynamic array is a
 range. The lexer is going to need to take a range of dchar (which may or
 may not be an array), and it's probably going to need to return a range
 of tokens. The parser would then take a range of tokens and then output
 the AST in some form or other - it probably couldn't be range, but I'm
 not sure. And while the lexer would need to operate on generic ranges of
 dchar, it would probably have to be special-cased for strings in a
 number
 of places in order to make it faster (e.g. checking the first char in a
 string rather than using front when it's known that the value being
 checked against is an ASCII character and will therefore fit in a single
 char - front has to decode the next character, which is less efficient).

Simply put, the decisison on decoding should belong to lexer. Thus strings should be wrapped as input range of char, wchar& dchar respectively.

??? The normal way to handle this is to simply special-case certain operations. e.g. static if(Unqual!(isElementEncodingType!R) == char) { ... }

Does isElementEncodingType aware of anything other then string/wstring?

No. It uses isNarrowString. So, what you'd end up doing is having the lexer accept a generic range of dchar and then have specializations where appropriate for narrow strings. Nothing in Phobos' string handling really supports the idea of dealing with generic char or wchar ranges. It all processes ranges of dchar and specializes on narrow strings where appropriate. But is there really a use case for a generic range of char or wchar? I don't know. In general, I really don't think that there is. When dealing with ranges of characters, they're essentially always either strings or strings which have been wrapped in other ranges (generally by calling some other range-based function such as take or filter). And those range-based functions pretty much inevitably need to treat the strings as ranges of dchar to do what they do (potentially with specific optimizations for strings). They aren't designed to operate on ranges of char or wchar, and the only reason to make them do so would be if there were a use case where you needed a range of char or wchar which was not an array. But they're all either arrays or wrapped arrays. So, no such use case currently exists with Phobos, and I really question the usefulness of trying to optimize on a generic range of char or wchar - especially when many of the optimizations for arrays involve random access ranges, and if you have a random access range of char or wchar, I _really_ question that it would ever be anything other than an array. So, I'd advise against trying to operate on ranges of char or wchar and just stick to operating on ranges of dchar with optimizations for narrow strings where appropriate.
 Now, speaking outside of this specific problem.
 Basically I would propose formalizing a kind of range that current
 string/wstring is. And that is a VariableLengthEncoding range (VLE
 range), a two in one - random access codeunit range and bidirectional
 'codepoint' range. I've heard of attempts on this concept before, but
 now with a use case at hand it could be become more important.

There has been some talk of VLE ranges but really only with regards to the fact that strings are a case of that. Nothing has been done to generalize it. It may be something that should be looked into, but until we've formalized that, I don't think that something like the lexer should be built around the idea. It would be safer to stick with what we've been doing - operating on ranges of dchar and special-casing for narrow strings where appropriate. If need be, it should be possible to adjust it to use VLEs later.
 The problem is, I think, that current InputRange range is insufficent as
 it requres to calculate length of first element twice: one time in front
 and extra one in popFront.

Perhaps an optional function should be added to input ranges which both returns and front and pops it. But VLE ranges such as narrow strings are probably the only ones which would really care, and we can already special case for strings, so it would probably only be of real value for general VLEs. However, since VLEs would arguably be a new range type, they could just implement the new function, and anything which special-cases VLEs can take advantage of it, while they still work just fine as input ranges. We could even add a free function which both pops and returns front which uses front and popFront for most ranges and the new function for VLEs, so you can code with that free function in cases where you intend to both take front and pop it off without caring about what type of range you're dealing with and without forcing ranges in general to implement an extra function. There's definitely more work and designing to be done here, but there are definitely possibilities. Still, in the interim, I'd advise simply special casing on narrow strings in the lexer rather than trying to use VLEs initially. - Jonathan M Davis
Mar 08 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 19:36:32 UTC, Jacob Carlborg wrote:
 This is great news. I was really worried that the license was 
 etched in
 stone. I'll need help finding out who owns the code, plus 
 legal advice
 if the process is more than just getting a simple confirmation 
 email
 from each of the original authors.

You have my blessing for the small changes I contributed with.

Great!
Mar 08 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 07:49:57 UTC, Jonathan M Davis wrote:
 Regardless, you need to familiarize yourself with ranges if you 
 want to get
 the lexer and parser ready for inclusion in Phobos.

I have to admit that I don't currently feel competent to do this work. I'm too green. But I do think that the program is already very close to what is needed. I also would assume that those experienced with dmd's C++ incarnation will find themselves on rather familiar soil if they set to work on the D version, so that might speed things up a little if they also know about ranges.
Mar 08 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, March 09, 2012 11:53:51 Dmitry Olshansky wrote:
 The goal is to make std.algorithm general when it comes to UTF-x ranges,
 VLE range seems a best suited abstraction level so far. Other things
 like base64 encoded stuff could be there, though it needs some thought.

My point is that it doesn't make sense to try and design the lexer around an as yet undesigned and unused VLE range. Either the lexer should just move forward with how things are currently done and then be adjusted later to use VLE ranges once they've been sorted out, or it should be postponed until VLE ranges are sorted out. - Jonathan M Davis
Mar 08 2012
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
 Now, there is interest in having a D parser and lexer in Phobos. I don't  
 know
 if your version will fit the bill (e.g. it must have a range-based API),  
 but we
 need one at some point. The original idea was to more or less directly  
 port
 dmd's lexer and parser with some adjustments to the API as necessary
 (primarily to make it range-based). But no one has had the time to  
 complete
 such a project yet (I originally volunteered to do it, but I just  
 haven't had
 the time).

CTFE heavy for phobos though. https://gist.github.com/1255439 - generic lexer generator https://gist.github.com/1262321 - D lexer
Mar 09 2012