www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - D port of dmd: Lexer, Parser, AND CodeGenerator fully operational

reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
Check it out:
https://github.com/zachthemystic/ddmd-clean/

This program is an adaptation of the work done by the ddmd team:
http://www.dsource.org/projects/ddmd

I described most of it in the README. I hope it runs smoothly for 
you. I only ran it on MAC OSX, and I don't know much about github 
or about how to get things running smoothly for others.

Don't expect miracles. The parser is NOT up to date, e.g. => with 
the lastest lambda syntax.

I'll gladly put a license on it if the leaders of the community 
tell me which one to use ( Artistic, libpng, Boost ).

Onward and upward to IDE functionality!

Zach
Mar 07 2012
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Hi,

which version of the compiler can this be built with?

I get this with 2.058:
dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit of type
bool() overrides but is not covariant with
dmd.expression.Expression.isBit of type int()
dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit does not
override any function
Mar 07 2012
parent reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 20:46:40 UTC, Andrej Mitrovic wrote:

 Hi,

 which version of the compiler can this be built with?

 I get this with 2.058:
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit of 
 type
 bool() overrides but is not covariant with
 dmd.expression.Expression.isBit of type int()
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit 
 does not
 override any function


Well, I was using 2.057

But obviously you've found a bug I can fix rather quickly!
Mar 07 2012
next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 21:06:25 UTC, Zach the Mystic wrote:

 On Wednesday, 7 March 2012 at 20:46:40 UTC, Andrej Mitrovic 
 wrote:

 Hi,

 which version of the compiler can this be built with?

 I get this with 2.058:
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit 
 of type
 bool() overrides but is not covariant with
 dmd.expression.Expression.isBit of type int()
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit 
 does not
 override any function


 Well, I was using 2.057

 But obviously you've found a bug I can fix rather quickly!


OK I think it's fixed.

Zach
Mar 07 2012
prev sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in message 
news:cibxxwrywnorlxwthent forum.dlang.org...

 On Wednesday, 7 March 2012 at 20:46:40 UTC, Andrej Mitrovic wrote:

 Hi,

 which version of the compiler can this be built with?

 I get this with 2.058:
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit of type
 bool() overrides but is not covariant with
 dmd.expression.Expression.isBit of type int()
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit does not
 override any function


 Well, I was using 2.057

 But obviously you've found a bug I can fix rather quickly!


You should check, but I think isBit is dead code anyway.
Mar 07 2012
parent reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 01:38:43 UTC, Daniel Murphy wrote:

 You should check, but I think isBit is dead code anyway.


I think it is. I've left a number of dead codes because it helped 
me understand the whole system better. There's a lot of isXXX() 
functions which come in handy, so maybe isBit will too. I have no 
reason to cut it until I fully understand that it's not going to 
be useful.
Mar 07 2012
parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in message 
news:duefgfqidzxwcfvgefac forum.dlang.org...

 On Thursday, 8 March 2012 at 01:38:43 UTC, Daniel Murphy wrote:

 You should check, but I think isBit is dead code anyway.


 I think it is. I've left a number of dead codes because it helped me 
 understand the whole system better. There's a lot of isXXX() functions 
 which come in handy, so maybe isBit will too. I have no reason to cut it 
 until I fully understand that it's not going to be useful.


It's a relic from when 'bit' was a basic type in D.  I removed a bunch of 
bit-related stuff about six months ago but missed this one.
Mar 07 2012
parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 05:05:46 UTC, Daniel Murphy wrote:

 "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in 
 message
 news:duefgfqidzxwcfvgefac forum.dlang.org...

 On Thursday, 8 March 2012 at 01:38:43 UTC, Daniel Murphy wrote:

 You should check, but I think isBit is dead code anyway.


 I think it is. I've left a number of dead codes because it 
 helped me understand the whole system better. There's a lot of 
 isXXX() functions which come in handy, so maybe isBit will 
 too. I have no reason to cut it until I fully understand that 
 it's not going to be useful.


 It's a relic from when 'bit' was a basic type in D.  I removed 
 a bunch of
 bit-related stuff about six months ago but missed this one.


Okay, great. Now I DO fully understand that it's not going to be 
useful. Thanks for the info I couldn't possibly have known 
without consulting someone whose been here much longer than I!
Mar 07 2012
prev sibling next sibling parent reply dnewbie <run3 myopera.com> writes:
Zach the Mystic - I can't compile it.
dmd\binExp.d(115): Error: function dmd.binExp.AndAndExp.isBit of type bool()
overrides but is no
t covariant with dmd.expression.Expression.isBit of type int()
dmd\binExp.d(115): Error: function dmd.binExp.AndAndExp.isBit does not override
any function


On Wed, Mar 7, 2012, at 10:09 PM, Zach the Mystic wrote:

 On Wednesday, 7 March 2012 at 21:06:25 UTC, Zach the Mystic wrote:

 On Wednesday, 7 March 2012 at 20:46:40 UTC, Andrej Mitrovic 
 wrote:

 Hi,

 which version of the compiler can this be built with?

 I get this with 2.058:
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit 
 of type
 bool() overrides but is not covariant with
 dmd.expression.Expression.isBit of type int()
 dmd\binExp.d(324): Error: function dmd.binExp.EqualExp.isBit 
 does not
 override any function


 Well, I was using 2.057

 But obviously you've found a bug I can fix rather quickly!


 
 OK I think it's fixed.
 
 Zach
Mar 07 2012
parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 21:15:46 UTC, dnewbie wrote:

 Zach the Mystic - I can't compile it.


Still not working?
Mar 07 2012
prev sibling next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 07/03/2012 20:02, Zach the Mystic wrote:

 Check it out:
 https://github.com/zachthemystic/ddmd-clean/

 This program is an adaptation of the work done by the ddmd team:
 http://www.dsource.org/projects/ddmd

 I described most of it in the README. I hope it runs smoothly for you. I
 only ran it on MAC OSX, and I don't know much about github or about how
 to get things running smoothly for others.

 Don't expect miracles. The parser is NOT up to date, e.g. => with the
 lastest lambda syntax.

 I'll gladly put a license on it if the leaders of the community tell me
 which one to use ( Artistic, libpng, Boost ).

 Onward and upward to IDE functionality!

 Zach


Just because I'm curious - how does it fare against the dmd 
testsuite/druntime unittests/phobos unittests?

-- 
Robert
http://octarineparrot.com/
Mar 07 2012
next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 21:26:46 UTC, Robert Clipsham wrote:

 Just because I'm curious - how does it fare against the dmd 
 testsuite/druntime unittests/phobos unittests?


Just because I'm ignorant, I have no idea whatsoever!

I'm not even sure how to test these things. You totally gone way 
past my level!
Mar 07 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
You have a duplicate definition of:

static int isTPL(Parameter[] arguments)

in dmd.parameters.d
Mar 07 2012
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
After fixing that dup definition the project compiles and runs on Win7
x64. Nice! :)
Mar 07 2012
parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 21:33:23 UTC, Andrej Mitrovic wrote:

 After fixing that dup definition the project compiles and runs 
 on Win7
 x64. Nice! :)


Thanks :-)

And thanks for being the first to try it out, too!
Mar 07 2012
prev sibling next sibling parent reply dnewbie <run3 myopera.com> writes:
It's working..

Pretty cool :)



On Thu, Mar 8, 2012, at 12:07 AM, Zach the Mystic wrote:

 On Wednesday, 7 March 2012 at 21:15:46 UTC, dnewbie wrote:

 Zach the Mystic - I can't compile it.


 
 Still not working?
Mar 07 2012
parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Wednesday, 7 March 2012 at 23:35:48 UTC, dnewbie wrote:

 It's working..

 Pretty cool :)


I think you're starting new threads each time you respond. You 
should find a way to respond to only the posts you want and not 
start a new thread each time. Not that I don't like the free 
advertising, but yeah.

Thanks for trying out ddmd-clean!
Mar 07 2012
prev sibling next sibling parent Manfred Nowak <svv1999 hotmail.com> writes:
Zach the Mystic wrote:


 Check it out:


Done. Congrats!

-manfred
Mar 07 2012
prev sibling next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in message 
news:afqmbmvuvizvgfooefqj forum.dlang.org...


 I'll gladly put a license on it if the leaders of the community tell me 
 which one to use ( Artistic, libpng, Boost ).

 Zach


It will need to be the same license as the frontend (GPL/Artistic).  It 
should be at the top of each c++ source file.
Mar 07 2012
parent reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:

 "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in 
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...


 I'll gladly put a license on it if the leaders of the 
 community tell me which one to use ( Artistic, libpng, Boost ).

 Zach


 It will need to be the same license as the frontend 
 (GPL/Artistic).  It
 should be at the top of each c++ source file.


It looks like the license is going to have to be GPL because it 
says so strictly in dmd's readme.txt. Somehow that license scares 
me, though. The "Free Software Foundation" seems like a very 
Orwellian institution to me. I hope it doesn't scare users away.

So be it.
Mar 07 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 03:12:48 Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:

 "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> wrote in
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...
 

 I'll gladly put a license on it if the leaders of the
 community tell me which one to use ( Artistic, libpng, Boost ).
 
 Zach


 
 It will need to be the same license as the frontend
 (GPL/Artistic).  It
 should be at the top of each c++ source file.


 
 It looks like the license is going to have to be GPL because it
 says so strictly in dmd's readme.txt. Somehow that license scares
 me, though. The "Free Software Foundation" seems like a very
 Orwellian institution to me. I hope it doesn't scare users away.


If you took it from ddmd, then it's definitely going to have to be GPL.

Now, there is interest in having a D parser and lexer in Phobos. I don't know 
if your version will fit the bill (e.g. it must have a range-based API), but we 
need one at some point. The original idea was to more or less directly port 
dmd's lexer and parser with some adjustments to the API as necessary 
(primarily to make it range-based). But no one has had the time to complete 
such a project yet (I originally volunteered to do it, but I just haven't had 
the time).

When that project was proposed, Walter agreed to let that port be Boost rather 
than GPL (since he holds the copyright and the port would be going in Phobos, 
which uses boost).

The problem with what you have (even if the API and implementation were 
perfect) is that it comes from ddmd, which had other contributors working on 
it. So, you would have to get permission from not only Walter but all of the 
relevant ddmd contributors. If you were able to _that_, and it could get 
passed the review process, then what you've done could be put into Phobos. But 
that requires that you take the time and effort to take care of getting the 
appropriate permissions, making sure that the API and implementation are 
acceptable for Phobos, and putting it through the Phobos review process. It 
would be great if you could do that though.

- Jonathan M Davis
Mar 07 2012
next sibling parent reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis wrote:

 If you took it from ddmd, then it's definitely going to have to 
 be GPL.

 Now, there is interest in having a D parser and lexer in 
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a 
 range-based API), but we
 need one at some point. The original idea was to more or less 
 directly port
 dmd's lexer and parser with some adjustments to the API as 
 necessary
 (primarily to make it range-based). But no one has had the time 
 to complete
 such a project yet (I originally volunteered to do it, but I 
 just haven't had
 the time).

 When that project was proposed, Walter agreed to let that port 
 be Boost rather
 than GPL (since he holds the copyright and the port would be 
 going in Phobos,
 which uses boost).

 The problem with what you have (even if the API and 
 implementation were
 perfect) is that it comes from ddmd, which had other 
 contributors working on
 it. So, you would have to get permission from not only Walter 
 but all of the
 relevant ddmd contributors. If you were able to _that_, and it 
 could get
 passed the review process, then what you've done could be put 
 into Phobos. But
 that requires that you take the time and effort to take care of 
 getting the
 appropriate permissions, making sure that the API and 
 implementation are
 acceptable for Phobos, and putting it through the Phobos review 
 process. It
 would be great if you could do that though.

 - Jonathan M Davis


This is great news. I was really worried that the license was 
etched in stone. I'll need help finding out who owns the code, 
plus legal advice if the process is more than just getting a 
simple confirmation email from each of the original authors.

I have some comments I feel are very interesting regarding the 
lexer and pointers. There are no pointers in any of the code 
besides the lexer, so I think that will be very satisfying to 
you. Now I don't know everything about ranges, but if you simply 
mean dynamic arrays, then yes, everything except the lexer uses 
arrays when necessary, although there's simply a lot of code 
which doesn't need them because most of the objects are really 
just lists of members, many of which are not arrays.

About the lexer, one thing I realized about the Wild-West pointer 
style as I was porting it is that it must be blazing fast. To my 
understanding, to call p.popFront() requires two operations, ++p; 
followed by --p.length; plus possibly array bounds checking, I 
don't know.

++p is all that the current lexer needs. It used to only check 
for EOF at each junction, but since I'm parsing little chunks of 
code instead of whole files now, it checks "if ( p >= endBuf )" 
at the beginning of each token scan, which gets pretty close to 
not going out of bounds, since most tokens aren't very long. That 
lexer is a tribute to very fast programming of an old school 
which will go away if it changes. Still, I can sense a tidal wave 
of RANGES coming, and I fear I'll just have to bid the little 
thing goodbye! :-(
Mar 07 2012
next sibling parent reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 07:21:19 UTC, Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis 
 wrote:

 If you took it from ddmd, then it's definitely going to have 
 to be GPL.

 Now, there is interest in having a D parser and lexer in 
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a 
 range-based API), but we
 need one at some point. The original idea was to more or less 
 directly port
 dmd's lexer and parser with some adjustments to the API as 
 necessary
 (primarily to make it range-based). But no one has had the 
 time to complete
 such a project yet (I originally volunteered to do it, but I 
 just haven't had
 the time).




I have another question. The parser need to parse the tokens into 
some kind of objects, which I presume will be the current dmd 
object structure? Many of the class members are only of value to 
semantic and can be cut. But as much value as a parser is, you 
need just as much work designing the structures of the objects it 
creates, which is mostly done, by the way, it's just I'm not sure 
what exactly you're trying to put into phobos.
Mar 07 2012
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 08:45:13 Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 07:21:19 UTC, Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis
 
 wrote:

 If you took it from ddmd, then it's definitely going to have
 to be GPL.
 
 Now, there is interest in having a D parser and lexer in
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a
 range-based API), but we
 need one at some point. The original idea was to more or less
 directly port
 dmd's lexer and parser with some adjustments to the API as
 necessary
 (primarily to make it range-based). But no one has had the
 time to complete
 such a project yet (I originally volunteered to do it, but I
 just haven't had
 the time).




 
 I have another question. The parser need to parse the tokens into
 some kind of objects, which I presume will be the current dmd
 object structure? Many of the class members are only of value to
 semantic and can be cut. But as much value as a parser is, you
 need just as much work designing the structures of the objects it
 creates, which is mostly done, by the way, it's just I'm not sure
 what exactly you're trying to put into phobos.


Well, the first concern is the lexer. As far as getting stuff into Phobos goes, 
I'd advise concentrating on getting the lexer ready rather than necessarily 
trying to do it all at once - _especially_ since the lexer needs to be usable 
separately from the parser.

As for what the parser should output, it's going to need to be the Abstract 
Syntax Tree. What exactly that looks like in dmd, I don't know. But we're 
pretty much going to want all of that information outputted by the parser. 
Ideally, it would be possible to build a compiler on top of the lexer and 
parser in Phobos. But I can't really give you specifics without studying the 
code in detail.

- Jonathan M Davis
Mar 07 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 08:21:17 Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis wrote:

 If you took it from ddmd, then it's definitely going to have to
 be GPL.
=20
 Now, there is interest in having a D parser and lexer in
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a
 range-based API), but we
 need one at some point. The original idea was to more or less
 directly port
 dmd's lexer and parser with some adjustments to the API as
 necessary
 (primarily to make it range-based). But no one has had the time
 to complete
 such a project yet (I originally volunteered to do it, but I
 just haven't had
 the time).
=20
 When that project was proposed, Walter agreed to let that port
 be Boost rather
 than GPL (since he holds the copyright and the port would be
 going in Phobos,
 which uses boost).
=20
 The problem with what you have (even if the API and
 implementation were
 perfect) is that it comes from ddmd, which had other
 contributors working on
 it. So, you would have to get permission from not only Walter
 but all of the
 relevant ddmd contributors. If you were able to _that_, and it
 could get
 passed the review process, then what you've done could be put
 into Phobos. But
 that requires that you take the time and effort to take care of
 getting the
 appropriate permissions, making sure that the API and
 implementation are
 acceptable for Phobos, and putting it through the Phobos review
 process. It
 would be great if you could do that though.
=20
 - Jonathan M Davis


=20
 This is great news. I was really worried that the license was
 etched in stone. I'll need help finding out who owns the code,
 plus legal advice if the process is more than just getting a
 simple confirmation email from each of the original authors.
=20
 I have some comments I feel are very interesting regarding the
 lexer and pointers. There are no pointers in any of the code
 besides the lexer, so I think that will be very satisfying to
 you. Now I don't know everything about ranges, but if you simply
 mean dynamic arrays, then yes, everything except the lexer uses
 arrays when necessary, although there's simply a lot of code
 which doesn't need them because most of the objects are really
 just lists of members, many of which are not arrays.
=20
 About the lexer, one thing I realized about the Wild-West pointer
 style as I was porting it is that it must be blazing fast. To my
 understanding, to call p.popFront() requires two operations, ++p;
 followed by --p.length; plus possibly array bounds checking, I
 don't know.
=20
 ++p is all that the current lexer needs. It used to only check
 for EOF at each junction, but since I'm parsing little chunks of
 code instead of whole files now, it checks "if ( p >=3D endBuf )"
 at the beginning of each token scan, which gets pretty close to
 not going out of bounds, since most tokens aren't very long. That
 lexer is a tribute to very fast programming of an old school
 which will go away if it changes. Still, I can sense a tidal wave
 of RANGES coming, and I fear I'll just have to bid the little
 thing goodbye! :-(


A range is not necessarily a dynamic array, though a dynamic array is a=
 range.=20
The lexer is going to need to take  a range of dchar (which may or may =
not be=20
an array), and it's probably going  to need to return a range of tokens=
. The=20
parser would then take a range of  tokens and then output the AST in so=
me form=20
or other - it probably couldn't be  range, but I'm not sure. And while =
the=20
lexer would need to operate on generic ranges of dchar, it would probab=
ly have=20
to be special-cased for strings in a number of places in order to make =
it=20
faster (e.g. checking the first char in a string rather than using fron=
t when=20
it's known that the value being checked against is an ASCII character a=
nd will=20
therefore fit in a single char - front has to decode the next character=
, which=20
is less efficient).

So, if you're not familiar with ranges, you probably have a fair bit of=
=20
learning ahead of you, and you're probably going to have to make a numb=
er of=20
changes to your lexer and parser (though the majority of it will probab=
ly be=20
able to stay intact). Unfortunately, a proper article and tutorial on t=
hem is=20
currently lacking in spite of the fact that Phobos uses them heavily.=20=

Fortunately however, in a book that Ali =C3=87ehreli is writing on D, h=
e has a=20
chapter on ranges that should help get you started:

http://ddili.org/ders/d.en/ranges.html

But I'd suggest that you play around with ranges a fair bit (especially=
 with=20
strings) before trying to change what you have to use them. std.algorit=
hm in=20
particular makes heavy use of ranges. And it wouldn't surprise me at al=
l if=20
some portions of your lexer and parser really should be using some of P=
hobos'=20
functions but isn't currently, because it's originally a port from C++.=
 You=20
should also make sure that you understand the basics of Unicode fairly =
well -=20
especially with how they pertain to char, wchar, and dchar - since that=
 will=20
affect your ability to correctly translate code to use ranges as well a=
s=20
properly optimize them.

It would probably help if other D developers who are more familiar with=
 ranges=20
took a look at what you have and maybe even helped you start adjusting =
your=20
code, but I don't know how many will both have the time and be interest=
ed. If=20
I have time, I'll probably start poking at it, but I don't know that I'=
ll have=20
time any time soon, much as I'd like to.

Regardless, you need to familiarize yourself with ranges if you want to=
 get=20
the lexer and parser ready for inclusion in Phobos. And you really shou=
ld=20
familiarize yourself with them anyway, since they're heavily used in D =
code in=20
general. Not being able to use ranges in D would be like not being able=
 to use=20
iterators in C++. You can program in it, but you'd be fairly crippled -=
=20
particularly when dealing with the standard library.

- Jonathan M Davis
Mar 07 2012
next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 07:49:57 UTC, Jonathan M Davis wrote:

 The lexer is going to need to take a range of dchar (which may 
 or may not be an array),
 And while the lexer would need to operate on generic ranges of 
 dchar, it would probably have to be special-cased for strings 
 in a number of places


I know what you mean. I actually cut out ddmd's conversion stuff 
because I had glanced over phobos I saw plenty of functions 
designed for this! I must have intuited what you are saying. dmd 
does all conversion to char* prior to sending the buffer to the 
lexer. I doubt there's a reason to change this procedure, only to 
put that conversion code directly into module dmd.lexer instead.


 The parser would then take a range of  tokens and then output 
 the AST in some form  or other - it probably couldn't be  
 range, but I'm not sure.


Dmd's AST is pretty idiosyncratic.
Example: class FuncDeclaration (function declaration ) has a 
bunch of named members:
{
Identifier ident; // the function's name
Parameter[] parameters; // its parameters
Statement frequire; // the in{} contract, if present
Statement fbody; // function body
etc.

Each one has its own name. I actually was working on how to turn 
it into a more iterable format, since if you want to edit the AST 
directly you're going to need to cursor down or up to the element 
you want. It's actually doable, but it's not a natural range-ish 
format. That's where I'm confused about the licensing issues, 
since I'm not sure if the particular object structure which gets 
parsed is also going to be in phobos or if it must remain GPL, 
which I'm not sure I want to continue using.



 So, if you're not familiar with ranges, you probably have a 
 fair bit of
 learning ahead of you, and you're probably going to have to 
 make a number of
 changes to your lexer and parser (though the majority of it 
 will probably be
 able to stay intact). Unfortunately, a proper article and 
 tutorial on them is
 currently lacking in spite of the fact that Phobos uses them 
 heavily.
 Fortunately however, in a book that Ali Çehreli is writing on 
 D, he has a
 chapter on ranges that should help get you started:

 http://ddili.org/ders/d.en/ranges.html

 But I'd suggest that you play around with ranges a fair bit 
 (especially with
 strings) before trying to change what you have to use them. 
 std.algorithm in
 particular makes heavy use of ranges. And it wouldn't surprise 
 me at all if
 some portions of your lexer and parser really should be using 
 some of Phobos'
 functions but isn't currently, because it's originally a port 
 from C++. You
 should also make sure that you understand the basics of Unicode 
 fairly well -
 especially with how they pertain to char, wchar, and dchar - 
 since that will
 affect your ability to correctly translate code to use ranges 
 as well as
 properly optimize them.

 It would probably help if other D developers who are more 
 familiar with ranges
 took a look at what you have and maybe even helped you start 
 adjusting your
 code, but I don't know how many will both have the time and be 
 interested. If
 I have time, I'll probably start poking at it, but I don't know 
 that I'll have
 time any time soon, much as I'd like to.

 Regardless, you need to familiarize yourself with ranges if you 
 want to get
 the lexer and parser ready for inclusion in Phobos. And you 
 really should
 familiarize yourself with them anyway, since they're heavily 
 used in D code in
 general. Not being able to use ranges in D would be like not 
 being able to use
 iterators in C++. You can program in it, but you'd be fairly 
 crippled -
 particularly when dealing with the standard library.

 - Jonathan M Davis
Mar 08 2012
prev sibling next sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
I hit "send" before I was done writing!

On Thursday, 8 March 2012 at 07:49:57 UTC, Jonathan M Davis wrote:

 Fortunately however, in a book that Ali Çehreli is writing on 
 D, he has a
 chapter on ranges that should help get you started:

 http://ddili.org/ders/d.en/ranges.html


Thanks. This is a really helpful guide.

Okay I'm done!
Mar 08 2012
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 08.03.2012 11:48, Jonathan M Davis wrote:

 On Thursday, March 08, 2012 08:21:17 Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis wrote:

 If you took it from ddmd, then it's definitely going to have to
 be GPL.

 Now, there is interest in having a D parser and lexer in
 Phobos. I don't know
 if your version will fit the bill (e.g. it must have a
 range-based API), but we
 need one at some point. The original idea was to more or less
 directly port
 dmd's lexer and parser with some adjustments to the API as
 necessary
 (primarily to make it range-based). But no one has had the time
 to complete
 such a project yet (I originally volunteered to do it, but I
 just haven't had
 the time).

 When that project was proposed, Walter agreed to let that port
 be Boost rather
 than GPL (since he holds the copyright and the port would be
 going in Phobos,
 which uses boost).

 The problem with what you have (even if the API and
 implementation were
 perfect) is that it comes from ddmd, which had other
 contributors working on
 it. So, you would have to get permission from not only Walter
 but all of the
 relevant ddmd contributors. If you were able to _that_, and it
 could get
 passed the review process, then what you've done could be put
 into Phobos. But
 that requires that you take the time and effort to take care of
 getting the
 appropriate permissions, making sure that the API and
 implementation are
 acceptable for Phobos, and putting it through the Phobos review
 process. It
 would be great if you could do that though.

 - Jonathan M Davis


 This is great news. I was really worried that the license was
 etched in stone. I'll need help finding out who owns the code,
 plus legal advice if the process is more than just getting a
 simple confirmation email from each of the original authors.

 I have some comments I feel are very interesting regarding the
 lexer and pointers. There are no pointers in any of the code
 besides the lexer, so I think that will be very satisfying to
 you. Now I don't know everything about ranges, but if you simply
 mean dynamic arrays, then yes, everything except the lexer uses
 arrays when necessary, although there's simply a lot of code
 which doesn't need them because most of the objects are really
 just lists of members, many of which are not arrays.

 About the lexer, one thing I realized about the Wild-West pointer
 style as I was porting it is that it must be blazing fast. To my
 understanding, to call p.popFront() requires two operations, ++p;
 followed by --p.length; plus possibly array bounds checking, I
 don't know.

 ++p is all that the current lexer needs. It used to only check
 for EOF at each junction, but since I'm parsing little chunks of
 code instead of whole files now, it checks "if ( p>= endBuf )"
 at the beginning of each token scan, which gets pretty close to
 not going out of bounds, since most tokens aren't very long. That
 lexer is a tribute to very fast programming of an old school
 which will go away if it changes. Still, I can sense a tidal wave
 of RANGES coming, and I fear I'll just have to bid the little
 thing goodbye! :-(


 A range is not necessarily a dynamic array, though a dynamic array is a range.
 The lexer is going to need to take  a range of dchar (which may or may not be
 an array), and it's probably going  to need to return a range of tokens. The
 parser would then take a range of  tokens and then output the AST in some form
 or other - it probably couldn't be  range, but I'm not sure. And while the
 lexer would need to operate on generic ranges of dchar, it would probably have
 to be special-cased for strings in a number of places in order to make it
 faster (e.g. checking the first char in a string rather than using front when
 it's known that the value being checked against is an ASCII character and will
 therefore fit in a single char - front has to decode the next character, which
 is less efficient).


Simply put, the decisison on decoding should belong to lexer. Thus 
strings should be wrapped as input range of char, wchar & dchar 
respectively.


 So, if you're not familiar with ranges, you probably have a fair bit of
 learning ahead of you, and you're probably going to have to make a number of
 changes to your lexer and parser (though the majority of it will probably be
 able to stay intact). Unfortunately, a proper article and tutorial on them is
 currently lacking in spite of the fact that Phobos uses them heavily.
 Fortunately however, in a book that Ali Çehreli is writing on D, he has a
 chapter on ranges that should help get you started:

 http://ddili.org/ders/d.en/ranges.html

 But I'd suggest that you play around with ranges a fair bit (especially with
 strings) before trying to change what you have to use them. std.algorithm in
 particular makes heavy use of ranges. And it wouldn't surprise me at all if
 some portions of your lexer and parser really should be using some of Phobos'
 functions but isn't currently, because it's originally a port from C++. You
 should also make sure that you understand the basics of Unicode fairly well -
 especially with how they pertain to char, wchar, and dchar - since that will
 affect your ability to correctly translate code to use ranges as well as
 properly optimize them.

 It would probably help if other D developers who are more familiar with ranges
 took a look at what you have and maybe even helped you start adjusting your
 code, but I don't know how many will both have the time and be interested. If
 I have time, I'll probably start poking at it, but I don't know that I'll have
 time any time soon, much as I'd like to.

 Regardless, you need to familiarize yourself with ranges if you want to get
 the lexer and parser ready for inclusion in Phobos. And you really should
 familiarize yourself with them anyway, since they're heavily used in D code in
 general. Not being able to use ranges in D would be like not being able to use
 iterators in C++. You can program in it, but you'd be fairly crippled -
 particularly when dealing with the standard library.

 - Jonathan M Davis



-- 
Dmitry Olshansky
Mar 08 2012
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 22:03:12 Dmitry Olshansky wrote:

 On 08.03.2012 11:48, Jonathan M Davis wrote:

 A range is not necessarily a dynamic array, though a dynamic array is a
 range. The lexer is going to need to take a range of dchar (which may or
 may not be an array), and it's probably going to need to return a range
 of tokens. The parser would then take a range of tokens and then output
 the AST in some form or other - it probably couldn't be range, but I'm
 not sure. And while the lexer would need to operate on generic ranges of
 dchar, it would probably have to be special-cased for strings in a number
 of places in order to make it faster (e.g. checking the first char in a
 string rather than using front when it's known that the value being
 checked against is an ASCII character and will therefore fit in a single
 char - front has to decode the next character, which is less efficient).


 
 Simply put, the decisison on decoding should belong to lexer. Thus
 strings should be wrapped as input range of char, wchar & dchar
 respectively.


??? The normal way to handle this is to simply special-case certain 
operations. e.g.

static if(Unqual!(isElementEncodingType!R) == char)
{ ... }
else
{ ... }

I'm not sure that wrapping char and wchar arrays in structs that treat them as 
ranges of char or wchar is a good idea. At minimum, I'm not aware of anything 
in Phobos currently working that way (unless you did something like that in 
std.regex?). Everything either treats them as generic ranges of dchar or 
special cases them. And when you want to be most efficient with string 
processing, I would think that you'd want to treat them exactly as the arrays 
of code units that they are rather than ranges - in which case treating them 
as generic ranges of dchar in most places and then special casing certain 
sections of code which can take advantage of the fact that they're arrays of 
code units seems like the way to go. The lexer is then choosing when something 
decodes, though the default is to decode, since it requires special-casing to 
avoid it.

- Jonathan M Davis
Mar 08 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 08.03.2012 22:46, Jonathan M Davis wrote:

 On Thursday, March 08, 2012 22:03:12 Dmitry Olshansky wrote:

 On 08.03.2012 11:48, Jonathan M Davis wrote:

 A range is not necessarily a dynamic array, though a dynamic array is a
 range. The lexer is going to need to take a range of dchar (which may or
 may not be an array), and it's probably going to need to return a range
 of tokens. The parser would then take a range of tokens and then output
 the AST in some form or other - it probably couldn't be range, but I'm
 not sure. And while the lexer would need to operate on generic ranges of
 dchar, it would probably have to be special-cased for strings in a number
 of places in order to make it faster (e.g. checking the first char in a
 string rather than using front when it's known that the value being
 checked against is an ASCII character and will therefore fit in a single
 char - front has to decode the next character, which is less efficient).


 Simply put, the decisison on decoding should belong to lexer. Thus
 strings should be wrapped as input range of char, wchar&  dchar
 respectively.


 ??? The normal way to handle this is to simply special-case certain
 operations. e.g.

 static if(Unqual!(isElementEncodingType!R) == char)
 { ... }


Does isElementEncodingType aware of anything other then string/wstring?


 else
 { ... }

 I'm not sure that wrapping char and wchar arrays in structs that treat them as
 ranges of char or wchar is a good idea. At minimum, I'm not aware of anything
 in Phobos currently working that way (unless you did something like that in
 std.regex?).


Well it's not that I'm insisting of _wrapping_ the arrays or some such 
in general sense.  And yes, I had some hard experience in std.regex with 
UTF decoding and range design that doesn't exactly fits.

What I'm truly against is going overly generic and automatically 
stomping on your performance. That being said the general design of 
string ranges has unnessary pessimization already as it's popFront does 
a UTF length lookup, identical operation is  performed when decoding the 
first codepoint (.front).
At any rate building lexer on top of ranges in current setting means 
using abstraction that _hides_ decoding inside. That's a bad idea, it's 
loosing a fight from the start. Why? Because in this case range can't 
know if decoding is needed at this particular state of lexer or not, it 
is generality that kills this by definition.

Yeah, in general auto-magically decoding ranges are OK, as long as the 
stuff you do has cost much higher then decoding (~10 times would be 
fine) or things that you strictly can't do otherwise. Lexer doesn't fall 
into this criteria.

Everything either treats them as generic ranges of dchar or

 special cases them. And when you want to be most efficient with string
 processing, I would think that you'd want to treat them exactly as the arrays
 of code units that they are rather than ranges - in which case treating them
 as generic ranges of dchar in most places and then special casing certain
 sections of code which can take advantage of the fact that they're arrays of
 code units seems like the way to go.



Yeah, no arguing that. The thing is that lexer as a whole is precisely 
one of these special cases. It's good as long as it's fast and that 
requires more control then "a generic input range of dchar".

Now, speaking outside of this specific problem.
Basically I would propose formalizing a kind of range that current 
string/wstring is. And that is a VariableLengthEncoding range (VLE 
range), a two in one - random access codeunit range and bidirectional 
'codepoint' range. I've heard of attempts on this concept before, but 
now with a use case at hand it could be become more important.
The problem is, I think, that current InputRange range is insufficent as 
it requres to calculate length of first element twice: one time in front 
and extra one in popFront.



-- 
Dmitry Olshansky
Mar 08 2012
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, March 09, 2012 00:54:48 Dmitry Olshansky wrote:

 On 08.03.2012 22:46, Jonathan M Davis wrote:

 On Thursday, March 08, 2012 22:03:12 Dmitry Olshansky wrote:

 On 08.03.2012 11:48, Jonathan M Davis wrote:

 A range is not necessarily a dynamic array, though a dynamic array is a
 range. The lexer is going to need to take a range of dchar (which may or
 may not be an array), and it's probably going to need to return a range
 of tokens. The parser would then take a range of tokens and then output
 the AST in some form or other - it probably couldn't be range, but I'm
 not sure. And while the lexer would need to operate on generic ranges of
 dchar, it would probably have to be special-cased for strings in a
 number
 of places in order to make it faster (e.g. checking the first char in a
 string rather than using front when it's known that the value being
 checked against is an ASCII character and will therefore fit in a single
 char - front has to decode the next character, which is less efficient).


 
 Simply put, the decisison on decoding should belong to lexer. Thus
 strings should be wrapped as input range of char, wchar& dchar
 respectively.


 
 ??? The normal way to handle this is to simply special-case certain
 operations. e.g.
 
 static if(Unqual!(isElementEncodingType!R) == char)
 { ... }


 
 Does isElementEncodingType aware of anything other then string/wstring?


No. It uses isNarrowString. So, what you'd end up doing is having the lexer 
accept a generic range of dchar and then have specializations where 
appropriate for narrow strings. Nothing in Phobos' string handling really 
supports the idea of dealing with generic char or wchar ranges. It all 
processes ranges of dchar and specializes on narrow strings where appropriate.

But is there really a use case for a generic range of char or wchar? I don't 
know. In general, I really don't think that there is. When dealing with ranges 
of characters, they're essentially always either strings or strings which have 
been wrapped in other ranges (generally by calling some other range-based 
function such as take or filter). And those range-based functions pretty much 
inevitably need to treat the strings as ranges of dchar to do what they do 
(potentially with specific optimizations for strings). They aren't designed to 
operate on ranges of char or wchar, and the only reason to make them do so 
would be if there were a use case where you needed a range of char or wchar 
which was not an array. But they're all either arrays or wrapped arrays. So, 
no such use case currently exists with Phobos, and I really question the 
usefulness of trying to optimize on a generic range of char or wchar - 
especially when many of the optimizations for arrays involve random access 
ranges, and if you have a random access range of char or wchar, I _really_ 
question that it would ever be anything other than an array.

So, I'd advise against trying to operate on ranges of char or wchar and just 
stick to operating on ranges of dchar with optimizations for narrow strings 
where appropriate.


 Now, speaking outside of this specific problem.
 Basically I would propose formalizing a kind of range that current
 string/wstring is. And that is a VariableLengthEncoding range (VLE
 range), a two in one - random access codeunit range and bidirectional
 'codepoint' range. I've heard of attempts on this concept before, but
 now with a use case at hand it could be become more important.


There has been some talk of VLE ranges but really only with regards to the 
fact that strings are a case of that. Nothing has been done to generalize it. 
It may be something that should be looked into, but until we've formalized 
that, I don't think that something like the lexer should be built around the 
idea. It would be safer to stick with what we've been doing - operating on 
ranges of dchar and special-casing for narrow strings where appropriate. If 
need be, it should be possible to adjust it to use VLEs later.


 The problem is, I think, that current InputRange range is insufficent as
 it requres to calculate length of first element twice: one time in front
 and extra one in popFront.


Perhaps an optional function should be added to input ranges which both 
returns and front and pops it. But VLE ranges such as narrow strings are 
probably the only ones which would really care, and we can already special 
case for strings, so it would probably only be of real value for general VLEs. 
However, since VLEs would arguably be a new range type, they could just 
implement the new function, and anything which special-cases VLEs can take 
advantage of it, while they still work just fine as input ranges. We could even 
add a free function which both pops and returns front which uses front and 
popFront for most ranges and the new function for VLEs, so you can code with 
that free function in cases where you intend to both take front and pop it off 
without caring about what type of range you're dealing with and without 
forcing ranges in general to implement an extra function.

There's definitely more work and designing to be done here, but there are 
definitely possibilities.

Still, in the interim, I'd advise simply special casing on narrow strings in 
the lexer rather than trying to use VLEs initially.

- Jonathan M Davis
Mar 08 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 09.03.2012 1:12, Jonathan M Davis wrote:

 On Friday, March 09, 2012 00:54:48 Dmitry Olshansky wrote:

 On 08.03.2012 22:46, Jonathan M Davis wrote:

 On Thursday, March 08, 2012 22:03:12 Dmitry Olshansky wrote:

 On 08.03.2012 11:48, Jonathan M Davis wrote:

 A range is not necessarily a dynamic array, though a dynamic array is a
 range. The lexer is going to need to take a range of dchar (which may or
 may not be an array), and it's probably going to need to return a range
 of tokens. The parser would then take a range of tokens and then output
 the AST in some form or other - it probably couldn't be range, but I'm
 not sure. And while the lexer would need to operate on generic ranges of
 dchar, it would probably have to be special-cased for strings in a
 number
 of places in order to make it faster (e.g. checking the first char in a
 string rather than using front when it's known that the value being
 checked against is an ASCII character and will therefore fit in a single
 char - front has to decode the next character, which is less efficient).


 Simply put, the decisison on decoding should belong to lexer. Thus
 strings should be wrapped as input range of char, wchar&  dchar
 respectively.


 ??? The normal way to handle this is to simply special-case certain
 operations. e.g.

 static if(Unqual!(isElementEncodingType!R) == char)
 { ... }


 Does isElementEncodingType aware of anything other then string/wstring?


 No. It uses isNarrowString. So, what you'd end up doing is having the lexer
 accept a generic range of dchar and then have specializations where
 appropriate for narrow strings. Nothing in Phobos' string handling really
 supports the idea of dealing with generic char or wchar ranges. It all
 processes ranges of dchar and specializes on narrow strings where appropriate.


The concept that *only* strings need special casing is broken. I recall 
somebody already stomped on this: i.e. any range that returns char (a 
wrapper range, over say memory-mapped file) passes by all the intricate 
special casing that does decoding of std.algorithm and friends. So to 
put it simply there is no way to tap into this *decoding by default* 
infrastructure.


 But is there really a use case for a generic range of char or wchar? I don't
 know. In general, I really don't think that there is.


Memory mapped file wrapper is one, it's just from the top of my head. 
There could be plenty of them, one needs just to look. 'I don't
  know' is not a solid argument, sorry.

When dealing with ranges

 of characters, they're essentially always either strings or strings which have
 been wrapped in other ranges (generally by calling some other range-based
 function such as take or filter). And those range-based functions pretty much
 inevitably need to treat the strings as ranges of dchar to do what they do
 (potentially with specific optimizations for strings). They aren't designed to
 operate on ranges of char or wchar, and the only reason to make them do so
 would be if there were a use case where you needed a range of char or wchar
 which was not an array. But they're all either arrays or wrapped arrays. So,
 no such use case currently exists with Phobos, and I really question the
 usefulness of trying to optimize on a generic range of char or wchar -
 especially when many of the optimizations for arrays involve random access
 ranges, and if you have a random access range of char or wchar, I _really_
 question that it would ever be anything other than an array.


And that is problem, if you fail to see why we need to stop pretending 
that all char/wchar ranges are arrays, then I failed somewhere. In the 
same vane one would argue that there is no other random access range but 
array, yet there is. For instance if we explore containers there could 
be Tries, Dequeues  and whatnot of char/wchar with their respective ranges.


 So, I'd advise against trying to operate on ranges of char or wchar and just
 stick to operating on ranges of dchar with optimizations for narrow strings
 where appropriate.


Probably the fact that in lexer it's not 'some place for optimizations, 
it's the whole thing' was missed. That's why I'm looking for more or 
less generic yet efficient way.


 Now, speaking outside of this specific problem.
 Basically I would propose formalizing a kind of range that current
 string/wstring is. And that is a VariableLengthEncoding range (VLE
 range), a two in one - random access codeunit range and bidirectional
 'codepoint' range. I've heard of attempts on this concept before, but
 now with a use case at hand it could be become more important.


 There has been some talk of VLE ranges but really only with regards to the
 fact that strings are a case of that. Nothing has been done to generalize it.
 It may be something that should be looked into, but until we've formalized
 that, I don't think that something like the lexer should be built around the
 idea. It would be safer to stick with what we've been doing - operating on
 ranges of dchar and special-casing for narrow strings where appropriate. If
 need be, it should be possible to adjust it to use VLEs later.


 The problem is, I think, that current InputRange range is insufficent as
 it requres to calculate length of first element twice: one time in front
 and extra one in popFront.


 Perhaps an optional function should be added to input ranges which both
 returns and front and pops it. But VLE ranges such as narrow strings are
 probably the only ones which would really care, and we can already special
 case for strings, so it would probably only be of real value for general VLEs.


The goal is to make std.algorithm general when it comes to UTF-x ranges, 
VLE range seems a best suited abstraction level so far. Other things 
like base64 encoded stuff could be there, though it needs some thought.


 However, since VLEs would arguably be a new range type, they could just
 implement the new function, and anything which special-cases VLEs can take
 advantage of it, while they still work just fine as input ranges. We could even
 add a free function which both pops and returns front which uses front and
 popFront for most ranges and the new function for VLEs, so you can code with
 that free function in cases where you intend to both take front and pop it off
 without caring about what type of range you're dealing with and without
 forcing ranges in general to implement an extra function.

 There's definitely more work and designing to be done here, but there are
 definitely possibilities.

 Still, in the interim, I'd advise simply special casing on narrow strings in
 the lexer rather than trying to use VLEs initially.


Aye, and write 2 copies of lexer, yikes. I would just ignore *default* 
array ranges for the moment and do whatever is convenient & fast. Then 
convince people to provide proper abstraction.
The problem of *default* ranges is that they have to do extra work to 
maintain their invariants, e.g. even array should adjust it's length.
Like OP mentioned popFront is --length, ++ptr; whereas ptr++ is all 
required. So clearly a wrapper of array that doesn't need to adjust 
length could be faster. A-la:
struct Wrapped(E){
private://not visible
	E[] arr;
	size_t idx;
public:
	 property auto front(){ arr[idx]; }
...
}
Or even 2 pointer (current+end) version.

-- 
Dmitry Olshansky
Mar 08 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, March 09, 2012 11:53:51 Dmitry Olshansky wrote:

 The goal is to make std.algorithm general when it comes to UTF-x ranges,
 VLE range seems a best suited abstraction level so far. Other things
 like base64 encoded stuff could be there, though it needs some thought.


My point is that it doesn't make sense to try and design the lexer around an 
as yet undesigned and unused VLE range. Either the lexer should just move 
forward with how things are currently done and then be adjusted later to use 
VLE ranges once they've been sorted out, or it should be postponed until VLE 
ranges are sorted out.

- Jonathan M Davis
Mar 08 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 09.03.2012 11:58, Jonathan M Davis wrote:

 On Friday, March 09, 2012 11:53:51 Dmitry Olshansky wrote:

 The goal is to make std.algorithm general when it comes to UTF-x ranges,
 VLE range seems a best suited abstraction level so far. Other things
 like base64 encoded stuff could be there, though it needs some thought.


 My point is that it doesn't make sense to try and design the lexer around an
 as yet undesigned and unused VLE range.


Interface-wise, yes.

  Either the lexer should just move

 forward with how things are currently done and then be adjusted later to use
 VLE ranges once they've been sorted out, or it should be postponed until VLE
 ranges are sorted out.


Or it could use whatever abstraction it needs internally, providing (for 
starters) (w|d)string interface.

-- 
Dmitry Olshansky
Mar 09 2012
prev sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 07:49:57 UTC, Jonathan M Davis wrote:

 Regardless, you need to familiarize yourself with ranges if you 
 want to get
 the lexer and parser ready for inclusion in Phobos.


I have to admit that I don't currently feel competent to do this 
work. I'm too green. But I do think that the program is already 
very close to what is needed. I also would assume that those 
experienced with dmd's C++ incarnation will find themselves on 
rather familiar soil if they set to work on the D version, so 
that might speed things up a little if they also know about 
ranges.
Mar 08 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-03-08 08:21, Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 04:56:07 UTC, Jonathan M Davis wrote:

 If you took it from ddmd, then it's definitely going to have to be GPL.

 Now, there is interest in having a D parser and lexer in Phobos. I
 don't know
 if your version will fit the bill (e.g. it must have a range-based
 API), but we
 need one at some point. The original idea was to more or less directly
 port
 dmd's lexer and parser with some adjustments to the API as necessary
 (primarily to make it range-based). But no one has had the time to
 complete
 such a project yet (I originally volunteered to do it, but I just
 haven't had
 the time).

 When that project was proposed, Walter agreed to let that port be
 Boost rather
 than GPL (since he holds the copyright and the port would be going in
 Phobos,
 which uses boost).

 The problem with what you have (even if the API and implementation were
 perfect) is that it comes from ddmd, which had other contributors
 working on
 it. So, you would have to get permission from not only Walter but all
 of the
 relevant ddmd contributors. If you were able to _that_, and it could get
 passed the review process, then what you've done could be put into
 Phobos. But
 that requires that you take the time and effort to take care of
 getting the
 appropriate permissions, making sure that the API and implementation are
 acceptable for Phobos, and putting it through the Phobos review
 process. It
 would be great if you could do that though.

 - Jonathan M Davis


 This is great news. I was really worried that the license was etched in
 stone. I'll need help finding out who owns the code, plus legal advice
 if the process is more than just getting a simple confirmation email
 from each of the original authors.


You have my blessing for the small changes I contributed with.


-- 
/Jacob Carlborg
Mar 08 2012
parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 19:36:32 UTC, Jacob Carlborg wrote:

 This is great news. I was really worried that the license was 
 etched in
 stone. I'll need help finding out who owns the code, plus 
 legal advice
 if the process is more than just getting a simple confirmation 
 email
 from each of the original authors.


 You have my blessing for the small changes I contributed with.


Great!
Mar 08 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-03-08 05:54, Jonathan M Davis wrote:

 On Thursday, March 08, 2012 03:12:48 Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:

 "Zach the Mystic"<reachMINUSTHISzachgmail dot.com>  wrote in
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...


 I'll gladly put a license on it if the leaders of the
 community tell me which one to use ( Artistic, libpng, Boost ).

 Zach


 It will need to be the same license as the frontend
 (GPL/Artistic).  It
 should be at the top of each c++ source file.


 It looks like the license is going to have to be GPL because it
 says so strictly in dmd's readme.txt. Somehow that license scares
 me, though. The "Free Software Foundation" seems like a very
 Orwellian institution to me. I hope it doesn't scare users away.


 If you took it from ddmd, then it's definitely going to have to be GPL.

 Now, there is interest in having a D parser and lexer in Phobos. I don't know
 if your version will fit the bill (e.g. it must have a range-based API), but we
 need one at some point. The original idea was to more or less directly port
 dmd's lexer and parser with some adjustments to the API as necessary
 (primarily to make it range-based). But no one has had the time to complete
 such a project yet (I originally volunteered to do it, but I just haven't had
 the time).

 When that project was proposed, Walter agreed to let that port be Boost rather
 than GPL (since he holds the copyright and the port would be going in Phobos,
 which uses boost).

 The problem with what you have (even if the API and implementation were
 perfect) is that it comes from ddmd, which had other contributors working on
 it. So, you would have to get permission from not only Walter but all of the
 relevant ddmd contributors. If you were able to _that_, and it could get
 passed the review process, then what you've done could be put into Phobos. But
 that requires that you take the time and effort to take care of getting the
 appropriate permissions, making sure that the API and implementation are
 acceptable for Phobos, and putting it through the Phobos review process. It
 would be great if you could do that though.

 - Jonathan M Davis


It would be nice if the frontend written in D (which ever it will be) 
could be used by DMD. Then there wouldn't be any problems of being out 
of sync.

-- 
/Jacob Carlborg
Mar 08 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, March 08, 2012 09:11:03 Jacob Carlborg wrote:

 On 2012-03-08 05:54, Jonathan M Davis wrote:

 On Thursday, March 08, 2012 03:12:48 Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:

 "Zach the Mystic"<reachMINUSTHISzachgmail dot.com>  wrote in
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...
 

 I'll gladly put a license on it if the leaders of the
 community tell me which one to use ( Artistic, libpng, Boost ).
 
 Zach


 
 It will need to be the same license as the frontend
 (GPL/Artistic).  It
 should be at the top of each c++ source file.


 
 It looks like the license is going to have to be GPL because it
 says so strictly in dmd's readme.txt. Somehow that license scares
 me, though. The "Free Software Foundation" seems like a very
 Orwellian institution to me. I hope it doesn't scare users away.


 
 If you took it from ddmd, then it's definitely going to have to be GPL.
 
 Now, there is interest in having a D parser and lexer in Phobos. I don't
 know if your version will fit the bill (e.g. it must have a range-based
 API), but we need one at some point. The original idea was to more or
 less directly port dmd's lexer and parser with some adjustments to the
 API as necessary (primarily to make it range-based). But no one has had
 the time to complete such a project yet (I originally volunteered to do
 it, but I just haven't had the time).
 
 When that project was proposed, Walter agreed to let that port be Boost
 rather than GPL (since he holds the copyright and the port would be going
 in Phobos, which uses boost).
 
 The problem with what you have (even if the API and implementation were
 perfect) is that it comes from ddmd, which had other contributors working
 on it. So, you would have to get permission from not only Walter but all
 of the relevant ddmd contributors. If you were able to _that_, and it
 could get passed the review process, then what you've done could be put
 into Phobos. But that requires that you take the time and effort to take
 care of getting the appropriate permissions, making sure that the API and
 implementation are acceptable for Phobos, and putting it through the
 Phobos review process. It would be great if you could do that though.
 
 - Jonathan M Davis


 
 It would be nice if the frontend written in D (which ever it will be)
 could be used by DMD. Then there wouldn't be any problems of being out
 of sync.


Well, having it be easier to keep in sync is one of the reasons that it was 
originally proposed to simply port the lexer in dmd's frontend to D. But 
having it _be_ the frontend of dmd would be even better. I don't know how 
interested Walter is or isn't in that, but having it be a port of the current 
frontend would likely make convincing him easier than it would be if it were 
one written from scratch.

- Jonathan M davis
Mar 08 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-03-08 09:20, Jonathan M Davis wrote:

 On Thursday, March 08, 2012 09:11:03 Jacob Carlborg wrote:

 On 2012-03-08 05:54, Jonathan M Davis wrote:

 On Thursday, March 08, 2012 03:12:48 Zach the Mystic wrote:

 On Thursday, 8 March 2012 at 01:43:26 UTC, Daniel Murphy wrote:

 "Zach the Mystic"<reachMINUSTHISzachgmail dot.com>   wrote in
 message
 news:afqmbmvuvizvgfooefqj forum.dlang.org...


 I'll gladly put a license on it if the leaders of the
 community tell me which one to use ( Artistic, libpng, Boost ).

 Zach


 It will need to be the same license as the frontend
 (GPL/Artistic).  It
 should be at the top of each c++ source file.


 It looks like the license is going to have to be GPL because it
 says so strictly in dmd's readme.txt. Somehow that license scares
 me, though. The "Free Software Foundation" seems like a very
 Orwellian institution to me. I hope it doesn't scare users away.


 If you took it from ddmd, then it's definitely going to have to be GPL.

 Now, there is interest in having a D parser and lexer in Phobos. I don't
 know if your version will fit the bill (e.g. it must have a range-based
 API), but we need one at some point. The original idea was to more or
 less directly port dmd's lexer and parser with some adjustments to the
 API as necessary (primarily to make it range-based). But no one has had
 the time to complete such a project yet (I originally volunteered to do
 it, but I just haven't had the time).

 When that project was proposed, Walter agreed to let that port be Boost
 rather than GPL (since he holds the copyright and the port would be going
 in Phobos, which uses boost).

 The problem with what you have (even if the API and implementation were
 perfect) is that it comes from ddmd, which had other contributors working
 on it. So, you would have to get permission from not only Walter but all
 of the relevant ddmd contributors. If you were able to _that_, and it
 could get passed the review process, then what you've done could be put
 into Phobos. But that requires that you take the time and effort to take
 care of getting the appropriate permissions, making sure that the API and
 implementation are acceptable for Phobos, and putting it through the
 Phobos review process. It would be great if you could do that though.

 - Jonathan M Davis


 It would be nice if the frontend written in D (which ever it will be)
 could be used by DMD. Then there wouldn't be any problems of being out
 of sync.


 Well, having it be easier to keep in sync is one of the reasons that it was
 originally proposed to simply port the lexer in dmd's frontend to D. But
 having it _be_ the frontend of dmd would be even better. I don't know how
 interested Walter is or isn't in that, but having it be a port of the current
 frontend would likely make convincing him easier than it would be if it were
 one written from scratch.

 - Jonathan M davis


Exactly, that is what I'm thinking as well. Maybe it then can gradually 
be modified to better support an API useful for other things than just 
the compiler. Or building an higher level API on top of it that the 
compiler may not need to use.

-- 
/Jacob Carlborg
Mar 08 2012
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
 Now, there is interest in having a D parser and lexer in Phobos. I don't  
 know
 if your version will fit the bill (e.g. it must have a range-based API),  
 but we
 need one at some point. The original idea was to more or less directly  
 port
 dmd's lexer and parser with some adjustments to the API as necessary
 (primarily to make it range-based). But no one has had the time to  
 complete
 such a project yet (I originally volunteered to do it, but I just  
 haven't had
 the time).


I did wrote a complete D lexer some time ago. I'd consider it a little too  
CTFE heavy for phobos though.
https://gist.github.com/1255439 - generic lexer generator
https://gist.github.com/1262321 - D lexer
Mar 09 2012
prev sibling next sibling parent reply Ary Manzana <ary esperanto.org.ar> writes:
On 3/7/12 5:02 PM, Zach the Mystic wrote:

 Check it out:
 https://github.com/zachthemystic/ddmd-clean/

 This program is an adaptation of the work done by the ddmd team:
 http://www.dsource.org/projects/ddmd

 I described most of it in the README. I hope it runs smoothly for you. I
 only ran it on MAC OSX, and I don't know much about github or about how
 to get things running smoothly for others.

 Don't expect miracles. The parser is NOT up to date, e.g. => with the
 lastest lambda syntax.

 I'll gladly put a license on it if the leaders of the community tell me
 which one to use ( Artistic, libpng, Boost ).

 Onward and upward to IDE functionality!

 Zach


Very nice!

I wonder, how did you port it?

And also, how are you going to maintain it synchronized to the lasted 
dmd version?

I ask, because I ported dmd to Java and it was really hard to maintain. 
I almost had to do it manually.

Of course, it will be much easier to diff, copy to the D code and then 
clean the code a bit. But since the files are different (the names, the 
extensions, where things are located), I guess it will be a PITA and 
you'll eventually quit.
Mar 07 2012
parent reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 02:32:11 UTC, Ary Manzana wrote:

 Very nice!

 I wonder, how did you port it?


Manually!
It was hard, but I'm new enough to programming in general that it 
was just as much of a learning process as a tedious grind or 
something.


 And also, how are you going to maintain it synchronized to the 
 lasted dmd version?


I'm under the impression that the syntax of the language changes 
much more rarely than the semantic and backend. While I don't 
have much experience with software maintenance, it will 
definitely be easier to maintain a mere parser than an entire 
semantic and backend.

I have two more things to say here, actually. One, you know, in a 
perfect world there would not even _be_ a D compiler written in 
C++ (i.e. only one written in D!), so I don't see why additions 
to this compiler have to be precisely like they are in dmd. They 
just need to be consistent with the language specification.

But two, I have a prospective use for this program which I am 
going to explore further. The idea is that the closer any given 
part of a compiler is to the human-readable code structure - i.e. 
the lexer is very close, the parser is a little further away, 
semantic even further, etc. - the more useful it will be as a 
development tool. As long as you can reverse any given 
compilation process - e.g. a parser is reversed by a code 
generator, a lexer by a "token generator", etc. - you can go 
backward and show the current state of the program to the user on 
the screen.

So instead of focusing on getting ddmd to work as a full-fledged 
compiler, I ( we? ) could focus on adding features which are 
useful for the development process. For example, say we are able 
to implement templates, which happen pretty soon after parsing, I 
believe, and also a reverser which allows us to put their current 
state to screen. Now we could look at the structure which WOULD 
be generated when compiled. CTFE may also be within reach, but 
I'm not at all sure because I know very little about it.

My immediate goal is to make an editor which doesn't work so much 
with text files as with fully parsed syntax trees, and merely 
converts them to code for displaying on the screen, and for 
writing to file to be sent to the actual D compiler.

Continuing with this theme, I've thought about some of the 
problems I might encounter ahead. But I'm rather excited about 
the solutions too. One obvious problem is that a parsed program 
is likely to be much more memory intensive than a text file. The 
solution I imagine to this problem is to literally convert chunks 
of a program BACK into text when they go offscreen. In other 
words, you're storing it as text, but editing it as a program 
tree. Not only would this text take up less memory, but it would 
also be easy to search through as text if that were the preferred 
editing style for the moment. It would also make writing the 
whole thing to file much faster, which will be extremely 
necessary because the whole process hinges on being able to 
quickly convert your tree to file so that it can be picked up by 
dmd, for example. Also, you could easily use any text editing 
capabilities you'd already implemented on non-D files.

Another thing I enjoy fantasizing about in this regard, is a 
program which duplicates its already parsed objects, hands them 
directly to its built-in compiler, and waits for the result 
without even needing to write to a file.

Anyway, the first thing I need is a gui, and a code generator 
capable of coloring its output appropriately, so I'm working on 
that, but it's not (even close to) ready for show yet!
Mar 07 2012
parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 04:53:20 UTC, Zach the Mystic wrote:

 Anyway, the first thing I need is a gui, and a code generator 
 capable of coloring its output appropriately, so I'm working on 
 that, but it's not (even close to) ready for show yet!


By "Code Generator" I actually mean pretty-printer, a distinction 
I was recently made aware of.
Mar 08 2012
prev sibling next sibling parent reply dolive <dolive89 sina.com> writes:
Zach the Mystic Wrote:


 Check it out:
 https://github.com/zachthemystic/ddmd-clean/
 
 This program is an adaptation of the work done by the ddmd team:
 http://www.dsource.org/projects/ddmd
 
 I described most of it in the README. I hope it runs smoothly for 
 you. I only ran it on MAC OSX, and I don't know much about github 
 or about how to get things running smoothly for others.
 
 Don't expect miracles. The parser is NOT up to date, e.g. => with 
 the lastest lambda syntax.
 
 I'll gladly put a license on it if the leaders of the community 
 tell me which one to use ( Artistic, libpng, Boost ).
 
 Onward and upward to IDE functionality!
 
 Zach



Great work !


Ask a few questions :
Is there documentation?  can it parse dmd c source file ? ( Auto- complete
feature of the ide will involve dmd c source file ? )

thank's

Dolive
Mar 07 2012
parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 05:41:46 UTC, dolive wrote:

 Great work !

 Ask a few questions :
 Is there documentation?  can it parse dmd c source file ? ( 
 Auto- complete feature of the ide will involve dmd c source 
 file ? )

 thank's

 Dolive


No documentation. Even the API is inconsistent between similar 
parsing calls. The only one called from outside originally was 
p.parseModule(), and that was after the buffer had been primed 
with detectByteOrderMarkandConvertToAscii (still not 

that you currently have to prime each buffer you load with 
p.nextToken() before calling p.parseXXX() to return what you 
want. Most of the parsing calls are made internally by the parser 
itself. It's going to take a little more effort to worked out a 
reasonable API for it, but I suppose that will also be the ideal 
time to document it.

As far as parsing C source files, I think you'll need a C 
compiler for that!
Mar 07 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-03-07 21:02, Zach the Mystic wrote:

 Check it out:
 https://github.com/zachthemystic/ddmd-clean/

 This program is an adaptation of the work done by the ddmd team:
 http://www.dsource.org/projects/ddmd

 I described most of it in the README. I hope it runs smoothly for you. I
 only ran it on MAC OSX, and I don't know much about github or about how
 to get things running smoothly for others.

 Don't expect miracles. The parser is NOT up to date, e.g. => with the
 lastest lambda syntax.

 I'll gladly put a license on it if the leaders of the community tell me
 which one to use ( Artistic, libpng, Boost ).

 Onward and upward to IDE functionality!

 Zach


This looks awesome.

-- 
/Jacob Carlborg
Mar 08 2012
prev sibling parent reply "David Nadlinger" <see klickverbot.at> writes:
On Wednesday, 7 March 2012 at 20:02:57 UTC, Zach the Mystic wrote:

 https://github.com/zachthemystic/ddmd-clean/


By the way, in compilers, »code generation« is commonly used to 
refer to the generation of machine code; so using the term to 
refer to .di file generation/pretty-printing could be misleading 
for some (it was for me, at least^^).

David
Mar 08 2012
next sibling parent James Miller <james aatch.net> writes:
On 8 March 2012 22:05, David Nadlinger <see klickverbot.at> wrote:

 On Wednesday, 7 March 2012 at 20:02:57 UTC, Zach the Mystic wrote:

 https://github.com/zachthemystic/ddmd-clean/




I would like to see the parser output an AST for use in other
situations. It would be nice to have a tool that can analyse the AST
and do thing with it (like autocompletion). Clang has done some pretty
cool things in this respect, to the point that it can practically do
code completion itself.

--
James Miller
Mar 08 2012
prev sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Thursday, 8 March 2012 at 09:05:05 UTC, David Nadlinger wrote:

 On Wednesday, 7 March 2012 at 20:02:57 UTC, Zach the Mystic 
 wrote:

 https://github.com/zachthemystic/ddmd-clean/


 By the way, in compilers, »code generation« is commonly used 
 to refer to the generation of machine code; so using the term 
 to refer to .di file generation/pretty-printing could be 
 misleading for some (it was for me, at least^^).

 David


I'm sorry. I see your point. I guess I'll call it 
pretty-printing, although it also occurred to me to call it 
"disparsing", which then humorously led to the notion 
"dyslexing". Wait, did I spell that right? :-)
Mar 08 2012