www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Looking for champion - std.lang.d.lex

reply Walter Bright <newshound2 digitalmars.com> writes:
As we all know, tool support is important for D's success. Making tools easier 
to build will help with that.

To that end, I think we need a lexer for the standard library - std.lang.d.lex. 
It would be helpful in writing color syntax highlighting filters, pretty 
printers, repl, doc generators, static analyzers, and even D compilers.

It should:

1. support a range interface for its input, and a range interface for its output
2. optionally not generate lexical errors, but just try to recover and continue
3. optionally return comments and ddoc comments as tokens
4. the tokens should be a value type, not a reference type
5. generally follow along with the C++ one so that they can be maintained in
tandem

It can also serve as the basis for creating a javascript implementation that
can 
be embedded into web pages for syntax highlighting, and eventually an 
std.lang.d.parse.

Anyone want to own this?
Oct 21 2010
next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
and how about

6. ctfe compatible

?
Oct 21 2010
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday 21 October 2010 15:12:41 Ellery Newcomer wrote:
 and how about
 
 6. ctfe compatible
 
 ?

That would seem like a good idea (though part of me cringes at the idea of a program specifically running the lexer (and possibly the parser) as part of its own compilation process), but for the main purpose of being used for tools for D, that would seem completely unnecessary. So, I'd say that it would be a good idea to make it CTFE-able if it is at all reasonable to do so but that if making it CTFE-able would harm the design for more typical use, then it shouldn't be made CTFE-able. Personally, I don't have a good feel for exactly what is CTFE- able though, so I have no idea how easy it would be to make it CTFE-able. However, it does seem like a good idea if it's reasonable to do so. And if it's not, hopefully as dmd's CTFE capabilities become more advanced, it will become possible to do so. - Jonathan M Davis
Oct 21 2010
parent Don <nospam nospam.com> writes:
Jonathan M Davis wrote:
 On Thursday 21 October 2010 15:12:41 Ellery Newcomer wrote:
 and how about

 6. ctfe compatible

 ?

That would seem like a good idea (though part of me cringes at the idea of a program specifically running the lexer (and possibly the parser) as part of its own compilation process), but for the main purpose of being used for tools for D, that would seem completely unnecessary. So, I'd say that it would be a good idea to make it CTFE-able if it is at all reasonable to do so but that if making it CTFE-able would harm the design for more typical use, then it shouldn't be made CTFE-able. Personally, I don't have a good feel for exactly what is CTFE- able though, so I have no idea how easy it would be to make it CTFE-able. However, it does seem like a good idea if it's reasonable to do so. And if it's not, hopefully as dmd's CTFE capabilities become more advanced, it will become possible to do so. - Jonathan M Davis

In the long term, the requirements for CTFE will be pretty much: 1. the function must be safe (eg, no asm). 2. the function must be pure 3. the compiler must have access to the source code You'll probably satisfy all those requirements anyway.
Oct 22 2010
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, October 21, 2010 15:01:21 Walter Bright wrote:
 As we all know, tool support is important for D's success. Making tools
 easier to build will help with that.
 
 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax highlighting
 filters, pretty printers, repl, doc generators, static analyzers, and even
 D compilers.
 
 It should:
 
 1. support a range interface for its input, and a range interface for its
 output 2. optionally not generate lexical errors, but just try to recover
 and continue 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be maintained
 in tandem
 
 It can also serve as the basis for creating a javascript implementation
 that can be embedded into web pages for syntax highlighting, and
 eventually an std.lang.d.parse.
 
 Anyone want to own this?

You mean that you're going to make someone actually pull out their compiler book? ;) I'd love to do this (lexers and parsers are great fun IMHO - it's the code generation that isn't so fun), but I'm afraid that I'm busy enough at the moment that if I take it on, it won't get done very quickly. It is so very tempting though... So, as long as you're not in a hurry, I'm up for it, but I can't guarantee anything even approaching fast delivery. - Jonathan M Davis
Oct 21 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Jonathan M Davis:

 So, as long as you're not in a hurry, I'm up for it, but I can't guarantee 
 anything even approaching fast delivery.

You may open the project here: http://github.com/ And then other people may help you along the way. Bye, bearophile
Oct 21 2010
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, October 21, 2010 17:24:34 Russel Winder wrote:
 On Thu, 2010-10-21 at 19:51 -0400, bearophile wrote:
 Jonathan M Davis:
 So, as long as you're not in a hurry, I'm up for it, but I can't
 guarantee anything even approaching fast delivery.

You may open the project here: http://github.com/ And then other people may help you along the way.

Of course using BitBucket or Launchpad may well be more likely to get support as Mercurial and Bazaar are so much more usable that Git.

I've never actually used Mercurial or Bazaar. I do use git all the time though. I quite like it. Now, it could be Mercurial or Bazaar is better (like I said, I haven't used them), but I do find git to be quite useable. The simple fact that I can just create a repository in place instead of having to set up a separate location for a repository (like you have to do with svn) is a _huge_ improvement. I didn't really use source control on my personal projects before git. git actually makes it easy enough to do so that I do it all the time now. - Jonathan M Davis
Oct 21 2010
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Jonathan M Davis wrote:
 You mean that you're going to make someone actually pull out their compiler 
 book? ;)

Not really, you can just use the dmd lexer source as a guide. Should be straightforward.
 So, as long as you're not in a hurry, I'm up for it, but I can't guarantee 
 anything even approaching fast delivery.

As long as it gets done!
Oct 21 2010
prev sibling next sibling parent Russel Winder <russel russel.org.uk> writes:
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, 2010-10-21 at 19:51 -0400, bearophile wrote:
 Jonathan M Davis:
=20
 So, as long as you're not in a hurry, I'm up for it, but I can't guaran=


 anything even approaching fast delivery.

You may open the project here: http://github.com/ And then other people may help you along the way.

Of course using BitBucket or Launchpad may well be more likely to get support as Mercurial and Bazaar are so much more usable that Git. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel russel.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Oct 21 2010
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday 21 October 2010 15:01:21 Walter Bright wrote:
 5. generally follow along with the C++ one so that they can be maintained
 in tandem

Does this mean that you want a pseudo-port of the C++ front end's lexer to D for this? Or are you looking for just certain pieces of it to be similar? I haven't looked at the front end code yet, so I don't know how it works there, but I wouldn't expect it to uses ranges, for instance, so I would expect that the basic design would naturally stray a bit from whatever was done in C++ simply by doing things in fairly idiomatic D. And if I do look at the front end to see how that's done, there's the issue of the license. As I understand it, the front end is LGPL, and Phobos is generally Boost, which would mean that I would be looking at LGPL-licensed code when designing Boost-licensed, even though it wouldn't really be copying the code per se since it's a change of language (though if you did the whole front end, obviously the license issue can be waved quite easily). License issues aside, however, I do think that it would make sense for std.lang.d.lex to do things similiarly to the C++ front end, even if there are a number of basic differences. - Jonathan M Davis
Oct 21 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Jonathan M Davis wrote:
 On Thursday 21 October 2010 15:01:21 Walter Bright wrote:
 5. generally follow along with the C++ one so that they can be maintained
 in tandem

Does this mean that you want a pseudo-port of the C++ front end's lexer to D for this? Or are you looking for just certain pieces of it to be similar?

Yes, but not a straight port. The C++ version has things in it that are unnecessary for the D version, like the external string table (should use an associative array instead), the support for lookahead can be put in the parser, doesn't tokenize comments, etc. Essentially I'd like the D lexer to be self-contained in one file.
 I haven't looked at the front end code yet, so I don't know how it works
there, 
 but I wouldn't expect it to uses ranges, for instance, so I would expect that 
 the basic design would naturally stray a bit from whatever was done in C++ 
 simply by doing things in fairly idiomatic D. And if I do look at the front
end 
 to see how that's done, there's the issue of the license. As I understand it, 
 the front end is LGPL, and Phobos is generally Boost, which would mean that I 
 would be looking at LGPL-licensed code when designing Boost-licensed, even 
 though it wouldn't really be copying the code per se since it's a change of 
 language (though if you did the whole front end, obviously the license issue
can 
 be waved quite easily).

Since the license is mine, I can change the D version to the Boost license, no problem.
 License issues aside, however, I do think that it would make sense for 
 std.lang.d.lex to do things similiarly to the C++ front end, even if there are
a 
 number of basic differences.

Yup. The idea is the D version lexes exactly the same grammar as the dmd one. The easiest way to ensure that is to do equivalent logic.
Oct 21 2010
next sibling parent Lutger <lutger.blijdestijn gmail.com> writes:
Jonathan M Davis wrote:

...
 Okay. Good to know. I'll start looking at the C++ front end some time in
 the next few days, but like I said, I really don't know how much time I'm
 going to be able to spend on it, so it won't necessarily be quick.
 However, porting logic should be much faster than doing it from scratch.
 
 - Jonathan M Davis

If you are gonna port from the C++ front end, there is already a port called ddmd which may give you a head start: www.dsource.org/projects/ddmd
Oct 22 2010
prev sibling parent dolive <dolive89 sina.com> writes:
Walter Bright 写到:

 Jonathan M Davis wrote:
 On Thursday 21 October 2010 15:01:21 Walter Bright wrote:
 5. generally follow along with the C++ one so that they can be maintained
 in tandem

Does this mean that you want a pseudo-port of the C++ front end's lexer to D for this? Or are you looking for just certain pieces of it to be similar?

Yes, but not a straight port. The C++ version has things in it that are unnecessary for the D version, like the external string table (should use an associative array instead), the support for lookahead can be put in the parser, doesn't tokenize comments, etc. Essentially I'd like the D lexer to be self-contained in one file.
 I haven't looked at the front end code yet, so I don't know how it works
there, 
 but I wouldn't expect it to uses ranges, for instance, so I would expect that 
 the basic design would naturally stray a bit from whatever was done in C++ 
 simply by doing things in fairly idiomatic D. And if I do look at the front
end 
 to see how that's done, there's the issue of the license. As I understand it, 
 the front end is LGPL, and Phobos is generally Boost, which would mean that I 
 would be looking at LGPL-licensed code when designing Boost-licensed, even 
 though it wouldn't really be copying the code per se since it's a change of 
 language (though if you did the whole front end, obviously the license issue
can 
 be waved quite easily).

Since the license is mine, I can change the D version to the Boost license, no problem.
 License issues aside, however, I do think that it would make sense for 
 std.lang.d.lex to do things similiarly to the C++ front end, even if there are
a 
 number of basic differences.

Yup. The idea is the D version lexes exactly the same grammar as the dmd one. The easiest way to ensure that is to do equivalent logic.

dmd2.050 October will release it ? thank's
Oct 22 2010
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday 21 October 2010 23:55:42 Walter Bright wrote:
 Jonathan M Davis wrote:
 On Thursday 21 October 2010 15:01:21 Walter Bright wrote:
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

Does this mean that you want a pseudo-port of the C++ front end's lexer to D for this? Or are you looking for just certain pieces of it to be similar?

Yes, but not a straight port. The C++ version has things in it that are unnecessary for the D version, like the external string table (should use an associative array instead), the support for lookahead can be put in the parser, doesn't tokenize comments, etc. Essentially I'd like the D lexer to be self-contained in one file.
 I haven't looked at the front end code yet, so I don't know how it works
 there, but I wouldn't expect it to uses ranges, for instance, so I would
 expect that the basic design would naturally stray a bit from whatever
 was done in C++ simply by doing things in fairly idiomatic D. And if I
 do look at the front end to see how that's done, there's the issue of
 the license. As I understand it, the front end is LGPL, and Phobos is
 generally Boost, which would mean that I would be looking at
 LGPL-licensed code when designing Boost-licensed, even though it
 wouldn't really be copying the code per se since it's a change of
 language (though if you did the whole front end, obviously the license
 issue can be waved quite easily).

Since the license is mine, I can change the D version to the Boost license, no problem.
 License issues aside, however, I do think that it would make sense for
 std.lang.d.lex to do things similiarly to the C++ front end, even if
 there are a number of basic differences.

Yup. The idea is the D version lexes exactly the same grammar as the dmd one. The easiest way to ensure that is to do equivalent logic.

Okay. Good to know. I'll start looking at the C++ front end some time in the next few days, but like I said, I really don't know how much time I'm going to be able to spend on it, so it won't necessarily be quick. However, porting logic should be much faster than doing it from scratch. - Jonathan M Davis
Oct 22 2010
prev sibling next sibling parent reply dolive <dolive89 sina.com> writes:
Walter Bright 写到:

 As we all know, tool support is important for D's success. Making tools easier 
 to build will help with that.
 
 To that end, I think we need a lexer for the standard library -
std.lang.d.lex. 
 It would be helpful in writing color syntax highlighting filters, pretty 
 printers, repl, doc generators, static analyzers, and even D compilers.
 
 It should:
 
 1. support a range interface for its input, and a range interface for its
output
 2. optionally not generate lexical errors, but just try to recover and continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be maintained in
tandem
 
 It can also serve as the basis for creating a javascript implementation that
can 
 be embedded into web pages for syntax highlighting, and eventually an 
 std.lang.d.parse.
 
 Anyone want to own this?

Do you have Scintilla for D ?
Oct 22 2010
next sibling parent dolive <dolive89 sina.com> writes:
dolive 写到:

 Walter Bright 写到:
 
 As we all know, tool support is important for D's success. Making tools easier 
 to build will help with that.
 
 To that end, I think we need a lexer for the standard library -
std.lang.d.lex. 
 It would be helpful in writing color syntax highlighting filters, pretty 
 printers, repl, doc generators, static analyzers, and even D compilers.
 
 It should:
 
 1. support a range interface for its input, and a range interface for its
output
 2. optionally not generate lexical errors, but just try to recover and continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be maintained in
tandem
 
 It can also serve as the basis for creating a javascript implementation that
can 
 be embedded into web pages for syntax highlighting, and eventually an 
 std.lang.d.parse.
 
 Anyone want to own this?

Do you have Scintilla for D ?

Should be port Scintilla to D.
Oct 22 2010
prev sibling parent dolive <dolive89 sina.com> writes:
dolive 写到:

 Walter Bright 写到:
 
 As we all know, tool support is important for D's success. Making tools easier 
 to build will help with that.
 
 To that end, I think we need a lexer for the standard library -
std.lang.d.lex. 
 It would be helpful in writing color syntax highlighting filters, pretty 
 printers, repl, doc generators, static analyzers, and even D compilers.
 
 It should:
 
 1. support a range interface for its input, and a range interface for its
output
 2. optionally not generate lexical errors, but just try to recover and continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be maintained in
tandem
 
 It can also serve as the basis for creating a javascript implementation that
can 
 be embedded into web pages for syntax highlighting, and eventually an 
 std.lang.d.parse.
 
 Anyone want to own this?

Do you have Scintilla for D ?

Should be port Scintilla to D.
Oct 22 2010
prev sibling next sibling parent reply BLS <windevguy hotmail.de> writes:
Why not creating a DLL/so based Lexer/Parser based on the existing DMD 
front end.? It could be always up to date. Necessary Steps. functional 
wrappers around C++ classes, Implementing the visitor pattern (AST), 
create std.lex and std.parse..

my 2 cents

On 22/10/2010 00:01, Walter Bright wrote:
 As we all know, tool support is important for D's success. Making tools
 easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax highlighting
 filters, pretty printers, repl, doc generators, static analyzers, and
 even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript implementation
 that can be embedded into web pages for syntax highlighting, and
 eventually an std.lang.d.parse.

 Anyone want to own this?

Oct 22 2010
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
BLS wrote:
 Why not creating a DLL/so based Lexer/Parser based on the existing DMD 
 front end.? It could be always up to date. Necessary Steps. functional 
 wrappers around C++ classes, Implementing the visitor pattern (AST), 
 create std.lex and std.parse..

I've done things like that before, they're even more work.
Oct 22 2010
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2010-10-22 17:37, BLS wrote:
 Why not creating a DLL/so based Lexer/Parser based on the existing DMD
 front end.? It could be always up to date. Necessary Steps. functional
 wrappers around C++ classes, Implementing the visitor pattern (AST),
 create std.lex and std.parse..

 my 2 cents

 On 22/10/2010 00:01, Walter Bright wrote:
 As we all know, tool support is important for D's success. Making tools
 easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax highlighting
 filters, pretty printers, repl, doc generators, static analyzers, and
 even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript implementation
 that can be embedded into web pages for syntax highlighting, and
 eventually an std.lang.d.parse.

 Anyone want to own this?


I think it would be better to create a lexer/parser in D and have it in the standard library. Then one could begin the process of porting the DMD frontend using this library. Then hopefully the DMD frontend will be written in D and use this new library, being one code base and will always be up to date. -- /Jacob Carlborg
Oct 22 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:i9spln$lbj$1 digitalmars.com...
 On 2010-10-22 17:37, BLS wrote:
 Why not creating a DLL/so based Lexer/Parser based on the existing DMD
 front end.? It could be always up to date. Necessary Steps. functional
 wrappers around C++ classes, Implementing the visitor pattern (AST),
 create std.lex and std.parse..

 my 2 cents

 On 22/10/2010 00:01, Walter Bright wrote:
 As we all know, tool support is important for D's success. Making tools
 easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax highlighting
 filters, pretty printers, repl, doc generators, static analyzers, and
 even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript implementation
 that can be embedded into web pages for syntax highlighting, and
 eventually an std.lang.d.parse.

 Anyone want to own this?


I think it would be better to create a lexer/parser in D and have it in the standard library. Then one could begin the process of porting the DMD frontend using this library. Then hopefully the DMD frontend will be written in D and use this new library, being one code base and will always be up to date.

*cough* DDMD
Oct 22 2010
parent Jacob Carlborg <doob me.com> writes:
On 2010-10-22 22:42, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:i9spln$lbj$1 digitalmars.com...
 On 2010-10-22 17:37, BLS wrote:
 Why not creating a DLL/so based Lexer/Parser based on the existing DMD
 front end.? It could be always up to date. Necessary Steps. functional
 wrappers around C++ classes, Implementing the visitor pattern (AST),
 create std.lex and std.parse..

 my 2 cents

 On 22/10/2010 00:01, Walter Bright wrote:
 As we all know, tool support is important for D's success. Making tools
 easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax highlighting
 filters, pretty printers, repl, doc generators, static analyzers, and
 even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript implementation
 that can be embedded into web pages for syntax highlighting, and
 eventually an std.lang.d.parse.

 Anyone want to own this?


I think it would be better to create a lexer/parser in D and have it in the standard library. Then one could begin the process of porting the DMD frontend using this library. Then hopefully the DMD frontend will be written in D and use this new library, being one code base and will always be up to date.

*cough* DDMD

I know, I would more than love to see DDMD becoming the official D compiler but if that will happen I would still like that the frontend is based on the lexer/parser library in phobos. -- /Jacob Carlborg
Oct 23 2010
prev sibling next sibling parent reply =?iso-8859-2?B?VG9tZWsgU293afFza2k=?= <just ask.me> writes:
Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 digitalmars.com>  =

napisa=B3(a):

 As we all know, tool support is important for D's success. Making tool=

 easier to build will help with that.

 To that end, I think we need a lexer for the standard library -  =

 std.lang.d.lex. It would be helpful in writing color syntax highlighti=

 filters, pretty printers, repl, doc generators, static analyzers, and =

 even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for =

 its output
 2. optionally not generate lexical errors, but just try to recover and=

 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be  =

 maintained in tandem

 It can also serve as the basis for creating a javascript implementatio=

 that can be embedded into web pages for syntax highlighting, and  =

 eventually an std.lang.d.parse.

 Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORBA, = Thrift, etc, so lexers will have to be written all over to grok interfac= e = files. Perhaps a generic tokenizer which can be parametrized with a = lexical grammar would bring more ROI, I got a hunch D's templates are = strong enough to pull this off without any source code generation ala = JavaCC. The books I read on compilers say tokenization is a solved = problem, so the theory part on what a good abstraction should be is done= . = What you think? -- Tomek
Oct 22 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Tomek Sowi駍ki wrote:
 Interesting idea. Here's another: D will soon need bindings for CORBA, 
 Thrift, etc, so lexers will have to be written all over to grok 
 interface files. Perhaps a generic tokenizer which can be parametrized 
 with a lexical grammar would bring more ROI, I got a hunch D's templates 
 are strong enough to pull this off without any source code generation 
 ala JavaCC. The books I read on compilers say tokenization is a solved 
 problem, so the theory part on what a good abstraction should be is 
 done. What you think?

Lexers are so simple, it is less work to just build them by hand than use lexer generator tools.
Oct 22 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/22/10 14:17 CDT, Walter Bright wrote:
 Tomek Sowi艅ski wrote:
 Interesting idea. Here's another: D will soon need bindings for CORBA,
 Thrift, etc, so lexers will have to be written all over to grok
 interface files. Perhaps a generic tokenizer which can be parametrized
 with a lexical grammar would bring more ROI, I got a hunch D's
 templates are strong enough to pull this off without any source code
 generation ala JavaCC. The books I read on compilers say tokenization
 is a solved problem, so the theory part on what a good abstraction
 should be is done. What you think?

Lexers are so simple, it is less work to just build them by hand than use lexer generator tools.

I wrote a C++ lexer. It wasn't at all easy except if I compared it against the work necessary to build a full compiler. Andrei
Oct 22 2010
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/22/10 14:02 CDT, Tomek Sowi艅ski wrote:
 Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 digitalmars.com>
 napisa艂(a):

 As we all know, tool support is important for D's success. Making
 tools easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax
 highlighting filters, pretty printers, repl, doc generators, static
 analyzers, and even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript
 implementation that can be embedded into web pages for syntax
 highlighting, and eventually an std.lang.d.parse.

 Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORBA, Thrift, etc, so lexers will have to be written all over to grok interface files. Perhaps a generic tokenizer which can be parametrized with a lexical grammar would bring more ROI, I got a hunch D's templates are strong enough to pull this off without any source code generation ala JavaCC. The books I read on compilers say tokenization is a solved problem, so the theory part on what a good abstraction should be is done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer generator. I have in mind the entire implementation of a simple design, but never had the time to execute on it. The tokenizer would work like this: alias Lexer!( "+", "PLUS", "-", "MINUS", "+=", "PLUS_EQ", ... "if", "IF", "else", "ELSE" ... ) DLexer; Such a declaration generates numeric values DLexer.PLUS etc. and generates an efficient code that extracts a stream of tokens from a stream of text. Each token in the token stream has the ID and the text. Comments, strings etc. can be handled in one of several ways but that's a longer discussion. The undertaking is doable but nontrivial. Andrei
Oct 22 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message 
news:i9spsa$ll0$1 digitalmars.com...
 On 10/22/10 14:02 CDT, Tomek Sowinski wrote:
 Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 digitalmars.com>
 napisal(a):

 As we all know, tool support is important for D's success. Making
 tools easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax
 highlighting filters, pretty printers, repl, doc generators, static
 analyzers, and even D compilers.

Interesting idea. Here's another: D will soon need bindings for CORBA, Thrift, etc, so lexers will have to be written all over to grok interface files. Perhaps a generic tokenizer which can be parametrized with a lexical grammar would bring more ROI, I got a hunch D's templates are strong enough to pull this off without any source code generation ala JavaCC. The books I read on compilers say tokenization is a solved problem, so the theory part on what a good abstraction should be is done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer generator.

FWIW, I've been converting my Goldie lexing/parsing library/toolset ( http://www.dsource.org/projects/goldie ) to D2/Phobos, and that should have a release sometime in the next couple months or so. I'm not sure it would really be appropriate for Phobos since it's pretty big, comes with a bunch of tools (one of which is built in C# ATM), it's not range-ified yet, probably doesn't use Phobos coding conventions, and relies on one of my other libraries/tools. But it does do generalized lexing/parsing (LALR) via the GOLD ( http://www.devincook.com/goldparser/ ) grammar file formats, can optionally generate source files for better compile-time checking (for instance, so Token!"<Statemnt>" will generate a compile-time error), has full documentation, and I'm working on a tool/lib that will compile the grammars without having to use the Windows/GUI-based GOLD Parser Builder tool.
Oct 22 2010
prev sibling next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Andrei Alexandrescu Wrote:
 
 I have in mind the entire implementation of a simple design, but never 
 had the time to execute on it. The tokenizer would work like this:
 
 alias Lexer!(
      "+", "PLUS",
      "-", "MINUS",
      "+=", "PLUS_EQ",
      ...
      "if", "IF",
      "else", "ELSE"
      ...
 ) DLexer;
 
 Such a declaration generates numeric values DLexer.PLUS etc. and 
 generates an efficient code that extracts a stream of tokens from a 
 stream of text. Each token in the token stream has the ID and the text.

What about, say, floating-point literals? It seems like the first element of a pair might have to be a regex pattern.
Oct 22 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/22/10 16:28 CDT, Sean Kelly wrote:
 Andrei Alexandrescu Wrote:
 I have in mind the entire implementation of a simple design, but never
 had the time to execute on it. The tokenizer would work like this:

 alias Lexer!(
       "+", "PLUS",
       "-", "MINUS",
       "+=", "PLUS_EQ",
       ...
       "if", "IF",
       "else", "ELSE"
       ...
 ) DLexer;

 Such a declaration generates numeric values DLexer.PLUS etc. and
 generates an efficient code that extracts a stream of tokens from a
 stream of text. Each token in the token stream has the ID and the text.

What about, say, floating-point literals? It seems like the first element of a pair might have to be a regex pattern.

Yah, with regard to such regular patterns (strings, comments, numbers, identifiers) there are at least two possibilities that I see: 1. Go the full route of allowing regexen in the definition. This is very hard because you need to generate an efficient (N|D)FA during compilation. 2. Pragmatically allow "fallthrough" routines, i.e. if nothing in the compile-time table matches, just call onUnrecognizedString(). In conjunction with a few simple specialized functions, that makes it very simple to define arbitrarily complex lexers where the bulk of the work (and the most tedious part) is done by the D compiler. Andrei
Oct 22 2010
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 On 10/22/10 16:28 CDT, Sean Kelly wrote:
 Andrei Alexandrescu Wrote:
 
 I have in mind the entire implementation of a simple design, but
 never
 had the time to execute on it. The tokenizer would work like this:
 
 alias Lexer!(
       "+", "PLUS",
       "-", "MINUS",
       "+=", "PLUS_EQ",
       ...
       "if", "IF",
       "else", "ELSE"
       ...
 ) DLexer;
 
 Such a declaration generates numeric values DLexer.PLUS etc. and
 generates an efficient code that extracts a stream of tokens from a
 stream of text. Each token in the token stream has the ID and the
 text.

What about, say, floating-point literals? It seems like the first element of a pair might have to be a regex pattern.

Yah, with regard to such regular patterns (strings, comments, numbers, identifiers) there are at least two possibilities that I see: 1. Go the full route of allowing regexen in the definition. This is very hard because you need to generate an efficient (N|D)FA during compilation. 2. Pragmatically allow "fallthrough" routines, i.e. if nothing in the compile-time table matches, just call onUnrecognizedString(). In conjunction with a few simple specialized functions, that makes it very simple to define arbitrarily complex lexers where the bulk of the work (and the most tedious part) is done by the D compiler.

For the second, that may push the work of recognizing some lexical elements into the parser. For example, a comment may be defined as /**/, which if there is no lexical definition of a comment means that it parses as four distinct valid tokens, div mul mul div.
Oct 23 2010
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Sean Kelly <sean invisibleduck.org> wrote:
 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 On 10/22/10 16:28 CDT, Sean Kelly wrote:
 Andrei Alexandrescu Wrote:
 
 I have in mind the entire implementation of a simple design, but
 never
 had the time to execute on it. The tokenizer would work like this:
 
 alias Lexer!(
       "+", "PLUS",
       "-", "MINUS",
       "+=", "PLUS_EQ",
       ...
       "if", "IF",
       "else", "ELSE"
       ...
 ) DLexer;
 
 Such a declaration generates numeric values DLexer.PLUS etc. and
 generates an efficient code that extracts a stream of tokens from a
 stream of text. Each token in the token stream has the ID and the
 text.

What about, say, floating-point literals? It seems like the first element of a pair might have to be a regex pattern.

Yah, with regard to such regular patterns (strings, comments, numbers, identifiers) there are at least two possibilities that I see: 1. Go the full route of allowing regexen in the definition. This is very hard because you need to generate an efficient (N|D)FA during compilation. 2. Pragmatically allow "fallthrough" routines, i.e. if nothing in the compile-time table matches, just call onUnrecognizedString(). In conjunction with a few simple specialized functions, that makes it very simple to define arbitrarily complex lexers where the bulk of the work (and the most tedious part) is done by the D compiler.

For the second, that may push the work of recognizing some lexical elements into the parser. For example, a comment may be defined as /**/, which if there is no lexical definition of a comment means that it parses as four distinct valid tokens, div mul mul div.

Or maybe not. A /* could be CommentBegin. I'll have to think on it a bit more.
Oct 23 2010
parent Sean Kelly <sean invisibleduck.org> writes:
Sean Kelly <sean invisibleduck.org> wrote:
 Sean Kelly <sean invisibleduck.org> wrote:
 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 On 10/22/10 16:28 CDT, Sean Kelly wrote:
 Andrei Alexandrescu Wrote:
 
 I have in mind the entire implementation of a simple design, but
 never
 had the time to execute on it. The tokenizer would work like this:
 
 alias Lexer!(
       "+", "PLUS",
       "-", "MINUS",
       "+=", "PLUS_EQ",
       ...
       "if", "IF",
       "else", "ELSE"
       ...
 ) DLexer;
 
 Such a declaration generates numeric values DLexer.PLUS etc. and
 generates an efficient code that extracts a stream of tokens from
 a
 stream of text. Each token in the token stream has the ID and the
 text.

What about, say, floating-point literals? It seems like the first element of a pair might have to be a regex pattern.

Yah, with regard to such regular patterns (strings, comments, numbers, identifiers) there are at least two possibilities that I see: 1. Go the full route of allowing regexen in the definition. This is very hard because you need to generate an efficient (N|D)FA during compilation. 2. Pragmatically allow "fallthrough" routines, i.e. if nothing in the compile-time table matches, just call onUnrecognizedString(). In conjunction with a few simple specialized functions, that makes it very simple to define arbitrarily complex lexers where the bulk of the work (and the most tedious part) is done by the D compiler.

For the second, that may push the work of recognizing some lexical elements into the parser. For example, a comment may be defined as /**/, which if there is no lexical definition of a comment means that it parses as four distinct valid tokens, div mul mul div.

Or maybe not. A /* could be CommentBegin. I'll have to think on it a bit more.

I still think it won't work. The stuff inside the comment would come through as a string of random tokens. Also, the // comment is EOL sensitive, and this info Ian normally communicated to the parser.
Oct 23 2010
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/23/10 11:44 CDT, Sean Kelly wrote:
 Andrei Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:
 On 10/22/10 16:28 CDT, Sean Kelly wrote:
 Andrei Alexandrescu Wrote:
 I have in mind the entire implementation of a simple design, but
 never
 had the time to execute on it. The tokenizer would work like this:

 alias Lexer!(
        "+", "PLUS",
        "-", "MINUS",
        "+=", "PLUS_EQ",
        ...
        "if", "IF",
        "else", "ELSE"
        ...
 ) DLexer;

 Such a declaration generates numeric values DLexer.PLUS etc. and
 generates an efficient code that extracts a stream of tokens from a
 stream of text. Each token in the token stream has the ID and the
 text.

What about, say, floating-point literals? It seems like the first element of a pair might have to be a regex pattern.

Yah, with regard to such regular patterns (strings, comments, numbers, identifiers) there are at least two possibilities that I see: 1. Go the full route of allowing regexen in the definition. This is very hard because you need to generate an efficient (N|D)FA during compilation. 2. Pragmatically allow "fallthrough" routines, i.e. if nothing in the compile-time table matches, just call onUnrecognizedString(). In conjunction with a few simple specialized functions, that makes it very simple to define arbitrarily complex lexers where the bulk of the work (and the most tedious part) is done by the D compiler.

For the second, that may push the work of recognizing some lexical elements into the parser. For example, a comment may be defined as /**/, which if there is no lexical definition of a comment means that it parses as four distinct valid tokens, div mul mul div.

I was thinking comments could be easily caught by simple routines: alias Lexer!( "+", "PLUS", "-", "MINUS", "+=", "PLUS_EQ", ... "/*", q{parseNonNestedComment("*/")}, "/+", q{parseNestedComment("+/")}, "//", q{parseOneLineComment()}, ... "if", "IF", "else", "ELSE", ... ) DLexer; During compilation, such non-tokens are recognized as code by the lexer generator and called appropriately. A comprehensive library of such routines completes a useful library. Andrei
Oct 23 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Andrei Alexandrescu wrote:
 During compilation, such non-tokens are recognized as code by the lexer 
 generator and called appropriately. A comprehensive library of such 
 routines completes a useful library.

I agree, a set of "canned" and heavily optimized lexing functions for common things like identifiers, numbers, comments, etc., would make a lexing library much more practical. Those will work great for inventing DSLs, but for existing languages, the trouble is that the different languages have subtle variations on how they handle them. For example, D's numeric literals allow embedded underscores. Go doesn't overflow on numeric literals. Javascript has some wacky rules to distinguish a comment from a regex. The \uNNNN letters allowed in identifiers in some languages. So while a general purpose lexing library will be very useful, for lexing D code (and Java, Javascript, etc.) a custom one will probably be much more practical.
Oct 23 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/23/10 13:41 CDT, Walter Bright wrote:
 Andrei Alexandrescu wrote:
 During compilation, such non-tokens are recognized as code by the
 lexer generator and called appropriately. A comprehensive library of
 such routines completes a useful library.

I agree, a set of "canned" and heavily optimized lexing functions for common things like identifiers, numbers, comments, etc., would make a lexing library much more practical. Those will work great for inventing DSLs, but for existing languages, the trouble is that the different languages have subtle variations on how they handle them. For example, D's numeric literals allow embedded underscores. Go doesn't overflow on numeric literals. Javascript has some wacky rules to distinguish a comment from a regex. The \uNNNN letters allowed in identifiers in some languages. So while a general purpose lexing library will be very useful, for lexing D code (and Java, Javascript, etc.) a custom one will probably be much more practical.

I don't see these two in tension. "General" does not need entail "unsuitable for subtle particularities". It is more difficult, but not impossible. Again, a general parser that takes care of the 90% of the drudgework and gives enough hooks to do the remaining 10%, all as efficient as hand-written code. Andrei
Oct 23 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Andrei Alexandrescu wrote:
 I don't see these two in tension. "General" does not need entail 
 "unsuitable for subtle particularities". It is more difficult, but not 
 impossible. Again, a general parser that takes care of the 90% of the 
 drudgework and gives enough hooks to do the remaining 10%, all as 
 efficient as hand-written code.

In general I agree with you, but that is a major project to do that and make it general, efficient, and easy to use - and then, one has to make a D lexer out of it. In the meantime, we have a lexer for D that would be straightforward to adapt to be a D library module. The only decisions that have to be made is what the API to it will be.
Oct 23 2010
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 On 10/23/10 11:44 CDT, Sean Kelly wrote:
 Andrei Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:
 On 10/22/10 16:28 CDT, Sean Kelly wrote:
 Andrei Alexandrescu Wrote:
 
 I have in mind the entire implementation of a simple design, but
 never
 had the time to execute on it. The tokenizer would work like this:
 
 alias Lexer!(
        "+", "PLUS",
        "-", "MINUS",
        "+=", "PLUS_EQ",
        ...
        "if", "IF",
        "else", "ELSE"
        ...
 ) DLexer;
 
 Such a declaration generates numeric values DLexer.PLUS etc. and
 generates an efficient code that extracts a stream of tokens from
 a
 stream of text. Each token in the token stream has the ID and the
 text.

What about, say, floating-point literals? It seems like the first element of a pair might have to be a regex pattern.

Yah, with regard to such regular patterns (strings, comments, numbers, identifiers) there are at least two possibilities that I see: 1. Go the full route of allowing regexen in the definition. This is very hard because you need to generate an efficient (N|D)FA during compilation. 2. Pragmatically allow "fallthrough" routines, i.e. if nothing in the compile-time table matches, just call onUnrecognizedString(). In conjunction with a few simple specialized functions, that makes it very simple to define arbitrarily complex lexers where the bulk of the work (and the most tedious part) is done by the D compiler.

For the second, that may push the work of recognizing some lexical elements into the parser. For example, a comment may be defined as /**/, which if there is no lexical definition of a comment means that it parses as four distinct valid tokens, div mul mul div.

I was thinking comments could be easily caught by simple routines: alias Lexer!( "+", "PLUS", "-", "MINUS", "+=", "PLUS_EQ", ... "/*", q{parseNonNestedComment("*/")}, "/+", q{parseNestedComment("+/")}, "//", q{parseOneLineComment()}, ... "if", "IF", "else", "ELSE", ... ) DLexer; During compilation, such non-tokens are recognized as code by the lexer generator and called appropriately. A comprehensive library of such routines completes a useful library.

Ah so the only issue is identifying the first set for a lexical element, is essence. That works.
Oct 23 2010
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message 
news:i9v8vq$2gvh$1 digitalmars.com...
 On 10/23/10 11:44 CDT, Sean Kelly wrote:
 Andrei Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:
 On 10/22/10 16:28 CDT, Sean Kelly wrote:
 Andrei Alexandrescu Wrote:
 I have in mind the entire implementation of a simple design, but
 never
 had the time to execute on it. The tokenizer would work like this:

 alias Lexer!(
        "+", "PLUS",
        "-", "MINUS",
        "+=", "PLUS_EQ",
        ...
        "if", "IF",
        "else", "ELSE"
        ...
 ) DLexer;

 Such a declaration generates numeric values DLexer.PLUS etc. and
 generates an efficient code that extracts a stream of tokens from a
 stream of text. Each token in the token stream has the ID and the
 text.

What about, say, floating-point literals? It seems like the first element of a pair might have to be a regex pattern.

Yah, with regard to such regular patterns (strings, comments, numbers, identifiers) there are at least two possibilities that I see: 1. Go the full route of allowing regexen in the definition. This is very hard because you need to generate an efficient (N|D)FA during compilation. 2. Pragmatically allow "fallthrough" routines, i.e. if nothing in the compile-time table matches, just call onUnrecognizedString(). In conjunction with a few simple specialized functions, that makes it very simple to define arbitrarily complex lexers where the bulk of the work (and the most tedious part) is done by the D compiler.

For the second, that may push the work of recognizing some lexical elements into the parser. For example, a comment may be defined as /**/, which if there is no lexical definition of a comment means that it parses as four distinct valid tokens, div mul mul div.

I was thinking comments could be easily caught by simple routines: alias Lexer!( "+", "PLUS", "-", "MINUS", "+=", "PLUS_EQ", ... "/*", q{parseNonNestedComment("*/")}, "/+", q{parseNestedComment("+/")}, "//", q{parseOneLineComment()}, ... "if", "IF", "else", "ELSE", ... ) DLexer; During compilation, such non-tokens are recognized as code by the lexer generator and called appropriately. A comprehensive library of such routines completes a useful library.

What's wrong with regexes? That's pretty typical for lexers.
Oct 23 2010
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/23/10 16:39 CDT, Nick Sabalausky wrote:
 "Andrei Alexandrescu"<SeeWebsiteForEmail erdani.org>  wrote in message
 news:i9v8vq$2gvh$1 digitalmars.com...
 What's wrong with regexes? That's pretty typical for lexers.

I mentioned that using regexes is possible but would make it much more difficult to generate good quality lexers. Besides, regexen are IMHO quite awkward at expressing certain things that can be easily parsed by hand, such as comments or recursive comments. Andrei
Oct 23 2010
parent "Nick Sabalausky" <a a.a> writes:
"Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message 
news:i9vlep$8ao$1 digitalmars.com...
 On 10/23/10 16:39 CDT, Nick Sabalausky wrote:
 "Andrei Alexandrescu"<SeeWebsiteForEmail erdani.org>  wrote in message
 news:i9v8vq$2gvh$1 digitalmars.com...
 What's wrong with regexes? That's pretty typical for lexers.

I mentioned that using regexes is possible but would make it much more difficult to generate good quality lexers.

I see. Maybe a lexer 2.0 thing.
 Besides, regexen are IMHO quite awkward at expressing certain things that 
 can be easily parsed by hand, such as comments

//[^\n]*\n /\*(.|\*[^/])*\*/ Pretty simple as far as regexes go, and I'm far from a regex expert. Plus there's nothing stopping the use of a vastly improved regex syntax like GOLD uses ( http://www.devincook.com/goldparser/doc/grammars/define-terminals.htm ). In that, the two regexes above would look like: {LineCommentChar} = {Printable} - {LF} LineComment = '//' {LineCommentChar}* {LF} {BlockCommentChar} = {Printable} - [*] {BlockCommentCharNoSlash} = {BlockCommentChar} - [/] BlockComment = '/*' ({BlockCommentChar} | '*' {BlockCommentCharNoSlash})* '*/' And further syntactical improvement is easy to imagine, such as in-line character set creation.
 or recursive comments.

Granted, although I think there is precident for regex engines that can handle matched nested pairs just fine.
Oct 23 2010
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 What's wrong with regexes?

They don't handle recursion.
Oct 23 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:i9vn3l$bd1$2 digitalmars.com...
 Nick Sabalausky wrote:
 What's wrong with regexes?

They don't handle recursion.

Neither do plain-old strings. But regexes will get you farther than plain strings before needing to resort to customized lexing. But I'm a big data-driven fan anyway. If you're not than I can see why it wouldn't seem as appealing as it does to me. In any case, if I have a chance I might see about adapting my Goldie ( www.dsource.org/projects/goldie ) library to more Phobos-friendly requirements. It's already a fully-usable lexer/parser (and the lexer/parser parts can be used independantly), with a complete grammar description language and I already have misc related tools written. And it's mostly working on D2 already (just need the next DMD because it has a fix for a bug that's a breaker for one of the tools). So if I can get it into a state more suitable for Phobos then that might end up putting things ahead of where they would be if someone just started from scratch. The initial versions might not be completely Phobos-ified, but it could definitely get there (especially if I had some guidance from people with more Phobos2 experience than me). Would Walter & co be interested in this? If not, I won't bother, but if so, then I may give it a shot.
Oct 23 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:ia01q3$1i1a$1 digitalmars.com...
 "Walter Bright" <newshound2 digitalmars.com> wrote in message 
 news:i9vn3l$bd1$2 digitalmars.com...
 Nick Sabalausky wrote:
 What's wrong with regexes?

They don't handle recursion.

Neither do plain-old strings. But regexes will get you farther than plain strings before needing to resort to customized lexing. But I'm a big data-driven fan anyway. If you're not than I can see why it wouldn't seem as appealing as it does to me. In any case, if I have a chance I might see about adapting my Goldie ( www.dsource.org/projects/goldie ) library to more Phobos-friendly requirements. It's already a fully-usable lexer/parser (and the lexer/parser parts can be used independantly), with a complete grammar description language and I already have misc related tools written. And it's mostly working on D2 already (just need the next DMD because it has a fix for a bug that's a breaker for one of the tools). So if I can get it into a state more suitable for Phobos then that might end up putting things ahead of where they would be if someone just started from scratch. The initial versions might not be completely Phobos-ified, but it could definitely get there (especially if I had some guidance from people with more Phobos2 experience than me). Would Walter & co be interested in this? If not, I won't bother, but if so, then I may give it a shot.

And FWIW, I was already thnking about making some improvements to Goldie's API enyway.
Oct 23 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:ia01sk$1i7s$1 digitalmars.com...
 "Nick Sabalausky" <a a.a> wrote in message 
 news:ia01q3$1i1a$1 digitalmars.com...
 "Walter Bright" <newshound2 digitalmars.com> wrote in message 
 news:i9vn3l$bd1$2 digitalmars.com...
 Nick Sabalausky wrote:
 What's wrong with regexes?

They don't handle recursion.

Neither do plain-old strings. But regexes will get you farther than plain strings before needing to resort to customized lexing. But I'm a big data-driven fan anyway. If you're not than I can see why it wouldn't seem as appealing as it does to me. In any case, if I have a chance I might see about adapting my Goldie ( www.dsource.org/projects/goldie ) library to more Phobos-friendly requirements. It's already a fully-usable lexer/parser (and the lexer/parser parts can be used independantly), with a complete grammar description language and I already have misc related tools written. And it's mostly working on D2 already (just need the next DMD because it has a fix for a bug that's a breaker for one of the tools). So if I can get it into a state more suitable for Phobos then that might end up putting things ahead of where they would be if someone just started from scratch. The initial versions might not be completely Phobos-ified, but it could definitely get there (especially if I had some guidance from people with more Phobos2 experience than me). Would Walter & co be interested in this? If not, I won't bother, but if so, then I may give it a shot.

And FWIW, I was already thnking about making some improvements to Goldie's API enyway.

But that's all if you want generalized lexing or parsing though. If you just want "lexing D code"/"parsing D code", then IMO anything other than adapting parts of DDMD would be the wrong way to go.
Oct 23 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Nick Sabalausky:

 But that's all if you want generalized lexing or parsing though. If you just 
 want "lexing D code"/"parsing D code", then IMO anything other than adapting 
 parts of DDMD would be the wrong way to go.

Is the DDMD licence compatible with the Phobos one? Is the DDMD author(s) willing? Bye, bearophile
Oct 23 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"bearophile" <bearophileHUGS lycos.com> wrote in message 
news:ia0410$1lju$1 digitalmars.com...
 Nick Sabalausky:

 But that's all if you want generalized lexing or parsing though. If you 
 just
 want "lexing D code"/"parsing D code", then IMO anything other than 
 adapting
 parts of DDMD would be the wrong way to go.

Is the DDMD licence compatible with the Phobos one? Is the DDMD author(s) willing?

I'd certainly hope so. If it isn't, then that would probably mean DMD's FE license is incompatible with Phobos. Which would be rather...weird. In any case, I asked that and a couple other Q's here, but haven't gotten an answer yet: http://www.dsource.org/forums/viewtopic.php?t=5627
Oct 23 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Denis Koroskin" <2korden gmail.com> wrote in message 
news:op.vk2na9bpo7cclz korden-pc...
 On Sun, 24 Oct 2010 06:55:22 +0400, Nick Sabalausky <a a.a> wrote:

 "bearophile" <bearophileHUGS lycos.com> wrote in message
 news:ia0410$1lju$1 digitalmars.com...
 Nick Sabalausky:

 But that's all if you want generalized lexing or parsing though. If you
 just
 want "lexing D code"/"parsing D code", then IMO anything other than
 adapting
 parts of DDMD would be the wrong way to go.

Is the DDMD licence compatible with the Phobos one? Is the DDMD author(s) willing?

I'd certainly hope so. If it isn't, then that would probably mean DMD's FE license is incompatible with Phobos. Which would be rather...weird. In any case, I asked that and a couple other Q's here, but haven't gotten an answer yet: http://www.dsource.org/forums/viewtopic.php?t=5627

Sorry, I wasn't checking the forum. IIRC DMD license is GPL so DDMD must be GPL too but I'm all for relicensing it as Boost.

According to a random file I picked out of trunk, it's dual-licensed with GPL (not sure which version) and Artistic (also not sure which version) http://www.dsource.org/projects/dmd/browser/trunk/src/access.c
Oct 24 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:ia0v9p$11p$1 digitalmars.com...
 "Denis Koroskin" <2korden gmail.com> wrote in message 
 news:op.vk2na9bpo7cclz korden-pc...
 On Sun, 24 Oct 2010 06:55:22 +0400, Nick Sabalausky <a a.a> wrote:

 "bearophile" <bearophileHUGS lycos.com> wrote in message
 news:ia0410$1lju$1 digitalmars.com...
 Nick Sabalausky:

 But that's all if you want generalized lexing or parsing though. If 
 you
 just
 want "lexing D code"/"parsing D code", then IMO anything other than
 adapting
 parts of DDMD would be the wrong way to go.

Is the DDMD licence compatible with the Phobos one? Is the DDMD author(s) willing?

I'd certainly hope so. If it isn't, then that would probably mean DMD's FE license is incompatible with Phobos. Which would be rather...weird. In any case, I asked that and a couple other Q's here, but haven't gotten an answer yet: http://www.dsource.org/forums/viewtopic.php?t=5627

Sorry, I wasn't checking the forum. IIRC DMD license is GPL so DDMD must be GPL too but I'm all for relicensing it as Boost.

According to a random file I picked out of trunk, it's dual-licensed with GPL (not sure which version) and Artistic (also not sure which version) http://www.dsource.org/projects/dmd/browser/trunk/src/access.c

That does surprise me though, since I'm pretty sure Phobos is Boost License. Anyone know why the difference?
Oct 24 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 That does surprise me though, since I'm pretty sure Phobos is Boost License. 
 Anyone know why the difference?

Phobos is Boost licensed to enable maximum usage for any purpose. The dmd front end is GPL licensed in order to ensure it stays open source and to discourage closed source forks.
Oct 24 2010
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-10-24 04:55, Nick Sabalausky wrote:
 "bearophile"<bearophileHUGS lycos.com>  wrote in message
 news:ia0410$1lju$1 digitalmars.com...
 Nick Sabalausky:

 But that's all if you want generalized lexing or parsing though. If you
 just
 want "lexing D code"/"parsing D code", then IMO anything other than
 adapting
 parts of DDMD would be the wrong way to go.

Is the DDMD licence compatible with the Phobos one? Is the DDMD author(s) willing?

I'd certainly hope so. If it isn't, then that would probably mean DMD's FE license is incompatible with Phobos. Which would be rather...weird. In any case, I asked that and a couple other Q's here, but haven't gotten an answer yet: http://www.dsource.org/forums/viewtopic.php?t=5627

As Walter wrote in the first post of this thread: "generally follow along with the C++ one so that they can be maintained in tandem" and in another post: "Since the license is mine, I can change the D version to the Boost license, no problem." http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=120221 -- /Jacob Carlborg
Oct 24 2010
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 Would Walter & co be interested in this? If not, I won't bother, 
 but if so, then I may give it a shot.

The problem is I never have used parser/lexer generators, so I am not really in a good position to review it.
Oct 23 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:ia0cfv$22kp$1 digitalmars.com...
 Nick Sabalausky wrote:
 Would Walter & co be interested in this? If not, I won't bother, but if 
 so, then I may give it a shot.

The problem is I never have used parser/lexer generators, so I am not really in a good position to review it.

Understandable. FWIW though, Goldie isn't really lexer/parse generator per se. Traditional lexer/parser generators like lex/yacc or ANTLR will actually generate the source code for a lexer or parser. Goldie just has a single lexer and parser, both already pre-written. They're just completely data-driven: Compared to the generators, Goldie's lexer is more like a general regex engine that simultaneously matches against multiple pre-compiled "regexes". By pre-compiled, I mean turned into a DFA - which is currently done by a separate non-source-available tool I didn't write, but I'm going to be writing my own version soon. By "regexes", I mean they're functionally regexes, but they're written in a much easier-to-read syntax than the typical PCRE. Goldie's parser is really just a rather typical (from what I understand) LALR parser. I don't know how much you know about LALR's, but the parser itself is naturally grammar-independent (at least as described in CS texts). Using an LALR involves converting the grammar completely into a table of states and lookaheads (single-token lookahead; unlike LL, any more than that is never really needed), and then the actual parser is directed entirely by that table (much like how regexes are converted to data, ie DFA, and then processed generically), so it's completely grammar-independent. And of, course, the actual lexer and parser can be optimized/rewritten/whatever with minimal impact on everything else. If anyone's interested, further details are here(1): http://www.devincook.com/goldparser/ Goldie does have optional code-generation capabilities, but it's entirely for the sake of providing a better statically-checked API tailored to your grammar (ex: to use D's type system to ensure at compile-time, instead of run-time, that token names are valid and that BNF rules you reference actually exist). It doesn't actually affect the lexer/parser in any non-trivial way. (1): By that site's terminology, Goldie would technically be a "GOLD Engine", plus some additional tools. But, my current work on Goldie will cut that actual "GOLD Parser Builder" program completely out-of-the-loop (but it will still maintain compatibility with it for anyone who wants to use it).
Oct 23 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 If anyone's interested, further details are here(1):
 http://www.devincook.com/goldparser/

It looks nice, but in clicking around on FAQ, documentation, getting started, etc., I can't find any example code.
Oct 24 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:ia0pce$2pbk$1 digitalmars.com...
 Nick Sabalausky wrote:
 If anyone's interested, further details are here(1):
 http://www.devincook.com/goldparser/

It looks nice, but in clicking around on FAQ, documentation, getting started, etc., I can't find any example code.

Well, that's because that program (GOLD Parser Builder) is just a tool that takes in a grammar description file and spits out the lexer/parser DFA/LALR tables. Then you use any GOLD-compatible engine in any langauge (such as Goldie) to load the DFA/LALR tables and use them to lex/parse. (But again, I'm currently working on code that will do that without having to use GOLD Parser Builder.) Here's some specific links for Goldie, and keep in mind that 1. I already have it pretty much converted to D2/Phobos in trunk (it used to be D1/Tango), 2. The API is not final and definitely open to suggestions (I have a few ideas already), 3. Any suggestions for improvements to the documentation, are, of course, welcome too, 4. Like I've said, in the next official release, using "GOLD Parser Builder" won't actually be required. Main Goldie Project page: http://www.dsource.org/projects/goldie Documentation for latest official release: http://www.semitwist.com/goldiedocs/current/Docs/ Samples directory in trunk: http://www.dsource.org/projects/goldie/browser/trunk/src/samples Slightly old documentation for the samples: http://www.semitwist.com/goldiedocs/current/Docs/SampleApps/ There's two "calculator" samples. They're the same, but correspond to the two different styles Goldie supports. One, "dynamic", doesn't involve any source-code-generation step and can load and use any arbitrary grammar at runtime (neat usages of this are shown in the "ParseAnything" sample and in the "Parse" tool http://www.semitwist.com/goldiedocs/current/Docs/Tools/Parse/ ). The other, "static", does involve generating some source-code (via a comand-line tool), but that gives you an API that's statically-checked against the grammar. The differences and pros/cons between these two styles are explained here (let me know if it's unclear): http://www.semitwist.com/goldiedocs/current/Docs/APIOver/StatVsDyn/
Oct 24 2010
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
It looks like a solid engine, and a nice tool. Does it belong as part of
Phobos? 
I don't know. What do other D users think?
Oct 24 2010
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
     http://www.semitwist.com/goldiedocs/current/Docs/APIOver/StatVsDyn/

One question I have is how does it compare with Spirit? That would be its main counterpart in the C++ space.
Oct 24 2010
next sibling parent div0 <div0 sourceforge.net> writes:
On 24/10/2010 18:19, Walter Bright wrote:
 Nick Sabalausky wrote:
 http://www.semitwist.com/goldiedocs/current/Docs/APIOver/StatVsDyn/

One question I have is how does it compare with Spirit? That would be its main counterpart in the C++ space.

Spirit is a LL parser, so it's not really suitable for human edited input as doing exact error reporting is tricky. -- My enormous talent is exceeded only by my outrageous laziness. http://www.ssTk.co.uk
Oct 24 2010
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:ia1ps7$1fq5$2 digitalmars.com...
 Nick Sabalausky wrote:
     http://www.semitwist.com/goldiedocs/current/Docs/APIOver/StatVsDyn/

One question I have is how does it compare with Spirit? That would be its main counterpart in the C++ space.

Can't say I'm really familiar with Spirit. From a brief lookover, these are my impresions of the differences: Spirit: Grammar is embedded into your source code as actual C++ code. Goldie: Grammar is defined in a domain-specfic language. But either one could probably have a wrapper to work the other way. Spirit: Uses (abuses?) operator overloading (Although, apperently SpiritD doesn't inherit Spirit's operator overloading: http://www.sstk.co.uk/spiritd.php ) Goldie: Operator overloading isn't really applicable, because of using a DSL. As they stand, Spirit seems like it could be pretty handly for simple, quick little DSLs, ex, things for which Goldie might seem like overkill. But Goldie's interface could probably be improved to compete pretty well in those cases. OTOH, Goldie's approach (being based on GOLD) has a deliberate separation between grammar and parsing, which has it's own benefits; for instance, grammar definitions can be re-used for any purpose.
Oct 24 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 Can't say I'm really familiar with Spirit. From a brief lookover, these are 
 my impresions of the differences:
 
 Spirit: Grammar is embedded into your source code as actual C++ code.
 Goldie: Grammar is defined in a domain-specfic language.
 But either one could probably have a wrapper to work the other way.
 
 Spirit: Uses (abuses?) operator overloading (Although, apperently SpiritD 
 doesn't inherit Spirit's operator overloading: 
 http://www.sstk.co.uk/spiritd.php )
 Goldie: Operator overloading isn't really applicable, because of using a 
 DSL.
 
 As they stand, Spirit seems like it could be pretty handly for simple, quick 
 little DSLs, ex, things for which Goldie might seem like overkill. But 
 Goldie's interface could probably be improved to compete pretty well in 
 those cases. OTOH, Goldie's approach (being based on GOLD) has a deliberate 
 separation between grammar and parsing, which has it's own benefits; for 
 instance, grammar definitions can be re-used for any purpose.
 
 

Does Goldie have (like Spirit) a set of canned routines for things like numeric literals? Can the D version of Goldie be turned into one file?
Oct 24 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:ia2duj$2j7e$1 digitalmars.com...
 Nick Sabalausky wrote:
 Can't say I'm really familiar with Spirit. From a brief lookover, these 
 are my impresions of the differences:

 Spirit: Grammar is embedded into your source code as actual C++ code.
 Goldie: Grammar is defined in a domain-specfic language.
 But either one could probably have a wrapper to work the other way.

 Spirit: Uses (abuses?) operator overloading (Although, apperently SpiritD 
 doesn't inherit Spirit's operator overloading: 
 http://www.sstk.co.uk/spiritd.php )
 Goldie: Operator overloading isn't really applicable, because of using a 
 DSL.

 As they stand, Spirit seems like it could be pretty handly for simple, 
 quick little DSLs, ex, things for which Goldie might seem like overkill. 
 But Goldie's interface could probably be improved to compete pretty well 
 in those cases. OTOH, Goldie's approach (being based on GOLD) has a 
 deliberate separation between grammar and parsing, which has it's own 
 benefits; for instance, grammar definitions can be re-used for any 
 purpose.

Does Goldie have (like Spirit) a set of canned routines for things like numeric literals?

No, but such things can easily be provided in the docs for simple copy-paste. For instance: DecimalLiteral = {Number} ({Number} | '_')* HexLiteral = '0' [xX] ({Number} | [ABCDEFabcdef_])+ Identifier = ('_' | {Letter}) ('_' | {AlphaNumeric})* {StringChar} = {Printable} - ["] StringLiteral = '"' ({StringChar} | '\' {Printable})* '"' All one would need to do to use those is copy-paste them into their grammar definition. Some sort of import mechanism could certainly be added though, to allow for selective import of pre-defined things like that. There are many pre-defined character sets though (and others can be manually-created, of course): http://www.devincook.com/goldparser/doc/grammars/character-sets.htm
 Can the D version of Goldie be turned into one file?

Assuming just the library and not the included tools (many of which could be provided as part of the library, though), and not counting files generated for the static-style, then yes, but it would probably be a bit long.
Oct 24 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message 
 Does Goldie have (like Spirit) a set of canned routines for things like 
 numeric literals?

No, but such things can easily be provided in the docs for simple copy-paste. For instance: DecimalLiteral = {Number} ({Number} | '_')* HexLiteral = '0' [xX] ({Number} | [ABCDEFabcdef_])+ Identifier = ('_' | {Letter}) ('_' | {AlphaNumeric})* {StringChar} = {Printable} - ["] StringLiteral = '"' ({StringChar} | '\' {Printable})* '"' All one would need to do to use those is copy-paste them into their grammar definition. Some sort of import mechanism could certainly be added though, to allow for selective import of pre-defined things like that.

In the regexp code, I provided special regexes for email addresses and URLs. Those are hard to get right, so it's a large convenience to provide them. Also, many literals can be fairly complex, and evaluating them can produce errors (such as integer overflow in the numeric literals). Having canned ones makes it much quicker for a user to get going. I'm guessing that a numeric literal is returned as a string. Is this string allocated on the heap? If so, it's a performance problem. Storage allocation costs figure large when trying to lex millions of lines.
 There are many pre-defined character sets though (and others can be 
 manually-created, of course): 
 http://www.devincook.com/goldparser/doc/grammars/character-sets.htm
 
 Can the D version of Goldie be turned into one file?

Assuming just the library and not the included tools (many of which could be provided as part of the library, though), and not counting files generated for the static-style, then yes, but it would probably be a bit long.

Long files aren't a problem. That's why we have .di files! I worry more about clutter.
Oct 24 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:ia34up$ldb$1 digitalmars.com...
 In the regexp code, I provided special regexes for email addresses and 
 URLs. Those are hard to get right, so it's a large convenience to provide 
 them.

 Also, many literals can be fairly complex, and evaluating them can produce 
 errors (such as integer overflow in the numeric literals). Having canned 
 ones makes it much quicker for a user to get going.

I'm not sure what exectly you're suggesting in these two paragraphs? (Or just commenting?)
 I'm guessing that a numeric literal is returned as a string. Is this 
 string allocated on the heap? If so, it's a performance problem. Storage 
 allocation costs figure large when trying to lex millions of lines.

Good point. I've just checked and there is allocation going on for each terminal lexed. But thanks to D's awesomeness, I can easily fix that to just use a slice of the original source string. I'll do that...
 Long files aren't a problem. That's why we have .di files! I worry more 
 about clutter.

I really find long files to be a pain to read and edit. It would be nice if #5005 ( http://d.puremagic.com/issues/show_bug.cgi?id=5005 ) could get done. Then, modules with a lot of code could be broken down as appropriate for their maintainers without having to bother the users with the "module blah.all" workaround (which Goldie currently uses, but I realize isn't normal Phobos style). AIUI, .di files don't really solve that. There is one other other minor related issue, though. One of my big principles for Goldie is flexibility. So in addition to the basic API that most people would use, I like to expose lower-level APIs for people who might want to sidestep certain parts of Goldie, or provide other less-typical but potentially useful things. But such things shouldn't be automatically imported for typical users, so that sort of stuff would be best left to a separate-but-related module. Maybe it's just too late over here for me, but can you be more specific on "clutter"? Do you mean like API clutter?
Oct 24 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message 
 news:ia34up$ldb$1 digitalmars.com...
 In the regexp code, I provided special regexes for email addresses and 
 URLs. Those are hard to get right, so it's a large convenience to provide 
 them.

 Also, many literals can be fairly complex, and evaluating them can produce 
 errors (such as integer overflow in the numeric literals). Having canned 
 ones makes it much quicker for a user to get going.

I'm not sure what exectly you're suggesting in these two paragraphs? (Or just commenting?)

Does Goldie's lexer not convert numeric literals to integer values?
 I'm guessing that a numeric literal is returned as a string. Is this 
 string allocated on the heap? If so, it's a performance problem. Storage 
 allocation costs figure large when trying to lex millions of lines.

Good point. I've just checked and there is allocation going on for each terminal lexed. But thanks to D's awesomeness, I can easily fix that to just use a slice of the original source string. I'll do that...

Are all tokens returned as strings?
 Long files aren't a problem. That's why we have .di files! I worry more 
 about clutter.

I really find long files to be a pain to read and edit. It would be nice if #5005 ( http://d.puremagic.com/issues/show_bug.cgi?id=5005 ) could get done. Then, modules with a lot of code could be broken down as appropriate for their maintainers without having to bother the users with the "module blah.all" workaround (which Goldie currently uses, but I realize isn't normal Phobos style). AIUI, .di files don't really solve that. There is one other other minor related issue, though. One of my big principles for Goldie is flexibility. So in addition to the basic API that most people would use, I like to expose lower-level APIs for people who might want to sidestep certain parts of Goldie, or provide other less-typical but potentially useful things. But such things shouldn't be automatically imported for typical users, so that sort of stuff would be best left to a separate-but-related module.

If I may suggest, leave the low level stuff out of the api until demand for it justifies it. It's hard to predict just what will be useful, so I suggest conservatism rather than kitchen sink. It can always be added later, but it's really hard to remove.
 Maybe it's just too late over here for me, but can you be more specific on 
 "clutter"? Do you mean like API clutter?

That too, but I meant a clutter of files. Long files aren't a problem with D.
Oct 25 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:ia3c3r$14k8$1 digitalmars.com...
 Does Goldie's lexer not convert numeric literals to integer values?

 Are all tokens returned as strings?

Goldie's lexer (and parser) are based on the GOLD system ( http://www.devincook.com/goldparser/ ) which is deliberately independent of both grammar and implementation language. As such, it doesn't know anything about what the specific terminals actually represent (There are 4 exceptions though: Comment tokens, Whitespace tokens, an "Error" token (ie, for lex errors), and the EOF token.) So the lexed data is always represented as a string. Although, the lexer actually returns an array of "class Token" ( http://www.semitwist.com/goldiedocs/current/Docs/APIRef/Token/#Token ). To get the original data that got lexed or parsed into that token, you call "toString()". (BTW, there are currently different "modes" of "toString()" for non-terminals, but I'm considering just ripping them all out and replacing them with a single "return a slice from the start of the first terminal to the end of the last terminal" - unless you think it would be useful to get a representation of the non-terminal's original data sans comments/whitespace, or with comments/whitespace converted to a single space.) I'm not sure that calling "to!whatever(token.toString())" is really all that much of a problem for user code.
 If I may suggest, leave the low level stuff out of the api until demand 
 for it justifies it. It's hard to predict just what will be useful, so I 
 suggest conservatism rather than kitchen sink. It can always be added 
 later, but it's really hard to remove.

That may be a good idea.
 That too, but I meant a clutter of files. Long files aren't a problem with 
 D.

Well, again, it may not be a problem with DMD, but I really think reading/editing a long file is a pain regardless of language. Maybe we just have different ideas of "long file"? To put it into numbers: At the moment, Goldie's library (not counting tools and the optional generated "static-mode" files) is about 3200 lines, including comment/blank lines. That size would be pretty unwieldy to maintain as a single source file, particularly since Goldie has a natural internal organization. Personally, I'd much rather have a clutter of source files than a cluttered source file. (But of course, I don't go to Java extremes and put *every* tiny little thing in a separate file.) As long as the complexity of having multiple files isn't passed along to user code (hence the frequent "module foo.all" idiom), then I can't say I really see a problem.
Oct 25 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message 
 news:ia3c3r$14k8$1 digitalmars.com...
 Does Goldie's lexer not convert numeric literals to integer values?

 Are all tokens returned as strings?

Goldie's lexer (and parser) are based on the GOLD system ( http://www.devincook.com/goldparser/ ) which is deliberately independent of both grammar and implementation language. As such, it doesn't know anything about what the specific terminals actually represent (There are 4 exceptions though: Comment tokens, Whitespace tokens, an "Error" token (ie, for lex errors), and the EOF token.) So the lexed data is always represented as a string. Although, the lexer actually returns an array of "class Token" ( http://www.semitwist.com/goldiedocs/current/Docs/APIRef/Token/#Token ). To get the original data that got lexed or parsed into that token, you call "toString()". (BTW, there are currently different "modes" of "toString()" for non-terminals, but I'm considering just ripping them all out and replacing them with a single "return a slice from the start of the first terminal to the end of the last terminal" - unless you think it would be useful to get a representation of the non-terminal's original data sans comments/whitespace, or with comments/whitespace converted to a single space.) I'm not sure that calling "to!whatever(token.toString())" is really all that much of a problem for user code.

Consider a string literal, say "abc\"def". With Goldie's method, I infer this string has to be scanned twice. Once to find its limits, and the second to convert it to the actual string. The latter is user code and will have to replicate whatever Goldie did.
 If I may suggest, leave the low level stuff out of the api until demand 
 for it justifies it. It's hard to predict just what will be useful, so I 
 suggest conservatism rather than kitchen sink. It can always be added 
 later, but it's really hard to remove.

That may be a good idea.

What Goldie will be compared against is Spirit. Spirit is a reasonably successful add-on to C++. Goldie doesn't have to do things the same way as Spirit (expression templates - ugh), but it should be as easy to use and at least as powerful.
 That too, but I meant a clutter of files. Long files aren't a problem with 
 D.

Well, again, it may not be a problem with DMD, but I really think reading/editing a long file is a pain regardless of language. Maybe we just have different ideas of "long file"? To put it into numbers: At the moment, Goldie's library (not counting tools and the optional generated "static-mode" files) is about 3200 lines, including comment/blank lines. That size would be pretty unwieldy to maintain as a single source file, particularly since Goldie has a natural internal organization.

Actually, I think 3200 lines is of moderate, not large, size :-)
 Personally, I'd much rather have a clutter of source files than a cluttered 
 source file. (But of course, I don't go to Java extremes and put *every* 
 tiny little thing in a separate file.) As long as the complexity of having 
 multiple files isn't passed along to user code (hence the frequent "module 
 foo.all" idiom), then I can't say I really see a problem.

I tend to just not like having to constantly grep to see which file XXX is in.
Oct 25 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:ia59si$1r0j$1 digitalmars.com...
 Consider a string literal, say "abc\"def". With Goldie's method, I infer 
 this string has to be scanned twice. Once to find its limits, and the 
 second to convert it to the actual string.

Yea, that is true. With that string in the input, the value given to the user code will be: assert(tokenObtainedFromGoldie.toString() == q{"abc\"def"}); That's a consequence of the grammar being separated from lexing/parsing implementation. You're right that that does seem less than ideal. Although I'm not sure how to remedy that without loosing the independence between grammar and lex/parse implementation that is the main point of the GOLD-based style. But there's something I don't quite understand about the approach you're suggesting: You seem to be suggesting that a terminal be progressively converted into its final form *as* it's still in the process of being recognized by the DFA. Which means, you don't know *what* you're supposed to be converting it into *while* you're converting it. Which means, you have to be speculatively converting it into all types of tokens that the current DFA state could possibly be on its way towards accepting (also, the DFA would need to contain a record of possible terminals for each DFA state). And then the result is thrown away if it turns out to be a different terminal. Is this correct? If so, is there generally enough lexical difference between the terminals that need such treatment to compensate for the extra processing needed in situations that are closer to worst-case (that is, in comparison to Goldie's current approach)? If all of that is so, then what would be your thoughts on this approach?: Suppose Goldie had a way to associate an optional "simultaneous/lockstep conversion" to a type of terminal. For instance: myLanguage.associateConversion("StringLiteral", new StringLiteralConverter()); Then, 'StringLiteralConverter' would be something that could be either user-provided or offered by Goldie (both ways would be supported). It would be some sort of class or something that had three basic functions: class StringLiteralConverter : ITerminalConverter { void process(dchar c) {...} // Or maybe this to make it possible to minimize allocations // in certain circumstances by utilizing slices: void process(dchar c, size_t indexIntoSource, string fullOrignalSource) {...} Variant emit() {...} void clear() {...} } Each state in the lexer's DFA would know which terminals it could possibly be processing. And for each of those terminals that has an associated converter, the lexer will call 'process()'. If a terminal is accepted, 'emit' is called to get the final result (and maybe do any needed finalization first), and then 'clear' is called on all converters that had been used. This feature would preclude the use of the actual "GOLD Parser Builder" program, but since I'm writing a tool to handle that functionality anyway, I'm not too concerned about that. Do you think that would work? Would its benefits be killed by the overhead introduced? If so, could those overheads be sufficiently reduced without scrapping the general idea?
 What Goldie will be compared against is Spirit. Spirit is a reasonably 
 successful add-on to C++. Goldie doesn't have to do things the same way as 
 Spirit (expression templates - ugh), but it should be as easy to use and 
 at least as powerful.

Understood.
 Personally, I'd much rather have a clutter of source files than a 
 cluttered source file. (But of course, I don't go to Java extremes and 
 put *every* tiny little thing in a separate file.) As long as the 
 complexity of having multiple files isn't passed along to user code 
 (hence the frequent "module foo.all" idiom), then I can't say I really 
 see a problem.

I tend to just not like having to constantly grep to see which file XXX is in.

Diff'rent strokes, I guess. I've only ever had that problem with Tango, which seems to kinda follow from the Java-STD-lib school of API design (no offense intended, Tango guys). But if I'm working on something that involves different sections of a codebase, which is very frequent, then I find it to be quite a pain to constantly scroll all around instead of just Ctrl-Tabbing between open files in different tabs.
Oct 25 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message 
 news:ia59si$1r0j$1 digitalmars.com...
 Consider a string literal, say "abc\"def". With Goldie's method, I infer 
 this string has to be scanned twice. Once to find its limits, and the 
 second to convert it to the actual string.

Yea, that is true. With that string in the input, the value given to the user code will be: assert(tokenObtainedFromGoldie.toString() == q{"abc\"def"}); That's a consequence of the grammar being separated from lexing/parsing implementation. You're right that that does seem less than ideal. Although I'm not sure how to remedy that without loosing the independence between grammar and lex/parse implementation that is the main point of the GOLD-based style. But there's something I don't quite understand about the approach you're suggesting: You seem to be suggesting that a terminal be progressively converted into its final form *as* it's still in the process of being recognized by the DFA. Which means, you don't know *what* you're supposed to be converting it into *while* you're converting it. Which means, you have to be speculatively converting it into all types of tokens that the current DFA state could possibly be on its way towards accepting (also, the DFA would need to contain a record of possible terminals for each DFA state). And then the result is thrown away if it turns out to be a different terminal. Is this correct? If so, is there generally enough lexical difference between the terminals that need such treatment to compensate for the extra processing needed in situations that are closer to worst-case (that is, in comparison to Goldie's current approach)?

Probably that's why I don't use lexer generators. Building lexers is the simplest part of building a compiler, and I've always been motivated by trying to make it as fast as possible. To specifically answer your question, yes, in the lexers I make, you know you're parsing a string, so you process it as you parse it.
 If all of that is so, then what would be your thoughts on this approach?:
 
 Suppose Goldie had a way to associate an optional "simultaneous/lockstep 
 conversion" to a type of terminal. For instance:
 
 myLanguage.associateConversion("StringLiteral", new 
 StringLiteralConverter());
 
 Then, 'StringLiteralConverter' would be something that could be either 
 user-provided or offered by Goldie (both ways would be supported). It would 
 be some sort of class or something that had three basic functions:
 
 class StringLiteralConverter : ITerminalConverter
 {
     void process(dchar c) {...}
 
     // Or maybe this to make it possible to minimize allocations
     // in certain circumstances by utilizing slices:
     void process(dchar c, size_t indexIntoSource, string fullOrignalSource) 
 {...}
 
     Variant emit() {...}
     void clear() {...}
 }
 
 Each state in the lexer's DFA would know which terminals it could possibly 
 be processing. And for each of those terminals that has an associated 
 converter, the lexer will call 'process()'. If a terminal is accepted, 
 'emit' is called to get the final result (and maybe do any needed 
 finalization first), and then 'clear' is called on all converters that had 
 been used.
 
 This feature would preclude the use of the actual "GOLD Parser Builder" 
 program, but since I'm writing a tool to handle that functionality anyway, 
 I'm not too concerned about that.
 
 Do you think that would work? Would its benefits be killed by the overhead 
 introduced? If so, could those overheads be sufficiently reduced without 
 scrapping the general idea?

I don't know. I'd have to study the issue for a while. I suggest taking a look at dmd's lexer and compare. I'm not sure what Spirit's approach to this is.
 What Goldie will be compared against is Spirit. Spirit is a reasonably 
 successful add-on to C++. Goldie doesn't have to do things the same way as 
 Spirit (expression templates - ugh), but it should be as easy to use and 
 at least as powerful.

Understood.

Oct 25 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:ia5j41$2bnk$1 digitalmars.com...
 To specifically answer your question, yes, in the lexers I make, you know 
 you're parsing a string, so you process it as you parse it.

...

 I don't know. I'd have to study the issue for a while. I suggest taking a 
 look at dmd's lexer and compare. I'm not sure what Spirit's approach to 
 this is.

I've taken a deeper look at Spirit's docs: In the older Spirit 1.x, the lexing is handled as part of the parsing. The structure of it definitely suggests it should be easy for it to do all token-conversion right as the string is being lexed, although I couldn't tell whether or not it actually did so (I'd have to look at the source). But, since Spirit 1.x doesn't handle lexing separately from parsing, I *think* backtracking (it *is* a backtracking parser) results in re-lexing, even for terminals that never get special processing, such as keywords (But I'm not completely certain because I don't have much experience with LL). In Spirit 2.x, standard usage involves having the lexing separate from parsing. I didn't see anything at all in the docs for Spirit 2.x that seemed to suggest even the possibility of it processing tokens as they're lexed. However, Spirit is designed with heavy policy-based customizability in mind, so such a thing might still possible in Spirit 2.x...But if so, it's definitely an advanced feature (or just really poorly documented). I have thought of another way to get such an ability into Goldie, and it would be very easy-to-use, but it would also be a fairly non-trivial to implement. And really, I'm starting to question again how important it would *really* be, at least initially. When I think of typical code, usually only a small amount of it is made up of the the sorts of terminals that would need extra processing. I have to admit, I still have no idea whether or not it would be worth it to get Goldie into Phobos. Maybe, maybe not, I dunno. I think popular opinion would probably be the best gauge of that. It seems like we're the only ones still in this thread, though...maybe that's a bad sign? ;) I do still think that if your primary goal is to provide parsing of D code through Phobos, then adapting DDMD would be the best best. Goldie would be more appropriate if customized lexing/parsing is the goal.
Oct 26 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Nick Sabalausky:

 I've taken a deeper look at Spirit's docs:

I have not used Spirit, but from what I have read, it doesn't scale (the compilation becomes too much slower when the system you have built becomes bigger). Bye, bearophile
Oct 26 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"bearophile" <bearophileHUGS lycos.com> wrote in message 
news:ia6a0h$nst$1 digitalmars.com...
 Nick Sabalausky:

 I've taken a deeper look at Spirit's docs:

I have not used Spirit, but from what I have read, it doesn't scale (the compilation becomes too much slower when the system you have built becomes bigger).

I think that's just because it's C++ though. I'd bet a D lib that worked the same way would probably do a lot better. In any case, I started writing a comparison of the main fundamental differences between Spirit and Goldie, and it ended up kinda rambling and not so just-the-main-fundamentals. But the gist was: Spirit is very flexible in how grammars are defined and processed, and Goldie is very flexible in what you can do with a given grammar once it's written (ie, how much mileage you can get out of it without changing one line of grammar and without designing it from the start to be flexible). Goldie does get some of that "flexibility in what you can do with it" though by tossing in some features and some limitations/requirements that Spirit leaves as "if you want it, put it in yourself (more or less manually), otherwise you don't pay a price for it." I think both approaches have their merits. Although I haven't a clue which is best for Phobos, or if Phobos even needs either.
Oct 26 2010
prev sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 26.10.2010 15:55, schrieb Leandro Lucarella:
 bearophile, el 26 de octubre a las 06:20 me escribiste:
  Nick Sabalausky:

  >  I've taken a deeper look at Spirit's docs:

  I have not used Spirit, but from what I have read, it doesn't scale
  (the compilation becomes too much slower when the system you have
  built becomes bigger).

I can confirm that, at least for Spirit 1, and for simple things it looks "nice" (in the C++ scale), but for real more complex things, the resulting code is really a mess.

yupp - Spirit feels right on the integration-side, but becomes more and more evil when stuff gets bigger a compiletime-ebnf-script parser would do better, especially when the ebnf-script comes through compiletime-file-include and can be used/developed from outside in an ide like gold parsers a compiletime-parse could "generated" the stub code like Spirit do but without beeing to much inside the language itselfe
Oct 26 2010
parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 26.10.2010 16:48, schrieb dennis luehring:
 Am 26.10.2010 15:55, schrieb Leandro Lucarella:
  bearophile, el 26 de octubre a las 06:20 me escribiste:
   Nick Sabalausky:

   >   I've taken a deeper look at Spirit's docs:

   I have not used Spirit, but from what I have read, it doesn't scale
   (the compilation becomes too much slower when the system you have
   built becomes bigger).

I can confirm that, at least for Spirit 1, and for simple things it looks "nice" (in the C++ scale), but for real more complex things, the resulting code is really a mess.

yupp - Spirit feels right on the integration-side, but becomes more and more evil when stuff gets bigger a compiletime-ebnf-script parser would do better, especially when the ebnf-script comes through compiletime-file-include and can be used/developed from outside in an ide like gold parsers a compiletime-parse could "generated" the stub code like Spirit do but without beeing to much inside the language itselfe

that combined with compiletime-features something like the bsn-parse do http://code.google.com/p/bsn-goldparser/ i think this all is very very doable in D
Oct 26 2010
parent "Nick Sabalausky" <a a.a> writes:
"dennis luehring" <dl.soluz gmx.net> wrote in message 
news:ia6s3b$1q90$1 digitalmars.com...
 Am 26.10.2010 16:48, schrieb dennis luehring:
 Am 26.10.2010 15:55, schrieb Leandro Lucarella:

 yupp - Spirit feels right on the integration-side, but becomes more and
 more evil when stuff gets bigger


Goldie (and any GOLD-based system, really) should scale up pretty well. The only possible scaling-up issues would be: 1. Splitting a large grammar across multiple files is not yet supported (and if I do add support for that in Goldie, and I may, then the "GOLD Parser Builder" IDE wouldn't know how to handle it). 2. Somewhat related to #1, grammars composed of multiple languages (such as classic-style ASP, or anything that involves a preprocessing step that hasn't already been done) aren't really supported yet. Spirit 1.x should be able to handle that, at least in some cases. I think Spirit 2.x's separation of lexing and parsing may have some trouble with it though. 3. I haven't had a chance to add any sort of character set optimization yet, so grammars that allow a large amount of Unicode characters will probably be slow to generate into tables and slow to lex. At least until I get around to taking care of that. I've never actually used Spirit, but its scaling up issues do seem to be a fairly fundamental issue with it's design (particularly so since it's C++). Although they do say on their site that some C++ compilers can handle Spirit without compile time growing exponentially in relation to grammar complexity.
 a compiletime-ebnf-script parser would do better, especially when
 the ebnf-script comes through compiletime-file-include and can be
 used/developed from outside in an ide like gold parsers


There's one problem with doing things via CTFE that us D folks often overlook: You can't use a build tool like make/rake/scons to detect when that particular data doesn't need to be recomputed and can thus be skipped. (Although it may be possible to manually make it work that way *if* CTFE gains the ability to access the filesystem.) I'm not opposed to the idea of making Goldie's compiling-a-grammar (ie, "process a grammar into the appropriate tables") ctfe-able, but it does already work in a way that you only need to compile a grammar into tables when you change the grammar (and changing a grammar is needed less frequently in Goldie than in Spirit because in Goldie no processing code is ever embedded into the grammar.).
 that combined with compiletime-features something like the bsn-parse do

 http://code.google.com/p/bsn-goldparser/

 i think this all is very very doable in D

Yea, I was pretty impressed with BSN. I definitely want to do something like that for Goldie, but I have a somewhat different idea in mind: I'm thinking of enhancing the grammar definition language so that all the information on how to construct an AST is right there in the grammar definition itself, and can thus be completely automated by Goldie. This would be in line with GOLD's philosophy and benefits of keeping the grammar definition separate from the processing code. And it would also be a step towards the idea I've had in mind since before Goldie was Goldie of being able to automate (or partially automate) generalized language translation/transformation.
Oct 26 2010
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-10-26 04:44, Nick Sabalausky wrote:
 "Walter Bright"<newshound2 digitalmars.com>  wrote in message
 news:ia59si$1r0j$1 digitalmars.com...
 Consider a string literal, say "abc\"def". With Goldie's method, I infer
 this string has to be scanned twice. Once to find its limits, and the
 second to convert it to the actual string.

Yea, that is true. With that string in the input, the value given to the user code will be: assert(tokenObtainedFromGoldie.toString() == q{"abc\"def"}); That's a consequence of the grammar being separated from lexing/parsing implementation. You're right that that does seem less than ideal. Although I'm not sure how to remedy that without loosing the independence between grammar and lex/parse implementation that is the main point of the GOLD-based style. But there's something I don't quite understand about the approach you're suggesting: You seem to be suggesting that a terminal be progressively converted into its final form *as* it's still in the process of being recognized by the DFA. Which means, you don't know *what* you're supposed to be converting it into *while* you're converting it.

I don't have much knowledge in this area but isn't this what a look-ahead is for? Just look ahead (hopefully) one character and decide what to convert to. -- /Jacob Carlborg
Oct 26 2010
prev sibling parent Leandro Lucarella <luca llucax.com.ar> writes:
bearophile, el 26 de octubre a las 06:20 me escribiste:
 Nick Sabalausky:
 
 I've taken a deeper look at Spirit's docs:

I have not used Spirit, but from what I have read, it doesn't scale (the compilation becomes too much slower when the system you have built becomes bigger).

I can confirm that, at least for Spirit 1, and for simple things it looks "nice" (in the C++ scale), but for real more complex things, the resulting code is really a mess. -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- A can of diet coke will float in water While a can of regular coke will sink
Oct 26 2010
prev sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Sun, 24 Oct 2010 06:55:22 +0400, Nick Sabalausky <a a.a> wrote:

 "bearophile" <bearophileHUGS lycos.com> wrote in message
 news:ia0410$1lju$1 digitalmars.com...
 Nick Sabalausky:

 But that's all if you want generalized lexing or parsing though. If you
 just
 want "lexing D code"/"parsing D code", then IMO anything other than
 adapting
 parts of DDMD would be the wrong way to go.

Is the DDMD licence compatible with the Phobos one? Is the DDMD author(s) willing?

I'd certainly hope so. If it isn't, then that would probably mean DMD's FE license is incompatible with Phobos. Which would be rather...weird. In any case, I asked that and a couple other Q's here, but haven't gotten an answer yet: http://www.dsource.org/forums/viewtopic.php?t=5627

Sorry, I wasn't checking the forum. IIRC DMD license is GPL so DDMD must be GPL too but I'm all for relicensing it as Boost.
Oct 24 2010
prev sibling next sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 22/10/2010 20:48, Andrei Alexandrescu wrote:
 On 10/22/10 14:02 CDT, Tomek Sowi艅ski wrote:
 Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 digitalmars.com>
 napisa艂(a):

 As we all know, tool support is important for D's success. Making
 tools easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax
 highlighting filters, pretty printers, repl, doc generators, static
 analyzers, and even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript
 implementation that can be embedded into web pages for syntax
 highlighting, and eventually an std.lang.d.parse.

 Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORBA, Thrift, etc, so lexers will have to be written all over to grok interface files. Perhaps a generic tokenizer which can be parametrized with a lexical grammar would bring more ROI, I got a hunch D's templates are strong enough to pull this off without any source code generation ala JavaCC. The books I read on compilers say tokenization is a solved problem, so the theory part on what a good abstraction should be is done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer generator.

Agreed, of all the things desired for D, a D tokenizer would rank pretty low I think. Another thing, even though a tokenizer generator would be much more desirable, I wonder if it is wise to have that in the standard library? It does not seem to be of wide enough interest to be in a standard library. (Out of curiosity, how many languages have such a thing in their standard library?) -- Bruno Medeiros - Software Engineer
Nov 19 2010
next sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 19/11/2010 21:27, Jonathan M Davis wrote:
 On Friday 19 November 2010 13:03:53 Bruno Medeiros wrote:
 On 22/10/2010 20:48, Andrei Alexandrescu wrote:
 On 10/22/10 14:02 CDT, Tomek Sowi艅ski wrote:
 Dnia 22-10-2010 o 00:01:21 Walter Bright<newshound2 digitalmars.com>

 napisa艂(a):
 As we all know, tool support is important for D's success. Making
 tools easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax
 highlighting filters, pretty printers, repl, doc generators, static
 analyzers, and even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript
 implementation that can be embedded into web pages for syntax
 highlighting, and eventually an std.lang.d.parse.

 Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORBA, Thrift, etc, so lexers will have to be written all over to grok interface files. Perhaps a generic tokenizer which can be parametrized with a lexical grammar would bring more ROI, I got a hunch D's templates are strong enough to pull this off without any source code generation ala JavaCC. The books I read on compilers say tokenization is a solved problem, so the theory part on what a good abstraction should be is done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer generator.

Agreed, of all the things desired for D, a D tokenizer would rank pretty low I think. Another thing, even though a tokenizer generator would be much more desirable, I wonder if it is wise to have that in the standard library? It does not seem to be of wide enough interest to be in a standard library. (Out of curiosity, how many languages have such a thing in their standard library?)

We want to make it easy for tools to be built to work on and deal with D code. An IDE, for example, needs to be able to tokenize and parse D code. A program like lint needs to be able to tokenize and parse D code. By providing a lexer and parser in the standard library, we are making it far easier for such tools to be written, and they could be of major benefit to the D community. Sure, the average program won't need to lex or parse D, but some will, and making it easy to do will make it a lot easier for such programs to be written. - Jonathan M Davis

And by providing a lexer and a parser outside the standard library, wouldn't it make it just as easy for those tools to be written? What's the advantage of being in the standard library? I see only disadvantages: to begin with it potentially increases the time that Walter or other Phobos contributors may have to spend on it, even if it's just reviewing patches or making sure the code works. -- Bruno Medeiros - Software Engineer
Nov 19 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 19/11/2010 22:02, Jonathan M Davis wrote:
 On Friday, November 19, 2010 13:53:12 Bruno Medeiros wrote:
 On 19/11/2010 21:27, Jonathan M Davis wrote:

 And by providing a lexer and a parser outside the standard library,
 wouldn't it make it just as easy for those tools to be written? What's
 the advantage of being in the standard library? I see only
 disadvantages: to begin with it potentially increases the time that
 Walter or other Phobos contributors may have to spend on it, even if
 it's just reviewing patches or making sure the code works.

If nothing, else, it makes it easier to keep in line with dmd itself. Since the dmd front end is LGPL, it's not possible to have a Boost port of it (like the Phobos version will be) without Walter's consent. And I'd be surprised if he did that for a third party library (though he seems to be pretty open on a lot of that kind of stuff). Not to mention, Walter and the core developers are _exactly_ the kind of people that you want working on a lexer or parser of the language itself, because they're the ones who work on it. - Jonathan M Davis

Eh? That license argument doesn't make sense: if the lexer and parser were to be based on DMD itself, then putting it in the standard library is equivalent (in licensing terms) to licensing the lexer and parser parts of DMD in Boost. More correctly, what I mean by equivalent, is that there no reason why Walter would allow one thing and not the other... (because on both cases he would have to issue that license) As for your second argument, yes, Walter and the core developers would be the most qualified people to work in it, no question about it. But my point is, I don't think Walter and Phobos core devs should be working on it, because it takes time away from other things that are much more important. Their time is precious. I think our main point of disagreement is just how important a D lexer and/or parser would be. I think it would be of very low interest, definitely not a "major benefit to the D community". For starters, regarding its use in IDEs: I think we are *ages* away from the point were an IDE based on D only will be able to compete with IDEs based in Eclipse/Visual-Studio/Xcode/etc.. I think much sooner we will have a full D compiler written in D than a (competitive) D IDE written in D. We barely have mature GUI libraries from what I understand. (What may be more realistic is an IDE partially written in D, and otherwise based on Eclipse/Visual-Studio/etc., but even so, I think it would be hard to compete with other non-D IDEs) -- Bruno Medeiros - Software Engineer
Nov 19 2010
next sibling parent reply Todd VanderVeen <TDVanderVeen gmail.com> writes:
== Quote from Bruno Medeiros (brunodomedeiros+spam com.gmail)'s article
 I think much sooner we will
 have a full D compiler written in D than a (competitive) D IDE written
 in D.

I agree. I do like the suggestion for developing the D grammar in Antlr though and it is something I would be interested in working on. With this in hand, the prospect of adding D support as was done for C++ to Eclipse or Netbeans becomes much more feasible. Has a complete grammar been defined/compiled or is anyone currently working in this direction? Having a robust IDE seems far more important than whether it is written in D itself.
Nov 19 2010
parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 19/11/2010 23:45, Todd VanderVeen wrote:
 == Quote from Bruno Medeiros (brunodomedeiros+spam com.gmail)'s article
 I think much sooner we will
 have a full D compiler written in D than a (competitive) D IDE written
 in D.

I agree. I do like the suggestion for developing the D grammar in Antlr though and it is something I would be interested in working on. With this in hand, the prospect of adding D support as was done for C++ to Eclipse or Netbeans becomes much more feasible. Has a complete grammar been defined/compiled or is anyone currently working in this direction? Having a robust IDE seems far more important than whether it is written in D itself.

See the comment I made below, to Michael Stover. ( news://news.digitalmars.com:119/ic71pa$1lev$1 digitalmars.com ) -- Bruno Medeiros - Software Engineer
Nov 19 2010
prev sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 20/11/2010 01:29, Jonathan M Davis wrote:
 On Friday, November 19, 2010 15:17:35 Bruno Medeiros wrote:
 On 19/11/2010 22:02, Jonathan M Davis wrote:
 On Friday, November 19, 2010 13:53:12 Bruno Medeiros wrote:
 On 19/11/2010 21:27, Jonathan M Davis wrote:

 And by providing a lexer and a parser outside the standard library,
 wouldn't it make it just as easy for those tools to be written? What's
 the advantage of being in the standard library? I see only
 disadvantages: to begin with it potentially increases the time that
 Walter or other Phobos contributors may have to spend on it, even if
 it's just reviewing patches or making sure the code works.

If nothing, else, it makes it easier to keep in line with dmd itself. Since the dmd front end is LGPL, it's not possible to have a Boost port of it (like the Phobos version will be) without Walter's consent. And I'd be surprised if he did that for a third party library (though he seems to be pretty open on a lot of that kind of stuff). Not to mention, Walter and the core developers are _exactly_ the kind of people that you want working on a lexer or parser of the language itself, because they're the ones who work on it. - Jonathan M Davis

Eh? That license argument doesn't make sense: if the lexer and parser were to be based on DMD itself, then putting it in the standard library is equivalent (in licensing terms) to licensing the lexer and parser parts of DMD in Boost. More correctly, what I mean by equivalent, is that there no reason why Walter would allow one thing and not the other... (because on both cases he would have to issue that license)

It's very different to have D implementation of something - which is based on a C++ version but definitely different in some respects - be under Boost and generally available, and having the C++ implementation be under Boost - particularly when the C++ version covers far more than just a lexer and parser. Someone _could_ port the D code back to C++ and have that portion useable under Boost, but that's a lot more work than just taking the C++ code and using it, and it's only the portions of the compiler which were ported to D to which could be re-used that way. And since the Boost code could be used in a commercial product while the LGPL is more restricted, it could make a definite difference. I'm not a licensing expert, and I'm not an expert on what Walter does and doesn't want done with his code, but he put the compiler front end under the LGPL, not Boost, and he's given his permission to have the lexer alone ported to D and put under the Boost license in the standard library, which is very different from putting the entire front end under Boost. I expect that the parser will follow eventually, but even if it does, that's still not the entire front end. So, there is a difference in licenses does have a real impact. And no one can take the LGPL C++ code and port it to D - for the standard library or otherwise - without Walter's permission, because its his copyright on the code.

There are some misunderstandings here. First, the DMD front-end is licenced under the GPL, not LGPL. Second, more importantly, it is actually also licensed under the Artistic license, a very permissible license. This is the basis for me stating that almost certainly Walter would not mind licensing the DMD parser and lexer under Boost, as it's actually not that different from the Artistic license.
 Ideally, Phobos would be huge in a manner similar to how C# or Java's libraries
 are huge. It will take time to get there, and we'll need more developers, but I

That point actually works in my favor. C# and Java's libraries are much bigger than Phobos, and yet they have no functionality for lexing/parsing their own languages (or any other for that matter)! -- Bruno Medeiros - Software Engineer
Nov 24 2010
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/19/10 1:03 PM, Bruno Medeiros wrote:
 On 22/10/2010 20:48, Andrei Alexandrescu wrote:
 On 10/22/10 14:02 CDT, Tomek Sowi艅ski wrote:
 Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 digitalmars.com>
 napisa艂(a):

 As we all know, tool support is important for D's success. Making
 tools easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax
 highlighting filters, pretty printers, repl, doc generators, static
 analyzers, and even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript
 implementation that can be embedded into web pages for syntax
 highlighting, and eventually an std.lang.d.parse.

 Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORBA, Thrift, etc, so lexers will have to be written all over to grok interface files. Perhaps a generic tokenizer which can be parametrized with a lexical grammar would bring more ROI, I got a hunch D's templates are strong enough to pull this off without any source code generation ala JavaCC. The books I read on compilers say tokenization is a solved problem, so the theory part on what a good abstraction should be is done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer generator.

Agreed, of all the things desired for D, a D tokenizer would rank pretty low I think. Another thing, even though a tokenizer generator would be much more desirable, I wonder if it is wise to have that in the standard library? It does not seem to be of wide enough interest to be in a standard library. (Out of curiosity, how many languages have such a thing in their standard library?)

Even C has strtok. Andrei
Nov 19 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 19/11/2010 23:39, Andrei Alexandrescu wrote:
 On 11/19/10 1:03 PM, Bruno Medeiros wrote:
 On 22/10/2010 20:48, Andrei Alexandrescu wrote:
 On 10/22/10 14:02 CDT, Tomek Sowi艅ski wrote:
 Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 digitalmars.com>
 napisa艂(a):

 As we all know, tool support is important for D's success. Making
 tools easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax
 highlighting filters, pretty printers, repl, doc generators, static
 analyzers, and even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript
 implementation that can be embedded into web pages for syntax
 highlighting, and eventually an std.lang.d.parse.

 Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORBA, Thrift, etc, so lexers will have to be written all over to grok interface files. Perhaps a generic tokenizer which can be parametrized with a lexical grammar would bring more ROI, I got a hunch D's templates are strong enough to pull this off without any source code generation ala JavaCC. The books I read on compilers say tokenization is a solved problem, so the theory part on what a good abstraction should be is done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer generator.

Agreed, of all the things desired for D, a D tokenizer would rank pretty low I think. Another thing, even though a tokenizer generator would be much more desirable, I wonder if it is wise to have that in the standard library? It does not seem to be of wide enough interest to be in a standard library. (Out of curiosity, how many languages have such a thing in their standard library?)

Even C has strtok. Andrei

That's just a fancy splitter, I wouldn't call that a proper tokenizer. I meant something that, at the very least, would tokenize based on regular expressions (and have heterogenous tokens). -- Bruno Medeiros - Software Engineer
Nov 24 2010
parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 24/11/2010 13:30, Bruno Medeiros wrote:
 On 19/11/2010 23:39, Andrei Alexandrescu wrote:
 On 11/19/10 1:03 PM, Bruno Medeiros wrote:
 On 22/10/2010 20:48, Andrei Alexandrescu wrote:
 On 10/22/10 14:02 CDT, Tomek Sowi艅ski wrote:
 Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 digitalmars.com>
 napisa艂(a):

 As we all know, tool support is important for D's success. Making
 tools easier to build will help with that.

 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax
 highlighting filters, pretty printers, repl, doc generators, static
 analyzers, and even D compilers.

 It should:

 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover
 and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem

 It can also serve as the basis for creating a javascript
 implementation that can be embedded into web pages for syntax
 highlighting, and eventually an std.lang.d.parse.

 Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORBA, Thrift, etc, so lexers will have to be written all over to grok interface files. Perhaps a generic tokenizer which can be parametrized with a lexical grammar would bring more ROI, I got a hunch D's templates are strong enough to pull this off without any source code generation ala JavaCC. The books I read on compilers say tokenization is a solved problem, so the theory part on what a good abstraction should be is done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer generator.

Agreed, of all the things desired for D, a D tokenizer would rank pretty low I think. Another thing, even though a tokenizer generator would be much more desirable, I wonder if it is wise to have that in the standard library? It does not seem to be of wide enough interest to be in a standard library. (Out of curiosity, how many languages have such a thing in their standard library?)

Even C has strtok. Andrei

That's just a fancy splitter, I wouldn't call that a proper tokenizer. I meant something that, at the very least, would tokenize based on regular expressions (and have heterogenous tokens).

In other words, a lexer, that might be a better term in this context. -- Bruno Medeiros - Software Engineer
Nov 24 2010
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday 19 November 2010 13:03:53 Bruno Medeiros wrote:
 On 22/10/2010 20:48, Andrei Alexandrescu wrote:
 On 10/22/10 14:02 CDT, Tomek Sowi=C5=84ski wrote:
 Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 digitalmars.com>
=20
 napisa=C5=82(a):
 As we all know, tool support is important for D's success. Making
 tools easier to build will help with that.
=20
 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax
 highlighting filters, pretty printers, repl, doc generators, static
 analyzers, and even D compilers.
=20
 It should:
=20
 1. support a range interface for its input, and a range interface for
 its output
 2. optionally not generate lexical errors, but just try to recover and
 continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem
=20
 It can also serve as the basis for creating a javascript
 implementation that can be embedded into web pages for syntax
 highlighting, and eventually an std.lang.d.parse.
=20
 Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORBA, Thrift, etc, so lexers will have to be written all over to grok interface files. Perhaps a generic tokenizer which can be parametrized with a lexical grammar would bring more ROI, I got a hunch D's templat=



 are strong enough to pull this off without any source code generation
 ala JavaCC. The books I read on compilers say tokenization is a solved
 problem, so the theory part on what a good abstraction should be is
 done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer generator.

Agreed, of all the things desired for D, a D tokenizer would rank pretty low I think. =20 Another thing, even though a tokenizer generator would be much more desirable, I wonder if it is wise to have that in the standard library? It does not seem to be of wide enough interest to be in a standard library. (Out of curiosity, how many languages have such a thing in their standard library?)

We want to make it easy for tools to be built to work on and deal with D co= de.=20 An IDE, for example, needs to be able to tokenize and parse D code. A progr= am=20 like lint needs to be able to tokenize and parse D code. By providing a lex= er=20 and parser in the standard library, we are making it far easier for such to= ols=20 to be written, and they could be of major benefit to the D community. Sure,= the=20 average program won't need to lex or parse D, but some will, and making it = easy=20 to do will make it a lot easier for such programs to be written. =2D Jonathan M Davis
Nov 19 2010
prev sibling parent =?iso-8859-2?B?VG9tZWsgU293afFza2k=?= <just ask.me> writes:
Dnia 22-10-2010 o 21:48:49 Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> napisa=B3(a):

 On 10/22/10 14:02 CDT, Tomek Sowi=F1ski wrote:
 Interesting idea. Here's another: D will soon need bindings for CORBA=


 Thrift, etc, so lexers will have to be written all over to grok
 interface files. Perhaps a generic tokenizer which can be parametrize=


 with a lexical grammar would bring more ROI, I got a hunch D's templa=


 are strong enough to pull this off without any source code generation=


 ala JavaCC. The books I read on compilers say tokenization is a solve=


 problem, so the theory part on what a good abstraction should be is
 done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenize=

 generator.

 I have in mind the entire implementation of a simple design, but never=

 had the time to execute on it. The tokenizer would work like this:

 alias Lexer!(
      "+", "PLUS",
      "-", "MINUS",
      "+=3D", "PLUS_EQ",
      ...
      "if", "IF",
      "else", "ELSE"
      ...
 ) DLexer;

Yes. One remark: native language constructs scale better for a grammar: enum TokenDef : string { Digit =3D "[0-9]", Letter =3D "[a-zA-Z_]", Identifier =3D Letter~'('~Letter~'|'~Digit~')', ... Plus =3D "+", Minus =3D "-", PlusEq =3D "+=3D", ... If =3D "if", Else =3D "else", ... } alias Lexer!TokenDef DLexer; BTW, there's a bug related: http://d.puremagic.com/issues/show_bug.cgi?id=3D2950
 Such a declaration generates numeric values DLexer.PLUS etc. and  =

 generates an efficient code that extracts a stream of tokens from a  =

 stream of text. Each token in the token stream has the ID and the text=

All good ideas.
 Comments, strings etc. can be handled in one of several ways but that'=

 a longer discussion.

The discussion's started anyhow. So what're the options? -- = Tomek
Oct 22 2010
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:i9qd8q$1ls4$1 digitalmars.com...
 4. the tokens should be a value type, not a reference type

I'm curious, is your reason for this purely to avoid allocations during lexing, or are there other reasons too? If it's mainly to avoid allocations during lexing then, maybe I've understood wrong, but isn't D2 getting the ability to construct class objects in-place into pre-allocated memory? (or already has the ability?) If so, do you think just creating the tokens that way would likely be close enough?
Oct 26 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message 
 news:i9qd8q$1ls4$1 digitalmars.com...
 4. the tokens should be a value type, not a reference type

I'm curious, is your reason for this purely to avoid allocations during lexing, or are there other reasons too?

It's one big giant reason. Storage allocation gets unbelievably costly in a lexer. Another is it makes tokens easy to copy. Another one is that classes are for polymorphic behavior. What kind of polymorphic behavior would one want with tokens?
 If it's mainly to avoid allocations during lexing then, maybe I've 
 understood wrong, but isn't D2 getting the ability to construct class 
 objects in-place into pre-allocated memory?

If you do that, might as well make them value types. The only reason classes exist is to support runtime polymorphism. C++ made a vast mistake in failing to distinguish between value types and reference types. Java made a related mistake by failing to acknowledge that value types have any useful purpose at all (unless they are built-in).
Oct 26 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter:

 Java made a related mistake by failing to acknowledge that 
 value types have any useful purpose at all (unless they are built-in).

Java was designed to be simple! Simple means to have a more uniform semantics. Removing value types was a good idea if you want to simplify a language (and remove a mountain of details from C++). And from the huge success of Java, I think the designers were right (later C# has added values too, but C# is indeed a more complex than Java). The Java VM also is now often able to allocate not escaping objects on the stack (escape analysis) regaining some of the lost performance. What I miss more in Java is not single structs (single values), but a way to build an array of values (structs). Because using parallel arrays is not nice at all. Even in Python using numPy you may create an array of structs (compound value items). Bye, bearophile
Oct 26 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
bearophile wrote:
 Walter:
 
 Java made a related mistake by failing to acknowledge that value types have
 any useful purpose at all (unless they are built-in).

Java was designed to be simple! Simple means to have a more uniform semantics.

So was Pascal. See the thread about how useless it was as a result. A hatchet is a very simple tool, easy to understand, and I could build a house with nothing but a hatchet. But it would make the house several times as expensive to build, and it would look like it was built with a hatchet.
 Removing value types was a good idea if you want to simplify a
 language (and remove a mountain of details from C++). And from the huge
 success of Java, I think the designers were right (later C# has added values
 too, but C# is indeed a more complex than Java). The Java VM also is now
 often able to allocate not escaping objects on the stack (escape analysis)
 regaining some of the lost performance.

The issue isn't just about lost performance. It's about proper encapsulation of a type. Value types and polymorphic types are different, have different purposes, different behaviors, etc. Conflating the two into the same construct makes for poor and confusing abstractions. It shifts the problem out of the language and onto the programmer. It does NOT make the complexity go away.
 What I miss more in Java is not single structs (single values),

There's a lot more to miss than that. I find Java code tends to be excessively complex, and that's because it lacks expressive power. It was summed up for me by a colleague who said that one needs an IDE to program in Java because with one button it will auto-generate 100 lines of boilerplate.
Oct 26 2010
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
retard wrote:
 have you even started
 studying the list of type system concepts I listed few days ago.

Java has proved that such things aren't useful in programming languages :-)
 Adding structs to Java wouldn't fix that.  You probably know that.
 Unifying structs and classes in a language like D and adding good escape 
 analysis wouldn't worsen the performance that badly in general purpose 
 applications. Java is mostly used for general purpose programming so your 
 claims about usefulness and the need for extreme performance look silly.

If that were true, why are Java char/int/double types value types, not a reference type derived from Object?
Oct 27 2010
prev sibling next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter:

 So was Pascal. See the thread about how useless it was as a result.

But Java is probably currently the most used language, so I guess they have created a simpler language, but not too much simple as Pascal was.
 Value types and polymorphic types are different, have different

Right.
Conflating the two into the same construct makes for poor and confusing
abstractions.<

In Python there are (more or less) only objects, and they are managed "by name" (similar a "by reference") and it works well enough.
It shifts the problem out of the language and onto the programmer. It does NOT
make the complexity go away.<

This is partially true. The presence of just objects doesn't solve all problems, so part of the complexity doesn't go away, it goes into the program. On the other hand value semantics introduces a large amount of complexity by itself (in C++ there is a huge amount of stuff about this semantics, and even in D the design is unfinished still after ten years and after all the experience with C++). So in my opinion in the end the net result is that removing structs makes the language+programs simpler.
 There's a lot more to miss than that. I find Java code tends to be excessively
 complex, and that's because it lacks expressive power. It was summed up for me
 by a colleague who said that one needs an IDE to program in Java because with
 one button it will auto-generate 100 lines of boilerplate.

Yes, clearly Java has several faults. It's far from perfect. But introducing structs inside Java is in my opinion not going to solve those problems much. (Anyway, C# is like a Java+Structs and it too works well enough).
 [...] If that were true, why are Java char/int/double types value types,
 not a reference type derived from Object?

For performance reasons, because originally Java didn't have the advanced compilation strategies used today. Languages like Clojure that run on the JavaVM use more reference types (for integer numbers too). After all this discussion I want to remind you that I am here because I like D and I like D structs, unions and all that :-) I prefer to use D many times over Java. And I agree that structs (or tagged unions) are better in D for the lexer if you want the lexer to be quite fast. Bye, bearophile
Oct 27 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
bearophile wrote:
 After all this discussion I want to remind you that I am here because I like
 D and I like D structs, unions and all that :-) I prefer to use D many times
 over Java. And I agree that structs (or tagged unions) are better in D for
 the lexer if you want the lexer to be quite fast.

So, there is "value" in value types after all. I confess I have no idea why you argue against them.
Oct 27 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Walter:

 So, there is "value" in value types after all. I confess I have no idea why
you 
 argue against them.

I am not arguing against them in absolute. They are good in some situations and not so good in other situations :-) Compound value types are very useful in a certain imperative low-level language. While if you are designing a simpler language, it's better to not add structs to it (and yes, in practice the world needs simpler languages too, not everyone needs a Ferrari in every moment. And I believe that removing structs makes on average simpler the sum of the language+its programs). Bye, bearophile
Oct 27 2010
prev sibling next sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 27/10/2010 05:39, Walter Bright wrote:
 What I miss more in Java is not single structs (single values),

There's a lot more to miss than that. I find Java code tends to be excessively complex, and that's because it lacks expressive power. It was summed up for me by a colleague who said that one needs an IDE to program in Java because with one button it will auto-generate 100 lines of boilerplate.

I've been hearing that a lot, but I find this to be excessively exaggerated. Can you give some concrete examples? Because regarding excessive verbosity in Java, I cab only remember tree significant things at the moment (at least disregarding meta programming), and one of them is nearly as verbose in D as in Java: 1) writing getters and setters for fields 2) verbose syntax for closures. (need to use an anonymous class, outer variables must be final, and wrapped in an array if write access is needed) 3) writing trivial constructors whose parameters mirror the fields, and then constructors assign the parameters to the fields. I don't think 1 and 2 happens that often to be that much of an annoyance. (unless you're one of those Java persons that thinks that directly accessing the public field of another class is a sin, instead every single field must have getters/setters and never ever be public...) As an additional note, I don't think having an IDE auto-generate X lines of boilerplate code is necessarily a bad thing. It's only bad if the alternative of having a better language feature would actually save me coding time (whether initial coding, or subsequent modifications) or improve code understanding. _Isn't this what matters?_ -- Bruno Medeiros - Software Engineer
Nov 19 2010
prev sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 27/10/2010 05:39, Walter Bright wrote:
 bearophile wrote:
 Walter:
 Java was designed to be simple! Simple means to have a more uniform
 semantics.

So was Pascal. See the thread about how useless it was as a result.

There's good simple, and there's bad simple... -- Bruno Medeiros - Software Engineer
Nov 19 2010
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:ia8321$vug$1 digitalmars.com...
 Nick Sabalausky wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message 
 news:i9qd8q$1ls4$1 digitalmars.com...
 4. the tokens should be a value type, not a reference type

I'm curious, is your reason for this purely to avoid allocations during lexing, or are there other reasons too?

It's one big giant reason. Storage allocation gets unbelievably costly in a lexer. Another is it makes tokens easy to copy. Another one is that classes are for polymorphic behavior. What kind of polymorphic behavior would one want with tokens?

Honestly, I'm not entirely certain whether or not Goldie actually needs its tokens to be classes instead of structs, but I'll explain my current usage: In the basic "dynamic" style, every token in Goldie, terminal or nonterminal, is of type "class Token" no matter what its symbol or part of grammar it represents. But Goldie has an optional "static" style which creates a class hierarchy, for the sake of compile-time type safety, with "Token" as the base. Suppose you have a grammar named "simple" that's a series of one or more a's and b's separated by plus signs: <Item> ::= 'a' | 'b' <List> ::= <List> '+' <Item> | <Item> So there's three terminals: "a", "b", and "+" And two nonterminals: "<Item>" and "<List>", each with exactly two possible reductions. So in dynamic-style, all of those are "class Token", and that's all that exists. But with the optional static-style, the following class hierarchy is effectively created: class Token; class Token_simple : Token; class Token_simple!"a" : Token_simple; class Token_simple!"b" : Token_simple; class Token_simple!"+" : Token_simple; class Token_simple!"<Item>" : Token_simple; class Token_simple!"<List>" : Token_simple; class Token_simple!("<Item>", "a") : Token_simple!"<Item>"; class Token_simple!("<Item>", "b") : Token_simple!"<Item>"; class Token_simple!("<List>", "<List>", "+", "<Item>") : Token_simple!"<List>"; class Token_simple!("<List>", "<Item>") : Token_simple!"<List>"; So rules inherit from the nonterminal they reduce to; terminals and nonterminals inherit from a dummy class dedicated specifically to the given grammar; and that inherits from plain old dynamic-style "Token". All of those template parameters are validated at compile-time. (At some point I'd like to make it possible to specify the rule-based tokens as something like: Token!("<List> ::= <List> '+' <Item>"), but I haven't gotten to it yet.) There's one more trick: The plain old Token exposes a member "subX" which can be numerically indexed to obtain the sub-tokens (for terminals, subX.length==0): void foo(Token token) { if(token.matches("<List>", "<List>", "+", "<Item>")) { auto leftSide = token.subX[0]; auto rightSide = token.subX[2]; //auto dummy = token.subX[10]; // Run-time error static assert(is(typeof(leftSide) == Token)); static assert(is(typeof(rightSide) == Token)); } } Note that it's impossible for the "matches" function to verify at compile-time that its arguments are valid. All of the static-style token types retain all of that for total compatibility with the dynamic-style. But the static-style nonterminals provide an additional member, "sub", that can be used like this: void foo(Token_simple!("<List>", "<List>", "+", "<Item>") token) { auto leftSide = token.sub!0; auto rightSide = token.sub!2; //auto dummy = token.sub!10; // Compile-time error static assert(is(typeof(leftSide) == Token_simple!"<List>")); static assert(is(typeof(rightSide) == Token_simple!"<Item>")); } As for whether or not this effect can be reasonably accomplished with structs: I have no idea, I haven't really looked into it.
Oct 26 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 As for whether or not this effect can be reasonably accomplished with 
 structs: I have no idea, I haven't really looked into it.

I use a tagged variant for the token struct. This doesn't make any difference if one is parsing small pieces of code. But when you're trying to stuff millions of lines of code down its maw, avoiding an allocation per token is a big deal. The indirect calls to the member functions of a class also perform poorly relative to tagged variants.
Oct 26 2010
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
retard wrote:
 This is why the basic data structure in functional languages, algebraic 
 data types, suits better for this purpose.

I think you recently demonstrated otherwise, as proven by the widespread use of Java :-)
Oct 27 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
retard wrote:
 Wed, 27 Oct 2010 12:08:19 -0700, Walter Bright wrote:
 
 retard wrote:
 This is why the basic data structure in functional languages, algebraic
 data types, suits better for this purpose.

use of Java :-)

I don't understand your logic -- Widespread use of Java proves that algebraic data types aren't a better suited way for expressing compiler's data structures such as syntax trees?

You told me that widespread use of Java proved that nothing more complex than what Java provides is useful: "Java is mostly used for general purpose programming so your claims about usefulness and the need for extreme performance look silly." I'd be surprised if you seriously meant that, as it implies that Java is the pinnacle of computer language design, but I can't resist teasing you about it. :-)
Oct 27 2010
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
retard wrote:
 I only meant that the widespead adoption of Java shows how the public at 
 large cares very little about the performance issues you mentioned  Java
 is one of the most widely used languages and it's also successful in many 
 fields. Things could be better from programming language theory's point 
 of view, but the business world is more interesting in profits and the 
 large pool of Java coders has given better benefits than more expressive 
 languages. I don't think that says anything against my notes about 
 algebraic data types.

Choice of a language has numerous factors, so you cannot dismiss one factor because the other factors still make it an attractive choice. For example: "the widespread adoption of horses shows how the public at large cares very little about the cars you mentioned."
Oct 27 2010
prev sibling next sibling parent reply Don <nospam nospam.com> writes:
retard wrote:
 Wed, 27 Oct 2010 16:04:34 -0600, Todd D. VanderVeen wrote:
 
 Legacy in the sense that C is perhaps.

 http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

Probably the top 10 names are more or less correct there, but some funny notes: 33. D 36. Scratch 40. Haskell 42. JavaFX Script 49. Scala Scratch is an educational tool. It isn't really suitable for any real world applications. It slows down considerably with too many expressions. There are several books about Haskell and Scala. Both have several books on them, active mailing lists, and also very many active community projects. Haven't heard much about JavaFX outside Sun/Oracle. These statistics look really weird.

I reckon Fortran is the one to look at it. If Tiobe's stats were sensible, the Fortran numbers would be solid as a rock. And ADA ought to be pretty stable too. But look at this: http://www.tiobe.com/index.php/paperinfo/tpci/Ada.html Laughable.
Oct 28 2010
parent Matthias Pleh <sufu alter.de> writes:
Am 28.10.2010 16:46, schrieb Don:
 retard wrote:
 Wed, 27 Oct 2010 16:04:34 -0600, Todd D. VanderVeen wrote:

 Legacy in the sense that C is perhaps.

 http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

Probably the top 10 names are more or less correct there, but some funny notes: 33. D 36. Scratch 40. Haskell 42. JavaFX Script 49. Scala Scratch is an educational tool. It isn't really suitable for any real world applications. It slows down considerably with too many expressions. There are several books about Haskell and Scala. Both have several books on them, active mailing lists, and also very many active community projects. Haven't heard much about JavaFX outside Sun/Oracle. These statistics look really weird.

I reckon Fortran is the one to look at it. If Tiobe's stats were sensible, the Fortran numbers would be solid as a rock. And ADA ought to be pretty stable too. But look at this: http://www.tiobe.com/index.php/paperinfo/tpci/Ada.html Laughable.

There was an article in the Ct-Magazin (German) where they took a closer look at this rankings. For example: - search for 'C' you got 3080 M - search for 'Java' you got 167 M 'Java' only competes with the island Java 'C' competes with C&A, c't-Magazin, C-Quadrat, C+C, char 'c', ... and many many more .... So, to correct this, only the first *100* (hundred) results are reviewed and the resulting factor applied to the sum of results. Just look at 1-100, then at 101-200, 201-300 and so on .. You get complete different factors. So this numbers at tiobe are really lying!!!!!! source: http://www.heise.de/developer/artikel/Traue-keiner-Statistik-993137.html greets Matthias
Oct 28 2010
prev sibling next sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 27/10/2010 22:04, retard wrote:
 Wed, 27 Oct 2010 13:52:29 -0700, Walter Bright wrote:

 retard wrote:
 Wed, 27 Oct 2010 12:08:19 -0700, Walter Bright wrote:

 retard wrote:
 This is why the basic data structure in functional languages,
 algebraic data types, suits better for this purpose.

widespread use of Java :-)

I don't understand your logic -- Widespread use of Java proves that algebraic data types aren't a better suited way for expressing compiler's data structures such as syntax trees?

You told me that widespread use of Java proved that nothing more complex than what Java provides is useful: "Java is mostly used for general purpose programming so your claims about usefulness and the need for extreme performance look silly." I'd be surprised if you seriously meant that, as it implies that Java is the pinnacle of computer language design, but I can't resist teasing you about it. :-)

I only meant that the widespead adoption of Java shows how the public at large cares very little about the performance issues you mentioned. Java is one of the most widely used languages and it's also successful in many fields. Things could be better from programming language theory's point of view, but the business world is more interesting in profits and the large pool of Java coders has given better benefits than more expressive languages. I don't think that says anything against my notes about algebraic data types.

"the widespead adoption of Java shows how the public at large cares very little about the performance issues you mentioned" WTF? The widespead adoption of Java means that _Java developers_ at large don't care about those performance issues (mostly because they work on stuff where they don't need to). But it's no statement about all the pool of developers. Java is hugely popular, but not in a "it's practically the only language people use" way. It's not like Windows on the desktop. -- Bruno Medeiros - Software Engineer
Nov 19 2010
prev sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 19/11/2010 22:25, Michael Stover wrote:
 As for D lexers and tokenizers, what would be nice is to
 A) build an antlr grammar for D
 B) build D targets for antlr so that antlr can generate lexers and
 parsers in the D language.

 For B) I found http://www.mbutscher.de/antlrd/index.html

 For A) A good list of antlr grammars is at
 http://www.antlr.org/grammar/list, but there isn't a D grammar.

 These things wouldn't be an enormous amount of work to create and
 maintain, and, if done, anyone could parse D code in many languages,
 including Java and C which would make providing IDE features for D
 development easier in those languages (eclipse for instance), and you
 could build lexers and parsers in D using antlr grammars.

 -Mike

Yes, that would be much better. It would be directly and immediately useful for the DDT project: "But better yet would be to start coding our own custom parser (using a parser generator like ANTLR for example), that could really be tailored for IDE needs. In the medium/long term, that's probably what needs to be done. " in http://www.digitalmars.com/d/archives/digitalmars/D/ide/Future_of_Descent_and_D_Eclipse_IDE_635.html -- Bruno Medeiros - Software Engineer
Nov 19 2010
next sibling parent Matthias Pleh <gonzo web.at> writes:
Am 20.11.2010 00:56, schrieb Michael Stover:
 so that was 4 months ago - how do things currently stand on that initiative?

 -Mike

 On Fri, Nov 19, 2010 at 6:37 PM, Bruno Medeiros
 <brunodomedeiros+spam com.gmail> wrote:

     On 19/11/2010 22:25, Michael Stover wrote:

         As for D lexers and tokenizers, what would be nice is to
         A) build an antlr grammar for D
         B) build D targets for antlr so that antlr can generate lexers and
         parsers in the D language.

         For B) I found http://www.mbutscher.de/antlrd/index.html

         For A) A good list of antlr grammars is at
         http://www.antlr.org/grammar/list, but there isn't a D grammar.

         These things wouldn't be an enormous amount of work to create and
         maintain, and, if done, anyone could parse D code in many languages,
         including Java and C which would make providing IDE features for D
         development easier in those languages (eclipse for instance),
         and you
         could build lexers and parsers in D using antlr grammars.

         -Mike


     Yes, that would be much better. It would be directly and immediately
     useful for the DDT project:

     "But better yet would be to start coding our own custom parser
     (using a parser generator like ANTLR for example), that could really
     be tailored for IDE needs. In the medium/long term, that's probably
     what needs to be done. "
     in
     http://www.digitalmars.com/d/archives/digitalmars/D/ide/Future_of_Descent_and_D_Eclipse_IDE_635.html

     --
     Bruno Medeiros - Software Engineer

There is a project with an antlr D-grammar in work. http://code.google.com/p/vs-d-integration/ Maybe this can be finished? matthias
Nov 20 2010
prev sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 19/11/2010 23:56, Michael Stover wrote:
 so that was 4 months ago - how do things currently stand on that initiative?

 -Mike

 On Fri, Nov 19, 2010 at 6:37 PM, Bruno Medeiros
 <brunodomedeiros+spam com.gmail> wrote:

     On 19/11/2010 22:25, Michael Stover wrote:

         As for D lexers and tokenizers, what would be nice is to
         A) build an antlr grammar for D
         B) build D targets for antlr so that antlr can generate lexers and
         parsers in the D language.

         For B) I found http://www.mbutscher.de/antlrd/index.html

         For A) A good list of antlr grammars is at
         http://www.antlr.org/grammar/list, but there isn't a D grammar.

         These things wouldn't be an enormous amount of work to create and
         maintain, and, if done, anyone could parse D code in many languages,
         including Java and C which would make providing IDE features for D
         development easier in those languages (eclipse for instance),
         and you
         could build lexers and parsers in D using antlr grammars.

         -Mike


     Yes, that would be much better. It would be directly and immediately
     useful for the DDT project:

     "But better yet would be to start coding our own custom parser
     (using a parser generator like ANTLR for example), that could really
     be tailored for IDE needs. In the medium/long term, that's probably
     what needs to be done. "
     in
     http://www.digitalmars.com/d/archives/digitalmars/D/ide/Future_of_Descent_and_D_Eclipse_IDE_635.html

     --
     Bruno Medeiros - Software Engineer

I don't know about Ellery, as you can see in that thread he/she(?) mentioned interest in working on that, but I don't know anything more. As for me, I didn't work on that, nor did I plan to. Nor am I planning to anytime soon, DDT can handle things with the current parser for now (bugs can be fixed on the current code, perhaps some limitations can be resolved by merging some more code from DMD), so I'll likely work on other more important features before I go there. For example, I'll likely work on debugger integration, and code completion improvements before I would go on writing a new parser from scratch. Plus, it gives more time to hopefully someone else work on it. :P Unlike Walter, I can't write a D parser in a weekend... :) Not even on a week, especially since I never done anything of this kind before. -- Bruno Medeiros - Software Engineer
Nov 24 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 11/24/2010 09:13 AM, Bruno Medeiros wrote:
 I don't know about Ellery, as you can see in that thread he/she(?)
 mentioned interest in working on that, but I don't know anything more.

Normally I go by 'it'. Been pretty busy this semester, so I haven't been doing much. But the bottom line is, yes I have working antlr grammars for D1 and D2 if you don't mind 1) they're slow 2) they're tied to a hacked-out version of the netbeans fork of ANTLR2 3) they're tied to some custom java code 4) I haven't been keeping the tree grammars so up to date I've not released them for those reasons. Semester will be over in about 3 weeks, though, and I'll have time then.
 As for me, I didn't work on that, nor did I plan to.
 Nor am I planning to anytime soon, DDT can handle things with the
 current parser for now (bugs can be fixed on the current code, perhaps
 some limitations can be resolved by merging some more code from DMD), so
 I'll likely work on other more important features before I go there. For
 example, I'll likely work on debugger integration, and code completion
 improvements before I would go on writing a new parser from scratch.
 Plus, it gives more time to hopefully someone else work on it. :P

 Unlike Walter, I can't write a D parser in a weekend... :) Not even on a
 week, especially since I never done anything of this kind before.

It took me like 3 months to read his parser to figure out what was going on.
Nov 24 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 24/11/2010 16:19, Ellery Newcomer wrote:
 On 11/24/2010 09:13 AM, Bruno Medeiros wrote:
 I don't know about Ellery, as you can see in that thread he/she(?)
 mentioned interest in working on that, but I don't know anything more.

Normally I go by 'it'.

I didn't meant to offend or anything, I was just unsure of that. To me Ellery seems like a female name (but that can be a bias due to English not being my first language, or some other cultural thing). On the other hand, I would be surprised if a person of the female variety would be that interested in D, to the point of contributing in such way.
 Been pretty busy this semester, so I haven't been doing much.

 But the bottom line is, yes I have working antlr grammars for D1 and D2
 if you don't mind
 1) they're slow
 2) they're tied to a hacked-out version of the netbeans fork of ANTLR2
 3) they're tied to some custom java code
 4) I haven't been keeping the tree grammars so up to date

 I've not released them for those reasons. Semester will be over in about
 3 weeks, though, and I'll have time then.

Hum, doesn't sound like it might be suitable for DDT, but I wasn't counting on that either.
 As for me, I didn't work on that, nor did I plan to.
 Nor am I planning to anytime soon, DDT can handle things with the
 current parser for now (bugs can be fixed on the current code, perhaps
 some limitations can be resolved by merging some more code from DMD), so
 I'll likely work on other more important features before I go there. For
 example, I'll likely work on debugger integration, and code completion
 improvements before I would go on writing a new parser from scratch.
 Plus, it gives more time to hopefully someone else work on it. :P

 Unlike Walter, I can't write a D parser in a weekend... :) Not even on a
 week, especially since I never done anything of this kind before.

It took me like 3 months to read his parser to figure out what was going on.

Not 3 man-months for sure!, right? (Man-month in the sense of someone working 40 hours per week during a month.) -- Bruno Medeiros - Software Engineer
Nov 24 2010
next sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 11/24/2010 02:09 PM, Bruno Medeiros wrote:
 I didn't meant to offend or anything, I was just unsure of that.

None taken; I'm just laughing at you. As I understand it, though, 'Ellery' is a unisex name, so it is entirely ambiguous.
 It took me like 3 months to read his parser to figure out what was going
 on.

Not 3 man-months for sure!, right? (Man-month in the sense of someone working 40 hours per week during a month.)

Probably not
Nov 24 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Bruno Medeiros:

 On the other hand, I would be surprised if a person of the female variety
 would be that interested in D, to the point of contributing in such way.

In Python newsgroups I have seen few women, now and then, but in the D newsgroup so far... not many. So far D seems a male thing. I don't know why. At the university at the Computer Science course there are a good enough number of female students (and few female teachers too). Bye, bearophile
Nov 24 2010
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
bearophile schrieb:
 Bruno Medeiros:
 
 On the other hand, I would be surprised if a person of the female variety
 would be that interested in D, to the point of contributing in such way.

In Python newsgroups I have seen few women, now and then, but in the D newsgroup so far... not many. So far D seems a male thing. I don't know why. At the university at the Computer Science course there are a good enough number of female students (and few female teachers too). Bye, bearophile

At my university there are *very* few woman studying computer science. Most women sitting in CS lectures here are studying maths and have to do some basic CS lectures (I don't think they're the kind that would try D voluntarily). We have two female professors though.
Nov 24 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Daniel Gibson" <metalcaedes gmail.com> wrote in message 
news:icjv6l$p1r$2 digitalmars.com...
 bearophile schrieb:
 Bruno Medeiros:

 On the other hand, I would be surprised if a person of the female 
 variety
 would be that interested in D, to the point of contributing in such way.

In Python newsgroups I have seen few women, now and then, but in the D newsgroup so far... not many. So far D seems a male thing. I don't know why. At the university at the Computer Science course there are a good enough number of female students (and few female teachers too). Bye, bearophile

At my university there are *very* few woman studying computer science. Most women sitting in CS lectures here are studying maths and have to do some basic CS lectures (I don't think they're the kind that would try D voluntarily). We have two female professors though.

See, that's the #1 worst thing about the field of programming: Total sausage fest.
Nov 25 2010
prev sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 24/11/2010 21:12, Daniel Gibson wrote:
 bearophile schrieb:
 Bruno Medeiros:

 On the other hand, I would be surprised if a person of the female
 variety
 would be that interested in D, to the point of contributing in such way.

In Python newsgroups I have seen few women, now and then, but in the D newsgroup so far... not many. So far D seems a male thing. I don't know why. At the university at the Computer Science course there are a good enough number of female students (and few female teachers too). Bye, bearophile

At my university there are *very* few woman studying computer science. Most women sitting in CS lectures here are studying maths and have to do some basic CS lectures (I don't think they're the kind that would try D voluntarily). We have two female professors though.

It is well know that there is a big gender gap in CS with regards to students and professionals. Something like 5-20% I guess, depending on university, company, etc.. But the interesting thing (although also quite unfortunate), is that this gap takes a even greater dip downwards when you consider the communities of FOSS developers/contributors. It must be well below 1%! (note that I'm not talking about *users* of FOSS software, but only people who actually contribute code, whether for FOSS projects, or for their own indie/toy projects) -- Bruno Medeiros - Software Engineer
Nov 26 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Tue, 26 Oct 2010 19:32:44 -0700, Walter Bright wrote:

 Nick Sabalausky wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:i9qd8q$1ls4$1 digitalmars.com...
 4. the tokens should be a value type, not a reference type

I'm curious, is your reason for this purely to avoid allocations during lexing, or are there other reasons too?

It's one big giant reason. Storage allocation gets unbelievably costly in a lexer. Another is it makes tokens easy to copy. Another one is that classes are for polymorphic behavior. What kind of polymorphic behavior would one want with tokens?

This is why the basic data structure in functional languages, algebraic data types, suits better for this purpose.
Oct 27 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Tue, 26 Oct 2010 21:39:32 -0700, Walter Bright wrote:

 bearophile wrote:
 Walter:
 
 Java made a related mistake by failing to acknowledge that value types
 have any useful purpose at all (unless they are built-in).

Java was designed to be simple! Simple means to have a more uniform semantics.

So was Pascal. See the thread about how useless it was as a result.

Blablabla.. this nostalgic lesson reminded me, have you even started studying the list of type system concepts I listed few days ago. A new version with links is coming at some point of time.
 What I miss more in Java is not single structs (single values),

There's a lot more to miss than that. I find Java code tends to be excessively complex, and that's because it lacks expressive power.

Adding structs to Java wouldn't fix that. You probably know that. Unifying structs and classes in a language like D and adding good escape analysis wouldn't worsen the performance that badly in general purpose applications. Java is mostly used for general purpose programming so your claims about usefulness and the need for extreme performance look silly.
Oct 27 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Wed, 27 Oct 2010 12:08:19 -0700, Walter Bright wrote:

 retard wrote:
 This is why the basic data structure in functional languages, algebraic
 data types, suits better for this purpose.

I think you recently demonstrated otherwise, as proven by the widespread use of Java :-)

I don't understand your logic -- Widespread use of Java proves that algebraic data types aren't a better suited way for expressing compiler's data structures such as syntax trees?
Oct 27 2010
prev sibling next sibling parent reply retard <re tard.com.invalid> writes:
Wed, 27 Oct 2010 13:52:29 -0700, Walter Bright wrote:

 retard wrote:
 Wed, 27 Oct 2010 12:08:19 -0700, Walter Bright wrote:
 
 retard wrote:
 This is why the basic data structure in functional languages,
 algebraic data types, suits better for this purpose.

widespread use of Java :-)

I don't understand your logic -- Widespread use of Java proves that algebraic data types aren't a better suited way for expressing compiler's data structures such as syntax trees?

You told me that widespread use of Java proved that nothing more complex than what Java provides is useful: "Java is mostly used for general purpose programming so your claims about usefulness and the need for extreme performance look silly." I'd be surprised if you seriously meant that, as it implies that Java is the pinnacle of computer language design, but I can't resist teasing you about it. :-)

I only meant that the widespead adoption of Java shows how the public at large cares very little about the performance issues you mentioned. Java is one of the most widely used languages and it's also successful in many fields. Things could be better from programming language theory's point of view, but the business world is more interesting in profits and the large pool of Java coders has given better benefits than more expressive languages. I don't think that says anything against my notes about algebraic data types.
Oct 27 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"retard" <re tard.com.invalid> wrote in message 
news:iaa44v$17sf$2 digitalmars.com...
 I only meant that the widespead adoption of Java shows how the public at
 large cares very little about the performance issues you mentioned.

The public at large is convinced that "Java is fast now, really!". So I'm not certain widespread adoption of Java necessarily indicates they don't care so much about performance. Of course, Java is quickly becoming a legacy language anyway (the next COBOL, IMO), so that throws another wrench into the works.
Oct 27 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 27/10/2010 22:43, Nick Sabalausky wrote:
 "retard"<re tard.com.invalid>  wrote in message
 news:iaa44v$17sf$2 digitalmars.com...
 I only meant that the widespead adoption of Java shows how the public at
 large cares very little about the performance issues you mentioned.

The public at large is convinced that "Java is fast now, really!". So I'm not certain widespread adoption of Java necessarily indicates they don't care so much about performance. Of course, Java is quickly becoming a legacy language anyway (the next COBOL, IMO), so that throws another wrench into the works.

Java is quickly becoming a legacy language? the next COBOL? SRSLY?... Just two years ago, the now hugely popular Android platform choose Java as it's language of choice, and you think Java is becoming legacy?... The development of the Java language itself has stagnated over the last 6 years or so (especially due to corporate politics, which now has become even worse and uncertain with all the shit Oracle is doing), but that's a completely different statement from saying Java is becoming legacy. In fact, all the uproar and concern about the future of Java under Oracle, of the JVM, of the JCP (the body that regulates changes to Java),etc., is a testament to the huge popularity of Java. Otherwise people (and corporations) wouldn't care, they would just let it wither away with much less concern. -- Bruno Medeiros - Software Engineer
Nov 19 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Bruno Medeiros:

 Java is quickly becoming a legacy language? the next COBOL? SRSLY?...
 Just two years ago, the now hugely popular Android platform choose Java 
 as it's language of choice, and you think Java is becoming legacy?...

Java on Adroid is not going well, there is a Oracle->Google lawsuit in progress. Google is interested in using a variant of NaCL on Android too. Bye, bearophile
Nov 19 2010
prev sibling parent "Todd D. VanderVeen" <tdv part.net> writes:
Legacy in the sense that C is perhaps.

http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html



-----Original Message-----
From: digitalmars-d-bounces puremagic.com
[mailto:digitalmars-d-bounces puremagic.com] On Behalf Of Nick Sabalausky
Sent: Wednesday, October 27, 2010 3:43 PM
To: digitalmars-d puremagic.com
Subject: Re: Looking for champion - std.lang.d.lex

"retard" <re tard.com.invalid> wrote in message 
news:iaa44v$17sf$2 digitalmars.com...
 I only meant that the widespead adoption of Java shows how the public at
 large cares very little about the performance issues you mentioned.

The public at large is convinced that "Java is fast now, really!". So I'm not certain widespread adoption of Java necessarily indicates they don't care so much about performance. Of course, Java is quickly becoming a legacy language anyway (the next COBOL, IMO), so that throws another wrench into the works.
Oct 27 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Wed, 27 Oct 2010 14:15:04 -0700, Walter Bright wrote:

 retard wrote:
 I only meant that the widespead adoption of Java shows how the public
 at large cares very little about the performance issues you mentioned 
 Java is one of the most widely used languages and it's also successful
 in many fields. Things could be better from programming language
 theory's point of view, but the business world is more interesting in
 profits and the large pool of Java coders has given better benefits
 than more expressive languages. I don't think that says anything
 against my notes about algebraic data types.

Choice of a language has numerous factors

I know that.
, so you cannot dismiss one
 factor because the other factors still make it an attractive choice.

I don't think I said anything that contradicts that.
 For example:
 
 "the widespread adoption of horses shows how the public at large cares
 very little about the cars you mentioned."

I meant caring in a way that results in masses of programmers migrating their code from Java to a language with those performance issues solved (e.g. D). A layman can make general remarks from people switching from Java/C++/C to "new" languages such as Groovy, Javascript, Python, PHP, and Ruby. The people want "simpler" languages. For example Ruby has terrible performance, but the performance becomes a non-issue once the web service framework is built in a scalable way.
Oct 27 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Wed, 27 Oct 2010 16:04:34 -0600, Todd D. VanderVeen wrote:

 Legacy in the sense that C is perhaps.
 
 http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

Probably the top 10 names are more or less correct there, but some funny notes: 33. D 36. Scratch 40. Haskell 42. JavaFX Script 49. Scala Scratch is an educational tool. It isn't really suitable for any real world applications. It slows down considerably with too many expressions. There are several books about Haskell and Scala. Both have several books on them, active mailing lists, and also very many active community projects. Haven't heard much about JavaFX outside Sun/Oracle. These statistics look really weird.
Oct 27 2010
prev sibling next sibling parent Michael Stover <michael.r.stover gmail.com> writes:
--20cf3005dd4495329404956f6160
Content-Type: text/plain; charset=ISO-8859-1

As for D lexers and tokenizers, what would be nice is to
A) build an antlr grammar for D
B) build D targets for antlr so that antlr can generate lexers and parsers
in the D language.

For B) I found http://www.mbutscher.de/antlrd/index.html

For A) A good list of antlr grammars is at http://www.antlr.org/grammar/list,
but there isn't a D grammar.

These things wouldn't be an enormous amount of work to create and maintain,
and, if done, anyone could parse D code in many languages, including Java
and C which would make providing IDE features for D development easier in
those languages (eclipse for instance), and you could build lexers and
parsers in D using antlr grammars.

-Mike



On Fri, Nov 19, 2010 at 5:09 PM, Bruno Medeiros
<brunodomedeiros+spam com.gmail> wrote:

 On 27/10/2010 22:43, Nick Sabalausky wrote:

 "retard"<re tard.com.invalid>  wrote in message
 news:iaa44v$17sf$2 digitalmars.com...

 I only meant that the widespead adoption of Java shows how the public at
 large cares very little about the performance issues you mentioned.

The public at large is convinced that "Java is fast now, really!". So I'm not certain widespread adoption of Java necessarily indicates they don't care so much about performance. Of course, Java is quickly becoming a legacy language anyway (the next COBOL, IMO), so that throws another wrench into the works.

Just two years ago, the now hugely popular Android platform choose Java as it's language of choice, and you think Java is becoming legacy?... The development of the Java language itself has stagnated over the last 6 years or so (especially due to corporate politics, which now has become even worse and uncertain with all the shit Oracle is doing), but that's a completely different statement from saying Java is becoming legacy. In fact, all the uproar and concern about the future of Java under Oracle, of the JVM, of the JCP (the body that regulates changes to Java),etc., is a testament to the huge popularity of Java. Otherwise people (and corporations) wouldn't care, they would just let it wither away with much less concern. -- Bruno Medeiros - Software Engineer

--20cf3005dd4495329404956f6160 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable As for D lexers and tokenizers, what would be nice is to<div>A) build an an= tlr grammar for D</div><div>B) build D targets for antlr so that antlr can = generate lexers and parsers in the D language.</div><div><br></div><div> For B) I found=A0<a href=3D"http://www.mbutscher.de/antlrd/index.html">http= ://www.mbutscher.de/antlrd/index.html</a></div><div><br></div><div>For A) A= good list of antlr grammars is at=A0<a href=3D"http://www.antlr.org/gramma= r/list">http://www.antlr.org/grammar/list</a>, but there isn&#39;t a D gram= mar.</div> <div><br></div><div>These things wouldn&#39;t be an enormous amount of work= to create and maintain, and, if done, anyone could parse D code in many la= nguages, including Java and C which would make providing IDE features for D= development easier in those languages (eclipse for instance), and you coul= d build lexers and parsers in D using antlr grammars.</div> <div><br></div><div>-Mike</div><div><br></div><div><br><br><div class=3D"gm= ail_quote">On Fri, Nov 19, 2010 at 5:09 PM, Bruno Medeiros <span dir=3D"ltr= ">&lt;brunodomedeiros+spam com.gmail&gt;</span> wrote:<br><blockquote class= =3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd= ing-left:1ex;"> On 27/10/2010 22:43, Nick Sabalausky wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> &quot;retard&quot;&lt;re tard.com.invalid&gt; =A0wrote in message<br> news:iaa44v$17sf$2 digitalmars.com...<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <br> I only meant that the widespead adoption of Java shows how the public at<br=

</blockquote> <br> The public at large is convinced that &quot;Java is fast now, really!&quot;= . So I&#39;m<br> not certain widespread adoption of Java necessarily indicates they don&#39;= t<br> care so much about performance. Of course, Java is quickly becoming a legac= y<br> language anyway (the next COBOL, IMO), so that throws another wrench into<b= r> the works.<br> <br> <br> </blockquote> <br> Java is quickly becoming a legacy language? the next COBOL? SRSLY?...<br> Just two years ago, the now hugely popular Android platform choose Java as = it&#39;s language of choice, and you think Java is becoming legacy?...<br> <br> The development of the Java language itself has stagnated over the last 6 y= ears or so (especially due to corporate politics, which now has become even= worse and uncertain with all the shit Oracle is doing), but that&#39;s a c= ompletely different statement from saying Java is becoming legacy.<br> In fact, all the uproar and concern about the future of Java under Oracle, = of the JVM, of the JCP (the body that regulates changes to Java),etc., is a= testament to the huge popularity of Java. Otherwise people (and corporatio= ns) wouldn&#39;t care, they would just let it wither away with much less co= ncern.<br> <font color=3D"#888888"> <br> <br> -- <br> Bruno Medeiros - Software Engineer<br> </font></blockquote></div><br></div> --20cf3005dd4495329404956f6160--
Nov 19 2010
prev sibling next sibling parent reply Andrew Wiley <debio264 gmail.com> writes:
--001636c59a2e1a389f04956f669b
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Nov 19, 2010 at 4:20 PM, bearophile <bearophileHUGS lycos.com>wrote:

 Bruno Medeiros:

 Java is quickly becoming a legacy language? the next COBOL? SRSLY?...
 Just two years ago, the now hugely popular Android platform choose Java
 as it's language of choice, and you think Java is becoming legacy?...

Java on Adroid is not going well, there is a Oracle->Google lawsuit in progress. Google is interested in using a variant of NaCL on Android too.

I have to agree with Bruno here, Java isn't going anywhere soon. It has an active community, corporations that very actively support it, and an open source effort that's probably the largest of any language (check out the Apache Foundation project lists). Toss in Clojure, Scala, Groovy, and their friends that can build on top of Java libraries, and you wind up with a package that isn't becoming obsolete any time soon. --001636c59a2e1a389f04956f669b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Fri, Nov 19, 2010 at 4:20 PM, bearophile <spa= n dir=3D"ltr">&lt;<a href=3D"mailto:bearophileHUGS lycos.com">bearophileHUG= S lycos.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" styl= e=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> Bruno Medeiros:<br> <div class=3D"im"><br> &gt; Java is quickly becoming a legacy language? the next COBOL? SRSLY?...<= br> &gt; Just two years ago, the now hugely popular Android platform choose Jav= a<br> &gt; as it&#39;s language of choice, and you think Java is becoming legacy?= ...<br> <br> </div>Java on Adroid is not going well, there is a Oracle-&gt;Google lawsui= t in progress. Google is interested in using a variant of NaCL on Android t= oo.<br></blockquote><div><br></div><div>I have to agree with Bruno here, Ja= va isn&#39;t going anywhere soon. It has an active community, corporations = that very actively support it, and an open source effort that&#39;s probabl= y the largest of any language (check out the Apache Foundation project list= s). Toss in Clojure, Scala, Groovy, and their friends that can build on top= of Java libraries, and you wind up with a package that isn&#39;t becoming = obsolete any time soon.</div> </div> --001636c59a2e1a389f04956f669b--
Nov 19 2010
parent "Nick Sabalausky" <a a.a> writes:
"Andrew Wiley" <debio264 gmail.com> wrote in message 
news:mailman.501.1290205603.21107.digitalmars-d puremagic.com...
 On Fri, Nov 19, 2010 at 4:20 PM, bearophile 
 <bearophileHUGS lycos.com>wrote:

 Bruno Medeiros:

 Java is quickly becoming a legacy language? the next COBOL? SRSLY?...
 Just two years ago, the now hugely popular Android platform choose Java
 as it's language of choice, and you think Java is becoming legacy?...

Java on Adroid is not going well, there is a Oracle->Google lawsuit in progress. Google is interested in using a variant of NaCL on Android too.

I have to agree with Bruno here, Java isn't going anywhere soon. It has an active community, corporations that very actively support it, and an open source effort that's probably the largest of any language (check out the Apache Foundation project lists). Toss in Clojure, Scala, Groovy, and their friends that can build on top of Java libraries, and you wind up with a package that isn't becoming obsolete any time soon.

To be clear, I meant Java the language, not Java the VM. But yea, you're right, I probably overstated my point. What I have noticed though is, like Bruno said, a slowdown in Java language development and I can certainly imagine complications from the Oracle takeover of Sun. I've also been noticing decreasing interest in using Java (the language) for new projects (although, yes, not *completely* stalled interest) compared to 5-10 years ago, sharply increased awareness and recognition of Java's drawbacks compared to 5-10 years ago, and greatly increased interest and usage of other JVM languages besides Java. Ten years from now, I wouldn't at all be surprised to see Java (the language) being used primarily for maintenance on existing software that had already been written in Java. In fact, I'd be surprised if that doesn't end up being the case at that point. But I do imagine seeing things like D, Nemerle, Scala and Python thriving at that point.
Nov 23 2010
prev sibling parent Michael Stover <michael.r.stover gmail.com> writes:
--0015175caae652baea049570a669
Content-Type: text/plain; charset=ISO-8859-1

so that was 4 months ago - how do things currently stand on that initiative?

-Mike

On Fri, Nov 19, 2010 at 6:37 PM, Bruno Medeiros
<brunodomedeiros+spam com.gmail> wrote:

 On 19/11/2010 22:25, Michael Stover wrote:

 As for D lexers and tokenizers, what would be nice is to
 A) build an antlr grammar for D
 B) build D targets for antlr so that antlr can generate lexers and
 parsers in the D language.

 For B) I found http://www.mbutscher.de/antlrd/index.html

 For A) A good list of antlr grammars is at
 http://www.antlr.org/grammar/list, but there isn't a D grammar.

 These things wouldn't be an enormous amount of work to create and
 maintain, and, if done, anyone could parse D code in many languages,
 including Java and C which would make providing IDE features for D
 development easier in those languages (eclipse for instance), and you
 could build lexers and parsers in D using antlr grammars.

 -Mike

Yes, that would be much better. It would be directly and immediately useful for the DDT project: "But better yet would be to start coding our own custom parser (using a parser generator like ANTLR for example), that could really be tailored for IDE needs. In the medium/long term, that's probably what needs to be done. " in http://www.digitalmars.com/d/archives/digitalmars/D/ide/Future_of_Descent_and_D_Eclipse_IDE_635.html -- Bruno Medeiros - Software Engineer

--0015175caae652baea049570a669 Content-Type: text/html; charset=ISO-8859-1 so that was 4 months ago - how do things currently stand on that initiative?<div><br></div><div>-Mike<br><br><div class="gmail_quote">On Fri, Nov 19, 2010 at 6:37 PM, Bruno Medeiros <span dir="ltr">&lt;brunodomedeiros+spam com.gmail&gt;</span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On 19/11/2010 22:25, Michael Stover wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> As for D lexers and tokenizers, what would be nice is to<br> A) build an antlr grammar for D<br> B) build D targets for antlr so that antlr can generate lexers and<br> parsers in the D language.<br> <br> For B) I found <a href="http://www.mbutscher.de/antlrd/index.html" target="_blank">http://www.mbutscher.de/antlrd/index.html</a><br> <br> For A) A good list of antlr grammars is at<br> <a href="http://www.antlr.org/grammar/list" target="_blank">http://www.antlr.org/grammar/list</a>, but there isn&#39;t a D grammar.<br> <br> These things wouldn&#39;t be an enormous amount of work to create and<br> maintain, and, if done, anyone could parse D code in many languages,<br> including Java and C which would make providing IDE features for D<br> development easier in those languages (eclipse for instance), and you<br> could build lexers and parsers in D using antlr grammars.<br> <br> -Mike<br> </blockquote> <br></div> Yes, that would be much better. It would be directly and immediately useful for the DDT project:<br> <br> &quot;But better yet would be to start coding our own custom parser (using a parser generator like ANTLR for example), that could really be tailored for IDE needs. In the medium/long term, that&#39;s probably what needs to be done. &quot;<br> in <a href="http://www.digitalmars.com/d/archives/digitalmars/D/ide/Future_of_Descent_and_D_E lipse_IDE_635.html" target="_blank">http://www.digitalmars.com/d/archives/digitalmars/D/ide/Future_of_Descent_and_D_Eclipse_IDE_635.html</a><br> <font color="#888888"> <br> -- <br> Bruno Medeiros - Software Engineer<br> </font></blockquote></div><br></div> --0015175caae652baea049570a669--
Nov 19 2010
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, November 19, 2010 13:53:12 Bruno Medeiros wrote:
 On 19/11/2010 21:27, Jonathan M Davis wrote:
 On Friday 19 November 2010 13:03:53 Bruno Medeiros wrote:
 On 22/10/2010 20:48, Andrei Alexandrescu wrote:
 On 10/22/10 14:02 CDT, Tomek Sowi=C5=84ski wrote:
 Dnia 22-10-2010 o 00:01:21 Walter Bright<newshound2 digitalmars.com>
=20
 napisa=C5=82(a):
 As we all know, tool support is important for D's success. Making
 tools easier to build will help with that.
=20
 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax
 highlighting filters, pretty printers, repl, doc generators, static
 analyzers, and even D compilers.
=20
 It should:
=20
 1. support a range interface for its input, and a range interface f=






 its output
 2. optionally not generate lexical errors, but just try to recover
 and continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be
 maintained in tandem
=20
 It can also serve as the basis for creating a javascript
 implementation that can be embedded into web pages for syntax
 highlighting, and eventually an std.lang.d.parse.
=20
 Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORB=





 Thrift, etc, so lexers will have to be written all over to grok
 interface files. Perhaps a generic tokenizer which can be parametriz=





 with a lexical grammar would bring more ROI, I got a hunch D's
 templates are strong enough to pull this off without any source code
 generation ala JavaCC. The books I read on compilers say tokenization
 is a solved problem, so the theory part on what a good abstraction
 should be is done. What you think?

Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokeniz=




 generator.

Agreed, of all the things desired for D, a D tokenizer would rank pret=



 low I think.
=20
 Another thing, even though a tokenizer generator would be much more
 desirable, I wonder if it is wise to have that in the standard library?
 It does not seem to be of wide enough interest to be in a standard
 library. (Out of curiosity, how many languages have such a thing in
 their standard library?)

We want to make it easy for tools to be built to work on and deal with D code. An IDE, for example, needs to be able to tokenize and parse D code. A program like lint needs to be able to tokenize and parse D code. By providing a lexer and parser in the standard library, we are making it far easier for such tools to be written, and they could be of major benefit to the D community. Sure, the average program won't need to lex or parse D, but some will, and making it easy to do will make it a lot easier for such programs to be written. =20 - Jonathan M Davis

And by providing a lexer and a parser outside the standard library, wouldn't it make it just as easy for those tools to be written? What's the advantage of being in the standard library? I see only disadvantages: to begin with it potentially increases the time that Walter or other Phobos contributors may have to spend on it, even if it's just reviewing patches or making sure the code works.

If nothing, else, it makes it easier to keep in line with dmd itself. Since= the=20 dmd front end is LGPL, it's not possible to have a Boost port of it (like t= he=20 Phobos version will be) without Walter's consent. And I'd be surprised if h= e did=20 that for a third party library (though he seems to be pretty open on a lot = of=20 that kind of stuff). Not to mention, Walter and the core developers are _ex= actly_=20 the kind of people that you want working on a lexer or parser of the langua= ge=20 itself, because they're the ones who work on it. =2D Jonathan M Davis
Nov 19 2010
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, November 19, 2010 15:17:35 Bruno Medeiros wrote:
 On 19/11/2010 22:02, Jonathan M Davis wrote:
 On Friday, November 19, 2010 13:53:12 Bruno Medeiros wrote:
 On 19/11/2010 21:27, Jonathan M Davis wrote:
 
 And by providing a lexer and a parser outside the standard library,
 wouldn't it make it just as easy for those tools to be written? What's
 the advantage of being in the standard library? I see only
 disadvantages: to begin with it potentially increases the time that
 Walter or other Phobos contributors may have to spend on it, even if
 it's just reviewing patches or making sure the code works.

If nothing, else, it makes it easier to keep in line with dmd itself. Since the dmd front end is LGPL, it's not possible to have a Boost port of it (like the Phobos version will be) without Walter's consent. And I'd be surprised if he did that for a third party library (though he seems to be pretty open on a lot of that kind of stuff). Not to mention, Walter and the core developers are _exactly_ the kind of people that you want working on a lexer or parser of the language itself, because they're the ones who work on it. - Jonathan M Davis

Eh? That license argument doesn't make sense: if the lexer and parser were to be based on DMD itself, then putting it in the standard library is equivalent (in licensing terms) to licensing the lexer and parser parts of DMD in Boost. More correctly, what I mean by equivalent, is that there no reason why Walter would allow one thing and not the other... (because on both cases he would have to issue that license)

It's very different to have D implementation of something - which is based on a C++ version but definitely different in some respects - be under Boost and generally available, and having the C++ implementation be under Boost - particularly when the C++ version covers far more than just a lexer and parser. Someone _could_ port the D code back to C++ and have that portion useable under Boost, but that's a lot more work than just taking the C++ code and using it, and it's only the portions of the compiler which were ported to D to which could be re-used that way. And since the Boost code could be used in a commercial product while the LGPL is more restricted, it could make a definite difference. I'm not a licensing expert, and I'm not an expert on what Walter does and doesn't want done with his code, but he put the compiler front end under the LGPL, not Boost, and he's given his permission to have the lexer alone ported to D and put under the Boost license in the standard library, which is very different from putting the entire front end under Boost. I expect that the parser will follow eventually, but even if it does, that's still not the entire front end. So, there is a difference in licenses does have a real impact. And no one can take the LGPL C++ code and port it to D - for the standard library or otherwise - without Walter's permission, because its his copyright on the code. As for the usefulness of a D lexer and parser, I've already had several programs or functions which I've wanted to write which would require it, and the lack has made them infeasible. For instance, I was considering posting a version of my datetime code without the unit tests in it, so that it would be easier to read the actual code (given the large number of unit tests), but I found that to accurately do that, you need a lexer for D, so I gave up on it for the time being. Having a function which stripped out unnecessary whitespace (and especially newlines) for string mixins would be great (particularly since line numbers get messed up with multi-line string mixins), but that would require CTFE-able D lexer to work correctly (though you might be able to hack together something which would mostly work), which we don't have. The D lexer won't be CTFE-able initially (though hopefully it will be once the CTFE capabilites of dmd improve), so you still won't be able to do that once the lexer is done, but it is a case where a lexer would be useful. Ideally, Phobos would be huge in a manner similar to how C# or Java's libraries are huge. It will take time to get there, and we'll need more developers, but I don't think that it really makes sense to not put things in the standard library because it might take more dev time - particularly when a D lexer is the sort of thing that likely won't need much changing once it's done, since it would only need to be changed when the language changed or when a bug with it was found (which would likely equate to a bug in the compiler anyway), so ultimately, the developer cost is likely fairly low. Additionally, Walter thinks that the development costs will be lower to have it in the standard library with an implementation similar to dmd's rather than having it separate. And it's his call. So, it's going to get done. There are several people around here who lament the lack of D parser in Phobos at least periodically, and I think that it will be good to have an appropriate lexer and parser for D in Phobos. Having other 3rd party stuff - like antlr - is great too, but that's no reason not to put it in the standard library. I think that we're just going to have to agree to disagree on this one. - Jonathan M Davis
Nov 19 2010
prev sibling parent reply dolive <dolive89 sina.com> writes:
Walter Bright 写到:

 As we all know, tool support is important for D's success. Making tools easier 
 to build will help with that.
 
 To that end, I think we need a lexer for the standard library -
std.lang.d.lex. 
 It would be helpful in writing color syntax highlighting filters, pretty 
 printers, repl, doc generators, static analyzers, and even D compilers.
 
 It should:
 
 1. support a range interface for its input, and a range interface for its
output
 2. optionally not generate lexical errors, but just try to recover and continue
 3. optionally return comments and ddoc comments as tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be maintained in
tandem
 
 It can also serve as the basis for creating a javascript implementation that
can 
 be embedded into web pages for syntax highlighting, and eventually an 
 std.lang.d.parse.
 
 Anyone want to own this?

intense support! Someone to do it?
Feb 26 2011
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 26 February 2011 02:06:18 dolive wrote:
 Walter Bright =D0=B4=B5=BD:
 As we all know, tool support is important for D's success. Making tools
 easier to build will help with that.
=20
 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax highlighting
 filters, pretty printers, repl, doc generators, static analyzers, and
 even D compilers.
=20
 It should:
=20
 1. support a range interface for its input, and a range interface for i=


 output 2. optionally not generate lexical errors, but just try to
 recover and continue 3. optionally return comments and ddoc comments as
 tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be maintain=


 in tandem
=20
 It can also serve as the basis for creating a javascript implementation
 that can be embedded into web pages for syntax highlighting, and
 eventually an std.lang.d.parse.
=20
 Anyone want to own this?

intense support! Someone to do it?

I'm working on it, but I have enough else going on right now that I'm no be= ing=20 very quick about it. I don't know when it will be done. =2D Jonathan M Davis
Feb 26 2011
parent dolive <dolive89 sina.com> writes:
Jonathan M Davis 写到:

 On Saturday 26 February 2011 02:06:18 dolive wrote:
 Walter Bright 写到:
 As we all know, tool support is important for D's success. Making tools
 easier to build will help with that.
 
 To that end, I think we need a lexer for the standard library -
 std.lang.d.lex. It would be helpful in writing color syntax highlighting
 filters, pretty printers, repl, doc generators, static analyzers, and
 even D compilers.
 
 It should:
 
 1. support a range interface for its input, and a range interface for its
 output 2. optionally not generate lexical errors, but just try to
 recover and continue 3. optionally return comments and ddoc comments as
 tokens
 4. the tokens should be a value type, not a reference type
 5. generally follow along with the C++ one so that they can be maintained
 in tandem
 
 It can also serve as the basis for creating a javascript implementation
 that can be embedded into web pages for syntax highlighting, and
 eventually an std.lang.d.parse.
 
 Anyone want to own this?

intense support! Someone to do it?

I'm working on it, but I have enough else going on right now that I'm no being very quick about it. I don't know when it will be done. - Jonathan M Davis

thanks, make an all out effort !
Feb 26 2011