www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - So... let's document dmd

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
https://github.com/D-Programming-Language/dlang.org/pull/1196
https://github.com/D-Programming-Language/dmd/pull/5352

Destroy!!

Andrei
Jan 14 2016
next sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Friday, 15 January 2016 at 04:53:16 UTC, Andrei Alexandrescu 
wrote:
 https://github.com/D-Programming-Language/dlang.org/pull/1196
 https://github.com/D-Programming-Language/dmd/pull/5352

 Destroy!!

 Andrei
Since dmd is now in D, is there any reason to not allow people to import from it while using dmd? Wouldn't it make anyone's life who wanted to parse D code easier by just doing import ddmd.parser; (or what ever it is) and use the actual parser?
Jan 15 2016
parent reply Joakim <dlang joakim.fea.st> writes:
On Friday, 15 January 2016 at 23:58:32 UTC, Jack Stouffer wrote:
 On Friday, 15 January 2016 at 04:53:16 UTC, Andrei Alexandrescu 
 wrote:
 https://github.com/D-Programming-Language/dlang.org/pull/1196
 https://github.com/D-Programming-Language/dmd/pull/5352

 Destroy!!

 Andrei
Since dmd is now in D, is there any reason to not allow people to import from it while using dmd? Wouldn't it make anyone's life who wanted to parse D code easier by just doing import ddmd.parser; (or what ever it is) and use the actual parser?
The ddmd lexer is already on dub, just have to add the parser next: http://code.dlang.org/packages/ddmd
Jan 15 2016
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 16 January 2016 at 05:11:57 UTC, Joakim wrote:
 The ddmd lexer is already on dub, just have to add the parser 
 next:

 http://code.dlang.org/packages/ddmd
You missed the main point of my question.
Jan 15 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Saturday, 16 January 2016 at 05:29:03 UTC, Jack Stouffer wrote:
 On Saturday, 16 January 2016 at 05:11:57 UTC, Joakim wrote:
 The ddmd lexer is already on dub, just have to add the parser 
 next:

 http://code.dlang.org/packages/ddmd
You missed the main point of my question.
I'm confused as to your point as well. Perhaps you could rephrase?
Jan 15 2016
next sibling parent Joakim <dlang joakim.fea.st> writes:
On Saturday, 16 January 2016 at 05:36:23 UTC, tsbockman wrote:
 On Saturday, 16 January 2016 at 05:29:03 UTC, Jack Stouffer 
 wrote:
 On Saturday, 16 January 2016 at 05:11:57 UTC, Joakim wrote:
 The ddmd lexer is already on dub, just have to add the parser 
 next:

 http://code.dlang.org/packages/ddmd
You missed the main point of my question.
I'm confused as to your point as well. Perhaps you could rephrase?
I think he's trying to emphasize that he'd like it to come with dmd. There was talk of putting the lexer/parser into phobos, but I believe dub was chosen as a staging ground for people to play with it first. So if you want to use the lexer/parser, use it from dub for now, then push for its inclusion into phobos later.
Jan 15 2016
prev sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 16 January 2016 at 05:36:23 UTC, tsbockman wrote:
 On Saturday, 16 January 2016 at 05:29:03 UTC, Jack Stouffer 
 wrote:
 On Saturday, 16 January 2016 at 05:11:57 UTC, Joakim wrote:
 The ddmd lexer is already on dub, just have to add the parser 
 next:

 http://code.dlang.org/packages/ddmd
You missed the main point of my question.
I'm confused as to your point as well. Perhaps you could rephrase?
dmd is now in D; theoretically that should allow for other projects to import from it like a normal D project. So why not make all of the ddmd modules available from any code that is complied with it, i.e. just like Phobos? The parser was just an example, there are any number of things that one could use from the code base.
Jan 15 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Saturday, 16 January 2016 at 07:52:11 UTC, Jack Stouffer wrote:
 dmd is now in D; theoretically that should allow for other 
 projects to import from it like a normal D project. So why not 
 make all of the ddmd modules available from any code that is 
 complied with it, i.e. just like Phobos?

 The parser was just an example, there are any number of things 
 that one could use from the code base.
That would be cool in theory, but it would likely become a major drag on development of the compiler itself. Once you start promoting the compiler's internals as part of the standard library, you can expect sooner or later that pressure will mount to bring them up to the same high standards of API design and stability as Phobos. While this process would be beneficial to the overall quality of the DMD code being exposed, it would also turn internal refactorings into public breaking changes. I really doubt that's a good trade-off at this point in DMD's development.
Jan 16 2016
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Jan 16, 2016 at 08:35:40AM +0000, tsbockman via Digitalmars-d wrote:
 On Saturday, 16 January 2016 at 07:52:11 UTC, Jack Stouffer wrote:
dmd is now in D; theoretically that should allow for other projects
to import from it like a normal D project. So why not make all of the
ddmd modules available from any code that is complied with it, i.e.
just like Phobos?

The parser was just an example, there are any number of things that
one could use from the code base.
That would be cool in theory, but it would likely become a major drag on development of the compiler itself. Once you start promoting the compiler's internals as part of the standard library, you can expect sooner or later that pressure will mount to bring them up to the same high standards of API design and stability as Phobos. While this process would be beneficial to the overall quality of the DMD code being exposed, it would also turn internal refactorings into public breaking changes. I really doubt that's a good trade-off at this point in DMD's development.
I disagree. I think having the dmd itself (lexer, parser, etc.) as a library (with the dmd executable merely being the default frontend) will do D a lot of good. For one thing, IDE's will no longer need to reinvent a D parser for the purposes of syntax highlighting; they can reuse the actual parser the compiler itself uses, and thus actually be correct (whereas their own parser may actually only parse a subset of D correctly). They will also be able to support instant feedback as the user types in D code (highlight syntax errors on the fly, etc.). This will also help with writing various tools that operate on D code, e.g., pretty-printers like dfmt won't need to roll its own lexer/parser anymore. Many other tools also become possible (automatic refactorings, function call tree analysis, dependency analysis, etc.). You could even use libdmd in your own D code, thus being able to have the equivalent of the `eval` available in many scripting languages (except that it will have native performance(!)). The problem of dmd refactorings becoming public breaking changes is not as big a problem if the public API is properly designed. If the compiler code itself is refactored to interface between its sub-components via cleanly-designed APIs with proper encapsulations, I don't see that it will become prohibitively difficult to minimize public API changes without hampering compiler development efforts. This is the way to go for better compiler code quality anyway, so even if libdmd never happens, the code should *still* be refactored in this way. Besides, I don't think anyone is sanely suggesting that we allow dmd source files to be free-for-all externally-importable modules. The more likely scenario is that there will be a clearly-defined external interface that is unlikely to change much, whereas the implementation underneath is still free to be changed, refactored, reorganized, rewritten, etc.. T -- BREAKFAST.COM halted...Cereal Port Not Responding. -- YHL
Jan 16 2016
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Saturday, 16 January 2016 at 15:13:28 UTC, H. S. Teoh wrote:
 On Sat, Jan 16, 2016 at 08:35:40AM +0000, tsbockman via 
 Digitalmars-d wrote:
 On Saturday, 16 January 2016 at 07:52:11 UTC, Jack Stouffer 
 wrote:
dmd is now in D; theoretically that should allow for other 
projects to import from it like a normal D project. So why 
not make all of the ddmd modules available from any code that 
is complied with it, i.e. just like Phobos?

The parser was just an example, there are any number of 
things that one could use from the code base.
That would be cool in theory, but it would likely become a major drag on development of the compiler itself. Once you start promoting the compiler's internals as part of the standard library, you can expect sooner or later that pressure will mount to bring them up to the same high standards of API design and stability as Phobos. While this process would be beneficial to the overall quality of the DMD code being exposed, it would also turn internal refactorings into public breaking changes. I really doubt that's a good trade-off at this point in DMD's development.
I disagree. I think having the dmd itself (lexer, parser, etc.) as a library (with the dmd executable merely being the default frontend) will do D a lot of good. For one thing, IDE's will no longer need to reinvent a D parser for the purposes of syntax highlighting; they can reuse the actual parser the compiler itself uses, and thus actually be correct (whereas their own parser may actually only parse a subset of D correctly). They will also be able to support instant feedback as the user types in D code (highlight syntax errors on the fly, etc.). This will also help with writing various tools that operate on D code, e.g., pretty-printers like dfmt won't need to roll its own lexer/parser anymore. Many other tools also become possible (automatic refactorings, function call tree analysis, dependency analysis, etc.). You could even use libdmd in your own D code, thus being able to have the equivalent of the `eval` available in many scripting languages (except that it will have native performance(!)). The problem of dmd refactorings becoming public breaking changes is not as big a problem if the public API is properly designed. If the compiler code itself is refactored to interface between its sub-components via cleanly-designed APIs with proper encapsulations, I don't see that it will become prohibitively difficult to minimize public API changes without hampering compiler development efforts. This is the way to go for better compiler code quality anyway, so even if libdmd never happens, the code should *still* be refactored in this way. Besides, I don't think anyone is sanely suggesting that we allow dmd source files to be free-for-all externally-importable modules. The more likely scenario is that there will be a clearly-defined external interface that is unlikely to change much, whereas the implementation underneath is still free to be changed, refactored, reorganized, rewritten, etc..
While this would all be nice in principle, the reality is that dmd is mostly worked on by two people these days (https://github.com/D-Programming-Language/dmd/graphs/contributors) and they're unlikely to refactor ddmd to put forth a libdmd. It will depend on someone caring enough to take it up, as Daniel did with the port of the frontend to D. As tsbockman said, I don't think we're at the stage where anybody will put in that much effort into this.
Jan 16 2016
parent Daniel Murphy <yebbliesnospam gmail.com> writes:
On 17/01/2016 2:29 AM, Joakim wrote:
 While this would all be nice in principle, the reality is that dmd is
 mostly worked on by two people these days
 (https://github.com/D-Programming-Language/dmd/graphs/contributors) and
 they're unlikely to refactor ddmd to put forth a libdmd.  It will depend
 on someone caring enough to take it up, as Daniel did with the port of
 the frontend to D.  As tsbockman said, I don't think we're at the stage
 where anybody will put in that much effort into this.
I've already spent a huge amount of time on refactoring the frontend, there's just so much more to do. More important than allowing new third-party uses of the frontend, is that well defined interfaces will make maintaining GDC and LDC less tricky. Having to maintain a stable API would probably hurt more than it helps at this point.
Jan 16 2016
prev sibling next sibling parent tsbockman <thomas.bockman gmail.com> writes:
On Saturday, 16 January 2016 at 15:13:28 UTC, H. S. Teoh wrote:
 I disagree. I think having the dmd itself (lexer, parser, etc.) 
 as a
 library (with the dmd executable merely being the default 
 frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D parser 
 for the purposes of syntax highlighting; they can reuse the 
 actual parser the compiler itself uses, and thus actually be 
 correct (whereas their own parser may actually only parse a 
 subset of D correctly).  They will also be able to support 
 instant feedback as the user types in D code (highlight syntax 
 errors on the fly, etc.).

 This will also help with writing various tools that operate on 
 D code, e.g., pretty-printers like dfmt won't need to roll its 
 own lexer/parser anymore. Many other tools also become possible 
 (automatic refactorings, function call tree analysis, 
 dependency analysis, etc.).

 You could even use libdmd in your own D code, thus being able 
 to have
 the equivalent of the `eval` available in many scripting 
 languages
 (except that it will have native performance(!)).

 The problem of dmd refactorings becoming public breaking 
 changes is not as big a problem if the public API is properly 
 designed. If the compiler code itself is refactored to 
 interface between its sub-components via cleanly-designed APIs 
 with proper encapsulations, I don't see that it will become 
 prohibitively difficult to minimize public API changes without 
 hampering compiler development efforts. This is the way to go 
 for better compiler code quality anyway, so even if libdmd 
 never happens, the code should *still* be refactored in this 
 way.

 Besides, I don't think anyone is sanely suggesting that we 
 allow dmd source files to be free-for-all externally-importable 
 modules. The more likely scenario is that there will be a 
 clearly-defined external interface that is unlikely to change 
 much, whereas the implementation underneath is still free to be 
 changed, refactored, reorganized, rewritten, etc..


 T
You're sort of making my point for me. I fully agree about all the benefits of making the lexer, parser, etc. available as a library. HOWEVER, as you say, this would first require a large refactoring of the code, to make it suitable for use as a library. Time spent on that refactoring is time that could have been spent improving the compiler in other ways. D is short of development resources as it is; I think fixing the many bugs in the compiler and fleshing out important complex features like C++ inter-op is of more value to the language in the long run. Also worth mentioning, is that the SDC project was actually designed from the beginning for the use you're advocating. I'm sure deadalnix would love to have more help with that...
Jan 16 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/16/2016 7:13 AM, H. S. Teoh via Digitalmars-d wrote:
 I disagree. I think having the dmd itself (lexer, parser, etc.) as a
 library (with the dmd executable merely being the default frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D parser for the
 purposes of syntax highlighting;
On the other hand, using lexer.d and parse.d as a guide to build your own is a trivial undertaking. The Boost license is designed so this can be done without worrying about making a derived work. I looked into doing syntax highlighting for my editor, MicroEmacs. It turns out it is not so easy to just use a compiler lexer/parser for it. For one thing, the one used in the compiler is optimized for speed in a forward pass through the text. But a syntax highlighter in a text editor is different. Suppose I change a character in the middle of a line. All the highlighting from that point forward may change. And to figure out what that change is, the parser/lexer has to start over from the beginning of the file! (Think string literals, nested comments, quoted string literals, etc.) This would make editing slow. I surmised that a solution is to have each line in the editor be tagged with a state to show the lexing state at the beginning of that line. Then, when a character in a line changes, the lexer can be restarted from that state, and it continues forward until the next line state matches the new computed state. This would make it enormously faster. But the compiler's lexer is not designed to be restartable in the middle. Similarly, a source code formatter would be different in a different way. It would need, for example, extra information about comments, token start/end positions, etc. Such data collection would be irrelevant to a compiler, and would slow it down and consume unneeded memory.
Apr 05 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 5 April 2016 at 08:46:30 UTC, Walter Bright wrote:
 On 1/16/2016 7:13 AM, H. S. Teoh via Digitalmars-d wrote:
 I disagree. I think having the dmd itself (lexer, parser, 
 etc.) as a
 library (with the dmd executable merely being the default 
 frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D 
 parser for the
 purposes of syntax highlighting;
On the other hand, using lexer.d and parse.d as a guide to build your own is a trivial undertaking. The Boost license is designed so this can be done without worrying about making a derived work. I looked into doing syntax highlighting for my editor, MicroEmacs. It turns out it is not so easy to just use a compiler lexer/parser for it. For one thing, the one used in the compiler is optimized for speed in a forward pass through the text. But a syntax highlighter in a text editor is different. Suppose I change a character in the middle of a line. All the highlighting from that point forward may change. And to figure out what that change is, the parser/lexer has to start over from the beginning of the file! (Think string literals, nested comments, quoted string literals, etc.) This would make editing slow.
This is how CE highlither works. The lexer used to highlight processes line by line. For each line infos about the previous line are available (nested comments count, region kind like quoted string, raw quoted string, asm, etc). Also keyword detection might use different dictionaries, up to 3 (one for the keywords, another for special keywords like __FILE__, a third for asm opcodes). Operators doesn't need to have a special token (tokxorequ, toplusplus, etc) there's just one. Also lexing number doesn't need to be as accurate as the front-end of the compiler (especially if the HL doesnt have a token type for the illegal "lexem". Another big difference is that the lexer used by an highlighter doesn't store the identifier associated to a token. Using the front-end (or even libdparse) would require a modularisation. But lexing is not hard so I don't think it's worse. At last it would only be used by 4 or 5 softwares.
Apr 05 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/5/2016 6:47 AM, Basile B. wrote:
 Also lexing number doesn't need to be as accurate as the
 front-end of the compiler (especially if the HL doesnt have a token type for
the
 illegal "lexem".
That is an interesting design point. If I was doing a highlighter, I'd highlight in red tokens that the compiler would reject, meaning I'd do the accurate number lexing. Lexing numbers correctly is not trivial, but since the compiler lexer's implementation can be cut/pasted, it is trivial in practice.
Apr 05 2016
parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 5 April 2016 at 21:37:09 UTC, Walter Bright wrote:
 On 4/5/2016 6:47 AM, Basile B. wrote:
 Also lexing number doesn't need to be as accurate as the
 front-end of the compiler (especially if the HL doesnt have a 
 token type for the
 illegal "lexem".
That is an interesting design point. If I was doing a highlighter, I'd highlight in red tokens that the compiler would reject, meaning I'd do the accurate number lexing. Lexing numbers correctly is not trivial, but since the compiler lexer's implementation can be cut/pasted, it is trivial in practice.
Even if when the most naive lexer see a number and consumes until a blank, a symbol or an operator, it's clear that this can be done: http://i.imgur.com/ehjps04.png Actually numbers is the only part of the D lexer where errors can be detected. There's no possible syntax errors otherwise. But one thing I forget to say in my previous post is that lexing can be "multi-pass". The D front-end does everything in a single pass, for example it direclty detects tokPlusPlus or tokXorEqu, but actually a multi pass lexer can work in 3 sub phases: 1/ split words 2/ detects token families in the words; identifier, keyword, operator, etc. 3/ specialize tokens: tokOp.data == "++" -> tokPlusPlus
Apr 05 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/5/2016 7:59 PM, Basile B. wrote:
 Actually numbers is the only part of the D lexer where errors can be detected.
 There's no possible syntax errors otherwise.
D's lexer.d detects many errors. Search lexer.d for "error".
Apr 05 2016
prev sibling parent Basile B. <b2.temp gmx.com> writes:
On Tuesday, 5 April 2016 at 13:47:55 UTC, Basile B. wrote:
 [...]
 by 4 or 5 softwares.
Believe or not, in 2012 I've registered a company with the same error (<stuff>.softwares). I don't know why...it hasnot been a commercial success.
Apr 05 2016
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05.04.2016 10:46, Walter Bright wrote:
 On 1/16/2016 7:13 AM, H. S. Teoh via Digitalmars-d wrote:
 I disagree. I think having the dmd itself (lexer, parser, etc.) as a
 library (with the dmd executable merely being the default frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D parser for the
 purposes of syntax highlighting;
On the other hand, using lexer.d and parse.d as a guide to build your own is a trivial undertaking. The Boost license is designed so this can be done without worrying about making a derived work. I looked into doing syntax highlighting for my editor, MicroEmacs. It turns out it is not so easy to just use a compiler lexer/parser for it. For one thing, the one used in the compiler is optimized for speed in a forward pass through the text. But a syntax highlighter in a text editor is different. Suppose I change a character in the middle of a line. All the highlighting from that point forward may change. And to figure out what that change is, the parser/lexer has to start over from the beginning of the file! (Think string literals, nested comments, quoted string literals, etc.) This would make editing slow. ...
Also, tools might want to parse a more intuitive grammar in order to be able to give suggestions on how to adapt the code such that DMDs parser will accept it.
 I surmised that a solution is to have each line in the editor be tagged
 with a state to show the lexing state at the beginning of that line.
 Then, when a character in a line changes, the lexer can be restarted
 from that state, and it continues forward until the next line state
 matches the new computed state. This would make it enormously faster.
 ...
Some additional care will need to be taken, e.g. /+ <backspace> <backspace> or even simply /+ comment +/
 But the compiler's lexer is not designed to be restartable in the middle.

 Similarly, a source code formatter would be different in a different
 way. It would need, for example, extra information about comments, token
 start/end positions, etc. Such data collection would be irrelevant to a
 compiler, and would slow it down and consume unneeded memory.
Apr 05 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/5/2016 4:08 PM, Timon Gehr wrote:
 Some additional care will need to be taken, e.g.

 /+ <backspace> <backspace>

 or even simply

 /+ comment +/
that should not be a problem.
Apr 05 2016
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 06.04.2016 06:16, Walter Bright wrote:
 On 4/5/2016 4:08 PM, Timon Gehr wrote:
 Some additional care will need to be taken, e.g.

 /+ <backspace> <backspace>

 or even simply

 /+ comment +/
that should not be a problem.
I think it can be for large files if the highlighter implements the described strategy in a literal fashion. /+ changes the lexer state at each position until the end of the file, and all the soon-to-be reusable information is replaced by irrelevant garbage by the reprocessing step. I think it is better to sometimes keep multiple lexer states per location, and maybe not necessarily run the lexer to completion each time. Maybe this is one reason why some editors choose to close delimiters for you automatically by default. One idea would be to handle all states simultaneously by keeping an efficient summary of the effect certain existing code snippets have on the lexer state. Summaries should be efficiently composable to compute effects (and hence the lexer state) quickly for any prefix of the code, while allowing code updates. I think it should be possible to fill in the details.
Apr 06 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-16 16:13, H. S. Teoh via Digitalmars-d wrote:

 I disagree. I think having the dmd itself (lexer, parser, etc.) as a
 library (with the dmd executable merely being the default frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D parser for the
 purposes of syntax highlighting; they can reuse the actual parser the
 compiler itself uses, and thus actually be correct (whereas their own
 parser may actually only parse a subset of D correctly).  They will also
 be able to support instant feedback as the user types in D code
 (highlight syntax errors on the fly, etc.).
I agree very much with this. Unfortunately many (most?) editors use their own weird syntax to describe grammars for syntax highlighting and cannot use a proper lexer/parser. -- /Jacob Carlborg
Apr 05 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/5/2016 4:34 AM, Jacob Carlborg wrote:
 I agree very much with this. Unfortunately many (most?) editors use their own
 weird syntax to describe grammars for syntax highlighting and cannot use a
 proper lexer/parser.
Haha, MicroEmacs is written in D, so just use D code :-) I don't really understand why IDE makers don't use an actual programming language for plugins - I'd use javascript as there's a D implementation of a javascript engine you can use. (Using D as a plugin language would require making DLLs which is a pain or making a D interpreter, which would be a major project (CTFE in theory should work for that, but it consumes too much memory).)
Apr 05 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-04-05 23:40, Walter Bright wrote:

 Haha, MicroEmacs is written in D, so just use D code :-)

 I don't really understand why IDE makers don't use an actual programming
 language for plugins - I'd use javascript as there's a D implementation
 of a javascript engine you can use. (Using D as a plugin language would
 require making DLLs which is a pain or making a D interpreter, which
 would be a major project (CTFE in theory should work for that, but it
 consumes too much memory).)
TextMate, Sublime and Atom all use the same syntax to describe the grammar for a language. All of them supports plugins (to various degrees), but none of them uses plugins to syntax highlight code, as far as I know. I guess it's easier for most of us, I know you write a lexer in two days :), to use the custom syntax to describe the grammar than to create a proper lexer and parser. Note that a lexer is not enough, these grammars can describe how a function (and other constructs) look like. -- /Jacob Carlborg
Apr 06 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/6/2016 1:09 AM, Jacob Carlborg wrote:
 TextMate, Sublime and Atom all use the same syntax to describe the grammar for
a
 language. All of them supports plugins (to various degrees), but none of them
 uses plugins to syntax highlight code, as far as I know.

 I guess it's easier for most of us, I know you write a lexer in two days :), to
 use the custom syntax to describe the grammar than to create a proper lexer and
 parser.
I doubt any of them could lex C/C++ correctly (trigraphs, macros, backslash line splicing, wysiwyg string literals). Even D has some issues - token string literals - that would defeat most grammar attempts. A straightforward lexer (i.e. one not heavily optimized for speed) written in a reasonable high level language does not look all that different from BNF, and has the advantage of not being borked if something is not quite expressible in the grammar language.
 Note that a lexer is not enough, these grammars can describe how a
 function (and other constructs) look like.
Are you talking about an AST? (Good luck doing an AST for C++ without a real C++ compiler front end! Not that hard for D, though.)
Apr 06 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-04-07 02:53, Walter Bright wrote:

 I doubt any of them could lex C/C++ correctly (trigraphs, macros,
 backslash line splicing, wysiwyg string literals).
No. But it doesn't need to be perfect. There's also nothing that tells what's wrong and what's right. For example, in TextMate the identifier in a function declaration is syntax highlighted, but in Eclipse it's not syntax highlighted.
 Are you talking about an AST?

 (Good luck doing an AST for C++ without a real C++ compiler front end!
 Not that hard for D, though.)
Yes. In TextMate one describes a grammar and assigns different scopes to the different rules. These scopes are then used by the theme engine to syntax highlight the code. The scopes are also used to enable/disable different bundle commands. For example, pressing Ctrl+B inside a comment could make the selected text bold (replace the text with an HTML or Markdown tag) while outside comments something else happens. TextMate also adds top level declarations like classes, structs and functions to a jump-list, for easy navigation. Note that I'm not defending the editors, I would love if one could use a proper front end to do the lexing and parsing. I'm just saying it's not as easy as it sounded in the post I originally replied to. -- /Jacob Carlborg
Apr 06 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/6/2016 11:36 PM, Jacob Carlborg wrote:
 On 2016-04-07 02:53, Walter Bright wrote:

 I doubt any of them could lex C/C++ correctly (trigraphs, macros,
 backslash line splicing, wysiwyg string literals).
No. But it doesn't need to be perfect.
I suppose that being a compiler guy I wouldn't care to build a half-assed one. It's not *that* hard to get it perfect, but you will need to use a programming language to do it.
Apr 07 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-04-07 09:28, Walter Bright wrote:

 I suppose that being a compiler guy I wouldn't care to build a
 half-assed one. It's not *that* hard to get it perfect, but you will
 need to use a programming language to do it.
I usually feel the same, but that would require me modify the editor. I've many times thought about create a plugin which uses a proper lexer and parser but it's a lot of work for something that doesn't have a plugin API. Also, GitHub is using the TextMate grammar to do syntax highlighting for D. If TextMate used a plugin for this instead, there would most likely be no one maintaining the the grammar GitHub is using. -- /Jacob Carlborg
Apr 07 2016
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 04/05/2016 05:40 PM, Walter Bright wrote:
 I don't really understand why IDE makers don't use an actual programming
 language for plugins
Programmer's Notepad 2 uses Python for plugins. Although the syntax highlighting is done through Scintilla which has it's own separate (non-programming-language IIRC) system for lex/parse.
Apr 07 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/14/2016 11:53 PM, Andrei Alexandrescu wrote:
 https://github.com/D-Programming-Language/dlang.org/pull/1196
 https://github.com/D-Programming-Language/dmd/pull/5352

 Destroy!!

 Andrei
Finally got around to updating this. To make it work, please review and pull https://github.com/D-Programming-Language/dmd/pull/5617 and then https://github.com/D-Programming-Language/dlang.org/pull/1196. BTW I'd forgotten we set up undefined macros to emit <!--UNDEFINED MACRO: "Xyz"--> in the generated html. That has been of great help. Andrei
Apr 01 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Alright, we're on! Entry point ain't sexy and the pages need all work, 
but it's a start. http://dlang.org/dmd-prerelease/ -- Andrei
Apr 01 2016