digitalmars.D - D lexer, parser, and ImportC parser are now standalone
- Walter Bright (12/12) Apr 09 2023 I've finally managed to sever its dependencies on the rest of the compil...
- Richard (Rikki) Andrew Cattermole (12/12) Apr 09 2023 I wanted to double check something about this, but I think that this
- Andrej Mitrovic (40/43) Apr 10 2023 Appreciate the effort!
- Richard (Rikki) Andrew Cattermole (5/7) Apr 10 2023 There is: https://github.com/dlang/dmd/tree/master/compiler/test/dub_pac...
- Andrej Mitrovic (4/13) Apr 10 2023 Oh thanks, that's better. I see there's a call to initDMD(), it's
- max haughton (3/12) Apr 10 2023 Formatting off an AST directly is quite hard to get perfect.
- Andrej Mitrovic (14/24) Apr 13 2023 Are the modules in this folder tested by the CI?
- Richard (Rikki) Andrew Cattermole (3/5) Apr 13 2023 You don't need to build dmd, so digger won't be helpful anyway.
- Andrej Mitrovic (7/12) Apr 13 2023 It's been fixed in the meantime:
- Andrej Mitrovic (19/22) Apr 13 2023 ```
- RazvanN (5/26) Apr 14 2023 Try with this fix: https://github.com/dlang/dmd/pull/15106
- Andrej Mitrovic (2/3) Apr 14 2023 Thanks for the quick fix!
- RazvanN (5/8) Apr 14 2023 You're welcome! Also, note that those test are not run for some
I've finally managed to sever its dependencies on the rest of the compiler. You can now build a standalone lexer, or a standalone D parser, or a standalone ImportC parser. You can use it to build: 1. syntax highligher 2. source code formatter 3. syntax checker 4. your own semantic processor 5. something that annotates D code Even without 1..5, having it be separate from the rest of the compiler makes it much easier to understand and work with. https://github.com/dlang/dmd/pull/15091
Apr 09 2023
"Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> I wanted to double check something about this, but I think that this will only be temporary unfortunately (assuming it has been successfully split, which we can't know because it hasn't been verified at the CI level). Both build.d and dub will allow things to bleed through and inter-dependency can be added accidentally. Dub is simple, as long as its in the same directory structure, it'll be -I'd and hence available (although it may not link). build.d compiles the frontend (minus the lexer, common and root) in one go. So therefore ImportC parser and the D one, can become inter-dependent. Along with all the semantic analysis ext. https://github.com/dlang/dmd/blob/master/compiler/src/build.d#L1544 Not quite time to start the celebrations just yet :/
Apr 09 2023
On Monday, 10 April 2023 at 02:02:49 UTC, Walter Bright wrote:I've finally managed to sever its dependencies on the rest of the compiler. You can now build a standalone lexer, or a standalone D parser, or a standalone ImportC parser.Appreciate the effort! From my brief attempts I've ran into a few problems: - `Parser` requires `ErrorSink` in its constructor, but `dmd.errorsink` is not in DMD's `dub.sdl`. So using `Parser` directly doesn't seem to work (errorsink module not found). Maybe it's an easy fix to dub.sdl..? - On Linux I'm getting linking errors. While on Windows I get a segfault. Probably because I am missing some important initialization somewhere. ----- import std.stdio; import dmd.dmodule; import dmd.identifier; import dmd.location; void main() { auto mod = new Module(`C:\DMD\dmd2\src\phobos\std\file.d`, new Identifier("file"), false, false); auto loc = Loc("app.d", 0, 0); if (!mod.read(loc)) assert(0, "can't read"); } ----- Linux: ----- Linking dparser /usr/bin/ld: /home/drey/.dub/cache/dmd/~master/+lexer/build/library-debug-linux.posix-x86_64-dmd_v2.102.0-9CFF4D64344FD218B941F4DE6FAF1B94F5E5640E5F33D1921150D7105AD91F2E/libdmd_lexer.a(errors_66_6da.o):(.data._D3dmd6errors17ErrorSinkComp ler7__ClassZ+0x50): undefined reference to `_D3dmd9errorsink9ErrorSink7__ClassZ' /usr/bin/ld: /home/drey/.dub/cache/dmd/~master/+lexer/build/library-debug-linux.posix-x86_64-dmd_v2.102.0-9CFF4D64344FD218B941F4DE6FAF1B94F5E5640E5F33D1921150D7105AD91F2E/libdmd_lexer a(errors_88_732.o): in function `_D3dmd6errors18colorHighlightCodeFNbKSQBk6common9outbuffer9OutBufferZv': /home/drey/.dub/packages/dmd-master/dmd/compiler/src/dmd/errors.d:832: undefined reference to `_D3dmd9errorsink13ErrorSinkNull7__ClassZ' ----- On Windows it segfaults somewhere in `StringTable` on an invalid pointer, so probably something needs to be initialized. But the real problem here is the lack of documentation, or a lack of examples of how to use these classes outside of the compiler itself. It would be nice to have a very minimal sample dub project that uses these Module / Parser classes.
Apr 10 2023
On 11/04/2023 12:17 AM, Andrej Mitrovic wrote:It would be nice to have a very minimal sample dub project that uses these Module / Parser classes.There is: https://github.com/dlang/dmd/tree/master/compiler/test/dub_package But yeah, there needs to be working examples for a formatter + auto completion engine. That is both tested and with good documentation; those two are probably the most common use cases for such a tool.
Apr 10 2023
On Monday, 10 April 2023 at 12:30:53 UTC, Richard (Rikki) Andrew Cattermole wrote:On 11/04/2023 12:17 AM, Andrej Mitrovic wrote:Oh thanks, that's better. I see there's a call to initDMD(), it's probably what I was missing.It would be nice to have a very minimal sample dub project that uses these Module / Parser classes.There is: https://github.com/dlang/dmd/tree/master/compiler/test/dub_package But yeah, there needs to be working examples for a formatter + auto completion engine. That is both tested and with good documentation; those two are probably the most common use cases for such a tool.
Apr 10 2023
On Monday, 10 April 2023 at 12:30:53 UTC, Richard (Rikki) Andrew Cattermole wrote:On 11/04/2023 12:17 AM, Andrej Mitrovic wrote:Formatting off an AST directly is quite hard to get perfect.It would be nice to have a very minimal sample dub project that uses these Module / Parser classes.There is: https://github.com/dlang/dmd/tree/master/compiler/test/dub_package But yeah, there needs to be working examples for a formatter + auto completion engine. That is both tested and with good documentation; those two are probably the most common use cases for such a tool.
Apr 10 2023
On Monday, 10 April 2023 at 12:30:53 UTC, Richard (Rikki) Andrew Cattermole wrote:On 11/04/2023 12:17 AM, Andrej Mitrovic wrote:Are the modules in this folder tested by the CI? I'm trying out this file in v2.103.0: https://github.com/dlang/dmd/blob/05e91cdcf88a922465ff3604e28f54dc385d7f65/compiler/test/dub_package/lexer.d ``` $ dmd --versionIt would be nice to have a very minimal sample dub project that uses these Module / Parser classes.There is: https://github.com/dlang/dmd/tree/master/compiler/test/dub_package But yeah, there needs to be working examples for a formatter + auto completion engine. That is both tested and with good documentation; those two are probably the most common use cases for such a tool.DMD64 D Compiler v2.103.0$ dmd app.d app.d(3): Error: unable to read module `globals` ``` It does not seem to work. Maybe I could try with DMD `master`. Although `digger` no longer seems to work so I bet it's gonna be a pain building everything from scratch.
Apr 13 2023
On 14/04/2023 1:24 AM, Andrej Mitrovic wrote:Maybe I could try with DMD `master`. Although `digger` no longer seems to work so I bet it's gonna be a pain building everything from scratch.You don't need to build dmd, so digger won't be helpful anyway. The frontend is pretty isolated as far as overall requirements go.
Apr 13 2023
On Thursday, 13 April 2023 at 13:29:01 UTC, Richard (Rikki) Andrew Cattermole wrote:On 14/04/2023 1:24 AM, Andrej Mitrovic wrote:It's been fixed in the meantime: https://github.com/CyberShadow/Digger/issues/98 And yes now I realize this isn't supposed to be used with just a regular D install, as it needs to import into DMD modules which aren't part of the release package.Maybe I could try with DMD `master`. Although `digger` no longer seems to work so I bet it's gonna be a pain building everything from scratch.You don't need to build dmd, so digger won't be helpful anyway.
Apr 13 2023
On Thursday, 13 April 2023 at 14:27:33 UTC, Andrej Mitrovic wrote:And yes now I realize this isn't supposed to be used with just a regular D install, as it needs to import into DMD modules which aren't part of the release package.``` $ dmd -I/c/dev/dmd/compiler/src/ app.d app.d(17): Error: none of the overloads of `this` are callable using argument types `(string, immutable(char)*, int, ulong, int, int)` /c/dev/dmd/compiler/src/dmd/lexer.d(111): Candidates are: `dmd.lexer.Lexer.this(const(char)* filename, const(char)* base, ulong begoffset, ulong endoffset, bool doDocComment, bool commentToken, ErrorSink errorSink, const(char)[] vendor = "DLF", uint versionNumber = 1u)` /c/dev/dmd/compiler/src/dmd/lexer.d(170): `dmd.lexer.Lexer.this(const(char)* filename, const(char)* base, ulong begoffset, ulong endoffset, bool doDocComment, bool commentToken, bool whitespaceToken, ErrorSink errorSink)` /c/dev/dmd/compiler/src/dmd/lexer.d(182): `dmd.lexer.Lexer.this(ErrorSink errorSink)` ``` I'll submit to bugzilla.
Apr 13 2023
On Thursday, 13 April 2023 at 14:29:56 UTC, Andrej Mitrovic wrote:On Thursday, 13 April 2023 at 14:27:33 UTC, Andrej Mitrovic wrote:Try with this fix: https://github.com/dlang/dmd/pull/15106 Note that you should use the dub package by simply using ./lexer , ./parser etc. That way all the dependencies are being taken care of.[...]``` $ dmd -I/c/dev/dmd/compiler/src/ app.d app.d(17): Error: none of the overloads of `this` are callable using argument types `(string, immutable(char)*, int, ulong, int, int)` /c/dev/dmd/compiler/src/dmd/lexer.d(111): Candidates are: `dmd.lexer.Lexer.this(const(char)* filename, const(char)* base, ulong begoffset, ulong endoffset, bool doDocComment, bool commentToken, ErrorSink errorSink, const(char)[] vendor = "DLF", uint versionNumber = 1u)` /c/dev/dmd/compiler/src/dmd/lexer.d(170): `dmd.lexer.Lexer.this(const(char)* filename, const(char)* base, ulong begoffset, ulong endoffset, bool doDocComment, bool commentToken, bool whitespaceToken, ErrorSink errorSink)` /c/dev/dmd/compiler/src/dmd/lexer.d(182): `dmd.lexer.Lexer.this(ErrorSink errorSink)` ``` I'll submit to bugzilla.
Apr 14 2023
On Friday, 14 April 2023 at 10:58:55 UTC, RazvanN wrote:Try with this fix: https://github.com/dlang/dmd/pull/15106Thanks for the quick fix!
Apr 14 2023
On Friday, 14 April 2023 at 12:18:04 UTC, Andrej Mitrovic wrote:On Friday, 14 April 2023 at 10:58:55 UTC, RazvanN wrote:You're welcome! Also, note that those test are not run for some reason. If you want to see up to date examples check the compiler/test/unit directory. Those tests are ran by the compiler test suite, therefore they should always be correct.Try with this fix: https://github.com/dlang/dmd/pull/15106Thanks for the quick fix!
Apr 14 2023