digitalmars.D - Who can make Phobos faster to import?
- Seb (25/25) Dec 15 2017 Following the discussion at
- Andrei Alexandrescu (26/56) Dec 20 2017 Dmitry wrote a nice PR for that, and I wrote two:
- Joakim (13/74) Dec 20 2017 I'm on it. I tried using a version of Seb's frontend library
- Andrei Alexandrescu (2/27) Dec 21 2017 That's an awesome start, thanks! -- Andrei
- RazvanN (15/72) Dec 26 2017 I ran into the same issue. The problem was that currently the
- Joakim (11/43) Dec 26 2017 I doubt that's it, as I noted I was running the standard dmd main
Following the discussion at https://github.com/dlang/phobos/pull/5916, we realized that there's quite some overhead in importing some Phobos modules. Ideally importing a Phobos module without using it, should be cost-free (i.e. pay for what you use). Test case: consider a file with `import std.X` and compile it with `dmd -c -o-`. A quick overview of the worst of the worst: std.regex 1.03s std.net.curl 0.28s std.concurrency 0.23s std.encoding 0.18s std.zip 0.16s std.path 0.14s std.file 0.14s std.mmfile 0.14s std.datetime 0.14s std.socket 0.12s std.string 0.10s std.uni 0.09s std.process 0.05s This following discussions have already brought a huge improvements to its light: https://github.com/dlang/phobos/pull/5931 Can you do a another one?
Dec 15 2017
On 12/15/2017 02:10 PM, Seb wrote:Following the discussion at https://github.com/dlang/phobos/pull/5916, we realized that there's quite some overhead in importing some Phobos modules. Ideally importing a Phobos module without using it, should be cost-free (i.e. pay for what you use). Test case: consider a file with `import std.X` and compile it with `dmd -c -o-`. A quick overview of the worst of the worst: std.regex 1.03s std.net.curl 0.28s std.concurrency 0.23s std.encoding 0.18s std.zip 0.16s std.path 0.14s std.file 0.14s std.mmfile 0.14s std.datetime 0.14s std.socket 0.12s std.string 0.10s std.uni 0.09s std.process 0.05s This following discussions have already brought a huge improvements to its light: https://github.com/dlang/phobos/pull/5931 Can you do a another one?Dmitry wrote a nice PR for that, and I wrote two: https://github.com/dlang/phobos/pull/5942 https://github.com/dlang/phobos/pull/5931 https://github.com/dlang/phobos/pull/5948 My approach has been the following: * First I tried to clarify which module imports what by means of replacing "import std.xxx;" with "import std.xxx : yyy, zzz, ...;" That has been of tremendous help in assessing each module's dependency liability. * Then I tried to figure which imports are most costly. I inserted __EOF__ in the module at different parts and measure how long it takes to just import that module. After a few attempts it becomes clear which imports are heaviest. * Then I pushed those heavy imports down where they're needed. An import placed in a function will not be opened while that function's module gets imported. Same goes about template types, but not structs or classes. (That should be fixed; see Seb's comment in my last PR.) Not only such work reduces import times, but it's illuminating for maintenance because it represents dependencies in a simple, fine-grained matter. A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d Andrei
Dec 20 2017
On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:On 12/15/2017 02:10 PM, Seb wrote:I'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again: https://github.com/dlang/dmd/pull/7425 https://github.com/dlang/dmd/blob/master/src/dmd/gluelayer.d#L47 https://gist.github.com/joakim-noah/09cf49bee3d82b03a54f Once I have something basic working, I'll put it up on github, so others who are interested can play with it and pitch in to make it a real tool.Following the discussion at https://github.com/dlang/phobos/pull/5916, we realized that there's quite some overhead in importing some Phobos modules. Ideally importing a Phobos module without using it, should be cost-free (i.e. pay for what you use). Test case: consider a file with `import std.X` and compile it with `dmd -c -o-`. A quick overview of the worst of the worst: std.regex 1.03s std.net.curl 0.28s std.concurrency 0.23s std.encoding 0.18s std.zip 0.16s std.path 0.14s std.file 0.14s std.mmfile 0.14s std.datetime 0.14s std.socket 0.12s std.string 0.10s std.uni 0.09s std.process 0.05s This following discussions have already brought a huge improvements to its light: https://github.com/dlang/phobos/pull/5931 Can you do a another one?Dmitry wrote a nice PR for that, and I wrote two: https://github.com/dlang/phobos/pull/5942 https://github.com/dlang/phobos/pull/5931 https://github.com/dlang/phobos/pull/5948 My approach has been the following: * First I tried to clarify which module imports what by means of replacing "import std.xxx;" with "import std.xxx : yyy, zzz, ...;" That has been of tremendous help in assessing each module's dependency liability. * Then I tried to figure which imports are most costly. I inserted __EOF__ in the module at different parts and measure how long it takes to just import that module. After a few attempts it becomes clear which imports are heaviest. * Then I pushed those heavy imports down where they're needed. An import placed in a function will not be opened while that function's module gets imported. Same goes about template types, but not structs or classes. (That should be fixed; see Seb's comment in my last PR.) Not only such work reduces import times, but it's illuminating for maintenance because it represents dependencies in a simple, fine-grained matter. A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d Andrei
Dec 20 2017
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> On 12/20/2017 10:31 PM, Joakim wrote:On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:That's an awesome start, thanks! -- AndreiA tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d AndreiI'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again: https://github.com/dlang/dmd/pull/7425 https://github.com/dlang/dmd/blob/master/src/dmd/gluelayer.d#L47 https://gist.github.com/joakim-noah/09cf49bee3d82b03a54f Once I have something basic working, I'll put it up on github, so others who are interested can play with it and pitch in to make it a real tool.
Dec 21 2017
On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:I ran into the same issue. The problem was that currently the frontend doesn't set the version identifiers. There's a PR which tries to fix it [1], but if you want to go back at using the compiler library via Seb's interface you just have to call this method [2] in the initialization step. Anyway I am working on this sort of tool also, so maybe we can unite our efforts. Ping me if you wish so, [1] https://github.com/dlang/dmd/pull/7524 [2] https://github.com/dlang/dmd/blob/master/src/dmd/mars.d#L1314On 12/15/2017 02:10 PM, Seb wrote:I'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:[...]Dmitry wrote a nice PR for that, and I wrote two: https://github.com/dlang/phobos/pull/5942 https://github.com/dlang/phobos/pull/5931 https://github.com/dlang/phobos/pull/5948 My approach has been the following: * First I tried to clarify which module imports what by means of replacing "import std.xxx;" with "import std.xxx : yyy, zzz, ...;" That has been of tremendous help in assessing each module's dependency liability. * Then I tried to figure which imports are most costly. I inserted __EOF__ in the module at different parts and measure how long it takes to just import that module. After a few attempts it becomes clear which imports are heaviest. * Then I pushed those heavy imports down where they're needed. An import placed in a function will not be opened while that function's module gets imported. Same goes about template types, but not structs or classes. (That should be fixed; see Seb's comment in my last PR.) Not only such work reduces import times, but it's illuminating for maintenance because it represents dependencies in a simple, fine-grained matter. A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d Andreihttps://github.com/dlang/dmd/pull/7425 https://github.com/dlang/dmd/blob/master/src/dmd/gluelayer.d#L47 https://gist.github.com/joakim-noah/09cf49bee3d82b03a54f Once I have something basic working, I'll put it up on github, so others who are interested can play with it and pitch in to make it a real tool.Best regards, RazvanN
Dec 26 2017
On Tuesday, 26 December 2017 at 13:27:38 UTC, RazvanN wrote:On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:I doubt that's it, as I noted I was running the standard dmd main from mars.d. More likely it tries to run asmSemantic on an asm block in core.atomic and asserts because that function's stubbed out when there's no backend.On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:I ran into the same issue. The problem was that currently the frontend doesn't set the version identifiers. There's a PR which tries to fix it [1], but if you want to go back at using the compiler library via Seb's interface you just have to call this method [2] in the initialization step.A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d AndreiI'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:Anyway I am working on this sort of tool also, so maybe we can unite our efforts.As in the `depend` import-scoping tool that Andrei wanted above? If so, that'd be great, as you know the frontend much better than me. I've been meaning to investigate how to extract a scope tree from the frontend, as I've never done much with dmd. Let me know how far you've gotten and what you want it to do initially, and maybe I can pitch in with your effort.
Dec 26 2017
On Tuesday, 26 December 2017 at 15:38:14 UTC, Joakim wrote:On Tuesday, 26 December 2017 at 13:27:38 UTC, RazvanN wrote:Yes, that was the tool I was talking about. I was thinking that the first version of the tool just outputs for each import what the used symbols are; for example : if you import std.stdio and use 3 symbols from it, the tool will output "std.stdio: sym1, sym2, sym2" and then the programmer can update the code accordingly. After developing this initial version we can think on how to improve it. For now, my tool only identifies imported non-templated functions, but I plan on adding support variables, types etc. We can chat on slack more about this if you are interested.On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:I doubt that's it, as I noted I was running the standard dmd main from mars.d. More likely it tries to run asmSemantic on an asm block in core.atomic and asserts because that function's stubbed out when there's no backend.On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:I ran into the same issue. The problem was that currently the frontend doesn't set the version identifiers. There's a PR which tries to fix it [1], but if you want to go back at using the compiler library via Seb's interface you just have to call this method [2] in the initialization step.A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d AndreiI'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:Anyway I am working on this sort of tool also, so maybe we can unite our efforts.As in the `depend` import-scoping tool that Andrei wanted above? If so, that'd be great, as you know the frontend much better than me. I've been meaning to investigate how to extract a scope tree from the frontend, as I've never done much with dmd. Let me know how far you've gotten and what you want it to do initially, and maybe I can pitch in with your effort.
Dec 27 2017
On Wednesday, 27 December 2017 at 10:40:38 UTC, RazvanN wrote:On Tuesday, 26 December 2017 at 15:38:14 UTC, Joakim wrote:I think I did all this with a dead-simple patch to dmd three years ago: http://forum.dlang.org/thread/pbpckzwmfglzgwqveoza forum.dlang.org I used that tweaked dmd to scope imports for some modules in phobos last summer: https://github.com/dlang/phobos/pulls?utf8=✓&q=is%3Apr+author%3Ajoakim-noah+is%3Aclosed+"selective+imports" That's much better than the alternative of tracking the imported symbols down yourself, but it's still a pain to edit the files manually. I'd rather it were integrated with Brian's libdparse to modify the source for you: https://github.com/dlang-community/libdparse/On Tuesday, 26 December 2017 at 13:27:38 UTC, RazvanN wrote:Yes, that was the tool I was talking about. I was thinking that the first version of the tool just outputs for each import what the used symbols are; for example : if you import std.stdio and use 3 symbols from it, the tool will output "std.stdio: sym1, sym2, sym2" and then the programmer can update the code accordingly. After developing this initial version we can think on how to improve it. For now, my tool only identifies imported non-templated functions, but I plan on adding support variables, types etc.On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:I doubt that's it, as I noted I was running the standard dmd main from mars.d. More likely it tries to run asmSemantic on an asm block in core.atomic and asserts because that function's stubbed out when there's no backend.On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:I ran into the same issue. The problem was that currently the frontend doesn't set the version identifiers. There's a PR which tries to fix it [1], but if you want to go back at using the compiler library via Seb's interface you just have to call this method [2] in the initialization step.A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d AndreiI'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:Anyway I am working on this sort of tool also, so maybe we can unite our efforts.As in the `depend` import-scoping tool that Andrei wanted above? If so, that'd be great, as you know the frontend much better than me. I've been meaning to investigate how to extract a scope tree from the frontend, as I've never done much with dmd. Let me know how far you've gotten and what you want it to do initially, and maybe I can pitch in with your effort.We can chat on slack more about this if you are interested.I don't use slack or any other chat platform. Probably best to stick to the forum or github, so that anyone else who's interested can follow along.
Dec 27 2017
On Wednesday, 27 December 2017 at 15:12:28 UTC, Joakim wrote:On Wednesday, 27 December 2017 at 10:40:38 UTC, RazvanN wrote:Hey Razvan, are you doing anything with this? Never heard back after my last post, so figured you were going your own way with this tool.[...]I think I did all this with a dead-simple patch to dmd three years ago: [...]
Aug 30 2018