www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Who can make Phobos faster to import?

reply Seb <seb wilzba.ch> writes:
Following the discussion at 
https://github.com/dlang/phobos/pull/5916, we realized that 
there's quite some overhead in importing some Phobos modules. 
Ideally importing a Phobos module without using it, should be 
cost-free (i.e. pay for what you use).

Test case: consider a file with `import std.X` and compile it 
with `dmd -c -o-`.

A quick overview of the worst of the worst:

std.regex	1.03s
std.net.curl	0.28s
std.concurrency	0.23s
std.encoding	0.18s
std.zip	        0.16s
std.path	0.14s
std.file	0.14s
std.mmfile	0.14s
std.datetime	0.14s
std.socket	0.12s
std.string	0.10s
std.uni	        0.09s
std.process	0.05s

This following discussions have already brought a huge 
improvements to its light:

https://github.com/dlang/phobos/pull/5931

Can you do a another one?
Dec 15 2017
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/15/2017 02:10 PM, Seb wrote:
 Following the discussion at https://github.com/dlang/phobos/pull/5916, 
 we realized that there's quite some overhead in importing some Phobos 
 modules. Ideally importing a Phobos module without using it, should be 
 cost-free (i.e. pay for what you use).
 
 Test case: consider a file with `import std.X` and compile it with `dmd 
 -c -o-`.
 
 A quick overview of the worst of the worst:
 
 std.regex    1.03s
 std.net.curl    0.28s
 std.concurrency    0.23s
 std.encoding    0.18s
 std.zip            0.16s
 std.path    0.14s
 std.file    0.14s
 std.mmfile    0.14s
 std.datetime    0.14s
 std.socket    0.12s
 std.string    0.10s
 std.uni            0.09s
 std.process    0.05s
 
 This following discussions have already brought a huge improvements to 
 its light:
 
 https://github.com/dlang/phobos/pull/5931
 
 Can you do a another one?
Dmitry wrote a nice PR for that, and I wrote two: https://github.com/dlang/phobos/pull/5942 https://github.com/dlang/phobos/pull/5931 https://github.com/dlang/phobos/pull/5948 My approach has been the following: * First I tried to clarify which module imports what by means of replacing "import std.xxx;" with "import std.xxx : yyy, zzz, ...;" That has been of tremendous help in assessing each module's dependency liability. * Then I tried to figure which imports are most costly. I inserted __EOF__ in the module at different parts and measure how long it takes to just import that module. After a few attempts it becomes clear which imports are heaviest. * Then I pushed those heavy imports down where they're needed. An import placed in a function will not be opened while that function's module gets imported. Same goes about template types, but not structs or classes. (That should be fixed; see Seb's comment in my last PR.) Not only such work reduces import times, but it's illuminating for maintenance because it represents dependencies in a simple, fine-grained matter. A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d Andrei
Dec 20 2017
parent reply Joakim <dlang joakim.fea.st> writes:
On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei 
Alexandrescu wrote:
 On 12/15/2017 02:10 PM, Seb wrote:
 Following the discussion at 
 https://github.com/dlang/phobos/pull/5916, we realized that 
 there's quite some overhead in importing some Phobos modules. 
 Ideally importing a Phobos module without using it, should be 
 cost-free (i.e. pay for what you use).
 
 Test case: consider a file with `import std.X` and compile it 
 with `dmd -c -o-`.
 
 A quick overview of the worst of the worst:
 
 std.regex    1.03s
 std.net.curl    0.28s
 std.concurrency    0.23s
 std.encoding    0.18s
 std.zip            0.16s
 std.path    0.14s
 std.file    0.14s
 std.mmfile    0.14s
 std.datetime    0.14s
 std.socket    0.12s
 std.string    0.10s
 std.uni            0.09s
 std.process    0.05s
 
 This following discussions have already brought a huge 
 improvements to its light:
 
 https://github.com/dlang/phobos/pull/5931
 
 Can you do a another one?
Dmitry wrote a nice PR for that, and I wrote two: https://github.com/dlang/phobos/pull/5942 https://github.com/dlang/phobos/pull/5931 https://github.com/dlang/phobos/pull/5948 My approach has been the following: * First I tried to clarify which module imports what by means of replacing "import std.xxx;" with "import std.xxx : yyy, zzz, ...;" That has been of tremendous help in assessing each module's dependency liability. * Then I tried to figure which imports are most costly. I inserted __EOF__ in the module at different parts and measure how long it takes to just import that module. After a few attempts it becomes clear which imports are heaviest. * Then I pushed those heavy imports down where they're needed. An import placed in a function will not be opened while that function's module gets imported. Same goes about template types, but not structs or classes. (That should be fixed; see Seb's comment in my last PR.) Not only such work reduces import times, but it's illuminating for maintenance because it represents dependencies in a simple, fine-grained matter. A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d Andrei
I'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again: https://github.com/dlang/dmd/pull/7425 https://github.com/dlang/dmd/blob/master/src/dmd/gluelayer.d#L47 https://gist.github.com/joakim-noah/09cf49bee3d82b03a54f Once I have something basic working, I'll put it up on github, so others who are interested can play with it and pitch in to make it a real tool.
Dec 20 2017
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/2017 10:31 PM, Joakim wrote:
 On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:
 A tool (call it depend - heh) to automate that would be awesome. For 
 example, this run would make all imported names explicit:

 depend --explicit *.d

 This run would push all imports down to the innermost scope of usage:

 depend --pushdown *.d


 Andrei
I'm on it.  I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again: https://github.com/dlang/dmd/pull/7425 https://github.com/dlang/dmd/blob/master/src/dmd/gluelayer.d#L47 https://gist.github.com/joakim-noah/09cf49bee3d82b03a54f Once I have something basic working, I'll put it up on github, so others who are interested can play with it and pitch in to make it a real tool.
That's an awesome start, thanks! -- Andrei
Dec 21 2017
prev sibling parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:
 On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei 
 Alexandrescu wrote:
 On 12/15/2017 02:10 PM, Seb wrote:
 [...]
Dmitry wrote a nice PR for that, and I wrote two: https://github.com/dlang/phobos/pull/5942 https://github.com/dlang/phobos/pull/5931 https://github.com/dlang/phobos/pull/5948 My approach has been the following: * First I tried to clarify which module imports what by means of replacing "import std.xxx;" with "import std.xxx : yyy, zzz, ...;" That has been of tremendous help in assessing each module's dependency liability. * Then I tried to figure which imports are most costly. I inserted __EOF__ in the module at different parts and measure how long it takes to just import that module. After a few attempts it becomes clear which imports are heaviest. * Then I pushed those heavy imports down where they're needed. An import placed in a function will not be opened while that function's module gets imported. Same goes about template types, but not structs or classes. (That should be fixed; see Seb's comment in my last PR.) Not only such work reduces import times, but it's illuminating for maintenance because it represents dependencies in a simple, fine-grained matter. A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit: depend --explicit *.d This run would push all imports down to the innermost scope of usage: depend --pushdown *.d Andrei
I'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:
I ran into the same issue. The problem was that currently the frontend doesn't set the version identifiers. There's a PR which tries to fix it [1], but if you want to go back at using the compiler library via Seb's interface you just have to call this method [2] in the initialization step. Anyway I am working on this sort of tool also, so maybe we can unite our efforts. Ping me if you wish so, [1] https://github.com/dlang/dmd/pull/7524 [2] https://github.com/dlang/dmd/blob/master/src/dmd/mars.d#L1314
 https://github.com/dlang/dmd/pull/7425
 https://github.com/dlang/dmd/blob/master/src/dmd/gluelayer.d#L47
 https://gist.github.com/joakim-noah/09cf49bee3d82b03a54f

 Once I have something basic working, I'll put it up on github, 
 so others who are interested can play with it and pitch in to 
 make it a real tool.
Best regards, RazvanN
Dec 26 2017
parent reply Joakim <dlang joakim.fea.st> writes:
On Tuesday, 26 December 2017 at 13:27:38 UTC, RazvanN wrote:
 On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:
 On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei 
 Alexandrescu wrote:
 A tool (call it depend - heh) to automate that would be 
 awesome. For example, this run would make all imported names 
 explicit:

 depend --explicit *.d

 This run would push all imports down to the innermost scope 
 of usage:

 depend --pushdown *.d


 Andrei
I'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:
I ran into the same issue. The problem was that currently the frontend doesn't set the version identifiers. There's a PR which tries to fix it [1], but if you want to go back at using the compiler library via Seb's interface you just have to call this method [2] in the initialization step.
I doubt that's it, as I noted I was running the standard dmd main from mars.d. More likely it tries to run asmSemantic on an asm block in core.atomic and asserts because that function's stubbed out when there's no backend.
 Anyway I am working on this sort of tool also, so maybe we can 
 unite our efforts.
As in the `depend` import-scoping tool that Andrei wanted above? If so, that'd be great, as you know the frontend much better than me. I've been meaning to investigate how to extract a scope tree from the frontend, as I've never done much with dmd. Let me know how far you've gotten and what you want it to do initially, and maybe I can pitch in with your effort.
Dec 26 2017
parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Tuesday, 26 December 2017 at 15:38:14 UTC, Joakim wrote:
 On Tuesday, 26 December 2017 at 13:27:38 UTC, RazvanN wrote:
 On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:
 On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei 
 Alexandrescu wrote:
 A tool (call it depend - heh) to automate that would be 
 awesome. For example, this run would make all imported names 
 explicit:

 depend --explicit *.d

 This run would push all imports down to the innermost scope 
 of usage:

 depend --pushdown *.d


 Andrei
I'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:
I ran into the same issue. The problem was that currently the frontend doesn't set the version identifiers. There's a PR which tries to fix it [1], but if you want to go back at using the compiler library via Seb's interface you just have to call this method [2] in the initialization step.
I doubt that's it, as I noted I was running the standard dmd main from mars.d. More likely it tries to run asmSemantic on an asm block in core.atomic and asserts because that function's stubbed out when there's no backend.
 Anyway I am working on this sort of tool also, so maybe we can 
 unite our efforts.
As in the `depend` import-scoping tool that Andrei wanted above? If so, that'd be great, as you know the frontend much better than me. I've been meaning to investigate how to extract a scope tree from the frontend, as I've never done much with dmd. Let me know how far you've gotten and what you want it to do initially, and maybe I can pitch in with your effort.
Yes, that was the tool I was talking about. I was thinking that the first version of the tool just outputs for each import what the used symbols are; for example : if you import std.stdio and use 3 symbols from it, the tool will output "std.stdio: sym1, sym2, sym2" and then the programmer can update the code accordingly. After developing this initial version we can think on how to improve it. For now, my tool only identifies imported non-templated functions, but I plan on adding support variables, types etc. We can chat on slack more about this if you are interested.
Dec 27 2017
parent Joakim <dlang joakim.fea.st> writes:
On Wednesday, 27 December 2017 at 10:40:38 UTC, RazvanN wrote:
 On Tuesday, 26 December 2017 at 15:38:14 UTC, Joakim wrote:
 On Tuesday, 26 December 2017 at 13:27:38 UTC, RazvanN wrote:
 On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:
 On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei 
 Alexandrescu wrote:
 A tool (call it depend - heh) to automate that would be 
 awesome. For example, this run would make all imported 
 names explicit:

 depend --explicit *.d

 This run would push all imports down to the innermost scope 
 of usage:

 depend --pushdown *.d


 Andrei
I'm on it. I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:
I ran into the same issue. The problem was that currently the frontend doesn't set the version identifiers. There's a PR which tries to fix it [1], but if you want to go back at using the compiler library via Seb's interface you just have to call this method [2] in the initialization step.
I doubt that's it, as I noted I was running the standard dmd main from mars.d. More likely it tries to run asmSemantic on an asm block in core.atomic and asserts because that function's stubbed out when there's no backend.
 Anyway I am working on this sort of tool also, so maybe we 
 can unite our efforts.
As in the `depend` import-scoping tool that Andrei wanted above? If so, that'd be great, as you know the frontend much better than me. I've been meaning to investigate how to extract a scope tree from the frontend, as I've never done much with dmd. Let me know how far you've gotten and what you want it to do initially, and maybe I can pitch in with your effort.
Yes, that was the tool I was talking about. I was thinking that the first version of the tool just outputs for each import what the used symbols are; for example : if you import std.stdio and use 3 symbols from it, the tool will output "std.stdio: sym1, sym2, sym2" and then the programmer can update the code accordingly. After developing this initial version we can think on how to improve it. For now, my tool only identifies imported non-templated functions, but I plan on adding support variables, types etc.
I think I did all this with a dead-simple patch to dmd three years ago: http://forum.dlang.org/thread/pbpckzwmfglzgwqveoza forum.dlang.org I used that tweaked dmd to scope imports for some modules in phobos last summer: https://github.com/dlang/phobos/pulls?utf8=✓&q=is%3Apr+author%3Ajoakim-noah+is%3Aclosed+"selective+imports" That's much better than the alternative of tracking the imported symbols down yourself, but it's still a pain to edit the files manually. I'd rather it were integrated with Brian's libdparse to modify the source for you: https://github.com/dlang-community/libdparse/
 We can chat on slack more about this if you are interested.
I don't use slack or any other chat platform. Probably best to stick to the forum or github, so that anyone else who's interested can follow along.
Dec 27 2017