www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - phobos dependencies

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1) 
for each module in phobos. Those include druntime dependencies.

With https://github.com/D-Programming-Language/phobos/pull/1768 
dependencies would be generated automatically, which should be a good 
step toward reining in.

The zsh command I ran was  wc -w generated/**/*.deps | sort -nr.


Andrei
Dec 18 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
19-Dec-2013 01:06, Andrei Alexandrescu пишет:
 http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1)
 for each module in phobos. Those include druntime dependencies.

So the bill of using pretty much anything in Phobos is pulling in 87 modules. Pretty much what I feared it is. -- Dmitry Olshansky
Dec 18 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/18/13 1:16 PM, Dmitry Olshansky wrote:
 19-Dec-2013 01:06, Andrei Alexandrescu пишет:
 http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1)
 for each module in phobos. Those include druntime dependencies.

So the bill of using pretty much anything in Phobos is pulling in 87 modules. Pretty much what I feared it is.

There are several directions we can take this. 1. Improve the compiler to handle imports lazily, i.e. an unused import is never opened. That's unlikely to help a lot of uses because most unqualified name lookups require all imports to be loaded (even after the name if resolved, the compiler must still look for ambiguities). 2. Push imports from top level into the entities (functions, classes etc) that use them. 3. Apply classic dependency management (break larger modules in smaller ones, accept some code duplication etc). I favor (2). Andrei
Dec 18 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
19-Dec-2013 01:40, Andrei Alexandrescu пишет:
 On 12/18/13 1:16 PM, Dmitry Olshansky wrote:
 19-Dec-2013 01:06, Andrei Alexandrescu пишет:
 http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1)
 for each module in phobos. Those include druntime dependencies.

So the bill of using pretty much anything in Phobos is pulling in 87 modules. Pretty much what I feared it is.

There are several directions we can take this. 1. Improve the compiler to handle imports lazily, i.e. an unused import is never opened. That's unlikely to help a lot of uses because most unqualified name lookups require all imports to be loaded (even after the name if resolved, the compiler must still look for ambiguities). 2. Push imports from top level into the entities (functions, classes etc) that use them. 3. Apply classic dependency management (break larger modules in smaller ones, accept some code duplication etc). I favor (2).

I'd add something that might help make (2) possible in case of templates. 4. Split modules at least into a package of with at least 2 parts: - Meta (traits) - constraints, type definitions etc. - API - functions, globals and other "meat" of the module. One prime example is std.range - isInputRange, isForwardRange etc. need not to be "bought together" with all of range adapters and std.algorithm dependency. -- Dmitry Olshansky
Dec 18 2013
parent Shammah Chancellor <anonymous coward.com> writes:
On 2013-12-18 22:19:14 +0000, H. S. Teoh said:

 I've been saying this over and over: now we have package.d, let's make
 use of it. Jonathan has been working on splitting up std.datetime
 (though that has yet to materialize); I think std.algorithm and
 std.range are due, too.

+1 on this. When I have some time I was going to make a pull request on some of the modules. -Shammah
Dec 19 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/18/2013 1:40 PM, Andrei Alexandrescu wrote:
 1. Improve the compiler to handle imports lazily, i.e. an unused import is
never
 opened. That's unlikely to help a lot of uses because most unqualified name
 lookups require all imports to be loaded (even after the name if resolved, the
 compiler must still look for ambiguities).

 2. Push imports from top level into the entities (functions, classes etc) that
 use them.

 3. Apply classic dependency management (break larger modules in smaller ones,
 accept some code duplication etc).

 I favor (2).

(1) is impractical for the reasons you mentioned. I favor (2), and also (4): 4. Break kitchen sink modules like std.algorithm into one module per algorithm. This should not result in code duplication.
Dec 18 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/18/13 3:08 PM, Walter Bright wrote:
 On 12/18/2013 1:40 PM, Andrei Alexandrescu wrote:
 1. Improve the compiler to handle imports lazily, i.e. an unused
 import is never
 opened. That's unlikely to help a lot of uses because most unqualified
 name
 lookups require all imports to be loaded (even after the name if
 resolved, the
 compiler must still look for ambiguities).

 2. Push imports from top level into the entities (functions, classes
 etc) that
 use them.

 3. Apply classic dependency management (break larger modules in
 smaller ones,
 accept some code duplication etc).

 I favor (2).

(1) is impractical for the reasons you mentioned. I favor (2), and also (4): 4. Break kitchen sink modules like std.algorithm into one module per algorithm. This should not result in code duplication.

That's (3), which enumerates a number of possible effects that may or may not overlap/ Andrei
Dec 18 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-19 08:26, Meta wrote:

 Maybe it's not possible, I'm not well versed on how the compiler works.
 But I think that one module per algorithm may be too granular. Hasn't
 someone suggested splitting it up by category, e.g., sorting, mutation,
 searching, etc.?

Sounds more reasonable. -- /Jacob Carlborg
Dec 19 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-18 22:40, Andrei Alexandrescu wrote:

 There are several directions we can take this.

 1. Improve the compiler to handle imports lazily, i.e. an unused import
 is never opened. That's unlikely to help a lot of uses because most
 unqualified name lookups require all imports to be loaded (even after
 the name if resolved, the compiler must still look for ambiguities).

 2. Push imports from top level into the entities (functions, classes
 etc) that use them.

 3. Apply classic dependency management (break larger modules in smaller
 ones, accept some code duplication etc).

 I favor (2).

I would guess that not a single of these solutions alone would solve the problem. Most likely all will be needed. -- /Jacob Carlborg
Dec 18 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/13 1:06 AM, Meta wrote:
 On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei Alexandrescu wrote:
 2. Push imports from top level into the entities (functions, classes
 etc) that use them.

2 comes with its own problems, though. With imports at the top, you can easily tell what the module imports at a glance. Putting imports into the deepest possible scopes where they are used will make it a huge chore to hunt all the imports down. You can of course grep for "import" (or ctrl+f for more plebeian editors), but you still don't have that singular list in one place that you can easily look at to tell what the module imports.

I think that's a price we need to pay the same way cars replaced carriages in spite of them having a lot of disadvantages at the time. It's simply a mindset to do away with. Transitive imports can be figured with: dmd -c -o- -v module | grep '^import ' I felt the need for non-transitive imports, and I'm sure a tool will show up before long. Local imports are the future. Better get used to them.
 Also, as was mentioned before, some imports (or lack of) will go
 unnoticed until the particular template or template path that it resides
 in gets instantiated. I don't relish the idea of thinking that my code
 is all clear, then getting some missing dependency error down the line
 after refactoring. This will make code more brittle.

Unittests/coverage to the rescue. Tooling tooling tooling. Andrei
Dec 20 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-20 10:37, Andrei Alexandrescu wrote:

 I think that's a price we need to pay the same way cars replaced
 carriages in spite of them having a lot of disadvantages at the time.
 It's simply a mindset to do away with.

 Transitive imports can be figured with:

 dmd -c -o- -v module | grep '^import '

 I felt the need for non-transitive imports, and I'm sure a tool will
 show up before long.

I agree. It would be nice if an IDE could show a list of all imports, including locals, of a module. -- /Jacob Carlborg
Dec 20 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/13 5:45 AM, Joseph Rushton Wakeling wrote:
 On 20/12/13 10:06, Meta wrote:
      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && isUniformRNG!RandomGen)  // <--- needs
 std.random.isUniformRNG
      {
          static assert(ss == SwapStrategy.unstable,
                  "Stable topN not yet implemented");
          while (r.length > nth)
          {
              auto pivot = uniform(0, r.length, rng);
              // ... etc. ...
          }
      }

I had this idea fot a while, and Walter is favorable of it as well - extend "import" for one-shot use. With that feature the example would become: void topN(alias less = "a < b", SwapStrategy ss = SwapStrategy.unstable, Range, RandomGen)(Range r, size_t nth, ref RandomGen rng) if (isRandomAccessRange!(Range) && hasLength!Range && import.std.random.isUniformRNG!RandomGen) { ... } In this case "import" would syntactically be placed at the beginning of a qualified name, meaning "import this module lazily and look up the symbol in it". This would simplify quite a lot of two-liners into one-liners in other places, too. Andrei
Dec 20 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/13 9:38 AM, H. S. Teoh wrote:
 Hmm. Why do we need to incorporate the 'import' keyword in the first
 place?

To make the intent explicit. But that's an interesting idea. I'd probably require a leading '.' just to make it unambiguous that lookup must start from top level.
 What about extending symbol lookup, so that if a fully-qualified
 symbol x.y.z can't be found in the current symbol tables, and x/y exists
 in the current import path, then implicitly try to import x.y and lookup
 z in that module. Then you could just write:

 	void f(T)(T t) if (std.range.isInputRange!T) ...

Due to a bug that's actually the case today to some extent :o). Andrei
Dec 20 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/20/2013 9:57 AM, Andrei Alexandrescu wrote:
 On 12/20/13 9:38 AM, H. S. Teoh wrote:
 Hmm. Why do we need to incorporate the 'import' keyword in the first
 place?

To make the intent explicit. But that's an interesting idea. I'd probably require a leading '.' just to make it unambiguous that lookup must start from top level.

Yes, it's interesting, but I think it's a bit too clever :-) I think importing should be explicit, not implicit.
Dec 20 2013
prev sibling next sibling parent captaindet <2krnk gmx.net> writes:
On 2013-12-20 11:38, H. S. Teoh wrote:
 On Fri, Dec 20, 2013 at 09:27:55AM -0800, Andrei Alexandrescu wrote:
 On 12/20/13 5:45 AM, Joseph Rushton Wakeling wrote:
 On 20/12/13 10:06, Meta wrote:
      void topN(alias less = "a<  b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range)&&  hasLength!Range
              &&  isUniformRNG!RandomGen)  //<--- needs
 std.random.isUniformRNG
      {
          static assert(ss == SwapStrategy.unstable,
                  "Stable topN not yet implemented");
          while (r.length>  nth)
          {
              auto pivot = uniform(0, r.length, rng);
              // ... etc. ...
          }
      }

I had this idea fot a while, and Walter is favorable of it as well - extend "import" for one-shot use. With that feature the example would become: void topN(alias less = "a< b", SwapStrategy ss = SwapStrategy.unstable, Range, RandomGen)(Range r, size_t nth, ref RandomGen rng) if (isRandomAccessRange!(Range)&& hasLength!Range && import.std.random.isUniformRNG!RandomGen) { ... } In this case "import" would syntactically be placed at the beginning of a qualified name, meaning "import this module lazily and look up the symbol in it".

Hmm. Why do we need to incorporate the 'import' keyword in the first place? What about extending symbol lookup, so that if a fully-qualified symbol x.y.z can't be found in the current symbol tables, and x/y exists in the current import path, then implicitly try to import x.y and lookup z in that module. Then you could just write: void f(T)(T t) if (std.range.isInputRange!T) ... and the compiler will automatically import std.range within that scope. Obviously, it's a bad idea to import std.range into module scope, since it would pollute the module namespace, but it seems a good idea to do a one-shot import automatically, since the qualified symbol itself already says which module the symbol is supposed to be defined in. There's no need to add another "import." prefix to it, IMO. T

one could make it a bit more explicit by requiring import std; at module level, or even more explicit auto import std; to enable lazy import of everything that is hierarchically below. could then be used for user libraries as well: import mybreadandbutter; // has several sub folders / packages /det
Dec 20 2013
prev sibling next sibling parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-12-20 17:27:55 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 I had this idea fot a while, and Walter is favorable of it as well - 
 extend "import" for one-shot use. With that feature the example would 
 become:
 
      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }
 
 In this case "import" would syntactically be placed at the beginning of 
 a qualified name, meaning "import this module lazily and look up the 
 symbol in it".
 
 This would simplify quite a lot of two-liners into one-liners in other 
 places, too.

How do you solve the problem that in D you can't tell the module name from a fully qualified names? For instance: import.std.something.somethingelse.anotherthing = 1; Is the module std? std.something? std.something.somethingelse? It could be any of these answers. module std.something; struct somethingelse { static int anotherthing = 0; } or module std.something.somethingelse; int anotherthing = 0; -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Dec 20 2013
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-12-20 19:36:28 +0000, "Meta" <jared771 gmail.com> said:

 On Friday, 20 December 2013 at 19:34:10 UTC, Patrick Down wrote:
 On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh wrote:
 in the current import path, then implicitly try to import x.y and lookup
 z in that module. Then you could just write:
 
 	void f(T)(T t) if (std.range.isInputRange!T) ...
 
 and the compiler will automatically import std.range within that scope.

How about: scope import std.range; // lazy import std.range; ? void f(T)(T t) if (std.range.isInputRange!T) ...

I think the best keyword to use in this situation would be stati... Oh, dammit, not again.

Actually, "static import" already exists. And semantically it's pretty much the same thing as the above: you have to use the symbol's full name. But currently the compiler will import eagerly. I doubt there'd be any breakage if "static" changed to mean "lazily imported". -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Dec 20 2013
parent Martin Nowak <code dawg.eu> writes:
On 12/20/2013 08:40 PM, Michel Fortin wrote:
 Actually, "static import" already exists. And semantically it's pretty
 much the same thing as the above: you have to use the symbol's full
 name. But currently the compiler will import eagerly. I doubt there'd be
 any breakage if "static" changed to mean "lazily imported".

Ah, same idea :).
Dec 20 2013
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 12/20/2013 06:27 PM, Andrei Alexandrescu wrote:
 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example would
 become:

      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }

 In this case "import" would syntactically be placed at the beginning of
 a qualified name, meaning "import this module lazily and look up the
 symbol in it".

 This would simplify quite a lot of two-liners into one-liners in other
 places, too.

The fact that template constraints use the module scope is indeed a root cause for a lot of module dependencies, so we should tackle this problem specifically. Couldn't static imports be made lazy without breaking any code? The above example would read. static import std.range. void foo(R)(R range) if (std.range.isForwardRange!R) { }
Dec 20 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/13 12:43 PM, Martin Nowak wrote:
 Couldn't static imports be made lazy without breaking any code?

One simple idea is to make all imports lazy - no change in semantics. Andrei
Dec 20 2013
parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-12-20 21:36:15 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 On 12/20/13 12:43 PM, Martin Nowak wrote:
 Couldn't static imports be made lazy without breaking any code?

One simple idea is to make all imports lazy - no change in semantics.

Sure, why not. Except that upon reaching the first non-fully-qualified symbol you'd have to load them all because they could all contain a symbol with that name. So in practice it'd change nothing, except for static imports. Beside that, loading modules eagerly is more parallelizable. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Dec 20 2013
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
21-Dec-2013 00:43, Martin Nowak пишет:
 On 12/20/2013 06:27 PM, Andrei Alexandrescu wrote:
 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example would
 become:

      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }

 In this case "import" would syntactically be placed at the beginning of
 a qualified name, meaning "import this module lazily and look up the
 symbol in it".

 This would simplify quite a lot of two-liners into one-liners in other
 places, too.

The fact that template constraints use the module scope is indeed a root cause for a lot of module dependencies, so we should tackle this problem specifically. Couldn't static imports be made lazy without breaking any code? The above example would read. static import std.range. void foo(R)(R range) if (std.range.isForwardRange!R) { }

I seriously doubt that we'd get anything better then:
 import std.range.traits;

 void foo(R)(R range) if (isForwardRange!R)
 {

...
 }

Stays practical w.r.t. cutting down dependencies and no need to uglify constraints. They are not that readable already. -- Dmitry Olshansky
Dec 21 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/21/13 12:08 AM, Dmitry Olshansky wrote:
 21-Dec-2013 00:43, Martin Nowak пишет:
 On 12/20/2013 06:27 PM, Andrei Alexandrescu wrote:
 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example would
 become:

      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }

 In this case "import" would syntactically be placed at the beginning of
 a qualified name, meaning "import this module lazily and look up the
 symbol in it".

 This would simplify quite a lot of two-liners into one-liners in other
 places, too.

The fact that template constraints use the module scope is indeed a root cause for a lot of module dependencies, so we should tackle this problem specifically. Couldn't static imports be made lazy without breaking any code? The above example would read. static import std.range. void foo(R)(R range) if (std.range.isForwardRange!R) { }


Yah but only if the symbol foo is actually used. Andrei
Dec 21 2013
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
21-Dec-2013 21:10, Andrei Alexandrescu пишет:
 On 12/21/13 12:08 AM, Dmitry Olshansky wrote:
 21-Dec-2013 00:43, Martin Nowak пишет:


 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


Yah but only if the symbol foo is actually used.

That assuming static import becomes lazy (if/when). In such a case I'd be against the idom still if only because of extra verbosity in constraints - it's a place where we'd want to have less of it. Second point is that even if import becomes lazy it's still analyze-the-whole-module at the first reference required. No escaping the fact that both constraints and bodies of templates function need to use fine grained imports (more . The more can be shifted inside of the body, and the more specific it gets the better dependency management we have. The latter implies the well-known benefit of having less stuff to analyze, compile and link. Thus I conclude that introducing lazy loading of symbols accomplishes too little for the amount of changes it entails. -- Dmitry Olshansky
Dec 21 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/21/13 1:06 PM, Dmitry Olshansky wrote:
 21-Dec-2013 21:10, Andrei Alexandrescu пишет:
 On 12/21/13 12:08 AM, Dmitry Olshansky wrote:
 21-Dec-2013 00:43, Martin Nowak пишет:


 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


Yah but only if the symbol foo is actually used.

That assuming static import becomes lazy (if/when). In such a case I'd be against the idom still if only because of extra verbosity in constraints - it's a place where we'd want to have less of it.

That's why I'm saying: make all imports lazy!!!! Andrei
Dec 21 2013
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
22-Dec-2013 01:16, Andrei Alexandrescu пишет:
 On 12/21/13 1:06 PM, Dmitry Olshansky wrote:
 21-Dec-2013 21:10, Andrei Alexandrescu пишет:
 On 12/21/13 12:08 AM, Dmitry Olshansky wrote:
 21-Dec-2013 00:43, Martin Nowak пишет:


 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


Yah but only if the symbol foo is actually used.

That assuming static import becomes lazy (if/when). In such a case I'd be against the idom still if only because of extra verbosity in constraints - it's a place where we'd want to have less of it.

That's why I'm saying: make all imports lazy!!!!

Unless language defines a way to tell apart and split off a group of declarations inside of a module as independent block laziness doesn't help any. The whole reason is to avoid analyzing the whole module and pulling in its globals. If lazy import can pull only pieces (per symbol dependencies) of module that are actually required - cool, but it's seems very distant possibility. As it stands the only thing lazy buys us is "pay as you touch" contrary to "pay as you name the intent to touch". The problem is that the payment is for the whole stock of the said "shop". I see second problem (granularity of imports) as far more critical then the first (condition under which the pieces are imported). The second problem seems solvable within the current implementation, the first seems like it would need arbitrary amount of time to fix and gains are marginal. -- Dmitry Olshansky
Dec 21 2013
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
22-Dec-2013 02:16, monarch_dodra пишет:
 On Saturday, 21 December 2013 at 21:38:59 UTC, Dmitry Olshansky wrote:
 As it stands the only thing lazy buys us is "pay as you touch"
 contrary to "pay as you name the intent to touch". The problem is that
 the payment is for the whole stock of the said "shop". I see second
 problem (granularity of imports) as far more critical then the first
 (condition under which the pieces are imported). The second problem
 seems solvable within the current implementation, the first seems like
 it would need arbitrary amount of time to fix and gains are marginal.

We should also keep in mind that as we split up modules and split apart dependencies, it also means that *as* we import a specific package, we are increasing our "use/import" ratio, further diminishing the issue of "import things we don't need." (who would import "std.foo.bar.baz", if they weren't planning to use baz?).

The main gain is to the library code be it Phobos itself or any 3rd party code. Application code is import-happy by definition and only few folks at this level would care to import things in such a fine garined manner.
 Arguably, we'd get "quadratic" effectiveness ;)

In mind the constant factor is increasing (as in more files, more syscalls) but as the total LOCs per package version of a module stays the same I fail to see any substantial efficiency loss. -- Dmitry Olshansky
Dec 21 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/21/13 1:38 PM, Dmitry Olshansky wrote:
 As it stands the only thing lazy buys us is "pay as you touch" contrary
 to "pay as you name the intent to touch". The problem is that the
 payment is for the whole stock of the said "shop". I see second problem
 (granularity of imports) as far more critical then the first (condition
 under which the pieces are imported). The second problem seems solvable
 within the current implementation, the first seems like it would need
 arbitrary amount of time to fix and gains are marginal.

The way I see it is we can improve the compilation speed for everyone without having them change the code in any way, as opposed to toiling over phobos to the benefit of nothing else but phobos. Which is the better win? Andrei
Dec 21 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
22-Dec-2013 08:23, Andrei Alexandrescu пишет:
 On 12/21/13 1:38 PM, Dmitry Olshansky wrote:
 As it stands the only thing lazy buys us is "pay as you touch" contrary
 to "pay as you name the intent to touch". The problem is that the
 payment is for the whole stock of the said "shop". I see second problem
 (granularity of imports) as far more critical then the first (condition
 under which the pieces are imported). The second problem seems solvable
 within the current implementation, the first seems like it would need
 arbitrary amount of time to fix and gains are marginal.

The way I see it is we can improve the compilation speed for everyone without having them change the code in any way, as opposed to toiling over phobos to the benefit of nothing else but phobos.

Because of poor organization of Phobos we have to. Anyhow work on std library is a major benefit to everyone who uses it (including compilation speed). And nobody have to change code to get the benefits e.g. the proverbial size of hello world is going to be smaller (you surely recall that recent problem). Unlike Phobos many projects already have sane package structure but the moment they use this tiny primitive (e.g. writeln) in Phobos compile times drop.
 Which is the
 better win?

I'd take both :o) I'm not against lazy imports per see. The problem is that the "instant" benefit is going to be smaller then you seem to imply. I'd like to be proven otherwise but there is no experimental compiler to see yet. If Phobos stays pretty much as is lazy imports is just a new (potentially slow and bogus) feature. And we know how new stuff works - see -allinst switch. On the other hand restructuring Phobos is a bonus that users don't have to suffer for. -- Dmitry Olshansky
Dec 22 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On 12/20/2013 09:43 PM, Martin Nowak wrote:
 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }

Furthermore selective import can always be made lazily without changing code. import std.algorithm : min; Only if `min` or `std.algorithm` are used the compiler needs to open std.algorithm.
Jan 05 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/5/14 5:22 PM, Martin Nowak wrote:
 On 12/20/2013 09:43 PM, Martin Nowak wrote:
 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }

Furthermore selective import can always be made lazily without changing code. import std.algorithm : min; Only if `min` or `std.algorithm` are used the compiler needs to open std.algorithm.

I thought more about this and I think that's wrong. Making all imports lazy would be a net large win. The missing link is transitivity: if one imports only min from std.algorithm, std.algorithm itself indeed needs to be opened and parsed, but ONLY transitive imports of std.algorithm that min itself needs would be opened. We can do lazy importing with no change to the language semantics/ Andrei
Jan 05 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/5/14 5:22 PM, Martin Nowak wrote:
 On 12/20/2013 09:43 PM, Martin Nowak wrote:
 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }

Furthermore selective import can always be made lazily without changing code. import std.algorithm : min; Only if `min` or `std.algorithm` are used the compiler needs to open std.algorithm.

I thought more about this and I think that's wrong. Making all imports lazy would be a net large win. The missing link is transitivity: if one imports only min from std.algorithm, std.algorithm itself indeed needs to be opened and parsed, but ONLY transitive imports of std.algorithm that min itself needs would be opened. We can do lazy importing with no change to the language semantics. Andrei
Jan 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/5/14 8:44 PM, Kenji Hara wrote:
 Honestly, "lazy import" (== defer loading module file and running
 semantic analysis for symbol search) would improve compilation speed for
 selective imports and static imports, but it would have no merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." is not correct.

Consider: import std.stdio; void main() { writeln("Hello, world!"); } Currently std.stdio and all modules transitively imported by it would be opened. With lazy imports, std.stdio gets opened and then writeln() gets semantically analyzed. Only modules required by writeln() itself will actually be opened. Big difference. Andrei
Jan 05 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/5/2014 9:22 PM, Andrei Alexandrescu wrote:
 On 1/5/14 8:44 PM, Kenji Hara wrote:
 Honestly, "lazy import" (== defer loading module file and running
 semantic analysis for symbol search) would improve compilation speed for
 selective imports and static imports, but it would have no merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." is not correct.

Consider: import std.stdio; void main() { writeln("Hello, world!"); } Currently std.stdio and all modules transitively imported by it would be opened. With lazy imports, std.stdio gets opened and then writeln() gets semantically analyzed. Only modules required by writeln() itself will actually be opened. Big difference.

import bar; import foo; Importing foo can never be done lazily if there are unqualified references to symbols in bar.
Jan 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/5/14 10:31 PM, Walter Bright wrote:
 On 1/5/2014 9:22 PM, Andrei Alexandrescu wrote:
 On 1/5/14 8:44 PM, Kenji Hara wrote:
 Honestly, "lazy import" (== defer loading module file and running
 semantic analysis for symbol search) would improve compilation speed for
 selective imports and static imports, but it would have no merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." is not
 correct.

Consider: import std.stdio; void main() { writeln("Hello, world!"); } Currently std.stdio and all modules transitively imported by it would be opened. With lazy imports, std.stdio gets opened and then writeln() gets semantically analyzed. Only modules required by writeln() itself will actually be opened. Big difference.

import bar; import foo; Importing foo can never be done lazily if there are unqualified references to symbols in bar.

Yah, but modules transitively imported in foo and bar need not be loaded eagerly. That's where the win comes from. Took me a while to figure. Andrei
Jan 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/7/14 3:25 AM, monarch_dodra wrote:
 Given that you *probably* imported "foo" with the plan to *use* one of
 its functions, you'll encounter an unqualified call sooner rather than
 later, and any "win" will promptly be lost.

It's not a one- or two-levels win, it's a transitive win. An unqualified call in one implementation would trigger only one level of import. That said I agree it's suboptimal, but it's a net improvement in the compiler that requires zero changes to source code. According to Walter it would also get rid of some forward declarations issues. Andrei
Jan 07 2014
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 01/07/2014 05:20 PM, Andrei Alexandrescu wrote:
 On 1/7/14 3:25 AM, monarch_dodra wrote:
 Given that you *probably* imported "foo" with the plan to *use* one of
 its functions, you'll encounter an unqualified call sooner rather than
 later, and any "win" will promptly be lost.

It's not a one- or two-levels win, it's a transitive win. An unqualified call in one implementation would trigger only one level of import. That said I agree it's suboptimal, but it's a net improvement in the compiler that requires zero changes to source code. According to Walter it would also get rid of some forward declarations issues. Andrei

Without introducing new ones? The problem with getting rid of some forward declarations issues in the past has been that some new ones were often introduced due to the analysis order changing in ad-hoc ways. What is the status of getting rid of _all_ forward declarations issues except those in a precisely specified set of uninterpretable usages? E.g. I have no precise image of how arbitrary forward referencing of enum members is actually supposed to work in a consistent way without base type annotation.
Jan 07 2014
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 12/20/2013 06:27 PM, Andrei Alexandrescu wrote:
 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example would
 become:

      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }

 In this case "import" would syntactically be placed at the beginning of
 a qualified name, meaning "import this module lazily and look up the
 symbol in it".

 This would simplify quite a lot of two-liners into one-liners in other
 places, too.


 Andrei

This is problematic for dependency generation à la rdmd. The compiler won't know the import until the lazy symbol is used, thereby making it impossible to build all dependencies into a library using rdmd.
Dec 20 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jan 05, 2014 at 09:22:09PM -0800, Andrei Alexandrescu wrote:
 On 1/5/14 8:44 PM, Kenji Hara wrote:
Honestly, "lazy import" (== defer loading module file and running
semantic analysis for symbol search) would improve compilation speed
for selective imports and static imports, but it would have no merit
for basic imports.

So precisely, "all imports lazy would be a net large win." is not
correct.

Consider: import std.stdio; void main() { writeln("Hello, world!"); } Currently std.stdio and all modules transitively imported by it would be opened. With lazy imports, std.stdio gets opened and then writeln() gets semantically analyzed. Only modules required by writeln() itself will actually be opened. Big difference.

It would be even better if only those modules required by the specific instantiation of writeln() being used would be loaded. So if you only instantiate writeln() with string arguments, then std.format doesn't even need to be pulled in. My guess is that this alone would reduce std.stdio template bloat by about 10-20% or so, possibly more, because most uses of writeln() actually only need a small part of the entire code that implements it. T -- Never step over a puddle, always step around it. Chances are that whatever made it is still dripping.
Jan 05 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 19, 2013 at 01:47:26AM +0400, Dmitry Olshansky wrote:
 19-Dec-2013 01:40, Andrei Alexandrescu пишет:
On 12/18/13 1:16 PM, Dmitry Olshansky wrote:
19-Dec-2013 01:06, Andrei Alexandrescu пишет:
http://chopapp.com/#fvepfd8 shows the number of dependencies (plus
1) for each module in phobos. Those include druntime dependencies.

So the bill of using pretty much anything in Phobos is pulling in 87 modules. Pretty much what I feared it is.

There are several directions we can take this. 1. Improve the compiler to handle imports lazily, i.e. an unused import is never opened. That's unlikely to help a lot of uses because most unqualified name lookups require all imports to be loaded (even after the name if resolved, the compiler must still look for ambiguities). 2. Push imports from top level into the entities (functions, classes etc) that use them. 3. Apply classic dependency management (break larger modules in smaller ones, accept some code duplication etc). I favor (2).


I vote for (2) as well. And for the first part of (3): I think std.algorithm is due for a breakup. There is no reason why using find, for example, should pull in sorting algorithms, and why using sorting algorithms should pull in cartesianProduct.
 I'd add something that might help make (2) possible in case of
 templates.
 
 4. Split modules at least into a package of with at least 2 parts:
 - Meta (traits) - constraints, type definitions etc.
 - API - functions, globals and other "meat" of the module.
 
 One prime example is std.range - isInputRange, isForwardRange etc.
 need not to be "bought together" with all of range adapters and
 std.algorithm dependency.

I've been saying this over and over: now we have package.d, let's make use of it. Jonathan has been working on splitting up std.datetime (though that has yet to materialize); I think std.algorithm and std.range are due, too. T -- "Hi." "'Lo."
Dec 18 2013
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Wednesday, 18 December 2013 at 23:08:21 UTC, Walter Bright 
wrote:
 I favor (2), and also (4):

 4. Break kitchen sink modules like std.algorithm into one 
 module per algorithm. This should not result in code 
 duplication.

That seems a little over the top. While I guess it's not necessarily a bad thing to increase modularity, users would have to pull in a large number of imports to do anything nontrivial. For example, when you want to use sort, find, splitter and swap all together, you either have the choice of doing: import std.algorithm; And pull in everything, or doing: import std.algorithm.sort, std.algorithm.find, std.algorithm.splitter, std.algorithm.swap; Or is there something I'm missing here? Of course, another large boon would be to correct the implementation of: import std.algorithm: sort, find, splitter, swap; So that it actually works in a sane way as opposed to pulling in everything.
Dec 18 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei 
Alexandrescu wrote:
 1. Improve the compiler to handle imports lazily, i.e. an 
 unused import is never opened. That's unlikely to help a lot of 
 uses because most unqualified name lookups require all imports 
 to be loaded (even after the name if resolved, the compiler 
 must still look for ambiguities).

 2. Push imports from top level into the entities (functions, 
 classes etc) that use them.

 3. Apply classic dependency management (break larger modules in 
 smaller ones, accept some code duplication etc).

 I favor (2).


 Andrei

2 + 3 (breaking modules into packages part) should do the tricky as far as I can see. Lot of current dependency bloat comes from importing whole std.range / std.algorithm to use one or two utility functions in few places. On topic of (2) I propose to add it to Phobos style guide and start enforcing during PR review.
Dec 18 2013
prev sibling next sibling parent Justin Whear <justin economicmodeling.com> writes:
On Thu, 19 Dec 2013 00:51:04 +0100, Meta wrote:

 On Wednesday, 18 December 2013 at 23:08:21 UTC, Walter Bright wrote:
 I favor (2), and also (4):

 4. Break kitchen sink modules like std.algorithm into one module per
 algorithm. This should not result in code duplication.

That seems a little over the top. While I guess it's not necessarily a bad thing to increase modularity, users would have to pull in a large number of imports to do anything nontrivial. For example, when you want to use sort, find, splitter and swap all together, you either have the choice of doing: import std.algorithm; And pull in everything, or doing: import std.algorithm.sort, std.algorithm.find, std.algorithm.splitter, std.algorithm.swap; Or is there something I'm missing here? Of course, another large boon would be to correct the implementation of: import std.algorithm: sort, find, splitter, swap; So that it actually works in a sane way as opposed to pulling in everything.

Presumably there would still be a std.algorithm package module; the questions is: does import std.algorithm : joiner; still work if std.algorithm is actually a package module and joiner is in a std.algorithm.joiner module?
Dec 18 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 19 December 2013 at 00:15:12 UTC, Justin Whear wrote:
 Presumably there would still be a std.algorithm package module;
  the
 questions is: does
   import std.algorithm : joiner;
 still work if std.algorithm is actually a package module and 
 joiner is in
 a std.algorithm.joiner module?

Intended to work by matching DIP.
Dec 18 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 18 December 2013 at 23:51:06 UTC, Meta wrote:
 Of course, another large boon would be to correct the 
 implementation of:

 import std.algorithm: sort, find, splitter, swap;

 So that it actually works in a sane way as opposed to pulling 
 in everything.

I don't see how it is even theoretically possible. You still need to lex/parse whole file to locate needed symbols (and semantic phases are already lazy in most cases afaik)
Dec 18 2013
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Thursday, 19 December 2013 at 00:23:47 UTC, Dicebot wrote:
 On Wednesday, 18 December 2013 at 23:51:06 UTC, Meta wrote:
 Of course, another large boon would be to correct the 
 implementation of:

 import std.algorithm: sort, find, splitter, swap;

 So that it actually works in a sane way as opposed to pulling 
 in everything.

I don't see how it is even theoretically possible. You still need to lex/parse whole file to locate needed symbols (and semantic phases are already lazy in most cases afaik)

Maybe it's not possible, I'm not well versed on how the compiler works. But I think that one module per algorithm may be too granular. Hasn't someone suggested splitting it up by category, e.g., sorting, mutation, searching, etc.?
Dec 18 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/19/13, Meta <jared771 gmail.com> wrote:
 For example, when you want to use sort, find, splitter and swap
 all together, you either have the choice of doing:

 import std.algorithm;

 And pull in everything, or doing:

 import std.algorithm.sort, std.algorithm.find,
 std.algorithm.splitter, std.algorithm.swap;

 Or is there something I'm missing here?

Hopefully we would add a feature that implements selective module imports, e.g.: // either a symbol in the algorithm.d/package.d module or // another module in the algorithm package *if* std.algorithm is a package. import std.algorithm : sort; I think there's a bugzilla enhancement for this.
Dec 19 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 18/12/13 22:40, Andrei Alexandrescu wrote:
 2. Push imports from top level into the entities (functions, classes etc) that
 use them.

This is probably going to be very productive. Example: a while back we had a discussion about how std.stdio wound up pulling in std.complex. John Colvin worked out that the module dependency goes something like this: std.stdio -> std.algorithm -> std.random -> std.numeric -> std.complex ... so I looked a bit deeper and found that there are at least 2 points where that dependency chain could be broken: * std.algorithm relies on std.random I think only one actual function, namely topN (where it calls std.random.uniform). All other uses of std.random are inside unittests. (Incidentally it's a bit bad that topN makes use of randomness but does not AFAICS allow you to specify a RNG.) * std.numeric relies on std.complex for quite a few functions and objects, but by no means all, and I doubt that std.random's dependency on std.numeric makes use of any of those. The problem here is that those functions need awareness of std.complex.Complex at the module level, e.g. because they have Complex return-types. So, it might be productive to separate out std.numeric based on std.complex dependency. I can provide a patch for std.random very easily (I already have it un-committed on my system right now as a result of re-running the above analysis). I also think it should be policy for Phobos that if an import is required only for unittests, it must be imported in those individual unittests, not for the module as a whole (and not in a version(unittest) block, which would disguise the problem and cause failures between unittest release builds vs. release builds).
Dec 19 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 19/12/13 00:08, Walter Bright wrote:
 4. Break kitchen sink modules like std.algorithm into one module per algorithm.
 This should not result in code duplication.

That might be desirable for other purposes anyway (std.algorithm is big!), but how does this differ from the existing possibility of doing, e.g., import std.algorithm : sort;
Dec 19 2013
prev sibling next sibling parent =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On 2013-12-19 10:56, Joseph Rushton Wakeling wrote:
 On 19/12/13 00:08, Walter Bright wrote:
 4. Break kitchen sink modules like std.algorithm into one module per
 algorithm.
 This should not result in code duplication.

That might be desirable for other purposes anyway (std.algorithm is big!), but how does this differ from the existing possibility of doing, e.g., import std.algorithm : sort;

Your line of code imports all of std.algorithm, as well as all modules imported by std.algorithm, etc. This includes functions used by topN, largestPartialIntersectionWeighted, findSplit, and so on. A modularized std.algorithm.sort would only import what sort needs (and what sort's needs need, and so on, but likely a much small graph). "But why can't we make the compiler smrter, so import std.algorithm : sort; only imports what's needed?" you might ask. I believe it's already been covered in this thread: overloads, for one (knowledge of all overloads is required). Second, if an imported type is used, how should the compiler guess which module contains the type? There may be more reasons, too. -- Simen
Dec 19 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 19/12/13 12:33, Simen Kjærås wrote:
 Your line of code imports all of std.algorithm, as well as all modules imported
 by std.algorithm, etc. This includes functions used by topN,
 largestPartialIntersectionWeighted, findSplit, and so on.

Ah, OK. Thanks for the clarification. A shame, but understandable.
Dec 19 2013
prev sibling next sibling parent "Idan Arye" <GenericNPC gmail.com> writes:
On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei 
Alexandrescu wrote:
 On 12/18/13 1:16 PM, Dmitry Olshansky wrote:
 19-Dec-2013 01:06, Andrei Alexandrescu пишет:
 http://chopapp.com/#fvepfd8 shows the number of dependencies 
 (plus 1)
 for each module in phobos. Those include druntime 
 dependencies.

So the bill of using pretty much anything in Phobos is pulling in 87 modules. Pretty much what I feared it is.

There are several directions we can take this. 1. Improve the compiler to handle imports lazily, i.e. an unused import is never opened. That's unlikely to help a lot of uses because most unqualified name lookups require all imports to be loaded (even after the name if resolved, the compiler must still look for ambiguities). 2. Push imports from top level into the entities (functions, classes etc) that use them. 3. Apply classic dependency management (break larger modules in smaller ones, accept some code duplication etc). I favor (2). Andrei

I suggested 2 a half a year ago - http://forum.dlang.org/thread/rrjaopnvapwhbatlsyvo forum.dlang.org One of the replys(http://forum.dlang.org/thread/rrjaopnvapwhbatlsyvo forum.dlang.org#post-heaeasepzokrwdsvytb :40forum.dlang.org) mentioned an issue(https://d.puremagic.com/issues/show_bug.cgi?id=7016) that blocks this solution.
Dec 19 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 19/12/13 13:55, Jacob Carlborg wrote:
 Sounds more reasonable.

Is there any reason why one couldn't have subpackages of packages, and so allow imports to be as fine-grained as the user wants? std.algorithm -> std.algorithm.sorting -> std.algorithm.sorting.quickSort ...
Dec 19 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 18 December 2013 at 21:48:14 UTC, Dmitry Olshansky 
wrote:
 4. Split modules at least into a package of with at least 2 
 parts:
 - Meta (traits) - constraints, type definitions etc.
 - API - functions, globals and other "meat" of the module.

Yes, yes, yes. This could also help with encapsulation, since private works across module boundaries. The type listing only has minimal functions for its interface, then other functions are used, with UFCS, from the meat module.
Dec 19 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 19 December 2013 at 13:20:30 UTC, Joseph Rushton 
Wakeling wrote:
 Is there any reason why one couldn't have subpackages of 
 packages, and so allow imports to be as fine-grained as the 
 user wants?

On the other hand, going too far might backfire, as more modules might lead to additional bloat with moduleinfo in the exe and more compile time cuz of more file loading/seeking. I don't know if this will matter or not, but something we should test to know for sure.
Dec 19 2013
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 19 December 2013 at 07:26:54 UTC, Meta wrote:
 Maybe it's not possible, I'm not well versed on how the 
 compiler works. But I think that one module per algorithm may 
 be too granular. Hasn't someone suggested splitting it up by 
 category, e.g., sorting, mutation, searching, etc.?

Previous discussions about this have come up with just using the categories already present in std.algorithm's documentation so it'd be: std.algorithm.searching std.algorithm.iteration std.algorithm.sorting std.algorithm.set std.algorithm.mutation Module tenses and exact naming could change, of course, but just those categories would break it up pretty evenly while keeping similar things close together. Jonathan could probably give some guidance here because as far as I know he's the only person that has any experience attempting to break up a large phobos module into a package (std.datetime). I'm not sure what he has left or what blockers remain for finishing that up.
Dec 19 2013
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei 
Alexandrescu wrote:
 2. Push imports from top level into the entities (functions, 
 classes etc) that use them.

2 comes with its own problems, though. With imports at the top, you can easily tell what the module imports at a glance. Putting imports into the deepest possible scopes where they are used will make it a huge chore to hunt all the imports down. You can of course grep for "import" (or ctrl+f for more plebeian editors), but you still don't have that singular list in one place that you can easily look at to tell what the module imports. Also, as was mentioned before, some imports (or lack of) will go unnoticed until the particular template or template path that it resides in gets instantiated. I don't relish the idea of thinking that my code is all clear, then getting some missing dependency error down the line after refactoring. This will make code more brittle.
Dec 20 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/20/13, Meta <jared771 gmail.com> wrote:
 2 comes with its own problems, though. With imports at the top,
 you can easily tell what the module imports at a glance. Putting
 imports into the deepest possible scopes where they are used will
 make it a huge chore to hunt all the imports down.

Why is that important though? You only need to worry about these scoped imports if you actually use the code that has the scoped imports, rather than preemptively worry about imports at the top.
 I don't relish the idea of thinking
 that my code is all clear, then getting some missing dependency
 error down the line after refactoring. This will make code more
 brittle.

If you don't unittest your templates (which would catch these types of errors), then you have much more to worry about.
Dec 20 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/20/13, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 Local imports are the future. Better get used to them.

To be fair, it's modularization that is the future. And that implies modules that don't stand tall at 8K, 16K, 32K lines+. If the modules have less code and the code is limited in scope then you automatically need less imports.
Dec 20 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 20/12/13 10:06, Meta wrote:
 2 comes with its own problems, though. With imports at the top, you can easily
 tell what the module imports at a glance. Putting imports into the deepest
 possible scopes where they are used will make it a huge chore to hunt all the
 imports down. You can of course grep for "import" (or ctrl+f for more plebeian
 editors), but you still don't have that singular list in one place that you can
 easily look at to tell what the module imports.

Conversely, one problem of imports at the top is that it's easy for unnecessary imports to hang around just because you don't have a direct association between the import and what it's actually used for. I think the real issue we face is that it's not possible to really gain the advantages of deeply-nested-as-possible imports without _also_ breaking up the modules; we can't have Andrei's strategy (2) without some of strategy (3). Example: std.algorithm.topN, the only part of std.algorithm (apart from unittests) that requires std.random: void topN(alias less = "a < b", SwapStrategy ss = SwapStrategy.unstable, Range)(Range r, size_t nth) if (isRandomAccessRange!(Range) && hasLength!Range) { static assert(ss == SwapStrategy.unstable, "Stable topN not yet implemented"); while (r.length > nth) { auto pivot = uniform(0, r.length); // ... etc. ... } } Now, at first glance it's easy to just insert an extra line before "auto pivot = ...": import std.random : uniform; ... but it quickly becomes non-trivial if you want to do what really ought to be an option here, and allow a non-default RNG to be passed to the function: void topN(alias less = "a < b", SwapStrategy ss = SwapStrategy.unstable, Range, RandomGen)(Range r, size_t nth, ref RandomGen rng) if (isRandomAccessRange!(Range) && hasLength!Range && isUniformRNG!RandomGen) // <--- needs std.random.isUniformRNG { static assert(ss == SwapStrategy.unstable, "Stable topN not yet implemented"); while (r.length > nth) { auto pivot = uniform(0, r.length, rng); // ... etc. ... } } // New function to support old 2-parameter version using default RNG void topN(alias less = "a < b", SwapStrategy ss = SwapStrategy.unstable, Range)(Range r, size_t nth) if (isRandomAccessRange!(Range) && hasLength!Range) { topN(r, ss, rndGen); <---- needs std.random.rndGen; } The second of the two needed imports is easy and can be nested inside the function, but the first -- the isUniformRNG template -- requires the import in the module's own scope. So, to really allow std.algorithm to do away with its dependency on std.random, you need to break isUniformRNG and similar templates away from the rest of the std.random functionality. (Yes, I know, you could break topN out from the rest of std.random, but remember that topN itself is only dependent on std.random for the case where you're using the unstable swap strategy.) I'd hypothesize that probably it would be productive to start modularizing Phobos modules by separating out all the various helper templates like isUniformRNG, after which it should be much easier to avoid needing top-level module imports.
Dec 20 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 10:06:01AM +0100, Meta wrote:
 On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei Alexandrescu
 wrote:
2. Push imports from top level into the entities (functions,
classes etc) that use them.

2 comes with its own problems, though. With imports at the top, you can easily tell what the module imports at a glance. Putting imports into the deepest possible scopes where they are used will make it a huge chore to hunt all the imports down. You can of course grep for "import" (or ctrl+f for more plebeian editors), but you still don't have that singular list in one place that you can easily look at to tell what the module imports.

But we're only doing this for Phobos. You can still use global imports in your own modules. I think for Phobos this makes sense, because it's the standard *library*, which means all of its pieces will always be available (being, y'know, standard), and you really only care about what that particular piece of functionality you're using imports. When you import std.algorithm.find, you don't really care about what std.algorithm.cartesianProduct imports, do you?
 Also, as was mentioned before, some imports (or lack of) will go
 unnoticed until the particular template or template path that it
 resides in gets instantiated. I don't relish the idea of thinking
 that my code is all clear, then getting some missing dependency
 error down the line after refactoring. This will make code more
 brittle.

In your own code you can still use imports at the top of the file. We never said we're going to get rid of that. T -- IBM = I Blame Microsoft
Dec 20 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 09:27:55AM -0800, Andrei Alexandrescu wrote:
 On 12/20/13 5:45 AM, Joseph Rushton Wakeling wrote:
On 20/12/13 10:06, Meta wrote:
     void topN(alias less = "a < b",
             SwapStrategy ss = SwapStrategy.unstable,
             Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
         if (isRandomAccessRange!(Range) && hasLength!Range
             && isUniformRNG!RandomGen)  // <--- needs
std.random.isUniformRNG
     {
         static assert(ss == SwapStrategy.unstable,
                 "Stable topN not yet implemented");
         while (r.length > nth)
         {
             auto pivot = uniform(0, r.length, rng);
             // ... etc. ...
         }
     }

I had this idea fot a while, and Walter is favorable of it as well - extend "import" for one-shot use. With that feature the example would become: void topN(alias less = "a < b", SwapStrategy ss = SwapStrategy.unstable, Range, RandomGen)(Range r, size_t nth, ref RandomGen rng) if (isRandomAccessRange!(Range) && hasLength!Range && import.std.random.isUniformRNG!RandomGen) { ... } In this case "import" would syntactically be placed at the beginning of a qualified name, meaning "import this module lazily and look up the symbol in it".

Hmm. Why do we need to incorporate the 'import' keyword in the first place? What about extending symbol lookup, so that if a fully-qualified symbol x.y.z can't be found in the current symbol tables, and x/y exists in the current import path, then implicitly try to import x.y and lookup z in that module. Then you could just write: void f(T)(T t) if (std.range.isInputRange!T) ... and the compiler will automatically import std.range within that scope. Obviously, it's a bad idea to import std.range into module scope, since it would pollute the module namespace, but it seems a good idea to do a one-shot import automatically, since the qualified symbol itself already says which module the symbol is supposed to be defined in. There's no need to add another "import." prefix to it, IMO. T -- Today's society is one of specialization: as you grow, you learn more and more about less and less. Eventually, you know everything about nothing.
Dec 20 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 09:57:46AM -0800, Andrei Alexandrescu wrote:
 On 12/20/13 9:38 AM, H. S. Teoh wrote:
Hmm. Why do we need to incorporate the 'import' keyword in the first
place?

To make the intent explicit. But that's an interesting idea. I'd probably require a leading '.' just to make it unambiguous that lookup must start from top level.
What about extending symbol lookup, so that if a fully-qualified
symbol x.y.z can't be found in the current symbol tables, and x/y
exists in the current import path, then implicitly try to import x.y
and lookup z in that module. Then you could just write:

	void f(T)(T t) if (std.range.isInputRange!T) ...

Due to a bug that's actually the case today to some extent :o).

"That's not a bug, that's an unintentional feature!" :-P T -- By understanding a machine-oriented language, the programmer will tend to use a much more efficient method; it is much closer to reality. -- D. Knuth
Dec 20 2013
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 20 December 2013 at 17:57:46 UTC, Andrei Alexandrescu 
wrote:
 On 12/20/13 9:38 AM, H. S. Teoh wrote:
 What about extending symbol lookup, so that if a 
 fully-qualified
 symbol x.y.z can't be found in the current symbol tables, and 
 x/y exists
 in the current import path, then implicitly try to import x.y 
 and lookup
 z in that module. Then you could just write:

 	void f(T)(T t) if (std.range.isInputRange!T) ...

Due to a bug that's actually the case today to some extent :o). Andrei

In regards to template restraints, it *would* be kind of nice to be able to trigger import only when overload resolution begins. For example, just because a function requires "std.range.isInputRange" to validate its inputs/overloads, doesn't mean it *has* to be globally imported into the entire module. In this sense, it would be nice to be able to define an "import" block for such a function: //---- module wow; void bar(T)(T t) if (isInputRange!T) import { std.range; } body { .... } void foo() { .... } //---- With such an approach, "std.range" only gets imported if a call is "attempted" to bar. if no such call is even attempted, then range isn't imported at all. In this example, "foo" simply does not know about range. It is scoped to bar. As a client, *if* I make no calls to bar, then I will not trigger a dependency to range in wow at all.
Dec 20 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 01:17:29PM -0600, captaindet wrote:
 On 2013-12-20 11:38, H. S. Teoh wrote:

Hmm. Why do we need to incorporate the 'import' keyword in the first
place? What about extending symbol lookup, so that if a
fully-qualified symbol x.y.z can't be found in the current symbol
tables, and x/y exists in the current import path, then implicitly
try to import x.y and lookup z in that module. Then you could just
write:

	void f(T)(T t) if (std.range.isInputRange!T) ...

and the compiler will automatically import std.range within that
scope.  Obviously, it's a bad idea to import std.range into module
scope, since it would pollute the module namespace, but it seems a
good idea to do a one-shot import automatically, since the qualified
symbol itself already says which module the symbol is supposed to be
defined in. There's no need to add another "import." prefix to it,
IMO.


T

one could make it a bit more explicit by requiring import std; at module level, or even more explicit auto import std; to enable lazy import of everything that is hierarchically below. could then be used for user libraries as well:

I like this idea. It could also be denoted as 'lazy import std', that is, if something references 'std.xyz.abc', then import the module std.xyz and lookup the symbol 'abc'. Then you could just use std.range.isInputRange in your signature constraints, and the import will only happen if you actually instantiate that template. A more restricted variant, which may be more palatable to Walter, is to require explicit modules, i.e., you can't just say 'lazy import std', but you have to say 'lazy import std.range': lazy import std.range; void func(T)(T t) if (std.range.isInputRange!T) { ... } So the compiler at least knows that 'std.range' refers to some as-yet unimported module, then when you actually reference a symbol under std.range, the compiler will go and import the module to find its definition. T -- IBM = I'll Buy Microsoft!
Dec 20 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 02:20:12PM -0500, Michel Fortin wrote:
 On 2013-12-20 17:27:55 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:
 
I had this idea fot a while, and Walter is favorable of it as well
- extend "import" for one-shot use. With that feature the example
would become:

     void topN(alias less = "a < b",
             SwapStrategy ss = SwapStrategy.unstable,
             Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
         if (isRandomAccessRange!(Range) && hasLength!Range
             && import.std.random.isUniformRNG!RandomGen)
     { ... }

In this case "import" would syntactically be placed at the
beginning of a qualified name, meaning "import this module lazily
and look up the symbol in it".

This would simplify quite a lot of two-liners into one-liners in
other places, too.

How do you solve the problem that in D you can't tell the module name from a fully qualified names? For instance: import.std.something.somethingelse.anotherthing = 1; Is the module std? std.something? std.something.somethingelse? It could be any of these answers.

Using captaindet's idea of lazy imports, we could solve this problem: // Tell compiler that 'std.range' is a module, but don't import // it just yet. lazy import std.range; void func() { // Since the compiler already knows that 'std.range' is // a module, it knows to now import std.range and look // up 'InputRange.front' in it. auto x = std.range.InputRange.front; } T -- EMACS = Extremely Massive And Cumbersome System
Dec 20 2013
prev sibling next sibling parent "Patrick Down" <patrick.down gmail.com> writes:
On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh wrote:
 in the current import path, then implicitly try to import x.y 
 and lookup
 z in that module. Then you could just write:

 	void f(T)(T t) if (std.range.isInputRange!T) ...

 and the compiler will automatically import std.range within 
 that scope.

How about: scope import std.range; // lazy import std.range; ? void f(T)(T t) if (std.range.isInputRange!T) ...
Dec 20 2013
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Friday, 20 December 2013 at 19:34:10 UTC, Patrick Down wrote:
 On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh wrote:
 in the current import path, then implicitly try to import x.y 
 and lookup
 z in that module. Then you could just write:

 	void f(T)(T t) if (std.range.isInputRange!T) ...

 and the compiler will automatically import std.range within 
 that scope.

How about: scope import std.range; // lazy import std.range; ? void f(T)(T t) if (std.range.isInputRange!T) ...

I think the best keyword to use in this situation would be stati... Oh, dammit, not again.
Dec 20 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 02:40:22PM -0500, Michel Fortin wrote:
 On 2013-12-20 19:36:28 +0000, "Meta" <jared771 gmail.com> said:
 
On Friday, 20 December 2013 at 19:34:10 UTC, Patrick Down wrote:
On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh wrote:
in the current import path, then implicitly try to import x.y and
lookup z in that module. Then you could just write:

	void f(T)(T t) if (std.range.isInputRange!T) ...

and the compiler will automatically import std.range within that
scope.

How about: scope import std.range; // lazy import std.range; ? void f(T)(T t) if (std.range.isInputRange!T) ...

I think the best keyword to use in this situation would be stati... Oh, dammit, not again.

Actually, "static import" already exists. And semantically it's pretty much the same thing as the above: you have to use the symbol's full name. But currently the compiler will import eagerly. I doubt there'd be any breakage if "static" changed to mean "lazily imported".

Hmm. In that case, looks like we already have the solution. :) T -- "Real programmers can write assembly code in any language. :-)" -- Larry Wall
Dec 20 2013
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 20 December 2013 at 19:51:03 UTC, H. S. Teoh wrote:
 On Fri, Dec 20, 2013 at 02:40:22PM -0500, Michel Fortin wrote:
 On 2013-12-20 19:36:28 +0000, "Meta" <jared771 gmail.com> said:
 
On Friday, 20 December 2013 at 19:34:10 UTC, Patrick Down 
wrote:
On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh 
wrote:
in the current import path, then implicitly try to import 
x.y and
lookup z in that module. Then you could just write:

	void f(T)(T t) if (std.range.isInputRange!T) ...

and the compiler will automatically import std.range within 
that
scope.

How about: scope import std.range; // lazy import std.range; ? void f(T)(T t) if (std.range.isInputRange!T) ...

I think the best keyword to use in this situation would be stati... Oh, dammit, not again.

Actually, "static import" already exists. And semantically it's pretty much the same thing as the above: you have to use the symbol's full name. But currently the compiler will import eagerly. I doubt there'd be any breakage if "static" changed to mean "lazily imported".

Hmm. In that case, looks like we already have the solution. :) T

Yes, that's actually quite smart. I like it. +1.
Dec 20 2013
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 20 December 2013 at 21:36:15 UTC, Andrei Alexandrescu 
wrote:
 On 12/20/13 12:43 PM, Martin Nowak wrote:
 Couldn't static imports be made lazy without breaking any code?

One simple idea is to make all imports lazy - no change in semantics. Andrei

If the import is lazy, how do you know which import to load when you see an unknown symbol? lazy static import works, because they have to be fully qualified.
Dec 20 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 10:51:13PM +0100, monarch_dodra wrote:
 On Friday, 20 December 2013 at 21:36:15 UTC, Andrei Alexandrescu
 wrote:
On 12/20/13 12:43 PM, Martin Nowak wrote:
Couldn't static imports be made lazy without breaking any code?

One simple idea is to make all imports lazy - no change in semantics. Andrei

If the import is lazy, how do you know which import to load when you see an unknown symbol? lazy static import works, because they have to be fully qualified.

Yeah, much as I like the idea of all imports being lazy by default, I see a lot of potential problems in the implementation. If you have 10 lazy imports and you reference an unknown symbol, will the compiler now go and import all 10 modules just so it can search them for the unknown symbol? Sounds like recipe for disaster. T -- It only takes one twig to burn down a forest.
Dec 20 2013
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Friday, 20 December 2013 at 19:40:22 UTC, Michel Fortin wrote:
 Actually, "static import" already exists. And semantically it's 
 pretty much the same thing as the above: you have to use the 
 symbol's full name. But currently the compiler will import 
 eagerly. I doubt there'd be any breakage if "static" changed to 
 mean "lazily imported".

Just a joke =)
Dec 20 2013
prev sibling next sibling parent "Chris Cain" <clcain uncg.edu> writes:
On Saturday, 21 December 2013 at 08:08:42 UTC, Dmitry Olshansky 
wrote:
 That has the disadvantage of importing the whole std.range.
 I seriously doubt that we'd get anything better then:

 import std.range.traits;

 void foo(R)(R range) if (isForwardRange!R)
 {

...
 }

Stays practical w.r.t. cutting down dependencies and no need to uglify constraints. They are not that readable already.

I agree, to an extent. That's definitely the ultimate solution that should be taken. Smaller modules are better in general. That said, supporting a lazy static import feature might not be a bad idea. Then a hybrid approach could be taken, which would help things in the short-run. Maybe doing something like this: static import std.range; alias isForwardRange = std.range.isForwardRange; void foo(R)(R range) if (isForwardRange!R) { ... } That would be readable now and would support easy changes to the better packaged approach later. I'd think this would be something possible to do in 2.066 whereas splitting up everything will likely take several versions.
Dec 21 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 21 December 2013 at 08:08:42 UTC, Dmitry Olshansky 
wrote:
 That has the disadvantage of importing the whole std.range.
 I seriously doubt that we'd get anything better then:

 import std.range.traits;

 void foo(R)(R range) if (isForwardRange!R)
 {

...
 }

Stays practical w.r.t. cutting down dependencies and no need to uglify constraints. They are not that readable already.

I think it was mostly agreed here that we should do both for best result.
Dec 21 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 21 December 2013 at 21:16:23 UTC, Andrei 
Alexandrescu wrote:
 That's why I'm saying: make all imports lazy!!!!

How? It has been already mentioned in this thread that this does not seem possible, at least withing existing language.
Dec 21 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/21/13, Dicebot <public dicebot.lv> wrote:
 On Saturday, 21 December 2013 at 21:16:23 UTC, Andrei
 Alexandrescu wrote:
 That's why I'm saying: make all imports lazy!!!!

How? It has been already mentioned in this thread that this does not seem possible, at least withing existing language.

The first step is to avoid reading the imports at all *unless* there's symbols missing. And then try to match selective imports first if there are any missing symbols. To demonstrate, here's the first test-case: ----- module test; import std.stdio : writeln; import std.algorithm; void main() { } ----- The compiler *does not* need to import any other modules (other than object.d, of course), because it doesn't find any missing symbols referenced from "test.d". Test-case 2: ----- import std.stdio : writeln; import std.algorithm; void main() { writeln(""); } ----- The compiler matches the missing symbol with the selective import "writeln", so it knows it only has to load std.stdio, *but not* std.algorithm. Test-case 3: ----- import std.stdio : writeln; import std.algorithm : map; import std.range; // might be here (compiler doesn't know) import std.container; // or here void main() { "foo".front; } ----- The compiler tries to find a selective import "front", but it doesn't find it. What's important here is that it still does not have to load std.stdio or std.algorithm, as we're explicitly loading only a set of symbols which were never referenced from the test.d module. So the next step here is for the compiler to try and load each module in sequence (probably via the declaration order, first std.container), and if there's a match the compiler would stop loading other modules (no need to load std.container if std.range has "front"). ----- I could think of more optimizations, for example if we had a way of exporting a list of module-level symbols into some kind of intermediary format (say JSON), the compiler could look up this list rather than to have to eagerly load every module in search of a symbol. For example: Test-case 4: ----- import std.stdio; import std.algorithm; import std.range; import std.container; void main() { "foo".front; } ----- If we had a "symbols.json", it might list things like: "front": { std.range, std.stdio }, so the compiler would know only to look for this symbol in these modules (and /if/ they are actually imported in the test.d module).
Dec 21 2013
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Saturday, 21 December 2013 at 21:38:59 UTC, Dmitry Olshansky 
wrote:
 As it stands the only thing lazy buys us is "pay as you touch" 
 contrary to "pay as you name the intent to touch". The problem 
 is that the payment is for the whole stock of the said "shop". 
 I see second problem (granularity of imports) as far more 
 critical then the first (condition under which the pieces are 
 imported). The second problem seems solvable within the current 
 implementation, the first seems like it would need arbitrary 
 amount of time to fix and gains are marginal.

We should also keep in mind that as we split up modules and split apart dependencies, it also means that *as* we import a specific package, we are increasing our "use/import" ratio, further diminishing the issue of "import things we don't need." (who would import "std.foo.bar.baz", if they weren't planning to use baz?). Arguably, we'd get "quadratic" effectiveness ;)
Dec 21 2013
prev sibling next sibling parent "Paolo Invernizzi" <paolo.invernizzi gmail.com> writes:
On Saturday, 21 December 2013 at 22:16:13 UTC, monarch_dodra 
wrote:
 We should also keep in mind that as we split up modules and 
 split apart dependencies, it also means that *as* we import a 
 specific package, we are increasing our "use/import" ratio, 
 further diminishing the issue of "import things we don't need." 
 (who would import "std.foo.bar.baz", if they weren't planning 
 to use baz?).

 Arguably, we'd get "quadratic" effectiveness ;)

I agree: I would be happy to have a warning for a module import when any of its symbols is used. That would lead to cleanup in our code... /Paolo
Dec 22 2013
prev sibling next sibling parent Kenji Hara <k.hara.pg gmail.com> writes:
--047d7b5d98ab47cbe504ef45eb83
Content-Type: text/plain; charset=UTF-8

2014/1/6 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>

 On 1/5/14 5:22 PM, Martin Nowak wrote:

 On 12/20/2013 09:43 PM, Martin Nowak wrote:

 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }

Furthermore selective import can always be made lazily without changing code. import std.algorithm : min; Only if `min` or `std.algorithm` are used the compiler needs to open std.algorithm.

I thought more about this and I think that's wrong. Making all imports lazy would be a net large win.

Honestly, "lazy import" (== defer loading module file and running semantic analysis for symbol search) would improve compilation speed for selective imports and static imports, but it would have no merit for basic imports. So precisely, "all imports lazy would be a net large win." is not correct. Kenji Hara --047d7b5d98ab47cbe504ef45eb83 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">2014= /1/6 Andrei Alexandrescu <span dir=3D"ltr">&lt;<a href=3D"mailto:SeeWebsite= ForEmail erdani.org" target=3D"_blank">SeeWebsiteForEmail erdani.org</a>&gt= ;</span><br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"HOEnZb"><div class=3D"h5">On 1= /5/14 5:22 PM, Martin Nowak wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> On 12/20/2013 09:43 PM, Martin Nowak wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <br> Couldn&#39;t static imports be made lazy without breaking any code?<br> The above example would read.<br> <br> static import std.range.<br> <br> void foo(R)(R range) if (std.range.isForwardRange!R)<br> {<br> }<br> </blockquote> <br> Furthermore selective import can always be made lazily without changing<br> code.<br> <br> import std.algorithm : min;<br> <br> Only if `min` or `std.algorithm` are used the compiler needs to open<br> std.algorithm.<br> </blockquote> <br></div></div> I thought more about this and I think that&#39;s wrong. Making all imports = lazy would be a net large win.<br></blockquote><div><br></div><div><div>Hon= estly, &quot;lazy import&quot; (=3D=3D defer loading module file and runnin= g semantic analysis for symbol search) would improve compilation speed for = selective imports and static imports, but it would have no merit for basic = imports.</div> <div><br></div><div>So precisely, &quot;all imports lazy would be a net lar= ge win.&quot; is not correct.</div></div><div><br></div><div>Kenji Hara</di= v></div></div></div> --047d7b5d98ab47cbe504ef45eb83--
Jan 05 2014
prev sibling next sibling parent Kenji Hara <k.hara.pg gmail.com> writes:
--f46d044287f6f7354a04ef4707d7
Content-Type: text/plain; charset=UTF-8

2014/1/6 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>

 On 1/5/14 8:44 PM, Kenji Hara wrote:

 Honestly, "lazy import" (== defer loading module file and running
 semantic analysis for symbol search) would improve compilation speed for
 selective imports and static imports, but it would have no merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." is not correct.

Consider: import std.stdio; void main() { writeln("Hello, world!"); } Currently std.stdio and all modules transitively imported by it would be opened. With lazy imports, std.stdio gets opened and then writeln() gets semantically analyzed. Only modules required by writeln() itself will actually be opened. Big difference.

Thanks for explanation. Indeed it would be big difference. Kenji Hara --f46d044287f6f7354a04ef4707d7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">2014= /1/6 Andrei Alexandrescu <span dir=3D"ltr">&lt;<a href=3D"mailto:SeeWebsite= ForEmail erdani.org" target=3D"_blank">SeeWebsiteForEmail erdani.org</a>&gt= ;</span><br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"im">On 1/5/14 8:44 PM, Kenji H= ara wrote:<br> </div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l= eft:1px #ccc solid;padding-left:1ex"><div class=3D"im"> Honestly, &quot;lazy import&quot; (=3D=3D defer loading module file and run= ning<br> semantic analysis for symbol search) would improve compilation speed for<br=

basic imports.<br> <br></div> So precisely, &quot;all imports lazy would be a net large win.&quot; is not= correct.<br> </blockquote> <br> Consider:<br> <br> import std.stdio;<br> void main() { writeln(&quot;Hello, world!&quot;); }<br> <br> Currently std.stdio and all modules transitively imported by it would be op= ened.<br> <br> With lazy imports, std.stdio gets opened and then writeln() gets semantical= ly analyzed. Only modules required by writeln() itself will actually be ope= ned. Big difference.</blockquote><div><br></div><div>Thanks for explanation= . Indeed it would be big difference.=C2=A0</div> <div><br></div><div>Kenji Hara</div></div></div></div> --f46d044287f6f7354a04ef4707d7--
Jan 05 2014
prev sibling next sibling parent Robert Schadek <realburner gmx.de> writes:
On 12/21/2013 10:43 PM, Andrej Mitrovic wrote:
 The compiler matches the missing symbol with the selective import
 "writeln", so it knows it only has to load std.stdio, *but not*
 std.algorithm.

 Test-case 3:
 -----
 import std.stdio : writeln;
 import std.algorithm : map;
 import std.range;  // might be here (compiler doesn't know)
 import std.container;  // or here

 void main()
 {
     "foo".front;
 }
 -----

 The compiler tries to find a selective import "front", but it doesn't
 find it. What's important here is that it still does not have to load
 std.stdio or std.algorithm, as we're explicitly loading only a set of
 symbols which were never referenced from the test.d module.

 So the next step here is for the compiler to try and load each module
 in sequence (probably via the declaration order, first std.container),
 and if there's a match the compiler would stop loading other modules
 (no need to load std.container if std.range has "front").

have a matching .front? If you stop after the first you'll not find that conflict. If you search both you'll not have any speed gain.
Jan 07 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 6 January 2014 at 07:02:03 UTC, Andrei Alexandrescu 
wrote:
 On 1/5/14 10:31 PM, Walter Bright wrote:
 On 1/5/2014 9:22 PM, Andrei Alexandrescu wrote:
 On 1/5/14 8:44 PM, Kenji Hara wrote:
 Honestly, "lazy import" (== defer loading module file and 
 running
 semantic analysis for symbol search) would improve 
 compilation speed for
 selective imports and static imports, but it would have no 
 merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." 
 is not
 correct.

Consider: import std.stdio; void main() { writeln("Hello, world!"); } Currently std.stdio and all modules transitively imported by it would be opened. With lazy imports, std.stdio gets opened and then writeln() gets semantically analyzed. Only modules required by writeln() itself will actually be opened. Big difference.

import bar; import foo; Importing foo can never be done lazily if there are unqualified references to symbols in bar.

Yah, but modules transitively imported in foo and bar need not be loaded eagerly. That's where the win comes from. Took me a while to figure. Andrei

Right, but this fails quite quickly. Consider this: //---- module Foo; import tons_of_imports_here; void foo() { unqualified_call(); } //---- import Foo; void main() { foo(); } //---- The *instant* the implementation makes an unqualified call, you *have* to import *all* the imports of the module, to figure out what to call. Given that you *probably* imported "foo" with the plan to *use* one of its functions, you'll encounter an unqualified call sooner rather than later, and any "win" will promptly be lost.
Jan 07 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 1/7/14, Robert Schadek <realburner gmx.de> wrote:
 There is a flaw in that design. What if std.range and std.container both
 have a matching .front? If you stop after the first you'll not find that
 conflict. If you search both you'll not have any speed gain.

Ah damn, you're right. It was too good to be true.
Jan 07 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 7 January 2014 at 12:02:32 UTC, Andrej Mitrovic wrote:
 On 1/7/14, Robert Schadek <realburner gmx.de> wrote:
 There is a flaw in that design. What if std.range and 
 std.container both
 have a matching .front? If you stop after the first you'll not 
 find that
 conflict. If you search both you'll not have any speed gain.

Ah damn, you're right. It was too good to be true.

Still pretty good with static / selective imports, just no magic ;)
Jan 07 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-18 22:06, Andrei Alexandrescu wrote:
 http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1)
 for each module in phobos. Those include druntime dependencies.

What is this page, it takes 15 seconds to load the content. -- /Jacob Carlborg
Dec 19 2013
next sibling parent reply "Craig Dillabaugh" <cdillaba cg.scs.carleton.ca> writes:
On Thursday, 19 December 2013 at 13:00:04 UTC, Jacob Carlborg
wrote:
 On 2013-12-18 22:06, Andrei Alexandrescu wrote:
 http://chopapp.com/#fvepfd8 shows the number of dependencies 
 (plus 1)
 for each module in phobos. Those include druntime dependencies.

What is this page, it takes 15 seconds to load the content.

I noticed the same thing. When I loaded the page I heard my disk crank up and the computer working like crazy. I was expecting some magnificent graphical display, and then I got a list of 100 or so lines of plain text. Maybe it uses the Javascript hypenator or something. After all it was Andrei that posted it :o)
Dec 19 2013
parent Martin Nowak <code dawg.eu> writes:
On 12/19/2013 04:13 PM, Craig Dillabaugh wrote:
 Maybe it uses the Javascript hypenator or something.  After all
 it was Andrei that posted it :o)

Hyphenate with D. http://code.dlang.org/packages/hyphenate If someone would wire it up with http://code.dlang.org/packages/gumbo-d it could run on HTML files.
Dec 19 2013
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/20/13, Martin Nowak <code dawg.eu> wrote:
 On 12/19/2013 04:13 PM, Craig Dillabaugh wrote:
 Maybe it uses the Javascript hypenator or something.  After all
 it was Andrei that posted it :o)

Hyphenate with D. http://code.dlang.org/packages/hyphenate

P.S. That homepage link leads to a dead link. The repo link works though.
Dec 20 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 19 Dec 2013 10:56:19 +0100
schrieb Joseph Rushton Wakeling <joseph.wakeling webdrake.net>:

 On 19/12/13 00:08, Walter Bright wrote:
 4. Break kitchen sink modules like std.algorithm into one module per algorithm.
 This should not result in code duplication.

That might be desirable for other purposes anyway (std.algorithm is big!), but how does this differ from the existing possibility of doing, e.g., import std.algorithm : sort;

Selective imports are even slower than normal imports. You still have to load all of std.algorithm to find and disambiguate the symbol. Only when you split up the file physically it will reduce the compiler's workload. -- Marco
Dec 19 2013
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 21 Dec 2013 22:43:58 +0100
schrieb Andrej Mitrovic <andrej.mitrovich gmail.com>:

 I could think of more optimizations, for example if we had a way of
 exporting a list of module-level symbols into some kind of
 intermediary format (say JSON), the compiler could look up this list
 rather than to have to eagerly load every module in search of a
 symbol.

We have .di files already, which could cut on the amount of parsing required. Other than that your idea for partially lazy imports looks like a good first optimization to me. Selective imports always override symbols of the same name in other imported modules, right? -- Marco
Dec 22 2013