www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Time to destroy Walter: breaking modules into packages

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
https://github.com/D-Programming-Language/dmd/pull/2139

Andrei
Jun 05 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 https://github.com/D-Programming-Language/dmd/pull/2139

Probably you already know the following things because they see, similar solutions. Python uses "__init__.py" instead of "package.d", it can also be empty: http://docs.python.org/2/tutorial/modules.html __init__.py can also execute initialization code for the package, like static this() in D. Inside __init__.py you can also define a "__all__" variable where you list all the modules imported when "from package import *" is used.
If __all__ is not defined, the statement from sound.effects 
import * does not import all submodules from the package 
sound.effects into the current namespace; it only ensures that 
the package sound.effects has been imported (possibly running 
any initialization code in __init__.py) and then imports 
whatever names are defined in the package. This includes any 
names defined (and submodules explicitly loaded) by __init__.py. 
It also includes any submodules of the package that were 
explicitly loaded by previous import statements.<

Bye, bearophile
Jun 05 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 3:31 PM, Andrei Alexandrescu wrote:
 https://github.com/D-Programming-Language/dmd/pull/2139

Well, I am destroyed. I need to do some more engineering. The problem is that packages and modules have become conflated with this change.
Jun 05 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 5:13 PM, Walter Bright wrote:
 On 6/5/2013 3:31 PM, Andrei Alexandrescu wrote:
 https://github.com/D-Programming-Language/dmd/pull/2139

Well, I am destroyed. I need to do some more engineering. The problem is that packages and modules have become conflated with this change.

Fixed and reopened.
Jun 05 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 05 Jun 2013 18:31:22 -0400
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 https://github.com/D-Programming-Language/dmd/pull/2139
 
 Andrei

To those who don't want to see a < 1000 lines module: Please consider that as soon as we'd merge multiple compression algorithms into one file for the sake of making the module "large enough", it will turn back on us when we decide to implement any larger number of those: http://en.wikipedia.org/wiki/Category:Lossless_compression_algorithms I see one down side though: Some algorithms are alterations of others and could reuse code. For the sake of speaking module names I'd still split the compression algorithms into multiple modules and probably let one import the other. -- Marco
Jun 05 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 04:10:26 Marco Leise wrote:
 Am Wed, 05 Jun 2013 18:31:22 -0400
 
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:
 https://github.com/D-Programming-Language/dmd/pull/2139
 
 Andrei

To those who don't want to see a < 1000 lines module: Please consider that as soon as we'd merge multiple compression algorithms into one file for the sake of making the module "large enough", it will turn back on us when we decide to implement any larger number of those: http://en.wikipedia.org/wiki/Category:Lossless_compression_algorithms I see one down side though: Some algorithms are alterations of others and could reuse code. For the sake of speaking module names I'd still split the compression algorithms into multiple modules and probably let one import the other.

If each of the compression algorithms is in its own module which is in the same package as other compression modules, then they can use package access to share stuff that would be specific to them but not belong in Phobos in general. But I believe that package level access only works on the same level, so you couldn't have separate modules for compressing and decompressing as was being suggested. It would need to be more like std.compress.zlib; std.compress.lzw; At that point, it would be trivial to have a common module of some kind with shared functionality which has package access level. - Jonathan M Davis
Jun 05 2013
prev sibling next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Thursday, 6 June 2013 at 02:36:12 UTC, Jonathan M Davis wrote:

 But I believe that package level access only works on the same 
 level, so you couldn't have separate modules for compressing 
 and decompressing as was being suggested. It would need to be 
 more like

 std.compress.zlib;
 std.compress.lzw;

 At that point, it would be trivial to have a common module of 
 some kind with
 shared functionality which has package access level.

 - Jonathan M Davis

+1 That seems to be the right level of granularity.
Jun 05 2013
prev sibling next sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 6 June 2013 at 02:36:12 UTC, Jonathan M Davis wrote:

 But I believe that package level access only works on the same 
 level, so you
 couldn't have separate modules for compressing and 
 decompressing as was being
 suggested.

'package' should be fixed so that 'package' declarations are accessible within nested packages.
Jun 05 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 08:09:38 Max Samukha wrote:
 On Thursday, 6 June 2013 at 02:36:12 UTC, Jonathan M Davis wrote:
 But I believe that package level access only works on the same
 level, so you
 couldn't have separate modules for compressing and
 decompressing as was being
 suggested.

'package' should be fixed so that 'package' declarations are accessible within nested packages.

Well, it _is_ debatable as to whether that's desirable or not. With the current behavior, you can have a package which shares stuff within itself but not with its sub-packages, but there's no way to share with the sub-packages without making the symbols public; whereas if sub-packages have access to their parent packages' package functions, then packages _can_ share with their sub-packages, but they can't restrict anything to just the package. Both ways have their pros and cons. I don't know which is ultimately better. - Jonathan M Davis
Jun 05 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 05 Jun 2013 22:35:59 -0400
schrieb "Jonathan M Davis" <jmdavisProg gmx.com>:

 std.compress.zlib;
 std.compress.lzw;
 
 At that point, it would be trivial to have a common module of some kind with 
 shared functionality which has package access level.
 
 - Jonathan M Davis

+1 If package is ambiguous, maybe it needs to be split in two or take a parameter package(std.algorithm) or something like that to serve both use cases. -- Marco
Jun 05 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 08:31:57 Marco Leise wrote:
 If package is ambiguous, maybe it needs to be split in two or
 take a parameter package(std.algorithm) or something like that
 to serve both use cases.

While that might be nice, I'm inclined to think that it would be better to avoid the extra complication and just pick the behavior that seems the most useful. In neither case is the loss all that large, particularly since package is likely to be used fairly rarely. It's quite useful to be sure, but it has a narrow enough set of typical use cases that complicating it probably isn't worth it. - Jonathan M Davis
Jun 06 2013
prev sibling next sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 6 June 2013 at 06:26:17 UTC, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 08:09:38 Max Samukha wrote:
 On Thursday, 6 June 2013 at 02:36:12 UTC, Jonathan M Davis 
 wrote:
 But I believe that package level access only works on the 
 same
 level, so you
 couldn't have separate modules for compressing and
 decompressing as was being
 suggested.

'package' should be fixed so that 'package' declarations are accessible within nested packages.

Well, it _is_ debatable as to whether that's desirable or not. With the current behavior, you can have a package which shares stuff within itself but not with its sub-packages, but there's no way to share with the sub-packages without making the symbols public; whereas if sub-packages have access to their parent packages' package functions, then packages _can_ share with their sub-packages, but they can't restrict anything to just the package. Both ways have their pros and cons. I don't know which is ultimately better.

There is not much to debate. Get your hands dirty with a real project having more than two levels of packages and you'll see that the first is ultimately better. While limiting 'package' to one level may be sometimes desirable, forcing package-private members to be world-public is highly undesirable. As a package author I can tolerate loose access policies within a package tree that *I* control but I definitely don't want to expose to the user of my package what he shouldn't have access to.
Jun 06 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 6/6/13, Max Samukha <maxsamukha gmail.com> wrote:
 There is not much to debate. Get your hands dirty with a real
 project having more than two levels of packages and you'll see
 that the first is ultimately better.

I've argued the same thing before. And as a cream on top I'd like allowing package on virtual methods. That way I can have internal virtual methods which can be extended in other modules or modules in subpackages (if the two features are supported). This allows me to specialize behavior in subclasses, but at the same time disallows the user from calling such methods (a final protected override comes close, except it will still be callable in user-code). Currently the way I work around this is to prepend an underscore to a public virtual and not document the method. It's very much a convention thing. I could instead use some kind of template mixin.. but those are so full of bugs that I eagerly avoid them.
Jun 06 2013
prev sibling next sibling parent reply "nazriel" <spam dzfl.pl> writes:
On Wednesday, 5 June 2013 at 22:31:21 UTC, Andrei Alexandrescu 
wrote:
 https://github.com/D-Programming-Language/dmd/pull/2139

 Andrei

I had a spin[1] with it long before Adam Wilson made it so popular topic. Feature was frequently mentioned on IRC as desired. Unfortunately I faced few holes in both Martin's and Andrei's DIPs and gave up on implementing it. Few issues I remember were: --- module std.datetime.core; class Foo {} module std.datetime.calendar; class Foo {} module std.datetime; public import std.datetime.core; public import std.datetime.calendar; module usage; import std.datetime; Foo x; // is it std.datetime.core or std.datetime.calender? // ok workaround.. std.datetime.Foo x; // same issue as above std.datetime.core.Foo x; // works atleast --- My point is, are directions Jonathan wants to go are right? I think std.datetime.Foo shouldn't be available. I remember also having some issues with ambiguous symbols. I had more corner cases in IRC Logs, but can't find it now. Also I see we are going with Andrei's DIP route. It is worth noting that Martin's one may have some additional functionality. Like glueing modules. Ie. --- module std.net; public import std.net.http; public import std.net.uri; shared(Uri) uri = new shared(Uri); --- In Martin's proposal uri symbol is reachable via std.net.__init__.uri in case of ambiguous. In Andrei's it is unreachable due to package being keyword. I am glad Walter is tackling it, because it is really useful feature, but please take a chill pill and rethink all corner cases. [1] https://github.com/nazriel/dmd/tree/dip16++
Jun 06 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 11:09 AM, Jonathan M Davis wrote:
 We've already discussed this at length. It's possible that we missed
 something, but this proposal is not something that we just jumped into without
 thinking it through first. In fact, it actually took quite a bit to talk Walter
 into the necessity of it in the first place.

The only reason I'm for it is because we need to break up std.algorithm (for the canonical example) without breaking existing code. I'm sure the problem with: simple module => kitchen sink monstrosity => break up module is a recurring issue that most everyone has sooner or later. We need to support it.
Jun 06 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 13:47:29 Andrej Mitrovic wrote:
 I've argued the same thing before. And as a cream on top I'd like
 allowing package on virtual methods. That way I can have internal
 virtual methods which can be extended in other modules or modules in
 subpackages (if the two features are supported). This allows me to
 specialize behavior in subclasses, but at the same time disallows the
 user from calling such methods (a final protected override comes
 close, except it will still be callable in user-code).

Well, since Walter seems to have been convinced in the "Slow performance compared to C++, ideas?" thread to make it so that non-virtual is the default, the presumably it could be changed so that package could be virtual when it's marked as virtual. - Jonathan M Davis
Jun 06 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 16:53:37 nazriel wrote:
 Few issues I remember were:
 
 module std.datetime.core;
 class Foo {}
 
 module std.datetime.calendar;
 class Foo {}
 
 module std.datetime;
 
 public import std.datetime.core;
 public import std.datetime.calendar;
 
 module usage;
 
 import std.datetime;
 
 Foo x; // is it std.datetime.core or std.datetime.calender?
 
 // ok workaround..
 std.datetime.Foo x; // same issue as above
 
 std.datetime.core.Foo x; // works atleast

If you have that problem, then you don't publicly import the entire module. It's up to the package designer to decide which portions of the package get publicly imported. But since std.datetime.Foo would be ambiguous, I don't think that it really matters anyway. In that case, you're simply forced to refer to it by its actual module rather than by the package. The only time that something like this is likely to be a concern is when you add a conflicting function to another module in the package, because then std.datime.Foo was working and then ceased to work. That's arguably cause for being careful about what you choose to publicly import in package.d, but I don't think that it invalidates the feature design at all.
 Also I see we are going with Andrei's DIP route.

How so? What Walter has done is almost identical to DIP 37. I believe that the only difference is that std.datetime.package would have module std.datetime; at the top, whereas DIP 37 currently says that it would have module std.datetime.package;
 I am glad Walter is tackling it, because it is really useful
 feature, but please take a chill pill and rethink all corner
 cases.

We've already discussed this at length. It's possible that we missed something, but this proposal is not something that we just jumped into without thinking it through first. In fact, it actually took quite a bit to talk Walter into the necessity of it in the first place. - Jonathan M Davis
Jun 06 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 11:16:03 Walter Bright wrote:
 On 6/6/2013 11:09 AM, Jonathan M Davis wrote:
 We've already discussed this at length. It's possible that we missed
 something, but this proposal is not something that we just jumped into
 without thinking it through first. In fact, it actually took quite a bit
 to talk Walter into the necessity of it in the first place.

The only reason I'm for it is because we need to break up std.algorithm (for the canonical example) without breaking existing code. I'm sure the problem with: simple module => kitchen sink monstrosity => break up module is a recurring issue that most everyone has sooner or later. We need to support it.-

Yes. If it weren't for the need to break up modules in-place without breaking code, then I'd argue for simply using the foo/all.d idiom where all.d publicly imports all of the modules in foo. Then you just import foo, and you get everything. IMHO, that solves the need for importing packages well enough to avoid needing a language change for it. People have been using it for that already. But it _doesn't_ allow you to break up modules in place without breaking code, which is why I think that this DIP is worth implementing. So, I think that we're in pretty strong agreement about it at this point. As the C# article on virtuality that we've been discussing underlines, features which promote being able to evolve code can be very valuable, and I think that we have some good additions in that camp over languages like C++ or Java (alias this and deprecated in particular come to mind) - maybe not as many as would be nice, but we do have some solid features for code evolution. And I think that this will be a good addition to that. - Jonathan M Davis
Jun 06 2013
prev sibling next sibling parent "Sad panda" <sunspyre gmail.com> writes:
Can we use some other word for counterargument than destroy 
please. :(

(resume discussion)
Jun 06 2013
prev sibling next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Thu, 06 Jun 2013 22:59:41 +0200, Sad panda <sunspyre gmail.com> wrote:

 Can we use some other word for counterargument than destroy please. :(

 (resume discussion)

But we like destroy. -- Simen
Jun 07 2013
prev sibling next sibling parent "nazriel" <spam dzfl.pl> writes:
On Thursday, 6 June 2013 at 18:10:11 UTC, Jonathan M Davis wrote:
 If you have that problem, then you don't publicly import the 
 entire module.
 It's up to the package designer to decide which portions of the 
 package get
 publicly imported. But since std.datetime.Foo would be 
 ambiguous, I don't
 think that it really matters anyway. In that case, you're 
 simply forced to
 refer to it by its actual module rather than by the package. 
 The only time
 that something like this is likely to be a concern is when you 
 add a
 conflicting function to another module in the package, because 
 then
 std.datime.Foo was working and then ceased to work. That's 
 arguably cause for
 being careful about what you choose to publicly import in 
 package.d, but I
 don't think that it invalidates the feature design at all.

Ok, after carefully rereading thread now I understand your point. Preventing breakage when std.datetime is splited into sub packages - ok. But still it looks weird to me that allowing access to std.datetime.time.Clock via std.datetime.Clock, even though Clock is defined in std.datetime.time and std.datetime.package is only publicly importing it, is ok. I guess whatever suits you :))
 Also I see we are going with Andrei's DIP route.

How so? What Walter has done is almost identical to DIP 37. I believe that the only difference is that std.datetime.package would have module std.datetime; at the top, whereas DIP 37 currently says that it would have module std.datetime.package;

Ach pardon, I am lost in all those DIPs. I recalled that Andrei's proposed package.d file, those I referred to your DIP37 (which seems to be summary of Andrei's DIP?) as Andrei's
 I am glad Walter is tackling it, because it is really useful
 feature, but please take a chill pill and rethink all corner
 cases.

We've already discussed this at length. It's possible that we missed something, but this proposal is not something that we just jumped into without thinking it through first. In fact, it actually took quite a bit to talk Walter into the necessity of it in the first place.

Sorry, it is just the impression that this feature was requested very long time ago, Martin's and Andrei's DIPs were staging and recently talk by Adam Wilson made it interesting to the D "crew". If it was carefully discussed and I somehow missed those discussion or I am not allowed to see them, then I am sorry and please ignore this and my previous post in this topic.
 - Jonathan M Davis

Best regards, Damian Ziemba
Jun 07 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 6/7/13, nazriel <spam dzfl.pl> wrote:
 But still it looks weird to me that allowing access to
 std.datetime.time.Clock via std.datetime.Clock, even though Clock
 is defined in std.datetime.time and std.datetime.package is only
 publicly importing it, is ok.

This could come as a benefit. For example, a user might not know exactly which modules a package module is composed of, so he'll instinctively try to type "std.datetime.time" and it will work.
Jun 07 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, June 07, 2013 10:55:36 nazriel wrote:
 If it was carefully discussed and I somehow missed those
 discussion or I am not allowed to see them, then I am sorry and
 please ignore this and my previous post in this topic.

Actually, Daniel Murphy, Martin Nowak, and I discussed it with Walter and Andrei at dconf the night of Adam's talk, so it wasn't on the newsgroup. There's nothing secret about it though. It's just that it was discussed in person, so there is no record. I believe that some further discussion has occured in the newsgroup in the two pull requests for it as well as the DIP announcement, but we went into a fair bit of detail in the discussion at dconf of what exactly the pros and cons would be as well as the side effects. What's really nice about DIP 37 is that it takes advantage of a number of existing features rather than requiring much new. It's essentially just taking what currently works with a module like all.d (publicly importing the everything in the package in the one module) and making it work in a manner that we can avoid code breakage. And naming the file package.d is a particular stroke of genius (provided by someone in the original discussion on Martion Nowak's DIP on the issue IIRC), because then the file can't even conflict with any existing code. So, while the implementation does have some definite nuances to it (like making sure that using the explict path to something like std.datetime.DosFileTime still works), it's ultimately fairly simple and straightforward. And given the need to break up modules in place, we decided that it was worth implementing. - Jonathan M Davis
Jun 07 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Thursday, 6 June 2013 at 06:09:41 UTC, Max Samukha wrote:
 'package' should be fixed so that 'package' declarations are 
 accessible within nested packages.

I don't think the 'package' access specifier should give privileges to *all* nested packages. Rather, I think we just need to invent a new word, and then specifying what 'package' access specifier does becomes easy. My suggestion for this new word is "module-pack"... Module-pack: "A folder which has a file called package.d inside it. Can be imported as if it was a module. etc." ...then, the definition of 'package' access modifier would be: "The label package can be specified at class level, outside all classes (module-level), or inside a struct. In all contexts, package introduces protection that allows access to the symbol to all modules and module-packs which are either within the same directory as the current module or within the same directory as the module-pack which holds the current module within. Non-module-pack subdirectories and the parent directory of the current module’s directory have no special privileges." Basically we'd pretend that module-packs are like modules.
Jun 18 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, June 19, 2013 06:49:49 TommiT wrote:
 On Thursday, 6 June 2013 at 06:09:41 UTC, Max Samukha wrote:
 'package' should be fixed so that 'package' declarations are
 accessible within nested packages.

I don't think the 'package' access specifier should give privileges to *all* nested packages. Rather, I think we just need to invent a new word, and then specifying what 'package' access specifier does becomes easy. My suggestion for this new word is "module-pack"... =20 Module-pack: "A folder which has a file called package.d inside it. Can be imported as if it was a module. etc." =20 ...then, the definition of 'package' access modifier would be: =20 "The label package can be specified at class level, outside all classes (module-level), or inside a struct. In all contexts, package introduces protection that allows access to the symbol to all modules and module-packs which are either within the same directory as the current module or within the same directory as the module-pack which holds the current module within. Non-module-pack subdirectories and the parent directory of the current module=E2=80=99s directory have no special privileges." =20 Basically we'd pretend that module-packs are like modules.

I _really_ don't like the idea of having the presence of package.d affe= ct the=20 package modifier, and I don't think that anything which controls how ma= ny=20 levels deep the package modifier gives the package access to is worth t= he extra=20 complication. If it's worth letting nested packages have access to thei= r=20 parent packages package-level stuff, then let's just do that for everyt= hing.=20 Trying to make it configurable really doesn't buy us much and adds yet = more=20 complication to an already complicated language. - Jonathan M Davis
Jun 18 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 05:35:17 UTC, Jonathan M Davis 
wrote:
 I _really_ don't like the idea of having the presence of 
 package.d affect the package modifier, and I don't think
 that anything which controls how many levels deep the
 package modifier gives the package access to is worth the extra 
 complication.

I don't like the extra complication either, but I think it's the lesser of two evils. Worse is a situation where you would want to break multiple modules, but you can't do it easily because of the way you've structured your code up to that point. On Wednesday, 19 June 2013 at 05:35:17 UTC, Jonathan M Davis wrote:
 If it's worth letting nested packages have access to their 
 parent packages package-level stuff, then
 let's just do that for everything.

But this is not enough. Let's say you have modules A.d and B.d under the same package. Now, if A.d uses something that is marked 'package' inside B.d, you can't break both A.d and B.d into these new module-package thingies without re-structuring your code.
Jun 18 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 06:25:02 UTC, TommiT wrote:
 Now, if A.d uses something that is marked 'package' inside B.d, 
 you can't break both A.d and B.d into these new module-package 
 thingies without re-structuring your code.

And it may not be even possible to re-structure the code, like when the thing marked 'package' is a member function for example.
Jun 18 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, June 19, 2013 08:25:00 TommiT wrote:
 If it's worth letting nested packages have access to their
 parent packages package-level stuff, then
 let's just do that for everything.

But this is not enough. Let's say you have modules A.d and B.d under the same package. Now, if A.d uses something that is marked 'package' inside B.d, you can't break both A.d and B.d into these new module-package thingies without re-structuring your code.

Then just put whatever is supposed to be shared between them in the package that they were both in originally. package stuff is not part of the public API, so when you're refactoring package code, you're refactoring code that you own entirely, so it shouldn't be an issue if you have to rearrange it a bit, especially since it's all localized to a single package by definition. It's only when something is part of a public API that refactoring becomes a problem. - Jonathan M Davis
Jun 18 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 06:25:02 UTC, TommiT wrote:
 I don't like the extra complication either, ...

I should have said, I don't like the added complexity. But I think this is one of those things that becomes totally obvious once you use it for a while and get used to thinking about these packaged-module thingies as just simply modules.
Jun 18 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 06:47:05 UTC, Jonathan M Davis 
wrote:
 Then just put whatever is supposed to be shared between them in 
 the package
 that they were both in originally. package stuff is not part of 
 the public API,
 so when you're refactoring package code, you're refactoring 
 code that you own
 entirely, so it shouldn't be an issue if you have to rearrange 
 it a bit,
 especially since it's all localized to a single package by 
 definition. It's
 only when something is part of a public API that refactoring 
 becomes a
 problem.

 - Jonathan M Davis

I can't put a member function into a different file if it's supposed to have access to private data of its enclosing class/struct.
Jun 18 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, June 19, 2013 08:51:38 TommiT wrote:
 On Wednesday, 19 June 2013 at 06:47:05 UTC, Jonathan M Davis
 
 wrote:
 Then just put whatever is supposed to be shared between them in
 the package
 that they were both in originally. package stuff is not part of
 the public API,
 so when you're refactoring package code, you're refactoring
 code that you own
 entirely, so it shouldn't be an issue if you have to rearrange
 it a bit,
 especially since it's all localized to a single package by
 definition. It's
 only when something is part of a public API that refactoring
 becomes a
 problem.
 
 - Jonathan M Davis

I can't put a member function into a different file if it's supposed to have access to private data of its enclosing class/struct.

Then you can put the class in a different file. But even if you can't, I don't think that it's worth it to complicate the package attribute any further. Even if we lose something in the process, we really don't lose much, and since currently, package only applies to modules directly in the same package and not nested packages, you'd still be gaining over what we currently have. - Jonathan M Davis
Jun 19 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis 
wrote:
 On Wednesday, June 19, 2013 08:51:38 TommiT wrote:
 I can't put a member function into a different file if it's
 supposed to have access to private data of its enclosing
 class/struct.

Then you can put the class in a different file. But even if you can't, I don't think that it's worth it to complicate the package attribute any further.

Let's now assume that we change the language specification so that 'package' access specifier allows access also to modules in nested packages. I'll show how difficult it can be to refactor your code in order to break it up into a package: The file structure: ------------------- /my_lib/mod_x.d /my_lib/mod_y.d File: /my_lib/mod_x.d ---------------------- module my_lib.mod_x; import my_lib.mod_y; void fun() { my_lib.mod_y.S1 s1; s1.detail(); } File: /my_lib/mod_y.d ---------------------- module my_lib.mod_y; struct S1 { public void api() { S2 s2; s2.detail(); } package void detail() { } } struct S2 { public void api() { } package void detail() { } } ################################################################ Now, I want to break up both mod_x.d and mod_y.d into packages. I'll start by doing it naively and end up with this: The file structure: ------------------- /my_lib/mod_x/submod.d /my_lib/mod_x/package.d /my_lib/mod_y/submod_s1.d /my_lib/mod_y/submod_s2.d /my_lib/mod_y/package.d File: /my_lib/mod_x/submod.d ----------------------------- module my_lib.mod_x.submod; import my_lib.mod_y; void fun() { my_lib.mod_y.S1 s1; s1.detail(); } File: /my_lib/mod_x/package.d ------------------------------ module my_lib.mod_x; public import my_lib.mod_x.submod; File: /my_lib/mod_y/submod_s1.d -------------------------------- module my_lib.mod_y.submod_s1; import my_lib.mod_y.submod_s2; struct S1 { public void api() { S2 s2; s2.detail(); } package void detail() { } } File: /my_lib/mod_y/submod_s2.d -------------------------------- module my_lib.mod_y.submod_s2; struct S2 { public void api() { } package void detail() { } } File: /my_lib/mod_y/package.d ------------------------------ module my_lib.mod_y; public import my_lib.mod_y.submod_s1; public import my_lib.mod_y.submod_s2; ################################################################ At this point I notice this doesn't work because my_lib.mod_x.submod.fun tries to call my_lib.mod_y.S1.detail which is marked package and now in different package. In order to fix this without changing the public API, I'll create a new file called shared_s1.d, move the struct S1 there, and public import shared_s1.d back into the S1's original submod_s1.d file: The file structure: ------------------- /my_lib/mod_x/submod.d /my_lib/mod_x/package.d /my_lib/mod_y/submod_s1.d /my_lib/mod_y/submod_s2.d /my_lib/mod_y/package.d /my_lib/shared_s1.d File: /my_lib/shared_s1.d -------------------------- module my_lib.shared_s1; import my_lib.mod_y.submod_s2; struct S1 { public void api() { S2 s2; s2.detail(); } package void detail() { } } File: /my_lib/mod_y/submod_s1.d -------------------------------- module my_lib.mod_y.submod_s1; public import my_lib.shared_s1; (other files stay the same) ################################################################ Now my_lib.mod_x.submod.fun has access rights and is able to call my_lib.mod_y.S1.detail because it's now an alias to my_lib.shared_s1.S1.detail. But then we notice that it doesn't work, because by moving the definition of S1 one folder up, we made it so that my_lib.shared_s1.S1.api isn't able to call my_lib.mod_y.submod_s2.S2.detail. So, we must do the same trick for the struct S2 now. Notice how viral this effect is. We end up with: The file structure: ------------------- /my_lib/mod_x/submod.d /my_lib/mod_x/package.d /my_lib/mod_y/submod_s1.d /my_lib/mod_y/submod_s2.d /my_lib/mod_y/package.d /my_lib/shared_s1.d /my_lib/shared_s2.d File: /my_lib/shared_s2.d -------------------------- module my_lib.shared_s2; struct S2 { public void api() { } package void detail() { } } File: /my_lib/mod_y/submod_s2.d -------------------------------- module my_lib.mod_y.submod_s2; public import my_lib.shared_s2; ################################################################ And now it all works. But what did we learn from all this: Your suggestion makes breaking modules into packages easier to understand but harder to do. My suggestion makes breaking modules into packages harder to understand but easier to do. On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis wrote:
 Even if we lose something in the process, we really don't lose
 much, and since currently, package only applies to modules
 directly in the same package and not nested packages, you'd
 still be gaining over what we currently have.

I don't want 'package' symbols to give away access to nested packages.
Jun 19 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 09:55:32 UTC, TommiT wrote:
 Your suggestion makes breaking modules into packages easier to 
 understand but harder to do.

...and not to mention, just plain ugly. I mean look at how those files that are part of a package end up being in different folders and you have those dummy (shared) files which the end user is not supposed to import.
Jun 19 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis 
wrote:
 [..] But even if you can't, I don't think that it's worth it to
 complicate the package attribute any further. [..]

I'd like to quote the TDPL foreword by Walter: "To the best of my knowledge, D offers an unprecedentedly adroit integration of several powerful programming paradigms: imperative, object-oriented, functional, and meta. At first blush, it would appear that such a language could not be simple. And indeed, D is not a simple language. But I’d argue that is the wrong way to view a language. A more useful view is, what do programming solutions in that language look like? Are D programs complicated and obtuse, or simple and elegant?" -Walter Bright Thus, we shouldn't strive so much to make the language simple (by having the simplest possible definition for the 'package' keyword), but rather, we should try to make programs, which use modules broken into packages, look simple, elegant and easy to understand.
Jun 19 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, June 19, 2013 13:13:54 TommiT wrote:
 On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis
 
 wrote:
 [..] But even if you can't, I don't think that it's worth it to
 complicate the package attribute any further. [..]

I'd like to quote the TDPL foreword by Walter: "To the best of my knowledge, D offers an unprecedentedly adroit integration of several powerful programming paradigms: imperative, object-oriented, functional, and meta. At first blush, it would appear that such a language could not be simple. And indeed, D is not a simple language. But I’d argue that is the wrong way to view a language. A more useful view is, what do programming solutions in that language look like? Are D programs complicated and obtuse, or simple and elegant?" -Walter Bright Thus, we shouldn't strive so much to make the language simple (by having the simplest possible definition for the 'package' keyword), but rather, we should try to make programs, which use modules broken into packages, look simple, elegant and easy to understand.

Sure, but there's a limit to how much complexity is reasonable, and every feature has to pull its own weight. And while I agree that there are cases where refactoring packages may cause complications with the package modifier specfically giving nested packages access to their parent packages' package stuff but not do anything like what you're suggesting, I don't agree that it's a big enough deal to merit complicating the package access level any further. If it's a problem, then maybe the code just plain needs to be redesigned rather than simply shuffled around. If we were really looking for full power with rearranging stuff like that, we'd have friend functions and classes, and D specifically elected _not_ to go for that level of complication. And overall, it's worked great for us thus far. - Jonathan M Davis
Jun 19 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 20:05:56 UTC, Jonathan M Davis
wrote:
 On Wednesday, June 19, 2013 13:13:54 TommiT wrote:
 On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis
 
 wrote:
 [..] But even if you can't, I don't think that it's worth it 
 to
 complicate the package attribute any further. [..]

I'd like to quote the TDPL foreword by Walter: "To the best of my knowledge, D offers an unprecedentedly adroit integration of several powerful programming paradigms: imperative, object-oriented, functional, and meta. At first blush, it would appear that such a language could not be simple. And indeed, D is not a simple language. But I’d argue that is the wrong way to view a language. A more useful view is, what do programming solutions in that language look like? Are D programs complicated and obtuse, or simple and elegant?" -Walter Bright Thus, we shouldn't strive so much to make the language simple (by having the simplest possible definition for the 'package' keyword), but rather, we should try to make programs, which use modules broken into packages, look simple, elegant and easy to understand.

Sure, but there's a limit to how much complexity is reasonable, and every feature has to pull its own weight. And while I agree that there are cases where refactoring packages may cause complications with the package modifier specfically giving nested packages access to their parent packages' package stuff but not do anything like what you're suggesting, I don't agree that it's a big enough deal to merit complicating the package access level any further. If it's a problem, then maybe the code just plain needs to be redesigned rather than simply shuffled around. If we were really looking for full power with rearranging stuff like that, we'd have friend functions and classes, and D specifically elected _not_ to go for that level of complication. And overall, it's worked great for us thus far. - Jonathan M Davis

No, we are not hoping to rearrange stuff like that. We are _forced_ to rearrange stuff like that if we go with your suggestion. With my suggestion, what I presented as the first attempt at breaking modules into packages would just work. (in case you actually read my post) And no, we don't want friend function, I never said that. Also, I'm just curious why do you keep saying "we don't want to complicate the package access specifier any further"? Because isn't the current specification of the package access specifier the simplest possible that it could ever be? "Everything under the same folder has access to symbols labeled package". It takes just 11 words to define it. You're talking about it like it's already somehow complicated.
Jun 19 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 20, 2013 00:21:45 TommiT wrote:
 Also, I'm just curious why do you keep saying "we don't want to
 complicate the package access specifier any further"? Because
 isn't the current specification of the package access specifier
 the simplest possible that it could ever be? "Everything under
 the same folder has access to symbols labeled package". It takes
 just 11 words to define it. You're talking about it like it's
 already somehow complicated.

No, it's not complicated, but the lanugage as a whole is complicated, and any new feature that's added to it increases its complexity. As such, it needs to pull its weight, and I really don't believe that that's the case here. I just don't think that complicating the package access modifier any further is worth the gain. There is some gain, but I think that it's ultimately quite small, and I'd much prefer that access modifiers stay simple. Obviously, you're entitled to think that the extra complexity is worth it, but I don't agree. - Jonathan M Davis
Jun 19 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 22:40:47 UTC, Jonathan M Davis 
wrote:
 On Thursday, June 20, 2013 00:21:45 TommiT wrote:
 Also, I'm just curious why do you keep saying "we don't want to
 complicate the package access specifier any further"? Because
 isn't the current specification of the package access specifier
 the simplest possible that it could ever be? "Everything under
 the same folder has access to symbols labeled package". It 
 takes
 just 11 words to define it. You're talking about it like it's
 already somehow complicated.

No, it's not complicated, but the lanugage as a whole is complicated, and any new feature that's added to it increases its complexity. As such, it needs to pull its weight, and I really don't believe that that's the case here. I just don't think that complicating the package access modifier any further is worth the gain. There is some gain, but I think that it's ultimately quite small, and I'd much prefer that access modifiers stay simple. Obviously, you're entitled to think that the extra complexity is worth it, but I don't agree. - Jonathan M Davis

You're willing to add extra complexity and inconvenience to programming in D just in order to keep the language specification simple. I don't think it's a good trade-off. Learning the complex details of language is a one-time cost that all programmers must pay when they start with the language. Whereas complexity and inconvenience in actually programming with the language is a running cost and may be a source of bugs as well. The running cost should clearly over-weight the one-time cost here.
Jun 19 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, June 20, 2013 05:39:30 TommiT wrote:
 On Wednesday, 19 June 2013 at 22:40:47 UTC, Jonathan M Davis
 
 wrote:
 On Thursday, June 20, 2013 00:21:45 TommiT wrote:
 Also, I'm just curious why do you keep saying "we don't want to
 complicate the package access specifier any further"? Because
 isn't the current specification of the package access specifier
 the simplest possible that it could ever be? "Everything under
 the same folder has access to symbols labeled package". It
 takes
 just 11 words to define it. You're talking about it like it's
 already somehow complicated.

No, it's not complicated, but the lanugage as a whole is complicated, and any new feature that's added to it increases its complexity. As such, it needs to pull its weight, and I really don't believe that that's the case here. I just don't think that complicating the package access modifier any further is worth the gain. There is some gain, but I think that it's ultimately quite small, and I'd much prefer that access modifiers stay simple. Obviously, you're entitled to think that the extra complexity is worth it, but I don't agree. - Jonathan M Davis

You're willing to add extra complexity and inconvenience to programming in D just in order to keep the language specification simple. I don't think it's a good trade-off. Learning the complex details of language is a one-time cost that all programmers must pay when they start with the language. Whereas complexity and inconvenience in actually programming with the language is a running cost and may be a source of bugs as well. The running cost should clearly over-weight the one-time cost here.

I don't think that this adds much complication to programs, because I don't think that the sorts of problems that you're worried about when refactoring code are likely to be common, and they can be gotten around by reworking how your code is designed. Worst case, you just make the functionality public but undocumented. And while that may not be ideal, I really don't think that it will happen enough to be worth the complication to the language required to avoid it. And consider the fact that even with your suggested changes, there are going to be ways that you can't refactor your code and still keep it using the same API and/or the same access level. We can pretty much always find more ways to refactor code that won't work with the access specifiers unless you introduce something like friend which allows you to explicitly specify who can access who rather than using general groups like public and private. And you have to draw the line somewhere. I think that making it so that the package access specifier lets nested packages access it in addition to the exact package is a good balance between simplicity and power. However, if you want to try and convince Walter, feel free. But from what he's said in the past, I suspect that he leans more towards getting rid of package entirely than thinking that it's worth making it more complicated in the manner that you're looking for. - Jonathan M Davis
Jun 19 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
This is how I see the logic of this new feature you're adding:

When you have a folder named "mod" which contains a file named 
package.d, the mod folder becomes conceptually a file with same 
name and the extension "d". I.e. the folder conceptually becomes 
a module, which you can then import to other modules. All the 
code contained within all the files contained within the mod 
folder becomes conceptually the content of this imaginary mod.d 
file. Therefore it is only logical that all the code within this 
imaginary mod.d file would have access to 'package'-labeled 
symbols that are in files which are located under the same folder 
under which this imaginary mod.d file is.
Jun 19 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
Also, if we loosen the encapsulation provided by the package 
access specifier so that it allows access also to modules in 
nested packages, then why is this keyword even named "package". 
Newcomers to the language would be asking us why is this keyword 
names "package", when it provides access beyond the package where 
the module is in. Loosening the encapsulation provided by 
'package' is not logical nor called for.
Jun 19 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Thursday, 20 June 2013 at 05:24:12 UTC, TommiT wrote:
 This is how I see the logic of this new feature you're adding:

 When you have a folder named "mod" which contains a file named 
 package.d, the mod folder becomes conceptually a file with same 
 name and the extension "d". I.e. the folder conceptually 
 becomes a module, which you can then import to other modules. 
 All the code contained within all the files contained within 
 the mod folder becomes conceptually the content of this 
 imaginary mod.d file. Therefore it is only logical that all the 
 code within this imaginary mod.d file would have access to 
 'package'-labeled symbols that are in files which are located 
 under the same folder under which this imaginary mod.d file is.

...and following that thread, I would also argue, that 'private'-labeled symbols in files that are inside this mod folder should be accessible to all the other files that are inside this mod folder, because conceptually all those files are part of the same (imaginary) module (mod.d).
Jun 19 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, June 20, 2013 08:45:21 TommiT wrote:
 On Thursday, 20 June 2013 at 05:24:12 UTC, TommiT wrote:
 This is how I see the logic of this new feature you're adding:
 
 When you have a folder named "mod" which contains a file named
 package.d, the mod folder becomes conceptually a file with same
 name and the extension "d". I.e. the folder conceptually
 becomes a module, which you can then import to other modules.
 All the code contained within all the files contained within
 the mod folder becomes conceptually the content of this
 imaginary mod.d file. Therefore it is only logical that all the
 code within this imaginary mod.d file would have access to
 'package'-labeled symbols that are in files which are located
 under the same folder under which this imaginary mod.d file is.

...and following that thread, I would also argue, that 'private'-labeled symbols in files that are inside this mod folder should be accessible to all the other files that are inside this mod folder, because conceptually all those files are part of the same (imaginary) module (mod.d).

The _only_ reason that we added package.d was to allow us to split up modules in place without breaking existing code. And it works as well as it does, because the _only_ new feature that it provides is making it so that when you import a package, you import package.d. All other aspects about it were already part of the language, and we weren't looking to add any new functionality whatsoever beyond being able to break up modules like std.algorithm or std.datetime. Trying to treat a package as if it were a module because it has a package.d file is contorting things considerably. That was never the intention of package.d at all. It's to help us evolve code without breaking public APIs. package is not part of the public API. private is not part of the public API. As such, they have _nothing_ to do with the purpose of package.d. And it was never intended that packages be treated as modules due to the presence of package.d save for the purpose of avoiding code breakage. IMHO, you are trying to take this _way_ too far. - Jonathan M Davis
Jun 20 2013
prev sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Thursday, 20 June 2013 at 07:59:57 UTC, Jonathan M Davis wrote:
 The _only_ reason that we added package.d was to allow us to 
 split up modules
 in place without breaking existing code. And it works as well 
 as it does,
 because the _only_ new feature that it provides is making it so 
 that when you
 import a package, you import package.d. All other aspects about 
 it were
 already part of the language, and we weren't looking to add any 
 new
 functionality whatsoever beyond being able to break up modules 
 like
 std.algorithm or std.datetime. Trying to treat a package as if 
 it were a
 module because it has a package.d file is contorting things 
 considerably. That
 was never the intention of package.d at all. It's to help us 
 evolve code
 without breaking public APIs. package is not part of the public 
 API. private
 is not part of the public API. As such, they have _nothing_ to 
 do with the
 purpose of package.d. And it was never intended that packages 
 be treated as
 modules due to the presence of package.d save for the purpose 
 of avoiding code
 breakage. IMHO, you are trying to take this _way_ too far.

 - Jonathan M Davis

Ok. Then this feature is not really about being able to easily break an existing module into a package, but rather about providing a different way to organize your code from the get-go. You _can_ break existing modules into packages, but you may need to go through some hoops to do so, and that's fine, because that's not the primary purpose of this feature anyway. I'd say let's keep the package access specifier as it is now.
Jun 20 2013