www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Fix #2529: explicit protection package #3651

reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/19/2014 7:01 AM, Dicebot wrote:
 Walter, now that release is out can you please state your opinion about
 https://github.com/D-Programming-Language/dmd/pull/3651 ? It is blocking Phobos
 module split and decoupling.
I keep thinking there's gotta be a way to do this without language changes.
Aug 19 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:
 On 8/19/2014 7:01 AM, Dicebot wrote:
 Walter, now that release is out can you please state your
opinion about
 https://github.com/D-Programming-Language/dmd/pull/3651 ? It
is blocking Phobos
 module split and decoupling.
I keep thinking there's gotta be a way to do this without language changes.
Any specific ideas? I can't imagine any clean solution - and proposed language extensions fits naturally into existing system without introducing any new concepts. It is also somewhat frequently asked about in NG.
Aug 19 2014
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, 19 August 2014 at 17:11:19 UTC, Dicebot wrote:
 On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:
 On 8/19/2014 7:01 AM, Dicebot wrote:
 Walter, now that release is out can you please state your
opinion about
 https://github.com/D-Programming-Language/dmd/pull/3651 ? It
is blocking Phobos
 module split and decoupling.
I keep thinking there's gotta be a way to do this without language changes.
Any specific ideas? I can't imagine any clean solution - and proposed language extensions fits naturally into existing system without introducing any new concepts. It is also somewhat frequently asked about in NG.
Yeah, I don't see how this could be done without a language change. Currently, modules in sub-packages are treated no differently from modules in completely different packages, so anything you did to give access to a module in a super-package to one in a sub-package would give access to any module. - Jonathan M Davis
Aug 19 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 19 August 2014 at 17:11:19 UTC, Dicebot wrote:
 On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:
 On 8/19/2014 7:01 AM, Dicebot wrote:
 Walter, now that release is out can you please state your
opinion about
 https://github.com/D-Programming-Language/dmd/pull/3651 ? It
is blocking Phobos
 module split and decoupling.
I keep thinking there's gotta be a way to do this without language changes.
Any specific ideas? I can't imagine any clean solution - and proposed language extensions fits naturally into existing system without introducing any new concepts. It is also somewhat frequently asked about in NG.
To put it differently - this issue is a very real blocker for some of planned Phobos changes (pretty much anything that is going to use package.d) and it has been hanging around for a long time. If you are going to propose alternative solution I'd be very glad to see it sooner than later as everyone else who has commented on that topic seems to be satisfied with the proposal.
Aug 19 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Tuesday, 19 August 2014 at 17:11:19 UTC, Dicebot wrote:
 Any specific ideas? I can't imagine any clean solution - and 
 proposed language extensions fits naturally into existing 
 system without introducing any new concepts. It is also 
 somewhat frequently asked about in NG.
--- module std.internal.mod1; package(std) void foo() {} module std.mod2; import std.internal.mod2; void bar() { foo(); } --- Isn't it equivalent to --- module std.internal_mod1; package void foo() {} module std.mod2; import std.internal_mod1; void bar() { foo(); } ---
Aug 20 2014
parent reply ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Wed, 20 Aug 2014 07:25:43 +0000
Kagamin via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

but it requires to dump all 'subpackages' into one flat directory and
with ugly names.

proposed extension will not break any existing code, yet will allow much
nicer hierarchy.
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 07:47:36 UTC, ketmar via 
Digitalmars-d-announce wrote:
 but it requires to dump all 'subpackages' into one flat 
 directory and
 with ugly names.
Huh? If std.datetime and std.regex are subpackages of `std` package, it usually means, they should reside in `std` directory, no? And how explicit package protection can change that?
Aug 20 2014
next sibling parent reply ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Wed, 20 Aug 2014 09:26:36 +0000
Kagamin via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

 Huh? If std.datetime and std.regex are subpackages of `std`=20
 package, it usually means, they should reside in `std` directory,=20
 no? And how explicit package protection can change that?
imagine mybiglib, mybiglib.internal, mybiglib.other, and each of them not single module, but package itself. so we have module mybiglib.other.thingy which wants to use functions from module mybiglib.internal.wisdom, which is part of mybiglib.internal package.
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 09:40:37 UTC, ketmar via 
Digitalmars-d-announce wrote:
 imagine mybiglib, mybiglib.internal, mybiglib.other, and each 
 of them
 not single module, but package itself. so we have module
 mybiglib.other.thingy which wants to use functions from
 module mybiglib.internal.wisdom, which is part of 
 mybiglib.internal
 package.
Do we need a hierarchy of internals, is the problem this big? Why mybiglib.wisdom is not good?
Aug 20 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:33:53 UTC, Kagamin wrote:
 On Wednesday, 20 August 2014 at 09:40:37 UTC, ketmar via 
 Digitalmars-d-announce wrote:
 imagine mybiglib, mybiglib.internal, mybiglib.other, and each 
 of them
 not single module, but package itself. so we have module
 mybiglib.other.thingy which wants to use functions from
 module mybiglib.internal.wisdom, which is part of 
 mybiglib.internal
 package.
Do we need a hierarchy of internals, is the problem this big? Why mybiglib.wisdom is not good?
As I see on the realistic example of datetime, which is BIG, we only need to split it into a flat set of files without an overly deep package hierarchy.
Aug 20 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 14:36:59 UTC, Kagamin wrote:
 As I see on the realistic example of datetime, which is BIG, we 
 only need to split it into a flat set of files without an 
 overly deep package hierarchy.
We _may_ split it into flat set files (solving only part of the problem) but it is desirable to have a deeper package hierarchy. Package hierarchy is not just an encapsulation tool, it is also a great way to simplify navigation and finding needed modules - something that Phobos is current terrible at exactly because of flat hierarchies.
Aug 20 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:40:34 UTC, Dicebot wrote:
 On Wednesday, 20 August 2014 at 14:36:59 UTC, Kagamin wrote:
 As I see on the realistic example of datetime, which is BIG, 
 we only need to split it into a flat set of files without an 
 overly deep package hierarchy.
We _may_ split it into flat set files (solving only part of the problem) but it is desirable to have a deeper package hierarchy. Package hierarchy is not just an encapsulation tool, it is also a great way to simplify navigation and finding needed modules - something that Phobos is current terrible at exactly because of flat hierarchies.
What is difficult to find? With flat structure you have all files right before your eyes. If you need std.datetime.systime module, you open std/datetime/systime.d file - that's the reason of needlessly restricting code structure with modules as if one size fits all.
Aug 23 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 23 August 2014 at 09:00:30 UTC, Kagamin wrote:
 What is difficult to find? With flat structure you have all 
 files right before your eyes. If you need std.datetime.systime 
 module, you open std/datetime/systime.d file - that's the 
 reason of needlessly restricting code structure with modules as 
 if one size fits all.
It is the same reasoning as with deep filesystem hierarchies and, well, any data hierarchies - once the element (module / file) count becomes bigger than ~dozen you only really notice things you know to look for. Contrary to that deeply nested categorized hierarchies are easy to casually to search through if you don't know exact module name - just iteratively pick whatever package fits the theme until you find what you want. I remember coding a bit in C#/.NET platform ages ago - it was totally possible to find relevant modules without even looking in docs, just using auto-complete through suggested package names for import. It was really positive experience for a newbie I was. At the same time a lot of people have no idea how many cool things Phobos actually has.
Aug 23 2014
next sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Sun, 24 Aug 2014 02:39:39 +0000
Dicebot via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

 It is the same reasoning as with deep filesystem hierarchies and,=20
 well, any data hierarchies
i already told that, but without any effects.
Aug 23 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Sunday, 24 August 2014 at 02:22:41 UTC, Dicebot wrote:
 Well difference is that "internal" substring in the fully 
 qualified name that is much more likely to tell user he is 
 better to not touch it. However, original Kagamin proposal of 
 embedding it into module names themselves does address that if 
 you are ok with resulting uglyness.
Both ways it's a convention, and I don't see, why such convention should exist in the first place, member protection has nothing to do with module name, which reflects module's functionality. On Sunday, 24 August 2014 at 02:34:01 UTC, Dicebot wrote:
 It can be a philosophical matter, but in my experience 
 grouping by functionality is genuine, and access is an 
 afterthought, so grouping by access doesn't work in the long 
 term, as code naturally (spontaneously) groups by 
 functionality.
That does contradict your statement that any stuff used in parent packages must go to up the hierarchy which is exactly grouping by access :)
Access there means protection, usage reflects functionality.
 I can probably give a more practical example I have just 
 recently encountered when re-designing one of our internal 
 library packages. It was a "serialization" package and 
 sub-packages defined different (de)serialization models, some 
 of those defining special wrapper types for resulting data that 
 enforce certain semantics specific to that serialization model 
 via the type system. It is not surprising that most internal 
 details of such wrappers were kept package protected as 
 exposing those to user could have easily violated all 
 assumptions (but was needed to implement (de)serialization 
 efficiently).

 It has become more complicated when "meta-serializers" have 
 been added to parent package - templated stuff that took any of 
 sub-package implementations and added some functionality (like 
 versioning support) on top. Being generic thing it reside in 
 higher level "serialization" package but to be implemented 
 efficiently it needs access to those internal wrapper fields. 
 Moving wrapper modules to parent package is not an option here 
 because those are closely related to specific serialization 
 model. Exposing stuff as public is not an option because anyone 
 not deeply familiar with package implementation can easily 
 break all type system guarantees that way. Moving 
 "meta-serializers" to sub-packages is quite a code duplication.

 Right now I keep that stuff public and say in docs "please 
 don't use this" which is hardly a good solution.
1. Making wrapper protected will preclude writing new serializer. 2. Using wrapper methods can be meaningless without serializer. 3. Serializer may just not expose wrapper, then user will have no way to access it. 4. .net has quite a lot of things like http://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.configuredtaskawaitabl %28v=vs.110%29.aspx and nothing explodes even though .net programmers are believed to be really stupid and evil. It's a virtue of Stackoverflow Driven Development. On Sunday, 24 August 2014 at 02:39:40 UTC, Dicebot wrote:
 On Saturday, 23 August 2014 at 09:00:30 UTC, Kagamin wrote:
 What is difficult to find? With flat structure you have all 
 files right before your eyes. If you need std.datetime.systime 
 module, you open std/datetime/systime.d file - that's the 
 reason of needlessly restricting code structure with modules 
 as if one size fits all.
It is the same reasoning as with deep filesystem hierarchies and, well, any data hierarchies - once the element (module / file) count becomes bigger than ~dozen you only really notice things you know to look for. Contrary to that deeply nested categorized hierarchies are easy to casually to search through if you don't know exact module name - just iteratively pick whatever package fits the theme until you find what you want.
I'm afraid, hierarchies make things harder to find. If you don't know what is where, flat organization will present you with everything readily available right before your eyes. With deep hierarchy you're left with abstract or poorly chosen categories at every hierarchy level, so effectively you have to walk the entire hierarchy, which is much more tedious than scroll a flat list of modules viewing ten modules per scroll. Badly named list of modules (like what we have now in phobos) scales well up to 100, well-named list is much more manageable: if you need xml, you already know it's near the end of the list - it's easy to find even among 1000 files - you don't ever need to scroll entire list. If it's not there, where do you go? There's no obvious answer. So even shallow hierarchy is more troublesome than a flat list of 1000 modules. I don't believe hierarchy will magically solve navigation problems just because it has folders.
 I remember coding a bit in C#/.NET platform ages ago - it was 
 totally possible to find relevant modules without even looking 
 in docs, just using auto-complete through suggested package 
 names for import. It was really positive experience for a 
 newbie I was. At the same time a lot of people have no idea how 
 many cool things Phobos actually has.
And System.Xml is not System.Text.Xml, System.Net is not System.IO.Net - where is deep hierarchy? If you want System.Xml, you type `s.x` and it doesn't matter, how many namespaces are there, you're presented with System.Xml. If it's not there, where to find it, in System.Text, System.Formats, System.Parsers, System.Lang, System.Markup, System.Sgml?
Aug 24 2014
parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 24 August 2014 at 14:15:25 UTC, Kagamin wrote:
 On Sunday, 24 August 2014 at 02:22:41 UTC, Dicebot wrote:
 Well difference is that "internal" substring in the fully 
 qualified name that is much more likely to tell user he is 
 better to not touch it. However, original Kagamin proposal of 
 embedding it into module names themselves does address that if 
 you are ok with resulting uglyness.
Both ways it's a convention, and I don't see, why such convention should exist in the first place, member protection has nothing to do with module name, which reflects module's functionality.
Because we have no other means of enforcing this protection - this is why my PR event exists.
 On Sunday, 24 August 2014 at 02:34:01 UTC, Dicebot wrote:
 It can be a philosophical matter, but in my experience 
 grouping by functionality is genuine, and access is an 
 afterthought, so grouping by access doesn't work in the long 
 term, as code naturally (spontaneously) groups by 
 functionality.
That does contradict your statement that any stuff used in parent packages must go to up the hierarchy which is exactly grouping by access :)
Access there means protection, usage reflects functionality.
And how that makes your statement less contradictive?
 I can probably give a more practical example <...>
 code duplication.
1. Making wrapper protected will preclude writing new serializer. 2. Using wrapper methods can be meaningless without serializer. 3. Serializer may just not expose wrapper, then user will have no way to access it. 4. .net has quite a lot of things like http://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.configuredtaskawaitabl %28v=vs.110%29.aspx and nothing explodes even though .net programmers are believed to be really stupid and evil. It's a virtue of Stackoverflow Driven Development.
I am afraid I don't really understand what your point here is - that my design is wrong and must be banned or that package concept is flawed and must be reworked? Neither is really an acceptable argument. There is a simple and strict requirement - no one must be able to extend serializers but few developers that routinely maintain that package (and all sub-packages). Never in user code. This idea is not a subject to reconsideration - if language says such design is illegal I am going to call that language bullshit.
 I'm afraid, hierarchies make things harder to find. If you 
 don't know what is where, flat organization will present you 
 with everything readily available right before your eyes. With 
 deep hierarchy you're left with abstract or poorly chosen 
 categories at every hierarchy level, so effectively you have to 
 walk the entire hierarchy, which is much more tedious than 
 scroll a flat list of modules viewing ten modules per scroll. 
 Badly named list of modules (like what we have now in phobos) 
 scales well up to 100, well-named list is much more manageable: 
 if you need xml, you already know it's near the end of the list 
 - it's easy to find even among 1000 files - you don't ever need 
 to scroll entire list. If it's not there, where do you go? 
 There's no obvious answer. So even shallow hierarchy is more 
 troublesome than a flat list of 1000 modules. I don't believe 
 hierarchy will magically solve navigation problems just because 
 it has folders.
You are speaking about _finding_ things. I am speaking about _discovering_ things. See the difference? There is no way you can quickly tell "aha, so this data formats are supported out of the box" when looking at plain flat list of 1000 files. With solid hierarchy it becomes trivial.
 And System.Xml is not System.Text.Xml, System.Net is not 
 System.IO.Net - where is deep hierarchy? If you want 
 System.Xml, you type `s.x` and it doesn't matter, how many 
 namespaces are there, you're presented with System.Xml. If it's 
 not there, where to find it, in System.Text, System.Formats, 
 System.Parsers, System.Lang, System.Markup, System.Sgml?
You pick few that make most sense to you and ignore the rest. If routinely needed module is not exposed via obvious path it is an issue worth filing in the bug tracker. More rarely used / obscure modules can have more surprising paths. I don't see a problem here. Again, searching for a known package is never a problem both ways - you can just, well, run the search query. Discoverability of uknowns is the key.
Aug 25 2014
prev sibling parent "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:40:34 UTC, Dicebot wrote:
 On Wednesday, 20 August 2014 at 14:36:59 UTC, Kagamin wrote:
 As I see on the realistic example of datetime, which is BIG, 
 we only need to split it into a flat set of files without an 
 overly deep package hierarchy.
We _may_ split it into flat set files (solving only part of the problem) but it is desirable to have a deeper package hierarchy. Package hierarchy is not just an encapsulation tool, it is also a great way to simplify navigation and finding needed modules - something that Phobos is current terrible at exactly because of flat hierarchies.
And with an excessively deep folder structure it will be much more tedious to walk the whole thing. It's deep nesting, which will make things hard to find.
Aug 23 2014
prev sibling next sibling parent reply Rory McGuire via Digitalmars-d-announce writes:
On Wed, Aug 20, 2014 at 4:33 PM, Kagamin via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:

 Do we need a hierarchy of internals, is the problem this big? Why
 mybiglib.wisdom is not good?
It gets really unwieldy when you want to put a whole bunch of things into one project and give control of different packages to different teams. Yes you could use multiple libraries in a lot of cases but if you're wanting it in a single setup / package (like phobos), especially one that is internal to another package you need sub-packages. Examples that would use this are: - kitchen sink libraries like phobos - frameworks such as vibe.d - libraries with lots of internal code such as Ogre3d if it were made in D Well that seems to be good reasons to me.
Aug 20 2014
parent "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:44:21 UTC, Rory McGuire via 
Digitalmars-d-announce wrote:
 On Wed, Aug 20, 2014 at 4:33 PM, Kagamin via 
 Digitalmars-d-announce <
 digitalmars-d-announce puremagic.com> wrote:

 Do we need a hierarchy of internals, is the problem this big? 
 Why
 mybiglib.wisdom is not good?
It gets really unwieldy when you want to put a whole bunch of things into one project and give control of different packages to different teams. Yes you could use multiple libraries in a lot of cases but if you're wanting it in a single setup / package (like phobos), especially one that is internal to another package you need sub-packages.
Yes, public interfaces of packages can form a hierarchy (which still shouldn't be more complex than necessary), but internals?
 Examples that would use this are:
 - kitchen sink libraries like phobos
I proposed an example of datetime module, that I don't see a problem with internals there. Can you point to an exact line in source, which suffers from this?
Aug 23 2014
prev sibling parent reply ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Wed, 20 Aug 2014 14:33:52 +0000
Kagamin via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

 Do we need a hierarchy of internals, is the problem this big? Why=20
 mybiglib.wisdom is not good?
ah, why we need such things as subdirectories at all? CP/M was fine without concept of subdirectories!
Aug 20 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/20/14, 7:49 AM, ketmar via Digitalmars-d-announce wrote:
 On Wed, 20 Aug 2014 14:33:52 +0000
 Kagamin via Digitalmars-d-announce
 <digitalmars-d-announce puremagic.com> wrote:

 Do we need a hierarchy of internals, is the problem this big? Why
 mybiglib.wisdom is not good?
ah, why we need such things as subdirectories at all? CP/M was fine without concept of subdirectories!
No need to demean the question. It is valid. -- Andrei
Aug 20 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 17:19:58 UTC, Andrei Alexandrescu 
wrote:
 On 8/20/14, 7:49 AM, ketmar via Digitalmars-d-announce wrote:
 On Wed, 20 Aug 2014 14:33:52 +0000
 Kagamin via Digitalmars-d-announce
 <digitalmars-d-announce puremagic.com> wrote:

 Do we need a hierarchy of internals, is the problem this big? 
 Why
 mybiglib.wisdom is not good?
ah, why we need such things as subdirectories at all? CP/M was fine without concept of subdirectories!
No need to demean the question. It is valid. -- Andrei
Originally flat Phobos hierarchy was considered "good enough". Now we can see that such approach doesn't scale well - features are often missed because of non-intuitive module placement, compile times suffer because of many cross-module dependencies (flat hierarchy encourages big modules). Is there any reason to think that same logic applied to sub-packages will scale any better as Phobos size grows? I doubt so. In fact I feel it is already beyond the size where it is convenient and only reason why even smaller deeply nested modules are not an option is exactly because it is too hard to keep both existing protection attribute relation and provide easy to navigate hierarchy at the same time. The fact that some of people who have actually tried to use package.d support this language change is pretty good anecdotal evidence that there is a problem to be solved.
Aug 20 2014
prev sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Wed, 20 Aug 2014 10:19:59 -0700
Andrei Alexandrescu via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

 No need to demean the question. It is valid. -- Andrei
sorry, i don't mean to insult anyone, just trying to make people see analogies. hierarchies are everywhere, it's convient way to store alot of things. modules itself is a great example of hierarchies -- we can dump all source code in one file instead of using modules. i.e. file system hierarchy is handy what there are many files. module hierarchy is handy when there are many modules in library. the ability to have fine-grained visibility control in module hierarchy is handy. this was written numerous times in this thread, yet he continues to ask the same question again and again as if he just don't bother to read any answers. that's why i asked about subdirs.
Aug 20 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 09:26:37 UTC, Kagamin wrote:
 On Wednesday, 20 August 2014 at 07:47:36 UTC, ketmar via 
 Digitalmars-d-announce wrote:
 but it requires to dump all 'subpackages' into one flat 
 directory and
 with ugly names.
Huh? If std.datetime and std.regex are subpackages of `std` package, it usually means, they should reside in `std` directory, no? And how explicit package protection can change that?
Your proposal makes such module layout illegal: std.package1.module1 std.package1.module2 std.package1.subpackage.module1 std.package1.subpackage.module2 (can't have symbols in subpackage that are also available to package1 but not whole std / public, must have `subpackage_module1` instead, `package_1_subpackage_module1` if it must be available for whole std) It severely limits expressive power of package system for code organization and encourages flat hierarchies with minimal nesting. It also looks ugly - anyone seeing module name with package name mangled in it will feel that there is something wrong with the language if it needs to resort to such hacks.
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 13:05:54 UTC, Dicebot wrote:
 Your proposal makes such module layout illegal:

 std.package1.module1
 std.package1.module2
 std.package1.subpackage.module1
 std.package1.subpackage.module2

 (can't have symbols in subpackage that are also available to 
 package1 but not whole std / public, must have 
 `subpackage_module1` instead, `package_1_subpackage_module1` if 
 it must be available for whole std)
If some utility is internal to std.package1 an used in the entire std.package1, shouldn't it sit in std.package1? Why push it to subpackage? std.package1.module1 std.package1.module2 std.package1.internal <- package1's internals go here std.package1.subpackage.module1 std.package1.subpackage.module2
Aug 20 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 14:25:59 UTC, Kagamin wrote:
 If some utility is internal to std.package1 an used in the 
 entire std.package1, shouldn't it sit in std.package1? Why push 
 it to subpackage?

 std.package1.module1
 std.package1.module2
 std.package1.internal <- package1's internals go here
 std.package1.subpackage.module1
 std.package1.subpackage.module2
It may semantically belong to subpackage but still needs to be available to package1, something not uncommon in templated code (subpackage is generic implementation, package1 is specialization that still needs access to non-public functions). Also you seem to imply that "internal" is always small tightly coupled module which is hardly true for any higher level functionality. So in practice it will be more like this: std.package1.module1 std.package1.module2 std.package1.internal1 std.package1.internal2 ... std.package1.internal20 Not really encouraging.. What you propose is effectively prohibiting to use packages to organize your code and requiring to design your module hierarchy based on desired protection relations, not other way around. I think it is conceptually wrong approach and unnecessarily restrictive compared to overall D design principles (no single "true" way of doing things)
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:35:31 UTC, Dicebot wrote:
 It may semantically belong to subpackage but still needs to be 
 available to package1, something not uncommon in templated code 
 (subpackage is generic implementation, package1 is 
 specialization that still needs access to non-public functions).
If generic implementation is designed to be customized, it probably means, it should be widely available for customization, like xml document and base64 encoder. Also it's probably a logical error if base type is less visible than the derived type.
 What you propose is effectively prohibiting to use packages to 
 organize your code and requiring to design your module 
 hierarchy based on desired protection relations, not other way 
 around. I think it is conceptually wrong approach and 
 unnecessarily restrictive compared to overall D design 
 principles (no single "true" way of doing things)
It's ok for packages to exchange public interface, but internals? If a subpackage has internals, they are for its usage, and consumable functionality should be provided through public interface.
Aug 20 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 15:13:06 UTC, Kagamin wrote:
 On Wednesday, 20 August 2014 at 14:35:31 UTC, Dicebot wrote:
 It may semantically belong to subpackage but still needs to be 
 available to package1, something not uncommon in templated 
 code (subpackage is generic implementation, package1 is 
 specialization that still needs access to non-public 
 functions).
If generic implementation is designed to be customized, it probably means, it should be widely available for customization, like xml document and base64 encoder. Also it's probably a logical error if base type is less visible than the derived type.
This is just your design preference and hardly a good "one size fits them all" decision. I also don't speak about inheritance but about composition - customization may be implemented via unsafe field exposure and unsuitable for general public usage but useful for simplifying internal implementation maintenance. We should not force users into single good package structure based on certain design beliefs.
 What you propose is effectively prohibiting to use packages to 
 organize your code and requiring to design your module 
 hierarchy based on desired protection relations, not other way 
 around. I think it is conceptually wrong approach and 
 unnecessarily restrictive compared to overall D design 
 principles (no single "true" way of doing things)
It's ok for packages to exchange public interface, but internals? If a subpackage has internals, they are for its usage, and consumable functionality should be provided through public interface.
Same here. As library size grows you completely lose the distinction between "public" and "internal", certain parts of the library may become hidden from other parts and public but available for others. It is pretty much a necessity to keep up with maintenance when there is a large team working on it simultaneously. Right now people mostly rely on convention, I have seem quite some comments like "should have been private but needs to be accessed from module X, don't ever touch it" (public-but-undocumented functions of Phobos sometimes fall into the same category). Telling people that they design applications in a wrong way and need to change their habits is a good approach to alienate them against the language.
Aug 20 2014
prev sibling parent reply "Chris Nicholson-Sauls" <ibisbasenji gmail.com> writes:
module foo.bar.one;
module foo.bar.internals; // package-protected utilities
module foo.bar.subpkg.two;
module foo.bar.subpkg.internals; // package-protected utilities


Current situation: module 'one' cannot access the 'bar.subpkg' 
utilities, and module 'two' cannot access the 'bar' utilities.  
This is needlessly limiting, forcing design choices that should 
not be dictated by the ability/inability to separate public and 
private API.  It also precludes many valid and good uses of 
nested package.d modules.

I really don't see any equally strong counter-argument.  But 
then, I've wanted this exact fix for literally years now.

It also is not limited to internal utility modules.  It can be 
useful for systems that select at compile time from one of a 
number of system-specific implementations of a given 
interface/api.  It can be useful for granting privileged access 
to certain api's and/or resources to specific module(s).  An 
example being: grant access to unsafe but versitile data 
manipulators solely to the subpackage containing the loaders.

Currently, there is no absolutely enforceable way of doing such 
things; so one ends up falling back on convention... and 
conventions, sadly, never hold in the real world for long.
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 20:39:06 UTC, Chris 
Nicholson-Sauls wrote:
 module foo.bar.one;
 module foo.bar.internals; // package-protected utilities
 module foo.bar.subpkg.two;
 module foo.bar.subpkg.internals; // package-protected utilities


 and module 'two' cannot access the 'bar' utilities.
Really? Looks like a bug. You'd better check if it's intended to work that way.
 This is needlessly limiting, forcing design choices that should 
 not be dictated by the ability/inability to separate public and 
 private API.  It also precludes many valid and good uses of 
 nested package.d modules.
If some thing is used in entire foo.bar package, then it's foo.bar's internal, not foo.bar.subpkg's internal. I think, it's natural when the wider the thing is used, the higher in hierarchy it sits.
 I really don't see any equally strong counter-argument.  But 
 then, I've wanted this exact fix for literally years now.
I don't really see, what it blocks.
 It also is not limited to internal utility modules.  It can be 
 useful for systems that select at compile time from one of a 
 number of system-specific implementations of a given 
 interface/api.  It can be useful for granting privileged access 
 to certain api's and/or resources to specific module(s).  An 
 example being: grant access to unsafe but versitile data 
 manipulators solely to the subpackage containing the loaders.
If those manipulators are used solely in loaders subpackage, they should belong to that package. Though I don't really see a problem here: there was an implementation of xml document, which allowed unsafe (but fast) specialization, which was believed to be useful for public usage. It fits well into D philosophy that safe things should be available by default and unsafe/fast things should be possible.
Aug 23 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-23 11:12, Kagamin wrote:

 If some thing is used in entire foo.bar package, then it's foo.bar's
 internal, not foo.bar.subpkg's internal. I think, it's natural when the
 wider the thing is used, the higher in hierarchy it sits.
A symbol declared "package" can only be accessed within its own package. It cannot be accessed from sub packages or super packages. -- /Jacob Carlborg
Aug 23 2014
parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 23 August 2014 at 09:24:04 UTC, Jacob Carlborg wrote:
 A symbol declared "package" can only be accessed within its own 
 package. It cannot be accessed from sub packages or super 
 packages.
I didn't know about that. Should it really work that way?
Aug 23 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-23 11:35, Kagamin wrote:

 I didn't know about that. Should it really work that way?
That's what we're trying to fix with this change. -- /Jacob Carlborg
Aug 23 2014
parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 23 August 2014 at 11:04:35 UTC, Jacob Carlborg wrote:
 That's what we're trying to fix with this change.
The proposal is to make internals visible to super packages, I only wonder whether internals visible to some package should be hidden from its subpackages.
Aug 23 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 23 August 2014 at 11:12:34 UTC, Kagamin wrote:
 On Saturday, 23 August 2014 at 11:04:35 UTC, Jacob Carlborg 
 wrote:
 That's what we're trying to fix with this change.
The proposal is to make internals visible to super packages, I only wonder whether internals visible to some package should be hidden from its subpackages.
It addresses both - explicitly supplied package identifiers imply access to all subpackages. Current implementation is very striaghtforward and naive in that regard.
Aug 23 2014
prev sibling next sibling parent reply "Dmitry Olshansky" <dmitry.olsh gmail.com> writes:
On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:
 On 8/19/2014 7:01 AM, Dicebot wrote:
 Walter, now that release is out can you please state your
opinion about
 https://github.com/D-Programming-Language/dmd/pull/3651 ? It
is blocking Phobos
 module split and decoupling.
I keep thinking there's gotta be a way to do this without language changes.
For what it's worth I don't care how things turn out w.r.t. syntax, language extension or convention but the problem must be solved. The way I see the problem: There must be nice and clean way to have internal stuff that is invisible for user. Preferably this stuff must not be monolithic and have form of package. Example structure: mod/package.d mod/internal/bar.d mod/... I actually face it in Phobos right now, my blocked pull: https://github.com/D-Programming-Language/phobos/pull/2412 Options: 1. Use current package attribute. Disadvantages: a) Must retain internals in the same package, user may still import them no problem at all. At least it does not do much harm as nothing is accessible. b) Can't use std.mod.internal.bar 'package' stuff in std.mod (upper one). That makes it next to unusable. Basically it means we have to push internals up, instead of pushing them down. 2. Use separate public internal package. Main disadvantage: a) Now user may import and actually USE internal stuff. b) Not only that but also internal methods of equally internal structs and classes leak out. In other words: a struct defined in internal module may never be exposed without some kind of wrapping. Ugly and unnecessary complication. I'd rather have a language feature, but whatever you guys invent as 3rd option that doesn't suck will have my vote. --- Dmitry Olshansky
Aug 22 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
All I want is that whatever decision Walter makes to happen 
sooner than in few years from now.
Aug 22 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Dicebot:

 All I want is that whatever decision Walter makes to happen 
 sooner than in few years from now.
There are other pending patches, like the support for the nice [$] syntax by Kenji. "I keep thinking there's gotta be a way to do this without language changes." or "I keep thinking there must be a better way than [$]." are the kind of standard answers Walter gives when he doesn't have time to think about some design (because he has time he usually needs only hours or very few days to implement features, like when he made UDAs or C++ namespeces). So I guess Walter is currently busy with something (like finishing the 2.066 release and its regressions, or developing Warp for Facebook, or something else), so you have to wait some time for him to come back to the language development side, to receive a good answer and a good solution. Bye, bearophile
Aug 22 2014
next sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Fri, 22 Aug 2014 20:48:22 +0000
bearophile via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

or (smart plan!) just apply the necessary PRs, build dmd from sources
and start using the features. if authors of popular libraries will do
this, there will be no choice and maintainers will be forced to apply
this patches. this is called 'driven by the community'. ;-)

ok-ok-ok, you got me, this was a silly joke! or maybe not?..
Aug 22 2014
prev sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Sat, 23 Aug 2014 04:50:36 +0300
ketmar via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

to be honest, i myself is a happy user of Kenji's $ patch and
package(...) patch. yes, it costs some efforts to keep 'em up-to-date,
but hey, i have no life anyway.
Aug 22 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Friday, 22 August 2014 at 15:06:14 UTC, Dmitry Olshansky wrote:
 Example structure:
 mod/package.d
 mod/internal/bar.d
 mod/...

 I actually face it in Phobos right now, my blocked pull:
 https://github.com/D-Programming-Language/phobos/pull/2412
I think, it should have structure std.regex.package; std.regex.backtracking; std.regex.ir; What is the reason to have regex stuff in a remote directory? That would make things hard to find and navigate.
Aug 23 2014
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
23-Aug-2014 12:57, Kagamin пишет:
 On Friday, 22 August 2014 at 15:06:14 UTC, Dmitry Olshansky wrote:
 Example structure:
 mod/package.d
 mod/internal/bar.d
 mod/...

 I actually face it in Phobos right now, my blocked pull:
 https://github.com/D-Programming-Language/phobos/pull/2412
I think, it should have structure std.regex.package; std.regex.backtracking; std.regex.ir; What is the reason to have regex stuff in a remote directory? That would make things hard to find and navigate.
Yes, that was my initial pull. The stuff inside was marked as `package`, as only package.d currently has the same public API. Then Dicebot suggested that since all modules are for the moment internal they have to be moved somewhere until public API is fleshed out (as it may change module name etc.). 2 options that both do NOT currently work: std.regex.internal std.internal.regex -- Dmitry Olshansky
Aug 23 2014
parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 23 August 2014 at 09:14:13 UTC, Dmitry Olshansky 
wrote:
 Yes, that was my initial pull. The stuff inside was marked as 
 `package`, as only package.d currently has the same public API.

 Then Dicebot suggested that since all modules are for the 
 moment internal they have to be moved somewhere until public 
 API is fleshed out (as it may change module name etc.).

 2 options that both do NOT currently work:
 std.regex.internal
 std.internal.regex
If the user relies that a symbol is found in std.internal.regex.backtracking, moving it will break the code too. So what's the difference?
Aug 23 2014
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
23-Aug-2014 13:33, Kagamin пишет:
 On Saturday, 23 August 2014 at 09:14:13 UTC, Dmitry Olshansky wrote:
 Yes, that was my initial pull. The stuff inside was marked as
 `package`, as only package.d currently has the same public API.

 Then Dicebot suggested that since all modules are for the moment
 internal they have to be moved somewhere until public API is fleshed
 out (as it may change module name etc.).

 2 options that both do NOT currently work:
 std.regex.internal
 std.internal.regex
If the user relies that a symbol is found in std.internal.regex.backtracking, moving it will break the code too. So what's the difference?
I don't see the difference. In any case user should see public aliases and never know where they lead to. Dicebot - any objections? However since my pull hang there for months, I decide to appease puristic requests and move things to internals, only to find out that it doesn't work. For what its worth I'd _prefer_ std.regex.internal as in my opinion internals got to be well contained. But as discussed, `package` doesn't work this way. -- Dmitry Olshansky
Aug 23 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 23 August 2014 at 10:04:40 UTC, Dmitry Olshansky 
wrote:
 For what its worth I'd _prefer_ std.regex.internal as in my 
 opinion internals got to be well contained. But as discussed, 
 `package` doesn't work this way.
It can be a philosophical matter, but in my experience grouping by functionality is genuine, and access is an afterthought, so grouping by access doesn't work in the long term, as code naturally (spontaneously) groups by functionality. The reason is when you maintain code, you traverse it according to functionality, because you work with implementation, and protection is interface, so protection is ignored. If you add a public functionality to internal module (ranges often end up like that), you can either just qualify it with public access, as it's done in normal modules, or refactor and move it to public module, so you end up with logically grouped functionality separated between two or more modules, which try to enforce grouping by access, while in normal modules you have logically grouped functionality in the same module even if they have different access. Also you can end up with public functionality in internal module, if the member proved to be stable, robust and use case for it arises.
Aug 23 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 23 August 2014 at 11:08:31 UTC, Kagamin wrote:
 On Saturday, 23 August 2014 at 10:04:40 UTC, Dmitry Olshansky 
 wrote:
 For what its worth I'd _prefer_ std.regex.internal as in my 
 opinion internals got to be well contained. But as discussed, 
 `package` doesn't work this way.
It can be a philosophical matter, but in my experience grouping by functionality is genuine, and access is an afterthought, so grouping by access doesn't work in the long term, as code naturally (spontaneously) groups by functionality.
That does contradict your statement that any stuff used in parent packages must go to up the hierarchy which is exactly grouping by access :) I can probably give a more practical example I have just recently encountered when re-designing one of our internal library packages. It was a "serialization" package and sub-packages defined different (de)serialization models, some of those defining special wrapper types for resulting data that enforce certain semantics specific to that serialization model via the type system. It is not surprising that most internal details of such wrappers were kept package protected as exposing those to user could have easily violated all assumptions (but was needed to implement (de)serialization efficiently). It has become more complicated when "meta-serializers" have been added to parent package - templated stuff that took any of sub-package implementations and added some functionality (like versioning support) on top. Being generic thing it reside in higher level "serialization" package but to be implemented efficiently it needs access to those internal wrapper fields. Moving wrapper modules to parent package is not an option here because those are closely related to specific serialization model. Exposing stuff as public is not an option because anyone not deeply familiar with package implementation can easily break all type system guarantees that way. Moving "meta-serializers" to sub-packages is quite a code duplication. Right now I keep that stuff public and say in docs "please don't use this" which is hardly a good solution.
Aug 23 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 23 August 2014 at 10:04:40 UTC, Dmitry Olshansky 
wrote:
 If the user relies that a symbol is found in
 std.internal.regex.backtracking, moving it will break the code 
 too. So
 what's the difference?
I don't see the difference. In any case user should see public aliases and never know where they lead to. Dicebot - any objections?
Well difference is that "internal" substring in the fully qualified name that is much more likely to tell user he is better to not touch it. However, original Kagamin proposal of embedding it into module names themselves does address that if you are ok with resulting uglyness.
Aug 23 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:
 On 8/19/2014 7:01 AM, Dicebot wrote:
 Walter, now that release is out can you please state your
opinion about
 https://github.com/D-Programming-Language/dmd/pull/3651 ? It
is blocking Phobos
 module split and decoupling.
I keep thinking there's gotta be a way to do this without language changes.
In the meanwhile I have updated https://github.com/D-Programming-Language/dmd/pull/3651 to address latest Walter comments and be based on recent master
Aug 25 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
..and it has just been merged! ^_^

Thanks Walter!
Aug 26 2014
next sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Tue, 26 Aug 2014 16:46:19 +0000
Dicebot via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

 ..and it has just been merged! ^_^
WOW! that's great!
 Thanks Walter!
second that!
Aug 26 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/26/14, 9:46 AM, Dicebot wrote:
 ...and it has just been merged! ^_^

 Thanks Walter!
Congratulations for a job well done. -- Andrei
Aug 26 2014
parent "Andrew Godfrey" <X y.com> writes:
On 8/26/14, 9:46 AM, Dicebot wrote:
 ...and it has just been merged! ^_^

 Thanks Walter!
Nice! This felt like a glaring hole, and you've filled it!
Aug 28 2014