www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Feature Idea: hidden modules

reply Mike Parker <aldacron71 yahoo.com> writes:
I recently came across a design situation where the concept of 'hidden' 
modules would prove useful, particularly since we can't have package and 
  modules with the same at the same level. Consider this package structure:

mypackage.platform;
mypackage.platformimpl.win32;
mypackage.platformimpl.linux;
mypackage.platformimpl.sdl;
mypackage.platformimpl.glfw;

Platform is setup to import the proper implementation based upon a 
version conditional. The problem is that all of the modules in the 
platofrmimpl package are visible to the client and can be imported 
freely. One might consider this as a solution:

mypackage.platform.platform;
mypackage.platform.win32;
mypackage.platform.linux;
...

Then the implementation modules could each be wrapped with package 
protection so that they are only visible to the platform.platform 
module. This works well if you are dealing with predefined interfaces. 
But in this case, there's no sense in using an interface - an app will 
only be compiled to use one platform at a time. From a design 
standpoint, it makes more sense in this case that each module just 
implement the necessary interface without an actual interface definition 
- perhaps as a struct with static methods. Then, the platform module 
just does this:

version(Win32App)
    import mypackage.platformimpl.win32;
else version(LinuxApp)
    import mypackage.platformimpl.linux;

Package protection doesn't work in this case because the contents of 
each platform file need to be exposed to the client, sense there is no 
abstract interface defined. Without some mechanism to allow you to hide 
a particular module from the client, there is no way around the need to 
use an interface + package protection if you wish to hide the imlementation.

So why not a new level of module protection that hides a given module 
from the client - meaning, it can't be imported at all outside of a 
particular part of the package tree. Perhaps something along these lines:

// module declaration using new 'hidden' keyword
hidden module mypackage.platformimpl.win32;

This would make the win32 module visible only to modules that meet the 
following criteria:

* any module that is in a subpackage of platformimpl can import it. ex: 
mypackage.platformimpl.anotherpackage.somemodule

* any module that is in the same package as the hidden module can import 
it. ex: mypackage.platformimpl.somemodule

* any module that is exactly one level above the hidden module's package 
can import it. ex: mypackage.somemodule

Regarding the third point, it might be useful to allow any module above 
the hidden module up to the root package to import it. For example:

foo.bar.baz.mymodule (hidden)
can be imported by foo.bar.module and foo.module

However, I think it better if it would be restricted only to one level 
above the hidden module's package. This allows you to provide an 
'adapter module' which imports the hidden modules, and any module that 
needs to use the hidden module's functionality must go through the 
adapter. This allows better encapsulation and avoids issues that might 
arise from importing the hidden modules directly in the root package.

Just something I thought might prove useful in providing a little more 
design flexibility over what we have already (and adds more weight to 
the need for a module statement, which some people have questioned 
recently).
Jul 10 2005
next sibling parent reply Mike Parker <aldacron71 yahoo.com> writes:
That didn't come out as clearly as it did in my head. If you're 
confused, I'll try to clarify it :)
Jul 10 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Sun, 10 Jul 2005 16:38:32 +0900, Mike Parker <aldacron71 yahoo.com>  
wrote:
 That didn't come out as clearly as it did in my head. If you're  
 confused, I'll try to clarify it :)

I may be confused. What is wrong with: [mypackage.platform.d] module mypackage.platform version(Win32) import mypackage.platformimpl.win32; version(linux) import mypackage.platformimpl.linux; version(sdl) import mypackage.platformimpl.sdl; version(glfw) mypackage.platformimpl.glfw; [mypackage.platformimpl.win32.d] module mypackage.platformimpl.win32 version(Win32): ..etc.. and so on for each module. This prevents an app compiled on "linux" from importing "mypackage.platformimpl.win32". Or is the real problem with hiding parts of the implementation? Does "mypackage.platform" need access to parts of "mypackage.platformimpl.win32"? are you trying to prevent users from having access to those same parts? Perhaps a small example will help, then I can experiment too. Regan
Jul 10 2005
parent reply Mike Parker <aldacron71 yahoo.com> writes:
Regan Heath wrote:
 On Sun, 10 Jul 2005 16:38:32 +0900, Mike Parker <aldacron71 yahoo.com>  
 wrote:
 
 That didn't come out as clearly as it did in my head. If you're  
 confused, I'll try to clarify it :)

I may be confused. What is wrong with: [mypackage.platform.d] module mypackage.platform version(Win32) import mypackage.platformimpl.win32; version(linux) import mypackage.platformimpl.linux; version(sdl) import mypackage.platformimpl.sdl; version(glfw) mypackage.platformimpl.glfw; [mypackage.platformimpl.win32.d] module mypackage.platformimpl.win32 version(Win32): ..etc.. and so on for each module. This prevents an app compiled on "linux" from importing "mypackage.platformimpl.win32".

No it doesn't. An app compiled on linux could still directly import mypackage.platformimpl.win32 directly, bypassing mypackage.platform altogether.
 
 Or is the real problem with hiding parts of the implementation? 

That's it exactly. The platform stuff was probably not the best example, but it's what I was working on when the idea hit me. To break it down simply: mypackage.foo.bar.module1 mypackage.foo.bar.module2 mypackage.foo.bar.module3 mypackage.foo.moduleselector The goal is to prevent any module outside of mypackage.foo (including anything in mypackage) from importing module1, module2, and module3 directly, forcing them to access that functionality through moduleselector. Obviously, you could wrap module1, module2, and module3 all in package protection statements and do this: mypackage.foo.bar.moduleselector With moduleselector in the same package, it is allowed access to declarations in all 3 modules. Outside modules will be forced to go through moduleselector since package protection hides everything. Really this solution is fine. But there's a minor drawback. All modules that moduleselector manages must be in the same package. From a design perspective, this prevents logical grouping of modules when multiple modules are involved. Consider this: mypackage.foo.bar.module1a mypackage.foo.bar.module1b mypackage.foo.bar.module1c mypackage.foo.bar.module1 If modules 2 & 3 have multiple modules that go together, now you have a big jumbled mess. With the hidden module concept as I proposed it, you could do this: mypackage.foo.implone.module1a mypackage.foo.implone.module1b mypackage.foo.implone.module1c mypackage.foo.implone.module1 mypackage.foo.impltwo.module2a ... mypackage.foo.moduleselector So the benefits from such a mechanism are design flexibility, and the ability to prevent some modules from being imported outside of a particular package tree. This can be very useful when designing library. It's not something that we can't live without, but I do think it's something that could prove useful.
Jul 10 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Mon, 11 Jul 2005 10:40:10 +0900, Mike Parker <aldacron71 yahoo.com>  
wrote:
 Regan Heath wrote:
 On Sun, 10 Jul 2005 16:38:32 +0900, Mike Parker <aldacron71 yahoo.com>   
 wrote:

 That didn't come out as clearly as it did in my head. If you're   
 confused, I'll try to clarify it :)

[mypackage.platform.d] module mypackage.platform version(Win32) import mypackage.platformimpl.win32; version(linux) import mypackage.platformimpl.linux; version(sdl) import mypackage.platformimpl.sdl; version(glfw) mypackage.platformimpl.glfw; [mypackage.platformimpl.win32.d] module mypackage.platformimpl.win32 version(Win32): ..etc.. and so on for each module. This prevents an app compiled on "linux" from importing "mypackage.platformimpl.win32".

No it doesn't. An app compiled on linux could still directly import mypackage.platformimpl.win32 directly, bypassing mypackage.platform altogether.

True, I wasn't meant to say that exactly. What I meant was, they can do it but it causes no problems. No compile errors. No access to things that are supposed to be hidden/protected (assuming "package" is used).
  Or is the real problem with hiding parts of the implementation?

That's it exactly. The platform stuff was probably not the best example, but it's what I was working on when the idea hit me. To break it down simply: mypackage.foo.bar.module1 mypackage.foo.bar.module2 mypackage.foo.bar.module3 mypackage.foo.moduleselector The goal is to prevent any module outside of mypackage.foo (including anything in mypackage) from importing module1, module2, and module3 directly, forcing them to access that functionality through moduleselector. Obviously, you could wrap module1, module2, and module3 all in package protection statements and do this: mypackage.foo.bar.moduleselector With moduleselector in the same package, it is allowed access to declarations in all 3 modules. Outside modules will be forced to go through moduleselector since package protection hides everything. Really this solution is fine. But there's a minor drawback.

This is the solution I was suggesting/hinting at.
 All modules that moduleselector manages must be in the same package.  
  From a design perspective, this prevents logical grouping of modules  
 when multiple modules are involved. Consider this:

 mypackage.foo.bar.module1a
 mypackage.foo.bar.module1b
 mypackage.foo.bar.module1c
 mypackage.foo.bar.module1

 If modules 2 & 3 have multiple modules that go together, now you have a  
 big jumbled mess.

I must be dense today.. why is it a jumbled mess? [mypackage.foo.bar.module1.d] module mypackage.foo.bar.module1 version(1A) import mypackage.foo.bar.module1a [mypackage.foo.bar.module1a.d] version(1A): package: ..etc.. [mypackage.foo.bar.module2.d] module mypackage.foo.bar.module1 version(2A) import mypackage.foo.bar.module1a [mypackage.foo.bar.module2a.d] version(2A): package: ..etc.. [program.d] import mypackage.foo.bar.module1; import mypackage.foo.bar.module2; Seems fine to me.
 With the hidden module concept as I proposed it, you could do this:

 mypackage.foo.implone.module1a
 mypackage.foo.implone.module1b
 mypackage.foo.implone.module1c
 mypackage.foo.implone.module1
 mypackage.foo.impltwo.module2a
 ...
 mypackage.foo.moduleselector

 So the benefits from such a mechanism are design flexibility, and the  
 ability to prevent some modules from being imported outside of a  
 particular package tree. This can be very useful when designing library.

 It's not something that we can't live without, but I do think it's  
 something that could prove useful.

I'm not sure we need to "prevent" people importing things, we just have to have a method of protecting code while also making it possible to divide a module into several logical units (files). "package" is the method which was added to facilitate this. Perhaps I'm just not getting it? In which case can you give a more concrete example of a specific problem? Regan
Jul 10 2005
parent reply Mike Parker <aldacron71 yahoo.com> writes:
Regan Heath wrote:

 mypackage.foo.bar.module1a
 mypackage.foo.bar.module1b
 mypackage.foo.bar.module1c
 mypackage.foo.bar.module1

 If modules 2 & 3 have multiple modules that go together, now you have 
 a  big jumbled mess.

I must be dense today.. why is it a jumbled mess?

Because if you have multiple 'groups' of modules (1a/b/c/d, 2/a/b/c/d, 3a/b/c/d) in the same package, there's no logical organization - all of them are jumbled together in the same package (necessary when using package protection). What is much cleaner is this: mypackage.implone.module1a/b/c/d mypackage.impltwo.module2a/b/c/d mypackage.implthree.module3a/b/c/d mypackage.moduleselector By allowing the modules in the implone/two/three packages to be hidden to oustsiders (except those below and one level above), you can now logically structure your packages and keeo a clean separation of module groups, rather than having them all bunched up in one package.
 Perhaps I'm just not getting it? In which case can you give a more  
 concrete example of a specific problem?

It's not a problem, per se. Nothing's really broken here. It's about adding more flexibility to the design through package structure. The package keyword is a big improvement over when we didn't have it, but it is still limited in that it forces everything to be in the same package or lower, and other modules can still access anything not protected by the package keyword when they import a particular module.
Jul 10 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Mon, 11 Jul 2005 14:41:35 +0900, Mike Parker <aldacron71 yahoo.com>  
wrote:
 Regan Heath wrote:

 mypackage.foo.bar.module1a
 mypackage.foo.bar.module1b
 mypackage.foo.bar.module1c
 mypackage.foo.bar.module1

 If modules 2 & 3 have multiple modules that go together, now you have  
 a  big jumbled mess.


Because if you have multiple 'groups' of modules (1a/b/c/d, 2/a/b/c/d, 3a/b/c/d) in the same package, there's no logical organization - all of them are jumbled together in the same package (necessary when using package protection).

So, you're saying you cannot do this: mypackage/implone/a mypackage/implone/b mypackage/implone/c mypackage/impltwo/a mypackage/impltwo/b mypackage/impltwo/c .. because implone and impltwo require access to each others internal data? This seems odd to me, no offense but it suggests either bad design, or that they should be in the same package or even the same file if they're that tightly bound. Without a concrete example I can only speculate.
 What is much cleaner is this:

 mypackage.implone.module1a/b/c/d
 mypackage.impltwo.module2a/b/c/d
 mypackage.implthree.module3a/b/c/d
 mypackage.moduleselector

 By allowing the modules in the implone/two/three packages to be hidden  
 to oustsiders (except those below and one level above), you can now  
 logically structure your packages and keeo a clean separation of module  
 groups, rather than having them all bunched up in one package.

Making them 'hidden' would probably achieve your goal, but, there are likely several solutions and I think I'd prefer one that was more like "package". I mean there is no point prohibiting an import if you can instead ensure it is meaningless (i.e. they cannot gain access to things they should't) thats my opinion anyway.
 Perhaps I'm just not getting it? In which case can you give a more   
 concrete example of a specific problem?

It's not a problem, per se. Nothing's really broken here.

I realise that.
 It's about adding more flexibility to the design through package  
 structure. The package keyword is a big improvement over when we didn't  
 have it, but it is still limited in that it forces everything to be in  
 the same package or lower,

I haven't had a problem with this restriction as yet, I think a concrete example i.e. show me exactly what you tried to do when you found this.
 and other modules can still access anything not protected by the package  
 keyword when they import a particular module.

That is because without "package" or another protection attribute it defaults to "public". We should probably get into the habit of putting: package: at the top of all our source (which suggests to me that this should be the default, not public) Regan
Jul 11 2005
parent reply Mike Parker <aldacron71 yahoo.com> writes:
Regan Heath wrote:

 Regan Heath wrote:

 mypackage.foo.bar.module1a
 mypackage.foo.bar.module1b
 mypackage.foo.bar.module1c
 mypackage.foo.bar.module1

 If modules 2 & 3 have multiple modules that go together, now you 
 have  a  big jumbled mess.

I must be dense today.. why is it a jumbled mess?

Because if you have multiple 'groups' of modules (1a/b/c/d, 2/a/b/c/d, 3a/b/c/d) in the same package, there's no logical organization - all of them are jumbled together in the same package (necessary when using package protection).

So, you're saying you cannot do this: mypackage/implone/a mypackage/implone/b mypackage/implone/c mypackage/impltwo/a mypackage/impltwo/b mypackage/impltwo/c .. because implone and impltwo require access to each others internal data?

No, that's not what I'm saying at all. I'm not sure if I'm not explaining myself clearly or not, but you are completely missing my point. Imagine you have a single module that provides an interface to implone and impltwo in mypackage.myinterfacemodule. You only want clients of your package to access implone and impltwo through the interface module, and not allow them to import them directly, thereby hiding the implementation from the client. Right now, there is no mechanism in D that allows this. The workaround is to put the interface module and all of the implementatoins in the same directory, but then you lose logical separation of the implementations. Imagine a team of programmers - Bob, Joe and Bill - who are working on a D library. One of the subsytems in the library is the foo subsystem, of which there could be several possible implementations. So, they develop a foo interface and each programmer develops his own custom implementation. This is the resulting package structure: teamlib.foomodule teamlib.bobfoo.modulesa/b/c/d/e teamlib.joefoo.modulea/b/c teamlib.billfoo.modulea/b/c/d Perhaps the implemetation can be chosen at compile time, perhaps at runtime, it doesn' matter. Currently in D, any part of the implementation that is not package protected can be imported directly and accessed by the library client. But, for whatever reason, the team doesn't want the client to access anything directly - they want would prefer if the client goes through foomodule instead. Because there is currently no way in D to achieve this goal with the given package structure, they'll need to wrap their implementation modules in package protection and do this: teamlib.foo.foomodule teamlib.foo.bobfooa/b/c/d/e teamlib.foo.joefooa/b/c teamlib.foo.billfooa/b/c/d So now instead of a logical separation of implementations, you have all 12 modules in the same package. Additionally, the implementaion modules have access to each other (which may not be desirable). The first method is structurally cleaner, easier to maintain (when you are talking about numerous files), and hides implementations from each other. That's what I'm trying to get across, and I hope I have succeeded this time. Having the ability to prevent modules from being imported outside of a particular scope could come in handy for library designs, I think. I know I can see uses for such a feature. It's something I can live without, but would love to have.
Jul 11 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
I was thinking instead of prohibiting the import we add a "protection  
attribute" to handle this.

Trying to create a concrete example and use "package" shows that package  
cannot handle it. Instead an attribute ("library") which allowed access if  
the module belonged to the same top level package would work I believe. eg.

[yourlib/foo.d]
module yourlib.foo;

[yourlib/_foo/_bill/a.d]
module yourlib._foo._bill.a;
library: //access for all yourlib.* module/packages

If a client imports "yourlib/_foo/_bill/a.d" they cannot access anything,  
yet at the same time "yourlib/foo.d" has access to everything in the  
yourlib.* tree.

That said, in my concrete example I was using 'version' and 'alias' in  
"yourlib/foo.d" to select the implementation, which of course then  
requires access to that implementation from outside the tree, defeating  
the initial purpose.

Perhaps prohibition is the only solution. Instead of "hidden" could we use  
the existing protection attributes? eg.

public module foo.bar;    //importable by everyone
protected module foo.bar; //importable by foo.*
package module foo.bar;   //importable by foo.bar.*
private module foo.bar;   //not importable

Or similar.

Regan
Jul 11 2005
parent Mike Parker <aldacron71 yahoo.com> writes:
Regan Heath wrote:
 
 Perhaps prohibition is the only solution. Instead of "hidden" could we 
 use  the existing protection attributes? eg.
 
 public module foo.bar;    //importable by everyone
 protected module foo.bar; //importable by foo.*
 package module foo.bar;   //importable by foo.bar.*
 private module foo.bar;   //not importable
 
 Or similar.

Call it anything you'd like :) I'm only interested in the feature itself. What you suggest though extends the idea much further, which I like. But looking at your example... I'm thinking this is what you meant: public module foo.bar.baz; //importable by everyone protected module foo.bar.baz; //importable by foo.* package module foo.bar.baz; //importable by foo.bar.* private module foo.bar.baz; //not importable So 'protected' modules would be accessible up to the root package, while 'package' modules would be accessible within the same package. Actually, package protection now extends to subpackages: module foo.module; package const int MYINT = 1; module foo.bar.module; import foo.module; // has access to MYINT So, if we were to apply the same sort of protection to modules, perhaps we should say that a 'package' module could be imported by modules in the same package or in subpackages but not those in super packages, while a 'private' module could only be imported by modules within the same package.
Jul 12 2005
prev sibling parent reply Vathix <chris dprogramming.com> writes:
On Sun, 10 Jul 2005 03:35:29 -0400, Mike Parker <aldacron71 yahoo.com>  
wrote:

 So why not a new level of module protection that hides a given module  
 from the client - meaning, it can't be imported at all outside of a  
 particular part of the package tree.

I did think of something like this before, but the syntax I thought of was: package module foo.bar; meaning the module itself has package access.
Jul 10 2005
parent Mike Parker <aldacron71 yahoo.com> writes:
Vathix wrote:

 I did think of something like this before, but the syntax I thought of was:
    package module foo.bar;
 meaning the module itself has package access.

That works too, particularly since we already have the package keyword. But 'package' implies that it is only available in the same package and subpackages. What I'm looking for is giving access one level up as well, but I suppose either would be acceptable.
Jul 10 2005