www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Feature Idea: hidden modules

reply Mike Parker <aldacron71 yahoo.com> writes:
I recently came across a design situation where the concept of 'hidden' 
modules would prove useful, particularly since we can't have package and 
  modules with the same at the same level. Consider this package structure:

mypackage.platform;
mypackage.platformimpl.win32;
mypackage.platformimpl.linux;
mypackage.platformimpl.sdl;
mypackage.platformimpl.glfw;

Platform is setup to import the proper implementation based upon a 
version conditional. The problem is that all of the modules in the 
platofrmimpl package are visible to the client and can be imported 
freely. One might consider this as a solution:

mypackage.platform.platform;
mypackage.platform.win32;
mypackage.platform.linux;
...

Then the implementation modules could each be wrapped with package 
protection so that they are only visible to the platform.platform 
module. This works well if you are dealing with predefined interfaces. 
But in this case, there's no sense in using an interface - an app will 
only be compiled to use one platform at a time. From a design 
standpoint, it makes more sense in this case that each module just 
implement the necessary interface without an actual interface definition 
- perhaps as a struct with static methods. Then, the platform module 
just does this:

version(Win32App)
    import mypackage.platformimpl.win32;
else version(LinuxApp)
    import mypackage.platformimpl.linux;

Package protection doesn't work in this case because the contents of 
each platform file need to be exposed to the client, sense there is no 
abstract interface defined. Without some mechanism to allow you to hide 
a particular module from the client, there is no way around the need to 
use an interface + package protection if you wish to hide the imlementation.

So why not a new level of module protection that hides a given module 
from the client - meaning, it can't be imported at all outside of a 
particular part of the package tree. Perhaps something along these lines:

// module declaration using new 'hidden' keyword
hidden module mypackage.platformimpl.win32;

This would make the win32 module visible only to modules that meet the 
following criteria:

* any module that is in a subpackage of platformimpl can import it. ex: 
mypackage.platformimpl.anotherpackage.somemodule

* any module that is in the same package as the hidden module can import 
it. ex: mypackage.platformimpl.somemodule

* any module that is exactly one level above the hidden module's package 
can import it. ex: mypackage.somemodule

Regarding the third point, it might be useful to allow any module above 
the hidden module up to the root package to import it. For example:

foo.bar.baz.mymodule (hidden)
can be imported by foo.bar.module and foo.module

However, I think it better if it would be restricted only to one level 
above the hidden module's package. This allows you to provide an 
'adapter module' which imports the hidden modules, and any module that 
needs to use the hidden module's functionality must go through the 
adapter. This allows better encapsulation and avoids issues that might 
arise from importing the hidden modules directly in the root package.

Just something I thought might prove useful in providing a little more 
design flexibility over what we have already (and adds more weight to 
the need for a module statement, which some people have questioned 
recently).
Jul 10 2005
next sibling parent reply Mike Parker <aldacron71 yahoo.com> writes:
That didn't come out as clearly as it did in my head. If you're 
confused, I'll try to clarify it :)
Jul 10 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Sun, 10 Jul 2005 16:38:32 +0900, Mike Parker <aldacron71 yahoo.com>  
wrote:

 That didn't come out as clearly as it did in my head. If you're  
 confused, I'll try to clarify it :)


I may be confused. What is wrong with:

[mypackage.platform.d]
module mypackage.platform
version(Win32) import mypackage.platformimpl.win32;
version(linux) import mypackage.platformimpl.linux;
version(sdl) import mypackage.platformimpl.sdl;
version(glfw) mypackage.platformimpl.glfw;

[mypackage.platformimpl.win32.d]
module mypackage.platformimpl.win32
version(Win32):
..etc..

and so on for each module.

This prevents an app compiled on "linux" from importing  
"mypackage.platformimpl.win32".

Or is the real problem with hiding parts of the implementation? Does  
"mypackage.platform" need access to parts of  
"mypackage.platformimpl.win32"? are you trying to prevent users from  
having access to those same parts?

Perhaps a small example will help, then I can experiment too.

Regan
Jul 10 2005
parent reply Mike Parker <aldacron71 yahoo.com> writes:
Regan Heath wrote:

 On Sun, 10 Jul 2005 16:38:32 +0900, Mike Parker <aldacron71 yahoo.com>  
 wrote:
 

 That didn't come out as clearly as it did in my head. If you're  
 confused, I'll try to clarify it :)


 
 
 I may be confused. What is wrong with:
 
 [mypackage.platform.d]
 module mypackage.platform
 version(Win32) import mypackage.platformimpl.win32;
 version(linux) import mypackage.platformimpl.linux;
 version(sdl) import mypackage.platformimpl.sdl;
 version(glfw) mypackage.platformimpl.glfw;
 
 [mypackage.platformimpl.win32.d]
 module mypackage.platformimpl.win32
 version(Win32):
 ..etc..
 
 and so on for each module.
 
 This prevents an app compiled on "linux" from importing  
 "mypackage.platformimpl.win32".


No it doesn't. An app compiled on linux could still directly import 
mypackage.platformimpl.win32 directly, bypassing mypackage.platform 
altogether.


 
 Or is the real problem with hiding parts of the implementation? 


That's it exactly. The platform stuff was probably not the best example, 
but it's what I was working on when the idea hit me. To break it down 
simply:

mypackage.foo.bar.module1
mypackage.foo.bar.module2
mypackage.foo.bar.module3
mypackage.foo.moduleselector

The goal is to prevent any module outside of mypackage.foo (including 
anything in mypackage) from importing module1, module2, and module3 
directly, forcing them to access that functionality through moduleselector.

Obviously, you could wrap module1, module2, and module3 all in package 
protection statements and do this:

mypackage.foo.bar.moduleselector

With moduleselector in the same package, it is allowed access to 
declarations in all 3 modules. Outside modules will be forced to go 
through moduleselector since package protection hides everything. Really 
this solution is fine. But there's a minor drawback.

All modules that moduleselector manages must be in the same package. 
 From a design perspective, this prevents logical grouping of modules 
when multiple modules are involved. Consider this:

mypackage.foo.bar.module1a
mypackage.foo.bar.module1b
mypackage.foo.bar.module1c
mypackage.foo.bar.module1

If modules 2 & 3 have multiple modules that go together, now you have a 
big jumbled mess. With the hidden module concept as I proposed it, you 
could do this:

mypackage.foo.implone.module1a
mypackage.foo.implone.module1b
mypackage.foo.implone.module1c
mypackage.foo.implone.module1
mypackage.foo.impltwo.module2a
...
mypackage.foo.moduleselector

So the benefits from such a mechanism are design flexibility, and the 
ability to prevent some modules from being imported outside of a 
particular package tree. This can be very useful when designing library.

It's not something that we can't live without, but I do think it's 
something that could prove useful.
Jul 10 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Mon, 11 Jul 2005 10:40:10 +0900, Mike Parker <aldacron71 yahoo.com>  
wrote:

 Regan Heath wrote:

 On Sun, 10 Jul 2005 16:38:32 +0900, Mike Parker <aldacron71 yahoo.com>   
 wrote:


 That didn't come out as clearly as it did in my head. If you're   
 confused, I'll try to clarify it :)


   I may be confused. What is wrong with:
  [mypackage.platform.d]
 module mypackage.platform
 version(Win32) import mypackage.platformimpl.win32;
 version(linux) import mypackage.platformimpl.linux;
 version(sdl) import mypackage.platformimpl.sdl;
 version(glfw) mypackage.platformimpl.glfw;
  [mypackage.platformimpl.win32.d]
 module mypackage.platformimpl.win32
 version(Win32):
 ..etc..
  and so on for each module.
  This prevents an app compiled on "linux" from importing   
 "mypackage.platformimpl.win32".


 No it doesn't. An app compiled on linux could still directly import  
 mypackage.platformimpl.win32 directly, bypassing mypackage.platform  
 altogether.


True, I wasn't meant to say that exactly. What I meant was, they can do it  
but it causes no problems. No compile errors. No access to things that are  
supposed to be hidden/protected (assuming "package" is used).


  Or is the real problem with hiding parts of the implementation?


 That's it exactly. The platform stuff was probably not the best example,  
 but it's what I was working on when the idea hit me. To break it down  
 simply:

 mypackage.foo.bar.module1
 mypackage.foo.bar.module2
 mypackage.foo.bar.module3
 mypackage.foo.moduleselector

 The goal is to prevent any module outside of mypackage.foo (including  
 anything in mypackage) from importing module1, module2, and module3  
 directly, forcing them to access that functionality through  
 moduleselector.

 Obviously, you could wrap module1, module2, and module3 all in package  
 protection statements and do this:

 mypackage.foo.bar.moduleselector

 With moduleselector in the same package, it is allowed access to  
 declarations in all 3 modules. Outside modules will be forced to go  
 through moduleselector since package protection hides everything. Really  
 this solution is fine. But there's a minor drawback.


This is the solution I was suggesting/hinting at.


 All modules that moduleselector manages must be in the same package.  
  From a design perspective, this prevents logical grouping of modules  
 when multiple modules are involved. Consider this:

 mypackage.foo.bar.module1a
 mypackage.foo.bar.module1b
 mypackage.foo.bar.module1c
 mypackage.foo.bar.module1

 If modules 2 & 3 have multiple modules that go together, now you have a  
 big jumbled mess.


I must be dense today.. why is it a jumbled mess?

[mypackage.foo.bar.module1.d]
module mypackage.foo.bar.module1
version(1A) import mypackage.foo.bar.module1a

[mypackage.foo.bar.module1a.d]
version(1A): package:
..etc..

[mypackage.foo.bar.module2.d]
module mypackage.foo.bar.module1
version(2A) import mypackage.foo.bar.module1a

[mypackage.foo.bar.module2a.d]
version(2A): package:
..etc..

[program.d]
import mypackage.foo.bar.module1;
import mypackage.foo.bar.module2;

Seems fine to me.


 With the hidden module concept as I proposed it, you could do this:

 mypackage.foo.implone.module1a
 mypackage.foo.implone.module1b
 mypackage.foo.implone.module1c
 mypackage.foo.implone.module1
 mypackage.foo.impltwo.module2a
 ...
 mypackage.foo.moduleselector

 So the benefits from such a mechanism are design flexibility, and the  
 ability to prevent some modules from being imported outside of a  
 particular package tree. This can be very useful when designing library.

 It's not something that we can't live without, but I do think it's  
 something that could prove useful.


I'm not sure we need to "prevent" people importing things, we just have to  
have a method of protecting code while also making it possible to divide a  
module into several logical units (files). "package" is the method which  
was added to facilitate this.

Perhaps I'm just not getting it? In which case can you give a more  
concrete example of a specific problem?

Regan
Jul 10 2005
parent reply Mike Parker <aldacron71 yahoo.com> writes:
Regan Heath wrote:


 mypackage.foo.bar.module1a
 mypackage.foo.bar.module1b
 mypackage.foo.bar.module1c
 mypackage.foo.bar.module1

 If modules 2 & 3 have multiple modules that go together, now you have 
 a  big jumbled mess.


 
 
 I must be dense today.. why is it a jumbled mess?


Because if you have multiple 'groups' of modules (1a/b/c/d, 2/a/b/c/d, 
3a/b/c/d) in the same package, there's no logical organization - all of 
them are jumbled together in the same package (necessary when using 
package protection). What is much cleaner is this:

mypackage.implone.module1a/b/c/d
mypackage.impltwo.module2a/b/c/d
mypackage.implthree.module3a/b/c/d
mypackage.moduleselector

By allowing the modules in the implone/two/three packages to be hidden 
to oustsiders (except those below and one level above), you can now 
logically structure your packages and keeo a clean separation of module 
groups, rather than having them all bunched up in one package.



 Perhaps I'm just not getting it? In which case can you give a more  
 concrete example of a specific problem?


It's not a problem, per se. Nothing's really broken here. It's about 
adding more flexibility to the design through package structure. The 
package keyword is a big improvement over when we didn't have it, but it 
is still limited in that it forces everything to be in the same package 
or lower, and other modules can still access anything not protected by 
the package keyword when they import a particular module.
Jul 10 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Mon, 11 Jul 2005 14:41:35 +0900, Mike Parker <aldacron71 yahoo.com>  
wrote:

 Regan Heath wrote:


 mypackage.foo.bar.module1a
 mypackage.foo.bar.module1b
 mypackage.foo.bar.module1c
 mypackage.foo.bar.module1

 If modules 2 & 3 have multiple modules that go together, now you have  
 a  big jumbled mess.


   I must be dense today.. why is it a jumbled mess?


 Because if you have multiple 'groups' of modules (1a/b/c/d, 2/a/b/c/d,  
 3a/b/c/d) in the same package, there's no logical organization - all of  
 them are jumbled together in the same package (necessary when using  
 package protection).


So, you're saying you cannot do this:

mypackage/implone/a
mypackage/implone/b
mypackage/implone/c

mypackage/impltwo/a
mypackage/impltwo/b
mypackage/impltwo/c

..

because implone and impltwo require access to each others internal data?

This seems odd to me, no offense but it suggests either bad design, or  
that they should be in the same package or even the same file if they're  
that tightly bound.

Without a concrete example I can only speculate.


 What is much cleaner is this:

 mypackage.implone.module1a/b/c/d
 mypackage.impltwo.module2a/b/c/d
 mypackage.implthree.module3a/b/c/d
 mypackage.moduleselector

 By allowing the modules in the implone/two/three packages to be hidden  
 to oustsiders (except those below and one level above), you can now  
 logically structure your packages and keeo a clean separation of module  
 groups, rather than having them all bunched up in one package.


Making them 'hidden' would probably achieve your goal, but, there are  
likely several solutions and I think I'd prefer one that was more like  
"package". I mean there is no point prohibiting an import if you can  
instead ensure it is meaningless (i.e. they cannot gain access to things  
they should't) thats my opinion anyway.


 Perhaps I'm just not getting it? In which case can you give a more   
 concrete example of a specific problem?


 It's not a problem, per se. Nothing's really broken here.


I realise that.


 It's about adding more flexibility to the design through package  
 structure. The package keyword is a big improvement over when we didn't  
 have it, but it is still limited in that it forces everything to be in  
 the same package or lower,


I haven't had a problem with this restriction as yet, I think a concrete  
example i.e. show me exactly what you tried to do when you found this.


 and other modules can still access anything not protected by the package  
 keyword when they import a particular module.


That is because without "package" or another protection attribute it  
defaults to "public". We should probably get into the habit of putting:

package:

at the top of all our source (which suggests to me that this should be the  
default, not public)

Regan
Jul 11 2005
parent reply Mike Parker <aldacron71 yahoo.com> writes:
Regan Heath wrote:


 Regan Heath wrote:


 mypackage.foo.bar.module1a
 mypackage.foo.bar.module1b
 mypackage.foo.bar.module1c
 mypackage.foo.bar.module1

 If modules 2 & 3 have multiple modules that go together, now you 
 have  a  big jumbled mess.


   I must be dense today.. why is it a jumbled mess?



 Because if you have multiple 'groups' of modules (1a/b/c/d, 
 2/a/b/c/d,  3a/b/c/d) in the same package, there's no logical 
 organization - all of  them are jumbled together in the same package 
 (necessary when using  package protection).


 
 
 So, you're saying you cannot do this:
 
 mypackage/implone/a
 mypackage/implone/b
 mypackage/implone/c
 
 mypackage/impltwo/a
 mypackage/impltwo/b
 mypackage/impltwo/c
 
 ..
 
 because implone and impltwo require access to each others internal data?


No, that's not what I'm saying at all. I'm not sure if I'm not 
explaining myself clearly or not, but you are completely missing my 
point. Imagine you have a single module that provides an interface to 
implone and impltwo in mypackage.myinterfacemodule. You only want 
clients of your package to access implone and impltwo through the 
interface module, and not allow them to import them directly, thereby 
hiding the implementation from the client. Right now, there is no 
mechanism in D that allows this. The workaround is to put the interface 
module and all of the implementatoins in the same directory, but then 
you lose logical separation of the implementations.

Imagine a team of programmers - Bob, Joe and Bill - who are working on a 
D library. One of the subsytems in the library is the foo subsystem, of 
which there could be several possible implementations. So, they develop 
a foo interface and each programmer develops his own custom 
implementation. This is the resulting package structure:

teamlib.foomodule
teamlib.bobfoo.modulesa/b/c/d/e
teamlib.joefoo.modulea/b/c
teamlib.billfoo.modulea/b/c/d

Perhaps the implemetation can be chosen at compile time, perhaps at 
runtime, it doesn' matter. Currently in D, any part of the 
implementation that is not package protected can be imported directly 
and accessed by the library client. But, for whatever reason, the team 
doesn't want the client to access anything directly - they want would 
prefer if the client goes through foomodule instead. Because there is 
currently no way in D to achieve this goal with the given package 
structure, they'll need to wrap their implementation modules in package 
protection and do this:

teamlib.foo.foomodule
teamlib.foo.bobfooa/b/c/d/e
teamlib.foo.joefooa/b/c
teamlib.foo.billfooa/b/c/d

So now instead of a logical separation of implementations, you have all 
12 modules in the same package. Additionally, the implementaion modules 
have access to each other (which may not be desirable). The first method 
is structurally cleaner, easier to maintain (when you are talking about 
numerous files), and hides implementations from each other.

That's what I'm trying to get across, and I hope I have succeeded this 
time. Having the ability to prevent modules from being imported outside 
of a particular scope could come in handy for library designs, I think. 
I know I can see uses for such a feature. It's something I can live 
without, but would love to have.
Jul 11 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
I was thinking instead of prohibiting the import we add a "protection  
attribute" to handle this.

Trying to create a concrete example and use "package" shows that package  
cannot handle it. Instead an attribute ("library") which allowed access if  
the module belonged to the same top level package would work I believe. eg.

[yourlib/foo.d]
module yourlib.foo;

[yourlib/_foo/_bill/a.d]
module yourlib._foo._bill.a;
library: //access for all yourlib.* module/packages

If a client imports "yourlib/_foo/_bill/a.d" they cannot access anything,  
yet at the same time "yourlib/foo.d" has access to everything in the  
yourlib.* tree.

That said, in my concrete example I was using 'version' and 'alias' in  
"yourlib/foo.d" to select the implementation, which of course then  
requires access to that implementation from outside the tree, defeating  
the initial purpose.

Perhaps prohibition is the only solution. Instead of "hidden" could we use  
the existing protection attributes? eg.

public module foo.bar;    //importable by everyone
protected module foo.bar; //importable by foo.*
package module foo.bar;   //importable by foo.bar.*
private module foo.bar;   //not importable

Or similar.

Regan
Jul 11 2005
parent Mike Parker <aldacron71 yahoo.com> writes:
Regan Heath wrote:

 
 Perhaps prohibition is the only solution. Instead of "hidden" could we 
 use  the existing protection attributes? eg.
 
 public module foo.bar;    //importable by everyone
 protected module foo.bar; //importable by foo.*
 package module foo.bar;   //importable by foo.bar.*
 private module foo.bar;   //not importable
 
 Or similar.


Call it anything you'd like :) I'm only interested in the feature 
itself. What you suggest though extends the idea much further, which I 
like. But looking at your example... I'm thinking this is what you meant:

public module foo.bar.baz;    //importable by everyone
protected module foo.bar.baz; //importable by foo.*
package module foo.bar.baz;   //importable by foo.bar.*
private module foo.bar.baz;   //not importable

So 'protected' modules would be accessible up to the root package, while 
'package' modules would be accessible within the same package. Actually, 
  package protection now extends to subpackages:

module foo.module;
package const int MYINT = 1;

module foo.bar.module;
import foo.module;

// has access to MYINT

So, if we were to apply the same sort of protection to modules, perhaps 
we should say that a 'package' module could be imported by modules in 
the same package or in subpackages but not those in super packages, 
while a 'private' module could only be imported by modules within the 
same package.
Jul 12 2005
prev sibling parent reply Vathix <chris dprogramming.com> writes:
On Sun, 10 Jul 2005 03:35:29 -0400, Mike Parker <aldacron71 yahoo.com>  
wrote:


 So why not a new level of module protection that hides a given module  
 from the client - meaning, it can't be imported at all outside of a  
 particular part of the package tree.


I did think of something like this before, but the syntax I thought of was:
    package module foo.bar;
meaning the module itself has package access.
Jul 10 2005
parent Mike Parker <aldacron71 yahoo.com> writes:
Vathix wrote:


 I did think of something like this before, but the syntax I thought of was:
    package module foo.bar;
 meaning the module itself has package access.


That works too, particularly since we already have the package keyword. 
But 'package' implies that it is only available in the same package and 
subpackages. What I'm looking for is giving access one level up as well, 
but I suppose either would be acceptable.
Jul 10 2005