www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 11330] New: Directory named as imported module should not stop module search

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330

           Summary: Directory named as imported module should not stop
                    module search
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: regression
          Priority: P2
         Component: DMD
        AssignedTo: nobody puremagic.com
        ReportedBy: thecybershadow gmail.com



17:47:53 EEST ---
This is a regression that appears to be caused by implementation of DIP37.

Directory structure:
├───b
│       (empty directory)
└───src
        a.d - contains "import b;" 
        b.d - empty file

Command line:
dmd -c -Isrc src/a

DMD 2.063.2 succeeds.

DMD from git complains:

src\a.d(1): Error: importing package 'b' requires a 'package.d' file which
cannot be found in 'b\package.d'

DMD shouldn't stop searching for a b.d or b\package.d just because it finds a
"b" directory first.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 23 2013
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330


Kenji Hara <k.hara.pg gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID



This is an invalid issue, as same as bug 11241 and bug 11243.

In your project, the module/package name 'b' should be resolved to the
first-found file system entry "./b/". The new package module feature looks for
"./b/package.d", then compiler correctly fails to search the file.

In other words: if a fully-qualified module name 'xx.yy.zz' would match to
multiple file/directories through the whole import paths, it is essentially not
good.

See also: http://d.puremagic.com/issues/show_bug.cgi?id=11241#c1

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 23 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330


Vladimir Panteleev <thecybershadow gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|INVALID                     |



18:37:39 EEST ---
If the behavior is by design, then the design is broken and needs to be fixed.
Reopening.

Compilers should not treat source files, which use an extension specifically
chosen for the programming language, the same as directories, which can contain
anything at all.

The current behavior is equivalent to the compiler complaining that there is a
file called "b.txt", because its base name is the same as a module/package used
in the program. It's not the compiler's business what directories there are in
the search path, because they can contain arbitrary files completely unrelated
to the task of building the program.

The generalized version of my situation is:

Let's say you have a program that manages foos. Your program consists of
several entry point modules, such as mangle-foos.d and unmangle-foos.d, and a
module which contains code that manages foos, foos.d. If you build the program
from its source directory, everything is fine. However, if you go to your
project directory, which incidentally contains a directory called "foos"
containing the data to be managed, and you run "rdmd
foosmangler/mangle-foos.d", the build breaks. In my opinion, this is not
acceptable.

My specific use case is a build tool used to configure, build and deploy a
software project. The tool contains a module called "source", which manages the
project's source code. If I run it (using rdmd) from the project root, which
contains a directory called "source", it breaks. I don't think I need to rename
my modules or my source directory to work around this design flaw.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 23 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330


Dicebot <public dicebot.lv> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |public dicebot.lv



It is also important to remember that DIP37 was intended as breakage-free
transparent change.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 23 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




Hmm. OK, confirmed your point. But I don't think that the current
behavior/design is broken. 

Up to 2.063, d and di files has been considered as a source of the compiler.
From 2.064, directory name will be added to the compiler's input by
implementing DIP37. That's intended change.

And, I'd like to explain why the current behavior is necessary.
If you have the following project:

./
 +--src1/
 |  +--a/
 |     +--...
 +--src2/
 |  +--a.d
 +--src3/
    +--b.d  (contains "import a;")

And you use the command line "dmd -Isrc1 -Isrc2 -Isrc3 ..." to compile project.
If compiler ignores the directory "src1/a/" when resolving "import a;", it
would be used to import "src2/a.d".
Then, what happen if you add a new file src1/a/package.d by using DIP37
feature? It will break existing the code "import a;'in "src3/b.d". The occurred
problem is contrary to the principle in DIP37- adding package.d should be
breakage-free transparent change.

To avoid the problem, if the directory "a" does not contain package.d file,
compiler should raise an error on "import a;" from the beginning.

========

On the other hand, in the OP case, I think the root issue is that dmd
implicitly adds the current directory in import paths by default. When you use
dmd as just a part of build-tool, the implicit addition could cause problems
like your case.

An idea based on my thought is: add a new compiler switch "--I." which removes
current directory from the import path list.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 23 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




20:06:50 EEST ---


 It will break existing the code "import a;'in "src3/b.d". The occurred
 problem is contrary to the principle in DIP37- adding package.d should be
 breakage-free transparent change.


I understand that the goal is to avoid hijacking when a package.d file is
added. If that's the case, then the issue can be resolved by making the
compiler complain if it finds both a.d and a/package.d in its search paths.

However, if you think about it more, shouldn't the compiler also complain if it
finds more than one match in its search path? Because that is also a hijacking
opportunity - remove the first match, and suddenly the compiler "silently" is
using some other version of the module. But would making the compiler complain
about ambiguous search path matches break too many environments?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 23 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330






 However, if you think about it more, shouldn't the compiler also complain if it
 finds more than one match in its search path? Because that is also a hijacking
 opportunity - remove the first match, and suddenly the compiler "silently" is
 using some other version of the module. But would making the compiler complain
 about ambiguous search path matches break too many environments?


Essentially yes. But it would introduce extra cost to search filesystem. If
import path list is huge - I think it's a possible case if you manage thousands
of third party projects by using package system - I cannot say the extra cost
is enough small. I'm not sure it is a compiler's work.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 23 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330


Walter Bright <bugzilla digitalmars.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
                 CC|                            |bugzilla digitalmars.com
         Resolution|                            |WONTFIX



14:02:10 PDT ---
1. The compiler has always relied on a "first found" algorithm when searching
the import paths. That's like using nested scopes when searching for a symbol.
This is a valuable feature as it allows for overriding (and I use this,
especially when debugging). There's a clear hierarchy of this. (The
anti-hijacking support in D comes into play for cases where there is no clear
hierarchy.)

2. The import system is designed to map onto the file system. Using
file/directory names that match module/package names is how it is supposed to
work, if there are matching names that have nothing to do with D will cause
problems.

I don't see any reasonable resolution for the regression issue you identified.
The only thing I can think of is don't name things with the same names as D
modules/packages when they are in the import path.

I'm going to resolve as wontfix for now. If a workable solution appears, please
reopen.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 26 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




00:04:50 EEST ---

 If a workable solution appears, please reopen.


What conditions must be satisfied for a solution to be deemed as "workable"?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 26 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




00:37:00 EEST ---


 1. The compiler has always relied on a "first found" algorithm when searching
 the import paths. That's like using nested scopes when searching for a symbol.
 This is a valuable feature as it allows for overriding (and I use this,
 especially when debugging). There's a clear hierarchy of this. (The
 anti-hijacking support in D comes into play for cases where there is no clear
 hierarchy.)


I don't see how a list of search paths is like a number of nested scopes. One
is a flat list, the other has clear hierarchy.

This is besides my argument, anyway, so I understand you're replying about the
suggestion for module hijack protection? This quote:


 I understand that the goal is to avoid hijacking when a package.d file is
 added.


If that's not the case, then it's even more unclear to me why this change was
introduced, and why it is considered beneficial.


 2. The import system is designed to map onto the file system. Using
 file/directory names that match module/package names is how it is supposed to
 work, if there are matching names that have nothing to do with D will cause
 problems.




Also, without context, your quote can be used to defend a hypothetical change
in DMD to reject "modulename.txt" files, if they exist in the search path and
"modulename" is imported somewhere, which is absurd. Which is the main point of
my argument: D compilers should not treat filesystem directories that match
imported D modules as if they are made for use for D. ".d" files are for D
compilers, directories are for everyone.

As far as I know, no other implementation of a package system in any other
language behaves as DMD behaves now.


 I'm going to resolve as wontfix for now. If a workable solution appears, please
 reopen.


I hope this isn't an attempt to sweep a regression under the rug to get 2.064
out quicker:
- I have stumbled upon this problem twice, in two different situations.
- This change affected two other users (issue 11241 and issue 11243).

I would appreciate a prompt formulation of the problem that this change was
supposed to solve, and the constraints that solutions must meet, so we can work
on a solution that satisfies all use cases and have it implemented before
2.064's release. Right now I don't have much to work with.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 26 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330


Vladimir Panteleev <thecybershadow gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WONTFIX                     |



01:59:14 EET ---
Since the ball's in the maintainers' court (per my requests for clarification
above) and there has been no reply in 24 hours, reopening so this doesn't get
lost.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




23:45:52 PDT ---


 I don't see how a list of search paths is like a number of nested scopes.


It's always been that way, and C and C++ compilers operate the same way. To
override, you put a path in front of the list of paths. The clear hierarchy is
the ones earlier in the list override the ones later, i.e. the "first found"
rule. Just like nested scopes.



 This is besides my argument, anyway, so I understand you're replying about the
 suggestion for module hijack protection?


This was in response to the idea that errors should be generated if a file
appears in more than one path.



 If that's not the case, then it's even more unclear to me why this change was
 introduced, and why it is considered beneficial.


The change was made so that existing modules, like std.algorithm, could be
split into multiple modules without breaking existing code.

See http://wiki.dlang.org/DIP37



 2. The import system is designed to map onto the file system. Using
 file/directory names that match module/package names is how it is supposed to
 work, if there are matching names that have nothing to do with D will cause
 problems.


 

 
 Also, without context, your quote can be used to defend a hypothetical change
 in DMD to reject "modulename.txt" files, if they exist in the search path and
 "modulename" is imported somewhere, which is absurd. Which is the main point of
 my argument: D compilers should not treat filesystem directories that match
 imported D modules as if they are made for use for D. ".d" files are for D
 compilers, directories are for everyone.


Directory names have always been regarded as having a 1:1 correspondence with
package names in D. This is not new. std.algorithm corresponds to
std/algorithm.d



 As far as I know, no other implementation of a package system in any other
 language behaves as DMD behaves now.


As I recall, this was debated at length and was implemented months ago.



 I hope this isn't an attempt to sweep a regression under the rug to get 2.064
 out quicker:
 - I have stumbled upon this problem twice, in two different situations.
 - This change affected two other users (issue 11241 and issue 11243).


I am sorry it breaks some existing setups, but the behavior is designed, and is
not a bug. There were several proposals, and this one broke the least and
before this everyone has liked it.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




15:33:14 EET ---


 The change was made so that existing modules, like std.algorithm, could be
 split into multiple modules without breaking existing code.


This does not explain or justify the problem this bug report is describing.


 See http://wiki.dlang.org/DIP37


The DIP specification says NOTHING about treating the first directory in its
search path as the match for a package. As far as I can see, the behavior in
question is an artifact of a poor implementation. In fact, the DIP says the
following:


 Also, having a package and module with the same name 
 will result in an ambiguity error when you try and 
 import them (e.g. foo/bar/package.d and foo/bar.d).


So the current behavior actually deviates from the design!


 Directory names have always been regarded as having a 1:1 correspondence with
 package names in D. This is not new. std.algorithm corresponds to
 std/algorithm.d


This does not explain or justify the problem this bug report is describing.


 As I recall, this was debated at length and was implemented months ago.


I do not recall any discussion that reached a consensus that the first
directory in D's search path should be treated as the match for an import of
the same name.


 I am sorry it breaks some existing setups, but the behavior is designed, and is
 not a bug. There were several proposals, and this one broke the least and
 before this everyone has liked it.


I am sorry, but judging by your arguments quoted above, I must conclude that
you either don't understand the problem, or are unable to justify the current
behavior. I must ask you to read my arguments carefully and respond to my
clarification requests directly.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 28 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




15:50:06 EET ---
Assuming good faith and that the problem is miscommunication, here's a
clarification of the problem this bug report is describing:

1) I am not against anything written on http://wiki.dlang.org/DIP37.

2) I only have a problem with treating directories (disregarding their
contents) as matches on the search path.

3) My suggestion is to also check for the "package.d" file inside the directory
when considering a directory as a search path match. If it does not exist, then
continue search as if that directory did not exist.

Would this be acceptable? If not, why not?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 28 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




23:13:35 PDT ---


 3) My suggestion is to also check for the "package.d" file inside the directory
 when considering a directory as a search path match. If it does not exist, then
 continue search as if that directory did not exist.


Makes sense.

https://github.com/D-Programming-Language/dmd/pull/2696

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 28 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




08:16:04 EET ---
Phew :) I'm really glad this was just a miscommunication. Thank you!

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 28 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




Commits pushed to master at https://github.com/D-Programming-Language/dmd

https://github.com/D-Programming-Language/dmd/commit/671161e4d5340afcb618c6cf3ded2c6d991a8c02
fix Issue 11330 - Directory named as imported module should not stop module
search

https://github.com/D-Programming-Language/dmd/commit/e5a0c64af40264744f0eed497b7ed517b1df7c4e


fix Issue 11330 - Directory named as imported module should not stop module
search

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 30 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330




Commit pushed to 2.064 at https://github.com/D-Programming-Language/dmd

https://github.com/D-Programming-Language/dmd/commit/eb1b00441491f1be554892963358462506c78403


fix Issue 11330 - Directory named as imported module should not stop module
search

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 30 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11330


Walter Bright <bugzilla digitalmars.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |FIXED


-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 30 2013