www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compile Imported Modules

reply Jonathan Marler <johnnymarler gmail.com> writes:
Wanted to get peoples thoughts on this.  The idea is to have a 
way to tell the compiler (probably with a command line option) 
that you'd like to "compile imported modules".  Say you have a 
program "prog" that depends on modules "foo" and "bar".

     import foo;
     import bar;

Compilation could look like:

     dmd prog.d foo.d bar.d

Or it could look like

     dmd -c foo.d
     dmd -c bar.d
     dmd prog.d foo.obj bar.obj

With this command line option, let's call it "-compile-imports" 
for now, you could do something like:

     dmd -compile-imports prog.d

This tells the compiler that after it has processed all the input 
files (source code/object files/library files), if it is missing 
modules, it should go back and look for those modules in it's 
list of imported modules, then compile them from there.  It's 
important that it only checks this after processing all the input 
files so that precompiled modules take precedence.  So you could 
still do something like this:

     dmd -c foo.d
     dmd -compile-imports prog.d foo.obj

In this example we use the precompiled foo module and then the 
compiler notices that the bar module is missing.  So it looks for 
the source in it's list of imports, then includes that in it's 
list of files to compile essentialy behaving as if that file was 
passed on the command line.

This is a simple example with only 2 modules, but for projects 
that use alot of libraries it could turn into something like this:

     dmd prog.d -Isomelib somelib\foo\module1.d 
somelib\foo\module2.d somelib\foo\module3.d somelib\foo\module4.d 
somelib\foo\module5.d somelib\foo\module6.d -Ianotherlib 
anotherlib\bar\module1.d anotherlib\bar\module2.d 
anotherlib\bar\module3.d anotherlib\bar\module4.d 
anotherlib\bar\module5.d

into this:

     dmd -compile-imports prog.d -Isomelib -Ianotherlib

This would also simplify rdmd and make it "less brittle" because 
it will not need to duplicate the logic inside the compiler that 
locates and selects which module files to compile.  Instead, it 
can simply use the -compile-imports switch leave that logic 
completely in the compiler.
Aug 24 2017
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Aug 24, 2017 at 03:53:05PM +0000, Jonathan Marler via Digitalmars-d
wrote:

 Wanted to get peoples thoughts on this.  The idea is to have a way to
 tell the compiler (probably with a command line option) that you'd
 like to "compile imported modules".


[...]

Isn't this what rdmd already does?


T

-- 
Do not reason with the unreasonable; you lose by definition.
Aug 24 2017
next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 24 August 2017 at 15:56:32 UTC, H. S. Teoh wrote:

 On Thu, Aug 24, 2017 at 03:53:05PM +0000, Jonathan Marler via 
 Digitalmars-d wrote:

 Wanted to get peoples thoughts on this.  The idea is to have a 
 way to tell the compiler (probably with a command line option) 
 that you'd like to "compile imported modules".


 [...]

 Isn't this what rdmd already does?


 T


That is one thing that rdmd does (as I mentioned in the original 
post).

I just looked through the rdmd code 
(https://github.com/dlang/tools/blob/master/rdmd.d) and it looks 
like it invokes the compiler using "dmd -v" to get the list of 
modules and then invokes the compiler again with the modules it 
found to perform the full compile.  So my original thought that 
the logic to find modules is duplicated was incorrect.  Instead 
we just pay a performance hit to get the correct list of imports 
since running "dmd -v" seems to take almost as long as the actual 
compile itself.  So this method comes close to doubling the time 
it takes to compile than if the feature was implemented in the 
compiler itself.

In any case, the idea is to allow the compiler to resolve this on 
it's own without help from rdmd.  This would remove the need to 
invoke the compiler twice, once to find the imports and once to 
compile.  It would also allow some projects/applications that 
don't use rdmd to take advantage of this feature, this may or may 
not include dub (not sure on that one).
Aug 24 2017
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 24 August 2017 at 16:32:32 UTC, Jonathan Marler 
wrote:

 On Thursday, 24 August 2017 at 15:56:32 UTC, H. S. Teoh wrote:

 On Thu, Aug 24, 2017 at 03:53:05PM +0000, Jonathan Marler via 
 Digitalmars-d wrote:

 Wanted to get peoples thoughts on this.  The idea is to have 
 a way to tell the compiler (probably with a command line 
 option) that you'd like to "compile imported modules".


 [...]

 Isn't this what rdmd already does?


 T


 That is one thing that rdmd does (as I mentioned in the 
 original post).

 I just looked through the rdmd code 
 (https://github.com/dlang/tools/blob/master/rdmd.d) and it 
 looks like it invokes the compiler using "dmd -v" to get the 
 list of modules and then invokes the compiler again with the 
 modules it found to perform the full compile.  So my original 
 thought that the logic to find modules is duplicated was 
 incorrect.  Instead we just pay a performance hit to get the 
 correct list of imports since running "dmd -v" seems to take 
 almost as long as the actual compile itself.  So this method 
 comes close to doubling the time it takes to compile than if 
 the feature was implemented in the compiler itself.

 In any case, the idea is to allow the compiler to resolve this 
 on it's own without help from rdmd.  This would remove the need 
 to invoke the compiler twice, once to find the imports and once 
 to compile.  It would also allow some projects/applications 
 that don't use rdmd to take advantage of this feature, this may 
 or may not include dub (not sure on that one).


rdmd is really bad in terms of performance. If you call a single 
D file with rdmd, it will always compile it twice. There was an 
attempt to fix this (https://github.com/dlang/tools/pull/194), 
but this has been reverted as it introduced a regression and no 
one had time to look at the regression.
Moving rdmd into DMD has been on the TODO list for quite a while 
and there is a consensus that the performance overhead if rdmd 
isn't nice. However, IIRC there was no clear consensus on how the 
integration should happen. I recall that the plan was to do try 
this with "dmd as a library", but I'm not sure whether that's 
really feasible ATM.
Aug 24 2017
next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Aug 24, 2017 at 04:49:08PM +0000, Seb via Digitalmars-d wrote:
[...]

 rdmd is really bad in terms of performance. If you call a single D
 file with rdmd, it will always compile it twice. There was an attempt
 to fix this (https://github.com/dlang/tools/pull/194), but this has
 been reverted as it introduced a regression and no one had time to
 look at the regression.  Moving rdmd into DMD has been on the TODO
 list for quite a while and there is a consensus that the performance
 overhead if rdmd isn't nice. However, IIRC there was no clear
 consensus on how the integration should happen. I recall that the plan
 was to do try this with "dmd as a library", but I'm not sure whether
 that's really feasible ATM.


Hmm. An interesting thought occurred to me: dmd already has a -run
option, so perhaps it wouldn't be too hard to add an auto-import option
like Jonathan proposes, then dmd would essentially have the
functionality of rdmd?  Well, other than caching the executable, that
is.  But once auto-import is in, redundant compilation will become a
thing of the past, as rdmd could just invoke dmd, and the only thing
extra it would do is the executable caching.


T

-- 
What doesn't kill me makes me stranger.
Aug 24 2017
prev sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 24 August 2017 at 16:49:08 UTC, Seb wrote:

 On Thursday, 24 August 2017 at 16:32:32 UTC, Jonathan Marler 
 wrote:

 On Thursday, 24 August 2017 at 15:56:32 UTC, H. S. Teoh wrote:

 On Thu, Aug 24, 2017 at 03:53:05PM +0000, Jonathan Marler via 
 Digitalmars-d wrote:

 Wanted to get peoples thoughts on this.  The idea is to have 
 a way to tell the compiler (probably with a command line 
 option) that you'd like to "compile imported modules".


 [...]

 Isn't this what rdmd already does?


 T


 That is one thing that rdmd does (as I mentioned in the 
 original post).

 I just looked through the rdmd code 
 (https://github.com/dlang/tools/blob/master/rdmd.d) and it 
 looks like it invokes the compiler using "dmd -v" to get the 
 list of modules and then invokes the compiler again with the 
 modules it found to perform the full compile.  So my original 
 thought that the logic to find modules is duplicated was 
 incorrect.  Instead we just pay a performance hit to get the 
 correct list of imports since running "dmd -v" seems to take 
 almost as long as the actual compile itself.  So this method 
 comes close to doubling the time it takes to compile than if 
 the feature was implemented in the compiler itself.

 In any case, the idea is to allow the compiler to resolve this 
 on it's own without help from rdmd.  This would remove the 
 need to invoke the compiler twice, once to find the imports 
 and once to compile.  It would also allow some 
 projects/applications that don't use rdmd to take advantage of 
 this feature, this may or may not include dub (not sure on 
 that one).


 rdmd is really bad in terms of performance. If you call a 
 single D file with rdmd, it will always compile it twice. There 
 was an attempt to fix this 
 (https://github.com/dlang/tools/pull/194), but this has been 
 reverted as it introduced a regression and no one had time to 
 look at the regression.
 Moving rdmd into DMD has been on the TODO list for quite a 
 while and there is a consensus that the performance overhead if 
 rdmd isn't nice. However, IIRC there was no clear consensus on 
 how the integration should happen. I recall that the plan was 
 to do try this with "dmd as a library", but I'm not sure 
 whether that's really feasible ATM.


Well this should solve the rdmd performance problem as well as 
make other user cases easier that don't necessarilly use rdmd.

I had another thought that instead of making this an "opt-in" 
feature, it would probably make more sense to be an "opt-out" 
feature.  So by default the compiler would compile missing 
imported modules unless you indicate otherwise, maybe a command 
line switch like "-dont-compile-imports".  And I don't see how 
this would break anything.  Everything should work the same as it 
did before, it's just now you can omit imported module files from 
the command line and it should just work.
Aug 24 2017
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Aug 24, 2017 at 05:37:12PM +0000, Jonathan Marler via Digitalmars-d
wrote:
[...]

 I had another thought that instead of making this an "opt-in" feature,
 it would probably make more sense to be an "opt-out" feature.  So by
 default the compiler would compile missing imported modules unless you
 indicate otherwise, maybe a command line switch like
 "-dont-compile-imports".  And I don't see how this would break
 anything.  Everything should work the same as it did before, it's just
 now you can omit imported module files from the command line and it
 should just work.


Uh, no.  This will definitely break separate compilation, and some
people will be very unhappy about that.  I think it's good enough to
leave it as an opt-in feature.


T

-- 
Today's society is one of specialization: as you grow, you learn more and more
about less and less. Eventually, you know everything about nothing.
Aug 24 2017
parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 24 August 2017 at 17:49:27 UTC, H. S. Teoh wrote:

 Uh, no.  This will definitely break separate compilation, and 
 some people will be very unhappy about that.


I couldn't think of a case that it would break.  Can you share 
the cases you thought of?
Aug 24 2017
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Aug 24, 2017 at 06:00:15PM +0000, Jonathan Marler via Digitalmars-d
wrote:

 On Thursday, 24 August 2017 at 17:49:27 UTC, H. S. Teoh wrote:

 Uh, no.  This will definitely break separate compilation, and some
 people will be very unhappy about that.


 
 I couldn't think of a case that it would break.  Can you share the
 cases you thought of?


Suppose you have main.d and module.d, and you want to compile them
separately:

	dmd -c main.d
	dmd -c module.d
	dmd -ofmyprogram main.o module.o

If dmd defaulted to auto-importing, then `dmd -c main.d` would also
compile module.d (assuming main.d imports `module`), contrary to what
was intended in a separate compilation scenario, and the last command
will produce a linker error from duplicated symbols.

This is just a simple case, of course. But in general, changing the
meaning of `dmd -c source.d` will break existing build scripts.  Sure,
you could ask people to update their build scripts to include
`-no-auto-imports`, but that requires effort from users, who will be
unhappy that upgrading dmd broke their build scripts.  For large
projects, such a change may not be trivial as in the above example.


T

-- 
Life is too short to run proprietary software. -- Bdale Garbee
Aug 24 2017
parent Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 24 August 2017 at 18:12:03 UTC, H. S. Teoh wrote:

 On Thu, Aug 24, 2017 at 06:00:15PM +0000, Jonathan Marler via 
 Digitalmars-d wrote:

 On Thursday, 24 August 2017 at 17:49:27 UTC, H. S. Teoh wrote:

 Uh, no.  This will definitely break separate compilation, 
 and some people will be very unhappy about that.


 
 I couldn't think of a case that it would break.  Can you share 
 the cases you thought of?


 Suppose you have main.d and module.d, and you want to compile 
 them separately:

 	dmd -c main.d
 	dmd -c module.d
 	dmd -ofmyprogram main.o module.o

 If dmd defaulted to auto-importing, then `dmd -c main.d` would 
 also compile module.d (assuming main.d imports `module`), 
 contrary to what was intended in a separate compilation 
 scenario, and the last command will produce a linker error from 
 duplicated symbols.

 This is just a simple case, of course. But in general, changing 
 the meaning of `dmd -c source.d` will break existing build 
 scripts.  Sure, you could ask people to update their build 
 scripts to include `-no-auto-imports`, but that requires effort 
 from users, who will be unhappy that upgrading dmd broke their 
 build scripts.  For large projects, such a change may not be 
 trivial as in the above example.


 T


Actually this feature is mutually exclusive with the "-c" case.  
It doesn't make sense to compile imported modules unless you are 
also linking an executable.  So your example would work as 
expected.

Do you have any other cases you thought of that would not work? 
Like I said I couldn't think of any.  I'm not saying that that's 
enough reason to make it an "opt-out" feature, it's just 
something to think about.  The feature could also be an "opt-in" 
feature at first and eventually made "opt-out" if it makes sense. 
  But I'd still like to know people's thoughts/concerns either way.
Aug 24 2017
prev sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 24 August 2017 at 17:37:12 UTC, Jonathan Marler 
wrote:

 On Thursday, 24 August 2017 at 16:49:08 UTC, Seb wrote:

 On Thursday, 24 August 2017 at 16:32:32 UTC, Jonathan Marler 
 wrote:

 On Thursday, 24 August 2017 at 15:56:32 UTC, H. S. Teoh wrote:

 On Thu, Aug 24, 2017 at 03:53:05PM +0000, Jonathan Marler 
 via Digitalmars-d wrote:

 Wanted to get peoples thoughts on this.  The idea is to 
 have a way to tell the compiler (probably with a command 
 line option) that you'd like to "compile imported modules".


 [...]

 Isn't this what rdmd already does?


 T


 That is one thing that rdmd does (as I mentioned in the 
 original post).

 I just looked through the rdmd code 
 (https://github.com/dlang/tools/blob/master/rdmd.d) and it 
 looks like it invokes the compiler using "dmd -v" to get the 
 list of modules and then invokes the compiler again with the 
 modules it found to perform the full compile.  So my original 
 thought that the logic to find modules is duplicated was 
 incorrect.  Instead we just pay a performance hit to get the 
 correct list of imports since running "dmd -v" seems to take 
 almost as long as the actual compile itself.  So this method 
 comes close to doubling the time it takes to compile than if 
 the feature was implemented in the compiler itself.

 In any case, the idea is to allow the compiler to resolve 
 this on it's own without help from rdmd.  This would remove 
 the need to invoke the compiler twice, once to find the 
 imports and once to compile.  It would also allow some 
 projects/applications that don't use rdmd to take advantage 
 of this feature, this may or may not include dub (not sure on 
 that one).


 rdmd is really bad in terms of performance. If you call a 
 single D file with rdmd, it will always compile it twice. 
 There was an attempt to fix this 
 (https://github.com/dlang/tools/pull/194), but this has been 
 reverted as it introduced a regression and no one had time to 
 look at the regression.
 Moving rdmd into DMD has been on the TODO list for quite a 
 while and there is a consensus that the performance overhead 
 if rdmd isn't nice. However, IIRC there was no clear consensus 
 on how the integration should happen. I recall that the plan 
 was to do try this with "dmd as a library", but I'm not sure 
 whether that's really feasible ATM.


 Well this should solve the rdmd performance problem as well as 
 make other user cases easier that don't necessarilly use rdmd.

 I had another thought that instead of making this an "opt-in" 
 feature, it would probably make more sense to be an "opt-out" 
 feature.  So by default the compiler would compile missing 
 imported modules unless you indicate otherwise, maybe a command 
 line switch like "-dont-compile-imports".  And I don't see how 
 this would break anything.  Everything should work the same as 
 it did before, it's just now you can omit imported module files 
 from the command line and it should just work.


I've looked through the DMD code to see how this could be 
implemented and I've run into a problem.  The solution I came up 
with was to go through all the imported modules and then 
determine which ones need to be compiled that haven't been given 
on the command line. The problem is, I don't know how to 
determine whether a module was already compiled and given in an 
obj/lib file.  For example,

dmd something.obj anotherthing.lib prog.d

As far as I know, the compiler has no idea which modules are 
contained in "something.obj" and "anotherthing.lib".  It just 
compiles the source given on then command line, then passes all 
the object files and libraries to the linker, at which point the 
concept of modules is lost.

Am I correct in saying that the compiler has no idea which 
modules an obj/lib file contains?
Aug 24 2017
parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Friday, 25 August 2017 at 01:50:00 UTC, Jonathan Marler wrote:

 On Thursday, 24 August 2017 at 17:37:12 UTC, Jonathan Marler 
 wrote:

 On Thursday, 24 August 2017 at 16:49:08 UTC, Seb wrote:

 On Thursday, 24 August 2017 at 16:32:32 UTC, Jonathan Marler 
 wrote:

 [...]


 rdmd is really bad in terms of performance. If you call a 
 single D file with rdmd, it will always compile it twice. 
 There was an attempt to fix this 
 (https://github.com/dlang/tools/pull/194), but this has been 
 reverted as it introduced a regression and no one had time to 
 look at the regression.
 Moving rdmd into DMD has been on the TODO list for quite a 
 while and there is a consensus that the performance overhead 
 if rdmd isn't nice. However, IIRC there was no clear 
 consensus on how the integration should happen. I recall that 
 the plan was to do try this with "dmd as a library", but I'm 
 not sure whether that's really feasible ATM.


 Well this should solve the rdmd performance problem as well as 
 make other user cases easier that don't necessarilly use rdmd.

 I had another thought that instead of making this an "opt-in" 
 feature, it would probably make more sense to be an "opt-out" 
 feature.  So by default the compiler would compile missing 
 imported modules unless you indicate otherwise, maybe a 
 command line switch like "-dont-compile-imports".  And I don't 
 see how this would break anything.  Everything should work the 
 same as it did before, it's just now you can omit imported 
 module files from the command line and it should just work.


 I've looked through the DMD code to see how this could be 
 implemented and I've run into a problem.  The solution I came 
 up with was to go through all the imported modules and then 
 determine which ones need to be compiled that haven't been 
 given on the command line. The problem is, I don't know how to 
 determine whether a module was already compiled and given in an 
 obj/lib file.  For example,

 dmd something.obj anotherthing.lib prog.d

 As far as I know, the compiler has no idea which modules are 
 contained in "something.obj" and "anotherthing.lib".  It just 
 compiles the source given on then command line, then passes all 
 the object files and libraries to the linker, at which point 
 the concept of modules is lost.

 Am I correct in saying that the compiler has no idea which 
 modules an obj/lib file contains?


I created a prototype implementation here 
(https://github.com/dlang/dmd/pull/7099).

It uses the same logic that rdmd uses to determine if a module 
exists in a given object/library file.  Pretty cool that now I 
can compile any code without having to list the import module 
files!

Instead of:

dmd prog.d -Isomelib somelib\foo\module1.d somelib\foo\module2.d 
somelib\foo\module3.d somelib\foo\module4.d somelib\foo\module5.d 
somelib\foo\module6.d -Ianotherlib anotherlib\bar\module1.d 
anotherlib\bar\module2.d anotherlib\bar\module3.d 
anotherlib\bar\module4.d anotherlib\bar\module5.d

I can do:

dmd -ci prog.d -Isomelib -Ianotherlib
Aug 25 2017
parent reply Daniel N <no public.email> writes:
On Friday, 25 August 2017 at 13:03:11 UTC, Jonathan Marler wrote:

 I can do:

 dmd -ci prog.d -Isomelib -Ianotherlib


I love it, thanks for doing this!
Aug 25 2017
parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Friday, 25 August 2017 at 13:15:35 UTC, Daniel N wrote:

 On Friday, 25 August 2017 at 13:03:11 UTC, Jonathan Marler 
 wrote:

 I can do:

 dmd -ci prog.d -Isomelib -Ianotherlib


 I love it, thanks for doing this!


Thanks, I think this is a really nice feature.  I've uploaded my 
build so that people can try it out here 
(https://github.com/marler8997/dmd/releases/tag/preview-compileimports).

Just download and unzip dmd2.zip, then run the addtopath.bat 
script in any shell you want to use this compiler in.  Thanks in 
advance to anyone who gives it a try and shares their thoughts.
Aug 25 2017
parent reply user1234 <user1234 12.hu> writes:
On Friday, 25 August 2017 at 19:20:15 UTC, Jonathan Marler wrote:

 On Friday, 25 August 2017 at 13:15:35 UTC, Daniel N wrote:

 On Friday, 25 August 2017 at 13:03:11 UTC, Jonathan Marler 
 wrote:

 I can do:

 dmd -ci prog.d -Isomelib -Ianotherlib


 I love it, thanks for doing this!


 Thanks, I think this is a really nice feature.  I've uploaded 
 my build so that people can try it out here 
 (https://github.com/marler8997/dmd/releases/tag/preview-compileimports).

 Just download and unzip dmd2.zip, then run the addtopath.bat 
 script in any shell you want to use this compiler in.  Thanks 
 in advance to anyone who gives it a try and shares their 
 thoughts.


How does that mix with implicit imports (public imports located 
in an explicit import) ?
Aug 25 2017
parent Jonathan Marler <johnnymarler gmail.com> writes:
On Saturday, 26 August 2017 at 06:31:11 UTC, user1234 wrote:

 On Friday, 25 August 2017 at 19:20:15 UTC, Jonathan Marler 
 wrote:

 On Friday, 25 August 2017 at 13:15:35 UTC, Daniel N wrote:

 On Friday, 25 August 2017 at 13:03:11 UTC, Jonathan Marler 
 wrote:

 I can do:

 dmd -ci prog.d -Isomelib -Ianotherlib


 I love it, thanks for doing this!


 Thanks, I think this is a really nice feature.  I've uploaded 
 my build so that people can try it out here 
 (https://github.com/marler8997/dmd/releases/tag/preview-compileimports).

 Just download and unzip dmd2.zip, then run the addtopath.bat 
 script in any shell you want to use this compiler in.  Thanks 
 in advance to anyone who gives it a try and shares their 
 thoughts.


 How does that mix with implicit imports (public imports located 
 in an explicit import) ?


All imported modules need to be compiled whether they were 
imported "explicitly" or "implicitly".  So both kinds work the 
same when it comes to this feature.

P.S. There are some cases where you can "get away" with not 
compiling a module, if it only contains declarations and 
templates for example.  In this case it kind of behaves like a 
header file in C/C++.
Aug 26 2017
prev sibling parent zabruk70 <sorry noem.ail> writes:
On Thursday, 24 August 2017 at 15:56:32 UTC, H. S. Teoh wrote:

 Isn't this what rdmd already does?


This is my favorite rdmd feature.
But there is local import bug :(
https://issues.dlang.org/show_bug.cgi?id=15533
Aug 25 2017