www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - When compiling multiple source files

reply "ProgrammingGhost" <dsioafiseghvfawklncfskzdcf sdifjsdiovgfdisjcisj.com> writes:
How does the compiler do static typing of multiple source files?
I heard D malloc memory and doesn't free to speed up compilation
but I am guessing every instance doesn't compile just one source
file? My question is if I have a function in this file and
another in a different file what does the compiler do when both
files needs to know the definition of another? Also how does it
handle modules?

   From another thing I heard text parsing can be ridiculously fast
so there may be no need for a binary representation of each file
parsed. Does the D compiler read all source files into memory
generate the AST then starts compiling each file? I know there
more than one compiler but I wouldn't mind hearing from either or
both if they differ.
Aug 18 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-08-18 17:31, ProgrammingGhost wrote:
 How does the compiler do static typing of multiple source files?
 I heard D malloc memory and doesn't free to speed up compilation
 but I am guessing every instance doesn't compile just one source
 file? My question is if I have a function in this file and
 another in a different file what does the compiler do when both
 files needs to know the definition of another? Also how does it
 handle modules?

    From another thing I heard text parsing can be ridiculously fast
 so there may be no need for a binary representation of each file
 parsed. Does the D compiler read all source files into memory
 generate the AST then starts compiling each file? I know there
 more than one compiler but I wouldn't mind hearing from either or
 both if they differ.

The compiler will start compiling the files passed on the command line. It will read the files asynchronously and then lex, parse build an AST and do semantic analyze. When the semantic analyze is done it will have access to all import declarations. It basically starts the same processes for all these imports, recursively. The reason for waiting until semantic analyze is done because you can have code looking like this: mixin("import foo;"); The expansion of the mixin and other similar features are done in the semantic analyze phase. -- /Jacob Carlborg
Aug 19 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-08-20 00:27, ProgrammingGhost wrote:

 Is it possible that if I just try to compile 1 file it could imports
 enough libraries that import/need the definitions for additional large
 libraries which in turn also imports everything causing ram issues? I'm
 sure in practice this will almost never happen. But I don't doubt there
 are main libraries that use other large libraries and everything
 imports/uses everything

It's theoretically possible. But one big difference between D and C/C++, is that D uses symbolic inclusion where C/C++ uses textual inclusion. In C/C++ you end up with these enormous translation units due to this. This won't happen in D. In C/C++ when you see "include <stdio.h>", for example, the preprocessor will basically copy-paste the content of stdio.h to where the include was located. In D the compiler just makes a note that a given file includes another, no content is copied. -- /Jacob Carlborg
Aug 19 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-08-20 00:27, ProgrammingGhost wrote:

 Is it possible that if I just try to compile 1 file it could imports
 enough libraries that import/need the definitions for additional large
 libraries which in turn also imports everything causing ram issues? I'm
 sure in practice this will almost never happen. But I don't doubt there
 are main libraries that use other large libraries and everything
 imports/uses everything

I guess I should add that we do have some problems with RAM running the unit tests in Phobos (the standard library). But this is rather due to the heavy use of templates and other compile time features. Not because there is too much code/text in the files. -- /Jacob Carlborg
Aug 19 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-08-20 12:45, John Colvin wrote:

 Hah, ram problems running the unittests..... This old laptop can't even
 summon enough ram to compile phobos at all!

Haha, that's bad. How much RAM do you have? -- /Jacob Carlborg
Aug 20 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-08-20 13:21, John Colvin wrote:

 Only 2GB

 It can work... but I have to close everything else running first,
 otherwise it locks everything up and then crashes out complaining about
 not having enough memory to fork.

Hmm, I'm pretty sure I have compiled Phobos on my MacBook, which only has 2GB. -- /Jacob Carlborg
Aug 20 2013
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08/20/2013 08:50 AM, Jacob Carlborg wrote:
 On 2013-08-20 00:27, ProgrammingGhost wrote:

 Is it possible that if I just try to compile 1 file it could imports
 enough libraries that import/need the definitions for additional large
 libraries which in turn also imports everything causing ram issues? I'm
 sure in practice this will almost never happen. But I don't doubt there
 are main libraries that use other large libraries and everything
 imports/uses everything

I guess I should add that we do have some problems with RAM running the unit tests in Phobos (the standard library). But this is rather due to the heavy use of templates and other compile time features. Not because there is too much code/text in the files.

I was under the impression that this is a limitation of DMD and will be fixed, right?
Aug 20 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-08-20 15:35, Timon Gehr wrote:

 I was under the impression that this is a limitation of DMD and will be
 fixed, right?

I think they (Walter, Don, possibly others) are working on it. CTFE apparently does unnecessary copying as well. -- /Jacob Carlborg
Aug 20 2013
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 08/20/2013 12:27 AM, ProgrammingGhost wrote:
 Is it possible that if I just try to compile 1 file it could imports
 enough libraries that import/need the definitions for additional large
 libraries which in turn also imports everything causing ram issues? I'm
 sure in practice this will almost never happen. But I don't doubt there
 are main libraries that use other large libraries and everything
 imports/uses everything
 ...

The compiler only has to pull in what is indispensable for semantically analyzing an imported module as far as to determine all relevant type signatures. So while it should be possible to create a project as large that the complete annotated syntax tree exhausts your RAM (projects are not that large in practice), it still would require careful construction of dependencies in order to actually exhaust RAM during separate compilation. (And it is much easier to do so by other means, eg. by instantiating templates in a loop.)
Aug 20 2013
prev sibling next sibling parent "ProgrammingGhost" <dsioafiseghvfawklncfskzdcf sdifjsdiovgfdisjcisj.com> writes:
On Monday, 19 August 2013 at 11:01:54 UTC, Jacob Carlborg wrote:
 The compiler will start compiling the files passed on the 
 command line. It will read the files asynchronously and then 
 lex, parse build an AST and do semantic analyze.

 When the semantic analyze is done it will have access to all 
 import declarations. It basically starts the same processes for 
 all these imports, recursively.

 The reason for waiting until semantic analyze is done because 
 you can have code looking like this:

 mixin("import foo;");

 The expansion of the mixin and other similar features are done 
 in the semantic analyze phase.

So everything is parsed once and kept in memory until the compiler finish every source file? Is there any ram problems when compiling large codebases? My experience with D is limited. Are libraries the same as C libraries? From my understanding the linker figures that part out and the compiler needs a separate file for the definition. If I build a library in D is it the same as a C library or different which includes function definitions? Sorry if I'm confused I know almost nothing about D. I stick to .NET, java and C++
Aug 19 2013
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 19 August 2013 at 17:15:35 UTC, ProgrammingGhost wrote:
 On Monday, 19 August 2013 at 11:01:54 UTC, Jacob Carlborg wrote:
 The compiler will start compiling the files passed on the 
 command line. It will read the files asynchronously and then 
 lex, parse build an AST and do semantic analyze.

 When the semantic analyze is done it will have access to all 
 import declarations. It basically starts the same processes 
 for all these imports, recursively.

 The reason for waiting until semantic analyze is done because 
 you can have code looking like this:

 mixin("import foo;");

 The expansion of the mixin and other similar features are done 
 in the semantic analyze phase.

So everything is parsed once and kept in memory until the compiler finish every source file? Is there any ram problems when compiling large codebases?

Unfortunately, yes, if you give dmd a very large number of files all at once, it will chew through all your free RAM. But dmd does support separate compilation: $dmd file1.d -c $dmd file2.d -c $dmd file1.o file2.o which alleviates the problem.
 My experience with D is limited. Are libraries the same as C 
 libraries? From my understanding the linker figures that part 
 out and the compiler needs a separate file for the definition. 
 If I build a library in D is it the same as a C library or 
 different which includes function definitions?

 Sorry if I'm confused I know almost nothing about D. I stick to 
 .NET, java and C++

Libraries in D use the same formats as C/C++ libraries.
Aug 19 2013
prev sibling next sibling parent "ProgrammingGhost" <dsioafiseghvfawklncfskzdcf sdifjsdiovgfdisjcisj.com> writes:
On Monday, 19 August 2013 at 17:35:39 UTC, John Colvin wrote:
 On Monday, 19 August 2013 at 17:15:35 UTC, ProgrammingGhost 
 wrote:
 On Monday, 19 August 2013 at 11:01:54 UTC, Jacob Carlborg 
 wrote:
 The compiler will start compiling the files passed on the 
 command line. It will read the files asynchronously and then 
 lex, parse build an AST and do semantic analyze.

 When the semantic analyze is done it will have access to all 
 import declarations. It basically starts the same processes 
 for all these imports, recursively.

 The reason for waiting until semantic analyze is done because 
 you can have code looking like this:

 mixin("import foo;");

 The expansion of the mixin and other similar features are 
 done in the semantic analyze phase.

So everything is parsed once and kept in memory until the compiler finish every source file? Is there any ram problems when compiling large codebases?

Unfortunately, yes, if you give dmd a very large number of files all at once, it will chew through all your free RAM. But dmd does support separate compilation: $dmd file1.d -c $dmd file2.d -c $dmd file1.o file2.o which alleviates the problem.
 My experience with D is limited. Are libraries the same as C 
 libraries? From my understanding the linker figures that part 
 out and the compiler needs a separate file for the definition. 
 If I build a library in D is it the same as a C library or 
 different which includes function definitions?

 Sorry if I'm confused I know almost nothing about D. I stick 
 to .NET, java and C++

Libraries in D use the same formats as C/C++ libraries.

Is it possible that if I just try to compile 1 file it could imports enough libraries that import/need the definitions for additional large libraries which in turn also imports everything causing ram issues? I'm sure in practice this will almost never happen. But I don't doubt there are main libraries that use other large libraries and everything imports/uses everything
Aug 19 2013
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 20 August 2013 at 06:50:12 UTC, Jacob Carlborg wrote:
 On 2013-08-20 00:27, ProgrammingGhost wrote:

 Is it possible that if I just try to compile 1 file it could 
 imports
 enough libraries that import/need the definitions for 
 additional large
 libraries which in turn also imports everything causing ram 
 issues? I'm
 sure in practice this will almost never happen. But I don't 
 doubt there
 are main libraries that use other large libraries and 
 everything
 imports/uses everything

I guess I should add that we do have some problems with RAM running the unit tests in Phobos (the standard library). But this is rather due to the heavy use of templates and other compile time features. Not because there is too much code/text in the files.

Hah, ram problems running the unittests..... This old laptop can't even summon enough ram to compile phobos at all!
Aug 20 2013
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 20 August 2013 at 11:08:51 UTC, Jacob Carlborg wrote:
 On 2013-08-20 12:45, John Colvin wrote:

 Hah, ram problems running the unittests..... This old laptop 
 can't even
 summon enough ram to compile phobos at all!

Haha, that's bad. How much RAM do you have?

Only 2GB It can work... but I have to close everything else running first, otherwise it locks everything up and then crashes out complaining about not having enough memory to fork.
Aug 20 2013
prev sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 20 August 2013 at 13:30:11 UTC, Jacob Carlborg wrote:
 On 2013-08-20 13:21, John Colvin wrote:

 Only 2GB

 It can work... but I have to close everything else running 
 first,
 otherwise it locks everything up and then crashes out 
 complaining about
 not having enough memory to fork.

Hmm, I'm pretty sure I have compiled Phobos on my MacBook, which only has 2GB.

Just from looking at htop while compiling, it looks like it uses approx. 800MB to compile phobos. It's not unusual for me to have a lot less than that free at any given time. The interesting thing is that it crashes instead of just swapping...
Aug 20 2013