• ProgrammingGhost (13/13) Aug 18 2013 How does the compiler do static typing of multiple source files?
• Jacob Carlborg (14/27) Aug 19 2013 The compiler will start compiling the files passed on the command line.
• ProgrammingGhost (10/21) Aug 19 2013 So everything is parsed once and kept in memory until the
• John Colvin (9/35) Aug 19 2013 Unfortunately, yes, if you give dmd a very large number of files
• ProgrammingGhost (7/46) Aug 19 2013 Is it possible that if I just try to compile 1 file it could
• Jacob Carlborg (11/17) Aug 19 2013 It's theoretically possible. But one big difference between D and C/C++,...
• Jacob Carlborg (7/13) Aug 19 2013 I guess I should add that we do have some problems with RAM running the
• John Colvin (3/20) Aug 20 2013 Hah, ram problems running the unittests..... This old laptop
• Jacob Carlborg (4/6) Aug 20 2013 Haha, that's bad. How much RAM do you have?
• John Colvin (5/10) Aug 20 2013 Only 2GB
• Jacob Carlborg (5/9) Aug 20 2013 Hmm, I'm pretty sure I have compiled Phobos on my MacBook, which only
• John Colvin (6/16) Aug 20 2013 Just from looking at htop while compiling, it looks like it uses
• Timon Gehr (3/14) Aug 20 2013 I was under the impression that this is a limitation of DMD and will be
• Jacob Carlborg (5/7) Aug 20 2013 I think they (Walter, Don, possibly others) are working on it. CTFE
• Timon Gehr (9/16) Aug 20 2013 The compiler only has to pull in what is indispensable for semantically

"ProgrammingGhost" <dsioafiseghvfawklncfskzdcf sdifjsdiovgfdisjcisj.com> writes:

How does the compiler do static typing of multiple source files?
I heard D malloc memory and doesn't free to speed up compilation
but I am guessing every instance doesn't compile just one source
file? My question is if I have a function in this file and
another in a different file what does the compiler do when both
files needs to know the definition of another? Also how does it
handle modules?

   From another thing I heard text parsing can be ridiculously fast
so there may be no need for a binary representation of each file
parsed. Does the D compiler read all source files into memory
generate the AST then starts compiling each file? I know there
more than one compiler but I wouldn't mind hearing from either or
both if they differ.

Jacob Carlborg <doob me.com> writes:

On 2013-08-18 17:31, ProgrammingGhost wrote:
 How does the compiler do static typing of multiple source files?
 I heard D malloc memory and doesn't free to speed up compilation
 but I am guessing every instance doesn't compile just one source
 file? My question is if I have a function in this file and
 another in a different file what does the compiler do when both
 files needs to know the definition of another? Also how does it
 handle modules?

    From another thing I heard text parsing can be ridiculously fast
 so there may be no need for a binary representation of each file
 parsed. Does the D compiler read all source files into memory
 generate the AST then starts compiling each file? I know there
 more than one compiler but I wouldn't mind hearing from either or
 both if they differ.

The compiler will start compiling the files passed on the command line. 
It will read the files asynchronously and then lex, parse build an AST 
and do semantic analyze.

When the semantic analyze is done it will have access to all import 
declarations. It basically starts the same processes for all these 
imports, recursively.

The reason for waiting until semantic analyze is done because you can 
have code looking like this:

mixin("import foo;");

The expansion of the mixin and other similar features are done in the 
semantic analyze phase.

-- 
/Jacob Carlborg

"ProgrammingGhost" <dsioafiseghvfawklncfskzdcf sdifjsdiovgfdisjcisj.com> writes:

On Monday, 19 August 2013 at 11:01:54 UTC, Jacob Carlborg wrote:
 The compiler will start compiling the files passed on the 
 command line. It will read the files asynchronously and then 
 lex, parse build an AST and do semantic analyze.

 When the semantic analyze is done it will have access to all 
 import declarations. It basically starts the same processes for 
 all these imports, recursively.

 The reason for waiting until semantic analyze is done because 
 you can have code looking like this:

 mixin("import foo;");

 The expansion of the mixin and other similar features are done 
 in the semantic analyze phase.

So everything is parsed once and kept in memory until the 
compiler finish every source file? Is there any ram problems when 
compiling large codebases? My experience with D is limited. Are 
libraries the same as C libraries? From my understanding the 
linker figures that part out and the compiler needs a separate 
file for the definition. If I build a library in D is it the same 
as a C library or different which includes function definitions?

Sorry if I'm confused I know almost nothing about D. I stick to 
.NET, java and C++

"John Colvin" <john.loughran.colvin gmail.com> writes:

On Monday, 19 August 2013 at 17:15:35 UTC, ProgrammingGhost wrote:
 On Monday, 19 August 2013 at 11:01:54 UTC, Jacob Carlborg wrote:
 The compiler will start compiling the files passed on the 
 command line. It will read the files asynchronously and then 
 lex, parse build an AST and do semantic analyze.

 When the semantic analyze is done it will have access to all 
 import declarations. It basically starts the same processes 
 for all these imports, recursively.

 The reason for waiting until semantic analyze is done because 
 you can have code looking like this:

 mixin("import foo;");

 The expansion of the mixin and other similar features are done 
 in the semantic analyze phase.

 So everything is parsed once and kept in memory until the 
 compiler finish every source file? Is there any ram problems 
 when compiling large codebases?

Unfortunately, yes, if you give dmd a very large number of files 
all at once, it will chew through all your free RAM. But dmd does 
support separate compilation:

$dmd file1.d -c
$dmd file2.d -c
$dmd file1.o file2.o

which alleviates the problem.

 My experience with D is limited. Are libraries the same as C 
 libraries? From my understanding the linker figures that part 
 out and the compiler needs a separate file for the definition. 
 If I build a library in D is it the same as a C library or 
 different which includes function definitions?

 Sorry if I'm confused I know almost nothing about D. I stick to 
 .NET, java and C++

Libraries in D use the same formats as C/C++ libraries.

"ProgrammingGhost" <dsioafiseghvfawklncfskzdcf sdifjsdiovgfdisjcisj.com> writes:

On Monday, 19 August 2013 at 17:35:39 UTC, John Colvin wrote:
 On Monday, 19 August 2013 at 17:15:35 UTC, ProgrammingGhost 
 wrote:
 On Monday, 19 August 2013 at 11:01:54 UTC, Jacob Carlborg 
 wrote:
 The compiler will start compiling the files passed on the 
 command line. It will read the files asynchronously and then 
 lex, parse build an AST and do semantic analyze.

 When the semantic analyze is done it will have access to all 
 import declarations. It basically starts the same processes 
 for all these imports, recursively.

 The reason for waiting until semantic analyze is done because 
 you can have code looking like this:

 mixin("import foo;");

 The expansion of the mixin and other similar features are 
 done in the semantic analyze phase.

 So everything is parsed once and kept in memory until the 
 compiler finish every source file? Is there any ram problems 
 when compiling large codebases?

 Unfortunately, yes, if you give dmd a very large number of 
 files all at once, it will chew through all your free RAM. But 
 dmd does support separate compilation:

 $dmd file1.d -c
 $dmd file2.d -c
 $dmd file1.o file2.o

 which alleviates the problem.

 My experience with D is limited. Are libraries the same as C 
 libraries? From my understanding the linker figures that part 
 out and the compiler needs a separate file for the definition. 
 If I build a library in D is it the same as a C library or 
 different which includes function definitions?

 Sorry if I'm confused I know almost nothing about D. I stick 
 to .NET, java and C++

 Libraries in D use the same formats as C/C++ libraries.

Is it possible that if I just try to compile 1 file it could 
imports enough libraries that import/need the definitions for 
additional large libraries which in turn also imports everything 
causing ram issues? I'm sure in practice this will almost never 
happen. But I don't doubt there are main libraries that use other 
large libraries and everything imports/uses everything

Jacob Carlborg <doob me.com> writes:

On 2013-08-20 00:27, ProgrammingGhost wrote:

 Is it possible that if I just try to compile 1 file it could imports
 enough libraries that import/need the definitions for additional large
 libraries which in turn also imports everything causing ram issues? I'm
 sure in practice this will almost never happen. But I don't doubt there
 are main libraries that use other large libraries and everything
 imports/uses everything

It's theoretically possible. But one big difference between D and C/C++, 
is that D uses symbolic inclusion where C/C++ uses textual inclusion. In 
C/C++ you end up with these enormous translation units due to this. This 
won't happen in D.

In C/C++ when you see "include <stdio.h>", for example, the preprocessor 
will basically copy-paste the content of stdio.h to where the include 
was located. In D the compiler just makes a note that a given file 
includes another, no content is copied.

-- 
/Jacob Carlborg

Jacob Carlborg <doob me.com> writes:

On 2013-08-20 00:27, ProgrammingGhost wrote:

 Is it possible that if I just try to compile 1 file it could imports
 enough libraries that import/need the definitions for additional large
 libraries which in turn also imports everything causing ram issues? I'm
 sure in practice this will almost never happen. But I don't doubt there
 are main libraries that use other large libraries and everything
 imports/uses everything

I guess I should add that we do have some problems with RAM running the 
unit tests in Phobos (the standard library). But this is rather due to 
the heavy use of templates and other compile time features. Not because 
there is too much code/text in the files.

-- 
/Jacob Carlborg

"John Colvin" <john.loughran.colvin gmail.com> writes:

On Tuesday, 20 August 2013 at 06:50:12 UTC, Jacob Carlborg wrote:
 On 2013-08-20 00:27, ProgrammingGhost wrote:

 Is it possible that if I just try to compile 1 file it could 
 imports
 enough libraries that import/need the definitions for 
 additional large
 libraries which in turn also imports everything causing ram 
 issues? I'm
 sure in practice this will almost never happen. But I don't 
 doubt there
 are main libraries that use other large libraries and 
 everything
 imports/uses everything

 I guess I should add that we do have some problems with RAM 
 running the unit tests in Phobos (the standard library). But 
 this is rather due to the heavy use of templates and other 
 compile time features. Not because there is too much code/text 
 in the files.

Hah, ram problems running the unittests..... This old laptop 
can't even summon enough ram to compile phobos at all!

Jacob Carlborg <doob me.com> writes:

On 2013-08-20 12:45, John Colvin wrote:

 Hah, ram problems running the unittests..... This old laptop can't even
 summon enough ram to compile phobos at all!

Haha, that's bad. How much RAM do you have?

-- 
/Jacob Carlborg

"John Colvin" <john.loughran.colvin gmail.com> writes:

On Tuesday, 20 August 2013 at 11:08:51 UTC, Jacob Carlborg wrote:
 On 2013-08-20 12:45, John Colvin wrote:

 Hah, ram problems running the unittests..... This old laptop 
 can't even
 summon enough ram to compile phobos at all!

 Haha, that's bad. How much RAM do you have?

Only 2GB

It can work... but I have to close everything else running first, 
otherwise it locks everything up and then crashes out complaining 
about not having enough memory to fork.

Jacob Carlborg <doob me.com> writes:

On 2013-08-20 13:21, John Colvin wrote:

 Only 2GB

 It can work... but I have to close everything else running first,
 otherwise it locks everything up and then crashes out complaining about
 not having enough memory to fork.

Hmm, I'm pretty sure I have compiled Phobos on my MacBook, which only 
has 2GB.

-- 
/Jacob Carlborg

"John Colvin" <john.loughran.colvin gmail.com> writes:

On Tuesday, 20 August 2013 at 13:30:11 UTC, Jacob Carlborg wrote:
 On 2013-08-20 13:21, John Colvin wrote:

 Only 2GB

 It can work... but I have to close everything else running 
 first,
 otherwise it locks everything up and then crashes out 
 complaining about
 not having enough memory to fork.

 Hmm, I'm pretty sure I have compiled Phobos on my MacBook, 
 which only has 2GB.

Just from looking at htop while compiling, it looks like it uses 
approx. 800MB to compile phobos. It's not unusual for me to have 
a lot less than that free at any given time.

The interesting thing is that it crashes instead of just 
swapping...

Timon Gehr <timon.gehr gmx.ch> writes:

On 08/20/2013 08:50 AM, Jacob Carlborg wrote:
 On 2013-08-20 00:27, ProgrammingGhost wrote:

 Is it possible that if I just try to compile 1 file it could imports
 enough libraries that import/need the definitions for additional large
 libraries which in turn also imports everything causing ram issues? I'm
 sure in practice this will almost never happen. But I don't doubt there
 are main libraries that use other large libraries and everything
 imports/uses everything

 I guess I should add that we do have some problems with RAM running the
 unit tests in Phobos (the standard library). But this is rather due to
 the heavy use of templates and other compile time features. Not because
 there is too much code/text in the files.

I was under the impression that this is a limitation of DMD and will be 
fixed, right?

Jacob Carlborg <doob me.com> writes:

On 2013-08-20 15:35, Timon Gehr wrote:

 I was under the impression that this is a limitation of DMD and will be
 fixed, right?

I think they (Walter, Don, possibly others) are working on it. CTFE 
apparently does unnecessary copying as well.

-- 
/Jacob Carlborg

Timon Gehr <timon.gehr gmx.ch> writes:

On 08/20/2013 12:27 AM, ProgrammingGhost wrote:
 Is it possible that if I just try to compile 1 file it could imports
 enough libraries that import/need the definitions for additional large
 libraries which in turn also imports everything causing ram issues? I'm
 sure in practice this will almost never happen. But I don't doubt there
 are main libraries that use other large libraries and everything
 imports/uses everything
 ...

The compiler only has to pull in what is indispensable for semantically 
analyzing an imported module as far as to determine all relevant type 
signatures. So while it should be possible to create a project as large 
that the complete annotated syntax tree exhausts your RAM (projects are 
not that large in practice), it still would require careful construction 
of dependencies in order to actually exhaust RAM during separate 
compilation. (And it is much easier to do so by other means, eg. by 
instantiating templates in a loop.)