www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Visual D Build + DMD Bugginess = Bad

reply dsimcha <dsimcha yahoo.com> writes:
I've noticed that, when passed multiple files at once, DMD is generally buggy
in ways that I can't reproduce in small test cases.  This includes things like
magically ignoring __gshared variables, and not being able to convert a type
to an alias to the same type, for example, not being able to convert a size_t
to a uint on 32-bit, or not being able to convert a float[] to an R where R is
a template parameter instantiated as float[].

I have two questions:

1.  Has anyone experienced similar things and if so are they further along
than me at creating decent test cases?

2.  I'm used to using Code::Blocks, but want to switch to Visual D because it
seems to work a lot better for the most part.  However, by default it seems to
pass all project files to the compiler at once, where Code::Blocks compiles
one file at a time.  This means I have to deal with DMD's bugginess when
passed mutliple files at once.  Does anyone know how to make Visual D work
Code::Blocks style, i.e. compile each file to an object file and link them
afterwords?
Oct 13 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"dsimcha" <dsimcha yahoo.com> wrote in message 
news:i9529m$2d4b$1 digitalmars.com...
 I've noticed that, when passed multiple files at once, DMD is generally 
 buggy
 in ways that I can't reproduce in small test cases.  This includes things 
 like
 magically ignoring __gshared variables, and not being able to convert a 
 type
 to an alias to the same type, for example, not being able to convert a 
 size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where 
 R is
 a template parameter instantiated as float[].

 I have two questions:

 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?

 2.  I'm used to using Code::Blocks, but want to switch to Visual D because 
 it
 seems to work a lot better for the most part.  However, by default it 
 seems to
 pass all project files to the compiler at once, where Code::Blocks 
 compiles
 one file at a time.  This means I have to deal with DMD's bugginess when
 passed mutliple files at once.  Does anyone know how to make Visual D work
 Code::Blocks style, i.e. compile each file to an object file and link them
 afterwords?

Don't they support setting up custom tools? I always do that and just define my own command line calls.
Oct 13 2010
prev sibling next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
dsimcha wrote:
 I've noticed that, when passed multiple files at once, DMD is generally buggy
 in ways that I can't reproduce in small test cases.  This includes things like
 magically ignoring __gshared variables, and not being able to convert a type
 to an alias to the same type, for example, not being able to convert a size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where R is
 a template parameter instantiated as float[].
 
 I have two questions:
 
 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?

I haven't run into something similar lately, but what you describe sounds like issues with forward references, maybe in combination with circular imports. Building multiple files at once can change the point where you enter the vicious import circle. Could it be related to any of these? http://d.puremagic.com/issues/show_bug.cgi?id=190 http://d.puremagic.com/issues/show_bug.cgi?id=3979
 
 2.  I'm used to using Code::Blocks, but want to switch to Visual D because it
 seems to work a lot better for the most part.  However, by default it seems to
 pass all project files to the compiler at once, where Code::Blocks compiles
 one file at a time.  This means I have to deal with DMD's bugginess when
 passed mutliple files at once.  Does anyone know how to make Visual D work
 Code::Blocks style, i.e. compile each file to an object file and link them
 afterwords?

I have that option on my todo list, but didn't implement it so far because I was not aware of any problems with compiling multiple files. I don't know if this is really an option for a larger project, but what you can do is select "Custom Build Tool" for all files in the "Common Properties" setting of the project property dialog and enter your own command line and output file. But be warned: you will not get any automatic dependency detection. As editing the settings for a lot of files might be very annoying, you might want to set it for one file and then edit the project file with a text editor to copy the settings to other files. Rainer
Oct 13 2010
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Rainer Schuetze (r.sagitario gmx.de)'s article
 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979

Yep, these look like at least part of the problem. Glad someone else has already reduced them to sane test cases so I don't have to. Reducing compiler bugs that only seem to occur in non-trivial, multi-module projects to decent test cases is a huge PITA. Anyhow, the project in question is a very messy codebase because it grew very organically. It's basically a haphazard collection of research prototype algorithms for predicting gene expression from DNA sequence, and every time I think of a new idea, I tend to just put it wherever I can make it fit and almost never bother with non-trivial refactoring. Therefore, there are cyclic imports **everywhere**.
 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer

Since build process automation is by far the biggest reason why I use an IDE instead of a plain old editor, I'd rather just stick with Code::Blocks for now. Could you please bump this up the todo list, given that building multiple files simultaneously is buggy in ways that probably aren't going to get fixed too soon?
Oct 13 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"dsimcha" <dsimcha yahoo.com> wrote in message 
news:i957sg$2otm$1 digitalmars.com...
 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article
 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer

Since build process automation is by far the biggest reason why I use an IDE instead of a plain old editor, I'd rather just stick with Code::Blocks for now. Could you please bump this up the todo list, given that building multiple files simultaneously is buggy in ways that probably aren't going to get fixed too soon?

I find that rdmd works just as well as any IDE (if not better) for automating the build process of D programs. Particularly if you apply the "Combined" patch here: http://d.puremagic.com/issues/show_bug.cgi?id=4930 Although it doesn't do one-at-a-time building ATM, so I guess that wouldn't help in your case.
Oct 13 2010
prev sibling next sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
I'll push the issue up the todo list. Maybe I'm not getting a lot of 
these errors because I have applied the dmd patches in the bug reports 
190 and 4753...


dsimcha wrote:
 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article
 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979

Yep, these look like at least part of the problem. Glad someone else has already reduced them to sane test cases so I don't have to. Reducing compiler bugs that only seem to occur in non-trivial, multi-module projects to decent test cases is a huge PITA. Anyhow, the project in question is a very messy codebase because it grew very organically. It's basically a haphazard collection of research prototype algorithms for predicting gene expression from DNA sequence, and every time I think of a new idea, I tend to just put it wherever I can make it fit and almost never bother with non-trivial refactoring. Therefore, there are cyclic imports **everywhere**.
 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer

Since build process automation is by far the biggest reason why I use an IDE instead of a plain old editor, I'd rather just stick with Code::Blocks for now. Could you please bump this up the todo list, given that building multiple files simultaneously is buggy in ways that probably aren't going to get fixed too soon?

Oct 14 2010
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Here's a preliminary new version that includes an option on the general 
project configuration page to switch to single file compilation:

http://www.dsource.org/projects/visuald/browser/downloads/VisualD-v0.3.18rc1.exe

I've used ddmd as a medium sized project for testing. It has more than 
400 files with most modules mutually importing each other. Some remarks:

- compilation speed is a lot worse, even when compared to what you would 
expect from a similar C/C++ project. It takes almost 8 minutes to 
compile ddmd on my system with single file compilation, while the 
standard compilation takes about 10 seconds.

- the dependency files written by dmd are so bloated that the check that 
selects the files to compile takes almost 20 seconds. This is because 
the size of each dependency file is about 700 kB, summing up to 220 MB 
to scan for imported files. (I had to do some optimizations and file 
operation caching to get the time down from several minutes).

- I consider the single compilation mode only sensible for a large 
collection of modules that have only limited dependencies.

- probably, compilation speed could be way better if semantic analysis 
would only be done on symbols that are actually directly or indirectly 
referred to by the modules passed on the command line. This would need a 
slight modification of the semantic analysis, but would also put the 
promise of independence of lexical order to the test - and would expose 
more of the problems you are currently getting.

This new version also includes some experimental debug/version statement 
highlighting, which has not yet been extensively tested. If it causes 
too many troubles, it can be disabled on the global property page 
Tools->Options->TextEditor->D->Colorizer.

Rainer


dsimcha wrote:
 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article
 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979

Yep, these look like at least part of the problem. Glad someone else has already reduced them to sane test cases so I don't have to. Reducing compiler bugs that only seem to occur in non-trivial, multi-module projects to decent test cases is a huge PITA. Anyhow, the project in question is a very messy codebase because it grew very organically. It's basically a haphazard collection of research prototype algorithms for predicting gene expression from DNA sequence, and every time I think of a new idea, I tend to just put it wherever I can make it fit and almost never bother with non-trivial refactoring. Therefore, there are cyclic imports **everywhere**.
 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer

Since build process automation is by far the biggest reason why I use an IDE instead of a plain old editor, I'd rather just stick with Code::Blocks for now. Could you please bump this up the todo list, given that building multiple files simultaneously is buggy in ways that probably aren't going to get fixed too soon?

Oct 17 2010
next sibling parent dsimcha <dsimcha yahoo.com> writes:
Awesome.  I frankly don't care about compile speeds because the projects in
question are fairly small, so even with a horribly inefficient build process
build
times are negligible.  FWIW I use the one at a time builds in CodeBlocks, too,
though I parallelize them on my quad core.

I only wanted the one at a time compilation feature as a temporary workaround
for
compiler bugs that will hopefully be fixed by the time people start using D2 for
larger projects.  I'll try Visual D again tomorrow and provide some feedback.
From my limited playing around with it, I've already noticed that the debugger
works a ton better than any other D debugger I've tried, i.e. possibly good
enough
that I'll stop using writeln as my main debugging method.

== Quote from Rainer Schuetze (r.sagitario gmx.de)'s article
 Here's a preliminary new version that includes an option on the general
 project configuration page to switch to single file compilation:
 http://www.dsource.org/projects/visuald/browser/downloads/VisualD-v0.3.18rc1.exe
 I've used ddmd as a medium sized project for testing. It has more than
 400 files with most modules mutually importing each other. Some remarks:
 - compilation speed is a lot worse, even when compared to what you would
 expect from a similar C/C++ project. It takes almost 8 minutes to
 compile ddmd on my system with single file compilation, while the
 standard compilation takes about 10 seconds.
 - the dependency files written by dmd are so bloated that the check that
 selects the files to compile takes almost 20 seconds. This is because
 the size of each dependency file is about 700 kB, summing up to 220 MB
 to scan for imported files. (I had to do some optimizations and file
 operation caching to get the time down from several minutes).
 - I consider the single compilation mode only sensible for a large
 collection of modules that have only limited dependencies.
 - probably, compilation speed could be way better if semantic analysis
 would only be done on symbols that are actually directly or indirectly
 referred to by the modules passed on the command line. This would need a
 slight modification of the semantic analysis, but would also put the
 promise of independence of lexical order to the test - and would expose
 more of the problems you are currently getting.
 This new version also includes some experimental debug/version statement
 highlighting, which has not yet been extensively tested. If it causes
 too many troubles, it can be disabled on the global property page
 Tools->Options->TextEditor->D->Colorizer.
 Rainer
 dsimcha wrote:
 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article
 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979

Yep, these look like at least part of the problem. Glad someone else has already reduced them to sane test cases so I don't have to. Reducing compiler bugs that only seem to occur in non-trivial, multi-module projects to decent test cases is a huge PITA. Anyhow, the project in question is a very messy codebase because it grew very organically. It's basically a haphazard collection of research prototype algorithms for predicting gene expression from DNA sequence, and every time I think of a new idea, I tend to just put it wherever I can make it fit and almost never bother with non-trivial refactoring. Therefore, there are cyclic imports **everywhere**.
 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer

Since build process automation is by far the biggest reason why I use an IDE instead of a plain old editor, I'd rather just stick with Code::Blocks for now. Could you please bump this up the todo list, given that building multiple files simultaneously is buggy in ways that probably aren't going to get fixed too soon?


Oct 17 2010
prev sibling next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Andrej Mitrovic wrote:
 Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
 with newer versions of VisualD (mago?). Do I need to set it up to be
 used in VS over the default one or is it selected by default after
 installation?
 

If you did not uncheck the mago debugger when installing, you can simply select the debug engine to be used on the project property page "Debugging" (visual studio internal debugger or mago).
Oct 17 2010
parent Rainer Schuetze <r.sagitario gmx.de> writes:
That's an unfortunate regression in the linker that comes with dmd 
2.049. There's already an updated linker available 
(http://ftp.digitalmars.com/link.8.00.8.zip). See also the project page 
http://dsource.org/projects/mago_debugger for more info.

Andrej Mitrovic wrote:
 I've just tried it. It doesn't seem to be able to find any locals,
 even if I add a variable to a watchlist it can't find the symbol. I
 guess this is still experimental. Anyway, thanks for the info.
 
 On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:
 Andrej Mitrovic wrote:
 Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
 with newer versions of VisualD (mago?). Do I need to set it up to be
 used in VS over the default one or is it selected by default after
 installation?

select the debug engine to be used on the project property page "Debugging" (visual studio internal debugger or mago).


Oct 17 2010
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Rainer Schuetze" <r.sagitario gmx.de> wrote in message 
news:i9f2ce$30re$1 digitalmars.com...
 I've used ddmd as a medium sized project for testing. It has more than 400 
 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would 
 expect from a similar C/C++ project. It takes almost 8 minutes to compile 
 ddmd on my system with single file compilation, while the standard 
 compilation takes about 10 seconds.

Whaa...? Standard compilation of ddmd takes me a full two minutes.
Oct 18 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:i9ibi0$ddn$1 digitalmars.com...
 "Rainer Schuetze" <r.sagitario gmx.de> wrote in message 
 news:i9f2ce$30re$1 digitalmars.com...
 I've used ddmd as a medium sized project for testing. It has more than 
 400 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would 
 expect from a similar C/C++ project. It takes almost 8 minutes to compile 
 ddmd on my system with single file compilation, while the standard 
 compilation takes about 10 seconds.

Whaa...? Standard compilation of ddmd takes me a full two minutes.

As a sidebar, it's the *only* D program I've compiled that's ever taken more than a few seconds for me.
Oct 18 2010
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Denis Koroskin" <2korden gmail.com> wrote in message 
news:op.vkshh4bdo7cclz korden-pc...
 On Tue, 19 Oct 2010 00:41:54 +0400, Nick Sabalausky <a a.a> wrote:

 "Rainer Schuetze" <r.sagitario gmx.de> wrote in message
 news:i9f2ce$30re$1 digitalmars.com...
 I've used ddmd as a medium sized project for testing. It has more than 
 400
 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would
 expect from a similar C/C++ project. It takes almost 8 minutes to 
 compile
 ddmd on my system with single file compilation, while the standard
 compilation takes about 10 seconds.

Whaa...? Standard compilation of ddmd takes me a full two minutes.

Do you compile the code on 486 or something? :)

Heh :) Compared to your and Rainer's systems it probably might as well be.
 Running the build script  takes 7.967 seconds for me (timed with timeit, 
 part of the Windows Server  2003 Resource Kit Tools).

My guess is it may be the multi-threaded "process multiple modules in parallel" optimization. DDMD does have a *lot* of source files. And I'm on a single-core (yea, yea, I know, but it works for me and I have no money). I'm curious, is it possible for you or Rainer to time it running dmd in a "force single-core" mode? Ideally using the main system/OS core (if there even is one)? I would expect a normal out-of-the-box dmd would be noticably faster for you two just because of that multi-thread/core optimization and the cores themselves just simply being faster, but 120 sec vs 8 sec still seems an excessive difference. And then considering also, outside of dmd bugs, I've never once had any D program besides ddmd take more than a few seconds - despite the fact that much of what I do complie is fairly template/mixin/ctfe-heavy, and ddmd appears to use very little (if any) of that - it's certainly still a conceivable speed difference, but it does make me wonder if something else might be going on. BTW, anyone know of an easy way to check how much time in a process is spent waiting on disk IO? It's entirely possible that my disk is just fragmented all to hell.
Oct 18 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Denis Koroskin" <2korden gmail.com> wrote in message 
news:op.vksxyn15o7cclz korden-pc...
 On Tue, 19 Oct 2010 06:26:23 +0400, Nick Sabalausky <a a.a> wrote:

 "Denis Koroskin" <2korden gmail.com> wrote in message
 Running the build script  takes 7.967 seconds for me (timed with timeit,
 part of the Windows Server  2003 Resource Kit Tools).

My guess is it may be the multi-threaded "process multiple modules in parallel" optimization. DDMD does have a *lot* of source files. And I'm on a single-core (yea, yea, I know, but it works for me and I have no money). I'm curious, is it possible for you or Rainer to time it running dmd in a "force single-core" mode? Ideally using the main system/OS core (if there even is one)? I would expect a normal out-of-the-box dmd would be noticably faster for you two just because of that multi-thread/core optimization and the cores themselves just simply being faster, but 120 sec vs 8 sec still seems an excessive difference. And then considering also, outside of dmd bugs, I've never once had any D program besides ddmd take more than a few seconds - despite the fact that much of what I do complie is fairly template/mixin/ctfe-heavy, and ddmd appears to use very little (if any) of that - it's certainly still a conceivable speed difference, but it does make me wonder if something else might be going on. BTW, anyone know of an easy way to check how much time in a process is spent waiting on disk IO? It's entirely possible that my disk is just fragmented all to hell.

dmd doesn't use multiple core for anything BUT asynchronous reading from disk. Everything else is done is a single thread.

Ahh, I see. So much for that theory then.
 I don't think it has  THAT big of impact, but I'll try to recompile with 
 all the cores disabled.

Thanks, that'll be interesting. I did go ahead and re-time the compile. Apparently I must have been remembering it wrong, because this time it only took 1 min 20 sec (and this was with a ton of stuff running - bunch of misc apps, FF2 with 20 tabs, an HDD SMART monitor, a torrent manager and a bunch of other servers (but no clients connected)). But that's still quite a lot of time to compile a D app, even for my machine. I also tried grabbing the latest ddmd, rebooted, killed all non-essential processes, and tried that way. Got it down to just slightly under one minute. I thought about maybe it being a limited-memory issue (remembering that dmd never frees anything until it's done - or is that just CTFE?), but I don't think that's it - the highest memory usage it ever got was about 200MB, and I have 1GB, and it still took a whole minute with almost everything besides XP shut down, so I'm not sure that was it. (I could have sworn I had 2GB at one point, but I think I probably cannibalized one of the sticks when I built my linux box - not that that's really relevant ;) ) FWIW, this is all with compiling the just the debug version of ddmd only. Ie, not including building the release version or the one-time initial setup of building dmd.lib.
Oct 18 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Denis Koroskin" <2korden gmail.com> wrote in message 
news:op.vks9nlljo7cclz korden-pc...
 On Tue, 19 Oct 2010 07:53:45 +0400, Nick Sabalausky <a a.a> wrote:

 "Denis Koroskin" <2korden gmail.com> wrote in message
 news:op.vksxyn15o7cclz korden-pc...
 I don't think it has  THAT big of impact, but I'll try to recompile with
 all the cores disabled.

Thanks, that'll be interesting. I did go ahead and re-time the compile. Apparently I must have been remembering it wrong, because this time it only took 1 min 20 sec (and this was with a ton of stuff running - bunch of misc apps, FF2 with 20 tabs, an HDD SMART monitor, a torrent manager and a bunch of other servers (but no clients connected)). But that's still quite a lot of time to compile a D app, even for my machine. I also tried grabbing the latest ddmd, rebooted, killed all non-essential processes, and tried that way. Got it down to just slightly under one minute. I thought about maybe it being a limited-memory issue (remembering that dmd never frees anything until it's done - or is that just CTFE?), but I don't think that's it - the highest memory usage it ever got was about 200MB, and I have 1GB, and it still took a whole minute with almost everything besides XP shut down, so I'm not sure that was it. (I could have sworn I had 2GB at one point, but I think I probably cannibalized one of the sticks when I built my linux box - not that that's really relevant ;) ) FWIW, this is all with compiling the just the debug version of ddmd only. Ie, not including building the release version or the one-time initial setup of building dmd.lib.

It's 8.6 seconds for a single cores, 8.3s for all 4 cores (Core2 Quad Q8300 2.5Ghz, Windows).

Hmm... that does make my situation seem odd then. Mine's a 1.7 GHz Celeron, so a little more than half the clock speed of yours. Of course, I'm well aware that my older architecture and less cache make mine slower than (2.5/1.7) of your speed, but that still seems like a strangely large difference. I wonder if maybe RAM speed could account for it.
Oct 19 2010
parent "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:i9jr29$6af$1 digitalmars.com...
 "Denis Koroskin" <2korden gmail.com> wrote in message 
 news:op.vks9nlljo7cclz korden-pc...
 On Tue, 19 Oct 2010 07:53:45 +0400, Nick Sabalausky <a a.a> wrote:

 "Denis Koroskin" <2korden gmail.com> wrote in message
 news:op.vksxyn15o7cclz korden-pc...
 I don't think it has  THAT big of impact, but I'll try to recompile 
 with
 all the cores disabled.

Thanks, that'll be interesting. I did go ahead and re-time the compile. Apparently I must have been remembering it wrong, because this time it only took 1 min 20 sec (and this was with a ton of stuff running - bunch of misc apps, FF2 with 20 tabs, an HDD SMART monitor, a torrent manager and a bunch of other servers (but no clients connected)). But that's still quite a lot of time to compile a D app, even for my machine. I also tried grabbing the latest ddmd, rebooted, killed all non-essential processes, and tried that way. Got it down to just slightly under one minute. I thought about maybe it being a limited-memory issue (remembering that dmd never frees anything until it's done - or is that just CTFE?), but I don't think that's it - the highest memory usage it ever got was about 200MB, and I have 1GB, and it still took a whole minute with almost everything besides XP shut down, so I'm not sure that was it. (I could have sworn I had 2GB at one point, but I think I probably cannibalized one of the sticks when I built my linux box - not that that's really relevant ;) ) FWIW, this is all with compiling the just the debug version of ddmd only. Ie, not including building the release version or the one-time initial setup of building dmd.lib.

It's 8.6 seconds for a single cores, 8.3s for all 4 cores (Core2 Quad Q8300 2.5Ghz, Windows).

Hmm... that does make my situation seem odd then. Mine's a 1.7 GHz Celeron, so a little more than half the clock speed of yours. Of course, I'm well aware that my older architecture and less cache make mine slower than (2.5/1.7) of your speed, but that still seems like a strangely large difference. I wonder if maybe RAM speed could account for it.

s/(2.5/1.7)/(1.7/2.5) Too late in the day for math...
Oct 19 2010
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
with newer versions of VisualD (mago?). Do I need to set it up to be
used in VS over the default one or is it selected by default after
installation?

On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:
 Here's a preliminary new version that includes an option on the general
 project configuration page to switch to single file compilation:

 http://www.dsource.org/projects/visuald/browser/downloads/VisualD-v0.3.18rc1.exe

 I've used ddmd as a medium sized project for testing. It has more than
 400 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would
 expect from a similar C/C++ project. It takes almost 8 minutes to
 compile ddmd on my system with single file compilation, while the
 standard compilation takes about 10 seconds.

 - the dependency files written by dmd are so bloated that the check that
 selects the files to compile takes almost 20 seconds. This is because
 the size of each dependency file is about 700 kB, summing up to 220 MB
 to scan for imported files. (I had to do some optimizations and file
 operation caching to get the time down from several minutes).

 - I consider the single compilation mode only sensible for a large
 collection of modules that have only limited dependencies.

 - probably, compilation speed could be way better if semantic analysis
 would only be done on symbols that are actually directly or indirectly
 referred to by the modules passed on the command line. This would need a
 slight modification of the semantic analysis, but would also put the
 promise of independence of lexical order to the test - and would expose
 more of the problems you are currently getting.

 This new version also includes some experimental debug/version statement
 highlighting, which has not yet been extensively tested. If it causes
 too many troubles, it can be disabled on the global property page
 Tools->Options->TextEditor->D->Colorizer.

 Rainer


 dsimcha wrote:
 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article
 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979

Yep, these look like at least part of the problem. Glad someone else has already reduced them to sane test cases so I don't have to. Reducing compiler bugs that only seem to occur in non-trivial, multi-module projects to decent test cases is a huge PITA. Anyhow, the project in question is a very messy codebase because it grew very organically. It's basically a haphazard collection of research prototype algorithms for predicting gene expression from DNA sequence, and every time I think of a new idea, I tend to just put it wherever I can make it fit and almost never bother with non-trivial refactoring. Therefore, there are cyclic imports **everywhere**.
 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer

Since build process automation is by far the biggest reason why I use an IDE instead of a plain old editor, I'd rather just stick with Code::Blocks for now. Could you please bump this up the todo list, given that building multiple files simultaneously is buggy in ways that probably aren't going to get fixed too soon?


Oct 17 2010
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I've just tried it. It doesn't seem to be able to find any locals,
even if I add a variable to a watchlist it can't find the symbol. I
guess this is still experimental. Anyway, thanks for the info.

On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:
 Andrej Mitrovic wrote:
 Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
 with newer versions of VisualD (mago?). Do I need to set it up to be
 used in VS over the default one or is it selected by default after
 installation?

If you did not uncheck the mago debugger when installing, you can simply select the debug engine to be used on the project property page "Debugging" (visual studio internal debugger or mago).

Oct 17 2010
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Ah, ok. I'll have a look, thanks.

On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:
 That's an unfortunate regression in the linker that comes with dmd
 2.049. There's already an updated linker available
 (http://ftp.digitalmars.com/link.8.00.8.zip). See also the project page
 http://dsource.org/projects/mago_debugger for more info.

 Andrej Mitrovic wrote:
 I've just tried it. It doesn't seem to be able to find any locals,
 even if I add a variable to a watchlist it can't find the symbol. I
 guess this is still experimental. Anyway, thanks for the info.

 On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:
 Andrej Mitrovic wrote:
 Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
 with newer versions of VisualD (mago?). Do I need to set it up to be
 used in VS over the default one or is it selected by default after
 installation?

select the debug engine to be used on the project property page "Debugging" (visual studio internal debugger or mago).



Oct 17 2010
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 19 Oct 2010 00:41:54 +0400, Nick Sabalausky <a a.a> wrote:

 "Rainer Schuetze" <r.sagitario gmx.de> wrote in message
 news:i9f2ce$30re$1 digitalmars.com...
 I've used ddmd as a medium sized project for testing. It has more than  
 400
 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would
 expect from a similar C/C++ project. It takes almost 8 minutes to  
 compile
 ddmd on my system with single file compilation, while the standard
 compilation takes about 10 seconds.

Whaa...? Standard compilation of ddmd takes me a full two minutes.

Do you compile the code on 486 or something? :) Running the build script takes 7.967 seconds for me (timed with timeit, part of the Windows Server 2003 Resource Kit Tools).
Oct 18 2010
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 19 Oct 2010 06:26:23 +0400, Nick Sabalausky <a a.a> wrote:

 "Denis Koroskin" <2korden gmail.com> wrote in message
 news:op.vkshh4bdo7cclz korden-pc...
 On Tue, 19 Oct 2010 00:41:54 +0400, Nick Sabalausky <a a.a> wrote:

 "Rainer Schuetze" <r.sagitario gmx.de> wrote in message
 news:i9f2ce$30re$1 digitalmars.com...
 I've used ddmd as a medium sized project for testing. It has more than
 400
 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you  
 would
 expect from a similar C/C++ project. It takes almost 8 minutes to
 compile
 ddmd on my system with single file compilation, while the standard
 compilation takes about 10 seconds.

Whaa...? Standard compilation of ddmd takes me a full two minutes.

Do you compile the code on 486 or something? :)

Heh :) Compared to your and Rainer's systems it probably might as well be.
 Running the build script  takes 7.967 seconds for me (timed with timeit,
 part of the Windows Server  2003 Resource Kit Tools).

My guess is it may be the multi-threaded "process multiple modules in parallel" optimization. DDMD does have a *lot* of source files. And I'm on a single-core (yea, yea, I know, but it works for me and I have no money). I'm curious, is it possible for you or Rainer to time it running dmd in a "force single-core" mode? Ideally using the main system/OS core (if there even is one)? I would expect a normal out-of-the-box dmd would be noticably faster for you two just because of that multi-thread/core optimization and the cores themselves just simply being faster, but 120 sec vs 8 sec still seems an excessive difference. And then considering also, outside of dmd bugs, I've never once had any D program besides ddmd take more than a few seconds - despite the fact that much of what I do complie is fairly template/mixin/ctfe-heavy, and ddmd appears to use very little (if any) of that - it's certainly still a conceivable speed difference, but it does make me wonder if something else might be going on. BTW, anyone know of an easy way to check how much time in a process is spent waiting on disk IO? It's entirely possible that my disk is just fragmented all to hell.

dmd doesn't use multiple core for anything BUT asynchronous reading from disk. Everything else is done is a single thread. I don't think it has THAT big of impact, but I'll try to recompile with all the cores disabled.
Oct 18 2010
prev sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 19 Oct 2010 07:53:45 +0400, Nick Sabalausky <a a.a> wrote:

 "Denis Koroskin" <2korden gmail.com> wrote in message
 news:op.vksxyn15o7cclz korden-pc...
 On Tue, 19 Oct 2010 06:26:23 +0400, Nick Sabalausky <a a.a> wrote:

 "Denis Koroskin" <2korden gmail.com> wrote in message
 Running the build script  takes 7.967 seconds for me (timed with  
 timeit,
 part of the Windows Server  2003 Resource Kit Tools).

My guess is it may be the multi-threaded "process multiple modules in parallel" optimization. DDMD does have a *lot* of source files. And I'm on a single-core (yea, yea, I know, but it works for me and I have no money). I'm curious, is it possible for you or Rainer to time it running dmd in a "force single-core" mode? Ideally using the main system/OS core (if there even is one)? I would expect a normal out-of-the-box dmd would be noticably faster for you two just because of that multi-thread/core optimization and the cores themselves just simply being faster, but 120 sec vs 8 sec still seems an excessive difference. And then considering also, outside of dmd bugs, I've never once had any D program besides ddmd take more than a few seconds - despite the fact that much of what I do complie is fairly template/mixin/ctfe-heavy, and ddmd appears to use very little (if any) of that - it's certainly still a conceivable speed difference, but it does make me wonder if something else might be going on. BTW, anyone know of an easy way to check how much time in a process is spent waiting on disk IO? It's entirely possible that my disk is just fragmented all to hell.

dmd doesn't use multiple core for anything BUT asynchronous reading from disk. Everything else is done is a single thread.

Ahh, I see. So much for that theory then.
 I don't think it has  THAT big of impact, but I'll try to recompile with
 all the cores disabled.

Thanks, that'll be interesting. I did go ahead and re-time the compile. Apparently I must have been remembering it wrong, because this time it only took 1 min 20 sec (and this was with a ton of stuff running - bunch of misc apps, FF2 with 20 tabs, an HDD SMART monitor, a torrent manager and a bunch of other servers (but no clients connected)). But that's still quite a lot of time to compile a D app, even for my machine. I also tried grabbing the latest ddmd, rebooted, killed all non-essential processes, and tried that way. Got it down to just slightly under one minute. I thought about maybe it being a limited-memory issue (remembering that dmd never frees anything until it's done - or is that just CTFE?), but I don't think that's it - the highest memory usage it ever got was about 200MB, and I have 1GB, and it still took a whole minute with almost everything besides XP shut down, so I'm not sure that was it. (I could have sworn I had 2GB at one point, but I think I probably cannibalized one of the sticks when I built my linux box - not that that's really relevant ;) ) FWIW, this is all with compiling the just the debug version of ddmd only. Ie, not including building the release version or the one-time initial setup of building dmd.lib.

It's 8.6 seconds for a single cores, 8.3s for all 4 cores (Core2 Quad Q8300 2.5Ghz, Windows).
Oct 18 2010
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
dsimcha wrote:
 I've noticed that, when passed multiple files at once, DMD is generally buggy
 in ways that I can't reproduce in small test cases.  This includes things like
 magically ignoring __gshared variables, and not being able to convert a type
 to an alias to the same type, for example, not being able to convert a size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where R is
 a template parameter instantiated as float[].
 
 I have two questions:
 
 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?

Phobos and Druntime are built by passing all the files to dmd at once, and it works without problems. No attempt was made to tweak the order the files were presented to dmd to make it work, I just aggregated them all.
Oct 13 2010
parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
On Wed, 13 Oct 2010 14:15:00 -0700, Walter Bright wrote:

 dsimcha wrote:
 I've noticed that, when passed multiple files at once, DMD is generally
 buggy in ways that I can't reproduce in small test cases.  This
 includes things like magically ignoring __gshared variables, and not
 being able to convert a type to an alias to the same type, for example,
 not being able to convert a size_t to a uint on 32-bit, or not being
 able to convert a float[] to an R where R is a template parameter
 instantiated as float[].
 
 I have two questions:
 
 1.  Has anyone experienced similar things and if so are they further
 along than me at creating decent test cases?

Phobos and Druntime are built by passing all the files to dmd at once, and it works without problems. No attempt was made to tweak the order the files were presented to dmd to make it work, I just aggregated them all.

It's not quite true that it works without problems. I reported bug 3979 because I was unable to include the new std.process in Phobos without changing the order of the files passed to DMD. -Lars
Oct 13 2010
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
I'm also having problems with VisualD + DMD, I made a posting in the 
bugs forums about it, but unfortunately no one did answer yet:

http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D.bugs&artnum=25536

Am 13.10.2010 21:45, schrieb dsimcha:
 I've noticed that, when passed multiple files at once, DMD is generally buggy
 in ways that I can't reproduce in small test cases.  This includes things like
 magically ignoring __gshared variables, and not being able to convert a type
 to an alias to the same type, for example, not being able to convert a size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where R is
 a template parameter instantiated as float[].

 I have two questions:

 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?

 2.  I'm used to using Code::Blocks, but want to switch to Visual D because it
 seems to work a lot better for the most part.  However, by default it seems to
 pass all project files to the compiler at once, where Code::Blocks compiles
 one file at a time.  This means I have to deal with DMD's bugginess when
 passed mutliple files at once.  Does anyone know how to make Visual D work
 Code::Blocks style, i.e. compile each file to an object file and link them
 afterwords?

-- Kind Regards Benjamin Thaut
Oct 13 2010
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday 13 October 2010 23:28:40 Benjamin Thaut wrote:
 I'm also having problems with VisualD + DMD, I made a posting in the
 bugs forums about it, but unfortunately no one did answer yet:
 
 http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group
 =digitalmars.D.bugs&artnum=25536

The bugs forum is not for posting to. The bugtracker sends information on the creation of and changes to bug reports. You sign up for the list if you want to see those messages. Pretty much no one is going to pay attention to anything that you post to the bug list directly. Typically, you'd post to the D list or the D-Learn list (D being for general stuff and Learn for questions about how the language works and anything related to learning D). - Jonathan M Davis
Oct 13 2010
prev sibling next sibling parent klickverbot <see klickverbot.at> writes:
On 10/14/10 8:28 AM, Benjamin Thaut wrote:
 I'm also having problems with VisualD + DMD, I made a posting in the
 bugs forums about it, but unfortunately no one did answer yet:

Uh, afaik the digitalmars.D.bugs forum just mirrors the changes from the puremagic D bugzilla, and thus is not quite the best place for discussions.
Oct 14 2010
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday 13 October 2010 23:58:16 Jonathan M Davis wrote:
 The bugs forum is not for posting to.

Ack. I meant the bug mailing list, not forum. Oh well, it should be fairly obvious what I meant. - Jonathan M Davis
Oct 14 2010
prev sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <ludwig informatik.uni-luebeck.de> writes:
Am 13.10.2010 21:45, schrieb dsimcha:
 I've noticed that, when passed multiple files at once, DMD is generally buggy
 in ways that I can't reproduce in small test cases.  This includes things like
 magically ignoring __gshared variables, and not being able to convert a type
 to an alias to the same type, for example, not being able to convert a size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where R is
 a template parameter instantiated as float[].

 I have two questions:

 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?

 2.  I'm used to using Code::Blocks, but want to switch to Visual D because it
 seems to work a lot better for the most part.  However, by default it seems to
 pass all project files to the compiler at once, where Code::Blocks compiles
 one file at a time.  This means I have to deal with DMD's bugginess when
 passed mutliple files at once.  Does anyone know how to make Visual D work
 Code::Blocks style, i.e. compile each file to an object file and link them
 afterwords?

Right now, I'm in exactly the same situation. However, the problems might be a bit more obvious: Sometimes symbols are missing at link time that should not and sometimes some symbols (most of the time phobos symbols) are defined twice but with exactly the same module path. (Error: multiple definition of conv_208_199f: _D53C:\D\dmd2\windows\bin\..\..\src\phobos\std\conv.d.52012__ModuleInfoZ and conv: _D53C:\D\dmd2\windows\bin\..\..\src\phobos\std\conv.d.52012__ModuleInfoZ) On Mac OS, using all-at-once compilation I get defective .lib files (the ar tool complains that the .o file is smaller than it reports or something similar) so I have to compile the code of all static libs together with the application code. The problem is that it is really difficult (almost impossible) to isolate the problem in any way because sometimes appears erratically and the code base is quite large. And also I cannot share the source code for debugging.
Oct 14 2010