www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Concern about dmd memory usage on win32

reply "monarch_dodra" <monarchdodra gmail.com> writes:
In particular, when compiling "-unittest std\algorithm.d", dmd 
uses *nearly* 1 GB (it uses about 1,051,176K on my machine).

Problem is that when it reaches 1GB, it crashes. I have a pull 
request which adds a few unittests to algorithm, and it is 
consistently crashing on win32 with an out of memory error.

In layman's terms: std\algorithm.d is full. You literally can't 
add any more unittests to it, without crashing dmd on win32.

I'd have recommended splitting the unittests in sub-modules or 
whatnot, but come to think about it, I'm actually more concern 
that a module could *singlehandedly* make the compiler crash on 
out of memory...

Also, I'm no expert, but why is my dmd limited to 1 GB memory on 
my 64 bit machine...?
Dec 07 2012
next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 7 December 2012 at 14:23:39 UTC, monarch_dodra wrote:
 [SNIP]

Come to think about it, what's funny is that the pull *used* to pass, but it doesn't anymore, and I haven't changed anything in the mean-time. There *have* been changes to algorithm, so that may be it, but it may also be a performance regression of dmd. Again, I'm no expert, so all I can do is report :/
Dec 07 2012
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12/7/2012 6:23 PM, monarch_dodra пишет:
 In particular, when compiling "-unittest std\algorithm.d", dmd uses
 *nearly* 1 GB (it uses about 1,051,176K on my machine).

 Problem is that when it reaches 1GB, it crashes. I have a pull request
 which adds a few unittests to algorithm, and it is consistently crashing
 on win32 with an out of memory error.

Yup, it dies the same way on auto-tester for my pull.
 In layman's terms: std\algorithm.d is full. You literally can't add any
 more unittests to it, without crashing dmd on win32.

 I'd have recommended splitting the unittests in sub-modules or whatnot,
 but come to think about it, I'm actually more concern that a module
 could *singlehandedly* make the compiler crash on out of memory...

 Also, I'm no expert, but why is my dmd limited to 1 GB memory on my 64
 bit machine...?

It's not large address-space aware (a matter of a proper bit set in PE header) thus is limited to 2Gb. The other part of problem has to do with the way DMC run-time allocates virtual memory. Somebody fixed it but the patch failed to get any recognition. -- Dmitry Olshansky
Dec 07 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, December 07, 2012 15:23:36 monarch_dodra wrote:
 In particular, when compiling "-unittest std\algorithm.d", dmd
 uses *nearly* 1 GB (it uses about 1,051,176K on my machine).
 
 Problem is that when it reaches 1GB, it crashes. I have a pull
 request which adds a few unittests to algorithm, and it is
 consistently crashing on win32 with an out of memory error.
 
 In layman's terms: std\algorithm.d is full. You literally can't
 add any more unittests to it, without crashing dmd on win32.
 
 I'd have recommended splitting the unittests in sub-modules or
 whatnot, but come to think about it, I'm actually more concern
 that a module could *singlehandedly* make the compiler crash on
 out of memory...
 
 Also, I'm no expert, but why is my dmd limited to 1 GB memory on
 my 64 bit machine...?

If you look in win32.mak, you'll see that the source files are split into separate groups (STD_1_HEAVY, STD_2_HEAVY, STD_3, STD_4, etc.). This is specifically to combat this problem. Every time that we reach the point that the compilation starts running out of memory again, we add more groups and/or rearrange them. It's suboptimal, but I don't know what else we can do at this point given dmd's limitations on 32-bit Windows. - Jonathan M Davis
Dec 07 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/7/12 1:43 PM, Jonathan M Davis wrote:
 The GC didn't break things per se. It was just made compilation much slower,
 and Walter didn't have time to fix it at the time (as dmd was close to a
 release), so it was disable. But someone needs to take the time to work on it
 and make it efficient enough to use (possibly doing stuff like making it so
that
 it only kicks in at least a certain amount of memory is used to keep the
 common case fast but make the memory-intensive cases work). And no one has
 done that.

I suggested this several times: work the GC so it only intervenes if the consumed memory would otherwise be prohibitively large. That way there's never a collection during normal compilation. Andrei
Dec 07 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/7/12 5:37 PM, deadalnix wrote:
 On Friday, 7 December 2012 at 22:30:59 UTC, Andrei Alexandrescu wrote:
 On 12/7/12 1:43 PM, Jonathan M Davis wrote:
 The GC didn't break things per se. It was just made compilation much
 slower,
 and Walter didn't have time to fix it at the time (as dmd was close to a
 release), so it was disable. But someone needs to take the time to
 work on it
 and make it efficient enough to use (possibly doing stuff like making
 it so that
 it only kicks in at least a certain amount of memory is used to keep the
 common case fast but make the memory-intensive cases work). And no
 one has
 done that.

I suggested this several times: work the GC so it only intervenes if the consumed memory would otherwise be prohibitively large. That way there's never a collection during normal compilation. Andrei

Nobody told you that the GC was THAT SLOW that it as even slower than the swap ? You really know nothing about optimization, don't you ?

This is not even remotely appropriate. Where did it come from? Andrei
Dec 07 2012
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/7/2012 2:51 PM, deadalnix wrote:
 I'm working on a program that now require more than 2.5Gb of RAM to compile,
 where separate compilation is not possible due to bug 8997 and that randomly
 fails to compile due to bug 8596. It is NOT fast and that insane memory
 consumption is a major cause of slowness.

I'm pretty sure the memory consumption happens with CTFE and Don is working on it.
Dec 10 2012
parent evansl <cppljevans suddenlink.net> writes:
On 12/10/12 06:45, Walter Bright wrote:
 On 12/7/2012 2:51 PM, deadalnix wrote:
 I'm working on a program that now require more than 2.5Gb of RAM to
 compile,
 where separate compilation is not possible due to bug 8997 and that
 randomly
 fails to compile due to bug 8596. It is NOT fast and that insane memory
 consumption is a major cause of slowness.

I'm pretty sure the memory consumption happens with CTFE and Don is working on it.

it also gives very detailed information that indicates which parts of your program are responsible for allocating the heap memory. from here: http://valgrind.org/docs/manual/ms-manual.html suggests massif might be some help in narrowing down the cause.
Dec 10 2012
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Fri, 07 Dec 2012 08:23:06 -0800
Jonathan M Davis <jmdavisProg gmx.com> wrote:

 On Friday, December 07, 2012 15:23:36 monarch_dodra wrote:
 In particular, when compiling "-unittest std\algorithm.d", dmd
 uses *nearly* 1 GB (it uses about 1,051,176K on my machine).
 
 Problem is that when it reaches 1GB, it crashes. I have a pull
 request which adds a few unittests to algorithm, and it is
 consistently crashing on win32 with an out of memory error.
 
 In layman's terms: std\algorithm.d is full. You literally can't
 add any more unittests to it, without crashing dmd on win32.
 
 I'd have recommended splitting the unittests in sub-modules or
 whatnot, but come to think about it, I'm actually more concern
 that a module could *singlehandedly* make the compiler crash on
 out of memory...
 
 Also, I'm no expert, but why is my dmd limited to 1 GB memory on
 my 64 bit machine...?

If you look in win32.mak, you'll see that the source files are split into separate groups (STD_1_HEAVY, STD_2_HEAVY, STD_3, STD_4, etc.). This is specifically to combat this problem. Every time that we reach the point that the compilation starts running out of memory again, we add more groups and/or rearrange them. It's suboptimal, but I don't know what else we can do at this point given dmd's limitations on 32-bit Windows.

Sooo...what's the status of fixing DMD's forever-standing memory usage issues? My understanding is that the big issues are: 1. CTFE allocates every time a CTFE variable's value is changed. 2. The GC inside DMD is disabled because it broke things, so it never releases memory. Is this correct? If so, what's the current status of fixes? It seems to me this would be something that should be creeping higher and higher up the priority list (if it hasn't already been doing so).
Dec 07 2012
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 7 December 2012 at 16:23:49 UTC, Jonathan M Davis 
wrote:
 If you look in win32.mak, you'll see that the source files are 
 split into
 separate groups (STD_1_HEAVY, STD_2_HEAVY, STD_3, STD_4, etc.). 
 This is
 specifically to combat this problem. Every time that we reach 
 the point that
 the compilation starts running out of memory again, we add more 
 groups and/or
 rearrange them. It's suboptimal, but I don't know what else we 
 can do at this
 point given dmd's limitations on 32-bit Windows.

 - Jonathan M Davis

I had actually been through this before, and someone told me about that. The problem at this point is that this isn't even an option anymore, since std/algorithm.d is in a group *alone*. On Friday, 7 December 2012 at 17:07:10 UTC, Nick Sabalausky wrote:
 [SNIP]
 Is this correct? If so, what's the current status of fixes? It 
 seems to
 me this would be something that should be creeping higher and 
 higher up
 the priority list (if it hasn't already been doing so).

What he said.
Dec 07 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, December 07, 2012 18:18:43 monarch_dodra wrote:
 I had actually been through this before, and someone told me
 about that. The problem at this point is that this isn't even an
 option anymore, since std/algorithm.d is in a group *alone*.

Then I'd have two suggestions then: 1. Figure out which tests are too expensive. Either disable them or make them less expensive. If we can't test as much as we need to right now, then the less critical tests will just need to be disabled as ugly as that may be. 2. version stuff. Either specifically version some of it out on Windows (or maybe just 32-bit Windows now that we have 64-bit Windows), or put some of the less critical stuff in version blocks that can be explicitly enabled by someone working on Phobos. The two should probably be combined though. Figure out which tests are problematic and version out the less critical ones on 32-bit Windows. Then you can create a bug report for those tests specifically and anyone working on the memory problem with have somethingh specific to run. Much of std.datetime's tests used to have to be disabled with version blocks on Windows until the compiler was improved enough and/or the tests were adjusted enough that they could be run on Windows (adjusting the makefile probably helped as well). Templates and CTFE are particularly expensive, so fantastic tricks like using foreach with TypeTuple can really cost a lot with dmd's current memory issues, and std.algorithm may just not be able to afford some of the better tests right now as much as that sucks. - Jonathan M Davis
Dec 07 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, December 07, 2012 12:07:06 Nick Sabalausky wrote:
 Sooo...what's the status of fixing DMD's forever-standing memory usage
 issues?
 
 My understanding is that the big issues are:
 
 1. CTFE allocates every time a CTFE variable's value is changed.
 
 2. The GC inside DMD is disabled because it broke things, so it never
 releases memory.
 
 Is this correct? If so, what's the current status of fixes? It seems to
 me this would be something that should be creeping higher and higher up
 the priority list (if it hasn't already been doing so).

The GC didn't break things per se. It was just made compilation much slower, and Walter didn't have time to fix it at the time (as dmd was close to a release), so it was disable. But someone needs to take the time to work on it and make it efficient enough to use (possibly doing stuff like making it so that it only kicks in at least a certain amount of memory is used to keep the common case fast but make the memory-intensive cases work). And no one has done that. Walter has been busy with other stuff and has made it clear that it's likely going to need to be someone else who steps up and fixes it, so we're stuck until soemone does that. As for CTFE, I don't know what the current state is. Don has plans, but I get the impression that he's too busy to get much done with them these days. We're dealing with a problem that requires some of our key developers (or someone to put in enough time and effort to learn much of what they know) in order to get it done, so it's fallen by the wayside thus far. - Jonathan M Davis
Dec 07 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, December 07, 2012 19:43:50 Jonathan M Davis wrote:
 Then I'd have two suggestions then:

I really need to reread my posts more before actually posting them... - Jonathan M Davis
Dec 07 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/7/12, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 Then I'd have two suggestions then:

Actually there is one other way, use version specifiers and then re-compile using different versions, e.g.: <line 1> version(StdAlgTest1) { <..lots of tests...> } <line 1000> version(StdAlgTest2) { <..lots of tests...> } <line 2000> version(StdAlgTest3) { <..lots of tests...> } Then the makefile would have to compile algorithm.d 3 times, via something like: $ rdmd --unittest -version=StdAlgTest1 --main std\algorithm.d $ rdmd --unittest -version=StdAlgTest2 --main std\algorithm.d $ rdmd --unittest -version=StdAlgTest3 --main std\algorithm.d In fact, why aren't we taking advantage and using rdmd already instead of using a seperate unittest.d file? I've always used rdmd to test my Phobos changes, it works very simple this way. All it takes to test a module is to pass the --unittest and --main flags and the module name.
Dec 07 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, December 07, 2012 20:04:54 Andrej Mitrovic wrote:
 On 12/7/12, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 Then I'd have two suggestions then:

re-compile using different versions, e.g.: <line 1> version(StdAlgTest1) { <..lots of tests...> } <line 1000> version(StdAlgTest2) { <..lots of tests...> } <line 2000> version(StdAlgTest3) { <..lots of tests...> } Then the makefile would have to compile algorithm.d 3 times, via something like: $ rdmd --unittest -version=StdAlgTest1 --main std\algorithm.d $ rdmd --unittest -version=StdAlgTest2 --main std\algorithm.d $ rdmd --unittest -version=StdAlgTest3 --main std\algorithm.d In fact, why aren't we taking advantage and using rdmd already instead of using a seperate unittest.d file? I've always used rdmd to test my Phobos changes, it works very simple this way. All it takes to test a module is to pass the --unittest and --main flags and the module name.

The windows build purposefully creates one executable in order to catch stuff like circular dependencies and to give dmd a larger project to compile at once in order to test dmd. Clearly, we're running into issues with that due to dmd's lack of capabilities when it comes to memory. But Walter has rejected all proposals to change it, and to some extent, I think that he's right. If anything, this is just highlighting an area that dmd really needs to be improved. All of the messing around that we've done with the makefiles is just hiding the problem. The POSIX builds do build the modules separately for unit tests, though they don't use dmd. I would point out though that as it stands, it wouldn't work to use rdmd, because it's in the tools project and not in dmd, druntime, or Phobos. Rather, it depends on them, so they can't depend on it. - Jonathan M Davis
Dec 07 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 7 December 2012 at 16:23:49 UTC, Jonathan M Davis 
wrote:
 If you look in win32.mak, you'll see that the source files are 
 split into
 separate groups (STD_1_HEAVY, STD_2_HEAVY, STD_3, STD_4, etc.). 
 This is
 specifically to combat this problem. Every time that we reach 
 the point that
 the compilation starts running out of memory again, we add more 
 groups and/or
 rearrange them. It's suboptimal, but I don't know what else we 
 can do at this
 point given dmd's limitations on 32-bit Windows.

 - Jonathan M Davis

I don't know? Maybe disabling the GC because it slowed down dmd wasn't a good idea after all. Who care about a fast compiler is that one crashes ? It does crash ! Yes but at least, it is fast !
Dec 07 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/7/12, deadalnix <deadalnix gmail.com> wrote:
 It does crash ! Yes but at least, it is fast !

Except it's not fast. It's slow and it crashes. It's a Yugo.
Dec 07 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, December 07, 2012 21:21:17 deadalnix wrote:
 On Friday, 7 December 2012 at 16:23:49 UTC, Jonathan M Davis
 
 wrote:
 If you look in win32.mak, you'll see that the source files are
 split into
 separate groups (STD_1_HEAVY, STD_2_HEAVY, STD_3, STD_4, etc.).
 This is
 specifically to combat this problem. Every time that we reach
 the point that
 the compilation starts running out of memory again, we add more
 groups and/or
 rearrange them. It's suboptimal, but I don't know what else we
 can do at this
 point given dmd's limitations on 32-bit Windows.
 
 - Jonathan M Davis

I don't know? Maybe disabling the GC because it slowed down dmd wasn't a good idea after all. Who care about a fast compiler is that one crashes ? It does crash ! Yes but at least, it is fast !

Most programs compile just fine as things are, and Walter cares a _lot_ about speed of compilation, so doing something harms the common case in favor of a less common one that doesn't even work right now didn't seem like a good idea. But really what it comes down to is that it was an experimental feature that clearly had problems, so it was temporarily disabled until it could be sorted out. All that means is that things were left exactly as they were rather than introducing a new element that could have caused problems. Further investigation and work _does_ need to be done, but without proper testing and further work being done on it, it probably _isn't_ a good idea to enable it. As with many things around here, the trick is that someone needs to spend time working on the problem, and no one has done so yet. - Jonathan M Davis
Dec 07 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 7 December 2012 at 22:30:59 UTC, Andrei Alexandrescu 
wrote:
 On 12/7/12 1:43 PM, Jonathan M Davis wrote:
 The GC didn't break things per se. It was just made 
 compilation much slower,
 and Walter didn't have time to fix it at the time (as dmd was 
 close to a
 release), so it was disable. But someone needs to take the 
 time to work on it
 and make it efficient enough to use (possibly doing stuff like 
 making it so that
 it only kicks in at least a certain amount of memory is used 
 to keep the
 common case fast but make the memory-intensive cases work). 
 And no one has
 done that.

I suggested this several times: work the GC so it only intervenes if the consumed memory would otherwise be prohibitively large. That way there's never a collection during normal compilation. Andrei

Nobody told you that the GC was THAT SLOW that it as even slower than the swap ? You really know nothing about optimization, don't you ?
Dec 07 2012
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
deadalnix:

 Nobody told you that the GC was THAT SLOW that it as even 
 slower than the swap ? You really know nothing about 
 optimization, don't you ?

Please be gentle in the forums, even with a person as strong as Andrei. Thank you. Bye, bearophile
Dec 07 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 7 December 2012 at 22:39:09 UTC, Andrei Alexandrescu 
wrote:
 Nobody told you that the GC was THAT SLOW that it as even 
 slower than
 the swap ? You really know nothing about optimization, don't 
 you ?

This is not even remotely appropriate. Where did it come from?

Sorry, I was trying to be ironic. People keep telling that the GC was removed because it was too slow, but it is pretty clear that, however how slow it is, it is not slower than swapping. I more and more irritated by the fact that we justify everything here because it have to be fast or whatever, and at the end, we have unreliable software that aren't even that fast in many cases. I'm working on a program that now require more than 2.5Gb of RAM to compile, where separate compilation is not possible due to bug 8997 and that randomly fails to compile due to bug 8596. It is NOT fast and that insane memory consumption is a major cause of slowness. make; make; make; make; make is the new make.
Dec 07 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 7 December 2012 at 22:42:16 UTC, bearophile wrote:
 deadalnix:

 Nobody told you that the GC was THAT SLOW that it as even 
 slower than the swap ? You really know nothing about 
 optimization, don't you ?

Please be gentle in the forums, even with a person as strong as Andrei. Thank you. Bye, bearophile

I never meant to say that Andrei was wrong, that the complete opposite. I made my point poorly and I apologize for that. I seemed obvious to me that the GC couldn't be slower than the swap and that nobody would take it seriously.
Dec 07 2012
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 7 December 2012 at 18:44:08 UTC, Jonathan M Davis 
wrote:
 On Friday, December 07, 2012 18:18:43 monarch_dodra wrote:
 I had actually been through this before, and someone told me
 about that. The problem at this point is that this isn't even 
 an
 option anymore, since std/algorithm.d is in a group *alone*.

Then I'd have two suggestions then: [SNIP] - Jonathan M Davis

I just tried to version each unittest (for win32) into two different objects (using version blocks). This worked relatively well, up until the final link step, where I was greeted with: //---- unittest5b.obj(unittest5b) Offset 00D68H Record Type 0091 Error 1: Previous Definition Different : _D3std9algorithm6EditOp6__initZ unittest5b.obj(unittest5b) Offset 685C4H Record Type 0091 Error 1: Previous Definition Different : _D3std9algorithm12__ModuleInfoZ unittest5b.obj(unittest5b) Offset 7ACEBH Record Type 00C3 Error 1: Previous Definition Different : __D3std9algorithm9__modtestFZv unittest5b.obj(unittest5b) Offset 7AFB4H Record Type 00C3 Error 1: Previous Definition Different : _D3std9algorithm7__arrayZ unittest5b.obj(unittest5b) Offset 7AFE0H Record Type 00C3 Error 1: Previous Definition Different : _D3std9algorithm8__assertFiZv unittest5b.obj(unittest5b) Offset 7B00CH Record Type 00C3 Error 1: Previous Definition Different : _D3std9algorithm15__unittest_failFiZv //---- The one I'm *really* concerned about is "ModuleInfo": My guess is that I'll never get rid of this error :/ I figure the "easy workaround", it to create a new dedicated executable, which tests just algorithm...? I don't think deactivating unit tests is a great move anyways...
Dec 10 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, December 10, 2012 13:34:19 monarch_dodra wrote:
 On Friday, 7 December 2012 at 18:44:08 UTC, Jonathan M Davis
 
 wrote:
 On Friday, December 07, 2012 18:18:43 monarch_dodra wrote:
 I had actually been through this before, and someone told me
 about that. The problem at this point is that this isn't even
 an
 option anymore, since std/algorithm.d is in a group *alone*.

Then I'd have two suggestions then: [SNIP] - Jonathan M Davis

I just tried to version each unittest (for win32) into two different objects (using version blocks). This worked relatively well, up until the final link step, where I was greeted with:

Different versions of the same module have to be done in separate builds. They couldn't all be in the same build. The Windows build is one executable, and it's been rejected to change that (for some good reasons - though it does cause problems here), so versioning the tests means that some of them won't be run as part of the normal unittest build. They'll have to be run manually or will only be run on other OSes which can handle the memory consumption. For quite a while, a lot of std.datetime's unit tests were just outright disabled on Windows, because the Windows version of dmd couldn't handle it. It's sounding like we're going to have to do the same with some of std.algorithm's unit tests until dmd's issues can be sorted out. - Jonathan M Davis
Dec 10 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 10 December 2012 at 12:46:19 UTC, Walter Bright wrote:
 On 12/7/2012 2:51 PM, deadalnix wrote:
 I'm working on a program that now require more than 2.5Gb of 
 RAM to compile,
 where separate compilation is not possible due to bug 8997 and 
 that randomly
 fails to compile due to bug 8596. It is NOT fast and that 
 insane memory
 consumption is a major cause of slowness.

I'm pretty sure the memory consumption happens with CTFE and Don is working on it.

I don't have a lot of it, but surely some.
Dec 10 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, December 07, 2012 15:23:36 monarch_dodra wrote:
 In particular, when compiling "-unittest std\algorithm.d", dmd
 uses *nearly* 1 GB (it uses about 1,051,176K on my machine).
 
 Problem is that when it reaches 1GB, it crashes. I have a pull
 request which adds a few unittests to algorithm, and it is
 consistently crashing on win32 with an out of memory error.
 
 In layman's terms: std\algorithm.d is full. You literally can't
 add any more unittests to it, without crashing dmd on win32.
 
 I'd have recommended splitting the unittests in sub-modules or
 whatnot, but come to think about it, I'm actually more concern
 that a module could *singlehandedly* make the compiler crash on
 out of memory...
 
 Also, I'm no expert, but why is my dmd limited to 1 GB memory on
 my 64 bit machine...?

Don just made some changes to Tuple so that it no longer uses std.metastrings.Format (which is _insanely_ inefficient), which may fix the memory problem that you were running into. Obviously, dmd still has issues, but a major one in the library was fixed (though we should really look at killing std.metastrings entirely, because it's too inefficient to reasonably use and the new format should now work with CTFE, unlike the old one - which is why Format exists in the first place). - Jonathan M Davis
Dec 11 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 11 December 2012 at 04:30:38 UTC, evansl wrote:
 On 12/10/12 06:45, Walter Bright wrote:
 On 12/7/2012 2:51 PM, deadalnix wrote:
 I'm working on a program that now require more than 2.5Gb of 
 RAM to
 compile,
 where separate compilation is not possible due to bug 8997 
 and that
 randomly
 fails to compile due to bug 8596. It is NOT fast and that 
 insane memory
 consumption is a major cause of slowness.

I'm pretty sure the memory consumption happens with CTFE and Don is working on it.

it also gives very detailed information that indicates which parts of your program are responsible for allocating the heap memory. from here: http://valgrind.org/docs/manual/ms-manual.html suggests massif might be some help in narrowing down the cause.

The problem with valgrind is that it does increase quite a lot the memory consumption of the program.
Dec 11 2012
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/7/2012 6:23 AM, monarch_dodra wrote:
 Also, I'm no expert, but why is my dmd limited to 1 GB memory on my 64 bit
 machine...?

The latest beta I uploaded increases the limit to 2 GB (thanks to a patch by Rainer Schuetze).
Dec 11 2012
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/14/2013 1:35 PM, monarch_dodra wrote:
 On Tuesday, 11 December 2012 at 22:11:55 UTC, Walter Bright wrote:
 On 12/7/2012 6:23 AM, monarch_dodra wrote:
 Also, I'm no expert, but why is my dmd limited to 1 GB memory on my 64 bit
 machine...?

The latest beta I uploaded increases the limit to 2 GB (thanks to a patch by Rainer Schuetze).

Is this patch in the main github release, or is there something special to change in the DMD makefile? I'm still having trouble, and am now having to deactivate some of algorithm's unittest to compile it, even without any changes. Any idea what I'm doing wrong?

Durn, I don't remember what the patch was.
Jan 14 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 14.01.2013 23:35, Walter Bright wrote:
 On 1/14/2013 1:35 PM, monarch_dodra wrote:
 On Tuesday, 11 December 2012 at 22:11:55 UTC, Walter Bright wrote:
 On 12/7/2012 6:23 AM, monarch_dodra wrote:
 Also, I'm no expert, but why is my dmd limited to 1 GB memory on my
 64 bit
 machine...?

The latest beta I uploaded increases the limit to 2 GB (thanks to a patch by Rainer Schuetze).

Is this patch in the main github release, or is there something special to change in the DMD makefile? I'm still having trouble, and am now having to deactivate some of algorithm's unittest to compile it, even without any changes. Any idea what I'm doing wrong?

Durn, I don't remember what the patch was.

The patch was in the heap allocation of dmc's runtime library snn.lib. The new lib seems to be in dmd 2.061, but it's probably not picked up when linking with dmc, only when building D files. It has to be copied to dmc's lib path aswell to have an effect on dmd itself.
Jan 14 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/14/2013 2:57 PM, Rainer Schuetze wrote:
 The patch was in the heap allocation of dmc's runtime library snn.lib. The new
 lib seems to be in dmd 2.061, but it's probably not picked up when linking with
 dmc, only when building D files. It has to be copied to dmc's lib path aswell
to
 have an effect on dmd itself.

Ah, that's right. Need to make sure you're using \dmd\windows\lib\snn.lib.
Jan 14 2013
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 11 December 2012 at 22:11:55 UTC, Walter Bright wrote:
 On 12/7/2012 6:23 AM, monarch_dodra wrote:
 Also, I'm no expert, but why is my dmd limited to 1 GB memory 
 on my 64 bit
 machine...?

The latest beta I uploaded increases the limit to 2 GB (thanks to a patch by Rainer Schuetze).

Is this patch in the main github release, or is there something special to change in the DMD makefile? I'm still having trouble, and am now having to deactivate some of algorithm's unittest to compile it, even without any changes. Any idea what I'm doing wrong?
Jan 14 2013
prev sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 15 January 2013 at 00:17:26 UTC, Walter Bright wrote:
 On 1/14/2013 2:57 PM, Rainer Schuetze wrote:
 The patch was in the heap allocation of dmc's runtime library 
 snn.lib. The new
 lib seems to be in dmd 2.061, but it's probably not picked up 
 when linking with
 dmc, only when building D files. It has to be copied to dmc's 
 lib path aswell to
 have an effect on dmd itself.

Ah, that's right. Need to make sure you're using \dmd\windows\lib\snn.lib.

Ah. I was still running with the lbs supplied in 2.060. That explains it then. Thanks!
Jan 14 2013