www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 10906] New: [2.064 git-head] Out of memory compiling Phobos on Windows

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906

           Summary: [2.064 git-head] Out of memory compiling Phobos on
                    Windows
           Product: D
           Version: unspecified
          Platform: All
        OS/Version: Windows
            Status: NEW
          Severity: regression
          Priority: P2
         Component: Phobos
        AssignedTo: nobody puremagic.com
        ReportedBy: cbkbbejeap mailinator.com



15:36:56 PDT ---
Using the latest github masters of DMD, druntime and phobos, compiling Phobos
on my Windows machine with 4GB RAM fails with "Error: out of memory". Despite
that, compiling the same versions inside a Linux (Debian 6) VirtualBox VM
*inside the same Windows machine* works fine.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 26 2013
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906


hsteoh quickfur.ath.cx changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hsteoh quickfur.ath.cx



Did you figure out what was causing the OOM? I'll bet it's either std.datetime
or std.algorithm (likely the latter if you're not running unittests --
std.datetime is very unittest-heavy, but it's std.algorithm that has
exponential template expansion in some places *cough* namely cartesianProduct,
written by yours truly).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906




17:26:58 PDT ---
The OOM is occurring during this (rather large) command:

---------------------------------
dmd -lib -ofphobos.lib -Xfphobos.json -O -release -w -d std\stdio.d
std\stdiobase.d  std\string.d std\format.d  std\file.d std\range.d std\array.d
std\functional.d std\path.d std\outbuffer.d std\utf.d  std\csv.d std\math.d
std\complex.d std\numeric.d std\bigint.d  std\metastrings.d std\bitmanip.d
std\typecons.d  std\uni.d std\base64.d std\md5.d std\ascii.d  std\demangle.d
std\uri.d std\mmfile.d std\getopt.d std\signals.d std\typetuple.d std\traits.d 
std\encoding.d std\xml.d  std\random.d  std\exception.d  std\compiler.d 
std\system.d std\concurrency.d std\datetime.d std\uuid.d std\digest\crc.d
std\digest\sha.d std\digest\md.d  std\digest\ripemd.d std\digest\digest.d 
std\algorithm.d std\variant.d  std\syserror.d std\zlib.d  std\stream.d
std\socket.d std\socketstream.d  std\container.d std\conv.d  std\zip.d
std\cstream.d std\regex.d  std\stdint.d  std\json.d  std\parallelism.d 
std\mathspecial.d  std\process.d  crc32.d  std\net\isemail.d std\net\curl.d 
std\c\process.d std\c\stdlib.d std\c\time.d std\c\stdio.d  std\c\math.d
std\c\stdarg.d std\c\stddef.d std\c\fenv.d std\c\string.d  std\c\locale.d
std\c\wcharh.d  std\windows\registry.d  std\windows\iunknown.d
std\windows\syserror.d std\windows\charset.d  std\c\windows\windows.d
std\c\windows\com.d  std\c\windows\winsock.d std\c\windows\stat.d 
std\internal\processinit.d std\internal\uni.d std\internal\uni_tab.d
std\internal\unicode_tables.d  std\internal\digest\sha_SSSE3.d 
std\internal\math\biguintcore.d  std\internal\math\biguintnoasm.d
std\internal\math\biguintx86.d  std\internal\math\gammafunction.d
std\internal\math\errorfunction.d  std\internal\windows\advapi32.d   
etc\c\zlib.d etc\c\curl.d etc\c\sqlite3.d  etc\c\zlib\zlib.lib
..\druntime\lib\druntime.lib
---------------------------------

Beyond that, I really don't know. Maybe the posix makefile splits it into
smaller chunks than that? Maybe a compiler change resulted in greater memory
usage on Windows? Maybe posix just handles low memory better? I'm not sure I'd
even know where to start.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906




No, in Linux it also runs that huge compile line.

One way to test is perhaps to copy-n-paste that command, omitting likely
culprits like std.algorithm and std.datetime (or std.regex, due to ctRegex, but
it's less likely) to see if it makes a difference.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906




Oh, and add -c to the command line as well, to prevent it from erroring out
from undefined symbols.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906




If the compiler attempts to use more than 2GB of virtual memory (including all
loaded libraries), then windows will throw an OOM error for exe's which aren't
marked as Large Address Aware. (such as dmd) As far as I know, posix systems
don't have this limitation and can use the full 4gb of the 32-bit address space
without needing to set a flag on the executable. I've managed to trigger this
error compiling one of my larger libraries (the arguments actually have to be
written to file before they can be passed to dmd, otherwise the command line is
too long) it ended up taking roughly 2.5gb of RAM (after I set the Large
Address Aware flag) for 2.63.2 to compile my 3.8mb across 1.3k heavily
interdependent modules into a static library. Perhaps the Large Address Aware
flag should be set by default on dmd?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906




Another thing to try, based on a wild gueses, is to edit std/algorithm.d and
add version(none) to cartesianProduct's unittests. One of them, I believe, has
a triple cartesian product, which in the current implementation causes an
explosion in the number of templates that must be instantiated.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906




17:43:25 PDT ---
Running the original command but with both std.algorthm and std.datetime
removed does succeed. Also, running the same command, but with *only*
std.algorthm and std.datetime *also* succeeds. So it would seem to be the
combination of everything that pushes it over.

However, it still seems a little strange, though: When I run the original
command, with everything, the memory usage only goes up to just over 600MB
before failing with "out of memory". And I have a couple gigs free, plus
virtual memory. Additionally, it works fine on Linux, on the same physical
machine, running *inside* a VM (set up with only 768MB RAM system memory).
(FWIW, I *am* still getting the OOM on Windows even when the Linux VM isn't
running - so it's definitely not a case of the VM using up too much of the
available RAM.)

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906




17:47:36 PDT ---
How do I set the Large Address Aware flag? I'm having trouble finding that info
through a web search.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906




18:06:59 PDT ---
Actually, if it works with std.algo and atd.datetime done separately (even
without the large address aware flag), and if those are notorious for being
large, perhaps I should just toss together a pull request adjusting phobos's
makefiles to do those separately? Or are there known issues with that?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906


yebblies <yebblies gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yebblies gmail.com



Like this, maybe:

http://www.pso-world.com/forums/showthread.php?t=199993

It is a single bit in the exe header, you could do it with a hex editor if you
really wanted to.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906






 Actually, if it works with std.algo and atd.datetime done separately (even
 without the large address aware flag), and if those are notorious for being
 large, perhaps I should just toss together a pull request adjusting phobos's
 makefiles to do those separately? Or are there known issues with that?


You should toss together a pull request that splits std.algorithm into
std.algorithm.package, std.algorithm.search, std.algorithm.sort,
std.algorithm.set, and std.algorithm.mutation.

:-)

Seriously, it's time we cut up that monster into more manageable pieces, now
that we have package.d.

I would say the same for std.datetime as well, but I believe Jonathan is
already working on that one.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906


Jonathan M Davis <jmdavisProg gmx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jmdavisProg gmx.com



PDT ---

 I would say the same for std.datetime as well, but I believe Jonathan is
 already working on that one.


I have it most of the way done already, but I'm quite busy at the moment, so
finishing it is going slowing.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 27 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906




13:09:44 PDT ---




 Actually, if it works with std.algo and atd.datetime done separately (even
 without the large address aware flag), and if those are notorious for being
 large, perhaps I should just toss together a pull request adjusting phobos's
 makefiles to do those separately? Or are there known issues with that?


 
 You should toss together a pull request that splits std.algorithm into
 std.algorithm.package, std.algorithm.search, std.algorithm.sort,
 std.algorithm.set, and std.algorithm.mutation.
 


That wouldn't address this out-of-memory issue though, would it?

I started modifying the makefiles to break up the compilation into smaller
steps, but I don't know how to do it without breaking the output of
phobos.json. We don't have any way to build phobos.json progressively, do we?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 28 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=10906


Nick Sabalausky <cbkbbejeap mailinator.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED



19:45:24 PDT ---
I'm closing this because the immediate issue is fixed by:

https://github.com/D-Programming-Language/dmd/pull/2507

However, that may not be a permanent solution: As phobos continues to grow, the
problem may resurface unless we address the issues of DMD memory usage and
adjusting the makefile to not build all of Phobos at once.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 01 2013