www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - memory allocation in dmd

reply Walter Bright <newshound2 digitalmars.com> writes:
Compiling std.algorithm for unittests consumes all the memory on many machines. 
I've been looking into what is allocating all that memory, and it isn't so easy 
without adding instrumentation code anywhere.

Anyone know of a convenient tool to do this on Linux?

(valgrind just hangs, or at least I gave up on it after 6 hours)
Jun 22 2013
next sibling parent reply "Geancarlo Rocha" <asdf mailinator.com> writes:
Maybe Dr. Memory could work out well as an alternative to 
valgrind with this issue?
http://www.drmemory.org/

On Saturday, 22 June 2013 at 21:41:15 UTC, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory 
 on many machines. I've been looking into what is allocating all 
 that memory, and it isn't so easy without adding 
 instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

Jun 22 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/22/2013 4:18 PM, Geancarlo Rocha wrote:
 Maybe Dr. Memory could work out well as an alternative to valgrind with this
issue?
 http://www.drmemory.org/

The web page indicates it is designed for looking for memory allocation bugs. That isn't the issue I'm having.
Jun 22 2013
prev sibling next sibling parent reply "Kiith-Sa" <kiithsacmp gmail.com> writes:
With valgrind; did you use massif?
That would be the right tool to use instead of memcheck (the 
default tool).
Jun 22 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/22/2013 4:24 PM, Kiith-Sa wrote:
 With valgrind; did you use massif?
 That would be the right tool to use instead of memcheck (the default tool).

I've never heard of massif. Will check it out.
Jun 22 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 06/23/2013 03:19 AM, Walter Bright wrote:
 On 6/22/2013 4:24 PM, Kiith-Sa wrote:
 With valgrind; did you use massif?
 That would be the right tool to use instead of memcheck (the default
 tool).

I've never heard of massif. Will check it out.

I tried it, works nice. No graphical output though. 1. compile dmd with debug symbols (-g) 2. valgrind --tool=massif ../dmd/src/dmd -main -unittest std/algorithm 3. ms_print massif.out.%PID% > outfile http://valgrind.org/docs/manual/ms-manual.html
Jun 24 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
24-Jun-2013 22:47, Martin Nowak пишет:
 On 06/23/2013 03:19 AM, Walter Bright wrote:
 On 6/22/2013 4:24 PM, Kiith-Sa wrote:
 With valgrind; did you use massif?
 That would be the right tool to use instead of memcheck (the default
 tool).

I've never heard of massif. Will check it out.

I tried it, works nice. No graphical output though. 1. compile dmd with debug symbols (-g) 2. valgrind --tool=massif ../dmd/src/dmd -main -unittest std/algorithm 3. ms_print massif.out.%PID% > outfile http://valgrind.org/docs/manual/ms-manual.html

There is a massif-visualizer tool that works nicely for me. -- Dmitry Olshansky
Jun 24 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-06-22 23:41, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory on many
 machines. I've been looking into what is allocating all that memory, and
 it isn't so easy without adding instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

There's a tool called dtrace on Mac OS X and *BSD which possibly could be used for this. Don't know if there's a corresponding tool for Linux. Hmm, seems to be available on Oracle Linux. http://en.wikipedia.org/wiki/DTrace -- /Jacob Carlborg
Jun 23 2013
next sibling parent reply "qznc" <qznc web.de> writes:
On Sunday, 23 June 2013 at 09:37:26 UTC, Jacob Carlborg wrote:
 On 2013-06-22 23:41, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory 
 on many
 machines. I've been looking into what is allocating all that 
 memory, and
 it isn't so easy without adding instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 
 hours)

There's a tool called dtrace on Mac OS X and *BSD which possibly could be used for this. Don't know if there's a corresponding tool for Linux. Hmm, seems to be available on Oracle Linux. http://en.wikipedia.org/wiki/DTrace

That would be SystemTap on Linux. However, I wonder if it is the right tool for the job.
Jun 23 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-06-23 15:12, qznc wrote:

 That would be SystemTap on Linux. However, I wonder if it is the right
 tool for the job.

Mac OS X has Instruments as well, which is used for these type of tasks. It's built on top of Dtrace: http://en.wikipedia.org/wiki/Instruments_(application) -- /Jacob Carlborg
Jun 23 2013
prev sibling next sibling parent "Nick B" <nick.barbalich gmail.com> writes:
On Sunday, 23 June 2013 at 15:22:22 UTC, Jacob Carlborg wrote:
 On 2013-06-23 15:12, qznc wrote:

 That would be SystemTap on Linux. However, I wonder if it is 
 the right
 tool for the job.

Mac OS X has Instruments as well, which is used for these type of tasks. It's built on top of Dtrace: http://en.wikipedia.org/wiki/Instruments_(application)

here is a comparion of Systemtap and DTrace http://sourceware.org/systemtap/wiki/SystemtapDtraceComparison
Jun 27 2013
prev sibling parent "Nick B" <nick.barbalich gmail.com> writes:
On Thursday, 27 June 2013 at 22:12:49 UTC, Nick B wrote:
 On Sunday, 23 June 2013 at 15:22:22 UTC, Jacob Carlborg wrote:
 On 2013-06-23 15:12, qznc wrote:

 That would be SystemTap on Linux. However, I wonder if it is 
 the right
 tool for the job.



 here is a comparion of Systemtap and DTrace


 http://sourceware.org/systemtap/wiki/SystemtapDtraceComparison

here is the SystemTap FAQ http://sourceware.org/systemtap/wiki/SystemTapFAQ it looks to be a very usefull tool. Nick
Jun 27 2013
prev sibling next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 22.06.2013 23:41, schrieb Walter Bright:
 Compiling std.algorithm for unittests consumes all the memory on many machines.
 I've been looking into what is allocating all that memory, and it isn't so easy
 without adding instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

there are "sometimes" better tools on windows or commercials - you should try these http://www.softwareverify.com/cpp-memory.php - i like this one http://mtuner.net/ http://www.puredevsoftware.com/MemPro.htm all for memory leak detection, but also doing memory profiling
Jun 23 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/23/2013 5:21 AM, dennis luehring wrote:
 Am 22.06.2013 23:41, schrieb Walter Bright:
 Compiling std.algorithm for unittests consumes all the memory on many machines.
 I've been looking into what is allocating all that memory, and it isn't so easy
 without adding instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

there are "sometimes" better tools on windows or commercials - you should try these http://www.softwareverify.com/cpp-memory.php - i like this one http://mtuner.net/ http://www.puredevsoftware.com/MemPro.htm all for memory leak detection, but also doing memory profiling

Thanks!
Jun 23 2013
prev sibling parent reply "Richard Webb" <webby beardmouse.org.uk> writes:
On Sunday, 23 June 2013 at 12:23:14 UTC, dennis luehring wrote:
 http://www.softwareverify.com/cpp-memory.php - i like this one

I've used that at work, though mainly for looking for leaks rather than memory profiling. Might try running DMD thruogh it if i get some time.
Jun 24 2013
parent reply Richard Webb <richard.webb boldonjames.com> writes:
On 24/06/2013 13:11, Richard Webb wrote:
 On Sunday, 23 June 2013 at 12:23:14 UTC, dennis luehring wrote:
 http://www.softwareverify.com/cpp-memory.php - i like this one

I've used that at work, though mainly for looking for leaks rather than memory profiling. Might try running DMD thruogh it if i get some time.

I had a go at running the VC build of DMD through Memory Validator, using "-release -unittest -c D:\DTesting\dmd.2.063\src\phobos\std\algorithm.d". It ran for about 45 mins, completed, and then locked up :-( I've put a screen shot of the object types list <http://tinyurl.com/qybudb4> as an example of what it shows - didn't get a chance to look at anything else before it fell over.
Jun 25 2013
parent dennis luehring <dl.soluz gmx.net> writes:
Am 25.06.2013 16:18, schrieb Richard Webb:
 On 24/06/2013 13:11, Richard Webb wrote:
 On Sunday, 23 June 2013 at 12:23:14 UTC, dennis luehring wrote:
 http://www.softwareverify.com/cpp-memory.php - i like this one

I've used that at work, though mainly for looking for leaks rather than memory profiling. Might try running DMD thruogh it if i get some time.

I had a go at running the VC build of DMD through Memory Validator, using "-release -unittest -c D:\DTesting\dmd.2.063\src\phobos\std\algorithm.d". It ran for about 45 mins, completed, and then locked up :-( I've put a screen shot of the object types list <http://tinyurl.com/qybudb4> as an example of what it shows - didn't get a chance to look at anything else before it fell over.

MTuner works - but i can't find a export or report feature
Jun 25 2013
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Saturday, 22 June 2013 at 21:41:15 UTC, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory 
 on many machines. I've been looking into what is allocating all 
 that memory, and it isn't so easy without adding 
 instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

Just wanted to add that with the latest release, I can't compile algorithm in unittest mode with -inline -O -release. This is actually an important test to run, as algorithm has a tendency to create "unreachable statement" error due to better flow analysis.
Jun 23 2013
prev sibling next sibling parent "Kiith-Sa" <kiithsacmp gmail.com> writes:
On Monday, 24 June 2013 at 18:47:17 UTC, Martin Nowak wrote:
 On 06/23/2013 03:19 AM, Walter Bright wrote:
 On 6/22/2013 4:24 PM, Kiith-Sa wrote:
 With valgrind; did you use massif?
 That would be the right tool to use instead of memcheck (the 
 default
 tool).

I've never heard of massif. Will check it out.

I tried it, works nice. No graphical output though. 1. compile dmd with debug symbols (-g) 2. valgrind --tool=massif ../dmd/src/dmd -main -unittest std/algorithm 3. ms_print massif.out.%PID% > outfile http://valgrind.org/docs/manual/ms-manual.html

sudo apt-get install massif-visualizer (on a newish debian/ubuntu/mint/etc)
Jun 24 2013
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
Another possibility is to comment some tests and see if valgrind 
finishes.
Jun 25 2013
prev sibling parent reply "Don" <turnyourkidsintocash nospam.com> writes:
On Saturday, 22 June 2013 at 21:41:15 UTC, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory 
 on many machines. I've been looking into what is allocating all 
 that memory, and it isn't so easy without adding 
 instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

I did a check on the CTFE part of it. You can do this by setting SHOWPERFORMANCE to 1 in the top of interpret.c. This produces: $ dmd -c -unittest std.algorithm ---- CTFE Performance ---- max call depth = 20 max stack = 63 array allocs = 356 assignments = 45356 That's actually not so terrible. The number of assignments gives a rough idea of how many CTFE statements are executed (almost everything interesting is an assignment). Most of the assignments are IntegerExpressions, of about 8 bytes, so it's under a megabyte in total. Note that there are very few array allocations, they are the thing that can really chew up memory quickly in CTFE. So although CTFE generally leaks memory like the Exxon Valdez leaks oil, I don't think it's to blame in this particular case.
Jul 02 2013
parent Richard Webb <richard.webb boldonjames.com> writes:
On 02/07/2013 11:30, Don wrote:
 $ dmd -c -unittest std.algorithm
          ---- CTFE Performance ----
 max call depth = 20    max stack = 63
 array allocs = 356    assignments = 45356

 That's actually not so terrible. The number of assignments gives a rough
 idea of how many CTFE statements are executed (almost everything
 interesting is an assignment). Most of the assignments are
 IntegerExpressions, of about 8 bytes, so it's under a megabyte in total.
 Note that there are very few array allocations, they are the thing that
 can really chew up memory quickly in CTFE.

 So although CTFE generally leaks memory like the Exxon Valdez leaks oil,
 I don't think it's to blame in this particular case.

I had another go at running memory validator on an MSVC build of the latest source, and that says that there are 619008 instances of IntegerExp, at 30 bytes each (it's the 3rd most numerous named object, though only uses 2.3% of the memory). I'm not sure how accurate that is as the wrapping of malloc etc in DMD confuse the results - e.g. it just lists 2159119 instances of 'rmem.c:46' rather than saying what they actually are. fwiw, it says there are 14,828,139 allocations totalling 814.7MB from '-c 'unittest' algorithm.d .
Jul 04 2013