digitalmars.D.bugs - [Issue 1804] New: Severe GC leaks with repetitive array allocations
- d-bugmail puremagic.com (28/28) Jan 24 2008 http://d.puremagic.com/issues/show_bug.cgi?id=1804
- Sean Kelly (8/22) Jan 24 2008 Interestingly, this app crashes in the same place if 'tmp' is deleted on
- d-bugmail puremagic.com (23/23) Jan 24 2008 http://d.puremagic.com/issues/show_bug.cgi?id=1804
- d-bugmail puremagic.com (16/16) Feb 05 2008 http://d.puremagic.com/issues/show_bug.cgi?id=1804
- Sean Kelly (5/5) Feb 07 2008 I've created a ticket for this in Tango:
- d-bugmail puremagic.com (4/4) Mar 31 2008 http://d.puremagic.com/issues/show_bug.cgi?id=1804
- d-bugmail puremagic.com (6/6) Apr 03 2008 http://d.puremagic.com/issues/show_bug.cgi?id=1804
- d-bugmail puremagic.com (12/12) Apr 01 2013 http://d.puremagic.com/issues/show_bug.cgi?id=1804
- d-bugmail puremagic.com (16/16) Jun 01 2013 http://d.puremagic.com/issues/show_bug.cgi?id=1804
http://d.puremagic.com/issues/show_bug.cgi?id=1804 Summary: Severe GC leaks with repetitive array allocations Product: D Version: 1.026 Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: P2 Component: DMD AssignedTo: bugzilla digitalmars.com ReportedBy: webmaster villagersonline.com Linux (Fedora 6), 64 bit The following program will run out of memory when the allocations reach about 8 MB. The problem happens on *both* 1.026 and 2.010 FYI: I tried running fullCollect() periodically, just in case...no luck. import std.stdio; void main() { for(int i=1; ; i++) { auto tmp = new char[i*1024]; if(i % 1_000 == 0) writefln("%d iterations, roughly %.1f MB", i, tmp.length/(1024*1024.0)); } } --
Jan 24 2008
d-bugmail puremagic.com wrote:FYI: I tried running fullCollect() periodically, just in case...no luck. import std.stdio; void main() { for(int i=1; ; i++) { auto tmp = new char[i*1024]; if(i % 1_000 == 0) writefln("%d iterations, roughly %.1f MB", i, tmp.length/(1024*1024.0)); } }Interestingly, this app crashes in the same place if 'tmp' is deleted on each iteration. If I had to guess, I'd say that empty pools aren't being added to the free list after collections, even though there's code in place that's supposed to do this. Perhaps it's just pools dedicated to big allocations that are the problem. This portion of the GC isn't really my area of expertise, but I'll submit a patch if I can figure this one out.
Jan 24 2008
http://d.puremagic.com/issues/show_bug.cgi?id=1804 This came up in a program where I had a large array, containing all of the prime numbers through some value. As I found more primes, I would append to the array. This works just fine, often (meaning that *sometimes* buffers are getting correctly collected), but after we hit some point the memory usage would spike because, whenever an append operation required a copy, the old array wasn't getting GC'd. I originally theorized that the prime numbers in the array were happening to match addresses of arrays (and that the bug was that the GC was scanning non-pointer memory). But that doesn't play out, exactly. My gut is still that there is a false pointer out there somewhere. Are the memory allocator metadata structures scanned by the GC? Maybe, after lots of allocations, we by random get metadata that looks like a pointer, which pins some memory....and then it gradually builds from there. Dunno. RECREATE UPDATE: Update the writefln line to also trace the address: writefln("%d iterations, roughly %.1f MB at %x", i, ..... , cast(uint)tmp.ptr); Then change the loop to start at 1014, 1015, or 1016 (maybe others) and the 1st buffer to be allocated will only get garbage collected once: after it is allocated the 2nd time, it doesn't appear that it is ever collected again. --
Jan 24 2008
http://d.puremagic.com/issues/show_bug.cgi?id=1804 Okay, I think I know what's going on. For big allocations, the GC will look for a pre-existing set of contiguous pages within a single pool to hold the block. If one cannot be found, then it creates a new pool of the requested size to hold the block. However, in this app the allocation size just continuously increases, so as time goes on the GC is unable to re-use existing pools and so allocates successively larger pools to hold the requested block. The problem is that all these old pools which were only used once are held by the GC for future use, so the total memory used by the application steadily increases, with most of this memory going completely unused. The simplest fix in this case would be for the GC to always release pools obtained for big allocations back to the OS when they are no longer needed. This should address the pathological case demonstrated in this test app. I'm going to make this change in Tango and see if it helps. --
Feb 05 2008
I've created a ticket for this in Tango: http://www.dsource.org/projects/tango/ticket/878 The fix is done and will be checked in once DSource is back online. The diff is pretty succinct, but I'd be happy to supply a patch for Phobos if that would help.
Feb 07 2008
http://d.puremagic.com/issues/show_bug.cgi?id=1804 FYI: I hit this problem on Phobos, not Tango. --
Mar 31 2008
http://d.puremagic.com/issues/show_bug.cgi?id=1804 Sure. But the same problem existed in Tango so I thought I'd diagnose it there and report my findings to save Walter some legwork with Phobos. Just trying to be helpful here :-) --
Apr 03 2008
http://d.puremagic.com/issues/show_bug.cgi?id=1804 Martin Krejcirik <mk krej.cz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED CC| |mk krej.cz Resolution| |FIXED Looks like it has been fixed somewhere between v1.056 and current v1.076. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Apr 01 2013
http://d.puremagic.com/issues/show_bug.cgi?id=1804 Martin Krejcirik <mk krej.cz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Component|DMD |Phobos Version|1.026 |D1 & D2 Resolution|FIXED | OS/Version|Linux |Windows I'm reopening this, because I noticed the problem (core.exception.OutOfMemoryError after about 20000 iterations) still exists on Windows XP 32bit with latest dmd 2.063. Works ok on Linux though. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Jun 01 2013