www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 11021] New: Huge GC leak leads to crash; GC uses 7x more memory

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021

           Summary: Huge GC leak leads to crash; GC uses 7x more memory
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: blocker
          Priority: P2
         Component: druntime
        AssignedTo: nobody puremagic.com
        ReportedBy: temtaime gmail.com


--- Comment #0 from Temtaime <temtaime gmail.com> 2013-09-13 07:09:22 PDT ---
import core.memory;
import std.traits;

void main() {
    while(true) {
        GC.collect();
        new ubyte[40 * 1024 * 1024];
    }
}

Application's memory usage grows and it crashes with message:
core.exception.OutOfMemoryError

If i comment import std.traits; everything goes OK, but now application uses
300 MM of memory.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021


monarchdodra gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |monarchdodra gmail.com


--- Comment #1 from monarchdodra gmail.com 2013-09-13 09:54:03 PDT ---
Are you, by any chance, on win32?

What is your metric for "GC uses 7x more memory"?

Also, you say: "but now application uses 300 MM of memory." But is that a
problem?

I don't know what to make about your "import" issue, as I can't reproduce on
linux. I'll try on windows.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #2 from monarchdodra gmail.com 2013-09-13 10:01:26 PDT ---
(In reply to comment #1)
 Are you, by any chance, on win32?
I can reproduce on win32, all version of dmd.
 What is your metric for "GC uses 7x more memory"?
 
 Also, you say: "but now application uses 300 MM of memory." But is that a
 problem?
Never mind, I understood what you meant.
 I don't know what to make about your "import" issue, as I can't reproduce on
 linux. I'll try on windows.
I don't know how you did it, but I run out of memory regardless. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #3 from Temtaime <temtaime gmail.com> 2013-09-13 10:07:59 PDT ---
Yes, it's on win32.

I think app should use about 50 mb as i'm allocating 40 mb array, not 300.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021


Orvid King <blah38621 gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |blah38621 gmail.com


--- Comment #4 from Orvid King <blah38621 gmail.com> 2013-09-13 10:10:20 PDT ---
(In reply to comment #3)
 Yes, it's on win32.
 
 I think app should use about 50 mb as i'm allocating 40 mb array, not 300.
I would actually expect it to use 90mb, your allocating a 40mb array inside a loop, the previous 40mb array will still be referenced when you go to allocate the new 40mb array, meaning that it can't be collected if the second allocation triggers a GC. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #5 from Temtaime <temtaime gmail.com> 2013-09-13 10:12:56 PDT ---
In loop i calling collect, so gc should frees the memory.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #6 from Orvid King <blah38621 gmail.com> 2013-09-13 10:19:48 PDT ---
(In reply to comment #5)
 In loop i calling collect, so gc should frees the memory.
You should really be clearing the existing array rather than allocating a new one, but that's beside the point. When you call the GC manually, the array is still referenced in local variable, if you were to reset that local variable before calling the GC, then it should collect it correctly. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #7 from monarchdodra gmail.com 2013-09-13 10:38:13 PDT ---
(In reply to comment #4)
 I would actually expect it to use 90mb, your allocating a 40mb array inside a
 loop, the previous 40mb array will still be referenced when you go to allocate
 the new 40mb array, meaning that it can't be collected if the second allocation
 triggers a GC.
(In reply to comment #6)
 You should really be clearing the existing array rather than allocating a new
 one, but that's beside the point. When you call the GC manually, the array is
 still referenced in local variable, if you were to reset that local variable
 before calling the GC, then it should collect it correctly.
He keeps no references to the array, so it should be collected immediatly on each call to GC.collect(). OR at least, there is no reason for it not to be collected. That said, I'm unsure "GC.collect()" promises it'll acutally *do* anything, it's more of a hint "now would be a good time to run a collection" (AFAIK). -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021


Dmitry Olshansky <dmitry.olsh gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dmitry.olsh gmail.com


--- Comment #8 from Dmitry Olshansky <dmitry.olsh gmail.com> 2013-09-13
11:51:16 PDT ---
(In reply to comment #0)
 import core.memory;
 import std.traits;
 
 void main() {
     while(true) {
         GC.collect();
         new ubyte[40 * 1024 * 1024];
     }
 }
 
 Application's memory usage grows and it crashes with message:
 core.exception.OutOfMemoryError
 
 If i comment import std.traits; everything goes OK, but now application uses
 300 MM of memory.
32bits? Allocating large blocks in a loop? Good luck with that... sooner or later they would be pinned by something that looks like a pointer to GC. We are back to false pointers and precise collection topic. How std.traits affects that is interesting thing, maybe inflates executable size. Anyway this is game of chance so any slight change may avoid it or bring it back. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #9 from monarchdodra gmail.com 2013-09-13 12:07:28 PDT ---
(In reply to comment #8)
 32bits? Allocating large blocks in a loop? 
 Good luck with that... sooner or later they would be pinned by something that
 looks like a pointer to GC.  We are back to false pointers and precise
 collection topic.
 
 How std.traits affects that is interesting thing, maybe inflates executable
 size. Anyway this is game of chance so any slight change may avoid it or bring
 it back.
2 things I don't understand: 1. New initializes, so the block contains nothing but 0x00's. How could that contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as pointer? 2. Besides, the GC knows about NO_SCAN, doesn't it? We are explicitly asking for new ubytes, which is a non-pointer type. Why would the GC scan any of that at all anywas? If replace the call with GC.malloc(BIG_SIZE, GC.BlkAttr.NO_SCAN), then my win32 runs forever... -------- TLDR, I don't see how the false pointer issue in 32 bit environment would produce this failure. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #10 from Dmitry Olshansky <dmitry.olsh gmail.com> 2013-09-13
13:22:18 PDT ---
(In reply to comment #9)
 (In reply to comment #8)
 32bits? Allocating large blocks in a loop? 
 Good luck with that... sooner or later they would be pinned by something that
 looks like a pointer to GC.  We are back to false pointers and precise
 collection topic.
 
 How std.traits affects that is interesting thing, maybe inflates executable
 size. Anyway this is game of chance so any slight change may avoid it or bring
 it back.
2 things I don't understand: 1. New initializes, so the block contains nothing but 0x00's. How could that contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as
False pointers that HAPPEN TO POINT INTO THAT BLOCK. I can't imagine we are having this conversation again. The larger the block the better target it makes.
 pointer?
 
 2. Besides, the GC knows about NO_SCAN, doesn't it? We are explicitly asking
 for new ubytes, which is a non-pointer type. Why would the GC scan any of that
 at all anywas?
 
Yes it SHOULDN'T scan THAT BLOCK.
 If replace the call with GC.malloc(BIG_SIZE, GC.BlkAttr.NO_SCAN), then my win32
 runs forever...
 
Or change something else and it might not... ? This is an erratic issue.
 --------
 
 TLDR, I don't see how the false pointer issue in 32 bit environment would
 produce this failure.
-- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #11 from monarchdodra gmail.com 2013-09-13 13:53:36 PDT ---
(In reply to comment #10)
 (In reply to comment #9)
 2 things I don't understand:
 
 1. New initializes, so the block contains nothing but 0x00's. How could that
 contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as
False pointers that HAPPEN TO POINT INTO THAT BLOCK. I can't imagine we are having this conversation again. The larger the block the better target it makes.
Ah. Right. I apologize for not being there the last time you had this conversation... -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #12 from Temtaime <temtaime gmail.com> 2013-09-13 23:43:45 PDT ---
It's array of ubytes, not pointers. GC mustn't scan its content.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021


safety0ff.bugz gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |safety0ff.bugz gmail.com


--- Comment #13 from safety0ff.bugz gmail.com 2013-10-03 06:45:06 PDT ---
(In reply to comment #12)
 It's array of ubytes, not pointers. GC mustn't scan its content.
I think you misunderstand, something in std.traits gets scanned and it is treated as a pointer into the array of ubytes. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Oct 03 2013