www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 11021] New: Huge GC leak leads to crash; GC uses 7x more memory

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021

           Summary: Huge GC leak leads to crash; GC uses 7x more memory
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: blocker
          Priority: P2
         Component: druntime
        AssignedTo: nobody puremagic.com
        ReportedBy: temtaime gmail.com



import core.memory;
import std.traits;

void main() {
    while(true) {
        GC.collect();
        new ubyte[40 * 1024 * 1024];
    }
}

Application's memory usage grows and it crashes with message:
core.exception.OutOfMemoryError

If i comment import std.traits; everything goes OK, but now application uses
300 MM of memory.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021


monarchdodra gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |monarchdodra gmail.com



Are you, by any chance, on win32?

What is your metric for "GC uses 7x more memory"?

Also, you say: "but now application uses 300 MM of memory." But is that a
problem?

I don't know what to make about your "import" issue, as I can't reproduce on
linux. I'll try on windows.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021






 Are you, by any chance, on win32?


I can reproduce on win32, all version of dmd.


 What is your metric for "GC uses 7x more memory"?
 
 Also, you say: "but now application uses 300 MM of memory." But is that a
 problem?


Never mind, I understood what you meant.


 I don't know what to make about your "import" issue, as I can't reproduce on
 linux. I'll try on windows.


I don't know how you did it, but I run out of memory regardless.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021




Yes, it's on win32.

I think app should use about 50 mb as i'm allocating 40 mb array, not 300.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021


Orvid King <blah38621 gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |blah38621 gmail.com





 Yes, it's on win32.
 
 I think app should use about 50 mb as i'm allocating 40 mb array, not 300.


I would actually expect it to use 90mb, your allocating a 40mb array inside a
loop, the previous 40mb array will still be referenced when you go to allocate
the new 40mb array, meaning that it can't be collected if the second allocation
triggers a GC.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021




In loop i calling collect, so gc should frees the memory.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021






 In loop i calling collect, so gc should frees the memory.


You should really be clearing the existing array rather than allocating a new
one, but that's beside the point. When you call the GC manually, the array is
still referenced in local variable, if you were to reset that local variable
before calling the GC, then it should collect it correctly.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021






 I would actually expect it to use 90mb, your allocating a 40mb array inside a
 loop, the previous 40mb array will still be referenced when you go to allocate
 the new 40mb array, meaning that it can't be collected if the second allocation
 triggers a GC.




 You should really be clearing the existing array rather than allocating a new
 one, but that's beside the point. When you call the GC manually, the array is
 still referenced in local variable, if you were to reset that local variable
 before calling the GC, then it should collect it correctly.


He keeps no references to the array, so it should be collected immediatly on
each call to GC.collect(). OR at least, there is no reason for it not to be
collected.

That said, I'm unsure "GC.collect()" promises it'll acutally *do* anything,
it's more of a hint "now would be a good time to run a collection" (AFAIK).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021


Dmitry Olshansky <dmitry.olsh gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dmitry.olsh gmail.com



11:51:16 PDT ---


 import core.memory;
 import std.traits;
 
 void main() {
     while(true) {
         GC.collect();
         new ubyte[40 * 1024 * 1024];
     }
 }
 
 Application's memory usage grows and it crashes with message:
 core.exception.OutOfMemoryError
 
 If i comment import std.traits; everything goes OK, but now application uses
 300 MM of memory.


32bits? Allocating large blocks in a loop? 
Good luck with that... sooner or later they would be pinned by something that
looks like a pointer to GC.  We are back to false pointers and precise
collection topic.

How std.traits affects that is interesting thing, maybe inflates executable
size. Anyway this is game of chance so any slight change may avoid it or bring
it back.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021






 32bits? Allocating large blocks in a loop? 
 Good luck with that... sooner or later they would be pinned by something that
 looks like a pointer to GC.  We are back to false pointers and precise
 collection topic.
 
 How std.traits affects that is interesting thing, maybe inflates executable
 size. Anyway this is game of chance so any slight change may avoid it or bring
 it back.


2 things I don't understand:

1. New initializes, so the block contains nothing but 0x00's. How could that
contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as
pointer?

2. Besides, the GC knows about NO_SCAN, doesn't it? We are explicitly asking
for new ubytes, which is a non-pointer type. Why would the GC scan any of that
at all anywas?

If replace the call with GC.malloc(BIG_SIZE, GC.BlkAttr.NO_SCAN), then my win32
runs forever...

--------

TLDR, I don't see how the false pointer issue in 32 bit environment would
produce this failure.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021




13:22:18 PDT ---




 32bits? Allocating large blocks in a loop? 
 Good luck with that... sooner or later they would be pinned by something that
 looks like a pointer to GC.  We are back to false pointers and precise
 collection topic.
 
 How std.traits affects that is interesting thing, maybe inflates executable
 size. Anyway this is game of chance so any slight change may avoid it or bring
 it back.


 
 2 things I don't understand:
 
 1. New initializes, so the block contains nothing but 0x00's. How could that
 contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as


False pointers that HAPPEN TO POINT INTO THAT BLOCK. I can't imagine we are
having this conversation again. The larger the block the better target it
makes.


 pointer?
 
 2. Besides, the GC knows about NO_SCAN, doesn't it? We are explicitly asking
 for new ubytes, which is a non-pointer type. Why would the GC scan any of that
 at all anywas?
 


Yes it SHOULDN'T scan THAT BLOCK.


 If replace the call with GC.malloc(BIG_SIZE, GC.BlkAttr.NO_SCAN), then my win32
 runs forever...
 


Or change something else and it might not... ? This is an erratic issue.


 --------
 
 TLDR, I don't see how the false pointer issue in 32 bit environment would
 produce this failure.


-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021








 2 things I don't understand:
 
 1. New initializes, so the block contains nothing but 0x00's. How could that
 contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as


 
 False pointers that HAPPEN TO POINT INTO THAT BLOCK. I can't imagine we are
 having this conversation again. The larger the block the better target it
 makes.


Ah. Right.

I apologize for not being there the last time you had this conversation...

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021




It's array of ubytes, not pointers. GC mustn't scan its content.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 13 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11021


safety0ff.bugz gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |safety0ff.bugz gmail.com





 It's array of ubytes, not pointers. GC mustn't scan its content.


I think you misunderstand, something in std.traits gets scanned and it is
treated as a pointer into the array of ubytes.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 03 2013