www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 5326] New: GC -- 99% CPU in gc gcx Gcx mark()

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5326

           Summary: GC -- 99% CPU in gc gcx Gcx mark()
           Product: D
           Version: D2
          Platform: x86_64
        OS/Version: Windows
            Status: NEW
          Severity: normal
          Priority: P2
         Component: druntime
        AssignedTo: sean invisibleduck.org
        ReportedBy: wfunction hotmail.com



I was running a piece of code that allocated a huge block of memory at once,
then allocated small chunks after that inside a block, deleting them at the end
of the scope. I managed to get the code down to this:

auto temp = new void[0x8080 * 4096];
for (int i = 0; i < 0x8080 * 4; i++)
{ scope auto temp2 = new void[0x10100]; }

What happens is, after the first few ten thousands of collections (for me, it
was around after 62,000 collections), the time spent in the mark() method
shoots up to around 99% of the application's time, and the program effectively
hangs at the allocation.

The numbers above were specific to my system; they're rather arbitrary. All you
need to do is:
(1) Run the code, and break into the process before around 1 second. Notice
that, when you do this, it's often inside memset() or some other function, and
not the GC.
(2) Run the code, and break into the process after several seconds have passed.
At this time, it's practically guaranteed that you're in the mark() method --
and in fact, for me, it froze my application.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 05 2010
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5326


nfxjfg gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |nfxjfg gmail.com



That's somewhat excpected. The mark() function is the core of the garbage
collector, where it basically follows all references in the program's heap.

void[] arrays can potentially contain references, so the GC has to mark() these
as well. (Why are you using void[]? 10 dollars on you have no real reason.)
What makes things worse: each time there's a pointer into a large memory block,
the GC has to do a linear scan to find the beginning of the memory block. And
you are alocating LARGE blocks (32k pages!), so that might become significant.

I think what's happening is that your heap is exploding due to false pointers.
The memory allocations are so large that there's always something that looks
like a pointer pointing into it, so it never gets free'd. A larger heap means
higher collection times.

So, this is probably not a bug.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 06 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5326




PS: scope doesn't work on arrays. Your example code never explicitly free's the
memory.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 06 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5326





 PS: scope doesn't work on arrays. Your example code never explicitly free's the


memory.

That explains everything... but then, how do we delete arrays?



 void[] arrays can potentially contain references, so the GC has to mark() these


as well.

I had NO IDEA that was the case!! (see below)



 (Why are you using void[]? 10 dollars on you have no real reason.)


I actually did have a reason, but I was misunderstanding the situation. I used
void[]'s in places where I did not want ubyte[]s to be interpreted as numbers,
so that they would not be dereferenceable. (To me, a void[] could contain
binary *data*, whereas a ubyte[] logically means that it contains numbers --
and while it's OK to reference a number in a list, it's *not* OK to me to just
randomly reference an arbitrary block of data, which might contain code inside
a buffer.)
Of course, I wasn't understanding their meaning correctly, so I won't do that
anymore; thank you very much for the explanation!!

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 10 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5326






 PS: scope doesn't work on arrays. Your example code never explicitly free's the


 memory.
 
 That explains everything... but then, how do we delete arrays?


You can free heap allocated arrays by "delete somearray;" (deprecated in D2).

I'm not sure what's the proper way of deleting an array is. As you can see in
the implementation for "delete <array>;", it's not that trivial:
http://www.dsource.org/projects/druntime/browser/trunk/src/rt/lifetime.d#L937
PS: That code may contain a bug. It may pass a pointer to gc_free() that is not
the start of a memory block, but gc_free() could have trouble with that and
have undefined effects on the GC heap integrity, but I digress.

When you allocate a class using scope on classes, it is normally allocated on
the stack. Doing this with arrays is hard; you need to use alloca() and turn
the resulting void* into an array...

Also, the idea with void[] was probably that it could contain *anything* that
can exist in memory. Thus the GC has to assume that it may contain pointers.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 10 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5326




Ah, all right, thank you for the information!

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 11 2010
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5326


Brad Roberts <braddr puremagic.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Platform|x86_64                      |x86



---
Mass migration of bugs marked as x86-64 to just x86.  The platform run on isn't
what's relevant, it's if the app is a 32 or 64 bit app.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Feb 06 2011