www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Garbage collector noob

reply torhu <fake address.dude> writes:
I haven't dealt directly with the garbage collector much before, but 
just used delete and scope to deal with deallocation.  And left some of 
the minor allocations to be freed by the GC.

Now I want to work more closely with the GC.  Can anyone explain to me 
why fullCollect doesn't seem to do anything here?  This example just 
keeps allocating memory, until the GC seems to kick in to stop it from 
growing.  It stabilizes at 426MB.  The same example using Tango instead 
stops at 597MB.

---
import std.gc;

void main()
{
     int[] a;

     for (;;) {
         a = new int[1024 * 1024 * 10];
         a = null;
         assert(a.ptr is null);
         fullCollect();
     }
}
---

Removing the call to fullCollect() doesn't change the behavior.  I 
expected fullCollect to collect as much memory as it can, and then reuse 
it for the next allocation.  Am I on the wrong track here?
Mar 06 2007
next sibling parent reply Sean Kelly <sean f4.ca> writes:
torhu wrote:
 I haven't dealt directly with the garbage collector much before, but 
 just used delete and scope to deal with deallocation.  And left some of 
 the minor allocations to be freed by the GC.
 
 Now I want to work more closely with the GC.  Can anyone explain to me 
 why fullCollect doesn't seem to do anything here?  This example just 
 keeps allocating memory, until the GC seems to kick in to stop it from 
 growing.  It stabilizes at 426MB.  The same example using Tango instead 
 stops at 597MB.

When I run the test on Tango, memory juse jumps to 467k and stays steady there. Doing some quick math with a calculator shows an array 1024^2*10 integers should occupy ~409k of memory, so the rest is just program overhead. For what it's worth, running this app on Tango without the call to gc.collect() brings memory use up to 594k. I'd have to debug the GC to figure out why.
 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then reuse 
 it for the next allocation.  Am I on the wrong track here?

Nope. That's what it should do. But be aware that the current GC allocates memory in pages and those pages are devoted to a specific size of data. And once so assigned, I don't think the GC ever attempts to reclaim empty pages if another size allocation is required. It doesn't matter in this case because anything over a page (4k) is the same "size" as far as the GC is concerned, but it may matter in an app that allocates a ton of small objects and discards them, then allocates a ton of larger objects and discards them, etc. Sean
Mar 06 2007
parent reply torhu <fake address.dude> writes:
Sean Kelly wrote:
 When I run the test on Tango, memory juse jumps to 467k and stays steady 
 there.  Doing some quick math with a calculator shows an array 1024^2*10 
 integers should occupy ~409k of memory, so the rest is just program 
 overhead.  For what it's worth, running this app on Tango without the 
 call to gc.collect() brings memory use up to 594k.  I'd have to debug 
 the GC to figure out why.

Uh... but 1024^2*10 ints is 40 MB exactly. Are you talking about a different test program?
 
 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then reuse 
 it for the next allocation.  Am I on the wrong track here?

Nope. That's what it should do. But be aware that the current GC allocates memory in pages and those pages are devoted to a specific size of data. And once so assigned, I don't think the GC ever attempts to reclaim empty pages if another size allocation is required. It doesn't matter in this case because anything over a page (4k) is the same "size" as far as the GC is concerned, but it may matter in an app that allocates a ton of small objects and discards them, then allocates a ton of larger objects and discards them, etc.

Are you on Windows or Linux? I'm on windows, and I'm not seeing the behavior you describe. The test I posted allocates the same amount of memory over and over again. But the GC doesn't seem to start reusing it until somewhere around the eleventh time through the loop. Until then it just grows. Just for the record, the number I'm looking at is 'private bytes' in process explorer, which is the same as 'VM size' in task manager. Which shows allocated heap memory, and probably some other stuff. And I use dmd 1.007.
Mar 07 2007
parent reply Sean Kelly <sean f4.ca> writes:
torhu wrote:
 Sean Kelly wrote:
 When I run the test on Tango, memory juse jumps to 467k and stays 
 steady there.  Doing some quick math with a calculator shows an array 
 1024^2*10 integers should occupy ~409k of memory, so the rest is just 
 program overhead.  For what it's worth, running this app on Tango 
 without the call to gc.collect() brings memory use up to 594k.  I'd 
 have to debug the GC to figure out why.

Uh... but 1024^2*10 ints is 40 MB exactly. Are you talking about a different test program?

No, you're right. I misread the output to contain one more zero than it does.
 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then 
 reuse it for the next allocation.  Am I on the wrong track here?

Nope. That's what it should do. But be aware that the current GC allocates memory in pages and those pages are devoted to a specific size of data. And once so assigned, I don't think the GC ever attempts to reclaim empty pages if another size allocation is required. It doesn't matter in this case because anything over a page (4k) is the same "size" as far as the GC is concerned, but it may matter in an app that allocates a ton of small objects and discards them, then allocates a ton of larger objects and discards them, etc.

Are you on Windows or Linux? I'm on windows, and I'm not seeing the behavior you describe. The test I posted allocates the same amount of memory over and over again. But the GC doesn't seem to start reusing it until somewhere around the eleventh time through the loop. Until then it just grows.

Windows. The app jumps to 467k almost immediately and stays there. I haven't tried stepping through the loop, so it may just be happening too fast for me to see.
 Just for the record, the number I'm looking at is 'private bytes' in 
 process explorer, which is the same as 'VM size' in task manager.  Which 
 shows allocated heap memory, and probably some other stuff.  And I use 
 dmd 1.007.

Thanks, I'll check that number to be sure. Sean
Mar 07 2007
parent torhu <fake address.dude> writes:
Sean Kelly wrote:
 Windows.  The app jumps to 467k almost immediately and stays there.  I 
 haven't tried stepping through the loop, so it may just be happening too 
 fast for me to see.

I assume you mean 467MB? Anyway, the point is not how fast it happens. The problem is that it happens at all. I expected to be able to use the garbage collector to keep the test app's memory at ~40MB. If std.gc.minimize doesn't do that, then how is it different from std.gc.fullCollect? By the way, the limit for how much memory size grows when running the test app depends on the size of the array. So the GC seems to set a limit based on the app's typical allocation sizes. Which makes sense and probably keeps allocation speeds up. It might be that I should start viewing the GC more as a leak preventor than a simpler way of keeping memory usage down. Simpler than explicit deallocation, that is. The GC might decide to let memory usage grow to what the programmer or the user consider unreasonable sizes. So basically, you should always use delete or scope to free memory, since when the GC does it, it will be a bit late.
Mar 07 2007
prev sibling next sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
torhu wrote:
 I haven't dealt directly with the garbage collector much before, but 
 just used delete and scope to deal with deallocation.  And left some of 
 the minor allocations to be freed by the GC.
 
 Now I want to work more closely with the GC.  Can anyone explain to me 
 why fullCollect doesn't seem to do anything here?  This example just 
 keeps allocating memory, until the GC seems to kick in to stop it from 
 growing.  It stabilizes at 426MB.  The same example using Tango instead 
 stops at 597MB.
 
 ---
 import std.gc;
 
 void main()
 {
     int[] a;
 
     for (;;) {
         a = new int[1024 * 1024 * 10];
         a = null;
         assert(a.ptr is null);
         fullCollect();
     }
 }
 ---
 
 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then reuse 
 it for the next allocation.  Am I on the wrong track here?

Not being exactly an expert on garbage collection myself, I did encounter the same behaviour which could be changed with a call to std.gc.minimize. This led me to think fullCollect does in fact not always clean up all unused memory, but just forces a GC scan, which may decide there is no need to collect if enough memory is available. Can anybody confirm this?
Mar 07 2007
parent torhu <fake address.dude> writes:
Lutger wrote:
 torhu wrote:
 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then reuse 
 it for the next allocation.  Am I on the wrong track here?

Not being exactly an expert on garbage collection myself, I did encounter the same behaviour which could be changed with a call to std.gc.minimize. This led me to think fullCollect does in fact not always clean up all unused memory, but just forces a GC scan, which may decide there is no need to collect if enough memory is available. Can anybody confirm this?

I tried minimize, but now the memory usage goes even higher, up to 469 MB instead of 426. So none of these functions seem to do what the docs say. Same goes for tango.core.Memory.gc.collect().
Mar 07 2007
prev sibling next sibling parent reply Paul Findlay <r.lph50+d gmail.com> writes:
Unfortunately this is unlikely to help but..

 This example just keeps allocating memory,  until the GC seems to kick
 in to stop it from growing. It stabilizes at 426MB.

used instead of fullCollect the memory usage stabilised at ~93MB This is with 512MB of total RAM - Paul
Mar 08 2007
parent reply torhu <fake address.dude> writes:
Paul Findlay wrote:
 Unfortunately this is unlikely to help but..

No, it's very interesting to hear what results other people get.
 Using plain phobos, my testing stabilises at ~51MB. When minimize was 
 used instead of fullCollect the memory usage stabilised at ~93MB
 
 This is with 512MB of total RAM

Ok, I have 896 MB. I have a feeling that doesn't affect the GC, but I can't be sure. I use dmd 1.007 on winxp sp2. What are you using?
Mar 08 2007
parent Paul Findlay <r.lph50+d gmail.com> writes:
 I use dmd 1.007 on winxp sp2.  What are you using?

My virtual memory is 768MB (I have no idea if this is revelevant, but in my spurious thinking, it may effect how the GC gives memory back to the OS) - Paul
Mar 08 2007
prev sibling parent reply Derek Parnell <derek psych.ward> writes:
On Wed, 07 Mar 2007 06:15:16 +0100, torhu wrote:

 I haven't dealt directly with the garbage collector much before, but 
 just used delete and scope to deal with deallocation.  And left some of 
 the minor allocations to be freed by the GC.
 
 Now I want to work more closely with the GC.  Can anyone explain to me 
 why fullCollect doesn't seem to do anything here? 

There is quite a large portion of the fullCollect source which is commented out in phobos, mainly in the section which reclaims RAM (I think). -- Derek Parnell Melbourne, Australia "Justice for David Hicks!" skype: derek.j.parnell
Mar 08 2007
parent reply torhu <fake address.dude> writes:
Derek Parnell wrote:
 On Wed, 07 Mar 2007 06:15:16 +0100, torhu wrote:
 Now I want to work more closely with the GC.  Can anyone explain to me 
 why fullCollect doesn't seem to do anything here? 

There is quite a large portion of the fullCollect source which is commented out in phobos, mainly in the section which reclaims RAM (I think).

Do you mean gcx.d, line 1619? That would explain some things. There's also something else of interest in that file: void minimize() // minimize physical memory usage { // Not implemented, ignore } Is this really the one being called as std.gc.minimize? Looks like it. I wish I had known this before. I guess I should file a bug report; either implement it or make the docs state that it is currently not implemented. If I knew what fullCollect() really was supposed to do, I could include that in the bug report too. I'm hoping someone with a better understanding of this than me can help out here.
Mar 08 2007
parent reply Sean Kelly <sean f4.ca> writes:
torhu wrote:
 Derek Parnell wrote:
 On Wed, 07 Mar 2007 06:15:16 +0100, torhu wrote:
 Now I want to work more closely with the GC.  Can anyone explain to 
 me why fullCollect doesn't seem to do anything here? 

There is quite a large portion of the fullCollect source which is commented out in phobos, mainly in the section which reclaims RAM (I think).

Do you mean gcx.d, line 1619? That would explain some things. There's also something else of interest in that file: void minimize() // minimize physical memory usage { // Not implemented, ignore } Is this really the one being called as std.gc.minimize? Looks like it. I wish I had known this before. I guess I should file a bug report; either implement it or make the docs state that it is currently not implemented. If I knew what fullCollect() really was supposed to do, I could include that in the bug report too. I'm hoping someone with a better understanding of this than me can help out here.

fullCollect performs a full collection of orphaned memory. With the current GC, this memory is recovered for future use by the GC, but it not returned to the OS.
Mar 08 2007
parent reply torhu <fake address.dude> writes:
Sean Kelly wrote:
 fullCollect performs a full collection of orphaned memory.  With the 
 current GC, this memory is recovered for future use by the GC, but it 
 not returned to the OS.

It's supposed to recover memory for future use, but it doesn't do that. Is that a correct description of the current situation? I'm not sure what you're really saying, you seem to imply that it does actually does recover memory. With my test app it sure doesn't reuse it, at least not for the next allocation that is made.
Mar 08 2007
parent Sean Kelly <sean f4.ca> writes:
torhu wrote:
 With my test app it sure doesn't reuse it, at least not 
 for the next allocation that is made.

Yeah. This is weird because I'd expect it to, but then I didn't write the GC. When I get some time I'm going to debug the thing and see what's going on.
Mar 08 2007