www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Get largest heap object at runtime? ...tracking the leak

reply Andres Clari <andres steelcode.net> writes:
Hi, is there any way to get from the GC all allocated objects, so 
I can see their size and find where I'm leaking memory? Or 
perhaps a good tool to help with this issue...

I tried building my program with "profile-gc" but I got an 
invalid MemoryOperationError with no stack trace... so no luck 
using that. And the leak is only obvious when running in 
production, where after 6 hours I've seen 1.5Gb of memory usage.

Right now, I'm able to extend the life of my program by injecting 
calls to:
GC.collect
GCminimize

Within an event that happens every 5 seconds, which I estimate 
will give me about a week before the program's memory is again at 
+1.5Gb.
Jan 21
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Sunday, 21 January 2018 at 17:28:13 UTC, Andres Clari wrote:
 Hi, is there any way to get from the GC all allocated objects, 
 so I can see their size and find where I'm leaking memory? Or 
 perhaps a good tool to help with this issue...

 I tried building my program with "profile-gc" but I got an 
 invalid MemoryOperationError with no stack trace... so no luck 
 using that. And the leak is only obvious when running in 
 production, where after 6 hours I've seen 1.5Gb of memory usage.
Are you by chance using vibe.d? In versions 0.8.x I observe a huge memory leak. Might be your problem.
 Right now, I'm able to extend the life of my program by 
 injecting calls to:
 GC.collect
 GCminimize

 Within an event that happens every 5 seconds, which I estimate 
 will give me about a week before the program's memory is again 
 at +1.5Gb.
Jan 21
parent reply Andres Clari <andres steelcode.net> writes:
On Monday, 22 January 2018 at 06:15:24 UTC, Dmitry Olshansky 
wrote:
 On Sunday, 21 January 2018 at 17:28:13 UTC, Andres Clari wrote:
 Hi, is there any way to get from the GC all allocated objects, 
 so I can see their size and find where I'm leaking memory? Or 
 perhaps a good tool to help with this issue...

 I tried building my program with "profile-gc" but I got an 
 invalid MemoryOperationError with no stack trace... so no luck 
 using that. And the leak is only obvious when running in 
 production, where after 6 hours I've seen 1.5Gb of memory 
 usage.
Are you by chance using vibe.d? In versions 0.8.x I observe a huge memory leak. Might be your problem.
 Right now, I'm able to extend the life of my program by 
 injecting calls to:
 GC.collect
 GCminimize

 Within an event that happens every 5 seconds, which I estimate 
 will give me about a week before the program's memory is again 
 at +1.5Gb.
Yeah. Although after doing some manual tracking and isolation of the suspect routines, I found that using "std.concurrency.spawn" to run a http post request with the "requests" library was the leaking suspect. I'm triggering some 15 requests every 5 seconds in production, so it's obvious there, but querying "GC.usedSize" I was able to observe the same on my dev environment. After removing the "spawn" call, and wrapping up the post code inside a "runTask", it seems to work properly now, and so far I have a consistently small memory size in production. Not sure why "spawn" would leak like that tho. I would assume that once the thread exits, it would get destroyed and it's resources reclaimed, specially when I have calls to "GC.collect and GC.minimize".
Jan 21
parent reply thedeemon <dlang thedeemon.com> writes:
On Monday, 22 January 2018 at 06:48:00 UTC, Andres Clari wrote:
 Not sure why "spawn" would leak like that tho. I would assume 
 that once the thread exits, it would get destroyed and it's 
 resources reclaimed, specially when I have calls to "GC.collect 
 and GC.minimize".
All threads withing a process share the same heap, so whatever one thread allocated in that heap is not freed or reclaimed automatically when thread dies, it stays in the heap.
Jan 22
parent reply Andres Clari <andres steelcode.net> writes:
On Monday, 22 January 2018 at 15:56:53 UTC, thedeemon wrote:
 On Monday, 22 January 2018 at 06:48:00 UTC, Andres Clari wrote:
 Not sure why "spawn" would leak like that tho. I would assume 
 that once the thread exits, it would get destroyed and it's 
 resources reclaimed, specially when I have calls to 
 "GC.collect and GC.minimize".
All threads withing a process share the same heap, so whatever one thread allocated in that heap is not freed or reclaimed automatically when thread dies, it stays in the heap.
Well the destructor of some Json objects and strings should be called when the thread exits tho. Why wouldn’t the GC take care of this?
Jan 22
parent thedeemon <dlang thedeemon.com> writes:
On Monday, 22 January 2018 at 16:37:19 UTC, Andres Clari wrote:
 All threads withing a process share the same heap, so whatever 
 one thread allocated in that heap is not freed or reclaimed 
 automatically when thread dies, it stays in the heap.
Well the destructor of some Json objects and strings should be called when the thread exits tho.
If some objects were on the stack of the thread proc, their destructors run, yes, as usual for leaving a scope. Strings don't have destructors, afaik. Thread-local storage gets cleaned up too. But everything that lives in heap, stays there, until GC finds them. Objects in heap don't have any information on them which thread they belong to.
Jan 22