www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - GC collecting "too much"..

reply simendsjo <simendsjo gmail.com> writes:
I'm doing some coding against a c library, and Ds GC keeps collecting c  
owned objects (I think - disabling the GC makes everything work)
But how can I figure out what the GC is (read: I am) fucking up?
I have some to!string(c_struct_field) and format("%s", c_struct_field) and  
field = c_fields[i] etc etc
Mar 25 2012
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
On Sunday, 25 March 2012 at 19:15:05 UTC, simendsjo wrote:
 I'm doing some coding against a c library, and Ds GC keeps 
 collecting c owned objects (I think - disabling the GC makes 
 everything work)

Three alternative solutions: - Allocate from the C heap the memory that C will need to use, and free it manually or in a struct destructor (RAII) or with scope(exit). - Keep a pointer to the D-GC memory in the D code too. - In core.memory there are ways to disable scanning of a memory zone. Maybe it's usable for your purposes too. Bye, bearophile
Mar 25 2012
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 03/26/2012 11:55 AM, simendsjo wrote:
 It seems threads created in the c library is totally unknown to D. How
 can I make D aware of these threads when there is no library support for
 it?

You may be looking for this: http://dlang.org/phobos/core_thread.html#thread_attachThis
Mar 26 2012
prev sibling next sibling parent simendsjo <simendsjo gmail.com> writes:
On Sun, 25 Mar 2012 22:18:02 +0200, bearophile <bearophileHUGS lycos.com>  
wrote:

 On Sunday, 25 March 2012 at 19:15:05 UTC, simendsjo wrote:
 I'm doing some coding against a c library, and Ds GC keeps collecting c  
 owned objects (I think - disabling the GC makes everything work)

Three alternative solutions: - Allocate from the C heap the memory that C will need to use, and free it manually or in a struct destructor (RAII) or with scope(exit). - Keep a pointer to the D-GC memory in the D code too. - In core.memory there are ways to disable scanning of a memory zone. Maybe it's usable for your purposes too. Bye, bearophile

I've been able to find where the code fails, but now I don't understand what's happening at all. Is calling GC.collect() from an extern(C) function undefined? The following code just starts a mongoose web server and tries to run GC.collect in the handler. extern(C) void* cb(mg_event event, mg_connection* conn, mg_request_info* request_info) { GC.collect(); // segfault return null; } void main() { auto opts = ["listening_ports", "6969"].map!(toUTFz!(char*))().array(); mg_start(&cb, null, cast(const(char**))opts); GC.collect(); // no problem readln(); } If I collect memory from main(), it works as expected (removing the collect from cb of course) while(readln().chomp() != "q") { GC.collect(); } The documentation in core.memory is a bit sparse on how the GC works. Are there any articles on the D GC?
Mar 26 2012
prev sibling next sibling parent simendsjo <simendsjo gmail.com> writes:
On Mon, 26 Mar 2012 10:13:35 +0200, simendsjo <simendsjo gmail.com> wrote:

 On Sun, 25 Mar 2012 22:18:02 +0200, bearophile  
 <bearophileHUGS lycos.com> wrote:

 On Sunday, 25 March 2012 at 19:15:05 UTC, simendsjo wrote:
 I'm doing some coding against a c library, and Ds GC keeps collecting  
 c owned objects (I think - disabling the GC makes everything work)

Three alternative solutions: - Allocate from the C heap the memory that C will need to use, and free it manually or in a struct destructor (RAII) or with scope(exit). - Keep a pointer to the D-GC memory in the D code too. - In core.memory there are ways to disable scanning of a memory zone. Maybe it's usable for your purposes too. Bye, bearophile

I've been able to find where the code fails, but now I don't understand what's happening at all. Is calling GC.collect() from an extern(C) function undefined? The following code just starts a mongoose web server and tries to run GC.collect in the handler. extern(C) void* cb(mg_event event, mg_connection* conn, mg_request_info* request_info) { GC.collect(); // segfault return null; } void main() { auto opts = ["listening_ports", "6969"].map!(toUTFz!(char*))().array(); mg_start(&cb, null, cast(const(char**))opts); GC.collect(); // no problem readln(); } If I collect memory from main(), it works as expected (removing the collect from cb of course) while(readln().chomp() != "q") { GC.collect(); } The documentation in core.memory is a bit sparse on how the GC works. Are there any articles on the D GC?

It seems threads created in the c library is totally unknown to D. How can I make D aware of these threads when there is no library support for it?
Mar 26 2012
prev sibling next sibling parent simendsjo <simendsjo gmail.com> writes:
On Mon, 26 Mar 2012 17:10:34 +0200, Timon Gehr <timon.gehr gmx.ch> wrote:

 On 03/26/2012 11:55 AM, simendsjo wrote:
 It seems threads created in the c library is totally unknown to D. How
 can I make D aware of these threads when there is no library support for
 it?

You may be looking for this: http://dlang.org/phobos/core_thread.html#thread_attachThis

Thanks, but I tried that too and couldn't get it to work. I added the following: extern(C) handler() { synchronized // needed here to avoid the GC to collect while attaching thread? { if(!Thread.getThis()) // thread unknown to D { thread_attachThis(); assert(Thread.getThis()); // now D knows about it } } GC.collect(); // still segfaults } Actually, using attachThis segfaults GC.collect() outside the thread handling code too.
Mar 26 2012
prev sibling next sibling parent simendsjo <simendsjo gmail.com> writes:
On Mon, 26 Mar 2012 20:15:40 +0200, simendsjo <simendsjo gmail.com> wrote:

 On Mon, 26 Mar 2012 17:10:34 +0200, Timon Gehr <timon.gehr gmx.ch> wrote:

 On 03/26/2012 11:55 AM, simendsjo wrote:
 It seems threads created in the c library is totally unknown to D. How
 can I make D aware of these threads when there is no library support  
 for
 it?

You may be looking for this: http://dlang.org/phobos/core_thread.html#thread_attachThis

Thanks, but I tried that too and couldn't get it to work. I added the following: extern(C) handler() { synchronized // needed here to avoid the GC to collect while attaching thread? { if(!Thread.getThis()) // thread unknown to D { thread_attachThis(); assert(Thread.getThis()); // now D knows about it } } GC.collect(); // still segfaults } Actually, using attachThis segfaults GC.collect() outside the thread handling code too.

Some info from gdb: [New Thread 0xf7cfcb70 (LWP 18614)] [New Thread 0xf74fbb70 (LWP 18615)] Program received signal SIGUSR1, User defined signal 1. 0xf7fdd430 in __kernel_vsyscall () (gdb) info threads Id Target Id Frame 3 Thread 0xf74fbb70 (LWP 18615) "main" 0xf7fdd430 in __kernel_vsyscall () 2 Thread 0xf7cfcb70 (LWP 18614) "main" 0xf7fdd430 in __kernel_vsyscall () * 1 Thread 0xf7dfd9c0 (LWP 18611) "main" 0xf7fdd430 in __kernel_vsyscall () (gdb) c Continuing. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xf74fbb70 (LWP 18615)] 0x080855b4 in gc.gcx.Gcx.mark() () (gdb) info threads Id Target Id Frame * 3 Thread 0xf74fbb70 (LWP 18615) "main" 0x080855b4 in gc.gcx.Gcx.mark() () 2 Thread 0xf7cfcb70 (LWP 18614) "main" 0xf7fdd430 in __kernel_vsyscall () 1 Thread 0xf7dfd9c0 (LWP 18611) "main" 0xf7fdd430 in __kernel_vsyscall () ------------ And if I change the code to: extern(C) handler() { GC.collect(); } [New Thread 0xf7cfcb70 (LWP 21454)] [New Thread 0xf74fbb70 (LWP 21455)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xf74fbb70 (LWP 21455)] 0x0808970e in core.thread.thread_suspendAll() () (gdb) info threads Id Target Id Frame * 3 Thread 0xf74fbb70 (LWP 21455) "main" 0x0808970e in core.thread.thread_suspendAll() () 2 Thread 0xf7cfcb70 (LWP 21454) "main" 0xf7fdd430 in __kernel_vsyscall () 1 Thread 0xf7dfd9c0 (LWP 21451) "main" 0xf7fdd430 in __kernel_vsyscall () ----- Any idea where I should go from here?
Mar 29 2012
prev sibling parent simendsjo <simendsjo gmail.com> writes:
On Thu, 29 Mar 2012 14:14:10 +0200, simendsjo <simendsjo gmail.com> wrote:

 On Mon, 26 Mar 2012 20:15:40 +0200, simendsjo <simendsjo gmail.com>  
 wrote:

 On Mon, 26 Mar 2012 17:10:34 +0200, Timon Gehr <timon.gehr gmx.ch>  
 wrote:

 On 03/26/2012 11:55 AM, simendsjo wrote:
 It seems threads created in the c library is totally unknown to D. How
 can I make D aware of these threads when there is no library support  
 for
 it?

You may be looking for this: http://dlang.org/phobos/core_thread.html#thread_attachThis

Thanks, but I tried that too and couldn't get it to work. I added the following: extern(C) handler() { synchronized // needed here to avoid the GC to collect while attaching thread? { if(!Thread.getThis()) // thread unknown to D { thread_attachThis(); assert(Thread.getThis()); // now D knows about it } } GC.collect(); // still segfaults } Actually, using attachThis segfaults GC.collect() outside the thread handling code too.


I compiled with a debug build of druntime, and it fails on dereferencing a pointer, pbot in gcx.d:2511: void mark(void *pbot, void *ptop, int nRecurse) void **p1 = cast(void **)pbot; (...) for (; p1 < p2; p1++) { auto p = cast(byte *)(*p1); // segfault! (...) gdb) print p1 $10 = (void **) 0xf74fc000 (gdb) print *p1 $11 = (void *) 0x0 Some more dbg info: (gdb) bt #0 0x08098753 in gc.gcx.Gcx.mark() (this=0x80df030, nRecurse=64, ptop=0xffffd190, pbot=0xf74f4df4) at src/gc/gcx.d:2511 #1 0x080986d6 in gc.gcx.Gcx.mark() (this=0x80df030, ptop=0xffffd190, pbot=0xf74f4df4) at src/gc/gcx.d:2494 #2 0x0809ef95 in core.thread.thread_scanAll() (this=0xf74f4cf8, p2=0xffffd190, p1=0xf74f4df4, type=<incomplete type>) at src/core/thread.d:2743 #3 0x080a6488 in thread_scanAllType (scan=..., curStackTop=0xf74f4df4) at src/core/thread.d:2698 #4 0x0809ef68 in thread_scanAll (scan=..., curStackTop=0xf74f4df4) at src/core/thread.d:2746 #5 0x08098c34 in gc.gcx.Gcx.fullcollect() (this=0x80df030, stackTop=0xf74f4df4) at src/gc/gcx.d:2776 #6 0x080989f9 in gc.gcx.Gcx.fullcollectshell() (this=0x80df030) at src/gc/gcx.d:2662 #7 0x08096da1 in gc.gcx.GC.fullCollect() (this=0x80df018) at src/gc/gcx.d:1372 #8 0x08086da6 in gc_collect () at src/gc/gc.d:154 #9 0x08085380 in core.memory.GC.collect() () at src/core/memory.d:101 #10 0x08080e2c in c_callback (event=<incomplete type>, conn=0x80ebec0, request_info=0x80ebec0) at main.d:54 #11 0xf7fd28c1 in handle_request(mg_connection*) () from ./libmongoose.so #12 0xf7fd4b7d in worker_thread(mg_context*) () from ./libmongoose.so #13 0xf7f8dd1a in start_thread (arg=0xf74fbb70) at pthread_create.c:304 #14 0xf7ed5e4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further (gdb) l 2511 2506 uint changes = 0; 2507 2508 //printf("marking range: %p -> %p\n", pbot, ptop); 2509 for (; p1 < p2; p1++) 2510 { 2511 auto p = cast(byte *)(*p1); 2512 2513 //if (log) debug(PRINTF) printf("\tmark %p\n", p); 2514 if (p >= minAddr && p < maxAddr) 2515 {
 And if I change the code to:
 extern(C) handler()
 {
      GC.collect();
 }

Checking with debug build of druntime, and I see it tries to suspend(thread_getThis()). This gives a null reference error as thread_getThis() returns null when it's not attached.
Mar 29 2012