www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - threading issues with D -> C -> Python

reply "Michael" <triorph gmail.com> writes:
Hi. I'm new here and this is my first post. I'm not sure this is 
the right subforum for it, but wasn't sure where else to put it 
either.

I've written a library to talk to some external hardware using a 
socket. It uses the std.concurrency threads to send messages 
between the main D-object for the hardware and the D-object for 
the sockets. I then wanted to be able to call these functions 
from Python. PyD appeared to be out of date, so I've been using a 
D -> C interface, and a C -> Python interface. The python code 
will often run from different python threads, so I then added yet 
another message-passing layer between the D->C interface and the 
D->hardware interface.

My problem is that this code routinely causes segmentation 
faults. I've spent a long time going through trying to figure out 
exactly what the causes are. I think there have been some related 
to D-exceptions not being handled gracefully by the C/Python 
code. Some more by stdout writing from multiple threads (which 
surprised me).

I'm fairly sure I have tackled both of these issues, but it still 
seems like Python threads and D threads don't mix well. When 
running the same functions from D, I am able to get no errors, 
but when run from Python/C it causes segfaults reliably.

Sorry for the large exposition. I am currently at the point of 
suspecting bugs in Phobos, but I am unskilled enough to tell for 
sure, and would appreciate any help.

The latest core dump gives a backtrace of almost entirely phobos 
commands:

#0  0x00007fe789ad3b97 in gc.gc.Gcx.fullcollect() () from 
/lib/libphobos2.so.0.66
#1  0x00007fe789ad3294 in gc.gc.Gcx.bigAlloc() () from 
/lib/libphobos2.so.0.66
#2  0x00007fe789ad0df1 in gc.gc.GC.mallocNoSync() () from 
/lib/libphobos2.so.0.66
#3  0x00007fe789ad0c15 in gc.gc.GC.malloc() () from 
/lib/libphobos2.so.0.66
#4  0x00007fe789ad6470 in gc_malloc () from 
/lib/libphobos2.so.0.66
#5  0x00007fe789ae6d36 in _d_newitemT () from 
/lib/libphobos2.so.0.66
#6  0x00007fe789e57112 in 
std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from 
/usr/lib/libv5camera.so
#7  0x00007fe789e570b5 in 
std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from 
/usr/lib/libv5camera.so
#8  0x00007fe789e562dc in 
std.array.__T8AppenderTAaZ.Appender.__T3putTAaZ.put() () from 
/usr/lib/libv5camera.so
#9  0x00007fe789e561ea in 
std.array.__T8AppenderTAaZ.Appender.__T3putTxwZ.put() () from 
/usr/lib/libv5camera.so
#10 0x00007fe789e5617d in 
std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() 
() from /usr/lib/libv5camera.so
#11 0x00007fe789e56132 in 
std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() 
() from /usr/lib/libv5camera.so
#12 0x00007fe789e61f09 in 
std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so
#13 0x00007fe789e5b4ac in 
std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so
#14 0x00007fe789e57e8d in 
std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T6__ctorTS3std8typecons24__T5TupleTAyaTiTG655
6kZ5TupleZ.__ctor() 
()
    from /usr/lib/libv5camera.so
#15 0x00007fe789e581f1 in 
std.variant.__T8VariantNVmi32Z.VariantN.__T7handlerTS3std8typecons24__T5TupleTAyaTiTG6553
kZ5TupleZ.handler() 
()
    from /usr/lib/libv5camera.so
#16 0x00007fe789e57d0f in 
std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T8opEqualsTS3std8typecons24__T5TupleTAyaTiTG65536
Z5TupleZ.opEquals() 
()
    from /usr/lib/libv5camera.so
#17 0x00007fe789e57ba8 in 
std.typecons.__T5TupleTAyaTiTG65536kZ.injectNamedFields() () from 
/usr/lib/libv5camera.so
#18 0x00007fe789e62087 in 
std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so
#19 0x00007fe789e621a3 in 
std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so
#20 0x00007fe789e5b7f6 in 
std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so
#21 0x00007fe789ac7d51 in core.thread.Thread.run() () from 
/lib/libphobos2.so.0.66
#22 0x00007fe789ac6f95 in thread_entryPoint () from 
/lib/libphobos2.so.0.66
#23 0x00007fe79cee5182 in start_thread (arg=0x7fe77aca5700) at 
pthread_create.c:312
#24 0x00007fe79cc11fbd in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Cheers,
Michael.
Dec 02 2014
next sibling parent reply ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 01:07:42 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

 I'm fairly sure I have tackled both of these issues, but it still=20
 seems like Python threads and D threads don't mix well. When=20
 running the same functions from D, I am able to get no errors,=20
 but when run from Python/C it causes segfaults reliably.
you are right, D threads and other language/library threads aren't mix well. at least you have to use `thread_attachThis()` and `thread_detachThis()` from core.threads module to make sure that GC is aware of "alien" threads. and i assume that calling this functions from python will not be very easy. but it's better to not mix 'em at all if it is possible.
Dec 02 2014
parent reply "Michael" <triorph gmail.com> writes:
Thanks for this. Its definitely a step in the right direction. 
Would you mind explaining a bit more about the problem here, if 
you can? I don't fully understand why the garbage collector needs 
to know about the threads, and if so for how long does it need to 
know? If I put in 
"thread_attachThis();scope(exit)thread_detachThis();" it doesn't 
appear to fix my problems, so I'm definitely curious as to what 
is going on under the hood.

Cheers,
Michael.

On Wednesday, 3 December 2014 at 01:17:43 UTC, ketmar via 
Digitalmars-d-learn wrote:
 On Wed, 03 Dec 2014 01:07:42 +0000
 Michael via Digitalmars-d-learn 
 <digitalmars-d-learn puremagic.com>
 wrote:

 I'm fairly sure I have tackled both of these issues, but it 
 still seems like Python threads and D threads don't mix well. 
 When running the same functions from D, I am able to get no 
 errors, but when run from Python/C it causes segfaults 
 reliably.
you are right, D threads and other language/library threads aren't mix well. at least you have to use `thread_attachThis()` and `thread_detachThis()` from core.threads module to make sure that GC is aware of "alien" threads. and i assume that calling this functions from python will not be very easy. but it's better to not mix 'em at all if it is possible.
Dec 02 2014
parent reply ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 02:21:45 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

 Thanks for this. Its definitely a step in the right direction.=20
 Would you mind explaining a bit more about the problem here, if=20
 you can? I don't fully understand why the garbage collector needs=20
 to know about the threads, and if so for how long does it need to=20
 know? If I put in=20
 "thread_attachThis();scope(exit)thread_detachThis();" it doesn't=20
 appear to fix my problems, so I'm definitely curious as to what=20
 is going on under the hood.
you have to call `thread_attachThis();` in "alien" thread, not in D thread. i.e. if you created thread from python code, you have to call `thread_attachThis();` in that python thread (i don't know how you'll do that, but you must ;-). and you must call `thread_detachThis();` from the same python thread before exiting from it. garbage collector must know about all running threads so it can scan their stacks, variables and so on. as there is no portable way to set application-wide hooks on thread creation and termination, you must inform GC about that events manually. the other thing you can do is to not use any D allocated data in "alien" threads. i.e. don't pass anything that was allocated by D code to python thread and vice versa. if you want to pass some data to "alien" thread, `malloc()` the necessary space, copy data to it and pass malloc'ed pointer. don't forget to free that data in "alien" thread. but i think that this is not what you really want, as it means alot of allocations and copying, and complicates the whole thing alot. "alien" is the thread that was created outside of D code.
Dec 02 2014
parent reply "Michael" <triorph gmail.com> writes:
On Wednesday, 3 December 2014 at 02:41:11 UTC, ketmar via 
Digitalmars-d-learn wrote:
 On Wed, 03 Dec 2014 02:21:45 +0000
 Michael via Digitalmars-d-learn 
 <digitalmars-d-learn puremagic.com>
 wrote:

 Thanks for this. Its definitely a step in the right direction. 
 Would you mind explaining a bit more about the problem here, 
 if you can? I don't fully understand why the garbage collector 
 needs to know about the threads, and if so for how long does 
 it need to know? If I put in 
 "thread_attachThis();scope(exit)thread_detachThis();" it 
 doesn't appear to fix my problems, so I'm definitely curious 
 as to what is going on under the hood.
you have to call `thread_attachThis();` in "alien" thread, not in D thread. i.e. if you created thread from python code, you have to call `thread_attachThis();` in that python thread (i don't know how you'll do that, but you must ;-). and you must call `thread_detachThis();` from the same python thread before exiting from it. garbage collector must know about all running threads so it can scan their stacks, variables and so on. as there is no portable way to set application-wide hooks on thread creation and termination, you must inform GC about that events manually. the other thing you can do is to not use any D allocated data in "alien" threads. i.e. don't pass anything that was allocated by D code to python thread and vice versa. if you want to pass some data to "alien" thread, `malloc()` the necessary space, copy data to it and pass malloc'ed pointer. don't forget to free that data in "alien" thread. but i think that this is not what you really want, as it means alot of allocations and copying, and complicates the whole thing alot. "alien" is the thread that was created outside of D code.
Okay. Well I am already not passing any D-allocated data. I'm specifically creating variables/arrays on the C-stack, and then passing the pointer of that to D and overwriting the data of the C-stack pointer for any return values. I was worried about that specific problem and I thought this would be a solution. I am then able to tell python to use the C-stack variable without having to worry about D trying to run any garbage collection on it. Going the other way, I probably am passing some python strings etc.. into D, but I would assume they are valid for the lifetime of the function call, and that D would have no reason to try and perform any garbage collection on them.
Dec 02 2014
next sibling parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 02:52:27 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

 Okay. Well I am already not passing any D-allocated data. I'm=20
 specifically creating variables/arrays on the C-stack, and then=20
 passing the pointer of that to D and overwriting the data of the=20
 C-stack pointer for any return values. I was worried about that=20
 specific problem and I thought this would be a solution. I am=20
 then able to tell python to use the C-stack variable without=20
 having to worry about D trying to run any garbage collection on=20
 it.
if D code has any pointer to that data stored anywhere, GC will walk it and hit another thread's stack. and now it doesn't know where it is, and it can't pause that uknown thread so it will not mutate the area GC is scanning now. this *may* work, but it will segfault sooner or later.
 Going the other way, I probably am passing some python strings=20
 etc.. into D, but I would assume they are valid for the lifetime=20
 of the function call, and that D would have no reason to try and=20
 perform any garbage collection on them.
D has conservative GC, so it will try to walk with the unknown data just in case that data contains some pointers. and GC can hit "false positives" there (something that *looks* like a pointer to some area) and other things. so to make the long story short: you should either register and deregister *all* your threads in GC (for D threads it's automatic process; for other threads you must do it manually), or don't use GC at all. besides, if you are using your D library from C code, you must call `rt_init()` once before calling any D code. this function will initialize D runtime. and you have to call `rt_term()` before exiting your program to deinitialize D runtime.
Dec 02 2014
prev sibling parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 02:52:27 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

by "using from C code" i mean that your main program is not written in
D, it has no D `main()` and so on. i.e. you wrote, for example, some
.a library in D and now you want to use that library in C code.
Dec 02 2014
prev sibling next sibling parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 01:07:42 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

all in all, you'd better not mixing D code with "alien" mulththreaded
code and not using .a/.so libraries written in D in another language
until you are familiar with D runtime and GC. those mixes are very
fragile.
Dec 02 2014
prev sibling next sibling parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 01:07:42 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

btw, Adam Ruppe's "D Cookbook" has a chapter which describes how to
call D library from C code. don't remember if it describes threading,
though.
Dec 02 2014
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/02/2014 05:07 PM, Michael wrote:
 Hi. I'm new here and this is my first post. I'm not sure this is the
 right subforum for it, but wasn't sure where else to put it either.

 I've written a library to talk to some external hardware using a socket.
 It uses the std.concurrency threads to send messages between the main
 D-object for the hardware and the D-object for the sockets. I then
 wanted to be able to call these functions from Python. PyD appeared to
 be out of date, so I've been using a D -> C interface, and a C -> Python
 interface. The python code will often run from different python threads,
 so I then added yet another message-passing layer between the D->C
 interface and the D->hardware interface.
are you looking at this pyd: https://bitbucket.org/ariovistus/pyd
Dec 02 2014
parent reply "Michael" <triorph gmail.com> writes:
On Wednesday, 3 December 2014 at 06:11:56 UTC, Ellery Newcomer 
wrote:
 are you looking at this pyd: 
 https://bitbucket.org/ariovistus/pyd
I'm looking at this one, which is what came up when googling "python to D" http://pyd.dsource.org/
Dec 03 2014
parent reply ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 20:41:46 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

 On Wednesday, 3 December 2014 at 06:11:56 UTC, Ellery Newcomer=20
 wrote:
 are you looking at this pyd:=20
 https://bitbucket.org/ariovistus/pyd
=20 I'm looking at this one, which is what came up when googling=20 "python to D" http://pyd.dsource.org/
ah, dsource strikes back! that vile site keep biting us again and again. let's hope that new admins will kill it for good.
Dec 03 2014
parent reply "Michael" <triorph gmail.com> writes:
On Wednesday, 3 December 2014 at 21:35:48 UTC, ketmar via 
Digitalmars-d-learn wrote:
 ah, dsource strikes back! that vile site keep biting us again 
 and
 again. let's hope that new admins will kill it for good.
Yeah. I've got the new PyD and it compiles and does everything I want much nicer, but it appears to have the exact same problems. When calling a python thread to my code, it can cause segfaults and hanging issues. Cheers, Michael.
Dec 03 2014
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/03/2014 04:43 PM, Michael wrote:
 On Wednesday, 3 December 2014 at 21:35:48 UTC, ketmar via
 Digitalmars-d-learn wrote:
 ah, dsource strikes back! that vile site keep biting us again and
 again. let's hope that new admins will kill it for good.
Yeah. I've got the new PyD and it compiles and does everything I want much nicer, but it appears to have the exact same problems. When calling a python thread to my code, it can cause segfaults and hanging issues. Cheers, Michael.
okay. that's not too surprising. If you can get me a minimal example, I'd be happy to have a look since pyd should probably support this case.
Dec 03 2014
parent reply "Michael" <triorph gmail.com> writes:
On Thursday, 4 December 2014 at 02:31:51 UTC, Ellery Newcomer 
wrote:
 okay. that's not too surprising.

 If you can get me a minimal example, I'd be happy to have a 
 look since pyd should probably support this case.
Cool. Unfortunately most of the times I've attempted to reduce this down it always seems to work, but I think that's because I often did the example code in D. I'll play around with it and try to send you an example. Cheers, Michael.
Dec 03 2014
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/03/2014 06:56 PM, Michael wrote:
 On Thursday, 4 December 2014 at 02:31:51 UTC, Ellery Newcomer wrote:
 okay. that's not too surprising.

 If you can get me a minimal example, I'd be happy to have a look since
 pyd should probably support this case.
Cool. Unfortunately most of the times I've attempted to reduce this down it always seems to work, but I think that's because I often did the example code in D. I'll play around with it and try to send you an example. Cheers, Michael.
dustmite?
Dec 03 2014
parent reply "Michael" <triorph gmail.com> writes:
On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer 
wrote:
 dustmite?
Not sure what went wrong with dustmite, but every time I tried it it just started deleting all the files in the directory and setup.py would give errors. I manually deleted a reasonable chunk of the code and I'm left with these files which still seem to cause segfaults: Main code: http://pastebin.com/zqgNTk9w PyD definitions: http://pastebin.com/6mRH3KZZ setup.py: http://pastebin.com/i9Ph78UC test code that causes segfaults: http://pastebin.com/1ukzShVh Cheers, Michael.
Dec 04 2014
next sibling parent "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Thu, Dec 04, 2014 at 10:11:53PM +0000, Michael via Digitalmars-d-learn wrote:
 On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote:
dustmite?
Not sure what went wrong with dustmite, but every time I tried it it just started deleting all the files in the directory and setup.py would give errors.
[...] Nothing is wrong with dustmite; one of its standard techniques to reduce code is to delete source files and see if the problem still happens (i.e., it's independent of that file). What you need to do is to craft a test script such that only the specific error you're looking for will return a success status, and everything else won't. Then when dustmite deletes an essential file, it will know that that was a wrong step and backtrack. T -- Food and laptops don't mix.
Dec 04 2014
prev sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/04/2014 02:11 PM, Michael wrote:
 On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote:
 dustmite?
Not sure what went wrong with dustmite, but every time I tried it it just started deleting all the files in the directory and setup.py would give errors. I manually deleted a reasonable chunk of the code and I'm left with these files which still seem to cause segfaults: Main code: http://pastebin.com/zqgNTk9w PyD definitions: http://pastebin.com/6mRH3KZZ setup.py: http://pastebin.com/i9Ph78UC test code that causes segfaults: http://pastebin.com/1ukzShVh Cheers, Michael.
hmm.. looks like here it originates in python when it tries to acquire the GIL. specifically, pthread_cond_timedwait is segfaulting. in your code, execution inside a python thread makes it to receiveTimeout in get_image. it made it past receiveTimeout in acquire. then I guess there is a context switch. the main python thread throws an exception, but a number of things trigger the segfault. I think it's just the interpreter loop calling RestoreThread. backtrace looks like #0 pthread_cond_timedwait GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000003799d07bb3 in PyCOND_TIMEDWAIT (cond=0x379a063220 <gil_cond>, mut=0x379a0631e0 <gil_mutex>, us=5000) at /usr/src/debug/Python-3.3.2/Python/condvar.h:103 #2 take_gil (tstate=tstate entry=0x604410) at /usr/src/debug/Python-3.3.2/Python/ceval_gil.h:224 #3 0x0000003799d081fb in PyEval_RestoreThread (tstate=tstate entry=0x604410) ... It looks like this is the python main thread. I see two other threads. (i took out one of your python spawns) #2 looks to be your listener thread. std.concurrency.send seems to have gotten it into a gc_malloc, but it looks like it's just waiting. it's currently in sem_wait. this one would have been spawned in D code by #3 #3 is your other python thread. it is also in pthread_cond_timedwait. by its stack trace, receiveTimeout is just waiting. I guess tomorrow I can try messing around with thread_attachThis, as the fullcollect happening in #2 might be screwing with python data. But you aren't really passing anything from python to d or vice versa, so I'm not sure why the gc would need to know about the python threads. not exactly my area of expertise, this.
Dec 04 2014
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/04/2014 10:55 PM, Ellery Newcomer wrote:

 I guess tomorrow I can try messing around with thread_attachThis, as the
 fullcollect happening in #2 might be screwing with python data. But you
 aren't really passing anything from python to d or vice versa, so I'm
 not sure why the gc would need to know about the python threads.
by gum, thread_attachThis and thread_detachThis fix the issue! now to figure out how to use them in the general case.
Dec 05 2014
parent reply "Michael" <triorph gmail.com> writes:
On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer 
wrote:
 On 12/04/2014 10:55 PM, Ellery Newcomer wrote:

 I guess tomorrow I can try messing around with 
 thread_attachThis, as the
 fullcollect happening in #2 might be screwing with python 
 data. But you
 aren't really passing anything from python to d or vice versa, 
 so I'm
 not sure why the gc would need to know about the python 
 threads.
by gum, thread_attachThis and thread_detachThis fix the issue! now to figure out how to use them in the general case.
This is great.. Thank you. I'm looking forward to being able to try the finished result.
Dec 07 2014
next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/07/2014 03:12 PM, Michael wrote:
 On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote:
 On 12/04/2014 10:55 PM, Ellery Newcomer wrote:

 I guess tomorrow I can try messing around with thread_attachThis, as the
 fullcollect happening in #2 might be screwing with python data. But you
 aren't really passing anything from python to d or vice versa, so I'm
 not sure why the gc would need to know about the python threads.
by gum, thread_attachThis and thread_detachThis fix the issue! now to figure out how to use them in the general case.
This is great.. Thank you. I'm looking forward to being able to try the finished result.
It would be great if there were some convenient hook in python to stick these calls. Near as I can tell, there isn't. That leaves you with either explicitly calling attach and detach with an exposed api, or pyd obsessively checking whether the current thread is registered. Actually, I suppose with a thread local flag the latter wouldn't be too bad. Mind if I incorporate your example into pyd's test suite?
Dec 07 2014
parent "Michael" <triorph gmail.com> writes:
On Monday, 8 December 2014 at 01:17:16 UTC, Ellery Newcomer wrote:
 On 12/07/2014 03:12 PM, Michael wrote:
 On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer 
 wrote:
 On 12/04/2014 10:55 PM, Ellery Newcomer wrote:

 I guess tomorrow I can try messing around with 
 thread_attachThis, as the
 fullcollect happening in #2 might be screwing with python 
 data. But you
 aren't really passing anything from python to d or vice 
 versa, so I'm
 not sure why the gc would need to know about the python 
 threads.
by gum, thread_attachThis and thread_detachThis fix the issue! now to figure out how to use them in the general case.
This is great.. Thank you. I'm looking forward to being able to try the finished result.
It would be great if there were some convenient hook in python to stick these calls. Near as I can tell, there isn't. That leaves you with either explicitly calling attach and detach with an exposed api, or pyd obsessively checking whether the current thread is registered. Actually, I suppose with a thread local flag the latter wouldn't be too bad. Mind if I incorporate your example into pyd's test suite?
Not at all. Go for it.
Dec 08 2014
prev sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/07/2014 03:12 PM, Michael wrote:
 now to figure out how to use them in the general case.
This is great.. Thank you. I'm looking forward to being able to try the finished result.
My build servers are broken at the moment, but I think I have this fixed, on linux at least.
Dec 17 2014
prev sibling parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 2014-12-03 at 01:07 +0000, Michael via Digitalmars-d-learn
wrote:
 Hi. I'm new here and this is my first post. I'm not sure this is=20
 the right subforum for it, but wasn't sure where else to put it=20
 either.
=20
 I've written a library to talk to some external hardware using a=20
 socket. It uses the std.concurrency threads to send messages=20
 between the main D-object for the hardware and the D-object for=20
 the sockets. I then wanted to be able to call these functions=20
 from Python. PyD appeared to be out of date, so I've been using a=20
As far as I can tell PyD is still active, but in a non-funded FOSS way, i.e. work happens as and when volunteers put time and effort in. I haven't tried PyD recently but it worked fine last time I did. If can set out what you tried and what didn't work, maybe there is a PyD solution, or a fix to PyD to give a solution?
 D -> C interface, and a C -> Python interface. The python code=20
 will often run from different python threads, so I then added yet=20
 another message-passing layer between the D->C interface and the=20
 D->hardware interface.
D's "big problem" is shared objects/dynamic link libraries. Without them you cannot interwork with Python at all. I have tried experiments on Linux creating shared libraries from D code with C linkage entry points to create classic Python extensions, and it appears to work fine. Except for having to start up the D heap and thread management, should they be needed. But that is what PyD is there for. If I took my experiments any further I would end up recreating PyD or something like it. It sounds like you are in a similar situation except that you appear to have an extra layer of C code. I am not sure a layer of C is needed between Python and D, it would be good to know more about why you seem to need it. Your use case is interesting as I have more or less given up on using D in a Python context. CPython naturally requires C linkage shared objects for non-Python code so C, C++ or D would be fine except that everyone does it in C or C++, very few people in the arena have even heard of D. PyPy brings it's own issues with non-Python code but the now have a C capability. With Cython, Pythran, and more recently Numba there is increasing less and less need for any user written non-Python code. Hardware control would be done in C and PyPy now has a C linkage mechanism. Of course for Python networking there is Twisted, Asyncio and Tornado so no Python folk are using non-Python code for handling networking.
 My problem is that this code routinely causes segmentation=20
 faults. I've spent a long time going through trying to figure out=20
 exactly what the causes are. I think there have been some related=20
 to D-exceptions not being handled gracefully by the C/Python=20
 code. Some more by stdout writing from multiple threads (which=20
 surprised me).
=20
 I'm fairly sure I have tackled both of these issues, but it still=20
 seems like Python threads and D threads don't mix well. When=20
 running the same functions from D, I am able to get no errors,=20
 but when run from Python/C it causes segfaults reliably.
Without seeing your code, it is difficult to say what may or may not be the problem, but I would guess it is about D infrastructure start up. I recollect being able to reliably get segfaults this way. Are you using ctypes or CFFI on the Python side?
 Sorry for the large exposition. I am currently at the point of=20
 suspecting bugs in Phobos, but I am unskilled enough to tell for=20
 sure, and would appreciate any help.
=20
 The latest core dump gives a backtrace of almost entirely phobos=20
 commands:
=20
 #0  0x00007fe789ad3b97 in gc.gc.Gcx.fullcollect() () from=20
 /lib/libphobos2.so.0.66
 #1  0x00007fe789ad3294 in gc.gc.Gcx.bigAlloc() () from=20
 /lib/libphobos2.so.0.66
 #2  0x00007fe789ad0df1 in gc.gc.GC.mallocNoSync() () from=20
 /lib/libphobos2.so.0.66
 #3  0x00007fe789ad0c15 in gc.gc.GC.malloc() () from=20
 /lib/libphobos2.so.0.66
 #4  0x00007fe789ad6470 in gc_malloc () from=20
 /lib/libphobos2.so.0.66
 #5  0x00007fe789ae6d36 in _d_newitemT () from=20
 /lib/libphobos2.so.0.66
 #6  0x00007fe789e57112 in=20
 std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from=20
 /usr/lib/libv5camera.so
 #7  0x00007fe789e570b5 in=20
 std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from=20
 /usr/lib/libv5camera.so
 #8  0x00007fe789e562dc in=20
 std.array.__T8AppenderTAaZ.Appender.__T3putTAaZ.put() () from=20
 /usr/lib/libv5camera.so
 #9  0x00007fe789e561ea in=20
 std.array.__T8AppenderTAaZ.Appender.__T3putTxwZ.put() () from=20
 /usr/lib/libv5camera.so
 #10 0x00007fe789e5617d in=20
 std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.format=
Char()=20
 () from /usr/lib/libv5camera.so
 #11 0x00007fe789e56132 in=20
 std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.format=
Char()=20
 () from /usr/lib/libv5camera.so
 #12 0x00007fe789e61f09 in=20
 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get()=
=20
 () from /usr/lib/libv5camera.so
 #13 0x00007fe789e5b4ac in=20
 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ=
.get()=20
 () from /usr/lib/libv5camera.so
 #14 0x00007fe789e57e8d in=20
 std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T6__ctorTS3std8typecons24__=
T5TupleTAyaTiTG65536kZ5TupleZ.__ctor()=20
 ()
     from /usr/lib/libv5camera.so
 #15 0x00007fe789e581f1 in=20
 std.variant.__T8VariantNVmi32Z.VariantN.__T7handlerTS3std8typecons24__T5T=
upleTAyaTiTG65536kZ5TupleZ.handler()=20
 ()
     from /usr/lib/libv5camera.so
 #16 0x00007fe789e57d0f in=20
 std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T8opEqualsTS3std8typecons24=
__T5TupleTAyaTiTG65536kZ5TupleZ.opEquals()=20
 ()
     from /usr/lib/libv5camera.so
 #17 0x00007fe789e57ba8 in=20
 std.typecons.__T5TupleTAyaTiTG65536kZ.injectNamedFields() () from=20
 /usr/lib/libv5camera.so
 #18 0x00007fe789e62087 in=20
 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get()=
=20
 () from /usr/lib/libv5camera.so
 #19 0x00007fe789e621a3 in=20
 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get()=
=20
 () from /usr/lib/libv5camera.so
 #20 0x00007fe789e5b7f6 in=20
 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ=
.get()=20
 () from /usr/lib/libv5camera.so
 #21 0x00007fe789ac7d51 in core.thread.Thread.run() () from=20
 /lib/libphobos2.so.0.66
 #22 0x00007fe789ac6f95 in thread_entryPoint () from=20
 /lib/libphobos2.so.0.66
 #23 0x00007fe79cee5182 in start_thread (arg=3D0x7fe77aca5700) at=20
 pthread_create.c:312
 #24 0x00007fe79cc11fbd in clone () at=20
 ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
My guess would be not properly initializing the D infrastructure from the incoming Python thread. I would suggest that you want to avoid threads crossing the boundaries and just pass data via a shared channel. Unix pipes seem to work well in this context since they provide a language independent data channel. =20 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Dec 02 2014
parent "Michael" <triorph gmail.com> writes:
On Wednesday, 3 December 2014 at 06:30:07 UTC, Russel Winder via 
Digitalmars-d-learn wrote:
 As far as I can tell PyD is still active, but in a non-funded 
 FOSS way,
 i.e. work happens as and when volunteers put time and effort 
 in. I
 haven't tried PyD recently but it worked fine last time I did. 
 If can
 set out what you tried and what didn't work, maybe there is a 
 PyD
 solution, or a fix to PyD to give a solution?
Yeah apparently I might have used the wrong PyD. This might end up being all I need to do to fix my problem.
 D's "big problem" is shared objects/dynamic link libraries. 
 Without them
 you cannot interwork with Python at all. I have tried 
 experiments on
 Linux creating shared libraries from D code with C linkage 
 entry points
 to create classic Python extensions, and it appears to work 
 fine. Except
 for having to start up the D heap and thread  management, 
 should they be
 needed. But that is what PyD is there for. If I took my 
 experiments any
 further I would end up recreating PyD or something like it.
This is what I'm doing. I'm using the rt_init() function to setup the heap/thread management. Am I missing anything else here? It seems the issue is definitely more complicated than that. I'll also point out that: http://dlang.org/interfaceToC had no information on calling rt_init first, although it seems like it should.
 It sounds like you are in a similar situation except that you 
 appear to
 have an extra layer of C code. I am not sure a layer of C is 
 needed
 between Python and D, it would be good to know more about why 
 you seem
 to need it.
Well I wanted to compile D code and directly call it in Python, and since PyD didn't work for me, I instead tried the python-> C interface I already knew (https://docs.python.org/2/c-api/index.html) and worked my way through a C->D interface, which I understood to be relatively simple.
 My guess would be not properly initializing the D 
 infrastructure from
 the incoming Python thread.

 I would suggest that you want to avoid threads crossing the 
 boundaries
 and just pass data via a shared channel. Unix pipes seem to 
 work well in
 this context since they provide a language independent data 
 channel.
Yeah I'm leaning in that direction myself, although I might try the other PyD library first. I wanted to be able to use the D message-passing libraries to do the thread-safety stuff at first, because it was much easier than the alternative, but I'm not sure that's true anymore.
Dec 03 2014