www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Embedding D Shared Library in WSGI Web Server

reply "John McFarlane" <dlang john.mcfarlane.name> writes:
I posted this question a while back in D.learn 
[http://forum.dlang.org/thread/rbozdetvuepfeftxkbac forum.dlang.org] 
but figured it might be more appropriate to ask here...

I've written a modest shared library in D that I'd like to call
directly from a Python web server (Linux/OS X, Apache, WSGI,
Pyramid). I can call it reliably from within Python unit tests
but on a running server, the library causes a SIGSEGV as soon as
I try anything as complicated as a writeln call. Under Linux,
I've tried using Pyd, CFFI and ctypes to bind the native .so to
Python with the same result. I've tried calling
Runtime.initialize() from an init function on server startup and
from the D function being called as part of the web request. I've
even tried writing a shim in C to make these calls and call
through to the D.

I've compiled with DMD and GDC with a variety of switches.
Currently my build command looks like:

     dmd ./d/*.d -version=library -release -shared -O -inline
-defaultlib=libphobos2.so -fPIC -oflibgusteau.so

The extern(C) function takes a string and returns a string. I've
tried holding onto a pointer to the returned string and passing
in an adequately sized byte array from Python. Smaller strings
seem to cause a crash with lower probability but typically the
input and output strings are a few KB in size and consistently
crash. I taken measures to ensure that they are correctly encoded
and null terminated. I've also tried disabling the GC, calling
terminate on function exit (after copying result into received
buffer) and various other measures in an attempt to get something
working.

I'm guessing that the web server spawns/relies on a thread or
process for each request and I've read that runtime
initialization should be invoked from the main thread. Does this
always mean the first thread that was created on process start up
or does it just have to be consistent? If calling from a thread
other than the one used to initialize, is that a problem? Are
other threads created when GC is invoked which might last past
the extern function being called and if so, can I prevent this by
controlling collection explicitly?

Thanks, John
Nov 07 2014
parent reply "Laeeth Isharc" <laeethnospam spammenot_laeeth.com> writes:
since you didn't get an answer.

https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading


you can turn off threads and processes to aid debugging:

https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading


StartServers 1
ServerLimit 1

With this configuration, only one process will be started, with 
no additional processes ever being created. The WSGI environment 
key/value pairs indicating how processes and threads are being 
used will for this configuration be as follows.

wsgi.multithread	False
wsgi.multiprocess	False

In effect, this configuration has the result of serialising all 
requests through a single process. This will allow an interactive 
browser based debugger to be used, but may prevent more complex 
WSGI applications which make use of AJAX techniques from working. 
This could occur where a web page initiates a sequence of AJAX 
requests and expects later requests to be able to complete while 
a response for an initial request is still pending. In other 
words, problems may occur where requests overlap, as subsequent 
requests will not be able to be executed until the initial 
request has completed.
-----
although obviously that is not a solution to your problem, it 
might help pinpoint it.

did you try tracking init state in your library and have entry 
points call D runtime if not already initialized?
Nov 07 2014
parent "John McFarlane" <dlang john.mcfarlane.name> writes:
On Friday, 7 November 2014 at 22:14:41 UTC, Laeeth Isharc wrote:

 since you didn't get an answer.

 https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading


 you can turn off threads and processes to aid debugging:

 https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading


 StartServers 1
 ServerLimit 1

 With this configuration, only one process will be started, with 
 no additional processes ever being created. The WSGI 
 environment key/value pairs indicating how processes and 
 threads are being used will for this configuration be as 
 follows.

 wsgi.multithread	False
 wsgi.multiprocess	False

 In effect, this configuration has the result of serialising all 
 requests through a single process. This will allow an 
 interactive browser based debugger to be used, but may prevent 
 more complex WSGI applications which make use of AJAX 
 techniques from working. This could occur where a web page 
 initiates a sequence of AJAX requests and expects later 
 requests to be able to complete while a response for an initial 
 request is still pending. In other words, problems may occur 
 where requests overlap, as subsequent requests will not be able 
 to be executed until the initial request has completed.
 -----
 although obviously that is not a solution to your problem, it 
 might help pinpoint it.

 did you try tracking init state in your library and have entry 
 points call D runtime if not already initialized?


Thanks. I'll use that information about WSGI to test whether it 
really is a thread/process issue. Multiple requests causing 
re-entry definitely isn't the problem yet but worth being wary of 
down the road. Also intrigued by the idea of using a 
browser-based debugger.

I tried a variety of combinations of initialize and terminate but 
not specifically calling initialize from the request thread only 
once and without calling terminate. I'll give that a go but I'm 
no too hopeful as initialize/terminate pairs don't work, 
initialize from the start-up thread doesn't work and the docs are 
unclear whether initialize doesn't already do said state checking 
for me.
Nov 07 2014