www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - building shared library from D code to import into cython

reply "Laeeth Isharc" <laeethnospam spammenot_laeeth.com> writes:
Hi.

I am trying to create a shared library in D linked against phobos 
so that I may use this in a cython extension module for Python.  
Ultimately I would like to be able to use a D class or struct 
(via the C++ interface) and call it from within cython, since 
cython classes cannot be instantiated without the gil (and this 
prevents easy parallelisation).

I feel a bit foolish asking the question as there is a nice 
example here for working with plain C using dmd as the linker, 
and using dmd and gcc to create a DMD shared library statically 
linked to phobos.  However, I have not succeeded in creating a D 
library statically linked to phobos that works with cython and 
python,
http://dlang.org/dll-linux.html#dso7

I tried it first with test C code to make sure I am able to get 
the C library/cython/Python interaction working.

pytest.c:
#include <stdio.h>

long pytest(long a)
{
	return a+1;
}

int main()
{
	long a =pytest(100);
	printf("%ld",a);
	return 0;
}


pytestpy.pyx:
cdef extern long pytest(long a)

cpdef pytestpy():
	return pytest(109)


setup.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

setup(
		ext_modules = cythonize([
  				 Extension("pytestpy", ["pytestpy.pyx"],
				libraries=["pytest"],
				)
  	]))


command line:
gcc -shared -o libpytest.so pytest.o
python setup.py build_ext -i
<copied libpytest.so to /usr/local/lib>
python
import pytestpy
pytestpy.pytestpy()
<it works>

----
now try pytest.d
import std.stdio;

extern (C) long pytest(long a)
{
	return a*2;
}

void main()
{
	auto a =pytest(100);
	writefln("%d",a);
}

command line:
rm pytestd.o
rm libpytest.so
rm /usr/local/lib/libpytest.so
dmd -c pytest.d -fPIC
gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so 
-L-rpath=/usr/local/lib
python

 import pytestpy
Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /usr/local/lib/libpytest.so: undefined symbol: _D3std5stdio12__ModuleInfoZ I guess it is not linking to the D runtime, but I am not sure what I should be doing to fix. Any thoughts appreciated. (The next step I was going to try when this works was C++ interface vs importing as a Cython class, but I thought best to start simple). I am running this on 64 bit Fedora 20. Thanks. Laeeth.
Oct 07 2014
parent reply "Freddy" <Hexagonalstar64 gmail.com> writes:
On Tuesday, 7 October 2014 at 20:55:59 UTC, Laeeth Isharc wrote:
 Hi.

 I am trying to create a shared library in D linked against 
 phobos so that I may use this in a cython extension module for 
 Python.  Ultimately I would like to be able to use a D class or 
 struct (via the C++ interface) and call it from within cython, 
 since cython classes cannot be instantiated without the gil 
 (and this prevents easy parallelisation).

 I feel a bit foolish asking the question as there is a nice 
 example here for working with plain C using dmd as the linker, 
 and using dmd and gcc to create a DMD shared library statically 
 linked to phobos.  However, I have not succeeded in creating a 
 D library statically linked to phobos that works with cython 
 and python,
 http://dlang.org/dll-linux.html#dso7

 I tried it first with test C code to make sure I am able to get 
 the C library/cython/Python interaction working.

 pytest.c:
 #include <stdio.h>

 long pytest(long a)
 {
 	return a+1;
 }

 int main()
 {
 	long a =pytest(100);
 	printf("%ld",a);
 	return 0;
 }


 pytestpy.pyx:
 cdef extern long pytest(long a)

 cpdef pytestpy():
 	return pytest(109)


 setup.py:

 from distutils.core import setup
 from distutils.extension import Extension
 from Cython.Build import cythonize

 setup(
 		ext_modules = cythonize([
  				 Extension("pytestpy", ["pytestpy.pyx"],
 				libraries=["pytest"],
 				)
  	]))


 command line:
 gcc -shared -o libpytest.so pytest.o
 python setup.py build_ext -i
 <copied libpytest.so to /usr/local/lib>
 python
 import pytestpy
 pytestpy.pytestpy()
 <it works>

 ----
 now try pytest.d
 import std.stdio;

 extern (C) long pytest(long a)
 {
 	return a*2;
 }

 void main()
 {
 	auto a =pytest(100);
 	writefln("%d",a);
 }

 command line:
 rm pytestd.o
 rm libpytest.so
 rm /usr/local/lib/libpytest.so
 dmd -c pytest.d -fPIC
 gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so 
 -L-rpath=/usr/local/lib
 python

 import pytestpy
Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /usr/local/lib/libpytest.so: undefined symbol: _D3std5stdio12__ModuleInfoZ I guess it is not linking to the D runtime, but I am not sure what I should be doing to fix. Any thoughts appreciated. (The next step I was going to try when this works was C++ interface vs importing as a Cython class, but I thought best to start simple). I am running this on 64 bit Fedora 20. Thanks. Laeeth.
Since when does gcc have a -defaultlib option? --- $ man gcc | grep defaultlib object-file-name -llibrary -nostartfiles -nodefaultlibs This is useful when you use -nostdlib or -nodefaultlibs but you do -nodefaultlibs is used. -nodefaultlibs -nodefaultlibs is libgcc.a, a library of internal subroutines which -nodefaultlibs you should usually specify -lgcc as well. This --- and why do you have a main function when compiling a shared library? Anyway, I think the problem is that "libpytest.so" can't find "libphobos2.so". Try adding "libphobos2.so"'s location (/usr/lib/x86_64-linux-gnu/libphobos2.so on my linux mint) to your -rpath .
Oct 07 2014
parent reply "Laeeth Isharc" <laeethnospam spammenot_laeeth.com> writes:
Hi.

Thanks for the quick response.

The -defaultlib was left around from trying all kinds of 
combinations of dmd and gcc.  I am not used to gcc, and it will 
take me some time to become properly acquainted with all the 
options.

I simply could not get it to recognize libphobos no matter what 
path settings I tried passing.

http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=4

I don't know whether this was necessary, but I did manage to get 
it to work after looking at a thread from a couple of years back. 
  I recompiled the D runtime and Phobos with position independent 
code (PIC) and pointed everything to that version instead, and it 
worked fine.  (Just export PIC=True, rename the posix.mak to 
Makefile, change the DMD path to /usr/sbin/dmd - in my case, and 
it all worked).

Next step is to try calling D interface from Cython.


Laeeth

In case anyone else should struggle with this in future.  
Makefile here:

PHOBOS_PATH=/opt/dmd2/src/phobos/generated/linux/release/64


pytestpy.so: pytest.d pytestpy.o
	dmd -fPIC -c pytest.d  #-shared -defaultlib=libphobos2.so 
-L-rpath=.:${PHOBOS_PATH}
	gcc -shared pytest.o pytestpy.o -o pytest.so -lphobos2 -lpthread 
-lrt -L. -L${PHOBOS_PATH} -Wl,-rpath=.:${PHOBOS_PATH} -o 
pytestpy.so -defaultlib=libphobos2.so -L-rpath=.:${PHOBOS_PATH}

pytest.o: %.d
	dmd -c $<

pytestpy.o: pytestpy.pyx
	cython pytestpy.pyx
	gcc -fPIC -c pytestpy.c -o pytestpy.o -I/usr/include/python2.7


clean:
	rm -f pytestpy.c pytest.o pytestpy.o pytest.so


pytest.d

extern (C) long pytest(long a)
{
	return a*2;
}

pytestpy.pyx
cdef extern long pytest(long a)

cpdef pytestpy():
	return pytest(109)

print pytest(102)


setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

setup(
		ext_modules = cythonize([
  				 Extension("pytestpy", ["pytestpy.pyx"],
				libraries=["pytest"],
				)
  	]))



On Tuesday, 7 October 2014 at 22:46:09 UTC, Freddy wrote:
 On Tuesday, 7 October 2014 at 20:55:59 UTC, Laeeth Isharc wrote:
 Hi.

 I am trying to create a shared library in D linked against 
 phobos so that I may use this in a cython extension module for 
 Python.  Ultimately I would like to be able to use a D class 
 or struct (via the C++ interface) and call it from within 
 cython, since cython classes cannot be instantiated without 
 the gil (and this prevents easy parallelisation).

 I feel a bit foolish asking the question as there is a nice 
 example here for working with plain C using dmd as the linker, 
 and using dmd and gcc to create a DMD shared library 
 statically linked to phobos.  However, I have not succeeded in 
 creating a D library statically linked to phobos that works 
 with cython and python,
 http://dlang.org/dll-linux.html#dso7

 I tried it first with test C code to make sure I am able to 
 get the C library/cython/Python interaction working.

 pytest.c:
 #include <stdio.h>

 long pytest(long a)
 {
 	return a+1;
 }

 int main()
 {
 	long a =pytest(100);
 	printf("%ld",a);
 	return 0;
 }


 pytestpy.pyx:
 cdef extern long pytest(long a)

 cpdef pytestpy():
 	return pytest(109)


 setup.py:

 from distutils.core import setup
 from distutils.extension import Extension
 from Cython.Build import cythonize

 setup(
 		ext_modules = cythonize([
 				 Extension("pytestpy", ["pytestpy.pyx"],
 				libraries=["pytest"],
 				)
 	]))


 command line:
 gcc -shared -o libpytest.so pytest.o
 python setup.py build_ext -i
 <copied libpytest.so to /usr/local/lib>
 python
 import pytestpy
 pytestpy.pytestpy()
 <it works>

 ----
 now try pytest.d
 import std.stdio;

 extern (C) long pytest(long a)
 {
 	return a*2;
 }

 void main()
 {
 	auto a =pytest(100);
 	writefln("%d",a);
 }

 command line:
 rm pytestd.o
 rm libpytest.so
 rm /usr/local/lib/libpytest.so
 dmd -c pytest.d -fPIC
 gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so 
 -L-rpath=/usr/local/lib
 python

 import pytestpy
Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /usr/local/lib/libpytest.so: undefined symbol: _D3std5stdio12__ModuleInfoZ I guess it is not linking to the D runtime, but I am not sure what I should be doing to fix. Any thoughts appreciated. (The next step I was going to try when this works was C++ interface vs importing as a Cython class, but I thought best to start simple). I am running this on 64 bit Fedora 20. Thanks. Laeeth.
Since when does gcc have a -defaultlib option? --- $ man gcc | grep defaultlib object-file-name -llibrary -nostartfiles -nodefaultlibs This is useful when you use -nostdlib or -nodefaultlibs but you do -nodefaultlibs is used. -nodefaultlibs -nodefaultlibs is libgcc.a, a library of internal subroutines which -nodefaultlibs you should usually specify -lgcc as well. This --- and why do you have a main function when compiling a shared library? Anyway, I think the problem is that "libpytest.so" can't find "libphobos2.so". Try adding "libphobos2.so"'s location (/usr/lib/x86_64-linux-gnu/libphobos2.so on my linux mint) to your -rpath .
Oct 07 2014
parent reply "Ellery Newcomer" <ellery-newcomer utulsa.edu> writes:
On Wednesday, 8 October 2014 at 00:25:57 UTC, Laeeth Isharc wrote:
 Hi.

 Thanks for the quick response.

 The -defaultlib was left around from trying all kinds of 
 combinations of dmd and gcc.  I am not used to gcc, and it will 
 take me some time to become properly acquainted with all the 
 options.
I managed to get it to compile with default dmd rpm: https://github.com/ariovistus/cythonic_d fedora 20, x86_64 Are you aware of pyd? (https://bitbucket.org/ariovistus/pyd) It knows how to build shared libraries, so I recommend you play with it, if only to watch how it does that. Biggest gotcha is starting up druntime.
Oct 09 2014
parent reply "Laeeth Isharc" <laeethnospam spammenot_laeeth.com> writes:
Thanks for this.

I am aware of pyd and will take a look at source/build process.

Any thoughts on speed in 2014  of pyd vs using cython to talk to 
D directly via C/C++ interface?  I saw this old coment here:

prabhuramachandran.blogspot.co.uk/2008/09/python-vs-cython-vs-d-pyd-vs-c-swig

" predict that the D version's relative slowness might have 
something to do with Pyd's somewhat awful handling of arrays (at 
least in part). More low-level and verbose, but possibly faster 
code could be written to compensate for this if this is indeed 
the problem. However, this is not a very attractive solution. 
(Optimally, Pyd should be capable of directly pointing D arrays 
at numpy arrays, but this is not actually implemented.)"


On Friday, 10 October 2014 at 02:19:17 UTC, Ellery Newcomer wrote:
 On Wednesday, 8 October 2014 at 00:25:57 UTC, Laeeth Isharc 
 wrote:
 Hi.

 Thanks for the quick response.

 The -defaultlib was left around from trying all kinds of 
 combinations of dmd and gcc.  I am not used to gcc, and it 
 will take me some time to become properly acquainted with all 
 the options.
I managed to get it to compile with default dmd rpm: https://github.com/ariovistus/cythonic_d fedora 20, x86_64 Are you aware of pyd? (https://bitbucket.org/ariovistus/pyd) It knows how to build shared libraries, so I recommend you play with it, if only to watch how it does that. Biggest gotcha is starting up druntime.
Oct 12 2014
parent "Ellery Newcomer" <ellery-newcomer utulsa.edu> writes:
On Sunday, 12 October 2014 at 16:07:19 UTC, Laeeth Isharc wrote:
 Any thoughts on speed in 2014  of pyd vs using cython to talk 
 to D directly via C/C++ interface?  I saw this old coment here:
pyd is basically just a convenience layer on top of the C interface. The part that would most likely screw with performance would be conversion from D objects to python objects and vice versa since that normally does a value copy. At the time of that comment, pyd's strategy for arrays was just iterate and convert each element. Now pyd can take advantage of the buffer protocol if it is exposed (2.7+), so converting e.g. numpy arrays should be just a memcopy. If you want by ref semantics instead of by value, use PydObject instead of a D type in your function signature. You can also turn off all conveniences and just use the C api with raw_only (https://bitbucket.org/ariovistus/pyd/wiki/CeleriD)
Oct 12 2014