www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Pyd on Linux building woes

reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
This is a lengthy one, for which I apologize. I've tried to lay the 
problem out as best I can, but I am not sure exactly where the problem 
lies. I am reasonably certain that it is a compilation or linkage 
problem, and not a problem with the code.

I have done some work with CeleriD and added support for GDC. This is 
required for Linux support, as DMD can't build shared objects on Linux. 
However, I've run into some basic building issues that I could use some 
help with.

Before I started mucking around with it, CeleriD simply took all of the 
code files needed, jammed them all on a command line with the 
appropriate options (-c, mostly), and compiled them. Then it linked them 
together. Fine.

When adding gdc support, the first thing I realised was that gdc (and 
gcc) doesn't support DMD's -od option, and distutils requires that all 
of the object files be placed in a specific directory. My first stab at 
getting around this was to compile the sources one at a time, specifying 
the location of the object files one by one.

When I tested this with DMD, it compiled but did not link. It complained 
of undefined symbols. The symbols given were the big long mangles of 
template instances. I am still not sure why this happened. I fixed it by 
putting it back the way it was, as a single call to the compiler.

To get rid of the need for the -od option, I changed the current working 
directory to where the object files needed to go, and specified the full 
path to each of the source files. By not specifying a location, the 
object files just get dumped in the cwd. Problem solved.

Except that gdc still doesn't work. The .so compiles and links nicely 
enough, except that when it is imported by Python, Python issues this error:

ImportError: /full/path/to/testdll.so: undefined symbol: 
_D3pyd9func_wrap56__T9func_wrapS25_D7testdll7dg_testFDFZvZvVk1TvTPFDFZvZvZ4funcUPS6python8PyObjectPS6python8PyObjectZPS6python8PyObject

Which is the mangle of a template instance. One of the very ones, I 
might add, that Optlink wasn't finding when I compiled the source files 
one by one.

For reference, the calls to the compiler and linker are:

DMD in Windows:
compiler:
C:\dmd\dmd\bin\dmd.exe

-debug -unittest -I<location of the Python header> -I<location of Pyd> <all of the source files: Python, Pyd, boilerplate, user code> linker:
C:\dmd\dmd\bin\dmd.exe

-ofbuild\lib.win32-2.4\testdll.pyd <the python DLL's .lib file> <a list of the .obj files> <the generated .def file> That /works/. This doesn't: GDC in Linux: compiler: $ gdc -fPIC -c -fdebug -funittest -I <location of the Python header> -I <location of Pyd> <all of the source files: Python, Pyd, boilerplate, user code> linker: $ gdc -fPIC -nostartfiles -shared -fdebug -funittest -o build/lib.linux-i686-2.4/testdll.so <a list of the .o files> -Wl,-soname,testdll.so -lpython2.4 -nostartfiles is needed to override the _init and _fini functions, which are called when the module is loaded and unloaded (they are defined in the boilerplate code). -shared and the -soname option are similar to the .def file in Windows. (They tell the linker to make a .so file.) The final option is like the .lib file in Windows; it links it against Python. It is possible that this is a problem with gdc? Or am I making some stupid mistake? -- Kirk McDonald Pyd: Wrapping Python with D http://dsource.org/projects/pyd/wiki
Aug 08 2006
next sibling parent reply Lars Ivar Igesund <larsivar igesund.net> writes:
Kirk McDonald wrote:

 This is a lengthy one, for which I apologize. I've tried to lay the
 problem out as best I can, but I am not sure exactly where the problem
 lies. I am reasonably certain that it is a compilation or linkage
 problem, and not a problem with the code.

 It is possible that this is a problem with gdc? Or am I making some
 stupid mistake?
 

No, you don't make any stupid mistakes. I call this a rather bad bug in the DMD frontend (which is why it is possible to see it with both compilers). The problem is that the template instances usually don't belong in the object/source file of the template itself, but in the client code. Thus the same instance (that is, the template is instantiated with the same type/parameters), might be needed in several object files. AFAIK, this is not handled (or definately not handled correctly) when compiling one by one object file, and which is why you won't see any D libs (that is, foo.lib, or libfoo.a files) with templates around. However, DMD is able to process all the input .d files at once and in this mode it obviously operate differently, and when producing the executable, it know everything it needs to know about the template instance, and thus it is no longer a problem. So, there a couple of non-perfect options, and the least perfect is not generating the symbols at all :P The other is duplicating them over the object files that needs the instance, but then the linker usually don't know which one to choose. Similar problems can be found with the moduleinfo symbols in some cases and when used with libraries. The problems seems to be worse on linux, probably because Walter knows better how to use the OMF format to the fullest compared to the ELF format. There should be quite a few bug reports around concerning both of these issues. -- Lars Ivar Igesund blog at http://larsivi.net DSource & #D: larsivi
Aug 08 2006
parent reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Lars Ivar Igesund wrote:
 Kirk McDonald wrote:
 
 
This is a lengthy one, for which I apologize. I've tried to lay the
problem out as best I can, but I am not sure exactly where the problem
lies. I am reasonably certain that it is a compilation or linkage
problem, and not a problem with the code.

<snip>
It is possible that this is a problem with gdc? Or am I making some
stupid mistake?

No, you don't make any stupid mistakes. I call this a rather bad bug in the DMD frontend (which is why it is possible to see it with both compilers). The problem is that the template instances usually don't belong in the object/source file of the template itself, but in the client code. Thus the same instance (that is, the template is instantiated with the same type/parameters), might be needed in several object files. AFAIK, this is not handled (or definately not handled correctly) when compiling one by one object file, and which is why you won't see any D libs (that is, foo.lib, or libfoo.a files) with templates around. However, DMD is able to process all the input .d files at once and in this mode it obviously operate differently, and when producing the executable, it know everything it needs to know about the template instance, and thus it is no longer a problem.

Curses. I was afraid it was something like that. And it seems plain that GDC doesn't operate in this mode, for whatever reason, and so the problem doesn't go away when I specify the source files all at once.
 
 So, there a couple of non-perfect options, and the least perfect is not
 generating the symbols at all :P The other is duplicating them over the
 object files that needs the instance, but then the linker usually don't
 know which one to choose. Similar problems can be found with the moduleinfo
 symbols in some cases and when used with libraries. The problems seems to
 be worse on linux, probably because Walter knows better how to use the OMF
 format to the fullest compared to the ELF format.
 
 There should be quite a few bug reports around concerning both of these
 issues.
 

I suppose now I will try to find them, and pester Walter about them. Well, Pyd should be ready with Linux support whenever a compiler is available that resolves this problem. The code is now in place for it. -- Kirk McDonald Pyd: Wrapping Python with D http://dsource.org/projects/pyd/wiki
Aug 08 2006
parent reply kris <foo bar.com> writes:
Kirk McDonald wrote:
 Curses. I was afraid it was something like that. And it seems plain that 
 GDC doesn't operate in this mode, for whatever reason, and so the 
 problem doesn't go away when I specify the source files all at once.

IIRC, David had once said gdc operates upon only one module at a time
Aug 08 2006
parent Lars Ivar Igesund <larsivar igesund.net> writes:
kris wrote:

 Kirk McDonald wrote:
 Curses. I was afraid it was something like that. And it seems plain that
 GDC doesn't operate in this mode, for whatever reason, and so the
 problem doesn't go away when I specify the source files all at once.

IIRC, David had once said gdc operates upon only one module at a time

And the reason for this, is gcc which operates upon only one module at the time, IIRC. -- Lars Ivar Igesund blog at http://larsivi.net DSource & #D: larsivi
Aug 08 2006
prev sibling parent reply Carlos Santander <csantander619 gmail.com> writes:
Kirk McDonald escribió:
 
 Before I started mucking around with it, CeleriD simply took all of the 
 code files needed, jammed them all on a command line with the 
 appropriate options (-c, mostly), and compiled them. Then it linked them 
 together. Fine.
 
 When adding gdc support, the first thing I realised was that gdc (and 
 gcc) doesn't support DMD's -od option, and distutils requires that all 
 of the object files be placed in a specific directory. My first stab at 
 getting around this was to compile the sources one at a time, specifying 
 the location of the object files one by one.
 
 When I tested this with DMD, it compiled but did not link. It complained 
 of undefined symbols. The symbols given were the big long mangles of 
 template instances. I am still not sure why this happened. I fixed it by 
 putting it back the way it was, as a single call to the compiler.
 
 To get rid of the need for the -od option, I changed the current working 
 directory to where the object files needed to go, and specified the full 
 path to each of the source files. By not specifying a location, the 
 object files just get dumped in the cwd. Problem solved.
 

Not too helpful, but anyway: if you call gdc as "gdmd", it accepts dmd's parameters (instead of gcc's). gdmd is just a wrapper script. -- Carlos Santander Bernal
Aug 08 2006
parent Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Carlos Santander wrote:
 Kirk McDonald escribió:
 
 Before I started mucking around with it, CeleriD simply took all of 
 the code files needed, jammed them all on a command line with the 
 appropriate options (-c, mostly), and compiled them. Then it linked 
 them together. Fine.

 When adding gdc support, the first thing I realised was that gdc (and 
 gcc) doesn't support DMD's -od option, and distutils requires that all 
 of the object files be placed in a specific directory. My first stab 
 at getting around this was to compile the sources one at a time, 
 specifying the location of the object files one by one.

 When I tested this with DMD, it compiled but did not link. It 
 complained of undefined symbols. The symbols given were the big long 
 mangles of template instances. I am still not sure why this happened. 
 I fixed it by putting it back the way it was, as a single call to the 
 compiler.

 To get rid of the need for the -od option, I changed the current 
 working directory to where the object files needed to go, and 
 specified the full path to each of the source files. By not specifying 
 a location, the object files just get dumped in the cwd. Problem solved.

Not too helpful, but anyway: if you call gdc as "gdmd", it accepts dmd's parameters (instead of gcc's). gdmd is just a wrapper script.

I'm well aware of gdmd, but I have chosen to not use it. It is a perl script, and will not typically work on Windows. I eventually want to support using GDC on Windows. -- Kirk McDonald Pyd: Wrapping Python with D http://dsource.org/projects/pyd/wiki
Aug 08 2006