www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Problems with shared library (-fPIC) and tango

reply Marius Muja <mariusm cs.ubc.ca> writes:
Hi,

I'm working on a project that needs to be compiled both as a standalone 
executable and (part of it) as a shared library. It's been working like 
that on 32 bit machines without any problems, but now I need to compile 
it on a 64 bit machine (to be able to access more the 3GB of memory).

When I tried to build the project on the 64 bit machine I was unable to 
build the shared library because of the following error:

libgphobos.a(ti_AC.o): relocation R_X86_64_32 against 
`_D17TypeInfo_C6Object6__initZ' can not be used when making a shared 
object; recompile with -fPIC

Since the error refers to an object in the standard library (I'm using 
tango) I tried to build the tango library using -fPIC. Doing that I 
didn't get the above error when I build the project, but it get the 
following assertion when I run it:

tango.core.Exception.AssertException lifetime.d(514): Assertion failure

This makes me think that the tango library doesn't work properly when 
compiled with -fPIC.

Does anyone know a solution to this problem? Is it possible to build a D 
shared library on a 64 bit machine when using tango?


Marius
Jul 04 2008
next sibling parent reply Lars Ivar Igesund <larsivar igesund.net> writes:
Marius Muja wrote:

 Hi,
 
 I'm working on a project that needs to be compiled both as a standalone
 executable and (part of it) as a shared library. It's been working like
 that on 32 bit machines without any problems, but now I need to compile
 it on a 64 bit machine (to be able to access more the 3GB of memory).
 
 When I tried to build the project on the 64 bit machine I was unable to
 build the shared library because of the following error:
 
 libgphobos.a(ti_AC.o): relocation R_X86_64_32 against
 `_D17TypeInfo_C6Object6__initZ' can not be used when making a shared
 object; recompile with -fPIC
 
 Since the error refers to an object in the standard library (I'm using
 tango) I tried to build the tango library using -fPIC. Doing that I
 didn't get the above error when I build the project, but it get the
 following assertion when I run it:
 
 tango.core.Exception.AssertException lifetime.d(514): Assertion failure
 
 This makes me think that the tango library doesn't work properly when
 compiled with -fPIC.
 
 Does anyone know a solution to this problem? Is it possible to build a D
 shared library on a 64 bit machine when using tango?
 
 
 Marius

There is a ticket related to this somewhere. I believe I was told once that part of the problem is that _all_ of the runtime can't be part of the dynamic library, but that there needs to be a static portion too with at least main. The above issue may have more to do with the effect of PIC on the code though. It would be very good if it could be resolved. -- Lars Ivar Igesund blog at http://larsivi.net DSource, #d.tango & #D: larsivi Dancing the Tango
Jul 05 2008
next sibling parent reply e-t172 <e-t172 akegroup.org> writes:
Lars Ivar Igesund a écrit :
 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.

IIRC, there is only one problem when trying to build Tango with -fPIC : the garbage collector, which segfaults on collection. Fix the GC and you have your Tango shared library.
Jul 05 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from e-t172 (e-t172 akegroup.org)'s article
 Lars Ivar Igesund a écrit :
 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.

the garbage collector, which segfaults on collection. Fix the GC and you have your Tango shared library.

I believe this is an issue with the static data segments not being represented in the way that the GC expects in a shared library. I really need to do some research to figure out how things are different in this instance. Though if someone knows then that would be good too :-) Sean
Jul 05 2008
next sibling parent reply e-t172 <e-t172 akegroup.org> writes:
Sean Kelly a écrit :
 == Quote from e-t172 (e-t172 akegroup.org)'s article
 Lars Ivar Igesund a écrit :
 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.

the garbage collector, which segfaults on collection. Fix the GC and you have your Tango shared library.

I believe this is an issue with the static data segments not being represented in the way that the GC expects in a shared library. I really need to do some research to figure out how things are different in this instance. Though if someone knows then that would be good too :-)

When I tried to build Tango in a .so (all Tango, not just the runtime), there was another (much less important) problem: when the executable starts, all the static main() functions of all modules were executed, even if I didn't use them. This can be a problem. When I executed my "Hello Word", I got messages about cluster initialization...
Jul 06 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from e-t172 (e-t172 akegroup.org)'s article
 Sean Kelly a écrit :
 == Quote from e-t172 (e-t172 akegroup.org)'s article
 Lars Ivar Igesund a écrit :
 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.

the garbage collector, which segfaults on collection. Fix the GC and you have your Tango shared library.

I believe this is an issue with the static data segments not being represented in the way that the GC expects in a shared library. I really need to do some research to figure out how things are different in this instance. Though if someone knows then that would be good too :-)

there was another (much less important) problem: when the executable starts, all the static main() functions of all modules were executed, even if I didn't use them. This can be a problem. When I executed my "Hello Word", I got messages about cluster initialization...

I'm not sure I understand. The only module with a main() function should be in the runtime. Did you mean static this()? I'll admit that if so that's still pretty weird. Sean
Jul 06 2008
parent e-t172 <e-t172 akegroup.org> writes:
Sean Kelly a écrit :
 == Quote from e-t172 (e-t172 akegroup.org)'s article
 Sean Kelly a écrit :
 == Quote from e-t172 (e-t172 akegroup.org)'s article
 Lars Ivar Igesund a écrit :
 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.

the garbage collector, which segfaults on collection. Fix the GC and you have your Tango shared library.

in the way that the GC expects in a shared library. I really need to do some research to figure out how things are different in this instance. Though if someone knows then that would be good too :-)

there was another (much less important) problem: when the executable starts, all the static main() functions of all modules were executed, even if I didn't use them. This can be a problem. When I executed my "Hello Word", I got messages about cluster initialization...

I'm not sure I understand. The only module with a main() function should be in the runtime. Did you mean static this()? I'll admit that if so that's still pretty weird.

Sorry. I of course meant static this() (module constructors). This was a typo.
Jul 06 2008
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Thomas Leonard wrote:
 On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:
 
 == Quote from e-t172 (e-t172 akegroup.org)'s article
 Lars Ivar Igesund a écrit :
 There is a ticket related to this somewhere. I believe I was told
 once that part of the problem is that _all_ of the runtime can't be
 part of the dynamic library, but that there needs to be a static
 portion too with at least main. The above issue may have more to do
 with the effect of PIC on the code though. It would be very good if
 it could be resolved.

the garbage collector, which segfaults on collection. Fix the GC and you have your Tango shared library.

represented in the way that the GC expects in a shared library. I really need to do some research to figure out how things are different in this instance. Though if someone knows then that would be good too :-)

Could that be why this program crashes? import std.gc: fullCollect; import mi = std.moduleinit; int main(string[] args) { //printf("%p\n", mi._moduleinfo_dtors[6]); fullCollect(); return 0; } If I uncomment the printf line, it works (I guess because the address gets left on the stack). Without calling fullCollect() explicitly, my programs work until the GC gets called, then they crash. I'm using a (rather modified) GDC on Linux, with libgphobos2 as a shared library. How are module variables like _moduleinfo_dtors supposed to get registered as roots with the GC?

The shared data segment(s) of a binary are identified by the runtime at startup and the runtime passes them to the GC for scanning. For example, look at initStaticDataPtrs() in this file: http://dsource.org/projects/tango/browser/trunk/lib/compiler/gdc/memory.d However, I think the way that the static data segment is identified in a shared library is different from in an executable app. I'm pretty sure the Boehm GC handles this properly and have been meaning to look at it for clues, but that hasn't made it to the top of my priority list yet. Sean
Oct 23 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
Thomas Leonard wrote:
 On Thu, 23 Oct 2008 19:59:30 -0700, Sean Kelly wrote:
 Thomas Leonard wrote:
 On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:
 == Quote from e-t172 (e-t172 akegroup.org)'s article
 IIRC, there is only one problem when trying to build Tango with -fPIC
 : the garbage collector, which segfaults on collection. Fix the GC
 and you have your Tango shared library.

represented in the way that the GC expects in a shared library. I really need to do some research to figure out how things are different in this instance. Though if someone knows then that would be good too :-)

import std.gc: fullCollect; import mi = std.moduleinit; int main(string[] args) { //printf("%p\n", mi._moduleinfo_dtors[6]); fullCollect(); return 0; } If I uncomment the printf line, it works (I guess because the address gets left on the stack). Without calling fullCollect() explicitly, my programs work until the GC gets called, then they crash. I'm using a (rather modified) GDC on Linux, with libgphobos2 as a shared library. How are module variables like _moduleinfo_dtors supposed to get registered as roots with the GC?

startup and the runtime passes them to the GC for scanning. For example, look at initStaticDataPtrs() in this file: http://dsource.org/projects/tango/browser/trunk/lib/compiler/gdc/

 However, I think the way that the static data segment is identified in a
 shared library is different from in an executable app.  I'm pretty sure
 the Boehm GC handles this properly and have been meaning to look at it
 for clues, but that hasn't made it to the top of my priority list yet.

Ah, thanks for the hints. I found some disabled code in gcgcc.d that tried to scan /proc/self/maps, and with a bit of tweaking it now works for me (tested on x86_64): http://repo.or.cz/w/delight/core.git? a=commitdiff;h=83befaa3f72973eb8afd8589d2e1cee4b37ee4fb;hp=03f581cf37409b35521bcfd6f239a14f1fa73221

Cool! I'll see about integrating these changes into druntime.
 BTW, would it be possible to get rid of this moduleinfo stuff? It seems 
 to cause a lot of problems, because when a program imports a library it 
 might not use exactly the same flags that were used when the library was 
 compiled (e.g. program is compiled with unit-tests, library wasn't). So 
 the program might try to use the moduleinfo, even though it doesn't 
 actually exist.

I don't think there's any way around moduleinfo in general, since it's the way that static ctors and dtors are called. However, I'd expect everything to work correctly regardless of the flags passed because the runtime code simply iterates across a list of ModuleInfo objects and calls whatever functions it needs to (after checking if the function pointers are non-null).
 Doesn't the dynamic linker already provide for calling static 
 initialisers? I'm not sure.

Possibly the static initializers that C uses, if any, but not the D initializers. However, I've started working on some new runtime functions to do dynamic loading and unloading of libraries on druntime, and this will do everything needed to properly initialize the library. Sean
Oct 24 2008
parent =?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sean Kelly wrote:
 Thomas Leonard wrote:
 Doesn't the dynamic linker already provide for calling static
 initialisers? I'm not sure.

Possibly the static initializers that C uses, if any, but not the D initializers.

AFAIK, all you need for some code to be run automatically on module loading/unloading is to put it in the correct section in the object file or in a properly named function (depending on the linker). For example, for GNU ld, sections called .init and .fini are executed upon module loading and unloading respectively. If the D initializers were put into these, they would be called automatically. You can look here for more details: http://www.delorie.com/gnu/docs/gcc/gccint_149.html Jerome <back to lurk mode/> - -- +------------------------- Jerome M. BERGER ---------------------+ | mailto:jeberger free.fr | ICQ: 238062172 | | http://jeberger.free.fr/ | Jabber: jeberger jabber.fr | +---------------------------------+------------------------------+ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkkC2AMACgkQd0kWM4JG3k88uQCgmW6hYyo93BlaugnUOLzAsisR hRIAn3AF1nay7Brp347mpZjTbZ4GzWiY =fbwC -----END PGP SIGNATURE-----
Oct 25 2008
prev sibling next sibling parent Thomas Leonard <talex5+d gmail.com> writes:
On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:

 == Quote from e-t172 (e-t172 akegroup.org)'s article
 Lars Ivar Igesund a écrit :
 There is a ticket related to this somewhere. I believe I was told
 once that part of the problem is that _all_ of the runtime can't be
 part of the dynamic library, but that there needs to be a static
 portion too with at least main. The above issue may have more to do
 with the effect of PIC on the code though. It would be very good if
 it could be resolved.

the garbage collector, which segfaults on collection. Fix the GC and you have your Tango shared library.

I believe this is an issue with the static data segments not being represented in the way that the GC expects in a shared library. I really need to do some research to figure out how things are different in this instance. Though if someone knows then that would be good too :-)

Could that be why this program crashes? import std.gc: fullCollect; import mi = std.moduleinit; int main(string[] args) { //printf("%p\n", mi._moduleinfo_dtors[6]); fullCollect(); return 0; } If I uncomment the printf line, it works (I guess because the address gets left on the stack). Without calling fullCollect() explicitly, my programs work until the GC gets called, then they crash. I'm using a (rather modified) GDC on Linux, with libgphobos2 as a shared library. How are module variables like _moduleinfo_dtors supposed to get registered as roots with the GC? Thanks,
Oct 23 2008
prev sibling parent Thomas Leonard <talex5+d gmail.com> writes:
On Thu, 23 Oct 2008 19:59:30 -0700, Sean Kelly wrote:
 Thomas Leonard wrote:
 On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:
 == Quote from e-t172 (e-t172 akegroup.org)'s article
 IIRC, there is only one problem when trying to build Tango with -fPIC
 : the garbage collector, which segfaults on collection. Fix the GC
 and you have your Tango shared library.

represented in the way that the GC expects in a shared library. I really need to do some research to figure out how things are different in this instance. Though if someone knows then that would be good too :-)

Could that be why this program crashes? import std.gc: fullCollect; import mi = std.moduleinit; int main(string[] args) { //printf("%p\n", mi._moduleinfo_dtors[6]); fullCollect(); return 0; } If I uncomment the printf line, it works (I guess because the address gets left on the stack). Without calling fullCollect() explicitly, my programs work until the GC gets called, then they crash. I'm using a (rather modified) GDC on Linux, with libgphobos2 as a shared library. How are module variables like _moduleinfo_dtors supposed to get registered as roots with the GC?

The shared data segment(s) of a binary are identified by the runtime at startup and the runtime passes them to the GC for scanning. For example, look at initStaticDataPtrs() in this file: http://dsource.org/projects/tango/browser/trunk/lib/compiler/gdc/

 
 However, I think the way that the static data segment is identified in a
 shared library is different from in an executable app.  I'm pretty sure
 the Boehm GC handles this properly and have been meaning to look at it
 for clues, but that hasn't made it to the top of my priority list yet.

Ah, thanks for the hints. I found some disabled code in gcgcc.d that tried to scan /proc/self/maps, and with a bit of tweaking it now works for me (tested on x86_64): http://repo.or.cz/w/delight/core.git? a=commitdiff;h=83befaa3f72973eb8afd8589d2e1cee4b37ee4fb;hp=03f581cf37409b35521bcfd6f239a14f1fa73221 BTW, would it be possible to get rid of this moduleinfo stuff? It seems to cause a lot of problems, because when a program imports a library it might not use exactly the same flags that were used when the library was compiled (e.g. program is compiled with unit-tests, library wasn't). So the program might try to use the moduleinfo, even though it doesn't actually exist. Doesn't the dynamic linker already provide for calling static initialisers? I'm not sure.
Oct 24 2008
prev sibling next sibling parent reply e-t172 <e-t172 akegroup.org> writes:
Marius Muja a écrit :
 I'm working on a project that needs to be compiled both as a standalone 
 executable and (part of it) as a shared library. It's been working like 
 that on 32 bit machines without any problems, but now I need to compile 
 it on a 64 bit machine (to be able to access more the 3GB of memory).
 
 When I tried to build the project on the 64 bit machine I was unable to 
 build the shared library because of the following error:
 
 libgphobos.a(ti_AC.o): relocation R_X86_64_32 against 
 `_D17TypeInfo_C6Object6__initZ' can not be used when making a shared 
 object; recompile with -fPIC

You can build a D shared library without including Tango in it (there will be undefined references in the shared library, which will be resolved when Tango is linked with the main executable). If you're using GDC, use the -nophoboslib switch when creating your shared library to avoid linking to Tango. Some problems can occur, though. If you happen to modify your shared library in such a way that the library makes use of a Tango symbol it didn't use before, it won't work unless you relink your executable to include the new symbol. That kind of defeats the whole purpose of having shared libraries. Of course, one could include all Tango symbols in the executable to solve the problem, but then you will end with a huge executable because all Tango will be included in it.
Jul 05 2008
parent Marius Muja <mariusm cs.ubc.ca> writes:
e-t172 wrote:

 
 You can build a D shared library without including Tango in it (there 
 will be undefined references in the shared library, which will be 
 resolved when Tango is linked with the main executable).
 
 If you're using GDC, use the -nophoboslib switch when creating your 
 shared library to avoid linking to Tango.

That won't really work for me, my shared library is in fact a matlab MEX file, so I'm not creating a main executable in the end, matlab will load it when needed. I'm also building a python binary extension this way, where I have the same problems.
 
 Some problems can occur, though. If you happen to modify your shared 
 library in such a way that the library makes use of a Tango symbol it 
 didn't use before, it won't work unless you relink your executable to 
 include the new symbol. That kind of defeats the whole purpose of having 
 shared libraries. Of course, one could include all Tango symbols in the 
 executable to solve the problem, but then you will end with a huge 
 executable because all Tango will be included in it.

Jul 05 2008
prev sibling parent e-t172 <e-t172 akegroup.org> writes:
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Thomas Leonard wrote:
 On Thu, 23 Oct 2008 19:59:30 -0700, Sean Kelly wrote:
 Thomas Leonard wrote:
 On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:
 == Quote from e-t172 (e-t172 akegroup.org)'s article
 IIRC, there is only one problem when trying to build Tango with -fPIC
 : the garbage collector, which segfaults on collection. Fix the GC
 and you have your Tango shared library.

represented in the way that the GC expects in a shared library. I really need to do some research to figure out how things are different in this instance. Though if someone knows then that would be good too :-)

import std.gc: fullCollect; import mi = std.moduleinit; int main(string[] args) { //printf("%p\n", mi._moduleinfo_dtors[6]); fullCollect(); return 0; } If I uncomment the printf line, it works (I guess because the address gets left on the stack). Without calling fullCollect() explicitly, my programs work until the GC gets called, then they crash. I'm using a (rather modified) GDC on Linux, with libgphobos2 as a shared library. How are module variables like _moduleinfo_dtors supposed to get registered as roots with the GC?

startup and the runtime passes them to the GC for scanning. For example, look at initStaticDataPtrs() in this file: http://dsource.org/projects/tango/browser/trunk/lib/compiler/gdc/

 However, I think the way that the static data segment is identified in a
 shared library is different from in an executable app.  I'm pretty sure
 the Boehm GC handles this properly and have been meaning to look at it
 for clues, but that hasn't made it to the top of my priority list yet.

Ah, thanks for the hints. I found some disabled code in gcgcc.d that tried to scan /proc/self/maps, and with a bit of tweaking it now works for me (tested on x86_64): http://repo.or.cz/w/delight/core.git? a=commitdiff;h=83befaa3f72973eb8afd8589d2e1cee4b37ee4fb;hp=03f581cf37409b35521bcfd6f239a14f1fa73221 BTW, would it be possible to get rid of this moduleinfo stuff? It seems to cause a lot of problems, because when a program imports a library it might not use exactly the same flags that were used when the library was compiled (e.g. program is compiled with unit-tests, library wasn't). So the program might try to use the moduleinfo, even though it doesn't actually exist. Doesn't the dynamic linker already provide for calling static initialisers? I'm not sure.

Sorry for the ten days delay, I was very busy lately. When I did some tests compiling Tango as a shared library, I noticed that all D static constructors present in the shared library were being called, which was kind of annoying at the time (for example, I got messages about "tina initialization" for a simple Hello world...). -- Etienne Dechamps / e-t172 - AKE Group Website: http://www.e-t172.net/ Contact: e-t172 akegroup.org Phone: +33547414942
Dec 03 2008