www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Dynamic libraries, again

reply Jacob Carlborg <doob me.com> writes:
I've been working the whole day on getting Tango to work built as a 
dynamic library with dmd on Mac OS X. I have one problem (for now) that 
I can't seem to solve.

The problem is that the module constructors are not called. That is 
because the module info array in the object module, which contains all 
the module constructors that should be called, is (almost) empty. It 
seems like that the compiler or the linker doesn't put the module 
constructors that are dynamically linked into the executable. I've also 
tried to move the _minfo_beg and _minfo_end variables, which are the 
beginning and the end of the sections containing the module info array, 
from the object module to a statically linked module and then passed 
them as arguments to the _moduleCtor function but without any success.

Does anyone have any idea have to solve the above problem? Does the 
compiler need to be modified?


/Jacob Carlborg
Mar 14 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Jacob Carlborg wrote:
 I've been working the whole day on getting Tango to work built as a 
 dynamic library with dmd on Mac OS X. I have one problem (for now) that 
 I can't seem to solve.
 
 The problem is that the module constructors are not called. That is 
 because the module info array in the object module, which contains all 
 the module constructors that should be called, is (almost) empty. It 
 seems like that the compiler or the linker doesn't put the module 
 constructors that are dynamically linked into the executable. I've also 
 tried to move the _minfo_beg and _minfo_end variables, which are the 
 beginning and the end of the sections containing the module info array, 
 from the object module to a statically linked module and then passed 
 them as arguments to the _moduleCtor function but without any success.
 
 Does anyone have any idea have to solve the above problem? Does the 
 compiler need to be modified?

There will be an _minfo_beg and _minfo_end for each loaded shared library. These must then be combined at runtime to form an aggregate array of all the modules. I don't know how to search the shared libraries to find the _minfo_beg/end sections, though I've never researched it, either.
Mar 14 2010
next sibling parent Jacob Carlborg <doob me.com> writes:
On 3/14/10 23:36, Walter Bright wrote:
 Jacob Carlborg wrote:
 I've been working the whole day on getting Tango to work built as a
 dynamic library with dmd on Mac OS X. I have one problem (for now)
 that I can't seem to solve.

 The problem is that the module constructors are not called. That is
 because the module info array in the object module, which contains all
 the module constructors that should be called, is (almost) empty. It
 seems like that the compiler or the linker doesn't put the module
 constructors that are dynamically linked into the executable. I've
 also tried to move the _minfo_beg and _minfo_end variables, which are
 the beginning and the end of the sections containing the module info
 array, from the object module to a statically linked module and then
 passed them as arguments to the _moduleCtor function but without any
 success.

 Does anyone have any idea have to solve the above problem? Does the
 compiler need to be modified?

There will be an _minfo_beg and _minfo_end for each loaded shared library. These must then be combined at runtime to form an aggregate array of all the modules.

I've looked at the output dumpobj gives on Tango and there are no __minfo_beg and __minfo_end sections but the __minfodata section is there.
 I don't know how to search the shared libraries to find the
 _minfo_beg/end sections, though I've never researched it, either.

Haven't you written dumpobj? It does just that.
Mar 15 2010
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 3/14/10 23:36, Walter Bright wrote:
 Jacob Carlborg wrote:
 I've been working the whole day on getting Tango to work built as a
 dynamic library with dmd on Mac OS X. I have one problem (for now)
 that I can't seem to solve.

 The problem is that the module constructors are not called. That is
 because the module info array in the object module, which contains all
 the module constructors that should be called, is (almost) empty. It
 seems like that the compiler or the linker doesn't put the module
 constructors that are dynamically linked into the executable. I've
 also tried to move the _minfo_beg and _minfo_end variables, which are
 the beginning and the end of the sections containing the module info
 array, from the object module to a statically linked module and then
 passed them as arguments to the _moduleCtor function but without any
 success.

 Does anyone have any idea have to solve the above problem? Does the
 compiler need to be modified?

There will be an _minfo_beg and _minfo_end for each loaded shared library. These must then be combined at runtime to form an aggregate array of all the modules.

Wouldn't this include all the modules and not just the modules that are used?
 I don't know how to search the shared libraries to find the
 _minfo_beg/end sections, though I've never researched it, either.

Mar 15 2010
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 3/14/10 23:36, Walter Bright wrote:
 Jacob Carlborg wrote:
 I've been working the whole day on getting Tango to work built as a
 dynamic library with dmd on Mac OS X. I have one problem (for now)
 that I can't seem to solve.

 The problem is that the module constructors are not called. That is
 because the module info array in the object module, which contains all
 the module constructors that should be called, is (almost) empty. It
 seems like that the compiler or the linker doesn't put the module
 constructors that are dynamically linked into the executable. I've
 also tried to move the _minfo_beg and _minfo_end variables, which are
 the beginning and the end of the sections containing the module info
 array, from the object module to a statically linked module and then
 passed them as arguments to the _moduleCtor function but without any
 success.

 Does anyone have any idea have to solve the above problem? Does the
 compiler need to be modified?

There will be an _minfo_beg and _minfo_end for each loaded shared library. These must then be combined at runtime to form an aggregate array of all the modules. I don't know how to search the shared libraries to find the _minfo_beg/end sections, though I've never researched it, either.

Latest update: I've found the __minfodata section and the content looks fine. The new problem I have is that an exception is thrown in _moduleCtor2 and I can't catch the exception. The problem is that it can't find any handler table. I think it's something wrong with _deh_beg and _deh_end in the deh module. If I have the _deh_beg and _deh_end as globals (as they are by default) it doesn't find a handler table. If I instead get the __deh_beg and the __deh_end sections from the main executable it does find a table handler but the table looks very different compared to a statically linked Tango. The length of the table is close to 600 compared to 1 when Tango is statically linked. It also contains a lot of zeros.
Mar 15 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Jacob Carlborg wrote:
 Latest update:
 I've found the __minfodata section and the content looks fine. The new 
 problem I have is that an exception is thrown in _moduleCtor2 and I 
 can't catch the exception. The problem is that it can't find any handler 
 table. I think it's something wrong with _deh_beg and _deh_end in the 
 deh module.
 
 If I have the _deh_beg and _deh_end as globals (as they are by default) 
 it doesn't find a handler table. If I instead get the __deh_beg and the 
 __deh_end sections from the main executable it does find a table handler 
 but the table looks very different compared to a statically linked 
 Tango. The length of the table is close to 600 compared to 1 when Tango 
 is statically linked. It also contains a lot of zeros.

Here's how things work. For each module, a reference to that module's ModuleInfo instance is stored in a segment named __minfodata. But, also, two empty segments are created: __minfo_beg and __minfo_end. A symbol, _minfo_beg, is defined for section __minfo_beg, and _minfo_end for __minfo_end. A "sandwich" is created: segment __minfo_beg _minfo_beg is defined here segment __minfodata ModuleInfo pointers go here segment __minfo_end _minfo_end is defined here The linker helpfully groups all the __minfodata segments together, and voila, by examining &_minfo_beg and &_minfo_end, we know the beginning and end of the ModuleInfo array! The exception handling tables, __deh_beg, __deh_eh, __deh_end are handled exactly the same way. The code to generate these tables is in backend/machobj.c. So, for an application, this all works great. The trouble with a shared library is now there are two sets of these triples, and the runtime doesn't know about the shared library set. Yes, dumpobj knows about .o files, but it doesn't know about linked files.
Mar 15 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 3/15/10 19:50, Walter Bright wrote:
 Jacob Carlborg wrote:
 Latest update:
 I've found the __minfodata section and the content looks fine. The new
 problem I have is that an exception is thrown in _moduleCtor2 and I
 can't catch the exception. The problem is that it can't find any
 handler table. I think it's something wrong with _deh_beg and _deh_end
 in the deh module.

 If I have the _deh_beg and _deh_end as globals (as they are by
 default) it doesn't find a handler table. If I instead get the
 __deh_beg and the __deh_end sections from the main executable it does
 find a table handler but the table looks very different compared to a
 statically linked Tango. The length of the table is close to 600
 compared to 1 when Tango is statically linked. It also contains a lot
 of zeros.

Here's how things work. For each module, a reference to that module's ModuleInfo instance is stored in a segment named __minfodata. But, also, two empty segments are created: __minfo_beg and __minfo_end. A symbol, _minfo_beg, is defined for section __minfo_beg, and _minfo_end for __minfo_end. A "sandwich" is created: segment __minfo_beg _minfo_beg is defined here segment __minfodata ModuleInfo pointers go here segment __minfo_end _minfo_end is defined here

The beg and end sections are not in the dynamic library. But it's possible to get the data with out those globals. This is how: http://tango.pastebin.com/YbmyZksx . This will also pull in all the module infos from the dynamic libraries regardless if they are used or not. BTW these globals are just causing trouble.
 The linker helpfully groups all the __minfodata segments together, and
 voila, by examining
 &_minfo_beg and &_minfo_end, we know the beginning and end of the
 ModuleInfo array!

I've already figured that out. I'm pretty sure the ModuleInfo array is working.
 The exception handling tables, __deh_beg, __deh_eh, __deh_end are
 handled exactly the same
 way. The code to generate these tables is in backend/machobj.c.

This is the problem. I did the same with __deh_eh as with __minfodata (see the link above) but it doesn't work.
 So, for an application, this all works great. The trouble with a shared
 library
 is now there are two sets of these triples, and the runtime doesn't know
 about the
 shared library set.

I think this is the problem with the __deh_eh section. I don't know from where I should get the __deh_eh section, the executable, the dynamic library or both. If both, how would I do that?
 Yes, dumpobj knows about .o files, but it doesn't know about linked files.

dumpobj works fine with executables and shared libraries.
Mar 15 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Jacob Carlborg wrote:
 On 3/15/10 19:50, Walter Bright wrote:
 Jacob Carlborg wrote:
 Latest update:
 I've found the __minfodata section and the content looks fine. The new
 problem I have is that an exception is thrown in _moduleCtor2 and I
 can't catch the exception. The problem is that it can't find any
 handler table. I think it's something wrong with _deh_beg and _deh_end
 in the deh module.

 If I have the _deh_beg and _deh_end as globals (as they are by
 default) it doesn't find a handler table. If I instead get the
 __deh_beg and the __deh_end sections from the main executable it does
 find a table handler but the table looks very different compared to a
 statically linked Tango. The length of the table is close to 600
 compared to 1 when Tango is statically linked. It also contains a lot
 of zeros.

Here's how things work. For each module, a reference to that module's ModuleInfo instance is stored in a segment named __minfodata. But, also, two empty segments are created: __minfo_beg and __minfo_end. A symbol, _minfo_beg, is defined for section __minfo_beg, and _minfo_end for __minfo_end. A "sandwich" is created: segment __minfo_beg _minfo_beg is defined here segment __minfodata ModuleInfo pointers go here segment __minfo_end _minfo_end is defined here

The beg and end sections are not in the dynamic library.

That doesn't make sense, because they are in the .o files (you can verify this with dumpobj). Did you generate a map file for the .so?
 But it's 
 possible to get the data with out those globals. This is how: 
 http://tango.pastebin.com/YbmyZksx . This will also pull in all the 
 module infos from the dynamic libraries regardless if they are used or 
 not. BTW these globals are just causing trouble.

How do they cause trouble, especially if they aren't even in the shared library?
 The linker helpfully groups all the __minfodata segments together, and
 voila, by examining
 &_minfo_beg and &_minfo_end, we know the beginning and end of the
 ModuleInfo array!

I've already figured that out. I'm pretty sure the ModuleInfo array is working.
 The exception handling tables, __deh_beg, __deh_eh, __deh_end are
 handled exactly the same
 way. The code to generate these tables is in backend/machobj.c.

This is the problem. I did the same with __deh_eh as with __minfodata (see the link above) but it doesn't work.

Does your code look in the deh for both the application and the shared library?
 So, for an application, this all works great. The trouble with a shared
 library
 is now there are two sets of these triples, and the runtime doesn't know
 about the
 shared library set.

I think this is the problem with the __deh_eh section. I don't know from where I should get the __deh_eh section, the executable, the dynamic library or both. If both, how would I do that?

As I wrote, if you have a shared library, it will have its own deh section that is completely distinct from the application's. The reason is because the linker doesn't know what dynamic libraries will be loaded, so there's no way it can combine the deh sections. I don't think the runtime loader is capable of doing a link. Therefore, the exception handling code will have to search *both*. To do both, the shared library will have to 'register' its deh code with the application code that searches deh, so the application code will know there is another deh to search. Either that, or the application code will have to do a runtime search for all loaded deh sections.
 Yes, dumpobj knows about .o files, but it doesn't know about linked 
 files.

dumpobj works fine with executables and shared libraries.

I've never tried that.
Mar 15 2010
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 3/15/10 20:36, Walter Bright wrote:
 Jacob Carlborg wrote:
 On 3/15/10 19:50, Walter Bright wrote:
 Jacob Carlborg wrote:
 Latest update:
 I've found the __minfodata section and the content looks fine. The new
 problem I have is that an exception is thrown in _moduleCtor2 and I
 can't catch the exception. The problem is that it can't find any
 handler table. I think it's something wrong with _deh_beg and _deh_end
 in the deh module.

 If I have the _deh_beg and _deh_end as globals (as they are by
 default) it doesn't find a handler table. If I instead get the
 __deh_beg and the __deh_end sections from the main executable it does
 find a table handler but the table looks very different compared to a
 statically linked Tango. The length of the table is close to 600
 compared to 1 when Tango is statically linked. It also contains a lot
 of zeros.

Here's how things work. For each module, a reference to that module's ModuleInfo instance is stored in a segment named __minfodata. But, also, two empty segments are created: __minfo_beg and __minfo_end. A symbol, _minfo_beg, is defined for section __minfo_beg, and _minfo_end for __minfo_end. A "sandwich" is created: segment __minfo_beg _minfo_beg is defined here segment __minfodata ModuleInfo pointers go here segment __minfo_end _minfo_end is defined here

The beg and end sections are not in the dynamic library.

That doesn't make sense, because they are in the .o files (you can verify this with dumpobj). Did you generate a map file for the .so?

They are in the object files but not in the dynamic library. I verified that with dumpobj.
 But it's possible to get the data with out those globals. This is how:
 http://tango.pastebin.com/YbmyZksx . This will also pull in all the
 module infos from the dynamic libraries regardless if they are used or
 not. BTW these globals are just causing trouble.

How do they cause trouble, especially if they aren't even in the shared library?

1. They are globals 2. They cause linker errors when building dynamic libraries which requires workarounds 3. They're not put into the dynamic libraries as they should
 The linker helpfully groups all the __minfodata segments together, and
 voila, by examining
 &_minfo_beg and &_minfo_end, we know the beginning and end of the
 ModuleInfo array!

I've already figured that out. I'm pretty sure the ModuleInfo array is working.
 The exception handling tables, __deh_beg, __deh_eh, __deh_end are
 handled exactly the same
 way. The code to generate these tables is in backend/machobj.c.

This is the problem. I did the same with __deh_eh as with __minfodata (see the link above) but it doesn't work.

Does your code look in the deh for both the application and the shared library?

Replying further down
 So, for an application, this all works great. The trouble with a shared
 library
 is now there are two sets of these triples, and the runtime doesn't know
 about the
 shared library set.

I think this is the problem with the __deh_eh section. I don't know from where I should get the __deh_eh section, the executable, the dynamic library or both. If both, how would I do that?

As I wrote, if you have a shared library, it will have its own deh section that is completely distinct from the application's. The reason is because the linker doesn't know what dynamic libraries will be loaded, so there's no way it can combine the deh sections. I don't think the runtime loader is capable of doing a link. Therefore, the exception handling code will have to search *both*. To do both, the shared library will have to 'register' its deh code with the application code that searches deh, so the application code will know there is another deh to search. Either that, or the application code will have to do a runtime search for all loaded deh sections.

I will try this now. Does the order matter?
 Yes, dumpobj knows about .o files, but it doesn't know about linked
 files.

dumpobj works fine with executables and shared libraries.

I've never tried that.

Mar 15 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
Jacob Carlborg wrote:
 On 3/15/10 20:36, Walter Bright wrote:
 Jacob Carlborg wrote:
 On 3/15/10 19:50, Walter Bright wrote:
 Jacob Carlborg wrote:
 Latest update:
 I've found the __minfodata section and the content looks fine. The new
 problem I have is that an exception is thrown in _moduleCtor2 and I
 can't catch the exception. The problem is that it can't find any
 handler table. I think it's something wrong with _deh_beg and _deh_end
 in the deh module.

 If I have the _deh_beg and _deh_end as globals (as they are by
 default) it doesn't find a handler table. If I instead get the
 __deh_beg and the __deh_end sections from the main executable it does
 find a table handler but the table looks very different compared to a
 statically linked Tango. The length of the table is close to 600
 compared to 1 when Tango is statically linked. It also contains a lot
 of zeros.

Here's how things work. For each module, a reference to that module's ModuleInfo instance is stored in a segment named __minfodata. But, also, two empty segments are created: __minfo_beg and __minfo_end. A symbol, _minfo_beg, is defined for section __minfo_beg, and _minfo_end for __minfo_end. A "sandwich" is created: segment __minfo_beg _minfo_beg is defined here segment __minfodata ModuleInfo pointers go here segment __minfo_end _minfo_end is defined here

The beg and end sections are not in the dynamic library.

That doesn't make sense, because they are in the .o files (you can verify this with dumpobj). Did you generate a map file for the .so?

They are in the object files but not in the dynamic library. I verified that with dumpobj.

Perhaps that's related to the linker errors.
 
 But it's possible to get the data with out those globals. This is how:
 http://tango.pastebin.com/YbmyZksx . This will also pull in all the
 module infos from the dynamic libraries regardless if they are used or
 not. BTW these globals are just causing trouble.

How do they cause trouble, especially if they aren't even in the shared library?

1. They are globals 2. They cause linker errors when building dynamic libraries which requires workarounds 3. They're not put into the dynamic libraries as they should

I don't understand. What kind of linker errors?
 So, for an application, this all works great. The trouble with a shared
 library
 is now there are two sets of these triples, and the runtime doesn't 
 know
 about the
 shared library set.

I think this is the problem with the __deh_eh section. I don't know from where I should get the __deh_eh section, the executable, the dynamic library or both. If both, how would I do that?

As I wrote, if you have a shared library, it will have its own deh section that is completely distinct from the application's. The reason is because the linker doesn't know what dynamic libraries will be loaded, so there's no way it can combine the deh sections. I don't think the runtime loader is capable of doing a link. Therefore, the exception handling code will have to search *both*. To do both, the shared library will have to 'register' its deh code with the application code that searches deh, so the application code will know there is another deh to search. Either that, or the application code will have to do a runtime search for all loaded deh sections.

I will try this now. Does the order matter?

No.
Mar 15 2010
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 3/15/10 20:36, Walter Bright wrote:
 Jacob Carlborg wrote:
 On 3/15/10 19:50, Walter Bright wrote:
 Jacob Carlborg wrote:
 Latest update:
 I've found the __minfodata section and the content looks fine. The new
 problem I have is that an exception is thrown in _moduleCtor2 and I
 can't catch the exception. The problem is that it can't find any
 handler table. I think it's something wrong with _deh_beg and _deh_end
 in the deh module.

 If I have the _deh_beg and _deh_end as globals (as they are by
 default) it doesn't find a handler table. If I instead get the
 __deh_beg and the __deh_end sections from the main executable it does
 find a table handler but the table looks very different compared to a
 statically linked Tango. The length of the table is close to 600
 compared to 1 when Tango is statically linked. It also contains a lot
 of zeros.

Here's how things work. For each module, a reference to that module's ModuleInfo instance is stored in a segment named __minfodata. But, also, two empty segments are created: __minfo_beg and __minfo_end. A symbol, _minfo_beg, is defined for section __minfo_beg, and _minfo_end for __minfo_end. A "sandwich" is created: segment __minfo_beg _minfo_beg is defined here segment __minfodata ModuleInfo pointers go here segment __minfo_end _minfo_end is defined here

The beg and end sections are not in the dynamic library.

That doesn't make sense, because they are in the .o files (you can verify this with dumpobj). Did you generate a map file for the .so?
 But it's possible to get the data with out those globals. This is how:
 http://tango.pastebin.com/YbmyZksx . This will also pull in all the
 module infos from the dynamic libraries regardless if they are used or
 not. BTW these globals are just causing trouble.

How do they cause trouble, especially if they aren't even in the shared library?
 The linker helpfully groups all the __minfodata segments together, and
 voila, by examining
 &_minfo_beg and &_minfo_end, we know the beginning and end of the
 ModuleInfo array!

I've already figured that out. I'm pretty sure the ModuleInfo array is working.
 The exception handling tables, __deh_beg, __deh_eh, __deh_end are
 handled exactly the same
 way. The code to generate these tables is in backend/machobj.c.

This is the problem. I did the same with __deh_eh as with __minfodata (see the link above) but it doesn't work.

Does your code look in the deh for both the application and the shared library?
 So, for an application, this all works great. The trouble with a shared
 library
 is now there are two sets of these triples, and the runtime doesn't know
 about the
 shared library set.

I think this is the problem with the __deh_eh section. I don't know from where I should get the __deh_eh section, the executable, the dynamic library or both. If both, how would I do that?

As I wrote, if you have a shared library, it will have its own deh section that is completely distinct from the application's. The reason is because the linker doesn't know what dynamic libraries will be loaded, so there's no way it can combine the deh sections. I don't think the runtime loader is capable of doing a link. Therefore, the exception handling code will have to search *both*. To do both, the shared library will have to 'register' its deh code with the application code that searches deh, so the application code will know there is another deh to search. Either that, or the application code will have to do a runtime search for all loaded deh sections.
 Yes, dumpobj knows about .o files, but it doesn't know about linked
 files.

dumpobj works fine with executables and shared libraries.

I've never tried that.

I managed to get it to work now. I had to reverse the order of the module info array in object_.d and the function table array in deh.d. Thank you very much for all the help.
Mar 15 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
Jacob Carlborg wrote:
 I managed to get it to work now. I had to reverse the order of the 
 module info array in object_.d and the function table array in deh.d. 
 Thank you very much for all the help.

Awesome! Thanks!
Mar 15 2010