www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dmd's linking order

reply Jonathan M Davis <jmdavisProg gmx.com> writes:
This stackoverflow question raises an interesting issue with linking order and 
dmd:

http://stackoverflow.com/questions/10095150/std-net-curl-linker-errors-in-linux

I don't know if it's this way on Linux machines in general, but the OP had to 
link his program manually to be able to use std.curl. dmd appears to put the -
L argument before any of its own linker arguments, and in this case, the 
linking argument for curl needs to go on the end.

So, my question is whether dmd should be changed to put any -L arguments 
passed to it after the arguments that it passes to the linker itself. I'm far 
from an expert on this and am quite surprised that the order of arguments to 
the linker matters, but since it does, it seems to me that we should find the 
optimal order for dmd to use if we can, since it's not terribly user friendly 
to force people to call gcc or ld directly rather than using dmd to link just 
because they want to pass an argument to the linker. That's what the -L flag is 
supposed to be for.

- Jonathan M Davis
Apr 10 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-04-11 03:46, Jonathan M Davis wrote:
 This stackoverflow question raises an interesting issue with linking order and
 dmd:

 http://stackoverflow.com/questions/10095150/std-net-curl-linker-errors-in-linux

 I don't know if it's this way on Linux machines in general, but the OP had to
 link his program manually to be able to use std.curl. dmd appears to put the -
 L argument before any of its own linker arguments, and in this case, the
 linking argument for curl needs to go on the end.

 So, my question is whether dmd should be changed to put any -L arguments
 passed to it after the arguments that it passes to the linker itself. I'm far
 from an expert on this and am quite surprised that the order of arguments to
 the linker matters, but since it does, it seems to me that we should find the
 optimal order for dmd to use if we can, since it's not terribly user friendly
 to force people to call gcc or ld directly rather than using dmd to link just
 because they want to pass an argument to the linker. That's what the -L flag is
 supposed to be for.

 - Jonathan M Davis

Could it happen that the linker arguments need to be placed first sometimes ? -- /Jacob Carlborg
Apr 11 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-04-11 10:37, Jonathan M Davis wrote:

 I have no idea. If they always need to be in a particular order, then we can
 just make dmd put the user provide ones in the correct spot, but if it varies
 depending on the flags, then either dmd is going to have to know which flags go
 in which order (assuming that it can), or there's no way to solve the problem
 without introducing new flags to dmd to give you more control over linking
 order or just using the linker directly.

 Unfortunately, I have no idea why the linking order even matters in the first
 place, so I can't really say what we need to do here. Hopefully, someone else
 around here _does_ know. But the issue does seem to need to be brought up.

I also finds it strange that linking order should matter. -- /Jacob Carlborg
Apr 11 2012
parent "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com> writes:
"Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
news:op.wcln1rh9eav7ka localhost.localdomain...
 I'm not a linker expert, but I found this page which describes the changes 
 and the reasoning:

 See this post: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition

That says the order-dependent behavior is caused by --as-needed being default. I'm no linker expert either, but the description of the purpose of --as-needed sounds...goofy and pointless (although maybe it makes sense for C's lack of a proper module system?). It can be disabled, apperently, by using --no-as-needed. That page tries to discourage people from doing that though, but the reason it gives seems vague. Personally, I'd wonder whether going along with --as-needed is really even worth doing. (Of course, if it turns out to be easy to fix the ordering, then we may as well.)
Apr 11 2012
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 11, 2012 at 01:37:48AM -0700, Jonathan M Davis wrote:
[...]
 Unfortunately, I have no idea why the linking order even matters in
 the first place, so I can't really say what we need to do here.
 Hopefully, someone else around here _does_ know. But the issue does
 seem to need to be brought up.

This is because many linkers (including *nix ld, IIRC) resolve symbols in the order the libraries were specified on the command-line. I don't remember the exact reason for this, but it probably has to do with improving the performance of the symbol resolution algorithm. It's a legacy from the early days of linker technology. T -- VI = Visual Irritation
Apr 11 2012
parent Mike Wey <mike-wey example.com> writes:
On 04/11/2012 09:54 PM, eles wrote:
 if you tried to link:

 gcc my_program.o -lB -lA (so, in reverse order)

read: gcc my_program.o -lB -lA (so, in unnatural order)
 this looked a bit like:

 gcc my_program.o -lA -lB

read: "it looked like the order is arbitrary"
 BUT what really happened was that the true link command was now:

 gcc my_program.o -lA -lB -lC -lB

read: gcc my_program.o -lB -lA -lB -lC (so that the symbols of libA *were* resolved)
 where the "-lA -lB -lC" is the expansion of the previous "-lA".


And to tell ld to not depend on the order in which the libs are specifies, you can use --start-group and --end-group. From the ld man page: --start-group archives --end-group The archives should be a list of archive files. They may be either explicit file names, or -l options. The specified archives are searched repeatedly until no new undefined references are created. Normally, an archive is searched only once in the order that it is specified on the command line. If a symbol in that archive is needed to resolve an undefined symbol referred to by an object in an archive that appears later on the command line, the linker would not be able to resolve that reference. By grouping the archives, they all be searched repeatedly until all possible references are resolved. Using this option has a significant performance cost. It is best to use it only when there are unavoidable circular references between two or more archives. -- Mike Wey
Apr 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 11 Apr 2012 13:04:59 -0400, H. S. Teoh <hsteoh quickfur.ath.cx>  
wrote:

 On Wed, Apr 11, 2012 at 01:37:48AM -0700, Jonathan M Davis wrote:
 [...]
 Unfortunately, I have no idea why the linking order even matters in
 the first place, so I can't really say what we need to do here.
 Hopefully, someone else around here _does_ know. But the issue does
 seem to need to be brought up.

This is because many linkers (including *nix ld, IIRC) resolve symbols in the order the libraries were specified on the command-line. I don't remember the exact reason for this, but it probably has to do with improving the performance of the symbol resolution algorithm. It's a legacy from the early days of linker technology.

Not exactly. For example, Ubuntu 10 was perfectly happy accepting libraries in any order. Only with Ubuntu 11 did this "revert" to the old way. I'm not sure that the reasoning was that it's "simpler", because clearly it's possible (and implemented!) Look for my other post for Ubuntu's explanation why, I don't really understand it. -Steve
Apr 11 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-04-11 16:59, Sean Kelly wrote:
 On Apr 11, 2012, at 1:25 AM, Jacob Carlborg<doob me.com>  wrote:

 Could it happen that the linker arguments need to be placed first sometimes ?

If it's a user-created library then maybe. The general rule on Unix is that dependent objects need to be listed before the object they depend on. I think the linker only does a single pass. Optlink doesn't have this problem--it's way nicer in this regard.

That would be the only case where optlink is nicer :) -- /Jacob Carlborg
Apr 11 2012
prev sibling next sibling parent "eles" <eles eles.com> writes:
 Not exactly.  For example, Ubuntu 10 was perfectly happy 
 accepting libraries in any order.  Only with Ubuntu 11 did this 
 "revert" to the old way.

Actually, it looked like that. What happenned behind the doors it was that the linking proceeded with --as-needed and the expansion was on the spot. Assume that you had libA depending on libB (but also on some other library that you do not use, namely libC). if you tried to link: gcc my_program.o -lB -lA (so, in reverse order) this looked a bit like: gcc my_program.o -lA -lB BUT what really happened was that the true link command was now: gcc my_program.o -lA -lB -lC -lB where the "-lA -lB -lC" is the expansion of the previous "-lA".
Apr 11 2012
prev sibling next sibling parent "eles" <eles eles.com> writes:
 if you tried to link:

 gcc my_program.o -lB -lA (so, in reverse order)

read: gcc my_program.o -lB -lA (so, in unnatural order)
 this looked a bit like:

 gcc my_program.o -lA -lB

read: "it looked like the order is arbitrary"
 BUT what really happened was that the true link command was now:

 gcc my_program.o -lA -lB -lC -lB

read: gcc my_program.o -lB -lA -lB -lC (so that the symbols of libA *were* resolved)
 where the "-lA -lB -lC" is the expansion of the previous "-lA".

Apr 11 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 04/11/12 20:30, Jacob Carlborg wrote:
 On 2012-04-11 16:59, Sean Kelly wrote:
 On Apr 11, 2012, at 1:25 AM, Jacob Carlborg<doob me.com>  wrote:

 Could it happen that the linker arguments need to be placed first sometimes ?

If it's a user-created library then maybe. The general rule on Unix is that dependent objects need to be listed before the object they depend on. I think the linker only does a single pass. Optlink doesn't have this problem--it's way nicer in this regard.

That would be the only case where optlink is nicer :)

Umm, "-( -llib1 -llib2 -)". But using the correct order would be the right solution. artur
Apr 11 2012
prev sibling parent "eles" <eles eles.com> writes:
 And to tell ld to not depend on the order in which the libs are 
 specifies, you can use --start-group and --end-group.

 From the ld man page:

 --start-group archives --end-group
     The archives should be a list of archive files.  They may 
 be either
     explicit file names, or -l options.

     The specified archives are searched repeatedly until no new
     undefined references are created.

Thanks. I knew that possibility exists, I just did not remember the parameters. Thank you for documenting this here.
Apr 12 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, April 11, 2012 10:25:37 Jacob Carlborg wrote:
 On 2012-04-11 03:46, Jonathan M Davis wrote:
 This stackoverflow question raises an interesting issue with linking order
 and dmd:
 
 http://stackoverflow.com/questions/10095150/std-net-curl-linker-errors-in-
 linux
 
 I don't know if it's this way on Linux machines in general, but the OP had
 to link his program manually to be able to use std.curl. dmd appears to
 put the - L argument before any of its own linker arguments, and in this
 case, the linking argument for curl needs to go on the end.
 
 So, my question is whether dmd should be changed to put any -L arguments
 passed to it after the arguments that it passes to the linker itself. I'm
 far from an expert on this and am quite surprised that the order of
 arguments to the linker matters, but since it does, it seems to me that
 we should find the optimal order for dmd to use if we can, since it's not
 terribly user friendly to force people to call gcc or ld directly rather
 than using dmd to link just because they want to pass an argument to the
 linker. That's what the -L flag is supposed to be for.
 
 - Jonathan M Davis

Could it happen that the linker arguments need to be placed first sometimes ?

I have no idea. If they always need to be in a particular order, then we can just make dmd put the user provide ones in the correct spot, but if it varies depending on the flags, then either dmd is going to have to know which flags go in which order (assuming that it can), or there's no way to solve the problem without introducing new flags to dmd to give you more control over linking order or just using the linker directly. Unfortunately, I have no idea why the linking order even matters in the first place, so I can't really say what we need to do here. Hopefully, someone else around here _does_ know. But the issue does seem to need to be brought up. - Jonathan M Davis
Apr 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 11 Apr 2012 04:47:19 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2012-04-11 10:37, Jonathan M Davis wrote:

 I have no idea. If they always need to be in a particular order, then  
 we can
 just make dmd put the user provide ones in the correct spot, but if it  
 varies
 depending on the flags, then either dmd is going to have to know which  
 flags go
 in which order (assuming that it can), or there's no way to solve the  
 problem
 without introducing new flags to dmd to give you more control over  
 linking
 order or just using the linker directly.

 Unfortunately, I have no idea why the linking order even matters in the  
 first
 place, so I can't really say what we need to do here. Hopefully,  
 someone else
 around here _does_ know. But the issue does seem to need to be brought  
 up.

I also finds it strange that linking order should matter.

It was a recent change in ld, it happened to me when I upgraded to ubutu 11.10. See this bug: http://d.puremagic.com/issues/show_bug.cgi?id=6822 I remember this behavior from old systems I used to work on (really old, like SunOS 4). I'm not a linker expert, but I found this page which describes the changes and the reasoning: See this post: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition My suggestion is to allow specifying in the dmd.conf file the specific location of phobos/druntime in the link order, then leave the -L options at the front. -Steve
Apr 11 2012
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
 Unfortunately, I have no idea why the linking order even matters in the  
 first
 place, so I can't really say what we need to do here. Hopefully, someone  
 else
 around here _does_ know. But the issue does seem to need to be brought  
 up.

i.e. phobos2 needs to precede curl. It seems to work when linking against shared libraries though.
Apr 11 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Apr 11, 2012, at 1:25 AM, Jacob Carlborg <doob me.com> wrote:

 Could it happen that the linker arguments need to be placed first sometime=

If it's a user-created library then maybe. The general rule on Unix is that d= ependent objects need to be listed before the object they depend on. I thin= k the linker only does a single pass. Optlink doesn't have this problem--it'= s way nicer in this regard.=20=
Apr 11 2012
prev sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 11 Apr 2012 14:51:14 -0400
schrieb "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com>:

 "Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
 news:op.wcln1rh9eav7ka localhost.localdomain...
 I'm not a linker expert, but I found this page which describes the changes 
 and the reasoning:

 See this post: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition

That says the order-dependent behavior is caused by --as-needed being default. I'm no linker expert either, but the description of the purpose of --as-needed sounds...goofy and pointless (although maybe it makes sense for C's lack of a proper module system?). It can be disabled, apperently, by using --no-as-needed. That page tries to discourage people from doing that though, but the reason it gives seems vague. Personally, I'd wonder whether going along with --as-needed is really even worth doing. (Of course, if it turns out to be easy to fix the ordering, then we may as well.)

I didn't know that --as-needed causes so much trouble. We've been using it on Gentoo a while before Ubuntu made that step and I know it took a while (~a year) until all packages would compile with it. The problem that this solved was that libraries and applications are linked to other libraries they don't actually call into. On a long stream of dependencies an application would dynamically load way more libraries than it actually uses, increasing the start time. So the linker lacked the intelligence to filter out unnecessary libraries before. Since there are some cases, where you must link to a library anyway, you can put --no-as-needed in front of them. Look at the positive side: You can now by default link against all libraries on your system without bloating the executable! :) -- Marco
Apr 11 2012
parent "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com> writes:
"Marco Leise" <Marco.Leise gmx.de> wrote in message 
news:20120412074942.0c6bb2cd marco-leise...
 Am Wed, 11 Apr 2012 14:51:14 -0400
 schrieb "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com>:

 "Steven Schveighoffer" <schveiguy yahoo.com> wrote in message
 news:op.wcln1rh9eav7ka localhost.localdomain...
 I'm not a linker expert, but I found this page which describes the 
 changes
 and the reasoning:

 See this post: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition

That says the order-dependent behavior is caused by --as-needed being default. I'm no linker expert either, but the description of the purpose of --as-needed sounds...goofy and pointless (although maybe it makes sense for C's lack of a proper module system?). It can be disabled, apperently, by using --no-as-needed. That page tries to discourage people from doing that though, but the reason it gives seems vague. Personally, I'd wonder whether going along with --as-needed is really even worth doing. (Of course, if it turns out to be easy to fix the ordering, then we may as well.)

I didn't know that --as-needed causes so much trouble. We've been using it on Gentoo a while before Ubuntu made that step and I know it took a while (~a year) until all packages would compile with it. The problem that this solved was that libraries and applications are linked to other libraries they don't actually call into. On a long stream of dependencies an application would dynamically load way more libraries than it actually uses, increasing the start time. So the linker lacked the intelligence to filter out unnecessary libraries before. Since there are some cases, where you must link to a library anyway, you can put --no-as-needed in front of them. Look at the positive side: You can now by default link against all libraries on your system without bloating the executable! :)

I was under the impression ld already stripped out unused symbols anyway?
Apr 11 2012