Digital Mars - digitalmars.D - Lib change leads to larger executables

↑ ↓ ← → Walter Bright <newshound digitalmars.com> writes:

 It does, but increases the exe size of the first example from 180kb to 617kb!
 180kb is when compiled using build/rebuild/jake etc (no library) and 


 is when using dmd+lib only. Same flags in both cases: none at all


Let's say you have a template instance, TI. It is declared in two 
modules, M1 and M2:

-----------M1------------
TI
A
-----------M2------------
TI
B
-------------------------

M1 also declares A, and M2 also declares B. Now, the linker is looking 
to resolve TI, and the first one it finds is one in M1, and so links in 
M1. Later on, it needs to resolve B, and so links in M2. The redundant 
TI is discarded (because it's a COMDAT).

However, suppose the program never references A, and A is a chunk of 
code that pulls in lots of other bloat. This could make the executable 
much larger than if, in resolving TI, it had picked M2 instead.

You can control which module containing TI will be pulled in by the 
linker to resolve TI, by specifying that module first to lib.exe.

You can also put TI in a third module that has neither A nor B in it. 
When compiling M1 and M2, import that third module, so TI won't be 
generated for M1 or M2.

Feb 21 2007

↑ ↓ ← → kris <foo bar.com> writes:

Walter Bright wrote:
 It does, but increases the exe size of the first example from 180kb to 
 617kb!

  > 180kb is when compiled using build/rebuild/jake etc (no library) and 
 the 617kb
  > is when using dmd+lib only. Same flags in both cases: none at all

 Let's say you have a template instance, TI. It is declared in two 
 modules, M1 and M2:

 -----------M1------------
 TI
 A
 -----------M2------------
 TI
 B
 -------------------------

 M1 also declares A, and M2 also declares B. Now, the linker is looking 
 to resolve TI, and the first one it finds is one in M1, and so links in 
 M1. Later on, it needs to resolve B, and so links in M2. The redundant 
 TI is discarded (because it's a COMDAT).

 However, suppose the program never references A, and A is a chunk of 
 code that pulls in lots of other bloat. This could make the executable 
 much larger than if, in resolving TI, it had picked M2 instead.

 You can control which module containing TI will be pulled in by the 
 linker to resolve TI, by specifying that module first to lib.exe.

 You can also put TI in a third module that has neither A nor B in it. 
 When compiling M1 and M2, import that third module, so TI won't be 
 generated for M1 or M2.

This is definately useful; thanks for this and for being so expediant 
with the lib change.

In this particular case I suspect something else is the cause, since (a) 
Tango is deliberately very granular (b) the map file for the huge exe is 
showing gobs of data that shouldn't exist <g>

Feb 21 2007

↑ ↓ ← → kris <foo bar.com> writes:

Walter Bright wrote:
 kris wrote:
 
 Walter Bright wrote:

 For a quick & dirty test, try reversing the order the object files 
 are presented to lib.



 there's a couple of hundred :-D


 
 Do the ones that were giving the undefined reference before the changes 
 to lib.


The lib itself is actually built via a single D module, which imports 
all others. This is then given to Build to construct the library. Thus I 
don't have direct control over the ordering. Having said that, it 
appears Build does something different when the modules are reordered; 
Likely changing the order in which modules are presented to the lib.

By moving things around, I see a change in size on the target executable 
between -4kb to +5kb

Feb 21 2007

↑ ↓ ← → kris <foo bar.com> writes:

kris wrote:
 Walter Bright wrote:
 
 kris wrote:

 Walter Bright wrote:

 For a quick & dirty test, try reversing the order the object files 
 are presented to lib.




 there's a couple of hundred :-D




 Do the ones that were giving the undefined reference before the 
 changes to lib.


 
 The lib itself is actually built via a single D module, which imports 
 all others. This is then given to Build to construct the library. Thus I 
 don't have direct control over the ordering. Having said that, it 
 appears Build does something different when the modules are reordered; 
 Likely changing the order in which modules are presented to the lib.
 
 By moving things around, I see a change in size on the target executable 
 between -4kb to +5kb
 


I've been messing with the response file handed to the librarian (via 
lib  foo); moving modules around here and there, reordering big chunks 
etc. Have yet to see a notable change in the resulting exe after 
relinking against each lib version.

Feb 21 2007

↑ ↓ ← → kris <foo bar.com> writes:

kris wrote:
 Walter Bright wrote:
 
 kris wrote:

 I've been messing with the response file handed to the librarian (via 
 lib  foo); moving modules around here and there, reordering big 
 chunks etc. Have yet to see a notable change in the resulting exe 
 after relinking against each lib version.




 Then look at the .map file to see what was linked in to the larger 
 file that wasn't in the smaller. Remove that module from the library. 
 Link again, and see what was unresolved. Rinse, repeat, and you'll see 
 what was pulling it all in.


 
 That's exactly what I'm doing, and I agree there seems to be something 
 odd going on here. With ~200 modules, it's no slam-dunk to isolate it :)



OK: narrowed it down to one obj file. The vast sea of data is coming 
from the 'locale' package, which is stuffed to the gills with I18N content.

There's a half dozen modules in 'locale', none of which are used or 
referenced by any other code in Tango, or by the example at hand. It is 
an entirely isolated package (for obvious reason).

Yes, there's the /potential/ for symbolic collision between 'locale' and 
some other module/package. Let's consider that in a moment.

In the meantime, I whittled the dependency down to one single module 
listed right at the very end of the lib-response file (the very last one 
to be added to the lib). This module is called Core.

When Core is added to the lib, the linker emits "missing symbol" errors 
since the rest of the locale package is missing. When Core is removed, 
there are no link errors. This indicates some kind of symbolic 
collision; one that is triggered by the very last module added to the lib?

So, sifting through the obj2asm output for Core, I see all the publics 
are correctly prefixed by the package name. Thus, each symbol exposed 
appears to be unique across the entire library. Except for one that 
stands out. It is noted as an 'extern', yet is also listed amongst the 
full set of publics in the obj2asm output; and it looks a bit 
suspicious. Here's a small snippet from the ocean of public symbols in Core:

====================
_D5tango4text6locale4Core18DaylightSavingTime5_ctorMFS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8TimeSpanZC5tango4text6locale4Core
8DaylightSavingTime 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core18DaylightSavingTime5startMFZS5tango4text6
ocale4Core8DateTime 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core18DaylightSavingTime3endMFZS5tango4text6
ocale4Core8DateTime 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core18DaylightSavingTime6changeMFZS5tango4text6
ocale4Core8TimeSpan 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core
8DaylightSavingTime 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core18DaylightSavingTime9getSundayMFiiiiiiiiZS5tango4text6
ocale4Core8DateTime 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone12getLocalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6
ocale4Core8DateTime 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone16getUniversalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6
ocale4Core8DateTime 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone12getUtcOffsetMFS5tango4text6locale4Core8DateTimeZS5tango4text6
ocale4Core8TimeSpan 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone20isDaylightSavingTimeMFS5tango4text6lo
ale4Core8DateTimeZb 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone7currentFZC5tango4text6locale4Core8TimeZone 
COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone5_ctorMFZC5tango4text6locale4Core8TimeZone 
COMDAT flags=x0 attr=x0 align=x0
_D47TypeInfo_C5tango4text6locale4Core12NumberFormat6__initZ	COMDAT 
flags=x0 attr=x10 align=x0
_D49TypeInfo_C5tango4text6locale4Core14DateTimeFormat6__initZ	COMDAT 
flags=x0 attr=x10 align=x0
_D5tango4text6locale4Core14__T7arrayOfTiZ7arrayOfFAiXAi	COMDAT flags=x0 
attr=x10 align=x0
_D5tango4text6locale4Core15__T7arrayOfTAaZ7arrayOfFAAaXAAa	COMDAT 
flags=x0 attr=x10 align=x0
_D12TypeInfo_AAa6__initZ	COMDAT flags=x0 attr=x10 align=x0
__D5tango4text6locale4Core9__modctorFZv	COMDAT flags=x0 attr=x0 align=x0
__D5tango4text6locale4Core9__moddtorFZv	COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8__assertFiZv	COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core7__arrayZ	COMDAT flags=x0 attr=x0 align=x0

====================

You see the odd one out? That cursed _D12TypeInfo_AAa6__initZ again?

Feb 21 2007

↑ ↓ ← → Pragma <ericanderton yahoo.removeme.com> writes:

kris wrote:
 kris wrote:
 Walter Bright wrote:

 kris wrote:

 I've been messing with the response file handed to the librarian 
 (via lib  foo); moving modules around here and there, reordering big 
 chunks etc. Have yet to see a notable change in the resulting exe 
 after relinking against each lib version.




 Then look at the .map file to see what was linked in to the larger 
 file that wasn't in the smaller. Remove that module from the library. 
 Link again, and see what was unresolved. Rinse, repeat, and you'll 
 see what was pulling it all in.



 That's exactly what I'm doing, and I agree there seems to be something 
 odd going on here. With ~200 modules, it's no slam-dunk to isolate it :)


 
 OK: narrowed it down to one obj file. The vast sea of data is coming 
 from the 'locale' package, which is stuffed to the gills with I18N content.
 
 There's a half dozen modules in 'locale', none of which are used or 
 referenced by any other code in Tango, or by the example at hand. It is 
 an entirely isolated package (for obvious reason).
 
 Yes, there's the /potential/ for symbolic collision between 'locale' and 
 some other module/package. Let's consider that in a moment.
 
 In the meantime, I whittled the dependency down to one single module 
 listed right at the very end of the lib-response file (the very last one 
 to be added to the lib). This module is called Core.
 
 When Core is added to the lib, the linker emits "missing symbol" errors 
 since the rest of the locale package is missing. When Core is removed, 
 there are no link errors. This indicates some kind of symbolic 
 collision; one that is triggered by the very last module added to the lib?
 
 So, sifting through the obj2asm output for Core, I see all the publics 
 are correctly prefixed by the package name. Thus, each symbol exposed 
 appears to be unique across the entire library. Except for one that 
 stands out. It is noted as an 'extern', yet is also listed amongst the 
 full set of publics in the obj2asm output; and it looks a bit 
 suspicious. Here's a small snippet from the ocean of public symbols in 
 Core:
 
 ====================
 _D5tango4text6locale4Core18DaylightSavingTime5_ctorMFS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8TimeSpanZC5tango4text6locale4Core
8DaylightSavingTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core18DaylightSavingTime5startMFZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core18DaylightSavingTime3endMFZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core18DaylightSavingTime6changeMFZS5tango4text6
ocale4Core8TimeSpan 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core
8DaylightSavingTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core18DaylightSavingTime9getSundayMFiiiiiiiiZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone12getLocalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone16getUniversalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone12getUtcOffsetMFS5tango4text6locale4Core8DateTimeZS5tango4text6
ocale4Core8TimeSpan 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone20isDaylightSavingTimeMFS5tango4text6lo
ale4Core8DateTimeZb 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone7currentFZC5tango4text6locale4Core8TimeZone 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone5_ctorMFZC5tango4text6locale4Core8TimeZone 
 COMDAT flags=x0 attr=x0 align=x0
 _D47TypeInfo_C5tango4text6locale4Core12NumberFormat6__initZ    COMDAT 
 flags=x0 attr=x10 align=x0
 _D49TypeInfo_C5tango4text6locale4Core14DateTimeFormat6__initZ    COMDAT 
 flags=x0 attr=x10 align=x0
 _D5tango4text6locale4Core14__T7arrayOfTiZ7arrayOfFAiXAi    COMDAT 
 flags=x0 attr=x10 align=x0
 _D5tango4text6locale4Core15__T7arrayOfTAaZ7arrayOfFAAaXAAa    COMDAT 
 flags=x0 attr=x10 align=x0
 _D12TypeInfo_AAa6__initZ    COMDAT flags=x0 attr=x10 align=x0
 __D5tango4text6locale4Core9__modctorFZv    COMDAT flags=x0 attr=x0 align=x0
 __D5tango4text6locale4Core9__moddtorFZv    COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8__assertFiZv    COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core7__arrayZ    COMDAT flags=x0 attr=x0 align=x0
 
 ====================
 
 You see the odd one out? That cursed _D12TypeInfo_AAa6__initZ again?


Sorry if I'm stating the obvious, but it seems to me that the linker is finding
this typeinfo COMDAT in Core first, 
rather than somewhere else, and is thereby forcing the inclusion of the rest of
it's containing module.

Does moving core.obj to the end of the .lib solve the problem?

-- 
- EricAnderton at yahoo

Feb 21 2007

↑ ↓ ← → kris <foo bar.com> writes:

Pragma wrote:
 kris wrote:
 
 kris wrote:

 Walter Bright wrote:

 kris wrote:

 I've been messing with the response file handed to the librarian 
 (via lib  foo); moving modules around here and there, reordering 
 big chunks etc. Have yet to see a notable change in the resulting 
 exe after relinking against each lib version.





 Then look at the .map file to see what was linked in to the larger 
 file that wasn't in the smaller. Remove that module from the 
 library. Link again, and see what was unresolved. Rinse, repeat, and 
 you'll see what was pulling it all in.




 That's exactly what I'm doing, and I agree there seems to be 
 something odd going on here. With ~200 modules, it's no slam-dunk to 
 isolate it :)




 OK: narrowed it down to one obj file. The vast sea of data is coming 
 from the 'locale' package, which is stuffed to the gills with I18N 
 content.

 There's a half dozen modules in 'locale', none of which are used or 
 referenced by any other code in Tango, or by the example at hand. It 
 is an entirely isolated package (for obvious reason).

 Yes, there's the /potential/ for symbolic collision between 'locale' 
 and some other module/package. Let's consider that in a moment.

 In the meantime, I whittled the dependency down to one single module 
 listed right at the very end of the lib-response file (the very last 
 one to be added to the lib). This module is called Core.

 When Core is added to the lib, the linker emits "missing symbol" 
 errors since the rest of the locale package is missing. When Core is 
 removed, there are no link errors. This indicates some kind of 
 symbolic collision; one that is triggered by the very last module 
 added to the lib?

 So, sifting through the obj2asm output for Core, I see all the publics 
 are correctly prefixed by the package name. Thus, each symbol exposed 
 appears to be unique across the entire library. Except for one that 
 stands out. It is noted as an 'extern', yet is also listed amongst the 
 full set of publics in the obj2asm output; and it looks a bit 
 suspicious. Here's a small snippet from the ocean of public symbols in 
 Core:

 ====================
 _D5tango4text6locale4Core18DaylightSavingTime5_ctorMFS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8TimeSpanZC5tango4text6locale4Core
8DaylightSavingTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core18DaylightSavingTime5startMFZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core18DaylightSavingTime3endMFZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core18DaylightSavingTime6changeMFZS5tango4text6
ocale4Core8TimeSpan 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core
8DaylightSavingTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core18DaylightSavingTime9getSundayMFiiiiiiiiZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone12getLocalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone16getUniversalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6
ocale4Core8DateTime 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone12getUtcOffsetMFS5tango4text6locale4Core8DateTimeZS5tango4text6
ocale4Core8TimeSpan 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone20isDaylightSavingTimeMFS5tango4text6lo
ale4Core8DateTimeZb 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone7currentFZC5tango4text6locale4Core8TimeZone 
 COMDAT flags=x0 attr=x0 align=x0
 _D5tango4text6locale4Core8TimeZone5_ctorMFZC5tango4text6locale4Core8TimeZone 
 COMDAT flags=x0 attr=x0 align=x0
 _D47TypeInfo_C5tango4text6locale4Core12NumberFormat6__initZ    COMDAT 
 flags=x0 attr=x10 align=x0
 _D49TypeInfo_C5tango4text6locale4Core14DateTimeFormat6__initZ    
 COMDAT flags=x0 attr=x10 align=x0
 _D5tango4text6locale4Core14__T7arrayOfTiZ7arrayOfFAiXAi    COMDAT 
 flags=x0 attr=x10 align=x0
 _D5tango4text6locale4Core15__T7arrayOfTAaZ7arrayOfFAAaXAAa    COMDAT 
 flags=x0 attr=x10 align=x0
 _D12TypeInfo_AAa6__initZ    COMDAT flags=x0 attr=x10 align=x0
 __D5tango4text6locale4Core9__modctorFZv    COMDAT flags=x0 attr=x0 
 align=x0
 __D5tango4text6locale4Core9__moddtorFZv    COMDAT flags=x0 attr=x0 
 align=x0
 _D5tango4text6locale4Core8__assertFiZv    COMDAT flags=x0 attr=x0 
 align=x0
 _D5tango4text6locale4Core7__arrayZ    COMDAT flags=x0 attr=x0 align=x0

 ====================

 You see the odd one out? That cursed _D12TypeInfo_AAa6__initZ again?


 
 Sorry if I'm stating the obvious, but it seems to me that the linker is 
 finding this typeinfo COMDAT in Core first, rather than somewhere else, 
 and is thereby forcing the inclusion of the rest of it's containing module.
 
 Does moving core.obj to the end of the .lib solve the problem?
 


Heya Eric

That's what it seems like and (as noted above) core.obj is already the 
very last one added to the lib ;)

The only way to resolve at this point is to remove core.obj entirely.

Feb 21 2007

↑ ↓ ← → Pragma <ericanderton yahoo.removeme.com> writes:

Frits van Bommel wrote:
 kris wrote:
 Pragma wrote:
 Sorry if I'm stating the obvious, but it seems to me that the linker 
 is finding this typeinfo COMDAT in Core first, rather than somewhere 
 else, and is thereby forcing the inclusion of the rest of it's 
 containing module.

 Does moving core.obj to the end of the .lib solve the problem?


 Heya Eric

 That's what it seems like and (as noted above) core.obj is already the 
 very last one added to the lib ;)

 The only way to resolve at this point is to remove core.obj entirely.


 Did you try putting it at the front of the lib? You never know, maybe it 
 picks the last one instead of the first one.
 
 Unless it just happens to be the only module to define 
 _D12TypeInfo_AAa6__initZ ...


I thought about that too, but that doesn't seem to be the case. As Kris also
stated, the lib compiles when Core is 
removed.  That implies that the TypeInfo mentioned lives elsewhere - and it
very likely does, as it's a "char[char[]]".

Just a hunch: does the .lib's dictionary play a role in OPTLINK's use of
finding COMDAT symbol matches in a .lib file? 
Maybe there's some non-.obj-order-dependent behavior going on between the two.

-- 
- EricAnderton at yahoo

Feb 21 2007

↑ ↓ ← → kris <foo bar.com> writes:

Pragma wrote:
 Frits van Bommel wrote:
 
 kris wrote:

 Pragma wrote:

 Sorry if I'm stating the obvious, but it seems to me that the linker 
 is finding this typeinfo COMDAT in Core first, rather than somewhere 
 else, and is thereby forcing the inclusion of the rest of it's 
 containing module.

 Does moving core.obj to the end of the .lib solve the problem?


 Heya Eric

 That's what it seems like and (as noted above) core.obj is already 
 the very last one added to the lib ;)

 The only way to resolve at this point is to remove core.obj entirely.



 Did you try putting it at the front of the lib? You never know, maybe 
 it picks the last one instead of the first one.

 Unless it just happens to be the only module to define 
 _D12TypeInfo_AAa6__initZ ...


 
 I thought about that too, but that doesn't seem to be the case. As Kris 
 also stated, the lib compiles when Core is removed.  That implies that 
 the TypeInfo mentioned lives elsewhere - and it very likely does, as 
 it's a "char[char[]]".
 


Perhaps a question is this: why the heck is that symbol exposed from 
Core, when it should instead be exposed via the TypeInfo class for 
char[][] instead ... linked via the TypeInfo classes in Object.d? A 
large number of those are present in every D executable.

Feb 21 2007

↑ ↓ ← → kris <foo bar.com> writes:

Walter Bright wrote:
 kris wrote:
 
 Perhaps a question is this: why the heck is that symbol exposed from 
 Core, when it should instead be exposed via the TypeInfo class for 
 char[][] instead ... linked via the TypeInfo classes in Object.d? A 
 large number of those are present in every D executable.


 
 The standard TypeInfo's in Phobos cover only basic types and arrays of 
 basic types. The rest are generated by the compiler.


Yep, I've seen that to be the case. However, the current strategy 
clearly leads to a somewhat haphazard mechanism for resolving such 
symbols: in this case, the exe is dogged by a suite of code that it 
doesn't want or need ... all for the sake of a typedef init?

Sure, we don't want such things being replicated all over the place, but 
I think it has been shown that the current approach is unrealistic in 
practice.

Isn't there some way to isolate the typeinfo such that only a segment is 
linked, rather than the entire "hosting" module (the one that just 
happened to be found first in the lib) ?

Feb 21 2007

↑ ↓ ← → Walter Bright <newshound digitalmars.com> writes:

Frits van Bommel wrote:
 kris wrote:
 Isn't there some way to isolate the typeinfo such that only a segment 
 is linked, rather than the entire "hosting" module (the one that just 
 happened to be found first in the lib) ?




No, the linker deals with .obj files as a unit.

 The obvious solution would be to always generate typeinfo even if it can 
 be determined imported modules will already supply it. The current 
 approach seems to confuse the linker, causing it to link in unrelated 
 objects that happen to supply the symbol even though the compiler 
 "meant" for another object file to supply it.


I wish to be precise - there is no "seems" or "confuse" with linking. It 
simply follows the algorithm I outlined previously - have an unresolved 
symbol, find the first .obj module in the library which resolves it. It 
does this in a loop until there are no further unreferenced symbols.

Most of the complexity in a linker stems from:

1) trying to make it fast
2) the over-complicated .obj file format

Conceptually, it is a very simple program.

 Yes, that will "bloat" object files, but the current approach apparently 
 bloats applications. Care to guess which are distributed most often? ;)


TypeInfo's are only going to grow, and this could create gigantic obj files.

Feb 21 2007

↑ ↓ ← → Derek Parnell <derek nomail.afraid.org> writes:

Walter,
do we (the developer community) have a problem here? 

If yes, will you be actively trying to find a satisfactory resolution in
the near future?

On Wed, 21 Feb 2007 13:22:09 -0800, Walter Bright wrote:

 Frits van Bommel wrote:
 kris wrote:
 Isn't there some way to isolate the typeinfo such that only a segment 
 is linked, rather than the entire "hosting" module (the one that just 
 happened to be found first in the lib) ?




 No, the linker deals with .obj files as a unit.


This has been pointed out a few times now; if any single item in an .OBJ
file is referenced in the program, the whole .OBJ file is linked into the
executable.

This implies that in order to make small executable files, we need to
ensure that .OBJ files are as atomic as possible and to minimize references
to other modules. Yes, these are at conflict with each other so a
compromise made be made somehow.

A better link editor would be able to only link in the portions of the .OBJ
file that are needed, but until someone writes a replacement for OptLink,
we are pretty well stuck with Walter's approach.
 
 The obvious solution would be to always generate typeinfo even if it can 
 be determined imported modules will already supply it. The current 
 approach seems to confuse the linker, causing it to link in unrelated 
 objects that happen to supply the symbol even though the compiler 
 "meant" for another object file to supply it.


 I wish to be precise - there is no "seems" or "confuse" with linking. It 
 simply follows the algorithm I outlined previously - have an unresolved 
 symbol, find the first .obj module in the library which resolves it. It 
 does this in a loop until there are no further unreferenced symbols.


Walter, I know that you are not going to change OptLink, so this next
question is purely theoretical ... instead of finding the 'first' object
file that resolves it, is there a better algorithm ... maybe the smallest
object file that resolves it, or ... I don't know ... but it might be worth
thinking about.

 
 Most of the complexity in a linker stems from:
 
 1) trying to make it fast


How fast is fast enough?

 2) the over-complicated .obj file format


Can we improve the OBJ file format?

 Conceptually, it is a very simple program.


And that might be a part of the problem.
 
 Yes, that will "bloat" object files, but the current approach apparently 
 bloats applications. Care to guess which are distributed most often? ;)


 TypeInfo's are only going to grow, and this could create gigantic obj files.


So, have we got a problem or not?

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Justice for David Hicks!"
22/02/2007 10:01:57 AM

Feb 21 2007

↑ ↓ ← → Walter Bright <newshound digitalmars.com> writes:

Derek Parnell wrote:
 If yes, will you be actively trying to find a satisfactory resolution in
 the near future?


I posted some suggestions to Kris.

 This has been pointed out a few times now; if any single item in an .OBJ
 file is referenced in the program, the whole .OBJ file is linked into the
 executable.


That's right. COMDATs make things slightly more complicated, as 
unreferenced COMDATs get discarded by the linker.

 This implies that in order to make small executable files, we need to
 ensure that .OBJ files are as atomic as possible and to minimize references
 to other modules. Yes, these are at conflict with each other so a
 compromise made be made somehow.
 
 A better link editor would be able to only link in the portions of the .OBJ
 file that are needed, but until someone writes a replacement for OptLink,
 we are pretty well stuck with Walter's approach.


It's important to work with existing tools (like linkers and 
librarians), which (among other things) helps ensure that D programs can 
link with the output of other compilers (like gcc).

 Walter, I know that you are not going to change OptLink, so this next
 question is purely theoretical ... instead of finding the 'first' object
 file that resolves it, is there a better algorithm ... maybe the smallest
 object file that resolves it, or ... I don't know ... but it might be worth
 thinking about.


The 'smallest' doesn't do what you ask, either, because even the 
smallest obj file could contain a reference to something big.

 Most of the complexity in a linker stems from:
 1) trying to make it fast




It's never fast enough. I know a fellow who made his fortune just 
writing a faster linker than MS-LINK. (You can guess the name of that 
linker!) Borland based their whole company's existence on fast 
compile-link times. Currently, ld is pig slow, it's a big bottleneck on 
the edit-compile-link-debug cycle on Linux.


 2) the over-complicated .obj file format




Only if we want to write a replacement for every tool out there that 
manipulates object files, and if we want to give up linking with the 
output of C compilers (or any other compilers).


 Conceptually, it is a very simple program.




Might be, but there also shouldn't be any confusion or mystery about 
what it's doing. Understanding how it works makes it possible to build a 
professional quality library. You can't really escape understanding it - 
and there's no reason to, it *is* a simple program.


 Yes, that will "bloat" object files, but the current approach apparently 
 bloats applications. Care to guess which are distributed most often? ;)




 So, have we got a problem or not?


Given limited resources, we have to deal with what we have.

Feb 21 2007

↑ ↓ ← → Bill Baxter <dnewsgroup billbaxter.com> writes:

Anders F Björklund wrote:
 John Reimer wrote:
 
 That's not a good argument. ld is pig slow?  I'm sorry but I don't get
 that.  It works; it works as intended; and, strangely, I don't hear 
 people
 complain about its apparent lack of speed.
 So what if a linker is blitzingly fast. If it's outdated and broken,
 there's not much to get excited about.  I'll choose the slow working one
 any day.


 I've find OPTLINK to hang and crash a lot when linking the wxD programs
 on Windows XP. But every time I try to reproduce it, it goes away... :-(
 
 So now I just run the "make" like three times in a row, and it usually
 succeeds in building everything. And yeah, it's rather fast in doing so.
 
 But I prefer the MinGW gdc/ld, since it works the first time but slower?
 (well that and that I have problems getting DMC to work with SDL / GL)


I see hangs occasionally even for small programs.  Even on single files 
compiled with dmd -run.  Every time it happens if I Ctrl-C kill it and 
run the same command again, everything is fine.  Frequency is maybe like 
1 out of every 50 compiles.

--bb

Feb 22 2007

↑ ↓ ← → Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:

Walter Bright wrote:
 Walter Bright wrote:
 Bill Baxter wrote:
 I see hangs occasionally even for small programs.  Even on single 
 files compiled with dmd -run.  Every time it happens if I Ctrl-C kill 
 it and run the same command again, everything is fine.  Frequency is 
 maybe like 1 out of every 50 compiles.


 I've never seen optlink crash except on known cases where there's a 
 gigantic amount of static data. If you've got more conventional cases, 
 please post a bug report with a reproducible case.


 I'd forgotten, there is a problem with optlink running on multicore 
 machines. There's supposed to be a way to tell Windows to run an exe 
 using only one core, but I can't think of it at the moment.


Is it a threaded application? (That would explain why the error is sporadic)
If so, the obvious workaround would be to tell it to only use one thread.
If not, why does multicore matter?

Feb 23 2007

↑ ↓ ← → Walter Bright <newshound digitalmars.com> writes:

Frits van Bommel wrote:
 Walter Bright wrote:
 Walter Bright wrote:
 Bill Baxter wrote:
 I see hangs occasionally even for small programs.  Even on single 
 files compiled with dmd -run.  Every time it happens if I Ctrl-C 
 kill it and run the same command again, everything is fine.  
 Frequency is maybe like 1 out of every 50 compiles.




 machines. There's supposed to be a way to tell Windows to run an exe 
 using only one core, but I can't think of it at the moment.


 Is it a threaded application?


Yes.

(That would explain why the error is sporadic)


Yes.

 If so, the obvious workaround would be to tell it to only use one thread.
 If not, why does multicore matter?


The way mutexes are done for multi core is different than for single 
core. I think optlink multithreads correctly only for single core machines.

Feb 23 2007

↑ ↓ ← → Bill Baxter <dnewsgroup billbaxter.com> writes:

Walter Bright wrote:
 Walter Bright wrote:
 Bill Baxter wrote:
 I see hangs occasionally even for small programs.  Even on single 
 files compiled with dmd -run.  Every time it happens if I Ctrl-C kill 
 it and run the same command again, everything is fine.  Frequency is 
 maybe like 1 out of every 50 compiles.


 I've never seen optlink crash except on known cases where there's a 
 gigantic amount of static data. If you've got more conventional cases, 
 please post a bug report with a reproducible case.


 I'd forgotten, there is a problem with optlink running on multicore 
 machines. There's supposed to be a way to tell Windows to run an exe 
 using only one core, but I can't think of it at the moment.


Hmm, one of my machines is core duo, so that could maybe be it.

--bb

Feb 23 2007

↑ ↓ ← → Bill Baxter <dnewsgroup billbaxter.com> writes:

Walter Bright wrote:
 Bill Baxter wrote:
 I see hangs occasionally even for small programs.  Even on single 
 files compiled with dmd -run.  Every time it happens if I Ctrl-C kill 
 it and run the same command again, everything is fine.  Frequency is 
 maybe like 1 out of every 50 compiles.


 I've never seen optlink crash except on known cases where there's a 
 gigantic amount of static data. If you've got more conventional cases, 
 please post a bug report with a reproducible case.


Hmm, well it's not crashing just hanging.  And it may not be optlink but 
dmd.  I do all my work on an external usb/firewire drive so that could 
be an issue (like optlink waiting for a file lock to be released?). 
Also it doesn't happen very often.  Maybe 1 out of 50 is too high an 
estimate.  More like once every few days (and I recompile things a lot).

It always works if I Ctrl-C and try again, so a repro will be difficult.

Other possibilities -- I also started using a cmd wrapper called 
"Console" about the same time as getting into D, so maybe that could 
also be related I suppose. [http://sourceforge.net/projects/console/]. 
It is a bit flaky at times.

In short it's really not a big issue.  Just something I see 
occasionally.  I posted it mostly to see if anyone else was silently 
seeing the same sorts of things.  If no-one else has seen this then it's 
most likely due to something in my particular setup.

--bb

Feb 23 2007

↑ ↓ ← → David Gileadi <foo bar.com> writes:

Bill Baxter wrote:
 Walter Bright wrote:
 Bill Baxter wrote:
 I see hangs occasionally even for small programs.  Even on single 
 files compiled with dmd -run.  Every time it happens if I Ctrl-C kill 
 it and run the same command again, everything is fine.  Frequency is 
 maybe like 1 out of every 50 compiles.


 I've never seen optlink crash except on known cases where there's a 
 gigantic amount of static data. If you've got more conventional cases, 
 please post a bug report with a reproducible case.


 Hmm, well it's not crashing just hanging.  And it may not be optlink but 
 dmd.  I do all my work on an external usb/firewire drive so that could 
 be an issue (like optlink waiting for a file lock to be released?). Also 
 it doesn't happen very often.  Maybe 1 out of 50 is too high an 
 estimate.  More like once every few days (and I recompile things a lot).
 
 It always works if I Ctrl-C and try again, so a repro will be difficult.
 
 Other possibilities -- I also started using a cmd wrapper called 
 "Console" about the same time as getting into D, so maybe that could 
 also be related I suppose. [http://sourceforge.net/projects/console/]. 
 It is a bit flaky at times.
 
 In short it's really not a big issue.  Just something I see 
 occasionally.  I posted it mostly to see if anyone else was silently 
 seeing the same sorts of things.  If no-one else has seen this then it's 
 most likely due to something in my particular setup.
 
 --bb


I've seen it before compiling the wxD samples.  I'm running a Pentium 4 
hyperthreaded.  As you say, it's a hang, it's occasional, but I wouldn't 
say 1 in 50--it bites me just about every time I compile all the wxD 
samples, so more like 1 in 15 for me.

-Dave

Feb 26 2007

↑ ↓ ← → Sean Kelly <sean f4.ca> writes:

John Reimer wrote:
 On Wed, 21 Feb 2007 16:12:10 -0800, Walter Bright wrote:
 
 Derek Parnell wrote:
 Most of the complexity in a linker stems from:
 1) trying to make it fast




 writing a faster linker than MS-LINK. (You can guess the name of that 
 linker!) Borland based their whole company's existence on fast 
 compile-link times. Currently, ld is pig slow, it's a big bottleneck on 
 the edit-compile-link-debug cycle on Linux.


 That's not a good argument. ld is pig slow?  I'm sorry but I don't get
 that.  It works; it works as intended; and, strangely, I don't hear people
 complain about its apparent lack of speed. 
 
 So what if a linker is blitzingly fast. If it's outdated and broken,
 there's not much to get excited about.  I'll choose the slow working one
 any day.


Ideally, perhaps a linker could provide both options: link fast and 
potentially bloat the exe or link carefully (and slowly) for a lean exe. 
  I'd use the fast link for debugging and the slow link for releases. 
Assuming, of course, that the linker were reliable enough that there was 
no risk of changing app behavior between the two.


Sean

Feb 22 2007

↑ ↓ ← → "Kristian Kilpi" <kjkilpi gmail.com> writes:

On Thu, 22 Feb 2007 18:32:18 +0200, Frits van Bommel  
<fvbommel REMwOVExCAPSs.nl> wrote:

 Sean Kelly wrote:
 Ideally, perhaps a linker could provide both options: link fast and  
 potentially bloat the exe or link carefully (and slowly) for a lean  
 exe.  I'd use the fast link for debugging and the slow link for  
 releases. Assuming, of course, that the linker were reliable enough  
 that there was no risk of changing app behavior between the two.


 That might not be the case here: if a module's object file is pulled in,  
 that module's static constructors and destructors are called at runtime,  
 right? So if different modules are pulled in with those options,  
 different static constructors/destructors get called.
 (Same goes for unit tests, if enabled, by the way)


Hmm, yes, but how that's different from the today's situation? Currently  
the linker chooses *arbitrary* object modules that happen to contain the  
needed typeinfo.

Feb 22 2007

↑ ↓ ← → Sean Kelly <sean f4.ca> writes:

Kristian Kilpi wrote:
 On Thu, 22 Feb 2007 18:32:18 +0200, Frits van Bommel 
 <fvbommel REMwOVExCAPSs.nl> wrote:
 
 Sean Kelly wrote:
 Ideally, perhaps a linker could provide both options: link fast and 
 potentially bloat the exe or link carefully (and slowly) for a lean 
 exe.  I'd use the fast link for debugging and the slow link for 
 releases. Assuming, of course, that the linker were reliable enough 
 that there was no risk of changing app behavior between the two.


 That might not be the case here: if a module's object file is pulled 
 in, that module's static constructors and destructors are called at 
 runtime, right? So if different modules are pulled in with those 
 options, different static constructors/destructors get called.
 (Same goes for unit tests, if enabled, by the way)


 Hmm, yes, but how that's different from the today's situation? Currently 
 the linker chooses *arbitrary* object modules that happen to contain the 
 needed typeinfo.


Because as long as the list of dependencies remains unchanged, the same 
arbitrary choices should be made.


Sean

Feb 22 2007

↑ ↓ ← → "Kristian Kilpi" <kjkilpi gmail.com> writes:

On Thu, 22 Feb 2007 22:08:46 +0200, Sean Kelly <sean f4.ca> wrote:

 Kristian Kilpi wrote:
 On Thu, 22 Feb 2007 18:32:18 +0200, Frits van Bommel  
 <fvbommel REMwOVExCAPSs.nl> wrote:

 Sean Kelly wrote:
 Ideally, perhaps a linker could provide both options: link fast and  
 potentially bloat the exe or link carefully (and slowly) for a lean  
 exe.  I'd use the fast link for debugging and the slow link for  
 releases. Assuming, of course, that the linker were reliable enough  
 that there was no risk of changing app behavior between the two.


 That might not be the case here: if a module's object file is pulled  
 in, that module's static constructors and destructors are called at  
 runtime, right? So if different modules are pulled in with those  
 options, different static constructors/destructors get called.
 (Same goes for unit tests, if enabled, by the way)


 Currently the linker chooses *arbitrary* object modules that happen to  
 contain the needed typeinfo.


 Because as long as the list of dependencies remains unchanged, the same  
 arbitrary choices should be made.


 Sean


Well yes, except there is no guarantees of that, in the specs I mean.  
Another linker may (and likely will) produce a different result. And the  
same can happen when a library is rebuild. The order of object modules  
affect how the linker will choose modules to be linked in.

Feb 22 2007

↑ ↓ ← → jcc7 <technocrat7 gmail.com> writes:

== Quote from Frits van Bommel (fvbommel REMwOVExCAPSs.nl)'s article
kris wrote:
Isn't there some way to isolate the typeinfo such that only a
segment is linked, rather than the entire "hosting" module (the
one that just happened to be found first in the lib) ?

can be determined imported modules will already supply it. The
current approach seems to confuse the linker, causing it to link in
unrelated objects that happen to supply the symbol even though the
compiler "meant" for another object file to supply it.

Yes, that will "bloat" object files, but the current approach
apparently bloats applications. Care to guess which are distributed
most often? ;)

I think your idea could work. It makes sense to me, but I'd like to go one
better:
Let's have DMD postpone creating TypeInfo until an .exe or .dll is being created
and only include them with the .obj for the "main" module (i.e. the module with
the main or DllMain function).

Surely, the compiler can figure out which TypeInfo's it needs at the point of
compiling an .exe or .dll. If not, even if we have to wait for linker to spit
out
a list of missing TypeInfo's and then generate the TypeInfo (trial-and-error), I
think that would be a small price to pay for eliminating all of this bloat of
unneeded module that Kris has discovered.

There seems to be a lot more concern around here about .exe-size than there is
about the speed of compiling and linking. Let's fix what's broken -- even if we
have to give up a little compile/link speed

My idea seems too obvious to be the solution. Does anyone else think this would
work?

By the way, Kris has a thorough description of the problem here:
http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=49257

jcc7

Feb 22 2007

↑ ↓ ← → Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:

jcc7 wrote:
== Quote from Frits van Bommel (fvbommel REMwOVExCAPSs.nl)'s article
kris wrote:
Isn't there some way to isolate the typeinfo such that only a
segment is linked, rather than the entire "hosting" module (the
one that just happened to be found first in the lib) ?

Yes, that will "bloat" object files, but the current approach
apparently bloats applications. Care to guess which are distributed
most often? ;)

I think your idea could work. It makes sense to me, but I'd like to go one
better:
Let's have DMD postpone creating TypeInfo until an .exe or .dll is being
created
and only include them with the .obj for the "main" module (i.e. the module with
the main or DllMain function).

Not all libraries may have a DllMain, IIRC it's completely optional.
On Windows it's required for D DLLs if you want to use the GC from
within the DLL, or have static constructors/destructors in the DLL --
but otherwise you may get by without. I think if you write C-style D you
may well get away without it.

Surely, the compiler can figure out which TypeInfo's it needs at the point of
compiling an .exe or .dll.

Not necessarily. Any modules that are linked in but not called by other
modules (e.g. code only reachable from static constructors and/or static
destructors) may not be seen when main/DllMain is compiled, if there
even is one of these (see above point about DllMain being optional).

If not, even if we have to wait for linker to spit out
a list of missing TypeInfo's and then generate the TypeInfo (trial-and-error),
I
think that would be a small price to pay for eliminating all of this bloat of
unneeded module that Kris has discovered.

This would mean you can't "manually" link stuff together, using
optlink/ld/whatever directly. I don't know how many people want to do
this, but Walter has made it pretty clear he wants to be able to use a
generic linker[1] (i.e. one that doesn't require specialized knowledge
of D) and I agree with that.
Consider this: if every (or even more than one) language required a
special way of linking, that would mean you couldn't link together code
written in those languages without writing a linker (or perhaps wrapper)
that supports both...
Though arguably the situation with DMD/Windows is already worse when it
comes to that, since almost nobody else uses OMF anymore...

I agree.

My idea seems too obvious to be the solution. Does anyone else think this
would work?

For above-mentioned reasons, I don't think it will work for all
(corner)cases.

By the way, Kris has a thorough description of the problem here:
http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=49257

I've seen it.

Feb 22 2007

↑ ↓ ← → jcc7 <technocrat7 gmail.com> writes:

Frits van Bommel (fvbommel REMwOVExCAPSs.nl) wrote:
 jcc7 wrote:
 Frits van Bommel (fvbommel REMwOVExCAPSs.nl) wrote:
 kris wrote:
 Isn't there some way to isolate the typeinfo such that only a
 segment is linked, rather than the entire "hosting" module (the
 one that just happened to be found first in the lib) ?


 it can be determined imported modules will already supply it. The
 current approach seems to confuse the linker, causing it to link
 in unrelated objects that happen to supply the symbol even though
 the compiler "meant" for another object file to supply it.

 Yes, that will "bloat" object files, but the current approach
 apparently bloats applications. Care to guess which are
 distributed most often? ;)


 I think your idea could work. It makes sense to me, but I'd like
 to go one better:




(By the way, this topic is mostly over-my-head, so I'll probably have to quit
offering ideas pretty soon lest I embarrass myselft more than I already may
have.)

 Let's have DMD postpone creating TypeInfo until an .exe or .dll is
 being created and only include them with the .obj for the "main"
 module (i.e. the module with the main or DllMain function).


 On Windows it's required for D DLLs if you want to use the GC from
 within the DLL, or have static constructors/destructors in the DLL
 but otherwise you may get by without. I think if you write C-style D
 you may well get away without it.


Well, I don't want to prevent anyone from playing by their own rules, so my
proposed TypeInfo-postponing compiler could have a switch to add the TypeInfo as
it's compiling any arbitrary code into an .obj file. But in usual circumstances,
I'd think that the TypeInfo would only be needed when producing an .exe or .dll.

 Surely, the compiler can figure out which TypeInfo's it needs at
 the point of compiling an .exe or .dll.


 other modules (e.g. code only reachable from static constructors
 and/or static destructors) may not be seen when main/DllMain is
 compiled, if there even is one of these (see above point about
 DllMain being optional).


I don't see how static constructors and/or destructors interferes with the
compiler detecting which TypeInfo's would be necessary, but I don't think such a
problem would be insurmountable. Perhaps, it'd be a question of "Is it worth the
effort?".

But then again, I don't know much about what the compiler and linker do "under
the
hood". It's mostly a black box for me. But from reading Walter and Kris discuss
the issues involved, I'm convinced there has to be a less haphazard way for DMD
and optlink to interact.


 If not, even if we have to wait for linker to spit out
 a list of missing TypeInfo's and then generate the TypeInfo
 (trial-and-error), I think that would be a small price to pay for
 eliminating all of this bloat of unneeded module that Kris has
 discovered.


 optlink/ld/whatever directly. I don't know how many people want to
 do this, but Walter has made it pretty clear he wants to be able to
 use a generic linker[1] (i.e. one that doesn't require specialized
 knowledge of D) and I agree with that.


Isn't there still a question of whether anyone has found a "generic linker" for
OMF (other than OptLink) that can work with DMD anyway?


 Consider this: if every (or even more than one) language required a
 special way of linking, that would mean you couldn't link together
 code written in those languages without writing a linker (or perhaps
 wrapper) that supports both...


Yeah, that doesn't sound like fun.


 Though arguably the situation with DMD/Windows is already worse when
 it comes to that, since almost nobody else uses OMF anymore...


Right. We seem to be on our own when it comes to using OMF.

I think we're mostly trying to find a fix for the problem with the OMF files
generated by DMD right now. Apparently, GDC doesn't have these same problems (or
if GDC does have linker problems, Walter isn't the one responsible for fixing
them). So I think the problem is limited to using DMD's OMF files on Windows.
(Doesn't DMD on Linux use ELF? I think that's the case.)


 For above-mentioned reasons, I don't think it will work for all
 (corner)cases.


You might be right, but I haven't given up hope yet.


jcc7

Feb 22 2007

↑ ↓ ← → Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:

jcc7 wrote:
 Frits van Bommel (fvbommel REMwOVExCAPSs.nl) wrote:
 jcc7 wrote:
 Surely, the compiler can figure out which TypeInfo's it needs at
 the point of compiling an .exe or .dll.


 other modules (e.g. code only reachable from static constructors
 and/or static destructors) may not be seen when main/DllMain is
 compiled, if there even is one of these (see above point about
 DllMain being optional).


 I don't see how static constructors and/or destructors interferes with the
 compiler detecting which TypeInfo's would be necessary, but I don't think such
a
 problem would be insurmountable.


How static constructors could interfere:
---
module selfcontained;

static this() {
     // some code that requires TypeInfo not used in other modules
     // (including Phobos), perhaps for a type defined in this module.
}
---
(Change 'static this' to 'static ~this' or 'unittest' for similar problems)

If this module isn't imported (directly or indirectly) from the file 
defining main() the compiler can't possibly know what TypeInfo needs to 
be generated for it when compiling main(), simply because it doesn't 
parse that file when pointed at the file containing main().

Yes, this could be "fixed" by having the module containing main() import 
all such modules, but it shouldn't have to. We shouldn't need to work 
around toolchain limitations, especially if there's a way to make it 
Just Work(TM).

 Perhaps, it'd be a question of "Is it worth the
 effort?".


It'll be worth the effort when one of _your_ projects fail to compile 
because of it :P.

 But then again, I don't know much about what the compiler and linker do "under
the
 hood". It's mostly a black box for me. But from reading Walter and Kris discuss
 the issues involved, I'm convinced there has to be a less haphazard way for DMD
 and optlink to interact.


Like I've mentioned earlier: I'm pretty sure this problem would go away 
entirely if the compiler simply generated all TypeInfo used in the 
module. If that generates larger intermediate object files I'm okay with 
that. In fact, that was how I thought it worked until I started reading 
about this problem...

 If not, even if we have to wait for linker to spit out
 a list of missing TypeInfo's and then generate the TypeInfo
 (trial-and-error), I think that would be a small price to pay for
 eliminating all of this bloat of unneeded module that Kris has
 discovered.


 optlink/ld/whatever directly. I don't know how many people want to
 do this, but Walter has made it pretty clear he wants to be able to
 use a generic linker[1] (i.e. one that doesn't require specialized
 knowledge of D) and I agree with that.


 Isn't there still a question of whether anyone has found a "generic linker" for
 OMF (other than OptLink) that can work with DMD anyway?


I believe I mentioned that a bit later ;).

[snip special linker discussion]
 Though arguably the situation with DMD/Windows is already worse when
 it comes to that, since almost nobody else uses OMF anymore...


 Right. We seem to be on our own when it comes to using OMF.


Well, it seems OpenWatcom supports it. From what I've read here the 
linker doesn't like DMD object files though. Walter claims it's buggy. I 
don't know enough about OMF to say one way or the other.

 I think we're mostly trying to find a fix for the problem with the OMF files
 generated by DMD right now. Apparently, GDC doesn't have these same problems
(or
 if GDC does have linker problems, Walter isn't the one responsible for fixing
 them). So I think the problem is limited to using DMD's OMF files on Windows.
 (Doesn't DMD on Linux use ELF? I think that's the case.)


Yes, DMD/Linux uses ELF. It just calls ld (through gcc) to link instead 
of using optlink.

I'm not sure if ld (or the mingw port of it) can use ELF to create 
Windows executables, but if it can that may be an option: just switch to 
ELF entirely and trash optlink. (this paragraph wasn't entirely serious, 
in case you hadn't noticed :P)

Feb 22 2007

↑ ↓ ← → Justin C Calvarese <technocrat7 gmail.com> writes:

Frits van Bommel wrote:
 jcc7 wrote:
 Frits van Bommel (fvbommel REMwOVExCAPSs.nl) wrote:
 jcc7 wrote:
 Surely, the compiler can figure out which TypeInfo's it needs at
 the point of compiling an .exe or .dll.


 other modules (e.g. code only reachable from static constructors
 and/or static destructors) may not be seen when main/DllMain is
 compiled, if there even is one of these (see above point about
 DllMain being optional).


 I don't see how static constructors and/or destructors interferes with 
 the
 compiler detecting which TypeInfo's would be necessary, but I don't 
 think such a
 problem would be insurmountable.


 How static constructors could interfere:
 ---
 module selfcontained;
 
 static this() {
     // some code that requires TypeInfo not used in other modules
     // (including Phobos), perhaps for a type defined in this module.
 }
 ---
 (Change 'static this' to 'static ~this' or 'unittest' for similar problems)
 
 If this module isn't imported (directly or indirectly) from the file 
 defining main() the compiler can't possibly know what TypeInfo needs to 
 be generated for it when compiling main(), simply because it doesn't 
 parse that file when pointed at the file containing main().


Oh, I thought the .obj file included mentions of things that are needed, 
but not contained in a particular .obj. I thought that's why "Error 42: 
Symbol Undefined" will appear if I don't give the compiler enough source 
files.

If that's not right, that would be a serious flaw in my proposal.


 Yes, this could be "fixed" by having the module containing main() import 
 all such modules, but it shouldn't have to. We shouldn't need to work 
 around toolchain limitations, especially if there's a way to make it 
 Just Work(TM).
 
 Perhaps, it'd be a question of "Is it worth the
 effort?".


 It'll be worth the effort when one of _your_ projects fail to compile 
 because of it :P.


Well, of course, my plan is contingent upon my projects successfully 
compiling. ;)

[snip my older thoughts]

 Like I've mentioned earlier: I'm pretty sure this problem would go away 
 entirely if the compiler simply generated all TypeInfo used in the 
 module. If that generates larger intermediate object files I'm okay with 
 that. In fact, that was how I thought it worked until I started reading 
 about this problem...


If that'd solve the problem, that'd be an improvement from the status quo.

But I had the understanding that there is a problem with the linker 
picking the TypeInfo from an arbitrary .obj (such as a large module that 
isn't needed for a particular program)? I'm afraid the linker might 
continue to choose an inappropriate TypeInfo. Or do you plan for all of 
the TypeInfo's to be unique, thus probably still bloating the .exe (but 
in a different way)?


 [snip special linker discussion]
 Though arguably the situation with DMD/Windows is already worse when
 it comes to that, since almost nobody else uses OMF anymore...


 Right. We seem to be on our own when it comes to using OMF.


 Well, it seems OpenWatcom supports it. From what I've read here the 
 linker doesn't like DMD object files though. Walter claims it's buggy. I 
 don't know enough about OMF to say one way or the other.


Well, it doesn't really matter to me if DMD continues to use OMF if the 
format doesn't cause a bunch of bloat or other broken features. But I 
still wonder Walter needs to stay so close to the "official" format if 
DMC/DMD's OMF doesn't seems to be compatible with any other compiler.

[snip my older thoughts]

 Yes, DMD/Linux uses ELF. It just calls ld (through gcc) to link instead 
 of using optlink.
 
 I'm not sure if ld (or the mingw port of it) can use ELF to create 
 Windows executables, but if it can that may be an option: just switch to 
 ELF entirely and trash optlink. (this paragraph wasn't entirely serious, 
 in case you hadn't noticed :P)


I suspect the option of ELF output would be welcomed by OMF's harshest 
critics. Not that I know anything about ELF.

-- 
jcc7

Feb 22 2007

↑ ↓ ← → Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:

Justin C Calvarese wrote:
[snip]
 Oh, I thought the .obj file included mentions of things that are needed, 
 but not contained in a particular .obj. I thought that's why "Error 42: 
 Symbol Undefined" will appear if I don't give the compiler enough source 
 files.
 
 If that's not right, that would be a serious flaw in my proposal.


Oh, you want the compiler to parse the .obj files to generate some extra 
stuff before it tries to link?
So basically that would be Walters "Put all the TypeInfo instantiations 
into one special module and import it everywhere you need TypeInfo", but 
automated by the compiler when it gets to the point of linking an 
executable?
I'm not sure if that's workable either since I'm pretty sure it'll need 
to parse the source files to generate the TypeInfo, at the very least 
for user-defined types. At that point it may not even have access to 
those source files, especially when someone is using a closed-source 
library (via .di header files). This is also a problem with what I 
thought you meant, by the way.

[snip]
 Perhaps, it'd be a question of "Is it worth the
 effort?".


 It'll be worth the effort when one of _your_ projects fail to compile 
 because of it :P.


 Well, of course, my plan is contingent upon my projects successfully 
 compiling. ;)


Thought so :).

 Like I've mentioned earlier: I'm pretty sure this problem would go 
 away entirely if the compiler simply generated all TypeInfo used in 
 the module. If that generates larger intermediate object files I'm 
 okay with that. In fact, that was how I thought it worked until I 
 started reading about this problem...


 If that'd solve the problem, that'd be an improvement from the status quo.
 
 But I had the understanding that there is a problem with the linker 
 picking the TypeInfo from an arbitrary .obj (such as a large module that 
 isn't needed for a particular program)? I'm afraid the linker might 
 continue to choose an inappropriate TypeInfo. Or do you plan for all of 
 the TypeInfo's to be unique, thus probably still bloating the .exe (but 
 in a different way)?


I was kind of hoping the compiler wouldn't go looking for a new object 
file that includes a symbol it hasn't seen yet if it's present in the 
object file that needs it.
After seeing some more discussion on hash tables used in OMF linkers, 
I'm not sure if that's what would happen. It'd depend on how the linker 
is implemented, I guess.

 [snip special linker discussion]
 Though arguably the situation with DMD/Windows is already worse when
 it comes to that, since almost nobody else uses OMF anymore...


 Right. We seem to be on our own when it comes to using OMF.


 Well, it seems OpenWatcom supports it. From what I've read here the 
 linker doesn't like DMD object files though. Walter claims it's buggy. 
 I don't know enough about OMF to say one way or the other.


 Well, it doesn't really matter to me if DMD continues to use OMF if the 
 format doesn't cause a bunch of bloat or other broken features.


Well, obviously it doesn't really matter if it works (and works well) :P.
Unfortunately, that doesn't currently seem to be the case...

 But I
 still wonder Walter needs to stay so close to the "official" format if
 DMC/DMD's OMF doesn't seems to be compatible with any other compiler.


Those bugs in OW might be fixed someday, or someone might re-implement 
OMF for the GNU toolchain (IIRC it was removed). Or someone else might 
want to implement a better linker for the Digital Mars compilers.
It's usually better to stick to published standards where they exist.

 Yes, DMD/Linux uses ELF. It just calls ld (through gcc) to link 
 instead of using optlink.

 I'm not sure if ld (or the mingw port of it) can use ELF to create 
 Windows executables, but if it can that may be an option: just switch 
 to ELF entirely and trash optlink. (this paragraph wasn't entirely 
 serious, in case you hadn't noticed :P)


 I suspect the option of ELF output would be welcomed by OMF's harshest 
 critics. Not that I know anything about ELF.


Thinking about it, I seem to recall "PE operations on non-PE file" to be 
a common error when I was trying to link ELF files on Windows 
(cross-compiled). MinGW-ld didn't seem to like ELF object files.
That was when linking to ELF binaries though, not to Windows executables.

Feb 22 2007

↑ ↓ ← → jcc7 <technocrat7 gmail.com> writes:

== Quote from Frits van Bommel (fvbommel REMwOVExCAPSs.nl)'s article
 Justin C Calvarese wrote:
 [snip]
 Oh, I thought the .obj file included mentions of things that are
 needed, but not contained in a particular .obj. I thought that's
 why "Error 42: Symbol Undefined" will appear if I don't give the
 compiler enough source files.

 If that's not right, that would be a serious flaw in my proposal.


 extra stuff before it tries to link?
 So basically that would be Walters "Put all the TypeInfo
 instantiations into one special module and import it everywhere you
 need TypeInfo", but automated by the compiler when it gets to the
 point of linking an executable?
 I'm not sure if that's workable either since I'm pretty sure it'll
 need to parse the source files to generate the TypeInfo, at the very
 least for user-defined types. At that point it may not even have
 access to those source files, especially when someone is using a
 closed-source library (via .di header files). This is also a problem
 with what I thought you meant, by the way.


Doh! I forgot that the compiler doesn't read the .obj/.lib files, but just hands
them over to the linker.

Yeah, I'm seeing the big problems with my idea now. It's sounding like a lot
more
work than we could expect Walter or anyone else to undertake. There might still
be
an easy solution (e.g. having the compiler produce a text file that lists the
needed TypeInfo's for each .obj that it outputs), but every solution seems to
bring with it another risk (e.g. too many files floating around that have to be
kept track of).

jcc7

Feb 23 2007

↑ ↓ ← → Daniel Keep <daniel.keep.lists gmail.com> writes:

(I'm just going to interject here because WOW this thread is getting
long; I can't even read the subject line anymore it's gone so far
sideways...)

There's one suggestion I haven't seen yet, so I'll make it:

I assume from the discussion about segment-based linking that it's
possible to pull out one particular section from an object file, and
just link that into the executable.

So, why not make a small modification to OPTLINK such that if a switch
is thrown, and it encounters any missing symbol of the form
/_D11TypeInfo_.*/, then it will link only the segment that symbol is in.
 In other words, it does what it currently does in all cases except
where there's a TypeInfo involved; in which case it links *just* the
TypeInfo, not the whole object file.

This doesn't break compatibility with OMF or any other tool; it's simply
an optimisation for reducing executable bloat in D programs.  This way,
we don't need a new object format, or a whole new linker.

Or have I just got it all wrong? :P

	-- Daniel

-- 
Unlike Knuth, I have neither proven or tried the above; it may not even
make sense.

v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D
i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP  http://hackerkey.com/

Feb 22 2007

↑ ↓ ← → kris <foo bar.com> writes:

Daniel Keep wrote:
 (I'm just going to interject here because WOW this thread is getting
 long; I can't even read the subject line anymore it's gone so far
 sideways...)
 
 There's one suggestion I haven't seen yet, so I'll make it:
 
 I assume from the discussion about segment-based linking that it's
 possible to pull out one particular section from an object file, and
 just link that into the executable.
 
 So, why not make a small modification to OPTLINK such that if a switch
 is thrown, and it encounters any missing symbol of the form
 /_D11TypeInfo_.*/, then it will link only the segment that symbol is in.
  In other words, it does what it currently does in all cases except
 where there's a TypeInfo involved; in which case it links *just* the
 TypeInfo, not the whole object file.
 
 This doesn't break compatibility with OMF or any other tool; it's simply
 an optimisation for reducing executable bloat in D programs.  This way,
 we don't need a new object format, or a whole new linker.
 
 Or have I just got it all wrong? :P
 
 	-- Daniel
 



On the face of it, that sounds like a reaonable solution. One would 
assume it would be legit to pull typeinfo from a segment instead ...

Feb 23 2007

↑ ↓ ← → Dave <Dave_member pathlink.com> writes:

kris wrote:
 Daniel Keep wrote:
 (I'm just going to interject here because WOW this thread is getting
 long; I can't even read the subject line anymore it's gone so far
 sideways...)

 There's one suggestion I haven't seen yet, so I'll make it:

 I assume from the discussion about segment-based linking that it's
 possible to pull out one particular section from an object file, and
 just link that into the executable.

 So, why not make a small modification to OPTLINK such that if a switch
 is thrown, and it encounters any missing symbol of the form
 /_D11TypeInfo_.*/, then it will link only the segment that symbol is in.
  In other words, it does what it currently does in all cases except
 where there's a TypeInfo involved; in which case it links *just* the
 TypeInfo, not the whole object file.

 This doesn't break compatibility with OMF or any other tool; it's simply
 an optimisation for reducing executable bloat in D programs.  This way,
 we don't need a new object format, or a whole new linker.

 Or have I just got it all wrong? :P

     -- Daniel


 
 On the face of it, that sounds like a reaonable solution. One would 
 assume it would be legit to pull typeinfo from a segment instead ...


Great idea if it's feasible... Would it then make sense that the switch be
thrown by default for DMD?

Feb 23 2007

↑ ↓ ← → jcc7 <technocrat7 gmail.com> writes:

== Quote from Daniel Keep (daniel.keep.lists gmail.com)'s article
 (I'm just going to interject here because WOW this thread is getting
 long; I can't even read the subject line anymore it's gone so far
 sideways...)

 There's one suggestion I haven't seen yet, so I'll make it:

 I assume from the discussion about segment-based linking that it's
 possible to pull out one particular section from an object file, and
 just link that into the executable.

 So, why not make a small modification to OPTLINK such that if a
 switch is thrown, and it encounters any missing symbol of the form
 /_D11TypeInfo_.*/, then it will link only the segment that symbol is
 in.  In other words, it does what it currently does in all cases
 except where there's a TypeInfo involved; in which case it links
 *just* the TypeInfo, not the whole object file.

 This doesn't break compatibility with OMF or any other tool; it's
 simply
 an optimisation for reducing executable bloat in D programs.  This
 way, we don't need a new object format, or a whole new linker.

 Or have I just got it all wrong? :P
 	-- Daniel


I don't know enough about how linkers work to know if OPTLINK can just include a
TypeInfo segment like that, but it sounds like a good idea to me. I think it's
more feasible than my idea was (since Frits helped me see some serious
challenges
with my idea).

And if it works, it should directly address Kris's problem.

jcc7

Feb 23 2007