www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Big problem with Small programs

reply kris <foo bar.com> writes:
I have a helloworld.d program. I compile it on both Win32 and on linux. 
On linux, the executable is 72KB whereas on Win32 it is 141KB. Why is this?

Turns out that D programs importing comprehensive Win32 D headers will 
wind up with executable space occupied for *all* constants and *all* 
struct initializers for the darned Win32 decls; where none of them are 
actually used.

That amounts to ~70KB of junk in the executable -- almost a full 100% 
increase in size beyond what it should be.

Does OptLink remove this? I've had no success with it. Walter? Can you 
help with this?

- Kris
Jan 23 2007
parent reply Walter Bright <newshound digitalmars.com> writes:
kris wrote:
 I have a helloworld.d program. I compile it on both Win32 and on linux. 
 On linux, the executable is 72KB whereas on Win32 it is 141KB. Why is this?
 
 Turns out that D programs importing comprehensive Win32 D headers will 
 wind up with executable space occupied for *all* constants and *all* 
 struct initializers for the darned Win32 decls; where none of them are 
 actually used.
 
 That amounts to ~70KB of junk in the executable -- almost a full 100% 
 increase in size beyond what it should be.
 
 Does OptLink remove this? I've had no success with it. Walter? Can you 
 help with this?
 
 - Kris

Try using enums instead of const variables, they don't take up any space in the object file.
Jan 23 2007
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Walter Bright wrote:
 kris wrote:
 I have a helloworld.d program. I compile it on both Win32 and on 
 linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. 
 Why is this?

 Turns out that D programs importing comprehensive Win32 D headers will 
 wind up with executable space occupied for *all* constants and *all* 
 struct initializers for the darned Win32 decls; where none of them are 
 actually used.

 That amounts to ~70KB of junk in the executable -- almost a full 100% 
 increase in size beyond what it should be.

 Does OptLink remove this? I've had no success with it. Walter? Can you 
 help with this?

Try using enums instead of const variables, they don't take up any space in the object file.

That shaved off maybe 30k. I think the remaining 40k are mostly static struct initializers, all of which are unused in the final app. As optlink is an optimizing linker, shouldn't it discard whatever is unused by the application? Or does that only apply to certain classes of information? Sean
Jan 23 2007
parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Sean Kelly wrote:
 Walter Bright wrote:
 Try using enums instead of const variables, they don't take up any 
 space in the object file.

That shaved off maybe 30k. I think the remaining 40k are mostly static struct initializers, all of which are unused in the final app. As optlink is an optimizing linker, shouldn't it discard whatever is unused by the application? Or does that only apply to certain classes of information?

I'm not sure about Windows, but I just tried it on Linux. It seems struct init data is stored in .rodata instead of some separate section[1]. That means if *any* read-only data from a module is referenced, all of its struct initializers are kept, presuming optlink works similarly to 'ld --gc-sections'. [1]: Like .gnu.linkonce.*, where TypeInfo init data is stored (as well as functions).
Jan 23 2007
prev sibling next sibling parent reply kris <foo bar.com> writes:
Walter Bright wrote:
 kris wrote:
 
 I have a helloworld.d program. I compile it on both Win32 and on 
 linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. 
 Why is this?

 Turns out that D programs importing comprehensive Win32 D headers will 
 wind up with executable space occupied for *all* constants and *all* 
 struct initializers for the darned Win32 decls; where none of them are 
 actually used.

 That amounts to ~70KB of junk in the executable -- almost a full 100% 
 increase in size beyond what it should be.

 Does OptLink remove this? I've had no success with it. Walter? Can you 
 help with this?

 - Kris

Try using enums instead of const variables, they don't take up any space in the object file.

Thanks, but that makes only a dent in the overhead. One might assume that dmd and optlink have no facilities to remove unreferenced entities such as these? If so, doesn't this seem to be a notable weak point in the toolchain? Why is this not an issue with C/C++ ? 1) C used true compile-time constants via #define. From your comment above, it would appear enum is the equivalent in D, rather than const int. Probably worth a note in the doc about that? 2) C does not have struct inititalizers. Why has this not come to light before? Well, the Phobos Win32 headers are pretty darned slim compared with the real mcCoy. There's been a lot of clamour for 'real' win32 headers for years. Now that we have them, we're penalized for using 'em :p Surely there must be some way for the linker to identify and discard unused entities? Another approach would be to make the win32 headers be .di files, such that no obj files are generated. However, this breaks quickly whenever one uses a struct requiring an initializer (link errors). Enum is only a small partial resolution here, and we need to complete the picture somehow. Any other suggestions, Walter? Can you tag the structs internally somehow to make the linker discard the unused ones? Can you explain exactly what's going on, such that others might be able to suggest something? - Kris
Jan 23 2007
next sibling parent kris <foo bar.com> writes:
kris wrote:
 Walter Bright wrote:
 
 kris wrote:

 I have a helloworld.d program. I compile it on both Win32 and on 
 linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. 
 Why is this?

 Turns out that D programs importing comprehensive Win32 D headers 
 will wind up with executable space occupied for *all* constants and 
 *all* struct initializers for the darned Win32 decls; where none of 
 them are actually used.

 That amounts to ~70KB of junk in the executable -- almost a full 100% 
 increase in size beyond what it should be.

 Does OptLink remove this? I've had no success with it. Walter? Can 
 you help with this?

 - Kris

Try using enums instead of const variables, they don't take up any space in the object file.

Thanks, but that makes only a dent in the overhead. One might assume that dmd and optlink have no facilities to remove unreferenced entities such as these? If so, doesn't this seem to be a notable weak point in the toolchain? Why is this not an issue with C/C++ ? 1) C used true compile-time constants via #define. From your comment above, it would appear enum is the equivalent in D, rather than const int. Probably worth a note in the doc about that? 2) C does not have struct inititalizers. Why has this not come to light before? Well, the Phobos Win32 headers are pretty darned slim compared with the real mcCoy. There's been a lot of clamour for 'real' win32 headers for years. Now that we have them, we're penalized for using 'em :p Surely there must be some way for the linker to identify and discard unused entities? Another approach would be to make the win32 headers be .di files, such that no obj files are generated. However, this breaks quickly whenever one uses a struct requiring an initializer (link errors). Enum is only a small partial resolution here, and we need to complete the picture somehow. Any other suggestions, Walter? Can you tag the structs internally somehow to make the linker discard the unused ones? Can you explain exactly what's going on, such that others might be able to suggest something?

For example (off the top of my head to get things rolling), what would be the ramifications of placing each struct initializer into its own data-segment? Would the linker manage to remove the unused ones then? If so, could a compiler flag be used to enable that, or could it be done behind the scenes? Other suggestions?
Jan 23 2007
prev sibling parent reply "Todor Totev" <umbra.tenebris list.ru> writes:
 Walter Bright wrote:
   Try using enums instead of const variables, they don't take up any =


 space in the object file.

Thanks, but that makes only a dent in the overhead.

Actually using enums instead of consts have a very nice side effect. Consider this declaration from win32: HANDLE CreateFileW(LPCWSTR, DWORD, DWORD, LPSECURITY_ATTRIBUTES, DWORD, = = DWORD, HANDLE); Now, which invocation is correct? CreateFile("a", OPEN_EXISTING, FILE_SHARE_READ, null, GENERIC_READ ...) or CreateFile("a", OPEN_EXISTING, GENERIC_READ, null, FILE_SHARE_READ ...) or even CreateFile("a", GENERIC_READ, OPEN_EXISTING, null, FILE_SHARE_READ ...)?= The compiler happily accepts all of them. Using enums, we have: enum FILE_SHARE { READ =3D 1, WRITE =3D 2, BOTH =3D READ | WRITE } enum DISPOSITION { CREATE_ALWAYS =3D 1, OPEN_EXISTING =3D 2 } void CreateFile(char[] fileName, DISPOSITION disposition, FILE_SHARE = share) {} int main() { // compiler allows unknown flags CreateFile("filename", cast(DISPOSITION)4, FILE_SHARE.READ); // when we swap the arguments by mistake CreateFile("filename", FILE_SHARE.READ, DISPOSITION.CREATE_ALWAYS); return 0; } This way if I make an error the compiler will very helpfully tell me what exactly is happening. Consider: CreateFont has 14 parameters, if the IDE does not help me i'd make an error when I use it, trust me On the other hand, when a new version of Windows come with expanded opti= ons I can just use the new numbers without breaking anything while the d = package is updated. The dotnet framework uses enums and I really like the idea. Regards, Todor
Jan 23 2007
next sibling parent reply kris <foo bar.com> writes:
Todor Totev wrote:
 Walter Bright wrote:

   Try using enums instead of const variables, they don't take up any  
 space in the object file.

Thanks, but that makes only a dent in the overhead.

Actually using enums instead of consts have a very nice side effect. Consider this declaration from win32: HANDLE CreateFileW(LPCWSTR, DWORD, DWORD, LPSECURITY_ATTRIBUTES, DWORD, DWORD, HANDLE); Now, which invocation is correct? CreateFile("a", OPEN_EXISTING, FILE_SHARE_READ, null, GENERIC_READ ...) or CreateFile("a", OPEN_EXISTING, GENERIC_READ, null, FILE_SHARE_READ ...) or even CreateFile("a", GENERIC_READ, OPEN_EXISTING, null, FILE_SHARE_READ ...)? The compiler happily accepts all of them. Using enums, we have: enum FILE_SHARE { READ = 1, WRITE = 2, BOTH = READ | WRITE } enum DISPOSITION { CREATE_ALWAYS = 1, OPEN_EXISTING = 2 } void CreateFile(char[] fileName, DISPOSITION disposition, FILE_SHARE share) {} int main() { // compiler allows unknown flags CreateFile("filename", cast(DISPOSITION)4, FILE_SHARE.READ); // when we swap the arguments by mistake CreateFile("filename", FILE_SHARE.READ, DISPOSITION.CREATE_ALWAYS); return 0; } This way if I make an error the compiler will very helpfully tell me what exactly is happening. Consider: CreateFont has 14 parameters, if the IDE does not help me i'd make an error when I use it, trust me On the other hand, when a new version of Windows come with expanded options I can just use the new numbers without breaking anything while the d package is updated. The dotnet framework uses enums and I really like the idea. Regards, Todor

Certainly, and I think you'll find many who agree, along with some who can't bear to type a few extra characters :) But that's a different type of issue than the one we're facing. I wish you had a fix for that one :p
Jan 23 2007
parent "Todor Totev" <umbra.tenebris list.ru> writes:
 Certainly, and I think you'll find many who agree, along with some who  
 can't bear to type a few extra characters :)

 But that's a different type of issue than the one we're facing. I wish  
 you had a fix for that one :p
 No, nada, nein, не, όχι :-(

Jan 23 2007
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
Todor Totev wrote:
 Walter Bright wrote:
   Try using enums instead of const variables, they don't take up any 
 space in the object file.

Thanks, but that makes only a dent in the overhead.

Actually using enums instead of consts have a very nice side effect. Consider this declaration from win32: HANDLE CreateFileW(LPCWSTR, DWORD, DWORD, LPSECURITY_ATTRIBUTES, DWORD, DWORD, HANDLE); Now, which invocation is correct? CreateFile("a", OPEN_EXISTING, FILE_SHARE_READ, null, GENERIC_READ ...) or CreateFile("a", OPEN_EXISTING, GENERIC_READ, null, FILE_SHARE_READ ...) or even CreateFile("a", GENERIC_READ, OPEN_EXISTING, null, FILE_SHARE_READ ...)? The compiler happily accepts all of them. Using enums, we have: enum FILE_SHARE { READ = 1, WRITE = 2, BOTH = READ | WRITE } enum DISPOSITION { CREATE_ALWAYS = 1, OPEN_EXISTING = 2 } void CreateFile(char[] fileName, DISPOSITION disposition, FILE_SHARE share) {} int main() { // compiler allows unknown flags CreateFile("filename", cast(DISPOSITION)4, FILE_SHARE.READ); // when we swap the arguments by mistake CreateFile("filename", FILE_SHARE.READ, DISPOSITION.CREATE_ALWAYS); return 0; } This way if I make an error the compiler will very helpfully tell me what exactly is happening. Consider: CreateFont has 14 parameters, if the IDE does not help me i'd make an error when I use it, trust me On the other hand, when a new version of Windows come with expanded options I can just use the new numbers without breaking anything while the d package is updated. The dotnet framework uses enums and I really like the idea.

I agree that this is a great feature of D. And if you're willing to adapt the Win32 headers to use this convention then I would gladly accept them :-) Sean
Jan 23 2007
parent Don Clugston <dac nospam.com.au> writes:
Sean Kelly wrote:
 Todor Totev wrote:
 Walter Bright wrote:
   Try using enums instead of const variables, they don't take up any 
 space in the object file.

Thanks, but that makes only a dent in the overhead.

Actually using enums instead of consts have a very nice side effect. Consider this declaration from win32: HANDLE CreateFileW(LPCWSTR, DWORD, DWORD, LPSECURITY_ATTRIBUTES, DWORD, DWORD, HANDLE); Now, which invocation is correct? CreateFile("a", OPEN_EXISTING, FILE_SHARE_READ, null, GENERIC_READ ...) or CreateFile("a", OPEN_EXISTING, GENERIC_READ, null, FILE_SHARE_READ ...) or even CreateFile("a", GENERIC_READ, OPEN_EXISTING, null, FILE_SHARE_READ ...)? The compiler happily accepts all of them. Using enums, we have: enum FILE_SHARE { READ = 1, WRITE = 2, BOTH = READ | WRITE } enum DISPOSITION { CREATE_ALWAYS = 1, OPEN_EXISTING = 2 } void CreateFile(char[] fileName, DISPOSITION disposition, FILE_SHARE share) {} int main() { // compiler allows unknown flags CreateFile("filename", cast(DISPOSITION)4, FILE_SHARE.READ); // when we swap the arguments by mistake CreateFile("filename", FILE_SHARE.READ, DISPOSITION.CREATE_ALWAYS); return 0; } This way if I make an error the compiler will very helpfully tell me what exactly is happening. Consider: CreateFont has 14 parameters, if the IDE does not help me i'd make an error when I use it, trust me On the other hand, when a new version of Windows come with expanded options I can just use the new numbers without breaking anything while the d package is updated. The dotnet framework uses enums and I really like the idea.

I agree that this is a great feature of D. And if you're willing to adapt the Win32 headers to use this convention then I would gladly accept them :-) Sean

As far as possible, we did that, for the ones that actually are enums. The problem is, most of those things aren't actually enums! They are complicated bitfields that get ORed together.
Jan 24 2007
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
Walter Bright wrote:
 kris wrote:
 I have a helloworld.d program. I compile it on both Win32 and on 
 linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. 
 Why is this?

 Turns out that D programs importing comprehensive Win32 D headers will 
 wind up with executable space occupied for *all* constants and *all* 
 struct initializers for the darned Win32 decls; where none of them are 
 actually used.

 That amounts to ~70KB of junk in the executable -- almost a full 100% 
 increase in size beyond what it should be.

 Does OptLink remove this? I've had no success with it. Walter? Can you 
 help with this?

Try using enums instead of const variables, they don't take up any space in the object file.

For reference, the "hello world" app test was compiled with Bud, which basically just recursively compiled all .d files and links the objects together. However, I just tried creating a library out of the Win32 modules and renamed them to .di files to keep Bud from building them, and the resulting app was 90k instead of 144k. I don't suppose you can explain how linking a lib and linking individual object files can have such glaringly different effects? I'm going to try a manual build without Bud and see if the app is 144k now too. Sean
Jan 23 2007
parent reply Sean Kelly <sean f4.ca> writes:
Sean Kelly wrote:
 Walter Bright wrote:
 kris wrote:
 I have a helloworld.d program. I compile it on both Win32 and on 
 linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. 
 Why is this?

 Turns out that D programs importing comprehensive Win32 D headers 
 will wind up with executable space occupied for *all* constants and 
 *all* struct initializers for the darned Win32 decls; where none of 
 them are actually used.

 That amounts to ~70KB of junk in the executable -- almost a full 100% 
 increase in size beyond what it should be.

 Does OptLink remove this? I've had no success with it. Walter? Can 
 you help with this?

Try using enums instead of const variables, they don't take up any space in the object file.

For reference, the "hello world" app test was compiled with Bud, which basically just recursively compiled all .d files and links the objects together. However, I just tried creating a library out of the Win32 modules and renamed them to .di files to keep Bud from building them, and the resulting app was 90k instead of 144k. I don't suppose you can explain how linking a lib and linking individual object files can have such glaringly different effects? I'm going to try a manual build without Bud and see if the app is 144k now too.

I'm beginning to suspect a compiler/linker bug. Here is my first test supplying all modules on one line: -------------------------------------------------------------------------------- C:\code\src\d\test\appsize>dmd -inline -release -O hello.d \code\import\tango\tango\io\Console.d \code\import\tango\tango\sys\Common.d \code\import\tango\tango\sys\win32\Types.d \code\import\tango\tango\sys\win32\Common.d \code\import\tango\tango\io\Buffer.d \code\import\tango\tango\io\Exception.d \code\import\tango\tango\io\model\IBuffer.d \code\import\tango\tango\io\model\IConduit.d \code\import\tango\tango\io\DeviceConduit.d \code\import\tango\tango\io\Conduit.d c:\bin\dmd\bin\..\..\dm\bin\link.exe hello+Console+Common+Types+Common+Buffer+Ex ception+IBuffer+IConduit+DeviceConduit+Conduit,,,user32+kernel32/noi; C:\code\src\d\test>dir hello.exe Volume in drive C is System Volume Serial Number is 58B0-B8C5 Directory of C:\code\src\d\test\appsize 01/23/2007 12:05 PM 141,852 hello.exe -------------------------------------------------------------------------------- And my second test compiling all modules separately and then linking into an app: -------------------------------------------------------------------------------- dmd -inline -release -O -c hello.d dmd -inline -release -O -c \code\import\tango\tango\io\Console.d dmd -inline -release -O -c \code\import\tango\tango\sys\Common.d dmd -inline -release -O -c \code\import\tango\tango\sys\win32\Types.d dmd -inline -release -O -c \code\import\tango\tango\sys\win32\Common.d -ofCommonWin32.obj dmd -inline -release -O -c \code\import\tango\tango\io\Buffer.d dmd -inline -release -O -c \code\import\tango\tango\io\Exception.d dmd -inline -release -O -c \code\import\tango\tango\io\model\IBuffer.d dmd -inline -release -O -c \code\import\tango\tango\io\model\IConduit.d dmd -inline -release -O -c \code\import\tango\tango\io\DeviceConduit.d dmd -inline -release -O -c \code\import\tango\tango\io\Conduit.d C:\code\src\d\test\appsize>dmd hello *.obj c:\bin\dmd\bin\..\..\dm\bin\link.exe hello+*,,,user32+kernel32/noi; C:\code\src\d\test\appsize>dir hello.exe Volume in drive C is System Volume Serial Number is 58B0-B8C5 Directory of C:\code\src\d\test\appsize 01/23/2007 12:13 PM 141,340 hello.exe -------------------------------------------------------------------------------- And now my third test where tango/sys/win32/*.* is compiled into a library (win32.lib) and the lib is linked instead of the module object files: -------------------------------------------------------------------------------- C:\code\src\d\test\appsize>lib -c -n Win32.lib CommonWin32.obj Types.obj Digital Mars Librarian Version 8.00n Copyright (C) Digital Mars 2000-2002 All Rights Reserved www.digitalmars.com Digital Mars Librarian complete. C:\code\src\d\test\appsize>del CommonWin32.obj Types.obj C:\code\src\d\test\appsize>dmd hello *.obj Win32.lib c:\bin\dmd\bin\..\..\dm\bin\link.exe hello+*,,,Win32.lib+user32+kernel32/noi; C:\code\src\d\test\appsize>dir hello.exe Volume in drive C is System Volume Serial Number is 58B0-B8C5 Directory of C:\code\src\d\test\appsize 01/23/2007 12:20 PM 90,140 hello.exe -------------------------------------------------------------------------------- As you can see, simply putting the Win32 object files into a library prior to linking reduced the application size by 51,100 bytes, but separate compilation had the same (bad) result as compiling all modules on one line. I grant that 51k may not be a huge issue for the average Windows app, but it could be a killer for handheld (WinCE) applications. Could you please explain the size difference if this is indeed not a bug? I don't mind making these files .di files and adding a pragma(lib, "Win32.lib"), but this seems like something that must be addressed in a more general manner at some point. Sean
Jan 23 2007
parent reply Walter Bright <newshound digitalmars.com> writes:
Sean Kelly wrote:
 As you can see, simply putting the Win32 object files into a library 
 prior to linking reduced the application size by 51,100 bytes, but 
 separate compilation had the same (bad) result as compiling all modules 
 on one line.

I suggest linking with /MAP, which will generate a .map file listing all the modules linked in, and all the global symbols linked in. It's a lot easier to see what's happening with that than guessing based on the file size.
Jan 23 2007
next sibling parent kris <foo bar.com> writes:
Walter Bright wrote:
 Sean Kelly wrote:
 
 As you can see, simply putting the Win32 object files into a library 
 prior to linking reduced the application size by 51,100 bytes, but 
 separate compilation had the same (bad) result as compiling all 
 modules on one line.

I suggest linking with /MAP, which will generate a .map file listing all the modules linked in, and all the global symbols linked in. It's a lot easier to see what's happening with that than guessing based on the file size.

Done that, Walter. The difference is entirey Win32 struct inits, consts and so on. Remove the int consts, and you still have the struct inits ... almost enough to hand out at Christmas :) What shall we do next?
Jan 23 2007
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
Walter Bright wrote:
 Sean Kelly wrote:
 As you can see, simply putting the Win32 object files into a library 
 prior to linking reduced the application size by 51,100 bytes, but 
 separate compilation had the same (bad) result as compiling all 
 modules on one line.

I suggest linking with /MAP, which will generate a .map file listing all the modules linked in, and all the global symbols linked in. It's a lot easier to see what's happening with that than guessing based on the file size.

hello_small.map: Start Length Name Class 0002:00000000 00011570H _TEXT CODE 32-bit 0002:00011570 00000198H ICODE ICODE 32-bit 0003:00000000 00000004H .CRT$XIA DATA 32-bit 0003:00000010 00000004H .CRT$XIZ DATA 32-bit 0003:00000020 00000004H .CRT$XCA DATA 32-bit 0003:00000030 00000004H .CRT$XCZ DATA 32-bit 0003:00000040 00000004H .CRT$XPA DATA 32-bit 0003:00000050 00000004H .CRT$XPZ DATA 32-bit 0003:00000060 00000004H .CRT$XTA DATA 32-bit 0003:00000070 00000004H .CRT$XTZ DATA 32-bit 0003:00000074 00000000H IMP__DATA IMP__DATA 32-bit 0003:00000080 00005100H _DATA DATA 32-bit 0003:00005180 00000000H FMB DATA 32-bit 0003:00005180 00000024H FM DATA 32-bit 0003:000051A4 00000000H FME DATA 32-bit 0003:000051A4 00000000H XIB DATA 32-bit 0003:000051A4 0000001CH XI DATA 32-bit 0003:000051C0 00000000H XIE DATA 32-bit 0003:000051C0 00000000H XCB DATA 32-bit 0003:000051C0 00000014H XC DATA 32-bit 0003:000051D4 00000000H XCE DATA 32-bit 0003:000051D4 00000000H XIFCB DATA 32-bit 0003:000051D4 00000004H XIFU DATA 32-bit 0003:000051D8 00000000H XIFL DATA 32-bit 0003:000051D8 00000004H XIFM DATA 32-bit 0003:000051DC 00000000H XIFCE DATA 32-bit 0003:000051E0 00000000H CONST CONST 32-bit 0003:000051E0 00000000H EEND ENDBSS 32-bit 0003:000051E0 000016ECH _BSS BSS 32-bit 0003:000068CC 00000000H XOB BSS 32-bit 0003:000068CC 00000004H XO BSS 32-bit 0003:000068D0 00000000H XOE BSS 32-bit 0003:000068D0 00000000H XOFB BSS 32-bit 0003:000068D0 00000108H XOF BSS 32-bit 0003:000069D8 00000000H XOFE BSS 32-bit 0003:000069E0 00000419H c_common BSS 32-bit 0003:00006E00 00000000H STACK STACK 32-bit Program entry point at 0000A2AC -------------------------------------------------------------------------------- hello_large.map: Start Length Name Class 0002:00000000 00011570H _TEXT CODE 32-bit 0002:00011570 00000198H ICODE ICODE 32-bit 0003:00000000 00000004H .CRT$XIA DATA 32-bit 0003:00000010 00000004H .CRT$XIZ DATA 32-bit 0003:00000020 00000004H .CRT$XCA DATA 32-bit 0003:00000030 00000004H .CRT$XCZ DATA 32-bit 0003:00000040 00000004H .CRT$XPA DATA 32-bit 0003:00000050 00000004H .CRT$XPZ DATA 32-bit 0003:00000060 00000004H .CRT$XTA DATA 32-bit 0003:00000070 00000004H .CRT$XTZ DATA 32-bit 0003:00000074 00000000H IMP__DATA IMP__DATA 32-bit 0003:00000080 00011660H _DATA DATA 32-bit 0003:000116E0 00000000H FMB DATA 32-bit 0003:000116E0 00000024H FM DATA 32-bit 0003:00011704 00000000H FME DATA 32-bit 0003:00011704 00000000H XIB DATA 32-bit 0003:00011704 0000001CH XI DATA 32-bit 0003:00011720 00000000H XIE DATA 32-bit 0003:00011720 00000000H XCB DATA 32-bit 0003:00011720 00000014H XC DATA 32-bit 0003:00011734 00000000H XCE DATA 32-bit 0003:00011734 00000000H XIFCB DATA 32-bit 0003:00011734 00000004H XIFU DATA 32-bit 0003:00011738 00000000H XIFL DATA 32-bit 0003:00011738 00000004H XIFM DATA 32-bit 0003:0001173C 00000000H XIFCE DATA 32-bit 0003:00011740 00000000H CONST CONST 32-bit 0003:00011740 00000000H EEND ENDBSS 32-bit 0003:00011740 0000499CH _BSS BSS 32-bit 0003:000160DC 00000000H XOB BSS 32-bit 0003:000160DC 00000004H XO BSS 32-bit 0003:000160E0 00000000H XOE BSS 32-bit 0003:000160E0 00000000H XOFB BSS 32-bit 0003:000160E0 00000108H XOF BSS 32-bit 0003:000161E8 00000000H XOFE BSS 32-bit 0003:000161F0 00000419H c_common BSS 32-bit 0003:00016610 00000000H STACK STACK 32-bit Program entry point at 0000A2AC
Jan 23 2007
parent reply Walter Bright <newshound digitalmars.com> writes:
Sean Kelly wrote:
 I suggest linking with /MAP, which will generate a .map file listing 
 all the modules linked in, and all the global symbols linked in. It's 
 a lot easier to see what's happening with that than guessing based on 
 the file size.

hello_small.map:

Please be sure you are linking with the /MAP switch, as it will give all the symbols, not just the segments.
Jan 23 2007
parent reply Sean Kelly <sean f4.ca> writes:
Walter Bright wrote:
 Sean Kelly wrote:
 I suggest linking with /MAP, which will generate a .map file listing 
 all the modules linked in, and all the global symbols linked in. It's 
 a lot easier to see what's happening with that than guessing based on 
 the file size.

hello_small.map:

Please be sure you are linking with the /MAP switch, as it will give all the symbols, not just the segments.

Oops, I missed the leading slash. The output is obviously quite large so I'm just going to note some of the larger blocks that are present in hello_large.map that are missing from hello_small.map: 0003:0001172C _D5tango3sys5win325Types10ACE_HEADER6__initZ 0042372C ... 0003:00002C08 _D5tango3sys5win325Types9cSTARTDOCi 00414C08 0003:00001074 _D5tango3sys5win325Types10RASCS_DONEi 00413074 ... 0003:00014604 _D5tango3sys5win325Types10WINDOWINFO6__initZ 00426604 The first block goes from line 410 to line 6719 in the map file, and the second block goes from line 8047 to line 14559 in the map file. All told, that's 12821 separate constants or initializers in the manually linked version that seem absent from the library-based version.
Jan 23 2007
parent reply Walter Bright <newshound digitalmars.com> writes:
Sean Kelly wrote:
 The first block goes from line 410 to line 6719 in the map file, and the 
 second block goes from line 8047 to line 14559 in the map file.  All 
 told, that's 12821 separate constants or initializers in the manually 
 linked version that seem absent from the library-based version.

That means that, in the library based version, nothing in the explicitly linked .obj files referenced them. If you want to find out what reference is pulling in a particular library module, use lib to remove that library module, relink, and note the undefined reference.
Jan 23 2007
parent reply Sean Kelly <sean f4.ca> writes:
Walter Bright wrote:
 Sean Kelly wrote:
 The first block goes from line 410 to line 6719 in the map file, and 
 the second block goes from line 8047 to line 14559 in the map file.  
 All told, that's 12821 separate constants or initializers in the 
 manually linked version that seem absent from the library-based version.

That means that, in the library based version, nothing in the explicitly linked .obj files referenced them.

If that's true then nothing in either version referenced them. The application is identical in both cases but for some of the objects living in a library in the small case.
 If you want to find out what reference is pulling in a particular 
 library module, use lib to remove that library module, relink, and note 
 the undefined reference.

See above. Sean
Jan 24 2007
parent Sean Kelly <sean f4.ca> writes:
Sean Kelly wrote:
 Walter Bright wrote:
 Sean Kelly wrote:
 The first block goes from line 410 to line 6719 in the map file, and 
 the second block goes from line 8047 to line 14559 in the map file.  
 All told, that's 12821 separate constants or initializers in the 
 manually linked version that seem absent from the library-based version.

That means that, in the library based version, nothing in the explicitly linked .obj files referenced them.

If that's true then nothing in either version referenced them. The application is identical in both cases but for some of the objects living in a library in the small case.

Just some further discussion. After exhaustive experimentation yesterday with Tango the following seems to be true: Using the Bud/Build approach of compiling all modules to object files and then linking them always works, however on Win32 it may result in a surplus of data in the executable. This may actually be legitimate behavior based on how the linker works and how the OMF blocks are segmented in the object files at link time, but it seems odd. Using the library approach of compiling all library code into a static library and then linking it does not always work, but it results in much leaner executables on Win32. Little or no surplus data is added to the executable, including data that is added using the Bud/Build method, even for the exact same application. This may again be legitimate behavior based on how the lib tool splits/joins/moves OMF blocks when assembling the library (assuming it does so), or it may have something to do with the linker having "all the information at its fingertips" when assembling the application. Where the library approach fails is if the library contains template code. Results are inconsistent, but yesterday we were seeing link errors in some cases and GPFs in others just for different library modules using the same template code, all of which work perfectly using the Bud/Build method. For the link errors, as near as we could tell the object files contained all the relevant functions (many of the link errors referred to class TypeInfo instances) but for some reason they didn't appear to be available at link time. It is possible that the lib tool is throwing them out for some reason, but I could not speculate on why. This leaves us in an awkward situation for releasing Tango. Suggesting everyone use Bud/Build seems the most reliable but some may not be happy with large executables or with having to use Bud/Build at all. Assembling everything into a single static library seems unreliable at best, but it will undoubtedly be preferred by many people. At the moment, I suspect we will adopt a hybrid approach out of necessity, placing some of the larger packages (like the Win32 package) in static libraries and linking them either explicitly or implicitly via pragma(lib) (which is problematic in itself since it doesn't work if placed in include files, at least with DMD--Bud may be different, I have yet to test this and plan to do so today). I am hoping that we can produce a small test case for the template issues so we can move to a full library approach once the problems have been addressed. Alternately, perhaps we can find a way to address linking so that less unnecessary data is included while linking objects directly. I'll admit that at this point I'm of half a mind to simply write an OMF linker that takes as long as necessary to produce a working application and be done with it. Out of curiosity, I suppose it would be a substantial amount of work for DMD (and I suppose DMC) to produce COFF object files instead of OMF? And I don't suppose that some work in this area may be necessary for generating 64-bit executables? If there's a chance we could move to a newer object file format for Win32, we would at least gain access to a broader array of tools, even if COFF itself is a bit problematic. Sean
Jan 25 2007