www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Executable size affected by module count?

reply kris <foo bar.com> writes:
Given a (fixed) body of code, it appears that retaining it all within 
one module, and splitting it into multiple modules, results in different 
executable sizes?

There's no real surprise that this would happen, but it's the actual 
difference that is cause for a little concern -- it appears that each 
module consumes 512 bytes minimum. This may actually be a linker thing, 
but perhaps not?

Why would this matter? Well, if you wind up using the 158 modules in the 
  Win32 project, that adds nearly 80KB to an application. Purely in 
module overhead. And, those headers are almost all enum, const, and struct.

This is in addition to the ~70KB of unused initializer from the Win32 
headers, dicussed in the other topic. That's a whole lot of overhead for 
Win32 programs to carry -- especially if the target is mobile devices.

Obviously, you'd be doing something truly serious if you were actually 
using all those header modules! However, Win32 headers are not exactly a 
model in decoupled design (in C also), so you wind up with large numbers 
of them unintentionally.

Any ideas, Walter?
Jan 24 2007
next sibling parent reply Walter Bright <newshound digitalmars.com> writes:
kris wrote:
 There's no real surprise that this would happen, but it's the actual 
 difference that is cause for a little concern -- it appears that each 
 module consumes 512 bytes minimum. This may actually be a linker thing, 
 but perhaps not?

The way to see what's in an object file is to run obj2asm on it. It'll show exactly what's submitted to the linker.
Jan 24 2007
parent reply kris <foo bar.com> writes:
Walter Bright wrote:
 kris wrote:
 
 There's no real surprise that this would happen, but it's the actual 
 difference that is cause for a little concern -- it appears that each 
 module consumes 512 bytes minimum. This may actually be a linker 
 thing, but perhaps not?

The way to see what's in an object file is to run obj2asm on it. It'll show exactly what's submitted to the linker.

I'm aware of that, thanks. Do you think you could comment on why it might be the linker? And how to compensate? You're probably one of the two ppl in the world who know OptLink ... Also: are dmd obj files compatible with any other linker?
Jan 24 2007
next sibling parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
kris wrote:
 Walter Bright wrote:
 kris wrote:

 There's no real surprise that this would happen, but it's the actual 
 difference that is cause for a little concern -- it appears that each 
 module consumes 512 bytes minimum. This may actually be a linker 
 thing, but perhaps not?

The way to see what's in an object file is to run obj2asm on it. It'll show exactly what's submitted to the linker.

I'm aware of that, thanks. Do you think you could comment on why it might be the linker?

Suggesting you look at what's submitted to the linker doesn't necessarily imply the linker is the cause. If it receives object files with more data in it (generated by the compiler) and dutifully links them, the output is still bigger ;).
 Also: are dmd obj files compatible with any other linker?

IIRC OMF (the format of dmd obj files) used to be the "standard" object format on Windows[1], and perhaps other OSs as well. That was a while back though. It wouldn't surprise me if optlink is the only recent linker to support it, but even if so you may be able to find some old versions of other linkers that support it. [1]: Or was it still DOS back then? The MS OS at the time, anyway.
Jan 24 2007
parent kris <foo bar.com> writes:
Frits van Bommel wrote:
 kris wrote:
 
 Walter Bright wrote:

 kris wrote:

 There's no real surprise that this would happen, but it's the actual 
 difference that is cause for a little concern -- it appears that 
 each module consumes 512 bytes minimum. This may actually be a 
 linker thing, but perhaps not?

The way to see what's in an object file is to run obj2asm on it. It'll show exactly what's submitted to the linker.

I'm aware of that, thanks. Do you think you could comment on why it might be the linker?

Suggesting you look at what's submitted to the linker doesn't necessarily imply the linker is the cause. If it receives object files with more data in it (generated by the compiler) and dutifully links them, the output is still bigger ;).

That's right. But I'd already looked at the obj file content, and subsequently discounted it :p
Jan 24 2007
prev sibling parent reply Walter Bright <newshound digitalmars.com> writes:
kris wrote:
 Walter Bright wrote:
 kris wrote:

 There's no real surprise that this would happen, but it's the actual 
 difference that is cause for a little concern -- it appears that each 
 module consumes 512 bytes minimum. This may actually be a linker 
 thing, but perhaps not?

The way to see what's in an object file is to run obj2asm on it. It'll show exactly what's submitted to the linker.

I'm aware of that, thanks.

The words used suggested an unfamiliarity with the tools. I believe it's well worth the effort to master what's going on with object files and linking, especially for professional developers, and the tools obj2asm, /MAP, and dumpexe are marvelous aids.
 Do you think you could comment on why it might be the linker? And how to 
 compensate? You're probably one of the two ppl in the world who know 
 OptLink ...

I'd first look at the contents of the .obj file, and see if that is what is expected. I'd also check the optlink instructions http://www.digitalmars.com/ctg/ctgLinkSwitches.html#alignment as there are quite a lot of switches that offer a great deal of control over the linking process.
 Also: are dmd obj files compatible with any other linker?

Any linker that supports the Microsoft OMF format. I know Microsoft linkers dropped support for it when they went to 32 bits, but I am not very familiar with other linkers. Pharlap did, but I think they went out of business.
Jan 24 2007
parent reply kris <foo bar.com> writes:
Walter Bright wrote:
 kris wrote:
 
 Walter Bright wrote:

 kris wrote:

 There's no real surprise that this would happen, but it's the actual 
 difference that is cause for a little concern -- it appears that 
 each module consumes 512 bytes minimum. This may actually be a 
 linker thing, but perhaps not?

The way to see what's in an object file is to run obj2asm on it. It'll show exactly what's submitted to the linker.

I'm aware of that, thanks.

The words used suggested an unfamiliarity with the tools.

Forgive me. They were chosen to cause the least amount of conflict? But, as far as OptLink goes -- yes, I have no clue about it, and there's more switches than a power-station. Best to ask the expert, it would seem?
 I believe it's 
 well worth the effort to master what's going on with object files and 
 linking, especially for professional developers, and the tools obj2asm, 
 /MAP, and dumpexe are marvelous aids.

Thank you. Indeed they are. We've both been using these kinds of tools for approximately the same period of time.
 
 Do you think you could comment on why it might be the linker? And how 
 to compensate? You're probably one of the two ppl in the world who 
 know OptLink ...

I'd first look at the contents of the .obj file, and see if that is what is expected.

Did that first. There's nothing that jumps out. I should note that this was first noticed perhaps two years ago ... it's not something that suddenly changed. But, it has become more important recently; vis-a-vis win32 headers
 I'd also check the optlink instructions 
 http://www.digitalmars.com/ctg/ctgLinkSwitches.html#alignment
 as there are quite a lot of switches that offer a great deal of control 
 over the linking process.

Yes, classic stuff! That's why I'm asking the expert.
 
 Also: are dmd obj files compatible with any other linker?

Any linker that supports the Microsoft OMF format. I know Microsoft linkers dropped support for it when they went to 32 bits, but I am not very familiar with other linkers. Pharlap did, but I think they went out of business.

Thanks. This means it's not exactly feasible to check the concern via another linker.
Jan 24 2007
parent reply jcc7 <technocrat7 gmail.com> writes:
== Quote from kris (foo bar.com)'s article
 Also: are dmd obj files compatible with any other linker?

Any linker that supports the Microsoft OMF format. I know Microsoft linkers dropped support for it when they went to 32 bits, but I am not very familiar with other linkers. Pharlap did, but I think they went out of business.

another linker.

It's probably a longshot that wouldn't work, but it's possible that the Standalone OpenWatcom Tools would be of some use: http://cmeerw.org/prog/owtools/ I haven't tried using any of the tools myself (and the webpage implies that development stopped in November 2003), but I thought I'd point this page out in case it's of use to you. It mentions: "I am currently working on getting OpenWatcom's tools working with Digital Mars C++ to provide an alternative to the already dated Digital Mars tools." And one of the tools is called "Open Watcom OMF Dump Utility". jcc7
Jan 25 2007
next sibling parent kris <foo bar.com> writes:
jcc7 wrote:
 == Quote from kris (foo bar.com)'s article
 
Also: are dmd obj files compatible with any other linker?

Any linker that supports the Microsoft OMF format. I know Microsoft linkers dropped support for it when they went to 32 bits, but I am not very familiar with other linkers. Pharlap did, but I think they went out of business.

Thanks. This means it's not exactly feasible to check the concern via another linker.

It's probably a longshot that wouldn't work, but it's possible that the Standalone OpenWatcom Tools would be of some use: http://cmeerw.org/prog/owtools/ I haven't tried using any of the tools myself (and the webpage implies that development stopped in November 2003), but I thought I'd point this page out in case it's of use to you. It mentions: "I am currently working on getting OpenWatcom's tools working with Digital Mars C++ to provide an alternative to the already dated Digital Mars tools." And one of the tools is called "Open Watcom OMF Dump Utility". jcc7

That's a good pointer -- thanks, jcc7 :)
Jan 25 2007
prev sibling parent Pragma <ericanderton yahoo.removeme.com> writes:
John Reimer wrote:
 dmd's largest problem here continues to be the omf format.  On the
 other hand, dmd would do well to have a new linker and object system to
 help it support new language features.

As someone who has hacked away on OMF handling for a while now, I must agree. It has quite a few strikes against it for prospective D toolchain developers: * Hard to follow documentation that contains errors and makes *lots* of assumptions * Is unsupported by just about everyone else except DigitalMars (no offense intended) * Is *not* 64-bit ready * Contains gobs of legacy cruft (specification moreso than .obj files) * Is limited in how it can represent module inter-dependencies Don't get me wrong. I like DMD and OPTLINK in particular - I would love to see OPTLINK modified to work with ELF files or some new D object file format. -- - EricAnderton at yahoo
Jan 25 2007
prev sibling next sibling parent reply Thomas Kuehne <thomas-dloop kuehne.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

kris schrieb am 2007-01-24:
 Given a (fixed) body of code, it appears that retaining it all within 
 one module, and splitting it into multiple modules, results in different 
 executable sizes?

 There's no real surprise that this would happen, but it's the actual 
 difference that is cause for a little concern -- it appears that each 
 module consumes 512 bytes minimum. This may actually be a linker thing, 
 but perhaps not?

 Any ideas, Walter?

Every non-trivial module contains (numbers are for Linux) _D5module7__arrayZ 23 bytes code, 19 bytes stringtab, 18 bytes symtab _D5module8__assertFiZv 24 bytes code, 23 bytes stringtab, 18 bytes symtab _D5module9__modctorFZv 11+ bytes code, 23 bytes stringtab, 18 bytes symtab In total 177 bytes, after stripping (strip --strip-all) 58 bytes. The minimum overhead of an object file is about 800 bytes, most of those are discared at link time. Thomas -----BEGIN PGP SIGNATURE----- iD8DBQFFt9xsLK5blCcjpWoRAiFWAJ9TP0DUTgcs67gE1XPFwhub90HJlgCfZrxE cmnrlBB68I2DbBUf61ekwY4= =czhH -----END PGP SIGNATURE-----
Jan 24 2007
parent reply Sean Kelly <sean f4.ca> writes:
Thomas Kuehne wrote:
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 kris schrieb am 2007-01-24:
 Given a (fixed) body of code, it appears that retaining it all within 
 one module, and splitting it into multiple modules, results in different 
 executable sizes?

 There's no real surprise that this would happen, but it's the actual 
 difference that is cause for a little concern -- it appears that each 
 module consumes 512 bytes minimum. This may actually be a linker thing, 
 but perhaps not?

 Any ideas, Walter?

Every non-trivial module contains (numbers are for Linux) _D5module7__arrayZ 23 bytes code, 19 bytes stringtab, 18 bytes symtab _D5module8__assertFiZv 24 bytes code, 23 bytes stringtab, 18 bytes symtab

These are the outstanding problem for exposing templates from library code. And I don't understand why they are generated, since it seems like the code will be identical for each instance generated. Couldn't they just have a static definition in the runtime?
 _D5module9__modctorFZv
 11+ bytes code, 23 bytes stringtab, 18 bytes symtab

Only if the module as a static ctor though, right?
 In total 177 bytes, after stripping (strip --strip-all) 58 bytes.
 The minimum overhead of an object file is about 800 bytes,
 most of those are discared at link time.

Thanks for the info. Sean
Jan 24 2007
next sibling parent Thomas Kuehne <thomas-dloop kuehne.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sean Kelly schrieb am 2007-01-24:
 Thomas Kuehne wrote:
 kris schrieb am 2007-01-24:


 _D5module9__modctorFZv
 11+ bytes code, 23 bytes stringtab, 18 bytes symtab

Only if the module as a static ctor though, right?

Also if a class/struct has a static ctor. Thomas -----BEGIN PGP SIGNATURE----- iD8DBQFFt+wfLK5blCcjpWoRAlVJAJ9hCLNd1aHPSAuRQ5pd9LoLqXQdKQCdHNwo symVzMwZP6+zbDfllfI64v0= =eyWh -----END PGP SIGNATURE-----
Jan 24 2007
prev sibling parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Sean Kelly wrote:
 Thomas Kuehne wrote:
 Every non-trivial module contains (numbers are for Linux)
 _D5module7__arrayZ
 23 bytes code, 19 bytes stringtab, 18 bytes symtab

 _D5module8__assertFiZv
 24 bytes code, 23 bytes stringtab, 18 bytes symtab

These are the outstanding problem for exposing templates from library code. And I don't understand why they are generated, since it seems like the code will be identical for each instance generated. Couldn't they just have a static definition in the runtime?

In their current form they're not identical for each module, for the simple reason that the code (after linking) has the reference to the module name string hardcoded. For two extra instructions per call that could be avoided, though at that point you might as well just call _d_assert/_d_arraybounds directly instead of using an intermediary function...
Jan 24 2007
parent reply Sean Kelly <sean f4.ca> writes:
Frits van Bommel wrote:
 Sean Kelly wrote:
 Thomas Kuehne wrote:
 Every non-trivial module contains (numbers are for Linux)
 _D5module7__arrayZ
 23 bytes code, 19 bytes stringtab, 18 bytes symtab

 _D5module8__assertFiZv
 24 bytes code, 23 bytes stringtab, 18 bytes symtab

These are the outstanding problem for exposing templates from library code. And I don't understand why they are generated, since it seems like the code will be identical for each instance generated. Couldn't they just have a static definition in the runtime?

In their current form they're not identical for each module, for the simple reason that the code (after linking) has the reference to the module name string hardcoded. For two extra instructions per call that could be avoided, though at that point you might as well just call _d_assert/_d_arraybounds directly instead of using an intermediary function...

Exactly my point. Why not define _d_assert and _d_arraybounds somewhere and simply call those? Since, as far as I can tell, the function bodies never vary. Sean
Jan 24 2007
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Sean Kelly wrote:
 Frits van Bommel wrote:
 Sean Kelly wrote:
 Thomas Kuehne wrote:
 Every non-trivial module contains (numbers are for Linux)
 _D5module7__arrayZ
 23 bytes code, 19 bytes stringtab, 18 bytes symtab

 _D5module8__assertFiZv
 24 bytes code, 23 bytes stringtab, 18 bytes symtab

These are the outstanding problem for exposing templates from library code. And I don't understand why they are generated, since it seems like the code will be identical for each instance generated. Couldn't they just have a static definition in the runtime?

In their current form they're not identical for each module, for the simple reason that the code (after linking) has the reference to the module name string hardcoded. For two extra instructions per call that could be avoided, though at that point you might as well just call _d_assert/_d_arraybounds directly instead of using an intermediary function...

Exactly my point. Why not define _d_assert and _d_arraybounds somewhere

Please look at phobos/std/asserterror.d and phobos/std/array.d[1]. I didn't just pick those names out of a hat ;). What the generated functions do is basically: asm { push EAX; // caller puts line number there push name_ptr; push name_length; call _d_assert; // or _d_array_bounds } Like I said, this can easily be inlined. Replacing "mov EAX, linenr" with three pushes and using a different address is all it takes... In fact, it would seem calls to _d_assert_msg are already done like this (for asserts with the optional char[] second argument). [1]: It would seem I made a typo, it's _d_array_bounds.
 and simply call those?  Since, as far as I can tell, the function bodies 
 never vary.

Jan 25 2007
parent reply Sean Kelly <sean f4.ca> writes:
Frits van Bommel wrote:
 Sean Kelly wrote:
 Frits van Bommel wrote:
 Sean Kelly wrote:
 Thomas Kuehne wrote:
 Every non-trivial module contains (numbers are for Linux)
 _D5module7__arrayZ
 23 bytes code, 19 bytes stringtab, 18 bytes symtab

 _D5module8__assertFiZv
 24 bytes code, 23 bytes stringtab, 18 bytes symtab

These are the outstanding problem for exposing templates from library code. And I don't understand why they are generated, since it seems like the code will be identical for each instance generated. Couldn't they just have a static definition in the runtime?

In their current form they're not identical for each module, for the simple reason that the code (after linking) has the reference to the module name string hardcoded. For two extra instructions per call that could be avoided, though at that point you might as well just call _d_assert/_d_arraybounds directly instead of using an intermediary function...

Exactly my point. Why not define _d_assert and _d_arraybounds somewhere

Please look at phobos/std/asserterror.d and phobos/std/array.d[1]. I didn't just pick those names out of a hat ;).

It's been too long since I've messed with this portion of Phobos. I knew they rang a bell! :-p
 What the generated functions do is basically:
 asm {
     push EAX;           // caller puts line number there
     push name_ptr;
     push name_length;
     call _d_assert;     // or _d_array_bounds
 }
 Like I said, this can easily be inlined. Replacing "mov EAX, linenr" 
 with three pushes and using a different address is all it takes...
 
 In fact, it would seem calls to _d_assert_msg are already done like this 
 (for asserts with the optional char[] second argument).

Makes perfect sense. Well... doing this would eliminate one of the last quantifiable issues with placing templates in a library, so it has my vote. Sean
Jan 25 2007
next sibling parent kris <foo bar.com> writes:
Sean Kelly wrote:
 Frits van Bommel wrote:
 
 Sean Kelly wrote:

 Frits van Bommel wrote:

 Sean Kelly wrote:

 Thomas Kuehne wrote:

 Every non-trivial module contains (numbers are for Linux)
 _D5module7__arrayZ
 23 bytes code, 19 bytes stringtab, 18 bytes symtab

 _D5module8__assertFiZv
 24 bytes code, 23 bytes stringtab, 18 bytes symtab

These are the outstanding problem for exposing templates from library code. And I don't understand why they are generated, since it seems like the code will be identical for each instance generated. Couldn't they just have a static definition in the runtime?

In their current form they're not identical for each module, for the simple reason that the code (after linking) has the reference to the module name string hardcoded. For two extra instructions per call that could be avoided, though at that point you might as well just call _d_assert/_d_arraybounds directly instead of using an intermediary function...

Exactly my point. Why not define _d_assert and _d_arraybounds somewhere

Please look at phobos/std/asserterror.d and phobos/std/array.d[1]. I didn't just pick those names out of a hat ;).

It's been too long since I've messed with this portion of Phobos. I knew they rang a bell! :-p
 What the generated functions do is basically:
 asm {
     push EAX;           // caller puts line number there
     push name_ptr;
     push name_length;
     call _d_assert;     // or _d_array_bounds
 }
 Like I said, this can easily be inlined. Replacing "mov EAX, linenr" 
 with three pushes and using a different address is all it takes...

 In fact, it would seem calls to _d_assert_msg are already done like 
 this (for asserts with the optional char[] second argument).

Makes perfect sense. Well... doing this would eliminate one of the last quantifiable issues with placing templates in a library, so it has my vote. Sean

Mine too
Jan 25 2007
prev sibling parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Sean Kelly wrote:
 Makes perfect sense.  Well... doing this would eliminate one of the last 
 quantifiable issues with placing templates in a library, so it has my vote.

Well, there's still another bug that seems to crop up in source files with only templates: Good ol' bug #22 (http://d.puremagic.com/issues/show_bug.cgi?id=22) Ran into that one again today. But you should know about it, since you were the one to report it. Though perhaps that one is more of a general templates problem, unrelated to libraries. Still annoying though.
Jan 25 2007
parent Sean Kelly <sean f4.ca> writes:
Frits van Bommel wrote:
 Sean Kelly wrote:
 Makes perfect sense.  Well... doing this would eliminate one of the 
 last quantifiable issues with placing templates in a library, so it 
 has my vote.

Well, there's still another bug that seems to crop up in source files with only templates: Good ol' bug #22 (http://d.puremagic.com/issues/show_bug.cgi?id=22) Ran into that one again today. But you should know about it, since you were the one to report it. Though perhaps that one is more of a general templates problem, unrelated to libraries. Still annoying though.

Good to know it's still there I suppose :-/ I'd gotten rid of all occurrences of it in Tango and wasn't sure if it was still an issue. Sean
Jan 25 2007
prev sibling parent John Reimer <terminal.node gmail.com> writes:
On Thu, 25 Jan 2007 15:49:28 +0000, jcc7 wrote:

 == Quote from kris (foo bar.com)'s article
 Also: are dmd obj files compatible with any other linker?

Any linker that supports the Microsoft OMF format. I know Microsoft linkers dropped support for it when they went to 32 bits, but I am not very familiar with other linkers. Pharlap did, but I think they went out of business.

another linker.

It's probably a longshot that wouldn't work, but it's possible that the Standalone OpenWatcom Tools would be of some use: http://cmeerw.org/prog/owtools/ I haven't tried using any of the tools myself (and the webpage implies that development stopped in November 2003), but I thought I'd point this page out in case it's of use to you. It mentions: "I am currently working on getting OpenWatcom's tools working with Digital Mars C++ to provide an alternative to the already dated Digital Mars tools." And one of the tools is called "Open Watcom OMF Dump Utility". jcc7

That statement has been there for years. I'm not sure if anywone has found any benefit to using the modified openwatcom linker. There doesn't seem to be any ongoing development for dmc compatibility. The one benefit (and primary goal, I think) was the ability to link with coff files, but since dmc/dmd spits out omf only, there's not much advantage beyond that... and then even coff files from different compiler venders are incompatible. Object formats have been a mess on windows for well over a decade... dmd's largest problem here continues to be the omf format. On the other hand, dmd would do well to have a new linker and object system to help it support new language features. Depending on old C technology is restrictive to D's progress. Alas, easier said than done. -JJR
Jan 25 2007