www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - What's causing program bloat?

reply "Kris" <fu bar.com> writes:
I saw the post earlier about the Phobos-based "Hello World" being almost 
300KB, and the note from Sean that Ares does the same thing at around 64KB.

Using Mango, I see the size of the executable has little bearing on the 
amount of code proper ~ instead it seems to be terribly inflated by other 
things. For example, I've seen an executable grow in size by 40KB+ when the 
total instruction/data imported is not even vaguely close to that number ~ 
in that particular case I roughly determined the actual code-size by 
instantiating an alternate instance via templating (char vs dchar, for 
instance), and the executable size went up by just 5KB with each distinct 
templated instance. The latter is within the expected range.

Noticed that importing a module, without otherwise referencing it, will 
generally cause that module to be linked. That might have something to do 
with static/module ctors?

Also noticed that linking via a library causes the executable size to be 
roughly 50% larger than linking without that library. That's with the same 
compile options in both cases.

However, what prompted this post is the following. I added a simple inner 
function and the executable size jumped up by 2KB. Moving the inner function 
outside the containing method, to a regular private one instead, bumps the 
executable size down by 2KB. This is a little troubling, and might be worthy 
of a little TLC?
Nov 23 2005
parent reply "Kris" <fu bar.com> writes:
Just noticed that the library C function frexp(double, int) causes the 
executable to increase by 6KB, when the function itself is but 110 bytes, 
with no other calls and no data references. There seems to be something 
fundamentally wrong.

BTW; the function lives in snn.lib



"Kris" <fu bar.com> wrote in message news:dm2itb$1oso$1 digitaldaemon.com...
I saw the post earlier about the Phobos-based "Hello World" being almost 
300KB, and the note from Sean that Ares does the same thing at around 64KB.

 Using Mango, I see the size of the executable has little bearing on the 
 amount of code proper ~ instead it seems to be terribly inflated by other 
 things. For example, I've seen an executable grow in size by 40KB+ when 
 the total instruction/data imported is not even vaguely close to that 
 number ~ in that particular case I roughly determined the actual code-size 
 by instantiating an alternate instance via templating (char vs dchar, for 
 instance), and the executable size went up by just 5KB with each distinct 
 templated instance. The latter is within the expected range.

 Noticed that importing a module, without otherwise referencing it, will 
 generally cause that module to be linked. That might have something to do 
 with static/module ctors?

 Also noticed that linking via a library causes the executable size to be 
 roughly 50% larger than linking without that library. That's with the same 
 compile options in both cases.

 However, what prompted this post is the following. I added a simple inner 
 function and the executable size jumped up by 2KB. Moving the inner 
 function outside the containing method, to a regular private one instead, 
 bumps the executable size down by 2KB. This is a little troubling, and 
 might be worthy of a little TLC?


 

Nov 23 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dm3ehn$2ebr$1 digitaldaemon.com...
 Just noticed that the library C function frexp(double, int) causes the
 executable to increase by 6KB, when the function itself is but 110 bytes,
 with no other calls and no data references. There seems to be something
 fundamentally wrong.

 BTW; the function lives in snn.lib

Linking with /map will generate a .map file, and you can see all the symbols linked in, along with their sizes. This helps a lot in determining where bloat is coming from.
Nov 26 2005
parent reply "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote in message 
news:dmap2s$19rf$1 digitaldaemon.com...
 "Kris" <fu bar.com> wrote in message 
 news:dm3ehn$2ebr$1 digitaldaemon.com...
 Just noticed that the library C function frexp(double, int) causes the
 executable to increase by 6KB, when the function itself is but 110 bytes,
 with no other calls and no data references. There seems to be something
 fundamentally wrong.

 BTW; the function lives in snn.lib

Linking with /map will generate a .map file, and you can see all the symbols linked in, along with their sizes. This helps a lot in determining where bloat is coming from.

Thanks for the pointer ~ I wish that would help :-( There are no other symbols associated with frexp() since it is completely standalone (as noted above). I checked this by renaming snn.lib, thus causing a list of missing symbols to be produced ~ there was nothing unexpected listed. I then added my own version of frexp(), and the code size dropped by 6KB, as expected. For whatever reason, the linker is including something unrelated/unreferenced. Perhaps due to the "common-segment" notion mentioned before?
Nov 26 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dmatmh$1u1l$1 digitaldaemon.com...
 "Walter Bright" <newshound digitalmars.com> wrote in message
 news:dmap2s$19rf$1 digitaldaemon.com...
 Linking with /map will generate a .map file, and you can see all the
 symbols
 linked in, along with their sizes. This helps a lot in determining where
 bloat is coming from.

Thanks for the pointer ~ I wish that would help :-( There are no other symbols associated with frexp() since it is completely standalone (as noted above). I checked this by renaming snn.lib, thus causing a list of missing symbols to be produced ~ there was nothing unexpected listed. I then added my own version of frexp(), and the code

 dropped by 6KB, as expected.

 For whatever reason, the linker is including something
 unrelated/unreferenced. Perhaps due to the "common-segment" notion

 before?

The .map file will list all the sizes of the sections brought in. This should narrow down where the 6Kb is coming from.
Nov 26 2005
parent reply "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote>
 For whatever reason, the linker is including something
 unrelated/unreferenced. Perhaps due to the "common-segment" notion

 before?

The .map file will list all the sizes of the sections brought in. This should narrow down where the 6Kb is coming from.

FWIW, here the seg map with frexp() Start Length Name Class 0002:00000000 00012EF0H _TEXT CODE 32-bit 0002:00012EF0 0000017AH ICODE ICODE 32-bit 0003:00000000 00000004H .CRT$XIA DATA 32-bit 0003:00000010 00000004H .CRT$XIZ DATA 32-bit 0003:00000020 00000004H .CRT$XCA DATA 32-bit 0003:00000030 00000004H .CRT$XCZ DATA 32-bit 0003:00000040 00000004H .CRT$XPA DATA 32-bit 0003:00000050 00000004H .CRT$XPZ DATA 32-bit 0003:00000060 00000004H .CRT$XTA DATA 32-bit 0003:00000070 00000004H .CRT$XTZ DATA 32-bit 0003:00000074 00000000H IMP__DATA IMP__DATA 32-bit 0003:00000080 00005404H _DATA DATA 32-bit 0003:00005484 00000000H FMB DATA 32-bit 0003:00005484 00000038H FM DATA 32-bit 0003:000054BC 00000000H FME DATA 32-bit 0003:000054BC 00000000H XIB DATA 32-bit 0003:000054BC 00000018H XI DATA 32-bit 0003:000054D4 00000000H XIE DATA 32-bit 0003:000054D4 00000000H XCB DATA 32-bit 0003:000054D4 00000010H XC DATA 32-bit 0003:000054E4 00000000H XCE DATA 32-bit 0003:000054E4 00000000H XIFCB DATA 32-bit 0003:000054E4 00000004H XIFU DATA 32-bit 0003:000054E8 00000000H XIFL DATA 32-bit 0003:000054E8 00000004H XIFM DATA 32-bit 0003:000054EC 00000000H XIFCE DATA 32-bit 0003:000054F0 00000000H CONST CONST 32-bit 0003:000054F0 00000000H EEND ENDBSS 32-bit 0003:000054F0 00001A00H _BSS BSS 32-bit 0003:00006EF0 00000000H XOB BSS 32-bit 0003:00006EF0 00000004H XO BSS 32-bit 0003:00006EF4 00000000H XOE BSS 32-bit 0003:00006EF4 00000000H XOFB BSS 32-bit 0003:00006EF4 00000108H XOF BSS 32-bit 0003:00006FFC 00000000H XOFE BSS 32-bit 0003:00007000 0000041DH c_common BSS 32-bit 0003:00007420 00000000H STACK STACK 32-bit and here it is without frexp() Start Length Name Class 0002:00000000 00011BB0H _TEXT CODE 32-bit 0002:00011BB0 0000017AH ICODE ICODE 32-bit 0003:00000000 00000004H .CRT$XIA DATA 32-bit 0003:00000010 00000004H .CRT$XIZ DATA 32-bit 0003:00000020 00000004H .CRT$XCA DATA 32-bit 0003:00000030 00000004H .CRT$XCZ DATA 32-bit 0003:00000040 00000004H .CRT$XPA DATA 32-bit 0003:00000050 00000004H .CRT$XPZ DATA 32-bit 0003:00000060 00000004H .CRT$XTA DATA 32-bit 0003:00000070 00000004H .CRT$XTZ DATA 32-bit 0003:00000074 00000000H IMP__DATA IMP__DATA 32-bit 0003:00000080 000052D0H _DATA DATA 32-bit 0003:00005350 00000000H FMB DATA 32-bit 0003:00005350 00000038H FM DATA 32-bit 0003:00005388 00000000H FME DATA 32-bit 0003:00005388 00000000H XIB DATA 32-bit 0003:00005388 00000018H XI DATA 32-bit 0003:000053A0 00000000H XIE DATA 32-bit 0003:000053A0 00000000H XCB DATA 32-bit 0003:000053A0 00000010H XC DATA 32-bit 0003:000053B0 00000000H XCE DATA 32-bit 0003:000053B0 00000000H XIFCB DATA 32-bit 0003:000053B0 00000004H XIFU DATA 32-bit 0003:000053B4 00000000H XIFL DATA 32-bit 0003:000053B4 00000004H XIFM DATA 32-bit 0003:000053B8 00000000H XIFCE DATA 32-bit 0003:000053C0 00000000H CONST CONST 32-bit 0003:000053C0 00000000H EEND ENDBSS 32-bit 0003:000053C0 00001A00H _BSS BSS 32-bit 0003:00006DC0 00000000H XOB BSS 32-bit 0003:00006DC0 00000004H XO BSS 32-bit 0003:00006DC4 00000000H XOE BSS 32-bit 0003:00006DC4 00000000H XOFB BSS 32-bit 0003:00006DC4 00000108H XOF BSS 32-bit 0003:00006ECC 00000000H XOFE BSS 32-bit 0003:00006ED0 00000419H c_common BSS 32-bit 0003:000072F0 00000000H STACK STACK 32-bit Does that tell you anything of value? ================================= A related question: The map file tells me that the entire C I/O subsystem (including sprintf, printf, floating point formatting, file I/O, etc, etc) is being linked, even though there are no obvious references to any of it when using a stripped-down version of Ares. It seems that "__acrtused_con" pulls all of that in? Is there a way I can eliminate all the C I/O from the executable? Following is a list of everything linked from the C runtime lib. You can see there's actually very little being used at this point. I'd like to shrink the footprint down a bit further. OPTLINK (R) for Win32 Release 7.50B1 Copyright (C) Digital Mars 1989 - 2001 All Rights Reserved SNN.lib Warning 2: File Not Found SNN.lib C:\D\mango\mango\convert\Integer.obj(Integer) Error 42: Symbol Undefined __ULDIV C:\D\mango\mango\convert\Format.obj(Format) Error 42: Symbol Undefined __fltused template.obj(template) Error 42: Symbol Undefined __acrtused_con C:\D\mango\mango\io\Buffer.obj(Buffer) Error 42: Symbol Undefined _memcpy C:\D\mango\mango\io\Resource.obj(Resource) Error 42: Symbol Undefined __except_list C:\d\dmd\bin\..\lib\phobos.lib(object) Error 42: Symbol Undefined _memcmp C:\d\dmd\bin\..\lib\phobos.lib(gc) Error 42: Symbol Undefined _memset C:\d\dmd\bin\..\lib\phobos.lib(gc) Error 42: Symbol Undefined _malloc C:\d\dmd\bin\..\lib\phobos.lib(deh) Error 42: Symbol Undefined __local_except_handler C:\d\dmd\bin\..\lib\phobos.lib(deh) Error 42: Symbol Undefined _strlen C:\d\dmd\bin\..\lib\phobos.lib(deh) Error 42: Symbol Undefined __global_unwind C:\d\dmd\bin\..\lib\phobos.lib(dmain2) Error 42: Symbol Undefined ___alloca C:\d\dmd\bin\..\lib\phobos.lib(dmain2) Error 42: Symbol Undefined _exit C:\d\dmd\bin\..\lib\phobos.lib(monitor) Error 42: Symbol Undefined _free C:\d\dmd\bin\..\lib\phobos.lib(monitor) Error 42: Symbol Undefined __assert C:\d\dmd\bin\..\lib\phobos.lib(monitor) Error 42: Symbol Undefined _calloc C:\d\dmd\bin\..\lib\phobos.lib(gcx) Error 42: Symbol Undefined _realloc C:\d\dmd\bin\..\lib\phobos.lib(gcx) Error 42: Symbol Undefined _memmove C:\d\dmd\bin\..\lib\phobos.lib(win32) Error 42: Symbol Undefined __end C:\d\dmd\bin\..\lib\phobos.lib(win32) Error 42: Symbol Undefined __xi_a C:\d\dmd\bin\..\lib\phobos.lib(thread) Error 42: Symbol Undefined __beginthreadex OPTLINK : Warning 134: No Start Address --- errorlevel 21
Nov 27 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dmdilh$v33$1 digitaldaemon.com...
 "Walter Bright" <newshound digitalmars.com> wrote>
 For whatever reason, the linker is including something
 unrelated/unreferenced. Perhaps due to the "common-segment" notion

 before?

The .map file will list all the sizes of the sections brought in. This should narrow down where the 6Kb is coming from.

FWIW, here the seg map with frexp()

Link with /map which will list all the symbols in the .map file (not just the segments), along with their addresses. If you really want to do a shrink, it'll be necessary to get comfortable with this. Also, the linker is quite a simple-minded program. It knows nothing about C, or the C standard library, etc. All it does is recursively look at its list of undefined symbols, and look for them in the supplied libraries. If an object file is being included, then it was specified explicitly or it was pulled in from a library because that object file defined a symbol that was undefined.
 A related question:

 The map file tells me that the entire C I/O subsystem (including sprintf,
 printf, floating point formatting, file I/O, etc, etc) is being linked,

 though there are no obvious references to any of it when using a
 stripped-down version of Ares.

 It seems that "__acrtused_con" pulls all of that in? Is there a way I can
 eliminate all the C I/O from the executable?

You can define stubs for the symbols in your own code - that way they don't get pulled in from the library. You can also yank out obj modules from SNN.LIB (using lib.exe) and see what errors happen when linking. I'm not trying to be flip, I'm trying to explain how to figure out how these things work. __acrtused_con pulls in constart.obj from SNN.LIB. The source to it is \dm\src\win32\constart.c.
 Following is a list of everything linked from the C runtime lib. You can

 there's actually very little being used at this point. I'd like to shrink
 the footprint down a bit further.

_exit() pulls in the C I/O subsystem, as it needs to flush & shut it down gracefully. Trying to eliminate all traces of the C I/O subsystem is not so easy, as there tend to be a lot of dependencies on it, distributed throughout. Back in the bad old 16 bit days, this was worth the effort. But now I don't see the point. At most it adds 10k to your final program (and adds nothing to your library). These days, performance matters far more.
Nov 28 2005
parent reply "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote in message 
news:dmgm0u$h4s$1 digitaldaemon.com...
 "Kris" <fu bar.com> wrote in message 
 news:dmdilh$v33$1 digitaldaemon.com...
 "Walter Bright" <newshound digitalmars.com> wrote>
 For whatever reason, the linker is including something
 unrelated/unreferenced. Perhaps due to the "common-segment" notion

 before?

The .map file will list all the sizes of the sections brought in. This should narrow down where the 6Kb is coming from.

FWIW, here the seg map with frexp()

Link with /map which will list all the symbols in the .map file (not just the segments), along with their addresses. If you really want to do a shrink, it'll be necessary to get comfortable with this.

Indeed :-) There was just too much content to post (about 140KB each). Tried to do a diff between them, but the code locations were all different, so pretty much every line was tagged as being different. So I did it the slow and painful way instead. Given the code: extern (C) double frexp(double, int*); void main() { int i; frexp(1.0, &i); } I did a manual diff between frexp() and no frexp() ~ it apparently requires the following symbols: 0002:00009538 _FDIV_DETECT 0040B538 0002:0000E23A Imp _LCMapStringA 24 (KERNEL32.DLL.LCMapStringA) 0003:00002780 __8087 00413780 0002:00009580 __8087_init 0040B580 0003:00002780 __80x87 00413780 0002:0000DC2A __87TOPSW 0040FC2A 0002:0000DC3A __DBLINT87 0040FC3A 0002:0000DC5B __DBLLNG87 0040FC5B 0002:0000DC2F __DBLTO87 0040FC2F 0002:00006FCA __DBLULNG 00408FCA 0002:0000DC26 __DTST87 0040FC26 0002:0000DBD0 __FCOMPP 0040FBD0 0002:0000951C __FDIVP 0040B51C 0002:000055B4 __FEEXCEPT 004075B4 0002:000055C8 __FEROUND 004075C8 0002:00009158 __FLOATCVT 0040B158 0002:0000DC77 __FLTTO87 0040FC77 0002:0000DBEE __FTEST0 0040FBEE 0002:0000DC0E __FTEST 0040FC0E 0002:00005D7C __SCANFLOAT 00407D7C 0002:000084E8 __STI_io_ctor 0040A4E8 0002:00006FB5 __ULNGDBL 00408FB5 0002:000094BC __WDOSIGN 0040B4BC 0002:00006914 __WSCANFLOAT 00408914 0002:00006F5B ___dtype 00408F5B 0002:000055D4 ___floatfmt 004075D4 0002:0000415D ___flt_rounds 0040615D 0002:00009354 ___fpclassify_ld 0040B354 0003:00002BBC ___locale_chars 00413BBC 0003:000025D8 ___locale_decimal_const 004135D8 0003:000025DC ___locale_decpoint 004135DC 0003:00005034 ___pscanfloat 00416034 0002:000060C8 ___wfloatfmt 004080C8 0003:00005030 ___wpfloatfmt 00416030 0003:0000502C ___wpscanfloat 0041602C 0002:0000DC83 __clear87 0040FC83 0002:0000DC8C __control87 0040FC8C 0002:00003CA0 __fltused 00405CA0 0002:0000DCB3 __fpreset 0040FCB3 0001:00000570 __imp__LCMapStringA 24 00401570 0002:0000DC7E __status87 0040FC7E 0003:000051F4 _fdiv_chk_flag 004161F4 0002:0000BA92 _fdiv_m16i 0040DA92 0002:0000B9F6 _fdiv_m32 0040D9F6 0002:0000BAC6 _fdiv_m32i 0040DAC6 0002:0000BA44 _fdiv_m64 0040DA44 0002:0000B4D8 _fdiv_r 0040D4D8 0002:0000BB96 _fdivr_m16i 0040DB96 0002:0000BAFA _fdivr_m32 0040DAFA 0002:0000BBCA _fdivr_m32i 0040DBCA 0002:0000BB48 _fdivr_m64 0040DB48 0002:0000B290 _fegetenv 0040D290 0002:0000B2E8 _fesetenv 0040D2E8 0002:00003DC0 _frexp 00405DC0 0002:000093D8 _mbtowc 0040B3D8 0002:0000B330 _memicmp 0040D330 0002:00003E41 _nextafter 00405E41 0002:0000400D _rint 0040600D 0002:000088EC _strtold 0040A8EC 0002:000090A4 _tolower 0040B0A4 0002:000094F4 _wcscpy 0040B4F4 0002:00006F09 dget_dtype 00408F09 0002:00006E75 dget_dtype_pair 00408E75 0002:00006C94 dleft_justify 00408C94 0002:00006D75 dnorm 00408D75 0002:00006CA8 dround 00408CA8 0002:00006C70 dunnorm 00408C70 0002:00006E2E exception 00408E2E 0002:000072BF fget_dtype 004092BF 0002:0000715A fleft_justify 0040915A 0002:000070E3 fnorm 004090E3 0002:0000716A fright_justify 0040916A 0002:000071AE fround 004091AE 0002:000070B8 funnorm 004090B8 0003:00003558 roundto0 00414558 No surprise that some of that is required for x8087 configuration, but ... isn't there a lot of totally unrelated stuff there? The extra ~6KB is in addition to the most basic FP support.
 Also, the linker is quite a simple-minded program. It knows nothing about 
 C,
 or the C standard library, etc. All it does is recursively look at its 
 list
 of undefined symbols, and look for them in the supplied libraries. If an
 object file is being included, then it was specified explicitly or it was
 pulled in from a library because that object file defined a symbol that 
 was
 undefined.

Yes. In this case the frexp() code itself does not make any calls, and does not reference any static data (that might come from somewhere else). If I were to guess, there's probably more that just frexp() in that linkable section; and/or the other parts make external references. I think this is a question of modularity more than anything else. Shouldn't library members be as small (self contained) as possible?
 A related question:

You can define stubs for the symbols in your own code - that way they don't get pulled in from the library. You can also yank out obj modules from SNN.LIB (using lib.exe) and see what errors happen when linking. I'm not trying to be flip, I'm trying to explain how to figure out how these things work.

Appreciate that, and have often stubbed out C library symbols from D code. But it's hard to guess which symbols to stub.
 __acrtused_con pulls in constart.obj from SNN.LIB. The source to it is
 \dm\src\win32\constart.c.

Thanks!
 _exit() pulls in the C I/O subsystem, as it needs to flush & shut it down
 gracefully.

That's the one. Thank you. I will attempt to stub _exit() ... [clickety click] ... hmmm ... made a small difference but the IO subsystem is still pulled in. Any other internal references you know of?
 Trying to eliminate all traces of the C I/O subsystem is not so easy, as
 there tend to be a lot of dependencies on it, distributed throughout. Back
 in the bad old 16 bit days, this was worth the effort. But now I don't see
 the point. At most it adds 10k to your final program (and adds nothing to
 your library). These days, performance matters far more.

No disagreement here. I'm simply trying to get a minimal set of requirements.
Nov 28 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
It could very well be a modularity issue with frexp. The source is in
\dm\src\core32\mathtrn.asm.

(Some of the bloat has to do with the addition of the workarounds for the
infamous Pentium floating point bug.)
Nov 29 2005
parent reply "Kris" <fu bar.com> writes:
OK. How do I get the source to track this down?


"Walter Bright" <newshound digitalmars.com> wrote in message 
news:dmhuh1$1nat$1 digitaldaemon.com...
 It could very well be a modularity issue with frexp. The source is in
 \dm\src\core32\mathtrn.asm.

 (Some of the bloat has to do with the addition of the workarounds for the
 infamous Pentium floating point bug.)

 

Nov 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
Email me your address.

"Kris" <fu bar.com> wrote in message news:dmj2gm$2l13$1 digitaldaemon.com...
 OK. How do I get the source to track this down?


 "Walter Bright" <newshound digitalmars.com> wrote in message
 news:dmhuh1$1nat$1 digitaldaemon.com...
 It could very well be a modularity issue with frexp. The source is in
 \dm\src\core32\mathtrn.asm.

 (Some of the bloat has to do with the addition of the workarounds for


 infamous Pentium floating point bug.)


Dec 02 2005
parent "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com>

 Email me your address.

You have mail ~ "Re: code bloat"
Dec 05 2005