www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - How to stop DMD from exploding the executable file size?

reply ryuukk_ <ryuukk.dev gmail.com> writes:
Having this simple code makes the executable gigantic!

```D
struct Big
{
     int[1024 * 1024] big = 0;
}

Big big;
```

Only with DMD, LDC produces a tiny executable

Is there a flag available to fix this behavior? it's not 
sustainable
Aug 30 2022
next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
```
$ ldc2 -m64 app.d
$ ll
total 4545
-rw-r--r-- 1 ryuukk 197121      69 Aug 31 00:57 app.d
-rwxr-xr-x 1 ryuukk 197121  453120 Aug 31 00:57 app.exe*
-rw-r--r-- 1 ryuukk 197121 4195865 Aug 31 00:57 app.obj
```

```
$ dmd -m64 app.d
$ ll
total 8617
-rw-r--r-- 1 ryuukk 197121      69 Aug 31 00:57 app.d
-rwxr-xr-x 1 ryuukk 197121 4623872 Aug 31 00:57 app.exe*
-rw-r--r-- 1 ryuukk 197121 4196322 Aug 31 00:57 app.obj
```
Aug 30 2022
parent reply solidstate1991 <laszloszeremi outlook.com> writes:
On Tuesday, 30 August 2022 at 22:58:44 UTC, ryuukk_ wrote:

 ```
 $ ldc2 -m64 app.d
 $ ll
 total 4545
 -rw-r--r-- 1 ryuukk 197121      69 Aug 31 00:57 app.d
 -rwxr-xr-x 1 ryuukk 197121  453120 Aug 31 00:57 app.exe*
 -rw-r--r-- 1 ryuukk 197121 4195865 Aug 31 00:57 app.obj
 ```

 ```
 $ dmd -m64 app.d
 $ ll
 total 8617
 -rw-r--r-- 1 ryuukk 197121      69 Aug 31 00:57 app.d
 -rwxr-xr-x 1 ryuukk 197121 4623872 Aug 31 00:57 app.exe*
 -rw-r--r-- 1 ryuukk 197121 4196322 Aug 31 00:57 app.obj
 ```


LDC has way better optimization features, so it's kinda expected.
Sep 01 2022
parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Thu, Sep 01, 2022 at 10:15:03PM +0000, solidstate1991 via Digitalmars-d
wrote:

 On Tuesday, 30 August 2022 at 22:58:44 UTC, ryuukk_ wrote:

 ```
 $ ldc2 -m64 app.d
 $ ll
 total 4545
 -rw-r--r-- 1 ryuukk 197121      69 Aug 31 00:57 app.d
 -rwxr-xr-x 1 ryuukk 197121  453120 Aug 31 00:57 app.exe*
 -rw-r--r-- 1 ryuukk 197121 4195865 Aug 31 00:57 app.obj
 ```
 
 ```
 $ dmd -m64 app.d
 $ ll
 total 8617
 -rw-r--r-- 1 ryuukk 197121      69 Aug 31 00:57 app.d
 -rwxr-xr-x 1 ryuukk 197121 4623872 Aug 31 00:57 app.exe*
 -rw-r--r-- 1 ryuukk 197121 4196322 Aug 31 00:57 app.obj
 ```


 
 LDC has way better optimization features, so it's kinda expected.


Yep.  LDC is my go-to compiler for release builds.  I use dmd for
development because the turnaround time is faster, but I also don't care
about performance/executable size when using dmd.  For release builds
when I *do* care about those things, I use LDC; dmd isn't even on my
radar for that.


T

-- 
Amateurs built the Ark; professionals built the Titanic.
Sep 01 2022
parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Thursday, 1 September 2022 at 22:31:50 UTC, H. S. Teoh wrote:

 On Thu, Sep 01, 2022 at 10:15:03PM +0000, solidstate1991 via 
 Digitalmars-d wrote:

 On Tuesday, 30 August 2022 at 22:58:44 UTC, ryuukk_ wrote:

 ```
 $ ldc2 -m64 app.d
 $ ll
 total 4545
 -rw-r--r-- 1 ryuukk 197121      69 Aug 31 00:57 app.d
 -rwxr-xr-x 1 ryuukk 197121  453120 Aug 31 00:57 app.exe*
 -rw-r--r-- 1 ryuukk 197121 4195865 Aug 31 00:57 app.obj
 ```
 
 ```
 $ dmd -m64 app.d
 $ ll
 total 8617
 -rw-r--r-- 1 ryuukk 197121      69 Aug 31 00:57 app.d
 -rwxr-xr-x 1 ryuukk 197121 4623872 Aug 31 00:57 app.exe*
 -rw-r--r-- 1 ryuukk 197121 4196322 Aug 31 00:57 app.obj
 ```


 
 LDC has way better optimization features, so it's kinda 
 expected.


 Yep.  LDC is my go-to compiler for release builds.  I use dmd 
 for development because the turnaround time is faster, but I 
 also don't care about performance/executable size when using 
 dmd.  For release builds when I *do* care about those things, I 
 use LDC; dmd isn't even on my radar for that.


 T


LDC is great, but note that the object files are equally large 
for both compilers. Apparently it is the linker that should be 
attributed the size reduction. Which leads me to wonder what the 
main function looks like, because it might have removed the 
entire initialisation as dead code…

It would have been better if OP would have posted complete code.

— Bastiaan.
Sep 02 2022
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Tue, Aug 30, 2022 at 10:51:46PM +0000, ryuukk_ via Digitalmars-d wrote:

 Having this simple code makes the executable gigantic!
 
 ```D
 struct Big
 {
     int[1024 * 1024] big = 0;
 }
 
 Big big;
 ```
 
 Only with DMD, LDC produces a tiny executable
 
 Is there a flag available to fix this behavior? it's not sustainable


Which version of dmd are you using? I tried to compile the above code
with 2.098 (old git master) and the total executable size is only 900KB.

I remember about a year or two ago there was a bug where dmd would
generate 1024 * 1024 store instructions to initialize the array, but I
just checked the disassembly, it's now a rep stos, which is very small.


T

-- 
Life is unfair. Ask too much from it, and it may decide you don't deserve what
you have now either.
Aug 30 2022
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Tuesday, 30 August 2022 at 23:05:16 UTC, H. S. Teoh wrote:

 On Tue, Aug 30, 2022 at 10:51:46PM +0000, ryuukk_ via 
 Digitalmars-d wrote:

 Having this simple code makes the executable gigantic!
 
 ```D
 struct Big
 {
     int[1024 * 1024] big = 0;
 }
 
 Big big;
 ```
 
 Only with DMD, LDC produces a tiny executable
 
 Is there a flag available to fix this behavior? it's not 
 sustainable


 Which version of dmd are you using? I tried to compile the 
 above code with 2.098 (old git master) and the total executable 
 size is only 900KB.

 I remember about a year or two ago there was a bug where dmd 
 would generate 1024 * 1024 store instructions to initialize the 
 array, but I just checked the disassembly, it's now a rep stos, 
 which is very small.


 T


dmd --version prints: `DMD32 D Compiler v2.100.1`

It's on Windows, maybe that problem is windows specific?
Aug 30 2022
parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Tue, Aug 30, 2022 at 11:09:55PM +0000, ryuukk_ via Digitalmars-d wrote:

 On Tuesday, 30 August 2022 at 23:05:16 UTC, H. S. Teoh wrote:


[...]

 Which version of dmd are you using? I tried to compile the above
 code with 2.098 (old git master) and the total executable size is
 only 900KB.
 
 I remember about a year or two ago there was a bug where dmd would
 generate 1024 * 1024 store instructions to initialize the array, but
 I just checked the disassembly, it's now a rep stos, which is very
 small.




[...]

 dmd --version prints: `DMD32 D Compiler v2.100.1`
 
 It's on Windows, maybe that problem is windows specific?


Possibly, Windows uses a different backend path (obviously). Try to look
at the disassembly to see if it's the abovementioned problem?


T

-- 
If blunt statements had a point, they wouldn't be blunt...
Aug 30 2022
prev sibling next sibling parent reply max haughton <maxhaton gmail.com> writes:
On Tuesday, 30 August 2022 at 22:51:46 UTC, ryuukk_ wrote:

 Having this simple code makes the executable gigantic!

 ```D
 struct Big
 {
     int[1024 * 1024] big = 0;
 }

 Big big;
 ```

 Only with DMD, LDC produces a tiny executable

 Is there a flag available to fix this behavior? it's not 
 sustainable


You really shouldn't have structs this big unless absolutely 
necessary.

The thing being elided is the initializer and pointer bitmap for 
the runtime to use.
Sep 02 2022
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Friday, 2 September 2022 at 17:53:51 UTC, max haughton wrote:

 On Tuesday, 30 August 2022 at 22:51:46 UTC, ryuukk_ wrote:

 Having this simple code makes the executable gigantic!

 ```D
 struct Big
 {
     int[1024 * 1024] big = 0;
 }

 Big big;
 ```

 Only with DMD, LDC produces a tiny executable

 Is there a flag available to fix this behavior? it's not 
 sustainable


 You really shouldn't have structs this big unless absolutely 
 necessary.

 The thing being elided is the initializer and pointer bitmap 
 for the runtime to use.


My use case should not makes us blind about the issue DMD is 
having

I tested with both C and Zig, and they both produce small 
executable

If DMD can do it on linux, there is no reason it can't do it on 
windows, if LDC can do it for both

I want to promote D, i can't promote D if i tell people "yeah but 
X, Y, Z, then you should go with X, when Z, and when stars are 
aligned, maybe"

It just feels bad; and the list of things that feels bad starts 
to stack up to a point it becomes sad
Sep 02 2022
next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
Just look: https://i.imgur.com/5JheSIA.png

5mb for DMD

200kb for LDC

Both release build

It is very sad, what is walter doing?
Sep 02 2022
next sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Sat, Sep 03, 2022 at 12:51:09AM +0000, ryuukk_ via Digitalmars-d wrote:

 Just look: https://i.imgur.com/5JheSIA.png
 
 5mb for DMD
 
 200kb for LDC
 
 Both release build
 
 It is very sad, what is walter doing?


Somebody already pointed out, it's LDC's linker that's eliminating the
excess weight.  You can prove this yourself by using `dmd -c` and then
linking the object files yourself, supplying additional LTO options to
discard unreferenced sections.


T

-- 
The diminished 7th chord is the most flexible and fear-instilling chord. Use it
often, use it unsparingly, to subdue your listeners into submission!
Sep 02 2022
prev sibling parent Don Allen <donaldcallen gmail.com> writes:
On Saturday, 3 September 2022 at 00:51:09 UTC, ryuukk_ wrote:

 Just look: https://i.imgur.com/5JheSIA.png

 5mb for DMD

 200kb for LDC

 Both release build

 It is very sad, what is walter doing?


Working. Building us great stuff that he gives to all of us free.

If the dmd executables are too large for you, use ldc for release 
builds. Or you can get smaller release builds with dmd by linking 
the object files yourself. I don't understand why you are still 
complaining.
Sep 02 2022
prev sibling parent reply max haughton <maxhaton gmail.com> writes:
On Saturday, 3 September 2022 at 00:49:36 UTC, ryuukk_ wrote:

 On Friday, 2 September 2022 at 17:53:51 UTC, max haughton wrote:

 On Tuesday, 30 August 2022 at 22:51:46 UTC, ryuukk_ wrote:

 Having this simple code makes the executable gigantic!

 ```D
 struct Big
 {
     int[1024 * 1024] big = 0;
 }

 Big big;
 ```

 Only with DMD, LDC produces a tiny executable

 Is there a flag available to fix this behavior? it's not 
 sustainable


 You really shouldn't have structs this big unless absolutely 
 necessary.

 The thing being elided is the initializer and pointer bitmap 
 for the runtime to use.


 My use case should not makes us blind about the issue DMD is 
 having

 I tested with both C and Zig, and they both produce small 
 executable

 If DMD can do it on linux, there is no reason it can't do it on 
 windows, if LDC can do it for both

 I want to promote D, i can't promote D if i tell people "yeah 
 but X, Y, Z, then you should go with X, when Z, and when stars 
 are aligned, maybe"

 It just feels bad; and the list of things that feels bad starts 
 to stack up to a point it becomes sad


If this is what stops them from using D they're using it as a toy 
- it's a completely nonsensical pattern 99% of the time. Huge 
structs usually imply a lack of understanding, for example you 
could easily blow up the stack.

I will at very least try and fix this in dmd, it's just the 
object emission code being overly conservative (ldc will emit the 
initializer if you do something that actually needs it), but if 
this is what stops them using D I would be very skeptical. C 
doesn't even have struct default-initializers.

In reality, the stars have to align the other way - we have lots 
of people writing D code every day, some of which might not be 
entirely sure what an object file even is, 99% of the time this 
kind of stuff just doesn't matter.

In line with the above statement, the actual answer here is to 
try and compress the big initializers rather than faff around 
with whether they are used or not - should the compiler focus on 
dead code elimination rather than optimizing the stuff that 
actually runs?

Most of these initializers will be zero, god forbid someone does 
statically allocate their tensor in a structure potentially we 
can give them a speed/size tradeoff.

Everyone making a living out of D's workflow that I'm aware of is 
prototype with dmd, ship with ldc or gdc. This should be 
enshrined in the toolchain, it is not currently but maybe it will 
be sooner rather than later.

I do like small binaries but I can count on 1 hand the number of 
times it's seemed important and on zero hands the number of time 
it's actually mattered in practice.

If it, in your estimation, does make it a difference it may be of 
note that PGO can make the inlining much more intelligent and 
thus save you potentially even double digit percentages of the 
non-PGO build's size.
Sep 02 2022
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Saturday, 3 September 2022 at 01:59:56 UTC, max haughton wrote:

 On Saturday, 3 September 2022 at 00:49:36 UTC, ryuukk_ wrote:

 On Friday, 2 September 2022 at 17:53:51 UTC, max haughton 
 wrote:

 On Tuesday, 30 August 2022 at 22:51:46 UTC, ryuukk_ wrote:

 Having this simple code makes the executable gigantic!

 ```D
 struct Big
 {
     int[1024 * 1024] big = 0;
 }

 Big big;
 ```

 Only with DMD, LDC produces a tiny executable

 Is there a flag available to fix this behavior? it's not 
 sustainable


 You really shouldn't have structs this big unless absolutely 
 necessary.

 The thing being elided is the initializer and pointer bitmap 
 for the runtime to use.


 My use case should not makes us blind about the issue DMD is 
 having

 I tested with both C and Zig, and they both produce small 
 executable

 If DMD can do it on linux, there is no reason it can't do it 
 on windows, if LDC can do it for both

 I want to promote D, i can't promote D if i tell people "yeah 
 but X, Y, Z, then you should go with X, when Z, and when stars 
 are aligned, maybe"

 It just feels bad; and the list of things that feels bad 
 starts to stack up to a point it becomes sad


 If this is what stops them from using D they're using it as a 
 toy - it's a completely nonsensical pattern 99% of the time. 
 Huge structs usually imply a lack of understanding, for example 
 you could easily blow up the stack.


It contributes, and give them arguments


 I will at very least try and fix this in dmd, it's just the 
 object emission code being overly conservative (ldc will emit 
 the initializer if you do something that actually needs it), 
 but if this is what stops them using D I would be very 
 skeptical. C doesn't even have struct default-initializers.


If LDC did it, DMD can do it, specially if it can already do it 
in linux


 In reality, the stars have to align the other way - we have 
 lots of people writing D code every day, some of which might 
 not be entirely sure what an object file even is, 99% of the 
 time this kind of stuff just doesn't matter.


Assumptions, we have to stop with assuming what people like, and 
do things the right way

We have to stop putting the burden on the users



 In line with the above statement, the actual answer here is to 
 try and compress the big initializers rather than faff around 
 with whether they are used or not - should the compiler focus 
 on dead code elimination rather than optimizing the stuff that 
 actually runs?

 Most of these initializers will be zero, god forbid someone 
 does statically allocate their tensor in a structure 
 potentially we can give them a speed/size tradeoff.

 Everyone making a living out of D's workflow that I'm aware of 
 is prototype with dmd, ship with ldc or gdc. This should be 
 enshrined in the toolchain, it is not currently but maybe it 
 will be sooner rather than later.

 I do like small binaries but I can count on 1 hand the number 
 of times it's seemed important and on zero hands the number of 
 time it's actually mattered in practice.

 If it, in your estimation, does make it a difference it may be 
 of note that PGO can make the inlining much more intelligent 
 and thus save you potentially even double digit percentages of 
 the non-PGO build's size.



This is a DMD problem, i am a user, my expectation is it should 
produce a non bloated executable, that's it, as a user i think 
that's fair to expect that from a compiler that produce native 
executables

And i show you the release build, on debug it is worse, building 
10+mbs files constantly, it'll worn the disk, and it makes file 
transfer painfully slow over the net

Again, i am a user, not a compiler person
Sep 05 2022
next sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
Thanks to  Herringway on IRC, he made me check dmd/ldc linker 
invocation and check the difference


LDC seem to add:

```
     /OPT:REF
     /OPT:ICF
```

Adding both and now DMD produce small file! (1mb vs 10mb before)

Can we have them enabled by default?
Sep 05 2022
prev sibling parent wjoe <invalid example.com> writes:
On Monday, 5 September 2022 at 17:31:10 UTC, ryuukk_ wrote:

 This is a DMD problem, i am a user, my expectation is it should 
 produce a non bloated executable, that's it, as a user i think 
 that's fair to expect that from a compiler that produce native 
 executables

 And i show you the release build, on debug it is worse, 
 building 10+mbs files constantly, it'll worn the disk, and it 
 makes file transfer painfully slow over the net

 Again, i am a user, not a compiler person


Not a compiler person but a user, too.
Since DMD is the reference compiler my expectations are:
1) the code it compiles, any other D compiler will do as well, 
without requiring modifications to the code
2) the result won't be optimized, neither size nor speed, as well 
as using e.g. LDC or GDC
3) fast
4) it may not support every platform and/or OS

If you're concerned about wearing out your hard drive why don't 
you build on a RAM drive ?
You can add a step to your build config to compress your 
executable for faster file transfers over the net.
If you don't have a build file you can always write a quick make 
file with a default target and add any flag you want - defining 
your own defaults - as well as any steps you need to have done 
pre/post compilation.

When you promote D what stops you to mention that DMD is the 
reference compiler and as such won't produce the best results for 
deployment and there's LDC and GDC, etc. which about everyone 
uses for that purpose?
Sep 08 2022
prev sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
I went ahead and sent a PR: 
https://github.com/dlang/dmd/pull/14418

Never give up
Sep 05 2022