www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - For the adventurous: News from the LDC/Linux front

reply "David Nadlinger" <code klickverbot.at> writes:
Hi all,

I am excited to share news about two changes that recently made 
their way into the development version of LDC, changes that might 
be interesting for many of you Linux users out there.

The first is that LDC now supports linker-level dead code 
elimination on Linux. If you happen to be familiar with the 
-f{function,data}-sections/--gc-sections options of the GNU 
toolchain, their equivalent is now enabled by default. For a set 
of small-ish programs that make use of different parts of Phobos, 
I've seen executable size improvements of close to 4x (!) in my 
tests. However, as --gc-sections is known to occasionally cause 
trouble with third-party code that relies on specific linker 
behavior, this optimization can be disabled with a new LDC 
switch, -disable-linker-strip-dead.

And secondly, proper support for building druntime/Phobos as 
shared libraries and loading D shared objects dynamically has now 
arrived in LDC! As you might be aware, Martin Nowak has spent a 
considerable amount of effort on adding runtime loading 
capabilities to DMD and druntime during the last year or so. LDC 
now offers the same level of functionality, to a good part based 
on Martin's solid work. To build a copy of LDC with 
druntime/Phobos as a shared library, which is also required for 
proper runtime loading, simply pass -DBUILD_SHARED_LIBS=ON to 
CMake. Oh, and for now this is Linux-only too, sorry.

Even though I am currently not aware of any remaining issues, 
there are likely a few rough edges still. To be able to 
confidently ship these features as part of the next release, we'd 
very much appreciate early feedback. Just grab LDC from Git 
master and let us know how things went over on digitalmars.D.ldc. 
If you run into a specific bug, you can also directly open a 
ticket at https://github.com/ldc-developers/ldc/issues.

Cheers,
David
Jul 08 2014
next sibling parent "Joseph Rushton Wakeling" <joseph.wakeling webdrake.net> writes:
On Tuesday, 8 July 2014 at 17:54:48 UTC, David Nadlinger wrote:

 I am excited to share news about two changes that recently made 
 their way into the development version of LDC, changes that 
 might be interesting for many of you Linux users out there.


Sounds great -- congratulations to all the LDC team!  I'll try 
and get this installed and see how it goes.
Jul 08 2014
prev sibling next sibling parent "Joseph Rushton Wakeling" <joseph.wakeling webdrake.net> writes:
On Tuesday, 8 July 2014 at 17:54:48 UTC, David Nadlinger wrote:

 The first is that LDC now supports linker-level dead code 
 elimination on Linux. If you happen to be familiar with the 
 -f{function,data}-sections/--gc-sections options of the GNU 
 toolchain, their equivalent is now enabled by default. For a 
 set of small-ish programs that make use of different parts of 
 Phobos, I've seen executable size improvements of close to 4x 
 (!) in my tests.


Matches my experience trying things just now -- 612 KB vs 3.2 MB 
for one example on my system (the hap.random benchmark 
executable), 453 KB vs 2.3 MB for another.

No linking problems so far, but then, I haven't exactly tried to 
stretch things ;-)

Now to try out the shared library support ...
Jul 08 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
This is all great news, congratulations!

On the gc-sections front, Martin had gotten it to work for DMD on Linux but
then 
had to back it out because it failed with the ld.gold linker.

If it works for ldc with ld.gold, can you please help explain what went wrong 
for dmd?
Jul 08 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/8/2014 1:39 PM, Walter Bright wrote:

 This is all great news, congratulations!

 On the gc-sections front, Martin had gotten it to work for DMD on Linux but
then
 had to back it out because it failed with the ld.gold linker.

 If it works for ldc with ld.gold, can you please help explain what went wrong
 for dmd?


https://github.com/D-Programming-Language/dmd/pull/3715
Jul 08 2014
parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 8 July 2014 at 20:40:02 UTC, Walter Bright wrote:

 On 7/8/2014 1:39 PM, Walter Bright wrote:

 This is all great news, congratulations!

 On the gc-sections front, Martin had gotten it to work for DMD 
 on Linux but then
 had to back it out because it failed with the ld.gold linker.

 If it works for ldc with ld.gold, can you please help explain 
 what went wrong
 for dmd?


 https://github.com/D-Programming-Language/dmd/pull/3715


If you pay attention to my comment there, original PR has never 
actually changed a single bit about generated binaries :(
Jul 09 2014
prev sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 8 July 2014 at 20:39:34 UTC, Walter Bright wrote:

 This is all great news, congratulations!


Thanks. There was quite a bit of staring at obscure backtraces 
and object file dumps involved. The most persistent issue was 
what revealed itself to be a nasty race condition in 
core.thread.Fiber due to the way GCC and LLVM implement the ELF 
general-dynamic TLS model only after I had spent almost two days 
looking for it in the wrong place (quite fittingly, this now is 

enjoy these kinds of challenges from time to time, though.


 On the gc-sections front, Martin had gotten it to work for DMD 
 on Linux but then had to back it out because it failed with the 
 ld.gold linker.

 If it works for ldc with ld.gold, can you please help explain 
 what went wrong for dmd?


As far as I can tell at this point, LDC works equally well with 
both ld.bfd and ld.gold. Martin and I were discussing some of the 
specifics at the main --gc-sections ticket already [1], but it 
boils down to the fact that I chose to implement the ModuleInfo 
reference pinning in a much less beautiful, but more commonplace 
way.

Specifically, I didn't even try to utilize R_arch_NONE 
relocations to fake the section dependencies, as this would have 
required dropping down to the machine code emission layer of 
LLVM. Sticking with a solution that is representable in LLVM IR 
meant that we can continue to use the LLVM-provided command line 
tools when investigating issues in the optimization/machine code 
generation pipeline. As a side-effect, there isn't anything 
special about the emitted code, making it less much probable to 
hit a weird edge case in one of the linkers. The current solution 
is ugly, though, and leads to some tens of bytes of extra junk 
per module linked, so we might eventually replace it with 
something more sophisticated.

David


[1] https://issues.dlang.org/show_bug.cgi?id=879
Jul 08 2014
prev sibling next sibling parent reply "Joseph Rushton Wakeling" <joseph.wakeling webdrake.net> writes:
On Tuesday, 8 July 2014 at 17:54:48 UTC, David Nadlinger wrote:

 And secondly, proper support for building druntime/Phobos as 
 shared libraries and loading D shared objects dynamically has 
 now arrived in LDC! As you might be aware, Martin Nowak has 
 spent a considerable amount of effort on adding runtime loading 
 capabilities to DMD and druntime during the last year or so. 
 LDC now offers the same level of functionality, to a good part 
 based on Martin's solid work. To build a copy of LDC with 
 druntime/Phobos as a shared library, which is also required for 
 proper runtime loading, simply pass -DBUILD_SHARED_LIBS=ON to 
 CMake. Oh, and for now this is Linux-only too, sorry.


Tried building hap.random's benchmarknew with the shared-library 
build of ldc, and got this error when I tried to run the 
resulting executable:

Fatal Error while loading '/opt/ldc/lib/libphobos2-ldc.so.65':
	The module 'std.range' is already defined in './benchmarknew'.
Jul 08 2014
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 8 July 2014 at 20:43:19 UTC, Joseph Rushton Wakeling 
wrote:

 Tried building hap.random's benchmarknew with the 
 shared-library build of ldc, and got this error when I tried to 
 run the resulting executable:

 Fatal Error while loading '/opt/ldc/lib/libphobos2-ldc.so.65':
 	The module 'std.range' is already defined in './benchmarknew'.


As strange as it might sound, I'm actually rather relieved that 
you ran into this issue. I previously thought Martin's collision 
detection code was at fault (see 
https://github.com/D-Programming-Language/druntime/pull/791#iss
ecomment-48090942), 
but suddenly couldn't reproduce the issue any longer. I'll look 
into it.

Oh, and it seems like you might be able to work around it for now 
by (ironically, given Martin's experiences) using ld.gold instead 
of ld.bfd.

David
Jul 08 2014
next sibling parent reply "Joseph Rushton Wakeling" <joseph.wakeling webdrake.net> writes:
On Tuesday, 8 July 2014 at 22:44:30 UTC, David Nadlinger wrote:

 As strange as it might sound, I'm actually rather relieved that 
 you ran into this issue. I previously thought Martin's 
 collision detection code was at fault (see 
 https://github.com/D-Programming-Language/druntime/pull/791#iss
ecomment-48090942), 
 but suddenly couldn't reproduce the issue any longer. I'll look 
 into it.


Thanks -- let me know if there's anything you'd like me to test.


 Oh, and it seems like you might be able to work around it for 
 now by (ironically, given Martin's experiences) using ld.gold 
 instead of ld.bfd.


Well, personally I don't have any need for a workaround (it's no 
problem to keep using static libraries for the moment).  But I'm 
very happy to keep trying stuff out in the process of resolving 
this issue.
Jul 08 2014
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 8 July 2014 at 23:15:18 UTC, Joseph Rushton Wakeling 
wrote:

 On Tuesday, 8 July 2014 at 22:44:30 UTC, David Nadlinger wrote:

 As strange as it might sound, I'm actually rather relieved 
 that you ran into this issue. I previously thought Martin's 
 collision detection code was at fault (see 
 https://github.com/D-Programming-Language/druntime/pull/791#iss
ecomment-48090942), 
 but suddenly couldn't reproduce the issue any longer. I'll 
 look into it.


 Thanks -- let me know if there's anything you'd like me to test.


A few lost hairs and an implementation of weak symbols later, 
this should now be fixed in Git master.

In other words, there are currently no known issues with shared 
libraries, so everybody test away. ;)

David
Jul 09 2014
next sibling parent Joseph Rushton Wakeling via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10/07/14 01:35, David Nadlinger via Digitalmars-d wrote:

 A few lost hairs and an implementation of weak symbols later, this should now
be
 fixed in Git master.

 In other words, there are currently no known issues with shared libraries, so
 everybody test away. ;)


Hmm, I just rebuild current git master from a fresh pull, and the problem is 
still there. :-(

This is a fresh build on Ubuntu 14.04 with cmake called via:

     cmake -DCMAKE_INSTALL_PREFIX=/opt/ldc -DBUILD_SHARED_LIBS=ON ..

Then if I try to build and run the hap.random benchmark with

     ldmd2 -I./source -O -inline -release -ofbenchmarknew benchmarknew.d 
source/hap/random/adaptor.d source/hap/random/distribution.d 
source/hap/random/generator.d source/hap/random/package.d
source/hap/random/traits.d

     ./benchmarknew

.... the latter falls over with the error previously mentioned:
error while loading shared libraries: libphobos2-ldc.so.65: cannot open shared 
object file: No such file or directory.

I note that when "make install" is run, then after libphobos2-ldc.so is 
installed, I see the following:

-- Removed runtime path from "/opt/ldc/lib/libphobos2-ldc.so.2.0.65"
-- Installing: /opt/ldc/lib/libdruntime-ldc-debug.so.2.0.65
-- Installing: /opt/ldc/lib/libdruntime-ldc-debug.so.65
-- Installing: /opt/ldc/lib/libdruntime-ldc-debug.so
-- Installing: /opt/ldc/lib/libphobos2-ldc-debug.so.2.0.65
-- Installing: /opt/ldc/lib/libphobos2-ldc-debug.so.65
-- Installing: /opt/ldc/lib/libphobos2-ldc-debug.so
-- Removed runtime path from "/opt/ldc/lib/libphobos2-ldc-debug.so.2.0.65"

... is this relevant?
Jul 12 2014
prev sibling parent Joseph Rushton Wakeling via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12/07/14 19:12, Joseph Rushton Wakeling via Digitalmars-d wrote:

 Hmm, I just rebuild current git master from a fresh pull, and the problem is
 still there. :-(


I'm so sorry, this is completely wrong.  I'd just forgotten to put back my 
/etc/ld.so.conf.d/ldc2.conf file pointing to /opt/ldc/lib.

With this in place, everything seems to work fine.  (Before, it really was a 
problem, as I was still getting the error even with the ldc2.conf file in
place;-)
Jul 12 2014
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 8 July 2014 at 22:44:30 UTC, David Nadlinger wrote:

 On Tuesday, 8 July 2014 at 20:43:19 UTC, Joseph Rushton 
 Wakeling wrote:

 Tried building hap.random's benchmarknew with the 
 shared-library build of ldc, and got this error when I tried 
 to run the resulting executable:

 Fatal Error while loading '/opt/ldc/lib/libphobos2-ldc.so.65':
 	The module 'std.range' is already defined in './benchmarknew'.


 […]


In an unexpected turn of events, it seems like you can also avoid 
this problem with ld.bfd by simply building with 
"-disable-linker-strip-dead".

This isn't supposed to happen. Under any circumstances. At all.

Running the program in a debugger, it seems like the 
__bss_start/_end symbols are off, just as I encountered 
previously. In fact, they apparently point to an address range 
that isn't mapped into the process at all!

I couldn't track down why this is happening yet. Here is 
something odd I noticed about the bfd-linked binary, though:
---
$ readelf --dyn-syms benchmarknew.bfd

Symbol table '.dynsym' contains 407 entries:
    Num:    Value          Size Type    Bind   Vis      Ndx Name
      0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
      1: 000000000065c518     0 NOTYPE  LOCAL  DEFAULT   29 _edata
      2: 000000000065d228     0 NOTYPE  LOCAL  DEFAULT   30 _end
      3: 000000000065c518     0 NOTYPE  LOCAL  DEFAULT   30 
__bss_start
[…]
---
The __bss_start/_end symbols have somehow turned LOCAL, whereas 
they should be globally visible to override the ones defined in 
libdruntime.so. I suspect that this is involved in the 
nonsensical values of 0x50b4d8/0x50b728 I get inside druntime. 
Note also that the offset between these two addresses is also not 
remotely correct.

David
Jul 08 2014
parent "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 8 July 2014 at 23:23:40 UTC, David Nadlinger wrote:

 The __bss_start/_end symbols have somehow turned LOCAL, whereas 
 they should be globally visible to override the ones defined in 
 libdruntime.so.


Sorry for the monologue, but this might be related: 
https://sourceware.org/bugzilla/show_bug.cgi?id=13683

David
Jul 08 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 08/07/14 19:54, David Nadlinger wrote:

 Hi all,

 I am excited to share news about two changes that recently made their
 way into the development version of LDC, changes that might be
 interesting for many of you Linux users out there.

 The first is that LDC now supports linker-level dead code elimination on
 Linux. If you happen to be familiar with the
 -f{function,data}-sections/--gc-sections options of the GNU toolchain,
 their equivalent is now enabled by default. For a set of small-ish
 programs that make use of different parts of Phobos, I've seen
 executable size improvements of close to 4x (!) in my tests. However, as
 --gc-sections is known to occasionally cause trouble with third-party
 code that relies on specific linker behavior, this optimization can be
 disabled with a new LDC switch, -disable-linker-strip-dead.


Is this only for Linux or Posix in general?

-- 
/Jacob Carlborg
Jul 08 2014
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 9 July 2014 at 06:44:10 UTC, Jacob Carlborg wrote:

 On 08/07/14 19:54, David Nadlinger wrote:

 The first is that LDC now supports linker-level dead code 
 elimination on
 Linux. […]


 Is this only for Linux or Posix in general?


So far it is only enabled on Linux, as I made the necessary 
adjustments to the druntime module registration code and so on 
while working on shared library support there.

That being said, supporting this should be considerably easier 
for platforms that still use the "old" __Dmodule_ref module 
registration mechanism, as the latter doesn't depend on 
unreferenced symbols in a magic section being kept by the linker. 
I can't think of any issues that would preclude using e.g. the 
equivalent linker option on OS X (-dead_strip) right now, or 
---gc-sections on FreeBSD.

But as with all improvements, this simply depends on somebody to 
sit down and make it happen. Getting the thing to work shouldn't 
be too hard, and I'd be happy to help anyone attempting this if 
any tough issues crop up, but if you are just waiting for me to 
find the time to investigate other platforms, you'll probably be 
in for quite a wait.

Cheers,
David
Jul 10 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-07-10 15:51, David Nadlinger wrote:


 So far it is only enabled on Linux, as I made the necessary adjustments
 to the druntime module registration code and so on while working on
 shared library support there.

 That being said, supporting this should be considerably easier for
 platforms that still use the "old" __Dmodule_ref module registration
 mechanism, as the latter doesn't depend on unreferenced symbols in a
 magic section being kept by the linker. I can't think of any issues that
 would preclude using e.g. the equivalent linker option on OS X
 (-dead_strip) right now, or ---gc-sections on FreeBSD.

 But as with all improvements, this simply depends on somebody to sit
 down and make it happen. Getting the thing to work shouldn't be too
 hard, and I'd be happy to help anyone attempting this if any tough
 issues crop up, but if you are just waiting for me to find the time to
 investigate other platforms, you'll probably be in for quite a wait.


I see, thanks.

-- 
/Jacob Carlborg
Jul 10 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 8 July 2014 at 17:54:48 UTC, David Nadlinger wrote:

 Hi all,

 I am excited to share news about two changes that recently made 
 their way into the development version of LDC, changes that 
 might be interesting for many of you Linux users out there.

 The first is that LDC now supports linker-level dead code 
 elimination on Linux. If you happen to be familiar with the 
 -f{function,data}-sections/--gc-sections options of the GNU 
 toolchain, their equivalent is now enabled by default. For a 
 set of small-ish programs that make use of different parts of 
 Phobos, I've seen executable size improvements of close to 4x 
 (!) in my tests. However, as --gc-sections is known to 
 occasionally cause trouble with third-party code that relies on 
 specific linker behavior, this optimization can be disabled 
 with a new LDC switch, -disable-linker-strip-dead.

 And secondly, proper support for building druntime/Phobos as 
 shared libraries and loading D shared objects dynamically has 
 now arrived in LDC! As you might be aware, Martin Nowak has 
 spent a considerable amount of effort on adding runtime loading 
 capabilities to DMD and druntime during the last year or so. 
 LDC now offers the same level of functionality, to a good part 
 based on Martin's solid work. To build a copy of LDC with 
 druntime/Phobos as a shared library, which is also required for 
 proper runtime loading, simply pass -DBUILD_SHARED_LIBS=ON to 
 CMake. Oh, and for now this is Linux-only too, sorry.

 Even though I am currently not aware of any remaining issues, 
 there are likely a few rough edges still. To be able to 
 confidently ship these features as part of the next release, 
 we'd very much appreciate early feedback. Just grab LDC from 
 Git master and let us know how things went over on 
 digitalmars.D.ldc. If you run into a specific bug, you can also 
 directly open a ticket at 
 https://github.com/ldc-developers/ldc/issues.

 Cheers,
 David


Those are probably best news from D world I have heard in last 
few months! Has anyone tried that on largish projects? Using GDC 
with --gc-sections has resulted in broken binaries sometimes but 
most likely it was the same ModuleInfo issue. Wonder how reliable 
it is in LDC.
Jul 09 2014
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 9 July 2014 at 10:01:15 UTC, Dicebot wrote:

 Those are probably best news from D world I have heard in last 
 few months! Has anyone tried that on largish projects? Using 
 GDC with --gc-sections has resulted in broken binaries 
 sometimes but most likely it was the same ModuleInfo issue. 
 Wonder how reliable it is in LDC.


So far, I'm not aware of any conceptual problems or 
implementation issues. I tested it on a some larger projects 
using Thrift and a few other libraries. But as always, there 
might be some odd cases out there I simply missed, especially as 
linker behavior tends to vary quite a bit between different 
versions of binutils (and interestingly also different 
distributions, as some tend to come with patched versions or 
adjusted defaults).

All in all, I can only encourage you to test this on your own 
projects and report any issues. This is also the idea behind 
turning the feature on by default for now. Note that in contrast 
to GDC, we actually consider any issues caused by this to be a 
bug (Iain and I agree to disagree on this).

David
Jul 10 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
LDC is officially my favorite D compiler from now <3

..now I only need to finish D2 migration of our code base *caugh*
Jul 10 2014
prev sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 July 2014 15:00, David Nadlinger via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Wednesday, 9 July 2014 at 10:01:15 UTC, Dicebot wrote:

 Those are probably best news from D world I have heard in last few months!
 Has anyone tried that on largish projects? Using GDC with --gc-sections has
 resulted in broken binaries sometimes but most likely it was the same
 ModuleInfo issue. Wonder how reliable it is in LDC.



 So far, I'm not aware of any conceptual problems or implementation issues. I
 tested it on a some larger projects using Thrift and a few other libraries.
 But as always, there might be some odd cases out there I simply missed,
 especially as linker behavior tends to vary quite a bit between different
 versions of binutils (and interestingly also different distributions, as
 some tend to come with patched versions or adjusted defaults).

 All in all, I can only encourage you to test this on your own projects and
 report any issues. This is also the idea behind turning the feature on by
 default for now. Note that in contrast to GDC, we actually consider any
 issues caused by this to be a bug (Iain and I agree to disagree on this).

 David


Indeed we do!

Instead I am consulting with the GCC devs to fix LTO support. >:)

Iain
Jul 10 2014
prev sibling parent Joseph Rushton Wakeling via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 08/07/14 19:54, David Nadlinger via Digitalmars-d wrote:

 And secondly, proper support for building druntime/Phobos as shared libraries
 and loading D shared objects dynamically has now arrived in LDC! As you might
be
 aware, Martin Nowak has spent a considerable amount of effort on adding runtime
 loading capabilities to DMD and druntime during the last year or so. LDC now
 offers the same level of functionality, to a good part based on Martin's solid
 work. To build a copy of LDC with druntime/Phobos as a shared library, which is
 also required for proper runtime loading, simply pass -DBUILD_SHARED_LIBS=ON to
 CMake. Oh, and for now this is Linux-only too, sorry.


So, now that I have this running, I thought I'd give it a go trying out 
hap.random's benchmarks with shared and with static druntime/phobos.

Most of hap.random is pretty standalone apart from reliance on std.range and 
std.traits (and I think that largely for compile-time checks), so, as can be 
expected, most functions are not meaningfully affected by being compiled
against 
a shared library.  The handful of functions that depend on std.math seem to
take 
a small speed hit that might not be significant.

However, the dice() function, which relies on std.algorithm.reduce, takes a 
substantial speed hit, taking 20% more time to run compared to when the 
benchmarks are compiled against a static phobos.

This is probably within the realm of normalcy, and I guess the speed hit might 
turn out to be smaller on a larger overall timescale/number of calls, but it 
seems a shame (and unexpected, as reduce is obviously templated, so I wouldn't 
expect to have a speed hit because of shared library linkage).  Can anyone 
suggest an explanation?
Jul 12 2014