www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The D ecosystem in Debian with free-as-in-freedom DMD

reply Matthias Klumpp <mak debian.org> writes:
Hi there!
These are probably questions directed mostly at Walter and others 
shaping D's goals, but this could be of general interest for many 
people, so to the forum it goes :-)

DMD is completely free software now and we can legally distribute 
it in Debian main - yay! This is an awesome achievement and will 
make D adoption in Linux distributions much easier (in fact, Red 
Hat is working on getting good support into Fedora too already).

There are a two issues though that we will be facing in Debian 
soon, and I would like to get some opinion and maybe perspective 
on from the D community on them.

Naturally, when the reference compiler is available in Debian, we 
would compile everything with that, as it is the development 
focus and the thing many people test with.
We do, however, have quite a bit of bioinformatics and other D 
software in the archive where performance matters - so our users 
and the developers of that software (like BioD, potentially Mir, 
maybe even Vibe.d) will want the fastest performance and will ask 
us to compile the libraries with LDC or GDC.

If we do that, we will run into the D ABI trap: Libraries 
compiled with compiler X can not be used from software compiled 
with D compiler Y. There is actually no ABI stability guarantee 
even between DMD releases.
This will make integrating D a huge pain. Recompiling the 
dependency-chain of a software from source when compiling a 
package using the "right" compiler and statically adding the code 
is forbidden by distro policy. Having static libraries in the 
dependencies doesn't solve the issue. Compiling each library with 
all D compilers is highly impractical and not really feasible.
So, how should we proceed here? We could make it "DMD is the only 
thing on the highway" compiling everything with DMD with zero 
exceptions, which would leave us with only DMD-internal ABI 
breakage and bad D code performance for some libraries. We could 
also continue using LDC for everything, but that comes with it's 
own issues (I am hitting quite a bunch of LDC bugs and upstream 
projects usually test with DMD or use features which are only in 
the latest DMD).

The other issue is architecture support. In Debian we are 
strongly encouraged to support as many architectures as possible, 
to the point of having to justify why arch X is not supported.
LDC runs on at least armhf and ppc64el additionally to 
amd64/ia32, while DMD says it's specifically only for ia32/amd64.
This means we might end up compiling stuff with different D 
compilers on different architectures, or we will need to drop 
architectures and request arch-specific package removals for 
things that currently build on architectures not supported by 
DMD, which will trigger resistance.

So, in summary:
   1) Is there some perspective on D getting a defined ABI that 
works with all major D compilers?
   2) What would the D community recommend on how to deal with the 
ABI issues currently? A Linux distribution is a bunch of tightly 
integrated software, and changing one piece in an incompatible 
way (e.g. by building it with LDC instead of DMD) will have 
consequences.
   3) Will DMD support more architectures in the near future? How 
should the architecture issue be handled?

I am interested in some feedback here, since I currently can't 
see a good way to address these issues.

Also: If you want to help out Debian's D team, feel free to ping 
me or any other D team member (we are very short handed and are 
only two active people right now). See https://wiki.debian.org/D 
(caution, wiki page is very outdated, last touched in 2012)
Apr 10 2017
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
 Recompiling the dependency-chain of a software from source when 
 compiling a package using the "right" compiler and statically 
 adding the code is forbidden by distro policy.
This is the part that I do not understand. Who came up with those policies and decided that they apply to D? Because I really don't think they should. A lot of D code is generic. Very extremely few exceptions, it is a given that any time a library is used, at least some part of the library's code is going to be compiled not when the library is "built", but when programs using the library are. The notion of D libraries preserving ABI compatibility of built shared libraries is pretty much unheard of (it would have to be an explicit design goal), with Phobos and everything else going for source-compatibility only. Heck, there are packages out there that do not even produce useful object files (such as header-only C++ libraries). As far as I can see, the main obstacle of D in Debian is the extreme dogmatism of the distribution's policies. D is not C (where these rules make much more sense), so I really don't think the same rules should apply. Can we treat it more like an interpreted language instead?
Apr 10 2017
next sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 12:40:33 UTC, Vladimir Panteleev 
wrote:
 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
 Recompiling the dependency-chain of a software from source 
 when compiling a package using the "right" compiler and 
 statically adding the code is forbidden by distro policy.
This is the part that I do not understand. Who came up with those policies and decided that they apply to D? Because I really don't think they should.
They are the result of years of experience in building complex systems and keeping them secure. If you have a dependency chain "X -> Y -> Z" (-> meaning "depends on"), and you find a security bug in Z, you the security team will just need to fix the bug in Z to resolve it in the whole distribution. But if the code which has this issue is compiled into all of the packages that depend on them, you will need to rebuild the full dependency chain to actually fix the security issue, which is not only time intensive but also a huge maintenance effort. In this simple example it doesn't look like much, but those dependency chains can grow massively large and complicated, and the only way to keep the large software stack maintainable and secure is by splitting pieces cleanly. Embedded code copies are allowed in rare events, but in these cases the security team needs to be aware of them. Sometimes, the licenses also explicitly prevent embedded code copies. Aside from these issues, splitting things cleanly also makes general package maintenance much easier, and adds flexibility for our users who can mix and match parts of the distribution as they like and combine them with their own code. You need to see here that D is not the center of the world and we will need to make it work nicely with the rest of the system. The technical policies work for everything else, so there is nothing that really justifies an exception for D here (if 10% of Debian's code was written in D and the Debian D team was really large we could maybe get one, but not the way it is now). And tbh, I think finding a good solution here is entirely possible.
Apr 10 2017
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 10 April 2017 at 12:59:37 UTC, Matthias Klumpp wrote:
 Who came up with those policies and decided that they apply to 
 D? Because I really don't think they should.
They are the result of years of experience in building complex systems and keeping them secure. If you have a dependency chain "X -> Y -> Z" (-> meaning "depends on"), and you find a security bug in Z, you the security team will just need to fix the bug in Z to resolve it in the whole distribution. But if the code which has this issue is compiled into all of the packages that depend on them, you will need to rebuild the full dependency chain to actually fix the security issue, which is not only time intensive but also a huge maintenance effort. In this simple example it doesn't look like much, but those dependency chains can grow massively large and complicated, and the only way to keep the large software stack maintainable and secure is by splitting pieces cleanly. Embedded code copies are allowed in rare events, but in these cases the security team needs to be aware of them. Sometimes, the licenses also explicitly prevent embedded code copies. Aside from these issues, splitting things cleanly also makes general package maintenance much easier, and adds flexibility for our users who can mix and match parts of the distribution as they like and combine them with their own code.
No, I understand all of this. What I'm saying that in the case of D, these rules, though making sense, will just not work. You can't replace a piece of code in a template instantiation in a compiled program, shared libraries and stable ABI or not.
 You need to see here that D is not the center of the world and 
 we will need to make it work nicely with the rest of the system.
The opposite is also true: requiring a stable shared library API of every packaged D library is just as unreasonable. In fact, to make these rules useful and applicable to all D programs, you'd have to completely forbid templates in the library's public interface, which would immediately exclude Phobos for one.
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 13:07:22 UTC, Vladimir Panteleev 
wrote:
 On Monday, 10 April 2017 at 12:59:37 UTC, Matthias Klumpp wrote:
 Who came up with those policies and decided that they apply 
 to D? Because I really don't think they should.
[...] You need to see here that D is not the center of the world and we will need to make it work nicely with the rest of the system.
The opposite is also true: requiring a stable shared library API of every packaged D library is just as unreasonable. In fact, to make these rules useful and applicable to all D programs, you'd have to completely forbid templates in the library's public interface, which would immediately exclude Phobos for one.
There is a really easy way to fix this: SONAMEs. Whenever you change something in the library breaking ABI or API, you bump it's SOVERSION, which will force the distribution to perform a transition and rebuild the dependency chain. If you give absolutely zero stability guarantees, you just set the SOVERSION equal to the project's version and trigger a transition every time (incredibly annoying, but, well, okay). This has worked nicely for every language. If you don't have templates in your API or don't change the templates between releases, you can survive with one library for a long time. This is working really great on the level of individual libraries, but if the whole language is ABI-unstable, the issues are much bigger and harder to track. Btw, at time we are just ignore the ABI issues, and surprisingly nothing broke yet, indicating that ABI breakage isn't very common or not affecting commonly used interfaces much.
Apr 10 2017
next sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 April 2017 at 15:20, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 10 April 2017 at 13:07:22 UTC, Vladimir Panteleev wrote:
 On Monday, 10 April 2017 at 12:59:37 UTC, Matthias Klumpp wrote:
 Who came up with those policies and decided that they apply to D?
 Because I really don't think they should.
[...] You need to see here that D is not the center of the world and we will need to make it work nicely with the rest of the system.
The opposite is also true: requiring a stable shared library API of every packaged D library is just as unreasonable. In fact, to make these rules useful and applicable to all D programs, you'd have to completely forbid templates in the library's public interface, which would immediately exclude Phobos for one.
There is a really easy way to fix this: SONAMEs. Whenever you change something in the library breaking ABI or API, you bump it's SOVERSION, which will force the distribution to perform a transition and rebuild the dependency chain. If you give absolutely zero stability guarantees, you just set the SOVERSION equal to the project's version and trigger a transition every time (incredibly annoying, but, well, okay).
I believe GDC in Debian is done correctly on this part. Last time someone else looked, it seemed like LDC and DMD make use of SOVERSION, but do so in an incorrect manner.
 This has worked nicely for every language. If you don't have templates in
 your API or don't change the templates between releases, you can survive
 with one library for a long time.

 This is working really great on the level of individual libraries, but if
 the whole language is ABI-unstable, the issues are much bigger and harder to
 track.

 Btw, at time we are just ignore the ABI issues, and surprisingly nothing
 broke yet, indicating that ABI breakage isn't very common or not affecting
 commonly used interfaces much.
Everyone should follow GDC's ABI, rather than trying to mimic DMD calling convention. ;-)
Apr 10 2017
next sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:
 [...]
 Everyone should follow GDC's ABI, rather than trying to mimic 
 DMD calling convention. ;-)
GDC is working very well, and using it would actually be the natural choice for us as GCC is the default compiler. However, there are a few problems with that: 1) GDC doesn't compile a large amount of D code currently out there due to having an outdated standard library and runtime. While we would like to use it, it doesn't help if it just can't compile the things. 2) GDC isn't part of the official GCC. This has quite wide reaching implications, most importantly other distros not offering it - in Fedora / Red Hat, this is the reason why GDC isn't available. This means fewer people test with GDC and we're basically compiling upstream projects with a compiler they never ever tested, potentially creating new issues that we can't easily forward upstream. This is less of an issue with LDC as LDC is more widely available in other distributions.
Apr 10 2017
parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 April 2017 at 19:48, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:
 [...]
 Everyone should follow GDC's ABI, rather than trying to mimic DMD calling
 convention. ;-)
GDC is working very well, and using it would actually be the natural choice for us as GCC is the default compiler. However, there are a few problems with that: 1) GDC doesn't compile a large amount of D code currently out there due to having an outdated standard library and runtime. While we would like to use it, it doesn't help if it just can't compile the things.
Master sports Phobos 2.071. Someone will have to see whether latter versions can be built using it.
  2) GDC isn't part of the official GCC. This has quite wide reaching
 implications, most importantly other distros not offering it - in Fedora /
 Red Hat, this is the reason why GDC isn't available. This means fewer people
 test with GDC and we're basically compiling upstream projects with a
 compiler they never ever tested, potentially creating new issues that we
 can't easily forward upstream. This is less of an issue with LDC as LDC is
 more widely available in other distributions.
This is an unfortunate distribution problem, things would be different if GCC were more like a library. It's not like anyone is helping me with the push. I am ultimately the one who is doing the tens of thousands of lines of code changes, and writing up the thousands of lines of documentation just to make it lean and presentable. ;-)
Apr 10 2017
next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:
 Master sports Phobos 2.071.  Someone will have to see whether 
 latter versions can be built using it.
… and some weird Frankensteinian mix of several frontend versions, I take it, maybe enough to build Phobos, but not necessarily compatible for user code? Or did you port all the changes since 2.068.2 back to C++? — David
Apr 10 2017
parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 April 2017 at 23:52, David Nadlinger via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:
 Master sports Phobos 2.071.  Someone will have to see whether latter
 versions can be built using it.
… and some weird Frankensteinian mix of several frontend versions, I take it, maybe enough to build Phobos, but not necessarily compatible for user code? Or did you port all the changes since 2.068.2 back to C++? — David
All the regression fixes and none of the bugs! The current situation is that it should be link-compatible with current upstream/stable. Enough so that when someone has the time to test, it should just be a case switching the sources and building the D version.
Apr 10 2017
next sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:
 On 10 April 2017 at 23:52, David Nadlinger via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:
 Master sports Phobos 2.071.  Someone will have to see whether 
 latter versions can be built using it.
… and some weird Frankensteinian mix of several frontend versions, I take it, maybe enough to build Phobos, but not necessarily compatible for user code? Or did you port all the changes since 2.068.2 back to C++? — David
All the regression fixes and none of the bugs! The current situation is that it should be link-compatible with current upstream/stable. Enough so that when someone has the time to test, it should just be a case switching the sources and building the D version.
First of all, thank you for your tremendous work on GDC! Fellow developers and me were also pretty stunned by you maintaining a quite large amount of different GDC versions in parallel without a huge team - that's some impressive work! What is the thing that's blocking GDCs GCC inclusion? Just manpower? Also, you were talking about "bugs" on several occasions, what's the thing with that? Is it GCC or general Phobos bugs? It would probably be awesome to have a summary blogpost or similar on the state of GDC, that could potentially also attract volunteers. Anyway, all a bit off-topic :-)
Apr 11 2017
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 11 April 2017 at 12:11:10 UTC, Matthias Klumpp wrote:
 On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:
 All the regression fixes and none of the bugs!

 The current situation is that it should be link-compatible 
 with current upstream/stable.  Enough so that when someone has 
 the time to test, it should just be a case switching the 
 sources and building the D version.
First of all, thank you for your tremendous work on GDC! Fellow developers and me were also pretty stunned by you maintaining a quite large amount of different GDC versions in parallel without a huge team - that's some impressive work! What is the thing that's blocking GDCs GCC inclusion? Just manpower? Also, you were talking about "bugs" on several occasions, what's the thing with that? Is it GCC or general Phobos bugs? It would probably be awesome to have a summary blogpost or similar on the state of GDC, that could potentially also attract volunteers. Anyway, all a bit off-topic :-)
I assume the bugs would be DMDFE bugs. Iain is giving talk at DConf on GDC (http://dconf.org/2017/talks/buclaw.html) which should help.
Apr 11 2017
prev sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 14:11, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:
 On 10 April 2017 at 23:52, David Nadlinger via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:
 Master sports Phobos 2.071.  Someone will have to see whether latter
 versions can be built using it.
… and some weird Frankensteinian mix of several frontend versions, I take it, maybe enough to build Phobos, but not necessarily compatible for user code? Or did you port all the changes since 2.068.2 back to C++? — David
All the regression fixes and none of the bugs! The current situation is that it should be link-compatible with current upstream/stable. Enough so that when someone has the time to test, it should just be a case switching the sources and building the D version.
First of all, thank you for your tremendous work on GDC! Fellow developers and me were also pretty stunned by you maintaining a quite large amount of different GDC versions in parallel without a huge team - that's some impressive work!
It was worse when we were using `#if D_GCC_VER == 33' macros to support all versions in the same branch. ;-) Having a really good CI helps too for when the periodic merges are done.
 What is the thing that's blocking GDCs GCC inclusion? Just manpower? Also,
 you were talking about "bugs" on several occasions, what's the thing with
 that? Is it GCC or general Phobos bugs? It would probably be awesome to have
 a summary blogpost or similar on the state of GDC, that could potentially
 also attract volunteers.
 Anyway, all a bit off-topic :-)
Manpower has got something to do with it when you have two fast ships sailing and you have one foot in each. I think I only mentioned bugs once here, and was talking strictly about the D front-end, which deals with the parsing and semantic analysis of D sources. Every release there are listed new features, regressions fixed, etc. The regressions fixed have a higher precedent for me at least.
Apr 11 2017
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:
 All the regression fixes and none of the bugs!
That's an interesting approach (spoken with British language sensibilities). Is anybody using GDC on a big "modern" D2 codebase (metaprogramming-heavy, …) right now? From my experience with Thrift, Weka, and so on, even regression fixes have a habit of breaking that sort of code in weird and wonderful ways. And that's not even mentioning other improvements/fixes/additions people might be relying on. Right now, the frontend version serves as a convenient aid for users to navigate those issues (apart, of course, from us trying to improve things by making semantic analysis more deterministic, etc). If something works on DMD 2.072.2, then users can also expect it to work on the corresponding LDC version as well. I'm not sure you are helping anybody by introducing another set of weird in-between versions. — David
Apr 11 2017
parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 20:13, David Nadlinger via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:
 All the regression fixes and none of the bugs!
That's an interesting approach (spoken with British language sensibilities). Is anybody using GDC on a big "modern" D2 codebase (metaprogramming-heavy, …) right now? From my experience with Thrift, Weka, and so on, even regression fixes have a habit of breaking that sort of code in weird and wonderful ways. And that's not even mentioning other improvements/fixes/additions people might be relying on. Right now, the frontend version serves as a convenient aid for users to navigate those issues (apart, of course, from us trying to improve things by making semantic analysis more deterministic, etc). If something works on DMD 2.072.2, then users can also expect it to work on the corresponding LDC version as well. I'm not sure you are helping anybody by introducing another set of weird in-between versions. — David
And DMD didn't help by switching codebase to D without first ironing out all implementation regressions and interoperability. Every release between 2.069 and 2.074 is a broken release as far I'm concerned. The most recent is getting closer, but there are likely still many things missing that need to be exposed to C++ (mostly ::create functions).
Apr 11 2017
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 1:43 PM, Iain Buclaw via Digitalmars-d wrote:
 This is an unfortunate distribution problem, things would be different
 if GCC were more like a library.  It's not like anyone is helping me
 with the push.  I am ultimately the one who is doing the tens of
 thousands of lines of code changes, and writing up the thousands of
 lines of documentation just to make it lean and presentable.  ;-)
Iain, thank you for doing this herculean task all by yourself. I know that you need help - anyone want to step up? I hope that making D fully open source will have a side effect of motivating people to help you get gdc into the main gcc distribution.
Apr 10 2017
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:
 Last time someone else looked, it seemed like LDC and DMD make 
 use of SOVERSION, but do so in an incorrect manner.
You know what exactly is the problem? Any suggestion what to use instead? It's currently using libphobos2.so.0.74, where using major version 0 to mean unstable might be a misapplication of semver (http://semver.org/#spec-item-4) for SONAME. Our point releases might also contain small ABI incompatibilities, so they aren't really eligible as patch version. https://github.com/dlang/phobos/blob/c648097621db225aa59e07d775f54dbaa847c68a/posix.mak#L151
Apr 12 2017
next sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 12 Apr 2017 07:42:42 +0000
schrieb Martin Nowak <code dawg.eu>:

 On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:
 Last time someone else looked, it seemed like LDC and DMD make 
 use of SOVERSION, but do so in an incorrect manner.  
You know what exactly is the problem? Any suggestion what to use instead? It's currently using libphobos2.so.0.74, where using major version 0 to mean unstable might be a misapplication of semver (http://semver.org/#spec-item-4) for SONAME.
I still haven't found some definitive documentation about this, but it seems linux shared library working essentially works like this: There's a major and a minor number. There's sometimes a patch version, but there's no conclusive documentation. Reading the ldconfig source code you can have as many version levels as you want [3] (though essentially this is treated like one minor version, it only affects comparing minor versions). But there are different ldconfig implementations (linux, bsd) so I don't know if multi-level minor versions are portable. * Major version reflects ABI level. A new major version can break ABI or add new stuff to ABI. * Minor versions can only extend the ABI of the major version but should not break any ABI. * Micro / Patch version is mostly unused in ldconfig. It only affects comparing minor versions, i.e. when you have libphobos.so.74.0.1 and libphobos.so.74.0.2 ldconfig will link libphobos.so.74=>libphobos.so.74.0.2 Filename format: lib[name].so.[major].[minor][.patch] Soname format: always lib[name].so.[major] It is possible to install and use multiple major versions. Every major version will always use the last installed minor version. (The distribution will manage symlinks for libfoo.[major] to libfoo.[major].[minor] for the largest minor version). Additionally a libfoo.so is installed (for linking / development only, might even be in -dev packages) to point to the latest libfoo.[major] symlink. The libfoo.[major] symlink is used when linking with -lfoo. The dependency encoded in the executable will use the soname though, so it will encode libfoo.[major]. If you install a new major library version, all existing executables will continue to use the major version hardcoded in the executable. If you update a major version to a new minor version, all executables using the major version soname will use the new minor version. This means: * Increase major every time you break ABI * Increase some minor level every time you only extend ABI So DMD should not keep the major version fixed as 0 (every time you update libphobos you break all existing binaries, as you break the ABI of libphobos.so.0). In GDC we use libtool which encodes like this [1][2]: libgphobos.so.[major].0.[release] e.g. libgphobos.74.0.2 This is not 100% safe if a minor release breaks ABI though. BTW: Interestingly even with these complicated rules you can end up in situation where versioning does not work: If you link application APP against libfoo.so on a system with libfoo.so.1.2 the soname will only encode libfoo.so.1. Now ship the library to a system with libfoo.so.1.1 and you may have missing symbols due to extended ABI in libfoo.so.1.2. TLDR: Downgrading libraries is not safe with this versioning approach. [1] https://autotools.io/libtool/version.html [2] https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html ldconfig is responsible for maintaining the symlinks, here's how it determines which version of a library is newer: [3] https://github.com/lattera/glibc/blob/master/elf/ldconfig.c#L939 https://github.com/lattera/glibc/blob/master/elf/dl-cache.c#L138
Apr 12 2017
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 12 Apr 2017 07:42:42 +0000
schrieb Martin Nowak <code dawg.eu>:

 Our point releases might also contain small ABI 
 incompatibilities, so they aren't really eligible as patch 
 version.
I've actually been hit by this in one point release on Gentoo, where I used dynamic linking for Dlang as soon as it was possible. Programs would suddenly complain about a mismatch between the size of some compiler generated item after a point release update of dmd. Just posting this to give the discussion a bit more substance. :) -- Marco
May 02 2017
prev sibling next sibling parent reply Johan Engelen <j j.nl> writes:
On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
 
 Btw, at time we are just ignore the ABI issues, and 
 surprisingly nothing broke yet, indicating that ABI breakage 
 isn't very common or not affecting commonly used interfaces 
 much.
One big ABI change was in 2.071: https://issues.dlang.org/show_bug.cgi?id=15644. And it involved interfaces. ;-) Nothing broke because of that? cheers, Johan
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 16:58:05 UTC, Johan Engelen wrote:
 On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
 
 Btw, at time we are just ignore the ABI issues, and 
 surprisingly nothing broke yet, indicating that ABI breakage 
 isn't very common or not affecting commonly used interfaces 
 much.
One big ABI change was in 2.071: https://issues.dlang.org/show_bug.cgi?id=15644. And it involved interfaces. ;-) Nothing broke because of that?
I am reading release notes, so we rebuilt dependencies of LDC - I have no idea about GDC-depending D code though. But since no bugs were reported, I assume no issues are present :-)
Apr 10 2017
next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 April 2017 at 19:50, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 10 April 2017 at 16:58:05 UTC, Johan Engelen wrote:
 On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
 Btw, at time we are just ignore the ABI issues, and surprisingly nothing
 broke yet, indicating that ABI breakage isn't very common or not affecting
 commonly used interfaces much.
One big ABI change was in 2.071: https://issues.dlang.org/show_bug.cgi?id=15644. And it involved interfaces. ;-) Nothing broke because of that?
I am reading release notes, so we rebuilt dependencies of LDC - I have no idea about GDC-depending D code though. But since no bugs were reported, I assume no issues are present :-)
That change is in gdc-7.x - so still in Debian unstable.
Apr 10 2017
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 17:50:08 UTC, Matthias Klumpp wrote:
 I am reading release notes, so we rebuilt dependencies of LDC -
(I assume you mean reverse dependencies.)
 […] But since no bugs were reported, I assume no issues are 
 present :-)
So do we need to put a reminder about the ABI being unstable into set of every release notes to make sure we won't get angry bug reports once users actually build their own D code against your packages? ;) — David
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 22:15:53 UTC, David Nadlinger wrote:
 On Monday, 10 April 2017 at 17:50:08 UTC, Matthias Klumpp wrote:
 I am reading release notes, so we rebuilt dependencies of LDC -
(I assume you mean reverse dependencies.)
 […] But since no bugs were reported, I assume no issues are 
 present :-)
So do we need to put a reminder about the ABI being unstable into set of every release notes to make sure we won't get angry bug reports once users actually build their own D code against your packages? ;)
Nah, there are several options here, one would simply be to tell people not to use the distro packages with anything but the default D compiler used in the respective Debian release. Go apparently tells people not to use Debian-shipped go code in their own projects at all.
Apr 11 2017
next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 12:03 +0000, Matthias Klumpp via Digitalmars-d
wrote:
=20
[=E2=80=A6]
 Nah, there are several options here, one would simply be to tell=C2=A0
 people not to use the distro packages with anything but the=C2=A0
 default D compiler used in the respective Debian release.
 Go apparently tells people not to use Debian-shipped go code in=C2=A0
 their own projects at all.
The vendoring systems that Go folk have invented are effectively mandatory for projects that want reproducible builds, and using platform specific code is not feasible. It suprises me that Debian and Fedora are going flat out trying to package Go stuff. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 11 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 12:42:13 UTC, Russel Winder wrote:
 On Tue, 2017-04-11 at 12:03 +0000, Matthias Klumpp via 
 Digitalmars-d wrote:
 
[…]
 Nah, there are several options here, one would simply be to 
 tell
 people not to use the distro packages with anything but the
 default D compiler used in the respective Debian release.
 Go apparently tells people not to use Debian-shipped go code in
 their own projects at all.
The vendoring systems that Go folk have invented are effectively mandatory for projects that want reproducible builds, and using platform specific code is not feasible. It suprises me that Debian and Fedora are going flat out trying to package Go stuff.
That's false. Debian is leading the effort on reproducible builds that many other projects (including Fedora) have joined, and a large chunk of packages is already reproducible[1]. It's actually quite the opposite: Build systems downloading random stuff from the internet make the system more likely to produce different build results. But in any case, the primary use for Debian packages is to be used by the distribution. [1]: https://tests.reproducible-builds.org/debian/reproducible.html
Apr 11 2017
parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 12:48 +0000, Matthias Klumpp via Digitalmars-d
wrote:
[=E2=80=A6]
 That's false. Debian is leading the effort on reproducible builds=C2=A0
 that many other projects (including Fedora) have joined, and a=C2=A0
 large chunk of packages is already reproducible[1].
 It's actually quite the opposite: Build systems downloading=C2=A0
 random stuff from the internet make the system more likely to=C2=A0
 produce different build results.
You are closer the that community that I am and so have better knowledge there. My comment was really reflecting the cant about the OS packages.
 But in any case, the primary use for Debian packages is to be=C2=A0
 used by the distribution.
Where Debian and Fedora provide applications written in Go they will be statically linked. So is the use case that everything used to create the application has to be packaged?
 [1]:=C2=A0
 https://tests.reproducible-builds.org/debian/reproducible.html
--=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 11 2017
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Tuesday, 11 April 2017 at 12:03:27 UTC, Matthias Klumpp wrote:
 On Monday, 10 April 2017 at 22:15:53 UTC, David Nadlinger wrote:
 So do we need to put a reminder about the ABI being unstable 
 into set of every release notes to make sure we won't get 
 angry bug reports once users actually build their own D code 
 against your packages? ;)
Nah, there are several options here, one would simply be to tell people not to use the distro packages with anything but the default D compiler used in the respective Debian release.
So as long as one sticks to packages in the official apt repos, all the libraries are guaranteed to be built with the distributed compiler as well? When you mentioned that you'd read the release notes regarding the ABI change, I got the impression that you had to manually rebuild the world for that to happen – hence my tongue-in-cheek remark about reminding you to do this in the release notes. Otherwise, you might get lucky as far as the distributed applications are concerned (i.e. happen not to hit any ABI issues), but users might still be hosed when it comes to their own code. — David
Apr 11 2017
parent Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 18:14:40 UTC, David Nadlinger wrote:
 On Tuesday, 11 April 2017 at 12:03:27 UTC, Matthias Klumpp 
 wrote:
 On Monday, 10 April 2017 at 22:15:53 UTC, David Nadlinger 
 wrote:
 So do we need to put a reminder about the ABI being unstable 
 into set of every release notes to make sure we won't get 
 angry bug reports once users actually build their own D code 
 against your packages? ;)
Nah, there are several options here, one would simply be to tell people not to use the distro packages with anything but the default D compiler used in the respective Debian release.
So as long as one sticks to packages in the official apt repos, all the libraries are guaranteed to be built with the distributed compiler as well?
Yes. Unfortunately there will be three of them which aren't compatible with each other, so we will kind of have to settle with one as default.
 When you mentioned that you'd read the release notes regarding 
 the ABI change, I got the impression that you had to manually 
 rebuild the world for that to happen – hence my tongue-in-cheek 
 remark about reminding you to do this in the release notes.
Well, it's a matter of telling the build admins or making a proper transition package (doesn't exit yet for D), but yeah, technically we'd need to rebuild all D stuff on ABI changes.
Apr 11 2017
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
 This has worked nicely for every language. If you don't have 
 templates in your API or don't change the templates between 
 releases, you can survive with one library for a long time.
But the vast majority of D libraries _do_ have templates (starting with Phobos). How should this situation be dealt with? How does Debian deal with, e.g., fixes to the templated code in Boost, which impact on other packages built using those header-only libraries?
Apr 10 2017
parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 22:26:46 UTC, Joseph Rushton Wakeling 
wrote:
 On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
 This has worked nicely for every language. If you don't have 
 templates in your API or don't change the templates between 
 releases, you can survive with one library for a long time.
But the vast majority of D libraries _do_ have templates (starting with Phobos). How should this situation be dealt with? How does Debian deal with, e.g., fixes to the templated code in Boost, which impact on other packages built using those header-only libraries?
Boost's soversion is changed on every release, and the version is included in it's -dev package as well. That's why we have libboost1.62-dev: https://packages.debian.org/de/sid/libboost1.62-dev (and possibly more). There is also a boost-defaults package setting the current default Boos version for packages to depend on. If a new Boost comes out, it's soversion and -dev package name changes, triggering a package transition and subsequently a full rebuild of all stuff depending on Boost. Doing something like this with D libraries would obviously be possible as well.
Apr 11 2017
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 5:59 AM, Matthias Klumpp wrote:
 You need to see here that D is not the center of the world and we will need to
 make it work nicely with the rest of the system. The technical policies work
for
 everything else, so there is nothing that really justifies an exception for D
 here (if 10% of Debian's code was written in D and the Debian D team was really
 large we could maybe get one, but not the way it is now).
 And tbh, I think finding a good solution here is entirely possible.
I think it is possible, too, and thank you for your efforts helping us do this. The ABI differences between the compilers is unfortunate, and is largely the result of historical accident. The first problem is that my idea originally was for D to have its own function calling convention, which would free us to innovate to have a more efficient calling convention than the C one. This hasn't panned out in practice, and ldc/gdc decided to sensibly stick with the C ABI. At some point, we should just crowbar dmd to generate the C ABI, but this has its own problems - it'll break code that uses the inline assembler. No obvious solution there. The next problem is that dmd occasionally changes the interface to the D runtime. Or more accurately, with about every release. This has not been an issue historically for us, as the two have always been a matched set. I'm a lot less sure how to deal with this. I also do not know how the gdc/lds druntime interfaces differ.
Apr 10 2017
next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 01:27, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 The next problem is that dmd occasionally changes the interface to the D
 runtime. Or more accurately, with about every release. This has not been an
 issue historically for us, as the two have always been a matched set. I'm a
 lot less sure how to deal with this. I also do not know how the gdc/lds
 druntime interfaces differ.
If you are referring to things such as _d_throw, _d_throwc, _d_throwdwarf, others... In most cases, GDC has always done things in the way that works for it, and as a result has almost never changed ABI of library functions that it is tied to. So we just have the one _d_throw function for throwing dwarf EH, while DMD has gone through several iterations over the years before finding themselves doing things the same way as GDC - by which point you've given the function a different name. ;-)
Apr 10 2017
prev sibling next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 23:27:35 UTC, Walter Bright wrote:
 The next problem is that dmd occasionally changes the interface 
 to the D runtime. […] I also do not know how the gdc/lds 
 druntime
 interfaces differ.
Just to make this very clear to everybody reading this thread: It's not even just that, but also the fact that we guarantee API-, but not ABI-stability for Phobos. Every time we continue to improve the pure/nothrow/ nogc situation by marking up some more code, we are breaking the ABI, because the mangled names of the involved symbols change. The ongoing work on `scope` also breaks the ABI when enabled. Yes, there are also some differences between the DMD/GDC/LDC runtime interfaces, but this is a comparatively very minor issue that doesn't make sense worrying too much about as long as ABI compatibility between releases isn't going to happen for other reasons such as the above. [1] — David [1] In fact, it looks like – for example with DMD moving to libunwind-based EH as well – the issue is slowly resolving itself anyway and at some point we'll merely have to sit down for a week and iron out the last few kinks.
Apr 10 2017
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 4:43 PM, David Nadlinger wrote:
 [1] In fact, it looks like – for example with DMD moving to libunwind-based
EH
 as well – the issue is slowly resolving itself anyway and at some point we'll
 merely have to sit down for a week and iron out the last few kinks.
dmd is not moving to a libunwind-based EH, it already has moved to it! But as an example of ABI change, the latest EH nogc proposal changes the layout of the Throwable object, and adds some code to the _d_throwdwarf() function. https://github.com/dlang/druntime/pull/1804
Apr 10 2017
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-04-11 02:20, Walter Bright wrote:
 On 4/10/2017 4:43 PM, David Nadlinger wrote:
 [1] In fact, it looks like – for example with DMD moving to
 libunwind-based EH
 as well – the issue is slowly resolving itself anyway and at some
 point we'll
 merely have to sit down for a week and iron out the last few kinks.
dmd is not moving to a libunwind-based EH, it already has moved to it! But as an example of ABI change, the latest EH nogc proposal changes the layout of the Throwable object, and adds some code to the _d_throwdwarf() function. https://github.com/dlang/druntime/pull/1804
It is possible to solve. For example, in Objective-C they have a non-fragile ABI where you can freely add instance variables to base classes and the offset will be adjusted automatically at load time for the subclasses [1]. The methods are already handled with dynamic message dispatch. [1] http://www.sealiesoftware.com/blog/archive/2009/01/27/objc_explain_Non-fragile_ivars.html -- /Jacob Carlborg
Apr 11 2017
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2017 6:46 AM, Jacob Carlborg wrote:
 http://www.sealiesoftware.com/blog/archive/2009/01/27/objc_explain_Non-fragile_ivars.html
Looks like it solves it by adding another layer of indirection: "the performance overhead of the extra ivar offset variable is small."
Apr 11 2017
parent Jacob Carlborg <doob me.com> writes:
On 2017-04-11 19:34, Walter Bright wrote:

 Looks like it solves it by adding another layer of indirection:

 "the performance overhead of the extra ivar offset variable is small."
Yes, there are always tradeoffs. But as you've said, exceptions are already slow ;) -- /Jacob Carlborg
Apr 11 2017
prev sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 23:43:04 UTC, David Nadlinger wrote:
 On Monday, 10 April 2017 at 23:27:35 UTC, Walter Bright wrote:
 The next problem is that dmd occasionally changes the 
 interface to the D runtime. […] I also do not know how the 
 gdc/lds druntime
 interfaces differ.
Just to make this very clear to everybody reading this thread: It's not even just that, but also the fact that we guarantee API-, but not ABI-stability for Phobos. Every time we continue to improve the pure/nothrow/ nogc situation by marking up some more code, we are breaking the ABI, because the mangled names of the involved symbols change. The ongoing work on `scope` also breaks the ABI when enabled.
If you could change the SOVERSION with every one of these changes, or simply just tie it to the respective Phobos release, distributions would automatically do the right thing and compile all D code using Phobos against the new version. This might give Phobos a large soversion or and ugly one like "2.074", but it would work. (LDC's Phobos and GDC's Phobos already have a soversion set in Debian...)
Apr 11 2017
parent reply David Nadlinger <code klickverbot.at> writes:
On Tuesday, 11 April 2017 at 12:38:01 UTC, Matthias Klumpp wrote:
 If you could change the SOVERSION with every one of these 
 changes, or simply just tie it to the respective Phobos 
 release, distributions would automatically do the right thing 
 and compile all D code using Phobos against the new version.
As you mention, this is already done in LDC; not just the Debian packages, but also upstream. The soname will always be `libphobos2-ldc.so.74` or what have you. (Thinking about it, it should probably include the patch version as well, although we haven't had a situation so far where we would have wanted to release multiple LDC versions for ABI-incompatible patch releases.) — David
Apr 11 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 15:31:46 UTC, David Nadlinger wrote:
 On Tuesday, 11 April 2017 at 12:38:01 UTC, Matthias Klumpp 
 wrote:
 If you could change the SOVERSION with every one of these 
 changes, or simply just tie it to the respective Phobos 
 release, distributions would automatically do the right thing 
 and compile all D code using Phobos against the new version.
As you mention, this is already done in LDC; not just the Debian packages, but also upstream. The soname will always be `libphobos2-ldc.so.74` or what have you. (Thinking about it, it should probably include the patch version as well, although we haven't had a situation so far where we would have wanted to release multiple LDC versions for ABI-incompatible patch releases.)
Phobos is versioned properly in both GDC and LDC, and as long as that continues, no problems exist at all :-) https://packages.debian.org/search?suite=stretch&keywords=phobos This would - as said - also work for other D shared libraries, unless the D ABI in general breaks or someone tries to build a program with GDC and uses a library that was built with LDC or DMD (or any other possible compiler combination).
Apr 11 2017
parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 17:48, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 11 April 2017 at 15:31:46 UTC, David Nadlinger wrote:
 On Tuesday, 11 April 2017 at 12:38:01 UTC, Matthias Klumpp wrote:
 If you could change the SOVERSION with every one of these changes, or
 simply just tie it to the respective Phobos release, distributions would
 automatically do the right thing and compile all D code using Phobos against
 the new version.
As you mention, this is already done in LDC; not just the Debian packages, but also upstream. The soname will always be `libphobos2-ldc.so.74` or what have you. (Thinking about it, it should probably include the patch version as well, although we haven't had a situation so far where we would have wanted to release multiple LDC versions for ABI-incompatible patch releases.)
Phobos is versioned properly in both GDC and LDC, and as long as that continues, no problems exist at all :-) https://packages.debian.org/search?suite=stretch&keywords=phobos This would - as said - also work for other D shared libraries, unless the D ABI in general breaks or someone tries to build a program with GDC and uses a library that was built with LDC or DMD (or any other possible compiler combination).
I thought I might try a "will it blend" experiment: $ cat foo.d module foo; int times2(int a) { return a * 2; } void throwIt(Throwable t) { throw t; } $ cat test.d import foo; void main() { import std.stdio; writeln("2 x 2 = ", times2(2)); Throwable t = new Exception("Die!"); try { throwIt(t); } catch (Exception e) { writeln("Caught Exception for: ", e.msg); } } 1. Naive: $ dmd -g -shared foo.d $ ldd foo.so linux-vdso.so.1 => (0x00007ffc1a293000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0f78bed000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0f789e5000) libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f0f787ce000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0f78405000) /lib64/ld-linux-x86-64.so.2 (0x0000557aa8c5f000) $ gdc -g test.d foo.so -shared-libphobos $ ldd a.out linux-vdso.so.1 => (0x00007fff020b9000) foo.so => ./foo.so (0x00007f262d66d000) libgphobos.so.71 => //usr/lib64/libgphobos.so.71 (0x00007f262cfd7000) libgdruntime.so.71 => /usr/lib64/libgdruntime.so.71 (0x00007f262ccde000) libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f262cac8000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f262c6d6000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f262c3cc000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f262c1af000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f262bfab000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f262bda2000) /lib64/ld-linux-x86-64.so.2 (0x0000564d0bc19000) $ ./a.out Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7bc01ae in gc_disable () from ./foo.so Ouch, DMD's statically linked libphobos has overridden GDC's gc_disable(). 2. Mixing shared libs. $ dmd -g -defaultlib=phobos2 -shared foo.d $ ldd foo.so linux-vdso.so.1 => (0x00007ffc84dfb000) --> libphobos2.so.0.74 => /usr/lib/x86_64-linux-gnu/libphobos2.so.0.74 (0x00007f7574f53000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7574b8a000) --> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7574985000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7574768000) --> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f757445f000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7574256000) libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f7574040000) /lib64/ld-linux-x86-64.so.2 (0x000055c715c3e000) $ gdc -g test.d foo.so -shared-libphobos $ ldd a.out linux-vdso.so.1 => (0x00007ffd9e3eb000) foo.so => ./foo.so (0x00007fd09d9cc000) libgphobos.so.71 => /usr/lib64/libgphobos.so.71 (0x00007fd09d336000) libgdruntime.so.71 => /usr/lib64/libgdruntime.so.71 (0x00007fd09d03d000) libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fd09ce27000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd09ca35000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd09c72b000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd09c50e000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd09c30a000) --> libphobos2.so.0.74 => /usr/lib/x86_64-linux-gnu/libphobos2.so.0.74 (0x00007fd09bae2000) /lib64/ld-linux-x86-64.so.2 (0x000056135abd3000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd09b8da000) $ ./a.out Fatal Error while loading '/usr/lib/x86_64-linux-gnu/libphobos2.so.0.74': The module 'std.array' is already defined in '/usr/lib64/libgphobos.so.71'. This seems bad, but it's actually a good sign. GDC's copy of rt.sections.checkModuleCollisions() has loaded DMD's shared libphobos and has correctly determined that in there exists a module that shares the same name. 3. Telling DMD not to link libphobos $ dmd -g -defaultlib= -shared foo.d $ ldd foo.so linux-vdso.so.1 => (0x00007ffdc2155000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f44ede1d000) /lib64/ld-linux-x86-64.so.2 (0x000055e14d836000) $ gdc -g test.d foo.so -shared-libphobos foo.so: undefined reference to `_d_throwdwarf' collect2: error: ld returned 1 exit status This is expected, GDC's libgdruntime doesn't have this. So I added it as a stub: extern(C) void _d_throwdwarf(Throwable o) { throw o; } $ gdc -g test.d foo.so -shared-libphobos $ ldd a.out linux-vdso.so.1 => (0x00007fffb05ee000) foo.so => ./foo.so (0x00007fe818644000) libgphobos.so.71 => /usr/lib64/libgphobos.so.71 (0x00007fe817fae000) libgdruntime.so.71 => /usr/lib64/libgdruntime.so.71 (0x00007fe817cb5000) libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fe817a9f000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe8176ad000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe8173a3000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe817186000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe816f82000) /lib64/ld-linux-x86-64.so.2 (0x00005644b677f000) $ ./a.out 2 x 2 = 4 Caught Exception for: Die! Success! There may be some hope, but you currently need a compatibility library for linking DMD compiled objects with GDC programs. And you must cut out Phobos out of the third party D library so that the library for the compiler building the application holds the one and only reference to symbols.
Apr 11 2017
prev sibling parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 23:27:35 UTC, Walter Bright wrote:
 On 4/10/2017 5:59 AM, Matthias Klumpp wrote:
 You need to see here that D is not the center of the world and 
 we will need to
 make it work nicely with the rest of the system. The technical 
 policies work for
 everything else, so there is nothing that really justifies an 
 exception for D
 here (if 10% of Debian's code was written in D and the Debian 
 D team was really
 large we could maybe get one, but not the way it is now).
 And tbh, I think finding a good solution here is entirely 
 possible.
I think it is possible, too, and thank you for your efforts helping us do this. The ABI differences between the compilers is unfortunate, and is largely the result of historical accident. The first problem is that my idea originally was for D to have its own function calling convention, which would free us to innovate to have a more efficient calling convention than the C one. This hasn't panned out in practice, and ldc/gdc decided to sensibly stick with the C ABI. At some point, we should just crowbar dmd to generate the C ABI, but this has its own problems - it'll break code that uses the inline assembler. No obvious solution there. The next problem is that dmd occasionally changes the interface to the D runtime. Or more accurately, with about every release. This has not been an issue historically for us, as the two have always been a matched set. I'm a lot less sure how to deal with this. I also do not know how the gdc/lds druntime interfaces differ.
Thank you very much for this context! It's really good to know why things are the way they are to properly understand the problem (I am no compiler developer and ABIs are not my expertise - as outsider to this I see pages like https://dlang.org/spec/abi.html though and wonder whether the incompatibilities are seen as an issue in the D community and whether there is a chance to address them before putting work into setting up infrastructure to rebuild the world on compiler updates).
Apr 11 2017
prev sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 12:40:33 UTC, Vladimir Panteleev 
wrote:
 [...]
 Can we treat it more like an interpreted language instead?
An interpreted language would interpret the code on the target system at runtime. This is not what D does, so we can't really treat it like we treat Python (where it is possible no neatly separate Python modules into separate packages - transitions triggering rebuild cascades only exist when we jump to the next major CPython version, which is something distributions are well prepared for. Transitions are only an issue if they happen constantly). At time, D is treated like C++, since it has much of the same challenges and we know how to deal with C++ - additionally to C++, D unfortunately though also has the unique issues outlined above, which complicate things. I also want to stress that having a single C++ library like Boost compiled into stuff and rolling dependency transitions when its API/ABI changes with a major release is less of a problem than having the entire language give zero stability and interoperability guarantees on anything that is compiled with it.
Apr 10 2017
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 6:08 AM, Matthias Klumpp wrote:
 I also want to stress that having a single C++ library like Boost compiled into
 stuff and rolling dependency transitions when its API/ABI changes with a major
 release is less of a problem than having the entire language give zero
stability
 and interoperability guarantees on anything that is compiled with it.
How is the g++/clang++ issue handled?
Apr 10 2017
next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 01:33, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 4/10/2017 6:08 AM, Matthias Klumpp wrote:
 I also want to stress that having a single C++ library like Boost compiled
 into
 stuff and rolling dependency transitions when its API/ABI changes with a
 major
 release is less of a problem than having the entire language give zero
 stability
 and interoperability guarantees on anything that is compiled with it.
How is the g++/clang++ issue handled?
I suspect that clang++ keeps ABI compatibility with g++ for both mangling and calling convention, except when functions are not callable from outside the compilation unit (static functions, templates). Not sure about libstdc++, I suspect they are either compatible for certain things.
Apr 10 2017
prev sibling parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 23:33:17 UTC, Walter Bright wrote:
 On 4/10/2017 6:08 AM, Matthias Klumpp wrote:
 I also want to stress that having a single C++ library like 
 Boost compiled into
 stuff and rolling dependency transitions when its API/ABI 
 changes with a major
 release is less of a problem than having the entire language 
 give zero stability
 and interoperability guarantees on anything that is compiled 
 with it.
How is the g++/clang++ issue handled?
The C ABI is 100% compatible, the C++ ABI is "mostly" compatible, there is some deliberate breakage from the Clang guys though. The issue isn't actually handled in Debian as all our code is always compiled with GCC, I am not aware of anything defaulting to Clang (although it might exist, but definitely not for library packages). With only one dominant compiler, things are way easier ^^
Apr 11 2017
prev sibling next sibling parent reply qznc <qznc web.de> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
 So, in summary:
   1) Is there some perspective on D getting a defined ABI that 
 works with all major D compilers?
   2) What would the D community recommend on how to deal with 
 the ABI issues currently? A Linux distribution is a bunch of 
 tightly integrated software, and changing one piece in an 
 incompatible way (e.g. by building it with LDC instead of DMD) 
 will have consequences.
   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?

 I am interested in some feedback here, since I currently can't 
 see a good way to address these issues.
How do Debian and C++ go along? There is no ABI compatibility between GCC and Clang afaik. I general, I would favor to build stuff with LDC instead of DMD, because of performance.
Apr 10 2017
parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 12:59:58 UTC, qznc wrote:
 [...]
 How do Debian and C++ go along? There is no ABI compatibility 
 between GCC and Clang afaik.
Clang offers compatibility for most basic features. There are some ABI compatibility issues though and you find them reported in the Clang/libc++ bugtrackers, and it's a pain (but the Clang/LLVM guys think they can do some ABI better/faster than what GCC offers, so some breakage is deliberate). In Debian, GCC compiles everything as the system's default compiler, so at least inside the distribution we don't have to worry about potential incompatibilities. Since GCC also supports an enormous amount of architectures and has strong optimization, the case is different there. In terms of "what happens when users use the OSes C++ libraries and compile with Clang instead of GCC" the situation is similar though: They might run into ABI issues (rarer though than with D). For the distro itself the problem doesn't exist though.
Apr 10 2017
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
 Hi there!
 [...]
 If we do that, we will run into the D ABI trap: Libraries 
 compiled with compiler X can not be used from software compiled 
 with D compiler Y. There is actually no ABI stability guarantee 
 even between DMD releases.
 This will make integrating D a huge pain. Recompiling the 
 dependency-chain of a software from source when compiling a 
 package using the "right" compiler and statically adding the 
 code is forbidden by distro policy. Having static libraries in 
 the dependencies doesn't solve the issue. Compiling each 
 library with all D compilers is highly impractical and not 
 really feasible.
 So, how should we proceed here? We could make it "DMD is the 
 only thing on the highway" compiling everything with DMD with 
 zero exceptions, which would leave us with only DMD-internal 
 ABI breakage and bad D code performance for some libraries.
I am working on a project that will require to be used almost exclusively with LDC (targeting the GPU backends of LLVM), and while technically only the mangling needs to match using any other complier will likely be a major workflow impediment. Perhaps it is reasonable to make them compile the stdlibs for LDC, but if it becomes as popular as I hope (dominating it's niché) it may be worthwhile considering shipping prebuilts.
Apr 10 2017
prev sibling next sibling parent reply Gerald <gerald.b.nunn gmail.com> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
 There are a two issues though that we will be facing in Debian 
 soon, and I would like to get some opinion and maybe 
 perspective on from the D community on them.
First I would like to say thank you for all the work you did in getting Tilix packaged for Debian, it is very much appreciated. While I'm not an expert in this topic I'll throw in my two cents anyway :) So to the topic at end, I tend to agree with Vladimir in that I just don't see it as feasible to make every dependency a shared library. Personally I would suggest that only libphobos and libdruntime be managed as shared libraries by a distro. Otherwise, any other dependencies used in DUB or elsewhere would be statically compiled. If in time a new D library ends up becoming a keystone foundation for many packages it could be considered for inclusion as a shared library. Otherwise, trying to manage every DUB dependency as a potential shared library is a huge amount of work and I don't feel most of them are mature or well maintained enough to support this approach. If you just concentrate on compiler, libphobos and libdruntime, you can have separate packages for each compiler toolchain which was Arch does. It has a libphobos package for DMD and a liblphobos for LDC. This then enables developers to specify the tool chain they prefer without interference. In addition to the question about D and C++. what do distros typically do for Rust and Cargo dependencies, or Java and Maven? Wouldn't that be a similar paradigm to D and DUB dependencies?
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 14:21:43 UTC, Gerald wrote:
 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
 There are a two issues though that we will be facing in Debian 
 soon, and I would like to get some opinion and maybe 
 perspective on from the D community on them.
First I would like to say thank you for all the work you did in getting Tilix packaged for Debian, it is very much appreciated. While I'm not an expert in this topic I'll throw in my two cents anyway :)
Thanks!
 So to the topic at end, I tend to agree with Vladimir in that I 
 just don't see it as feasible to make every dependency a shared 
 library. Personally I would suggest that only libphobos and 
 libdruntime be managed as shared libraries by a distro. 
 Otherwise, any other dependencies used in DUB or elsewhere 
 would be statically compiled.
This would raise the maintenance effort for D software, and require policy exceptions, which is something we won't do. It's more likely that we'll see less D in the distro than going down this route. DUB is unusable for Debian packaging anyway, and alternatives like Automake or Meson handle shared libraries really well, so in itself there is no reason not to do it.
 If in time a new D library ends up becoming a keystone 
 foundation for many packages it could be considered for 
 inclusion as a shared library. Otherwise, trying to manage 
 every DUB dependency as a potential shared library is a huge 
 amount of work and I don't feel most of them are mature or well 
 maintained enough to support this approach.
We'll take care of that, it's what we do as a distro and we have a fair amount of experience in handling these kinds of upstream projects. The goal is to reduce the maintenance cost for things once they are *in* the distro, any prior work to get it into the distribution is very well invested. That's why I have been writing a lot of Makefiles and Meson build definitions lately.
 If you just concentrate on compiler, libphobos and libdruntime, 
 you can have separate packages for each compiler toolchain 
 which was Arch does.
That's a necessity already and we do the same. But with an unstable ABI the standard library is also affected, which would trigger us to do Haskell-style versioning (which mangles dependencies to depend on a virtual package containing a hash of the GHC version), and that not only sucks but also requires quite a lot of manpower.
 It has a libphobos package for DMD and a liblphobos for LDC. 
 This then enables developers to specify the tool chain they 
 prefer without interference.
Yes, but that's not feasible for use *in the distribution itself* as we can't just pick and choose the right compiler per package if something up in the dependency chain was compiled with the "wrong" compiler.
 In addition to the question about D and C++. what do distros 
 typically do for Rust and Cargo dependencies, or Java and 
 Maven? Wouldn't that be a similar paradigm to D and DUB 
 dependencies?
Maven dependencies are also separated out into smaller packages and maven (unlike dub) plays relatively well with Debian. It also has no ABI issues and interface stability can be tracked. Rust only has one compiler which strongly optimizes, so we don't have the problem of choosing the right one. Cargo is/was an issue but it's being worked on and seems to work well now: https://wiki.debian.org/Teams/RustPackaging/Cargo (not that I am not involved in Haskell, Rust or Java packaging). Just because it makes sense: We do have an upstream guide showing some best practices for upstream projects which help us to maintain software for long periods. It also contains language-specific advice and some general info which is valid for all languages: https://wiki.debian.org/UpstreamGuide
Apr 10 2017
parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 17:27:28 UTC, Matthias Klumpp wrote:
 That's why I have been writing a lot of Makefiles and Meson 
 build definitions lately.
It seems like doing so without having a closer look at the realities of D software (no stable ABI, etc.) might not have been the best use of your time. How would Dub be such fundamentally unfixable that making the official language tools play nice with distro packaging (and the other way round) wouldn't be preferable over manually re-writing build systems all over the place?
 But with an unstable ABI the standard library is also affected, 
 which would trigger us to do Haskell-style versioning (which 
 mangles dependencies to depend on a virtual package containing 
 a hash of the GHC version), and that not only sucks but also 
 requires quite a lot of manpower.
So it sounds like there is a solution already for other languages. Could you elaborate some more on the problems with it? I suppose there is some wiki page documenting the process somewhere?
 Rust only has one compiler which strongly optimizes, so we 
 don't have the problem of choosing the right one. Cargo is/was 
 an issue but it's being worked on and seems to work well now: 
 https://wiki.debian.org/Teams/RustPackaging/Cargo
Rust doesn't have a stable ABI either, and it doesn't look like there is any movement in that direction (not that I think that there should be). That the people driving the effort might not be aware of it yet doesn't mean it isn't an issue for them. IIRC OCaml is also very much a statically linked affair. And how does Debian distribute Go binaries? Is there any issue with those being linked statically? If not, let's just distribute D libraries as source and compile/link them statically when building binaries, and problem solved. Some of the compiler developers, myself included, understand the issues involving ABI stability and distro packaging quite well (although the latter admittedly only on a general level). In fact, one of my earliest open source memories is of some work in the trenches ensuring ABI stability of some bits of KDE across releases. Yet we are still going to tell you that the D ABI is going to remain unstable for the foreseeable future. This is not something that just requires a man-week or month to "fix" in the compiler, but would impact many other areas as well, for example language evolution. If you somehow got the impression that this is just due to D developers "not getting it", just have a look at the other recent compiled languages. Go and Rust don't fare any differently, and even Swift, with all its development manpower, doesn't have a stable ABI yet [1]. And I believe header-only C++ template libraries have been mentioned already as well. I can only speak for myself, of course, but I certainly see the strategic importance of integration into the Linux distribution ecosystem for D, and I'm very happy to work with packagers wherever possible. However, you also need to acknowledge the properties of the ecosystem *you* are working with. If you see a big stretch of difficult terrain in front of you, closing your eyes won't make it go away; you'll only lose time you could spend working around it. ABI instability is something you'll have to work around one way or the other. — David [1] Actually, the Swift guys tried to pin it down leading up to version 4, but in the end decided it was too much work to get done in time and possibly not a wise idea at this point altogether.
Apr 10 2017
next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Monday, April 10, 2017 23:08:17 David Nadlinger via Digitalmars-d wrote:
 IIRC OCaml is also very much a statically linked affair. And how
 does Debian distribute Go binaries? Is there any issue with those
 being linked statically? If not, let's just distribute D
 libraries as source and compile/link them statically when
 building binaries, and problem solved.
Honestly, I don't see how it really makes much sense to use shared libraries with D except in cases where you have no choice. The lack of ABI compatibility makes them almost useless. Also, what are we even looking to distribute in debian? I would have thought that the normal thing to do would be to build with dub, in which case, having the compiler and dub be debian packages makes sense but not really anything else. If you're looking to package an application that was written in D, then that becomes another question, but then if you just statically link it, the ABI compatibility problem goes away as does any need to package any D library dependencies. - Jonathan M Davis
Apr 10 2017
next sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 00:47:34 UTC, Jonathan M Davis wrote:
 On Monday, April 10, 2017 23:08:17 David Nadlinger via [...]
 Also, what are we even looking to distribute in debian? I would 
 have thought that the normal thing to do would be to build with 
 dub, in which case, having the compiler and dub be debian 
 packages makes sense but not really anything else. If you're 
 looking to package an application that was written in D, then 
 that becomes another question, but then if you just statically 
 link it, the ABI compatibility problem goes away as does any 
 need to package any D library dependencies.
You will have static-library packages which have the exact same ABI issues shared libraries have. And yeah, this is obviously about stuff being built with D compilers in the distro, such as Tilix, BioD, AppStream Generator and all future things which might emerge and be useful to have in the OS.
Apr 11 2017
parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, April 11, 2017 12:40:34 Matthias Klumpp via Digitalmars-d wrote:
 On Tuesday, 11 April 2017 at 00:47:34 UTC, Jonathan M Davis wrote:
 On Monday, April 10, 2017 23:08:17 David Nadlinger via [...]
 Also, what are we even looking to distribute in debian? I would
 have thought that the normal thing to do would be to build with
 dub, in which case, having the compiler and dub be debian
 packages makes sense but not really anything else. If you're
 looking to package an application that was written in D, then
 that becomes another question, but then if you just statically
 link it, the ABI compatibility problem goes away as does any
 need to package any D library dependencies.
You will have static-library packages which have the exact same ABI issues shared libraries have. And yeah, this is obviously about stuff being built with D compilers in the distro, such as Tilix, BioD, AppStream Generator and all future things which might emerge and be useful to have in the OS.
My point was that there really isn't a reason to package D libraries in the distro. ABI compatibility makes including D libraries in a distro problematic, and dub takes care dependencies for you, making it completely unnecessary to include any D libraries in a distro. Now, there may be a reason to include a D _program_ in a distro, but with a D program, you can just build it with dub and statically link in its D libraries so that they don't need to be part of the distro. And if we go that route, you pretty much avoid this whole mess. - Jonathan M Davis
Apr 11 2017
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-04-11 02:47, Jonathan M Davis via Digitalmars-d wrote:

 Honestly, I don't see how it really makes much sense to use shared libraries
 with D except in cases where you have no choice. The lack of ABI
 compatibility makes them almost useless.

 Also, what are we even looking to distribute in debian? I would have thought
 that the normal thing to do would be to build with dub, in which case,
 having the compiler and dub be debian packages makes sense but not really
 anything else. If you're looking to package an application that was written
 in D, then that becomes another question, but then if you just statically
 link it, the ABI compatibility problem goes away as does any need to package
 any D library dependencies.
I agree, I don't see any point in distributing libraries. just applications. But I do know some people will refuse to install anything that doesn't come through the system package manager. -- /Jacob Carlborg
Apr 11 2017
parent Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 13:43:38 UTC, Jacob Carlborg wrote:
 On 2017-04-11 02:47, Jonathan M Davis via Digitalmars-d wrote:

 Honestly, I don't see how it really makes much sense to use 
 shared libraries
 with D except in cases where you have no choice. The lack of 
 ABI
 compatibility makes them almost useless.

 Also, what are we even looking to distribute in debian? I 
 would have thought
 that the normal thing to do would be to build with dub, in 
 which case,
 having the compiler and dub be debian packages makes sense but 
 not really
 anything else. If you're looking to package an application 
 that was written
 in D, then that becomes another question, but then if you just 
 statically
 link it, the ABI compatibility problem goes away as does any 
 need to package
 any D library dependencies.
I agree, I don't see any point in distributing libraries. just applications. But I do know some people will refuse to install anything that doesn't come through the system package manager.
Every single bit of software that is available in the distribution needs to be packaged in it so you can replicate its build using only what is available in the distro. Fetching things from the internet is not allowed. (That's actually a hard no in the policy, while other things like the use of static linking are rather "you shouldn't do it if you can avoid it") The fundamental thing a distribution does is integrating software and creating a consistent whole out of many moving parts. In order to do that, you absolutely can not rely on site-specific package managers like pip, dub, npm, etc. as they are not built for that purpose and only see their own ecosystem. You could still build with them though in case they integrate well with the distro.
Apr 11 2017
prev sibling parent Enrico Weigelt, metux IT consult <enrico.weigelt gr13.net> writes:
On Tuesday, 11 April 2017 at 00:47:34 UTC, Jonathan M Davis wrote:

Hi folks,

 Also, what are we even looking to distribute in debian? I would 
 have thought that the normal thing to do would be to build with 
 dub, in which case, having the compiler and dub be debian 
 packages makes sense but not really anything else.
Are you only targeting manually operated inhouse applications where operator = developer ? I haven't done any serious D programming yet - the language itself looks quite interesting to me, eg. for embedded systems. Unfortunately, the current situation makes it pretty unusable to me.
 but then if you just statically link it, the ABI compatibility 
 problem goes away as does any need to package any D library 
 dependencies.
Technically correct, but then the maintainers of individual packages have to keep an eye on each single package in the dependency chain, eg. on security issues and rebuild in those cases - basically defeating one of the major goals of the concept of distros. --mtx
May 19 2017
prev sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 23:08:17 UTC, David Nadlinger wrote:
 On Monday, 10 April 2017 at 17:27:28 UTC, Matthias Klumpp wrote:
 That's why I have been writing a lot of Makefiles and Meson 
 build definitions lately.
It seems like doing so without having a closer look at the realities of D software (no stable ABI, etc.) might not have been the best use of your time. How would Dub be such fundamentally unfixable that making the official language tools play nice with distro packaging (and the other way round) wouldn't be preferable over manually re-writing build systems all over the place?
Dub is not unfixable, but there are many more issues with it which make it very very hard to use in a distribution context - I tried that and it didn't work well. I am treating D in Debian like I would treat C++, which is quite fair given the similar featuresets and challenges.
 But with an unstable ABI the standard library is also 
 affected, which would trigger us to do Haskell-style 
 versioning (which mangles dependencies to depend on a virtual 
 package containing a hash of the GHC version), and that not 
 only sucks but also requires quite a lot of manpower.
So it sounds like there is a solution already for other languages. Could you elaborate some more on the problems with it? I suppose there is some wiki page documenting the process somewhere?
Haskell and OCaml permanently rebuild the whole stack on every new compiler release, which is why they have permanent transition trackers[1], so they basically continuously rebuild. I want to avoid this at all cost for D, as this is very very maintenance intensive and painful, and will require much more people to work on it than D has available in Debian. [1]: https://release.debian.org/transitions/
 Rust only has one compiler which strongly optimizes, so we 
 don't have the problem of choosing the right one. Cargo is/was 
 an issue but it's being worked on and seems to work well now: 
 https://wiki.debian.org/Teams/RustPackaging/Cargo
Rust doesn't have a stable ABI either, and it doesn't look like there is any movement in that direction (not that I think that there should be). That the people driving the effort might not be aware of it yet doesn't mean it isn't an issue for them.
It's being worked on[2], but it's not a super-high priority. There doesn't seem to be a definitive answer on how Rust is handled in Debian yet (but to know for certain, I would need to ask the Rust team). [2]: https://github.com/rust-lang/rfcs/issues/600
 IIRC OCaml is also very much a statically linked affair. And 
 how does Debian distribute Go binaries? Is there any issue with 
 those being linked statically? If not, let's just distribute D 
 libraries as source and compile/link them statically when 
 building binaries, and problem solved.
Surprisingly it looks like many Go packages are indeed provided as source-installations. Doing something like this with D would require Makefiles and Dub to pick up sources from system locations properly which isn't really done yet...
 Some of the compiler developers, myself included, understand 
 the issues involving ABI stability and distro packaging quite 
 well (although the latter admittedly only on a general level). 
 In fact, one of my earliest open source memories is of some 
 work in the trenches ensuring ABI stability of some bits of KDE 
 across releases. Yet we are still going to tell you that the D 
 ABI is going to remain unstable for the foreseeable future. 
 This is not something that just requires a man-week or month to 
 "fix" in the compiler, but would impact many other areas as 
 well, for example language evolution.

 If you somehow got the impression that this is just due to D 
 developers "not getting it", just have a look at the other 
 recent compiled languages. Go and Rust don't fare any 
 differently, and even Swift, with all its development manpower, 
 doesn't have a stable ABI yet [1]. And I believe header-only 
 C++ template libraries have been mentioned already as well.

 I can only speak for myself, of course, but I certainly see the 
 strategic importance of integration into the Linux distribution 
 ecosystem for D, and I'm very happy to work with packagers 
 wherever possible. However, you also need to acknowledge the 
 properties of the ecosystem *you* are working with. If you see 
 a big stretch of difficult terrain in front of you, closing 
 your eyes won't make it go away; you'll only lose time you 
 could spend working around it. ABI instability is something 
 you'll have to work around one way or the other.
That's the whole point of this thread, I want to find the best solution to deal with this issue and also one the D community can live with. I am not set on any particular solution yet, at the moment I see the problem and I am thinking about how to deal with it in the best possible way (ideally not doing what OCaml/Haskell are doing, although pulling a "soft Haskell" where we rebuild everything when the D ABI changes would be way easier).
Apr 11 2017
next sibling parent reply Gerald <gerald.b.nunn gmail.com> writes:
Looking into Rust and Cargo Debian packaging, it looks like they 
are looking to go with source packages to work around the ABI 
issue:

"Because Rust doesn't have a stable shared-library ABI, and we 
don't want to rebuild every Rust library package for every 
architecture every time we upload a new version of rustc, we 
package library crates as source code, installed into 
/usr/share/cargo/registry/cratename-version/. Thus, the Debian 
"binary" package for a library crate contains the same source as 
the source package."

https://internals.rust-lang.org/t/debian-rust-packaging-policy-draft/4453
Apr 11 2017
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 12:38 +0000, Gerald via Digitalmars-d wrote:
 Looking into Rust and Cargo Debian packaging, it looks like they=C2=A0
 are looking to go with source packages to work around the ABI=C2=A0
 issue:
=20
 "Because Rust doesn't have a stable shared-library ABI, and we=C2=A0
 don't want to rebuild every Rust library package for every=C2=A0
 architecture every time we upload a new version of rustc, we=C2=A0
 package library crates as source code, installed into=C2=A0
 /usr/share/cargo/registry/cratename-version/. Thus, the Debian=C2=A0
 "binary" package for a library crate contains the same source as=C2=A0
 the source package."
=20
 https://internals.rust-lang.org/t/debian-rust-packaging-policy-draft/
 4453
So why bother, no Rust people will use it, they just use Cargo and it gets stuff for them. Why bother packaging anything when people will not use it? All you need to have a working Rust system on Debian is curl and bash. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 11 2017
parent Gerald <gerald.b.nunn gmail.com> writes:
On Tuesday, 11 April 2017 at 12:47:13 UTC, Russel Winder wrote:
 So why bother, no Rust people will use it, they just use Cargo 
 and it gets stuff for them. Why bother packaging anything when 
 people will not use it? All you need to have a working Rust 
 system on Debian is curl and bash.
Users of an application written in Rust are not developers, they just want to install an app using whatever packaging mechanism the distro provides. Also, I would expect that the source library packages for Rust are likely just a build dependency and users are not building Rust binaries on their system. This is just a way for the Debian packagers to build packages and track dependencies using the tools and processes they already have in place. Having said that, I don't disagree with you as I think it's a bit silly to replicate what cargo, maven and dub already provide OOTB and essentially duplicate their existing repositories. However, if Rust is going in this direction it can make sense for D to follow and just make the necessary improvements in Dub to support the process.
Apr 11 2017
prev sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, April 11, 2017 12:25:09 Matthias Klumpp via Digitalmars-d wrote:
 Haskell and OCaml permanently rebuild the whole stack on every
 new compiler release, which is why they have permanent transition
 trackers[1], so they basically continuously rebuild. I want to
 avoid this at all cost for D, as this is very very maintenance
 intensive and painful, and will require much more people to work
 on it than D has available in Debian.
That's basically what's required with D. It is not ABI compatible across releases, and while ABI compatibility might be nice, it really isn't reasonable with D - especially with how attributes work and how template heavy D code is. If the D compiler gets upgraded, then anything that gets built needs everything that it depends on to have been built with that exact compiler version. But if we just use dub - which _is_ the official packaging and build tool - then we avoid these issues. Ideally, the compiler and dub would be part of the distro, but libraries don't need to be. And it sounds like that's basically how the Go and Rust folks want to function as well. So, it would make sense for these languages to simply not have their libraries be included in distros. The build tools are plenty. - Jonathan M Davis
Apr 11 2017
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-04-11 14:56, Jonathan M Davis via Digitalmars-d wrote:

 That's basically what's required with D. It is not ABI compatible across
 releases, and while ABI compatibility might be nice, it really isn't
 reasonable with D - especially with how attributes work and how template
 heavy D code is. If the D compiler gets upgraded, then anything that gets
 built needs everything that it depends on to have been built with that exact
 compiler version.

 But if we just use dub - which _is_ the official packaging and build tool -
 then we avoid these issues. Ideally, the compiler and dub would be part of
 the distro, but libraries don't need to be. And it sounds like that's
 basically how the Go and Rust folks want to function as well. So, it would
 make sense for these languages to simply not have their libraries be
 included in distros. The build tools are plenty.
I agree, but I think that they want to avoid having packages that require dependencies outside of the Debian tree to build. That is, a user end application written in D, that is built using Dub. To solve that and still use Dub they would need to package the dependent libraries as source packages and somehow point Dub to the library in the Debian tree and not code.dlang.org. I don't see why that wouldn't be possible and if anything needs to be changed in Dub for that, I think that's reasonable. -- /Jacob Carlborg
Apr 11 2017
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 11/04/2017 3:01 PM, Jacob Carlborg wrote:
 On 2017-04-11 14:56, Jonathan M Davis via Digitalmars-d wrote:

 That's basically what's required with D. It is not ABI compatible across
 releases, and while ABI compatibility might be nice, it really isn't
 reasonable with D - especially with how attributes work and how template
 heavy D code is. If the D compiler gets upgraded, then anything that gets
 built needs everything that it depends on to have been built with that
 exact
 compiler version.

 But if we just use dub - which _is_ the official packaging and build
 tool -
 then we avoid these issues. Ideally, the compiler and dub would be
 part of
 the distro, but libraries don't need to be. And it sounds like that's
 basically how the Go and Rust folks want to function as well. So, it
 would
 make sense for these languages to simply not have their libraries be
 included in distros. The build tools are plenty.
I agree, but I think that they want to avoid having packages that require dependencies outside of the Debian tree to build. That is, a user end application written in D, that is built using Dub. To solve that and still use Dub they would need to package the dependent libraries as source packages and somehow point Dub to the library in the Debian tree and not code.dlang.org. I don't see why that wouldn't be possible and if anything needs to be changed in Dub for that, I think that's reasonable.
/usr/share/source/D/package-name-version Add a search path like that to Dub and create source only library packages and that is pretty much all the distribution we need for libraries I reckon. Destroy!
Apr 11 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 14:04:44 UTC, rikki cattermole wrote:
 [...]
 /usr/share/source/D/package-name-version

 Add a search path like that to Dub and create source only 
 library packages and that is pretty much all the distribution 
 we need for libraries I reckon.
It's more likely that the path would be `/usr/include/d` because that's the place the D sources seem to usually reside, especially if they are also used as "headers" for shared libraries. At time I am playing around with the idea of using pkg-config[1] files to enlist the sources a D library consists of. By doing that, we would have a very build-system agnostic way of doing storing that information that can be used by Automake (native), Dub, Meson and even plain Makefiles. It's also very widely used (although it is commonly not used to list sources). This would also jive really well with the existing D packages in Debian and we could - if we go down the source-only route - still decide to precompile a few selected libraries, for example in case they are very time consuming to compile or really large. This will *not* solve the issues with Phobos breakage though, as Phobos is a shared library. But doing that would pretty much work around all other problems, except for the cost of having statically linked binaries, but that seems to be inevitable anyway given the state of things. (Disclaimer: This is not a D team policy draft yet, just an opinion at the moment that might be refined or changed entirely) [1]: https://www.freedesktop.org/wiki/Software/pkg-config/
Apr 11 2017
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 11/04/2017 3:21 PM, Matthias Klumpp wrote:
 On Tuesday, 11 April 2017 at 14:04:44 UTC, rikki cattermole wrote:
 [...]
 /usr/share/source/D/package-name-version

 Add a search path like that to Dub and create source only library
 packages and that is pretty much all the distribution we need for
 libraries I reckon.
It's more likely that the path would be `/usr/include/d` because that's the place the D sources seem to usually reside, especially if they are also used as "headers" for shared libraries. At time I am playing around with the idea of using pkg-config[1] files to enlist the sources a D library consists of. By doing that, we would have a very build-system agnostic way of doing storing that information that can be used by Automake (native), Dub, Meson and even plain Makefiles. It's also very widely used (although it is commonly not used to list sources). This would also jive really well with the existing D packages in Debian and we could - if we go down the source-only route - still decide to precompile a few selected libraries, for example in case they are very time consuming to compile or really large. This will *not* solve the issues with Phobos breakage though, as Phobos is a shared library. But doing that would pretty much work around all other problems, except for the cost of having statically linked binaries, but that seems to be inevitable anyway given the state of things. (Disclaimer: This is not a D team policy draft yet, just an opinion at the moment that might be refined or changed entirely) [1]: https://www.freedesktop.org/wiki/Software/pkg-config/
The problem with /usr/include/d is that is where .di files would be located not .d. This would also match up with the c/c++ usage of it.
Apr 11 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:
 [...]
 The problem with /usr/include/d is that is where .di files 
 would be located not .d. This would also match up with the 
 c/c++ usage of it.
When I asked about this a while back, I was told to just install the sources into the include D as "almost nobody uses .di files except for proprietary libraries" (and do those even exist?). But in any case, any path would be fine with me as long as people can settle on using it - `/usr/share/d` would be available ^^
Apr 11 2017
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 11/04/2017 3:33 PM, Matthias Klumpp wrote:
 On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:
 [...]
 The problem with /usr/include/d is that is where .di files would be
 located not .d. This would also match up with the c/c++ usage of it.
When I asked about this a while back, I was told to just install the sources into the include D as "almost nobody uses .di files except for proprietary libraries" (and do those even exist?). But in any case, any path would be fine with me as long as people can settle on using it - `/usr/share/d` would be available ^^
Satoshi is developing one and is having a load of trouble in doing so. That says everything, we got a lot of work to do.
Apr 11 2017
prev sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, April 11, 2017 14:33:01 Matthias Klumpp via Digitalmars-d wrote:
 On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:
 [...]
 The problem with /usr/include/d is that is where .di files
 would be located not .d. This would also match up with the
 c/c++ usage of it.
When I asked about this a while back, I was told to just install the sources into the include D as "almost nobody uses .di files except for proprietary libraries" (and do those even exist?). But in any case, any path would be fine with me as long as people can settle on using it - `/usr/share/d` would be available ^^
Putting .di files in the include directory makes sense when compared to C++, but it's definitely true that they're used quite rarely. They gain you very little and cost you a lot (e.g. no CTFE). But unless someone were looking to put both .di and .d files for the same module on the same system, it wouldn't really be an issue to put them both in the include directory. I would expect open source librares to use .di files very rarely though. - Jonathan M Davis
Apr 11 2017
parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 11 Apr 2017 07:44:45 -0700
schrieb Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com>:

 On Tuesday, April 11, 2017 14:33:01 Matthias Klumpp via Digitalmars-d
 wrote:
 On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:  
 [...]
 The problem with /usr/include/d is that is where .di files
 would be located not .d. This would also match up with the
 c/c++ usage of it.  
When I asked about this a while back, I was told to just install the sources into the include D as "almost nobody uses .di files except for proprietary libraries" (and do those even exist?). But in any case, any path would be fine with me as long as people can settle on using it - `/usr/share/d` would be available ^^
Putting .di files in the include directory makes sense when compared to C++, but it's definitely true that they're used quite rarely. They gain you very little and cost you a lot (e.g. no CTFE). But unless someone were looking to put both .di and .d files for the same module on the same system, it wouldn't really be an issue to put them both in the include directory. I would expect open source librares to use .di files very rarely though. - Jonathan M Davis
I'd think of .d files as a superset of .di files. There's no reason to install both (assuming same library version) and having the .d files will produce better cross-module inlining so these are preferred. Of course for proprietary libraries .di files are required. But nevertheless .di and .d can be mixed in /usr/include/d. The only downside is that inlining from /usr/include could cause licensing problems (e.g. LGPL allows only linking, not sure how inlining would be affected. But it seems they have an Exception for inlining) -- Johannes
Apr 11 2017
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 11 April 2017 at 21:08:05 UTC, Johannes Pfau wrote:
 I'd think of .d files as a superset of .di files.
Note that there is zero difference between them. The compiler treats both *exactly* the same way (it just considers the extensions to be aliases of each other). It is just a naming convention.
Apr 11 2017
prev sibling next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, April 11, 2017 14:21:57 Matthias Klumpp via Digitalmars-d wrote:
 This will *not* solve the issues with Phobos breakage though, as
 Phobos is a shared library.
It could always just be distributed as a static library. There arguably isn't much point in distributing it as a shared library anyway - particularly when it's not ABI compatible across versions. I actively avoid having phobos as a shared library on my systems, because it just causes versioning problems when programs are built against it. All of those problems go away with a static library. And so much of Phobos is templated anyway that there isn't even much to share. - Jonathan M Davis
Apr 11 2017
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 11 Apr 2017 07:40:12 -0700
schrieb Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com>:

 It could always just be distributed as a static library. There arguably
 isn't much point in distributing it as a shared library anyway -
 particularly when it's not ABI compatible across versions. I actively avoid
 having phobos as a shared library on my systems, because it just causes
 versioning problems when programs are built against it. All of those
 problems go away with a static library. And so much of Phobos is templated
 anyway that there isn't even much to share.
 
 - Jonathan M Davis
I see what you're doing there, but your last point is wishful thinking. Dynamically linked binaries can share megabytes of code. Even Phobos - although heavily templated - has proven to be very amenable to sharing. For example, a "Hello world!" program using `writeln()` has these sizes when compiled with `dmd -O -release -inline`: static Phobos2 : 806968 bytes dynamic Phobos2 : 18552 bytes That's about 770 KiB to share or 97.7% of its total size! Awesome! Another package we have on Gentoo now is the tiling terminal emulator Tilix, compiled with its default `dmd -O`: static Phobos2+GtkD : 14828272 bytes dynamic Phobos2+GtkD : 6126464 bytes So here we get ~8.3 megabytes in binary size reduction due to sharing. (Though 6 MiB for a terminal is still a lot - are we slim yet? :P ) Some forms of proprietary code and GPL code need clean shared library interfaces, free from templates and attribute inference. I mentioned this before: As far as feasible have full attribute inference for non-exported functions and reliance on only explicit attributes for exported functions, so that code changes can't break the public API (due to mangling changes). When Dlang's ABI changes, release a major version and make a public announcement so that everyone can schedule the system update to a convenient time as is done with GCC C++ ABI changes. -- Marco
May 02 2017
next sibling parent David Nadlinger <code klickverbot.at> writes:
On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:
      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!
By the way, using LDC: 402736 bytes for the static build (Linux x86_64). ;) — David
May 02 2017
prev sibling parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:
 I see what you're doing there, but your last point is wishful 
 thinking. Dynamically linked binaries can share megabytes of 
 code. Even Phobos - although heavily templated - has proven to 
 be very amenable to sharing. For example, a "Hello world!" 
 program using `writeln()` has these sizes when compiled with 
 `dmd -O -release -inline`:

      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!
Is all of that active code, or is some of that (statically knowable) never getting executed (as in could've been removed at compile/link time)?
May 02 2017
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 02 May 2017 20:53:50 +0000
schrieb Moritz Maxeiner <moritz ucworks.org>:

 On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:
 I see what you're doing there, but your last point is wishful 
 thinking. Dynamically linked binaries can share megabytes of 
 code. Even Phobos - although heavily templated - has proven to 
 be very amenable to sharing. For example, a "Hello world!" 
 program using `writeln()` has these sizes when compiled with 
 `dmd -O -release -inline`:

      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!  
Is all of that active code, or is some of that (statically knowable) never getting executed (as in could've been removed at compile/link time)?
I guess David gave you the answer. So it's just 95.4% of its total size. :p By the way, is the fully dynamic linking version possible with ldc2 now as well ? -- Marco
May 02 2017
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 2 May 2017 at 23:27:28 UTC, Marco Leise wrote:
 Am Tue, 02 May 2017 20:53:50 +0000
 schrieb Moritz Maxeiner <moritz ucworks.org>:

 On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:
 I see what you're doing there, but your last point is 
 wishful thinking. Dynamically linked binaries can share 
 megabytes of code. Even Phobos - although heavily templated 
 - has proven to be very amenable to sharing. For example, a 
 "Hello world!" program using `writeln()` has these sizes 
 when compiled with `dmd -O -release -inline`:

      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!
Is all of that active code, or is some of that (statically knowable) never getting executed (as in could've been removed at compile/link time)?
I guess David gave you the answer. So it's just 95.4% of its total size. :p
Under the assumption that ldc2 produces no dead code in the output; is that a reasonable assumption (I'm not sure)?
 By the way, is the fully dynamic linking version possible with
 ldc2 now as well ?
I did have a modified ebuild to try that out a while back and it seemed to work fine in my limited testing scope. Since I quite often change installed d compilers and don't want my programs (like tilix) to stop working (or have old d compiler versions being kept installed *just* because some programs were built with them), I generally link against phobos statically, anyway, so I my tests weren't exhaustive. The cmake flag to to use in the ebuild would be BUILD_SHARED [1]. [1] https://github.com/ldc-developers/ldc/blob/v1.2.0/CMakeLists.txt#L522
May 02 2017
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 03 May 2017 01:02:38 +0000
schrieb Moritz Maxeiner <moritz ucworks.org>:

 On Tuesday, 2 May 2017 at 23:27:28 UTC, Marco Leise wrote:
 Am Tue, 02 May 2017 20:53:50 +0000
 schrieb Moritz Maxeiner <moritz ucworks.org>:
  
 On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:  
 I see what you're doing there, but your last point is 
 wishful thinking. Dynamically linked binaries can share 
 megabytes of code. Even Phobos - although heavily templated 
 - has proven to be very amenable to sharing. For example, a 
 "Hello world!" program using `writeln()` has these sizes 
 when compiled with `dmd -O -release -inline`:

      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!  
Is all of that active code, or is some of that (statically knowable) never getting executed (as in could've been removed at compile/link time)?
I guess David gave you the answer. So it's just 95.4% of its total size. :p
Under the assumption that ldc2 produces no dead code in the output; is that a reasonable assumption (I'm not sure)?
 By the way, is the fully dynamic linking version possible with
 ldc2 now as well ?  
I did have a modified ebuild to try that out a while back and it seemed to work fine in my limited testing scope. Since I quite often change installed d compilers and don't want my programs (like tilix) to stop working (or have old d compiler versions being kept installed *just* because some programs were built with them), I generally link against phobos statically, anyway, so I my tests weren't exhaustive. The cmake flag to to use in the ebuild would be BUILD_SHARED [1]. [1] https://github.com/ldc-developers/ldc/blob/v1.2.0/CMakeLists.txt#L522
I remember seeing this line and thinking "aww .. no way to build both versions?". I.e. enable the "static-libs" USE flag to _also_ build static libraries accompanying the shared ones. I'll create an issue asking about that. -- Marco
May 03 2017
prev sibling next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 14:21 +0000, Matthias Klumpp via Digitalmars-d
wrote:
=20
[=E2=80=A6]
 At time I am playing around with the idea of using pkg-config[1]=C2=A0
 files to enlist the sources a D library consists of.
 By doing that, we would have a very build-system agnostic way of=C2=A0
 doing storing that information that can be used by Automake=C2=A0
 (native), Dub, Meson and even plain Makefiles. It's also very=C2=A0
 widely used (although it is commonly not used to list sources).
And SCons. What about CMake? (Which is the CLion build system using Make underneath, but they are rapidly moving to supporting Ninja.) BTW, I'd like to reiterate that the D support in Meson is great, thanks for getting that in place. Meson and CMake are the only systems that can install stuff sensibly.=20
 [=E2=80=A6]
--=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 11 2017
parent Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 14:43:15 UTC, Russel Winder wrote:
 On Tue, 2017-04-11 at 14:21 +0000, Matthias Klumpp via 
 Digitalmars-d wrote:
 
[…]
 At time I am playing around with the idea of using 
 pkg-config[1]
 files to enlist the sources a D library consists of.
 By doing that, we would have a very build-system agnostic way 
 of
 doing storing that information that can be used by Automake
 (native), Dub, Meson and even plain Makefiles. It's also very
 widely used (although it is commonly not used to list sources).
And SCons. What about CMake? (Which is the CLion build system using Make underneath, but they are rapidly moving to supporting Ninja.)
CMake supports this as well: https://cmake.org/cmake/help/v3.0/module/FindPkgConfig.html The current idea is in case a library "foobar" would be packaged, to have a "foobar-src.pc" pkgconfig file (additionally to a potentially also existing "foobar.pc" file), containing something like this: ``` prefix=/usr/local exec_prefix=${prefix} includedir=${prefix}/include/d/foo Name: foobar Description: The foo library (sources) Version: 1.0.0 Cflags: -I${includedir} -d-version=blahblub Sources: ${includedir}/alpha.d ${includedir}/beta.d ${includedir}/gamma.d ``` Build systems would then need to explicitly fetch the Sources field and add its expanded values to the project's sources. (Using Cflags for this would be messy, as the build system might want to deal with flags and sources separately). Alternatively dub could also define a layout and we write plugins for each build-system to make it work. This will be really annoying with large libraries like GtkD though, which will require substantially longer to build. Maybe it's worth keeping some libraries precompiled (thereby locking all their reverse-dependencies to whatever D compiler was used to compile the library). One problem with the pkg-config approach is that to support the precompiled case too, build systems would need to search for both "foobar" and "foobar-src" and only fail dependency detection if both of them are missing. At least with CMake/Meson that's easy to do though (although it's a bit messy).
Apr 11 2017
prev sibling next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 07:40 -0700, Jonathan M Davis via Digitalmars-d
wrote:
 On Tuesday, April 11, 2017 14:21:57 Matthias Klumpp via Digitalmars-d=20
 wrote:
 This will *not* solve the issues with Phobos breakage though, as
 Phobos is a shared library.
=20 It could always just be distributed as a static library. There arguably isn't much point in distributing it as a shared library anyway - particularly when it's not ABI compatible across versions. I actively avoid having phobos as a shared library on my systems, because it just causes versioning problems when programs are built against it. All of those problems go away with a static library. And so much of Phobos is templated anyway that there isn't even much to share. =20 - Jonathan M Davis
Having played a bit with GtkD, you always want this as a shared library for development.=20 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 11 2017
next sibling parent Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 14:49:03 UTC, Russel Winder wrote:
 [...]
 Having played a bit with GtkD, you always want this as a shared 
 library for development.
Yeah, GtkD is pretty massive and takes quite a large amount of time to compile... Redoing that for each software depending on it is pretty wasteful.
Apr 11 2017
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Tuesday, 11 April 2017 at 14:49:03 UTC, Russel Winder wrote:
 Having played a bit with GtkD, you always want this as a shared 
 library for development.
Why would a shared library be preferable to a static library for that (which might still be re-used between different projects, etc.)? — David
Apr 11 2017
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 17:02 +0000, David Nadlinger via Digitalmars-d
wrote:
 On Tuesday, 11 April 2017 at 14:49:03 UTC, Russel Winder wrote:
 Having played a bit with GtkD, you always want this as a shared=C2=A0
 library for development.
=20 Why would a shared library be preferable to a static library for=C2=A0 that (which might still be re-used between different projects,=C2=A0 etc.)? =20
I have only the data that compiling and linking a GtkD application against a shared library is a lot shorter than against a static library. I'll get proper data, but tomorrow =E2=80=93 that upstairs have a water leak and our kitchen ceiling is about to fall in (again, third time) means I am not competent just now. Hassle me Thursday if I have presented data. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 11 2017
parent David Nadlinger <code klickverbot.at> writes:
On Tuesday, 11 April 2017 at 18:13:11 UTC, Russel Winder wrote:
 I have only the data that compiling and linking a GtkD 
 application against a shared library is a lot shorter than 
 against a static library.
Sure, but that might be easily fixed, and if you really want to use shared libraries, you can always make Dub create/use a library in a local directory (and set the rpath appropriately, …). I don't quite see how any of this is related to the topic at hand (distro packaging). I'd indeed be interested in some data on the linking times, though. Be sure to test with ld.gold as well, if you are not using it already. — David
Apr 11 2017
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 11 Apr 2017 14:21:57 +0000
schrieb Matthias Klumpp <mak debian.org>:

 can be used by Automake 
 (native),
Do you maintain D support for automake? I wrote some basic D support for autoconf and libtool (https://github.com/D-Programming-GDC/GDC/tree/master/libphobos/m4) but no automake support except for some hacks in https://github.com/D-Programming-GDC/GDC/blob/master/libphobos/d_rules.am I guess I should upstream this some time. -- Johannes
Apr 11 2017
parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 23:02, Johannes Pfau via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Am Tue, 11 Apr 2017 14:21:57 +0000
 schrieb Matthias Klumpp <mak debian.org>:

 can be used by Automake
 (native),
Do you maintain D support for automake? I wrote some basic D support for autoconf and libtool (https://github.com/D-Programming-GDC/GDC/tree/master/libphobos/m4) but no automake support except for some hacks in https://github.com/D-Programming-GDC/GDC/blob/master/libphobos/d_rules.am I guess I should upstream this some time. -- Johannes
Yeah, you should. I upstreamed D support in DejaGNU back when I added D tests to GDB. https://www.mail-archive.com/dejagnu gnu.org/msg00723.html
May 01 2017
prev sibling next sibling parent reply qznc <qznc web.de> writes:
On Tuesday, 11 April 2017 at 12:56:59 UTC, Jonathan M Davis wrote:
 But if we just use dub - which _is_ the official packaging and 
 build tool - then we avoid these issues. Ideally, the compiler 
 and dub would be part of the distro, but libraries don't need 
 to be. And it sounds like that's basically how the Go and Rust 
 folks want to function as well. So, it would make sense for 
 these languages to simply not have their libraries be included 
 in distros. The build tools are plenty.
This is not compatible with Debian. Debian requires to include *everything*. You must be able to build every package without network access only from source packages. Essentially, somebody must fix this dub issue: https://github.com/dlang/dub/issues/838
Apr 11 2017
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 11 Apr 2017 15:03:36 +0000
schrieb qznc <qznc web.de>:

 On Tuesday, 11 April 2017 at 12:56:59 UTC, Jonathan M Davis wrote:
 But if we just use dub - which _is_ the official packaging and 
 build tool - then we avoid these issues. Ideally, the compiler 
 and dub would be part of the distro, but libraries don't need 
 to be. And it sounds like that's basically how the Go and Rust 
 folks want to function as well. So, it would make sense for 
 these languages to simply not have their libraries be included 
 in distros. The build tools are plenty.  
This is not compatible with Debian. Debian requires to include *everything*. You must be able to build every package without network access only from source packages. Essentially, somebody must fix this dub issue: https://github.com/dlang/dub/issues/838
I created a similar issue 3 years ago: https://github.com/dlang/dub/issues/342 A key point is how environment variables that everyone agreed upon made "make" a flexible build tool, being able to use a user specified compiler, installation directory, compiler flags and more. This flexibility is relied on by Linux packaging tools to produce binaries for a specific target. Then there is the problem that at some point you rely on non-Dlang libraries that only the system package manager can provide and update. In the past this always ended in the system package manager winning over the language tool as the common program to update and clean the system. Dicebot and bioinfornatics (Fedora) felt that dub was for developers and wasn't tailored to system administration at that time. p0nce, s-ludwig and markos (Debian) also gave input and the net result was: A Makefile generator for dub shall be written that exposes the desired environment variables and can be used by existing Linux package managers. -- Marco
May 02 2017
prev sibling parent Enrico Weigelt, metux IT consult <enrico.weigelt gr13.net> writes:
On Tuesday, 11 April 2017 at 12:56:59 UTC, Jonathan M Davis wrote:

 But if we just use dub - which _is_ the official packaging and 
 build tool - then we avoid these issues. Ideally, the compiler 
 and dub would be part of the distro, but libraries don't need 
 to be.
But that would defeat a primary goal of distros: reproducable builds, w/o depending on downloading anything from the internet. Distros are doing that for good reasons: saves operators from uncountable working hours (one of *the* major reason of Linux' success).
 And it sounds like that's basically how the Go and Rust folks 
 want to function as well.
Right. Exactly why we completely banned Go from a project just a few days ago. --mtx
May 19 2017
prev sibling next sibling parent reply qznc <qznc web.de> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
   1) Is there some perspective on D getting a defined ABI that 
 works with all major D compilers?
   2) What would the D community recommend on how to deal with 
 the ABI issues currently? A Linux distribution is a bunch of 
 tightly integrated software, and changing one piece in an 
 incompatible way (e.g. by building it with LDC instead of DMD) 
 will have consequences.
   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?
My prediction for Walters reply: 1) No. Not worth it, because templates, ctfe, etc. 3) No. Not worth it, because D frontend stuff is more important.
 Also: If you want to help out Debian's D team, feel free to 
 ping me or any other D team member (we are very short handed 
 and are only two active people right now). See 
 https://wiki.debian.org/D (caution, wiki page is very outdated, 
 last touched in 2012)
Tentative ping, but that Wiki page is not helpful. The linked svn repo is empty. Where and how do you work? I guess the issue are still the same as you wrote here (except 1. is solved): https://gist.github.com/ximion/fe6264481319dd94c8308b1ea4e8207a So, mostly dub needs work, I guess.
Apr 10 2017
parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 14:33:34 UTC, qznc wrote:
 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
   1) Is there some perspective on D getting a defined ABI that 
 works with all major D compilers?
   2) What would the D community recommend on how to deal with 
 the ABI issues currently? A Linux distribution is a bunch of 
 tightly integrated software, and changing one piece in an 
 incompatible way (e.g. by building it with LDC instead of DMD) 
 will have consequences.
   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?
My prediction for Walters reply: 1) No. Not worth it, because templates, ctfe, etc.
That's short-sighted IMHO, because if the template doesn't change, ABI/API doesn't change. Also, some projects use D as better C and don't expose this functionality. It should be up to the project to set the level of API/ABI stability, and not to the compiler to make everything unstable by default.
 [...]
 Tentative ping, but that Wiki page is not helpful. The linked 
 svn repo is empty. Where and how do you work?
Yeah, the page is really poor, it was last touched in 2012. I made a few updates to at least link to the current Git repo. We generally work on various Git repositories, but not all of them are run by the D team (e.g. libundead and libbiod as well as several games are things I am aware of that aren't D-team maintained but are part of other team's work). One can find all stuff using D by testing the reverse-build-depends on the LDC and GDC compilers.
 I guess the issue are still the same as you wrote here (except 
 1. is solved):
 https://gist.github.com/ximion/fe6264481319dd94c8308b1ea4e8207a

 So, mostly dub needs work, I guess.
Yes, but since Meson is working well and Meson scripts are easy to write, it's not a super high priority item anymore. As I said earlier, work as a distribution developer is pretty much always about reducing long-term maintenance cost, and not about less work short-term, which means we will gladly write Meson or Automake scripts to integrate software into Debian if there is a demand for it. On Monday, 10 April 2017 at 15:11:01 UTC, Jack Stouffer wrote:
 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?
This can be definitively answered as "no", https://issues.dlang.org/show_bug.cgi?id=15108
 *Walter Bright*:
 Doing an ARM back end wouldn't be that hard. It's much less 
 complex than x86. Most of the work would be deleting about half 
 of the x86 code generator :-)
:D - doesn't sound like a flat-out no, much more like there just wasn't someone doing the work yet.
Apr 10 2017
prev sibling next sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?
This can be definitively answered as "no", https://issues.dlang.org/show_bug.cgi?id=15108
Apr 10 2017
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 10 April 2017 at 15:11:01 UTC, Jack Stouffer wrote:
 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?
This can be definitively answered as "no", https://issues.dlang.org/show_bug.cgi?id=15108
When templates are done I might give it a shot :)
Apr 10 2017
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 8:11 AM, Jack Stouffer wrote:
 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
   3) Will DMD support more architectures in the near future? How should the
 architecture issue be handled?
This can be definitively answered as "no", https://issues.dlang.org/show_bug.cgi?id=15108
With it being open source now, this could change if someone wants to pick up the flag.
Apr 10 2017
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via Digitalmars-d
wrote:
[...]
 Naturally, when the reference compiler is available in Debian, we
 would compile everything with that, as it is the development focus and
 the thing many people test with.

 We do, however, have quite a bit of bioinformatics and other D
 software in the archive where performance matters - so our users and
 the developers of that software (like BioD, potentially Mir, maybe
 even Vibe.d) will want the fastest performance and will ask us to
 compile the libraries with LDC or GDC.
 
 If we do that, we will run into the D ABI trap: Libraries compiled
 with compiler X can not be used from software compiled with D compiler
 Y. There is actually no ABI stability guarantee even between DMD
 releases.  This will make integrating D a huge pain. Recompiling the
 dependency-chain of a software from source when compiling a package
 using the "right" compiler and statically adding the code is forbidden
 by distro policy.  Having static libraries in the dependencies doesn't
 solve the issue.  Compiling each library with all D compilers is
 highly impractical and not really feasible.
This is not a hard problem to solve, IMO. Just build the library into two separate binaries, each with a SONAME that encodes the ABI it is compatible with. The resulting two .so's can either be distributed as separate packages (for minimum bloat, if that's a concern), or as a single package that contains both binaries (since they have different SONAMEs this should not be a problem). Then if you compile some software X that depends on this library, it will pick up the correct version of the library depending on which compiler you compiled with.
 The other issue is architecture support. In Debian we are strongly
 encouraged to support as many architectures as possible, to the point
 of having to justify why arch X is not supported.
I'd say this makes a good case for my proposed solution above. DMD is unlikely to support other archs than amd64/ia32 in the foreseeable future, so the justification for dmd being unavailable for arch X would be that upstream DMD simply doesn't support it. This, however, should not prevent us from using gdc/ldc on those other archs, so that we can still ship packages for those archs. They will merely require ldc rather than dmd. And obviously, libraries built for that arch will only support the ldc SONAME, not the dmd one. (This may be an argument for bundling both SONAMEs in a single package -- it gets messy if we start shipping multiple libraries, some of which may not be available on all archs. By shipping a single package that includes both versions for ia32/amd64, we can simply omit the DMD-compiled version from other archs.) Unfortunately, I realize that this means some packages that require the latest DMD would not be available on all archs, if they require features unavailable in gdc/ldc. But this problem may not be a huge one, now that ldc is mostly up-to-date with dmd (at most 1 release behind, IIRC). GDC may lag behind a bit more because it is unfortunately tied to GCC releases, so we may have to forego using gdc for building newer D packages. But we should be able to ship most D packages compiled with both. Furthermore, I wonder if we should standardize on ldc for most D software instead of dmd, unless that software absolutely depends on features only available on dmd. My reasons are: - While dmd compiles very fast, it consistently (IME) produces code that runs about 20-30% slower than code produced by gdc (and I presume ldc as well). Since we're talking about building Debian packages on Debian's buildd's, which are background batch processes, compilation speed is no big deal, but the performance of the executable *is* a big deal. The last thing we want is to give a poor impression of D by shipping official Debian packages that have subpar performance. - DMD is unlikely to target other archs than ia32/amd64 in the foreseeable future, AFAIK, unless the recent relicensing triumph of dmd's backend makes this future more likely. There will definitely be resistance among DDs because of lack of support for other archs. LDC, in contrast, supports all of Debian's supported archs. - LDC is already available in Debian, meaning that we can already start adding D packages built with ldc without needing to work through the red tape involved in adding a new compiler to the archive. - The only case where I can see we'd want to compile with dmd is if the target software uses features that are only available on dmd. Since LDC's release cycle is now very close to DMD's, I don't foresee there will be too many D projects out there that can't be compiled with LDC. So there should really only be a small number of packages that we'd have to fallback to dmd, which will also be limited in supported archs -- meaning that it will probably be challenging to get it into the Debian archive in the first place. - If we standardize on using ldc to compile Debian packages, we can sidestep the ABI compatibility issue (though the SONAME hack will probably still be necessary in order to provide users with libraries that are usable with dmd). Because of this, my first inclination is to standardize on ldc rather than dmd, where building Debian packages are concerned. Of course, this in no way diminishes the priority in *distributing* dmd in the Debian archive as -dev packages. Just don't build other Debian packages with it. :-D [...]
 Also: If you want to help out Debian's D team, feel free to ping me or
 any other D team member (we are very short handed and are only two
 active people right now). See https://wiki.debian.org/D (caution, wiki
 page is very outdated, last touched in 2012)
I'd like to be involved, since I *am* technically a Debian developer (though haven't been very active for the past long while -- I do try to at least keep the meager few packages I maintain up-to-date, though). T -- "Holy war is an oxymoron." -- Lazarus Long
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 17:29:04 UTC, H. S. Teoh wrote:
 On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via 
 Digitalmars-d wrote: [...]
 [...]
 If we do that, we will run into the D ABI trap: Libraries 
 compiled with compiler X can not be used from software 
 compiled with D compiler Y. There is actually no ABI stability 
 guarantee even between DMD releases.  This will make 
 integrating D a huge pain. Recompiling the dependency-chain of 
 a software from source when compiling a package using the 
 "right" compiler and statically adding the code is forbidden 
 by distro policy.  Having static libraries in the dependencies 
 doesn't solve the issue.  Compiling each library with all D 
 compilers is highly impractical and not really feasible.
This is not a hard problem to solve, IMO. Just build the library into two separate binaries, each with a SONAME that encodes the ABI it is compatible with.
This would require to hack each and every build system to support this *and* if there is a pkgconfig file for the shared library, to change all depending software to check for multiple library names which is a bit crazy...
 The resulting two .so's can either be distributed as separate 
 packages (for minimum bloat, if that's a concern), or as a 
 single package that contains both binaries (since they have 
 different SONAMEs this should not be a problem).
Since one library == one package in Debian, it would have to be multiple packages, otherwise we would need to override Lintian errors/warnings, which is always a bad idea.
 Then if you compile some software X that depends on this 
 library, it will pick up the correct version of the library 
 depending on which compiler you compiled with.
Unfortunately not without special support in the software's build system :-/
 [...]
 DMD is unlikely to support other archs than amd64/ia32 in the 
 foreseeable future, so the justification for dmd being 
 unavailable for arch X would be that upstream DMD simply 
 doesn't support it.  This, however, should not prevent us from 
 using gdc/ldc on those other archs, so that we can still ship 
 packages for those archs.  They will merely require ldc rather 
 than dmd. And obviously, libraries built for that arch will 
 only support the ldc SONAME, not the dmd one. (This may be an 
 argument for bundling both SONAMEs in a single package -- it 
 gets messy if we start shipping multiple libraries, some of 
 which may not be available on all archs. By shipping a single 
 package that includes both versions for ia32/amd64, we can 
 simply omit the DMD-compiled version from other archs.)
Conditional build-dependencies are a bit annoying, but with a metapackage "d-compiler" or similar, using different D compilers on different architectures would definitely be possible.
 Unfortunately, I realize that this means some packages that 
 require the latest DMD would not be available on all archs, if 
 they require features unavailable in gdc/ldc.  But this problem 
 may not be a huge one, now that ldc is mostly up-to-date with 
 dmd (at most 1 release behind, IIRC). GDC may lag behind a bit 
 more because it is unfortunately tied to GCC releases, so we 
 may have to forego using gdc for building newer D packages. But 
 we should be able to ship most D packages compiled with both.
Compiling with multiple compilers is a really big effort with rather questionable gain, IMO. But as far as LDCs compatibility with other D projects goes: That is really good, the only reason you sometimes can't compile some random D code with LDC might be bugs, but not old standard libraries.
 Furthermore, I wonder if we should standardize on ldc for most 
 D software instead of dmd, unless that software absolutely 
 depends on features only available on dmd.  My reasons are:

 - While dmd compiles very fast, it consistently (IME) produces 
 code that
   runs about 20-30% slower than code produced by gdc (and I 
 presume ldc
   as well). Since we're talking about building Debian packages 
 on
   Debian's buildd's, which are background batch processes, 
 compilation
   speed is no big deal, but the performance of the executable 
 *is* a big
   deal. The last thing we want is to give a poor impression of 
 D by
   shipping official Debian packages that have subpar 
 performance.

 - DMD is unlikely to target other archs than ia32/amd64 in the
   foreseeable future, AFAIK, unless the recent relicensing 
 triumph of
   dmd's backend makes this future more likely.  There will 
 definitely be
   resistance among DDs because of lack of support for other 
 archs. LDC,
   in contrast, supports all of Debian's supported archs.

 - LDC is already available in Debian, meaning that we can 
 already start
   adding D packages built with ldc without needing to work 
 through the
   red tape involved in adding a new compiler to the archive.
I agree with all of that, I think sticking with LDC might indeed be the least painful thing to do...
 - The only case where I can see we'd want to compile with dmd 
 is if the
   target software uses features that are only available on dmd.
Oddly, it usually is the other way round :P - some projects explicitly want LDC or GDC.
 [...]
 Of course, this in no way diminishes the priority in 
 *distributing* dmd in the Debian archive as -dev packages. Just 
 don't build other Debian packages with it. :-D
Indeed. It needs license review (urgh... ^^), but when it's through that there is no reason not to have it in Debian main and Ubuntu.
 [...]
 Also: If you want to help out Debian's D team, feel free to 
 ping me or any other D team member (we are very short handed 
 and are only two active people right now). See 
 https://wiki.debian.org/D (caution, wiki page is very 
 outdated, last touched in 2012)
I'd like to be involved, since I *am* technically a Debian developer (though haven't been very active for the past long while -- I do try to at least keep the meager few packages I maintain up-to-date, though).
It's unlikely that I will maintain the dmd package (I might co-maintain it or sponsor uploads though - reason is simply lack of time and -ETOOMANYPROJECTS), so any help there is highly appreciated and I'd help with it whenever I can (I just can't make long-term commitments). If DMD is uploaded as part of the D team, spreading the maintenance cost will be rather simple too :-)
Apr 10 2017
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Mon, Apr 10, 2017 at 06:12:26PM +0000, Matthias Klumpp via Digitalmars-d
wrote:
 On Monday, 10 April 2017 at 17:29:04 UTC, H. S. Teoh wrote:
 On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via
 Digitalmars-d wrote: [...]
 [...]
 If we do that, we will run into the D ABI trap: Libraries compiled
 with compiler X can not be used from software compiled with D
 compiler Y. There is actually no ABI stability guarantee even
 between DMD releases.  This will make integrating D a huge pain.
 Recompiling the dependency-chain of a software from source when
 compiling a package using the "right" compiler and statically
 adding the code is forbidden by distro policy.  Having static
 libraries in the dependencies doesn't solve the issue.  Compiling
 each library with all D compilers is highly impractical and not
 really feasible.
This is not a hard problem to solve, IMO. Just build the library into two separate binaries, each with a SONAME that encodes the ABI it is compatible with.
This would require to hack each and every build system to support this *and* if there is a pkgconfig file for the shared library, to change all depending software to check for multiple library names which is a bit crazy...
Hmm. I guess there's no easy way to make dmd/ldc emit dependencies with modified SONAMEs? So yeah, you're right, every software that depends on said libraries would have to explicitly depend on a different SONAME depending on what they were built with. OK, crazy idea, nevermind. :-( [...]
 - The only case where I can see we'd want to compile with dmd is if
   the target software uses features that are only available on dmd.
Oddly, it usually is the other way round :P - some projects explicitly want LDC or GDC.
Well, if they explicitly want ldc/gdc, then we'll just have to do with that. But I think ldc should be able to compile anything that compiles with gdc, so we might still be able to standardize on ldc by default. [...]
 Of course, this in no way diminishes the priority in *distributing*
 dmd in the Debian archive as -dev packages. Just don't build other
 Debian packages with it. :-D
Indeed. It needs license review (urgh... ^^), but when it's through that there is no reason not to have it in Debian main and Ubuntu.
One issue, though: if we standardize on compiling Debian packages with ldc, then what do we do with libraries that are not ABI-compatible with dmd? Since users would expect that if they need libfoo, they'd just `apt-get install libfoo-dev` and then they should be able to just run` dmd -L-lfoo` and it should all magically "just work". [...]
 I'd like to be involved, since I *am* technically a Debian developer
 (though haven't been very active for the past long while -- I do try
 to at least keep the meager few packages I maintain up-to-date,
 though).
It's unlikely that I will maintain the dmd package (I might co-maintain it or sponsor uploads though - reason is simply lack of time and -ETOOMANYPROJECTS), so any help there is highly appreciated and I'd help with it whenever I can (I just can't make long-term commitments). If DMD is uploaded as part of the D team, spreading the maintenance cost will be rather simple too :-)
Yeah, the reason I haven't been very active in Debian is also due to ENOTIME, so I doubt I'd be able to keep up with maintaining dmd. But if it's spread among more D team members, it could be more workable. T -- Heuristics are bug-ridden by definition. If they didn't have bugs, they'd be algorithms.
Apr 10 2017
next sibling parent reply Wyatt <wyatt.epp gmail.com> writes:
On Monday, 10 April 2017 at 18:46:31 UTC, H. S. Teoh wrote:
 Hmm.  I guess there's no easy way to make dmd/ldc emit 
 dependencies with modified SONAMEs?  So yeah, you're right, 
 every software that depends on said libraries would have to 
 explicitly depend on a different SONAME depending on what they 
 were built with.  OK, crazy idea, nevermind. :-(
Doesn't sounds that crazy; you already do it with GCC versions, right? (Debian _does_ have something like that, right? Where you can pick your C compiler.) -Wyatt
Apr 10 2017
parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 20:11:28 UTC, Wyatt wrote:
 On Monday, 10 April 2017 at 18:46:31 UTC, H. S. Teoh wrote:
 Hmm.  I guess there's no easy way to make dmd/ldc emit 
 dependencies with modified SONAMEs?  So yeah, you're right, 
 every software that depends on said libraries would have to 
 explicitly depend on a different SONAME depending on what they 
 were built with.  OK, crazy idea, nevermind. :-(
Doesn't sounds that crazy; you already do it with GCC versions, right? (Debian _does_ have something like that, right? Where you can pick your C compiler.)
Yes, that's why all packages need to honor the CFLAGS/CC env var somehow or get the default C compiler from dpkg at build time, so we can easily apply new C flags globally, and projects to build the distro with Clang work. We do never change the SONAME of anything, however. We do track symbols and might change the SOVERSION occasionally if breakage is found (making 3 -> 3a etc.), but the soname isn't changed. The good thing is that Clang and GCC are (with few exceptions) very compatible.
Apr 11 2017
prev sibling parent reply Matthias Klumpp <mak debian.org> writes:
I apologize in advance for the large amount of mail that will 
likely follow, but I want to address all comments.

On Monday, 10 April 2017 at 18:46:31 UTC, H. S. Teoh wrote:
 [...]
 One issue, though: if we standardize on compiling Debian 
 packages with ldc, then what do we do with libraries that are 
 not ABI-compatible with dmd?  Since users would expect that if 
 they need libfoo, they'd just `apt-get install libfoo-dev` and 
 then they should be able to just run` dmd -L-lfoo` and it 
 should all magically "just work".
That's the problem I would like to see addressed (but given Walter's comment, it won't be feasible to resolve it in the near future). We could simply do away with "don't use distro packages for your D programming", at least that's what Go recommends.
Apr 11 2017
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 11:55 +0000, Matthias Klumpp via Digitalmars-d
wrote:
=20
[=E2=80=A6]
 That's the problem I would like to see addressed (but given=C2=A0
 Walter's comment, it won't be feasible to resolve it in the near=C2=A0
 future).
 We could simply do away with "don't use distro packages for your=C2=A0
 D programming", at least that's what Go recommends.
The Go community advice against platform packages is principally because they want up-to-date packages on old distributions. Most programming languages are going this route, used programming language tools and libraries are decreasingly packaged by the OS. Given things like Anaconda and PyPI, all the Fedora and Debian Python packaging is increasingly wasted =E2=80=93 except for those packages used b= y the distribution itself. So I can see Fedora and Debian packaging Python and some bits and pieces, but otherwise it's not needed. Same is true for Ruby, Go, Groovy, Java, etc.=20 Debian and Fedora really do need to reassess their packaging strategy with respect to software development. Far too much effort is going into packaging when rolling release is far better handled by language specific systems. Go is going this route, Python likewise. I suggest getting the compilers onto the distributions as a marketing thing not as a real use thing. For me the way Rust works by having it's own shell-based installer is the right way of installing languages. SDKMAN! is an even more extensive system for all JVM-related systems. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 11 2017
parent Enrico Weigelt, metux IT consult <enrico.weigelt gr13.net> writes:
On Tuesday, 11 April 2017 at 12:36:41 UTC, Russel Winder wrote:

 Most programming languages are going this route, used 
 programming language tools and libraries are decreasingly 
 packaged by the OS.
Exactly the reason why we banned Go quite recently. Just far too expensive, especially w/ crosscompile.
 Given things like Anaconda and PyPI, all the Fedora and Debian 
 Python packaging is increasingly wasted – except for those 
 packages used by the distribution itself.
Not at all. These folks just completely ignore the whole purpose of binary distros. And I regularily see that at operating costs. One of the major reasons why Windows operating is so expensive is the lack of proper package management and distro infrastructures. Few years ago, I've introduced apt-based package management for Zimbra extensions (for some solution provider who also develops lots of own extensions and customizations). Cut down deployment costs to less than 10%.
 So I can see Fedora and Debian packaging Python and some bits
 and pieces, but otherwise it's not needed. Same is true for 
 Ruby, Go, Groovy, Java, etc.
With ruby, just given up completely - the community has shown itself really hostile against distros, as well as any kind of decent release engineering, which makes TOC very very expensive. Python and Java are manageable (thanks to the hard work of distro folks). No idea about Groovy. Go is horrible - banned it from all projects.
 Debian and Fedora really do need to reassess their packaging 
 strategy with respect to software development.
Software developers should reassess their methodologies with respect to distros and operating - these are the folks who make the software actually usable for the arbitrary user, and they get all the basting if something goes wrong again.
 Far too much effort is going into packaging when rolling release
 is far better handled by language specific systems.
Far too much effort is going into proprietary / language specific homebrewn tools, by people who believe their own little subspace bubble was the whole universe and have no idea what matters in professional operating. Much of my professional work is being HoD - the interface between development, operating and PMO - or training development teams. The numbers on my table tell a clear story.
 I suggest getting the compilers onto the distributions as a 
 marketing thing not as a real use thing.
Why should Distros like Debian do any marketing ? Selling stuff isn't any of their objectives.
 For me the way Rust works by having it's own shell-based 
 installer
 is the right way of installing languages.
For me (as HoD, release manager, auditor, embedded engineer, software architect, etc) such things are an complete showstopper - reproducable build and deployment is a hard requirement - anything else is just dangerous and expensive. --mtx
May 19 2017
prev sibling parent David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
 Recompiling the dependency-chain of a software from source when 
 compiling a package using the "right" compiler and statically 
 adding the code is forbidden by distro policy.
Yet, from what I could find after a brief search, the Rust packaging guys are planning to do exactly that: https://wiki.debian.org/Teams/RustPackaging/Policy#Library_package_structure You might want to get in touch with them, particularly regarding
 […] require policy exceptions, which is something we won't do 
 […]
before either side wastes more time on this than necessary. — David
Apr 10 2017