www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ldc - Add targets for release binaries? (1.3+)

reply Johan Engelen <j j.nl> writes:
Would it make sense to release 1.3 with more targets enabled than 
the host?
X86, ARM, AArch64 ?

- Johan
Apr 22
next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:
 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?
I don't really think it makes sense for end users, as we would need to ship the respective standard libraries and a way to select between cross-compilation targets as well. That being said, purely from a "marketing" perspective it might be nice to increase visibility for ARM support in LDC. (Users might read the -help output or just try to specify a target triple.) The primary motivation behind shipping host-only releases was binary size - have you checked recently how the two build compare? -- David
Apr 22
next sibling parent Johan Engelen <j j.nl> writes:
On Saturday, 22 April 2017 at 11:55:39 UTC, David Nadlinger wrote:
 On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:
 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?
I don't really think it makes sense for end users, as we would need to ship the respective standard libraries and a way to select between cross-compilation targets as well.
The extra standard libraries could be separate downloads (hosted by whoever), but there is also `-betterC`. We could include a script that builds Phobos for a certain triple (I was already thinking about including such a script for building Phobos with LTO), but for druntime we don't include all the source...
 The primary motivation behind shipping host-only releases was 
 binary size - have you checked recently how the two build 
 compare?
Just tested: adding ARM and AArch64 to X86 on Mac adds 5 MB (LTO build).
Apr 22
prev sibling parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Saturday, 22 April 2017 at 11:55:39 UTC, David Nadlinger wrote:
 On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:
 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?
I don't really think it makes sense for end users, as we would need to ship the respective standard libraries and a way to select between cross-compilation targets as well.
Moreover, "ARM" alone doesn't mean anything in this context. The sheer number of platforms using different ABIs, architecture versions, library versions and configurations makes it virtually impossible to even build druntime and Phobos in a way that they could later be at least linked correctly for all ARM-based platforms. That being said, having binary packages that work out-of-the-box for a few popular targets, such as Raspbian, Android and OS X, wouldn't be a bad thing.
Apr 28
parent reply David Nadlinger via digitalmars-d-ldc <digitalmars-d-ldc puremagic.com> writes:
On 28 Apr 2017, at 9:23, Adrian Matoga via digitalmars-d-ldc wrote:
 Moreover, "ARM" alone doesn't mean anything in this context. The sheer 
 number of platforms using different ABIs, architecture versions, 
 library versions and configurations makes it virtually impossible to 
 even build druntime and Phobos in a way that they could later be at 
 least linked correctly for all ARM-based platforms.

 That being said, having binary packages that work out-of-the-box for a 
 few popular targets, such as Raspbian, Android and OS X, wouldn't be a 
 bad thing.
True. But as you mention, there are a couple of popular targets. For example, Debian/Ubuntu/Fedora/Arch/… all have an armhf distribution which targets ARMv7 without NEON. A binary package for that would already cover a huge portion of people just looking to try out LDC on their ARM dev board. (RP1 is of course its own thing, unfortunately.) — David
Apr 30
parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Sunday, 30 April 2017 at 15:44:50 UTC, David Nadlinger wrote:
 On 28 Apr 2017, at 9:23, Adrian Matoga via digitalmars-d-ldc 
 wrote:
 Moreover, "ARM" alone doesn't mean anything in this context. 
 The sheer number of platforms using different ABIs, 
 architecture versions, library versions and configurations 
 makes it virtually impossible to even build druntime and 
 Phobos in a way that they could later be at least linked 
 correctly for all ARM-based platforms.

 That being said, having binary packages that work 
 out-of-the-box for a few popular targets, such as Raspbian, 
 Android and OS X, wouldn't be a bad thing.
True. But as you mention, there are a couple of popular targets. For example, Debian/Ubuntu/Fedora/Arch/… all have an armhf distribution which targets ARMv7 without NEON. A binary package for that would already cover a huge portion of people just looking to try out LDC on their ARM dev board. (RP1 is of course its own thing, unfortunately.)
Yeah, GDC has a binary release for cross-compiling to armhf, which works well for RPi (including v1) out of the box. It's really a huge time saver. What I miss the most in LDC is the ability to build and use druntime/Phobos for different targets (including subtargets, e.g. different ARM architecture versions and ABIs) with the same compiler. I'd like to add it myself, but I don't really know where to start. Perhaps it's an idea to be discussed at this DConf's hackathon.
May 05
parent reply David Nadlinger via digitalmars-d-ldc <digitalmars-d-ldc puremagic.com> writes:
On 5 May 2017, at 9:52, Adrian Matoga via digitalmars-d-ldc wrote:
 What I miss the most in LDC is the ability to build and use 
 druntime/Phobos for different targets (including subtargets, e.g. 
 different ARM architecture versions and ABIs) with the same compiler. 
 I'd like to add it myself, but I don't really know where to start. 
 Perhaps it's an idea to be discussed at this DConf's hackathon.
The biggest issue to solve here is the rest of the toolchain (linker, system libraries, …). LDC's part can easily be done by adding a small piece of code to the config file handling such that different sections are matched depending on the chosen target (where you can then set library lookup paths for druntime/Phobos, the GCC excecutable to use for cross linking, etc.). Unfortunately I'm not at DConf this year, but I'd be happy to help out with any questions you might have implementing something like that. — David
May 05
next sibling parent reply Johan Engelen <j j.nl> writes:
On Friday, 5 May 2017 at 12:49:32 UTC, David Nadlinger wrote:
 On 5 May 2017, at 9:52, Adrian Matoga via digitalmars-d-ldc 
 wrote:
 What I miss the most in LDC is the ability to build and use 
 druntime/Phobos for different targets (including subtargets, 
 e.g. different ARM architecture versions and ABIs) with the 
 same compiler.
Perhaps what would be nice is a script that we can ship with LDC, that then compiles Phobos/druntime with specific compiler options: target triple, LTO flag (!), ... LDC will be able to link libraries in the future, so the script will not have to deal with some of the platform issues. -Johan
May 05
next sibling parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Friday, 5 May 2017 at 13:43:31 UTC, Johan Engelen wrote:
 On Friday, 5 May 2017 at 12:49:32 UTC, David Nadlinger wrote:
 On 5 May 2017, at 9:52, Adrian Matoga via digitalmars-d-ldc 
 wrote:
 What I miss the most in LDC is the ability to build and use 
 druntime/Phobos for different targets (including subtargets, 
 e.g. different ARM architecture versions and ABIs) with the 
 same compiler.
Perhaps what would be nice is a script that we can ship with LDC, that then compiles Phobos/druntime with specific compiler options: target triple, LTO flag (!), ... LDC will be able to link libraries in the future, so the script will not have to deal with some of the platform issues. -Johan
Yes, and I believe the "script" is doable in a few lines inserted in CMakeLists we already have. I need to improve my CMakese to prove it. For my target, I only had to change few things in runtime/CMakeLists.txt: set CMAKE_C_COMPILER, and add -triple and --sysroot. I believe in most cases it'll be just a matter of iterating over a list of targets, each having its specific flags and paths, and for each of them rebuilding druntime and Phobos in a separate directory. The other part of it is using the LDC you've built this way later. You should be able to conveniently specify which of these targets you want. Technically, you can already use -mtriple, -gcc, -defaultlib etc., but it could be made easier.
May 05
parent kinke <noone nowhere.com> writes:
On Friday, 5 May 2017 at 15:59:21 UTC, Adrian Matoga wrote:
 Yes, and I believe the "script" is doable in a few lines 
 inserted in CMakeLists we already have. I need to improve my 
 CMakese to prove it.
Our CMake scripts are a holy mess. An approach patching the runtime's CMakeLists.txt in the few required places, re-running CMake (with `-DCMAKE_C_COMPILER=... -DD_FLAGS=-w;-mtriple=...` etc.) and then building the new runtime libs (obviously using the already built LDC), all of this for each desired cross-target, might be a simpler approach.
 The other part of it is using the LDC you've built this way 
 later. You should be able to conveniently specify which of 
 these targets you want. Technically, you can already use 
 -mtriple, -gcc, -defaultlib etc., but it could be made easier.
See my post above; all you need is an extended config file and then specifying `-mtriple=...` for the LDC command line(s).
May 05
prev sibling parent David Nadlinger via digitalmars-d-ldc <digitalmars-d-ldc puremagic.com> writes:
On 5 May 2017, at 14:43, Johan Engelen via digitalmars-d-ldc wrote:
 LDC will be able to link libraries in the future, so the script will 
 not have to deal with some of the platform issues.
I believe you'd still need to have the target toolchain around at build time for shared libraries. — David
May 05
prev sibling parent reply kinke <noone nowhere.com> writes:
On Friday, 5 May 2017 at 12:49:32 UTC, David Nadlinger wrote:
 LDC's part can easily be done by adding a small piece of code 
 to the config file handling such that different sections are 
 matched depending on the chosen target (where you can then set 
 library lookup paths for druntime/Phobos, the GCC excecutable 
 to use for cross linking, etc.).
Rainer (IIRC) already implemented that a year ago or so. Just add additional sections, named exactly like the target triple, to the config file, e.g.: ` default: { switches = [ "-I%%ldcbinarypath%%/../import/ldc", "-I%%ldcbinarypath%%/../import", "-L-L%%ldcbinarypath%%/../lib64", "-defaultlib=phobos2-ldc,druntime-ldc", "-debuglib=phobos2-ldc-debug,druntime-ldc-debug" ]; }; armv7-none-linux-android: { switches = [ "-I%%ldcbinarypath%%/../import/ldc", "-I%%ldcbinarypath%%/../import", "-gcc=MY_CROSS_LINKING_GCC", "-L-LMY_ARM_LIBS_DIR", "-defaultlib=phobos2-ldc,druntime-ldc", "-debuglib=phobos2-ldc-debug,druntime-ldc-debug" ]; }; ` We could spice it up by allowing wildcards in the section names at some point, but cross-compiling druntime/Phobos, installing an appropriate cross-gcc and modifying the LDC config file is really all what's needed.
May 05
parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Friday, 5 May 2017 at 15:52:44 UTC, kinke wrote:
 On Friday, 5 May 2017 at 12:49:32 UTC, David Nadlinger wrote:
 LDC's part can easily be done by adding a small piece of code 
 to the config file handling such that different sections are 
 matched depending on the chosen target (where you can then set 
 library lookup paths for druntime/Phobos, the GCC excecutable 
 to use for cross linking, etc.).
Rainer (IIRC) already implemented that a year ago or so. Just add additional sections, named exactly like the target triple, to the config file, e.g.: ` default: { switches = [ "-I%%ldcbinarypath%%/../import/ldc", "-I%%ldcbinarypath%%/../import", "-L-L%%ldcbinarypath%%/../lib64", "-defaultlib=phobos2-ldc,druntime-ldc", "-debuglib=phobos2-ldc-debug,druntime-ldc-debug" ]; }; armv7-none-linux-android: { switches = [ "-I%%ldcbinarypath%%/../import/ldc", "-I%%ldcbinarypath%%/../import", "-gcc=MY_CROSS_LINKING_GCC", "-L-LMY_ARM_LIBS_DIR", "-defaultlib=phobos2-ldc,druntime-ldc", "-debuglib=phobos2-ldc-debug,druntime-ldc-debug" ]; }; ` We could spice it up by allowing wildcards in the section names at some point, but cross-compiling druntime/Phobos, installing an appropriate cross-gcc and modifying the LDC config file is really all what's needed.
Is this included in the releases? Or at least in master? I'm trying it out with 1.2.0 and LDC doesn't seem to pick up anything but "default", whatever -mtriple I pass to it.
May 05
parent reply kinke <noone nowhere.com> writes:
On Friday, 5 May 2017 at 16:27:10 UTC, Adrian Matoga wrote:
 Is this included in the releases? Or at least in master?
 I'm trying it out with 1.2.0 and LDC doesn't seem to pick up 
 anything but "default", whatever -mtriple I pass to it.
This is how our Win64 multilib package handles linking Win32 and Win64 binaries, so it's in master for at least a year or so. I just tested our prebuilt 1.2 Linux x64 package just to make sure and it works. By adding `-v` to the LDC command line, you see the used config file and, in parentheses, the section being looked up (falling back to `default` if it's missing), e.g. (when specifying `-m32` or `-mtriple=i686-unknown-linux-gnu`): binary /home/martin/ldc2-1.2.0-linux-x86_64/bin/ldc2 version 1.2.0 (DMD v2.072.2, LLVM 4.0.0) config /home/martin/ldc2-1.2.0-linux-x86_64/etc/ldc2.conf (i686-unknown-linux-gnu) ...
May 05
parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Friday, 5 May 2017 at 16:43:01 UTC, kinke wrote:
 On Friday, 5 May 2017 at 16:27:10 UTC, Adrian Matoga wrote:
 Is this included in the releases? Or at least in master?
 I'm trying it out with 1.2.0 and LDC doesn't seem to pick up 
 anything but "default", whatever -mtriple I pass to it.
This is how our Win64 multilib package handles linking Win32 and Win64 binaries, so it's in master for at least a year or so. I just tested our prebuilt 1.2 Linux x64 package just to make sure and it works. By adding `-v` to the LDC command line, you see the used config file and, in parentheses, the section being looked up (falling back to `default` if it's missing), e.g. (when specifying `-m32` or `-mtriple=i686-unknown-linux-gnu`): binary /home/martin/ldc2-1.2.0-linux-x86_64/bin/ldc2 version 1.2.0 (DMD v2.072.2, LLVM 4.0.0) config /home/martin/ldc2-1.2.0-linux-x86_64/etc/ldc2.conf (i686-unknown-linux-gnu) ...
Thank you. The "config" line is very informative. It seems that LDC doesn't use the specified triple verbatim, but may modify it, e.g. "arm-linux-gnueabi" becomes "arm--linux-gnueabi". Taking this into account, it works and it looks like it's a part of the solution I'm looking for.
May 05
parent reply kinke <noone nowhere.com> writes:
On Friday, 5 May 2017 at 16:56:57 UTC, Adrian Matoga wrote:
 It seems that LDC doesn't use the specified triple verbatim, 
 but may modify it, e.g. "arm-linux-gnueabi" becomes 
 "arm--linux-gnueabi".
That 2nd component of the 4-component 'triple' bothers me too. I've only seen the values `pc`, `unknown`, `none` and now, thanks to you ;), even '' for it so far.
May 05
next sibling parent Adrian Matoga <dlang.spam matoga.info> writes:
On Friday, 5 May 2017 at 18:49:46 UTC, kinke wrote:
 On Friday, 5 May 2017 at 16:56:57 UTC, Adrian Matoga wrote:
 It seems that LDC doesn't use the specified triple verbatim, 
 but may modify it, e.g. "arm-linux-gnueabi" becomes 
 "arm--linux-gnueabi".
That 2nd component of the 4-component 'triple' bothers me too. I've only seen the values `pc`, `unknown`, `none` and now, thanks to you ;), even '' for it so far.
It seems that any number of components is accepted as a 'triple', and if there's more than four, some of them are moved from the inside to the back of it. :) I'd like to have a couple of targets side by side that would all use "arm-linux-gnueabi" (or maybe "arm-tizen-linux-gnueabi"?), but would differ in their sysroot paths. I could add extra components to the triple to distinguish these targets and it'd work with the current LDC, but it seems a bit hackish to me. What do you think?
May 05
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Friday, 5 May 2017 at 18:49:46 UTC, kinke wrote:
 On Friday, 5 May 2017 at 16:56:57 UTC, Adrian Matoga wrote:
 It seems that LDC doesn't use the specified triple verbatim, 
 but may modify it, e.g. "arm-linux-gnueabi" becomes 
 "arm--linux-gnueabi".
That 2nd component of the 4-component 'triple' bothers me too. I've only seen the values `pc`, `unknown`, `none` and now, thanks to you ;), even '' for it so far.
the second component of the triple is the vendor, so hardware-vendor-os/environment[-abi]. but yeah a blank '' is weird. Most OSX triples are $hardware-apple-$os. All the NVPTX triples are $hardware-nvidia-(cuda|opencl).
May 14
prev sibling next sibling parent reply kinke <noone nowhere.com> writes:
On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:
 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?

 - Johan
FWIW, the MSVC packages already feature a bunch of other targets incl. ARM and PowerPC [https://github.com/ldc-developers/ldc-scripts/blob/master/ldc2-msvc/RELEASE.proj#L67].
Apr 22
parent reply Johan Engelen <j j.nl> writes:
On Saturday, 22 April 2017 at 12:51:23 UTC, kinke wrote:
 On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:
 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?

 - Johan
FWIW, the MSVC packages already feature a bunch of other targets incl. ARM and PowerPC
Ah, so let's use the same list for the ldc-scripts script?
Apr 22
parent kinke <noone nowhere.com> writes:
On Saturday, 22 April 2017 at 14:53:46 UTC, Johan Engelen wrote:
 On Saturday, 22 April 2017 at 12:51:23 UTC, kinke wrote:
 On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen 
 wrote:
 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?

 - Johan
FWIW, the MSVC packages already feature a bunch of other targets incl. ARM and PowerPC
Ah, so let's use the same list for the ldc-scripts script?
I don't have a strong opinion on this (and I didn't create that list), but a couple of MBytes (uncompressed) seem totally worth it to me (at least for x86 hosts), even if mainly for advertising the cross-compilation capabilities and target variety via `ldc2 --version`. With 1.3 able to generate entire static libs for all supported targets, it'd make some more sense I think.
Apr 22
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:
 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?

 - Johan
Not for 1.3, but once the compiler dcompute stuff is finished it'd be great to have NVPTX (and SPIRV once I get it into LLVM and we release with a version that has it). The increased visibility would be very god to have.
Apr 22