www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ldc - Testing out the new cross-compilation support in ldc master

reply Joakim <dlang joakim.fea.st> writes:
I merged kinke's pull two weeks ago that adds a small D build 
tool which makes it easy to download and cross-compile the 
standard libary for other platforms, so you can easily use the 
official ldc releases for {Mac,Win,Linux}/{x86,x64} as a 
cross-compiler for newer platforms that ldc is coming to, like 
Android/ARM or Linux/PPC.  It can also be used for LTO or 
sanitizer-enabled builds of the standard library, basically 
anything where you want the stdlib built differently:

https://github.com/ldc-developers/ldc/pull/2253

Right now, it has to be built from the git repo, as the first 
official release it will ship with is the upcoming ldc 1.4 beta, 
would be good if more people tried it out and let us know how 
it's working. Generally speaking, you need all the same dev tools 
you'd need if you were building from source before, such as git, 
CMake, Make or Ninja, and a C cross-compiler/linker toolchain for 
the platform you want to target (needed because of a few 
C/assembly files in the stdlib).

Specifically, I'm interested in how well it works on Mac and 
Windows to cross-compile for Android/ARM, as I haven't used 
Mac/Windows in years and don't have those platforms available.  
You can wait till the ldc 1.4 beta is out, but if you're 
comfortable building ldc master from source, here are some 
instructions on how to try it.  You will be following the same 
routine as the first third of this wiki page, except you won't be 
limited to cross-compiling from linux anymore:

https://wiki.dlang.org/Build_LDC_for_Android

The first three steps are the differences from that wiki page, 
then instructions on building and using the new tool:

1. All the same prerequisites, except you don't need the Android 
SDK nor obviously the linux/x64 shell.  See the steps to download 
the NDK for Windows 10, for a simple way on how to get it.
2. For Android/ARM, you need to compile a lightly tweaked llvm, 
as shown in the instructions there (with one exception, leave off 
the default target triple option).  For other targets, like 
Linux/PPC, a stock llvm is fine.
3. Build ldc and druntime/phobos as you would normally and shown 
there, except using the master branch and no need to apply those 
Android patches.
4. Once you've built ldc, druntime and phobos, build kinke's new 
tool with this command:

make ldc-build-runtime

5. Finally, you use this new tool to cross-compile the standard 
library and possibly the test runners for your target.  Here's an 
example from the comments of that PR, for Android/ARM, all one 
command that I've expanded for legibility:

CC=/home/joakim/ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang

./bin/ldc-build-runtime --ldcSrcDir=/home/joakim/ldc 
BUILD_SHARED_LIBS=OFF C_SYSTEM_LIBS="m;c"

--cFlags="-gcc-toolchain;/home/joakim/ndk-r15c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64;-fpic;-ffunction-sections;-funwind-tables;-fstack-protector-strong;-Wno-invalid-command-line-argument;-Wno-unused-command-line-argument;-no-canonical-prefixes;-fno-integrated-as;-target;armv7-none-linux-androideabi;-march=armv7-a;-mfloat-abi=softfp;-mfpu=vfpv3-d16;-mthumb;-Os;-g;-DNDEBUG;-fomit-frame-pointer;-fno-strict-aliasing;-DANDROID;-Wa,--noexecstack;-Wformat;-Werror=format-security;-isystem;/home/joakim/ndk-r15c/platforms/android-21/arch-arm/usr/include"

--linkerFlags="-Wl,-z,nocopyreloc;--sysroot=/home/joakim/ndk-r15c/platforms/android-21/arch-arm;-lgcc;-gcc-toolchain;/home/joakim/ndk-r15c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64;-target;armv7-none-linux-androideabi;-no-canonical-prefixes;-fuse-ld=bfd;-Wl,--fix-cortex-a8;-Wl,--no-undefined;-Wl,-z,noexecstack;-Wl,-z,relro;-Wl,-z,now;-Wl,--warn-shared-textrel;-Wl,--fatal-warnings;-fPIE;-pie"

--dFlags="-w;-mtriple=armv7-none-linux-android" --ninja 
--testrunners

CC specifies the path to your C cross-compiler, which is assumed 
to invoke the right linker for your target.  For Android/ARM, 
replace all uses of /home/joakim/ndk-r15c with the path to your 
NDK.  I think you have to specify --ldcSrcDir, the path to your 
ldc source, since you're building from the git repo.  The Android 
port doesn't support building the stdlib as a shared library yet, 
so I turn it off.  It also uses a different set of C system 
libraries, because Android includes pthreads in libc, for example.

The --cFlags and --linkerFlags were taken from running the 
Android ndk-build script in verbose mode.  Use --dFlags to set 
the llvm target triple for your platform, as I do here for 
Android/ARM.  Leave off --ninja if you'd like to use Make 
instead, same for --testrunners if you don't want to run the 
stdlib tests.  Run ./ldc-build-runtime -h to see the full list of 
flags.

Once that builds, assuming all goes well, try downloading 
druntime-test-runner and phobos2-test-runner to a terminal 
emulator on your Android/ARM device and running them, if you 
built the test runners.  Otherwise, you can follow the 
instructions here to cross-compile a sample D program, though 
you'll need to add the -mtriple=armv7-none-linux-android flag 
when calling ldc:

https://wiki.dlang.org/Build_D_for_Android#Build_a_command-line_executable

kinke, will setting CC first work on Windows and other shells?  
If not, I guess it will work if passed after the invocation of 
ldc-build-runtime, like all the other flags.

I'll build a native ldc 1.4 for Android/ARM, to make available 
with the upcoming official beta release, just as I do now for ldc 
1.3 on my own github repo.  Maybe that termux-packages build 
script can be merged into the official packaging ldc-scripts 
eventually.

I'd like to make it easier to cross-compile for supported 
platforms, by being able to pass a target OS and architecture 
alone to this build tool and CMake, which would call some default 
C and linker flags in the official CMake config.  You will still 
be able to override the defaults with custom C/linker flags.  
I'll submit a PR for Android/ARM, hopefully we'll have more 
defaults for Linux/PPC, Linux/armhf, and other more well-tested 
platforms.  I hope you're not waiting on that PR to push the ldc 
1.4 beta out, kinke.
Aug 18 2017
next sibling parent reply kinke <noone nowhere.com> writes:
Thanks for the write-up!

On Friday, 18 August 2017 at 07:34:04 UTC, Joakim wrote:

 Right now, it has to be built from the git repo, as the first 
 official release it will ship with is the upcoming ldc 1.4 beta


Not entirely true; we have CI builds for Windows, which happen to 
include our tailored (Android-ready) LLVM already. It obviously 
also includes the ldc-build-runtime tool (which btw is part of 
the `all` build target, no need to be built manually). So for a 
quick test on Windows, a recent CI build is already enough, and 
one only needs the prerequisites (Android NDK, CMake, ninja) and 
LDC source and can then start with the last step (5).


 kinke, will setting CC first work on Windows and other shells?


Nope, a combined command `CC=... ldc-build-runtime ...` doesn't 
work in the regular Windows shell. Environment variables need to 
be set (exported in Posix lingo) explicitly: `set CC=...`, then 
invoke the tool. That being said, I have no idea whether CMake 
honors that environment variable on Windows; if it doesn't, one 
should be able to fall back to `ldc-build-runtime 
CMAKE_C_COMPILER=... CMAKE_LINKER=... ...`.


 I hope you're not waiting on that PR to push the ldc 1.4 beta 
 out, kinke.


Nope, don't worry ;) - I think we're pretty much ready for 
1.4-beta1. DMD has already pushed out a 2.076 beta...
Aug 18 2017
parent Joakim <dlang joakim.fea.st> writes:
On Friday, 18 August 2017 at 16:12:25 UTC, kinke wrote:

 Thanks for the write-up!

 On Friday, 18 August 2017 at 07:34:04 UTC, Joakim wrote:

 Right now, it has to be built from the git repo, as the first 
 official release it will ship with is the upcoming ldc 1.4 beta


 Not entirely true; we have CI builds for Windows, which happen 
 to include our tailored (Android-ready) LLVM already. It 
 obviously also includes the ldc-build-runtime tool (which btw 
 is part of the `all` build target, no need to be built 
 manually). So for a quick test on Windows, a recent CI build is 
 already enough, and one only needs the prerequisites (Android 
 NDK, CMake, ninja) and LDC source and can then start with the 
 last step (5).


OK, didn't know about those.  Here's the wiki page with more 
info, if anyone wants to try it out on Windows:

https://wiki.dlang.org/Latest_LDC_binaries_for_Windows
Aug 18 2017
prev sibling next sibling parent kinke <noone nowhere.com> writes:
I created a Wiki page about ldc-build-runtime: 
https://wiki.dlang.org/Building_LDC_runtime_libraries
Aug 19 2017
prev sibling next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Friday, 18 August 2017 at 07:34:04 UTC, Joakim wrote:

 I merged kinke's pull two weeks ago that adds a small D build 
 tool which makes it easy to download and cross-compile the 
 standard libary for other platforms, so you can easily use the 
 official ldc releases for {Mac,Win,Linux}/{x86,x64} as a 
 cross-compiler for newer platforms that ldc is coming to, like 
 Android/ARM or Linux/PPC.  It can also be used for LTO or 
 sanitizer-enabled builds of the standard library, basically 
 anything where you want the stdlib built differently:

 [...]


Crap, this small patch to omit building core.stdc.tgmath is still 
needed:

https://gist.github.com/joakim-noah/d74af3cf1355492557a9c56ef1bf2636#file-ldc_1-3_android_arm-L18

I always apply it so forgot about that until I just tried 
cross-compiling the stdlib for Android/ARM with the new ldc 1.4 
beta1.  Also, I wanted to get a small PR in for that and to allow 
cross-compiling by just specifying the OS and Arch, but I got 
side-tracked with other stuff and didn't get that done before the 
beta1 release.

Just add that one line to the runtime CMake config when the build 
fails, then rerun the ldc-build-runtime command from the wiki 
page kinke just added and it should work.
Aug 27 2017
parent David Nadlinger via digitalmars-d-ldc <digitalmars-d-ldc puremagic.com> writes:
On 27 Aug 2017, at 8:49, Joakim via digitalmars-d-ldc wrote:

 Just add that one line to the runtime CMake config when the build 
 fails, then rerun the ldc-build-runtime command from the wiki page 
 kinke just added and it should work.


Sounds like there should be a beta branch PR for this. ;) —David
Aug 27 2017
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Friday, 18 August 2017 at 07:34:04 UTC, Joakim wrote:

 5. Finally, you use this new tool to cross-compile the standard 
 library and possibly the test runners for your target.  Here's 
 an example from the comments of that PR, for Android/ARM, all 
 one command that I've expanded for legibility:

 CC=/home/joakim/ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang

 ./bin/ldc-build-runtime --ldcSrcDir=/home/joakim/ldc 
 BUILD_SHARED_LIBS=OFF C_SYSTEM_LIBS="m;c"

 --cFlags="-gcc-toolchain;/home/joakim/ndk-r15c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64;-fpic;-ffunction-sections;-funwind-tables;-fstack-protector-strong;-Wno-invalid-command-line-argument;-Wno-unused-command-line-argument;-no-canonical-prefixes;-fno-integrated-as;-target;armv7-none-linux-androideabi;-march=armv7-a;-mfloat-abi=softfp;-mfpu=vfpv3-d16;-mthumb;-Os;-g;-DNDEBUG;-fomit-frame-pointer;-fno-strict-aliasing;-DANDROID;-Wa,--noexecstack;-Wformat;-Werror=format-security;-isystem;/home/joakim/ndk-r15c/platforms/android-21/arch-arm/usr/include"

 --linkerFlags="-Wl,-z,nocopyreloc;--sysroot=/home/joakim/ndk-r15c/platforms/android-21/arch-arm;-lgcc;-gcc-toolchain;/home/joakim/ndk-r15c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64;-target;armv7-none-linux-androideabi;-no-canonical-prefixes;-fuse-ld=bfd;-Wl,--fix-cortex-a8;-Wl,--no-undefined;-Wl,-z,noexecstack;-Wl,-z,relro;-Wl,-z,now;-Wl,--warn-shared-textrel;-Wl,--fatal-warnings;-fPIE;-pie"

 --dFlags="-w;-mtriple=armv7-none-linux-android" --ninja 
 --testrunners

 CC specifies the path to your C cross-compiler, which is 
 assumed to invoke the right linker for your target.  For 
 Android/ARM, replace all uses of /home/joakim/ndk-r15c with the 
 path to your NDK.  I think you have to specify --ldcSrcDir, the 
 path to your ldc source, since you're building from the git 
 repo.  The Android port doesn't support building the stdlib as 
 a shared library yet, so I turn it off.  It also uses a 
 different set of C system libraries, because Android includes 
 pthreads in libc, for example.

 The --cFlags and --linkerFlags were taken from running the 
 Android ndk-build script in verbose mode.  Use --dFlags to set 
 the llvm target triple for your platform, as I do here for 
 Android/ARM.  Leave off --ninja if you'd like to use Make 
 instead, same for --testrunners if you don't want to run the 
 stdlib tests.  Run ./ldc-build-runtime -h to see the full list 
 of flags.


I have updated the Android wiki page with full instructions on 
using the 1.4 release to cross-compile to Android/ARM from linux 
and Windows PCs:

https://wiki.dlang.org/Build_D_for_Android

The giant ldc-build-runtime command above has been integrated 
into the CMake config and boiled down to just `ldc-build-runtime 
--targetPreset=Android-arm`:

https://wiki.dlang.org/Build_D_for_Android#linux

If someone can try cross-compiling from a mac, which I haven't 
had in almost a decade, by using the Android NDK and either 
filling in the equivalent commands on the wiki or posting them 
here, that should complete it.


 Once that builds, assuming all goes well, try downloading 
 druntime-test-runner and phobos2-test-runner to a terminal 
 emulator on your Android/ARM device and running them, if you 
 built the test runners.  Otherwise, you can follow the 
 instructions here to cross-compile a sample D program, though 
 you'll need to add the -mtriple=armv7-none-linux-android flag 
 when calling ldc:

 https://wiki.dlang.org/Build_D_for_Android#Build_a_command-line_executable


It's not so simple to build the samples, I've updated that page 
with all that was needed, including using ldc with -Xcc.


 I'd like to make it easier to cross-compile for supported 
 platforms, by being able to pass a target OS and architecture 
 alone to this build tool and CMake, which would call some 
 default C and linker flags in the official CMake config.  You 
 will still be able to override the defaults with custom 
 C/linker flags.  I'll submit a PR for Android/ARM, hopefully 
 we'll have more defaults for Linux/PPC, Linux/armhf, and other 
 more well-tested platforms.  I hope you're not waiting on that 
 PR to push the ldc 1.4 beta out, kinke.


Let's get cross-compilation presets for other supported platforms 
in before the 1.5 release:

https://github.com/ldc-developers/ldc/blob/master/runtime/PresetRuntimeConfiguration.cmake

For example, it'd be good to get kinke's linux->Windows 
cross-compilation config in there, and instructions on setting it 
up and using it into the ldc-build-runtime wiki page.  The same 
for Linux/armhf, which I think some have been cross-compiling.
Sep 18 2017