www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DMD as cross-compiler

reply Walter Bright <newshound2 digitalmars.com> writes:
I realized that DMD is pretty close to being a cross-compiler, and won't need 
much work to go the rest of the way. I anticipate it could be very convenient 
when inconvenient to get the native DMD installed. It also should be useful for 
debugging problems with test suite failures :-/

Much of the static versioning in the backend is the result of converting it
from 
C with its #if statements. Being done with all that preprocessor legacy will be 
nice.

Working on a series of incremental PRs to make this happen.
Nov 11 2020
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Wednesday, 11 November 2020 at 10:49:54 UTC, Walter Bright 
wrote:
 I realized that DMD is pretty close to being a cross-compiler, 
 and won't need much work to go the rest of the way. I 
 anticipate it could be very convenient when inconvenient to get 
 the native DMD installed. It also should be useful for 
 debugging problems with test suite failures :-/

 Much of the static versioning in the backend is the result of 
 converting it from C with its #if statements. Being done with 
 all that preprocessor legacy will be nice.

 Working on a series of incremental PRs to make this happen.
Jonathon raised today that FreeBSD 11 -> FreeBSD 12 as been an issue for cross-compiling since the build scripts were replaced by a build.d source file. It seems that cross-compiling between incompatible versions of libc on the same platform still needs to be sorted out.
Nov 11 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 11 November 2020 at 12:28:17 UTC, Iain Buclaw wrote:

 Jonathon raised today that FreeBSD 11 -> FreeBSD 12 as been an 
 issue for cross-compiling since the build scripts were replaced 
 by a build.d source file.
What's the issue with that? Use LDC and compile build.d to target FreeBSD 12. Sure, you need the SDK, but you should be able to copy that from a FreeBSD 12 machine. -- /Jacob Carlborg
Nov 11 2020
parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On Thursday, 12 November 2020 at 06:45:37 UTC, Jacob Carlborg 
wrote:
 On Wednesday, 11 November 2020 at 12:28:17 UTC, Iain Buclaw 
 wrote:

 Jonathon raised today that FreeBSD 11 -> FreeBSD 12 as been an 
 issue for cross-compiling since the build scripts were 
 replaced by a build.d source file.
What's the issue with that? Use LDC and compile build.d to target FreeBSD 12. Sure, you need the SDK, but you should be able to copy that from a FreeBSD 12 machine.
Judging from the state of druntime, LDC doesn't work on FreeBSD at all. https://github.com/ldc-developers/druntime/blob/412467a452e2d12f561a2eace933dd44014af3c6/src/core/sys/freebsd/sys/mount.d#L291-L302 What's broken? pragma(mangle) string is invalid. What needs to be fixed? There needs to be a new pragma(symver) added to support symbol versioning feature of ELF. --- If it was me porting, I'd do it in the stages of: 1. Build FBSD11 compiler on FBSD11. 2. Cross-compile FBSD12 compiler on FBSD11 using compiler built in stage 1. 3. Copy FBSD12 compiler over to FBSD12. 4. Build FBSD12 compiler on FBSD12 using compiler built in stage 2. I think where Jonathon is going wrong is in skipping step 2 and jumping straight to three. In the old make build system this just worked because (as I understand) there's a FBSD11 compatibility layer, allowing old binaries to run on the new system. But as that is no more, building build.d using the stage 1 compiler on FBSD12 will result in a broken application (as it's linking against FBSD12 libc).
Nov 12 2020
prev sibling next sibling parent Andrea Fontana <nospam example.com> writes:
On Wednesday, 11 November 2020 at 10:49:54 UTC, Walter Bright 
wrote:
 I realized that DMD is pretty close to being a cross-compiler, 
 and won't need much work to go the rest of the way. I 
 anticipate it could be very convenient when inconvenient to get 
 the native DMD installed. It also should be useful for 
 debugging problems with test suite failures :-/

 Much of the static versioning in the backend is the result of 
 converting it from C with its #if statements. Being done with 
 all that preprocessor legacy will be nice.

 Working on a series of incremental PRs to make this happen.
Wow nice! But will it work out-of-the-box? I mean: what about linking with druntime, for example? Andrea
Nov 11 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 11 November 2020 at 10:49:54 UTC, Walter Bright 
wrote:
 I realized that DMD is pretty close to being a cross-compiler, 
 and won't need much work to go the rest of the way. I 
 anticipate it could be very convenient when inconvenient to get 
 the native DMD installed. It also should be useful for 
 debugging problems with test suite failures :-/

 Much of the static versioning in the backend is the result of 
 converting it from C with its #if statements. Being done with 
 all that preprocessor legacy will be nice.

 Working on a series of incremental PRs to make this happen.
But what would be a target? ARM? MIPS?
Nov 11 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 11 November 2020 at 16:57:53 UTC, Stefan Koch wrote:
 But what would be a target?
 ARM?
 MIPS?
Windows, Linux, and Mac I'd imagine. I tend to just run `wine dmd` to do linux-> windows but it might still be useful anyway. this reminds me i still need to make my generic ldc x-compile setup program.
Nov 11 2020
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Nov 11, 2020 at 05:11:51PM +0000, Adam D. Ruppe via Digitalmars-d wrote:
 On Wednesday, 11 November 2020 at 16:57:53 UTC, Stefan Koch wrote:
 But what would be a target?
 ARM?
 MIPS?
Windows, Linux, and Mac I'd imagine. I tend to just run `wine dmd` to do linux-> windows but it might still be useful anyway.
I prefer LDC for cross-compiling, as it can do both Windows and Android cross-compile.
 this reminds me i still need to make my generic ldc x-compile setup
 program.
I just download and unpack both the Linux and Windows versions of LDC, then add this to ldc2.conf (for Linux version): "(i686|x86_64)-.*-windows.msvc": { switches = [ "-defaultlib=phobos2-ldc,druntime-ldc", "-link-defaultlib-shared=false", ]; lib-dirs = [ "/path/to/ldc2-1.24.0-windows-x64/lib", ]; }; Then just compile with `-mtriple=x84_64-windows-msvc` and it Just Works(tm). It's been awesome. T -- VI = Visual Irritation
Nov 11 2020
prev sibling parent reply =?UTF-8?B?Um9iZXJ0IE0uIE3DvG5jaA==?= <robert.muench saphirion.com> writes:
On 11 Nov 2020 at 11:49:54 CET, "Walter Bright" <newshound2 digitalmars.com>
wrote:

 I realized that DMD is pretty close to being a cross-compiler, and won't need
 
 much work to go the rest of the way. I anticipate it could be very convenient
 
 when inconvenient to get the native DMD installed.
Digging into Go since some weeks, and the x-compiling of it is just super: GOOS=windows GOARCH=amd64 go build And you get a Windows EXE from OSX. That's it, zero config setup. The simpler these things work, the better for the whole language. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Nov 11 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 11 November 2020 at 21:09:14 UTC, Robert M. Münch 
wrote:

 Digging into Go since some weeks, and the x-compiling of it is 
 just super:

 GOOS=windows GOARCH=amd64 go build

 And you get a Windows EXE from OSX. That's it, zero config 
 setup. The simpler these things work, the better for the whole 
 language.
The reason why it works for Go is because they have their own full toolchain. That means, their own compiler (including backend), their own assembler, their own object format and their own linker. All of these are written to be cross-target. In addition to that, Go only interfaces with the kernel API. On Linux, that means direct system calls, they're not even going through the C wrappers. That's perfect for cross-compiling, since there's nothing to link against. They used to do this on macOS as well, but had to back down, because on macOS the ABI is not the kernel, but the C wrappers for the system calls. Zig is doing something similar, they're actually doing one better. Zig supports cross-compiling of C and C++ code as well (I don't think Go does this). Zig ships with Clang and the C standard library for various targets. But Zig uses their own frontend but the LLVM backend and the LLD linker. They do seem to have some trouble with linking some things on macOS. Also, on macOS they're cheating. Because they cannot ship the SDK, they solve the linking problem by allowing undefined symbols. I don't know if Go does the same. Unfortunately it won't be that simple for D. D took the easy way out and relies both on the C standard library and other platform specific libraries implemented in C. While it's easy to cross-compile and object file, linking will not be as easy. There's LLD, which works on Windows and Linux, but it doesn't work for macOS. You also need the SDK to have something to link with. Targeting Windows, it might be this easy, using LDC, because it's already relying on LLD and the MinGW libraries. -- /Jacob Carlborg
Nov 11 2020
next sibling parent reply Elronnd <elronnd elronnd.net> writes:
On Thursday, 12 November 2020 at 06:42:00 UTC, Jacob Carlborg 
wrote:
 Unfortunately it won't be that simple for D. D took the easy 
 way out and relies both on the C standard library and other 
 platform specific libraries implemented in C. While it's easy 
 to cross-compile and object file, linking will not be as easy.
FWIW I started working on some stubs for a pure-d libc a while back - https://github.com/moon-chilled/dlibc/. If it were completed and integrated into druntime--or, at least, if enough of it were completed to support druntime+phobos--then it would make for a much nicer cross-compiling experience.
 Also, on macOS they're cheating. Because they cannot ship the 
 SDK, they solve the linking problem by allowing undefined 
 symbols. I don't know if Go does the same.
Perhaps a better solution would be to link to a stub dylib of libc? (Assuming macos libc has a stable ABI.)
 There's LLD, which works on Windows and Linux, but it doesn't 
 work for macOS.
As of about a year and a half ago, its status was
 LLD for Mach-O has bitrotted significantly and isn't really 
 functional anymore (beyond the most basic programs). I'd 
 recommend sticking with ld64 as the linker.
(From http://lists.llvm.org/pipermail/cfe-dev/2019-March/061666.html) so the situation may have improved in the mean time; looking at the logs (https://github.com/llvm/llvm-project/commits/master/lld/MachO) I see a fair amount of activity. (In particular, commit 2e8e1bd adds support for stub dylibs, relevant to my above proposal.)
Nov 12 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Thursday, 12 November 2020 at 18:58:56 UTC, Elronnd wrote:

 FWIW I started working on some stubs for a pure-d libc a while 
 back - https://github.com/moon-chilled/dlibc/.  If it were 
 completed and integrated into druntime--or, at least, if enough 
 of it were completed to support druntime+phobos--then it would 
 make for a much nicer cross-compiling experience.
That's nice. But I would prefer if druntime and Phobos did not depend on the C standard library, even if it's implemented in D. You're constrained by C and by the API of the C standard library. Implementing druntime and Phobos without the C standard library will give a big opportunity to do things better.
 Perhaps a better solution would be to link to a stub dylib of 
 libc?  (Assuming macos libc has a stable ABI.)
That's already what I'm working on [1].
 LLD for Mach-O has bitrotted significantly and isn't really 
 functional anymore (beyond the most basic programs). I'd 
 recommend sticking with ld64 as the linker.
Same here, working on that as well [2]. [1] https://github.com/d-cross-compiler/sdk-apple/releases/tag/v0.0.1 [2] https://github.com/d-cross-compiler/cctools-port/releases/tag/cctools-949.0.1-ld64-530 -- /Jacob Carlborg
Nov 12 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2020 11:15 AM, Jacob Carlborg wrote:
 Implementing druntime and Phobos without the 
 C standard library will give a big opportunity to do things better.
How so? (Not that you're wrong, I'm just genuinely curious.)
Nov 12 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Friday, 13 November 2020 at 03:33:05 UTC, Walter Bright wrote:
 On 11/12/2020 11:15 AM, Jacob Carlborg wrote:
 Implementing druntime and Phobos without the C standard 
 library will give a big opportunity to do things better.
How so? (Not that you're wrong, I'm just genuinely curious.)
It might not necessary mean a better or different API for Phobos. But the lower level APIs could be better. Classic examples are: * Use arrays instead of pointers (or pointer length pair). `memcpy` would be: void[] memcpy(void[] source, void[] destination); // I think it's more natural to pass the source first * Better error handling. I think returning error codes or using `errno` is pretty bad error handling strategy. Even though exceptions might not be the right choice (unless something like this is used [1]), something like returning: `Result!(Value, Error)` might be better. * No global symbols, i.e. not having to worry about conflicting names across multiple files In short, all the reasons for which you created D in the first place ;) Unfortunately there will still be some limitations due to how the kernel APIs use pointers and error codes/errno. [1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0709r0.pdf -- /Jacob Carlborg
Nov 14 2020
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Saturday, 14 November 2020 at 19:19:02 UTC, Jacob Carlborg 
wrote:
 On Friday, 13 November 2020 at 03:33:05 UTC, Walter Bright 
 wrote:
 On 11/12/2020 11:15 AM, Jacob Carlborg wrote:
 Implementing druntime and Phobos without the C standard 
 library will give a big opportunity to do things better.
How so? (Not that you're wrong, I'm just genuinely curious.)
It might not necessary mean a better or different API for Phobos. But the lower level APIs could be better. Classic examples are: * Use arrays instead of pointers (or pointer length pair). `memcpy` would be: void[] memcpy(void[] source, void[] destination); // I think it's more natural to pass the source first
Copying code will be same.
 * Better error handling. I think returning error codes or using 
 `errno` is pretty bad error handling strategy. Even though 
 exceptions might not be the right choice (unless something like 
 this is used [1]), something like returning: `Result!(Value, 
 Error)` might be better.
Errno can be covered by druntime like in: https://dlang.org/phobos/core_memory.html#.pureMalloc
 * No global symbols, i.e. not having to worry about conflicting 
 names across multiple files
You don't have to worry even now, even if there is a conflict, this can be resolved by linking settings (I propose to switch from DUB to Meson :-)) Really, libc is not (only) C library, but de-facto standard for accessing to OS syscalls. It is possible to drop it but it doesn't make any sense, it just adds extra work to maintain a huge compatibility layer for all supported OSes. I once thought differently, but recently delved deeper into druntime and looked how it works. In total, I think, there are about ~15 calls to libc, not so many.
Nov 14 2020
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Sunday, 15 November 2020 at 07:21:28 UTC, Denis Feklushkin 
wrote:
 [...]

 Really, libc is not (only) C library, but de-facto standard for 
 accessing to OS syscalls. It is possible to drop it but it 
 doesn't make any sense, it just adds extra work to maintain a 
 huge compatibility layer for all supported OSes.

 I once thought differently, but recently delved deeper into 
 druntime and looked how it works. In total, I think, there are 
 about ~15 calls to libc, not so many.
Only for the OSes where C was born on, UNIX and POSIX clones. There are still several platforms around where this is not the case, because their main systems programming language isn't C, e.g. IBM and Unisys mainframes. Embedded platforms that use their own toolchains without any kind of POSIX compatibility. And desktop OSes like Windows, which follows the tradition of C and C++ compilers outside the UNIX world, where each compiler vendor ships their own ISO compliant library, and the OS syscalls are completly unrelated, e.g. ZeroMemory() instead of memset(). Amiga, Atari, OS/2, BeOS, Symbian, Brew, Mac OS pre OS X, Newton, PalmOS, are all examples of this tradition.
Nov 14 2020
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Sunday, 15 November 2020 at 07:51:52 UTC, Paulo Pinto wrote:
 On Sunday, 15 November 2020 at 07:21:28 UTC, Denis Feklushkin 
 wrote:
 [...]

 Really, libc is not (only) C library, but de-facto standard 
 for accessing to OS syscalls. It is possible to drop it but it 
 doesn't make any sense, it just adds extra work to maintain a 
 huge compatibility layer for all supported OSes.

 I once thought differently, but recently delved deeper into 
 druntime and looked how it works. In total, I think, there are 
 about ~15 calls to libc, not so many.
Only for the OSes where C was born on, UNIX and POSIX clones. There are still several platforms around where this is not the case, because their main systems programming language isn't C, e.g. IBM and Unisys mainframes. Embedded platforms that use their own toolchains without any kind of POSIX compatibility. And desktop OSes like Windows, which follows the tradition of C and C++ compilers outside the UNIX world, where each compiler vendor ships their own ISO compliant library, and the OS syscalls are completly unrelated, e.g. ZeroMemory() instead of memset(). Amiga, Atari, OS/2, BeOS, Symbian, Brew, Mac OS pre OS X, Newton, PalmOS, are all examples of this tradition.
Agree. Also threads are not part of the libc, but Posix: if 3-rd party libc provides C standard functions it does not necessarily provides pthread interface. In this regard, I want to propose related idea to move this dependent from libc and threads and etc parts into a separate druntime package ("backend") so that those who wish can themselves implement interfaces to these or another more appropriate OS calls. Something like: https://github.com/denizzzka/druntime/blob/non_posix/src/core/thread/osthread.d#L217 ===> https://github.com/denizzzka/d_c_arm_test/blob/master/d/freertos_druntime_backend/external/core/thread.d#L396 but without all these "version (DruntimeAbstractRt)" branches. This will simplify druntime main code (get rid of many versions branches inside of the druntime) and make it possible to easily implement support for various not common operating systems such as HaikuOS or druntime for bare-metal.
Nov 15 2020
parent Kagamin <spam here.lot> writes:
On Sunday, 15 November 2020 at 08:12:01 UTC, Denis Feklushkin 
wrote:
 Agree. Also threads are not part of the libc, but Posix: if 
 3-rd party libc provides C standard functions it does not 
 necessarily provides pthread interface.
It's catching on https://en.cppreference.com/w/c/thread/thrd_create
Nov 16 2020
prev sibling parent =?UTF-8?B?Um9iZXJ0IE0uIE3DvG5jaA==?= <robert.muench saphirion.com> writes:
On 12 Nov 2020 at 07:42:00 CET, "Jacob Carlborg" <doob me.com> wrote:

 The reason why it works for Go is because they have their own 
 full toolchain. That means, their own compiler (including 
 backend), their own assembler, their own object format and their 
 own linker. All of these are written to be cross-target.
And being x-target is a key design goal, which is really an amazing out-of-the-box experience. Of course you have some (silent) different behavior for some Go std lib stuff... which can be of surprise. The rust guys take a much more pedantic approach here, which IMO is the cleaner way to do it. However, Go's goal is getting stuff done. And it's really delivering on that. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Nov 12 2020