www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - TLS + LDC + Android (ARM) = FAIL

reply Igor Shirkalin <mathsoft inbox.ru> writes:
We solved the subject with modifying druntime source related with 
tls. Imaging, we have lost a lot of D's features.
As far as I know DMD or GDC are not available for ARM 
architecture. So we need LDC.
A short story: we have big C/C++ project that links D (LDC) code 
for different platforms.

Does new "-betterC" mean we may use parallelism with using 
separate linker?

- IS
Nov 01
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 1 November 2017 at 18:24, Igor Shirkalin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 We solved the subject with modifying druntime source related with tls.
 Imaging, we have lost a lot of D's features.
 As far as I know DMD or GDC are not available for ARM architecture. So we
 need LDC.
GDC supports the same or maybe more platforms than LDC. :-)
Nov 01
next sibling parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw wrote:
 On 1 November 2017 at 18:24, Igor Shirkalin via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 We solved the subject with modifying druntime source related 
 with tls.
 Imaging, we have lost a lot of D's features.
 As far as I know DMD or GDC are not available for ARM 
 architecture. So we
 need LDC.
GDC supports the same or maybe more platforms than LDC. :-)
Could you please give the reference how to build it in linux or windows?
Nov 01
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw wrote:
 GDC supports the same or maybe more platforms than LDC. :-)
Or quite possibly fewer, depending on what one understands "platform" and "support" to mean. ;) What is the state of GDC on Android/ARM – has anyone been using it recently? — David
Nov 01
parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 01 Nov 2017 17:42:22 +0000
schrieb David Nadlinger <code klickverbot.at>:

 On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw wrote:
 GDC supports the same or maybe more platforms than LDC. :-) =20
=20 Or quite possibly fewer, depending on what one understands=20 "platform" and "support" to mean. ;) =20 What is the state of GDC on Android/ARM =E2=80=93 has anyone been using=20 it recently? =20 =E2=80=94=C2=A0David =20
ARM: Fine. Android: probably won't work well. AFAIK we're only missing emulated TLS / GC integration, so most test will pass but real apps will crash because of GC memory corruption. I guess I should finally get back to fixing that problem ;-) OTOH Android doesn't even support GCC anymore, so I don't really see much benefit in maintaining GDC Android support. -- Johannes
Nov 01
next sibling parent reply Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau 
wrote:
 Am Wed, 01 Nov 2017 17:42:22 +0000
 schrieb David Nadlinger <code klickverbot.at>:

 On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw 
 wrote:
 [...]
Or quite possibly fewer, depending on what one understands "platform" and "support" to mean. ;) What is the state of GDC on Android/ARM – has anyone been using it recently? — David
ARM: Fine. Android: probably won't work well. AFAIK we're only missing emulated TLS / GC integration, so most test will pass but real apps will crash because of GC memory corruption. I guess I should finally get back to fixing that problem ;-) OTOH Android doesn't even support GCC anymore,
 so I don't really see much benefit in maintaining GDC Android 
 support.
 -- Johannes
That's too bad. I'd do it here for food. - Igor
Nov 01
parent Johannes Pfau <nospam example.com> writes:
Am Wed, 01 Nov 2017 18:32:37 +0000
schrieb Igor Shirkalin <mathsoft inbox.ru>:

 On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau=20
 wrote:
 Am Wed, 01 Nov 2017 17:42:22 +0000
 schrieb David Nadlinger <code klickverbot.at>:
 =20
 On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw=20
 wrote: =20
 [...] =20
=20 Or quite possibly fewer, depending on what one understands=20 "platform" and "support" to mean. ;) =20 What is the state of GDC on Android/ARM =E2=80=93 has anyone been=20 using it recently? =20 =E2=80=94=C2=A0David =20
ARM: Fine. Android: probably won't work well. AFAIK we're only=20 missing emulated TLS / GC integration, so most test will pass=20 but real apps will crash because of GC memory corruption. I=20 guess I should finally get back to fixing that problem ;-) OTOH=20 Android doesn't even support GCC anymore, =20
=20
 so I don't really see much benefit in maintaining GDC Android=20
 support.
 -- Johannes =20
=20 That's too bad. I'd do it here for food. =20 - Igor =20
I understand that D support for Android is important. I just think that if google now supports only LLVM for Android, focusing on a LLVM based compiler such as LDC is a more reasonable way to support Android. -- Johannes
Nov 01
prev sibling next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 1 November 2017 at 19:28, Johannes Pfau via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Am Wed, 01 Nov 2017 17:42:22 +0000
 schrieb David Nadlinger <code klickverbot.at>:

 On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw wrote:
 GDC supports the same or maybe more platforms than LDC. :-)
Or quite possibly fewer, depending on what one understands "platform" and "support" to mean. ;) What is the state of GDC on Android/ARM – has anyone been using it recently? — David
ARM: Fine. Android: probably won't work well. AFAIK we're only missing emulated TLS / GC integration, so most test will pass but real apps will crash because of GC memory corruption. I guess I should finally get back to fixing that problem ;-) OTOH Android doesn't even support GCC anymore, so I don't really see much benefit in maintaining GDC Android support. -- Johannes
What's the thread model on Android? You could perhaps get away with --enable-threads=single.
Nov 01
parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 18:33:45 UTC, Iain Buclaw wrote:
 On 1 November 2017 at 19:28, Johannes Pfau via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 Am Wed, 01 Nov 2017 17:42:22 +0000
 schrieb David Nadlinger <code klickverbot.at>:

[...]
ARM: Fine. Android: probably won't work well. AFAIK we're only missing emulated TLS / GC integration, so most test will pass but real apps will crash because of GC memory corruption. I guess I should finally get back to fixing that problem ;-) OTOH Android doesn't even support GCC anymore, so I don't really see much benefit in maintaining GDC Android support. -- Johannes
What's the thread model on Android? You could perhaps get away with --enable-threads=single.
I didn't pick this option up. We need multithreading.
Nov 01
prev sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau 
wrote:
 ARM: Fine. Android: probably won't work well. AFAIK we're only 
 missing emulated TLS / GC integration, so most test will pass 
 but real apps will crash because of GC memory corruption. I 
 guess I should finally get back to fixing that problem ;-) OTOH 
 Android doesn't even support GCC anymore, so I don't really see 
 much benefit in maintaining GDC Android support.
I don't see what their deciding to drop gcc has to do with whether gdc should support Android. On Wednesday, 1 November 2017 at 18:56:00 UTC, Igor Shirkalin wrote:
 Everything was beautiful for Win/Linux/iOS/Android.
 Some day we had to use LDC (for some obvious reasons). I marked 
 every function with  nogc and pure (it helped me to optimize 
 the code). The problem is TLS. Android doesn't support it. At 
 all. If you understand I talk about -betterC feature.
Android doesn't have native TLS, but __thread-annotated variables will work with clang: it uses emulated TLS. betterC is not going to save you, many parts of the standard library won't work with it. Please open an issue for ldc. In it, give us as much relevant detail as you can. Are you building the D component as a separate shared library or does it just get linked into a single binary with your C/C++ code? Are you making sure to initialize the D runtime using rt_init, as described here? https://wiki.dlang.org/Runtime_internals Have you followed the three linking rules for emulated TLS that I laid out in an earlier post? Put that info in an issue on the ldc github and we can help you out.
Nov 01
parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 01 Nov 2017 19:24:42 +0000
schrieb Joakim <dlang joakim.fea.st>:

 On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau 
 wrote:
 ARM: Fine. Android: probably won't work well. AFAIK we're only 
 missing emulated TLS / GC integration, so most test will pass 
 but real apps will crash because of GC memory corruption. I 
 guess I should finally get back to fixing that problem ;-) OTOH 
 Android doesn't even support GCC anymore, so I don't really see 
 much benefit in maintaining GDC Android support.  
I don't see what their deciding to drop gcc has to do with whether gdc should support Android.
If there's a backend bug in GCC related to Android nobody will take responsibility as it's not officially supported. All tutorials and documentation will focus on LLVM based compilers. It's certainly still interesting to use GDC for Android, but it is more work (especially considering Android is one of the few systems requiring emutls) for little benefit, especially if most users are going to use LLVM anyway. With the limited time available I think GDC should focus on systems where GCC is a first class or even the preferred compiler. X86/MIPS/ARM/PPC/Linux as some distributions such as debian might prefer a GCC based compiler. Then there are embedded toolchains which primarily use GCC: MSP and ARM (GCC Arm Embedded project). Also many console homebrew toolchains exclusively use GCC. I just don't think we have to support two compilers for any target with the little resources we have. -- Johannes
Nov 01
parent Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 21:50:09 UTC, Johannes Pfau 
wrote:
 Am Wed, 01 Nov 2017 19:24:42 +0000
 schrieb Joakim <dlang joakim.fea.st>:

 On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau 
 wrote:
 ARM: Fine. Android: probably won't work well. AFAIK we're 
 only missing emulated TLS / GC integration, so most test 
 will pass but real apps will crash because of GC memory 
 corruption. I guess I should finally get back to fixing that 
 problem ;-) OTOH Android doesn't even support GCC anymore, 
 so I don't really see much benefit in maintaining GDC 
 Android support.
I don't see what their deciding to drop gcc has to do with whether gdc should support Android.
If there's a backend bug in GCC related to Android nobody will take responsibility as it's not officially supported. All tutorials and documentation will focus on LLVM based compilers. It's certainly still interesting to use GDC for Android, but it is more work (especially considering Android is one of the few systems requiring emutls) for little benefit, especially if most users are going to use LLVM anyway. With the limited time available I think GDC should focus on systems where GCC is a first class or even the preferred compiler. X86/MIPS/ARM/PPC/Linux as some distributions such as debian might prefer a GCC based compiler. Then there are embedded toolchains which primarily use GCC: MSP and ARM (GCC Arm Embedded project). Also many console homebrew toolchains exclusively use GCC. I just don't think we have to support two compilers for any target with the little resources we have.
Here's the thing though: Android is basically just linux/ARM without native TLS and replacing glibc with the bionic C runtime. If you're going to support linux/ARM with glibc, it's not going to take much to support Android too, particularly since I upstreamed all my Bionic declarations and support into stock druntime. Far be it from me to tell you what to work on, and I realize getting emulated TLS working with the garbage-collector is a pain, but it won't take much more than emulated TLS for gdc to support Android. If you grep the llvm codebase for Android, you find almost nothing, basically just emulated TLS. I agree that we don't need two D compilers for every platform, but it'd be nice to have for the most widely deployed OS platform on the planet. ;)
Nov 01
prev sibling next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
wrote:
 We solved the subject with modifying druntime source related 
 with tls. Imaging, we have lost a lot of D's features.
You'd have been better off opening an issue for ldc: https://github.com/ldc-developers/ldc/issues TLS should work fine, though it's emulated, as Android doesn't support native TLS. You have to be careful how you link the emulated TLS, and you have to include a main function even for a shared library: https://wiki.dlang.org/Build_LDC_for_Android#Directions_for_future_work You'd be better off talking to the ldc devs- this is the first I'm hearing about this- rather than going in and making changes to druntime.
 As far as I know DMD or GDC are not available for ARM 
 architecture. So we need LDC.
 A short story: we have big C/C++ project that links D (LDC) 
 code for different platforms.

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?
You'll need to expand on this: you want to use std.parallelism, which isn't working for you now, or something else to do with the linker is going wrong?
Nov 01
parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 17:44:11 UTC, Joakim wrote:
 On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
 wrote:
 We solved the subject with modifying druntime source related 
 with tls. Imaging, we have lost a lot of D's features.
You'd have been better off opening an issue for ldc: https://github.com/ldc-developers/ldc/issues TLS should work fine, though it's emulated, as Android doesn't support native TLS. You have to be careful how you link the emulated TLS, and you have to include a main function even for a shared library: https://wiki.dlang.org/Build_LDC_for_Android#Directions_for_future_work You'd be better off talking to the ldc devs- this is the first I'm hearing about this- rather than going in and making changes to druntime.
 As far as I know DMD or GDC are not available for ARM 
 architecture. So we need LDC.
 A short story: we have big C/C++ project that links D (LDC) 
 code for different platforms.

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?
You'll need to expand on this: you want to use std.parallelism, which isn't working for you now, or something else to do with
We use external D libraries in C project. Using parallelisms means we have to initialize druntime. But tls stops it (therefore we had changed it). Sure, we have tried to build pure D part and it didnt work.
 the linker is going wrong?
of course the linker was working properly.
Nov 01
prev sibling next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
wrote:
 Does new "-betterC" mean we may use parallelism with using 
 separate linker?
`-betterC` does not add any emulation of missing platform features — on the contrary, it *removes* language runtime functionality! Thus, if TLS doesn't work for you (IIRC, emulated TLS should work on Android following Joakim's work!), adding `-betterC` won't improve the situation. Could you please open an ticket on the LDC GitHub tracker with details on the issue? — David
Nov 01
parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 17:46:29 UTC, David Nadlinger 
wrote:
 On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
 wrote:
 Does new "-betterC" mean we may use parallelism with using 
 separate linker?
`-betterC` does not add any emulation of missing platform features — on the contrary, it *removes* language runtime functionality! Thus, if TLS doesn't work for you (IIRC, emulated TLS should work on Android following Joakim's work!), adding `-betterC` won't improve the situation.
I undrastand that.
 Could you please open an ticket on the LDC GitHub tracker with 
 details on the issue?
I'd try with help.
  — David
Nov 01
prev sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
wrote:
 We solved the subject with modifying druntime source related 
 with tls. Imaging, we have lost a lot of D's features.
 As far as I know DMD or GDC are not available for ARM 
 architecture. So we need LDC.
 A short story: we have big C/C++ project that links D (LDC) 
 code for different platforms.

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?

 - IS
If you're having problems with the emulated TLS I put together for Android, it is most likely because I didn't document well what needs to be done when linking for Android. Specifically, there are three rules that _must_ be followed: 1. You must use the ld.bfd linker, ld.gold won't do. 2. You must have a D main function, even for a shared library (which can be put next to android_main, if you're using the default Android wrapper from my D android library). 3. The ELF object with the D main function must be passed to the linker first. If you look at my examples on the wiki, you'll see that they all follow these rules: https://wiki.dlang.org/Build_D_for_Android I should have called these rules out separately though, like I'm doing here, a documentation oversight.
Nov 01
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 01 Nov 2017 18:06:29 +0000
schrieb Joakim <dlang joakim.fea.st>:

 On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
 wrote:
 We solved the subject with modifying druntime source related 
 with tls. Imaging, we have lost a lot of D's features.
 As far as I know DMD or GDC are not available for ARM 
 architecture. So we need LDC.
 A short story: we have big C/C++ project that links D (LDC) 
 code for different platforms.

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?

 - IS  
If you're having problems with the emulated TLS I put together for Android, it is most likely because I didn't document well what needs to be done when linking for Android. Specifically, there are three rules that _must_ be followed: 1. You must use the ld.bfd linker, ld.gold won't do. 2. You must have a D main function, even for a shared library (which can be put next to android_main, if you're using the default Android wrapper from my D android library). 3. The ELF object with the D main function must be passed to the linker first. If you look at my examples on the wiki, you'll see that they all follow these rules: https://wiki.dlang.org/Build_D_for_Android I should have called these rules out separately though, like I'm doing here, a documentation oversight.
Also when mixing D and C code, you can't access extern TLS variables across the language boundary. Maybe OP tries to do that as he mixes D/C code? -- Johannes
Nov 01
parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 18:30:28 UTC, Johannes Pfau 
wrote:
 Am Wed, 01 Nov 2017 18:06:29 +0000
 schrieb Joakim <dlang joakim.fea.st>:

 On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
 wrote:
 [...]
If you're having problems with the emulated TLS I put together for Android, it is most likely because I didn't document well what needs to be done when linking for Android. Specifically, there are three rules that _must_ be followed: 1. You must use the ld.bfd linker, ld.gold won't do. 2. You must have a D main function, even for a shared library (which can be put next to android_main, if you're using the default Android wrapper from my D android library). 3. The ELF object with the D main function must be passed to the linker first. If you look at my examples on the wiki, you'll see that they all follow these rules: https://wiki.dlang.org/Build_D_for_Android I should have called these rules out separately though, like I'm doing here, a documentation oversight.
Also when mixing D and C code, you can't access extern TLS variables across the language boundary. Maybe OP tries to do that as he mixes D/C code? -- Johannes
Everything was beautiful for Win/Linux/iOS/Android. Some day we had to use LDC (for some obvious reasons). I marked every function with nogc and pure (it helped me to optimize the code). The problem is TLS. Android doesn't support it. At all. If you understand I talk about -betterC feature. - Igor
Nov 01
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 18:06:29 UTC, Joakim wrote:
 If you're having problems with the emulated TLS I put together 
 for Android, it is most likely because I didn't document well 
 what needs to be done when linking for Android.  Specifically, 
 there are three rules that _must_ be followed:

 1. You must use the ld.bfd linker, ld.gold won't do.
 2. You must have a D main function, even for a shared library 
 (which can be put next to android_main, if you're using the 
 default Android wrapper from my D android library).
 3. The ELF object with the D main function must be passed to 
 the linker first.

 If you look at my examples on the wiki, you'll see that they 
 all follow these rules:

 https://wiki.dlang.org/Build_D_for_Android

 I should have called these rules out separately though, like 
 I'm doing here, a documentation oversight.
I've added a new section to the Android wiki page with this info, and some more relevant details: https://wiki.dlang.org/Build_D_for_Android#Changes_for_Android
Nov 01