www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D with minimal runtime

reply Dibyendu Majumdar <d.majumdar gmail.com> writes:
Hi,

When writing compiler tools, a GC seems desirable as managing 
memory for the data structures used by compilers is a pain. I 
guess the better C option has no runtime, whereas normal D has 
the runtime with GC. I was wondering whether the runtime can be 
minimized to only support the GC. Or is that already the case, 
i.e. the D runtime is already the minimum needed for GC?

Thanks and Regards
Dibyendu
Jun 15
next sibling parent Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Monday, 15 June 2020 at 08:54:45 UTC, Dibyendu Majumdar wrote:
 When writing compiler tools, a GC seems desirable as managing 
 memory for the data structures used by compilers is a pain. I 
 guess the better C option has no runtime, whereas normal D has 
 the runtime with GC. I was wondering whether the runtime can be 
 minimized to only support the GC. Or is that already the case, 
 i.e. the D runtime is already the minimum needed for GC?
Just to clarify the context - the compiler needs to be built as a library that can be used in any project.
Jun 15
prev sibling next sibling parent reply IGotD- <nise nise.com> writes:
On Monday, 15 June 2020 at 08:54:45 UTC, Dibyendu Majumdar wrote:
 Hi,

 When writing compiler tools, a GC seems desirable as managing 
 memory for the data structures used by compilers is a pain. I 
 guess the better C option has no runtime, whereas normal D has 
 the runtime with GC. I was wondering whether the runtime can be 
 minimized to only support the GC. Or is that already the case, 
 i.e. the D runtime is already the minimum needed for GC?

 Thanks and Regards
 Dibyendu
C does almost require a standard library which is clib. You can't do memcpy or memset without clib so C becomes kind of dependable on clib. However, you can use C without clib as well and D can that too with the -betterC switch. I'm looking for a minimal druntime too. Right now it is all or nothing for druntime and it expect a full rich OS. There have been talks to partion druntime better so that you can incremental support, for example leave networking out if you want to. Also right now for all operating systems, druntime depends on clib itself. Nim has a compiler switch -os:any which means that the runtime will only depend on clib which makes it very portable. That would have been nice in druntime too.
Jun 15
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/15/2020 3:01 AM, IGotD- wrote:
 I'm looking for a minimal druntime too. Right now it is all or nothing for 
 druntime and it expect a full rich OS. There have been talks to partion
druntime 
 better so that you can incremental support, for example leave networking out
if 
 you want to. Also right now for all operating systems, druntime depends on
clib 
 itself. Nim has a compiler switch -os:any which means that the runtime will
only 
 depend on clib which makes it very portable. That would have been nice in 
 druntime too.
Since druntime is a library, only the functions actually referenced are pulled out of the library when linking.
Jun 15
parent reply IGotD- <nise nise.com> writes:
On Monday, 15 June 2020 at 10:07:10 UTC, Walter Bright wrote:
 Since druntime is a library, only the functions actually 
 referenced are pulled out of the library when linking.
Still you need to compile druntime and if it finds any missing symbol there, the build will fail. Also the in druntime files often include other files in druntime which doesn't make it that easy. As we have discussed before, the best way for this in my opinion is to have an abstract interface in druntime for the OS functions. This way we could stub those functions that the programmer doesn't need.
Jun 15
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/15/2020 3:16 AM, IGotD- wrote:
 As we have discussed before, the best way for this in my opinion is to have an 
 abstract interface in druntime for the OS functions. This way we could stub 
 those functions that the programmer doesn't need.
If you include the line: extern (C) void printf() { } somewhere in your code, printf() will *not* get linked in. An extra interface layer is not necessary.
Jun 15
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/20 4:50 PM, Walter Bright wrote:
 On 6/15/2020 3:16 AM, IGotD- wrote:
 As we have discussed before, the best way for this in my opinion is to 
 have an abstract interface in druntime for the OS functions. This way 
 we could stub those functions that the programmer doesn't need.
If you include the line:    extern (C) void printf() { } somewhere in your code, printf() will *not* get linked in. An extra interface layer is not necessary.
Word. Any work on reducing druntime needs to be based on a good understanding of how linking works.
Jun 15
parent reply dangbinghoo <dangbinghoo google-mail.com> writes:
On Tuesday, 16 June 2020 at 01:16:27 UTC, Andrei Alexandrescu 
wrote:
 On 6/15/20 4:50 PM, Walter Bright wrote:
 On 6/15/2020 3:16 AM, IGotD- wrote:
 As we have discussed before, the best way for this in my 
 opinion is to have an abstract interface in druntime for the 
 OS functions. This way we could stub those functions that the 
 programmer doesn't need.
If you include the line:    extern (C) void printf() { } somewhere in your code, printf() will *not* get linked in. An extra interface layer is not necessary.
Word. Any work on reducing druntime needs to be based on a good understanding of how linking works.
hi, many pepole is requiring that the D runtime needs to be minimal. Yes, this "is a linking problem" just if you're always using "normal" or "rich" OS, things matter if you trying to using D in the microcontroller target such as STM32F4 or something similar, these platfrom has hundreds of KB ram and maybe 1MB of flash memory, it's powerfull enogh for running the D language. if you trying to build a betterC programing on these platforms(only RTOS or no OS at all), it shows that D runtime is requring some thing that only Rich-OS have. the rust and zig programming language is doing better in this situation, the D runtime can be made only need to requiring ANSI C stdlib (like nim 'any') and then the D runtime is more portable. thanks! ---- binghoo dang
Jun 16
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/16/20 10:09 PM, dangbinghoo wrote:
 the D runtime can be made only need to requiring ANSI C stdlib (like nim 
 'any') and then the D runtime is more portable.
That would be nice, and possible if most parts of druntime would be templated and others would only be pulled in if used. It boils down to finding the talented folks to make it happen.
Jun 16
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 17 June 2020 at 02:29:44 UTC, Andrei Alexandrescu 
wrote:
 possible if most parts of druntime would be templated and 
 others would only be pulled in if used.
I actually don't think it is even necessary to template it. If you had independent modules written with minimal dependencies, there's several things that make it work automatically: dmd -i can compile as-needed. Linkers can leave things out of exes from static libs. Etc. My arsd package as like 78 modules now. But I'm strict about them not importing each other unless absolutely necessary. You can use most of them individually without caring about the other 77 (but if you do reference it, it gets auto-imported, e.g. Document.fromUrl brings in http... but if you never call that, no need for the http module at all in your build) There's been some decent work in dmd to support this for language features too, but from the other direction. If a feature isn't present in object.d, it won't reference it. That little glue layer might be templatized so it is available, but not used unless needed... then from there it just imports a traditional module. Pretty sure good chunks of that would work. druntime and phobos too have problems of intertwined imports that would take a lot of work to clean up though.
Jun 16
prev sibling parent reply IGotD- <nise nise.com> writes:
On Wednesday, 17 June 2020 at 02:09:47 UTC, dangbinghoo wrote:
 many pepole is requiring that the D runtime needs to be 
 minimal. Yes, this "is a linking problem" just if you're always 
 using "normal" or "rich" OS, things matter if you trying to 
 using D in the microcontroller target such as STM32F4 or 
 something similar, these platfrom has hundreds of KB ram and 
 maybe 1MB of flash memory, it's powerfull enogh for running the 
 D language.

 if you trying to build a betterC programing on these 
 platforms(only RTOS or no OS at all), it shows that D runtime 
 is requring some thing that only Rich-OS have.

 the rust and zig programming language is doing better in this 
 situation, the D runtime can be made only need to requiring 
 ANSI C stdlib (like nim 'any') and then the D runtime is more 
 portable.


 thanks!
 ----
 binghoo dang
Indeed and it is not only about "not linking in what you don't need" as the druntime expects a lot of things from the OS. This includes reading elf section information, which shouldn't be needed in a minimal system as we don't even know if the programmer wants to use elf/pe. Maybe the programmer just want to use a binary blob. Typically RTOS in small platforms like STM32F4 or similar often don't implement TLS. For these kind of systems it would be nice if druntime didn't have any TLS variables at all and then the programmer must promise not to use any TLS variables. Essentially where you obtain the TLS area for each thread must be stubbed and just report empty TLS area. Full D isn't really adapted for limited OS/RTOS today and it has painted itself into a corner by implementing a lot of expected OS support in the druntime. There isn't any wrong with it, it's just that D isn't a systems programming language and the question is if D should go that route. I'd say that D missed that train and D shouldn't peruse that any further.
Jun 17
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Wednesday, 17 June 2020 at 10:39:38 UTC, IGotD- wrote:
 Typically RTOS in small platforms like STM32F4 or similar often 
 don't implement TLS.
Don't worry - TLS implementation is trivial for each platform.
Jun 17
parent IGotD- <nise nise.com> writes:
On Thursday, 18 June 2020 at 05:50:38 UTC, Denis Feklushkin wrote:
 Don't worry - TLS implementation is trivial for each platform.
For a static system which doesn't change, then yes it might be trivial. For a system that allows runtime loaded shared libraries, then no.
Jun 18
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/20 6:01 AM, IGotD- wrote:
 I'm looking for a minimal druntime too. Right now it is all or nothing 
 for druntime and it expect a full rich OS. There have been talks to 
 partion druntime better so that you can incremental support, for example 
 leave networking out if you want to. Also right now for all operating 
 systems, druntime depends on clib itself. Nim has a compiler switch 
 -os:any which means that the runtime will only depend on clib which 
 makes it very portable. That would have been nice in druntime too.
We really need to merge druntime and phobos in one. The entire distinction has long become nonsensical. The new merged library ("hall" in honor of Asaph Hall who discovered Phobos and Deimos) would be built under a strict pay-as-you-go regime. That means an empty main uses virtually nothing, a main with a writeln() uses a little I/O, a main that allocates memory brings the GC in play (which auto-initializes on the first call), etc. Templates and deduction must be heavily favored over old crummy C bindings. E.g. writeln() should actually link little else than a call to the fwrite() function. We'd need at least 2 solid engineers for that.
Jun 15
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 16 June 2020 at 01:08:16 UTC, Andrei Alexandrescu 
wrote:
 The new merged library ("hall" in honor of Asaph Hall who 
 discovered Phobos and Deimos) would be built under a strict 
 pay-as-you-go regime. That means an empty main uses virtually 
 nothing, a main with a writeln() uses a little I/O, a main that 
 allocates memory brings the GC in play (which auto-initializes 
 on the first call), etc.
How would that interact with architecture-based constraints (e.g. issues porting libraries in their entirety because of architecture-specific limitations, missing assembly implementations, etc.)? Is it possible to ensure there is a well-defined "pay for what you can" separation of concerns, as well as "pay-as-you-go"? P.S. I think you should call the library Asaph, it's more distinctive ;-)
Jun 18
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/18/20 8:10 AM, Joseph Rushton Wakeling wrote:
 On Tuesday, 16 June 2020 at 01:08:16 UTC, Andrei Alexandrescu wrote:
 The new merged library ("hall" in honor of Asaph Hall who discovered 
 Phobos and Deimos) would be built under a strict pay-as-you-go regime. 
 That means an empty main uses virtually nothing, a main with a 
 writeln() uses a little I/O, a main that allocates memory brings the 
 GC in play (which auto-initializes on the first call), etc.
How would that interact with architecture-based constraints (e.g. issues porting libraries in their entirety because of architecture-specific limitations, missing assembly implementations, etc.)?
I assume porting a pay-as-you-go library would be easier to port than a monolithic one.
 Is it possible to ensure there is a well-defined "pay for what you can" 
 separation of concerns, as well as "pay-as-you-go"?
I don't know. That would be for the crack team that hasn't been assembled yet :o). What I can tell is that in a large project, whatever people pay attention to will work well. Whatever people don't pay attention to will work poorly. For example: in Phobos I paid little attention to things like object file size, linking in code, mutual dependencies, or separating memory allocation from other work. These aspects did not evolve well.
 P.S. I think you should call the library Asaph, it's more distinctive ;-)
Another decision for them!
Jun 18
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Thursday, 18 June 2020 at 13:17:14 UTC, Andrei Alexandrescu 
wrote:
 I don't know. That would be for the crack team that hasn't been 
 assembled yet :o)
There are times I rather regret that I'm in a very busy job with lots to do ... :-)
 P.S. I think you should call the library Asaph, it's more 
 distinctive ;-)
Another decision for them!
Sure, but it's the RIGHT decision ;-)
Jun 18
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 15 June 2020 at 08:54:45 UTC, Dibyendu Majumdar wrote:
 I was wondering whether the runtime can be minimized to only 
 support the GC. Or is that already the case, i.e. the D runtime 
 is already the minimum needed for GC?
Eh, a good bulk of the d runtime is GC support, it would be hard to pull the gc out without most the rest of it. In general, if you create a file called "object.d" in your work directory and compile it in, you can override druntime to nothing and build it back up from there just... that's a pain to get from nothing back up to GC point.
Jun 15
next sibling parent reply IGotD- <nise nise.com> writes:
On Monday, 15 June 2020 at 12:21:08 UTC, Adam D. Ruppe wrote:
 Eh, a good bulk of the d runtime is GC support, it would be 
 hard to pull the gc out without most the rest of it.
I think he means the opposite, remove everything but GC. You bascially need GC in order to get the benefits of D but you don't necessarily need everything else.
Jun 15
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 15 June 2020 at 16:45:21 UTC, IGotD- wrote:
 I think he means the opposite, remove everything but GC.
That's exactly what I meant... just I awkwardly worded it. I meant pull the GC out of full druntime and drop it into your minimal runtime so you have just it without the other stuff.
 You bascially need GC in order to get the benefits of D but you 
 don't necessarily need everything else.
eh GC is a big win with D, but you can get lots of D benefits without it too.
Jun 15
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/20 8:21 AM, Adam D. Ruppe wrote:
 On Monday, 15 June 2020 at 08:54:45 UTC, Dibyendu Majumdar wrote:
 I was wondering whether the runtime can be minimized to only support 
 the GC. Or is that already the case, i.e. the D runtime is already the 
 minimum needed for GC?
Eh, a good bulk of the d runtime is GC support, it would be hard to pull the gc out without most the rest of it.
How difficult would be to arrange things such that absolutely nothing in the GC makes it in the final ninary if not used? My recollection is that we currently have a layering on top of the GC that is based on a misunderstanding of how linkers work, and also awkward and unnecessary. Cutting that off and putting some good work into a self-loading GC (linked in if and only if used) would be amazing work.
Jun 15
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 16 June 2020 at 01:12:10 UTC, Andrei Alexandrescu 
wrote:
 How difficult would be to arrange things such that absolutely 
 nothing in the GC makes it in the final ninary if not used?
I don't know... I haven't looked at it all recently. I know the GC is used in part of the program initialization right now though (as are threads and other things) so that'd probably all have to change.
 My recollection is that we currently have a layering on top of 
 the GC that is based on a misunderstanding of how linkers work, 
 and also awkward and unnecessary.
the gc proxy thing is a mess yah.
Jun 16
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/16/20 10:47 PM, Adam D. Ruppe wrote:
 On Tuesday, 16 June 2020 at 01:12:10 UTC, Andrei Alexandrescu wrote:
 How difficult would be to arrange things such that absolutely nothing 
 in the GC makes it in the final ninary if not used?
I don't know... I haven't looked at it all recently. I know the GC is used in part of the program initialization right now though (as are threads and other things) so that'd probably all have to change.
There was some work done to remove shared static this() for the GC.
Jun 16
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jun 17, 2020 at 02:47:50AM +0000, Adam D. Ruppe via Digitalmars-d wrote:
 On Tuesday, 16 June 2020 at 01:12:10 UTC, Andrei Alexandrescu wrote:
 How difficult would be to arrange things such that absolutely
 nothing in the GC makes it in the final ninary if not used?
I don't know... I haven't looked at it all recently. I know the GC is used in part of the program initialization right now though (as are threads and other things) so that'd probably all have to change.
[...] As of recent releases, GC initialization is actually lazy now, it will not initialize anything until you actually use it. Still one step away from being totally optional, granted, but it shouldn't be *too* difficult to make it so that, e.g., the GC code isn't even linked unless there's a reference to one of its symbols somewhere. T -- You have to expect the unexpected. -- RL
Jun 16
parent reply Jacob Carlborg <doob me.com> writes:
On 2020-06-17 06:08, H. S. Teoh wrote:

 As of recent releases, GC initialization is actually lazy now, it will
 not initialize anything until you actually use it.  Still one step away
 from being totally optional, granted, but it shouldn't be *too*
 difficult to make it so that, e.g., the GC code isn't even linked unless
 there's a reference to one of its symbols somewhere.
How would the GC know if it needs to initialize itself if you allocate the object at compile time, i.e.: __gshared a = new Object; // module scope Or is it expected that the above object fill never be freed? What about the TLS data, isn't that used for the GC? This data is retrieved eagerly when a new thread is started. But since you'll always have at least one thread the data will always be retrieved? -- /Jacob Carlborg
Jun 17
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/17/20 5:06 AM, Jacob Carlborg wrote:
 On 2020-06-17 06:08, H. S. Teoh wrote:
 
 As of recent releases, GC initialization is actually lazy now, it will
 not initialize anything until you actually use it.  Still one step away
 from being totally optional, granted, but it shouldn't be *too*
 difficult to make it so that, e.g., the GC code isn't even linked unless
 there's a reference to one of its symbols somewhere.
How would the GC know if it needs to initialize itself if you allocate the object at compile time, i.e.: __gshared a = new Object; // module scope
Wouldn't that be allocated during compilation, and frozen at runtime (i.e. essentially static data)? That's my recollection of what happens.
 Or is it expected that the above object fill never be freed?
I don't think it's ever allocated actually.
Jun 17
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 17 June 2020 at 12:48:03 UTC, Andrei Alexandrescu 
wrote:
 Wouldn't that be allocated during compilation, and frozen at 
 runtime (i.e. essentially static data)?
Yeah. You can modify the object (it is not const), but its memory address is statically allocated and will never change / be freed, even if you overwrite all the references. It is part of the exe file. (this fact is actually useful for many things but it can be confusing)
Jun 17
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 17 Jun 2020 11:06:26 +0200 schrieb Jacob Carlborg:

 What about the TLS data, isn't that used for the GC? This data is
 retrieved eagerly when a new thread is started. But since you'll always
 have at least one thread the data will always be retrieved?
TLS memory is scanned by the GC, but it is managed / freed by the C library when a thread is stopped. Similar for static memory, which is as far as I know never freed. So as long as you don't have any heap memory, there's really no GC work to do. -- Johannes
Jun 18
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 17 Jun 2020 02:47:50 +0000 schrieb Adam D. Ruppe:

 On Tuesday, 16 June 2020 at 01:12:10 UTC, Andrei Alexandrescu wrote:
 
 My recollection is that we currently have a layering on top of the GC
 that is based on a misunderstanding of how linkers work,
 and also awkward and unnecessary.
the gc proxy thing is a mess yah.
Isn't the GC proxy only used for windows DLL support? IIRC we need it as we can't build druntime as a DLL right now. -- Johannes
Jun 18