www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - druntime redesign

reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
Hi!

This post is inspired by: 
https://forum.dlang.org/thread/scklbdrgjiypknuephfj forum.dlang.org

On Saturday, 22 February 2020 at 13:20:40 UTC, IGotD- wrote:
 Will there be a future improvement of this so that OS specific 
 support can be moved to separate files and selected on module 
 level instead? For example.

 void DSpecificLibraryFunction(...)
 {
     OSDependentLibraryFunction(...)
 }

 so that there are predefined OS specific functions that are 
 selected on module level. Also version() switches where 
 everything is consolidated into one file makes it more 
 difficult to merge.
More specific proposal how to achieve this step by step: Сommunity may agree to add druntime module "core.sys.abstract" and everywhere in druntime code what references to a specific OSes and architecures (all these thousands of "version(){} else version(){}") will add also obtain code like: version(Abstract_Platform){ core.sys.abstract.some_call(); } This "core.sys.abstract" code for every exotic uncommon (or just "every"?) platform will be placed outside of druntime "src" dir and will be used for compilation only if appropriate compiler flag "-I" is set and version=Abstract_Platform is defined. Then some uncommon platforms can be moved to "abstract" implementation (for purpose of code coverage for this new "abstract subsystem"). And at next step maybe every platforms can be implemented as abstract (and "core.sys.abstract" will be renamed into "core.sys"). For me it looks like it can be done gradually and at the same time not to break anything. How do you like this idea? Do you see missings/flaws in it?
Feb 24
next sibling parent reply IGotD- <nise nise.com> writes:
On Tuesday, 25 February 2020 at 00:05:05 UTC, Denis Feklushkin 
wrote:
 And at next step maybe every platforms can be implemented as 
 abstract (and "core.sys.abstract" will be renamed into 
 "core.sys").

 For me it looks like it can be done gradually and at the same 
 time not to break anything.

 How do you like this idea? Do you see missings/flaws in it?
This really opens up a second question, with your proposal with an abstract interface we can start to reduce the dependency on the C library. The C library is just there as baggage in order to give basic OS functionality. With an abstract interface we can use the OS primitives directly without going through any C library. This is a huge task but this offers a gradual approach. We can have the C library as an option through the abstract interface if we still need it. I see a double benefit of this.
Feb 24
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 25 February 2020 at 00:22:19 UTC, IGotD- wrote:

 This really opens up a second question, with your proposal with 
 an abstract interface we can start to reduce the dependency on 
 the C library. The C library is just there as baggage in order 
 to give basic OS functionality. With an abstract interface we 
 can use the OS primitives directly without going through any C 
 library. This is a huge task but this offers a gradual approach.

 We can have the C library as an option through the abstract 
 interface if we still need it.
Yes! Perhaps need to introduce option DRUNTIME_LIBC_AVAILABLE for druntime so that we can gradually get rid of all libc's symbols like "errno" inside of new abstract implementations by wrapping some druntime code pieces into version() branches.
Feb 24
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 25 February 2020 at 01:21:33 UTC, Denis Feklushkin 
wrote:
 More specific proposal how to achieve this step by step:
 Сommunity may agree to add druntime module "core.sys.abstract" 
 and everywhere in druntime code what references to a specific 
 OSes and architecures (all these thousands of "version(){} else 
 version(){}") will add also obtain code like
Probably I seem to know easiest way: Many modules are scattered by version() keywords that implement common calls necessary when using any platform. For brief example look into: core/sync/event.d We can improve Makefiles (and cmake in case of LDC fork of druntime) to include only needed platform-dependend subdirs what will contain these modules but without "versions", just code for each supported platform. (Do not confuse with core.sys.platform_name section!) (And I couldn’t do it myself right now because I don’t know well enough make and cmake systems. I am need "initial sample template" to do things like in it.) After such replacement it will be very easy to clear out remaining libc calls (if it will be need - all supported platforms actually have libc interfaces), and also support for new arbitrary platforms will be ready - for supporting any new platform will be enough to replace platform-specific section with your own at build stage by build system. What do you think about this? Is that good idea?
Mar 15
next sibling parent reply IGotD- <nise nise.com> writes:
On Sunday, 15 March 2020 at 10:29:11 UTC, Denis Feklushkin wrote:
 On Tuesday, 25 February 2020 at 01:21:33 UTC, Denis Feklushkin 
 wrote:

 Probably I seem to know easiest way:

 Many modules are scattered by version() keywords that implement 
 common calls necessary when using any platform.
 For brief example look into: core/sync/event.d

 We can improve Makefiles (and cmake in case of LDC fork of 
 druntime) to include only needed platform-dependend subdirs 
 what will contain these modules but without "versions", just 
 code for each supported platform.
 (Do not confuse with core.sys.platform_name section!)

 (And I couldn’t do it myself right now because I don’t know 
 well enough make and cmake systems. I am need "initial sample 
 template" to do things like in it.)

 After such replacement it will be very easy to clear out 
 remaining libc calls (if it will be need - all supported 
 platforms actually have libc interfaces), and also support for 
 new arbitrary platforms will be ready - for supporting any new 
 platform will be enough to replace platform-specific section 
 with your own at build stage by build system.

 What do you think about this? Is that good idea?
If take src/core/sync/semaphore.d (class Semaphore) for example this is an example with a lot of version switches that could be a good candidate for an abstract interface. I'm not sure what type of abstraction it should be, inherited SemaphoreImpl class, mixin member of a name given by OS module, fixed member name? There is alot of confusion with the OS and clib as OS support functionality is mixed together with clib functionality in the code. This is very confusing. Instead if an abstract OS want to use clib, it should do so separately. For example Windows should use Microsoft Clib, Linux could use a variety of clib like glibc, musl etc. Clib support should be under OS support and not mixed into the generic code. Another thing I've noticed is how elf code is duplicated for each OS. As elf is a standard we don't need to duplicate this and we can have a generic solution instead. We should have an ABI for this as well. First I think we need to determine what type of "polymorphism" to use ie. how to select the OS specific implementation, then start to implement an ABI. Much of the ABI is straight forward I think. BTW, isn't it time to retire make files and go for cmake only? Perhaps OT, I should start a new thread.
Mar 15
parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Sunday, 15 March 2020 at 12:08:54 UTC, IGotD- wrote:
 On Sunday, 15 March 2020 at 10:29:11 UTC, Denis Feklushkin
 What do you think about this? Is that good idea?
If take src/core/sync/semaphore.d (class Semaphore) for example this is an example with a lot of version switches that could be a good candidate for an abstract interface. I'm not sure what type of abstraction it should be, inherited SemaphoreImpl class, mixin member of a name given by OS module, fixed member name?
It’s definitely not a class because in this case it is not required to provide polymorphism - there is only one platform will be presented in result binary. (If someone occasionally wants to make a multi-platform binaries it will still be more difficult than just several implementations of these base class.) Just fixed member is preferred, as for me.
 There is alot of confusion with the OS and clib as OS support 
 functionality is mixed together with clib functionality in the 
 code. This is very confusing. Instead if an abstract OS want to 
 use clib, it should do so separately. For example Windows 
 should use Microsoft Clib, Linux could use a variety of clib 
 like glibc, musl etc. Clib support should be under OS support 
 and not mixed into the generic code.
My investigation showed me that there is not so much of this libc code. I even think that it will be possible to introduce temporary abstract libc version(CRuntime_AbstractLibc) to cover this case.
 Another thing I've noticed is how elf code is duplicated for 
 each OS. As elf is a standard we don't need to duplicate this 
 and we can have a generic solution instead. We should have an 
 ABI for this as well.
And, probably, not all platforms need ELF support in druntime.
 First I think we need to determine what type of "polymorphism" 
 to use ie. how to select the OS specific implementation, then 
 start to implement an ABI. Much of the ABI is straight forward 
 I think.
Just improve makefiles to be able select needed subdirectories like in posix.mak/win64.mak and LDC's druntime CMakeLists.txt: file(GLOB_RECURSE DRUNTIME_D_BIONIC ${RUNTIME_DIR}/src/core/sys/bionic/*.d) file(GLOB_RECURSE DRUNTIME_D_DARWIN ${RUNTIME_DIR}/src/core/sys/darwin/*.d) file(GLOB_RECURSE DRUNTIME_D_DRAGONFLYBSD ${RUNTIME_DIR}/src/core/sys/dragonflybsd/*.d) file(GLOB_RECURSE DRUNTIME_D_FREEBSD ${RUNTIME_DIR}/src/core/sys/freebsd/*.d) file(GLOB_RECURSE DRUNTIME_D_LINUX ${RUNTIME_DIR}/src/core/sys/linux/*.d) file(GLOB_RECURSE DRUNTIME_D_NETBSD ${RUNTIME_DIR}/src/core/sys/netbsd/*.d) file(GLOB_RECURSE DRUNTIME_D_OPENBSD ${RUNTIME_DIR}/src/core/sys/openbsd/*.d) file(GLOB_RECURSE DRUNTIME_D_POSIX ${RUNTIME_DIR}/src/core/sys/posix/*.d) file(GLOB_RECURSE DRUNTIME_D_SOLARIS ${RUNTIME_DIR}/src/core/sys/solaris/*.d) file(GLOB_RECURSE DRUNTIME_D_WINDOWS ${RUNTIME_DIR}/src/core/sys/windows/*.d)
 BTW, isn't it time to retire make files and go for cmake only? 
 Perhaps OT, I should start a new thread.
Do you know a mind state when you recognize a new tool and it starts to seem perfect and you want to use it everywhere and always? So for me now this tool is Meson build system. Therefore, I am biased. Maybe we should consider switching to it. It is ideal for D in terms of the ability to simplified link programs written in various languages. In fact, I'm not going to use druntime with existing built-in make/cmake scripts in my Dlang MCU projects because it are too complex and it is very difficult to configure to support dozens of all kinds of options that microcontroller binaries need.
Mar 15
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Sunday, 15 March 2020 at 10:29:11 UTC, Denis Feklushkin wrote:
 What do you think about this? Is that good idea?
Sounds like what Walter wanted to do in the first place: https://github.com/dlang/druntime/pull/2616#issuecomment-496312793 https://github.com/dlang/druntime/pull/2616#issuecomment-503503846
Mar 15
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Sunday, 15 March 2020 at 16:31:35 UTC, Vladimir Panteleev 
wrote:
 On Sunday, 15 March 2020 at 10:29:11 UTC, Denis Feklushkin 
 wrote:
 What do you think about this? Is that good idea?
Sounds like what Walter wanted to do in the first place: https://github.com/dlang/druntime/pull/2616#issuecomment-496312793 https://github.com/dlang/druntime/pull/2616#issuecomment-503503846
Yes, it seems like that. Just in case, to exclude discrepancies, I detail purpose of my proposal: My proposal is about exclude any version() from platform-independent code. It is too hard to maintain all versions in one module file because it is too many types of platforms is available. (Really, I think we have near of infinity number of them, if we treat as version various sets of types aliases, etc.)
Mar 15
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Sunday, 15 March 2020 at 18:05:50 UTC, Denis Feklushkin wrote:
 On Sunday, 15 March 2020 at 16:31:35 UTC, Vladimir Panteleev 
 wrote:
 On Sunday, 15 March 2020 at 10:29:11 UTC, Denis Feklushkin 
 wrote:
 What do you think about this? Is that good idea?
https://github.com/dlang/druntime/pull/2994
Mar 16
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Monday, 16 March 2020 at 22:30:05 UTC, Denis Feklushkin wrote:
 On Sunday, 15 March 2020 at 18:05:50 UTC, Denis Feklushkin 
 wrote:
 On Sunday, 15 March 2020 at 16:31:35 UTC, Vladimir Panteleev 
 wrote:
 On Sunday, 15 March 2020 at 10:29:11 UTC, Denis Feklushkin 
 wrote:
 What do you think about this? Is that good idea?
https://github.com/dlang/druntime/pull/2994
Problem: It is need to set default import directory depending on the platform. (Like currently compiler does default import for Phobos.) Otherwise Phobos and all other software isn't builds because platform specific code isn't imported.
Mar 16
parent reply Johan <j j.nl> writes:
On Tuesday, 17 March 2020 at 03:08:41 UTC, Denis Feklushkin wrote:
 On Monday, 16 March 2020 at 22:30:05 UTC, Denis Feklushkin 
 wrote:
 On Sunday, 15 March 2020 at 18:05:50 UTC, Denis Feklushkin 
 wrote:
 On Sunday, 15 March 2020 at 16:31:35 UTC, Vladimir Panteleev 
 wrote:
 On Sunday, 15 March 2020 at 10:29:11 UTC, Denis Feklushkin 
 wrote:
 What do you think about this? Is that good idea?
https://github.com/dlang/druntime/pull/2994
Problem: It is need to set default import directory depending on the platform. (Like currently compiler does default import for Phobos.) Otherwise Phobos and all other software isn't builds because platform specific code isn't imported.
Rather than setting the import directory, I think a much nicer solution is something like this: https://github.com/weka-io/mecca/blob/master/src/mecca/platform/os/package.d This works for different OS types, different CPU types, etc. And it removes the need to specify a whole list of specific import directories for each OS/cpu combination in the config file (please keep LDC crosscompilation in mind). -Johan
Mar 17
parent reply IGotD- <nise nise.com> writes:
On Tuesday, 17 March 2020 at 16:25:22 UTC, Johan wrote:
 Rather than setting the import directory, I think a much nicer 
 solution is something like this:
 https://github.com/weka-io/mecca/blob/master/src/mecca/platform/os/package.d
 This works for different OS types, different CPU types, etc. 
 And it removes the need to specify a whole list of specific 
 import directories for each OS/cpu combination in the config 
 file (please keep LDC crosscompilation in mind).

 -Johan
How is this different from using version?
Mar 17
next sibling parent Johan <j j.nl> writes:
On Tuesday, 17 March 2020 at 17:12:39 UTC, IGotD- wrote:
 On Tuesday, 17 March 2020 at 16:25:22 UTC, Johan wrote:
 Rather than setting the import directory, I think a much nicer 
 solution is something like this:
 https://github.com/weka-io/mecca/blob/master/src/mecca/platform/os/package.d
 This works for different OS types, different CPU types, etc. 
 And it removes the need to specify a whole list of specific 
 import directories for each OS/cpu combination in the config 
 file (please keep LDC crosscompilation in mind).

 -Johan
How is this different from using version?
It isn't. But it puts `version` in only a few places, instead of spread across many different files. We shouldn't be replacing version, but utilize it. -Johan
Mar 17
prev sibling parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 17 March 2020 at 17:12:39 UTC, IGotD- wrote:
 On Tuesday, 17 March 2020 at 16:25:22 UTC, Johan wrote:
 Rather than setting the import directory, I think a much nicer 
 solution is something like this:
Yes, already implemented in new PR: https://github.com/dlang/druntime/pull/2997
 https://github.com/weka-io/mecca/blob/master/src/mecca/platform/os/package.d
 This works for different OS types, different CPU types, etc. 
 And it removes the need to specify a whole list of specific 
 import directories for each OS/cpu combination in the config 
 file (please keep LDC crosscompilation in mind).

 -Johan
How is this different from using version?
I used version (CRuntime_Abstract) to import platform dependent code. This strategy looks good, but again there is a problem that is not directly related to the code: changing one module required changing only 8 lines in the code, but 15 lines in build scripts. And this is not all of needed changes - LDC uses its own cmake build scripts. And each module will require approximately same changes. It will be like a disaster. Thus, at first stage we need to clear up build scripts.
Mar 17
next sibling parent reply IGotD- <nise nise.com> writes:
On Tuesday, 17 March 2020 at 17:38:41 UTC, Denis Feklushkin wrote:
 I used version (CRuntime_Abstract) to import platform dependent 
 code.

 This strategy looks good, but again there is a problem that is 
 not directly related to the code: changing one module required 
 changing only 8 lines in the code, but 15 lines in build 
 scripts. And this is not all of needed changes - LDC uses its 
 own cmake build scripts.

 And each module will require approximately same changes. It 
 will be like a disaster. Thus, at first stage we need to clear 
 up build scripts.
This is where something like import core.sys.<compiletime identifier>.semaphore; would be an absolute great help.
Mar 17
next sibling parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 17 March 2020 at 19:34:45 UTC, IGotD- wrote:

 And each module will require approximately same changes. It 
 will be like a disaster. Thus, at first stage we need to clear 
 up build scripts.
This is where something like import core.sys.<compiletime identifier>.semaphore; would be an absolute great help.
This is not fundamental difference with version()
Mar 17
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-03-17 20:34, IGotD- wrote:

 This is where something like
 
 import core.sys.<compiletime identifier>.semaphore;
 
 would be an absolute great help.
Can be easily handled with a string mixin and a utility function: version (linux) enum platform = "linux"; else version (Windows) enum platform = "windows"; string import_(string[] imports ...) { string result; foreach (i, imp; imports) { if (i != 0) result ~= '.'; result ~= imp; } return "import " ~ result ~ ';'; } mixin(import_("core", "sys", platform, "semaphore")); -- /Jacob Carlborg
Mar 17
next sibling parent IGotD- <nise nise.com> writes:
On Tuesday, 17 March 2020 at 20:37:04 UTC, Jacob Carlborg wrote:
 On 2020-03-17 20:34, IGotD- wrote:

 This is where something like
 
 import core.sys.<compiletime identifier>.semaphore;
 
 would be an absolute great help.
Can be easily handled with a string mixin and a utility function: version (linux) enum platform = "linux"; else version (Windows) enum platform = "windows"; string import_(string[] imports ...) { string result; foreach (i, imp; imports) { if (i != 0) result ~= '.'; result ~= imp; } return "import " ~ result ~ ';'; } mixin(import_("core", "sys", platform, "semaphore"));
Thank you, didn't know that mixins even works for imports. This greatly would help a generic ABI as the import would just select the specific files for the particular OS. The OS implementations could then have dedicated names that like SemapholeImpl as type and dedicated methods like SemapholeImpl.wait(). This really open ups the degrees of freedom.
Mar 17
prev sibling parent reply IGotD- <nise nise.com> writes:
On Tuesday, 17 March 2020 at 20:37:04 UTC, Jacob Carlborg wrote:
 Can be easily handled with a string mixin and a utility 
 function:

 version (linux)
     enum platform = "linux";
 else version (Windows)
     enum platform = "windows";

 string import_(string[] imports ...)
 {
     string result;

     foreach (i, imp; imports)
     {
         if (i != 0)
             result ~= '.';

         result ~= imp;
     }

     return "import " ~ result ~ ';';
 }

 mixin(import_("core", "sys", platform, "semaphore"));
I see many occurrences like this insinde the version blocks: import core.sys.platform.subfunctionality : subFuncImpl; alias functionality = subFuncImpl; We could provide with an abstract interface like this as you suggested. mixin(import_("core", "sys", platform, "semaphore")) : SemaphoreAcquireImpl; The Semaphore class can use SemaphoreAcquireImpl. SemaphoreAcquireImpl would have a type like this where SemaphoreOSType also needs to be imported in similar fashion: void SemaphoreAcquireImpl(SemaphoreOSType sem), but this function would be the same for every system. I think this type of interface could be repeated for almost all OS dependent functionality. The type of the semaphore descriptor must also be be provided, as some are a descriptor on certain systems or a pointer type on others. Some functions can just be aliased mixin(import_("core", "sys", platform, "process")) : getPidImpl; alias getpid = getPidImpl; This type of interface could just be added under one version block like: version(druntimeAbstractInterface) { ... } Something that we all can work on is a generic C library interface for "C library style OS" for compatibility. This of course would have limited functionality but could provide basic functionality. This also makes important to selectively choose what to support. A stub would be an alternative, that just goes through silently or raises a "not implemented exception". Nim has an --os:any option which means it will just use the C library.
May 06
parent Jacob Carlborg <doob me.com> writes:
On 2020-05-06 14:18, IGotD- wrote:

 I see many occurrences like this insinde the version blocks:
 
 import core.sys.platform.subfunctionality : subFuncImpl;
 alias functionality = subFuncImpl;
 
 We could provide with an abstract interface like this as you suggested.
 
 mixin(import_("core", "sys", platform, "semaphore")) : 
 SemaphoreAcquireImpl;
 The Semaphore class can use SemaphoreAcquireImpl.
It doesn't need to be an abstract interface. Since the implementation will be selected at compile time the only thing that is required that all implementation implements the same API. The API can be functions, structs, classes, whatever. For an existing example see [1]. [1] https://github.com/dlang/druntime/blob/master/src/rt/sections.d -- /Jacob Carlborg
May 06
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-03-17 18:38, Denis Feklushkin wrote:
 On Tuesday, 17 March 2020 at 17:12:39 UTC, IGotD- wrote:
 On Tuesday, 17 March 2020 at 16:25:22 UTC, Johan wrote:
 Rather than setting the import directory, I think a much nicer 
 solution is something like this:
Yes, already implemented in new PR: https://github.com/dlang/druntime/pull/2997
That PR kind of misses the point. What I did with Mecca was to manage to remove, more or less, all version identifiers from outside of the `platform` package. If you're doing this for druntime I recommend you have a package for each platform. Currently you still have all version identifiers in the file for Posix. -- /Jacob Carlborg
Mar 17
parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 17 March 2020 at 20:27:21 UTC, Jacob Carlborg wrote:
 On 2020-03-17 18:38, Denis Feklushkin wrote:
 On Tuesday, 17 March 2020 at 17:12:39 UTC, IGotD- wrote:
 On Tuesday, 17 March 2020 at 16:25:22 UTC, Johan wrote:
 Rather than setting the import directory, I think a much 
 nicer solution is something like this:
Yes, already implemented in new PR: https://github.com/dlang/druntime/pull/2997
That PR kind of misses the point. What I did with Mecca was to manage to remove, more or less, all version identifiers from outside of the `platform` package. If you're doing this for druntime I recommend you have a package for each platform. Currently you still have all version identifiers in the file for Posix.
This is not a problem as long as all of these platforms are supported by druntime. But yes, some another Posix files can be divided into smallest platform parts, but such small file as posix's time.d can be left and so - there will no be problem. I didn’t think about staying on this state with splitted core.stdc.time. I wanted to divide core.time too and some of its parts maybe become divided to smallest sys modules as you say. But now I stuck into these Makefiles and don't know how to solve this gracefully.
Mar 17
prev sibling parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 17 March 2020 at 17:38:41 UTC, Denis Feklushkin wrote:

 This strategy looks good, but again there is a problem that is 
 not directly related to the code: changing one module required 
 changing only 8 lines in the code, but 15 lines in build 
 scripts. And this is not all of needed changes - LDC uses its 
 own cmake build scripts.

 And each module will require approximately same changes. It 
 will be like a disaster. Thus, at first stage we need to clear 
 up build scripts.
This problem is solved by Rainer Schuetze, thank you! https://github.com/dlang/druntime/pull/3026 Discussion related to this topic continues for a little in this PR: https://github.com/dlang/druntime/pull/2997
Apr 26
prev sibling next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 25 February 2020 at 00:05:05 UTC, Denis Feklushkin 
wrote:
 Hi!

 This post is inspired by: 
 https://forum.dlang.org/thread/scklbdrgjiypknuephfj forum.dlang.org

 [...]
Somewhat related that might be of interest to you. I wrote my own standard library that doesn't depend on libc, doesn't use druntime and is compatible with betterC. https://github.com/dragon-lang/mar
Feb 25
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 25 February 2020 at 16:42:04 UTC, Jonathan Marler 
wrote:
 On Tuesday, 25 February 2020 at 00:05:05 UTC, Denis Feklushkin 
 wrote:
 Hi!

 This post is inspired by: 
 https://forum.dlang.org/thread/scklbdrgjiypknuephfj forum.dlang.org

 [...]
Somewhat related that might be of interest to you. I wrote my own standard library that doesn't depend on libc, doesn't use druntime and is compatible with betterC. https://github.com/dragon-lang/mar
This project is very interesting to me I believe it may be extremely valuable to D as a whole! I have seen it a few months ago, but back then I didn't have much time to check it out. What is the current status? Is it self-hosting? E.g. can you build dmd only with it? What are your goals (in addition to minimal external dependencies)? How much of Phobos and druntime do you want reimplement or offer alternative features to? Threading, ranges, containers and algorithms, IO, std.math, etc.?
Feb 25
parent Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 25 February 2020 at 22:39:54 UTC, Petar Kirov 
[ZombineDev] wrote:
 On Tuesday, 25 February 2020 at 16:42:04 UTC, Jonathan Marler 
 wrote:
 On Tuesday, 25 February 2020 at 00:05:05 UTC, Denis Feklushkin 
 wrote:
 Hi!

 This post is inspired by: 
 https://forum.dlang.org/thread/scklbdrgjiypknuephfj forum.dlang.org

 [...]
Somewhat related that might be of interest to you. I wrote my own standard library that doesn't depend on libc, doesn't use druntime and is compatible with betterC. https://github.com/dragon-lang/mar
This project is very interesting to me I believe it may be extremely valuable to D as a whole! I have seen it a few months ago, but back then I didn't have much time to check it out. What is the current status? Is it self-hosting? E.g. can you build dmd only with it? What are your goals (in addition to minimal external dependencies)? How much of Phobos and druntime do you want reimplement or offer alternative features to? Threading, ranges, containers and algorithms, IO, std.math, etc.?
I wanted to be able to use D without relying on libc. One use case I had for this was to write my own rootfs purely in D (https://github.com/marler8997/maros). I was very glad I did this as it was instrumental in learning how most linux distributions work. I want my standard library to work with or without GC, druntime, libc, phobos, and betterC. I want the API to be as similar as possible whether or not I'm using these features. I want to be able to enable/disable these like features without rewriting my code. In order to do this, the standard library needs to work with or without them. Also note that the "mar" library can be used alongside druntime/phobos as well. It's in its own namespace so there's no issues in using them together. I was able to gradually port code to use "mar" instead of druntime/phobos piece by piece by referencing both. Another big feature is that "mar" tracks null-terminated strings in the type system. Pretty much all OS APIs use null-terminated (aka sentinel) strings, and it seems like a good idea to enforce them at compile-time rather than runtime. Anyway, there was alot of reasons I made it. Even if all it becomes is an educational experience for me then I'm still glad I did it, and if other poeple can find more uses for it then even better.
Feb 25
prev sibling parent reply IGotD- <nise nise.com> writes:
Trying to adapt druntime has been a cascade of version (this) {} 
else version (that). druntime and phobos is a ball of wool.

After porting druntime I realized that I need Phobos as well 
because there is a lot of simple stuff like converting int to 
string and so on. Converting an int to string doesn't rely on an 
operating system. However, I'm forced to go through this version 
cascade over again which is even bigger that druntime.

The situation is similar to the C library which also might have 
dependencies all over. Some C library implementations are more 
simple and OS support can be stubbed. What D needs is a stubbed 
ABI for druntime/phobos. This means no file support, no threading 
etc. This ties into the "pay as you go" goal of D. D should be 
able to compile with minimal OS support instead of now require a 
rich OS.

What we need is an ABI. Then we also need a stubbed 
implementation of the ABI. This gives any implementer a good 
overview what needs to be implemented as well as that the OS 
support can be done incrementally. Some OSes don't support 
everything like memory mapped files and we should support this 
possibility as well.

We should also try to isolate the OS independent functionality 
more. Convenience functions that doesn't need an OS should be 
broken out, or at least with switches be compiled isolated from 
the OS dependent code.

If we are able to do this, D will make a rocket career in 
embedded systems.
May 02
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-05-02 22:24, IGotD- wrote:

 Some OSes don't support everything like memory mapped files and we 
 should support this possibility as well.
Yes, I noticed this the hard way when I added support for iOS. It was assumed to be Posix compatible but there are a couple of edge cases where some system calls are not available.
 We should also try to isolate the OS independent functionality more. 
 Convenience functions that doesn't need an OS should be broken out, or 
 at least with switches be compiled isolated from the OS dependent code.
I agree. Everything that is platform specific should be in its own package, with one sub-package for each platform. I managed to do this when I added support for macOS to Mecca [1], turned out quite nice. The only remaining version blocks outside the platform package was to select the correct platform specific import. [1] https://github.com/weka-io/mecca/pull/12 -- /Jacob Carlborg
May 06
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Wednesday, 6 May 2020 at 07:54:31 UTC, Jacob Carlborg wrote:
 On 2020-05-02 22:24, IGotD- wrote:

 Some OSes don't support everything like memory mapped files 
 and we should support this possibility as well.
Yes, I noticed this the hard way when I added support for iOS. It was assumed to be Posix compatible but there are a couple of edge cases where some system calls are not available.
POSIX has kind of lost relevance, hence why Microsoft went with Linux for their new new POSIX subsystem. "POSIX Has Become Outdated" http://www.cs.columbia.edu/~vatlidak/resources/POSIXmagazine.pdf
May 06
parent reply Jacob Carlborg <doob me.com> writes:
On 2020-05-06 10:40, Paulo Pinto wrote:

 POSIX has kind of lost relevance, hence why Microsoft went with Linux 
 for their new new POSIX subsystem.
The Linux subsystem is now actually a proper Linux kernel, running side by side with the Windows kernel, and not a reimplementation of the Linux kernel API. -- /Jacob Carlborg
May 06
parent Paulo Pinto <pjmlp progtools.org> writes:
On Wednesday, 6 May 2020 at 17:38:40 UTC, Jacob Carlborg wrote:
 On 2020-05-06 10:40, Paulo Pinto wrote:

 POSIX has kind of lost relevance, hence why Microsoft went 
 with Linux for their new new POSIX subsystem.
The Linux subsystem is now actually a proper Linux kernel, running side by side with the Windows kernel, and not a reimplementation of the Linux kernel API.
Indeed, which was the point I was making, Linux compatibility has become more valuable than plain old POSIX. They are also not the first vendor to go for that approach, other UNIXes and mainframes also offer similar compatibility, through various means.
May 06
prev sibling next sibling parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Saturday, 2 May 2020 at 20:24:46 UTC, IGotD- wrote:
 Trying to adapt druntime has been a cascade of version (this) 
 {} else version (that). druntime and phobos is a ball of wool.
This fork of ldc version of druntime is close to success: https://github.com/denizzzka/druntime/tree/non_posix Not so much has been added there (look at the diff between ldc and this branch).
 The situation is similar to the C library which also might have 
 dependencies all over. Some C library implementations are more 
 simple and OS support can be stubbed. What D needs is a stubbed 
 ABI for druntime/phobos. This means no file support, no 
 threading etc. This ties into the "pay as you go" goal of D. D 
 should be able to compile with minimal OS support instead of 
 now require a rich OS.
It is important to understand that libc library is not actually a library for use C language. Yes, it contains some C string and math functions, but OS API functions what provided by libc is convient wrapper for OS syscalls what used by everyone, because no one wants to support all number of OSes syscalls. So, libc de facto is OS API. It would be great to replace them. But this does not look as a priority, because you can always use parts of any libc as a static library for baremetal. Perhaps it is better to wait for the "worldwide libc replacement" to appear? Personally, I don't like the concept of errno in libc.
 We should also try to isolate the OS independent functionality 
 more. Convenience functions that doesn't need an OS should be 
 broken out, or at least with switches be compiled isolated from 
 the OS dependent code.
$ grep -r " core.stdc" --include='*.d' druntime/src/core/ |grep import displays that at least druntime contains not so much number of libc calls.
May 06
parent reply Jacob Carlborg <doob me.com> writes:
On 2020-05-06 13:42, Denis Feklushkin wrote:

 It is important to understand that libc library is not actually a 
 library for use C language.
 
 Yes, it contains some C string and math functions, but OS API functions 
 what provided by libc is convient wrapper for OS syscalls what used by 
 everyone, because no one wants to support all number of OSes syscalls.
I'm not sure that I agree. I would say that libc is the library containing the functions and types defined by the C standard. Then it just happens so that on most platforms they will bake in the OS APIs in the same library. I just checked, on macOS all programs are linked with libSystem. This is mostly an umbrella library that brings in other libraries. The library itself only contains 36 symbols, but it depends on 33 other libraries. One of them being libsystem_c, which seems to contain the actual C standard library. Then there's libsystem_kernel, which seems to contain the OS APIs.
 So, libc de facto is OS API.
That depends on the platform as well. On Linux the system calls provide a stable API. On macOS they don't. Apple explicitly specifies that the stable API is the syscall wrappers.
 It would be great to replace them. But this does not look as a priority, 
 because you can always use parts of any libc as a static library for 
 baremetal.
You cannot. glibc doesn't fully support static linking, not even on Linux. For example, DNS lookup does not work/is not fully supported in a fully statically linked binary.
 Perhaps it is better to wait for the "worldwide libc replacement" to 
 appear? Personally, I don't like the concept of errno in libc.
Yeah, I don't like it either. -- /Jacob Carlborg
May 06
next sibling parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Wednesday, 6 May 2020 at 18:03:40 UTC, Jacob Carlborg wrote:

 It would be great to replace them. But this does not look as a 
 priority, because you can always use parts of any libc as a 
 static library for baremetal.
You cannot. glibc doesn't fully support static linking, not even on Linux. For example, DNS lookup does not work/is not fully supported in a fully statically linked binary.
"any libc" -> "any suitable libc" :-) (picolibc is my choise for this purposes)
May 06
parent reply dangbinghoo <dangbinghoo gmail.com> writes:
On Wednesday, 6 May 2020 at 18:56:55 UTC, Denis Feklushkin wrote:
 On Wednesday, 6 May 2020 at 18:03:40 UTC, Jacob Carlborg wrote:

 It would be great to replace them. But this does not look as 
 a priority, because you can always use parts of any libc as a 
 static library for baremetal.
You cannot. glibc doesn't fully support static linking, not even on Linux. For example, DNS lookup does not work/is not fully supported in a fully statically linked binary.
"any libc" -> "any suitable libc" :-) (picolibc is my choise for this purposes)
for druntime to working with MCU platform, we probably can rely only on [newlib] or even [newlib-nano] style `libC`, and make all other things to OS APIs, although some RTOS has implemented several POSIX api, it's all only a subset of POSIX. but the embedded programmer will happy if we support newlib-nano + FreeRTOS or NuttX or Zephyr.
May 06
parent dangbinghoo <dangbinghoo gmail.com> writes:
On Thursday, 7 May 2020 at 06:46:27 UTC, dangbinghoo wrote:
 On Wednesday, 6 May 2020 at 18:56:55 UTC, Denis Feklushkin 
 wrote:
 On Wednesday, 6 May 2020 at 18:03:40 UTC, Jacob Carlborg wrote:

 [...]
You cannot. glibc doesn't fully support static linking, not even on Linux. For example, DNS lookup does not work/is not fully supported in a fully statically linked binary.
"any libc" -> "any suitable libc" :-) (picolibc is my choise for this purposes)
for druntime to working with MCU platform, we probably can rely only on [newlib] or even [newlib-nano] style `libC`, and make all other things to OS APIs, although some RTOS has implemented several POSIX api, it's all only a subset of POSIX. but the embedded programmer will happy if we support newlib-nano + FreeRTOS or NuttX or Zephyr.
sorry, I don't know [newlib-nano] is now called [picolibc]! ^_^
May 07
prev sibling parent IGotD- <nise nise.com> writes:
On Wednesday, 6 May 2020 at 18:03:40 UTC, Jacob Carlborg wrote:
 You cannot. glibc doesn't fully support static linking, not 
 even on Linux. For example, DNS lookup does not work/is not 
 fully supported in a fully statically linked binary.
Networking is not really part of the C library standard, despite it is often linked into libc. Supporting C library functionality gives you a limited system but at least a system that will be able to work on most system without any modifications, like lowest common denominator.
May 06
prev sibling next sibling parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Saturday, 2 May 2020 at 20:24:46 UTC, IGotD- wrote:

 After porting druntime I realized that I need Phobos as well 
 because there is a lot of simple stuff like converting int to 
 string and so on. Converting an int to string doesn't rely on 
 an operating system. However, I'm forced to go through this 
 version cascade over again which is even bigger that druntime.
I briefly looked (grep) which "versions" are used in Phobos. It seems to me that everything that is about different OSes support can be transferred to druntime without problem. (And in druntime can be added ability to select arbitrary OS code as repeatedly described above.) ...And then druntime could become "the new libc"
May 06
prev sibling parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
Ok, now my project runs druntime and can blink by LED. Binary 
size is 200Kb :-)

https://github.com/denizzzka/d_c_arm_test/

On Saturday, 2 May 2020 at 20:24:46 UTC, IGotD- wrote:
 Trying to adapt druntime has been a cascade of version (this) 
 {} else version (that). druntime and phobos is a ball of wool.
About Phobos: here is only one issue with it and supporting any platform for "blinking LED project": file std/ascii.d: /// Newline sequence for this system. version (Windows) immutable newline = "\r\n"; else version (Posix) immutable newline = "\n"; else static assert(0, "Unsupported OS"); Compilation fails here because "OS" isn't Windows and isn't Posix. It is need to clarify way to pass external OS version into Phobos.
May 18
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
https://dlang.org/spec/version.html#predefined-versions

FreeStanding

\n is probably correct for this version, but it probably should be 
discussed in a PR.
May 18
parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 19 May 2020 at 05:10:01 UTC, rikki cattermole wrote:
 https://dlang.org/spec/version.html#predefined-versions

 FreeStanding

 \n is probably correct for this version, but it probably should 
 be discussed in a PR.
newline declaration can be moved into druntime, because druntime depends from OS.
May 19
prev sibling parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 19 May 2020 at 05:02:41 UTC, Denis Feklushkin wrote:

 file std/ascii.d:

 /// Newline sequence for this system.
 version (Windows)
     immutable newline = "\r\n";
 else version (Posix)
     immutable newline = "\n";
 else
     static assert(0, "Unsupported OS");
I just accidentally used dmd version of Phobos without LDC extension, this is already fixed for LDC fork
May 19