www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How you guys go about -BetterC Multithreading?

reply ParticlePeter <ParticlePeter gmx.de> writes:
Any experience reports or general suggestions?
I've used only D threads so far.
Nov 09
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter wrote:
 Any experience reports or general suggestions?
 I've used only D threads so far.
It would be far easier if you use druntime + nogc and/or de-register latency-sensitive threads from druntime [1], so they're not interrupted even if some other thread calls the GC. Probably the path of least resistance is to call [2] and queue nogc tasks on [3]. If you really want to pursue the version(D_BetterC) route, then you're essentially on your own to use the threading facilities provided by your target OS, e.g.: https://linux.die.net/man/3/pthread_create https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx Though you need to be extra careful not to use thread-local storage (e.g. only shared static and __gshared) and not to rely on (shared) static {con|de}structors, dynamic arrays, associative arrays, exceptions, classes, RAII, etc., which is really not worth it, unless you're writing very low-level code (e.g. OS kernels and drivers). [1]: https://dlang.org/phobos/core_thread#.thread_detachThis [2]: https://dlang.org/phobos/core_memory#.GC.disable [3]: https://dlang.org/phobos/std_parallelism#.taskPool
Nov 09
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 09/11/2017 12:19 PM, Petar Kirov [ZombineDev] wrote:
 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter wrote:
 Any experience reports or general suggestions?
 I've used only D threads so far.
It would be far easier if you use druntime + nogc and/or de-register latency-sensitive threads from druntime [1], so they're not interrupted even if some other thread calls the GC. Probably the path of least resistance is to call [2] and queue nogc tasks on [3]. If you really want to pursue the version(D_BetterC) route, then you're essentially on your own to use the threading facilities provided by your target OS, e.g.: https://linux.die.net/man/3/pthread_create https://msdn.microsoft.com/en-us/library/windows/desktop/ms 82516(v=vs.85).aspx
You can use a library like libuv to handle threads (non-language based TLS too, not sure that it can be tied in unfortunately).
Nov 09
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 12:30:49 UTC, rikki cattermole 
wrote:
 On 09/11/2017 12:19 PM, Petar Kirov [ZombineDev] wrote:
 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter 
 wrote:
 Any experience reports or general suggestions?
 I've used only D threads so far.
It would be far easier if you use druntime + nogc and/or de-register latency-sensitive threads from druntime [1], so they're not interrupted even if some other thread calls the GC. Probably the path of least resistance is to call [2] and queue nogc tasks on [3]. If you really want to pursue the version(D_BetterC) route, then you're essentially on your own to use the threading facilities provided by your target OS, e.g.: https://linux.die.net/man/3/pthread_create https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx
You can use a library like libuv to handle threads (non-language based TLS too, not sure that it can be tied in unfortunately).
Yeah, any cross-platform thread-pool / event loop library with C interface should obviously be preferred than manual use of raw thread primitives. Essentially, try to follow Sean Parent's advice on "No Raw/Incidental *": https://www.youtube.com/watch?v=zULU6Hhp42w
Nov 09
parent ParticlePeter <ParticlePeter gmx.de> writes:
On Thursday, 9 November 2017 at 12:43:54 UTC, Petar Kirov 
[ZombineDev] wrote:
 On Thursday, 9 November 2017 at 12:30:49 UTC, rikki cattermole 
 wrote:
 On 09/11/2017 12:19 PM, Petar Kirov [ZombineDev] wrote:
 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter 
 wrote:
 Any experience reports or general suggestions?
 I've used only D threads so far.
It would be far easier if you use druntime + nogc and/or de-register latency-sensitive threads from druntime [1], so they're not interrupted even if some other thread calls the GC. Probably the path of least resistance is to call [2] and queue nogc tasks on [3]. If you really want to pursue the version(D_BetterC) route, then you're essentially on your own to use the threading facilities provided by your target OS, e.g.: https://linux.die.net/man/3/pthread_create https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx
You can use a library like libuv to handle threads (non-language based TLS too, not sure that it can be tied in unfortunately).
Yeah, any cross-platform thread-pool / event loop library with C interface should obviously be preferred than manual use of raw thread primitives. Essentially, try to follow Sean Parent's advice on "No Raw/Incidental *": https://www.youtube.com/watch?v=zULU6Hhp42w
This all is good input, thanks. I was looking into: https://github.com/GerHobbelt/pthread-win32 Anyone used this?
Nov 09
prev sibling next sibling parent reply ParticlePeter <ParticlePeter gmx.de> writes:
On Thursday, 9 November 2017 at 12:19:00 UTC, Petar Kirov 
[ZombineDev] wrote:
 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter 
 wrote:
 Any experience reports or general suggestions?
 I've used only D threads so far.
It would be far easier if you use druntime + nogc and/or de-register latency-sensitive threads from druntime [1], so they're not interrupted even if some other thread calls the GC. Probably the path of least resistance is to call [2] and queue nogc tasks on [3]. If you really want to pursue the version(D_BetterC) route, then you're essentially on your own to use the threading facilities provided by your target OS, e.g.: https://linux.die.net/man/3/pthread_create https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx Though you need to be extra careful not to use thread-local storage (e.g. only shared static and __gshared) and not to rely on (shared) static {con|de}structors, dynamic arrays, associative arrays, exceptions, classes, RAII, etc., which is really not worth it, unless you're writing very low-level code (e.g. OS kernels and drivers). [1]: https://dlang.org/phobos/core_thread#.thread_detachThis [2]: https://dlang.org/phobos/core_memory#.GC.disable [3]: https://dlang.org/phobos/std_parallelism#.taskPool
Forgot to mention, I'll try this first, I think its a good first step towards -BetterC usage. But in the end I want to see how far I can get with the -BetterC feature.
Nov 09
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 13:00:15 UTC, ParticlePeter wrote:
 On Thursday, 9 November 2017 at 12:19:00 UTC, Petar Kirov 
 [ZombineDev] wrote:
 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter 
 wrote:
 Any experience reports or general suggestions?
 I've used only D threads so far.
It would be far easier if you use druntime + nogc and/or de-register latency-sensitive threads from druntime [1], so they're not interrupted even if some other thread calls the GC. Probably the path of least resistance is to call [2] and queue nogc tasks on [3]. If you really want to pursue the version(D_BetterC) route, then you're essentially on your own to use the threading facilities provided by your target OS, e.g.: https://linux.die.net/man/3/pthread_create https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx Though you need to be extra careful not to use thread-local storage (e.g. only shared static and __gshared) and not to rely on (shared) static {con|de}structors, dynamic arrays, associative arrays, exceptions, classes, RAII, etc., which is really not worth it, unless you're writing very low-level code (e.g. OS kernels and drivers). [1]: https://dlang.org/phobos/core_thread#.thread_detachThis [2]: https://dlang.org/phobos/core_memory#.GC.disable [3]: https://dlang.org/phobos/std_parallelism#.taskPool
Forgot to mention, I'll try this first, I think its a good first step towards -BetterC usage. But in the end I want to see how far I can get with the -BetterC feature.
In short, the cost / benefit of going all the way version(D_BetterC) is incredibly poor for regular applications, as you end up a bit more limited than with modern C++ (> 11) for prototyping. For example, even writers of D real-time audio plugins don't go as far. If you're writing libraries, especially math-heavy template code, CTFE and generic code in general, then version(D_BetterC) is a useful tool for verifying that your library doesn't need unnecessary dependencies preventing it from being trivially integrated in foreign language environments. Well if you like generic code as much as I do, you can surely do great with version(D_BetterC) even for application code, but you would have to make alomst every non-builtin type that you use in your code a template parameter (or alternatively an extern (C++/COM) interface if that works in -betterC), so you can easily swap druntime/phobos-based implementations for your custom ones, one by one, but I guess few people would be interested in following this path.
Nov 09
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 09/11/2017 4:00 PM, Petar Kirov [ZombineDev] wrote:
 On Thursday, 9 November 2017 at 13:00:15 UTC, ParticlePeter wrote:
 On Thursday, 9 November 2017 at 12:19:00 UTC, Petar Kirov [ZombineDev] 
 wrote:
 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter wrote:
 Any experience reports or general suggestions?
 I've used only D threads so far.
It would be far easier if you use druntime + nogc and/or de-register latency-sensitive threads from druntime [1], so they're not interrupted even if some other thread calls the GC. Probably the path of least resistance is to call [2] and queue nogc tasks on [3]. If you really want to pursue the version(D_BetterC) route, then you're essentially on your own to use the threading facilities provided by your target OS, e.g.: https://linux.die.net/man/3/pthread_create https://msdn.microsoft.com/en-us/library/windows/desktop/ms 82516(v=vs.85).aspx Though you need to be extra careful not to use thread-local storage (e.g. only shared static and __gshared) and not to rely on (shared) static {con|de}structors, dynamic arrays, associative arrays, exceptions, classes, RAII, etc., which is really not worth it, unless you're writing very low-level code (e.g. OS kernels and drivers). [1]: https://dlang.org/phobos/core_thread#.thread_detachThis [2]: https://dlang.org/phobos/core_memory#.GC.disable [3]: https://dlang.org/phobos/std_parallelism#.taskPool
Forgot to mention, I'll try this first, I think its a good first step towards -BetterC usage. But in the end I want to see how far I can get with the -BetterC feature.
In short, the cost / benefit of going all the way version(D_BetterC) is incredibly poor for regular applications, as you end up a bit more limited than with modern C++ (> 11) for prototyping. For example, even writers of D real-time audio plugins don't go as far.
I just did some work for Guillaume Piolat (p0nce author of dplug), guess what is going to be used again, druntime!
Nov 09
prev sibling parent reply Guillaume Piolat <contact spam.com> writes:
On Thursday, 9 November 2017 at 16:00:36 UTC, Petar Kirov 
[ZombineDev] wrote:
 In short, the cost / benefit of going all the way 
 version(D_BetterC) is incredibly poor for regular applications, 
 as you end up a bit more limited than with modern C++ (> 11) 
 for prototyping. For example, even writers of D real-time audio 
 plugins don't go as far.
For now we do have some nogc alternatives for mutex, condition variables, thread-pool, file reading, etc... (dplug:core package) for use with the runtime disabled - the middle ground that's way more usable than -betterC. They may, or not, be applicable to -betterC.
Nov 10
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 10 November 2017 at 11:55:57 UTC, Guillaume Piolat 
wrote:
 On Thursday, 9 November 2017 at 16:00:36 UTC, Petar Kirov 
 [ZombineDev] wrote:
 In short, the cost / benefit of going all the way 
 version(D_BetterC) is incredibly poor for regular 
 applications, as you end up a bit more limited than with 
 modern C++ (> 11) for prototyping. For example, even writers 
 of D real-time audio plugins don't go as far.
For now we do have some nogc alternatives for mutex, condition variables, thread-pool, file reading, etc... (dplug:core package) for use with the runtime disabled - the middle ground that's way more usable than -betterC. They may, or not, be applicable to -betterC.
Interesting, thanks for the info. It looks you made good progress since the last time I read about your project (quite a while ago). I'll be sure to check what you have in dplug:core ;)
Nov 10
prev sibling parent ParticlePeter <ParticlePeter gmx.de> writes:
On Friday, 10 November 2017 at 11:55:57 UTC, Guillaume Piolat 
wrote:

 For now we do have some  nogc alternatives for mutex, condition 
 variables, thread-pool, file reading, etc... (dplug:core 
 package) for use with the runtime disabled - the middle ground 
 that's way more usable than -betterC. They may, or not, be 
 applicable to -betterC.
Your thread module (among others) is an amazing read and a very nice starting point for my endeavor. Thanks for pointing me in this direction!
Nov 27
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-11-09 13:19, Petar Kirov [ZombineDev] wrote:

 Though you need to be extra careful not to use thread-local storage
I think TLS should work, it's the OS that handles TLS, not druntime. -- /Jacob Carlborg
Nov 09
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 16:08:20 UTC, Jacob Carlborg 
wrote:
 On 2017-11-09 13:19, Petar Kirov [ZombineDev] wrote:

 Though you need to be extra careful not to use thread-local 
 storage
I think TLS should work, it's the OS that handles TLS, not druntime.
Thanks for reminding me, I keep forgetting that it should just work (minus initialization?).
Nov 09
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-11-09 17:52, Petar Kirov [ZombineDev] wrote:

 Thanks for reminding me, I keep forgetting that it should just work 
 (minus initialization?).
What do you mean "initialization"? Any type that can be used in C in TLS should work in D as well (except for macOS 32bit, if anyone cares). -- /Jacob Carlborg
Nov 09
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 19:42:55 UTC, Jacob Carlborg 
wrote:
 On 2017-11-09 17:52, Petar Kirov [ZombineDev] wrote:

 Thanks for reminding me, I keep forgetting that it should just 
 work (minus initialization?).
What do you mean "initialization"?
static constructors
Nov 10
parent Jacob Carlborg <doob me.com> writes:
On 2017-11-10 14:30, Petar Kirov [ZombineDev] wrote:

 static constructors
Yeah, those won't work. I don't think that's really related to TLS, hence my confusion. -- /Jacob Carlborg
Nov 10