digitalmars.D.learn - GDC - program runs in one thread, DMD - in 4 threads, why?
- eugene (45/45) Sep 10 2021 Here is test program (which is using DList aggressively)
- drug (6/9) Sep 10 2021 It is off-topic a bit but I think none can compare gdc 4.9.2 to dmd
- eugene (6/8) Sep 10 2021 I am newbie - have been learning D for about 2 months or so.
- Mike Parker (9/11) Sep 10 2021 GDC is stuck on a much older version of D. Iain has backported
- Basile B. (3/12) Sep 10 2021 most recent dmd has a GC that collect in threads. it can be tuned
- bauss (24/37) Sep 10 2021 Here's the specific change:
- eugene (2/4) Sep 10 2021 thanx a lot!
- eugene (16/18) Sep 10 2021 but it does not:
- Steven Schveighoffer (7/31) Sep 10 2021 `--DRT...` is a d runtime switch, which is processed while running your
- Adam D Ruppe (13/19) Sep 10 2021 Yeah, in newer versions, when the garbage collector does its
- eugene (2/4) Sep 10 2021 oops... :)
- eugene (5/10) Sep 10 2021 ps xH | grep [e]cho
- Adam D Ruppe (1/1) Sep 10 2021 btw why do the threads cause you trouble?
Here is test program (which is using DList aggressively)
```d
import std.stdio : writeln;
import core.sys.posix.unistd : sleep;
//import std.container.dlist; // dmd (v2.097.2)
import std.container: DList; // gdc (4.9.2)
void main() {
auto list = DList!(int)();
int k;
while (true) {
// 1.sleep;
k.writeln;
list.insertBack(k);
list.insertBack(k);
list.removeFront();
list.removeFront();
k++;
}
}
```
When compiled with gdc, it runs in one thread:
dexp ~ $ ps xH | grep [t]est
3823 pts/12 R+ 0:07 ./test
But when with dmd, program creates 3 additional threads:
dexp ~ $ ps xH | grep [t]est
3839 pts/12 Rl+ 0:00 ./test
3839 pts/12 Sl+ 0:00 ./test
3839 pts/12 Sl+ 0:00 ./test
3839 pts/12 Sl+ 0:00 ./test
What are these extra threads for?
To work with list (which is strange)???
GC?
Something else?
Observations:
- without DList there is no extra-threads
- with sleep() there is also no extra-threads
So I think it is **very aggressive usage** of DList that causes
this.
Can this (really unwanted) behavior be disabled in DMD?
I do not want to have multiple threads,
a program (real program, not the test above) has to be
single-threaded.
system used:
dexp ~ $ cat /etc/debian_version
8.11
Sep 10 2021
10.09.2021 12:27, eugene пишет://import std.container.dlist; // dmd (v2.097.2) import std.container: DList; // gdc (4.9.2)It is off-topic a bit but I think none can compare gdc 4.9.2 to dmd 2.097.2 because gdc has older version than dmd. I would compare gdc to appropriate dmd version, it's about 2.078. But note that gdc backported some code from more recent dmd-fe versions so it has 2.078-dirty version in fact.
Sep 10 2021
On Friday, 10 September 2021 at 10:20:52 UTC, drug wrote:It is off-topic a bitI am newbie - have been learning D for about 2 months or so. I understand that my question is not about the language itself, just picked forum for new users.but I think none can compare gdc 4.9.2 tosame picture with gdc 8.4.0 - one thread, no pthread_create() behind the scenes.
Sep 10 2021
Mike Parker <aldacron gmail.com> On Friday, 10 September 2021 at 11:20:10 UTC, eugene wrote:same picture with gdc 8.4.0 - one thread, no pthread_create() behind the scenes.GDC is stuck on a much older version of D. Iain has backported some bugfixes and optimizations, but featurewise it's mostly D 2.076. This is because that's the last version of the D frontend implemented in C++, and he needed that to get gdc into gcc. As bauss pointed out, the library feature you're talking about was implemented in 2.087. Iain is working on porting gdc over to the D version of the D frontend, at which point it will be much more up-to-date.
Sep 10 2021
On Friday, 10 September 2021 at 09:27:49 UTC, eugene wrote:Here is test program (which is using DList aggressively) [...] Can this (really unwanted) behavior be disabled in DMD? I do not want to have multiple threads, a program (real program, not the test above) has to be single-threaded. system used: dexp ~ $ cat /etc/debian_version 8.11most recent dmd has a GC that collect in threads. it can be tuned but I dont find the documentation for this right now.
Sep 10 2021
On Friday, 10 September 2021 at 10:39:48 UTC, Basile B. wrote:On Friday, 10 September 2021 at 09:27:49 UTC, eugene wrote:Here's the specific change: https://dlang.org/changelog/2.087.0.html#gc_parallel ``` GC now marks the heap with multiple threads The garbage collector now uses available CPU cores to mark the heap faster. This reduces pause times for a collection considerably. By default, the GC uses all available logical cores of your CPU. This might affect your application if it has threads that are not suspended during the mark phase of the collection. You can configure the number of additional threads used for marking by DRT option parallel to the GC configuration, e.g. by passing --DRT-gcopt=parallel:2 on the command line. A value of 0 disables parallel marking completely. As usual, you can also embed the configuration into the application by redefining rt_options, e.g. extern(C) __gshared string[] rt_options = [ "gcopt=parallel:0" ]; ``` ______________ So for OP what you want to use is: ```d --DRT-gcopt=parallel:0 ```Here is test program (which is using DList aggressively) [...] Can this (really unwanted) behavior be disabled in DMD? I do not want to have multiple threads, a program (real program, not the test above) has to be single-threaded. system used: dexp ~ $ cat /etc/debian_version 8.11most recent dmd has a GC that collect in threads. it can be tuned but I dont find the documentation for this right now.
Sep 10 2021
On Friday, 10 September 2021 at 11:09:10 UTC, bauss wrote:Here's the specific change: https://dlang.org/changelog/2.087.0.html#gc_parallelthanx a lot!
Sep 10 2021
On Friday, 10 September 2021 at 11:09:10 UTC, bauss wrote:--DRT-gcopt=parallel:2 on the command line. A value of 0 disables parallel marking completely.but it does not: make -f Makefile-dmd dmd --DRT-gcopt=parallel:0 engine/*.d common-sm/*.d server-sm/*.d pool.d echo_server.d -ofecho-server dmd --DRT-gcopt=parallel:0 engine/*.d common-sm/*.d client-sm/*.d pool.d echo_client.d -ofecho-client ps xH | grep [e]cho 5460 pts/14 Sl+ 0:00 ./echo-server 5460 pts/14 Sl+ 0:00 ./echo-server 5460 pts/14 Sl+ 0:00 ./echo-server 5460 pts/14 Sl+ 0:00 ./echo-server 5466 pts/15 Sl+ 0:00 ./echo-client 5466 pts/15 Sl+ 0:00 ./echo-client 5466 pts/15 Sl+ 0:00 ./echo-client 5466 pts/15 Sl+ 0:00 ./echo-client
Sep 10 2021
On 9/10/21 7:47 AM, eugene wrote:On Friday, 10 September 2021 at 11:09:10 UTC, bauss wrote:`--DRT...` is a d runtime switch, which is processed while running your program, not by the compiler. Try `./echo-client --DRT-gcopt=parallel:0` There is also a way to add this to your program so it's not needed on the command line. -Steve--DRT-gcopt=parallel:2 on the command line. A value of 0 disables parallel marking completely.but it does not: make -f Makefile-dmd dmd --DRT-gcopt=parallel:0 engine/*.d common-sm/*.d server-sm/*.d pool.d echo_server.d -ofecho-server dmd --DRT-gcopt=parallel:0 engine/*.d common-sm/*.d client-sm/*.d pool.d echo_client.d -ofecho-client ps xH | grep [e]cho 5460 pts/14 Sl+ 0:00 ./echo-server 5460 pts/14 Sl+ 0:00 ./echo-server 5460 pts/14 Sl+ 0:00 ./echo-server 5460 pts/14 Sl+ 0:00 ./echo-server 5466 pts/15 Sl+ 0:00 ./echo-client 5466 pts/15 Sl+ 0:00 ./echo-client 5466 pts/15 Sl+ 0:00 ./echo-client 5466 pts/15 Sl+ 0:00 ./echo-client
Sep 10 2021
On Friday, 10 September 2021 at 09:27:49 UTC, eugene wrote:What are these extra threads for? GC? So I think it is **very aggressive usage** of DList that causes this.Yeah, in newer versions, when the garbage collector does its first collect, it spawns some helper threads to speed up its mark process. When the GC is not actively marking, those extra threads are 100% idle, so they don't affect the rest of your program.Can this (really unwanted) behavior be disabled in DMD? I do not want to have multiple threads,here's the page on it https://dlang.org/spec/garbage.html#gc_config You either pass as an argument *to your application* --DRT-gcopt=parallel:0 or to make it permanent, in your source file with the main() function, add this: extern(C) __gshared string[] rt_options = [ "gcopt=parallel:0" ]; and it will stop doing that.
Sep 10 2021
On Friday, 10 September 2021 at 11:32:02 UTC, Adam D Ruppe wrote:You either pass as an argument *to your application* --DRT-gcopt=parallel:0oops... :)
Sep 10 2021
On Friday, 10 September 2021 at 11:53:04 UTC, eugene wrote:On Friday, 10 September 2021 at 11:32:02 UTC, Adam D Ruppe wrote:ps xH | grep [e]cho 5727 pts/14 S+ 0:13 ./echo-server --DRT-gcopt=parallel:0 5762 pts/15 S+ 0:12 ./echo-client --DRT-gcopt=parallel:0 thanks everybody for answers!You either pass as an argument *to your application* --DRT-gcopt=parallel:0oops... :)
Sep 10 2021
btw why do the threads cause you trouble?
Sep 10 2021
On Friday, 10 September 2021 at 12:10:58 UTC, Adam D Ruppe wrote:btw why do the threads cause you trouble?Well... probably it is subjective thing - just do not 'like' when a program is doing something that is not explicitly in it's source (I am C coder, you guessed). More specifically - I have event driven state machine engine (reactor pattern, written in C), which I have been using over recent 5 years in various data acquisition and network client/servers programs. All these programs by their nature are single-threaded, since 'event/message driven' + 'state machines' combination gives me very high-grained concurrency within single thread (and I don't like fibers/greenlets/coroutines). Recently I tried to re-implement that engine in D 'problematic'). I can give a link to sources of those echo-client/server pair if someone is interested (but there is a 'little' problem remained - when compliled with dmd, it sometimes crashes upon receiving SIGINT).
Sep 10 2021
On Friday, 10 September 2021 at 12:46:36 UTC, eugene wrote:On Friday, 10 September 2021 at 12:10:58 UTC, Adam D Ruppe wrote:Well the program you're writing is still single-threaded. The GC only uses the other threads when sweeping so the program flow will still be single-threaded and the exact same. It's just used to speed-up the GC.btw why do the threads cause you trouble?Well... probably it is subjective thing - just do not 'like' when a program is doing something that is not explicitly in it's source (I am C coder, you guessed). More specifically - I have event driven state machine engine (reactor pattern, written in C), which I have been using over recent 5 years in various data acquisition and network client/servers programs. All these programs by their nature are single-threaded, since 'event/message driven' + 'state machines' combination gives me very high-grained concurrency within single thread (and I don't like fibers/greenlets/coroutines). Recently I tried to re-implement that engine in D 'problematic'). I can give a link to sources of those echo-client/server pair if someone is interested (but there is a 'little' problem remained - when compliled with dmd, it sometimes crashes upon receiving SIGINT).
Sep 10 2021
On Friday, 10 September 2021 at 12:59:08 UTC, bauss wrote:It's just used to speed-up the GC.Yeah, I got the point, but to be absolutely honest, I (>20 years of C coding) do not like GC as such. I believe manual free() is not that 'hard'. And one must still release other resources. (in C I write just close(fd), in D I write client.destroy() - in both cases I have to do this manually, if that fd was obtained in absolutely another scope) Yes, I know, I can do without GC in D, just did not tried yet (started learning D two months ago or so) And again - I've chosen D (not Rust), because 'borrow checker' and especially 'explicit lifetimes' is a nightmare :) - i would rather accept GC.
Sep 10 2021