www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Terminating multiple processes

reply Russel Winder <russel winder.org.uk> writes:
So, I have an application which has a sort of nano-services
architecture, basically it is a set of communicating processes.
Terminating those processes blocked on an input channel is quite easy,
send a terminate message on the input channel. But what about a process
that has no input channel, one that is blocked on OS events?

Is there a way of forcibly, but nicely, terminating a spawned process
that never executes `receive()`?

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Jan 31
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/31/18 12:44 PM, Russel Winder wrote:
 So, I have an application which has a sort of nano-services
 architecture, basically it is a set of communicating processes.
 Terminating those processes blocked on an input channel is quite easy,
 send a terminate message on the input channel. But what about a process
 that has no input channel, one that is blocked on OS events?
 
 Is there a way of forcibly, but nicely, terminating a spawned process
 that never executes `receive()`?
 
You talking about processes or threads? `receive` I believe is an inter-thread channel, no? Terminating processes is generally done via signals or in the case of windows, calling the right system call. Threads are another story. Typically, you need to have the thread check periodically for a termination event. There's no "nice" way to do it out of band. In my experience, the best way to do it is to never block, but use some sort of "wait on input" for any would-be-blocking operation. You can use a large timeout, like 1 second, if immediate termination isn't important. If you are using Fibers, and all your i/o is done using some event-based system (e.g. vibe.d), then things can be easier. -Steve
Jan 31
parent Russel Winder <russel winder.org.uk> writes:
On Wed, 2018-01-31 at 13:26 -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:
=20
[=E2=80=A6]
 You talking about processes or threads? `receive` I believe is an=20
 inter-thread channel, no?
These are threads with no shared memory, just message passing via channels. I think of them as processes, but not OS processes. My poor choose of names, I should just have said threads.
 Terminating processes is generally done via signals or in the case
 of=20
 windows, calling the right system call.
Indeed. usually SIGHUP or SIGTERM, but never SIGKILL.
 Threads are another story. Typically, you need to have the thread
 check=20
 periodically for a termination event. There's no "nice" way to do it
 out=20
 of band.
This is what I was fearing. Of course from a theoretical standpoint it is the right way to do things.
 In my experience, the best way to do it is to never block, but use
 some=20
 sort of "wait on input" for any would-be-blocking operation. You can
 use=20
 a large timeout, like 1 second, if immediate termination isn't
 important.
The thread in question is usually blocked in an inotify wait state having no input channel. It seems I will have to manufacture a select. I think this is what is missing in D, or at least my knowledge of it. In Go, I can set up a select between channels and block, no need for a timeout, an input on either channel causes a wakeup. The code then has to work out which channel caused the wakeup. On the other hand DInotify doesn't provide a channel style blocking wait, so select isn't an option.
 If you are using Fibers, and all your i/o is done using some event-
 based=20
 system (e.g. vibe.d), then things can be easier.
I am not sure I want to get into single-threaded event queues, but maybe I should. I like the dataflow approach, it is simpler for me to reason about. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
Feb 01
prev sibling parent reply Arek <arychlinski gmail.com> writes:
On Wednesday, 31 January 2018 at 17:44:37 UTC, Russel Winder 
wrote:
 So, I have an application which has a sort of nano-services 
 architecture, basically it is a set of communicating processes. 
 Terminating those processes blocked on an input channel is 
 quite easy, send a terminate message on the input channel. But 
 what about a process that has no input channel, one that is 
 blocked on OS events?

 Is there a way of forcibly, but nicely, terminating a spawned 
 process that never executes `receive()`?
Assuming your're talking about threads: there's no secure method of forcing the thread to stop. Threads share the state (eg. can hold the locks) and killing them is always risky. If your threads are blocked reading the socket, you probably can close these sockets and exit after the read error. Another way is to use atomic flag indicating that thread needs to be interrupted. After any blocking operation, the thread have to check this flag and finish the job. It's good to use timeouts (eg socket timeout) in such scenario (if possible). Arek
Jan 31
parent reply Russel Winder <russel winder.org.uk> writes:
On Wed, 2018-01-31 at 22:15 +0000, Arek via Digitalmars-d-learn wrote:
=20
[=E2=80=A6]
 Assuming your're talking about threads: there's no secure method=20
 of forcing the thread to stop. Threads share the state (eg. can=20
 hold the locks) and killing them is always risky.
I am indeed talking threads not OS processes. I just like working with (processes|tasks) submitted to a thread pool, but this can lead to jargon use problems.
 If your threads are blocked reading the socket, you probably can=20
 close these sockets and exit after the read error.
The problem is actually a thread blocked in an inotify blocking read. As both Steven and yourself have pointed out I am going to have to use a timeout to check the state of the application.=20
 Another way is to use atomic flag indicating that thread needs to=20
 be interrupted.
 After any blocking operation, the thread have to check this flag=20
 and finish the job.
 It's good to use timeouts (eg socket timeout) in such scenario=20
 (if possible).
I guess there is a choice here between shared memory to set the termination flag, or using an input channel and sending the termination message. I think the latter may be preferable, and certainly more consistent with how the other threads terminate. Thanks to you and Steven for confirming what I had been fearing, but wondered if there was an easier (albeit less theoretically sound) one. =20 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
Feb 01
next sibling parent reply Arek <arychlinski gmail.com> writes:
On Thursday, 1 February 2018 at 11:42:32 UTC, Russel Winder wrote:
 On Wed, 2018-01-31 at 22:15 +0000, Arek via Digitalmars-d-learn 
 wrote:
 
[…] The problem is actually a thread blocked in an inotify blocking read. As both Steven and yourself have pointed out I am going to have to use a timeout to check the state of the application.
Try to use inotify in non-blocking mode (an example here: https://gist.github.com/pkrnjevic/6016356) with select or epoll and timeouts.
 I guess there is a choice here between shared memory to set the 
 termination flag, or using an input channel and sending the 
 termination message. I think the latter may be preferable, and 
 certainly more consistent with how the other threads terminate.
I would use shared memory here (eg. atomic bool) because any communication channel introduces possibility of further blocking problems. Arek
Feb 01
parent reply Russel Winder <russel winder.org.uk> writes:
On Thu, 2018-02-01 at 12:15 +0000, Arek via Digitalmars-d-learn wrote:
=20
[=E2=80=A6]
 Try to use inotify in non-blocking mode (an example here:=20
 https://gist.github.com/pkrnjevic/6016356) with select or epoll=20
 and timeouts.
Isn't there a C++ binding for the C API? I am using DInotify which is a D binding. I will be checking soon but I am assuming there is a timeout version so I can loop to check the application state.
=20
 I guess there is a choice here between shared memory to set the=20
 termination flag, or using an input channel and sending the=20
 termination message. I think the latter may be preferable, and=20
 certainly more consistent with how the other threads terminate.
=20
=20 I would use shared memory here (eg. atomic bool) because any=20 communication channel introduces possibility of further blocking=20 problems.
A priori I am not convinced. I have used a state variable in C++ and Python code where there is no channel system, but in Go, Groovy/GPars, using channels is always preferable. Given the channel has a "read if there is something to read" there can't be a blocking problem =E2=80=93 if = the channel system is a good one. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
Feb 01
parent reply Arek <arychlinski gmail.com> writes:
On Thursday, 1 February 2018 at 12:30:24 UTC, Russel Winder wrote:
 On Thu, 2018-02-01 at 12:15 +0000, Arek via Digitalmars-d-learn 
 wrote:
 
[…]
 Try to use inotify in non-blocking mode (an example here: 
 https://gist.github.com/pkrnjevic/6016356) with select or 
 epoll and timeouts.
Isn't there a C++ binding for the C API? I am using DInotify which is a D binding. I will be checking soon but I am assuming there is a timeout version so I can loop to check the application state.
DInitify doesn't cover full capabilities of the inotify API. Especially it doesn't utilize newer inotify_init1 syscall and doesn't expose 'select' interface. This C++ wrapper may be interesting for you: https://github.com/erikzenker/inotify-cpp But I haven't used it.
 
 I would use shared memory here (eg. atomic bool) because any 
 communication channel introduces possibility of further 
 blocking problems.
A priori I am not convinced. I have used a state variable in C++ and Python code where there is no channel system, but in Go, Groovy/GPars, using channels is always preferable. Given the channel has a "read if there is something to read" there can't be a blocking problem – if the channel system is a good one.
You may be right. But interrupting the thread is very low level mechanism. Eg in Java, it is incorporated into Thread class: thread has method 'interrupt()' which sets the flag, and there is property 'isInterrupted()'. Usually your thread code have something like while(!Thread.currentThread().isInterrupted()) { /* do something*/ } I would use similar pattern. Arek
Feb 01
parent Russel Winder <russel winder.org.uk> writes:
On Thu, 2018-02-01 at 20:13 +0000, Arek via Digitalmars-d-learn wrote:
=20
[=E2=80=A6]
 DInitify doesn't cover full capabilities of the inotify API.
 Especially it doesn't utilize newer inotify_init1 syscall and=20
 doesn't expose 'select' interface.
 This C++ wrapper may be interesting for you:=20
 https://github.com/erikzenker/inotify-cpp
 But I haven't used it.
Hummm=E2=80=A6 sounds like I need to look to create a pull request to get t= o dinotify 0.2.3 or even 0.3.0.
=20
[=E2=80=A6]
 You may be right. But interrupting the thread is very low level=20
 mechanism.
 Eg in Java, it is incorporated into Thread class: thread has=20
 method 'interrupt()' which
 sets the flag, and there is property 'isInterrupted()'. Usually=20
 your thread code have something like =20
 while(!Thread.currentThread().isInterrupted()) { /* do=20
 something*/ }
 I would use similar pattern.
I do want to avoid such low-level things, they may be required for building libraries, but they seem totally the wrong level of concept for an application. I am definitely headed to the (select|poll|epoll) on the inotify file descriptor to get the timeout, and then either read or check the input channel for termination or an atomic Boolean state variable. Channels seem most likely since parent threads send a terminating token. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
Feb 02
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Thursday, 1 February 2018 at 11:42:32 UTC, Russel Winder wrote:
 The problem is actually a thread blocked in an inotify blocking 
 read. As both Steven and yourself have pointed out I am going 
 to have to use a timeout to check the state of the application.
There are better solutions (select/...), But couldn't you in theory just send a custom signal to the thread (which you ignore), and then check for the exit flag after the syscall returned EINTR? The DInotify wrapper might of course have the retry loop hardcoded; I didn't check. —David
Feb 01
next sibling parent Russel Winder <russel winder.org.uk> writes:
On Fri, 2018-02-02 at 01:09 +0000, David Nadlinger via Digitalmars-d-
learn wrote:
 On Thursday, 1 February 2018 at 11:42:32 UTC, Russel Winder wrote:
 The problem is actually a thread blocked in an inotify blocking=20
 read. As both Steven and yourself have pointed out I am going=20
 to have to use a timeout to check the state of the application.
=20 There are better solutions (select/...), But couldn't you in=20 theory just send a custom signal to the thread (which you=20 ignore), and then check for the exit flag after the syscall=20 returned EINTR?
I think I am going to go the (select|poll|epoll) on the file descriptor with a timeout and then the read if that is appropriate or check the input channel for the terminate token if that is appropriate. The question whether to do this in my code or create a "read with timeout" in dinotify
 The DInotify wrapper might of course have the retry loop=20
 hardcoded; I didn't check.
Dinotify adds the tree monitoring as an extra over the inotify API but U do not see any timeout related things. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
Feb 02
prev sibling parent Russel Winder <russel winder.org.uk> writes:
On Fri, 2018-02-02 at 20:14 +0000, David Nadlinger wrote:
=20
[=E2=80=A6]
 That sounds entirely sensible. Your original question was whether it
 was=20
 possible to terminate threads blocked in a syscall, though. Signals=20
 allow you to do that on POSIX for many "slow" syscalls, by making it=20
 return EINTR. You would check whatever channel/atomic flag/=E2=80=A6
 mechanism=20
 you use to signal termination before continuing to block by
 reissuing=20
 the syscall.
=20
 I haven't tested how `read` behaves for inotify myself, but
 inotify(7)=20
 suggests that it indeed handles signals this way.
[=E2=80=A6] After some discussions, Dmitry acted very quickly and added a timeout read to the blocking read in dinotify. This solves my problem "at a stroke" as I can now block waiting for a specific time on inotify and then check the input channel for a time and repeat. Not really busy wait just a sequence of blocking reads with timeouts. The important thing is that With very little latency, the thread is checking two things. DInotify 0.3.0 will likely appear soon in Dub. It would be good to get it packaged for Debian. I will take up an invitation to submit a ready made package for the Debian D packaging team. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
Feb 05