www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - can clone a thread?

reply "Gianni Pisetta" <pisetta.gianni alice.it> writes:
Hi,
i want to duplicate/clone a thread with the stack and tls, like 
the standard function fork() in c. Is in the phobos library a 
method that do the same thing?

Gianni Pisetta
Dec 10 2013
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 12/10/2013 04:04 AM, Gianni Pisetta wrote:

 Hi,
 i want to duplicate/clone a thread with the stack and tls, like the
 standard function fork() in c. Is in the phobos library a method that do
 the same thing?

 Gianni Pisetta


Apparently, there isn't. However, you can use fork() directly from the 
core.sys.posix.unistd module.

Ali
Dec 10 2013
parent reply "Gianni Pisetta" <pisetta.gianni alice.it> writes:
On Tuesday, 10 December 2013 at 13:39:17 UTC, Ali Çehreli wrote:

 On 12/10/2013 04:04 AM, Gianni Pisetta wrote:

 Hi,
 i want to duplicate/clone a thread with the stack and tls, 
 like the
 standard function fork() in c. Is in the phobos library a 
 method that do
 the same thing?

 Gianni Pisetta


 Apparently, there isn't. However, you can use fork() directly 
 from the core.sys.posix.unistd module.

 Ali


I had the same idea, but assumed (maybe wrongly) that it messes 
up the garbage collection because it creates a copy of the 
allocated objects in tls that the gc is not aware of. I searched 
the core.thread reference to see if it has a similar method but 
found none. Then i posted this question. So is it safe to use 
fork()? I now realized that i don't know how the gc works.

Gianni Pisetta
Dec 10 2013
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 12/10/2013 06:56 AM, Gianni Pisetta wrote:

 On Tuesday, 10 December 2013 at 13:39:17 UTC, Ali Çehreli wrote:

 On 12/10/2013 04:04 AM, Gianni Pisetta wrote:

 Hi,
 i want to duplicate/clone a thread with the stack and tls, like the
 standard function fork() in c. Is in the phobos library a method that do
 the same thing?

 Gianni Pisetta


 Apparently, there isn't. However, you can use fork() directly from the
 core.sys.posix.unistd module.

 Ali


 I had the same idea, but assumed (maybe wrongly) that it messes up the
 garbage collection because it creates a copy of the allocated objects in
 tls that the gc is not aware of. I searched the core.thread reference to
 see if it has a similar method but found none. Then i posted this
 question. So is it safe to use fork()? I now realized that i don't know
 how the gc works.

 Gianni Pisetta


I don't know such details but I presume fork() is safe. Grepping in 
Phobos modules, process.d uses it.

Ali
Dec 10 2013
parent reply "Gianni Pisetta" <pisetta.gianni alice.it> writes:
On Tuesday, 10 December 2013 at 17:14:47 UTC, Ali Çehreli wrote:

 On 12/10/2013 06:56 AM, Gianni Pisetta wrote:

 On Tuesday, 10 December 2013 at 13:39:17 UTC, Ali Çehreli 
 wrote:

 On 12/10/2013 04:04 AM, Gianni Pisetta wrote:

 Hi,
 i want to duplicate/clone a thread with the stack and tls, 
 like the
 standard function fork() in c. Is in the phobos library a 
 method that do
 the same thing?

 Gianni Pisetta


 Apparently, there isn't. However, you can use fork() directly 
 from the
 core.sys.posix.unistd module.

 Ali


 I had the same idea, but assumed (maybe wrongly) that it 
 messes up the
 garbage collection because it creates a copy of the allocated 
 objects in
 tls that the gc is not aware of. I searched the core.thread 
 reference to
 see if it has a similar method but found none. Then i posted 
 this
 question. So is it safe to use fork()? I now realized that i 
 don't know
 how the gc works.

 Gianni Pisetta


 I don't know such details but I presume fork() is safe. 
 Grepping in Phobos modules, process.d uses it.

 Ali


Ok, thanks for the reply. I will give it a try.

Gianni Pisetta
Dec 10 2013
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 10 Dec 2013 20:36:11 +0100
schrieb "Gianni Pisetta" <pisetta.gianni alice.it>:


 On Tuesday, 10 December 2013 at 17:14:47 UTC, Ali =C3=87ehreli wrote:

 On 12/10/2013 06:56 AM, Gianni Pisetta wrote:

 On Tuesday, 10 December 2013 at 13:39:17 UTC, Ali =C3=87ehreli=20
 wrote:

 On 12/10/2013 04:04 AM, Gianni Pisetta wrote:

 Hi,
 i want to duplicate/clone a thread with the stack and tls,=20
 like the
 standard function fork() in c. Is in the phobos library a=20
 method that do
 the same thing?

 Gianni Pisetta


 Apparently, there isn't. However, you can use fork() directly=20
 from the
 core.sys.posix.unistd module.

 Ali


 I had the same idea, but assumed (maybe wrongly) that it=20
 messes up the
 garbage collection because it creates a copy of the allocated=20
 objects in
 tls that the gc is not aware of. I searched the core.thread=20
 reference to
 see if it has a similar method but found none. Then i posted=20
 this
 question. So is it safe to use fork()? I now realized that i=20
 don't know
 how the gc works.

 Gianni Pisetta


 I don't know such details but I presume fork() is safe.=20
 Grepping in Phobos modules, process.d uses it.

 Ali


=20
 Ok, thanks for the reply. I will give it a try.
=20
 Gianni Pisetta


D is cross-platform, so to have a wrapper for fork() would
mean to get it working on Windows, too.
std.process, can must use fork() of course, since it is the
way to spawn a child process on Posix systems. fork() doesn't
clone a thread, but a whole process. So there is no issue
with the GC, as it will be cloned as well. Until Posix threads
were introduced, it was common to create "threads" using
fork(), so it probably works ok for you. Maybe we can find a
better solution though if you tell us some more.

For example this exists in D:

static this()
{
  // Initialize TLS variables
}

That is a module constructor that is run once for each new
Thread. You could prepare your TLS variables there as a copy
of global data or other function calls.
If you really need an automated copy of all TLS variables for
a new thread the situation looks dim I guess :p

--=20
Marco
Dec 10 2013
parent reply "Gianni Pisetta" <pisetta.gianni alice.it> writes:
 D is cross-platform, so to have a wrapper for fork() would
 mean to get it working on Windows, too.
 std.process, can must use fork() of course, since it is the
 way to spawn a child process on Posix systems. fork() doesn't
 clone a thread, but a whole process. So there is no issue
 with the GC, as it will be cloned as well. Until Posix threads
 were introduced, it was common to create "threads" using
 fork(), so it probably works ok for you. Maybe we can find a
 better solution though if you tell us some more.

 For example this exists in D:

 static this()
 {
   // Initialize TLS variables
 }

 That is a module constructor that is run once for each new
 Thread. You could prepare your TLS variables there as a copy
 of global data or other function calls.
 If you really need an automated copy of all TLS variables for
 a new thread the situation looks dim I guess :p


No, i think fork or something similar is the way i prefer to go.
I'm working on a simple top-down parser, so i save all the data 
references in the context of the function and at the end of it i 
create the syntax tree object with all his childrens. So i have 
the already allocated objects in the tls and the info on the 
realtionships of these objects plus the parser state in the call 
stack. I want to duplicate the parser because i want all the 
possible syntax trees as a result and when the tokenizer will 
find in the input two or more possible tokens, it must fork and 
return for each thread one token from the possible ones. At the 
end if one or more threads made to the end, each thread copy his 
syntax tree on a shared array and the main thread then can work 
on the results.

Another way to solve this is to move all the info from the call 
stack to a stack managed by me, have a standard procedure that 
fills the stack. When i must duplicate the thread i can create a 
new one, make a deep copy of the objects in the stack and the 
stack itself, then call the standard procedure again. But i think 
this is a bottom-up parser and a major change in the application 
design.

For now i will stick with fork, if in the future a similar 
function is implemented in core.thread, i will use it.
Dec 11 2013
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 11 Dec 2013 09:10:09 +0100
schrieb "Gianni Pisetta" <pisetta.gianni alice.it>:


 No, i think fork or something similar is the way i prefer to go.
 I'm working on a simple top-down parser, so i save all the data 
 references in the context of the function and at the end of it i 
 create the syntax tree object with all his childrens. So i have 
 the already allocated objects in the tls and the info on the 
 realtionships of these objects plus the parser state in the call 
 stack. I want to duplicate the parser because i want all the 
 possible syntax trees as a result and when the tokenizer will 
 find in the input two or more possible tokens, it must fork and 
 return for each thread one token from the possible ones. At the 
 end if one or more threads made to the end, each thread copy his 
 syntax tree on a shared array and the main thread then can work 
 on the results.
 
 Another way to solve this is to move all the info from the call 
 stack to a stack managed by me, have a standard procedure that 
 fills the stack. When i must duplicate the thread i can create a 
 new one, make a deep copy of the objects in the stack and the 
 stack itself, then call the standard procedure again. But i think 
 this is a bottom-up parser and a major change in the application 
 design.

 For now i will stick with fork, if in the future a similar 
 function is implemented in core.thread, i will use it.


Alright, fork is optimized for this kind of stuff and it
should work fine. Personally I would likely have tried a
manged stack instead of cloning the whole application. How do
you write back the results from the child processes? Do you
use pipes or shared memory?

-- 
Marco
Dec 11 2013
parent reply "Gianni Pisetta" <pisetta.gianni alice.it> writes:
On Wednesday, 11 December 2013 at 08:54:18 UTC, Marco Leise wrote:

 Am Wed, 11 Dec 2013 09:10:09 +0100
 schrieb "Gianni Pisetta" <pisetta.gianni alice.it>:


 No, i think fork or something similar is the way i prefer to 
 go.
 I'm working on a simple top-down parser, so i save all the 
 data references in the context of the function and at the end 
 of it i create the syntax tree object with all his childrens. 
 So i have the already allocated objects in the tls and the 
 info on the realtionships of these objects plus the parser 
 state in the call stack. I want to duplicate the parser 
 because i want all the possible syntax trees as a result and 
 when the tokenizer will find in the input two or more possible 
 tokens, it must fork and return for each thread one token from 
 the possible ones. At the end if one or more threads made to 
 the end, each thread copy his syntax tree on a shared array 
 and the main thread then can work on the results.
 
 Another way to solve this is to move all the info from the 
 call stack to a stack managed by me, have a standard procedure 
 that fills the stack. When i must duplicate the thread i can 
 create a new one, make a deep copy of the objects in the stack 
 and the stack itself, then call the standard procedure again. 
 But i think this is a bottom-up parser and a major change in 
 the application design.

 For now i will stick with fork, if in the future a similar 
 function is implemented in core.thread, i will use it.


 Alright, fork is optimized for this kind of stuff and it
 should work fine. Personally I would likely have tried a
 manged stack instead of cloning the whole application. How do
 you write back the results from the child processes? Do you
 use pipes or shared memory?


I also like the managed stack idea, but i think the main problem 
with that is that it is harder to understand what's going on when 
you have to debug.
I'm not at the point yet of passing the results to the main 
thread, but i think at the end i will make the entire tree 
immutable(or maybe already create it with immutable) and add the 
reference to a shared array or use a message passing pattern like 
the one in std.concurrency.

Gianni Pisetta
Dec 11 2013
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 11 Dec 2013 11:04:29 +0100
schrieb "Gianni Pisetta" <pisetta.gianni alice.it>:


 On Wednesday, 11 December 2013 at 08:54:18 UTC, Marco Leise wrote:

 Am Wed, 11 Dec 2013 09:10:09 +0100
 schrieb "Gianni Pisetta" <pisetta.gianni alice.it>:


 No, i think fork or something similar is the way i prefer to 
 go.
 I'm working on a simple top-down parser, so i save all the 
 data references in the context of the function and at the end 
 of it i create the syntax tree object with all his childrens. 
 So i have the already allocated objects in the tls and the 
 info on the realtionships of these objects plus the parser 
 state in the call stack. I want to duplicate the parser 
 because i want all the possible syntax trees as a result and 
 when the tokenizer will find in the input two or more possible 
 tokens, it must fork and return for each thread one token from 
 the possible ones. At the end if one or more threads made to 
 the end, each thread copy his syntax tree on a shared array 
 and the main thread then can work on the results.
 
 Another way to solve this is to move all the info from the 
 call stack to a stack managed by me, have a standard procedure 
 that fills the stack. When i must duplicate the thread i can 
 create a new one, make a deep copy of the objects in the stack 
 and the stack itself, then call the standard procedure again. 
 But i think this is a bottom-up parser and a major change in 
 the application design.

 For now i will stick with fork, if in the future a similar 
 function is implemented in core.thread, i will use it.


 Alright, fork is optimized for this kind of stuff and it
 should work fine. Personally I would likely have tried a
 manged stack instead of cloning the whole application. How do
 you write back the results from the child processes? Do you
 use pipes or shared memory?


 
 I also like the managed stack idea, but i think the main problem 
 with that is that it is harder to understand what's going on when 
 you have to debug.
 I'm not at the point yet of passing the results to the main 
 thread, but i think at the end i will make the entire tree 
 immutable(or maybe already create it with immutable) and add the 
 reference to a shared array or use a message passing pattern like 
 the one in std.concurrency.
 
 Gianni Pisetta


Then wait a second! fork() creates a new _process_, not a
thread. The two options I gave are pretty much the only ones
available for communication between local processes. Maybe add
a TCP socket connection to that...

Shared arrays and std.concurrency work only for _threads_
inside the same process.

-- 
Marco
Dec 12 2013