www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Condition Mutexes

reply dsimcha <dsimcha yahoo.com> writes:
I'm messing around w/ core.sync.  Does anyone know what I'm doing wrong in

(right now, I'm using busy spinning), I might have a decent implementation of
parallelForeach over ranges.

import core.sync.mutex, core.sync.condition, core.thread, std.stdio;

__gshared Condition condition;

void waitThenPrint() {
    condition.wait();
    writeln("FOO");
}

void main() {
    condition = new Condition( new Mutex() );
    auto T = new Thread(&waitThenPrint);
    T.start();
    condition.notify();  // Never wakes up and prints FOO.
}
Oct 20 2009
next sibling parent Graham St Jack <Graham.StJack internode.on.net> writes:
On Wed, 21 Oct 2009 00:56:13 +0000, dsimcha wrote:

 I'm messing around w/ core.sync.  Does anyone know what I'm doing wrong

 mutexes (right now, I'm using busy spinning), I might have a decent
 implementation of parallelForeach over ranges.
 
 import core.sync.mutex, core.sync.condition, core.thread, std.stdio;
 
 __gshared Condition condition;
 
 void waitThenPrint() {
     condition.wait();
     writeln("FOO");
 }
 
 void main() {
     condition = new Condition( new Mutex() ); auto T = new
     Thread(&waitThenPrint);
     T.start();
     condition.notify();  // Never wakes up and prints FOO.
 }
There are a few problems. The most serious is that you have to lock the mutex before calling condition.wait(). The underlying operating-system stuff atomically This means that the mutex needs to be an attribute of the class, and waitThenPrint() should be more like this: void waitThenPrint() { synchronized(myMutex) { condition.wait(); } writeln("FOO"); } While it isn't strictly necessary in this case, you should also: Put the condition.notify() call into a synchronized(myMutex) block. When some state variables are involved in the condition, you should do something like this: void waitThenPrint() { synchronized(myMutex) { while (state_not_right()) { condition.wait(); } } writeln("FOO"); } and synchronized(myMutex) { set_state_to_right(); condition.notify(); }
Oct 20 2009
prev sibling parent reply Bartosz Milewski <bartosz-nospam relisoft.com> writes:
dsimcha Wrote:


 void main() {
     condition = new Condition( new Mutex() );
     auto T = new Thread(&waitThenPrint);
     T.start();
     condition.notify();  // Never wakes up and prints FOO.
 }
Your program terminates immediately after sending the notification. You need to stall the exit until the other thread has a chance to wake up.
Oct 21 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Bartosz Milewski (bartosz-nospam relisoft.com)'s article
 dsimcha Wrote:
 void main() {
     condition = new Condition( new Mutex() );
     auto T = new Thread(&waitThenPrint);
     T.start();
     condition.notify();  // Never wakes up and prints FOO.
 }
Your program terminates immediately after sending the notification. You need to
stall the exit until the other thread has a chance to wake up. Thanks. I've implemented this, along w/ one other suggestion from another poster. Here's the new program. It still doesn't work. Has anyone successfully used core.sync.condition from druntime *on D2, not the Tango version on D1*? If it works, then it should be better documented so people who aren't already threading gurus can figure out how to use it. If it doesn't work, then as soon as I can confirm that I'm not the problem, I'll go file a bug report. Bartosz, since you're a threading guru, could you please write a simple test program using core.sync.condition and see if it works, and if not either file a bug report or let me know? import core.sync.mutex, core.sync.condition, core.thread, std.stdio; __gshared Condition condition; __gshared Mutex mutex; void waitThenPrint() { mutex.lock; condition.wait(); mutex.unlock; writeln("FOO"); } void main() { mutex = new Mutex; condition = new Condition(mutex); auto T = new Thread(&waitThenPrint); T.start(); condition.notify(); // Never wakes up and prints FOO. T.join; }
Oct 21 2009
next sibling parent reply Jason House <jason.james.house gmail.com> writes:
dsimcha Wrote:

 == Quote from Bartosz Milewski (bartosz-nospam relisoft.com)'s article
 dsimcha Wrote:
 void main() {
     condition = new Condition( new Mutex() );
     auto T = new Thread(&waitThenPrint);
     T.start();
     condition.notify();  // Never wakes up and prints FOO.
 }
Your program terminates immediately after sending the notification. You need to
stall the exit until the other thread has a chance to wake up. Thanks. I've implemented this, along w/ one other suggestion from another poster. Here's the new program. It still doesn't work. Has anyone successfully used core.sync.condition from druntime *on D2, not the Tango version on D1*? If it works, then it should be better documented so people who aren't already threading gurus can figure out how to use it. If it doesn't work, then as soon as I can confirm that I'm not the problem, I'll go file a bug report. Bartosz, since you're a threading guru, could you please write a simple test program using core.sync.condition and see if it works, and if not either file a bug report or let me know? import core.sync.mutex, core.sync.condition, core.thread, std.stdio; __gshared Condition condition; __gshared Mutex mutex; void waitThenPrint() { mutex.lock; condition.wait(); mutex.unlock; writeln("FOO"); } void main() { mutex = new Mutex; condition = new Condition(mutex); auto T = new Thread(&waitThenPrint); T.start(); condition.notify(); // Never wakes up and prints FOO. T.join; }
You should lock condition before calling notify. Even if you did that, you have a race for which thread gets the lock first. If the main thread gets it, the spawned thread will never receive the notify and hang forever.
Oct 21 2009
parent dsimcha <dsimcha yahoo.com> writes:
== Quote from Jason House (jason.james.house gmail.com)'s article
 dsimcha Wrote:
 == Quote from Bartosz Milewski (bartosz-nospam relisoft.com)'s article
 dsimcha Wrote:
 void main() {
     condition = new Condition( new Mutex() );
     auto T = new Thread(&waitThenPrint);
     T.start();
     condition.notify();  // Never wakes up and prints FOO.
 }
Your program terminates immediately after sending the notification. You need to
stall the exit until the other thread has a chance to wake up. Thanks. I've implemented this, along w/ one other suggestion from another poster. Here's the new program. It still doesn't work. Has anyone successfully used core.sync.condition from druntime *on D2, not the Tango version on D1*? If it works, then it should be better documented so people who aren't already threading gurus can figure out how to use it. If it doesn't work, then as soon as I can confirm that I'm not the problem, I'll go file a bug report. Bartosz, since you're a threading guru, could you please write a simple test program using core.sync.condition and see if it works, and if not either file a bug report or let me know? import core.sync.mutex, core.sync.condition, core.thread, std.stdio; __gshared Condition condition; __gshared Mutex mutex; void waitThenPrint() { mutex.lock; condition.wait(); mutex.unlock; writeln("FOO"); } void main() { mutex = new Mutex; condition = new Condition(mutex); auto T = new Thread(&waitThenPrint); T.start(); condition.notify(); // Never wakes up and prints FOO. T.join; }
You should lock condition before calling notify. Even if you did that, you have
a race for which thread gets the lock first. If the main thread gets it, the spawned thread will never receive the notify and hang forever. Thanks, but at a more general level, where can I find documentation on how to use conditions properly? I realize that I have absolutely no fundamental understanding of how they work or what assumptions they make, but it seems like there is very little written material on this, either for the general case or for the specific case of core.sync. For example, for some reason I assumed that a condition's methods would be already synchronized internally.
Oct 21 2009
prev sibling next sibling parent =?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:
dsimcha wrote:
 =3D=3D Quote from Bartosz Milewski (bartosz-nospam relisoft.com)'s arti=
cle
 dsimcha Wrote:
 void main() {
     condition =3D new Condition( new Mutex() );
     auto T =3D new Thread(&waitThenPrint);
     T.start();
     condition.notify();  // Never wakes up and prints FOO.
 }
Your program terminates immediately after sending the notification. Yo=
u need to
 stall the exit until the other thread has a chance to wake up.
=20
 Thanks.  I've implemented this, along w/ one other suggestion from anot=
her poster.
  Here's the new program.  It still doesn't work.  Has anyone successful=
ly used
 core.sync.condition from druntime *on D2, not the Tango version on D1*?=
If it
 works, then it should be better documented so people who aren't already=
threading
 gurus can figure out how to use it.  If it doesn't work, then as soon a=
s I can
 confirm that I'm not the problem, I'll go file a bug report.
=20
 Bartosz, since you're a threading guru, could you please write a simple=
test
 program using core.sync.condition and see if it works, and if not eithe=
r file a
 bug report or let me know?
=20
 import core.sync.mutex, core.sync.condition, core.thread, std.stdio;
=20
 __gshared Condition condition;
 __gshared Mutex mutex;
=20
 void waitThenPrint() {
     mutex.lock;
     condition.wait();
     mutex.unlock;
     writeln("FOO");
 }
=20
 void main() {
     mutex =3D new Mutex;
     condition =3D new Condition(mutex);
     auto T =3D new Thread(&waitThenPrint);
     T.start();
     condition.notify();  // Never wakes up and prints FOO.
     T.join;
 }
Well, you should lock the mutex before calling condition.notify and=20 release it afterwards. Note that there is always a chance that "notify" will be called=20 before the thread starts waiting. In that case your program will=20 deadlock (randomly). Jerome --=20 mailto:jeberger free.fr http://jeberger.free.fr Jabber: jeberger jabber.fr
Oct 21 2009
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 21 Oct 2009 14:50:32 -0400, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Bartosz Milewski (bartosz-nospam relisoft.com)'s article
 dsimcha Wrote:
 void main() {
     condition = new Condition( new Mutex() );
     auto T = new Thread(&waitThenPrint);
     T.start();
     condition.notify();  // Never wakes up and prints FOO.
 }
Your program terminates immediately after sending the notification. You need to
stall the exit until the other thread has a chance to wake up. Thanks. I've implemented this, along w/ one other suggestion from another poster. Here's the new program. It still doesn't work.
Here is a major flaw in your logic: A condition is a *signal* not a *flag*. In order to catch the signal you have to be listening for it. It's not like a semaphore. What you need in order to use a condition is a flag that is protected by the lock. Generally, the model is: thread1() { lock(mutex) { set(flag); condition.notify(); } } thread2() { lock(mutex) { while(!flag) condition.wait(); unset(flag); // we received the signal, clear it. } } The condition is basically a way to give control of waking up a thread to another thread. But the condition is not the, um... condition you are waiting for :) You still need a state variable to say "hey, you should continue now, the state is correctly set". Think of the flag like a mailbox flag. You only put it up *after* you put mail in the mailbox, and the mailman lowers the flag when he gets the mail out. There are lots of threading tutorials you can probably read to get it. But basically, I can break down what exactly happens, I'll do 2 scenarios: Scenario 1: 1. thread2 locks the mutex, which protects the flag. It sees that the flag is unset, so it waits on the condition. This *atomically* unlocks the mutex and enters the thread into the list of threads to wake up when the condition is signaled. 2. thread1 now can lock the mutex, and sets the flag. It notifies the condition, which wakes up thread2. 3. thread2 *remains asleep* until it can reacquire the lock. thread1 unlocks the mutex after leaving the scope. 4. thread2 wakes up after reacquiring the mutex, and repeats the while-loop, seeing that the flag is now set. 5. thread2 unsets the flag and unlocks the mutex, continuing. Scenario 2: 1. thread1 locks the mutex. 2. thread2 sleeps because it cannot lock the mutex. 3. thread1 sets the flag, then signals the condition. Since nobody is listening *this doesn't affect anything*. 4. thread1 exits the scope, releasing the mutex. 5. thread2 now acquires the lock, sees the flag is set, and doesn't even wait on the condition, unsets the flag, exits the scope, and unlocks the mutex. You can see how the locking is important to protect the atomicity of setting the flag, and it is *really* important that the condition wait atomically unlocks the mutex and enters the thread into the condition's wakeup queue. So generally speaking: rule 1, don't do anything with a condition unless the mutex it uses is locked. rule 2, always have a state that is protected by the same lock that the condition is waiting with. You also may wonder why there is even a while loop, I mean, why recheck the flag after the condition is signalled? Well, in this case, it's not required, but it's very good practice. When you have a case where the flag is not a flag, but a multi-state variable, and you are waiting for a specific state, one thread might signal every time the state changes. Well, you don't want to continue until you get the state that you want, so you need to re-check the state. Another case is if you have several instances of thread2, and you only want to release one of them with the signal. Hope this helps. -Steve
Oct 21 2009
parent dsimcha <dsimcha yahoo.com> writes:
== Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 On Wed, 21 Oct 2009 14:50:32 -0400, dsimcha <dsimcha yahoo.com> wrote:
 == Quote from Bartosz Milewski (bartosz-nospam relisoft.com)'s article
 dsimcha Wrote:
 void main() {
     condition = new Condition( new Mutex() );
     auto T = new Thread(&waitThenPrint);
     T.start();
     condition.notify();  // Never wakes up and prints FOO.
 }
Your program terminates immediately after sending the notification. You need to
stall the exit until the other thread has a chance to wake up. Thanks. I've implemented this, along w/ one other suggestion from another poster. Here's the new program. It still doesn't work.
Here is a major flaw in your logic: A condition is a *signal* not a *flag*. In order to catch the signal you have to be listening for it. It's not like a semaphore. What you need in order to use a condition is a flag that is protected by the lock. Generally, the model is: thread1() { lock(mutex) { set(flag); condition.notify(); } } thread2() { lock(mutex) { while(!flag) condition.wait(); unset(flag); // we received the signal, clear it. } } The condition is basically a way to give control of waking up a thread to another thread. But the condition is not the, um... condition you are waiting for :) You still need a state variable to say "hey, you should continue now, the state is correctly set". Think of the flag like a mailbox flag. You only put it up *after* you put mail in the mailbox, and the mailman lowers the flag when he gets the mail out. There are lots of threading tutorials you can probably read to get it. But basically, I can break down what exactly happens, I'll do 2 scenarios: Scenario 1: 1. thread2 locks the mutex, which protects the flag. It sees that the flag is unset, so it waits on the condition. This *atomically* unlocks the mutex and enters the thread into the list of threads to wake up when the condition is signaled. 2. thread1 now can lock the mutex, and sets the flag. It notifies the condition, which wakes up thread2. 3. thread2 *remains asleep* until it can reacquire the lock. thread1 unlocks the mutex after leaving the scope. 4. thread2 wakes up after reacquiring the mutex, and repeats the while-loop, seeing that the flag is now set. 5. thread2 unsets the flag and unlocks the mutex, continuing. Scenario 2: 1. thread1 locks the mutex. 2. thread2 sleeps because it cannot lock the mutex. 3. thread1 sets the flag, then signals the condition. Since nobody is listening *this doesn't affect anything*. 4. thread1 exits the scope, releasing the mutex. 5. thread2 now acquires the lock, sees the flag is set, and doesn't even wait on the condition, unsets the flag, exits the scope, and unlocks the mutex. You can see how the locking is important to protect the atomicity of setting the flag, and it is *really* important that the condition wait atomically unlocks the mutex and enters the thread into the condition's wakeup queue. So generally speaking: rule 1, don't do anything with a condition unless the mutex it uses is locked. rule 2, always have a state that is protected by the same lock that the condition is waiting with. You also may wonder why there is even a while loop, I mean, why recheck the flag after the condition is signalled? Well, in this case, it's not required, but it's very good practice. When you have a case where the flag is not a flag, but a multi-state variable, and you are waiting for a specific state, one thread might signal every time the state changes. Well, you don't want to continue until you get the state that you want, so you need to re-check the state. Another case is if you have several instances of thread2, and you only want to release one of them with the signal. Hope this helps. -Steve
Thank you. This was *extremely* helpful. I'll read over it in more detail when I get back to hacking my futures/parallel foreach lib.
Oct 21 2009
prev sibling parent reply Bartosz Milewski <bartosz-nospam relisoft.com> writes:
dsimcha Wrote:

 Bartosz, since you're a threading guru, could you please write a simple test
 program using core.sync.condition and see if it works, and if not either file a
 bug report or let me know?
Conditions were implemented by Sean Kelly, so he's the guru. The file condition.d in core.sync has unit tests, which presumably still run (although I don't know if anybody still runs unit tests in druntime). If using conditions in D is harder than in Java, than we should rethink their implementation. Every D object has a built-in mutex. Conditon variables should be able to work with this mutex and with synchronized sections out of the box. The most common case should be the easiest. The fact that you are having all those technical problems proves my point. As for non-technical problems, rewrite your test so that you wait on a condition in main and let the new thread do the signalling. That way they won't be able to miss each other, as they currently do.
Oct 21 2009
next sibling parent dsimcha <dsimcha yahoo.com> writes:
== Quote from Bartosz Milewski (bartosz-nospam relisoft.com)'s article
 dsimcha Wrote:
 Bartosz, since you're a threading guru, could you please write a simple test
 program using core.sync.condition and see if it works, and if not either file a
 bug report or let me know?
Conditions were implemented by Sean Kelly, so he's the guru. The file
condition.d in core.sync has unit tests, which presumably still run (although I don't know if anybody still runs unit tests in druntime).
 If using conditions in D is harder than in Java, than we should rethink their
implementation. Every D object has a built-in mutex. Conditon variables should be able to work with this mutex and with synchronized sections out of the box. The most common case should be the easiest. The fact that you are having all those technical problems proves my point.
 As for non-technical problems, rewrite your test so that you wait on a
condition
in main and let the new thread do the signalling. That way they won't be able to miss each other, as they currently do. I got my test program to work and understand what I did wrong. I think the real problem is that: 1. core.sync isn't even in the Phobos/druntime docs yet, and is still kind of "here be dragons", so I originally thought it might legitimately just be broken. 2. The docs for core.sync.condition assume you already know what a condition variable is and how to use one and just provide a terse description of the interface to the druntime implementation with zero examples. On the other hand, the details of this primitive are somewhat arcane and hard to find. I'd vote for improving these and will do it myself if noone else does it by the time I get some higher priority hacking done.
Oct 21 2009
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 21 Oct 2009 17:13:48 -0400, Bartosz Milewski  
<bartosz-nospam relisoft.com> wrote:

 dsimcha Wrote:

 Bartosz, since you're a threading guru, could you please write a simple  
 test
 program using core.sync.condition and see if it works, and if not  
 either file a
 bug report or let me know?
Conditions were implemented by Sean Kelly, so he's the guru. The file condition.d in core.sync has unit tests, which presumably still run (although I don't know if anybody still runs unit tests in druntime). If using conditions in D is harder than in Java, than we should rethink their implementation. Every D object has a built-in mutex. Conditon variables should be able to work with this mutex and with synchronized sections out of the box. The most common case should be the easiest. The fact that you are having all those technical problems proves my point.
IIRC, Conditions should be able to use the in-object mutex. However, I think you still need a core.sync.Mutex object. I think the way it works is you initialize the Mutex with an object, and it inserts itself as the in-object mutex (hopefully before the on-demand mutex is created). So indirectly, you can do: new Condition(new Mutex(this)); Kinda odd, I agree... This should probably be shortcutted. Sean? -Steve
Oct 21 2009