www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - synchronized (this[.classinfo]) in druntime and phobos

reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
Hi,

I've seen several occurrences of synchronized (this) and synchronized 
(this.classinfo) in both druntime and phobos by now. I propose that we 
officially ban these patterns with extreme prejudice.

1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happens 
to use the object for locking will most likely end up with a deadlock on 
their hands.
2) Locking on something as fundamental as type info means that any 
arbitrary part of the application could cause a deadlock by doing the same.

The solution to (1) is to simply use a Mutex internally (or an Object 
with synchronized statements), and for (2), to simply use private global 
objects.

(Now, regarding (1), you might argue that anyone who locks on an 
arbitrary object is doing it wrong, but you can't really blame them - 
it's frankly D's fault for allowing monitors on arbitrary objects, which 
is a horrible mess.)

Anyone against this?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org
May 28 2012
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, May 29, 2012 00:36:13 Alex R=C3=B8nne Petersen wrote:
 Hi,
=20
 I've seen several occurrences of synchronized (this) and synchronized=

 (this.classinfo) in both druntime and phobos by now. I propose that w=

 officially ban these patterns with extreme prejudice.
=20
 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happ=

 to use the object for locking will most likely end up with a deadlock=

 their hands.
 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the=

=20
 The solution to (1) is to simply use a Mutex internally (or an Object=

 with synchronized statements), and for (2), to simply use private glo=

 objects.
=20
 (Now, regarding (1), you might argue that anyone who locks on an
 arbitrary object is doing it wrong, but you can't really blame them -=

 it's frankly D's fault for allowing monitors on arbitrary objects, wh=

 is a horrible mess.)
=20
 Anyone against this?

Don't synchronized classes synchronize on the object for all of their f= unction=20 calls? How is this any different? - Jonathan M Davis
May 28 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 00:42, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 00:36:13 Alex Rønne Petersen wrote:
 Hi,

 I've seen several occurrences of synchronized (this) and synchronized
 (this.classinfo) in both druntime and phobos by now. I propose that we
 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happens
 to use the object for locking will most likely end up with a deadlock on
 their hands.
 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the same.

 The solution to (1) is to simply use a Mutex internally (or an Object
 with synchronized statements), and for (2), to simply use private global
 objects.

 (Now, regarding (1), you might argue that anyone who locks on an
 arbitrary object is doing it wrong, but you can't really blame them -
 it's frankly D's fault for allowing monitors on arbitrary objects, which
 is a horrible mess.)

 Anyone against this?

Don't synchronized classes synchronize on the object for all of their function calls? How is this any different? - Jonathan M Davis

I have no idea how synchronized classes work; they are not a documented feature of the language. We have synchronized functions which synchronize on the this reference. Perhaps synchronized classes just make all functions in a class do this. Either way, this is a fundamental language design fallacy. This is anti-pattern number one when it comes to locking: * http://stackoverflow.com/a/251668/438034 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and SyncLock Keywords section) -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 28 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

Per TDPL, having individually synchronized functions is illegal. Either all of the functions in a class are synchronized or none of them are. Putting synchronized on a function should actually be illegal. It belongs only on classes or in synchronized blocks, never on functions. Now, unfortuntately, I don't believe that the compiler enforces any of that right now, so you end up synchronizing indivdual functions rather than whole classes, but the synchronized functions themselves should function the same.
 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

Well, then you should probably be arguing about the design of synchronized classes/functions rather than synchronized(this). However, given the design of synchronized classes, synchronized(this) would probably never be appropriate. If you're using a synchronized class, then you don't need it. And if you're not using a synchronized class, then you shouldn't be synchronizing your class. I would definitely think that synchronized blocks should synchronize on something else, since they're trying to lock a much smaller area than an entire class. - Jonathan M Davis

I don't think arguing about them makes sense at this point. Way too much code would break if we changed the semantics. I'd consider it a mistake and a lesson learned, rather. But I take it you agree that synchronized (this) and similar "uncontrollable" synchronization blocks should be avoided? -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 28 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 01:35, Alex Rønne Petersen wrote:
 On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

Per TDPL, having individually synchronized functions is illegal. Either all of the functions in a class are synchronized or none of them are. Putting synchronized on a function should actually be illegal. It belongs only on classes or in synchronized blocks, never on functions. Now, unfortuntately, I don't believe that the compiler enforces any of that right now, so you end up synchronizing indivdual functions rather than whole classes, but the synchronized functions themselves should function the same.
 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

Well, then you should probably be arguing about the design of synchronized classes/functions rather than synchronized(this). However, given the design of synchronized classes, synchronized(this) would probably never be appropriate. If you're using a synchronized class, then you don't need it. And if you're not using a synchronized class, then you shouldn't be synchronizing your class. I would definitely think that synchronized blocks should synchronize on something else, since they're trying to lock a much smaller area than an entire class. - Jonathan M Davis

I don't think arguing about them makes sense at this point. Way too much code would break if we changed the semantics. I'd consider it a mistake and a lesson learned, rather.

I should probably add that Java learned it long ago, and yet we adopted it anyway... blergh.
 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 28 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

The "lesson learned" from Java that TDPL enumerates is the mistake of having synchronized on functions rather than entire classes, but clearly even that is currently TDPL-only and not actually properly implemented yet. - Jonathan M Davis

But synchronized on entire classes is exactly equally flawed... the fundamental problem is still that they synchronize on a public resource. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 28 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 18:41, Jess wrote:
 On Monday, 28 May 2012 at 23:55:04 UTC, Alex Rønne Petersen wrote:
 On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

The "lesson learned" from Java that TDPL enumerates is the mistake of having synchronized on functions rather than entire classes, but clearly even that is currently TDPL-only and not actually properly implemented yet. - Jonathan M Davis

But synchronized on entire classes is exactly equally flawed... the fundamental problem is still that they synchronize on a public resource.

Sorry, I don't quite follow... 1) locking on a public resource => BAD, high risk. 2) Adding GC.callLocked(), allowing a generic delegate to claim the GC lock https://github.com/D-Programming-Language/druntime/pull/213 => no-brainer, i.e. no risk?

druntime is the low-level runtime library of D. This discussion is about high-level class design. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 18:57, Alex Rønne Petersen wrote:
 On 29-05-2012 18:41, Jess wrote:
 On Monday, 28 May 2012 at 23:55:04 UTC, Alex Rønne Petersen wrote:
 On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

The "lesson learned" from Java that TDPL enumerates is the mistake of having synchronized on functions rather than entire classes, but clearly even that is currently TDPL-only and not actually properly implemented yet. - Jonathan M Davis

But synchronized on entire classes is exactly equally flawed... the fundamental problem is still that they synchronize on a public resource.

Sorry, I don't quite follow... 1) locking on a public resource => BAD, high risk. 2) Adding GC.callLocked(), allowing a generic delegate to claim the GC lock https://github.com/D-Programming-Language/druntime/pull/213 => no-brainer, i.e. no risk?

druntime is the low-level runtime library of D. This discussion is about high-level class design.

I should also add that the reason I started this thread was to prevent common locking errors induced by the anti-patterns synchronized causes. If you're calling *anything* in the core.memory module, I sure hope you know what you're doing in general. I really don't think these two issues are connected at all. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 19:37, Jess wrote:
 On Tuesday, 29 May 2012 at 17:01:01 UTC, Alex Rønne Petersen wrote:

druntime is the low-level runtime library of D. This discussion is about high-level class design.

I should also add that the reason I started this thread was to prevent common locking errors induced by the anti-patterns synchronized causes. If you're calling *anything* in the core.memory module, I sure hope you know what you're doing in general. I really don't think these two issues are connected at all.

ok, maybe I read too much into your statement below? it sounded as if you wanted to ban this usage from druntime? Alex Rønne Petersen wrote: "I've seen several occurrences of synchronized (this) and synchronized (this.classinfo) in both druntime and phobos by now. I propose that we officially ban these patterns with extreme prejudice."

Yes, I do. But I'm not sure what that has to do with my GC.callLocked() PR? -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 On 29-05-2012 01:35, Alex Rønne Petersen wrote:
 On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

Per TDPL, having individually synchronized functions is illegal. Either all of the functions in a class are synchronized or none of them are. Putting synchronized on a function should actually be illegal. It belongs only on classes or in synchronized blocks, never on functions. Now, unfortuntately, I don't believe that the compiler enforces any of that right now, so you end up synchronizing indivdual functions rather than whole classes, but the synchronized functions themselves should function the same.
 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

Well, then you should probably be arguing about the design of synchronized classes/functions rather than synchronized(this). However, given the design of synchronized classes, synchronized(this) would probably never be appropriate. If you're using a synchronized class, then you don't need it. And if you're not using a synchronized class, then you shouldn't be synchronizing your class. I would definitely think that synchronized blocks should synchronize on something else, since they're trying to lock a much smaller area than an entire class. - Jonathan M Davis

I don't think arguing about them makes sense at this point. Way too much code would break if we changed the semantics. I'd consider it a mistake and a lesson learned, rather.

I should probably add that Java learned it long ago, and yet we adopted it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat previously done errors.
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat previously done errors.

So what is the lesson Java learned, and how does it address multithreaded programming in wake of that lesson? Andrei
May 29 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat previously done errors.

So what is the lesson Java learned, and how does it address multithreaded programming in wake of that lesson? Andrei

It learned that allowing locking on arbitrary objects makes controlling locking (and thus reducing the chance for deadlocks) impossible. It goes against our TLS-by-default and message-passing concurrency design. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat previously done errors.

So what is the lesson Java learned, and how does it address multithreaded programming in wake of that lesson? Andrei

It learned that allowing locking on arbitrary objects makes controlling locking (and thus reducing the chance for deadlocks) impossible.

And how does Java address multithreading in general, and those issues in particular, today? Andrei
May 29 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat previously done errors.

So what is the lesson Java learned, and how does it address multithreaded programming in wake of that lesson? Andrei

It learned that allowing locking on arbitrary objects makes controlling locking (and thus reducing the chance for deadlocks) impossible.

And how does Java address multithreading in general, and those issues in particular, today? Andrei

It doesn't, and neither does C#. Java still encourages using synchronized, and C# still encourages using lock, but many prominent figures in those programming language communities have written blog posts on why these language constructs are evil and should be avoided. Besides, it seems to me that D can't quite make up its mind. We have TLS by default, and we encourage message-passing (through a library mechanism), and then we have the synchronized statement and attribute. It just seems so incredibly inconsistent. synchronized encourages doing the wrong thing (locks and synchronization). -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 3:01 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat previously done errors.

So what is the lesson Java learned, and how does it address multithreaded programming in wake of that lesson? Andrei

It learned that allowing locking on arbitrary objects makes controlling locking (and thus reducing the chance for deadlocks) impossible.

And how does Java address multithreading in general, and those issues in particular, today? Andrei

It doesn't, and neither does C#. Java still encourages using synchronized, and C# still encourages using lock, but many prominent figures in those programming language communities have written blog posts on why these language constructs are evil and should be avoided.

Some citations (beyond the known fallacies of Java 1.0) would be great. Thanks.
 Besides, it seems to me that D can't quite make up its mind. We have TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

Each paradigm has its place. Lock-based programming is definitely here to stay, and when the paradigm is needed, doing it with synchronized objects is better than most alternatives. Andrei
May 29 2012
parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 00:53, Andrei Alexandrescu a écrit :
 On 5/29/12 3:01 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat previously done errors.

So what is the lesson Java learned, and how does it address multithreaded programming in wake of that lesson? Andrei

It learned that allowing locking on arbitrary objects makes controlling locking (and thus reducing the chance for deadlocks) impossible.

And how does Java address multithreading in general, and those issues in particular, today? Andrei

It doesn't, and neither does C#. Java still encourages using synchronized, and C# still encourages using lock, but many prominent figures in those programming language communities have written blog posts on why these language constructs are evil and should be avoided.

Some citations (beyond the known fallacies of Java 1.0) would be great. Thanks.
 Besides, it seems to me that D can't quite make up its mind. We have TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

Each paradigm has its place. Lock-based programming is definitely here to stay, and when the paradigm is needed, doing it with synchronized objects is better than most alternatives. Andrei

No ! You don't want to synchronize on ANY object. You want to synchronize on explicit mutexes. See that link for instance : http://msdn.microsoft.com/en-us/library/ms173179.aspx It is from microsoft, you can't assume it is C# bashing. It says : "Generally, it is best to avoid locking on a public type, or on object instances beyond the control of your application. For example, lock(this) can be problematic if the instance can be accessed publicly, because code beyond your control may lock on the object as well. This could create deadlock situations where two or more threads wait for the release of the same object. Locking on a public data type, as opposed to an object, can cause problems for the same reason." Given example also create dumb object to lock on them instead of using this/any object. Time has proven it is the way to go. I have tracked race condition in large java codebase, and I conclude the same thing as microsoft's guys.
May 30 2012
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 14:32, Regan Heath a écrit :
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com> wrote:

 Le 30/05/2012 00:53, Andrei Alexandrescu a écrit :
 On 5/29/12 3:01 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat previously done errors.

So what is the lesson Java learned, and how does it address multithreaded programming in wake of that lesson? Andrei

It learned that allowing locking on arbitrary objects makes controlling locking (and thus reducing the chance for deadlocks) impossible.

And how does Java address multithreading in general, and those issues in particular, today? Andrei

It doesn't, and neither does C#. Java still encourages using synchronized, and C# still encourages using lock, but many prominent figures in those programming language communities have written blog posts on why these language constructs are evil and should be avoided.

Some citations (beyond the known fallacies of Java 1.0) would be great. Thanks.
 Besides, it seems to me that D can't quite make up its mind. We have
 TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

Each paradigm has its place. Lock-based programming is definitely here to stay, and when the paradigm is needed, doing it with synchronized objects is better than most alternatives. Andrei

No ! You don't want to synchronize on ANY object. You want to synchronize on explicit mutexes.

+1 .. this is the key point/issue. If all objects can be locked, and if synchronized classes/methods use the /same/ mutex you can easily/accidentally have hard to find deadlocks. The deadlocks in Q are usually (at least) two threads locking (at least) two objects in the opposite order and with synchronized classes/methods the fact that a lock is being taken is not always obvious from the code (at the call site) so the problem code can look fairly innocuous. **[Example 1]** class C { synchronized void ccc() { } } class D { synchronized void ddd() { } } C c = new C(); D d = new D(); [thread1] ..lock(c) // locks c { // deadlock window d.ddd(); // locks d } [thread2] ..lock(d) // locks d { // deadlock window c.ccc(); // locks c } **[Example 2]** class A { B b; .. synchronized void foo() // locks a { // deadlock window b.bar(); // locks b } } class B { A a; .. synchronized void bar() // locks b { // deadlock window a.foo(); // locks a } } A a = new A(); B b = new B(); a.b = b; b.a = a; [thread1] a.foo(); // locks a then b [thread2] b.bar(); // locks b then a So, the root cause of the problem is that synchronized classes/methods use a publicly visible mutex (the object/this pointer) to perform their locking. The solution is to lock on a private mutex, which no-one else can lock unexpectedly as described in the link below:
 See that link for instance :
 http://msdn.microsoft.com/en-us/library/ms173179.aspx

Now, ideally we want to make it hard for people to screw this up in this manner and there are several ways to do it. 1. Prevent locking on any/every object. People would have to create a separate mutex object for locking. This is a little tedious, but it does make people think about the scope of the locking (where to put the mutex, who can see/use it, etc). On the flipside it can make people re-use a mutex for multiple conceptual critical section areas of code, which is less than ideal, but at least it's not 'broken'.

It is suboptimal, but correct.
 2. Allow locking on any/every object but use a different mutex for
 synchronized class/methods. So 'synchronized' on a method call locks a
 private mutex object, not the object itself. And synchronized(object)
 locks the public mutex object. The downside here is that now every
 object potentially has 2 mutex objects/handles internally - a public and
 a private one.

It doesn't address most of the drawback cited. Notably the fact that every object have a monitor field, but most of them will not use it.
May 30 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 14:54, Thiez wrote:
 On Wednesday, 30 May 2012 at 12:43:15 UTC, deadalnix wrote:
 It doesn't address most of the drawback cited. Notably the fact that
 every object have a monitor field, but most of them will not use it.

That could be solved without abandoning object locks. If locking on all objects is allowed, objects in TLS could simply omit the monitor field. Locking would then be performed as: if (object not in TLS) { insert expensive locking operation here } return Yes, this would introduce an extra conditional jump whenever a lock is acquired/released, but the overhead is very small compared to the relatively expensive lock/unlock operation.

I don't understand how that would work. Where would you store the monitor reference? -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 15:10, Regan Heath a écrit :
 IMO, the wasted space used by the monitor isn't ideal, but it's not a
 "problem" for the general case. It is a problem for people with limited
 memory environments but that's another topic. Talking about both is only
 clouding the discussional waters so to speak.

 R

As the GC allocate by power of 2, this can go pretty bad. It did happen to me (once to be fair, it is probably not the every day problem) that my memory consumption exploded because of the monitor field.
May 30 2012
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 5:43 AM, deadalnix wrote:
 It doesn't address most of the drawback cited. Notably the fact that
 every object have a monitor field, but most of them will not use it.

Once the TDPL design is implemented, we could look into eliminating that field for all but "synchronized" class. In the TDPL design, "synchronized" is a class-level attribute. Andrei
May 30 2012
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 6:10 AM, Regan Heath wrote:
 Ultimately the "problem" that needs solving is the fact that
 synchronized classes/methods use a public mutex and this exposure makes
 deadlocks more likely. I was hoping that by giving some examples in
 code, it would become clearer to the people who don't see what all the
 fuss is about.

How do free-floating mutexes that are not explicitly associated with _any_ data make deadlocks less likely? In my experience the less structured mutexes are worse, not better.
 Yes, having to manually create a mutex is a pain, and as Andrei has
 mentioned a few times having 2 separate things (one object to lock, one
 mutex) which conceptually should be one thing is worse from an
 understanding/maintenance point of view, however..

 1. If the mutex is intended to lock a single object then we can make the
 following options available:
 a. A Lockable class to derive from.
 b. A Lockable struct to compose with (assuming here that the struct will
 be created/destroyed with the object).
 c. A Lockable!(T) wrapper template/struct/class to wrap objects to be
 locked.
 d. A Lockable interface which classes may implement.
 e. Some sort of compiler magic where it supplies the mutex for
 objects/methods marked with "synchronized" or involved in a
 synchronized(object) statement.

 2. If the scope of the locking is greater than a single object, then a
 separate mutex is required/desired anyway.

But all of these are available in the current design. You have synchronized classes for the likes of 1a and the core mutex for creating all others except e.
 1a. would contain a mutex primitive and lock/tryLock/unlock methods (it
 would implement the interface in 1d)

 1b. would also contain a mutex primitive and alias could be used to
 expose the lock/tryLock/unlock methods (it would implement the interface
 in 1d)

 1c. is actually a technique I have used a lot in C++ where the template
 class is created on the stack of a function/critical section and it
 (optionally) locks on creation (or later when lock() is called) and
 always unlocks on destruction as it leaves scope (by any means). It's a
 very robust pattern, and is similar (the inspiration/or result) of/to
 the synchronized block itself.

Yah, we use it a lot at Facebook. We probably should have such a wrapper in the standard library.
 IMO, the wasted space used by the monitor isn't ideal, but it's not a
 "problem" for the general case. It is a problem for people with limited
 memory environments but that's another topic. Talking about both is only
 clouding the discussional waters so to speak.

I agree. Andrei
May 30 2012
prev sibling parent =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 30-05-2012 15:10, Regan Heath wrote:
 On Wed, 30 May 2012 13:43:14 +0100, deadalnix <deadalnix gmail.com> wrote:
 Le 30/05/2012 14:32, Regan Heath a écrit :
 1. Prevent locking on any/every object. People would have to create a
 separate mutex object for locking. This is a little tedious, but it does
 make people think about the scope of the locking (where to put the
 mutex, who can see/use it, etc). On the flipside it can make people
 re-use a mutex for multiple conceptual critical section areas of code,
 which is less than ideal, but at least it's not 'broken'.

It is suboptimal, but correct.

Indeed, and can be made optimal with some fine-tuning/work.
 2. Allow locking on any/every object but use a different mutex for
 synchronized class/methods. So 'synchronized' on a method call locks a
 private mutex object, not the object itself. And synchronized(object)
 locks the public mutex object. The downside here is that now every
 object potentially has 2 mutex objects/handles internally - a public and
 a private one.

It doesn't address most of the drawback cited. Notably the fact that every object have a monitor field, but most of them will not use it.

True, I was just mentioning it as an option that solves the key issue/problem - not as the best solution to that problem. I think #1 is better. Ultimately the "problem" that needs solving is the fact that synchronized classes/methods use a public mutex and this exposure makes deadlocks more likely. I was hoping that by giving some examples in code, it would become clearer to the people who don't see what all the fuss is about. Yes, having to manually create a mutex is a pain, and as Andrei has mentioned a few times having 2 separate things (one object to lock, one mutex) which conceptually should be one thing is worse from an understanding/maintenance point of view, however.. 1. If the mutex is intended to lock a single object then we can make the following options available: a. A Lockable class to derive from. b. A Lockable struct to compose with (assuming here that the struct will be created/destroyed with the object). c. A Lockable!(T) wrapper template/struct/class to wrap objects to be locked. d. A Lockable interface which classes may implement. e. Some sort of compiler magic where it supplies the mutex for objects/methods marked with "synchronized" or involved in a synchronized(object) statement. 2. If the scope of the locking is greater than a single object, then a separate mutex is required/desired anyway. 1a. would contain a mutex primitive and lock/tryLock/unlock methods (it would implement the interface in 1d) 1b. would also contain a mutex primitive and alias could be used to expose the lock/tryLock/unlock methods (it would implement the interface in 1d) 1c. is actually a technique I have used a lot in C++ where the template class is created on the stack of a function/critical section and it (optionally) locks on creation (or later when lock() is called) and always unlocks on destruction as it leaves scope (by any means). It's a very robust pattern, and is similar (the inspiration/or result) of/to the synchronized block itself.

In D, you can kind of already do that: void foo() { mtx.lock(); scope (exit) mtx.unlock(); // ... } It does involve that one extra scope (exit) line, but I'm of the opinion that being explicit about that is a good thing.
 1d. could be a requirement for classes which are to be used in
 synchronized statements i.e.

 class A : Lockable
 {
 void lock() {}
 void unlock() {}
 }

 A a = new A();
 synchronized(a)
 { // calls a.lock()
 ...
 } // calls a.unlock()

 1e. I leave as a opener to Walter/Andrei/the reader .. I've seen many a
 genius idea pop out of nowhere on this forum.

 IMO, the wasted space used by the monitor isn't ideal, but it's not a
 "problem" for the general case. It is a problem for people with limited
 memory environments but that's another topic. Talking about both is only
 clouding the discussional waters so to speak.

 R

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com> wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

+1 .. this is the key point/issue.

TDPL's design only allows for entire synchronized classes (not separate synchronized and unsynchronized methods), which pair mutexes with the data they protect. This is more restrictive than exposing mutexes, but in a good way. We use such a library artifact in C++ at Facebook all the time, to great success. People shouldn't create designs that have synchronized classes referring to one another naively. Designing with mutexes (explicit or implicit) will always create the possibility of deadlock, so examples how that could happen are easy to come across. TDPL improves on deadlocks by introducing synchronized statements with more than one argument, see 13.15. The implicit mutexes used by synchronized classes are recursive. Andrei
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 9:43 AM, Regan Heath wrote:
 On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>
 wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

+1 .. this is the key point/issue.

TDPL's design only allows for entire synchronized classes (not separate synchronized and unsynchronized methods), which pair mutexes with the data they protect. This is more restrictive than exposing mutexes, but in a good way. We use such a library artifact in C++ at Facebook all the time, to great success.

Can you call pass them to a synchronized statement? i.e. TDPLStyleSynchClass a = new TDPLStyleSynchClass(); synchronized(a) { }

Yes. Well I recommend acquiring the text! :o)
 ... because, if you can, then you're exposing the mutex.

No.
 People shouldn't create designs that have synchronized classes
 referring to one another naively. Designing with mutexes (explicit or
 implicit) will always create the possibility of deadlock, so examples
 how that could happen are easy to come across.

True. But in my Example 1

Your example 1 should not compile.
 TDPL improves on deadlocks by introducing synchronized statements with
 more than one argument, see 13.15.

Is there anywhere I can see this online? (for free :p)

http://goo.gl/ZhPM2
 The implicit mutexes used by synchronized classes are recursive.

:) why would you want anything else.

Efficiency. Andrei
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 10:40 AM, Regan Heath wrote:
 On Wed, 30 May 2012 18:16:38 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 9:43 AM, Regan Heath wrote:
 On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>
 wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

+1 .. this is the key point/issue.

TDPL's design only allows for entire synchronized classes (not separate synchronized and unsynchronized methods), which pair mutexes with the data they protect. This is more restrictive than exposing mutexes, but in a good way. We use such a library artifact in C++ at Facebook all the time, to great success.

Can you call pass them to a synchronized statement? i.e. TDPLStyleSynchClass a = new TDPLStyleSynchClass(); synchronized(a) { }

Yes. Well I recommend acquiring the text! :o)
 ... because, if you can, then you're exposing the mutex.

No.

For the purposes of this thread, and the anti-pattern/problem we're discussing, you are.

No. I explained in my previous post that the synchronized statement does not expose locks. This is not a matter of opinion.
 It is the combination of synchronized
 classes/methods (implicit locking) and external synchronized statements
 (explicit locking) which result in the unexpected, accidental, and hard
 to see deadlocks we're talking about here.

You can have deadlocks but with synchronized you can't leak locks or doubly-unlock them. With free mutexes you have all of the above.
 People shouldn't create designs that have synchronized classes
 referring to one another naively. Designing with mutexes (explicit or
 implicit) will always create the possibility of deadlock, so examples
 how that could happen are easy to come across.

True. But in my Example 1

Your example 1 should not compile.

o.. k.. I expected you would get my meaning with the simplified example. Here you are:

[snip]
 Runs and deadlocks immediately.

Sure. As I said, synchronized helps with scoping the locks and unlocks, but not with deadlocks. You can rewrite the example with two bare mutexes just as well. Andrei
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

I'm not suggesting using free mutexes. I'm suggesting keeping the mutex private inside the object.

Ergo, you are suggesting using free mutexes. Your second sentence destroys the first. Andrei
May 31 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 31/05/2012 11:55, Regan Heath a écrit :
 On Thu, 31 May 2012 10:48:51 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

I'm not suggesting using free mutexes. I'm suggesting keeping the mutex private inside the object.

Ergo, you are suggesting using free mutexes. Your second sentence destroys the first.

Depends on your definition of "free". You appear to have meant as an instance/pointer/object even one in a class, I initially read it as meaning as a separate object from the class you're locking. In any case, you're right the compiler doesn't get synchronized() statements/classes/methods wrong and a programmer can. The trade-off is the cause of this thread of discussion. R

I think you misunderstand each other. Here is what I nderstand : Andrei is talking about free mutex in the sense of a mutex with lock and unlock function available. Such a mutex is error prone, because you can lock/unlock in an incorrect way. synchronized is here a better alternative, because lock and unlock always goes by pair. Regan, by free mutex, you mean a mutex that can be locked and unlocked by everybody. This is problematic too. The ideal is to lock in a way that isn't a free mutex in both definition of the term.
May 31 2012
prev sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 00:35, Jonathan M Davis wrote:
 On Wednesday, May 30, 2012 00:01:49 Alex Rønne Petersen wrote:
 Besides, it seems to me that D can't quite make up its mind. We have TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

It allows multiple approaches at multiple levels. TLS and message passing is the preferred way, and shared combined with synchronized or mutexes to protect it is available when you need it. synchronized classes are more straightforward approach to locking when you need to protect an entire object (and less error-prone in some respects, because you then have the guarantee that it happens for all functions and don't run the risk of not locking in some functions; it also works with inheritance that way, unlike mutexes). sychronized blocks on the other hand are a quick and easy way to protect specific sections of code without having to bother creating mutexes for it. And mutexes are available when you need full control. Which approach you go with depends on what you're doing and what your needs are.

I agree on your point about synchronized blocks, but not about synchronized classes. Synchronized classes lock on the this reference, which is exactly what should be avoided.
 Now, I could definitely see an argument that using shared is more low level and
 that it would be just simpler to only have mutexes and no synchronized, but we
 ended up with a more tiered approach, and that's not all bad. Each approach
 has its advantages and disadvantages, and D gives you all of them, so you can
 pick and choose what works best for you.

 - Jonathan M Davis

To be clear, I have nothing against synchronized blocks per se. I think it's fine to have them in the language. But I think that making *every* object a monitor is a horrible mistake (for reasons I outlined in another post in this thread). Synchronized blocks are good because they operate on an implicit, hidden, global mutex. You can't screw up with that. Synchronized blocks that operate on a specific object should be limited somehow so not every object has to have a monitor field. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
  Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with that.

I think there's quite some disconnect here. If there's any anti-pattern in this discussion, it's operating on an implicit, hidden, global mutex. Walter agreed to eliminate that from D, but never got around to it. Andrei
May 29 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

I think there's quite some disconnect here. If there's any anti-pattern in this discussion, it's operating on an implicit, hidden, global mutex. Walter agreed to eliminate that from D, but never got around to it. Andrei

I'd love to hear why you think this design is problematic as opposed to one that lets users accidentally expose synchronization issues to consumers of their API surface, which is what many people end up doing since synchronized (this) or even synchronized (this.classinfo) are allowed at all. (I've seen countless cases of those two horrible abuses of synchronized especially from people asking questions on e.g. IRC.) -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 4:17 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

I think there's quite some disconnect here. If there's any anti-pattern in this discussion, it's operating on an implicit, hidden, global mutex. Walter agreed to eliminate that from D, but never got around to it. Andrei

I'd love to hear why you think this design is problematic as opposed to one that lets users accidentally expose synchronization issues to consumers of their API surface, which is what many people end up doing since synchronized (this) or even synchronized (this.classinfo) are allowed at all. (I've seen countless cases of those two horrible abuses of synchronized especially from people asking questions on e.g. IRC.)

I think the most egregious example is synchronization by global lock done in Python and other languages. It has very nice semantics (stop the world, do something, resume the world), but scales poorly enough to be universally considered a failure. Quite honest I'm shocked that you're advocating it. Andrei
May 29 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 01:22, Andrei Alexandrescu wrote:
 On 5/29/12 4:17 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

I think there's quite some disconnect here. If there's any anti-pattern in this discussion, it's operating on an implicit, hidden, global mutex. Walter agreed to eliminate that from D, but never got around to it. Andrei

I'd love to hear why you think this design is problematic as opposed to one that lets users accidentally expose synchronization issues to consumers of their API surface, which is what many people end up doing since synchronized (this) or even synchronized (this.classinfo) are allowed at all. (I've seen countless cases of those two horrible abuses of synchronized especially from people asking questions on e.g. IRC.)

I think the most egregious example is synchronization by global lock done in Python and other languages. It has very nice semantics (stop the world, do something, resume the world), but scales poorly enough to be universally considered a failure. Quite honest I'm shocked that you're advocating it. Andrei

I'm not advocating it. I don't use it myself. I'm just saying that I find it to be less of an issue than synchronizing on arbitrary objects, because the latter is error-prone. And yes, Python is a textbook example of synchronization gone completely wrong. But that doesn't mean that you don't sometimes have to synchronize on some global resource, and in those cases, it can be useful syntactic sugar (but certainly not essential). -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 01:25, Alex Rønne Petersen a écrit :
 On 30-05-2012 01:22, Andrei Alexandrescu wrote:
 On 5/29/12 4:17 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

I think there's quite some disconnect here. If there's any anti-pattern in this discussion, it's operating on an implicit, hidden, global mutex. Walter agreed to eliminate that from D, but never got around to it. Andrei

I'd love to hear why you think this design is problematic as opposed to one that lets users accidentally expose synchronization issues to consumers of their API surface, which is what many people end up doing since synchronized (this) or even synchronized (this.classinfo) are allowed at all. (I've seen countless cases of those two horrible abuses of synchronized especially from people asking questions on e.g. IRC.)

I think the most egregious example is synchronization by global lock done in Python and other languages. It has very nice semantics (stop the world, do something, resume the world), but scales poorly enough to be universally considered a failure. Quite honest I'm shocked that you're advocating it. Andrei

I'm not advocating it. I don't use it myself. I'm just saying that I find it to be less of an issue than synchronizing on arbitrary objects, because the latter is error-prone. And yes, Python is a textbook example of synchronization gone completely wrong. But that doesn't mean that you don't sometimes have to synchronize on some global resource, and in those cases, it can be useful syntactic sugar (but certainly not essential).

I have to agree with Andrei here. The global mutex isn't the solution either. The mutex should be explicit, and under control (not this).
May 30 2012
parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 11:15, deadalnix wrote:
 Le 30/05/2012 01:25, Alex Rønne Petersen a écrit :
 On 30-05-2012 01:22, Andrei Alexandrescu wrote:
 On 5/29/12 4:17 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

I think there's quite some disconnect here. If there's any anti-pattern in this discussion, it's operating on an implicit, hidden, global mutex. Walter agreed to eliminate that from D, but never got around to it. Andrei

I'd love to hear why you think this design is problematic as opposed to one that lets users accidentally expose synchronization issues to consumers of their API surface, which is what many people end up doing since synchronized (this) or even synchronized (this.classinfo) are allowed at all. (I've seen countless cases of those two horrible abuses of synchronized especially from people asking questions on e.g. IRC.)

I think the most egregious example is synchronization by global lock done in Python and other languages. It has very nice semantics (stop the world, do something, resume the world), but scales poorly enough to be universally considered a failure. Quite honest I'm shocked that you're advocating it. Andrei

I'm not advocating it. I don't use it myself. I'm just saying that I find it to be less of an issue than synchronizing on arbitrary objects, because the latter is error-prone. And yes, Python is a textbook example of synchronization gone completely wrong. But that doesn't mean that you don't sometimes have to synchronize on some global resource, and in those cases, it can be useful syntactic sugar (but certainly not essential).

I have to agree with Andrei here. The global mutex isn't the solution either. The mutex should be explicit, and under control (not this).

I've been arguing for being explicit all along. :) What I've been saying in this particular part of the thread is just that synchronized blocks that don't operate on a specific monitor object are less error-prone. Nothing else. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 10:20 AM, Martin Nowak wrote:
 On Wed, 30 May 2012 01:10:41 +0200, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

I think there's quite some disconnect here. If there's any anti-pattern in this discussion, it's operating on an implicit, hidden, global mutex. Walter agreed to eliminate that from D, but never got around to it.

They're not really global, it's one per synchronized block.

That means global. They're terrible and should be eliminated from the language. Andrei
May 30 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 19:22, Andrei Alexandrescu wrote:
 On 5/30/12 10:20 AM, Martin Nowak wrote:
 On Wed, 30 May 2012 01:10:41 +0200, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

I think there's quite some disconnect here. If there's any anti-pattern in this discussion, it's operating on an implicit, hidden, global mutex. Walter agreed to eliminate that from D, but never got around to it.

They're not really global, it's one per synchronized block.

That means global. They're terrible and should be eliminated from the language. Andrei

So you do agree that explicit synchronization is better? Or only in this particular case? -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 10:34 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 19:22, Andrei Alexandrescu wrote:
 On 5/30/12 10:20 AM, Martin Nowak wrote:
 On Wed, 30 May 2012 01:10:41 +0200, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

I think there's quite some disconnect here. If there's any anti-pattern in this discussion, it's operating on an implicit, hidden, global mutex. Walter agreed to eliminate that from D, but never got around to it.

They're not really global, it's one per synchronized block.

That means global. They're terrible and should be eliminated from the language. Andrei

So you do agree that explicit synchronization is better?

No. It's worse most of the time, and usefully more flexible sometimes. Andrei
May 30 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, May 30, 2012 00:01:49 Alex Rønne Petersen wrote:
 Besides, it seems to me that D can't quite make up its mind. We have TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

It allows multiple approaches at multiple levels. TLS and message passing is the preferred way, and shared combined with synchronized or mutexes to protect it is available when you need it. synchronized classes are more straightforward approach to locking when you need to protect an entire object (and less error-prone in some respects, because you then have the guarantee that it happens for all functions and don't run the risk of not locking in some functions; it also works with inheritance that way, unlike mutexes). sychronized blocks on the other hand are a quick and easy way to protect specific sections of code without having to bother creating mutexes for it. And mutexes are available when you need full control. Which approach you go with depends on what you're doing and what your needs are. Now, I could definitely see an argument that using shared is more low level and that it would be just simpler to only have mutexes and no synchronized, but we ended up with a more tiered approach, and that's not all bad. Each approach has its advantages and disadvantages, and D gives you all of them, so you can pick and choose what works best for you. - Jonathan M Davis
May 29 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 29/05/2012 23:54, Andrei Alexandrescu a écrit :
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat previously done errors.

So what is the lesson Java learned, and how does it address multithreaded programming in wake of that lesson? Andrei

It learned that allowing locking on arbitrary objects makes controlling locking (and thus reducing the chance for deadlocks) impossible.

And how does Java address multithreading in general, and those issues in particular, today? Andrei

In java, you basically have no concurency built-in. Everything is shared by default, you can lock on anything (which is deadlock and liquid lock prone), you have dumb primitive like wait and notify that are perfect to cause race conditions. Don't get me wrong, I do a lot of java code, and it isn't that bad in general. But for concurrency, this isn't the model we want. In a more general scope, the problem of java is to believe that everything is OOP, including concurrency, when they are, in fact, different topics. D already have much better tools that the one java provide (std.concurency, std.parallelism, TLS by default, transitive type qualifiers, . . .) that most these thing taken from java don't make any sense now. For instance, what is the point of being able to lock on any object when most of them are thread local ??
May 30 2012
parent mta`chrono <chrono mta-international.net> writes:
Am 30.05.2012 11:11, schrieb deadalnix:
 
 D already have much better tools that the one java provide
 (std.concurency, std.parallelism, TLS by default, transitive type
 qualifiers, . . .) that most these thing taken from java don't make any
 sense now.
 
 For instance, what is the point of being able to lock on any object when
 most of them are thread local ??

Right! Locking on non-TLS objects doesn't make sense. Perhaps only shared objects should be synchronizeable and thus contain a monitor / pointer to a monitor.
Jun 03 2012
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Wed, 30 May 2012 01:10:41 +0200, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/29/12 4:06 PM, Alex R=C3=B8nne Petersen wrote:
  Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with=


 that.

I think there's quite some disconnect here. If there's any anti-patter=

 in this discussion, it's operating on an implicit, hidden, global mute=

 Walter agreed to eliminate that from D, but never got around to it.

They're not really global, it's one per synchronized block. https://github.com/D-Programming-Language/druntime/blob/master/src/rt/cr= itical_.d They're actually pretty limited/safe because you can't access/compose th= e = underlying lock. import std.stdio, core.thread; void foo() { synchronized { auto t =3D new Thread(&bar); t.start(); t.join(); writeln("foo"); } } void bar() { synchronized writeln("bar"); } void main() { foo(); }
May 30 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, May 29, 2012 01:38:25 Alex R=C3=B8nne Petersen wrote:
 I should probably add that Java learned it long ago, and yet we adopt=

 it anyway... blergh.

The "lesson learned" from Java that TDPL enumerates is the mistake of h= aving=20 synchronized on functions rather than entire classes, but clearly even = that is=20 currently TDPL-only and not actually properly implemented yet. - Jonathan M Davis
May 28 2012
prev sibling next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 01:41, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:35:23 Alex Rønne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way too much
 code would break if we changed the semantics. I'd consider it a mistake
 and a lesson learned, rather.

 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

I'm not an expert on threading stuf, but it would be my opinion that if you're not intending to protect the entire class with locks that it makes no sense to lock on the class itself. You're locking for something specific, in which case, your sychronized block should be locking on something else specific to what you're trying to protect. Certainly, that's how I'd approach it with mutexes. You don't have a mutex for an entire class unless it's actually used for all of the class' functions. Rather, you use mutexes specific to what you're trying to protect. - Jonathan M Davis

Right, but even if you really *are* protecting the entire class, you can still create mysterious deadlocks if users of your code lock on your class. So I'm arguing that no matter the use case, never lock on a 'this' reference exposed outside of some API layer. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 28 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 29.05.2012 16:07, Regan Heath wrote:
 According to the docs here:
 http://dlang.org/class.html#synchronized-functions

 A synchronized function locks "this" and as "this" is exposed
 publicly... In the following code the "lock" statement and "synchronized
 void bar" lock the same mutex.

 class Foo {
 synchronized void bar() { ...statements... }
 }

 void main()
 {
 Foo foo = new Foo();
 lock(foo)
 {
 ...statements...
 }
 }

 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.


I'd be darned but every Object in D has monitor fields. If I'm not mistaken it's the mutex you are looking for ;) -- Dmitry Olshansky
May 29 2012
parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 29-05-2012 14:19, Dmitry Olshansky wrote:
 On 29.05.2012 16:07, Regan Heath wrote:
 According to the docs here:
 http://dlang.org/class.html#synchronized-functions

 A synchronized function locks "this" and as "this" is exposed
 publicly... In the following code the "lock" statement and "synchronized
 void bar" lock the same mutex.

 class Foo {
 synchronized void bar() { ...statements... }
 }

 void main()
 {
 Foo foo = new Foo();
 lock(foo)
 {
 ...statements...
 }
 }

 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.


I'd be darned but every Object in D has monitor fields. If I'm not mistaken it's the mutex you are looking for ;)

Indeed they do, and therefore each object must eat an entire word of memory for questionable gain. Generalized object monitors is the worst idea in programming language and virtual machine design, ever. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 29.05.2012 16:26, Alex Rønne Petersen wrote:
 On 29-05-2012 14:19, Dmitry Olshansky wrote:
 On 29.05.2012 16:07, Regan Heath wrote:
 According to the docs here:
 http://dlang.org/class.html#synchronized-functions

 A synchronized function locks "this" and as "this" is exposed
 publicly... In the following code the "lock" statement and "synchronized
 void bar" lock the same mutex.

 class Foo {
 synchronized void bar() { ...statements... }
 }

 void main()
 {
 Foo foo = new Foo();
 lock(foo)
 {
 ...statements...
 }
 }

 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.


I'd be darned but every Object in D has monitor fields. If I'm not mistaken it's the mutex you are looking for ;)

Indeed they do, and therefore each object must eat an entire word of memory for questionable gain. Generalized object monitors is the worst idea in programming language and virtual machine design, ever.

Agreed, awfuly crippled design for a language with Thread-local by default. Looks like we have 'oh my god, what were they thinking' moment here. If anything I'd rather re-implement the whole v-table infrastructure via mixins, templates & friends. -- Dmitry Olshansky
May 29 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-05-29 14:29, Dmitry Olshansky wrote:

 Agreed, awfuly crippled design for a language with Thread-local by default.
 Looks like we have 'oh my god, what were they thinking' moment here.
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

I'm pretty sure that was decided long before D 1.0 and way way longer before D2 where TLS was introduced. -- /Jacob Carlborg
May 29 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 29.05.2012 17:59, Jacob Carlborg wrote:
 On 2012-05-29 14:29, Dmitry Olshansky wrote:

 Agreed, awfuly crippled design for a language with Thread-local by
 default.
 Looks like we have 'oh my god, what were they thinking' moment here.
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

I'm pretty sure that was decided long before D 1.0 and way way longer before D2 where TLS was introduced.

I know. If anything it's hardly a good excuse, it should have been revisited once TLS was introduced. It still can be now. -- Dmitry Olshansky
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 12:41 PM, Dmitry Olshansky wrote:
 On 29.05.2012 17:59, Jacob Carlborg wrote:
 On 2012-05-29 14:29, Dmitry Olshansky wrote:

 Agreed, awfuly crippled design for a language with Thread-local by
 default.
 Looks like we have 'oh my god, what were they thinking' moment here.
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

I'm pretty sure that was decided long before D 1.0 and way way longer before D2 where TLS was introduced.

I know. If anything it's hardly a good excuse, it should have been revisited once TLS was introduced. It still can be now.

Don't be hatin'. You'd appreciate the matter considerably more after you defined your own language and had its users yell at you for changing even a small detail. The situation is at it is, and there's no need to get agitated. We're all on the same boat, trying to make things work out best. Andrei
May 29 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 1:48, Andrei Alexandrescu wrote:

 I know. If anything it's hardly a good excuse, it should have been
 revisited once TLS was introduced. It still can be now.

Don't be hatin'. You'd appreciate the matter considerably more after you defined your own language and had its users yell at you for changing even a small detail. The situation is at it is, and there's no need to get agitated. We're all on the same boat, trying to make things work out best.

Yeah, no problem. I mean I wasn't bashing D for bugs or flaws (before this point at least) anyway. And it's not like I'm cursing here ;) I just wish it was different. To set things straight I still believe that OOP part of language is not what I want it to be, and thus I just don't use it. Like in the old saying: there is always a choice. Things that stopped me from using it are: - hidden v-table design like C++ and Java, doesn't help things much. Encapsulation is good, but if it has no advantages other then hiding things it's bad. - GCed by default, no provision for custom allocation until very recently - monitor field, and who knows what else (v-table obviously, maybe something else?) - no tail-const, no solutions aside from rebindable and casts. Pointers to struct look so much better in this regard ;) - special slow built-in protocols for equality (it's robust, but I don't need it) - opEquals signature madness (probably fixed?) - final vs virtual, no inlining of known ahead-of-time virtual call - nowdays enforced purity, safe, nothrow whatever. Why should I follow these restrictions - dubious. Observe that all of the above have very few advantages brought to me by compiler: - polymorphism - inheritance - contract inheritance Of the above I only ever needed the first 2. Call me dinosaur. -- Dmitry Olshansky
May 29 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 2:27, Dmitry Olshansky wrote:
 I just wish it was different. To set things straight I still believe
 that OOP part of language is not what I want it to be, and thus I just
 don't use it. Like in the old saying: there is always a choice.

Minor correction, when I first come to try D I used classes, I confess :). The toy project of 2D graphics engine needed resource manager, thus I used Flyweight pattern with lazy-loading/unloding behind the scenes. The concrete resource were struct on C heap but handles were small objects were derived from Resource fro obvious reasons. Brings us back to extra word per object problem, flyweight is exactly kind of pattern that is hit by collateral damage of this decision! OT: Actually algorithms to lazily cache resources are very similar to virtual memory management, same strategies, same action just replace "swap-out page" with "unload resource". TL;DR all of this worked out quite well then I lost interest in the project but certainly not D itself.
 Things that stopped me from using it are:
 - hidden v-table design like C++ and Java, doesn't help things much.
 Encapsulation is good, but if it has no advantages other then hiding
 things it's bad.
 - GCed by default, no provision for custom allocation until very recently
 - monitor field, and who knows what else (v-table obviously, maybe
 something else?)
 - no tail-const, no solutions aside from rebindable and casts. Pointers
 to struct look so much better in this regard ;)
 - special slow built-in protocols for equality (it's robust, but I don't
 need it)
 - opEquals signature madness (probably fixed?)
 - final vs virtual, no inlining of known ahead-of-time virtual call
 - nowdays enforced purity,  safe, nothrow whatever. Why should I follow
 these restrictions - dubious.

 Observe that all of the above have very few advantages brought to me by
 compiler:
 - polymorphism
 - inheritance
 - contract inheritance

 Of the above I only ever needed the first 2. Call me dinosaur.

Throwing in this one: -useless RTTi that still occupies space. (the useless part might have been fixed recently? Some customizable field added to Typeinfo - dunno) -- Dmitry Olshansky
May 30 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-05-29 23:48, Andrei Alexandrescu wrote:

 Don't be hatin'. You'd appreciate the matter considerably more after you
 defined your own language and had its users yell at you for changing
 even a small detail.

 The situation is at it is, and there's no need to get agitated. We're
 all on the same boat, trying to make things work out best.

It seems more and more that D2 is not a designed language. Instead new features are just slapped on without considering how it would impact the rest of the language. -- /Jacob Carlborg
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 2:14 AM, Jacob Carlborg wrote:
 On 2012-05-29 23:48, Andrei Alexandrescu wrote:

 Don't be hatin'. You'd appreciate the matter considerably more after you
 defined your own language and had its users yell at you for changing
 even a small detail.

 The situation is at it is, and there's no need to get agitated. We're
 all on the same boat, trying to make things work out best.

It seems more and more that D2 is not a designed language. Instead new features are just slapped on without considering how it would impact the rest of the language.

What features are you referring to? Andrei
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 11:41 AM, Jacob Carlborg wrote:
 On Wednesday, 30 May 2012 at 15:45:05 UTC, Andrei Alexandrescu wrote:
 On 5/30/12 2:14 AM, Jacob Carlborg wrote:
 It seems more and more that D2 is not a designed language. Instead new
 features are just slapped on without considering how it would impact the
 rest of the language.

What features are you referring to?

The concurrency model as this thread shows. There is no bridge between shared and unshared data like const is to immutable and mutable.

We considered that (maybe_synchronized) , but decided not to go with it amid fear of overcomplicating things.
 Perhaps
 the monitor on every object should have been removed when the new
 concurrency model was designed, as this thread suggests.

This thread does a good job at arguing that scoped locking does not prevent deadlocks, but this is not new or interesting. Unfortunately I fail to derive significant proposed value. There exist type systems that avoid locks. They are very restrictive and difficult to work with.
 "inout" was added long after the const system was added to D. If done
 correctly this should have come up as a problem when designing the const
 system. You cannot apply const/immutable to an object reference in the
 same way as you can to a pointer.

 "const" doesn't play nice with ranges.

I see how these can be annoying, but they're not the result of us not designing things. We designed things best we could. Andrei
May 30 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 21:10, Andrei Alexandrescu wrote:
 On 5/30/12 11:41 AM, Jacob Carlborg wrote:
 On Wednesday, 30 May 2012 at 15:45:05 UTC, Andrei Alexandrescu wrote:
 On 5/30/12 2:14 AM, Jacob Carlborg wrote:
 It seems more and more that D2 is not a designed language. Instead new
 features are just slapped on without considering how it would impact
 the
 rest of the language.

What features are you referring to?

The concurrency model as this thread shows. There is no bridge between shared and unshared data like const is to immutable and mutable.

We considered that (maybe_synchronized) , but decided not to go with it amid fear of overcomplicating things.

The result is a feature that is arguably only useful in small laboratory cases. Are you sure we shouldn't revisit shared and try to work out a bridge between the unshared and shared world? Not to mention that shared is fundamentally x86-biased, which seems to be annoying trend in D development in general...
 Perhaps
 the monitor on every object should have been removed when the new
 concurrency model was designed, as this thread suggests.

This thread does a good job at arguing that scoped locking does not prevent deadlocks, but this is not new or interesting. Unfortunately I fail to derive significant proposed value. There exist type systems that avoid locks. They are very restrictive and difficult to work with.

I can think of 3 languages that have monitors in the "type" system: D, C# (as a result of the CLR's design; mostly because they wanted Java interoperability), and Java. Are you really saying all other type systems are difficult to work with just because of this little thing?
 "inout" was added long after the const system was added to D. If done
 correctly this should have come up as a problem when designing the const
 system. You cannot apply const/immutable to an object reference in the
 same way as you can to a pointer.

 "const" doesn't play nice with ranges.

I see how these can be annoying, but they're not the result of us not designing things. We designed things best we could. Andrei

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 12:17 PM, Alex Rønne Petersen wrote:
 The result is a feature that is arguably only useful in small laboratory
 cases. Are you sure we shouldn't revisit shared and try to work out a
 bridge between the unshared and shared world?

We're as always open to suggestions.
 There exist type systems that avoid locks. They are very restrictive and
 difficult to work with.

I can think of 3 languages that have monitors in the "type" system: D, C# (as a result of the CLR's design; mostly because they wanted Java interoperability), and Java. Are you really saying all other type systems are difficult to work with just because of this little thing?

Apologies, meant "avoid deadlocks" instead of "avoid locks". Andrei
May 30 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-05-30 21:10, Andrei Alexandrescu wrote:

 I see how these can be annoying, but they're not the result of us not
 designing things. We designed things best we could.

I would say it's not good enough. The whole approach of designing the language is wrong. This is how it works today, which is bad: 1. Designing feature X 2. Implementing in the compiler 3. Shipped with the next release Andrei, Walter: "Hey look at this new feature X in the next release, it's great". Community: "Say what now. What did that come from?". Some time later Community: "X is broken in these different ways. X can't integrate with Y, Z, W. Since Phobos doesn't use the feature we can't use it, making it useless" Andrei, Walter: "No, it's perfectly designed" Community: "But it's not working in practice" Andrei, Walter: "Sure it is, end of discussion" This is the bad approach, but there's also a worse approach: 1. Designing feature X 2. Document the feature in TDPL 3. Wait wait wait 4. Community: "Where is feature X, it's in TDPL" 5. Andrei, Walter: "It's not implemented yet/correctly" 6. Repeat step 3-5 a couple of times 7. Implement feature X in the compiler 8. Ship with the next release Community: "X is broken in these different ways. X can't integrate with Y, Z, W. Since Phobos doesn't use the feature we can't use it, making it useless" Andrei, Walter: "No, it's perfectly designed" Community: "But it's not working in practice" Andrei, Walter: "It can't be changed, it's already in TDPL, it's written in stone" What should have been done is something like this: 1. Designing feature X 2. Show the new feature for the community 3. Consider the feedback and possible tweak/redesign 4. Implementing in an experimental branch of the compiler 5. Release an experimental version with just this feature 6. Repeat step 3-5 until satisfied or put on hold/drop the idea 7. Prepare Phobos and druntime for the new feature 8. Move the implementation to the main branch 9. Ship feature X with the next release 10. wait 11. Fix bugs for feature X 12. Repeat step 10-11 a couple of times 13. Write about it in TDPL -- /Jacob Carlborg
May 30 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 11:47 PM, Jacob Carlborg wrote:
 On 2012-05-30 21:10, Andrei Alexandrescu wrote:

 I see how these can be annoying, but they're not the result of us not
 designing things. We designed things best we could.

I would say it's not good enough. The whole approach of designing the language is wrong.

I understand how frustrating this is. In fact even the way you consider "good" is not nearly good enough. What we need is really more formalization of the language design, something that we're sorely missing. I am sometimes frustrated out of my mind at the lack of rigor and discipline in the process. On the other hand, we march with the troops we have. Andrei
May 31 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 2:12 AM, foobar wrote:
 On Thursday, 31 May 2012 at 08:01:14 UTC, Andrei Alexandrescu wrote:
 On 5/30/12 11:47 PM, Jacob Carlborg wrote:
 On 2012-05-30 21:10, Andrei Alexandrescu wrote:

 I see how these can be annoying, but they're not the result of us not
 designing things. We designed things best we could.

I would say it's not good enough. The whole approach of designing the language is wrong.

I understand how frustrating this is. In fact even the way you consider "good" is not nearly good enough. What we need is really more formalization of the language design, something that we're sorely missing. I am sometimes frustrated out of my mind at the lack of rigor and discipline in the process. On the other hand, we march with the troops we have. Andrei

Please no. This is how C++ is designed and we all know how fucked up that is.

Not at all. This is either a misunderstanding, or you lack the faintest idea about the history of C++.
 Writing a [rigorous] spec is almost always incorrect since requirements
 change and unforeseen things come about. Jacob's post illustrates this
 when the spec is written [in TDPL] before implementing, testing and
 integrating it.

 By making a rigorous spec you exacerbate the problem - it takes more
 time to write such a spec thus making the time-frame for unforeseen
 changes larger.

No. Andrei
May 31 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 31/05/2012 11:24, Andrei Alexandrescu a écrit :
 By making a rigorous spec you exacerbate the problem - it takes more
 time to write such a spec thus making the time-frame for unforeseen
 changes larger.

No.

Can you elaborate on what you are thinking as a process ?
May 31 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-05-31 10:01, Andrei Alexandrescu wrote:

 I understand how frustrating this is. In fact even the way you consider
 "good" is not nearly good enough. What we need is really more
 formalization of the language design, something that we're sorely
 missing. I am sometimes frustrated out of my mind at the lack of rigor
 and discipline in the process. On the other hand, we march with the
 troops we have.

Yeah, "good" should have been "better". I'm not sure what you mean with "we march with the troops we have" but it would be fairly easy to at least improve the process somewhat. It would require any extra manpower. It's just that you, Walter and the community needs to be willing to do the changes. -- /Jacob Carlborg
May 31 2012
prev sibling parent mta`chrono <chrono mta-international.net> writes:
Am 31.05.2012 08:47, schrieb Jacob Carlborg:
 What should have been done is something like this:
 
 1. Designing feature X
 2. Show the new feature for the community
 3. Consider the feedback and possible tweak/redesign
 4. Implementing in an experimental branch of the compiler
 5. Release an experimental version with just this feature
 6. Repeat step 3-5 until satisfied or put on hold/drop the idea
 7. Prepare Phobos and druntime for the new feature
 8. Move the implementation to the main branch
 9. Ship feature X with the next release
 10. wait
 11. Fix bugs for feature X
 12. Repeat step 10-11 a couple of times
 13. Write about it in TDPL
 

+1
Jun 03 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 5:29 AM, Dmitry Olshansky wrote:
 Agreed, awfuly crippled design for a language with Thread-local by default.
 Looks like we have 'oh my god, what were they thinking' moment here.

The synchronized class feature was copied by Walter from Java while he at the time had only little understanding of multithreading. The thread-local by default notion was added much later. Indeed we'd design things very differently if synchronized came about later. The design of "synchronized" described in TDPL and not yet implemented is a conciliation of the two.
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

Could you please elaborate how that would help multithreading? Andrei
May 29 2012
next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 29-05-2012 23:41, Andrei Alexandrescu wrote:
 On 5/29/12 5:29 AM, Dmitry Olshansky wrote:
 Agreed, awfuly crippled design for a language with Thread-local by
 default.
 Looks like we have 'oh my god, what were they thinking' moment here.

The synchronized class feature was copied by Walter from Java while he at the time had only little understanding of multithreading. The thread-local by default notion was added much later. Indeed we'd design things very differently if synchronized came about later. The design of "synchronized" described in TDPL and not yet implemented is a conciliation of the two.
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

Could you please elaborate how that would help multithreading? Andrei

The whole concept of synchronization needs to be ripped out of Object with *extreme prejudice*. We have core.sync.mutex for a reason. I think the vtable infrastructure as templates/mixins is an orthogonal point. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 2:50 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:41, Andrei Alexandrescu wrote:
 On 5/29/12 5:29 AM, Dmitry Olshansky wrote:
 Agreed, awfuly crippled design for a language with Thread-local by
 default.
 Looks like we have 'oh my god, what were they thinking' moment here.

The synchronized class feature was copied by Walter from Java while he at the time had only little understanding of multithreading. The thread-local by default notion was added much later. Indeed we'd design things very differently if synchronized came about later. The design of "synchronized" described in TDPL and not yet implemented is a conciliation of the two.
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

Could you please elaborate how that would help multithreading? Andrei

The whole concept of synchronization needs to be ripped out of Object with *extreme prejudice*. We have core.sync.mutex for a reason.

That's more of a restating than an elaboration. Andrei
May 29 2012
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 1:41, Andrei Alexandrescu wrote:
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

Could you please elaborate how that would help multithreading?

It's unrelated to "ease multithreading part strictly speaking. My observation is that it leads to at least removal of 1 word per object. Not a small thing if you happen to use GC, that "sells his memory" in 16 byte chunks, chances are you'd waste 2 instead of one. Again strictly speaking I'm of an opinion that having mutex together with object guarded by it is at least better then 2 separate entities bound together by virtue of code comments :) In any case if mutex is desired, object could have had some other base type say SyncObject. Or use "synchronized class" to that end, what it does now by the way - locks on each method? More about the actual point is that I've come to believe that there is satisfactory way to implement whatever scheme of polymorphism* we want within the language on top of structs without 'class' and 'interface' keywords, special "flawed" pointer type (i.e. tail-const anyone?), and last but not least without new/delete/finalizes (new/delete are still overridable, hint-hint) madness. Ideally I think it should be possible to lower the whole interface/object/class infrastructure to code that uses structs with direct function pointer tables, etc. Multiple alias this is the key, sadly so, otherwise subtyping to multiple interfaces seem not likely. Then some future compiler may even chose to not provide OOP as built-in but lower to this manual implementation on top of struct(!). *I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as ordinal (index inside one global master v-table) instead of pointless _hidden_ v-table pointer could be interesting in certain designs. Another idea is to try tackling multi-methods via some form of compressed 2-stage v-table. (my recent work on generalized tries in D sparked some ideas) -- Dmitry Olshansky
May 29 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 3:06 PM, Dmitry Olshansky wrote:
 On 30.05.2012 1:41, Andrei Alexandrescu wrote:
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

Could you please elaborate how that would help multithreading?

It's unrelated to "ease multithreading part strictly speaking. My observation is that it leads to at least removal of 1 word per object. Not a small thing if you happen to use GC, that "sells his memory" in 16 byte chunks, chances are you'd waste 2 instead of one.

So there is concern about the word per object wasted by the possible mutex. I understand.
 Again strictly speaking I'm of an opinion that having mutex together
 with object guarded by it is at least better then 2 separate entities
 bound together by virtue of code comments :)

Absolutely. I hope you agree that this essentially means you're advocating a Java-style approach in which the mutex is implicitly present...
 In any case if mutex is desired, object could have had some other base
 type say SyncObject.

... albeit not in all objects, only a subhierarchy thereof. I posted the same thing. Nice :o).
 Or use "synchronized class" to that end, what it does now by the way
 - locks on each method?

TDPL's design of synchronized still hasn't been implemented. The design indeed prescribes that all public access to the resource is synchronized.
 More about the actual point is that I've come to believe that there is
 satisfactory way to implement whatever scheme of polymorphism* we want
 within the language on top of structs without 'class' and 'interface'
 keywords, special "flawed" pointer type (i.e. tail-const anyone?), and
 last but not least without new/delete/finalizes (new/delete are still
 overridable, hint-hint) madness.

 Ideally I think it should be possible to lower the whole
 interface/object/class infrastructure to code that uses structs with
 direct function pointer tables, etc. Multiple alias this is the key,
 sadly so, otherwise subtyping to multiple interfaces seem not likely.
 Then some future compiler may even chose to not provide OOP as built-in
 but lower to this manual implementation on top of struct(!).

Well all of these are nice thoughts but at some point we must acknowledge we're operating within the confines of an already-defined language.
 *I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as
 ordinal (index inside one global master v-table) instead of pointless
 _hidden_ v-table pointer could be interesting in certain designs.
 Another idea is to try tackling multi-methods via some form of
 compressed 2-stage v-table. (my recent work on generalized tries in D
 sparked some ideas)

Any post that starts with taking an issue against the waste of one word and ends advocating Smalltalk and Obj-C is... ho-hum. Andrei
May 29 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 3:02, Andrei Alexandrescu wrote:
 On 5/29/12 3:06 PM, Dmitry Olshansky wrote:
 Again strictly speaking I'm of an opinion that having mutex together
 with object guarded by it is at least better then 2 separate entities
 bound together by virtue of code comments :)

Absolutely. I hope you agree that this essentially means you're advocating a Java-style approach in which the mutex is implicitly present...
 In any case if mutex is desired, object could have had some other base
 type say SyncObject.

... albeit not in all objects, only a subhierarchy thereof. I posted the same thing. Nice :o).

Great.
 Or use "synchronized class" to that end, what it does now by the way
 - locks on each method?

TDPL's design of synchronized still hasn't been implemented. The design indeed prescribes that all public access to the resource is synchronized.

So sad, I recall when I was reading about it it made a lot of sense.
 More about the actual point is that I've come to believe that there is
 satisfactory way to implement whatever scheme of polymorphism* we want
 within the language on top of structs without 'class' and 'interface'
 keywords, special "flawed" pointer type (i.e. tail-const anyone?), and
 last but not least without new/delete/finalizes (new/delete are still
 overridable, hint-hint) madness.

 Ideally I think it should be possible to lower the whole
 interface/object/class infrastructure to code that uses structs with
 direct function pointer tables, etc. Multiple alias this is the key,
 sadly so, otherwise subtyping to multiple interfaces seem not likely.
 Then some future compiler may even chose to not provide OOP as built-in
 but lower to this manual implementation on top of struct(!).

Well all of these are nice thoughts but at some point we must acknowledge we're operating within the confines of an already-defined language.
 *I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as
 ordinal (index inside one global master v-table) instead of pointless
 _hidden_ v-table pointer could be interesting in certain designs.
 Another idea is to try tackling multi-methods via some form of
 compressed 2-stage v-table. (my recent work on generalized tries in D
 sparked some ideas)

Any post that starts with taking an issue against the waste of one word and ends advocating Smalltalk and Obj-C is... ho-hum.

Well going from practical matters to personal dreams is remarkably easy at 2 AM :) -- Dmitry Olshansky
May 29 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-05-30 00:06, Dmitry Olshansky wrote:

 *I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as
 ordinal (index inside one global master v-table) instead of pointless
 _hidden_ v-table pointer could be interesting in certain designs.
 Another idea is to try tackling multi-methods via some form of
 compressed 2-stage v-table. (my recent work on generalized tries in D
 sparked some ideas)

I like that classes are first class citizens in Objective-C and Ruby. Class methods instead of static methods. Easily extendable from outside the class. But I still like to have the "class" and "interface" keywords. Would this solve the problem with const: https://github.com/D-Programming-Language/dmd/pull/3 -- /Jacob Carlborg
May 30 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 13:28, Jacob Carlborg wrote:
 On 2012-05-30 00:06, Dmitry Olshansky wrote:

 *I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as
 ordinal (index inside one global master v-table) instead of pointless
 _hidden_ v-table pointer could be interesting in certain designs.
 Another idea is to try tackling multi-methods via some form of
 compressed 2-stage v-table. (my recent work on generalized tries in D
 sparked some ideas)

I like that classes are first class citizens in Objective-C and Ruby. Class methods instead of static methods. Easily extendable from outside the class. But I still like to have the "class" and "interface" keywords.

Yup. Even Java has Class as object. If D copied most of OOP from Java it seems strange that it was replaced by halfhearted factory method in object.di kind of saying "you have RTTI but it's crippled just in case if it is of no use". Still I'd think it's reasonable course to enact slow but steady evolution of existing OOP design, changing synchronized/monitors seems as good start as any.
 Would this solve the problem with const:

 https://github.com/D-Programming-Language/dmd/pull/3

I know, and hoped it would happen sooner. But at least it's possible. -- Dmitry Olshansky
May 30 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-05-30 12:21, Dmitry Olshansky wrote:

 Yup. Even Java has Class as object. If D copied most of OOP from Java it
 seems strange that it was replaced by halfhearted factory method in
 object.di
 kind of saying "you have RTTI but it's crippled just in case if it is of
 no use".

I would not compare Java's Class with classes in Objective-C and Ruby. In Objective-C and Ruby classes are objects with their own hierarchy of inheritance. Example in Ruby: class Foo def bar instance_variable = 1 class_variable = 2 end # class method def self.bar instance_variable_on_the_class = 1 class_variable = 3 end end a = Foo instance_of_foo = a.new
 Still I'd think it's reasonable course to enact slow but steady
 evolution of existing OOP design, changing synchronized/monitors seems
 as good start as any.

I agree. -- /Jacob Carlborg
May 30 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 17:58, Jacob Carlborg wrote:
 On 2012-05-30 12:21, Dmitry Olshansky wrote:

 Yup. Even Java has Class as object. If D copied most of OOP from Java it
 seems strange that it was replaced by halfhearted factory method in
 object.di
 kind of saying "you have RTTI but it's crippled just in case if it is of
 no use".

I would not compare Java's Class with classes in Objective-C and Ruby. In Objective-C and Ruby classes are objects with their own hierarchy of inheritance. Example in Ruby: class Foo def bar instance_variable = 1 class_variable = 2 end # class method def self.bar instance_variable_on_the_class = 1 class_variable = 3 end end a = Foo instance_of_foo = a.new

Yes, it's OOP classics I think. Dynamic languages are easier in this regard. But let's not detract from original topic - locking. -- Dmitry Olshansky
May 30 2012
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
Le 29/05/2012 14:26, Alex Rønne Petersen a écrit :
 On 29-05-2012 14:19, Dmitry Olshansky wrote:
 On 29.05.2012 16:07, Regan Heath wrote:
 According to the docs here:
 http://dlang.org/class.html#synchronized-functions

 A synchronized function locks "this" and as "this" is exposed
 publicly... In the following code the "lock" statement and "synchronized
 void bar" lock the same mutex.

 class Foo {
 synchronized void bar() { ...statements... }
 }

 void main()
 {
 Foo foo = new Foo();
 lock(foo)
 {
 ...statements...
 }
 }

 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.


I'd be darned but every Object in D has monitor fields. If I'm not mistaken it's the mutex you are looking for ;)

Indeed they do, and therefore each object must eat an entire word of memory for questionable gain. Generalized object monitors is the worst idea in programming language and virtual machine design, ever.

Every fucking things MUST BE AN OBJECT :D So let's lock on ANY OBJECT !
May 29 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 5:26 AM, Alex Rønne Petersen wrote:
 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

I think that's an exaggeration. Care to elaborate a bit? Andrei
May 29 2012
parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 29-05-2012 23:38, Andrei Alexandrescu wrote:
 On 5/29/12 5:26 AM, Alex Rønne Petersen wrote:
 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

I think that's an exaggeration. Care to elaborate a bit? Andrei

1) You waste an entire word of memory in *every single object you ever create*. And for the majority of objects, this word will always be zero/null. Not to mention that when you do allocate a monitor, you also get the actual memory of that monitor in addition. 2) Anyone can lock on any object meaning it's near impossible to see where a deadlock might come from. 3) Encapsulation is completely broken as a result of (2). 4) There's a whole bunch of runtime and GC plumbing (which is even broken in druntime right now) to support object monitors. To fix this entire mess, we'd need weak references. 5) This whole object monitor model goes against our "thread-local by default" model. Synchronization is evil and should be explicit with core.sync.mutex. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 2:56 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:38, Andrei Alexandrescu wrote:
 On 5/29/12 5:26 AM, Alex Rønne Petersen wrote:
 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

I think that's an exaggeration. Care to elaborate a bit? Andrei

1) You waste an entire word of memory in *every single object you ever create*. And for the majority of objects, this word will always be zero/null. Not to mention that when you do allocate a monitor, you also get the actual memory of that monitor in addition.

I agree. It would be better if synchronizable objects would be part of a sub-hierarchy such that not everyone pays for the word.
 2) Anyone can lock on any object meaning it's near impossible to see
 where a deadlock might come from.

What would be the alternative? Deadlocks are a natural consequence of the fact that indeed any thread can wait on a resource.
 3) Encapsulation is completely broken as a result of (2).

Again, what would be the alternative?
 4) There's a whole bunch of runtime and GC plumbing (which is even
 broken in druntime right now) to support object monitors. To fix this
 entire mess, we'd need weak references.

That's more like an argument the implementation is imperfect, not that the idea is the worst ever.
 5) This whole object monitor model goes against our "thread-local by
 default" model. Synchronization is evil and should be explicit with
 core.sync.mutex.

But doing things with core.sync.mutex is a definite step backwards as it is prey to all of the issues above. For example, "anyone can lock any mutex". How is that progress? Andrei
May 29 2012
next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:56 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:38, Andrei Alexandrescu wrote:
 On 5/29/12 5:26 AM, Alex Rønne Petersen wrote:
 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

I think that's an exaggeration. Care to elaborate a bit? Andrei

1) You waste an entire word of memory in *every single object you ever create*. And for the majority of objects, this word will always be zero/null. Not to mention that when you do allocate a monitor, you also get the actual memory of that monitor in addition.

I agree. It would be better if synchronizable objects would be part of a sub-hierarchy such that not everyone pays for the word.
 2) Anyone can lock on any object meaning it's near impossible to see
 where a deadlock might come from.

What would be the alternative? Deadlocks are a natural consequence of the fact that indeed any thread can wait on a resource.

But mutexes allow proper encapsulation by hiding the mutex resource. As I've proven in the very OP of this thread, both druntime and phobos suffer from the anti-pattern that is locking on a public resource when you shouldn't. The language encourages doing it with this synchronized business.
 3) Encapsulation is completely broken as a result of (2).

Again, what would be the alternative?
 4) There's a whole bunch of runtime and GC plumbing (which is even
 broken in druntime right now) to support object monitors. To fix this
 entire mess, we'd need weak references.

That's more like an argument the implementation is imperfect, not that the idea is the worst ever.

Of course. But it's enough reason for me to have banned synchronized from my source base entirely. This point is a practical matter more than it's a design matter.
 5) This whole object monitor model goes against our "thread-local by
 default" model. Synchronization is evil and should be explicit with
 core.sync.mutex.

But doing things with core.sync.mutex is a definite step backwards as it is prey to all of the issues above. For example, "anyone can lock any mutex". How is that progress?

You encapsulate the mutex and thereby avoid the anti-pattern I pointed out in the OP, which has evidently found its way into both druntime and phobos.
 Andrei

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 01:10, Jonathan M Davis wrote:
 On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

 From your comments, it sounds to me like the problem is entirely with

the issue of having externals lock on them without the explicit use of synchronized blocks. They're the equivalent of locking a mutex in every public function and releasing it afterwards. The problem is that that mutex is effectively public such that when a synchronized block is used on it, it locks on the same mutex. If no synchronized blocks are used, as far as I can tell, it's a non-issue. As such, wouldn't simply making the use of a sychronized block on a synchronized object illegal fix the problem? - Jonathan M Davis

Possibly. I haven't thought that through, but I think that that could work. In fact, it would probably be a good way to ensure safety statically, since a synchronized class tells its user "I take care of synchronization". We still have the issue of every object having a monitor, but that is probably a separate issue... -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent Michel Fortin <michel.fortin michelf.com> writes:
On 2012-05-29 23:22:38 +0000, Alex Rønne Petersen <alex lycus.org> said:

 Possibly. I haven't thought that through, but I think that that could 
 work. In fact, it would probably be a good way to ensure safety 
 statically, since a synchronized class tells its user "I take care of 
 synchronization".

More or less. My biggest gripe about synchronized functions is that calling other functions within them is prone to create deadlocks. Can you see the deadlock in this example? synchronized class Node { private Node[] peers; void addPeer(Node peer) // implicitly synchronized { peers ~= peer; } void poke() // implicitly synchronized { writeln("blip!"); } void pokeNext() // implicitly synchronized { if (!peers.empty) peers.front.poke(); } } shared Node node1; shared Node node2; void pokeThread(Node node) { node.pokeNext(); } void main() { // setup node1 = new Node; node2 = new Node; node1.addPeer(node2); node2.addPeer(node1); // start two threads spawn(&pokeThread, node1); spawn(&pokeThread, node2); } The correct way to write the pokeNext function would be this, assuming you could remove the implicit synchronization: void pokeNext() // not implicitly synchronized { Node next; synchronized (this) { if (!peers.empty) next = peers.front; } if (next) next.poke(); } Here you only synchronize access to the variable, which cannot deadlock in any way. Keeping the lock while calling other functions is dangerous (other functions can take their own lock and thus deadlock) and should be avoided as long as you hold a lock. The easiest way to avoid deadlocks is to never take two locks at the same time. The problem is that implicit synchronized blocks makes it too easy to take many locks at the same time without even noticing, which is deadlock prone. I think that is a reason enough not to use synchronized classes in the general case. They're fine as long as your member functions (which are implicitly synchronized) only access what is contained in the class, but you must be sure not to call another function that takes another lock, or then you need to be very careful about what those functions do. (Also, being forced to take locks longer than necessary because the whole function is always synchronized can be a performance problem, but that's a relatively minor issue compared to deadlocks.) -- Michel Fortin michel.fortin michelf.com http://michelf.com/
May 29 2012
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 16:43, Regan Heath wrote:
 On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

 From your comments, it sounds to me like the problem is entirely with

don't have the issue of having externals lock on them without the explicit use of synchronized blocks. They're the equivalent of locking a mutex in every public function and releasing it afterwards. The problem is that that mutex is effectively public such that when a synchronized block is used on it, it locks on the same mutex. If no synchronized blocks are used, as far as I can tell, it's a non-issue. As such, wouldn't simply making the use of a sychronized block on a synchronized object illegal fix the problem?

Not in all cases. If you have a synchronized class, say a thread-safe container, which has a synchronized lookup and synchronized insert function, code like.. if (!o.lookup(...)) o.insert(...) obtains and releases the mutex twice each, and in between another thread could call o.insert() and insert the missing object - resulting in 2 copies being inserted (or a duplicate insert exception being throw, or...). Now, the above is bad design, but the only other choice is to expose lock() and unlock() methods (or the mutex - back to square 1one) to allow locking around multiple class, but now the caller has to be careful to call unlock in all code paths. scope helps us in D, but this is actually the whole point of synchronized blocks - 1. ensuring the unlock always happens and 2. making the scope of the locking visible in the code.

{ void applyLocked(scope delegate void(Container _this)); ... } -- Dmitry Olshansky
May 30 2012
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 17:14, Regan Heath wrote:
 synchronized class Container
 {

 void applyLocked(scope delegate void(Container _this));
 ...
 }

Neat :)

synchronized class Locked!(C) { void applyLocked(scope delegate void(C _this) dg) {//should have lock/unlock already, as every sync'ed class gd(_this); } final auto opDispatch(string mtd, Args...)(Args args) {//can't go alias this, it won't be locked I think return mixin("_this."~mtd~"("~args~")"); } private: C _this; } -- Dmitry Olshansky
May 30 2012
prev sibling parent deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 14:55, Dmitry Olshansky a écrit :
 On 30.05.2012 16:43, Regan Heath wrote:
 On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

 From your comments, it sounds to me like the problem is entirely with

don't have the issue of having externals lock on them without the explicit use of synchronized blocks. They're the equivalent of locking a mutex in every public function and releasing it afterwards. The problem is that that mutex is effectively public such that when a synchronized block is used on it, it locks on the same mutex. If no synchronized blocks are used, as far as I can tell, it's a non-issue. As such, wouldn't simply making the use of a sychronized block on a synchronized object illegal fix the problem?

Not in all cases. If you have a synchronized class, say a thread-safe container, which has a synchronized lookup and synchronized insert function, code like.. if (!o.lookup(...)) o.insert(...) obtains and releases the mutex twice each, and in between another thread could call o.insert() and insert the missing object - resulting in 2 copies being inserted (or a duplicate insert exception being throw, or...). Now, the above is bad design, but the only other choice is to expose lock() and unlock() methods (or the mutex - back to square 1one) to allow locking around multiple class, but now the caller has to be careful to call unlock in all code paths. scope helps us in D, but this is actually the whole point of synchronized blocks - 1. ensuring the unlock always happens and 2. making the scope of the locking visible in the code.

{ void applyLocked(scope delegate void(Container _this)); ... }

I often ends up doing similar stuff when manipulating pair of actions. lock/unlock, connect/disconect, begin/commit/rollback, etc . . .
May 30 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 14:43, Regan Heath a écrit :
 On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

 From your comments, it sounds to me like the problem is entirely with

don't have the issue of having externals lock on them without the explicit use of synchronized blocks. They're the equivalent of locking a mutex in every public function and releasing it afterwards. The problem is that that mutex is effectively public such that when a synchronized block is used on it, it locks on the same mutex. If no synchronized blocks are used, as far as I can tell, it's a non-issue. As such, wouldn't simply making the use of a sychronized block on a synchronized object illegal fix the problem?

Not in all cases. If you have a synchronized class, say a thread-safe container, which has a synchronized lookup and synchronized insert function, code like.. if (!o.lookup(...)) o.insert(...) obtains and releases the mutex twice each, and in between another thread could call o.insert() and insert the missing object - resulting in 2 copies being inserted (or a duplicate insert exception being throw, or...). Now, the above is bad design, but the only other choice is to expose lock() and unlock() methods (or the mutex - back to square 1one) to allow locking around multiple class, but now the caller has to be careful to call unlock in all code paths. scope helps us in D, but this is actually the whole point of synchronized blocks - 1. ensuring the unlock always happens and 2. making the scope of the locking visible in the code. R

The correct solution is an insertIfNotPresent function, or to throw/return an error when inserting an item. Exposing the lock is just spreading the mess.
May 30 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 6:39 AM, deadalnix wrote:
 Le 30/05/2012 14:43, Regan Heath a écrit :
 Not in all cases. If you have a synchronized class, say a thread-safe
 container, which has a synchronized lookup and synchronized insert
 function, code like..

 if (!o.lookup(...))
 o.insert(...)

throw/return an error when inserting an item. Exposing the lock is just spreading the mess.

Agreed on both counts. Andrei
May 30 2012
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 5:43 AM, Regan Heath wrote:
 Not in all cases. If you have a synchronized class, say a thread-safe
 container, which has a synchronized lookup and synchronized insert
 function, code like..

 if (!o.lookup(...))
 o.insert(...)

 obtains and releases the mutex twice each, and in between another thread
 could call o.insert() and insert the missing object - resulting in 2
 copies being inserted (or a duplicate insert exception being throw,
 or....).

 Now, the above is bad design, but the only other choice is to expose
 lock() and unlock() methods (or the mutex - back to square 1one) to
 allow locking around multiple class, but now the caller has to be
 careful to call unlock in all code paths. scope helps us in D, but this
 is actually the whole point of synchronized blocks - 1. ensuring the
 unlock always happens and 2. making the scope of the locking visible in
 the code.

There's a hybrid design available - a lock() operation may return a Locked!Type object which has a richer set of primitives. The implied understanding is that for the lifetime of that object, the underlying mutex is locked. Andrei
May 30 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

From your comments, it sounds to me like the problem is entirely with 

the issue of having externals lock on them without the explicit use of synchronized blocks. They're the equivalent of locking a mutex in every public function and releasing it afterwards. The problem is that that mutex is effectively public such that when a synchronized block is used on it, it locks on the same mutex. If no synchronized blocks are used, as far as I can tell, it's a non-issue. As such, wouldn't simply making the use of a sychronized block on a synchronized object illegal fix the problem? - Jonathan M Davis
May 29 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis <jmdavisProg gmx.co=
m>  =

wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex R=F8nne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. =


 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource whe=


 you shouldn't. The language encourages doing it with this synchronize=


 business.

 From your comments, it sounds to me like the problem is entirely with=


 synchronized blocks, not sychronized classes. Synchronized classes don=

 have
 the issue of having externals lock on them without the explicit use of=

 synchronized blocks. They're the equivalent of locking a mutex in ever=

 public
 function and releasing it afterwards. The problem is that that mutex i=

 effectively public such that when a synchronized block is used on it, =

 locks
 on the same mutex. If no synchronized blocks are used, as far as I can=

 tell,
 it's a non-issue.

 As such, wouldn't simply making the use of a sychronized block on a
 synchronized object illegal fix the problem?

Not in all cases. If you have a synchronized class, say a thread-safe = container, which has a synchronized lookup and synchronized insert = function, code like.. if (!o.lookup(...)) o.insert(...) obtains and releases the mutex twice each, and in between another thread= = could call o.insert() and insert the missing object - resulting in 2 = copies being inserted (or a duplicate insert exception being throw, or..= .). Now, the above is bad design, but the only other choice is to expose = lock() and unlock() methods (or the mutex - back to square 1one) to allo= w = locking around multiple class, but now the caller has to be careful to = call unlock in all code paths. scope helps us in D, but this is actuall= y = the whole point of synchronized blocks - 1. ensuring the unlock always = happens and 2. making the scope of the locking visible in the code. R -- = Using Opera's revolutionary email client: http://www.opera.com/mail/
May 30 2012
prev sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 30 May 2012 13:55:10 +0100, Dmitry Olshansky  =

<dmitry.olsh gmail.com> wrote:

 On 30.05.2012 16:43, Regan Heath wrote:
 On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex R=F8nne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource=




 As
 I've proven in the very OP of this thread, both druntime and phobos=




 suffer from the anti-pattern that is locking on a public resource w=




 you shouldn't. The language encourages doing it with this synchroni=




 business.

 From your comments, it sounds to me like the problem is entirely wi=




 synchronized blocks, not sychronized classes. Synchronized classes
 don't have
 the issue of having externals lock on them without the explicit use =



 synchronized blocks. They're the equivalent of locking a mutex in
 every public
 function and releasing it afterwards. The problem is that that mutex=



 effectively public such that when a synchronized block is used on it=



 it locks
 on the same mutex. If no synchronized blocks are used, as far as I c=



 tell,
 it's a non-issue.

 As such, wouldn't simply making the use of a sychronized block on a
 synchronized object illegal fix the problem?

Not in all cases. If you have a synchronized class, say a thread-safe=


 container, which has a synchronized lookup and synchronized insert
 function, code like..

 if (!o.lookup(...))
 o.insert(...)

 obtains and releases the mutex twice each, and in between another thr=


 could call o.insert() and insert the missing object - resulting in 2
 copies being inserted (or a duplicate insert exception being throw,  =


 or...).

 Now, the above is bad design, but the only other choice is to expose
 lock() and unlock() methods (or the mutex - back to square 1one) to
 allow locking around multiple class, but now the caller has to be
 careful to call unlock in all code paths. scope helps us in D, but th=


 is actually the whole point of synchronized blocks - 1. ensuring the
 unlock always happens and 2. making the scope of the locking visible =


 the code.

{ void applyLocked(scope delegate void(Container _this)); ... }

Neat :) R -- = Using Opera's revolutionary email client: http://www.opera.com/mail/
May 30 2012
prev sibling parent =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 29-05-2012 14:07, Regan Heath wrote:
 On Tue, 29 May 2012 01:08:59 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Tuesday, May 29, 2012 01:54:59 Alex Rønne Petersen wrote:
 On 29-05-2012 01:41, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:35:23 Alex Rønne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way


 code would break if we changed the semantics. I'd consider it a


 and a lesson learned, rather.

 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

I'm not an expert on threading stuf, but it would be my opinion

 you're not intending to protect the entire class with locks that it

 no sense to lock on the class itself. You're locking for something
 specific, in which case, your sychronized block should be locking on
 something else specific to what you're trying to protect. Certainly,
 that's how I'd approach it with mutexes. You don't have a mutex for an
 entire class unless it's actually used for all of the class'

 Rather, you use mutexes specific to what you're trying to protect.

 - Jonathan M Davis

Right, but even if you really *are* protecting the entire class, you can still create mysterious deadlocks if users of your code lock on your class. So I'm arguing that no matter the use case, never lock on a 'this' reference exposed outside of some API layer.

Well, it seems pretty abysmal to me to be locking on something that you don't control. Making a mutex that your class used public would just be stupid.

Interestingly this is what C#s Array type does with SyncRoot (intentionally).

Microsoft has officially dismissed the SyncRoot approach to synchronization. I don't have a link handy, but there's an article *somewhere* on MSDN about it.
 With synchronized classes/functions, you're basically creating and
 using an
 implicit mutex for the whole class, which would then be the same as if
 you locked it at the beginning of every member function call and
 unlocked it at its end, which doesn't expose the mutex at all. So, I
 don't really see te
 problem there would have to study the matter more to see what the
 issue there is.

According to the docs here: http://dlang.org/class.html#synchronized-functions A synchronized function locks "this" and as "this" is exposed publicly.... In the following code the "lock" statement and "synchronized void bar" lock the same mutex. class Foo { synchronized void bar() { ...statements... } } void main() { Foo foo = new Foo(); lock(foo) { ...statements... } }
 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.

Yep. It commonly arises where you have a class/object which is either not synchronized itself (because it might be used in single threaded situations and you want performance) like a collection class for example, or is synchronized but you need to lock a sequence of member function calls i.e. a lookup and insert on the collection. What happens then, is people just call lock(<object>) and end up locking the same mutex as the class does internally. What happens next to cause a deadlock is that inside that lock they call one or more functions or methods which internally call methods on another synchronized object. This results in a locking pattern of object1, object2. In another piece of code, running in another thread, they do something similar which results in a locking pattern of object2, object1 and eventually these threads will deadlock each other. R

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
prev sibling next sibling parent "Nathan M. Swan" <nathanmswan gmail.com> writes:
On Monday, 28 May 2012 at 23:55:04 UTC, Alex Rønne Petersen 
wrote:
 On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet 
 we adopted
 it anyway... blergh.

The "lesson learned" from Java that TDPL enumerates is the mistake of having synchronized on functions rather than entire classes, but clearly even that is currently TDPL-only and not actually properly implemented yet. - Jonathan M Davis

But synchronized on entire classes is exactly equally flawed... the fundamental problem is still that they synchronize on a public resource.

The fundamental problem is shared memory ;) NMS
May 28 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 29/05/2012 01:35, Alex Rønne Petersen a écrit :
 On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

Per TDPL, having individually synchronized functions is illegal. Either all of the functions in a class are synchronized or none of them are. Putting synchronized on a function should actually be illegal. It belongs only on classes or in synchronized blocks, never on functions. Now, unfortuntately, I don't believe that the compiler enforces any of that right now, so you end up synchronizing indivdual functions rather than whole classes, but the synchronized functions themselves should function the same.
 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

Well, then you should probably be arguing about the design of synchronized classes/functions rather than synchronized(this). However, given the design of synchronized classes, synchronized(this) would probably never be appropriate. If you're using a synchronized class, then you don't need it. And if you're not using a synchronized class, then you shouldn't be synchronizing your class. I would definitely think that synchronized blocks should synchronize on something else, since they're trying to lock a much smaller area than an entire class. - Jonathan M Davis

I don't think arguing about them makes sense at this point. Way too much code would break if we changed the semantics. I'd consider it a mistake and a lesson learned, rather. But I take it you agree that synchronized (this) and similar "uncontrollable" synchronization blocks should be avoided?

I would say that breaking things here, with the right deprecation process, is the way to go. shared isn't working properly ATM, so anyway, things will have to change in regard of shared memory support in the language.
May 29 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 10:37, deadalnix wrote:
 Le 29/05/2012 01:35, Alex Rønne Petersen a écrit :
 On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

Per TDPL, having individually synchronized functions is illegal. Either all of the functions in a class are synchronized or none of them are. Putting synchronized on a function should actually be illegal. It belongs only on classes or in synchronized blocks, never on functions. Now, unfortuntately, I don't believe that the compiler enforces any of that right now, so you end up synchronizing indivdual functions rather than whole classes, but the synchronized functions themselves should function the same.
 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

Well, then you should probably be arguing about the design of synchronized classes/functions rather than synchronized(this). However, given the design of synchronized classes, synchronized(this) would probably never be appropriate. If you're using a synchronized class, then you don't need it. And if you're not using a synchronized class, then you shouldn't be synchronizing your class. I would definitely think that synchronized blocks should synchronize on something else, since they're trying to lock a much smaller area than an entire class. - Jonathan M Davis

I don't think arguing about them makes sense at this point. Way too much code would break if we changed the semantics. I'd consider it a mistake and a lesson learned, rather. But I take it you agree that synchronized (this) and similar "uncontrollable" synchronization blocks should be avoided?

I would say that breaking things here, with the right deprecation process, is the way to go.

Well, we could deprecate it over time, but it certainly can't be a day-to-day thing.
 shared isn't working properly ATM, so anyway, things will have to change
 in regard of shared memory support in the language.

And it never will. ;) -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply deadalnix <deadalnix gmail.com> writes:
Le 29/05/2012 14:26, Alex Rønne Petersen a écrit :
 shared isn't working properly ATM, so anyway, things will have to change
 in regard of shared memory support in the language.

And it never will. ;)

It has been successful done in other languages and has proven itself useful. This is a difficult backend problem, but it can be solved.
May 29 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 17:52, deadalnix wrote:
 Le 29/05/2012 14:26, Alex Rønne Petersen a écrit :
 shared isn't working properly ATM, so anyway, things will have to change
 in regard of shared memory support in the language.

And it never will. ;)

It has been successful done in other languages and has proven itself useful. This is a difficult backend problem, but it can be solved.

The problem is not in the compiler back end, the problem is that shared is completely useless as it is. There's no bridge from shared to unshared, making it impossible to work with any real world code base. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 29/05/2012 18:56, Alex Rønne Petersen a écrit :
 On 29-05-2012 17:52, deadalnix wrote:
 Le 29/05/2012 14:26, Alex Rønne Petersen a écrit :
 shared isn't working properly ATM, so anyway, things will have to
 change
 in regard of shared memory support in the language.

And it never will. ;)

It has been successful done in other languages and has proven itself useful. This is a difficult backend problem, but it can be solved.

The problem is not in the compiler back end, the problem is that shared is completely useless as it is. There's no bridge from shared to unshared, making it impossible to work with any real world code base.

I'd argue that shared shouldn't contains big stuff. Integers, pointer to immutable data, flags, stuff like that. For such usage, this would work. Still, you comment show that the current state of things needs some work, and it is likely to impact both shared and synchronized.
May 29 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei
May 29 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

core.sync.mutex -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

core.sync.mutex

That's worse. Andrei
May 29 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

core.sync.mutex

That's worse. Andrei

I don't agree. Also, it is faster than object monitors. This was proven by David Simcha recently where he sped up GC allocations by some 40%-ish factor just by changing to a final class derived from core.sync.mutex. And no, that's not a bug in the monitor implementation. It's a limitation of the design. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 3:58 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

core.sync.mutex

That's worse. Andrei

I don't agree.

One simple thing to understand is that core.sync.mutex does everything synchronized objects do, in a much less structured way. So it's tenuous to build an argument that synchronized classes do something wrong but bare, unstructured mutexes do something good.
 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

We'd need to take a look at that. I recall at a point Bartosz was working on cheap locks using the mutex word as a spin lock in certain circumstances. Anyhow, it's something that would be interesting to look at. Andrei
May 29 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 01:07, Andrei Alexandrescu wrote:
 On 5/29/12 3:58 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

core.sync.mutex

That's worse. Andrei

I don't agree.

One simple thing to understand is that core.sync.mutex does everything synchronized objects do, in a much less structured way. So it's tenuous to build an argument that synchronized classes do something wrong but bare, unstructured mutexes do something good.

Okay, let's get something straight here: The Mutex class in core.sync.mutex is *not* final. It does *not* force you to use it in a compositional fashion. You can inherit it just as any other non-final class. And when you use synchronized (mtx), where mtx is an instance of Mutex or a derived class, it will even do the Right Thing (TM) and lock on the mutex. Mutex is effectively the synchronized object type that you want. Why not just use it?
 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

We'd need to take a look at that. I recall at a point Bartosz was working on cheap locks using the mutex word as a spin lock in certain circumstances. Anyhow, it's something that would be interesting to look at. Andrei

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 08:23, Sean Kelly a écrit :
 On May 29, 2012, at 4:07 PM, Andrei
Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:

 On 5/29/12 3:58 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

core.sync.mutex

That's worse. Andrei

I don't agree.

One simple thing to understand is that core.sync.mutex does everything synchronized objects do, in a much less structured way. So it's tenuous to build an argument that synchronized classes do something wrong but bare, unstructured mutexes do something good.
 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

We'd need to take a look at that. I recall at a point Bartosz was working on cheap locks using the mutex word as a spin lock in certain circumstances. Anyhow, it's something that would be interesting to look at.

I bet this is because monitors are lazily initialized, so the cost of acquiring a lock is more than just locking the underlying mutex. The implementation for built-in monitors really isnt great. I've been meaning to do something about that.

And it use double check locking the wrong way, so can eventually explode sometime.
May 30 2012
prev sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 08:23, Sean Kelly wrote:
 On May 29, 2012, at 4:07 PM, Andrei
Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:

 On 5/29/12 3:58 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

core.sync.mutex

That's worse. Andrei

I don't agree.

One simple thing to understand is that core.sync.mutex does everything synchronized objects do, in a much less structured way. So it's tenuous to build an argument that synchronized classes do something wrong but bare, unstructured mutexes do something good.
 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

We'd need to take a look at that. I recall at a point Bartosz was working on cheap locks using the mutex word as a spin lock in certain circumstances. Anyhow, it's something that would be interesting to look at.

I bet this is because monitors are lazily initialized, so the cost of acquiring a lock is more than just locking the underlying mutex. The implementation for built-in monitors really isnt great. I've been meaning to do something about that.

We also need to fix the monitor memory leak that sometimes manifests itself... -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.
May 30 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

But in this design anyone can lock such an object, which was something you advocated against. Andrei
May 30 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 9:03 AM, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

But in this design anyone can lock such an object, which was something you advocated against.

I think there is some confusion here as to what the "problem" is and is not. The problem is /not/ that you can lock any object. The problem is /not/ that we have synchronized(object) {} The problem is /not/ that we have synchronized classes/methods.

Several posts in this thread assert that such are problems.
 The problem /is/ that synchronized classes/methods use a mutex which is
 exposed publicly, and it the same mutex as used by synchronized(object)
 {}. This exposure/re-use makes deadlocks more likely to happen, and
 harder to spot.

This is news to me. How do you publicly access the mutex of a synchronized class object? Andrei
May 30 2012
parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 On 5/30/12 9:03 AM, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

But in this design anyone can lock such an object, which was something you advocated against.

I think there is some confusion here as to what the "problem" is and is not. The problem is /not/ that you can lock any object. The problem is /not/ that we have synchronized(object) {} The problem is /not/ that we have synchronized classes/methods.

Several posts in this thread assert that such are problems.
 The problem /is/ that synchronized classes/methods use a mutex which is
 exposed publicly, and it the same mutex as used by synchronized(object)
 {}. This exposure/re-use makes deadlocks more likely to happen, and
 harder to spot.

This is news to me. How do you publicly access the mutex of a synchronized class object?

Generally in two ways: 1) synchronized (obj) 2) obj.__monitor (1) accesses it in order to actually do locking, but if obj is an instantiation of a class that uses the this reference internally for locking, you end up with potential deadlocks. Therefore, you can say that obj's mutex is exposed. This isn't necessarily bad, because if obj does *not* use its this reference for synchronized statements, nothing will blow up. This is why the OP was about banning synchronized (this). (2) is an undocumented feature of the language that druntime (and maybe some phobos code) makes use of to create/alter/delete monitors.
 Andrei

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
next sibling parent =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 30-05-2012 18:42, Iain Buclaw wrote:
 On 30 May 2012 17:23, Alex Rønne Petersen<alex lycus.org>  wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 On 5/30/12 9:03 AM, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org>  wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T&& is(typeof(T.init.lock()))&& is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

But in this design anyone can lock such an object, which was something you advocated against.

I think there is some confusion here as to what the "problem" is and is not. The problem is /not/ that you can lock any object. The problem is /not/ that we have synchronized(object) {} The problem is /not/ that we have synchronized classes/methods.

Several posts in this thread assert that such are problems.
 The problem /is/ that synchronized classes/methods use a mutex which is
 exposed publicly, and it the same mutex as used by synchronized(object)
 {}. This exposure/re-use makes deadlocks more likely to happen, and
 harder to spot.

This is news to me. How do you publicly access the mutex of a synchronized class object?

Generally in two ways: 1) synchronized (obj) 2) obj.__monitor (1) accesses it in order to actually do locking, but if obj is an instantiation of a class that uses the this reference internally for locking, you end up with potential deadlocks. Therefore, you can say that obj's mutex is exposed. This isn't necessarily bad, because if obj does *not* use its this reference for synchronized statements, nothing will blow up. This is why the OP was about banning synchronized (this). (2) is an undocumented feature of the language that druntime (and maybe some phobos code) makes use of to create/alter/delete monitors.

I'm pretty certain the use of .__monitor is stricted only to: object_.d - The frontend implementation which contains the underlying functions that are invoked when you synchronise(obj). monitor_.d - The backend implementation which contains the platform separated bits. Regards

I use it in my source base currently to implement weak references. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 9:23 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

Generally in two ways: 1) synchronized (obj)

This is not accessing the mutex for arbitrary operations.
 2) obj.__monitor

All symbols starting with two underscores are reserved by the implementation.
 (1) accesses it in order to actually do locking, but if obj is an
 instantiation of a class that uses the this reference internally for
 locking, you end up with potential deadlocks.

Deadlocks are endemic to mutex-based programming and are not avoided inherently or preferentially by the alternative you discussed (exposing unrestricted mutex).
 Therefore, you can say
 that obj's mutex is exposed.

Nothing could stop one, but that claim is incorrect. Andrei
May 30 2012
next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 30-05-2012 19:12, Andrei Alexandrescu wrote:
 On 5/30/12 9:23 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

Generally in two ways: 1) synchronized (obj)

This is not accessing the mutex for arbitrary operations.

No, it is indeed not. You didn't explicitly say you wanted to do "arbitrary operations", so I assumed that by accessing the mutex you meant doing the only two things that are sensible for a mutex: Locking and unlocking it. What else would you want to do? This discussion has always been about locking and unlocking publicly exposed mutexes. I don't think arguing about "arbitrary operations" is going to get the discussion anywhere exactly because of what you proceed to say ...
 2) obj.__monitor

All symbols starting with two underscores are reserved by the implementation.

... since that effectively means there is no way, from an implementation-independent standpoint, to alter the memory of a mutex (or whatever) (see my answer below), thus making the only sensible operations on a mutex locking and unlocking. But I may be misunderstanding you. If so, please clarify and/or correct me. (Just so we're clear, I included (2) only for completeness. I know it's a dirty hack and I agree that it should never be in the language 'spec' or TDPL.)
 (1) accesses it in order to actually do locking, but if obj is an
 instantiation of a class that uses the this reference internally for
 locking, you end up with potential deadlocks.

Deadlocks are endemic to mutex-based programming and are not avoided inherently or preferentially by the alternative you discussed (exposing unrestricted mutex).
 Therefore, you can say
 that obj's mutex is exposed.

Nothing could stop one, but that claim is incorrect.

It is exposed as far as using it for anything sensible goes. No, synchronized (obj) does not let you delete the monitor of obj or alter it or something along these lines, but I don't think that's worth bringing into this argument. This has always been about locking and unlocking a publicly exposed mutex, and about preventing common deadlock sources.
 Andrei

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 10:28 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 19:12, Andrei Alexandrescu wrote:
 On 5/30/12 9:23 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

Generally in two ways: 1) synchronized (obj)

This is not accessing the mutex for arbitrary operations.

No, it is indeed not. You didn't explicitly say you wanted to do "arbitrary operations", so I assumed that by accessing the mutex you meant doing the only two things that are sensible for a mutex: Locking and unlocking it. What else would you want to do?

For example, locking it in one function and unlocking it in another. This is impossible with synchronized(obj). To summarize, the synchronized(obj) operation is not exposing the mutex, only manipulate it in a specific way. Andrei
May 30 2012
parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 30-05-2012 20:25, Andrei Alexandrescu wrote:
 On 5/30/12 10:28 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 19:12, Andrei Alexandrescu wrote:
 On 5/30/12 9:23 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

Generally in two ways: 1) synchronized (obj)

This is not accessing the mutex for arbitrary operations.

No, it is indeed not. You didn't explicitly say you wanted to do "arbitrary operations", so I assumed that by accessing the mutex you meant doing the only two things that are sensible for a mutex: Locking and unlocking it. What else would you want to do?

For example, locking it in one function and unlocking it in another. This is impossible with synchronized(obj). To summarize, the synchronized(obj) operation is not exposing the mutex, only manipulate it in a specific way.

It clearly exposes it *enough* to cause potential for deadlocks by manipulating it in this specific way, which is what this is all about. I don't think arguing about what "exposed" means in this specific context (or perhaps in general?) or what levels of manipulation justify the use of the term "expose" will get the discussion anywhere...
 Andrei

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 11:38 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 20:25, Andrei Alexandrescu wrote:
 To summarize, the synchronized(obj) operation is not exposing the mutex,
 only manipulate it in a specific way.

It clearly exposes it *enough* to cause potential for deadlocks by manipulating it in this specific way, which is what this is all about.

Nobody claimed synchronized protects against deadlocks. Unfortunately, raw mutexes are just as bad in that regard.
 I don't think arguing about what "exposed" means in this specific
 context (or perhaps in general?) or what levels of manipulation justify
 the use of the term "expose" will get the discussion anywhere...

I simply pointed out a factual mistake in your argument. Andrei
May 30 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

I don't think so. Synchronized classes are the unit of scoped locking in D. If you want to do all scoped locking internally, make the class private.
 Besides, I really like
 the synchronized block keyword thing. It's a great mechanism for locking.

Me too! Andrei
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

I don't think so. Synchronized classes are the unit of scoped locking in D. If you want to do all scoped locking internally, make the class private.

Maybe you didn't read thoroughly the first part of my post (the example that shows a deadlock that is easily preventable if the mutex isn't exposed).

The mutex is not exposed.
 The issue is that it's possible to hijack the mutex that you
 are using to protect the class data in order to protect *external* data.
 Hijacking is not possible if the mutex is private (i.e. a private member).

 So great! Just don't use the builtin object mutex, because it exposes
 possible race conditions. But then why do we have it (the object hidden
 mutex)? And now I can't use synchronized classes to ensure the whole
 thing is protected, I have to write all my functions like this:

 void foo()
 {
 synchronized(this.mutex) // lock a private mutex
 {
 }
 }

This is a good example. But many synchronized classes are actually meaningful (e.g. the classic synchronized queue etc). Andrei
May 30 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 21:12, Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

I don't think so. Synchronized classes are the unit of scoped locking in D. If you want to do all scoped locking internally, make the class private.

Maybe you didn't read thoroughly the first part of my post (the example that shows a deadlock that is easily preventable if the mutex isn't exposed).

The mutex is not exposed.

I'm trying really hard to not to be rather impolite here, but I don't know how else to put it: You seem to be the only one who subscribes to this definition of "exposed". I like to think that there are varying degrees of exposing resources in programming. The mutex may not be directly exposed in the sense that you can obtain a reference (though in reality in all compiler implementations, you can), but it is exposed in the sense that you can lock and unlock it, which is a mutation of state and program flow. I think we need to set aside this matter about what exposing something really means. The mere fact is that you can lock and unlock the mutex of any object.
 The issue is that it's possible to hijack the mutex that you
 are using to protect the class data in order to protect *external* data.
 Hijacking is not possible if the mutex is private (i.e. a private
 member).

 So great! Just don't use the builtin object mutex, because it exposes
 possible race conditions. But then why do we have it (the object hidden
 mutex)? And now I can't use synchronized classes to ensure the whole
 thing is protected, I have to write all my functions like this:

 void foo()
 {
 synchronized(this.mutex) // lock a private mutex
 {
 }
 }

This is a good example. But many synchronized classes are actually meaningful (e.g. the classic synchronized queue etc). Andrei

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
next sibling parent deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 21:25, Alex Rønne Petersen a écrit :
 On 30-05-2012 21:12, Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

I don't think so. Synchronized classes are the unit of scoped locking in D. If you want to do all scoped locking internally, make the class private.

Maybe you didn't read thoroughly the first part of my post (the example that shows a deadlock that is easily preventable if the mutex isn't exposed).

The mutex is not exposed.

I'm trying really hard to not to be rather impolite here, but I don't know how else to put it: You seem to be the only one who subscribes to this definition of "exposed". I like to think that there are varying degrees of exposing resources in programming. The mutex may not be directly exposed in the sense that you can obtain a reference (though in reality in all compiler implementations, you can), but it is exposed in the sense that you can lock and unlock it, which is a mutation of state and program flow. I think we need to set aside this matter about what exposing something really means. The mere fact is that you can lock and unlock the mutex of any object.

Expliciting terms is always useful, even if it seems sometime irritating.
May 30 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 12:25 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 21:12, Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

I don't think so. Synchronized classes are the unit of scoped locking in D. If you want to do all scoped locking internally, make the class private.

Maybe you didn't read thoroughly the first part of my post (the example that shows a deadlock that is easily preventable if the mutex isn't exposed).

The mutex is not exposed.

I'm trying really hard to not to be rather impolite here, but I don't know how else to put it: You seem to be the only one who subscribes to this definition of "exposed".

I don't care as long as it's the correct one. The mutex is not exposed. Andrei
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 2:45 AM, Regan Heath wrote:
 Ok, how about you tell us what term you want to use for the current
 state of affairs then, because arguing about this is pointless and I'm
 happy to use whatever term you deem appropriate (because I couldn't care
 less what the term is - as long as we all know what is meant, which I
 believe is the case here).

The idiom is called "scoped locking". Andrei
May 31 2012
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 3:27 AM, Regan Heath wrote:
 On Thu, 31 May 2012 10:49:27 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:45 AM, Regan Heath wrote:
 Ok, how about you tell us what term you want to use for the current
 state of affairs then, because arguing about this is pointless and I'm
 happy to use whatever term you deem appropriate (because I couldn't care
 less what the term is - as long as we all know what is meant, which I
 believe is the case here).

The idiom is called "scoped locking".

So.. the mutex is "scoped locking"?

No, the Java-style approach with the synchronized statement etc. Andrei
May 31 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 3:27 AM, Regan Heath wrote:
 I think
 the mutex is "available for locking and unlocking" <- need a word for
 that, which is not "exposed". How about accessible, available, usable,
 or just plain lockable .. So, the problem here is that the mutex is
 lockable by external code via synchronized() and this means a certain
 type of deadlock is possible.

But this is a protection/visibility issue, which is orthogonal on the locking capability. It's as if you say "int is not good because anyone can overflow it." Okay! Make it private inside a CheckedInt class.
 Moving on..

 I think the change mentioned in TDPL to restrict synchronized to
 synchronized classes is a step in the right direction WRT wasted monitor
 space and people freely locking anything. But, it is exactly the case
 which results in more possible deadlocks (the cause of this thread) AND
 I think it's actually far more likely people will want to use a
 synchronized statement on a class which is not itself synchronized, like
 for example an existing container class.

 Given that, restricting synchronized statements to synchronized classes
 seems entirely wrong to me.

So where's the mutex that would be used to synchronize objects that are not synchronizable?
 In fact, I would say you almost want to stop
 people using synchronized statements on synchronized classes because:
 1. If a synchronized class is written correctly it should not be
 necessary in the general case(*)
 2. It raises the chances of deadlocks (the cause of this thread).
 3. It means that classes in general will be simpler to write (no need to
 worry about synchronization) and also more lightweight for use in
 non-threaded/non-shared cases. (because we'd provide a template wrapper
 to make them synchronizable)

There are cases in which you want to do multiple operations under a single critical section, even though the API is otherwise well-designed. That may be for correctness, efficiency, or both. I don't see why we'd want to disallow that, it's a good idiom.
 So, more and more I'm thinking it would be better to provide a
 library/runtime co-operative solution where we have an interface which
 is required for synchronized statements, and a wrapper template to
 implement that interface for any existing non-synchronized class.
 Meaning, the choice is either to write a synchronized class (rare) or a
 non-synchronized class - knowing it can easily be synchronized if/when
 needed.

Looking forward for a fleshed out proposal. Make sure you motivate it properly.
 The "liquid lock" problem mentioned earlier is an interesting one that I
 have not personally experienced, perhaps because I don't lock anything
 but mutex primitives and I never to re-assign these.

 (*) Locking on a larger scope can be achieved by providing a method
 taking a delegate (see earlier thread/replies) and/or exposing
 lock/unlock for those few situations where the delegate method cannot be
 used. These are the few cases in which this cannot be avoided as the
 lock/unlock are separated by more than a single scope (so synchronized
 statements don't help in these cases either).

On first look, the inversion of control using delegates delegates has similar liabilities as straight scoped locking. Andrei
May 31 2012
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 31/05/2012 13:13, Andrei Alexandrescu a écrit :
 On 5/31/12 3:27 AM, Regan Heath wrote:
 I think
 the mutex is "available for locking and unlocking" <- need a word for
 that, which is not "exposed". How about accessible, available, usable,
 or just plain lockable .. So, the problem here is that the mutex is
 lockable by external code via synchronized() and this means a certain
 type of deadlock is possible.

But this is a protection/visibility issue, which is orthogonal on the locking capability. It's as if you say "int is not good because anyone can overflow it." Okay! Make it private inside a CheckedInt class.
 Moving on..

 I think the change mentioned in TDPL to restrict synchronized to
 synchronized classes is a step in the right direction WRT wasted monitor
 space and people freely locking anything. But, it is exactly the case
 which results in more possible deadlocks (the cause of this thread) AND
 I think it's actually far more likely people will want to use a
 synchronized statement on a class which is not itself synchronized, like
 for example an existing container class.

 Given that, restricting synchronized statements to synchronized classes
 seems entirely wrong to me.

So where's the mutex that would be used to synchronize objects that are not synchronizable?
 In fact, I would say you almost want to stop
 people using synchronized statements on synchronized classes because:
 1. If a synchronized class is written correctly it should not be
 necessary in the general case(*)
 2. It raises the chances of deadlocks (the cause of this thread).
 3. It means that classes in general will be simpler to write (no need to
 worry about synchronization) and also more lightweight for use in
 non-threaded/non-shared cases. (because we'd provide a template wrapper
 to make them synchronizable)

There are cases in which you want to do multiple operations under a single critical section, even though the API is otherwise well-designed. That may be for correctness, efficiency, or both. I don't see why we'd want to disallow that, it's a good idiom.

Because it is now unclear who is controlling the lock. The solution consisting in passing a delegate as parameter or as template is superior, because it is now clear who is in charge of the synchronization, reducing greatly chances of deadlock.
May 31 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

It can also be a lot clunkier for certain abstractions. Say I want a ProducerConsumerQueue. It's much more convenient to simply make it a synchronized class with the classic primitives, instead of primitives that accept delegates etc. Nevertheless I think there's merit in this idea. One thing to point out is that the idiom can easily be done today with a regular class holding a synchronized class private member. So we got everything we need. Andrei
May 31 2012
parent reply deadalnix <deadalnix gmail.com> writes:
Le 31/05/2012 20:17, Andrei Alexandrescu a écrit :
 On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

It can also be a lot clunkier for certain abstractions. Say I want a ProducerConsumerQueue. It's much more convenient to simply make it a synchronized class with the classic primitives, instead of primitives that accept delegates etc. Nevertheless I think there's merit in this idea. One thing to point out is that the idiom can easily be done today with a regular class holding a synchronized class private member. So we got everything we need. Andrei

I was thinking about that. Here is what I ended up to think is the best solution : synchronized classes exists. By default, they can't be use as parameter for synchronized(something) . synchronized(something) will be valid is something provide opSynchronized(scope delegate void()) or something similar. Think opApply here. The synchronized statement is rewritten in a call to that delegate. Here are the benefit of such an approach : 1/ Lock and unlock are not exposed. You can only use them by pair. 2/ You cannot lock on any object, so you avoid most liquid locks and don't waste memory. 3/ synchronized classes ensure that a class can be shared and it internal are protected from concurrent access. 4/ It is not possible possible by default to lock on synchronized classes's instances. It grant better control over the lock and it is now clear which piece of code is responsible of it. 5/ The design allow the programmer to grant the permission to lock on synchronized classes's instances if he/she want to. 6/ It is now possible to synchronize on a broader range of user defined stuffs. The main drawback is the same as opApply : return (and break/continue but it is less relevant for opSynchronized). Solution to this problem have been proposed in the past using compiler and stack magic. It open door for stuff like : ReadWriteLock rw; synchronized(rw.read) { } synchronized(rw.write) { } And many types of lock : spin lock, interprocesses locks, semaphores, . . . And all can be used with the synchronized syntax, and without exposing locking and unlocking primitives. What do people think ?
Jun 01 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 01.06.2012 16:26, deadalnix wrote:
  Here is what I ended up to think is the best
 solution :

 synchronized classes exists. By default, they can't be use as parameter
 for synchronized(something) .

 synchronized(something) will be valid is something provide
 opSynchronized(scope delegate void()) or something similar. Think
 opApply here. The synchronized statement is rewritten in a call to that
 delegate.

 Here are the benefit of such an approach :
 1/ Lock and unlock are not exposed. You can only use them by pair.
 2/ You cannot lock on any object, so you avoid most liquid locks and
 don't waste memory.
 3/ synchronized classes ensure that a class can be shared and it
 internal are protected from concurrent access.
 4/ It is not possible possible by default to lock on synchronized
 classes's instances. It grant better control over the lock and it is now
 clear which piece of code is responsible of it.
 5/ The design allow the programmer to grant the permission to lock on
 synchronized classes's instances if he/she want to.
 6/ It is now possible to synchronize on a broader range of user defined
 stuffs.

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem
 have been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 And many types of lock : spin lock, interprocesses locks, semaphores, .
 . . And all can be used with the synchronized syntax, and without
 exposing locking and unlocking primitives.

 What do people think ?

+1. Works for me. It refines what I believe the shadow cabinet (loosely: me, you, Alex, Regan Heath and Steven) propose. P.S. Removing monitor from non-synced/shared classes would be good too. As a separate matter. -- Dmitry Olshansky
Jun 01 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 01/06/2012 14:52, Steven Schveighoffer a écrit :
 On Fri, 01 Jun 2012 08:38:45 -0400, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:

 On 01.06.2012 16:26, deadalnix wrote:
 Here is what I ended up to think is the best
 solution :

 synchronized classes exists. By default, they can't be use as parameter
 for synchronized(something) .

 synchronized(something) will be valid is something provide
 opSynchronized(scope delegate void()) or something similar. Think
 opApply here. The synchronized statement is rewritten in a call to that
 delegate.

 Here are the benefit of such an approach :
 1/ Lock and unlock are not exposed. You can only use them by pair.
 2/ You cannot lock on any object, so you avoid most liquid locks and
 don't waste memory.
 3/ synchronized classes ensure that a class can be shared and it
 internal are protected from concurrent access.
 4/ It is not possible possible by default to lock on synchronized
 classes's instances. It grant better control over the lock and it is now
 clear which piece of code is responsible of it.
 5/ The design allow the programmer to grant the permission to lock on
 synchronized classes's instances if he/she want to.
 6/ It is now possible to synchronize on a broader range of user defined
 stuffs.

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem
 have been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 And many types of lock : spin lock, interprocesses locks, semaphores, .
 . . And all can be used with the synchronized syntax, and without
 exposing locking and unlocking primitives.

 What do people think ?

+1. Works for me. It refines what I believe the shadow cabinet (loosely: me, you, Alex, Regan Heath and Steven) propose.

Is this really necessary? When is opSynchronized going to be written any way other than: _mutex.lock(); scope(exit) _mutex.unlock(); dg(); I'll note that it's easier to forget to lock or unlock if the compiler isn't enforcing it. You might even naively do this: _mutex.lock(); dg(); _mutex.unlock(); // not called on exception thrown! I kind of like the __lock() __unlock() pair that the compiler always calls both in the right place/way. Yes, you could just leave those implementations blank, but very unlikely. Plus, we already have issues with inout and delegates for opApply, this would have the same issues.
 P.S. Removing monitor from non-synced/shared classes would be good
 too. As a separate matter.

I think at this point, we should leave it there until we can really figure out a detailed plan on how to deal with it. It currently affects all runtime code which does virtual function lookups or interface lookups, and alignment. We would have to change a lot of compiler and runtime code to remove it.

A lot of work have to be done, for sure. But concurency is the next big thing D will have to face IMO. When const/immutable are fixed :D
Jun 01 2012
prev sibling next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 01-06-2012 14:26, deadalnix wrote:
 Le 31/05/2012 20:17, Andrei Alexandrescu a écrit :
 On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

It can also be a lot clunkier for certain abstractions. Say I want a ProducerConsumerQueue. It's much more convenient to simply make it a synchronized class with the classic primitives, instead of primitives that accept delegates etc. Nevertheless I think there's merit in this idea. One thing to point out is that the idiom can easily be done today with a regular class holding a synchronized class private member. So we got everything we need. Andrei

I was thinking about that. Here is what I ended up to think is the best solution : synchronized classes exists. By default, they can't be use as parameter for synchronized(something) . synchronized(something) will be valid is something provide opSynchronized(scope delegate void()) or something similar. Think opApply here. The synchronized statement is rewritten in a call to that delegate. Here are the benefit of such an approach : 1/ Lock and unlock are not exposed. You can only use them by pair. 2/ You cannot lock on any object, so you avoid most liquid locks and don't waste memory. 3/ synchronized classes ensure that a class can be shared and it internal are protected from concurrent access. 4/ It is not possible possible by default to lock on synchronized classes's instances. It grant better control over the lock and it is now clear which piece of code is responsible of it. 5/ The design allow the programmer to grant the permission to lock on synchronized classes's instances if he/she want to. 6/ It is now possible to synchronize on a broader range of user defined stuffs. The main drawback is the same as opApply : return (and break/continue but it is less relevant for opSynchronized). Solution to this problem have been proposed in the past using compiler and stack magic. It open door for stuff like : ReadWriteLock rw; synchronized(rw.read) { } synchronized(rw.write) { } And many types of lock : spin lock, interprocesses locks, semaphores, . .. . And all can be used with the synchronized syntax, and without exposing locking and unlocking primitives. What do people think ?

Your idea is great, but it has one (or arguably many) fundamental flaw, the same one that opApply does: The delegate's type is fixed. That is, you can't call opSynchronized() in a pure function, for example (same goes for nothrow, safe, ...). -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jun 01 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 01/06/2012 14:52, Alex Rønne Petersen a écrit :
 On 01-06-2012 14:26, deadalnix wrote:
 Le 31/05/2012 20:17, Andrei Alexandrescu a écrit :
 On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

It can also be a lot clunkier for certain abstractions. Say I want a ProducerConsumerQueue. It's much more convenient to simply make it a synchronized class with the classic primitives, instead of primitives that accept delegates etc. Nevertheless I think there's merit in this idea. One thing to point out is that the idiom can easily be done today with a regular class holding a synchronized class private member. So we got everything we need. Andrei

I was thinking about that. Here is what I ended up to think is the best solution : synchronized classes exists. By default, they can't be use as parameter for synchronized(something) . synchronized(something) will be valid is something provide opSynchronized(scope delegate void()) or something similar. Think opApply here. The synchronized statement is rewritten in a call to that delegate. Here are the benefit of such an approach : 1/ Lock and unlock are not exposed. You can only use them by pair. 2/ You cannot lock on any object, so you avoid most liquid locks and don't waste memory. 3/ synchronized classes ensure that a class can be shared and it internal are protected from concurrent access. 4/ It is not possible possible by default to lock on synchronized classes's instances. It grant better control over the lock and it is now clear which piece of code is responsible of it. 5/ The design allow the programmer to grant the permission to lock on synchronized classes's instances if he/she want to. 6/ It is now possible to synchronize on a broader range of user defined stuffs. The main drawback is the same as opApply : return (and break/continue but it is less relevant for opSynchronized). Solution to this problem have been proposed in the past using compiler and stack magic. It open door for stuff like : ReadWriteLock rw; synchronized(rw.read) { } synchronized(rw.write) { } And many types of lock : spin lock, interprocesses locks, semaphores, . .. . And all can be used with the synchronized syntax, and without exposing locking and unlocking primitives. What do people think ?

Your idea is great, but it has one (or arguably many) fundamental flaw, the same one that opApply does: The delegate's type is fixed. That is, you can't call opSynchronized() in a pure function, for example (same goes for nothrow, safe, ...).

I was also thinking about passing the delegate as template parameter but wanted to keep design proposal consistent with what already exists. It solve some issues. Maybe opApply have to be modified that way, or, better, both could work. Anyway, some issue are known and have to be fixed with opApply, and the same issue are present here.I did mention that in my proposal. Still, I think it is better to have consistent mecanisms in the language, and it is no more work to fix that for opApply than to fix that for both opApply and opSynchronized .
Jun 01 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 01/06/2012 22:55, Sean Kelly a écrit :
 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:
 The main drawback is the same as opApply : return (and break/continue but it
is less relevant for opSynchronized). Solution to this problem have been
proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

Opens the door? This works today exactly as outlined above. Or am I missing a part of your argument?
 And many types of lock : spin lock, interprocesses locks, semaphores, . . .
And all can be used with the synchronized syntax, and without exposing locking
and unlocking primitives.

All works today.

Unless you do some monitor magic, it doesn't.
Jun 03 2012
parent reply deadalnix <deadalnix gmail.com> writes:
Le 03/06/2012 21:40, Andrew Wiley a écrit :
 On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

     Le 01/06/2012 22:55, Sean Kelly a écrit :

         On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


             The main drawback is the same as opApply : return (and
             break/continue but it is less relevant for opSynchronized).
             Solution to this problem have been proposed in the past
             using compiler and stack magic.

             It open door for stuff like :
             ReadWriteLock rw;
             synchronized(rw.read) {

             }

             synchronized(rw.write) {

             }


         Opens the door?  This works today exactly as outlined above.  Or
         am I missing a part of your argument?

             And many types of lock : spin lock, interprocesses locks,
             semaphores, . . . And all can be used with the synchronized
             syntax, and without exposing locking and unlocking primitives.


         All works today.


     Unless you do some monitor magic, it doesn't.

 Yes, it does.
 -----
 class Something {
      private:
          ReadWriteLock _rw;
      public:
          this() {
              _rw = new ReadWriteLock();
          }
          void doSomething() shared {
              synchronized(_rw.read) {
                  // do things
              }
          }
 }
 -----
 I've used this pattern in code. There might be some casting required
 because the core synchronization primitives haven't been updated to use
 shared yet.

And where is that ReadWriteLock ?
Jun 03 2012
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 04/06/2012 02:03, Andrew Wiley a écrit :
 On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

     Le 03/06/2012 21:40, Andrew Wiley a écrit :

         On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
         <mailto:deadalnix gmail.com>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>> wrote:

             Le 01/06/2012 22:55, Sean Kelly a écrit :

                 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


                     The main drawback is the same as opApply : return (and
                     break/continue but it is less relevant for
         opSynchronized).
                     Solution to this problem have been proposed in the past
                     using compiler and stack magic.

                     It open door for stuff like :
                     ReadWriteLock rw;
                     synchronized(rw.read) {

                     }

                     synchronized(rw.write) {

                     }


                 Opens the door?  This works today exactly as outlined
         above.  Or
                 am I missing a part of your argument?

                     And many types of lock : spin lock, interprocesses
         locks,
                     semaphores, . . . And all can be used with the
         synchronized
                     syntax, and without exposing locking and unlocking
         primitives.


                 All works today.


             Unless you do some monitor magic, it doesn't.

         Yes, it does.
         -----
         class Something {
              private:
                  ReadWriteLock _rw;
              public:
                  this() {
                      _rw = new ReadWriteLock();
                  }
                  void doSomething() shared {
                      synchronized(_rw.read) {
                          // do things
                      }
                  }
         }
         -----
         I've used this pattern in code. There might be some casting required
         because the core synchronization primitives haven't been updated
         to use
         shared yet.


     And where is that ReadWriteLock ?

 On the GC heap, just like the Monitor object pointed to by __monitor if
 you mark a method or class as synchronized.

I meant where is the code ?
Jun 03 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 04/06/2012 02:21, Andrew Wiley a écrit :
 On Sun, Jun 3, 2012 at 5:13 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

     Le 04/06/2012 02:03, Andrew Wiley a écrit :

         On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com
         <mailto:deadalnix gmail.com>

         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>> wrote:

             Le 03/06/2012 21:40, Andrew Wiley a écrit :

                 On Sun, Jun 3, 2012 at 12:29 PM, deadalnix
         <deadalnix gmail.com <mailto:deadalnix gmail.com>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>>> wrote:

                     Le 01/06/2012 22:55, Sean Kelly a écrit :

                         On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


                             The main drawback is the same as opApply :
         return (and
                             break/continue but it is less relevant for
                 opSynchronized).
                             Solution to this problem have been proposed
         in the past
                             using compiler and stack magic.

                             It open door for stuff like :
                             ReadWriteLock rw;
                             synchronized(rw.read) {

                             }

                             synchronized(rw.write) {

                             }


                         Opens the door?  This works today exactly as
         outlined
                 above.  Or
                         am I missing a part of your argument?

                             And many types of lock : spin lock,
         interprocesses
                 locks,
                             semaphores, . . . And all can be used with the
                 synchronized
                             syntax, and without exposing locking and
         unlocking
                 primitives.


                         All works today.


                     Unless you do some monitor magic, it doesn't.

                 Yes, it does.
                 -----
                 class Something {
                      private:
                          ReadWriteLock _rw;
                      public:
                          this() {
                              _rw = new ReadWriteLock();
                          }
                          void doSomething() shared {
                              synchronized(_rw.read) {
                                  // do things
                              }
                          }
                 }
                 -----
                 I've used this pattern in code. There might be some
         casting required
                 because the core synchronization primitives haven't been
         updated
                 to use
                 shared yet.


             And where is that ReadWriteLock ?

         On the GC heap, just like the Monitor object pointed to by
         __monitor if
         you mark a method or class as synchronized.


     I meant where is the code ?

 My apologies, it's actually ReadWriteMutex:
 http://dlang.org/phobos/core_sync_rwmutex.html

Which does use monitor magic.
Jun 03 2012
prev sibling parent =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 04-06-2012 02:03, Andrew Wiley wrote:
 On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

     Le 03/06/2012 21:40, Andrew Wiley a écrit :

         On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
         <mailto:deadalnix gmail.com>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>> wrote:

             Le 01/06/2012 22:55, Sean Kelly a écrit :

                 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


                     The main drawback is the same as opApply : return (and
                     break/continue but it is less relevant for
         opSynchronized).
                     Solution to this problem have been proposed in the past
                     using compiler and stack magic.

                     It open door for stuff like :
                     ReadWriteLock rw;
                     synchronized(rw.read) {

                     }

                     synchronized(rw.write) {

                     }


                 Opens the door?  This works today exactly as outlined
         above.  Or
                 am I missing a part of your argument?

                     And many types of lock : spin lock, interprocesses
         locks,
                     semaphores, . . . And all can be used with the
         synchronized
                     syntax, and without exposing locking and unlocking
         primitives.


                 All works today.


             Unless you do some monitor magic, it doesn't.

         Yes, it does.
         -----
         class Something {
              private:
                  ReadWriteLock _rw;
              public:
                  this() {
                      _rw = new ReadWriteLock();
                  }
                  void doSomething() shared {
                      synchronized(_rw.read) {
                          // do things
                      }
                  }
         }
         -----
         I've used this pattern in code. There might be some casting required
         because the core synchronization primitives haven't been updated
         to use
         shared yet.


     And where is that ReadWriteLock ?

 On the GC heap, just like the Monitor object pointed to by __monitor if
 you mark a method or class as synchronized.

The monitor in obj.__monitor is actually allocated on the native C heap, and, in some cases, leaked... one of the many reasons I want it gone. -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jun 03 2012
prev sibling parent travert phare.normalesup.org (Christophe Travert) writes:
deadalnix , dans le message (digitalmars.D:169136), a écrit :
 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {
 
 }
 
 synchronized(rw.write) {
 
 }
 
 And many types of lock : spin lock, interprocesses locks, semaphores, . 
 . . And all can be used with the synchronized syntax, and without 
 exposing locking and unlocking primitives.
 

synchronize (foo) { ... } could do anything. You might as well propose a syntax sugar to call any function or method taking a delegate. You could make it do any operation with the idiom: struct MyType { void foo(void delegate() dg) { this.open(); scope(exit) this.close(); dg(); } ... } a.foo { // maybe some keyword should be necessary ... } Same problem as opApply, should take different type of delegates, etc...
Jun 09 2012
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
Le 31/05/2012 16:01, Regan Heath a écrit :
 True, it's basically the same as a synchronized block in that respect.
 What we actually want is a way to limit the calls made by the delegate
 to methods of the object itself. If it could not call a synchronized
 method on a 2nd object, you could not possibly deadlock. Except, that is
 to say, unless you held a separate lock beforehand - but, the important
 point here is that you would have to take both locks explicitly, rather
 than by an implicit synchronized method call, making the bug far more
 obvious to code inspection.

 R

I'm not sure I follow you here. Can you elaborate on that ?
May 31 2012
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
 But this is a protection/visibility issue, which is orthogonal on the
 locking capability. It's as if you say "int is not good because anyone
 can overflow it." Okay! Make it private inside a CheckedInt class.

Sorry, that's a bad comparison. CheckedInt is to int, what CheckedMutex is to mutex - but I'm not suggesting anything like a CheckedMutex. I'm suggesting "mutex" but kept private inside the class /that it locks/. Yes, it's a visibility issue, the issue is that the mutex used by synchronized classes/methods is too visible/accessible and this opens it up for deadlocks which are otherwise impossible.

Sure it's awful comparison.
 So where's the mutex that would be used to synchronize objects that
 are not synchronizable?

In the wrapper class/struct/object which derives a synchronized class/struct from the original. My D foo is not strong enough to just come up with valid D code for the idiom on the fly, but essentially you wrap the original object in a new object using a template which adds the mutex member and the interface methods (lock, tryLock, and unlock) required. No, this doesn't work with "final" classes.. but it shouldn't, they're final after all. For them you need to add/manage the mutex manually - the price you pay for "final".

OK let me land you a hand here. My proposal, that I think fits your ideas quite favorably. I'll enumerate certain assumptions beforehand since it's one of most confusing threads I ever followed. 1. synchronized class means: always allocate hidden monitor mutex, lock it on every public method of this class. 2. synchronized(x,y,z) is lowered to (I use Steven's idea): auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z sorted FOR EACH item IN sorted tuple add code in [[ ... ]] [[// conceptual item.__lock(); scope(exit) item.__unlock(); ]] In other words it works for every object that defines lock/unlock. Multiple object version works only if there is opCmp defined. (by address whatever, any total ordering should work) Per definition above synchronized classes AS IS can't be *synchronized* on. Instead their public methods are implicitly synchronized. The end result is: User can synchronize on various synchronization entities and even plug in custom synchronization primitives (say OS Y provides have this fancy lock type Z). It's explicit in as sense that object supports it via __lock__/unlock. Obviously one still can use __lock/__unlock explicitly just don't forget to wear big HERE BE DRAGONS banner. Synchronized class encapsulate mutex completely - nobody aside from designer of the class may (a)use it. Does it makes sense? -- Dmitry Olshansky
May 31 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 31.05.2012 19:11, Steven Schveighoffer wrote:
 On Thu, 31 May 2012 10:49:52 -0400, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your
 ideas quite favorably.

[snip]
 Does it makes sense?

Everything but the ordering. I like the idea of ordering the locks based on an intrinsic value, but opCmp isn't it. Objects can mutate, and opCmp may produce different results. Imagine: synchronized(a, b) // at this point a < b { a.makeGreaterThan(b); assert(a > b); }

No way! I live in world where victim's hand is cut off as a punishment for mutating a lock after creation ;)
 You *never* want the ordering to change.

 But I think we can probably work that out. What about comparing handles
 of the mutexes? So you sort based on some property __mutex_id() which
 must return a unique size_t that can be used to always define an
 ordering of locking mutexes? Most mutexes are allocated by the OS, and
 so their handles won't be affected by a moving GC, so you likely will
 use the handle value.

This could work. In fact I imagined comparing handles ... As with math at large you may either project (biject) your domain to a well-known one (like numbers, mutex_id in your case) and work with it or define all relevant properties for whatever your values in this domain are (providing custom ordering for user defined types). -- Dmitry Olshansky
May 31 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 31.05.2012 20:54, Regan Heath wrote:

 No way! I live in world where victim's hand is cut off as a punishment
 for mutating a lock after creation ;)

This is related to the "liquid lock" problem raised/mentioned elsewhere in the thread. If the reference you lock can change, then one thread/iteration may lock one instance, the lock instance mutates, and a 2nd thread/iteration locks the new instance and both execute more or less in parallel over code which was supposed to be serialized. It's not a problem if the lock object is the object being mutated/used by the code, but it is a problem if the lock object is protecting other shared objects. It's not something I have ever bumped into myself, because the bulk of my locking experience is in C/C++ and I use a locking primitive which is initialised once and never mutates.

Yup, C/C++, locking only special OS provided stuff, it's my background exactly.
 You *never* want the ordering to change.

 But I think we can probably work that out. What about comparing handles
 of the mutexes? So you sort based on some property __mutex_id() which
 must return a unique size_t that can be used to always define an
 ordering of locking mutexes? Most mutexes are allocated by the OS, and
 so their handles won't be affected by a moving GC, so you likely will
 use the handle value.

This could work. In fact I imagined comparing handles ... As with math at large you may either project (biject) your domain to a well-known one (like numbers, mutex_id in your case) and work with it or define all relevant properties for whatever your values in this domain are (providing custom ordering for user defined types).

This ordering idea is present in TDPL.. have you both read the link Andrei posted? http://goo.gl/ZhPM2 (see 13.15)

Yes, in fact I was going by TDPL. I just propose to extend it to entities other then class instances ( more specifically only those with __lock/__unlock ). -- Dmitry Olshansky
May 31 2012
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 31.05.2012 19:05, Regan Heath wrote:
 Yes, and it's all more intentional/flexible than what we have now, but.. :)

 Does it address what I thought was the main "problem" case. That is, as
 soon as you lock 2 objects, one via a synchronized() statement and the
 other via a synchronized method you can get a non-obvious deadlock. e.g.

 synchronized class A
 {
 void foo() {} // implicitly locks instance of A
 }

 class B
 {
 void __lock() {} // locks instance of B
 void __unlock() {} // unlocks instance of B
 }

 shared A a;
 shared B b;

 void main()
 {
 a = new shared(A)();
 b = new shared(B)();
 ..start thread which locks a then b..
 synchronized(b) // locks b 'explicitly'
 {
 a.foo(); // locks a 'implicitly'
 }
 }

 .. but, hang on, can a thread actually lock a and then b? If 'a' cannot
 participate in a synchronized statement (which it can't under this
 proposal) then no, there is no way to lock 'a' except by calling a
 member. So, provided 'a' does not have a member which locks 'b' - were
 deadlock safe!

 So.. problem solved; by preventing external/public lock/unlock on a
 synchronized class. (I think the proposal should enforce this
 restriction; synchronized classes cannot define __lock/__unlock).

Surprisingly it is. And if A can't access B it's fool-proof. As to __lock/__unlock defined for synchronized classes, yes I believe compiler should catch it. -- Dmitry Olshansky
May 31 2012
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 7:49 AM, Dmitry Olshansky wrote:
 But this is a protection/visibility issue, which is orthogonal on the
 locking capability. It's as if you say "int is not good because anyone
 can overflow it." Okay! Make it private inside a CheckedInt class.

Sorry, that's a bad comparison. CheckedInt is to int, what CheckedMutex is to mutex - but I'm not suggesting anything like a CheckedMutex. I'm suggesting "mutex" but kept private inside the class /that it locks/. Yes, it's a visibility issue, the issue is that the mutex used by synchronized classes/methods is too visible/accessible and this opens it up for deadlocks which are otherwise impossible.

Sure it's awful comparison.

It's a great comparison. Andrei
May 31 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 7:49 AM, Dmitry Olshansky wrote:
 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

Great.
 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)

Great. What are regular calls to synchronized class methods lowered into?
 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

 Does it makes sense?

It does make sense, but I think we need a Lock struct type that makes sure code cannot screw up the number of lock and unlock calls. We shouldn't just expose bare lock() and unlock(). Andrei
May 31 2012
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 31.05.2012 22:33, Andrei Alexandrescu wrote:
 On 5/31/12 7:49 AM, Dmitry Olshansky wrote:
 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

Great.
 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)

Great. What are regular calls to synchronized class methods lowered into?

Posted in another reply, basically same lock(); scope(exit)unlock() of hidden monitor.
 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

 Does it makes sense?

It does make sense, but I think we need a Lock struct type that makes sure code cannot screw up the number of lock and unlock calls. We shouldn't just expose bare lock() and unlock().

Dunno. In fact __lock/__unlock could be artificially hidden from user but not compiler with his magic rewrites. But I believe there is value in having access to lock/unlock for those select cases where it's beneficial. -- Dmitry Olshansky
May 31 2012
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 12:27 PM, Sean Kelly wrote:
 On May 31, 2012, at 11:33 AM, Andrei Alexandrescu wrote:
 It does make sense, but I think we need a Lock struct type that
 makes sure code cannot screw up the number of lock and unlock
 calls. We shouldn't just expose bare lock() and unlock().

synchronized already works with classes that implement Object.Monitor, and we have scope guards. Do we really need an RTTI Lock struct as well?

At this point it's unclear to me what different branches of this sprawling thread are proposing. I have extracted one thing that we should definitely look into - formalizing the lowering of the synchronized statement. Other than that I failed to derive much signal. Andrei
May 31 2012
prev sibling next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 31-05-2012 16:49, Dmitry Olshansky wrote:
 But this is a protection/visibility issue, which is orthogonal on the
 locking capability. It's as if you say "int is not good because anyone
 can overflow it." Okay! Make it private inside a CheckedInt class.

Sorry, that's a bad comparison. CheckedInt is to int, what CheckedMutex is to mutex - but I'm not suggesting anything like a CheckedMutex. I'm suggesting "mutex" but kept private inside the class /that it locks/. Yes, it's a visibility issue, the issue is that the mutex used by synchronized classes/methods is too visible/accessible and this opens it up for deadlocks which are otherwise impossible.

Sure it's awful comparison.
 So where's the mutex that would be used to synchronize objects that
 are not synchronizable?

In the wrapper class/struct/object which derives a synchronized class/struct from the original. My D foo is not strong enough to just come up with valid D code for the idiom on the fly, but essentially you wrap the original object in a new object using a template which adds the mutex member and the interface methods (lock, tryLock, and unlock) required. No, this doesn't work with "final" classes.. but it shouldn't, they're final after all. For them you need to add/manage the mutex manually - the price you pay for "final".

OK let me land you a hand here. My proposal, that I think fits your ideas quite favorably. I'll enumerate certain assumptions beforehand since it's one of most confusing threads I ever followed. 1. synchronized class means: always allocate hidden monitor mutex, lock it on every public method of this class.

If this means that the monitor field in regular objects goes away, then I'm all for this, at least.
 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)


 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

 Does it makes sense?

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 31 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 31.05.2012 22:34, Alex Rønne Petersen wrote:
 On 31-05-2012 16:49, Dmitry Olshansky wrote:
 But this is a protection/visibility issue, which is orthogonal on the
 locking capability. It's as if you say "int is not good because anyone
 can overflow it." Okay! Make it private inside a CheckedInt class.

Sorry, that's a bad comparison. CheckedInt is to int, what CheckedMutex is to mutex - but I'm not suggesting anything like a CheckedMutex. I'm suggesting "mutex" but kept private inside the class /that it locks/. Yes, it's a visibility issue, the issue is that the mutex used by synchronized classes/methods is too visible/accessible and this opens it up for deadlocks which are otherwise impossible.

Sure it's awful comparison.
 So where's the mutex that would be used to synchronize objects that
 are not synchronizable?

In the wrapper class/struct/object which derives a synchronized class/struct from the original. My D foo is not strong enough to just come up with valid D code for the idiom on the fly, but essentially you wrap the original object in a new object using a template which adds the mutex member and the interface methods (lock, tryLock, and unlock) required. No, this doesn't work with "final" classes.. but it shouldn't, they're final after all. For them you need to add/manage the mutex manually - the price you pay for "final".

OK let me land you a hand here. My proposal, that I think fits your ideas quite favorably. I'll enumerate certain assumptions beforehand since it's one of most confusing threads I ever followed. 1. synchronized class means: always allocate hidden monitor mutex, lock it on every public method of this class.

If this means that the monitor field in regular objects goes away, then I'm all for this, at least.

Yes, it removes monitor in non-synchronized object. Glad you we are in agreement here. And all public methods lower to : <signature> { this.hidden__monitor.lock(); scope(exit) this.hidden__monitor.unlock(); ... //code } -- Dmitry Olshansky
May 31 2012
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 31.05.2012 18:49, Dmitry Olshansky wrote:
 I'll enumerate certain assumptions beforehand since it's one of most
 confusing threads I ever followed.

 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)


 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

It could even go one step forward. Though it require a great deal of though, here is an initial idea. So with the above we have synchronized class with monitor that is eagerly initialized (it's the propose of sync-ed class after all to use this mutex safely) and ordinary class that don't have it. Now if we let shared classes to have monitor field lazily initialized something interesting shows up. For the moment let's skip painful detail of cast(shared). It's then possible to introduce a restricted version of ownership system. Since ever member of shared class is shared or synchronized(?) it should be OK to access member of any depth in it by following the chain of monitor locks: (i.e. I'm assuming composition implies "owns" relationship, this ownership relation comes only for shared types) shared class A { shared B b; ... } shared class B { shared C c; ... } then: { ... A a = ....; // locks A's monitor, then B's monitor, unlocks A's // then C's monitor, unlocks B's // then calls do_smth, then unlocks C's //all of the above lock/unlock sequence has to be exception proof a.b.c.do_smth(); } Grunted assuming compositon == ownership is wrong in general, but could serve as a sensible default. Then weak non-owning references would be provided via dirty casting in special wrappers like { NotOwning!(shared T) x;//shared but doesn't own it, thus if //calling any resource inside of x, we should keep x.monitor locked //through out the operation. (unlike in the above case) } ... Obviously it's iteration 1 of idea, it needs to deal with possible atomic modifications, cast(shared), etc. -- Dmitry Olshansky
May 31 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 01.06.2012 0:39, Dmitry Olshansky wrote:
 On 31.05.2012 18:49, Dmitry Olshansky wrote:
 I'll enumerate certain assumptions beforehand since it's one of most
 confusing threads I ever followed.

 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)


 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

It could even go one step forward. Though it require a great deal of though, here is an initial idea. So with the above we have synchronized class with monitor that is eagerly initialized (it's the propose of sync-ed class after all to use this mutex safely) and ordinary class that don't have it. Now if we let shared classes to have monitor field lazily initialized something interesting shows up. For the moment let's skip painful detail of cast(shared). It's then possible to introduce a restricted version of ownership system. Since ever member of shared class is shared or synchronized(?) it should be OK to access member of any depth in it by following the chain of monitor locks: (i.e. I'm assuming composition implies "owns" relationship, this ownership relation comes only for shared types) shared class A { shared B b; ... } shared class B { shared C c; ... } then: { ... A a = ....; // locks A's monitor, then B's monitor, unlocks A's // then C's monitor, unlocks B's // then calls do_smth, then unlocks C's //all of the above lock/unlock sequence has to be exception proof a.b.c.do_smth(); } Grunted assuming compositon == ownership is wrong in general, but could

s/Grunted/Granted
 serve as a sensible default. Then weak non-owning references would be
 provided via dirty casting in special wrappers like
 {
 NotOwning!(shared T) x;//shared but doesn't own it, thus if
 //calling any resource inside of x, we should keep x.monitor locked
 //through out the operation. (unlike in the above case)
 }
 ...

 Obviously it's iteration 1 of idea, it needs to deal with possible
 atomic modifications, cast(shared), etc.

-- Dmitry Olshansky
May 31 2012
prev sibling parent reply mta`chrono <chrono mta-international.net> writes:
Am 31.05.2012 17:05, schrieb Regan Heath:
 .. but, hang on, can a thread actually lock a and then b?  If 'a' cannot
 participate in a synchronized statement (which it can't under this
 proposal) then no, there is no way to lock 'a' except by calling a
 member.  So, provided 'a' does not have a member which locks 'b' - were
 deadlock safe!
 
 So.. problem solved; by preventing external/public lock/unlock on a
 synchronized class.  (I think the proposal should enforce this
 restriction; synchronized classes cannot define __lock/__unlock).
 
 R
 

I think it doesn't matter whether you expose your mointor / locking / unlocking to the public or not. You can always unhappily create deadlocks that are hard to debug between tons of spaghetti code. shared A a; shared B b; void thread1() { synchronized(a) // locks A { synchronized(b) // ... then B { // ..... code .... } } } void thread2() { synchronized(b) // locks B { synchronized(a) // ... then A { // ..... code .... } } }
Jun 04 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 04/06/2012 10:56, Jonathan M Davis a écrit :
 On Monday, June 04, 2012 10:51:08 mta`chrono wrote:
 Am 31.05.2012 17:05, schrieb Regan Heath:
 .. but, hang on, can a thread actually lock a and then b?  If 'a' cannot
 participate in a synchronized statement (which it can't under this
 proposal) then no, there is no way to lock 'a' except by calling a
 member.  So, provided 'a' does not have a member which locks 'b' - were
 deadlock safe!

 So.. problem solved; by preventing external/public lock/unlock on a
 synchronized class.  (I think the proposal should enforce this
 restriction; synchronized classes cannot define __lock/__unlock).

 R

I think it doesn't matter whether you expose your mointor / locking / unlocking to the public or not. You can always unhappily create deadlocks that are hard to debug between tons of spaghetti code.

You can always create deadlocks, but if there's something which gives you little benefit but significantly increases the risk of deadlocks (e.g. making it easy to lock on a synchronized class' internal mutex via a synchronized block), then it's valuable to make it illegal. Because while it won't prevent all deadlocking, it _does_ eliminate one case where it's overly easy to deadlock. - Jonathan M Davis

At least illegal by default. The programmer may enable it by him/herself, but not fall in the trap inadvertently.
Jun 04 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 7:01 AM, Regan Heath wrote:
 I'm
 suggesting "mutex" but kept private inside the class /that it locks/.
 Yes, it's a visibility issue, the issue is that the mutex used by
 synchronized classes/methods is too visible/accessible and this opens it
 up for deadlocks which are otherwise impossible.

You're suggesting an idiom based on an unrestricted abstraction ("mutex") combined with a language feature ("private"). That's no progress because: 1. People can still misuse the unrestricted "mutex" as they have for the past 50 years. 2. The idiom can be done TODAY with a regular class that has a member of type synchronized class. Alternatively, you could use the mutex type in core directly. So your suggestion actually makes the language worse and adds no power. Your impression being that "synchronized" is too deadlock-prone, and you believe the solution is taking one step BACK and eliminate it in favor of the "private"-based idiom in conjunction with the "mutex" type which is deadlock-prone PLUS prone to a variety of other issues.
 So where's the mutex that would be used to synchronize objects that
 are not synchronizable?

In the wrapper class/struct/object which derives a synchronized class/struct from the original. My D foo is not strong enough to just come up with valid D code for the idiom on the fly, but essentially you wrap the original object in a new object using a template which adds the mutex member and the interface methods (lock, tryLock, and unlock) required. No, this doesn't work with "final" classes.. but it shouldn't, they're final after all. For them you need to add/manage the mutex manually - the price you pay for "final".

Is there anything here that can't be done today, and pronto?
 There are cases in which you want to do multiple operations under a
 single critical section, even though the API is otherwise
 well-designed. That may be for correctness, efficiency, or both. I
 don't see why we'd want to disallow that, it's a good idiom.

Who suggested disallowing this? No-one. There are 3 main use cases I see for this; 1. Several disparate objects locked by a single mutex - in which case the correct solution is a separate mutex/monitor object.

There's also "13.14.3 Forcing Identical Mutexes" in TDPL.
 2. A single object, locked for serveral method calls - in which case the
 method-passed-a-delegate idea (mentioned below/ described in a separate
 thread) works, unless..

Here the synchronized-based idiom works well if the cost of reacquiring an already-owned mutex is sufficiently low.
 3. The calls span several scopes, in which case good-old manual
 lock/unlock is required (synchronized blocks don't help here)

Right. In these rare cases core.sync.mutex would be recommendable.
 Looking forward for a fleshed out proposal. Make sure you motivate it
 properly.

Sorry, I have no spare time to spare. You're getting free ideas/thoughts from me, feel free to ignore them.

Thanks. Let me know if I understand correctly that your idea boils down to "I don't like synchronized, let's deprecate it and get back to core.sync.mutex and recommend the private thingamaroo." In that case, I disagree. I believe synchronized has good merits that are being ignored.
 On first look, the inversion of control using delegates delegates has
 similar liabilities as straight scoped locking.

True, it's basically the same as a synchronized block in that respect. What we actually want is a way to limit the calls made by the delegate to methods of the object itself. If it could not call a synchronized method on a 2nd object, you could not possibly deadlock. Except, that is to say, unless you held a separate lock beforehand - but, the important point here is that you would have to take both locks explicitly, rather than by an implicit synchronized method call, making the bug far more obvious to code inspection.

Right. This is very D-unlike. I don't see anything in the current language that can limit the things a lambda/delegate can do. Andrei
May 31 2012
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 11:35 AM, Steven Schveighoffer wrote:
 On Thu, 31 May 2012 14:29:27 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 7:01 AM, Regan Heath wrote:

 Sorry, I have no spare time to spare. You're getting free ideas/thoughts
 from me, feel free to ignore them.

Thanks. Let me know if I understand correctly that your idea boils down to "I don't like synchronized, let's deprecate it and get back to core.sync.mutex and recommend the private thingamaroo." In that case, I disagree. I believe synchronized has good merits that are being ignored.

No, this is definitely *not* what we are saying.

Then probably a more formal proposal is in order. Andrei
May 31 2012
prev sibling parent deadalnix <deadalnix gmail.com> writes:
Le 01/06/2012 14:55, Regan Heath a écrit :
 On Thu, 31 May 2012 19:29:27 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 On 5/31/12 7:01 AM, Regan Heath wrote:
 Sorry, I have no spare time to spare. You're getting free ideas/thoughts
 from me, feel free to ignore them.

Thanks. Let me know if I understand correctly that your idea boils down to "I don't like synchronized, let's deprecate it and get back to core.sync.mutex and recommend the private thingamaroo." In that case, I disagree. I believe synchronized has good merits that are being ignored.

To present this another way.. Your motivation for the construct: "synchronized(a, b, ...)" was to prevent deadlocks caused by: [thread1] synchronized(a) { synchronized(b) { } } [thread2] synchronized(b) { synchronized(a) { } } right? Well, this is the same problem expressed several other less-obvious (to code inspection) ways: 1. [thread1] synchronized(a) { b.foo(); // where b.foo is { synchronized(this) { ... } } } [thread2] synchronized(b) { a.foo(); // where a.foo is { synchronized(this) { ... } } } 2. [thread1] synchronized(a) { b.foo(); // where b.foo is synchronized void foo() { ... } } [thread2] synchronized(b) { a.foo(); // where a.foo is synchronized void foo() { ... } } #1 can be solved (in most/many cases) by doing 2 things, first by disallowing that idiom completely in favour of synchronized classes/class methods (which I think TDPL does?), and second by adding more control as described below in(#2) #2 can be solved (in most/many cases) by allowing greater control over who can participate in synchronized statements. If either 'a' or 'b' were not allowed to participate in a synchronized statement, then either thread1 or thread2 would be invalid code and a deadlock involving these 2 objects would be impossible(*). There will still exist some synchronized classes which want to participate in synchronized statements but I'm thinking/hoping this is rare and if the default for D is 'not allowed' then it becomes a conscious choice and we can supply the developer with a warning in the docs which describe how to do it, introduce the synchronized(a, b, ...) construct, etc. From another angle.. I'm guessing it's either impossible or very hard to detect the 2 cases presented above at compile time? Essentially the compiler would need to know which code could execute in separate threads, then determine lock ordering for all shared/lockable objects, then detect cases of both (lock a, b) and (lock b, a) in separate threads. Sounds tricky. R (*)using synchronized statements - one could still keep a reference to the other internally and call a synchronized member function from within a synchronized member function

I think it is unrealistic to prevent all deadlock, unless you can come up with a radically new approach. It is still possible to provide interface that prevent common traps.
Jun 01 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
To clarify:

On 5/30/12 12:25 PM, Alex Rønne Petersen wrote:
  The mutex may not be
 directly exposed in the sense that you can obtain a reference (though in
 reality in all compiler implementations, you can), but it is exposed in
 the sense that you can lock and unlock it, which is a mutation of state
 and program flow.

synchronized (object) { writeln("about to unlock the object"); XXX writeln("unlocked the object"); } Replace "XXX" with a construct that unlocks the object. Andrei
May 30 2012
next sibling parent deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 22:31, Steven Schveighoffer a écrit :
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 To clarify:

 On 5/30/12 12:25 PM, Alex Rønne Petersen wrote:
 The mutex may not be
 directly exposed in the sense that you can obtain a reference (though in
 reality in all compiler implementations, you can), but it is exposed in
 the sense that you can lock and unlock it, which is a mutation of state
 and program flow.

synchronized (object) { writeln("about to unlock the object"); XXX writeln("unlocked the object"); } Replace "XXX" with a construct that unlocks the object.

This is not what we are talking about. I guess the easiest way to say it is, the API used to "lock and then subsequently unlock" the mutex. That is, even this: Object o = new Object; synchronized(o) { } is exposing the mutex in such a way that you can interfere with the semantic meaning of the mutex, and cause preventable deadlocks. It would be preferrable for: synchronized(o) { } to be an error, unless typeof(o) allows it explicitly. I gave you a case where you can have a deadlock, even with simple fully synchronized classes. You might say, "yeah, but all mutexes can have deadlocks!". My contention is that if the default is you do *not* expose the mutex to external callers, there *cannot* be a deadlock. Here is the example again: synchronized class A { private int _foo; property int foo() { return _foo;} } synchronized class B { private A _a; property A a() { return _a;} void fun() {} } void thread(B b) { synchronized(b.a) { b.fun(); } } If synchronized(b.a) is valid, deadlock can occur. If it's invalid, deadlock *cannot* occur. -Steve

This cannot remove all deadlocks, but yes, it goes in the right direction.
May 30 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 1:31 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

This is not what we are talking about.

Oh yes it is. "Expose the mutex" means "make the two mutex primitive operations lock() and unlock() freely usable against the mutex object".
 I guess the easiest way to say it is, the API used to "lock and then
 subsequently unlock" the mutex. That is, even this:

 Object o = new Object;

 synchronized(o)
 {
 }

 is exposing the mutex in such a way that you can interfere with the
 semantic meaning of the mutex, and cause preventable deadlocks.

 It would be preferrable for:

 synchronized(o)
 {
 }

 to be an error, unless typeof(o) allows it explicitly.

Yah, it should be only allowed for synchronized classes.
 I gave you a case where you can have a deadlock, even with simple fully
 synchronized classes. You might say, "yeah, but all mutexes can have
 deadlocks!".

Not only I might say that, I'm actually gonna say it. Yeah, but all mutexes can have deadlocks!
 My contention is that if the default is you do *not* expose the mutex to
 external callers, there *cannot* be a deadlock.

That's shuffling the fundamental issue from client code to library code. This can be done in either approach (synchronized vs. raw mutex), except it's clunkier with raw mutex.
 Here is the example again:

 synchronized class A
 {
 private int _foo;
  property int foo() { return _foo;}
 }

 synchronized class B
 {
 private A _a;
  property A a() { return _a;}
 void fun() {}
 }

 void thread(B b)
 {
 synchronized(b.a)
 {
 b.fun();
 }
 }

 If synchronized(b.a) is valid, deadlock can occur. If it's invalid,
 deadlock *cannot* occur.

I don't get what this is supposed to prove. Andrei
May 30 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 23:45, Andrei Alexandrescu a écrit :
 On 5/30/12 1:31 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

This is not what we are talking about.

Oh yes it is. "Expose the mutex" means "make the two mutex primitive operations lock() and unlock() freely usable against the mutex object".

The more I think of it, the more I think it should go throw a synchrnizable property or something.
May 31 2012
prev sibling next sibling parent =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 30-05-2012 18:03, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

But in this design anyone can lock such an object, which was something you advocated against.

I think there is some confusion here as to what the "problem" is and is not. The problem is /not/ that you can lock any object. The problem is /not/ that we have synchronized(object) {} The problem is /not/ that we have synchronized classes/methods. The problem /is/ that synchronized classes/methods use a mutex which is exposed publicly, and it the same mutex as used by synchronized(object) {}. This exposure/re-use makes deadlocks more likely to happen, and harder to spot.

Thanks. I think the discussion was getting a bit derailed, myself being guilty of going off on tangential issues. Yes, the issue is exactly what you point out here: Exposing a locking resource publicly. The issue with a monitor on every object can probably be solved completely separately.
 Removal of this "problem" will not stop deadlocks from happening as
 they're a problem multi-threaded/lock based code will always have (*).
 It will however make them far less likely to happen accidentally. It
 will make programmers think about what/where/when and how to lock things
 rather than blithely using synchronized everywhere. It will mean using
 locks is not as easy as currently, though I think we can make it fairly
 nice with good library code and/or some co-operation between the runtime
 and synchronized() statements i.e. requiring the class implement a
 common Lockable interface for example.

One gripe I have with using interfaces is speed. David Simcha showed that virtual mutex calls actually *do* impact speed significantly, particularly in the GC in his case. As I pointed out earlier in this thread, core.sync.mutex.Mutex allows both composition and inheritance, and works with the synchronized statement. I still think it satisfies what everyone wants, given those two facts, and I have yet to see any counter-arguments to that (but would love to hear some to get a better understanding of why some people are against mutexes).
 The fact that every object can be locked, and this means they all use
 more memory is a side issue - arguably as important for some.

 R

 (*) the best you can do to "solve" the deadlock problem is to impose
 lock acquisition ordering on your code, as in all locks must be acquired
 in a defined order, and if not an assertion/error/exception is thrown.
 This requires some sort of static/singelton/overlord class which is
 aware of all mutexes and is involved in all acquisitions/releases.

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 17:46, Andrei Alexandrescu a écrit :
 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

But in this design anyone can lock such an object, which was something you advocated against. Andrei

I was advocating on being able to lock ANY object, which is out of control. Such an object is known to be lockable, and most object will not be. It will avoid liquid lock, because most thing will not be lockable. Combined with encapsulation capabilities that D already have, exposing the lock to the entire worlds is easy. It will avoid unexpected lock from 3rd party code, which is a source of tedious deadlock. It also open door for locking on struct, that is easily embeded in a class as value (so constructed and destructed with the class). The fact that anyone or not can lock/unlock a mutex is a encapsulation problem and we already have solution in D for that.
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 9:56 AM, deadalnix wrote:
 Le 30/05/2012 17:46, Andrei Alexandrescu a écrit :
 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

But in this design anyone can lock such an object, which was something you advocated against. Andrei

I was advocating on being able to lock ANY object, which is out of control.

Must've been another poster. Anyhow, in TDPL's design only synchronized classes can be used with the synchronized statement.
 Such an object is known to be lockable, and most object will not be. It
 will avoid liquid lock, because most thing will not be lockable.

I'm celebrating day 2 of having no idea what a liquid lock is.
 Combined with encapsulation capabilities that D already have, exposing
 the lock to the entire worlds is easy. It will avoid unexpected lock
 from 3rd party code, which is a source of tedious deadlock.

 It also open door for locking on struct, that is easily embeded in a
 class as value (so constructed and destructed with the class).

 The fact that anyone or not can lock/unlock a mutex is a encapsulation
 problem and we already have solution in D for that.

... which can be used with synchronized classes. Andrei
May 30 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 19:19, Andrei Alexandrescu wrote:
 On 5/30/12 9:56 AM, deadalnix wrote:
 Le 30/05/2012 17:46, Andrei Alexandrescu a écrit :
 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

But in this design anyone can lock such an object, which was something you advocated against. Andrei

I was advocating on being able to lock ANY object, which is out of control.

Must've been another poster. Anyhow, in TDPL's design only synchronized classes can be used with the synchronized statement.
 Such an object is known to be lockable, and most object will not be. It
 will avoid liquid lock, because most thing will not be lockable.

I'm celebrating day 2 of having no idea what a liquid lock is.

I have to confess to this as well. I'm starting to suspect that he really means deadlock. Searching on Google for "liquid lock mutex" literally brings up this very NG thread. :)
 Combined with encapsulation capabilities that D already have, exposing
 the lock to the entire worlds is easy. It will avoid unexpected lock
 from 3rd party code, which is a source of tedious deadlock.

 It also open door for locking on struct, that is easily embeded in a
 class as value (so constructed and destructed with the class).

 The fact that anyone or not can lock/unlock a mutex is a encapsulation
 problem and we already have solution in D for that.

.... which can be used with synchronized classes. Andrei

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
[snip]
 Such an object is known to be lockable, and most object will not be. It
 will avoid liquid lock, because most thing will not be lockable.

I'm celebrating day 2 of having no idea what a liquid lock is.

I have to confess to this as well. I'm starting to suspect that he really means deadlock. Searching on Google for "liquid lock mutex" literally brings up this very NG thread. :)

Maybe it's livelock? P.S. Just trying to snatched the prize in this little guess-game :) -- Dmitry Olshansky
May 30 2012
parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 20:03, cal a écrit :
 On Wednesday, 30 May 2012 at 17:46:11 UTC, Dmitry Olshansky wrote:
 [snip]
 Such an object is known to be lockable, and most object will not
 be. It will avoid liquid lock, because most thing will not be
 lockable.

I'm celebrating day 2 of having no idea what a liquid lock is.

I have to confess to this as well. I'm starting to suspect that he really means deadlock. Searching on Google for "liquid lock mutex" literally brings up this very NG thread. :)

P.S. Just trying to snatched the prize in this little guess-game :)

FWIW, I recently came across the term here: http://schneide.wordpress.com/tag/liquid-lock/

I explained that in another post a few minutes ago, and yes, this is it. Maybe i should write an article on that, I though it was more well known.
May 30 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 12:22 PM, deadalnix wrote:
 Le 30/05/2012 20:03, cal a écrit :
 On Wednesday, 30 May 2012 at 17:46:11 UTC, Dmitry Olshansky wrote:
 [snip]
 Such an object is known to be lockable, and most object will not
 be. It will avoid liquid lock, because most thing will not be
 lockable.

I'm celebrating day 2 of having no idea what a liquid lock is.

I have to confess to this as well. I'm starting to suspect that he really means deadlock. Searching on Google for "liquid lock mutex" literally brings up this very NG thread. :)

P.S. Just trying to snatched the prize in this little guess-game :)

FWIW, I recently came across the term here: http://schneide.wordpress.com/tag/liquid-lock/

I explained that in another post a few minutes ago, and yes, this is it. Maybe i should write an article on that, I though it was more well known.

As a side note, I'm highly weary of inventing new terminology. This is a general remark, not related to this particular instance. Most often when I personally found the need to invent a new term, it was because of not having done my homework of looking at related work. Andrei
May 30 2012
parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 21:45, Andrei Alexandrescu a écrit :
 On 5/30/12 12:22 PM, deadalnix wrote:
 Le 30/05/2012 20:03, cal a écrit :
 On Wednesday, 30 May 2012 at 17:46:11 UTC, Dmitry Olshansky wrote:
 [snip]
 Such an object is known to be lockable, and most object will not
 be. It will avoid liquid lock, because most thing will not be
 lockable.

I'm celebrating day 2 of having no idea what a liquid lock is.

I have to confess to this as well. I'm starting to suspect that he really means deadlock. Searching on Google for "liquid lock mutex" literally brings up this very NG thread. :)

P.S. Just trying to snatched the prize in this little guess-game :)

FWIW, I recently came across the term here: http://schneide.wordpress.com/tag/liquid-lock/

I explained that in another post a few minutes ago, and yes, this is it. Maybe i should write an article on that, I though it was more well known.

As a side note, I'm highly weary of inventing new terminology. This is a general remark, not related to this particular instance. Most often when I personally found the need to invent a new term, it was because of not having done my homework of looking at related work. Andrei

I didn't invented that term. But this is off topic. How would you call that phenomena ?
May 30 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 12:51 PM, deadalnix wrote:
 http://schneide.wordpress.com/tag/liquid-lock/

I explained that in another post a few minutes ago, and yes, this is it. Maybe i should write an article on that, I though it was more well known.

As a side note, I'm highly weary of inventing new terminology. This is a general remark, not related to this particular instance. Most often when I personally found the need to invent a new term, it was because of not having done my homework of looking at related work. Andrei

I didn't invented that term. But this is off topic.

I know. Apparently Daniel Lindner invented it in a blog post that made no inroads on the net in five months.
 How would you call
 that phenomena ?

"Researching related work". It's very common outside the academic circles, and very frowned upon inside them. Andrei
May 30 2012
prev sibling parent deadalnix <deadalnix gmail.com> writes:
Le 31/05/2012 07:18, Martin Nowak a écrit :
 FWIW, I recently came across the term here:
 http://schneide.wordpress.com/tag/liquid-lock/

I explained that in another post a few minutes ago, and yes, this is it. Maybe i should write an article on that, I though it was more well known.

 The main problem here is the object the lock is acquired upon: the
 reference of lastMessage is mutable! We call this a liquid lock,
 because the lock isn’t as solid as it should be. It’s one of the more
 hideous multithreading pitfalls as it looks like everything’s fine at
 first glance.

You should try to name the real root of the bug. It's trying to serialize access to a channel by locking the messages. There is nothing fancy about this, it has nothing to do with mutability it's just about locking the wrong resource.

When you lock on a non final resource, you ends up with trouble at some point. A have wasted enough time on that type of problem to know it for sure.
May 31 2012
prev sibling parent deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 19:19, Andrei Alexandrescu a écrit :
 Must've been another poster. Anyhow, in TDPL's design only synchronized
 classes can be used with the synchronized statement.

Which is good.
 Such an object is known to be lockable, and most object will not be. It
 will avoid liquid lock, because most thing will not be lockable.

I'm celebrating day 2 of having no idea what a liquid lock is.

I just explained that in another post.
 Combined with encapsulation capabilities that D already have, exposing
 the lock to the entire worlds is easy. It will avoid unexpected lock
 from 3rd party code, which is a source of tedious deadlock.

 It also open door for locking on struct, that is easily embeded in a
 class as value (so constructed and destructed with the class).

 The fact that anyone or not can lock/unlock a mutex is a encapsulation
 problem and we already have solution in D for that.

... which can be used with synchronized classes.

As said, I have nothing against synchronized classes.
May 30 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Thu, 31 May 2012 16:23:29 +0100, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 On 31.05.2012 19:11, Steven Schveighoffer wrote:
 On Thu, 31 May 2012 10:49:52 -0400, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your
 ideas quite favorably.

[snip]
 Does it makes sense?

Everything but the ordering. I like the idea of ordering the locks based on an intrinsic value, but opCmp isn't it. Objects can mutate, and opCmp may produce different results. Imagine: synchronized(a, b) // at this point a < b { a.makeGreaterThan(b); assert(a > b); }

No way! I live in world where victim's hand is cut off as a punishment for mutating a lock after creation ;)

This is related to the "liquid lock" problem raised/mentioned elsewhere in the thread. If the reference you lock can change, then one thread/iteration may lock one instance, the lock instance mutates, and a 2nd thread/iteration locks the new instance and both execute more or less in parallel over code which was supposed to be serialized. It's not a problem if the lock object is the object being mutated/used by the code, but it is a problem if the lock object is protecting other shared objects. It's not something I have ever bumped into myself, because the bulk of my locking experience is in C/C++ and I use a locking primitive which is initialised once and never mutates.
 You *never* want the ordering to change.

 But I think we can probably work that out. What about comparing handles
 of the mutexes? So you sort based on some property __mutex_id() which
 must return a unique size_t that can be used to always define an
 ordering of locking mutexes? Most mutexes are allocated by the OS, and
 so their handles won't be affected by a moving GC, so you likely will
 use the handle value.

This could work. In fact I imagined comparing handles ... As with math at large you may either project (biject) your domain to a well-known one (like numbers, mutex_id in your case) and work with it or define all relevant properties for whatever your values in this domain are (providing custom ordering for user defined types).

This ordering idea is present in TDPL.. have you both read the link Andrei posted? http://goo.gl/ZhPM2 (see 13.15) R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 31 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 31 May 2012 14:55:31 -0400, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 Dunno. In fact __lock/__unlock could be artificially hidden from user  
 but not compiler with his magic rewrites.
 But I believe there is value in having access to lock/unlock for those  
 select cases where it's beneficial.

__ctor and __dtor are accessible, but nobody abuses them. I think if we use the double-underscore terminology, this would be fine. -Steve
May 31 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 01 Jun 2012 08:38:45 -0400, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 On 01.06.2012 16:26, deadalnix wrote:
  Here is what I ended up to think is the best
 solution :

 synchronized classes exists. By default, they can't be use as parameter
 for synchronized(something) .

 synchronized(something) will be valid is something provide
 opSynchronized(scope delegate void()) or something similar. Think
 opApply here. The synchronized statement is rewritten in a call to that
 delegate.

 Here are the benefit of such an approach :
 1/ Lock and unlock are not exposed. You can only use them by pair.
 2/ You cannot lock on any object, so you avoid most liquid locks and
 don't waste memory.
 3/ synchronized classes ensure that a class can be shared and it
 internal are protected from concurrent access.
 4/ It is not possible possible by default to lock on synchronized
 classes's instances. It grant better control over the lock and it is now
 clear which piece of code is responsible of it.
 5/ The design allow the programmer to grant the permission to lock on
 synchronized classes's instances if he/she want to.
 6/ It is now possible to synchronize on a broader range of user defined
 stuffs.

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem
 have been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 And many types of lock : spin lock, interprocesses locks, semaphores, .
 . . And all can be used with the synchronized syntax, and without
 exposing locking and unlocking primitives.

 What do people think ?

+1. Works for me. It refines what I believe the shadow cabinet (loosely: me, you, Alex, Regan Heath and Steven) propose.

Is this really necessary? When is opSynchronized going to be written any way other than: _mutex.lock(); scope(exit) _mutex.unlock(); dg(); I'll note that it's easier to forget to lock or unlock if the compiler isn't enforcing it. You might even naively do this: _mutex.lock(); dg(); _mutex.unlock(); // not called on exception thrown! I kind of like the __lock() __unlock() pair that the compiler always calls both in the right place/way. Yes, you could just leave those implementations blank, but very unlikely. Plus, we already have issues with inout and delegates for opApply, this would have the same issues.
 P.S. Removing monitor from non-synced/shared classes would be good too.  
 As a separate matter.

I think at this point, we should leave it there until we can really figure out a detailed plan on how to deal with it. It currently affects all runtime code which does virtual function lookups or interface lookups, and alignment. We would have to change a lot of compiler and runtime code to remove it. I personally don't see it as a huge issue, we are already allocating on power-of-two boundaries which can almost double required space. I feel class size is really one of those things you should be oblivious to. That being said, I'm all for performance improvement, even small ones, if they are free and someone is willing to do all the leg work :) -Steve
Jun 01 2012
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 01 Jun 2012 10:09:01 -0400, deadalnix <deadalnix gmail.com> wrote:


 I think it is unrealistic to prevent all deadlock, unless you can come  
 up with a radically new approach.

 It is still possible to provide interface that prevent common traps.

Right, the idea is not to exterminate all deadlock, it's to *make it possible to* prevent deadlock. Currently, it's not possible unless you avoid using synchronized(this). -Steve
Jun 01 2012
prev sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Thu, 31 May 2012 19:35:50 +0100, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Thu, 31 May 2012 14:29:27 -0400, Andrei Alexandrescu  
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 7:01 AM, Regan Heath wrote:

 Sorry, I have no spare time to spare. You're getting free  
 ideas/thoughts
 from me, feel free to ignore them.

Thanks. Let me know if I understand correctly that your idea boils down to "I don't like synchronized, let's deprecate it and get back to core.sync.mutex and recommend the private thingamaroo." In that case, I disagree. I believe synchronized has good merits that are being ignored.

No, this is definitely *not* what we are saying. The idea is that synchronized(x) is still present, but what objects you can call this on, and more importantly, *who* can do this is restricted.

Exactly.
 Nobody is advocating abandoning synchronized in favor of manual locks.   
 In fact, I think we all want to *avoid* manual locks as much as  
 possible.  It's all about controlling access.  If it comes down to "you  
 must use a private, error-prone mutex member in order to prevent  
 deadlocks," then I think we have room for improvement.

Indeed. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Jun 01 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I missed that message. I don't think synchronized classes should go away. I think synchronize on non synchronized classes should go away.
May 30 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 21:17, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I missed that message. I don't think synchronized classes should go away. I think synchronize on non synchronized classes should go away.

This is something I can agree on. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 12:26 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 21:17, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I missed that message. I don't think synchronized classes should go away. I think synchronize on non synchronized classes should go away.

This is something I can agree on.

Then I'm unclear what we disagree about. Andrei
May 30 2012
parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 21:49, Andrei Alexandrescu wrote:
 On 5/30/12 12:26 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 21:17, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I missed that message. I don't think synchronized classes should go away. I think synchronize on non synchronized classes should go away.

This is something I can agree on.

Then I'm unclear what we disagree about. Andrei

OK, so we agree that synchronizing on non-synchronized classes is a Bad Thing (TM) and should be disallowed. We also agree that (while it is a somewhat tangential issue) the extra word used in every object is wasteful (right?). Now, it's clear that a synchronized class has to store its mutex somewhere. We could have synchronized classes add an implicit, hidden field that holds the mutex. Personally, I feel that this might add a little too much magic (and also forces using the GC), but maybe it's just me. What I pointed out earlier in this thread is that core.sync.mutex.Mutex can be used both in a compositional style and through inheritance. For example: class MySynchronizedObject : Mutex { } auto mso = new MySynchronizedObject; synchronized (mso) // this calls lock and unlock on the mutex that mso *is* since it inherits Mutex directly, i.e. no implicit monitor is created. { // ... } So basically, I think the question is this: Should synchronized classes implicitly have core.sync.mutex.Mutex as their base class instead of object.Object, or should they contain a hidden field? Both options involve magic, which might be undesirable. The inheritance solution obviously places restrictions on what base type can be used for synchronized classes. The hidden field solution means forcing use of the GC. Am I making sense? Thoughts on this? -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 12:17 PM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I missed that message. I don't think synchronized classes should go away. I think synchronize on non synchronized classes should go away.

I agree. Andrei
May 30 2012
parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 21:42, Andrei Alexandrescu a écrit :
 On 5/30/12 12:17 PM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I missed that message. I don't think synchronized classes should go away. I think synchronize on non synchronized classes should go away.

I agree. Andrei

Ha, this is getting somewhere :D
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 12:52 PM, deadalnix wrote:
 Le 30/05/2012 21:42, Andrei Alexandrescu a écrit :
 On 5/30/12 12:17 PM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I missed that message. I don't think synchronized classes should go away. I think synchronize on non synchronized classes should go away.

I agree. Andrei

Ha, this is getting somewhere :D

It's getting where I was the whole time. Andrei
May 30 2012
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 21:58, Andrei Alexandrescu a écrit :
 It's getting where I was the whole time.

 Andrei

So we have a communication problem.
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 1:13 PM, deadalnix wrote:
 Le 30/05/2012 21:58, Andrei Alexandrescu a écrit :
 It's getting where I was the whole time.

 Andrei

So we have a communication problem.

Not me! :o) Andrei
May 30 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 23:41, Andrei Alexandrescu a écrit :
 On 5/30/12 1:13 PM, deadalnix wrote:
 Le 30/05/2012 21:58, Andrei Alexandrescu a écrit :
 It's getting where I was the whole time.

 Andrei

So we have a communication problem.

Not me! :o) Andrei

Happy to know that you understand yourself <:o) . Now let's simply work on a mind to mind broadcasting system !
May 30 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-05-30 21:58, Andrei Alexandrescu wrote:
 On 5/30/12 12:52 PM, deadalnix wrote:
 Ha, this is getting somewhere :D

It's getting where I was the whole time. Andrei

No, the original problem was that you can synchronize on ALL objects, regardless if they're marked with "synchronized" or not. This it how it works today. -- /Jacob Carlborg
May 31 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/12 12:28 AM, Jacob Carlborg wrote:
 On 2012-05-30 21:58, Andrei Alexandrescu wrote:
 On 5/30/12 12:52 PM, deadalnix wrote:
 Ha, this is getting somewhere :D

It's getting where I was the whole time. Andrei

No, the original problem was that you can synchronize on ALL objects, regardless if they're marked with "synchronized" or not. This it how it works today.

I was the whole time where TDPL is, which is not where the implementation is today. Andrei
May 31 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 31/05/2012 09:49, Andrei Alexandrescu a écrit :
 On 5/31/12 12:28 AM, Jacob Carlborg wrote:
 On 2012-05-30 21:58, Andrei Alexandrescu wrote:
 On 5/30/12 12:52 PM, deadalnix wrote:
 Ha, this is getting somewhere :D

It's getting where I was the whole time. Andrei

No, the original problem was that you can synchronize on ALL objects, regardless if they're marked with "synchronized" or not. This it how it works today.

I was the whole time where TDPL is, which is not where the implementation is today. Andrei

And which isn't what dlang.org say too. You can agree that this is confusing for everybody.
May 31 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Thu, 31 May 2012 15:06:14 +0100, deadalnix <deadalnix gmail.com> wrot=
e:

 Le 31/05/2012 16:01, Regan Heath a =E9crit :
 True, it's basically the same as a synchronized block in that respect=


 What we actually want is a way to limit the calls made by the delegat=


 to methods of the object itself. If it could not call a synchronized
 method on a 2nd object, you could not possibly deadlock. Except, that=


 to say, unless you held a separate lock beforehand - but, the importa=


 point here is that you would have to take both locks explicitly, rath=


 than by an implicit synchronized method call, making the bug far more=


 obvious to code inspection.

 R

I'm not sure I follow you here. Can you elaborate on that ?

Apologies in advance, my D code skillz have rusted significantly.. class A { synchronized void foo() { } } class B {} void main() { A a =3D new A(); B b =3D new B(); synchronized(b) // locks B { a.foo(); // locks A } } .. this deadlocks if another thread locks A then B anywhere. The = "problem" is that this can be hard to spot and easy to do accidentally a= s = a.foo() does not scream "I'm going to lock something". This is the same problem with the synchronized method calling a supplied= = delegate.. class A { synchronized void foo() { } } class B { synchronized void syncDelegate(void delegate(void) dg) { .. } } void main() { A a =3D new A(); B b =3D new B(); b.syncDelegate(.. code which calls a.foo() .. ); // locks B = (syncDelegate), then locks A (a.foo) } The only way to avoid the deadlock is to prevent calls to synchronized = methods inside the delegate. R -- = Using Opera's revolutionary email client: http://www.opera.com/mail/
May 31 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Thu, 31 May 2012 15:49:52 +0100, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your  
 ideas quite favorably.

 I'll enumerate certain assumptions beforehand since it's one of most  
 confusing threads I ever followed.

 1. synchronized class means: always allocate hidden monitor mutex, lock  
 it on every public method of this class.

 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 	auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z  
 sorted
 	FOR EACH item IN sorted tuple add code in [[ ... ]]
 	[[// conceptual
 		item.__lock();
 		scope(exit) item.__unlock();
 	]]
 In other words it works for every object that defines lock/unlock.  
 Multiple object version works only if there is opCmp defined. (by  
 address whatever, any total ordering should work)


 Per definition above synchronized classes AS IS can't be *synchronized*  
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug  
 in custom synchronization primitives (say OS Y provides have this fancy  
 lock type Z). It's explicit in as sense that object supports it via  
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly  
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from  
 designer of the class may (a)use it.

 Does it makes sense?

Yes, and it's all more intentional/flexible than what we have now, but.. :) Does it address what I thought was the main "problem" case. That is, as soon as you lock 2 objects, one via a synchronized() statement and the other via a synchronized method you can get a non-obvious deadlock. e.g. synchronized class A { void foo() {} // implicitly locks instance of A } class B { void __lock() {} // locks instance of B void __unlock() {} // unlocks instance of B } shared A a; shared B b; void main() { a = new shared(A)(); b = new shared(B)(); ..start thread which locks a then b.. synchronized(b) // locks b 'explicitly' { a.foo(); // locks a 'implicitly' } } .. but, hang on, can a thread actually lock a and then b? If 'a' cannot participate in a synchronized statement (which it can't under this proposal) then no, there is no way to lock 'a' except by calling a member. So, provided 'a' does not have a member which locks 'b' - were deadlock safe! So.. problem solved; by preventing external/public lock/unlock on a synchronized class. (I think the proposal should enforce this restriction; synchronized classes cannot define __lock/__unlock). R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 31 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 31 May 2012 10:49:52 -0400, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your  
 ideas quite favorably.

[snip]
 Does it makes sense?

Everything but the ordering. I like the idea of ordering the locks based on an intrinsic value, but opCmp isn't it. Objects can mutate, and opCmp may produce different results. Imagine: synchronized(a, b) // at this point a < b { a.makeGreaterThan(b); assert(a > b); } You *never* want the ordering to change. But I think we can probably work that out. What about comparing handles of the mutexes? So you sort based on some property __mutex_id() which must return a unique size_t that can be used to always define an ordering of locking mutexes? Most mutexes are allocated by the OS, and so their handles won't be affected by a moving GC, so you likely will use the handle value. -Steve
May 31 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 31 May 2012 14:29:27 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 7:01 AM, Regan Heath wrote:

 Sorry, I have no spare time to spare. You're getting free ideas/thoughts
 from me, feel free to ignore them.

Thanks. Let me know if I understand correctly that your idea boils down to "I don't like synchronized, let's deprecate it and get back to core.sync.mutex and recommend the private thingamaroo." In that case, I disagree. I believe synchronized has good merits that are being ignored.

No, this is definitely *not* what we are saying. The idea is that synchronized(x) is still present, but what objects you can call this on, and more importantly, *who* can do this is restricted. Nobody is advocating abandoning synchronized in favor of manual locks. In fact, I think we all want to *avoid* manual locks as much as possible. It's all about controlling access. If it comes down to "you must use a private, error-prone mutex member in order to prevent deadlocks," then I think we have room for improvement. -Steve
May 31 2012
prev sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Thu, 31 May 2012 19:29:27 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:
 On 5/31/12 7:01 AM, Regan Heath wrote:
 Sorry, I have no spare time to spare. You're getting free ideas/thoughts
 from me, feel free to ignore them.

Thanks. Let me know if I understand correctly that your idea boils down to "I don't like synchronized, let's deprecate it and get back to core.sync.mutex and recommend the private thingamaroo." In that case, I disagree. I believe synchronized has good merits that are being ignored.

To present this another way.. Your motivation for the construct: "synchronized(a, b, ...)" was to prevent deadlocks caused by: [thread1] synchronized(a) { synchronized(b) { } } [thread2] synchronized(b) { synchronized(a) { } } right? Well, this is the same problem expressed several other less-obvious (to code inspection) ways: 1. [thread1] synchronized(a) { b.foo(); // where b.foo is { synchronized(this) { ... } } } [thread2] synchronized(b) { a.foo(); // where a.foo is { synchronized(this) { ... } } } 2. [thread1] synchronized(a) { b.foo(); // where b.foo is synchronized void foo() { ... } } [thread2] synchronized(b) { a.foo(); // where a.foo is synchronized void foo() { ... } } #1 can be solved (in most/many cases) by doing 2 things, first by disallowing that idiom completely in favour of synchronized classes/class methods (which I think TDPL does?), and second by adding more control as described below in(#2) #2 can be solved (in most/many cases) by allowing greater control over who can participate in synchronized statements. If either 'a' or 'b' were not allowed to participate in a synchronized statement, then either thread1 or thread2 would be invalid code and a deadlock involving these 2 objects would be impossible(*). There will still exist some synchronized classes which want to participate in synchronized statements but I'm thinking/hoping this is rare and if the default for D is 'not allowed' then it becomes a conscious choice and we can supply the developer with a warning in the docs which describe how to do it, introduce the synchronized(a, b, ...) construct, etc. From another angle.. I'm guessing it's either impossible or very hard to detect the 2 cases presented above at compile time? Essentially the compiler would need to know which code could execute in separate threads, then determine lock ordering for all shared/lockable objects, then detect cases of both (lock a, b) and (lock b, a) in separate threads. Sounds tricky. R (*)using synchronized statements - one could still keep a reference to the other internally and call a synchronized member function from within a synchronized member function -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Jun 01 2012
prev sibling next sibling parent Andrew Wiley <wiley.andrew.j gmail.com> writes:
--f46d042f9c62ada38904c19a85a5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Sun, Jun 3, 2012 at 5:13 PM, deadalnix <deadalnix gmail.com> wrote:

 Le 04/06/2012 02:03, Andrew Wiley a =E9crit :

 On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com

 <mailto:deadalnix gmail.com>> wrote:

    Le 03/06/2012 21:40, Andrew Wiley a =E9crit :

        On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
        <mailto:deadalnix gmail.com>
        <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>> wrote:

            Le 01/06/2012 22:55, Sean Kelly a =E9crit :

                On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


                    The main drawback is the same as opApply : return (an=


                    break/continue but it is less relevant for
        opSynchronized).
                    Solution to this problem have been proposed in the pa=


                    using compiler and stack magic.

                    It open door for stuff like :
                    ReadWriteLock rw;
                    synchronized(rw.read) {

                    }

                    synchronized(rw.write) {

                    }


                Opens the door?  This works today exactly as outlined
        above.  Or
                am I missing a part of your argument?

                    And many types of lock : spin lock, interprocesses
        locks,
                    semaphores, . . . And all can be used with the
        synchronized
                    syntax, and without exposing locking and unlocking
        primitives.


                All works today.


            Unless you do some monitor magic, it doesn't.

        Yes, it does.
        -----
        class Something {
             private:
                 ReadWriteLock _rw;
             public:
                 this() {
                     _rw =3D new ReadWriteLock();
                 }
                 void doSomething() shared {
                     synchronized(_rw.read) {
                         // do things
                     }
                 }
        }
        -----
        I've used this pattern in code. There might be some casting
 required
        because the core synchronization primitives haven't been updated
        to use
        shared yet.


    And where is that ReadWriteLock ?

 On the GC heap, just like the Monitor object pointed to by __monitor if
 you mark a method or class as synchronized.

I meant where is the code ?

My apologies, it's actually ReadWriteMutex: http://dlang.org/phobos/core_sync_rwmutex.html --f46d042f9c62ada38904c19a85a5 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Sun, Jun 3, 2012 at 5:13 PM, deadalnix <span = dir=3D"ltr">&lt;<a href=3D"mailto:deadalnix gmail.com" target=3D"_blank">de= adalnix gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote= " style=3D"margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(= 204,204,204);border-left-width:1px;border-left-style:solid"> Le 04/06/2012 02:03, Andrew Wiley a =E9crit :<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;padding= -left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-l= eft-style:solid"> On Sun, Jun 3, 2012 at 4:39 PM, deadalnix &lt;<a href=3D"mailto:deadalnix g= mail.com" target=3D"_blank">deadalnix gmail.com</a><div class=3D"im"><br> &lt;mailto:<a href=3D"mailto:deadalnix gmail.com" target=3D"_blank">deadaln= ix gmail.com</a>&gt;&gt; wrote:<br> <br> =A0 =A0Le 03/06/2012 21:40, Andrew Wiley a =E9crit :<br> <br> =A0 =A0 =A0 =A0On Sun, Jun 3, 2012 at 12:29 PM, deadalnix &lt;<a href=3D"m= ailto:deadalnix gmail.com" target=3D"_blank">deadalnix gmail.com</a><br> =A0 =A0 =A0 =A0&lt;mailto:<a href=3D"mailto:deadalnix gmail.com" target=3D= "_blank">deadalnix gmail.com</a>&gt;<br></div><div><div class=3D"h5"> =A0 =A0 =A0 =A0&lt;mailto:<a href=3D"mailto:deadalnix gmail.com" target=3D= "_blank">deadalnix gmail.com</a> &lt;mailto:<a href=3D"mailto:deadalnix gma= il.com" target=3D"_blank">deadalnix gmail.com</a>&gt;&gt;&gt; wrote:<br> <br> =A0 =A0 =A0 =A0 =A0 =A0Le 01/06/2012 22:55, Sean Kelly a =E9crit :<br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0On Jun 1, 2012, at 5:26 AM, deadalnix wrote= :<br> <br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0The main drawback is the same as op= Apply : return (and<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0break/continue but it is less relev= ant for<br> =A0 =A0 =A0 =A0opSynchronized).<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Solution to this problem have been = proposed in the past<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0using compiler and stack magic.<br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0It open door for stuff like :<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ReadWriteLock rw;<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0synchronized(rw.read) {<br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}<br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0synchronized(rw.write) {<br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}<br> <br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Opens the door? =A0This works today exactly= as outlined<br> =A0 =A0 =A0 =A0above. =A0Or<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0am I missing a part of your argument?<br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0And many types of lock : spin lock,= interprocesses<br> =A0 =A0 =A0 =A0locks,<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0semaphores, . . . And all can be us= ed with the<br> =A0 =A0 =A0 =A0synchronized<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0syntax, and without exposing lockin= g and unlocking<br> =A0 =A0 =A0 =A0primitives.<br> <br> <br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0All works today.<br> <br> <br> =A0 =A0 =A0 =A0 =A0 =A0Unless you do some monitor magic, it doesn&#39;t.<b= r> <br> =A0 =A0 =A0 =A0Yes, it does.<br> =A0 =A0 =A0 =A0-----<br> =A0 =A0 =A0 =A0class Something {<br> =A0 =A0 =A0 =A0 =A0 =A0 private:<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ReadWriteLock _rw;<br> =A0 =A0 =A0 =A0 =A0 =A0 public:<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 this() {<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 _rw =3D new ReadWriteLock();<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 void doSomething() shared {<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 synchronized(_rw.read) {<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 // do things<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }<br> =A0 =A0 =A0 =A0}<br> =A0 =A0 =A0 =A0-----<br> =A0 =A0 =A0 =A0I&#39;ve used this pattern in code. There might be some cas= ting required<br> =A0 =A0 =A0 =A0because the core synchronization primitives haven&#39;t bee= n updated<br> =A0 =A0 =A0 =A0to use<br> =A0 =A0 =A0 =A0shared yet.<br> <br> <br> =A0 =A0And where is that ReadWriteLock ?<br> <br></div></div><div class=3D"im"> On the GC heap, just like the Monitor object pointed to by __monitor if<br> you mark a method or class as synchronized.<br> </div></blockquote> <br> I meant where is the code ?<br> </blockquote></div><div>=A0</div><div>My apologies, it&#39;s actually ReadW= riteMutex: <a href=3D"http://dlang.org/phobos/core_sync_rwmutex.html">http:= //dlang.org/phobos/core_sync_rwmutex.html</a><br></div> --f46d042f9c62ada38904c19a85a5--
Jun 03 2012
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jun 3, 2012, at 5:24 PM, Alex R=F8nne Petersen wrote:
=20
 The monitor in obj.__monitor is actually allocated on the native C =

gone. The benefit of it being on the native C heap is that you can use = synchronized{} in the object's dtor. You can't do this if the monitor = is on the GC heap, as Mutex is.=
Jun 06 2012
prev sibling next sibling parent "Jess" <Jess.Sandgren hotmail.com> writes:
On Monday, 28 May 2012 at 23:55:04 UTC, Alex Rønne Petersen 
wrote:
 On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet 
 we adopted
 it anyway... blergh.

The "lesson learned" from Java that TDPL enumerates is the mistake of having synchronized on functions rather than entire classes, but clearly even that is currently TDPL-only and not actually properly implemented yet. - Jonathan M Davis

But synchronized on entire classes is exactly equally flawed... the fundamental problem is still that they synchronize on a public resource.

Sorry, I don't quite follow... 1) locking on a public resource => BAD, high risk. 2) Adding GC.callLocked(), allowing a generic delegate to claim the GC lock https://github.com/D-Programming-Language/druntime/pull/213 => no-brainer, i.e. no risk?
May 29 2012
prev sibling next sibling parent "Jess" <Jess.Sandgren hotmail.com> writes:
On Tuesday, 29 May 2012 at 17:01:01 UTC, Alex Rønne Petersen 
wrote:

druntime is the low-level runtime library of D. This discussion is about high-level class design.

I should also add that the reason I started this thread was to prevent common locking errors induced by the anti-patterns synchronized causes. If you're calling *anything* in the core.memory module, I sure hope you know what you're doing in general. I really don't think these two issues are connected at all.

ok, maybe I read too much into your statement below? it sounded as if you wanted to ban this usage from druntime? Alex Rønne Petersen wrote: "I've seen several occurrences of synchronized (this) and synchronized (this.classinfo) in both druntime and phobos by now. I propose that we officially ban these patterns with extreme prejudice."
May 29 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Thu, 31 May 2012 12:06:45 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 3:27 AM, Regan Heath wrote:
 On Thu, 31 May 2012 10:49:27 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:45 AM, Regan Heath wrote:
 Ok, how about you tell us what term you want to use for the current
 state of affairs then, because arguing about this is pointless and I'm
 happy to use whatever term you deem appropriate (because I couldn't  
 care
 less what the term is - as long as we all know what is meant, which I
 believe is the case here).

The idiom is called "scoped locking".

So.. the mutex is "scoped locking"?

No, the Java-style approach with the synchronized statement etc.

*sigh* I give up, this is pointless. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 31 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 May 2012 17:42:47 -0400, Jos=C3=A9 Armando Garc=C3=ADa Sancio=
  =

<jsancio gmail.com> wrote:

 On Wed, May 30, 2012 at 5:30 PM, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:
 All deadlocks involving A and B mutexes.  Or if not, can you show how=


  I
 can't see it.

 -Steve

This can cause a deadlock because locking order is not guaranteed, no?=

 synchronized class A {
   void delegate(B b) { b.call() }
   void call() {}
 }

 synchronized class B {
   void delegate(A a) { a.call() }
   void call() {}
 }

 Thanks,
 -Jose
 PS. This may not compile not sure if it should be 'void
 delegate(shared B b)', etc.

A and B are not modifiable. Given the implementation of A and B that I = = gave, how can you achieve deadlock? The big problem I see with allowing anyone to synchronize on A and B is = = that one cannot achieve the property of "not possible to use me in a = deadlock." -Steve
May 31 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Thu, 31 May 2012 12:13:00 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:
 On 5/31/12 3:27 AM, Regan Heath wrote:
 I think
 the mutex is "available for locking and unlocking" <- need a word for
 that, which is not "exposed". How about accessible, available, usable,
 or just plain lockable .. So, the problem here is that the mutex is
 lockable by external code via synchronized() and this means a certain
 type of deadlock is possible.

But this is a protection/visibility issue, which is orthogonal on the locking capability. It's as if you say "int is not good because anyone can overflow it." Okay! Make it private inside a CheckedInt class.

Sorry, that's a bad comparison. CheckedInt is to int, what CheckedMutex is to mutex - but I'm not suggesting anything like a CheckedMutex. I'm suggesting "mutex" but kept private inside the class /that it locks/. Yes, it's a visibility issue, the issue is that the mutex used by synchronized classes/methods is too visible/accessible and this opens it up for deadlocks which are otherwise impossible.
 I think the change mentioned in TDPL to restrict synchronized to
 synchronized classes is a step in the right direction WRT wasted monitor
 space and people freely locking anything. But, it is exactly the case
 which results in more possible deadlocks (the cause of this thread) AND
 I think it's actually far more likely people will want to use a
 synchronized statement on a class which is not itself synchronized, like
 for example an existing container class.

 Given that, restricting synchronized statements to synchronized classes
 seems entirely wrong to me.

So where's the mutex that would be used to synchronize objects that are not synchronizable?

In the wrapper class/struct/object which derives a synchronized class/struct from the original. My D foo is not strong enough to just come up with valid D code for the idiom on the fly, but essentially you wrap the original object in a new object using a template which adds the mutex member and the interface methods (lock, tryLock, and unlock) required. No, this doesn't work with "final" classes.. but it shouldn't, they're final after all. For them you need to add/manage the mutex manually - the price you pay for "final".
 In fact, I would say you almost want to stop
 people using synchronized statements on synchronized classes because:
 1. If a synchronized class is written correctly it should not be
 necessary in the general case(*)
 2. It raises the chances of deadlocks (the cause of this thread).
 3. It means that classes in general will be simpler to write (no need to
 worry about synchronization) and also more lightweight for use in
 non-threaded/non-shared cases. (because we'd provide a template wrapper
 to make them synchronizable)

There are cases in which you want to do multiple operations under a single critical section, even though the API is otherwise well-designed. That may be for correctness, efficiency, or both. I don't see why we'd want to disallow that, it's a good idiom.

Who suggested disallowing this? No-one. There are 3 main use cases I see for this; 1. Several disparate objects locked by a single mutex - in which case the correct solution is a separate mutex/monitor object. 2. A single object, locked for serveral method calls - in which case the method-passed-a-delegate idea (mentioned below/ described in a separate thread) works, unless.. 3. The calls span several scopes, in which case good-old manual lock/unlock is required (synchronized blocks don't help here)
 So, more and more I'm thinking it would be better to provide a
 library/runtime co-operative solution where we have an interface which
 is required for synchronized statements, and a wrapper template to
 implement that interface for any existing non-synchronized class.
 Meaning, the choice is either to write a synchronized class (rare) or a
 non-synchronized class - knowing it can easily be synchronized if/when
 needed.

Looking forward for a fleshed out proposal. Make sure you motivate it properly.

Sorry, I have no spare time to spare. You're getting free ideas/thoughts from me, feel free to ignore them.
 The "liquid lock" problem mentioned earlier is an interesting one that I
 have not personally experienced, perhaps because I don't lock anything
 but mutex primitives and I never to re-assign these.

 (*) Locking on a larger scope can be achieved by providing a method
 taking a delegate (see earlier thread/replies) and/or exposing
 lock/unlock for those few situations where the delegate method cannot be
 used. These are the few cases in which this cannot be avoided as the
 lock/unlock are separated by more than a single scope (so synchronized
 statements don't help in these cases either).

On first look, the inversion of control using delegates delegates has similar liabilities as straight scoped locking.

True, it's basically the same as a synchronized block in that respect. What we actually want is a way to limit the calls made by the delegate to methods of the object itself. If it could not call a synchronized method on a 2nd object, you could not possibly deadlock. Except, that is to say, unless you held a separate lock beforehand - but, the important point here is that you would have to take both locks explicitly, rather than by an implicit synchronized method call, making the bug far more obvious to code inspection. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 31 2012
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On May 31, 2012, at 11:33 AM, Andrei Alexandrescu wrote:
=20
 It does make sense, but I think we need a Lock struct type that makes =

shouldn't just expose bare lock() and unlock(). synchronized already works with classes that implement Object.Monitor, = and we have scope guards. Do we really need an RTTI Lock struct as = well?=
May 31 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 May 2012 16:56:51 -0400, deadalnix <deadalnix gmail.com> wrot=
e:

 Le 30/05/2012 22:31, Steven Schveighoffer a =C3=A9crit :
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 To clarify:

 On 5/30/12 12:25 PM, Alex R=C3=B8nne Petersen wrote:
 The mutex may not be
 directly exposed in the sense that you can obtain a reference (thou=




 in
 reality in all compiler implementations, you can), but it is expose=




 in
 the sense that you can lock and unlock it, which is a mutation of  =




 state
 and program flow.

synchronized (object) { writeln("about to unlock the object"); XXX writeln("unlocked the object"); } Replace "XXX" with a construct that unlocks the object.

This is not what we are talking about. I guess the easiest way to say it is, the API used to "lock and then subsequently unlock" the mutex. That is, even this: Object o =3D new Object; synchronized(o) { } is exposing the mutex in such a way that you can interfere with the semantic meaning of the mutex, and cause preventable deadlocks. It would be preferrable for: synchronized(o) { } to be an error, unless typeof(o) allows it explicitly. I gave you a case where you can have a deadlock, even with simple ful=


 synchronized classes. You might say, "yeah, but all mutexes can have
 deadlocks!".

 My contention is that if the default is you do *not* expose the mutex=


 external callers, there *cannot* be a deadlock.

 Here is the example again:

 synchronized class A
 {
 private int _foo;
  property int foo() { return _foo;}
 }

 synchronized class B
 {
 private A _a;
  property A a() { return _a;}
 void fun() {}
 }

 void thread(B b)
 {
 synchronized(b.a)
 {
 b.fun();
 }
 }

 If synchronized(b.a) is valid, deadlock can occur. If it's invalid,
 deadlock *cannot* occur.

 -Steve

This cannot remove all deadlocks, but yes, it goes in the right =

 direction.

All deadlocks involving A and B mutexes. Or if not, can you show how? = I = can't see it. -Steve
May 30 2012
prev sibling next sibling parent =?ISO-8859-1?Q?Jos=E9_Armando_Garc=EDa_Sancio?= <jsancio gmail.com> writes:
On Wed, May 30, 2012 at 5:30 PM, Steven Schveighoffer
<schveiguy yahoo.com> wrote:
 All deadlocks involving A and B mutexes. =A0Or if not, can you show how? =

 can't see it.

 -Steve

This can cause a deadlock because locking order is not guaranteed, no? synchronized class A { void delegate(B b) { b.call() } void call() {} } synchronized class B { void delegate(A a) { a.call() } void call() {} } Thanks, -Jose PS. This may not compile not sure if it should be 'void delegate(shared B b)', etc.
May 30 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Thu, 31 May 2012 10:48:51 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

I'm not suggesting using free mutexes. I'm suggesting keeping the mutex private inside the object.

Ergo, you are suggesting using free mutexes. Your second sentence destroys the first.

Depends on your definition of "free". You appear to have meant as an instance/pointer/object even one in a class, I initially read it as meaning as a separate object from the class you're locking. In any case, you're right the compiler doesn't get synchronized() statements/classes/methods wrong and a programmer can. The trade-off is the cause of this thread of discussion. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 31 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Thu, 31 May 2012 10:49:27 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:45 AM, Regan Heath wrote:
 Ok, how about you tell us what term you want to use for the current
 state of affairs then, because arguing about this is pointless and I'm
 happy to use whatever term you deem appropriate (because I couldn't care
 less what the term is - as long as we all know what is meant, which I
 believe is the case here).

The idiom is called "scoped locking".

So.. the mutex is "scoped locking"? Doesn't sound right to me. I think the mutex is "available for locking and unlocking" <- need a word for that, which is not "exposed". How about accessible, available, usable, or just plain lockable .. So, the problem here is that the mutex is lockable by external code via synchronized() and this means a certain type of deadlock is possible. Moving on.. I think the change mentioned in TDPL to restrict synchronized to synchronized classes is a step in the right direction WRT wasted monitor space and people freely locking anything. But, it is exactly the case which results in more possible deadlocks (the cause of this thread) AND I think it's actually far more likely people will want to use a synchronized statement on a class which is not itself synchronized, like for example an existing container class. Given that, restricting synchronized statements to synchronized classes seems entirely wrong to me. In fact, I would say you almost want to stop people using synchronized statements on synchronized classes because: 1. If a synchronized class is written correctly it should not be necessary in the general case(*) 2. It raises the chances of deadlocks (the cause of this thread). 3. It means that classes in general will be simpler to write (no need to worry about synchronization) and also more lightweight for use in non-threaded/non-shared cases. (because we'd provide a template wrapper to make them synchronizable) So, more and more I'm thinking it would be better to provide a library/runtime co-operative solution where we have an interface which is required for synchronized statements, and a wrapper template to implement that interface for any existing non-synchronized class. Meaning, the choice is either to write a synchronized class (rare) or a non-synchronized class - knowing it can easily be synchronized if/when needed. The "liquid lock" problem mentioned earlier is an interesting one that I have not personally experienced, perhaps because I don't lock anything but mutex primitives and I never to re-assign these. (*) Locking on a larger scope can be achieved by providing a method taking a delegate (see earlier thread/replies) and/or exposing lock/unlock for those few situations where the delegate method cannot be used. These are the few cases in which this cannot be avoided as the lock/unlock are separated by more than a single scope (so synchronized statements don't help in these cases either). R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 31 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 May 2012 17:45:32 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 1:31 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

This is not what we are talking about.

Oh yes it is. "Expose the mutex" means "make the two mutex primitive operations lock() and unlock() freely usable against the mutex object".

Stop that! You are blatantly arguing minutia when you should be trying to understand what we are saying! This is like dealing with a stubborn child.
 I guess the easiest way to say it is, the API used to "lock and then
 subsequently unlock" the mutex. That is, even this:

 Object o = new Object;

 synchronized(o)
 {
 }

 is exposing the mutex in such a way that you can interfere with the
 semantic meaning of the mutex, and cause preventable deadlocks.

 It would be preferrable for:

 synchronized(o)
 {
 }

 to be an error, unless typeof(o) allows it explicitly.

Yah, it should be only allowed for synchronized classes.

Not good enough. Whether I want all accesses to methods and members to be synchronized should be orthogonal to whether I want to allow arbitrary scoped locking and unlocking of my object.
 I gave you a case where you can have a deadlock, even with simple fully
 synchronized classes. You might say, "yeah, but all mutexes can have
 deadlocks!".

Not only I might say that, I'm actually gonna say it. Yeah, but all mutexes can have deadlocks!

Right, if used incorrectly. In my example, you cannot have a deadlock unless the ability to scope-lock the object is exposed to the public.
 My contention is that if the default is you do *not* expose the mutex to
 external callers, there *cannot* be a deadlock.

That's shuffling the fundamental issue from client code to library code. This can be done in either approach (synchronized vs. raw mutex), except it's clunkier with raw mutex.

Not talking about raw mutexes, my code is based on synchronized classes. Period.
 Here is the example again:

 synchronized class A
 {
 private int _foo;
  property int foo() { return _foo;}
 }

 synchronized class B
 {
 private A _a;
  property A a() { return _a;}
 void fun() {}
 }

 void thread(B b)
 {
 synchronized(b.a)
 {
 b.fun();
 }
 }

 If synchronized(b.a) is valid, deadlock can occur. If it's invalid,
 deadlock *cannot* occur.

I don't get what this is supposed to prove.

If you cannot synchronize on b.a, as an *external* entity, then you cannot deadlock. This is an important property. The author of A and B can say "deadlock-proof". How do you not see this as valuable? Right now, we have a choice: a. use synchronized -> prone to deadlocks that you cannot control or prevent. b. roll your own locking -> possible to prevent deadlocks in all cases, but prone to error since you could accidentally miss an unlock or lock. I want: c. use synchronized with *no* public access -> possible to prevent deadlocks in all cases, safe to use without forgetting to unlock/lock. -Steve
May 31 2012
prev sibling next sibling parent "foobar" <foo bar.com> writes:
On Thursday, 31 May 2012 at 11:24:56 UTC, Steven Schveighoffer 
wrote:
 On Wed, 30 May 2012 17:45:32 -0400, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 1:31 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

This is not what we are talking about.

Oh yes it is. "Expose the mutex" means "make the two mutex primitive operations lock() and unlock() freely usable against the mutex object".

Stop that! You are blatantly arguing minutia when you should be trying to understand what we are saying! This is like dealing with a stubborn child.

 -Steve

This reminds me one of my favorite Zen stories: Nan-in, a Japanese master during the Meiji era (1868-1912), received a university professor who came to inquire about Zen. Nan-in served tea. He poured his visitor's cup full, and then kept on pouring. The professor watched the overflow until he no longer could restrain himself. "It is overfull. No more will go in!" "Like this cup," Nan-in said, "you are full of your own opinions and speculations. How can I show you Zen unless you first empty your cup?"
May 31 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jun 1, 2012, at 5:26 AM, deadalnix wrote:
=20
 The main drawback is the same as opApply : return (and break/continue =

have been proposed in the past using compiler and stack magic.
=20
 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {
=20
 }
=20
 synchronized(rw.write) {
=20
 }

Opens the door? This works today exactly as outlined above. Or am I = missing a part of your argument?
 And many types of lock : spin lock, interprocesses locks, semaphores, =

exposing locking and unlocking primitives. All works today.=
Jun 01 2012
prev sibling next sibling parent Andrew Wiley <wiley.andrew.j gmail.com> writes:
--f46d04016b0306121d04c1969986
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com> wrote:

 Le 01/06/2012 22:55, Sean Kelly a =E9crit :

 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem h=



 been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

Opens the door? This works today exactly as outlined above. Or am I missing a part of your argument? And many types of lock : spin lock, interprocesses locks, semaphores, .
 . . And all can be used with the synchronized syntax, and without expos=



 locking and unlocking primitives.

All works today.

Unless you do some monitor magic, it doesn't.

Yes, it does. ----- class Something { private: ReadWriteLock _rw; public: this() { _rw =3D new ReadWriteLock(); } void doSomething() shared { synchronized(_rw.read) { // do things } } } ----- I've used this pattern in code. There might be some casting required because the core synchronization primitives haven't been updated to use shared yet. --f46d04016b0306121d04c1969986 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <span= dir=3D"ltr">&lt;<a href=3D"mailto:deadalnix gmail.com" target=3D"_blank">d= eadalnix gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quot= e" style=3D"margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb= (204,204,204);border-left-width:1px;border-left-style:solid"> Le 01/06/2012 22:55, Sean Kelly a =E9crit :<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;padding= -left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-l= eft-style:solid"><div class=3D"im"> On Jun 1, 2012, at 5:26 AM, deadalnix wrote:<br> </div><div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0= px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border= -left-width:1px;border-left-style:solid"> <br> The main drawback is the same as opApply : return (and break/continue but i= t is less relevant for opSynchronized). Solution to this problem have been = proposed in the past using compiler and stack magic.<br> <br> It open door for stuff like :<br> ReadWriteLock rw;<br> synchronized(rw.read) {<br> <br> }<br> <br> synchronized(rw.write) {<br> <br> }<br> </blockquote> <br></div><div class=3D"im"> Opens the door? =A0This works today exactly as outlined above. =A0Or am I m= issing a part of your argument?<br> <br> </div><div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0= px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border= -left-width:1px;border-left-style:solid"> And many types of lock : spin lock, interprocesses locks, semaphores, . . .= And all can be used with the synchronized syntax, and without exposing loc= king and unlocking primitives.<br> </blockquote> <br></div> All works today.<br> </blockquote> <br> Unless you do some monitor magic, it doesn&#39;t.<br> </blockquote><div>=A0</div><div>Yes, it does.=A0</div><div>-----</div><div>= class Something=A0{</div><div>=A0=A0=A0 private:</div></div><div>=A0=A0=A0= =A0=A0=A0=A0 ReadWriteLock _rw;</div><div>=A0=A0=A0 public:</div><div>=A0= =A0=A0=A0=A0=A0=A0 this() {</div><div>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 _rw= =3D new ReadWriteLock();</div> <div>=A0=A0=A0=A0=A0=A0=A0 }</div><div>=A0=A0=A0=A0=A0=A0=A0 void doSomethi= ng() shared {</div><div>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 synchronized(_rw.= read) {</div><div>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 // do thing= s</div><div>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 }</div><div>=A0=A0=A0=A0=A0= =A0=A0 }</div><div>}</div><div>-----</div> <div>=A0</div><div>I&#39;ve used this pattern in code. There might be some = casting required because the core synchronization primitives haven&#39;t be= en updated to use shared yet.<br></div> --f46d04016b0306121d04c1969986--
Jun 03 2012
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jun 3, 2012, at 12:29 PM, deadalnix wrote:

 Le 01/06/2012 22:55, Sean Kelly a =E9crit :
 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:
=20
 The main drawback is the same as opApply : return (and =



this problem have been proposed in the past using compiler and stack = magic.
=20
 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {
=20
 }
=20
 synchronized(rw.write) {
=20
 }

Opens the door? This works today exactly as outlined above. Or am I =


=20
 And many types of lock : spin lock, interprocesses locks, =



without exposing locking and unlocking primitives.
=20
 All works today.

Unless you do some monitor magic, it doesn't.

There is a bit of cleverness in there.
Jun 06 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, May 29, 2012 01:11:49 Alex R=C3=B8nne Petersen wrote:
 I have no idea how synchronized classes work; they are not a document=

 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

Per TDPL, having individually synchronized functions is illegal. Either= _all_=20 of the functions in a class are synchronized or _none_ of them are. Put= ting=20 synchronized on a function should actually be illegal. It belongs only = on=20 classes or in synchronized blocks, never on functions. Now, unfortuntately, I don't believe that the compiler enforces any of = that=20 right now, so you end up synchronizing indivdual functions rather than = whole=20 classes, but the synchronized functions themselves should function the = same. - Jonathan M Davis
May 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, May 29, 2012 01:11:49 Alex R=C3=B8nne Petersen wrote:
 I have no idea how synchronized classes work; they are not a document=

 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

Per TDPL, having individually synchronized functions is illegal. Either= all=20 of the functions in a class are synchronized or none of them are. Putti= ng=20 synchronized on a function should actually be illegal. It belongs only = on=20 classes or in synchronized blocks, never on functions. Now, unfortuntately, I don't believe that the compiler enforces any of = that=20 right now, so you end up synchronizing indivdual functions rather than = whole=20 classes, but the synchronized functions themselves should function the = same.
 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:
=20
 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and=

 SyncLock Keywords section)

Well, then you should probably be arguing about the design of synchroni= zed=20 classes/functions rather than synchronized(this). However, given the de= sign of=20 synchronized classes, synchronized(this) would probably never be approp= riate.=20 If you're using a synchronized class, then you don't need it. And if yo= u're=20 not using a synchronized class, then you shouldn't be synchronizing you= r=20 class. I would definitely think that synchronized blocks should synchro= nize on=20 something else, since they're trying to lock a much smaller area than a= n=20 entire class. - Jonathan M Davis
May 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, May 29, 2012 01:35:23 Alex R=C3=B8nne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way too m=

 code would break if we changed the semantics. I'd consider it a mista=

 and a lesson learned, rather.
=20
 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

I'm not an expert on threading stuf, but it would be my opinion that if= you're=20 not intending to protect the entire class with locks that it makes no s= ense to=20 lock on the class itself. You're locking for something specific, in whi= ch case,=20 your sychronized block should be locking on something else specific to = what=20 you're trying to protect. Certainly, that's how I'd approach it with mu= texes.=20 You don't have a mutex for an entire class unless it's actually used fo= r all=20 of the class' functions. Rather, you use mutexes specific to what you'r= e trying=20 to protect. - Jonathan M Davis
May 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, May 29, 2012 01:54:59 Alex R=C3=B8nne Petersen wrote:
 On 29-05-2012 01:41, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:35:23 Alex R=C3=B8nne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way to=



 code would break if we changed the semantics. I'd consider it a mi=



 and a lesson learned, rather.
=20
 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

I'm not an expert on threading stuf, but it would be my opinion tha=


 you're not intending to protect the entire class with locks that it=


 no sense to lock on the class itself. You're locking for something
 specific, in which case, your sychronized block should be locking o=


 something else specific to what you're trying to protect. Certainly=


 that's how I'd approach it with mutexes. You don't have a mutex for=


 entire class unless it's actually used for all of the class' functi=


 Rather, you use mutexes specific to what you're trying to protect.
=20
 - Jonathan M Davis

Right, but even if you really *are* protecting the entire class, you =

 still create mysterious deadlocks if users of your code lock on your
 class. So I'm arguing that no matter the use case, never lock on a
 'this' reference exposed outside of some API layer.

Well, it seems pretty abysmal to me to be locking on something that you= don't=20 control. Making a mutex that your class used public would just be stupi= d. With=20 synchronized classes/functions, you're basically creating and using an=20= implicit mutex for the whole class, which would then be the same as if = you=20 locked it at the beginning of every member function call and unlocked i= t at=20 its end, which doesn't expose the mutex at all. So, I don't really see = te=20 problem there would have to study the matter more to see what the issue= there=20 is. But locking on another class rather than something specifically int= ended as=20 a mutex does seem to me like it's asking for trouble. But maybe that mi= ndset=20 comes from using mutexes primarily as opposed to synchronized statement= s (as=20 most of that sort of programming that I've done has been in C++), and I= don't=20 necessarily understand all of the ways that programmers screw up with=20= synchronized blocks. - Jonathan M Davis
May 28 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 29/05/2012 00:36, Alex Rønne Petersen a écrit :
 Hi,

 I've seen several occurrences of synchronized (this) and synchronized
 (this.classinfo) in both druntime and phobos by now. I propose that we
 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happens
 to use the object for locking will most likely end up with a deadlock on
 their hands.
 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the same.

 The solution to (1) is to simply use a Mutex internally (or an Object
 with synchronized statements), and for (2), to simply use private global
 objects.

 (Now, regarding (1), you might argue that anyone who locks on an
 arbitrary object is doing it wrong, but you can't really blame them -
 it's frankly D's fault for allowing monitors on arbitrary objects, which
 is a horrible mess.)

 Anyone against this?

I already did some comment about this. Making any object synchronization is a very bad design decision. It is deadlock prone, it is liquid lock prone, it cause an overhead on any object, and even worse, it is useless most of the time as D promote thread locality (which is very good). OOP and concurrency are 2 orthogonal topics, and should be handled as such.
May 29 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

Actually I think such a characterization is superficial and biased to the extent it becomes wrong. Locks are deadlock-prone by design, whether used with objects or with classic procedural programs. In fact they are better confined to objects because the association between the protected data and the synchronization object is clear.
 OOP and concurrency are 2 orthogonal topics, and should be handled as
 such.

Well there would be active objects that contradict that. But really it is very natural to encapsulate together some data along with the mutex that must be acquired to manipulate it. There are known issues with inheritance, but they are not introduced by the approach, only exposed by it. Andrei
May 29 2012
parent reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

Actually I think such a characterization is superficial and biased to the extent it becomes wrong.

Really ? I think he's spot on.
 Locks are deadlock-prone by design, whether used with objects or with
 classic procedural programs. In fact they are better confined to objects
 because the association between the protected data and the
 synchronization object is clear.

There is no doubt that synchronization of any kind is hard and error-prone no matter how you do it. But do you really think that the situation is made better by allowing locking on *anything*, meaning that locks are effectively resources shared publicly? Besides, what's wrong with core.sync.mutex?
 OOP and concurrency are 2 orthogonal topics, and should be handled as
 such.

Well there would be active objects that contradict that. But really it is very natural to encapsulate together some data along with the mutex that must be acquired to manipulate it. There are known issues with inheritance, but they are not introduced by the approach, only exposed by it. Andrei

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

Actually I think such a characterization is superficial and biased to the extent it becomes wrong.

Really ? I think he's spot on.

I'd be glad to think the same. Good arguments might be helpful. So far I've seen exaggerated statements and posturing. Such are entertaining but are of limited effectiveness in furthering a point.
 Locks are deadlock-prone by design, whether used with objects or with
 classic procedural programs. In fact they are better confined to objects
 because the association between the protected data and the
 synchronization object is clear.

There is no doubt that synchronization of any kind is hard and error-prone no matter how you do it. But do you really think that the situation is made better by allowing locking on *anything*, meaning that locks are effectively resources shared publicly? Besides, what's wrong with core.sync.mutex?

Your very argument that anyone can lock on it. To that, I'll add that core.sync.mutex is not properly associated with the resource it protects, so coding with bare mutexes is in fact inferior to coding with synchronized objects. Andrei
May 29 2012
next sibling parent reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 30-05-2012 00:50, Andrei Alexandrescu wrote:
 On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

Actually I think such a characterization is superficial and biased to the extent it becomes wrong.

Really ? I think he's spot on.

I'd be glad to think the same. Good arguments might be helpful. So far I've seen exaggerated statements and posturing. Such are entertaining but are of limited effectiveness in furthering a point.
 Locks are deadlock-prone by design, whether used with objects or with
 classic procedural programs. In fact they are better confined to objects
 because the association between the protected data and the
 synchronization object is clear.

There is no doubt that synchronization of any kind is hard and error-prone no matter how you do it. But do you really think that the situation is made better by allowing locking on *anything*, meaning that locks are effectively resources shared publicly? Besides, what's wrong with core.sync.mutex?

Your very argument that anyone can lock on it. To that, I'll add that core.sync.mutex is not properly associated with the resource it protects, so coding with bare mutexes is in fact inferior to coding with synchronized objects. Andrei

I'm not sure I follow. A mutex can be stored privately. Any object can be locked on, meaning no lock is effectively protected or private or... anything. It encourages shared locks, which is seriously the worst deadlock inducing anti-pattern to have ever manifested in multithreaded programming. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/29/12 3:56 PM, Alex Rønne Petersen wrote:
 A mutex can be stored privately.

It can, but that doesn't mean it will.
 Any object can be locked on, meaning no
 lock is effectively protected or private or... anything.

To paraphrase you, "An object can be stored privately".
 It encourages
 shared locks, which is seriously the worst deadlock inducing
 anti-pattern to have ever manifested in multithreaded programming.

I'll ignore the hyperbole and continued posturing. Please understand it does absolutely nothing in carrying your point. A technical question I have is - what are shared locks, and what are superior alternatives to them? Andrei
May 29 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 3:04, Andrei Alexandrescu wrote:
 On 5/29/12 3:56 PM, Alex Rønne Petersen wrote:
 A mutex can be stored privately.

It can, but that doesn't mean it will.
 Any object can be locked on, meaning no
 lock is effectively protected or private or... anything.

To paraphrase you, "An object can be stored privately".
 It encourages
 shared locks, which is seriously the worst deadlock inducing
 anti-pattern to have ever manifested in multithreaded programming.

I'll ignore the hyperbole and continued posturing. Please understand it does absolutely nothing in carrying your point. A technical question I have is - what are shared locks, and what are superior alternatives to them?

I'll intervene. Following famous nerd motto "talk is cheap, show me the code" I strongly suggest discussing concrete scenarios. All "prone to deadlock sentiment" is trivia as in buyer beware. That being said: class iMutexed {//implicit intent, explicit details Mutex mutex; } vs class iMutexed {//implicit intent, implicit details //implicit mutex } Makes little to no difference. EXCEPT that designer of the class has no control what so ever over hidden mutex! Thus I believe the best way to fix it is to (say) require obj be a subclass of Mutex in order to use synchronized(obj). i.e. class iMutexed: Mutex {//explicit intent, implicit details } -- Dmitry Olshansky
May 29 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 12:17, Thiez wrote:
 On Wednesday, 30 May 2012 at 06:31:32 UTC, Dmitry Olshansky wrote:
 I'll intervene. Following famous nerd motto "talk is cheap, show me
 the code" I strongly suggest discussing concrete scenarios. All "prone
 to deadlock sentiment" is trivia as in buyer beware.

 That being said:

 class iMutexed
 {//implicit intent, explicit details
 Mutex mutex;
 }

 vs

 class iMutexed
 {//implicit intent, implicit details

 //implicit mutex
 }

 Makes little to no difference. EXCEPT that designer of the class has
 no control what so ever over hidden mutex! Thus I believe the best way
 to fix it is to (say) require obj be a subclass of Mutex in order to
 use synchronized(obj). i.e.

 class iMutexed: Mutex
 {//explicit intent, implicit details

 }

Forcing synchronized classes to extend Mutex would make it impossible to create a synchronized subclass of a pre-existing unsynchronized class, would it not?

Composition. Obviously synchronized subclasses of existing class sounds strange. The other way around even worse ;) Wrapping synchronized classes obviously fine. Unless you want to introduce multiple inheritence. If you
 want to go this way at least make it an interface that has special
 meaning to the compiler, instead of a class that must be inherited from.

No thanks, multiple inheritance brings too much "coolness". Interface is no go, locks should be effective. -- Dmitry Olshansky
May 30 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30.05.2012 12:33, Dmitry Olshansky wrote:
 On 30.05.2012 12:17, Thiez wrote:
 On Wednesday, 30 May 2012 at 06:31:32 UTC, Dmitry Olshansky wrote:
 I'll intervene. Following famous nerd motto "talk is cheap, show me
 the code" I strongly suggest discussing concrete scenarios. All "prone
 to deadlock sentiment" is trivia as in buyer beware.

 That being said:

 class iMutexed
 {//implicit intent, explicit details
 Mutex mutex;
 }

 vs

 class iMutexed
 {//implicit intent, implicit details

 //implicit mutex
 }

 Makes little to no difference. EXCEPT that designer of the class has
 no control what so ever over hidden mutex! Thus I believe the best way
 to fix it is to (say) require obj be a subclass of Mutex in order to
 use synchronized(obj). i.e.

 class iMutexed: Mutex
 {//explicit intent, implicit details

 }

Forcing synchronized classes to extend Mutex would make it impossible to create a synchronized subclass of a pre-existing unsynchronized class, would it not?

Composition. Obviously synchronized subclasses of existing class sounds strange. The other way around even worse ;) Wrapping synchronized classes obviously fine.

Wrapping by synchronized class.... -- Dmitry Olshansky
May 30 2012
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 00:50, Andrei Alexandrescu a écrit :
 On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

Actually I think such a characterization is superficial and biased to the extent it becomes wrong.

Really ? I think he's spot on.

I'd be glad to think the same. Good arguments might be helpful. So far I've seen exaggerated statements and posturing. Such are entertaining but are of limited effectiveness in furthering a point.

I have provided link in other posts to document the point. Not to mention that my personal experience back up this too. I don't like what you are doing here, because you don't provide anything and simply discard what don't fit the current design. For instance, you didn't even considered the liquid lock problem.
May 30 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/12 2:40 AM, deadalnix wrote:
 Le 30/05/2012 00:50, Andrei Alexandrescu a écrit :
 On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

Actually I think such a characterization is superficial and biased to the extent it becomes wrong.

Really ? I think he's spot on.

I'd be glad to think the same. Good arguments might be helpful. So far I've seen exaggerated statements and posturing. Such are entertaining but are of limited effectiveness in furthering a point.

I have provided link in other posts to document the point. Not to mention that my personal experience back up this too. I don't like what you are doing here, because you don't provide anything and simply discard what don't fit the current design. For instance, you didn't even considered the liquid lock problem.

It would help if I knew what a liquid lock is. The first page of Google search doesn't help. I'm only discarding content-less posturing a la "worst idea ever", "useless", "very bad design decision" etc. Such is just immature. It is encouraging you have started sending content (the msdn paper), thanks, and please keep it coming. Thanks, Andrei
May 30 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 30/05/2012 17:50, Andrei Alexandrescu a écrit :
 On 5/30/12 2:40 AM, deadalnix wrote:
 Le 30/05/2012 00:50, Andrei Alexandrescu a écrit :
 On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision.
 It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

Actually I think such a characterization is superficial and biased to the extent it becomes wrong.

Really ? I think he's spot on.

I'd be glad to think the same. Good arguments might be helpful. So far I've seen exaggerated statements and posturing. Such are entertaining but are of limited effectiveness in furthering a point.

I have provided link in other posts to document the point. Not to mention that my personal experience back up this too. I don't like what you are doing here, because you don't provide anything and simply discard what don't fit the current design. For instance, you didn't even considered the liquid lock problem.

It would help if I knew what a liquid lock is. The first page of Google search doesn't help. I'm only discarding content-less posturing a la "worst idea ever", "useless", "very bad design decision" etc. Such is just immature. It is encouraging you have started sending content (the msdn paper), thanks, and please keep it coming. Thanks, Andrei

Ok let's me explain what liquid lock is. It is a common error in languages that allow lock on any object. The phenomena is fairly simple, but lead to really tedious debugging session. It happen when you use an object that is useful for some functionality of your program as a mutex. This is OK because the instance is always the same. Then, later on, when adding new functionality/refactoring/whatever, you introduce a possibility to change the instance you lock on. In such situation, you seems to lock in an appropriate manner, but you fail to ensure correct locking when it occur at the exact moment the object referred is changed. This is really tedious to track down, and quite easy to create when several developers are working on the same codebase for an extended period of time. To avoid this, you must ensure that any reference on object you mutate isn't used somewhere as a mutex. It is considered good practice in Java to not lock on any object and use a specific object to lock on. The reference to that object will be final, so it is guaranteed to not change. For instance, Intellij issue a warning about that : http://www.coderanch.com/t/233943/threads/java/Synchronization-non-final-field At the end, locking on any object cause trouble. Locking on any object has proven itself to be a very bad design decision in Java and C#. I don't think it is immature to say that. This is well known in java community, and I guess it is too in C# (I have way less experience with C# ).
May 30 2012
prev sibling parent "Thiez" <thiezz gmail.com> writes:
On Wednesday, 30 May 2012 at 06:31:32 UTC, Dmitry Olshansky wrote:
 I'll intervene. Following famous nerd motto "talk is cheap, 
 show me the code" I strongly suggest discussing concrete 
 scenarios. All "prone to deadlock sentiment" is trivia as in 
 buyer beware.

 That being said:

 class iMutexed
 {//implicit intent, explicit details
 	Mutex mutex;
 }

 vs

 class iMutexed
 {//implicit intent, implicit details

 //implicit mutex
 }

 Makes little to no difference. EXCEPT that designer of the 
 class has no control what so ever over hidden mutex! Thus I 
 believe the best way to fix it is to (say) require obj be a 
 subclass of Mutex in order to use synchronized(obj). i.e.

 class iMutexed: Mutex
 {//explicit intent, implicit details

 }

Forcing synchronized classes to extend Mutex would make it impossible to create a synchronized subclass of a pre-existing unsynchronized class, would it not? Unless you want to introduce multiple inheritence. If you want to go this way at least make it an interface that has special meaning to the compiler, instead of a class that must be inherited from.
May 30 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Tue, 29 May 2012 01:08:59 +0100, Jonathan M Davis <jmdavisProg gmx.co=
m>  =

wrote:

 On Tuesday, May 29, 2012 01:54:59 Alex R=F8nne Petersen wrote:
 On 29-05-2012 01:41, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:35:23 Alex R=F8nne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way to=




 much
 code would break if we changed the semantics. I'd consider it a  =




 mistake
 and a lesson learned, rather.

 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

I'm not an expert on threading stuf, but it would be my opinion tha=



 if
 you're not intending to protect the entire class with locks that it=



 makes
 no sense to lock on the class itself. You're locking for something
 specific, in which case, your sychronized block should be locking o=



 something else specific to what you're trying to protect. Certainly=



 that's how I'd approach it with mutexes. You don't have a mutex for=



 entire class unless it's actually used for all of the class'  =



 functions.
 Rather, you use mutexes specific to what you're trying to protect.

 - Jonathan M Davis

Right, but even if you really *are* protecting the entire class, you =


 still create mysterious deadlocks if users of your code lock on your
 class. So I'm arguing that no matter the use case, never lock on a
 'this' reference exposed outside of some API layer.

Well, it seems pretty abysmal to me to be locking on something that yo=

 don't control. Making a mutex that your class used public would just b=

 stupid.

Interestingly this is what C#s Array type does with SyncRoot = (intentionally).
 With synchronized classes/functions, you're basically creating and usi=

 an
 implicit mutex for the whole class, which would then be the same as if=

 you locked it at the beginning of every member function call and  =

 unlocked it at its end, which doesn't expose the mutex at all. So, I  =

 don't really see te
 problem there would have to study the matter more to see what the issu=

 there is.

According to the docs here: http://dlang.org/class.html#synchronized-functions A synchronized function locks "this" and as "this" is exposed publicly..= . = In the following code the "lock" statement and "synchronized void bar" = lock the same mutex. class Foo { synchronized void bar() { ...statements... } } void main() { Foo foo =3D new Foo(); lock(foo) { ...statements... } }
 But locking on another class rather than something specifically intend=

 as a mutex does seem to me like it's asking for trouble.

Yep. It commonly arises where you have a class/object which is either n= ot = synchronized itself (because it might be used in single threaded = situations and you want performance) like a collection class for example= , = or is synchronized but you need to lock a sequence of member function = calls i.e. a lookup and insert on the collection. What happens then, is= = people just call lock(<object>) and end up locking the same mutex as the= = class does internally. What happens next to cause a deadlock is that = inside that lock they call one or more functions or methods which = internally call methods on another synchronized object. This results in= a = locking pattern of object1, object2. In another piece of code, running = in = another thread, they do something similar which results in a locking = pattern of object2, object1 and eventually these threads will deadlock = each other. R -- = Using Opera's revolutionary email client: http://www.opera.com/mail/
May 29 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Tue, 29 May 2012 13:07:29 +0100, Regan Heath <regan netmail.co.nz>  
wrote:
 void main()
 {
    Foo foo = new Foo();
    lock(foo)
    {
      ...statements...
    }
 }

Apologies..I've been writing more C# than D lately. "lock" above should be "synchronized" for D. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 29 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On May 29, 2012, at 4:07 PM, Andrei Alexandrescu <SeeWebsiteForEmail erdani.=
org> wrote:

 On 5/29/12 3:58 PM, Alex R=C3=B8nne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex R=C3=B8nne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? =20 Andrei

core.sync.mutex

That's worse. =20 Andrei =20

I don't agree.

One simple thing to understand is that core.sync.mutex does everything syn=

d an argument that synchronized classes do something wrong but bare, unstruc= tured mutexes do something good.
=20
 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

We'd need to take a look at that. I recall at a point Bartosz was working o=

nyhow, it's something that would be interesting to look at. I bet this is because monitors are lazily initialized, so the cost of acquir= ing a lock is more than just locking the underlying mutex. The implementatio= n for built-in monitors really isnt great. I've been meaning to do something= about that.=20=
May 29 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com> wrot=
e:

 Le 30/05/2012 00:53, Andrei Alexandrescu a =E9crit :
 On 5/29/12 3:01 PM, Alex R=F8nne Petersen wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex R=F8nne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex R=F8nne Petersen a =E9crit :
 I should probably add that Java learned it long ago, and yet we=








 adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to=







 repeat
 previously done errors.

So what is the lesson Java learned, and how does it address multithreaded programming in wake of that lesson? Andrei

It learned that allowing locking on arbitrary objects makes =





 controlling
 locking (and thus reducing the chance for deadlocks) impossible.

And how does Java address multithreading in general, and those issu=




 in
 particular, today?

 Andrei

It doesn't, and neither does C#. Java still encourages using synchronized, and C# still encourages using lock, but many prominent=



 figures in those programming language communities have written blog
 posts on why these language constructs are evil and should be avoide=



 Some citations (beyond the known fallacies of Java 1.0) would be grea=


 Thanks.

 Besides, it seems to me that D can't quite make up its mind. We have=



 TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribut=



 It just seems so incredibly inconsistent. synchronized encourages do=



 the wrong thing (locks and synchronization).

Each paradigm has its place. Lock-based programming is definitely her=


 to stay, and when the paradigm is needed, doing it with synchronized
 objects is better than most alternatives.


 Andrei

No ! You don't want to synchronize on ANY object. You want to synchronize o=

 explicit mutexes.

+1 .. this is the key point/issue. If all objects can be locked, and if= = synchronized classes/methods use the /same/ mutex you can = easily/accidentally have hard to find deadlocks. The deadlocks in Q are= = usually (at least) two threads locking (at least) two objects in the = opposite order and with synchronized classes/methods the fact that a loc= k = is being taken is not always obvious from the code (at the call site) so= = the problem code can look fairly innocuous. **[Example 1]** class C { synchronized void ccc() { } } class D { synchronized void ddd() { } } C c =3D new C(); D d =3D new D(); [thread1] ..lock(c) // locks c { // deadlock window d.ddd(); // locks d } [thread2] ..lock(d) // locks d { // deadlock window c.ccc(); // locks c } **[Example 2]** class A { B b; .. synchronized void foo() // locks a { // deadlock window b.bar(); // locks b } } class B { A a; .. synchronized void bar() // locks b { // deadlock window a.foo(); // locks a } } A a =3D new A(); B b =3D new B(); a.b =3D b; b.a =3D a; [thread1] a.foo(); // locks a then b [thread2] b.bar(); // locks b then a So, the root cause of the problem is that synchronized classes/methods u= se = a publicly visible mutex (the object/this pointer) to perform their = locking. The solution is to lock on a private mutex, which no-one else = = can lock unexpectedly as described in the link below:
 See that link for instance :  =

 http://msdn.microsoft.com/en-us/library/ms173179.aspx

Now, ideally we want to make it hard for people to screw this up in this= = manner and there are several ways to do it. 1. Prevent locking on any/every object. People would have to create a = separate mutex object for locking. This is a little tedious, but it doe= s = make people think about the scope of the locking (where to put the mutex= , = who can see/use it, etc). On the flipside it can make people re-use a = mutex for multiple conceptual critical section areas of code, which is = less than ideal, but at least it's not 'broken'. 2. Allow locking on any/every object but use a different mutex for = synchronized class/methods. So 'synchronized' on a method call locks a = = private mutex object, not the object itself. And synchronized(object) = locks the public mutex object. The downside here is that now every obje= ct = potentially has 2 mutex objects/handles internally - a public and a = private one. .. there are probably a number of other ways to do it. I favour #1, = especially as thread-local is the default storage type now. 100% of = objects in a single threaded application don't need a built-in mutex, mo= st = objects (80%??) in a threaded application will be thread-local and don't= = need one. Only those few shared objects, which are actually manually = locked with synchronized or have synchronized class/methods actually nee= d = a mutex. It seems that removing this will save memory and avoid tricky = = deadlock bugs and the cost is having to manually create a mutex object = (1/2 lines of code) to lock instead.. it seems like a good idea to me. R -- = Using Opera's revolutionary email client: http://www.opera.com/mail/
May 30 2012
prev sibling next sibling parent "Thiez" <thiezz gmail.com> writes:
On Wednesday, 30 May 2012 at 12:43:15 UTC, deadalnix wrote:
 It doesn't address most of the drawback cited. Notably the fact 
 that every object have a monitor field, but most of them will 
 not use it.

That could be solved without abandoning object locks. If locking on all objects is allowed, objects in TLS could simply omit the monitor field. Locking would then be performed as: if (object not in TLS) { insert expensive locking operation here } return Yes, this would introduce an extra conditional jump whenever a lock is acquired/released, but the overhead is very small compared to the relatively expensive lock/unlock operation.
May 30 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 30 May 2012 13:43:14 +0100, deadalnix <deadalnix gmail.com> wrot=
e:
 Le 30/05/2012 14:32, Regan Heath a =E9crit :
 1. Prevent locking on any/every object. People would have to create a=


 separate mutex object for locking. This is a little tedious, but it d=


 make people think about the scope of the locking (where to put the
 mutex, who can see/use it, etc). On the flipside it can make people
 re-use a mutex for multiple conceptual critical section areas of code=


 which is less than ideal, but at least it's not 'broken'.

It is suboptimal, but correct.

Indeed, and can be made optimal with some fine-tuning/work.
 2. Allow locking on any/every object but use a different mutex for
 synchronized class/methods. So 'synchronized' on a method call locks =


 private mutex object, not the object itself. And synchronized(object)=


 locks the public mutex object. The downside here is that now every
 object potentially has 2 mutex objects/handles internally - a public =


 a private one.

It doesn't address most of the drawback cited. Notably the fact that =

 every object have a monitor field, but most of them will not use it.

True, I was just mentioning it as an option that solves the key = issue/problem - not as the best solution to that problem. I think #1 is= = better. Ultimately the "problem" that needs solving is the fact that synchronize= d = classes/methods use a public mutex and this exposure makes deadlocks mor= e = likely. I was hoping that by giving some examples in code, it would = become clearer to the people who don't see what all the fuss is about. Yes, having to manually create a mutex is a pain, and as Andrei has = mentioned a few times having 2 separate things (one object to lock, one = = mutex) which conceptually should be one thing is worse from an = understanding/maintenance point of view, however.. 1. If the mutex is intended to lock a single object then we can make the= = following options available: a. A Lockable class to derive from. b. A Lockable struct to compose with (assuming here that the struct w= ill = be created/destroyed with the object). c. A Lockable!(T) wrapper template/struct/class to wrap objects to be= = locked. d. A Lockable interface which classes may implement. e. Some sort of compiler magic where it supplies the mutex for = objects/methods marked with "synchronized" or involved in a = synchronized(object) statement. 2. If the scope of the locking is greater than a single object, then a = separate mutex is required/desired anyway. 1a. would contain a mutex primitive and lock/tryLock/unlock methods (it = = would implement the interface in 1d) 1b. would also contain a mutex primitive and alias could be used to expo= se = the lock/tryLock/unlock methods (it would implement the interface in 1d)= 1c. is actually a technique I have used a lot in C++ where the template = = class is created on the stack of a function/critical section and it = (optionally) locks on creation (or later when lock() is called) and alwa= ys = unlocks on destruction as it leaves scope (by any means). It's a very = robust pattern, and is similar (the inspiration/or result) of/to the = synchronized block itself. 1d. could be a requirement for classes which are to be used in = synchronized statements i.e. class A : Lockable { void lock() {} void unlock() {} } A a =3D new A(); synchronized(a) { // calls a.lock() ... } // calls a.unlock() 1e. I leave as a opener to Walter/Andrei/the reader .. I've seen many a = = genius idea pop out of nowhere on this forum. IMO, the wasted space used by the monitor isn't ideal, but it's not a = "problem" for the general case. It is a problem for people with limited= = memory environments but that's another topic. Talking about both is onl= y = clouding the discussional waters so to speak. R -- = Using Opera's revolutionary email client: http://www.opera.com/mail/
May 30 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 05/30/12 10:17, Thiez wrote:
 On Wednesday, 30 May 2012 at 06:31:32 UTC, Dmitry Olshansky wrote:
 I'll intervene. Following famous nerd motto "talk is cheap, show me the code"
I strongly suggest discussing concrete scenarios. All "prone to deadlock
sentiment" is trivia as in buyer beware.

 That being said:

 class iMutexed
 {//implicit intent, explicit details
     Mutex mutex;
 }

 vs

 class iMutexed
 {//implicit intent, implicit details

 //implicit mutex
 }

 Makes little to no difference. EXCEPT that designer of the class has no
control what so ever over hidden mutex! Thus I believe the best way to fix it
is to (say) require obj be a subclass of Mutex in order to use
synchronized(obj). i.e.

 class iMutexed: Mutex
 {//explicit intent, implicit details

 }

Forcing synchronized classes to extend Mutex would make it impossible to create a synchronized subclass of a pre-existing unsynchronized class, would it not? Unless you want to introduce multiple inheritence. If you want to go this way at least make it an interface that has special meaning to the compiler, instead of a class that must be inherited from.

Yeah, that leads to multiple inheritance, or cheap imitations thereof. Just create lock+unlock (or acquire+release, whatever) operators that can be overloaded, ie void opUnary!("lock")() {} void opUnary!("unlock")() {} and make 'synchronize (struct|class){...}' use them, ditto for implicitly synchronized public methods (note: by lowering directly. You can already do something like this right now, but it would have a large runtime overhead.) Full backward compatibility retained, and the user can actually implement the locking in a sane manner - something that is not likely to happen any time soon as a built-in language feature. And yes, removing the monitors from classes is desirable and would have back-compat issues, but it is then just an optimization, and does not need to happen immediately. On 05/30/12 11:34, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate synchronized classes? Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable = isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

Yep, a way to mark objects usable for locking would also help. I'd drop the 'isShared' check - it's redundant and doesn't play well with D 'shared' concept in its current form; it can always be introduced back, when 'shared' evolves to something more sane. On 05/30/12 11:14, Jacob Carlborg wrote:
 On 2012-05-29 23:48, Andrei Alexandrescu wrote:
 
 Don't be hatin'. You'd appreciate the matter considerably more after you
 defined your own language and had its users yell at you for changing
 even a small detail.

 The situation is at it is, and there's no need to get agitated. We're
 all on the same boat, trying to make things work out best.

It seems more and more that D2 is not a designed language. Instead new features are just slapped on without considering how it would impact the rest of the language.

Yes. But it is not so much the adhoc features that are the problem, as long as they don't significantly break the rest of the language. Not fixing known deficiencies in the name of backward compatibility is. The gain from having a new, useful and working feature more than makes up for the cost of changes to app code. Having features that work for trivial hello-world programs, but fail for most real code, does much more harm than good. artur
May 30 2012
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 28 May 2012 18:36:13 -0400, Alex R=C3=B8nne Petersen <alex lycus=
.org>  =

wrote:

 Hi,

 I've seen several occurrences of synchronized (this) and synchronized =

 (this.classinfo) in both druntime and phobos by now. I propose that we=

 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happe=

 to use the object for locking will most likely end up with a deadlock =

 their hands.
 2) Locking on something as fundamental as type info means that any  =

 arbitrary part of the application could cause a deadlock by doing the =

 same.

Reading all of this thread, I think I agree with you. Now, I'll point out a couple things: 1. Most code that is written with synchronized is *not* abused. 2. using synchronized(x) should not go away. Here is what I'm thinking: First, we give control of the mutex to the author of the type. So inste= ad = of an implicit location, and implicit lazy construction, we make = everything explicit: Currently: synchronized(x) { } translates to if(x.mutex =3D=3D null) x.mutex =3D createmutex(); lock(x.mutex) try { } finally { unlock(x.mutex); } So I propose it does this instead (entirely via lowering): synchronized(x) { } translates to: auto l =3D x.__getMutex(); l.lock(); try { } finally { l.unlock(); } So we have a few benefits here: 1. if typeof(x) does not define __getMutex, the above is a compiler erro= r. 2. We can control the visibility of the mutex, because we can attribute = = private or protected to the __getMutex function. 3. We can control what kinds of objects of typeof(x) use mutexes. e.g. = = define __getMutex() shared only. 4. We have control over the type of the lock, it can be anything and doe= s = not need to fit into some runtime function's interface. Second, I propose a somewhat gradual transition. I think on first = release, we *still* keep the allocated word in the class instance, and = keep the druntime function which lazily allocates the mutex. Therefore,= = changing an object that currently *properly* can use synchronized(this) = = can be made to work: class Synchronizable { xxx __getMutex() shared { return _d_getObjectMonitor(this);} } We can deprecate that functionality, and the allocated space for the mut= ex = later if desired. -Steve
May 30 2012
parent Artur Skawina <art.08.09 gmail.com> writes:
On 05/30/12 17:50, Steven Schveighoffer wrote:
 So I propose it does this instead (entirely via lowering):
 
 synchronized(x)
 {
 }
 
 translates to:
 
 auto l = x.__getMutex();
 l.lock();
 try
 {
 }
 finally
 {
    l.unlock();
 }

I like this, it's slightly more complex, but can reduce the amount of boilerplate (by not requiring lock/unlock forwarders etc) and is more powerful than my proposal.
 So we have a few benefits here:
 
 1. if typeof(x) does not define __getMutex, the above is a compiler error.
 2. We can control the visibility of the mutex, because we can attribute
private or protected to the __getMutex function.
 3. We can control what kinds of objects of typeof(x) use mutexes.  e.g. define
__getMutex() shared only.
 4. We have control over the type of the lock, it can be anything and does not
need to fit into some runtime function's interface.

5. The lock doesn't have to be a real lock, but eg a lock manager. Hmm, lowering to l.lock(x) and l.unlock(x) might be better; inlining will make it zero-cost for the simple case, but this will make certain schemes much simplier (no need to encode 'x' inside the returned 'l').
 Second, I propose a somewhat gradual transition.  I think on first release, we
*still* keep the allocated word in the class instance, and keep the druntime
function which lazily allocates the mutex.  Therefore, changing an object that
currently *properly* can use synchronized(this) can be made to work:
 
 class Synchronizable
 {
    xxx __getMutex() shared { return _d_getObjectMonitor(this);}
 }
 
 We can deprecate that functionality, and the allocated space for the mutex
later if desired.

Yes, UFCS could help too. artur
May 30 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a =E9crit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecat=



 synchronized classes?

 Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable =3D isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is t=


 we lock explicit stuff.

But in this design anyone can lock such an object, which was something=

 you advocated against.

I think there is some confusion here as to what the "problem" is and is = = not. The problem is /not/ that you can lock any object. The problem is /not/ that we have synchronized(object) {} The problem is /not/ that we have synchronized classes/methods. The problem /is/ that synchronized classes/methods use a mutex which is = = exposed publicly, and it the same mutex as used by synchronized(object) = = {}. This exposure/re-use makes deadlocks more likely to happen, and = harder to spot. Removal of this "problem" will not stop deadlocks from happening as = they're a problem multi-threaded/lock based code will always have (*). = It = will however make them far less likely to happen accidentally. It will = = make programmers think about what/where/when and how to lock things rath= er = than blithely using synchronized everywhere. It will mean using locks i= s = not as easy as currently, though I think we can make it fairly nice with= = good library code and/or some co-operation between the runtime and = synchronized() statements i.e. requiring the class implement a common = Lockable interface for example. The fact that every object can be locked, and this means they all use mo= re = memory is a side issue - arguably as important for some. R (*) the best you can do to "solve" the deadlock problem is to impose loc= k = acquisition ordering on your code, as in all locks must be acquired in a= = defined order, and if not an assertion/error/exception is thrown. This = = requires some sort of static/singelton/overlord class which is aware of = = all mutexes and is involved in all acquisitions/releases. -- = Using Opera's revolutionary email client: http://www.opera.com/mail/
May 30 2012
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 30 May 2012 17:23, Alex R=F8nne Petersen <alex lycus.org> wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 On 5/30/12 9:03 AM, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a =E9crit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecat=






 synchronized classes?

 Andrei

I think something similar to range design here is the way to go. It is easy to define something like template isLockable(T) { enum isLockable =3D isShared!T && is(typeof(T.init.lock())) && is(typeof(T.init.release())); } And allow locking only if(isLockable!Type) . Now we can create SyncObject or any structure we want. The point is that we lock explicit stuff.

But in this design anyone can lock such an object, which was something you advocated against.

I think there is some confusion here as to what the "problem" is and is not. The problem is /not/ that you can lock any object. The problem is /not/ that we have synchronized(object) {} The problem is /not/ that we have synchronized classes/methods.

Several posts in this thread assert that such are problems.
 The problem /is/ that synchronized classes/methods use a mutex which is
 exposed publicly, and it the same mutex as used by synchronized(object)
 {}. This exposure/re-use makes deadlocks more likely to happen, and
 harder to spot.

This is news to me. How do you publicly access the mutex of a synchronized class object?

Generally in two ways: 1) synchronized (obj) 2) obj.__monitor (1) accesses it in order to actually do locking, but if obj is an instantiation of a class that uses the this reference internally for locking, you end up with potential deadlocks. Therefore, you can say that obj's mutex is exposed. This isn't necessarily bad, because if obj does *not* use its this reference for synchronized statements, nothing will bl=

 up. This is why the OP was about banning synchronized (this).

 (2) is an undocumented feature of the language that druntime (and maybe s=

 phobos code) makes use of to create/alter/delete monitors.

I'm pretty certain the use of .__monitor is stricted only to: object_.d - The frontend implementation which contains the underlying functions that are invoked when you synchronise(obj). monitor_.d - The backend implementation which contains the platform separated bits. Regards --=20 Iain Buclaw *(p < e ? p++ : p) =3D (c & 0x0f) + '0';
May 30 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>  
 wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

+1 .. this is the key point/issue.

TDPL's design only allows for entire synchronized classes (not separate synchronized and unsynchronized methods), which pair mutexes with the data they protect. This is more restrictive than exposing mutexes, but in a good way. We use such a library artifact in C++ at Facebook all the time, to great success.

Can you call pass them to a synchronized statement? i.e. TDPLStyleSynchClass a = new TDPLStyleSynchClass(); synchronized(a) { } .. because, if you can, then you're exposing the mutex. If you can't, how do you do the above with these classes? By explicitly calling a lock() or tryLock() method? (do they implement a known interface?)
 People shouldn't create designs that have synchronized classes referring  
 to one another naively. Designing with mutexes (explicit or implicit)  
 will always create the possibility of deadlock, so examples how that  
 could happen are easy to come across.

True. But in my Example 1 the classes could be "entire" synchronized classes, and they do not refer to each other naively. Instead, two threads have shared access to them and in each case they explicitly acquire one mutex (via a synchronized() statement), but implicitly acquire another (by calling a method). It's the implicit acquisition which makes the bug hard to see and easy to do accidentally.
 TDPL improves on deadlocks by introducing synchronized statements with  
 more than one argument, see 13.15.

Is there anywhere I can see this online? (for free :p)
 The implicit mutexes used by synchronized classes are recursive.

:) why would you want anything else. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 30 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 30 May 2012 18:16:38 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 9:43 AM, Regan Heath wrote:
 On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>
 wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

+1 .. this is the key point/issue.

TDPL's design only allows for entire synchronized classes (not separate synchronized and unsynchronized methods), which pair mutexes with the data they protect. This is more restrictive than exposing mutexes, but in a good way. We use such a library artifact in C++ at Facebook all the time, to great success.

Can you call pass them to a synchronized statement? i.e. TDPLStyleSynchClass a = new TDPLStyleSynchClass(); synchronized(a) { }

Yes. Well I recommend acquiring the text! :o)
 ... because, if you can, then you're exposing the mutex.

No.

For the purposes of this thread, and the anti-pattern/problem we're discussing, you are. It is the combination of synchronized classes/methods (implicit locking) and external synchronized statements (explicit locking) which result in the unexpected, accidental, and hard to see deadlocks we're talking about here.
 People shouldn't create designs that have synchronized classes
 referring to one another naively. Designing with mutexes (explicit or
 implicit) will always create the possibility of deadlock, so examples
 how that could happen are easy to come across.

True. But in my Example 1

Your example 1 should not compile.

o.. k.. I expected you would get my meaning with the simplified example. Here you are: import core.thread; import std.random; import std.stdio; class C { synchronized void ccc() { writefln("c.ccc()"); } } class D { synchronized void ddd() { writefln("d.ddd()"); } } shared C c; shared D d; void main() { c = new shared(C)(); d = new shared(D)(); Thread thread1 = new Thread( &threadFunc1 ); Thread thread2 = new Thread( &threadFunc2 ); thread1.start(); thread2.start(); Thread.sleep(dur!("seconds")(60)); } void threadFunc1() { Random gen = rndGen(); while(1) { synchronized(c) { printf("threadFunc1 sync(c) %p, obtain d.. %p\n", c, d); d.ddd(); } Thread.sleep(dur!("usecs")(gen.front())); gen.popFront(); } } void threadFunc2() { Random gen = rndGen(); while(1) { synchronized(d) { printf("threadFunc1 sync(d) %p, obtain c.. %p\n", d, c); c.ccc(); } Thread.sleep(dur!("usecs")(gen.front())); gen.popFront(); } } Runs and deadlocks immediately.
 TDPL improves on deadlocks by introducing synchronized statements with
 more than one argument, see 13.15.

Is there anywhere I can see this online? (for free :p)

http://goo.gl/ZhPM2

Thanks. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 30 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 May 2012 13:12:47 -0400, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 9:23 AM, Alex R=C3=B8nne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

Generally in two ways: 1) synchronized (obj)

This is not accessing the mutex for arbitrary operations.

If I might interject, I think the issue is that as the author of a class= , = you cannot enforce that *only* the data in your class is protected by th= e = mutex. Yes, you can enforce that the data is protected, but there is nothing = stopping a 3rd party caller from hijacking your mutex to protect other = things it shouldn't be involved with. For instance, let's say you have: synchronized class A { private int _foo; property int foo() { return _foo;} } synchronized class B { private A _a; property A a() { return _a;} void fun() {} } void thread(B b) { synchronized(b.a) { b.fun(); } } Run multiple instances of this thread, and it will deadlock. As the = author of A, you have no control for preventing external code from = hijacking your mutex for its own purposes in an incorrect way. But take out this public access to your mutex, and you *can* enforce A's= = locking semantics correctly, and therefore B's semantics. Absolutely no= = way can A or B's lock participate in a deadlock. I think it is worth making this enforceable. In other words, the = synchronized(b.a) statement shouldn't compile. Yes, you can just use a private mutex. But doesn't that just lead to = recommending not using a feature of the language? Besides, I really lik= e = the synchronized block keyword thing. It's a great mechanism for lockin= g. -Steve
May 30 2012
prev sibling next sibling parent "cal" <callumenator gmail.com> writes:
On Wednesday, 30 May 2012 at 17:46:11 UTC, Dmitry Olshansky wrote:
 [snip]
 Such an object is known to be lockable, and most object will 
 not be. It will avoid liquid lock, because most thing will 
 not be lockable.

I'm celebrating day 2 of having no idea what a liquid lock is.

I have to confess to this as well. I'm starting to suspect that he really means deadlock. Searching on Google for "liquid lock mutex" literally brings up this very NG thread. :)

P.S. Just trying to snatched the prize in this little guess-game :)

FWIW, I recently came across the term here: http://schneide.wordpress.com/tag/liquid-lock/
May 30 2012
prev sibling next sibling parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Tue, 29 May 2012 00:36:13 +0200, Alex R=C3=B8nne Petersen <alex lycus=
.org>  =

wrote:

 Hi,

 I've seen several occurrences of synchronized (this) and synchronized =

 (this.classinfo) in both druntime and phobos by now. I propose that we=

 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happe=

 to use the object for locking will most likely end up with a deadlock =

 their hands.

 2) Locking on something as fundamental as type info means that any  =

 arbitrary part of the application could cause a deadlock by doing the =

 same.

= mutex. It's a little ugly but I fail to see fundamental issues. synchronized(Stacktrace.classinfo) synchronized(typeid(ArrayAllocLengthLock)) // uses an exclusive type as = = named mutex synchronized(TaskPool.classinfo) synchronized(this.classinfo) // this a.k.a. Socket Then there is std.concurrency.registryLock which is used in the exact sa= me = ways as the occurences above. To do that it requires 6 lines of code, ha= s = to successfully avoid 2 pitfalls and allocates the mutex eagerly. Then there is core.thread which has a lot of issues with locking. There is some suspicious code in object.rt_attachDisposeEvent which synchronizes on it's argument. I'm inclined to say that this is a very thin backing for such a rant. There was one interesting argument by Regan Heath:
 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.

 The problem /is/ that synchronized classes/methods use a mutex which i=

 exposed publicly, and it the same mutex as used by synchronized(object=

 {}.  This exposure/re-use makes deadlocks more likely to happen, and  =

 harder to spot.

Now I do not see anyone synchronizing it's code on arbitrary objects or = any other kind of abuse that would increase the inherent possibility for = deadlocks.
May 30 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 20:26, Martin Nowak wrote:
 On Tue, 29 May 2012 00:36:13 +0200, Alex Rønne Petersen
 <alex lycus..org> wrote:

 Hi,

 I've seen several occurrences of synchronized (this) and synchronized
 (this.classinfo) in both druntime and phobos by now. I propose that we
 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who
 happens to use the object for locking will most likely end up with a
 deadlock on their hands.

 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the
 same.

mutex. It's a little ugly but I fail to see fundamental issues. synchronized(Stacktrace.classinfo) synchronized(typeid(ArrayAllocLengthLock)) // uses an exclusive type as named mutex synchronized(TaskPool.classinfo)

And what happens if I synchronize on TaskPool.classinfo in my code? The point here is that deadlocks *can* happen. If we went by the logic that "it's not likely to go wrong, so let's not care", why do we have slices? scope statements? exception handling?
 synchronized(this.classinfo) // this a.k.a. Socket

 Then there is std.concurrency.registryLock which is used in the exact
 same ways as the occurences above. To do that it requires 6 lines of
 code, has to successfully avoid 2 pitfalls and allocates the mutex eagerly.

 Then there is core.thread which has a lot of issues with locking.

What issues specifically...?
 There is some suspicious code in object.rt_attachDisposeEvent
 which synchronizes on it's argument.

 I'm inclined to say that this is a very thin backing for such a rant.


 There was one interesting argument by Regan Heath:

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.

 The problem /is/ that synchronized classes/methods use a mutex which
 is exposed publicly, and it the same mutex as used by
 synchronized(object) {}. This exposure/re-use makes deadlocks more
 likely to happen, and harder to spot.

Now I do not see anyone synchronizing it's code on arbitrary objects or any other kind of abuse that would increase the inherent possibility for deadlocks.

As I pointed out in another post in this thread, I've seen the pattern used several times by people asking questions on media such as IRC. IRC has much higher traffic than e.g. D.learn. -- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
prev sibling next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 30-05-2012 20:26, Martin Nowak wrote:
 On Tue, 29 May 2012 00:36:13 +0200, Alex Rønne Petersen
 <alex lycus..org> wrote:

 Hi,

 I've seen several occurrences of synchronized (this) and synchronized
 (this.classinfo) in both druntime and phobos by now. I propose that we
 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who
 happens to use the object for locking will most likely end up with a
 deadlock on their hands.

 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the
 same.

mutex. It's a little ugly but I fail to see fundamental issues. synchronized(Stacktrace.classinfo) synchronized(typeid(ArrayAllocLengthLock)) // uses an exclusive type as named mutex synchronized(TaskPool.classinfo) synchronized(this.classinfo) // this a.k.a. Socket Then there is std.concurrency.registryLock which is used in the exact same ways as the occurences above. To do that it requires 6 lines of code, has to successfully avoid 2 pitfalls and allocates the mutex eagerly. Then there is core.thread which has a lot of issues with locking. There is some suspicious code in object.rt_attachDisposeEvent which synchronizes on it's argument. I'm inclined to say that this is a very thin backing for such a rant.

Also, I don't think calling this a rant is fair. I gave specific reasons for why the patterns I mentioned are bad in _libraries_ of all things (notice that I was talking about druntime and phobos!).
 There was one interesting argument by Regan Heath:

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.

 The problem /is/ that synchronized classes/methods use a mutex which
 is exposed publicly, and it the same mutex as used by
 synchronized(object) {}. This exposure/re-use makes deadlocks more
 likely to happen, and harder to spot.

Now I do not see anyone synchronizing it's code on arbitrary objects or any other kind of abuse that would increase the inherent possibility for deadlocks.

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 30 2012
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
 And what happens if I synchronize on TaskPool.classinfo in my code?

extended so why would you do that. Likewise the C# article [1] said it'd be a bad idea to lock a literal string and the liquid-lock bug [2] just locks nearby data to synchronize it's code. That this should be a common source of bugs strikes me as rather odd, but then one could already wonder why 'lock' is used intransitively. Saying 'lock on sth.' when you actually want to lock sth. (data or resource) just pinpoints the underlying issue. For example the analysis in [2] misses the actual point of the bug. He wanted to achieve mutual exclusive usage of a channel but instead of locking the channel he locked a message. [1] http://msdn.microsoft.com/en-us/library/ms173179.aspx [2] http://schneide.wordpress.com/tag/liquid-lock/ For me this whole discussion boils down to the following. Locks are quite safe if you use them to actually protect a resource. Locks are very unsafe if you use them as part of an implicit protocol.
May 30 2012
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Wed, 30 May 2012 20:45:51 +0200, Alex R=C3=B8nne Petersen <alex lycus=
.org>  =

wrote:

 Also, I don't think calling this a rant is fair. I gave specific reaso=

 for why the patterns I mentioned are bad in _libraries_ of all things =

 (notice that I was talking about druntime and phobos!).

Sorry, I didn't meant to be harsh. But please reread your initial post. It's important to emphasize the specific reasons in an objective way to start off a solid discussion. Words like extreme, horrible, mess, ban, blame or fault are counterproductive.
May 30 2012
prev sibling next sibling parent "Jacob Carlborg" <doob me.com> writes:
On Wednesday, 30 May 2012 at 15:45:05 UTC, Andrei Alexandrescu 
wrote:
 On 5/30/12 2:14 AM, Jacob Carlborg wrote:
 It seems more and more that D2 is not a designed language. 
 Instead new
 features are just slapped on without considering how it would 
 impact the
 rest of the language.

What features are you referring to?

The concurrency model as this thread shows. There is no bridge between shared and unshared data like const is to immutable and mutable. Perhaps the monitor on every object should have been removed when the new concurrency model was designed, as this thread suggests. "inout" was added long after the const system was added to D. If done correctly this should have come up as a problem when designing the const system. You cannot apply const/immutable to an object reference in the same way as you can to a pointer. It took some pretty good convincing for you to accept that strings weren't enough as a substitute for lambdas. Fortunately we have proper lambdas now :) "const" doesn't play nice with ranges. All these points have been mentioned before. -- /Jacob Carlborg
May 30 2012
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

I don't think so. Synchronized classes are the unit of scoped locking in D. If you want to do all scoped locking internally, make the class private.

Maybe you didn't read thoroughly the first part of my post (the example that shows a deadlock that is easily preventable if the mutex isn't exposed). The issue is that it's possible to hijack the mutex that you are using to protect the class data in order to protect *external* data. Hijacking is not possible if the mutex is private (i.e. a private member). So great! Just don't use the builtin object mutex, because it exposes possible race conditions. But then why do we have it (the object hidden mutex)? And now I can't use synchronized classes to ensure the whole thing is protected, I have to write all my functions like this: void foo() { synchronized(this.mutex) // lock a private mutex { } } -Steve
May 30 2012
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, May 30, 2012 12:12:15 Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 
 <SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

I don't think so. Synchronized classes are the unit of scoped locking in D. If you want to do all scoped locking internally, make the class private.

Maybe you didn't read thoroughly the first part of my post (the example that shows a deadlock that is easily preventable if the mutex isn't exposed).

The mutex is not exposed.

No, you can't directly access the mutex externally, but a synchronized block used on a synchronized object uses the same mutex that the object does. So, if you have synchronized class C { } then this synchronized block C c; synchronized(c) { } will use the exact same mutex that any functions on C itself use. So, it's possible for code outside of your class to lock that mutex using a synchronized statement. They do _not_ have direct access to the mutex to do whatever they want with it, but they _can_ lock via synchronized statements, which means that the synchronized class does _not_ have total control over what code locks its mutex. All that's required to fix this particular problem is to make it illegal to use a synchronized block on a synchronized object - either that or give synchronized object's two mutexes (one to use internally for its functions and one to be locked externally using synchronized blocks). The doesn't necessarily solve all of the issues being discussed here, but it _does_ solve the issue of a synchronized class not being the only entity with the ability to lock its mutex. - Jonathan M Davis
May 30 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 To clarify:

 On 5/30/12 12:25 PM, Alex R=C3=B8nne Petersen wrote:
  The mutex may not be
 directly exposed in the sense that you can obtain a reference (though=


 reality in all compiler implementations, you can), but it is exposed =


 the sense that you can lock and unlock it, which is a mutation of sta=


 and program flow.

synchronized (object) { writeln("about to unlock the object"); XXX writeln("unlocked the object"); } Replace "XXX" with a construct that unlocks the object.

This is not what we are talking about. I guess the easiest way to say it is, the API used to "lock and then = subsequently unlock" the mutex. That is, even this: Object o =3D new Object; synchronized(o) { } is exposing the mutex in such a way that you can interfere with the = semantic meaning of the mutex, and cause preventable deadlocks. It would be preferrable for: synchronized(o) { } to be an error, unless typeof(o) allows it explicitly. I gave you a case where you can have a deadlock, even with simple fully = = synchronized classes. You might say, "yeah, but all mutexes can have = deadlocks!". My contention is that if the default is you do *not* expose the mutex to= = external callers, there *cannot* be a deadlock. Here is the example again: synchronized class A { private int _foo; property int foo() { return _foo;} } synchronized class B { private A _a; property A a() { return _a;} void fun() {} } void thread(B b) { synchronized(b.a) { b.fun(); } } If synchronized(b.a) is valid, deadlock can occur. If it's invalid, = deadlock *cannot* occur. -Steve
May 30 2012
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
 FWIW, I recently came across the term here:
 http://schneide.wordpress.com/tag/liquid-lock/

I explained that in another post a few minutes ago, and yes, this is i=

 Maybe i should write an article on that, I though it was more well kno=

 The main problem here is the object the lock is acquired upon: the  =

 reference of lastMessage is mutable! We call this a liquid lock, becau=

 the lock isn=E2=80=99t as solid as it should be. It=E2=80=99s one of t=

 multithreading pitfalls as it looks like everything=E2=80=99s fine at =

 glance.

You should try to name the real root of the bug. It's trying to serialize access to a channel by locking the messages. There is nothing fancy about this, it has nothing to do with mutability it's just about locking the wrong resource.
May 30 2012
prev sibling next sibling parent "foobar" <foo bar.com> writes:
On Thursday, 31 May 2012 at 08:01:14 UTC, Andrei Alexandrescu 
wrote:
 On 5/30/12 11:47 PM, Jacob Carlborg wrote:
 On 2012-05-30 21:10, Andrei Alexandrescu wrote:

 I see how these can be annoying, but they're not the result 
 of us not
 designing things. We designed things best we could.

I would say it's not good enough. The whole approach of designing the language is wrong.

I understand how frustrating this is. In fact even the way you consider "good" is not nearly good enough. What we need is really more formalization of the language design, something that we're sorely missing. I am sometimes frustrated out of my mind at the lack of rigor and discipline in the process. On the other hand, we march with the troops we have. Andrei

Please no. This is how C++ is designed and we all know how fucked up that is. Writing a [rigorous] spec is almost always incorrect since requirements change and unforeseen things come about. Jacob's post illustrates this when the spec is written [in TDPL] before implementing, testing and integrating it. By making a rigorous spec you exacerbate the problem - it takes more time to write such a spec thus making the time-frame for unforeseen changes larger.
May 31 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 10:40 AM, Regan Heath wrote:
 On Wed, 30 May 2012 18:16:38 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 9:43 AM, Regan Heath wrote:
 On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>
 wrote:
 You don't want to synchronize on ANY object. You want to  
 synchronize
 on explicit mutexes.

+1 .. this is the key point/issue.

TDPL's design only allows for entire synchronized classes (not separate synchronized and unsynchronized methods), which pair mutexes with the data they protect. This is more restrictive than exposing mutexes, but in a good way. We use such a library artifact in C++ at Facebook all the time, to great success.

Can you call pass them to a synchronized statement? i.e. TDPLStyleSynchClass a = new TDPLStyleSynchClass(); synchronized(a) { }

Yes. Well I recommend acquiring the text! :o)
 ... because, if you can, then you're exposing the mutex.

No.

For the purposes of this thread, and the anti-pattern/problem we're discussing, you are.

No. I explained in my previous post that the synchronized statement does not expose locks. This is not a matter of opinion.

Well, it seems some people disagree here. You are exposing the ability to lock and unlock them, which is the cause of more frequent deadlocks (this assertion comes from the M$ article, my experience with them and it seems from other peoples experience of them as well). You're not exposing the mutex primitive in any other way, but that's irrelevant for this discussion. Now, you and I have probably done enough multi-threaded programming with mutexes that we no longer fall into this trap easily, but someone just starting in the world of threading/locking is going to, repeatedly. If we make it impossible to do it "by default", so they have to think about when/where to apply the mutex, and make the code doing the locking more obvious, then I think we'd go a long way to making it much nicer.
 It is the combination of synchronized
 classes/methods (implicit locking) and external synchronized statements
 (explicit locking) which result in the unexpected, accidental, and hard
 to see deadlocks we're talking about here.

You can have deadlocks but with synchronized you can't leak locks or doubly-unlock them. With free mutexes you have all of the above.

I'm not suggesting using free mutexes. I'm suggesting keeping the mutex private inside the object. Private in this case means cannot be locked by external code. Which basically means synchronized() is a no go "by default". This will avoid all the careless accidental deadlock cases involving synchronized w/ synchronized classes/methods and make people actually think about when/where to use their mutexes, instead of just throwing synchronized all about the place. It is sad, because I really like the scopedness of synchronized but I think it encourages lazy programming in this case. Note that I said "by default" above, we can definitely provide library interfaces, structs, classes, etc to make synchronized work and if we did it would become a conscious choice the programmer makes to use them, and that should hopefully (because I cannot predict the future) mean less accidental deadlocks.
 People shouldn't create designs that have synchronized classes
 referring to one another naively. Designing with mutexes (explicit or
 implicit) will always create the possibility of deadlock, so examples
 how that could happen are easy to come across.

True. But in my Example 1

Your example 1 should not compile.

o.. k.. I expected you would get my meaning with the simplified example. Here you are:

[snip]
 Runs and deadlocks immediately.

Sure. As I said, synchronized helps with scoping the locks and unlocks, but not with deadlocks. You can rewrite the example with two bare mutexes just as well.

Again, I'm not suggesting using "bare" or "free" mutexes. I'm suggesting keeping the mutex private, so that it cannot be locked without it being obvious that that's whats happening. If it were more obvious there would be less chance of an accidental deadlock, and more chance people would actually use an idea like an "atomic lock of multiple sync primitives" (your synchronized with multiple mutexes and lock ordering), where possible. Sadly even that idea only works when both locks are taken in the same method/function and only if they can and should be taken at the same time .. which I think is a fairly rare case. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
May 31 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 30 May 2012 20:46:17 +0100, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 12:25 PM, Alex R=F8nne Petersen wrote:
 On 30-05-2012 21:12, Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead=






 recommending not using a feature of the language?

I don't think so. Synchronized classes are the unit of scoped lock=





 in D. If you want to do all scoped locking internally, make the cl=





 private.

Maybe you didn't read thoroughly the first part of my post (the =




 example
 that shows a deadlock that is easily preventable if the mutex isn't=




 exposed).

The mutex is not exposed.

I'm trying really hard to not to be rather impolite here, but I don't=


 know how else to put it: You seem to be the only one who subscribes t=


 this definition of "exposed".

I don't care as long as it's the correct one. The mutex is not exposed=

Ok, how about you tell us what term you want to use for the current stat= e = of affairs then, because arguing about this is pointless and I'm happy t= o = use whatever term you deem appropriate (because I couldn't care less wha= t = the term is - as long as we all know what is meant, which I believe is t= he = case here). R -- = Using Opera's revolutionary email client: http://www.opera.com/mail/
May 31 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On May 31, 2012, at 2:48 AM, Andrei Alexandrescu <SeeWebsiteForEmail erdani.=
org> wrote:

 On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

I'm not suggesting using free mutexes. I'm suggesting keeping the mutex private inside the object.

Ergo, you are suggesting using free mutexes. Your second sentence destroys=

To be fair: auto m =3D new Mutex; synchronized (m) {...} Free mutexes but still safe. Scope guards obviously work too. That said, I t= hink the point of contention here is that because synchronized can take an a= rbitrary object, it's possible to lock on a class for stuff completely unrel= ated to what the class does internally. It's certainly bad form though, and I= don't know a good way to prevent it without crippling synchronized.=20=
May 31 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 31 May 2012 11:39:06 -0400, Sean Kelly <sean invisibleduck.org>  
wrote:

 On May 31, 2012, at 2:48 AM, Andrei Alexandrescu  
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

I'm not suggesting using free mutexes. I'm suggesting keeping the mutex private inside the object.

Ergo, you are suggesting using free mutexes. Your second sentence destroys the first.

To be fair: auto m = new Mutex; synchronized (m) {...} Free mutexes but still safe. Scope guards obviously work too. That said, I think the point of contention here is that because synchronized can take an arbitrary object, it's possible to lock on a class for stuff completely unrelated to what the class does internally. It's certainly bad form though, and I don't know a good way to prevent it without crippling synchronized.

See my suggestion and Dmitry's in other parts of this thread. The idea is that you still allow synchronized(x), but only if the author of x allows it. If we use lowering, and make the lock and unlock methods of some entity, then you can also use attributes to limit scope of synchronized. For example, if we make synchronized(x) equivalent to calling x.lock() and x.unlock() correctly, if lock() and unlock() are private, then you can only call synchronized(x) from inside the instance. -Steve
May 31 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 31 May 2012 12:54:07 -0400, Regan Heath <regan netmail.co.nz>  
wrote:

 On Thu, 31 May 2012 16:23:29 +0100, Dmitry Olshansky  
 <dmitry.olsh gmail.com> wrote:

 On 31.05.2012 19:11, Steven Schveighoffer wrote:
 On Thu, 31 May 2012 10:49:52 -0400, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your
 ideas quite favorably.

[snip]
 Does it makes sense?

Everything but the ordering. I like the idea of ordering the locks based on an intrinsic value, but opCmp isn't it. Objects can mutate, and opCmp may produce different results. Imagine: synchronized(a, b) // at this point a < b { a.makeGreaterThan(b); assert(a > b); }

No way! I live in world where victim's hand is cut off as a punishment for mutating a lock after creation ;)

This is related to the "liquid lock" problem raised/mentioned elsewhere in the thread. If the reference you lock can change, then one thread/iteration may lock one instance, the lock instance mutates, and a 2nd thread/iteration locks the new instance and both execute more or less in parallel over code which was supposed to be serialized. It's not a problem if the lock object is the object being mutated/used by the code, but it is a problem if the lock object is protecting other shared objects. It's not something I have ever bumped into myself, because the bulk of my locking experience is in C/C++ and I use a locking primitive which is initialised once and never mutates.

Well, I thought the original proposal was to use the opCmp of the *data*, i.e. the object being protected. At least that's what it seems like from the example. Apologies if this was not the case.
 You *never* want the ordering to change.

 But I think we can probably work that out. What about comparing handles
 of the mutexes? So you sort based on some property __mutex_id() which
 must return a unique size_t that can be used to always define an
 ordering of locking mutexes? Most mutexes are allocated by the OS, and
 so their handles won't be affected by a moving GC, so you likely will
 use the handle value.

This could work. In fact I imagined comparing handles ... As with math at large you may either project (biject) your domain to a well-known one (like numbers, mutex_id in your case) and work with it or define all relevant properties for whatever your values in this domain are (providing custom ordering for user defined types).

This ordering idea is present in TDPL.. have you both read the link Andrei posted? http://goo.gl/ZhPM2 (see 13.15)

No, I haven't read it yet. I did review an early version of TDPL, but I have not re-read the released version (though I do have a copy). Whether it was in there or not, it's been a while :) -Steve
May 31 2012
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 01 Jun 2012 15:34:20 +0100, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Fri, 01 Jun 2012 10:09:01 -0400, deadalnix <deadalnix gmail.com>  
 wrote:


 I think it is unrealistic to prevent all deadlock, unless you can come  
 up with a radically new approach.

 It is still possible to provide interface that prevent common traps.

Right, the idea is not to exterminate all deadlock, it's to *make it possible to* prevent deadlock. Currently, it's not possible unless you avoid using synchronized(this).

What he ^ said :) R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Jun 01 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 06/01/12 14:26, deadalnix wrote:
 Le 31/05/2012 20:17, Andrei Alexandrescu a écrit :
 On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

It can also be a lot clunkier for certain abstractions. Say I want a ProducerConsumerQueue. It's much more convenient to simply make it a synchronized class with the classic primitives, instead of primitives that accept delegates etc. Nevertheless I think there's merit in this idea. One thing to point out is that the idiom can easily be done today with a regular class holding a synchronized class private member. So we got everything we need. Andrei

I was thinking about that. Here is what I ended up to think is the best solution : synchronized classes exists. By default, they can't be use as parameter for synchronized(something) . synchronized(something) will be valid is something provide opSynchronized(scope delegate void()) or something similar. Think opApply here. The synchronized statement is rewritten in a call to that delegate.

This has similar issues as opApply. It would have to be a template and always inlined; The opLock/opUnlock approach lets you do the same things w/o the delegate overhead and signature restrictions while making it a bit harder to screw up the locking.
 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {
 
 }
 
 synchronized(rw.write) {
 
 }
 
 And many types of lock : spin lock, interprocesses locks, semaphores, . . .
And all can be used with the synchronized syntax, and without exposing locking
and unlocking primitives.
 
 What do people think ?

It can already be done using 'synchronized', it's *only* an issue of efficiency and syntax. Eg right now i'm doing { scope s = somesemaphore.sync; whatever(); } and a properly lowered 'synchronized' would turn that into synchronized (somesemaphore) { whatever(); } Currently, the latter is probably possible, but not w/o a huge perf hit; the former is practically free, just as if it was written as: somesemaphore.wait(); try whatever(); finally somesemaphore.post(); Lowering 'synchronized' is about making the second form possible, for anything resembling some kind of synchronization primitive. OpSynchronized isn't necessary. artur
Jun 01 2012
prev sibling next sibling parent Andrew Wiley <wiley.andrew.j gmail.com> writes:
--bcaec5523980dbd79704c19a442e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com> wrote:

 Le 03/06/2012 21:40, Andrew Wiley a =E9crit :

 On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

    Le 01/06/2012 22:55, Sean Kelly a =E9crit :

        On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


            The main drawback is the same as opApply : return (and
            break/continue but it is less relevant for opSynchronized).
            Solution to this problem have been proposed in the past
            using compiler and stack magic.

            It open door for stuff like :
            ReadWriteLock rw;
            synchronized(rw.read) {

            }

            synchronized(rw.write) {

            }


        Opens the door?  This works today exactly as outlined above.  Or
        am I missing a part of your argument?

            And many types of lock : spin lock, interprocesses locks,
            semaphores, . . . And all can be used with the synchronized
            syntax, and without exposing locking and unlocking primitives=


        All works today.


    Unless you do some monitor magic, it doesn't.

 Yes, it does.
 -----
 class Something {
     private:
         ReadWriteLock _rw;
     public:
         this() {
             _rw =3D new ReadWriteLock();
         }
         void doSomething() shared {
             synchronized(_rw.read) {
                 // do things
             }
         }
 }
 -----
 I've used this pattern in code. There might be some casting required
 because the core synchronization primitives haven't been updated to use
 shared yet.

And where is that ReadWriteLock ?

On the GC heap, just like the Monitor object pointed to by __monitor if you mark a method or class as synchronized. --bcaec5523980dbd79704c19a442e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <span = dir=3D"ltr">&lt;<a href=3D"mailto:deadalnix gmail.com" target=3D"_blank">de= adalnix gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote= " style=3D"margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(= 204,204,204);border-left-width:1px;border-left-style:solid"> Le 03/06/2012 21:40, Andrew Wiley a =E9crit :<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;padding= -left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-l= eft-style:solid"><div class=3D"im"> On Sun, Jun 3, 2012 at 12:29 PM, deadalnix &lt;<a href=3D"mailto:deadalnix = gmail.com" target=3D"_blank">deadalnix gmail.com</a><br></div><div><div cla= ss=3D"h5"> &lt;mailto:<a href=3D"mailto:deadalnix gmail.com" target=3D"_blank">deadaln= ix gmail.com</a>&gt;&gt; wrote:<br> <br> =A0 =A0Le 01/06/2012 22:55, Sean Kelly a =E9crit :<br> <br> =A0 =A0 =A0 =A0On Jun 1, 2012, at 5:26 AM, deadalnix wrote:<br> <br> <br> =A0 =A0 =A0 =A0 =A0 =A0The main drawback is the same as opApply : return (= and<br> =A0 =A0 =A0 =A0 =A0 =A0break/continue but it is less relevant for opSynchr= onized).<br> =A0 =A0 =A0 =A0 =A0 =A0Solution to this problem have been proposed in the = past<br> =A0 =A0 =A0 =A0 =A0 =A0using compiler and stack magic.<br> <br> =A0 =A0 =A0 =A0 =A0 =A0It open door for stuff like :<br> =A0 =A0 =A0 =A0 =A0 =A0ReadWriteLock rw;<br> =A0 =A0 =A0 =A0 =A0 =A0synchronized(rw.read) {<br> <br> =A0 =A0 =A0 =A0 =A0 =A0}<br> <br> =A0 =A0 =A0 =A0 =A0 =A0synchronized(rw.write) {<br> <br> =A0 =A0 =A0 =A0 =A0 =A0}<br> <br> <br> =A0 =A0 =A0 =A0Opens the door? =A0This works today exactly as outlined abo= ve. =A0Or<br> =A0 =A0 =A0 =A0am I missing a part of your argument?<br> <br> =A0 =A0 =A0 =A0 =A0 =A0And many types of lock : spin lock, interprocesses = locks,<br> =A0 =A0 =A0 =A0 =A0 =A0semaphores, . . . And all can be used with the sync= hronized<br> =A0 =A0 =A0 =A0 =A0 =A0syntax, and without exposing locking and unlocking = primitives.<br> <br> <br> =A0 =A0 =A0 =A0All works today.<br> <br> <br> =A0 =A0Unless you do some monitor magic, it doesn&#39;t.<br> <br></div></div><div class=3D"im"> Yes, it does.<br> -----<br> class Something {<br> =A0 =A0 private:<br> =A0 =A0 =A0 =A0 ReadWriteLock _rw;<br> =A0 =A0 public:<br> =A0 =A0 =A0 =A0 this() {<br> =A0 =A0 =A0 =A0 =A0 =A0 _rw =3D new ReadWriteLock();<br> =A0 =A0 =A0 =A0 }<br> =A0 =A0 =A0 =A0 void doSomething() shared {<br> =A0 =A0 =A0 =A0 =A0 =A0 synchronized(_rw.read) {<br> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 // do things<br> =A0 =A0 =A0 =A0 =A0 =A0 }<br> =A0 =A0 =A0 =A0 }<br> }<br> -----<br> I&#39;ve used this pattern in code. There might be some casting required<br=

<br> shared yet.<br> </div></blockquote> <br> And where is that ReadWriteLock ?<br> </blockquote></div><div>=A0</div><div>On the GC heap, just like the Monitor= object pointed to by __monitor if you mark a method or class as synchroniz= ed.<br></div> --bcaec5523980dbd79704c19a442e--
Jun 03 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, June 04, 2012 10:51:08 mta`chrono wrote:
 Am 31.05.2012 17:05, schrieb Regan Heath:
 .. but, hang on, can a thread actually lock a and then b?  If 'a' cannot
 participate in a synchronized statement (which it can't under this
 proposal) then no, there is no way to lock 'a' except by calling a
 member.  So, provided 'a' does not have a member which locks 'b' - were
 deadlock safe!
 
 So.. problem solved; by preventing external/public lock/unlock on a
 synchronized class.  (I think the proposal should enforce this
 restriction; synchronized classes cannot define __lock/__unlock).
 
 R

I think it doesn't matter whether you expose your mointor / locking / unlocking to the public or not. You can always unhappily create deadlocks that are hard to debug between tons of spaghetti code.

You can always create deadlocks, but if there's something which gives you little benefit but significantly increases the risk of deadlocks (e.g. making it easy to lock on a synchronized class' internal mutex via a synchronized block), then it's valuable to make it illegal. Because while it won't prevent all deadlocking, it _does_ eliminate one case where it's overly easy to deadlock. - Jonathan M Davis
Jun 04 2012
prev sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Tuesday, 29 May 2012 at 22:01:50 UTC, Alex Rønne Petersen 
wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:

It doesn't, and neither does C#. Java still encourages using synchronized, and C# still encourages using lock, but many prominent figures in those programming language communities have written blog posts on why these language constructs are evil and should be avoided. Besides, it seems to me that D can't quite make up its mind. We have TLS by default, and we encourage message-passing (through a library mechanism), and then we have the synchronized statement and attribute. It just seems so incredibly inconsistent. synchronized encourages doing the wrong thing (locks and synchronization).

I can't remember having read those blog posts. I agree message passing is a better abstraction, but I haven't read Josh Bloch, for example, saying "don't use synchronized".
Jun 06 2012