www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Should core.sync.mutex.Mutex, core.sync.condition.Condition,

reply Jonathan M Davis <jmdavisProg gmx.com> writes:
I don't know if we can answer this for sure at the moment given the ongoing 
discussion on shared, but looking at core.sync, it occurred to me that there's 
a major problem with the classes in there. None of the work with shared. And 
unless I'm missing something here, I don't see how many of them are even 
useful as anything other than shared. After all, what good is a mutex which is 
thread-local? But none of the methods on Mutex or its friends are shared.

So, the question is should the all have their methods shared? And if they 
should, is there any reason to have non-shared overloads for them? What good 
are they as anything other than shared? How is anyone using them right now?

- Jonathan M Davis
Nov 17 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 18-11-2012 05:46, Jonathan M Davis wrote:
 I don't know if we can answer this for sure at the moment given the ongoing
 discussion on shared, but looking at core.sync, it occurred to me that there's
 a major problem with the classes in there. None of the work with shared. And
 unless I'm missing something here, I don't see how many of them are even
 useful as anything other than shared. After all, what good is a mutex which is
 thread-local? But none of the methods on Mutex or its friends are shared.

 So, the question is should the all have their methods shared? And if they
 should, is there any reason to have non-shared overloads for them? What good
 are they as anything other than shared? How is anyone using them right now?

 - Jonathan M Davis

Not at this point in time. It would break a ridiculous amount of code if we did this, given the current extremely annoying nature of shared. Most D code I have seen in the wild just shares mutexes, conditions, etc with __gshared or some other mechanism anyway, so I don't think there's anything to gain. Like, what would shared actually buy you here? -- Alex Rønne Petersen alex lycus.org http://lycus.org
Nov 17 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 18-11-2012 05:58, Jonathan M Davis wrote:
 On Sunday, November 18, 2012 05:51:00 Alex Rønne Petersen wrote:
 On 18-11-2012 05:46, Jonathan M Davis wrote:
 I don't know if we can answer this for sure at the moment given the
 ongoing
 discussion on shared, but looking at core.sync, it occurred to me that
 there's a major problem with the classes in there. None of the work with
 shared. And unless I'm missing something here, I don't see how many of
 them are even useful as anything other than shared. After all, what good
 is a mutex which is thread-local? But none of the methods on Mutex or its
 friends are shared.

 So, the question is should the all have their methods shared? And if they
 should, is there any reason to have non-shared overloads for them? What
 good are they as anything other than shared? How is anyone using them
 right now?

 - Jonathan M Davis

Not at this point in time. It would break a ridiculous amount of code if we did this, given the current extremely annoying nature of shared. Most D code I have seen in the wild just shares mutexes, conditions, etc with __gshared or some other mechanism anyway, so I don't think there's anything to gain. Like, what would shared actually buy you here?

__gshared is a good reason for leaving non-shared overloads, but isn't code really supposed to be using shared and not __gshared unless it's specifically extern(C)? That being the case, I'd expect shared to be the correct thing to use with mutexes normally, and right now, that won't work without a ton of casting or adding shared overloads. - Jonathan M Davis

I don't know what anything that has to do with shared is supposed or not supposed to do. It is not clear, even to the primary language designers, what shared actually is, so I really can't say anything to this... All I can say is, let's wait with this until we know what shared is going to actually do. We don't want to do something we'll regret later. -- Alex Rønne Petersen alex lycus.org http://lycus.org
Nov 17 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11/18/2012 11:13 AM, Alex Rønne Petersen пишет:
 On 18-11-2012 05:58, Jonathan M Davis wrote:
 On Sunday, November 18, 2012 05:51:00 Alex Rønne Petersen wrote:
 On 18-11-2012 05:46, Jonathan M Davis wrote:
 I don't know if we can answer this for sure at the moment given the
 ongoing
 discussion on shared, but looking at core.sync, it occurred to me that
 there's a major problem with the classes in there. None of the work
 with
 shared. And unless I'm missing something here, I don't see how many of
 them are even useful as anything other than shared. After all, what
 good
 is a mutex which is thread-local? But none of the methods on Mutex
 or its
 friends are shared.

 So, the question is should the all have their methods shared? And if
 they
 should, is there any reason to have non-shared overloads for them? What
 good are they as anything other than shared? How is anyone using them
 right now?

 - Jonathan M Davis

Not at this point in time. It would break a ridiculous amount of code if we did this, given the current extremely annoying nature of shared. Most D code I have seen in the wild just shares mutexes, conditions, etc with __gshared or some other mechanism anyway, so I don't think there's anything to gain. Like, what would shared actually buy you here?

__gshared is a good reason for leaving non-shared overloads, but isn't code really supposed to be using shared and not __gshared unless it's specifically extern(C)? That being the case, I'd expect shared to be the correct thing to use with mutexes normally, and right now, that won't work without a ton of casting or adding shared overloads. - Jonathan M Davis

I don't know what anything that has to do with shared is supposed or not supposed to do. It is not clear, even to the primary language designers, what shared actually is, so I really can't say anything to this...

Let's please stop this mantra. It buy us nothing but FUD. One thing is clear - shared is not going away over night. Yes, having auto-magic atomic ops for various builtins is discussed, the kind of memory model is discussed. Everything else stays. It's the globally shared data and it has tight restrictions on what you can do with it. As such it is (just like immutable) is safely passable between threads.
 All I can say is, let's wait with this until we know what shared is
 going to actually do. We don't want to do something we'll regret later.

Obviously we know what shared is. It's just certain details need straightening up. And certainly druntime/std.concurrency should add more support for it where meaningful. -- Dmitry Olshansky
Nov 18 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 18-11-2012 10:58, Dmitry Olshansky wrote:
 11/18/2012 11:13 AM, Alex Rønne Petersen пишет:
 On 18-11-2012 05:58, Jonathan M Davis wrote:
 On Sunday, November 18, 2012 05:51:00 Alex Rønne Petersen wrote:
 On 18-11-2012 05:46, Jonathan M Davis wrote:
 I don't know if we can answer this for sure at the moment given the
 ongoing
 discussion on shared, but looking at core.sync, it occurred to me that
 there's a major problem with the classes in there. None of the work
 with
 shared. And unless I'm missing something here, I don't see how many of
 them are even useful as anything other than shared. After all, what
 good
 is a mutex which is thread-local? But none of the methods on Mutex
 or its
 friends are shared.

 So, the question is should the all have their methods shared? And if
 they
 should, is there any reason to have non-shared overloads for them?
 What
 good are they as anything other than shared? How is anyone using them
 right now?

 - Jonathan M Davis

Not at this point in time. It would break a ridiculous amount of code if we did this, given the current extremely annoying nature of shared. Most D code I have seen in the wild just shares mutexes, conditions, etc with __gshared or some other mechanism anyway, so I don't think there's anything to gain. Like, what would shared actually buy you here?

__gshared is a good reason for leaving non-shared overloads, but isn't code really supposed to be using shared and not __gshared unless it's specifically extern(C)? That being the case, I'd expect shared to be the correct thing to use with mutexes normally, and right now, that won't work without a ton of casting or adding shared overloads. - Jonathan M Davis

I don't know what anything that has to do with shared is supposed or not supposed to do. It is not clear, even to the primary language designers, what shared actually is, so I really can't say anything to this...

Let's please stop this mantra. It buy us nothing but FUD. One thing is clear - shared is not going away over night. Yes, having auto-magic atomic ops for various builtins is discussed, the kind of memory model is discussed. Everything else stays. It's the globally shared data and it has tight restrictions on what you can do with it. As such it is (just like immutable) is safely passable between threads.

There have been proposals to a completely different meaning of shared that has nothing to do with memory models or atomic ops. There is no FUD here. Shoehorning shared in its current state into core.sync is just plain unreasonable given that nothing has been set in stone about the meaning of shared at all yet.
 All I can say is, let's wait with this until we know what shared is
 going to actually do. We don't want to do something we'll regret later.

Obviously we know what shared is. It's just certain details need straightening up. And certainly druntime/std.concurrency should add more support for it where meaningful.

We don't know at all what shared is. Have you been following the other thread? People have wildly different ideas of what it should do. I don't see why we should be in a hurry to add shared to core.sync in particular. Practically all of the runtime and standard library can't work properly with shared anyway, so it's not going to change a whole lot. I think it's much more sensible to wait until shared's semantics are set in stone and *then* start adapting the runtime and standard library. I can't stop anyone from just doing this, of course, but it seems like a premature change to me. -- Alex Rønne Petersen alex lycus.org http://lycus.org
Nov 18 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11/18/2012 10:36 PM, Alex Rønne Petersen пишет:
[snip]
 Everything else stays. It's the globally shared data and it has tight
 restrictions on what you can do with it. As such it is (just like
 immutable) is safely passable between threads.

There have been proposals to a completely different meaning of shared that has nothing to do with memory models or atomic ops. There is no FUD here. Shoehorning shared in its current state into core.sync is just plain unreasonable given that nothing has been set in stone about the meaning of shared at all yet.

I'd argue that shared on these objects has nothing to do with memory models and atomics. It's about global visibility and access to API that is interlocked on OS-level anyway. It works with any sane definition that doesn't forget to add that 'shared' implies global visibility.
 All I can say is, let's wait with this until we know what shared is
 going to actually do. We don't want to do something we'll regret later.

Obviously we know what shared is. It's just certain details need straightening up. And certainly druntime/std.concurrency should add more support for it where meaningful.

We don't know at all what shared is. Have you been following the other thread? People have wildly different ideas of what it should do.

I admit dismissing a lot of proposals from that thread on sight. The ones being about shared somehow 'just working' and getting you safe way to sharing and the one about auto-magically attaching mutexes to shared stuff. Given now that the thread is mostly spent. If there many good different concepts I'd appreciate if you list these here.
 I don't see why we should be in a hurry to add shared to core.sync in
 particular.

 Practically all of the runtime and standard library can't
 work properly with shared anyway, so it's not going to change a whole
 lot.

In part because even things that are supposed to always be shared in fact are not working as shared. And that's because nobody have found the time to go on and do this work. Add the fact that say class instances have monitor field when everything is TLS by default for me is indication of one thing only - during a move to TLS by default and shared a lot of things were not updated accordingly because of the lack of time and other issues.
 I think it's much more sensible to wait until shared's semantics
 are set in stone and *then* start adapting the runtime and standard
 library.

 I can't stop anyone from just doing this, of course, but it seems like a
 premature change to me.

sense only as a shared object? I mean the kind of change that can realistically can happen not some vague idea. -- Dmitry Olshansky
Nov 18 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
To separate an important notion I'm trying to express.

I assume that:

class X{
...
	shared foo(...);
	void bar(...);
...
}

shared X sharedStuff;
X localStuff;

Implies under any viable definition of shared that (including the 
current one and any pending clarifications):

a) foo can be called with sharedStuff in multiple threads at the same 
time. Thus it has to be thread safe (irregardless of possible 
 system/ trusted hackery inside).

b) localStuff on the other hand can called with  bar (and any non-shared 
methods) only in a single thread at any time. If localStuff is defined 
at global scope each thread has its own copy thus this requirement is 
fulfilled kind of automatically.

The major benefits of shared lie in cleaning up the opposite - unshared 
stuff. Having shared enables us to archieve great things we haven't 
looked at yet (maybe because of bugs/issues in the other departments). 
I'll name a few.

We can have lock-less and _safe_ file I/O  by default unlike e.g. C and 
pretty much everybody else. I should have said safe & _faster_  by 
default. This is why I believe D should stop relying on C's I/O as 
anything but fallback. We even had a partially complete design for new 
std.io that does it.

Same goes for memory allocations - lock-less thread-local allocators are 
possible _safely_ and by default (though it would have to go to the OS 
from time to time and likely to lock something here). It's not explored 
and remains as unused potential.

-- 
Dmitry Olshansky
Nov 18 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, November 18, 2012 05:51:00 Alex R=C3=B8nne Petersen wrote:
 On 18-11-2012 05:46, Jonathan M Davis wrote:
 I don't know if we can answer this for sure at the moment given the=


 ongoing
 discussion on shared, but looking at core.sync, it occurred to me t=


 there's a major problem with the classes in there. None of the work=


 shared. And unless I'm missing something here, I don't see how many=


 them are even useful as anything other than shared. After all, what=


 is a mutex which is thread-local? But none of the methods on Mutex =


 friends are shared.
=20
 So, the question is should the all have their methods shared? And i=


 should, is there any reason to have non-shared overloads for them? =


 good are they as anything other than shared? How is anyone using th=


 right now?
=20
 - Jonathan M Davis

Not at this point in time. It would break a ridiculous amount of code=

 we did this, given the current extremely annoying nature of shared.
=20
 Most D code I have seen in the wild just shares mutexes, conditions, =

 with __gshared or some other mechanism anyway, so I don't think there=

 anything to gain. Like, what would shared actually buy you here?

__gshared is a good reason for leaving non-shared overloads, but isn't = code=20 really supposed to be using shared and not __gshared unless it's specif= ically=20 extern(C)? That being the case, I'd expect shared to be the correct thi= ng to=20 use with mutexes normally, and right now, that won't work without a ton= of=20 casting or adding shared overloads. - Jonathan M Davis
Nov 17 2012
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11/18/2012 8:46 AM, Jonathan M Davis пишет:
 I don't know if we can answer this for sure at the moment given the ongoing
 discussion on shared, but looking at core.sync, it occurred to me that there's
 a major problem with the classes in there. None of the work with shared. And
 unless I'm missing something here, I don't see how many of them are even
 useful as anything other than shared. After all, what good is a mutex which is
 thread-local? But none of the methods on Mutex or its friends are shared.

None of them are useful without shared.They should work with shared, but that'll break code outright. I think 2 overloads per method with TLS versions going through the deprecation path.
 So, the question is should the all have their methods shared? And if they
 should, is there any reason to have non-shared overloads for them? What good
 are they as anything other than shared? How is anyone using them right now?

I do suspect that most people use _gshared with them as it is the less painful way. Another way is to have them shared and cast them to TLS as needed. Since mutex, condition etc. are opaque types around OS primitives TLS doesn't make sense at all but currently allows to call their methods. -- Dmitry Olshansky
Nov 18 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, November 18, 2012 23:28:03 Dmitry Olshansky wrote:
 What change in semantics of shared could stop mutex object from making
 sense only as a shared object? I mean the kind of change that can
 realistically can happen not some vague idea.

I thought that it was blindingly clear from TDPL and even the online spec what shared is for. It marks variables as being shared across threads rather than thread local. I don't see how that could possibly be changing. __gshared isn't even _mentioned_ in TDPL, and I think that it gets used as much as it does is entirely because of some of the current issues with shared's implementation in the compiler and the lack of support in druntime and Phobos. There are definitely questions as to what exactly should happen when a variable is marked as shared, but they have to do with what protections you get and what conversions are possible. They still don't fundamentally change the fact that the point of shared is to share an instance across threads rather than have it be thread local like everything is by default. So, I concur in that I don't see how the semantics of shared could possibly not be appropriate for mutexes. I started this thread primarily because I was shocked that Mutex, Condition, etc. weren't marked with shared already, and I didn't see how they could even be used without it (and apparently, the answer to that is that almost everyone cops out and uses __gshared). - Jonathan M Davis
Nov 18 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Nov 18, 2012, at 12:44 PM, Jonathan M Davis <jmdavisProg gmx.com> =
wrote:
=20
 So, I concur in that I don't see how the semantics of shared could =

 not be appropriate for mutexes. I started this thread primarily =

 shocked that Mutex, Condition, etc. weren't marked with shared =

 didn't see how they could even be used without it (and apparently, the =

 to that is that almost everyone cops out and uses __gshared).

I tried this once and it cascaded to requiring modifications of various = definitions on core.sys.posix to add a "shared" qualifier, and since I = wasn't ready to do that I rolled back the changes. I guess the = alternative would be to have a shared equivalent for every operation = that basically just casts away shared and then calls the non-shared = function, but that's such a terrible design I've been resisting it.=
Nov 18 2012
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, November 18, 2012 14:48:02 Sean Kelly wrote:
 On Nov 18, 2012, at 12:44 PM, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 So, I concur in that I don't see how the semantics of shared could
 possibly
 not be appropriate for mutexes. I started this thread primarily because I
 was shocked that Mutex, Condition, etc. weren't marked with shared
 already, and I didn't see how they could even be used without it (and
 apparently, the answer to that is that almost everyone cops out and uses
 __gshared).

I tried this once and it cascaded to requiring modifications of various definitions on core.sys.posix to add a "shared" qualifier, and since I wasn't ready to do that I rolled back the changes. I guess the alternative would be to have a shared equivalent for every operation that basically just casts away shared and then calls the non-shared function, but that's such a terrible design I've been resisting it.

Well, this is certainly going to need to be sorted out. And given that it's not unreasonable that both shared and __gshared be used with the type is core.sync, I don't think we have much choice to duplicate most of those functions (probably using casting internally to avoid actually duplicating the implementation in most cases) unless __gshared becomes implicitly convertible to shared. - Jonathan M Davis
Nov 20 2012