www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - valid uses of shared

reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
I am having a quite interesting debate on pure and shared with Artur  
Skawina in another thread, and I thought about how horrible a state shared  
is in.  It's not implemented as designed, and the design really leaves  
more questions than it has answers.  In addition, it has not real  
connection with thread synchronization whatsoever, and Michel Fortin  
suggested some improvements to synchronized that look really cool that  
would involve shared.

So I thought about, what are the truly valid uses of shared?  And then I  
thought, more importantly, what are the *invalid* uses of shared?

Because I think one of the biggest confusing pieces of shared is, we have  
no idea when I should use it, or how to use it.  So far, the only benefit  
I've seen from it is when you mark something as not shared, the things you  
can assume about it.

I think a couple usages of shared make very little sense:

1. having a shared piece of data on the stack.
2. shared value types.

1 makes little sense because a stack is a wholly-owned subsidiary of a  
thread.  Its existence depends completely on the stack frame staying  
around.  If I share a piece of my stack with another thread, then I return  
 from that function, I have just sent a dangling pointer over to the other  
thread.

2 makes little sense because when you pass around a value type, it's  
inherently not shared, you are making a copy!  What is the point of  
passing a shared int to another thread?  Might as well pass an int (this  
is one of the sticking points I have with pure functions accepting or  
dealing with shared data).

I have an idea that might fix *both* of these problems.

What if we disallowed declarations of shared type constructors on any  
value type?  So shared(int) x is an error, but shared(int)* x is not (and  
actually shared(int *) x is also an error, because the pointer is passed  
by value).  However, the type shared(int) is valid, it just can't be used  
to declare anything.

The only types that could be shared, would be:

ref shared T => local reference to shared data
shared(T) * => local pointer to shared data
shared(C) => local reference to shared class

And that's it.

The following would be illegal:

struct X
{
   shared int x; // illegal
   shared(int)* y; // legal

   shared(X) *next; // legal
}

shared class C  // legal, C is always a reference type
{
    shared int x; // illegal, but useless, since C is already shared
}

If you notice, I never allow shared values to be stored on the stack, they  
are always going to be stored on the heap.  We can use this to our  
advantage -- using special allocators that are specific to shared data, we  
can ensure the synchronization tools necessary to protect this data gets  
allocated on the heap along side it.  I'm not sure exactly how this could  
work, but I was thinking, instead of allocating a monitor based on the  
*type* (i.e. a class), you allocate it based on whether it's *shared* or  
not.  Since I can never create a shared struct X on the stack, it must be  
in the heap, so...

struct X
{
    int y;
}

shared(X) *x = new shared(X);

synchronized(x) // scope-locks hidden allocated monitor object
{
    x.y = 5;
}

x.y = 5; // should we disallow this, or maybe even auto-lock x?

Hm... another idea -- you can't extract any piece of an aggregate.  That  
is, it would be illegal to do:

shared(int)* myYptr = &x.y;

because that would work around the synchronization.

auto would have to strip shared:

auto myY = x.y; // typeof(myY) == int.

This is definitely not a complete proposal.  But I wonder if this is the  
right direction?

-Steve
Jun 07 2012
next sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 06/08/12 01:51, Steven Schveighoffer wrote:
 I am having a quite interesting debate on pure and shared with Artur Skawina
in another thread, and I thought about how horrible a state shared is in.  It's
not implemented as designed, and the design really leaves more questions than
it has answers.  In addition, it has not real connection with thread
synchronization whatsoever, and Michel Fortin suggested some improvements to
synchronized that look really cool that would involve shared.
 
 So I thought about, what are the truly valid uses of shared?  And then I
thought, more importantly, what are the *invalid* uses of shared?
 
 Because I think one of the biggest confusing pieces of shared is, we have no
idea when I should use it, or how to use it.  So far, the only benefit I've
seen from it is when you mark something as not shared, the things you can
assume about it.
 
 I think a couple usages of shared make very little sense:
 
 1. having a shared piece of data on the stack.
 2. shared value types.
 
 1 makes little sense because a stack is a wholly-owned subsidiary of a thread.
 Its existence depends completely on the stack frame staying around.  If I
share a piece of my stack with another thread, then I return from that
function, I have just sent a dangling pointer over to the other thread.
 
 2 makes little sense because when you pass around a value type, it's
inherently not shared, you are making a copy!  What is the point of passing a
shared int to another thread?  Might as well pass an int (this is one of the
sticking points I have with pure functions accepting or dealing with shared
data).
 
 I have an idea that might fix *both* of these problems.
 
 What if we disallowed declarations of shared type constructors on any value
type?  So shared(int) x is an error, but shared(int)* x is not (and actually
shared(int *) x is also an error, because the pointer is passed by value). 
However, the type shared(int) is valid, it just can't be used to declare
anything.
 
 The only types that could be shared, would be:
 
 ref shared T => local reference to shared data
 shared(T) * => local pointer to shared data
 shared(C) => local reference to shared class
 
 And that's it.
 
 The following would be illegal:
 
 struct X
 {
   shared int x; // illegal
   shared(int)* y; // legal
 
   shared(X) *next; // legal
 }

Note that the type of 'x' in shared struct S { int x; } should probably be 'shared(int)'. Which lets you safely take an address of an aggregates field. And I'm not sure if marking a struct and class as shared would work correctly right now, it's probably too easy to lose the 'shared' qualifier.
 shared class C  // legal, C is always a reference type
 {
    shared int x; // illegal, but useless, since C is already shared
 }

Redundant, but should be accepted, just like 'static' is.
 If you notice, I never allow shared values to be stored on the stack, they are
always going to be stored on the heap.  We can use this to our advantage --
using special allocators that are specific to shared data, we can ensure the
synchronization tools necessary to protect this data gets allocated on the heap
along side it.  I'm not sure exactly how this could work, but I was thinking,
instead of allocating a monitor based on the *type* (i.e. a class), you
allocate it based on whether it's *shared* or not.  Since I can never create a
shared struct X on the stack, it must be in the heap, so...
 
 struct X
 {
    int y;
 }
 
 shared(X) *x = new shared(X);
 
 synchronized(x) // scope-locks hidden allocated monitor object
 {
    x.y = 5;
 }
 
 x.y = 5; // should we disallow this, or maybe even auto-lock x?
 
 Hm... another idea -- you can't extract any piece of an aggregate.  That is,
it would be illegal to do:
 
 shared(int)* myYptr = &x.y;
 
 because that would work around the synchronization.

That's too restrictive. It would overload 'shared' even more. If you want that kind of synchronize magic to work, just allow: shared synchronized(optional_locking_primitive) struct S { ... } And *now* 'x.y = 5' can do its magic, while '&x.y' can be disallowed. shared struct S { Atomic!int i; } shared(S)* p = ... p.i += 1; should work, so accessing fields must remain possible.
 auto would have to strip shared:
 
 auto myY = x.y; // typeof(myY) == int.

Hmm, i'm not sure about this, maybe it should be disallowed; it could only strip 'shared' from the head anyway.
 This is definitely not a complete proposal.  But I wonder if this is the right
direction?

I think it is. artur
Jun 07 2012
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 08.06.2012 8:03, Steven Schveighoffer wrote:
 On Thu, 07 Jun 2012 20:58:13 -0400, Artur Skawina <art.08.09 gmail.com>
 wrote:

 On 06/08/12 01:51, Steven Schveighoffer wrote:
 The following would be illegal:

 struct X
 {
 shared int x; // illegal
 shared(int)* y; // legal

 shared(X) *next; // legal
 }

Note that the type of 'x' in shared struct S { int x; } should probably be 'shared(int)'. Which lets you safely take an address of an aggregates field.

That's one of the things I'm wondering about. Should it be allowed? I agree that the type should be shared(int), but the type should not transfer to function calls or auto, it should be sticky to the particular variable. Only references should be sticky typed.
 And I'm not sure if marking a struct and class as shared would work
 correctly right now, it's probably too easy to lose the 'shared'
 qualifier.

Right, I was thinking shared structs do not make sense, since I don't think shared members do not make sense. Either a whole struct/class is shared or it is not. Because you can only put classes on the heap, shared makes sense as an attribute for a class. But then again, it might make sense to say "this struct is only ever shared, so it should be required to go on the heap". I like your idea later about identifying shared struct types that should use synchronization.
 shared class C // legal, C is always a reference type
 {
 shared int x; // illegal, but useless, since C is already shared
 }

Redundant, but should be accepted, just like 'static' is.

That's probably fine.
 If you notice, I never allow shared values to be stored on the stack,
 they are always going to be stored on the heap. We can use this to
 our advantage -- using special allocators that are specific to shared
 data, we can ensure the synchronization tools necessary to protect
 this data gets allocated on the heap along side it. I'm not sure
 exactly how this could work, but I was thinking, instead of
 allocating a monitor based on the *type* (i.e. a class), you allocate
 it based on whether it's *shared* or not. Since I can never create a
 shared struct X on the stack, it must be in the heap, so...

 struct X
 {
 int y;
 }

 shared(X) *x = new shared(X);

 synchronized(x) // scope-locks hidden allocated monitor object
 {
 x.y = 5;
 }

 x.y = 5; // should we disallow this, or maybe even auto-lock x?

 Hm... another idea -- you can't extract any piece of an aggregate.
 That is, it would be illegal to do:

 shared(int)* myYptr = &x.y;

 because that would work around the synchronization.

That's too restrictive. It would overload 'shared' even more. If you want that kind of synchronize magic to work, just allow: shared synchronized(optional_locking_primitive) struct S { ... } And *now* 'x.y = 5' can do its magic, while '&x.y' can be disallowed. shared struct S { Atomic!int i; } shared(S)* p = ... p.i += 1; should work, so accessing fields must remain possible.

OK. You are right, synchronized may be overkill for basic types.
 auto would have to strip shared:

 auto myY = x.y; // typeof(myY) == int.

Hmm, i'm not sure about this, maybe it should be disallowed; it could only strip 'shared' from the head anyway.

Yes, head stripping. I think it should be allowed. For instance, if you wanted to read a shared double, and take the cosine of it, this should be allowed: auto n = cos(sharedValue);

what if it's a auto n = cos(sharedValues[k]); //type is shared(double) right? and somebody in the other thread is halfway through writing it?
 If it's not, the alternatives are:

 auto n = cos(cast()sharedValue);

 or

 double v = sharedValue; // explicit removal of shared
 auto m = cos(v);

 Neither of these look necessary, I think just allowing shared value
 types to automatically convert to non-shared versions works the best.

 This is definitely not a complete proposal. But I wonder if this is
 the right direction?

I think it is.

good.

Indeed. -- Dmitry Olshansky
Jun 08 2012
prev sibling next sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 06/08/12 06:03, Steven Schveighoffer wrote:
 On Thu, 07 Jun 2012 20:58:13 -0400, Artur Skawina <art.08.09 gmail.com> wrote:
 
 On 06/08/12 01:51, Steven Schveighoffer wrote:
 The following would be illegal:

 struct X
 {
   shared int x; // illegal
   shared(int)* y; // legal

   shared(X) *next; // legal
 }

Note that the type of 'x' in shared struct S { int x; } should probably be 'shared(int)'. Which lets you safely take an address of an aggregates field.

That's one of the things I'm wondering about. Should it be allowed?

Must be. We're talking about the next generation of a high level assembler, not logo. :)
 
 I agree that the type should be shared(int), but the type should not transfer
to function calls or auto, it should be sticky to the particular variable. 
Only references should be sticky typed.

The problem with this is that it should be symmetrical, IOW the conversion from non-shared to shared would also have to be (implicitly) allowed. A type that converts to both would be better, even if harder to implement.
 And I'm not sure if marking a struct and class as shared would work
 correctly right now, it's probably too easy to lose the 'shared' qualifier.

Right, I was thinking shared structs do not make sense, since I don't think shared members do not make sense. Either a whole struct/class is shared or it is not. Because you can only put classes on the heap, shared makes sense as an attribute for a class. But then again, it might make sense to say "this struct is only ever shared, so it should be required to go on the heap". I like your idea later about identifying shared struct types that should use synchronization.

Of course shared structs make sense, it's what allows implementing any non-trivial shared type. static Atomic!int counter; inside a function is perfectly fine. And, as somebody already mentioned in this thread, omitting 'static' should cause a build failure; right now it is accepted, even when written as shared Atomic!int counter; The problem? 'shared' is silently dropped. Move the counter from a struct into a function after realizing it's only accessed from one place, forget to add 'static' - and the result will compile w/o even a warning.
 
 If you notice, I never allow shared values to be stored on the stack, they are
always going to be stored on the heap.  We can use this to our advantage --
using special allocators that are specific to shared data, we can ensure the
synchronization tools necessary to protect this data gets allocated on the heap
along side it.  I'm not sure exactly how this could work, but I was thinking,
instead of allocating a monitor based on the *type* (i.e. a class), you
allocate it based on whether it's *shared* or not.  Since I can never create a
shared struct X on the stack, it must be in the heap, so...

 struct X
 {
    int y;
 }

 shared(X) *x = new shared(X);

 synchronized(x) // scope-locks hidden allocated monitor object
 {
    x.y = 5;
 }

 x.y = 5; // should we disallow this, or maybe even auto-lock x?

 Hm... another idea -- you can't extract any piece of an aggregate.  That is,
it would be illegal to do:

 shared(int)* myYptr = &x.y;

 because that would work around the synchronization.

That's too restrictive. It would overload 'shared' even more. If you want that kind of synchronize magic to work, just allow: shared synchronized(optional_locking_primitive) struct S { ... } And *now* 'x.y = 5' can do its magic, while '&x.y' can be disallowed. shared struct S { Atomic!int i; } shared(S)* p = ... p.i += 1; should work, so accessing fields must remain possible.

OK. You are right, synchronized may be overkill for basic types.
 auto would have to strip shared:

 auto myY = x.y; // typeof(myY) == int.

Hmm, i'm not sure about this, maybe it should be disallowed; it could only strip 'shared' from the head anyway.

Yes, head stripping. I think it should be allowed. For instance, if you wanted to read a shared double, and take the cosine of it, this should be allowed: auto n = cos(sharedValue); If it's not, the alternatives are: auto n = cos(cast()sharedValue); or double v = sharedValue; // explicit removal of shared auto m = cos(v); Neither of these look necessary, I think just allowing shared value types to automatically convert to non-shared versions works the best.

If 'shared(VT)' implicitly converts to VT, then auto myY = x.y; // typeof(myY) == shared(int) would still be fine. So would auto n = cos(x,y); // assuming some weird cos() that works on ints ;) But I'm not sure allowing these implicit conversions is a good idea. At least not yet. :)
 This is definitely not a complete proposal.  But I wonder if this is the right
direction?

I think it is.

good.

It's a small step in the right direction. artur
Jun 08 2012
next sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 06/08/12 19:13, Steven Schveighoffer wrote:
 On Fri, 08 Jun 2012 10:57:15 -0400, Artur Skawina <art.08.09 gmail.com> wrote:
 
 On 06/08/12 06:03, Steven Schveighoffer wrote:

 I agree that the type should be shared(int), but the type should not transfer
to function calls or auto, it should be sticky to the particular variable. 
Only references should be sticky typed.

The problem with this is that it should be symmetrical, IOW the conversion from non-shared to shared would also have to be (implicitly) allowed. A type that converts to both would be better, even if harder to implement.

It should be allowed (and is today).

Hmm. I think it shouldn't be. This is how it is today: shared Atomic!int ai; shared Atomic!(void*) ap; void f(Atomic!int i) {} // Atomic() struct template temporarily made unshared for this test. void fp(Atomic!(void*) i) {} void main() { f(ai); fp(ap); } Error: function f (Atomic!(int) i) is not callable using argument types (shared(Atomic!(int))) Error: cannot implicitly convert expression (ai) of type shared(Atomic!(int)) to Atomic!(int) Error: function fp (Atomic!(void*) i) is not callable using argument types (shared(Atomic!(void*))) Error: cannot implicitly convert expression (ap) of type shared(Atomic!(void*)) to Atomic!(void*) It seems to work for built-in value types, which i didn't even realize, because the thought of using them with 'shared' never crossed my mind. I don't really see why those should be treated differently from user defined types, which should not allow implicit shared<>unshared conversions.
 I am talking about stripping head-shared, so shared(int *) automatically
converts to shared(int)* when used as an rvalue.

Where would the 'shared(int*) type come from? IOW, given 'shared struct S { int i; } S s;' what would the type of '&s.i' be? In your model; because right now it is 'shared(int)*'.
 Right, I was thinking shared structs do not make sense, since I don't think
shared members do not make sense.  Either a whole struct/class is shared or it
is not.  Because you can only put classes on the heap, shared makes sense as an
attribute for a class.

 But then again, it might make sense to say "this struct is only ever shared,
so it should be required to go on the heap".  I like your idea later about
identifying shared struct types that should use synchronization.

Of course shared structs make sense, it's what allows implementing any non-trivial shared type. static Atomic!int counter; inside a function is perfectly fine. And, as somebody already mentioned in this thread, omitting 'static' should cause a build failure; right now it is accepted, even when written as shared Atomic!int counter; The problem? 'shared' is silently dropped. Move the counter from a struct into a function after realizing it's only accessed from one place, forget to add 'static' - and the result will compile w/o even a warning.

The difference is that static is not a type constructor.

The problem is that 'shared' is lost, resulting in an incorrect program. When you explicitly declare something as shared the compiler better treat it as such, or fail to compile it; silently changing the meaning is never acceptable.
 e.g.:
 
 shared int x; // typeof(x) == int

This could be made illegal, but if it is accepted then it should retain its type.
 void foo(shared int *n){...}
 
 foo(&x); // compiler error?  huh?
 
 I think this is a no-go.  Shared has to be statically disallowed for local
variables.

It's a possibility. Except _static_ local variables, those must work.
 Yes, head stripping.  I think it should be allowed.  For instance, if you
wanted to read a shared double, and take the cosine of it, this should be
allowed:

 auto n = cos(sharedValue);

 If it's not, the alternatives are:

 auto n = cos(cast()sharedValue);

 or

 double v = sharedValue; // explicit removal of shared
 auto m = cos(v);

 Neither of these look necessary, I think just allowing shared value types to
automatically convert to non-shared versions works the best.

If 'shared(VT)' implicitly converts to VT, then auto myY = x.y; // typeof(myY) == shared(int) would still be fine.

No, because then &myY yields a reference to shared data on the stack, which is what I think should be disallowed.

The only problem with shared data on the stack i can think of is portability. But this is something that can be decided at a much later time, it wouldn't be used much in practice anyway.
 My recommendation is that typeof(myY) == int.
 
 But I'm not sure allowing these implicit conversions is a good idea.
 At least not yet. :)

Implicit conversions to and from shared already are valid. i.e. int x = sharedInt; is valid code.

yes, but see above. shared(BVT)->BVT and shared(P*)->shared(P)* are allowed, and i don't think the latter is necessarily sound. Yes, the current shared model practically requires this, but i don't think raw access to shared data is the best approach. Note that inside synchronized() statements such conversions would be fine. And this also reminds me - mixed shared/unshared fields inside an aggregate should be legal. Why? Consider a mutex-like field that protects other data. Of course, synchronized() must imply a compiler barrier for this to work, but it needs it anyway. Memory barriers should always be explicit, btw; but for some architectures that might not be very practical.
 I'm talking about changing the types of expressions, such that the expression
type is always the tail-shared version.  In fact, simply using a shared piece
of data as an rvalue will convert it into a tail-shared version.

Could you provide an example? Because I'm not sure what problem this is supposed to solve. Eg. what is "a shared piece of data" and where does it come from? artur
Jun 08 2012
next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 06/08/12 23:59, Steven Schveighoffer wrote:
 On Fri, 08 Jun 2012 15:30:26 -0400, Artur Skawina <art.08.09 gmail.com> wrote:
 
 On 06/08/12 19:13, Steven Schveighoffer wrote:
 On Fri, 08 Jun 2012 10:57:15 -0400, Artur Skawina <art.08.09 gmail.com> wrote:

 On 06/08/12 06:03, Steven Schveighoffer wrote:

 I agree that the type should be shared(int), but the type should not transfer
to function calls or auto, it should be sticky to the particular variable. 
Only references should be sticky typed.

The problem with this is that it should be symmetrical, IOW the conversion from non-shared to shared would also have to be (implicitly) allowed. A type that converts to both would be better, even if harder to implement.

It should be allowed (and is today).

Hmm. I think it shouldn't be. This is how it is today: shared Atomic!int ai; shared Atomic!(void*) ap; void f(Atomic!int i) {} // Atomic() struct template temporarily made unshared for this test. void fp(Atomic!(void*) i) {} void main() { f(ai); fp(ap); } Error: function f (Atomic!(int) i) is not callable using argument types (shared(Atomic!(int))) Error: cannot implicitly convert expression (ai) of type shared(Atomic!(int)) to Atomic!(int) Error: function fp (Atomic!(void*) i) is not callable using argument types (shared(Atomic!(void*))) Error: cannot implicitly convert expression (ap) of type shared(Atomic!(void*)) to Atomic!(void*)

This is a bug (a regression, actually). I tested this simple code: struct S { int i; } void main() { shared(S) s; S s2 = s; } which fails on 2.057-2.059, but passes on 2.056 Looking at the changelog, looks like there were some changes related to shared and inout in 2.057. Those probably had a hand in this. I'll file a bug to document this regression.

FWIW i can't think of a specific case where allowing the implicit conversions would cause problems, it's just that I'm not sure they are absolutely always safe. Things like having *both* shared and unshared versions of a type shouldn't be impossible just because of this.
 It seems to work for built-in value types, which i didn't even realize,
because the thought of
 using them with 'shared' never crossed my mind. I don't really see why those
should be treated
 differently from user defined types, which should not allow implicit
shared<>unshared conversions.

They shouldn't be treated differently, everything should implicitly convert. Here is why: shared means "shared". If you make a copy of the value, that's your private copy, it's not shared! So there is no reason to automatically mark it as shared (you can explicitly mark it as shared if you want, but I contend this shouldn't be valid on stack data). Now, a shared *Reference* needs to keep the referred data as shared. So for instance, shared(int) * should *not* implicitly cast to int *. It's the same rules for immutable. You can assign an immutable int to an int no problem, because you aren't affecting the original's type, and the two are not referencing the same data.

That's all obvious; it's not the trivial scenarios that i'm worried about. It's things like having both "global" and "local" versions of a type, which are then accessed differently. Implicitly converting between those would be unsafe.
 I am talking about stripping head-shared, so shared(int *) automatically
converts to shared(int)* when used as an rvalue.

Where would the 'shared(int*) type come from? IOW, given 'shared struct S { int i; } S s;' what would the type of '&s.i' be? In your model; because right now it is 'shared(int)*'.

struct S { int *i; } void main() { shared(S) s; auto x = s.i; pragma(msg, typeof(x).stringof); // prints shared(int *) }

Right. Would you have a problem with disallowing accessing 's.i' like that? ;) Because that's (part of) my point - "raw" access to shared data (the pointer in this case) is not really safe. Which is why I'd prefer not to drop the shared qualifier from the head unless absolutely necessary. That operation is of course safe in itself - the issue is that it encourages writing code like in your example. Where the better alternatives would be either wrapping the type (pointer, here), or using some kind of accessor. Imagine auditing code that contains lots if these direct shared accesses; could you assume that every person who touched it knew what he/she was doing, that all required memory barriers are there etc?.. If you're saying that forbidding direct access can't be done by default for backward compatibility reasons, then I'm not really disagreeing. It's just that you are proposing a new shared model, and in that case i think the order should be 1) figuring out how a perfect one should look like and 2) making necessary compromises. And we're IMHO not at that second stage yet. :)
 Right, I was thinking shared structs do not make sense, since I don't think
shared members do not make sense.  Either a whole struct/class is shared or it
is not.  Because you can only put classes on the heap, shared makes sense as an
attribute for a class.

 But then again, it might make sense to say "this struct is only ever shared,
so it should be required to go on the heap".  I like your idea later about
identifying shared struct types that should use synchronization.

Of course shared structs make sense, it's what allows implementing any non-trivial shared type. static Atomic!int counter; inside a function is perfectly fine. And, as somebody already mentioned in this thread, omitting 'static' should cause a build failure; right now it is accepted, even when written as shared Atomic!int counter; The problem? 'shared' is silently dropped. Move the counter from a struct into a function after realizing it's only accessed from one place, forget to add 'static' - and the result will compile w/o even a warning.

The difference is that static is not a type constructor.

The problem is that 'shared' is lost, resulting in an incorrect program. When you explicitly declare something as shared the compiler better treat it as such, or fail to compile it; silently changing the meaning is never acceptable.

later:
 That was misleading; "shared" isn't actually lost, but as the variable is
 placed on the stack it becomes effectively thread local, which can be very
 unintuitive. But i can't think of an easy way to prevent this mistake, while
 still allowing shared data to be placed on the stack. And the latter can
 be useful sometimes...

I don't think it's unintuitive at all. shared *is* lost because it's *no longer shared*. It makes perfect sense to me. I also don't think it is a mistake. I frequently use the pattern of capturing the current state of a shared variable to work with locally within a function. Normally, in C or C++, there is no type difference between shared and unshared data, so it's just an int in both cases. However, while I'm working with my local copy, I don't want it changing, and it shouldn't be. A mistake is to mark it shared, because then I can send it to another thread possibly inadvertently.

The 'unintuitive' thing about it is having data typed as shared that in reality is thread local, that's all. It's unintuitive, but not actually wrong.
 e.g.:

 shared int x; // typeof(x) == int

This could be made illegal, but if it is accepted then it should retain its type.
 void foo(shared int *n){...}

 foo(&x); // compiler error?  huh?

 I think this is a no-go.  Shared has to be statically disallowed for local
variables.

It's a possibility. Except _static_ local variables, those must work.

static is different, because they are not local, they are global. Again, this comes down to a storage class vs. a type constructor. All that is different is that the symbol is local, it's still put in the global segment.

I'm apparently not making myself clear, sorry about that. I'm only trying to make sure that you don't propose to ban "shared" from local static data. It seemed you wanted to disallow a lot of things for apparently no, or no substantial, reason.
 If 'shared(VT)' implicitly converts to VT, then

    auto myY = x.y; // typeof(myY) == shared(int)

 would still be fine.

No, because then &myY yields a reference to shared data on the stack, which is what I think should be disallowed.

The only problem with shared data on the stack i can think of is portability. But this is something that can be decided at a much later time, it wouldn't be used much in practice anyway.

It's the same problem as taking addresses of stack variables. It's allowed, and can be valid in some cases, but it will cost you dearly if you get it wrong. You are better off allocating on the heap, or using library constructs that know what they are doing.

In fact your argument for forbidding shared data on the stack inspired me, so i created a relatively safe API that allows me do to exactly that... I originally wanted to ban it too, because of the 'unintuitiveness' of it, but now actually have code that uses it and works. Convincing me now that it shouldn't be allowed won't work. :) Thanks for the idea; it's not something i would have even considered doing in another language; the fact that D makes implementing it possible (and efficient) in about half a page of code and a one-liner in the callers is why I'm still around here, despite all the language holes.
 But I'm not sure allowing these implicit conversions is a good idea.
 At least not yet. :)

Implicit conversions to and from shared already are valid. i.e. int x = sharedInt; is valid code.

yes, but see above. shared(BVT)->BVT and shared(P*)->shared(P)* are allowed, and i don't think the latter is necessarily sound. Yes, the current shared model practically requires this, but i don't think raw access to shared data is the best approach.

It's not raw access, as soon as you create an rvalue, it's no longer aliased to the shared data. shared(P)* is it's own copy of the pointer. In other words, it's no longer shared, so shared should be stripped. However, what it *points* to is still shared, and still maintains the shard attribute.

It's the act of retrieving that pointer that I'd like to make safer.
 Making shared storage illegal on the stack is somewhat orthogonal to this. 
While I can see where having shared stack data is useful, it's completely
incorrect to forward shared attributes on copies of data.
 
 But it's so hard to guarantee that the stack variable storage does not go
away, especially when you have now thrown it out to another thread, which may
or may not tell you when it's done with it, that I think it should be made
illegal.

No! ;)
 At the very least, it should be illegal in  safe code.

Most certainly.
 Note that inside synchronized() statements such conversions would be fine.

I think you are not understanding the storage aspect.

I don't care about the storage aspect. :) We're talking about different things, maybe my explanation above made things clearer, at least I hope it did. The reason for which conversions inside a synchronized block are safer is the fact that it could be seen as if the current thread "owns" the data; at least if there's a clear monitor<->data dependency. But I think that approach wouldn't necessarily work, for example what dealing with semaphores, hence i must retract that statement; they are not always "fine".
 I'm talking about changing the types of expressions, such that the expression
type is always the tail-shared version.  In fact, simply using a shared piece
of data as an rvalue will convert it into a tail-shared version.

Could you provide an example? Because I'm not sure what problem this is supposed to solve. Eg. what is "a shared piece of data" and where does it come from?

If the above replies haven't responded enough, I will elaborate, let me know (responded while reading, I probably should have read the whole post first ;)

I was just wondering if you had any other case in mind, other than directly reading 'shared' data. Adding compiler magic to do that safely, which seems to be something that is still seriously considered, would add way too much overhead. It would make 'shared' fine for toy examples, but inappropriate for real code. artur
Jun 08 2012
prev sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 06/09/12 04:01, mta`chrono wrote:
      private static shared int counter; // shared across all instances
           auto i = ++counter;

What would you expect to happen here? Every thread to receive an unique value, at least until the counter wraps around? Then evaluating '++counter' needs to be atomic. How would that be implemented? Should the compiler do this automatically? Would this be expected behavior from an increment operator? It gets even worse in the postincrement case - the naive rewriting that then happens means it is practically impossible to implement it correctly in a user defined type. So anything that relies on the result of 'counter++' will be buggy - and you can't prevent this bug from happening. shared struct S { int x; disable this(this); } shared S s; Error: cannot implicitly convert expression (this) of type shared(S) to S Which may be related to the "bug" Steven filed, but fixing this by allowing the implicit conversion would just hide the real problem. artur
Jun 09 2012
prev sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 06/08/12 21:30, Artur Skawina wrote:
 in this thread, omitting 'static' should cause a build failure; right
 now it is accepted, even when written as

    shared Atomic!int counter;

 The problem? 'shared' is silently dropped. Move the counter from a struct
 into a function after realizing it's only accessed from one place, forget
 to add 'static' - and the result will compile w/o even a warning.

The difference is that static is not a type constructor.

The problem is that 'shared' is lost, resulting in an incorrect program. When you explicitly declare something as shared the compiler better treat it as such, or fail to compile it; silently changing the meaning is never acceptable.

That was misleading; "shared" isn't actually lost, but as the variable is placed on the stack it becomes effectively thread local, which can be very unintuitive. But i can't think of an easy way to prevent this mistake, while still allowing shared data to be placed on the stack. And the latter can be useful sometimes... artur
Jun 08 2012
prev sibling next sibling parent reply mta`chrono <chrono mta-international.net> writes:
Would this be legal?

class A
{
     private static shared int counter; // shared across all instances

     this()
     {
          auto i = ++counter;
          pragma(msg, typeof(i)); // prints int
     }
}
Jun 08 2012
parent Mike Wey <mike-wey example.com> writes:
On 06/09/2012 04:01 AM, mta`chrono wrote:
 Would this be legal?

 class A
 {
       private static shared int counter; // shared across all instances

       this()
       {
            auto i = ++counter;
            pragma(msg, typeof(i)); // prints int
       }
 }

Would it also be legal if the variable wasn't static? int opApply(int delegate(ref Pixels) dg) { shared(int) progress; foreach ( row; taskPool.parallel(iota(extent.y, extent.y + extent.height)) ) { int result = dg(Pixels(image, extent.x, row, extent.width, 1)); if ( result ) return result; if ( image.monitor !is null ) { atomicOp!"+="(progress, 1); image.monitor()("ImageView/" ~ image.filename, progress, extent.height); } } return 0; } -- Mike Wey
Jun 09 2012
prev sibling next sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 06/10/12 10:51, Mehrdad wrote:
 On Friday, 8 June 2012 at 04:03:08 UTC, Steven Schveighoffer wrote:
 I agree that the type should be shared(int), but the type should not transfer
to function calls or auto, it should be sticky to the particular variable.

I think that's a pretty fundamental problem: Shared is a type constructor, but you just mentioned that it has to be a property of the _variable_, i.e. a storage class.

Actually, no. What Steven is saying is that a _copy_ of the variable (the result of an expression, in general) does not need to retain the same qualifiers, such as 'shared' or 'const'. It means that the result does have a different type, but that's ok, as it's a different entity. This is obviously fine: const int[8] a; auto p = &a[0]; // typeof(p)==const(int)* as we just created the pointer, there's no need for it to be const, it just needs to point to 'const'. Now consider: const int*[8] a; auto p = a[0]; // typeof(p)==const(int)* Here we *also* create a new pointer, so it does not need to be const either. The only difference is in how the *value* of this new pointer is obtained. In the 'const' case the data (pointer value) can just be read from the memory location and the operation is always perfectly safe. The reason I don't want this to happen when dealing with 'shared' is not that it's wrong, it isn't. It's because it would make writing unsafe/confusing/buggy code too easy, as you then can completely ignore the 'shared' aspect. artur PS. That second const example does not describe current behavior - the compiler infers 'p' as const, which is wrong.
Jun 10 2012
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 06/10/2012 04:22 PM, Artur Skawina wrote:
 On 06/10/12 10:51, Mehrdad wrote:
 On Friday, 8 June 2012 at 04:03:08 UTC, Steven Schveighoffer wrote:
 I agree that the type should be shared(int), but the type should not transfer
to function calls or auto, it should be sticky to the particular variable.

I think that's a pretty fundamental problem: Shared is a type constructor, but you just mentioned that it has to be a property of the _variable_, i.e. a storage class.

Actually, no. What Steven is saying is that a _copy_ of the variable (the result of an expression, in general) does not need to retain the same qualifiers, such as 'shared' or 'const'. It means that the result does have a different type, but that's ok, as it's a different entity. This is obviously fine: const int[8] a; auto p =&a[0]; // typeof(p)==const(int)* as we just created the pointer, there's no need for it to be const, it just needs to point to 'const'. Now consider: const int*[8] a; auto p = a[0]; // typeof(p)==const(int)* Here we *also* create a new pointer, so it does not need to be const either. The only difference is in how the *value* of this new pointer is obtained. In the 'const' case the data (pointer value) can just be read from the memory location and the operation is always perfectly safe. The reason I don't want this to happen when dealing with 'shared' is not that it's wrong, it isn't. It's because it would make writing unsafe/confusing/buggy code too easy, as you then can completely ignore the 'shared' aspect. artur

I can't follow the line of reasoning here.
 PS. That second const example does not describe current behavior - the compiler
 infers 'p' as const, which is wrong.

It is by design. However, it would usually be more convenient if it was as you describe.
Jun 10 2012
prev sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 06/11/12 12:35, Steven Schveighoffer wrote:
 On Sun, 10 Jun 2012 10:22:50 -0400, Artur Skawina <art.08.09 gmail.com> wrote:
 This is obviously fine:

    const int[8] a; auto p = &a[0]; // typeof(p)==const(int)*

 as we just created the pointer, there's no need for it to be const, it just
needs
 to point to 'const'.

 Now consider:

    const int*[8] a; auto p = a[0]; // typeof(p)==const(int)*

 Here we *also* create a new pointer, so it does not need to be const either.
The
 only difference is in how the *value* of this new pointer is obtained. In the
 'const' case the data (pointer value) can just be read from the memory location
 and the operation is always perfectly safe.

The worst is this one: const int[8] a; auto e = a[0]; // typeof(p) == const(int) (in current compiler)

Yeah, and it makes using 'auto' less convenient - because then you have to cast away const.
 The reason I don't want this to happen when dealing with 'shared' is not that
it's
 wrong, it isn't. It's because it would make writing unsafe/confusing/buggy code
 too easy, as you then can completely ignore the 'shared' aspect.

I wholly disagree. In fact, keeping the full qualifier intact *enforces* incorrect code, because you are forcing shared semantics on literally unshared data. Never would this start ignoring shared on data that is truly shared. This is why I don't really get your argument. If you could perhaps explain with an example, it might be helpful.

*The programmer* can then treat shared data just like unshared. Because every load and every store will "magically" work. I'm afraid that after more than two or three people touch the code, the chances of it being correct would be less than 50%... The fact that you can not (or shouldn't be able to) mix shared and unshared freely is one of the main advantages of shared-annotation. artur
Jun 11 2012
parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 06/11/12 14:11, Steven Schveighoffer wrote:
 On Mon, 11 Jun 2012 07:56:12 -0400, Artur Skawina <art.08.09 gmail.com> wrote:
 
 On 06/11/12 12:35, Steven Schveighoffer wrote:

 I wholly disagree.  In fact, keeping the full qualifier intact *enforces*
incorrect code, because you are forcing shared semantics on literally unshared
data.

 Never would this start ignoring shared on data that is truly shared.  This is
why I don't really get your argument.

 If you could perhaps explain with an example, it might be helpful.

*The programmer* can then treat shared data just like unshared. Because every load and every store will "magically" work. I'm afraid that after more than two or three people touch the code, the chances of it being correct would be less than 50%... The fact that you can not (or shouldn't be able to) mix shared and unshared freely is one of the main advantages of shared-annotation.

If shared variables aren't doing the right thing with loads and stores, then we should fix that.

Where do you draw the line? shared struct S { int i void* p; SomeStruct s; ubyte[256] a; } shared(S)* p = ... ; auto v1 = p.i; auto v2 = p.p; auto v3 = p.s; auto v4 = p.a; auto v5 = p.i++; Are these operations on shared data all safe? Note that if these accesses would be protected by some lock, then the 'shared' qualifier wouldn't really be needed - compiler barriers, that make sure it all happens while this thread holds the lock, would be enough. (even the order of operations doesn't usually matter in that case and enforcing one would in fact add overhead)
 But leaving things "the way they are" doesn't fix any problems:
 
 shared int x;
 
 void main()
 {
     auto y = x; // typeof(y) == shared(int)
     y += 5;
     x = y;
 }
 
 How is this any more "correct" than if y is int?

Not allowing the implicit conversions does not fix all cases, yes. It will catch /some/ invalid uses. It's a step towards a saner model. Where "raw" access like in your example won't be allowed. Yes, it would need a compiler flag. When I say that I think your ideas are a step in the right direction, it's because I know where the journey will eventually lead us... :) I'm not sure forbidding direct manipulation is the ultimate goal, but it is a safe base for the next stage, which would be figuring out which cases can be allowed, because the compiler will always be able to do the right thing. Starting out by allowing everything is what got us into this mess in the first place. artur
Jun 11 2012
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
 Are these operations on shared data all safe? Note that if these
 accesses would be protected by some lock, then the 'shared' qualifier
 wouldn't really be needed - compiler barriers, that make sure it all
 happens while this thread holds the lock, would be enough. (even the
 order of operations doesn't usually matter in that case and enforcing
 one would in fact add overhead)

No, they should not be all safe, I never suggested that. It's impossible to engineer a one-size-fits-all for accessing shared variables, because it doesn't know what mechanism you are going to use to protect it. As you say, once this data is protected by a lock, memory barriers aren't needed. But requiring a lock is too heavy handed for all cases. This is a good point to make about the current memory-barrier attempts, they just aren't comprehensive enough, nor do they guarantee pretty much anything except simple loads and stores. Perhaps the correct way to implement shared semantics is to not allow access *whatsoever* (except taking the address of a shared piece of data), unless you: a) lock the block that contains it b) use some library feature that uses casting-away of shared to accomplish the correct thing. For example, atomicOp.

atomic. Yet things like this are funky trap: shread int x; //global ... x = x + func(); //Booom! read-modify-write and not atomic, should have used x+= func() So a-b set of rules could be more reasonable then it seems.
 None of this can prevent deadlocks, but it does create a way to prevent
 deadlocks.

 If this was the case, stack data would be able to be marked shared, and
 you'd have to use option b (it would not be in a block). Perhaps for
 simple data types, when memory barriers truly are enough, and a
 shared(int) is on the stack (and not part of a container), straight
 loads and stores would be allowed.

 Now, would you agree that:

 auto v1 = synchronized p.i;

 might be a valid mechanism? In other words, assuming p is lockable,
 synchronized p.i locks p, then reads i, then unlocks p, and the result
 type is unshared?

 Also, inside synchronized(p), p becomes tail-shared, meaning all data
 contained in p is unshared, all data referred to by p remains shared.

 In this case, we'd need a new type constructor (e.g. locked) to
 formalize the type.

 Make sense?

While I've missed a good portion of this thread I think we should explore this direction. Shared has to be connected with locks/synchronized. -- Dmitry Olshansky
Jun 11 2012
prev sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 06/11/12 19:27, Steven Schveighoffer wrote:
 On Mon, 11 Jun 2012 09:41:37 -0400, Artur Skawina <art.08.09 gmail.com> wrote:
 
 On 06/11/12 14:11, Steven Schveighoffer wrote:
 On Mon, 11 Jun 2012 07:56:12 -0400, Artur Skawina <art.08.09 gmail.com> wrote:

 On 06/11/12 12:35, Steven Schveighoffer wrote:

 I wholly disagree.  In fact, keeping the full qualifier intact *enforces*
incorrect code, because you are forcing shared semantics on literally unshared
data.

 Never would this start ignoring shared on data that is truly shared.  This is
why I don't really get your argument.

 If you could perhaps explain with an example, it might be helpful.

*The programmer* can then treat shared data just like unshared. Because every load and every store will "magically" work. I'm afraid that after more than two or three people touch the code, the chances of it being correct would be less than 50%... The fact that you can not (or shouldn't be able to) mix shared and unshared freely is one of the main advantages of shared-annotation.

If shared variables aren't doing the right thing with loads and stores, then we should fix that.

Where do you draw the line? shared struct S { int i void* p; SomeStruct s; ubyte[256] a; } shared(S)* p = ... ; auto v1 = p.i; auto v2 = p.p; auto v3 = p.s; auto v4 = p.a; auto v5 = p.i++; Are these operations on shared data all safe? Note that if these accesses would be protected by some lock, then the 'shared' qualifier wouldn't really be needed - compiler barriers, that make sure it all happens while this thread holds the lock, would be enough. (even the order of operations doesn't usually matter in that case and enforcing one would in fact add overhead)

No, they should not be all safe, I never suggested that. It's impossible to engineer a one-size-fits-all for accessing shared variables, because it doesn't know what mechanism you are going to use to protect it. As you say, once this data is protected by a lock, memory barriers aren't needed. But requiring a lock is too heavy handed for all cases. This is a good point to make about the current memory-barrier attempts, they just aren't comprehensive enough, nor do they guarantee pretty much anything except simple loads and stores. Perhaps the correct way to implement shared semantics is to not allow access *whatsoever* (except taking the address of a shared piece of data), unless you: a) lock the block that contains it b) use some library feature that uses casting-away of shared to accomplish the correct thing. For example, atomicOp.

Exactly; this is what I'm after the whole time. And I think it can be done in most cases without casting away shared. For example by allowing the safe conversions from/to shared of results of expression involving shared data, but only under certain circumstances. Eg in methods with a shared 'this'.
 None of this can prevent deadlocks, but it does create a way to prevent
deadlocks.
 
 If this was the case, stack data would be able to be marked shared, and you'd
have to use option b (it would not be in a block).  Perhaps for simple data
types, when memory barriers truly are enough, and a shared(int) is on the stack
(and not part of a container), straight loads and stores would be allowed.

Why? Consider the case of function that directly or indirectly launches a few threads and gives them the address of some local shared object. If the current thread also accesses this object, which has to be possible, then it must obey the same rules.
 Now, would you agree that:
 
 auto v1 = synchronized p.i;
 
 might be a valid mechanism?  In other words, assuming p is lockable,
synchronized p.i locks p, then reads i, then unlocks p, and the result type is
unshared?

I think I would prefer auto v1 = synchronized(p).i; ie for the synchronized expression to lock the object, return an unshared reference, and the object be unlocked once this ref goes away. RLII. ;) Which would then also allow for { auto unshared_p = synchronized(p); auto v1 = unshared_p.i; auto v2 = unshared_p.p; // etc } and with a little more syntax sugar it could turn into synchronized (unshared_p = p) { auto v1 = unshared_p.i; auto v2 = unshared_p.p; // etc } The problem with this is that it only unshares the head, which I think isn't enough. Hmm. One approach would be to allow shared struct S { ubyte* data; AStruct *s1; shared AnotherStruct *s2; shared S* next; } and for synchronized(s){} to drop 'shared' from any field that isn't also marked as shared. IOW treat any 'unshared' field as owned by the object. (an alternative could be to tag the fields that should be unshared instead)
 Also, inside synchronized(p), p becomes tail-shared, meaning all data
contained in p is unshared, all data referred to by p remains shared.
 
 In this case, we'd need a new type constructor (e.g. locked) to formalize the
type.

I should have read to the end i guess. :) You mean something like I described above, only done by mutating the type of 'p'? That might work too. But I need to think about this some more. Why would we need 'locked'?
 Make sense?

More and more. artur
Jun 11 2012
next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 06/11/12 22:21, Steven Schveighoffer wrote:
 Now, would you agree that:

 auto v1 = synchronized p.i;

 might be a valid mechanism?  In other words, assuming p is lockable,
synchronized p.i locks p, then reads i, then unlocks p, and the result type is
unshared?

I think I would prefer auto v1 = synchronized(p).i;

This kind of makes synchronized a type constructor, which it is not.

Yes; the suggestion was to also allow synchronized /expressions/, in addition to statements.
 ie for the synchronized expression to lock the object, return an unshared
 reference, and the object be unlocked once this ref goes away. RLII. ;)

 Which would then also allow for

    {
       auto unshared_p = synchronized(p);
       auto v1 = unshared_p.i;
       auto v2 = unshared_p.p;
       // etc
    }

I think this can be done, but I would not want to use synchronized. One of the main benefits of synchronized is it's a block attribute, not a type attribute. So you can't actually abuse it.

There's a precedent, mixin expressions. However, there's no need to invent new constructs, as this already works: { auto unshared_p = p.locked; auto v1 = unshared_p.i; auto v2 = unshared_p.p; // etc } and does not require compiler or language changes. I'm using this idiom with mutexes and semaphores; the 'locked' implementation is *extremely* fragile, it's very easy to confuse the compiler, which then spits out nonsensical error messages and refuses to cooperate. But the above should already be possible, only the return type could be problematic; keeping 'p' opaque would be best. I'll play with this when I find some time. But 'synchronized' and 'shared' are really two different things, I probably shouldn't have used your original example as a base, as it only added to the confusion, sorry. 'synchronized' allows you to implement critical sections. 'shared' is just a way to mark some data as needing special treatment. If all accesses to an object are protected by 'synchronized', either explicitly or implicitly (by using a struct or class marked as synchronized) then you don't need to mark the data as 'shared' at all. It would be pointless - the thread that owns the lock also owns the data. 'shared' is what lets you implement the locking primitives used by synchronized and various lock-free schemes. (right now 'shared' alone isn't powerful enough, yes) You can use one or the other, sometimes even both, but they are not directly tied to each other. So there's no need for 'synchronized' to unshare anything, at least not in the simple mutex case. Accessing objects both with and without holding a lock is extremely rare.
 The locked type I specify below might fit the bill.  But it would have to be
hard-tied to the block.  In other words, we would have to make *very* certain
it would not escape the block.  Kind of like inout.     

void f(scope S*); ... { auto locked_p = p.locked; f(locked_p.s); } Requiring the signature to be 'void f(locked S*);' would not be a good idea; this must continue to work and introducing another type would exclude all code not specifically written with it in mind, like practically all libraries.
 This is important -- you don't want to escape a reference to the unlocked type
somewhere.

Yes, but it needs another solution. 'scope' might be enough, but right now we'd have to trust the programmer completely... (It's about not leaking refs to *inside* the locked object, not just 'p' (or 'locked_p') itself) artur
Jun 11 2012
prev sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 06/12/12 00:00, Artur Skawina wrote:
 On 06/11/12 22:21, Steven Schveighoffer wrote:
 Now, would you agree that:

 auto v1 = synchronized p.i;

 might be a valid mechanism?  In other words, assuming p is lockable,
synchronized p.i locks p, then reads i, then unlocks p, and the result type is
unshared?




What I think you want is relatively simple, something like this: struct synchronized(m) S { int i; void *p; Mutex m; } and then for S to be completely opaque, unless inside a synchronized statement. So S* s = ... auto v1 = s.i; // "Error: access to 's.i' requires synchronization" synchronized (s) { auto v2 = s.i; // ... } auto v3 = s.p; // "Error: access to 's.p' requires synchronization" and there's no 'shared' involved at all. Provided that no reference to a locked 's' can escape this should be enough to solve this problem. Preventing the leaks while not unnecessarily restricting what can be done inside the synchronized block would be a different problem. The obvious solution would be to treat all refs gotten from or via 's' as scoped (and trust the programmer; with time the enforcing can be improved), but sometimes you will actually want to remove objects from a synchronized container - so that must be possible too. artur
Jun 11 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 06/11/12 12:26, Steven Schveighoffer wrote:
 On Sat, 09 Jun 2012 08:31:01 -0400, Mike Wey <mike-wey example.com> wrote:
 
 On 06/09/2012 04:01 AM, mta`chrono wrote:
 Would this be legal?

 class A
 {
       private static shared int counter; // shared across all instances

       this()
       {
            auto i = ++counter;
            pragma(msg, typeof(i)); // prints int
       }
 }

Would it also be legal if the variable wasn't static?

No.

Why? What if this class would like to launch a few threads to do some work, export the address of the counter and have them report back by updating it? Unlike the shared-on-stack case, this wouldn't even be unsafe (the memory won't be freed until all threads stop using it.) The alternative is to have to split the class into two, more heap allocations etc.
 int opApply(int delegate(ref Pixels) dg)
 {
      shared(int) progress;

      foreach ( row; taskPool.parallel(iota(extent.y, extent.y +
extent.height)) )
      {
          int result = dg(Pixels(image, extent.x, row, extent.width, 1));

          if ( result )
               return result;

          if ( image.monitor !is null )
          {
              atomicOp!"+="(progress, 1);
              image.monitor()("ImageView/" ~ image.filename, progress,
extent.height);
          }
      }
      return 0;
 }

AFAIK, if you removed shared from progress, it would work. I don't think std.parallel is as strict as std.concurrency (and for pretty good reason). I think a better way to mark progress is to make it an atomic integer type (like Artur has developed).

Yes. The problem with that however is that I never managed to make this do the right thing: Atomic!int a; // somewhere in a shared struct/class. ... int x = s.a; // OK, access via getter. auto y = s.a; // Oops, we just copied the whole struct. void f(T)(T arg); f(s.a); // Ditto. Which may happen to work for properly aligned small structs because accessing those are atomic anyway, but is wrong. artur
Jun 11 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 06/11/12 14:07, Steven Schveighoffer wrote:
 However, allocating another heap block to do sharing, in my opinion, is worth
the extra cost.  This way, you have clearly separated what is shared and what
isn't.
 
 You can always cast to get around the limitations.

"clearly separating what is shared and what isn't" *is* exactly what tagging the data with 'shared' does.
 I think a better way to mark progress is to make it an atomic integer type
(like Artur has developed).

Yes. The problem with that however is that I never managed to make this do the right thing: Atomic!int a; // somewhere in a shared struct/class. ... int x = s.a; // OK, access via getter. auto y = s.a; // Oops, we just copied the whole struct. void f(T)(T arg); f(s.a); // Ditto. Which may happen to work for properly aligned small structs because accessing those are atomic anyway, but is wrong.

You can disable copying with disable this(this);

I wish. shared struct S { int x; disable this(this); } shared S s; Error: cannot implicitly convert expression (this) of type shared(S) to S The post-dec/inc rewriting together with this bug also means you cannot prevent the bogus atomic++ operation from succeeding. And that is not the only problem with 'shared' and structs. http://www.digitalmars.com/d/archives/digitalmars/D/Disabling_copy_constructor_in_shared_structs_157638.html http://www.digitalmars.com/d/archives/digitalmars/D/dtors_in_shared_structs_fail_to_compile_157978.html artur
Jun 11 2012
prev sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 06/11/12 16:57, Steven Schveighoffer wrote:
 On Mon, 11 Jun 2012 09:39:40 -0400, Artur Skawina <art.08.09 gmail.com> wrote:
 
 On 06/11/12 14:07, Steven Schveighoffer wrote:
 However, allocating another heap block to do sharing, in my opinion, is worth
the extra cost.  This way, you have clearly separated what is shared and what
isn't.

 You can always cast to get around the limitations.

"clearly separating what is shared and what isn't" *is* exactly what tagging the data with 'shared' does.

I posted a response, it showed up in the online forums, but for some reason didn't show up in my nntp client... If you missed it, it is here. http://forum.dlang.org/post/op.wfqtz5u0eav7ka steves-laptop

The mailing list delivered it too. I'm against disallowing things that are not unsafe as such and have valid use cases, so we will probably not agree about that. I considered the GC/mempool implications before arguing for allowing 'shared' fields inside unshared aggregates - the compiler has enough knowledge to pick the right pool, if it ever decides to treat "local" data differently. I'm not sure doing that would be good idea, in cases where the lifetime of an object cannot be determined statically. But deciding to use a global pool can always be done by checking if a shared field exists. artur
Jun 11 2012
prev sibling next sibling parent reply "Robert DaSilva" <spunit262 yahoo.com> writes:
On Thursday, 7 June 2012 at 23:51:27 UTC, Steven Schveighoffer 
wrote:
 I am having a quite interesting debate on pure and shared with 
 Artur Skawina in another thread, and I thought about how 
 horrible a state shared is in.  It's not implemented as 
 designed, and the design really leaves more questions than it 
 has answers.  In addition, it has not real connection with 
 thread synchronization whatsoever, and Michel Fortin suggested 
 some improvements to synchronized that look really cool that 
 would involve shared.

 So I thought about, what are the truly valid uses of shared?  
 And then I thought, more importantly, what are the *invalid* 
 uses of shared?

 Because I think one of the biggest confusing pieces of shared 
 is, we have no idea when I should use it, or how to use it.  So 
 far, the only benefit I've seen from it is when you mark 
 something as not shared, the things you can assume about it.

 I think a couple usages of shared make very little sense:

 1. having a shared piece of data on the stack.
 2. shared value types.

 1 makes little sense because a stack is a wholly-owned 
 subsidiary of a thread.  Its existence depends completely on 
 the stack frame staying around.  If I share a piece of my stack 
 with another thread, then I return from that function, I have 
 just sent a dangling pointer over to the other thread.

 2 makes little sense because when you pass around a value type, 
 it's inherently not shared, you are making a copy!  What is the 
 point of passing a shared int to another thread?  Might as well 
 pass an int (this is one of the sticking points I have with 
 pure functions accepting or dealing with shared data).

 I have an idea that might fix *both* of these problems.

 What if we disallowed declarations of shared type constructors 
 on any value type?  So shared(int) x is an error, but 
 shared(int)* x is not (and actually shared(int *) x is also an 
 error, because the pointer is passed by value).  However, the 
 type shared(int) is valid, it just can't be used to declare 
 anything.

 The only types that could be shared, would be:

 ref shared T => local reference to shared data
 shared(T) * => local pointer to shared data
 shared(C) => local reference to shared class

 And that's it.

 The following would be illegal:

 struct X
 {
   shared int x; // illegal
   shared(int)* y; // legal

   shared(X) *next; // legal
 }

 shared class C  // legal, C is always a reference type
 {
    shared int x; // illegal, but useless, since C is already 
 shared
 }

 If you notice, I never allow shared values to be stored on the 
 stack, they are always going to be stored on the heap.  We can 
 use this to our advantage -- using special allocators that are 
 specific to shared data, we can ensure the synchronization 
 tools necessary to protect this data gets allocated on the heap 
 along side it.  I'm not sure exactly how this could work, but I 
 was thinking, instead of allocating a monitor based on the 
 *type* (i.e. a class), you allocate it based on whether it's 
 *shared* or not.  Since I can never create a shared struct X on 
 the stack, it must be in the heap, so...

 struct X
 {
    int y;
 }

 shared(X) *x = new shared(X);

 synchronized(x) // scope-locks hidden allocated monitor object
 {
    x.y = 5;
 }

 x.y = 5; // should we disallow this, or maybe even auto-lock x?

 Hm... another idea -- you can't extract any piece of an 
 aggregate.  That is, it would be illegal to do:

 shared(int)* myYptr = &x.y;

 because that would work around the synchronization.

 auto would have to strip shared:

 auto myY = x.y; // typeof(myY) == int.

 This is definitely not a complete proposal.  But I wonder if 
 this is the right direction?

 -Steve

You're forgetting about Global data. I think rather the head shared should be striped as this fits better with how D treats meaningless specifiers. And trying to put structs that contain shared data on the stack should be illegal. shared int global_shared_int; // type is shared(int); struct struct_with_shared_data { shared int shared_data_in_a_struct; // type is shared(int) } int main() { shared int not_really_shared; // type is int shared int* unshared_ptr_to_shared_int; // type is shared(int)*; struct_with_shared_data foo; // illegal, shared data can't stored on the stack // nor can the sharedness be striped. }
Jun 07 2012
parent travert phare.normalesup.org (Christophe Travert) writes:
"Steven Schveighoffer" , dans le message (digitalmars.D:169568), a
 écrit :
 On Thu, 07 Jun 2012 22:16:21 -0400, Robert DaSilva <spunit262 yahoo.com>  
 wrote:
 
 
 You're forgetting about Global data.

I wasn't so much forgetting it as I was ignoring it :) My thought on that is that the shared keyword in that case is truly a storage class. It's the one place where having a value-type based shared value makes sense. If we had some kind of synchronized/shared pairing, the compiler would have to allocate mutex space for that too.
 I think rather the head shared should be striped as this fits better  
 with how D treats meaningless specifiers.

I don't think we can do that with type constructors, but I'm not sure. I'm certainly against it, as I am against the current abuses of that methodology.
 And trying to put structs that contain shared data on the stack should  
 be illegal.


The compiler can already heap-allocate function variables that should be on the stack. So why disallowing shared for function variables? void foo() { shared int test; // allocates test on shared memory block. } Just like: int delegate(int) adder(int a) { return b => (a+b); // allocates a on the heap to make a closure. } -- Christophe
Jun 18 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 07 Jun 2012 22:16:21 -0400, Robert DaSilva <spunit262 yahoo.com>  
wrote:


 You're forgetting about Global data.

I wasn't so much forgetting it as I was ignoring it :) My thought on that is that the shared keyword in that case is truly a storage class. It's the one place where having a value-type based shared value makes sense. If we had some kind of synchronized/shared pairing, the compiler would have to allocate mutex space for that too.
 I think rather the head shared should be striped as this fits better  
 with how D treats meaningless specifiers.

I don't think we can do that with type constructors, but I'm not sure. I'm certainly against it, as I am against the current abuses of that methodology.
 And trying to put structs that contain shared data on the stack should  
 be illegal.

First, I can't imagine that you'd actually need a block on the heap that was partially thread-local and partially shared. So allowing partially-shared types does not make sense to me. For the cases where it may make sense, a value-type + shared pointer might work, or a heap block of TLS data which has a member pointing to another heap block of shared data could make sense. Can you think of good use cases that show it is important to have hybrid blocks? Second, I kind of like the idea that the monitor for the data in the block protects the entire block. This would not make sense if part of the block is not shared. -Steve
Jun 07 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 07 Jun 2012 20:58:13 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/08/12 01:51, Steven Schveighoffer wrote:
 The following would be illegal:

 struct X
 {
   shared int x; // illegal
   shared(int)* y; // legal

   shared(X) *next; // legal
 }

Note that the type of 'x' in shared struct S { int x; } should probably be 'shared(int)'. Which lets you safely take an address of an aggregates field.

That's one of the things I'm wondering about. Should it be allowed? I agree that the type should be shared(int), but the type should not transfer to function calls or auto, it should be sticky to the particular variable. Only references should be sticky typed.
 And I'm not sure if marking a struct and class as shared would work
 correctly right now, it's probably too easy to lose the 'shared'  
 qualifier.

Right, I was thinking shared structs do not make sense, since I don't think shared members do not make sense. Either a whole struct/class is shared or it is not. Because you can only put classes on the heap, shared makes sense as an attribute for a class. But then again, it might make sense to say "this struct is only ever shared, so it should be required to go on the heap". I like your idea later about identifying shared struct types that should use synchronization.
 shared class C  // legal, C is always a reference type
 {
    shared int x; // illegal, but useless, since C is already shared
 }

Redundant, but should be accepted, just like 'static' is.

That's probably fine.
 If you notice, I never allow shared values to be stored on the stack,  
 they are always going to be stored on the heap.  We can use this to our  
 advantage -- using special allocators that are specific to shared data,  
 we can ensure the synchronization tools necessary to protect this data  
 gets allocated on the heap along side it.  I'm not sure exactly how  
 this could work, but I was thinking, instead of allocating a monitor  
 based on the *type* (i.e. a class), you allocate it based on whether  
 it's *shared* or not.  Since I can never create a shared struct X on  
 the stack, it must be in the heap, so...

 struct X
 {
    int y;
 }

 shared(X) *x = new shared(X);

 synchronized(x) // scope-locks hidden allocated monitor object
 {
    x.y = 5;
 }

 x.y = 5; // should we disallow this, or maybe even auto-lock x?

 Hm... another idea -- you can't extract any piece of an aggregate.   
 That is, it would be illegal to do:

 shared(int)* myYptr = &x.y;

 because that would work around the synchronization.

That's too restrictive. It would overload 'shared' even more. If you want that kind of synchronize magic to work, just allow: shared synchronized(optional_locking_primitive) struct S { ... } And *now* 'x.y = 5' can do its magic, while '&x.y' can be disallowed. shared struct S { Atomic!int i; } shared(S)* p = ... p.i += 1; should work, so accessing fields must remain possible.

OK. You are right, synchronized may be overkill for basic types.
 auto would have to strip shared:

 auto myY = x.y; // typeof(myY) == int.

Hmm, i'm not sure about this, maybe it should be disallowed; it could only strip 'shared' from the head anyway.

Yes, head stripping. I think it should be allowed. For instance, if you wanted to read a shared double, and take the cosine of it, this should be allowed: auto n = cos(sharedValue); If it's not, the alternatives are: auto n = cos(cast()sharedValue); or double v = sharedValue; // explicit removal of shared auto m = cos(v); Neither of these look necessary, I think just allowing shared value types to automatically convert to non-shared versions works the best.
 This is definitely not a complete proposal.  But I wonder if this is  
 the right direction?

I think it is.

good. -Steve
Jun 07 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 08 Jun 2012 03:13:04 -0400, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 On 08.06.2012 8:03, Steven Schveighoffer wrote:

 Yes, head stripping. I think it should be allowed. For instance, if you
 wanted to read a shared double, and take the cosine of it, this should
 be allowed:

 auto n = cos(sharedValue);

what if it's a auto n = cos(sharedValues[k]); //type is shared(double) right? and somebody in the other thread is halfway through writing it?

At this point, I am not sure shared means synchronized or atomic. It just means multiple threads can see it. Perhaps you have already taken the lock that protects this variable. My point is, the transfer of a shared type into an rvalue should strip the head-shared part of it *automatically*. Especially since it would be illegal to store such a type on the stack (and quite literally, a shared rvalue makes no sense whatsoever), we don't want to force casting everywhere. -Steve
Jun 08 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 08 Jun 2012 10:57:15 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/08/12 06:03, Steven Schveighoffer wrote:

 I agree that the type should be shared(int), but the type should not  
 transfer to function calls or auto, it should be sticky to the  
 particular variable.  Only references should be sticky typed.

The problem with this is that it should be symmetrical, IOW the conversion from non-shared to shared would also have to be (implicitly) allowed. A type that converts to both would be better, even if harder to implement.

It should be allowed (and is today). I am talking about stripping head-shared, so shared(int *) automatically converts to shared(int)* when used as an rvalue.
 Right, I was thinking shared structs do not make sense, since I don't  
 think shared members do not make sense.  Either a whole struct/class is  
 shared or it is not.  Because you can only put classes on the heap,  
 shared makes sense as an attribute for a class.

 But then again, it might make sense to say "this struct is only ever  
 shared, so it should be required to go on the heap".  I like your idea  
 later about identifying shared struct types that should use  
 synchronization.

Of course shared structs make sense, it's what allows implementing any non-trivial shared type. static Atomic!int counter; inside a function is perfectly fine. And, as somebody already mentioned in this thread, omitting 'static' should cause a build failure; right now it is accepted, even when written as shared Atomic!int counter; The problem? 'shared' is silently dropped. Move the counter from a struct into a function after realizing it's only accessed from one place, forget to add 'static' - and the result will compile w/o even a warning.

The difference is that static is not a type constructor. e.g.: shared int x; // typeof(x) == int void foo(shared int *n){...} foo(&x); // compiler error? huh? I think this is a no-go. Shared has to be statically disallowed for local variables.
 Yes, head stripping.  I think it should be allowed.  For instance, if  
 you wanted to read a shared double, and take the cosine of it, this  
 should be allowed:

 auto n = cos(sharedValue);

 If it's not, the alternatives are:

 auto n = cos(cast()sharedValue);

 or

 double v = sharedValue; // explicit removal of shared
 auto m = cos(v);

 Neither of these look necessary, I think just allowing shared value  
 types to automatically convert to non-shared versions works the best.

If 'shared(VT)' implicitly converts to VT, then auto myY = x.y; // typeof(myY) == shared(int) would still be fine.

No, because then &myY yields a reference to shared data on the stack, which is what I think should be disallowed. My recommendation is that typeof(myY) == int.
 But I'm not sure allowing these implicit conversions is a good idea.
 At least not yet. :)

Implicit conversions to and from shared already are valid. i.e. int x = sharedInt; is valid code. I'm talking about changing the types of expressions, such that the expression type is always the tail-shared version. In fact, simply using a shared piece of data as an rvalue will convert it into a tail-shared version. -Steve
Jun 08 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 08 Jun 2012 15:30:26 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/08/12 19:13, Steven Schveighoffer wrote:
 On Fri, 08 Jun 2012 10:57:15 -0400, Artur Skawina <art.08.09 gmail.com>  
 wrote:

 On 06/08/12 06:03, Steven Schveighoffer wrote:

 I agree that the type should be shared(int), but the type should not  
 transfer to function calls or auto, it should be sticky to the  
 particular variable.  Only references should be sticky typed.

The problem with this is that it should be symmetrical, IOW the conversion from non-shared to shared would also have to be (implicitly) allowed. A type that converts to both would be better, even if harder to implement.

It should be allowed (and is today).

Hmm. I think it shouldn't be. This is how it is today: shared Atomic!int ai; shared Atomic!(void*) ap; void f(Atomic!int i) {} // Atomic() struct template temporarily made unshared for this test. void fp(Atomic!(void*) i) {} void main() { f(ai); fp(ap); } Error: function f (Atomic!(int) i) is not callable using argument types (shared(Atomic!(int))) Error: cannot implicitly convert expression (ai) of type shared(Atomic!(int)) to Atomic!(int) Error: function fp (Atomic!(void*) i) is not callable using argument types (shared(Atomic!(void*))) Error: cannot implicitly convert expression (ap) of type shared(Atomic!(void*)) to Atomic!(void*)

This is a bug (a regression, actually). I tested this simple code: struct S { int i; } void main() { shared(S) s; S s2 = s; } which fails on 2.057-2.059, but passes on 2.056 Looking at the changelog, looks like there were some changes related to shared and inout in 2.057. Those probably had a hand in this. I'll file a bug to document this regression.
 It seems to work for built-in value types, which i didn't even realize,  
 because the thought of
 using them with 'shared' never crossed my mind. I don't really see why  
 those should be treated
 differently from user defined types, which should not allow implicit  
 shared<>unshared conversions.

They shouldn't be treated differently, everything should implicitly convert. Here is why: shared means "shared". If you make a copy of the value, that's your private copy, it's not shared! So there is no reason to automatically mark it as shared (you can explicitly mark it as shared if you want, but I contend this shouldn't be valid on stack data). Now, a shared *Reference* needs to keep the referred data as shared. So for instance, shared(int) * should *not* implicitly cast to int *. It's the same rules for immutable. You can assign an immutable int to an int no problem, because you aren't affecting the original's type, and the two are not referencing the same data.
 I am talking about stripping head-shared, so shared(int *)  
 automatically converts to shared(int)* when used as an rvalue.

Where would the 'shared(int*) type come from? IOW, given 'shared struct S { int i; } S s;' what would the type of '&s.i' be? In your model; because right now it is 'shared(int)*'.

struct S { int *i; } void main() { shared(S) s; auto x = s.i; pragma(msg, typeof(x).stringof); // prints shared(int *) }
 Right, I was thinking shared structs do not make sense, since I don't  
 think shared members do not make sense.  Either a whole struct/class  
 is shared or it is not.  Because you can only put classes on the  
 heap, shared makes sense as an attribute for a class.

 But then again, it might make sense to say "this struct is only ever  
 shared, so it should be required to go on the heap".  I like your  
 idea later about identifying shared struct types that should use  
 synchronization.

Of course shared structs make sense, it's what allows implementing any non-trivial shared type. static Atomic!int counter; inside a function is perfectly fine. And, as somebody already mentioned in this thread, omitting 'static' should cause a build failure; right now it is accepted, even when written as shared Atomic!int counter; The problem? 'shared' is silently dropped. Move the counter from a struct into a function after realizing it's only accessed from one place, forget to add 'static' - and the result will compile w/o even a warning.

The difference is that static is not a type constructor.

The problem is that 'shared' is lost, resulting in an incorrect program. When you explicitly declare something as shared the compiler better treat it as such, or fail to compile it; silently changing the meaning is never acceptable.

later:
 That was misleading; "shared" isn't actually lost, but as the variable is
 placed on the stack it becomes effectively thread local, which can be  
 very
 unintuitive. But i can't think of an easy way to prevent this mistake,  
 while
 still allowing shared data to be placed on the stack. And the latter can
 be useful sometimes...

I don't think it's unintuitive at all. shared *is* lost because it's *no longer shared*. It makes perfect sense to me. I also don't think it is a mistake. I frequently use the pattern of capturing the current state of a shared variable to work with locally within a function. Normally, in C or C++, there is no type difference between shared and unshared data, so it's just an int in both cases. However, while I'm working with my local copy, I don't want it changing, and it shouldn't be. A mistake is to mark it shared, because then I can send it to another thread possibly inadvertently.
 e.g.:

 shared int x; // typeof(x) == int

This could be made illegal, but if it is accepted then it should retain its type.
 void foo(shared int *n){...}

 foo(&x); // compiler error?  huh?

 I think this is a no-go.  Shared has to be statically disallowed for  
 local variables.

It's a possibility. Except _static_ local variables, those must work.

static is different, because they are not local, they are global. Again, this comes down to a storage class vs. a type constructor. All that is different is that the symbol is local, it's still put in the global segment.
 If 'shared(VT)' implicitly converts to VT, then

    auto myY = x.y; // typeof(myY) == shared(int)

 would still be fine.

No, because then &myY yields a reference to shared data on the stack, which is what I think should be disallowed.

The only problem with shared data on the stack i can think of is portability. But this is something that can be decided at a much later time, it wouldn't be used much in practice anyway.

It's the same problem as taking addresses of stack variables. It's allowed, and can be valid in some cases, but it will cost you dearly if you get it wrong. You are better off allocating on the heap, or using library constructs that know what they are doing.
 But I'm not sure allowing these implicit conversions is a good idea.
 At least not yet. :)

Implicit conversions to and from shared already are valid. i.e. int x = sharedInt; is valid code.

yes, but see above. shared(BVT)->BVT and shared(P*)->shared(P)* are allowed, and i don't think the latter is necessarily sound. Yes, the current shared model practically requires this, but i don't think raw access to shared data is the best approach.

It's not raw access, as soon as you create an rvalue, it's no longer aliased to the shared data. shared(P)* is it's own copy of the pointer. In other words, it's no longer shared, so shared should be stripped. However, what it *points* to is still shared, and still maintains the shard attribute. Making shared storage illegal on the stack is somewhat orthogonal to this. While I can see where having shared stack data is useful, it's completely incorrect to forward shared attributes on copies of data. But it's so hard to guarantee that the stack variable storage does not go away, especially when you have now thrown it out to another thread, which may or may not tell you when it's done with it, that I think it should be made illegal. At the very least, it should be illegal in safe code.
 Note that inside synchronized() statements such conversions would be  
 fine.

I think you are not understanding the storage aspect.
 I'm talking about changing the types of expressions, such that the  
 expression type is always the tail-shared version.  In fact, simply  
 using a shared piece of data as an rvalue will convert it into a  
 tail-shared version.

Could you provide an example? Because I'm not sure what problem this is supposed to solve. Eg. what is "a shared piece of data" and where does it come from?

If the above replies haven't responded enough, I will elaborate, let me know (responded while reading, I probably should have read the whole post first ;) -Steve
Jun 08 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 08 Jun 2012 17:59:35 -0400, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 I'll file a bug to document this regression.

http://d.puremagic.com/issues/show_bug.cgi?id=8212 -Steve
Jun 08 2012
prev sibling next sibling parent "Mehrdad" <wfunction hotmail.com> writes:
On Friday, 8 June 2012 at 04:03:08 UTC, Steven Schveighoffer 
wrote:
 I agree that the type should be shared(int), but the type 
 should not transfer to function calls or auto, it should be 
 sticky to the particular variable.

I think that's a pretty fundamental problem: Shared is a type constructor, but you just mentioned that it has to be a property of the _variable_, i.e. a storage class. Just thought I'd throw that out there.
Jun 10 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 08 Jun 2012 22:01:41 -0400, mta`chrono  
<chrono mta-international.net> wrote:

 Would this be legal?

 class A
 {
      private static shared int counter; // shared across all instances

      this()
      {
           auto i = ++counter;
           pragma(msg, typeof(i)); // prints int
      }
 }

Yes, but it would not automatically make ++counter atomic. My proposal is simply to avoid sharing data that shouldn't be shared (i.e. stack data and temporaries), not to make all operations thread-safe. -Steve
Jun 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 09 Jun 2012 08:31:01 -0400, Mike Wey <mike-wey example.com> wrote:

 On 06/09/2012 04:01 AM, mta`chrono wrote:
 Would this be legal?

 class A
 {
       private static shared int counter; // shared across all instances

       this()
       {
            auto i = ++counter;
            pragma(msg, typeof(i)); // prints int
       }
 }

Would it also be legal if the variable wasn't static?

No.
 int opApply(int delegate(ref Pixels) dg)
 {
      shared(int) progress;

      foreach ( row; taskPool.parallel(iota(extent.y, extent.y +  
 extent.height)) )
      {
          int result = dg(Pixels(image, extent.x, row, extent.width, 1));

          if ( result )
               return result;

          if ( image.monitor !is null )
          {
              atomicOp!"+="(progress, 1);
              image.monitor()("ImageView/" ~ image.filename, progress,  
 extent.height);
          }
      }
      return 0;
 }

AFAIK, if you removed shared from progress, it would work. I don't think std.parallel is as strict as std.concurrency (and for pretty good reason). I think a better way to mark progress is to make it an atomic integer type (like Artur has developed). -Steve
Jun 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sun, 10 Jun 2012 10:22:50 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/10/12 10:51, Mehrdad wrote:
 On Friday, 8 June 2012 at 04:03:08 UTC, Steven Schveighoffer wrote:
 I agree that the type should be shared(int), but the type should not  
 transfer to function calls or auto, it should be sticky to the  
 particular variable.

I think that's a pretty fundamental problem: Shared is a type constructor, but you just mentioned that it has to be a property of the _variable_, i.e. a storage class.

Actually, no. What Steven is saying is that a _copy_ of the variable (the result of an expression, in general) does not need to retain the same qualifiers, such as 'shared' or 'const'. It means that the result does have a different type, but that's ok, as it's a different entity.

Yes, this is correct.
 This is obviously fine:

    const int[8] a; auto p = &a[0]; // typeof(p)==const(int)*

 as we just created the pointer, there's no need for it to be const, it  
 just needs
 to point to 'const'.

 Now consider:

    const int*[8] a; auto p = a[0]; // typeof(p)==const(int)*

 Here we *also* create a new pointer, so it does not need to be const  
 either. The
 only difference is in how the *value* of this new pointer is obtained.  
 In the
 'const' case the data (pointer value) can just be read from the memory  
 location
 and the operation is always perfectly safe.

The worst is this one: const int[8] a; auto e = a[0]; // typeof(p) == const(int) (in current compiler)
 The reason I don't want this to happen when dealing with 'shared' is not  
 that it's
 wrong, it isn't. It's because it would make writing  
 unsafe/confusing/buggy code
 too easy, as you then can completely ignore the 'shared' aspect.

I wholly disagree. In fact, keeping the full qualifier intact *enforces* incorrect code, because you are forcing shared semantics on literally unshared data. Never would this start ignoring shared on data that is truly shared. This is why I don't really get your argument. If you could perhaps explain with an example, it might be helpful. -Steve
Jun 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 11 Jun 2012 07:51:37 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/11/12 12:26, Steven Schveighoffer wrote:
 On Sat, 09 Jun 2012 08:31:01 -0400, Mike Wey <mike-wey example.com>  
 wrote:

 On 06/09/2012 04:01 AM, mta`chrono wrote:
 Would this be legal?

 class A
 {
       private static shared int counter; // shared across all  
 instances

       this()
       {
            auto i = ++counter;
            pragma(msg, typeof(i)); // prints int
       }
 }

Would it also be legal if the variable wasn't static?

No.

Why? What if this class would like to launch a few threads to do some work, export the address of the counter and have them report back by updating it? Unlike the shared-on-stack case, this wouldn't even be unsafe (the memory won't be freed until all threads stop using it.) The alternative is to have to split the class into two, more heap allocations etc.

The interesting thing here is, then you have both shared and unshared data in the same heap block. Because a class is heap-allocated by default, you are right, you have a much smaller chance of sharing stack data. However, allocating another heap block to do sharing, in my opinion, is worth the extra cost. This way, you have clearly separated what is shared and what isn't. You can always cast to get around the limitations. auto sharedCounter = cast(shared int *)&counter; dispatchThreadsToUpdateCounter(sharedCounter); waitForThreadsToExit();
 I think a better way to mark progress is to make it an atomic integer  
 type (like Artur has developed).

Yes. The problem with that however is that I never managed to make this do the right thing: Atomic!int a; // somewhere in a shared struct/class. ... int x = s.a; // OK, access via getter. auto y = s.a; // Oops, we just copied the whole struct. void f(T)(T arg); f(s.a); // Ditto. Which may happen to work for properly aligned small structs because accessing those are atomic anyway, but is wrong.

You can disable copying with disable this(this); -Steve
Jun 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 11 Jun 2012 07:56:12 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/11/12 12:35, Steven Schveighoffer wrote:

 I wholly disagree.  In fact, keeping the full qualifier intact  
 *enforces* incorrect code, because you are forcing shared semantics on  
 literally unshared data.

 Never would this start ignoring shared on data that is truly shared.   
 This is why I don't really get your argument.

 If you could perhaps explain with an example, it might be helpful.

*The programmer* can then treat shared data just like unshared. Because every load and every store will "magically" work. I'm afraid that after more than two or three people touch the code, the chances of it being correct would be less than 50%... The fact that you can not (or shouldn't be able to) mix shared and unshared freely is one of the main advantages of shared-annotation.

If shared variables aren't doing the right thing with loads and stores, then we should fix that. But leaving things "the way they are" doesn't fix any problems: shared int x; void main() { auto y = x; // typeof(y) == shared(int) y += 5; x = y; } How is this any more "correct" than if y is int? -Steve
Jun 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 11 Jun 2012 09:39:40 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/11/12 14:07, Steven Schveighoffer wrote:
 However, allocating another heap block to do sharing, in my opinion, is  
 worth the extra cost.  This way, you have clearly separated what is  
 shared and what isn't.

 You can always cast to get around the limitations.

"clearly separating what is shared and what isn't" *is* exactly what tagging the data with 'shared' does.

There are special GC considerations for shared as well. For instance, unshared data can go into a "local" heap. But I feel uneasy passing around pointers to heap data to other threads where some of my thread-local data is present. It's bound to lead to unwanted effects. For example, if I share some piece of my class, and the other thread holds onto it forever, it means my class (which may hold large resources that aren't shared) will not be dealloced even when there's no reference to it outside that piece of shared data. Also, don't forget, you can cast to get the behavior you desire. You should always be able to work around these limitations with casting (and casting unshared to shared should be well-defined for the compiler as long as you don't ever treat the data as unshared again, similar to immutable). I think it's reasonable to make it easier to write good designs, and harder to write questionable ones. There's a lot of rules in D that are like that, just the whole notion of marking shared data is one of them. I want to stress that this idea of preventing member variables from being marked shared is not necessarily a *requirement*, it's merely something I think fosters good design. There's nothing technically wrong with it, I just think code is better off not doing it. But marking stack variables as shared I think has to go -- there are too many pitfalls, a cast should be required.
 I think a better way to mark progress is to make it an atomic integer  
 type (like Artur has developed).

Yes. The problem with that however is that I never managed to make this do the right thing: Atomic!int a; // somewhere in a shared struct/class. ... int x = s.a; // OK, access via getter. auto y = s.a; // Oops, we just copied the whole struct. void f(T)(T arg); f(s.a); // Ditto. Which may happen to work for properly aligned small structs because accessing those are atomic anyway, but is wrong.

You can disable copying with disable this(this);

I wish. shared struct S { int x; disable this(this); } shared S s; Error: cannot implicitly convert expression (this) of type shared(S) to S

This *definitely* is a bug.
 The post-dec/inc rewriting together with this bug also means you cannot
 prevent the bogus atomic++ operation from succeeding.

 And that is not the only problem with 'shared' and structs.

 http://www.digitalmars.com/d/archives/digitalmars/D/Disabling_copy_constructor_in_shared_structs_157638.html
 http://www.digitalmars.com/d/archives/digitalmars/D/dtors_in_shared_structs_fail_to_compile_157978.html

Haven't read these, but if there are bugs in the compiler, make sure you file those. disable this(this) *should* work, if it doesn't its a bug. -Steve
Jun 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 11 Jun 2012 09:39:40 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/11/12 14:07, Steven Schveighoffer wrote:
 However, allocating another heap block to do sharing, in my opinion, is  
 worth the extra cost.  This way, you have clearly separated what is  
 shared and what isn't.

 You can always cast to get around the limitations.

"clearly separating what is shared and what isn't" *is* exactly what tagging the data with 'shared' does.

I posted a response, it showed up in the online forums, but for some reason didn't show up in my nntp client... If you missed it, it is here. http://forum.dlang.org/post/op.wfqtz5u0eav7ka steves-laptop -Steve
Jun 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 11 Jun 2012 09:41:37 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/11/12 14:11, Steven Schveighoffer wrote:
 On Mon, 11 Jun 2012 07:56:12 -0400, Artur Skawina <art.08.09 gmail.com>  
 wrote:

 On 06/11/12 12:35, Steven Schveighoffer wrote:

 I wholly disagree.  In fact, keeping the full qualifier intact  
 *enforces* incorrect code, because you are forcing shared semantics  
 on literally unshared data.

 Never would this start ignoring shared on data that is truly shared.   
 This is why I don't really get your argument.

 If you could perhaps explain with an example, it might be helpful.

*The programmer* can then treat shared data just like unshared. Because every load and every store will "magically" work. I'm afraid that after more than two or three people touch the code, the chances of it being correct would be less than 50%... The fact that you can not (or shouldn't be able to) mix shared and unshared freely is one of the main advantages of shared-annotation.

If shared variables aren't doing the right thing with loads and stores, then we should fix that.

Where do you draw the line? shared struct S { int i void* p; SomeStruct s; ubyte[256] a; } shared(S)* p = ... ; auto v1 = p.i; auto v2 = p.p; auto v3 = p.s; auto v4 = p.a; auto v5 = p.i++; Are these operations on shared data all safe? Note that if these accesses would be protected by some lock, then the 'shared' qualifier wouldn't really be needed - compiler barriers, that make sure it all happens while this thread holds the lock, would be enough. (even the order of operations doesn't usually matter in that case and enforcing one would in fact add overhead)

No, they should not be all safe, I never suggested that. It's impossible to engineer a one-size-fits-all for accessing shared variables, because it doesn't know what mechanism you are going to use to protect it. As you say, once this data is protected by a lock, memory barriers aren't needed. But requiring a lock is too heavy handed for all cases. This is a good point to make about the current memory-barrier attempts, they just aren't comprehensive enough, nor do they guarantee pretty much anything except simple loads and stores. Perhaps the correct way to implement shared semantics is to not allow access *whatsoever* (except taking the address of a shared piece of data), unless you: a) lock the block that contains it b) use some library feature that uses casting-away of shared to accomplish the correct thing. For example, atomicOp. None of this can prevent deadlocks, but it does create a way to prevent deadlocks. If this was the case, stack data would be able to be marked shared, and you'd have to use option b (it would not be in a block). Perhaps for simple data types, when memory barriers truly are enough, and a shared(int) is on the stack (and not part of a container), straight loads and stores would be allowed. Now, would you agree that: auto v1 = synchronized p.i; might be a valid mechanism? In other words, assuming p is lockable, synchronized p.i locks p, then reads i, then unlocks p, and the result type is unshared? Also, inside synchronized(p), p becomes tail-shared, meaning all data contained in p is unshared, all data referred to by p remains shared. In this case, we'd need a new type constructor (e.g. locked) to formalize the type. Make sense? -Steve
Jun 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 11 Jun 2012 13:42:37 -0400, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 a) lock the block that contains it
 b) use some library feature that uses casting-away of shared to
 accomplish the correct thing. For example, atomicOp.

atomic. Yet things like this are funky trap: shread int x; //global ... x = x + func(); //Booom! read-modify-write and not atomic, should have used x+= func()

We cannot prevent data races such as these (though we may be able to disable specific cases like this), since you can always split out this expression into multiple valid ones. Also, you can hide details in functions: x = func(x); But we can say that you cannot *read or write* a shared variable non-atomically. That is a goal I think is achievable by the type system and the language. That arguably has no real-world value, ever, whereas the above may be valid in some cases (maybe you know more semantically about the application than the compiler can glean).
 While I've missed a good portion of this thread I think we should  
 explore this direction. Shared has to be connected with  
 locks/synchronized.

Yes, I agree. If shared and synchronized are not connected somehow, the point of both seems rather lost. As this was mostly a brainstorming post, I'll restate what I think as a reply to the original post, since my views have definitely changed. -Steve
Jun 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 11 Jun 2012 15:23:56 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 06/11/12 19:27, Steven Schveighoffer wrote:

 Perhaps the correct way to implement shared semantics is to not allow  
 access *whatsoever* (except taking the address of a shared piece of  
 data), unless you:

 a) lock the block that contains it
 b) use some library feature that uses casting-away of shared to  
 accomplish the correct thing.  For example, atomicOp.

Exactly; this is what I'm after the whole time. And I think it can be done in most cases without casting away shared. For example by allowing the safe conversions from/to shared of results of expression involving shared data, but only under certain circumstances. Eg in methods with a shared 'this'.

Good, I'm glad we are starting to come together.
 None of this can prevent deadlocks, but it does create a way to prevent  
 deadlocks.

 If this was the case, stack data would be able to be marked shared, and  
 you'd have to use option b (it would not be in a block).  Perhaps for  
 simple data types, when memory barriers truly are enough, and a  
 shared(int) is on the stack (and not part of a container), straight  
 loads and stores would be allowed.

Why? Consider the case of function that directly or indirectly launches a few threads and gives them the address of some local shared object. If the current thread also accesses this object, which has to be possible, then it must obey the same rules.

I think this is possible for what I prescribed. You need a special construct for locking and using shared data on the stack (for instance Lockable!S). Another possible option is to consider the stack frame as the "container", and if it contains any shared data, put in a hidden mutex. In order to do this correctly, we need a way to hook synchronized properly from library code.
 Now, would you agree that:

 auto v1 = synchronized p.i;

 might be a valid mechanism?  In other words, assuming p is lockable,  
 synchronized p.i locks p, then reads i, then unlocks p, and the result  
 type is unshared?

I think I would prefer auto v1 = synchronized(p).i;

This kind of makes synchronized a type constructor, which it is not.
 ie for the synchronized expression to lock the object, return an unshared
 reference, and the object be unlocked once this ref goes away. RLII. ;)

 Which would then also allow for

    {
       auto unshared_p = synchronized(p);
       auto v1 = unshared_p.i;
       auto v2 = unshared_p.p;
       // etc
    }

I think this can be done, but I would not want to use synchronized. One of the main benefits of synchronized is it's a block attribute, not a type attribute. So you can't actually abuse it. The locked type I specify below might fit the bill. But it would have to be hard-tied to the block. In other words, we would have to make *very* certain it would not escape the block. Kind of like inout.
 Also, inside synchronized(p), p becomes tail-shared, meaning all data  
 contained in p is unshared, all data referred to by p remains shared.

 In this case, we'd need a new type constructor (e.g. locked) to  
 formalize the type.

I should have read to the end i guess. :) You mean something like I described above, only done by mutating the type of 'p'? That might work too.

Right, any accesses to p *inside* the block "magically" become locked(S) instead of shared(S). We have to make certain locked(S) instances cannot escape, and we already do something like this with inout -- just don't allow members or static variables to be typed as locked(T). I like replacing the symbol because then it doesn't allow you access to the outer symbol (although you can get around this, it should be made difficult). As long as the locks are reentrant, it shouldn't pose a large problem, but obviously you should try and avoid locking the same data over and over again. One interesting thing: synchronized methods now would mark this as locked(typeof(this)) instead of typeof(this). So you can *avoid* the locking and unlocking code while calling member functions, while preserving it for the first call. This is important -- you don't want to escape a reference to the unlocked type somewhere. -Steve
Jun 11 2012
prev sibling parent deadalnix <deadalnix gmail.com> writes:
Le 08/06/2012 01:51, Steven Schveighoffer a écrit :
 2. shared value types.

2. You can have value type on heap. Or value types that point to shared data.
Jun 12 2012