Manu <turkeyman gmail.com> writes:

Okay, so I've been thinking on this for a while... I think I have a
pretty good feel for how shared is meant to be.

1. shared should behave exactly like const, except in addition to
inhibiting write access, it also inhibits read access.

I think this is the foundation for a useful definition for shared, and
it's REALLY easy to understand and explain.

Current situation where you can arbitrarily access shared members
undermines any value it has. Shared must assure you don't access
members unsafely, and the only way to do that with respect to data
members, is to inhibit access completely.
I think shared is just const without read access.

Assuming this world... how do you use shared?

1. traditional; assert that the object become thread-local by
acquiring a lock, cast shared away
2. object may have shared methods; such methods CAN be called on
shared instances. such methods may internally implement
synchronisation to perform their function. perhaps methods of a
lock-free queue structure for instance, or operator overloads on
`Atomic!int`, etc.

In practise, there is no functional change in usage from the current
implementation, except we disallow unsafe accesses (which will make
the thing useful).

From there, it opens up another critical opportunity; T* -> shared(T)*

promotion.
Const would be useless without T* -> const(T)* promotion. Shared
suffers a similar problem.
If you write a lock-free queue for instance, and all the methods are
`shared` (ie, threadsafe), then under the current rules, you can't
interact with the object when it's not shared, and that's fairly
useless.

Assuming the rules above: "can't read or write to members", and the
understanding that `shared` methods are expected to have threadsafe
implementations (because that's the whole point), what are the risks
from allowing T* -> shared(T)* conversion?

All the risks that I think have been identified previously assume that
you can arbitrarily modify the data. That's insanity... assume we fix
that... I think the promotion actually becomes safe now...?

Destroy...

Peter Alexander <peter.alexander.au gmail.com> writes:

On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 2. object may have shared methods; such methods CAN be called on
 shared instances. such methods may internally implement
 synchronisation to perform their function. perhaps methods of a
 lock-free queue structure for instance, or operator overloads on
 `Atomic!int`, etc.

Just checking my understanding: are you saying here that shared 
methods can effectively do anything and the burden of correctness 
is on the author? Or do you still have to cast the shared away 
first?

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 12:15 PM Peter Alexander via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 2. object may have shared methods; such methods CAN be called on
 shared instances. such methods may internally implement
 synchronisation to perform their function. perhaps methods of a
 lock-free queue structure for instance, or operator overloads on
 `Atomic!int`, etc.

 Just checking my understanding: are you saying here that shared
 methods can effectively do anything and the burden of correctness
 is on the author? Or do you still have to cast the shared away
 first?

Burden of correctness that a `shared` method is indeed threadsafe is
absolutely on the author; there's nothing we can do to guarantee this
(multithreading is hard!), but in this new world, a user of an API
will be able to see shared methods and assume that they're threadsafe,
since that would be (and should be!) the whole point.

Since `shared` has no read or write access, at the bottom of the
stack, it may be necessary that the function cast shared away to
implement its magic. If you lock a mutex for instance, then you
naturally need to cast it away; but that's solving the problem with a
sledge-hammer.
In my experience, most such functions are implemented with atomics;
and the atomic API already receives shared args, ie:
https://github.com/dlang/druntime/blob/master/src/core/atomic.d#L379
So, if you do your work with atomics, then you don't need any casts.
Also, if the object is a composite, then a `shared` method is able to
call `shared` methods on its members, and in that case, you are able
to do useful aggregate work without casting.

I think the interactions I describe above are all correct.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Monday, 15 October 2018 at 19:14:58 UTC, Peter Alexander wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 2. object may have shared methods; such methods CAN be called 
 on
 shared instances. such methods may internally implement
 synchronisation to perform their function. perhaps methods of a
 lock-free queue structure for instance, or operator overloads 
 on
 `Atomic!int`, etc.

 Just checking my understanding: are you saying here that shared 
 methods can effectively do anything and the burden of 
 correctness is on the author? Or do you still have to cast the 
 shared away first?

Well `this` is still shared so you would be prevented from doing 
anything non-atomically or lock cast away shared, but that will 
be enforced by the type system. So the burden of  correctness is 
on the author, but the complier enforces correct behaviour.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 12:15 PM Peter Alexander via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 2. object may have shared methods; such methods CAN be called on
 shared instances. such methods may internally implement
 synchronisation to perform their function. perhaps methods of a
 lock-free queue structure for instance, or operator overloads on
 `Atomic!int`, etc.

 Just checking my understanding: are you saying here that shared
 methods can effectively do anything and the burden of correctness
 is on the author? Or do you still have to cast the shared away
 first?

Just to be clear:

 are you saying here that shared methods can effectively do anything

No, quite the opposite. I am saying that shared objects have neither
read nor write access to members. Shared means "no read or write
access". You can only call shared methods.

jmh530 <john.michael.hall gmail.com> writes:

On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Okay, so I've been thinking on this for a while... I think I 
 have a pretty good feel for how shared is meant to be.

 1. shared should behave exactly like const, except in addition 
 to inhibiting write access, it also inhibits read access.

Are you familiar with reference capabilities[1] in the pony 
language? They describe many of them in terms of read/write 
uniqueness. Another way they describe them [2] is in denying 
aliases, like deny global read alias.


[1] 
https://tutorial.ponylang.io/capabilities/reference-capabilities.html
[2] See page 12-14: http://www.doc.ic.ac.uk/~scd/Pony-WG2.16.pdf

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Destroy...

Keep in mind immutable is implicitly shared (i.e. not in tls) 
because nobody can change it. It should stay readable for this 
reason.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 12:45 PM Nicholas Wilson via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Destroy...

 Keep in mind immutable is implicitly shared (i.e. not in tls)
 because nobody can change it. It should stay readable for this
 reason.

Are you saying `is(immutable(int) == shared) == true)` ??

jmh530 <john.michael.hall gmail.com> writes:

On Monday, 15 October 2018 at 20:44:35 UTC, Manu wrote:
 snip

 Are you saying `is(immutable(int) == shared) == true)` ??

 From the spec:
"Applying any qualifier to immutable T results in immutable T. 
This makes immutable a fixed point of qualifier combinations and 
makes types such as const(immutable(shared T)) impossible to 
create."

Example:

import std.stdio : writeln;

void main()
{
     writeln(is(immutable(int) == shared immutable(int)) == true); 
//prints true
}

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Monday, 15 October 2018 at 20:54:12 UTC, jmh530 wrote:
 On Monday, 15 October 2018 at 20:44:35 UTC, Manu wrote:
 snip

 Are you saying `is(immutable(int) == shared) == true)` ??

 From the spec:
 "Applying any qualifier to immutable T results in immutable T. 
 This makes immutable a fixed point of qualifier combinations 
 and makes types such as const(immutable(shared T)) impossible 
 to create."

 Example:

 import std.stdio : writeln;

 void main()
 {
     writeln(is(immutable(int) == shared immutable(int)) == 
 true); //prints true
 }

The philosophy of this is that: the value never changes, 
therefore only one copy of the variable needs to exist (i.e. 
immutable variables declared at module scope are _not_ thread 
local) and can be shared between threads with no race conditions.

I'm saying that while what you propose sounds very reasonable, we 
should make sure that reading immutable variables is still good ;)

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 2:05 PM Nicholas Wilson via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 20:54:12 UTC, jmh530 wrote:
 On Monday, 15 October 2018 at 20:44:35 UTC, Manu wrote:
 snip

 Are you saying `is(immutable(int) == shared) == true)` ??

 From the spec:
 "Applying any qualifier to immutable T results in immutable T.
 This makes immutable a fixed point of qualifier combinations
 and makes types such as const(immutable(shared T)) impossible
 to create."

 Example:

 import std.stdio : writeln;

 void main()
 {
     writeln(is(immutable(int) == shared immutable(int)) ==
 true); //prints true
 }

 The philosophy of this is that: the value never changes,
 therefore only one copy of the variable needs to exist (i.e.
 immutable variables declared at module scope are _not_ thread
 local) and can be shared between threads with no race conditions.

 I'm saying that while what you propose sounds very reasonable, we
 should make sure that reading immutable variables is still good ;)

I don't think what I describe affects immutable in any way;
`is(immutable(int) == shared) == false`, as it should be. Immutable
wouldn't have its rules affected by any change to shared.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Monday, 15 October 2018 at 21:08:38 UTC, Manu wrote:
 On Mon, Oct 15, 2018 at 2:05 PM Nicholas Wilson via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 I'm saying that while what you propose sounds very reasonable, 
 we should make sure that reading immutable variables is still 
 good ;)

 I don't think what I describe affects immutable in any way;
 `is(immutable(int) == shared) == false`, as it should be. 
 Immutable
 wouldn't have its rules affected by any change to shared.

OK, just making sure you've got this covered.

Peter Alexander <peter.alexander.au gmail.com> writes:

On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Destroy...

What you describe sounds better than what we currently have.

I have at least two concerns:

1. A single producer, single consumer (SPSC) queue is necessarily 
shared, but is only safe if there is one writing thread and one 
reading thread. Is it ok if shared also requires user discipline 
and/or runtime checks to ensure correct usage?

2. In your scheme (as I understand), a struct composed entirely 
of atomics would be able to implement shared methods without any 
casts, but also be completely thread *unsafe*. Is this okay?



struct TwoInts {
   Atomic!int x, y;

   void swap() shared {
     int z = x.load;
     x.store(y.load);
     y.store(z);
   }
}

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 1:05 PM Peter Alexander via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Destroy...

 What you describe sounds better than what we currently have.

 I have at least two concerns:

 1. A single producer, single consumer (SPSC) queue is necessarily
 shared, but is only safe if there is one writing thread and one
 reading thread. Is it ok if shared also requires user discipline
 and/or runtime checks to ensure correct usage?

I think you can model this differently... perhaps rather than a single
object, it's a coupled pair.
For instance, a thread-local enque object, and a separate thread-local
deque object. Those would express the 2 thread-local points of access
to queue and dequeue, and the implementation shunts between them in a
threadsafe way. I mean, it's not *really* a threadsafe primitive, so
for the whole object to be 'shared', I think that might be a
design-fail.

 2. In your scheme (as I understand), a struct composed entirely
 of atomics would be able to implement shared methods without any
 casts, but also be completely thread *unsafe*. Is this okay?

It would be as safe as the design intends.
A struct with a bunch of public atomics effectively presents a set of
distinct atomics, and each one is thread-safe relative to eachother.
If the members are actually coupled into aggregate state, then you
make then private and methods implement the state transitions in such
a way guaranteeing atomicity.
Like I say before, the language can't "make it threadsafe" for you...
be producing a shared method, you have a responsibility to make sure
it works right.



 struct TwoInts {
    Atomic!int x, y;

    void swap() shared {
      int z = x.load;
      x.store(y.load);
      y.store(z);
    }
 }

Your swap function is plain broken; it doesn't do what the API promises.
You can write all sorts of broken code, and this is a good example of
just plain broken code.
Also, `x` and `y` probably shouldn't be public, or it effectively
communicates that they're de-coupled state.

Peter Alexander <peter.alexander.au gmail.com> writes:

On Monday, 15 October 2018 at 20:53:32 UTC, Manu wrote:
 On Mon, Oct 15, 2018 at 1:05 PM Peter Alexander via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 1. A single producer, single consumer (SPSC) queue is 
 necessarily shared, but is only safe if there is one writing 
 thread and one reading thread. Is it ok if shared also 
 requires user discipline and/or runtime checks to ensure 
 correct usage?

 I think you can model this differently... perhaps rather than a 
 single
 object, it's a coupled pair.

That's a nice design.


 Your swap function is plain broken; it doesn't do what the API 
 promises.
 You can write all sorts of broken code, and this is a good 
 example of
 just plain broken code.

If it is broken then why allow it? Why do we need to cast shared 
away if they weren't atomic and why do we allow it if they are 
atomic?

I understand that shared can't magically tell you when code is 
thread safe or not. It does make sense to disallow almost 
everything and require casts. I'm just not seeing the value of 
allowing shared methods to access shared members if it isn't 
thread safe. Make it require casts.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 4:25 PM Peter Alexander via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 20:53:32 UTC, Manu wrote:
 On Mon, Oct 15, 2018 at 1:05 PM Peter Alexander via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 1. A single producer, single consumer (SPSC) queue is
 necessarily shared, but is only safe if there is one writing
 thread and one reading thread. Is it ok if shared also
 requires user discipline and/or runtime checks to ensure
 correct usage?

 I think you can model this differently... perhaps rather than a
 single
 object, it's a coupled pair.

 That's a nice design.


 Your swap function is plain broken; it doesn't do what the API
 promises.
 You can write all sorts of broken code, and this is a good
 example of
 just plain broken code.

 If it is broken then why allow it? Why do we need to cast shared
 away if they weren't atomic and why do we allow it if they are
 atomic?

It's not that it's 'allowed' other than, yes, access to atomic int's
is allowed in a threadsafe way because atomic access to individual
int's is threadsafe, so the primitive operation is perfectly
acceptable.

Your function models a higher-level concept; which is an atomic swap.
You need to make sure that a threadsafe API you're authoring does
actually deliver on the promise it makes.

 I understand that shared can't magically tell you when code is
 thread safe or not. It does make sense to disallow almost
 everything and require casts. I'm just not seeing the value of
 allowing shared methods to access shared members if it isn't
 thread safe. Make it require casts.

Because in an awful lot of cases, it is threadsafe. Implementing
low-level machinery, and consuming such machinery should have a 1:many
relationship. You're talking about adding friction to the 'many' such
that the '1' knows that they need to implement their function right? I
mean, they already know that, because they wrote 'shared' after their
function declaration.

In the common case, perhaps my object might aggregate a threadsafe
queue, and my shared method prepares an item and then adds it to the
queue. I think the vast majority case will be making use of utility
functionality and that shouldn't present undue friction.
It's not like atomic int's that are class members are going to be
accidentally twiddled; you added Atomic!int's to your class, and that
wasn't an accident... and if you're making use of the lowest-level
atomic primitives, it's fair to presume you know how to use them.
You're writing a shared method, which means you already encapsulate
the promise that you are implementing a function that deals with
thread-safety. I don't know what the cast in this case would add.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:

 Assuming the rules above: "can't read or write to members", and 
 the understanding that `shared` methods are expected to have 
 threadsafe implementations (because that's the whole point), 
 what are the risks from allowing T* -> shared(T)* conversion?

 All the risks that I think have been identified previously 
 assume that you can arbitrarily modify the data. That's 
 insanity... assume we fix that... I think the promotion 
 actually becomes safe now...?

You're still talking about implicit promotion? No, it does not 
become safe no matter what restrictions you put on `shared` 
instances, because the caller of any function that takes `shared` 
arguments remains blissfully unaware of this promotion.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 1:15 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:

 Assuming the rules above: "can't read or write to members", and
 the understanding that `shared` methods are expected to have
 threadsafe implementations (because that's the whole point),
 what are the risks from allowing T* -> shared(T)* conversion?

 All the risks that I think have been identified previously
 assume that you can arbitrarily modify the data. That's
 insanity... assume we fix that... I think the promotion
 actually becomes safe now...?

 You're still talking about implicit promotion?

Absolutely. This is critical to make shared useful, and I think
there's a path to make it work.

 No, it does not
 become safe no matter what restrictions you put on `shared`
 instances, because the caller of any function that takes `shared`
 arguments remains blissfully unaware of this promotion.

It doesn't matter... what danger is there of distributing a shared pointer?
Are you concerned that someone with a shared reference might call a
threadsafe method? If that's an invalid operation, then the method has
broken it's basic agreed contract.

And I agree that it's conceivable that you could contrive a bad
program, but you can contrive a bad program with literally any
language feature!

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Monday, 15 October 2018 at 20:57:46 UTC, Manu wrote:
 On Mon, Oct 15, 2018 at 1:15 PM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:

 Assuming the rules above: "can't read or write to members", 
 and the understanding that `shared` methods are expected to 
 have threadsafe implementations (because that's the whole 
 point), what are the risks from allowing T* -> shared(T)* 
 conversion?

 All the risks that I think have been identified previously 
 assume that you can arbitrarily modify the data. That's 
 insanity... assume we fix that... I think the promotion 
 actually becomes safe now...?

 You're still talking about implicit promotion?

 Absolutely. This is critical to make shared useful, and I think 
 there's a path to make it work.

 No, it does not
 become safe no matter what restrictions you put on `shared`
 instances, because the caller of any function that takes 
 `shared`
 arguments remains blissfully unaware of this promotion.

 It doesn't matter... what danger is there of distributing a 
 shared pointer? Are you concerned that someone with a shared 
 reference might call a threadsafe method? If that's an invalid 
 operation, then the method has broken it's basic agreed 
 contract.

No, on the contrary. Someone with an unshared pointer may call 
unshared method or read/write data while someone else accesses it 
via `shared` interface precisely because you allow T to escape to 
shared(T). You *need* an explicit cast for this.

 And I agree that it's conceivable that you could contrive a bad 
 program, but you can contrive a bad program with literally any 
 language feature!

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 2:25 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 20:57:46 UTC, Manu wrote:
 On Mon, Oct 15, 2018 at 1:15 PM Stanislav Blinov via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:

 Assuming the rules above: "can't read or write to members",
 and the understanding that `shared` methods are expected to
 have threadsafe implementations (because that's the whole
 point), what are the risks from allowing T* -> shared(T)*
 conversion?

 All the risks that I think have been identified previously
 assume that you can arbitrarily modify the data. That's
 insanity... assume we fix that... I think the promotion
 actually becomes safe now...?

 You're still talking about implicit promotion?

 Absolutely. This is critical to make shared useful, and I think
 there's a path to make it work.

 No, it does not
 become safe no matter what restrictions you put on `shared`
 instances, because the caller of any function that takes
 `shared`
 arguments remains blissfully unaware of this promotion.

 It doesn't matter... what danger is there of distributing a
 shared pointer? Are you concerned that someone with a shared
 reference might call a threadsafe method? If that's an invalid
 operation, then the method has broken it's basic agreed
 contract.

 No, on the contrary. Someone with an unshared pointer may call
 unshared method or read/write data while someone else accesses it
 via `shared` interface precisely because you allow T to escape to
 shared(T). You *need* an explicit cast for this.

If a shared method is incompatible with an unshared method, your class
is broken.
Explicit casting doesn't magically implement thread-safety, it
basically just guarantees failure. What I suggest are rules that lead
to proper behaviour with respect to writing a thread-safe API.
You can write bad code with any feature in any number of ways.

I see it this way:
If your object has shared methods, then it is distinctly and
*deliberately* involved in thread-safety. You have deliberately
opted-in to writing a thread-safe object, and you must deliver on your
promise.
The un-shared API of an object that supports `shared` are not exempt
from the thread-safety commitment, they are simply the subset of the
API that may not be called from a shared context.
If your shared method is incompatible with other methods, your class
is broken, and you violate your promise.

Nobody writes methods of an object such that they don't work with each
other... methods are part of a deliberately crafted and packaged
entity. If you write a shared object, you do so deliberately, and you
buy responsibility of making sure your objects API is thread-safe.
If your object is not thread-safe, don't write shared methods.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Monday, 15 October 2018 at 21:51:43 UTC, Manu wrote:

 If a shared method is incompatible with an unshared method, 
 your class is broken.

What?!? So... my unshared methods should also perform all that's 
necessary for `shared` methods?

 Explicit casting doesn't magically implement thread-safety, it
 basically just guarantees failure.

It doesn't indeed. It does, however, at least help prevent silent 
bugs. Via that same guaranteed failure. That failure is about all 
the help we can get from the compiler anyway.

 What I suggest are rules that lead to proper behaviour with 
 respect to writing a thread-safe API.
 You can write bad code with any feature in any number of ways.

Yup. For example, passing an int* to a function expecting shared 
int*.

 I see it this way:
 If your object has shared methods, then it is distinctly and
 *deliberately* involved in thread-safety. You have deliberately
 opted-in to writing a thread-safe object, and you must deliver 
 on your promise.

 The un-shared API of an object that supports `shared` are not 
 exempt from the thread-safety commitment, they are simply the 
 subset of the API that may not be called from a shared context.

And therefore they lack any synchronization. So I don't see how 
they *can* be "compatible" with `shared` methods.

 If your shared method is incompatible with other methods, your 
 class is broken, and you violate your promise.

Nope.

class BigCounter {

     this() { /* don't even need the mutex if I'm not sharing this 
*/ }

     this(Mutex m = null) shared {
         this.m = m ? m : new Mutex;
     }

     void increment() { value += 1; }
     void increment() shared { synchronized(m) 
*value.assumeUnshared += 1; }

private:
     Mutex m;
     BigInt value;
}

They're not "compatible" in any shape or form. Or would you have 
the unshared ctor also create the mutex and unshared increment 
also take the lock? What's the point of having them  then? Better 
disallow mixed implementations altogether (which is actually not 
that bad of an idea).

 Nobody writes methods of an object such that they don't work 
 with each other... methods are part of a deliberately crafted 
 and packaged
 entity. If you write a shared object, you do so deliberately, 
 and you buy responsibility of making sure your objects API is 
 thread-safe.
 If your object is not thread-safe, don't write shared methods.

Ahem... Okay...

import std.concurrency;
import core.atomic;

void thread(shared int* x) {
     (*x).atomicOp!"+="(1);
}

shared int c;

void main() {
     int x;
     auto tid = spawn(&thread, &x); // "just" a typo
}

You're saying that's ok, it should "just" compile. It shouldn't. 
It should produce an error and a mild electric discharge into the 
developer's chair.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Monday, 15 October 2018 at 23:30:43 UTC, Stanislav Blinov 
wrote:
 On Monday, 15 October 2018 at 21:51:43 UTC, Manu wrote:

 If a shared method is incompatible with an unshared method, 
 your class is broken.

 What?!? So... my unshared methods should also perform all 
 that's necessary for `shared` methods?

No, its the other way around: a shared method that does extra 
synchronisation should work irrespective of wether or not the 
object needs that synchronisation. e.g. atomic loading a TLS 
variable is fine.

 Explicit casting doesn't magically implement thread-safety, it
 basically just guarantees failure.

 It doesn't indeed. It does, however, at least help prevent 
 silent bugs. Via that same guaranteed failure. That failure is 
 about all the help we can get from the compiler anyway.

 What I suggest are rules that lead to proper behaviour with 
 respect to writing a thread-safe API.
 You can write bad code with any feature in any number of ways.

 Yup. For example, passing an int* to a function expecting 
 shared int*.

That is a reasonable thing to do if shared is const + no 
unsynched reads.

 I see it this way:
 If your object has shared methods, then it is distinctly and
 *deliberately* involved in thread-safety. You have deliberately
 opted-in to writing a thread-safe object, and you must deliver 
 on your promise.

 The un-shared API of an object that supports `shared` are not 
 exempt from the thread-safety commitment, they are simply the 
 subset of the API that may not be called from a shared context.

 And therefore they lack any synchronization. So I don't see how 
 they *can* be "compatible" with `shared` methods.

I think Manu means you have a shared object with some shared 
methods and some unshared methods. The shared methods deal with 
synchronisation and can therefore be call from anywhere by 
anyone, whereas the unshared methods must be called on a locked 
object.

 snip
 Nobody writes methods of an object such that they don't work 
 with each other... methods are part of a deliberately crafted 
 and packaged
 entity. If you write a shared object, you do so deliberately, 
 and you buy responsibility of making sure your objects API is 
 thread-safe.
 If your object is not thread-safe, don't write shared methods.

 Ahem... Okay...

 import std.concurrency;
 import core.atomic;

 void thread(shared int* x) {
     (*x).atomicOp!"+="(1);
 }

 shared int c;

 void main() {
     int x;
     auto tid = spawn(&thread, &x); // "just" a typo
 }

 You're saying that's ok, it should "just" compile. It 
 shouldn't. It should produce an error and a mild electric 
 discharge into the developer's chair.

Indeed that is just a typo, just as that is a contrived example. 
You'd notice that pretty quick in a debugger.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 5:10 PM Nicholas Wilson via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 23:30:43 UTC, Stanislav Blinov
 wrote:
 On Monday, 15 October 2018 at 21:51:43 UTC, Manu wrote:

 I see it this way:
 If your object has shared methods, then it is distinctly and
 *deliberately* involved in thread-safety. You have deliberately
 opted-in to writing a thread-safe object, and you must deliver
 on your promise.

 The un-shared API of an object that supports `shared` are not
 exempt from the thread-safety commitment, they are simply the
 subset of the API that may not be called from a shared context.

 And therefore they lack any synchronization. So I don't see how
 they *can* be "compatible" with `shared` methods.

 I think Manu means you have a shared object with some shared
 methods and some unshared methods. The shared methods deal with
 synchronisation and can therefore be call from anywhere by
 anyone, whereas the unshared methods must be called on a locked
 object.

Yes, except maybe I didn't make it clear that I DO expect the
un-shared methods to be aware that a sibling shared method does exist
(you wrote it!), and that it may manipulate some state, so *if* the
un-shared method does interact with the same data that the shared
method may manipulate (in many cases, it won't; it's likely only a
small subset of an object's functionality that may have thread-safe
access), then the un-shared method does need to acknowledge that
functional overlap.
So even though a method is un-shared, it still needs to be aware that
it may have sibling methods that are shared. If they don't access an
overlapping data-set, no special handling is required. If they do
overlap, they may need to coordinate appropriately.

Isaac S. <spam-no-reply-isaac outlook.com> writes:

On Tuesday, 16 October 2018 at 00:36:12 UTC, Manu wrote:
 *snip*

 Yes, except maybe I didn't make it clear that I DO expect the
 un-shared methods to be aware that a sibling shared method does 
 exist
 (you wrote it!), and that it may manipulate some state, so *if* 
 the
 un-shared method does interact with the same data that the 
 shared
 method may manipulate (in many cases, it won't; it's likely 
 only a
 small subset of an object's functionality that may have 
 thread-safe
 access), then the un-shared method does need to acknowledge that
 functional overlap.
 So even though a method is un-shared, it still needs to be 
 aware that
 it may have sibling methods that are shared. If they don't 
 access an
 overlapping data-set, no special handling is required. If they 
 do
 overlap, they may need to coordinate appropriately.

I understand your point but I think the current shared (no 
implicit conversion) has its uses. It can be quite useful to have 
one interface for when an object is shared and one for when it is 
not (one with and without the synchronization cost). Sure, 
something as trivial as a counter can be re-implemented in both 
ways but more complex objects would easily result in extreme code 
duplication.

It's also easy to acknowledge that implicit conversion to shared 
has its uses.

Instead of forcing one way or another, how about we leave the 
decision up to the programmer? Say something like "alias this 
shared;" enables implicit conversion to shared.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 7:15 PM Isaac S. via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 16 October 2018 at 00:36:12 UTC, Manu wrote:
 *snip*

 Yes, except maybe I didn't make it clear that I DO expect the
 un-shared methods to be aware that a sibling shared method does
 exist
 (you wrote it!), and that it may manipulate some state, so *if*
 the
 un-shared method does interact with the same data that the
 shared
 method may manipulate (in many cases, it won't; it's likely
 only a
 small subset of an object's functionality that may have
 thread-safe
 access), then the un-shared method does need to acknowledge that
 functional overlap.
 So even though a method is un-shared, it still needs to be
 aware that
 it may have sibling methods that are shared. If they don't
 access an
 overlapping data-set, no special handling is required. If they
 do
 overlap, they may need to coordinate appropriately.

 I understand your point but I think the current shared (no
 implicit conversion) has its uses. It can be quite useful to have
 one interface for when an object is shared and one for when it is
 not (one with and without the synchronization cost). Sure,
 something as trivial as a counter can be re-implemented in both
 ways but more complex objects would easily result in extreme code
 duplication.

If you can give a single 'use', I'm all ears ;)

 It's also easy to acknowledge that implicit conversion to shared
 has its uses.

I actually know of many real uses for this case.

 Instead of forcing one way or another, how about we leave the
 decision up to the programmer? Say something like "alias this
 shared;" enables implicit conversion to shared.

That might be fine.
Like I said in OP, the first point that I think needs to be agreed on,
is that shared can not read or write members. I think that's a
pre-requisite for any interesting development.

Isaac S. <spam-no-reply-isaac outlook.com> writes:

On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote:
 I understand your point but I think the current shared (no 
 implicit conversion) has its uses. It can be quite useful to 
 have one interface for when an object is shared and one for 
 when it is not (one with and without the synchronization 
 cost). Sure, something as trivial as a counter can be 
 re-implemented in both ways but more complex objects would 
 easily result in extreme code duplication.

 If you can give a single 'use', I'm all ears ;)

My usages are a custom ref counted template and list types (which 
are built on top of the former). The ref counted template will 
initialize naively when not shared but utilize compare-and-set 
and thread yielding if needed when shared (ensureInitialized can 
occur any time after declaration). The list types (which are not 
shared-compatible *yet*) will utilize a read-write mutex only 
when shared (no cost beyond a little unused memory to non-shared 
objects). As such, casting any of the non-shared versions of 
these types to shared would be unsafe.

(Technically the types can be safely casted to shared so long as 
no non-shared
reference to them exists and vice versa.)

 It's also easy to acknowledge that implicit conversion to 
 shared has its uses.

 I actually know of many real uses for this case.

(There was a time when I wanted it as well. I've forgotten what 
for by now.)

 That might be fine.
 Like I said in OP, the first point that I think needs to be 
 agreed on,
 is that shared can not read or write members. I think that's a
 pre-requisite for any interesting development.

I will agree that *outsiders* should not be able to read/write 
members of a shared object. I'm not sure whether your design 
means even within a shared function such members cannot be read 
from or written to. If it does, the casting required may get a 
little annoying but isn't unworkable (especially since an 
unshared template can do the cast in an  trusted manner).

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 8:55 PM Isaac S. via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote:
 I understand your point but I think the current shared (no
 implicit conversion) has its uses. It can be quite useful to
 have one interface for when an object is shared and one for
 when it is not (one with and without the synchronization
 cost). Sure, something as trivial as a counter can be
 re-implemented in both ways but more complex objects would
 easily result in extreme code duplication.

 If you can give a single 'use', I'm all ears ;)

 My usages are a custom ref counted template and list types (which
 are built on top of the former). The ref counted template will
 initialize naively when not shared but utilize compare-and-set
 and thread yielding if needed when shared (ensureInitialized can
 occur any time after declaration). The list types (which are not
 shared-compatible *yet*) will utilize a read-write mutex only
 when shared (no cost beyond a little unused memory to non-shared
 objects). As such, casting any of the non-shared versions of
 these types to shared would be unsafe.

 (Technically the types can be safely casted to shared so long as
 no non-shared
 reference to them exists and vice versa.)

Can you link to code? It doesn't sound incompatible with my proposal.
Mutexes are blunt instruments, and easy to model. My whole suggestion
has virtually no effect on the mutex protected case, which is what
most people seem to talk about.

 That might be fine.
 Like I said in OP, the first point that I think needs to be
 agreed on,
 is that shared can not read or write members. I think that's a
 pre-requisite for any interesting development.

 I will agree that *outsiders* should not be able to read/write
 members of a shared object. I'm not sure whether your design
 means even within a shared function such members cannot be read
 from or written to.

Absolutely. Shared references can not access members, that's the rule.
It's completely unsafe to read or write to members of a shared thing. Period.
Attributing a method shared doesn't change that fact.

You can create a thread-local (ie, accessible) object from a shared
object by acquiring a lock on it for some duration, and it would be
helpful if the mechanism that fetches the lock also cast away shared
as part of the lock operation.

 If it does, the casting required may get a
 little annoying but isn't unworkable (especially since an
 unshared template can do the cast in an  trusted manner).

I agree users shouldn't write casts explicitly, they should use a lock
helper or something that grabs the lock and returns the un-shared
reference.
The existing behaviour is totally unacceptable though; shared objects
can read/write arbitrarily... what's the point of that? It gives the
impression that it's okay, but it's completely broken.
There is no situation where a shared thing can arbitrarily access its members.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 4:35 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 21:51:43 UTC, Manu wrote:

 If a shared method is incompatible with an unshared method,
 your class is broken.

 What?!? So... my unshared methods should also perform all that's
 necessary for `shared` methods?

Of course! You're writing a threadsafe object... how could you expect otherwise?

 Explicit casting doesn't magically implement thread-safety, it
 basically just guarantees failure.

 It doesn't indeed. It does, however, at least help prevent silent
 bugs. Via that same guaranteed failure. That failure is about all
 the help we can get from the compiler anyway.

Just to be clear, what I'm suggesting is a significant *restriction*
to what shared already does... there will be a whole lot more safety
under my proposal.
The cast gives exactly nothing that attributing a method as shared
doesn't give you, except that attributing a method shared is so much
more sanitary and clearly communicates intent at the API level.

 What I suggest are rules that lead to proper behaviour with
 respect to writing a thread-safe API.
 You can write bad code with any feature in any number of ways.

 Yup. For example, passing an int* to a function expecting shared
 int*.

I don't understand your example. What's the problem you're suggesting?

 I see it this way:
 If your object has shared methods, then it is distinctly and
 *deliberately* involved in thread-safety. You have deliberately
 opted-in to writing a thread-safe object, and you must deliver
 on your promise.

 The un-shared API of an object that supports `shared` are not
 exempt from the thread-safety commitment, they are simply the
 subset of the API that may not be called from a shared context.

 And therefore they lack any synchronization. So I don't see how
 they *can* be "compatible" with `shared` methods.

I don't understand this statement either. Who said they lack
synchronisation? If they need it, they will have it.
There's a good chance they don't need it though, they might not
interact with a thread-unsafe portion of the class.

 If your shared method is incompatible with other methods, your
 class is broken, and you violate your promise.

 Nope.

So certain...

 class BigCounter {

      this() { /* don't even need the mutex if I'm not sharing this
 */ }

      this(Mutex m = null) shared {
          this.m = m ? m : new Mutex;
      }

      void increment() { value += 1; }
      void increment() shared { synchronized(m)
 *value.assumeUnshared += 1; }

 private:
      Mutex m;
      BigInt value;
 }

You've just conflated 2 classes into one. One is a threadlocal
counter, the other is a threadsafe counter. Which is it?
Like I said before: "you can contrive a bad program with literally any
language feature!"

 They're not "compatible" in any shape or form.

Correct, you wrote 2 different things and mashed them together.

 Or would you have
 the unshared ctor also create the mutex and unshared increment
 also take the lock? What's the point of having them  then? Better
 disallow mixed implementations altogether (which is actually not
 that bad of an idea).

Right. This is key to my whole suggestion. If you write a shared
thing, you accept that it's shared! You don't just accept it, you jam
the stake in the ground.
There's a relatively small number of things that need to be
threadsafe, you won't see `shared` methods appearing at random. If you
use shared, you promise threadsafety OR the members of the thing are
inaccessible without some sort of lock-&-cast-away treatment.

 Nobody writes methods of an object such that they don't work
 with each other... methods are part of a deliberately crafted
 and packaged
 entity. If you write a shared object, you do so deliberately,
 and you buy responsibility of making sure your objects API is
 thread-safe.
 If your object is not thread-safe, don't write shared methods.

 Ahem... Okay...

 import std.concurrency;
 import core.atomic;

 void thread(shared int* x) {
      (*x).atomicOp!"+="(1);
 }

 shared int c;

 void main() {
      int x;
      auto tid = spawn(&thread, &x); // "just" a typo
 }

 You're saying that's ok, it should "just" compile. It shouldn't.
 It should produce an error and a mild electric discharge into the
 developer's chair.

Yup. It's a typo. You passed a stack pointer to a scope that outlives
the caller.
That class of issue is not on trial here. There's DIP1000, and all
sorts of things to try and improve safety in terms of lifetimes.
You only managed to contrive this by spawning a thread. If it were
just a normal function, this would be perfectly legitimate, and again,
that's my whole point.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Tuesday, 16 October 2018 at 00:15:54 UTC, Manu wrote:
 On Mon, Oct 15, 2018 at 4:35 PM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 What?!? So... my unshared methods should also perform all 
 that's necessary for `shared` methods?


 Of course! You're writing a threadsafe object... how could you 
 expect otherwise?

See below.

 Just to be clear, what I'm suggesting is a significant 
 *restriction*
 to what shared already does... there will be a whole lot more 
 safety under my proposal.

I don't see how an *implicit* cast can be a restriction. At all.

 The cast gives exactly nothing that attributing a method as 
 shared
 doesn't give you, except that attributing a method shared is so 
 much
 more sanitary and clearly communicates intent at the API level.

It's like we're talking about wholly different things here. 
Casting should be done by the caller, i.e. a programmer that uses 
some API. If that API expects shared arguments, the caller better 
make sure they pass shared values. Implicit conversion destroys 
any obligations between the caller and the API.

 You can write bad code with any feature in any number of 
 ways.

 Yup. For example, passing an int* to a function expecting 
 shared int*.

 I don't understand your example. What's the problem you're 
 suggesting?

The problem that I'm suggesting is exactly that: an `int*` is 
not, and can not, be a `shared int*` at the same time. Substitute 
int for any type. But D is not Rust and it can't statically 
prevent that, except for disallowing trivial programming 
mistakes, which, with implicit conversion introduced, would also 
go away.

 ...And therefore they lack any synchronization. So I don't see 
 how they *can* be "compatible" with `shared` methods.

 I don't understand this statement either. Who said they lack 
 synchronisation? If they need it, they will have it. There's a 
 good chance they don't need it though, they might not  interact 
 with a thread-unsafe portion of the class.

Or they might.

 If your shared method is incompatible with other methods, 
 your class is broken, and you violate your promise.

 Nope.

 So certain...

 class BigCounter {

      this() { /* don't even need the mutex if I'm not sharing 
 this
 */ }

      this(Mutex m = null) shared {
          this.m = m ? m : new Mutex;
      }

      void increment() { value += 1; }
      void increment() shared { synchronized(m)
 *value.assumeUnshared += 1; }

 private:
      Mutex m;
      BigInt value;
 }

 You've just conflated 2 classes into one. One is a threadlocal
 counter, the other is a threadsafe counter. Which is it?
 Like I said before: "you can contrive a bad program with 
 literally any language feature!"

Because that is exactly the code that a good amount of 
"developers" will write. Especially those of the "don't think 
about it" variety. Don't be mistaken for a second: if the 
language allows it, they'll write it.

 They're not "compatible" in any shape or form.

 Correct, you wrote 2 different things and mashed them together.

Can you actually provide an example of a mixed shared/unshared 
class that even makes sense then? As I said, at this point I'd 
rather see such definitions prohibited entirely.

 Or would you have
 the unshared ctor also create the mutex and unshared increment 
 also take the lock? What's the point of having them  then? 
 Better disallow mixed implementations altogether (which is 
 actually not that bad of an idea).


 Right. This is key to my whole suggestion. If you write a 
 shared thing, you accept that it's shared! You don't just 
 accept it, you jam the stake in the ground.

Then, once more, `shared` should then just be a type qualifier 
exclusively, and mixing shared/unshared methods should just not 
be allowed.

 There's a relatively small number of things that need to be
 threadsafe, you won't see `shared` methods appearing at random. 
 If you use shared, you promise threadsafety OR the members of 
 the thing are inaccessible without some sort of 
 lock-&-cast-away treatment.

As above.

 import std.concurrency;
 import core.atomic;

 void thread(shared int* x) {
      (*x).atomicOp!"+="(1);
 }

 shared int c;

 void main() {
      int x;
      auto tid = spawn(&thread, &x); // "just" a typo
 }

 You're saying that's ok, it should "just" compile. It 
 shouldn't. It should produce an error and a mild electric 
 discharge into the developer's chair.

 Yup. It's a typo. You passed a stack pointer to a scope that 
 outlives the caller.
 That class of issue is not on trial here. There's DIP1000, and 
 all sorts of things to try and improve safety in terms of 
 lifetimes.

I'm sorry, I'm not very good at writing "real" examples for 
things that don't exist or don't compile. End of sarcasm.

Let's come back to DIP1000 when it's actually implemented in it's 
entirety, ok? Anyway, you're nitpicking while actually missing 
the point altogether. The way `shared` is "implemented" today, 
the API (`thread` function) *requires* the caller to pass a 
`shared int*`. Implicit conversion breaks that contract.

At the highest level, the only reason for taking a `shared` 
argument is to pass that argument to another thread. That is the 
*only* way to communicate that intent via the type system for the 
time being. You're suggesting to ignore that fact. `shared` was 
supposed to protect from unshared aliasing, not silently allow it.
If you allow implicit conversion, there would literally be no way 
of knowing whether some API will access your data concurrently, 
other than plain old documentation (or sifting through it's code, 
which may not be available). This makes `shared` useless as a 
type qualifier.

 You only managed to contrive this by spawning a thread. If it 
 were just a normal function, this would be perfectly 
 legitimate, and again, that's my whole point.

I think you will agree that passing a pointer to a thread-local 
variable to another thread is not always a safe thing to do. 
Conditions do apply, which are on you (the programmer) to uphold, 
and the compiler can't help you with that. The only way the 
compiler *can* help you here is make sure you don't do that 
unintentionally. Which it won't be able to do if you allow such 
implicit conversion.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 15, 2018 at 7:25 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 16 October 2018 at 00:15:54 UTC, Manu wrote:
 On Mon, Oct 15, 2018 at 4:35 PM Stanislav Blinov via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 What?!? So... my unshared methods should also perform all
 that's necessary for `shared` methods?


 Of course! You're writing a threadsafe object... how could you
 expect otherwise?

 See below.

 Just to be clear, what I'm suggesting is a significant
 *restriction*
 to what shared already does... there will be a whole lot more
 safety under my proposal.

 I don't see how an *implicit* cast can be a restriction. At all.

Because a shared pointer can't access anything.
You can't do anything with a shared instance, so the can be no harm done.

Only if there are shared methods (that promise thread-safety) is it
that shared gets interesting.
Without that, it's just a market for the existing recommended use of
shared; which is lock and cast away.

 The cast gives exactly nothing that attributing a method as
 shared
 doesn't give you, except that attributing a method shared is so
 much
 more sanitary and clearly communicates intent at the API level.

 It's like we're talking about wholly different things here.
 Casting should be done by the caller, i.e. a programmer that uses
 some API. If that API expects shared arguments, the caller better
 make sure they pass shared values. Implicit conversion destroys
 any obligations between the caller and the API.

Why? What could a function do with shared arguments?

 You can write bad code with any feature in any number of
 ways.

 Yup. For example, passing an int* to a function expecting
 shared int*.

 I don't understand your example. What's the problem you're
 suggesting?

 The problem that I'm suggesting is exactly that: an `int*` is
 not, and can not, be a `shared int*` at the same time. Substitute
 int for any type. But D is not Rust and it can't statically
 prevent that, except for disallowing trivial programming
 mistakes, which, with implicit conversion introduced, would also
 go away.

Why not? The guy who receives the argument receives an argument that
*may be shared*, and as such, he's restricted access to it
appropriately.
Just like if you receive a const thing, you can't write to it, even if
the caller's thing isn't const.
If you receive a shared thing, you can't read or write to it.

 ...And therefore they lack any synchronization. So I don't see
 how they *can* be "compatible" with `shared` methods.

 I don't understand this statement either. Who said they lack
 synchronisation? If they need it, they will have it. There's a
 good chance they don't need it though, they might not  interact
 with a thread-unsafe portion of the class.

 Or they might.

Then you will implement synchronisation, or have violated your
thread-safety promise.

 If your shared method is incompatible with other methods,
 your class is broken, and you violate your promise.

 Nope.

 So certain...

 class BigCounter {

      this() { /* don't even need the mutex if I'm not sharing
 this
 */ }

      this(Mutex m = null) shared {
          this.m = m ? m : new Mutex;
      }

      void increment() { value += 1; }
      void increment() shared { synchronized(m)
 *value.assumeUnshared += 1; }

 private:
      Mutex m;
      BigInt value;
 }

 You've just conflated 2 classes into one. One is a threadlocal
 counter, the other is a threadsafe counter. Which is it?
 Like I said before: "you can contrive a bad program with
 literally any language feature!"

 Because that is exactly the code that a good amount of
 "developers" will write. Especially those of the "don't think
 about it" variety. Don't be mistaken for a second: if the
 language allows it, they'll write it.

This is not even an argument.
Atomic!int must be used with care. Any threading of ANY KIND must be
handled with care.
Saying we shouldn't make shared useful because someone can do
something wrong is like saying we shouldn't have atomic int's and we
shouldn't have spawn(). They're simply too dangerous to give to
users...

 They're not "compatible" in any shape or form.

 Correct, you wrote 2 different things and mashed them together.

 Can you actually provide an example of a mixed shared/unshared
 class that even makes sense then? As I said, at this point I'd
 rather see such definitions prohibited entirely.

I think this is a typical sort of construction:

struct ThreadsafeQueue(T)
{
  void QueueItem(T*) shared;
  T* UnqueueItem() shared;
}

struct SpecialWorkList
{
  struct Job { ... }

  void MakeJob(int x, float y, string z) shared  // <- any thread may
produce a job
  {
    Job* job = new Job; // <- this is thread-local
    PopulateJob(job, x, y, z); // <- preparation of a job might be
complex, and worthy of the SpecialWorkList implementation

    jobList.QueueItem(job);  // <- QueueItem encapsulates
thread-safety, no need for blunt casts
  }

  void Flush() // <- not shared, thread-local consumer
  {
    Job* job;
    while (job = jobList.UnqueueItem()) // <- it's obviously safe for
a thread-local to call UnqueueItem even though the implementation is
threadsafe
    {
      // thread-local dispatch of work...
      // perhaps rendering, perhaps deferred destruction, perhaps
deferred resource creation... whatever!
    }
  }

  void GetSpecialSystemState() // <- this has NOTHING to do with the
threadsafe part of SpecialWorkList
  {
    return os.functionThatChecksSystemState();
  }

  // there may be any number of utility functions that don't interact
with jobList.

private:
  void PopulateJob(ref Job job, ...)
  {
    // expensive function; not thread-safe, and doesn't have any
interaction with threading.
  }

  ThreadsafeQueue!Job jobList;
}


This isn't an amazing example, but it's typical of a thing that's
mostly thread-local, and only a small controlled part of it's
functionality is thread-safe.
The thread-local method Flush() also deals with thread-safety
internally... because it flushes a thread-safe queue.

All thread-safety concerns are composed by a utility object, so
there's no need for locks, magic, or casts here.

 Or would you have
 the unshared ctor also create the mutex and unshared increment
 also take the lock? What's the point of having them  then?
 Better disallow mixed implementations altogether (which is
 actually not that bad of an idea).


 Right. This is key to my whole suggestion. If you write a
 shared thing, you accept that it's shared! You don't just
 accept it, you jam the stake in the ground.

 Then, once more, `shared` should then just be a type qualifier
 exclusively, and mixing shared/unshared methods should just not
 be allowed.

1. No.
2. I would have to repeat literally everything I've ever said on this
topic to respond to this comment.

 There's a relatively small number of things that need to be
 threadsafe, you won't see `shared` methods appearing at random.
 If you use shared, you promise threadsafety OR the members of
 the thing are inaccessible without some sort of
 lock-&-cast-away treatment.

 As above.

I don't understand.

 import std.concurrency;
 import core.atomic;

 void thread(shared int* x) {
      (*x).atomicOp!"+="(1);
 }

 shared int c;

 void main() {
      int x;
      auto tid = spawn(&thread, &x); // "just" a typo
 }

 You're saying that's ok, it should "just" compile. It
 shouldn't. It should produce an error and a mild electric
 discharge into the developer's chair.

 Yup. It's a typo. You passed a stack pointer to a scope that
 outlives the caller.
 That class of issue is not on trial here. There's DIP1000, and
 all sorts of things to try and improve safety in terms of
 lifetimes.

 I'm sorry, I'm not very good at writing "real" examples for
 things that don't exist or don't compile. End of sarcasm.

 Let's come back to DIP1000 when it's actually implemented in it's
 entirety, ok? Anyway, you're nitpicking while actually missing
 the point altogether. The way `shared` is "implemented" today,
 the API (`thread` function) *requires* the caller to pass a
 `shared int*`. Implicit conversion breaks that contract.

So?
What does it mean to pass a `shared int*`?

 At the highest level, the only reason for taking a `shared`
 argument is to pass that argument to another thread.

Not even. This is the most un-useful application I can think of. It
doesn't really model the problem at all.
Transfer of ownership is a job for move semantics.
shared is for interacting with objects that are *already* owned by many threads.
shared needs to model mechanics to do a limited set of thread-safe
interactions with shared objects that are shared. That would make
shared a useful thing, rather than a giant stain.

 That is the
 *only* way to communicate that intent via the type system for the
 time being.

...but that's shit. And it doesn't communicate that intent at all.
Bluntly casting attributes on things is a terrible solution to that
proposed problem.

 You're suggesting to ignore that fact.

Yes; everything we think about shared today is completely worthless.
Under my proposal, some existing applications might remain untouched
(they do), but they're not worth worrying about from a design point of
view, because they're not really 'designs'.
Focus on making shared a useful thing, and then see where we're at.

 `shared` was
 supposed to protect from unshared aliasing, not silently allow it.

Inhibiting all access satisfies that protection. It doesn't matter if
a pointer is distributed if you can't access the contents.
Now from there, we need a way to make interacting with guaranteed
thread-safe API's interesting and useful, and I'm describing how to do
that.

 If you allow implicit conversion, there would literally be no way
 of knowing whether some API will access your data concurrently,
 other than plain old documentation (or sifting through it's code,
 which may not be available). This makes `shared` useless as a
 type qualifier.

That's the whole point though.
A thread-safe think couldn't care less if the data is shared or not,
because it's threadsafe.
Now we're able to describe what's thread-safe, and what's not. This
makes shared *useful* as a type qualifier.

 You only managed to contrive this by spawning a thread. If it
 were just a normal function, this would be perfectly
 legitimate, and again, that's my whole point.

 I think you will agree that passing a pointer to a thread-local
 variable to another thread is not always a safe thing to do.

That's the problem I'm trying to resolve by removing all access.
I'm trying to make that interaction safe, and that's the key to moving
forward as I see it.
If the object is thread-local, then no other thread can access the
object in any way, and it's just a fancy int.

 Conditions do apply, which are on you (the programmer) to uphold,
 and the compiler can't help you with that. The only way the
 compiler *can* help you here is make sure you don't do that
 unintentionally. Which it won't be able to do if you allow such
 implicit conversion.

You need to demonstrate how the implicit conversion may lead to chaos.
The conversion is immensely useful, and I haven't thought how it's a
problem yet.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Tuesday, 16 October 2018 at 03:00:21 UTC, Manu wrote:

 I don't see how an *implicit* cast can be a restriction. At 
 all.


 Because a shared pointer can't access anything.
 You can't do anything with a shared instance, so the can be no 
 harm done.

That just doesn't compute. You obviously *can* do "anything" with 
a shared instance, per your own requirements all you need is 
methods. But at this point, the caller has an unshared reference, 
and your API possibliy stole away a shared one without the caller 
ever knowing of it happening.

 It's like we're talking about wholly different things here. 
 Casting should be done by the caller, i.e. a programmer that 
 uses some API. If that API expects shared arguments, the 
 caller better make sure they pass shared values. Implicit 
 conversion destroys any obligations between the caller and the 
 API.

 Why? What could a function do with shared arguments?

How, exactly, do you propose getting shared state from one thread 
to another? Exclusively through globals?

 The problem that I'm suggesting is exactly that: an `int*` is 
 not, and can not, be a `shared int*` at the same time. 
 Substitute int for any type. But D is not Rust and it can't 
 statically prevent that, except for disallowing trivial 
 programming mistakes, which, with implicit conversion 
 introduced, would also go away.

 Why not? The guy who receives the argument receives an argument 
 that
 *may be shared*, and as such, he's restricted access to it 
 appropriately.

But the guy that *provides* that argument may have no idea of 
this happening. This is unshared aliasing.

module yourapi;

// My code does not know about this function at all
void giveToThread(shared int* ptr) { /* ... */ }

void yourAPI(int* ptr) {
     giveToThread(ptr);
}

module mycode;

int x;

void main() {
     yourAPI(&x);
}

 Just like if you receive a const thing, you can't write to it, 
 even if the caller's thing isn't const.

You do know why those Rust guys disallowed mutable aliasing? That 
is, having both mutable and immutable references at the same 
time. That's a long known problem that in C++ and D is only 
"solved" by programmer discipline and not much else.

 If you receive a shared thing, you can't read or write to it.

You can, per your model, through methods. All the while the 
original is being read and written freely.

 a good chance they don't need it though, they might not  
 interact with a thread-unsafe portion of the class.

 Or they might.

 Then you will implement synchronisation, or have violated your 
 thread-safety promise.

Such a good promise it is when it's simply ignored by an implicit 
cast.

 Because that is exactly the code that a good amount of 
 "developers" will write. Especially those of the "don't think 
 about it" variety. Don't be mistaken for a second: if the 
 language allows it, they'll write it.

 This is not even an argument.
 Atomic!int must be used with care. Any threading of ANY KIND 
 must be
 handled with care.

Implicit casts are in another galaxy as far as handling with care 
is concerned.

 Saying we shouldn't make shared useful because someone can do 
 something wrong is like saying we shouldn't have atomic int's 
 and we shouldn't have spawn(). They're simply too dangerous to 
 give to users...

We should make `shared` useful. We shouldn't allow unshared 
aliasing.

 Can you actually provide an example of a mixed shared/unshared 
 class that even makes sense then? As I said, at this point I'd 
 rather see such definitions prohibited entirely.

 I think this is a typical sort of construction:

 struct ThreadsafeQueue(T)
 {
   void QueueItem(T*) shared;
   T* UnqueueItem() shared;
 }

That's an interface for a multi-producer multi-consumer, yet 
you're using it as a multi-producer single-consumer. Those would 
have very different implementations.

 struct SpecialWorkList
 {
   struct Job { ... }

   void MakeJob(int x, float y, string z) shared  // <- any 
 thread may produce a job
   {
     Job* job = new Job; // <- this is thread-local
     PopulateJob(job, x, y, z); // <- preparation of a job might 
 be complex, and worthy of the SpecialWorkList implementation

...except you can't call PopulateJob from MakeJob.

The two methods after compiler rewrite are (pseudocode):

void struct_method_SpecialWorkList_MakeJob(shared 
SpecialWorkList* this, int x, float y, string z);
void struct_method_SpecialWorkList_PopulateJob(SpecialWorkList* 
this, Job* job, ...);

Or are you also suggesting to allow implicitly demoting 
shared(T)* to T* ?!? Then you can just throw away `shared`.

     jobList.QueueItem(job);  // <- QueueItem encapsulates 
 thread-safety, no need for blunt casts

None were needed here regardless.

   void Flush() // <- not shared, thread-local consumer
   {

I.e. this can only be called by the thread that owns this 
instance, because only that thread can have an unshared reference 
to it.

   void GetSpecialSystemState() // <- this has NOTHING to do 
 with the threadsafe part of SpecialWorkList
   {
     return os.functionThatChecksSystemState();
   }

Therefore it's either static or out of SpecialWorkList. As 
someone in strong opposition to conflation you should've seen 
that.

   // there may be any number of utility functions that don't 
 interact with jobList.

So have a separate interface for them, why put them in the same 
namespace then?

 private:
   void PopulateJob(ref Job job, ...)
   {
     // expensive function; not thread-safe, and doesn't have any
 interaction with threading.

And cannot be called from `shared` functions without at least 
casting away `shared`.

 This isn't an amazing example, but it's typical of a thing  
 that's
 mostly thread-local, and only a small controlled part of it's
 functionality is thread-safe.
 The thread-local method Flush() also deals with thread-safety
 internally... because it flushes a thread-safe queue.



 All thread-safety concerns are composed by a utility object, so 
 there's no need for locks, magic, or casts here.

// Casts:
auto assumeUnshared(T)(ref shared(T) x) { /* ... */ }

struct MPSCQueue(T) {
     void put(T*) shared;
     T* remove();
     // perhaps also this one:
     T* steal() shared;
}

struct Job { /* ... */ }

struct JobList {

     void addJob(Args...)(Args args) shared {
         auto job = new Job;
         this.assumeUnshared.populateJob(*job, forward!args);
         jobs.put(job);
     }

     Job* removeJob() {
         return jobs.remove();
     }

     Job* stealJob() shared {
        return jobs.steal();
     }

private:

     void populateJob(ref Job job, ...) {
         /* ... */
     }

     MPSCQueue!Job jobs;
}

struct SpecialWorkList {

     shared JobList* getJobList() { return jobList; }

     // here's an implicit cast to the actually "shared" data if 
you need one:
     //alias getJobList this;

     void flush() {
         auto list = jobList.assumeUnshared;
         Job* job;
         while ((job = list.removeJob) !is null) {
             // ...
         }
     }

     // put anything non-shared here...

private:

     // Even though your 'SpecialWorkList' is unshared,
     // you're sharing at least this reference with other threads:
     shared JobList* jobList;
}

SpecialWorkList makeSpecialWorkList() {
     SpecialWorkList result;
     // whatever, allocate the jobList, initialize other state, 
etc.
     return result;
}

void worker(shared int* keepMeltingCPU, shared JobList* jobList) {
     while (true) {
         if (!keepMeltingCPU.atomicLoad()) break;

         // Do some heavy processing...

         jobList.addJob(/* arguments */);
         jobList.addJob(/* arguments */);
     }
}

void thief(shared int* keepMeltingCPU, shared JobList* jobList) {
     while (true) {
         if (!keepMeltingCPU.atomicLoad()) break;
         jobList.addJob(/* arguments */);
         jobList.addJob(/* arguments */);
         /* ... */
         // help with jobs
         Job* job;
         while ((job = jobList.steal) !is null) {
             // ...
         }
     }
}

bool readInput() { /* ... */ }

void main() {
     Thread[2] threads;
     shared int[2] flags = [ 1, 1 ];
     SpecialWorkList list = makeSpecialWorkList();
     threads[0] = someFunctionThatMakesAThread(&worker, &flags[0], 
list.getJobList);
     threads[1] = someFunctionThatMakesAThread(&thief, &flags[1], 
list.getJobList);

     while (true) {
         list.flush();
         if (!readInput()) break;
     }

     foreach (ref flag; flags) {
         flag.atomicStore(0);
     }

     foreach (t; threads) t.join();
}

As you can see, the amount of casting is minimal. It is explicit 
and documenting the intent. And all the casts are actually the 
opposite way of what you're proposing. That's because the design 
lends to just not letting you share thread-local data. The 
"problem" here is that you can't add jobs from main thread 
without adding an unshared overload. Which is trivial, but would 
require a cast.

There are, however, deeper problems that don't have anything to 
do with the `shared` keyword, and start with the call "new Job". 
I could assume it's just for the sake of example, and you'd 
actually store the jobs:

a) not as pointers
b) in aligned and padded storage

but since we're nitpicking, and you're very specific in that 
approach, I'm not sure if I'm safe making such an assumption.

 missing the point altogether. The way `shared` is 
 "implemented" today, the API (`thread` function) *requires* 
 the caller to pass a `shared int*`. Implicit conversion breaks 
 that contract.

 So?
 What does it mean to pass a `shared int*`?

Exactly what it says: passing a pointer to a shared int. Which is 
what int* is not. Adding `shared` to a type is not the same thing 
as "promoting" a mutable to a const, which, the way that's done 
in D, isn't that awesome a deal to begin with.

 At the highest level, the only reason for taking a `shared` 
 argument is to pass that argument to another thread.

 Not even. This is the most un-useful application I can think 
 of. It doesn't really model the problem at all.

There's no feasible alternative in the current type system.

 Transfer of ownership is a job for move semantics.

There is no transfer of ownership when you're *sharing* data.

 shared is for interacting with objects that are *already* owned 
 by many threads.

There's only one owner, usually a thread that instantiated the 
thing in the first place. Ergo, there needs to be a way to 
"share" that thing. Explicitly.

 shared needs to model mechanics to do a limited set of 
 thread-safe
 interactions with shared objects that are shared. That would 
 make
 shared a useful thing, rather than a giant stain.

Agreed.

 That is the *only* way to communicate that intent via the type 
 system for the
 time being.

 ...but that's shit. And it doesn't communicate that intent at 
 all. Bluntly casting attributes on things is a terrible 
 solution to that proposed problem.

It's not a solution, it's a crutch. Presumably the programmer 
would know whether or not that cast is safe in that particular 
context.
Can you propose a better way to distinguish shared and 
thread-local instances without introducing "thread" built-in type 
(at which point we should all probably just stop using D)? So 
long as threading is purely library code, there isn't exactly 
much room for that.

 Yes; everything we think about shared today is completely  
 worthless.
 Under my proposal, some existing applications might remain 
 untouched
 (they do), but they're not worth worrying about from a design 
 point of view, because they're not really 'designs'.
 Focus on making shared a useful thing, and then see where we're 
 at.

I quite like the idea of disallowing member access wholesale. For 
example, things like assignment, copy/postblit should be 
 disabled for anything `shared` by default, shared destructors 
should not exist. But this doesn't *really* solve the problem.

 `shared` was
 supposed to protect from unshared aliasing, not silently allow 
 it.

 Inhibiting all access satisfies that protection. It doesn't 
 matter if
 a pointer is distributed if you can't access the contents.
 Now from there, we need a way to make interacting with 
 guaranteed
 thread-safe API's interesting and useful, and I'm describing 
 how to do that.

An API cannot be guaranteed thread-safe without some kind of 
contract in it's signature.

 If you allow implicit conversion, there would literally be no 
 way of knowing whether some API will access your data 
 concurrently, other than plain old documentation (or sifting 
 through it's code, which may not be available). This makes 
 `shared` useless as a type qualifier.


 That's the whole point though.
 A thread-safe think couldn't care less if the data is shared or 
 not,
 because it's threadsafe.
 Now we're able to describe what's thread-safe, and what's not. 
 This makes shared *useful* as a type qualifier.

You're looking at it backwards, I assume because you keep 
thinking about `shared` in terms of `const`. `shared` thing cares 
if the data is shared or not, to which with implicit casting in 
place there is no compile-time check.

 I think you will agree that passing a pointer to a 
 thread-local variable to another thread is not always a safe 
 thing to do.


 That's the problem I'm trying to resolve by removing all access.
 I'm trying to make that interaction safe, and that's the key to 
 moving forward as I see it.
 If the object is thread-local, then no other thread can access 
 the object in any way, and it's just a fancy int.

Not true if you allow to silently cast a pointer to that fancy 
int to a pointer to a shared int.

 Conditions do apply, which are on you (the programmer) to 
 uphold, and the compiler can't help you with that. The only 
 way the compiler *can* help you here is make sure you don't do 
 that unintentionally. Which it won't be able to do if you 
 allow such implicit conversion.


 You need to demonstrate how the implicit conversion may lead to 
 chaos. The conversion is immensely useful, and I haven't 
 thought how it's a problem yet.

I already have. With your implicit promotions, you can literally 
pass off anything as shared. No amount of DIP1000s will safeguard 
from this, because the only ways to pass data to threads is to 
either only use globals, or escape it. Therefore the language 
needs some way of annotating that intent. Currently, `shared` 
does that. If you don't like that, there needs to be some 
alternative.

Timon Gehr <timon.gehr gmx.ch> writes:

On 15.10.2018 23:51, Manu wrote:
 If a shared method is incompatible with an unshared method, your class
 is broken.

Then what you want is not implicit unshared->shared conversion. What you 
want is a different way to type shared member access. You want a setup 
where shared methods are only allowed to access shared members and 
unshared methods are only allowed to access unshared members.

I.e., what you want is that shared is not transitive. You want that if 
you have a shared(C) c, then it is an error to access c.m iff m is not 
shared. This way you can have partially shared classes, where part of 
the class is thread-local, and other parts are shared with other threads.

Is this it?

Timon Gehr <timon.gehr gmx.ch> writes:

On 17.10.2018 14:24, Timon Gehr wrote:
 On 15.10.2018 23:51, Manu wrote:
 If a shared method is incompatible with an unshared method, your class
 is broken.

 
 Then what you want is not implicit unshared->shared conversion. What you 
 want is a different way to type shared member access. You want a setup 
 where shared methods are only allowed to access shared members and 
 unshared methods are only allowed to access unshared members.
 
 I.e., what you want is that shared is not transitive. You want that if 
 you have a shared(C) c, then it is an error to access c.m iff m is not 
 shared. This way you can have partially shared classes, where part of 
 the class is thread-local, and other parts are shared with other threads.
 
 Is this it?

(Also, with this new definition of 'shared', unshared -> shared 
conversion would of course become sound.)

Timon Gehr <timon.gehr gmx.ch> writes:

On 17.10.2018 14:29, Timon Gehr wrote:
 to access c.m iff m is not shared

Unfortunate typo. This should be if, not iff (if and only if).

Timon Gehr <timon.gehr gmx.ch> writes:

On 17.10.2018 14:24, Timon Gehr wrote:
 and unshared methods are only allowed to access unshared members.

This is actually not necessary, let me reformulate:

You want:

- if you have a C c and a shared(C) s, typeof(s.x) == typeof(c.x).
- shared methods are not allowed to access unshared members.
- shared is not transitive, and therefore unshared class references 
implicitly convert to shared class references

Applied to pointers, this would mean that you can implicitly convert 
int* -> shared(int*), but not shared(int*)->int*, int* -> shared(int)* 
or shared(int)* -> int*. shared(int*) and shared(shared(int)*) would be 
different types, such that shared(int*) cannot be dereferenced but 
shared(shared(int)*) can.

Manu <turkeyman gmail.com> writes:

On Wed, Oct 17, 2018 at 6:15 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 17.10.2018 14:24, Timon Gehr wrote:
 and unshared methods are only allowed to access unshared members.

 This is actually not necessary, let me reformulate:

 You want:

 - if you have a C c and a shared(C) s, typeof(s.x) == typeof(c.x).

No. c.x is mutable, s.x is shared (and inaccessible).

 - shared methods are not allowed to access unshared members.

No. Shared methods are not allowed to access ANY members. Shared data
has no access.
I don't believe there's any rules that can make raw access to shared
data safe. We must depend on threadsafe tooling (libraries).

 - shared is not transitive, and therefore unshared class references
 implicitly convert to shared class references

No. Shared is transitive.

 Applied to pointers, this would mean that you can implicitly convert
 int* -> shared(int*), but not shared(int*)->int*, int* -> shared(int)*
 or shared(int)* -> int*.

Correct. This is acceptable because shared(int)* can not be accessed;
can't produce to a race.
shared -> unshared conversion produces an unshared alias, and is
obviously invalid.

 shared(int*) and shared(shared(int)*) would be
 different types, such that shared(int*) cannot be dereferenced but
 shared(shared(int)*) can.

No. No manner of shared(T) is safely accessible under any
circumstance. This is just the reality, and there's no way it can be
allowed.
The escape hatch is that T may have some shared methods which
implement threadsafe interaction with T, and THAT is the only safe way
to interact with shared data.

12345swordy <alexanderheistermann gmail.com> writes:

On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 If you write a lock-free queue for instance, and all the 
 methods are
 `shared` (ie, threadsafe), then under the current rules, you 
 can't
 interact with the object when it's not shared, and that's fairly
 useless.

Unless the compiler can show that it is ok to implicit/explicity 
convert the object to share without any unintended consequences. 
It should reject it.
It seems that the better solution would to implement a 

Operators
https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/statements-expressions-operators/using-conversion-operators

But that itself requires an DIP itself.

-Alex

ag0aep6g <anonymous example.com> writes:

On 10/15/2018 08:46 PM, Manu wrote:
 1. traditional; assert that the object become thread-local by
 acquiring a lock, cast shared away
 2. object may have shared methods; such methods CAN be called on
 shared instances. such methods may internally implement
 synchronisation to perform their function. perhaps methods of a
 lock-free queue structure for instance, or operator overloads on
 `Atomic!int`, etc.

[...]
 Assuming the rules above: "can't read or write to members", and the
 understanding that `shared` methods are expected to have threadsafe
 implementations (because that's the whole point), what are the risks
 from allowing T* -> shared(T)* conversion?

As far as I understand, the rule "can't read or write to members" is for 
the compiler, right? I can still read and write members, but I have to 
cast `shared` away and ensure thread-safety myself?

If that's so, then my example from the last thread might still apply:

----
struct Bob
{
   int* p;
   void doThing() shared
   {
     p = &s; /* Might need a cast or two here, and a lock or an atomic 
store or whatever. */
   }
}
shared int s;
----

When the needed casts etc. are added, is `doThing` allowed? If not, I 
think you have to specify more precisely what a method can and can't do.

If `doThing` is ok, you can't allow T* -> shared(T)*. You'd be allowing 
aliasing an unqualified int* with a shared(int*):

----
void main()
{
   Bob* b = new Bob;
   shared(Bob)* sb = b; /* You'd allow this line. */
   sb.doThing();
   /* Now the unqualified int* b.p points to the shared int s. */
}
----

Jacob Carlborg <doob me.com> writes:

On 2018-10-15 20:46, Manu wrote:

 1. traditional; assert that the object become thread-local by
 acquiring a lock, cast shared away

Instead of having to explicitly cast away shared we could leverage the 
synchronized statement. It could be enhanced to allow the following:

shared int a;
Mutex mutex;

synchronized(tlsA; a, mutex)
{
     // within this scope "tlsA" is the same as "a" but without
     // the "shared" qualifier. "tlsA" is not allowed to escape the block
}

This could also be implemented as a library function, but that would 
require casting away shared inside the implementation:

guard(a, mutex, (tlsA){

});

void guard(T)(scope shared T value, Mutex mutex, scope void delegate 
(scope T) block)
{
     mutex.lock_nothrow();
     scope (exit)
         mutex.unlock_nothrow();

     block(cast(T) value);
}


-- 
/Jacob Carlborg

Kagamin <spam here.lot> writes:

On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Current situation where you can arbitrarily access shared 
 members
 undermines any value it has.

The value of shared is existence of thread-local data that's 
guaranteed to be not shared, so you don't need to worry about 
thread-local data being shared, and shared protects that value 
well.

 Assuming this world... how do you use shared?

Unique solution for each case.

 If you write a lock-free queue for instance, and all the 
 methods are
 `shared` (ie, threadsafe), then under the current rules, you 
 can't
 interact with the object when it's not shared, and that's fairly
 useless.

Create it as shared.

 Assuming the rules above: "can't read or write to members", and 
 the understanding that `shared` methods are expected to have 
 threadsafe implementations (because that's the whole point), 
 what are the risks from allowing T* -> shared(T)* conversion?

All data becomes possibly shared, so you can't assume it's 
unshared, effectively C-style sharing. BTW D supports the latter 
already.

Manu <turkeyman gmail.com> writes:

On Tue, Oct 16, 2018 at 2:25 AM Kagamin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Current situation where you can arbitrarily access shared
 members
 undermines any value it has.

 The value of shared is existence of thread-local data that's
 guaranteed to be not shared, so you don't need to worry about
 thread-local data being shared, and shared protects that value
 well.

This isn't really an argument in favour of shared being able to
arbitrarily access all its members though. I'm not sure how your
points addresses my claim?

this sounds like that old quote "there can not be peace without war".
It's not shared that makes not-shared thread-local by default, it's
just that's how D is defined.
Shared allows you to break that core assumption. Without shared, it
would still be thread-local by default... you just wouldn't have an
escape hatch ;)

 Assuming this world... how do you use shared?

 Unique solution for each case.

 If you write a lock-free queue for instance, and all the
 methods are
 `shared` (ie, threadsafe), then under the current rules, you
 can't
 interact with the object when it's not shared, and that's fairly
 useless.

 Create it as shared.

Then you can't use it locally.

 Assuming the rules above: "can't read or write to members", and
 the understanding that `shared` methods are expected to have
 threadsafe implementations (because that's the whole point),
 what are the risks from allowing T* -> shared(T)* conversion?

 All data becomes possibly shared, so you can't assume it's
 unshared, effectively C-style sharing. BTW D supports the latter
 already.

No data becomes 'possibly shared', because it's all inaccessible.
Only if your object specifies a threadsafe API may some controlled
data become shared... and that's the whole point of writing a
threadsafe object.
If the threadsafe object doesn't support sharing its own data, then
it's not really a threadsafe object.

Timon Gehr <timon.gehr gmx.ch> writes:

On 15.10.2018 20:46, Manu wrote:
 
 Assuming the rules above: "can't read or write to members", and the
 understanding that `shared` methods are expected to have threadsafe
 implementations (because that's the whole point), what are the risks
 from allowing T* -> shared(T)* conversion?
 

Unshared becomes useless, and in turn, shared becomes useless. You can't 
have unshared/shared aliasing.

 All the risks that I think have been identified previously assume that
 you can arbitrarily modify the data. That's insanity... assume we fix
 that... I think the promotion actually becomes safe now...?

But useless, because there is no way to ensure thread safety of reads 
and writes if only one party to the shared state knows about the sharing.

Dominikus Dittes Scherkl <dominikus.scherkl continental-corporation.com> writes:

On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote:
 On 15.10.2018 20:46, Manu wrote:
 
 Assuming the rules above: "can't read or write to members", 
 and the
 understanding that `shared` methods are expected to have 
 threadsafe
 implementations (because that's the whole point), what are the 
 risks
 from allowing T* -> shared(T)* conversion?
 

 Unshared becomes useless, and in turn, shared becomes useless.

why is unshared useless?
Unshared means you can read an write to it.
If you give it to a function that expect something shared,
the function you had given it to can't read or write it, so it
can't do any harm.
Of course it can handle it threadsave, but as it is local,
that is only overhead - reading or changing the value can't do
any harm either. I like the idea.

 But useless, because there is no way to ensure thread safety of 
 reads and writes if only one party to the shared state knows 
 about the sharing.

Of course there is. Giving an unshared value to a function that
even can handle shared values may create some overhead, but is
indeed threadsave.

Timon Gehr <timon.gehr gmx.ch> writes:

On 16.10.2018 13:04, Dominikus Dittes Scherkl wrote:
 On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote:
 On 15.10.2018 20:46, Manu wrote:
 Assuming the rules above: "can't read or write to members", and the
 understanding that `shared` methods are expected to have threadsafe
 implementations (because that's the whole point), what are the risks
 from allowing T* -> shared(T)* conversion?

 Unshared becomes useless, and in turn, shared becomes useless.

 why is unshared useless?
 Unshared means you can read an write to it.
 If you give it to a function that expect something shared,
 the function you had given it to can't read or write it, so it
 can't do any harm.

It can do harm to others who hold an unshared alias to the same data and 
are operating on it concurrently.

 Of course it can handle it threadsave, but as it is local,
 that is only overhead - reading or changing the value can't do
 any harm either. I like the idea.
 
 But useless, because there is no way to ensure thread safety of reads 
 and writes if only one party to the shared state knows about the sharing.

 Of course there is.

Please do enlighten me. You have two processors operating 
(reading/writing) on the same address space on a modern computer 
architecture with a weak memory model, and you are using an optimizing 
compiler. How do you ensure sensible results without cooperation from 
both of them? (Hint: you don't.)

 Giving an unshared value to a function that
 even can handle shared values may create some overhead, but is
 indeed threadsave.
 

Yes, if you give it to one function only, that is the case. However, as 
you may know, concurrency means that there may be multiple functions 
operating on the data _at the same time_. If one of them operates on the 
data as if it was not shared, you will run into trouble.

You are arguing as if there was either no concurrency or no mutable 
aliasing.

Manu <turkeyman gmail.com> writes:

On Tue, Oct 16, 2018 at 6:25 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 16.10.2018 13:04, Dominikus Dittes Scherkl wrote:
 On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote:
 On 15.10.2018 20:46, Manu wrote:
 Assuming the rules above: "can't read or write to members", and the
 understanding that `shared` methods are expected to have threadsafe
 implementations (because that's the whole point), what are the risks
 from allowing T* -> shared(T)* conversion?

 Unshared becomes useless, and in turn, shared becomes useless.

 why is unshared useless?
 Unshared means you can read an write to it.
 If you give it to a function that expect something shared,
 the function you had given it to can't read or write it, so it
 can't do any harm.

 It can do harm to others who hold an unshared alias to the same data and
 are operating on it concurrently.

Nobody else holds an unshared alias.
If you pass a value as const, you don't fear that it will become mutable.

 Of course it can handle it threadsave, but as it is local,
 that is only overhead - reading or changing the value can't do
 any harm either. I like the idea.

 But useless, because there is no way to ensure thread safety of reads
 and writes if only one party to the shared state knows about the sharing.

 Of course there is.

 Please do enlighten me. You have two processors operating
 (reading/writing) on the same address space on a modern computer
 architecture with a weak memory model, and you are using an optimizing
 compiler. How do you ensure sensible results without cooperation from
 both of them? (Hint: you don't.)

What? This is a weird statement.
So, you're saying that nobody has successfully written any threadsafe
code, ever... we should stop trying, and we should admit that
threadsafe queues and atomics, and mutexes and stuff all don't exist?

 without cooperation from both of them?

Perhaps this is the key to your statement?
Yes. 'cooperation from both of them' in this case means, they are both
interacting with a threadsafe api, and they are blocked from accessing
members, or any non-threadsafe api.

 Giving an unshared value to a function that
 even can handle shared values may create some overhead, but is
 indeed threadsave.

 Yes, if you give it to one function only, that is the case. However, as
 you may know, concurrency means that there may be multiple functions
 operating on the data _at the same time_. If one of them operates on the
 data as if it was not shared, you will run into trouble.

Who's doing this, and how?

 You are arguing as if there was either no concurrency or no mutable
 aliasing.

If a class has no shared methods, there's no possibility for mutable aliasing.
If the class has shared methods, then the class was carefully designed
to be threadsafe.

Timon Gehr <timon.gehr gmx.ch> writes:

On 16.10.2018 20:07, Manu wrote:
 On Tue, Oct 16, 2018 at 6:25 AM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 16.10.2018 13:04, Dominikus Dittes Scherkl wrote:
 On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote:
 On 15.10.2018 20:46, Manu wrote:
 Assuming the rules above: "can't read or write to members", and the
 understanding that `shared` methods are expected to have threadsafe
 implementations (because that's the whole point), what are the risks
 from allowing T* -> shared(T)* conversion?

 Unshared becomes useless, and in turn, shared becomes useless.

 why is unshared useless?
 Unshared means you can read an write to it.
 If you give it to a function that expect something shared,
 the function you had given it to can't read or write it, so it
 can't do any harm.

 It can do harm to others who hold an unshared alias to the same data and
 are operating on it concurrently.

 
 Nobody else holds an unshared alias.

How so? If you allow implicit conversions from unshared to shared, then 
you immediately get this situation.

 If you pass a value as const, you don't fear that it will become mutable.
 ...

No, but as I already explained last time, mutable -> const is not at all 
like unshared -> shared.

const only takes away capabilities, shared adds new capabilities, such 
as sending a reference to another thread. If you have two threads that 
share data, you need cooperation from both to properly synchronize accesses.

 Of course it can handle it threadsave, but as it is local,
 that is only overhead - reading or changing the value can't do
 any harm either. I like the idea.

 But useless, because there is no way to ensure thread safety of reads
 and writes if only one party to the shared state knows about the sharing.

 Of course there is.

 Please do enlighten me. You have two processors operating
 (reading/writing) on the same address space on a modern computer
 architecture with a weak memory model, and you are using an optimizing
 compiler. How do you ensure sensible results without cooperation from
 both of them? (Hint: you don't.)

 
 What? This is a weird statement.
 So, you're saying that nobody has successfully written any threadsafe
 code, ever... we should stop trying, and we should admit that
 threadsafe queues and atomics, and mutexes and stuff all don't exist?
 

Obviously I am not saying that.

 without cooperation from both of them?

 
 Perhaps this is the key to your statement?

Yes.

 Yes. 'cooperation from both of them' in this case means, they are both
 interacting with a threadsafe api, and they are blocked from accessing
 members, or any non-threadsafe api.
 ...

Yes. Your proposal only enforces this for the shared alias.

 Giving an unshared value to a function that
 even can handle shared values may create some overhead, but is
 indeed threadsave.

 Yes, if you give it to one function only, that is the case. However, as
 you may know, concurrency means that there may be multiple functions
 operating on the data _at the same time_. If one of them operates on the
 data as if it was not shared, you will run into trouble.

 
 Who's doing this,

Anyone, it really does not matter. One major point of the type system is 
to ensure that _all_  safe code has defined behavior. You can convert 
between shared and unshared, just not in  safe code.

 and how?
 ...

They create a mutable instance of a class, they create a shared alias 
using one of your proposed holes, then send the shared alias to another 
thread, call some methods on it in both threads and get race conditions.

 You are arguing as if there was either no concurrency or no mutable
 aliasing.

 
 If a class has no shared methods, there's no possibility for mutable aliasing.
 If the class has shared methods, then the class was carefully designed
 to be threadsafe.
 

Not necessarily. Counterexample:

 safe:
class C{
     int x;
     void foo(){
         x+=1; // this can still race with atomicIncrement
     }
     void bar()shared{
         atomicIncrement(x); // presumably you want to allow this
     }
}

void main(){
     auto c=new C();
     shared s=c; // depending on your exact proposed rules, this step 
may be more cumbersome
     spawn!(()=>s.bar());
     s.foo(); // race
}

Now, if a class has only shared members, that is another story. In this 
case, all references should implicitly convert to shared. There's a DIP 
I meant to write about this. (For all qualifiers, not just shared).

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/17/18 8:02 AM, Timon Gehr wrote:
 Now, if a class has only shared members, that is another story. In this 
 case, all references should implicitly convert to shared. There's a DIP 
 I meant to write about this. (For all qualifiers, not just shared).

When you say "shared members", you mean all the data is shared too or 
just the methods are shared?

If not the data, D has a problem with encapsulation. Not only all the 
methods on the class must be shared, but ALL code in the entire module 
must be marked as using a shared class instance. Otherwise, other 
functions could modify the private data without using the proper synch 
mechanisms.

We are better off requiring the cast, or enforcing that one must use a 
shared object to begin with.

I think any sometimes-shared object is in any case going to benefit from 
parallel implementations for when the thing is unshared.

-Steve

Timon Gehr <timon.gehr gmx.ch> writes:

On 17.10.2018 15:40, Steven Schveighoffer wrote:
 On 10/17/18 8:02 AM, Timon Gehr wrote:
 Now, if a class has only shared members, that is another story. In 
 this case, all references should implicitly convert to shared. There's 
 a DIP I meant to write about this. (For all qualifiers, not just shared).

 
 When you say "shared members", you mean all the data is shared too or 
 just the methods are shared?
 
 If not the data, D has a problem with encapsulation. Not only all the 
 methods on the class must be shared, but ALL code in the entire module 
 must be marked as using a shared class instance. Otherwise, other 
 functions could modify the private data without using the proper synch 
 mechanisms.
 
 We are better off requiring the cast, or enforcing that one must use a 
 shared object to begin with.
 
 I think any sometimes-shared object is in any case going to benefit from 
 parallel implementations for when the thing is unshared.
 
 -Steve

The specific proposal was that, for example, if a class is defined like 
this:

shared class C{
     // ...
}

then shared(C) and C are implicitly convertible to each other. The 
change is not fully backwards-compatible, because right now, this 
annotation just makes all members (data and methods) shared, but child 
classes may introduce unshared members.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/17/18 10:18 AM, Timon Gehr wrote:
 On 17.10.2018 15:40, Steven Schveighoffer wrote:
 On 10/17/18 8:02 AM, Timon Gehr wrote:
 Now, if a class has only shared members, that is another story. In 
 this case, all references should implicitly convert to shared. 
 There's a DIP I meant to write about this. (For all qualifiers, not 
 just shared).

 When you say "shared members", you mean all the data is shared too or 
 just the methods are shared?

 If not the data, D has a problem with encapsulation. Not only all the 
 methods on the class must be shared, but ALL code in the entire module 
 must be marked as using a shared class instance. Otherwise, other 
 functions could modify the private data without using the proper synch 
 mechanisms.

 We are better off requiring the cast, or enforcing that one must use a 
 shared object to begin with.

 I think any sometimes-shared object is in any case going to benefit 
 from parallel implementations for when the thing is unshared.

 -Steve

 
 The specific proposal was that, for example, if a class is defined like 
 this:
 
 shared class C{
      // ...
 }
 
 then shared(C) and C are implicitly convertible to each other. The 
 change is not fully backwards-compatible, because right now, this 
 annotation just makes all members (data and methods) shared, but child 
 classes may introduce unshared members.

OK, so the proposal is that all data and function members are shared. 
That makes sense.

In one sense, because the class reference is conflated with the type 
modifier, having a C that isn't shared, actually have it's class data be 
shared, would be useful.

-Steve

Manu <turkeyman gmail.com> writes:

On Wed, Oct 17, 2018 at 5:05 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 [... all text ...]

OMFG, I just spent about 3 hours writing a super-detailed reply to all
of Timon's posts in aggregate... I clicked send... and it's gone.
I don't know if this is a gmail thing, a mailing list thing... no
idea... but it's... gone.
I can't repeat that effort :(

Paolo Invernizzi <paolo.invernizzi gmail.com> writes:

On Thursday, 18 October 2018 at 06:20:02 UTC, Manu wrote:
 On Wed, Oct 17, 2018 at 5:05 AM Timon Gehr via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 [... all text ...]

 OMFG, I just spent about 3 hours writing a super-detailed reply 
 to all
 of Timon's posts in aggregate... I clicked send... and it's 
 gone.
 I don't know if this is a gmail thing, a mailing list thing... 
 no
 idea... but it's... gone.
 I can't repeat that effort :(

Never never write something super-detailed in a web-based "thing"!
Native application, and copy-past!

:-O

But' now I'm curious about your reply! Timon argumentation are 
really strong (IMHO), so it's a double effort! :-/

/Paolo

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 2:20 AM, Manu wrote:
 On Wed, Oct 17, 2018 at 5:05 AM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 [... all text ...]

 
 OMFG, I just spent about 3 hours writing a super-detailed reply to all
 of Timon's posts in aggregate... I clicked send... and it's gone.
 I don't know if this is a gmail thing, a mailing list thing... no
 idea... but it's... gone.
 I can't repeat that effort :(
 

If it's gmail, it should be in sent folder, no?

I've never had a gmail message that got sent fail to go into the sent box.

-Steve

Manu <turkeyman gmail.com> writes:

On Tue, Oct 16, 2018 at 3:20 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 15.10.2018 20:46, Manu wrote:
 Assuming the rules above: "can't read or write to members", and the
 understanding that `shared` methods are expected to have threadsafe
 implementations (because that's the whole point), what are the risks
 from allowing T* -> shared(T)* conversion?

 Unshared becomes useless, and in turn, shared becomes useless. You can't
 have unshared/shared aliasing.

What aliasing? Please show a reasonable and likely construction of the
problem. I've been trying to think of it.

 All the risks that I think have been identified previously assume that
 you can arbitrarily modify the data. That's insanity... assume we fix
 that... I think the promotion actually becomes safe now...?

 But useless, because there is no way to ensure thread safety of reads
 and writes if only one party to the shared state knows about the sharing.

What? I don't understand this sentence.

If a shared method is not threadsafe, then it's an implementation
error. A user should expect that a shared method is threadsafe,
otherwise it shouldn't be a shared method! Thread-local (ie, normal)
methods are for not-threadsafe functionality.

Timon Gehr <timon.gehr gmx.ch> writes:

On 16.10.2018 19:25, Manu wrote:
 On Tue, Oct 16, 2018 at 3:20 AM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 15.10.2018 20:46, Manu wrote:
 Assuming the rules above: "can't read or write to members", and the
 understanding that `shared` methods are expected to have threadsafe
 implementations (because that's the whole point), what are the risks
 from allowing T* -> shared(T)* conversion?

 Unshared becomes useless, and in turn, shared becomes useless. You can't
 have unshared/shared aliasing.

 
 What aliasing?

Aliasing means you have two references to the same data. The two 
references are then said to alias. An implicit conversion from unshared 
to shared by definition introduces aliasing, where one of the two 
references is unshared and the other is shared.

 Please show a reasonable and likely construction of the
 problem. I've been trying to think of it.
 ...

I have given you an example. I don't care whether it is "reasonable" or 
"likely". The point of  safe is to have a subset of the language with a 
sound type system.

 All the risks that I think have been identified previously assume that
 you can arbitrarily modify the data. That's insanity... assume we fix
 that... I think the promotion actually becomes safe now...?

 But useless, because there is no way to ensure thread safety of reads
 and writes if only one party to the shared state knows about the sharing.

 
 What? I don't understand this sentence.
 
 If a shared method is not threadsafe, then it's an implementation
 error. A user should expect that a shared method is threadsafe,
 otherwise it shouldn't be a shared method! Thread-local (ie, normal)
 methods are for not-threadsafe functionality.
 

Your function can be thread safe all you want. If you have a thread 
unsafe function also operating on the same state, you will still get 
race conditions. E.g. mutual exclusion is based on cooperation from all 
threads. If one of them forgets to lock, it does not matter how 
threadsafe all the others are.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/15/18 2:46 PM, Manu wrote:
 Okay, so I've been thinking on this for a while... I think I have a
 pretty good feel for how shared is meant to be.
 
 1. shared should behave exactly like const, except in addition to
 inhibiting write access, it also inhibits read access.
 
 I think this is the foundation for a useful definition for shared, and
 it's REALLY easy to understand and explain.
 
 Current situation where you can arbitrarily access shared members
 undermines any value it has. Shared must assure you don't access
 members unsafely, and the only way to do that with respect to data
 members, is to inhibit access completely.
 I think shared is just const without read access.
 
 Assuming this world... how do you use shared?
 
 1. traditional; assert that the object become thread-local by
 acquiring a lock, cast shared away
 2. object may have shared methods; such methods CAN be called on
 shared instances. such methods may internally implement
 synchronisation to perform their function. perhaps methods of a
 lock-free queue structure for instance, or operator overloads on
 `Atomic!int`, etc.
 
 In practise, there is no functional change in usage from the current
 implementation, except we disallow unsafe accesses (which will make
 the thing useful).
 
From there, it opens up another critical opportunity; T* -> shared(T)*

 promotion.
 Const would be useless without T* -> const(T)* promotion. Shared
 suffers a similar problem.
 If you write a lock-free queue for instance, and all the methods are
 `shared` (ie, threadsafe), then under the current rules, you can't
 interact with the object when it's not shared, and that's fairly
 useless.
 
 Assuming the rules above: "can't read or write to members", and the
 understanding that `shared` methods are expected to have threadsafe
 implementations (because that's the whole point), what are the risks
 from allowing T* -> shared(T)* conversion?
 
 All the risks that I think have been identified previously assume that
 you can arbitrarily modify the data. That's insanity... assume we fix
 that... I think the promotion actually becomes safe now...?
 
 Destroy...
 

This is a step in the right direction. But there is still one problem -- 
shared is inherently transitive.

So casting away shared is super-dangerous, even if you lock the shared 
data, because any of the subreferences will become unshared and 
read/writable.

For instance:

struct S
{
    int x;
    int *y;
}

shared int z;

auto s1 = shared(S)(1, &z);

auto s2 = shared(S)(2, &z);

S* s1locked = s1.lock;

Now I have access to z via s1locked as an unshared int, and I never 
locked z. Potentially one could do the same thing via s2, and now there 
are 2 mutable references, potentially in 2 threads.

All of this, of course, is manual. So technically we could manually 
implement it properly inside S. But this means shared doesn't help us much.

We really need on top of shared, a way to specify something is 
tail-shared. That is, all the data in S is unshared, but anything it 
points to is still shared. That at least helps the person implementing 
the manual locking from doing stupid things himself.

-Steve

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
 On 10/15/18 2:46 PM, Manu wrote:

 From there, it opens up another critical opportunity; T* -> shared(T)*

 promotion.
 Const would be useless without T* -> const(T)* promotion. Shared
 suffers a similar problem.
 If you write a lock-free queue for instance, and all the methods are
 `shared` (ie, threadsafe), then under the current rules, you can't
 interact with the object when it's not shared, and that's fairly
 useless.


Oh, I didn't see this part. Completely agree with Timon on this, no 
implicit conversions should be allowed.

If you want to have a lock-free implementation of something, you can 
abstract the assignments and reads behind the proper mechanisms anyway, 
and still avoid locking (casting is not locking).

-Steve

Manu <turkeyman gmail.com> writes:

On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
 On 10/15/18 2:46 PM, Manu wrote:

 From there, it opens up another critical opportunity; T* -> shared(T)*

 promotion.
 Const would be useless without T* -> const(T)* promotion. Shared
 suffers a similar problem.
 If you write a lock-free queue for instance, and all the methods are
 `shared` (ie, threadsafe), then under the current rules, you can't
 interact with the object when it's not shared, and that's fairly
 useless.


 Oh, I didn't see this part. Completely agree with Timon on this, no
 implicit conversions should be allowed.

Why?

 If you want to have a lock-free implementation of something, you can
 abstract the assignments and reads behind the proper mechanisms anyway,
 and still avoid locking (casting is not locking).

Sorry, I don't understand what you're saying. Can you clarify?

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/16/18 2:10 PM, Manu wrote:
 On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
 On 10/15/18 2:46 PM, Manu wrote:

  From there, it opens up another critical opportunity; T* -> shared(T)*

 promotion.
 Const would be useless without T* -> const(T)* promotion. Shared
 suffers a similar problem.
 If you write a lock-free queue for instance, and all the methods are
 `shared` (ie, threadsafe), then under the current rules, you can't
 interact with the object when it's not shared, and that's fairly
 useless.


 Oh, I didn't see this part. Completely agree with Timon on this, no
 implicit conversions should be allowed.

 
 Why?

int x;

shared int *p = &x; // allow implicit conversion, currently error

passToOtherThread(p);

useHeavily(&x);

How is this safe? Thread1 is using x without locking, while the other 
thread has to lock. In order for synchronization to work, both sides 
have to agree on a synchronization technique and abide by it.

 If you want to have a lock-free implementation of something, you can
 abstract the assignments and reads behind the proper mechanisms anyway,
 and still avoid locking (casting is not locking).

 
 Sorry, I don't understand what you're saying. Can you clarify?
 

I'd still mark a lock-free implementation shared, and all its methods 
shared. shared does not mean you have to lock, just cast away shared. A 
lock-free container still has to do some special things to make sure it 
avoids races, and having an "unusable" state aids in enforcing this.

-Steve

Manu <turkeyman gmail.com> writes:

On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via
Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/16/18 2:10 PM, Manu wrote:
 On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
 On 10/15/18 2:46 PM, Manu wrote:

  From there, it opens up another critical opportunity; T* -> shared(T)*

 promotion.
 Const would be useless without T* -> const(T)* promotion. Shared
 suffers a similar problem.
 If you write a lock-free queue for instance, and all the methods are
 `shared` (ie, threadsafe), then under the current rules, you can't
 interact with the object when it's not shared, and that's fairly
 useless.


 Oh, I didn't see this part. Completely agree with Timon on this, no
 implicit conversions should be allowed.

 Why?

 int x;

 shared int *p = &x; // allow implicit conversion, currently error

 passToOtherThread(p);

 useHeavily(&x);

What does this mean? It can't do anything... that's the whole point here.
I think I'm struggling here with people bringing presumptions to the
thread. You need to assume the rules I define in the OP for the
experiment to work.

 How is this safe?

Because useHeavily() can't read or write to x.

 Thread1 is using x without locking, while the other
 thread has to lock. In order for synchronization to work, both sides
 have to agree on a synchronization technique and abide by it.

Only the owning thread can access x, the shared instance can't access x at all.

If some code somewhere decides it wants to cast-away shared, then it
needs to determine ownership via some other means. It needs to be
confident that the original owner yielded ownership, and any API that
leads to this behaviour would need to be designed in such a way to
encourage correct behaviour.
That's *exactly* how it is now... I haven't changed anything from this
perspective.

 If you want to have a lock-free implementation of something, you can
 abstract the assignments and reads behind the proper mechanisms anyway,
 and still avoid locking (casting is not locking).

 Sorry, I don't understand what you're saying. Can you clarify?

 I'd still mark a lock-free implementation shared, and all its methods
 shared. shared does not mean you have to lock, just cast away shared.

Shared *should* mean that the function is threadsafe, and you are safe
to call it from a shared instance.
If a function is not shared, then you MUST cast away shared, and that
implies that you need to use external means to create a context where
you have thread-local ownership of the instance (usually with a
mutex).

You shouldn't need to cast-away shared to make a safe function call.
By casting away shared, you also gain access to all the non-threadsafe
methods and members.
Casting shared away to call a shared method makes a safe access into a
potentially unsafe access.

It's unacceptable to case-away shared. It should be as unacceptable as
casting const away. The only situation where it's okay is where you're
externally verifying thread-locality by external means, and that's
subject to your broader systemic design.

 A lock-free container still has to do some special things to make sure it
 avoids races, and having an "unusable" state aids in enforcing this.

Can you explain how my proposal doesn't model this very neatly?

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/16/18 4:26 PM, Manu wrote:
 On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/16/18 2:10 PM, Manu wrote:
 On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
 On 10/15/18 2:46 PM, Manu wrote:

   From there, it opens up another critical opportunity; T* -> shared(T)*

 promotion.
 Const would be useless without T* -> const(T)* promotion. Shared
 suffers a similar problem.
 If you write a lock-free queue for instance, and all the methods are
 `shared` (ie, threadsafe), then under the current rules, you can't
 interact with the object when it's not shared, and that's fairly
 useless.


 Oh, I didn't see this part. Completely agree with Timon on this, no
 implicit conversions should be allowed.

 Why?

 int x;

 shared int *p = &x; // allow implicit conversion, currently error

 passToOtherThread(p);

 useHeavily(&x);

 
 What does this mean? It can't do anything... that's the whole point here.
 I think I'm struggling here with people bringing presumptions to the
 thread. You need to assume the rules I define in the OP for the
 experiment to work.

OK, I wrote a whole big response to this, and I went and re-quoted the 
above, and now I think I understand what the point of your statement is.

I'll first say that if you don't want to allow implicit casting of 
shared to mutable, then you can't allow implicit casting from mutable to 
shared. Because it's mutable, races can happen.

There is in fact, no difference between:

int *p;
shared int *p2 = p;
int *p3 = cast(int*)p2;

and this:

int *p;
shared int *p2 = p;
int *p3 = p;

So really, the effort to prevent the reverse cast is defeated by 
allowing the implicit cast.

There is a reason we disallow assigning from mutable to immutable 
without a cast. Yet, it is done in many cases, because you are sometimes 
building an immutable object with mutable pieces, and want to cast the 
final result.

In this case, it's ON YOU to make sure it's correct, and the traditional 
mechanism for the compiler giving you the responsibility is to require a 
cast.

-----

OK, so here is where I think I misunderstood your point. When you said a 
lock-free queue would be unusable if it wasn't shared, I thought you 
meant it would be unusable if we didn't allow the implicit cast. But I 
realize now, you meant you should be able to use a lock-free queue 
without it being actually shared anywhere.

What I say to this is that it doesn't need to be usable. I don't care to 
use a lock-free queue in a thread-local capacity. I'll just use a normal 
queue, which is easy to implement, and doesn't have to worry about race 
conditions or using atomics. A lock free queue is a special thing, very 
difficult to get right, and only really necessary if you are going to 
share it. And used for performance reasons!

Why would I want to incur performance penalties when using a lock-free 
queue in an unshared mode? I would actually expect 2 separate 
implementations of the primitives, one for shared one for unshared.

What about primitives that would be implemented the same? In that case, 
the shared method becomes:

auto method() { return (cast(Queue*)&this).method; }

Is this "unusable"? Without a way to say, you can call this on shared or 
unshared instances, then we need to do it this way.

But I would trust the queue to handle this properly depending on whether 
it was typed shared or not.

-Steve

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer 
wrote:
 There is in fact, no difference between:

 int *p;
 shared int *p2 = p;
 int *p3 = cast(int*)p2;

 and this:

 int *p;
 shared int *p2 = p;
 int *p3 = p;

If I understand Manu correctly the first should compile, and the 
second should error, just like if you replaces shared with const 
in the above.

Manu <turkeyman gmail.com> writes:

On Tue, Oct 16, 2018 at 3:25 PM Nicholas Wilson via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer
 wrote:
 There is in fact, no difference between:

 int *p;
 shared int *p2 = p;
 int *p3 = cast(int*)p2;

 and this:

 int *p;
 shared int *p2 = p;
 int *p3 = p;

 If I understand Manu correctly the first should compile, and the
 second should error, just like if you replaces shared with const
 in the above.

Why is the second an error?

Right, for the sake of the thought experiment, replace shared with const.
The first is an abomination (casting away const)... the second is just a no-op.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Wednesday, 17 October 2018 at 00:29:04 UTC, Manu wrote:
 On Tue, Oct 16, 2018 at 3:25 PM Nicholas Wilson via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven 
 Schveighoffer wrote:
 There is in fact, no difference between:

 int *p;
 shared int *p2 = p;
 int *p3 = cast(int*)p2;

 and this:

 int *p;
 shared int *p2 = p;
 int *p3 = p;

 If I understand Manu correctly the first should compile, and 
 the second should error, just like if you replaces shared with 
 const in the above.

 Why is the second an error?

 Right, for the sake of the thought experiment, replace shared 
 with const.
 The first is an abomination (casting away const)... the second 
 is just a no-op.

I missed that the third example was *p3 = p; not *p3 = p2;

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer 
wrote:
 OK, so here is where I think I misunderstood your point. When 
 you said a lock-free queue would be unusable if it wasn't 
 shared, I thought you meant it would be unusable if we didn't 
 allow the implicit cast. But I realize now, you meant you 
 should be able to use a lock-free queue without it being 
 actually shared anywhere.

 What I say to this is that it doesn't need to be usable. I 
 don't care to use a lock-free queue in a thread-local capacity. 
 I'll just use a normal queue, which is easy to implement, and 
 doesn't have to worry about race conditions or using atomics. A 
 lock free queue is a special thing, very difficult to get 
 right, and only really necessary if you are going to share it. 
 And used for performance reasons!

I think this comes up where the queue was originally shared, you 
acquired a lock on the thing it is a member of, and you want to 
continue using it through your exclusive reference.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/16/18 6:24 PM, Nicholas Wilson wrote:
 On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote:
 OK, so here is where I think I misunderstood your point. When you said 
 a lock-free queue would be unusable if it wasn't shared, I thought you 
 meant it would be unusable if we didn't allow the implicit cast. But I 
 realize now, you meant you should be able to use a lock-free queue 
 without it being actually shared anywhere.

 What I say to this is that it doesn't need to be usable. I don't care 
 to use a lock-free queue in a thread-local capacity. I'll just use a 
 normal queue, which is easy to implement, and doesn't have to worry 
 about race conditions or using atomics. A lock free queue is a special 
 thing, very difficult to get right, and only really necessary if you 
 are going to share it. And used for performance reasons!

 
 I think this comes up where the queue was originally shared, you 
 acquired a lock on the thing it is a member of, and you want to continue 
 using it through your exclusive reference.
 

Isn't that a locking queue? I thought we were talking lock-free?

-Steve

Manu <turkeyman gmail.com> writes:

On Tue, Oct 16, 2018 at 2:20 PM Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/16/18 4:26 PM, Manu wrote:
 On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/16/18 2:10 PM, Manu wrote:
 On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
 On 10/15/18 2:46 PM, Manu wrote:

   From there, it opens up another critical opportunity; T* -> shared(T)*

 promotion.
 Const would be useless without T* -> const(T)* promotion. Shared
 suffers a similar problem.
 If you write a lock-free queue for instance, and all the methods are
 `shared` (ie, threadsafe), then under the current rules, you can't
 interact with the object when it's not shared, and that's fairly
 useless.


 Oh, I didn't see this part. Completely agree with Timon on this, no
 implicit conversions should be allowed.

 Why?

 int x;

 shared int *p = &x; // allow implicit conversion, currently error

 passToOtherThread(p);

 useHeavily(&x);

 What does this mean? It can't do anything... that's the whole point here.
 I think I'm struggling here with people bringing presumptions to the
 thread. You need to assume the rules I define in the OP for the
 experiment to work.

 OK, I wrote a whole big response to this, and I went and re-quoted the
 above, and now I think I understand what the point of your statement is.

 I'll first say that if you don't want to allow implicit casting of
 shared to mutable,

It's critical that this is not allowed. It's totally unreasonable to
cast from shared to thread-local without synchronisation.
It's as bad as casting away const.

 then you can't allow implicit casting from mutable to
 shared. Because it's mutable, races can happen.

I don't follow...

 There is in fact, no difference between:

 int *p;
 shared int *p2 = p;
 int *p3 = cast(int*)p2;

Totally illegal!! You casted away shared. That's as bad as casting away const.

 and this:

 int *p;
 shared int *p2 = p;
 int *p3 = p;

There's nothing wrong with this... I don't understand the point?

 So really, the effort to prevent the reverse cast is defeated by
 allowing the implicit cast.

Only the caller has the thread-local instance. You can take a
thread-local pointer to a thread-local within the context of a single
thread.
So, it's perfectly valid for `p` and `p3` to exist in a single scope.
`p2` is fine here too... and if that shared pointer were to escape to
another thread, it wouldn't be a threat, because it's not readable or
writable, and you can't make it back into a thread-local pointer
without carefully/deliberately deployed machinery.

 There is a reason we disallow assigning from mutable to immutable
 without a cast. Yet, it is done in many cases, because you are sometimes
 building an immutable object with mutable pieces, and want to cast the
 final result.

I don't think analogy to immutable has a place in this discussion, or
at least, I don't understand the relevance...
I think the reasonable analogy is const.

 In this case, it's ON YOU to make sure it's correct, and the traditional
 mechanism for the compiler giving you the responsibility is to require a
 cast.

I think what you're talking about are behaviours relating to casting
shared *away*, and that's some next-level shit. Handling in that case
is no different to the way it exists today. You must guarantee that
the pointer you possess becomes thread-local before casting it to a
thread-local pointer.
In my application framework, I will never cast shared away under my
proposed design. We don't have any such global locks.

 -----

 OK, so here is where I think I misunderstood your point. When you said a
 lock-free queue would be unusable if it wasn't shared, I thought you
 meant it would be unusable if we didn't allow the implicit cast. But I
 realize now, you meant you should be able to use a lock-free queue
 without it being actually shared anywhere.

Right, a lock-free queue is a threadsafe object, and it's methods work
whether the queue is shared or not.
The methods are attributed shared because they can be called on shared
instances... but they can ALSO be called from a thread-local instance,
and under my suggested promotion rules, it's fine for the this-pointer
to promote to shared to make the call.

 What I say to this is that it doesn't need to be usable. I don't care to
 use a lock-free queue in a thread-local capacity. I'll just use a normal
 queue, which is easy to implement, and doesn't have to worry about race
 conditions or using atomics. A lock free queue is a special thing, very
 difficult to get right, and only really necessary if you are going to
 share it. And used for performance reasons!

I'm more interested in the object that has that lock-free queue as a
member... it is probably a mostly thread-local object, but may have a
couple of shared methods.
I have a whole lot of objects which have 3 tiers of API access; the
thread-local part, the threadsafe part, and the const part. Just as a
mutable instance can call a const method, there's no reason a
thread-local instance can't call a threadsafe method.

You can ask 'why', and I can't give any more satisfactory answer than
"you can call a const method from a mutable object", this is
commonsense, and should be possible.
We want to call shared methods from threadlocal objects all the time.
It's possible, and it's safe... there's no reason we shouldn't be able
to.

 Why would I want to incur performance penalties when using a lock-free
 queue in an unshared mode? I would actually expect 2 separate
 implementations of the primitives, one for shared one for unshared.

Overloading for shared and unshared is possible, and may be desirable
in many cases.
There are also many cases where the code duplication and tech-debt
does not carry its weight. It should not be required, because it's not
technically required.

 What about primitives that would be implemented the same? In that case,
 the shared method becomes:

 auto method() { return (cast(Queue*)&this).method; }

 Is this "unusable"? Without a way to say, you can call this on shared or
 unshared instances, then we need to do it this way.

I don't understand here...?
I'm saying, we *don't* need a way to say "you can call on shared or
unshared instances*, because that's always safe to do. If it wasn't
safe to call a const method with a mutable instance, something is
terrible wrong; same applies here, it is always safe to call a
threadsafe method with a thread-local instance. If that's not true,
the method is objectively not threadsafe.

 But I would trust the queue to handle this properly depending on whether
 it was typed shared or not.

The queue is less interesting than the object that aggregates the queue.

Isaac S. <spam-no-reply-isaac outlook.com> writes:

On Tuesday, 16 October 2018 at 06:21:22 UTC, Manu wrote:
 On Mon, Oct 15, 2018 at 8:55 PM Isaac S. via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote:
 I understand your point but I think the current shared (no 
 implicit conversion) has its uses.
 *snip*

 If you can give a single 'use', I'm all ears ;)

 My usages are a custom ref counted template and list types 
 (which are built on top of the former). The ref counted 
 template will initialize naively when not shared but utilize 
 compare-and-set and thread yielding if needed when shared 
 (ensureInitialized can occur any time after declaration). The 
 list types (which are not shared-compatible *yet*) will 
 utilize a read-write mutex only when shared (no cost beyond a 
 little unused memory to non-shared objects). As such, casting 
 any of the non-shared versions of these types to shared would 
 be unsafe.

 (Technically the types can be safely casted to shared so long 
 as no non-shared reference to them exists and vice versa.)

 Can you link to code? It doesn't sound incompatible with my 
 proposal. Mutexes are blunt instruments, and easy to model. My 
 whole suggestion has virtually no effect on the mutex protected 
 case, which is what most people seem to talk about.

On Wednesday, 17 October 2018 at 00:26:58 UTC, Manu wrote:
 *snip*

 Overloading for shared and unshared is possible, and may be 
 desirable in many cases.
 There are also many cases where the code duplication and 
 tech-debt
 does not carry its weight. It should not be required, because 
 it's not technically required.

Overloading for shared and unshared is my reason for not allowing 
implicit conversion on my types (I have no problems with implicit 
conversion being optional or disableable). The unshared function 
performs no synchronization of any kind while the shared function 
always does. This means that having an unshared and shared 
reference to the same object is unsafe.

As for an actual link to code, I can really only link to my ref 
counted template
as I haven't gotten around to making the containers 
shared-compatible yet. The main point of interest is at lines 
103-131:

https://github.com/isaacs-dev/Familiar/blob/66f1a94fc099601465e755d40a2c68bf4200cabd/containers/familiar/containers/safe_ref_counted.d#L103

The unshared ensureInitialized() doesn't worry about threads 
simultaneously calling it. The shared version of it uses 
compare-and-set to prevent two threads from initializing it at 
the same time. An unsafe example would require a shared 
reference/pointer to an unshared SafeRefCounted (which would be 
fairly weird to do) and both calling ensureInitialized. While an 
example using the ref counted template is fairly contrived, a 
list-type that only uses a read-write mutex when it's shared 
isn't.


As to the larger part of preventing reading/writing of shared 
objects, I agree with its purpose and have no problems with it.

Manu <turkeyman gmail.com> writes:

On Tue, Oct 16, 2018 at 8:20 PM Isaac S. via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 16 October 2018 at 06:21:22 UTC, Manu wrote:
 On Mon, Oct 15, 2018 at 8:55 PM Isaac S. via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote:
 I understand your point but I think the current shared (no
 implicit conversion) has its uses.
 *snip*

 If you can give a single 'use', I'm all ears ;)

 My usages are a custom ref counted template and list types
 (which are built on top of the former). The ref counted
 template will initialize naively when not shared but utilize
 compare-and-set and thread yielding if needed when shared
 (ensureInitialized can occur any time after declaration). The
 list types (which are not shared-compatible *yet*) will
 utilize a read-write mutex only when shared (no cost beyond a
 little unused memory to non-shared objects). As such, casting
 any of the non-shared versions of these types to shared would
 be unsafe.

 (Technically the types can be safely casted to shared so long
 as no non-shared reference to them exists and vice versa.)

 Can you link to code? It doesn't sound incompatible with my
 proposal. Mutexes are blunt instruments, and easy to model. My
 whole suggestion has virtually no effect on the mutex protected
 case, which is what most people seem to talk about.

 On Wednesday, 17 October 2018 at 00:26:58 UTC, Manu wrote:
 *snip*

 Overloading for shared and unshared is possible, and may be
 desirable in many cases.
 There are also many cases where the code duplication and
 tech-debt
 does not carry its weight. It should not be required, because
 it's not technically required.

 Overloading for shared and unshared is my reason for not allowing
 implicit conversion on my types (I have no problems with implicit
 conversion being optional or disableable). The unshared function
 performs no synchronization of any kind while the shared function
 always does. This means that having an unshared and shared
 reference to the same object is unsafe.

Okay, so just to be clear, you're objecting to an immensely useful
behaviour because you exploit the current design as an optimisation
potential?
That said, I'm still not taking anything away from you... you can
still implement your class that way if you like, no change will affect
that decision.

You obviously have mechanisms in place to guarantee the
thread-local-ness of your object (otherwise it wouldn't work), so,
assuming you implement the unshared overload to be unsafe, then
situation hasn't changed for you at all...?
I don't think a robust middleware would make that trade-off, but an
application can do that if it wishes. Trading implicit safety
(enforcing it externally via application context) for perf is not
unusual at all.

 As to the larger part of preventing reading/writing of shared
 objects, I agree with its purpose and have no problems with it.

Right, the implicit cast thing is secondary to this fundamental part.
I think it's worth first focusing on getting that rule right.
shared = no read + no write .. I don't think that's objectionable.

Isaac S. <spam-no-reply-isaac outlook.com> writes:

On Wednesday, 17 October 2018 at 03:50:44 UTC, Manu wrote:
 On Tue, Oct 16, 2018 at 8:20 PM Isaac S. via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 *snip*

 Overloading for shared and unshared is my reason for not 
 allowing implicit conversion on my types (I have no problems 
 with implicit conversion being optional or disableable). The 
 unshared function performs no synchronization of any kind 
 while the shared function always does. This means that having 
 an unshared and shared reference to the same object is unsafe.

 Okay, so just to be clear, you're objecting to an immensely 
 useful
 behaviour because you exploit the current design as an 
 optimisation potential?
 That said, I'm still not taking anything away from you... you 
 can still implement your class that way if you like, no change 
 will affect that decision.

 You obviously have mechanisms in place to guarantee the
 thread-local-ness of your object (otherwise it wouldn't work), 
 so, assuming you implement the unshared overload to be unsafe, 
 then
 situation hasn't changed for you at all...?
 I don't think a robust middleware would make that trade-off, 
 but an application can do that if it wishes. Trading implicit 
 safety
 (enforcing it externally via application context) for perf is 
 not unusual at all.

I'm not objecting to the behavior (I actually really want it in 
the language as it allows for some really useful designs) but 
more-so saying it would be nice if it could be enabled or 
disabled. If its enabled-by-default, the  disable attribute could 
be used somehow (maybe " disable alias this shared;").

Even without the ability to enable/disable shared implicit 
conversion it won't break my code, it'll just allow a potential 
bug. Since it does allow for extremely useful designs, I'll 
support this change (*even if it can't be enabled/disabled*).

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/16/18 8:26 PM, Manu wrote:
 On Tue, Oct 16, 2018 at 2:20 PM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/16/18 4:26 PM, Manu wrote:
 On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 int x;

 shared int *p = &x; // allow implicit conversion, currently error

 passToOtherThread(p);

 useHeavily(&x);

 What does this mean? It can't do anything... that's the whole point here.
 I think I'm struggling here with people bringing presumptions to the
 thread. You need to assume the rules I define in the OP for the
 experiment to work.

 OK, I wrote a whole big response to this, and I went and re-quoted the
 above, and now I think I understand what the point of your statement is.

 I'll first say that if you don't want to allow implicit casting of
 shared to mutable,

 
 It's critical that this is not allowed. It's totally unreasonable to
 cast from shared to thread-local without synchronisation.

OK, so even with synchronization in the second thread when you cast, you 
still have a thread-local pointer in the originating thread WITHOUT 
synchronization.

 It's as bad as casting away const.

Of course! But shared has a different problem from const. Const allows 
the data to change through another reference, shared cannot allow 
changes without synchronization.

Changes without synchronization are *easy* with an unshared reference. 
Data can't be shared and unshared at the same time.

 then you can't allow implicit casting from mutable to
 shared. Because it's mutable, races can happen.

 
 I don't follow...

You seem to be saying that shared data is unusable. But why the hell 
have it then? At some point it has to be usable. And the agreed-upon use 
is totally defeated if you also have some stray non-shared reference to it.

 
 There is in fact, no difference between:

 int *p;
 shared int *p2 = p;
 int *p3 = cast(int*)p2;

 
 Totally illegal!! You casted away shared. That's as bad as casting away const.

But if you can't do anything with shared data, how do you use it?

 
 and this:

 int *p;
 shared int *p2 = p;
 int *p3 = p;

 
 There's nothing wrong with this... I don't understand the point?

It's identical to the top one. You now have a new unshared reference to 
shared data. This is done WITHOUT any agreed-upon synchronization.

 So really, the effort to prevent the reverse cast is defeated by
 allowing the implicit cast.

 
 Only the caller has the thread-local instance. You can take a
 thread-local pointer to a thread-local within the context of a single
 thread.
 So, it's perfectly valid for `p` and `p3` to exist in a single scope.
 `p2` is fine here too... and if that shared pointer were to escape to
 another thread, it wouldn't be a threat, because it's not readable or
 writable, and you can't make it back into a thread-local pointer
 without carefully/deliberately deployed machinery.

Huh? If shared data can never be used, why have it?

Pretend that p is not a pointer to an int, but a pointer to an UNSHARED 
type that has shared methods on it and unshared methods (for when you 
don't need any sync).

Now the shared methods will obey the sync, but the unshared ones won't. 
The result is races. I can't understand how you don't see that.

 There is a reason we disallow assigning from mutable to immutable
 without a cast. Yet, it is done in many cases, because you are sometimes
 building an immutable object with mutable pieces, and want to cast the
 final result.

 
 I don't think analogy to immutable has a place in this discussion, or
 at least, I don't understand the relevance...
 I think the reasonable analogy is const.

No, immutable is more akin to shared because immutable and mutable are 
completely different. const can point at mutable or immutable data. 
shared can't be both shared and unshared. There's no comparison. Data is 
either shared or not shared, there is no middle ground. There is no 
equivalent of const to say "this data could be shared, or could be 
unshared".

 In this case, it's ON YOU to make sure it's correct, and the traditional
 mechanism for the compiler giving you the responsibility is to require a
 cast.

 
 I think what you're talking about are behaviours relating to casting
 shared *away*, and that's some next-level shit. Handling in that case
 is no different to the way it exists today. You must guarantee that
 the pointer you possess becomes thread-local before casting it to a
 thread-local pointer.
 In my application framework, I will never cast shared away under my
 proposed design. We don't have any such global locks.

OK, so how does shared data actually operate? Somewhere, the magic has 
to turn into real code. If not casting away shared, what do you suggest?

 -----

 OK, so here is where I think I misunderstood your point. When you said a
 lock-free queue would be unusable if it wasn't shared, I thought you
 meant it would be unusable if we didn't allow the implicit cast. But I
 realize now, you meant you should be able to use a lock-free queue
 without it being actually shared anywhere.

 
 Right, a lock-free queue is a threadsafe object, and it's methods work
 whether the queue is shared or not.
 The methods are attributed shared because they can be called on shared
 instances... but they can ALSO be called from a thread-local instance,
 and under my suggested promotion rules, it's fine for the this-pointer
 to promote to shared to make the call.

It's fine in terms of a specific object we are talking about, with 
pre-determined agreements as to whether the data will be passed to other 
threads. But the compiler has no idea about these agreements. It can't 
logically determine that this is safe without you telling it, hence the 
requirement for casting.

 What I say to this is that it doesn't need to be usable. I don't care to
 use a lock-free queue in a thread-local capacity. I'll just use a normal
 queue, which is easy to implement, and doesn't have to worry about race
 conditions or using atomics. A lock free queue is a special thing, very
 difficult to get right, and only really necessary if you are going to
 share it. And used for performance reasons!

 
 I'm more interested in the object that has that lock-free queue as a
 member... it is probably a mostly thread-local object, but may have a
 couple of shared methods.

I get that, I think the shared methods just need to have unshared 
versions for the case when the queue is fully thread-local.

 I have a whole lot of objects which have 3 tiers of API access; the
 thread-local part, the threadsafe part, and the const part. Just as a
 mutable instance can call a const method, there's no reason a
 thread-local instance can't call a threadsafe method.

If you call a const method, it can squirrel away a const pointer to the 
otherwise mutable data, which is then usable later (and safe to do so).

The same cannot be said for a shared pointer. That can be then easily 
moved to another thread, WITHOUT the expectation that it was. And in 
that case, the now thread-local queue is actually shared between 
threads, and calling the thread-local API will cause races.

I didn't respond to the rest of the comments, because they were simply 
another form of "shared is similar to const", which is not true.

threadsafe use of shared data depends on ALL threads using those same 
mechanisms. If one uses simple thread-local use, then it all falls apart.

-Steve

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven 
Schveighoffer wrote:
 On 10/16/18 8:26 PM, Manu wrote:
 On Tue, Oct 16, 2018 at 2:20 PM Steven Schveighoffer via 
 Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 There is in fact, no difference between:

 int *p;
 shared int *p2 = p;
 int *p3 = cast(int*)p2;

 
 Totally illegal!! You casted away shared. That's as bad as 
 casting away const.

 But if you can't do anything with shared data, how do you use 
 it?

 
 and this:

 int *p;
 shared int *p2 = p;
 int *p3 = p;

 
 There's nothing wrong with this... I don't understand the 
 point?

 It's identical to the top one. You now have a new unshared 
 reference to shared data. This is done WITHOUT any agreed-upon 
 synchronization.

It isn't, you typo'd it (I originally missed it too).
 int *p3 = cast(int*)p2;

vs

 int *p3 = p;

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/17/18 9:58 AM, Nicholas Wilson wrote:
 On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer wrote:
 It's identical to the top one. You now have a new unshared reference 
 to shared data. This is done WITHOUT any agreed-upon synchronization.

 
 It isn't, you typo'd it (I originally missed it too).
 int *p3 = cast(int*)p2;

 
 vs
 
 int *p3 = p;


It wasn't a typo.

It's identical in that both result in a thread-local pointer equivalent 
to p. Effectively, you can "cast" away shared without having to write a 
cast.

I was trying to demonstrate the ineffectiveness of preventing implicit 
casting from shared to mutable if you allow unshared data to implicitly 
cast to shared.

It's the same problem with mutable and immutable. It's why we can't 
allow the implicit casting. Explicit casting is OK as long as you don't 
later modify the data.

In the same vein, explicit casting of local to shared is OK as long as 
you don't ever treat the data as local again. Which should requires a 
cast to say "I know what I'm doing, compiler".

-Steve

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Wednesday, 17 October 2018 at 15:51:04 UTC, Steven 
Schveighoffer wrote:
 On 10/17/18 9:58 AM, Nicholas Wilson wrote:
 On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven 
 Schveighoffer wrote:
 It's identical to the top one. You now have a new unshared 
 reference to shared data. This is done WITHOUT any 
 agreed-upon synchronization.

 
 It isn't, you typo'd it (I originally missed it too).
 int *p3 = cast(int*)p2;

 
 vs
 
 int *p3 = p;


 It wasn't a typo.

The first example assigns p2, the second assigns p (which is 
thread local) _not_ p2 (which is shared), I'm confused.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/17/18 12:27 PM, Nicholas Wilson wrote:
 On Wednesday, 17 October 2018 at 15:51:04 UTC, Steven Schveighoffer wrote:
 On 10/17/18 9:58 AM, Nicholas Wilson wrote:
 On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer 
 wrote:
 It's identical to the top one. You now have a new unshared reference 
 to shared data. This is done WITHOUT any agreed-upon synchronization.

 It isn't, you typo'd it (I originally missed it too).
 int *p3 = cast(int*)p2;

 vs

 int *p3 = p;


 It wasn't a typo.

 
 The first example assigns p2, the second assigns p (which is thread 
 local) _not_ p2 (which is shared), I'm confused.
 

Here they are again:

int *p;
shared int *p2 = p;
int *p3 = cast(int*)p2;

int *p;
shared int *p2 = p;
int *p3 = p;


I'll put some asserts in that show they accomplish the same thing:

assert(p3 is p2);
assert(p3 is p);
assert(p2 is p);

What the example demonstrates is that while you are trying to disallow 
implicit casting of a shared pointer to an unshared pointer, you have 
inadvertently allowed it by leaving behind an unshared pointer that is 
the same thing.

While we do implicitly allow mutable to cast to const, it's because 
const is a weak guarantee. It's a guarantee that the data may not change 
via *this* reference, but could change via other references.

Shared doesn't have the same characteristics. In order for a datum to be 
safely shared, it must be accessed with synchronization or atomics by 
ALL parties. If you have one party that can simply change it without 
those, you will get races.

That's why shared/unshared is more akin to mutable/immutable than 
mutable/const.

It's true that only one thread will have thread-local access. It's not 
valid any more than having one mutable alias to immutable data.

-Steve

Manu <turkeyman gmail.com> writes:

On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via
Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/17/18 12:27 PM, Nicholas Wilson wrote:
 On Wednesday, 17 October 2018 at 15:51:04 UTC, Steven Schveighoffer wrote:
 On 10/17/18 9:58 AM, Nicholas Wilson wrote:
 On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer
 wrote:
 It's identical to the top one. You now have a new unshared reference
 to shared data. This is done WITHOUT any agreed-upon synchronization.

 It isn't, you typo'd it (I originally missed it too).
 int *p3 = cast(int*)p2;

 vs

 int *p3 = p;


 It wasn't a typo.

 The first example assigns p2, the second assigns p (which is thread
 local) _not_ p2 (which is shared), I'm confused.

 Here they are again:

 int *p;
 shared int *p2 = p;
 int *p3 = cast(int*)p2;

 int *p;
 shared int *p2 = p;
 int *p3 = p;


 I'll put some asserts in that show they accomplish the same thing:

 assert(p3 is p2);
 assert(p3 is p);
 assert(p2 is p);

 What the example demonstrates is that while you are trying to disallow
 implicit casting of a shared pointer to an unshared pointer, you have
 inadvertently allowed it by leaving behind an unshared pointer that is
 the same thing.

This doesn't make sense... you're showing a thread-local program.
The thread owning the unshared pointer is entitled to the unshared
pointer. It can make as many copies at it likes. They are all
thread-local.
There's only one owning thread, and you can't violate that without unsafe casts.

 In order for a datum to be
 safely shared, it must be accessed with synchronization or atomics by
 ALL parties.

** Absolutely **

 If you have one party that can simply change it without
 those, you will get races.

*** THIS IS NOT WHAT I'M PROPOSING ***

I've explained it a few times now, but people aren't reading what I
actually write, and just assume based on what shared already does that
they know what I'm suggesting.
You need to eject all presumptions from your mind, take the rules I
offer as verbatim, and do thought experiments from there.

 That's why shared/unshared is more akin to mutable/immutable than
 mutable/const.

Only if you misrepresent my suggestion.

 It's true that only one thread will have thread-local access. It's not
 valid any more than having one mutable alias to immutable data.

And this is why the immutable analogy is invalid. It's like const.
shared offers restricted access (like const), not a different class of
thing.
There is one thread with thread-local access, and many threads with
shared access.

If a shared (threadsafe) method can be defeated by threadlocal access,
then it's **not threadsafe**, and the program is invalid.

struct NotThreadsafe
{
  int x;
  void local()
  {
    ++x; // <- invalidates the method below, you violate the other
function's `shared` promise
  }
  void notThreadsafe() shared
  {
    atomicIncrement(&x);
  }
}

struct Atomic(T)
{
  void opUnary(string op : "++")() shared { atomicIncrement(&val); }
  private T val;
}
struct Threadsafe
{
  Atomic!int x;
  void local()
  {
    ++x;
  }
  void threadsafe() shared
  {
    ++x;
  }
}

Naturally, local() is redundant, and it's perfectly fine for a
thread-local to call threadsafe() via implicit conversion.

Here's another one, where only a subset of the object is modeled to be
threadsafe (this is particularly interesting to me):

struct Threadsafe
{
  int x;
  Atomic!int y;

  void notThreadsafe()
  {
    ++x;
    ++y;
  }
  void threadsafe() shared
  {
    ++y;
  }
}

In these examples, the thread-local function *does not* undermine the
threadsafety of threadsafe(), it MUST NOT undermine the threadsafety
of threadsafe(), or else threadsafe() **IS NOT THREADSAFE**.
In the second example, you can see how it's possible and useful to do
thread-local work without invalidating the objects threadsafety
commitments.


I've said this a bunch of times, there are 2 rules:
1. shared inhibits read and write access to members
2. `shared` methods must be threadsafe

From there, shared becomes interesting and useful.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:

 I've said this a bunch of times, there are 2 rules:
 1. shared inhibits read and write access to members
 2. `shared` methods must be threadsafe

From there, shared becomes interesting and useful.


Oh God...

void atomicInc(shared int* i) { /* ... */ }

Now what? There are no "methods" for ints, only UFCS. Those 
functions can be as safe as you like, but if you allow implicit 
promotion of int* to shared int*, you *allow implicit races*.

Manu <turkeyman gmail.com> writes:

On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:

 I've said this a bunch of times, there are 2 rules:
 1. shared inhibits read and write access to members
 2. `shared` methods must be threadsafe

From there, shared becomes interesting and useful.


 Oh God...

 void atomicInc(shared int* i) { /* ... */ }

 Now what? There are no "methods" for ints, only UFCS. Those
 functions can be as safe as you like, but if you allow implicit
 promotion of int* to shared int*, you *allow implicit races*.

This function is effectively an intrinsic. It's unsafe by definition.
It's a tool for implementing threadsafe machinery.
No user can just start doing atomic operations on random ints and say
"it's threadsafe", you must encapsulate the threadsafe functionality
into some sort of object that aggregates all concerns and presents an
intellectually sound api.


Let me try one:

void free(void*) { ... }

Now what? I might have dangling pointers... it's a catastrophe!
It's essentially the same argument.
This isn't a function that professes to do something that people might
misunderstand and try to use in an unsafe way, it's a low-level
implementation device, which is used to build larger *useful*
constructs.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
 On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:

 I've said this a bunch of times, there are 2 rules:
 1. shared inhibits read and write access to members
 2. `shared` methods must be threadsafe

From there, shared becomes interesting and useful.


 Oh God...

 void atomicInc(shared int* i) { /* ... */ }

 Now what? There are no "methods" for ints, only UFCS. Those 
 functions can be as safe as you like, but if you allow 
 implicit promotion of int* to shared int*, you *allow implicit 
 races*.

 This function is effectively an intrinsic. It's unsafe by 
 definition.

Only if implicit conversion is allowed. If it isn't, that's 
likely  trusted, and this:

void atomicInc(ref shared int);

can even be  safe.

 It's a tool for implementing threadsafe machinery.
 No user can just start doing atomic operations on random ints 
 and say
 "it's threadsafe", you must encapsulate the threadsafe 
 functionality
 into some sort of object that aggregates all concerns and 
 presents an
 intellectually sound api.

Threadsafety starts and ends with the programmer. By your logic 
*all* functions operating on `shared` are unsafe then. As far as 
compiler is concerned, there would be no difference between these 
two:

struct S {}
void atomicInc(ref shared S);

and

struct S { void atomicInc() shared { /* ... */ } }

The signatures of those two functions are exactly the same. How 
is that different from a function taking a shared int pointer or 
reference?

 Let me try one:

 void free(void*) { ... }

 Now what? I might have dangling pointers... it's a catastrophe!

One could argue that it should be void free(ref void* p) { /* ... 
*/ p = null; }
As a matter of fact, in my own allocators memory blocks allocated 
by them are passed by value and are non-copyable, they're not 
just void[] as in std.experimental.allocator. One must 'move' 
them to pass ownership, and that includes deallocation. But 
that's another story altogether.

 It's essentially the same argument.
 This isn't a function that professes to do something that 
 people might
 misunderstand and try to use in an unsafe way, it's a low-level
 implementation device, which is used to build larger *useful*
 constructs.

You're missing the point, again. You have an int. You pass a 
pointer to it to some API that takes an int*. You continue to use 
your int as just an int. The API changes, and now the function 
you called previously takes a shared int*. Implicit conversion 
works, everything compiles, you have a race. Now, that's of 
course an extremely stupid scenario. The point is: the caller of 
some API *must* assert that they indeed pass shared data. It's 
insufficient for the API alone to "promise" taking shared data. 
That's the difference with promotion to `const`.

Manu <turkeyman gmail.com> writes:

On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
 On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:

 I've said this a bunch of times, there are 2 rules:
 1. shared inhibits read and write access to members
 2. `shared` methods must be threadsafe

From there, shared becomes interesting and useful.


 Oh God...

 void atomicInc(shared int* i) { /* ... */ }

 Now what? There are no "methods" for ints, only UFCS. Those
 functions can be as safe as you like, but if you allow
 implicit promotion of int* to shared int*, you *allow implicit
 races*.

 This function is effectively an intrinsic. It's unsafe by
 definition.

 Only if implicit conversion is allowed. If it isn't, that's
 likely  trusted, and this:

 void atomicInc(ref shared int);

 can even be  safe.

In this case, with respect to the context (a single int) atomicInc()
is ALWAYS safe, even with implicit conversion. You can atomicInc() a
thread-local int perfectly safely.

 It's a tool for implementing threadsafe machinery.
 No user can just start doing atomic operations on random ints
 and say
 "it's threadsafe", you must encapsulate the threadsafe
 functionality
 into some sort of object that aggregates all concerns and
 presents an
 intellectually sound api.

 Threadsafety starts and ends with the programmer. By your logic
 *all* functions operating on `shared` are unsafe then. As far as
 compiler is concerned, there would be no difference between these
 two:

 struct S {}
 void atomicInc(ref shared S);

 and

 struct S { void atomicInc() shared { /* ... */ } }

 The signatures of those two functions are exactly the same. How
 is that different from a function taking a shared int pointer or
 reference?

It's not, atomicInc() of an int is always safe with respect to the int itself.
You can call atomicInc() on an unshared int and it's perfectly fine,
but now you need to consider context, and that's a problem for the
design of the higher-level scope.

To maintain thread-safety, the int in question must be appropriately contained.

The problem is that the same as the example I presented before, which
I'll repeat:

struct InvalidProgram
{
  int x;
  void fun() { ++x; }
  void gun() shared { atomicInc(&x); }
}

The method gun() (and therefore the whole object) is NOT threadsafe by
my definition, because fun() violates the threadsafety of gun().
The situation applies equally here that:
int x;
atomicInc(&x);
++x; // <- by my definition, this 'API' (increment an int) violates
the threadsafety of atomicInc(), and atomicInc() is therefore not
threadsafe.

`int` doesn't present a threadsafe API, so int is by definition, NOT
threadsafe. atomicInc() should be  system, and not  trusted.

If you intend to share an int, use Atomic!int, because it has a threadsafe API.
atomicInc(shared int*) is effectively just an unsafe intrinsic, and
its only use is at ground-level implementation of threadsafe
machinery, like malloc() and free().

 Let me try one:

 void free(void*) { ... }

 Now what? I might have dangling pointers... it's a catastrophe!

 One could argue that it should be void free(ref void* p) { /* ...
 */ p = null; }

void *p2 = p;
free(p);
p2.crash();

 As a matter of fact, in my own allocators memory blocks allocated
 by them are passed by value and are non-copyable, they're not
 just void[] as in std.experimental.allocator. One must 'move'
 them to pass ownership, and that includes deallocation. But
 that's another story altogether.

Right, now you're talking about move semantics to implement transfer
of ownership... you might recall I was arguing this exact case to
express transferring ownership of objects between threads earlier.
This talk of blunt casts and "making sure everything is good" is all
just great, but it doesn't mean anything interesting with respect to
`shared`. It should be interesting even without unsafe casts.

 It's essentially the same argument.
 This isn't a function that professes to do something that
 people might
 misunderstand and try to use in an unsafe way, it's a low-level
 implementation device, which is used to build larger *useful*
 constructs.

 You're missing the point, again. You have an int. You pass a
 pointer to it to some API that takes an int*. You continue to use
 your int as just an int.

You have written an invalid program. I can think of an infinite number
of ways to write an invalid program.
In this case, don't have an `int`, instead, have an Atomic!int; you
now guarantee appropriate access, problem solved!
If you do have an int, don't pass it to other threads at random when
you don't have any idea what they intend to do with it! That's basic
common sense. You don't pass a pointer to a function if you don't know
what it does with the pointer!

 The API changes, and now the function
 you called previously takes a shared int*. Implicit conversion
 works, everything compiles, you have a race. Now, that's of
 course an extremely stupid scenario.

Yes.

 The point is: the caller of
 some API *must* assert that they indeed pass shared data. It's
 insufficient for the API alone to "promise" taking shared data.
 That's the difference with promotion to `const`.

The caller doesn't care if it's true that the callee can't do anything
with it that's unsafe anyway.
We effect that state by removing all non-threadsafe access.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
 On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
 On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:

 I've said this a bunch of times, there are 2 rules:
 1. shared inhibits read and write access to members
 2. `shared` methods must be threadsafe

From there, shared becomes interesting and useful.


 Oh God...

 void atomicInc(shared int* i) { /* ... */ }

 Now what? There are no "methods" for ints, only UFCS. Those 
 functions can be as safe as you like, but if you allow 
 implicit promotion of int* to shared int*, you *allow 
 implicit races*.

 This function is effectively an intrinsic. It's unsafe by 
 definition.

 Only if implicit conversion is allowed. If it isn't, that's 
 likely  trusted, and this:

 void atomicInc(ref shared int);

 can even be  safe.

 In this case, with respect to the context (a single int) 
 atomicInc()
 is ALWAYS safe, even with implicit conversion. You can 
 atomicInc() a
 thread-local int perfectly safely.

Yes, *you* can. *Another* function can't unless *you* allow for 
it to be safe. You can't do that if that function silently 
assumes you gave it shared data, when in fact you did not.

 The signatures of those two functions are exactly the same. 
 How is that different from a function taking a shared int 
 pointer or reference?

 It's not, atomicInc() of an int is always safe with respect to 
 the int itself.
 You can call atomicInc() on an unshared int and it's perfectly 
 fine,
 but now you need to consider context, and that's a problem for  
 the
 design of the higher-level scope.

 To maintain thread-safety, the int in question must be 
 appropriately contained.

Exactly. And that means it can't convert to shared without my say 
so :)

 The problem is that the same as the example I presented before, 
 which I'll repeat:

 struct InvalidProgram
 {
   int x;
   void fun() { ++x; }
   void gun() shared { atomicInc(&x); }
 }

 The method gun() (and therefore the whole object) is NOT 
 threadsafe by
 my definition, because fun() violates the threadsafety of gun().
 The situation applies equally here that:
 int x;
 atomicInc(&x);
 ++x; // <- by my definition, this 'API' (increment an int) 
 violates the threadsafety of atomicInc(), and atomicInc() is 
 therefore not threadsafe.

No. The 'API' is just the atomicInc function. You, the user of 
that API, own the int. If the API wants a shared int from you, 
you have to be in agreement. You can't have any agreement if the 
API is only making promises and assumptions.

 `int` doesn't present a threadsafe API, so int is by 
 definition, NOT threadsafe. atomicInc() should be  system, and 
 not  trusted.

Exactly. `int` isn't threadsafe and therefore cannot 
automatically convert to `shared int`.

 If you intend to share an int, use Atomic!int, because it has a 
 threadsafe API.

No. With current implementation of `shared`, which disallows your 
automatic promotion,
your intent is enforced. You cannot share a local `int` unless 
*you know* it's safe to do so and therefore can cast that int to 
shared.

 atomicInc(shared int*) is effectively just an unsafe intrinsic, 
 and

It is only unsafe if you allow int* to silently convert to shared 
int*. If you can't do that, you can't call `atomicInc` on an int*.

 One could argue that it should be void free(ref void* p) { /* 
 ... */ p = null; }


 void *p2 = p;
 free(p);
 p2.crash();

That's exactly analogous to what you're proposing: leaking 
`shared` references while keeping unshared data.

 As a matter of fact, in my own allocators memory blocks 
 allocated by them are passed by value and are non-copyable, 
 they're not just void[] as in std.experimental.allocator. One 
 must 'move' them to pass ownership, and that includes 
 deallocation. But that's another story altogether.

 Right, now you're talking about move semantics to implement 
 transfer of ownership... you might recall I was arguing this 
 exact case to express transferring ownership of objects between 
 threads earlier.

You went off on an irrelevant tangent there, and I feel like you 
didn't even see my reply. You don't pass any ownership when you 
share. You just share. As an owner, you get to access the 
un-`shared` interface freely. Others do not.

 This talk of blunt casts and "making sure everything is good" 
 is all just great, but it doesn't mean  anything interesting 
 with respect to `shared`. It should be interesting even without 
 unsafe casts.

Again, I feel like you didn't see my reply. It's not talk about 
just blunt casts to make sure everything is good. You either have 
shared data to begin with, and so can share it freely, or you 
*know* that you can share this particular piece of data, even if 
it itself isn't marked `shared`, and you *assert* that by 
deliberately casting. *You* know, *you* cast, not "some function 
expects you to know, and just auto-casts".

 You're missing the point, again. You have an int. You pass a 
 pointer to it to some API that takes an int*. You continue to 
 use your int as just an int.


 You have written an invalid program. I can think of an infinite 
 number of ways to write an invalid program.

No, I have not. I didn't make any promises that my data was 
shared, and I wasn't expecting it to be treated as such. I didn't 
even author that API. The other guy (the API) made wrong 
assumptions. Don't you see the difference here?

 In this case, don't have an `int`, instead, have an Atomic!int; 
 you now guarantee appropriate access, problem solved!

No, it isn't. My int *is* thread-local, *I* don't need an 
Atomic!int, I didn't sanction that int's use as shared in any 
way. Yet the automatic conversion presumes that I have.

As I stated previously, there's no difference between Atomic!int 
and free functions operating on shared int* (or ref shared int). 
Struct methods are sugared versions of those free functions, 
*nothing more*. That's why we have UFCS.

 If you do have an int, don't pass it to other threads at random 
 when you don't have any idea what they intend to do with it!

*I* don't pass it to other threads at random, and I expect other 
code to not do so without *my* approval. What approval can I give 
if *other* code can silently assume I'm giving it shared data, 
when in fact I'm not?

 That's basic common sense. You don't pass a pointer to a 
 function if you don't know what it does with the pointer!

I should know what the function does with the pointer from it's 
signature. Now, currently in D that very blurry. *Hopefully* with 
'scope', DIP25 and DIP1000 this becomes more common. But that's 
at least what we should strive for.
If a function takes `shared`, I better be sure I'm giving it 
`shared`. The only way to do so is either have `shared` to begin 
with, or explicitly cast when I know I can do so.

 The point is: the caller of
 some API *must* assert that they indeed pass shared data. It's
 insufficient for the API alone to "promise" taking shared data.
 That's the difference with promotion to `const`.


 The caller doesn't care if it's true that the callee can't do 
 anything with it that's unsafe anyway.
 We effect that state by removing all non-threadsafe access.

You allow non-threadsafe access with implicit cast, not remove it.

This broken record is getting very tiresome... Let me ask you 
this once again: *why* are you so bent on this implicit 
conversion from mutable to shared? So far the only reason I've 
seen is to just avoid writing additional methods that forward to 
`shared` methods.

Most of your casts will be the other way around, will have to be 
explicit and there's nothing that can be done about that. You'll 
only have mutable->shared casts in few select cases, exactly 
because they're corner cases where you *need* to make the 
decision clear.

Erik van Velzen <erik evanv.nl> writes:

I don't have anything to add that hasn't been said yet but it's 
good to see some thinking on this subject. It feels like progress.

Manu <turkeyman gmail.com> writes:

On Wed, Oct 17, 2018 at 5:35 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
 On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
 On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:

 I've said this a bunch of times, there are 2 rules:
 1. shared inhibits read and write access to members
 2. `shared` methods must be threadsafe

From there, shared becomes interesting and useful.


 Oh God...

 void atomicInc(shared int* i) { /* ... */ }

 Now what? There are no "methods" for ints, only UFCS. Those
 functions can be as safe as you like, but if you allow
 implicit promotion of int* to shared int*, you *allow
 implicit races*.

 This function is effectively an intrinsic. It's unsafe by
 definition.

 Only if implicit conversion is allowed. If it isn't, that's
 likely  trusted, and this:

 void atomicInc(ref shared int);

 can even be  safe.

 In this case, with respect to the context (a single int)
 atomicInc()
 is ALWAYS safe, even with implicit conversion. You can
 atomicInc() a
 thread-local int perfectly safely.

 Yes, *you* can. *Another* function can't unless *you* allow for
 it to be safe. You can't do that if that function silently
 assumes you gave it shared data, when in fact you did not.

 The signatures of those two functions are exactly the same.
 How is that different from a function taking a shared int
 pointer or reference?

 It's not, atomicInc() of an int is always safe with respect to
 the int itself.
 You can call atomicInc() on an unshared int and it's perfectly
 fine,
 but now you need to consider context, and that's a problem for
 the
 design of the higher-level scope.

 To maintain thread-safety, the int in question must be
 appropriately contained.

 Exactly. And that means it can't convert to shared without my say
 so :)

It can though, because `int` doesn't have any shared methods; it's
inaccessible. You can't do anything with a shared int, so it's safe to
distribute regardless.

 The problem is that the same as the example I presented before,
 which I'll repeat:

 struct InvalidProgram
 {
   int x;
   void fun() { ++x; }
   void gun() shared { atomicInc(&x); }
 }

 The method gun() (and therefore the whole object) is NOT
 threadsafe by
 my definition, because fun() violates the threadsafety of gun().
 The situation applies equally here that:
 int x;
 atomicInc(&x);
 ++x; // <- by my definition, this 'API' (increment an int)
 violates the threadsafety of atomicInc(), and atomicInc() is
 therefore not threadsafe.

 No. The 'API' is just the atomicInc function. You, the user of
 that API, own the int. If the API wants a shared int from you,
 you have to be in agreement. You can't have any agreement if the
 API is only making promises and assumptions.

Okay, here's an idea... atomicInc() should NOT take `shared int*`,
change atomicInt() to receive int*.
We agree that `int` is not a threadsafe type, atomicInc() is not a
threadsafe method, and therefore shouldn't be shared.

If you have a shared(int)*, and you want to do un-threadsafe
atomicInc(), you need to cast to int*.
I think this is actually correct by my definition, because we
recognise int is not a threadsafe type, and atomicInc() is not a
threadsafe method, so hard-cast is required, and programmer must
appropriately validate the context when doing an unsafe operation.

 `int` doesn't present a threadsafe API, so int is by
 definition, NOT threadsafe. atomicInc() should be  system, and
 not  trusted.

 Exactly. `int` isn't threadsafe and therefore cannot
 automatically convert to `shared int`.

It can convert to shared int, because it's inaccessible... you can't
do anything with a `shared int` except unsafe cast shared away. Once
you're in unsafe land, you are responsible for assuring correct
conditions.

 If you intend to share an int, use Atomic!int, because it has a
 threadsafe API.

 No. With current implementation of `shared`, which disallows your
 automatic promotion,
 your intent is enforced. You cannot share a local `int` unless
 *you know* it's safe to do so and therefore can cast that int to
 shared.

Yes, but we're not talking about current definition of shared. Stop
confusing the conversation.
This thread is about the rules I define.

 atomicInc(shared int*) is effectively just an unsafe intrinsic,
 and

 It is only unsafe if you allow int* to silently convert to shared
 int*. If you can't do that, you can't call `atomicInc` on an int*.

I allow conversion to shared. Assume that, and then solve.

Actually, I think I realise the problem. We're not stuck on a failure
of definition, we're stressing over an invalid API!
It's incorrect for atomicInt to receive a shared(int)*, because it
doesn't (and can't) promise thread-safety.
It must receive int*.
The implementation of Atomic which encapsulates access (guards against
unsafe access) to the int will do the proper cast.

 One could argue that it should be void free(ref void* p) { /*
 ... */ p = null; }


 void *p2 = p;
 free(p);
 p2.crash();

 That's exactly analogous to what you're proposing: leaking
 `shared` references while keeping unshared data.

Right, and I'm saying it's completely acceptable and normal.
We don't need to be blocked on this matter.
It's a low-level intrinsic, it isn't a general use API. Can we please
look past it.

void atomicInc(int*); // <- we'll assume this moving forward.

As far as I can tell, this is the ONLY issue you have with the design.
Please find another issue.

 This talk of blunt casts and "making sure everything is good"
 is all just great, but it doesn't mean  anything interesting
 with respect to `shared`. It should be interesting even without
 unsafe casts.

 Again, I feel like you didn't see my reply. It's not talk about
 just blunt casts to make sure everything is good. You either have
 shared data to begin with, and so can share it freely, or you
 *know* that you can share this particular piece of data, even if
 it itself isn't marked `shared`, and you *assert* that by
 deliberately casting. *You* know, *you* cast, not "some function
 expects you to know, and just auto-casts".

Sorry, I can't understand this paragraph...

 You're missing the point, again. You have an int. You pass a
 pointer to it to some API that takes an int*. You continue to
 use your int as just an int.


 You have written an invalid program. I can think of an infinite
 number of ways to write an invalid program.

 No, I have not. I didn't make any promises that my data was
 shared, and I wasn't expecting it to be treated as such. I didn't
 even author that API. The other guy (the API) made wrong
 assumptions. Don't you see the difference here?

I do not. I see it black-and-white, it you write `shared` on a method,
you made a promise to do threadsafety.
You must deliver on that promise. If you do not deliver that promise,
the program is invalid.
If you *can not* deliver on that promise, then you can not attribute
the method shared (or receive a shared argument).

 As I stated previously, there's no difference between Atomic!int
 and free functions operating on shared int* (or ref shared int).
 Struct methods are sugared versions of those free functions,
 *nothing more*. That's why we have UFCS.

I'm not sure what you're saying. Atomic encapsulates guaranteed
threadsafe access to an int, whereas shared(int)* doesn't and can't.
Any threadsafe API that receives an int will necessarily receive an
Atomic!int, unless it's deliberately doing un-safety.

 That's basic common sense. You don't pass a pointer to a
 function if you don't know what it does with the pointer!

 I should know what the function does with the pointer from it's
 signature. Now, currently in D that very blurry. *Hopefully* with
 'scope', DIP25 and DIP1000 this becomes more common. But that's
 at least what we should strive for.
 If a function takes `shared`, I better be sure I'm giving it
 `shared`.

You don't care, because you are certain that it can't do anything
that's unsafe with your object.

 The point is: the caller of
 some API *must* assert that they indeed pass shared data. It's
 insufficient for the API alone to "promise" taking shared data.
 That's the difference with promotion to `const`.


 The caller doesn't care if it's true that the callee can't do
 anything with it that's unsafe anyway.
 We effect that state by removing all non-threadsafe access.

 You allow non-threadsafe access with implicit cast, not remove it.

Why do you keep saying this?

threadsafe methods!

I **ONLY** allow threadsafe access by allowing implicit conversion.

 This broken record is getting very tiresome...

It is. I don't know how to be clearer than I have been.

 Let me ask you
 this once again: *why* are you so bent on this implicit
 conversion from mutable to shared? So far the only reason I've
 seen is to just avoid writing additional methods that forward to
 `shared` methods.

Because users of an API shouldn't have to riddle their code with
manual casts to call perfectly safe functions.
That would be a plain-as-day indicator that the design is wrong.

I need a some real examples of how the implicit cast violates threadsafety...
We've identified the issue with atomicInc(), and it's the only thing
you're stuck on, and you've never picked any general holes in the
design. Please find a legitimate hole in the design...

 Most of your casts will be the other way around, will have to be
 explicit and there's nothing that can be done about that. You'll
 only have mutable->shared casts in few select cases, exactly
 because they're corner cases where you *need* to make the
 decision clear.

If shared means "threadsafe or no access", then you don't need to make
the decision clear... you don't need to make the decision at all. It's
safe by definition.
I'm trying to create a situation where users are not concerned that
they were able to use an API correctly unless they're *implementing*
threadsafety; that's when they need to think, and any requirement for
un-safety and casting should be factored into that space, not the
calling space.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

Manu,

how is it that you can't see what *your own* proposal means??? 
Implicit casting from mutable to shared means that everything is 
shared by default! Precisely the opposite of what D proclaims.

You also essentially forbid defining *any* functions that take 
`shared T*` argument(s). You keep asking for concrete "holes". 
Don't you see what the previous "atomicInc" example implies???

If *any* free function `foo(shared T* bar)`, per your definition, 
is not threadsafe, then no other function with shared argument(s) 
can be threadsafe at all. So how do you call functions on shared 
data then? You keep saying "methods, methods..."

struct Other { /* ... */ }

struct S {
     void foo(shared Other*) shared;
}

Per your rules, there would be *nothing* in the language to 
prevent calling S.foo with an unshared Other.

So the only way to make your proposal work would be to forbid all 
functions from taking `shared T*` or `ref shared T` argument. 
Except we can't do that, because a method is just a function with 
an implicit first argument. The code above is the same as this:

void foo(ref shared S, shared Other*);

It's literally *the same signature*. So there's nothing in the 
language to prevent calling that on an unshared S either.

To sum up, things you implied but never specified in your 
proposal:

1. Primitive types can't be explicitly `shared`.
2. Free functions taking `shared` arguments are not allowed.
3. Only `shared` methods can implement threadsafe operations on 
`shared` data (which contradicts (2) already) <- this one you did 
specify.
4. Every variable is implicitly shared, whether intended so or 
not.

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Thursday, 18 October 2018 at 10:08:48 UTC, Stanislav Blinov 
wrote:
 Manu,

 how is it that you can't see what *your own* proposal means??? 
 Implicit casting from mutable to shared means that everything 
 is shared by default! Precisely the opposite of what D 
 proclaims.

Well, sorta. But that's not a problem, because you can't do 
anything that's not threadsafe to something that's shared.


 You also essentially forbid defining *any* functions that take 
 `shared T*` argument(s). You keep asking for concrete "holes". 
 Don't you see what the previous "atomicInc" example implies???

I certainly don't. Please do elucidate.


 If *any* free function `foo(shared T* bar)`, per your 
 definition, is not threadsafe, then no other function with 
 shared argument(s) can be threadsafe at all. So how do you call 
 functions on shared data then? You keep saying "methods, 
 methods..."

 struct Other { /* ... */ }

 struct S {
     void foo(shared Other*) shared;
 }

 Per your rules, there would be *nothing* in the language to 
 prevent calling S.foo with an unshared Other.

That's true. And you can't do anything to it, so that's fine.


 So the only way to make your proposal work would be to forbid 
 all functions from taking `shared T*` or `ref shared T` 
 argument.

No. Please read this thread again. From the beginning, every 
word. Actually, don't do that, because Manu's proposal is simple 
and elegant:

 1. the rule must be applied that shared object can not be read 
 or written
 2. attributing a method shared is a statement and a promise 
 that the
 method is threadsafe
 
 The rest just follows naturally.

There's actually one more thing: The one and only thing you can 
do (without unsafe casting) with a shared object, is call shared 
methods and free functions on it.


 To sum up, things you implied but never specified in your 
 proposal:

 1. Primitive types can't be explicitly `shared`.

Sure they can, they just can't present a thread-safe interface, 
so you can't do anything with a shared(int).


 2. Free functions taking `shared` arguments are not allowed.

Yes, they are. They would be using other shared methods or free 
functions on the shared argument, and would thus be thread-safe. 
If defined in the same module as the type on which they operate, 
they would have access to the internal state of the object, and 
would have to be written in such a way as to not violate the 
thread-safety of other methods and free functions that operate on 
it.


 3. Only `shared` methods can implement threadsafe operations on 
 `shared` data (which contradicts (2) already) <- this one you 
 did specify.

Non-shared methods are perfectly free to be thread-safe (and they 
should be, in the sense that they shouldn't interfere with shared 
methods). A better way to state this is that only shared methods 
may be called on a shared object. A shared object may also be 
passed to a function taking a shared parameter.


 4. Every variable is implicitly shared, whether intended so or 
 not.

Well, yes, in the same sense that every variable is also 
implicitly const, whether intended so or not.

--
   Simen

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 11:35:21 UTC, Simen Kjærås wrote:
 On Thursday, 18 October 2018 at 10:08:48 UTC, Stanislav Blinov 
 wrote:
 Manu,

 how is it that you can't see what *your own* proposal means??? 
 Implicit casting from mutable to shared means that everything 
 is shared by default! Precisely the opposite of what D 
 proclaims.

 Well, sorta. But that's not a problem, because you can't do 
 anything that's not threadsafe to something that's shared.

Yes you can. You silently agree to another function's assumption 
that you pass shared data, while actually passing thread-local 
data and keeping treating it as thread-local. I.e. you silently 
agree to a race.

 You also essentially forbid defining *any* functions that take 
 `shared T*` argument(s). You keep asking for concrete "holes". 
 Don't you see what the previous "atomicInc" example implies???

 I certainly don't. Please do elucidate.

What the hell? I do in the very next paragraph. Do people read 
sentence by sentence and assume context does not exist or what?

 If *any* free function `foo(shared T* bar)`, per your 
 definition, is not threadsafe, then no other function with 
 shared argument(s) can be threadsafe at all. So how do you 
 call functions on shared data then? You keep saying "methods, 
 methods..."

 struct Other { /* ... */ }

 struct S {
     void foo(shared Other*) shared;
 }

 Per your rules, there would be *nothing* in the language to 
 prevent calling S.foo with an unshared Other.

 That's true. And you can't do anything to it, so that's fine.

Yes you can do "anything" to it. If you couldn't, you wouldn't be 
able to implement `shared` at all. Forbidding reads and writes 
isn't enough to guarantee that you "can't do anything with it". 
*Unless* you forbid implicit conversion from mutable to shared. 
Then, and only then, your statement can hold.

 So the only way to make your proposal work would be to forbid 
 all functions from taking `shared T*` or `ref shared T` 
 argument.

 No. Please read this thread again. From the beginning, every 
 word.

Are you kidding me? Maybe it's *you* who should do that?..

 Actually, don't do that, because Manu's proposal is simple and 
 elegant:

 1. the rule must be applied that shared object can not be read 
 or written


No objection there, I fully support that. I even stated multiple 
times how it can be extended and why.

 2. attributing a method shared is a statement and a promise 
 that the method is threadsafe


No objection here either.

 The rest just follows naturally.


Nothing follows naturally. The proposal doesn't talk at all about 
the fact that you can't have "methods" on primitives, that you 
can't distinguish between shared and unshared data if that 
proposal is realized, that you absolutely destroy D's 
TLS-by-default treatment...

 There's actually one more thing: The one and only thing you can 
 do (without unsafe casting) with a shared object, is call 
 shared methods and free functions on it.

Functions that you must not be allowed to write per this same 
proposal. How quaint.

 To sum up, things you implied but never specified in your 
 proposal:

 1. Primitive types can't be explicitly `shared`.

 Sure they can, they just can't present a thread-safe interface, 
 so you can't do anything with a shared(int).

Ergo... you can't have functions taking pointers to shared 
primitives. Ergo, `shared <primitive type>` becomes a useless 
language construct.

 2. Free functions taking `shared` arguments are not allowed.

 Yes, they are. They would be using other shared methods or free 
 functions on the shared argument, and would thus be 
 thread-safe. If defined in the same module as the type on which 
 they operate, they would have access to the internal state of 
 the object, and would have to be written in such a way as to 
 not violate the thread-safety of other methods and free 
 functions that operate on it.

This contradicts (1). Either you can have functions taking shared 
T* arguments, thus
creating threadsafe interface for them, or you can't. If, per (1) 
as you say, you can't

 3. Only `shared` methods can implement threadsafe operations 
 on `shared` data (which contradicts (2) already) <- this one 
 you did specify.

 Non-shared methods are perfectly free to be thread-safe (and 
 they should be, in the sense that they shouldn't interfere with 
 shared methods). A better way to state this is that only shared 
 methods may be called on a shared object. A shared object may 
 also be passed to a function taking a shared parameter.


 4. Every variable is implicitly shared, whether intended so or 
 not.


 Well, yes, in the same sense that every variable is also 
 implicitly const, whether intended so or not.

I sort of expected that answer. No, nothing is implicitly const. 
When you pass a reference to a function taking const, *you keep 
mutable reference*, the function agrees to that, and it's only 
"promise" is to not modify data through the reference you gave 
it. But *you still keep mutable reference*. Just as you would 
keep *unshared mutable* reference if implicit conversion from 
mutable to shared existed.

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Thursday, 18 October 2018 at 12:15:07 UTC, Stanislav Blinov 
wrote:
 On Thursday, 18 October 2018 at 11:35:21 UTC, Simen Kjærås 
 wrote:
 On Thursday, 18 October 2018 at 10:08:48 UTC, Stanislav Blinov 
 wrote:
 Manu,

 how is it that you can't see what *your own* proposal 
 means??? Implicit casting from mutable to shared means that 
 everything is shared by default! Precisely the opposite of 
 what D proclaims.

 Well, sorta. But that's not a problem, because you can't do 
 anything that's not threadsafe to something that's shared.

 Yes you can. You silently agree to another function's 
 assumption that you pass shared data, while actually passing 
 thread-local data and keeping treating it as thread-local. I.e. 
 you silently agree to a race.

No, you don't. If I give you a locked box with no obvious way to 
open it, I can expect you not to open it. It's the same thing. If 
you have a shared(T), and it doesn't define a thread-safe 
interface, you can do nothing with it. If you are somehow able to 
cause a race with something with which you can do nothing, please 
tell me how, because I'm pretty sure that implies the very laws 
of logic are invalid.


 If *any* free function `foo(shared T* bar)`, per your 
 definition, is not threadsafe, then no other function with 
 shared argument(s) can be threadsafe at all. So how do you 
 call functions on shared data then? You keep saying "methods, 
 methods..."

 struct Other { /* ... */ }

 struct S {
     void foo(shared Other*) shared;
 }

 Per your rules, there would be *nothing* in the language to 
 prevent calling S.foo with an unshared Other.

 That's true. And you can't do anything to it, so that's fine.

 Yes you can do "anything" to it.

No, you can't. You can do thread-safe things to it. That's 
nothing, *unless* Other defines a shared (thread-safe) interface, 
in which case it's safe, and everything is fine.

Example:

struct Other {
     private Data payload;

     // shared function. Thread-safe, can be called from a
     // shared object, or from an unshared object.
     void twiddle() shared { payload.doSharedStuff(); }

     // unshared function. Cannot be called from a shared object.
     // Promises not to interfere with shared data, or to so only
     // in thread-safe ways (by calling thread-safe methods, or
     // by taking a mutex or equivalent).
     void twaddle() { payload.doSharedThings(); }

     // Bad function. Promises not to interfere with shared data,
     // but does so anyway.
     // Give the programmer a stern talking-to.
     void twank() {
         payload.fuckWith();
     }
}

struct S {
    void foo(shared Other* o) shared {
        // No can do - can't call non-shared functions on shared 
object.
        // o.twaddle();

        // Can do - twiddle is always safe to call.
        o.twiddle();
    }
}

 If you couldn't, you wouldn't be able to implement `shared` at 
 all. Forbidding reads and writes isn't enough to guarantee that 
 you "can't do anything with it".

Alright, so I have this shared object that I can't read from, and 
can't write to. It has no public shared members. What can I do 
with it? I can pass it to other guys, who also can't do anything 
with it. Are there other options?


 The rest just follows naturally.


 Nothing follows naturally. The proposal doesn't talk at all 
 about the fact that you can't have "methods" on primitives,

You can't have thread-safe methods operating directly on 
primitives, because they already present a non-thread-safe 
interface. This is true. This follows naturally from the rules.


 that you can't distinguish between shared and unshared data if 
 that proposal is realized,

And you can't do that currently either. Just like today, 
shared(T) means the T may or may not be shared with other thread. 
Nothing more, nothing less.


 that you absolutely destroy D's TLS-by-default treatment...

I'm unsure what you mean by this.


 There's actually one more thing: The one and only thing you 
 can do (without unsafe casting) with a shared object, is call 
 shared methods and free functions on it.

 Functions that you must not be allowed to write per this same 
 proposal. How quaint.

What? Which functions can't I write?

// Safe, regular function operating on shared data.
void foo(shared(Other)* o) {
     o.twiddle(); // Works great!
}

// Unsafe function. Should belong somewhere deep in druntime
// and only be used by certified wizards.
void bar(shared(int)* i) {
     atomicOp!"++"(i);
}

 1. Primitive types can't be explicitly `shared`.

 Sure they can, they just can't present a thread-safe 
 interface, so you can't do anything with a shared(int).

 Ergo... you can't have functions taking pointers to shared 
 primitives. Ergo, `shared <primitive type>` becomes a useless 
 language construct.

Yup, this is correct. But wrap it in a struct, like e.g. 
Atomic!int, and everything's hunky-dory.


 2. Free functions taking `shared` arguments are not allowed.

 Yes, they are. They would be using other shared methods or 
 free functions on the shared argument, and would thus be 
 thread-safe. If defined in the same module as the type on 
 which they operate, they would have access to the internal 
 state of the object, and would have to be written in such a 
 way as to not violate the thread-safety of other methods and 
 free functions that operate on it.

 This contradicts (1). Either you can have functions taking 
 shared T* arguments, thus
 creating threadsafe interface for them, or you can't. If, per 
 (1) as you say, you can't

I have no idea where I or Manu have said you can't make functions 
that take shared(T)*.


 4. Every variable is implicitly shared, whether intended so 
 or not.


 Well, yes, in the same sense that every variable is also 
 implicitly const, whether intended so or not.

 I sort of expected that answer. No, nothing is implicitly 
 const. When you pass a reference to a function taking const, 
 *you keep mutable reference*, the function agrees to that, and 
 it's only "promise" is to not modify data through the reference 
 you gave it. But *you still keep mutable reference*. Just as 
 you would keep *unshared mutable* reference if implicit 
 conversion from mutable to shared existed.

Yup, and that's perfectly fine, because 'shared' means 
'thread-safe'. I think Manu might have mentioned that once.

If a type presents both a shared and a non-shared interface, and 
the non-shared interface may do things that impact the shared 
part, these things must be done in a thread-safe manner. If 
that's not the case, you have a bug. The onus is on the creator 
of a type to do this.

Let's say it together: for a type to be thread-safe, all of its 
public members must be written in a thread-safe way. If a 
non-shared method may jeopardize this, the type is not 
thread-safe, and shouldn't provide a shared interface.

--
   Simen

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Thursday, 18 October 2018 at 13:09:10 UTC, Simen Kjærås wrote:
 Ergo... you can't have functions taking pointers to shared 
 primitives. Ergo, `shared <primitive type>` becomes a useless 
 language construct.

 Yup, this is correct. But wrap it in a struct, like e.g. 
 Atomic!int, and everything's hunky-dory.

Sorry, small mistake here. You're correct that shared <primitive 
type> becomes useless, except as a way to signal that there's 
something there, and you can't touch it. I was not replying to 
the part saying 'you can't have functions taking pointers to 
shared primitives'. That's just patently false. However, I don't 
see why you'd want such a function, since it can't do anything 
with what you pass it.

--
   Simen

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 13:09:10 UTC, Simen Kjærås wrote:

 Well, sorta. But that's not a problem, because you can't do 
 anything that's not threadsafe to something that's shared.

 Yes you can. You silently agree to another function's 
 assumption that you pass shared data, while actually passing 
 thread-local data and keeping treating it as thread-local. 
 I.e. you silently agree to a race.

 No, you don't. If I give you a locked box with no obvious way 
 to open it, I can expect you not to open it.

You contradict yourself and don't even notice it. Per your rules, 
the way to open that locked box is have shared methods that 
access data via casting. Also per your rules, there is absolutely 
no way for the programmer to control whether they're actually 
sharing the data. Therefore, some API can steal a shared 
reference without your approval, use that with your "safe" shared 
methods, while you're continuing to threat your data as not 
shared.

 It's the same thing. If you have a shared(T), and it doesn't 
 define a thread-safe interface, you can do nothing with it. If 
 you are somehow able to cause a race with something with which 
 you can do nothing, please tell me how, because I'm pretty sure 
 that implies the very laws of logic are invalid.

You and Manu both seem to think that methods allow you to "define 
a thread-safe interface".

struct S {
     void foo() shared;
}

Per your rules, S.foo is thread-safe. It is here that I remind 
you, *again*, what S.foo actually looks like, given made-up 
easy-to-read mangling:

void struct_S_foo(ref shared S);

And yet, for some reason, you think that these are not 
thread-safe:

void foo(shared int*);
void bar(ref shared int);

I mean, how does that logic even work with you?

 Per your rules, there would be *nothing* in the language to 
 prevent calling S.foo with an unshared Other.

 That's true. And you can't do anything to it, so that's fine.

 Yes you can do "anything" to it.

 No, you can't. You can do thread-safe things to it. That's 
 nothing, *unless* Other defines a shared (thread-safe) 
 interface, in which case it's safe, and everything is fine.

 Example:

 struct Other {
     private Data payload;

     // shared function. Thread-safe, can be called from a
     // shared object, or from an unshared object.
     void twiddle() shared { payload.doSharedStuff(); }

     // unshared function. Cannot be called from a shared object.
     // Promises not to interfere with shared data, or to so only
     // in thread-safe ways (by calling thread-safe methods, or
     // by taking a mutex or equivalent).
     void twaddle() { payload.doSharedThings(); }

     // Bad function. Promises not to interfere with shared data,
     // but does so anyway.
     // Give the programmer a stern talking-to.
     void twank() {
         payload.fuckWith();
     }
 }

 struct S {
    void foo(shared Other* o) shared {
        // No can do - can't call non-shared functions on shared 
 object.
        // o.twaddle();

        // Can do - twiddle is always safe to call.
        o.twiddle();
    }
 }

That's already wrong starting at line 2. It should be:

struct Other {
     private shared Data payload; // shared, there's no question 
about it

     // shared function. Thread-safe, can be called from a
     // shared object, or from an unshared object.
     void twiddle() shared { payload.doSharedStuff(); }

     // unshared function. Cannot be called from a shared object.
     // Promises not to interfere with shared data, or to so only
     // in thread-safe ways (by calling thread-safe methods, or
     // by taking a mutex or equivalent).
     void twaddle() {
         // fine so long as there's a
         // 'auto doSharedThings(ref shared Data)'
         // or an equivalent method for Data.
         // Otherwise it just wouldn't compile, as it should.
         payload.doSharedThings();
     }

     // No longer a bad function, because it doesn't compile, and 
the
     // programmer can do their own auto-spanking.
     void twank() {
         payload.fuckWith(); // Error: cannot fuckWith() 'shared 
Data'
     }
}

struct S {
    void foo(shared Other* o) shared {
        // No can do - can't call non-shared functions on shared 
object.
        // o.twaddle();
        // ^Yep, agreed

        // Can do - twiddle is always safe to call.
        o.twiddle();
    }
}

Well, that was easy, wasn't it?

Your implementation of 'twaddle' is *unsafe*, because the 
compiler doesn't know that 'payload' is shared. For example, when 
inlining, it may reorder the calls in it and cause races or other 
UB. At least one of the reasons behind `shared` *was* to serve as 
compiler barrier.
What I don't see in your example is where it would be necessary 
to cast mutable to shared, let alone auto-cast it. And that's the 
heart of this discussion.

If you just do this:

auto other = new /*shared*/ Other;

...then at the moment, per current rules, you can either twiddle 
or twaddle (depending on whether you remove the comment or not), 
but not both, despite being a sole owner of 'Other'. This is the 
only place where I can see *some small* value in automatic 
conversion. But I'd much rather have a language that strictly 
forbids me to do nasty things than provides small conveniences in 
corner cases.

 If you couldn't, you wouldn't be able to implement `shared` at 
 all. Forbidding reads and writes isn't enough to guarantee 
 that you "can't do anything with it".

 Alright, so I have this shared object that I can't read from, 
 and can't write to. It has no public shared members. What can I 
 do with it? I can pass it to other guys, who also can't do 
 anything with it. Are there other options?

It can have any number of public shared "members" per UFCS. The 
fact that you forget is that there's no difference between a 
method and a free function, other than syntax sugar. Well, OK, 
there's guaranteed private access for methods, but same is true 
for module members.

 The rest just follows naturally.


 Nothing follows naturally. The proposal doesn't talk at all 
 about the fact that you can't have "methods" on primitives,

 You can't have thread-safe methods operating directly on 
 primitives, because they already present a non-thread-safe 
 interface. This is true. This follows naturally from the rules.

Everything in D already presents a non-threadsafe interface. 
Things that you advocate included.

struct S {
     void foo() shared;
}

That is not threadsafe. This *sort of* is:

struct S {
      disable this(this);
      disable void opAssign(S);

     void foo() shared;
}

...except not quite still. Now, if the compiler generated above 
in the presence of any `shared` members or methods, then we could 
begin talking about it being threadsafe. But that part is 
mysteriously missing from Manu's proposal, even though I keep 
reminding of this in what feels like every third post or so (I'm 
probably grossly exaggerating).

 that you can't distinguish between shared and unshared data if 
 that proposal is realized,


 And you can't do that currently either. Just like today, 
 shared(T) means the T may or may not be shared with other 
 thread. Nothing more, nothing less.

I don't think it means what you think it means. "May or may not 
be shared with other thread" means "you MUST treat it as if it's 
shared with other thread". That's it. That's why automatic 
conversion doesn't make *any* sense, and that's why compiler 
error on attempting to pass over mutable as shared makes 
*perfect* sense.

 that you absolutely destroy D's TLS-by-default treatment...

 I'm unsure what you mean by this.

You lose the ability to distinguish thread-local and shared data.

 There's actually one more thing: The one and only thing you 
 can do (without unsafe casting) with a shared object, is call 
 shared methods and free functions on it.

 Functions that you must not be allowed to write per this same 
 proposal. How quaint.

 What? Which functions can't I write?

 // Safe, regular function operating on shared data.
 void foo(shared(Other)* o) {
     o.twiddle(); // Works great!
 }
 // Unsafe function. Should belong somewhere deep in druntime
 // and only be used by certified wizards.
 void bar(shared(int)* i) {
     atomicOp!"++"(i);
 }

Uh-huh, only due to some weird convention that "methods" are 
somehow safer than free functions. Which they're not.

 1. Primitive types can't be explicitly `shared`.

 Sure they can, they just can't present a thread-safe 
 interface, so you can't do anything with a shared(int).

 Ergo... you can't have functions taking pointers to shared 
 primitives. Ergo, `shared <primitive type>` becomes a useless 
 language construct.

 Yup, this is correct. But wrap it in a struct, like e.g. 
 Atomic!int, and everything's hunky-dory.

So again,

void atomicInc(shared int*); // is "not safe", but
void struct_Atomic_int_opUnary_plus_plus(ref shared Atomic); // 
is "safe"

just because latter is a "method". And that, by you, is 
hunky-dory? Whether it's a method or a free function, it's 
written to work on *shared* data. Of course it wouldn't be safe 
if you allow any non-shared data to become shared without the 
programmer having a say in this.

 I have no idea where I or Manu have said you can't make 
 functions that take shared(T)*.

Because you keep saying they're unsafe and that you should wrap 
them up in a struct for no other reason than just "because 
methods are kosher".

 I sort of expected that answer. No, nothing is implicitly 
 const. When you pass a reference to a function taking const, 
 *you keep mutable reference*, the function agrees to that, and 
 it's only "promise" is to not modify data through the 
 reference you gave it. But *you still keep mutable reference*. 
 Just as you would keep *unshared mutable* reference if 
 implicit conversion from mutable to shared existed.

 Yup, and that's perfectly fine, because 'shared' means 
 'thread-safe'. I think Manu might have mentioned that once.

 If a type presents both a shared and a non-shared interface, 
 and the non-shared interface may do things that impact the 
 shared part, these things must be done in a thread-safe manner.
 If that's not the case, you have a bug. The onus is on the 
 creator of a type to do this.

Yes, that part is perfectly fine with me.

 Let's say it together: for a type to be thread-safe, all of its 
 public members must be written in a thread-safe way.

It's shared private parts also must be written in a thread-safe 
way. Yes, they're private, but they still may be shared. Welcome 
to the communism of multithreading.

 If a non-shared method may jeopardize this, the type is not 
 thread-safe, and shouldn't provide a shared interface.

It can't jeopardize anything so long as you actually treat your 
shared data accordingly, and don't just magically assume unshared 
parts to be shared for some reason.

aliak <something something.com> writes:

On Thursday, 18 October 2018 at 16:31:02 UTC, Stanislav Blinov 
wrote:
 So again,

 void atomicInc(shared int*); // is "not safe", but
 void struct_Atomic_int_opUnary_plus_plus(ref shared Atomic); // 
 is "safe"

 just because latter is a "method". And that, by you, is 
 hunky-dory? Whether it's a method or a free function, it's 
 written to work on *shared* data. Of course it wouldn't be safe 
 if you allow any non-shared data to become shared without the 
 programmer having a say in this.

Out of curiosity, when it comes to primitives, what could you do 
under MP in void "atomicInc(shared int*)" that would be 
problematic?

void atomicInc(shared int*) {
   // i.e. what goes here?
}

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 17:10:03 UTC, aliak wrote:

 Out of curiosity, when it comes to primitives, what could you 
 do under MP in void "atomicInc(shared int*)" that would be 
 problematic?

 void atomicInc(shared int*) {
   // i.e. what goes here?
 }

1. Anything if int* implicitly converts to shared int* (per MP), 
because then that function is indeed unsafe.
2. Only actual platform-specific implementation bugs otherwise, 
and these are beyond what `shared` can provide.

aliak <something something.com> writes:

On Thursday, 18 October 2018 at 17:23:36 UTC, Stanislav Blinov 
wrote:
 On Thursday, 18 October 2018 at 17:10:03 UTC, aliak wrote:

 Out of curiosity, when it comes to primitives, what could you 
 do under MP in void "atomicInc(shared int*)" that would be 
 problematic?

 void atomicInc(shared int*) {
   // i.e. what goes here?
 }

 1. Anything if int* implicitly converts to shared int* (per 
 MP), because then that function is indeed unsafe.

Right, but the argument is a shared int*, so from what I've 
understood... you can't do anything with it since it has no 
shared members. i.e. you can't read or write to it. No?

 2. Only actual platform-specific implementation bugs otherwise, 
 and these are beyond what `shared` can provide.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 18:05:51 UTC, aliak wrote:

 Right, but the argument is a shared int*, so from what I've 
 understood... you can't do anything with it since it has no 
 shared members. i.e. you can't read or write to it. No?

Obviously the implementation would cast `shared` away, just like 
it would if it were Atomic!int. But for some reason, Manu thinks 
that latter is OK doing that, but former is voodoo. Go figure.

aliak <something something.com> writes:

On Thursday, 18 October 2018 at 18:12:03 UTC, Stanislav Blinov 
wrote:
 On Thursday, 18 October 2018 at 18:05:51 UTC, aliak wrote:

 Right, but the argument is a shared int*, so from what I've 
 understood... you can't do anything with it since it has no 
 shared members. i.e. you can't read or write to it. No?

 Obviously the implementation would cast `shared` away, just 
 like it would if it were Atomic!int. But for some reason, Manu 
 thinks that latter is OK doing that, but former is voodoo. Go 
 figure.

Sounds like one is encapsulated within a box that carefully 
handles thread safety and makes promises with the API and the 
other is not.

I don't think you can apply shared on a free function, i.e.:

void increment(shared int*) shared;

in which case increment would not, and cannot be a threadsafe api 
in Manu's world.

So once you throw an Object in to shared land all you could do is 
call shared methods on it, and since they'd have been carefully 
written with sharing in mind... it does seem a lot more usable.

On these two cases:

increment(shared int* p1) {
  // I have no guarantees that protecting and accessing p1 will 
not cause problems
  //
  // but you don't have this guarantee in any world (current nor 
MP) because you can
  // never be sure that p1 was not cast from a mutable.
}

int* p2;
increment(p2);
// I have no guarantee that accessing p2 is safe anymore.
//
// But that would apply only if the author of increment was being 
unsafe.
// and "increment" cannot be marked as shared.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 21:51:52 UTC, aliak wrote:
 On Thursday, 18 October 2018 at 18:12:03 UTC, Stanislav Blinov 
 wrote:
 On Thursday, 18 October 2018 at 18:05:51 UTC, aliak wrote:

 Right, but the argument is a shared int*, so from what I've 
 understood... you can't do anything with it since it has no 
 shared members. i.e. you can't read or write to it. No?

 Obviously the implementation would cast `shared` away, just 
 like it would if it were Atomic!int. But for some reason, Manu 
 thinks that latter is OK doing that, but former is voodoo. Go 
 figure.

 Sounds like one is encapsulated within a box that carefully

Unit of "encapsulation" in D is either a module or a package, not 
a struct. Free functions are a very valid way of accessing 
"encapsulated" data.

 handles thread safety and makes promises with the API and the 
 other is not.

Nope.

void foo(const T* x);

makes a promise to not write through x. It assumes '*x' itself 
may not be const.

void foo(shared T* x);

makes a promise to threat '*x' in a thread-safe manner. But per 
MP, it *assumes* that '*x' is shared. And if it isn't, good luck 
finding that spot in your code.

 I don't think you can apply shared on a free function, i.e.:

 void increment(shared int*) shared;

 in which case increment would not, and cannot be a threadsafe 
 api in Manu's world.

Wrong. In Manu's "world", this is somehow considered "safe":

void T_method_increment(ref shared T);

...because that is what a method is, while this:

void increment(shared T*);
void increment(ref shared T);

...is considered "unsafe" because reasons. Do you see the 
difference in signatures? I sure don't.

 So once you throw an Object in to shared land all you could do 
 is call shared methods on it, and since they'd have been 
 carefully written with sharing in mind... it does seem a lot 
 more usable.

Same goes with free functions.

 On these two cases:

 increment(shared int* p1) {
  // I have no guarantees that protecting and accessing p1 will 
 not cause problems
  //
  // but you don't have this guarantee in any world (current nor 
 MP) because you can
  // never be sure that p1 was not cast from a mutable.
 }

Except that you *have to* *explicitly* cast it, which is:
a) documentation
b) greppable
c) easily fails review for people not authorized to do so

 int* p2;
 increment(p2);
 // I have no guarantee that accessing p2 is safe anymore.
 // But that would apply only if the author of increment was 
 being unsafe.
 // and "increment" cannot be marked as shared.

No. *You*, the caller of an API (the "increment"), do not 
necessarily control that API. By allowing implicit conversion you 
waive all claims on your own data. In Manu's world, "increment" 
*assumes* you're doing the right thing. Yet at the same time, 
Manu happily talks about how only "experts" can do the right 
thing. How these two things co-exist in his world, I have no idea.

The "have no guarantee" holds in both cases. Except case (1) 
would require actually checking what the hell you're doing before 
making a cast, while in case (2) you just blindly write unsafe 
code.

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Thursday, 18 October 2018 at 16:31:02 UTC, Stanislav Blinov 
wrote:
 You contradict yourself and don't even notice it. Per your 
 rules, the way to open that locked box is have shared methods 
 that access data via casting. Also per your rules, there is 
 absolutely no way for the programmer to control whether they're 
 actually sharing the data. Therefore, some API can steal a 
 shared reference without your approval, use that with your 
 "safe" shared methods, while you're continuing to threat your 
 data as not shared.

Yes, and that's fine. Because it's thread-safe, remember?


 You and Manu both seem to think that methods allow you to 
 "define a thread-safe interface".

 struct S {
     void foo() shared;
 }

 Per your rules, S.foo is thread-safe. It is here that I remind 
 you, *again*, what S.foo actually looks like, given made-up 
 easy-to-read mangling:

 void struct_S_foo(ref shared S);

 And yet, for some reason, you think that these are not 
 thread-safe:

 void foo(shared int*);
 void bar(ref shared int);

Again, no. No. No, no, no, no, no. We have not said that, we are 
not saying that, and we will not say that. Because it's not true, 
and I pointed out exactly this in a previous post. I have no idea 
where you got this idea, and I hope we can excise it. There is 
absolutely nothing wrong with void foo(shared int*). (apart from 
the fact it can't safely do anything)

For clarity: the interface of a type is any method, function, 
delegate or otherwise that may affect its internals. That means 
any free function in the same module, and any non-private members.


 Your implementation of 'twaddle' is *unsafe*, because the 
 compiler doesn't know that 'payload' is shared. For example, 
 when inlining, it may reorder the calls in it and cause races 
 or other UB. At least one of the reasons behind `shared` *was* 
 to serve as compiler barrier.

Ah, now this is a good point - thanks! That does seem like it's a 
harder problem than has come up thus far.


 Alright, so I have this shared object that I can't read from, 
 and can't write to. It has no public shared members. What can 
 I do with it? I can pass it to other guys, who also can't do 
 anything with it. Are there other options?

 It can have any number of public shared "members" per UFCS. The 
 fact that you forget is that there's no difference between a 
 method and a free function, other than syntax sugar. Well, OK, 
 there's guaranteed private access for methods, but same is true 
 for module members.

This again? See point 2, above. I hope we can stop this silliness 
soon.


 The rest just follows naturally.


 Nothing follows naturally. The proposal doesn't talk at all 
 about the fact that you can't have "methods" on primitives,

 You can't have thread-safe methods operating directly on 
 primitives, because they already present a non-thread-safe 
 interface. This is true. This follows naturally from the rules.

 Everything in D already presents a non-threadsafe interface. 
 Things that you advocate included.

 struct S {
     void foo() shared;
 }

 That is not threadsafe. This *sort of* is:

 struct S {
      disable this(this);
      disable void opAssign(S);

     void foo() shared;
 }

 ...except not quite still. Now, if the compiler generated above 
 in the presence of any `shared` members or methods, then we 
 could begin talking about it being threadsafe. But that part is 
 mysteriously missing from Manu's proposal, even though I keep 
 reminding of this in what feels like every third post or so 
 (I'm probably grossly exaggerating).

Again, this is good stuff. This is an actual example of what can 
go wrong. Thanks!


 that you can't distinguish between shared and unshared data 
 if that proposal is realized,


 And you can't do that currently either. Just like today, 
 shared(T) means the T may or may not be shared with other 
 thread. Nothing more, nothing less.

 I don't think it means what you think it means. "May or may not 
 be shared with other thread" means "you MUST treat it as if 
 it's shared with other thread". That's it.

Yup, hence 'shared' on a method meaning 'thread-safe'. So it's 
fine. I think this has been mentioned before.


 That's why automatic conversion doesn't make *any* sense, and 
 that's why compiler error on attempting to pass over mutable as 
 shared makes *perfect* sense.

No. Because shared access is thread-safe.


 that you absolutely destroy D's TLS-by-default treatment...

 I'm unsure what you mean by this.

 You lose the ability to distinguish thread-local and shared 
 data.

And when is this a problem? Again, anything that has shared 
access to something is incapable of doing anything 
non-thread-safe to it.


 Functions that you must not be allowed to write per this same 
 proposal. How quaint.

 What? Which functions can't I write?

 Uh-huh, only due to some weird convention that "methods" are 
 somehow safer than free functions. Which they're not.

No. Again, point 2. Nobody says this.


 Yup, this is correct. But wrap it in a struct, like e.g. 
 Atomic!int, and everything's hunky-dory.

 So again,

 void atomicInc(shared int*); // is "not safe", but
 void struct_Atomic_int_opUnary_plus_plus(ref shared Atomic); // 
 is "safe"

No, void atomicInc(shared int*) is perfectly safe, as long as it 
doesn't cast away shared. Again, the problem is int already has a 
non-thread-safe interface, which Atomic!int doesn't. And once 
more, for clarity, this interface includes any function that has 
access to its private members, free function, method, delegate 
return value from a function/method, what have you. Since D's 
unit of encapsulation is the module, this has to be the case. For 
int, the members of that interface include all operators. For 
pointers, it includes deref and pointer arithmetic. For arrays 
indexing, slicing, access to .ptr, etc. None of these lists are 
necessarily complete.


 I have no idea where I or Manu have said you can't make 
 functions that take shared(T)*.

 Because you keep saying they're unsafe and that you should wrap 
 them up in a struct for no other reason than just "because 
 methods are kosher".

Again, point 2. I think we have been remiss in the explanation of 
what we consider the interface.


 Let's say it together: for a type to be thread-safe, all of 
 its public members must be written in a thread-safe way.

 It's shared private parts also must be written in a thread-safe 
 way. Yes, they're private, but they still may be shared. 
 Welcome to the communism of multithreading.

For the public members to be thread-safe, yes, the private parts 
must be thread-safe. That's always the case. If I'm going to 
build a skyscraper, I will not make the foundation out of 
cardboard.


Now, Two very good points came up in this post, and I think it's 
worth stating them again, because they do present possible issues 
with MP:

1) How does MP deal with reorderings in non-shared methods?

I don't know. I'd hide behind 'that's for the type implementor to 
handle', but it's a subtle enough problem that I'm not happy with 
that answer.


2) What about default members like opAssign and postblit?

The obvious solution is for the compiler to not generate these 
when a type has a shared method or is taken as shared by a free 
function in the same module. I don't like the latter part of 
that, but it should work.

--
   Simen

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 22:08:14 UTC, Simen Kjærås wrote:
 On Thursday, 18 October 2018 at 16:31:02 UTC, Stanislav Blinov

 Now, if the compiler generated above in the presence of any 
 `shared` members or methods, then we could begin talking about 
 it being threadsafe...


 Again, this is good stuff. This is an actual example of what 
 can go wrong. Thanks!

You're welcome.

 No, void atomicInc(shared int*) is perfectly safe, as long as 
 it doesn't cast away shared.

Eh? If you can't read or write to your shared members, how *do* 
you implement your "safe" shared methods without casting away 
shared? Magic?!

 Again, the problem is int already has a non-thread-safe 
 interface, which Atomic!int doesn't.

*All* structs and classes have a non-thread-safe interface, as I 
have demonstrated, and thankfully you agree with that. But *there 
is no mention of it* in the OP, nor there was *any recognition* 
of it up to this point. When I asked about copying, assignment 
and destructors in the previous thread (thread, not post), Manu 
quite happily proclaimed those were fine given some arbitrary 
conditions. Yet those are quite obviously *not fine*, especially 
*if* you want to have a "safe" implicit conversion. *That* 
prompted me to assume that Manu didn't actually think long and 
hard about his proposal and what it implies. Without recognizing 
those issues:

struct S {
     private int x;
     void foo() shared;
}

void shareWithThread(shared S* s);

auto s = make!S;     // or new, whatever
shareWithThread(&s); // Manu's implicit conversion

// 10 kLOC below, written by some other guy 2 years later:
*s = S.init;

^ that is a *terrible*, and un-greppable BUG. How fast would you 
spot that in a review? Pretty fast if you saw or wrote the other 
code yesterday. A week later? A month? A year?..

Again, that is assuming *only* what I'm *certain about* in Manu's 
proposal, not something he or you assumed but didn't mention.

 And once more, for clarity, this interface includes any 
 function that has access to its private members, free function, 
 method, delegate return value from a function/method, what have 
 you. Since D's unit of encapsulation is the module, this has to 
 be the case. For int, the members of that interface include all 
 operators. For pointers, it includes deref and pointer 
 arithmetic. For arrays indexing, slicing, access to .ptr, etc. 
 None of these lists are necessarily complete.

Aight, now, *now* I can perhaps try to reason about this from 
your point of view. Still, I would need some experimentation to 
see if such approach could actually work. And that would mean 
digging out old non-`shared`-aware code and performing some... 
dubious... activities.

 I have no idea where I or Manu have said you can't make 
 functions that take shared(T)*.

Because that was the only way to reason about your 
interpretations of various examples until you said this:

 I think we have been remiss in the explanation of what we 
 consider the interface.
 For clarity: the interface of a type is any method, function, 
 delegate or otherwise that may affect its internals. That means 
 any free function in the same module, and any non-private 
 members.

Now compare that to what is stated in the OP and correlate with 
what I'm saying, you might understand where my opposition comes 
from.

 Now, Two very good points came up in this post, and I think 
 it's worth stating them again, because they do present possible 
 issues with MP:

 1) How does MP deal with reorderings in non-shared methods?

 I don't know. I'd hide behind 'that's for the type implementor 
 to handle', but it's a subtle enough problem that I'm not happy 
 with that answer.


 2) What about default members like opAssign and postblit?

 The obvious solution is for the compiler to not generate these 
 when a type has a shared method or is taken as shared by a free 
 function in the same module. I don't like the latter part of 
 that, but it should work.

Something I didn't yet stress about (I think only mentioned 
briefly somewhere) is, sigh, destructors. Right now, `shared` 
allows you to either have a `~this()` or a `~this() shared`, but 
not both. In my mind, `~this() shared` is an abomination. One 
should either:

1) have data that starts life shared (a global, or e.g. new 
shared(T)), and simply MUST NOT have a destructor. Such data is 
ownerless, or you can say that everybody owns it. Therefore 
there's no deterministic way of knowing whether or not or when to 
call the destructor. You can think of it as an analogy with 
current stance on finalizers with GC.

2) have data that starts life locally (e.g. it's not declared 
`shared`, but converted later). Such types MAY have a destructor, 
because they always have a cleanly defined owner: whoever holds 
the non-`shared` reference (recall that copying MUST be 
*disabled* for any shared-aware type). But that destructor MUST 
NOT be `shared`.

Consequently, types such as these:

shared struct S { /* ... */ }

MUST NOT define a destructor, either explicitly, or implicitly 
through members, i.e. it's a compile error if an __xdtor needs to 
be generated for such type.

However, in practice this would mean that practically all types 
couldn't have a destructor:

struct S {
     private shared X x; // X must not have a destructor, even 
though S can?..
}

Perhaps, an exception to (1) above could be made for such cases, 
but I'm too tired to think about wording at the moment.

Also, the proposal has no mention of interaction of `shared` and 
the GC, which AFAIK is also missing pretty much everywhere you 
can even get some information on current state of `shared`. This 
*needs* to be addressed.

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 3:10 PM Simen Kjærås via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Now, Two very good points came up in this post, and I think it's
 worth stating them again, because they do present possible issues
 with MP:

It is easy to respond to these.

 1) How does MP deal with reorderings in non-shared methods?

 I don't know. I'd hide behind 'that's for the type implementor to
 handle', but it's a subtle enough problem that I'm not happy with
 that answer.

This is a red-herring. Opaque function calls are never reordered.
If they are inlined, the compiler has full visibility to the internal
machinery present.

If you call one function that performs an atomic op, then another that
performs an atomic op, it is impossible for the CPU to reorder atomic
op's around eachother, that would defeat the entire point of hardware
atomic operations.
If the functions used mutexes, or semaphores, exactly the same
applies; they are either function calls (not reordered), or they use
hardware primitives, which have scheduling fences.

If none of those things are present in the threadsafe functions, then
the functions aren't threadsafe in the first place!

In short, he made up this issue, it doesn't exist.

 2) What about default members like opAssign and postblit?

 The obvious solution is for the compiler to not generate these
 when a type has a shared method or is taken as shared by a free
 function in the same module. I don't like the latter part of
 that, but it should work.

These aren't issues either. There's nothing wrong with atomic
assignment; you just have to implement an atomic assignment.
Postblit is being replaced with copy-ctor's and `shared` is one of the
explicit reasons why! Copy-ctor's are also fine, it would express an
atomic assignment.


This is just hot air, and only strengthen my conviction.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Friday, 19 October 2018 at 01:22:53 UTC, Manu wrote:
 On Thu, Oct 18, 2018 at 3:10 PM Simen Kjærås via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 Now, Two very good points came up in this post, and I think 
 it's worth stating them again, because they do present 
 possible issues with MP:

 It is easy to respond to these.

 1) How does MP deal with reorderings in non-shared methods?

 I don't know. I'd hide behind 'that's for the type implementor 
 to handle', but it's a subtle enough problem that I'm not 
 happy with that answer.

 This is a red-herring. Opaque function calls are never 
 reordered.
 If they are inlined, the compiler has full visibility to the 
 internal machinery present.

You don't say?.. And what, exactly, stops the optimizer from 
removing "unnecessary" reads or rearranging them with stores, 
given the original code, which, if you freaking read it, you'd 
see there's no indication that it's not allowed to do so.

 If you call one function that performs an atomic op, then 
 another that performs an atomic op, it is impossible for the 
 CPU to reorder atomic op's around eachother, that would defeat 
 the entire point of hardware atomic operations.

I'm not talking about CPU reordering at all. I'm talking about 
the optimizer.

 In short, he made up this issue, it doesn't exist.

Yeeees, of course I have. What else have I made up, can you tell? 
You know what doesn't exist though? Even one example of a useful 
implicit conversion form mutable to shared from you. Not even one.

 2) What about default members like opAssign and postblit?

 The obvious solution is for the compiler to not generate these 
 when a type has a shared method or is taken as shared by a 
 free function in the same module. I don't like the latter part 
 of that, but it should work.

 These aren't issues either. There's nothing wrong with atomic
 assignment; you just have to implement an atomic assignment.

You just haven't read the code. Those members aren't even 
`shared`. The *least* you can do is disable them *if* you're 
going to cast your variable to `shared`. Otherwise your 
"interface" remains non-threadsafe.

 Postblit is being replaced with copy-ctor's and `shared` is one 
 of the explicit reasons why! Copy-ctor's are also fine, it 
 would express an atomic assignment.

And this strengthens *my* belief that you haven't at all thought 
about this. There is literally *no* purpose for any `shared` 
types to have any copy-ctors. The only feasible copy primitives 
are from shared to local and from local to shared. Not to mention 
that again, to even talk about your "implicit" conversions, you 
must first think about what can happen to the *owned* 
(non-`shared`) reference after the conversion. Hint: you can't 
copy it. You can't assign *to it*. Not via default-generated 
postblits and opAssigns, which are not, and can not, be "atomic".
I'm fully aware about postblits being "replaced" by copy-ctors, 
I'm also fully aware how "much" thought was put into that wrt. 
`shared`.

 This is just hot air, and only strengthen my conviction.

You know what, I'm fed up with you too. Just show me one, *one* 
non-contrived example of useful implicit conversion from mutable 
to shared. So far you haven't produced *any at all*. Then we can 
talk about what is hot air here. Produce, or drop this 
presumptious crap.

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 6:50 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Friday, 19 October 2018 at 01:22:53 UTC, Manu wrote:
 On Thu, Oct 18, 2018 at 3:10 PM Simen Kjærås via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Now, Two very good points came up in this post, and I think
 it's worth stating them again, because they do present
 possible issues with MP:

 It is easy to respond to these.

 1) How does MP deal with reorderings in non-shared methods?

 I don't know. I'd hide behind 'that's for the type implementor
 to handle', but it's a subtle enough problem that I'm not
 happy with that answer.

 This is a red-herring. Opaque function calls are never
 reordered.
 If they are inlined, the compiler has full visibility to the
 internal machinery present.

 You don't say?.. And what, exactly, stops the optimizer from
 removing "unnecessary" reads or rearranging them with stores,
 given the original code, which, if you freaking read it, you'd
 see there's no indication that it's not allowed to do so.

 If you call one function that performs an atomic op, then
 another that performs an atomic op, it is impossible for the
 CPU to reorder atomic op's around eachother, that would defeat
 the entire point of hardware atomic operations.

 I'm not talking about CPU reordering at all. I'm talking about
 the optimizer.

 In short, he made up this issue, it doesn't exist.

 Yeeees, of course I have. What else have I made up, can you tell?
 You know what doesn't exist though? Even one example of a useful
 implicit conversion form mutable to shared from you. Not even one.

 2) What about default members like opAssign and postblit?

 The obvious solution is for the compiler to not generate these
 when a type has a shared method or is taken as shared by a
 free function in the same module. I don't like the latter part
 of that, but it should work.

 These aren't issues either. There's nothing wrong with atomic
 assignment; you just have to implement an atomic assignment.

 You just haven't read the code. Those members aren't even
 `shared`. The *least* you can do is disable them *if* you're
 going to cast your variable to `shared`. Otherwise your
 "interface" remains non-threadsafe.

 Postblit is being replaced with copy-ctor's and `shared` is one
 of the explicit reasons why! Copy-ctor's are also fine, it
 would express an atomic assignment.

 And this strengthens *my* belief that you haven't at all thought
 about this. There is literally *no* purpose for any `shared`
 types to have any copy-ctors. The only feasible copy primitives
 are from shared to local and from local to shared. Not to mention
 that again, to even talk about your "implicit" conversions, you
 must first think about what can happen to the *owned*
 (non-`shared`) reference after the conversion. Hint: you can't
 copy it. You can't assign *to it*. Not via default-generated
 postblits and opAssigns, which are not, and can not, be "atomic".
 I'm fully aware about postblits being "replaced" by copy-ctors,
 I'm also fully aware how "much" thought was put into that wrt.
 `shared`.

 This is just hot air, and only strengthen my conviction.

 You know what, I'm fed up with you too. Just show me one, *one*
 non-contrived example of useful implicit conversion from mutable
 to shared. So far you haven't produced *any at all*. Then we can
 talk about what is hot air here. Produce, or drop this
 presumptious crap.

You are an obscene person. I'm out.
You win.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Friday, 19 October 2018 at 01:53:00 UTC, Manu wrote:

 This is a red-herring.
 In short, he made up this issue, it doesn't exist.
 This is just hot air, and only strengthen my conviction.

 Produce, or drop this presumptious crap.


 You are an obscene person. I'm out.

Oooh, I'm soooorry, come baack!
Really though, what is it that you wanted to achieve here? You 
ask for counter-arguments, are given them on *17 pages already*, 
are asked numerous times to actually demonstrate the value of a 
small contained portion of your proposal, and all you do is shrug 
this all off just because you presume to "know better", and on 
top of that have the audacity to call someone else *obscene*? 
Wow... just... wow!

 You win.

I didn't know it was a contest.

Manu <turkeyman gmail.com> writes:

On Thu., 18 Oct. 2018, 7:10 pm Stanislav Blinov via Digitalmars-d, <
digitalmars-d puremagic.com> wrote:

 On Friday, 19 October 2018 at 01:53:00 UTC, Manu wrote:

 This is a red-herring.
 In short, he made up this issue, it doesn't exist.
 This is just hot air, and only strengthen my conviction.

 Produce, or drop this presumptious crap.


 You are an obscene person. I'm out.

 Oooh, I'm soooorry, come baack!
 Really though, what is it that you wanted to achieve here? You
 ask for counter-arguments, are given them on *17 pages already*,
 are asked numerous times to actually demonstrate the value of a
 small contained portion of your proposal, and all you do is shrug
 this all off just because you presume to "know better", and on
 top of that have the audacity to call someone else *obscene*?
 Wow... just... wow!

 You win.

 I didn't know it was a contest.

I've given use cases constantly, about taking object ownership, promotions,
and distribution for periods (think parallel for), I can achieve all my
goals with full  safety, absolutely no casts in user code, and I have
infrastructure in production that applies these patterns successfully. It's
worth pursuing.

I've spent years thinking on this, I'm trying to move the needle on this
issue for the first time in over a decade at least, and you have violently
opposed, in principle, from the very first post, and make no effort to
actually understand the proposition.

It's clearly a contest from your insurance that my proposal in worthless in
every single post you've made. You want me to admit defeat and desist.
Fuck you. You win. I don't have the time or energy to argue against a wall.

You are obscene, you're complete unproductive, and destructive from no
apparent reason.
I hope you continue to love shared, just the way it is... useless.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Friday, 19 October 2018 at 02:20:22 UTC, Manu wrote:

 I've given use cases constantly, about taking object ownership, 
 promotions, and distribution for periods (think parallel for),

Manu, you haven't shown *any* code in which conversion from 
mutable to shared, an *implicit* one at that, was even present, 
let alone useful. While at the same time blatantly dismissing 
*all* examples to the contrary as "bad code" or any other reason, 
especially the "you just don't understand". How do you expect us 
to understand if all you do is talk about how it's useful, but 
don't explain why?
You can talk and brag all you want, but until you do show at 
least one example, I'm sorry but you have no case. And what's 
this all about? A *small*, localized portion of your proposal, 
that isn't event the source of the problems you're talking about 
in the current state of affairs.
In all examples flying about in this thread, all of *required* 
casts were *the opposite*, from shared to mutable, and you 
yourself acknowledge that those are indeed required on the lowest 
level.
As I've said numerous times, from all that's been said and 
implied in this thread, the *only* use of implicit casting that 
comes to mind is avoiding writing some forwarding methods, that's 
about it. I honestly can't see how this is valuable to warrant 
such a drastic change in the language. Now maybe it's that I'm 
dumb, then just say so already and we'll all move on. Just don't 
act like you're the only one who knows it all and everyone else 
is an unwashed pleb clinging to worshiping the faces in the trees.

 I can achieve all my goals with full  safety, absolutely no 
 casts in user code, and I have infrastructure in production 
 that applies these patterns successfully. It's worth pursuing.

Okay... Then please do show *one* example of useful implicit 
conversion from mutable to shared.

 I've spent years thinking on this, I'm trying to move the 
 needle on this issue for the first time in over a decade at 
 least,

And you're behaving in this thread as if everyone else, myself 
included, were sitting on our thumbs counting flies for this same 
decade.

 and you have violently opposed, in principle, from the very 
 first post, and make no effort to actually understand the 
 proposition.

I haven't done such a thing. I have asked you, numerous times, 
one, and only one question, and you never so much as replied to 
*that question*. What is the big value of this implicit 
conversion that it would warrant changing current language rules 
regarding type conversions?

 It's clearly a contest from your insurance that my proposal in 
 worthless in every single post you've made. You want me to 
 admit defeat and desist.

And here you are, continuing to presume. Now you suddenly know 
what I want. Marvelous.

I *don't* want you to admit any defeat, *or* desist. I *want* you 
to succeed. I *want* to help make `shared` useful so that I and 
everyone else can actually start writing code with it, not in 
spite of it. I've agreed with you on pretty much everything in 
your proposal, *except one thing*. I want you to demonstrate the 
practical value of *that thing*, and it's benefits over the 
current state of affairs, and I asked you several times to 
explain to us mere unwashed mortals exactly how it's useful. What 
have we been doing for 17 pages? Discussing the benefits of 
disabling reads/writes on shared? No. Estimating how much 
existing code could go to trash, what parts of DRuntime/Phobos 
would need a rewrite? No. We were in constant back-and-forth of 
"But... nope... but... nope" about this implicit conversion, 
which you value so much yet for some reason fail to defend. 
Saying "I had good time with it" is not a very practical defense, 
not for a language construct anyway.

 Fuck you. You win. I don't have the time or energy to argue 
 against a wall.

If you ask to destroy, be prepared for a fight. Or don't ask. 
Just stop appealing to your own authority.

 You are obscene, you're complete unproductive, and destructive 
 from no apparent reason.

Give it all you've got, please. Let it all out all at once.

 I hope you continue to love shared, just the way it is... 
 useless.

Yet another presumption. Good on you.

Norm <norm.rowtree gmail.com> writes:

On Friday, 19 October 2018 at 02:20:22 UTC, Manu wrote:
 On Thu., 18 Oct. 2018, 7:10 pm Stanislav Blinov via 
 Digitalmars-d, < digitalmars-d puremagic.com> wrote:

 On Friday, 19 October 2018 at 01:53:00 UTC, Manu wrote:

 This is a red-herring.
 In short, he made up this issue, it doesn't exist.
 This is just hot air, and only strengthen my conviction.

 Produce, or drop this presumptious crap.


 You are an obscene person. I'm out.

 Oooh, I'm soooorry, come baack!
 Really though, what is it that you wanted to achieve here? You
 ask for counter-arguments, are given them on *17 pages 
 already*,
 are asked numerous times to actually demonstrate the value of a
 small contained portion of your proposal, and all you do is 
 shrug
 this all off just because you presume to "know better", and on
 top of that have the audacity to call someone else *obscene*?
 Wow... just... wow!

 You win.

 I didn't know it was a contest.

 I've given use cases constantly, about taking object ownership, 
 promotions, and distribution for periods (think parallel for), 
 I can achieve all my goals with full  safety, absolutely no 
 casts in user code, and I have infrastructure in production 
 that applies these patterns successfully. It's worth pursuing.

 I've spent years thinking on this, I'm trying to move the 
 needle on this issue for the first time in over a decade at 
 least, and you have violently opposed, in principle, from the 
 very first post, and make no effort to actually understand the 
 proposition.

 It's clearly a contest from your insurance that my proposal in 
 worthless in every single post you've made. You want me to 
 admit defeat and desist. Fuck you. You win. I don't have the 
 time or energy to argue against a wall.

 You are obscene, you're complete unproductive, and destructive 
 from no
 apparent reason.
 I hope you continue to love shared, just the way it is... 
 useless.

There's another way; Stanislav isn't one you need to convince so 
if that particular discussion is unproductive and disruptive just 
ignore it. I.e technical discussions should be robust but once 
they become personal just ignore that input and move on. Isn't 
always possible I know but in this case I reckon you can.

Convincing Walter, Andrei and the rest of the core dev team of 
course will require a DIP.

Keep going on this, it is the first hint of movement with shared 
since like foreva!

bye,
norm

bye,
Norm

rikki cattermole <rikki cattermole.co.nz> writes:

On 19/10/2018 7:09 PM, Norm wrote:
 There's another way; Stanislav isn't one you need to convince so if that 
 particular discussion is unproductive and disruptive just ignore it. I.e 
 technical discussions should be robust but once they become personal 
 just ignore that input and move on. Isn't always possible I know but in 
 this case I reckon you can.

Trust me, I think a few of us at least have already figured that out.

 Convincing Walter, Andrei and the rest of the core dev team of course 
 will require a DIP.
 
 Keep going on this, it is the first hint of movement with shared since 
 like foreva!

As long as it doesn't look like my idea[0] (Andrei doesn't like it, I 
may have asked) it should have some sort of legs.

[0] https://github.com/rikkimax/DIPs/blob/shared/DIPs/DIP1xxx-RC2.md

Dominikus Dittes Scherkl <dominikus.scherkl continental-corporation.com> writes:

On Friday, 19 October 2018 at 06:25:00 UTC, rikki cattermole 
wrote:
 On 19/10/2018 7:09 PM, Norm wrote:

 [0] 
 https://github.com/rikkimax/DIPs/blob/shared/DIPs/DIP1xxx-RC2.md

This document provide no reasoning about what usecases it 
supports:

Is it possible to create objects that are shared just for short 
periods during their livetime and guarantee that they can be used 
threadsave like Manu want it to be?

Does it prohibit misuse any better than Manus proposal (that 
requires the "Expert" to implement all theadsave API)?

Is the "normal" User still enforced to do some unsave casts?

Has the normal User to have high knownledge of how a threadsave 
API is to be used or can the compiler provide any guarantees that 
using them can only fail if the implementation behind the API has 
bugs (e.g. provide some encapsulation)?

Or any other usecases why and how this design is better than what 
we have now?

And also some ideas how to implement some useacases (examples) 
are completely missing.

Manu <turkeyman gmail.com> writes:

On Fri., 19 Oct. 2018, 6:10 am Dominikus Dittes Scherkl via Digitalmars-d, <
digitalmars-d puremagic.com> wrote:

 On Friday, 19 October 2018 at 06:25:00 UTC, rikki cattermole
 wrote:
 On 19/10/2018 7:09 PM, Norm wrote:

 [0]
 https://github.com/rikkimax/DIPs/blob/shared/DIPs/DIP1xxx-RC2.md

 This document provide no reasoning about what usecases it
 supports:

 Is it possible to create objects that are shared just for short
 periods during their livetime and guarantee that they can be used
 threadsave like Manu want it to be?

 Does it prohibit misuse any better than Manus proposal (that
 requires the "Expert" to implement all theadsave API)?

No, a key misunderstanding. My proposal is  safe. The only thing an expert
must do is write the few  trusted implementations that live at the very
bottom of the stack.
That would always be in a lib. When was the last time you rewrote std::map
because you thought you could do better?

The whole stack from there on up (the user stack) is safe, and you can have
confidence in the  safe-ty.
My goal is to make it safe, clearly communicate how a user interact with
the API, and mechanically confirm that users do the right stuff.
My proposal is specifically structured to not require *any* unsafe
interactions at the user level. Only core machinery that is  trusted needs
expert attention.

I don't think it's possible to invent a proposal with a higher degree of
verifiable safety.

rikki cattermole <rikki cattermole.co.nz> writes:

On 20/10/2018 2:07 AM, Dominikus Dittes Scherkl wrote:
 This document provide no reasoning about what usecases it supports:

It was a basic idea of mine... It was never meant to be PR'd.

Joakim <dlang joakim.fea.st> writes:

On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
 On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
 On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:

 I've said this a bunch of times, there are 2 rules:
 1. shared inhibits read and write access to members
 2. `shared` methods must be threadsafe

From there, shared becomes interesting and useful.


 Oh God...

 void atomicInc(shared int* i) { /* ... */ }

 Now what? There are no "methods" for ints, only UFCS. Those 
 functions can be as safe as you like, but if you allow 
 implicit promotion of int* to shared int*, you *allow 
 implicit races*.

 This function is effectively an intrinsic. It's unsafe by 
 definition.

 Only if implicit conversion is allowed. If it isn't, that's 
 likely  trusted, and this:

 void atomicInc(ref shared int);

 can even be  safe.

 In this case, with respect to the context (a single int) 
 atomicInc()
 is ALWAYS safe, even with implicit conversion. You can 
 atomicInc() a
 thread-local int perfectly safely.

 It's a tool for implementing threadsafe machinery.
 No user can just start doing atomic operations on random ints
 and say
 "it's threadsafe", you must encapsulate the threadsafe
 functionality
 into some sort of object that aggregates all concerns and
 presents an
 intellectually sound api.

 Threadsafety starts and ends with the programmer. By your 
 logic *all* functions operating on `shared` are unsafe then. 
 As far as compiler is concerned, there would be no difference 
 between these two:

 struct S {}
 void atomicInc(ref shared S);

 and

 struct S { void atomicInc() shared { /* ... */ } }

 The signatures of those two functions are exactly the same. 
 How is that different from a function taking a shared int 
 pointer or reference?

 It's not, atomicInc() of an int is always safe with respect to 
 the int itself.
 You can call atomicInc() on an unshared int and it's perfectly 
 fine,
 but now you need to consider context, and that's a problem for 
 the
 design of the higher-level scope.

 To maintain thread-safety, the int in question must be 
 appropriately contained.

 The problem is that the same as the example I presented before, 
 which I'll repeat:

 struct InvalidProgram
 {
   int x;
   void fun() { ++x; }
   void gun() shared { atomicInc(&x); }
 }

 The method gun() (and therefore the whole object) is NOT 
 threadsafe by
 my definition, because fun() violates the threadsafety of gun().
 The situation applies equally here that:
 int x;
 atomicInc(&x);
 ++x; // <- by my definition, this 'API' (increment an int) 
 violates
 the threadsafety of atomicInc(), and atomicInc() is therefore 
 not
 threadsafe.

 `int` doesn't present a threadsafe API, so int is by 
 definition, NOT threadsafe. atomicInc() should be  system, and 
 not  trusted.

 If you intend to share an int, use Atomic!int, because it has a 
 threadsafe API.
 atomicInc(shared int*) is effectively just an unsafe intrinsic, 
 and
 its only use is at ground-level implementation of threadsafe
 machinery, like malloc() and free().

 Let me try one:

 void free(void*) { ... }

 Now what? I might have dangling pointers... it's a 
 catastrophe!

 One could argue that it should be void free(ref void* p) { /* 
 ...
 */ p = null; }

 void *p2 = p;
 free(p);
 p2.crash();

 As a matter of fact, in my own allocators memory blocks 
 allocated by them are passed by value and are non-copyable, 
 they're not just void[] as in std.experimental.allocator. One 
 must 'move' them to pass ownership, and that includes 
 deallocation. But that's another story altogether.

 Right, now you're talking about move semantics to implement 
 transfer of ownership... you might recall I was arguing this 
 exact case to express transferring ownership of objects between 
 threads earlier. This talk of blunt casts and "making sure 
 everything is good" is all just great, but it doesn't mean 
 anything interesting with respect to `shared`. It should be 
 interesting even without unsafe casts.

 It's essentially the same argument.
 This isn't a function that professes to do something that
 people might
 misunderstand and try to use in an unsafe way, it's a 
 low-level
 implementation device, which is used to build larger *useful*
 constructs.

 You're missing the point, again. You have an int. You pass a 
 pointer to it to some API that takes an int*. You continue to 
 use your int as just an int.

 You have written an invalid program. I can think of an infinite 
 number
 of ways to write an invalid program.
 In this case, don't have an `int`, instead, have an Atomic!int; 
 you
 now guarantee appropriate access, problem solved!
 If you do have an int, don't pass it to other threads at random 
 when
 you don't have any idea what they intend to do with it! That's 
 basic
 common sense. You don't pass a pointer to a function if you 
 don't know
 what it does with the pointer!

 The API changes, and now the function
 you called previously takes a shared int*. Implicit conversion
 works, everything compiles, you have a race. Now, that's of
 course an extremely stupid scenario.

 Yes.

 The point is: the caller of
 some API *must* assert that they indeed pass shared data. It's
 insufficient for the API alone to "promise" taking shared data.
 That's the difference with promotion to `const`.

 The caller doesn't care if it's true that the callee can't do 
 anything
 with it that's unsafe anyway.
 We effect that state by removing all non-threadsafe access.

Rather than answering all objections in this thread in detail, a 
mistake I've made before too, I suggest you put up a 3-5 page 
document somewhere explaining your proposal in detail. You can 
use the feedback here as a guide on what to explain more and what 
not to. Be sure to include code samples of how you see everything 
ultimately working in your external document.

Two mistakes you may be making in writing responses in this 
thread, which I've made before too:

1. Assuming people read anything more than fragments of what 
you're writing. My experience suggests people just read bits and 
pieces quickly, then compose them together in the way that makes 
most sense to them based on how _past_ technology works.

2. Assuming people have any idea how the underlying 
implementation of shared and multi-threading works. I know little 
to nothing about how shared systems work and how that interacts 
with type systems, which is why I haven't responded to your 
proposal, but that doesn't stop others from responding who may 
not know much more.

Neither of these will be solved by writing an external document, 
but at least Walter and others more seriously interested will 
benefit from a better-motivated explanation with more detail. I 
won't read it, ;) but I think we can avoid another long thread 
like this if you go this route. I'm planning the same approach 
with the technical topic I raised before, along with working code.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/17/18 2:46 PM, Manu wrote:
 On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via

 What the example demonstrates is that while you are trying to disallow
 implicit casting of a shared pointer to an unshared pointer, you have
 inadvertently allowed it by leaving behind an unshared pointer that is
 the same thing.

 
 This doesn't make sense... you're showing a thread-local program.
 The thread owning the unshared pointer is entitled to the unshared
 pointer. It can make as many copies at it likes. They are all
 thread-local.

It's assumed that shared int pointer can be passed to another thread, 
right? Do I have to write a full program to demonstrate?

 There's only one owning thread, and you can't violate that without unsafe
casts.

The what is the point of shared? Like why would you share data that 
NOBODY CAN USE?

At SOME POINT, shared data needs to be readable and writable. Any 
correct system is going to dictate how that works. It's a good start to 
make shared data unusable unless you cast. But then to make it 
implicitly castable from unshared defeats the whole purpose.

 In order for a datum to be
 safely shared, it must be accessed with synchronization or atomics by
 ALL parties.

 
 ** Absolutely **
 
 If you have one party that can simply change it without
 those, you will get races.

 
 *** THIS IS NOT WHAT I'M PROPOSING ***
 
 I've explained it a few times now, but people aren't reading what I
 actually write, and just assume based on what shared already does that
 they know what I'm suggesting.
 You need to eject all presumptions from your mind, take the rules I
 offer as verbatim, and do thought experiments from there.

What seems to be a mystery here is how one is to actually manipulate 
shared data. If it's not usable as shared data, how does one use it?

 
 That's why shared/unshared is more akin to mutable/immutable than
 mutable/const.

 
 Only if you misrepresent my suggestion.

It's not misrepresentation, I'm trying to fill in the holes with the 
only logical possibilities I can think of.

 
 It's true that only one thread will have thread-local access. It's not
 valid any more than having one mutable alias to immutable data.

 
 And this is why the immutable analogy is invalid. It's like const.
 shared offers restricted access (like const), not a different class of
 thing.

No, not at all. Somehow one must manipulate shared data. If shared data 
cannot be read or written, there is no reason to share it.

So LOGICALLY, we have to assume, yes there actually IS a way to 
manipulate shared data through these very carefully constructed and 
guarded things.

 There is one thread with thread-local access, and many threads with
 shared access.
 
 If a shared (threadsafe) method can be defeated by threadlocal access,
 then it's **not threadsafe**, and the program is invalid.
 
 struct NotThreadsafe
 {
    int x;
    void local()
    {
      ++x; // <- invalidates the method below, you violate the other
 function's `shared` promise
    }
    void notThreadsafe() shared
    {
      atomicIncrement(&x);
    }
 }

So the above program is invalid. Is it compilable with your added 
allowance of implicit casting to shared? If it's not compilable, why 
not? If it is compilable, how in the hell does your proposal help 
anything? I get the exact behavior today without any changes (except 
today, I need to explicitly cast, which puts the onus on me).

 
 struct Atomic(T)
 {
    void opUnary(string op : "++")() shared { atomicIncrement(&val); }
    private T val;
 }
 struct Threadsafe
 {
    Atomic!int x;
    void local()
    {
      ++x;
    }
    void threadsafe() shared
    {
      ++x;
    }
 }
 
 Naturally, local() is redundant, and it's perfectly fine for a
 thread-local to call threadsafe() via implicit conversion.

In this case, yes. But that's not because of anything the compiler can 
prove.

How does Atomic work? I thought shared data was not usable? I'm being 
pedantic because every time I say "well at some point you must be able 
to modify things", you explode.

Complete the sentence: "In order to read or write shared data, you have 
to ..."

 
 Here's another one, where only a subset of the object is modeled to be
 threadsafe (this is particularly interesting to me):
 
 struct Threadsafe
 {
    int x;
    Atomic!int y;
 
    void notThreadsafe()
    {
      ++x;
      ++y;
    }
    void threadsafe() shared
    {
      ++y;
    }
 }
 
 In these examples, the thread-local function *does not* undermine the
 threadsafety of threadsafe(), it MUST NOT undermine the threadsafety
 of threadsafe(), or else threadsafe() **IS NOT THREADSAFE**.
 In the second example, you can see how it's possible and useful to do
 thread-local work without invalidating the objects threadsafety
 commitments.
 
 
 I've said this a bunch of times, there are 2 rules:
 1. shared inhibits read and write access to members
 2. `shared` methods must be threadsafe
 
From there, shared becomes interesting and useful.

 

Given rule 1, how does Atomic!int actually work, if it can't read or 
write shared members?
For rule 2, how does the compiler actually prove this?

Any programming by convention, we can do today. We can implement 
Atomic!int with the current compiler, using unsafe casts inside  trusted 
blocks.

-Steve

Manu <turkeyman gmail.com> writes:

On Wed, Oct 17, 2018 at 12:35 PM Steven Schveighoffer via
Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/17/18 2:46 PM, Manu wrote:
 On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via

 What the example demonstrates is that while you are trying to disallow
 implicit casting of a shared pointer to an unshared pointer, you have
 inadvertently allowed it by leaving behind an unshared pointer that is
 the same thing.

 This doesn't make sense... you're showing a thread-local program.
 The thread owning the unshared pointer is entitled to the unshared
 pointer. It can make as many copies at it likes. They are all
 thread-local.

 It's assumed that shared int pointer can be passed to another thread,
 right? Do I have to write a full program to demonstrate?

And that shared(int)* provides no access. No other thread with that
pointer can do anything with it.

 There's only one owning thread, and you can't violate that without unsafe
casts.

 The what is the point of shared? Like why would you share data that
 NOBODY CAN USE?

You can call shared methods. They promise threadsafety.
That's a small subset of the program, but that's natural; only a very
small subset of the program is safe to be called from a shared
context.

In addition, traditional unsafe interactions which may involve
acquiring locks and doing casts remain exactly the same, and the exact
same design patterns must apply which assure that the object is
handled correctly.
I'm not suggesting any changes that affect that workflow.

 At SOME POINT, shared data needs to be readable and writable. Any
 correct system is going to dictate how that works. It's a good start to
 make shared data unusable unless you cast. But then to make it
 implicitly castable from unshared defeats the whole purpose.

No. No casting! This is antiquated workflow.. I'm not trying to take
it away from you, but it's not an interesting model for the future.
`shared` can model more than just that.
You can call threadsafe methods. Shared methods explicitly dictate how
the system works, and in a very clear and obvious/intuitive way.

The implicit cast makes using threadsafe objects more convenient when
you only have one, which is extremely common.

 In order for a datum to be
 safely shared, it must be accessed with synchronization or atomics by
 ALL parties.

 ** Absolutely **

 If you have one party that can simply change it without
 those, you will get races.

 *** THIS IS NOT WHAT I'M PROPOSING ***

 I've explained it a few times now, but people aren't reading what I
 actually write, and just assume based on what shared already does that
 they know what I'm suggesting.
 You need to eject all presumptions from your mind, take the rules I
 offer as verbatim, and do thought experiments from there.

 What seems to be a mystery here is how one is to actually manipulate
 shared data. If it's not usable as shared data, how does one use it?

Call shared methods. It's not like I haven't been saying this in every
post since my OP.
The only possible thing that you can safely do with a shared object is
call a method that has been carefully designed for thread-safe
calling. Any other access is invalid under any circumstance.

 It's true that only one thread will have thread-local access. It's not
 valid any more than having one mutable alias to immutable data.

 And this is why the immutable analogy is invalid. It's like const.
 shared offers restricted access (like const), not a different class of
 thing.

 No, not at all. Somehow one must manipulate shared data. If shared data
 cannot be read or written, there is no reason to share it.

Call shared methods.

 So LOGICALLY, we have to assume, yes there actually IS a way to
 manipulate shared data through these very carefully constructed and
 guarded things.

Yes, call the methods, they were carefully constructed to be threadsafe.
Only functions that have made a promise to be threadsafe and
implemented that complexity are valid interactions.

 There is one thread with thread-local access, and many threads with
 shared access.

 If a shared (threadsafe) method can be defeated by threadlocal access,
 then it's **not threadsafe**, and the program is invalid.

 struct NotThreadsafe
 {
    int x;
    void local()
    {
      ++x; // <- invalidates the method below, you violate the other
 function's `shared` promise
    }
    void notThreadsafe() shared
    {
      atomicIncrement(&x);
    }
 }

 So the above program is invalid. Is it compilable with your added
 allowance of implicit casting to shared? If it's not compilable, why
 not?

All my examples assume my implicit conversion rule. But regardless,
under my proposal, the above program is invalid. This violates the
threadsafe promise.

 If it is compilable, how in the hell does your proposal help
 anything? I get the exact behavior today without any changes (except
 today, I need to explicitly cast, which puts the onus on me).

My proposal doesn't help this program, it's invalid. I'm just
demonstrating what an invalid program looks like.

 struct Atomic(T)
 {
    void opUnary(string op : "++")() shared { atomicIncrement(&val); }
    private T val;
 }
 struct Threadsafe
 {
    Atomic!int x;
    void local()
    {
      ++x;
    }
    void threadsafe() shared
    {
      ++x;
    }
 }

 Naturally, local() is redundant, and it's perfectly fine for a
 thread-local to call threadsafe() via implicit conversion.

 In this case, yes. But that's not because of anything the compiler can
 prove.

The compiler can't 'prove' anything at all related to threadsafety. We
need to make one assumption; that `shared` methods are expected to be
threadsafe, and from there the compiler can prove correctness with
respect to that assumption.
In this case, `Atomic.opUnary("++")` promises that it's threadsafe,
and as such, the aggregate can safely use that tool to implement
higher-level logic.

 How does Atomic work? I thought shared data was not usable? I'm being
 pedantic because every time I say "well at some point you must be able
 to modify things", you explode.

Atomic implements a safe utility using unsafe primitives (atomic
increment intrinsic).
Atomic wraps the unsafe call to an intrinsic into a box that's safe,
and can be used by clients.
In my worldview, atomic is at the bottom of the chain-of-trust. It's
effectively a  trusted implementation of a foundational tool.
Almost every low level tool is of this nature.

 Complete the sentence: "In order to read or write shared data, you have
 to ..."

Call a shared method, or at the bottom of the stack, you need to do
unsafe ( trusted?) implementations of the foundational machinery
(possibly using casts), and package into boxes that are safe to
interact with and build out from.

 Here's another one, where only a subset of the object is modeled to be
 threadsafe (this is particularly interesting to me):

 struct Threadsafe
 {
    int x;
    Atomic!int y;

    void notThreadsafe()
    {
      ++x;
      ++y;
    }
    void threadsafe() shared
    {
      ++y;
    }
 }

 In these examples, the thread-local function *does not* undermine the
 threadsafety of threadsafe(), it MUST NOT undermine the threadsafety
 of threadsafe(), or else threadsafe() **IS NOT THREADSAFE**.
 In the second example, you can see how it's possible and useful to do
 thread-local work without invalidating the objects threadsafety
 commitments.


 I've said this a bunch of times, there are 2 rules:
 1. shared inhibits read and write access to members
 2. `shared` methods must be threadsafe

From there, shared becomes interesting and useful.


 Given rule 1, how does Atomic!int actually work, if it can't read or
 write shared members?

It's an intrinsic. You use it the same as malloc() or free(). It's a
piece of low-level mechanical tooling which you use in an unsafe way,
but you then wrap it in a layer that introduces the type-safety.
malloc() returns a void*, which you cast to the intended type, and
then perform construction.
You can't implement a typesafe new without malloc() at the bottom of the stack.

You need to raise your vision one-level higher to users of Atomic to
see interesting interactions.

 For rule 2, how does the compiler actually prove this?

 Any programming by convention, we can do today. We can implement
 Atomic!int with the current compiler, using unsafe casts inside  trusted
 blocks.

It can't. I don't know what can be done to mechanically enforce this
requirement, but I would suggest that it's a goal to work towards in
the future with any technology possible.
In the meantime though, if we accept that the user writing a
threadsafe tool is responsible for delivering on their promise, then
the system that emerges is widely useful. The higher-level becomes
generally interesting, the low-level will remain to be implemented by
experts, and is no change from the situation today.
The low level doesn't really care much about type-safety, it's the
high-level I'm interested in. We can tell a MUCH better story about
how users can interact with shared machinery, and we should.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/17/18 6:37 PM, Manu wrote:
 On Wed, Oct 17, 2018 at 12:35 PM Steven Schveighoffer via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/17/18 2:46 PM, Manu wrote:
 On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via

 What the example demonstrates is that while you are trying to disallow
 implicit casting of a shared pointer to an unshared pointer, you have
 inadvertently allowed it by leaving behind an unshared pointer that is
 the same thing.

 This doesn't make sense... you're showing a thread-local program.
 The thread owning the unshared pointer is entitled to the unshared
 pointer. It can make as many copies at it likes. They are all
 thread-local.

 It's assumed that shared int pointer can be passed to another thread,
 right? Do I have to write a full program to demonstrate?

 
 And that shared(int)* provides no access. No other thread with that
 pointer can do anything with it.

So then it's a misnomer -- it's not really shared, because I can't do 
anything with it.

 
 There's only one owning thread, and you can't violate that without unsafe
casts.

 The what is the point of shared? Like why would you share data that
 NOBODY CAN USE?

 
 You can call shared methods. They promise threadsafety.
 That's a small subset of the program, but that's natural; only a very
 small subset of the program is safe to be called from a shared
 context.

All I can see is that a shared method promises to be callable on shared 
or unshared data. In essence, it promises nothing.

It's the programmer who must implement the thread safety, and there 
really is no help at all from the compiler for this. At some level, 
there will be either casts, or intrinsics, both of which are unsafe 
without knowing all the context of the object. In any case, it's simply 
a false guarantee of thread safety, which might as well be a convention 
of "any function which starts with TS_ is supposed to be thread safe".

shared in the current form promises one thing and one thing only -- data 
marked as shared is actually sharable between threads, and data not 
marked as shared is actually not shared between threads. This new regime 
you are proposing does nothing extra or new, except break that guarantee.

 At SOME POINT, shared data needs to be readable and writable. Any
 correct system is going to dictate how that works. It's a good start to
 make shared data unusable unless you cast. But then to make it
 implicitly castable from unshared defeats the whole purpose.

 
 No. No casting! This is antiquated workflow.. I'm not trying to take
 it away from you, but it's not an interesting model for the future.
 `shared` can model more than just that.
 You can call threadsafe methods. Shared methods explicitly dictate how
 the system works, and in a very clear and obvious/intuitive way.
 
 The implicit cast makes using threadsafe objects more convenient when
 you only have one, which is extremely common.

The implicit cast means that you have to look at more than just your 
method. You have to look at the entire module, and figure out all the 
interactions, to see if the thread safe method actually is thread safe. 
That's programming by convention, and fully trusting the programmer.

I don't think this thread is going anywhere, so I'll just have to wait 
and see if someone else can explain it better. I'm a firm no on implicit 
casting from mutable to shared.

-Steve

Manu <turkeyman gmail.com> writes:

On Wed, Oct 17, 2018 at 6:50 PM Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/17/18 6:37 PM, Manu wrote:
 On Wed, Oct 17, 2018 at 12:35 PM Steven Schveighoffer via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/17/18 2:46 PM, Manu wrote:
 On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via

 What the example demonstrates is that while you are trying to disallow
 implicit casting of a shared pointer to an unshared pointer, you have
 inadvertently allowed it by leaving behind an unshared pointer that is
 the same thing.

 This doesn't make sense... you're showing a thread-local program.
 The thread owning the unshared pointer is entitled to the unshared
 pointer. It can make as many copies at it likes. They are all
 thread-local.

 It's assumed that shared int pointer can be passed to another thread,
 right? Do I have to write a full program to demonstrate?

 And that shared(int)* provides no access. No other thread with that
 pointer can do anything with it.

 So then it's a misnomer -- it's not really shared, because I can't do
 anything with it.

**EXACTLY**.. this is the key to the entire design that allows for the
implicit promotion.

 There's only one owning thread, and you can't violate that without unsafe
casts.

 The what is the point of shared? Like why would you share data that
 NOBODY CAN USE?

 You can call shared methods. They promise threadsafety.
 That's a small subset of the program, but that's natural; only a very
 small subset of the program is safe to be called from a shared
 context.

 All I can see is that a shared method promises to be callable on shared
 or unshared data. In essence, it promises nothing.

How is that nothing? It promises threadsafety, such that it can be
called on shared data. Normally, you can do NOTHING with shared data
short of unsafe operations.
It is always safe to call a threadsafe function on an unshared thing,
so it should be possible.
`shared` opts into a strong commitment to threadsafety that allows for
a shared thing to interact with it, otherwise, no interaction is
possible.

I don't understand how you can claim it promises nothing; this is an
extremely strong promise!

 It's the programmer who must implement the thread safety, and there
 really is no help at all from the compiler for this.

I hope we can work on this moving forwards. There are probably some
things we can do... but even without compiler support, in reality,
there are probably only a small number of core pieces of tooling, but
most authors of a shared thing will just aggregate core tools and call
through to their API's.
Most shared code will use tools written by experts, and implement
nothing fancy or magical themselves.

 At some level,
 there will be either casts, or intrinsics, both of which are unsafe
 without knowing all the context of the object.

This will tend to be in the core tooling, written by an expert.
The whole point of my design is to remove friction from the USERS of
those tools, such that they can create aggregate functionality safely
without doing anything unsafe.

 In any case, it's simply
 a false guarantee of thread safety, which might as well be a convention
 of "any function which starts with TS_ is supposed to be thread safe".

What makes it a false guarantee? It allows you to have confidence in
the stack built on top of some core tools.
The whole point here is to minimise the number of people writing code
like that. I'm trying to make shared generally useful so that people
don't have to engage with it at a low-level.

If you see an unsafe case in a shared function (beyond the core
tooling), then you should immediately be suspicious, and this is a
deliberate part of my goal here. The point here is to produce safer
and more reliable code by defining access rules that lead to confident
threadsafety.

What you describe is what we have now.

 shared in the current form promises one thing and one thing only -- data
 marked as shared is actually sharable between threads, and data not
 marked as shared is actually not shared between threads.

Ummm. The current form might *say* that, but you still have full
unregulated access to all members, and it's completely unsafe, with no
attempt to mitigate that.
This is a new definition. Take it for what it's worth. The old
definition is mostly worthless.
I'm trying to make something that's interesting and useful.

 This new regime you are proposing does nothing extra or new, except break that
guarantee.

There are heaps of advantages:
1. If I have a shared thing, I know I can't ruin its state by
manipulating it arbitrarily like I can now
2. I am now able to describe threadsafe objects
3. I relieve users from mental strain of trying to understand how to
correctly and safely interact with shared API's, because they can have
confidence that a shared thing can only perform threadsafe activity
4. I'm trying to give shared a simple, meaningful and *useful*
definition, that's easy to understand and communicate. I believe the
only reason people are having trouble, because they are taking it in
contrast, rather than as it is.

 At SOME POINT, shared data needs to be readable and writable. Any
 correct system is going to dictate how that works. It's a good start to
 make shared data unusable unless you cast. But then to make it
 implicitly castable from unshared defeats the whole purpose.

 No. No casting! This is antiquated workflow.. I'm not trying to take
 it away from you, but it's not an interesting model for the future.
 `shared` can model more than just that.
 You can call threadsafe methods. Shared methods explicitly dictate how
 the system works, and in a very clear and obvious/intuitive way.

 The implicit cast makes using threadsafe objects more convenient when
 you only have one, which is extremely common.

 The implicit cast means that you have to look at more than just your
 method. You have to look at the entire module, and figure out all the
 interactions, to see if the thread safe method actually is thread safe.
 That's programming by convention, and fully trusting the programmer.

I don't understand... how can the outer context affect the
threadsafety of a properly encapsulated thing?
Back to this trivial example:

struct Atomic(T)
{
  void opUnary(string op : "++")() shared { atomicInc(cast(T*)&val); }
  private T val;
}

This is properly encapsulated... the promise is firm.
If some threadsafe machinery is sufficiently complex that it has a
wide roam and you can't reason about whether the method is actually
threadsafe, then you have no business writing threadsafe machinery;
use a library.
This design, nor the current design, nor any other design can possibly help you.

 I don't think this thread is going anywhere, so I'll just have to wait
 and see if someone else can explain it better. I'm a firm no on implicit
 casting from mutable to shared.

You need to take it for an intellectual spin. Show me how it's corrupt
rather than just presenting discomfort with the idea in theory.
You're addicted to some concepts that you've carried around for a long
time. There is no value in requiring casts, they're just a funky
smell, and force the user to perform potentially unsafe manual
conversions, or interactions that they don't understand.

Implicit conversion is for allowing/encouraging what is safe. Calling
threadsafe functions is safe. You should be able to call threadsafe
functions.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/17/18 10:26 PM, Manu wrote:
 On Wed, Oct 17, 2018 at 6:50 PM Steven Schveighoffer via Digitalmars-d
 The implicit cast means that you have to look at more than just your
 method. You have to look at the entire module, and figure out all the
 interactions, to see if the thread safe method actually is thread safe.
 That's programming by convention, and fully trusting the programmer.

 
 I don't understand... how can the outer context affect the
 threadsafety of a properly encapsulated thing?

[snip]

 You need to take it for an intellectual spin. Show me how it's corrupt
 rather than just presenting discomfort with the idea in theory.
 You're addicted to some concepts that you've carried around for a long
 time. There is no value in requiring casts, they're just a funky
 smell, and force the user to perform potentially unsafe manual
 conversions, or interactions that they don't understand.

For example (your example):

struct NotThreadsafe
{
   private int x;
   void local()
   {
     ++x; // <- invalidates the method below, you violate the other
function's `shared` promise
   }
   void notThreadsafe() shared
   {
     atomicIncrement(&x);
   }
}

First, note the comment. I can't look ONLY at the implementation of 
"notThreadSafe" (assuming the function name is less of a giveaway) in 
order to guarantee that it's actually thread safe. I have to look at the 
WHOLE MODULE. Anything could potentially do what local() does. I added 
private to x to at least give the appearance of thread safety.

But on top of that, if I can't implicitly cast mutable to shared, then 
this ACTUALLY IS thread safe, as long as all the casting in the module 
is sound (easy to search and verify), and hopefully all the casting is 
encapsulated in primitives like you have written. Because someone on the 
outside would have to cast a mutable item into a shared item, and this 
puts the responsibility on them to make sure it works.

I'm ALL FOR having shared be completely unusable as-is unless you cast 
(thanks for confirming what I suspected in your last post). It's the 
implicit casting which I think makes things way more difficult, and 
completely undercuts the utility of the compiler's mechanical checking.

And on top of that, I WANT that implementation. If I know something is 
not shared, why would I ever want to use atomics on it? I don't like 
needlessly throwing away performance. This is how I would write it:

struct ThreadSafe
{
    private int x;
    void increment()
    {
       ++x; // I know this is not shared, so no reason to use atomics
    }
    void increment() shared
    {
       atomicIncrement(&x); // use atomics, to avoid races
    }
}

The beauty of shared not being implicitly castable, is it allows you to 
focus on the implementation at hand, with the knowledge that nothing 
else can meddle with it. The goal of mechanical checking should be to 
narrow the focus of what needs to be proven correct.

-Steve

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 9:35 AM, Steven Schveighoffer wrote:
 
 struct NotThreadsafe
 {
    private int x;
    void local()
    {
      ++x; // <- invalidates the method below, you violate the other
 function's `shared` promise
    }
    void notThreadsafe() shared
    {
      atomicIncrement(&x);
    }
 }
 

[snip]

 But on top of that, if I can't implicitly cast mutable to shared, then 
 this ACTUALLY IS thread safe, as long as all the casting in the module 
 is sound (easy to search and verify), and hopefully all the casting is 
 encapsulated in primitives like you have written. Because someone on the 
 outside would have to cast a mutable item into a shared item, and this 
 puts the responsibility on them to make sure it works.
 

Another thing to point out -- I can make x public (not private), and 
it's STILL THREAD SAFE.

-Steve

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 6:50 AM Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/18/18 9:35 AM, Steven Schveighoffer wrote:
 struct NotThreadsafe
 {
    private int x;
    void local()
    {
      ++x; // <- invalidates the method below, you violate the other
 function's `shared` promise
    }
    void notThreadsafe() shared
    {
      atomicIncrement(&x);
    }
 }

 [snip]

 But on top of that, if I can't implicitly cast mutable to shared, then
 this ACTUALLY IS thread safe, as long as all the casting in the module
 is sound (easy to search and verify), and hopefully all the casting is
 encapsulated in primitives like you have written. Because someone on the
 outside would have to cast a mutable item into a shared item, and this
 puts the responsibility on them to make sure it works.

 Another thing to point out -- I can make x public (not private), and
 it's STILL THREAD SAFE.

I'm not sure that's an interesting design goal though.
Most things don't have shared methods. If you're writing a thing with
shared methods, you're very in the business of implementing
threadsafety... you're going to want to make it the tightest,
most-unlikely-to-have-threading-bugs thing you can write.
I predict that you're not going to be upset about the restriction.

And if you are, AND you're confident in your application to maintain a
mutually-exclusive shared/TL separation, then you can do this to your
hearts content! Nobody will stop you, and it will be fine.
But I don't think it should be default, because the rules as designed
that way, enforce *users* to perform unsafe casts when they're, on
average, not qualified to make those decisions.

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Thursday, 18 October 2018 at 13:35:22 UTC, Steven 
Schveighoffer wrote:
 struct ThreadSafe
 {
    private int x;
    void increment()
    {
       ++x; // I know this is not shared, so no reason to use 
 atomics
    }
    void increment() shared
    {
       atomicIncrement(&x); // use atomics, to avoid races
    }
 }

But this isn't thread-safe, for the exact reasons described 
elsewhere in this thread (and in fact, incorrectly leveled at 
Manu's proposal). Someone could write this code:

void foo() {
     ThreadSafe* a = new ThreadSafe();
     shareAllOver(a);
     a.increment(); // unsafe, non-shared method call
}

When a.increment() is being called, you have no idea if anyone 
else is using the shared interface.

This is one of the issues that MP (Manu's Proposal) tries to deal 
with. Under MP, your code would *not* be considered thread-safe, 
because the non-shared portion may interfere with the shared 
portion. You'd need to write two types:

struct ThreadSafe {
     private int x;
     void increment() shared {
         atomicIncrement(&x);
     }
}

struct NotThreadSafe {
     private int x;
     void increment() {
         ++x;
     }
}

These two are different types with different semantics, and 
forcing them both into the same struct is an abomination.

In your case, the user of your type will need to ensure 
thread-safety. You may not have any control over how he's doing 
things, while you *do* control the code in your own type (and 
module, since that also affects things). Under MP, the type is 
what needs to be thread-safe, and once it is, the chance of a 
user mucking things up is much lower.

--
   Simen

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 10:11 AM, Simen Kjærås wrote:
 On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
 struct ThreadSafe
 {
    private int x;
    void increment()
    {
       ++x; // I know this is not shared, so no reason to use atomics
    }
    void increment() shared
    {
       atomicIncrement(&x); // use atomics, to avoid races
    }
 }

 
 But this isn't thread-safe, for the exact reasons described elsewhere in 
 this thread (and in fact, incorrectly leveled at Manu's proposal). 
 Someone could write this code:
 
 void foo() {
      ThreadSafe* a = new ThreadSafe();
      shareAllOver(a);

Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type 
ThreadSafe *

      a.increment(); // unsafe, non-shared method call
 }
 
 When a.increment() is being called, you have no idea if anyone else is 
 using the shared interface.

I do, because unless you have cast the type to shared, I'm certain there 
is only thread-local aliasing to it.

 This is one of the issues that MP (Manu's Proposal) tries to deal with. 
 Under MP, your code would *not* be considered thread-safe, because the 
 non-shared portion may interfere with the shared portion. You'd need to 
 write two types:
 
 struct ThreadSafe {
      private int x;
      void increment() shared {
          atomicIncrement(&x);
      }
 }
 
 struct NotThreadSafe {
      private int x;
      void increment() {
          ++x;
      }
 }
 
 These two are different types with different semantics, and forcing them 
 both into the same struct is an abomination.

Why? What if I wanted to have an object that is local for a while, but 
then I want it to be shared (and I ensure carefully when I cast to 
shared that there are no other aliases to that)?

 In your case, the user of your type will need to ensure thread-safety. 

No, the contract the type provides is: if you DON'T cast unshared to 
shared or vice versa, the type is thread-safe.

If you DO cast unshared to shared, then the type is thread-safe as long 
as you no longer use the unshared reference.

This is EXACTLY how immutable works.

 You may not have any control over how he's doing things, while you *do* 
 control the code in your own type (and module, since that also affects 
 things). Under MP, the type is what needs to be thread-safe, and once it 
 is, the chance of a user mucking things up is much lower.

Under MP, the type is DEFENSIVELY thread-safe, locking or using atomics 
unnecessarily when it's thread-local.

-Steve

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/18/18 10:11 AM, Simen Kjærås wrote:
 On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
 struct ThreadSafe
 {
    private int x;
    void increment()
    {
       ++x; // I know this is not shared, so no reason to use atomics
    }
    void increment() shared
    {
       atomicIncrement(&x); // use atomics, to avoid races
    }
 }

 But this isn't thread-safe, for the exact reasons described elsewhere in
 this thread (and in fact, incorrectly leveled at Manu's proposal).
 Someone could write this code:

 void foo() {
      ThreadSafe* a = new ThreadSafe();
      shareAllOver(a);

 Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type
 ThreadSafe *

And here you expect a user to perform an unsafe-cast (which they may
not understand), and we have no language semantics to enforce the
transfer of ownership. How do you assure that the user yields the
thread-local instance?
I think requiring the cast is un-principled in every way that D values.

My proposal doesn't rely on convention (that we *hope* the user does
correctly yield the thread-local instance)... it assures a set of safe
rules by default.

This is the core value proposition of my proposal. It's literally the
entire point.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 2:55 PM, Manu wrote:
 On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/18/18 10:11 AM, Simen Kjærås wrote:
 On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
 struct ThreadSafe
 {
     private int x;
     void increment()
     {
        ++x; // I know this is not shared, so no reason to use atomics
     }
     void increment() shared
     {
        atomicIncrement(&x); // use atomics, to avoid races
     }
 }

 But this isn't thread-safe, for the exact reasons described elsewhere in
 this thread (and in fact, incorrectly leveled at Manu's proposal).
 Someone could write this code:

 void foo() {
       ThreadSafe* a = new ThreadSafe();
       shareAllOver(a);

 Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type
 ThreadSafe *

 
 And here you expect a user to perform an unsafe-cast (which they may
 not understand), and we have no language semantics to enforce the
 transfer of ownership. How do you assure that the user yields the
 thread-local instance?

No, I expect them to do:

auto a = new shared(ThreadSafe)();

 I think requiring the cast is un-principled in every way that D values.

No cast is required. If you have shared data, it's shared. If you have 
thread local data, it's unshared. Allocate the data the way you expect 
to use it.

It's only if you intend to turn unshared data into shared data where you 
need an unsafe cast.

It's not even as difficult as immutable, because you can still modify 
shared data. For instance, the shared constructor doesn't have to have 
special rules about initialization, it can just assume shared from the 
beginning.

-Steve

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 12:15 PM Steven Schveighoffer via
Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/18/18 2:55 PM, Manu wrote:
 On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/18/18 10:11 AM, Simen Kjærås wrote:
 On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
 struct ThreadSafe
 {
     private int x;
     void increment()
     {
        ++x; // I know this is not shared, so no reason to use atomics
     }
     void increment() shared
     {
        atomicIncrement(&x); // use atomics, to avoid races
     }
 }

 But this isn't thread-safe, for the exact reasons described elsewhere in
 this thread (and in fact, incorrectly leveled at Manu's proposal).
 Someone could write this code:

 void foo() {
       ThreadSafe* a = new ThreadSafe();
       shareAllOver(a);

 Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type
 ThreadSafe *

 And here you expect a user to perform an unsafe-cast (which they may
 not understand), and we have no language semantics to enforce the
 transfer of ownership. How do you assure that the user yields the
 thread-local instance?

 No, I expect them to do:

 auto a = new shared(ThreadSafe)();

I don't have any use for this design in my application.
I can't use the model you prescribe, at all.

 I think requiring the cast is un-principled in every way that D values.

 No cast is required. If you have shared data, it's shared. If you have
 thread local data, it's unshared. Allocate the data the way you expect
 to use it.

All data is thread-local, and occasionally becomes shared during periods.
I can't make use of the model you describe.

My proposal is more permissive, and allows a wider range of
application designs. What are the disadvantages?

 It's only if you intend to turn unshared data into shared data where you
 need an unsafe cast.

It's unnecessary though, because threadsafe functions are threadsafe!
You're pointlessly forcing un-safety. Why would I prefer a design that
forces unsafe interactions to perform safe operations?

 It's not even as difficult as immutable, because you can still modify
 shared data. For instance, the shared constructor doesn't have to have
 special rules about initialization, it can just assume shared from the
 beginning.

Your design us immutable, mine is const.
Tell me, how many occurrences of 'immutable' can you find in your
software? ... how about const?
Which is more universally useful? If you had to choose one or the
other, which one could you live without?

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 5:22 PM, Manu wrote:
 On Thu, Oct 18, 2018 at 12:15 PM Steven Schveighoffer via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/18/18 2:55 PM, Manu wrote:
 On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/18/18 10:11 AM, Simen Kjærås wrote:
 On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
 struct ThreadSafe
 {
      private int x;
      void increment()
      {
         ++x; // I know this is not shared, so no reason to use atomics
      }
      void increment() shared
      {
         atomicIncrement(&x); // use atomics, to avoid races
      }
 }

 But this isn't thread-safe, for the exact reasons described elsewhere in
 this thread (and in fact, incorrectly leveled at Manu's proposal).
 Someone could write this code:

 void foo() {
        ThreadSafe* a = new ThreadSafe();
        shareAllOver(a);

 Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type
 ThreadSafe *

 And here you expect a user to perform an unsafe-cast (which they may
 not understand), and we have no language semantics to enforce the
 transfer of ownership. How do you assure that the user yields the
 thread-local instance?

 No, I expect them to do:

 auto a = new shared(ThreadSafe)();

 
 I don't have any use for this design in my application.
 I can't use the model you prescribe, at all.

Huh? This is the same thing you are asking for. How were you intending 
to make a thread-safe thing sharable? Surely it will be typed as shared, 
right? How else will you pass it to multiple threads?

 
 I think requiring the cast is un-principled in every way that D values.

 No cast is required. If you have shared data, it's shared. If you have
 thread local data, it's unshared. Allocate the data the way you expect
 to use it.

 
 All data is thread-local, and occasionally becomes shared during periods.
 I can't make use of the model you describe.

If data is shared, it is shared. Once it is shared, it never goes back.

In your model, everything is *assumed* shared, so that's what you need 
to do, initialize it as shared. It still works just as you like. Even if 
you never actually share it, or share it periodically.

 My proposal is more permissive, and allows a wider range of
 application designs. What are the disadvantages?

The opposite is true. More designs are allowed by restricting casting as 
I have demonstrated many times.

 It's only if you intend to turn unshared data into shared data where you
 need an unsafe cast.

 
 It's unnecessary though, because threadsafe functions are threadsafe!
 You're pointlessly forcing un-safety. Why would I prefer a design that
 forces unsafe interactions to perform safe operations?

No unsafe interactions are required for a type that defensively is 
shared. Just make it always shared, and you don't have any problems.

 It's not even as difficult as immutable, because you can still modify
 shared data. For instance, the shared constructor doesn't have to have
 special rules about initialization, it can just assume shared from the
 beginning.

 
 Your design us immutable, mine is const.

No, your design is not const, const works on normal types. It's 
applicable to anything.

Your design is only applicable to special types that experts write. It's 
not applicable to int, for instance. It feels more like a special 
library than a compiler feature.

 Tell me, how many occurrences of 'immutable' can you find in your
 software? ... how about const?

I generally use inout whenever possible, or const when that is more 
appropriate. But that is for methods.

For data, I generally use immutable when I want a constant.

But like I said, something can't be both shared and unshared. So having 
shared pointers point at unshared data makes no sense -- once it's 
shared, it's shared. So shared really can't be akin to const.

 Which is more universally useful? If you had to choose one or the
 other, which one could you live without?

I would hate to have a const where you couldn't read the data, I 
probably would rather have immutable.

I said I would stop commenting on this thread, and I didn't keep that 
promise. I really am going to stop now. I'm pretty sure Walter will not 
agree with this mechanism, so until you convince him, I don't really 
need to be spending time on this.

We seem to be completely understanding each others mechanisms, but not 
agreeing which one is correct, based on (from both sides) hypothetical 
types and usages.

-Steve

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 3:40 PM Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/18/18 5:22 PM, Manu wrote:
 On Thu, Oct 18, 2018 at 12:15 PM Steven Schveighoffer via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/18/18 2:55 PM, Manu wrote:
 On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/18/18 10:11 AM, Simen Kjærås wrote:
 On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
 struct ThreadSafe
 {
      private int x;
      void increment()
      {
         ++x; // I know this is not shared, so no reason to use atomics
      }
      void increment() shared
      {
         atomicIncrement(&x); // use atomics, to avoid races
      }
 }

 But this isn't thread-safe, for the exact reasons described elsewhere in
 this thread (and in fact, incorrectly leveled at Manu's proposal).
 Someone could write this code:

 void foo() {
        ThreadSafe* a = new ThreadSafe();
        shareAllOver(a);

 Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type
 ThreadSafe *

 And here you expect a user to perform an unsafe-cast (which they may
 not understand), and we have no language semantics to enforce the
 transfer of ownership. How do you assure that the user yields the
 thread-local instance?

 No, I expect them to do:

 auto a = new shared(ThreadSafe)();

 I don't have any use for this design in my application.
 I can't use the model you prescribe, at all.

 Huh? This is the same thing you are asking for. How were you intending
 to make a thread-safe thing sharable? Surely it will be typed as shared,
 right? How else will you pass it to multiple threads?

Things get promoted to shared and distributed on occasion, but there
is only one owner, and he's the only guy with thread-local access.

 I think requiring the cast is un-principled in every way that D values.

 No cast is required. If you have shared data, it's shared. If you have
 thread local data, it's unshared. Allocate the data the way you expect
 to use it.

 All data is thread-local, and occasionally becomes shared during periods.
 I can't make use of the model you describe.

 If data is shared, it is shared. Once it is shared, it never goes back.

I'm suggesting that; data *may be shared* (ie, has valid threadsafe
interaction), or it may not (has no access).
NOT that data "is shared", or "is not". I don't see value in that
proposition. I don't know how to make that useful to me. It's
impossible for me to interact with that model safely.
The ONLY way to interact with that model safely is to allocate objects
shared, or not... there is no safe transition, and I am totally
concerned with the transition.

 In your model, everything is *assumed* shared, so that's what you need
 to do, initialize it as shared. It still works just as you like. Even if
 you never actually share it, or share it periodically.

You can't do that... cus you can't call anything's unshared methods.

 My proposal is more permissive, and allows a wider range of
 application designs. What are the disadvantages?

 The opposite is true. More designs are allowed by restricting casting as
 I have demonstrated many times.

No you haven't.
What design is enabled by it that is inhibited by my design? I can't see this...
Your demonstrated design if fundamentally unsafe, unless you
*allocate* things as shared, and accept that no transition is
possible.
This is not a flexible design. My design also allows that if that's
what you want. Under my proposal, you can equally allocate things
shared to the exact same effect; I haven't taken anything away.

 It's only if you intend to turn unshared data into shared data where you
 need an unsafe cast.

 It's unnecessary though, because threadsafe functions are threadsafe!
 You're pointlessly forcing un-safety. Why would I prefer a design that
 forces unsafe interactions to perform safe operations?

 No unsafe interactions are required for a type that defensively is
 shared. Just make it always shared, and you don't have any problems.

Then you completely lose access to the unshared API. This is not workable.

 It's not even as difficult as immutable, because you can still modify
 shared data. For instance, the shared constructor doesn't have to have
 special rules about initialization, it can just assume shared from the
 beginning.

 Your design us immutable, mine is const.

 No, your design is not const, const works on normal types. It's
 applicable to anything.

But... that's the same here.
const restricts you to calling const methods... shared restricts you
to calling shared methods.

 Tell me, how many occurrences of 'immutable' can you find in your
 software? ... how about const?

 I generally use inout whenever possible, or const when that is more
 appropriate. But that is for methods.

 For data, I generally use immutable when I want a constant.

You're dodging the question... I think you know that nothing can work
without const, but we could live without immutable.
Certainly, immutable wouldn't work without const!
The exact same connundrum applies here. Imagine that we only had
immutable, and any time you wanted to call a const method, you had to
cast to immutable...

 But like I said, something can't be both shared and unshared. So having
 shared pointers point at unshared data makes no sense -- once it's
 shared, it's shared. So shared really can't be akin to const.

Yes, but I'm saying the mutually-exclusive state isn't useful. No
transition is possible, and necessitates unsafety to do anything (that
I want to do).
I'm trying to create a world where the mutual-exclusion is effected in
practise (can define and perform threadsafe interaction), but not in
strict terms, such that transition is impossible... I'm very carefully
designing a solution where the worlds aren't isolated, and that world
is 100% more useful to me and my entire ecosystem.
I don't know how to use shared safely if it's not designed that way,
just like I don't know how to generally use immutable in lieu of
const. immutable wouldn't be practical without const.

 Which is more universally useful? If you had to choose one or the
 other, which one could you live without?

 I would hate to have a const where you couldn't read the data, I
 probably would rather have immutable.

If an object has no const methods, you can't do anything with it. The
usefulness of const depends on people recognising and support const.

 I said I would stop commenting on this thread, and I didn't keep that
 promise. I really am going to stop now. I'm pretty sure Walter will not
 agree with this mechanism, so until you convince him, I don't really
 need to be spending time on this.

 We seem to be completely understanding each others mechanisms, but not
 agreeing which one is correct, based on (from both sides) hypothetical
 types and usages.

I'm still not at all clear on how I'm excluding any interesting use
cases... I have deliberately tried to preserve all the valuable use
cases I'm aware of.
I don't understand your perspective, because I don't understand your
sense of loss.

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/18/18 10:11 AM, Simen Kjærås wrote:
      a.increment(); // unsafe, non-shared method call
 }

 When a.increment() is being called, you have no idea if anyone else is
 using the shared interface.

 I do, because unless you have cast the type to shared, I'm certain there
 is only thread-local aliasing to it.

No, you can never be sure. Your assumption depends on the *user*
engaging in an unsafe operation (the cast), and correctly perform a
conventional act; they must correctly the safely transfer ownership.
My proposal puts all requirements on the author, not the user. I think
this is a much more trustworthy relationship, and in terms of
cognitive load, author:users is a 1:many relationship, and I place the
load on the '1', not the 'many.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 2:59 PM, Manu wrote:
 On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/18/18 10:11 AM, Simen Kjærås wrote:
       a.increment(); // unsafe, non-shared method call
 }

 When a.increment() is being called, you have no idea if anyone else is
 using the shared interface.

 I do, because unless you have cast the type to shared, I'm certain there
 is only thread-local aliasing to it.

 
 No, you can never be sure. Your assumption depends on the *user*
 engaging in an unsafe operation (the cast), and correctly perform a
 conventional act; they must correctly the safely transfer ownership.

Not at all. No transfer of ownership is needed, no cast is needed. If 
you want to share something declare it shared.

 My proposal puts all requirements on the author, not the user. I think
 this is a much more trustworthy relationship, and in terms of
 cognitive load, author:users is a 1:many relationship, and I place the
 load on the '1', not the 'many.

Sure, but we can create a system today where smart people make objects 
that do the right thing without compiler help. We don't need to break 
the guarantees of shared to do it.

-Steve

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Thursday, 18 October 2018 at 14:19:41 UTC, Steven 
Schveighoffer wrote:
 On 10/18/18 10:11 AM, Simen Kjærås wrote:
 On Thursday, 18 October 2018 at 13:35:22 UTC, Steven 
 Schveighoffer wrote:
 struct ThreadSafe
 {
    private int x;
    void increment()
    {
       ++x; // I know this is not shared, so no reason to use 
 atomics
    }
    void increment() shared
    {
       atomicIncrement(&x); // use atomics, to avoid races
    }
 }

 
 But this isn't thread-safe, for the exact reasons described 
 elsewhere in this thread (and in fact, incorrectly leveled at 
 Manu's proposal). Someone could write this code:
 
 void foo() {
      ThreadSafe* a = new ThreadSafe();
      shareAllOver(a);

 Error: cannot call function shareAllOver(shared(ThreadSafe) *) 
 with type ThreadSafe *

Sorry, typo. Should of course have been 
shareAllOver(cast(shared)a);

      a.increment(); // unsafe, non-shared method call
 }
 
 When a.increment() is being called, you have no idea if anyone 
 else is using the shared interface.

 I do, because unless you have cast the type to shared, I'm 
 certain there is only thread-local aliasing to it.



 This is one of the issues that MP (Manu's Proposal) tries to 
 deal with. Under MP, your code would *not* be considered 
 thread-safe, because the non-shared portion may interfere with 
 the shared portion. You'd need to write two types:
 
 struct ThreadSafe {
      private int x;
      void increment() shared {
          atomicIncrement(&x);
      }
 }
 
 struct NotThreadSafe {
      private int x;
      void increment() {
          ++x;
      }
 }
 
 These two are different types with different semantics, and 
 forcing them both into the same struct is an abomination.

 Why? What if I wanted to have an object that is local for a 
 while, but then I want it to be shared (and I ensure carefully 
 when I cast to shared that there are no other aliases to that)?

 In your case, the user of your type will need to ensure 
 thread-safety.

 No, the contract the type provides is: if you DON'T cast 
 unshared to shared or vice versa, the type is thread-safe.

 If you DO cast unshared to shared, then the type is thread-safe 
 as long as you no longer use the unshared reference.

 This is EXACTLY how immutable works.

Yes, and that means the user of the type will need to follow 
these rules to ensure thread-safety. Which is what I said. Under 
MP, it's simply safe.


 You may not have any control over how he's doing things, while 
 you *do* control the code in your own type (and module, since 
 that also affects things). Under MP, the type is what needs to 
 be thread-safe, and once it is, the chance of a user mucking 
 things up is much lower.

 Under MP, the type is DEFENSIVELY thread-safe, locking or using 
 atomics unnecessarily when it's thread-local.

Yes, because safety >> efficiency. There's nothing stopping you 
from making fast, unsafe functions under MP. Call them 
unsafe_<foo> to make them easy to grep for. Since only the type 
implementer, not the users, write this code, it's not an undue 
burden.

One of the greatest benefits of MP is that all the potential 
problem points are in one place. In a large codebase with 
multiple developers, anyone anywhere could be using a dangling 
unshared reference under the current schema. Under MP, that's 
confined to the type, not its uses.

--
   Simen

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 6:40 AM Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/17/18 10:26 PM, Manu wrote:
 On Wed, Oct 17, 2018 at 6:50 PM Steven Schveighoffer via Digitalmars-d
 The implicit cast means that you have to look at more than just your
 method. You have to look at the entire module, and figure out all the
 interactions, to see if the thread safe method actually is thread safe.
 That's programming by convention, and fully trusting the programmer.

 I don't understand... how can the outer context affect the
 threadsafety of a properly encapsulated thing?

 [snip]

 You need to take it for an intellectual spin. Show me how it's corrupt
 rather than just presenting discomfort with the idea in theory.
 You're addicted to some concepts that you've carried around for a long
 time. There is no value in requiring casts, they're just a funky
 smell, and force the user to perform potentially unsafe manual
 conversions, or interactions that they don't understand.

 For example (your example):

 struct NotThreadsafe
 {
    private int x;
    void local()
    {
      ++x; // <- invalidates the method below, you violate the other
 function's `shared` promise
    }
    void notThreadsafe() shared
    {
      atomicIncrement(&x);
    }
 }

 First, note the comment. I can't look ONLY at the implementation of
 "notThreadSafe" (assuming the function name is less of a giveaway) in
 order to guarantee that it's actually thread safe. I have to look at the
 WHOLE MODULE. Anything could potentially do what local() does. I added
 private to x to at least give the appearance of thread safety.

 But on top of that, if I can't implicitly cast mutable to shared, then
 this ACTUALLY IS thread safe, as long as all the casting in the module
 is sound (easy to search and verify), and hopefully all the casting is
 encapsulated in primitives like you have written. Because someone on the
 outside would have to cast a mutable item into a shared item, and this
 puts the responsibility on them to make sure it works.

 I'm ALL FOR having shared be completely unusable as-is unless you cast
 (thanks for confirming what I suspected in your last post). It's the
 implicit casting which I think makes things way more difficult, and
 completely undercuts the utility of the compiler's mechanical checking.

 And on top of that, I WANT that implementation. If I know something is
 not shared, why would I ever want to use atomics on it? I don't like
 needlessly throwing away performance. This is how I would write it:

 struct ThreadSafe
 {
     private int x;
     void increment()
     {
        ++x; // I know this is not shared, so no reason to use atomics
     }
     void increment() shared
     {
        atomicIncrement(&x); // use atomics, to avoid races
     }
 }

 The beauty of shared not being implicitly castable, is it allows you to
 focus on the implementation at hand, with the knowledge that nothing
 else can meddle with it. The goal of mechanical checking should be to
 narrow the focus of what needs to be proven correct.

 -Steve

I understand your argument, and I used to think this too... but I
concluded differently for 1 simple reason: usability.
I have demonstrated these usability considerations in production. I am
confident it's the right balance.

I propose:
 1. Normal people don't write thread-safety, a very small number of
unusual people do this. I feel very good about biasing 100% of the
cognitive load INSIDE the shared method. This means the expert, and
ONLY the expert, must make decisions about thread-safety
implementation.
 2. Implicit conversion allows users to safely interact with safe
things without doing unsafe casts. I think it's a complete design fail
if you expect any user anywhere to perform an unsafe cast to call a
perfectly thread-safe function. The user might not properly understand
their obligations.
 3. The practical result of the above is, any complexity relating to
safety is completely owned by the threadsafe author, and not cascaded
to the user. You can't expect users to understand, and make correct
decisions about threadsafety. Safety should be default position.

I recognise the potential loss of an unsafe optimised thread-local path.
1. This truly isn't a big deal. If this is really hurting you, you
will notice on the profiler, and deploy a thread-exclusive path
assuming the context supports it.
2. I will trade that for confidence in safe interaction every day of
the week. Safety is the right default position here.
2. You just need to make the unsafe thread-exclusive variant explicit, eg:

 struct ThreadSafe
 {
     private int x;
     void unsafeIncrement() // <- make it explicit
     {
        ++x; // User has asserted that no sharing is possible, no reason to use
atomics
     }
     void increment() shared
     {
        atomicIncrement(&x); // object may be shared
     }
 }

I think this is quiet a reasonable and clearly documented compromise.
I think absolutely-reliably-threadsafe-by-default is the right default
position. And if you want to accept unsafe operations for optimsation
circumstances, then you're welcome to deploy that in your code as you
see fit.

If the machinery is not a library for distribution and local to your
application, and you know for certain that your context is such that
thread-local and shared are mutually exclusive, then you're free to
make the unshared overload not-threadsafe; you can do this because you
know your application context.
You just shouldn't make widely distributed tooling this way.

I will indeed do this myself in some cases, because I know those facts
about my application.
But I wouldn't compromise the default design of shared for this
optimisation potential... deliberately deployed optimisation is okay
to be unsafe when taken in context.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 18:24:47 UTC, Manu wrote:

 I have demonstrated these usability considerations in 
 production. I am
 confident it's the right balance.

Then convince us. So far you haven't.

 I propose:
  1. Normal people don't write thread-safety, a very small 
 number of
 unusual people do this. I feel very good about biasing 100% of 
 the
 cognitive load INSIDE the shared method. This means the expert, 
 and
 ONLY the expert, must make decisions about thread-safety 
 implementation.

No argument.

  2. Implicit conversion allows users to safely interact with 
 safe
 things without doing unsafe casts. I think it's a complete 
 design fail
 if you expect any user anywhere to perform an unsafe cast to 
 call a
 perfectly thread-safe function. The user might not properly 
 understand
 their obligations.

Disagreed. "Normal" people wouldn't be doing any unsafe casts and 
*must not be able to* do something as unsafe as silent promotion 
of thread-local mutable data to shared. But it's perfectly fine 
for the "expert" user to do such a promotion, explicitly.

  3. The practical result of the above is, any complexity 
 relating to
 safety is completely owned by the threadsafe author, and not 
 cascaded
 to the user. You can't expect users to understand, and make 
 correct
 decisions about threadsafety. Safety should be default position.

Exactly. And an implicit conversion from mutable to shared isn't 
safe at all.

 I recognise the potential loss of an unsafe optimised 
 thread-local path.
 1. This truly isn't a big deal. If this is really hurting you, 
 you
 will notice on the profiler, and deploy a thread-exclusive path
 assuming the context supports it.

 2. I will trade that for confidence in safe interaction every 
 day of
 the week. Safety is the right default position here.

Does not compute. Either you're an "expert" from above and live 
with that burden, or you're not.

 2. You just need to make the unsafe thread-exclusive variant 
 explicit, eg:

 struct ThreadSafe
 {
     private int x;
     void unsafeIncrement() // <- make it explicit
     {
        ++x; // User has asserted that no sharing is possible, 
 no reason to use atomics
     }
     void increment() shared
     {
        atomicIncrement(&x); // object may be shared
     }
 }


No. The above code is not thread-safe at all. The private int 
*must* be declared shared. Then it becomes:

struct ThreadSafe {
     // These must be *required* if you want to assert any thread 
safety. Be nice
     // if the compiler did that for us.
      disable this(this);
      disable void opAssign(typeof(this));

     private shared int x; // <- *must* be shared

     void unsafeIncrement()  system {
         x.assumeUnshared += 1;
     }

     // or deduced:

     void unsafeIncrement()() // assumeUnshared must be  system, 
thus this unsafeIncrement will also be deduced  system
     {
         x.assumeUnshared += 1;
     }

     void increment() shared {
         x.atomicOp!"+="(1);
     }
}

 I think this is quiet a reasonable and clearly documented 
 compromise.

With the fixes above, it is. Without them, it will only be 
apparent from documentation, and who writes, or reads, that?..

 I think absolutely-reliably-threadsafe-by-default is the right 
 default position.

But it is exactly the opposite of automatic promotions from 
mutable to shared!

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 2:24 PM, Manu wrote:
 I understand your argument, and I used to think this too... but I
 concluded differently for 1 simple reason: usability.

You have not demonstrated why your proposal is usable, and the proposal 
to simply make shared not accessible while NOT introducing implicit 
conversion is somehow not usable.

I find quite the opposite -- the implicit conversion introduces more 
pitfalls and less guarantees from the compiler.

 I have demonstrated these usability considerations in production. I am
 confident it's the right balance.

Are these considerations the list below, or are they something else? If 
so, can you list them?

 I propose:
   1. Normal people don't write thread-safety, a very small number of
 unusual people do this. I feel very good about biasing 100% of the
 cognitive load INSIDE the shared method. This means the expert, and
 ONLY the expert, must make decisions about thread-safety
 implementation.

Thread safety is not easy. But it's also not generic.

In terms of low-level things like atomics and lock-free implementations, 
those ARE generic and SHOULD only be written by experts. But other than 
that, you can't know how someone has designed all the conditions in 
their code.

For example, you can have an expert write mutex locks and semaphores. 
But they can't tell you the proper order to lock different objects to 
ensure there's no deadlock. That's application specific.

   2. Implicit conversion allows users to safely interact with safe
 things without doing unsafe casts. I think it's a complete design fail
 if you expect any user anywhere to perform an unsafe cast to call a
 perfectly thread-safe function. The user might not properly understand
 their obligations.

I also do not expect anyone to perform unsafe casts in normal use. I 
expect them to use more generic well-written types in a shared-object 
library. Casting should be very rare.

   3. The practical result of the above is, any complexity relating to
 safety is completely owned by the threadsafe author, and not cascaded
 to the user. You can't expect users to understand, and make correct
 decisions about threadsafety. Safety should be default position.

I think these are great rules, and none are broken by keeping the 
explicit cast requirement in place.

 I recognise the potential loss of an unsafe optimised thread-local path.
 1. This truly isn't a big deal. If this is really hurting you, you
 will notice on the profiler, and deploy a thread-exclusive path
 assuming the context supports it.

This is a mischaracterization. The thread-local path is perfectly safe 
because only one thread can be accessing the data. That's why it's 
thread-local and not shared.

 2. I will trade that for confidence in safe interaction every day of
 the week. Safety is the right default position here.

You can be confident that any shared data is properly synchronized via 
the API provided. No confidence should be lost here.

 2. You just need to make the unsafe thread-exclusive variant explicit, eg:

It is explicit, the thread-exclusive variant is not marked shared, and 
cannot be called on data that is actually shared and needs synchronization.

 
 struct ThreadSafe
 {
      private int x;
      void unsafeIncrement() // <- make it explicit
      {
         ++x; // User has asserted that no sharing is possible, no reason to
use atomics
      }
      void increment() shared
      {
         atomicIncrement(&x); // object may be shared
      }
 }


This is more design by convention.

 
 I think this is quiet a reasonable and clearly documented compromise.
 I think absolutely-reliably-threadsafe-by-default is the right default
 position. And if you want to accept unsafe operations for optimsation
 circumstances, then you're welcome to deploy that in your code as you
 see fit.

All thread-local operations are thread-safe by default, because there 
can be only one thread using it. That is the beauty of the current 
regime, regardless of how broken shared is -- unshared is solid. We 
shouldn't want to break that guarantee.

 If the machinery is not a library for distribution and local to your
 application, and you know for certain that your context is such that
 thread-local and shared are mutually exclusive, then you're free to
 make the unshared overload not-threadsafe; you can do this because you
 know your application context.
 You just shouldn't make widely distributed tooling this way.

I can make widely distributed tooling that does both shared and unshared 
versions of the code, and ALL are thread safe. No choices are necessary, 
no compromise on performance, and no design by convention.

 I will indeed do this myself in some cases, because I know those facts
 about my application.
 But I wouldn't compromise the default design of shared for this
 optimisation potential... deliberately deployed optimisation is okay
 to be unsafe when taken in context.
 

Except it's perfectly thread safe to use data without synchronization in 
one thread -- which is supported by having unshared data. Unshared means 
only one thread. In your proposal, anything can be seen from one or more 
threads.

-Steve

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 12:10 PM Steven Schveighoffer via
Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 10/18/18 2:24 PM, Manu wrote:
 I understand your argument, and I used to think this too... but I
 concluded differently for 1 simple reason: usability.

 You have not demonstrated why your proposal is usable, and the proposal
 to simply make shared not accessible while NOT introducing implicit
 conversion is somehow not usable.

 I find quite the opposite -- the implicit conversion introduces more
 pitfalls and less guarantees from the compiler.

I don't think it introduces *more*, I think it's the same number of
compiler guarantees, but it rearranges them into what I feel are a
more satisfactory and reliable configuration.
Ie, I rearrange such that the compiler guarantees are applicable to
the 'many' case rather than the '1' case, and from that perspective,
it means the compiler guarantees are more widely deployed.

 I have demonstrated these usability considerations in production. I am
 confident it's the right balance.

 Are these considerations the list below, or are they something else? If
 so, can you list them?

 I propose:
   1. Normal people don't write thread-safety, a very small number of
 unusual people do this. I feel very good about biasing 100% of the
 cognitive load INSIDE the shared method. This means the expert, and
 ONLY the expert, must make decisions about thread-safety
 implementation.

 Thread safety is not easy. But it's also not generic.

They are generic in lots of instances. A lock-free queue object is a
generic container object which can be deployed safely.
A threadsafe state-machine implementation which implements reliable
and valid state transitions using atomics is generic.
Most of the threadsafe tooling I've ever seen expose at the
user-facing level is absolutely generic, and can be packaged as a safe
and user-friendly abstraction.

 In terms of low-level things like atomics and lock-free implementations,
 those ARE generic and SHOULD only be written by experts. But other than
 that, you can't know how someone has designed all the conditions in
 their code.

That guy is implementing the machinery, and he is the best possible
person to validate that he delivered on his promises. Nobody should
have to perform un-safety to interact with his promises. There is
greater chance of user error than expert failure (users are, by
definition, more numerous in number, and almost certainly less
qualified).

 For example, you can have an expert write mutex locks and semaphores.
 But they can't tell you the proper order to lock different objects to
 ensure there's no deadlock. That's application specific.

A mutex-style API has an element of un-safety by definition. My
proposal doesn't affect lock and cast-away workflows.
That workflow remains the same, and depends on unsafe interactions,
and I don't think it's possible to arrange that any other way.
What I'm trying to do is express another form of safe interaction
tools with threadsafe devices, and in my work, I would use those
exclusively. I have no use or desire for unsafe lock-and-cast
workflows in our ecosystem.
I'm trying to add a new possibility for expressing threadsafety that
doesn't exist with strong guarantees today.

   2. Implicit conversion allows users to safely interact with safe
 things without doing unsafe casts. I think it's a complete design fail
 if you expect any user anywhere to perform an unsafe cast to call a
 perfectly thread-safe function. The user might not properly understand
 their obligations.

 I also do not expect anyone to perform unsafe casts in normal use. I
 expect them to use more generic well-written types in a shared-object
 library. Casting should be very rare.

You're resistant to implicit conversion to shared. Casting to shared
is unsafe, and depending on them to yield thread-local ownership is a
'hope' at best; your worldview depends on users performing unsafe
interactions with otherwise safe (threadsafe) API's.
I'm trying to reposition away from that world into a safe-by-default place.

   3. The practical result of the above is, any complexity relating to
 safety is completely owned by the threadsafe author, and not cascaded
 to the user. You can't expect users to understand, and make correct
 decisions about threadsafety. Safety should be default position.

 I think these are great rules, and none are broken by keeping the
 explicit cast requirement in place.

They are expressly broken whenever anyone has to cast to shared. That
is unsafe, and they're expected to yield the thread-local ownership by
convention.
That is the cancer at the core of my worldview, which I'm trying to
factor away. I think my proposal delivers on that very elegantly.
There will be *no casts* anywhere, outside of the low-level
implementation methods written by the expert.
I am aiming for safe-by-default. If you want to step outside that
place, you do so deliberate, and carefully, and it's easy to search
for.

 I recognise the potential loss of an unsafe optimised thread-local path.
 1. This truly isn't a big deal. If this is really hurting you, you
 will notice on the profiler, and deploy a thread-exclusive path
 assuming the context supports it.

 This is a mischaracterization. The thread-local path is perfectly safe
 because only one thread can be accessing the data. That's why it's
 thread-local and not shared.

It's not safe. There may be a thread-local instance at any time,
because we have no mechanism to concretely transfer ownership. You
rely on unsafe cast and convention to yield ownership to perform the
transition.
I'm saying, I don't find that acceptable, and I'm designing for that
reality, rather than wishful thinking.

 2. I will trade that for confidence in safe interaction every day of
 the week. Safety is the right default position here.

 You can be confident that any shared data is properly synchronized via
 the API provided. No confidence should be lost here.

But you can't safely call a threadsafe method, and you can't
transition TL -> shared data. My proposal addresses those issues.

 2. You just need to make the unsafe thread-exclusive variant explicit, eg:

 It is explicit, the thread-exclusive variant is not marked shared, and
 cannot be called on data that is actually shared and needs synchronization.

It's defeated by the other considerations though. Implicit conversion
is the only way to allow data to become shared safely, and the design
works elegantly.

 struct ThreadSafe
 {
      private int x;
      void unsafeIncrement() // <- make it explicit
      {
         ++x; // User has asserted that no sharing is possible, no reason to
use atomics
      }
      void increment() shared
      {
         atomicIncrement(&x); // object may be shared
      }
 }


 This is more design by convention.

You want to do an unsafe thing. You need to prescribe convention when
you want to do unsafe things.
I don't recommend this, I'm telling you that you can do it in your
case of desired optimisation.

 I think this is quiet a reasonable and clearly documented compromise.
 I think absolutely-reliably-threadsafe-by-default is the right default
 position. And if you want to accept unsafe operations for optimsation
 circumstances, then you're welcome to deploy that in your code as you
 see fit.

 All thread-local operations are thread-safe by default, because there
 can be only one thread using it. That is the beauty of the current
 regime, regardless of how broken shared is -- unshared is solid. We
 shouldn't want to break that guarantee.

Right, but you just said it; shared is useless by extension. I'm
trying to reconcile shared with the current design in such a way to
express a useful concept.
I'm not undermining how thread-local is threadsafe by default, that
promise still exists unchanged.
What I'm saying is, promise of threadsafety must be true with respect
to the threadlocal implementation, that is all. I think it's a
reasonable definition for threadsafety, and the value is evident; you
don't need to perform unsafe operations and convention to do
interaction with threadsafe machinery (as it should be, because it is
*threadsafe*).

 If the machinery is not a library for distribution and local to your
 application, and you know for certain that your context is such that
 thread-local and shared are mutually exclusive, then you're free to
 make the unshared overload not-threadsafe; you can do this because you
 know your application context.
 You just shouldn't make widely distributed tooling this way.

 I can make widely distributed tooling that does both shared and unshared
 versions of the code, and ALL are thread safe. No choices are necessary,
 no compromise on performance, and no design by convention.

You provide no path to make an unshared thing shared. I don't think
the language has any tools to express this; we can't express an
ownership transfer.
The mutually-exclusive shared-ness design fails here. I'm designing for reality.

 I will indeed do this myself in some cases, because I know those facts
 about my application.
 But I wouldn't compromise the default design of shared for this
 optimisation potential... deliberately deployed optimisation is okay
 to be unsafe when taken in context.

 Except it's perfectly thread safe to use data without synchronization in
 one thread -- which is supported by having unshared data. Unshared means
 only one thread. In your proposal, anything can be seen from one or more
 threads.

Right, but as I've tried to demonstrate, shared can't exist safely
with this construction, no transition is possible.
I consider that a non-starter, and my design addresses that reality.

ag0aep6g <anonymous example.com> writes:

On 17.10.18 20:46, Manu wrote:
 struct NotThreadsafe
 {
    int x;
    void local()
    {
      ++x; // <- invalidates the method below, you violate the other
 function's `shared` promise
    }
    void notThreadsafe() shared
    {
      atomicIncrement(&x);
    }
 }

In the `shared` method you'd get a nice error when attempting `++x;`, 
because it's not thread-safe. With your proposal, it's just as unsafe in 
`local`, but you don't get any help from the compiler in spotting it. 
`local` is effectively ` trusted` without being marked as such.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/15/2018 11:46 AM, Manu wrote:
 [...]

Shared has one incredibly valuable feature - it allows you, the programmer, to 
identify data that can be accessed by multiple threads. There are so many ways 
that data can be shared, the only way to comprehend what is going on is to
build 
a wall around shared data.

(The exception to this is immutable data. Immutable data does not need 
synchronization, so there is no need to distinguish between shared and unshared 
immutable data.)

This falls completely apart if shared and unshared data can be aliased. When 
Andrei and I came up with the rules for:

    mutable
    const
    shared
    const shared
    immutable

and which can be implicitly converted to what, so far nobody has found a fault 
in those rules. Timon Gehr has done a good job showing that they still stand 
unbreached.

So, how can mutable data become shared, and vice versa? I've never found a way 
to do that that is proveably safe. Therefore, it's up to the programmer.

If a programmer does the following:

     T* p;
     shared(T)* sp;

     p = cast(T*)sp;           (1)
     sp = cast(shared(T)*) p;  (2)

those casts will be rejected in  safe code. They'll have to be in  trusted or 
 system code. It will be up to the programmer to ensure (via mutexes, locks, 
uniqueness, etc.) that:

     (1) sp is a unique pointer and that ownership of the data is thus 
transferred for the lifetime of p

     (2) p is a unique pointer and that ownership of the data is thus 
transferred for the lifetime of sp

A sensible program should try to minimize the places where there are these 
"gates" through the wall, as those gates will be where you'll be looking when 
threading bugs appear.

Allowing implicit aliasing between shared and unshared data means the entire 
program needs to reviewed for threading bugs, rather than just the shared
parts. 
One might as well just not have shared as a language feature at all. D being a 
systems programming language, one can certainly write code that way, just like 
one does with C, and with all of the threading bugs one gets doing that.

----------

Currently, the compiler allows you to read/write shared data without any locks 
or atomics. I.e. the compiler doesn't attempt to insert any synchronization on 
your behalf. Given all the ways synchronization can be done, it just seems 
presumptive and restrictive to impose a particular scheme. It's pretty much
left 
up to the user.

Which is why I recommend minimizing the amount of code that actually has to
deal 
with shared data.

Manu <turkeyman gmail.com> writes:

On Tue, Oct 16, 2018 at 10:45 PM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/15/2018 11:46 AM, Manu wrote:
 [...]

 Shared has one incredibly valuable feature - it allows you, the programmer, to
 identify data that can be accessed by multiple threads. There are so many ways
 that data can be shared, the only way to comprehend what is going on is to
build
 a wall around shared data.

 (The exception to this is immutable data. Immutable data does not need
 synchronization, so there is no need to distinguish between shared and unshared
 immutable data.)

 This falls completely apart if shared and unshared data can be aliased.

What does it mean 'aliased' precisely? I'm trying to prevent that in
practical terms.
If you can't read/write to a shared object... is it actually aliased?
Or do you just have a fancy int that happens to look like a pointer?

 When Andrei and I came up with the rules for:

     mutable
     const
     shared
     const shared
     immutable

 and which can be implicitly converted to what, so far nobody has found a fault
 in those rules.

Oh bollocks... everyone has been complaining about this for at least
the 10 years I've been here!
Shared is uninteresting and mostly useless as spec-ed, everyone knows this.
Interaction with shared via barely-controlled blunt casting in
 trusted blocks is feeble and boring. It doesn't really give us
anything in practice that we don't have in C++. It's an uninspired
design.

I am working against a design where *everything* may be shared, and
it's a very interesting design space. I have no doubt it represents
the future of my field.
We are limited by what the type system can express on this front. I'm
trying to explore opportunities to make shared interesting and
useful... and I'm mostly being met with prejudice here.
Eject what you think you know about shared from your brain, and try
and take a fresh look from the perspective I'm proposing.
I think you'll find that the current wisdom and interactions with
shared are actually preserved in practice, but new opportunities
become available. If not, I want to understand how the design is
broken, so I can continue to iterate.

If D offered a real advantage here over C++, this would really
represent a strong attracting force.

 Timon Gehr has done a good job showing that they still stand
 unbreached.

His last comment was applied to a different proposal.
His only comment on this thread wasn't in response to the proposal in
this thread.
If you nominate Timon as your proxy, then he needs to destroy my
proposal, or at least comment on it, rather than make some prejudiced
comment generally.

 So, how can mutable data become shared, and vice versa? I've never found a way
 to do that that is proveably safe. Therefore, it's up to the programmer.

I'm proposing a way, and that is:
1. the rule must be applied that shared object can not be read or written
2. attributing a method shared is a statement and a promise that the
method is threadsafe

The rest just follows naturally.

 If a programmer does the following:

      T* p;
      shared(T)* sp;

      p = cast(T*)sp;           (1)
      sp = cast(shared(T)*) p;  (2)

 those casts will be rejected in  safe code. They'll have to be in  trusted or
  system code. It will be up to the programmer to ensure (via mutexes, locks,
 uniqueness, etc.) that:

      (1) sp is a unique pointer and that ownership of the data is thus
 transferred for the lifetime of p

      (2) p is a unique pointer and that ownership of the data is thus
 transferred for the lifetime of sp

'Transfer' is a nice idea, but it's not what's happening here. That's
not what these operations do.
Transferring ownership is a job for move semantics, and that's nowhere in sight.

Also, there's no need to 'transfer' a this pointer to shared to call a
threadsafe method. You can always just call it safely.

 A sensible program should try to minimize the places where there are these
 "gates" through the wall, as those gates will be where you'll be looking when
 threading bugs appear.

I agree, that's my goal here. Shared right now is an unsafe mess; I
want to apply aggressive restriction so that you can't do unsafe
operations without explicit casts.

 Allowing implicit aliasing between shared and unshared data means the entire
 program needs to reviewed for threading bugs, rather than just the shared
parts.

That's backwards, I'm suggesting conversion TO shared, such that you
are able to call threadsafe methods on thread-local things. There's no
possible harm in that.
I'm trying to eliminate aliasing by removing read/write access.
The result is, the only thing you are able to do with a shared
reference are explicitly threadsafe things. Any other operation is
inaccessible without deliberate blunt casting, and the rules
surrounding that are the same as always; those that you listed above.

 One might as well just not have shared as a language feature at all. D being a
 systems programming language, one can certainly write code that way, just like
 one does with C, and with all of the threading bugs one gets doing that.

And this is a silly claim that's been made before in this thread. It
doesn't make sense, and I kinda have to presume here you have either
not read or do not understand my proposal...
Are you suggesting that I hate shared, and I want to make it
worthless, therefore I'm trying to change the rules to eliminate it
from practical existence?
I promise you, I'd *really* like it if shared were a _useful_ language
feature. The suggestion that I intend to undermine it such that it not
be a language feature at all is obviously ridiculous.


 ----------

 Currently, the compiler allows you to read/write shared data without any locks
 or atomics.

Right, this is the core of the problem, and it must change for shared
to model anything useful.

 I.e. the compiler doesn't attempt to insert any synchronization on
 your behalf. Given all the ways synchronization can be done, it just seems
 presumptive and restrictive to impose a particular scheme. It's pretty much
left
 up to the user.

But it's a useless model; it's just a boring sentinel tag with no
functional application. The shared rules don't model any useful
behaviour.
I'm trying to express how shared could model a useful behaviour, and
coincidentally be *more* conceptually safe, and intuitively model and
communicate appropriate interactions with threadsafe objects.

 Which is why I recommend minimizing the amount of code that actually has to
deal
 with shared data.

Well, the future is SMP. 100% of our code potentially deals with
shared data, and it would be idea to do so in a typesafe manner.
We can't model this with C++. We can't model it in D either, because
D's `shared` doesn't model anything... but with just the one change
that shared can't read or write, we would have the foundation of
something useful.
Being able to read/write to shared objects is weird; it's completely
unsafe in every event. It's a useless access right, and changing it
such that shared can not read/write would improve the model in such a
way that it starts to become useful.

jmh530 <john.michael.hall gmail.com> writes:

On Wednesday, 17 October 2018 at 07:20:20 UTC, Manu wrote:
 [snip]

 Oh bollocks... everyone has been complaining about this for at 
 least
 the 10 years I've been here!
 [snip]

As far as I had known from reading the forums, shared was not 
feature complete.

Also, are you familiar with Atila's fearless library for safe 
sharing?
https://github.com/atilaneves/fearless

Timon Gehr <timon.gehr gmx.ch> writes:

On 17.10.2018 09:20, Manu wrote:
 Timon Gehr has done a good job showing that they still stand
 unbreached.

 His last comment was applied to a different proposal.
 His only comment on this thread wasn't in response to the proposal in
 this thread.
 If you nominate Timon as your proxy, then he needs to destroy my
 proposal, or at least comment on it, rather than make some prejudiced
 comment generally.
 

There is no "prejudice", just reasoning. Your proposal was "disallow 
member access on shared aggregates, allow implicit conversion from 
unshared to shared and keep everything else the same". This is a bad 
proposal. There may be a good proposal that allows the things you want, 
but you have not stated what they are, your OP was just: "look at this 
bad proposal, it might work, no?" I said no, then was met with some 
hostility.

You should focus on finding a good proposal that achieves what you want 
without breaking the type system instead of attacking me.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

Jesus Manu, it's soon 8 pages of dancing around a trivial issue.

Implicit casting from mutable to shared is unsafe, case closed.
Explicit cast from mutable to unsafe, on the other hand:

- is an assertion (on programmer's behalf) that this instance is 
indeed unique
- is self-documenting
- is greppable (especially if implemented as assumeShared or 
other descriptive name).

I don't understand why are you so fixated on this. The other 
parts of your proposal are much more important.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Wednesday, 17 October 2018 at 13:36:53 UTC, Stanislav Blinov 
wrote:

 Explicit cast from mutable to unsafe, on the other hand:

Blargh, to shared of course.

Guillaume Piolat <spam smam.org> writes:

On Wednesday, 17 October 2018 at 07:20:20 UTC, Manu wrote:
 Shared is uninteresting and mostly useless as spec-ed, everyone 
 knows this.
 Interaction with shared via barely-controlled blunt casting in
  trusted blocks is feeble and boring. It doesn't really give us
 anything in practice that we don't have in C++. It's an 
 uninspired
 design.

+1

I'm pretty sure it's great in theory, in practice it's another 
invasive type constructor, and not a particular well-understood 
or easy to deal with one.

The fact that this _type constructor_ finds its way into 
_identifiers_ create some concern: 
https://github.com/dlang/phobos/blob/656798f2b385437c239246b59e0433148190938c/std/experimental/allocator/package.d#L642

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Wednesday, 17 October 2018 at 14:44:19 UTC, Guillaume Piolat 
wrote:

 The fact that this _type constructor_ finds its way into 
 _identifiers_ create some concern: 
 https://github.com/dlang/phobos/blob/656798f2b385437c239246b59e0433148190938c/std/experimental/allocator/package.d#L642

Well, ISharedAllocator is indeed overboard, but at least 
`allocateShared` would've been a very useful identifier indeed.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/17/2018 12:20 AM, Manu wrote:
 What does it mean 'aliased' precisely?

Aliasing means there are two paths to the same piece of data. That could be two 
pointers pointing to the same data, or one pointer to a variable that is 
accessible by name.

 It doesn't really give us
 anything in practice that we don't have in C++.

It provides a standard, enforced way to distinguish shared data from unshared 
data, and no way to bypass it in  safe code. There's no way to do that in C++.

Manu <turkeyman gmail.com> writes:

On Fri., 19 Oct. 2018, 3:10 am Walter Bright via Digitalmars-d, <
digitalmars-d puremagic.com> wrote:

 On 10/17/2018 12:20 AM, Manu wrote:
 What does it mean 'aliased' precisely?

 Aliasing means there are two paths to the same piece of data. That could
 be two
 pointers pointing to the same data, or one pointer to a variable that is
 accessible by name.

The reason I ask is because, by my definition, if you have:
int* a;
shared(int)* b = a;

While you have 2 numbers that address the same data, it is not actually
aliased because only `a` can access it. It is not aliased in any practical
sense.

 It doesn't really give us
 anything in practice that we don't have in C++.

 It provides a standard, enforced way to distinguish shared data from
 unshared
 data, and no way to bypass it in  safe code. There's no way to do that in
 C++.

Right, but we can do so much better. I want shared to model "what is
thread- safe to do", because that models what you are able to do, and what
API's should encourage when operating on `shared` things.

Exclusively distinguishing shared and unshared data is not an interesting
distinction if shared data has no access.
I've been trying to say over and over; ignore what you think you know about
that definition, accept my rules strictly as given (they're very simple and
concise, there's only 2 rules), such that shared will mean "is threadsafe
to call with this data" when applied to function args...
Build the thought experiment outward from there.
That's an interesting and useful definition for shared, and it leads to a
framework where shared is useful in a fully  safe SMP program, and even
models  safe transitions across unshared -> shared boundaries (parallel
for, map/reduce, etc, fork and join style workloads), which are typical
lock-free patterns.

Lock-and-cast semantics are preserved unchanged for those that interact in
the way shared is prescribed today, but strictly modeling that workflow is
uninteresting, because it's unsafe by definition. I'm not losing that, but
I'm trying to introduce a safe workflow that exists in complement, and my
model works.
I don't even know if we have a mutex defined in our codebase, we don't use
them... but we can max out a 64core thread ripper.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/19/2018 11:18 PM, Manu wrote:
 The reason I ask is because, by my definition, if you have:
 int* a;
 shared(int)* b = a;
 
 While you have 2 numbers that address the same data, it is not actually
aliased 
 because only `a` can access it.

They are aliased, by code that believes it is unshared, and code that believes 
it is shared. This is not going to work.


 Exclusively distinguishing shared and unshared data is not an interesting 
 distinction if shared data has no access.

Somehow, you still have to find a way to give the shared path access, through a 
gate or a cast or a lock or whatever. And then it breaks, because two different 
threads are accessing the same data each thinking that data is not shared.

aliak <something something.com> writes:

On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright wrote:
 Somehow, you still have to find a way to give the shared path 
 access, through a gate or a cast or a lock or whatever. And 
 then it breaks, because two different threads are accessing the 
 same data each thinking that data is not shared.

When you say that, then under Manu's proposal and the code below:

class C {
   void f();
   void g() shared;
}

void t1(shared C c) {
   c.g; // ok
   c.f; // error
}

void t2(shared C c) {
   c.g; // ok
   c.f; // error
}

auto c = new C();
spawn(&t1, c);
spawn(&t2, c);
c.f; // ok
c.g; // ok

Do you mean the implementation of C.g? Since that is shared 
wouldn't that just be a normal understanding that you'd need to 
synchronize the data access in shared since it's a shared method? 
And if you mean C.f, then if that accessed data (that was 
accessed by C.g) unsafely, then that's just a bad implementation 
no? Or?

Cheers,
- Ali

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Saturday, 20 October 2018 at 16:18:53 UTC, aliak wrote:

 class C {
   void f();
   void g() shared;
 }

 void t1(shared C c) {
   c.g; // ok
   c.f; // error
 }

 void t2(shared C c) {
   c.g; // ok
   c.f; // error
 }

 auto c = new C();
 spawn(&t1, c);
 spawn(&t2, c);
 c.f; // ok
 c.g; // ok

Those are not "ok". They're only "ok" under Manu's proposal so 
long as the author of C promises (via documentation) that that's 
indeed "ok". There can be no statically-enforced guarantees that 
those calls are "ok", or that issuing them in that order is "ok". 
Yet Manu keeps insisting that somehow there is.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Saturday, 20 October 2018 at 16:41:41 UTC, Stanislav Blinov 
wrote:
 On Saturday, 20 October 2018 at 16:18:53 UTC, aliak wrote:

 class C {
   void f();
   void g() shared;
 }

 void t1(shared C c) {
   c.g; // ok
   c.f; // error
 }

 void t2(shared C c) {
   c.g; // ok
   c.f; // error
 }

 auto c = new C();
 spawn(&t1, c); // line 20
 spawn(&t2, c); // line 21
 c.f; // ok
 c.g; // ok // line 23

 Those are not "ok". They're only "ok" under Manu's proposal so 
 long as the author of C promises (via documentation) that 
 that's indeed "ok". There can be no statically-enforced 
 guarantees that those calls are "ok", or that issuing them in 
 that order is "ok". Yet Manu keeps insisting that somehow there 
 is.

Backing up a bit and making a few observations (after adding 
imports and wrapping the bottom code in a function):

1. the code above currently does not compile, error messages are:
     i.  line 20 & 21: spawn fails to instantiate because c is not 
shared
     ii. line 23: shared method C.g is not callable using a 
non-shared object
     iii. the lines already marked // error

2. in order to fix 1.i, one must cast c to shared at the call 
site, this is not  safe

3 fixing 1.ii requires doing `(cast(shared)c).g`, this is also 
not  safe

4 fixing 1.iii fixing requires casting away shared, this is not 
only not  safe, but also wrong. c is a class so one could try 
locking it although I'm not sure what the implications are for 
doing that when another thread owns the data, probably bad.

5 the current means of dealing with shared with lock and cast 
away shared is also not  safe

6 under Manu's proposal reading and writing shared objects 
results in compilation error

7 The static guarantees we have in the language are type safety 
and  safe

8 under Manu's proposal to do anything one must call shared 
functions on said object, this implies a " trusted" 
implementation at the bottom of the stack for ensuring thread 
safety (atomics and lock + cast (assuming it is not wrong), other 
sync primitives) that are not  safe, but not outright wrong 
either.

The question then becomes: assuming the implementation _is_  safe 
type correct and thread safe etc., can the author of C provide 
guarantees of  safe and type correctness? and can this guarantee 
be free of false positives?

Currently the answer is no: the requirement to cast to and from 
shared is un- safe and that burden is on the user which means 
that they must understand the inner workings of C to know it that 
is the case.

Manu's proposal is slightly more interesting. shared becomes a 
guarantee that accesses to that object will not race, assuming 
that the  trusted implementation at the bottom of the stack are 
correct. In the above if t1 and t2 took `const shared C` and `g` 
was also const shared, then I think that it could.

Manu <turkeyman gmail.com> writes:

On Sat, Oct 20, 2018 at 9:45 AM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 20 October 2018 at 16:18:53 UTC, aliak wrote:

 class C {
   void f();
   void g() shared;
 }

 void t1(shared C c) {
   c.g; // ok
   c.f; // error
 }

 void t2(shared C c) {
   c.g; // ok
   c.f; // error
 }

 auto c = new C();
 spawn(&t1, c);
 spawn(&t2, c);
 c.f; // ok
 c.g; // ok

 Those are not "ok". They're only "ok" under Manu's proposal so
 long as the author of C promises (via documentation) that that's
 indeed "ok". There can be no statically-enforced guarantees that
 those calls are "ok", or that issuing them in that order is "ok".
 Yet Manu keeps insisting that somehow there is.

I only insist that if you write a shared method, you promise that it
is threadsafe.
If f() undermines g() threadsafety, then **g() is NOT threadsafe**,
and you just write an invalid program.

You can write an invalid program in any imaginable number of ways;
that's just not an interesting discussion. An interesting discussion
is what we might to to help prevent writing such an invalid program...
I don't suggest here what we can do to statically encorce this, but I
suspect there does exist *some* options which may help, which can be
further developments.

What I also assert is that *this unsafe code is rare*... it exists
only at the bottom of the tooling stack, and anyone else using a
shared object will not do unsafe, and therefore will not be able to
create the problem. If you do unsafety anywhere near `shared`, you
should feel nervous. I'm trying to make a world where you aren't
*required* to do unsafety at every single interaction.

Understand: f() can only undermine g() promise of threadsafety **if
f() is not  safe**. Users won't create this situation accidentally,
they can only do it deliberately.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Saturday, 20 October 2018 at 18:30:59 UTC, Manu wrote:
 On Sat, Oct 20, 2018 at 9:45 AM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Saturday, 20 October 2018 at 16:18:53 UTC, aliak wrote:

 class C {
   void f();
   void g() shared;
 }

 Those are not "ok". They're only "ok" under Manu's proposal so 
 long as the author of C promises (via documentation) that 
 that's indeed "ok". There can be no statically-enforced 
 guarantees that those calls are "ok", or that issuing them in 
 that order is "ok". Yet Manu keeps insisting that somehow 
 there is.

 I only insist that if you write a shared method, you promise 
 that it is threadsafe.
 If f() undermines g() threadsafety, then **g() is NOT 
 threadsafe**, and you just write an invalid program.

 You can write an invalid program in any imaginable number of 
 ways; that's just not an interesting discussion. An interesting 
 discussion is what we might to to help prevent writing such an 
 invalid program... I don't suggest here what we can do to 
 statically encorce this, but I suspect there does exist *some* 
 options which may help, which can be further developments.

 What I also assert is that *this unsafe code is rare*... it 
 exists only at the bottom of the tooling stack, and anyone else 
 using a shared object will not do unsafe, and therefore will 
 not be able to create the problem. If you do unsafety anywhere 
 near `shared`, you should feel nervous. I'm trying to make a 
 world where you aren't *required* to do unsafety at every 
 single interaction.

 Understand: f() can only undermine g() promise of threadsafety 
 **if f() is not  safe**. Users won't create this situation 
 accidentally, they can only do it deliberately.

---

module expertcode;

 safe:

struct FileHandle {
      safe:

     void[] read(void[] storage) shared;
     void[] write(const(void)[] buffer) shared;
}

FileHandle openFile(string path);
// only the owner can close
void closeFile(ref FileHandle);

void shareWithThreads(shared FileHandle*); // i.e. generate a 
number of jobs in some queue
void waitForThreads();                     // waits until all 
processing is done

module usercode;

import expertcode;

void processHugeFile(string path) {
     FileHandle file = openFile(path);
     shareWithThreads(&file);    // implicit cast
     waitForThreads();
     file.closeFile();
}

---

Per your proposal, everything in 'expertcode' can be written 
 safe, i.e. not violating any of the points that  safe forbids, 
or doing so only in a  trusted manner. As far as the language is 
concerned, this would mean that processHugeFile can be  safe as 
well.

Remove the call to `waitForThreads()` (assume user just forgot 
that, i.e. the "accident"). Nothing would change for the 
compiler: all calls remain  safe. And yet, if we're lucky, we get 
a consistent instacrash. If we're unlucky, we get memory 
corruption, or an unsolicited write to another currently open 
file, either of which can go unnoticed for some time.

Of course the program becomes invalid if you do that, there's no 
question about it, this goes for all buggy code. The problem is, 
definition of "valid" lies beyond the type system: it's an 
agreement between different parts of code, i.e. between expert 
programmers who wrote FileHandle et al., and users who write 
processHugeFile(). The main issue is that certain *runtime* 
conditions can still violate  safe-ty.

Your proposal makes the language more strict wrt. to writing 
 safe 'expertmodule', thanks to disallowing reads and writes 
through `shared`, which is great.
However the implicit conversion to `shared` doesn't in any way 
improve the situation as far as user code is concerned, unless 
I'm still missing something.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/20/2018 11:30 AM, Manu wrote:
 You can write an invalid program in any imaginable number of ways;
 that's just not an interesting discussion.

What we're discussing is not an invalid program, but what guarantees the type 
system can provide.

D's current type system guarantees that a T* and a shared(T)* do not point to 
the same memory location in  safe code.

To get them to point to the same memory location, you've got to dip into
 system 
code, where *you* become responsible for maintaining the guarantees.

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Sunday, 21 October 2018 at 09:04:34 UTC, Walter Bright wrote:
 On 10/20/2018 11:30 AM, Manu wrote:
 You can write an invalid program in any imaginable number of 
 ways;
 that's just not an interesting discussion.

 What we're discussing is not an invalid program, but what 
 guarantees the type system can provide.

 D's current type system guarantees that a T* and a shared(T)* 
 do not point to the same memory location in  safe code.

 To get them to point to the same memory location, you've got to 
 dip into  system code, where *you* become responsible for 
 maintaining the guarantees.

The only difference between this and Manu's proposal is when you 
need to dip into  system code - in MP it's perfectly fine for the 
pointers to be equal, but when you want to read from or write to 
the address, you'll need to use  system. In other words, the dip 
into  system happens deeper in the codebase, meaning more code 
can be  safe.

--
   Simen

Manu <turkeyman gmail.com> writes:

On Sun., 21 Oct. 2018, 2:05 am Walter Bright via Digitalmars-d, <
digitalmars-d puremagic.com> wrote:

 On 10/20/2018 11:30 AM, Manu wrote:
 You can write an invalid program in any imaginable number of ways;
 that's just not an interesting discussion.

 What we're discussing is not an invalid program, but what guarantees the
 type
 system can provide.

 D's current type system guarantees that a T* and a shared(T)* do not point
 to
 the same memory location in  safe code.

My proposal guarantees that too, but in a more interesting way, because it
opens the door to a whole working model. And it's totally  safe.

To get them to point to the same memory location, you've got to dip into
  system
 code, where *you* become responsible for maintaining the guarantees.

My model preserves that property. Why do you think I'm running that static
guarantee?

It's all irrelevant if you don't express any mechanism to *do* anything.
Shared today does not have any use. It simply expresses that data *is*
shared, and says nothing about what you can do with it.
If you don't express a safe mechanism for interacting with shared data,
then simply expressing the distinction of shared data really is completely
uninteresting.
It's just a marker that's mixed up in a bunch of unsafe code. I'm no more
satisfied than I am with C++.

Shared needs to do something; I propose that it strictly models operations
that are threadsafe and semantic restrictions required to support that, and
then you have a *usage* scheme, which is safe, and API conveys proper
interaction.. not just an uninteresting marker.

I'm genuinely amazed that you're not intrigued by a  safe shared
proposition. Nobly likes  safe more than you.

I could run our entire SMP stack 100%  safe.

I am going to fork D with this feature one way or another. It's the most
meaningful and compelling opportunity I've seen in ever. If there's ever
been a single thing that could truly move a bunch of C++ programmers, this
is it. C++ can do a crappy job of modelling most stuff in D, but it simply
can't go anywhere near this, and I've been working on competing C++ models
for months.
SMP is the future, we're going all-in this generation. Almost every
function in our codebase runs in an SMP environment... And I was staggered
that I was able to work this definition through to such a simple and
elegant set of rules.
I can't get my head around why people aren't more excited about this...
fully  safe SMP is huge!

rikki cattermole <rikki cattermole.co.nz> writes:

On 21/10/2018 10:41 PM, Manu wrote:
 On Sun., 21 Oct. 2018, 2:05 am Walter Bright via Digitalmars-d, 
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:
 
     On 10/20/2018 11:30 AM, Manu wrote:
      > You can write an invalid program in any imaginable number of ways;
      > that's just not an interesting discussion.
 
     What we're discussing is not an invalid program, but what guarantees
     the type
     system can provide.
 
     D's current type system guarantees that a T* and a shared(T)* do not
     point to
     the same memory location in  safe code.
 
 
 My proposal guarantees that too, but in a more interesting way, because 
 it opens the door to a whole working model. And it's totally  safe.
 
     To get them to point to the same memory location, you've got to dip
     into  system
     code, where *you* become responsible for maintaining the guarantees.
 
 
 My model preserves that property. Why do you think I'm running that 
 static guarantee?
 
 It's all irrelevant if you don't express any mechanism to *do* anything. 
 Shared today does not have any use. It simply expresses that data *is* 
 shared, and says nothing about what you can do with it.
 If you don't express a safe mechanism for interacting with shared data, 
 then simply expressing the distinction of shared data really is 
 completely uninteresting.
 It's just a marker that's mixed up in a bunch of unsafe code. I'm no 
 more satisfied than I am with C++.
 
 Shared needs to do something; I propose that it strictly models 
 operations that are threadsafe and semantic restrictions required to 
 support that, and then you have a *usage* scheme, which is safe, and API 
 conveys proper interaction.. not just an uninteresting marker.
 
 I'm genuinely amazed that you're not intrigued by a  safe shared 
 proposition. Nobly likes  safe more than you.
 
 I could run our entire SMP stack 100%  safe.
 
 I am going to fork D with this feature one way or another. It's the most 
 meaningful and compelling opportunity I've seen in ever. If there's ever 
 been a single thing that could truly move a bunch of C++ programmers, 
 this is it. C++ can do a crappy job of modelling most stuff in D, but it 
 simply can't go anywhere near this, and I've been working on competing 
 C++ models for months.
 SMP is the future, we're going all-in this generation. Almost every 
 function in our codebase runs in an SMP environment... And I was 
 staggered that I was able to work this definition through to such a 
 simple and elegant set of rules.
 I can't get my head around why people aren't more excited about this... 
 fully  safe SMP is huge!

I'm excited, but you need to write a DIP even if preliminary which shows 
both new semantics but also shows both working and current code to 
compare them.

aliak <something something.com> writes:

On Saturday, 20 October 2018 at 16:41:41 UTC, Stanislav Blinov 
wrote:
 Those are not "ok". They're only "ok" under Manu's proposal so 
 long as the author of C promises (via documentation) that 
 that's indeed "ok". There can be no statically-enforced 
 guarantees that those calls are "ok", or that issuing them in 
 that order is "ok". Yet Manu keeps insisting that somehow there 
 is.

No he is not insisting you can statically enforce thread safety.

When I say ok, I mean assuming the implementer actually wrote 
correct code. This applies to any shared method today as well.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Sunday, 21 October 2018 at 11:25:16 UTC, aliak wrote:
 On Saturday, 20 October 2018 at 16:41:41 UTC, Stanislav Blinov 
 wrote:
 Those are not "ok". They're only "ok" under Manu's proposal so 
 long as the author of C promises (via documentation) that 
 that's indeed "ok". There can be no statically-enforced 
 guarantees that those calls are "ok", or that issuing them in 
 that order is "ok". Yet Manu keeps insisting that somehow 
 there is.

 No he is not insisting you can statically enforce thread safety.

I stand corrected, it would seem so.

 When I say ok, I mean assuming the implementer actually wrote 
 correct code. This applies to any shared method today as well.

This ("ok") can only be achieved if the "implementor" (the 
"expert") writes every function self-contained, at which point 
sharing something from user code becomes a non-issue (i.e. it 
becomes unnecessary). But that's not a very useful API. As soon 
as you have more than one function operating on the same data, 
the onus is on the user (the caller) to call those functions in 
correct order, or, more generally, without invalidating the state 
of shared data.

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Sunday, 21 October 2018 at 13:24:49 UTC, Stanislav Blinov 
wrote:
 On Sunday, 21 October 2018 at 11:25:16 UTC, aliak wrote:
 When I say ok, I mean assuming the implementer actually wrote 
 correct code. This applies to any shared method today as well.

 This ("ok") can only be achieved if the "implementor" (the 
 "expert") writes every function self-contained, at which point 
 sharing something from user code becomes a non-issue (i.e. it 
 becomes unnecessary). But that's not a very useful API. As soon 
 as you have more than one function operating on the same data, 
 the onus is on the user (the caller) to call those functions in 
 correct order, or, more generally, without invalidating the 
 state of shared data.

The onus is *always* on the user to write function calls in the 
correct order, multi-threading or not. We expect programmers to 
be able to figure out why this doesn't print 'Hello world!':

void main() {
     import std.stdio;
     string hello;
     writeln(hello);
     hello = "Hello world!";
}

We also expect writeln to be written in such a way that it 
doesn't corrupt random data or cause life-threatening situations 
just because hello was uninitialized upon calling writeln, and 
assigned afterwards. We should expect the same of multi-threaded 
programs.

This places the onus of writing thread-safe code on the writer of 
the multi-threaded equivalent of writeln and string.opAssign. 
Only this way can the user of the library write code and not 
expect things to blow up in their face.

--
   Simen

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright wrote:
 On 10/19/2018 11:18 PM, Manu wrote:
 The reason I ask is because, by my definition, if you have:
 int* a;
 shared(int)* b = a;
 
 While you have 2 numbers that address the same data, it is not 
 actually aliased because only `a` can access it.

 They are aliased,

Quoting Wikipedia:

two pointers A and B which have the same value, then the name 
A[0] aliases the name B[0]. In this case we say the pointers A 
and B alias each other. Note that the concept of pointer 
aliasing is not very well-defined – two pointers A and B may or 
may not alias each other, depending on what operations are 
performed in the function using A and B.

In this case given the above: `a[0]` does not alias `b[0]` 
because `b[0]` is ill defined under Manu's proposal, because the 
memory referenced by `a` is not reachable through `b` because you 
can't read or write through `b`.

 by code that believes it is unshared

you cannot ` safe`ly modify the memory  through `b`, `a`'s view 
of the memory is unchanged in  safe code.

 and, code that believes it is shared.

you cannot have non-atomic access though `b`, `b` has no  safe 
view of the memory, unless it is atomic (which by definition is 
synchronised).

This is not going to work.

Aú contraire.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Saturday, 20 October 2018 at 16:48:05 UTC, Nicholas Wilson 
wrote:
 On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright 
 wrote:
 On 10/19/2018 11:18 PM, Manu wrote:
 The reason I ask is because, by my definition, if you have:
 int* a;
 shared(int)* b = a;
 
 While you have 2 numbers that address the same data, it is 
 not actually aliased because only `a` can access it.

 They are aliased,

 Quoting Wikipedia:

two pointers A and B which have the same value, then the name 
A[0] aliases the name B[0]. In this case we say the pointers A 
and B alias each other. Note that the concept of pointer 
aliasing is not very well-defined – two pointers A and B may or 
may not alias each other, depending on what operations are 
performed in the function using A and B.

 In this case given the above: `a[0]` does not alias `b[0]` 
 because `b[0]` is ill defined under Manu's proposal, because 
 the memory referenced by `a` is not reachable through `b` 
 because you can't read or write through `b`.

 by code that believes it is unshared

 you cannot ` safe`ly modify the memory  through `b`, `a`'s view 
 of the memory is unchanged in  safe code.

And that's already a bug, because the language can't enforce 
threadsafe access through `a`, regardless of presence of `b`. 
Only the programmer can.

 and, code that believes it is shared.

 you cannot have non-atomic access though `b`, `b` has no  safe 
 view of the memory, unless it is atomic (which by definition is 
 synchronised).

Synchronized with what? You still have `a`, which isn't `shared` 
and doesn't require any atomic access or synchronization. At this 
point it doesn't matter if it's an int or a struct. As soon as 
you share `a`, you can't just pretend that reading or writing `a` 
is safe. Encapsulate it all you want, safety only remains a 
contract of convention, the language can't enforce it.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Saturday, 20 October 2018 at 17:06:22 UTC, Stanislav Blinov 
wrote:
 On Saturday, 20 October 2018 at 16:48:05 UTC, Nicholas Wilson 
 wrote:
 On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright 
 wrote:
 by code that believes it is unshared

 you cannot ` safe`ly modify the memory  through `b`, `a`'s 
 view of the memory is unchanged in  safe code.

 And that's already a bug, because the language can't enforce 
 threadsafe access through `a`, regardless of presence of `b`. 
 Only the programmer can.

  access through `a` is through the owned reference threadsafety 
through a does't mean anything, all _other_ access must ensure 
that the are ordered correctly.

 and, code that believes it is shared.

 you cannot have non-atomic access though `b`, `b` has no  safe 
 view of the memory, unless it is atomic (which by definition 
 is synchronised).

 Synchronized with what? You still have `a`, which isn't 
 `shared` and doesn't require any atomic access or 
 synchronization.

Synchronized w.r.t any writes to that memory, e.g. from `a`.

At this point it doesn't matter if it's an int
 or a struct.

Yes.

 As soon as you share `a`, you can't just pretend that reading 
 or writing `a` is safe.

You can if no-one else writes to it, which is the whole point of 
Manu's proposal. Perhaps it should be const shared instead of 
shared but still.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/20/2018 11:08 AM, Nicholas Wilson wrote:
 You can if no-one else writes to it, which is the whole point of Manu's 
 proposal. Perhaps it should be const shared instead of shared but still.

There is no purpose whatsoever to data that can be neither read nor written. 
Shared data is only useful if, at some point, it is read/written, presumably by 
casting it to unshared in  trusted code. As soon as that is done, you've got a 
data race with the other existing unshared aliases.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Sunday, 21 October 2018 at 09:58:18 UTC, Walter Bright wrote:
 On 10/20/2018 11:08 AM, Nicholas Wilson wrote:
 You can if no-one else writes to it, which is the whole point 
 of Manu's proposal. Perhaps it should be const shared instead 
 of shared but still.

 There is no purpose whatsoever to data that can be neither read 
 nor written. Shared data is only useful if, at some point, it 
 is read/written, presumably by casting it to unshared in 
  trusted code. As soon as that is done, you've got a data race 
 with the other existing unshared aliases.

Just a thought: if a hard requirement is made on `shared` data to 
be non-copyable, a  safe conversion could be guaranteed. But it 
can't be implicit either:

shared(T) share(T)(T value) if (!is(T == shared) && 
!isCopyable!T) {
     shared(T) result = move(value);
     return result;
}

struct ShareableData {
      disable <postblit and/or copy ctor>; // Generated by 
compiler in presence of `shared` members and/or `shared` methods

     /* ... */
}

void sendToThread(T)(shared T* ptr)  safe;

void usage()  safe {
     int x;
     sendToThread(&x); // Error: 'x' is not shared
     shared y = x;     // Ok
     sendToThread(&y); // Ok


     ShareableData data;
     sendToThread(&data); // Error: 'data' is not shared
     auto p = &data;
     sendToThread(p);     // Error: *p is not shared

     auto sharedData = share(move(data));
     sendToThread(&sharedData); // Ok

     auto yCopy = y;   // Error: cannot copy 'shared' y
     auto dataCopy = sharedData; // Error: cannot copy 'shared' 
sharedData

     ShareableData otherData;
     sendToThread(cast(shared(ShareableData)*) &otherData); // 
Error non- safe cast in  safe code
}

And again, we're back to 'once it's shared, it can't be  safe-ly 
unshared', which ruins the distinction between owned and shared 
references, which is one of the nicer properties that Manu's 
proposal seems to want to achieve :(

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Sunday, 21 October 2018 at 09:58:18 UTC, Walter Bright wrote:
 On 10/20/2018 11:08 AM, Nicholas Wilson wrote:
 You can if no-one else writes to it, which is the whole point 
 of Manu's proposal. Perhaps it should be const shared instead 
 of shared but still.

 There is no purpose whatsoever to data that can be neither read 
 nor written. Shared data is only useful if, at some point, it 
 is read/written, presumably by casting it to unshared in 
  trusted code. As soon as that is done, you've got a data race 
 with the other existing unshared aliases.

No, because every part of the public interface has to work 
together to ensure thread-safety.

This code is invalid (but compiles) under MP:

module A;
struct S {
     private int n;
     void foo()  safe {
         n--; // Not thread-safe
     }
     void bar() shared  trusted {
         atomicOp!"++"(n.assumeUnshared);
     }
}

module B;
import A;
void passToOtherThread(shared(S)*); // Calls S.bar()

void main() {
     S* s = new S();
     passToOtherThread(s);
     s.foo();
}

The reason: foo() breaks bar()s promise of thread-safety. This 
means that S does not provide a thread-safe interface. It would 
be nice if the compiler could statically notice that, but I don't 
see how that'd work.

Now, for a thread-safe version:

module A;
struct S {
     int n;
     void foo()  trusted {
         atomicOp!"--"(n); // Thread-safe
     }
     void bar() shared  trusted {
         atomicOp!"++"(n.assumeUnshared);
     }
}

module B;
import A;
void passToOtherThread(shared(S)*); // Calls S.bar()

void main() {
     S* s = new S();
     passToOtherThread(s);
     s.foo();
}

In this case, passToOtherThread is free to call S.bar as often as 
it may feel like, since atomic operations are used in every 
possible access to S.n. This is true even though one thread has 
unshared access and other threads have shared access.

--
   Simen

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Sunday, 21 October 2018 at 09:58:18 UTC, Walter Bright wrote:
 On 10/20/2018 11:08 AM, Nicholas Wilson wrote:
 You can if no-one else writes to it, which is the whole point 
 of Manu's proposal. Perhaps it should be const shared instead 
 of shared but still.

 There is no purpose whatsoever to data that can be neither read 
 nor written.

Indeed but there is a subtle difference between that and Manu's 
proposal: access through the shared variable may not have 
non-atomic reads, not no reads.

 Shared data is only useful if, at some point, it is 
 read/written,

Yes

 presumably by casting it to unshared in  trusted code.

That is one way to do it, others include atomics  and other 
 trusted primitives

 As soon as that is done, you've got a data race with the other 
 existing unshared aliases.

You're in  trusted code, that is the whole point. The onus is on 
the programmer to make that correct, same with regular 
 safe/ trusted system code.

Timon Gehr <timon.gehr gmx.ch> writes:

On 21.10.18 17:54, Nicholas Wilson wrote:
 
 As soon as that is done, you've got a data race with the other 
 existing unshared aliases.

 
 You're in  trusted code, that is the whole point. The onus is on the 
 programmer to make that correct, same with regular  safe/ trusted system 
 code.

Not all of the parties that participate in the data race are in  trusted 
code. The point of  trusted is modularity: you manually check  trusted 
code according to some set of restrictions and then you are sure that 
there is no memory corruption.

Note that you are not allowed to look at any of the  safe code while 
checking your  trusted code. You will only see an opaque interface to 
the  safe code that you call and all you know is that all the  safe code 
type checks according to  safe rules. Note that there might be an 
arbitrary number of  safe functions and methods that you do not see.

Think about it this way: you first write all the  trusted and  system 
code, and some evil guy who does not like you comes in after you looks 
at your code and writes all the  safe code. If there is any memory 
corruption, it will be your fault and you will face harsh consequences. 
Now, design the  safe type checking rules. It won't be MP!

Note that there may well be a good way to get the good properties of MP 
without breaking the type system, but MP itself is not good because it 
breaks  safe.

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 21.10.18 17:54, Nicholas Wilson wrote:
 As soon as that is done, you've got a data race with the other
 existing unshared aliases.

 You're in  trusted code, that is the whole point. The onus is on the
 programmer to make that correct, same with regular  safe/ trusted system
 code.

 Not all of the parties that participate in the data race are in  trusted
 code. The point of  trusted is modularity: you manually check  trusted
 code according to some set of restrictions and then you are sure that
 there is no memory corruption.

 Note that you are not allowed to look at any of the  safe code while
 checking your  trusted code. You will only see an opaque interface to
 the  safe code that you call and all you know is that all the  safe code
 type checks according to  safe rules. Note that there might be an
 arbitrary number of  safe functions and methods that you do not see.

 Think about it this way: you first write all the  trusted and  system
 code, and some evil guy who does not like you comes in after you looks
 at your code and writes all the  safe code. If there is any memory
 corruption, it will be your fault and you will face harsh consequences.
 Now, design the  safe type checking rules. It won't be MP!

 Note that there may well be a good way to get the good properties of MP
 without breaking the type system, but MP itself is not good because it
 breaks  safe.

Show me. Nobody has been able to show that yet. I'd really like to know this.

Neia Neutuladh <neia ikeran.org> writes:

On Sun, 21 Oct 2018 12:04:16 -0700, Manu wrote:
 On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Note that there may well be a good way to get the good properties of MP
 without breaking the type system, but MP itself is not good because it
 breaks  safe.

 
 Show me. Nobody has been able to show that yet. I'd really like to know
 this.

If we only used your proposal and only used  safe code, we wouldn't have 
any data races, but that's only because we wouldn't have any shared data. 
We'd have shared *variables*, but they would not contain any data we could 
read or alter, and that's pretty much useless.

To use your proposal, we need to cast data back from shared to unshared. 
When it's unshared, we need to make sure that exactly one thread has a 
reference to that data as unshared. And  safe *should* help us with that. 
Currently, it helps because casting unshared to shared is not  safe, 
because it makes it trivial to get multiple threads with unshared 
references to the same data. And that's when you're using shared as 
expected rather than doing something weird.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Sunday, 21 October 2018 at 22:12:18 UTC, Neia Neutuladh wrote:
 On Sun, 21 Oct 2018 12:04:16 -0700, Manu wrote:
 On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 Note that there may well be a good way to get the good 
 properties of MP without breaking the type system, but MP 
 itself is not good because it breaks  safe.

 
 Show me. Nobody has been able to show that yet. I'd really 
 like to know this.

 If we only used your proposal and only used  safe code, we 
 wouldn't have any data races,

Only of the  trusted implementation is thread safe, which it 
_should_ be. This is the same caveat as non threaded 
- safe/ tusted code.

 but that's only because we wouldn't have any shared data. We'd 
 have shared *variables*, but they would not contain any data we 
 could read

Reads must be atomic.

 or alter, and that's pretty much useless.

 To use your proposal, we need to cast data back from shared to 
 unshared.

Yes but this is in the  trusted implementation that forms the 
basis of your threadsafety.

 When it's unshared, we need to make sure that exactly one 
 thread has a reference to that data as unshared.

Nod.

And  safe *should* help us with that.

Nod.

 Currently, it helps because casting unshared to shared is not 
  safe,

This remains the case, and should be done (enforced by the 
compiler) only in  trusted/ system code as a basis for thread 
safe,  safe code.

 because it makes it trivial to get multiple threads with 
 unshared references to the same data.

That is  trusted or  system code and therefore is the programmers 
responsibility.

 And that's when you're using shared as expected rather than 
 doing something weird.

That forms the basis of your thread safe stack. From there on, 
the basis that shared arguments to functions are treated safely 
in the presence of threading means that code that calls the 
 trusted implementations is  safe.

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 3:15 PM Neia Neutuladh via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sun, 21 Oct 2018 12:04:16 -0700, Manu wrote:
 On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Note that there may well be a good way to get the good properties of MP
 without breaking the type system, but MP itself is not good because it
 breaks  safe.

 Show me. Nobody has been able to show that yet. I'd really like to know
 this.

 If we only used your proposal and only used  safe code, we wouldn't have
 any data races, but that's only because we wouldn't have any shared data.
 We'd have shared *variables*, but they would not contain any data we could
 read or alter, and that's pretty much useless.

I've shown an implementation of Atomic(T) 3 times now... no response
any time. Why is it being dismissed? Do I need to write it more times?
This is a clear demonstration of how to build the foundation of the
 safe threadsafe stack.

 To use your proposal, we need to cast data back from shared to unshared.
 When it's unshared, we need to make sure that exactly one thread has a
 reference to that data as unshared.

No, you just need to make sure that access is atomic or synchronised
in a proper way, and if you do cast away shared in some low-level
 trusted function, make sure that reference doesn't escape.
You can do it, I have faith.

 And  safe *should* help us with that.

Totally.

 Currently, it helps because casting unshared to shared is not  safe,
 because it makes it trivial to get multiple threads with unshared
 references to the same data.

No no, that's a massive smell. That means anytime anyone wants to
distribute something, they need to perform unsafe casts. That's not
okay.
Modeling shared-ness/unshared-ness is not *useful* in any way that I
have been able to identify. Modelling what it means to be threadsafe
is useful in every application I've ever written.
100% of my SMP code works with my proposal, and something close to 0%
works with shared as it is today. (assuming we desire  safe
interaction, which we do, because threading is hard enough already!)

 And that's when you're using shared as
 expected rather than doing something weird.

No, I *expect* to use shared in  safe code, and not write any unsafe
code ever. shared doesn't model a useful interaction now, not in any
way.

Today, access to shared data members are unrestricted and completely
unsafe, passing data into something like a parallel-for requires
unsafe casts.

Neia Neutuladh <neia ikeran.org> writes:

On Sun, 21 Oct 2018 17:35:38 -0700, Manu wrote:

 On Sun, Oct 21, 2018 at 3:15 PM Neia Neutuladh via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 If we only used your proposal and only used  safe code, we wouldn't
 have any data races, but that's only because we wouldn't have any
 shared data. We'd have shared *variables*, but they would not contain
 any data we could read or alter, and that's pretty much useless.

 
 I've shown an implementation of Atomic(T) 3 times now... no response any
 time. Why is it being dismissed? Do I need to write it more times?
 This is a clear demonstration of how to build the foundation of the
  safe threadsafe stack.

Yes, Atomic can be implemented here as  trusted code, and maybe using 
 safe compiler intrinsics for some situations. That's useful! But it's not 
*enough*.

It would require a lot of work to refit existing code to using only atomic 
operations, and it would end up looking rather clunky a lot of the time. 
If you're doing anything complex with a complex object graph, you're going 
to have a terrible time. You need to copy that object graph (atomically), 
which is going to be expensive and provides non-trivial restrictions on 
what you can store.

So I'm not keen on a world in which all multithreading is using atomic 
structs.

Unless, when you were talking about Atomic, you actually meant a wrapper 
containing some sort of lock, allowing you to submit arbitrary delegates 
to mutate the data within in a serialized way, similar to Atila Neves's 
fearless library. Which really stretches the definition of "atomic".

 Currently, it helps because casting unshared to shared is not  safe,
 because it makes it trivial to get multiple threads with unshared
 references to the same data.

 
 No no, that's a massive smell. That means anytime anyone wants to
 distribute something, they need to perform unsafe casts. That's not
 okay.

Casting thread-local to shared makes it easy to cause errors, and that's 
why it's a massive smell. Making it silent doesn't eliminate the smell.

 100% of my SMP code works with my proposal, and something close to 0%
 works with shared as it is today. (assuming we desire  safe interaction,
 which we do, because threading is hard enough already!)

You want un-shared things to implicitly cast to shared. You don't want to 
have to allocate anything as shared, and you do want to pass absolutely 
anything to any thread.

You can write a  trusted assumeShared function, analogous to assumeUnique, 
to accomplish that. It would be slightly more awkward than what you're 
proposing, but it accomplishes one of your two goals: convert non-shared 
things to shared things in  safe code.

The other goal in your proposal is for some code that currently compiles 
not to. So you should be able to write the code you want, albeit with an 
increased risk of bugs. Or you could write a template that wraps a shared 
thing and forbids field access and assignment.

 And that's when you're using shared as expected rather than doing
 something weird.

 
 No, I *expect* to use shared in  safe code, and not write any unsafe
 code ever. shared doesn't model a useful interaction now, not in any
 way.
 
 Today, access to shared data members are unrestricted and completely
 unsafe

Yes, and that's bad and should be changed.

 passing data into something like a parallel-for requires unsafe
 casts.

Or allocating data as shared, which is the recommended way, because that 
makes absolutely certain that, from the start, no code has an un-shared 
copy of that data. No casts needed.

Timon Gehr <timon.gehr gmx.ch> writes:

On 21.10.18 21:04, Manu wrote:
 On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 21.10.18 17:54, Nicholas Wilson wrote:
 As soon as that is done, you've got a data race with the other
 existing unshared aliases.

 You're in  trusted code, that is the whole point. The onus is on the
 programmer to make that correct, same with regular  safe/ trusted system
 code.

 Not all of the parties that participate in the data race are in  trusted
 code. The point of  trusted is modularity: you manually check  trusted
 code according to some set of restrictions and then you are sure that
 there is no memory corruption.

 Note that you are not allowed to look at any of the  safe code while
 checking your  trusted code. You will only see an opaque interface to
 the  safe code that you call and all you know is that all the  safe code
 type checks according to  safe rules. Note that there might be an
 arbitrary number of  safe functions and methods that you do not see.

 Think about it this way: you first write all the  trusted and  system
 code, and some evil guy who does not like you comes in after you looks
 at your code and writes all the  safe code. If there is any memory
 corruption, it will be your fault and you will face harsh consequences.
 Now, design the  safe type checking rules. It won't be MP!

 Note that there may well be a good way to get the good properties of MP
 without breaking the type system, but MP itself is not good because it
 breaks  safe.

 
 Show me. Nobody has been able to show that yet. I'd really like to know this.
 

I just did, but if you really need to, give me a non-trivial piece of 
correct multithreaded code that accesses some declared-unshared field 
from a shared method and I will show you how the evil guy would modify 
some  safe code in it and introduce race conditions. It needs to be your 
code, as otherwise you will just claim again that it is me who wrote bad 
 trusted code.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Monday, 22 October 2018 at 00:38:33 UTC, Timon Gehr wrote:
 I just did,

Link please?

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.10.18 02:46, Nicholas Wilson wrote:
 On Monday, 22 October 2018 at 00:38:33 UTC, Timon Gehr wrote:
 I just did,

 
 Link please?
 

https://forum.dlang.org/post/pqii8k$11u3$1 digitalmars.com

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Monday, 22 October 2018 at 00:55:00 UTC, Timon Gehr wrote:
 On 22.10.18 02:46, Nicholas Wilson wrote:
 On Monday, 22 October 2018 at 00:38:33 UTC, Timon Gehr wrote:
 I just did,

 
 Link please?
 

 https://forum.dlang.org/post/pqii8k$11u3$1 digitalmars.com

That contains no code.

Not all of the parties that participate in the data race are in 
 trusted code.

There are two cases of  trusted code, bad and good. Bad trusted 
code can indeed be misused by  safe code to corrupt memory. Good 
trusted code cannot.

What part of the proposal breaks the type system? The implicit 
conversion to shared implies that the passed to function is  safe 
iff all the functions it calls are  safe only call function with 
that parameter to other shared  safe functions _OR_ they are 
 trusted.

 The point of  trusted is modularity: you manually check 
  trusted code according to some set of restrictions and then 
 you are sure that there is no memory corruption.

Yes. And?

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 5:40 PM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 21.10.18 21:04, Manu wrote:
 On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 21.10.18 17:54, Nicholas Wilson wrote:
 As soon as that is done, you've got a data race with the other
 existing unshared aliases.

 You're in  trusted code, that is the whole point. The onus is on the
 programmer to make that correct, same with regular  safe/ trusted system
 code.

 Not all of the parties that participate in the data race are in  trusted
 code. The point of  trusted is modularity: you manually check  trusted
 code according to some set of restrictions and then you are sure that
 there is no memory corruption.

 Note that you are not allowed to look at any of the  safe code while
 checking your  trusted code. You will only see an opaque interface to
 the  safe code that you call and all you know is that all the  safe code
 type checks according to  safe rules. Note that there might be an
 arbitrary number of  safe functions and methods that you do not see.

 Think about it this way: you first write all the  trusted and  system
 code, and some evil guy who does not like you comes in after you looks
 at your code and writes all the  safe code. If there is any memory
 corruption, it will be your fault and you will face harsh consequences.
 Now, design the  safe type checking rules. It won't be MP!

 Note that there may well be a good way to get the good properties of MP
 without breaking the type system, but MP itself is not good because it
 breaks  safe.

 Show me. Nobody has been able to show that yet. I'd really like to know this.

 I just did,

There's no code there... just a presumption that the person who wrote
the  trusted code did not deliver the promise they made.

 but if you really need to, give me a non-trivial piece of> correct
multithreaded code that accesses some declared-unshared field
 from a shared method and I will show you how the evil guy would modify
 some  safe code in it and introduce race conditions. It needs to be your
 code, as otherwise you will just claim again that it is me who wrote bad
  trusted code.

You can pick on any of my prior code fragments. They've all been ignored so far.

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.10.18 02:54, Manu wrote:
 On Sun, Oct 21, 2018 at 5:40 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 21.10.18 21:04, Manu wrote:
 On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 21.10.18 17:54, Nicholas Wilson wrote:
 As soon as that is done, you've got a data race with the other
 existing unshared aliases.

 You're in  trusted code, that is the whole point. The onus is on the
 programmer to make that correct, same with regular  safe/ trusted system
 code.

 Not all of the parties that participate in the data race are in  trusted
 code. The point of  trusted is modularity: you manually check  trusted
 code according to some set of restrictions and then you are sure that
 there is no memory corruption.

 Note that you are not allowed to look at any of the  safe code while
 checking your  trusted code. You will only see an opaque interface to
 the  safe code that you call and all you know is that all the  safe code
 type checks according to  safe rules. Note that there might be an
 arbitrary number of  safe functions and methods that you do not see.

 Think about it this way: you first write all the  trusted and  system
 code, and some evil guy who does not like you comes in after you looks
 at your code and writes all the  safe code. If there is any memory
 corruption, it will be your fault and you will face harsh consequences.
 Now, design the  safe type checking rules. It won't be MP!

 Note that there may well be a good way to get the good properties of MP
 without breaking the type system, but MP itself is not good because it
 breaks  safe.

 Show me. Nobody has been able to show that yet. I'd really like to know this.

 I just did,

 
 There's no code there... just a presumption that the person who wrote
 the  trusted code did not deliver the promise they made.
 ...

Yes, because there is no way to write  trusted code that holds its 
promise while actually doing something interesting in multiple threads 
if  safe code can implicitly convert from unshared to shared.

 but if you really need to, give me a non-trivial piece of> correct
multithreaded code that accesses some declared-unshared field
 from a shared method and I will show you how the evil guy would modify
 some  safe code in it and introduce race conditions. It needs to be your
 code, as otherwise you will just claim again that it is me who wrote bad
  trusted code.

 
 You can pick on any of my prior code fragments. They've all been ignored so
far.
 

I don't want "code fragments". Show me the real code.

I manually browsed through posts now (thanks a lot) and found this 
implementation:

struct Atomic(T){
   void opUnary(string op : "++")() shared { atomicIncrement(&val); }
   private T val;
}

This is  system code. There is no  safe or  trusted here, so I am 
ignoring it.


Then I browsed some more, because I had nothing better to do, and I 
found this. I completed it so that it is actually compilable, except for 
the unsafe implicit conversion.

Please read this code, and then carefully read the comments below it 
before you respond. I will totally ignore any of your answers that 
arrive in the next two hours.

---
module borked;

void atomicIncrement(int* p) system{
     import core.atomic;
     atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
}

struct Atomic(T){
     private T val;
     void opUnary(string op : "++")() shared  trusted {
         atomicIncrement(cast(T*)&val);
     }
}
void main() safe{
     Atomic!int i;
     auto a=&[i][0];// was: Atomic!int* a = &i;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++i.val; // race
}
---


Oh no! The author of the  trusted function (i.e. you) did not deliver on 
the promise they made!

Now, before you go and tell me that I am stupid because I wrote bad 
code, consider the following:

- It is perfectly  safe to access private members from the same module.

- You may not blame the my  safe main function for the problem. It is 
 safe, so it cannot be blamed for UB. Any UB is the result of a bad 
 trusted function, a compiler bug, or hardware failure.

- The only  trusted function in this module was written by you.

You said that there is a third implementation somewhere. If that one 
actually works, I apologize and ask you to please paste it again in this 
subthread.

Aliak <something something.com> writes:

On Monday, 22 October 2018 at 10:26:14 UTC, Timon Gehr wrote:
 ---
 module borked;

 void atomicIncrement(int* p) system{
     import core.atomic;
     atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
 }

 struct Atomic(T){
     private T val;
     void opUnary(string op : "++")() shared  trusted {
         atomicIncrement(cast(T*)&val);
     }
 }
 void main() safe{
     Atomic!int i;
     auto a=&[i][0];// was: Atomic!int* a = &i;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++i.val; // race
 }
 ---


 Oh no! The author of the  trusted function (i.e. you) did not 
 deliver on the promise they made!

hi, if you change the private val in Atomic to be “private shared 
T val”, is the situation the same?

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.10.18 14:39, Aliak wrote:
 On Monday, 22 October 2018 at 10:26:14 UTC, Timon Gehr wrote:
 ---
 module borked;

 void atomicIncrement(int* p) system{
     import core.atomic;
     atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
 }

 struct Atomic(T){
     private T val;
     void opUnary(string op : "++")() shared  trusted {
         atomicIncrement(cast(T*)&val);
     }
 }
 void main() safe{
     auto a=new Atomic!int;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++a.val; // race
 }
 ---


 Oh no! The author of the  trusted function (i.e. you) did not deliver 
 on the promise they made!

 
 hi, if you change the private val in Atomic to be “private shared T 
 val”, is the situation the same?

It's a bit different, because then there is no implicit unshared->shared 
conversion happening, and this discussion is only about that. However, 
without further restrictions, you can probably construct cases where a 
 safe function in one module escapes a private shared(T)* member to 
somewhere else that expects a different synchronization strategy.

Therefore, even if we agree that unshared->shared conversion cannot be 
implicit in  safe code, the 'shared' design is not complete, but it 
would be a good first step to agree that this cannot happen, such that 
we can then move on to harder issues.

E.g. probably it would be good to have something like  trusted data that 
cannot be manipulated from  safe code, such that  trusted functions can 
rely on some invariants.

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.10.18 12:26, Timon Gehr wrote:
 ---
 module borked;
 
 void atomicIncrement(int* p) system{
      import core.atomic;
      atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
 }
 
 struct Atomic(T){
      private T val;
      void opUnary(string op : "++")() shared  trusted {
          atomicIncrement(cast(T*)&val);
      }
 }
 void main() safe{
      Atomic!int i;
      auto a=&[i][0];// was: Atomic!int* a = &i;
      import std.concurrency;
      spawn((shared(Atomic!int)* a){ ++*a; }, a);
      ++i.val; // race
 }
 ---

Obviously, this should have been:

---
module borked;

void atomicIncrement(int*p) system{
     import core.atomic;
     atomicOp!"+="(*cast(shared(int)*)p,1);
}
struct Atomic(T){
     private T val;
     void opUnary(string op:"++")()shared  trusted{
         atomicIncrement(cast(T*)&val);
     }
}
void main() safe{
     auto a=new Atomic!int;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++a.val; // race
}
---

(I was short on time and had to fix Manu's code because it was not 
actually compilable.)

Manu <turkeyman gmail.com> writes:

On Mon, Oct 22, 2018 at 6:00 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 22.10.18 12:26, Timon Gehr wrote:
 ---
 module borked;

 void atomicIncrement(int* p) system{
      import core.atomic;
      atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
 }

 struct Atomic(T){
      private T val;
      void opUnary(string op : "++")() shared  trusted {
          atomicIncrement(cast(T*)&val);
      }
 }
 void main() safe{
      Atomic!int i;
      auto a=&[i][0];// was: Atomic!int* a = &i;
      import std.concurrency;
      spawn((shared(Atomic!int)* a){ ++*a; }, a);
      ++i.val; // race
 }
 ---

 Obviously, this should have been:

 ---
 module borked;

 void atomicIncrement(int*p) system{
      import core.atomic;
      atomicOp!"+="(*cast(shared(int)*)p,1);
 }
 struct Atomic(T){
      private T val;
      void opUnary(string op:"++")()shared  trusted{
          atomicIncrement(cast(T*)&val);
      }
 }
 void main() safe{
      auto a=new Atomic!int;
      import std.concurrency;
      spawn((shared(Atomic!int)* a){ ++*a; }, a);
      ++a.val; // race
 }
 ---

 (I was short on time and had to fix Manu's code because it was not
 actually compilable.)

Nitpick; atomicOp does not receive a shared arg under my proposal,
it's not a threadsafe function by definition as discussed a few times.

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Monday, 22 October 2018 at 10:26:14 UTC, Timon Gehr wrote:
 module borked;

 void atomicIncrement(int* p) system{
     import core.atomic;
     atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
 }

 struct Atomic(T){
     private T val;
     void opUnary(string op : "++")() shared  trusted {
         atomicIncrement(cast(T*)&val);
     }
 }
 void main() safe{
     Atomic!int i;
     auto a=&[i][0];// was: Atomic!int* a = &i;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++i.val; // race
 }
 ---


 Oh no! The author of the  trusted function (i.e. you) did not 
 deliver on the promise they made!

 Now, before you go and tell me that I am stupid because I wrote 
 bad code, consider the following:

 - It is perfectly  safe to access private members from the same 
 module.

Yes, so you need to place the  trusted code in a separate module. 
The  trusted code will be very rare (Atomic!T, lockfree queue, 
mutex, semaphore...). This will be in the standard library, 
possibly some other library. You should not be writing your own 
implementations of these. You should not be writing  trusted code 
without very good reason. You should be using  safe functions all 
over your code if at all possible.


 - You may not blame the my  safe main function for the problem. 
 It is  safe, so it cannot be blamed for UB. Any UB is the 
 result of a bad  trusted function, a compiler bug, or hardware 
 failure.

No, we blame the fact you have not blocked off non-thread-safe 
access to Atomic!T.val. What you have made is this:

https://i.imgur.com/PnKMigl.jpg

The piece of code to blame is the  trusted function - you can't 
trust it, since non-thread-safe access to Atomic!T.val has not 
been blocked off. As long as anyone can extend its interface, 
Atomic!T can't be thread-safe.

I'll quote myself:

 For clarity: the interface of a type is any method, function,
 delegate or otherwise that may affect its internals. That
 means any free function in the same module, and any
 non-private members.

I've actually missed some possibilities there - member functions 
of other types in the same module must also count as part of the 
interface. Because of this wide net, modules that implement 
thread-safe types with shared methods should be short and sweet.


 - The only  trusted function in this module was written by you.

 You said that there is a third implementation somewhere. If 
 that one actually works, I apologize and ask you to please 
 paste it again in this subthread.

Here's the correct version:

module atomic;

void atomicIncrement(int* p)  system {
     import core.atomic;
     atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
}

struct Atomic(T) {
     // Should probably mark this shared for extra safety,
     // but it's not strictly necessary
     private T val;
     void opUnary(string op : "++")() shared  trusted {
         atomicIncrement(cast(T*)&val);
     }
}
---------
module unborked;
import atomic;

void main()  safe {
     auto a = new Atomic!int;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     //++i.val; // Cannot access private member
}

Once more, Joe Average Programmer should not be writing the 
 trusted code in Atomic!T.opUnary - he should be using libraries 
written by people who have studied the exact issues that make 
multithreading hard.

--
   Simen

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.10.18 15:26, Simen Kjærås wrote:
 Here's the correct version:
 
 module atomic;
 
 void atomicIncrement(int* p)  system {
      import core.atomic;
      atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
 }
 
 struct Atomic(T) {
      // Should probably mark this shared for extra safety,
      // but it's not strictly necessary
      private T val;
      void opUnary(string op : "++")() shared  trusted {
          atomicIncrement(cast(T*)&val);
      }
 }
 ---------
 module unborked;
 import atomic;
 
 void main()  safe {
      auto a = new Atomic!int;
      import std.concurrency;
      spawn((shared(Atomic!int)* a){ ++*a; }, a);
      //++i.val; // Cannot access private member
 }
 
 Once more, Joe Average Programmer should not be writing the  trusted 
 code in Atomic!T.opUnary - he should be using libraries written by 
 people who have studied the exact issues that make multithreading hard.
 
 -- 
    Simen

module reborked;
import atomic;

void main() safe{
     auto a=new Atomic!int;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++a.tupleof[0];
}

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
 module reborked;
 import atomic;

 void main() safe{
     auto a=new Atomic!int;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++a.tupleof[0];
 }

Finally! Proof that MP is impossible. On the other hand, why the 
hell is that  safe? It breaks all sorts of guarantees about 
 safety. At a minimum, that should be un- safe.

Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326

--
   Simen

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.10.18 16:09, Simen Kjærås wrote:
 On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
 module reborked;
 import atomic;

 void main() safe{
     auto a=new Atomic!int;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++a.tupleof[0];
 }

 
 Finally! Proof that MP is impossible. On the other hand, why the hell is 
 that  safe? It breaks all sorts of guarantees about  safety. At a 
 minimum, that should be un- safe.
 
 Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326
 
 -- 
    Simen

Even if this is changed (and it probably should be), it does not fix the 
case where the  safe function is in the same module. I don't think it is 
desirable to change the definition of  trusted such that you need to 
check the entire module if it contains a single  trusted function.

If I can break safety of some (previously correct) code by editing only 
 safe code, then that's a significant blow to  safe. I think we need a 
general way to protect data from being manipulated in  safe code in any 
way, same module or not.

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Monday, 22 October 2018 at 14:31:28 UTC, Timon Gehr wrote:
 On 22.10.18 16:09, Simen Kjærås wrote:
 On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
 module reborked;
 import atomic;

 void main() safe{
     auto a=new Atomic!int;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++a.tupleof[0];
 }

 
 Finally! Proof that MP is impossible. On the other hand, why 
 the hell is that  safe? It breaks all sorts of guarantees 
 about  safety. At a minimum, that should be un- safe.
 
 Filed in bugzilla: 
 https://issues.dlang.org/show_bug.cgi?id=19326
 
 --
    Simen

 Even if this is changed (and it probably should be), it does 
 not fix the case where the  safe function is in the same 
 module. I don't think it is desirable to change the definition 
 of  trusted such that you need to check the entire module if it 
 contains a single  trusted function.

 If I can break safety of some (previously correct) code by 
 editing only  safe code, then that's a significant blow to 
  safe. I think we need a general way to protect data from being 
 manipulated in  safe code in any way, same module or not.

What do you mean by 'previously correct'?

struct Array(T) {
      safe:
     private int* ptr;
     private int length;
      disable this();
     this(int n)  trusted {
         ptr = new int[n].ptr;
         length = n;
         foreach (ref e; ptr[0..length])
             e = 123;
     }
      trusted ref int get(int idx) {
         assert(idx < length);
         return ptr[idx];
     }
}

unittest {
     auto s = Array!int(1);
     assert(s.get(0) == 123);
}

Is this correct code?

What if I add this:

 safe void bork(T)(ref Array!T s) {
     s.length *= 2;
}

unittest {
     auto s = Array!int(1);
     bork(s);
     assert(s.get(1) == 123); // Out of bounds!
}

--
   Simen

Manu <turkeyman gmail.com> writes:

On Mon, Oct 22, 2018 at 7:35 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 22.10.18 16:09, Simen Kjærås wrote:
 On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
 module reborked;
 import atomic;

 void main() safe{
     auto a=new Atomic!int;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++a.tupleof[0];
 }

 Finally! Proof that MP is impossible. On the other hand, why the hell is
 that  safe? It breaks all sorts of guarantees about  safety. At a
 minimum, that should be un- safe.

 Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326

 --
    Simen

 Even if this is changed (and it probably should be), it does not fix the
 case where the  safe function is in the same module. I don't think it is
 desirable to change the definition of  trusted such that you need to
 check the entire module if it contains a single  trusted function.

 If I can break safety of some (previously correct) code by editing only
  safe code, then that's a significant blow to  safe. I think we need a
 general way to protect data from being manipulated in  safe code in any
 way, same module or not.

I'm all ears.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/22/2018 7:31 AM, Timon Gehr wrote:
 Even if this is changed (and it probably should be), it does not fix the case 
 where the  safe function is in the same module. I don't think it is desirable
to 
 change the definition of  trusted such that you need to check the entire
module 
 if it contains a single  trusted function.
 
 If I can break safety of some (previously correct) code by editing only  safe 
 code, then that's a significant blow to  safe. I think we need a general way
to 
 protect data from being manipulated in  safe code in any way, same module or
not.

One possible workaround is to use PIMPL. Another is to put the  trusted
privates 
in a separate module.

The rationale behind module access to privates is:

1. avoid the whole ugly C++ "friend" complexity

2. presumably someone with privileges to edit a module can be trusted to behave

3. the module is the level of encapsulation

12345swordy <alexanderheistermann gmail.com> writes:

On Monday, 22 October 2018 at 23:39:37 UTC, Walter Bright wrote:
 On 10/22/2018 7:31 AM, Timon Gehr wrote:
 [...]

 One possible workaround is to use PIMPL. Another is to put the 
  trusted privates in a separate module.

 The rationale behind module access to privates is:

 1. avoid the whole ugly C++ "friend" complexity

 2. presumably someone with privileges to edit a module can be 
 trusted to behave

 3. the module is the level of encapsulation

Quick question, if a class import a module in its scope, does it 
carry its own copy when creating multiple objects with it?

-Alex

Neia Neutuladh <neia ikeran.org> writes:

On Tue, 23 Oct 2018 00:08:04 +0000, 12345swordy wrote:
 Quick question, if a class import a module in its scope, does it carry
 its own copy when creating multiple objects with it?

Importing a module doesn't make a copy of any code. Each module contains 
only its own code. Importing modules makes it so you can access their 
code, nothing more.

Instances of a class don't contain private copies of their own code. 
There's only one copy per executable. So even if importing a module made a 
copy of the code, you'd still only have one copy per type, not per object.

12345swordy <alexanderheistermann gmail.com> writes:

On Tuesday, 23 October 2018 at 01:21:32 UTC, Neia Neutuladh wrote:
 On Tue, 23 Oct 2018 00:08:04 +0000, 12345swordy wrote:
 Quick question, if a class import a module in its scope, does 
 it carry its own copy when creating multiple objects with it?

 Importing a module doesn't make a copy of any code. Each module 
 contains only its own code. Importing modules makes it so you 
 can access their code, nothing more.

 Instances of a class don't contain private copies of their own 
 code. There's only one copy per executable. So even if 
 importing a module made a copy of the code, you'd still only 
 have one copy per type, not per object.

Huh, that what I figured. Though it not easy to test it as you 
literally have to create two files for this.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 22, 2018 at 7:10 AM Simen Kjærås via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
 module reborked;
 import atomic;

 void main() safe{
     auto a=new Atomic!int;
     import std.concurrency;
     spawn((shared(Atomic!int)* a){ ++*a; }, a);
     ++a.tupleof[0];
 }

 Finally! Proof that MP is impossible. On the other hand, why the
 hell is that  safe? It breaks all sorts of guarantees about
  safety. At a minimum, that should be un- safe.

 Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326

Yeah, that's shockingly dangerous for all sorts of reasons!
I mean, is this really an argument to destroy my proposal, or are you
just destroying  safe in general?

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.10.18 21:29, Manu wrote:
 On Mon, Oct 22, 2018 at 7:10 AM Simen Kjærås via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
 module reborked;
 import atomic;

 void main() safe{
      auto a=new Atomic!int;
      import std.concurrency;
      spawn((shared(Atomic!int)* a){ ++*a; }, a);
      ++a.tupleof[0];
 }

 Finally! Proof that MP is impossible. On the other hand, why the
 hell is that  safe? It breaks all sorts of guarantees about
  safety. At a minimum, that should be un- safe.

 Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326

 
 Yeah, that's shockingly dangerous for all sorts of reasons!
 I mean, is this really an argument to destroy my proposal,

It was a way to satisfy Simen's (imho, arbitrary) constraint that the 
 safe code should be in a different module.

 or are you just destroying  safe in general?
 

It is likely that there is  trusted code in the wild that is currently 
broken because of the assumption that private data cannot be modified by 
untrusted actors. I think if we can have e.g.  trusted data that cannot 
be manipulated at all from  safe code (including taking addresses), a 
lot more is possible. shared on variables/fields could then imply  trusted.

It is then however still not clear that it makes sense to allow implicit 
conversion from unshared to shared. It may still be error prone, or even 
impossible to realize. For example, it could hypothetically be the case 
that each processor has its own address space and there is additionally 
some shared address space, in which case allocation would differ for 
data that is shared and data that is unshared.

So I would again like to ask: why can't classes that want to be able to 
have their references implicitly converted to shared and then be sent to 
other threads not just make all members 'shared'? Under your proposal, 
they can never know that they have not already been implicitly converted 
to 'shared' anyway, so any unshared code already needs to take into 
account the possibility that there are other, concurrent accesses.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/22/2018 6:40 AM, Timon Gehr wrote:
      ++a.tupleof[0];

Yes, tupleof is known to break through the private access barrier. This is an 
issue, not sure if there's a bugzilla for it.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 22, 2018 at 3:30 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 22.10.18 02:54, Manu wrote:
 On Sun, Oct 21, 2018 at 5:40 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 21.10.18 21:04, Manu wrote:
 On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 21.10.18 17:54, Nicholas Wilson wrote:
 As soon as that is done, you've got a data race with the other
 existing unshared aliases.

 You're in  trusted code, that is the whole point. The onus is on the
 programmer to make that correct, same with regular  safe/ trusted system
 code.

 Not all of the parties that participate in the data race are in  trusted
 code. The point of  trusted is modularity: you manually check  trusted
 code according to some set of restrictions and then you are sure that
 there is no memory corruption.

 Note that you are not allowed to look at any of the  safe code while
 checking your  trusted code. You will only see an opaque interface to
 the  safe code that you call and all you know is that all the  safe code
 type checks according to  safe rules. Note that there might be an
 arbitrary number of  safe functions and methods that you do not see.

 Think about it this way: you first write all the  trusted and  system
 code, and some evil guy who does not like you comes in after you looks
 at your code and writes all the  safe code. If there is any memory
 corruption, it will be your fault and you will face harsh consequences.
 Now, design the  safe type checking rules. It won't be MP!

 Note that there may well be a good way to get the good properties of MP
 without breaking the type system, but MP itself is not good because it
 breaks  safe.

 Show me. Nobody has been able to show that yet. I'd really like to know this.

 I just did,

 There's no code there... just a presumption that the person who wrote
 the  trusted code did not deliver the promise they made.
 ...

 Yes, because there is no way to write  trusted code that holds its
 promise while actually doing something interesting in multiple threads
 if  safe code can implicitly convert from unshared to shared.

How do my examples prior fail to hold their promise?
struct S
{
  private int x;
  void method() shared  trusted { /* safely manipulate x */ }
}

How can you not trust that function? How can a 3rd party invalidate
that functions promise? `x` is inaccessible.
S can be thread-local or shared as much as you like, and it's safe,
and I don't know how a 3rd party could  safely undermine that?

 but if you really need to, give me a non-trivial piece of> correct
multithreaded code that accesses some declared-unshared field
 from a shared method and I will show you how the evil guy would modify
 some  safe code in it and introduce race conditions. It needs to be your
 code, as otherwise you will just claim again that it is me who wrote bad
  trusted code.

 You can pick on any of my prior code fragments. They've all been ignored so
far.

 I don't want "code fragments". Show me the real code.

 I manually browsed through posts now (thanks a lot) and found this
 implementation:

 struct Atomic(T){
    void opUnary(string op : "++")() shared { atomicIncrement(&val); }
    private T val;
 }

 This is  system code. There is no  safe or  trusted here, so I am
 ignoring it.


 Then I browsed some more, because I had nothing better to do, and I
 found this. I completed it so that it is actually compilable, except for
 the unsafe implicit conversion.

 Please read this code, and then carefully read the comments below it
 before you respond. I will totally ignore any of your answers that
 arrive in the next two hours.

 ---
 module borked;

 void atomicIncrement(int* p) system{
      import core.atomic;
      atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
 }

 struct Atomic(T){
      private T val;
      void opUnary(string op : "++")() shared  trusted {
          atomicIncrement(cast(T*)&val);
      }
 }
 void main() safe{
      Atomic!int i;
      auto a=&[i][0];// was: Atomic!int* a = &i;
      import std.concurrency;
      spawn((shared(Atomic!int)* a){ ++*a; }, a);
      ++i.val; // race
 }
 ---


 Oh no! The author of the  trusted function (i.e. you) did not deliver on
 the promise they made!

 Now, before you go and tell me that I am stupid because I wrote bad
 code, consider the following:

 - It is perfectly  safe to access private members from the same module.

 - You may not blame the my  safe main function for the problem. It is
  safe, so it cannot be blamed for UB. Any UB is the result of a bad
  trusted function, a compiler bug, or hardware failure.

 - The only  trusted function in this module was written by you.

 You said that there is a third implementation somewhere. If that one
 actually works, I apologize and ask you to please paste it again in this
 subthread.

Last time I checked, main does not go in core.atomic. I've never added
any code to core.atomic.
These things wouldn't live in peoples code.

I understand that my proposal relies on trusting a very small number
of low-level implementations at the bottom of the stack... but they're
strongly encapsulated, live in libraries, and if there existed a world
where you COULD describe threadsafe interaction with shared; you
would.
 trusted user code, especially where it related to `shared` would be
terrifying, and you wouldn't do it.
I understand that the situation you present is technically possible,
but why would it happen in reality? I find the proposition of a  safe
threading stack to far far outweight that risk.
It's also possible that tech may be improved to assist with this
particular problem, I haven't tried to address it; I'm interested in
if the rules are sound.

One suggestion was to make `private int val` shared, then external
functions have no access... that helps mitigate this particular
mistake.

It's back to this 1:many thing. There's one place you can possibly
make this mistake (and it's probably maintained by an expert author),
and it should be well encapsulated in a core lib; whereas the current
`shared` requires that _end-users_ do unsafe casts all the time among
user code, and they're almost certainly not experts.
That's rigged totally backwards. The values are all wrong.

If my scheme is sound above this issue, then it's reasonable to focus
on ideas to reduce the odds of mistake for the one implementer of the
low-level tool.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/21/2018 11:58 AM, Timon Gehr wrote:
 [...]

Thank you, Timon, for a nice explanation of what I was trying to express.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 22, 2018 at 12:55 AM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/21/2018 11:58 AM, Timon Gehr wrote:
 [...]

 Thank you, Timon, for a nice explanation of what I was trying to express.

You removed whatever comment you're referring to.
I don't understand any of Timon's posts in this thread at all, which
is unusual because he's usually pretty clear.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/22/2018 1:42 AM, Manu wrote:
 You removed whatever comment you're referring to.

If your newsreader cannot find the antecedent, you badly need to use a better 
one. Thunderbird handles this rather well, there's no reason to use an inferior
one.

Or just click the <- button:

https://digitalmars.com/d/archives/digitalmars/D/shared_-_i_need_it_to_be_useful_320165.html#N320607


 I don't understand any of Timon's posts in this thread at all, which
 is unusual because he's usually pretty clear.

I suspect Timon is equally frustrated at not getting his point across.

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 3:00 AM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/20/2018 11:08 AM, Nicholas Wilson wrote:
 You can if no-one else writes to it, which is the whole point of Manu's
 proposal. Perhaps it should be const shared instead of shared but still.

 There is no purpose whatsoever to data that can be neither read nor written.

There is no primitive type with implicit threadsafety... nor need there be.
Shared data simply can not be read or written in any threadsafe
manner. This is a rock-solid reality, and the type system needs to
reflect that reality as a fundamental premise.
Only from there can we start to define meaningful threadsafety.

All threadsafe interactions with anything involve calling functions.
It's completely reasonable to make `shared` inhibit all read and write
access to data. We can only call shared methods, because only
real-code can implement threadsafety.

 Shared data is only useful if, at some point, it is read/written, presumably by
 casting it to unshared in  trusted code. As soon as that is done, you've got a
 data race with the other existing unshared aliases.

If such a race is possible, then the  trusted function is not
threadsafe, so it is not  trusted by definition.
You wrote a bad  trusted function, and you should feel bad.

The simplest way to guarantee that no unsafe access is possible is to
use encapsulation to assure no unregulated access exists.

Timon Gehr <timon.gehr gmx.ch> writes:

On 21.10.18 20:46, Manu wrote:
 Shared data is only useful if, at some point, it is read/written, presumably by
 casting it to unshared in  trusted code. As soon as that is done, you've got a
 data race with the other existing unshared aliases.

 If such a race is possible, then the  trusted function is not
 threadsafe, so it is not  trusted by definition.
 You wrote a bad  trusted function, and you should feel bad.
 ...

I wonder where this "each piece of code is maintained by only one person 
and furthermore this is the only person that will suffer if the code has 
bugs" mentality comes from. It is very popular as well as obviously 
nonsense.

 The simplest way to guarantee that no unsafe access is possible is to
 use encapsulation to assure no unregulated access exists.

This only works if untrusted programmers (i.e. programmers who are only 
allowed to write/modify  safe code) are not allowed to change your 
class. I.e. it does not work.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Monday, 22 October 2018 at 00:32:35 UTC, Timon Gehr wrote:
 This only works if untrusted programmers (i.e. programmers who 
 are only allowed to write/modify  safe code) are not allowed to 
 change your class. I.e. it does not work.

This is the basis of the current  safe/ trusted/ system model.
Are you saying it is useless?

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 5:35 PM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 21.10.18 20:46, Manu wrote:
 Shared data is only useful if, at some point, it is read/written, presumably by
 casting it to unshared in  trusted code. As soon as that is done, you've got a
 data race with the other existing unshared aliases.

 If such a race is possible, then the  trusted function is not
 threadsafe, so it is not  trusted by definition.
 You wrote a bad  trusted function, and you should feel bad.
 ...

 I wonder where this "each piece of code is maintained by only one person
 and furthermore this is the only person that will suffer if the code has
 bugs" mentality comes from. It is very popular as well as obviously
 nonsense.

 The simplest way to guarantee that no unsafe access is possible is to
 use encapsulation to assure no unregulated access exists.

 This only works if untrusted programmers (i.e. programmers who are only
 allowed to write/modify  safe code) are not allowed to change your
 class. I.e. it does not work.

Have you ever cracked open std::map and 'fixed' it because you thought
it was bad?
Of course not. Same applies here. Nobody 'fixes' core.atomic.Atomic
without understanding what they're doing.

You seem to be stuck on the detail whether you can trust the  trusted
author though... that is a reasonable point of debate, but it's a
slightly separate topic. I am confident that the number of  trusted
functions required to found a useful stack are low, probably countable
with fingers, and always in a library.

If we can put aside that point of debate just for now; whether you
feel the  trusted author can be trusted, assuming that they can, does
the model work? Can you break the model as I have presented it?
If not; if the model is sound, then we can begin the discussion you're
alluding to and talk about opportunities to improve on static
guarantees for  trusted authors, or ways to communicate their
responsibility clearly, and patterns to assure success.

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.10.18 02:45, Manu wrote:
 On Sun, Oct 21, 2018 at 5:35 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 21.10.18 20:46, Manu wrote:
 Shared data is only useful if, at some point, it is read/written, presumably by
 casting it to unshared in  trusted code. As soon as that is done, you've got a
 data race with the other existing unshared aliases.

 If such a race is possible, then the  trusted function is not
 threadsafe, so it is not  trusted by definition.
 You wrote a bad  trusted function, and you should feel bad.
 ...

 I wonder where this "each piece of code is maintained by only one person
 and furthermore this is the only person that will suffer if the code has
 bugs" mentality comes from. It is very popular as well as obviously
 nonsense.

 The simplest way to guarantee that no unsafe access is possible is to
 use encapsulation to assure no unregulated access exists.

 This only works if untrusted programmers (i.e. programmers who are only
 allowed to write/modify  safe code) are not allowed to change your
 class. I.e. it does not work.

 
 Have you ever cracked open std::map and 'fixed' it because you thought
 it was bad?
 Of course not. Same applies here. Nobody 'fixes' core.atomic.Atomic
 without understanding what they're doing.
 

You are not proposing to let core.atomic.Atomic convert to shared 
implicitly, you are proposing to do that for all classes.

 You seem to be stuck on the detail whether you can trust the  trusted
 author though...

Again: the  safe author is the problem.

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 5:55 PM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 22.10.18 02:45, Manu wrote:
 On Sun, Oct 21, 2018 at 5:35 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 21.10.18 20:46, Manu wrote:
 Shared data is only useful if, at some point, it is read/written, presumably by
 casting it to unshared in  trusted code. As soon as that is done, you've got a
 data race with the other existing unshared aliases.

 If such a race is possible, then the  trusted function is not
 threadsafe, so it is not  trusted by definition.
 You wrote a bad  trusted function, and you should feel bad.
 ...

 I wonder where this "each piece of code is maintained by only one person
 and furthermore this is the only person that will suffer if the code has
 bugs" mentality comes from. It is very popular as well as obviously
 nonsense.

 The simplest way to guarantee that no unsafe access is possible is to
 use encapsulation to assure no unregulated access exists.

 This only works if untrusted programmers (i.e. programmers who are only
 allowed to write/modify  safe code) are not allowed to change your
 class. I.e. it does not work.

 Have you ever cracked open std::map and 'fixed' it because you thought
 it was bad?
 Of course not. Same applies here. Nobody 'fixes' core.atomic.Atomic
 without understanding what they're doing.

 You are not proposing to let core.atomic.Atomic convert to shared
 implicitly, you are proposing to do that for all classes.

You can always implicitly convert to shared.
Where did I ever say anything like that? I'm sure I've never said
this. How do these transformations of what I've said keep happening?

 You seem to be stuck on the detail whether you can trust the  trusted
 author though...

 Again: the  safe author is the problem.

I don't follow. The  safe author is incapable of doing threadsafety
violation. They can only combine threadsafe functions.
They can certainly produce a program that doesn't work, and they are
capable of ordering issues, but that's not the same as data-race
related crash bugs.

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.10.18 03:01, Manu wrote:
 On Sun, Oct 21, 2018 at 5:55 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 22.10.18 02:45, Manu wrote:
 On Sun, Oct 21, 2018 at 5:35 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 21.10.18 20:46, Manu wrote:
 Shared data is only useful if, at some point, it is read/written, presumably by
 casting it to unshared in  trusted code. As soon as that is done, you've got a
 data race with the other existing unshared aliases.

 If such a race is possible, then the  trusted function is not
 threadsafe, so it is not  trusted by definition.
 You wrote a bad  trusted function, and you should feel bad.
 ...

 I wonder where this "each piece of code is maintained by only one person
 and furthermore this is the only person that will suffer if the code has
 bugs" mentality comes from. It is very popular as well as obviously
 nonsense.

 The simplest way to guarantee that no unsafe access is possible is to
 use encapsulation to assure no unregulated access exists.

 This only works if untrusted programmers (i.e. programmers who are only
 allowed to write/modify  safe code) are not allowed to change your
 class. I.e. it does not work.

 Have you ever cracked open std::map and 'fixed' it because you thought
 it was bad?



(Also, yes, some people do that because std::map does not provide an 
interface to augment the binary search tree.)

 Of course not. Same applies here. Nobody 'fixes' core.atomic.Atomic
 without understanding what they're doing.

 You are not proposing to let core.atomic.Atomic convert to shared
 implicitly, you are proposing to do that for all classes.

 
 You can always implicitly convert to shared.

Yes, exactly what I said.

 Where did I ever say anything like that? I'm sure I've never said
 this.

???

I said that you are proposing to allow implicit conversions to shared 
for all classes, not only core.atomic.Atomic, and the last time you said 
it was the previous sentence of the same post.

 How do these transformations of what I've said keep happening?
 ...

You literally said that nobody changes core.atomic.Atomic. Anyway, even 
if I bought that  safe somehow should not be checked within druntime (I 
don't), bringing up this example does not make for a coherent argument 
why implicit conversion to shared should be allowed for all classes.

 You seem to be stuck on the detail whether you can trust the  trusted
 author though...

 Again: the  safe author is the problem.

 
 I don't follow. The  safe author is incapable of doing threadsafety
 violation.

They are capable of doing so as soon as you provide them a  trusted 
function that treats data as shared that they can access as unshared.

 They can only combine threadsafe functions.
 They can certainly produce a program that doesn't work, and they are
 capable of ordering issues, but that's not the same as data-race
 related crash bugs.
 

Accessing private members of aggregates in the same module is  safe. 
tupleof is  safe too.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 22, 2018 at 6:40 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 22.10.18 03:01, Manu wrote:
 Where did I ever say anything like that? I'm sure I've never said
 this.

 ???

 I said that you are proposing to allow implicit conversions to shared
 for all classes, not only core.atomic.Atomic, and the last time you said
 it was the previous sentence of the same post.

Sorry, I read the "not proposing to let" as something like "proposing
to not let".
So let me re-respond.

Yes, I propose implicit conversion to shared. That's what makes it usable.
Basically any argument to a fork/join, parallel-for, map/reduce,
etc... they all require this transition.
Our software is almost exclusively made up of those processes. Almost
every type we have transits to shared contexts (with a restricted
threadsafe API), and almost nothing we have could be allocated shared
to the exclusion of thread-local access (like Atila likes to suggest).
I don't know how to make use of the mutually-exclusive design of the
current model in any code I've ever written.

I've been writing SMP code since the xbox360 alpha-kit landed on my
desk in 2006. As we've matured, we've seen mutexes disappear
completely. Even discreet worker threads which used to use semaphores
to signal new work arrival are dispappearing as it's becoming
impossible to find enough distinct kinds of work for a discreet worker
threads to do that keeps all cores busy.
We break the work into tasks, build a DAG of the execution schedule
with respect to data access dependencies, and parallel-for/reduce
within tasks that operate over large volumes of data.
We used to hide data sharing detail behind walls, but it moves into
user-facing code as parallel-for appears, and that invokes the
necessity to be able to express what is threadsafe at the type system
level.

I can assure, in our architecture at least, I am not aware of any case
where a user would write a  trusted function under my proposal. We
could run our stack  safe.

 You seem to be stuck on the detail whether you can trust the  trusted
 author though...

 Again: the  safe author is the problem.

 I don't follow. The  safe author is incapable of doing threadsafety
 violation.

 They are capable of doing so as soon as you provide them a  trusted
 function that treats data as shared that they can access as unshared.

That function is not  trusted by definition.

 They can only combine threadsafe functions.
 They can certainly produce a program that doesn't work, and they are
 capable of ordering issues, but that's not the same as data-race
 related crash bugs.

 Accessing private members of aggregates in the same module is  safe.
 tupleof is  safe too.

I don't see any situation where any user would write code in the same
module as core.atomic, or core.mutex, or wherever these couple of
functions live.

Manu <turkeyman gmail.com> writes:

On Sat, Oct 20, 2018 at 10:10 AM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 20 October 2018 at 16:48:05 UTC, Nicholas Wilson
 wrote:
 On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright
 wrote:
 On 10/19/2018 11:18 PM, Manu wrote:
 The reason I ask is because, by my definition, if you have:
 int* a;
 shared(int)* b = a;

 While you have 2 numbers that address the same data, it is
 not actually aliased because only `a` can access it.

 They are aliased,

 Quoting Wikipedia:

two pointers A and B which have the same value, then the name
A[0] aliases the name B[0]. In this case we say the pointers A
and B alias each other. Note that the concept of pointer
aliasing is not very well-defined – two pointers A and B may or
may not alias each other, depending on what operations are
performed in the function using A and B.

 In this case given the above: `a[0]` does not alias `b[0]`
 because `b[0]` is ill defined under Manu's proposal, because
 the memory referenced by `a` is not reachable through `b`
 because you can't read or write through `b`.

 by code that believes it is unshared

 you cannot ` safe`ly modify the memory  through `b`, `a`'s view
 of the memory is unchanged in  safe code.

 And that's already a bug, because the language can't enforce
 threadsafe access through `a`, regardless of presence of `b`.
 Only the programmer can.

 and, code that believes it is shared.

 you cannot have non-atomic access though `b`, `b` has no  safe
 view of the memory, unless it is atomic (which by definition is
 synchronised).

 Synchronized with what? You still have `a`, which isn't `shared`
 and doesn't require any atomic access or synchronization. At this
 point it doesn't matter if it's an int or a struct. As soon as
 you share `a`, you can't just pretend that reading or writing `a`
 is safe.

`b` can't read or write `a`... accessing `a` is absolutely safe.
Someone must do something unsafe to undermine your threadsafety... and
if you write unsafe code and don't know what you're doing, there's
nothing that can help you.
Today, every interaction with shared is unsafe. Creating a safe
interaction with shared will lead to people not doing unsafe things at
every step.

 Encapsulate it all you want, safety only remains a
 contract of convention, the language can't enforce it.

You're talking about  trusted code again. You're fixated on unsafe
interactions... my proposal is about SAFE interactions. I'm trying to
obliterate unsafe interactions with shared.


 module expertcode;

  safe:

 struct FileHandle {
       safe:

      void[] read(void[] storage) shared;
      void[] write(const(void)[] buffer) shared;
 }

 FileHandle openFile(string path);
 // only the owner can close
 void closeFile(ref FileHandle);

 void shareWithThreads(shared FileHandle*); // i.e. generate a
 number of jobs in some queue
 void waitForThreads();                     // waits until all
 processing is done

 module usercode;

 import expertcode;

 void processHugeFile(string path) {
      FileHandle file = openFile(path);
      shareWithThreads(&file);    // implicit cast
      waitForThreads();
      file.closeFile();
 }

This is a very strange program... I'm dubious it is in fact
"expertcode"... but let's look into it.

File handle seems to have just 2 methods... and they are both threadsafe.
Open and Close are free-functions. Close does not promise threadsafety
itself (but of course, it doesn't violate read/write's promise, or the
program is invalid).

I expect the only possible way to achieve this is by an internal mutex
to make sure read/write/close calls are serialised. read and write
will appropriately check their file-open state each time they perform
their actions. What read/write do in the case of being called on a
closed file... anyones guess? I'm gonna say they do no-op... they
return a null pointer to indicate the error state.

Looking at the meat of the program; you open a file, and distribute it
to do accesses (I presume?)....
Naturally, this is a really weird thing to do, because even if the API
is threadsafe such that it doesn't crash and reads/writes are
serialised, the sequencing of reads/writes will be random, so I don't
believe any sane person (let alone an expert) would write this
program... but moving on.
Then you wait for them to finish, and close the file.

Fine. You have a file with randomly interleaved data... for whatever reason.

 Per your proposal, everything in 'expertcode' can be written
  safe, i.e. not violating any of the points that  safe forbids,
 or doing so only in a  trusted manner. As far as the language is
 concerned, this would mean that processHugeFile can be  safe as
 well.

This program does appear to be safe (assuming that the implementations
aren't invalid), but a very strange program nonetheless.

 Remove the call to `waitForThreads()` (assume user just forgot
 that, i.e. the "accident"). Nothing would change for the
 compiler: all calls remain  safe.

Yup.

 And yet, if we're lucky, we get
 a consistent instacrash. If we're unlucky, we get memory
 corruption, or an unsolicited write to another currently open
 file, either of which can go unnoticed for some time.

Woah! Now this is way off-piste..
Why would get a crash? Why would get memory corruption? None of those
things make sense.

So, you call closeFile immediately and read/write start returning null.
I'm going to assume that `shareWithThreads()` was implemented by an
'expert' who checked the function results for errors. It was detected
that the reads/write failed, and an error "failed to read file" was
emit, then the function returned promptly.
The uncertainty of what happens in this program is however
`shareWithThreads()` handles read/write emitting an error.

 Of course the program becomes invalid if you do that, there's no
 question about it, this goes for all buggy code.

In this case, I wouldn't say the program becomes 'invalid'; it is
valid for filesystem functions to return error states and you should
handle them.
In this case, read/write must return some "file not open" state, and
it should be handled properly.
This problem has nothing to do with threadsafety. It's a logic issue
related to threading, but that's got nothing to do with this.

 The problem is,
 definition of "valid" lies beyond the type system: it's an
 agreement between different parts of code, i.e. between expert
 programmers who wrote FileHandle et al., and users who write
 processHugeFile(). The main issue is that certain *runtime*
 conditions can still violate  safe-ty.

Perhaps you don't understand what  safe-ty means? It's a compiler
assertion that the code is memory-safe. It's not a magic attribute
that tells you that your program is right.
Runtime conditions being in a valid state is a high-level problem for
the program, and doesn't interacts with threadsafety in any
fundamental way, and not in any way that  safe has anything to do
with.
You're just describing normal high-level multi-threading logic
problems. `shared` does not and can not help you with that; you need
to look to libraries that offer threading support frameworks for that.
It can help you not write code that does invalid access to memory and
crash. That's the extent of its charter.

If a `shared` API is designed well, it can also offer strong implicit
advice about how to correctly interact with API's. The compiler will
coerce you to do the right things with error messages.

 Your proposal makes the language more strict wrt. to writing
  safe 'expertmodule', thanks to disallowing reads and writes
 through `shared`, which is great.
 However the implicit conversion to `shared` doesn't in any way
 improve the situation as far as user code is concerned, unless
 I'm still missing something.

It does, it eliminates unsafe user interactions. It must be that way to be safe.
There were no casts above, it's great! And your program is safe!
(although it's wrong)

FWIW, I doubt anybody in their right mind would attempt to write a
threadsafe filesystem API this way. Any such API would be structured
COMPLETELY differently; it would likely have one `shared` method that
would accept requests for deferred fulfillment, and handle unique
objects associated with each request.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Sunday, 21 October 2018 at 05:47:14 UTC, Manu wrote:
 On Sat, Oct 20, 2018 at 10:10 AM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Synchronized with what? You still have `a`, which isn't 
 `shared` and doesn't require any atomic access or 
 synchronization. At this point it doesn't matter if it's an 
 int or a struct. As soon as you share `a`, you can't just 
 pretend that reading or writing `a` is safe.


 `b` can't read or write `a`... accessing `a` is absolutely safe.

It's not, with or without your proposal. The purpose of sharing 
`a` into `b` is to allow someone to access `*a` in a threadsafe 
way (but un- safe, as it *will* require casting away `shared` 
from `b`). That is what's making keeping an unshared reference 
`a` un- safe: whoever accesses `*a` in their  trusted 
implementations via `*b` can't know that `*a` is being 
( safe-ly!) accessed in a non-threadsafe way at the same time.

 Someone must do something unsafe to undermine your 
 threadsafety... and
 if you write unsafe code and don't know what you're doing, 
 there's
 nothing that can help you.

Ergo, it follows that anyone that is making an implicit cast from 
mutable to shared better know what they're doing, which mere 
mortal users (not "experts") might not. I.e. it's a way to giving 
a loaded gun to someone who never held a weapon before.

 Today, every interaction with shared is unsafe.

Nod.

 Creating a safe interaction with shared will lead to people not 
 doing unsafe things at every step.

Triple nod.

 Encapsulate it all you want, safety only remains a
 contract of convention, the language can't enforce it.

 You're talking about  trusted code again. You're fixated on 
 unsafe interactions... my proposal is about SAFE interactions. 
 I'm trying to obliterate unsafe interactions with shared.

I know... Manu, I *know* what you're trying to do. We (me, Atila, 
Timon, Walter...) are not opposing your goals, we're pointing out 
the weakest spot of your proposal, which, it would seem, would 
require more changes to the language than just disallowing 
reading/writing `shared` members.

 module expertcode;

  safe:

 struct FileHandle {
       safe:

      void[] read(void[] storage) shared;
      void[] write(const(void)[] buffer) shared;
 }

 FileHandle openFile(string path);
 // only the owner can close
 void closeFile(ref FileHandle);

 void shareWithThreads(shared FileHandle*); // i.e. generate a
 number of jobs in some queue
 void waitForThreads();                     // waits until all
 processing is done

 module usercode;

 import expertcode;

 void processHugeFile(string path) {
      FileHandle file = openFile(path);
      shareWithThreads(&file);    // implicit cast
      waitForThreads();
      file.closeFile();
 }

 This is a very strange program...

Why? That's literally the purpose of being able to `share`: you 
create/acquire a resource, share it, but keep a non-`shared` 
reference to yourself. If that's not required, you'd just create 
the data `shared` to begin with.

 I'm dubious it is in fact "expertcode"... but let's look into 
 it.

You're fixating on it being file now. I give an abstract example, 
you dismiss it as contrived, I give a concrete one, you want to 
dismiss it as "strange".

Heh, replace 'FileHandle' with 'BackBuffer', 'openFile' with 
'acquireBackBuffer', 'shareWithThreads' with 
'generateDrawCommands', 'waitForThreads' with 
'gatherCommandsAndDraw', 'closeFile' with 'postProcessAndPresent' 
;)

 File handle seems to have just 2 methods... and they are both 
 threadsafe. Open and Close are free-functions.

It doesn't matter if they're free functions or not. What matters 
is signature: they're taking non-`shared` (i.e. 'owned') 
reference. Methods are free functions in disguise.

 Close does not promise threadsafety itself (but of course, it 
 doesn't violate read/write's promise, or the program is 
 invalid).

Yep, and that's the issue. It SHALL NOT violate threadsafety, but 
it can't promise such in any way :(

 I expect the only possible way to achieve this is by an 
 internal mutex to make sure read/write/close calls are 
 serialised.

With that particular interface, yes.

 read and write will appropriately check their file-open state 
 each time they perform their actions.

Why? The only purpose of giving someone a `shared` reference is 
to give a reference to an open file. `shared` references can't do 
anything with the file but read and write, they would expect to 
be able to do so.

 What read/write do in the case of being called on a closed 
 file... anyones guess? I'm gonna say they do no-op... they 
 return a null pointer to indicate the error state.

 Looking at the meat of the program; you open a file, and 
 distribute it to do accesses (I presume?)....

 Naturally, this is a really weird thing to do, because even if 
 the API is threadsafe such that it doesn't crash and 
 reads/writes are
 serialised, the sequencing of reads/writes will be random, so I 
 don't believe any sane person (let alone an expert) would write 
 this
 program... but moving on.

Um, that's literally what std.stdio does, for writes at least, 
except it doesn't advertise `File` as `shared`. That's how we get 
interleaved, but not corrupted, output even when writing from 
multiple threads. Now, that's not *universally* useful, but 
nonetheless that's a valid use case.

 Then you wait for them to finish, and close the file.
 Fine. You have a file with randomly interleaved data... for 
 whatever reason.

Or I have command lists, or images loaded in background...

 This program does appear to be safe (assuming that the 
 implementations aren't invalid), but a very strange program 
 nonetheless.

 Remove the call to `waitForThreads()` (assume user just forgot
 that, i.e. the "accident"). Nothing would change for the
 compiler: all calls remain  safe.

 Yup.

 And yet, if we're lucky, we get
 a consistent instacrash. If we're unlucky, we get memory
 corruption, or an unsolicited write to another currently open
 file, either of which can go unnoticed for some time.


 Woah! Now this is way off-piste..
 Why would get a crash? Why would get memory corruption? None of 
 those things make sense.

Because the whole reason to have `shared` is to avoid the 
extraneous checks that you mentioned above, and only write actual 
useful code (i.e. lock-write-unlock, or read-put-to-queue-repeat, 
or whatever), not busy-work (testing if the file is open on every 
call). If you have a `shared` reference, it better be to existing 
data. If it isn't, the program is invalid already: you've shared 
something that doesn't "exist" (good for marketing, not so good 
for multithreading). That's why having `shared` and un-`shared` 
references to the same data simultaneously is not safe: you can't 
guarantee in any way that the owning thread doesn't invalidate 
the data through it's non-`shared` reference while you're doing 
your threadsafe `shared` work; you can only "promise" that by 
convention (documentation).

 So, you call closeFile immediately and read/write start 
 returning null.

And I have partially-read or partially-written data. Or Maybe I 
call closeFile(), main thread continues and opens another file, 
which gives the same file descriptor, `shared` references to 
FileHandle which the user forgot to wait on continue to work 
oblivious to the fact that it's a different file now. It's a 
horrible, but still  safe, implementation of FileHandle, yes, but 
the caller (user) doesn't know that, and can't know that just 
from the interface. The only advice against that is "don't do 
that", but that's irrespective of your proposal.

 I'm going to assume that `shareWithThreads()` was implemented  
 by an
 'expert' who checked the function results for errors. It was 
 detected that the reads/write failed, and an error "failed to 
 read file" was emit, then the function returned promptly.
 The uncertainty of what happens in this program is however
 `shareWithThreads()` handles read/write emitting an error.

But you can only find out about these errors in `waitForThreads`, 
the very call that the user "forgot" to make!

 Of course the program becomes invalid if you do that, there's

 no question about it, this goes for all buggy code.

 In this case, I wouldn't say the program becomes 'invalid'; it 
 is
 valid for filesystem functions to return error states and you 
 should handle them.
 In this case, read/write must return some "file not open" 
 state, and it should be handled properly.
 This problem has nothing to do with threadsafety. It's a logic 
 issue related to threading, but that's got nothing to do with 
 this.

There's no question about it, it *is* a logic error. The point 
is, it's a logic error that ultimately can lead to UB despite 
being  safe. Just like this is: 
https://issues.dlang.org/show_bug.cgi?id=19316.

 The problem is,
 definition of "valid" lies beyond the type system: it's an
 agreement between different parts of code, i.e. between expert
 programmers who wrote FileHandle et al., and users who write
 processHugeFile(). The main issue is that certain *runtime*
 conditions can still violate  safe-ty.

 Perhaps you don't understand what  safe-ty means? It's a 
 compiler assertion that the code is memory-safe. It's not a 
 magic attribute that tells you that your program is right.

I know.

 Runtime conditions being in a valid state is a high-level 
 problem for the program, and doesn't interacts with 
 threadsafety in any
 fundamental way, and not in any way that  safe has anything to 
 do with.

Yep.

 You're just describing normal high-level multi-threading logic
 problems. `shared` does not and can not help you with that; you 
 need to look to libraries that offer threading support 
 frameworks for that.
 It can help you not write code that does invalid access to 
 memory and crash. That's the extent of its charter.

I understand that. So... it would seem that your proposal focuses 
more on  safe than on threadsafety?

 If a `shared` API is designed well, it can also offer strong 
 implicit advice about how to correctly interact with API's. The 
 compiler will coerce you to do the right things with error 
 messages.

 Your proposal makes the language more strict wrt. to writing
  safe 'expertmodule', thanks to disallowing reads and writes
 through `shared`, which is great.
 However the implicit conversion to `shared` doesn't in any way
 improve the situation as far as user code is concerned, unless
 I'm still missing something.


 It does, it eliminates unsafe user interactions. It must be 
 that way to be safe. There were no casts above, it's great! And 
 your program is safe!
 (although it's wrong)

It's  safe, but it's wrong because it's not threadsafe. Yay! :D

 FWIW, I doubt anybody in their right mind would attempt to 
 write a threadsafe filesystem API this way.

std.stdio ;) (yes, I know there's no `shared` there, but that's 
what it does).

 Any such API would be structured COMPLETELY differently; it 
 would likely have one `shared` method that would accept 
 requests for deferred fulfillment, and handle unique objects 
 associated with each request.

Perhaps. How would the user know that?

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Sunday, 21 October 2018 at 12:45:43 UTC, Stanislav Blinov 
wrote:
 On Sunday, 21 October 2018 at 05:47:14 UTC, Manu wrote:
 On Sat, Oct 20, 2018 at 10:10 AM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Synchronized with what? You still have `a`, which isn't 
 `shared` and doesn't require any atomic access or 
 synchronization. At this point it doesn't matter if it's an 
 int or a struct. As soon as you share `a`, you can't just 
 pretend that reading or writing `a` is safe.


 `b` can't read or write `a`... accessing `a` is absolutely 
 safe.

 It's not, with or without your proposal. The purpose of sharing 
 `a` into `b` is to allow someone to access `*a` in a threadsafe 
 way (but un- safe, as it *will* require casting away `shared` 
 from `b`). That is what's making keeping an unshared reference 
 `a` un- safe: whoever accesses `*a` in their  trusted 
 implementations via `*b` can't know that `*a` is being 
 ( safe-ly!) accessed in a non-threadsafe way at the same time.

Then someone has not done their job. Since the pieces of code 
that will actually use the un- safe building blocks at the bottom 
are few and far between, it is reasonable to assume that an 
expert will be writing this code, and that such code be placed in 
a separate module where all access to the shared type is 
controlled.

It seems you expect regular users to have calls to atomicOp!"++" 
scattered all over their code. I find this an unreasonable 
expectation, and fully agree that this will lead to problems.


 Someone must do something unsafe to undermine your 
 threadsafety... and
 if you write unsafe code and don't know what you're doing, 
 there's
 nothing that can help you.

 Ergo, it follows that anyone that is making an implicit cast 
 from mutable to shared better know what they're doing, which 
 mere mortal users (not "experts") might not. I.e. it's a way to 
 giving a loaded gun to someone who never held a weapon before.

No.


 Close does not promise threadsafety itself (but of course, it 
 doesn't violate read/write's promise, or the program is 
 invalid).

 Yep, and that's the issue. It SHALL NOT violate threadsafety, 
 but it can't promise such in any way :(

Can you demonstrate any system that can promise something like 
that? (apart from all-immutable)


 read and write will appropriately check their file-open state 
 each time they perform their actions.

 Why? The only purpose of giving someone a `shared` reference is 
 to give a reference to an open file. `shared` references can't 
 do anything with the file but read and write, they would expect 
 to be able to do so.

Because otherwise it's not thread-safe. Exactly as you point out, 
the owner could call closeFile before some other thread was 
finished writing. If the implementer of FileHandle fails to take 
this into account, then no, it's not thread-safe.


 I'm going to assume that `shareWithThreads()` was implemented  
 by an
 'expert' who checked the function results for errors. It was 
 detected that the reads/write failed, and an error "failed to 
 read file" was emit, then the function returned promptly.
 The uncertainty of what happens in this program is however
 `shareWithThreads()` handles read/write emitting an error.

 But you can only find out about these errors in 
 `waitForThreads`, the very call that the user "forgot" to make!

Of course not. You can throw exceptions, you could add a 
destructor that reports on these errors, you could set an error 
flag somewhere and check that every now and then. The fact that 
you've managed to write a horribly broken API under MP and can't 
see a way to do better inside that system does not necessarily 
mean the problem is with MP.

--
   Simen

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 5:50 AM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 21 October 2018 at 05:47:14 UTC, Manu wrote:

 And yet, if we're lucky, we get
 a consistent instacrash. If we're unlucky, we get memory
 corruption, or an unsolicited write to another currently open
 file, either of which can go unnoticed for some time.


 Woah! Now this is way off-piste..
 Why would get a crash? Why would get memory corruption? None of
 those things make sense.

 Because the whole reason to have `shared` is to avoid the
 extraneous checks that you mentioned above,

No, it is to assure that you write correct not-broken code.

 and only write actual
 useful code (i.e. lock-write-unlock, or read-put-to-queue-repeat,
 or whatever), not busy-work (testing if the file is open on every
 call).

`shared` is no comment on performance. You have written a slow locking API.
If you care about perf, you would write a completely different API
that favours perf.
This not impossible, nor even particularly hard.

 If you have a `shared` reference, it better be to existing
 data.

I mean, if I dereference a pointer, it had better not be null!

 If it isn't, the program is invalid already: you've shared
 something that doesn't "exist" (good for marketing, not so good
 for multithreading).

I mean, if I dereference a pointer, it had better not be null!

 That's why having `shared` and un-`shared`
 references to the same data simultaneously is not safe: you can't
 guarantee in any way that the owning thread doesn't invalidate
 the data through it's non-`shared` reference while you're doing
 your threadsafe `shared` work; you can only "promise" that by
 convention (documentation).

The owning thread is not a special actor. Your reasoning is wonky here.
Lets say there is no owning instance, only a collection of shared
instances... any one of them can theoretically do an operation that
interferes with the others.
That's issues for general threading, and no design for `shared` can
(or should) interact with that problem. That's a problem for
architecture.

 So, you call closeFile immediately and read/write start
 returning null.

 And I have partially-read or partially-written data.

I expect you flushed before killing the file.

 Or Maybe I
 call closeFile(), main thread continues and opens another file,
 which gives the same file descriptor, `shared` references to
 FileHandle which the user forgot to wait on continue to work
 oblivious to the fact that it's a different file now.

It's wild to suggest that ANY design for `shared` should somehow deal
with the OS recycling a file handle...
And it's still not an un- safe crash! It's just a program with a bug.

 It's a
 horrible, but still  safe, implementation of FileHandle, yes, but
 the caller (user) doesn't know that, and can't know that just
 from the interface. The only advice against that is "don't do
 that", but that's irrespective of your proposal.

No proposal can (or should) address these issues. You're concerned
with an issue that is sooooo far left-field at this stage.
Programming languages don't validate that you wrote a working bug-free program.

 I'm going to assume that `shareWithThreads()` was implemented
 by an
 'expert' who checked the function results for errors. It was
 detected that the reads/write failed, and an error "failed to
 read file" was emit, then the function returned promptly.
 The uncertainty of what happens in this program is however
 `shareWithThreads()` handles read/write emitting an error.

 But you can only find out about these errors in `waitForThreads`,
 the very call that the user "forgot" to make!

...what?

 [ ... snip ... ]

You have to concede defeat at this point.

Destroy my proposal with another legitimately faulty program.


 I understand that. So... it would seem that your proposal focuses
 more on  safe than on threadsafety?

I am trying to achieve  safe-ty _with respect to threadsafety_.

'threadsafety' with respect to "proper program operation" is a job for
programmers, and program architecture.
No language attribute can make your program right.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Sunday, 21 October 2018 at 19:22:45 UTC, Manu wrote:
 On Sun, Oct 21, 2018 at 5:50 AM Stanislav Blinov via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Because the whole reason to have `shared` is to avoid the 
 extraneous checks that you mentioned above,

 No, it is to assure that you write correct not-broken code.

You can do that without `shared`.

 and only write actual useful code (i.e. lock-write-unlock, or 
 read-put-to-queue-repeat, or whatever), not busy-work (testing 
 if the file is open on every call).

 `shared` is no comment on performance. You have written a slow 
 locking API.
 If you care about perf, you would write a completely different 
 API that favours perf.
 This not impossible, nor even particularly hard.

You're conflating your assumptions about the code with the topic 
of this discussion. I can write a million examples and you'll 
still find a million reasons to talk about how they're 
incorrectly implemented, instead of focusing on merits or 
disadvantages of your proposal with the code given as is.

 If you have a `shared` reference, it better be to existing 
 data.

 I mean, if I dereference a pointer, it had better not be null!

Why would you share a null pointer?

 That's why having `shared` and un-`shared`
 references to the same data simultaneously is not safe: you 
 can't guarantee in any way that the owning thread doesn't 
 invalidate
 the data through it's non-`shared` reference while you're doing
 your threadsafe `shared` work; you can only "promise" that by
 convention (documentation).

 The owning thread is not a special actor. Your reasoning is 
 wonky here.

Why have it then at all? If it's not a "special" actor, just make 
all shared data `shared`. But your proposal specifically targets 
the conversion, suggesting you *do* need a special actor.

 And I have partially-read or partially-written data.

 I expect you flushed before killing the file.

So? The threads still weren't done yet.

 Or Maybe I call closeFile(), main thread continues and opens 
 another file,
 which gives the same file descriptor, `shared` references to
 FileHandle which the user forgot to wait on continue to work
 oblivious to the fact that it's a different file now.


 It's wild to suggest that ANY design for `shared` should 
 somehow deal with the OS recycling a file handle...

I'm not suggesting that at all, you've completely misrepresenting 
what I'm saying by splitting a quote.

 And it's still not an un- safe crash! It's just a program with 
 a bug.

Ok, if you say that sticking  safe on code that can partially 
piece together data from unrelated sources is fine, then sure.

 I'm going to assume that `shareWithThreads()` was implemented
 by an 'expert' who checked the function results for errors...



 But you can only find out about these errors in 
 `waitForThreads`, the very call that the user "forgot" to make!

 ...what?

Please suggest another way of handling errors reported by other 
threads. More `shared` state?

 [ ... snip ... ]

 You have to concede defeat at this point.

I agree. No matter how hard I try or how many times I ask you to 
demonstrate, I still fail to see the value in assuming  safe 
implicitly conversion of mutable data to shared. Instead of 
defending your proposal, you chose to attack the opposition. 
You've defeated me, flawless victory.

 Destroy my proposal with another legitimately faulty program.

Is there a point? I post code, you: "nah, that's wrong". Steven 
posts code, you: "nah, that's wrong". Timon posts code, you: 
"nah, that's wrong". Walter posts code, you: "nah, that's 
wrong"... What's right then?

Neia Neutuladh <neia ikeran.org> writes:

On Sat, 20 Oct 2018 22:47:14 -0700, Manu wrote:
 Looking at the meat of the program; you open a file, and distribute it
 to do accesses (I presume?)....
 Naturally, this is a really weird thing to do, because even if the API
 is threadsafe such that it doesn't crash and reads/writes are
 serialised, the sequencing of reads/writes will be random, so I don't
 believe any sane person (let alone an expert) would write this
 program... but moving on.
 Then you wait for them to finish, and close the file.
 
 Fine. You have a file with randomly interleaved data... for whatever
 reason.

I'd expect almost every nontrivial multithreaded program to do this. It's 
called a log file.

You don't need to read data pointed at by a log file, but you do need to 
read the FILE* or the file descriptor.

Database-like things using a journal file might need a shared file for 
both reading and writing.

 So, you call closeFile immediately and read/write start returning null.

You start threads. You give them access to the log file. You wait for the 
threads to exit. Then you close the file.

Manu <turkeyman gmail.com> writes:

On Sat, Oct 20, 2018 at 2:05 AM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/19/2018 11:18 PM, Manu wrote:
 The reason I ask is because, by my definition, if you have:
 int* a;
 shared(int)* b = a;

 While you have 2 numbers that address the same data, it is not actually aliased
 because only `a` can access it.

 They are aliased, by code that believes it is unshared, and code that believes
 it is shared.

This situation could only occur if you do unsafe code badly.
Unlike today, where you must do unsafe code to do any interaction of
any kind, you will be able to do fully-safe interaction with the stack
of tooling.
In that world, any unsafe code and *particularly* where it interacts
with shared will be an obnoxious and scary code smell.
If it was possible to interact with shared safely, it would be
blindingly suspicious when people are likely to be shooting themselves
in the foot.

The situation you describe here is *exactly* what we have right now,
and I'm trying to prevent that.

 This is not going to work

This is an unfair dismissal. Have you tried it? I have.
Write me the rules in a patch that I can take for a drive and
demonstrate what the stack looks like.

 Exclusively distinguishing shared and unshared data is not an interesting
 distinction if shared data has no access.

 Somehow, you still have to find a way to give the shared path access, through a
 gate or a cast or a lock or whatever.

I'm not sure you've understood the proposal.
This is the reason for the implicit conversion. It provides safe
transition. That's why I'm so insistent on it.

 And then it breaks, because two different
 threads are accessing the same data each thinking that data is not shared.

This can only occur if you deliberately violate your  safety, and then mess up.
This is *exactly* the interaction prescribed to shared today. This is
what I'm fixing by making a fully  safe path!

I think you demonstrate here that you haven't understood the reason,
or the semantics of my proposal. I'm not sure how to clarify it, what
can I give you?

Walter Bright <newshound2 digitalmars.com> writes:

On 10/20/2018 11:24 AM, Manu wrote:
 This is an unfair dismissal.

It has nothing at all to do with fairness. It is about what the type system 
guarantees in  safe code. To repeat, the current type system guarantees in
 safe 
code that T* and shared(T)* do not point to the same memory location.

Does your proposal maintain that or not? It's a binary question.

 I'm not sure you've understood the proposal.
 This is the reason for the implicit conversion. It provides safe
 transition.

I don't see any way to make an implicit T* to shared(T)* safe, or vice versa. 
The T* code can create more aliases that the conversion doesn't know about, and 
the shared(T)* code can hand out aliases to other threads. So it all falls to 
pieces. Using a 'scope' qualifier won't work, because 'scope' isn't transitive, 
while shared is, i.e. U** and shared(U*)*.

 I'm not sure how to clarify it, what can I give you?

Write a piece of code that does such an implicit conversion that you argue is 
 safe. Make the code as small as possible. Your example:

 int* a;
 shared(int)* b = a;

This is not safe.

---- Manu's Proposal ---
 safe:
int i;
int* a = &i;
StartNewThread(a); // Compiles! Coder has no idea!

... in the new thread ...
void StartOfNewThread(shared(int)* b) {

     ... we have two threads accessing 'i',
     one thinks it is shared, the other unshared,
     and StartOfNewThread() has no idea and anyone
     writing code for StartOfNewThread() has no way
     to know anything is wrong ...

     lockedIncrement(b);  // Data Race!
}
--- Current D ---
 safe:
int i;
int* a = &i;
StartNewThread(a);   // Danger, Will Robinson! Does Not Compile!
StartNewThread(cast(shared(int)*) a) // Danger, Will Robinson!
                                      // Unsafe Cast! Does Not Compile!
---

Your proposal means that the person writing the lockedIncrement(), which is a 
perfectly reasonable thing to do, simply cannot write it in a way that has a 
 safe interface, because the person writing the lockedIncrement() library 
function has no way to know that the data it receives is actually unshared data.

I.e.  trusted code is obliged to proved a safe interface. Your proposal makes 
that impossible because the compiler would allow unshared data to be implicitly 
typed as shared.

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Sunday, 21 October 2018 at 09:50:09 UTC, Walter Bright wrote:
 On 10/20/2018 11:24 AM, Manu wrote:
 This is an unfair dismissal.

 It has nothing at all to do with fairness. It is about what the 
 type system guarantees in  safe code. To repeat, the current 
 type system guarantees in  safe code that T* and shared(T)* do 
 not point to the same memory location.

 Does your proposal maintain that or not? It's a binary question.

No. Instead, it proposes something more useful: once cast to 
shared(T)*, only thread-safe operations may be performed on it.


 int* a;
 shared(int)* b = a;

 This is not safe.

Under MP, this is perfectly safe - you can do nothing with a 
shared(int)*, except call un- safe, non-thread-safe functions on 
it, which will *fail to compile* under  safe.


 ---- Manu's Proposal ---
  safe:
 int i;
 int* a = &i;
 StartNewThread(a); // Compiles! Coder has no idea!

 ... in the new thread ...
 void StartOfNewThread(shared(int)* b) {

     ... we have two threads accessing 'i',
     one thinks it is shared, the other unshared,
     and StartOfNewThread() has no idea and anyone
     writing code for StartOfNewThread() has no way
     to know anything is wrong ...

     lockedIncrement(b);  // Data Race!
 }

Someone's messed up if they've marked lockedIncrement  safe - 
under MP, it shouldn't be. lockedIncrement is a very low-level 
piece of functionality, and should be  system. It also shouldn't 
take a shared(int)*, but a int*, forcing an unsafe cast and 
making it obvious the code is un safe.

--
   Simen

Manu <turkeyman gmail.com> writes:

On Sun., 21 Oct. 2018, 2:55 am Walter Bright via Digitalmars-d,
<digitalmars-d puremagic.com> wrote:
 On 10/20/2018 11:24 AM, Manu wrote:
 This is an unfair dismissal.

 It has nothing at all to do with fairness. It is about what the type system
 guarantees in  safe code. To repeat, the current type system guarantees in
 safe
 code that T* and shared(T)* do not point to the same memory location.

 Does your proposal maintain that or not? It's a binary question.

By the definition Nick pulled from Wikipedia and posted for you a few
posts back, yes, my proposal satisfies Wikipedia's definition of no
aliasing. I understand that property is critical, and I have carefully
designed for it.

 I'm not sure you've understood the proposal.
 This is the reason for the implicit conversion. It provides safe
 transition.

 I don't see any way to make an implicit T* to shared(T)* safe, or vice versa.
 The T* code can create more aliases that the conversion doesn't know about, and
 the shared(T)* code can hand out aliases to other threads. So it all falls to
 pieces.

T* can't make additional T* aliases on other threads; there can only
be one thread with T*.
shared(T)* can not make a T*.
shared(T)* has no read or write access, so it's not an alias of T* by
Wikipedia's definition.

Only threadsafe functions can do anything to T.
The leap of faith is; some  trusted utility functions at the bottom of
the shared stack makes a promise that it is threadsafe, and must
deliver that promise.
I don't think this is unreasonable; this is the nature of  trusted
functions, they make a promise, and they must keep it.
If the trusted function does not lie, then the chain of trust holds
upwards through the stack.

The are very few such trusted functions in practise. Like, similar to
the number of digits you have.

 Using a 'scope' qualifier won't work, because 'scope' isn't transitive,
 while shared is, i.e. U** and shared(U*)*.

I don't think I depend on scope in any way.
That was an earlier revision of thinking in an older thread.

  > I'm not sure how to clarify it, what can I give you?

 Write a piece of code that does such an implicit conversion that you argue is
  safe. Make the code as small as possible. Your example:

  > int* a;
  > shared(int)* b = a;

 This is not safe.

 ---- Manu's Proposal ---
  safe:
 int i;
 int* a = &i;
 StartNewThread(a); // Compiles! Coder has no idea!

 ... in the new thread ...
 void StartOfNewThread(shared(int)* b) {

      ... we have two threads accessing 'i',
      one thinks it is shared, the other unshared,
      and StartOfNewThread() has no idea and anyone
      writing code for StartOfNewThread() has no way
      to know anything is wrong ...

      lockedIncrement(b);  // Data Race!
 }

This program doesn't compile. You receive an error because it is not safe.
The function is `lockedIncrement(int*)`. It can't receive a shared
argument; the function is not threadsafe by my definition.
You have written a program that produces the expected error that
alerts you that you have tried to do un- safe and make a race.

Stanislav produced this same program, and I responded with the correct
program a few posts back.
I'll repeat it here; the  safe program to model this interaction is:

 safe:

// function is NOT threadsafe by my definition, can not be called on
shared arguments
void atomicIncrement(int*);

struct Atomic(T)
{
  // encapsulare the unsafe data so it's inaccessible by any unsafe means
  private T val;

  // perform the unsafe cast in a trusted function
  // we are able to assure a valid calling context by encapsulating
the data above
  void opUnary(string op : "++")() shared  trusted {
atomicIncrement(cast(T*)&val); }
}

Atomic!int i;
Atomic!int* a = &i;
StartNewThread(a); // Compiles, of course!
++i; // no race

... in the new thread ...
void StartOfNewThread(shared(Atomic!int)* b) {
  ... we have two threads accessing 'i', one has thread-local access,
this one has a restricted shared access.
  here, we have a shared instance, so we can only access `b` via
threadsafe functions.
  as such, we can manipulate `b` without fear.
  ++i; // no race!
}


 Your proposal means that the person writing the lockedIncrement(), which is a
 perfectly reasonable thing to do, simply cannot write it in a way that has a
  safe interface

Correct, the rules of my proposal apply to lockedIncrement(). They
apply to `shared` generally.
lockedIncrement() is not a threadsafe function. You can't call it on a
shared instance, because `int`s API (ie, all intrinsic operations) are
not threadsafe.
lockedIncrement() can't promise threadsafe access to `shared(int)*`,
so the argument is not shared.

Your program made the correct compile error about doing unsafety, but
the location of the compile error is different under my proposal;
complexity is worn by the shared library author, rather than every
calling user ever.
I think my proposal places the complexity in the right location.
`shared` is intrinsically dangerous; it's not reasonable to ask every
user that ever calls a shared API to write unsafe code when when
calling. That's just plain bad design.

 because the person writing the lockedIncrement() library
 function has no way to know that the data it receives is actually unshared
data.

The author of `shared` tooling must assure a valid context such that
its threadsafety promises are true. Atomic(T) does that with a private
member.
The  safe way to interact with atomics is to use the Atomic utility
type I showed above.
That is one such  trusted tool that I talk about as being "at the
bottom of the stack".
It is probably joined by a mutex/semaphore, and some
containers/queues. That is probably all the things, and other things
would be  safe compositions of those tools.

 I.e.  trusted code is obliged to proved a safe interface. Your proposal makes
 that impossible because the compiler would allow unshared data to be implicitly
 typed as shared.

What? No.
Please, try and understand my proposal...

Walter Bright <newshound2 digitalmars.com> writes:

I'd like to add that if the compiler can prove that a T* points to a unique T, 
then it can be implicitly cast to shared(T)*. And it does so, like the result
of 
.dup can be so converted.

12345swordy <alexanderheistermann gmail.com> writes:

On Sunday, 21 October 2018 at 18:45:15 UTC, Walter Bright wrote:
 I'd like to add that if the compiler can prove that a T* points 
 to a unique T, then it can be implicitly cast to shared(T)*. 
 And it does so, like the result of .dup can be so converted.

This can be achieved by using the unique struct and enforce the 
uniqueness at compile time.

https://github.com/dlang/phobos/blob/master/std/typecons.d#L130

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Sunday, 21 October 2018 at 09:50:09 UTC, Walter Bright wrote:
 ---- Manu's Proposal ---
  safe:
 int i;
 int* a = &i;
 StartNewThread(a); // Compiles! Coder has no idea!

 ... in the new thread ...
 void StartOfNewThread(shared(int)* b) {

     ... we have two threads accessing 'i',
     one thinks it is shared, the other unshared,
     and StartOfNewThread() has no idea and anyone
     writing code for StartOfNewThread() has no way
     to know anything is wrong ...

     lockedIncrement(b);  // Data Race!

No, does not compile, lockedIncrement takes an int*
Error cannot convert shared(int)* to int*

 Your proposal means that the person writing the 
 lockedIncrement(), which is a perfectly reasonable thing to do,

Indeed.

 simply cannot write it in a way that has a  safe interface, 
 because the person writing the lockedIncrement() library 
 function has no way to know that the data it receives is 
 actually unshared data.

It does, it takes an int* which is not implicitly convertible to 
given an shared(int)*

 I.e.  trusted code is obliged to proved a safe interface.

Yes.

 Your proposal makes that impossible because the compiler would 
 allow unshared data to be implicitly typed as shared.

Yes, but not the other way around.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Sunday, 21 October 2018 at 19:07:37 UTC, Nicholas Wilson wrote:
 On Sunday, 21 October 2018 at 09:50:09 UTC, Walter Bright wrote:
 Your proposal makes that impossible because the compiler would 
 allow unshared data to be implicitly typed as shared.

 Yes, but not the other way around.

Whoops that should read

 Your proposal makes that impossible


No

 because the compiler would allow unshared data to be 
 implicitly typed as shared.


Yes, but the problem you describe is arises from implicit 
conversion in the other direction, which is not part of the 
proposal.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
 Yes, but the problem you describe is arises from implicit conversion in the 
 other direction, which is not part of the proposal.

It's Manu's example.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/21/2018 2:08 PM, Walter Bright wrote:
 On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
 Yes, but the problem you describe is arises from implicit conversion in the 
 other direction, which is not part of the proposal.

 
 It's Manu's example.

Then I don't know what the proposal is. Pieces of it appear to be scattered
over 
numerous posts, mixed in with other text, opinions, and handwavy stuff. There's 
nothing to point to that is "the proposal".

I suggest you and Manu write up a proper proposal. Something that is complete, 
has nothing else in it, has a rationale, illuminating examples, and explains
why 
alternatives are inferior.

For examples of how to do it:

https://github.com/dlang/DIPs/tree/master/DIPs

Trying to rewrite the semantics of shared is not a simple task, doing 
multithreading correctly is a minefield of "OOPS! I didn't think of that!" and 
if anything cries out for a DIP, your and Manu's proposal does.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Sunday, 21 October 2018 at 21:32:14 UTC, Walter Bright wrote:
 On 10/21/2018 2:08 PM, Walter Bright wrote:
 On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
 Yes, but the problem you describe is arises from implicit 
 conversion in the other direction, which is not part of the 
 proposal.

 
 It's Manu's example.

 Then I don't know what the proposal is. Pieces of it appear to 
 be scattered over numerous posts, mixed in with other text, 
 opinions, and handwavy stuff. There's nothing to point to that 
 is "the proposal".

The proposal is:

Implicit conversion _to_ shared, e.g. passing it to a thread 
entry point, and not implicit conversion _from_ shared (just like 
implicit const conversions).

Shared disables reads and writes

  Your confusion results from the use of atomic add which, in 
Manu's examples had a different signature than before.

 I suggest you and Manu write up a proper proposal. Something 
 that is complete, has nothing else in it, has a rationale, 
 illuminating examples, and explains why alternatives are 
 inferior.

We will eventually. This started as a "please point out any 
problems with this" and has probably outlived that phase.

 Trying to rewrite the semantics of shared is not a simple task,

Not as much as trying to explain it! Having talked to Manu in 
person it is much easier to understand.

 doing multithreading correctly is a minefield of "OOPS! I 
 didn't think of that!"

The above case in point, this is about assuming your 
implementation of thread safe primitives are thread safe 
( trusted) that you use it correctly ( safe).

Walter Bright <newshound2 digitalmars.com> writes:

On 10/21/2018 4:12 PM, Nicholas Wilson wrote:
 On Sunday, 21 October 2018 at 21:32:14 UTC, Walter Bright wrote:
 On 10/21/2018 2:08 PM, Walter Bright wrote:
 On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
 Yes, but the problem you describe is arises from implicit conversion in the 
 other direction, which is not part of the proposal.

 It's Manu's example.

 Then I don't know what the proposal is. Pieces of it appear to be scattered 
 over numerous posts, mixed in with other text, opinions, and handwavy stuff. 
 There's nothing to point to that is "the proposal".

 
 The proposal is:
 
 Implicit conversion _to_ shared, e.g. passing it to a thread entry point, and 
 not implicit conversion _from_ shared (just like implicit const conversions).

That's what I was referring to, and Manu's example. It doesn't work, as I 
pointed out.


 I suggest you and Manu write up a proper proposal. Something that is complete, 
 has nothing else in it, has a rationale, illuminating examples, and explains 
 why alternatives are inferior.

 
 We will eventually. This started as a "please point out any problems with
this" 
 and has probably outlived that phase.

You'll need to address the issues raised here in the DIP.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Monday, 22 October 2018 at 00:46:04 UTC, Walter Bright wrote:
 That's what I was referring to, and Manu's example. It doesn't 
 work, as I pointed out.

I'm pretty sure it does, but please repeat it.

 We will eventually. This started as a "please point out any 
 problems with this" and has probably outlived that phase.

 You'll need to address the issues raised here in the DIP.

That is a given. You would do well to heed it for your own DIPs.

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 5:50 PM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/21/2018 4:12 PM, Nicholas Wilson wrote:
 On Sunday, 21 October 2018 at 21:32:14 UTC, Walter Bright wrote:
 On 10/21/2018 2:08 PM, Walter Bright wrote:
 On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
 Yes, but the problem you describe is arises from implicit conversion in the
 other direction, which is not part of the proposal.

 It's Manu's example.

 Then I don't know what the proposal is. Pieces of it appear to be scattered
 over numerous posts, mixed in with other text, opinions, and handwavy stuff.
 There's nothing to point to that is "the proposal".

 The proposal is:

 Implicit conversion _to_ shared, e.g. passing it to a thread entry point, and
 not implicit conversion _from_ shared (just like implicit const conversions).

 That's what I was referring to, and Manu's example. It doesn't work, as I
 pointed out.


 I suggest you and Manu write up a proper proposal. Something that is complete,
 has nothing else in it, has a rationale, illuminating examples, and explains
 why alternatives are inferior.

 We will eventually. This started as a "please point out any problems with this"
 and has probably outlived that phase.

 You'll need to address the issues raised here in the DIP.

Would you please respond to my messages, and specifically, respond to
the code that I presented to you in response to your broken example.
Or any of my earlier fragments throughout this thread. I've shared
quite a few, and so far, nobody has ever produced a criticism of any
of my fragments. They've just been skipped over.
But the one aimed directly at your own most recent sample program
addresses your program directly.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/21/2018 5:54 PM, Manu wrote:
 Would you please respond to my messages, and specifically, respond to
 the code that I presented to you in response to your broken example.
 Or any of my earlier fragments throughout this thread. I've shared
 quite a few, and so far, nobody has ever produced a criticism of any
 of my fragments. They've just been skipped over.

That's just the problem. You've posted 62 messages so far in this thread, and 
then there's all the ones Nicholas posted.

Trying to assemble the "earlier fragments throughout this thread" is not 
practical for readers, and the endless nature of this thread is ample evidence 
for it. The n.g. is a place to discuss a proposal, not the proposal itself.

This change is definitely merits an actual proposal DIP, so that one is assured 
of seeing the complete proposal, rationale, examples, etc., in one document, as 
well as not being distracted by sidebars, thread drift, and mistakes. This 
document can evolve with corrections and clarifications from the discussion,
and 
anyone can get up to speed quickly by just reading the latest version of it.


 But the one aimed directly at your own most recent sample program
 addresses your program directly.

My most recent sample program was a direct criticism of one of your fragments, 
so please don't say "nobody has ever ...". I do understand your frustration at 
finding it hard to get your point across, but the problem at least for me is 
trying to mine it from nuggets scattered across 62 posts. Mine it, refine it, 
cast it into an ingot, then present it as a DIP.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 22, 2018 at 12:50 AM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/21/2018 5:54 PM, Manu wrote:
 Would you please respond to my messages, and specifically, respond to
 the code that I presented to you in response to your broken example.
 Or any of my earlier fragments throughout this thread. I've shared
 quite a few, and so far, nobody has ever produced a criticism of any
 of my fragments. They've just been skipped over.

 That's just the problem. You've posted 62 messages so far in this thread, and
 then there's all the ones Nicholas posted.

I sent it twice... again just a short while ago right before this
one... but you responded to this one and not that one O_o

 Trying to assemble the "earlier fragments throughout this thread" is not
 practical for readers, and the endless nature of this thread is ample evidence
 for it. The n.g. is a place to discuss a proposal, not the proposal itself.

 This change is definitely merits an actual proposal DIP, so that one is assured
 of seeing the complete proposal, rationale, examples, etc., in one document, as
 well as not being distracted by sidebars, thread drift, and mistakes. This
 document can evolve with corrections and clarifications from the discussion,
and
 anyone can get up to speed quickly by just reading the latest version of it.

Okay, but I still want you to respond to my corrections of your
program, which were in direct response to you... twice.

  > But the one aimed directly at your own most recent sample program
  > addresses your program directly.

 My most recent sample program was a direct criticism of one of your fragments,
 so please don't say "nobody has ever ...". I do understand your frustration at
 finding it hard to get your point across, but the problem at least for me is
 trying to mine it from nuggets scattered across 62 posts. Mine it, refine it,
 cast it into an ingot, then present it as a DIP.

I posted it, twice... 2 messages, back to back, and you're responding
to this one, and not that one. I'll post it again...

Walter Bright <newshound2 digitalmars.com> writes:

On 10/22/2018 1:34 AM, Manu wrote:
 I posted it, twice... 2 messages, back to back, and you're responding
 to this one, and not that one. I'll post it again...


Posting it over and over is illustrative of the failure of posting proposal 
documents to the n.g. instead of posting it as a DIP which can be referred to:

1. nobody knows which of your 70 messages are the ones with the proposal in it

2. with multiple posts of the proposal, nobody knows which one is the most 
up-to-date one

Doing it this way does not work. Continuing to repost it is a waste of your 
time. Post it as a DIP and link to it.

rikki cattermole <rikki cattermole.co.nz> writes:

On 22/10/2018 10:28 PM, Walter Bright wrote:
 On 10/22/2018 1:34 AM, Manu wrote:
 I posted it, twice... 2 messages, back to back, and you're responding
 to this one, and not that one. I'll post it again...

 
 
 Posting it over and over is illustrative of the failure of posting 
 proposal documents to the n.g. instead of posting it as a DIP which can 
 be referred to:
 
 1. nobody knows which of your 70 messages are the ones with the proposal 
 in it
 
 2. with multiple posts of the proposal, nobody knows which one is the 
 most up-to-date one
 
 Doing it this way does not work. Continuing to repost it is a waste of 
 your time. Post it as a DIP and link to it.

As I've said previously, it doesn't need to be a good DIP or anywhere 
near complete. It just needs code examples comparing current and 
proposed behavior with some text about semantics changes.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/22/2018 2:31 AM, rikki cattermole wrote:
 As I've said previously, it doesn't need to be a good DIP or anywhere near 
 complete. It just needs code examples comparing current and proposed behavior 
 with some text about semantics changes.

That's right, and it can be evolved so everyone involved can easily see what
the 
latest is, and not get mired in and distracted by resolved issues.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 22, 2018 at 2:30 AM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/22/2018 1:34 AM, Manu wrote:
 I posted it, twice... 2 messages, back to back, and you're responding
 to this one, and not that one. I'll post it again...


 Posting it over and over is illustrative of the failure of posting proposal
 documents to the n.g. instead of posting it as a DIP which can be referred to:

 1. nobody knows which of your 70 messages are the ones with the proposal in it

 2. with multiple posts of the proposal, nobody knows which one is the most
 up-to-date one

It hasn't changed. Not one single bit. I haven't changed a single
detail in this thread.

 Doing it this way does not work. Continuing to repost it is a waste of your
 time. Post it as a DIP and link to it.

And you STILL ignored my post >_<
Please look at the proper code that implements your broken example. I
know you've seen it now.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/22/2018 2:44 AM, Manu wrote:
 It hasn't changed. Not one single bit. I haven't changed a single
 detail in this thread.

A reader would not know that. Reposting the same thing over and over in a
thread 
is not how threaded discussions work. You've indicated previously you're not 
even using a threaded newsreader.


 And you STILL ignored my post >_<

I'm going to until you post a DIP. The way this discussion is going is 
incredibly inefficient of your time and mine, and everyone else's who tries to 
engage constructively. It's exactly why we have a DIP process.

It's really not hard. You can even use Markdown to make it look much nicer.

https://github.com/dlang/DIPs

https://github.com/dlang/DIPs/blob/master/PROCEDURE.md

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/21/2018 2:08 PM, Walter Bright wrote:
 On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
 Yes, but the problem you describe is arises from implicit conversion in the
 other direction, which is not part of the proposal.

 It's Manu's example.

 Then I don't know what the proposal is. Pieces of it appear to be scattered
over
 numerous posts, mixed in with other text,

No no, they're repeated, not scattered, because I seem to have to keep
repeating it over and over, because nobody is reading the text, or
perhaps imaging there is a lot more text than there is.

 opinions, and handwavy stuff.

You mean like every post in opposition which disregards the rules and
baselessly asserts it's a terrible idea? :/

 There's nothing to point to that is "the proposal".

You can go back to the OP, not a single detail is changed at any
point, but I've repeated it a whole bunch of times (including in
direct response to your last post) and the summary has become more
concise, but not different.

1. Shared has no read or write access to data
2. Functions with shared arguments are threadsafe with respect to
those arguments
  a. This is a commitment that must be true in _absolute terms_ (there
exists discussion about the ways that neighbours must not undermine
this promise)
  b. There are numerous examples demonstrating how to configure this
(TL;DR: use encapsulation, and  trusted at the bottom of the stack)

If you can find a legitimate example where those rules don't hold, I
want to see it.
But every example so far has been based on a faulty premise where
those 2 simple rules were not actually applied.

I responded to your faulty program directly with the correct program,
and you haven't acknowledged it. Did you see it?

 I suggest you and Manu write up a proper proposal. Something that is complete,
 has nothing else in it, has a rationale, illuminating examples, and explains
why
 alternatives are inferior.

I have written this program a couple of times, including in direct
response to your last sample program.
You seem to have dismissed it... where is your response to that
program, or my last entire post?

 For examples of how to do it:

 https://github.com/dlang/DIPs/tree/master/DIPs

 Trying to rewrite the semantics of shared is not a simple task, doing
 multithreading correctly is a minefield of "OOPS! I didn't think of that!" and
 if anything cries out for a DIP, your and Manu's proposal does.

Yes, I agree it's DIP worthy. But given the almost nothing but overt
hostility I've received here, why on earth would I waste my time
writing a DIP?
I put months into my other DIP which sits gathering dust... if this
thread inspired any confidence that it would be well-received I would
make the effort, but the critical reception we've seen here is... a
bit strange.
It's a 2-point proposal, the rules are **SO SIMPLE**, which is why I
love it. How it can be misunderstood is something I'm having trouble
understanding, and I don't know how to make it any clearer than I
already have; numerous times over, including in my last reply to you,
which you have ignored and dismissed it seems.

Please go back and read my response to your last program.

Joakim <dlang joakim.fea.st> writes:

On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
 On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On 10/21/2018 2:08 PM, Walter Bright wrote:
 On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
 Yes, but the problem you describe is arises from implicit 
 conversion in the other direction, which is not part of the 
 proposal.

 It's Manu's example.

 Then I don't know what the proposal is. Pieces of it appear to 
 be scattered over numerous posts, mixed in with other text,

 No no, they're repeated, not scattered, because I seem to have 
 to keep repeating it over and over, because nobody is reading 
 the text, or perhaps imaging there is a lot more text than 
 there is.

I told you this is what happens with forum posts 4 days ago, yet 
you didn't listen:

https://forum.dlang.org/post/fokdcnzircoiuhrhzmgv forum.dlang.org

 opinions, and handwavy stuff.

 You mean like every post in opposition which disregards the 
 rules and baselessly asserts it's a terrible idea? :/

 There's nothing to point to that is "the proposal".

 You can go back to the OP, not a single detail is changed at 
 any point, but I've repeated it a whole bunch of times 
 (including in direct response to your last post) and the 
 summary has become more concise, but not different.

 1. Shared has no read or write access to data
 2. Functions with shared arguments are threadsafe with respect 
 to
 those arguments
   a. This is a commitment that must be true in _absolute terms_ 
 (there
 exists discussion about the ways that neighbours must not 
 undermine
 this promise)
   b. There are numerous examples demonstrating how to configure 
 this
 (TL;DR: use encapsulation, and  trusted at the bottom of the 
 stack)

 If you can find a legitimate example where those rules don't 
 hold, I
 want to see it.
 But every example so far has been based on a faulty premise 
 where
 those 2 simple rules were not actually applied.

Put it all together in a 2-3 page proposal elsewhere, so he 
doesn't have to hunt everything out in a blizzard of forum posts.

 I responded to your faulty program directly with the correct 
 program, and you haven't acknowledged it. Did you see it?

 I suggest you and Manu write up a proper proposal. Something 
 that is complete, has nothing else in it, has a rationale, 
 illuminating examples, and explains why alternatives are 
 inferior.

 I have written this program a couple of times, including in 
 direct
 response to your last sample program.
 You seem to have dismissed it... where is your response to that
 program, or my last entire post?

 For examples of how to do it:

 https://github.com/dlang/DIPs/tree/master/DIPs

 Trying to rewrite the semantics of shared is not a simple 
 task, doing multithreading correctly is a minefield of "OOPS! 
 I didn't think of that!" and if anything cries out for a DIP, 
 your and Manu's proposal does.

 Yes, I agree it's DIP worthy. But given the almost nothing but 
 overt
 hostility I've received here, why on earth would I waste my time
 writing a DIP?
 I put months into my other DIP which sits gathering dust... if 
 this
 thread inspired any confidence that it would be well-received I 
 would
 make the effort, but the critical reception we've seen here 
 is... a
 bit strange.
 It's a 2-point proposal, the rules are **SO SIMPLE**, which is 
 why I
 love it. How it can be misunderstood is something I'm having 
 trouble
 understanding, and I don't know how to make it any clearer than 
 I
 already have; numerous times over, including in my last reply 
 to you,
 which you have ignored and dismissed it seems.

 Please go back and read my response to your last program.

He did not say to write a full DIP, just a proposal, so he knows 
exactly what you mean, just as I said. It will require a DIP 
eventually, but he didn't ask you to write one now.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:

 No no, they're repeated, not scattered, because I seem to have 
 to keep repeating it over and over, because nobody is reading 
 the text, or perhaps imaging there is a lot more text than 
 there is.
 ...
 You mean like every post in opposition which disregards the 
 rules and baselessly asserts it's a terrible idea? :/
 ...
 I responded to your faulty program directly with the correct 
 program, and you haven't acknowledged it. Did you see it?

Right back at you.

Quote:

I think this is a typical sort of construction:

struct ThreadsafeQueue(T)
{
   void QueueItem(T*) shared;
   T* UnqueueItem() shared;
}

struct SpecialWorkList
{
   struct Job { ... }

   void MakeJob(int x, float y, string z) shared  // <- any thread 
may
produce a job
   {
     Job* job = new Job; // <- this is thread-local
     PopulateJob(job, x, y, z); // <- preparation of a job might be
complex, and worthy of the SpecialWorkList implementation

     jobList.QueueItem(job);  // <- QueueItem encapsulates
thread-safety, no need for blunt casts
   }

   void Flush() // <- not shared, thread-local consumer
   {
     Job* job;
     while (job = jobList.UnqueueItem()) // <- it's obviously safe 
for
a thread-local to call UnqueueItem even though the implementation 
is
threadsafe
     {
       // thread-local dispatch of work...
       // perhaps rendering, perhaps deferred destruction, perhaps
deferred resource creation... whatever!
     }
   }

   void GetSpecialSystemState() // <- this has NOTHING to do with 
the
threadsafe part of SpecialWorkList
   {
     return os.functionThatChecksSystemState();
   }

   // there may be any number of utility functions that don't 
interact
with jobList.

private:
   void PopulateJob(ref Job job, ...)
   {
     // expensive function; not thread-safe, and doesn't have any
interaction with threading.
   }

   ThreadsafeQueue!Job jobList;
}


This isn't an amazing example, but it's typical of a thing that's
mostly thread-local, and only a small controlled part of it's
functionality is thread-safe.
The thread-local method Flush() also deals with thread-safety
internally... because it flushes a thread-safe queue.

All thread-safety concerns are composed by a utility object, so 
there's no need for locks, magic, or casts here.

EndQuote;

The above:
1) Will not compile, not currently, not under your proposal 
(presumably you forgot in frustration to cast before calling 
PopulateJob?..)
2) Does not in any way demonstrate a practical  safe application 
of an implicit conversion. As I wrote in the original response to 
that code, with that particular code it seems more like you just 
need forwarding methods that call `shared` methods under the hood 
(i.e. MakeJob), and it'd be "nice" if you didn't have to write 
those and could just call `shared` MakeJob on an un-`shared` 
reference directly. But these are all assumptions without seeing 
the actual usage.

Please just stop acting like everyone here is opposing *you*. All 
you're doing is dismissing everyone with a "nuh-huh, you no 
understand, you bad". If it was just me, fine, it would mean I'm 
dumb and not worthy of this discussion. But this isn't the case, 
which means *you are not getting your point across*. And yet 
instead of trying to fix that, you're getting all snarky.

Manu <turkeyman gmail.com> writes:

On Mon, Oct 22, 2018 at 4:50 AM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:

 No no, they're repeated, not scattered, because I seem to have
 to keep repeating it over and over, because nobody is reading
 the text, or perhaps imaging there is a lot more text than
 there is.
 ...
 You mean like every post in opposition which disregards the
 rules and baselessly asserts it's a terrible idea? :/
 ...
 I responded to your faulty program directly with the correct
 program, and you haven't acknowledged it. Did you see it?

 Right back at you.

 Quote:

 I think this is a typical sort of construction:

 struct ThreadsafeQueue(T)
 {
    void QueueItem(T*) shared;
    T* UnqueueItem() shared;
 }

 struct SpecialWorkList
 {
    struct Job { ... }

    void MakeJob(int x, float y, string z) shared  // <- any thread
 may
 produce a job
    {
      Job* job = new Job; // <- this is thread-local
      PopulateJob(job, x, y, z); // <- preparation of a job might be
 complex, and worthy of the SpecialWorkList implementation

      jobList.QueueItem(job);  // <- QueueItem encapsulates
 thread-safety, no need for blunt casts
    }

    void Flush() // <- not shared, thread-local consumer
    {
      Job* job;
      while (job = jobList.UnqueueItem()) // <- it's obviously safe
 for
 a thread-local to call UnqueueItem even though the implementation
 is
 threadsafe
      {
        // thread-local dispatch of work...
        // perhaps rendering, perhaps deferred destruction, perhaps
 deferred resource creation... whatever!
      }
    }

    void GetSpecialSystemState() // <- this has NOTHING to do with
 the
 threadsafe part of SpecialWorkList
    {
      return os.functionThatChecksSystemState();
    }

    // there may be any number of utility functions that don't
 interact
 with jobList.

 private:
    void PopulateJob(ref Job job, ...)
    {
      // expensive function; not thread-safe, and doesn't have any
 interaction with threading.
    }

    ThreadsafeQueue!Job jobList;
 }


 This isn't an amazing example, but it's typical of a thing that's
 mostly thread-local, and only a small controlled part of it's
 functionality is thread-safe.
 The thread-local method Flush() also deals with thread-safety
 internally... because it flushes a thread-safe queue.

 All thread-safety concerns are composed by a utility object, so
 there's no need for locks, magic, or casts here.

 EndQuote;

 The above:
 1) Will not compile, not currently, not under your proposal
 (presumably you forgot in frustration to cast before calling
 PopulateJob?..)

I did correct that line (along with an apology) on my very next post;
it would probably be a member of Job... or any manner of other code.
That is the least interesting line in the program

 2) Does not in any way demonstrate a practical  safe application
 of an implicit conversion. As I wrote in the original response to
 that code, with that particular code it seems more like you just
 need forwarding methods that call `shared` methods under the hood
 (i.e. MakeJob), and it'd be "nice" if you didn't have to write
 those and could just call `shared` MakeJob on an un-`shared`
 reference directly. But these are all assumptions without seeing
 the actual usage.

 Please just stop acting like everyone here is opposing *you*.

You're right, it's mostly you.

  All
 you're doing is dismissing everyone with a "nuh-huh, you no
 understand, you bad". If it was just me, fine, it would mean I'm
 dumb and not worthy of this discussion. But this isn't the case,
 which means *you are not getting your point across*. And yet
 instead of trying to fix that, you're getting all snarky.

I mean, it's fair, but it's pretty bloody hypocritical coming from you!
I think it's fair to point out that your high-frequency, persistent,

least, until I told you to GF) is the primary reason I'm frustrated
here.
You can own part responsibility for my emotion.

Atila Neves <atila.neves gmail.com> writes:

On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
 On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 Then I don't know what the proposal is. Pieces of it appear to 
 be scattered over numerous posts, mixed in with other text,

 No no, they're repeated, not scattered, because I seem to have 
 to keep repeating it over and over, because nobody is reading 
 the text, or perhaps imaging there is a lot more text than 
 there is.

I've read every post on this thread and I also have the feeling 
that it's scattered. At the very least, I'm 90% confident I don't 
understand what it is you're proposing. Trust me, I'm trying.

I believe that you have a proposal which you believe results in 
 safe multithreaded code. I don't understand how what I've read 
so far would accomplish that. I'm conviced that shared data 
shouldn't be allowed to be written to, but I haven't yet been 
convinced of anything else.

I don't see how it's possible that implicit conversion from 
non-shared to shared can work at all. Yes, I know that in the 
proposal putting `shared` on anything makes it useless, but 
*somehow* that data gets to be used, even if it's by a  trusted 
function that casts away shared. At that point, nothing you do 
thread-safely to the shared data matters if you obtained the 
shared data from an implicit conversion. There may be many many 
aliases to it before it was converted, all of them able to write 
to that memory location believing it's not shared. And it would 
be  safe (but definitely not thread-safe) to do so! This has been 
explained a few times, by multiple people. I haven't seen anyone 
addressing this yet (it's possible it got lost in a sea of text).

I don't even understand why it is you want to cast anything to 
shared anyway - that'd always be a code smell if I saw it during 
code review. If it's shared, type it as such. Or better yet, if 
you can just use immutable.

I understand the frustration of not getting your point across. I 
would like to kindly point out that, if the replies have gotten 
to multiple dozen pages and several well-meaning people still 
don't get it, then the proposal probably isn't as simple as you 
believe it to be.

John Colvin <john.loughran.colvin gmail.com> writes:

On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
 On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On 10/21/2018 2:08 PM, Walter Bright wrote:
 On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
 Yes, but the problem you describe is arises from implicit 
 conversion in the other direction, which is not part of the 
 proposal.

 It's Manu's example.

 Then I don't know what the proposal is. Pieces of it appear to 
 be scattered over numerous posts, mixed in with other text,

 No no, they're repeated, not scattered, because I seem to have 
 to keep repeating it over and over, because nobody is reading 
 the text, or perhaps imaging there is a lot more text than 
 there is.

I can go look at the original post - and I have - but while it 
may strictly speaking contain all the information I need, the 
amount of time and brain power it would take for me to comprehend 
the consequences is pretty large. I assume you have done a lot of 
that work already and could save everyone a lot of time by 
putting together a quick document that covers some of that, with 
good examples. I'm not going to read {1,2,3}00 messages full of 
irritated bidirectional miscommunication to try and understand 
this unless I really have to, and I assume others feel similarly.

Manu <turkeyman gmail.com> writes:

On Sun, Oct 21, 2018 at 11:31 AM Manu <turkeyman gmail.com> wrote:
 On Sun., 21 Oct. 2018, 2:55 am Walter Bright via Digitalmars-d,
 <digitalmars-d puremagic.com> wrote:
 On 10/20/2018 11:24 AM, Manu wrote:
 This is an unfair dismissal.

 It has nothing at all to do with fairness. It is about what the type system
 guarantees in  safe code. To repeat, the current type system guarantees in
 safe
 code that T* and shared(T)* do not point to the same memory location.

 Does your proposal maintain that or not? It's a binary question.

 By the definition Nick pulled from Wikipedia and posted for you a few
 posts back, yes, my proposal satisfies Wikipedia's definition of no
 aliasing. I understand that property is critical, and I have carefully
 designed for it.

 I'm not sure you've understood the proposal.
 This is the reason for the implicit conversion. It provides safe
 transition.

 I don't see any way to make an implicit T* to shared(T)* safe, or vice versa.
 The T* code can create more aliases that the conversion doesn't know about, and
 the shared(T)* code can hand out aliases to other threads. So it all falls to
 pieces.

 T* can't make additional T* aliases on other threads; there can only
 be one thread with T*.
 shared(T)* can not make a T*.
 shared(T)* has no read or write access, so it's not an alias of T* by
 Wikipedia's definition.

 Only threadsafe functions can do anything to T.
 The leap of faith is; some  trusted utility functions at the bottom of
 the shared stack makes a promise that it is threadsafe, and must
 deliver that promise.
 I don't think this is unreasonable; this is the nature of  trusted
 functions, they make a promise, and they must keep it.
 If the trusted function does not lie, then the chain of trust holds
 upwards through the stack.

 The are very few such trusted functions in practise. Like, similar to
 the number of digits you have.

 Using a 'scope' qualifier won't work, because 'scope' isn't transitive,
 while shared is, i.e. U** and shared(U*)*.

 I don't think I depend on scope in any way.
 That was an earlier revision of thinking in an older thread.

  > I'm not sure how to clarify it, what can I give you?

 Write a piece of code that does such an implicit conversion that you argue is
  safe. Make the code as small as possible. Your example:

  > int* a;
  > shared(int)* b = a;

 This is not safe.

 ---- Manu's Proposal ---
  safe:
 int i;
 int* a = &i;
 StartNewThread(a); // Compiles! Coder has no idea!

 ... in the new thread ...
 void StartOfNewThread(shared(int)* b) {

      ... we have two threads accessing 'i',
      one thinks it is shared, the other unshared,
      and StartOfNewThread() has no idea and anyone
      writing code for StartOfNewThread() has no way
      to know anything is wrong ...

      lockedIncrement(b);  // Data Race!
 }

 This program doesn't compile. You receive an error because it is not safe.
 The function is `lockedIncrement(int*)`. It can't receive a shared
 argument; the function is not threadsafe by my definition.
 You have written a program that produces the expected error that
 alerts you that you have tried to do un- safe and make a race.

 Stanislav produced this same program, and I responded with the correct
 program a few posts back.
 I'll repeat it here; the  safe program to model this interaction is:

  safe:

 // function is NOT threadsafe by my definition, can not be called on
 shared arguments
 void atomicIncrement(int*);

 struct Atomic(T)
 {
   // encapsulare the unsafe data so it's inaccessible by any unsafe means
   private T val;

   // perform the unsafe cast in a trusted function
   // we are able to assure a valid calling context by encapsulating
 the data above
   void opUnary(string op : "++")() shared  trusted {
 atomicIncrement(cast(T*)&val); }
 }

 Atomic!int i;
 Atomic!int* a = &i;
 StartNewThread(a); // Compiles, of course!
 ++i; // no race

 ... in the new thread ...
 void StartOfNewThread(shared(Atomic!int)* b) {
   ... we have two threads accessing 'i', one has thread-local access,
 this one has a restricted shared access.
   here, we have a shared instance, so we can only access `b` via
 threadsafe functions.
   as such, we can manipulate `b` without fear.
   ++i; // no race!
 }


 Your proposal means that the person writing the lockedIncrement(), which is a
 perfectly reasonable thing to do, simply cannot write it in a way that has a
  safe interface

 Correct, the rules of my proposal apply to lockedIncrement(). They
 apply to `shared` generally.
 lockedIncrement() is not a threadsafe function. You can't call it on a
 shared instance, because `int`s API (ie, all intrinsic operations) are
 not threadsafe.
 lockedIncrement() can't promise threadsafe access to `shared(int)*`,
 so the argument is not shared.

 Your program made the correct compile error about doing unsafety, but
 the location of the compile error is different under my proposal;
 complexity is worn by the shared library author, rather than every
 calling user ever.
 I think my proposal places the complexity in the right location.
 `shared` is intrinsically dangerous; it's not reasonable to ask every
 user that ever calls a shared API to write unsafe code when when
 calling. That's just plain bad design.

 because the person writing the lockedIncrement() library
 function has no way to know that the data it receives is actually unshared
data.

 The author of `shared` tooling must assure a valid context such that
 its threadsafety promises are true. Atomic(T) does that with a private
 member.
 The  safe way to interact with atomics is to use the Atomic utility
 type I showed above.
 That is one such  trusted tool that I talk about as being "at the
 bottom of the stack".
 It is probably joined by a mutex/semaphore, and some
 containers/queues. That is probably all the things, and other things
 would be  safe compositions of those tools.

 I.e.  trusted code is obliged to proved a safe interface. Your proposal makes
 that impossible because the compiler would allow unshared data to be implicitly
 typed as shared.

 What? No.
 Please, try and understand my proposal...


Did you read this email? It seems you didn't read this email... Please read it.

Manu <turkeyman gmail.com> writes:

Third time's the charm.... maybe?

--------------------- repeated, 3rd time ------------------------

On Sun., 21 Oct. 2018, 2:55 am Walter Bright via Digitalmars-d,
<digitalmars-d puremagic.com> wrote:
 On 10/20/2018 11:24 AM, Manu wrote:
 This is an unfair dismissal.

 It has nothing at all to do with fairness. It is about what the type system
 guarantees in  safe code. To repeat, the current type system guarantees in
 safe
 code that T* and shared(T)* do not point to the same memory location.

 Does your proposal maintain that or not? It's a binary question.

By the definition Nick pulled from Wikipedia and posted for you a few
posts back, yes, my proposal satisfies Wikipedia's definition of no
aliasing. I understand that property is critical, and I have carefully
designed for it.

 I'm not sure you've understood the proposal.
 This is the reason for the implicit conversion. It provides safe
 transition.

 I don't see any way to make an implicit T* to shared(T)* safe, or vice versa.
 The T* code can create more aliases that the conversion doesn't know about, and
 the shared(T)* code can hand out aliases to other threads. So it all falls to
 pieces.

T* can't make additional T* aliases on other threads; there can only
be one thread with T*.
shared(T)* can not make a T*.
shared(T)* has no read or write access, so it's not an alias of T* by
Wikipedia's definition.

Only threadsafe functions can do anything to T.
The leap of faith is; some  trusted utility functions at the bottom of
the shared stack makes a promise that it is threadsafe, and must
deliver that promise.
I don't think this is unreasonable; this is the nature of  trusted
functions, they make a promise, and they must keep it.
If the trusted function does not lie, then the chain of trust holds
upwards through the stack.

The are very few such trusted functions in practise. Like, similar to
the number of digits you have.

 Using a 'scope' qualifier won't work, because 'scope' isn't transitive,
 while shared is, i.e. U** and shared(U*)*.

I don't think I depend on scope in any way.
That was an earlier revision of thinking in an older thread.

  > I'm not sure how to clarify it, what can I give you?

 Write a piece of code that does such an implicit conversion that you argue is
  safe. Make the code as small as possible. Your example:

  > int* a;
  > shared(int)* b = a;

 This is not safe.

 ---- Manu's Proposal ---
  safe:
 int i;
 int* a = &i;
 StartNewThread(a); // Compiles! Coder has no idea!

 ... in the new thread ...
 void StartOfNewThread(shared(int)* b) {

      ... we have two threads accessing 'i',
      one thinks it is shared, the other unshared,
      and StartOfNewThread() has no idea and anyone
      writing code for StartOfNewThread() has no way
      to know anything is wrong ...

      lockedIncrement(b);  // Data Race!
 }

This program doesn't compile. You receive an error because it is not safe.
The function is `lockedIncrement(int*)`. It can't receive a shared
argument; the function is not threadsafe by my definition.
You have written a program that produces the expected error that
alerts you that you have tried to do un- safe and make a race.

Stanislav produced this same program, and I responded with the correct
program a few posts back.
I'll repeat it here; the  safe program to model this interaction is:

 safe:

// function is NOT threadsafe by my definition, can not be called on
shared arguments
void atomicIncrement(int*);

struct Atomic(T)
{
  // encapsulare the unsafe data so it's inaccessible by any unsafe means
  private T val;

  // perform the unsafe cast in a trusted function
  // we are able to assure a valid calling context by encapsulating
the data above
  void opUnary(string op : "++")() shared  trusted {
atomicIncrement(cast(T*)&val); }
}

Atomic!int i;
Atomic!int* a = &i;
StartNewThread(a); // Compiles, of course!
++i; // no race

... in the new thread ...
void StartOfNewThread(shared(Atomic!int)* b) {
  //... we have two threads accessing 'i', one has thread-local
access, this one has a restricted shared access.
  // here, we have a shared instance, so we can only access `b` via
threadsafe functions.
  // as such, we can manipulate `b` without fear.
  ++i; // no race!
}


 Your proposal means that the person writing the lockedIncrement(), which is a
 perfectly reasonable thing to do, simply cannot write it in a way that has a
  safe interface

Correct, the rules of my proposal apply to lockedIncrement(). They
apply to `shared` generally.
lockedIncrement() is not a threadsafe function. You can't call it on a
shared instance, because `int`s API (ie, all intrinsic operations) are
not threadsafe.
lockedIncrement() can't promise threadsafe access to `shared(int)*`,
so the argument is not shared.

Your program made the correct compile error about doing unsafety, but
the location of the compile error is different under my proposal;
complexity is worn by the shared library author, rather than every
calling user ever.
I think my proposal places the complexity in the right location.
`shared` is intrinsically dangerous; it's not reasonable to ask every
user that ever calls a shared API to write unsafe code when when
calling. That's just plain bad design.

 because the person writing the lockedIncrement() library
 function has no way to know that the data it receives is actually unshared
data.

The author of `shared` tooling must assure a valid context such that
its threadsafety promises are true. Atomic(T) does that with a private
member.
The  safe way to interact with atomics is to use the Atomic utility
type I showed above.
That is one such  trusted tool that I talk about as being "at the
bottom of the stack".
It is probably joined by a mutex/semaphore, and some
containers/queues. That is probably all the things, and other things
would be  safe compositions of those tools.

 I.e.  trusted code is obliged to proved a safe interface. Your proposal makes
 that impossible because the compiler would allow unshared data to be implicitly
 typed as shared.

What? No.
Please, try and understand my proposal...

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright 
wrote:

 When Andrei and I came up with the rules for:

    mutable
    const
    shared
    const shared
    immutable

 and which can be implicitly converted to what, so far nobody 
 has found a fault in those rules...

Here's one: shared -> const shared shouldn't be allowed. Mutable 
aliasing in single-threaded code is bad enough as it is.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Wednesday, 17 October 2018 at 07:24:13 UTC, Stanislav Blinov 
wrote:
 On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright 
 wrote:

 When Andrei and I came up with the rules for:

    mutable
    const
    shared
    const shared
    immutable

 and which can be implicitly converted to what, so far nobody 
 has found a fault in those rules...

 Here's one: shared -> const shared shouldn't be allowed. 
 Mutable aliasing in single-threaded code is bad enough as it is.

Could you explain that a bit more? I don't understand it, mutable 
-> const is half the reason const exists.

I was thinking that mutable -> shared const as apposed to mutable 
-> shared would get around the issues that Timon posted.

Timon Gehr <timon.gehr gmx.ch> writes:

On 17.10.2018 16:14, Nicholas Wilson wrote:
 
 I was thinking that mutable -> shared const as apposed to mutable -> 
 shared would get around the issues that Timon posted.

Unfortunately not. For example, the thread with the mutable reference is 
not obliged to actually make the changes that are performed on that 
reference visible to other threads.

Nicholas Wilson <iamthewilsonator hotmail.com> writes:

On Wednesday, 17 October 2018 at 14:26:43 UTC, Timon Gehr wrote:
 On 17.10.2018 16:14, Nicholas Wilson wrote:
 
 I was thinking that mutable -> shared const as apposed to 
 mutable -> shared would get around the issues that Timon 
 posted.

 Unfortunately not. For example, the thread with the mutable 
 reference is not obliged to actually make the changes that are 
 performed on that reference visible to other threads.

Yes, but that is covered by not being able to read non-atomically 
from a shared reference.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/17/18 10:33 AM, Nicholas Wilson wrote:
 On Wednesday, 17 October 2018 at 14:26:43 UTC, Timon Gehr wrote:
 On 17.10.2018 16:14, Nicholas Wilson wrote:
 I was thinking that mutable -> shared const as apposed to mutable -> 
 shared would get around the issues that Timon posted.

 Unfortunately not. For example, the thread with the mutable reference 
 is not obliged to actually make the changes that are performed on that 
 reference visible to other threads.

 
 Yes, but that is covered by not being able to read non-atomically from a 
 shared reference.

All sides must participate in synchronization for it to make sense. The 
mutable side has no obligation to use atomics. It can use ++data, and 
race conditions will happen.

-Steve

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Wednesday, 17 October 2018 at 14:14:56 UTC, Nicholas Wilson 
wrote:
 On Wednesday, 17 October 2018 at 07:24:13 UTC, Stanislav Blinov 
 wrote:
 On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright 
 wrote:

 When Andrei and I came up with the rules for:

    mutable
    const
    shared
    const shared
    immutable

 and which can be implicitly converted to what, so far nobody 
 has found a fault in those rules...

 Here's one: shared -> const shared shouldn't be allowed. 
 Mutable aliasing in single-threaded code is bad enough as it 
 is.

 Could you explain that a bit more? I don't understand it, 
 mutable -> const is half the reason const exists.

Yes, but `shared` is not `const`. This 'might change' rule is 
poisonous for shared data: you essentially create extra work for 
the CPU for very little gain. There's absolutely no reason to 
share a read-only half-constant anyway. Either give immutable, or 
just flat out copy.

 I was thinking that mutable -> shared const as apposed to 
 mutable -> shared would get around the issues that Timon posted.

It wouldn't, as Timon already explained. Writes to not-`shared` 
mutables might just not be propagated beyond the core. Wouldn't 
happen on amd64, of course, but still. It's not about atomicity 
of reads, it just depends on architecture. On some systems you 
have to explicitly synchronize memory.

jmh530 <john.michael.hall gmail.com> writes:

On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright 
wrote:
 On 10/15/2018 11:46 AM, Manu wrote:
 [...]

 Shared has one incredibly valuable feature - it allows you, the 
 programmer, to identify data that can be accessed by multiple 
 threads. There are so many ways that data can be shared, the 
 only way to comprehend what is going on is to build a wall 
 around shared data.

 (The exception to this is immutable data. Immutable data does 
 not need synchronization, so there is no need to distinguish 
 between shared and unshared immutable data.)
 [snip]


Isn't that also true for isolated data (data that only allows one 
alias)?

Walter Bright <newshound2 digitalmars.com> writes:

On 10/17/2018 4:29 AM, jmh530 wrote:
 Isn't that also true for isolated data (data that only allows one alias)?

That's colloquially called "unique" data. And yes, it is also true for that. 
That's why casting the return value of malloc() to 'shared' is safe. It's just 
that the language has no way to semantically identify unique data with its 
current type system.

rikki cattermole <rikki cattermole.co.nz> writes:

On 19/10/2018 9:02 PM, Walter Bright wrote:
 On 10/17/2018 4:29 AM, jmh530 wrote:
 Isn't that also true for isolated data (data that only allows one alias)?

 
 That's colloquially called "unique" data. And yes, it is also true for 
 that. That's why casting the return value of malloc() to 'shared' is 
 safe. It's just that the language has no way to semantically identify 
 unique data with its current type system.

Actually we kind of have a way to do this (I think). Scope. It can't 
cross the thread boundary and it has a pretty clear owner as far as the 
stack is concerned. Given a bit of time and empowering of scope, we 
could do a  scopeonly type attribute for functions.

NX <nightmarex1337 hotmail.com> writes:

I don't see any problem with this proposal as long as these 
points hold:

- Shared <-> Unshared is never implicit, either requiring an 
explicit cast (both ways) or having a language support which 
allows the conversion gracefully.
- Shared methods are called by compiler if the type is shared or 
if there is no unshared equivalent.
- Programmer needs to guarantee that shared -> unshared 
cast/conversion is thread-safe by hand; such as acquiring a lock, 
atomic operations...
- Programmer needs to guarantee that when unshared -> shared 
cast/conversion happens, data is not accessed through unshared 
reference during the lifetime of shared reference(s). Effectively 
this means a data needs to be treated as shared everywhere at the 
same time otherwise all things fall apart.

Atila Neves <atila.neves gmail.com> writes:

On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 1. shared should behave exactly like const, except in addition 
 to inhibiting write access, it also inhibits read access.

How is this significantly different from now?

-----------------
shared int i;
++i;

Error: read-modify-write operations are not allowed for shared 
variables. Use core.atomic.atomicOp!"+="(i, 1) instead.
-----------------

There's not much one can do to modify a shared value as it is.

 Current situation where you can arbitrarily access shared 
 members
 undermines any value it has.

Unless I'm missing something, I can't arbitrarily do anything 
with shared members right now.

 Shared must assure you don't access
 members unsafely, and the only way to do that with respect to 
 data
 members, is to inhibit access completely.

Or use atomic operations.

 Assuming this world... how do you use shared?

https://github.com/atilaneves/fearless

or

https://dlang.org/phobos/core_atomic.html

 From there, it opens up another critical opportunity; T* -> 
 shared(T)*
 promotion.

I don't think that works. See below.

 If you write a lock-free queue for instance, and all the 
 methods are
 `shared` (ie, threadsafe), then under the current rules, you 
 can't
 interact with the object when it's not shared, and that's fairly
 useless.

Not really:

-----------
struct FancyQueue(E) {
     // ...
     void pushBack(this T)(E element) {
         static if(is(T == shared)) {
             // lock mutex or whatever is needed
             auto safeThis = ()  trusted { return cast(FancyQueue) 
this; }();
         } else {
             // no need to lock anything
             alias safeThis = this;
         }
         // profit
     }
}
-----------

Usable if shared or not.

 Assuming the rules above: "can't read or write to members", and 
 the understanding that `shared` methods are expected to have 
 threadsafe implementations (because that's the whole point), 
 what are the risks from allowing T* -> shared(T)* conversion?

int i;
tid.send(&i);
++i;  // oops, data race


 All the risks that I think have been identified previously 
 assume that you can arbitrarily modify the data.

Do you have any examples of arbitrarily modifying shared data? I 
can't think of any.

 That's  insanity... assume we fix that... I think the promotion 
 actually becomes safe now...?

I don't think so, no.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 1:17 PM, Atila Neves wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 1. shared should behave exactly like const, except in addition to 
 inhibiting write access, it also inhibits read access.

 
 How is this significantly different from now?
 
 -----------------
 shared int i;
 ++i;

  i = i + 1; // OK(!)

-Steve

Atila Neves <atila.neves gmail.com> writes:

On Thursday, 18 October 2018 at 17:43:40 UTC, Steven 
Schveighoffer wrote:
 On 10/18/18 1:17 PM, Atila Neves wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 1. shared should behave exactly like const, except in 
 addition to inhibiting write access, it also inhibits read 
 access.

 
 How is this significantly different from now?
 
 -----------------
 shared int i;
 ++i;

  i = i + 1; // OK(!)

 -Steve

Sigh... I'm sure there's a practical reason for it, but still.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 1. shared should behave exactly like const, except in addition 
 to inhibiting write access, it also inhibits read access.

 How is this significantly different from now?

 -----------------
 shared int i;
 ++i;

 Error: read-modify-write operations are not allowed for shared 
 variables. Use core.atomic.atomicOp!"+="(i, 1) instead.
 -----------------

 There's not much one can do to modify a shared value as it is.

i = 1;
int x = i;
shared int y = i;

And so on. The compiler needs to forbid this.

 Unless I'm missing something, I can't arbitrarily do anything 
 with shared members right now.

Except arbitrarily read and write them :)

 From there, it opens up another critical opportunity; T* -> 
 shared(T)*
 promotion.


 I don't think that works. See below.

Welcome to the club.

 Assuming the rules above: "can't read or write to members", 
 and the understanding that `shared` methods are expected to 
 have threadsafe implementations (because that's the whole 
 point), what are the risks from allowing T* -> shared(T)* 
 conversion?

 int i;
 tid.send(&i);
 ++i;  // oops, data race

Doesn't work. No matter what you show Manu or Simen here they 
think it's just a bad contrived example. You can't sway them by 
the fact that the compiler currently *prevents* this from 
happening.

 All the risks that I think have been identified previously 
 assume that you can arbitrarily modify the data.


 Do you have any examples of arbitrarily modifying shared data? 
 I can't think of any.

See near the beginning of this post ;)

 That's  insanity... assume we fix that... I think the 
 promotion actually becomes safe now...?

 I don't think so, no.

+100500.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 1:47 PM, Stanislav Blinov wrote:
 On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 1. shared should behave exactly like const, except in addition to 
 inhibiting write access, it also inhibits read access.

 How is this significantly different from now?

 -----------------
 shared int i;
 ++i;

 Error: read-modify-write operations are not allowed for shared 
 variables. Use core.atomic.atomicOp!"+="(i, 1) instead.
 -----------------

 There's not much one can do to modify a shared value as it is.

 
 i = 1;
 int x = i;
 shared int y = i;

This should be fine, y is not shared when being created.

However, this still is allowed, and shouldn't be:

y = 5;

-Steve

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 18:26:27 UTC, Steven 
Schveighoffer wrote:
 On 10/18/18 1:47 PM, Stanislav Blinov wrote:
 On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves 
 wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 1. shared should behave exactly like const, except in 
 addition to inhibiting write access, it also inhibits read 
 access.

 How is this significantly different from now?

 -----------------
 shared int i;
 ++i;

 Error: read-modify-write operations are not allowed for 
 shared variables. Use core.atomic.atomicOp!"+="(i, 1) instead.
 -----------------

 There's not much one can do to modify a shared value as it is.

 
 i = 1;
 int x = i;
 shared int y = i;

 This should be fine, y is not shared when being created.

'y' isn't, but 'i' is. It's fine on amd64, but that's incidental.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 2:42 PM, Stanislav Blinov wrote:
 On Thursday, 18 October 2018 at 18:26:27 UTC, Steven Schveighoffer wrote:
 On 10/18/18 1:47 PM, Stanislav Blinov wrote:
 On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 1. shared should behave exactly like const, except in addition to 
 inhibiting write access, it also inhibits read access.

 How is this significantly different from now?

 -----------------
 shared int i;
 ++i;

 Error: read-modify-write operations are not allowed for shared 
 variables. Use core.atomic.atomicOp!"+="(i, 1) instead.
 -----------------

 There's not much one can do to modify a shared value as it is.

 i = 1;
 int x = i;
 shared int y = i;

 This should be fine, y is not shared when being created.

 
 'y' isn't, but 'i' is. It's fine on amd64, but that's incidental.

OH, I didn't even notice that `i` didn't have a type, so it was a 
continuation of the original example! I read it as declaring y as shared 
and assigning it to a thread-local (which it isn't actually).

My bad.

-Steve

Timon Gehr <timon.gehr gmx.ch> writes:

On 18.10.18 20:26, Steven Schveighoffer wrote:
 i = 1;
 int x = i;
 shared int y = i;

 
 This should be fine, y is not shared when being created.
 
 However, this still is allowed, and shouldn't be:
 
 y = 5;
 
 -Steve

I'm pretty sure you will have to allow operations on shared local 
variables. Otherwise, how are you ever going to use a shared(C)? You 
can't even call a shared method on it because it involves reading the 
reference.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 23:47:56 UTC, Timon Gehr wrote:

 I'm pretty sure you will have to allow operations on shared 
 local variables. Otherwise, how are you ever going to use a 
 shared(C)? You can't even call a shared method on it because it 
 involves reading the reference.

Because you can't really "share" C (e.g. by value). You share a 
C*, or, rather a shared(C)*. The pointer itself, which you own, 
isn't shared at all, and shouldn't be: it's your own reference to 
shared data. You can read and write that pointer all you want. 
What you must not be able to do is read and write *c.

Although, when it's a global?.. I'm not sure. We can have the 
compiler always generate a by-reference access, i.e. make that 
part of the language spec. Because full-on read of C.sizeof can't 
be statically proven thread-safe generically anyway (that's why 
generated copying and assignment don't make any sense for 
`shared`).

Timon Gehr <timon.gehr gmx.ch> writes:

On 19.10.18 02:29, Stanislav Blinov wrote:
 On Thursday, 18 October 2018 at 23:47:56 UTC, Timon Gehr wrote:
 
 I'm pretty sure you will have to allow operations on shared local 
 variables. Otherwise, how are you ever going to use a shared(C)? You 
 can't even call a shared method on it because it involves reading the 
 reference.

 
 Because you can't really "share" C (e.g. by value). You share a C*, or, 
 rather a shared(C)*.

(Here, I intended C to be a class, if that was unclear.)

 The pointer itself, which you own, isn't shared at 
 all, and shouldn't be: it's your own reference to shared data. You can 
 read and write that pointer all you want. What you must not be able to 
 do is read and write *c.
 ...

Presumably you could have a local variable shared(C) c, then take its 
address &c and send it to a thread which will be terminated before the 
scope of the local variable ends.

So, basically, the lack of tail-shared is an issue.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Friday, 19 October 2018 at 00:36:11 UTC, Timon Gehr wrote:
 On 19.10.18 02:29, Stanislav Blinov wrote:
 On Thursday, 18 October 2018 at 23:47:56 UTC, Timon Gehr wrote:
 
 I'm pretty sure you will have to allow operations on shared 
 local variables. Otherwise, how are you ever going to use a 
 shared(C)? You can't even call a shared method on it because 
 it involves reading the reference.

 
 Because you can't really "share" C (e.g. by value). You share 
 a C*, or, rather a shared(C)*.

 (Here, I intended C to be a class, if that was unclear.)

In that case, it's already a pointer, and the only real issue is 
the interaction with GC, which I mentioned before *needs* to be 
addressed. And that is only when C was allocated by GC.

 The pointer itself, which you own, isn't shared at all, and 
 shouldn't be: it's your own reference to shared data. You can 
 read and write that pointer all you want. What you must not be 
 able to do is read and write *c.
 ...

 Presumably you could have a local variable shared(C) c, then 
 take its address &c and send it to a thread which will be 
 terminated before the scope of the local variable ends.

I assume you mean *after*, because if that thread terminates 
before there's no problem.

 So, basically, the lack of tail-shared is an issue.

Well, not exactly. Irrespective of Manu's proposal, it's just 
inherent in D: sharing implies escaping, there's really no way 
around it. Provisions must be made in DIP1000 and in the language 
in general. However, that is a good point *against* implicit 
conversions, let alone  safe ones.

Erik van Velzen <erik evanv.nl> writes:

On Thursday, 18 October 2018 at 17:47:29 UTC, Stanislav Blinov 
wrote:
 On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Assuming the rules above: "can't read or write to members", 
 and the understanding that `shared` methods are expected to 
 have threadsafe implementations (because that's the whole 
 point), what are the risks from allowing T* -> shared(T)* 
 conversion?

 int i;
 tid.send(&i);
 ++i;  // oops, data race

 Doesn't work. No matter what you show Manu or Simen here they 
 think it's just a bad contrived example. You can't sway them by 
 the fact that the compiler currently *prevents* this from 
 happening.

Manu said clearly that the receiving thread won't be able to read 
or write the pointer.

Because int or int* does not have threadsafe member functions.


You can still disagree on the merits, but so far it has been 
demonstrated as a sound idea.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 19:04:58 UTC, Erik van Velzen 
wrote:
 On Thursday, 18 October 2018 at 17:47:29 UTC, Stanislav Blinov 
 wrote:
 On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves 
 wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Assuming the rules above: "can't read or write to members", 
 and the understanding that `shared` methods are expected to 
 have threadsafe implementations (because that's the whole 
 point), what are the risks from allowing T* -> shared(T)* 
 conversion?

 int i;
 tid.send(&i);
 ++i;  // oops, data race

 Doesn't work. No matter what you show Manu or Simen here they 
 think it's just a bad contrived example. You can't sway them 
 by the fact that the compiler currently *prevents* this from 
 happening.

 Manu said clearly that the receiving thread won't be able to 
 read or write the pointer.

Yes it will, by casting `shared` away. *Just like* his proposed 
"wrap everything into" struct will. There's exactly no difference.

 Because int or int* does not have threadsafe member functions.

int doesn't have any member functions. Or it can have as many as 
you like per UFCS. Same goes for structs. Because "methods" are 
just free functions in disguise, so that whole distinction in 
Manu's proposal is a weaksauce convention at best.

 You can still disagree on the merits, but so far it has been 
 demonstrated as a sound idea.

No, it hasn't been.

Erik van Velzen <erik evanv.nl> writes:

On Thursday, 18 October 2018 at 19:26:39 UTC, Stanislav Blinov 
wrote:
 On Thursday, 18 October 2018 at 19:04:58 UTC, Erik van Velzen 
 wrote:
 On Thursday, 18 October 2018 at 17:47:29 UTC, Stanislav Blinov 
 wrote:
 Doesn't work. No matter what you show Manu or Simen here they 
 think it's just a bad contrived example. You can't sway them 
 by the fact that the compiler currently *prevents* this from 
 happening.

 Manu said clearly that the receiving thread won't be able to 
 read or write the pointer.

 Yes it will, by casting `shared` away. *Just like* his proposed 
 "wrap everything into" struct will. There's exactly no 
 difference.

Casting is inherently unsafe. Or at least, there's no threadsafe 
guarantee.

 Because int or int* does not have threadsafe member functions.

 int doesn't have any member functions. Or it can have as many 
 as you like per UFCS. Same goes for structs. Because "methods" 
 are just free functions in disguise, so that whole distinction 
 in Manu's proposal is a weaksauce convention at best.

 You can still disagree on the merits, but so far it has been 
 demonstrated as a sound idea.

 No, it hasn't been.

I think you are missing the wider point. I can write 
thread-unsafe code *right now*, no casts required. Just put 
shared at the declaration. The proposal would actually give some 
guarantees.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 19:51:17 UTC, Erik van Velzen 
wrote:
 On Thursday, 18 October 2018 at 19:26:39 UTC, Stanislav Blinov

 Manu said clearly that the receiving thread won't be able to 
 read or write the pointer.

 Yes it will, by casting `shared` away. *Just like* his 
 proposed "wrap everything into" struct will. There's exactly 
 no difference.


 Casting is inherently unsafe. Or at least, there's no 
 threadsafe guarantee.

So? That's the only way to implement required low-level access, 
especially if we imagine that the part of Manu's proposal about 
disabling reads and writes on `shared` values is a given. It's 
the only way to implement Manu's Atomic!int, or at least 
operation it requires, for example.

 You can still disagree on the merits, but so far it has been 
 demonstrated as a sound idea.

 No, it hasn't been.

 I think you are missing the wider point. I can write 
 thread-unsafe code *right now*, no casts required. Just put 
 shared at the declaration. The proposal would actually give 
 some guarantees.

No, I think you are missing the wider point. You can write 
thread-unsafe code regardless of using `shared`, and regardless 
of it's implementation details. Allowing *implicit automatic 
promotion* of *mutable thread-local* data to shared will allow 
you to write even more thread-unsafe code, not less.

The solid part of the proposal is about disabling reads and 
writes. The rest is pure convention: write structs instead of 
functions, and somehow (???) benefit from a totally unsafe 
implicit cast.

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 1:10 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Thursday, 18 October 2018 at 19:51:17 UTC, Erik van Velzen
 wrote:
 On Thursday, 18 October 2018 at 19:26:39 UTC, Stanislav Blinov

 Manu said clearly that the receiving thread won't be able to
 read or write the pointer.

 Yes it will, by casting `shared` away. *Just like* his
 proposed "wrap everything into" struct will. There's exactly
 no difference.


 Casting is inherently unsafe. Or at least, there's no
 threadsafe guarantee.

 So? That's the only way to implement required low-level access,
 especially if we imagine that the part of Manu's proposal about
 disabling reads and writes on `shared` values is a given. It's
 the only way to implement Manu's Atomic!int, or at least
 operation it requires, for example.

 trusted code exists, and it's the foundation of the  safe stack.
I think you're just trying to be obtuse at this point.

Erik van Velzen <erik evanv.nl> writes:

Manu I'm also making a plea for you to write a document with your 
proposal which aggregates all relevant examples and objections. 
Then you can easily refer to it and we can introduce ppl to idea 
without reading a megathread.

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 2:40 PM Erik van Velzen via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Manu I'm also making a plea for you to write a document with your
 proposal which aggregates all relevant examples and objections.
 Then you can easily refer to it and we can introduce ppl to idea
 without reading a megathread.

I can start on this. I wouldn't take the time before having confidence
the whole effort was not to be rejected in principle.
I'm still not sure on this.
I think WRT Steven, we've understood each others position, and the
only point of contention is on weighting a value-judgement.
I'm confident in my weighting strategy, but I need to convince.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

Manu, Erik, Simen... In what world can a person consciously say 
"casting is unsafe", and yet at the same time claim that 
*implicit casting* is safe? What the actual F, guys?

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Thursday, 18 October 2018 at 21:54:55 UTC, Stanislav Blinov 
wrote:
 Manu, Erik, Simen... In what world can a person consciously say 
 "casting is unsafe", and yet at the same time claim that 
 *implicit casting* is safe? What the actual F, guys?

In a world where the implicit casting always ends in a place 
where anything you can do is safe. As we have said ad nauseam.

--
   Simen

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 2:55 PM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Manu, Erik, Simen... In what world can a person consciously say
 "casting is unsafe", and yet at the same time claim that
 *implicit casting* is safe? What the actual F, guys?

Implicit casting exists in a world where the conversion is guaranteed
to be safe, because rules are defined appropriately.
Explicit casting exists in a world where those guarantees are not
present, because no such rules.

The 2 different strategies are 2 different worlds, one is my proposal,
the other is more like what we have now. They are 2 different
rule-sets.
You are super-attached to some presumptions, and appear to refuse to
analyse the proposal from the grounds it defines.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Thursday, 18 October 2018 at 22:09:02 UTC, Manu wrote:

 The 2 different strategies are 2 different worlds, one is my 
 proposal,
 the other is more like what we have now. They are 2 different
 rule-sets.
 You are super-attached to some presumptions, and appear to 
 refuse to
 analyse the proposal from the grounds it defines.

Please see my exchange with Simen in case you're skipping my 
posts.

Erik van Velzen <erik evanv.nl> writes:

On Thursday, 18 October 2018 at 20:07:54 UTC, Stanislav Blinov 
wrote:
 On Thursday, 18 October 2018 at 19:51:17 UTC, Erik van Velzen 
 wrote:
 On Thursday, 18 October 2018 at 19:26:39 UTC, Stanislav Blinov

 Manu said clearly that the receiving thread won't be able to 
 read or write the pointer.

 Yes it will, by casting `shared` away. *Just like* his 
 proposed "wrap everything into" struct will. There's exactly 
 no difference.


 Casting is inherently unsafe. Or at least, there's no 
 threadsafe guarantee.

 So? That's the only way to implement required low-level access, 
 especially if we imagine that the part of Manu's proposal about 
 disabling reads and writes on `shared` values is a given. It's 
 the only way to implement Manu's Atomic!int, or at least 
 operation it requires, for example.

 You can still disagree on the merits, but so far it has been 
 demonstrated as a sound idea.

 No, it hasn't been.

 I think you are missing the wider point. I can write 
 thread-unsafe code *right now*, no casts required. Just put 
 shared at the declaration. The proposal would actually give 
 some guarantees.

 No, I think you are missing the wider point. You can write 
 thread-unsafe code regardless of using `shared`, and regardless 
 of its implementation details. Allowing *implicit automatic 
 promotion* of *mutable thread-local* data to shared will allow 
 you to write even more thread-unsafe code, not less.

This may come as a surprise but I agree with this factually.

It's just that the extra guarantee provided by disallowing 
implicit casting to shared is not so valuable to me. If you have 
an object which can be used in both a thread-safe and a 
thread-unsafe way that's a bug or code smell. Like the earlier 
example with localInc() and sharedInc(). And we can't guarantee 
the safety of  trusted implementations anyways.

If the extra guarantee is not valuable, might as well allow it.

 The solid part of the proposal is about disabling reads and 
 writes. The rest is pure convention: write structs instead of 
 functions,

"Writing structs" seems acknowledging how it's usually done. And 
then you put these in a library with generic concurrent data 
structures.

 and somehow (???) benefit from a totally unsafe implicit cast.

(See first part)

Timon Gehr <timon.gehr gmx.ch> writes:

On 18.10.18 23:34, Erik van Velzen wrote:
 If you have an object which can be used in both a thread-safe and a 
 thread-unsafe way that's a bug or code smell.

Then why do you not just make all members shared? Because with Manu's 
proposal, as soon as you have a shared method, all members effectively 
become shared. It just seems pointless to type them as unshared anyway 
and then rely on convention within  safe code to prevent unsafe 
accesses. Because, why? It just makes no sense.

With the proposal I posted in the beginning, you would then not only get 
implicit conversion of class references to shared, but also back to 
unshared.

I think the conflation of shared member functions and thread safe member 
functions is confusing. shared on a member function just means that the 
`this` reference is shared. The only use case for this is overloading on 
shared. The D approach to multithreading is that /all/ functions should 
be thread safe, but it is easier for some of them because they don't 
even need to access any shared state. It is therefore helpful if the 
type system cleanly separates shared from unshared state.

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Friday, 19 October 2018 at 00:29:01 UTC, Timon Gehr wrote:
 On 18.10.18 23:34, Erik van Velzen wrote:
 If you have an object which can be used in both a thread-safe 
 and a thread-unsafe way that's a bug or code smell.

 Then why do you not just make all members shared? Because with 
 Manu's proposal, as soon as you have a shared method, all 
 members effectively become shared. It just seems pointless to 
 type them as unshared anyway and then rely on convention within 
  safe code to prevent unsafe accesses. Because, why? It just 
 makes no sense.

Let's assume you have something like this:

module foo;

private shared int sharedState;

struct Accessor {

     int flags;

     void addUser(this T)()   {
        static if (is(T == shared))
            sharedState.atomicInc(); // unconditionally increment 
when it's a shared reference
        else {
            // owner may optimize shared access based on it's own 
state
            if (!(flags & SKIP_LOCKS)) sharedState.atomicInc();
        }
     }

     void removeUser(this T)() {
         static if (is(T == shared))
             sharedState.atomicDec();
         else {
             if (!(flags & SKIP_LOCKS)) sharedState.atomicDec();
         }
     }

     void setFlags(int f) { flags = f; }
}

The 'Accessor' doesn't really hold any shared state, but it 
accesses a shared module global. Now, the *owner* (e.g. code that 
instantiated a local Accessor) may use non-`shared` interface to 
track additional state and make decisions whether or not access 
the global. A concrete example would be e.g. an I/O lock, where 
if you know you don't have any threads other than main, you can 
skip syscalls locking/unlocking the handle.

 With the proposal I posted in the beginning, you would then not 
 only get implicit conversion of class references to shared, but 
 also back to unshared.

Is it in your first posts in this thread? I must've skipped that.

 I think the conflation of shared member functions and thread 
 safe member functions is confusing. shared on a member function 
 just means that the `this` reference is shared.

Nod.

 The only use case for this is overloading on shared. The D 
 approach to multithreading is that /all/ functions should be 
 thread safe, but it is easier for some of them because they 
 don't even need to access any shared state. It is therefore 
 helpful if the type system cleanly separates shared from 
 unshared state.

Nod nod.

Manu <turkeyman gmail.com> writes:

On Thu, Oct 18, 2018 at 5:30 PM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 18.10.18 23:34, Erik van Velzen wrote:
 If you have an object which can be used in both a thread-safe and a
 thread-unsafe way that's a bug or code smell.

 Then why do you not just make all members shared? Because with Manu's
 proposal, as soon as you have a shared method, all members effectively
 become shared.

No they don't, only facets that overlap with the shared method.
I tried to present an example before:

struct Threadsafe
{
  int x;
  Atomic!int y;
  void foo() shared { ++y; } // <- shared interaction only affects 'y'
  void bar() { ++x; ++y; } // <- not threadsafe, but does not violate
foo's commitment; only interaction with 'y' has any commitment
associated with it
  void unrelated() { ++x; } // <- no responsibilities are transposed
here, you can continue to do whatever you like throughout the class
where 'y' is not concerned
}

In practise, and in my direct experience, classes tend to have exactly
one 'y', and either zero (pure utility), or many such 'x' members.
Threadsafe API interacts with 'y', and the rest is just normal
thread-local methods which interact with all members thread-locally,
and may also interact with 'y' while not violating any threadsafety
commitments.

 It just seems pointless to type them as unshared anyway
 and then rely on convention within  safe code to prevent unsafe
 accesses. Because, why? It just makes no sense.

The pattern, is almost 100% of cses is: owner does all sorts of
stuff... threadsafe interactions with things happen in high-frequency
worker things. Parallel-for type workloads happen in bursts, the rest
of the program works like normal.
Parallel for workloads are restricted to the threadsafe API.

 With the proposal I posted in the beginning, you would then not only get
 implicit conversion of class references to shared, but also back to
 unshared.

I'm not sure how your proposal (which felt way more complicated to me)
improved on my design.
Can you clarify how my very simple design is ineffective?

 I think the conflation of shared member functions and thread safe member
 functions is confusing.

What else can it possibly mean to be a shared method? If it's not
threadsafe, it's an insta-crash...

 shared on a member function just means that the
 `this` reference is shared.

And by extension, you can only interact with other threadsafe methods.

 The only use case for this is overloading on
 shared.

No, it's for separating threadsafe methods from those that aren't.
Just like const methods separate mutating methods from those that don't.

 The D approach to multithreading is that /all/ functions should
 be thread safe, but it is easier for some of them because they don't
 even need to access any shared state. It is therefore helpful if the
 type system cleanly separates shared from unshared state.

This is patently untrue. I can't implement any interesting shared
architecture with the design as is today.
I am also completely free to violate any sense of threadsafety with no
restrictions of any kind.
Write a shared method, access any members, call it from shared
instances, watch the fire burn.

I don't understand how you can claim that D's design says that all
functions are threadsafe, it's so plainly far from the truth...?

Steven Schveighoffer <schveiguy gmail.com> writes:

On 10/18/18 9:09 PM, Manu wrote:
 On Thu, Oct 18, 2018 at 5:30 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 18.10.18 23:34, Erik van Velzen wrote:
 If you have an object which can be used in both a thread-safe and a
 thread-unsafe way that's a bug or code smell.

 Then why do you not just make all members shared? Because with Manu's
 proposal, as soon as you have a shared method, all members effectively
 become shared.

 
 No they don't, only facets that overlap with the shared method.
 I tried to present an example before:
 
 struct Threadsafe
 {
    int x;
    Atomic!int y;
    void foo() shared { ++y; } // <- shared interaction only affects 'y'
    void bar() { ++x; ++y; } // <- not threadsafe, but does not violate
 foo's commitment; only interaction with 'y' has any commitment
 associated with it
    void unrelated() { ++x; } // <- no responsibilities are transposed
 here, you can continue to do whatever you like throughout the class
 where 'y' is not concerned
 }
 
 In practise, and in my direct experience, classes tend to have exactly
 one 'y', and either zero (pure utility), or many such 'x' members.
 Threadsafe API interacts with 'y', and the rest is just normal
 thread-local methods which interact with all members thread-locally,
 and may also interact with 'y' while not violating any threadsafety
 commitments.

I promised I wouldn't respond, I'm going to break that (obviously).

But that's because after reading this description I ACTUALLY understand 
what you are looking for.

I'm going to write a fuller post later, but I can't right now. But the 
critical thing here is, you want a system where you can divvy up a type 
into pieces you share and pieces you don't. But then you *don't* want to 
have to share only the shared pieces. You want to share the whole thing 
and be sure that it can't access your unshared pieces.

This critical requirement makes things a bit more interesting. For the 
record, the most difficult thing to reaching this understanding was that 
whenever I proposed anything, your answer was something like 'I just 
can't work with that', and when I asked why, you said 'because it's 
useless', etc. Fully explaining this point is very key to understanding 
your thinking.

To be continued...

-Steve

Manu <turkeyman gmail.com> writes:

On Fri, Oct 19, 2018 at 9:45 AM Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/18/18 9:09 PM, Manu wrote:
 On Thu, Oct 18, 2018 at 5:30 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 18.10.18 23:34, Erik van Velzen wrote:
 If you have an object which can be used in both a thread-safe and a
 thread-unsafe way that's a bug or code smell.

 Then why do you not just make all members shared? Because with Manu's
 proposal, as soon as you have a shared method, all members effectively
 become shared.

 No they don't, only facets that overlap with the shared method.
 I tried to present an example before:

 struct Threadsafe
 {
    int x;
    Atomic!int y;
    void foo() shared { ++y; } // <- shared interaction only affects 'y'
    void bar() { ++x; ++y; } // <- not threadsafe, but does not violate
 foo's commitment; only interaction with 'y' has any commitment
 associated with it
    void unrelated() { ++x; } // <- no responsibilities are transposed
 here, you can continue to do whatever you like throughout the class
 where 'y' is not concerned
 }

 In practise, and in my direct experience, classes tend to have exactly
 one 'y', and either zero (pure utility), or many such 'x' members.
 Threadsafe API interacts with 'y', and the rest is just normal
 thread-local methods which interact with all members thread-locally,
 and may also interact with 'y' while not violating any threadsafety
 commitments.

 I promised I wouldn't respond, I'm going to break that (obviously).

 But that's because after reading this description I ACTUALLY understand
 what you are looking for.

 I'm going to write a fuller post later, but I can't right now. But the
 critical thing here is, you want a system where you can divvy up a type
 into pieces you share and pieces you don't. But then you *don't* want to
 have to share only the shared pieces. You want to share the whole thing
 and be sure that it can't access your unshared pieces.

 This critical requirement makes things a bit more interesting. For the
 record, the most difficult thing to reaching this understanding was that
 whenever I proposed anything, your answer was something like 'I just
 can't work with that', and when I asked why, you said 'because it's
 useless', etc. Fully explaining this point is very key to understanding
 your thinking.

 To be continued...

I'm glad that there's movement here... but I'm still not 100%
convinced you understood me; perhaps getting close though.
I only say that because your description above has a whole lot more
words and complexity than is required to express my proposal. If you
perceive that complexity in structural terms, then I am still not
clearly understood.

 "divvy up a type into pieces"

This is an odd mental model of what I'm saying, and I can't sympathise
with those words, but if they work for you, and we agree on the
semantics, then sure...
If you write an object with some const methods and some non-const
methods, then take a const instance of the object... you can only call
the const methods. Have you 'divvied up the type' into a const portion
and a non-const portion? If the answer is yes, then I can accept your
description.

I would talk in terms of restriction:
An object has 4 functions, 2 are mutable, 2 are const... you apply
const to the type and you are *restricted* to only calling the 2 const
functions.
An object has 4 functions, 2 are unsahred, 2 are shared... you apply
shared to the type and you are *restricted* to only calling the 2
shared (threadsafe) functions.

I haven't 'broken the type up', I'm just restricting what you can do
to it from within a particular context. In the const context, you
can't mutate it. In the shared context, you can't do un-threadsafe to
it, and the guarantee of that is embedded in the rules:
1. shared data can not be read or written
2. shared methods must be threadsafe
  a. this may require that they be  trusted at the low-level, like the
methods of `Atomic(T)`
  b. no other method may violate the shared method's promise,
otherwise it does not actually deliver its promise
    i. I have extensive experience with this and it's just not a
problem in practise, but compiler technology to assist would be
welcome!
    ii. This does NOT mean the not-shared methods are somehow shared;
they just need to be careful when interacting with some (usually
small) subset of members

This design implies strong encapsulation, but that's a naturally
occurring tendency implementing anything that's threadsafe.
As a helpful best-practise to assure that non-shared methods don't
undermine a shared method's commitment; prefer to interact with
volatile members via accessors or properties that are themselves
shared (can be private if you like).
You will find that it's not actually hard to deliver on the object's commitment.

If you write tooling that is at the level one-up from the bottom of
the stack or higher, it should be unusual that you ever need to write
a  trusted method, in which case delivering on *your* shared methods
promise is implicit.

Atila Neves <atila.neves gmail.com> writes:

On Thursday, 18 October 2018 at 19:04:58 UTC, Erik van Velzen 
wrote:
 On Thursday, 18 October 2018 at 17:47:29 UTC, Stanislav Blinov 
 wrote:
 On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves 
 wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Assuming the rules above: "can't read or write to members", 
 and the understanding that `shared` methods are expected to 
 have threadsafe implementations (because that's the whole 
 point), what are the risks from allowing T* -> shared(T)* 
 conversion?

 int i;
 tid.send(&i);
 ++i;  // oops, data race

 Doesn't work. No matter what you show Manu or Simen here they 
 think it's just a bad contrived example. You can't sway them 
 by the fact that the compiler currently *prevents* this from 
 happening.

 Manu said clearly that the receiving thread won't be able to 
 read or write the pointer.

Not directly - but obviously there must be *some* way to using 
it, in this case since it's an int with one of the core.atomic 
functions. At that point the spawned thread will be accessing it 
correctly, but the parent thread can modify the int in a 
non-atomic fashion.

 Because int or int* does not have threadsafe member functions.

https://dlang.org/phobos/core_atomic.html

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Friday, 19 October 2018 at 18:00:47 UTC, Atila Neves wrote:
 Because int or int* does not have threadsafe member functions.

 https://dlang.org/phobos/core_atomic.html

Atomic and thread-safe are two very different concepts. 
Thread-safe is more of an ecosystem thing - if there are ways to 
do non-thread-safe operations on something, atomic operations are 
not enough to make things thread-safe. This is what core.atomic 
does:

https://i.imgur.com/PnKMigl.jpg

If you close the other openings, atomics are fantastic building 
blocks to making something thread-safe, but on their own they are 
not enough.

--
   Simen

jmh530 <john.michael.hall gmail.com> writes:

On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote:
 [snip]

 Assuming this world... how do you use shared?

 https://github.com/atilaneves/fearless

I had posted your library before to no response...

I had two questions, if you'll indulge me.

The first is perhaps more wrt automem. I noticed that I couldn't 
use automem's Unique with  safe currently. Is there any way to 
make it  safe, perhaps with dip1000?

Second, Rust's borrow checker is that you can only have one 
mutable borrow. This is kind of like Exclusive, but it does it at 
compile-time, rather than with GC/RC. Is this something that can 
be incorporated into fearless?

Atila Neves <atila.neves gmail.com> writes:

On Thursday, 18 October 2018 at 21:24:53 UTC, jmh530 wrote:
 On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote:
 [snip]

 Assuming this world... how do you use shared?

 https://github.com/atilaneves/fearless

 I had posted your library before to no response...

 I had two questions, if you'll indulge me.

 The first is perhaps more wrt automem. I noticed that I 
 couldn't use automem's Unique with  safe currently. Is there 
 any way to make it  safe, perhaps with dip1000?

Yeah, I punted on making anything there  safe. I have to go back 
and fix it. At least I wrote Vector from scratch to be  safe.

 Second, Rust's borrow checker is that you can only have one 
 mutable borrow. This is kind of like Exclusive, but it does it 
 at compile-time, rather than with GC/RC. Is this something that 
 can be incorporated into fearless?

Well, Rust's version of Exclusive is Mutex, and that's pretty 
much always used with Arc. The closest we have to a borrow 
checker is DIP1000 and that's what fearless relies on.

John Colvin <john.loughran.colvin gmail.com> writes:

On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Okay, so I've been thinking on this for a while... I think I 
 have a pretty good feel for how shared is meant to be.

 [...]

I don't understand how you can safely have simultaneously shared 
methods that can modify data and unshared references that can 
modify data.

struct S {
     int* x;
     void incX() shared { ... }
}

auto x = new int;
auto s = shared S(x);

Now I have two references to x, one shared, one unshared, both 
can be written to.

What am I missing?

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Monday, 22 October 2018 at 21:40:23 UTC, John Colvin wrote:
 On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
 Okay, so I've been thinking on this for a while... I think I 
 have a pretty good feel for how shared is meant to be.

 [...]

 I don't understand how you can safely have simultaneously 
 shared methods that can modify data and unshared references 
 that can modify data.

 struct S {
     int* x;
     void incX() shared { ... }
 }

 auto x = new int;
 auto s = shared S(x);

 Now I have two references to x, one shared, one unshared, both 
 can be written to.

 What am I missing?

S's constructor is not written in such a way as to disallow 
unsafe, mutable sharing of S.x. S.x is not private, and there's 
no indication that any attempt has been made to block off other 
avenues of access to S.x (other code in the same module could 
access it). S.incX is not  safe.

I don't know for sure what's inside S.incX, but let's assume it's 
this:

void incX() shared {
     atomicOp!"++"(x);
}

This will fail to compile under MP, since atomicOp would take a 
ref int, not a ref shared int as it does today (since no implicit 
conversion from shared to unshared is allowed).

To make it compile, you will need to explicitly cast it to 
unshared. Since incX is not  safe, you can do this without the 
compiler complaining. (this is why you should be writing all the 
code that you can, as  safe)

If we assume you perform no unsafe casts, then I have no idea 
what S.incX would be doing - int* can have no thread-safe, 
 trusted methods, so you can do absolutely nothing with S.x 
inside S's shared,  safe methods.

Most importantly though, why the hell are you trying to write 
such low-level code? The code you should be writing is this:

struct S {
     Atomic!int x;
     void incX()  safe shared {
         x++;
     }
}

Because Atomic!T has been written by an expert to be thread-safe.

This is why we've been repeating, over and over, again and again, 
that Joe Average Programmer should not be writing the 
foundational building blocks of MP.

--
   Simen

John Colvin <john.loughran.colvin gmail.com> writes:

On Tuesday, 23 October 2018 at 07:33:14 UTC, Simen Kjærås wrote:
 On Monday, 22 October 2018 at 21:40:23 UTC, John Colvin wrote:
 [...]

 S's constructor is not written in such a way as to disallow 
 unsafe, mutable sharing of S.x. S.x is not private, and there's 
 no indication that any attempt has been made to block off other 
 avenues of access to S.x (other code in the same module could 
 access it). S.incX is not  safe.

 I don't know for sure what's inside S.incX, but let's assume 
 it's this:

 void incX() shared {
     atomicOp!"++"(x);
 }

 This will fail to compile under MP, since atomicOp would take a 
 ref int, not a ref shared int as it does today (since no 
 implicit conversion from shared to unshared is allowed).

 To make it compile, you will need to explicitly cast it to 
 unshared. Since incX is not  safe, you can do this without the 
 compiler complaining. (this is why you should be writing all 
 the code that you can, as  safe)

 If we assume you perform no unsafe casts, then I have no idea 
 what S.incX would be doing - int* can have no thread-safe, 
  trusted methods, so you can do absolutely nothing with S.x 
 inside S's shared,  safe methods.

 Most importantly though, why the hell are you trying to write 
 such low-level code? The code you should be writing is this:

 struct S {
     Atomic!int x;
     void incX()  safe shared {
         x++;
     }
 }

 Because Atomic!T has been written by an expert to be 
 thread-safe.

 This is why we've been repeating, over and over, again and 
 again, that Joe Average Programmer should not be writing the 
 foundational building blocks of MP.

 --
   Simen

I don't think this direction of me providing examples and you 
examining them is going to work, as I clearly don't (or at least 
I hope I don't currently) understand the proposal.

Is there an example something useful that would work and have an 
 safe API under this proposal? Something that includes the 
implicit cast from T* to shared(T)* and some shared methods?

Neia Neutuladh <neia ikeran.org> writes:

On Mon, 15 Oct 2018 11:46:45 -0700, Manu wrote:
From there, it opens up another critical opportunity; T* -> shared(T)*

 promotion.
 Const would be useless without T* -> const(T)* promotion. Shared suffers
 a similar problem.
 If you write a lock-free queue for instance, and all the methods are
 `shared` (ie, threadsafe), then under the current rules, you can't
 interact with the object when it's not shared, and that's fairly
 useless.

Somehow I lost track of your reason behind the promotion.

Why not just ask to be able to call shared methods on a thread-local 
variable? No implicit casting; you need to take positive action to share 
anything. And if there's a non-shared overload, that's still preferred.

Would that be enough? Or do you have use cases that wouldn't fit there?

Stanislav Blinov <stanislav.blinov gmail.com> writes:

On Tuesday, 23 October 2018 at 01:26:57 UTC, Neia Neutuladh wrote:

 Somehow I lost track of your reason behind the promotion.

 Why not just ask to be able to call shared methods on a 
 thread-local variable? No implicit casting; you need to take 
 positive action to share anything. And if there's a non-shared 
 overload, that's still preferred.

Because these have the same signature:

struct S {
     void method() shared;
}

void func(ref shared S zis);

i.e. calling a shared method on a thread-local variable does 
require an implicit cast of the 'this' reference. Just like 
calling a 'const' method.

Atila Neves <atila.neves gmail.com> writes:

On Tuesday, 23 October 2018 at 01:26:57 UTC, Neia Neutuladh wrote:
 On Mon, 15 Oct 2018 11:46:45 -0700, Manu wrote:
From there, it opens up another critical opportunity; T* -> 
shared(T)*

 promotion.
 Const would be useless without T* -> const(T)* promotion. 
 Shared suffers
 a similar problem.
 If you write a lock-free queue for instance, and all the 
 methods are
 `shared` (ie, threadsafe), then under the current rules, you 
 can't
 interact with the object when it's not shared, and that's 
 fairly
 useless.

 Somehow I lost track of your reason behind the promotion.

 Why not just ask to be able to call shared methods on a 
 thread-local variable? No implicit casting; you need to take 
 positive action to share anything. And if there's a non-shared 
 overload, that's still preferred.

This is easily achievable with template this.