www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP 1024--Shared Atomics--Community Review Round 1

reply Mike Parker <aldacron gmail.com> writes:
This is the feedback thread for the first round of Community 
Review for DIP 1024, "Shared Atomics":

https://github.com/dlang/DIPs/blob/0b892dd99aba74b9631572ad3a53000f5975b7c2/DIPs/DIP1024.md

All review-related feedback on and discussion of the DIP should 
occur in this thread. The review period will end at 11:59 PM ET 
on October 15, or when I make a post declaring it complete.

At the end of Round 1, if further review is deemed necessary, the 
DIP will be scheduled for another round of Community Review. 
Otherwise, it will be queued for the Final Review and Formal 
Assessment.

Anyone intending to post feedback in this thread is expected to 
be familiar with the reviewer guidelines:

https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

**Thanks in advance for keeping all discussion on topic.**
Oct 01 2019
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
This DIP will need to detail how it does (and doesn't) interact with 
methods with the shared attribute on it, along with examples of the 
shared qualifier on a struct/class.

As it stands, if I was to implement it, I could interpret the DIP to 
only affect globals.
Oct 01 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 4:21 AM, rikki cattermole wrote:

 This DIP will need to detail how it does (and doesn't) interact with methods 
 with the shared attribute on it, along with examples of the shared qualifier
on 
 a struct/class.


The DIP does not change how `shared` is part of the type system, and does not 
change how `shared` affects the types of structs or classes or instances of
them.


 As it stands, if I was to implement it, I could interpret the DIP to only
affect 
 globals.


`shared` isn't a storage class, it's a type constructor, so I don't see how you 
could infer that.
Oct 01 2019
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 02/10/2019 9:31 AM, Walter Bright wrote:

 On 10/1/2019 4:21 AM, rikki cattermole wrote:

 This DIP will need to detail how it does (and doesn't) interact with 
 methods with the shared attribute on it, along with examples of the 
 shared qualifier on a struct/class.


 
 The DIP does not change how `shared` is part of the type system, and 
 does not change how `shared` affects the types of structs or classes or 
 instances of them.


Okay, if we get a statement to that affect in the DIP, maybe with an 
example then nobody can interpret it wrong :)


 As it stands, if I was to implement it, I could interpret the DIP to 
 only affect globals.


 
 `shared` isn't a storage class, it's a type constructor, so I don't see 
 how you could infer that.


Okay in that case, the spec needs updating. Because right now it is 
listed as part of `StorageClass` along with `const`.
Oct 01 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 1:54 PM, rikki cattermole wrote:

 On 02/10/2019 9:31 AM, Walter Bright wrote:

 On 10/1/2019 4:21 AM, rikki cattermole wrote:

 This DIP will need to detail how it does (and doesn't) interact with methods 
 with the shared attribute on it, along with examples of the shared qualifier 
 on a struct/class.


 The DIP does not change how `shared` is part of the type system, and does not 
 change how `shared` affects the types of structs or classes or instances of
them.


 
 Okay, if we get a statement to that affect in the DIP, maybe with an example 
 then nobody can interpret it wrong :)


Ok.



 As it stands, if I was to implement it, I could interpret the DIP to only 
 affect globals.


 `shared` isn't a storage class, it's a type constructor, so I don't see how 
 you could infer that.


 
 Okay in that case, the spec needs updating. Because right now it is listed as 
 part of `StorageClass` along with `const`.


While const appears in the grammar as a storage class, mainly for programmer 
convenience, it is a type constructor.
Oct 03 2019
prev sibling next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:

 [snip]


In terms of other languages that might take a similar approach, 
the DIP only mentions C's volatile. The Task Parallelism and 
Synchronization section of the Chapel Language specification [1] 
might be of interest.

[1] https://chapel-lang.org/docs/_downloads/chapelLanguageSpec.pdf
Oct 01 2019
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1024, "Shared Atomics":


Section "Alternatives" should be properly referenced as it makes 
unsubstantiated claims.(Also the references are not linked to 
from the main document).

The linked implementation (which is already merged(!)) appears to 
bear no correlation to the DIP, which appears to suffer a severe 
identity crisis, and as such completely fails the vengeful-ex 
principle:


 This change will require using core.atomic or equivalent 
 functions to read and write to shared memory objects. It will 
 prevent unintended, inadvertent non-use of atomic access.
 ...
 By prohibiting direct access to shared data, the user will be 
 required to use core.atomic and to consider the correctness of 
 their code.



 [The DIP is] just replacing calls to core.atomic with language 
 support...
 This is not pioneering new ground. The only innovation is 
 support by the language type system rather than library calls.
 x=1; // atomic store, release, happens after everything above


So which one is it?
If it is the first, all the references to apparent automatic 
rewriting should be dropped.
If it is the second, I fail to see why this is semantically any 
different to what C++ does with atomic<int>, which the DIP makes 
the claim is controversial.
Oct 01 2019
prev sibling next sibling parent reply IGotD- <nise nise.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:

"This change will require using core.atomic or equivalent 
functions to read and >write to shared memory objects. It will 
prevent unintended, inadvertent non-use of >atomic access."


What if the type is complex (like a struct) outside what the ISA 
can handle? Usually this only applies for integers 8 - 128 bits 
depending on architecture, anything beyond that is out of reach 
unless you want to dig into Intel TSX which the shared attribute 
isn't suitable for.

Another thing that is left out are the effects of operator 
overloading.

shared int s;

// What would this do? Atomically increment s or just atomically 
load s, add one
// and then atomically store s?
s = s + 1;
// Would this atomically increase s and return the value after 
increment?
s++;
Oct 01 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 7:14 AM, IGotD- wrote:

 On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:

 "This change will require using core.atomic or equivalent functions to read 
 and >write to shared memory objects. It will prevent unintended, inadvertent 
 non-use of >atomic access."


 
 What if the type is complex (like a struct) outside what the ISA can handle? 
 Usually this only applies for integers 8 - 128 bits depending on architecture, 
 anything beyond that is out of reach unless you want to dig into Intel TSX
which 
 the shared attribute isn't suitable for.
 
 Another thing that is left out are the effects of operator overloading.
 
 shared int s;
 
 // What would this do? Atomically increment s or just atomically load s, add
one
 // and then atomically store s?
 s = s + 1;
 // Would this atomically increase s and return the value after increment?
 s++;



Actually, shared operations would no longer be generated from the syntax.
They'd 
all be done with library functions, which could be implemented as intrinsics by 
the compiler.

If it's outside what the ISA could handle, then those library functions would 
not be intrinsics, and it's up to the implementation of those functions to make 
it work.
Oct 01 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 01.10.19 22:34, Walter Bright wrote:

 Actually, shared operations would no longer be generated from the 
 syntax. They'd all be done with library functions


To clarify, we're going to get this(?):

     shared int x;
     void main()
     {
         x = 42; /* error */
     }

You should change the first sentence of the DIP then. It says: "Reads 
and writes to data typed as shared are made atomic where the target CPU 
supports atomic operations".

 From that sentence I would very much expect the code be accepted and 
generate an atomic write.
Oct 01 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 3:07 PM, ag0aep6g wrote:

 On 01.10.19 22:34, Walter Bright wrote:

 Actually, shared operations would no longer be generated from the syntax. 
 They'd all be done with library functions


 
 To clarify, we're going to get this(?):
 
      shared int x;
      void main()
      {
          x = 42; /* error */
      }


Yes.



 You should change the first sentence of the DIP then. It says: "Reads and
writes 
 to data typed as shared are made atomic where the target CPU supports atomic 
 operations".
 
  From that sentence I would very much expect the code be accepted and generate 
 an atomic write.


I agree it's confusing.
Oct 03 2019
prev sibling next sibling parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1024, "Shared Atomics":

 [...]


There is no detail on how is this going to be implemented.

-Alex
Oct 01 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 9:08 AM, 12345swordy wrote:

 There is no detail on how is this going to be implemented.


It'll disallow direct translation of operators to code, instead the user will 
have to call core.atomic library functions to do it, and the compiler may treat 
those as intrinsics.
Oct 01 2019
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Tuesday, 1 October 2019 at 20:35:51 UTC, Walter Bright wrote:

 On 10/1/2019 9:08 AM, 12345swordy wrote:

 There is no detail on how is this going to be implemented.


 It'll disallow direct translation of operators to code, instead 
 the user will have to call core.atomic library functions to do 
 it, and the compiler may treat those as intrinsics.


Does this mean that the user can rollout their own custom 
implementation?

-Alex
Oct 01 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 1:47 PM, 12345swordy wrote:

 On Tuesday, 1 October 2019 at 20:35:51 UTC, Walter Bright wrote:

 On 10/1/2019 9:08 AM, 12345swordy wrote:

 There is no detail on how is this going to be implemented.


 It'll disallow direct translation of operators to code, instead the user will 
 have to call core.atomic library functions to do it, and the compiler may 
 treat those as intrinsics.


 
 Does this mean that the user can rollout their own custom implementation?


Yes, as they can today.
Oct 03 2019
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:

[...]

My feedback: What is the proposal?

"By prohibiting direct access to shared data, the user will be 
required to use core.atomic and to consider the correctness of 
their code."

"This change will require using core.atomic or equivalent 
functions to read and write to shared memory objects. It will 
prevent unintended, inadvertent non-use of atomic access."

"Since it's just replacing calls to core.atomic with language 
support, and atomics are well understood (at least by experts) in 
multiple languages, there should be little risk. This is not 
pioneering new ground. The only innovation is support by the 
language type system rather than library calls."

It's possible that I just don't understand this topic well 
enought, but in my opinion it's more of a sketch of things that 
should be discussed in a DIP rather than a DIP. An example would 
go a long way.
Oct 01 2019
parent reply ag0aep6g <anonymous example.com> writes:
On Tuesday, 1 October 2019 at 16:22:36 UTC, bachmeier wrote:

 My feedback: What is the proposal?


This is the proposal (first paragraph of the DIP / Abstract): 
"Reads and writes to data typed as shared are made atomic where 
the target CPU supports atomic operations and become erroneous 
when they cannot be done atomically."

The rest of the DIP sometimes seems to contradict the Abstract 
(e.g. "prohibiting direct access to shared data", "will require 
using core.atomic"). It should probably be clarified that using 
core.atomic will only be necessary when the compiler can't 
generate the atomic operation itself for whatever reason.
Oct 01 2019
parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 1 October 2019 at 16:49:25 UTC, ag0aep6g wrote:

 [snip]

 This is the proposal (first paragraph of the DIP / Abstract): 
 "Reads and writes to data typed as shared are made atomic where 
 the target CPU supports atomic operations and become erroneous 
 when they cannot be done atomically."

 The rest of the DIP sometimes seems to contradict the Abstract 
 (e.g. "prohibiting direct access to shared data", "will require 
 using core.atomic"). It should probably be clarified that using 
 core.atomic will only be necessary when the compiler can't 
 generate the atomic operation itself for whatever reason.


I assumed bachmeier's point was more an issue with the 
organization and presentation of the DIP than what the proposal 
actually is...
Oct 01 2019
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On Tue, Oct 1, 2019 at 3:45 AM Mike Parker via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 [...]


At best, this DIP is premature; we need to make shared correctly
restrict read/write access before trying to do fancy stuff.
That flag is now available, but the implementation has bugs, and it
needs to be enabled by default.

Assuming that the supporting material I mention above is available and
works properly, then I think it might be reasonable to open some
discussion of this kind, and on that note... I think this is a really
bad idea, and almost the opposite of what you want in virtually every
case.
I've written a whole lot of lock-free concurrent code, which involves
a lot of atomics, and the semantics in this DIP are not anything I
have ever seen or wanted (as I've argued here in the past). I think
the semantics described are misleading, and would only serve to
improperly educate an amateur user how to incorrectly write lock-free
algorithms, and increase their confidence that they're doing it right
(they're probably not).
1. Most atomic operations are NOT seq-cst. Seq-cst operations are *super rare*.
  a. I just searched our entire engine codebase; there are 971 calls
to atomic functions; there are *four* instances of seq-cst, and I
think at least some of them are actually wrong (over-synchronised),
possibly all of them, I would need to consider closer.
2. Many atomic operations appear in a series, in which case, it's
typical that exactly one operation in the series may be the
synchronisation point.
3. In reality, probably at least half of operations are 'relaxed'.
4. xchg and cas operations are fundamental to any atomic code, and
there are no syntax or operators for those operations; so, you're
undoubtedly using core.atomic aggressively anyway, and I think it's
just weird to litter naked arithmetic in between calls to core.atomic
functions. I feel that's misleading by making the naked arithmetic
look like NOT-atomic operations by contrast to a bunch of atomic
library calls.
5. In almost all cases when working with atomics, you tend to do
quirky stuff not captured by regular usage of data types, and not
conveniently facilitated by this DIP.

For instance; this is a common implementation for releasing a ref count:
  if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed))
  {
    // maybe acquire if necessary
    // do release logic...
  }

How do I write that code with this DIP?

We need to make shared correct before we do any stuff like this, and
then when it's correct, we can talk... but I will probably continue to
resist this, because the only result I see is that it will train
amateurs to write bad atomic algorithms with an improper sense of
confidence gained by being sanctioned by the language.

This is a bad idea.
Oct 01 2019
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On Tue, Oct 1, 2019 at 10:57 PM Manu <turkeyman gmail.com> wrote:

 On Tue, Oct 1, 2019 at 3:45 AM Mike Parker via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 [...]


 At best, this DIP is premature; we need to make shared correctly
 restrict read/write access before trying to do fancy stuff.
 That flag is now available, but the implementation has bugs, and it
 needs to be enabled by default.

 Assuming that the supporting material I mention above is available and
 works properly, then I think it might be reasonable to open some
 discussion of this kind, and on that note... I think this is a really
 bad idea, and almost the opposite of what you want in virtually every
 case.
 I've written a whole lot of lock-free concurrent code, which involves
 a lot of atomics, and the semantics in this DIP are not anything I
 have ever seen or wanted (as I've argued here in the past). I think
 the semantics described are misleading, and would only serve to
 improperly educate an amateur user how to incorrectly write lock-free
 algorithms, and increase their confidence that they're doing it right
 (they're probably not).
 1. Most atomic operations are NOT seq-cst. Seq-cst operations are *super rare*.
   a. I just searched our entire engine codebase; there are 971 calls
 to atomic functions; there are *four* instances of seq-cst, and I
 think at least some of them are actually wrong (over-synchronised),
 possibly all of them, I would need to consider closer.
 2. Many atomic operations appear in a series, in which case, it's
 typical that exactly one operation in the series may be the
 synchronisation point.
 3. In reality, probably at least half of operations are 'relaxed'.
 4. xchg and cas operations are fundamental to any atomic code, and
 there are no syntax or operators for those operations; so, you're
 undoubtedly using core.atomic aggressively anyway, and I think it's
 just weird to litter naked arithmetic in between calls to core.atomic
 functions. I feel that's misleading by making the naked arithmetic
 look like NOT-atomic operations by contrast to a bunch of atomic
 library calls.
 5. In almost all cases when working with atomics, you tend to do
 quirky stuff not captured by regular usage of data types, and not
 conveniently facilitated by this DIP.

 For instance; this is a common implementation for releasing a ref count:
   if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed))
   {
     // maybe acquire if necessary
     // do release logic...
   }

 How do I write that code with this DIP?

 We need to make shared correct before we do any stuff like this, and
 then when it's correct, we can talk... but I will probably continue to
 resist this, because the only result I see is that it will train
 amateurs to write bad atomic algorithms with an improper sense of
 confidence gained by being sanctioned by the language.

 This is a bad idea.


Sorry, that should have read:

  if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed) == 1)
  {
    // maybe acquire if necessary
    // do release logic...
  }
Oct 01 2019
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On Tue, Oct 1, 2019 at 11:01 PM Manu <turkeyman gmail.com> wrote:

 On Tue, Oct 1, 2019 at 10:57 PM Manu <turkeyman gmail.com> wrote:

 On Tue, Oct 1, 2019 at 3:45 AM Mike Parker via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 [...]


 At best, this DIP is premature; we need to make shared correctly
 restrict read/write access before trying to do fancy stuff.
 That flag is now available, but the implementation has bugs, and it
 needs to be enabled by default.

 Assuming that the supporting material I mention above is available and
 works properly, then I think it might be reasonable to open some
 discussion of this kind, and on that note... I think this is a really
 bad idea, and almost the opposite of what you want in virtually every
 case.
 I've written a whole lot of lock-free concurrent code, which involves
 a lot of atomics, and the semantics in this DIP are not anything I
 have ever seen or wanted (as I've argued here in the past). I think
 the semantics described are misleading, and would only serve to
 improperly educate an amateur user how to incorrectly write lock-free
 algorithms, and increase their confidence that they're doing it right
 (they're probably not).
 1. Most atomic operations are NOT seq-cst. Seq-cst operations are *super rare*.
   a. I just searched our entire engine codebase; there are 971 calls
 to atomic functions; there are *four* instances of seq-cst, and I
 think at least some of them are actually wrong (over-synchronised),
 possibly all of them, I would need to consider closer.
 2. Many atomic operations appear in a series, in which case, it's
 typical that exactly one operation in the series may be the
 synchronisation point.
 3. In reality, probably at least half of operations are 'relaxed'.
 4. xchg and cas operations are fundamental to any atomic code, and
 there are no syntax or operators for those operations; so, you're
 undoubtedly using core.atomic aggressively anyway, and I think it's
 just weird to litter naked arithmetic in between calls to core.atomic
 functions. I feel that's misleading by making the naked arithmetic
 look like NOT-atomic operations by contrast to a bunch of atomic
 library calls.
 5. In almost all cases when working with atomics, you tend to do
 quirky stuff not captured by regular usage of data types, and not
 conveniently facilitated by this DIP.

 For instance; this is a common implementation for releasing a ref count:
   if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed))
   {
     // maybe acquire if necessary
     // do release logic...
   }

 How do I write that code with this DIP?

 We need to make shared correct before we do any stuff like this, and
 then when it's correct, we can talk... but I will probably continue to
 resist this, because the only result I see is that it will train
 amateurs to write bad atomic algorithms with an improper sense of
 confidence gained by being sanctioned by the language.

 This is a bad idea.


 Sorry, that should have read:

   if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed) == 1)
   {
     // maybe acquire if necessary
     // do release logic...
   }


Wait up... it's possible that I've misunderstood this DIP...

There are 2 paragraphs that seem to contradict each other.

First there is A:
"""
By prohibiting direct access to shared data, the user will be required
to use core.atomic and to consider the correctness of their code.



This change will require using core.atomic or equivalent functions to
read and write to shared memory objects. It will prevent unintended,
inadvertent non-use of atomic access.
"""

I absolutely agree with this text.

Then there's B:
"""
Atomic reads perform an acquire operation, writes perform a release
operation, and read-modify-write performs an acquire, then a
modification, and then a release. The result is sequentially
consistent ordering, and each thread observes the same order of
operations (total order).

The code generated would be equivalent to that from the core.atomic
library for the supported operations.
"""

And then goes on with examples like this where `x` is clearly accessed
without a call to core.atomic:
```
__gshared int g=0;
shared int x=0;

void thread1(){
    g=1; // non-atomic store
    x=1; // atomic store, release, happens after everything above
}

void thread2(){
    if(x==1){ // atomic load, acquire, happens before everything below
        assert(g==1); // non-atomic load
    }
}
```

I'm confused, which is it, A or B? I presume B, and my prior commentary applies.
Oct 01 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 11:30 PM, Manu wrote:

 I'm confused, which is it, A or B? I presume B, and my prior commentary
applies.


A
Oct 02 2019
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 10/1/2019 11:30 PM, Manu wrote:

 I'm confused, which is it, A or B? I presume B, and my prior commentary
applies.


 A


In that case, I don't understand the only code shown in the DIP; you
show freely reading from and assigning to a shared int... that should
be an error, and there should be atomicLoad/astomicStore in those
locations?
Oct 02 2019
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 2 October 2019 at 17:21:59 UTC, Manu wrote:

 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 On 10/1/2019 11:30 PM, Manu wrote:

 I'm confused, which is it, A or B? I presume B, and my prior 
 commentary applies.


 A


 In that case, I don't understand the only code shown in the 
 DIP; you show freely reading from and assigning to a shared 
 int... that should be an error, and there should be 
 atomicLoad/astomicStore in those locations?


My understanding is that the compiler rewrites that code to use 
core.atomic.
Oct 02 2019
next sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Wednesday, 2 October 2019 at 17:51:09 UTC, Atila Neves wrote:

 On Wednesday, 2 October 2019 at 17:21:59 UTC, Manu wrote:

 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 On 10/1/2019 11:30 PM, Manu wrote:

 I'm confused, which is it, A or B? I presume B, and my 
 prior commentary applies.


 A


 In that case, I don't understand the only code shown in the 
 DIP; you show freely reading from and assigning to a shared 
 int... that should be an error, and there should be 
 atomicLoad/astomicStore in those locations?


 My understanding is that the compiler rewrites that code to use 
 core.atomic.


Why can't it import core.atomic automatically then?

-Alex
Oct 02 2019
prev sibling parent Manu <turkeyman gmail.com> writes:
On Wed, Oct 2, 2019 at 10:55 AM Atila Neves via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Wednesday, 2 October 2019 at 17:21:59 UTC, Manu wrote:

 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 10/1/2019 11:30 PM, Manu wrote:

 I'm confused, which is it, A or B? I presume B, and my prior
 commentary applies.


 A


 In that case, I don't understand the only code shown in the
 DIP; you show freely reading from and assigning to a shared
 int... that should be an error, and there should be
 atomicLoad/astomicStore in those locations?


 My understanding is that the compiler rewrites that code to use
 core.atomic.


I asked Walter a couple posts up whether or not that's what he meant
(A/B), and he said A:
"""
By prohibiting direct access to shared data, the user will be required
to use core.atomic and to consider the correctness of their code.
This change will require using core.atomic or equivalent functions to
read and write to shared memory objects. It will prevent unintended,
inadvertent non-use of atomic access.
"""
...which is distinctly not that.
Oct 02 2019
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/2/2019 10:21 AM, Manu wrote:

 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 10/1/2019 11:30 PM, Manu wrote:

 I'm confused, which is it, A or B? I presume B, and my prior commentary
applies.


 A


 
 In that case, I don't understand the only code shown in the DIP; you
 show freely reading from and assigning to a shared int... that should
 be an error, and there should be atomicLoad/astomicStore in those
 locations?
 


Yes. The trouble stems from the DIP being originally written to apply shared 
semantics to operators. This was changed to be function only, and some vestiges 
of the original remain. Hence the confusion.
Oct 03 2019
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On Wed, Oct 2, 2019 at 10:21 AM Manu <turkeyman gmail.com> wrote:

 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 10/1/2019 11:30 PM, Manu wrote:

 I'm confused, which is it, A or B? I presume B, and my prior commentary
applies.


 A


 In that case, I don't understand the only code shown in the DIP; you
 show freely reading from and assigning to a shared int... that should
 be an error, and there should be atomicLoad/astomicStore in those
 locations?


Also the text: "Atomic reads perform an acquire operation, writes
perform a release
operation, and read-modify-write performs an acquire, then a
modification, and then a release."

What does that mean? I don't know what that's talking about.
Please correct the DIP so we can understand it.
Oct 02 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/2/2019 10:25 AM, Manu wrote:

 Also the text: "Atomic reads perform an acquire operation, writes
 perform a release
 operation, and read-modify-write performs an acquire, then a
 modification, and then a release."
 
 What does that mean? I don't know what that's talking about.
 Please correct the DIP so we can understand it.


That follows the usage pattern in the C++ memory model documentation. I agree
it 
is hard to follow, but it is normal jargon for those familiar with the 
subtleties of multi-threaded memory models.

I have written a spec PR that starts to formally define the memory model for D, 
but it's been sitting there for a long time and nobody wants to touch it. I 
don't think a DIP is the right place for it.

I see no reason to deviate from the C++ memory model semantics.
Oct 03 2019
parent Manu <turkeyman gmail.com> writes:
On Thu., 3 Oct. 2019, 8:45 pm Walter Bright via Digitalmars-d, <
digitalmars-d puremagic.com> wrote:


 On 10/2/2019 10:25 AM, Manu wrote:

 Also the text: "Atomic reads perform an acquire operation, writes
 perform a release
 operation, and read-modify-write performs an acquire, then a
 modification, and then a release."

 What does that mean? I don't know what that's talking about.
 Please correct the DIP so we can understand it.


 That follows the usage pattern in the C++ memory model documentation. I
 agree it
 is hard to follow, but it is normal jargon for those familiar with the
 subtleties of multi-threaded memory models.


I understand the language used, I have no idea how it applies to this DIP,
specifically because I'm still not sure what it does, since various
paragraphs and the example contradict each other.

I look forward to a revision that's easy to follow.
Oct 04 2019
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On Tue, Oct 1, 2019 at 11:30 PM Manu <turkeyman gmail.com> wrote:

 [...]


And then I notice everyone else has pointed out the confusion too.
Sorry for the noise!
Please clarity the confusion, then I can revoke my criticisms, or not...
Oct 01 2019
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
Access to shared memory should be disallowed only for safe code, 
but not for system code. For safe code it's justified, because 
safety is unavoidable there, for system code there's a concern of 
ease of use of shared data, which is affected by littering and is 
not addressed in this DIP. Stripping shared qualifier off for 
assignment isn't necessarily correct if, say, the data type 
switches between atomic and nonatomic reference counting based on 
its type.
Oct 02 2019
next sibling parent reply Kagamin <spam here.lot> writes:
Also cast() doesn't strip qualifiers

shared int[] a;
pragma(msg,typeof(cast()a).stringof);

prints "shared(int)[]"
Oct 02 2019
parent Meta <jared771 gmail.com> writes:
On Wednesday, 2 October 2019 at 09:00:54 UTC, Kagamin wrote:

 Also cast() doesn't strip qualifiers

 shared int[] a;
 pragma(msg,typeof(cast()a).stringof);

 prints "shared(int)[]"


That's because `shared int[] a` is really `shared(int[] a)`, so 
cast() only strips the top level shared off.
Oct 02 2019
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:

 Access to shared memory should be disallowed only for safe 
 code, but not for system code.


That definition doesn't play nice with safety inference i.e. 
templates.

It should be that shared memory access is disabled in all 
contexts,
which must be worked around with casts,
which makes the function  system,
which must then be encapsulated with a  trusted interface in 
order to use in  safe code.
Oct 02 2019
next sibling parent Kagamin <spam here.lot> writes:
On Wednesday, 2 October 2019 at 10:42:48 UTC, Nicholas Wilson 
wrote:

 On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:

 Access to shared memory should be disallowed only for safe 
 code, but not for system code.


 That definition doesn't play nice with safety inference i.e. 
 templates.


I don't think so. How?
Oct 03 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/2/2019 3:42 AM, Nicholas Wilson wrote:

 On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:

 Access to shared memory should be disallowed only for safe code, but not for 
 system code.


 
 That definition doesn't play nice with safety inference i.e. templates.
 
 It should be that shared memory access is disabled in all contexts,
 which must be worked around with casts,
 which makes the function  system,
 which must then be encapsulated with a  trusted interface in order to use in 
  safe code.


Sounds right.
Oct 11 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via Digitalmars-d 
wrote:

 On 10/2/2019 3:42 AM, Nicholas Wilson wrote:

 On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:

 Access to shared memory should be disallowed only for safe code, but
 not for system code.


 That definition doesn't play nice with safety inference i.e. templates.

 It should be that shared memory access is disabled in all contexts,
 which must be worked around with casts,
 which makes the function  system,
 which must then be encapsulated with a  trusted interface in order to
 use in  safe code.


 Sounds right.


This is pretty much what several of us were arguing for during the
discussions about shared at dconf this year.

- Jonathan M Davis
Oct 11 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 06:31:18 UTC, Jonathan M Davis 
wrote:

 On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via 
 Digitalmars-d wrote:

 On 10/2/2019 3:42 AM, Nicholas Wilson wrote:





 It should be that shared memory access is disabled in all 
 contexts,
 which must be worked around with casts,
 which makes the function  system,
 which must then be encapsulated with a  trusted interface in 
 order to
 use in  safe code.


 Sounds right.


 This is pretty much what several of us were arguing for during 
 the discussions about shared at dconf this year.


How are you going to prove that  safe code does not retain 
nonshared-references after the lock has been released?

How does his work with array elements/slices.
Oct 12 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 3:15:26 AM MDT Ola Fosheim Grøstad via 
Digitalmars-d wrote:

 On Saturday, 12 October 2019 at 06:31:18 UTC, Jonathan M Davis

 wrote:

 On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via

 Digitalmars-d wrote:

 On 10/2/2019 3:42 AM, Nicholas Wilson wrote:

 It should be that shared memory access is disabled in all
 contexts,
 which must be worked around with casts,
 which makes the function  system,
 which must then be encapsulated with a  trusted interface in
 order to
 use in  safe code.


 Sounds right.


 This is pretty much what several of us were arguing for during
 the discussions about shared at dconf this year.


 How are you going to prove that  safe code does not retain
 nonshared-references after the lock has been released?

 How does his work with array elements/slices.


It's the same as with any  trusted code. It's up to the programmer to ensure
that what the code is doing is actually  safe, and if the code isn't able to
provide an API that is  safe, then it shouldn't be  trusted. Ultimately, if
the programmer casts away shared while a mutex is locked, it's up to the
programmer to ensure that no thread-local references to the shared data
escape (though scope can help with that) so that when the lock is released,
the only references left are shared.

- Jonathan M Davis
Oct 12 2019
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis 
wrote:

 It's the same as with any  trusted code. It's up to the 
 programmer to ensure that what the code is doing is actually 
  safe, and if the code isn't able to provide an API that is 
  safe, then it shouldn't be  trusted.


But didn't Walter in another thread some time ago say that he 
would add Rust-like linear typing for memory management?

If this is the goal, shouldn't you also use linear typing for 
allowing more flexible  safe/ trusted APIs?


 Ultimately, if the programmer casts away shared while a mutex 
 is locked, it's up to the programmer to ensure that no 
 thread-local references to the shared data escape (though scope 
 can help with that) so that when the lock is released, the only 
 references left are shared.


How does that work with arrays. E.g. processing that has 
coroutine/ranges style implementations.  How can you ensure that 
it is impossible for the lock to be released and also making sure 
that all  safe single-threaded array processing code can be used?

You often want to use "shared" for large sections of arrays.

Another thing is that it seems to me that things could go wrong 
if you have slices and dynamic arrays. E.g.:

a. you take a slice of the last element of an unshared dynamic 
array and hand it to one thread.

b. you reduce the length of the dynamic array

c. you hand out an unshared version of the dynamic array to 
another thread which then increase the length of the dynamic 
array in  safe code

Now you have two  safe unshared references to the last element in 
two different threads? I don't have much experience with 
extending dynamic arrays, because I don't like the D semantics of 
it, but isn't this roughly how things could go?
Oct 12 2019
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 10:22:38 UTC, Ola Fosheim Grøstad 
wrote:

 c. you hand out an unshared version of the dynamic array to 
 another thread which then increase the length of the dynamic 
 array in  safe code


I'm sorry. I mixed up Go and D dynamic array semantics... :-/

I thought they were exactly the same, but apparently D always 
reallocates while Go tries to avoid reallocation.  I dislike the 
semantics of Go too... that being said.

Still, it might be an issue for library types, even though it 
appears to work out for builtins (?).
Oct 12 2019
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 4:22:38 AM MDT Ola Fosheim Grøstad via 
Digitalmars-d wrote:

 On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis

 wrote:

 It's the same as with any  trusted code. It's up to the
 programmer to ensure that what the code is doing is actually
  safe, and if the code isn't able to provide an API that is
  safe, then it shouldn't be  trusted.


 But didn't Walter in another thread some time ago say that he
 would add Rust-like linear typing for memory management?

 If this is the goal, shouldn't you also use linear typing for
 allowing more flexible  safe/ trusted APIs?


I don't know anything about what you mean by linear types (I really don't
know much about Rust). So, I can't say how that would come into play, but in
general, with  trusted, the programmer must verify that what the code is
doing is  safe and must provide an API that is  safe. If the programmer
can't guarantee that what the code is doing is  safe, then they shouldn't be
marking it is  trusted. Any operations that the compiler can guarantee as
 safe are already considered  safe. By definition, once  system is
involved, it's because the compiler can't guarantee that the code is  safe,
and  trusted is the programmer's tool for telling the compiler that a
particular piece of code is actually  safe even though the compiler couldn't
verify that.

In the case of shared, in general, it's not thread-safe to read or write to
such a variable without either using atomics or some other form of thread
synchronization that is currently beyond the ability of the compiler to make
guarantees about and will likely always be beyond the ability of the
compiler to make guarantees about except maybe in fairly restricted
circumstances. The type system separates thread-local from shared, but it
doesn't have any concept of ownership and simply doesn't have the kind of
information required to figure out whether something is thread-safe. The
closest that anyone has proposed is TDPL's synchronized classes, and even
those could only safely remove the outer layer of shared - and even that
required locking down synchronized classes pretty thoroughly.

That's why it doesn't make sense to allow reading from or writing to shared
variables, and it has to be up to the programmer to deal with the
synchronization primitives and temporarily casting to thread-local to
operate on the shared data while it's protected by those synchronization
primitives. The casts automatically make such code  system as it should be,
and then the programmer is going to have to guarantee the thread-safety just
like they'd do in a language like Java or C++. The difference is that D's
type system helps the programmer, whereas languages like Java or C++ leave
it entirely up to the programmer to get it right. The type system segregates
the data that's shared, and the code that does all of the tricky threading
stuff is  system, requiring the programmer to verify that piece of code and
mark it as  trusted to then be used by the  safe portions of the program.
Yes, the casts make the code more verbose than it would be in C++, but other
than that, you're basically doing exactly the same thing that you'd be doing
in C++.

By no means does that mean that any of the threading stuff is easy, but as
long as the type system cannot make guarantees about thread-safety,
segregating the code that deals with the threading and requiring that the
programmer verify its correctness in order to have it work with  safe code
is the best that we can do. Regardless, even if we're able to later come up
with improvements that allow shared to be implicitly removed under some set
of circumstances (and thus have such code be  safe), shared in general still
needs to be locked down in a manner similar to what this DIP is proposing.
Any improvements that we later come up with would then just be laid on top
of that.

- Jonathan M Davis
Oct 12 2019
parent reply Gregor =?UTF-8?B?TcO8Y2ts?= <gregormueckl gmx.de> writes:
On Saturday, 12 October 2019 at 20:52:45 UTC, Jonathan M Davis 
wrote:

 In the case of shared, in general, it's not thread-safe to read 
 or write to such a variable without either using atomics or 
 some other form of thread synchronization that is currently 
 beyond the ability of the compiler to make guarantees about and 
 will likely always be beyond the ability of the compiler to 
 make guarantees about except maybe in fairly restricted 
 circumstances.


A shared variable may not need any synchronization at all, 
depending on the algorithm it is used in.

There is a class of optimized algorithms that act like gathering 
operations. You mostly find them on GPUs because they map quite 
naturally to that hardware architecture, but they can also be 
implemented on CPUs with multiple threads. The core idea is in 
every case that you generate a set of output values in parallel 
in such a way that each value in the output is generated by at 
most one of the running threads. So there is no need to 
synchronize memory writes when the underlying hardware 
architecture provides sufficient cache coherency guarantees. All 
the threads share the same input, which obviously must not be 
modified.

A compiler cannot possibly be smart enough to prove that 
synchronization is not required for these kinds of algorithms. 
And any form of enforced synchronization (explicit or implicit) 
would significantly affect performance. So how would you express 
this implicit synchronization by mathematical properties to the 
compiler? Would the whole implementation have to be marked as 
 safe?
Oct 15 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 15 October 2019 at 10:25:54 UTC, Gregor Mückl wrote:

 generated by at most one of the running threads. So there is no 
 need to synchronize memory writes when the underlying hardware 
 architecture provides sufficient cache coherency guarantees.


So what you basically is saying is that a low level language 
should be careful about assuming a particular hardware model and 
leave more to intrinsics and libraries. I think that is 
reasonable, because hardware does change and concurrent 
programming strategies change.

So D has to figure out whether it is a low level language for 
many architectures or a specific x86 centric one limited to 
common contemporary programming patterns.

My perception is that D does not need to lock down «shared» but 
can rather improve on these:

1. A set of basic lowlevel building blocks + intrinsics (which D 
has mostly).

2. Metaprogramming features to build libraries that support 
contemporary patterns (which D has mostly).

3. Language features that support clean static analysis tooling 
(perhaps lacking).


But given the history of D, it will probably go with what is 
demanded based on contemporary x86 patterns.
Oct 15 2019
parent reply Gregor =?UTF-8?B?TcO8Y2ts?= <gregormueckl gmx.de> writes:
On Tuesday, 15 October 2019 at 11:28:24 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 15 October 2019 at 10:25:54 UTC, Gregor Mückl wrote:

 generated by at most one of the running threads. So there is 
 no need to synchronize memory writes when the underlying 
 hardware architecture provides sufficient cache coherency 
 guarantees.


 So what you basically is saying is that a low level language 
 should be careful about assuming a particular hardware model 
 and leave more to intrinsics and libraries. I think that is 
 reasonable, because hardware does change and concurrent 
 programming strategies change.


I'm not sure if this is quite what I'm saying. I guess I'm fine 
with the compiler telling me that a piece of code tries to access 
shared data without any annotation that this is what is desired. 
In my opinion, it then needs to be up to the developer to deal 
with this. It is *not* OK for the compiler to go and secretly 
insert synchronization mechanisms (memory barriers, atomic 
operations, etc...) behind the developer's back. If such an 
automatism must exist, it also must be invoked by the developer 
explicitly ("Computer, generate automatic synchronization!"). And 
there must be a way to tell the compiler that the developer is a 
responsible adult who knows what he/she is doing and that the 
code is OK for reasons unknown to the compiler.
Oct 16 2019
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, October 16, 2019 12:43:30 PM MDT Gregor Mückl via Digitalmars-
d wrote:

 On Tuesday, 15 October 2019 at 11:28:24 UTC, Ola Fosheim Grøstad

 wrote:

 On Tuesday, 15 October 2019 at 10:25:54 UTC, Gregor Mückl wrote:

 generated by at most one of the running threads. So there is
 no need to synchronize memory writes when the underlying
 hardware architecture provides sufficient cache coherency
 guarantees.


 So what you basically is saying is that a low level language
 should be careful about assuming a particular hardware model
 and leave more to intrinsics and libraries. I think that is
 reasonable, because hardware does change and concurrent
 programming strategies change.


 I'm not sure if this is quite what I'm saying. I guess I'm fine
 with the compiler telling me that a piece of code tries to access
 shared data without any annotation that this is what is desired.
 In my opinion, it then needs to be up to the developer to deal
 with this. It is *not* OK for the compiler to go and secretly
 insert synchronization mechanisms (memory barriers, atomic
 operations, etc...) behind the developer's back. If such an
 automatism must exist, it also must be invoked by the developer
 explicitly ("Computer, generate automatic synchronization!"). And
 there must be a way to tell the compiler that the developer is a
 responsible adult who knows what he/she is doing and that the
 code is OK for reasons unknown to the compiler.


The DIP is not clear on the matter and needs to be updated, but Walter has
made it clear in his comments that there is no plan to even insert
core.atomic calls for you. Reading and writing to shared data is going to be
illegal, requiring that the programmer either explicitly use core.atomic or
that they cast away shared (rendering that part of the code  system, thereby
segregating the code that the programmer has to verify for thread safety).

It is conceivable that someone will come up with a feature that will allow
the compiler to implicitly remove shared under some set of circumstances,
because it's able to see that only one thread can access that data at that
point (e.g. TDPL's synchronized classes would be able to do this to the
outer layer of shared for the member variables of that class), but that
would just be an improvement on top of what Walter is proposing, and
honestly, I doubt that we'll ever see much along those lines, because making
such compiler guarantees is very difficult with D's type system (e.g. TDPL's
synchronized classes lock the type down quite a lot and yet are still only
able to remove a single layer of shared, making them borderline useless, and
AFAIK, no one has yet proposed anything that could do better). The type
system would likely need a concept of thread ownership to safely reason
about much with regards to shared, and even if the compiler _were_ able to
implicitly remove shared under some set of circumstances, there's no way
that it's going to understand all of the various threading mechanisms that
get used in code. So, at best, you'd be able to use a particular feature to
have a piece of code implicitly remove shared, because the compiler is able
to do it for that particular idiom. There's no question that the programmer
is going to have to cast away shared in many cases in order to actually
operate on shared data.

And once the programmer casts away shared, that code is then  system,
requiring the programmer to vet it and certify that it's actually  safe by
using  trusted. So, all of the code that operates on shared data should then
be in three camps:

1. It uses core.atomic.

2. It casts away shared and is thus  system.


internally.

Andrei has talked in the past about inserting magic to make shared "just
work" without casting, but I think that it's pretty clear at this point that
that isn't going to work. The best that we could do would be to make some
operations which are guaranteed to be atomic allowed on shared primitive
types, and that would be a questionable choice for a variety of reasons.
Either way, it's not currently the plan to do so.

- Jonathan M Davis
Oct 16 2019
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis 
wrote:

 It's the same as with any  trusted code. It's up to the 
 programmer to ensure that what the code is doing is actually 
  safe, and if the code isn't able to provide an API that is 
  safe, then it shouldn't be  trusted.


So, I am trying to figure out what that API would have to look 
like and the only solution I can think of right away is that you 
have are doing a callback back into  safe code from the 
lock-holding  trusted code?

So, if you have to obtain locks from multiple sources, it either 
won't work out, or you have to use some serious kludges, like a 
deep call chain:

while(tryagain) {
     trusted_code_obtains_lock_1-->
        safe_code_callback1 calls->
            trusted_code_obtains_lock2
                safe_code_callback2 processes and sets 
tryagain=false
}

Of course, that could lead to starvation.

So to avoid all of that multi-threaded libraries have to have 2 
APIs, one simple transactional wrapper-API for  safe code and 
another API for  trusted code that can be used to write a new 
wrapper that allows locking of multiple datastructures.

I hope people try to sketch out what such libraries will look 
like before the semantics are decided on. Maybe an experimental 
branch would be appropriate.
Oct 12 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 5:26:42 AM MDT Ola Fosheim Grøstad via 
Digitalmars-d wrote:

 I hope people try to sketch out what such libraries will look
 like before the semantics are decided on. Maybe an experimental
 branch would be appropriate.


The fact that we have thread-local in the language requires that the type
system then have the concept of shared. So, there should be no question
about that. As part of that, converting between thread-local and shared has
to require casts (otherwise, the compiler can't guarantee that the
thread-local stuff is actually thread-local). Those casts then have to be
 system, because they potentially violate the compiler's guarantees about
thread-local and shared data.

The question that then comes is what kinds of operations should be allowed
on shared, and there are basically two approaches that can be taken:

1. Not allow read-write operations that aren't guaranteed to be atomic.

2. Ignore thread-safety and just let people do whatever they want to with
shared other than implicitly convert between shared and thread-local (since
that would violate the compiler guarantees about which data is shared and
which is thread-local).


stuff in a manner that requires the programmer to get involved, because
operating on shared in a way that isn't guarantee to be atomic then becomes
 system. So, it becomes far easier to locate the pieces of code that need to
be verified as thread-safe by the programmer, and you get actual compiler
guarantees that any  safe code operating on shared variables doesn't have
any threading problems. Only the  trusted code has to be examined.


shared was thread-safe. You wouldn't have to do as much casting, because
basic operations would work on shared data, but you'd then have to examine
all code involving shared to determine whether what it was doing was
thread-safe. And because many types are designed to be thread-local rather
than shared, you'd still have to cast to thread-local in many cases, and
that code would still have to be  system, because the programmer is
potentially violating the compiler guarantees about thread-local when
casting to or from shared.


worse job segregating code that needed to be examined when dealing with
threading issues, and it doesn't really remove the casting and the like that
folks like to complain about with regards to shared. We were locked into
that once we had shared as part of the type system. Languages like C++ can
only get away with not having it because they ignore the issue. That means
that in such languages, you can just operate on shared stuff as if it were
thread-local, but the programmer then also has to worry about threading
issues throughout the entire program, because the type system isn't helping
them at all. Good programming practices help make that problem more
manageable, but by having shared as part of the type system in D, the
language and the compiler make the problem much more manageable by
segregating the threading code.




isn't guaranteed to be atomic, the compiler prevents you from doing it
unless you  use core.atomic or have  system code with casts. Even if we have



It may be that if we can come up with ways for the compiler to know that
it's thread-safe to implicitly remove shared in a piece of code, libraries
won't have to have as much  trusted code, but the low level semantics would
still need to be the same, and any code that wasn't using whatever idiom or
feature allowed the compiler to implicitly remove shared in a piece of code
would still need to do what it would need to do right now, which is what it
would need to do if this DIP were implemented.

All this DIP really does is force code that the compiler can't guarantee is
atomic to use either core.atomic or be  system/ trusted. The idioms and
design patterns involved with thread-safe code are the same either way.

- Jonathan M Davis
Oct 12 2019
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis 
wrote:

 The fact that we have thread-local in the language requires 
 that the type system then have the concept of shared. So, there 
 should be no question about that.


Well, my understanding from what you wrote is that the shared 
marker is 100% syntactical and has no semantic implications. In 
that case it can be done as a library meta programming construct 
(type wrapping). You just need a way to restrict application of 
the constructs  based on code structure ( safe). So one could 
establish a syntactical solution that is perfectly suitable for 
meta programming?

But you could probably do better by introducing an effectsbased 
typesystem.

Anyway, based on what you said shared does not have to be in the 
language and could be replaced by more expressive metaprogramming 
mechanisms. I think.
Oct 12 2019
parent reply Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 3:35 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis
 wrote:

 The fact that we have thread-local in the language requires
 that the type system then have the concept of shared. So, there
 should be no question about that.


 Well, my understanding from what you wrote is that the shared
 marker is 100% syntactical and has no semantic implications.


The semantic implication is that there is no read or write access...
just the same as how const says there is no write access.


 In that case it can be done as a library meta programming construct
 (type wrapping).


We can't define type constructors in a library sadly. You could apply
the same argument to const.

And aside from that, it belongs in the language due to the simple fact
that it defines D's thread-local by default.
If you think about it, it's not really the definition of thread-local
by default that leads to shared... it's actually that shared existing
in the language defines thread-local by default.
If there was no shared, then there couldn't possibly be thread-local
by default, as D is specified. shared is fundamental to D, it's just
that it's semantics have been poorly understood because it's
historically had relatively little use.


 You just need a way to restrict application of
 the constructs  based on code structure ( safe). So one could
 establish a syntactical solution that is perfectly suitable for
 meta programming?

 But you could probably do better by introducing an effectsbased
 typesystem.

 Anyway, based on what you said shared does not have to be in the
 language and could be replaced by more expressive metaprogramming
 mechanisms. I think.


It's like I said above; shared MUST be in the language, because it
defines one of the fundamental language semantics; thread-local by
default.
Oct 12 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 00:13:12 UTC, Manu wrote:

 On Sat, Oct 12, 2019 at 3:35 PM Ola Fosheim Grøstad via The 
 semantic implication is that there is no read or write 
 access... just the same as how const says there is no write 
 access.


Well, I understand what you mean, but if you can achieve the same 
effect by hiding, then it is syntactical in nature.

That would be difficult with const in D ( might work in other 
languages ).


 It's like I said above; shared MUST be in the language, because 
 it defines one of the fundamental language semantics; 
 thread-local by default.


Does the shared marker have any effect on generated code? If not, 
then it is just a way to bypass (break) the type system on a 
syntactical level.
Oct 12 2019
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 8:45 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Sunday, 13 October 2019 at 00:13:12 UTC, Manu wrote:

 On Sat, Oct 12, 2019 at 3:35 PM Ola Fosheim Grøstad via The
 semantic implication is that there is no read or write
 access... just the same as how const says there is no write
 access.


 Well, I understand what you mean, but if you can achieve the same
 effect by hiding, then it is syntactical in nature.


We can't though.


 That would be difficult with const in D ( might work in other
 languages ).


 It's like I said above; shared MUST be in the language, because
 it defines one of the fundamental language semantics;
 thread-local by default.


 Does the shared marker have any effect on generated code?


Not at this time. And I'm not sure what your point is... does const
have any effect in generated code? (it does not)


 If not,
 then it is just a way to bypass (break) the type system on a
 syntactical level.


I don't understand this.. how is it bypassing/breaking the type system?
Oct 12 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.no> writes:
On Sunday, 13 October 2019 at 03:49:17 UTC, Manu wrote:

 On Sat, Oct 12, 2019 at 8:45 PM Ola Fosheim Grøstad via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Well, I understand what you mean, but if you can achieve the 
 same effect by hiding, then it is syntactical in nature.


 We can't though.


Assume that symbols prefixed by  trusted_ are only visible in 
 trusted code. Then you can define  trusted_unwrap that returns a 
reference to the unwrapped type.

Then you have what you need for a simple solution to a library 
version of shared.



 Not at this time. And I'm not sure what your point is... does 
 const have any effect in generated code? (it does not)


Const could also be done as a library wrapper, but that would 
require full fledeged meta programming and strong deductive 
functionality. So not realistic for D.


 If not,
 then it is just a way to bypass (break) the type system on a
 syntactical level.


 I don't understand this.. how is it bypassing/breaking the type 
 system?


If you need a cast to make use of shared...
Oct 12 2019
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 9:41:00 PM MDT Ola Fosheim Grøstad via 
Digitalmars-d wrote:

 It's like I said above; shared MUST be in the language, because
 it defines one of the fundamental language semantics;
 thread-local by default.


 Does the shared marker have any effect on generated code? If not,
 then it is just a way to bypass (break) the type system on a
 syntactical level.


The compiler is free to assume that anything that isn't shared is
thread-local and optimize code accordingly. Also, it _does_ matter on some
level with how memory is laid out, because thread-local storage gets used
(IIRC, there were problems on older versions of OS X, because we had to fake
the TLS, because OS X didn't provide proper TLS). Module-level and static
variables which are shared only get initialized once, whereas module-level
and static variables which are thread-local get initialized once per thread,
because each thread gets their own copy. So, while the vast majority of D
objects are thread-local, the distinction between what's thread-local and
what's shared is very much built into how D functions.

That's part of why it's so risky to cast away shared, and why it needs to be
 system. If the programmer screws up that code and allows any shared data to
be treated as thread-local when any other thread could be operating on that
data, then not only does that incur all of the typical issues that come with
screwing up synchronizing data access, but the compiler could make things
worse, because it generated code based on the assumption that the data being
operated on was thread-local.

Fortunately, as long as shared disallows non-atomic, read/write operations
the code that needs to be examined for threading issues is restricted to
 system/ trusted code that involves shared, so the amount of code that the
programmer has to examine and verify to ensure that shared data doesn't end
up being treated as thread-local when multiple threads can operate on it is
limited and should be fairly easy to find.

- Jonathan M Davis
Oct 12 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 04:09:26 UTC, Jonathan M Davis 
wrote:

 The compiler is free to assume that anything that isn't shared 
 is thread-local and optimize code accordingly.


That has no real effect though, if you throw away shared before 
using it.

That said, if you had a formalization of the threads and what 
states different threads are in then you could get some 
performance benefits by eliding shared overhead when it can be 
proven that no other threads are accessing a shared variable.

But that is not on the table...


 Also, it _does_ matter on some level with how memory is laid 
 out, because thread-local storage gets used (IIRC, there were 
 problems on older versions of OS X, because we had to fake the 
 TLS, because OS X didn't provide proper TLS).


Actually, how the memory is laid out is not a big deal. Vendors 
can do this with a library solution too. What is a big deal with 
TLS is that you need different codegen for obtaining a reference, 
but that also mean that you cannot throw away that type 
information before you have a obtained an effective address 
(reference)?

Same thing with shared. On an architecture that requires a 
different instruction sequences for shared memory and local 
memory, you would need to retain that information. Not for 
allocation, which can be dealt with at the library level, but for 
accessing it when executing generic functions.

If anything that would be an argument for never being allowed to 
throw away the shared marker, and also an argument for being able 
to add more markers (for different types of memory or I/O).



 Module-level and static variables which are shared only get 
 initialized once, whereas module-level and static variables 
 which are thread-local get initialized once per thread, because 
 each thread gets their own copy.


I'd rather say that a TLS variable "value" is conceptually 
"value[thread_id]" or "value__12345".



 Fortunately, as long as shared disallows non-atomic, read/write 
 operations the code that needs to be examined for threading 
 issues is restricted to  system/ trusted code that involves 
 shared, so the amount of code that the programmer has to 
 examine and verify to ensure that shared data doesn't end up 
 being treated as thread-local when multiple threads can operate 
 on it is limited and should be fairly easy to find.


And you can achieve all that with a library type, because D 
doesn't actually deal with shared memory, it provides some 
instructions that results in barriers in the IR.

If you wrap all those barrier-generating instructions in library 
calls then they can just avoid emitting them if the type they are 
applied to is not wrapped in a  library provided Shared wrapper.

I don't see why the compiler has to know anything about shared 
for this to work the same way as described.

Anyway, I think D really needs to think twice about adding more 
and more narrow features and instead improve on the meta 
programming capabilities. Clearly many D features could be moved 
to the library (slicing, dicts etc).

If there is a limited pool of people working on the compiler it 
would make a lot of sense to put more power into the hands of 
library authors.  The meta-programming capabilities of C++ is 
evolving too...
Oct 13 2019
parent reply Manu <turkeyman gmail.com> writes:
On Sun, Oct 13, 2019 at 12:55 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Sunday, 13 October 2019 at 04:09:26 UTC, Jonathan M Davis
 wrote:

 The compiler is free to assume that anything that isn't shared
 is thread-local and optimize code accordingly.


 That has no real effect though, if you throw away shared before
 using it.

 That said, if you had a formalization of the threads and what
 states different threads are in then you could get some
 performance benefits by eliding shared overhead when it can be
 proven that no other threads are accessing a shared variable.


What does that even mean? You're just making things up.
Thread's don't have 'states', that's literally the point of threads!
No code-gen can ever assume any stateful details of any kind about
another thread.
Maybe there's a 10 year research project there, but that idea is so
ridiculous and unlikely to work that nobody would ever try.

I think what you're talking about is something more like a framework;
where different threads are running known code which is configured to
cooperate in particular ways, and implements safe data transmission
between threads.
This is what will exist, there will be elaborate libraries of this
kind. I don't expect many end-users would ever encounter a `shared`
object, it would likely be behind the curtains in all cases, unless
they're writing very low-level systems.


 But that is not on the table...


Definitely not. Not at the language level at least.
Oct 13 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 19:08:00 UTC, Manu wrote:

 On Sun, Oct 13, 2019 at 12:55 AM Ola Fosheim Grøstad via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:



 That said, if you had a formalization of the threads and what 
 states different threads are in then you could get some 
 performance benefits by eliding shared overhead when it can be 
 proven that no other threads are accessing a shared variable.


 What does that even mean? You're just making things up.


(Drop the ad hominems... Especially when your in an area that is 
not your field.)



 Thread's don't have 'states', that's literally the point of 
 threads!


All program execution is in a state at any given point in time.

So yes, you transition between states. That is the premise for 
building an advanced type system. That's what makes it possible 
to implement Rusts borrow checker.

Anything that moves in a discrete fashion will move between 
states, that is a basic computer science conceptualization of 
computing. It is the foundation that theories about computing is 
built on.


 Maybe there's a 10 year research project there, but that idea 
 is so
 ridiculous and unlikely to work that nobody would ever try.


Oh. People do it. Sure, it is hard in the most general case, but 
you do have languages with high level concurrency constructs that 
provide type systems that makes it possible to write provably 
correct concurrent programs.

It doesn't make much sense to claim that nobody would ever try to 
build something that actually exists... Does it?



 I think what you're talking about is something more like a 
 framework;


No. Languages/compilers that check concurrency states at compile 
time.



 But that is not on the table...


 Definitely not. Not at the language level at least.


That's right. So those benefits are not avilable... which was my 
point. :-)
Oct 14 2019
parent reply Manu <turkeyman gmail.com> writes:
On Mon, Oct 14, 2019 at 5:50 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Sunday, 13 October 2019 at 19:08:00 UTC, Manu wrote:

 On Sun, Oct 13, 2019 at 12:55 AM Ola Fosheim Grøstad via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:



 That said, if you had a formalization of the threads and what
 states different threads are in then you could get some
 performance benefits by eliding shared overhead when it can be
 proven that no other threads are accessing a shared variable.


 What does that even mean? You're just making things up.


 (Drop the ad hominems... Especially when your in an area that is
 not your field.)


Threading and synchronisation is exactly my field.

And it's not at hominem, I really do think you're just making stuff up
(although I'm sure informed by ideas in competing ecosystems)... I'm
sure you mean well and have some great idea, but you haven't presented
anything that I can think on.
The closest thing you've said to a real implementation proposal so far
is strong-ownership based message passing like rust, but that's not
really in contest with this DIP, and I can't imagine how `shared`
would have any influence on that toolbox.


 Thread's don't have 'states', that's literally the point of
 threads!


 All program execution is in a state at any given point in time.

 So yes, you transition between states. That is the premise for
 building an advanced type system. That's what makes it possible
 to implement Rusts borrow checker.


That's all thread-local concepts. There's nothing reasonable you can
determine between threads without synchronising the threads.
D is flirting with escape analysis, we have `scope` and `return` and
things like that, and those are the things that will eventually allow
us to do in-language transitions to/from shared without unsafe casts.


 Anything that moves in a discrete fashion will move between
 states, that is a basic computer science conceptualization of
 computing. It is the foundation that theories about computing is
 built on.


The only reasonable CS model of threading I've ever seen requires you
to start talking about strong ownership and transfering ownership
between threads.
While I agree that strong ownership and message passing is robust and
worth having in any async ecosystem, it's also tremendously
inefficient for certain workloads, and if that were all we had baked
into the language, it excludes other (useful) possibilities.


 Maybe there's a 10 year research project there, but that idea
 is so
 ridiculous and unlikely to work that nobody would ever try.


 Oh. People do it. Sure, it is hard in the most general case, but
 you do have languages with high level concurrency constructs that
 provide type systems that makes it possible to write provably
 correct concurrent programs.


That's not so much language at that point, it becomes blessed library,
or a 'framework' even.
Point me to some examples which are more than in-language message passing?
I don't think we're excluding those sorts of solutions from D.


 It doesn't make much sense to claim that nobody would ever try to
 build something that actually exists... Does it?


The way you initially described the premise was something along the
lines of "the language knows the *state* of interacting threads and
does magic thread-safe stuff", which I'm skeptical exists anywhere
without a very elaborate framework to manage any such orchestration.
I think now that you might be talking about ownership and message
passing... but it's still not clear. What are you actually talking
about? Do you have references?

Incidentally, my code that I linked to back up this thread is
effectively an implementation of a rust-like shared object where it
maintains a 'borrow' ref count, and guards access safely... it's easy
to write those tools in D.


 I think what you're talking about is something more like a
 framework;


 No. Languages/compilers that check concurrency states at compile
 time.


Show me. That is exactly the opposite of what threads are for :/
I can't imagine it, and if a _language_ did that, it almost certainly
comes at a high cost, and it's almost certainly not what I want it to
do... but I'm willing to be surprised.


 But that is not on the table...


 Definitely not. Not at the language level at least.


 That's right. So those benefits are not avilable... which was my
 point. :-)


I can't predict what you're talking about, and you haven't shown
anything substantial. I suspect if you show what you mean, I may find
the term 'benefits' to be problematic ;)
Oct 14 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 14 October 2019 at 18:04:28 UTC, Manu wrote:

 And it's not at hominem, I really do think you're just making 
 stuff up
 (although I'm sure informed by ideas in competing ecosystems)...


If you go after the person and not the statement then that is "ad 
hominem", isn't it?  Claiming that people are making stuff up 
does not compel me to expand on the topic, but I'll try to give 
you an idea of the landscape as I see it, anyway.

I'm not sure what you mean by «competing eco systems», but I am 
talking about theoretical and applied research. And it is not a 
field I am highly interested in, but I have a passing interest in 
it. It makes a lot of sense to use such approaches if you are 
designing communication systems or critical control systems. It 
might be wasted on games.

Anyway, there are various ways of reasoning formally about 
concurrency, and it has been done for a long time... And there 
are many approaches.  Ranging from dedicated tools for 
programming in various niches where it is beneficial, to formal 
methods and tools for analyzing distributed protocols. It is a 
big field, and I think the latter theme is easier to get into, 
and you'll find plenty on this if you search on keywords like: 
formal methods, formally verified, distributed computing, 
distributed protocols...


 The closest thing you've said to a real implementation proposal 
 so far
 is strong-ownership based message passing like rust, but that's 
 not
 really in contest with this DIP, and I can't imagine how 
 `shared`
 would have any influence on that toolbox.


My argument was that D would not have it, so it would not be 
possible to make significant optimization based on modelling 
concurrency. Therefore "shared" as a library type ought to be 
sufficient.  That was my line of argument.

I do not think D should model concurrency formally.

I belive such languages should be created from a clean slate. 
Though, some appears to do stuff with annotated C, but it does 
not sound very compelling to just slap it onto an existing 
language.



 That's all thread-local concepts. There's nothing reasonable 
 you can
 determine between threads without synchronising the threads.


You might need synchronization, but you can potentially have less 
of it if you have a proof that thread 1 is always in state B when 
thread 2 is in state A, or that thread 1 is never in state B if 
thread 2 is not in state D, etc.

It is not uncommon to modell processes as statemachines and 
analyze them formally. When you have that model, you can 
implement it at a lower level if you want, and make a 
correspondence proof that tie each chunk of low level code to the 
high level model. Then there are languages where everything is 
handled for you in a "regular" programming language, but they 
will have a particular focus, with a particular set of trade-offs.



 The only reasonable CS model of threading I've ever seen 
 requires you
 to start talking about strong ownership and transfering 
 ownership
 between threads.


There are many ways to model and reason about concurrency. One 
approach was actually inspired by the modelling of chemical 
reactions using PetriNets.

One "impractical", but interesting, low level 
instruction-by-instruction approach uses Hoare logic. While 
possible, you might be limited to very few instructions. Anyway, 
with this approach you might in theory be able to avoid explicit 
synchronization in some cases as you can then prove whether or 
not all possible interleaving of instructions during execution 
comply with the specification! Still, the combinatorial explosion 
makes this very limiting and hard to reason about.  Anyway, Hoare 
logic is a very low level approach that has been researched in 
the 70s or 80s, so as you can see there is a history of various 
approaches.  Maybe applicable to a tight low level scope like a 
simple high performance hardware driver.

Many problems are only computationally prohibitively expensive if 
you want the optimal solution always, so researchers sometimes 
find ways to solve problems that appears to be impossible due to 
computational complexity, but with the caveat that you sometimes 
get the result "I don't know".  In optimization, "I don't know" 
is good enough, because then you just  fall back on the 
conservative implementation. A solver that is insufficient for 
unguided verification might be good enough for optimization. If 
"impossible" problems can be dealt with "sometimes", then you 
will get optimizations "sometimes".

Anyway, applied solutions seem to focus more on high level 
representations and high level mechanisms, but I would not be 
surprised if you find applied solutions working on low level 
representations using low level mechanisms.



 While I agree that strong ownership and message passing is 
 robust and
 worth having in any async ecosystem, it's also tremendously
 inefficient for certain workloads, and if that were all we had 
 baked
 into the language, it excludes other (useful) possibilities.


Ok, but I didn't argue for it being built into the language...

I am arguing that if it is built into the language then there 
should be a semantic reason for doing so. And static analysis of 
locking patterns might be one such reason.

But in general I think D would be better off by providing generic 
mechanisms rather than adding more and more limited features.

So maybe not add borrowing, but perhaps add some sort of limited 
linear/affine typing mechanism than can be used for implementing 
libraries that provides it to some extent. So people can write 
libraries that statically check that files that are opened also 
are closed. Or that you follow specific patterns when setting up 
hardware (register protocols or something like that).

If meta-programming and library-authoring is D2's main selling 
point then that ought to be front and center.


 Oh. People do it. Sure, it is hard in the most general case, 
 but you do have languages with high level concurrency 
 constructs that provide type systems that makes it possible to 
 write provably correct concurrent programs.


 That's not so much language at that point, it becomes blessed 
 library,
 or a 'framework' even.


No, you cannot run a solver from a library. Well, I guess you 
could, but that is not how it is usually done.


 The way you initially described the premise was something along 
 the lines of "the language knows the *state* of interacting 
 threads and does magic thread-safe stuff", which I'm skeptical 
 exists anywhere without a very elaborate framework to manage 
 any such orchestration.


Not sure what you mean by elaborate framework. You need a 
language for expressing it. Then you need a solver for solving 
the constraints (analyzing/checking the model).



 I think now that you might be talking about ownership and 
 message passing...


It could be, but I am talking about formal verification of 
concurrency models. Or languages that are designed in such a way 
that they translates into such models that can be verified.

What you are talking about might be the other direction where you 
have languages that are designed in such a way that some of the 
issues provably cannot arise (e.g. Pony, maybe Go).

But that is not what I am referring to. I am referring to 
inference using a solver.



 Incidentally, my code that I linked to back up this thread is 
 effectively an implementation of a rust-like shared object 
 where it maintains a 'borrow' ref count, and guards access 
 safely... it's easy to write those tools in D.


Alright, but if you had access to more advanced typing machinery 
in D then you could go further down that road yourself. Maybe not 
to the extent that it could keep up with Rust, but closer.



 if a _language_ did that, it almost certainly
 comes at a high cost, and it's almost certainly not what I want 
 it to
 do... but I'm willing to be surprised.


I don't think you want to do it. You'd have to change how you 
approach it, and it is not really needed for more mundane tasks.



 That's right. So those benefits are not avilable... which was 
 my point. :-)


 I can't predict what you're talking about, and you haven't 
 shown anything substantial. I suspect if you show what you 
 mean, I may find the term 'benefits' to be problematic ;)


What do you mean by substantial?

The benefits is that you prove that the code follows the model, 
which in turn can be fed into the optimizer and assumed to always 
hold.

It is not realistic that D will ever go there.

It is more realistic that D would expand the type system or 
meta-programming mechanisms in ways that are useful to library 
authors and that they could create concurrent frameworks.
Oct 14 2019
parent reply Manu <turkeyman gmail.com> writes:
On Mon, Oct 14, 2019 at 2:50 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 14 October 2019 at 18:04:28 UTC, Manu wrote:

 And it's not at hominem, I really do think you're just making
 stuff up
 (although I'm sure informed by ideas in competing ecosystems)...


 If you go after the person and not the statement then that is "ad
 hominem", isn't it?  Claiming that people are making stuff up
 does not compel me to expand on the topic, but I'll try to give
 you an idea of the landscape as I see it, anyway.

 I'm not sure what you mean by «competing eco systems», but I am
 talking about theoretical and applied research. And it is not a
 field I am highly interested in, but I have a passing interest in
 it. It makes a lot of sense to use such approaches if you are
 designing communication systems or critical control systems. It
 might be wasted on games.

 Anyway, there are various ways of reasoning formally about
 concurrency, and it has been done for a long time... And there
 are many approaches.  Ranging from dedicated tools for
 programming in various niches where it is beneficial, to formal
 methods and tools for analyzing distributed protocols. It is a
 big field, and I think the latter theme is easier to get into,
 and you'll find plenty on this if you search on keywords like:
 formal methods, formally verified, distributed computing,
 distributed protocols...


I'm aware of a lot of work in this area, but I haven't seem how it
influences a language at the lowest level (beyond ownership based
solutions, where transfer of unique ownership is used to model
thread-safe data transitions); it's usually in the realm of what I
would call 'framework', although there are many examples of such being
blessed by the language.
I don't think D has a great history with blessing an implementation as
sacred and installing it into the syntax, and D tends to be expressive
enough to make library solutions for this sort of thing quite
satisfying.


 The closest thing you've said to a real implementation proposal
 so far
 is strong-ownership based message passing like rust, but that's
 not
 really in contest with this DIP, and I can't imagine how
 `shared`
 would have any influence on that toolbox.


 My argument was that D would not have it, so it would not be
 possible to make significant optimization based on modelling
 concurrency. Therefore "shared" as a library type ought to be
 sufficient.  That was my line of argument.

 I do not think D should model concurrency formally.


Then how do you define thread-local by default, and how do you
mark/identify data that violates that axiom?
D doesn't "model concurrency", that's not what `shared` does, it
simply establishes thread-locality.


 I belive such languages should be created from a clean slate.
 Though, some appears to do stuff with annotated C, but it does
 not sound very compelling to just slap it onto an existing
 language.


Are you talking about D here, or just a hypothetical application of your POV?


 That's all thread-local concepts. There's nothing reasonable
 you can
 determine between threads without synchronising the threads.


 You might need synchronization, but you can potentially have less
 of it if you have a proof that thread 1 is always in state B when
 thread 2 is in state A, or that thread 1 is never in state B if
 thread 2 is not in state D, etc.


This is well into 'framework' territory; a thread is associated with
static knowledge of a state machine, known threads are collected into
a registry, and then state transitions can assert validity against the
other threads in the pool.
This sounds super complex, hard to express, and not worth of language
syntax. It also has nothing to do with `shared`.


 It is not uncommon to modell processes as statemachines and
 analyze them formally.


Sure, but we don't do that, or anything that's even remotely in that
ballpark. It's not a functional language, and we don't have *code* or
data access introspection like that.


 When you have that model, you can
 implement it at a lower level if you want, and make a
 correspondence proof that tie each chunk of low level code to the
 high level model. Then there are languages where everything is
 handled for you in a "regular" programming language, but they
 will have a particular focus, with a particular set of trade-offs.


That sort of language goes all in on one very restrictive programming
model. It's not anything like D, and I don't think it's a
consideration when talking about `shared`.
If you want to resist `shared`'s existence, then I don't think ideas
like this will move us forward.


 The only reasonable CS model of threading I've ever seen
 requires you
 to start talking about strong ownership and transfering
 ownership
 between threads.


 There are many ways to model and reason about concurrency. One
 approach was actually inspired by the modelling of chemical
 reactions using PetriNets.

 One "impractical", but interesting, low level
 instruction-by-instruction approach uses Hoare logic. While
 possible, you might be limited to very few instructions. Anyway,
 with this approach you might in theory be able to avoid explicit
 synchronization in some cases as you can then prove whether or
 not all possible interleaving of instructions during execution
 comply with the specification! Still, the combinatorial explosion
 makes this very limiting and hard to reason about.  Anyway, Hoare
 logic is a very low level approach that has been researched in
 the 70s or 80s, so as you can see there is a history of various
 approaches.  Maybe applicable to a tight low level scope like a
 simple high performance hardware driver.


I've read about ideas like this before, but like, it's strictly
fantasy. I'm interested in what's real, where we are, what we have,
and where we need to be in practical terms.


 While I agree that strong ownership and message passing is
 robust and
 worth having in any async ecosystem, it's also tremendously
 inefficient for certain workloads, and if that were all we had
 baked
 into the language, it excludes other (useful) possibilities.


 Ok, but I didn't argue for it being built into the language...


There's gotta be something at the bottom of the stack, or you can't
write anything above.


 I am arguing that if it is built into the language then there
 should be a semantic reason for doing so. And static analysis of
 locking patterns might be one such reason.


There is; D specifies thread-local by default, and that absolutely
requires a marker (and associated semantics) to attribute the data
that violates that principle.

This has nothing to do with locking, that's a much higher-level
problem and completely outside the language.
Locking is this:
  * I have this shared data, I want to access it...
  * I must create a context where I have an exclusive window on the data.
  * One such implementation involves a mutex of some form, where I
assert that I am the unique lease-holder, and then in effect the data
becomes thread-local for the duration of that lease (ie, cast shared
away, because the data is thread-local for that duration).

In D, we can implement semantics like this by scope-guarded lock tools
of various kinds. It is also possible to achieve thread-locality by
deferring to a scheduling infrastructure with knowledge of
access-control. Such frameworks will cast to thread-local when issuing
access controlled data to the user.
We should make use of D's recent escape analysis tricks to guarantee
that thread-local leases remain properly scoped, and mutually
exclusive with other leases.


 But in general I think D would be better off by providing generic
 mechanisms rather than adding more and more limited features.


What does this mean to you? We can write very rich libraries with
`shared` as a safety mechanism. Nothing more is needed from the
language, except perhaps some future implicit cast rules.


 So maybe not add borrowing, but perhaps add some sort of limited
 linear/affine typing mechanism than can be used for implementing
 libraries that provides it to some extent. So people can write
 libraries that statically check that files that are opened also
 are closed. Or that you follow specific patterns when setting up
 hardware (register protocols or something like that).


What does this have to do with `shared`, and why is the DIP at odds with this?


 If meta-programming and library-authoring is D2's main selling
 point then that ought to be front and center.


I agree. I wrote that borrow-inspired guarded access type which I lank above.


 Oh. People do it. Sure, it is hard in the most general case,
 but you do have languages with high level concurrency
 constructs that provide type systems that makes it possible to
 write provably correct concurrent programs.


 That's not so much language at that point, it becomes blessed
 library,
 or a 'framework' even.


 No, you cannot run a solver from a library. Well, I guess you
 could, but that is not how it is usually done.


If this relates to the ideas you mentioned at the top; it's dependent
on threads with rich statically-associated data.
D could do this sort of thing in library; we have user-defined
attributes, and heavy-weight thread definition objects could carry
that around and be consumed by solvers.

In some ways it's a bit like Andrei's Big-O library, which carries
metadata around with definitions and used for static analysis.


 The way you initially described the premise was something along
 the lines of "the language knows the *state* of interacting
 threads and does magic thread-safe stuff", which I'm skeptical
 exists anywhere without a very elaborate framework to manage
 any such orchestration.


 Not sure what you mean by elaborate framework. You need a
 language for expressing it. Then you need a solver for solving
 the constraints (analyzing/checking the model).


This is exactly what I would call a framework library. An elaborated
and opinionated implementation of a particular model.


 I think now that you might be talking about ownership and
 message passing...


 It could be, but I am talking about formal verification of
 concurrency models. Or languages that are designed in such a way
 that they translates into such models that can be verified.

 What you are talking about might be the other direction where you
 have languages that are designed in such a way that some of the
 issues provably cannot arise (e.g. Pony, maybe Go).

 But that is not what I am referring to. I am referring to
 inference using a solver.


Your ideas rely on intensive attribution or extremely sophisticated
introspection. We can easily do such attribution in D, but not
introspection of code, and where to be practical, like you say, the
low-level set of operations would need to be radically simplified into
a small and reasonable set.
You're not talking about D at that point.


 Incidentally, my code that I linked to back up this thread is
 effectively an implementation of a rust-like shared object
 where it maintains a 'borrow' ref count, and guards access
 safely... it's easy to write those tools in D.


 Alright, but if you had access to more advanced typing machinery
 in D then you could go further down that road yourself. Maybe not
 to the extent that it could keep up with Rust, but closer.


I'm not sure what rust offers to my little lib up there that I can't
do; I mean, rust basically does exactly that, in basically the same
way.
The only advantage of rust is that it has some language to track
lifetime knowledge through function calls which can be used for escape
analysis of the scoped lease object.


 if a _language_ did that, it almost certainly
 comes at a high cost, and it's almost certainly not what I want
 it to
 do... but I'm willing to be surprised.


 I don't think you want to do it. You'd have to change how you
 approach it, and it is not really needed for more mundane tasks.


Is it needed for *any* task? I can't think of a more complex
interactive environment in software than simulation/video games.
Nowhere else are so many completely different distinct solutions all
interacting in an asynchronous (and realtime) environment.
Maybe you want a proof-able solution that guarantees progress, and
willing to sacrifice efficiency for it... but in that world, why do
you need threads at all? If such an environment exists, I think it's
very niche.


 That's right. So those benefits are not avilable... which was
 my point. :-)


 I can't predict what you're talking about, and you haven't
 shown anything substantial. I suspect if you show what you
 mean, I may find the term 'benefits' to be problematic ;)


 What do you mean by substantial?


By 'substantial' I mean 'at all'. You didn't reference examples or
specifics, or describe your vision until now.


 The benefits is that you prove that the code follows the model,
 which in turn can be fed into the optimizer and assumed to always
 hold.

 It is not realistic that D will ever go there.

 It is more realistic that D would expand the type system or
 meta-programming mechanisms in ways that are useful to library
 authors and that they could create concurrent frameworks.


Right. That's exactly what we're doing. I'm glad you agree :)
Oct 14 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 15 October 2019 at 04:09:52 UTC, Manu wrote:

 On Mon, Oct 14, 2019 at 2:50 PM Ola Fosheim Grøstad via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 I'm aware of a lot of work in this area, but I haven't seem how 
 it
 influences a language at the lowest level (beyond ownership 
 based
 solutions, where transfer of unique ownership is used to model
 thread-safe data transitions);


Well, that's because the research topic is «verification». So 
that is where the culture is, but most programmers have no 
interest in verification, because then they would have to think 
about what they are doing twice... They probably should!! But 
that takes time and we have a global programming culture of 
«fudging» things together :-).  Doesn't mean the ideas cannot be 
leveraged for more mundane purposes with more limited scope than 
verification though.

Rust is the outlier, and as such is quite impressive in the sense 
that they have managed to established a culture where programmers 
accept having to do extra (unnecessary) work to make their 
programs run in exchange for higher confidence of correctness. 
But as you can see in any discussion about Rust vs C++, a large 
number of programmers reject that and hold it against Rust. So 
establishing such a culture is not an everyday phenomenon.

You seemed to suggest that I use Rust, but I personally have no 
need for Rust, as I currently don't have issues that Rust can 
solve.  I might pick it up for mundane reasons like tight webasm 
code generation.

What I find interesting about Rust is the culture that builds up 
around the language semantics. Which certainly is a shift from 
programming cultures that preceded it as far as I can tell.



 it's usually in the realm of  what I
 would call 'framework', although there are many examples of 
 such being blessed by the language.


Ok, so basically anything that is unfamiliar in the D context is 
a «framework», including type systems. Fair enough. («framework» 
is usually used for denoting a set of library types and possibly 
runtime with tight coupling that aims to provide a foundation for 
writing applications in a limited category.)



 Then how do you define thread-local by default, and how do you
 mark/identify data that violates that axiom?


The only reason you need to mark TLS is because it is referenced 
using a different instruction sequence.

So to repeat myself:

Other than that, D does not provide anything beyond 
atomics/barriers. Does it? Everything else is purely syntactical. 
Slam a  -tag around your code and you get to reinterpret and 
thwart the type system. That's a syntactical approach in my mind. 
Not a sound type system approach. Therefore it offers nothing 
over a library type.


 static knowledge of a state machine, known threads are 
 collected into
 a registry, and then state transitions can assert validity 
 against the
 other threads in the pool.


You could do it dynamically, but also statically. Simple example, 
if you can establish that all Reader threads are spawned after 
the Writer thread has ended then you can ellide all the locking 
in the Reader threads. (without the programmer knowing about it).



 that ballpark. It's not a functional language, and we don't 
 have *code* or data access introspection like that.


Doesn't have to be FPL, people do interesting things on the IR 
level, but I am not saying D should, the manpower and knowhow 
isn't here.



 That sort of language goes all in on one very restrictive 
 programming model.


It might, but doesn't have to, programmers can restrict 
themselves to satisfy the verifier. It appears to work for Rust.



 If you want to resist `shared`'s existence, then I don't think 
 ideas like this will move us forward.


I am not resisting anything.

I am trying to figure out why you cannot do «shared» equally well 
as a library type and what is missing in the language that 
prevents that from being possible.

Because if that is not possible then I sense that there must be 
some glaring flaws in the D meta-programming capabilities that 
ought to be fixed.

For me this is an exercise in figuring out what works and does 
not work in D's meta programming model.



 I've read about ideas like this before, but like, it's strictly 
 fantasy. I'm interested in what's real, where we are, what we 
 have, and where we need to be in practical terms.


Ok, so I presented two ends of the spectrum that has a long 
history, one from 1939/1960s (PetriNets) and one from 1980 
(concurrent Hoare logic).  In between those ends you have many 
different approaches.

You wanted to discuss what is possible/not possible, and these 
two end points does say something about what people have been 
researching and therefore what might/might not be possible.

There are probably future applications for concurrent Hoare logic 
in software synthesis (let the computer find instruction 
sequences that satisfy a specification.). At least that seems 
plausible, so I wouldn't call it fantasy or useless.



 There is; D specifies thread-local by default, and that 
 absolutely requires a marker (and associated semantics) to 
 attribute the data that violates that principle.


That's just a claim. No, you only need lower level constructs 
like barriers.

You should be able to implement the marker in a library, if not 
then someone ought to take a hard look at the meta-programming 
capabilities.



 What does this mean to you? We can write very rich libraries 
 with `shared` as a safety mechanism. Nothing more is needed 
 from the language, except perhaps some future implicit cast 
 rules.


That is also not true. If you are allowed to turn "shared" into 
proper "local"  then you've got a type system that is so weak 
that there is no need for it to be in the language. It is not 
sound.

To get a proper and sound type system that requires language 
support you need something stronger than «D shared». Pony's take 
on this is to use more types to mark the legal transitions. Too 
late for D, so you might as well just figure out WHY the meta 
programming capabilities prevent you from implementing «shared» 
as a library type.

I have not been able to figure that out, because nobody has 
showed me why it cannot be done as a library type.


GOLDEN RULE OF THUMB FOR PROGRAMMING LANGUAGE DESIGN:

Before you implement at feature:

1. Implement it as a library solution.

2. Use it and describe why it does not feel right/works right.

3. Figure out what weaknesses the language has that prevents it 
from being done properly as a library solution.

4. Figure out how that can be improved then go back to 1.

5. Give up, evaluate the situation and start planning a language 
feature.



 It is more realistic that D would expand the type system or 
 meta-programming mechanisms in ways that are useful to library 
 authors and that they could create concurrent frameworks.


 Right. That's exactly what we're doing. I'm glad you agree :)


I agree with myself, but that is NOT exactly what you are 
doing... :) See the «golden rule of thumb» above.

Before adding a feature, provide an in-depth analysis for why a 
library solution is insufficient.

I don't see how any skilled language designer could disagree with 
that statement.
Oct 15 2019
prev sibling parent reply IGotD- <nise nise.com> writes:
On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis 
wrote:




 operate on shared data in a manner which isn't guaranteed to be 
 atomic, the compiler prevents you from doing it unless you  use 

 and thus no such compiler errors, the code should still have 

 then it isn't thread-safe.


With this DIP, shared integers/small types will be automatically 
atomic. For complex/large types, will you still be able to use 
them as before between threads and you have protect the type 
yourself at least for a transitional period?

"Atomic" here as I get it also mean atomically updating complex 
types. This usually means that you need to guard the operations 
with some kind of mutex. The compiler can of course detect this 
and issue a warning/error to the user which doesn't seem to be 
the scope of this DIP.

Correct me if I'm wrong but we have the following scenarios.
1. shared integer/simple type (size dependent?) -> automatically 
HW atomic operations
2. shared complex type -> write to any member must be protected 
with a mutex.
3. shared complex type -> read to any member must be protected 
with a mutex or read/write mutex allowing multiple reads.

The compiler is used the detect these scenarios so that the user 
doesn't forget protecting the shared types.

As I get it this DIP is just a baby step towards this bigger 
scope for the shared keyword.
Oct 12 2019
next sibling parent Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 4:40 PM IGotD- via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis
 wrote:




 operate on shared data in a manner which isn't guaranteed to be
 atomic, the compiler prevents you from doing it unless you  use

 and thus no such compiler errors, the code should still have

 then it isn't thread-safe.


 With this DIP, shared integers/small types will be automatically
 atomic.


That's not what this DIP says, as had been clarified by Walter a
number of times now.


 For complex/large types, will you still be able to use
 them as before between threads and you have protect the type
 yourself at least for a transitional period?

 "Atomic" here as I get it also mean atomically updating complex
 types. This usually means that you need to guard the operations
 with some kind of mutex. The compiler can of course detect this
 and issue a warning/error to the user which doesn't seem to be
 the scope of this DIP.

 Correct me if I'm wrong but we have the following scenarios.
 1. shared integer/simple type (size dependent?) -> automatically
 HW atomic operations


This is the misunderstanding through most of this thread.


 2. shared complex type -> write to any member must be protected
 with a mutex.
 3. shared complex type -> read to any member must be protected
 with a mutex or read/write mutex allowing multiple reads.

 The compiler is used the detect these scenarios so that the user
 doesn't forget protecting the shared types.

 As I get it this DIP is just a baby step towards this bigger
 scope for the shared keyword.


Useful libraries will follow.
Oct 12 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 5:37:50 PM MDT IGotD- via Digitalmars-d wrote:

 On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis

 wrote:




 operate on shared data in a manner which isn't guaranteed to be
 atomic, the compiler prevents you from doing it unless you  use

 and thus no such compiler errors, the code should still have

 then it isn't thread-safe.


 With this DIP, shared integers/small types will be automatically
 atomic. For complex/large types, will you still be able to use
 them as before between threads and you have protect the type
 yourself at least for a transitional period?

 "Atomic" here as I get it also mean atomically updating complex
 types. This usually means that you need to guard the operations
 with some kind of mutex. The compiler can of course detect this
 and issue a warning/error to the user which doesn't seem to be
 the scope of this DIP.

 Correct me if I'm wrong but we have the following scenarios.
 1. shared integer/simple type (size dependent?) -> automatically
 HW atomic operations
 2. shared complex type -> write to any member must be protected
 with a mutex.
 3. shared complex type -> read to any member must be protected
 with a mutex or read/write mutex allowing multiple reads.

 The compiler is used the detect these scenarios so that the user
 doesn't forget protecting the shared types.

 As I get it this DIP is just a baby step towards this bigger
 scope for the shared keyword.


When we're talking about atomic with regards to shared, we're talking about
what core.atomic does. They're operations that are atomic with regards to
CPU instructions. The most that that can work with is primitive types like
integers or pointers. More complex types require stuff like mutexes to
protect them in order to freely mutate them.

Walter needs to make the DIP clearer, but in the discussions in this thread,
he's made it clear that the intention is that read/write operations will
become illegal for all shared data, forcing code to either use core.atomic
to do atomic operations on the data or to use synchronization mechanisms
such as mutexes to allow for thread-safe reading and writing of data, with
the code needing to cast away shared in order to operate on the data
(meaning that the code will then be  system or  trusted).

In principle, the compiler could allow reading and writing shared variables
by inserting the core.atomic stuff for you, but that's not the current plan.
Either way, it's very difficult to have the compiler understand
synchronization primitives well enough and have enough guarantees about what
the code is doing to be able to do something like implicitly remove shared
for you. So, it's highly unlikely that we'll ever get much in the language
that would be able to implicitly remove shared for even simple pieces of
code let alone complex types. AFAIK, only construct along those lines that's
been proposed thus far that could work is TDPL's synchronized classes, and
they can only implicitly remove a single layer of shared - and that can only
do that much because of how restrictive they are. Having the compiler
magically handle thread synchronization primitives for you is likely a pipe
dream.

Rather, what's likely going to tend to happen is that complex objects that
are supposed to be used as shared will handle the appropriate atomics or
mutexes internally, providing an  safe API for the user. However, the
internals will still have to do the dirty stuff and be appropriately vetted
for thread-safety. That's already what you typically get in a language like
C++. It's just that the type system doesn't have shared, so you have to
manage everything yourself and don't have a way to segregate shared data and
the functions operating on it other than by convention, whereas D's shared
enforces it via the type system.

Ultimately, shared is about segregating the code that operates on shared
data - both so that most code can just be thread-local without worrying
about it and so that you can easily determine which pieces of the program
have to be examined for threading issues - just like  system/ trusted/ safe
isolates the portions of the program where you potentially have to worry
about memory safety. shared isn't about magically handling threading stuff
for you. If we can figure out how to add mechanisms on top of shared which
make things easier, then great, but shared has to be properly locked down
first, and given that D is a systems language (thus allowing all kinds of
crazy stuff), and its type system really has no concept of ownership, I
don't think that it's very likely that we're going to be able to add much to
the language that's going to allow shared to be implicitly removed or allow
you to otherwise operate on shared data without worrying about dealing with
the synchronization primitives yourself.

- Jonathan M Davis
Oct 12 2019
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 4:30 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis
 wrote:

 It's the same as with any  trusted code. It's up to the
 programmer to ensure that what the code is doing is actually
  safe, and if the code isn't able to provide an API that is
  safe, then it shouldn't be  trusted.


 So, I am trying to figure out what that API would have to look
 like and the only solution I can think of right away is that you
 have are doing a callback back into  safe code from the
 lock-holding  trusted code?

 So, if you have to obtain locks from multiple sources, it either
 won't work out, or you have to use some serious kludges, like a
 deep call chain:

 while(tryagain) {
      trusted_code_obtains_lock_1-->
         safe_code_callback1 calls->
             trusted_code_obtains_lock2
                 safe_code_callback2 processes and sets
 tryagain=false
 }

 Of course, that could lead to starvation.

 So to avoid all of that multi-threaded libraries have to have 2
 APIs, one simple transactional wrapper-API for  safe code and
 another API for  trusted code that can be used to write a new
 wrapper that allows locking of multiple datastructures.

 I hope people try to sketch out what such libraries will look
 like before the semantics are decided on. Maybe an experimental
 branch would be appropriate.


Here's a rough draft of one such sort of tool I use all the time in
shared-intensive code:
https://gist.github.com/TurkeyMan/c16db7a0be312e9a0a2083f5f4a6efec

I just hacked that together the other night to do some experiments. I
have a toolbox like that which we use at work.
This code here feels fine to me in D, but it all depends on
restrictive shared to work.
There's nothing useful you can do with shared safely as it exists in
the language today.
Oct 12 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 23:36:55 UTC, Manu wrote:

 Here's a rough draft of one such sort of tool I use all the 
 time in shared-intensive code: 
 https://gist.github.com/TurkeyMan/c16db7a0be312e9a0a2083f5f4a6efec


Thanks! That looks quite low level, but now I understand more 
what you are looking for.

What I had in mind was writing APIs that allows ordinary 
programmers to do parallell programming safely.

Like taking the single threaded code they have written for 
processing a single array or merging two arrays with each other 
and then use a library for speeding it up.


Anyway, my core argument is to put more meta-programming power in 
hands of library authors like you so that people who have the 
shoes on can define the semantics they are after. I really don't 
think this has to be done at the compiler level, given the right 
meta programming tools, based on the proposed semantics. And I 
don't think providing those meta programming tools are more work 
than hardwiring more stuff into the compiler. Clearly that is 
just my opinion. Others might feel differently.

(Other semantics do require compiler support to work well, like 
heterogeneous memory architectures, but the proposal does not 
work with that.)
Oct 13 2019
parent reply Manu <turkeyman gmail.com> writes:
On Sun, Oct 13, 2019 at 1:10 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Saturday, 12 October 2019 at 23:36:55 UTC, Manu wrote:

 Here's a rough draft of one such sort of tool I use all the
 time in shared-intensive code:
 https://gist.github.com/TurkeyMan/c16db7a0be312e9a0a2083f5f4a6efec


 Thanks! That looks quite low level, but now I understand more
 what you are looking for.


Well the implementation is low-level, obviously. But the tool is not.
It's quite approachable at user level, the only risk being to avoid
deadlock. If you want to systemically avoid dead-locks, then
higher-level tools appear.


 What I had in mind was writing APIs that allows ordinary
 programmers to do parallell programming safely.


That doesn't exist. I would call that a 'framework', and that's a very
high level library suite.
What we're doing here is making it possible to write such a thing safely.

'Ordinary' programmers don't do concurrency on a low level. Period.


 Like taking the single threaded code they have written for
 processing a single array or merging two arrays with each other
 and then use a library for speeding it up.


I'm working towards a safe parallel-for... this needs to land first
before I can attempt to argue for the next steps to make that work.


 Anyway, my core argument is to put more meta-programming power in
 hands of library authors like you so that people who have the
 shoes on can define the semantics they are after. I really don't
 think this has to be done at the compiler level, given the right
 meta programming tools, based on the proposed semantics. And I
 don't think providing those meta programming tools are more work
 than hardwiring more stuff into the compiler. Clearly that is
 just my opinion. Others might feel differently.


I've said this a few times now, and I'll say it again.
Your insistence that shared should be a library thing applies equally
to const. It behaves in an identical manner to const, and if you can't
make a compelling case for moving const to a library, you can't
possibly sell me on that idea with respect to shared.

1. `shared` defines D's thread-local by default semantics... without
shared, there is no thread-local by default. That decision is so deep,
there is absolutely no way to move away from that.
2. We must be able to overload on `shared`. We can't 'overload' on
template wrappers; we can try and specialise and stuff and abuse
clever inference and type deduction, but things break down quickly. I
have tried to simulate type qualifiers with template wrappers before.
It always goes *very* badly, and it's a mess to work with.
3. Most importantly, there is a future (not shown in this discussion)
where shared should allow some implicit promotions safely; for
instance:

void fun(scope ref shared T arg);
T x;
fun(x); // OK; promote to shared, strictly bound to the lifetime of
this call tree

This will allow some of the simple deployment of algorithms you refer to above.
Interaction with scope and other escape analysis tools is very
low-level and quite complex, and we need shared to have a solid
definition before we can experiment with that stuff.
This is the key that will unlock parallel-for and other algorithms.
Oct 13 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 19:23:45 UTC, Manu wrote:

 Your insistence that shared should be a library thing applies 
 equally
 to const. It behaves in an identical manner to const, and if 
 you can't
 make a compelling case for moving const to a library, you can't
 possibly sell me on that idea with respect to shared.


I'm not trying to sell anything, but you cannot easily implement 
D's const using metaprogramming as metaprogramming is realized in 
D.

You can easily extend the metaprogramming capabilities of D to 
implement shared.

Apples and Oranges.



 1. `shared` defines D's thread-local by default semantics... 
 without
 shared, there is no thread-local by default. That decision is 
 so deep,
 there is absolutely no way to move away from that.


That argument makes no sense to me. Thread local or not thread 
local is essentially only by social agreement if you have 
syntactical means to get around it.  It is predominantly a social 
contract.

It is a matter of agreement, interpretation, conceptualization, 
mindset... That is what defines what is thread local in D. Not 
machinery.

That is not how it is in a type system that is stronger like in 
Pony. Where it is strictly enforced and clearly defined fully 
semantically in the machinery with no means to break the type 
system.

In that regard, D is not much different from C++, it is 
predominantly social and syntactical. Except in D you might have 
stronger agreement, stronger social contract, more visible 
syntactical symbols. So it appears to be stronger.

But as long as D is going to interface tightly with C/C++ it will 
always rely on social contracts.


 2. We must be able to overload on `shared`. We can't 'overload' 
 on
 template wrappers; we can try and specialise and stuff and abuse
 clever inference and type deduction, but things break down 
 quickly. I
 have tried to simulate type qualifiers with template wrappers 
 before.
 It always goes *very* badly, and it's a mess to work with.


You have two alternatives:

1.  you can do a simple one with template wrappers.

2. you can extend the meta programming capabilities and have a 
more advanced  solution that is more in line with the current 
syntax

But it certainly is possible, and not very hard to envision. 
After all making something completely unavailable, is much easier 
than only making some aspects of an object unavailable. Which is 
why "shared" is relatively easy to do on the syntactical level in 
D, but "const" is harder. If meta-programming in D was more 
deductive then const would be fair game too, but that is not the 
case, so... it becomes less obvious and probably not worth the 
effort.



 3. Most importantly, there is a future (not shown in this 
 discussion)
 where shared should allow some implicit promotions safely; for
 instance:


Ok, not sure if that really impacts whether it can be done in a 
library or not, but it certainly is VERY important to list such 
use-cases with sufficient detail when designing foundational 
mechanisms that is supposed to enable good interfaces to 
parallell programming libraries!
Oct 13 2019
parent reply Manu <turkeyman gmail.com> writes:
On Sun, Oct 13, 2019 at 2:10 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Sunday, 13 October 2019 at 19:23:45 UTC, Manu wrote:

 Your insistence that shared should be a library thing applies
 equally
 to const. It behaves in an identical manner to const, and if
 you can't
 make a compelling case for moving const to a library, you can't
 possibly sell me on that idea with respect to shared.


 I'm not trying to sell anything, but you cannot easily implement
 D's const using metaprogramming as metaprogramming is realized in
 D.

 You can easily extend the metaprogramming capabilities of D to
 implement shared.

 Apples and Oranges.


How so?
`shared` is exactly like const, but with an additional restriction
that you can't read either.


 1. `shared` defines D's thread-local by default semantics...
 without
 shared, there is no thread-local by default. That decision is
 so deep,
 there is absolutely no way to move away from that.


 That argument makes no sense to me. Thread local or not thread
 local is essentially only by social agreement if you have
 syntactical means to get around it.  It is predominantly a social
 contract.


"if you have syntactical means to get around it" <- this is `shared`;
if we lose shared, we lose thread-local.


 It is a matter of agreement, interpretation, conceptualization,
 mindset... That is what defines what is thread local in D. Not
 machinery.


Yeah, but if there's no way to describe not-thread-local, then
thread-local doesn't really mean anything.
Assuming there were no shared, and "data is thread-local" is
specified, then you'd be doing UB anytime you spawn a second thread...
then the language would just need to tolerate that, and accept that
so-called thread-local data may or may not actually be thread-local
(and conservatively assume not thread-local), because no way to
distinguish.
I mean, I understand your point of view, but in practical terms, it's
not correct. thread-local by default exists because `shared` exists.


 2. We must be able to overload on `shared`. We can't 'overload'
 on
 template wrappers; we can try and specialise and stuff and abuse
 clever inference and type deduction, but things break down
 quickly. I
 have tried to simulate type qualifiers with template wrappers
 before.
 It always goes *very* badly, and it's a mess to work with.


 You have two alternatives:

 1.  you can do a simple one with template wrappers.


It goes very badly every time I've tried this.


 2. you can extend the meta programming capabilities and have a
 more advanced  solution that is more in line with the current
 syntax


That's a huge development, and much uncertainty.
And it's not like we're introducing shared here... he's been in D
forever, we're just making it mean something.


 But it certainly is possible, and not very hard to envision.
 After all making something completely unavailable, is much easier
 than only making some aspects of an object unavailable. Which is
 why "shared" is relatively easy to do on the syntactical level in
 D, but "const" is harder.


I don't understand... I think they're almost identical hard-ness.


 3. Most importantly, there is a future (not shown in this
 discussion)
 where shared should allow some implicit promotions safely; for
 instance:


 Ok, not sure if that really impacts whether it can be done in a
 library or not, but it certainly is VERY important to list such
 use-cases with sufficient detail when designing foundational
 mechanisms that is supposed to enable good interfaces to
 parallell programming libraries!


We're not designing foundational mechanisms, Walter designed shared
decades ago and it's firmly embedded in the language. We're just
making it not completely broken right now.
Designing a better shared is future work, which needs to start from
stable ground.
Oct 13 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 21:44:45 UTC, Manu wrote:

 On Sun, Oct 13, 2019 at 2:10 PM Ola Fosheim Grøstad via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 How so?
 `shared` is exactly like const, but with an additional 
 restriction
 that you can't read either.


Shared provides only identity of the top level.
Const provides recursively methods for reading.


 "if you have syntactical means to get around it" <- this is 
 `shared`; if we lose shared, we lose thread-local.


Nah, C++ memory was local... Threads came through a 
library/syscall. Didn't loose or win the concept of local 
memory... It is semiotic if not strictly enforced, a matter of 
interpretation.
A matter of intersubjectivity, e.g people sharing the same 
interpretation.


 Assuming there were no shared, and "data is thread-local" is
 specified, then you'd be doing UB anytime you spawn a second 
 thread...


No, they could have disjunct stores. Like JavaScript where you 
transfer unique ownership.

Conceptually close to using a mutex... Except in the mutex case 
there is no formal representation, the connection between mutex 
and object is in the head of the programmers.


 not correct. thread-local by default exists because `shared` 
 exists.


Nah, I'd rather say that you can transfer ownership. Shared just 
means that it is up for grabs, it is in transition. Just like in 
JavaScript. At some point the mediating system is in control.

No conflict.



 It goes very badly every time I've tried this.


I'll have to take your word for it... Something serious?


 2. you can extend the meta programming capabilities and have a 
 more advanced  solution that is more in line with the current 
 syntax


 That's a huge development, and much uncertainty.
 And it's not like we're introducing shared here... he's been in 
 D
 forever, we're just making it mean something.


Yes, there might be a cultural barrier to changing the 
syntax/expected features.



 I don't understand... I think they're almost identical 
 hard-ness.


You can a simple shared wrapper in 10 lines, so I don't 
understand the "identical" part...



 We're not designing foundational mechanisms, Walter designed 
 shared
 decades ago and it's firmly embedded in the language.


Designed? If it was, there would be no DIP.
Oct 13 2019
parent Manu <turkeyman gmail.com> writes:
On Sun, Oct 13, 2019 at 4:30 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Sunday, 13 October 2019 at 21:44:45 UTC, Manu wrote:

 On Sun, Oct 13, 2019 at 2:10 PM Ola Fosheim Grøstad via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 How so?
 `shared` is exactly like const, but with an additional
 restriction
 that you can't read either.


 Shared provides only identity of the top level.
 Const provides recursively methods for reading.


 "if you have syntactical means to get around it" <- this is
 `shared`; if we lose shared, we lose thread-local.


 Nah, C++ memory was local... Threads came through a
 library/syscall. Didn't loose or win the concept of local
 memory... It is semiotic if not strictly enforced, a matter of
 interpretation.
 A matter of intersubjectivity, e.g people sharing the same
 interpretation.


 Assuming there were no shared, and "data is thread-local" is
 specified, then you'd be doing UB anytime you spawn a second
 thread...


 No, they could have disjunct stores. Like JavaScript where you
 transfer unique ownership.


They could, but they don't..
You're talking about a lot of hypothetical, I'm talking about how D
is, and how it's always been.


 It goes very badly every time I've tried this.


 I'll have to take your word for it... Something serious?


It just doesn't scale, and relies extremely heavily on template constraints.
You lose; function pointers, virtual functions, binary libs, DLL's,
etc... you also lose auto-complete style features in the editor.
Templates should be the solution to generic code problems, not to
problems like this.


 2. you can extend the meta programming capabilities and have a
 more advanced  solution that is more in line with the current
 syntax


 That's a huge development, and much uncertainty.
 And it's not like we're introducing shared here... he's been in
 D
 forever, we're just making it mean something.


 Yes, there might be a cultural barrier to changing the
 syntax/expected features.


What's the point of that tangent though?
We're not seriously going to consider changing the way const is
defined in D, and make a proposal for user-defines type constructors
in a conversation about making shared work properly.


 We're not designing foundational mechanisms, Walter designed
 shared
 decades ago and it's firmly embedded in the language.


 Designed? If it was, there would be no DIP.


I agree it should have been like this from the start, but it exists,
it's the tool we have, and we need to make it work properly.
Oct 14 2019
prev sibling parent Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 2:20 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Saturday, 12 October 2019 at 06:31:18 UTC, Jonathan M Davis
 wrote:

 On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via
 Digitalmars-d wrote:

 On 10/2/2019 3:42 AM, Nicholas Wilson wrote:





 It should be that shared memory access is disabled in all
 contexts,
 which must be worked around with casts,
 which makes the function  system,
 which must then be encapsulated with a  trusted interface in
 order to
 use in  safe code.


 Sounds right.


 This is pretty much what several of us were arguing for during
 the discussions about shared at dconf this year.


 How are you going to prove that  safe code does not retain
 nonshared-references after the lock has been released?

 How does his work with array elements/slices.


I think `scope` and `return` have something to say about avoiding
escaping references. When the semantics are all working, we can
produce such test code.
Oct 12 2019
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:

 Access to shared memory should be disallowed only for safe 
 code, but not for system code. For safe code it's justified, 
 because safety is unavoidable there, for system code there's a 
 concern of ease of use of shared data, which is affected by 
 littering and is not addressed in this DIP. Stripping shared 
 qualifier off for assignment isn't necessarily correct if, say, 
 the data type switches between atomic and nonatomic reference 
 counting based on its type.


 system code can always cast away shared.
Oct 02 2019
parent reply Kagamin <spam here.lot> writes:
On Wednesday, 2 October 2019 at 11:49:48 UTC, Atila Neves wrote:

  system code can always cast away shared.


It's justified for  safe code, because safety is mandatory there, 
but not for system code. Access to shared data is well within 
semantics of system code. System code is even intended to access 
shared data because safe code can't do it in general case.
Oct 03 2019
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 3 October 2019 at 09:29:28 UTC, Kagamin wrote:

 On Wednesday, 2 October 2019 at 11:49:48 UTC, Atila Neves wrote:

  system code can always cast away shared.


 It's justified for  safe code, because safety is mandatory 
 there, but not for system code. Access to shared data is well 
 within semantics of system code. System code is even intended 
 to access shared data because safe code can't do it in general 
 case.


A coworker of mine (Don Clugston) came up with this analogy.
Multi-threading is a bit like radiating material, you put it in a 
safe-box, and put a warning sticker on it, such that only 
professionals may open it.
And even the professional would not want to be around the 
material for very long and certainly would want to be warned 
before he accesses the material.
Therefore the material should be in it's designated box sealed 
away fore the majority of the time.
Oct 03 2019
parent reply Kagamin <spam here.lot> writes:
On Thursday, 3 October 2019 at 12:14:30 UTC, Stefan Koch wrote:

 A coworker of mine (Don Clugston) came up with this analogy.
 Multi-threading is a bit like radiating material, you put it in 
 a safe-box, and put a warning sticker on it, such that only 
 professionals may open it.
 And even the professional would not want to be around the 
 material for very long and certainly would want to be warned 
 before he accesses the material.
 Therefore the material should be in it's designated box sealed 
 away fore the majority of the time.


Sounds like that coworker wants  safe code. The DIP is justified 
for safe code, but not for system code. Also the DIP provides 
memory safety, not thread safety.
Oct 04 2019
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 4 October 2019 at 10:12:45 UTC, Kagamin wrote:

 On Thursday, 3 October 2019 at 12:14:30 UTC, Stefan Koch wrote:

 A coworker of mine (Don Clugston) came up with this analogy.
 Multi-threading is a bit like radiating material, you put it 
 in a safe-box, and put a warning sticker on it, such that only 
 professionals may open it.
 And even the professional would not want to be around the 
 material for very long and certainly would want to be warned 
 before he accesses the material.
 Therefore the material should be in it's designated box sealed 
 away fore the majority of the time.


 Sounds like that coworker wants  safe code. The DIP is 
 justified for safe code, but not for system code. Also the DIP 
 provides memory safety, not thread safety.


The reason why you want shared access to be forbidden by default 
is to see where synchronization is required and force you to 
think about it.
In all your code not just safe code.
Oct 04 2019
parent reply Kagamin <spam here.lot> writes:
On Friday, 4 October 2019 at 12:22:16 UTC, Stefan Koch wrote:

 The reason why you want shared access to be forbidden by 
 default is to see where synchronization is required and force 
 you to think about it.
 In all your code not just safe code.


That's achieved by writing safe code, so the failure requires 
transition to trusted code, where there's type system and lack of 
conversion between shared and unshared data. Adding churn 
indefinitely isn't very practical, it increases cognitive load on 
programmer and diverts resources that can be spent on thinking 
about synchronization. Also the idea ignores the problem that 
people deem it difficult to work with shared data, adding more 
churn only worsens that problem, which is traded for an alleged 
solution to a hypothetical problem.
Oct 07 2019
parent reply Manu <turkeyman gmail.com> writes:
On Mon, Oct 7, 2019 at 1:05 AM Kagamin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Friday, 4 October 2019 at 12:22:16 UTC, Stefan Koch wrote:

 The reason why you want shared access to be forbidden by
 default is to see where synchronization is required and force
 you to think about it.
 In all your code not just safe code.


 That's achieved by writing safe code, so the failure requires
 transition to trusted code, where there's type system and lack of
 conversion between shared and unshared data. Adding churn
 indefinitely isn't very practical, it increases cognitive load on
 programmer and diverts resources that can be spent on thinking
 about synchronization. Also the idea ignores the problem that
 people deem it difficult to work with shared data, adding more
 churn only worsens that problem, which is traded for an alleged
 solution to a hypothetical problem.


This made exactly no sense to me. I don't know what you're saying here.
That said, based on your past posts, I think I need to assure you that
it is **NEVER** acceptable to freely read/write to shared data, and
that doesn't only apply to  safe code.

I wish Walter would update this DIP, but based on his comments, I
think it's going the right way...?
Oct 09 2019
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 9 October 2019 at 18:25:36 UTC, Manu wrote:

 This made exactly no sense to me. I don't know what you're 
 saying here. That said, based on your past posts, I think I 
 need to assure you that it is **NEVER** acceptable to freely 
 read/write to shared data, and that doesn't only apply to  safe 
 code.


Maybe set up a list of what problems and opportunities "shared" 
is supposed to deal with? What will it solve, and just as 
importantly, list explicitly which issues it won't deal with.

As far as I can tell, one could get better optimization if one 
could assume that certain memory areas are not accessed by other 
threads and avoid C++-ish sequencing points even in code that use 
concurrency-primitives. That seems to be out of scope?

If C++ is supposed to interface with D, then the spec ought to 
present an argument for how D's memory model is compatible with 
C++, which may be summed up as (quoting cppreference.com):

«Any thread can potentially access any object in the program 
(objects with automatic and thread-local storage duration may 
still be accessed by another thread through a pointer or by 
reference).

Different threads of execution are always allowed to access (read 
and modify) different memory locations concurrently, with no 
interference and no synchronization requirements.»
Oct 10 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 10 October 2019 at 07:03:26 UTC, Ola Fosheim Grøstad 
wrote:

 As far as I can tell, one could get better optimization if one 
 could assume that certain memory areas are not accessed by 
 other threads and avoid C++-ish sequencing points even in code 
 that use concurrency-primitives. That seems to be out of scope?


Or to be blunt, if "shared" has no effect on the IR then it ought 
to be removed from the language and replaced with mechanisms that 
makes it possible to implement the "shared" marker functionality 
as a template wrapper.
Oct 10 2019
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/9/2019 11:25 AM, Manu wrote:

 I wish Walter would update this DIP, but based on his comments, I
 think it's going the right way...?


I'll update it to clear up the ambiguity in it after this review cycle is over.

It'll be clear that reads and writes to shared data will not be allowed 
directly, you'll have to go through a function.
Oct 11 2019
prev sibling next sibling parent tn <no email.com> writes:
In my opinion, if a DIP is so confusingly written that most 
people don't understand what it is proposing, the review should 
be stopped, the DIP should be rewritten and then a new review 
should be started. Otherwise the review thread just gets full of 
irrelevant comments that are based on misunderstandings, and 
therefore becomes hard to follow.

This is the case for this DIP.

This comment by Walter made it more clear, where the 
inconsistencies originate from:

On Friday, 4 October 2019 at 03:35:45 UTC, Walter Bright wrote:

 Yes. The trouble stems from the DIP being originally written to 
 apply shared semantics to operators. This was changed to be 
 function only, and some vestiges of the original remain. Hence 
 the confusion.


It is not, however, an excuse not to fix the text before 
restarting the review.

***

When it comes to the actual content of this DIP, it should 
describe clearly both
1) what the current behavior is before this DIP
2) what the behavior would be after this DIP

(Currently 1 is either not addressed at all, or described very 
implicitly. And 1 is described inconsistently and not very 
clearly.)
Oct 04 2019
prev sibling parent Mike Parker <aldacron gmail.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1024, "Shared Atomics":

 https://github.com/dlang/DIPs/blob/0b892dd99aba74b9631572ad3a53000f5975b7c2/DIPs/DIP1024.md


Apologies to everyone who participated in the review. I didn't 
give the discussion my full attention, else I would have called 
off the review early so that the document could be corrected.

Generally, that's what the policy should be when such confusing 
ambiguities slip through the revision process. So if it does 
happen again in the future, I ask that whoever catches it please 
ping me (email or slack is the best bet) in case I miss it.

Walter is going to revise the DIP and we will do another round of 
community review soon.
Oct 17 2019