www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A Refcounted Array Type

reply Walter Bright <newshound2 digitalmars.com> writes:
This is pretty straightforward. More could be done:

1. small array optimization
2. support for ranges as constructor args
3. present a range interface
4. support for malloc/free instead of GC
5. bounds checking
6. the array[] and the count could be allocated together
7. array[] could be just a pointer

but the basic idea is there, I didn't want to hide it behind all the other
flesh 
a professional type would have.

Note the return in opIndex(). This is DIP25 at work!

Compile:
     dmd rcarray -unittest -main -dip25

===========================================

struct RCArray(E) {

     this(E[] a)
     {
         array = a.dup;
         start = 0;
         end = a.length;
         count = new int;
         *count = 1;
     }

     ~this()
     {
         if (count && --*count == 0)
             delete array;
     }

     this(this)
     {
         if (count)
             ++*count;
     }

     size_t length()
     {
         return end - start;
     }

     ref E opIndex(size_t i) return // here's the magic
     {
         return array[start + i];
     }

     RCArray opSlice(size_t lwr, size_t upr)
     {
         RCArray result = this;
         result.start = start + lwr;
         result.end = start + upr;
         return result;
     }

   private:
     E[] array;
     size_t start, end;
     int* count;
}

unittest
{
     static int[3] s = [7, 6, 4];
     auto r = RCArray!int(s);
     assert(r.length == 3);
     assert(r[0] == 7);
     assert(r[1] == 6);
     assert(r[2] == 4);
     assert(*r.count == 1);

     {
         auto r2 = r;
         assert(r2[0] == 7);
         assert(r2[1] == 6);
         assert(r2[2] == 4);
         assert(*r.count == 2);

         r[1] = 3;
         assert(r2[0] == 7);
         assert(r2[1] == 3);
         assert(r2[2] == 4);
     }
     assert(*r.count == 1);

     auto r3 = r[1 .. 3];
     r[2] = 9;
     assert(r3[0] == 3);
     assert(r3[1] == 9);

   /+
     ref int test(ref RCArray!int rr)
     {
         return rr[1]; // this gives error
     }
    +/
}
Feb 23 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/23/15 2:15 PM, Walter Bright wrote:

 This is pretty straightforward.


[snip]

The code builds if you slap a " safe:" at the top. There is one bug in 
the compiler: "delete" must not be allowed in  safe code. The destructor 
must be  trusted.

Understanding that this code (sans delete) is  safe (and the 
contribution of DIP25 to that) is of paramount importance.

Making it possible to define  safe structs that are still able to return 
reference to their internals is crucial. It paves the way for truly safe 
reference counted classes.


Andrei
Feb 23 2015
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Monday, February 23, 2015 15:28:21 Andrei Alexandrescu via Digitalmars-d
wrote:

 On 2/23/15 2:15 PM, Walter Bright wrote:

 This is pretty straightforward.


 [snip]

 The code builds if you slap a " safe:" at the top. There is one bug in
 the compiler: "delete" must not be allowed in  safe code. The destructor
 must be  trusted.


And delete is supposed to have been deprecated ages ago, but yeah, it
_definitely_ shouldn't be considered  safe.

- Jonathan M Davis
Feb 23 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/23/2015 9:59 PM, Jonathan M Davis via Digitalmars-d wrote:

 And delete is supposed to have been deprecated ages ago, but yeah, it
 _definitely_ shouldn't be considered  safe.


Managing memory always is going to be unsafe. The idea is to encapsulate that, 
which RCArray shows how to do.
Feb 23 2015
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Monday, February 23, 2015 23:02:03 Walter Bright via Digitalmars-d wrote:

 On 2/23/2015 9:59 PM, Jonathan M Davis via Digitalmars-d wrote:

 And delete is supposed to have been deprecated ages ago, but yeah, it
 _definitely_ shouldn't be considered  safe.


 Managing memory always is going to be unsafe. The idea is to encapsulate that,
 which RCArray shows how to do.


That's fine. The issue is that delete is considered  safe by the compiler,
not with the idea of RCArray. This code shouldn't compile:

void main()  safe
{
    auto i = new int(5);
    delete i;
}

and it does. Of course, delete is supposed to have been deprecated, but no
one has gotten around to doing it. So, maybe that's why no one made it so
that it was treated correctly with regards to  safe.

- Jonathan M Davis
Feb 24 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 1:32 AM, Jonathan M Davis via Digitalmars-d wrote:

 The issue is that delete is considered  safe by the compiler,


I thought more people would be interested in how to do a memory safe reference 
counted container.
Feb 24 2015
next sibling parent reply "matovitch" <camille.brugel laposte.net> writes:
On Tuesday, 24 February 2015 at 09:44:13 UTC, Walter Bright wrote:

 On 2/24/2015 1:32 AM, Jonathan M Davis via Digitalmars-d wrote:

 The issue is that delete is considered  safe by the compiler,


 I thought more people would be interested in how to do a memory 
 safe reference counted container.


This is really interesting ! Thought as a beginner, I am 
wondering why the counter need to be a pointer ?
Feb 24 2015
parent reply "Namespace" <rswhite4 gmail.com> writes:
On Tuesday, 24 February 2015 at 10:08:23 UTC, matovitch wrote:

 On Tuesday, 24 February 2015 at 09:44:13 UTC, Walter Bright 
 wrote:

 On 2/24/2015 1:32 AM, Jonathan M Davis via Digitalmars-d wrote:

 The issue is that delete is considered  safe by the compiler,


 I thought more people would be interested in how to do a 
 memory safe reference counted container.


 This is really interesting ! Thought as a beginner, I am 
 wondering why the counter need to be a pointer ?


That's why:
----
this(this)
{
     if (count)
         ++*count;
}
----
Feb 24 2015
parent reply "matovitch" <camille.brugel laposte.net> writes:
On Tuesday, 24 February 2015 at 10:11:02 UTC, Namespace wrote:

 On Tuesday, 24 February 2015 at 10:08:23 UTC, matovitch wrote:

 On Tuesday, 24 February 2015 at 09:44:13 UTC, Walter Bright 
 wrote:

 On 2/24/2015 1:32 AM, Jonathan M Davis via Digitalmars-d 
 wrote:

 The issue is that delete is considered  safe by the compiler,


 I thought more people would be interested in how to do a 
 memory safe reference counted container.


 This is really interesting ! Thought as a beginner, I am 
 wondering why the counter need to be a pointer ?


 That's why:
 ----
 this(this)
 {
     if (count)
         ++*count;
 }
 ----


Hmm, I don't see why that's why... :(
Feb 24 2015
next sibling parent "Namespace" <rswhite4 gmail.com> writes:
On Tuesday, 24 February 2015 at 10:13:36 UTC, matovitch wrote:

 On Tuesday, 24 February 2015 at 10:11:02 UTC, Namespace wrote:

 On Tuesday, 24 February 2015 at 10:08:23 UTC, matovitch wrote:

 On Tuesday, 24 February 2015 at 09:44:13 UTC, Walter Bright 
 wrote:

 On 2/24/2015 1:32 AM, Jonathan M Davis via Digitalmars-d 
 wrote:

 The issue is that delete is considered  safe by the 
 compiler,


 I thought more people would be interested in how to do a 
 memory safe reference counted container.


 This is really interesting ! Thought as a beginner, I am 
 wondering why the counter need to be a pointer ?


 That's why:
 ----
 this(this)
 {
    if (count)
        ++*count;
 }
 ----


 Hmm, I don't see why that's why... :(


It's more effective if you figure it out on your own.
But maybe this is a bit more clear?
https://github.com/Dgame/m3/blob/master/SmartPointer.d#L106
Feb 24 2015
prev sibling parent reply "Ulrich =?UTF-8?B?S8O8dHRsZXIi?= <kuettler gmail.com> writes:
On Tuesday, 24 February 2015 at 10:13:36 UTC, matovitch wrote:

 On Tuesday, 24 February 2015 at 10:11:02 UTC, Namespace wrote:

 On Tuesday, 24 February 2015 at 10:08:23 UTC, matovitch wrote:
 That's why:
 ----
 this(this)
 {
    if (count)
        ++*count;
 }
 ----


 Hmm, I don't see why that's why... :(


The counter is shared amount all "copies" of the array.
Feb 24 2015
parent "matovitch" <camille.brugel laposte.net> writes:
On Tuesday, 24 February 2015 at 10:16:00 UTC, Ulrich Küttler 
wrote:

 On Tuesday, 24 February 2015 at 10:13:36 UTC, matovitch wrote:

 On Tuesday, 24 February 2015 at 10:11:02 UTC, Namespace wrote:

 On Tuesday, 24 February 2015 at 10:08:23 UTC, matovitch wrote:
 That's why:
 ----
 this(this)
 {
   if (count)
       ++*count;
 }
 ----


 Hmm, I don't see why that's why... :(


 The counter is shared amount all "copies" of the array.


Ah yes, what an idiot ! Thanks !
Feb 24 2015
prev sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, February 24, 2015 01:44:07 Walter Bright via Digitalmars-d wrote:

 On 2/24/2015 1:32 AM, Jonathan M Davis via Digitalmars-d wrote:

 The issue is that delete is considered  safe by the compiler,


 I thought more people would be interested in how to do a memory safe reference
 counted container.


Oh, I think that that there's definitely something there and that we should
be interested. I don't think that either Andrei or I were trying to take
away from that. It's just that code showed a problem in that delete was
considered  safe when it should be  system, so Andrei pointed that out, and
then I agreed with him and pointed out that delete was really supposed to
have been deprecated by now anyway. But the concept is perfectly valid, and
delete can be replaced with destroy and core.memory.GC.free for the same
effect without actually needing delete, so it'll still work even if delete
is finally deprecated - though honestly, I would think that if you're going
to be doing manual memory management, it would just be better to use malloc
and free and avoid the GC altogether (though if the element type contains
any references or pointers, you probably need to tell the GC about the
memory so that it can scan it).

- Jonathan M Davis
Feb 25 2015
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 5. bounds checking


When you go past bounds of a built-in array you get an error 
located in the user code, while if you put a pre-condition in 
your Array struct to detect the same errors, you get a run-time 
error message located in that pre-condition instead. I'd like 
some way to solve this small problem of giving more correctly 
located error messages. In Contract programming lingo it's a 
problem of "blame management" (I'd also like a way to detect some 
compile-time out-of-bound errors for user-defined collections, 
but you said this is not worth the effort).

Bye,
bearophile
Feb 23 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/23/2015 4:13 PM, bearophile wrote:

 5. bounds checking


 When you go past bounds of a built-in array you get an error located in the
user
 code, while if you put a pre-condition in your Array struct to detect the same
 errors, you get a run-time error message located in that pre-condition instead.
 I'd like some way to solve this small problem of giving more correctly located
 error messages. In Contract programming lingo it's a problem of "blame
 management" (I'd also like a way to detect some compile-time out-of-bound
errors
 for user-defined collections, but you said this is not worth the effort).


This is off-topic. The point of the array class as presented is to show how a 
memory safe reference counted container can be built.
Feb 23 2015
prev sibling next sibling parent reply Max Klyga <max.klyga gmail.com> writes:
I thought that delete is deprecated, yet here Walter himself promotes a 
solution that uses it.
Can we *PLEASE* finally deprecate things that are supposed to be 
deprecated and remove things that are supposed to be removed?

On 2015-02-23 22:15:46 +0000, Walter Bright said:


 This is pretty straightforward. More could be done:
 
 1. small array optimization
 2. support for ranges as constructor args
 3. present a range interface
 4. support for malloc/free instead of GC
 5. bounds checking
 6. the array[] and the count could be allocated together
 7. array[] could be just a pointer
 
 but the basic idea is there, I didn't want to hide it behind all the 
 other flesh a professional type would have.
 
 Note the return in opIndex(). This is DIP25 at work!
 
 Compile:
      dmd rcarray -unittest -main -dip25
 
 ===========================================
 
 struct RCArray(E) {
 
      this(E[] a)
      {
          array = a.dup;
          start = 0;
          end = a.length;
          count = new int;
          *count = 1;
      }
 
      ~this()
      {
          if (count && --*count == 0)
              delete array;
      }
 
      this(this)
      {
          if (count)
              ++*count;
      }
 
      size_t length()
      {
          return end - start;
      }
 
      ref E opIndex(size_t i) return // here's the magic
      {
          return array[start + i];
      }
 
      RCArray opSlice(size_t lwr, size_t upr)
      {
          RCArray result = this;
          result.start = start + lwr;
          result.end = start + upr;
          return result;
      }
 
    private:
      E[] array;
      size_t start, end;
      int* count;
 }
 
 unittest
 {
      static int[3] s = [7, 6, 4];
      auto r = RCArray!int(s);
      assert(r.length == 3);
      assert(r[0] == 7);
      assert(r[1] == 6);
      assert(r[2] == 4);
      assert(*r.count == 1);
 
      {
          auto r2 = r;
          assert(r2[0] == 7);
          assert(r2[1] == 6);
          assert(r2[2] == 4);
          assert(*r.count == 2);
 
          r[1] = 3;
          assert(r2[0] == 7);
          assert(r2[1] == 3);
          assert(r2[2] == 4);
      }
      assert(*r.count == 1);
 
      auto r3 = r[1 .. 3];
      r[2] = 9;
      assert(r3[0] == 3);
      assert(r3[1] == 9);
 
    /+
      ref int test(ref RCArray!int rr)
      {
          return rr[1]; // this gives error
      }
     +/
 }
Feb 23 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/23/2015 5:01 PM, Max Klyga wrote:

 I thought that delete is deprecated, yet here Walter himself promotes a
solution
 that uses it.
 Can we *PLEASE* finally deprecate things that are supposed to be deprecated and
 remove things that are supposed to be removed?


That isn't the point. I could have well have used malloc/free, but I wanted the 
code example to be about how to make a memory safe ref counted container, not 
about the details of memory allocation. It demonstrates how 'return ref' is to 
be used.
Feb 23 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright wrote:

     ref E opIndex(size_t i) return // here's the magic


exactly what is this doing? I don't see this explained in DIP25 
at all.
Feb 23 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/23/2015 5:41 PM, weaselcat wrote:

 On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright wrote:

     ref E opIndex(size_t i) return // here's the magic


 exactly what is this doing? I don't see this explained in DIP25 at all.


The 'this' is passed by reference. So the 'return' is really 'return ref this', 
the rest is explained by DIP25.
Feb 23 2015
next sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Tuesday, 24 February 2015 at 02:06:07 UTC, Walter Bright wrote:

 On 2/23/2015 5:41 PM, weaselcat wrote:

 On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright 
 wrote:

    ref E opIndex(size_t i) return // here's the magic


 exactly what is this doing? I don't see this explained in 
 DIP25 at all.


 The 'this' is passed by reference. So the 'return' is really 
 'return ref this', the rest is explained by DIP25.


Oh, I see. That makes a lot more sense.
Feb 23 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/23/15 6:05 PM, Walter Bright wrote:

 On 2/23/2015 5:41 PM, weaselcat wrote:

 On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright wrote:

     ref E opIndex(size_t i) return // here's the magic


 exactly what is this doing? I don't see this explained in DIP25 at all.


 The 'this' is passed by reference. So the 'return' is really 'return ref
 this', the rest is explained by DIP25.


We should amend DIP25 to explain that "this" is handled like a 
parameter. -- Andrei
Feb 24 2015
parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 24 February 2015 at 15:51:42 UTC, Andrei Alexandrescu 
wrote:

 On 2/23/15 6:05 PM, Walter Bright wrote:

 On 2/23/2015 5:41 PM, weaselcat wrote:

 On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright 
 wrote:

    ref E opIndex(size_t i) return // here's the magic


 exactly what is this doing? I don't see this explained in 
 DIP25 at all.


 The 'this' is passed by reference. So the 'return' is really 
 'return ref
 this', the rest is explained by DIP25.


 We should amend DIP25 to explain that "this" is handled like a 
 parameter. -- Andrei


Note that I'm not surprised by that behavior (this is indeed ref 
for struct). But this is not ref for classes, which will defeat 
DIP25.
Feb 24 2015
prev sibling next sibling parent "Chris" <wendlec tcd.ie> writes:
On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright wrote:

 This is pretty straightforward. More could be done:

 1. small array optimization
 2. support for ranges as constructor args
 3. present a range interface
 4. support for malloc/free instead of GC
 5. bounds checking
 6. the array[] and the count could be allocated together
 7. array[] could be just a pointer

 but the basic idea is there, I didn't want to hide it behind 
 all the other flesh a professional type would have.

 Note the return in opIndex(). This is DIP25 at work!

 Compile:
     dmd rcarray -unittest -main -dip25

 ===========================================

 struct RCArray(E) {

     this(E[] a)
     {
         array = a.dup;
         start = 0;
         end = a.length;
         count = new int;
         *count = 1;
     }

     ~this()
     {
         if (count && --*count == 0)
             delete array;
     }

     this(this)
     {
         if (count)
             ++*count;
     }

     size_t length()
     {
         return end - start;
     }

     ref E opIndex(size_t i) return // here's the magic
     {
         return array[start + i];
     }

     RCArray opSlice(size_t lwr, size_t upr)
     {
         RCArray result = this;
         result.start = start + lwr;
         result.end = start + upr;
         return result;
     }

   private:
     E[] array;
     size_t start, end;
     int* count;
 }

 unittest
 {
     static int[3] s = [7, 6, 4];
     auto r = RCArray!int(s);
     assert(r.length == 3);
     assert(r[0] == 7);
     assert(r[1] == 6);
     assert(r[2] == 4);
     assert(*r.count == 1);

     {
         auto r2 = r;
         assert(r2[0] == 7);
         assert(r2[1] == 6);
         assert(r2[2] == 4);
         assert(*r.count == 2);

         r[1] = 3;
         assert(r2[0] == 7);
         assert(r2[1] == 3);
         assert(r2[2] == 4);
     }
     assert(*r.count == 1);

     auto r3 = r[1 .. 3];
     r[2] = 9;
     assert(r3[0] == 3);
     assert(r3[1] == 9);

   /+
     ref int test(ref RCArray!int rr)
     {
         return rr[1]; // this gives error
     }
    +/
 }


Thanks for the example.

I think we need more practical stuff like this compiled somewhere 
on the homepage, especially when a new feature is introduced, to 
answer questions like: What is it? When should I use it? And how 
do I implement it? Saves a lot of time and the frustration of 
finding out later that there _is_ such a feature, only I never 
heard of it until now.
Feb 24 2015
prev sibling next sibling parent reply "ponce" <contact g3mesfrommars.fr> writes:
Does DIP25 means we can turn resources into RC types and be done 
with non-deterministic destructor calls by the GC, without using 
RefCounted or Unique?

On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright wrote:

 This is pretty straightforward. More could be done:

 1. small array optimization
 2. support for ranges as constructor args
 3. present a range interface
 4. support for malloc/free instead of GC
 5. bounds checking
 6. the array[] and the count could be allocated together
 7. array[] could be just a pointer

 but the basic idea is there, I didn't want to hide it behind 
 all the other flesh a professional type would have.

 Note the return in opIndex(). This is DIP25 at work!

 Compile:
     dmd rcarray -unittest -main -dip25

 ===========================================

 struct RCArray(E) {

     this(E[] a)
     {
         array = a.dup;
         start = 0;
         end = a.length;
         count = new int;
         *count = 1;
     }

     ~this()
     {
         if (count && --*count == 0)
             delete array;
     }

     this(this)
     {
         if (count)
             ++*count;
     }

     size_t length()
     {
         return end - start;
     }

     ref E opIndex(size_t i) return // here's the magic
     {
         return array[start + i];
     }

     RCArray opSlice(size_t lwr, size_t upr)
     {
         RCArray result = this;
         result.start = start + lwr;
         result.end = start + upr;
         return result;
     }

   private:
     E[] array;
     size_t start, end;
     int* count;
 }

 unittest
 {
     static int[3] s = [7, 6, 4];
     auto r = RCArray!int(s);
     assert(r.length == 3);
     assert(r[0] == 7);
     assert(r[1] == 6);
     assert(r[2] == 4);
     assert(*r.count == 1);

     {
         auto r2 = r;
         assert(r2[0] == 7);
         assert(r2[1] == 6);
         assert(r2[2] == 4);
         assert(*r.count == 2);

         r[1] = 3;
         assert(r2[0] == 7);
         assert(r2[1] == 3);
         assert(r2[2] == 4);
     }
     assert(*r.count == 1);

     auto r3 = r[1 .. 3];
     r[2] = 9;
     assert(r3[0] == 3);
     assert(r3[1] == 9);

   /+
     ref int test(ref RCArray!int rr)
     {
         return rr[1]; // this gives error
     }
    +/
 }
Feb 24 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 3:14 AM, ponce wrote:

 Does DIP25 means we can turn resources into RC types and be done with
 non-deterministic destructor calls by the GC, without using RefCounted
 or Unique?


That's what we're aiming for. -- Andrei
Feb 24 2015
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2015-02-23 22:15:46 +0000, Walter Bright said:


 int* count;
 
 [...] if (count && --*count == 0) [...]


Careful!

This isn't memory safe and you have to thank the GC for it. If you ever 
use RCArray as a member variable in a class, the RCArray destructor is 
going to be called from a random thread when the class destructor is 
run. If some thread has a stack reference to the array you have a race.

You have to use an atomic counter unless you can prove the RCArray 
struct will never be put in a GC-managed context. It is rather sad that 
the language has no way to enforce such a restriction, and also that 
 safe cannot detect that this is a problem here.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 24 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/24/15 8:17 AM, Michel Fortin wrote:

 On 2015-02-23 22:15:46 +0000, Walter Bright said:


 int* count;

 [...] if (count && --*count == 0) [...]


 Careful!

 This isn't memory safe and you have to thank the GC for it. If you ever
 use RCArray as a member variable in a class, the RCArray destructor is
 going to be called from a random thread when the class destructor is
 run. If some thread has a stack reference to the array you have a race.

 You have to use an atomic counter unless you can prove the RCArray
 struct will never be put in a GC-managed context. It is rather sad that
 the language has no way to enforce such a restriction, and also that
  safe cannot detect that this is a problem here.


Actually, RCArray can never be allocated on GC, or you may corrupt 
memory. count may be non-null, and point at invalid memory when the dtor 
is called.

Only safe way to do this is to C malloc/free the count. And yes, at that 
point, you need atomics.

-Steve
Feb 24 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 6:56 AM, Steven Schveighoffer wrote:

 Actually, RCArray can never be allocated on GC, or you may corrupt memory.
count
 may be non-null, and point at invalid memory when the dtor is called.


Just set count to null after the delete.


 Only safe way to do this is to C malloc/free the count. And yes, at that point,
 you need atomics.


No, RCArray is not intended for shared access between threads.

Shared containers and local containers are different enough that they merit 
being different types with different implementations altogether. Trying to just 
slap 'shared' on a container isn't going to work.
Feb 24 2015
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 24 February 2015 at 20:35:04 UTC, Walter Bright wrote:

 Only safe way to do this is to C malloc/free the count. And 
 yes, at that point,
 you need atomics.


 No, RCArray is not intended for shared access between threads.

 Shared containers and local containers are different enough 
 that they merit being different types with different 
 implementations altogether. Trying to just slap 'shared' on a 
 container isn't going to work.


Maybe we want to fix the GC, exceptions and delegates or disable 
them in  safe code because they all cause implicit sharing.
Feb 24 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 2:34 PM, deadalnix wrote:

 Maybe we want to fix the GC, exceptions and delegates or disable them in  safe
 code because they all cause implicit sharing.


Andrei already filed a bug report on the GC issue. If there aren't bug reports 
on the other two, file them.
Feb 24 2015
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 25 February 2015 at 00:13:03 UTC, Walter Bright 
wrote:

 On 2/24/2015 2:34 PM, deadalnix wrote:

 Maybe we want to fix the GC, exceptions and delegates or 
 disable them in  safe
 code because they all cause implicit sharing.


 Andrei already filed a bug report on the GC issue. If there 
 aren't bug reports on the other two, file them.


Well I filled a bug on the delegate one ~2 years ago, and the 2 
others are features, not bugs, as far as I know.
Feb 24 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 4:32 PM, deadalnix wrote:

 On Wednesday, 25 February 2015 at 00:13:03 UTC, Walter Bright wrote:

 On 2/24/2015 2:34 PM, deadalnix wrote:

 Maybe we want to fix the GC, exceptions and delegates or disable them in  safe
 code because they all cause implicit sharing.


 Andrei already filed a bug report on the GC issue. If there aren't bug reports
 on the other two, file them.


 Well I filled a bug on the delegate one ~2 years ago, and the 2 others are
 features, not bugs, as far as I know.


Without examples and bug reports, I don't really know what you're talking
about. 
At least with a bug report, you have something easy to link to instead of
trying 
to explain it over and over again.
Feb 24 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/24/15 3:34 PM, Walter Bright wrote:

 On 2/24/2015 6:56 AM, Steven Schveighoffer wrote:

 Actually, RCArray can never be allocated on GC, or you may corrupt
 memory. count
 may be non-null, and point at invalid memory when the dtor is called.


 Just set count to null after the delete.


No, the GC may have already deallocated the memory count points at, and 
count is still non-null.

This is why you can never refer to GC-allocated memory in dtors, if they 
can be called from the GC.


 Only safe way to do this is to C malloc/free the count. And yes, at
 that point,
 you need atomics.


 No, RCArray is not intended for shared access between threads.


GC is what shares the access. For instance, if you have an RCArray in 
one object that is being collected, it will attempt to decrement count. 
However, that doesn't mean that another RCArray reference in the 
originating thread won't also be trying to adjust count. In reality, 
it's only shared between the GC-collection running thread and the 
originating thread. I was not talking at all about making RCArray 
shared-usable.

As talked about before, running dtors in the originating thread can 
solve this problem.

-Steve
Feb 26 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/26/15 8:51 AM, Steven Schveighoffer wrote:

 As talked about before, running dtors in the originating thread can
 solve this problem.


Yah, that will solve the nonatomic reference counting. What do you think 
about http://forum.dlang.org/thread/mcllre$1abs$1 digitalmars.com?

Andrei
Feb 26 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 11:57 AM, Andrei Alexandrescu wrote:

 On 2/26/15 8:51 AM, Steven Schveighoffer wrote:

 As talked about before, running dtors in the originating thread can
 solve this problem.


 Yah, that will solve the nonatomic reference counting. What do you think
 about http://forum.dlang.org/thread/mcllre$1abs$1 digitalmars.com?


I saw that, it sounds reasonable. I have to mull over what it means.

I think possibly a better solution is to have a "finalize" function, 
similar to how tango does it, so the dtor is only called from 
destroy/delete, and the finalize method is called from the GC.

-Steve
Feb 26 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 26 February 2015 at 18:04:11 UTC, Steven 
Schveighoffer wrote:

 I think possibly a better solution is to have a "finalize" 
 function, similar to how tango does it, so the dtor is only 
 called from destroy/delete, and the finalize method is called 
 from the GC.


Ugh, so you still have to identify objects before releasing the 
memory?

Put all objects that need cleanup on a separate GC heap then, to 
reduce impact.
Feb 26 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 26 February 2015 at 16:57:51 UTC, Andrei 
Alexandrescu wrote:

 On 2/26/15 8:51 AM, Steven Schveighoffer wrote:

 As talked about before, running dtors in the originating 
 thread can
 solve this problem.


 Yah, that will solve the nonatomic reference counting. What do 
 you think about 
 http://forum.dlang.org/thread/mcllre$1abs$1 digitalmars.com?

 Andrei


class BazingaException : Exception {

     RefCount!Stuff reallyImportantStuff;

     // ...

}

void main() {
     auto t = new Thread({
         RefCount!Stuff s = ...;
         throw new BazingaException(s);
     });

     t.start();
     t.join();
}
Feb 26 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/26/15 12:54 PM, deadalnix wrote:

 On Thursday, 26 February 2015 at 16:57:51 UTC, Andrei Alexandrescu wrote:

 On 2/26/15 8:51 AM, Steven Schveighoffer wrote:

 As talked about before, running dtors in the originating thread can
 solve this problem.


 Yah, that will solve the nonatomic reference counting. What do you
 think about http://forum.dlang.org/thread/mcllre$1abs$1 digitalmars.com?

 Andrei


 class BazingaException : Exception {

      RefCount!Stuff reallyImportantStuff;

      // ...

 }

 void main() {
      auto t = new Thread({
          RefCount!Stuff s = ...;
          throw new BazingaException(s);
      });

      t.start();
      t.join();
 }


Could you please walk me through what the matter is here. Thanks. -- Andrei
Feb 26 2015
next sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2015-02-26 21:07:26 +0000, Andrei Alexandrescu said:


 On 2/26/15 12:54 PM, deadalnix wrote:

 On Thursday, 26 February 2015 at 16:57:51 UTC, Andrei Alexandrescu wrote:

 On 2/26/15 8:51 AM, Steven Schveighoffer wrote:

 As talked about before, running dtors in the originating thread can
 solve this problem.


 
 Yah, that will solve the nonatomic reference counting. What do you
 think about http://forum.dlang.org/thread/mcllre$1abs$1 digitalmars.com?
 
 Andrei


 
 class BazingaException : Exception {
 
 RefCount!Stuff reallyImportantStuff;
 
 // ...
 
 }
 
 void main() {
 auto t = new Thread({
 RefCount!Stuff s = ...;
 throw new BazingaException(s);
 });
 
 t.start();
 t.join();
 }


 
 Could you please walk me through what the matter is here. Thanks. -- Andrei


The exception is thrown by t.join() in another thread, after the 
originating thread died. Thus, obviously, it cannot be destructed in 
the originating thread as stated above. But everyone already know that.

But the example doesn't make one problem quite clear: Unless the GC 
destroys the exception in the thread join() was called, there can be a 
race. That's because join() moves the exception to another thread, and 
the thread that now owns the exception could make copies of that 
reallyImportantStuff and access the counter beyond the exception's 
lifetime. So it turns out that the GC heap needs call the exception's 
destructor in the thread calling join() to avoid races.

Additionally, if the old thread has leftover objects still not yet 
collected, they'll need to be destroyed in the thread calling join() 
too. Otherwise you might get races when the exception is destroyed.

So you could solve all that by changing of ownership for things 
originating from the worker thread to the thread that is calling 
join(). Or if no one calls join(), then you can destroy objects 
originating from the dead thread in any thread, as long as they are all 
destroyed *in the same thread* (because objects originating from the 
same thread might all points to the same thread-local reference counts).

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 26 2015
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 26 February 2015 at 21:07:26 UTC, Andrei 
Alexandrescu wrote:

 Could you please walk me through what the matter is here. 
 Thanks. -- Andrei


1/ The originating thread is likely to be dead by the time we 
collect.
2/ The RefCounted Stuff crossed thread boundaries so even if we 
kept the thread alive somehow to be able to destruct as 
mentioned, this is still not thread safe.
Feb 26 2015
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 26 February 2015 at 16:51:30 UTC, Steven 
Schveighoffer wrote:

 As talked about before, running dtors in the originating thread 
 can solve this problem.


This probably implies forcibly destroying objects whose creating 
thread terminated.
Feb 26 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 12:56 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Thursday, 26 February 2015 at 16:51:30 UTC, Steven Schveighoffer wrote:

 As talked about before, running dtors in the originating thread can
 solve this problem.


 This probably implies forcibly destroying objects whose creating thread
 terminated.


I don't think so, those objects can just be destroyed by the 
GC-collection running thread. If the thread is no longer present, there 
can't be a race.

-Steve
Feb 26 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 26 February 2015 at 18:08:28 UTC, Steven 
Schveighoffer wrote:

 On 2/26/15 12:56 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 On Thursday, 26 February 2015 at 16:51:30 UTC, Steven 
 Schveighoffer wrote:

 As talked about before, running dtors in the originating 
 thread can
 solve this problem.


 This probably implies forcibly destroying objects whose 
 creating thread
 terminated.


 I don't think so, those objects can just be destroyed by the 
 GC-collection running thread. If the thread is no longer 
 present, there can't be a race.


That's true. However, what if the destructors access global 
(thread-local) variables? Is that already disallowed?
Feb 26 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 4:40 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Thursday, 26 February 2015 at 18:08:28 UTC, Steven Schveighoffer wrote:

 On 2/26/15 12:56 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?=
 <schuetzm gmx.net>" wrote:

 On Thursday, 26 February 2015 at 16:51:30 UTC, Steven Schveighoffer
 wrote:

 As talked about before, running dtors in the originating thread can
 solve this problem.


 This probably implies forcibly destroying objects whose creating thread
 terminated.


 I don't think so, those objects can just be destroyed by the
 GC-collection running thread. If the thread is no longer present,
 there can't be a race.


 That's true. However, what if the destructors access global
 (thread-local) variables? Is that already disallowed?


Hm... I don't know. Is it "disallowed?" I don't think so (a simple test 
would suffice), but if any code exists today that does it, it's very 
very wrong :) I would suspect that such code should be banned.

-Steve
Feb 26 2015
next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Thu, 26 Feb 2015 18:04:28 -0500, Steven Schveighoffer wrote:


 That's true. However, what if the destructors access global
 (thread-local) variables? Is that already disallowed?


=20
 Hm... I don't know. Is it "disallowed?" I don't think so (a simple test
 would suffice), but if any code exists today that does it, it's very
 very wrong :) I would suspect that such code should be banned.


but what if i'm doing memory managing myself and i *know* that it's ok to=20
use thread-locals in my destructor?=
Feb 26 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 7:34 PM, ketmar wrote:

 On Thu, 26 Feb 2015 18:04:28 -0500, Steven Schveighoffer wrote:


 That's true. However, what if the destructors access global
 (thread-local) variables? Is that already disallowed?


 Hm... I don't know. Is it "disallowed?" I don't think so (a simple test
 would suffice), but if any code exists today that does it, it's very
 very wrong :) I would suspect that such code should be banned.


 but what if i'm doing memory managing myself and i *know* that it's ok to
 use thread-locals in my destructor?


Accessing thread-local data in the GC-run destructor means you are 
accessing in the GC collecting thread. In the case where we make dtors 
run in the originating thread, it is OK. But if the thread is gone, then 
any running of that dtor may incorrectly use the TL data in that thread, 
which may not be correct.

In any case, accessing thread local data today is most certainly wrong, 
since there isn't even a likelihood the same thread will destroy the data.

I can't see a situation where this would be what you wanted. Accessing 
global data should be fine, but thread-local data is not.

-Steve
Feb 26 2015
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Thu, 26 Feb 2015 21:56:30 -0500, Steven Schveighoffer wrote:


 On 2/26/15 7:34 PM, ketmar wrote:

 On Thu, 26 Feb 2015 18:04:28 -0500, Steven Schveighoffer wrote:


 That's true. However, what if the destructors access global
 (thread-local) variables? Is that already disallowed?


 Hm... I don't know. Is it "disallowed?" I don't think so (a simple
 test would suffice), but if any code exists today that does it, it's
 very very wrong :) I would suspect that such code should be banned.


 but what if i'm doing memory managing myself and i *know* that it's ok
 to use thread-locals in my destructor?


 Accessing thread-local data in the GC-run destructor means you are
 accessing in the GC collecting thread. In the case where we make dtors
 run in the originating thread, it is OK. But if the thread is gone, then
 any running of that dtor may incorrectly use the TL data in that thread,
 which may not be correct.
=20
 In any case, accessing thread local data today is most certainly wrong,
 since there isn't even a likelihood the same thread will destroy the
 data.
=20
 I can't see a situation where this would be what you wanted. Accessing
 global data should be fine, but thread-local data is not.


let's assume that i have a program that has only one thread. i know it=20
for sure. and i don't like prepending ugly `__gshared` to my declarations.

sure, it all depends of GC code (it can be multithreaded itself, or it=20
can deliberately run finalisers in another thread), but why i should not=20
be allowed to do what i want? i can write that GC myself, and i *know*=20
that there will be no other threads.

this is the same as allocating in finaliser: as finalisers aren't made=20
implicitly  nogc, i can see no sense in restricting 'em in any other way,=20
as the same arguments applies in both cases.=
Feb 26 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 10:56 PM, ketmar wrote:

 On Thu, 26 Feb 2015 21:56:30 -0500, Steven Schveighoffer wrote:



 I can't see a situation where this would be what you wanted. Accessing
 global data should be fine, but thread-local data is not.


 let's assume that i have a program that has only one thread. i know it
 for sure. and i don't like prepending ugly `__gshared` to my declarations.


In that case, you shouldn't be subject to any kind of race conditions. 
But we can't make the library/language based on this assumption. Your 
case is the exceptional case.

I think it's going to be difficult to impossible to prevent people from 
using TLS in destructors anyway. It can probably be a guideline more 
than a compiler error.

-Steve
Feb 26 2015
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Thu, 26 Feb 2015 23:13:01 -0500, Steven Schveighoffer wrote:


 I think it's going to be difficult to impossible to prevent people from
 using TLS in destructors anyway. It can probably be a guideline more
 than a compiler error.


and a warning, please! a warning that i can silence, of course.=
Feb 26 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 27 February 2015 at 04:13:03 UTC, Steven Schveighoffer 
wrote:

 In that case, you shouldn't be subject to any kind of race 
 conditions. But we can't make the library/language based on 
 this assumption. Your case is the exceptional case.


His case is not  safe, so we can ignore that problem. However, 
this do not change the fact that the type qualifier do not 
provide the guarantee they are supposed to.
Feb 27 2015
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Fri, 27 Feb 2015 20:51:54 +0000, deadalnix wrote:


 On Friday, 27 February 2015 at 04:13:03 UTC, Steven Schveighoffer wrote:

 In that case, you shouldn't be subject to any kind of race conditions.
 But we can't make the library/language based on this assumption. Your
 case is the exceptional case.


 His case is not  safe, so we can ignore that problem.


the compiler tends to disagree:

=3D=3D=3D test.d =3D=3D=3D
int myglobal;

class A {
  ~this ()  safe { if (myglobal =3D=3D 42) assert(0); }
}

void main () {
  auto a =3D new A;
}
=3D=3D=3D=3D=3D=3D

dmd -w -c -o- test.d

wow! no warnings, no errors!=
Feb 27 2015
next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Saturday, 28 February 2015 at 04:18:38 UTC, ketmar wrote:

 On Fri, 27 Feb 2015 20:51:54 +0000, deadalnix wrote:


 On Friday, 27 February 2015 at 04:13:03 UTC, Steven 
 Schveighoffer wrote:

 In that case, you shouldn't be subject to any kind of race 
 conditions.
 But we can't make the library/language based on this 
 assumption. Your
 case is the exceptional case.


 His case is not  safe, so we can ignore that problem.


 the compiler tends to disagree:

 === test.d ===
 int myglobal;

 class A {
   ~this ()  safe { if (myglobal == 42) assert(0); }
 }

 void main () {
   auto a = new A;
 }
 ======

 dmd -w -c -o- test.d

 wow! no warnings, no errors!


Slighty off-topic,
If DIP74 is approved it would be nice if destructors were 
redesigned(and possibly dropped altogether for non-deterministic 
lifetimes.)
Feb 27 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 28 February 2015 at 04:27:36 UTC, weaselcat wrote:

 On Saturday, 28 February 2015 at 04:18:38 UTC, ketmar wrote:

 On Fri, 27 Feb 2015 20:51:54 +0000, deadalnix wrote:


 On Friday, 27 February 2015 at 04:13:03 UTC, Steven 
 Schveighoffer wrote:

 In that case, you shouldn't be subject to any kind of race 
 conditions.
 But we can't make the library/language based on this 
 assumption. Your
 case is the exceptional case.


 His case is not  safe, so we can ignore that problem.


 the compiler tends to disagree:

 === test.d ===
 int myglobal;

 class A {
  ~this ()  safe { if (myglobal == 42) assert(0); }
 }

 void main () {
  auto a = new A;
 }
 ======

 dmd -w -c -o- test.d

 wow! no warnings, no errors!


 Slighty off-topic,
 If DIP74 is approved it would be nice if destructors were 
 redesigned(and possibly dropped altogether for 
 non-deterministic lifetimes.)


Yeah, we should have distinct finalizers (with or without DIP74).
Feb 28 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/27/15 11:18 PM, ketmar wrote:

 On Fri, 27 Feb 2015 20:51:54 +0000, deadalnix wrote:


 On Friday, 27 February 2015 at 04:13:03 UTC, Steven Schveighoffer wrote:

 In that case, you shouldn't be subject to any kind of race conditions.
 But we can't make the library/language based on this assumption. Your
 case is the exceptional case.


 His case is not  safe, so we can ignore that problem.


 the compiler tends to disagree:

 === test.d ===
 int myglobal;

 class A {
    ~this ()  safe { if (myglobal == 42) assert(0); }
 }

 void main () {
    auto a = new A;
 }
 ======

 dmd -w -c -o- test.d

 wow! no warnings, no errors!


I think in this case, the compiler can probably statically reject the 
code. But there are definitely ways around that, so I don't think full 
compliance is possible. I also think it's wrong-headed to have the 
compiler reject such code based on a GC deficiency.

The only solution is to have GC-allocated data be destroyed in the same 
thread it's allocated in. This brings the multi-threaded program into 
the same realm as yours -- looks like one thread for unshared data :)

-Steve
Mar 02 2015
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Mon, 02 Mar 2015 15:43:42 -0500, Steven Schveighoffer wrote:


 On 2/27/15 11:18 PM, ketmar wrote:

 On Fri, 27 Feb 2015 20:51:54 +0000, deadalnix wrote:


 On Friday, 27 February 2015 at 04:13:03 UTC, Steven Schveighoffer
 wrote:

 In that case, you shouldn't be subject to any kind of race
 conditions.
 But we can't make the library/language based on this assumption. Your
 case is the exceptional case.


 His case is not  safe, so we can ignore that problem.


 the compiler tends to disagree:

 =3D=3D=3D test.d =3D=3D=3D
 int myglobal;

 class A {
    ~this ()  safe { if (myglobal =3D=3D 42) assert(0); }
 }

 void main () {
    auto a =3D new A;
 }
 =3D=3D=3D=3D=3D=3D

 dmd -w -c -o- test.d

 wow! no warnings, no errors!


 I think in this case, the compiler can probably statically reject the
 code.


it can't, 'cause it doesn't know how the given object is going to be=20
allocated. i may completely skip GC and use my own allocator, which has=20
no troubles with threads, for example.=
Mar 02 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/2/15 10:38 PM, ketmar wrote:

 On Mon, 02 Mar 2015 15:43:42 -0500, Steven Schveighoffer wrote:


 I think in this case, the compiler can probably statically reject the
 code.


 it can't, 'cause it doesn't know how the given object is going to be
 allocated. i may completely skip GC and use my own allocator, which has
 no troubles with threads, for example.


Classes are expected to be GC allocated. But the point is moot, because 
statically rejecting this does not guarantee abuse will not happen. And 
I don't think this restriction should be enshrined in the language 
anyway -- it's quite possible to write a GC that behaves correctly here.

-Steve
Mar 03 2015
parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 03 Mar 2015 08:53:03 -0500, Steven Schveighoffer wrote:


 Classes are expected to be GC allocated.


O_O

classes are expected to be heap-allocated, methinks, and heap is not=20
necessary garbage collected.


 it's quite possible to write a GC that behaves correctly here.


it's *almost* possible, i'd say. or, in another words, it's not possible=20
without turning threads to garbage-collected resource (at least it's not=20
possible without havy and dirty hackery and/or support from compiler=20
itself).=
Mar 03 2015
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 28 February 2015 at 04:18:38 UTC, ketmar wrote:

 His case is not  safe, so we can ignore that problem.


 the compiler tends to disagree:

 === test.d ===
 int myglobal;

 class A {
   ~this ()  safe { if (myglobal == 42) assert(0); }
 }

 void main () {
   auto a = new A;
 }
 ======

 dmd -w -c -o- test.d

 wow! no warnings, no errors!


And what's the problem?
Mar 03 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/3/15 9:22 AM, Kagamin wrote:

 On Saturday, 28 February 2015 at 04:18:38 UTC, ketmar wrote:

 His case is not  safe, so we can ignore that problem.


 the compiler tends to disagree:

 === test.d ===
 int myglobal;

 class A {
   ~this ()  safe { if (myglobal == 42) assert(0); }
 }

 void main () {
   auto a = new A;
 }
 ======

 dmd -w -c -o- test.d

 wow! no warnings, no errors!


 And what's the problem?


This:

void main()
{
    myglobal = 42;
    spawnThread((){myglobal = 0; auto a = new A; a = null; while(1){}});

    // trigger assert in A.dtor.
    int[] mem;
    while(1)
    {
       mem.length += 10000;
    }
}

-Steve
Mar 03 2015
parent reply "Kagamin" <spam here.lot> writes:
On Tuesday, 3 March 2015 at 14:42:40 UTC, Steven Schveighoffer 
wrote:

    // trigger assert in A.dtor.


You mean assert breaks something?
Mar 03 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/3/15 9:57 AM, Kagamin wrote:

 On Tuesday, 3 March 2015 at 14:42:40 UTC, Steven Schveighoffer wrote:

    // trigger assert in A.dtor.


 You mean assert breaks something?


OK, if you want to be pedantic:

int *myglobal;

repeat same code, but access what myglobal points to instead of setting 
to 42. Memory corruption is what  safe code is trying to prevent. This 
would lead to memory corruption.

-Steve
Mar 03 2015
parent reply "Kagamin" <spam here.lot> writes:
On Tuesday, 3 March 2015 at 18:38:11 UTC, Steven Schveighoffer 
wrote:

 int *myglobal;

 repeat same code, but access what myglobal points to instead of 
 setting to 42. Memory corruption is what  safe code is trying 
 to prevent. This would lead to memory corruption.


You mean, the class destructor accesses a GC-allocated memory? 
That's indeed unsafe, but unrelated to globals.
Mar 03 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/4/15 2:26 AM, Kagamin wrote:

 On Tuesday, 3 March 2015 at 18:38:11 UTC, Steven Schveighoffer wrote:

 int *myglobal;

 repeat same code, but access what myglobal points to instead of
 setting to 42. Memory corruption is what  safe code is trying to
 prevent. This would lead to memory corruption.


 You mean, the class destructor accesses a GC-allocated memory? That's
 indeed unsafe, but unrelated to globals.


Wow, you really aren't getting it. I give up, it's tiring.

-Steve
Mar 04 2015
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 4 March 2015 at 15:39:43 UTC, Steven Schveighoffer 
wrote:

 Wow, you really aren't getting it. I give up, it's tiring.


Tiring? Did it take that much effort to not say what you mean?
Mar 04 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/4/15 11:02 AM, Kagamin wrote:

 On Wednesday, 4 March 2015 at 15:39:43 UTC, Steven Schveighoffer wrote:

 Wow, you really aren't getting it. I give up, it's tiring.


 Tiring? Did it take that much effort to not say what you mean?


No tiring like it is to have to explain obvious details to a child.

In any case, either you are being intentionally obtuse or really 
ignorant. Either way, I don't have any desire to continue with this 
conversation.

-Steve
Mar 04 2015
parent "Kagamin" <spam here.lot> writes:
On Wednesday, 4 March 2015 at 17:12:38 UTC, Steven Schveighoffer 
wrote:

 No tiring like it is to have to explain obvious details to a 
 child.


Safety system is only partially proactive in a sense it doesn't 
cover all possible scenarios, of one finds a bug, he provides an 
example, illustrating the bug (did int myglobal do it? if it was 
a child thing, why it was got wrong?). And scenarios tend to be 
very non-obvious, especially if one speaks about yet unseen bug. 
How an unknown bug can be obvious? And you believe, that 
expressing yourself in Nostradamus style greatly helps make 
things obvious? What things in safety system are on a child level 
at all? You should have a longer way to go down to a child 
argument.
Mar 05 2015
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 03 Mar 2015 14:22:15 +0000, Kagamin wrote:


 On Saturday, 28 February 2015 at 04:18:38 UTC, ketmar wrote:

 His case is not  safe, so we can ignore that problem.


 the compiler tends to disagree:

 =3D=3D=3D test.d =3D=3D=3D
 int myglobal;

 class A {
   ~this ()  safe { if (myglobal =3D=3D 42) assert(0); }
 }

 void main () {
   auto a =3D new A;
 }
 =3D=3D=3D=3D=3D=3D

 dmd -w -c -o- test.d

 wow! no warnings, no errors!


=20
 And what's the problem?


the answer is in your quotation.=
Mar 03 2015
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 26 February 2015 at 23:04:28 UTC, Steven 
Schveighoffer wrote:

 On 2/26/15 4:40 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 On Thursday, 26 February 2015 at 18:08:28 UTC, Steven 
 Schveighoffer wrote:

 On 2/26/15 12:56 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?=
 <schuetzm gmx.net>" wrote:

 On Thursday, 26 February 2015 at 16:51:30 UTC, Steven 
 Schveighoffer
 wrote:

 As talked about before, running dtors in the originating 
 thread can
 solve this problem.


 This probably implies forcibly destroying objects whose 
 creating thread
 terminated.


 I don't think so, those objects can just be destroyed by the
 GC-collection running thread. If the thread is no longer 
 present,
 there can't be a race.


 That's true. However, what if the destructors access global
 (thread-local) variables? Is that already disallowed?


 Hm... I don't know. Is it "disallowed?" I don't think so (a 
 simple test would suffice), but if any code exists today that 
 does it, it's very very wrong :) I would suspect that such code 
 should be banned.


By "disallowed", I indeed mean "banned". Unfortunately, something 
that's just banned will sooner or later be used by someone, 
unless it's also statically prevented by the compiler.
Feb 27 2015
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 12:56 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Thursday, 26 February 2015 at 16:51:30 UTC, Steven Schveighoffer wrote:

 As talked about before, running dtors in the originating thread can
 solve this problem.


 This probably implies forcibly destroying objects whose creating thread
 terminated.


However, this has to be done carefully. For instance, you have to run 
dtors on next allocation, or when thread terminates, because if you do 
it inside stop-the-world signal handler, you still potentially have a race.

-Steve
Feb 26 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 5:17 AM, Michel Fortin wrote:

 On 2015-02-23 22:15:46 +0000, Walter Bright said:


 int* count;

 [...] if (count && --*count == 0) [...]


 Careful!

 This isn't memory safe and you have to thank the GC for it. If you ever
 use RCArray as a member variable in a class, the RCArray destructor is
 going to be called from a random thread when the class destructor is
 run. If some thread has a stack reference to the array you have a race.

 You have to use an atomic counter unless you can prove the RCArray
 struct will never be put in a GC-managed context. It is rather sad that
 the language has no way to enforce such a restriction, and also that
  safe cannot detect that this is a problem here.


Good point. This is already a danger in existing code, e.g. File uses 
reference counting. I just submitted 
https://issues.dlang.org/show_bug.cgi?id=14221.

Andrei
Feb 24 2015
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright wrote:

 struct RCArray(E) {

     this(E[] a)
     {
         array = a.dup;
         start = 0;
         end = a.length;
         count = new int;
         *count = 1;
     }


This may not be  safe depending on the type of E. Also, this do 
not really solve the garbage problem


     ~this()
     {
         if (count && --*count == 0)
             delete array;
     }

     this(this)
     {
         if (count)
             ++*count;
     }


Here, we are going to run a bunch of null check because we can't 
enforce proper construction of struct. This is not per se a 
limitation of this piece of code, simply a long standing issue 
that should up.

Also, there are no way to ensure that the array is going to be 
bound to one thread, even without shared (exception, delegate, 
destructor, pure return). You need at least atomic increment 
and/or decrement.

But worse, as this can be assigned to static variables, you need 
to lock the whole damn struct unless you can update it 
atomically. The struct is 320 bits large and it is not updatable 
atomically in any arch I know of.

You need a mutex.


     size_t length()
     {
         return end - start;
     }

     ref E opIndex(size_t i) return // here's the magic
     {
         return array[start + i];
     }


I have to admit, that is pretty cool. But it is only gonna work 
with value types.


     RCArray opSlice(size_t lwr, size_t upr)
     {
         RCArray result = this;
         result.start = start + lwr;
         result.end = start + upr;
         return result;
     }


You need bound check: end <= start <= array.length .


   private:
     E[] array;
     size_t start, end;
     int* count;
 }
Feb 24 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 2:27 PM, deadalnix wrote:

 This may not be  safe depending on the type of E.


The code is a template, and so relies on inference.



 Also, this do not really solve the garbage problem


Steven pointed out the correct fix for that.



 Also, there are no way to ensure that the array is going to be bound to one
 thread, even without shared (exception, delegate, destructor, pure return). You
 need at least atomic increment and/or decrement.


No. The correct solution is to fix any cases of implicit sharing. If they don't 
have bugzilla issues, file them.



 I have to admit, that is pretty cool. But it is only gonna work with value
types.


I don't see that limitation at all.



 You need bound check: end <= start <= array.length


See the antecedent:

"More could be done:
[...]
5. bounds checking"
Feb 24 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 25 February 2015 at 00:20:25 UTC, Walter Bright 
wrote:

 On 2/24/2015 2:27 PM, deadalnix wrote:

 I have to admit, that is pretty cool. But it is only gonna 
 work with value types.


 I don't see that limitation at all.


Just try to do the same with opSlice() and you will see what he 
means.
Feb 25 2015
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright wrote:

 struct RCArray(E) {

     this(E[] a)
     {
         array = a.dup;
         start = 0;
         end = a.length;
         count = new int;
         *count = 1;
     }


This may not be  safe depending on the type of E. Also, this do 
not really solve the garbage problem as you need a GCed slice to 
build the RCed slice. That mean that without a nicer way to 
construct this, this is not solving any problem.

Also, I guess that even if you use the GC in there, you want to 
make sure that the function is  nogc .


     ~this()
     {
         if (count && --*count == 0)
             delete array;
     }

     this(this)
     {
         if (count)
             ++*count;
     }


Here, we are going to run a bunch of null check because we can't 
enforce proper construction of struct. This is not per se a 
limitation of this piece of code, simply a long standing issue 
that should up.

Also, there are no way to ensure that the array is going to be 
bound to one thread, even without shared (exception, delegate, 
destructor, pure return). You need at least atomic increment 
and/or decrement.

But worse, as this can be assigned to static variables, you need 
to lock the whole damn struct unless you can update it 
atomically. The struct is 320 bits large and it is not updatable 
atomically in any arch I know of.

You need a mutex.


     size_t length()
     {
         return end - start;
     }

     ref E opIndex(size_t i) return // here's the magic
     {
         return array[start + i];
     }


I have to admit, that is pretty cool. But it is only gonna work 
with value types.


     RCArray opSlice(size_t lwr, size_t upr)
     {
         RCArray result = this;
         result.start = start + lwr;
         result.end = start + upr;
         return result;
     }


You need bound check: end <= start <= array.length .


   private:
     E[] array;
     size_t start, end;
     int* count;
 }


Generally, this need support for other array operations, is not 
usable in  nogc, has threading issues, but this is a cool thing 
you can do with "ref return".
Feb 24 2015
prev sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright wrote:

   private:
     E[] array;
     size_t start, end;
     int* count;


What is the point of keeping start/end? Aren't those baked into 
the array slice? Not storing start end means not having to do 
index arithmetic (minor), reducing struct size (always nice). But 
more importantly, it allows implicit (and conditional) bounds 
checking (awesome), which actually runs regardless anyways.

Or did I miss something?
Mar 05 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 5 March 2015 at 15:20:47 UTC, monarch_dodra wrote:

 On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright 
 wrote:

  private:
    E[] array;
    size_t start, end;
    int* count;


 What is the point of keeping start/end? Aren't those baked into 
 the array slice? Not storing start end means not having to do 
 index arithmetic (minor), reducing struct size (always nice). 
 But more importantly, it allows implicit (and conditional) 
 bounds checking (awesome), which actually runs regardless 
 anyways.

 Or did I miss something?


`GC.free()` needs a pointer to the start of the allocated block; 
it will not release memory if it gets an interior pointer. But as 
far as I can see, one pointer that stores the original address 
should be enough.
Mar 05 2015
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 5 March 2015 at 16:19:09 UTC, Marc Schütz wrote:

 On Thursday, 5 March 2015 at 15:20:47 UTC, monarch_dodra wrote:

 On Monday, 23 February 2015 at 22:15:54 UTC, Walter Bright 
 wrote:

 private:
   E[] array;
   size_t start, end;
   int* count;


 What is the point of keeping start/end? Aren't those baked 
 into the array slice? Not storing start end means not having 
 to do index arithmetic (minor), reducing struct size (always 
 nice). But more importantly, it allows implicit (and 
 conditional) bounds checking (awesome), which actually runs 
 regardless anyways.

 Or did I miss something?


 `GC.free()` needs a pointer to the start of the allocated 
 block; it will not release memory if it gets an interior 
 pointer. But as far as I can see, one pointer that stores the 
 original address should be enough.


Still, you shouldn't need "end", and bounds checking would "just
work".
Mar 05 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/5/15 8:50 AM, monarch_dodra wrote:

 Still, you shouldn't need "end", and bounds checking would "just
 work".


Correct. -- Andrei
Mar 05 2015