www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Reference counting for resource management

reply LMB <lmbarros gmail.com> writes:
Hello,

I've been reading the forums for some time, but this is my first post here :-)

I am trying to create yet another D2 wrapper for SQLite. As usual with many C
libraries, SQLite provides us some "objects" and functions to create and
destroy them. So, I decided to encapsulate these SQLite objects in a struct,
and use reference counting to destroys the objects as soon as I can safely do
so.

The problem is that I can't make it work correctly in all situations. It guess
that I am not increasing the reference count on every situation in which my
object is copied, because I got "negative" reference counts in some tests. But
this is just a guess, I am not sure at all.

So, is there any complete example on how to implement reference counting in D2?

I found some discussions about ref counting in D, like Bartoz's nice post
(http://bartoszmilewski.wordpress.com/2009/08/19/the-anatomy-of-r
ference-counting/), but not a complete working example.

If there is no example around, I'd be pleased if someone could give me any
advice. This is what one of my wrappers roughly looks like (for brevity, I
removed all code not related to reference counting):

struct Database
{
   public this(in string fileName)
   {
      sqlite3_open(toStringz(fileName), &db_);

      refCount_ = cast(uint*)(malloc(uint.sizeof));
      *refCount_ = 1;
   }

   this(this)
   body
   {
      ++(*refCount_);
   }

   Database opAssign(Database other)
   {
      db_ = other.db_;
      refCount_ = other.refCount_;
      ++(*refCount_);
      return this;
   }

   public ~this()
   {
      if (refCount_ !is null)
      {
         --(*refCount_);

         if (*refCount_ == 0)
         {
            free(refCount_);
            refCount_ = null;
            immutable status = sqlite3_close(db_);
         }
      }
   }

   private sqlite3* db_;
   private uint* refCount_;
}


Thanks!

LMB
Nov 27 2009
next sibling parent LMB <lmbarros gmail.com> writes:
LMB Wrote:

 Hello,
 
 [...]

Ouch! I thought I was posting to the "learn" group. My bad, sorry!
Nov 27 2009
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
------------ug9J4sIV6F1FVPZ4qTdT0x
Content-Type: text/plain; charset=koi8-r; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sat, 28 Nov 2009 01:27:41 +0300, LMB <lmbarros gmail.com> wrote:

 Hello,

 I've been reading the forums for some time, but this is my first post  
 here :-)

 I am trying to create yet another D2 wrapper for SQLite. As usual with  
 many C libraries, SQLite provides us some "objects" and functions to  
 create and destroy them. So, I decided to encapsulate these SQLite  
 objects in a struct, and use reference counting to destroys the objects  
 as soon as I can safely do so.

 The problem is that I can't make it work correctly in all situations. It  
 guess that I am not increasing the reference count on every situation in  
 which my object is copied, because I got "negative" reference counts in  
 some tests. But this is just a guess, I am not sure at all.

 So, is there any complete example on how to implement reference counting  
 in D2?

 I found some discussions about ref counting in D, like Bartoz's nice  
 post  
 (http://bartoszmilewski.wordpress.com/2009/08/19/the-anatomy-of-r
ference-counting/),  
 but not a complete working example.

 If there is no example around, I'd be pleased if someone could give me  
 any advice. This is what one of my wrappers roughly looks like (for  
 brevity, I removed all code not related to reference counting):

 struct Database
 {
    public this(in string fileName)
    {
       sqlite3_open(toStringz(fileName), &db_);

       refCount_ = cast(uint*)(malloc(uint.sizeof));
       *refCount_ = 1;
    }

    this(this)
    body
    {
       ++(*refCount_);
    }

    Database opAssign(Database other)
    {
       db_ = other.db_;
       refCount_ = other.refCount_;
       ++(*refCount_);
       return this;
    }

    public ~this()
    {
       if (refCount_ !is null)
       {
          --(*refCount_);

          if (*refCount_ == 0)
          {
             free(refCount_);
             refCount_ = null;
             immutable status = sqlite3_close(db_);
          }
       }
    }

    private sqlite3* db_;
    private uint* refCount_;
 }


 Thanks!

 LMB

I attached my implementation of RefCounted pattern. Feel free to use and improve it. Here is an excerpt from a letter I wrote about RefCounted. It covers its design, implementation and usage. There are two templates: RefCounted and RefObject (see implementation in net/core/RefCounted.d). They are not classes/structs but rather templates that are used for being mixed in other classes so that you don't have to inherit from it. RefObject defines the following methods: final public uint referenceCounter() const; final public void addRef(); final public void releaseRef(); (I'll probably also add final public void destroy(); methods that would do nothing). Once you mixin this template into your class, your class instances become reference-counted (explicitly at this moment), so it could be used without the other one (see below for details). Mixins also have useful ability to override default implementation. For example, destroy() would do nothing by default, and rely on garbage-collector to recycle memory instead of calling "delete this;". User may override it and free some resources immediately (e.g. disconnect from remote host) before memory is recycled. Automatic reference-counting is achieved using RefCounted template. It's use is as follows: struct DriverPtr { mixin RefCountedT!(Driver, DriverPtr); } struct LinkPtr { mixin RefCountedT!(Link, LinkPtr); } The reason is, again, flexibility. It is possible to override certain methods when needed. I'll probably also add a default RefCounted class, something like this: struct RefCounted!(T) { mixin RefCountedT!(T, RefCounted!(T)); } Users will be able to use it like this: alias RefCounted!(Link) LinkPtr; There are other also other approaches to implement ref-counting mechanism, but I found this one to be the most flexible one. The two classes are best used together, but you may use one without other, two. RefCountedT relies on the following 3 methods: uint referenceCounter(); void addRef(); void releaseRef(); and it doesn't really care the way they are implemented. The most convenient way is to use mixin RefObject, though. ------------ug9J4sIV6F1FVPZ4qTdT0x Content-Disposition: attachment; filename=RefCounted.d Content-Type: application/octet-stream; name=RefCounted.d Content-Transfer-Encoding: Base64 aW1wb3J0IHRocmVhZC5BdG9taWM7DQoNCnRlbXBsYXRlIFJlZk9iamVjdCgpDQp7 DQoJLy8vDQoJLy8vIE51bWJlciBvZiByZWZlcmVuY2VzDQoJLy8vDQoJZmluYWwg cHVibGljIHVpbnQgcmVmZXJlbmNlQ291bnRlcigpIGNvbnN0IHsgcmV0dXJuIF9y ZWZlcmVuY2VDb3VudGVyOyB9DQoNCglmaW5hbCBwdWJsaWMgdm9pZCBhZGRSZWYo KQ0KCXsNCgkJYXNzZXJ0KF9yZWZlcmVuY2VDb3VudGVyID49IDApOw0KICAgICAg ICB0aHJlYWQuQXRvbWljLmF0b21pY0luY3JlbWVudChfcmVmZXJlbmNlQ291bnRl cik7DQoJfQ0KDQoJZmluYWwgcHVibGljIHZvaWQgcmVsZWFzZVJlZigpDQoJew0K CQlhc3NlcnQoX3JlZmVyZW5jZUNvdW50ZXIgPiAwKTsNCiAgICAgICAgaW50IG9s ZFZhbHVlID0gdGhyZWFkLkF0b21pYy5hdG9taWNEZWNyZW1lbnQoX3JlZmVyZW5j ZUNvdW50ZXIpOw0KICAgICAgICBhc3NlcnQob2xkVmFsdWUgPiAwKTsNCiAgICAg ICAgaWYgKG9sZFZhbHVlID09IDEpIHsNCiAgICAgICAgICAgIC8vIG5vIG1vcmUg cmVmZXJlbmNlcyBsZWZ0DQoJCQkvLyBkZWxldGUgZHJpdmVyIG9yIGxlYXZlIGl0 IHRvIEdDPw0KICAgICAgICB9DQoJfQ0KDQoJcHJpdmF0ZSBpbnQgX3JlZmVyZW5j ZUNvdW50ZXIgPSAwOw0KfQ0KDQp0ZW1wbGF0ZSBSZWZDb3VudGVkVChULCBSZWZD b3VudGVkKQ0Kew0KCXRoaXMoVCBwdHIpDQoJew0KCQlhc3NlcnQocHRyICFpcyBu dWxsKTsNCgkJX3B0ciA9IHB0cjsNCgkJX3B0ci5hZGRSZWYoKTsNCgl9DQoNCgl0 aGlzKHRoaXMpDQoJew0KCQlpZiAoX3B0ciAhaXMgbnVsbCkgew0KCQkJX3B0ci5h ZGRSZWYoKTsNCgkJfQ0KCX0NCg0KCX50aGlzKCkNCgl7DQoJCWlmIChfcHRyICFp cyBudWxsKSB7DQoJCQlfcHRyLnJlbGVhc2VSZWYoKTsNCgkJfQ0KCX0NCg0KCXZv aWQgb3BBc3NpZ24ocmVmIFQgcHRyKQ0KICAgIHsNCiAgICAgICAgaWYgKF9wdHIg IWlzIF9wdHIpDQogICAgICAgIHsNCiAgICAgICAgICAgIGlmIChwdHIgIWlzIG51 bGwpIHsNCiAgICAgICAgICAgICAgICBwdHIuYWRkUmVmKCk7DQogICAgICAgICAg ICB9DQogICAgICAgICAgICBpZiAoX3B0ciAhaXMgbnVsbCkgew0KICAgICAgICAg ICAgICAgIF9wdHIucmVsZWFzZVJlZigpOw0KICAgICAgICAgICAgfQ0KICAgICAg ICAgICAgX3B0ciA9IHB0cjsNCiAgICAgICAgfQ0KICAgIH0NCg0KCXZvaWQgb3BB c3NpZ24ocmVmIFJlZkNvdW50ZWQgb3RoZXIpDQogICAgew0KICAgICAgICBpZiAo X3B0ciAhaXMgb3RoZXIuX3B0cikNCiAgICAgICAgew0KICAgICAgICAgICAgaWYg KG90aGVyLl9wdHIgIWlzIG51bGwpIHsNCiAgICAgICAgICAgICAgICBvdGhlci5f cHRyLmFkZFJlZigpOw0KICAgICAgICAgICAgfQ0KICAgICAgICAgICAgaWYgKF9w dHIgIWlzIG51bGwpIHsNCiAgICAgICAgICAgICAgICBfcHRyLnJlbGVhc2VSZWYo KTsNCiAgICAgICAgICAgIH0NCiAgICAgICAgICAgIF9wdHIgPSBvdGhlci5fcHRy Ow0KICAgICAgICB9DQogICAgfQ0KDQoJYm9vbCBvcEVxdWFscyhjb25zdChUKSBw dHIpIGNvbnN0DQoJew0KCQlyZXR1cm4gX3B0ciBpcyBwdHI7DQoJfQ0KDQogICAg Ym9vbCBvcEVxdWFscyhyZWYgY29uc3QoUmVmQ291bnRlZCkgb3RoZXIpIGNvbnN0 DQogICAgew0KICAgICAgICByZXR1cm4gX3B0ciBpcyBvdGhlci5fcHRyOw0KICAg IH0NCg0KCVQgZ2V0KCkNCgl7DQoJCXJldHVybiBfcHRyOw0KCX0NCg0KCWFsaWFz IGdldCB0aGlzOw0KDQoJLy8vDQogICAgLy8vIFJlbGVhc2VzIG9iamVjdCAoaWYg ZXhpc3RzKQ0KICAgIC8vLw0KICAgIHZvaWQgcmVzZXQoKQ0KICAgIHsNCiAgICAg ICAgaWYgKF9wdHIgIWlzIG51bGwpIHsNCiAgICAgICAgICAgIF9wdHIucmVsZWFz ZVJlZigpOw0KICAgICAgICB9DQoNCiAgICAgICAgX3B0ciA9IG51bGw7DQogICAg fQ0KDQoJcHJvdGVjdGVkIFQgX3B0ciA9IG51bGw7DQp9DQo= ------------ug9J4sIV6F1FVPZ4qTdT0x--
Nov 27 2009
prev sibling parent reply Bartosz Milewski <bartosz-nospam relisoft.com> writes:
I don't think you need opAssign. The compiler will automatically use bitblit
and postblit. 

Here's my implementation of a reference counted thread id structure for
comparison:

struct Tid
{
    this(HANDLE h) 
    {
        _h = h;
        Counter * cnt = cast(Counter *) core.stdc.stdlib.malloc(Counter.sizeof);
        cnt._n = 1;
        _cnt = cast(shared Counter *) cnt;
    }
    ~this()
    {
        release();
    }
    this(this)
    {
        _cnt.inc;
    }
    // invalidates current Tid object
    Tid transfer()
    {
        Tid tid = { _h, _cnt };
        _cnt = null;
        return tid;
    }
    void release()
    {
        if (_cnt !is null)
        {
            uint newCount = _cnt.dec;
            if (newCount == 0)
            {
                CloseHandle(_h);
                core.stdc.stdlib.free(cast(Counter *)_cnt);
                _cnt = null;
            }
        }
    }
    void start()
    {
        assert(_h != INVALID_HANDLE_VALUE);
        if (ResumeThread(_h) == -1)
            throw new ThreadException("Error resuming thread");
    }
    void join(bool rethrow = true) 
    {
        if (_h != INVALID_HANDLE_VALUE)
            if (WaitForSingleObject(_h, INFINITE) != WAIT_OBJECT_0)
                throw new ThreadException("Join failed");
    }
    void setPriority() // dummy for now
    {
    }
private:
    // Revisit: implement using atomic add
    struct Counter
    {
        uint inc() shared { return ++_n; }
        uint dec() shared { return --_n; }

        uint _n = 1;
    }
private:
    HANDLE             _h = INVALID_HANDLE_VALUE;
    shared Counter * _cnt;
}

LMB Wrote:

 Hello,
 
 I've been reading the forums for some time, but this is my first post here :-)
 
 I am trying to create yet another D2 wrapper for SQLite. As usual with many C
libraries, SQLite provides us some "objects" and functions to create and
destroy them. So, I decided to encapsulate these SQLite objects in a struct,
and use reference counting to destroys the objects as soon as I can safely do
so.
 
 The problem is that I can't make it work correctly in all situations. It guess
that I am not increasing the reference count on every situation in which my
object is copied, because I got "negative" reference counts in some tests. But
this is just a guess, I am not sure at all.
 
 So, is there any complete example on how to implement reference counting in D2?
 
 I found some discussions about ref counting in D, like Bartoz's nice post
(http://bartoszmilewski.wordpress.com/2009/08/19/the-anatomy-of-r
ference-counting/), but not a complete working example.
 
 If there is no example around, I'd be pleased if someone could give me any
advice. This is what one of my wrappers roughly looks like (for brevity, I
removed all code not related to reference counting):
 
 struct Database
 {
    public this(in string fileName)
    {
       sqlite3_open(toStringz(fileName), &db_);
 
       refCount_ = cast(uint*)(malloc(uint.sizeof));
       *refCount_ = 1;
    }
 
    this(this)
    body
    {
       ++(*refCount_);
    }
 
    Database opAssign(Database other)
    {
       db_ = other.db_;
       refCount_ = other.refCount_;
       ++(*refCount_);
       return this;
    }
 
    public ~this()
    {
       if (refCount_ !is null)
       {
          --(*refCount_);
 
          if (*refCount_ == 0)
          {
             free(refCount_);
             refCount_ = null;
             immutable status = sqlite3_close(db_);
          }
       }
    }
 
    private sqlite3* db_;
    private uint* refCount_;
 }
 
 
 Thanks!
 
 LMB
 

Nov 29 2009
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Mon, 30 Nov 2009 03:19:33 +0300, Bartosz Milewski  
<bartosz-nospam relisoft.com> wrote:

 I don't think you need opAssign. The compiler will automatically use  
 bitblit and postblit.

 Here's my implementation of a reference counted thread id structure for  
 comparison:

 struct Tid
 {
     this(HANDLE h)
     {
         _h = h;
         Counter * cnt = cast(Counter *)  
 core.stdc.stdlib.malloc(Counter.sizeof);
         cnt._n = 1;
         _cnt = cast(shared Counter *) cnt;
     }
     ~this()
     {
         release();
     }
     this(this)
     {
         _cnt.inc;
     }
     // invalidates current Tid object
     Tid transfer()
     {
         Tid tid = { _h, _cnt };
         _cnt = null;
         return tid;
     }
     void release()
     {
         if (_cnt !is null)
         {
             uint newCount = _cnt.dec;
             if (newCount == 0)
             {
                 CloseHandle(_h);
                 core.stdc.stdlib.free(cast(Counter *)_cnt);
                 _cnt = null;
             }
         }
     }
     void start()
     {
         assert(_h != INVALID_HANDLE_VALUE);
         if (ResumeThread(_h) == -1)
             throw new ThreadException("Error resuming thread");
     }
     void join(bool rethrow = true)
     {
         if (_h != INVALID_HANDLE_VALUE)
             if (WaitForSingleObject(_h, INFINITE) != WAIT_OBJECT_0)
                 throw new ThreadException("Join failed");
     }
     void setPriority() // dummy for now
     {
     }
 private:
     // Revisit: implement using atomic add
     struct Counter
     {
         uint inc() shared { return ++_n; }
         uint dec() shared { return --_n; }

         uint _n = 1;
     }
 private:
     HANDLE             _h = INVALID_HANDLE_VALUE;
     shared Counter * _cnt;
 }

 LMB Wrote:

 Hello,

 I've been reading the forums for some time, but this is my first post  
 here :-)

 I am trying to create yet another D2 wrapper for SQLite. As usual with  
 many C libraries, SQLite provides us some "objects" and functions to  
 create and destroy them. So, I decided to encapsulate these SQLite  
 objects in a struct, and use reference counting to destroys the objects  
 as soon as I can safely do so.

 The problem is that I can't make it work correctly in all situations.  
 It guess that I am not increasing the reference count on every  
 situation in which my object is copied, because I got "negative"  
 reference counts in some tests. But this is just a guess, I am not sure  
 at all.

 So, is there any complete example on how to implement reference  
 counting in D2?

 I found some discussions about ref counting in D, like Bartoz's nice  
 post  
 (http://bartoszmilewski.wordpress.com/2009/08/19/the-anatomy-of-r
ference-counting/),  
 but not a complete working example.

 If there is no example around, I'd be pleased if someone could give me  
 any advice. This is what one of my wrappers roughly looks like (for  
 brevity, I removed all code not related to reference counting):

 struct Database
 {
    public this(in string fileName)
    {
       sqlite3_open(toStringz(fileName), &db_);

       refCount_ = cast(uint*)(malloc(uint.sizeof));
       *refCount_ = 1;
    }

    this(this)
    body
    {
       ++(*refCount_);
    }

    Database opAssign(Database other)
    {
       db_ = other.db_;
       refCount_ = other.refCount_;
       ++(*refCount_);
       return this;
    }

    public ~this()
    {
       if (refCount_ !is null)
       {
          --(*refCount_);

          if (*refCount_ == 0)
          {
             free(refCount_);
             refCount_ = null;
             immutable status = sqlite3_close(db_);
          }
       }
    }

    private sqlite3* db_;
    private uint* refCount_;
 }


 Thanks!

 LMB


I think RefCounted should be flexible enough to re-usable for other data structures, too (e.g. so that it could be part of Phobos). By the way, your code has a bug: Tid tid; // or release() and exisiting Tid Tid tid2 = tid; // segfault since you don't check if _cnt is null in postblit
Nov 29 2009
parent Bartosz Milewski <bartosz-nospam relisoft.com> writes:
Denis Koroskin Wrote:

 I think RefCounted should be flexible enough to re-usable for other data  
 structures, too (e.g. so that it could be part of Phobos).
 

I agree. However this code goes into core, so it can't use Phobos.
 By the way, your code has a bug:
 
 Tid tid; // or release() and exisiting Tid
 Tid tid2 = tid; // segfault since you don't check if _cnt is null in  
 postblit

I realize that, but I'm not sure if this would be a library bug or a user bug. I wish one could just prevent such usage, e.g., by nulling the default constructor: struct Tid { this() = null; ... } This is a more general problem of the absence of user-defined default constructors for structs. This makes perfect sense for PODS, but not for structs that have a destructor (and postblit). Of course, if the default constructor is blocked, you wouldn't be able to have arrays of such structs, which, as a side effect, would eliminate the problem of exceptions thrown from default constructors and array unwinding. It's a bit of a minefield there.
Nov 30 2009