www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - weak references

reply Brian Hammond <d brianhammond.com> writes:
Does (or will) D support weak references?

See http://www.python.org/doc/current/lib/module-weakref.html for a
backgrounder.

Thanks!
May 07 2004
next sibling parent reply Ilya Minkov <minkov cs.tum.edu> writes:
Brian Hammond schrieb:
 Does (or will) D support weak references?
 
 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.

At a language level, it is technically not very feasible, since D has to be able to work with conservative garbage collectors, as well as other automatic memory management systems. In Python, it happens to be well implementable because memory is managed by reference- counting garbage collector. But for statically compiled languages, reference-counting has proven to be a major slowdown, expecially when it has to be thread-safe. There should be other ways to implement caches and such in D. Normally you would have something like a "central repository" for objects of some sort, and you "request" them and "put back", basically doing sort-of manual reference counting, which is though considerably much less overhead than automatic one. There might also be some other ways, e.g. by mangling pointers so that garbage collector doesn't find them - but then the object must maintain a complete list of its users. A library implementation is though definately more tedious to use, would have another advantage performance-wise: it may use a smarter clean-up strategy than a garbage collector and work asynchronously compared to it, thus reducing pauses. -eye
May 13 2004
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Ilya Minkov wrote:

<snip>
 But for statically compiled languages, reference-counting has 
 proven to be a major slowdown, expecially when it has to be thread-safe.

And a cause of memory leaks, when you try to use circular data structures.
 There should be other ways to implement caches and such in D. Normally 
 you would have something like a "central repository" for objects of some 
 sort, and you "request" them and "put back", basically doing sort-of 
 manual reference counting, which is though considerably much less 
 overhead than automatic one. There might also be some other ways, e.g. 
 by mangling pointers so that garbage collector doesn't find them - but 
 then the object must maintain a complete list of its users.

I thought of writing a class that would do something like that. The trouble is that we'd need a means of determining if an object is reachable by strong references. The only way we can really get near doing this is to have a common base class for weak-referenceable objects, which would clear the references in its destructor. A compromise would be to create an object wrapper enabling something to be WR'd, but that might lead to WRs being cleared while the object is still active. But with the base class proviso, a possibility is to assign a serial number to each object when it is first WR'd, and have the WR object hold this number. There would be a static mapping table of some sort between serial numbers and mangled pointers. When it's time to clear the WRs, it would simply set the pointer in the table to null. The only time you're going to run into trouble is if you WR more objects in the course of your program than the type of the serial number'll allow. Stewart. -- My e-mail is valid but not my primary mailbox, aside from its being the unfortunate victim of intensive mail-bombing at the moment. Please keep replies on the 'group where everyone may benefit.
May 13 2004
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
In article <c80726$1qq$1 digitaldaemon.com>, Stewart Gordon says...
..
I thought of writing a class that would do something like that.  The 
trouble is that we'd need a means of determining if an object is 
reachable by strong references.  The only way we can really get near 
doing this is to have a common base class for weak-referenceable 
objects, which would clear the references in its destructor.  A 
compromise would be to create an object wrapper enabling something to be 
WR'd, but that might lead to WRs being cleared while the object is still 
active.

The boehm-wheeler gc (and probably most others) has a provision to add non-scanned areas. (In D, malloced space is non-scanned, right?) If you add one of these areas (needs some language support) and put an array of weak pointers there, then the only piece missing is a way for the GC to clear pointers (from that area) for objects that it collects. So: Weak refs are a tiny 'smart pointer' like object that points at a slot in the table (or indexes into the unscanned space). That slot has a non-scanned pointer to the real object. When the GC is done reaping the heap, it does one last mark scan: it runs over the weak ref table: instead of marking objects as "not garbage", it just zeroes the weak pointers to them. If the object is already marked as "not garbage" it leaves the pointer alone (there must be a strong ref). - Extra accesses are needed to use weak refs (its a pointer to a pointer). - This is okay, because weak pointers are rarely used except for this kind of caching. Once you fetch the regular pointer, its as fast as any other. (Side note: The standard application for weak references (caching) seems a bit "small" to me. It seems like the number of objects kept in-cache should be performance tuned, not cleared every time the GC gets hungry, right? You can keep a fixed number of regular pointers, plus the weak set; but the weak set only helps for cache elements that are past the end of this initial size. The non-weak portion of the cache is probably as large as you can afford to begin with, so..) Kevin
May 13 2004
parent Ben Hinkle <bhinkle4 juno.com> writes:
 (Side note: The standard application for weak references (caching) seems a
 bit
 "small" to me.  It seems like the number of objects kept in-cache should
 be
 performance tuned, not cleared every time the GC gets hungry, right?  You
 can keep a fixed number of regular pointers, plus the weak set; but the
 weak set
 only helps for cache elements that are past the end of this initial size. 
 The non-weak portion of the cache is probably as large as you can afford
 to begin with, so..)

I agree - I've only seen weak references used to manage caches. It would be neat to have GC callbacks where an object can register a delegate that runs when the GC is about to full-collect. That way a cache can clear itself out (or clear out the extra elements of the cache) without needing weak references. I guess you'd have to be careful not to leak the cache since the GC would then have a reference to it. -Ben
May 14 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...
 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.

Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references. So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.
May 14 2004
parent reply Brian Hammond <d at brianhammond dot com> <Brian_member pathlink.com> writes:
Thanks for the response.  I was thinking of weakrefs as an exploitable feature
for a resource manager I am writing for a game engine in D.  

Weak refs seemed like a good fit for this problem.  The resource manager
maintains a cache of weak references to resource instances.  The manager knows
they exist but doesn't stop the gc of such from happening by holding (strong)
references to the resource instances.

Thanks, Brian

In article <c8440l$2j3e$1 digitaldaemon.com>, Walter says...
"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...
 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.

Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references. So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.

May 16 2004
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
I would think you can get essentially the same effect in D via destructors:

class Cache {
..
delete(int key) {...}
};

class GCwatch {
int key;
Cache cache;

this(int k, Cache c)
{
key = k;
cache = c;
}

~this()
{
Cache.delete(key);
}
};

add_weak_elem()
{
Cache c = new Cache;
int k = c.get_weak_elem();

new GCwatch(c,k);
..
}

The GCwatch is not held by a strong pointer, so the GC will delete it, causing
Cache cleanups to happen at the same time as GC cleanups.

Kevin

In article <c896ua$quh$1 digitaldaemon.com>, Brian Hammond <d at brianhammond
dot com> says...
Thanks for the response.  I was thinking of weakrefs as an exploitable feature
for a resource manager I am writing for a game engine in D.  

Weak refs seemed like a good fit for this problem.  The resource manager
maintains a cache of weak references to resource instances.  The manager knows
they exist but doesn't stop the gc of such from happening by holding (strong)
references to the resource instances.

Thanks, Brian

In article <c8440l$2j3e$1 digitaldaemon.com>, Walter says...
"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...
 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.

Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references. So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.


May 17 2004
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
I should amend this: the cached object needs a pointer to the GCwatch, to
prevent it from being deleted when a strong pointer exists, and the Cache needs
to disguise the pointer to the element.  Also the method should not be called
"delete".

Kevin

In article <c8ambr$2lq$1 digitaldaemon.com>, Kevin Bealer says...
I would think you can get essentially the same effect in D via destructors:

class Cache {
..
delete(int key) {...}
};

class GCwatch {
int key;
Cache cache;

this(int k, Cache c)
{
key = k;
cache = c;
}

~this()
{
Cache.delete(key);
}
};

add_weak_elem()
{
Cache c = new Cache;
int k = c.get_weak_elem();

new GCwatch(c,k);
..
}

The GCwatch is not held by a strong pointer, so the GC will delete it, causing
Cache cleanups to happen at the same time as GC cleanups.

Kevin

In article <c896ua$quh$1 digitaldaemon.com>, Brian Hammond <d at brianhammond
dot com> says...
Thanks for the response.  I was thinking of weakrefs as an exploitable feature
for a resource manager I am writing for a game engine in D.  

Weak refs seemed like a good fit for this problem.  The resource manager
maintains a cache of weak references to resource instances.  The manager knows
they exist but doesn't stop the gc of such from happening by holding (strong)
references to the resource instances.

Thanks, Brian

In article <c8440l$2j3e$1 digitaldaemon.com>, Walter says...
"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...
 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.

Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references. So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.



May 17 2004
parent reply Brian Hammond <d at brianhammond dot com> <Brian_member pathlink.com> writes:
Your idea is fine it's just manual.  I was mostly curious if my issue could be
solved using weakrefs in D.  As has been pointed out, weakrefs don't apply well
to D.  Thus, to implement this, the monitored object must notify interested
parties (say a cache) that it's been collected (notification is done in the
destructor). 

In article <c8ampu$3c0$1 digitaldaemon.com>, Kevin Bealer says...
I should amend this: the cached object needs a pointer to the GCwatch, to
prevent it from being deleted when a strong pointer exists, and the Cache needs
to disguise the pointer to the element.  Also the method should not be called
"delete".

Kevin

Keeping an explicit pointer won't keep the object alive if there are no references to it and the GC runs. E.g. import std.gc; class A { this() { printf("A.this() called.\n"); } ~this() { printf("A.~this() called.\n"); } } int main(char[][] args) { printf("creating a and pointing b at a\n"); A a = new A(); A* b = &a; printf("b's target object is %.*s\n", *b ? "not null" : "null"); printf("Nullifying a...\n"); a = null; printf("b's target object is %.*s\n", *b ? "not null" : "null"); printf("Running gc...\n"); std.gc.fullCollect(); printf("A has %.*s collected\n", !a ? "been" : "not been"); printf("b's target object is %.*s\n", *b ? "not null" : "null"); printf("end.\n"); return 0; } output: creating a and pointing b at a A.this() called. b's target object is not null Nullifying a... b's target object is null Running gc... A.~this() called. A has been collected b's target object is null end. Thanks! Brian
In article <c8ambr$2lq$1 digitaldaemon.com>, Kevin Bealer says...
I would think you can get essentially the same effect in D via destructors:

class Cache {
..
delete(int key) {...}
};

class GCwatch {
int key;
Cache cache;

this(int k, Cache c)
{
key = k;
cache = c;
}

~this()
{
Cache.delete(key);
}
};

add_weak_elem()
{
Cache c = new Cache;
int k = c.get_weak_elem();

new GCwatch(c,k);
..
}

The GCwatch is not held by a strong pointer, so the GC will delete it, causing
Cache cleanups to happen at the same time as GC cleanups.

Kevin

In article <c896ua$quh$1 digitaldaemon.com>, Brian Hammond <d at brianhammond
dot com> says...
Thanks for the response.  I was thinking of weakrefs as an exploitable feature
for a resource manager I am writing for a game engine in D.  

Weak refs seemed like a good fit for this problem.  The resource manager
maintains a cache of weak references to resource instances.  The manager knows
they exist but doesn't stop the gc of such from happening by holding (strong)
references to the resource instances.

Thanks, Brian

In article <c8440l$2j3e$1 digitaldaemon.com>, Walter says...
"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...
 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.

Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references. So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.




May 17 2004
parent Kevin Bealer <Kevin_member pathlink.com> writes:
In article <c8b8mj$v82$1 digitaldaemon.com>, Brian Hammond <d at brianhammond
dot com> says...
Your idea is fine it's just manual.  I was mostly curious if my issue could be
solved using weakrefs in D.  As has been pointed out, weakrefs don't apply well
to D.  Thus, to implement this, the monitored object must notify interested
parties (say a cache) that it's been collected (notification is done in the
destructor). 

My original thought was to use this technique to create a generic "weak pointer" object. But I have had a realization w.r.t finalization: weak pointers won't work properly from a "gc-dead" object because the finalizer cannot legally fiddle with sub-objects: they may also be garbage. Kevin
May 17 2004