www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The GC, destructors, exceptions and memory corruption

reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
Hi,

A while ago, I've tracked down the cause of an insidious memory corruption  
problem in one of my D programs. The problem was caused by an inadvertent  
allocation in a destructor (called by the GC). The current GC  
implementation is completely unprepared to handle such a situation - an  
allocation during a GC run will break the GC's invariants, and will  
ultimately result in memory corruption. I've filed this problem as issue  
5653.

I've created a simple test case which illustrates the problem:

//////////////////////////////////////////////////////////////////////////////
const message = "Hello, world!";

char[] s = null;

class C
{
	~this()
	{
		s = message.dup;
	}
}

version(D_Version2)
	import core.memory;
else
	import std.gc;

void main()
{
	C c;
	c = new C();
	c = new C(); // clobber any references to first instance

	version(D_Version2)
		GC.collect();
	else
		fullCollect();

	assert(s !is null, "Destructor wasn't called");
	assert(s == message, "Memory was corrupted");
}
//////////////////////////////////////////////////////////////////////////////

The exact reason the above program corrupts memory is that .dup will  
allocate memory by taking an item from a free list. However, after the  
destructor returns, the GC continues on to rebuild the free list with the  
information it had before the .dup allocation. The first machine word of  
the allocated region will be overwritten with a pointer to the next item  
in the free list.

I wrote a patch to the D1 GC to forbid allocations from destructors, and  
was considering to port it to D2 and wrap it in a pull request, but  
realized that my patch breaks the GC in case a destructor throws. However,  
looking at the GC code it doesn't look like the GC is prepared to handle  
that situation either... while I haven't noticed any ways in which it  
could lead to memory corruption, if the program would catch exceptions  
propagated through a GC run, it could lead to persistent memory leaks  
(inconsistency between flags and freelists) and destructors of other  
objects being called several times (due to free lists not being rebuilt).

Thus, my question is: what's the expected behavior of D programs when a  
destructor throws?

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 12 2011
next sibling parent reply Alexander <aldem+dmars nk7.net> writes:
On 13.05.2011 06:53, Vladimir Panteleev wrote:


 Thus, my question is: what's the expected behavior of D programs when a
destructor throws?



  I would say, the only expected (and correct, IMHO) behavior should be
termination of the program because of unhandled exception.

/Alexander
May 13 2011
next sibling parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 13.05.2011 09:57, schrieb Alexander:

 On 13.05.2011 06:53, Vladimir Panteleev wrote:
 

 Thus, my question is: what's the expected behavior of D programs when a
destructor throws?



   I would say, the only expected (and correct, IMHO) behavior should be
termination of the program because of unhandled exception.
 
 /Alexander



That sucks if there's no way to handle that exception (other than by try
{ ... } catch {} in the destructor itself)..

But probably the destructor should generally be forbidden to throw, so
if it does and it crashes the program it may be fine.

http://www.digitalmars.com/d/archives/digitalmars/D/What_is_nothrow_for_70451.html
and
http://www.digitalmars.com/d/archives/digitalmars/D/What_is_throwable_86055.html

discussed throwing in destructors a little bit.

Cheers,
- Daniel
May 13 2011
prev sibling next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 13.05.2011 10:12, schrieb Vladimir Panteleev:

 On Fri, 13 May 2011 10:57:37 +0300, Alexander <aldem+dmars nk7.net> wrote:
 

 On 13.05.2011 06:53, Vladimir Panteleev wrote:


 Thus, my question is: what's the expected behavior of D programs when
 a destructor throws?



   I would say, the only expected (and correct, IMHO) behavior should
 be termination of the program because of unhandled exception.



 Yes, but what if it's handled (there's a try/catch block around the
 allocation or fullCollect call that invoked the GC)?
 



I don't think the exception from a destructor should be thrown to an
allocation.. and probably not fullCollect either.
For clear() or delete it /may/ make sense to get exceptions thrown in
the destructor, but it'd be inconsistent if the exceptions would just
vanish or terminate the program otherwise.

It may be sane to just define that destructors are nothrow and if they
throw anyway to terminate the program.

What is the current behaviour anyway? ;)
May 13 2011
parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 13.05.2011 10:33, schrieb Vladimir Panteleev:

 On Fri, 13 May 2011 11:22:11 +0300, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 

 It may be sane to just define that destructors are nothrow and if they
 throw anyway to terminate the program.

 What is the current behaviour anyway? ;)



 D2 throws a FinalizeError (which is an Error, so "not recoverable").



So the program is terminated? In that case you don't really have to
worry about the state of the GC, right?


 D1 just allows the exception to propagate through the GC to whatever
 caused a GC to run.
 Both leave the GC in an indeterminate state, as far as I can tell.
May 13 2011
prev sibling next sibling parent reply Don <nospam nospam.com> writes:
Alexander wrote:

 On 13.05.2011 06:53, Vladimir Panteleev wrote:
 

 Thus, my question is: what's the expected behavior of D programs when a
destructor throws?



   I would say, the only expected (and correct, IMHO) behavior should be
termination of the program because of unhandled exception.
 
 /Alexander



Are you talking about *finalizers* or *destructors* ?

Throwing from inside a destructor should definitely work (unlike C++).
But finalizers should probably be nothrow.
May 13 2011
next sibling parent reply Alexander <aldem+dmars nk7.net> writes:
On 13.05.2011 10:25, Don wrote:


 Are you talking about *finalizers* or *destructors* ?



  Destructors as defined in D spec. There are no finalizers (yet), AFAIK.


 Throwing from inside a destructor should definitely work (unlike C++).



  How? Destructor is called by the GC when object is deleted. When and where
(which thread) this happens is unknown, it is done outside of regular flow of
execution - so who and where can catch this exception? So, like any other
uncatched exception,
it will terminate the program.


 But finalizers should probably be nothrow.



  Once they are implemented :) In any case, as long as there is no try/catch
around GC collection, I see little use for exceptions apart from terminating
the program.

  If, however, we are talking about using try/catch blocks inside of
destructors - then of course, this should be possible.

/Alexander
May 13 2011
parent reply Don <nospam nospam.com> writes:
Alexander wrote:

 On 13.05.2011 10:25, Don wrote:
 

 Are you talking about *finalizers* or *destructors* ?



   Destructors as defined in D spec. There are no finalizers (yet), AFAIK.



The things in classes which the spec calls "destructors" are finalizers, 
not destructors.


 

 Throwing from inside a destructor should definitely work (unlike C++).



   How? Destructor is called by the GC when object is deleted. When and where
(which thread) this happens is unknown, it is done outside of regular flow of
execution - so who and where can catch this exception? So, like any other
uncatched exception,
 it will terminate the program.



If the GC calls it, it's a finalizer, not a destructor.

Structs have destructors, and it's perfectly OK to throw inside them.
May 13 2011
next sibling parent Alexander <aldem+dmars nk7.net> writes:
On 13.05.2011 15:44, Don wrote:


 The things in classes which the spec calls "destructors" are finalizers, not
destructors.



  OK. Then - when finalizer is called and throws something, program must be
terminated - as there is no catch.

/Alexander
May 13 2011
prev sibling next sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2011-05-13 09:44:33 -0400, Don <nospam nospam.com> said:


 If the GC calls it, it's a finalizer, not a destructor.
 
 Structs have destructors, and it's perfectly OK to throw inside them.



Really? What if you allocated the struct on the heap, as a member of a 
class, or in an array? The struct will be on the heap and its 
destructor will become the finalizer.

What you say about structs makes sense only as long as structs are 
confined to the stack. In reality, structs can also be on the heap.

Discussion for bug 4621's has a long discussion about this issue.
<http://d.puremagic.com/issues/show_bug.cgi?id=4621>

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
May 13 2011
prev sibling parent reply Max Samukha <maxter spambox.com> writes:
On 05/13/2011 04:44 PM, Don wrote:

 Alexander wrote:

 On 13.05.2011 10:25, Don wrote:


 Are you talking about *finalizers* or *destructors* ?



 Destructors as defined in D spec. There are no finalizers (yet), AFAIK.



 The things in classes which the spec calls "destructors" are finalizers,
 not destructors.


 Throwing from inside a destructor should definitely work (unlike C++).



 How? Destructor is called by the GC when object is deleted. When and
 where (which thread) this happens is unknown, it is done outside of
 regular flow of execution - so who and where can catch this exception?
 So, like any other uncatched exception,
 it will terminate the program.



 If the GC calls it, it's a finalizer, not a destructor.

 Structs have destructors, and it's perfectly OK to throw inside them.



Destructor/finalizer confusion is one of D's weakest spots. GC and 
"delete" (now - "clear") should have never been designed to call the 
same function. Now that there are talks about making GC call destructors 
on structs, structs are at the threat of becoming as broken as classes are.
May 13 2011
parent Michel Fortin <michel.fortin michelf.com> writes:
On 2011-05-13 13:16:03 -0400, Max Samukha <maxter spambox.com> said:


 On 05/13/2011 04:44 PM, Don wrote:

 If the GC calls it, it's a finalizer, not a destructor.
 
 Structs have destructors, and it's perfectly OK to throw inside them.



 Destructor/finalizer confusion is one of D's weakest spots. GC and 
 "delete" (now - "clear") should have never been designed to call the 
 same function. Now that there are talks about making GC call 
 destructors on structs, structs are at the threat of becoming as broken 
 as classes are.



They're already broken in subtle and racy ways because of this. A few examples:
<http://d.puremagic.com/issues/show_bug.cgi?id=4624>

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
May 13 2011
prev sibling next sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2011-05-13 04:34:06 -0400, "Vladimir Panteleev" 
<vladimir thecybershadow.net> said:


 Come to think of it, SafeD shouldn't allow accessing anything on the 
 heap  in finalizers as well...



Back to bug 4621.
<http://d.puremagic.com/issues/show_bug.cgi?id=4621>

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
May 13 2011
prev sibling parent reply Don <nospam nospam.com> writes:
Vladimir Panteleev wrote:

 On Fri, 13 May 2011 11:25:01 +0300, Don <nospam nospam.com> wrote:
 

 Alexander wrote:

 On 13.05.2011 06:53, Vladimir Panteleev wrote:


 Thus, my question is: what's the expected behavior of D programs 
 when a destructor throws?



 be termination of the program because of unhandled exception.
  /Alexander



 Are you talking about *finalizers* or *destructors* ?

 Throwing from inside a destructor should definitely work (unlike C++).
 But finalizers should probably be nothrow.



 How would you distinguish the two in the language? Class destructors = 
 finalizers?



Yes.
struct destructor = destructor


 Come to think of it, SafeD shouldn't allow accessing anything on the 
 heap in finalizers as well...



Yeah. It's pretty hard to come up with a use case for a finalizer.
May 13 2011
next sibling parent dsimcha <dsimcha yahoo.com> writes:
== Quote from Don (nospam nospam.com)'s article

 Yeah. It's pretty hard to come up with a use case for a finalizer.



This is why I'm so interested in improving D's GC yet have shown so little
interest in this bug.  I can't even figure out what finalizers are for and
basically never use them.  Even in the cases where I have used them before to
get
around false pointer issues, I now realize that such uses were actually latent
race conditions.
May 13 2011
prev sibling parent Alexander <aldem+dmars nk7.net> writes:
On 13.05.2011 15:47, Don wrote:


 Yeah. It's pretty hard to come up with a use case for a finalizer.



  Why? If object holds some external resource, which needs to be released (like
file handle) - that makes perfect use case, IMHO.

  It *could* happen that explicit release didn't happen, that's why finalizer
should do this.

/Alexander
May 13 2011
prev sibling parent Alexander <aldem+dmars nk7.net> writes:
On 13.05.2011 10:12, Vladimir Panteleev wrote:


 Yes, but what if it's handled (there's a try/catch block around the allocation
or fullCollect call that invoked the GC)?



  Oh, that case... Sure, if GC is buggy and throw causes GC allocation in
turn... then we have a problem, of course, but of course, this should work -
normally.

/Alexander
May 13 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 13 May 2011 10:57:37 +0300, Alexander <aldem+dmars nk7.net> wrote:


 On 13.05.2011 06:53, Vladimir Panteleev wrote:


 Thus, my question is: what's the expected behavior of D programs when a  
 destructor throws?



   I would say, the only expected (and correct, IMHO) behavior should be  
 termination of the program because of unhandled exception.



Yes, but what if it's handled (there's a try/catch block around the  
allocation or fullCollect call that invoked the GC)?

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 13 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 13 May 2011 11:11:47 +0300, Daniel Gibson <metalcaedes gmail.com>  
wrote:


 http://www.digitalmars.com/d/archives/digitalmars/D/What_is_nothrow_for_70451.html
 and
 http://www.digitalmars.com/d/archives/digitalmars/D/What_is_throwable_86055.html

 discussed throwing in destructors a little bit.



Both threads refer to destructors in RAII, though.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 13 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 13 May 2011 11:25:01 +0300, Don <nospam nospam.com> wrote:


 Alexander wrote:

 On 13.05.2011 06:53, Vladimir Panteleev wrote:


 Thus, my question is: what's the expected behavior of D programs when  
 a destructor throws?



 be termination of the program because of unhandled exception.
  /Alexander



 Are you talking about *finalizers* or *destructors* ?

 Throwing from inside a destructor should definitely work (unlike C++).
 But finalizers should probably be nothrow.



How would you distinguish the two in a language? Class destructors =  
finalizers?

Come to think of it, SafeD shouldn't allow accessing anything on the heap  
in destructors as well...

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 13 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 13 May 2011 11:22:11 +0300, Daniel Gibson <metalcaedes gmail.com>  
wrote:


 It may be sane to just define that destructors are nothrow and if they
 throw anyway to terminate the program.

 What is the current behaviour anyway? ;)



D2 throws a FinalizeError (which is an Error, so "not recoverable").
D1 just allows the exception to propagate through the GC to whatever  
caused a GC to run.
Both leave the GC in an indeterminate state, as far as I can tell.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 13 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 13 May 2011 11:25:01 +0300, Don <nospam nospam.com> wrote:


 Alexander wrote:

 On 13.05.2011 06:53, Vladimir Panteleev wrote:


 Thus, my question is: what's the expected behavior of D programs when  
 a destructor throws?



 be termination of the program because of unhandled exception.
  /Alexander



 Are you talking about *finalizers* or *destructors* ?

 Throwing from inside a destructor should definitely work (unlike C++).
 But finalizers should probably be nothrow.



How would you distinguish the two in the language? Class destructors =  
finalizers?

Come to think of it, SafeD shouldn't allow accessing anything on the heap  
in finalizers as well...

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 13 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 13 May 2011 11:41:04 +0300, Daniel Gibson <metalcaedes gmail.com>  
wrote:


 Am 13.05.2011 10:33, schrieb Vladimir Panteleev:

 On Fri, 13 May 2011 11:22:11 +0300, Daniel Gibson
 <metalcaedes gmail.com> wrote:


 It may be sane to just define that destructors are nothrow and if they
 throw anyway to terminate the program.

 What is the current behaviour anyway? ;)



 D2 throws a FinalizeError (which is an Error, so "not recoverable").



 So the program is terminated? In that case you don't really have to
 worry about the state of the GC, right?



I guess, but this requires that the user does not use catch(Object) or  
catch(Throwable).
I don't know if this is acceptable for SafeD, for example.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 13 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 13 May 2011 12:28:01 +0300, Vladimir Panteleev  
<vladimir thecybershadow.net> wrote:


 I guess, but this requires that the user does not use catch(Object) or  
 catch(Throwable).



Also just catch { ... }, of course (I don't understand why that's in the  
language at all).

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 13 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 13 May 2011 11:41:04 +0300, Daniel Gibson <metalcaedes gmail.com>  
wrote:


 So the program is terminated? In that case you don't really have to
 worry about the state of the GC, right?



After looking again at it, my patch actually does "the right thing" and  
locks out the GC after a finalizer exception (any attempt to access the GC  
 from that point will instantly throw), so I've sent the pull requests.  
Thanks for the feedback.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 13 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On 2011-05-13 08:06, Michel Fortin wrote:

 On 2011-05-13 09:44:33 -0400, Don <nospam nospam.com> said:

 If the GC calls it, it's a finalizer, not a destructor.
 
 Structs have destructors, and it's perfectly OK to throw inside them.



 Really? What if you allocated the struct on the heap, as a member of a
 class, or in an array? The struct will be on the heap and its
 destructor will become the finalizer.
 
 What you say about structs makes sense only as long as structs are
 confined to the stack. In reality, structs can also be on the heap.
 
 Discussion for bug 4621's has a long discussion about this issue.
 <http://d.puremagic.com/issues/show_bug.cgi?id=4621>



Well, assuming that a struct's destructor is called when it's garbage 
collected from the heap (which as I recall, doesn't ever happen at this 
point), couldn't the GC just catch any exceptions that it throws and then 
throw the appropriate Error? Then the destructor could throw just fine while 
it's on the stack, but you'd get the FinalizerError (or whatever it's called) 
when it happens on the heap.

- Jonathan M Davis
May 13 2011
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
They effectively are. When a finalized throws its wrapped in a FinalizerErro=
r or something of the sort. This is old code from when Walter said that it s=
hould be illegal for a finalizer to throw (rule should be revised with new c=
haining rules, but it works for now).=20

Sent from my iPhone

On May 13, 2011, at 1:25 AM, Don <nospam nospam.com> wrote:


 Alexander wrote:

 On 13.05.2011 06:53, Vladimir Panteleev wrote:

 Thus, my question is: what's the expected behavior of D programs when a d=










  I would say, the only expected (and correct, IMHO) behavior should be te=







 /Alexander



 Are you talking about *finalizers* or *destructors* ?
=20
 Throwing from inside a destructor should definitely work (unlike C++).
 But finalizers should probably be nothrow.
May 13 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Fri, 13 May 2011 10:01:56 -0400, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Don (nospam nospam.com)'s article

 Yeah. It's pretty hard to come up with a use case for a finalizer.



 This is why I'm so interested in improving D's GC yet have shown so  
 little
 interest in this bug.  I can't even figure out what finalizers are for  
 and
 basically never use them.  Even in the cases where I have used them  
 before to get
 around false pointer issues, I now realize that such uses were actually  
 latent
 race conditions.



I use the finalizers of proxy classes to manage GPU memory.
May 13 2011
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 13 May 2011 17:01:56 +0300, dsimcha <dsimcha yahoo.com> wrote:


 == Quote from Don (nospam nospam.com)'s article

 Yeah. It's pretty hard to come up with a use case for a finalizer.



 This is why I'm so interested in improving D's GC yet have shown so  
 little
 interest in this bug.  I can't even figure out what finalizers are for  
 and
 basically never use them.  Even in the cases where I have used them  
 before to get
 around false pointer issues, I now realize that such uses were actually  
 latent
 race conditions.



In my case, it was mainly because D1 has no struct destructors.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 13 2011