www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Destruction Derby

reply Arcane Jill <Arcane_member pathlink.com> writes:
Destruction - Definitive test results.

Faced with uncertainty, I did what any scientist would do. I did an experiment.
Here's the source code:

//======== BEGIN =========

   import std.c.stdio;
   import std.c.stdlib;
   import std.gc;
   
   FILE * f;
   
   class A
   {
       new(uint n)
       {
           f = fopen("C:\\gctest.txt","wt");
           fprintf(f,"Allocator called\n");
           return malloc(n);
       }
       
       this()
       {
           fprintf(f,"Constructor called\n");
       }
       
       ~this()
       {
           fprintf(f,"Destructor called\n");
       }
       
       delete(void* p)
       {
           fprintf(f,"Deallocator called\n");
           fclose(f);
           free(p);
       }
       
       int t;
   }
   
   int main()
   {
       A a = new A();
       a = null;
       fullCollect();
       return 0;
   }


//========= END ==========

And here's the contents of the file after the program is run:

//======= BEGIN ========
Allocator called
Constructor called
//======== END =========

Conclusions:
(1) The destructor was not called.
(2) The class deallocator was not called.

Observe that the garbage collector WAS EXPLICITLY INVOKED, and that the variable
'a' was not reachable at the time the gc was run. This "destruction" is not
merely lazy, it seems to be actually on vacation.

Now, there are two possibilities. Either
(a) This is correct behavior
(b) This is a bug in the compiler

If it's (a) I will need to think very hard to try to find a workaround. If I
can't find one I will have to abandon my D project and write the thing in C++.
If it's (b) then it's critical to me that the bug be fixed asap, because, again,
this has to work, or D becomes useless for my purposes.
Jun 09 2004
next sibling parent reply J Anderson <REMOVEanderson badmama.com.au> writes:
Arcane Jill wrote:


If it's (a) I will need to think very hard to try to find a workaround. If I
can't find one I will have to abandon my D project and write the thing in C++.
If it's (b) then it's critical to me that the bug be fixed asap, because, again,
this has to work, or D becomes useless for my purposes.
  


Are you saying that the only reason you are using D is because of the 
gc?  If you use C++ you won't get automatic deconstruction either, you 
have to call delete (in D and C++) to get that.


-- 
-Anderson: http://badmama.com.au/~anderson/
Jun 09 2004
next sibling parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <ca6us1$27g7$1 digitaldaemon.com>, J Anderson says...

If you use C++ you won't get automatic deconstruction either, you 
have to call delete (in D and C++) to get that.


Not true.


    void f()  // C++
    {
        SecurePassword p;   // constructed on the stack

        /* do stuff with p which auto makes illegal in D */

    }
    // Destructor of p WILL be called, even in the event of an exception


Jill
Jun 09 2004
prev sibling parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <ca6us1$27g7$1 digitaldaemon.com>, J Anderson says...


Are you saying that the only reason you are using D is because of the 
gc?


No. I'm using D because D is brilliant. I beleive I've said that a few times
already. Please don't read into my words that which I have not said.

Destructors which refuse to execute are merely a recently discovered problem
which I'm currently trying either to solve or work around.

Jill
Jun 09 2004
prev sibling next sibling parent reply "Ivan Senji" <ivan.senji public.srce.hr> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:ca6rl9$22p9$1 digitaldaemon.com...

 Destruction - Definitive test results.

 Faced with uncertainty, I did what any scientist would do. I did an


experiment.

 Here's the source code:

 //======== BEGIN =========

   import std.c.stdio;
   import std.c.stdlib;
   import std.gc;

   FILE * f;

   class A
   {
       new(uint n)
       {
           f = fopen("C:\\gctest.txt","wt");
           fprintf(f,"Allocator called\n");
           return malloc(n);
       }

       this()
       {
           fprintf(f,"Constructor called\n");
       }

       ~this()
       {
           fprintf(f,"Destructor called\n");
       }

       delete(void* p)
       {
           fprintf(f,"Deallocator called\n");
           fclose(f);
           free(p);
       }

       int t;
   }

   int main()
   {
       A a = new A();
       a = null;
       fullCollect();
       return 0;
   }


 //========= END ==========

 And here's the contents of the file after the program is run:

 //======= BEGIN ========
 Allocator called
 Constructor called
 //======== END =========

 Conclusions:
 (1) The destructor was not called.
 (2) The class deallocator was not called.

 Observe that the garbage collector WAS EXPLICITLY INVOKED, and that the


variable

 'a' was not reachable at the time the gc was run. This "destruction" is


not

 merely lazy, it seems to be actually on vacation.


Maybe i am missing something but is it out of the question to use
delete a; ?


 Now, there are two possibilities. Either
 (a) This is correct behavior
 (b) This is a bug in the compiler

 If it's (a) I will need to think very hard to try to find a workaround. If


I

 can't find one I will have to abandon my D project and write the thing in


C++.

 If it's (b) then it's critical to me that the bug be fixed asap, because,


again,

 this has to work, or D becomes useless for my purposes.
Jun 09 2004
parent Ilya Minkov <minkov cs.tum.edu> writes:
Ivan Senji schrieb:


 Maybe i am missing something but is it out of the question to use
 delete a; ?


It is a question of *guarantees*. If a language with automatic memory 
management doesn't guarantee that destructors get called, it has to 
become a problem if more or less critical resources want to be managed 
in an advanced manner. And as shown, auto doesn't always help either.

-eye
Jun 09 2004
prev sibling next sibling parent reply hellcatv hotmail.com writes:
what if you explicitly call delete a; before setting it to null?
In article <ca6rl9$22p9$1 digitaldaemon.com>, Arcane Jill says...

Destruction - Definitive test results.

Faced with uncertainty, I did what any scientist would do. I did an experiment.
Here's the source code:

//======== BEGIN =========

   import std.c.stdio;
   import std.c.stdlib;
   import std.gc;
   
   FILE * f;
   
   class A
   {
       new(uint n)
       {
           f = fopen("C:\\gctest.txt","wt");
           fprintf(f,"Allocator called\n");
           return malloc(n);
       }
       
       this()
       {
           fprintf(f,"Constructor called\n");
       }
       
       ~this()
       {
           fprintf(f,"Destructor called\n");
       }
       
       delete(void* p)
       {
           fprintf(f,"Deallocator called\n");
           fclose(f);
           free(p);
       }
       
       int t;
   }
   
   int main()
   {
       A a = new A();
       a = null;
       fullCollect();
       return 0;
   }


//========= END ==========

And here's the contents of the file after the program is run:

//======= BEGIN ========
Allocator called
Constructor called
//======== END =========

Conclusions:
(1) The destructor was not called.
(2) The class deallocator was not called.

Observe that the garbage collector WAS EXPLICITLY INVOKED, and that the variable
'a' was not reachable at the time the gc was run. This "destruction" is not
merely lazy, it seems to be actually on vacation.

Now, there are two possibilities. Either
(a) This is correct behavior
(b) This is a bug in the compiler

If it's (a) I will need to think very hard to try to find a workaround. If I
can't find one I will have to abandon my D project and write the thing in C++.
If it's (b) then it's critical to me that the bug be fixed asap, because, again,
this has to work, or D becomes useless for my purposes.
Jun 09 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <ca78n4$2mq2$1 digitaldaemon.com>, hellcatv hotmail.com says...

what if you explicitly call delete a; before setting it to null?


You are not the only person who has suggested this, however please, please
understand that in the field in which I am working, this is simply not an
option. Yes OF COURSE it is possible to explicitly call delete() - it's not like
I hadn't thought of that. It's also possible to call a specially written
function called wipeThisSensitiveDataNow().

But that would be missing the point.

A class which manages sensitive data *MUST* be capable of managing that resource
completely, of preventing access to it except through authorized methods, and of
securely erasing that sensitive data, REGARDLESS of what calls a user of that
class does or does not make.

I could write pages and pages on this, on why it is important. But ... I'd
rather that you trusted me that I know what I'm doing. This is my area of
expertise. (Well, that and three decades of experience writing system software).
Computer security is fraught with dangers. There are holes everywhere you look.
One - single - omission ... could render sensitive data such as passwords,
encryption keys, etc., open to an attacker. (Well, to an attacker who knows as
much as I, anyway). And it could be YOUR data that is being protected by SOMEONE
ELSE'S application built using MY toolkit. And THEY might not have called
delete() (or even read the manual).

It's not that there's anything wrong with calling delete(). It's that I cannot
rely on callers of my API doing that. And I have to guard against the
possibility that they may not.

Fortunately, I am going to succeed, come what may. Even if I have to construct
things on the stack (so that they can be routinely erased just by declaring a
large array from time to time), I will find a way.

But all of this is irrelevant, and a side-issue. I can demonstrate, in a simple,
repeatable experiment, that a destructor is not always called. So either this is
correct behavior (in which case, fine, I'll figure out a way round it), or it's
a bug (in which case, it is critical to anyone doing resource management that it
be fixed). That's all.

I do recognise that you were trying to help though, and I do appreciate that.
Arcane Jill
Jun 09 2004
parent "Ivan Senji" <ivan.senji public.srce.hr> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:ca7hnm$3ed$1 digitaldaemon.com...

 In article <ca78n4$2mq2$1 digitaldaemon.com>, hellcatv hotmail.com says...

what if you explicitly call delete a; before setting it to null?


 You are not the only person who has suggested this, however please, please
 understand that in the field in which I am working, this is simply not an
 option. Yes OF COURSE it is possible to explicitly call delete() - it's


not like

 I hadn't thought of that. It's also possible to call a specially written
 function called wipeThisSensitiveDataNow().

 But that would be missing the point.

 A class which manages sensitive data *MUST* be capable of managing that


resource

 completely, of preventing access to it except through authorized methods,


and of

 securely erasing that sensitive data, REGARDLESS of what calls a user of


that

 class does or does not make.

 I could write pages and pages on this, on why it is important. But ... I'd
 rather that you trusted me that I know what I'm doing. This is my area of
 expertise. (Well, that and three decades of experience writing system


software).

 Computer security is fraught with dangers. There are holes everywhere you


look.

 One - single - omission ... could render sensitive data such as passwords,
 encryption keys, etc., open to an attacker. (Well, to an attacker who


knows as

 much as I, anyway). And it could be YOUR data that is being protected by


SOMEONE

 ELSE'S application built using MY toolkit. And THEY might not have called
 delete() (or even read the manual).

 It's not that there's anything wrong with calling delete(). It's that I


cannot

 rely on callers of my API doing that. And I have to guard against the
 possibility that they may not.

 Fortunately, I am going to succeed, come what may. Even if I have to


construct

 things on the stack (so that they can be routinely erased just by


declaring a

 large array from time to time), I will find a way.

 But all of this is irrelevant, and a side-issue. I can demonstrate, in a


simple,

 repeatable experiment, that a destructor is not always called. So either


this is

 correct behavior (in which case, fine, I'll figure out a way round it), or


it's

 a bug (in which case, it is critical to anyone doing resource management


that it

 be fixed). That's all.

 I do recognise that you were trying to help though, and I do appreciate


that.

 Arcane Jill


Thanks for a good explanation :) I mada a mistake of thinking only about
your
code were you can make sure that you call delete, but i wasn't thinking
about
the user code, and now i see the problem.
Jun 09 2004
prev sibling next sibling parent Andy Friesen <andy ikagames.com> writes:
Try this:


  int main()
  {
      {
          auto A a = new A();
          // use a here
      }
      // a's destructor should be called by this point
      fullCollect();
      return 0;
  }


I haven't tested this firsthand, but it should have the effect of 
finalizing a when its enclosing scope terminates. (even if an exception 
is thrown)

  -- andy
Jun 09 2004
prev sibling next sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
If you remove the allocator it works ok. The garbage collector does not scan
the C heap for objects to collect so an instance of class A (when it has the
custom allocator) will never get collected and ones has to call "delete a"
explicitly.
I think the point of custom allocators/deallocators is to allow classes to
avoid the GC memory pool. If you want GC then use the default GC allocator
or have your custom allocator allocate from the GC pool and not the C heap.

"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:ca6rl9$22p9$1 digitaldaemon.com...

 Destruction - Definitive test results.

 Faced with uncertainty, I did what any scientist would do. I did an


experiment.

 Here's the source code:

 //======== BEGIN =========

   import std.c.stdio;
   import std.c.stdlib;
   import std.gc;

   FILE * f;

   class A
   {
       new(uint n)
       {
           f = fopen("C:\\gctest.txt","wt");
           fprintf(f,"Allocator called\n");
           return malloc(n);
       }

       this()
       {
           fprintf(f,"Constructor called\n");
       }

       ~this()
       {
           fprintf(f,"Destructor called\n");
       }

       delete(void* p)
       {
           fprintf(f,"Deallocator called\n");
           fclose(f);
           free(p);
       }

       int t;
   }

   int main()
   {
       A a = new A();
       a = null;
       fullCollect();
       return 0;
   }


 //========= END ==========

 And here's the contents of the file after the program is run:

 //======= BEGIN ========
 Allocator called
 Constructor called
 //======== END =========

 Conclusions:
 (1) The destructor was not called.
 (2) The class deallocator was not called.

 Observe that the garbage collector WAS EXPLICITLY INVOKED, and that the


variable

 'a' was not reachable at the time the gc was run. This "destruction" is


not

 merely lazy, it seems to be actually on vacation.

 Now, there are two possibilities. Either
 (a) This is correct behavior
 (b) This is a bug in the compiler

 If it's (a) I will need to think very hard to try to find a workaround. If


I

 can't find one I will have to abandon my D project and write the thing in


C++.

 If it's (b) then it's critical to me that the bug be fixed asap, because,


again,

 this has to work, or D becomes useless for my purposes.
Jun 09 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <ca7it0$5b6$1 digitaldaemon.com>, Ben Hinkle says...

If you remove the allocator it works ok. The garbage collector does not scan
the C heap for objects to collect so an instance of class A (when it has the
custom allocator) will never get collected and ones has to call "delete a"
explicitly.


Thank you very much. That does work, and that's excellent.

I guess I just hadn't understood the relationship between custom allocators and
the GC. I'm really glad you pointed that out. Thanks again.




I think the point of custom allocators/deallocators is to allow classes to
avoid the GC memory pool. If you want GC then use the default GC allocator
or have your custom allocator allocate from the GC pool and not the C heap.


Yeah, I got that now.

It would seem a minor deficiency in the D language that a custom allocator can
only realistically be used with an auto class (unless it doesn't need a
destructor). But that's not a problem for me, so I'm not going to worry about
it.

I think the manual should probably mention this, but you've solved my problem so
I'm happy now. Thanks again,

Jill
Jun 09 2004
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
In article <ca7la5$90l$1 digitaldaemon.com>, Arcane Jill says...

In article <ca7it0$5b6$1 digitaldaemon.com>, Ben Hinkle says...

If you remove the allocator it works ok. The garbage collector does not scan
the C heap for objects to collect so an instance of class A (when it has the
custom allocator) will never get collected and ones has to call "delete a"
explicitly.


Thank you very much. That does work, and that's excellent.

I guess I just hadn't understood the relationship between custom allocators and
the GC. I'm really glad you pointed that out. Thanks again.




I think the point of custom allocators/deallocators is to allow classes to
avoid the GC memory pool. If you want GC then use the default GC allocator
or have your custom allocator allocate from the GC pool and not the C heap.


Yeah, I got that now.

It would seem a minor deficiency in the D language that a custom allocator can
only realistically be used with an auto class (unless it doesn't need a
destructor). But that's not a problem for me, so I'm not going to worry about
it.

I think the manual should probably mention this, but you've solved my problem so
I'm happy now. Thanks again,

Jill


I would caution you that GC will still miss objects where a stack pointer
happens to point to the object.  If you need absolute guarantees (as it seems
you do), a more forceful approach may still be required.  Maybe the object could
add itself to some kind of global "set" when constructed, and remove itself when
destructed.  At program end, the set would need to be iterated and destructed.

If the objects should be deleted by the GC during the program run, the pointer
would need to be masked somehow.  Also, it would be a mistake to add and remove
the object itself - the pointer must be a pointer to the exact allocation you
are concerned with.  If object A points to object B, the destructor for A cannot
be sure that B has not already be collected.  So the global set would need a
list of B pointers, where B is a struct or array.

Kevin
Jun 14 2004
parent Kevin Bealer <Kevin_member pathlink.com> writes:
In article <cake5h$1629$1 digitaldaemon.com>, Kevin Bealer says...

In article <ca7la5$90l$1 digitaldaemon.com>, Arcane Jill says...

In article <ca7it0$5b6$1 digitaldaemon.com>, Ben Hinkle says...

If you remove the allocator it works ok. The garbage collector does not scan
the C heap for objects to collect so an instance of class A (when it has the
custom allocator) will never get collected and ones has to call "delete a"
explicitly.


Thank you very much. That does work, and that's excellent.

I guess I just hadn't understood the relationship between custom allocators and
the GC. I'm really glad you pointed that out. Thanks again.




I think the point of custom allocators/deallocators is to allow classes to
avoid the GC memory pool. If you want GC then use the default GC allocator
or have your custom allocator allocate from the GC pool and not the C heap.


Yeah, I got that now.

It would seem a minor deficiency in the D language that a custom allocator can
only realistically be used with an auto class (unless it doesn't need a
destructor). But that's not a problem for me, so I'm not going to worry about
it.

I think the manual should probably mention this, but you've solved my problem so
I'm happy now. Thanks again,

Jill


I would caution you that GC will still miss objects where a stack pointer
happens to point to the object.  If you need absolute guarantees (as it seems
you do), a more forceful approach may still be required.  Maybe the object could
add itself to some kind of global "set" when constructed, and remove itself when
destructed.  At program end, the set would need to be iterated and destructed.

If the objects should be deleted by the GC during the program run, the pointer
would need to be masked somehow.  Also, it would be a mistake to add and remove
the object itself - the pointer must be a pointer to the exact allocation you
are concerned with.  If object A points to object B, the destructor for A cannot
be sure that B has not already be collected.  So the global set would need a
list of B pointers, where B is a struct or array.

Kevin


Another thought: it would not be hard for an attacker to determine likely
addresses that malloc may produce (by downloading the source and running similar
programs).  Also, some malloc() implementations have very predictable patterns
such as alignment to power of two for most allocations.

The attacker could then use input (to a web server for example) which contains
these addresses.  One technique would be to specify the memory address as a
spoofed IP address or find unicode characters that match these addresses.
Assuming a conservative GC, this will pin your secure data object in memory, for
whatever nefarious purpose, because the integer will look a lot like a pointer.

(Walter's technique of timing out the secure data will work against this.)

Kevin
Jun 14 2004
prev sibling next sibling parent reply "Walter" <newshound digitalmars.com> writes:
When you overload operators new and delete for your class, then YOU are
managing the memory for that class, not the GC. The GC only manages memory
allocated by new when new is NOT overloaded.

Overloading new and delete is saying "I want to manage the memory myself; GC
keep your hands off of it."


"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:ca6rl9$22p9$1 digitaldaemon.com...

 Destruction - Definitive test results.

 Faced with uncertainty, I did what any scientist would do. I did an


experiment.

 Here's the source code:

 //======== BEGIN =========

   import std.c.stdio;
   import std.c.stdlib;
   import std.gc;

   FILE * f;

   class A
   {
       new(uint n)
       {
           f = fopen("C:\\gctest.txt","wt");
           fprintf(f,"Allocator called\n");
           return malloc(n);
       }

       this()
       {
           fprintf(f,"Constructor called\n");
       }

       ~this()
       {
           fprintf(f,"Destructor called\n");
       }

       delete(void* p)
       {
           fprintf(f,"Deallocator called\n");
           fclose(f);
           free(p);
       }

       int t;
   }

   int main()
   {
       A a = new A();
       a = null;
       fullCollect();
       return 0;
   }


 //========= END ==========

 And here's the contents of the file after the program is run:

 //======= BEGIN ========
 Allocator called
 Constructor called
 //======== END =========

 Conclusions:
 (1) The destructor was not called.
 (2) The class deallocator was not called.

 Observe that the garbage collector WAS EXPLICITLY INVOKED, and that the


variable

 'a' was not reachable at the time the gc was run. This "destruction" is


not

 merely lazy, it seems to be actually on vacation.

 Now, there are two possibilities. Either
 (a) This is correct behavior
 (b) This is a bug in the compiler

 If it's (a) I will need to think very hard to try to find a workaround. If


I

 can't find one I will have to abandon my D project and write the thing in


C++.

 If it's (b) then it's critical to me that the bug be fixed asap, because,


again,

 this has to work, or D becomes useless for my purposes.
Jun 09 2004
parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <ca7mkb$b4b$1 digitaldaemon.com>, Walter says...

When you overload operators new and delete for your class, then YOU are
managing the memory for that class, not the GC. The GC only manages memory
allocated by new when new is NOT overloaded.

Overloading new and delete is saying "I want to manage the memory myself; GC
keep your hands off of it."


Gotcha. Ben explained that to me earlier. Thanks for being so helpful. :}
Jill
Jun 09 2004
prev sibling parent reply EricAnderton at yahoo dot com <EricAnderton_member pathlink.com> writes:
In article <ca6rl9$22p9$1 digitaldaemon.com>, Arcane Jill says...

Destruction - Definitive test results.

Faced with uncertainty, I did what any scientist would do. I did an experiment.
 [snip]
Conclusions:
(1) The destructor was not called.
(2) The class deallocator was not called.


Jill,
One thing ocurred to me with your test case here, that may have some hidden
dangers lurking in it.  I'm sure you've already thought of something like this,
but I thought I'd at least share it here anyway for completeness' sake (as well
as for my own education, since I'm drawing a huge blank here).

If this class were to hold a reference to a gc'd object, would that reference be
scanned by the collector even though this class is allocated "manually"?  

I know next to nothing about security programming, so I haven't a clue if
allowing the GC to scan your objects is even remotely acceptable.  But since
this class lives outside the GC, knowing when and how a GC'd object reference is
still valid keeps coming to my mind as a rather large hurdle.

So I guess the big question here is:
Is any product of a user-defined new() added to the GC automatically for root
scanning, or should it be added with a call to std.gc.AddRange()?

- Eric
Jun 09 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <ca7ohq$ee3$1 digitaldaemon.com>, EricAnderton at yahoo dot com
says...


So I guess the big question here is:
Is any product of a user-defined new() added to the GC automatically for root
scanning, or should it be added with a call to std.gc.AddRange()?


If you had have asked me that question yesterday I would not have known the
answer, but now, thanks to Ben and Walter...

Yup. You have to call addRange() if it's going to contain referenced to GC'ed
stuff. (If I've understood correctly).

Arcane Jill
Jun 09 2004
parent "Walter" <newshound digitalmars.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:ca7pjd$g0k$1 digitaldaemon.com...

 Yup. You have to call addRange() if it's going to contain referenced to


GC'ed

 stuff. (If I've understood correctly).


Call addRange() for any memory that will contain references to GC'd objects
that is not already in the GC pool, or the stack, registers, or static data.
Jun 09 2004