www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Garbage collector throwing during collection

reply =?ISO-8859-15?Q?S=F6nke_Ludwig?= <ludwig informatik.uni-luebeck.de> writes:
On DMD I got some strange out-of-memory errors with the current 2.054 
version. I tracked it down to allocations happening during garbage 
collection.

Has something fundamental in the GC implementation changed in the latest 
release? It used to work, but seems to be completely disallowed now. I 
think it is quite a strong requirement to not make any allocations from 
within finalizers and makes them just another bit more useless. (In my 
case I have some logging going on which needs dynamic allocations for 
formatting (Appender)).

Regards,
Sönke
Jul 13 2011
next sibling parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
Yes, they're currently disallowed.  Something about memory corruption in the 
gc.  This will probably be fixed eventually, but I wouldn't hold your 
breath.

"Sönke Ludwig" <ludwig informatik.uni-luebeck.de> wrote in message 
news:ivm4bo$2fsc$1 digitalmars.com...

 On DMD I got some strange out-of-memory errors with the current 2.054 
 version. I tracked it down to allocations happening during garbage 
 collection.

 Has something fundamental in the GC implementation changed in the latest 
 release? It used to work, but seems to be completely disallowed now. I 
 think it is quite a strong requirement to not make any allocations from 
 within finalizers and makes them just another bit more useless. (In my 
 case I have some logging going on which needs dynamic allocations for 
 formatting (Appender)).

 Regards,
 Sönke
Jul 14 2011
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thu, 14 Jul 2011 09:58:09 +0300, Sönke Ludwig  
<ludwig informatik.uni-luebeck.de> wrote:


 On DMD I got some strange out-of-memory errors with the current 2.054  
 version. I tracked it down to allocations happening during garbage  
 collection.

 Has something fundamental in the GC implementation changed in the latest  
 release? It used to work, but seems to be completely disallowed now. I  
 think it is quite a strong requirement to not make any allocations from  
 within finalizers and makes them just another bit more useless. (In my  
 case I have some logging going on which needs dynamic allocations for  
 formatting (Appender)).


Hi, I was the one who submitted the patch. The situation before the patch  
was that allocating memory from a finalizer of a collected object  
inevitably lead to memory corruption (which is, as you may know,  
devilishly hard to track down). Ideally, someone would rewrite the  
appropriate parts of the GC to allow safe throwing and allocations from  
finalizers, but that hasn't happened yet.

Here is the Bugzilla issue:  
http://d.puremagic.com/issues/show_bug.cgi?id=5653

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Jul 14 2011
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
On Jul 14, 2011, at 3:12 AM, Vladimir Panteleev wrote:


 On Thu, 14 Jul 2011 09:58:09 +0300, S=F6nke Ludwig =


<ludwig informatik.uni-luebeck.de> wrote:

=20

 On DMD I got some strange out-of-memory errors with the current 2.054 =




version. I tracked it down to allocations happening during garbage =
collection.

=20
 Has something fundamental in the GC implementation changed in the =




latest release? It used to work, but seems to be completely disallowed =
now. I think it is quite a strong requirement to not make any =
allocations from within finalizers and makes them just another bit more =
useless. (In my case I have some logging going on which needs dynamic =
allocations for formatting (Appender)).

=20
 Hi, I was the one who submitted the patch. The situation before the =


patch was that allocating memory from a finalizer of a collected object =
inevitably lead to memory corruption (which is, as you may know, =
devilishly hard to track down). Ideally, someone would rewrite the =
appropriate parts of the GC to allow safe throwing and allocations from =
finalizers, but that hasn't happened yet.

I started on the rewrite, but it's a pretty big change so I'm =
considering trying out Leandro's GC instead.  It would require some work =
as well, but the code is cleaner and it already supports precise =
scanning, so it may be a better starting point.
Jul 14 2011
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Sean Kelly (sean invisibleduck.org)'s article

 I started on the rewrite, but it's a pretty big change so I'm
 considering trying out Leandro's GC instead.  It would require some work
 as well, but the code is cleaner and it already supports precise
 scanning, so it may be a better starting point.


What's the scope of said rewrite?
Jul 14 2011
parent Sean Kelly <sean invisibleduck.org> writes:
On Jul 14, 2011, at 7:05 AM, dsimcha wrote:


 =3D=3D Quote from Sean Kelly (sean invisibleduck.org)'s article

 I started on the rewrite, but it's a pretty big change so I'm
 considering trying out Leandro's GC instead.  It would require some =




work

 as well, but the code is cleaner and it already supports precise
 scanning, so it may be a better starting point.


=20
 What's the scope of said rewrite?


During a collection, having allocations generate new pools instead of =
modifying the existing ones, then merging everything when the collection =
ends.  This is a bit tricky because the GC wasn't written with this =
level of encapsulation in mind, so a bunch of functions need to be =
modified.=
Jul 16 2011
prev sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Thu, 14 Jul 2011 08:39:06 -0400, Sean Kelly <sean invisibleduck.org> wrote:


 On Jul 14, 2011, at 3:12 AM, Vladimir Panteleev wrote:


 On Thu, 14 Jul 2011 09:58:09 +0300, Sönke Ludwig
<ludwig informatik.uni-luebeck.de> wrote:


 On DMD I got some strange out-of-memory errors with the current 2.054 version.
I tracked it down to allocations happening during garbage collection.

 Has something fundamental in the GC implementation changed in the latest
release? It used to work, but seems to be completely disallowed now. I think it
is quite a strong requirement to not make any allocations from within
finalizers and makes them just another bit more useless. (In my case I have
some logging going on which needs dynamic allocations for formatting
(Appender)).


 Hi, I was the one who submitted the patch. The situation before the patch was
that allocating memory from a finalizer of a collected object inevitably lead
to memory corruption (which is, as you may know, devilishly hard to track
down). Ideally, someone would rewrite the appropriate parts of the GC to allow
safe throwing and allocations from finalizers, but that hasn't happened yet.


 I started on the rewrite, but it's a pretty big change so I'm considering
trying out Leandro's GC instead.  It would require some work as well, but the
code is cleaner and it already supports precise scanning, so it may be a better
starting point.


IIRC Leandro's GC doesn't work on windows.
Jul 14 2011
parent reply Sean Kelly <sean invisibleduck.org> writes:
On Jul 14, 2011, at 7:21 AM, Robert Jacques wrote:


 On Thu, 14 Jul 2011 08:39:06 -0400, Sean Kelly =


<sean invisibleduck.org> wrote:

=20

 On Jul 14, 2011, at 3:12 AM, Vladimir Panteleev wrote:
=20

 On Thu, 14 Jul 2011 09:58:09 +0300, S=F6nke Ludwig =






<ludwig informatik.uni-luebeck.de> wrote:

=20

 On DMD I got some strange out-of-memory errors with the current =








2.054 version. I tracked it down to allocations happening during garbage =
collection.

=20
 Has something fundamental in the GC implementation changed in the =








latest release? It used to work, but seems to be completely disallowed =
now. I think it is quite a strong requirement to not make any =
allocations from within finalizers and makes them just another bit more =
useless. (In my case I have some logging going on which needs dynamic =
allocations for formatting (Appender)).

=20
 Hi, I was the one who submitted the patch. The situation before the =






patch was that allocating memory from a finalizer of a collected object =
inevitably lead to memory corruption (which is, as you may know, =
devilishly hard to track down). Ideally, someone would rewrite the =
appropriate parts of the GC to allow safe throwing and allocations from =
finalizers, but that hasn't happened yet.

=20
 I started on the rewrite, but it's a pretty big change so I'm =




considering trying out Leandro's GC instead.  It would require some work =
as well, but the code is cleaner and it already supports precise =
scanning, so it may be a better starting point.

=20
 IIRC Leandro's GC doesn't work on windows.


If it doesn't then it shouldn't be too hard to fix.  Leandro intended it =
to work on Windows but didn't have a test box.=
Jul 16 2011
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 16.07.2011 20:22, schrieb Sean Kelly:

 On Jul 14, 2011, at 7:21 AM, Robert Jacques wrote:


 On Thu, 14 Jul 2011 08:39:06 -0400, Sean Kelly<sean invisibleduck.org>  wrote:


 On Jul 14, 2011, at 3:12 AM, Vladimir Panteleev wrote:


 On Thu, 14 Jul 2011 09:58:09 +0300, Sönke
Ludwig<ludwig informatik.uni-luebeck.de>  wrote:


 On DMD I got some strange out-of-memory errors with the current 2.054 version.
I tracked it down to allocations happening during garbage collection.

 Has something fundamental in the GC implementation changed in the latest
release? It used to work, but seems to be completely disallowed now. I think it
is quite a strong requirement to not make any allocations from within
finalizers and makes them just another bit more useless. (In my case I have
some logging going on which needs dynamic allocations for formatting
(Appender)).


 Hi, I was the one who submitted the patch. The situation before the patch was
that allocating memory from a finalizer of a collected object inevitably lead
to memory corruption (which is, as you may know, devilishly hard to track
down). Ideally, someone would rewrite the appropriate parts of the GC to allow
safe throwing and allocations from finalizers, but that hasn't happened yet.


 I started on the rewrite, but it's a pretty big change so I'm considering
trying out Leandro's GC instead.  It would require some work as well, but the
code is cleaner and it already supports precise scanning, so it may be a better
starting point.


 IIRC Leandro's GC doesn't work on windows.


 If it doesn't then it shouldn't be too hard to fix.  Leandro intended it to
work on Windows but didn't have a test box.


I'm not sure, but I seem to remember it uses fork()?
fork() is not available on Windows (there are workarounds, but they are 
slow, unlike the copy-on-update implementation of modern Unix like 
operating systems).

Cheers,
- Daniel
Jul 22 2011
parent "Robert Jacques" <sandford jhu.edu> writes:
On Fri, 22 Jul 2011 17:22:24 -0400, Daniel Gibson <metalcaedes gmail.com> wrote:


 Am 16.07.2011 20:22, schrieb Sean Kelly:

 On Jul 14, 2011, at 7:21 AM, Robert Jacques wrote:


 On Thu, 14 Jul 2011 08:39:06 -0400, Sean Kelly<sean invisibleduck.org>  wrote:


 On Jul 14, 2011, at 3:12 AM, Vladimir Panteleev wrote:


 On Thu, 14 Jul 2011 09:58:09 +0300, Sönke
Ludwig<ludwig informatik.uni-luebeck.de>  wrote:


 On DMD I got some strange out-of-memory errors with the current 2.054 version.
I tracked it down to allocations happening during garbage collection.

 Has something fundamental in the GC implementation changed in the latest
release? It used to work, but seems to be completely disallowed now. I think it
is quite a strong requirement to not make any allocations from within
finalizers and makes them just another bit more useless. (In my case I have
some logging going on which needs dynamic allocations for formatting
(Appender)).


 Hi, I was the one who submitted the patch. The situation before the patch was
that allocating memory from a finalizer of a collected object inevitably lead
to memory corruption (which is, as you may know, devilishly hard to track
down). Ideally, someone would rewrite the appropriate parts of the GC to allow
safe throwing and allocations from finalizers, but that hasn't happened yet.


 I started on the rewrite, but it's a pretty big change so I'm considering
trying out Leandro's GC instead.  It would require some work as well, but the
code is cleaner and it already supports precise scanning, so it may be a better
starting point.


 IIRC Leandro's GC doesn't work on windows.


 If it doesn't then it shouldn't be too hard to fix.  Leandro intended it to
work on Windows but didn't have a test box.


 I'm not sure, but I seem to remember it uses fork()?
 fork() is not available on Windows (there are workarounds, but they are
 slow, unlike the copy-on-update implementation of modern Unix like
 operating systems).

 Cheers,
 - Daniel


Yes, CDGC is a snapshot GC inspired by this paper: Nonintrusive Cloning Garbage
Collector with Stock Operating System Support
(http://www.cs.purdue.edu/homes/grr/snapshot-gc.ps) And while CDGC doesn't
require fork per se, it does require OS support for dynamically marking memory
pages as copy-on-write. Windows doesn't allow this, which is one reason fork
can't be implemented efficiently on Windows.
Jul 22 2011
prev sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <ludwig informatik.uni-luebeck.de> writes:
Hi! I see, that could also explain some earlier crashes. I can 
definitely work around it for now and was just worrying that this could 
have been part of the general plan for the GC.

Regards,
Sönke

Am 14.07.2011 12:12, schrieb Vladimir Panteleev:

 On Thu, 14 Jul 2011 09:58:09 +0300, Sönke Ludwig
 <ludwig informatik.uni-luebeck.de> wrote:


 On DMD I got some strange out-of-memory errors with the current 2.054
 version. I tracked it down to allocations happening during garbage
 collection.

 Has something fundamental in the GC implementation changed in the
 latest release? It used to work, but seems to be completely disallowed
 now. I think it is quite a strong requirement to not make any
 allocations from within finalizers and makes them just another bit
 more useless. (In my case I have some logging going on which needs
 dynamic allocations for formatting (Appender)).


 Hi, I was the one who submitted the patch. The situation before the
 patch was that allocating memory from a finalizer of a collected object
 inevitably lead to memory corruption (which is, as you may know,
 devilishly hard to track down). Ideally, someone would rewrite the
 appropriate parts of the GC to allow safe throwing and allocations from
 finalizers, but that hasn't happened yet.

 Here is the Bugzilla issue:
 http://d.puremagic.com/issues/show_bug.cgi?id=5653
Jul 14 2011