www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Is Phobos's Garbage Collector utterly broken? (Phobos vs Tango)

reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
Attached is a simple program which creates 326*100000 objects, and periodically
prints out progress. I initially wrote it to try to find a memory leak in
Tango's GC (which was actually fixed at some point).

The results of this program are quite interesting.
The Tango version runs in 65-75 seconds on my machine, and never seems to take
more than 3MB of RAM.
The Phobos version, however, runs much more poorly - it starts to consume
hundreds of MB of RAM immediately, and the fastest run from my tests was 265
seconds. It also behaves quite randomly (memory usage patterns differ each run).
Here's a description of one of the not-so-great runs: it starts off with almost
instantly allocating 80 MB of memory, and after a while allocates about 30
more. It runs fine for a while, doing a 1000 in 2-3 seconds - until around
60000 it spontaneously starts eating a lot of memory. It goes on like this
until it eats ~340 MB of RAM in all, where it plateaus. At that point, the
program slows down to a crawl, doing a 1000 in a few minutes. I've attached a
screenshot of the RAM history of this run - I had to stop the program to keep
the run's history within view (don't think much would have changed had I left
it running, and it could have taken half an hour or more).

What could be the problem of this seemingly random behavior? Is the GC not
releasing the memory to the OS on purpose, by estimating that the OS doesn't
need it? Or is it a GC bug - if it is, are the warnings Valgrind outputs about
Phobos's GC referencing and using undefined memory somehow related to this?

-- 
Best regards,
  Vladimir                          mailto:thecybershadow gmail.com
Jul 31 2007
parent reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Wed, 01 Aug 2007 09:08:16 +0300, Vladimir Panteleev
<thecybershadow gmail.com> wrote:


 I initially wrote it to try to find a memory leak in Tango's GC (which was
actually fixed at some point).


Turns out it's still there, and it's the old "binary data" issue with
pointer-searching GCs, which was fixed in D/Phobos 1.001 by making the GC
type-aware. Check out the attached sample programs for a simple example - the
Tango version can't know there are no pointers in its GrowBuffer's data, and
thus leaks like crazy, while the Phobos version stays at 13MB.

-- 
Best regards,
  Vladimir                          mailto:thecybershadow gmail.com
Aug 01 2007
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 09:08:16 +0300, Vladimir Panteleev
<thecybershadow gmail.com> wrote:
 

 I initially wrote it to try to find a memory leak in Tango's GC (which was
actually fixed at some point).


 
 Turns out it's still there, and it's the old "binary data" issue with
pointer-searching GCs, which was fixed in D/Phobos 1.001 by making the GC
type-aware. Check out the attached sample programs for a simple example - the
Tango version can't know there are no pointers in its GrowBuffer's data, and
thus leaks like crazy, while the Phobos version stays at 13MB.


It turns out this is because GrowBuffer uses a void[] internally to 
store data.  The type should probably be changed to byte[].  I'll file a 
ticket for it.

Sean
Aug 01 2007
next sibling parent reply Deewiant <deewiant.doesnotlike.spam gmail.com> writes:
Sean Kelly wrote:

 Vladimir Panteleev wrote:

 Turns out it's still there, and it's the old "binary data" issue with
 pointer-searching GCs, which was fixed in D/Phobos 1.001 by making the
 GC type-aware. Check out the attached sample programs for a simple
 example - the Tango version can't know there are no pointers in its
 GrowBuffer's data, and thus leaks like crazy, while the Phobos version
 stays at 13MB.


 
 It turns out this is because GrowBuffer uses a void[] internally to
 store data.  The type should probably be changed to byte[].  I'll file a
 ticket for it.
 


Isn't this also a problem with all the IO stuff that use void[] for everything?
It starts right up there at IBuffer and IConduit.

The way I see it: void[] is for "pure memory" that you might want to access in
multiple different ways (akin to unions), ubyte[] is for the traditional model
of "arbitrary data", and byte[] is just weird, I've yet to figure out a use for
it.

-- 
Remove ".doesnotlike.spam" from the mail address.
Aug 01 2007
parent Sean Kelly <sean f4.ca> writes:
Deewiant wrote:

 Sean Kelly wrote:

 Vladimir Panteleev wrote:

 Turns out it's still there, and it's the old "binary data" issue with
 pointer-searching GCs, which was fixed in D/Phobos 1.001 by making the
 GC type-aware. Check out the attached sample programs for a simple
 example - the Tango version can't know there are no pointers in its
 GrowBuffer's data, and thus leaks like crazy, while the Phobos version
 stays at 13MB.


 It turns out this is because GrowBuffer uses a void[] internally to
 store data.  The type should probably be changed to byte[].  I'll file a
 ticket for it.


 
 Isn't this also a problem with all the IO stuff that use void[] for everything?
 It starts right up there at IBuffer and IConduit.


Well, passing data around as void[] is fine, it just can't be stored as 
void[].  Buffer is one of the few objects that actually stores data in 
this way.


 The way I see it: void[] is for "pure memory" that you might want to access in
 multiple different ways (akin to unions), ubyte[] is for the traditional model
 of "arbitrary data", and byte[] is just weird, I've yet to figure out a use
for it.


That sounds like a good distinction.  Feel free to add to the ticket if 
you think other uses of void[] should be changed to byte[] or ubyte[].


Sean
Aug 01 2007
prev sibling parent reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Wed, 01 Aug 2007 17:48:26 +0300, Sean Kelly <sean f4.ca> wrote:


 It turns out this is because GrowBuffer uses a void[] internally to
 store data.  The type should probably be changed to byte[].  I'll file a
 ticket for it.


Cheers, that indeed fixed it. And now it runs much faster than the Phobos
version, too!

What's the reasoning of scanning void[] - why would anyone keep pointers in a
void[] since it's supposed to mean "binary non-descript data"?

-- 
Best regards,
  Vladimir                          mailto:thecybershadow gmail.com
Aug 01 2007
next sibling parent Sean Kelly <sean f4.ca> writes:
Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 17:48:26 +0300, Sean Kelly <sean f4.ca> wrote:
 

 It turns out this is because GrowBuffer uses a void[] internally to
 store data.  The type should probably be changed to byte[].  I'll file a
 ticket for it.


 
 Cheers, that indeed fixed it. And now it runs much faster than the Phobos
version, too!
 
 What's the reasoning of scanning void[] - why would anyone keep pointers in a
void[] since it's supposed to mean "binary non-descript data"?


I think the idea was that a void array may contain /anything/ including 
structs, in-place constructed classes, array references, etc, and it was 
easier specifying that void arrays be scanned than expecting the user to 
call gc.hasPointers() or whatever on the memory block every time a 
reallocation occurs.


Sean
Aug 01 2007
prev sibling next sibling parent reply kenny <funisher gmail.com> writes:
ok you convinced me... I'll switch to tango now -- or at least start heading
that direction..

I use dmd 2.003 at the moment (it is fast enough for me, I guess... postgre is
the slow one actually), and I noticed that it definitely doesn't work with
2.003 at all.

Another thing that's really awesome about phobos / d is that the documentation
is 100% available offline. Personally, I have the worlds worst internet
connection on this planet. I would definitely be needing the documentation to
be available offline to be able to use tango effectively. I see some doc stuff,
but it looks like css and js files (I assume that the html is generated
somehow) -- but what about all of the good information on the wiki?

Other than that, I was checking out the docs and tango looks really really
good. I really like the threading stuff you've got. There are so many really
good features -- I'm actually pretty excited.

So here's the questions:
1. when is dmd 2.0 support? (I guess I can downgrade to dmd 1.0 for some time)
2. will all the documentation be available offline?
3. what other features are way faster than the phobos equiv?

Last thing... I really wish I had buffers and GrowBuffers right now... I could
really use them for some cool stuff... They look awesome! Dang... I've been
missing out. When I get converted over to tango, I'll provide before and after
benchmarks.

Good work guys.

Kenny

Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 17:48:26 +0300, Sean Kelly <sean f4.ca> wrote:
 

 It turns out this is because GrowBuffer uses a void[] internally to
 store data.  The type should probably be changed to byte[].  I'll file a
 ticket for it.


 
 Cheers, that indeed fixed it. And now it runs much faster than the Phobos
version, too!
 
 What's the reasoning of scanning void[] - why would anyone keep pointers in a
void[] since it's supposed to mean "binary non-descript data"?
Aug 01 2007
parent reply Sean Kelly <sean f4.ca> writes:
kenny wrote:

 ok you convinced me... I'll switch to tango now -- or at least start heading
that direction..
 
 I use dmd 2.003 at the moment (it is fast enough for me, I guess... postgre is
the slow one actually), and I noticed that it definitely doesn't work with
2.003 at all.


Tango hasn't transitioned to D 2.0 yet.  Maintaining parallel versions 
of the library isn't a terribly appealing notion, though I suppose it 
may become necessary at some point.  Also, I had been worried that some 
of the existing features might change (though it seems like this 
probably won't actually happen), and because the work involved in a port 
to 2.0 will likely be fairly significant, I don't want to have to do it 
more than once.


 Another thing that's really awesome about phobos / d is that the documentation
is 100% available offline. Personally, I have the worlds worst internet
connection on this planet. I would definitely be needing the documentation to
be available offline to be able to use tango effectively. I see some doc stuff,
but it looks like css and js files (I assume that the html is generated
somehow) -- but what about all of the good information on the wiki?


I believe there is some work underway to generate offline information 
from the wiki, but I'm not sure how far along it is.  I agree that 
offline documentation is very useful, as I do most of my D programming 
on the train.


 Other than that, I was checking out the docs and tango looks really really
good. I really like the threading stuff you've got. There are so many really
good features -- I'm actually pretty excited.
 
 So here's the questions:
 1. when is dmd 2.0 support? (I guess I can downgrade to dmd 1.0 for some time)


Not sure.  After the conference I may start looking into it.


 2. will all the documentation be available offline?


Yes, but no timetable yet.


 3. what other features are way faster than the phobos equiv?


IO is the stand-out in terms of performance.  I think regex would be 
faster with a rewrite if we (or someone else) can find the time for it 
(we use the Phobos version right now).  And I'm not sure if it matters, 
but I've been told our threading implementation is more robust, and 
certain calls there are definitely faster than Phobos given the way each 
are implemented.


Sean
Aug 02 2007
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Sean Kelly wrote:

 kenny wrote:

 2. will all the documentation be available offline?


 
 Yes, but no timetable yet.


You can do this yourself with a copy of wget.  It's not a *perfect*
copy, but it does work.  I did this a while back, so forgive the sketchy
instructions :)  Also, this is *just* the API reference, and doesn't
include the formatted source code.  That said, I found it useful when my
dialup was down.

NB: Thunderbird <3s wrapping lines, so be careful with the shell
commands (prefixed with a '$').

First, make a directory for the docs.  Then open up a shell (or command
line or whatever) and get into that directory.  Then run these commands
(note: these are for bash.  If you don't have bash, just expand the
curly braces: "a{b,c}d" ==> "abd acd", and join the lines together at
the end-of-line backslashes.)

$ wget -np -nH -m -p -k -E -x --cut-dirs=4 \
http://www.dsource.org/projects/tango/docs/current/

$ wget -np -nH -m -p -k -E -x --cut-dirs=6 \
http://svn.dsource.org/projects/tango/trunk/doc/html/candydoc/img/

$ wget -nH -x --cut-dirs=4 \
http://www.dsource.org/projects/tango/docs/current/js/\
{explorer,tree,util}.js \
http://www.dsource.org/projects/tango/chrome/common/js/\
{dsource,trac}.js

$ wget -nH -x --cut-dirs=3 \
http://www.dsource.org/projects/tango/themeengine/theme.js

That should do it.

Hope that helps.

	-- Daniel
Aug 02 2007
parent Brad Anderson <brad dsource.org> writes:
Daniel Keep wrote:

 
 Sean Kelly wrote:

 kenny wrote:

 2. will all the documentation be available offline?


 Yes, but no timetable yet.


 
 You can do this yourself with a copy of wget.  It's not a *perfect*
 copy, but it does work.  I did this a while back, so forgive the sketchy
 instructions :)  Also, this is *just* the API reference, and doesn't
 include the formatted source code.  That said, I found it useful when my
 dialup was down.
 
 NB: Thunderbird <3s wrapping lines, so be careful with the shell
 commands (prefixed with a '$').
 
 First, make a directory for the docs.  Then open up a shell (or command
 line or whatever) and get into that directory.  Then run these commands
 (note: these are for bash.  If you don't have bash, just expand the
 curly braces: "a{b,c}d" ==> "abd acd", and join the lines together at
 the end-of-line backslashes.)
 
 $ wget -np -nH -m -p -k -E -x --cut-dirs=4 \
 http://www.dsource.org/projects/tango/docs/current/
 
 $ wget -np -nH -m -p -k -E -x --cut-dirs=6 \
 http://svn.dsource.org/projects/tango/trunk/doc/html/candydoc/img/
 
 $ wget -nH -x --cut-dirs=4 \
 http://www.dsource.org/projects/tango/docs/current/js/\
 {explorer,tree,util}.js \
 http://www.dsource.org/projects/tango/chrome/common/js/\
 {dsource,trac}.js
 
 $ wget -nH -x --cut-dirs=3 \
 http://www.dsource.org/projects/tango/themeengine/theme.js
 
 That should do it.
 
 Hope that helps.
 
 	-- Daniel


Ah, that was you sucking my server dry, eh?

I do have it on my list of things to do for offline docs... and
potentially Gregor's DSSS could help here.

BA
Aug 02 2007
prev sibling parent reply Dave <Dave_member pathlink.com> writes:
Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 17:48:26 +0300, Sean Kelly <sean f4.ca> wrote:
 

 It turns out this is because GrowBuffer uses a void[] internally to
 store data.  The type should probably be changed to byte[].  I'll file a
 ticket for it.


 
 Cheers, that indeed fixed it. And now it runs much faster than the Phobos
version, too!
 


I thought the Phobos and Tango GC's were basically the same -- has that changed
over the last 1/2 
year or so?

Thanks,

- Dave


 What's the reasoning of scanning void[] - why would anyone keep pointers in a
void[] since it's supposed to mean "binary non-descript data"?
Aug 02 2007
parent Sean Kelly <sean f4.ca> writes:
Dave wrote:

 Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 17:48:26 +0300, Sean Kelly <sean f4.ca> wrote:


 It turns out this is because GrowBuffer uses a void[] internally to
 store data.  The type should probably be changed to byte[].  I'll file a
 ticket for it.


 Cheers, that indeed fixed it. And now it runs much faster than the 
 Phobos version, too!
 


 I thought the Phobos and Tango GC's were basically the same -- has that 
 changed over the last 1/2 year or so?


No.  They're still basically the same.  But some of the differences that 
exist both there and in the runtime could have a noticeable impact on 
performance in some situations, as this test shows.  Also, recent 
discussion about this issue has inspired me to make some additional 
changes that will further affect certain aspects of how memory is 
managed, etc, and these could have an impact on the performance of some 
corner cases (or at least that's my hope).

Eventually, I'd like to spend some time developing a new GC which is 
more oriented towards multithreaded programming.  But that promises to 
be a fairly large project, and I don't have time for it quite yet.


Sean
Aug 02 2007
prev sibling next sibling parent reply Jascha Wetzel <"[firstname]" mainia.de> writes:
Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 09:08:16 +0300, Vladimir Panteleev
<thecybershadow gmail.com> wrote:
 

 I initially wrote it to try to find a memory leak in Tango's GC (which was
actually fixed at some point).


 
 Turns out it's still there, and it's the old "binary data" issue with
pointer-searching GCs, which was fixed in D/Phobos 1.001 by making the GC
type-aware. Check out the attached sample programs for a simple example - the
Tango version can't know there are no pointers in its GrowBuffer's data, and
thus leaks like crazy, while the Phobos version stays at 13MB.


i ran into another issue with the phobos gc. i have constant array 
literals that are used in a constructor to initialize a member. the data 
isn't always used. each instance uses it at most once. the second time 
an instance uses that data, it has been overwritten. the problem 
disappears if i disable the gc.
it looks like the gc frees the memory that holds the constant 
initializer. unfortunately it's really hard to reproduce this problem in 
a small program.
Aug 06 2007
parent reply Sean Kelly <sean f4.ca> writes:
Jascha Wetzel wrote:

 Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 09:08:16 +0300, Vladimir Panteleev 
 <thecybershadow gmail.com> wrote:


 I initially wrote it to try to find a memory leak in Tango's GC 
 (which was actually fixed at some point).


 Turns out it's still there, and it's the old "binary data" issue with 
 pointer-searching GCs, which was fixed in D/Phobos 1.001 by making the 
 GC type-aware. Check out the attached sample programs for a simple 
 example - the Tango version can't know there are no pointers in its 
 GrowBuffer's data, and thus leaks like crazy, while the Phobos version 
 stays at 13MB.


 
 i ran into another issue with the phobos gc. i have constant array 
 literals that are used in a constructor to initialize a member. the data 
 isn't always used. each instance uses it at most once. the second time 
 an instance uses that data, it has been overwritten. the problem 
 disappears if i disable the gc.
 it looks like the gc frees the memory that holds the constant 
 initializer. unfortunately it's really hard to reproduce this problem in 
 a small program.


Have you tested this with Tango?  I would expect the same broken 
behavior but you never know, and any differences may help track down the 
issue.


Sean
Aug 06 2007
next sibling parent Jascha Wetzel <"[firstname]" mainia.de> writes:
Sean Kelly wrote:

 Have you tested this with Tango?  I would expect the same broken 
 behavior but you never know, and any differences may help track down the 
 issue.


i postponed that because it'll take a bit longer to my the D2.0 code 
compile with 1.018 again. but i will definitely test it.
Aug 06 2007
prev sibling parent Jascha Wetzel <firstname mainia.de> writes:
Sean Kelly wrote:

 Jascha Wetzel wrote:

 Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 09:08:16 +0300, Vladimir Panteleev 
 <thecybershadow gmail.com> wrote:


 I initially wrote it to try to find a memory leak in Tango's GC 
 (which was actually fixed at some point).


 Turns out it's still there, and it's the old "binary data" issue with 
 pointer-searching GCs, which was fixed in D/Phobos 1.001 by making 
 the GC type-aware. Check out the attached sample programs for a 
 simple example - the Tango version can't know there are no pointers 
 in its GrowBuffer's data, and thus leaks like crazy, while the Phobos 
 version stays at 13MB.


 i ran into another issue with the phobos gc. i have constant array 
 literals that are used in a constructor to initialize a member. the 
 data isn't always used. each instance uses it at most once. the second 
 time an instance uses that data, it has been overwritten. the problem 
 disappears if i disable the gc.
 it looks like the gc frees the memory that holds the constant 
 initializer. unfortunately it's really hard to reproduce this problem 
 in a small program.


 
 Have you tested this with Tango?  I would expect the same broken 
 behavior but you never know, and any differences may help track down the 
 issue.


Tango has the same problem. I couldn't find a small test program that 
provokes this problem, but i can reproduce it with a larger one.

The problem doesn't arise the second time it is used, as i guessed 
earlier, it's after the GC ran once. It seems to miss the root in B.data 
(see below), free that memory and re-assign it to some other block. When 
the program crashes, the memory pointed to by B.data has been 
overwritten with values (used as indeces) that cause the crash.
It's basically this:

align(1) struct C
{
    uint a, b, c;
    uint[] s;
}

abstract class A
{
   C[] data; // gets initialized in subclass' c'tor - see below

   void foo()
   {
     bar(&data[calcIndex()]);
   }

   void bar(C* a)
   {
     foreach ( s; a.s )
       doSomething(s);  // crashes because s is used as an index
   }
}

class B : A
{
   this()
   {
     data = [
         C(1,2,3,[4,5,6]),
         C(7,8,9,[1,3,6]),
         C(35,5,88,[1234,78,6])
     ];
   }
}
Sep 17 2007
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 09:08:16 +0300, Vladimir Panteleev
<thecybershadow gmail.com> wrote:
 

 I initially wrote it to try to find a memory leak in Tango's GC (which was
actually fixed at some point).


 
 Turns out it's still there, and it's the old "binary data" issue with
pointer-searching GCs, which was fixed in D/Phobos 1.001 by making the GC
type-aware. Check out the attached sample programs for a simple example - the
Tango version can't know there are no pointers in its GrowBuffer's data, and
thus leaks like crazy, while the Phobos version stays at 13MB.


The cause of this is somewhat an artifact of the OO design in Tango. 
The underlying buffer being allocated is a byte[], but the reference to 
it is a void[].  The problem occurs when GrowBuffer grows the buffer by 
increasing its length, which causes the buffer to be reallocated as a 
void[].  The reason this is a problem is that neither runtime, Tango or 
Phobos, preserves memory block attributes during a reallocation--they 
both simply key off the type being used to perform the reallocation. 
Obviously, this is a problem, and I've decided to change the behavior in 
Tango accordingly.  It will take some doing and I'm a bit over-busy at 
the moment, but before long the Tango runtime will preserve all block 
attributes on a reallocation.  In essence, this will occur by having the 
runtime call gc_realloc, but before this will work gc_realloc must be 
fixed to handle slices.


Sean
Aug 06 2007
parent reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Mon, 06 Aug 2007 18:20:22 +0300, Sean Kelly <sean f4.ca> wrote:


 Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 09:08:16 +0300, Vladimir Panteleev
<thecybershadow gmail.com> wrote:


 I initially wrote it to try to find a memory leak in Tango's GC (which was
actually fixed at some point).


 Turns out it's still there, and it's the old "binary data" issue with
pointer-searching GCs, which was fixed in D/Phobos 1.001 by making the GC
type-aware. Check out the attached sample programs for a simple example - the
Tango version can't know there are no pointers in its GrowBuffer's data, and
thus leaks like crazy, while the Phobos version stays at 13MB.


 The cause of this is somewhat an artifact of the OO design in Tango.
 The underlying buffer being allocated is a byte[], but the reference to
 it is a void[].  The problem occurs when GrowBuffer grows the buffer by
 increasing its length, which causes the buffer to be reallocated as a
 void[].  The reason this is a problem is that neither runtime, Tango or
 Phobos, preserves memory block attributes during a reallocation--they
 both simply key off the type being used to perform the reallocation.
 Obviously, this is a problem, and I've decided to change the behavior in
 Tango accordingly.  It will take some doing and I'm a bit over-busy at
 the moment, but before long the Tango runtime will preserve all block
 attributes on a reallocation.  In essence, this will occur by having the
 runtime call gc_realloc, but before this will work gc_realloc must be
 fixed to handle slices.


I'd still rather vote towards making the GC not scan void[] - it makes the most
sense.

-- 
Best regards,
  Vladimir                          mailto:thecybershadow gmail.com
Aug 07 2007
parent Sean Kelly <sean f4.ca> writes:
Vladimir Panteleev wrote:

 On Mon, 06 Aug 2007 18:20:22 +0300, Sean Kelly <sean f4.ca> wrote:
 

 Vladimir Panteleev wrote:

 On Wed, 01 Aug 2007 09:08:16 +0300, Vladimir Panteleev
<thecybershadow gmail.com> wrote:


 I initially wrote it to try to find a memory leak in Tango's GC (which was
actually fixed at some point).


 Turns out it's still there, and it's the old "binary data" issue with
pointer-searching GCs, which was fixed in D/Phobos 1.001 by making the GC
type-aware. Check out the attached sample programs for a simple example - the
Tango version can't know there are no pointers in its GrowBuffer's data, and
thus leaks like crazy, while the Phobos version stays at 13MB.


 The cause of this is somewhat an artifact of the OO design in Tango.
 The underlying buffer being allocated is a byte[], but the reference to
 it is a void[].  The problem occurs when GrowBuffer grows the buffer by
 increasing its length, which causes the buffer to be reallocated as a
 void[].  The reason this is a problem is that neither runtime, Tango or
 Phobos, preserves memory block attributes during a reallocation--they
 both simply key off the type being used to perform the reallocation.
 Obviously, this is a problem, and I've decided to change the behavior in
 Tango accordingly.  It will take some doing and I'm a bit over-busy at
 the moment, but before long the Tango runtime will preserve all block
 attributes on a reallocation.  In essence, this will occur by having the
 runtime call gc_realloc, but before this will work gc_realloc must be
 fixed to handle slices.


 
 I'd still rather vote towards making the GC not scan void[] - it makes the
most sense.


Others have expressed the same opinion.  I'll withhold my own thoughts 
but to say that I think the idea behind the current approach is twofold:

1. void[] is the 'any' buffer type for in-program data.  The type of the 
underlying data could be an array of bytes or it could be an array of 
structs containing pointers.  The 'any' buffer type for out-of-program 
data is byte[], because until such data is translated to D types, it's 
merely a stream of bytes.

2. It is nice to have an in-language option for specifying that an 'any' 
buffer type may contain pointers.

The most obvious counter-argument is that by assigning special behavior 
to void[], those who don't like that behavior must use something else 
and lose the implicit conversion that void[] provides.  This is 
extremely convenient in some cases.  Another being that because void[] 
does not specify a type, it is appropriate for out-of-program data as 
well, and scanning a stream of bytes read from a file for pointers is bad.

I personally don't think there is a solution to this that will make 
everyone happy, and am hoping that by preserving block attributes I will 
make void[] more usable as a reference type since then only the first 
allocation must be "new byte[x]".  It is also more consistent, since 
some reallocations obtain a new array (and lose block information), 
while others do not (and preserve block information).


Sean
Aug 07 2007