www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Can dynamic arrays move in memory?

reply Brad Beveridge <brad somewhere.net> writes:
I already know the answer - yes.  The question should really be
"Can dynamic arrays move in memory if you don't resize them?"

If they can't - fine, but it should be part of the specification & not 
up to the compiler implementation to decide.

If they can, then I think D has a big problem - which I will illustrate.
I am playing with OpenGL, and I have a bug - it may not be because 
arrays move, but if they could move then you would certainly run into 
problems.
Assume that the GC can move array references at will:
1) Create a dynamic array called points.
2) Pass points into a C library - in this case OpenGL
3) Create a dynamic array of normals
4) Pass normals into C lib (openGL)
5) Call a library function that does something with these arrays.

Depending on how the GC behaves - and if arrays can move - then the call 
to 5) is pretty undefined.  The C library is pointing at stale memory 
locations.

I get the impression that future (compacting) GCs may be able to move 
arrays at will, even if you don't resize the array that moves.  If this 
is the case I would like to propose "lock" and "unlock" properties for 
arrays.
When an array is locked you cannot resize it (runtime error), but you 
can be sure that the GC will not move it for you - ie, it is just a 
plain chunk of memory that C can mess about with.  After the array is 
unlocked, then any C code can no longer trust their pointers.

Thoughts?
Brad
Apr 27 2005
next sibling parent reply "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
"Brad Beveridge" <brad somewhere.net> wrote in message 
news:d4pgjl$2gho$1 digitaldaemon.com...
 If they can't - fine, but it should be part of the specification & not up 
 to the compiler implementation to decide.

The current GC doesn't, but you're absolutely right - it should be set in stone whether the GC is allowed to move memory or not. Hopefully the decision is _not_ to allow it. Most C/C++ libs require that the memory does not move. Things like associative arrays with class refs as keys would not work properly (unless the class refs were doubly indirect - which would just be stupid). And unless you're really really REALLY pressed for memory, not copying is usually just fine.
Apr 27 2005
parent Norbert Nemec <Norbert Nemec-online.de> writes:
Jarrett Billingsley schrieb:
 "Brad Beveridge" <brad somewhere.net> wrote in message 
 news:d4pgjl$2gho$1 digitaldaemon.com...
 
If they can't - fine, but it should be part of the specification & not up 
to the compiler implementation to decide.

The current GC doesn't, but you're absolutely right - it should be set in stone whether the GC is allowed to move memory or not.

Allowing memory to be moved by the GC would be a perfect source of subtle bugs, far worse than uninitialized variables in C: Imagine some code that erraneously depends on memory staying in some fixed position. The code will work for years until somebody actually implements a GC that exploits that possibility. Now the code will suddenly begin to behave absolutely unpredictable and crash in non-reproducible ways! And this possibly so rarely that people will not even believe there is a bug...
Apr 28 2005
prev sibling next sibling parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"Brad Beveridge" <brad somewhere.net> wrote in message 
news:d4pgjl$2gho$1 digitaldaemon.com...
I already know the answer - yes.  The question should really be
 "Can dynamic arrays move in memory if you don't resize them?"

 I get the impression that future (compacting) GCs may be able to move 
 arrays at will, even if you don't resize the array that moves.  If this is 
 the case I would like to propose "lock" and "unlock" properties for 
 arrays.
 When an array is locked you cannot resize it (runtime error), but you can 
 be sure that the GC will not move it for you - ie, it is just a plain 
 chunk of memory that C can mess about with.  After the array is unlocked, 
 then any C code can no longer trust their pointers.

 Thoughts?
 Brad

On the wish-list. see http://www.digitalmars.com/d/archives/digitalmars/D/9385.html
Apr 27 2005
parent Brad Beveridge <brad somewhere.net> writes:
Ben Hinkle wrote:

 On the wish-list. see 
 http://www.digitalmars.com/d/archives/digitalmars/D/9385.html 
 
 

Ah. At least I don't feel too bad about missing this message - it was a while ago :) Thanks Ben Brad
Apr 27 2005
prev sibling next sibling parent Benji Smith <dlanguage xxagg.com> writes:
Brad Beveridge wrote:
 I already know the answer - yes.  The question should really be
 "Can dynamic arrays move in memory if you don't resize them?"
 
 ...
 
 If they can, then I think D has a big problem...

Hmmmmmmmm. But if the GC can't move arrays, then won't it eventually degrade as memory becomes fragmented? It seems like, if the GC were allowed to move arrays, it would be able to keep a reasonably sized pool of contiguous memory available for allocation. The most recent incarnation of the Java GC has improved its performance quite a bit by creating memory "ergonomics", one of which is determining when (and if) the compiler is allowed to move objects from its "young" object pool to its "mature" object pool. Perhaps the lock & unlock idea proposed by Brad would work well to solve any problems when interacting with C libraries. But it seems like a job for the compiler/runtime. Not for the programmer to worry about. --BenjiSmith
Apr 28 2005
prev sibling next sibling parent Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
There is another thing to consider, which is similar, but not quite the 
same: if you don't keep a D reference to one of those arrays, then D 
might garbage collect the array away before GL uses it.

So, you either have to keep a D reference to those arrays hanging around 
until GL is done, or you have to disable the GC in that window of time.

Brad Beveridge wrote:
 I already know the answer - yes.  The question should really be
 "Can dynamic arrays move in memory if you don't resize them?"
 
 If they can't - fine, but it should be part of the specification & not 
 up to the compiler implementation to decide.
 
 If they can, then I think D has a big problem - which I will illustrate.
 I am playing with OpenGL, and I have a bug - it may not be because 
 arrays move, but if they could move then you would certainly run into 
 problems.
 Assume that the GC can move array references at will:
 1) Create a dynamic array called points.
 2) Pass points into a C library - in this case OpenGL
 3) Create a dynamic array of normals
 4) Pass normals into C lib (openGL)
 5) Call a library function that does something with these arrays.
 
 Depending on how the GC behaves - and if arrays can move - then the call 
 to 5) is pretty undefined.  The C library is pointing at stale memory 
 locations.
 
 I get the impression that future (compacting) GCs may be able to move 
 arrays at will, even if you don't resize the array that moves.  If this 
 is the case I would like to propose "lock" and "unlock" properties for 
 arrays.
 When an array is locked you cannot resize it (runtime error), but you 
 can be sure that the GC will not move it for you - ie, it is just a 
 plain chunk of memory that C can mess about with.  After the array is 
 unlocked, then any C code can no longer trust their pointers.
 
 Thoughts?
 Brad

Apr 28 2005
prev sibling parent bobef <bobef paintballforce.com> writes:
Locking/unlocking is much needed for every type of variable, I believe, 
not only for arrays. And this is because all the world uses C/C++ which 
doesn't have a GC... And since D depends on C/C++ so much and it always 
will it needs such properties... Just for example - Win32. Win32 will 
always lay on C and D will always have to support Win32 (or at least for 
next 15 years)...

Brad Beveridge wrote:
 I already know the answer - yes.  The question should really be
 "Can dynamic arrays move in memory if you don't resize them?"
 
 If they can't - fine, but it should be part of the specification & not 
 up to the compiler implementation to decide.
 
 If they can, then I think D has a big problem - which I will illustrate.
 I am playing with OpenGL, and I have a bug - it may not be because 
 arrays move, but if they could move then you would certainly run into 
 problems.
 Assume that the GC can move array references at will:
 1) Create a dynamic array called points.
 2) Pass points into a C library - in this case OpenGL
 3) Create a dynamic array of normals
 4) Pass normals into C lib (openGL)
 5) Call a library function that does something with these arrays.
 
 Depending on how the GC behaves - and if arrays can move - then the call 
 to 5) is pretty undefined.  The C library is pointing at stale memory 
 locations.
 
 I get the impression that future (compacting) GCs may be able to move 
 arrays at will, even if you don't resize the array that moves.  If this 
 is the case I would like to propose "lock" and "unlock" properties for 
 arrays.
 When an array is locked you cannot resize it (runtime error), but you 
 can be sure that the GC will not move it for you - ie, it is just a 
 plain chunk of memory that C can mess about with.  After the array is 
 unlocked, then any C code can no longer trust their pointers.
 
 Thoughts?
 Brad

May 01 2005