www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Append to dynamic array that was allocated via calloc

reply John Burton <john.burton jbmail.com> writes:
I can create a "slice" using non-gc allocated memory.

         int* ptr = cast(int*)calloc(int.sizeof, 10);
         int[] data = ptr[0..10];

If I don't want a memory leak I have to call free(ptr) somewhere 
as it won't be GC collected when data or ptr go out of scope. I 
presume there is nothing wrong with doing the above, other than 
perhaps there being better ways (and the memory leak if not 
free'd)

If I then write this :-

         data ~= 1;

What happens? It seems to successfully append an extra value to 
the array. It appears to "work" when I try it in my compiler but 
I don't understand how. Will this be trying to write beyond the 
memory I calloc'ed?
Jul 25
next sibling parent Dukc <ajieskola gmail.com> writes:
On Tuesday, 25 July 2017 at 12:40:13 UTC, John Burton wrote:
 What happens? It seems to successfully append an extra value to 
 the array. It appears to "work" when I try it in my compiler 
 but I don't understand how. Will this be trying to write beyond 
 the memory I calloc'ed?
The language makes no guarantee. It may append in-place if the operating system finds out that the memory after it is not in use. But it may also allocate a whole new array (with gc) and copy the whole thing there. And since the original array was allocated manually, that results in a memory leak. Unless you check for that by storing the appended array in a new variable and compare their pointers. The why part here is for efficiency reasons. We do not want to copy every time one appends, but if there's something in way that just needs to be done. I think the correct way here is to use realloc(). Or std.container.Array. It is well possible there are other good options too.
Jul 25
prev sibling next sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Tuesday, 25 July 2017 at 12:40:13 UTC, John Burton wrote:
 I can create a "slice" using non-gc allocated memory.

         int* ptr = cast(int*)calloc(int.sizeof, 10);
         int[] data = ptr[0..10];

 If I don't want a memory leak I have to call free(ptr) 
 somewhere as it won't be GC collected when data or ptr go out 
 of scope. I presume there is nothing wrong with doing the 
 above, other than perhaps there being better ways (and the 
 memory leak if not free'd)

 If I then write this :-

         data ~= 1;

 What happens? It seems to successfully append an extra value to 
 the array. It appears to "work" when I try it in my compiler 
 but I don't understand how. Will this be trying to write beyond 
 the memory I calloc'ed?
This should give you the answer: writefln("Before: ptr = %s capacity = %s", slice.ptr, slice.capacity); slice ~= 1; writefln("After: ptr = %s capacity = %s", slice.ptr, slice.capacity); It shows that before the append, the capacity is 0. That indicates that any append will cause a new allocation -- from the GC. The next writefln verifies this by showing a different value for ptr and a new capacity of 15. In order for this to work, you'll need to manually manage the length and track the capacity yourself. If all you want is to allocate space for 10 ints, but not 10 actual ints, then something like this: size_t capacity = 10; int* ints = cast(int*)malloc(int.sizeof * capacity); int[] slice = ints[0 .. 10]; slice.length = 0; slice ~= 1; --capacity; Then reallocate the array when capacity reaches 0. Or just use std.container.array.Array which does all this for you.
Jul 25
parent John Burton <john.burton jbmail.com> writes:
On Tuesday, 25 July 2017 at 13:24:36 UTC, Mike Parker wrote:
 On Tuesday, 25 July 2017 at 12:40:13 UTC, John Burton wrote:
 [...]
This should give you the answer: writefln("Before: ptr = %s capacity = %s", slice.ptr, slice.capacity); slice ~= 1; writefln("After: ptr = %s capacity = %s", slice.ptr, slice.capacity); It shows that before the append, the capacity is 0. That indicates that any append will cause a new allocation -- from the GC. The next writefln verifies this by showing a different value for ptr and a new capacity of 15. In order for this to work, you'll need to manually manage the length and track the capacity yourself. If all you want is to allocate space for 10 ints, but not 10 actual ints, then something like this: size_t capacity = 10; int* ints = cast(int*)malloc(int.sizeof * capacity); int[] slice = ints[0 .. 10]; slice.length = 0; slice ~= 1; --capacity; Then reallocate the array when capacity reaches 0. Or just use std.container.array.Array which does all this for you.
Ok so it sounds like this is "safe" in that it will copy my data into GC memory and all work safely. I'll need to somehow keep track of my original memory and free it to avoid a memory leak... (I don't plan to do any of this, but I wanted to understand)(
Jul 25
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 7/25/17 8:40 AM, John Burton wrote:
 I can create a "slice" using non-gc allocated memory.
 
          int* ptr = cast(int*)calloc(int.sizeof, 10);
          int[] data = ptr[0..10];
 
 If I don't want a memory leak I have to call free(ptr) somewhere as it 
 won't be GC collected when data or ptr go out of scope. I presume there 
 is nothing wrong with doing the above, other than perhaps there being 
 better ways (and the memory leak if not free'd)
 
 If I then write this :-
 
          data ~= 1;
 
 What happens? It seems to successfully append an extra value to the 
 array. It appears to "work" when I try it in my compiler but I don't 
 understand how. Will this be trying to write beyond the memory I calloc'ed?
What happens is the runtime detects that data is NOT pointing at an appendable GC-allocated block. So it reallocates the whole block in the GC, and appends your element. The original pointer is lost from data, and leaked (assuming you aren't manually freeing ptr somewhere). I'd recommend using std.container.Array[1], which should do the right thing with malloc'd memory, and give you the nice operators such as ~=. Or you have to resort to C calls for everything, if you don't want to deal with that. Optionally, if you have a defined lifetime for this array in a scope, you can do this: int *ptr = ...; // your calloc call scope(exit) free(ptr); // ensure it is freed. Make SURE you don't change ptr inside the function. Now you can use ~= on data, and it should still free the ptr at the end of the scope/function. -Steve [1] http://dlang.org/phobos/std_container_array.html#.Array
Jul 25