www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Best properly way to destroy a 2 dimensional array?

reply Jonathan Villa <jv_vortex msn.com> writes:
I wrote a little program that given some number it generates a 
list of different combinations (represented by a ubyte array), so 
in the end my function with name GenerateCombinations(int x) 
returns a ubyte[][] (list of arrays of ubytes).

Now the problem is, the quantity of combinations generated are 
pow(2, `x`), thus, giving 20 it returns me a list of 1_048_576 
arrays of ubyte, every array has length of `x` so, in the end 
this function allocates (seeing the Windows Task Manager) 42 MB 
of data after input `20` the first time.

When the list is ready it prints the quantity of combinations 
given (combinations.length), after that I added a foreach(ubyte[] 
a; combinations) calling destroy to every array and then 
destroy(combinations), and before to re-input a new number to 
generate a new combination list I call GC.Collect() and 
GC.minimize().

My question is if I'm doing good calling destroy to those arrays, 
because I don't see any change in the quantity of memory the 
program is using: In the start it uses 1496 KB, after input `20` 
it now have 43000 KB (approx.) and if I input again 20 then it 
sums up to 90000 KB.

In the end I don't see some kind of real memory free. I'm doing 
something wrong?
That's the reason I'm asking for your advice.

PD: All arrays are constructed with the `new` expression.
PD2: Don't ask me why I don't better do an `int count` instead of 
constructing arrays, this little program is going to be part of a 
more huge application and I need the combination list.

Regards. JV
Apr 06 2016
next sibling parent Jonathan Villa <jv_vortex msn.com> writes:
On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:

 I wrote a little program that given some number it generates a 
 list of different combinations (represented by a ubyte array), 
 so in the end my function with name GenerateCombinations(int x) 
 returns a ubyte[][] (list of arrays of ubytes).


Sample code.

void main()
{
	while(true)
	{
		write("Alternatives quantity: ");
		string value = chomp(readln());
		
		if (value == "x")
			break;
		
		int x = to!int(value);
		
		ubyte[][] combs = GenerateCombinations(x);
		writefln("There are %d combinations.", combs.length);
		
		foreach(ubyte[] a; combs)
			destroy(a);
		destroy(combs);
		
		writeln();
		
		GC.collect();
		GC.minimize();
	}
	
	return;
}
Apr 06 2016
prev sibling next sibling parent reply Jonathan Villa <jv_vortex msn.com> writes:
On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:

 I wrote a little program that given some number it generates a


TL;DR: My program generates a very large `ubyte[][]`, and after I 
call destroy and GC.collect() and GC.minimize(), the memory 
occuping doesn't seems to decrease.
Apr 06 2016
parent reply Andrea Fontana <nospam example.com> writes:
On Wednesday, 6 April 2016 at 20:30:33 UTC, Jonathan Villa wrote:

 On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa 
 wrote:

 I wrote a little program that given some number it generates a


 TL;DR: My program generates a very large `ubyte[][]`, and after 
 I call destroy and GC.collect() and GC.minimize(), the memory 
 occuping doesn't seems to decrease.


Anything change if you wrap your code like:


while ...
{
...
    {
       ubyte[][] ...
       ...
    }

    GC.collect();
    GC.minimieze();
}

?
Apr 07 2016
parent reply Andrea Fontana <nospam example.com> writes:
On Thursday, 7 April 2016 at 10:02:05 UTC, Andrea Fontana wrote:

 On Wednesday, 6 April 2016 at 20:30:33 UTC, Jonathan Villa 
 wrote:

 On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa 
 wrote:

 I wrote a little program that given some number it generates a


 TL;DR: My program generates a very large `ubyte[][]`, and 
 after I call destroy and GC.collect() and GC.minimize(), the 
 memory occuping doesn't seems to decrease.


 Anything change if you wrap your code like:


 while ...
 {
 ...
    {
       ubyte[][] ...
       ...
    }

    GC.collect();
    GC.minimieze();
 }

 ?


Or maybe you could try to do:

combs = null;

before collecting.
Apr 07 2016
parent Jonathan Villa <jv_vortex msn.com> writes:
On Thursday, 7 April 2016 at 10:05:02 UTC, Andrea Fontana wrote:

 On Thursday, 7 April 2016 at 10:02:05 UTC, Andrea Fontana wrote:

 On Wednesday, 6 April 2016 at 20:30:33 UTC, Jonathan Villa 
 wrote:

 Anything change if you wrap your code like:


 while ...
 {
 ...
    {
       ubyte[][] ...
       ...
    }

    GC.collect();
    GC.minimieze();
 }

 ?


 Or maybe you could try to do:

 combs = null;

 before collecting.



Yep, I enclosed the ubyte[][] allocation in a new scope, like our 
friend Alex suggested, but nothing seems to change. I'm going to 
try combs = null...

Big news, I've tested my code using Debian x64 8.2 and it really 
frees memory! even without adding `combs = null` (I'm going to 
add it anyway just in case).

So, looks like is some kind of Windows plataform problem. Next 
time I boot on Windows I'm going to add the null assignment and 
hope for results.

Regards!
Apr 07 2016
prev sibling next sibling parent reply Alex Parrill <initrd.gz gmail.com> writes:
On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:

 I wrote a little program that given some number it generates a 
 list of different combinations (represented by a ubyte array), 
 so in the end my function with name GenerateCombinations(int x) 
 returns a ubyte[][] (list of arrays of ubytes).

 ...


Why not make a range instead? No need to reserve memory for the 
entire array if you can compute the elements as-needed.

If you really want an array, std.experimental.allocator will let 
you manually allocate/release objects.
Apr 06 2016
parent reply Jonathan Villa <jv_vortex msn.com> writes:
On Wednesday, 6 April 2016 at 21:33:14 UTC, Alex Parrill wrote:

 On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa 
 wrote:

 I wrote a little program that given some number it generates a 
 ...


 Why not make a range instead? No need to reserve memory for the 
 entire array if you can compute the elements as-needed.

 If you really want an array, std.experimental.allocator will 
 let you manually allocate/release objects.


I'm going to try use a custom allocator for tomorrow.

My general idea is first to get the predicted quantity of 
combinations so I can divide and parallelize them.

What do you think can be the problem with the lack of 
deallocation?

Regards.
Apr 06 2016
parent Alex Parrill <initrd.gz gmail.com> writes:
On Wednesday, 6 April 2016 at 23:14:10 UTC, Jonathan Villa wrote:

 On Wednesday, 6 April 2016 at 21:33:14 UTC, Alex Parrill wrote:

 My general idea is first to get the predicted quantity of 
 combinations


Seems like you already know; your OP says you have 2^n 
combinations.


 so I can divide and parallelize them.


std.parallelism.parallel can do this for you, and works on ranges.




 What do you think can be the problem with the lack of 
 deallocation?


Don't know exactly. destroy just runs the destructors, it doesn't 
free any memory. Since the arrays are still in scope, it might 
not be able to free them.
Apr 06 2016
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/6/16 3:54 PM, Jonathan Villa wrote:

 I wrote a little program that given some number it generates a list of
 different combinations (represented by a ubyte array), so in the end my
 function with name GenerateCombinations(int x) returns a ubyte[][] (list
 of arrays of ubytes).

 Now the problem is, the quantity of combinations generated are pow(2,
 `x`), thus, giving 20 it returns me a list of 1_048_576 arrays of ubyte,
 every array has length of `x` so, in the end this function allocates
 (seeing the Windows Task Manager) 42 MB of data after input `20` the
 first time.

 When the list is ready it prints the quantity of combinations given
 (combinations.length), after that I added a foreach(ubyte[] a;
 combinations) calling destroy to every array and then
 destroy(combinations), and before to re-input a new number to generate a
 new combination list I call GC.Collect() and GC.minimize().

 My question is if I'm doing good calling destroy to those arrays,
 because I don't see any change in the quantity of memory the program is
 using: In the start it uses 1496 KB, after input `20` it now have 43000
 KB (approx.) and if I input again 20 then it sums up to 90000 KB.

 In the end I don't see some kind of real memory free. I'm doing
 something wrong?
 That's the reason I'm asking for your advice.

 PD: All arrays are constructed with the `new` expression.
 PD2: Don't ask me why I don't better do an `int count` instead of
 constructing arrays, this little program is going to be part of a more
 huge application and I need the combination list.

 Regards. JV


You are likely running into the GC being conservative. You are also 
possibly running into an issue where you are expecting the compiler to 
do something with the stack where it may do something else. A classic 
question that is asked on D forums all the time is why when you null 
some variable and then call GC collect, the memory isn't collected. The 
answer is because quite possibly the value is still in a register, and 
the registers must be scanned too.

As for conservatism, if you generate a 1-million element array of 
pointers (each element has a pointer in it), then you have some random 
stack variable that "happens" to point into that giant array (more 
likely in 32-bit systems), then both the array, and all it's pointing at 
gets saved.

Note that destroying an array is equivalent to setting it to null. It 
doesn't write any data to the array itself.

Ironically, your example of this:

foreach(ubyte[] a; combs) destroy(a);

does absolutely nothing. Because all you are doing is setting a to null, 
and a is a local variable.

The only thing that could make a difference is destroying combs, which 
is equivalent to setting combs to null. But like I said, it's possible 
there is still a pointer in registers, or that some other stack variable 
points at your huge array.

-Steve
Apr 07 2016
parent reply Jonathan Villa <jv_vortex msn.com> writes:
On Thursday, 7 April 2016 at 16:13:59 UTC, Steven Schveighoffer 
wrote:

 On 4/6/16 3:54 PM, Jonathan Villa wrote:

 You are likely running into the GC being conservative. You are 
 also possibly running into an issue where you are expecting the 
 compiler to do something with the stack where it may do 
 something else. A classic question that is asked on D forums 
 all the time is why when you null some variable and then call 
 GC collect, the memory isn't collected. The answer is because 
 quite possibly the value is still in a register, and the 
 registers must be scanned too.

 As for conservatism, if you generate a 1-million element array 
 of pointers (each element has a pointer in it), then you have 
 some random stack variable that "happens" to point into that 
 giant array (more likely in 32-bit systems), then both the 
 array, and all it's pointing at gets saved.
 
 -Steve


Good, I compiled the program in 64 bit mode (-m64) and now it 
release memory normally like in Debian.

I would like to know where can I learn more about the 'register' 
or the GC for the D so I can avoid future issues.

JV.
Apr 08 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/8/16 11:08 AM, Jonathan Villa wrote:

 On Thursday, 7 April 2016 at 16:13:59 UTC, Steven Schveighoffer wrote:

 On 4/6/16 3:54 PM, Jonathan Villa wrote:

 You are likely running into the GC being conservative. You are also
 possibly running into an issue where you are expecting the compiler to
 do something with the stack where it may do something else. A classic
 question that is asked on D forums all the time is why when you null
 some variable and then call GC collect, the memory isn't collected.
 The answer is because quite possibly the value is still in a register,
 and the registers must be scanned too.

 As for conservatism, if you generate a 1-million element array of
 pointers (each element has a pointer in it), then you have some random
 stack variable that "happens" to point into that giant array (more
 likely in 32-bit systems), then both the array, and all it's pointing
 at gets saved.


 Good, I compiled the program in 64 bit mode (-m64) and now it release
 memory normally like in Debian.


Definitely, 64-bit is less likely to randomly point at memory. This 
isn't 100% sure-fire way to fix the problem though.


 I would like to know where can I learn more about the 'register' or the
 GC for the D so I can avoid future issues.


Your best bet is to free the memory itself if it's possible.

import core.memory: GC;
GC.free(combs.ptr);

For more info on how GC works: dlang.org/spec/garbage.html

-Steve
Apr 08 2016
parent Jonathan Villa <jv_vortex msn.com> writes:
On Friday, 8 April 2016 at 15:21:50 UTC, Steven Schveighoffer 
wrote:

 On 4/8/16 11:08 AM, Jonathan Villa wrote:

 On Thursday, 7 April 2016 at 16:13:59 UTC, Steven 
 Schveighoffer wrote:


 Your best bet is to free the memory itself if it's possible.

 import core.memory: GC;
 GC.free(combs.ptr);

 For more info on how GC works: dlang.org/spec/garbage.html

 -Steve


Thank you very much! I think this is the answer I was looking for 
C:

JV
Apr 08 2016