digitalmars.D.learn - Best properly way to destroy a 2 dimensional array?
- Jonathan Villa (28/28) Apr 06 2016 I wrote a little program that given some number it generates a
- Jonathan Villa (29/33) Apr 06 2016 Sample code.
- Jonathan Villa (4/5) Apr 06 2016 TL;DR: My program generates a very large `ubyte[][]`, and after I
- Andrea Fontana (13/19) Apr 07 2016 Anything change if you wrap your code like:
- Andrea Fontana (4/25) Apr 07 2016 Or maybe you could try to do:
- Jonathan Villa (11/34) Apr 07 2016 Yep, I enclosed the ubyte[][] allocation in a new scope, like our
- Alex Parrill (5/10) Apr 06 2016 Why not make a range instead? No need to reserve memory for the
- Jonathan Villa (7/15) Apr 06 2016 I'm going to try use a custom allocator for tomorrow.
- Alex Parrill (8/14) Apr 06 2016 Seems like you already know; your OP says you have 2^n
- Steven Schveighoffer (24/50) Apr 07 2016 You are likely running into the GC being conservative. You are also
- Jonathan Villa (7/23) Apr 08 2016 Good, I compiled the program in 64 bit mode (-m64) and now it
- Steven Schveighoffer (8/29) Apr 08 2016 Definitely, 64-bit is less likely to randomly point at memory. This
- Jonathan Villa (5/13) Apr 08 2016 Thank you very much! I think this is the answer I was looking for
I wrote a little program that given some number it generates a list of different combinations (represented by a ubyte array), so in the end my function with name GenerateCombinations(int x) returns a ubyte[][] (list of arrays of ubytes). Now the problem is, the quantity of combinations generated are pow(2, `x`), thus, giving 20 it returns me a list of 1_048_576 arrays of ubyte, every array has length of `x` so, in the end this function allocates (seeing the Windows Task Manager) 42 MB of data after input `20` the first time. When the list is ready it prints the quantity of combinations given (combinations.length), after that I added a foreach(ubyte[] a; combinations) calling destroy to every array and then destroy(combinations), and before to re-input a new number to generate a new combination list I call GC.Collect() and GC.minimize(). My question is if I'm doing good calling destroy to those arrays, because I don't see any change in the quantity of memory the program is using: In the start it uses 1496 KB, after input `20` it now have 43000 KB (approx.) and if I input again 20 then it sums up to 90000 KB. In the end I don't see some kind of real memory free. I'm doing something wrong? That's the reason I'm asking for your advice. PD: All arrays are constructed with the `new` expression. PD2: Don't ask me why I don't better do an `int count` instead of constructing arrays, this little program is going to be part of a more huge application and I need the combination list. Regards. JV
Apr 06 2016
On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:I wrote a little program that given some number it generates a list of different combinations (represented by a ubyte array), so in the end my function with name GenerateCombinations(int x) returns a ubyte[][] (list of arrays of ubytes).Sample code. void main() { while(true) { write("Alternatives quantity: "); string value = chomp(readln()); if (value == "x") break; int x = to!int(value); ubyte[][] combs = GenerateCombinations(x); writefln("There are %d combinations.", combs.length); foreach(ubyte[] a; combs) destroy(a); destroy(combs); writeln(); GC.collect(); GC.minimize(); } return; }
Apr 06 2016
On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:I wrote a little program that given some number it generates aTL;DR: My program generates a very large `ubyte[][]`, and after I call destroy and GC.collect() and GC.minimize(), the memory occuping doesn't seems to decrease.
Apr 06 2016
On Wednesday, 6 April 2016 at 20:30:33 UTC, Jonathan Villa wrote:On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:Anything change if you wrap your code like: while ... { ... { ubyte[][] ... ... } GC.collect(); GC.minimieze(); } ?I wrote a little program that given some number it generates aTL;DR: My program generates a very large `ubyte[][]`, and after I call destroy and GC.collect() and GC.minimize(), the memory occuping doesn't seems to decrease.
Apr 07 2016
On Thursday, 7 April 2016 at 10:02:05 UTC, Andrea Fontana wrote:On Wednesday, 6 April 2016 at 20:30:33 UTC, Jonathan Villa wrote:Or maybe you could try to do: combs = null; before collecting.On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:Anything change if you wrap your code like: while ... { ... { ubyte[][] ... ... } GC.collect(); GC.minimieze(); } ?I wrote a little program that given some number it generates aTL;DR: My program generates a very large `ubyte[][]`, and after I call destroy and GC.collect() and GC.minimize(), the memory occuping doesn't seems to decrease.
Apr 07 2016
On Thursday, 7 April 2016 at 10:05:02 UTC, Andrea Fontana wrote:On Thursday, 7 April 2016 at 10:02:05 UTC, Andrea Fontana wrote:Yep, I enclosed the ubyte[][] allocation in a new scope, like our friend Alex suggested, but nothing seems to change. I'm going to try combs = null... Big news, I've tested my code using Debian x64 8.2 and it really frees memory! even without adding `combs = null` (I'm going to add it anyway just in case). So, looks like is some kind of Windows plataform problem. Next time I boot on Windows I'm going to add the null assignment and hope for results. Regards!On Wednesday, 6 April 2016 at 20:30:33 UTC, Jonathan Villa wrote: Anything change if you wrap your code like: while ... { ... { ubyte[][] ... ... } GC.collect(); GC.minimieze(); } ?Or maybe you could try to do: combs = null; before collecting.
Apr 07 2016
On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:I wrote a little program that given some number it generates a list of different combinations (represented by a ubyte array), so in the end my function with name GenerateCombinations(int x) returns a ubyte[][] (list of arrays of ubytes). ...Why not make a range instead? No need to reserve memory for the entire array if you can compute the elements as-needed. If you really want an array, std.experimental.allocator will let you manually allocate/release objects.
Apr 06 2016
On Wednesday, 6 April 2016 at 21:33:14 UTC, Alex Parrill wrote:On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:I'm going to try use a custom allocator for tomorrow. My general idea is first to get the predicted quantity of combinations so I can divide and parallelize them. What do you think can be the problem with the lack of deallocation? Regards.I wrote a little program that given some number it generates a ...Why not make a range instead? No need to reserve memory for the entire array if you can compute the elements as-needed. If you really want an array, std.experimental.allocator will let you manually allocate/release objects.
Apr 06 2016
On Wednesday, 6 April 2016 at 23:14:10 UTC, Jonathan Villa wrote:On Wednesday, 6 April 2016 at 21:33:14 UTC, Alex Parrill wrote: My general idea is first to get the predicted quantity of combinationsSeems like you already know; your OP says you have 2^n combinations.so I can divide and parallelize them.std.parallelism.parallel can do this for you, and works on ranges.What do you think can be the problem with the lack of deallocation?Don't know exactly. destroy just runs the destructors, it doesn't free any memory. Since the arrays are still in scope, it might not be able to free them.
Apr 06 2016
On 4/6/16 3:54 PM, Jonathan Villa wrote:I wrote a little program that given some number it generates a list of different combinations (represented by a ubyte array), so in the end my function with name GenerateCombinations(int x) returns a ubyte[][] (list of arrays of ubytes). Now the problem is, the quantity of combinations generated are pow(2, `x`), thus, giving 20 it returns me a list of 1_048_576 arrays of ubyte, every array has length of `x` so, in the end this function allocates (seeing the Windows Task Manager) 42 MB of data after input `20` the first time. When the list is ready it prints the quantity of combinations given (combinations.length), after that I added a foreach(ubyte[] a; combinations) calling destroy to every array and then destroy(combinations), and before to re-input a new number to generate a new combination list I call GC.Collect() and GC.minimize(). My question is if I'm doing good calling destroy to those arrays, because I don't see any change in the quantity of memory the program is using: In the start it uses 1496 KB, after input `20` it now have 43000 KB (approx.) and if I input again 20 then it sums up to 90000 KB. In the end I don't see some kind of real memory free. I'm doing something wrong? That's the reason I'm asking for your advice. PD: All arrays are constructed with the `new` expression. PD2: Don't ask me why I don't better do an `int count` instead of constructing arrays, this little program is going to be part of a more huge application and I need the combination list. Regards. JVYou are likely running into the GC being conservative. You are also possibly running into an issue where you are expecting the compiler to do something with the stack where it may do something else. A classic question that is asked on D forums all the time is why when you null some variable and then call GC collect, the memory isn't collected. The answer is because quite possibly the value is still in a register, and the registers must be scanned too. As for conservatism, if you generate a 1-million element array of pointers (each element has a pointer in it), then you have some random stack variable that "happens" to point into that giant array (more likely in 32-bit systems), then both the array, and all it's pointing at gets saved. Note that destroying an array is equivalent to setting it to null. It doesn't write any data to the array itself. Ironically, your example of this: foreach(ubyte[] a; combs) destroy(a); does absolutely nothing. Because all you are doing is setting a to null, and a is a local variable. The only thing that could make a difference is destroying combs, which is equivalent to setting combs to null. But like I said, it's possible there is still a pointer in registers, or that some other stack variable points at your huge array. -Steve
Apr 07 2016
On Thursday, 7 April 2016 at 16:13:59 UTC, Steven Schveighoffer wrote:On 4/6/16 3:54 PM, Jonathan Villa wrote: You are likely running into the GC being conservative. You are also possibly running into an issue where you are expecting the compiler to do something with the stack where it may do something else. A classic question that is asked on D forums all the time is why when you null some variable and then call GC collect, the memory isn't collected. The answer is because quite possibly the value is still in a register, and the registers must be scanned too. As for conservatism, if you generate a 1-million element array of pointers (each element has a pointer in it), then you have some random stack variable that "happens" to point into that giant array (more likely in 32-bit systems), then both the array, and all it's pointing at gets saved. -SteveGood, I compiled the program in 64 bit mode (-m64) and now it release memory normally like in Debian. I would like to know where can I learn more about the 'register' or the GC for the D so I can avoid future issues. JV.
Apr 08 2016
On 4/8/16 11:08 AM, Jonathan Villa wrote:On Thursday, 7 April 2016 at 16:13:59 UTC, Steven Schveighoffer wrote:Definitely, 64-bit is less likely to randomly point at memory. This isn't 100% sure-fire way to fix the problem though.On 4/6/16 3:54 PM, Jonathan Villa wrote: You are likely running into the GC being conservative. You are also possibly running into an issue where you are expecting the compiler to do something with the stack where it may do something else. A classic question that is asked on D forums all the time is why when you null some variable and then call GC collect, the memory isn't collected. The answer is because quite possibly the value is still in a register, and the registers must be scanned too. As for conservatism, if you generate a 1-million element array of pointers (each element has a pointer in it), then you have some random stack variable that "happens" to point into that giant array (more likely in 32-bit systems), then both the array, and all it's pointing at gets saved.Good, I compiled the program in 64 bit mode (-m64) and now it release memory normally like in Debian.I would like to know where can I learn more about the 'register' or the GC for the D so I can avoid future issues.Your best bet is to free the memory itself if it's possible. import core.memory: GC; GC.free(combs.ptr); For more info on how GC works: dlang.org/spec/garbage.html -Steve
Apr 08 2016
On Friday, 8 April 2016 at 15:21:50 UTC, Steven Schveighoffer wrote:On 4/8/16 11:08 AM, Jonathan Villa wrote:Thank you very much! I think this is the answer I was looking for C: JVOn Thursday, 7 April 2016 at 16:13:59 UTC, Steven Schveighoffer wrote:Your best bet is to free the memory itself if it's possible. import core.memory: GC; GC.free(combs.ptr); For more info on how GC works: dlang.org/spec/garbage.html -Steve
Apr 08 2016