www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - is it a bug or what?

reply "collerblade" <cccc cc.cc> writes:
Dear D user, i have this code:

import core.sys.windows.windows;

real[] a;
while(1) {
   a.length=4096*4096;
   a=null;
		
   Sleep(2000);
}

It allocates memory, but its never gets freed, just keep going 
up, and after 10 secs, memoryexception is thrown. I checked the 
pointer it is GC safe (attributes is 10). Am i missing something?

TY: collerblade
Jan 22 2015
next sibling parent reply "collerblade" <cccc cc.cc> writes:
Also forget to mention, that im using DMD 2.066.1.
If i alloc memory with GC.malloc(..), it is handled correcty (and 
freed).
Why GC doesn't frees dynamic arrays?
Jan 22 2015
parent "collerblade" <cccc cc.cc> writes:
i tested malloc, and it also run out of mem, except when i 
manually free the pointer.
There is no GC im DMD?
Jan 22 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/22/15 8:44 AM, collerblade wrote:
 Dear D user, i have this code:

 import core.sys.windows.windows;

 real[] a;
 while(1) {
    a.length=4096*4096;
    a=null;

    Sleep(2000);
 }

 It allocates memory, but its never gets freed, just keep going up, and
 after 10 secs, memoryexception is thrown. I checked the pointer it is GC
 safe (attributes is 10). Am i missing something?
Are you compiling 64-bit code or 32-bit? It makes a huge difference. A real is going to be 16-byte aligned. This means real[4096*4096] is allocating 1/4GB per allocation. This consumes 1/16th of your total address space on 32-bit address space. The GC in D is conservative. This means that on 32-bit architecture, any 4-byte segment scanned by the GC (whether it's a pointer or not) that happens to point into the array you allocated is going to keep that memory pinned. Because any 4-byte garbage has a 1:16 chance of pointing there, odds are good that this will happen. Then it keeps happening, you eventually run out of memory. Now, some possible explanation of why GC.malloc might work while this does not: GC.malloc allocates exactly 4096*4096*16 bytes. Setting a.length goes through the append code. Because extending length is considered an appending event, it *over-allocates* by a certain factor, expecting you to keep appending. (this is how appending can be amortized O(1) performance). This means the amount of memory allocated is larger than 1/4GB. Therefore the chance of failing by setting a.length is going to be higher than with using GC.malloc directly, but it could fail with either case. Answers? 1. Manually free such a large array when done with it. 2. Do not allocate such large arrays. 3. Use 64-bit code generation (-m64 on DMD windows, even if you are on Win64) if possible. -Steve
Jan 22 2015
parent reply "collerblade" <cccc cc.cc> writes:
On Thursday, 22 January 2015 at 14:19:10 UTC, Steven 
Schveighoffer wrote:
 On 1/22/15 8:44 AM, collerblade wrote:
 Dear D user, i have this code:

 import core.sys.windows.windows;

 real[] a;
 while(1) {
   a.length=4096*4096;
   a=null;

   Sleep(2000);
 }

 It allocates memory, but its never gets freed, just keep going 
 up, and
 after 10 secs, memoryexception is thrown. I checked the 
 pointer it is GC
 safe (attributes is 10). Am i missing something?
Are you compiling 64-bit code or 32-bit? It makes a huge difference. A real is going to be 16-byte aligned. This means real[4096*4096] is allocating 1/4GB per allocation. This consumes 1/16th of your total address space on 32-bit address space. The GC in D is conservative. This means that on 32-bit architecture, any 4-byte segment scanned by the GC (whether it's a pointer or not) that happens to point into the array you allocated is going to keep that memory pinned. Because any 4-byte garbage has a 1:16 chance of pointing there, odds are good that this will happen. Then it keeps happening, you eventually run out of memory. Now, some possible explanation of why GC.malloc might work while this does not: GC.malloc allocates exactly 4096*4096*16 bytes. Setting a.length goes through the append code. Because extending length is considered an appending event, it *over-allocates* by a certain factor, expecting you to keep appending. (this is how appending can be amortized O(1) performance). This means the amount of memory allocated is larger than 1/4GB. Therefore the chance of failing by setting a.length is going to be higher than with using GC.malloc directly, but it could fail with either case. Answers? 1. Manually free such a large array when done with it. 2. Do not allocate such large arrays. 3. Use 64-bit code generation (-m64 on DMD windows, even if you are on Win64) if possible. -Steve
TY for your answer. I checked malloc and it is the same. It runs out if memory too. Also tried with int[]->result is the same. Memory is initialized to 0-s, so there is no pointer pointing to memory. At least at my side. Are u saiying, that most likely other GC allocated memory points to my arrays? Hm thats maybe true. If thats a case i have to manage memory myself. Also what about scpoed alloc? scoped int[] a=new int[whatever_big_size]; It sould free memory but it does not :( :(. D is soo convinient. i never wanna use malloc again :D :D :D
Jan 22 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/22/15 9:40 AM, collerblade wrote:
 On Thursday, 22 January 2015 at 14:19:10 UTC, Steven Schveighoffer wrote:
 On 1/22/15 8:44 AM, collerblade wrote:
 Dear D user, i have this code:

 import core.sys.windows.windows;

 real[] a;
 while(1) {
   a.length=4096*4096;
   a=null;

   Sleep(2000);
 }

 It allocates memory, but its never gets freed, just keep going up, and
 after 10 secs, memoryexception is thrown. I checked the pointer it is GC
 safe (attributes is 10). Am i missing something?
Are you compiling 64-bit code or 32-bit? It makes a huge difference. A real is going to be 16-byte aligned. This means real[4096*4096] is allocating 1/4GB per allocation. This consumes 1/16th of your total address space on 32-bit address space. The GC in D is conservative. This means that on 32-bit architecture, any 4-byte segment scanned by the GC (whether it's a pointer or not) that happens to point into the array you allocated is going to keep that memory pinned. Because any 4-byte garbage has a 1:16 chance of pointing there, odds are good that this will happen. Then it keeps happening, you eventually run out of memory. Now, some possible explanation of why GC.malloc might work while this does not: GC.malloc allocates exactly 4096*4096*16 bytes. Setting a.length goes through the append code. Because extending length is considered an appending event, it *over-allocates* by a certain factor, expecting you to keep appending. (this is how appending can be amortized O(1) performance). This means the amount of memory allocated is larger than 1/4GB. Therefore the chance of failing by setting a.length is going to be higher than with using GC.malloc directly, but it could fail with either case. Answers? 1. Manually free such a large array when done with it. 2. Do not allocate such large arrays. 3. Use 64-bit code generation (-m64 on DMD windows, even if you are on Win64) if possible. -Steve
TY for your answer.
Before I respond, first question to answer is if you are using 32-bit compiled code. If not, then it's not typical for what you are doing to cause problems.
 I checked malloc and it is the same. It runs out if memory too.
 Also tried with int[]->result is the same.
 Memory is initialized to 0-s, so there is no pointer pointing to memory.
 At least at my side. Are u saiying, that most likely other GC allocated
 memory points to my arrays? Hm thats maybe true. If thats a case i have
 to manage memory myself.
Most likely it's a stack variable or global/TLS variable. Those are scanned too :) Also note that reals are NOT initialized to 0, but NaN.
 Also what about scpoed alloc?

 scoped int[] a=new int[whatever_big_size];

 It sould free memory but it does not :( :(.
I would not use this, I think it's scheduled for deprecation, or at least the meaning is scheduled to change. Note that a must go out of scope pointing at the array that was allocated in order to free the memory. You can't set it to null, because then nothing would be freed! -Steve
Jan 22 2015
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Steven Schveighoffer:

 scoped int[] a=new int[whatever_big_size];
 ...
I would not use this, I think it's scheduled for deprecation, or at least the meaning is scheduled to change.
As D introduces some limited static management of memory ownership, the semantics of scoped arrays could be fixed. Bye, bearophile
Jan 22 2015
prev sibling parent "collerblade" <cccc cc.cc> writes:
Thank all of u,
im going to use malloc.
Jan 22 2015