www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How to detect free/unfree memory segments?

reply Suliman <evermind live.ru> writes:
I would like to visualize how GC works and display free/not free 
memory segments.
How I can understand which of them are used and which not?

Could anybody explain what dangerous of memory fragmentation in 
languages without GC? Am I right understand that there is stay 
some small memory chunks that very hard to reuse?
Dec 22 2016
next sibling parent Nemanja Boric <4burgos gmail.com> writes:
I can just partially answer this part of the question:


 Could anybody explain what dangerous of memory fragmentation in 
 languages without GC? Am I right understand that there is stay 
 some small memory chunks that very hard to reuse?


On Unix-like systems, system call for allocating memory is 
traditionally `brk`/`sbrk` which just moves "program break" which 
indicates how much memory program has allocated.

Say that you're running your application, and your heap looks 
like this:

            4096
-------------------
|  heap  [myarr]  |
-------------------
                   ^
                   |
              program break

now you call `sbrk(2 * 4096)` as you need  which will allocate 
2*4096 for you:

            4096
--------------------------------------
|  heap  [myarr]    [my new array]   |
--------------------------------------
                                      ^
                                      |
                               program break

Now, if you want to discard [myarr], it will leave a hole in 
heap, since you don't want to move `[my new array]`. This hole 
can be reused, but sometimes it can't, depending on the size, etc.

That's why there's another system call you can use: mmap, which 
you would use to allocate space for `my new array`, 
non-contiguously and not adjacent to `myarr`,  and so when you 
discard myarr, you don't have fragmentation issue:

            4096
------------------                   --------------------
|  heap  [myarr]  |                  |  [my new array]   |
------------------                   --------------------
                   ^
                   |
                 program break


These, and many more facts (reusing freed area, when to use brk, 
when to use mmap, locking contention detection, for example) are 
used by the standard memory allocators you would use in non-GC 
program (such is malloc), so while it is possible that you will 
have memory fragmentation problems, it is very unlikely, unless 
you're writing something that's very different than vast majority 
of the programs. And in that case, there are different allocators 
that you can use, or you can write your own.

Nemanja

On Friday, 23 December 2016 at 06:18:02 UTC, Suliman wrote:

 I would like to visualize how GC works and display free/not 
 free memory segments.
 How I can understand which of them are used and which not?

 Could anybody explain what dangerous of memory fragmentation in 
 languages without GC? Am I right understand that there is stay 
 some small memory chunks that very hard to reuse?
Dec 23 2016
prev sibling parent reply jkpl <jkpl nowhere.de> writes:
On Friday, 23 December 2016 at 06:18:02 UTC, Suliman wrote:

 I would like to visualize how GC works and display free/not 
 free memory segments.
 How I can understand which of them are used and which not?

 Could anybody explain what dangerous of memory fragmentation in 
 languages without GC? Am I right understand that there is stay 
 some small memory chunks that very hard to reuse?


You start with a wrong assumption. The C malloc functions is not 
just a nasty and mean memory provider. Several implementations 
uses internally freelists. Which means that the gaps created by a 
free() may be filled again.

For example
- TCMallocator use free lists
- GCC C malloc use free lists (as stated here 
http://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html)
- for snn.lib malloc (used by DMD win32) I can't say.
Dec 23 2016
parent reply Suliman <evermind live.ru> writes:
On Saturday, 24 December 2016 at 01:15:43 UTC, jkpl wrote:

 On Friday, 23 December 2016 at 06:18:02 UTC, Suliman wrote:

 I would like to visualize how GC works and display free/not 
 free memory segments.
 How I can understand which of them are used and which not?

 Could anybody explain what dangerous of memory fragmentation 
 in languages without GC? Am I right understand that there is 
 stay some small memory chunks that very hard to reuse?


 You start with a wrong assumption. The C malloc functions is 
 not just a nasty and mean memory provider. Several 
 implementations uses internally freelists. Which means that the 
 gaps created by a free() may be filled again.

 For example
 - TCMallocator use free lists
 - GCC C malloc use free lists (as stated here 
 http://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html)
 - for snn.lib malloc (used by DMD win32) I can't say.


So there is no any problems with memory fragmentation and all 
used memory arr successfuly reuse?
Dec 23 2016
parent Adam Wilson <flyboynw gmail.com> writes:
On 12/23/16 8:39 PM, Suliman wrote:

 On Saturday, 24 December 2016 at 01:15:43 UTC, jkpl wrote:

 On Friday, 23 December 2016 at 06:18:02 UTC, Suliman wrote:

 I would like to visualize how GC works and display free/not free
 memory segments.
 How I can understand which of them are used and which not?

 Could anybody explain what dangerous of memory fragmentation in
 languages without GC? Am I right understand that there is stay some
 small memory chunks that very hard to reuse?


 You start with a wrong assumption. The C malloc functions is not just
 a nasty and mean memory provider. Several implementations uses
 internally freelists. Which means that the gaps created by a free()
 may be filled again.

 For example
 - TCMallocator use free lists
 - GCC C malloc use free lists (as stated here
 http://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html)

 - for snn.lib malloc (used by DMD win32) I can't say.


 So there is no any problems with memory fragmentation and all used
 memory arr successfuly reuse?


Strictly speaking fragmentation is a problem in D regardless of GC/nogc. 
The *only* way to reduce fragmentation is to use a compaction algorithm 
of some kind. Again this is separate from GC/nogc, because you could 
lock a memory pool you made and apply the same compaction algorithm, as 
opposed to using a GC algo.

For example, lets say you have the following unfragmented memory layout:
1111111122222333344444444444445555

Now delete the "four" memory:
11111111222223333-------------5555

And allocate a new "six" object, which just happens to fit inside the 
the space available on the freelist:
111111112222233336666666666---5555

Note the three dashes. Those will go unused until such time as an object 
small enough to fit inside is allocated. However, what happens if the 
object is not exactly the space available?

11111111222223333666666666677-5555

The dashes are what is commonly referred to as fragmentation. In that, 
small fragments of memory go unused because nothing fits inside them.

With a Precise GC you can implement a compacting collector that reorders 
blocks of memory within the address space to reduce or eliminate 
fragmentation.

-- 
Adam Wilson
IRC: LightBender
//quiet.dlang.dev
Dec 30 2016