www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - I want my Memory back ;-)

reply ABothe <info alexanderbothe.com> writes:
Hi everyone,

I don't know if it's a bug or not .. but D or at least its GC doesn't want to
free my allocated stuff! Also when I 'disable' the GC via GC.disable(); it
won't free it :-/

byte[] asdf=new byte[50_000_000]; // OK a bit much---but it's an example value

GC.free(&asdf); // It doesn't work!!
delete asdf; // Also, without GC.free, doesn't work!!!

So...please give me a solution for this, D people :-D


--Alex
Jul 12 2010
next sibling parent dsimcha <dsimcha yahoo.com> writes:
== Quote from ABothe (info alexanderbothe.com)'s article
 Hi everyone,
 I don't know if it's a bug or not .. but D or at least its GC doesn't want to
 free my allocated stuff! Also when I 'disable' the GC via GC.disable(); it
 won't free it :-/
 byte[] asdf=new byte[50_000_000]; // OK a bit much---but it's an example value
 GC.free(&asdf); // It doesn't work!!
 delete asdf; // Also, without GC.free, doesn't work!!!
 So...please give me a solution for this, D people :-D
 --Alex

GC.free(cast(void*) asdf.ptr);
Jul 12 2010
prev sibling next sibling parent reply Leandro Lucarella <luca llucax.com.ar> writes:
ABothe, el 12 de julio a las 23:44 me escribiste:
 Hi everyone,
 
 I don't know if it's a bug or not .. but D or at least its GC doesn't want to
 free my allocated stuff! Also when I 'disable' the GC via GC.disable(); it
 won't free it :-/

Disable that exactly that, force the GC *not* to run a collection.
 byte[] asdf=new byte[50_000_000]; // OK a bit much---but it's an example value

This is a clasical problem of the conservative GC. The problem is, false pointers would keep your large chunks of memory alive (the bigger the chunk, the higher the probability of a random word being confused as a pointer into it). There were some solutions proposed, but none accepted yet: http://d.puremagic.com/issues/show_bug.cgi?id=2927 http://d.puremagic.com/issues/show_bug.cgi?id=3463
 GC.free(&asdf); // It doesn't work!!
 delete asdf; // Also, without GC.free, doesn't work!!!

Those 2 *must* work. What are you doing to check if they work? -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- There is no pain you are receding A distant ship, smoke on the horizon. You are only coming through in waves. Your lips move but I can't hear what you're saying.
Jul 12 2010
next sibling parent BCS <none anon.com> writes:
Hello Leandro,

 ABothe, el 12 de julio a las 23:44 me escribiste:
 
 GC.free(&asdf); // It doesn't work!!
 delete asdf; // Also, without GC.free, doesn't work!!!


If, by chance, you are looking at the memory usage of your program (i.e. the amount of space the OS has mapped) as reported by the OS or the like, then you won't see anything as the GC never returns memory to the OS, OTOH unless you are VERY tight on memory, this will have negligible effect on most modern computers. -- ... <IXOYE><
Jul 12 2010
prev sibling parent reply ABothe <info asdf.com> writes:
First, thanks for the response!

When executing the program at your own, look at the task
manager... It simply doesn't do it!


Is there somebody who can make it possible for the GC to give
allocated memory right back??
Jul 13 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
ABothe:
 Is there somebody who can make it possible for the GC to give
 allocated memory right back??

In Python generally this doesn't happen. Bye, bearophile
Jul 13 2010
parent reply ABothe <info asdf.com> writes:
So there isn't any other method than calling C's free(); or malloc
(); ?
Jul 13 2010
next sibling parent reply FeepingCreature <default_357-line yahoo.de> writes:
On 13.07.2010 12:28, ABothe wrote:
 So there isn't any other method than calling C's free(); or malloc
 (); ?

That's how it is.
Jul 13 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
FeepingCreature:
 On 13.07.2010 12:28, ABothe wrote:
 So there isn't any other method than calling C's free(); or malloc
 (); ?

That's how it is.

But you can build some abstraction by yourself, for example inside a struct you can keep the pointer to a memory block in the C heap, and in the ~this() of the struct you can put the free(). Then D has to give you a way to disallow the "new" on this struct, to be sure its destructor is called when it goes out of scope (or the object it's contained into gets deallocated). Bye, bearophile
Jul 13 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 07/13/2010 03:36 PM, bearophile wrote:
 FeepingCreature:
 On 13.07.2010 12:28, ABothe wrote:
 So there isn't any other method than calling C's free(); or malloc
 (); ?

That's how it is.

But you can build some abstraction by yourself, for example inside a struct you can keep the pointer to a memory block in the C heap, and in the ~this() of the struct you can put the free(). Then D has to give you a way to disallow the "new" on this struct, to be sure its destructor is called when it goes out of scope (or the object it's contained into gets deallocated). Bye, bearophile

Also refer to RefCounted: http://www.dsource.org/projects/phobos/browser/trunk/phobos/std/typecons.d#L1991 Highly recommended. Andrei
Jul 13 2010
parent ABothe <info asdf.com> writes:
So it's not planned by you or Walter to let the GC be able to free
some memory? I mean this would be very helpful and would make
large programs possible
(Especially those ones are often working with a huge data
amount...think of running that program for a while...the memory
consumption would be enormously high:-/ and mess up the whole
system)
Jul 13 2010
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"ABothe" <info asdf.com> wrote in message 
news:i1hf3h$7sd$1 digitalmars.com...
 So there isn't any other method than calling C's free(); or malloc
 (); ?

GC.minimize() should do it: http://www.digitalmars.com/d/2.0/phobos/std_gc.html If it doesn't, then there's probably something holding on a reference to it (like the LRU cache Steven mentioned), or something that the GC mistakenly thinks is a reference to it. ------------------------------- Not sent from an iPhone.
Jul 13 2010
parent "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:i1iaan$1nbe$1 digitalmars.com...
 "ABothe" <info asdf.com> wrote in message 
 news:i1hf3h$7sd$1 digitalmars.com...
 So there isn't any other method than calling C's free(); or malloc
 (); ?

GC.minimize() should do it: http://www.digitalmars.com/d/2.0/phobos/std_gc.html If it doesn't, then there's probably something holding on a reference to it (like the LRU cache Steven mentioned), or something that the GC mistakenly thinks is a reference to it.

But, in a GC'ed environment, you shouldn't normally be trying to release memory back to the OS every time you delete something. In most cases, the GC deliberately saves freed memory to return it back to you the next time you allocate. So if you want to see whether or not you're *really* leaking, try something more like this: while(true) { byte[] asdf=new byte[50_000_000]; GC.free(cast(void*) asdf.ptr); } Then monitor the memory usage reported by the OS. If it keeps climbing higher and higher and higher, then something's going wrong. If it stays pretty much constant, then everything's working correctly.
Jul 13 2010
prev sibling parent =?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:
Content-Type: multipart/mixed;
 boundary="------------070005050409000208000703"

This is a multi-part message in MIME format.
--------------070005050409000208000703
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

ABothe wrote:
 So there isn't any other method than calling C's free(); or malloc
 (); ?

the memory to the OS either (*). The only way to be sure that the memory will be returned is to call mmap/munmap (or their equivalent on whatever OS you're using). (*) Most malloc implementations allocate large chunks of memory using mmap, which can return the memory to the OS, but they allocate small chunks with sbrk, which can't. As proof, see the attached C code which prints: At the beginning: VmSize: 3872 kB After allocating the memory: VmSize: 36608 kB After releasing the memory: VmSize: 36608 kB Jerome --=20 mailto:jeberger free.fr http://jeberger.free.fr Jabber: jeberger jabber.fr --------------070005050409000208000703 Content-Type: text/x-csrc; name="test.c" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline; filename="test.c" #define _GNU_SOURCE #include <stdio.h> #include <stdlib.h> #include <unistd.h> int startsWith (const char* s1, const char* s2) { while ((*s1 !=3D 0) && (*s2 !=3D 0) && (*s1 =3D=3D *s2)) { ++s1; ++s2; } return (*s2 =3D=3D 0); } void printMemInfo (void) { char* filename; FILE* status; char buffer[128]; asprintf (&filename, "/proc/%d/status", getpid()); status =3D fopen (filename, "r"); free (filename); while (fgets (buffer, sizeof (buffer), status) !=3D NULL) if (startsWith (buffer, "VmSize")) { printf ("%s", buffer); break; } fclose (status); } void** allocate (int n) { void** result =3D NULL; int i; for (i =3D 0; i < n; ++i) { void** next =3D malloc (sizeof (void**)); *next =3D result; result =3D next; } return result; } void release (void** list) { while (list !=3D NULL) { void** next =3D *list; free (list); list =3D next; } } int main (int argc, char** argv) { void** list; printf ("At the beginning:\n"); printMemInfo(); list =3D allocate (1024*1024); printf ("After allocating the memory:\n"); printMemInfo(); release (list); printf ("After releasing the memory:\n"); printMemInfo(); return 0; } --------------070005050409000208000703--
Jul 14 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 Jul 2010 03:50:19 -0400, ABothe <info asdf.com> wrote:

 First, thanks for the response!

 When executing the program at your own, look at the task
 manager... It simply doesn't do it!


 Is there somebody who can make it possible for the GC to give
 allocated memory right back??

The GC does not release memory back to the OS in all circumstances, it may hold onto the memory, even though it isn't allocated. Part of the reason for this is it may just not be possible. IIRC, on Linux systems, all your heap-allocated memory must be contiguous, meaning that if you deallocate holes, you can't release them back to the OS. There is also a design bug in the array allocator (which you probably have not encountered, since you did not append), where it will hold onto memory even if nothing is referencing it because the LRU cache is referencing it. I am going to fix that in the near future. -Steve
Jul 13 2010
prev sibling next sibling parent Leandro Lucarella <luca llucax.com.ar> writes:
ABothe, el 13 de julio a las 07:50 me escribiste:
 First, thanks for the response!
 
 When executing the program at your own, look at the task
 manager... It simply doesn't do it!
 
 Is there somebody who can make it possible for the GC to give
 allocated memory right back??

Calling gc.minimize() should free memory (as give it back to the OS) if there are any pools completely free, but IFAIK is only implemented in Tango and D2 (i.e., not implemented in Phobos/D1). -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- La esperanza es una amiga que nos presta la ilusión.
Jul 13 2010
prev sibling next sibling parent Leandro Lucarella <luca llucax.com.ar> writes:
ABothe, el 13 de julio a las 23:20 me escribiste:
 So it's not planned by you or Walter to let the GC be able to free
 some memory? I mean this would be very helpful and would make
 large programs possible
 (Especially those ones are often working with a huge data
 amount...think of running that program for a while...the memory
 consumption would be enormously high:-/ and mess up the whole
 system)

gc.minimize() is supposed to do that. But if you really need to work with huge datasets, I'd recommend to do manual memory management anyways, because of the current limitations of the GC. -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- Hoy estuvimos en el museo de antropología, pero yo voy a volver para estar por lo menos un día ahí adentro... es una locura, como Disney pero de indigenas -- Carla Lucarella (10/2008 contando de su viaje a México)
Jul 14 2010
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tue, 13 Jul 2010 02:44:49 +0300, ABothe <info alexanderbothe.com> wrote:

 Hi everyone,

 I don't know if it's a bug or not .. but D or at least its GC doesn't  
 want to
 free my allocated stuff! Also when I 'disable' the GC via GC.disable();  
 it won't free it :-/

Alex, please have a look at the module I posted to D.announce. It allows working with unmanaged memory without having to worry about manual deallocation, etc. For example: import std.stdio; import std.gc; import data; void main() { Data d = new Data(50_000_000); readln(); // check my memory usage now d = null; fullCollect(); readln(); // check my memory usage now } You can get my module from here: http://github.com/CyberShadow/data.d -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Jul 14 2010