www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - When is it time for a 1.0 feature freeze?

reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Among the many as-yet-unfulfilled prerequisites for being ready for 1.0 
is coming to a final decision on the feature set.

On releasing 0.93, Walter stated:

"Barring a very, very, compelling case, this is it for 1.0 language 
features.  There are a lot more things I want to do, but this has got to 
be enough for 1.0."

However, there have been lots of new features since then.  And as new 
features are added, it opens the door to more bugs and more 
imperfections in the spec.

I think that sometime soon, we should finally make it more firm that 
"this is it for 1.0 language features".  Once this is done, then we can 
finally concentrate more on fixing compiler bugs and getting the spec up 
to scratch (both cleaning up spelling/grammar and cleaning up what is 
ill-defined) without having to worry too much about creating even more 
work for ourselves in the process.

Stewart.
Aug 21 2006
parent reply "Søren J. Løvborg" <web kwi.dk> writes:
Stewart Gordon wrote:
 I think that sometime soon, we should finally make it more firm that "this 
 is it for 1.0 language features".  Once this is done, then we can finally 
 concentrate more on fixing compiler bugs and getting the spec up to 
 scratch (both cleaning up spelling/grammar and cleaning up what is 
 ill-defined) without having to worry too much about creating even more 
 work for ourselves in the process.

The only thing I can think off that should be fixed before 1.0 is the "auto" keyword. It's current double meaning is a mess, and it needs to be defined as either a RAII or a type deduction keyword (and another keyword/syntax needs to be introduced for the other meaning.) I'm leaning towards making "auto" mean type deduction, and introduce another keyword ("raii"? Unlikely to break old code, I'd venture) for RAII allocations. That'd be a change from the C meaning of "auto", but so is the D meaning of "typedef" (since C's "typedef" is D's "alias"), and "auto" is practically never used in C, even. Søren J. Løvborg web kwi.dk
Aug 21 2006
parent reply Fredrik Olsson <peylow gmail.com> writes:
Søren J. Løvborg skrev:
 Stewart Gordon wrote:
 I think that sometime soon, we should finally make it more firm that "this 
 is it for 1.0 language features".  Once this is done, then we can finally 
 concentrate more on fixing compiler bugs and getting the spec up to 
 scratch (both cleaning up spelling/grammar and cleaning up what is 
 ill-defined) without having to worry too much about creating even more 
 work for ourselves in the process.

The only thing I can think off that should be fixed before 1.0 is the "auto" keyword.

foo.bar += 42; Array literals: bar ~= [1,5,42]; Struct literals: baz = {"hej", 42}; Those and the lack of sets and ranges are top on my list. The small stuff that can reduce code size in half. The only 100% foolproof way to write bug-free code is; not to write any. So the closer to no code, the better :). // Fredrik Olsson
Aug 31 2006
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Fredrik Olsson wrote:
<snip>
 I can also think of allowing properties as lvalue:
 foo.bar += 42;
 
 Array literals:
 bar ~= [1,5,42];

The problem with that notation is that, in the general case, the type of the array cannot be guaranteed. There have been a number of proposed syntaxes; AFAIK the best so far is http://www.digitalmars.com/d/archives/digitalmars/D/39125.html So your answer to the subject of this thread is "Once these features have been added", right? Could be a long time.... :-) <snip>
 Those and the lack of sets and ranges are top on my list. The small 
 stuff that can reduce code size in half. The only 100% foolproof way to 
 write bug-free code is; not to write any. So the closer to no code, the 
 better :).

How does not writing code equate to writing code? But you remind me of this: http://www.stevemcconnell.com/cctune.htm (See from "A fast program is just as important as a correct one--False!" downwards.) Stewart. -- -----BEGIN GEEK CODE BLOCK----- Version: 3.1 GCS/M d- s:- C++ a->--- UB P+ L E W++ N+++ o K- w++ O? M V? PS- PE- Y? PGP- t- 5? X? R b DI? D G e++++ h-- r-- !y ------END GEEK CODE BLOCK------ My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Sep 01 2006
parent reply Ivan Senji <ivan.senji_REMOVE_ _THIS__gmail.com> writes:
Stewart Gordon wrote:
 Fredrik Olsson wrote:
 <snip>
 I can also think of allowing properties as lvalue:
 foo.bar += 42;

 Array literals:
 bar ~= [1,5,42];

The problem with that notation is that, in the general case, the type of the array cannot be guaranteed. There have been a number of proposed syntaxes; AFAIK the best so far is http://www.digitalmars.com/d/archives/digitalmars/D/39125.html

I agree! That is the best proposal so far. I know Walter wants that to be a 2.0 feature but it still doesn't make sense to me. Array and struct initializers are more fundamental feature than templates, delegates, lazyness, even classes, and they are going to be missing from D1.0? Crazy! These days each D project I start has T[] array(T)(T[] x...) { return x.dup; } right after the module statement and that should be an indication that something is wrong. Although the array template fixes (kind of) the array initialization problem, there is nothing that can be done for structs (not counting static opCall as a solution).
Sep 01 2006
parent reply Don Clugston <dac nospam.com.au> writes:
Ivan Senji wrote:
 Stewart Gordon wrote:
 Fredrik Olsson wrote:
 <snip>
 I can also think of allowing properties as lvalue:
 foo.bar += 42;

 Array literals:
 bar ~= [1,5,42];

The problem with that notation is that, in the general case, the type of the array cannot be guaranteed. There have been a number of proposed syntaxes; AFAIK the best so far is http://www.digitalmars.com/d/archives/digitalmars/D/39125.html

I agree! That is the best proposal so far. I know Walter wants that to be a 2.0 feature but it still doesn't make sense to me. Array and struct initializers are more fundamental feature than templates, delegates, lazyness, even classes, and they are going to be missing from D1.0? Crazy!

Those things would delay D1.0 by months. The idea is to declare D1.0 quickly, and then move onto those things.
Sep 01 2006
parent reply Ivan Senji <ivan.senji_REMOVE_ _THIS__gmail.com> writes:
Don Clugston wrote:
 Ivan Senji wrote:
 Stewart Gordon wrote:
 http://www.digitalmars.com/d/archives/digitalmars/D/39125.html

I agree! That is the best proposal so far. I know Walter wants that to be a 2.0 feature but it still doesn't make sense to me. Array and struct initializers are more fundamental feature than templates, delegates, lazyness, even classes, and they are going to be missing from D1.0? Crazy!

Those things would delay D1.0 by months. The idea is to declare D1.0 quickly, and then move onto those things.

I know, and I wouldn't like that to happen but many other features have been delaying D for years (not saying that I don't like them). I don't have anything against D1.1 od D1.5 or even D2.0 having these features but I am a little worried about what happens when D1.0 attracts new people and they must very quickly realize "what? D doesn't have array and struct literals? What kind of a language is that?" Is that kind of negative publicity bad or not I don't know but it might make D feel incomplete. When you stay around D for a while you realize it isn't that incomplete after all but for some people the first negative impression is going to be the last impression they make of D.
Sep 01 2006
parent reply Serg Kovrov <kovrov no.spam> writes:
* Ivan Senji:
 I don't have anything against D1.1 od D1.5 or even D2.0 having these 
 features but I am a little worried about what happens when D1.0 attracts 
 new people and they must very quickly realize "what? D doesn't have 
 array and struct literals? What kind of a language is that?"

I'm more concerning about memory management scheme. Most people I was talking to about D (and I agitate very fellow programmer i personally know to at least try D)... firstly complains about 'yet another language' syndrome. There is really nothing can be done about it, you just have to convince them to not to stop discussion =) Next, everyone from C/C++ camp (especially C) is just so-skeptical about GC. All sort of things were already beaten to death here on NG. It is a meter of personal taste and there is nothing can't be done either. And then usually happens following: on my clumsy allegations that it is not necessary to use GC, and with malloc/free one could manage memory manually. Well, you know... Whom I trying to deceive, myself? It is simply not so convenient to actually do so. And what about string operations (concatenations, assignment etc), they use GC and i don't know how to avoid it. Plus, standard library is hard-coupled with GC. My point is that if not use GC programming with D is even more awkward then C/C++. But that not worst part. Although I'm personally from C++ camp and I get used to GC. and see it as good thing. But, there is one big BUT! Currently GC do not return memory to OS, and that thing even I can't subdue. The only reason I choose 'native' (as opposite to VM) programming language is effective resource usage. I hate Java/.net gui applications because their memory consuming. I really appreciate developers that choose c/c++ to write small, memory effective, but yet feature-reach applications like FileZilla Server, uTorrent or Miranda IM. Most desktop applications (like text editors, news readers, web browsers, etc) must coexist with each other. That is, to use memory wisely and give it back if it not needed anymore. This 'upper water mark' approach in GC is just not acceptable. All I actually can say to my fellow programmers in defense of D, is that language still developing and eventually all good things happens. On that point discussion usually ends and the bottom line is "nice language, we will look at later, when/if it evolve". And for GC, to fee memory back to OS is crucial to every one who care about memory management. And ones who do not - whey will stay with their Java/.net VM monsters and will newer even look at D. Besides, this type of developers do not care about anything but their CV. Sure thing D will not help them. P.S. sorry for rather long post. I don't expect it myself to be so =) -- serg.
Sep 01 2006
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Serg Kovrov wrote:
 * Ivan Senji:
 I don't have anything against D1.1 od D1.5 or even D2.0 having these 
 features but I am a little worried about what happens when D1.0 
 attracts new people and they must very quickly realize "what? D 
 doesn't have array and struct literals? What kind of a language is that?"

I'm more concerning about memory management scheme. Most people I was talking to about D (and I agitate very fellow programmer i personally know to at least try D)... firstly complains about 'yet another language' syndrome. There is really nothing can be done about it, you just have to convince them to not to stop discussion =) Next, everyone from C/C++ camp (especially C) is just so-skeptical about GC. All sort of things were already beaten to death here on NG. It is a meter of personal taste and there is nothing can't be done either. And then usually happens following: on my clumsy allegations that it is not necessary to use GC, and with malloc/free one could manage memory manually. Well, you know... Whom I trying to deceive, myself? It is simply not so convenient to actually do so. And what about string operations (concatenations, assignment etc), they use GC and i don't know how to avoid it.

The same as you would in C. Manually allocate and free the buffer referenced by the array ptr. The layout of dynamic arrays is specified in the ABI, so you're guaranteed to have a .len and .ptr property to play with if you really want to.
 Plus, standard library is hard-coupled with GC.

It doesn't have to be. That's just how Phobos was written.
 My
 point is that if not use GC programming with D is even more awkward then 
 C/C++.

True. Though I think 'scope' statements help a lot, and stack allocation will help tremendously. Walter has already said he plans to implement: MyClass c = new MyClass(); // heap alloc MyClass d = MyClass(); // stack alloc And hopefully: char[] a = new char[32]; // heap alloc char[] b = char[32]; // stack alloc This would do away with the need for alloca(), which would conflict with dynamic closures anyway. An in-language solution, however, may be compatible in some cases.
 But that not worst part. Although I'm personally from C++ camp and I get 
 used to GC. and see it as good thing. But, there is one big BUT! 
 Currently GC do not return memory to OS, and that thing even I can't 
 subdue.

This is just the default GC shipped with DMD (and GDC). There's no reason it can't be replaced. In fact, I've taken steps to make this as easy as possible in Ares.
 Most desktop applications (like text editors, news readers, web 
 browsers, etc) must coexist with each other. That is, to use memory 
 wisely and give it back if it not needed anymore. This 'upper water 
 mark' approach in GC is just not acceptable.

I think it depends on the application. If an application behaves consistently across their run cycle, there's no reason to return memory to the OS because it will likely be needed again. If an application allocates a ton of memory, frees it, and never uses it again then what you say is true. Though in these cases the application could call malloc/free for these one-time memory allocations as well.
 All I actually can say to my fellow programmers in defense of D, is that 
 language still developing and eventually all good things happens. On 
 that point discussion usually ends and the bottom line is "nice 
 language, we will look at later, when/if it evolve".

I think it's important to separate discussions of the language with the behavior of accompanying library code. The GC occupies kind of a strange middle-ground here, but I still consider it library code because nothing inside is required to be compiler-specific. Sure the compiler runtime allocates and frees memory, but it doesn't need special access that is not available to the user. Personally, I think D as a language is shaping up quite nicely. The library could perhaps use a little work, but that can be done without changing the language spec. Sean
Sep 01 2006
parent reply Serg Kovrov <kovrov no.spam> writes:
Sean Kelly wrote:
 Serg Kovrov wrote:
 Most desktop applications (like text editors, news readers, web 
 browsers, etc) must coexist with each other. That is, to use memory 
 wisely and give it back if it not needed anymore. This 'upper water 
 mark' approach in GC is just not acceptable.

I think it depends on the application. If an application behaves consistently across their run cycle, there's no reason to return memory to the OS because it will likely be needed again. If an application allocates a ton of memory, frees it, and never uses it again then what you say is true. Though in these cases the application could call malloc/free for these one-time memory allocations as well.

I can't imagine an application that shouldn't free (return to OS) memory in runtime. Perhaps only really 'heavy' monopoly applications (like games, 3d modeling packages) and 'run and die' tools (like grep, less). Other application that possibly could running unlimited time and process some significant amount of data, simply MUST return unused memory to OS. For example web browser - user could open up to several tens of web pages simultaneously. Some maybe enormously big. If browser will no free memory consumed in such peak moments use will be forced to restart application (e.g. experience inconveniences) or will experience slowdown caused by swapping. Same goes to any application working with arbitrary-sized data - image viewers, mail notificators, IDEs or text publishing tools, p2p clients or servers... Rally. As user, I don't want my everyday applications to hold peak memory, or restart them constantly. In the end i will chose other software, that will satisfy my habits. I found it silly to shape behavior of an application by language (or it standard library) limitations. If only programmer could have some control over GC, it would be just fine. For example to query GC's memory pool and decide to free some amount if it exceed some threshold. Walter, any comments on this? -- serg.
Sep 01 2006
parent Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:
 If only programmer could have some control over GC, it would be just 
 fine. For example to query GC's memory pool and decide to free some 
 amount if it exceed some threshold. Walter, any comments on this?

Because, as I said in another post, malloc/free/new/delete don't return memory to the operating system, what apps often do if they need to allocate/free some small number of very large buffers is to do them separately, usually with a call straight into the OS API. You can do the same thing in D, using the same technique and pretty much identical code.
Sep 02 2006
prev sibling parent reply Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:
 But that not worst part. Although I'm personally from C++ camp and I get 
 used to GC. and see it as good thing. But, there is one big BUT! 
 Currently GC do not return memory to OS, and that thing even I can't 
 subdue. The only reason I choose 'native' (as opposite to VM) 
 programming language is effective resource usage. I hate Java/.net gui 
 applications because their memory consuming. I really appreciate 
 developers that choose c/c++ to write small, memory effective, but yet 
 feature-reach applications like FileZilla Server, uTorrent or Miranda IM.
 
 Most desktop applications (like text editors, news readers, web 
 browsers, etc) must coexist with each other. That is, to use memory 
 wisely and give it back if it not needed anymore. This 'upper water 
 mark' approach in GC is just not acceptable.

1) Few programmers realize that C/C++'s malloc/free/new/delete *never* (and I emphasize NEVER) return memory to the operating system. All free/delete do is return memory to the memory pool managed by the runtime library, not the operating system. In order to actually return memory to the operating system, one has to write their own memory management code, which is a significant (and necessarily non-portable) effort. Hardly anyone does that. 2) The GC's memory pool doesn't consist of physical memory, it consists of virtual memory. If that virtual memory is not being used, the operating system automatically swaps it out and allocates the physical memory to whoever actually uses it. The 'memory' held by the GC isn't memory at all, but nothing more than a range of addresses. You can run out of virtual address space, but that's rare and usually indicates you've got a bug, such as an infinite loop allocating memory. 3) The GC could be modified to return unused address space to the operating system, that it doesn't do so now is not a limitation on D itself. 4) Given the exact same code, a gc'd program will consume roughly twice as much memory as the same explicitly allocated one (though neither will return memory to the operating system). But a program written to take advantage of GC can often use less memory (because it needs to make far fewer copies of things, and because it doesn't have to allocate extra memory management blocks like shared_ptr<T> does). 5) malloc/free/new/delete use an 'upper water mark' system of managing their internal free memory pools just like D's gc does.
Sep 02 2006
next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Walter Bright wrote:
<snip>
 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.

Never? I can't believe you've tried every implementation out there, including your own, and found the same. Or does the spec forbid free or delete to return memory to the OS? I can't for the life of me see why this would be. Stewart. -- -----BEGIN GEEK CODE BLOCK----- Version: 3.1 GCS/M d- s:- C++ a->--- UB P+ L E W++ N+++ o K- w++ O? M V? PS- PE- Y? PGP- t- 5? X? R b DI? D G e++++ h-- r-- !y ------END GEEK CODE BLOCK------ My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Sep 02 2006
next sibling parent Gregor Richards <Richards codu.org> writes:
Stewart Gordon wrote:
 Walter Bright wrote:
 <snip>
 
 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.

<snip> Never? I can't believe you've tried every implementation out there, including your own, and found the same. Or does the spec forbid free or delete to return memory to the OS? I can't for the life of me see why this would be. Stewart.

In general changing the amount a process wishes to use (the data segment size) involves calling [s]brk. brk just controls the pointer to the end of the memory you have available. You can't get data segments in chunks, only one contiguous block. Because of this, it's in general impossible for free/delete to return the memory to the OS, as it cannot move memory arbitrarily (or all the pointers would go funky), and it in all likelyhood has something allocated at the end of the data segment. I don't know whether it's in any spec that this is how it MUST work, but this is how it USUALLY works. There's no real reason to do otherwise, especially since changing it would make the data segmentation much more complicated. - Gregor Richards
Sep 02 2006
prev sibling next sibling parent Derek Parnell <derek psyc.ward> writes:
On Sun, 03 Sep 2006 01:45:25 +0100, Stewart Gordon wrote:

 Walter Bright wrote:
 <snip>
 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.

Never? I can't believe you've tried every implementation out there, including your own, and found the same. Or does the spec forbid free or delete to return memory to the OS? I can't for the life of me see why this would be. Stewart.

From memory, I believe the Manx-C for the Amiga used to return memory to the operating system. -- Derek Parnell Melbourne, Australia "Down with mediocrity!"
Sep 02 2006
prev sibling next sibling parent Walter Bright <newshound digitalmars.com> writes:
Stewart Gordon wrote:
 Walter Bright wrote:
 <snip>
 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.

Never? I can't believe you've tried every implementation out there, including your own, and found the same.

You're right, I haven't tried every implementation, and don't know it for a fact. It's just true of every one I've looked at, and there are 3 good reasons for it to be so: 1) Speed 2) Speed 3) Speed <g> If your app isrelying on free/delete returning memory to the operating system, you really need to take a good hard look at the implementation of it, because it isn't. Ironically, it's much more likely that a gc will return memory to the os - because a gc can be a copying collector, meaning it can compact the used memory and free the rest. free/delete cannot do any compaction, meaning it's unlikely to be able to release any memory back to the os due to fragmentation, minimum page sizes, and the high water nature of allocating memory to a task. It's often moot anyway, since as I wrote before, runtime heap management is holding on to virtual memory, not physical memory. A gc again has the potential advantage here, as it can compact the actual used memory into as few pages as possible, reducing physical memory consumption, and increasing the working set that will fit into the cache.
 Or does the spec forbid free or delete to return memory to the OS?  I
 can't for the life of me see why this would be.

The spec has nothing to say about it.
Sep 03 2006
prev sibling parent Sean Kelly <sean f4.ca> writes:
Stewart Gordon wrote:
 Walter Bright wrote:
 <snip>
 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.

Never? I can't believe you've tried every implementation out there, including your own, and found the same. Or does the spec forbid free or delete to return memory to the OS? I can't for the life of me see why this would be.

Memory can only be allocated and freed in pages, and a memory page can obviously only be returned to the OS if it is completely empty. Not to mention the issue of holes in what is typically supposed to be a contiguous address space. Some allocators will re-purpose empty pages if a different allocation size is needed, but I don't know of any that actually return memory to the OS. I think huge allocations may be an exception, as allocators typically obtain the memory for these in a slightly different manner. But even then I'm not sure the memory is "returned to the OS" on release in a manner that would actually decrease the process virtual memory footprint. Sean
Sep 03 2006
prev sibling next sibling parent reply Serg Kovrov <kovrov no.spam> writes:
Walter, thanks for your kind and comprehensive answer, but I still do 
not get this 'malloc/free do not return memory to OS' part.

I just tried simplest example with GCC(mingw): I allocated with 'new' 
1Mb string. Then in process explorer (or in FAR's Process list) I can 
see my process usage of 'Private bytes' increases equivalently. When 
free this string, with 'delete', i see that 'Private bytes' usage 
dropped to same level it was before. Isn't this mean that memory 
returned to OS?

And second example, I just tried open text files of different size with 
Catch22 Neatpad example (http://catch22.net/tuts/ - compiled with 
VC++2005), and see exactly same  behavior. Private bytes usage changes 
proportionally to opened file size. I've looked at source code an can 
see that for file buffer and support buffers allocation/deallocation 
used new/delete from standard library.

Could someone please, explain me where I do not understand Walter? Or 
maybe point to appropriate article/reference.

-- 
serg.
Sep 03 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:
 Walter, thanks for your kind and comprehensive answer, but I still do 
 not get this 'malloc/free do not return memory to OS' part.
 
 I just tried simplest example with GCC(mingw): I allocated with 'new' 
 1Mb string. Then in process explorer (or in FAR's Process list) I can 
 see my process usage of 'Private bytes' increases equivalently. When 
 free this string, with 'delete', i see that 'Private bytes' usage 
 dropped to same level it was before. Isn't this mean that memory 
 returned to OS?

Not necessarily. There's a difference between physical memory and virtual memory. The OS will automatically take away from the process any unused physical memory, although the process retains it as virtual memory. If the process explorer watches physical memory, then that's what it's seeing.
Sep 03 2006
next sibling parent reply Serg Kovrov <kovrov no.spam> writes:
Walter Bright wrote:
 Serg Kovrov wrote:
 I just tried simplest example with GCC(mingw): I allocated with 'new' 
 1Mb string. Then in process explorer (or in FAR's Process list) I can 
 see my process usage of 'Private bytes' increases equivalently. When 
 free this string, with 'delete', i see that 'Private bytes' usage 
 dropped to same level it was before. Isn't this mean that memory 
 returned to OS?

Not necessarily. There's a difference between physical memory and virtual memory. The OS will automatically take away from the process any unused physical memory, although the process retains it as virtual memory. If the process explorer watches physical memory, then that's what it's seeing.

I'm not trying to argue here, but to understand. I done some search on subject of memory usage. So far, I see it as I described before - to measure application's memory usage, best way is to watch 'private bytes' counter in Perfmon (or via other tools that (I believe) uses Perfmon, such as Process Explorer or FAR's Process list plugin). For example, quote from http://shsc.info/WindowsMemoryManagement:
 ... is what's called "private bytes" by the Performance control panel.
 It's memory that's private to that process; generally it's the amount
 of RAM the process has asked for in order to store runtime data.

And as I said before C++'s 'delete' (not sure about all implementations, but at least GCC/mingw and VC++2005) do free memory from this 'private bytes' thing. You said it is not really returned to OS, but get in to some pool. Ok, could you explain further, is it my application's pool, or some shared pool? What I mean, is: my application allocated and used this memory, but not anymore - it has been 'delete'd. Application will not need a new memory from OS for some time, or even possibly never. Can this freed(deleted) memory be used elsewhere? Please note, I talking about mentioned here C++ implementation that actually affect 'private bytes' counter. Thanks -- serg.
Sep 03 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:
 Walter Bright wrote:
 Not necessarily. There's a difference between physical memory and 
 virtual memory. The OS will automatically take away from the process 
 any unused physical memory, although the process retains it as virtual 
 memory. If the process explorer watches physical memory, then that's 
 what it's seeing.

subject of memory usage. So far, I see it as I described before - to measure application's memory usage, best way is to watch 'private bytes' counter in Perfmon (or via other tools that (I believe) uses Perfmon, such as Process Explorer or FAR's Process list plugin).

If those tools are measuring physical memory, that has no connection to whether memory is returned to the operating system or not.
 For example, quote from http://shsc.info/WindowsMemoryManagement:
 ... is what's called "private bytes" by the Performance control panel.
 It's memory that's private to that process; generally it's the amount
 of RAM the process has asked for in order to store runtime data.

And as I said before C++'s 'delete' (not sure about all implementations, but at least GCC/mingw and VC++2005) do free memory from this 'private bytes' thing. You said it is not really returned to OS, but get in to some pool. Ok, could you explain further, is it my application's pool, or some shared pool?

Your application's pool, which is implemented by the runtime library. You can see a sample implementation of this in K+R's "The C Programming Language" book. malloc/free/new/delete are just functions that are part of your program.
 What I mean, is: my application allocated and used this memory, but not 
 anymore - it has been 'delete'd. Application will not need a new memory 
 from OS for some time, or even possibly never. Can this freed(deleted) 
 memory be used elsewhere? Please note, I talking about mentioned here 
 C++ implementation that actually affect 'private bytes' counter.

To understand this, you really need to understand about how demand paged virtual memory works, and the difference between virtual and physical memory: http://en.wikipedia.org/wiki/Virtual_memory
Sep 03 2006
parent reply Serg Kovrov <kovrov no.spam> writes:
Walter Bright wrote:
 If those tools are measuring physical memory, that has no connection to 
 whether memory is returned to the operating system or not.

Private Bytes is a virtual memory counter. -- serg.
Sep 03 2006
parent Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:
 Walter Bright wrote:
 If those tools are measuring physical memory, that has no connection 
 to whether memory is returned to the operating system or not.

Private Bytes is a virtual memory counter.

You might find this helpful from http://blogs.msdn.com/ricom/archive/2005/08/01/446329.aspx : "The Private Bytes counter reports the commit charge of the process. That is to say, the amount of space that has been allocated in the swap file to hold the contents of the private memory in the event that it is swapped out. Note: I'm avoiding the word "reserved" because of possible confusion with virtual memory in the reserved state which is not committed. So, if we are concerned with space allocated in the swap file then the Private Bytes counter is right on the money. However, that is not usually what we are concerned about. We're much more interested in memory pressure caused by copies of private bytes in multiple processes. That is to say we are concerned about the physical memory that has been allocated to hold those private bytes. Why might/does the operating system allocate space in the swap file to hold the contents of memory whose contents have never become resident and may never become resident? The answer is not actually so complicated: Windows cannot deliver an "out of memory" exception/error just because you tried writing to, for instance, a static variable. The swap space must be pre-allocated at a reasonable time (such as loading a DLL) so that we can deliver an error result at a reasonable time -- the time at which the virtual addresses changed from reserved to committed. So if we had some static data that could be modified, but usually isn't modified in practice, and maybe isn't even read, you would find that the Private Bytes counter overstated the cost of that storage. The only real cost is the small bit of housekeeping to track allocated space in the swap file. No private memory need be allocated. In contrast, the output of vadump deals directly in what is resident and so, if memory is tight, it may understate the true memory cost because some things may have already been swapped out by the time you ask for the current resident pages. However, if memory is abundant then swapping is not going to happen and you can get a true picture of all the required pages from vadump."
Sep 03 2006
prev sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Walter Bright wrote:
 Serg Kovrov wrote:
 Walter, thanks for your kind and comprehensive answer, but I still do 
 not get this 'malloc/free do not return memory to OS' part.

 I just tried simplest example with GCC(mingw): I allocated with 'new' 
 1Mb string. Then in process explorer (or in FAR's Process list) I can 
 see my process usage of 'Private bytes' increases equivalently. When 
 free this string, with 'delete', i see that 'Private bytes' usage 
 dropped to same level it was before. Isn't this mean that memory 
 returned to OS?

Not necessarily. There's a difference between physical memory and virtual memory. The OS will automatically take away from the process any unused physical memory, although the process retains it as virtual memory. If the process explorer watches physical memory, then that's what it's seeing.

I think Serg is right. I don't know what program he used to monitor the mem usage ("process explorer", is that the Windows Task Manager?), but my test confirms what he's saying. I made a small C program that allocs 100Mb and then frees it. I monitored it with Windows XP Task Manager process list, looking at the fields "Mem Usage" (physical mem) and "VM Size". Compiling with GCC (3.2.3), it does free back the memory to the OS (VM Size decreases after the free). I tried the program with DMC but the same does not happen: the memory is not returned after the free. I don't have visual C available right now so I didn't try that one. After a google search: http://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html "Occasionally, free can actually return memory to the operating system and make the process smaller." Evidently the Windows version of libc also does the same. And seems it's smart enough to return the mem not just when the top of the heap is all free (what I was expecting) but also with free pages in the middle of the heap. (the test program allocs two segments and frees the first only) ---- test program ---- #include <stdio.h> #include <assert.h> int main() { char buf[80]; char* ptr = (char*) malloc(100000000); assert(ptr != NULL); char* ptr2 = (char*) malloc(100000000); assert(ptr2 != NULL); assert(ptr2 > ptr); printf("Mem Allocated\n"); fflush(stdout); gets(buf); // Just to see the physical mem usage int i; for(i = 0; i < 20000; i++) ptr[i*4000] = 'X'; printf("Memory Writen\n"); fflush(stdout); gets(buf); free(ptr); printf("ptr1 free'd\n"); fflush(stdout); gets(buf); } -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 04 2006
next sibling parent Serg Kovrov <kovrov no.spam> writes:
* Bruno Medeiros:
 I don't know what program he used to monitor  the mem usage
 ("process explorer", is that the Windows Task Manager?), but 
 my test confirms what he's saying.

I use sysinternals process explorer as substitute to windows task manager. I believe it reads perfmon counters for that purpose. Great tool, BTW. Worth looking at: http://www.sysinternals.com/Utilities/ProcessExplorer.html -- serg.
Sep 04 2006
prev sibling parent reply Walter Bright <newshound digitalmars.com> writes:
Bruno Medeiros wrote:
 I think Serg is right. I don't know what program he used to monitor  the 
 mem usage ("process explorer", is that the Windows Task Manager?), but 
 my test confirms what he's saying.
 
 I made a small C program that allocs 100Mb and then frees it. I 
 monitored it with Windows XP Task Manager process list, looking at the 
 fields "Mem Usage" (physical mem) and "VM Size". Compiling with GCC 
 (3.2.3), it does free back the memory to the OS (VM Size decreases after 
 the free). I tried the program with DMC but the same does not happen: 
 the memory is not returned after the free. I don't have visual C 
 available right now so I didn't try that one.
 
 After a google search:
 http://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html
 "Occasionally, free can actually return memory to the operating system 
 and make the process smaller."
 Evidently the Windows version of libc also does the same. And seems it's 
 smart enough to return the mem not just when the top of the heap is all 
 free (what I was expecting) but also with free pages in the middle of 
 the heap. (the test program allocs two segments and frees the first only)

What your program is doing is allocating a gigantic chunk. It is a reasonable thing for new() to have a fork in it, and for gigantic chunks allocate/free them by calling the OS directly and not attempting to manage it. Thus you would see the behavior you see. Try allocating a large number of small chunks, and free them.
Sep 04 2006
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Walter Bright wrote:
 Bruno Medeiros wrote:
 I think Serg is right. I don't know what program he used to monitor  
 the mem usage ("process explorer", is that the Windows Task Manager?), 
 but my test confirms what he's saying.

 I made a small C program that allocs 100Mb and then frees it. I 
 monitored it with Windows XP Task Manager process list, looking at the 
 fields "Mem Usage" (physical mem) and "VM Size". Compiling with GCC 
 (3.2.3), it does free back the memory to the OS (VM Size decreases 
 after the free). I tried the program with DMC but the same does not 
 happen: the memory is not returned after the free. I don't have visual 
 C available right now so I didn't try that one.

 After a google search:
 http://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html 

 "Occasionally, free can actually return memory to the operating system 
 and make the process smaller."
 Evidently the Windows version of libc also does the same. And seems 
 it's smart enough to return the mem not just when the top of the heap 
 is all free (what I was expecting) but also with free pages in the 
 middle of the heap. (the test program allocs two segments and frees 
 the first only)

What your program is doing is allocating a gigantic chunk. It is a reasonable thing for new() to have a fork in it, and for gigantic chunks allocate/free them by calling the OS directly and not attempting to manage it. Thus you would see the behavior you see. Try allocating a large number of small chunks, and free them.

Hum, I modified the program and tried 4 more tests: Allocating 10000 (sequential) chunks of size 10000 bytes. Free them all. -> All the 100Mb of memory is returned to the OS. Allocating 100000 (sequential) chunks of size 1000 bytes. Free them all. -> All the 100Mb of memory is returned to the OS. Allocating 10000 (sequential) chunks of size 10000 bytes. Free only half of them, the even numbered ones, so that the total freed memory is not contiguous. -> Of the 50Mb memory free'd, about 30Mb of memory is returned to the OS. Expected due to page segmentantion/rounding. Allocating 100000 (sequential) chunks of size 1000 bytes. Again free only the even numbered ones. -> 50Mb memory is free'd, but no memory is returned to the OS. Expected due to page segmentantion/rounding. So it seems glibc does its best, it returns any page if it is all free. (hum, and I'm curious to what the results in VC++ are, if anyone tries it, do post the results) ------ test program ------ #include <stdio.h> #include <assert.h> #define CHUNKSIZE 10000 #define NUMCHUNKS 10000 #define CHUNKINC 1 //use CHUNKINC 2 for even numbered freeing int main() { char buf[666]; int i; char* ptrs[NUMCHUNKS]; printf("NumChunks: %d Size: %d Inc: %d\n", NUMCHUNKS, CHUNKSIZE, CHUNKINC); fflush(stdout); gets(buf); for(i = 0; i < NUMCHUNKS; ++i) { ptrs[i] = (char*) malloc(CHUNKSIZE); assert(ptrs[i] != NULL); } char* ptrs2[NUMCHUNKS]; for(i = 0; i < NUMCHUNKS; ++i) { ptrs2[i] = (char*) malloc(CHUNKSIZE); assert(ptrs2[i] != NULL); assert(ptrs2[i] > ptrs[NUMCHUNKS-1]); } printf("Mem Allocated\n"); fflush(stdout); gets(buf); /* for(i = 0; i < NUMCHUNKS; i++) ptrs[i][0] = 'X'; printf("Memory Writen\n"); fflush(stdout); gets(buf); */ for(i = 0; i < NUMCHUNKS; i += CHUNKINC) { free(ptrs[i]); } printf("Mem free'd\n"); fflush(stdout); gets(buf); } -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 05 2006
next sibling parent Serg Kovrov <kovrov no.spam> writes:
* Bruno Medeiros:
 Walter Bright wrote:
 What your program is doing is allocating a gigantic chunk. It is a 
 reasonable thing for new() to have a fork in it, and for gigantic 
 chunks allocate/free them by calling the OS directly and not 
 attempting to manage it. Thus you would see the behavior you see.

 Try allocating a large number of small chunks, and free them.

Hum, I modified the program and tried 4 more tests: Allocating 10000 (sequential) chunks of size 10000 bytes. Free them all. -> All the 100Mb of memory is returned to the OS. Allocating 100000 (sequential) chunks of size 1000 bytes. Free them all. -> All the 100Mb of memory is returned to the OS. Allocating 10000 (sequential) chunks of size 10000 bytes. Free only half of them, the even numbered ones, so that the total freed memory is not contiguous. -> Of the 50Mb memory free'd, about 30Mb of memory is returned to the OS. Expected due to page segmentantion/rounding. Allocating 100000 (sequential) chunks of size 1000 bytes. Again free only the even numbered ones. -> 50Mb memory is free'd, but no memory is returned to the OS. Expected due to page segmentantion/rounding. So it seems glibc does its best, it returns any page if it is all free. (hum, and I'm curious to what the results in VC++ are, if anyone tries it, do post the results)

I can't be sure, but it's possible that Microsoft runtime library 'delete' implementation calls _heapmin() [http://msdn2.microsoft.com/en-us/library/fc7etheh.aspx] for that purpose. I red about it some time ago in a post on comp.os.ms-windows.programmer. The actual discussion was about delete not returning memory to OS =) I think it really depends on some other factors. The point was that 'delete'(or 'free') *could* return memory to OS, but not guaranteed. Phobos' GC has a stub f-n std.gc.minimize(), which probably intended for same purpose. I hope Walter could provide an update on it real purpose and implementation plans. -- serg.
Sep 05 2006
prev sibling next sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
10_000 chunks of 10_000 (freeing everything - CHUNKINC = 1):

On startup...
	Mem usage: 612k
	VM Size: 240k

After alloc:
	Mem usage: 119,504k
	VM Size: 195k

After free:
	Mem usage: 60,272k
	VM Size 98,296k


DMD version... same constants.

Startup...
	Mem usage: 1,304k
	VM Size: 696k

After alloc (which takes much longer):
	Mem usage: 248,208k
	VM Size: 248,208k

After free: same as above.

I used the below source.

-[Unknown]


import std.c.stdio;

const CHUNKSIZE = 10_000;
const NUMCHUNKS = 10_000;
const CHUNKINC = 1; // Use CHUNKINC 2 for even numbered freeing.

int main()
{
     char buf[666];
     int i;
     ubyte*[NUMCHUNKS] ptrs;

     printf("NumChunks: %d Size: %d Inc: %d\n", NUMCHUNKS, CHUNKSIZE, 
CHUNKINC);
     fflush(stdout); gets(buf);

     for (i = 0; i < NUMCHUNKS; ++i)
     {
         ptrs[i] = new ubyte[CHUNKSIZE];
         assert (ptrs[i] != null);
     }

     ubyte*[NUMCHUNKS] ptrs2;

     for(i = 0; i < NUMCHUNKS; ++i) {
         ptrs2[i] = new ubyte[CHUNKSIZE];
         assert (ptrs2[i] != null);
         assert (ptrs2[i] > ptrs[NUMCHUNKS - 1]);
     }
     printf("Mem Allocated\n"); fflush(stdout); gets(buf);

/*
     for (i = 0; i < NUMCHUNKS; i++)
         ptrs[i][0] = 42;
     printf("Memory Writen\n"); fflush(stdout); gets(buf);
*/

     for (i = 0; i < NUMCHUNKS; i += CHUNKINC)
     {
         delete ptrs[i];
     }
     printf("Mem free'd\n"); fflush(stdout); gets(buf);

     return 0;
}

 Hum, I modified the program and tried 4 more tests:
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free only half 
 of them, the even numbered ones, so that the total freed memory is not 
 contiguous.
 -> Of the 50Mb memory free'd, about 30Mb of memory is returned to the 
 OS. Expected due to page segmentantion/rounding.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Again free 
 only the even numbered ones.
 -> 50Mb memory is free'd, but no memory is returned to the OS. Expected 
 due to page segmentantion/rounding.
 
 So it seems glibc does its best, it returns any page if it is all free. 
 (hum, and I'm curious to what the results in VC++ are, if anyone tries 
 it, do post the results)
 
 ------ test program ------
 #include <stdio.h>
 #include <assert.h>
 
 #define CHUNKSIZE 10000
 #define NUMCHUNKS 10000
 #define CHUNKINC 1 //use CHUNKINC 2 for even numbered freeing
 
 int main() {
     char buf[666];
     int i;
     char* ptrs[NUMCHUNKS];
     
     printf("NumChunks: %d Size: %d Inc: %d\n", NUMCHUNKS, CHUNKSIZE, 
 CHUNKINC);
     fflush(stdout); gets(buf);
     
     for(i = 0; i < NUMCHUNKS; ++i) {
         ptrs[i] = (char*) malloc(CHUNKSIZE);
         assert(ptrs[i] != NULL);
     }
 
     char* ptrs2[NUMCHUNKS];
 
     for(i = 0; i < NUMCHUNKS; ++i) {
         ptrs2[i] = (char*) malloc(CHUNKSIZE);
         assert(ptrs2[i] != NULL);
         assert(ptrs2[i] > ptrs[NUMCHUNKS-1]);
     }
     printf("Mem Allocated\n"); fflush(stdout); gets(buf);
     
 /*    for(i = 0; i < NUMCHUNKS; i++)
         ptrs[i][0] = 'X';
     printf("Memory Writen\n"); fflush(stdout);    gets(buf);
 */
 
     for(i = 0; i < NUMCHUNKS; i += CHUNKINC) {
         free(ptrs[i]);
     }
     printf("Mem free'd\n"); fflush(stdout); gets(buf);
 }
 

Sep 05 2006
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Unknown W. Brackets wrote:
 10_000 chunks of 10_000 (freeing everything - CHUNKINC = 1):
 
 On startup...
     Mem usage: 612k
     VM Size: 240k
 
 After alloc:
     Mem usage: 119,504k
     VM Size: 195k

 After free:
     Mem usage: 60,272k
     VM Size 98,296k
 

You forgot to mention which compiler(and compiler version) was used. *g*
 
 DMD version... same constants.

DMD? Why not just use DMC? -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 05 2006
parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Oops, my mistake.  Yes, 195,000k.  That's what I get for posting 
something when I wasn't feeling well.

I meant to say, VC++ as requested.  I'm using cl 13.10.3077 for 80x86. 
Default options.

I used DMD because the comparison with DMC had already been made, but I 
didn't see anything equivalent for DMD.  DMD 0.162, by the way.

Thanks,
-[Unknown]


 Unknown W. Brackets wrote:
 10_000 chunks of 10_000 (freeing everything - CHUNKINC = 1):

 On startup...
     Mem usage: 612k
     VM Size: 240k

 After alloc:
     Mem usage: 119,504k
     VM Size: 195k

Typo there, should be about VM Size: 195,000k.
 After free:
     Mem usage: 60,272k
     VM Size 98,296k

You forgot to mention which compiler(and compiler version) was used. *g*
 DMD version... same constants.

DMD? Why not just use DMC?

Sep 06 2006
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Unknown W. Brackets wrote:
 Oops, my mistake.  Yes, 195,000k.  That's what I get for posting 
 something when I wasn't feeling well.
 
 I meant to say, VC++ as requested.  I'm using cl 13.10.3077 for 80x86. 
 Default options.
 

Ah, Visual Studio 2003
 I used DMD because the comparison with DMC had already been made, but I 
 didn't see anything equivalent for DMD.  DMD 0.162, by the way.
 

Walter had mentioned before that DMD's memory manager use C's malloc and free to allocate, so no difference in return behavior is to be expected. -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 09 2006
parent reply Sean Kelly <sean f4.ca> writes:
Bruno Medeiros wrote:
 Walter had mentioned before that DMD's memory manager use C's malloc and 
 free to allocate, so no difference in return behavior is to be expected.

It actually uses VirtualAlloc, though malloc is a failover option in GDC's memory manager. Sean
Sep 09 2006
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Sean Kelly wrote:
 Bruno Medeiros wrote:
 Walter had mentioned before that DMD's memory manager use C's malloc 
 and free to allocate, so no difference in return behavior is to be 
 expected.

It actually uses VirtualAlloc, though malloc is a failover option in GDC's memory manager. Sean

Hum, didn't Walter once mention that nearly all compiler implementations of new and delete used malloc? I assumed his compilers where one of such (didn't look at Phobos). Perhaps that simply was not the case. Or perhaps he just meant C++ compilers? -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 22 2006
parent Sean Kelly <sean f4.ca> writes:
Bruno Medeiros wrote:
 Sean Kelly wrote:
 Bruno Medeiros wrote:
 Walter had mentioned before that DMD's memory manager use C's malloc 
 and free to allocate, so no difference in return behavior is to be 
 expected.

It actually uses VirtualAlloc, though malloc is a failover option in GDC's memory manager.

Hum, didn't Walter once mention that nearly all compiler implementations of new and delete used malloc? I assumed his compilers where one of such (didn't look at Phobos). Perhaps that simply was not the case. Or perhaps he just meant C++ compilers?

He meant C++ compilers. But the situation in D is somewhat different because of garbage collection. malloc is itself a memory allocator and D's GC is also an allocator, so having D's allocator call malloc would be like one retail store buying all their products from another retail store in bulk--it's much more efficient just to order from the distributer :-) Sean
Sep 22 2006
prev sibling parent Walter Bright <newshound digitalmars.com> writes:
Bruno Medeiros wrote:
 Hum, I modified the program and tried 4 more tests:
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free only half 
 of them, the even numbered ones, so that the total freed memory is not 
 contiguous.
 -> Of the 50Mb memory free'd, about 30Mb of memory is returned to the 
 OS. Expected due to page segmentantion/rounding.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Again free 
 only the even numbered ones.
 -> 50Mb memory is free'd, but no memory is returned to the OS. Expected 
 due to page segmentantion/rounding.
 
 So it seems glibc does its best, it returns any page if it is all free. 

So it does seem to, and so my statement is obsolete.
Sep 08 2006
prev sibling parent Helmut Leitner <leitner wikiservice.at> writes:
Walter Bright wrote:
 Serg Kovrov wrote:
 
 But that not worst part. Although I'm personally from C++ camp and I 
 get used to GC. and see it as good thing. But, there is one big BUT! 
 Currently GC do not return memory to OS, and that thing even I can't 
 subdue. The only reason I choose 'native' (as opposite to VM) 
 programming language is effective resource usage. I hate Java/.net gui 
 applications because their memory consuming. I really appreciate 
 developers that choose c/c++ to write small, memory effective, but yet 
 feature-reach applications like FileZilla Server, uTorrent or Miranda IM.

 Most desktop applications (like text editors, news readers, web 
 browsers, etc) must coexist with each other. That is, to use memory 
 wisely and give it back if it not needed anymore. This 'upper water 
 mark' approach in GC is just not acceptable.

1) Few programmers realize that C/C++'s malloc/free/new/delete *never* (and I emphasize NEVER) return memory to the operating system. All free/delete do is return memory to the memory pool managed by the runtime library, not the operating system. In order to actually return memory to the operating system, one has to write their own memory management code, which is a significant (and necessarily non-portable) effort. Hardly anyone does that.

I think this doesn't give a complete picture of the situation. There is the need to give memory back to the OS (e. g. if a user has just stopped working with a couple of 100 MB pictures). Operating systems sometimes do offer mechanisms for this (e. g. Windows has the GlobalAlloc/GlobalReAlloc/GlobalFree set of functions). If you use a wrapper for memory allocation it is very easy to use this efficiently (pure or mixed with malloc/realloc/free).
Sep 08 2006