www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - When is it time for a 1.0 feature freeze?

reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Among the many as-yet-unfulfilled prerequisites for being ready for 1.0 
is coming to a final decision on the feature set.

On releasing 0.93, Walter stated:

"Barring a very, very, compelling case, this is it for 1.0 language 
features.  There are a lot more things I want to do, but this has got to 
be enough for 1.0."

However, there have been lots of new features since then.  And as new 
features are added, it opens the door to more bugs and more 
imperfections in the spec.

I think that sometime soon, we should finally make it more firm that 
"this is it for 1.0 language features".  Once this is done, then we can 
finally concentrate more on fixing compiler bugs and getting the spec up 
to scratch (both cleaning up spelling/grammar and cleaning up what is 
ill-defined) without having to worry too much about creating even more 
work for ourselves in the process.

Stewart.
Aug 21 2006
parent reply "Søren J. Løvborg" <web kwi.dk> writes:
Stewart Gordon wrote:

 I think that sometime soon, we should finally make it more firm that "this 
 is it for 1.0 language features".  Once this is done, then we can finally 
 concentrate more on fixing compiler bugs and getting the spec up to 
 scratch (both cleaning up spelling/grammar and cleaning up what is 
 ill-defined) without having to worry too much about creating even more 
 work for ourselves in the process.


The only thing I can think off that should be fixed before 1.0 is the "auto" 
keyword.

It's current double meaning is a mess, and it needs to be defined as either 
a RAII or a type deduction keyword (and another keyword/syntax needs to be 
introduced for the other meaning.)

I'm leaning towards making "auto" mean type deduction, and introduce another 
keyword ("raii"? Unlikely to break old code, I'd venture) for RAII 
allocations.

That'd be a change from the C meaning of "auto", but so is the D meaning of 
"typedef" (since C's "typedef" is D's "alias"), and "auto" is practically 
never used in C, even.

Søren J. Løvborg
web kwi.dk
Aug 21 2006
parent reply Fredrik Olsson <peylow gmail.com> writes:
Søren J. Løvborg skrev:

 Stewart Gordon wrote:

 I think that sometime soon, we should finally make it more firm that "this 
 is it for 1.0 language features".  Once this is done, then we can finally 
 concentrate more on fixing compiler bugs and getting the spec up to 
 scratch (both cleaning up spelling/grammar and cleaning up what is 
 ill-defined) without having to worry too much about creating even more 
 work for ourselves in the process.


 
 The only thing I can think off that should be fixed before 1.0 is the "auto" 
 keyword.
 


I can also think of allowing properties as lvalue:
foo.bar += 42;

Array literals:
bar ~= [1,5,42];

Struct literals:
baz = {"hej", 42};


Those and the lack of sets and ranges are top on my list. The small 
stuff that can reduce code size in half. The only 100% foolproof way to 
write bug-free code is; not to write any. So the closer to no code, the 
better :).


// Fredrik Olsson
Aug 31 2006
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Fredrik Olsson wrote:
<snip>

 I can also think of allowing properties as lvalue:
 foo.bar += 42;
 
 Array literals:
 bar ~= [1,5,42];


The problem with that notation is that, in the general case, the type of 
the array cannot be guaranteed.  There have been a number of proposed 
syntaxes; AFAIK the best so far is

http://www.digitalmars.com/d/archives/digitalmars/D/39125.html

So your answer to the subject of this thread is "Once these features 
have been added", right?  Could be a long time.... :-)

<snip>

 Those and the lack of sets and ranges are top on my list. The small 
 stuff that can reduce code size in half. The only 100% foolproof way to 
 write bug-free code is; not to write any. So the closer to no code, the 
 better :).


How does not writing code equate to writing code?

But you remind me of this:

http://www.stevemcconnell.com/cctune.htm

(See from "A fast program is just as important as a correct one--False!" 
downwards.)

Stewart.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS/M d- s:-  C++  a->--- UB  P+ L E  W++  N+++ o K-  w++  O? M V? PS- 
PE- Y? PGP- t- 5? X? R b DI? D G e++++ h-- r-- !y
------END GEEK CODE BLOCK------

My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
Sep 01 2006
parent reply Ivan Senji <ivan.senji_REMOVE_ _THIS__gmail.com> writes:
Stewart Gordon wrote:

 Fredrik Olsson wrote:
 <snip>

 I can also think of allowing properties as lvalue:
 foo.bar += 42;

 Array literals:
 bar ~= [1,5,42];


 
 The problem with that notation is that, in the general case, the type of 
 the array cannot be guaranteed.  There have been a number of proposed 
 syntaxes; AFAIK the best so far is
 
 http://www.digitalmars.com/d/archives/digitalmars/D/39125.html


I agree! That is the best proposal so far. I know Walter wants that to 
be a 2.0 feature but it still doesn't make sense to me.
Array and struct initializers are more fundamental feature than 
templates, delegates, lazyness, even classes, and they are going to be 
missing from D1.0? Crazy!

These days each D project I start has
T[] array(T)(T[] x...)
{
    return x.dup;
}

right after the module statement and that should be an indication that 
something is wrong. Although the array template fixes (kind of) the 
array initialization problem, there is nothing that can be done for 
structs (not counting static opCall as a solution).
Sep 01 2006
parent reply Don Clugston <dac nospam.com.au> writes:
Ivan Senji wrote:

 Stewart Gordon wrote:

 Fredrik Olsson wrote:
 <snip>

 I can also think of allowing properties as lvalue:
 foo.bar += 42;

 Array literals:
 bar ~= [1,5,42];


 The problem with that notation is that, in the general case, the type 
 of the array cannot be guaranteed.  There have been a number of 
 proposed syntaxes; AFAIK the best so far is

 http://www.digitalmars.com/d/archives/digitalmars/D/39125.html


 
 I agree! That is the best proposal so far. I know Walter wants that to 
 be a 2.0 feature but it still doesn't make sense to me.
 Array and struct initializers are more fundamental feature than 
 templates, delegates, lazyness, even classes, and they are going to be 
 missing from D1.0? Crazy!


Those things would delay D1.0 by months. The idea is to declare D1.0 
quickly, and then move onto those things.
Sep 01 2006
parent reply Ivan Senji <ivan.senji_REMOVE_ _THIS__gmail.com> writes:
Don Clugston wrote:

 Ivan Senji wrote:

 Stewart Gordon wrote:

 http://www.digitalmars.com/d/archives/digitalmars/D/39125.html


 I agree! That is the best proposal so far. I know Walter wants that to 
 be a 2.0 feature but it still doesn't make sense to me.
 Array and struct initializers are more fundamental feature than 
 templates, delegates, lazyness, even classes, and they are going to be 
 missing from D1.0? Crazy!


 
 Those things would delay D1.0 by months. The idea is to declare D1.0 
 quickly, and then move onto those things.


I know, and I wouldn't like that to happen but many other features have 
been delaying D for years (not saying that I don't like them).

I don't have anything against D1.1 od D1.5 or even D2.0 having these 
features but I am a little worried about what happens when D1.0 attracts 
  new people and they must very quickly realize "what? D doesn't have 
array and struct literals? What kind of a language is that?"

Is that kind of negative publicity bad or not I don't know but it might 
make D feel incomplete. When you stay around D for a while you realize 
it isn't that incomplete after all but for some people the first 
negative impression is going to be the last impression they make of D.
Sep 01 2006
parent reply Serg Kovrov <kovrov no.spam> writes:
* Ivan Senji:

 I don't have anything against D1.1 od D1.5 or even D2.0 having these 
 features but I am a little worried about what happens when D1.0 attracts 
 new people and they must very quickly realize "what? D doesn't have 
 array and struct literals? What kind of a language is that?"


I'm more concerning about memory management scheme.

Most people I was talking to about D (and I agitate very fellow 
programmer i personally know to at least try D)... firstly complains 
about 'yet another language' syndrome. There is really nothing can be 
done about it, you just have to convince them to not to stop discussion =)

Next, everyone from C/C++ camp (especially C) is just so-skeptical about 
GC. All sort of things were already beaten to death here on NG. It is a 
meter of personal taste and there is nothing can't be done either.

And then usually happens following: on my clumsy allegations that it is 
not necessary to use GC, and with malloc/free one could manage memory 
manually. Well, you know... Whom I trying to deceive, myself? It is 
simply not so convenient to actually do so. And what about string 
operations (concatenations, assignment etc), they use GC and i don't 
know how to avoid it. Plus, standard library is hard-coupled with GC. 
My point is that if not use GC programming with D is even more awkward 
then C/C++.

But that not worst part. Although I'm personally from C++ camp and I get 
used to GC. and see it as good thing. But, there is one big BUT! 
Currently GC do not return memory to OS, and that thing even I can't 
subdue. The only reason I choose 'native' (as opposite to VM) 
programming language is effective resource usage. I hate Java/.net gui 
applications because their memory consuming. I really appreciate 
developers that choose c/c++ to write small, memory effective, but yet 
feature-reach applications like FileZilla Server, uTorrent or Miranda IM.

Most desktop applications (like text editors, news readers, web 
browsers, etc) must coexist with each other. That is, to use memory 
wisely and give it back if it not needed anymore. This 'upper water 
mark' approach in GC is just not acceptable.

All I actually can say to my fellow programmers in defense of D, is that 
language still developing and eventually all good things happens. On 
that point discussion usually ends and the bottom line is "nice 
language, we will look at later, when/if it evolve".

And for GC, to fee memory back to OS is crucial to every one who care 
about memory management. And ones who do not - whey will stay with their 
Java/.net VM monsters and will newer even look at D. Besides, this type 
of developers do not care about anything but their CV. Sure thing D will 
not help them.


P.S. sorry for rather long post. I don't expect it myself to be so =)
-- 
serg.
Sep 01 2006
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Serg Kovrov wrote:

 * Ivan Senji:

 I don't have anything against D1.1 od D1.5 or even D2.0 having these 
 features but I am a little worried about what happens when D1.0 
 attracts new people and they must very quickly realize "what? D 
 doesn't have array and struct literals? What kind of a language is that?"


 
 I'm more concerning about memory management scheme.
 
 Most people I was talking to about D (and I agitate very fellow 
 programmer i personally know to at least try D)... firstly complains 
 about 'yet another language' syndrome. There is really nothing can be 
 done about it, you just have to convince them to not to stop discussion =)
 
 Next, everyone from C/C++ camp (especially C) is just so-skeptical about 
 GC. All sort of things were already beaten to death here on NG. It is a 
 meter of personal taste and there is nothing can't be done either.
 
 And then usually happens following: on my clumsy allegations that it is 
 not necessary to use GC, and with malloc/free one could manage memory 
 manually. Well, you know... Whom I trying to deceive, myself? It is 
 simply not so convenient to actually do so. And what about string 
 operations (concatenations, assignment etc), they use GC and i don't 
 know how to avoid it.


The same as you would in C.  Manually allocate and free the buffer 
referenced by the array ptr.  The layout of dynamic arrays is specified 
in the ABI, so you're guaranteed to have a .len and .ptr property to 
play with if you really want to.


 Plus, standard library is hard-coupled with GC.


It doesn't have to be.  That's just how Phobos was written.


 My
 point is that if not use GC programming with D is even more awkward then 
 C/C++.


True.  Though I think 'scope' statements help a lot, and stack 
allocation will help tremendously.  Walter has already said he plans to 
implement:

   MyClass c = new MyClass(); // heap alloc
   MyClass d = MyClass();     // stack alloc

And hopefully:

   char[] a = new char[32]; // heap alloc
   char[] b = char[32];     // stack alloc

This would do away with the need for alloca(), which would conflict with 
dynamic closures anyway.  An in-language solution, however, may be 
compatible in some cases.


 But that not worst part. Although I'm personally from C++ camp and I get 
 used to GC. and see it as good thing. But, there is one big BUT! 
 Currently GC do not return memory to OS, and that thing even I can't 
 subdue.


This is just the default GC shipped with DMD (and GDC).  There's no 
reason it can't be replaced.  In fact, I've taken steps to make this as 
easy as possible in Ares.


 Most desktop applications (like text editors, news readers, web 
 browsers, etc) must coexist with each other. That is, to use memory 
 wisely and give it back if it not needed anymore. This 'upper water 
 mark' approach in GC is just not acceptable.


I think it depends on the application.  If an application behaves 
consistently across their run cycle, there's no reason to return memory 
to the OS because it will likely be needed again.  If an application 
allocates a ton of memory, frees it, and never uses it again then what 
you say is true.  Though in these cases the application could call 
malloc/free for these one-time memory allocations as well.


 All I actually can say to my fellow programmers in defense of D, is that 
 language still developing and eventually all good things happens. On 
 that point discussion usually ends and the bottom line is "nice 
 language, we will look at later, when/if it evolve".


I think it's important to separate discussions of the language with the 
behavior of accompanying library code.  The GC occupies kind of a 
strange middle-ground here, but I still consider it library code because 
nothing inside is required to be compiler-specific.  Sure the compiler 
runtime allocates and frees memory, but it doesn't need special access 
that is not available to the user.  Personally, I think D as a language 
is shaping up quite nicely.  The library could perhaps use a little 
work, but that can be done without changing the language spec.


Sean
Sep 01 2006
parent reply Serg Kovrov <kovrov no.spam> writes:
Sean Kelly wrote:

 Serg Kovrov wrote:

 Most desktop applications (like text editors, news readers, web 
 browsers, etc) must coexist with each other. That is, to use memory 
 wisely and give it back if it not needed anymore. This 'upper water 
 mark' approach in GC is just not acceptable.


 
 I think it depends on the application.  If an application behaves 
 consistently across their run cycle, there's no reason to return memory 
 to the OS because it will likely be needed again.  If an application 
 allocates a ton of memory, frees it, and never uses it again then what 
 you say is true.  Though in these cases the application could call 
 malloc/free for these one-time memory allocations as well.


I can't imagine an application that shouldn't free (return to OS) memory 
in runtime. Perhaps only really 'heavy' monopoly applications (like 
games, 3d modeling packages) and 'run and die' tools (like grep, less).

Other application that possibly could running unlimited time and process 
some significant amount of data, simply MUST return unused memory to OS.

For example web browser - user could open up to several tens of web 
pages simultaneously. Some maybe enormously big. If browser will no free 
memory consumed in such peak moments use will be forced to restart 
application (e.g. experience inconveniences) or will experience slowdown 
caused by swapping.

Same goes to any application working with arbitrary-sized data - image 
viewers, mail notificators, IDEs or text publishing tools, p2p clients 
or servers... Rally. As user, I don't want my everyday applications to 
hold peak memory, or restart them constantly. In the end i will chose 
other software, that will satisfy my habits.

I found it silly to shape behavior of an application by language (or it 
standard library) limitations.

If only programmer could have some control over GC, it would be just 
fine. For example to query GC's memory pool and decide to free some 
amount if it exceed some threshold. Walter, any comments on this?

-- 
serg.
Sep 01 2006
parent Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:

 If only programmer could have some control over GC, it would be just 
 fine. For example to query GC's memory pool and decide to free some 
 amount if it exceed some threshold. Walter, any comments on this?


Because, as I said in another post, malloc/free/new/delete don't return 
memory to the operating system, what apps often do if they need to 
allocate/free some small number of very large buffers is to do them 
separately, usually with a call straight into the OS API.

You can do the same thing in D, using the same technique and pretty much 
identical code.
Sep 02 2006
prev sibling parent reply Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:

 But that not worst part. Although I'm personally from C++ camp and I get 
 used to GC. and see it as good thing. But, there is one big BUT! 
 Currently GC do not return memory to OS, and that thing even I can't 
 subdue. The only reason I choose 'native' (as opposite to VM) 
 programming language is effective resource usage. I hate Java/.net gui 
 applications because their memory consuming. I really appreciate 
 developers that choose c/c++ to write small, memory effective, but yet 
 feature-reach applications like FileZilla Server, uTorrent or Miranda IM.
 
 Most desktop applications (like text editors, news readers, web 
 browsers, etc) must coexist with each other. That is, to use memory 
 wisely and give it back if it not needed anymore. This 'upper water 
 mark' approach in GC is just not acceptable.


1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
(and I emphasize NEVER) return memory to the operating system. All 
free/delete do is return memory to the memory pool managed by the 
runtime library, not the operating system. In order to actually return 
memory to the operating system, one has to write their own memory 
management code, which is a significant (and necessarily non-portable) 
effort. Hardly anyone does that.

2) The GC's memory pool doesn't consist of physical memory, it consists 
of virtual memory. If that virtual memory is not being used, the 
operating system automatically swaps it out and allocates the physical 
memory to whoever actually uses it. The 'memory' held by the GC isn't 
memory at all, but nothing more than a range of addresses.

You can run out of virtual address space, but that's rare and usually 
indicates you've got a bug, such as an infinite loop allocating memory.

3) The GC could be modified to return unused address space to the 
operating system, that it doesn't do so now is not a limitation on D itself.

4) Given the exact same code, a gc'd program will consume roughly twice 
as much memory as the same explicitly allocated one (though neither will 
return memory to the operating system). But a program written to take 
advantage of GC can often use less memory (because it needs to make far 
fewer copies of things, and because it doesn't have to allocate extra 
memory management blocks like shared_ptr<T> does).

5) malloc/free/new/delete use an 'upper water mark' system of managing 
their internal free memory pools just like D's gc does.
Sep 02 2006
next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Walter Bright wrote:
<snip>

 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.


<snip>

Never?  I can't believe you've tried every implementation out there, 
including your own, and found the same.

Or does the spec forbid free or delete to return memory to the OS?  I 
can't for the life of me see why this would be.

Stewart.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS/M d- s:-  C++  a->--- UB  P+ L E  W++  N+++ o K-  w++  O? M V? PS- 
PE- Y? PGP- t- 5? X? R b DI? D G e++++ h-- r-- !y
------END GEEK CODE BLOCK------

My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
Sep 02 2006
next sibling parent Gregor Richards <Richards codu.org> writes:
Stewart Gordon wrote:

 Walter Bright wrote:
 <snip>
 

 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.


 
 <snip>
 
 Never?  I can't believe you've tried every implementation out there, 
 including your own, and found the same.
 
 Or does the spec forbid free or delete to return memory to the OS?  I 
 can't for the life of me see why this would be.
 
 Stewart.
 


In general changing the amount a process wishes to use (the data segment 
size) involves calling [s]brk.  brk just controls the pointer to the end 
of the memory you have available.  You can't get data segments in 
chunks, only one contiguous block.  Because of this, it's in general 
impossible for free/delete to return the memory to the OS, as it cannot 
move memory arbitrarily (or all the pointers would go funky), and it in 
all likelyhood has something allocated at the end of the data segment.

I don't know whether it's in any spec that this is how it MUST work, but 
this is how it USUALLY works.  There's no real reason to do otherwise, 
especially since changing it would make the data segmentation much more 
complicated.

  - Gregor Richards
Sep 02 2006
prev sibling next sibling parent Derek Parnell <derek psyc.ward> writes:
On Sun, 03 Sep 2006 01:45:25 +0100, Stewart Gordon wrote:


 Walter Bright wrote:
 <snip>

 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.


 <snip>
 
 Never?  I can't believe you've tried every implementation out there, 
 including your own, and found the same.
 
 Or does the spec forbid free or delete to return memory to the OS?  I 
 can't for the life of me see why this would be.
 
 Stewart.


From memory, I believe the Manx-C for the Amiga used to return memory to
the operating system.

-- 
Derek Parnell
Melbourne, Australia
"Down with mediocrity!"
Sep 02 2006
prev sibling next sibling parent Walter Bright <newshound digitalmars.com> writes:
Stewart Gordon wrote:

 Walter Bright wrote:
 <snip>

 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.


 <snip>
 
 Never?  I can't believe you've tried every implementation out there, 
 including your own, and found the same.


You're right, I haven't tried every implementation, and don't know it 
for a fact. It's just true of every one I've looked at, and there are 3 
good reasons for it to be so:

1) Speed
2) Speed
3) Speed

<g>

If your app isrelying on free/delete returning memory to the operating 
system, you really need to take a good hard look at the implementation 
of it, because it isn't.

Ironically, it's much more likely that a gc will return memory to the os 
- because a gc can be a copying collector, meaning it can compact the 
used memory and free the rest. free/delete cannot do any compaction, 
meaning it's unlikely to be able to release any memory back to the os 
due to fragmentation, minimum page sizes, and the high water nature of 
allocating memory to a task.

It's often moot anyway, since as I wrote before, runtime heap management 
  is holding on to virtual memory, not physical memory. A gc again has 
the potential advantage here, as it can compact the actual used memory 
into as few pages as possible, reducing physical memory consumption, and 
increasing the working set that will fit into the cache.


 Or does the spec forbid free or delete to return memory to the OS?  I
 can't for the life of me see why this would be.


The spec has nothing to say about it.
Sep 03 2006
prev sibling parent Sean Kelly <sean f4.ca> writes:
Stewart Gordon wrote:

 Walter Bright wrote:
 <snip>

 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.


 <snip>
 
 Never?  I can't believe you've tried every implementation out there, 
 including your own, and found the same.
 
 Or does the spec forbid free or delete to return memory to the OS?  I 
 can't for the life of me see why this would be.


Memory can only be allocated and freed in pages, and a memory page can 
obviously only be returned to the OS if it is completely empty.  Not to 
mention the issue of holes in what is typically supposed to be a 
contiguous address space.  Some allocators will re-purpose empty pages 
if a different allocation size is needed, but I don't know of any that 
actually return memory to the OS.  I think huge allocations may be an 
exception, as allocators typically obtain the memory for these in a 
slightly different manner.  But even then I'm not sure the memory is 
"returned to the OS" on release in a manner that would actually decrease 
the process virtual memory footprint.


Sean
Sep 03 2006
prev sibling next sibling parent reply Serg Kovrov <kovrov no.spam> writes:
Walter, thanks for your kind and comprehensive answer, but I still do 
not get this 'malloc/free do not return memory to OS' part.

I just tried simplest example with GCC(mingw): I allocated with 'new' 
1Mb string. Then in process explorer (or in FAR's Process list) I can 
see my process usage of 'Private bytes' increases equivalently. When 
free this string, with 'delete', i see that 'Private bytes' usage 
dropped to same level it was before. Isn't this mean that memory 
returned to OS?

And second example, I just tried open text files of different size with 
Catch22 Neatpad example (http://catch22.net/tuts/ - compiled with 
VC++2005), and see exactly same  behavior. Private bytes usage changes 
proportionally to opened file size. I've looked at source code an can 
see that for file buffer and support buffers allocation/deallocation 
used new/delete from standard library.

Could someone please, explain me where I do not understand Walter? Or 
maybe point to appropriate article/reference.

-- 
serg.
Sep 03 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:

 Walter, thanks for your kind and comprehensive answer, but I still do 
 not get this 'malloc/free do not return memory to OS' part.
 
 I just tried simplest example with GCC(mingw): I allocated with 'new' 
 1Mb string. Then in process explorer (or in FAR's Process list) I can 
 see my process usage of 'Private bytes' increases equivalently. When 
 free this string, with 'delete', i see that 'Private bytes' usage 
 dropped to same level it was before. Isn't this mean that memory 
 returned to OS?


Not necessarily. There's a difference between physical memory and 
virtual memory. The OS will automatically take away from the process any 
unused physical memory, although the process retains it as virtual 
memory. If the process explorer watches physical memory, then that's 
what it's seeing.
Sep 03 2006
next sibling parent reply Serg Kovrov <kovrov no.spam> writes:
Walter Bright wrote:

 Serg Kovrov wrote:

 I just tried simplest example with GCC(mingw): I allocated with 'new' 
 1Mb string. Then in process explorer (or in FAR's Process list) I can 
 see my process usage of 'Private bytes' increases equivalently. When 
 free this string, with 'delete', i see that 'Private bytes' usage 
 dropped to same level it was before. Isn't this mean that memory 
 returned to OS?


 
 Not necessarily. There's a difference between physical memory and 
 virtual memory. The OS will automatically take away from the process any 
 unused physical memory, although the process retains it as virtual 
 memory. If the process explorer watches physical memory, then that's 
 what it's seeing.


I'm not trying to argue here, but to understand. I done some search on 
subject of memory usage. So far, I see it as I described before - to 
measure application's memory usage, best way is to watch 'private bytes' 
counter in Perfmon (or via other tools that (I believe) uses Perfmon, 
such as Process Explorer or FAR's Process list plugin).

For example, quote from http://shsc.info/WindowsMemoryManagement:

 ... is what's called "private bytes" by the Performance control panel.
 It's memory that's private to that process; generally it's the amount
 of RAM the process has asked for in order to store runtime data.


And as I said before C++'s 'delete' (not sure about all implementations, 
but at least GCC/mingw and VC++2005) do free memory from this 'private 
bytes' thing.

You said it is not really returned to OS, but get in to some pool. Ok, 
could you explain further, is it my application's pool, or some shared pool?

What I mean, is: my application allocated and used this memory, but not 
anymore - it has been 'delete'd. Application will not need a new memory 
from OS for some time, or even possibly never. Can this freed(deleted) 
memory be used elsewhere? Please note, I talking about mentioned here 
C++ implementation that actually affect 'private bytes' counter.

Thanks
-- 
serg.
Sep 03 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:

 Walter Bright wrote:

 Not necessarily. There's a difference between physical memory and 
 virtual memory. The OS will automatically take away from the process 
 any unused physical memory, although the process retains it as virtual 
 memory. If the process explorer watches physical memory, then that's 
 what it's seeing.


 I'm not trying to argue here, but to understand. I done some search on 
 subject of memory usage. So far, I see it as I described before - to 
 measure application's memory usage, best way is to watch 'private bytes' 
 counter in Perfmon (or via other tools that (I believe) uses Perfmon, 
 such as Process Explorer or FAR's Process list plugin).


If those tools are measuring physical memory, that has no connection to 
whether memory is returned to the operating system or not.


 For example, quote from http://shsc.info/WindowsMemoryManagement:

 ... is what's called "private bytes" by the Performance control panel.
 It's memory that's private to that process; generally it's the amount
 of RAM the process has asked for in order to store runtime data.


 
 And as I said before C++'s 'delete' (not sure about all implementations, 
 but at least GCC/mingw and VC++2005) do free memory from this 'private 
 bytes' thing.
 
 You said it is not really returned to OS, but get in to some pool. Ok, 
 could you explain further, is it my application's pool, or some shared 
 pool?


Your application's pool, which is implemented by the runtime library. 
You can see a sample implementation of this in K+R's "The C Programming 
Language" book. malloc/free/new/delete are just functions that are part 
of your program.



 What I mean, is: my application allocated and used this memory, but not 
 anymore - it has been 'delete'd. Application will not need a new memory 
 from OS for some time, or even possibly never. Can this freed(deleted) 
 memory be used elsewhere? Please note, I talking about mentioned here 
 C++ implementation that actually affect 'private bytes' counter.


To understand this, you really need to understand about how demand paged 
virtual memory works, and the difference between virtual and physical 
memory: http://en.wikipedia.org/wiki/Virtual_memory
Sep 03 2006
parent reply Serg Kovrov <kovrov no.spam> writes:
Walter Bright wrote:

 If those tools are measuring physical memory, that has no connection to 
 whether memory is returned to the operating system or not.


Private Bytes is a virtual memory counter.

-- 
serg.
Sep 03 2006
parent Walter Bright <newshound digitalmars.com> writes:
Serg Kovrov wrote:

 Walter Bright wrote:

 If those tools are measuring physical memory, that has no connection 
 to whether memory is returned to the operating system or not.


 
 Private Bytes is a virtual memory counter.


You might find this helpful from 
http://blogs.msdn.com/ricom/archive/2005/08/01/446329.aspx :

"The Private Bytes counter reports the commit charge of the process. 
That is to say, the
amount of space that has been allocated  in the swap file to hold the 
contents of the
private memory in the event that it is swapped out.  Note: I'm avoiding 
the word "reserved"
because of possible confusion with virtual memory in the reserved state 
which is not
committed.

So, if we are concerned with space allocated in the swap file then the 
Private Bytes counter
is right on the money.  However, that is not usually what we are 
concerned about.  We're
much more interested in memory pressure caused by copies of private 
bytes in multiple
processes.  That is to say we are concerned about the physical memory 
that has been allocated
to hold those private bytes.

Why might/does the operating system allocate space in the swap file to 
hold the contents of
memory whose contents have never become resident and may never become 
resident?  The answer
is not actually so complicated:  Windows cannot deliver an "out of 
memory" exception/error
just because you tried writing to, for instance, a static variable.  The 
swap space must be
pre-allocated at a reasonable time (such as loading a DLL) so that we 
can deliver an error
result at a reasonable time -- the time at which the virtual addresses 
changed from
reserved to committed.

So if we had some static data that could be modified, but usually isn't 
modified in
practice, and maybe isn't even read, you would find that the Private 
Bytes counter
overstated the cost of that storage.  The only real cost is the small 
bit of housekeeping
to track allocated space in the swap file.  No private memory need be 
allocated.

In contrast, the output of vadump deals directly in what is resident and 
so, if memory is
tight, it may understate the true memory cost because some things may 
have already been
swapped out by the time you ask for the current resident pages. 
However, if memory is
abundant then swapping is not going to happen and you can get a true 
picture of all the
required pages from vadump."
Sep 03 2006
prev sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Walter Bright wrote:

 Serg Kovrov wrote:

 Walter, thanks for your kind and comprehensive answer, but I still do 
 not get this 'malloc/free do not return memory to OS' part.

 I just tried simplest example with GCC(mingw): I allocated with 'new' 
 1Mb string. Then in process explorer (or in FAR's Process list) I can 
 see my process usage of 'Private bytes' increases equivalently. When 
 free this string, with 'delete', i see that 'Private bytes' usage 
 dropped to same level it was before. Isn't this mean that memory 
 returned to OS?


 
 Not necessarily. There's a difference between physical memory and 
 virtual memory. The OS will automatically take away from the process any 
 unused physical memory, although the process retains it as virtual 
 memory. If the process explorer watches physical memory, then that's 
 what it's seeing.


I think Serg is right. I don't know what program he used to monitor  the 
mem usage ("process explorer", is that the Windows Task Manager?), but 
my test confirms what he's saying.

I made a small C program that allocs 100Mb and then frees it. I 
monitored it with Windows XP Task Manager process list, looking at the 
fields "Mem Usage" (physical mem) and "VM Size". Compiling with GCC 
(3.2.3), it does free back the memory to the OS (VM Size decreases after 
the free). I tried the program with DMC but the same does not happen: 
the memory is not returned after the free. I don't have visual C 
available right now so I didn't try that one.

After a google search:
http://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html
"Occasionally, free can actually return memory to the operating system 
and make the process smaller."
Evidently the Windows version of libc also does the same. And seems it's 
smart enough to return the mem not just when the top of the heap is all 
free (what I was expecting) but also with free pages in the middle of 
the heap. (the test program allocs two segments and frees the first only)


---- test program ----
#include <stdio.h>
#include <assert.h>

int main() {
	char buf[80];
	
	char* ptr = (char*) malloc(100000000);
	assert(ptr != NULL);

	char* ptr2 = (char*) malloc(100000000);
	assert(ptr2 != NULL);

	assert(ptr2 > ptr);

	
	printf("Mem Allocated\n"); fflush(stdout);
	gets(buf);
	// Just to see the physical mem usage
	int i;
	for(i = 0; i < 20000; i++)
		ptr[i*4000] = 'X';

	printf("Memory Writen\n"); fflush(stdout);
	gets(buf);

	free(ptr);
	printf("ptr1 free'd\n"); fflush(stdout);

	gets(buf);
}

-- 
Bruno Medeiros - MSc in CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 04 2006
next sibling parent Serg Kovrov <kovrov no.spam> writes:
* Bruno Medeiros:

 I don't know what program he used to monitor  the mem usage
 ("process explorer", is that the Windows Task Manager?), but 
 my test confirms what he's saying.


I use sysinternals process explorer as substitute to windows task 
manager. I believe it reads perfmon counters for that purpose. Great 
tool, BTW. Worth looking at:
http://www.sysinternals.com/Utilities/ProcessExplorer.html

-- 
serg.
Sep 04 2006
prev sibling parent reply Walter Bright <newshound digitalmars.com> writes:
Bruno Medeiros wrote:

 I think Serg is right. I don't know what program he used to monitor  the 
 mem usage ("process explorer", is that the Windows Task Manager?), but 
 my test confirms what he's saying.
 
 I made a small C program that allocs 100Mb and then frees it. I 
 monitored it with Windows XP Task Manager process list, looking at the 
 fields "Mem Usage" (physical mem) and "VM Size". Compiling with GCC 
 (3.2.3), it does free back the memory to the OS (VM Size decreases after 
 the free). I tried the program with DMC but the same does not happen: 
 the memory is not returned after the free. I don't have visual C 
 available right now so I didn't try that one.
 
 After a google search:
 http://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html
 "Occasionally, free can actually return memory to the operating system 
 and make the process smaller."
 Evidently the Windows version of libc also does the same. And seems it's 
 smart enough to return the mem not just when the top of the heap is all 
 free (what I was expecting) but also with free pages in the middle of 
 the heap. (the test program allocs two segments and frees the first only)


What your program is doing is allocating a gigantic chunk. It is a 
reasonable thing for new() to have a fork in it, and for gigantic chunks 
allocate/free them by calling the OS directly and not attempting to 
manage it. Thus you would see the behavior you see.

Try allocating a large number of small chunks, and free them.
Sep 04 2006
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Walter Bright wrote:

 Bruno Medeiros wrote:

 I think Serg is right. I don't know what program he used to monitor  
 the mem usage ("process explorer", is that the Windows Task Manager?), 
 but my test confirms what he's saying.

 I made a small C program that allocs 100Mb and then frees it. I 
 monitored it with Windows XP Task Manager process list, looking at the 
 fields "Mem Usage" (physical mem) and "VM Size". Compiling with GCC 
 (3.2.3), it does free back the memory to the OS (VM Size decreases 
 after the free). I tried the program with DMC but the same does not 
 happen: the memory is not returned after the free. I don't have visual 
 C available right now so I didn't try that one.

 After a google search:
 http://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html 

 "Occasionally, free can actually return memory to the operating system 
 and make the process smaller."
 Evidently the Windows version of libc also does the same. And seems 
 it's smart enough to return the mem not just when the top of the heap 
 is all free (what I was expecting) but also with free pages in the 
 middle of the heap. (the test program allocs two segments and frees 
 the first only)


 
 What your program is doing is allocating a gigantic chunk. It is a 
 reasonable thing for new() to have a fork in it, and for gigantic chunks 
 allocate/free them by calling the OS directly and not attempting to 
 manage it. Thus you would see the behavior you see.
 
 Try allocating a large number of small chunks, and free them.


Hum, I modified the program and tried 4 more tests:

Allocating 10000 (sequential) chunks of size 10000 bytes. Free them all.
-> All the 100Mb of memory is returned to the OS.

Allocating 100000 (sequential) chunks of size 1000 bytes. Free them all.
-> All the 100Mb of memory is returned to the OS.

Allocating 10000 (sequential) chunks of size 10000 bytes. Free only half 
of them, the even numbered ones, so that the total freed memory is not 
contiguous.
-> Of the 50Mb memory free'd, about 30Mb of memory is returned to the 
OS. Expected due to page segmentantion/rounding.

Allocating 100000 (sequential) chunks of size 1000 bytes. Again free 
only the even numbered ones.
-> 50Mb memory is free'd, but no memory is returned to the OS. Expected 
due to page segmentantion/rounding.

So it seems glibc does its best, it returns any page if it is all free. 
(hum, and I'm curious to what the results in VC++ are, if anyone tries 
it, do post the results)

------ test program ------
#include <stdio.h>
#include <assert.h>

#define CHUNKSIZE 10000
#define NUMCHUNKS 10000
#define CHUNKINC 1 //use CHUNKINC 2 for even numbered freeing

int main() {
	char buf[666];
	int i;
	char* ptrs[NUMCHUNKS];
	
	printf("NumChunks: %d Size: %d Inc: %d\n", NUMCHUNKS, CHUNKSIZE, 
CHUNKINC);
	fflush(stdout); gets(buf);
	
	for(i = 0; i < NUMCHUNKS; ++i) {
		ptrs[i] = (char*) malloc(CHUNKSIZE);
		assert(ptrs[i] != NULL);
	}

	char* ptrs2[NUMCHUNKS];

	for(i = 0; i < NUMCHUNKS; ++i) {
		ptrs2[i] = (char*) malloc(CHUNKSIZE);
		assert(ptrs2[i] != NULL);
		assert(ptrs2[i] > ptrs[NUMCHUNKS-1]);
	}
	printf("Mem Allocated\n"); fflush(stdout); gets(buf);
	
/*	for(i = 0; i < NUMCHUNKS; i++)
		ptrs[i][0] = 'X';
	printf("Memory Writen\n"); fflush(stdout);	gets(buf);
*/

	for(i = 0; i < NUMCHUNKS; i += CHUNKINC) {
		free(ptrs[i]);
	}
	printf("Mem free'd\n"); fflush(stdout); gets(buf);
}

-- 
Bruno Medeiros - MSc in CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 05 2006
next sibling parent Serg Kovrov <kovrov no.spam> writes:
* Bruno Medeiros:

 Walter Bright wrote:

 What your program is doing is allocating a gigantic chunk. It is a 
 reasonable thing for new() to have a fork in it, and for gigantic 
 chunks allocate/free them by calling the OS directly and not 
 attempting to manage it. Thus you would see the behavior you see.

 Try allocating a large number of small chunks, and free them.


 
 Hum, I modified the program and tried 4 more tests:
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free only half 
 of them, the even numbered ones, so that the total freed memory is not 
 contiguous.
 -> Of the 50Mb memory free'd, about 30Mb of memory is returned to the 
 OS. Expected due to page segmentantion/rounding.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Again free 
 only the even numbered ones.
 -> 50Mb memory is free'd, but no memory is returned to the OS. Expected 
 due to page segmentantion/rounding.
 
 So it seems glibc does its best, it returns any page if it is all free. 
 (hum, and I'm curious to what the results in VC++ are, if anyone tries 
 it, do post the results)


I can't be sure, but it's possible that Microsoft runtime library 
'delete' implementation calls _heapmin() 
[http://msdn2.microsoft.com/en-us/library/fc7etheh.aspx] for that 
purpose. I red about it some time ago in a post on 
comp.os.ms-windows.programmer. The actual discussion was about delete 
not returning memory to OS =) I think it really depends on some other 
factors. The point was that 'delete'(or 'free') *could* return memory to 
OS, but not guaranteed.

Phobos' GC has a stub f-n std.gc.minimize(), which probably intended for 
same purpose. I hope Walter could provide an update on it real purpose 
and implementation plans.

-- 
serg.
Sep 05 2006
prev sibling next sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
10_000 chunks of 10_000 (freeing everything - CHUNKINC = 1):

On startup...
	Mem usage: 612k
	VM Size: 240k

After alloc:
	Mem usage: 119,504k
	VM Size: 195k

After free:
	Mem usage: 60,272k
	VM Size 98,296k


DMD version... same constants.

Startup...
	Mem usage: 1,304k
	VM Size: 696k

After alloc (which takes much longer):
	Mem usage: 248,208k
	VM Size: 248,208k

After free: same as above.

I used the below source.

-[Unknown]


import std.c.stdio;

const CHUNKSIZE = 10_000;
const NUMCHUNKS = 10_000;
const CHUNKINC = 1; // Use CHUNKINC 2 for even numbered freeing.

int main()
{
     char buf[666];
     int i;
     ubyte*[NUMCHUNKS] ptrs;

     printf("NumChunks: %d Size: %d Inc: %d\n", NUMCHUNKS, CHUNKSIZE, 
CHUNKINC);
     fflush(stdout); gets(buf);

     for (i = 0; i < NUMCHUNKS; ++i)
     {
         ptrs[i] = new ubyte[CHUNKSIZE];
         assert (ptrs[i] != null);
     }

     ubyte*[NUMCHUNKS] ptrs2;

     for(i = 0; i < NUMCHUNKS; ++i) {
         ptrs2[i] = new ubyte[CHUNKSIZE];
         assert (ptrs2[i] != null);
         assert (ptrs2[i] > ptrs[NUMCHUNKS - 1]);
     }
     printf("Mem Allocated\n"); fflush(stdout); gets(buf);

/*
     for (i = 0; i < NUMCHUNKS; i++)
         ptrs[i][0] = 42;
     printf("Memory Writen\n"); fflush(stdout); gets(buf);
*/

     for (i = 0; i < NUMCHUNKS; i += CHUNKINC)
     {
         delete ptrs[i];
     }
     printf("Mem free'd\n"); fflush(stdout); gets(buf);

     return 0;
}


 Hum, I modified the program and tried 4 more tests:
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free only half 
 of them, the even numbered ones, so that the total freed memory is not 
 contiguous.
 -> Of the 50Mb memory free'd, about 30Mb of memory is returned to the 
 OS. Expected due to page segmentantion/rounding.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Again free 
 only the even numbered ones.
 -> 50Mb memory is free'd, but no memory is returned to the OS. Expected 
 due to page segmentantion/rounding.
 
 So it seems glibc does its best, it returns any page if it is all free. 
 (hum, and I'm curious to what the results in VC++ are, if anyone tries 
 it, do post the results)
 
 ------ test program ------
 #include <stdio.h>
 #include <assert.h>
 
 #define CHUNKSIZE 10000
 #define NUMCHUNKS 10000
 #define CHUNKINC 1 //use CHUNKINC 2 for even numbered freeing
 
 int main() {
     char buf[666];
     int i;
     char* ptrs[NUMCHUNKS];
     
     printf("NumChunks: %d Size: %d Inc: %d\n", NUMCHUNKS, CHUNKSIZE, 
 CHUNKINC);
     fflush(stdout); gets(buf);
     
     for(i = 0; i < NUMCHUNKS; ++i) {
         ptrs[i] = (char*) malloc(CHUNKSIZE);
         assert(ptrs[i] != NULL);
     }
 
     char* ptrs2[NUMCHUNKS];
 
     for(i = 0; i < NUMCHUNKS; ++i) {
         ptrs2[i] = (char*) malloc(CHUNKSIZE);
         assert(ptrs2[i] != NULL);
         assert(ptrs2[i] > ptrs[NUMCHUNKS-1]);
     }
     printf("Mem Allocated\n"); fflush(stdout); gets(buf);
     
 /*    for(i = 0; i < NUMCHUNKS; i++)
         ptrs[i][0] = 'X';
     printf("Memory Writen\n"); fflush(stdout);    gets(buf);
 */
 
     for(i = 0; i < NUMCHUNKS; i += CHUNKINC) {
         free(ptrs[i]);
     }
     printf("Mem free'd\n"); fflush(stdout); gets(buf);
 }
Sep 05 2006
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Unknown W. Brackets wrote:

 10_000 chunks of 10_000 (freeing everything - CHUNKINC = 1):
 
 On startup...
     Mem usage: 612k
     VM Size: 240k
 
 After alloc:
     Mem usage: 119,504k
     VM Size: 195k


Typo there, should be about VM Size: 195,000k.


 After free:
     Mem usage: 60,272k
     VM Size 98,296k
 


You forgot to mention which compiler(and compiler version) was used. *g*



 
 DMD version... same constants.


DMD? Why not just use DMC?

-- 
Bruno Medeiros - MSc in CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 05 2006
parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Oops, my mistake.  Yes, 195,000k.  That's what I get for posting 
something when I wasn't feeling well.

I meant to say, VC++ as requested.  I'm using cl 13.10.3077 for 80x86. 
Default options.

I used DMD because the comparison with DMC had already been made, but I 
didn't see anything equivalent for DMD.  DMD 0.162, by the way.

Thanks,
-[Unknown]



 Unknown W. Brackets wrote:

 10_000 chunks of 10_000 (freeing everything - CHUNKINC = 1):

 On startup...
     Mem usage: 612k
     VM Size: 240k

 After alloc:
     Mem usage: 119,504k
     VM Size: 195k


  >
 Typo there, should be about VM Size: 195,000k.
 

 After free:
     Mem usage: 60,272k
     VM Size 98,296k


 
 You forgot to mention which compiler(and compiler version) was used. *g*
 
 

 DMD version... same constants.


 
 DMD? Why not just use DMC?
Sep 06 2006
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Unknown W. Brackets wrote:

 Oops, my mistake.  Yes, 195,000k.  That's what I get for posting 
 something when I wasn't feeling well.
 
 I meant to say, VC++ as requested.  I'm using cl 13.10.3077 for 80x86. 
 Default options.
 


Ah, Visual Studio 2003


 I used DMD because the comparison with DMC had already been made, but I 
 didn't see anything equivalent for DMD.  DMD 0.162, by the way.
 


Walter had mentioned before that DMD's memory manager use C's malloc and 
free to allocate, so no difference in return behavior is to be expected.

-- 
Bruno Medeiros - MSc in CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 09 2006
parent reply Sean Kelly <sean f4.ca> writes:
Bruno Medeiros wrote:

 Walter had mentioned before that DMD's memory manager use C's malloc and 
 free to allocate, so no difference in return behavior is to be expected.


It actually uses VirtualAlloc, though malloc is a failover option in 
GDC's memory manager.


Sean
Sep 09 2006
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Sean Kelly wrote:

 Bruno Medeiros wrote:

 Walter had mentioned before that DMD's memory manager use C's malloc 
 and free to allocate, so no difference in return behavior is to be 
 expected.


 
 It actually uses VirtualAlloc, though malloc is a failover option in 
 GDC's memory manager.
 
 
 Sean


Hum, didn't Walter once mention that nearly all compiler implementations 
of new and delete used malloc? I assumed his compilers where one of such 
(didn't look at Phobos). Perhaps that simply was not the case. Or 
perhaps he just meant C++ compilers?


-- 
Bruno Medeiros - MSc in CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 22 2006
parent Sean Kelly <sean f4.ca> writes:
Bruno Medeiros wrote:

 Sean Kelly wrote:

 Bruno Medeiros wrote:

 Walter had mentioned before that DMD's memory manager use C's malloc 
 and free to allocate, so no difference in return behavior is to be 
 expected.


 It actually uses VirtualAlloc, though malloc is a failover option in 
 GDC's memory manager.


 
 Hum, didn't Walter once mention that nearly all compiler implementations 
 of new and delete used malloc? I assumed his compilers where one of such 
 (didn't look at Phobos). Perhaps that simply was not the case. Or 
 perhaps he just meant C++ compilers?


He meant C++ compilers.  But the situation in D is somewhat different 
because of garbage collection.  malloc is itself a memory allocator and 
D's GC is also an allocator, so having D's allocator call malloc would 
be like one retail store buying all their products from another retail 
store in bulk--it's much more efficient just to order from the 
distributer :-)


Sean
Sep 22 2006
prev sibling parent Walter Bright <newshound digitalmars.com> writes:
Bruno Medeiros wrote:

 Hum, I modified the program and tried 4 more tests:
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Free them all.
 -> All the 100Mb of memory is returned to the OS.
 
 Allocating 10000 (sequential) chunks of size 10000 bytes. Free only half 
 of them, the even numbered ones, so that the total freed memory is not 
 contiguous.
 -> Of the 50Mb memory free'd, about 30Mb of memory is returned to the 
 OS. Expected due to page segmentantion/rounding.
 
 Allocating 100000 (sequential) chunks of size 1000 bytes. Again free 
 only the even numbered ones.
 -> 50Mb memory is free'd, but no memory is returned to the OS. Expected 
 due to page segmentantion/rounding.
 
 So it seems glibc does its best, it returns any page if it is all free. 


So it does seem to, and so my statement is obsolete.
Sep 08 2006
prev sibling parent Helmut Leitner <leitner wikiservice.at> writes:
Walter Bright wrote:

 Serg Kovrov wrote:
 

 But that not worst part. Although I'm personally from C++ camp and I 
 get used to GC. and see it as good thing. But, there is one big BUT! 
 Currently GC do not return memory to OS, and that thing even I can't 
 subdue. The only reason I choose 'native' (as opposite to VM) 
 programming language is effective resource usage. I hate Java/.net gui 
 applications because their memory consuming. I really appreciate 
 developers that choose c/c++ to write small, memory effective, but yet 
 feature-reach applications like FileZilla Server, uTorrent or Miranda IM.

 Most desktop applications (like text editors, news readers, web 
 browsers, etc) must coexist with each other. That is, to use memory 
 wisely and give it back if it not needed anymore. This 'upper water 
 mark' approach in GC is just not acceptable.


 
 
 1) Few programmers realize that C/C++'s malloc/free/new/delete *never* 
 (and I emphasize NEVER) return memory to the operating system. All 
 free/delete do is return memory to the memory pool managed by the 
 runtime library, not the operating system. In order to actually return 
 memory to the operating system, one has to write their own memory 
 management code, which is a significant (and necessarily non-portable) 
 effort. Hardly anyone does that.


I think this doesn't give a complete picture of the situation.

There is the need to give memory back to the OS (e. g. if a user
has just stopped working with a couple of 100 MB pictures).

Operating systems sometimes do offer mechanisms for this (e. g.
Windows has the GlobalAlloc/GlobalReAlloc/GlobalFree set of functions).
If you use a wrapper for memory allocation it is very easy to use
this efficiently (pure or mixed with malloc/realloc/free).
Sep 08 2006