www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - deprecated delete and manual memory management

reply Benjamin Thaut <code benjamin-thaut.de> writes:
I've been reading through various delete topics in the past hour, but 
couldn't find any statement on how manual memory management would look, 
if the delete operator is deprecated. Something like the following seems 
really odd:

class foo {
	public new(size_t sz){ //language support
		return malloc(sz);
	}

	public void Delete(){ // no language support ??
		this.__dtor();
		free(this);
	}
}

auto fooInst = new foo(); //language support
fooInst.Delete(); //no language support ??
-- 
Kind Regards
Benjamin Thaut
Apr 26 2011
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/26/11 12:36 PM, Benjamin Thaut wrote:
 I've been reading through various delete topics in the past hour, but
 couldn't find any statement on how manual memory management would look,
 if the delete operator is deprecated. Something like the following seems
 really odd:

 class foo {
 public new(size_t sz){ //language support
 return malloc(sz);
 }

 public void Delete(){ // no language support ??
 this.__dtor();
 free(this);
 }
 }

 auto fooInst = new foo(); //language support
 fooInst.Delete(); //no language support ??

You'd just define functions for object creation and disposal. Andrei
Apr 26 2011
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 26 Apr 2011 13:36:37 -0400, Benjamin Thaut  
<code benjamin-thaut.de> wrote:

 I've been reading through various delete topics in the past hour, but  
 couldn't find any statement on how manual memory management would look,  
 if the delete operator is deprecated. Something like the following seems  
 really odd:

 class foo {
 	public new(size_t sz){ //language support
 		return malloc(sz);
 	}

 	public void Delete(){ // no language support ??
 		this.__dtor();
 		free(this);
 	}
 }

 auto fooInst = new foo(); //language support
 fooInst.Delete(); //no language support ??

IIUC, the custom allocator will be deprecated as well. What I think you need to use instead is emplace (a phobos function, not sure where it is), clear, and GC.free. Andrei hinted at a not-yet-written drop-in replacement for delete, but I'm not sure how that looks. -Steve
Apr 26 2011
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 26.04.2011 19:59, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 13:36:37 -0400, Benjamin Thaut
 <code benjamin-thaut.de> wrote:

 I've been reading through various delete topics in the past hour, but
 couldn't find any statement on how manual memory management would
 look, if the delete operator is deprecated. Something like the
 following seems really odd:

 class foo {
 public new(size_t sz){ //language support
 return malloc(sz);
 }

 public void Delete(){ // no language support ??
 this.__dtor();
 free(this);
 }
 }

 auto fooInst = new foo(); //language support
 fooInst.Delete(); //no language support ??

IIUC, the custom allocator will be deprecated as well. What I think you need to use instead is emplace (a phobos function, not sure where it is), clear, and GC.free. Andrei hinted at a not-yet-written drop-in replacement for delete, but I'm not sure how that looks. -Steve

I don't want to use the GC at all, and clear() seems inefficent, since it reaintializes the object. This is unneccsary since I just want to throw it away anyway. emplace only works for structs, it does not work for classes. Could someone please write a small example how manual memory management would look without new / delete? When the new operator can not be overloaded anymore, what happens when I dedicde to manualy manage the memory for a certain class? I then would have to search and replace every single new statement for that class. What happens when I forgett one? Some of my instances will then be on the GC heap and some on the self managed heap? -- Kind Regards Benjamin Thaut
Apr 26 2011
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 26.04.2011 20:23, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 14:14:11 -0400, Benjamin Thaut
 <code benjamin-thaut.de> wrote:
 
 Am 26.04.2011 19:59, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 13:36:37 -0400, Benjamin Thaut
 <code benjamin-thaut.de> wrote:

 I've been reading through various delete topics in the past hour, but
 couldn't find any statement on how manual memory management would
 look, if the delete operator is deprecated. Something like the
 following seems really odd:

 class foo {
 public new(size_t sz){ //language support
 return malloc(sz);
 }

 public void Delete(){ // no language support ??
 this.__dtor();
 free(this);
 }
 }

 auto fooInst = new foo(); //language support
 fooInst.Delete(); //no language support ??

IIUC, the custom allocator will be deprecated as well. What I think you need to use instead is emplace (a phobos function, not sure where it is), clear, and GC.free. Andrei hinted at a not-yet-written drop-in replacement for delete, but I'm not sure how that looks. -Steve

I don't want to use the GC at all, and clear() seems inefficent, since it reaintializes the object. This is unneccsary since I just want to throw it away anyway.

This is no longer the case for classes. It actually runs the same function that delete runs (rt_finalize). Though, I'm not sure it's in 2.052...
 emplace only works for structs, it does not work for classes.

I have not used it, but I'm under the impression from Andrei that it is to replace scope classes, so I would guess it has to work for them.

It *does* work for classes (as of dmd 2.052).
 Could someone please write a small example how manual memory
 management would look without new / delete?

I think that is a good idea.

Not a tutorial, just a simple example using C malloc/free: http://pastebin.com/HSBrk5kA Cheers, - Daniel
Apr 26 2011
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 26.04.2011 21:26, schrieb Daniel Gibson:
 Am 26.04.2011 20:23, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 14:14:11 -0400, Benjamin Thaut
 <code benjamin-thaut.de>  wrote:

 Am 26.04.2011 19:59, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 13:36:37 -0400, Benjamin Thaut
 <code benjamin-thaut.de>  wrote:

 I've been reading through various delete topics in the past hour, but
 couldn't find any statement on how manual memory management would
 look, if the delete operator is deprecated. Something like the
 following seems really odd:

 class foo {
 public new(size_t sz){ //language support
 return malloc(sz);
 }

 public void Delete(){ // no language support ??
 this.__dtor();
 free(this);
 }
 }

 auto fooInst = new foo(); //language support
 fooInst.Delete(); //no language support ??

IIUC, the custom allocator will be deprecated as well. What I think you need to use instead is emplace (a phobos function, not sure where it is), clear, and GC.free. Andrei hinted at a not-yet-written drop-in replacement for delete, but I'm not sure how that looks. -Steve

I don't want to use the GC at all, and clear() seems inefficent, since it reaintializes the object. This is unneccsary since I just want to throw it away anyway.

This is no longer the case for classes. It actually runs the same function that delete runs (rt_finalize). Though, I'm not sure it's in 2.052...
 emplace only works for structs, it does not work for classes.

I have not used it, but I'm under the impression from Andrei that it is to replace scope classes, so I would guess it has to work for them.

It *does* work for classes (as of dmd 2.052).
 Could someone please write a small example how manual memory
 management would look without new / delete?

I think that is a good idea.

Not a tutorial, just a simple example using C malloc/free: http://pastebin.com/HSBrk5kA Cheers, - Daniel

Thanks for the example, thats exactly what I needed. I still don't understand why the delete operator is deprecated completely. It could be defined, that it is only useable if the new and delete operator have been overloaded in the class or struct that is tried to be deleted. -- Kind Regards Benjamin Thaut
Apr 26 2011
next sibling parent reply Alexander <aldem+dmars nk7.net> writes:
On 26.04.2011 21:48, Benjamin Thaut wrote:

 I still don't understand why the delete operator is deprecated completely.

Me too, to be honest. Any pointers to this discussion? Custom new/delete ops could be *very* useful (and natural) for system-level programming (kernels & co), also for just manual MM (which is useful sometimes). /Alexander
Apr 26 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
 On 26.04.2011 21:48, Benjamin Thaut wrote:

 I still don't understand why the delete operator is deprecated completely.

Me too, to be honest. Any pointers to this discussion? Custom new/delete ops >

co), also for just manual MM (which is useful sometimes).
 /Alexander

But you understand why it is deprecated for GC memory? The main thing to note is that the semantics of C++ 'new' and D 'new' are rather different. D 'new' performs allocation on the GC heap by default. The only sane overloads of 'new' would therefore allocate the object on a custom GC heap, which you never want to do. So there is absolutely no way to overload the 'new' operator meaningfully inside the D language. The argument for removing 'delete' overloading is trivial after taking that into consideration. You can still create custom allocators by the means of template functions. (I do not think this is optimal though because they duplicate code that needn't be) They feel a little bit less natural though, which is not a problem, since in D, custom allocators and manual memory management in general, _are_ less natural. Benjamin Thaut wrote:
 I still don't understand why the delete operator is deprecated
 completely. It could be defined, that it is only useable if the new and
 delete operator have been overloaded in the class or struct that is
 tried to be deleted.

You don't gain anything by overloadable new/delete in D, because you cannot use the feature to quickly patch in a custom allocator to existing code as possible in C++ as a means of optimization. (this binds that allocator to a specific type, which does not make much sense in other context) I agree that new/delete is nicer to look at than the template-based version, but as discussed above, that does not really help. Timon
Apr 26 2011
next sibling parent Alexander <aldem+dmars nk7.net> writes:
On 26.04.2011 23:43, Timon Gehr wrote:

 But you understand why it is deprecated for GC memory?

Sure, I do. Though, sometimes it is useful to immediately deallocate an object, instead of delegating this to GC (and most likely - to delay deallocation). There was a post concerning thoughts of Linus Torvalds about GC, and the keyword is - "cache". Another possible use - in case of large objects, which are used for short period of time - it makes little sense to keep them around until next GC cycle, as this may lead to another request to OS, which may be unnecessary otherwise.
 The main thing to note is that the semantics of C++ 'new' and D 'new' are
rather
 different.

From programmer's point of view - not really. 'new' will allocate (and create) an object, which is not really different from C++. Or?
 D 'new' performs allocation on the GC heap by default. The only sane overloads
of
 'new' would therefore allocate the object on a custom GC heap, which you never
 want to do.

Why it is the only "sane"? Why I couldn't allocate from somewhere else? If D is positioned as language for system programming (not only, but anyway), it is quite OK to use other (de)allocators.
 The argument for removing 'delete' overloading is trivial after taking that
into consideration.

Still, to be honest, I don't see, what is the argument. 'delete' looks quite OK for explicit deallocation, regardless of where the object was allocated.
 You can still create custom allocators by the means of template functions. (I
do
 not think this is optimal though because they duplicate code that needn't be)

This way, we delegate all this to the developer, as he has to care - where objects should be allocated. OTOH, some objects (in specific environments) may "know better", where they should be allocated - that's why I believe that overloading makes sense.
 They feel a little bit less natural though, which is not a problem, since in
D, custom
 allocators and manual memory management in general, _are_ less natural.

Why? In some cases (again - system and real-time programming) - explicit (de)allocation is a must. Why not leave things as is? Those who don't need it will never use them, those who do need - will always have the option to use.
 You don't gain anything by overloadable new/delete in D, because you cannot use
 the feature to quickly patch in a custom allocator to existing code as
possible in
 C++ as a means of optimization. (this binds that allocator to a specific type,
 which does not make much sense in other context)

Exactly the point - for specific (derived) types, for specific application, when D Runtime/Phobos are not used (partially or fully) - this makes sense, IMHO. /Alexander
Apr 26 2011
prev sibling next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 27.04.2011 22:13, schrieb so:
 One thing that is perhaps obvious, but eludes me; when dropping the
 delete-operator, are there any obvious reason to not also drop the
 "new" keyword? ("new" is a fairly good method/variable name, if
 nothing else)

 I could see two possible alternatives:

 Global (and defined-by-default) template new!(Type): (as suggested
 elsewhere in this thread by so)
   This is very close to the current situation, but makes it possible
 to use as a method-name, and clearly states there's nothing "magic"
 about it.

 Explicit Allocator, such as GC.new!(Type).
   Would have the benefit of clearly showing who did the allocation,
 and would map nicely to other allocators. (Malloc.new/free!(T)!(T)).

   It would also allow library-methods that might allocate instances,
 take an allocator as an optional template argument. I.E.
     auto obj = dict.createObject("key");
   OR
     auto obj = dict.createObject!(Malloc)("key");
     scope(exit) Malloc.free(obj);

 Possibly an obvious bad idea, but I haven't seen it discussed?

 Regards
 / Ulrik

With the deprecated "delete", custom allocators and such i don't see much of a reason other than saving an "!" (actually we don't save a char here, " " also a char :) ), and "()" if you are calling the default constructor. I don't consider these points negative, quite contrary, you just got rid of an arcane syntax. It is understandable for C++ has it. If nothing else, the lack of many basic features (maybe the reason of this syntax). Rather then removing all together, i would drop them from language and go for a library solution, if there is an obvious blocker, i also fail to see it :)

It'd create template bloat and uglier syntax (expecially confusing for people coming from about any other popular OO language) for a really common, standard feature. These drawbacks are acceptable for custom allocation and other stuff the average user shouldn't care about, but not for an elemental feature like "new". Cheers, - Daniel
Apr 27 2011
next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 27.04.2011 22:37, schrieb so:
 It'd create template bloat and uglier syntax (expecially confusing for
 people coming from about any other popular OO language) for a really
 common, standard feature.
 These drawbacks are acceptable for custom allocation and other stuff the
 average user shouldn't care about, but not for an elemental feature like
 "new".

 Cheers,
 - Daniel

For the template bloat, yes that would be a problem. But it is not ugly! Take it back! :) auto a = new A; auto a = new!A(); auto b = new B(5); auto b = new!B(5); For the confusion part, the real confusion (rather shock) awaits when they get to the part where they see "new" but no "delete". We could argue against this all the way, but to every single of us "new" and "delete" are a pair.

No, in Java and C# there's no delete.
Apr 27 2011
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 27.04.2011 22:41, schrieb Daniel Gibson:
 Am 27.04.2011 22:37, schrieb so:
 It'd create template bloat and uglier syntax (expecially confusing for
 people coming from about any other popular OO language) for a really
 common, standard feature.
 These drawbacks are acceptable for custom allocation and other stuff the
 average user shouldn't care about, but not for an elemental feature like
 "new".

 Cheers,
 - Daniel

For the template bloat, yes that would be a problem. But it is not ugly! Take it back! :) auto a = new A; auto a = new!A(); auto b = new B(5); auto b = new!B(5); For the confusion part, the real confusion (rather shock) awaits when they get to the part where they see "new" but no "delete". We could argue against this all the way, but to every single of us "new" and "delete" are a pair.

No, in Java and C# there's no delete.

Also, new (== creating a new Object on the heap) is a standard feature in D that is needed all the time, delete (== manually destroy and deallocate an Object) isn't.
Apr 27 2011
parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 27.04.2011 22:53, schrieb so:
 On Wed, 27 Apr 2011 23:42:59 +0300, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 
 Am 27.04.2011 22:41, schrieb Daniel Gibson:
 Am 27.04.2011 22:37, schrieb so:
 It'd create template bloat and uglier syntax (expecially confusing for
 people coming from about any other popular OO language) for a really
 common, standard feature.
 These drawbacks are acceptable for custom allocation and other
 stuff the
 average user shouldn't care about, but not for an elemental feature
 like
 "new".

 Cheers,
 - Daniel

For the template bloat, yes that would be a problem. But it is not ugly! Take it back! :) auto a = new A; auto a = new!A(); auto b = new B(5); auto b = new!B(5); For the confusion part, the real confusion (rather shock) awaits when they get to the part where they see "new" but no "delete". We could argue against this all the way, but to every single of us "new" and "delete" are a pair.

No, in Java and C# there's no delete.

Also, new (== creating a new Object on the heap) is a standard feature in D that is needed all the time, delete (== manually destroy and deallocate an Object) isn't.

As Steven also pointed out:
 For non-garbage-collected languages, yes.  For GC languages, delete is
 to be discouraged (that is what the GC is for)

Which makes D special, since it claims it can do both. One would expect it to work as it advertised, without writing a whole runtime of your own.

This claim is indeed confusing. It has been there for D1 as well and I found it kind of disappointing that you couldn't do proper manual memory management with Objects of *any* class but just with custom classes that defined new() and delete()... This is better with D2 and emplace(), enabling to write simple custom (de)allocators - as my example has proven that can be done with about 5 lines for the allocator and 2 lines for the deallocator - that can be used for any class. In this case however delete makes no sense because you need to use your custom deallocator anyway. However I think the standard use of D (both 1 and 2) is to use new and no delete and no custom (de)allocators, even though it may have been advertised as a core feature. Cheers, - Daniel
Apr 27 2011
prev sibling next sibling parent KennyTM~ <kennytm gmail.com> writes:
On Apr 28, 11 04:28, Daniel Gibson wrote:
 It'd create template bloat and uglier syntax (expecially confusing for
 people coming from about any other popular OO language) for a really
 common, standard feature.
 These drawbacks are acceptable for custom allocation and other stuff the
 average user shouldn't care about, but not for an elemental feature like
 "new".

 Cheers,
 - Daniel

Er elemental? If 'new' is deprecated then a 'class' and 'array' will be constructed with class K { ... } auto k = K(...); // no new auto a = int[] (k.length); // no new same as how a 'struct' is constructed now. The only situation left where 'new!T()' is really necessary is when you need to create a T* on heap (GC.malloc) e.g. int* ps = new!int(4); I fail to see this is elemental in D.
Apr 27 2011
prev sibling next sibling parent reply Alexander <aldem+dmars nk7.net> writes:
On 27.04.2011 22:42, Steven Schveighoffer wrote:

 For non-garbage-collected languages, yes.  For GC languages, delete is to be
discouraged (that is what the GC is for).

delete() is 99% of the cases O(1) operation (thanks to free lists), while invocation of GC is O(?) (no one knows how many objects are pending deallocation, and when exactly it will be invoked). I agree that in normal applications (mostly) this is rarely an issue, but there are not normal applications, which I mentioned previously - RT & OS, and some others. Additionally, memory management hooks supported by the compiler are faster than any other solution (templates & co).
 Java and C# code do not have much use for delete either

What about those coming from C++ and D1 (including D2 up to this point)? But, actually, I am really interested in only one thing... I agree, that some may feel discomfort using delete, but what is the reason to remove it from the language? Probably, I've missed something, but why not to leave it as is for those who need it? /Alexander
Apr 27 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-04-28 00:15, Alexander wrote:
 On 27.04.2011 22:42, Steven Schveighoffer wrote:

 For non-garbage-collected languages, yes.  For GC languages, delete is to be
discouraged (that is what the GC is for).

delete() is 99% of the cases O(1) operation (thanks to free lists), while invocation of GC is O(?) (no one knows how many objects are pending deallocation, and when exactly it will be invoked). I agree that in normal applications (mostly) this is rarely an issue, but there are not normal applications, which I mentioned previously - RT& OS, and some others.

If you're writing an OS you would need to write your own runtime anyway.
    Additionally, memory management hooks supported by the compiler are faster
than any other solution (templates&  co).

 Java and C# code do not have much use for delete either

What about those coming from C++ and D1 (including D2 up to this point)? But, actually, I am really interested in only one thing... I agree, that some may feel discomfort using delete, but what is the reason to remove it from the language? Probably, I've missed something, but why not to leave it as is for those who need it? /Alexander

-- /Jacob Carlborg
Apr 28 2011
prev sibling parent reply Alexander <aldem+dmars nk7.net> writes:
On 28.04.2011 16:52, Steven Schveighoffer wrote:

   delete() is 99% of the cases O(1) operation (thanks to free lists),

Not exactly, it still needs to update the metadata for the block, which involves a binary search (O(lgn)) + you have to take the global lock, which means you could be making your multi-threaded application worse than a single-threaded one.

Well, I was assuming that metadata is attached to the block - there is no need to search. Of course, it depends on implementation. OTOH, when there are deletes performed in a bunch (+ time to scan the data for references), it could really take time. Taking a lock, IMHO, is a bit less expensive comparing to pausing all threads. But I may be wrong...
 The point is that it runs infrequently enough to avoid hindering performance.

Then there is another catch lurking around - the less frequently it runs, the more objects will be collected, so performance is hindered even more.
 D's GC is pretty horrible at performance, so there are definitely improvements
yet to be seen there (David recently fixed some really bad ones, the next
version of dmd
 should be much faster at allocation).

Is it possible to adapt (or use directly) Boehm's GC? Just an idea...
 As mentioned, RT applications and OS kernels would need a specialized runtime,
D does not support that with the current GC/runtime.  This does not mean that
you would be using delete for those, you'd probably use a specialized runtime
function using
 that specialized runtime.

Right, but using "delete" is a bit more "natural". IMHO, of course :)
   Additionally, memory management hooks supported by the compiler are faster
than any other solution (templates & co).

Absolutely untrue. Runtime hooks cannot be inlined for instance.

Unless compiler *knows* that there are hooks. Some believe that compiler must not handle anything specifically, but, to be really effective, this is difficult to avoid. Intrinsic functions are very useful.
 Having a dedicated keyword makes it seem sanctioned and promoted by the
language, when in fact, it should almost never be used.

D has some features which are discouraged anyway (like pointers & co.), but - D is a tool, and I believe that decision how to use a tool should be left to the user, that's all :) /Alexander
Apr 28 2011
parent reply Alexander <aldem+dmars nk7.net> writes:
On 28.04.2011 17:32, Steven Schveighoffer wrote:

 This is not how it's implemented.  The metadata is kept in a separate page
from the memory itself.

Huh? Could you please elaborate, what do you mean by "metadata" and why it has to be kept on a separate page? I've seen at least one GC implementation with free-lists and metadata preceding allocated block, so even most of the allocations took O(1) time (unless there was a shortage in memory).
 Not really.  The more objects there are to collect, the less scanning needs to
be done.  You only process blocks that have references to them.

Theoretically, any heap/stack-allocated block may have references. Stack scanning is relatively cheap (we know its current size, and usually it is small), but number of objects on the heap may be quite big, and heap itself as well. Or do I mistunderstood something? /Alexander
Apr 28 2011
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Steven Schveighoffer wrote:
 Yes, the allocation and free performance could be improved, but it doesn't
 change the fact that delete manually is not a huge performance gain, if at
 all.

My benchmark shows that in the current implementation manual deletion of GC memory is at least 15 times faster than leaving it up to the GC. (better cache behavior of the data unmeasured)
 If you want to gain performance, don't use the GC at all (for
 example scope classes), or use a custom allocator that is tailored to your
 needs.

Agreed. But writing a very well performing custom allocator that scales well with the application is a not-exactly-trivial task.
Apr 28 2011
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/28/11 2:27 PM, Timon Gehr wrote:
 Steven Schveighoffer wrote:
 Yes, the allocation and free performance could be improved, but it doesn't
 change the fact that delete manually is not a huge performance gain, if at
 all.

My benchmark shows that in the current implementation manual deletion of GC memory is at least 15 times faster than leaving it up to the GC. (better cache behavior of the data unmeasured)

Thanks for the data point. Another good one would be comparing new/delete with malloc/free. Andrei
Apr 28 2011
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Steven Schveighoffer wrote:
 I would need to see what your benchmark is before I'd say it was
 conclusive.

Ok. Version with delete is GC_hints (delete does not give many semantic guarantees, ergo it is a hint), version without delete is GC. It is also attached for convenience. This benchmark tests quite high loads on the GC heap, so for usual applications the GC won't perform all that bad. But then again, usual applications don't come with a memory allocation bottleneck. Andrei: The benchmark also includes a comparison to malloc/free (version c_heap). Timon /* Benchmark GC vs GC & hints vs freelist vs C Heap * My measurements on Intel Core Duo 2x2.4 GhZ on Ubuntu 10.04 (lucid) Amd64 * version=GC: 1m45s * version=GC_hints: ~6.8s * //using custom allocators: * version=freelist: ~3.7s * version=c_heap: ~4.5s */ import std.stdio; import std.c.stdlib; import std.c.string; import std.algorithm:swap; version=GC;//default //version=GC_hints; //version=freelist; //version=c_heap; version(freelist) version=calldel; version(GC_hints) version=calldel; version(c_heap) version=calldel; class foo{ char data[1020]=void; static foo freelist; foo next; version(freelist){ new(size_t s){ if(freelist !is null){ void *r=cast(void*)freelist; freelist=freelist.next; return r; } return cast(void*)new char[s];//could use malloc(s); effects are similar } delete(void *p){ if(p){ foo oldfl=freelist; freelist=cast(foo)p; freelist.next=oldfl; } } } version(c_heap){ new(size_t s){return malloc(s);} delete(void *p){if(p) free(p);} } }; foo[100000] a; void main(){ srand(100); int top; int dir;//0: expected growing, 1: expected shrinking foreach(t;0..10000000){ if(!top){ a[top++]=new foo; dir=0; }else if(top==100000){ top--; version(calldel) delete a[top]; dir=1; }else if(dir^!(rand()%3)){ top--; version(calldel) delete a[top]; }else a[top++]=new foo; if(!rand()%100) for(int i=0;i<100;i++) swap(a[0],a[rand()%top]);//mess around } }
Apr 28 2011
parent Timon Gehr <timon.gehr gmx.ch> writes:
Steven Schveighoffer wrote:
 I'll point out a couple of issues here:

 1. you are never removing elements from the array when version(calldel) is
 not enabled.  That is, the GC might run, but does not clean any objects
 when you just do top--.  Although this might not make a huge difference,
 it is not an equivalent comparison.  I'd suggest adding else clauses to
 the version(calldel) where you set a[top] to null.

Whoops, good catch, this is a bug. I get 1m27 when reassigning a[top] to null, which is a little bit faster.
 2. the GC is conservative, and you are allocating swaths of 1K objects.
 It's quite possible that you are running into false pointer issues.  This
 might be helped with a more precise GC (which there is a patch for in
 bugzilla).  David Simcha has also checked into github improvements for
 allocation of large objects.  This may also help with your benchmark.

The data=void can be removed, but it actually slows down the benchmark a little bit on my machine.
 I too have seen the GC perform horribly in situations like this, where you
 are allocating lots and lots of data.  All you need to do to prove it is
 append an array for so many elements, you will see huge pauses where the
 GC runs.

 These are situations where delete helps because of the conservative
 implementation of the GC.  Yes, it is a valid point, but it doesn't prove
 that delete is better against *any* GC.  What we need to do is improve the
 GC

 I tried setting a[top] to null in the GC system.  For comparison, on my
 system your original GC test took 2m42s.

 With setting a[top] to null, the new run on my system is 2m36s.  So the
 time does not decrease significantly.

 Some more data:

 I added the following line to the foreach loop:

 if(!(t % 10000)) writeln(t, "\t", top);

 There is some interesting behavior, particularly for the GC and GC_hints
 versions.  Your code obviously goes through cycles in the array, first
 growing up to 100,000, then shrinking back down to 0, then repeating the
 cycle.  There is some noise with the rand % 3, but it is not enough to
 make the growth and shrinking truly random.  What is interesting in this
 display is, in both GC_hints and GC versions, the initial growth to
 100,000 is somewhat slow, with the GC_hints version being a bit faster.

The growing and shrinking is intended to work exactly that way, it makes the allocations more, say, dynamic. ;) If it was truly random, the memory usage would be kept more or less constant.
 However, after that, the difference is drastic.  The GC_hints version
 shrinks very quickly, and the GC version continues its slow pace.  This is
 mildly surprising, because setting the top element to null should be much
 faster than delete, and both are using the GC to allocate new nodes.
 However, what I find really interesting is the subsequent growth back up
 to 100,000 is still slow in the GC version, but is still very speedy in
 the GC_hints version.  Given that both are using the GC for allocation, I
 would expect even footing.  There is definitely something different going
 on when delete is called than when a collection cycle is called.  The GC
 version may leave more objects allocated than the GC_hints version, but it
 should speed up as it can collect more object blocks to reuse.  It's
 almost as if it's not reusing memory, but my system isn't running out of
 memory.  So I'm not sure exactly what's going on.

 This clearly needs some investigation, thank you for sharing your
 benchmark.

 -Steve

Apr 29 2011
prev sibling parent reply Alexander <aldem+dmars nk7.net> writes:
On 28.04.2011 18:17, Steven Schveighoffer wrote:

 It would seem making the metadata be a constant offset from the data page
would be better, but of course, it cannot be that way for multi-page blocks.

Why it cannot?
 If you have ideas on how to improve the performance, I encourage you to learn
how the GC works and submit some patches!  It actually can be pretty fun.

Probably I do, I'll take a look.
 Yes, the allocation and free performance could be improved, but it doesn't
change the fact that delete manually is not a huge performance gain, if at all.

I agree, just in some cases it is a gain, if not to performance, then to memory usage. Imagine a tight loop with aggressive GC , when collection is attempted on every allocation...
 If an object is to be collected, there will be no references to that object. 
This means you do not have to scan that object's memory to see if it points to
something else.  Less memory needs to be scanned.

Right, but I still have to scan other objects and roots, which are still alive, and there can be many. Good GC shouldn't attempt a collection unless memory is tight, though. Anyway, I'll take a look into GC code, probably will make a benchmark to find out its weakness... /Alexander
Apr 28 2011
parent Alexander <aldem+dmars nk7.net> writes:
On 29.04.2011 15:10, Steven Schveighoffer wrote:

 I think D's GC only attempts a collection if memory is tight, but I'm not sure
now (after investigating somewhat Timon's benchmarks).

I am not sure as well - running some code I've noticed that memory usage is always quite low (there were a lot of allocations in a tight loop). PS: Quickly peeked into D's GC code - oh... it will take time :) /Alexander
Apr 29 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-04-27 22:28, Daniel Gibson wrote:
 Am 27.04.2011 22:13, schrieb so:
 One thing that is perhaps obvious, but eludes me; when dropping the
 delete-operator, are there any obvious reason to not also drop the
 "new" keyword? ("new" is a fairly good method/variable name, if
 nothing else)

 I could see two possible alternatives:

 Global (and defined-by-default) template new!(Type): (as suggested
 elsewhere in this thread by so)
    This is very close to the current situation, but makes it possible
 to use as a method-name, and clearly states there's nothing "magic"
 about it.

 Explicit Allocator, such as GC.new!(Type).
    Would have the benefit of clearly showing who did the allocation,
 and would map nicely to other allocators. (Malloc.new/free!(T)!(T)).

    It would also allow library-methods that might allocate instances,
 take an allocator as an optional template argument. I.E.
      auto obj = dict.createObject("key");
    OR
      auto obj = dict.createObject!(Malloc)("key");
      scope(exit) Malloc.free(obj);

 Possibly an obvious bad idea, but I haven't seen it discussed?

 Regards
 / Ulrik

With the deprecated "delete", custom allocators and such i don't see much of a reason other than saving an "!" (actually we don't save a char here, " " also a char :) ), and "()" if you are calling the default constructor. I don't consider these points negative, quite contrary, you just got rid of an arcane syntax. It is understandable for C++ has it. If nothing else, the lack of many basic features (maybe the reason of this syntax). Rather then removing all together, i would drop them from language and go for a library solution, if there is an obvious blocker, i also fail to see it :)

It'd create template bloat and uglier syntax (expecially confusing for people coming from about any other popular OO language) for a really common, standard feature. These drawbacks are acceptable for custom allocation and other stuff the average user shouldn't care about, but not for an elemental feature like "new". Cheers, - Daniel

Ruby uses "ClassName.new" to create a new instance and Objective-C uses "[[ClassName alloc] init]". Not all languages use "new ClassName". -- /Jacob Carlborg
Apr 28 2011
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 28.04.2011 10:28, schrieb Jacob Carlborg:
 On 2011-04-27 22:28, Daniel Gibson wrote:
 Am 27.04.2011 22:13, schrieb so:
 One thing that is perhaps obvious, but eludes me; when dropping the
 delete-operator, are there any obvious reason to not also drop the
 "new" keyword? ("new" is a fairly good method/variable name, if
 nothing else)

 I could see two possible alternatives:

 Global (and defined-by-default) template new!(Type): (as suggested
 elsewhere in this thread by so)
 This is very close to the current situation, but makes it possible
 to use as a method-name, and clearly states there's nothing "magic"
 about it.

 Explicit Allocator, such as GC.new!(Type).
 Would have the benefit of clearly showing who did the allocation,
 and would map nicely to other allocators. (Malloc.new/free!(T)!(T)).

 It would also allow library-methods that might allocate instances,
 take an allocator as an optional template argument. I.E.
 auto obj = dict.createObject("key");
 OR
 auto obj = dict.createObject!(Malloc)("key");
 scope(exit) Malloc.free(obj);

 Possibly an obvious bad idea, but I haven't seen it discussed?

 Regards
 / Ulrik

With the deprecated "delete", custom allocators and such i don't see much of a reason other than saving an "!" (actually we don't save a char here, " " also a char :) ), and "()" if you are calling the default constructor. I don't consider these points negative, quite contrary, you just got rid of an arcane syntax. It is understandable for C++ has it. If nothing else, the lack of many basic features (maybe the reason of this syntax). Rather then removing all together, i would drop them from language and go for a library solution, if there is an obvious blocker, i also fail to see it :)

It'd create template bloat and uglier syntax (expecially confusing for people coming from about any other popular OO language) for a really common, standard feature. These drawbacks are acceptable for custom allocation and other stuff the average user shouldn't care about, but not for an elemental feature like "new". Cheers, - Daniel

Ruby uses "ClassName.new" to create a new instance and Objective-C uses "[[ClassName alloc] init]". Not all languages use "new ClassName".

Ok. And probably Python doesn't as well. However D has this C/C++/Java/C# like syntax (Objective-C probably has that as well, but anyway), i.e. curly braces for blocks, C-like syntax for functions, try {..} catch(Foo f) {..} (yeah I know C doesn't have that, but no alternative different syntax either), ... so IMHO it makes sense for D to have the same syntax as the majority of these languages for the same feature (if that syntax doesn't have big enough drawbacks like <> for templates). Cheers, - Daniel
Apr 28 2011
parent Alexander <aldem+dmars nk7.net> writes:
On 28.04.2011 12:46, Daniel Gibson wrote:

 ... so IMHO it makes sense for D to have the same syntax as the majority of
these languages for the same feature (if that syntax doesn't have big enough
drawbacks like <> for templates).

That's my point too. And especially this is true when some features are already there for long time. /Alexander
Apr 28 2011
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 27.04.2011 20:21, Ulrik Mikaelsson wrote:
 2011/4/26 Timon Gehr<timon.gehr gmx.ch>:
 But you understand why it is deprecated for GC memory?

 The main thing to note is that the semantics of C++ 'new' and D 'new' are
rather
 different.
 D 'new' performs allocation on the GC heap by default. The only sane overloads
of
 'new' would therefore allocate the object on a custom GC heap, which you never
 want to do. So there is absolutely no way to overload the 'new' operator
 meaningfully inside the D language. The argument for removing 'delete'
overloading
 is trivial after taking that into consideration.

 You can still create custom allocators by the means of template functions. (I
do
 not think this is optimal though because they duplicate code that needn't be)
They
 feel a little bit less natural though, which is not a problem, since in D,
custom
 allocators and manual memory management in general, _are_ less natural.

delete-operator, are there any obvious reason to not also drop the "new" keyword? ("new" is a fairly good method/variable name, if nothing else) I could see two possible alternatives: Global (and defined-by-default) template new!(Type): (as suggested elsewhere in this thread by so) This is very close to the current situation, but makes it possible to use as a method-name, and clearly states there's nothing "magic" about it. Explicit Allocator, such as GC.new!(Type). Would have the benefit of clearly showing who did the allocation, and would map nicely to other allocators. (Malloc.new/free!(T)!(T)). It would also allow library-methods that might allocate instances, take an allocator as an optional template argument. I.E. auto obj = dict.createObject("key"); OR auto obj = dict.createObject!(Malloc)("key"); scope(exit) Malloc.free(obj); Possibly an obvious bad idea, but I haven't seen it discussed? Regards / Ulrik

I presonally think of new in D as a shortcut for yours would be GC.new!T(...). It's the same thing as with 'foreach' and such, i.e. used often enough to deserve a keyword. As for your suggestion, I see some problems with it: - yours Malloc and GC artifacts already have asymmetry, imagine generic code. Also there is stack-like allocators e.g. new David Simcha's TempAlloc - there is an awful lot of operations which allocate GC memory implicitly aka arr1 ~ arr2, so it's not like you can pass the allocator and prevent the function from allocating _somewhere_ else - I still fail to see how passing allocator would allow the function to decide on whether it's GC-like allocation or not, i.e. manual memory management implies the certain way function does its job (more generally the whole program, framework, module etc. it's not something decided at single call point) -- Dmitry Olshansky
Apr 27 2011
parent reply Alexander <aldem+dmars nk7.net> writes:
On 27.04.2011 22:46, Dmitry Olshansky wrote:

 - there is an awful lot of operations which allocate GC memory implicitly aka
arr1 ~ arr2, so it's not like you can pass the allocator and prevent the
function from allocating _somewhere_ else

This is solvable by using different runtime or class hierarchy, thus overriding "~" & co. Another option is to avoid constructs which may invoke GC (those are well-described).
 - I still fail to see how passing allocator would allow the function to decide
on whether it's GC-like allocation or not, i.e. manual memory management
implies the certain way function does its job (more generally the whole
program, framework, module
 etc. it's not something decided at single call point)

Again - different runtime and/or specialized class hierarchy. Custom (de)allocators are inherited, so one base class will do. /Alexander
Apr 27 2011
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 28.04.2011 2:19, Alexander wrote:
 On 27.04.2011 22:46, Dmitry Olshansky wrote:

 - there is an awful lot of operations which allocate GC memory implicitly aka
arr1 ~ arr2, so it's not like you can pass the allocator and prevent the
function from allocating _somewhere_ else


I think you might be surprised how many there are.
 - I still fail to see how passing allocator would allow the function to decide
on whether it's GC-like allocation or not, i.e. manual memory management
implies the certain way function does its job (more generally the whole
program, framework, module
 etc. it's not something decided at single call point)


memory management in c++ and can't see other ways. So I gather, the solution is to dump GC ? Hardly very useful. Also dropping language constructs is no go. (this 'avoid' is too vague) OK, fighting implicit allocation. With what - refcounting? Imagine array slices and such, when can you tell there is no references to this memory block ? Well, if you know what you are doing you could try and hack druntime to use refcounting, there was talk about it. Anyway, I just want to make sure you get the idea that GC is the default thing in D, not 'possible under certain circumstances' thing like it is in C++.
 /Alexander

-- Dmitry Olshansky
Apr 28 2011
parent Alexander <aldem+dmars nk7.net> writes:
On 28.04.2011 10:14, Dmitry Olshansky wrote:

 That's your C++ experience speaking ;) It seems you just like manual memory
management in c++ and can't see other ways.

This is not true, actually :) I agree, that GC is nice thing for most applications, and I really like it, but, I simply want to have an *option* to *control* memory allocation, if I choose to do so. Rarely, but I do need it (and not only me).
 Also dropping language constructs is no go. (this 'avoid' is too vague)

I didn't say so - just referring to http://digitalmars.com/d/2.0/garbage.html, namely section "D Operations That Involve the Garbage Collector". RT & OS applications are constrained anyway, so I see no problem there. And no, "dropping GC" is not an option.
 OK, fighting implicit allocation. With what - refcounting?

Manual management - whatever is this. malloc() with following free(), for instance - when there is no need anymore. But I prefer a bit more "natural" (not to everyone, of course) way - using delete. Even with GC, explicit delete may be useful (I've explained already, why).
  Anyway, I just want to make sure you get the idea that GC is the default
thing in D, not 'possible under certain circumstances' thing like it is in C++.

Believe me, I do get that idea - using it for many years in Perl & C# :) And I am strictly "pro" GC in D as a default, just want to have more options, that's all - that's why I don't like when those options are phased out with no (obvious to me) reason (which I want to find out). /Alexander
Apr 28 2011
prev sibling next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 26.04.2011 21:48, schrieb Benjamin Thaut:
 Am 26.04.2011 21:26, schrieb Daniel Gibson:
 Am 26.04.2011 20:23, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 14:14:11 -0400, Benjamin Thaut
 <code benjamin-thaut.de>  wrote:
 Could someone please write a small example how manual memory
 management would look without new / delete?

I think that is a good idea.

Not a tutorial, just a simple example using C malloc/free: http://pastebin.com/HSBrk5kA Cheers, - Daniel

Thanks for the example, thats exactly what I needed. I still don't understand why the delete operator is deprecated completely. It could be defined, that it is only useable if the new and delete operator have been overloaded in the class or struct that is tried to be deleted.

This way it's much more flexible. Note that you don't have to change the classes at all (i.e. you don't have to provide new() and delete()) to use a custom allocator. This means 1. you can use custom allocators with classes you didn't write yourself 2. you can even use different custom allocators for the same class at the same time (you'd have to be really careful not to screw that up by using the wrong deallocator for an Object) 3. you only have to write an (de)allocator once and you can use it for any class, even if they're not derived from each other Also, and I think this is really a great feature for performance that isn't available with overloaded new and delete operators, you can create an array of objects that really lie next to each other in memory (like an array of structs), so you can benefit from CPU cache effects. This is a very simple example doing that: http://pastebin.com/98mGU7y1 However you have to be careful with it if, i.e. don't put new objects in it, don't slice it (you have to at least feed the original array to myDeleteArr()) etc. To really use something like this probably a struct that enforces this (by conservatively overloading opIndex, opAssign etc) would be better than a function returning a dynamic array that happens to point to objects that are adjacent in memory. Cheers, - Daniel
Apr 26 2011
next sibling parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 26.04.2011 23:49, schrieb Andrej Mitrovic:
 On 4/26/11, Daniel Gibson <metalcaedes gmail.com> wrote:
 2. you can even use different custom allocators for the same class at
 the same time (you'd have to be really careful not to screw that up by
 using the wrong deallocator for an Object)

Maybe the allocator could annotate the class somehow, and the deallocator could check which allocator was used to create the object so it can call the matching deallocator. One way this could be done is through subclassing, but maybe that would mess things up. And this would probably mean runtime checks as well. I've never dealt with this before, but some ideas come to mind..

Yeah, just having one int/enum flag that indicates which allocator was used would be sufficient. However for that to work you have to use custom classes, that are all derived from MyObject or something like that (which is derived from Object and contains this flag). Depending on what you're doing this may not be a problem at all, but it's not as generic as being able to use *any* class. (Maybe?) Another possibility: the allocator returns a struct/class that contains this flag + the object and uses alias this to redirect everything to the contained class? I haven't really thought this through, but something like this could work. Cheers, - Daniel
Apr 26 2011
prev sibling next sibling parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 27.04.2011 00:31, schrieb so:
 On Wed, 27 Apr 2011 00:01:58 +0300, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 
 Am 26.04.2011 21:48, schrieb Benjamin Thaut:
 Am 26.04.2011 21:26, schrieb Daniel Gibson:
 Am 26.04.2011 20:23, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 14:14:11 -0400, Benjamin Thaut
 <code benjamin-thaut.de>  wrote:
 Could someone please write a small example how manual memory
 management would look without new / delete?

I think that is a good idea.

Not a tutorial, just a simple example using C malloc/free: http://pastebin.com/HSBrk5kA Cheers, - Daniel

Thanks for the example, thats exactly what I needed. I still don't understand why the delete operator is deprecated completely. It could be defined, that it is only useable if the new and delete operator have been overloaded in the class or struct that is tried to be deleted.

This way it's much more flexible. Note that you don't have to change the classes at all (i.e. you don't have to provide new() and delete()) to use a custom allocator. This means 1. you can use custom allocators with classes you didn't write yourself 2. you can even use different custom allocators for the same class at the same time (you'd have to be really careful not to screw that up by using the wrong deallocator for an Object) 3. you only have to write an (de)allocator once and you can use it for any class, even if they're not derived from each other Also, and I think this is really a great feature for performance that isn't available with overloaded new and delete operators, you can create an array of objects that really lie next to each other in memory (like an array of structs), so you can benefit from CPU cache effects. This is a very simple example doing that: http://pastebin.com/98mGU7y1 However you have to be careful with it if, i.e. don't put new objects in it, don't slice it (you have to at least feed the original array to myDeleteArr()) etc. To really use something like this probably a struct that enforces this (by conservatively overloading opIndex, opAssign etc) would be better than a function returning a dynamic array that happens to point to objects that are adjacent in memory. Cheers, - Daniel

Those that come from other languages (and D1), come with some baggage, it is strange not being able to see the huge advantages over new/delete operators. I agree with others, we need the best documentation we can get on this subject. A challenge. Write a "new" wrapper (lets call it "mynew") that logs each allocation (without preprocessor magic and C++0x). D: T mynew(T, A...)(A a) { printf("hello"); // return new T(a); return new!T(a); } Any takers for the C++ version? Daniel, you could have named them new and delete then everyone would be happy!

I could, but dmd wouldn't compile that because new and delete are reserved keywords ;) And by the way, your code won't compile because new!T(a) is illegal - new isn't a template. Or is that with my hypothetical new template that doesn't compiler either? Cheers, - Daniel
Apr 26 2011
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
so:

 I agree with others, we need the best documentation we can get on this subject.

In-place allocation (or generally other forms of manual allocation and deallocation) of class instances in D2 is a topic well fit for a *long* article for the iPad2 contest (written for future D programmers coming from C++ too), and later this good article has to go among the D docs of the official documentation :-) Bye, bearophile
Apr 26 2011
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 27.04.2011 00:58, schrieb bearophile:
 so:
 
 I agree with others, we need the best documentation we can get on this subject.

In-place allocation (or generally other forms of manual allocation and deallocation) of class instances in D2 is a topic well fit for a *long* article for the iPad2 contest (written for future D programmers coming from C++ too), and later this good article has to go among the D docs of the official documentation :-) Bye, bearophile

The D2 documentation should stop advertising custom allocators via new() and delete() and the scope storage class. This means http://www.digitalmars.com/d/2.0/memory.html and http://www.digitalmars.com/d/2.0/comparison.html (possibly more) should be rewritten or adjusted. BTW the comparison also lists strong typedefs which are also to be deprecated AFAIK. Furthermore: Why is emplace() in std.conv? I really wouldn't expect it there.. more likely I'd look in std.typecons, because scoped!T() is already there and does something related (allocate an object without new). And while I'm complaining: http://www.digitalmars.com/d/2.0/phobos/phobos.html doesn't list core.stdc.* Cheers, - Daniel
Apr 26 2011
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 27.04.2011 01:22, schrieb Daniel Gibson:
 Am 27.04.2011 00:58, schrieb bearophile:
 so:

 I agree with others, we need the best documentation we can get on this subject.

In-place allocation (or generally other forms of manual allocation and deallocation) of class instances in D2 is a topic well fit for a *long* article for the iPad2 contest (written for future D programmers coming from C++ too), and later this good article has to go among the D docs of the official documentation :-) Bye, bearophile

The D2 documentation should stop advertising custom allocators via new() and delete() and the scope storage class. This means http://www.digitalmars.com/d/2.0/memory.html and http://www.digitalmars.com/d/2.0/comparison.html (possibly more) should be rewritten or adjusted. BTW the comparison also lists strong typedefs which are also to be deprecated AFAIK. Furthermore: Why is emplace() in std.conv? I really wouldn't expect it there.. more likely I'd look in std.typecons, because scoped!T() is already there and does something related (allocate an object without new). And while I'm complaining: http://www.digitalmars.com/d/2.0/phobos/phobos.html doesn't list core.stdc.* Cheers, - Daniel

Oh and http://www.digitalmars.com/d/2.0/phobos/core_memory.html should also be adjusted in GC.free(): "If finalization is desired, use delete instead." Should probably changed to "If finalization is desired, call clear() on the Object/struct before calling GC.free()" or something like that. I just noticed something regarding clear(): struct Bar { ... } Bar *b = new Bar(); This compiles without any warning, but doesn't call the destructor: clear(b); This compiles without any warning and actually does what you'd expect, i.e. calls destructor: clear(*b); Is this behavior expected/wanted? If not: Is it a known bug? I couldn't find it in the bugtracker, but maybe I searched for the wrong keywords. Cheers, - Daniel
Apr 26 2011
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 27.04.2011 02:03, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 19:45:21 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 
 I just noticed something regarding clear():
   struct Bar { ... }
   Bar *b = new Bar();
 This compiles without any warning, but doesn't call the destructor:
   clear(b);
 This compiles without any warning and actually does what you'd expect,
 i.e. calls destructor:
   clear(*b);

 Is this behavior expected/wanted? If not: Is it a known bug? I couldn't
 find it in the bugtracker, but maybe I searched for the wrong keywords.

Let's start with class references. Because class references cannot be separated from its reference, you have to finalize the class when finalizing the reference, because there's no way to say "clear what this reference refers to" vs. "clear this reference". So you have to give a way to finalize a class instance. With pointers, however, you can specify as you say, whether you want to clear the pointer or the struct itself. Now, is it much useful to clear a pointer without clearing what it points to? I'd say no, clearing a pointer is as easy as doing ptr = null. So I'm thinking, it should be filed as a bug. The obvious thing to decide is, what should be done on references to references? If you clear a double pointer, should it go through both pointers? Or a pointer to a class reference? I'd say no, but you have to take extra steps to ensure it is this way. -Steve

IMHO clear isn't needed for anything but structs and Objects. For any simple type or pointer you can just write x = x.init; instead of clear(x) or, as you already mentioned, x=null; for pointers. AFAIK the main purpose of clear() is to explicitly call the destructor - and that only makes sense for structs and classes. Allowing it for other types (especially pointers) just sneaks in non-obvious bugs, especially when it's considered a replacement for delete (which calls the destructor for both Object a *struct). BTW: clear() has often been mentioned in this NG but isn't documented in the Phobos documentation (which is no surprise because clear() doesn't have doc-comments). So I guess I'll report this as a bug. Cheers, - Daniel
Apr 26 2011
next sibling parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 27.04.2011 02:26, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 20:15:45 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 
 Am 27.04.2011 02:03, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 19:45:21 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:

 I just noticed something regarding clear():
   struct Bar { ... }
   Bar *b = new Bar();
 This compiles without any warning, but doesn't call the destructor:
   clear(b);
 This compiles without any warning and actually does what you'd expect,
 i.e. calls destructor:
   clear(*b);

 Is this behavior expected/wanted? If not: Is it a known bug? I couldn't
 find it in the bugtracker, but maybe I searched for the wrong keywords.

Let's start with class references. Because class references cannot be separated from its reference, you have to finalize the class when finalizing the reference, because there's no way to say "clear what this reference refers to" vs. "clear this reference". So you have to give a way to finalize a class instance. With pointers, however, you can specify as you say, whether you want to clear the pointer or the struct itself. Now, is it much useful to clear a pointer without clearing what it points to? I'd say no, clearing a pointer is as easy as doing ptr = null. So I'm thinking, it should be filed as a bug. The obvious thing to decide is, what should be done on references to references? If you clear a double pointer, should it go through both pointers? Or a pointer to a class reference? I'd say no, but you have to take extra steps to ensure it is this way. -Steve

IMHO clear isn't needed for anything but structs and Objects. For any simple type or pointer you can just write x = x.init; instead of clear(x) or, as you already mentioned, x=null; for pointers.

It is useful for it to work for generic code.
 AFAIK the main purpose of clear() is to explicitly call the destructor -
 and that only makes sense for structs and classes.
 Allowing it for other types (especially pointers) just sneaks in
 non-obvious bugs, especially when it's considered a replacement for
 delete (which calls the destructor for both Object a *struct).

I'm not sure it introduces subtle bugs for things besides pointers. Clearing an int should be harmless.
 BTW: clear() has often been mentioned in this NG but isn't documented in
 the Phobos documentation (which is no surprise because clear() doesn't
 have doc-comments).

 So I guess I'll report this as a bug.

If you mean that clear has no docs, I already reported that. http://d.puremagic.com/issues/show_bug.cgi?id=5855 Feel free to report the enhancement request for clear to dereference pointers. -Steve

Done: http://d.puremagic.com/issues/show_bug.cgi?id=5895
Apr 26 2011
prev sibling parent reply Francisco Almeida <francisco.m.almeida gmail.com> writes:
== Quote from Daniel Gibson (metalcaedes gmail.com)'s article
 Am 27.04.2011 02:03, schrieb Steven Schveighoffer:
 IMHO clear isn't needed for anything but structs and Objects.
 For any simple type or pointer you can just write x = x.init; instead of
 clear(x) or, as you already mentioned, x=null; for pointers.
 AFAIK the main purpose of clear() is to explicitly call the destructor -
 and that only makes sense for structs and classes.
 Allowing it for other types (especially pointers) just sneaks in
 non-obvious bugs, especially when it's considered a replacement for
 delete (which calls the destructor for both Object a *struct).
 BTW: clear() has often been mentioned in this NG but isn't documented in
 the Phobos documentation (which is no surprise because clear() doesn't
 have doc-comments).
 So I guess I'll report this as a bug.
 Cheers,
 - Daniel

Regarding clear(), has the implementation already been corrected so that it does *not* call the constructor after object destruction?
Apr 27 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-04-27 15:00, Steven Schveighoffer wrote:
 On Wed, 27 Apr 2011 03:09:41 -0400, Francisco Almeida
 <francisco.m.almeida gmail.com> wrote:

 == Quote from Daniel Gibson (metalcaedes gmail.com)'s article
 Am 27.04.2011 02:03, schrieb Steven Schveighoffer:
 IMHO clear isn't needed for anything but structs and Objects.
 For any simple type or pointer you can just write x = x.init; instead of
 clear(x) or, as you already mentioned, x=null; for pointers.
 AFAIK the main purpose of clear() is to explicitly call the destructor -
 and that only makes sense for structs and classes.
 Allowing it for other types (especially pointers) just sneaks in
 non-obvious bugs, especially when it's considered a replacement for
 delete (which calls the destructor for both Object a *struct).
 BTW: clear() has often been mentioned in this NG but isn't documented in
 the Phobos documentation (which is no surprise because clear() doesn't
 have doc-comments).
 So I guess I'll report this as a bug.
 Cheers,
 - Daniel

Regarding clear(), has the implementation already been corrected so that it does *not* call the constructor after object destruction?

It's fixed in the latest git repository (BTW, how does one refer to the git "trunk"?) It's not in 2.052 -Steve

"head" or to be exact "HEAD" if you were wondering what to call it. -- /Jacob Carlborg
Apr 27 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-04-27 16:25, Steven Schveighoffer wrote:
 On Wed, 27 Apr 2011 10:17:30 -0400, Jacob Carlborg <doob me.com> wrote:

 "head" or to be exact "HEAD" if you were wondering what to call it.

Yes, thank you (that's twice you answered my naming questions, perhaps I should just email you directly next time ;) -Steve

Hehe :) -- /Jacob Carlborg
Apr 27 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-04-27 18:27, Ulrik Mikaelsson wrote:
 2011/4/27 Jacob Carlborg<doob me.com>:
 "head" or to be exact "HEAD" if you were wondering what to call it.
 --
 /Jacob Carlborg

Sorry to be nit-picking, but HEAD is whatever branch or commit is currently checked out into your working-copy. The git-equivalent of SVN-"trunk" is "master". The default-name of the branch that usually contains published stuff. (Although just as in SVN it's a convention, not a rule.)

Yeah, that's correct. But in this case I think he actually was referring to the latest commit. I'm pretty sure I've heard the latest commit in SVN be referred to as "trunk". -- /Jacob Carlborg
Apr 27 2011
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 29 Apr 2011 17:02:55 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:

 Steven Schveighoffer wrote:

 2. the GC is conservative, and you are allocating swaths of 1K objects.
 It's quite possible that you are running into false pointer issues.   
 This
 might be helped with a more precise GC (which there is a patch for in
 bugzilla).  David Simcha has also checked into github improvements for
 allocation of large objects.  This may also help with your benchmark.

The data=void can be removed, but it actually slows down the benchmark a little bit on my machine.

Yes, that's because you are using char data, whose default is 0xff :) data = void doesn't actually make that data random, because it still has to initialize the other parts of the class data. The run time doesn't have some way to "skip" certain parts of the initializer data. So it likely just sets it to 0. If you changed your data type to ubyte[], it would become fast again, even without the =void, because it's memsetting to 0 vs copying an initializer with some bytes set to 0xff. -Steve
May 02 2011
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 26 Apr 2011 14:14:11 -0400, Benjamin Thaut  
<code benjamin-thaut.de> wrote:

 Am 26.04.2011 19:59, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 13:36:37 -0400, Benjamin Thaut
 <code benjamin-thaut.de> wrote:

 I've been reading through various delete topics in the past hour, but
 couldn't find any statement on how manual memory management would
 look, if the delete operator is deprecated. Something like the
 following seems really odd:

 class foo {
 public new(size_t sz){ //language support
 return malloc(sz);
 }

 public void Delete(){ // no language support ??
 this.__dtor();
 free(this);
 }
 }

 auto fooInst = new foo(); //language support
 fooInst.Delete(); //no language support ??

IIUC, the custom allocator will be deprecated as well. What I think you need to use instead is emplace (a phobos function, not sure where it is), clear, and GC.free. Andrei hinted at a not-yet-written drop-in replacement for delete, but I'm not sure how that looks. -Steve

I don't want to use the GC at all, and clear() seems inefficent, since it reaintializes the object. This is unneccsary since I just want to throw it away anyway.

This is no longer the case for classes. It actually runs the same function that delete runs (rt_finalize). Though, I'm not sure it's in 2.052...
 emplace only works for structs, it does not work for classes.

I have not used it, but I'm under the impression from Andrei that it is to replace scope classes, so I would guess it has to work for them.
 Could someone please write a small example how manual memory management  
 would look without new / delete?

I think that is a good idea.
 When the new operator can not be overloaded anymore, what happens when I  
 dedicde to manualy manage the memory for a certain class? I then would  
 have to search and replace every single new statement for that class.  
 What happens when I forgett one? Some of my instances will then be on  
 the GC heap and some on the self managed heap?

disable the constructor, then use a static factory method to initialize the class with the correct heap. If this isn't satisfactory, someone else will have to answer, I'm not really one of those who ever used custom allocators or emplace. -Steve
Apr 26 2011
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 28 Apr 2011 16:14:48 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:

 Steven Schveighoffer wrote:
 I would need to see what your benchmark is before I'd say it was
 conclusive.

Ok. Version with delete is GC_hints (delete does not give many semantic guarantees, ergo it is a hint), version without delete is GC. It is also attached for convenience. This benchmark tests quite high loads on the GC heap, so for usual applications the GC won't perform all that bad. But then again, usual applications don't come with a memory allocation bottleneck. Andrei: The benchmark also includes a comparison to malloc/free (version c_heap). Timon /* Benchmark GC vs GC & hints vs freelist vs C Heap * My measurements on Intel Core Duo 2x2.4 GhZ on Ubuntu 10.04 (lucid) Amd64 * version=GC: 1m45s * version=GC_hints: ~6.8s * //using custom allocators: * version=freelist: ~3.7s * version=c_heap: ~4.5s */ import std.stdio; import std.c.stdlib; import std.c.string; import std.algorithm:swap; version=GC;//default //version=GC_hints; //version=freelist; //version=c_heap; version(freelist) version=calldel; version(GC_hints) version=calldel; version(c_heap) version=calldel; class foo{ char data[1020]=void; static foo freelist; foo next; version(freelist){ new(size_t s){ if(freelist !is null){ void *r=cast(void*)freelist; freelist=freelist.next; return r; } return cast(void*)new char[s];//could use malloc(s); effects are similar } delete(void *p){ if(p){ foo oldfl=freelist; freelist=cast(foo)p; freelist.next=oldfl; } } } version(c_heap){ new(size_t s){return malloc(s);} delete(void *p){if(p) free(p);} } }; foo[100000] a; void main(){ srand(100); int top; int dir;//0: expected growing, 1: expected shrinking foreach(t;0..10000000){ if(!top){ a[top++]=new foo; dir=0; }else if(top==100000){ top--; version(calldel) delete a[top]; dir=1; }else if(dir^!(rand()%3)){ top--; version(calldel) delete a[top]; }else a[top++]=new foo; if(!rand()%100) for(int i=0;i<100;i++) swap(a[0],a[rand()%top]);//mess around } }

I'll point out a couple of issues here: 1. you are never removing elements from the array when version(calldel) is not enabled. That is, the GC might run, but does not clean any objects when you just do top--. Although this might not make a huge difference, it is not an equivalent comparison. I'd suggest adding else clauses to the version(calldel) where you set a[top] to null. 2. the GC is conservative, and you are allocating swaths of 1K objects. It's quite possible that you are running into false pointer issues. This might be helped with a more precise GC (which there is a patch for in bugzilla). David Simcha has also checked into github improvements for allocation of large objects. This may also help with your benchmark. I too have seen the GC perform horribly in situations like this, where you are allocating lots and lots of data. All you need to do to prove it is append an array for so many elements, you will see huge pauses where the GC runs. These are situations where delete helps because of the conservative implementation of the GC. Yes, it is a valid point, but it doesn't prove that delete is better against *any* GC. What we need to do is improve the GC. I tried setting a[top] to null in the GC system. For comparison, on my system your original GC test took 2m42s. With setting a[top] to null, the new run on my system is 2m36s. So the time does not decrease significantly. Some more data: I added the following line to the foreach loop: if(!(t % 10000)) writeln(t, "\t", top); There is some interesting behavior, particularly for the GC and GC_hints versions. Your code obviously goes through cycles in the array, first growing up to 100,000, then shrinking back down to 0, then repeating the cycle. There is some noise with the rand % 3, but it is not enough to make the growth and shrinking truly random. What is interesting in this display is, in both GC_hints and GC versions, the initial growth to 100,000 is somewhat slow, with the GC_hints version being a bit faster. However, after that, the difference is drastic. The GC_hints version shrinks very quickly, and the GC version continues its slow pace. This is mildly surprising, because setting the top element to null should be much faster than delete, and both are using the GC to allocate new nodes. However, what I find really interesting is the subsequent growth back up to 100,000 is still slow in the GC version, but is still very speedy in the GC_hints version. Given that both are using the GC for allocation, I would expect even footing. There is definitely something different going on when delete is called than when a collection cycle is called. The GC version may leave more objects allocated than the GC_hints version, but it should speed up as it can collect more object blocks to reuse. It's almost as if it's not reusing memory, but my system isn't running out of memory. So I'm not sure exactly what's going on. This clearly needs some investigation, thank you for sharing your benchmark. -Steve
Apr 29 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 4/26/11, Daniel Gibson <metalcaedes gmail.com> wrote:
 2. you can even use different custom allocators for the same class at
 the same time (you'd have to be really careful not to screw that up by
 using the wrong deallocator for an Object)

Maybe the allocator could annotate the class somehow, and the deallocator could check which allocator was used to create the object so it can call the matching deallocator. One way this could be done is through subclassing, but maybe that would mess things up. And this would probably mean runtime checks as well. I've never dealt with this before, but some ideas come to mind..
Apr 26 2011
prev sibling next sibling parent so <so so.so> writes:
On Wed, 27 Apr 2011 00:01:58 +0300, Daniel Gibson <metalcaedes gmail.com>  
wrote:

 Am 26.04.2011 21:48, schrieb Benjamin Thaut:
 Am 26.04.2011 21:26, schrieb Daniel Gibson:
 Am 26.04.2011 20:23, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 14:14:11 -0400, Benjamin Thaut
 <code benjamin-thaut.de>  wrote:
 Could someone please write a small example how manual memory
 management would look without new / delete?

I think that is a good idea.

Not a tutorial, just a simple example using C malloc/free: http://pastebin.com/HSBrk5kA Cheers, - Daniel

Thanks for the example, thats exactly what I needed. I still don't understand why the delete operator is deprecated completely. It could be defined, that it is only useable if the new and delete operator have been overloaded in the class or struct that is tried to be deleted.

This way it's much more flexible. Note that you don't have to change the classes at all (i.e. you don't have to provide new() and delete()) to use a custom allocator. This means 1. you can use custom allocators with classes you didn't write yourself 2. you can even use different custom allocators for the same class at the same time (you'd have to be really careful not to screw that up by using the wrong deallocator for an Object) 3. you only have to write an (de)allocator once and you can use it for any class, even if they're not derived from each other Also, and I think this is really a great feature for performance that isn't available with overloaded new and delete operators, you can create an array of objects that really lie next to each other in memory (like an array of structs), so you can benefit from CPU cache effects. This is a very simple example doing that: http://pastebin.com/98mGU7y1 However you have to be careful with it if, i.e. don't put new objects in it, don't slice it (you have to at least feed the original array to myDeleteArr()) etc. To really use something like this probably a struct that enforces this (by conservatively overloading opIndex, opAssign etc) would be better than a function returning a dynamic array that happens to point to objects that are adjacent in memory. Cheers, - Daniel

Those that come from other languages (and D1), come with some baggage, it is strange not being able to see the huge advantages over new/delete operators. I agree with others, we need the best documentation we can get on this subject. A challenge. Write a "new" wrapper (lets call it "mynew") that logs each allocation (without preprocessor magic and C++0x). D: T mynew(T, A...)(A a) { printf("hello"); // return new T(a); return new!T(a); } Any takers for the C++ version? Daniel, you could have named them new and delete then everyone would be happy!
Apr 26 2011
prev sibling next sibling parent so <so so.so> writes:
 Or is that with my hypothetical new template that
 doesn't compiler either?

Yes, and the commented one is for the current solution.
Apr 26 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 26 Apr 2011 19:45:21 -0400, Daniel Gibson <metalcaedes gmail.com>  
wrote:

 I just noticed something regarding clear():
   struct Bar { ... }
   Bar *b = new Bar();
 This compiles without any warning, but doesn't call the destructor:
   clear(b);
 This compiles without any warning and actually does what you'd expect,
 i.e. calls destructor:
   clear(*b);

 Is this behavior expected/wanted? If not: Is it a known bug? I couldn't
 find it in the bugtracker, but maybe I searched for the wrong keywords.

Let's start with class references. Because class references cannot be separated from its reference, you have to finalize the class when finalizing the reference, because there's no way to say "clear what this reference refers to" vs. "clear this reference". So you have to give a way to finalize a class instance. With pointers, however, you can specify as you say, whether you want to clear the pointer or the struct itself. Now, is it much useful to clear a pointer without clearing what it points to? I'd say no, clearing a pointer is as easy as doing ptr = null. So I'm thinking, it should be filed as a bug. The obvious thing to decide is, what should be done on references to references? If you clear a double pointer, should it go through both pointers? Or a pointer to a class reference? I'd say no, but you have to take extra steps to ensure it is this way. -Steve
Apr 26 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 26 Apr 2011 20:15:45 -0400, Daniel Gibson <metalcaedes gmail.com>  
wrote:

 Am 27.04.2011 02:03, schrieb Steven Schveighoffer:
 On Tue, 26 Apr 2011 19:45:21 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:

 I just noticed something regarding clear():
   struct Bar { ... }
   Bar *b = new Bar();
 This compiles without any warning, but doesn't call the destructor:
   clear(b);
 This compiles without any warning and actually does what you'd expect,
 i.e. calls destructor:
   clear(*b);

 Is this behavior expected/wanted? If not: Is it a known bug? I couldn't
 find it in the bugtracker, but maybe I searched for the wrong keywords.

Let's start with class references. Because class references cannot be separated from its reference, you have to finalize the class when finalizing the reference, because there's no way to say "clear what this reference refers to" vs. "clear this reference". So you have to give a way to finalize a class instance. With pointers, however, you can specify as you say, whether you want to clear the pointer or the struct itself. Now, is it much useful to clear a pointer without clearing what it points to? I'd say no, clearing a pointer is as easy as doing ptr = null. So I'm thinking, it should be filed as a bug. The obvious thing to decide is, what should be done on references to references? If you clear a double pointer, should it go through both pointers? Or a pointer to a class reference? I'd say no, but you have to take extra steps to ensure it is this way. -Steve

IMHO clear isn't needed for anything but structs and Objects. For any simple type or pointer you can just write x = x.init; instead of clear(x) or, as you already mentioned, x=null; for pointers.

It is useful for it to work for generic code.
 AFAIK the main purpose of clear() is to explicitly call the destructor -
 and that only makes sense for structs and classes.
 Allowing it for other types (especially pointers) just sneaks in
 non-obvious bugs, especially when it's considered a replacement for
 delete (which calls the destructor for both Object a *struct).

I'm not sure it introduces subtle bugs for things besides pointers. Clearing an int should be harmless.
 BTW: clear() has often been mentioned in this NG but isn't documented in
 the Phobos documentation (which is no surprise because clear() doesn't
 have doc-comments).

 So I guess I'll report this as a bug.

If you mean that clear has no docs, I already reported that. http://d.puremagic.com/issues/show_bug.cgi?id=5855 Feel free to report the enhancement request for clear to dereference pointers. -Steve
Apr 26 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 27 Apr 2011 03:09:41 -0400, Francisco Almeida  
<francisco.m.almeida gmail.com> wrote:

 == Quote from Daniel Gibson (metalcaedes gmail.com)'s article
 Am 27.04.2011 02:03, schrieb Steven Schveighoffer:
 IMHO clear isn't needed for anything but structs and Objects.
 For any simple type or pointer you can just write x = x.init; instead of
 clear(x) or, as you already mentioned, x=null; for pointers.
 AFAIK the main purpose of clear() is to explicitly call the destructor -
 and that only makes sense for structs and classes.
 Allowing it for other types (especially pointers) just sneaks in
 non-obvious bugs, especially when it's considered a replacement for
 delete (which calls the destructor for both Object a *struct).
 BTW: clear() has often been mentioned in this NG but isn't documented in
 the Phobos documentation (which is no surprise because clear() doesn't
 have doc-comments).
 So I guess I'll report this as a bug.
 Cheers,
 - Daniel

Regarding clear(), has the implementation already been corrected so that it does *not* call the constructor after object destruction?

It's fixed in the latest git repository (BTW, how does one refer to the git "trunk"?) It's not in 2.052 -Steve
Apr 27 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 27 Apr 2011 10:17:30 -0400, Jacob Carlborg <doob me.com> wrote:

 "head" or to be exact "HEAD" if you were wondering what to call it.

Yes, thank you (that's twice you answered my naming questions, perhaps I should just email you directly next time ;) -Steve
Apr 27 2011
prev sibling next sibling parent Ulrik Mikaelsson <ulrik.mikaelsson gmail.com> writes:
2011/4/26 Timon Gehr <timon.gehr gmx.ch>:
 But you understand why it is deprecated for GC memory?

 The main thing to note is that the semantics of C++ 'new' and D 'new' are
rather
 different.
 D 'new' performs allocation on the GC heap by default. The only sane overloads
of
 'new' would therefore allocate the object on a custom GC heap, which you never
 want to do. So there is absolutely no way to overload the 'new' operator
 meaningfully inside the D language. The argument for removing 'delete'
overloading
 is trivial after taking that into consideration.

 You can still create custom allocators by the means of template functions. (I
do
 not think this is optimal though because they duplicate code that needn't be)
They
 feel a little bit less natural though, which is not a problem, since in D,
custom
 allocators and manual memory management in general, _are_ less natural.

One thing that is perhaps obvious, but eludes me; when dropping the delete-operator, are there any obvious reason to not also drop the "new" keyword? ("new" is a fairly good method/variable name, if nothing else) I could see two possible alternatives: Global (and defined-by-default) template new!(Type): (as suggested elsewhere in this thread by so) This is very close to the current situation, but makes it possible to use as a method-name, and clearly states there's nothing "magic" about it. Explicit Allocator, such as GC.new!(Type). Would have the benefit of clearly showing who did the allocation, and would map nicely to other allocators. (Malloc.new/free!(T)!(T)). It would also allow library-methods that might allocate instances, take an allocator as an optional template argument. I.E. auto obj = dict.createObject("key"); OR auto obj = dict.createObject!(Malloc)("key"); scope(exit) Malloc.free(obj); Possibly an obvious bad idea, but I haven't seen it discussed? Regards / Ulrik
Apr 27 2011
prev sibling next sibling parent Ulrik Mikaelsson <ulrik.mikaelsson gmail.com> writes:
2011/4/27 Jacob Carlborg <doob me.com>:
 "head" or to be exact "HEAD" if you were wondering what to call it.
 --
 /Jacob Carlborg

Sorry to be nit-picking, but HEAD is whatever branch or commit is currently checked out into your working-copy. The git-equivalent of SVN-"trunk" is "master". The default-name of the branch that usually contains published stuff. (Although just as in SVN it's a convention, not a rule.)
Apr 27 2011
prev sibling next sibling parent so <so so.so> writes:
 One thing that is perhaps obvious, but eludes me; when dropping the
 delete-operator, are there any obvious reason to not also drop the
 "new" keyword? ("new" is a fairly good method/variable name, if
 nothing else)

 I could see two possible alternatives:

 Global (and defined-by-default) template new!(Type): (as suggested
 elsewhere in this thread by so)
   This is very close to the current situation, but makes it possible
 to use as a method-name, and clearly states there's nothing "magic"
 about it.

 Explicit Allocator, such as GC.new!(Type).
   Would have the benefit of clearly showing who did the allocation,
 and would map nicely to other allocators. (Malloc.new/free!(T)!(T)).

   It would also allow library-methods that might allocate instances,
 take an allocator as an optional template argument. I.E.
     auto obj = dict.createObject("key");
   OR
     auto obj = dict.createObject!(Malloc)("key");
     scope(exit) Malloc.free(obj);

 Possibly an obvious bad idea, but I haven't seen it discussed?

 Regards
 / Ulrik

With the deprecated "delete", custom allocators and such i don't see much of a reason other than saving an "!" (actually we don't save a char here, " " also a char :) ), and "()" if you are calling the default constructor. I don't consider these points negative, quite contrary, you just got rid of an arcane syntax. It is understandable for C++ has it. If nothing else, the lack of many basic features (maybe the reason of this syntax). Rather then removing all together, i would drop them from language and go for a library solution, if there is an obvious blocker, i also fail to see it :)
Apr 27 2011
prev sibling next sibling parent so <so so.so> writes:
 It'd create template bloat and uglier syntax (expecially confusing for
 people coming from about any other popular OO language) for a really
 common, standard feature.
 These drawbacks are acceptable for custom allocation and other stuff the
 average user shouldn't care about, but not for an elemental feature like
 "new".

 Cheers,
 - Daniel

For the template bloat, yes that would be a problem. But it is not ugly! Take it back! :) auto a = new A; auto a = new!A(); auto b = new B(5); auto b = new!B(5); For the confusion part, the real confusion (rather shock) awaits when they get to the part where they see "new" but no "delete". We could argue against this all the way, but to every single of us "new" and "delete" are a pair.
Apr 27 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 27 Apr 2011 16:37:49 -0400, so <so so.so> wrote:

 For the confusion part, the real confusion (rather shock) awaits when  
 they get to the part where they see "new" but no "delete".
 We could argue against this all the way, but to every single of us "new"  
 and "delete" are a pair.

For non-garbage-collected languages, yes. For GC languages, delete is to be discouraged (that is what the GC is for). Java and C# code do not have much use for delete either, so people coming from those languages will feel right at home without the delete keyword I would think. -Steve
Apr 27 2011
prev sibling next sibling parent so <so so.so> writes:
On Wed, 27 Apr 2011 23:42:59 +0300, Daniel Gibson <metalcaedes gmail.com>  
wrote:

 Am 27.04.2011 22:41, schrieb Daniel Gibson:
 Am 27.04.2011 22:37, schrieb so:
 It'd create template bloat and uglier syntax (expecially confusing for
 people coming from about any other popular OO language) for a really
 common, standard feature.
 These drawbacks are acceptable for custom allocation and other stuff  
 the
 average user shouldn't care about, but not for an elemental feature  
 like
 "new".

 Cheers,
 - Daniel

For the template bloat, yes that would be a problem. But it is not ugly! Take it back! :) auto a = new A; auto a = new!A(); auto b = new B(5); auto b = new!B(5); For the confusion part, the real confusion (rather shock) awaits when they get to the part where they see "new" but no "delete". We could argue against this all the way, but to every single of us "new" and "delete" are a pair.

No, in Java and C# there's no delete.

Also, new (== creating a new Object on the heap) is a standard feature in D that is needed all the time, delete (== manually destroy and deallocate an Object) isn't.

As Steven also pointed out:
 For non-garbage-collected languages, yes.  For GC languages, delete is  
 to be discouraged (that is what the GC is for)

Which makes D special, since it claims it can do both. One would expect it to work as it advertised, without writing a whole runtime of your own.
Apr 27 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 27 Apr 2011 18:15:26 -0400, Alexander <aldem+dmars nk7.net> wrote:

 On 27.04.2011 22:42, Steven Schveighoffer wrote:

 For non-garbage-collected languages, yes.  For GC languages, delete is  
 to be discouraged (that is what the GC is for).

delete() is 99% of the cases O(1) operation (thanks to free lists),

Not exactly, it still needs to update the metadata for the block, which involves a binary search (O(lgn)) + you have to take the global lock, which means you could be making your multi-threaded application worse than a single-threaded one. I have seen this in practice when one uses memory allocation or deallocation frequently. clear is definitely O(1) (well, as much as you thought delete was O(1), it depends on the dtor).
 while invocation of GC is O(?) (no one knows how many objects are  
 pending deallocation, and when exactly it will be invoked).

The point is that it runs infrequently enough to avoid hindering performance. D's GC is pretty horrible at performance, so there are definitely improvements yet to be seen there (David recently fixed some really bad ones, the next version of dmd should be much faster at allocation). Yes, if you run the collector every time you are done using an object, performance will be much worse than using delete to destroy the object, but that is also not encouraged.
   I agree that in normal applications (mostly) this is rarely an issue,  
 but there are not normal applications, which I mentioned previously - RT  
 & OS, and some others.

As mentioned, RT applications and OS kernels would need a specialized runtime, D does not support that with the current GC/runtime. This does not mean that you would be using delete for those, you'd probably use a specialized runtime function using that specialized runtime.
   Additionally, memory management hooks supported by the compiler are  
 faster than any other solution (templates & co).

Absolutely untrue. Runtime hooks cannot be inlined for instance.
 Java and C# code do not have much use for delete either

What about those coming from C++ and D1 (including D2 up to this point)?

C++ users will have to adjust to an environment that provides a GC. D1/D2 users should use delete sparingly anyways. Migrating those few cases to use a new method shouldn't be too difficult.
   But, actually, I am really interested in only one thing... I agree,  
 that some may feel discomfort using delete, but what is the reason to  
 remove it from the language? Probably, I've missed something, but why  
 not to leave it as is for those who need it?

I think (and I'm not sure, Andrei was the one behind this) that the issue is that a keyword operation is very simple/easy to write, and deleting memory should be discouraged. Having a dedicated keyword makes it seem sanctioned and promoted by the language, when in fact, it should almost never be used. -Steve
Apr 28 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 28 Apr 2011 11:11:41 -0400, Alexander <aldem+dmars nk7.net> wrote:

 On 28.04.2011 16:52, Steven Schveighoffer wrote:

   delete() is 99% of the cases O(1) operation (thanks to free lists),

Not exactly, it still needs to update the metadata for the block, which involves a binary search (O(lgn)) + you have to take the global lock, which means you could be making your multi-threaded application worse than a single-threaded one.

Well, I was assuming that metadata is attached to the block - there is no need to search. Of course, it depends on implementation.

This is not how it's implemented. The metadata is kept in a separate page from the memory itself.
   OTOH, when there are deletes performed in a bunch (+ time to scan the  
 data for references), it could really take time. Taking a lock, IMHO, is  
 a bit less expensive comparing to pausing all threads. But I may be  
 wrong...

The point is, you only do it once in a while. If you delete objects 100 times a second, and the GC runs only every 10 seconds, then you have to consider the cost of pausing all threads and running a collection cycle against 10,000 times taking the lock. IMO, the GC runs too often in D, but we can improve that metric.
 The point is that it runs infrequently enough to avoid hindering  
 performance.

Then there is another catch lurking around - the less frequently it runs, the more objects will be collected, so performance is hindered even more.

Not really. The more objects there are to collect, the less scanning needs to be done. You only process blocks that have references to them. There is also a certain amount of overhead that runs regardless of how many objects are ready to be collected.
 D's GC is pretty horrible at performance, so there are definitely  
 improvements yet to be seen there (David recently fixed some really bad  
 ones, the next version of dmd
 should be much faster at allocation).

Is it possible to adapt (or use directly) Boehm's GC? Just an idea...

Yes, the GC is swappable. You need to re-compile the runtime with a different GC, all the hooks are movable. In fact, there's a GC stub that just calls C malloc for all allocations.
 As mentioned, RT applications and OS kernels would need a specialized  
 runtime, D does not support that with the current GC/runtime.  This  
 does not mean that you would be using delete for those, you'd probably  
 use a specialized runtime function using
 that specialized runtime.

Right, but using "delete" is a bit more "natural". IMHO, of course :)

For some, free might be more "natural" :) I think you can find another term to free data that is not a keyword easily enough.
   Additionally, memory management hooks supported by the compiler are  
 faster than any other solution (templates & co).

Absolutely untrue. Runtime hooks cannot be inlined for instance.

Unless compiler *knows* that there are hooks. Some believe that compiler must not handle anything specifically, but, to be really effective, this is difficult to avoid. Intrinsic functions are very useful.

The compiler cannot possibly make any assumptions about how memory management is done. It must dutifully call the runtime function, or else you could not implement interesting things in the GC.
 Having a dedicated keyword makes it seem sanctioned and promoted by the  
 language, when in fact, it should almost never be used.

D has some features which are discouraged anyway (like pointers & co.), but - D is a tool, and I believe that decision how to use a tool should be left to the user, that's all :)

But delete doesn't allow you to do things that you can't already do otherwise. Delete is not going away without replacing it with an equivalent functionality. Pointers and casting are a necessary evil for optimizing code and forcing things in the type system. There is no way to implement pointers and casting in the runtime. -Steve
Apr 28 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 28 Apr 2011 11:57:00 -0400, Alexander <aldem+dmars nk7.net> wrote:

 On 28.04.2011 17:32, Steven Schveighoffer wrote:

 This is not how it's implemented.  The metadata is kept in a separate  
 page from the memory itself.

Huh? Could you please elaborate, what do you mean by "metadata" and why it has to be kept on a separate page?

Flags for the memory, such as whether it is free, or whether it's reachable (during a scan), whether it has pointers, etc. Why it is kept on a separate page, I'm not sure, I didn't write that. It would seem making the metadata be a constant offset from the data page would be better, but of course, it cannot be that way for multi-page blocks. If you have ideas on how to improve the performance, I encourage you to learn how the GC works and submit some patches! It actually can be pretty fun.
   I've seen at least one GC implementation with free-lists and metadata  
 preceding allocated block, so even most of the allocations took O(1)  
 time (unless there was a shortage in memory).

Allocations are going to be less, because the memory is hopefully in a free list, and hopefully the metadata is pointed at by the free list (not sure about current GC implementation). Yes, the allocation and free performance could be improved, but it doesn't change the fact that delete manually is not a huge performance gain, if at all. If you want to gain performance, don't use the GC at all (for example scope classes), or use a custom allocator that is tailored to your needs.
 Not really.  The more objects there are to collect, the less scanning  
 needs to be done.  You only process blocks that have references to them.

Theoretically, any heap/stack-allocated block may have references. Stack scanning is relatively cheap (we know its current size, and usually it is small), but number of objects on the heap may be quite big, and heap itself as well. Or do I mistunderstood something?

If an object is to be collected, there will be no references to that object. This means you do not have to scan that object's memory to see if it points to something else. Less memory needs to be scanned. You start from your roots, and then start scanning from there. Your roots are your thread local storage, stack space, globals, and roots added manually. -Steve
Apr 28 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 28 Apr 2011 15:27:16 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:

 Steven Schveighoffer wrote:
 Yes, the allocation and free performance could be improved, but it  
 doesn't
 change the fact that delete manually is not a huge performance gain, if  
 at
 all.

My benchmark shows that in the current implementation manual deletion of GC memory is at least 15 times faster than leaving it up to the GC. (better cache behavior of the data unmeasured)

I would need to see what your benchmark is before I'd say it was conclusive.
 If you want to gain performance, don't use the GC at all (for
 example scope classes), or use a custom allocator that is tailored to  
 your
 needs.

Agreed. But writing a very well performing custom allocator that scales well with the application is a not-exactly-trivial task.

True. Generally, GC has good enough performance for most applications. But again, all of this is somewhat moot since delete isn't exactly going away, it's just going away as a keyword. -Steve
Apr 28 2011
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 28 Apr 2011 19:59:52 -0400, Alexander <aldem+dmars nk7.net> wrote:

 On 28.04.2011 18:17, Steven Schveighoffer wrote:

 It would seem making the metadata be a constant offset from the data  
 page would be better, but of course, it cannot be that way for  
 multi-page blocks.

Why it cannot?

Because in the second or subsequent page of a multi-page memory block, the previous page cannot be metadata, it must be data (this is one contiguous block). So how do you find the metadata for such a block? In fact, thinking about it, the metadata is what tells the GC how big the block is, and where it starts, so you can't store the metadata in a constant offset, it's just not possible.
 Yes, the allocation and free performance could be improved, but it  
 doesn't change the fact that delete manually is not a huge performance  
 gain, if at all.

I agree, just in some cases it is a gain, if not to performance, then to memory usage. Imagine a tight loop with aggressive GC , when collection is attempted on every allocation...

Yes, if you have specific memory requirements, it's best to use an alternative strategy. That does not mean delete is the answer, and even if it is, the functionality that delete represents is not going away, just the keyword itself does.
 If an object is to be collected, there will be no references to that  
 object.  This means you do not have to scan that object's memory to see  
 if it points to something else.  Less memory needs to be scanned.

Right, but I still have to scan other objects and roots, which are still alive, and there can be many.

Yes, but your assertion was that more dead objects means the scan takes longer. It's actually the opposite (or at least, it should be). The sweep phase probably takes longer, though, but I wouldn't expect this to be the bottleneck.
 Good GC shouldn't attempt a collection unless memory is tight, though.

I think D's GC only attempts a collection if memory is tight, but I'm not sure now (after investigating somewhat Timon's benchmarks). -Steve
Apr 29 2011