www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.allocator ready for some abuse

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Hello,


I know it's been a long wait. Hopefully it was worth it. The alpha 
release of untyped allocators is ready for tire-kicking and a test drive.

Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

Warning: this is alpha quality. Unit tests are thin, and there are no 
benchmarks. Both would be appreciated, particularly benchmarks to 
validate the gains (which I speculate can be very sizable) of 
custom-built, special-purpose allocators compared to traditional allocators.

I acknowledge I'm clearly in no position to evaluate this design. I have 
been knocking around it for long enough to have no idea how easy it is 
to get into it from the outside, or how good it is. By all signs I could 
gather this feels like good design, and one of the best I've ever put 
together. The allocators defined have an archetypal feeling, are 
flexible both statically and dynamically, and morph and combine in 
infinite ways.

CAllocator and CAllocatorImpl make the link between the static and 
dynamic worlds. Once an allocator is assembled out of pieces and finely 
tuned, wrapping it in a dynamic API is a snap.

Please destroy! I've literally sweat as I'm sending this :o).


Andrei
Oct 24 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 and morph and combine in infinite ways.
Yes, those combination capabilities look very nice :-) I presume this module is not in "core" because it's meant to be used from normal D code. - Could some new/extra D language/compiler support/features help this module usage safety, efficiency, expressibility or usage syntax (beside Issue 11331)? - Can you create an hierarchical allocator with those? - Are some operating system memory functions (below C malloc) used/supported/addable/useful for those allocators? (Including awareness of virtual memory management from the OS). Bye, bearophile
Oct 24 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 1:46 PM, bearophile wrote:
 Andrei Alexandrescu:

 and morph and combine in infinite ways.
Yes, those combination capabilities look very nice :-) I presume this module is not in "core" because it's meant to be used from normal D code.
Interesting - moving to core is a possibility if we find it necessary.
 - Could some new/extra D language/compiler support/features help this
 module usage safety, efficiency, expressibility or usage syntax (beside
 Issue 11331)?
I thought about that a lot, too, while I was working on the design. Kinda analyzing my own going. It's generally been a pleasant experience - i.e. whenever I'd find myself in a bind there was some nice way out of it. One thing that I found myself wishing is uniformly aliasing something to something else. There's plenty of code like: static if (staticallyKnownAlignment!Primary && staticallyKnownAlignment!Fallback) enum uint alignment = min(Primary.alignment, Fallback.alignment); else uint alignment() { return min(primary.alignment, fallback.alignment); } This is aimed at making the alignment either a statically-known constant or a dynamic property, depending on two type parameters. There's not a lot of duplication but it's quite annoying to have to write this. Would be great if I could just write e.g. alias alignment = min(primary.alignment, fallback.alignment); and have the thing just automagically do the right thing. There have been some good idioms I used gainfully, i.e. I think the use of unbounded and chooseAtRuntime could become quite common. "Turtles all the way" are awesome because they allow me to get work done without having to pop up and down. Consider for example: bool reallocate(ref void[] b, size_t newSize) { bool crossAllocatorMove(From, To)(ref From from, ref To to) { auto b1 = to.allocate(newSize); if (!b1) return false; if (b.length < newSize) b1[0 .. b.length] = b[]; else b1[] = b[0 .. newSize]; static if (hasMember!(From, "deallocate")) from.deallocate(b); b = b1; return true; } if (primary.owns(b)) { if (primary.reallocate(b, newSize)) return true; // Move from primary to fallback return crossAllocatorMove(primary, fallback); } if (fallback.reallocate(b, newSize)) return true; // Interesting. Move from fallback to primary. return crossAllocatorMove(fallback, primary); } This is code that moves memory across two allocators, in either directions. The original version duplicated the actual work. Then I considered a version that would define crossAllocatorMove as a private method. Then I tried the above, which just worked awesomely nice. But above all there's the property of Things Just Working (tm). Compared to e.g. the time when I designed ranges, there's a lot less toil for a lot more effect. Consider e.g. the Segregator with multiple arguments: template Segregator(Args...) if (Args.length > 3) { // Binary search private enum cutPoint = ((Args.length - 2) / 4) * 2; static if (cutPoint >= 2) { alias Segregator = .Segregator!( Args[cutPoint], .Segregator!(Args[0 .. cutPoint], Args[cutPoint + 1]), .Segregator!(Args[cutPoint + 2 .. $]) ); } else { // Favor small sizes alias Segregator = .Segregator!( Args[0], Args[1], .Segregator!(Args[2 .. $]) ); } // Linear search //alias Segregator = .Segregator!( // Args[0], Args[1], // .Segregator!(Args[2 .. $]) //); } Granted, it's not easy to get into, but can't be made much easier because it does something highly non-trivial: a left-leaning binary search through a subset of the template arguments. That's in order to find the best size class to allocate. (Linear version shown in comment, probably we should enable that with a policy.) Such code would have been much harder to get going in the past because of random bugs and limitations in the compiler and inscrutable error messages. The entire code base stands at 4000 lines, documentation and all, yet allows for a lot of allocator designs.
 - Can you create an hierarchical allocator with those?
Not sure what that means (I recall the phrase has been discussed but forgot what it means).
 - Are some operating system memory functions (below C malloc)
 used/supported/addable/useful for those allocators? (Including awareness
 of virtual memory management from the OS).
Definitely! * http://stackoverflow.com/questions/3839922/aligned-malloc-in-gcc comes to mind. * http://en.wikipedia.org/wiki/Sbrk * http://en.wikipedia.org/wiki/Mmap These would be implemented as top allocators (a la Mallocator). On top thereof, nice block allocators etc. can be easily defined. Also the final version should define a few convenience types that are pre-assembled allocators that are known to be useful: heaps, reaps, freelist batteries come to mind. Andrei
Oct 24 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 - Can you create an hierarchical allocator with those?
Not sure what that means (I recall the phrase has been discussed but forgot what it means).
I meant something like this:
 (Including awareness of virtual memory management from the OS).

 Definitely!

 * 
 http://stackoverflow.com/questions/3839922/aligned-malloc-in-gcc 
 comes to mind.

 * http://en.wikipedia.org/wiki/Sbrk

 * http://en.wikipedia.org/wiki/Mmap
By that "awareness" I meant something like what the Azul JavaVM does, here a panoramic view of similar ideas: http://web.eece.maine.edu/~jyue/papers/iccci.pdf Bye, bearophile
Oct 24 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 4:01 PM, bearophile wrote:
 Andrei Alexandrescu:

 - Can you create an hierarchical allocator with those?
Not sure what that means (I recall the phrase has been discussed but forgot what it means).
I meant something like this:
I see. These are akin to regions to some extent, just a bit more dynamic. The design doesn't explicitly address the pattern, so I expect there would be some hiccups.
 (Including awareness of virtual memory management from the OS).

 Definitely!

 * http://stackoverflow.com/questions/3839922/aligned-malloc-in-gcc
 comes to mind.

 * http://en.wikipedia.org/wiki/Sbrk

 * http://en.wikipedia.org/wiki/Mmap
By that "awareness" I meant something like what the Azul JavaVM does, here a panoramic view of similar ideas: http://web.eece.maine.edu/~jyue/papers/iccci.pdf
Once we get to types and garbage, such stuff will become more relevant. Andrei
Oct 24 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2013 12:54 PM, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha release of
 untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
I'm impressed. I looked for 3 or 4 must haves, and they were there! It covers all the cases I've written custom allocators for.
Oct 24 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 2:08 PM, Walter Bright wrote:
 On 10/24/2013 12:54 PM, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of
 untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
I'm impressed. I looked for 3 or 4 must haves, and they were there! It covers all the cases I've written custom allocators for.
<big sigh of relief> Andrei
Oct 24 2013
parent "ChrisG" <christopher.gassib gmail.com> writes:
On Thursday, 24 October 2013 at 21:17:45 UTC, Andrei Alexandrescu 
wrote:
 On 10/24/13 2:08 PM, Walter Bright wrote:
 On 10/24/2013 12:54 PM, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The 
 alpha
 release of
 untyped allocators is ready for tire-kicking and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
I'm impressed. I looked for 3 or 4 must haves, and they were there! It covers all the cases I've written custom allocators for.
<big sigh of relief> Andrei
So, is the idea to add an optional allocator template parameter to various objects in the phobos library, or is this only intended to be used standalone via the with statement? -Chris
Nov 07 2013
prev sibling next sibling parent reply Artem Tarasov <lomereiter gmail.com> writes:
The design looks great. Easy to grasp and covers almost all imaginable use
cases.

'Almost' because it would be nice to also have ScopeAllocator!N ~
FallbackAllocator!(InsituRegion!N, Mallocator) which would automatically
deallocate all heap-allocated blocks when exiting the scope (i.e. no
deallocate method, array of pointers to be freed is kept on a C heap).
Oct 24 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 2:16 PM, Artem Tarasov wrote:
 The design looks great. Easy to grasp and covers almost all imaginable
 use cases.

 'Almost' because it would be nice to also have ScopeAllocator!N ~
 FallbackAllocator!(InsituRegion!N, Mallocator) which would automatically
 deallocate all heap-allocated blocks when exiting the scope (i.e. no
 deallocate method, array of pointers to be freed is kept on a C heap).
Great point, clearly you cut to the chase :o). Yes, I hope to integrate that, and I'm a bit pained I didn't have the time to do that. The way I see that could be done is by defining a ScopedAllocator that uses internally an AffixAllocator with a doubly-linked list tracking all allocations. In the destructor, ScopedAllocator would walk the list and deallocate everything allocated since its construction. Andrei
Oct 24 2013
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 24 October 2013 at 21:20:02 UTC, Andrei Alexandrescu 
wrote:
 The way I see that could be done is by defining a 
 ScopedAllocator that uses internally an AffixAllocator with a 
 doubly-linked list tracking all allocations. In the destructor, 
 ScopedAllocator would walk the list and deallocate everything 
 allocated since its construction.
Can this be generalized as a proxy allocator which simply implements deallocateAll, on top of its parent allocator's primitives? The list can be singly-linked if random deallocation is not needed.
Oct 24 2013
prev sibling next sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
Awesome! Will Appender get an option to use a suitable allocator?
Oct 24 2013
next sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?
A dream of me, that will probably never come true, would be also something like this: ---- with (Mallocator) { int[] arr; arr ~= 42; /// will use Mallocator.it.allocate internal } ----
Oct 24 2013
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/24/13, Namespace <rswhite4 googlemail.com> wrote:
 A dream of me, that will probably never come true, would be also
 something like this:
 ----
 with (Mallocator) {
      int[] arr;
      arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----
Actually this got me thinking.. import core.memory; GC.useAllocator!Mallocator; int[] arr; arr ~= 42; This could be a useful test-bed for the allocators by using them with existing D applications and libraries that depend on the GC (most apps I'd assume).
Oct 24 2013
parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Thursday, 24 October 2013 at 21:58:36 UTC, Andrej Mitrovic 
wrote:
 On 10/24/13, Namespace <rswhite4 googlemail.com> wrote:
 A dream of me, that will probably never come true, would be 
 also
 something like this:
 ----
 with (Mallocator) {
      int[] arr;
      arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----
Actually this got me thinking.. import core.memory; GC.useAllocator!Mallocator; int[] arr; arr ~= 42; This could be a useful test-bed for the allocators by using them with existing D applications and libraries that depend on the GC (most apps I'd assume).
That would be really awesome. I would test it immediately and extensively. :)
Oct 24 2013
parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Thursday, 24 October 2013 at 22:22:53 UTC, Namespace wrote:
 On Thursday, 24 October 2013 at 21:58:36 UTC, Andrej Mitrovic 
 wrote:
 On 10/24/13, Namespace <rswhite4 googlemail.com> wrote:
 A dream of me, that will probably never come true, would be 
 also
 something like this:
 ----
 with (Mallocator) {
     int[] arr;
     arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----
Actually this got me thinking.. import core.memory; GC.useAllocator!Mallocator; int[] arr; arr ~= 42; This could be a useful test-bed for the allocators by using them with existing D applications and libraries that depend on the GC (most apps I'd assume).
That would be really awesome. I would test it immediately and extensively. :)
Another idea would be that every built-in array has a Allocator property which can be set: ---- int[] arr; arr.allocator = Mallocator; ----
Oct 24 2013
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 25 Oct 2013 00:33:40 +0200
schrieb "Namespace" <rswhite4 googlemail.com>:

 Another idea would be that every built-in array has a Allocator 
 property which can be set:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----
The type of arr.allocator would have to be CAllocator to allow runtime swapping of allocators with polymorphism. Would you'd implement it like .capacity, as a virtual property that doesn't take up space in the slice? -- Marco
Oct 25 2013
parent "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 07:50:00 UTC, Marco Leise wrote:
 Am Fri, 25 Oct 2013 00:33:40 +0200
 schrieb "Namespace" <rswhite4 googlemail.com>:

 Another idea would be that every built-in array has a 
 Allocator property which can be set:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----
The type of arr.allocator would have to be CAllocator to allow runtime swapping of allocators with polymorphism. Would you'd implement it like .capacity, as a virtual property that doesn't take up space in the slice?
Yes I would, but it's not up to me. But I would prefer this solution because it is more flexible IMO.
Oct 25 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable allocator?
A dream of me, that will probably never come true, would be also something like this: ---- with (Mallocator) { int[] arr; arr ~= 42; /// will use Mallocator.it.allocate internal } ----
Oddly enough this can be actually done. with (setAllocator!Mallocator) { ... } setAllcator returns an rvalue that changes the global allocator to the Mallocator in the constructor, and restores it to whatever it was in the destructor. Andrei
Oct 24 2013
next sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu 
wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?
A dream of me, that will probably never come true, would be also something like this: ---- with (Mallocator) { int[] arr; arr ~= 42; /// will use Mallocator.it.allocate internal } ----
Oddly enough this can be actually done. with (setAllocator!Mallocator) { ... } setAllcator returns an rvalue that changes the global allocator to the Mallocator in the constructor, and restores it to whatever it was in the destructor. Andrei
Are you saying that this code: ---- with (setAllocator!Mallocator) { int[] arr; arr ~= 42; [1] } ---- use the mallocator for [1]? So no GC memory is needed? Another examaple: ---- with (setAllocator!ScopeAllocator) { int[] arr; arr ~= 42; [1] } ---- Did [1] use ScopeAllocator for memory allocation? And is the memory of arr automatically collected at the end of the scope with ScopeAllocator?
Oct 25 2013
parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Friday, 25 October 2013 at 07:19:48 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu 
 wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?
A dream of me, that will probably never come true, would be also something like this: ---- with (Mallocator) { int[] arr; arr ~= 42; /// will use Mallocator.it.allocate internal } ----
Oddly enough this can be actually done. with (setAllocator!Mallocator) { ... } setAllcator returns an rvalue that changes the global allocator to the Mallocator in the constructor, and restores it to whatever it was in the destructor. Andrei
Are you saying that this code: ---- with (setAllocator!Mallocator) { int[] arr; arr ~= 42; [1] } ---- use the mallocator for [1]? So no GC memory is needed? Another examaple: ---- with (setAllocator!ScopeAllocator) { int[] arr; arr ~= 42; [1] } ---- Did [1] use ScopeAllocator for memory allocation? And is the memory of arr automatically collected at the end of the scope with ScopeAllocator?
This is impossible because with() has already another meaning. However nothing stops from inventing another name for this. Compiler can insert calls to druntime as well to other context dependent functions. After allocator design is defined we can move on to step 2, implementing this idea.
Oct 25 2013
next sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
 Are you saying that this code:
 ----
 with (setAllocator!Mallocator) {
    int[] arr;
    arr ~= 42; [1]
 }
 ----
 use the mallocator for [1]? So no GC memory is needed?

 Another examaple:
 ----
 with (setAllocator!ScopeAllocator) {
    int[] arr;
    arr ~= 42; [1]
 }
 ----
 Did [1] use ScopeAllocator for memory allocation? And is the 
 memory of arr automatically collected at the end of the scope 
 with ScopeAllocator?
This is impossible because with() has already another meaning. However nothing stops from inventing another name for this. Compiler can insert calls to druntime as well to other context dependent functions. After allocator design is defined we can move on to step 2, implementing this idea.
You mena something like this? ---- use(Mallocator) { int[] arr; arr ~= 42; } ---- Or did I understand you wrong? Whats about the virtual property idea, that every array has internal an allocator? Wouldn't it be easier to implement such a thing?
Oct 25 2013
parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Friday, 25 October 2013 at 08:13:21 UTC, Namespace wrote:
 Are you saying that this code:
 ----
 with (setAllocator!Mallocator) {
   int[] arr;
   arr ~= 42; [1]
 }
 ----
 use the mallocator for [1]? So no GC memory is needed?

 Another examaple:
 ----
 with (setAllocator!ScopeAllocator) {
   int[] arr;
   arr ~= 42; [1]
 }
 ----
 Did [1] use ScopeAllocator for memory allocation? And is the 
 memory of arr automatically collected at the end of the scope 
 with ScopeAllocator?
This is impossible because with() has already another meaning. However nothing stops from inventing another name for this. Compiler can insert calls to druntime as well to other context dependent functions. After allocator design is defined we can move on to step 2, implementing this idea.
You mena something like this? ---- use(Mallocator) { int[] arr; arr ~= 42; } ---- Or did I understand you wrong?
It depends on how Mallocator is related to integer array (which again boils down to allocator design). If it is appropriate, then yes.
 Whats about the virtual property idea, that every array has 
 internal an allocator? Wouldn't it be easier to implement such 
 a thing?
Please provide example.
Oct 25 2013
parent reply "Namespace" <rswhite4 googlemail.com> writes:
 You mena something like this?
 ----
 use(Mallocator) {
    int[] arr;
    arr ~= 42;
 }
 ----
 Or did I understand you wrong?
It depends on how Mallocator is related to integer array (which again boils down to allocator design). If it is appropriate, then yes.
 Whats about the virtual property idea, that every array has 
 internal an allocator? Wouldn't it be easier to implement such 
 a thing?
Please provide example.
Something like that: http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=3#post-pfoxyfzyjxqcqwnvgnpi:40forum.dlang.org Every array has an internal allocator property which can be reset: ---- int[] arr; arr.allocator = Mallocator; ---- or ---- int[] arr; arr.useAllocator(Mallocator); ---- But maybe a design without some alias notation would be more preferable: ---- { ScopeAllocator m; int[] arr; arr.useAllocator(m); arr ~= 42; /// Use m.allocate } /// end of scope: ScopeAllocator collects all remaining memory. ---- And: ---- int[] arr; assert(arr is null); { ScopeAllocator m; arr.useAllocator(m); arr ~= 42; /// Use m.allocate } /// end of scope: ScopeAllocator collects all remaining memory. assert(arr is null); ----
Oct 25 2013
parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Friday, 25 October 2013 at 08:27:52 UTC, Namespace wrote:
 Something like that: 
 http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=3#post-pfoxyfzyjxqcqwnvgnpi:40forum.dlang.org

 Every array has an internal allocator property which can be 
 reset:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----

 or

 ----
 int[] arr;
 arr.useAllocator(Mallocator);
 ----
That's doable.
 But maybe a design without some alias notation would be more 
 preferable:
 ----
 {
     ScopeAllocator m;
     int[] arr;
     arr.useAllocator(m);

     arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 ----

 And:
 ----
 int[] arr;
 assert(arr is null);
 {
     ScopeAllocator m;
     arr.useAllocator(m);

     arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 assert(arr is null);
 ----
That's also doable. TypeInfo will be bloated more and there would be cost of some sort of scope exit, and, ideally, a check that reference does not escape, but this is doable.
Oct 25 2013
next sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 08:50:23 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 08:27:52 UTC, Namespace wrote:
 Something like that: 
 http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=3#post-pfoxyfzyjxqcqwnvgnpi:40forum.dlang.org

 Every array has an internal allocator property which can be 
 reset:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----

 or

 ----
 int[] arr;
 arr.useAllocator(Mallocator);
 ----
That's doable.
 But maybe a design without some alias notation would be more 
 preferable:
 ----
 {
    ScopeAllocator m;
    int[] arr;
    arr.useAllocator(m);

    arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 ----

 And:
 ----
 int[] arr;
 assert(arr is null);
 {
    ScopeAllocator m;
    arr.useAllocator(m);

    arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 assert(arr is null);
 ----
That's also doable. TypeInfo will be bloated more and there would be cost of some sort of scope exit, and, ideally, a check that reference does not escape, but this is doable.
A dream could come true.
Oct 25 2013
parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Friday, 25 October 2013 at 08:53:00 UTC, Namespace wrote:
 A dream could come true.
It depends on how Andrei design allocators, Walter's willingness to accept changes and developers to implement it.
Oct 25 2013
parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 09:26:46 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 08:53:00 UTC, Namespace wrote:
 A dream could come true.
It depends on how Andrei design allocators, Walter's willingness to accept changes and developers to implement it.
Therefore the 'could'. But if this would be implemented, I'm sure that this would give D good advantages to be a real opponent for C++. We would have then the possibility to manage our memory by ourself. One of D's promises is, that the GC can be disabled. Yes, it can, but then we have many many things which do not work. For example built-in arrays. With the ability of allocators the promise could come true.
Oct 25 2013
next sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be disabled. 
 Yes, it can, but then we have many many things which do not 
 work. For example built-in arrays.
Not only arrays, but classes, throwables, scope exits, new operator, nested structs, etc.
Oct 25 2013
parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be disabled. 
 Yes, it can, but then we have many many things which do not 
 work. For example built-in arrays.
Not only arrays, but classes, throwables, scope exits, new operator, nested structs, etc.
Thats right. But I often use temporary arrays, but I still don't like them because they are always consume so much GC memory. But with allocators that would end. Let us hope that Walter has the right intention and that Andrei design the allocators for this purpose.
Oct 25 2013
parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be disabled. 
 Yes, it can, but then we have many many things which do not 
 work. For example built-in arrays.
Not only arrays, but classes, throwables, scope exits, new operator, nested structs, etc.
Thats right. But I often use temporary arrays, but I still don't like them because they are always consume so much GC memory. But with allocators that would end. Let us hope that Walter has the right intention and that Andrei design the allocators for this purpose.
Why does it have to be the opaque druntime dynamic array? Why can't you use the hypothetical (planned, rather) std.container.Array that supports custom allocators, or a type of your own design? Garbage collected memory management is the only memory management paradigm that supports the infinite lifetime model (GC includes automatic reference counting). Swapping it out for a different memory management technique while libraries or parts of the program still rely on infinite lifetime for parts such as the slice concatenation operators and closures is disastrous. The best strategy for avoiding the global GC is to not use the language features associated with infinite lifetime, which includes slice concatenation and the `new` operator. For implicit allocations like closures, we don't have much of a choice, so here the capability is useful, but for slice operators? Just use a different type! Currently the `new` operator has capabilities that function calls cannot replicate - namely non-static nested class construction and construction of classes with non-public constructors - but these issues have to be solved anyway to support allocators in general.
Oct 25 2013
parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 11:21:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be 
 disabled. Yes, it can, but then we have many many things 
 which do not work. For example built-in arrays.
Not only arrays, but classes, throwables, scope exits, new operator, nested structs, etc.
Thats right. But I often use temporary arrays, but I still don't like them because they are always consume so much GC memory. But with allocators that would end. Let us hope that Walter has the right intention and that Andrei design the allocators for this purpose.
Why does it have to be the opaque druntime dynamic array? Why can't you use the hypothetical (planned, rather) std.container.Array that supports custom allocators, or a type of your own design?
Because Array!int looks a lot more ugly than such a nice thing as int[]. And if it is possible to change the allocator for some arrays, why shouldn't we implement it? The default allocator would stay the GC allocator. So if you don't want to swap the allocator of your arrays, don't do it.
Oct 25 2013
next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Friday, 25 October 2013 at 12:07:30 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 11:21:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be 
 disabled. Yes, it can, but then we have many many things 
 which do not work. For example built-in arrays.
Not only arrays, but classes, throwables, scope exits, new operator, nested structs, etc.
Thats right. But I often use temporary arrays, but I still don't like them because they are always consume so much GC memory. But with allocators that would end. Let us hope that Walter has the right intention and that Andrei design the allocators for this purpose.
Why does it have to be the opaque druntime dynamic array? Why can't you use the hypothetical (planned, rather) std.container.Array that supports custom allocators, or a type of your own design?
Because Array!int looks a lot more ugly than such a nice thing as int[].
If the template syntax is too ugly then we've really failed at designing an extensible development platform. Even so, with type inference and aliases, the need to write down involved names is all but eliminated (not that I think Array!int is a particularly involved name). Not everything belongs in the core language. Conflating slices with garbage collected dynamic arrays is a mistake we have to live with, but let's not make the situation even more complicated.
 And if it is possible to change the allocator for some arrays, 
 why shouldn't we implement it?
Because it has a significant cost.
 The default allocator would stay the GC allocator. So if you 
 don't want to swap the allocator of your arrays, don't do it.
That's fine as long as you only use the array locally and don't pass it to any code that depends on infinite lifetime semantics. It also enables circumvention of SafeD unless some language rules are changed.
Oct 25 2013
parent "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 14:09:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 12:07:30 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 11:21:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin 
 wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be 
 disabled. Yes, it can, but then we have many many things 
 which do not work. For example built-in arrays.
Not only arrays, but classes, throwables, scope exits, new operator, nested structs, etc.
Thats right. But I often use temporary arrays, but I still don't like them because they are always consume so much GC memory. But with allocators that would end. Let us hope that Walter has the right intention and that Andrei design the allocators for this purpose.
Why does it have to be the opaque druntime dynamic array? Why can't you use the hypothetical (planned, rather) std.container.Array that supports custom allocators, or a type of your own design?
Because Array!int looks a lot more ugly than such a nice thing as int[].
If the template syntax is too ugly then we've really failed at designing an extensible development platform. Even so, with type inference and aliases, the need to write down involved names is all but eliminated (not that I think Array!int is a particularly involved name). Not everything belongs in the core language. Conflating slices with garbage collected dynamic arrays is a mistake we have to live with, but let's not make the situation even more complicated.
I don't get your problem.
 And if it is possible to change the allocator for some arrays, 
 why shouldn't we implement it?
Because it has a significant cost.
Which cost?
 The default allocator would stay the GC allocator. So if you 
 don't want to swap the allocator of your arrays, don't do it.
That's fine as long as you only use the array locally and don't pass it to any code that depends on infinite lifetime semantics. It also enables circumvention of SafeD unless some language rules are changed.
The language cannot protect you from every mistake you can do. You should know what you do. If you don't, don't use such a feature. It's very simple.
Oct 25 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/25/13 5:07 AM, Namespace wrote:
 On Friday, 25 October 2013 at 11:21:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by ourself.
 One of D's promises is, that the GC can be disabled. Yes, it can,
 but then we have many many things which do not work. For example
 built-in arrays.
Not only arrays, but classes, throwables, scope exits, new operator, nested structs, etc.
Thats right. But I often use temporary arrays, but I still don't like them because they are always consume so much GC memory. But with allocators that would end. Let us hope that Walter has the right intention and that Andrei design the allocators for this purpose.
Why does it have to be the opaque druntime dynamic array? Why can't you use the hypothetical (planned, rather) std.container.Array that supports custom allocators, or a type of your own design?
Because Array!int looks a lot more ugly than such a nice thing as int[]. And if it is possible to change the allocator for some arrays, why shouldn't we implement it? The default allocator would stay the GC allocator. So if you don't want to swap the allocator of your arrays, don't do it.
Maintaining per-array-object allocators could be quite expensive. Two other possibilities are keeping one allocators for all array of a given type and of course keeping a global allocator for all arrays. Andrei
Oct 25 2013
parent "Namespace" <rswhite4 googlemail.com> writes:
 Because Array!int looks a lot more ugly than such a nice thing 
 as int[].
 And if it is possible to change the allocator for some arrays, 
 why
 shouldn't we implement it?
 The default allocator would stay the GC allocator. So if you 
 don't want
 to swap the allocator of your arrays, don't do it.
Maintaining per-array-object allocators could be quite expensive. Two other possibilities are keeping one allocators for all array of a given type and of course keeping a global allocator for all arrays. Andrei
Sounds also nice. As long as you can customize the way built-in arrays allocate, I'm fine with that. :)
Oct 25 2013
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 25/10/13 11:37, Namespace wrote:
 We would have then the possibility to manage our memory by ourself. One of D's
 promises is, that the GC can be disabled. Yes, it can, but then we have many
 many things which do not work. For example built-in arrays. With the ability of
 allocators the promise could come true.
That's something I'd really like to know more about. My D code almost invariably works with the "natural" way to handle memory in D, which is to use "new" where needed, plus stuff like array appending ~ and alterations to array lengths, with all allocations handled behind the scenes by the GC. I've always felt bad about the fact that this therefore imposes use of the GC on anyone who uses my code. It would be great if one could just write idiomatic D code and know that others using it could dictate different memory-management strategies and have them "just work".
Oct 26 2013
parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Saturday, 26 October 2013 at 15:42:48 UTC, Joseph Rushton 
Wakeling wrote:
 On 25/10/13 11:37, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's
 promises is, that the GC can be disabled. Yes, it can, but 
 then we have many
 many things which do not work. For example built-in arrays. 
 With the ability of
 allocators the promise could come true.
That's something I'd really like to know more about. My D code almost invariably works with the "natural" way to handle memory in D, which is to use "new" where needed, plus stuff like array appending ~ and alterations to array lengths, with all allocations handled behind the scenes by the GC. I've always felt bad about the fact that this therefore imposes use of the GC on anyone who uses my code. It would be great if one could just write idiomatic D code and know that others using it could dictate different memory-management strategies and have them "just work".
Read on from here to find out more: http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=5#post-uqolhuqqygquxnaxahkz:40forum.dlang.org :)
Oct 26 2013
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 26/10/13 17:46, Namespace wrote:
 Read on from here to find out more:
 http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=5#post-uqolhuqqygquxnaxahkz:40forum.dlang.org
I already did, and if I understand right, things are looking good ... but I wanted to be sure whether or not I understand :-)
Oct 26 2013
prev sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 08:50:23 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 08:27:52 UTC, Namespace wrote:
 Something like that: 
 http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=3#post-pfoxyfzyjxqcqwnvgnpi:40forum.dlang.org

 Every array has an internal allocator property which can be 
 reset:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----

 or

 ----
 int[] arr;
 arr.useAllocator(Mallocator);
 ----
That's doable.
 But maybe a design without some alias notation would be more 
 preferable:
 ----
 {
    ScopeAllocator m;
    int[] arr;
    arr.useAllocator(m);

    arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 ----

 And:
 ----
 int[] arr;
 assert(arr is null);
 {
    ScopeAllocator m;
    arr.useAllocator(m);

    arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 assert(arr is null);
 ----
That's also doable. TypeInfo will be bloated more and there would be cost of some sort of scope exit, and, ideally, a check that reference does not escape, but this is doable.
Why the cost of scope(exit)? If ScopeAllocator m get out of scope, the DTor is called and ideally the collect method is called. That's all. Or am I wrong?
Oct 25 2013
parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Friday, 25 October 2013 at 09:13:45 UTC, Namespace wrote:
 But maybe a design without some alias notation would be more 
 preferable:
 ----
 {
   ScopeAllocator m;
   int[] arr;
   arr.useAllocator(m);

   arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 ----

 And:
 ----
 int[] arr;
 assert(arr is null);
 {
   ScopeAllocator m;
   arr.useAllocator(m);

   arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 assert(arr is null);
 ----
That's also doable. TypeInfo will be bloated more and there would be cost of some sort of scope exit, and, ideally, a check that reference does not escape, but this is doable.
Why the cost of scope(exit)? If ScopeAllocator m get out of scope, the DTor is called and ideally the collect method is called. That's all. Or am I wrong?
Probably because there may be exception thrown between appending to arr and end of block. In this particular case compiler may be smart enough to optimize it away, but dmd optimization capabilities and nothrow is a separate story.
Oct 25 2013
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 25 October 2013 at 08:09:30 UTC, Maxim Fomin wrote:
 This is impossible because with() has already another meaning.
I believe Andrei meant using the existing "with" statement as it currently is defined in the language. The function setAllocator would set the given allocator, and return a structure with a destructor. The destructor should be called at the end of the "with" block, and it will set the current allocator back to its old value. However, it appears that currently this usage of "with" is buggy: http://d.puremagic.com/issues/show_bug.cgi?id=11351
Oct 25 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
     ...
 }

 setAllcator returns an rvalue that changes the global allocator to the
 Mallocator in the constructor, and restores it to whatever it was in the
 destructor.
Wouldn't this be very unsafe? Say you call another function inside the with-statement and that function assumes the standard GC for allocating memory. -- /Jacob Carlborg
Oct 25 2013
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
25-Oct-2013 16:52, Jacob Carlborg пишет:
 On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
     ...
 }

 setAllcator returns an rvalue that changes the global allocator to the
 Mallocator in the constructor, and restores it to whatever it was in the
 destructor.
Wouldn't this be very unsafe? Say you call another function inside the with-statement and that function assumes the standard GC for allocating memory.
Very true. To put it simply it's a disastrous idea that sadly is too easy to be ignored. IMHO we'd better start with containers and refitting Phobos from built-in AA/arrays to user-defined containers. One interesting way there is to accept/adopt containers as OutputRange. -- Dmitry Olshansky
Oct 25 2013
next sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky 
wrote:
 25-Oct-2013 16:52, Jacob Carlborg пишет:
 On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
    ...
 }

 setAllcator returns an rvalue that changes the global 
 allocator to the
 Mallocator in the constructor, and restores it to whatever it 
 was in the
 destructor.
Wouldn't this be very unsafe? Say you call another function inside the with-statement and that function assumes the standard GC for allocating memory.
Very true. To put it simply it's a disastrous idea that sadly is too easy to be ignored. IMHO we'd better start with containers and refitting Phobos from built-in AA/arrays to user-defined containers. One interesting way there is to accept/adopt containers as OutputRange.
Did you mean to get rid of built-in arrays / kill int[] and replace it with Array!T?
Oct 25 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
25-Oct-2013 22:41, Namespace пишет:
 On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky wrote:
 25-Oct-2013 16:52, Jacob Carlborg пишет:
 On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
    ...
 }

 setAllcator returns an rvalue that changes the global allocator to the
 Mallocator in the constructor, and restores it to whatever it was in
 the
 destructor.
Wouldn't this be very unsafe? Say you call another function inside the with-statement and that function assumes the standard GC for allocating memory.
Very true. To put it simply it's a disastrous idea that sadly is too easy to be ignored. IMHO we'd better start with containers and refitting Phobos from built-in AA/arrays to user-defined containers. One interesting way there is to accept/adopt containers as OutputRange.
Did you mean to get rid of built-in arrays / kill int[] and replace it with Array!T?
Hm, arrays? Kill? No, they are incredibly nice for prototyping + they are very useful even as just slices. What I mean is to make it easy to use Phobos stuff with other containers in place of built-ins, basically no hard-codding behind the scenes. Typical offender is Appender - it's rigid, has dangerous API and doesn't support building anything but T[]. For this particular case see: http://d.puremagic.com/issues/show_bug.cgi?id=11138 Another one: It's trivially expendable to any other Array-like type, yet ATM it's hardwired. O.T. I'd gladly kill built-in AA though just to save people a lot of time spent on debugging that crap. More precisely I'd keep AA _literals_ and give the user the means to construct any type of Key-->Value store out of it. It's too late probably. -- Dmitry Olshansky
Oct 25 2013
parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 25 October 2013 at 19:03:14 UTC, Dmitry Olshansky 
wrote:
 25-Oct-2013 22:41, Namespace пишет:
 On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky 
 wrote:
 25-Oct-2013 16:52, Jacob Carlborg пишет:
 On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
   ...
 }

 setAllcator returns an rvalue that changes the global 
 allocator to the
 Mallocator in the constructor, and restores it to whatever 
 it was in
 the
 destructor.
Wouldn't this be very unsafe? Say you call another function inside the with-statement and that function assumes the standard GC for allocating memory.
Very true. To put it simply it's a disastrous idea that sadly is too easy to be ignored. IMHO we'd better start with containers and refitting Phobos from built-in AA/arrays to user-defined containers. One interesting way there is to accept/adopt containers as OutputRange.
Did you mean to get rid of built-in arrays / kill int[] and replace it with Array!T?
Hm, arrays? Kill? No, they are incredibly nice for prototyping + they are very useful even as just slices. What I mean is to make it easy to use Phobos stuff with other containers in place of built-ins, basically no hard-codding behind the scenes. Typical offender is Appender - it's rigid, has dangerous API and doesn't support building anything but T[]. For this particular case see: http://d.puremagic.com/issues/show_bug.cgi?id=11138 Another one: It's trivially expendable to any other Array-like type, yet ATM it's hardwired. O.T. I'd gladly kill built-in AA though just to save people a lot of time spent on debugging that crap. More precisely I'd keep AA _literals_ and give the user the means to construct any type of Key-->Value store out of it. It's too late probably.
With which syntax? As far as e.g. int[string] (and not something ugly as Map!(string, int)) would stay what would be the problem to change the backend?
Oct 25 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
25-Oct-2013 23:26, Namespace пишет:
 On Friday, 25 October 2013 at 19:03:14 UTC, Dmitry Olshansky wrote:
 O.T. I'd gladly kill built-in AA though just to save people a lot of
 time spent on debugging that crap. More precisely I'd keep AA
 _literals_ and give the user the means to construct any type of
 Key-->Value store out of it. It's too late probably.
With which syntax? As far as e.g. int[string] (and not something ugly as Map!(string, int)) would stay what would be the problem to change the backend?
Map!(string,int) is fine actually as I'm no syntax fan anyway. It's not the problem with "backend" as much as with its state and the interface it imposes: a) It's stuck in between compiler intrinsic and library artifact. Being neither it has weaknesses of both (well that's fixable). b) Interface... I don't even know where to start. But first of all: it's a hash table and can't be anything else - see the rehash function. More interestingly it escapes pointers ('in' operator) that must stay valid until another insert. That's a tough constraint on hash table (HT) implementation. Being HT it doesn't expose nor allows to change the load-factor. c) Built-in HTs are one size fits all. Needless to say - it doesn't always fit and in particular there are many variations that trade insert/remove and/or space-efficiency for faster lookup and vise-versa. Also to stay on topic - memory management / deterministic cleanup? Trivial in UDT but not in built-ins. d) AA literals are of no use the moment one stops using built-in HTs least one constructs temporary built-in HTs on GC heap. In my opinion if that's left as is the AA literal feature doesn't quite pull its weight. All in all built-in AA are aging badly since D1 days. -- Dmitry Olshansky
Oct 25 2013
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 25/10/13 20:41, Namespace wrote:
 Did you mean to get rid of built-in arrays / kill int[] and replace it with
 Array!T?
Array!T is problematic as things stand -- e.g. you can't foreach over one. So, forgetting syntax preferences, there needs to be some work on containers before they can "just work" like the builtins. If it's possible, I'd rather see the converse -- that code that assumes the GC will "just work" with other allocation strategies, so one can use the builtins without worrying. But am I being naively hopeful in seeking that? :-)
Oct 26 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
26-Oct-2013 20:01, Joseph Rushton Wakeling пишет:
 On 25/10/13 20:41, Namespace wrote:
 Did you mean to get rid of built-in arrays / kill int[] and replace it
 with
 Array!T?
Array!T is problematic as things stand -- e.g. you can't foreach over one.
Sure you can. Try it and rejoice: void main() { import std.container, std.stdio; Array!int a = make!(Array!int)(1,2,3,4); //the rule is: if a can be sliced then slice it and use that slice foreach(v; a) { writeln(v); } }
 So, forgetting syntax preferences, there needs to be some work on
 containers before they can "just work" like the builtins.
Depends on what's required for you to consider it "just works". But in general they can't be complete replica of built-ins for many reasons, built-ins being designed with GC in mind is one. Other problems include having no user-land analog of implicit tail-const of arrays.
 If it's possible, I'd rather see the converse -- that code that assumes
 the GC will "just work" with other allocation strategies, so one can use
 the builtins without worrying.
Only if you are switching to from one GC kind to another. There is no way out of automatic memory management.
 But am I being naively hopeful in
 seeking that? :-)
Yes. -- Dmitry Olshansky
Oct 26 2013
next sibling parent "Tourist" <gravatar gravatar.com> writes:
On Saturday, 26 October 2013 at 16:10:46 UTC, Dmitry Olshansky 
wrote:
 If it's possible, I'd rather see the converse -- that code 
 that assumes
 the GC will "just work" with other allocation strategies, so 
 one can use
 the builtins without worrying.
Only if you are switching to from one GC kind to another. There is no way out of automatic memory management.
Waiting for ARC :)
Oct 26 2013
prev sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Saturday, 26 October 2013 at 16:10:46 UTC, Dmitry Olshansky 
wrote:
 26-Oct-2013 20:01, Joseph Rushton Wakeling пишет:
 On 25/10/13 20:41, Namespace wrote:
 Did you mean to get rid of built-in arrays / kill int[] and 
 replace it
 with
 Array!T?
Array!T is problematic as things stand -- e.g. you can't foreach over one.
Sure you can. Try it and rejoice: void main() { import std.container, std.stdio; Array!int a = make!(Array!int)(1,2,3,4); //the rule is: if a can be sliced then slice it and use that slice foreach(v; a) { writeln(v); } }
The fact that foreach with a ref item or foreach with an index doesn't work makes using them a lot more of a hassle than built-in arrays though. void main() { import std.container, std.stdio; Array!int a = make!(Array!int)(1,2,3,4); foreach(ref v; a) { v *= v; // no effect on a } writeln(a[]); // [1, 2, 3, 4] foreach(v, i; a) // Error: cannot infer argument types { a[i] *= v; } }
Oct 26 2013
prev sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Saturday, 26 October 2013 at 16:01:18 UTC, Joseph Rushton 
Wakeling wrote:
 On 25/10/13 20:41, Namespace wrote:
 Did you mean to get rid of built-in arrays / kill int[] and 
 replace it with
 Array!T?
Array!T is problematic as things stand -- e.g. you can't foreach over one. So, forgetting syntax preferences, there needs to be some work on containers before they can "just work" like the builtins. If it's possible, I'd rather see the converse -- that code that assumes the GC will "just work" with other allocation strategies, so one can use the builtins without worrying. But am I being naively hopeful in seeking that? :-)
I would never vote to replace the built-in arrays with something ugly as Array!T. If D would switch to Array!T and Map!(T, U) it would be the same hell as with C++. But I hope allocators enable the possibility that built-in arrays could use other memory mangaement besides the GC. That would be awesome.
Oct 26 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky 
wrote:
 Very true. To put it simply it's a disastrous idea that sadly 
 is too easy to be ignored.

 IMHO we'd better start with containers and refitting Phobos 
 from built-in AA/arrays to user-defined containers. One 
 interesting way there is to accept/adopt containers as 
 OutputRange.
Typed allocator on top of this should be defined before this is discussed. Right it doesn't make a lot of sense.
Oct 25 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
25-Oct-2013 23:50, deadalnix пишет:
 On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky wrote:
 Very true. To put it simply it's a disastrous idea that sadly is too
 easy to be ignored.

 IMHO we'd better start with containers and refitting Phobos from
 built-in AA/arrays to user-defined containers. One interesting way
 there is to accept/adopt containers as OutputRange.
Typed allocator on top of this should be defined before this is discussed. Right it doesn't make a lot of sense.
Yeah, that's another step before that. It's just I'm strongly opposed to "the strap this magical new allocator somewhere so that's it's global for everything, right now" movement and I had to plot the alternative course ;) -- Dmitry Olshansky
Oct 25 2013
prev sibling next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu 
wrote:
 with (setAllocator!Mallocator)
That'd be a mistake except perhaps in two circumstances: 1) if setAllocator was system with warnings attached. or 2) if the content inside the with block was all a pure function. function cannot escape any reference, since it isn't allowed to access outside mutable data at all. When the pure function returns, the only possible places one of the other allocated values could be are in one of its parameters - which are set right here on the same (or the immediately following) line as the call to setAllocator, so it is no surprise, or the return value, and ditto. Otherwise, by the pure rules, they must all be local and thus freeing them is fine. I'm not quite sold on replacing the global allocator. I'd generally prefer to use a container type so it is clear who the owner is, and then if anything, just prohibit slice concat (and preferably, storing them too, scope storage class!!!!111!one) so you don't accidentally reallocate without informing the owner. But if we did do it, signatures like system AllocatorResetStruct setAllocator(Allocator)() {} and safe void setAllocator(Allocator)(pure safe void function() code) {} just might work out.
Oct 25 2013
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-10-25 15:14:38 +0000, "Adam D. Ruppe" <destructionator gmail.com> said:

  safe void setAllocator(Allocator)(pure  safe void function() code) {}
 
 just might work out.
I wonder what kind of things can be done in a pure function that takes no parameter and returns void. Not much I guess. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Oct 25 2013
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 10/25/2013 08:50 PM, Michel Fortin wrote:
 On 2013-10-25 15:14:38 +0000, "Adam D. Ruppe"
 <destructionator gmail.com> said:

  safe void setAllocator(Allocator)(pure  safe void function() code) {}

 just might work out.
I wonder what kind of things can be done in a pure function that takes no parameter and returns void. Not much I guess.
It can e.g. allocate data and embed it in an exception. (Which is also why the above function is still unsafe.)
Oct 25 2013
prev sibling parent reply "bitwise" <bitwise.pvt gmail.com> writes:
On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu 
wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?
A dream of me, that will probably never come true, would be also something like this: ---- with (Mallocator) { int[] arr; arr ~= 42; /// will use Mallocator.it.allocate internal } ----
Oddly enough this can be actually done. with (setAllocator!Mallocator) { ... } setAllcator returns an rvalue that changes the global allocator to the Mallocator in the constructor, and restores it to whatever it was in the destructor. Andrei
Doesn't this race because the allocator instance is shared? I couldn't find 'setAllocator' in the source code. Bit
Sep 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/11/2015 06:32 PM, bitwise wrote:
 On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable allocator?
A dream of me, that will probably never come true, would be also something like this: ---- with (Mallocator) { int[] arr; arr ~= 42; /// will use Mallocator.it.allocate internal } ----
Oddly enough this can be actually done. with (setAllocator!Mallocator) { ... } setAllcator returns an rvalue that changes the global allocator to the Mallocator in the constructor, and restores it to whatever it was in the destructor. Andrei
Doesn't this race because the allocator instance is shared? I couldn't find 'setAllocator' in the source code.
Yah, that was a rumination, not something already implemented. -- Andrei
Sep 11 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
On Friday, 11 September 2015 at 23:13:16 UTC, Andrei Alexandrescu 
wrote:
 On 09/11/2015 06:32 PM, bitwise wrote:
 On Friday, 25 October 2013 at 00:00:36 UTC, Andrei 
 Alexandrescu wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace 
 wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?
A dream of me, that will probably never come true, would be also something like this: ---- with (Mallocator) { int[] arr; arr ~= 42; /// will use Mallocator.it.allocate internal } ----
Oddly enough this can be actually done. with (setAllocator!Mallocator) { ... } setAllcator returns an rvalue that changes the global allocator to the Mallocator in the constructor, and restores it to whatever it was in the destructor. Andrei
Doesn't this race because the allocator instance is shared? I couldn't find 'setAllocator' in the source code.
Yah, that was a rumination, not something already implemented. -- Andrei
Ok, thanks. I thought that may be the case. One more question: I'd like to integrate these into the containers I'm building, but I'm not clear about how to determine if GC.addRange() should be called. I've thought through this, and I'm pretty sure that I can call GC.addRange and GC.removeRange indiscriminately on any memory my container gets from an allocator, as long as the container finds that (hasIndirections!T == true). My reasoning is that if an allocator calls GC.addRange on it's own memory, then it should also reinitialize that memory when it gets deallocated, which would include calling GC.addRange again if it had to. Also, calling GC.addRange or GC.removeRange on GC allocated memory should have no effect. Does this sound right, or am I crazy? Bit
Sep 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/11/2015 07:46 PM, bitwise wrote:
 On Friday, 11 September 2015 at 23:13:16 UTC, Andrei Alexandrescu wrote:
 On 09/11/2015 06:32 PM, bitwise wrote:
 On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable allocator?
A dream of me, that will probably never come true, would be also something like this: ---- with (Mallocator) { int[] arr; arr ~= 42; /// will use Mallocator.it.allocate internal } ----
Oddly enough this can be actually done. with (setAllocator!Mallocator) { ... } setAllcator returns an rvalue that changes the global allocator to the Mallocator in the constructor, and restores it to whatever it was in the destructor. Andrei
Doesn't this race because the allocator instance is shared? I couldn't find 'setAllocator' in the source code.
Yah, that was a rumination, not something already implemented. -- Andrei
Ok, thanks. I thought that may be the case. One more question: I'd like to integrate these into the containers I'm building, but I'm not clear about how to determine if GC.addRange() should be called. I've thought through this, and I'm pretty sure that I can call GC.addRange and GC.removeRange indiscriminately on any memory my container gets from an allocator, as long as the container finds that (hasIndirections!T == true). My reasoning is that if an allocator calls GC.addRange on it's own memory, then it should also reinitialize that memory when it gets deallocated, which would include calling GC.addRange again if it had to. Also, calling GC.addRange or GC.removeRange on GC allocated memory should have no effect. Does this sound right, or am I crazy?
Say you have a container that uses its own allocator inside, yet offers the user to store objects with indirections that use the GC. Then indeed the container would need to call addRange and removeRange on its own internal structures. If, on the contrary, the container imposes that its own held objects use the container's allocator as well (a rare but not implausible design), it wouldn't need to do that. Andrei
Sep 11 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
On Saturday, 12 September 2015 at 00:31:27 UTC, Andrei 
Alexandrescu wrote:
 On 09/11/2015 07:46 PM, bitwise wrote:
 [...]
Say you have a container that uses its own allocator inside, yet offers the user to store objects with indirections that use the GC. Then indeed the container would need to call addRange and removeRange on its own internal structures.
This sounds like the default setup I'll be using. The containers will use a Mallocator by default, so I will have to add the ranges when the contained type is found to have indirections.
 If, on the contrary, the container imposes that its own held 
 objects use the container's allocator as well (a rare but not 
 implausible design), it wouldn't need to do that.
I don't understand exactly what you mean, but here is a more verbose version of my concern: Lets say someone's allocator calls GC.addRange on the memory it allocates before a container gets it. The container would call GC.addRange when it gets it, but then call GC.removeRange before calling allocator.deallocate(). The allocator which think's the block has already been registered with the GC could possibly reuse it, thinking it's registered with the GC already....but that allocator may pass the memory to a container which chooses not to call GC.addRange, because it thinks it's using an allocator that does that for it. That second container may now store types with indirections, which would be unreachable, and get freed, and......chaos. I suppose that as a general rule, a container could always add ranges it gets from an allocator to the GC, which would prevent the above problem...but only for containers that abide by the rule. It sounds like you're saying that although it may not be needed sometimes, that no harm would be done. I think a better solution though would be for the GC to have something like GC.hasRange(). The container could check the state of the memory it receives and take appropriate action. I don't see anything in the current GC docs which would allow you to tell if a range had already been added to the GC. Bit
Sep 11 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
On Saturday, 12 September 2015 at 06:30:42 UTC, bitwise wrote:
 On Saturday, 12 September 2015 at 00:31:27 UTC, Andrei 
 Alexandrescu wrote:
 [...]
This sounds like the default setup I'll be using. The containers will use a Mallocator by default, so I will have to add the ranges when the contained type is found to have indirections. [...]
Alternatively, GC.addRange() could return a value indicating whether or not the range had actually been added(for the first time) and should be removed. Bit
Sep 11 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Saturday, 12 September 2015 at 06:45:16 UTC, bitwise wrote:
 [...]

 Alternatively, GC.addRange() could return a value indicating 
 whether or not the range had actually been added(for the first 
 time) and should be removed.

    Bit
Maybe the solution is as simple as specifying the state of memory that should be received from an untyped allocator. It could be explicitly stated that an untyped allocator should give out raw memory, and should not initialize it's content or add any ranges to the GC. While it may seem obvious for a C++ allocator not to initialize it's contents, I think this makes sense in the presence of a GC. I would appreciate some feedback on this. Bit
Sep 14 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2015 10:46 AM, bitwise wrote:
 On Saturday, 12 September 2015 at 06:45:16 UTC, bitwise wrote:
 [...]

 Alternatively, GC.addRange() could return a value indicating whether
 or not the range had actually been added(for the first time) and
 should be removed.

    Bit
Maybe the solution is as simple as specifying the state of memory that should be received from an untyped allocator. It could be explicitly stated that an untyped allocator should give out raw memory, and should not initialize it's content or add any ranges to the GC. While it may seem obvious for a C++ allocator not to initialize it's contents, I think this makes sense in the presence of a GC. I would appreciate some feedback on this. Bit
I think what we need here is a GCConnectedAllocator that can be inserted at the bottom of the allocation foodchain to insert calls to addRange and removeRange appropriately. -- Andrei
Sep 14 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 2:31 PM, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable allocator?
Well there's the matter of defining typed allocators on top of these ones. But yes, once we have allocators in place, they will become part of the Phobos' APIs they can serve. Andrei
Oct 24 2013
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.
Very, very nice! One question that immediately jumps out for me: As a primitive, will deallocateAll always call the parent's deallocate/deallocateAll? Sometimes it's useful to tell a top-level allocator to forget it allocated anything, but not actually free memory, so that it can be reused - similarly to Appender's clear method. There doesn't seem to be an easy way to do it with Freelist, as it will always forward to the parent's deallocate or deallocateAll. Perhaps there could be a "clear" primitive too? Spied two typos: - s/AllocatorWithOptions/AllocatorWithStats/ - s/an the call/the call/
Oct 24 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 24 October 2013 at 21:44:43 UTC, Vladimir Panteleev 
wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei 
 Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.
A nitpick: FallbackAllocator!(InSituRegion!..., ...) can be faster by defining a specialized allocator as a Region, which is initialized with the slice of a static buffer, and once that is filled up, a new buffer. This avoids a conditional branch in FallbackAllocator (we'd be reusing the same branch that checks for a full buffer and returns null in Region). This combination will likely be used often so it might be worth optimizing this use case, but OTOH the FallbackAllocator conditional branch will likely be subject to good branch prediction, so at least on x86 the gain might be minimal.
Oct 24 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 3:08 PM, Vladimir Panteleev wrote:
 On Thursday, 24 October 2013 at 21:44:43 UTC, Vladimir Panteleev wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test
 drive.
A nitpick: FallbackAllocator!(InSituRegion!..., ...) can be faster by defining a specialized allocator as a Region, which is initialized with the slice of a static buffer, and once that is filled up, a new buffer. This avoids a conditional branch in FallbackAllocator (we'd be reusing the same branch that checks for a full buffer and returns null in Region). This combination will likely be used often so it might be worth optimizing this use case, but OTOH the FallbackAllocator conditional branch will likely be subject to good branch prediction, so at least on x86 the gain might be minimal.
Yah, good idea. At some point I have no doubt that intensively-used allocators will be tuned a fair amount. Andrei
Oct 24 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 2:44 PM, Vladimir Panteleev wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.
Very, very nice! One question that immediately jumps out for me: As a primitive, will deallocateAll always call the parent's deallocate/deallocateAll?
That's a gray area for the time being. What I fear (for e.g. HeapBlock) is a pathological case in which deallocateAll frees parent's buffer and then a reallocation grabs another one etc. Also I haven't forgotten your suggestion returnsZeroedMemory. I think we should implement it at a point.
 Sometimes it's useful to tell a top-level allocator to forget it
 allocated anything, but not actually free memory, so that it can be
 reused - similarly to Appender's clear method. There doesn't seem to be
 an easy way to do it with Freelist, as it will always forward to the
 parent's deallocate or deallocateAll. Perhaps there could be a "clear"
 primitive too?
I think that's a good idea.
 Spied two typos:
 - s/AllocatorWithOptions/AllocatorWithStats/
 - s/an the call/the call/
Fixed, thanks. Andrei
Oct 24 2013
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 24 Oct 2013 12:54:41 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 Please destroy! I've literally sweat as I'm sending this :o).
=20
 Andrei
1) =C2=BBFallbackAllocator is an the allocator equivalent of an "or" operator in algebra.=C2=AB Remove the "the" there. 2) FallbackAllocator!(InSituRegion!16384, GCAllocator) a; Awesome, I actually use stuff like this. 3) // Same as above, except the freelist only applies to the reap. What is a reap? Is my English just bad or did 'h' get replaced with 'r' here a few times? 4) =C2=BBFor example, (D HeapBlock!(Allocator, 4096) (described in detail below) =E2=80=A6=C2=AB Broken DDoc macro. The class wrapper at the end is a nice addition to the set of allocators. It seems like a good pattern to have in D with the constant mix of class hierarchies and value types. --=20 Marco
Oct 24 2013
next sibling parent "qznc" <qznc web.de> writes:
On Thursday, 24 October 2013 at 21:57:10 UTC, Marco Leise wrote:
 3) // Same as above, except the freelist only applies to the
 reap.

 What is a reap? Is my English just bad or did 'h' get replaced
 with 'r' here a few times?
A reap mixes region and heap. Previous discussion: http://forum.dlang.org/post/aikhlqoffaznpkugesev forum.dlang.org Paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.7.6505&rep=rep1&type=pdf
Oct 24 2013
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 24 October 2013 at 21:57:10 UTC, Marco Leise wrote:
 3) // Same as above, except the freelist only applies to the
 reap.

 What is a reap? Is my English just bad or did 'h' get replaced
 with 'r' here a few times?
Reap = region + heap. I believe the term originates from this 2002 paper: http://people.cs.umass.edu/~emery/pubs/berger-oopsla2002.pdf Andrei mentioned them in this 2008 talk: http://accu.org/content/conf2008/Alexandrescu-memory-allocation.screen.pdf
Oct 24 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 2:56 PM, Marco Leise wrote:
 Am Thu, 24 Oct 2013 12:54:41 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 Please destroy! I've literally sweat as I'm sending this :o).

 Andrei
1) »FallbackAllocator is an the allocator equivalent of an "or" operator in algebra.« Remove the "the" there.
Fixed, thanks. I see "reap" has been discussed. Andrei
Oct 24 2013
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there 
 are no benchmarks. Both would be appreciated, particularly 
 benchmarks to validate the gains (which I speculate can be very 
 sizable) of custom-built, special-purpose allocators compared 
 to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this 
 design. I have been knocking around it for long enough to have 
 no idea how easy it is to get into it from the outside, or how 
 good it is. By all signs I could gather this feels like good 
 design, and one of the best I've ever put together. The 
 allocators defined have an archetypal feeling, are flexible 
 both statically and dynamically, and morph and combine in 
 infinite ways.

 CAllocator and CAllocatorImpl make the link between the static 
 and dynamic worlds. Once an allocator is assembled out of 
 pieces and finely tuned, wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei
I'm nowhere near experienced enough to comment on how good this really is, but it sure looks like an impressive design. Congratulations :)
Oct 24 2013
prev sibling next sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
I admit, I don't use custom allocators as much as I should and am not as experienced in this topic as others, yet for my purposes this design seems excellent. The docs are solid, and I really like the idea of just being able to composite something like AllocatorWithStats. The use cases I had in mind (which are admittedly quite simple) are covered, and it's very clear how to add a custom allocator if desired. A couple of questions: Shouldn't most of these implementations be marked nothrow? I could be missing something here. For CAllocator, I notice setting alignment is a property yet does not return void. Does this not go against the idea of property? Perhaps it should be setAlignment instead?
Oct 24 2013
next sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Thursday, 24 October 2013 at 22:19:10 UTC, Kapps wrote:
 For CAllocator, I notice setting alignment is a property yet 
 does not return void. Does this not go against the idea of 
  property? Perhaps it should be setAlignment instead?
Actually, this doesn't make sense because then it would not be possible to use CAllocator instead of an Allocator in a template as the type would be wrong. It still feels a bit odd that it returns bool to me, but I definitely see the logic now.
Oct 24 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 3:19 PM, Kapps wrote:
 A couple of questions:
 Shouldn't most of these implementations be marked nothrow? I could be
 missing something here.
Oh yah, that's part of the whole "alpha" thing. Then, since most allocators are templated, I'm hoping language definition will catch up with me and just infer them most of the time.
 For CAllocator, I notice setting alignment is a property yet does not
 return void. Does this not go against the idea of  property? Perhaps it
 should be setAlignment instead?
Yah, I'll fix that. Thanks! Andrei
Oct 24 2013
prev sibling next sibling parent reply Brad Roberts <braddr puremagic.com> writes:
There was a comment in an earlier reply that's very relevant to mine.  A good
demonstration of the 
utility of these classes is how much of the current (and proposed) garbage
collector can be replaced 
by using this module.  For that to happen, the code needs to actually live in
the runtime (at least 
as things are currently layered).

On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha release of
untyped allocators is
 ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there are no
benchmarks. Both would be
 appreciated, particularly benchmarks to validate the gains (which I speculate
can be very sizable)
 of custom-built, special-purpose allocators compared to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this design. I have been
knocking around it for
 long enough to have no idea how easy it is to get into it from the outside, or
how good it is. By
 all signs I could gather this feels like good design, and one of the best I've
ever put together.
 The allocators defined have an archetypal feeling, are flexible both
statically and dynamically, and
 morph and combine in infinite ways.

 CAllocator and CAllocatorImpl make the link between the static and dynamic
worlds. Once an allocator
 is assembled out of pieces and finely tuned, wrapping it in a dynamic API is a
snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei
Oct 24 2013
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
25-Oct-2013 02:29, Brad Roberts пишет:
 There was a comment in an earlier reply that's very relevant to mine.  A
 good demonstration of the utility of these classes is how much of the
 current (and proposed) garbage collector can be replaced by using this
 module.  For that to happen, the code needs to actually live in the
 runtime (at least as things are currently layered).

 On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is
 ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there are no
 benchmarks. Both would be
 appreciated, particularly benchmarks to validate the gains (which I
 speculate can be very sizable)
 of custom-built, special-purpose allocators compared to traditional
 allocators.

 I acknowledge I'm clearly in no position to evaluate this design. I
 have been knocking around it for
 long enough to have no idea how easy it is to get into it from the
 outside, or how good it is. By
 all signs I could gather this feels like good design, and one of the
 best I've ever put together.
 The allocators defined have an archetypal feeling, are flexible both
 statically and dynamically, and
 morph and combine in infinite ways.

 CAllocator and CAllocatorImpl make the link between the static and
 dynamic worlds. Once an allocator
 is assembled out of pieces and finely tuned, wrapping it in a dynamic
 API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei
Looks incredibly cool so far :) Some thoughts: - I don't like the name CAllocator. Whatever that C stands for it's ambiguous (Class, Cee, Caramel?). - It may be only me but I _think_ it could be more useful to have a few specific interfaces then 1 fat CAllocator. I've come to dislike fat-interfaces with isSupported kludges but that's IMHO. - I see that shrink primitive didn't make it... Well, on the upside the primitives count is really low. - In description of chooseAtRuntime: HeapBlock!chooseAtRuntime --> HeapBlock!(Allocator, chooseAtRuntime) -- Dmitry Olshansky
Oct 25 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/25/13 12:19 PM, Dmitry Olshansky wrote:
 - I don't like the name CAllocator. Whatever that C stands for it's
 ambiguous (Class, Cee, Caramel?).
Class. Better names welcome.
 - It may be only me but I _think_ it could be more useful to have a few
 specific interfaces then 1 fat CAllocator. I've come to dislike
 fat-interfaces with isSupported kludges but that's IMHO.
Yah it's a good idea. I'll think of it.
 - I see that shrink primitive didn't make it... Well, on the upside the
 primitives count is really low.
Yah. "Getting away with as little as you can" is my mantra :o).
 - In description of chooseAtRuntime:
 HeapBlock!chooseAtRuntime  --> HeapBlock!(Allocator, chooseAtRuntime)
Fixed and pushed. https://github.com/andralex/phobos/blob/allocator/std/allocator.d Andrei
Oct 26 2013
next sibling parent "Jacob Carlborg" <doob me.com> writes:
On Sunday, 27 October 2013 at 06:15:52 UTC, Andrei Alexandrescu 
wrote:

 Class. Better names welcome.
Allocator. Or we could call CAllocator AllocatorBase and call CAllocatorImpl Allocator. It depends on which of these two classes is most likely to be used in API's. I think the one most likely to be used should be called Allocator. Other names could be DynamicAllocator or RuntimeAllocator. -- /Jacob Carlborg
Oct 27 2013
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Sunday, 27 October 2013 at 06:15:52 UTC, Andrei Alexandrescu 
wrote:
 On 10/25/13 12:19 PM, Dmitry Olshansky wrote:
 - I don't like the name CAllocator. Whatever that C stands for 
 it's
 ambiguous (Class, Cee, Caramel?).
Class. Better names welcome.
I think DynamicAllocator would be a better name than CAllocator, but regardless, I think they're both pretty bad. However, I think it's indicative of a deeper problem with the interface chosen here: CAllocator is probably a kind of god object. Like Dmitry, I think using `interface`s (maybe `wrap` can be leveraged for the adaptation role[1]) is an approach worth investigating. It lets the library statically enforce that it gets the functionality it needs as opposed to having to raise a runtime error, while still allowing the caller to implement that functionality dynamically (urgh, reaching for words!). [1] Assuming `wrap` works on structs.
Oct 27 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there 
 are no benchmarks. Both would be appreciated, particularly 
 benchmarks to validate the gains (which I speculate can be very 
 sizable) of custom-built, special-purpose allocators compared 
 to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this 
 design. I have been knocking around it for long enough to have 
 no idea how easy it is to get into it from the outside, or how 
 good it is. By all signs I could gather this feels like good 
 design, and one of the best I've ever put together. The 
 allocators defined have an archetypal feeling, are flexible 
 both statically and dynamically, and morph and combine in 
 infinite ways.

 CAllocator and CAllocatorImpl make the link between the static 
 and dynamic worlds. Once an allocator is assembled out of 
 pieces and finely tuned, wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei
First, I love it ! Now destroying :D The first thing I notice is that this module is dead unsafe. I'm not sure it belong to std. We should probably reserve that for a "guru" part of the standard lib. Putting some system function in the standard lib is OK, as they are packaged around concepts (range, array, algorithms, etc . . .) but an entire module dedicated to system stuff seems like it belong to another place. I see that you did update the whole stuff according to previous discussion, and I'm not sure I have something against the design anymore :D Now, typed allocators, generic tail const and we can get awesome collections!
Oct 24 2013
next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Thursday, 24 October 2013 at 23:22:19 UTC, deadalnix wrote:
 The first thing I notice is that this module is dead unsafe. 
 I'm not sure it belong to std. We should probably reserve that 
 for a "guru" part of the standard lib. Putting some system 
 function in the standard lib is OK, as they are packaged around 
 concepts (range, array, algorithms, etc . . .) but an entire 
 module dedicated to  system stuff seems like it belong to 
 another place.
std.unsafe.*?
Oct 24 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 25 October 2013 at 01:10:24 UTC, Meta wrote:
 On Thursday, 24 October 2013 at 23:22:19 UTC, deadalnix wrote:
 The first thing I notice is that this module is dead unsafe. 
 I'm not sure it belong to std. We should probably reserve that 
 for a "guru" part of the standard lib. Putting some system 
 function in the standard lib is OK, as they are packaged 
 around concepts (range, array, algorithms, etc . . .) but an 
 entire module dedicated to  system stuff seems like it belong 
 to another place.
std.unsafe.*?
uac.* Many bonus points for who get this.
Oct 24 2013
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
25-Oct-2013 03:22, deadalnix пишет:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
[snip]
 First, I love it !

 Now destroying :D

 The first thing I notice is that this module is dead unsafe. I'm not
 sure it belong to std. We should probably reserve that for a "guru" part
 of the standard lib. Putting some system function in the standard lib is
 OK, as they are packaged around concepts (range, array, algorithms, etc
 . . .) but an entire module dedicated to  system stuff seems like it
 belong to another place.
+1 on both counts. It's a system module, incredibly convenient but it belongs to core.allocator; -- Dmitry Olshansky
Oct 25 2013
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-10-24 19:54:41 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 Please destroy! I've literally sweat as I'm sending this :o).
Seems good. Although I'm no expert on allocators I find it very easy to grasp. One remark though. While it generally makes sense that you know the size of an allocation when you want to deallocate, in some case it might be suboptimal to have to provide the size. For instance, you could use an allocator to allocate an object (with a virtual table and all). When comes the time to deallocate, to get the size you might have to dereference two pointers to extract the value from the classinfo, then create the proper void[] range to feed deallocate(). If your allocator is a wrapper for malloc/free, the size bit is ignored when calling free and all that work for retrieving the actual size of the object is wasted. I don't know if this is something worth addressing. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Oct 24 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 5:13 PM, Michel Fortin wrote:
 On 2013-10-24 19:54:41 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:

 Please destroy! I've literally sweat as I'm sending this :o).
Seems good. Although I'm no expert on allocators I find it very easy to grasp. One remark though. While it generally makes sense that you know the size of an allocation when you want to deallocate, in some case it might be suboptimal to have to provide the size. For instance, you could use an allocator to allocate an object (with a virtual table and all). When comes the time to deallocate, to get the size you might have to dereference two pointers to extract the value from the classinfo, then create the proper void[] range to feed deallocate(). If your allocator is a wrapper for malloc/free, the size bit is ignored when calling free and all that work for retrieving the actual size of the object is wasted. I don't know if this is something worth addressing.
If that happens, perhaps AffixAllocator!(A, size_t) could be of use by sticking the allocated size just before the allocation. But then you have a different problem - tapping into possibly cold memory when deallocating. I have plans to define a type that allows user-defined extra data grouped together (e.g. in an array or a list). Due to packing, subword per-allocation data can be stored efficiently. Such a design would allow storing all object sizes together. Andrei
Oct 24 2013
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-10-25 00:20:41 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 If that happens, perhaps AffixAllocator!(A, size_t) could be of use by 
 sticking the allocated size just before the allocation. But then you 
 have a different problem - tapping into possibly cold memory when 
 deallocating.
The size is already available in the classinfo, but the underlying allocator doesn't need it when deallocating (the size part will get ignored). The goal is to not have to give the size to deallocate when it doesn't need it (to save you from retrieving it). All you need is to know that deallocate() ignores the size part of the given array telling you whether or not it's safe to call deallocate(pointer[0..0]). You could add an optional "deallocateIgnoresSize" property for that. By the way, I think the idea of adding a boolean property for this offers less room for misuse than adding an optional deallocate(void*b) overload. With an overload you're duplicating the API and it won't be immediately clear whether or not it's best to provide the size. But I'm still not sure it's worth the trouble. I'll leave others be the judge of that. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Oct 24 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 25 October 2013 at 01:00:04 UTC, Michel Fortin wrote:
 On 2013-10-25 00:20:41 +0000, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:

 If that happens, perhaps AffixAllocator!(A, size_t) could be 
 of use by sticking the allocated size just before the 
 allocation. But then you have a different problem - tapping 
 into possibly cold memory when deallocating.
The size is already available in the classinfo, but the underlying allocator doesn't need it when deallocating (the size part will get ignored). The goal is to not have to give the size to deallocate when it doesn't need it (to save you from retrieving it). All you need is to know that deallocate() ignores the size part of the given array telling you whether or not it's safe to call deallocate(pointer[0..0]). You could add an optional "deallocateIgnoresSize" property for that. By the way, I think the idea of adding a boolean property for this offers less room for misuse than adding an optional deallocate(void*b) overload. With an overload you're duplicating the API and it won't be immediately clear whether or not it's best to provide the size. But I'm still not sure it's worth the trouble. I'll leave others be the judge of that.
Many allocators have different policies for different size. So, if you don't pass the size, you ends up have to recover it, which can be a tedious constraint. If the length is not used, I hope that The compiler may be able to optimize it way if the allocator is simple (so the deallocate function is inlined). If the deallocate function is quite complex, then I guess this extra calculation is required anyway, and won't matter that much.
Oct 24 2013
prev sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 25 October 2013 at 01:00:04 UTC, Michel Fortin wrote:
 On 2013-10-25 00:20:41 +0000, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:

 If that happens, perhaps AffixAllocator!(A, size_t) could be 
 of use by sticking the allocated size just before the 
 allocation. But then you have a different problem - tapping 
 into possibly cold memory when deallocating.
The size is already available in the classinfo, but the underlying allocator doesn't need it when deallocating (the size part will get ignored). The goal is to not have to give the size to deallocate when it doesn't need it (to save you from retrieving it). All you need is to know that deallocate() ignores the size part of the given array telling you whether or not it's safe to call deallocate(pointer[0..0]). You could add an optional "deallocateIgnoresSize" property for that. By the way, I think the idea of adding a boolean property for this offers less room for misuse than adding an optional deallocate(void*b) overload. With an overload you're duplicating the API and it won't be immediately clear whether or not it's best to provide the size. But I'm still not sure it's worth the trouble. I'll leave others be the judge of that.
A void deallocate(void* b) overload might be good in these cases.
Oct 25 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2013 12:54 PM, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha release of
 untyped allocators is ready for tire-kicking and a test drive.
This is not a comment on the allocator design, but the module layout. Would it be possible that this use the "package" idea with one allocator per file instead of the all-in-one-file setup?
Oct 24 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-10-25 03:08, Walter Bright wrote:

 This is not a comment on the allocator design, but the module layout.

 Would it be possible that this use the "package" idea with one allocator
 per file instead of the all-in-one-file setup?
Yes, please. -- /Jacob Carlborg
Oct 24 2013
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 10/25/2013 03:08 AM, Walter Bright wrote:
 Would it be possible that this use the "package" idea with one allocator
 per file instead of the all-in-one-file setup?
I'd like to see that as well.
Oct 30 2013
prev sibling next sibling parent "inout" <inout gmail.com> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there 
 are no benchmarks. Both would be appreciated, particularly 
 benchmarks to validate the gains (which I speculate can be very 
 sizable) of custom-built, special-purpose allocators compared 
 to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this 
 design. I have been knocking around it for long enough to have 
 no idea how easy it is to get into it from the outside, or how 
 good it is. By all signs I could gather this feels like good 
 design, and one of the best I've ever put together. The 
 allocators defined have an archetypal feeling, are flexible 
 both statically and dynamically, and morph and combine in 
 infinite ways.

 CAllocator and CAllocatorImpl make the link between the static 
 and dynamic worlds. Once an allocator is assembled out of 
 pieces and finely tuned, wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei
Looks good at first glance. A few notes: - GC.realloc() is nothrow, but you are still catching OOM exception in GCAllocator.reallocate(). You don't in other methods (e.g. allocate) - methods aren't marked as nothrow, they probably should be - I don't understand how UntypedAllocator.collect() works - since it is not allowed to store any pointers in it, calling collect() on it should just free all allocated memory, right? In this case, it might be best to rename it. It will also be beneficial to pass proper bitmask attribute to GC.allocate() (e.g. BlkAttr.NO_SCAN). - Many constants are global lowercase (i.e. platformAlignment, unbounded etc). It confused me for a bit when I was reading code ("where the hell is this variable coming from?"). Will read the rest later.
Oct 24 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-10-24 21:54, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
As someone else mentioned. Shouldn't this possibly be place in druntime instead. It might take advantage of some allocator where it cannot use the GC. I think we can come up with better names for "it" and "goodAllocSize". Typo: In the table, the description of goodAllocSize: "This module defines a default mplementation" Missing an "i" in "implementation". -- /Jacob Carlborg
Oct 24 2013
prev sibling next sibling parent reply "simendsjo" <simendsjo gmail.com> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.
Does anyone has good resources on allocators in general? I've mostly used malloc/free and GCs. None of the books I've read has gone into allocators..
Oct 25 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/25/13 1:21 AM, simendsjo wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.
Does anyone has good resources on allocators in general? I've mostly used malloc/free and GCs. None of the books I've read has gone into allocators..
I don't know of any literature in book format. There are a few papers though on HeapLayers, Hoard, jemalloc, and probably more. Fetch a couple and see what they cite and who cites them. Andrei
Oct 25 2013
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/24/2013 09:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha release of
untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
Looks good.
 ...

 Please destroy! I've literally sweat as I'm sending this :o).
...
- Possible null dereference on line 3128. :o) (In general, CascadingAllocator is not yet very composable. It should return null allocations instead of failing in undocumented ways. It should probably also allow the 'make' function to return null.) - "CAllocator". The name is non-descriptive/misleading. - Preconditions should be in the corresponding section.
Oct 25 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/25/13 5:50 AM, Timon Gehr wrote:
 - Possible null dereference on line 3128. :o)
    (In general, CascadingAllocator is not yet very composable. It should
 return null allocations instead of failing in undocumented ways.
Good point. Fixed and pushed.
 It
 should probably also allow the 'make' function to return null.)
make() returns Allocator objects by value, not pointers.
 - "CAllocator". The name is non-descriptive/misleading.
"Class Allocator" :o)
 - Preconditions should be in the corresponding section.
Will do... Andrei
Oct 25 2013
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/25/2013 05:21 PM, Andrei Alexandrescu wrote:
 On 10/25/13 5:50 AM, Timon Gehr wrote:
 - Possible null dereference on line 3128. :o)
    (In general, CascadingAllocator is not yet very composable. It should
 return null allocations instead of failing in undocumented ways.
Good point. Fixed and pushed. ...
Thanks.
 It
 should probably also allow the 'make' function to return null.)
make() returns Allocator objects by value, not pointers. ...
'Allocator' could in general be a reference type.
 - "CAllocator". The name is non-descriptive/misleading.
"Class Allocator" :o) ...
Then its purpose must be allocating classes? :o)
Oct 25 2013
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 26 Oct 2013 00:11:13 +0200
schrieb Timon Gehr <timon.gehr gmx.ch>:

 - "CAllocator". The name is non-descriptive/misleading.
"Class Allocator" :o) ...
Then its purpose must be allocating classes? :o)
I'm actually fine with that name, probably because I tend to prefix my D reference types in that manner, too. 'I' for interfaces, 'C' for classes and maybe 'A' for abstract classes. -- Marco
Oct 26 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-10-26 11:03, Marco Leise wrote:

 I'm actually fine with that name, probably because I tend to
 prefix my D reference types in that manner, too. 'I' for
 interfaces, 'C' for classes and maybe 'A' for abstract classes.
That's just horrible. -- /Jacob Carlborg
Oct 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/26/13 4:02 AM, Jacob Carlborg wrote:
 On 2013-10-26 11:03, Marco Leise wrote:

 I'm actually fine with that name, probably because I tend to
 prefix my D reference types in that manner, too. 'I' for
 interfaces, 'C' for classes and maybe 'A' for abstract classes.
That's just horrible.
I think it's a commonly-used convention. Andrei
Oct 26 2013
next sibling parent "Jacob Carlborg" <doob me.com> writes:
On Saturday, 26 October 2013 at 15:00:12 UTC, Andrei Alexandrescu 
wrote:

 I think it's a commonly-used convention.
Is this something that comes from C++ which doesn't have explicit abstract classes and interfaces in the same sense that D, Java I dislike this as much as I dislike adding or appending m_ or _ for instance variables. -- /Jacob Carlborg
Oct 27 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-10-26 17:01, Andrei Alexandrescu wrote:

 I think it's a commonly-used convention.
Any book, covering the topic, will say that mangling types in names is bad practice. We don't use this naming convention anywhere in Phobos. -- /Jacob Carlborg
Oct 27 2013
prev sibling next sibling parent "ponce" <contact gam3sfrommars.fr> writes:
 Please destroy! I've literally sweat as I'm sending this :o).
This looks fantastic. Congratulations!
Oct 25 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 25 October 2013 05:54, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org
 wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha release
 of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/**phobos/blob/allocator/std/**
 allocator.d<https://github.com/andralex/phobos/blob/allocator/std/allocator.d>

 Dox: http://erdani.com/d/phobos-**prerelease/std_allocator.html<http://erdani.com/d/phobos-prerelease/std_allocator.html>

 Warning: this is alpha quality. Unit tests are thin, and there are no
 benchmarks. Both would be appreciated, particularly benchmarks to validate
 the gains (which I speculate can be very sizable) of custom-built,
 special-purpose allocators compared to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this design. I have
 been knocking around it for long enough to have no idea how easy it is to
 get into it from the outside, or how good it is. By all signs I could
 gather this feels like good design, and one of the best I've ever put
 together. The allocators defined have an archetypal feeling, are flexible
 both statically and dynamically, and morph and combine in infinite ways.

 CAllocator and CAllocatorImpl make the link between the static and dynamic
 worlds. Once an allocator is assembled out of pieces and finely tuned,
 wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).
I like it. It seems like a solid foundation. I'm still very curious to see how the next phase (deep language integration) will look though. My immediate reactions: 1. I'm still sad there are no allocAligned() functions or something of that type to request explicit alignment with allocations. I'm not sure there is sufficient support for requesting alignment with allocations. The set-able alignment property approach seems a little weird (and only seemed to be supported on one allocator?). I guess experience will tell if this is sufficient and/or convenient. I'd still like to see an allocWithAlignment() method or something, which may be implemented efficiently by allocators that can support it. 2. I see some lines like this: assert(parent.alignment >= X.alignof); What if parent.alignment < X.alignof? If 'parent' is something with an inflexible alignment, like malloc or the GC, what is the proper (convenient) way to reconcile the requirement? 3. FreeList has some options; minSize, maxSize, maxNodes. When I'm using a freelist, the most important option to me is to be able to allocate new nodes in batches. I'd like an option added to control the batch size, so multiple new nodes are allocated in contiguous blocks when the pool grows. Perhaps add a 4th parameter; minBatchSize = 1? (to retain support for your existing logic batching small allocations into larger allocated blocks) The main reasons for this are cache/spatial locality of small allocations, and minimising overhead burden on the upstream allocator. 4. OT: Working with D struct's, you often encounter, for instance: struct Region(uint minAlign = platformAlignment) { private BasicRegion!(minAlign) base; ... Doesn't this make you sad? It makes me sad quite regularly. Especially when the very next line is (often): alias base this;
Oct 25 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/25/13 7:23 AM, Manu wrote:
 My immediate reactions:

 1. I'm still sad there are no allocAligned() functions or something of
 that type to request explicit alignment with allocations. I'm not sure
 there is sufficient support for requesting alignment with allocations.
 The set-able alignment property approach seems a little weird (and only
 seemed to be supported on one allocator?). I guess experience will tell
 if this is sufficient and/or convenient.
 I'd still like to see an allocWithAlignment() method or something, which
 may be implemented efficiently by allocators that can support it.
Per-allocation alignment requests are currently allowed (by setting the property transitorily) but indeed not really nice. I can see how HeapBlock could implement a nice alignedAllocate call but the others would kind of boringly pass it along. Don't forget that it's always possible to define extra primitives for a given allocator (see e.g. relinquish or available). They should be migrated to official API status only if they could help composition in one way or another. In fact that's how I defined the API - I started with allocate()/deallocate() and a list of allocators I wanted to implement, and tried to get away with as few primitives as possible. For example FallbackAllocator makes owns() necessary etc.
 2. I see some lines like this:
 assert(parent.alignment>=X.alignof);
 What if parent.alignment < X.alignof? If 'parent' is something with an
 inflexible alignment, like malloc or the GC, what is the proper
 (convenient) way to reconcile the requirement?
That assert is in Freelist and requires that the parent allocator returns memory aligned to at least pointer alignment, so as to write pointers at the front of the allocation. That's a really low bar, I think there's no need to worry about it. (If you do, defining a UnalignedFreelist is always an option.)
 3. FreeList has some options; minSize, maxSize, maxNodes. When I'm using
 a freelist, the most important option to me is to be able to allocate
 new nodes in batches. I'd like an option added to control the batch
 size, so multiple new nodes are allocated in contiguous blocks when the
 pool grows. Perhaps add a 4th parameter; minBatchSize = 1? (to retain
 support for your existing logic batching small allocations into larger
 allocated blocks)
 The main reasons for this are cache/spatial locality of small
 allocations, and minimising overhead burden on the upstream allocator.
Good idea; it's what people often do anyway. Before sending this out I'd added at least the option to allocate several nodes at a time to fill memory more efficiently, see allocateFresh at https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L1988. Allowing the user to choose the number of nodes in a batch is a good extension of that. (I think the default should be unbounded, i.e. let the freelist allocator decide depending on goodMallocSize of the parent.)
 4. OT: Working with D struct's, you often encounter, for instance:

 structRegion(uintminAlign=platformAlignment)
 {
 privateBasicRegion!(minAlign)base;
 ...

 Doesn't this make you sad? It makes me sad quite regularly.
 Especially when the very next line is (often): alias base this;
I'm content with that and I think you should too. Andrei
Oct 25 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/25/13 7:23 AM, Manu wrote:
 1. I'm still sad there are no allocAligned() functions or something of
 that type to request explicit alignment with allocations. I'm not sure
 there is sufficient support for requesting alignment with allocations.
 The set-able alignment property approach seems a little weird (and only
 seemed to be supported on one allocator?).
Woke up this morning with the following idea. 1. We add two optional API functions: void[] alignedAllocate(size_t, uint); bool alignedReallocate(ref void[], size_t, uint); These would satisfy aligned allocation requests. The pointer thus allocated should be deallocated the usual way (there is no alignedDeallocate). 2. We make alignment a statically-known enum and delete all code that allows it to be get and set at runtime. Consequences: 1. The API gets larger due to the two new APIs. However, that is offset (probably more than enough) by the removal the option to set the alignment at runtime, which currently is causing severe ripple effects. I think the result will be a simpler overall design and implementation. 2. HeapBlock should be able to implement the two functions. All regions should be able to implement the first. 3. Mallocator could also implement these on Posix: http://man7.org/linux/man-pages/man3/posix_memalign.3.html. However, Windows requires a specific call for deallocating aligned memory. To accommodate both portably, we leave Mallocator as is and create AlignedMallocator that uses the _aligned_* family on Windows and the respective functions on Posix. On Windows, allocate() requests would pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc. Is this satisfactory for everyone? Andrei
Oct 26 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 3. Mallocator could also implement these on Posix:
 http://man7.org/linux/man-pages/man3/posix_memalign.3.html. However,
 Windows requires a specific call for deallocating aligned memory. To
 accommodate both portably, we leave Mallocator as is and create
 AlignedMallocator that uses the _aligned_* family on Windows and the
 respective functions on Posix. On Windows, allocate() requests would
 pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc.
Just implemented AlignedMallocator and pushed. Untested on Windows. Andrei
Oct 27 2013
next sibling parent reply "Tourist" <gravatar gravatar.com> writes:
On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu 
wrote:
 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 3. Mallocator could also implement these on Posix:
 http://man7.org/linux/man-pages/man3/posix_memalign.3.html. 
 However,
 Windows requires a specific call for deallocating aligned 
 memory. To
 accommodate both portably, we leave Mallocator as is and create
 AlignedMallocator that uses the _aligned_* family on Windows 
 and the
 respective functions on Posix. On Windows, allocate() requests 
 would
 pass a default of platformSize (i.e. 16 I suspect) to 
 _aligned_malloc.
Just implemented AlignedMallocator and pushed. Untested on Windows. Andrei
In the following line: ---------- On Posix, forwards to realloc. On Windows, calls _aligned_realloc(b.ptr, newSize, platformAlignment). ---------- Link is incorrect (http//, colon is missing).
Oct 27 2013
next sibling parent "Tourist" <gravatar gravatar.com> writes:
On Sunday, 27 October 2013 at 10:52:06 UTC, Tourist wrote:
 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu 
 wrote:
 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 3. Mallocator could also implement these on Posix:
 http://man7.org/linux/man-pages/man3/posix_memalign.3.html. 
 However,
 Windows requires a specific call for deallocating aligned 
 memory. To
 accommodate both portably, we leave Mallocator as is and 
 create
 AlignedMallocator that uses the _aligned_* family on Windows 
 and the
 respective functions on Posix. On Windows, allocate() 
 requests would
 pass a default of platformSize (i.e. 16 I suspect) to 
 _aligned_malloc.
Just implemented AlignedMallocator and pushed. Untested on Windows. Andrei
In the following line: ---------- On Posix, forwards to realloc. On Windows, calls _aligned_realloc(b.ptr, newSize, platformAlignment). ---------- Link is incorrect (http//, colon is missing).
That's how Firefox interprets it. In source code it's: http://http//msdn.microsoft.com/en-US/library/y69db7sx(v=vs.80).aspx
Oct 27 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/27/13 3:52 AM, Tourist wrote:
 In the following line:
 ----------
 On Posix, forwards to realloc. On Windows, calls _aligned_realloc(b.ptr,
 newSize, platformAlignment).
 ----------

 Link is incorrect (http//, colon is missing).
Thanks, ouch, there were a few more bugs around there as well in the untested code. Andrei
Oct 27 2013
prev sibling next sibling parent reply "safety0ff" <safety0ff.dev gmail.com> writes:
On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu 
wrote:
 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 AlignedMallocator that uses the _aligned_* family on Windows 
 and the
 respective functions on Posix. On Windows, allocate() requests 
 would
 pass a default of platformSize (i.e. 16 I suspect) to 
 _aligned_malloc.
Just implemented AlignedMallocator and pushed.
The constraints on the alignment parameter are neither documented nor checked. i.e. Alignment must be a power of two, and for posix_memalign it must additionally be greater than sizeof(void*). I can only think of one use case for needing runtime specified alignment: allocating operating system page sized chunks. Are there any other use cases?
Oct 27 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/27/13 11:26 AM, safety0ff wrote:
 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote:
 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 AlignedMallocator that uses the _aligned_* family on Windows and the
 respective functions on Posix. On Windows, allocate() requests would
 pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc.
Just implemented AlignedMallocator and pushed.
The constraints on the alignment parameter are neither documented nor checked. i.e. Alignment must be a power of two, and for posix_memalign it must additionally be greater than sizeof(void*).
Fixed, will push soon.
 I can only think of one use case for needing runtime specified
 alignment: allocating operating system page sized chunks.
 Are there any other use cases?
Cache-line aligned. http://stackoverflow.com/questions/794632/programmatically-get-the-cache-line-size/1900464 Also some I/O requires aligned buffers. http://goo.gl/ni860U Andrei
Oct 27 2013
parent reply Manu <turkeyman gmail.com> writes:
On 28 October 2013 04:36, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org
 wrote:
 On 10/27/13 11:26 AM, safety0ff wrote:

 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote:

 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:

 AlignedMallocator that uses the _aligned_* family on Windows and the
 respective functions on Posix. On Windows, allocate() requests would
 pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc.
Just implemented AlignedMallocator and pushed. The constraints on the alignment parameter are neither documented nor
checked. i.e. Alignment must be a power of two, and for posix_memalign it must additionally be greater than sizeof(void*).
Fixed, will push soon. I can only think of one use case for needing runtime specified
 alignment: allocating operating system page sized chunks.
 Are there any other use cases?
Cache-line aligned. http://stackoverflow.com/**questions/794632/**programmatically-get-the-** cache-line-size/1900464<http://stackoverflow.com/questions/794632/programmatically-get-the-cache-line-size/1900464> Also some I/O requires aligned buffers. http://goo.gl/ni860U
GPU's also tend to deal with strict alignments of various buffers. SIMD requires >= 16byte alignment, which is higher than the default 4-8 byte alignment of many allocators.
Oct 27 2013
parent reply "safety0ff" <safety0ff.dev gmail.com> writes:
On Monday, 28 October 2013 at 02:30:53 UTC, Manu wrote:
 GPU's also tend to deal with strict alignments of various 
 buffers.

 SIMD requires >= 16byte alignment, which is higher than the 
 default 4-8
 byte alignment of many allocators.
AFAIK the alignments for those cases are known a priori, the question was regarding alignments that are only known at runtime.
Oct 27 2013
parent Manu <turkeyman gmail.com> writes:
On 28 October 2013 12:47, safety0ff <safety0ff.dev gmail.com> wrote:

 On Monday, 28 October 2013 at 02:30:53 UTC, Manu wrote:

 GPU's also tend to deal with strict alignments of various buffers.

 SIMD requires >= 16byte alignment, which is higher than the default 4-8
 byte alignment of many allocators.
AFAIK the alignments for those cases are known a priori, the question was regarding alignments that are only known at runtime.
You don't know the runtime GPU at compile time.
Oct 27 2013
prev sibling parent "jerro" <a a.com> writes:
 i.e. Alignment must be a power of two, and for posix_memalign 
 it must additionally be greater than sizeof(void*).
It must be a power of two and a multiple of sizeof(void*), which means that it must be either zero or greater or equal to sizeof(void*).
Oct 27 2013
prev sibling next sibling parent reply "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu 
wrote:
 Just implemented AlignedMallocator and pushed.



 Untested on Windows.
It seems inconsistent that allocate() forwards to alignedAllocate(), while reallocate() does not forward to alignedReallocate(). Why is this? Lars
Oct 28 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/28/13 1:03 AM, Lars T. Kyllingstad wrote:
 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote:
 Just implemented AlignedMallocator and pushed.




 Untested on Windows.
It seems inconsistent that allocate() forwards to alignedAllocate(), while reallocate() does not forward to alignedReallocate(). Why is this? Lars
Two reasons: (1) Posix does not support aligned reallocation. That would make forwarding a pessimization on that OS; (2) On Windows, realloc() cannot be applied to memory gotten with _aligned_malloc and vice versa. Andrei
Oct 28 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/28/13 8:15 AM, Andrei Alexandrescu wrote:
 On 10/28/13 1:03 AM, Lars T. Kyllingstad wrote:
 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote:
 Just implemented AlignedMallocator and pushed.





 Untested on Windows.
It seems inconsistent that allocate() forwards to alignedAllocate(), while reallocate() does not forward to alignedReallocate(). Why is this? Lars
Two reasons: (1) Posix does not support aligned reallocation. That would make forwarding a pessimization on that OS; (2) On Windows, realloc() cannot be applied to memory gotten with _aligned_malloc and vice versa. Andrei
Oh, my mistake. On Windows reallocate() should forward to alignedReallocate. Andrei
Oct 28 2013
prev sibling parent Jerry <jlquinn optonline.net> writes:
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:

 Just implemented AlignedMallocator and pushed.


I'm having a vision of a perverse version called MalignedAllocator...
Nov 04 2013
prev sibling parent Jerry <jlquinn optonline.net> writes:
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

 On 10/25/13 7:23 AM, Manu wrote:
 Woke up this morning with the following idea.

 1. We add two optional API functions:

 void[] alignedAllocate(size_t, uint);
 bool alignedReallocate(ref void[], size_t, uint);
A bit of bikeshedding: Since you have allocate() allocateAll() I would use allocateAligned() reallocateAligned() to be more consistent. Jerry
Nov 04 2013
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/25/13 7:23 AM, Manu wrote:
 3. FreeList has some options; minSize, maxSize, maxNodes. When I'm using
 a freelist, the most important option to me is to be able to allocate
 new nodes in batches.
Pushed new version with batchSize parameter, defaulted to 8. Andrei
Oct 26 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/25/13 7:23 AM, Manu wrote:
 1. I'm still sad there are no allocAligned() functions or something of
 that type to request explicit alignment with allocations. I'm not sure
 there is sufficient support for requesting alignment with allocations.
 The set-able alignment property approach seems a little weird (and only
 seemed to be supported on one allocator?). I guess experience will tell
 if this is sufficient and/or convenient.
 I'd still like to see an allocWithAlignment() method or something, which
 may be implemented efficiently by allocators that can support it.
Code: http://erdani.com/d/phobos-prerelease/std_allocator.html Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html I've made alignedAllocate() and alignedReallocate() parts of the official API, added alignedAlloc to all regions, added AlignedMallocator which taps into the aligned system APIs, and eliminated all that dynamic alignment setting stuff. I think we're in better shape now. Thanks for the suggestion! I still need to: 1. add stats for aligned calls to AllocatorWithStats 2. define HeapBlock.alignedXxx (which should be interesting) 3. separate IOwns, IAlignedAllocate, IDeallocate from CAllocator and have CAllocatorImpl!alloc conditionally implement them depending on whether alloc implements the respective primitives. Andrei
Oct 27 2013
prev sibling next sibling parent reply "goughy" <andrew goughy.org> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
Not really qualified to comment, but looks cool. Looking at AllocatorWithStats, https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2913 it looks like an error, should be ... static if (flags & Options.callerTime) Goughy
Oct 25 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/25/13 6:17 PM, goughy wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
Not really qualified to comment, but looks cool. Looking at AllocatorWithStats, https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2913 it looks like an error, should be ... static if (flags & Options.callerTime) Goughy
Famous criminals Cut & Paste strike again. Thanks! Andrei
Oct 25 2013
prev sibling next sibling parent "Chad Joan" <chadjoan gmail.com> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 ... awesome stuff ...

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei
I like it a lot so far. I was really worried about being able to dynamically dispatch to an allocator determined at a previous place in the call stack, and it seems you're all over it with CAllocator. Hell yeah! I have an editing suggestion for the CAllocator comment: Instead of "Implementation of CAllocator using Allocator. [...]" I suggest "Implements CAllocator using the given Allocator. [...]" The current one read strangely to me at first, and I had to re-read it several times and notice that "Allocator" referred to the template parameter. I agree with others that say that the name CAllocator is too ambiguous or vague. When I scanned through the allocators, I initially dismissed it because I though it was a proxy for the system's underlying C allocator, with Mallocator being a D-based optimized reimplementation of the C allocator. Reading further clarified this, but it does probably harm skimming and searching. I suggest an alternative name for CAllocator: DispatchingAllocator. I believe this may represent what it does: dispatch allocation to another allocator that is behind a curtain. Something like AbstractAllocator might work too, but still seems slightly ambiguous to me (i.e. abstract in what sense?). I just hope that the future "top" allocator that handles language-builtin allocations will be one that can maintain a stack of allocators and push/pop the current default allocator, as well as prevent or redirect allocator choice made within calls to 3rd party libraries (assuming the libraries are written in D, of course).
Oct 26 2013
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 24 Oct 2013 12:54:41 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 Hello,
 
 
 I know it's been a long wait. Hopefully it was worth it. The alpha 
 release of untyped allocators is ready for tire-kicking and a test
 drive.
 
 Code:
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d
 
 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
 
 Warning: this is alpha quality. Unit tests are thin, and there are no 
 benchmarks. Both would be appreciated, particularly benchmarks to 
 validate the gains (which I speculate can be very sizable) of 
 custom-built, special-purpose allocators compared to traditional
 allocators.
 
 I acknowledge I'm clearly in no position to evaluate this design. I
 have been knocking around it for long enough to have no idea how easy
 it is to get into it from the outside, or how good it is. By all
 signs I could gather this feels like good design, and one of the best
 I've ever put together. The allocators defined have an archetypal
 feeling, are flexible both statically and dynamically, and morph and
 combine in infinite ways.
 
 CAllocator and CAllocatorImpl make the link between the static and 
 dynamic worlds. Once an allocator is assembled out of pieces and
 finely tuned, wrapping it in a dynamic API is a snap.
 
 Please destroy! I've literally sweat as I'm sending this :o).
 
 
 Andrei
The overall design looks great! Some small nitpicks: * byAllocation is not accessible from outside std.allocator * Is it intentional that AllocatorWithStats can't log the __FUNCTION__ / __PRETTY_FUNCTION__ ? * The documentation for byAllocation should probably state that it only lists 'alive' allocations and that this is especially useful for finding memory leaks. And one bigger problem: CAllocatorImpl doesn't work well with AllocatorWithStats. AllocatorWithStats special members (byAllocation, etc) are not directly accessible. Although we can access them by using .impl it'd be nice to have a special AllocatorWithStats class which directly exposes those members. The second problem is that the logged file/line is always in std.allocator.d. It's probably not easy to get this working in all cases (especially regarding polymorphism).
Oct 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/26/13 5:23 AM, Johannes Pfau wrote:
 Some small nitpicks:
 * byAllocation is not accessible from outside std.allocator
Fixed.
 * Is it intentional that AllocatorWithStats can't log the
    __FUNCTION__ / __PRETTY_FUNCTION__ ?
Just an omission, good idea! Fixed. Let's just record __FUNCTION__.
 * The documentation for byAllocation should probably state that it only
    lists 'alive' allocations and that this is especially useful for
    finding memory leaks.
Fixed.
 And one bigger problem: CAllocatorImpl doesn't work well with
 AllocatorWithStats.

 AllocatorWithStats special members (byAllocation, etc) are not
 directly accessible. Although we can access them by using .impl it'd be
 nice to have a special AllocatorWithStats class which directly exposes
 those members.
That shouldn't be a problem - just downcast to CAllocatorImpl!AllocatorWithStats and you got access.
 The second problem is that the logged file/line is always in
 std.allocator.d. It's probably not easy to get this working in all
 cases (especially regarding polymorphism).
Uhm, that /is/ a problem. Andrei
Oct 26 2013
parent reply Lionello Lunesu <lionello lunesu.remove.com> writes:
On 10/27/13, 7:10, Andrei Alexandrescu wrote:
 The second problem is that the logged file/line is always in
 std.allocator.d. It's probably not easy to get this working in all
 cases (especially regarding polymorphism).
Uhm, that /is/ a problem.
This is supported by using default values: 1:import std.stdio; 2:void w(int l = __LINE__){writeln(l);} 3:void main(){w();} outputs: 3 By design :)
Oct 31 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/31/13 8:57 AM, Lionello Lunesu wrote:
 On 10/27/13, 7:10, Andrei Alexandrescu wrote:
 The second problem is that the logged file/line is always in
 std.allocator.d. It's probably not easy to get this working in all
 cases (especially regarding polymorphism).
Uhm, that /is/ a problem.
This is supported by using default values: 1:import std.stdio; 2:void w(int l = __LINE__){writeln(l);} 3:void main(){w();} outputs: 3 By design :)
The problem is only in CAllocator. Andrei
Oct 31 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 24/10/13 21:54, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha release of
 untyped allocators is ready for tire-kicking and a test drive.

  ...

 Please destroy! I've literally sweat as I'm sending this :o).
Hmm, seems like I'm coming late to quite a party ... :-) Like John Colvin, I feel that much of this is over my head -- I don't have the experience/expertise to judge requirements or solutions. But it looks like very nice code indeed and I'm encouraged by the fact that most "Can it do ...?" questions seem to have a ready answer. My own interests have always been less in allocators per se and more in the functionality they unlock -- stuff like containers, for example. It may be unfamiliarity -- my own code has either used new/delete (the latter only in C++, not D...) or malloc/dealloc -- but looking at this module as it stands I don't really see how to practically make use of it; which makes me wonder whether the people calling for it to be in core rather than std have a point. Maybe it would help to have code examples that really show these allocators really being used to do something -- the existing examples are comprehensive and show what to do, but not really what to do _with_ the constructs thus created. Or, maybe I should just educate myself more ... ;-) Suffice to say that my overwhelming feeling is of gratitude and admiration for all the obvious thought and hard work put into creating this module. Thanks and congratulations, Andrei! :-) Best wishes, -- Joe
Oct 26 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.
About time! ;) But it was definitely worth waiting for. I don't have that much experience with using custom allocators, but API-wise, this looks great. Not overburdened, but not underpowered either. I would suggest a different name for goodAllocSize(), though. Maybe actualAllocSize() or just allocSize(). This may be a stupid question, but is expand() useful outside the allocator infrastructure itself? Would you ever use that instead of reallocate() in a container, for instance? Lars
Oct 28 2013
prev sibling next sibling parent "Joseph Cassman" <jc7919 outlook.com> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei
Going through the code today after reading the thread for the past few days I was blown away by the depth of the concepts involved. Now the information in the article "Memory Allocation: Either Love It or Hate It (or just think it’s okay)" makes a lot more sense. The composability and customizability of the design look good. This is not an area of coding that I am that familiar with but I am interested in trying out a couple of ideas using the code in the coming weeks. I had a good laugh when I saw the ASCII art before the definition of NullAllocator. Pretty cool! Thanks for this work. Joseph
Oct 28 2013
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 10/24/2013 09:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
This looks really promising. There are a lot of building blocks and the way different capabilities are modelled by optional methods nicely solves the biggest difficulty with allocators. I think it's important to put this in it's own github repo and add a dub package for it on code.dlang.org so that it's easy to test the implementation and to contribute improvements. That said it failed my litmus test. I previously used David Simcha's RegionAllocator https://github.com/dsimcha/TempAlloc/blob/master/std/allocators/region.d. The pattern is to allocate some metadata followed by allocating many fixed size tree nodes. When the tree is constructed it is used to render an image which is the result of that operation. The tree and all metadata is freed and the region allocator is reused for the next method invocation (it keeps the memory). I think the closest would be to use CascadingAllocator with Region but there are two issues. CascadingAllocator successively tries all allocators and if that fails creates a new region. So this runs in O(N) complexity even though most of the time only the last allocator will have memory available. There is no simple way to deallocateAll without freeing the regions. What I need is something similar to clear in appender. I also can't relinquish the memory from the inner regions because they are private. So for my use-case the only way that I found to use this module is to compute the upper bound of memory needed when the renderer is invoked. Then I have to relinquish the buffer from a region, reallocate it using Mallocator.it and construct a new region with the reallocated buffer. This works only because I can cheaply compute the upper bound of required memory. This wouldn't work in other scenarios. I think this a very important use-case, e.g. using an auto-growing thread local region is what I would use to serve HTTP requests. But for this one might also want to use nested regions.
Oct 30 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/30/13 1:02 PM, Martin Nowak wrote:
 This looks really promising.
 There are a lot of building blocks and the way different capabilities
 are modelled by optional methods nicely solves the biggest difficulty
 with allocators.
 I think it's important to put this in it's own github repo and add a dub
 package for it on code.dlang.org so that it's easy to test the
 implementation and to contribute improvements.
Tried to register github package andralex/phobos, and: ============================ 500 - Internal Server Error Internal Server Error Internal error information: object.Exception source/dubregistry/repositories/repository.d(43): Failed to read JSON from https://raw.github.com/andralex/phobos/master/package.json: Unexpected reply for 'https://raw.github.com/andralex/phobos/master/package.json': Not Found ---------------- ./dub-registry(dubregistry.repositories.repository.PackageVersionInfo dubregistry.repositories.github.GithubRepository.getVersionInfo(immut ble(char)[])+0x3b5) [0x6e8ec1] ./dub-registry(void dubregistry.registry.DubRegistry.addPackage(vibe.data.json.Json, vibe.data.bson.BsonObjectID)+0xa6) [0x62d372] ./dub-registry(void dubregistry.web.DubRegistryWebFrontend.addPackage(vibe.http.serve .HTTPServerRequest, vibe.http.server.HTTPServerResponse, userman.controller.User)+0x222) [0x6e4406] ./dub-registry(void delegate(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse) userman.web.UserManWebInterface.auth(void delegate(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse, userman.controller.User)).void requestHandler(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x10d) [0x800181] ./dub-registry(void vibe.http.router.URLRouter.handleRequest(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x179) [0x6fb8b5] ./dub-registry(bool vibe.http.server.handleRequest(vibe.core.stream.Stream, immutable(char)[], vibe.http.server.HTTPServerListener, ref vibe.http.server.HTTPServerSettings, ref bool)+0x16c8) [0x6f0344] ./dub-registry(void vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerListener)+0x143) [0x6eebb7] ./dub-registry(void vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings, void delegate(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)).void doListen(vibe.http.server.HTTPServerSettings, vibe.http.server.HTTPServerListener, immutable(char)[]).void __lambda54(vibe.core.net.TCPConnection)+0x2c) [0x6eb160] ./dub-registry(extern (C) nothrow void vibe.core.drivers.libevent2_tcp.onConnect(int, short, void*).void ClientTask.execute()+0x2d6) [0x70939a] ./dub-registry(void vibe.core.core.CoreTask.run()+0xf2) [0x7172fe] ./dub-registry(void core.thread.Fiber.run()+0x2a) [0x83eae2] ./dub-registry(fiber_entryPoint+0x61) [0x83e9ed] [(nil)] ============================ This is obviously because package.json is absent from the repo, but I'd say it shouldn't cause such an error. That makes me think probably Phobos should have a package.json so people can install updates via code.dlang.org.
 That said it failed my litmus test.
 I previously used David Simcha's RegionAllocator
 https://github.com/dsimcha/TempAlloc/blob/master/std/allocators/region.d.
Let's see!
 The pattern is to allocate some metadata followed by allocating many
 fixed size tree nodes. When the tree is constructed it is used to render
 an image which is the result of that operation.
 The tree and all metadata is freed and the region allocator is reused
 for the next method invocation (it keeps the memory).

 I think the closest would be to use CascadingAllocator with Region but
 there are two issues.

 CascadingAllocator successively tries all allocators and if that fails
 creates a new region. So this runs in O(N) complexity even though most
 of the time only the last allocator will have memory available.
Yah, I'd left a TODO in there when I first wrote the code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L3723 The newly-added allocator should come to the front of the list. My suspicion, however, is that you won't be able to measure a difference. A well-dimensioned cascade of regions will have a high ration of allocations within a regions to number of regions.
 There is no simple way to deallocateAll without freeing the regions.
 What I need is something similar to clear in appender.
Hm, interesting. But then again - do you think it makes a difference? Allocation of regions is a very small fraction of the work done on using the regions.
 I also can't relinquish the memory from the inner regions because they
 are private.
How do we formalize that?
 So for my use-case the only way that I found to use this module
 is to compute the upper bound of memory needed when the renderer is
 invoked. Then I have to relinquish the buffer from a region, reallocate
 it using Mallocator.it and construct a new region with the reallocated
 buffer.
I'd say just plow ahead with a straight region as implemented. If you measure any difference, let's talk. Andrei
Nov 01 2013
next sibling parent Martin Nowak <code dawg.eu> writes:
On 11/01/2013 10:00 PM, Andrei Alexandrescu wrote:
 On 10/30/13 1:02 PM, Martin Nowak wrote:
 This looks really promising.
 There are a lot of building blocks and the way different capabilities
 are modelled by optional methods nicely solves the biggest difficulty
 with allocators.
 I think it's important to put this in it's own github repo and add a dub
 package for it on code.dlang.org so that it's easy to test the
 implementation and to contribute improvements.
Tried to register github package andralex/phobos, and:
I transferred you a separate repo with all std.allocator commits. https://github.com/dawgfoto/std_allocator You can build the library with "dub" and run unittests with "dub -build=unittest -config=unittest". Creating docs is a problem, maybe some expert can chime in.
 This is obviously because package.json is absent from the repo, but I'd
 say it shouldn't cause such an error.
https://github.com/rejectedsoftware/dub-registry/issues/31
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L3723

 The newly-added allocator should come to the front of the list.

 My suspicion, however, is that you won't be able to measure a
 difference. A well-dimensioned cascade of regions will have a high
 ration of allocations within a regions to number of regions.
You're probably right.
Nov 09 2013
prev sibling parent "Martin Nowak" <code dawg.eu> writes:
 Tried to register github package andralex/phobos, and:

 ============================
 500 - Internal Server Error

 Internal Server Error
Now fixed https://github.com/rejectedsoftware/dub-registry/issues/31.
Nov 13 2013
prev sibling next sibling parent reply "safety0ff" <safety0ff.dev gmail.com> writes:
I noticed that the GCAllocator provides no way of controlling the 
memory block attributes 

allocations get the default (no attributes.) This is a leaky 
abstraction, a data structure or composed allocators may desire 
to control the attributes to reduce GC pressure.
Oct 31 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/31/13 7:26 PM, safety0ff wrote:
 I noticed that the GCAllocator provides no way of controlling the memory

 ,) all allocations get the default (no attributes.) This is a leaky
 abstraction, a data structure or composed allocators may desire to
 control the attributes to reduce GC pressure.
These attributes seem to be informed by the types stored, which would be above the charter of untyped allocator. Andrei
Oct 31 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 1 November 2013 at 02:43:00 UTC, Andrei Alexandrescu 
wrote:
 On 10/31/13 7:26 PM, safety0ff wrote:
 I noticed that the GCAllocator provides no way of controlling 
 the memory
 block attributes 

 ,) all allocations get the default (no attributes.) This is a 
 leaky
 abstraction, a data structure or composed allocators may 
 desire to
 control the attributes to reduce GC pressure.
These attributes seem to be informed by the types stored, which would be above the charter of untyped allocator. Andrei
The choice is for the typed allocator. That mean that untyped allocator should either provide the choice, or assume the worse (it may contain pointer).
Oct 31 2013
prev sibling parent reply "safety0ff" <safety0ff.dev gmail.com> writes:
On Friday, 1 November 2013 at 02:43:00 UTC, Andrei Alexandrescu 
wrote:
 On 10/31/13 7:26 PM, safety0ff wrote:
 I noticed that the GCAllocator provides no way of controlling 
 the memory
 block attributes 

 ,) all allocations get the default (no attributes.) This is a 
 leaky
 abstraction, a data structure or composed allocators may 
 desire to
 control the attributes to reduce GC pressure.
These attributes seem to be informed by the types stored, which would be above the charter of untyped allocator. Andrei
The attributes are informed by whatever code is calling the GC, the GC interface deals in void*'s. Consider an AA implementation that wishes to use FancyAllocator with fallback GCAllocator with block attributes NO_INTERIOR and NO_SCAN. With your proposed GCAllocator you either need to rewrite GCAllocator, or you need to add some nasty code to set the attributes depending on whether the primary allocator or secondary allocator own the memory. By fixing the leaky abstraction this use case can be coded as follows: FallbackAllocator!(FancyAllocator, GCAllocator!(GC.BLkAttr.NO_INTERIOR | GC.BLkAttr.NO_SCAN)) a;
Oct 31 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/31/13 8:37 PM, safety0ff wrote:
 On Friday, 1 November 2013 at 02:43:00 UTC, Andrei Alexandrescu wrote:
 On 10/31/13 7:26 PM, safety0ff wrote:
 I noticed that the GCAllocator provides no way of controlling the memory

 ,) all allocations get the default (no attributes.) This is a leaky
 abstraction, a data structure or composed allocators may desire to
 control the attributes to reduce GC pressure.
These attributes seem to be informed by the types stored, which would be above the charter of untyped allocator. Andrei
The attributes are informed by whatever code is calling the GC, the GC interface deals in void*'s. Consider an AA implementation that wishes to use FancyAllocator with fallback GCAllocator with block attributes NO_INTERIOR and NO_SCAN. With your proposed GCAllocator you either need to rewrite GCAllocator, or you need to add some nasty code to set the attributes depending on whether the primary allocator or secondary allocator own the memory. By fixing the leaky abstraction this use case can be coded as follows: FallbackAllocator!(FancyAllocator, GCAllocator!(GC.BLkAttr.NO_INTERIOR | GC.BLkAttr.NO_SCAN)) a;
Migrating the flags into the type is a possibility but maybe it's easiest to add flags as runtime parameters. Allocators can always define additional nonstandard routines. The standard routines concern mostly composition. Of course, it is also possible to make such flags standard (it may be the case that typed allocators require such). Andrei
Nov 01 2013
parent "safety0ff" <safety0ff.dev gmail.com> writes:
On Friday, 1 November 2013 at 15:15:10 UTC, Andrei Alexandrescu 
wrote:
 Migrating the flags into the type is a possibility but maybe 
 it's easiest to add flags as runtime parameters.
I was using that as an example to argue for its inclusion because it was concise. I'm not sure what the best interface would be, I'd have to think about it for a while.
Nov 01 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
Added SharedFreelist, a lock-free freelist. Andrei
Oct 31 2013
next sibling parent Martin Nowak <code dawg.eu> writes:
On 11/01/2013 03:34 AM, Andrei Alexandrescu wrote:
 Added SharedFreelist, a lock-free freelist.




 Andrei
Nice, that reminds me that I still want to polish my implementation of a lock free doubly-linked list in D. It would be great to collect these efforts in a library. I remember the request of Adam Wilson for a System.Collections.Concurrent in D. I put my stuff on github https://github.com/dawgfoto/lock-free. The doubly-linked list is based on this paper. IIRC the paper had a bug. http://dx.doi.org/10.1016/j.jpdc.2008.03.001 Recently I also found a C++ implementation. It's much simpler in D due to the GC. https://github.com/Kometes/Honeycomb/blob/master/src/common/Honey/Thread/LockFree/List.h
Nov 01 2013
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/31/13 7:34 PM, Andrei Alexandrescu wrote:
 On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
Added SharedFreelist, a lock-free freelist.
Added documentation clarifying the allocators' approach to shared memory. See section "Allocating memory for sharing across threads" near the top of http://erdani.com/d/phobos-prerelease/std_allocator.html Andrei
Nov 02 2013
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
01-Nov-2013 06:34, Andrei Alexandrescu пишет:
 On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
Added SharedFreelist, a lock-free freelist.
Neat. However it's missing from "Synopsis of predefined allocator building blocks" table. -- Dmitry Olshansky
Nov 02 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/2/13 1:15 PM, Dmitry Olshansky wrote:
 01-Nov-2013 06:34, Andrei Alexandrescu пишет:
 On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test
 drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
Added SharedFreelist, a lock-free freelist.
Neat. However it's missing from "Synopsis of predefined allocator building blocks" table.
Thanks, fixed and pushed. Andrei
Nov 02 2013
parent reply "Meta" <jared771 gmail.com> writes:
 From the "Allocating memory for sharing across threads" section, 
this paragraph:

"All allocators in this module accept and return void[] **(as 
opposed to )**. This is because at the time of allocation, 
deallocation, or reallocation, the memory is effectively not 
shared (**it** it were, it would be a bug at the application 
level)."

Starred items appear to be typos or missing a piece.
Nov 02 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/2/13 1:39 PM, Meta wrote:
  From the "Allocating memory for sharing across threads" section, this
 paragraph:

 "All allocators in this module accept and return void[] **(as opposed to
 )**. This is because at the time of allocation, deallocation, or
 reallocation, the memory is effectively not shared (**it** it were, it
 would be a bug at the application level)."

 Starred items appear to be typos or missing a piece.
Ew. Thanks, fixed and pushed. Andrei
Nov 02 2013
parent reply "Temtaime" <temtaime gmail.com> writes:
Hi, Andrei.

I cannot compile the allocator.

allocator.d(799): Error: undefined identifier returned

At 499 line:
returned alignedReallocate(b, newSize, alignment);

Ur mistake?
Nov 03 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/3/13 4:14 AM, Temtaime wrote:
 Hi, Andrei.

 I cannot compile the allocator.

 allocator.d(799): Error: undefined identifier returned

 At 499 line:
 returned alignedReallocate(b, newSize, alignment);

 Ur mistake?
Yes, mine. I don't have access to Windows so I expect a bunch of typos in the Windows-specific code. Fixed and pushed this one. Thanks, Andrei
Nov 03 2013
prev sibling parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Friday, 1 November 2013 at 02:33:57 UTC, Andrei Alexandrescu 
wrote:
 Added SharedFreelist, a lock-free freelist.




 Andrei
Hi Andrei, Please check this bug fix for SharedFreelist https://github.com/andralex/phobos/pull/21 . I have found that source code for bounded `maxNodes` version of SharedFreelist is commented out. I understand that `maxNodes` can be only approximate bound for _shared_ free list. However, approximate `maxNodes` bound is very useful too. Can I create PR for this feature? Best Regards, --Ilya
Sep 10 2015
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
I have another request despite putting this in it's own repo.

Often one want an exponentially (power of 2) growing step size for 
Bucketizer. Currently only equally spaced buckets are possible which 
isn't very practical to scale from 16b to 2Mb.

Nov 01 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/1/13 1:36 PM, Martin Nowak wrote:
 I have another request despite putting this in it's own repo.
I assume s/despite/in addition to/ :o).
 Often one want an exponentially (power of 2) growing step size for
 Bucketizer. Currently only equally spaced buckets are possible which
 isn't very practical to scale from 16b to 2Mb.

I considered the growth strategy as a policy. My personal favorite is "choose an approximate exponential growth strategy that keeps maximum internal fragmentation less than x%." That's how jemalloc is dimensioned. I decided to stick with linear at least for now, for a simple reason: it's easy enough to simply enumerate the strategy by hand by using Segregator. Exponentials quickly grow to span a bunch of memory, so there aren't a lot of terms involved. Nevertheless it would be a nice illustration of D's generative powers. Andrei
Nov 01 2013
parent reply "Dicebot" <public dicebot.lv> writes:
 That makes me think probably Phobos should have a package.json 
 so people
can install updates via code.dlang.org. How would you imagine that? Not-yet-in-Phobos packages are expected to be submitted as standalone ones, not as whole Phobos repo.
Nov 01 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/1/13 2:12 PM, Dicebot wrote:
 That makes me think probably Phobos should have a package.json so people
can install updates via code.dlang.org. How would you imagine that? Not-yet-in-Phobos packages are expected to be submitted as standalone ones, not as whole Phobos repo.
I meant people who'd want to just install Phobos from master. Probably not a good idea because Phobos changes often are in sync with compiler changes. Andrei
Nov 01 2013
prev sibling next sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
How about a stack allocator like this:
----
enum StackSize = 8192;

struct Stack {
	static Stack it;
	
	void[StackSize] _buffer = void;
	size_t _bufUsage;

	void[] take(size_t N) {
		if ((this._bufUsage + N) <= StackSize) {
			scope(exit) this._bufUsage += N;

			return _buffer[this._bufUsage .. this._bufUsage + N];
		}

		return null;
	}
	
	void reset() {
		this._bufUsage = 0;
	}
}
----
Would that fit in std.allocator?
Nov 07 2013
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 07.11.2013 11:32, schrieb Namespace:
 How about a stack allocator like this:
 ----
 enum StackSize = 8192;

 struct Stack {
      static Stack it;

      void[StackSize] _buffer = void;
      size_t _bufUsage;

      void[] take(size_t N) {
          if ((this._bufUsage + N) <= StackSize) {
              scope(exit) this._bufUsage += N;

              return _buffer[this._bufUsage .. this._bufUsage + N];
          }

          return null;
      }

      void reset() {
          this._bufUsage = 0;
      }
 }
 ----
 Would that fit in std.allocator?
That's std.allocator.InSituRegion, just that is misses the reset() method.
Nov 07 2013
next sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Thursday, 7 November 2013 at 13:15:03 UTC, Sönke Ludwig wrote:
 Am 07.11.2013 11:32, schrieb Namespace:
 How about a stack allocator like this:
 ----
 enum StackSize = 8192;

 struct Stack {
     static Stack it;

     void[StackSize] _buffer = void;
     size_t _bufUsage;

     void[] take(size_t N) {
         if ((this._bufUsage + N) <= StackSize) {
             scope(exit) this._bufUsage += N;

             return _buffer[this._bufUsage .. this._bufUsage + 
 N];
         }

         return null;
     }

     void reset() {
         this._bufUsage = 0;
     }
 }
 ----
 Would that fit in std.allocator?
That's std.allocator.InSituRegion, just that is misses the reset() method.
Nice! But I suggest two things: 1. Add a reset function for reusing the same storage 2. Should that: https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2907 not be replaced with ubyte[size] _store = void; ?
Nov 07 2013
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Nov-2013 17:30, Namespace пишет:
 On Thursday, 7 November 2013 at 13:15:03 UTC, Sönke Ludwig wrote:
 Am 07.11.2013 11:32, schrieb Namespace:
 How about a stack allocator like this:
 ----
 enum StackSize = 8192;

 struct Stack {
     static Stack it;

     void[StackSize] _buffer = void;
     size_t _bufUsage;

     void[] take(size_t N) {
         if ((this._bufUsage + N) <= StackSize) {
             scope(exit) this._bufUsage += N;

             return _buffer[this._bufUsage .. this._bufUsage + N];
         }

         return null;
     }

     void reset() {
         this._bufUsage = 0;
     }
 }
 ----
 Would that fit in std.allocator?
That's std.allocator.InSituRegion, just that is misses the reset() method.
Nice! But I suggest two things: 1. Add a reset function for reusing the same storage 2. Should that: https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2907 not be replaced with ubyte[size] _store = void; ?
Hm... Andrei has just been caught using internal pointers in structs :o) -- Dmitry Olshansky
Nov 07 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/7/13 5:30 AM, Namespace wrote:
 Nice! But I suggest two things:
   1. Add a reset function for reusing the same storage
freeAll().
   2. Should that:
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2907
 not be replaced with ubyte[size] _store = void; ?
Made the change and pushed, thanks. Unfortunately there's a performance bug in the front-end (if at least one field has initialization, all fields are initialized by bitblasting .init over the object). https://d.puremagic.com/issues/show_bug.cgi?id=11331 Andrei
Nov 07 2013
parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Thursday, 7 November 2013 at 16:28:42 UTC, Andrei Alexandrescu 
wrote:
 On 11/7/13 5:30 AM, Namespace wrote:
 Nice! But I suggest two things:
  1. Add a reset function for reusing the same storage
freeAll().
There is no freeAll. Did you mean deallocateAll? ---- void deallocateAll() { _crt = null; } ---- _crt is null and the same storage can't be used again.
Nov 07 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/7/13 9:02 AM, Namespace wrote:
 On Thursday, 7 November 2013 at 16:28:42 UTC, Andrei Alexandrescu wrote:
 On 11/7/13 5:30 AM, Namespace wrote:
 Nice! But I suggest two things:
  1. Add a reset function for reusing the same storage
freeAll().
There is no freeAll. Did you mean deallocateAll? ---- void deallocateAll() { _crt = null; } ---- _crt is null and the same storage can't be used again.
Sorry that's indeed the name and there's a bug in it! Andrei
Nov 07 2013
parent reply "Jack Applegame" <japplegame gmail.com> writes:
Passing -g flag to the linker causes a linker error on win32.
Very simple program:

import std.allocator;
void main() {}

Compilation commands:

dmd.exe -g -debug -c std/allocator.d -ofallocator.obj
dmd.exe -g -debug -c main.d -ofmain.obj
dmd.exe out.exe -g allocator.obj main.obj

Output:

dmd.exe out.exe -g allocator.obj main.obj
OPTLINK (R) for Win32  Release 8.00.13
Copyright (C) Digital Mars 1989-2010  All rights reserved.
http://www.digitalmars.com/ctg/optlink.html
allocator.obj(allocator)
  Error 42: Symbol Undefined __aligned_malloc
allocator.obj(allocator)
  Error 42: Symbol Undefined __aligned_free
allocator.obj(allocator)
  Error 42: Symbol Undefined __aligned_realloc
--- errorlevel 3

Without -g flag on the last command (dmd.exe out.exe 
allocator.obj main.obj) no linker error appears.
Nov 16 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/16/13 2:50 AM, Jack Applegame wrote:
 Passing -g flag to the linker causes a linker error on win32.
[snip] Maybe this would help? http://community.rti.com/kb/why-does-visual-studio-complain-about-alignedmalloc-and-alignedfree Andrei
Nov 16 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/7/13 5:15 AM, Sönke Ludwig wrote:
 That's std.allocator.InSituRegion, just that is misses the reset() method.
freeAll should take care of it. Andrei
Nov 07 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/7/13 2:32 AM, Namespace wrote:
 How about a stack allocator like this:
 ----
 enum StackSize = 8192;

 struct Stack {
      static Stack it;

      void[StackSize] _buffer = void;
      size_t _bufUsage;

      void[] take(size_t N) {
          if ((this._bufUsage + N) <= StackSize) {
              scope(exit) this._bufUsage += N;

              return _buffer[this._bufUsage .. this._bufUsage + N];
          }

          return null;
      }

      void reset() {
          this._bufUsage = 0;
      }
 }
 ----
 Would that fit in std.allocator?
It's there! Andrei
Nov 07 2013
prev sibling next sibling parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
Awesome!

Since we have SIMD instructions in core it will be nice to have 
AlignedGCallocator.

Best Regards,
Ilya
Nov 07 2013
prev sibling next sibling parent "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there 
 are no benchmarks. Both would be appreciated, particularly 
 benchmarks to validate the gains (which I speculate can be very 
 sizable) of custom-built, special-purpose allocators compared 
 to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this 
 design. I have been knocking around it for long enough to have 
 no idea how easy it is to get into it from the outside, or how 
 good it is. By all signs I could gather this feels like good 
 design, and one of the best I've ever put together. The 
 allocators defined have an archetypal feeling, are flexible 
 both statically and dynamically, and morph and combine in 
 infinite ways.

 CAllocator and CAllocatorImpl make the link between the static 
 and dynamic worlds. Once an allocator is assembled out of 
 pieces and finely tuned, wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei
All I can say is - I can't wait to see it in the run-time library! I read about the jmalloc last year, and your work makes it possible to have something similar in D. Kudos! System developers will definitely like std.allocator, no doubt! :)
Nov 07 2013
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
Andrei, what is current state of std.allocator? I am asking this 
in context of recent Walter Phobos proposal (ScopeBuffer) to 
evaluate how feasible is to define any relations between two at 
current stage.
Feb 17 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking this in
 context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how
 feasible is to define any relations between two at current stage.
Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei
Feb 17 2014
parent reply "ANtlord" <antlord92 gmail.com> writes:
On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu 
wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking 
 this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.
Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei
Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you.
Feb 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking this in
 context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how
 feasible is to define any relations between two at current stage.
Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei
Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you.
Interface with garbage collector is to be done. The rest is usable right now. -- Andrei
Feb 11 2015
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei 
Alexandrescu wrote:
 On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei 
 Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking 
 this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.
Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei
Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you.
Interface with garbage collector is to be done. The rest is usable right now. -- Andrei
Should it be in std.experimental? Or at least on code.dlang.org?
Feb 11 2015
parent "MrSmith" <mrsmith33 yandex.ru> writes:
On Wednesday, 11 February 2015 at 15:57:26 UTC, John Colvin wrote:
 Should it be in std.experimental? Or at least on code.dlang.org?
Yeah, dub package would be really nice!
Feb 11 2015
prev sibling parent reply "ANtlord" <antlord92 gmail.com> writes:
On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei 
Alexandrescu wrote:
 On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei 
 Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking 
 this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.
Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei
Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you.
Interface with garbage collector is to be done. The rest is usable right now. -- Andrei
I thought, that general purpose of allocator it is memory management without garbage collector. Because allocator will be used in containers, and they must work without garbage collector. If no, what is difference between containers and arrays in current implementation of D? I assumed, that containers are need for increased performance within saving safety code. But a using GC will not increase performance. Or Am I dead wrong?
Feb 12 2015
parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 12 February 2015 at 08:56:53 UTC, ANtlord wrote:
 On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei 
 Alexandrescu wrote:
 On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei 
 Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking 
 this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.
Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei
Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you.
Interface with garbage collector is to be done. The rest is usable right now. -- Andrei
I thought, that general purpose of allocator it is memory management without garbage collector. Because allocator will be used in containers, and they must work without garbage collector. If no, what is difference between containers and arrays in current implementation of D? I assumed, that containers are need for increased performance within saving safety code. But a using GC will not increase performance. Or Am I dead wrong?
You still need to cooperate with the GC and let it know which memory should be hands off, for example. Or just use it for the initial allocation and then ask it politely to release ownership of it. -- Paulo
Feb 12 2015
parent reply "ANtlord" <antlord92 gmail.com> writes:
On Thursday, 12 February 2015 at 11:10:28 UTC, Paulo  Pinto wrote:
 On Thursday, 12 February 2015 at 08:56:53 UTC, ANtlord wrote:
 On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei 
 Alexandrescu wrote:
 On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei 
 Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am 
 asking this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.
Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei
Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you.
Interface with garbage collector is to be done. The rest is usable right now. -- Andrei
I thought, that general purpose of allocator it is memory management without garbage collector. Because allocator will be used in containers, and they must work without garbage collector. If no, what is difference between containers and arrays in current implementation of D? I assumed, that containers are need for increased performance within saving safety code. But a using GC will not increase performance. Or Am I dead wrong?
You still need to cooperate with the GC and let it know which memory should be hands off, for example. Or just use it for the initial allocation and then ask it politely to release ownership of it. -- Paulo
Aaa, thank you for explanation. I've got it. If the rest is usable right now and interface with GC. Why is not this module in std? Is it in queue for review?
Feb 13 2015
parent reply "Piotrek" <no_data no_data.pl> writes:
Hi,

Sorry for putting it here but I don't know where to file a bug 
report for the allocator project.

On 32-bit linux with the latest dmd beta I get errors for ulong 
-> uint (size_t) conversions.

dmd -main -unittest allocator.d

allocator.d(2015): Error: cannot implicitly convert expression (i 
* 4096LU) of type ulong to uint
allocator.d(2015): Error: cannot implicitly convert expression 
((i + cast(ulong)blocks) * 4096LU) of type ulong to uint
allocator.d(1743): Error: template instance 
std.allocator.HeapBlock!(4096u, 4u) <cut the long line>
allocator.d(331):        instantiated from here: 
HeapBlock!(4096u, 4u)
allocator.d(334): Error: template instance Segregator! <cut the 
long line>
allocator.d(2015): Error: cannot implicitly convert expression (i 
* 128LU) of type ulong to uint
allocator.d(2015): Error: cannot implicitly convert expression 
((i + cast(ulong)blocks) * 128LU) of type ulong to uint
allocator.d(1743): Error: template instance 
std.allocator.HeapBlock!(128u, 4u) <cut the long line>
  , __ctmp2303).this(m)) error instantiating
allocator.d(1342):        instantiated from here: 
HeapBlock!(128u, 4u)
allocator.d(1493): Error: cannot implicitly convert expression (x 
/ 64LU) of type ulong to immutable(uint)
allocator.d(1495): Error: cannot implicitly convert expression (y 
/ 64LU) of type ulong to immutable(uint)
allocator.d(1520): Error: cannot implicitly convert expression (x 
/ 64LU) of type ulong to uint
allocator.d(1526): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1527): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1544): Error: cannot implicitly convert expression 
(w) of type ulong to uint
allocator.d(1553): Error: cannot implicitly convert expression 
(w) of type ulong to uint
allocator.d(1572): Error: cannot implicitly convert expression 
(w) of type ulong to uint
allocator.d(1582): Error: cannot implicitly convert expression 
(w) of type ulong to uint
allocator.d(1607): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1615): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1627): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1633): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(4143): Error: function 
std.allocator.roundUpToMultipleOf (uint s, uint base) is not 
callable using argument types (ulong, uint)


Is it a known issue? Or are there currently only 64-bit OSes 
supported by the allocator project?

Piotrek
Feb 26 2015
parent reply "ANtlord" <antlord92 gmail.com> writes:
On Thursday, 26 February 2015 at 21:01:27 UTC, Piotrek wrote:
 Hi,

 Sorry for putting it here but I don't know where to file a bug 
 report for the allocator project.

 On 32-bit linux with the latest dmd beta I get errors for ulong 
 -> uint (size_t) conversions.

 dmd -main -unittest allocator.d

 allocator.d(2015): Error: cannot implicitly convert expression 
 (i * 4096LU) of type ulong to uint
 allocator.d(2015): Error: cannot implicitly convert expression 
 ((i + cast(ulong)blocks) * 4096LU) of type ulong to uint
 allocator.d(1743): Error: template instance 
 std.allocator.HeapBlock!(4096u, 4u) <cut the long line>
 allocator.d(331):        instantiated from here: 
 HeapBlock!(4096u, 4u)
 allocator.d(334): Error: template instance Segregator! <cut the 
 long line>
 allocator.d(2015): Error: cannot implicitly convert expression 
 (i * 128LU) of type ulong to uint
 allocator.d(2015): Error: cannot implicitly convert expression 
 ((i + cast(ulong)blocks) * 128LU) of type ulong to uint
 allocator.d(1743): Error: template instance 
 std.allocator.HeapBlock!(128u, 4u) <cut the long line>
  , __ctmp2303).this(m)) error instantiating
 allocator.d(1342):        instantiated from here: 
 HeapBlock!(128u, 4u)
 allocator.d(1493): Error: cannot implicitly convert expression 
 (x / 64LU) of type ulong to immutable(uint)
 allocator.d(1495): Error: cannot implicitly convert expression 
 (y / 64LU) of type ulong to immutable(uint)
 allocator.d(1520): Error: cannot implicitly convert expression 
 (x / 64LU) of type ulong to uint
 allocator.d(1526): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1527): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1544): Error: cannot implicitly convert expression 
 (w) of type ulong to uint
 allocator.d(1553): Error: cannot implicitly convert expression 
 (w) of type ulong to uint
 allocator.d(1572): Error: cannot implicitly convert expression 
 (w) of type ulong to uint
 allocator.d(1582): Error: cannot implicitly convert expression 
 (w) of type ulong to uint
 allocator.d(1607): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1615): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1627): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1633): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(4143): Error: function 
 std.allocator.roundUpToMultipleOf (uint s, uint base) is not 
 callable using argument types (ulong, uint)


 Is it a known issue? Or are there currently only 64-bit OSes 
 supported by the allocator project?

 Piotrek
I think, that if use this project https://github.com/andralex/std_allocator/, than you can post the issue to related issue tracker. And I see, that types in traceback are different from source https://github.com/andralex/std_allocator/blob/master/source/s d/allocator.d#L857. Maybe you need to upgrade package. Sorry, if my english isn't clear.
Feb 27 2015
parent "Piotrek" <no_data no_data.pl> writes:
On Friday, 27 February 2015 at 08:18:53 UTC, ANtlord wrote:
 I think, that if use this project 
 https://github.com/andralex/std_allocator/, than you can post 
 the issue to related issue tracker.
Oh, I must be blind. I thought the issue tracker was disables on the repository in the same way as for Phobos. Thanks for checking. I submitted the issue.
 And I see, that types in traceback are different from source 
 https://github.com/andralex/std_allocator/blob/master/source/s
d/allocator.d#L857. 
 Maybe you need to upgrade package.
If you mean size_t and uint difference it's because size_t is an alias for uint on 32-bit os. roundUpToMultipleOf(size_t s, uint base) becomes roundUpToMultipleOf (uint s, uint base) Piotrek
Feb 27 2015
prev sibling parent reply "bitwise" <bitwise.pvt gmail.com> writes:
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
Am I the only one seeing dead links?
Sep 10 2015
parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
Am 11.09.2015 um 04:35 schrieb bitwise:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
Am I the only one seeing dead links?
The code has moved to std.experimental: https://github.com/andralex/phobos/tree/allocator/std/experimental/allocator http://erdani.com/d/phobos-prerelease/std_experimental_allocator.html
Sep 11 2015