www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - how to allocate large number of small object?

reply newbie d.com writes:
I need to allocate a large number (>10K) of small objects.  

class Small {
some field ...
}

Right now I call "new Small()" for each allocation, the overall performance of
the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.

char *pool = new char[ N * Small.size_of ];
char *next = pool;

Small myNew() {
Small obj = cast(Small)next;
next += Small.size_of;
}

is there any problem with it?  What's the correct way to do this in D?

Thanks.
Oct 04 2005
next sibling parent reply xs0 <xs0 xs0.com> writes:
newbie d.com wrote:
 I need to allocate a large number (>10K) of small objects.  
 
 class Small {
 some field ...
 }
 
 Right now I call "new Small()" for each allocation, the overall performance of
 the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.
 
 char *pool = new char[ N * Small.size_of ];
 char *next = pool;
 
 Small myNew() {
 Small obj = cast(Small)next;
 next += Small.size_of;
 }
 
 is there any problem with it?  What's the correct way to do this in D?
 
 Thanks.

You can write a custom allocator without any "hacks": http://www.digitalmars.com/d/memory.html#newdelete BTW, I think it should be Small.init.sizeof, Small.sizeof returns the size of a reference, not an instance.. xs0
Oct 05 2005
next sibling parent Sean Kelly <sean f4.ca> writes:
In article <di05kl$a40$1 digitaldaemon.com>, xs0 says...
BTW, I think it should be Small.init.sizeof, Small.sizeof returns the 
size of a reference, not an instance..

Really? But Small is the type name. Classes are just a special case all around it seems. Sean
Oct 05 2005
prev sibling parent "Ben Hinkle" <bhinkle mathworks.com> writes:
 BTW, I think it should be Small.init.sizeof, Small.sizeof returns the size 
 of a reference, not an instance..

I think you mean Small.classinfo.init.length Small.init.sizeof == Small.sizeof
Oct 05 2005
prev sibling next sibling parent pragma <pragma_member pathlink.com> writes:
In article <dhvr8g$vg$1 digitaldaemon.com>, newbie d.com says...
I need to allocate a large number (>10K) of small objects.  

class Small {
some field ...
}

Right now I call "new Small()" for each allocation, the overall performance of
the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.

char *pool = new char[ N * Small.size_of ];
char *next = pool;

Small myNew() {
Small obj = cast(Small)next;
next += Small.size_of;
}

is there any problem with it?  What's the correct way to do this in D?

Another solution apart from what has already been suggested:
 const uint POOL_SIZE = 1024*1024*5; // 5MB
 ubyte[POOL_SIZE] pool = void;
 ubyte* next = pool.ptr;
 Small obj = cast(Small)next;
 next += Small.size_of;

The above will get you a 5MB pool of memory, allocated to a ubyte array (which is slightly better form than char[]). The 'void' initalizer instructs the compiler to not initalize the entire array to zero, which can cost you time for big chunks of memory. - EricAnderton at yahoo
Oct 05 2005
prev sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
<newbie d.com> wrote in message news:dhvr8g$vg$1 digitaldaemon.com...
I need to allocate a large number (>10K) of small objects.

 class Small {
 some field ...
 }

 Right now I call "new Small()" for each allocation, the overall 
 performance of
 the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.

 char *pool = new char[ N * Small.size_of ];
 char *next = pool;

 Small myNew() {
 Small obj = cast(Small)next;
 next += Small.size_of;
 }

 is there any problem with it?  What's the correct way to do this in D?

 Thanks.

Be careful about casting a random pointer to a Small. The resulting "object" won't have a vtable or monitor or be initialized so you won't be able to call any methods or synchronize on the object. If you don't care about either of those I suggesting making Small a struct.
Oct 05 2005
next sibling parent Sean Kelly <sean f4.ca> writes:
In article <di0m9b$p23$1 digitaldaemon.com>, Ben Hinkle says...
<newbie d.com> wrote in message news:dhvr8g$vg$1 digitaldaemon.com...
I need to allocate a large number (>10K) of small objects.

 class Small {
 some field ...
 }

 Right now I call "new Small()" for each allocation, the overall 
 performance of
 the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.

 char *pool = new char[ N * Small.size_of ];
 char *next = pool;

 Small myNew() {
 Small obj = cast(Small)next;
 next += Small.size_of;
 }

 is there any problem with it?  What's the correct way to do this in D?

 Thanks.

Be careful about casting a random pointer to a Small. The resulting "object" won't have a vtable or monitor or be initialized so you won't be able to call any methods or synchronize on the object. If you don't care about either of those I suggesting making Small a struct.

It should be simple enough to do a placement new on the memory block though... Sean
Oct 05 2005
prev sibling parent reply newbie d.com writes:
In article <di0m9b$p23$1 digitaldaemon.com>, Ben Hinkle says...
Be careful about casting a random pointer to a Small. The resulting "object" 
won't have a vtable or monitor or be initialized so you won't be able to 
call any methods or synchronize on the object. If you don't care about 
either of those I suggesting making Small a struct. 

I cannot use struct. So under this constraint, can I do something as I showed earlier? And what's the best way to make sure the *object* I got from the pool is properly initialized? In article <di05kl$a40$1 digitaldaemon.com>, xs0 says...
You can write a custom allocator without any "hacks":
http://www.digitalmars.com/d/memory.html#newdelete

I checked that page, same question: what's the magic to make sure the *object* is properly initialize? new() { p = std.c.stdlib.malloc(sz); gc.addRange(p, p + sz); return p; } Thanks.
Oct 05 2005
parent "Ben Hinkle" <ben.hinkle gmail.com> writes:
 I checked that page, same question: what's the magic to make sure the 
 *object*
 is properly initialize?

 new() {
 p = std.c.stdlib.malloc(sz);
 gc.addRange(p, p + sz);
 return p;
 }

I haven't had to ever use custom allocators so I haven't tried it myself but presumably from the description on http://www.digitalmars.com/d/class.html#allocators the result of new() is initialized.
Oct 05 2005