www.digitalmars.com         C & C++   DMDScript  

D - dynamic array problems

reply "Alex A. B." <freshmind fromru.com> writes:
I have numerous problems with the program I wrote to test and benchmark D's
dynamic arrays.
The source code is attached. It compiles with D 0.79 and D 0.75(with minor
modification).

What it tries to do is to read file contents and fill dynamic array of
arrays(each of random length(1-20 bytes),
then reassemble the data back and write it to another file.

It seems to work Ok, but only on small(really small) files. On my system
under Win98 it
stops working with files bigger than 50kb. ;) Under XP it manages to work
with any size
I tried, but consuming catastrophic amounts of memory(up to 90mb per 800kb
file!) and time.
Of course program does nothing useful. ;) But it was intended to be
benchmark for memory
allocation and management speed.

Just out of curiousity I have implemented the same program in
Prolog(SWI-Prolog) and ran
it on the same files. Well the speed was just about the same and memory
consumption was
twice as small. This is unbelievable results, because prolog is fully
interpreted, linked-lists
are used instead of arrays and even file ops are done on element-by-element
basis
through tail recursion!

Probably my code is wrong somewhere or D's run-time is doing very bad job.
Please prove me wrong, because I was(and still) going to use D extensively.
Jan 29 2004
parent reply Paul Runde <prunde consolidated.net> writes:
Alex A. B. wrote:
 I have numerous problems with the program I wrote to test and benchmark D's
 dynamic arrays.
 The source code is attached. It compiles with D 0.79 and D 0.75(with minor
 modification).
 
 What it tries to do is to read file contents and fill dynamic array of
 arrays(each of random length(1-20 bytes),
 then reassemble the data back and write it to another file.
 
 It seems to work Ok, but only on small(really small) files. On my system
 under Win98 it
 stops working with files bigger than 50kb. ;) Under XP it manages to work
 with any size
 I tried, but consuming catastrophic amounts of memory(up to 90mb per 800kb
 file!) and time.
 Of course program does nothing useful. ;) But it was intended to be
 benchmark for memory
 allocation and management speed.
 
 Just out of curiousity I have implemented the same program in
 Prolog(SWI-Prolog) and ran
 it on the same files. Well the speed was just about the same and memory
 consumption was
 twice as small. This is unbelievable results, because prolog is fully
 interpreted, linked-lists
 are used instead of arrays and even file ops are done on element-by-element
 basis
 through tail recursion!
 
 Probably my code is wrong somewhere or D's run-time is doing very bad job.
 Please prove me wrong, because I was(and still) going to use D extensively.
 
 

This is probably the memory allocation issue that been reported recently. Here is the fix originally suggested by yaneurao. In std.thread add foward declaration: extern(Windows) export thread_hdl GetCurrentProcess(); In Thread.getCurrentThreadHandle() in std.thread change the line: thread_hdl currentProcess = cast(thread_hdl)-1; to thread_hdl currentProcess = GetCurrentProcess(); Then rebuild phobos.lib This fixed the problems I was encountering. Paul
Jan 29 2004
parent reply "Ben Hinkle" <bhinkle4 juno.com> writes:
Anyone else know what is going on with this? Has someone tried it who
recompiled phobos? I'd like to know if the GC is really sucking up this much
memory.
thanks,
-Ben

"Paul Runde" <prunde consolidated.net> wrote in message
news:bvb1v6$1l8o$1 digitaldaemon.com...
 Alex A. B. wrote:
 I have numerous problems with the program I wrote to test and benchmark


 dynamic arrays.
 The source code is attached. It compiles with D 0.79 and D 0.75(with


 modification).

 What it tries to do is to read file contents and fill dynamic array of
 arrays(each of random length(1-20 bytes),
 then reassemble the data back and write it to another file.

 It seems to work Ok, but only on small(really small) files. On my system
 under Win98 it
 stops working with files bigger than 50kb. ;) Under XP it manages to


 with any size
 I tried, but consuming catastrophic amounts of memory(up to 90mb per


 file!) and time.
 Of course program does nothing useful. ;) But it was intended to be
 benchmark for memory
 allocation and management speed.

 Just out of curiousity I have implemented the same program in
 Prolog(SWI-Prolog) and ran
 it on the same files. Well the speed was just about the same and memory
 consumption was
 twice as small. This is unbelievable results, because prolog is fully
 interpreted, linked-lists
 are used instead of arrays and even file ops are done on


 basis
 through tail recursion!

 Probably my code is wrong somewhere or D's run-time is doing very bad


 Please prove me wrong, because I was(and still) going to use D



This is probably the memory allocation issue that been reported recently. Here is the fix originally suggested by yaneurao. In std.thread add foward declaration: extern(Windows) export thread_hdl GetCurrentProcess(); In Thread.getCurrentThreadHandle() in std.thread change the line: thread_hdl currentProcess = cast(thread_hdl)-1; to thread_hdl currentProcess = GetCurrentProcess(); Then rebuild phobos.lib This fixed the problems I was encountering. Paul

Jan 29 2004
parent reply "Alex A. B." <freshmind fromru.com> writes:
Ben Hinkle <bhinkle4 juno.com> сообщил в новостях
следующее:bvbsbv$90$1 digitaldaemon.com...
 Anyone else know what is going on with this? Has someone tried it who
 recompiled phobos? I'd like to know if the GC is really sucking up this

 memory.
 thanks,
 -Ben

allocation problem with Windows 98 was gone, it still does consume HUGE amounts of memory. Even SWI-Prologs' GC does a much better job at the same task.
Jan 29 2004
next sibling parent reply "Matthew" <matthew.hat stlsoft.dot.org> writes:
Maybe it's time to consider that collecting background thread?

"Alex A. B." <freshmind fromru.com> wrote in message
news:bvc9eo$lj8$1 digitaldaemon.com...
 Ben Hinkle <bhinkle4 juno.com> сообщил в новостях
 следующее:bvbsbv$90$1 digitaldaemon.com...
 Anyone else know what is going on with this? Has someone tried it who
 recompiled phobos? I'd like to know if the GC is really sucking up this

 memory.
 thanks,
 -Ben

allocation problem with Windows 98 was gone, it still does consume HUGE amounts of memory. Even SWI-Prologs' GC does a much better job at the same task.

Jan 29 2004
parent "Ben Hinkle" <bhinkle4 juno.com> writes:
Glancing over the test code it doesn't look like it is doing much
collecting, only lots and lots of small allocation. There are some reallocs
being called by ~= but in general it looks like there are small (1 to 20
bytes) chunks and few 800K chunks (for an 800K input). Is it overhead on
small objects? Is it poor recylcing of realloced arrays? I notice the last
loop as it catenates all the small chunks back together the GC never has to
alloc a small chunk - it is always resizing the output array to something
larger so if the old array was just put on the free list it would never get
reused).
Does the GC ever merge free'd blocks? I can't tell from glancing at the
gcx.d what is causing this code to perform like it is. It would be a pity to
get D 1.0 and have it dismissed because of GC corner cases that make it look
bad.

-Ben

"Matthew" <matthew.hat stlsoft.dot.org> wrote in message
news:bvcaoe$nri$1 digitaldaemon.com...
 Maybe it's time to consider that collecting background thread?

 "Alex A. B." <freshmind fromru.com> wrote in message
 news:bvc9eo$lj8$1 digitaldaemon.com...
 Ben Hinkle <bhinkle4 juno.com> сообщил в новостях
 следующее:bvbsbv$90$1 digitaldaemon.com...
 Anyone else know what is going on with this? Has someone tried it who
 recompiled phobos? I'd like to know if the GC is really sucking up



 much
 memory.
 thanks,
 -Ben

allocation problem with Windows 98 was gone, it still does consume HUGE amounts of memory. Even SWI-Prologs' GC does a much better job at the same task.


Jan 29 2004
prev sibling parent "Ben Hinkle" <bhinkle4 juno.com> writes:
"Alex A. B." <freshmind fromru.com> wrote in message
news:bvc9eo$lj8$1 digitaldaemon.com...
 Ben Hinkle <bhinkle4 juno.com> сообщил в новостях
 следующее:bvbsbv$90$1 digitaldaemon.com...
 Anyone else know what is going on with this? Has someone tried it who
 recompiled phobos? I'd like to know if the GC is really sucking up this

 memory.
 thanks,
 -Ben

allocation problem with Windows 98 was gone, it still does consume HUGE amounts of memory. Even SWI-Prologs' GC does a much better job at the same task.

I've attached a small program to ferret out some GC behaviors. Playing around you'll see that 1) the smallest allocable chunks come in sizes 16, 32, 64 etc bytes 2) resizing an array dynamically one element at a time is very inefficient. I think these two factors contribute to the memory usage you saw with your test program. I haven't tried modifying your code exactly, but see what happens when you replace the "rand()%20+1" in your program with numbers around 16 or 32. Also try getting rid of the "~=" and replace it with explicitly setting the length so some large numbers. At one point Walter recommended setting the length to a large number and then back to zero in order to "preallocate" space. This didn't completely fix the problem but it helped a little. -Ben
Jan 30 2004