www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Allocating many object

reply Frank Benoit <frank nix.de> writes:
class Test
{
    int[10] values;
}


void main()
{
    Test[1000_000] arr;
    printf( "sizeof Test : %d\n\x00", Test.classinfo.init.length );
    printf( "sizeof Array: %d\n\x00", arr.sizeof );
    for( int j = 0; j < 1000; j++ )
    {
        for( int i = 0; i < 1000; i++ )
        {
            arr[j*1000+i] = new Test;
        }
        printf( "loop %d\n\x00", j );
    }
}

The program runs and gets slower and slower.
After 8min at loop 700 I canceled.
Is this speed acceptable for allocating 48MB (+4MB for the array)?

I tried to use a XML DOM parser, parsing 1,5MB xml. This seems to take
endless.

Is there a workaround? e.g. preallocating memory in one piece.
Mar 27 2006
next sibling parent Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Frank Benoit wrote:
 class Test
 {
     int[10] values;
 }
 
 
 void main()
 {
     Test[1000_000] arr;
     printf( "sizeof Test : %d\n\x00", Test.classinfo.init.length );
     printf( "sizeof Array: %d\n\x00", arr.sizeof );
     for( int j = 0; j < 1000; j++ )
     {
         for( int i = 0; i < 1000; i++ )
         {
             arr[j*1000+i] = new Test;
         }
         printf( "loop %d\n\x00", j );
     }
 }
 
 The program runs and gets slower and slower.
 After 8min at loop 700 I canceled.
 Is this speed acceptable for allocating 48MB (+4MB for the array)?
 
 I tried to use a XML DOM parser, parsing 1,5MB xml. This seems to take
 endless.
 
 Is there a workaround? e.g. preallocating memory in one piece.

I used the following re-write of your program: # private import std.stdio; # private import cashew.utils.timer; # # class Test { # int[10] values ; # } # # void main () { # Test[100_000] arr; # writefln(r"sizeof Test : %d"c, Test.classinfo.init.length); # writefln(r"sizeof Array: %d"c, arr.sizeof); # { auto Timer t0 = new Timer(r"loop"c); # foreach(i, inout x; arr) { # x = new Test; # if (((i + 1) % 100) == 0) # writefln(r"loop %d"c, i + 1); # } # }//timer # } Do note I dropped the size of 'arr' from 1000000 to 100000. This has nothing to do with with the issue, so far as I know, and everything to do with a stack size problem on the machine I'm sitting at right now. Don't ask... pos. Anyhow, the Timer reports from about thirty or so runs of that program spanned between 0.66 seconds and 0.77 seconds total, so I shouldn't expect your original to take more than a second or two. -- Chris Nicholson-Sauls
Mar 27 2006
prev sibling next sibling parent reply Frank Benoit <frank nix.de> writes:
To see the difference to Java...

public class Alloc {

	int[] val;
	public Alloc()
	{
		val = new int[10];
	}
	
	public static void main(String[] args) {
		Alloc[] arr = new Alloc[1000000];
		for( int j = 0; j < 1000; j++ )
		{
	        	for( int i = 0; i < 1000; i++ )
	        	{
	            		arr[j*1000+i] = new Alloc();
	        	}
	        	System.out.println( "loop "+j );
	    	}
		System.out.println( "complete " );
	}
}


Note, this Java program make a second allocation for the array and does
also initialize the memory.

The program was finished after ONE second.
This 1000 times faster like the D program.

So I have to say that again... The GC is a show stopper and need to be
changed.
Mar 27 2006
parent Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Frank Benoit wrote:
 	int[] val;
 	public Alloc()
 	{
 		val = new int[10];
 	}

Just for the record, you can use the same construct above in D. In fact, you could technically do such silliness as: # auto val = new int[10]; Madness. But occasionally useful madness. -- Chris Nicholson-Sauls
Mar 27 2006
prev sibling next sibling parent reply Frank Benoit <frank nix.de> writes:
I am completely sorry.
I used a wrong verson of libphobos.
With the original one it works fast.

Frank
Mar 27 2006
parent reply Frank Benoit <frank nix.de> writes:
Frank Benoit schrieb:
 I am completely sorry.
 I used a wrong verson of libphobos.
 With the original one it works fast.
 
 Frank

BTW: The only difference was in the makefiles: original: DFLAGS = -O -inline -release My one : DFLAGS = -g -debug With the original options and an own compiled libphobos.a it is also completelly OK. ~500ms
Mar 27 2006
parent reply "Andrew Fedoniouk" <news terrainformatica.com> writes:
"Frank Benoit" <frank nix.de> wrote in message 
news:e09ju8$28nc$1 digitaldaemon.com...
 Frank Benoit schrieb:
 I am completely sorry.
 I used a wrong verson of libphobos.
 With the original one it works fast.

 Frank

BTW: The only difference was in the makefiles: original: DFLAGS = -O -inline -release My one : DFLAGS = -g -debug With the original options and an own compiled libphobos.a it is also completelly OK. ~500ms

By the way: SGML tokenizer (aka HTML/XML pull parser) in Harmonia is capable to scan 30mb XML file in one second. It depends on hardware but order of magnitude is like this. It does not allocate any memory while parsing. Andrew.
Mar 27 2006
parent reply ZZ <ZZ zz.com> writes:
Andrew Fedoniouk wrote:
 By the way:
 SGML tokenizer (aka HTML/XML pull parser) in Harmonia is
 capable to scan 30mb XML file in one second. It depends on hardware but
 order of magnitude is like this.
 It does not allocate any memory while parsing.
 
 Andrew. 
 
 

in a string. Do you plan on adding a file stream as input? Zz
Mar 28 2006
parent "Andrew Fedoniouk" <news terrainformatica.com> writes:
"ZZ" <ZZ zz.com> wrote in message news:e0ba8l$20k5$1 digitaldaemon.com...
 Andrew Fedoniouk wrote:
 By the way:
 SGML tokenizer (aka HTML/XML pull parser) in Harmonia is
 capable to scan 30mb XML file in one second. It depends on hardware but
 order of magnitude is like this.
 It does not allocate any memory while parsing.

 Andrew.

a string. Do you plan on adding a file stream as input?

If you need to scan file then you can use MMFile. module harmonia.io.mmfile; ( http://harmonia.terrainformatica.com ) or the one from phobos. For stream parsing you need to create your own scanner and override StreamScanner:Scanner { get_char() switch_input() currentFragment() (optional, error reporting) }
 Zz 

Mar 28 2006
prev sibling parent Derek Parnell <derek psych.ward> writes:
On Mon, 27 Mar 2006 21:05:55 +0200, Frank Benoit wrote:

 class Test
 {
     int[10] values;
 }
 
 void main()
 {
     Test[1000_000] arr;
     printf( "sizeof Test : %d\n\x00", Test.classinfo.init.length );
     printf( "sizeof Array: %d\n\x00", arr.sizeof );
     for( int j = 0; j < 1000; j++ )
     {
         for( int i = 0; i < 1000; i++ )
         {
             arr[j*1000+i] = new Test;
         }
         printf( "loop %d\n\x00", j );
     }
 }
 
 The program runs and gets slower and slower.
 After 8min at loop 700 I canceled.
 Is this speed acceptable for allocating 48MB (+4MB for the array)?
 
 I tried to use a XML DOM parser, parsing 1,5MB xml. This seems to take
 endless.
 
 Is there a workaround? e.g. preallocating memory in one piece.

I made a slight change to the code and it worked well. I moved 'Test[1000_000] arr;' to the module level so it wouldn't be stack allocated, and I changed it to 'Test[1000_000] arr = void;' to avoid preinitializing it. -- Derek (skype: derek.j.parnell) Melbourne, Australia "Down with mediocracy!" 28/03/2006 10:30:30 AM
Mar 27 2006