www.digitalmars.com         C & C++   DMDScript  

c++.command-line - Handling large data structures.

reply rvw <rvw_member pathlink.com> writes:
I am writing programs that execute from the command line of Windows 2000.  I
want to be able to do calculations with large data files and structures.  I have
been compiling with the options:  -5 -o -A -r -v2 -w -mn.

I thought the -mn option would give me 4GB of data space.  However, if I create
and write to an array of something over 256,000 unsigned longs (x4 bytes = 1MB),
the program compiles but crashes.

How can I use the whole, or most of, the 4GB?

Thanks for you help.

rvw
May 02 2006
parent reply Jan Knepper <jan smartsoft.us> writes:
How do you create that array?
How about posting an example?



rvw wrote:
 I am writing programs that execute from the command line of Windows 2000.  I
 want to be able to do calculations with large data files and structures.  I
have
 been compiling with the options:  -5 -o -A -r -v2 -w -mn.
 
 I thought the -mn option would give me 4GB of data space.  However, if I create
 and write to an array of something over 256,000 unsigned longs (x4 bytes =
1MB),
 the program compiles but crashes.
 
 How can I use the whole, or most of, the 4GB?
 
 Thanks for you help.
 
 rvw
 
 

-- ManiaC++ Jan Knepper But as for me and my household, we shall use Mozilla... www.mozilla.org
May 02 2006
parent reply rvw <rvw_member pathlink.com> writes:
The following is my test program.  It compiles and executes for up to 256000 or
so, but compiles and fails at 260000.  The error message is "t.exe has generated
errors and will be closed by Windows."

#include <stdio.h>
#include <stdlib.h>

unsigned long i;

int main()
{
unsigned long Long[256000];

for(i=0;i<=255999;i+=10000) 
{
Long[i] = i;
printf("%u\n",Long[i]);
}

return EXIT_SUCCESS;
}


In article <e392o2$6l2$1 digitaldaemon.com>, Jan Knepper says...
How do you create that array?
How about posting an example?



rvw wrote:
 I am writing programs that execute from the command line of Windows 2000.  I
 want to be able to do calculations with large data files and structures.  I
have
 been compiling with the options:  -5 -o -A -r -v2 -w -mn.
 
 I thought the -mn option would give me 4GB of data space.  However, if I create
 and write to an array of something over 256,000 unsigned longs (x4 bytes =
1MB),
 the program compiles but crashes.
 
 How can I use the whole, or most of, the 4GB?
 
 Thanks for you help.
 
 rvw
 
 

-- ManiaC++ Jan Knepper But as for me and my household, we shall use Mozilla... www.mozilla.org

May 02 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
That's because Windows gives a limited amount of space to the stack, and 
you're allocating the array upon the stack. Try instead allocating the 
array with calloc().

rvw wrote:
 The following is my test program.  It compiles and executes for up to 256000 or
 so, but compiles and fails at 260000.  The error message is "t.exe has
generated
 errors and will be closed by Windows."
 
 #include <stdio.h>
 #include <stdlib.h>
 
 unsigned long i;
 
 int main()
 {
 unsigned long Long[256000];
 
 for(i=0;i<=255999;i+=10000) 
 {
 Long[i] = i;
 printf("%u\n",Long[i]);
 }
 
 return EXIT_SUCCESS;
 }
 
 
 In article <e392o2$6l2$1 digitaldaemon.com>, Jan Knepper says...
 How do you create that array?
 How about posting an example?



 rvw wrote:
 I am writing programs that execute from the command line of Windows 2000.  I
 want to be able to do calculations with large data files and structures.  I
have
 been compiling with the options:  -5 -o -A -r -v2 -w -mn.

 I thought the -mn option would give me 4GB of data space.  However, if I create
 and write to an array of something over 256,000 unsigned longs (x4 bytes =
1MB),
 the program compiles but crashes.

 How can I use the whole, or most of, the 4GB?

 Thanks for you help.

 rvw

-- ManiaC++ Jan Knepper But as for me and my household, we shall use Mozilla... www.mozilla.org


May 02 2006
parent reply rvw <rvw_member pathlink.com> writes:
Thanks for your help Walter.  CALLOC does work through 400 million unsigned ints
in the following program.  It fails at 500 million -- "Abnormal program
termination." 

My computer has 1GB of RAM. Is that the (first) limiting factor? (By the way,
there was a lot of hard drive activity during program execution -- Windows
finding working room?)

//t.c     This works at least through 400 million, as written.

#include <dos.h>
#include <stdio.h>
#include <stdlib.h>

unsigned long i;

int main()
{
unsigned long *A;

A = calloc (400000001, sizeof(unsigned long));
if (A == NULL)
{
fprintf (stderr, "Calloc failed");
abort();
}

for (i = 0;i<=399999999;i+=10000000)
{
A[i]=i;
printf("%-12u%-u   \n",i, A[i]);
}

return EXIT_SUCCESS;
}

_______________________________________________________________________
In article <e39ap8$jsg$1 digitaldaemon.com>, Walter Bright says...
That's because Windows gives a limited amount of space to the stack, and 
you're allocating the array upon the stack. Try instead allocating the 
array with calloc().

rvw wrote:
 The following is my test program.  It compiles and executes for up to 256000 or
 so, but compiles and fails at 260000.  The error message is "t.exe has
generated
 errors and will be closed by Windows."
 
 #include <stdio.h>
 #include <stdlib.h>
 
 unsigned long i;
 
 int main()
 {
 unsigned long Long[256000];
 
 for(i=0;i<=255999;i+=10000) 
 {
 Long[i] = i;
 printf("%u\n",Long[i]);
 }
 
 return EXIT_SUCCESS;
 }
 
 
 In article <e392o2$6l2$1 digitaldaemon.com>, Jan Knepper says...
 How do you create that array?
 How about posting an example?



 rvw wrote:
 I am writing programs that execute from the command line of Windows 2000.  I
 want to be able to do calculations with large data files and structures.  I
have
 been compiling with the options:  -5 -o -A -r -v2 -w -mn.

 I thought the -mn option would give me 4GB of data space.  However, if I create
 and write to an array of something over 256,000 unsigned longs (x4 bytes =
1MB),
 the program compiles but crashes.

 How can I use the whole, or most of, the 4GB?

 Thanks for you help.

 rvw

-- ManiaC++ Jan Knepper But as for me and my household, we shall use Mozilla... www.mozilla.org



May 03 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
500 million unsigned ints is 2Gb of memory.

rvw wrote:
 Thanks for your help Walter.  CALLOC does work through 400 million unsigned
ints
 in the following program.  It fails at 500 million -- "Abnormal program
 termination." 
 
 My computer has 1GB of RAM. Is that the (first) limiting factor? (By the way,
 there was a lot of hard drive activity during program execution -- Windows
 finding working room?)

May 06 2006
parent rvw <rvw_member pathlink.com> writes:
I know.  That's why I was surprised that 400 million worked -- that's 1.6GB.
But as I said, there was a lot of hard drive activity.  My guess is that Windows
automatically used hard drive space.  Again, thanks for your help.

In article <e3je8h$1gmu$1 digitaldaemon.com>, Walter Bright says...
500 million unsigned ints is 2Gb of memory.

rvw wrote:
 Thanks for your help Walter.  CALLOC does work through 400 million unsigned
ints
 in the following program.  It fails at 500 million -- "Abnormal program
 termination." 
 
 My computer has 1GB of RAM. Is that the (first) limiting factor? (By the way,
 there was a lot of hard drive activity during program execution -- Windows
 finding working room?)


May 08 2006