www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Death by design?

reply "Bob W" <nospam aol.com> writes:
Although D catches lots of runtime errors,
programs tend to die silently if the following
code is used with huge text files:

chararr=cast(char[])read(bigfile);

If I remember correctly, the problem starts
with files of approx. 6MB+ in size. In some
cases the file will load, the program will
finish all operations, but will never return to
the command line (DmD 0.121, Win version).

I always had the perception that range and
stack overflow checking is enabled by default
and the according exceptions would be flagged.
Right or wrong?
Apr 21 2005
parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"Bob W" <nospam aol.com> wrote in message 
news:d49lvn$19h7$1 digitaldaemon.com...
 Although D catches lots of runtime errors,
 programs tend to die silently if the following
 code is used with huge text files:

 chararr=cast(char[])read(bigfile);

 If I remember correctly, the problem starts
 with files of approx. 6MB+ in size. In some
 cases the file will load, the program will
 finish all operations, but will never return to
 the command line (DmD 0.121, Win version).

 I always had the perception that range and
 stack overflow checking is enabled by default
 and the according exceptions would be flagged.
 Right or wrong?

It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op. Currently it collects and given a large amount of memory I remember posts from long ago where the program could take a minute (maybe they or my memory were exagerating..) to exit while it runs though checking memory.
Apr 21 2005
next sibling parent "Walter" <newshound digitalmars.com> writes:
For huge files, using std.stream or std.mmfile to read it is a better
choice.
Apr 21 2005
prev sibling next sibling parent reply "Bob W" <nospam aol.com> writes:
"Ben Hinkle" <ben.hinkle gmail.com> wrote in message 
news:d49m8b$19lr$1 digitaldaemon.com...
 "Bob W" <nospam aol.com> wrote in message 
 news:d49lvn$19h7$1 digitaldaemon.com...
 Although D catches lots of runtime errors,
 programs tend to die silently if the following
 code is used with huge text files:

 chararr=cast(char[])read(bigfile);

 If I remember correctly, the problem starts
 with files of approx. 6MB+ in size. In some
 cases the file will load, the program will
 finish all operations, but will never return to
 the command line (DmD 0.121, Win version).

 I always had the perception that range and
 stack overflow checking is enabled by default
 and the according exceptions would be flagged.
 Right or wrong?

It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op. Currently it collects and given a large amount of memory I remember posts from long ago where the program could take a minute (maybe they or my memory were exagerating..) to exit while it runs though checking memory.

Thanks for your reply, but I cannot help to believe that there are some other things going wrong: Given the most simplistic program: read file to buffer, display buffer length, done. Examples: - File size 154,223,881: returns within the fraction of a sec. - File size 158,923,657: waits forever (i.e. until my patience is exhausted and/or ^C is pressed). (Before anyone asks: 1GB installed) If you have a good explanation why a 3% file size increase could possibly cause a maybe almost infinite delay increase, I'll do the Phobos recompilation. Otherwise I just patiently wait for further info.
Apr 21 2005
parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"Bob W" <nospam aol.com> wrote in message 
news:d49qpp$1dcp$1 digitaldaemon.com...
 "Ben Hinkle" <ben.hinkle gmail.com> wrote in message 
 news:d49m8b$19lr$1 digitaldaemon.com...
 "Bob W" <nospam aol.com> wrote in message 
 news:d49lvn$19h7$1 digitaldaemon.com...
 Although D catches lots of runtime errors,
 programs tend to die silently if the following
 code is used with huge text files:

 chararr=cast(char[])read(bigfile);

 If I remember correctly, the problem starts
 with files of approx. 6MB+ in size. In some
 cases the file will load, the program will
 finish all operations, but will never return to
 the command line (DmD 0.121, Win version).

 I always had the perception that range and
 stack overflow checking is enabled by default
 and the according exceptions would be flagged.
 Right or wrong?

It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op. Currently it collects and given a large amount of memory I remember posts from long ago where the program could take a minute (maybe they or my memory were exagerating..) to exit while it runs though checking memory.

Thanks for your reply, but I cannot help to believe that there are some other things going wrong: Given the most simplistic program: read file to buffer, display buffer length, done. Examples: - File size 154,223,881: returns within the fraction of a sec. - File size 158,923,657: waits forever (i.e. until my patience is exhausted and/or ^C is pressed). (Before anyone asks: 1GB installed) If you have a good explanation why a 3% file size increase could possibly cause a maybe almost infinite delay increase, I'll do the Phobos recompilation. Otherwise I just patiently wait for further info.

You are probably right - one would think that if the GC were to blame it would be linear. I don't know what the cause is. Maybe you can try putting some printfs in internal/dmain2.d after your main returns and see where it gets stuck.
Apr 22 2005
parent "Bob W" <nospam aol.com> writes:
 If you have a good explanation why a 3% file size increase
 could possibly cause a maybe almost infinite delay increase,
 I'll do the Phobos recompilation. Otherwise I just patiently
 wait for further info.

You are probably right - one would think that if the GC were to blame it would be linear. I don't know what the cause is. Maybe you can try putting some printfs in internal/dmain2.d after your main returns and see where it gets stuck.

I could but for now I will follow Walter's advice: Stay away from std.file.read() for huge files. He probably has had some info on these issues before. And, who knows, maybe the read() is already shining in DMD 0.122 or DMD 0.123? ; )
Apr 22 2005
prev sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
"Ben Hinkle" <ben.hinkle gmail.com> wrote:

[...]
 It could be the giant gc collection going on at program exit.
 Try recompiling phobos with the function gc_term in
 phobos/internal/gc/gc.d set to a no-op.

I remember that posts, but what is the purpose of a collect at exit anyway? There is another point: I once tried to allocate all memory but did not succeed because no outOfMemoryException was thrown. Instead the program did not return. I reported on that, but nobody seemed interested. -manfred
Apr 21 2005
parent reply Thomas Kuehne <thomas-dloop kuehne.thisisspam.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Manfred Nowak schrieb am Fri, 22 Apr 2005 03:39:05 +0000 (UTC):
 "Ben Hinkle" <ben.hinkle gmail.com> wrote:

 [...]
 It could be the giant gc collection going on at program exit.
 Try recompiling phobos with the function gc_term in
 phobos/internal/gc/gc.d set to a no-op.

I remember that posts, but what is the purpose of a collect at exit anyway? There is another point: I once tried to allocate all memory but did not succeed because no outOfMemoryException was thrown. Instead the program did not return.

How did your try to "allocate all memory"? Thomas -----BEGIN PGP SIGNATURE----- iD8DBQFCaHS53w+/yD4P9tIRAq7OAJ0VKdl78oinqzs5TDHlhTShgpuoegCfdv6/ OTfr+vjnpOF0UvzzxeXE1OM= =n59e -----END PGP SIGNATURE-----
Apr 21 2005
parent Manfred Nowak <svv1999 hotmail.com> writes:
Thomas Kuehne <thomas-dloop kuehne.thisisspam.cn> wrote:

[...]
 How did your try to "allocate all memory"?

See bugs group. -manfred
Apr 22 2005