digitalmars.D - Death by design?
- "Bob W" <nospam aol.com> Apr 21 2005
- "Ben Hinkle" <ben.hinkle gmail.com> Apr 21 2005
- "Walter" <newshound digitalmars.com> Apr 21 2005
- "Bob W" <nospam aol.com> Apr 21 2005
- "Ben Hinkle" <ben.hinkle gmail.com> Apr 22 2005
- "Bob W" <nospam aol.com> Apr 22 2005
- Manfred Nowak <svv1999 hotmail.com> Apr 21 2005
- Thomas Kuehne <thomas-dloop kuehne.thisisspam.cn> Apr 21 2005
- Manfred Nowak <svv1999 hotmail.com> Apr 22 2005
Although D catches lots of runtime errors, programs tend to die silently if the following code is used with huge text files: chararr=cast(char[])read(bigfile); If I remember correctly, the problem starts with files of approx. 6MB+ in size. In some cases the file will load, the program will finish all operations, but will never return to the command line (DmD 0.121, Win version). I always had the perception that range and stack overflow checking is enabled by default and the according exceptions would be flagged. Right or wrong?
Apr 21 2005
"Bob W" <nospam aol.com> wrote in message news:d49lvn$19h7$1 digitaldaemon.com...Although D catches lots of runtime errors, programs tend to die silently if the following code is used with huge text files: chararr=cast(char[])read(bigfile); If I remember correctly, the problem starts with files of approx. 6MB+ in size. In some cases the file will load, the program will finish all operations, but will never return to the command line (DmD 0.121, Win version). I always had the perception that range and stack overflow checking is enabled by default and the according exceptions would be flagged. Right or wrong?
It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op. Currently it collects and given a large amount of memory I remember posts from long ago where the program could take a minute (maybe they or my memory were exagerating..) to exit while it runs though checking memory.
Apr 21 2005
"Walter" <newshound digitalmars.com> For huge files, using std.stream or std.mmfile to read it is a better choice.
Apr 21 2005
"Ben Hinkle" <ben.hinkle gmail.com> wrote in message news:d49m8b$19lr$1 digitaldaemon.com..."Bob W" <nospam aol.com> wrote in message news:d49lvn$19h7$1 digitaldaemon.com...Although D catches lots of runtime errors, programs tend to die silently if the following code is used with huge text files: chararr=cast(char[])read(bigfile); If I remember correctly, the problem starts with files of approx. 6MB+ in size. In some cases the file will load, the program will finish all operations, but will never return to the command line (DmD 0.121, Win version). I always had the perception that range and stack overflow checking is enabled by default and the according exceptions would be flagged. Right or wrong?
It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op. Currently it collects and given a large amount of memory I remember posts from long ago where the program could take a minute (maybe they or my memory were exagerating..) to exit while it runs though checking memory.
Thanks for your reply, but I cannot help to believe that there are some other things going wrong: Given the most simplistic program: read file to buffer, display buffer length, done. Examples: - File size 154,223,881: returns within the fraction of a sec. - File size 158,923,657: waits forever (i.e. until my patience is exhausted and/or ^C is pressed). (Before anyone asks: 1GB installed) If you have a good explanation why a 3% file size increase could possibly cause a maybe almost infinite delay increase, I'll do the Phobos recompilation. Otherwise I just patiently wait for further info.
Apr 21 2005
"Bob W" <nospam aol.com> wrote in message news:d49qpp$1dcp$1 digitaldaemon.com..."Ben Hinkle" <ben.hinkle gmail.com> wrote in message news:d49m8b$19lr$1 digitaldaemon.com..."Bob W" <nospam aol.com> wrote in message news:d49lvn$19h7$1 digitaldaemon.com...Although D catches lots of runtime errors, programs tend to die silently if the following code is used with huge text files: chararr=cast(char[])read(bigfile); If I remember correctly, the problem starts with files of approx. 6MB+ in size. In some cases the file will load, the program will finish all operations, but will never return to the command line (DmD 0.121, Win version). I always had the perception that range and stack overflow checking is enabled by default and the according exceptions would be flagged. Right or wrong?
It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op. Currently it collects and given a large amount of memory I remember posts from long ago where the program could take a minute (maybe they or my memory were exagerating..) to exit while it runs though checking memory.
Thanks for your reply, but I cannot help to believe that there are some other things going wrong: Given the most simplistic program: read file to buffer, display buffer length, done. Examples: - File size 154,223,881: returns within the fraction of a sec. - File size 158,923,657: waits forever (i.e. until my patience is exhausted and/or ^C is pressed). (Before anyone asks: 1GB installed) If you have a good explanation why a 3% file size increase could possibly cause a maybe almost infinite delay increase, I'll do the Phobos recompilation. Otherwise I just patiently wait for further info.
You are probably right - one would think that if the GC were to blame it would be linear. I don't know what the cause is. Maybe you can try putting some printfs in internal/dmain2.d after your main returns and see where it gets stuck.
Apr 22 2005
If you have a good explanation why a 3% file size increase could possibly cause a maybe almost infinite delay increase, I'll do the Phobos recompilation. Otherwise I just patiently wait for further info.
You are probably right - one would think that if the GC were to blame it would be linear. I don't know what the cause is. Maybe you can try putting some printfs in internal/dmain2.d after your main returns and see where it gets stuck.
I could but for now I will follow Walter's advice: Stay away from std.file.read() for huge files. He probably has had some info on these issues before. And, who knows, maybe the read() is already shining in DMD 0.122 or DMD 0.123? ; )
Apr 22 2005
"Ben Hinkle" <ben.hinkle gmail.com> wrote: [...]It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op.
I remember that posts, but what is the purpose of a collect at exit anyway? There is another point: I once tried to allocate all memory but did not succeed because no outOfMemoryException was thrown. Instead the program did not return. I reported on that, but nobody seemed interested. -manfred
Apr 21 2005
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Manfred Nowak schrieb am Fri, 22 Apr 2005 03:39:05 +0000 (UTC):"Ben Hinkle" <ben.hinkle gmail.com> wrote: [...]It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op.
I remember that posts, but what is the purpose of a collect at exit anyway? There is another point: I once tried to allocate all memory but did not succeed because no outOfMemoryException was thrown. Instead the program did not return.
How did your try to "allocate all memory"? Thomas -----BEGIN PGP SIGNATURE----- iD8DBQFCaHS53w+/yD4P9tIRAq7OAJ0VKdl78oinqzs5TDHlhTShgpuoegCfdv6/ OTfr+vjnpOF0UvzzxeXE1OM= =n59e -----END PGP SIGNATURE-----
Apr 21 2005
Thomas Kuehne <thomas-dloop kuehne.thisisspam.cn> wrote: [...]How did your try to "allocate all memory"?
See bugs group. -manfred
Apr 22 2005