www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - About File.rawWrite

reply bearophile <bearophileHUGS lycos.com> writes:
This D2 program runs in about 5.13 seconds on my PC:


import std.stdio;
void main() {
    auto f = File("bytes_test.dat", "wb");
    ubyte[3] RGB;
    foreach (_; 0 .. 1_000_000)
        f.rawWrite(RGB);
}



While this C program runs in about 0.14 seconds:


#include <stdio.h>
int main() {
    FILE *f = fopen("bytes_test.dat", "wb");
    unsigned char RGB[3] = {0};
    int i;
    for (i = 0; i < 1000000; i++)
        fwrite(RGB, 1, 3, f);
    return 0;
}


Is my D2 program wrong, or is File.rawWrite in need of some improvements?

(Writing 3 bytes at a time is not efficient, but the C code shows that the
runtime is acceptable for me for small files).

Bye,
bearophile
Nov 29 2011
parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
29.11.2011 15:57, bearophile :
 This D2 program runs in about 5.13 seconds on my PC:


 import std.stdio;
 void main() {
      auto f = File("bytes_test.dat", "wb");
      ubyte[3] RGB;
      foreach (_; 0 .. 1_000_000)
          f.rawWrite(RGB);
 }



 While this C program runs in about 0.14 seconds:


 #include<stdio.h>
 int main() {
      FILE *f = fopen("bytes_test.dat", "wb");
      unsigned char RGB[3] = {0};
      int i;
      for (i = 0; i<  1000000; i++)
          fwrite(RGB, 1, 3, f);
      return 0;
 }


 Is my D2 program wrong, or is File.rawWrite in need of some improvements?

 (Writing 3 bytes at a time is not efficient, but the C code shows that the
runtime is acceptable for me for small files).

 Bye,
 bearophile

Your OS is Windows, right? On Windows, rawWrite and rawRead always flushes stream, sets binary mode, reads/writes, flushes stream again, sets previous mode. This is definitely unnecessary slow - at least it should change mode only if needed (the file is opened in a text mode). The better solution is to change all other functions so the file mode will be changed lazily (for a text file, raw* functions will change file mode and leave file in this mode until a call of a function, that really needs a file to be in a text mode). By the way, why this changing mode behaviour is Windows only? Yes, it is clearly documented that this is the case, but it isn't documented _why_ this is the case. Quick link to the current rawRead implementation (for happy owners of Google's Chrome, the fastest beautiful web browser that can BSOD your Windows even without administrator rights) (the link isn't for Firefox: it will slow down the browser and be displayed incorrectly): https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L458
Nov 29 2011
next sibling parent reply torhu <no spam.invalid> writes:
On 29.11.2011 16:00, Denis Shelomovskij wrote:
 Your OS is Windows, right? On Windows, rawWrite and rawRead always
 flushes stream, sets binary mode, reads/writes, flushes stream again,
 sets previous mode. This is definitely unnecessary slow - at least it
 should change mode only if needed (the file is opened in a text mode).
 The better solution is to change all other functions so the file mode
 will be changed lazily (for a text file, raw* functions will change file
 mode and leave file in this mode until a call of a function, that really
 needs a file to be in a text mode).

 By the way, why this changing mode behaviour is Windows only? Yes, it is
 clearly documented that this is the case, but it isn't documented _why_
 this is the case.

Text mode means that \n in strings is translated to \r\n when writing, and the other way when reading. That way you can use only LF (\n) to get linefeeds on all platforms, even if Windows uses CRLF.
Nov 29 2011
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 11/29/2011 01:20 PM, torhu wrote:
 On 29.11.2011 16:00, Denis Shelomovskij wrote:
 Your OS is Windows, right? On Windows, rawWrite and rawRead always
 flushes stream, sets binary mode, reads/writes, flushes stream again,
 sets previous mode. This is definitely unnecessary slow - at least it
 should change mode only if needed (the file is opened in a text mode).
 The better solution is to change all other functions so the file mode
 will be changed lazily (for a text file, raw* functions will change file
 mode and leave file in this mode until a call of a function, that really
 needs a file to be in a text mode).

 By the way, why this changing mode behaviour is Windows only? Yes, it is
 clearly documented that this is the case, but it isn't documented _why_
 this is the case.

Text mode means that \n in strings is translated to \r\n when writing, and the other way when reading. That way you can use only LF (\n) to get linefeeds on all platforms, even if Windows uses CRLF.

The "has no effect" part in the following paragraph has been news to me. From 'man fopen' on my Lubuntu 11 based system: <quote> The mode string can also include the letter 'b' either as a last char‐ acter or as a character between the characters in any of the two-char‐ acter strings described above. This is strictly for compatibility with C89 and has no effect; the 'b' is ignored on all POSIX conforming sys‐ tems, including Linux. (Other systems may treat text files and binary files differently, and adding the 'b' may be a good idea if you do I/O to a binary file and expect that your program may be ported to non-Unix environments.) </quote> Ali
Nov 29 2011
prev sibling parent Kai Meyer <kai unixlords.com> writes:
On 11/29/2011 08:00 AM, Denis Shelomovskij wrote:
 29.11.2011 15:57, bearophile :
 This D2 program runs in about 5.13 seconds on my PC:


 import std.stdio;
 void main() {
 auto f = File("bytes_test.dat", "wb");
 ubyte[3] RGB;
 foreach (_; 0 .. 1_000_000)
 f.rawWrite(RGB);
 }



 While this C program runs in about 0.14 seconds:


 #include<stdio.h>
 int main() {
 FILE *f = fopen("bytes_test.dat", "wb");
 unsigned char RGB[3] = {0};
 int i;
 for (i = 0; i< 1000000; i++)
 fwrite(RGB, 1, 3, f);
 return 0;
 }


 Is my D2 program wrong, or is File.rawWrite in need of some improvements?

 (Writing 3 bytes at a time is not efficient, but the C code shows that
 the runtime is acceptable for me for small files).

 Bye,
 bearophile

Your OS is Windows, right? On Windows, rawWrite and rawRead always flushes stream, sets binary mode, reads/writes, flushes stream again, sets previous mode. This is definitely unnecessary slow - at least it should change mode only if needed (the file is opened in a text mode). The better solution is to change all other functions so the file mode will be changed lazily (for a text file, raw* functions will change file mode and leave file in this mode until a call of a function, that really needs a file to be in a text mode). By the way, why this changing mode behaviour is Windows only? Yes, it is clearly documented that this is the case, but it isn't documented _why_ this is the case. Quick link to the current rawRead implementation (for happy owners of Google's Chrome, the fastest beautiful web browser that can BSOD your Windows even without administrator rights) (the link isn't for Firefox: it will slow down the browser and be displayed incorrectly): https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L458

core i7 with 16 GB of ram..... -Kai Meyer
Nov 29 2011