www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - BufferedFile bug?

reply Regan Heath <regan netmail.co.nz> writes:
Hey,

[DMD 2.010, windows]

So.. was doing a bit of simple D coding and got some really odd 
behaviour, namely this code creates a 4gb (4,294,975,895 bytes) file on 
my disk!

My question is, is this a bug or am I doing something stupid?

The idea is to open a file and overwrite random blocks with '0' (the 
ascii number, not the decimal value - though I get the same behaviour 
with either).

When executed this code seems to hang after outputting "Close.." and at 
that point the file is growing, and growing... ctrl+c does nothing 
immediately but seems to work after the file reaches 4gb, at a guess 
when close actually returns.

Opening the file in textpad only shows 8232 characters, all 'a' - so 
none of the modifications have been applied and it seems an eof 
character has been inserted at that point in the 4gb file.

Adding a return after the file creation before the modification shows a 
file which is as expected 50000 characters long, all 'a'.  So that at 
least is working as expected.

import std.stdio;
import std.stream;
import std.file;
import std.random;

int main()
{
	// Create small file
	auto 	f = new BufferedFile("test", FileMode.OutNew);
	int	size = 50000;
	
	for(int i = 0; i < size; i++)
		f.write('a');
		
	f.close;

	// Open and modify	
	f = new BufferedFile("test", FileMode.In | FileMode.Out);
	ulong	total = getSize("test");
	long	block = total/1000;
	long 	offset;
	long	bytes;
	Random 	gen;
	
	writefln("Total: %d,%d", total, block);
	
	for(int i = 0; i < 2; i++)
	{
		offset = uniform!(long)(gen, 0L, block);
		bytes  = uniform!(long)(gen, 0L, block);
		
		writefln("Seek : %d,%d,%d", f.position, offset, bytes);
		
		f.seekCur(offset);
		
		writefln("Mod  : %d,%d", f.position, bytes);
		
		for(long j = 0; j < bytes; j++)
			f.write('0');
			
		writefln("Done : %d", f.position);
	}
	
	writefln("Close: %d", f.position);
	
	f.close;
	
	writefln("Done");
	
	return 0;
}
Mar 06 2008
parent reply Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Regan Heath wrote:
 Hey,
 
 [DMD 2.010, windows]
 
 So.. was doing a bit of simple D coding and got some really odd 
 behaviour, namely this code creates a 4gb (4,294,975,895 bytes) file on 
 my disk!
 
 My question is, is this a bug or am I doing something stupid?

Congratulations, you found another stupid integer promotion rule bug. Those have been a personal gripe for me. Hopefully we can one day fix this horrible design mistake from C. The problem is that: uint l = 1; long c = -l; yields c == 4294967295 wohoo... (Did I mention that I hate those?) The error lies in BufferedStream.flush() BufferedStream has a uint bufferSourcePos and does: if (bufferSourcePos != 0 && seekable) { // move actual file pointer to front of buffer streamPos = s.seek(-bufferSourcePos, SeekPos.Current); and since seeks first argument is a long, it gets a value close to 4GB. -- Oskar
Mar 06 2008
next sibling parent reply "Janice Caron" <caron800 googlemail.com> writes:
On 06/03/2008, Oskar Linde <oskar.lindeREM ovegmail.com> wrote:
  The problem is that:

  uint x = 1;
  long y = -x;

  yields y == 4294967295

(I changed the variable names because lowercase L looks like one to me). Huh? Why doesn't y equal minus one? I wouldn't call that a bug in BufferedStream.flush(), I'd call it a bug in D's arithmetic generally. The way I see it, there are two possible fixes: (1) Disallow unary minus completely for all unsigned types. Thus uint x = 1; long y = -x; /*ERROR*/ forcing the user to explicitly write long y = -cast(int)x; or long y = -cast(long)x; either of which should be OK, or (2), when unary minus is applied to an unsigned type, promote ubyte to short, ushort to int, uint to long, ulong to cent (...best make that one illegal for now). That would mean uint x = 1; long y = -x; /* OK */ but uint x = 1; int y = -x; /* Error */ Either would work, but the status quo /can't/ be right!!!???
Mar 06 2008
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Janice Caron:
 I wouldn't call that a bug in BufferedStream.flush(), I'd call it a
 bug in D's arithmetic generally.

Another faster solution is to avoid unsigned integrals whenever you can, so you can avoid some bugs.
 (2), when unary minus is applied to an unsigned type, promote ubyte to
 short, ushort to int, uint to long, ulong to cent (...best make that
 one illegal for now). That would mean

Delphi has runtime cheeks for integer overflow too, they help. Bye, bearophile
Mar 06 2008
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
bearophile wrote:
 Delphi has runtime cheeks for integer overflow too, they help.

I'm sorry. I don't usually mind misspellings so much, but this one is really getting on my nerves. Maybe it's the fact you repeat it so often, combined with the fact the misspelling has an entirely different meaning; I don't know. It's "checks". Not "cheeks". Compare: http://dictionary.reference.com/search?q=check http://dictionary.reference.com/search?q=cheek
Mar 06 2008
parent bearophile <bearophileHUGS lycos.com> writes:
Frits van Bommel:

 I'm sorry. I don't usually mind misspellings so much, but this one is 
 really getting on my nerves. Maybe it's the fact you repeat it so often, 
 combined with the fact the misspelling has an entirely different 
 meaning; I don't know.
 It's "checks". Not "cheeks".

This language is my third one, and it shows. Don't be sorry, I'll try to remember it, checks! If you find other problems in my language please feel free to tell me. Thank you very much, bearophile
Mar 06 2008
prev sibling next sibling parent Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Janice Caron wrote:
 On 06/03/2008, Oskar Linde <oskar.lindeREM ovegmail.com> wrote:
  The problem is that:

  uint x = 1;
  long y = -x;

  yields y == 4294967295

(I changed the variable names because lowercase L looks like one to me). Huh? Why doesn't y equal minus one? I wouldn't call that a bug in BufferedStream.flush(), I'd call it a bug in D's arithmetic generally.

It is hard to classify something as a bug when it behaves exactly as specified. See "Usual Arithmetic Conversion" under: http://www.digitalmars.com/d/1.0/type.html This is exactly how it works in C too.
 The way I see it, there are two possible fixes:
 
 (1) Disallow unary minus completely for all unsigned types. Thus

Unary minus is not the only affected operator. All integer arithmetic is affected.
 (2), when unary minus is applied to an unsigned type, promote ubyte to
 short, ushort to int, uint to long, ulong to cent (...best make that
 one illegal for now). That would mean
 
     uint x = 1;
     long y = -x; /* OK */
 
 but
 
     uint x = 1;
     int y = -x; /* Error */
 
 Either would work, but the status quo /can't/ be right!!!???

There has been some previous discussions along those lines. For example: http://www.digitalmars.com/d/archives/digitalmars/D/unsigned_policy_47929.html#N47954 Today, all arithmetic operations are promoted to int/uint, and by the magic of the 2's-complement representation, things actually work out most of the time. for example: int a = (rand() % 3) - 1; works fine, while double b = (rand() % 3) - 1; doesn't. -- Oskar
Mar 06 2008
prev sibling next sibling parent reply "Lionello Lunesu" <lionello lunesu.remove.com> writes:
"Janice Caron" <caron800 googlemail.com> wrote in message 
news:mailman.118.1204810684.2351.digitalmars-d puremagic.com...
 On 06/03/2008, Oskar Linde <oskar.lindeREM ovegmail.com> wrote:
  The problem is that:

  uint x = 1;
  long y = -x;

  yields y == 4294967295

(I changed the variable names because lowercase L looks like one to me). Huh? Why doesn't y equal minus one? I wouldn't call that a bug in BufferedStream.flush(), I'd call it a bug in D's arithmetic generally. The way I see it, there are two possible fixes: (1) Disallow unary minus completely for all unsigned types. Thus uint x = 1; long y = -x; /*ERROR*/ forcing the user to explicitly write long y = -cast(int)x; or long y = -cast(long)x; either of which should be OK, or (2), when unary minus is applied to an unsigned type, promote ubyte to short, ushort to int, uint to long, ulong to cent (...best make that one illegal for now). That would mean uint x = 1; long y = -x; /* OK */ but uint x = 1; int y = -x; /* Error */ Either would work, but the status quo /can't/ be right!!!???

Andrei Alexandrescu posted a .pdf a while back with a graph containing all types and directed edges for all allowed implicit promotions.. I think he has it on his site, but I can't seem to access it (erdani.org) from China. Has he been politcally active lately?? : ) L.
Mar 06 2008
parent "Lionello Lunesu" <lionello lunesu.remove.com> writes:
Found it:

http://erdani.org/d-implicit-conversions.pdf
Mar 06 2008
prev sibling parent "Saaa" <empty needmail.com> writes:
Is there a place where it lists the specific specs which, as this promotion, 
are that error prone? 
Mar 06 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 06/03/2008, Oskar Linde <oskar.lindeREM ovegmail.com> wrote:
 There has been some previous discussions along those lines. For example:
  http://www.digitalmars.com/d/archives/digitalmars/D/unsigned_policy_47929.html#N47954

Gosh - I've clearly missed out on that rather interesting line of conversation. Thanks for the link.
Mar 06 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 07/03/2008, Lionello Lunesu <lionello lunesu.remove.com> wrote:
 Found it:

  http://erdani.org/d-implicit-conversions.pdf

I tried downloading that (unsuccessfully). However, it seems to me that we don't need a diagram. The rule can be easily expressed purely in logic. It is safe to implicitly convert from type T to type U, if and only if both of the following two rules apply: (1) There exists no value x of type T for which cast(T) cast(U) x != x (2) There exists no value x of type T, and no type V, for which cast(V) x != cast(V) cast(U) x Finding a single counterexample to either of those rules is sufficient to prove a conversion unsafe. For example, one cannot safely implicitly cast an int to a short, because rule (1) is violated (e.g. with counterexample x == 0x10000). Likewise, one cannot safely implicitly cast a uint to an int, because rule (2) is violated (e.g. with counterexample x == 0xFFFFFFFF and V == long) Well, that blows C integer promotion rules out of the window! :-)
Mar 07 2008
prev sibling parent Regan Heath <regan netmail.co.nz> writes:
Oskar Linde wrote:
 Regan Heath wrote:
 Hey,

 [DMD 2.010, windows]

 So.. was doing a bit of simple D coding and got some really odd 
 behaviour, namely this code creates a 4gb (4,294,975,895 bytes) file 
 on my disk!

 My question is, is this a bug or am I doing something stupid?

Congratulations, you found another stupid integer promotion rule bug. Those have been a personal gripe for me. Hopefully we can one day fix this horrible design mistake from C. The problem is that: uint l = 1; long c = -l; yields c == 4294967295 wohoo... (Did I mention that I hate those?) The error lies in BufferedStream.flush() BufferedStream has a uint bufferSourcePos and does: if (bufferSourcePos != 0 && seekable) { // move actual file pointer to front of buffer streamPos = s.seek(-bufferSourcePos, SeekPos.Current); and since seeks first argument is a long, it gets a value close to 4GB.

When the file ended up that close to 4gb each time I did suspect an signed/unsigned bug. Did anyone make a bug report or should I do so now? Regan
Mar 11 2008