www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - BufferedFile bug?

reply Regan Heath <regan netmail.co.nz> writes:
Hey,

[DMD 2.010, windows]

So.. was doing a bit of simple D coding and got some really odd 
behaviour, namely this code creates a 4gb (4,294,975,895 bytes) file on 
my disk!

My question is, is this a bug or am I doing something stupid?

The idea is to open a file and overwrite random blocks with '0' (the 
ascii number, not the decimal value - though I get the same behaviour 
with either).

When executed this code seems to hang after outputting "Close.." and at 
that point the file is growing, and growing... ctrl+c does nothing 
immediately but seems to work after the file reaches 4gb, at a guess 
when close actually returns.

Opening the file in textpad only shows 8232 characters, all 'a' - so 
none of the modifications have been applied and it seems an eof 
character has been inserted at that point in the 4gb file.

Adding a return after the file creation before the modification shows a 
file which is as expected 50000 characters long, all 'a'.  So that at 
least is working as expected.

import std.stdio;
import std.stream;
import std.file;
import std.random;

int main()
{
	// Create small file
	auto 	f = new BufferedFile("test", FileMode.OutNew);
	int	size = 50000;
	
	for(int i = 0; i < size; i++)
		f.write('a');
		
	f.close;

	// Open and modify	
	f = new BufferedFile("test", FileMode.In | FileMode.Out);
	ulong	total = getSize("test");
	long	block = total/1000;
	long 	offset;
	long	bytes;
	Random 	gen;
	
	writefln("Total: %d,%d", total, block);
	
	for(int i = 0; i < 2; i++)
	{
		offset = uniform!(long)(gen, 0L, block);
		bytes  = uniform!(long)(gen, 0L, block);
		
		writefln("Seek : %d,%d,%d", f.position, offset, bytes);
		
		f.seekCur(offset);
		
		writefln("Mod  : %d,%d", f.position, bytes);
		
		for(long j = 0; j < bytes; j++)
			f.write('0');
			
		writefln("Done : %d", f.position);
	}
	
	writefln("Close: %d", f.position);
	
	f.close;
	
	writefln("Done");
	
	return 0;
}
Mar 06 2008
parent reply Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Regan Heath wrote:

 Hey,
 
 [DMD 2.010, windows]
 
 So.. was doing a bit of simple D coding and got some really odd 
 behaviour, namely this code creates a 4gb (4,294,975,895 bytes) file on 
 my disk!
 
 My question is, is this a bug or am I doing something stupid?



Congratulations, you found another stupid integer promotion rule bug. 
Those have been a personal gripe for me. Hopefully we can one day fix 
this horrible design mistake from C.

The problem is that:

uint l = 1;
long c = -l;

yields c == 4294967295

wohoo... (Did I mention that I hate those?)

The error lies in BufferedStream.flush()

BufferedStream has a uint bufferSourcePos and does:

       if (bufferSourcePos != 0 && seekable) {
	// move actual file pointer to front of buffer
	streamPos = s.seek(-bufferSourcePos, SeekPos.Current);

and since seeks first argument is a long, it gets a value close to 4GB.

-- 
Oskar
Mar 06 2008
next sibling parent reply "Janice Caron" <caron800 googlemail.com> writes:
On 06/03/2008, Oskar Linde <oskar.lindeREM ovegmail.com> wrote:

  The problem is that:

  uint x = 1;
  long y = -x;

  yields y == 4294967295



(I changed the variable names because lowercase L looks like one to me).

Huh?

Why doesn't y equal minus one?

I wouldn't call that a bug in BufferedStream.flush(), I'd call it a
bug in D's arithmetic generally.

The way I see it, there are two possible fixes:

(1) Disallow unary minus completely for all unsigned types. Thus

    uint x = 1;
    long y = -x; /*ERROR*/

forcing the user to explicitly write

    long y = -cast(int)x;

or

    long y = -cast(long)x;

either of which should be OK, or

(2), when unary minus is applied to an unsigned type, promote ubyte to
short, ushort to int, uint to long, ulong to cent (...best make that
one illegal for now). That would mean

    uint x = 1;
    long y = -x; /* OK */

but

    uint x = 1;
    int y = -x; /* Error */

Either would work, but the status quo /can't/ be right!!!???
Mar 06 2008
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Janice Caron:

 I wouldn't call that a bug in BufferedStream.flush(), I'd call it a
 bug in D's arithmetic generally.



Another faster solution is to avoid unsigned integrals whenever you can, so you
can avoid some bugs.



 (2), when unary minus is applied to an unsigned type, promote ubyte to
 short, ushort to int, uint to long, ulong to cent (...best make that
 one illegal for now). That would mean



Delphi has runtime cheeks for integer overflow too, they help.

Bye,
bearophile
Mar 06 2008
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
bearophile wrote:

 Delphi has runtime cheeks for integer overflow too, they help.



I'm sorry. I don't usually mind misspellings so much, but this one is 
really getting on my nerves. Maybe it's the fact you repeat it so often, 
combined with the fact the misspelling has an entirely different 
meaning; I don't know.

It's "checks". Not "cheeks".

Compare:
http://dictionary.reference.com/search?q=check
http://dictionary.reference.com/search?q=cheek
Mar 06 2008
parent bearophile <bearophileHUGS lycos.com> writes:
Frits van Bommel:


 I'm sorry. I don't usually mind misspellings so much, but this one is 
 really getting on my nerves. Maybe it's the fact you repeat it so often, 
 combined with the fact the misspelling has an entirely different 
 meaning; I don't know.
 It's "checks". Not "cheeks".



This language is my third one, and it shows.
Don't be sorry, I'll try to remember it, checks!
If you find other problems in my language please feel free to tell me.
Thank you very much,
bearophile
Mar 06 2008
prev sibling next sibling parent Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Janice Caron wrote:

 On 06/03/2008, Oskar Linde <oskar.lindeREM ovegmail.com> wrote:

  The problem is that:

  uint x = 1;
  long y = -x;

  yields y == 4294967295



 (I changed the variable names because lowercase L looks like one to me).
 
 Huh?
 
 Why doesn't y equal minus one?
 
 I wouldn't call that a bug in BufferedStream.flush(), I'd call it a
 bug in D's arithmetic generally.



It is hard to classify something as a bug when it behaves exactly as 
specified. See "Usual Arithmetic Conversion" under:

http://www.digitalmars.com/d/1.0/type.html

This is exactly how it works in C too.


 The way I see it, there are two possible fixes:
 
 (1) Disallow unary minus completely for all unsigned types. Thus



Unary minus is not the only affected operator. All integer arithmetic is 
affected.


 (2), when unary minus is applied to an unsigned type, promote ubyte to
 short, ushort to int, uint to long, ulong to cent (...best make that
 one illegal for now). That would mean
 
     uint x = 1;
     long y = -x; /* OK */
 
 but
 
     uint x = 1;
     int y = -x; /* Error */
 
 Either would work, but the status quo /can't/ be right!!!???



There has been some previous discussions along those lines. For example:

http://www.digitalmars.com/d/archives/digitalmars/D/unsigned_policy_47929.html#N47954

Today, all arithmetic operations are promoted to int/uint, and by the 
magic of the 2's-complement representation, things actually work out 
most of the time.

for example:

int a = (rand() % 3) - 1;

works fine, while

double b = (rand() % 3) - 1;

doesn't.

-- 
Oskar
Mar 06 2008
prev sibling next sibling parent reply "Lionello Lunesu" <lionello lunesu.remove.com> writes:
"Janice Caron" <caron800 googlemail.com> wrote in message 
news:mailman.118.1204810684.2351.digitalmars-d puremagic.com...

 On 06/03/2008, Oskar Linde <oskar.lindeREM ovegmail.com> wrote:

  The problem is that:

  uint x = 1;
  long y = -x;

  yields y == 4294967295



 (I changed the variable names because lowercase L looks like one to me).

 Huh?

 Why doesn't y equal minus one?

 I wouldn't call that a bug in BufferedStream.flush(), I'd call it a
 bug in D's arithmetic generally.

 The way I see it, there are two possible fixes:

 (1) Disallow unary minus completely for all unsigned types. Thus

    uint x = 1;
    long y = -x; /*ERROR*/

 forcing the user to explicitly write

    long y = -cast(int)x;

 or

    long y = -cast(long)x;

 either of which should be OK, or

 (2), when unary minus is applied to an unsigned type, promote ubyte to
 short, ushort to int, uint to long, ulong to cent (...best make that
 one illegal for now). That would mean

    uint x = 1;
    long y = -x; /* OK */

 but

    uint x = 1;
    int y = -x; /* Error */

 Either would work, but the status quo /can't/ be right!!!???



Andrei Alexandrescu posted a .pdf a while back with a graph containing all 
types and directed edges for all allowed implicit promotions.. I think he 
has it on his site, but I can't seem to access it (erdani.org) from China. 
Has he been politcally active lately??  : )

L.
Mar 06 2008
parent "Lionello Lunesu" <lionello lunesu.remove.com> writes:
Found it:

http://erdani.org/d-implicit-conversions.pdf
Mar 06 2008
prev sibling parent "Saaa" <empty needmail.com> writes:
Is there a place where it lists the specific specs which, as this promotion, 
are that error prone?
Mar 06 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 06/03/2008, Oskar Linde <oskar.lindeREM ovegmail.com> wrote:

 There has been some previous discussions along those lines. For example:
  http://www.digitalmars.com/d/archives/digitalmars/D/unsigned_policy_47929.html#N47954



Gosh - I've clearly missed out on that rather interesting line of
conversation. Thanks for the link.
Mar 06 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 07/03/2008, Lionello Lunesu <lionello lunesu.remove.com> wrote:

 Found it:

  http://erdani.org/d-implicit-conversions.pdf



I tried downloading that (unsuccessfully). However, it seems to me
that we don't need a diagram. The rule can be easily expressed purely
in logic.

It is safe to implicitly convert from type T to type U, if and only if
both of the following two rules apply:

(1) There exists no value x of type T for which
    cast(T) cast(U) x != x

(2) There exists no value x of type T, and no type V, for which
    cast(V) x != cast(V) cast(U) x

Finding a single counterexample to either of those rules is sufficient
to prove a conversion unsafe. For example, one cannot safely
implicitly cast an int to a short, because rule (1) is violated (e.g.
with counterexample x == 0x10000). Likewise, one cannot safely
implicitly cast a uint to an int, because rule (2) is violated (e.g.
with counterexample x == 0xFFFFFFFF and V == long)

Well, that blows C integer promotion rules out of the window! :-)
Mar 07 2008
prev sibling parent Regan Heath <regan netmail.co.nz> writes:
Oskar Linde wrote:

 Regan Heath wrote:

 Hey,

 [DMD 2.010, windows]

 So.. was doing a bit of simple D coding and got some really odd 
 behaviour, namely this code creates a 4gb (4,294,975,895 bytes) file 
 on my disk!

 My question is, is this a bug or am I doing something stupid?



 Congratulations, you found another stupid integer promotion rule bug. 
 Those have been a personal gripe for me. Hopefully we can one day fix 
 this horrible design mistake from C.
 
 The problem is that:
 
 uint l = 1;
 long c = -l;
 
 yields c == 4294967295
 
 wohoo... (Did I mention that I hate those?)
 
 The error lies in BufferedStream.flush()
 
 BufferedStream has a uint bufferSourcePos and does:
 
       if (bufferSourcePos != 0 && seekable) {
     // move actual file pointer to front of buffer
     streamPos = s.seek(-bufferSourcePos, SeekPos.Current);
 
 and since seeks first argument is a long, it gets a value close to 4GB.



When the file ended up that close to 4gb each time I did suspect an 
signed/unsigned bug.  Did anyone make a bug report or should I do so now?

Regan
Mar 11 2008