www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 1985] New: import expression should return ubyte[] not string

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1985

           Summary: import expression should return ubyte[] not string
           Product: D
           Version: 1.028
          Platform: PC
        OS/Version: Windows
            Status: NEW
          Severity: normal
          Priority: P2
         Component: DMD
        AssignedTo: bugzilla digitalmars.com
        ReportedBy: wbaxter gmail.com


The compiler does not convert the encoding of the imported files to utf8, so it
should not pretend that it knows the contents of the file are utf8.

In fact, probably one of the most practically useful applications of the import
expression is to import binary files, which it is impossible for the compiler
to put in utf8 format.

So the conclusion is that import("foo.dat") should evaluate to ubyte[], not
char[].  It can be cast to char[] if the developer happens to know that the
file is, in fact, text.  Currently the situation is reversed -- the data must
be cast to ubyte[] if the developer knows it is not, in fact, utf8 text.


-- 
Apr 10 2008
next sibling parent reply "Janice Caron" <caron800 googlemail.com> writes:
On 11/04/2008, d-bugmail puremagic.com <d-bugmail puremagic.com> wrote:
    import expression should return ubyte[] not string

Shouldn't it return invariant(ubyte)[] ?
Apr 11 2008
parent Bill Baxter <dnewsgroup billbaxter.com> writes:
Janice Caron wrote:
 On 11/04/2008, d-bugmail puremagic.com <d-bugmail puremagic.com> wrote:
    import expression should return ubyte[] not string

Shouldn't it return invariant(ubyte)[] ?

In D2 probably so, but I filed the bug against D1.028. --bb
Apr 11 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 11/04/2008, Bill Baxter <dnewsgroup billbaxter.com> wrote:
 Shouldn't it return invariant(ubyte)[] ?

In D2 probably so, but I filed the bug against D1.028.

Makes sense. I suppose in D2 it depends on the answer to the following question. If I write auto a = import("filename"); auto b = import("filename"); do we get two separate copies, or do a and b both point to the same memory? If the latter, it should definitely be invariant(ubyte)[] in D2, though as you say, ubyte[] in D1.
Apr 11 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1985





------- Comment #3 from aarti interia.pl  2008-04-11 03:24 -------
Yes, I also think that implying that imported file is char[] is not best
decision. 

Maybe even imported type should be void[], so that it must be explicitly casted
to proper type.


-- 
Apr 11 2008
prev sibling next sibling parent "Janice Caron" <caron800 googlemail.com> writes:
On 11/04/2008, d-bugmail puremagic.com <d-bugmail puremagic.com> wrote:
  Maybe even imported type should be void[], so that it must be explicitly
casted
  to proper type.

No, it should be ubyte. The reason is that void arrays can contain pointers, and ubyte arrays can't. A void array means that the garbage collector has to scan it, looking for anything that looks like it might be an address, and if it finds such a collection of bits by accident, then something will be marked as "in use", that actually isn't. If the array came from a file, it can't very well have meaningful pointers into RAM, so I agree with the original poster that it should be ubyte[] for D1, and I would add invariant(ubyte)[] for D2.
Apr 11 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1985





------- Comment #5 from aarti interia.pl  2008-04-11 06:16 -------
Answer to comment #4

This isn't (direct) argument against my proposal. It would be if GC scanning
void[] is fundamentally "good thing". But in fact I don't think that in this
case it is such kind of design decision. (There were already proposals to
change this behavior.)

But in case of reading external files, the problem is that compiler just *don't
know* format of imported file. So IMHO the best thing to do is to reflect this
situation in language, and force user to cast content of file to real type.

In case there will stay default cast to some type in import it is really
difficult to justify which default behavior is better. I agree with Bill that
in most GUI application it would be better to have imported array of bytes. But
you can also think about DB framework in which external file is used to define
schema of database (see: U++ framework written in C++). In this case some text
format is much more natural...


-- 
Apr 11 2008
prev sibling next sibling parent reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1985





------- Comment #6 from wbaxter gmail.com  2008-04-11 07:36 -------
(In reply to comment #5)
 Answer to comment #4
 
 This isn't (direct) argument against my proposal. It would be if GC scanning
 void[] is fundamentally "good thing". But in fact I don't think that in this
 case it is such kind of design decision. (There were already proposals to
 change this behavior.)
 
 But in case of reading external files, the problem is that compiler just *don't
 know* format of imported file. So IMHO the best thing to do is to reflect this
 situation in language, and force user to cast content of file to real type.
 
 In case there will stay default cast to some type in import it is really
 difficult to justify which default behavior is better. I agree with Bill that
 in most GUI application it would be better to have imported array of bytes. But
 you can also think about DB framework in which external file is used to define
 schema of database (see: U++ framework written in C++). In this case some text
 format is much more natural...
 

Your argument is right on, but ubyte[] *is* the type that means "I don't know what the heck this data is, but it's just data, not pointers". void[] means "I don't know what the heck this data is, it might even be full of pointers". I don't see any way that file that is on disk at *compile time* could contain pointers that are relevant to the program later on at *run time*. So ubyte[] is the proper type. --
Apr 11 2008
parent reply Aarti_pl <aarti interia.pl> writes:
 Your argument is right on, but ubyte[] *is* the type that means "I don't know
 what the heck this data is, but it's just data, not pointers".  void[] means "I
 don't know what the heck this data is, it might even be full of pointers".  
 I don't see any way that file that is on disk at *compile time* could contain
 pointers that are relevant to the program later on at *run time*.  So ubyte[]
 is the proper type.

I think I understand your way of thinking. But the problem here is different. With void[] you can not do anything, so you have to cast. With ubyte[] you can use data as they are, because they have in fact specific type. But what if file contains array of int's? In such a case you are in exactly same situation as with char[]. Compiler chooses one type to which it casts by default, exactly like currently with char[]. And this choice can be wrong. If it is good or wrong depends only on application. In GUI ubyte[] is more appropriate (e.g. importing icon), but in DB framework string is better (importing db schema). Having written this all I slowly get to conclusion that current situation is not so bad :-) char[] would be usually much more appropriate for lower layers in application than ubyte[] (mostly for loading and compiling domain languages). Alternative with void[] is IMHO theoretically better, but practically you will have even more casts in you code... Tough decision... :-)
Apr 11 2008
parent Frank Benoit <keinfarbton googlemail.com> writes:
Aarti_pl schrieb:
 Your argument is right on, but ubyte[] *is* the type that means "I 
 don't know
 what the heck this data is, but it's just data, not pointers".  void[] 
 means "I
 don't know what the heck this data is, it might even be full of 
 pointers".  I don't see any way that file that is on disk at *compile 
 time* could contain
 pointers that are relevant to the program later on at *run time*.  So 
 ubyte[]
 is the proper type.

I think I understand your way of thinking. But the problem here is different. With void[] you can not do anything, so you have to cast. With ubyte[] you can use data as they are, because they have in fact specific type. But what if file contains array of int's? In such a case you are in exactly same situation as with char[]. Compiler chooses one type to which it casts by default, exactly like currently with char[]. And this choice can be wrong. If it is good or wrong depends only on application. In GUI ubyte[] is more appropriate (e.g. importing icon), but in DB framework string is better (importing db schema). Having written this all I slowly get to conclusion that current situation is not so bad :-) char[] would be usually much more appropriate for lower layers in application than ubyte[] (mostly for loading and compiling domain languages). Alternative with void[] is IMHO theoretically better, but practically you will have even more casts in you code... Tough decision... :-)

foreach( c; import("data")){ // do something } - cannot work with void[] - will always work in a reasonable way with ubyte[] - might throw UnicodeException with char[]
Apr 11 2008
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1985





------- Comment #8 from wbaxter gmail.com  2008-04-11 15:18 -------
(In reply to comment #7)
 I think I understand your way of thinking. But the problem here is 
 different. With void[] you can not do anything, so you have to cast. 
 With ubyte[] you can use data as they are, because they have in fact 
 specific type. 

Well, it's a specific type that represents raw memory. But I get your point too. If there were a void_without_pointers type that couldn't be used without casting, then I'd be happy for import("file") to use that. But with void[] getting scanned by the gc for pointers it's a no-go. If you do just one import("big_binary_file") it could add enough false pointers that your app would hold on to big gobs of memory it doesn't need to. --
Apr 11 2008