www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Stream mixins?

reply "Janice Caron" <caron800 googlemail.com> writes:
Hi, Walter, I just had a good idea. (Well, I think it's a good idea!)

Please could you separate out the member functions of
std.stream.Stream into three mixins: ReadMixin, WriteMixin and
SeekMixin?

ReadMixin to include every read function except readBlock()
WriteMixin to include every write function except writeBlock()
SeekMixin to include every seek function.

Then you could implement Stream by mixing in the mixins.

More importantly, I could write

class MyOutputStream : OutputStream
{
    this() { writeable = true; }
    uint writeBlock(const const void* buffer, uint size) { /*...*/ }
    mixin WriteMixin;
}

instead of having to derive from Stream (I don't want an input stream)
or implement writefln()
Nov 10 2007
next sibling parent "Kris" <foo bar.com> writes:
This is a fair example of why Tango does things differently. The I/O stream 
are discrete, they have interfaces for cases where inheritance won't 
suffice, and Seekable is a trivial interface.


"Janice Caron" <caron800 googlemail.com> wrote in message 
news:mailman.31.1194728178.2338.digitalmars-d puremagic.com...
 Hi, Walter, I just had a good idea. (Well, I think it's a good idea!)

 Please could you separate out the member functions of
 std.stream.Stream into three mixins: ReadMixin, WriteMixin and
 SeekMixin?

 ReadMixin to include every read function except readBlock()
 WriteMixin to include every write function except writeBlock()
 SeekMixin to include every seek function.

 Then you could implement Stream by mixing in the mixins.

 More importantly, I could write

 class MyOutputStream : OutputStream
 {
    this() { writeable = true; }
    uint writeBlock(const const void* buffer, uint size) { /*...*/ }
    mixin WriteMixin;
 }

 instead of having to derive from Stream (I don't want an input stream)
 or implement writefln() 

Nov 10 2007
prev sibling next sibling parent reply "Janice Caron" <caron800 googlemail.com> writes:
On 11/10/07, Kris <foo bar.com> wrote:
 This is a fair example of why Tango does things differently.

Thank you, but since I don't use Tango that's not much help. Allow me to bump my request. Walter, please could you separate out the member functions of std.stream.Stream into three mixins: ReadMixin, WriteMixin and SeekMixin, so I can write class MyOutputStream : OutputStream { uint writeBlock(const const void* buffer, uint size) { /*...*/ } mixin WriteMixin; } ? Without that, the interface OutputStream is impractical to implement - I'd have to implement printf() and writef(); I'd have to implement wunget() and opApply(); ... and not make any mistakes! It's too daunting, and a mixin would do the trick nicely. Also - a question. There are functions in the InputStream and OutputStream interfaces whose definitions are implementation dependent. For example, InputStream.readStringW(). The documentation says "The file format is implementation-specific and should not be used except as opposite actions to write". Now - forgive me for being confused here, but shouldn't an interface define what each function must do? Isn't that the whole point of an interface? I mean if I published: interface ConfusingInterface { int f(void[] a); } but then documented that the content of the input array a is implementation-specific, then I have an interface that doesn't mean anything! Moreover, the phrase "except as opposite actions to write" make no sense, because InputStream has no write functions. The only way I can make sense of this instruction is if I assume that every InputStream must have a /corresponding/ OutputStream. But if that's true - why not just do away with the separate interfaces and just have Stream? So, it seems to me that (1) All abstract functions whose purpose is not completely defined should be removed from InputStream and OutputStream, and (2) We need those mixins available. Disclaimer. My goal here is make std.stream better. /Please/ nobody turn this into another Tango-versus-Phobos thread. I'm so bored with those.
Nov 11 2007
next sibling parent BCS <ao pathlink.com> writes:
Reply to Janice,

 Also - a question. There are functions in the InputStream and
 OutputStream interfaces whose definitions are implementation
 dependent. For example, InputStream.readStringW(). The documentation
 says "The file format is implementation-specific and should not be
 used except as opposite actions to write".
 
 Now - forgive me for being confused here, but shouldn't an interface
 define what each function must do? Isn't that the whole point of an
 interface? I mean if I published:
 

It is defined, but something like this: "Write takes the argument an converts it to bytes that are pushed into a stream, read takes a stream at a location that write started at and read in some bytes and returns the data write was given" what is undefined is only the format of the data.
 interface ConfusingInterface
 {
 int f(void[] a);
 }
 but then documented that the content of the input array a is
 implementation-specific, then I have an interface that doesn't mean
 anything!
 

If it is defined where void[]a should come from it can be useful. Defining the format requires either looking stuff down to future change or adding all sorts of holes for extension.
 Moreover, the phrase "except as opposite actions to write" make no
 sense, because InputStream has no write functions. The only way I can
 make sense of this instruction is if I assume that every InputStream
 must have a /corresponding/ OutputStream. But if that's true - why not
 just do away with the separate interfaces and just have Stream?
 

What would you then do for an object that has no write abilities, or no read abilities? To be a bit silly, what good is a InputStrem without an OutputStream somewhere?
Nov 11 2007
prev sibling parent reply "Janice Caron" <caron800 googlemail.com> writes:
On 11/11/07, BCS <ao pathlink.com> wrote:
 What would you then do for an object that has no write abilities, or no read
 abilities? To be a bit silly, what good is a InputStrem without an OutputStream
 somewhere?

There are plenty of examples of those! /dev/random is an input stream with no output stream /dev/null is an output stream with no input stream What I was actually planning to do was implement a hash function (e.g. SHA-256) as an output stream. You write your bytes to the stream, then call a function to get the hash of those bytes. By definition this would /have/ to be an output-only stream. That's the whole POINT of a hash function - it's /one way/! But I don't see why, in order to do that, I should have to implement: abstract void write(char[] s); abstract void write(const(wchar)[] s); Writes a string, together with its length. The format is implementation-specific and should only be used in conjunction with read. Throw WriteException on error. If the format is implementation specific, then does that mean that I, as the creator of HashOutputStream, get to define the implementation? If so, how does a function like wstring f(InputStream is) { wstring s = is.readStringW(); /* do something with s */ return s; } know what to expect? Of course, the silly thing is that it's /easy/ for me to make a Stream (as opposed to an OutputSteam). That's because Stream is a class, not an interface. All I have to do is subclass Stream and override the three abstract functions - job done! And if I make readBlock() through a ReadException, and seek() throw a SeekException, then I do actually /have/ what I want. That, basically, is problem solved. Except ... it seems overkill because I don't need the read half. And it makes me question the purpose of the OutputStream interface. Why have it, if it so damned hard to use?
Nov 11 2007
next sibling parent "Kris" <foo bar.com> writes:
"Janice Caron" <caron800 googlemail.com>
[snip]
 What I was actually planning to do was implement a hash function (e.g.
 SHA-256) as an output stream. You write your bytes to the stream, then
 call a function to get the hash of those bytes. By definition this
 would /have/ to be an output-only stream. That's the whole POINT of a
 hash function - it's /one way/!

Indeed. You perhaps don't care about this, Janice, but others may since they won't have to write it themselves: Tango has a broad suite of digest classes, including MD2, MD4, MD5, SHA0, SHA01, SHA1, SHA256, and Tiger (contributed by NG regulars). You can use those independently, or combine with a DigestStream to slurp the digest as it flow by. For example: # auto stream = new DigestOutput (new GrowBuffer, new Sha256); # stream.write (somedata) # stream.write (moredata) # .... # auto digest = stream.digest.binaryDigest; And similarly for input, if you need that also: # auto stream = new DigestInput (new FileInput("myfile"), new Sha256); # stream.read (somedata) # stream.read (moredata) # .... # auto digest = stream.digest.binaryDigest;
 But I don't see why, in order to do that, I should have to implement:

    abstract void write(char[] s);
    abstract void write(const(wchar)[] s);

Yeah, I agree.
Nov 11 2007
prev sibling next sibling parent BCS <ao pathlink.com> writes:
Reply to Janice,

 On 11/11/07, BCS <ao pathlink.com> wrote:
 
 What would you then do for an object that has no write abilities, or
 no read abilities? To be a bit silly, what good is a InputStrem
 without an OutputStream somewhere?
 

/dev/random is an input stream with no output stream /dev/null is an output stream with no input stream What I was actually planning to do was implement a hash function (e.g. SHA-256) as an output stream. You write your bytes to the stream, then call a function to get the hash of those bytes. By definition this would /have/ to be an output-only stream. That's the whole POINT of a hash function - it's /one way/! But I don't see why, in order to do that, I should have to implement: abstract void write(char[] s); abstract void write(const(wchar)[] s); Writes a string, together with its length. The format is implementation-specific and should only be used in conjunction with read. Throw WriteException on error. If the format is implementation specific, then does that mean that I, as the creator of HashOutputStream, get to define the implementation?

No, in your case the write() functions are useless. What you seem to want is to pass a chunk of data. For this you should use the lower level functions. However, you may have a point in that the write functions may cause issues for some cases. Maybe there should be a WriteStreamLL that just has the non implementation specific stuff. and then have WriteStream extend it with the high level stuff. I think it all derives from a design choice of "don't say you will do more than you have to". As an example, if the format was specified, what big-vs.-little endian? Do you pick one and let the other side suffer the overhead? As soon as you start nailing things down, where do you stop? The only logical choices I see are no spec and full spec. At the base level, full spec doesn't work to well. You still can get all the benefits of nailing down the format with, as you pointed out, a mixin. You can even have several (big, endian, little endian, pick-one-at-write-time-endian) I think the point in the end is that if you need to be consistent about how you implement the stuff. If you ave a write only stream that never returns the stuff then impalement it however you want(*). If you have a reader and a writer, make sure they match and expect the user to use them in pairs. * come to think of it. as long as you don't need to match your hash with someone else using different code then the formatting is irrelevant.
Nov 11 2007
prev sibling parent reply "Janice Caron" <caron800 googlemail.com> writes:
On 11/11/07, BCS <ao pathlink.com> wrote:
 What you seem to want
 is to pass a chunk of data. For this you should use the lower level functions.

No, a stream of data. A property of hash functions is that you can pass the data all in one chunk, or you can pass it in lots of little chunks, but either way you'll get the same answer. If you can accept the data a byte at a time, then it should be trivial to turn it into a stream. This is, in fact, the case, with Tango. I'm happy to agree they got that right.
 However, you may have a point in that the write functions may cause issues
 for some cases. Maybe there should be a WriteStreamLL that just has the non
 implementation specific stuff. and then have WriteStream extend it with the
 high level stuff.

I may have been misunderstood. I have no problem with a /class/ doing high level (or low level) stuff. Putting it all into a class makes thing easier for the implementor, because all you have to do is derive from the class. My complaint was that these functions exist in an /interface/. This means that implementers aren't being told exactly what they need to implement, and functions which accept such interfaces won't know what to expect. If it's in an interface, we really need to nail down exactly what the functions are supposed to do.
 I think it all derives from a design choice of "don't say you will do more
 than you have to". As an example, if the format was specified, what
big-vs.-little
 endian? Do you pick one and let the other side suffer the overhead? As soon
 as you start nailing things down, where do you stop? The only logical choices
 I see are no spec and full spec. At the base level, full spec doesn't work
 to well.

Again, none of this is a problem for classes. The base class could, if desired, implement writeBigEndian() and writeLittleEndian(), and then they'd be available for all subclasses. But the minute you put stuff like that into an interface, with no handy class to derive from, you just make w-a-y too much work (and potential for errors) for the implementor.
 You still can get all the benefits of nailing down the format with, as you
 pointed out, a mixin.

Yes. That's what I was asking for. Although, even in that case, it would still be advisable to change the docs for those interfaces from "this is implementation specific" to "here is exactly what it must do".
 come to think of it. as long as you don't need to match your hash with
 someone else using different code then the formatting is irrelevant.

Exactly, yes. But the same argument works in general - not just for hash functions. For /any/ kind of output-only stream, you should not be expected to have hand-code your own printf()!
Nov 11 2007
parent BCS <ao pathlink.com> writes:
Reply to Janice,

 On 11/11/07, BCS <ao pathlink.com> wrote:
 
 What you seem to want
 is to pass a chunk of data. For this you should use the lower level
 functions.

No, a stream of data. A property of hash functions is that you can pass the data all in one chunk, or you can pass it in lots of little chunks, but either way you'll get the same answer. If you can accept the data a byte at a time, then it should be trivial to turn it into a stream.

that is in no way contrdictory with what I sugested. The lowlevel stuff just says "here is a lump of bytes to write" the high level stuff says "here is an array of uints to incode and write"
 
 However, you may have a point in that the write functions may cause
 issues for some cases. Maybe there should be a WriteStreamLL that
 just has the non implementation specific stuff. and then have
 WriteStream extend it with the high level stuff.
 

high level (or low level) stuff. Putting it all into a class makes thing easier for the implementor, because all you have to do is derive from the class. My complaint was that these functions exist in an /interface/. This means that implementers aren't being told exactly what they need to implement, and functions which accept such interfaces won't know what to expect. If it's in an interface, we really need to nail down exactly what the functions are supposed to do.

IMHO, they are nailed down exactly: The write functions will take the given data and convert it to a bit pattern that can be sent through a stream and then, with a corresponding call to a read, converted back to the original data. This /is/ exact provided that you consider the "data on the wire" to be below the level of abstraction you are working at. And I think that assumption is valid in most cases.
 
 I think it all derives from a design choice of "don't say you will do
 more than you have to". As an example, if the format was specified,
 what big-vs.-little endian? Do you pick one and let the other side
 suffer the overhead? As soon as you start nailing things down,
 where do you stop? The only logical choices I see are no spec
 and full spec. At the base level, full spec doesn't work to well.

Again, none of this is a problem for classes. The base class could, if desired, implement writeBigEndian() and writeLittleEndian(), and then they'd be available for all subclasses.

if you have a writeBigEndian() and writeLittleEndian() what do you do when you need to pass something to a function that shouldn't known what enden is being used? void WriteWhateverEndean(OutputStream str) { str. // what here?????? } really what is needs is something like multiple inheritance but without the problems.
 But the minute you put stuff like that into an interface, with no
 handy class to derive from, you just make w-a-y too much work (and
 potential for errors) for the implementor.

interface WithLotsOfStuff {...} template LotsOfStuff() {...} class Use : WithLotsOfStuff { mixin LotsOfStuff!(); } That's not much work. If they want something different than one of the defaults, then they need to do the work anyway
 You still can get all the benefits of nailing down the format with,
 as you pointed out, a mixin.
 

Although, even in that case, it would still be advisable to change the docs for those interfaces from "this is implementation specific" to "here is exactly what it must do".

Ok. Big or little endian? when sending an array should the length be 16, 32, 64 or 128 bits? when sending floats, should they be sent in IEEE format (Note some systems don't use IEEE) What about real, 80bit or 128bit floats? My point is not that this should not be defined, my point is that the INTERFACE is the wrong place to define it. The correct place is the code (in a mixin) that implements it. The questions mentioned above only scratch the surface of the issue. If you nail them down in the interface than immediately someone is out in the cold where what you picked doesn't work. The correct solution IMHO, as glossed over in the code example above, is to define this in the implementing code and then have the code using the interface ignore the differences. This does require that the user be carful to keep things paired up properly but that is no more of an issue with my solution than it is with yours.
 you should not
 be expected to have hand-code your own printf()!
 

No objection there :)
Nov 11 2007
prev sibling parent Sean Kelly <sean f4.ca> writes:
Janice Caron wrote:
 Hi, Walter, I just had a good idea. (Well, I think it's a good idea!)
 
 Please could you separate out the member functions of
 std.stream.Stream into three mixins: ReadMixin, WriteMixin and
 SeekMixin?

Not much help for Phobos I suppose, but I did this in an example stream implementation about three years ago: http://www.invisibleduck.org/~sean/code/io/stream.d Sean
Nov 11 2007