www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Streams, Exceptions and return codes

reply =?ISO-8859-1?Q?Sigbj=F8rn_Lund_Olsen?= <sigbjorn lundolsen.net> writes:
I'm rolling my own experimental streams library, but have run into a 
design choice where I'm doubting what is the best solution. In my 
IInputStream(T) interface, I've got this method:

uint read // out uint elementsRead
(
	inout T[] buffer,
	in int atLeast,
	in int atMost,
	in int bufferOffset
);

I've always thought that an end-of-stream doesn't really warrant an 
exception, but consider that we are at the end of a stream, and that I 
call it like this:

ubyte[] buffer;
uint elementsRead = stream.read(buffer, 10, 200, 0);

stream.read *cannot* return more than 0 elements, being as it is at the 
end of the stream. Should I at this point throw an EndOfStreamException, 
or simply return 0, leaving the caller to check whether elementsRead is 

= atLeast and <= atMost, and then follow up with a call to 


isEndOfStream() (or somesuch function)?

Cheers,
Sigbjørn Lund Olsen
Jun 14 2004
next sibling parent "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
"Sigbjørn Lund Olsen" <sigbjorn lundolsen.net> wrote in message
news:cakmg1$1j8f$1 digitaldaemon.com...

 I'm rolling my own experimental streams library, but have run into a
 design choice where I'm doubting what is the best solution. In my
 IInputStream(T) interface, I've got this method:

 uint read // out uint elementsRead
 (
 inout T[] buffer,
 in int atLeast,
 in int atMost,
 in int bufferOffset
 );

 I've always thought that an end-of-stream doesn't really warrant an
 exception, but consider that we are at the end of a stream, and that I
 call it like this:

 ubyte[] buffer;
 uint elementsRead = stream.read(buffer, 10, 200, 0);

 stream.read *cannot* return more than 0 elements, being as it is at the
 end of the stream. Should I at this point throw an EndOfStreamException,
 or simply return 0, leaving the caller to check whether elementsRead is
  >= atLeast and <= atMost, and then follow up with a call to
 isEndOfStream() (or somesuch function)?


Return code.
Jun 14 2004
prev sibling next sibling parent reply EricAnderton at yahoo dot com <EricAnderton_member pathlink.com> writes:
In article <cakmg1$1j8f$1 digitaldaemon.com>,
=?ISO-8859-1?Q?Sigbj=F8rn_Lund_Olsen?= says...

I'm rolling my own experimental streams library, but have run into a 
design choice where I'm doubting what is the best solution. In my 
IInputStream(T) interface, I've got this method:

uint read // out uint elementsRead
(
	inout T[] buffer,
	in int atLeast,
	in int atMost,
	in int bufferOffset
);

I've always thought that an end-of-stream doesn't really warrant an 
exception, but consider that we are at the end of a stream, and that I 
call it like this:

ubyte[] buffer;
uint elementsRead = stream.read(buffer, 10, 200, 0);

stream.read *cannot* return more than 0 elements, being as it is at the 
end of the stream. Should I at this point throw an EndOfStreamException, 
or simply return 0, leaving the caller to check whether elementsRead is 

= atLeast and <= atMost, and then follow up with a call to 


isEndOfStream() (or somesuch function)?

Cheers,
Sigbjørn Lund Olsen


Your design pretty much locks you into not using exceptions.  If an exception is
thrown from this function, there is no way to capture how many elements have
been read from the stream into the buffer; the return value is only available
upon success of the function.  You'd have to overload the function (or rewrite
it) with an inout parameter to obtain the actual number of elements read to the
buffer in an exception handler (or worse yet, attach it to the exception object
somehow).

Judging by the interface, (say if I were to not read your documentation for this
new library) I would expect read() to behave with these invariants:

- present stream is open
- buffer is not null
- atLeast >= 0 (should probably be a uint)
- atMost >= atLeast (should probably be a uint)
- bufferOffset >= 0 (should probably be a uint)
- bufferOffset + atMost <= buffer.length
- if the return value is less than atLeast, then there are only that many
elements left in the stream and the buffer is unchanged.
- if the return value >= atLeast then buffer now contains that many more
elements.
- only throws if the stream is not open (or if the stream is in some other
unusable state).
- blocks on read since there is no async callback parameter (assuming the stream
class itself doesn't have some sort of method/property/whatever for async
hooks).


Hope this helps.
- Eric
Jun 14 2004
parent reply =?ISO-8859-1?Q?Sigbj=F8rn_Lund_Olsen?= <sigbjorn lundolsen.net> writes:
EricAnderton at yahoo dot com wrote:


 In article <cakmg1$1j8f$1 digitaldaemon.com>,
 =?ISO-8859-1?Q?Sigbj=F8rn_Lund_Olsen?= says...
 

I'm rolling my own experimental streams library, but have run into a 
design choice where I'm doubting what is the best solution. In my 
IInputStream(T) interface, I've got this method:

uint read // out uint elementsRead
(
	inout T[] buffer,
	in int atLeast,
	in int atMost,
	in int bufferOffset
);

I've always thought that an end-of-stream doesn't really warrant an 
exception, but consider that we are at the end of a stream, and that I 
call it like this:

ubyte[] buffer;
uint elementsRead = stream.read(buffer, 10, 200, 0);

stream.read *cannot* return more than 0 elements, being as it is at the 
end of the stream. Should I at this point throw an EndOfStreamException, 
or simply return 0, leaving the caller to check whether elementsRead is 


= atLeast and <= atMost, and then follow up with a call to 


isEndOfStream() (or somesuch function)?

Cheers,
Sigbjørn Lund Olsen


 
 
 Your design pretty much locks you into not using exceptions.  If an exception
is
 thrown from this function, there is no way to capture how many elements have
 been read from the stream into the buffer; the return value is only available
 upon success of the function.  You'd have to overload the function (or rewrite
 it) with an inout parameter to obtain the actual number of elements read to the
 buffer in an exception handler (or worse yet, attach it to the exception object
 somehow).


If I were to use exceptions, I would only throw exceptions in the case 
where I would not be able to service the call. Ie, for an 
'EndOfStreamException', it would not be thrown the instant EOS is 
encountered, but *only* if read cannot provide between atLeast and 
atMost elements *because* it is at the end of the stream.

Other reasons for not being able to provide the requested number of 
elements could be things such as too few elements (in the example I 
provided, lets say there are only 9 ubytes left before end-of-stream), 
or too few elements being accessible (in the case of a file transfer, or 
audio recording, the caller may wish to get more elements than are 
practically available).

In all the mentioned cases the numElements would be 0.


 Judging by the interface, (say if I were to not read your documentation for
this
 new library) I would expect read() to behave with these invariants:
 
 - present stream is open


Constructor opens, Destructor closes.


 - buffer is not null


It could be an empty array (which iirc in D doesn't equal null). 
bufferOffset is provided for letting the caller specify that the read 
should start into a certain offset into the buffer. I was planning to 
provide an overloaded convenience function where bufferOffset = 0. D 
really could use default arguments.


 - atLeast >= 0 (should probably be a uint)


No, it should be able to read/skip/peek/check backwards. Rationale: I 
don't want to implement 'seek()' except as convenience functions for any 
*bounded* stream. In my mind, a stream is loosely defined as "a 
potentially infinite sequence of elements", that doesn't necessarily 
have a beginning or end. So by default the 'seeking' is relative to the 
current position, forward and backwards as far as the limits of 
accessible data. A separate interface would define standardised 
convenience functions which specifies that on any bounded stream (where 
beginning and end, or only beginning, are known values) you would be 
able to do absolute seeking.


 - atMost >= atLeast (should probably be a uint)


abs(atMost) >= abs(atLeast), most certainly


 - bufferOffset >= 0 (should probably be a uint)


Yes, and yes, it probably should.


 - bufferOffset + atMost <= buffer.length


buffer would be resized in read() if the length is found insufficient.


 - if the return value is less than atLeast, then there are only that many
 elements left in the stream and the buffer is unchanged.


No, it would return the number of elements *read*. If there are less 
than atLeast elements left in the stream read() would return 0 *or* 
throw an exception (which is what this is all about really).


 - if the return value >= atLeast then buffer now contains that many more
 elements.


Correct.


 - only throws if the stream is not open (or if the stream is in some other
 unusable state).


Would definetly throw something at someone in that case, yes.


 - blocks on read since there is no async callback parameter (assuming the
stream
 class itself doesn't have some sort of method/property/whatever for async
 hooks).


read() would block on read, yes, for async i/o there are similar methods 
beginRead and endRead. Like .net Stream classes either 'set' of methods 
would be defined in terms of the other, so that in case the Stream is 
*actually* using async i/o read() would call beginRead() and endRead() 
and then return, and in the case that the Stream is *actually* using 
sync i/o beginRead() would call read() and then return. In any case the 
caller never needs to know what is actually being used, the caller code 
will *work*, but won't be able to take advantage of async i/o unless of 
course the stream uses async i/o.

The idea behind is really to mix EIO 
(http://www.erights.org/elib/concurrency/eio/), Java new i/o and .net 
streams. I'm so original ;-)

Cheers,
Sigbjørn Lund Olsen
Jun 14 2004
parent reply EricAnderton at yahoo dot com <EricAnderton_member pathlink.com> writes:
In article <cakuj7$215k$1 digitaldaemon.com>,
=?ISO-8859-1?Q?Sigbj=F8rn_Lund_Olsen?= says...

If I were to use exceptions, I would only throw exceptions in the case 
where I would not be able to service the call. Ie, for an 
'EndOfStreamException', it would not be thrown the instant EOS is 
encountered, but *only* if read cannot provide between atLeast and 
atMost elements *because* it is at the end of the stream.


Gotcha.  That would make the most sense given the interface.


Other reasons for not being able to provide the requested number of 
elements could be things such as too few elements (in the example I 
provided, lets say there are only 9 ubytes left before end-of-stream), 
or too few elements being accessible (in the case of a file transfer, or 
audio recording, the caller may wish to get more elements than are 
practically available). 


[...]

 In my mind, a stream is loosely defined as "a 
potentially infinite sequence of elements", that doesn't necessarily 
have a beginning or end. So by default the 'seeking' is relative to the 
current position, forward and backwards as far as the limits of 
accessible data. A separate interface would define standardised 
convenience functions which specifies that on any bounded stream (where 
beginning and end, or only beginning, are known values) you would be 
able to do absolute seeking.


I think I understand what you're trying to do, and I think you've pretty much
worked out the major implications of your design.  Please let the NG know
when/if you plan to make this available; I have a project that might make use of
a good flexible stream class like this.

Also, I like the idea that you're trying to add forwards and backwards 'seeking'
into one simple interface. IMO, I really dislike how other stream authors have
decided to make 'unreading' a stream more cumbersome than is necessary.


The idea behind is really to mix EIO 
(http://www.erights.org/elib/concurrency/eio/), Java new i/o and .net 
streams. I'm so original ;-)


Now all is clear!  Maybe it's not original, per-se, but it is new to D.  I like
how some of the bulletpoints for EIO's design goals seem to be very much in line
with D's/Walter's.  Nifty.

Yours,
- Eric
Jun 15 2004
parent "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
Eric, everyone,

Perhaps you might consider checking out mango.io over at
http://www.dsource.org/forums/viewtopic.php?t=148 since it has almost
everything I've seen noted in this discussion:

-Buffered I/O for Files, Sockets etc
-A variety of Readers and Writers for formatted IO, endian conversion, etc
-Multiple reader/writers upon the same buffer
-Tokens and Tokenizers for loosely formatted input (text lines, words,
numbers, etc)
-RegExp token wrapper
-Chained operations
-EOS checking is almost completely redundant (you rarely, if ever, need to
check)
-Both put/get and <</>> syntax
-Bidi equivalent of std.outbuffer (can use Readers/Writers per usual)
-Memory-mapped IO seamlessly overloads buffered IO
-Class serialization (to file, network, etc)
-CompositeIO framework for bracketing IO (such as commit semantics)
-Simply mechanism to bind your own classes into Reader/Writer framework
-Exceptions thrown for exceptional conditions (per discussion)
-Printf wrapper
-Mango.io has very little overhead, so it's *fast*. You can optionally
enable array-slicing when using CompositeReader. Tokens are always sliced,
but you can .dup them
-Includes typical file management tools (FileProxy, FileConduit), file path
manipulation (FilePath), etc
-Includes an RFC 2396 compliant URI specification
-Includes a simple Properties file reader
-some <gasp> documentation!

There's a bunch more stuff in there also. Mango.io has been pretty well
beaten-up by three great guys since March, so it's quite stable and robust
now. Sure, there's things in there that could use improvement, but it's a
reasonable start. The design is somewhat reminiscent of Java NIO, but
without all the hideous IO warts that Java IO was originally saddled with. I
would contend that mango.io is dramatically cleaner than the Java design;
but that wouldn't be too hard, now would it? <g>

Mango.io is part of the Mango Tree, which currently includes these other
modules:

-exceptionally fast HTTP server (thanks mainly to D array slicing)
-Java like servlet engine (mango.io was built with this in mind)
-Log4J clone, including an HTML monitor/console for inspecting and setting
log/debug status at runtime (plus Chainsaw integration)
-HTTP client
-some simple caching mechanisms, including virtual caching to disk.

I invite you to take a look, and comment. It'd be great if we could turn
mango.io into a truly excellent IO platform for D ...

- Kris


"EricAnderton at yahoo dot com" <EricAnderton_member pathlink.com> wrote in
message news:camuri$23b0$1 digitaldaemon.com...

 In article <cakuj7$215k$1 digitaldaemon.com>,
 =?ISO-8859-1?Q?Sigbj=F8rn_Lund_Olsen?= says...

If I were to use exceptions, I would only throw exceptions in the case
where I would not be able to service the call. Ie, for an
'EndOfStreamException', it would not be thrown the instant EOS is
encountered, but *only* if read cannot provide between atLeast and
atMost elements *because* it is at the end of the stream.


 Gotcha.  That would make the most sense given the interface.


Other reasons for not being able to provide the requested number of
elements could be things such as too few elements (in the example I
provided, lets say there are only 9 ubytes left before end-of-stream),
or too few elements being accessible (in the case of a file transfer, or
audio recording, the caller may wish to get more elements than are
practically available).


 [...]

 In my mind, a stream is loosely defined as "a
potentially infinite sequence of elements", that doesn't necessarily
have a beginning or end. So by default the 'seeking' is relative to the
current position, forward and backwards as far as the limits of
accessible data. A separate interface would define standardised
convenience functions which specifies that on any bounded stream (where
beginning and end, or only beginning, are known values) you would be
able to do absolute seeking.


 I think I understand what you're trying to do, and I think you've pretty


much

 worked out the major implications of your design.  Please let the NG know
 when/if you plan to make this available; I have a project that might make


use of

 a good flexible stream class like this.

 Also, I like the idea that you're trying to add forwards and backwards


'seeking'

 into one simple interface. IMO, I really dislike how other stream authors


have

 decided to make 'unreading' a stream more cumbersome than is necessary.


The idea behind is really to mix EIO
(http://www.erights.org/elib/concurrency/eio/), Java new i/o and .net
streams. I'm so original ;-)


 Now all is clear!  Maybe it's not original, per-se, but it is new to D.  I


like

 how some of the bulletpoints for EIO's design goals seem to be very much


in line

 with D's/Walter's.  Nifty.

 Yours,
 - Eric
Jun 15 2004
prev sibling next sibling parent Regan Heath <regan netwin.co.nz> writes:
Return value.

Matthew mentioned earlier his idea/method of writing a non exception 
throwing base level I think this is a good idea.

This function could return 0 and flag eof somehow, then the next level up 
can decide if it's an exception and throw one, or not.

Regan

On Mon, 14 Jun 2004 19:23:14 +0200, Sigbjørn Lund Olsen 
<sigbjorn lundolsen.net> wrote:


 I'm rolling my own experimental streams library, but have run into a 
 design choice where I'm doubting what is the best solution. In my 
 IInputStream(T) interface, I've got this method:

 uint read // out uint elementsRead
 (
 	inout T[] buffer,
 	in int atLeast,
 	in int atMost,
 	in int bufferOffset
 );

 I've always thought that an end-of-stream doesn't really warrant an 
 exception, but consider that we are at the end of a stream, and that I 
 call it like this:

 ubyte[] buffer;
 uint elementsRead = stream.read(buffer, 10, 200, 0);

 stream.read *cannot* return more than 0 elements, being as it is at the 
 end of the stream. Should I at this point throw an EndOfStreamException, 
 or simply return 0, leaving the caller to check whether elementsRead is 
  >= atLeast and <= atMost, and then follow up with a call to 
 isEndOfStream() (or somesuch function)?

 Cheers,
 Sigbjørn Lund Olsen




-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Jun 14 2004
prev sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Sigbjørn Lund Olsen wrote:

<snip>

 stream.read *cannot* return more than 0 elements, being as it is at the 
 end of the stream. Should I at this point throw an EndOfStreamException, 
 or simply return 0, leaving the caller to check whether elementsRead is 
  >= atLeast and <= atMost, and then follow up with a call to 
 isEndOfStream() (or somesuch function)?


My thought is that determining whether there is any more data to read, 
and actually reading the data, should be two separate operations.

This would enable distinction between expected and unexpected EOF.  An 
expected EOF is, say, at the end of a text file.  There is nothing in a 
text file to say that there must be more data in order for it to be a 
valid text file.  Reaching the end of the file is then part of the 
normal program logic, and so a loop that checks for EOF and then exits 
is a sensible solution.

On the other hand, there are file formats that cannot end abruptly.  For 
example, a Windows BMP has its dimensions specified in the header, so if 
the file's valid there won't be an EOF while you're still reading it in. 
  (For simplicity, I'm ignoring RLE compressed bitmaps, which I'm not 
sure if complicate matters a bit.)  EOF is then an unexpected condition, 
one worthy of exception.  Otherwise, who knows what the app might try to 
do with the bad data?

Stewart.

-- 
My e-mail is valid but not my primary mailbox, aside from its being the 
unfortunate victim of intensive mail-bombing at the moment.  Please keep 
replies on the 'group where everyone may benefit.
Jun 15 2004
parent =?ISO-8859-1?Q?Sigbj=F8rn_Lund_Olsen?= <sigbjorn lundolsen.net> writes:
Stewart Gordon wrote:

 Sigbjørn Lund Olsen wrote:
 
 <snip>
 

 stream.read *cannot* return more than 0 elements, being as it is at 
 the end of the stream. Should I at this point throw an 
 EndOfStreamException, or simply return 0, leaving the caller to check 
 whether elementsRead is  >= atLeast and <= atMost, and then follow up 
 with a call to isEndOfStream() (or somesuch function)?


 
 
 My thought is that determining whether there is any more data to read, 
 and actually reading the data, should be two separate operations.
 
 This would enable distinction between expected and unexpected EOF.  An 
 expected EOF is, say, at the end of a text file.  There is nothing in a 
 text file to say that there must be more data in order for it to be a 
 valid text file.  Reaching the end of the file is then part of the 
 normal program logic, and so a loop that checks for EOF and then exits 
 is a sensible solution.
 
 On the other hand, there are file formats that cannot end abruptly.  For 
 example, a Windows BMP has its dimensions specified in the header, so if 
 the file's valid there won't be an EOF while you're still reading it in. 
  (For simplicity, I'm ignoring RLE compressed bitmaps, which I'm not 
 sure if complicate matters a bit.)  EOF is then an unexpected condition, 
 one worthy of exception.  Otherwise, who knows what the app might try to 
 do with the bad data?


The current interface 'draft' has a function IOStatus check(in int 
atLeast, in int atMost) or int check(out IOStatus ioStatus, in int 
atleast, in int atMost for that purpose. The interface is inspired by 
EIO (link in another post in the thread), which defines input by two 
'choices' for input - the nature of cursor movement and the nature of 
the result.

read() returns elements and advances the cursor position
skip() returns status and advances the cursor position
peek() returns elements and retains the cursor position
check() returns status and retains the cursor position

I'm not entirely confident that the mapping to output makes as much 
sense, though. Certainly, you might want to skip forward or backward, 
but lets say you skip past end-of-stream, would that be a legal 
operation? And certainly the equivalent of peek() (write() that retains 
the cursor position) doesn't seem immediatly useful (to me at least).

Which of course makes it a little moot - you *do* have a way of checking 
before catching, but the exceptions do need to be thrown in the case 
that checking doesn't occur, or is ignored. I guess this means I've made 
up my mind.

As for RLE-encoded bitmaps, I would imagine the way I'd do it, would be 
to write a filter stream and check the number of decoded bytes against 
the header.

Cheers,
Sigbjørn Lund Olsen
Jun 15 2004