www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - deprecating std.stream, std.cstream, std.socketstream

reply Walter Bright <newshound2 digitalmars.com> writes:
This discussion started in the thread "Getting the const-correctness of Object 
sorted once and for all", but it deserved its own thread.

These modules suffer from the following problems:

1. poor documentation, dearth of examples & rationale

2. toHash(), toString(), etc., all need to be const pure nothrow, but it's 
turning out to be problematic for doing it for these classes

3. overlapping functionality with std.stdio

4. they should present a range interface, not a streaming one
May 13 2012
next sibling parent =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 13-05-2012 23:38, Walter Bright wrote:
 This discussion started in the thread "Getting the const-correctness of
 Object sorted once and for all", but it deserved its own thread.

 These modules suffer from the following problems:

 1. poor documentation, dearth of examples & rationale

 2. toHash(), toString(), etc., all need to be const pure nothrow, but
 it's turning out to be problematic for doing it for these classes

 3. overlapping functionality with std.stdio

 4. they should present a range interface, not a streaming one
I'm all for it. I haven't used any of them, ever, and probably never will. Their APIs aren't particularly appealing, honestly. -- - Alex
May 13 2012
prev sibling next sibling parent "Nathan M. Swan" <nathanmswan gmail.com> writes:
On Sunday, 13 May 2012 at 21:39:07 UTC, Walter Bright wrote:
 4. they should present a range interface, not a streaming one
I was just about to make a post suggesting that! You could easily integrate std.io with std.algorithm to do some pretty cool things. NMS
May 13 2012
prev sibling next sibling parent reply "Kiith-Sa" <42 theanswer.com> writes:
On Sunday, 13 May 2012 at 21:39:07 UTC, Walter Bright wrote:
 This discussion started in the thread "Getting the 
 const-correctness of Object sorted once and for all", but it 
 deserved its own thread.

 These modules suffer from the following problems:

 1. poor documentation, dearth of examples & rationale

 2. toHash(), toString(), etc., all need to be const pure 
 nothrow, but it's turning out to be problematic for doing it 
 for these classes

 3. overlapping functionality with std.stdio

 4. they should present a range interface, not a streaming one
My D:YAML library (YAML parser) depends on std.stream (e.g. for cross-endian compatibility and loading from memory), and I've been waiting for a replacement since the first release. I support removing std.stream, but it needs a replacement with equivalent functionality. Actually, I've postponed a 1.0 release _until_ std.stream is replaced.
May 13 2012
next sibling parent reply "Nathan M. Swan" <nathanmswan gmail.com> writes:
On Sunday, 13 May 2012 at 21:53:58 UTC, Kiith-Sa wrote:
 My D:YAML library (YAML parser) depends on std.stream
 (e.g. for cross-endian compatibility and loading from memory),
 and I've been waiting for a replacement since the first release.
We also need better interfacing with UTFs in that. D is usually great at Unicode, but it doesn't interface well with I/O. For example, when working on the file-reader for SDC I had to hand-code the check-BOM-read-and-convert functions: http://bit.ly/J0QWVF . Trying to make it read lazily is even harder, as all std.utf functions work on arrays, not ranges. I think this should change. NMS
May 13 2012
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2012 3:16 PM, Nathan M. Swan wrote:
 Trying to make it read lazily is even harder, as all std.utf functions work on
 arrays, not ranges. I think this should change.
Yes, std.utf should be upgraded to present range interfaces.
May 13 2012
parent "Jonas Drewsen" <jdrewsen nospam.com> writes:
On Sunday, 13 May 2012 at 22:26:17 UTC, Walter Bright wrote:
 On 5/13/2012 3:16 PM, Nathan M. Swan wrote:
 Trying to make it read lazily is even harder, as all std.utf 
 functions work on
 arrays, not ranges. I think this should change.
Yes, std.utf should be upgraded to present range interfaces.
+1 on that. I really needed it when doing the std.net.curl stuff and would be happy to move it to a more generic handling in std.utf.
May 15 2012
prev sibling next sibling parent "Nathan M. Swan" <nathanmswan gmail.com> writes:
On Sunday, 13 May 2012 at 21:53:58 UTC, Kiith-Sa wrote:
 My D:YAML library (YAML parser) depends on std.stream
 (e.g. for cross-endian compatibility and loading from memory),
 and I've been waiting for a replacement since the first release.
We also need better interfacing with UTFs in that. D is usually great at Unicode, but it doesn't interface well with I/O. For example, when working on the file-reader for SDC I had to hand-code the check-BOM-read-and-convert functions: http://bit.ly/J0QWVF . Trying to make it read lazily is even harder, as all std.utf functions work on arrays, not ranges. I think this should change. NMS
May 13 2012
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 5/13/12, Kiith-Sa <42 theanswer.com> wrote:
 My D:YAML library (YAML parser) depends on std.stream
Also ae.xml depends on it.
May 14 2012
prev sibling next sibling parent Robert Clipsham <robert octarineparrot.com> writes:
On 13/05/2012 22:38, Walter Bright wrote:
 This discussion started in the thread "Getting the const-correctness of
 Object sorted once and for all", but it deserved its own thread.

 These modules suffer from the following problems:

 1. poor documentation, dearth of examples & rationale

 2. toHash(), toString(), etc., all need to be const pure nothrow, but
 it's turning out to be problematic for doing it for these classes

 3. overlapping functionality with std.stdio

 4. they should present a range interface, not a streaming one
I make use of std.stream quite a lot... It's horrible, it has to go. I'm not too bothered if replacements aren't available straight away, as it doesn't take much to drop 10 lines of replacement in for the functionality I use from it until the actual replacement appears. -- Robert http://octarineparrot.com/
May 13 2012
prev sibling next sibling parent reply "Oleg Kuporosov" <Oleg.Kuporosov gmail.com> writes:
On Sunday, 13 May 2012 at 21:39:07 UTC, Walter Bright wrote:

 3. overlapping functionality with std.stdio
unfortunatelly std.stdio under Windows couldn't handle UTF16(wchar)-based file names and text IO which are naturel there. The root of issues looks in both underlying DMC C-stdio (something wrong with w* based functions?) and std.format which provides only UTF8 strings. It make sense to depreciate for reasons but only after std.stdio would support UTF16 names/flows or good replacement (Steven's std.io?) would be ready. Currently std.[c]stream is only the way to work with UTF16 filesystems in Phobos. Or switch to Tango which looks supports it too (but I don't have expirience here).
May 13 2012
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2012 10:22 PM, Oleg Kuporosov wrote:
 unfortunatelly std.stdio under Windows couldn't handle UTF16(wchar)-based file
 names and text IO which are naturel there. The root of issues looks in both
 underlying DMC C-stdio (something wrong with w* based functions?) and
std.format
 which provides only UTF8 strings. It make sense to depreciate for reasons but
 only after std.stdio would support UTF16 names/flows or good replacement
 (Steven's std.io?) would be ready. Currently std.[c]stream is only the way to
 work with UTF16 filesystems in Phobos. Or switch to Tango which looks supports
 it too (but I don't have expirience here).
Why not just convert the UTF16 strings to UTF8 ones? They have the same information.
May 14 2012
prev sibling next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
 From the other thread....

On 13/05/2012 21:58, Walter Bright wrote:
 On 5/13/2012 1:48 PM, Stewart Gordon wrote:
 On 13/05/2012 20:42, Walter Bright wrote:
 <snip>
 I'd like to see std.stream dumped.  I don't see any reason for it
 to exist that std.stdio doesn't do (or should do).
So std.stdio.File is the replacement for the std.stream stuff?
Not exactly. Ranges are the replacement. std.stdio.File is merely a range that deals with files.
I don't see any of the required range methods in it. Moreover, I'm a bit confused about the means of retrieving multiple elements at once with the range API, such as a set number of bytes from a file. We have popFrontN, which advances the range but doesn't return the data from it. We have take and takeExactly, which seem to be the way to get a set number of elements from the range, but I'm confused about when/whether using these advances the original range. If I'm writing a library to read a binary file format, I want to allow the data to come from a file, a socket or a memory image. The stream API makes this straightforward. But it seems some work is needed before std.stdio and the range API are up to it. Stewart.
May 14 2012
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2012 4:43 AM, Stewart Gordon wrote:
 If I'm writing a library to read a binary file format, I want to allow the data
 to come from a file, a socket or a memory image. The stream API makes this
 straightforward. But it seems some work is needed before std.stdio and the
range
 API are up to it.
I agree. But that's where the effort needs to be made, not in carrying stream forward.
May 14 2012
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sun, 13 May 2012 17:38:23 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 This discussion started in the thread "Getting the const-correctness of  
 Object sorted once and for all", but it deserved its own thread.

 These modules suffer from the following problems:

 1. poor documentation, dearth of examples & rationale

 2. toHash(), toString(), etc., all need to be const pure nothrow, but  
 it's turning out to be problematic for doing it for these classes

 3. overlapping functionality with std.stdio

 4. they should present a range interface, not a streaming one
I keep trying to avoid talking about this, because I'm writing a replacement library for std.stream, and I don't want to step on any toes while it's still not accepted. But I have to say, ranges are *not* a good interface for generic data providers. They are *very* good for structured data providers. In other words, a stream of bytes, not a good range (who wants to get one byte at a time?). A stream of UTF text broken into lines, a very good range. I have no problem with getting rid of std.stream. I've never actually used it. Still, we absolutely need a non-range based low-level streaming interface to data. If nothing else, we need something we can build ranges upon, and I think my replacement does a very good job of that. -Steve
May 14 2012
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2012 8:02 AM, Steven Schveighoffer wrote:
 I keep trying to avoid talking about this, because I'm writing a replacement
 library for std.stream, and I don't want to step on any toes while it's still
 not accepted.

 But I have to say, ranges are *not* a good interface for generic data
providers.
 They are *very* good for structured data providers.

 In other words, a stream of bytes, not a good range (who wants to get one byte
 at a time?). A stream of UTF text broken into lines, a very good range.

 I have no problem with getting rid of std.stream. I've never actually used it.
 Still, we absolutely need a non-range based low-level streaming interface to
 data. If nothing else, we need something we can build ranges upon, and I think
 my replacement does a very good job of that.
I'll say in advance without seeing your design that it'll be a tough sell if it is not range based. I've been doing some range based work on the side. I'm convinced there is enormous potential there, despite numerous shortcomings with them I ran across in Phobos. Those shortcomings can be fixed, they are not fatal. The ability to do things like: void main() { stdin.byChunk(1024). map!(a => a.idup). // one of those shortcomings joiner(). stripComments(). copy(stdout.lockingTextWriter()); } is just kick ass.
May 14 2012
next sibling parent reply "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Tuesday, 15 May 2012 at 02:56:20 UTC, Walter Bright wrote:
 On 5/14/2012 8:02 AM, Steven Schveighoffer wrote:
 I keep trying to avoid talking about this, because I'm writing 
 a replacement
 library for std.stream, and I don't want to step on any toes 
 while it's still
 not accepted.

 But I have to say, ranges are *not* a good interface for 
 generic data providers.
 They are *very* good for structured data providers.

 In other words, a stream of bytes, not a good range (who wants 
 to get one byte
 at a time?). A stream of UTF text broken into lines, a very 
 good range.

 [...]
I'll say in advance without seeing your design that it'll be a tough sell if it is not range based. I've been doing some range based work on the side. I'm convinced there is enormous potential there, despite numerous shortcomings with them I ran across in Phobos. Those shortcomings can be fixed, they are not fatal. [...]
I have to say, I'm with Steve on this one. While I do believe ranges will have a very important role to play in D's future I/O paradigm, I also think there needs to be a layer beneath the ranges that more directly maps to OS primitives. And as D is a systems programming language, that layer needs to be publicly available. (Note that this is how std.stdio works now, more or less.) -Lars
May 15 2012
parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Tuesday, 15 May 2012 at 15:22:03 UTC, Lars T. Kyllingstad 
wrote:
 On Tuesday, 15 May 2012 at 02:56:20 UTC, Walter Bright wrote:
 On 5/14/2012 8:02 AM, Steven Schveighoffer wrote:
 I keep trying to avoid talking about this, because I'm 
 writing a replacement
 library for std.stream, and I don't want to step on any toes 
 while it's still
 not accepted.

 But I have to say, ranges are *not* a good interface for 
 generic data providers.
 They are *very* good for structured data providers.

 In other words, a stream of bytes, not a good range (who 
 wants to get one byte
 at a time?). A stream of UTF text broken into lines, a very 
 good range.

 [...]
I'll say in advance without seeing your design that it'll be a tough sell if it is not range based. I've been doing some range based work on the side. I'm convinced there is enormous potential there, despite numerous shortcomings with them I ran across in Phobos. Those shortcomings can be fixed, they are not fatal. [...]
I have to say, I'm with Steve on this one. While I do believe ranges will have a very important role to play in D's future I/O paradigm, I also think there needs to be a layer beneath the ranges that more directly maps to OS primitives. And as D is a systems programming language, that layer needs to be publicly available. (Note that this is how std.stdio works now, more or less.)
Also, I wouldn't mind std.*stream getting deprecated. Personally, I've never used those modules -- not even once. As a first step their documentation could be removed from dlang.org, so new users aren't tempted to start using them. No functionality is better than poor functionality, IMO. -Lars
May 15 2012
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 14 May 2012 22:56:08 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 5/14/2012 8:02 AM, Steven Schveighoffer wrote:
 I keep trying to avoid talking about this, because I'm writing a  
 replacement
 library for std.stream, and I don't want to step on any toes while it's  
 still
 not accepted.

 But I have to say, ranges are *not* a good interface for generic data  
 providers.
 They are *very* good for structured data providers.

 In other words, a stream of bytes, not a good range (who wants to get  
 one byte
 at a time?). A stream of UTF text broken into lines, a very good range.

 I have no problem with getting rid of std.stream. I've never actually  
 used it.
 Still, we absolutely need a non-range based low-level streaming  
 interface to
 data. If nothing else, we need something we can build ranges upon, and  
 I think
 my replacement does a very good job of that.
I'll say in advance without seeing your design that it'll be a tough sell if it is not range based. I've been doing some range based work on the side. I'm convinced there is enormous potential there, despite numerous shortcomings with them I ran across in Phobos. Those shortcomings can be fixed, they are not fatal. The ability to do things like: void main() { stdin.byChunk(1024). map!(a => a.idup). // one of those shortcomings joiner(). stripComments(). copy(stdout.lockingTextWriter()); }
I think we may have a misunderstanding. My design is not range-based, but supports ranges, and actually makes them very easy to implement. byChunk is a perfect example of good range -- it defines a specific criteria for determining an "element" of data, appropriate for specific situations. But it must be built on top of something that allows reading arbitrary amounts of data. At the lowest level, this is the OS file descriptor/HANDLE. To be efficient, it should be based on a buffering stream. That buffering stream *does not* need to be a range, and I don't think shoehorning such a construct into a range interface makes any sense. To make this clear, I can say that any range File supports, my design will support *as a range*. To make it even clearer, the current std.stdio.File structure, which you have shown to "kick ass" with ranges, is *NOT* range-based by my definition. I should note, the output range idiom is directly supported, because the output range definition exactly maps to an output stream definition. -Steve
May 16 2012
prev sibling parent reply "Nathan M. Swan" <nathanmswan gmail.com> writes:
On Monday, 14 May 2012 at 15:02:11 UTC, Steven Schveighoffer 
wrote:
 In other words, a stream of bytes, not a good range (who wants 
 to get one byte at a time?).  A stream of UTF text broken into 
 lines, a very good range.
There are several cases where one would want one byte at the time; e.g. as an input to another range that produces the utf text as an output. I do agree for e.g. with binary data some data can't be read with ranges (when you need to read small chunks of varying size), but that doesn't mean most things shouldn't be ranged-based. NMS
May 15 2012
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
On May 15, 2012, at 3:34 PM, "Nathan M. Swan" <nathanmswan gmail.com> wrote:=


 On Monday, 14 May 2012 at 15:02:11 UTC, Steven Schveighoffer wrote:
 In other words, a stream of bytes, not a good range (who wants to get one=
byte at a time?). A stream of UTF text broken into lines, a very good rang= e.
=20
 There are several cases where one would want one byte at the time; e.g. as=
an input to another range that produces the utf text as an output.
=20
 I do agree for e.g. with binary data some data can't be read with ranges (=
when you need to read small chunks of varying size), but that doesn't mean m= ost things shouldn't be ranged-based. You really want both, depending on the situation. I don't see what's weird a= bout this. C++ iostreams have input and output iterators built on top as wel= l, for much the same reason. The annoying part is that once you've moved to a= range interface it's hard to go back. Like say I want a ZipRange on top of a= FileRange. But now I wan to read structs as binary blobs from that uncompr= essed output.=20 One thing I'd like in a buffered input API is a way to perform transactional= reads such that if the full read can't be performed, the read state remains= unchanged. The best you can do with most APIs is to check for a desired len= gth, but what I'd I don't want to read until a full line is available, and I= don't know the exact length? Typically, you end up having to double buffer= , which stinks.=20=
May 15 2012
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 15 May 2012 19:43:05 -0400, Sean Kelly <sean invisibleduck.org>  
wrote:

 One thing I'd like in a buffered input API is a way to perform  
 transactional reads such that if the full read can't be performed, the  
 read state remains unchanged. The best you can do with most APIs is to  
 check for a desired length, but what I'd I don't want to read until a  
 full line is available, and I don't know the exact length?  Typically,  
 you end up having to double buffer, which stinks.
My new design supports this. I have a function called readUntil: https://github.com/schveiguy/phobos/blob/new-io2/std/io.d#L832 Essentially, it reads into its buffer until the condition is satisfied. Therefore, you are not double buffering. The return value is a slice of the buffer. There is a way to opt-out of reading any data if you determine you cannot do a full read. Just return 0 from the delegate. -Steve
May 16 2012
parent reply travert phare.normalesup.org (Christophe Travert) writes:
"Steven Schveighoffer" , dans le message (digitalmars.D:167548), a
 My new design supports this.  I have a function called readUntil:
 
 https://github.com/schveiguy/phobos/blob/new-io2/std/io.d#L832
 
 Essentially, it reads into its buffer until the condition is satisfied.   
 Therefore, you are not double buffering.  The return value is a slice of  
 the buffer.
 
 There is a way to opt-out of reading any data if you determine you cannot  
 do a full read.  Just return 0 from the delegate.
Maybe I already told this some time ago, but I am not very comfortable with this design. The process delegate has to maintain an internal state, if you want to avoid reading everything again. It will be difficult to implement those process delegates. Do you have an example of moderately complicated reading process to show us it is not too complicated? To avoid this issue, the design could be reversed: A method that would like to read a certain amount of character could take a delegate from the stream, which provides additionnal bytes of data. Example: // create a T by reading from stream. returns true if the T was // successfully created, and false otherwise. bool readFrom(const(ubyte)[] delegate(size_t consumed) stream, out T t); The stream delegate returns a buffer of data to read from when called with consumed==0. It must return additionnal data when called repeatedly. When it is called with a consumed != 0, the corresponding amount of consumed bytes can be discared from the buffer. This "stream" delegate (if should have a better name) should not be more difficult to implement than readUntil, but makes it more easy to use by the client. Did I miss some important information ? -- Christophe
May 16 2012
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 10:03:42 -0400, Christophe Travert  
<travert phare.normalesup.org> wrote:

 "Steven Schveighoffer" , dans le message (digitalmars.D:167548), a
 My new design supports this.  I have a function called readUntil:

 https://github.com/schveiguy/phobos/blob/new-io2/std/io.d#L832

 Essentially, it reads into its buffer until the condition is satisfied.
 Therefore, you are not double buffering.  The return value is a slice of
 the buffer.

 There is a way to opt-out of reading any data if you determine you  
 cannot
 do a full read.  Just return 0 from the delegate.
Maybe I already told this some time ago, but I am not very comfortable with this design. The process delegate has to maintain an internal state, if you want to avoid reading everything again. It will be difficult to implement those process delegates.
The delegate is given which portion has already been "processed", that is the 'start' parameter. If you can use this information, it's highly useful. If you need more context, yes, you have to store it elsewhere, but you do have a delegate which contains a context pointer. In a few places (take a look at TextStream's readln https://github.com/schveiguy/phobos/blob/new-io2/std/io.d#L2149) I use inner functions that have access to the function call's frame pointer in order to configure or store data.
 Do you have an example
 of moderately complicated reading process to show us it is not too
 complicated?
The most complicated I have so far is reading UTF data as a range of dchar: https://github.com/schveiguy/phobos/blob/new-io2/std/io.d#L2209 Note that I hand-inlined all the decoding because using std.utf or the runtime was too slow, so although it looks huge, it's pretty basic stuff, and can largely be ignored for the terms of this discussion. The interesting part is how it specifies what to consume and what not to. I realize it's a different way of thinking about how to do I/O, but it gives more control to the buffer, so it can reason about how best to buffer things. I look at as a way of the buffered stream saying "I'll read some data, you tell me when you see something interesting, and I'll give you a slice to it". The alternative is to double-buffer your data. Each call to read can invalidate the previously buffered data. But readUntil guarantees the data is contiguous and consumed all at once, no need to double-buffer
 To avoid this issue, the design could be reversed: A method that would
 like to read a certain amount of character could take a delegate from
 the stream, which provides additionnal bytes of data.

 Example:
 // create a T by reading from stream. returns true if the T was
 // successfully created, and false otherwise.
 bool readFrom(const(ubyte)[] delegate(size_t consumed) stream, out T t);

 The stream delegate returns a buffer of data to read from when called
 with consumed==0. It must return additionnal data when called
 repeatedly. When it is called with a consumed != 0, the corresponding
 amount of consumed bytes can be discared from the buffer.
I can see use cases for both your method and mine. I think I can implement your idea in terms of mine. I might just do that. The only thing missing is, you need a way to specify to the delegate that it needs more data. Probably using size_t.max as an argument. In fact, I need a peek function anyways, your function will provide that ability as well. -Steve
May 16 2012
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2012 4:43 PM, Sean Kelly wrote:
 One thing I'd like in a buffered input API is a way to perform transactional
 reads such that if the full read can't be performed, the read state remains
 unchanged. The best you can do with most APIs is to check for a desired
 length, but what I'd I don't want to read until a full line is available, and
 I don't know the exact length?  Typically, you end up having to double
 buffer, which stinks.
std.stdio.byLine()
May 16 2012
parent reply Sean Kelly <sean invisibleduck.org> writes:
On May 16, 2012, at 6:52 AM, Walter Bright <newshound2 digitalmars.com> wrot=
e:

 On 5/15/2012 4:43 PM, Sean Kelly wrote:
 One thing I'd like in a buffered input API is a way to perform transactio=
nal
 reads such that if the full read can't be performed, the read state remai=
ns
 unchanged. The best you can do with most APIs is to check for a desired
 length, but what I'd I don't want to read until a full line is available,=
and
 I don't know the exact length?  Typically, you end up having to double
 buffer, which stinks.
=20 std.stdio.byLine()
That was just an example. What if I want to do a formatted read and I'm read= ing from a file that someone else is writing to? I don't want to block or g= et a partial result and an EOF that needs to be reset.=20=
May 16 2012
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2012 7:49 AM, Sean Kelly wrote:
 On May 16, 2012, at 6:52 AM, Walter Bright<newshound2 digitalmars.com>
 wrote:

 On 5/15/2012 4:43 PM, Sean Kelly wrote:
 One thing I'd like in a buffered input API is a way to perform
 transactional reads such that if the full read can't be performed, the
 read state remains unchanged. The best you can do with most APIs is to
 check for a desired length, but what I'd I don't want to read until a
 full line is available, and I don't know the exact length?  Typically,
 you end up having to double buffer, which stinks.
std.stdio.byLine()
That was just an example. What if I want to do a formatted read and I'm reading from a file that someone else is writing to? I don't want to block or get a partial result and an EOF that needs to be reset.
Then you'll need an input range that can be reset - a ForwardRange.
May 16 2012
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, May 15, 2012 at 04:43:05PM -0700, Sean Kelly wrote:
[...]
 One thing I'd like in a buffered input API is a way to perform
 transactional reads such that if the full read can't be performed, the
 read state remains unchanged. The best you can do with most APIs is to
 check for a desired length, but what I'd I don't want to read until a
 full line is available, and I don't know the exact length?  Typically,
 you end up having to double buffer, which stinks. 
This would be very nice to have, but how would you go about implementing such a thing, though? Wouldn't you need OS-level support for it? T -- Let's eat some disquits while we format the biskettes.
May 15 2012
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
May 16 2012
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 09:50:12 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with  
 ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read? -Steve
May 16 2012
next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 16/05/2012 15:38, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with
 ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read? -Steve
A bit ugly but: ---- // Default to 4 byte chunks auto range = myStream.byChunks(4); foreach (chunk; range) { // Set the next chunk is 3 bytes // Chunk after is 4 bytes range.nextChunkSize = 3; // Next chunk is always 5 bytes range.chunkSize = 5; } ---- -- Robert http://octarineparrot.com/
May 16 2012
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 11:19:46 -0400, Robert Clipsham  
<robert octarineparrot.com> wrote:

 On 16/05/2012 15:38, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with
 ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read? -Steve
A bit ugly but: ---- // Default to 4 byte chunks auto range = myStream.byChunks(4); foreach (chunk; range) { // Set the next chunk is 3 bytes // Chunk after is 4 bytes range.nextChunkSize = 3; // Next chunk is always 5 bytes range.chunkSize = 5; }
Yeah, I've seen this before. It's not convincing. -Steve
May 16 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 16.05.2012 19:32, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 11:19:46 -0400, Robert Clipsham
 <robert octarineparrot.com> wrote:

 On 16/05/2012 15:38, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with
 ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read? -Steve
A bit ugly but: ---- // Default to 4 byte chunks auto range = myStream.byChunks(4); foreach (chunk; range) { // Set the next chunk is 3 bytes // Chunk after is 4 bytes range.nextChunkSize = 3; // Next chunk is always 5 bytes range.chunkSize = 5; }
Yeah, I've seen this before. It's not convincing.
Yes, It's obvious that files do *not* generally follow range of items semantic. I mean not even range of various items. In case of binary data it's most of the time header followed by various data. Or hierarchical structure. Or table of links + raw data. Or whatever. I've yet to see standard way to deal with binary formats :) -- Dmitry Olshansky
May 16 2012
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 11:48:32 -0400, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 On 16.05.2012 19:32, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 11:19:46 -0400, Robert Clipsham
 <robert octarineparrot.com> wrote:
 A bit ugly but:
 ----
 // Default to 4 byte chunks
 auto range = myStream.byChunks(4);
 foreach (chunk; range) {
 // Set the next chunk is 3 bytes
 // Chunk after is 4 bytes
 range.nextChunkSize = 3;

 // Next chunk is always 5 bytes
 range.chunkSize = 5;
 }
Yeah, I've seen this before. It's not convincing.
Yes, It's obvious that files do *not* generally follow range of items semantic. I mean not even range of various items. In case of binary data it's most of the time header followed by various data. Or hierarchical structure. Or table of links + raw data. Or whatever. I've yet to see standard way to deal with binary formats :)
The best solution would be a range that's specific to your format. My solution intends to support that. But that's only if your format fits within the "range of elements" model. Good old fashioned "read X bytes" needs to be supported, and insisting you do this range style is just plain wrong IMO. -Steve
May 16 2012
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2012 7:38 AM, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read?
std.byLine() does it. In general, you can read n bytes by calling empty, front, and popFront n times.
May 16 2012
next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
On 16/05/2012 16:59, Walter Bright wrote:
 On 5/16/2012 7:38 AM, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read?
std.byLine() does it.
And is what you want to do with a text file in many cases.
 In general, you can read n bytes by calling empty, front, and popFront n times.
Why would anybody want to read a large binary file _one byte at a time_? Stewart.
May 16 2012
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 16, 2012 at 05:41:49PM +0100, Stewart Gordon wrote:
 On 16/05/2012 16:59, Walter Bright wrote:
On 5/16/2012 7:38 AM, Steven Schveighoffer wrote:
On Wed, 16 May 2012 09:50:12 -0400, Walter Bright <newshound2 digitalmars.com>
wrote:

On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
I do agree for e.g. with binary data some data can't be read with
ranges (when you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read?
std.byLine() does it.
And is what you want to do with a text file in many cases.
In general, you can read n bytes by calling empty, front, and
popFront n times.
Why would anybody want to read a large binary file _one byte at a time_?
[...] import std.range; byte[] readNBytes(R)(R range, size_t n) if (isInputRange!R && hasSlicing!R) { return R[0..n]; } T -- MAS = Mana Ada Sistem?
May 16 2012
parent Stewart Gordon <smjg_1998 yahoo.com> writes:
On 16/05/2012 17:48, H. S. Teoh wrote:
 On Wed, May 16, 2012 at 05:41:49PM +0100, Stewart Gordon wrote:
<snip>
 Why would anybody want to read a large binary file _one byte at a
 time_?
[...] import std.range; byte[] readNBytes(R)(R range, size_t n) if (isInputRange!R&& hasSlicing!R) { return R[0..n]; }
What if I want it to work on ranges that don't have slicing? Stewart.
May 16 2012
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2012 9:41 AM, Stewart Gordon wrote:
 On 16/05/2012 16:59, Walter Bright wrote:
 On 5/16/2012 7:38 AM, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read?
std.byLine() does it.
And is what you want to do with a text file in many cases.
 In general, you can read n bytes by calling empty, front, and popFront n times.
Why would anybody want to read a large binary file _one byte at a time_?
You can have that range read from byChunk(). It's really the same thing that C's stdio does.
May 16 2012
next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 13:21:37 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 5/16/2012 9:41 AM, Stewart Gordon wrote:
 On 16/05/2012 16:59, Walter Bright wrote:
 On 5/16/2012 7:38 AM, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright  
 <newshound2 digitalmars.com>
 wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with  
 ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read?
std.byLine() does it.
And is what you want to do with a text file in many cases.
 In general, you can read n bytes by calling empty, front, and popFront  
 n times.
Why would anybody want to read a large binary file _one byte at a time_?
You can have that range read from byChunk(). It's really the same thing that C's stdio does.
This is very wrong. byChunk doesn't cut it. The number of bytes to consume from the stream can depend on any number of factors, including the actual data in the stream. For instance, I challenge you to write an efficient (meaning no extra buffering) byLine using byChunk as a base. -Steve
May 16 2012
prev sibling parent Stewart Gordon <smjg_1998 yahoo.com> writes:
On 16/05/2012 18:21, Walter Bright wrote:
<snip>
 You can have that range read from byChunk(). It's really the same thing that
C's stdio does.
And what if I want it to work on ranges that don't have a byChunk method? Stewart.
May 16 2012
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 11:59:37 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 5/16/2012 7:38 AM, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright  
 <newshound2 digitalmars.com>
 wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with  
 ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read?
std.byLine() does it.
Have you looked at how std.byLine works? It certainly does not use a range interface as a source.
 In general, you can read n bytes by calling empty, front, and popFront n  
 times.
I hope you are not serious! This will make D *the worst performing* i/o language. This should be evidence enough: steves steves-laptop:~$ time dd if=/dev/zero of=/dev/null bs=1 count=1000000 1000000+0 records in 1000000+0 records out 1000000 bytes (1.0 MB) copied, 0.74052 s, 1.4 MB/s real 0m0.744s user 0m0.176s sys 0m0.564s steves steves-laptop:~$ time dd if=/dev/zero of=/dev/null bs=1000 count=1000 1000+0 records in 1000+0 records out 1000000 bytes (1.0 MB) copied, 0.00194096 s, 515 MB/s real 0m0.006s user 0m0.000s sys 0m0.004s -Steve
May 16 2012
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2012 10:18 AM, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 11:59:37 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 5/16/2012 7:38 AM, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read?
std.byLine() does it.
Have you looked at how std.byLine works? It certainly does not use a range interface as a source.
It presents a range interface, though. Not a streaming one.
 In general, you can read n bytes by calling empty, front, and popFront n times.
I hope you are not serious! This will make D *the worst performing* i/o language.
You can read arbitrary numbers of bytes by tacking a range on after byChunk().
May 16 2012
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 13:23:07 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 5/16/2012 10:18 AM, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 11:59:37 -0400, Walter Bright  
 <newshound2 digitalmars.com>
 wrote:

 On 5/16/2012 7:38 AM, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 09:50:12 -0400, Walter Bright  
 <newshound2 digitalmars.com>
 wrote:

 On 5/15/2012 3:34 PM, Nathan M. Swan wrote:
 I do agree for e.g. with binary data some data can't be read with  
 ranges (when
 you need to read small chunks of varying size),
I don't see why that should be true.
How do you tell front and popFront how many bytes to read?
std.byLine() does it.
Have you looked at how std.byLine works? It certainly does not use a range interface as a source.
It presents a range interface, though. Not a streaming one.
But that is *the point*! The code deciding how much data to read (i.e. the entity I referenced above that 'tells front and popFront how many bytes to read') is *not* using a range interface. In other words, ranges aren't enough. Ranges can be built on top of streaming interfaces. But there is *still* a need for a comprehensive streaming toolkit. And C's streaming toolkit is not as good as a native D toolkit can be.
 In general, you can read n bytes by calling empty, front, and popFront  
 n times.
I hope you are not serious! This will make D *the worst performing* i/o language.
You can read arbitrary numbers of bytes by tacking a range on after byChunk().
No, this doesn't work in most cases. See my other post. You can't get everything you want out of just byChunk and byLine. what about byMySpecificPacketProtocol? -Steve
May 16 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/16/12 12:34 PM, Steven Schveighoffer wrote:
 In other words, ranges aren't enough.
This is copiously clear to me, but the way I like to think about it is by extending the notion of range (with notions such as e.g. BufferedRange, LookaheadRange, and such) instead of developing an abstraction independent from ranges and then working on stitching that with ranges. Andrei
May 16 2012
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 13:48:49 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/16/12 12:34 PM, Steven Schveighoffer wrote:
 In other words, ranges aren't enough.
This is copiously clear to me, but the way I like to think about it is by extending the notion of range (with notions such as e.g. BufferedRange, LookaheadRange, and such) instead of developing an abstraction independent from ranges and then working on stitching that with ranges.
What I think we would end up with is a streaming API with range primitives tacked on. - empty is clunky, but possible to implement. However, it may become invalid (think of reading a file that is being appended to by another process). - popFront and front do not have any clear definition of what they refer to. The only valid thing I can think of is bytes, and then nobody will use them. That's hardly saying it's "range based". I refuse to believe that people will be thrilled by having to 'pre-configure' each front and popFront call in order to get work done. If you want to try and convince me, I'm willing to listen, but so far I haven't seen anything that looks at all appetizing. -Steve
May 16 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/16/12 1:00 PM, Steven Schveighoffer wrote:
 What I think we would end up with is a streaming API with range
 primitives tacked on.

 - empty is clunky, but possible to implement. However, it may become
 invalid (think of reading a file that is being appended to by another
 process).
 - popFront and front do not have any clear definition of what they refer
 to. The only valid thing I can think of is bytes, and then nobody will
 use them.

 That's hardly saying it's "range based". I refuse to believe that people
 will be thrilled by having to 'pre-configure' each front and popFront
 call in order to get work done. If you want to try and convince me, I'm
 willing to listen, but so far I haven't seen anything that looks at all
 appetizing.
Where the two meet is in the notion of buffered streams. Ranges are by default buffered, i.e. user code can call front() several times without an intervening popFront() and get the same thing. So a range has by definition a buffer of at least one element. That makes the range interface unsuitable for strictly UNbuffered streams. On the other hand, a range could no problem offer OPTIONAL unbuffered reads (the existence of a buffer does not preclude availability of unbuffered transfers). So to tie it all nicely I think we need: 1. A STRICTLY UNBUFFERED streaming abstraction 2. A notion of range that supports unbuffered transfers. Andrei
May 16 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05/16/2012 11:08 PM, Andrei Alexandrescu wrote:
 On 5/16/12 1:00 PM, Steven Schveighoffer wrote:
 What I think we would end up with is a streaming API with range
 primitives tacked on.

 - empty is clunky, but possible to implement. However, it may become
 invalid (think of reading a file that is being appended to by another
 process).
 - popFront and front do not have any clear definition of what they refer
 to. The only valid thing I can think of is bytes, and then nobody will
 use them.

 That's hardly saying it's "range based". I refuse to believe that people
 will be thrilled by having to 'pre-configure' each front and popFront
 call in order to get work done. If you want to try and convince me, I'm
 willing to listen, but so far I haven't seen anything that looks at all
 appetizing.
Where the two meet is in the notion of buffered streams. Ranges are by default buffered, i.e. user code can call front() several times without an intervening popFront() and get the same thing. So a range has by definition a buffer of at least one element.
I don't think this necessarily holds. 'front' might be computed on the fly, as it is done for std.algorithm.map.
 That makes the range interface unsuitable for strictly UNbuffered
 streams. On the other hand, a range could no problem offer OPTIONAL
 unbuffered reads (the existence of a buffer does not preclude
 availability of unbuffered transfers).

 So to tie it all nicely I think we need:

 1. A STRICTLY UNBUFFERED streaming abstraction

 2. A notion of range that supports unbuffered transfers.


 Andrei
May 16 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/16/12 4:40 PM, Timon Gehr wrote:
 On 05/16/2012 11:08 PM, Andrei Alexandrescu wrote:
 On 5/16/12 1:00 PM, Steven Schveighoffer wrote:
 What I think we would end up with is a streaming API with range
 primitives tacked on.

 - empty is clunky, but possible to implement. However, it may become
 invalid (think of reading a file that is being appended to by another
 process).
 - popFront and front do not have any clear definition of what they refer
 to. The only valid thing I can think of is bytes, and then nobody will
 use them.

 That's hardly saying it's "range based". I refuse to believe that people
 will be thrilled by having to 'pre-configure' each front and popFront
 call in order to get work done. If you want to try and convince me, I'm
 willing to listen, but so far I haven't seen anything that looks at all
 appetizing.
Where the two meet is in the notion of buffered streams. Ranges are by default buffered, i.e. user code can call front() several times without an intervening popFront() and get the same thing. So a range has by definition a buffer of at least one element.
I don't think this necessarily holds. 'front' might be computed on the fly, as it is done for std.algorithm.map.
It used to be buffered in fact but that was too much trouble. The fair thing to say here is that map relies on the implicit buffering of its input. Andrei
May 16 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 16 May 2012 at 17:48:52 UTC, Andrei Alexandrescu 
wrote:
 This is copiously clear to me, but the way I like to think 
 about it is by extending the notion of range (with notions such 
 as e.g. BufferedRange, LookaheadRange, and such)
I tried this in cgi.d somewhat recently. It ended up only vaguely looking like a range. /** A slight difference from regular ranges is you can give it the maximum number of bytes to consume. IMPORTANT NOTE: the default is to consume nothing, so if you don't call consume() yourself and use a regular foreach, it will infinitely loop! */ void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0) {} I called that a "slight different" in the comment, but it is actually a pretty major difference. In practice, it is nothing like a regular range. If I defaulted to size_t.max, you could foreach() it, but then you don't really get to take advantage of the buffer, since it is cleared out entirely for each iteration. If it defaults to 0, you can put it in a foreach... but you have to manually say how much of it is consumed, which no other range does, meaning it won't work with std.algorithm or anything. It sorta looks like a range, but isn't actually one at all. I'm sure something better is possible, but I don't think the range abstraction is a good fit for this use case. Of course, providing optional ranges (like how file gives byChunk, byLine, etc.) is probably a good idea.
May 16 2012
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 16, 2012 at 12:48:49PM -0500, Andrei Alexandrescu wrote:
 On 5/16/12 12:34 PM, Steven Schveighoffer wrote:
In other words, ranges aren't enough.
This is copiously clear to me, but the way I like to think about it is by extending the notion of range (with notions such as e.g. BufferedRange, LookaheadRange, and such) instead of developing an abstraction independent from ranges and then working on stitching that with ranges.
[...] One direction that _could_ be helpful, perhaps, is to extend the concept of range to include, let's tentatively call it, a ChunkedRange. Basically a ChunkedRange implements the usual InputRange operations (empty, front, popfront) but adds the following new primitives: - bool hasAtLeast(R)(R range, int n) - true if underlying range has at least n elements left; - E[] frontN(R)(R range, int n) - returns a slice containing the front n elements from the range: this will buffer the next n elements from the range if they aren't already; repeated calls will just return the buffer; - void popN(R)(R range, int n) - discards the first n elements from the buffer, thus causing the next call to frontN() to fetch more data if necessary. These are all tentative names, of course. But the idea is that you can keep N elements of the range "in view" at a time, without having to individually read them out and save them in a second buffer, and you can advance this view once you're done with the current data and want to move on. Existing range operations like popFrontN, take, takeExactly, drop, etc., can be extended to take advantage of the extra functionality of ChunkedRanges. (Perhaps popFrontN can even be merged with popN, since they amount to the same thing.) Using a ChunkedRange allows you to write functions that parse a particular range and return a range of chunks (say, a deserializer that returns a range of objects given a range of bytes). Thinking on it a bit further, perhaps we can call this a WindowedRange, since it somewhat resembles the sliding window protocol where you keep a "window" of sequential packet ids in an active buffer, and remove them from the buffer as they get ack'ed (consumed by popN). The buffer thus acts like a "window" into the next n elements in the range, which can be "slid forward" as data is consumed. T -- Having a smoking section in a restaurant is like having a peeing section in a swimming pool. -- Edward Burr
May 16 2012
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 15:38:02 -0400, H. S. Teoh <hsteoh quickfur.ath.cx>  
wrote:

 On Wed, May 16, 2012 at 12:48:49PM -0500, Andrei Alexandrescu wrote:
 On 5/16/12 12:34 PM, Steven Schveighoffer wrote:
In other words, ranges aren't enough.
This is copiously clear to me, but the way I like to think about it is by extending the notion of range (with notions such as e.g. BufferedRange, LookaheadRange, and such) instead of developing an abstraction independent from ranges and then working on stitching that with ranges.
[...] One direction that _could_ be helpful, perhaps, is to extend the concept of range to include, let's tentatively call it, a ChunkedRange. Basically a ChunkedRange implements the usual InputRange operations (empty, front, popfront) but adds the following new primitives: - bool hasAtLeast(R)(R range, int n) - true if underlying range has at least n elements left; - E[] frontN(R)(R range, int n) - returns a slice containing the front n elements from the range: this will buffer the next n elements from the range if they aren't already; repeated calls will just return the buffer; - void popN(R)(R range, int n) - discards the first n elements from the buffer, thus causing the next call to frontN() to fetch more data if necessary.
On such ranges, what would popFront and front do? I'm assuming since frontN and popN are referring to how many elements, and since the most logical definition for elements is bytes, that front gets the next byte, and popFront discards the next byte. This seems useless to me. I still don't get the need to "add" this to ranges. The streaming API works fine on its own. But there is an omission with your proposed API regardless -- reading data is a mutating event. It destructively mutates the underlying data stream so that you cannot get the data again. This means you must double-buffer data in order to support frontN and popN that are not necessary with a simple read API. For example: auto buf = new ubyte[1000000]; stream.read(buf); does not need to first buffer the data inside the stream and then copy it to buf, it can read it from the OS *directly* into buf. -Steve
May 16 2012
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 16, 2012 at 04:15:22PM -0400, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 15:38:02 -0400, H. S. Teoh
 <hsteoh quickfur.ath.cx> wrote:
[...]
One direction that _could_ be helpful, perhaps, is to extend the concept
of range to include, let's tentatively call it, a ChunkedRange.
Basically a ChunkedRange implements the usual InputRange operations
(empty, front, popfront) but adds the following new primitives:

- bool hasAtLeast(R)(R range, int n) - true if underlying range has at
  least n elements left;

- E[] frontN(R)(R range, int n) - returns a slice containing the front n
  elements from the range: this will buffer the next n elements from the
  range if they aren't already; repeated calls will just return the
  buffer;

- void popN(R)(R range, int n) - discards the first n elements from the
  buffer, thus causing the next call to frontN() to fetch more data if
  necessary.
On such ranges, what would popFront and front do? I'm assuming since frontN and popN are referring to how many elements, and since the most logical definition for elements is bytes, that front gets the next byte, and popFront discards the next byte. This seems useless to me.
How so? It's still useful for implementing readByte, for example.
 I still don't get the need to "add" this to ranges.  The streaming API
 works fine on its own.
 
 But there is an omission with your proposed API regardless --
 reading data is a mutating event.  It destructively mutates the
 underlying data stream so that you cannot get the data again.  This
 means you must double-buffer data in order to support frontN and
 popN that are not necessary with a simple read API.
 
 For example:
 
 auto buf = new ubyte[1000000];
 stream.read(buf);
 
 does not need to first buffer the data inside the stream and then
 copy it to buf, it can read it from the OS *directly* into buf.
[...] The idea is that by asking for N elements at a time instead of calling front/popFront N times, the underlying implementation can optimize the request by creating a buffer of size N and have the OS read exactly N bytes directly into that buffer. // Reads 1,000,000 bytes into newly allocated buffer and returns // buffer. auto buf = stream.frontN(1_000_000); // Since 1,000,000 bytes is already read into the buffer, this // simply returns a slice of the same buffer: auto buf2 = stream.frontN(1_000_000); assert(buf is buf2); // This consumes the buffer: stream.popN(1_000_000); // This will read another 1,000,000 bytes into a new buffer auto buf3 = stream.frontN(1_000_000); // This returns the same buffer as buf3 since we already have // the data available. auto buf4 = stream.frontN(1_000_000); T -- English has the lovely word "defenestrate", meaning "to execute by throwing someone out a window", or more recently "to remove Windows from a computer and replace it with something useful". :-) -- John Cowan
May 16 2012
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 16:30:43 -0400, H. S. Teoh <hsteoh quickfur.ath.cx>  
wrote:

 On Wed, May 16, 2012 at 04:15:22PM -0400, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 15:38:02 -0400, H. S. Teoh
 <hsteoh quickfur.ath.cx> wrote:
[...]
One direction that _could_ be helpful, perhaps, is to extend the  
concept
of range to include, let's tentatively call it, a ChunkedRange.
Basically a ChunkedRange implements the usual InputRange operations
(empty, front, popfront) but adds the following new primitives:

- bool hasAtLeast(R)(R range, int n) - true if underlying range has at
  least n elements left;

- E[] frontN(R)(R range, int n) - returns a slice containing the front  
n
  elements from the range: this will buffer the next n elements from  
the
  range if they aren't already; repeated calls will just return the
  buffer;

- void popN(R)(R range, int n) - discards the first n elements from the
  buffer, thus causing the next call to frontN() to fetch more data if
  necessary.
On such ranges, what would popFront and front do? I'm assuming since frontN and popN are referring to how many elements, and since the most logical definition for elements is bytes, that front gets the next byte, and popFront discards the next byte. This seems useless to me.
How so? It's still useful for implementing readByte, for example.
readByte is covered by frontN(1). Why the need for front()? Let me answer that question for you -- so it can be treated as a normal range. But nobody will want to do that. i.e. copy to appender will read one byte at a time into the array.
 I still don't get the need to "add" this to ranges.  The streaming API
 works fine on its own.

 But there is an omission with your proposed API regardless --
 reading data is a mutating event.  It destructively mutates the
 underlying data stream so that you cannot get the data again.  This
 means you must double-buffer data in order to support frontN and
 popN that are not necessary with a simple read API.

 For example:

 auto buf = new ubyte[1000000];
 stream.read(buf);

 does not need to first buffer the data inside the stream and then
 copy it to buf, it can read it from the OS *directly* into buf.
[...] The idea is that by asking for N elements at a time instead of calling front/popFront N times, the underlying implementation can optimize the request by creating a buffer of size N and have the OS read exactly N bytes directly into that buffer. // Reads 1,000,000 bytes into newly allocated buffer and returns // buffer. auto buf = stream.frontN(1_000_000);
OK, so stream is providing data via return value and allocation.
 	// Since 1,000,000 bytes is already read into the buffer, this
 	// simply returns a slice of the same buffer:
 	auto buf2 = stream.frontN(1_000_000);
Is buf2 mutable? If so, this is no good, buf could have mutated this data. But this can be fixed by making the return value of frontN be const(ubyte)[].
 	assert(buf is buf2);

 	// This consumes the buffer:
 	stream.popN(1_000_000);
What does "consume" mean, discard? Obviously not "reuse", due to line below...
 	// This will read another 1,000,000 bytes into a new buffer
 	auto buf3 = stream.frontN(1_000_000);
OK, you definitely lost me here, this will not fly. The whole point of buffering is to avoid having to reallocate on every read. If you have to allocate every read, "buffering" is going to have a negative impact on performance! -Steve
May 16 2012
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 16, 2012 at 04:52:09PM -0400, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 16:30:43 -0400, H. S. Teoh
 <hsteoh quickfur.ath.cx> wrote:
 
On Wed, May 16, 2012 at 04:15:22PM -0400, Steven Schveighoffer wrote:
On Wed, 16 May 2012 15:38:02 -0400, H. S. Teoh
<hsteoh quickfur.ath.cx> wrote:
[...]
One direction that _could_ be helpful, perhaps, is to extend the
concept of range to include, let's tentatively call it, a
ChunkedRange.  Basically a ChunkedRange implements the usual
InputRange operations (empty, front, popfront) but adds the
following new primitives:

- bool hasAtLeast(R)(R range, int n) - true if underlying range has
  at least n elements left;

- E[] frontN(R)(R range, int n) - returns a slice containing the
  front n elements from the range: this will buffer the next n
  elements from the range if they aren't already; repeated calls
  will just return the buffer;

- void popN(R)(R range, int n) - discards the first n elements from
  the buffer, thus causing the next call to frontN() to fetch more
  data if necessary.
On such ranges, what would popFront and front do? I'm assuming since frontN and popN are referring to how many elements, and since the most logical definition for elements is bytes, that front gets the next byte, and popFront discards the next byte. This seems useless to me.
How so? It's still useful for implementing readByte, for example.
readByte is covered by frontN(1). Why the need for front()? Let me answer that question for you -- so it can be treated as a normal range. But nobody will want to do that. i.e. copy to appender will read one byte at a time into the array.
If this new type of range is recognized by std.range, then the relevant algorithms can be made to recognize the existence of frontN and make good use of it, instead of iterating front N times. Then front() can still be used by stuff that really only wants a single byte at a time. [...]
The idea is that by asking for N elements at a time instead of
calling front/popFront N times, the underlying implementation can
optimize the request by creating a buffer of size N and have the OS
read exactly N bytes directly into that buffer.

	// Reads 1,000,000 bytes into newly allocated buffer and returns
	// buffer.
	auto buf = stream.frontN(1_000_000);
OK, so stream is providing data via return value and allocation.
	// Since 1,000,000 bytes is already read into the buffer, this
	// simply returns a slice of the same buffer:
	auto buf2 = stream.frontN(1_000_000);
Is buf2 mutable? If so, this is no good, buf could have mutated this data. But this can be fixed by making the return value of frontN be const(ubyte)[].
	assert(buf is buf2);

	// This consumes the buffer:
	stream.popN(1_000_000);
What does "consume" mean, discard? Obviously not "reuse", due to line below...
Yes, discard. That's what popFront does right now for a single element.
	// This will read another 1,000,000 bytes into a new buffer
	auto buf3 = stream.frontN(1_000_000);
OK, you definitely lost me here, this will not fly. The whole point of buffering is to avoid having to reallocate on every read. If you have to allocate every read, "buffering" is going to have a negative impact on performance!
[...] I thought the whole point of buffering is to avoid excessive roundtrips to disk I/O. Though you do have a point that allocating on every read is a bad idea. T -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -- Michael Beibl
May 16 2012
prev sibling parent "jerro" <a a.com> writes:
 One direction that _could_ be helpful, perhaps, is to extend 
 the concept
 of range to include, let's tentatively call it, a ChunkedRange.
 Basically a ChunkedRange implements the usual InputRange 
 operations
 (empty, front, popfront) but adds the following new primitives:

 - bool hasAtLeast(R)(R range, int n) - true if underlying range 
 has at
   least n elements left;
I think it would be better to have a function that would return the number of elements left.
 - E[] frontN(R)(R range, int n) - returns a slice containing 
 the front n
   elements from the range: this will buffer the next n elements 
 from the
   range if they aren't already; repeated calls will just return 
 the
   buffer;

 - void popN(R)(R range, int n) - discards the first n elements 
 from the
   buffer, thus causing the next call to frontN() to fetch more 
 data if
   necessary.
I like the idea of frontN and popN. But is there any reason why a type that defines those (let's call it a stream) should also be a range? I would prefer to have a type that just defines those two functions, a function that returns the number of available elements and a functions that tells whether we are at the end of stream. If you need a range of elements with a blocking popFront, it's easy to build one on top of it. You can write a functions that takes any stream and returns a range of element. I think that's better than having to write front, popFront, and empty for every stream.
May 16 2012
prev sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 05/16/12 21:38, H. S. Teoh wrote:
 On Wed, May 16, 2012 at 12:48:49PM -0500, Andrei Alexandrescu wrote:
 On 5/16/12 12:34 PM, Steven Schveighoffer wrote:
 In other words, ranges aren't enough.
This is copiously clear to me, but the way I like to think about it is by extending the notion of range (with notions such as e.g. BufferedRange, LookaheadRange, and such) instead of developing an abstraction independent from ranges and then working on stitching that with ranges.
[...] One direction that _could_ be helpful, perhaps, is to extend the concept of range to include, let's tentatively call it, a ChunkedRange. Basically a ChunkedRange implements the usual InputRange operations (empty, front, popfront) but adds the following new primitives: - bool hasAtLeast(R)(R range, int n) - true if underlying range has at least n elements left; - E[] frontN(R)(R range, int n) - returns a slice containing the front n elements from the range: this will buffer the next n elements from the range if they aren't already; repeated calls will just return the buffer; - void popN(R)(R range, int n) - discards the first n elements from the buffer, thus causing the next call to frontN() to fetch more data if necessary. These are all tentative names, of course. But the idea is that you can keep N elements of the range "in view" at a time, without having to individually read them out and save them in a second buffer, and you can advance this view once you're done with the current data and want to move on. Existing range operations like popFrontN, take, takeExactly, drop, etc., can be extended to take advantage of the extra functionality of ChunkedRanges. (Perhaps popFrontN can even be merged with popN, since they amount to the same thing.) Using a ChunkedRange allows you to write functions that parse a particular range and return a range of chunks (say, a deserializer that returns a range of objects given a range of bytes). Thinking on it a bit further, perhaps we can call this a WindowedRange, since it somewhat resembles the sliding window protocol where you keep a "window" of sequential packet ids in an active buffer, and remove them from the buffer as they get ack'ed (consumed by popN). The buffer thus acts like a "window" into the next n elements in the range, which can be "slid forward" as data is consumed.
Right now, everybody reinvents this, with a slightly different interface... It's really obvious, needed and just has to be standardized. A few notes: hasAtLeast is redundant as it can be better expressed as .length; what would be the point of wrapping 'r.length>=n'? An '.available' property would be useful to find eg out how much can be consumed w/o blocking, but that one should return a size_t too. 'E[] frontN' should have a version that returns all available elements; i called it ' property E[] fronts()' here. It's more efficient that way and doesn't rely on the compiler to inline and optimize the limit checks away. PopN -- well, its signature here is 'void popFronts(size_t n)', other than that, there's no difference. Similar things are necessary for output ranges. Here, what i needed was: void put(ref E el) void puts(E[] els) property size_t free() // Not the most intuitive name w/o context; // returns the number of E's that can be 'put()' // w/o blocking. Note that all of this doesn't address the consume-variable-sized-chunks issue. But that can now be efficiently handled by another layer on top. On 05/16/12 22:15, Steven Schveighoffer wrote:
 I still don't get the need to "add" this to ranges.  The streaming API works
fine on its own.
This is not an argument against a streaming API (at least not for me), but for efficient ranges. With the API above I can shift tens of gigabytes of data per second between threads. And still use the 'std' range API and everything that works with it...
 But there is an omission with your proposed API regardless -- reading data is
a mutating event.  It destructively mutates the underlying data stream so that
you cannot get the data again.  This means you must double-buffer data in order
to support frontN and popN that are not necessary with a simple read API.
 
 For example:
 
 auto buf = new ubyte[1000000];
 stream.read(buf);
 
 does not need to first buffer the data inside the stream and then copy it to
buf, it can read it from the OS *directly* into buf.
Sometimes having the buffer managed by 'stream' and 'read()' returning a slice into it works (this is what 'fronts' above does). Reusing a caller managed buffer can be useful in other cases, yes. artur
May 16 2012
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 16 May 2012 16:38:54 -0400, Artur Skawina <art.08.09 gmail.com>  
wrote:

 On 05/16/12 22:15, Steven Schveighoffer wrote:
 I still don't get the need to "add" this to ranges.  The streaming API  
 works fine on its own.
This is not an argument against a streaming API (at least not for me), but for efficient ranges. With the API above I can shift tens of gigabytes of data per second between threads. And still use the 'std' range API and everything that works with it...
But you never would want to. Don't get me wrong, the primitives here could work for a streaming API (I haven't implemented it that way, but it could be made to work). But the idea that it must *also* be a std.range input range makes zero sense. To me, this is as obvious as not supporting linklist[index]; Sure, it can be done, but who would ever use it? -Steve
May 16 2012
parent Artur Skawina <art.08.09 gmail.com> writes:
On 05/16/12 22:58, Steven Schveighoffer wrote:
 On Wed, 16 May 2012 16:38:54 -0400, Artur Skawina <art.08.09 gmail.com> wrote:
 
 On 05/16/12 22:15, Steven Schveighoffer wrote:
 I still don't get the need to "add" this to ranges.  The streaming API works
fine on its own.
This is not an argument against a streaming API (at least not for me), but for efficient ranges. With the API above I can shift tens of gigabytes of data per second between threads. And still use the 'std' range API and everything that works with it...
But you never would want to. Don't get me wrong, the primitives here could work for a streaming API (I haven't implemented it that way, but it could be made to work). But the idea that it must *also* be a std.range input range makes zero sense.
Well, I do want to. For example, I can pass the produced data to *any* range consumer, it may be as efficient as mine, but will still work reasonably (I just did a quick test and the difference seems to be about 10G/s less for plain front+popFront consumer). The goal here is: if we could agree on a standard interface then *any* producer and consumer, including the ones in the std lib could take advantage of this (optional) feature. It's not so much about function call overhead as /syscall/ and /locking/ costs. Retrieving or writing 100 elements with only one lock-unlock sequence makes a large difference.
 To me, this is as obvious as not supporting linklist[index];  Sure, it can be
done, but who would ever use it?
This is not even related. Your 'read(ref ubyte[])' approach can actually mean that one more copy of the data is required. Think writer->range_or_stream->reader -- unless the reader is already waiting with an empty buffer, the stream has to copy the data to an internal buffer, which then has to be copied again when a reader comes around. The 'slice[] = fronts' solution avoids the second copy. Like I said, depending on the circumstances, sometimes you want one scheme, sometimes the other. (TBH, right now i can't think of a case where i'd prefer a non-range based approach; having the same i/f is just so convenient. But I'm sure there's one ;) ) artur
May 16 2012
prev sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
tbh, I've found byChunk to be less than worthless
in my experience; it's a liability because I still
have to wrap it somehow to real real world files.

Consider reading a series of strings in the format
<length><data>,[...].

I'd like it to be this simple (neglecting priming the loop):

string[] s;
while(!file.eof)) {
     ubyte length = file.read!ubyte;
     s ~= file.read!string(length);
}


The C fgetc/fread interface can do this reasonably
well.

string[] s;
while(!feof(fp)) {
    ubyte length = fgetc(fp);
    char[] buffer;
    buffer.length = length;
    fread(buffer.ptr, 1, length, fp);
    s ~= assumeUnique(buffer);
}


But, doing it with byChunk is an exercise in pain
that I don't even feel like writing here.




Another problem is consider a network interface. You
want to handle the packets as they come in.

byChunk doesn't work at all because it blocks until it
gets the chunk of the requested size.

foreach(chunk; socket.byChunk(1024))


suppose you get a packet of length 1000 and you have
to answer it. That will block forever.

So, if you use byChunk as the underlying thing to fill
your buffer... you don't get anywhere.


I think a better input primitive is byPacket(max_size).
This works more like the read primitive on the operating
system.

Moreover, I want it to buffer, and control how much is consumed.


auto packetSource = socket.byPacket(1024);
foreach(packet; packetSource) {
    // as soon as some data comes in we can get the length
    if(packet.length < 2) continue;
    auto length = packet.peek!(ushort); // neglect endian for now
    if(packet.length < length + 2) continue; // wait for more data

    packet.consume(2);
    handle(packet.consume(length));
}



In addition to the byChunk blocking problem...
what if the length straddles the edge?



byChunk is just a huge hassle to work with for every file
format I've tried so far. byLine is a little better
(some file formats are defined as being line based)
but still a bit of a pain for anything that can spill
into two lines.
May 16 2012
prev sibling parent reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 13-05-2012 23:38, Walter Bright wrote:
 This discussion started in the thread "Getting the const-correctness of
 Object sorted once and for all", but it deserved its own thread.

 These modules suffer from the following problems:

 1. poor documentation, dearth of examples & rationale

 2. toHash(), toString(), etc., all need to be const pure nothrow, but
 it's turning out to be problematic for doing it for these classes

 3. overlapping functionality with std.stdio

 4. they should present a range interface, not a streaming one
While we're at it, do we want to keep std.outbuffer? -- - Alex
May 14 2012
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2012 6:29 PM, Alex Rønne Petersen wrote:
 While we're at it, do we want to keep std.outbuffer?
Since it's not range based, probably not.
May 14 2012
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, May 14, 2012 at 07:57:28PM -0700, Walter Bright wrote:
 On 5/14/2012 6:29 PM, Alex Rønne Petersen wrote:
While we're at it, do we want to keep std.outbuffer?
Since it's not range based, probably not.
Why not just fold this into std.io? I'm surprised that this is a separate module, actually. It should either be folded into std.io, or developed to be more generic (i.e., have range-based API, have more features like auto-flushing past a certain size, etc.). T -- Prosperity breeds contempt, and poverty breeds consent. -- Suck.com
May 14 2012
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2012 9:54 PM, H. S. Teoh wrote:
 On Mon, May 14, 2012 at 07:57:28PM -0700, Walter Bright wrote:
 On 5/14/2012 6:29 PM, Alex Rønne Petersen wrote:
 While we're at it, do we want to keep std.outbuffer?
Since it's not range based, probably not.
Why not just fold this into std.io?
It's not I/O.
May 14 2012
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 15.05.2012 8:54, H. S. Teoh wrote:
 On Mon, May 14, 2012 at 07:57:28PM -0700, Walter Bright wrote:
 On 5/14/2012 6:29 PM, Alex Rønne Petersen wrote:
 While we're at it, do we want to keep std.outbuffer?
Since it's not range based, probably not.
Why not just fold this into std.io? I'm surprised that this is a separate module, actually. It should either be folded into std.io, or developed to be more generic (i.e., have range-based API, have more features like auto-flushing past a certain size, etc.).
It's std.array Appender. The only difference is text vs binary output form. -- Dmitry Olshansky
May 15 2012