www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - OutputRanges and slicing/save()

reply Justin Whear <justin economicmodeling.com> writes:
I've run into a design issue surrounding ranges and am looking for advice 
on the best way to proceed.  To illustrate the issue, consider the 
Shapefile format: a 100 byte header followed by variable-length records.  
The tricky bit is that the header includes a field which contains the 
total length of the file (as measured in 16-bit words, curiously).  The 
header must be written first, but the total length of the file isn't 
known until all the records have been encoded.  When writing to a File 
this isn't a problem: write 100 bytes of padding, write the records, use 
rewind(), and write the proper header.  It's in the context of an 
OutputRange that I don't know how to proceed.  Consider the most flexible 
range type: the array.  An array is not an OutputRange, so it needs to be 
wrapped in something like std.array.Appender.  Ideally I could save off 
the initial state of the range, write a bogus header, write the records, 
then jump back and write the proper header.  Unfortunately, Appender is 
not a ForwardRange, nor does it appear that the field of OutputRanges 
which are also ForwardRanges has been explored.  I'm using the excellent 
read, write, and append functions from std.bitmanip, so write() would fit 
the bill if only Appender supported slicing.

My current solution is require the user to construct the ShapeWriter 
output range (which supports `put(Shape)`) over two separate output 
ranges of ubyte: one for the header and a second for the records, then 
delay writing to the header range until the record range is complete.  
This is both needlessly complex and it leaves the proper combination of 
the two to the user, making ShapeWriter a very leaky abstraction.

So, for particular questions:
1) Am I missing something in Phobos that would provide an OutputRange of 
ubytes while also providing ForwardRange/slicing capabilities?
2) I know the relationship between streams and ranges has been discussed 
at least once before; is the concept of rewinding and overwriting simply 
incompatible with OutputRange in general?

Justin
Jan 03 2014
next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 3 January 2014 at 22:56:41 UTC, Justin Whear wrote:
 I've run into a design issue surrounding ranges and am looking 
 for advice
 on the best way to proceed.  To illustrate the issue, consider 
 the
 Shapefile format: a 100 byte header followed by variable-length 
 records.
 The tricky bit is that the header includes a field which 
 contains the
 total length of the file (as measured in 16-bit words, 
 curiously).  The
 header must be written first, but the total length of the file 
 isn't
 known until all the records have been encoded.  When writing to 
 a File
 this isn't a problem: write 100 bytes of padding, write the 
 records, use
 rewind(), and write the proper header.  It's in the context of 
 an
 OutputRange that I don't know how to proceed.  Consider the 
 most flexible
 range type: the array.  An array is not an OutputRange, so it 
 needs to be
 wrapped in something like std.array.Appender.

Actaully, dynamic arrays *are* output ranges. Each "put" places the element at the front of the range, and the range is then pop'ed front. Its' not an "expanding" output range, rather, a "fillable" output range, so not really something that would fit your need.
 Ideally I could save off
 the initial state of the range, write a bogus header, write the 
 records,
 then jump back and write the proper header.  Unfortunately, 
 Appender is
 not a ForwardRange, nor does it appear that the field of 
 OutputRanges
 which are also ForwardRanges has been explored.  I'm using the 
 excellent
 read, write, and append functions from std.bitmanip, so write() 
 would fit
 the bill if only Appender supported slicing.

 My current solution is require the user to construct the 
 ShapeWriter
 output range (which supports `put(Shape)`) over two separate 
 output
 ranges of ubyte: one for the header and a second for the 
 records, then
 delay writing to the header range until the record range is 
 complete.
 This is both needlessly complex and it leaves the proper 
 combination of
 the two to the user, making ShapeWriter a very leaky 
 abstraction.

 So, for particular questions:
 1) Am I missing something in Phobos that would provide an 
 OutputRange of
 ubytes while also providing ForwardRange/slicing capabilities?

<pedantic>ubyte[] is such a range</pedantic>. It won't do what you want though.
 2) I know the relationship between streams and ranges has been 
 discussed
 at least once before; is the concept of rewinding and 
 overwriting simply
 incompatible with OutputRange in general?

 Justin

I think the biggest issue is that there is actaully 0 relation between input ranges and output ranges. The two concepts are completely orthogonal. In particular, arguably, input ranges that are also output ranges are confusing, since rather than "growing as you add items to their tip (what you'd want)", they instead "shrink as you overwrite their front, until they are empty/full". When using put(range, item), the "put" primitive of "range" takes precedence over "front". You could design a range that uses that, which would look a bit more like what you want, but there is nothing that exists in phobos that does this that I know of anyways. It's a bit of a mess (IMO).
Jan 03 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 3 January 2014 at 23:14:31 UTC, monarch_dodra wrote:
 In particular, arguably, input ranges that are also output 
 ranges are confusing,  since rather than "growing as you add 
 items to their tip (what you'd want)", they instead "shrink as 
 you overwrite their front, until they are empty/full".

To add to that, it's *kind* of like an input/output stream, that you can only write to, after you have read it...
Jan 03 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-01-03 23:56, Justin Whear wrote:
 I've run into a design issue surrounding ranges and am looking for advice
 on the best way to proceed.  To illustrate the issue, consider the
 Shapefile format: a 100 byte header followed by variable-length records.
 The tricky bit is that the header includes a field which contains the
 total length of the file (as measured in 16-bit words, curiously).  The
 header must be written first, but the total length of the file isn't
 known until all the records have been encoded.  When writing to a File
 this isn't a problem: write 100 bytes of padding, write the records, use
 rewind(), and write the proper header.  It's in the context of an
 OutputRange that I don't know how to proceed.  Consider the most flexible
 range type: the array.  An array is not an OutputRange, so it needs to be
 wrapped in something like std.array.Appender.  Ideally I could save off
 the initial state of the range, write a bogus header, write the records,
 then jump back and write the proper header.  Unfortunately, Appender is
 not a ForwardRange, nor does it appear that the field of OutputRanges
 which are also ForwardRanges has been explored.  I'm using the excellent
 read, write, and append functions from std.bitmanip, so write() would fit
 the bill if only Appender supported slicing.

If you're satisfied with just using arrays and not a general output range you can do something similar to what you did with a file. Add the 100 extra bytes for the header to Appender, then add the contents. Extract the array used for backing in the Appender and fill the proper header. -- /Jacob Carlborg
Jan 04 2014
parent "Justin Whear" <justin economicmodeling.com> writes:
On Saturday, 4 January 2014 at 11:22:03 UTC, Jacob Carlborg wrote:
 On 2014-01-03 23:56, Justin Whear wrote:
 I've run into a design issue surrounding ranges and am looking 
 for advice
 on the best way to proceed.  To illustrate the issue, consider 
 the
 Shapefile format: a 100 byte header followed by 
 variable-length records.
 The tricky bit is that the header includes a field which 
 contains the
 total length of the file (as measured in 16-bit words, 
 curiously).  The
 header must be written first, but the total length of the file 
 isn't
 known until all the records have been encoded.  When writing 
 to a File
 this isn't a problem: write 100 bytes of padding, write the 
 records, use
 rewind(), and write the proper header.  It's in the context of 
 an
 OutputRange that I don't know how to proceed.  Consider the 
 most flexible
 range type: the array.  An array is not an OutputRange, so it 
 needs to be
 wrapped in something like std.array.Appender.  Ideally I could 
 save off
 the initial state of the range, write a bogus header, write 
 the records,
 then jump back and write the proper header.  Unfortunately, 
 Appender is
 not a ForwardRange, nor does it appear that the field of 
 OutputRanges
 which are also ForwardRanges has been explored.  I'm using the 
 excellent
 read, write, and append functions from std.bitmanip, so 
 write() would fit
 the bill if only Appender supported slicing.

If you're satisfied with just using arrays and not a general output range you can do something similar to what you did with a file. Add the 100 extra bytes for the header to Appender, then add the contents. Extract the array used for backing in the Appender and fill the proper header.

I guess my larger concern is providing an interface which avoids allocation. In that case, having an overload which accepts an array for writing into and signals if it was too small could work. It is a bit disappointing to lose the nice generic interface which would really give the user control.
Jan 04 2014