digitalmars.D.learn - iopipe: Writing output to std.io File
- Jesse Phillips (29/29) Jan 26 2020 I'd like to start utilizing IOPipe[1] more. Right now I have an
- Jesse Phillips (24/24) Jan 26 2020 Just as I'm hitting send the part I'm missing clicked:
- Jesse Phillips (2/14) Jan 26 2020 Unfortunately this did not write a text file as I expected.
- Steven Schveighoffer (42/59) Jan 27 2020 This is because output is buffered differently from input. With output
- Steven Schveighoffer (4/12) Jan 27 2020 Ugh, should have tested.
- Jesse Phillips (6/31) Jan 27 2020 I really feel like this is all very well thought out and clean, I
- Steven Schveighoffer (15/19) Jan 28 2020 Thanks! Tee is actually a pretty close approximation. For example, if
- Jesse Phillips (12/16) Jan 29 2020 That statement I think will be very helpful to me.
- Steven Schveighoffer (14/32) Jan 31 2020 The push function divides the buffer into 2 parts, the "writeable" items...
I'd like to start utilizing IOPipe[1] more. Right now I have an interest in utilizing it for buffering output (actually I don't have a need for buffering, I just want to utilize iopipe) Looking through some different examples[2][3] I thought I would have something with this: ```dlang /+ dub.sdl: name "iobuftofile" dependency "iopipe" version="~>0.1.7" dependency "io" version="~>0.2.4" +/ void main() { import iopipe.valve; import iopipe.textpipe; import iopipe.bufpipe; import iopipe.buffer; import std.io; import std.typecons; auto output() { return std.io.File("somefile.txt", mode!"w").refCounted; } auto outputBuffered = bufd!char .outputPipe(output()); } ``` 1. https://code.dlang.org/packages/iopipe 2. https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d#L13 3. https://youtu.be/9fzttyj4JCs?t=1210
Jan 26 2020
Just as I'm hitting send the part I'm missing clicked: I needed to add the text encoding because my buffer is `char` but File writes `ubyte` ```dlang /+ dub.sdl: name "iobuftofile" dependency "iopipe" version="~>0.1.7" dependency "io" version="~>0.2.4" +/ void main() { import iopipe.valve; import iopipe.textpipe; import iopipe.bufpipe; import iopipe.buffer; import std.io; import std.typecons; auto output() { return std.io.File("somefile.txt", mode!"w").refCounted; } auto outputBuffered = bufd!char .encodeText // Missing This part .outputPipe(output()); } ```
Jan 26 2020
On Monday, 27 January 2020 at 01:50:00 UTC, Jesse Phillips wrote:Just as I'm hitting send the part I'm missing clicked: I needed to add the text encoding because my buffer is `char` but File writes `ubyte` ```dlang auto output() { return std.io.File("somefile.txt", mode!"w").refCounted; } auto outputBuffered = bufd!char .encodeText // Missing This part .outputPipe(output()); } ```Unfortunately this did not write a text file as I expected.
Jan 26 2020
On 1/26/20 11:59 PM, Jesse Phillips wrote:On Monday, 27 January 2020 at 01:50:00 UTC, Jesse Phillips wrote:This is because output is buffered differently from input. With output you are writing to the buffer, then flushing the result as needed. But I constructed iopipe to be able to utilize all mechanisms as both input and output. Before I show you what to do, let me explain what the above actually does. 1. You constructed a buffer of characters. Good, this is the first step. 2. You used encodeText to convert the data to ubyte. Note that for char buffer, this basically is just a cast. Other encodings might do byteswapping. But you have done this step a bit early. At this point now, the window type is ubyte[]. You want to put data into the buffer as char[]. 3. You appended an outputPipe, which is a pass through while writing the data to a file (which is fed a stream of uninitialized data I think, so you probably got a file with garbage in it). Writing to this pipe's buffer is kind of pointless because the writing has already happened (pretty much all effects and actions performed on the buffer happen in the .extend function). What you need is to access the buffer for writing BEFORE encoding and streaming to the file. For this, you need the push [1] mechanism, which wraps up everything you want to happen to data you have written to the buffer into a lambda template: auto outputBuffered = bufd!char .push!(p => p .encodeText .outputPipe(output())); now, you write to the outputBuffered "buffer" as needed. releasing any data that is ready to go. As soon as it needs more buffer space and there is pending data, it will write the data, giving you back the buffer space. It also auto-flushes on the last reference destructor (this behavior is configurable). A simple writeln on such a pipe would look like: void writeln(Pipe)(ref Pipe sink, string text) { enforce(sink.ensureElems(text.size) >= text.length); // make sure there's enough buffer space to hold the text sink[0 .. text.length] = text; // write to the buffer sink.release(text.length); // release to be written } I really need to do an article on iopipe. I've been neglecting that... -Steve [1] http://schveiguy.github.io/iopipe/iopipe/valve/push.htmlJust as I'm hitting send the part I'm missing clicked: I needed to add the text encoding because my buffer is `char` but File writes `ubyte` ```dlang auto output() { return std.io.File("somefile.txt", mode!"w").refCounted; } auto outputBuffered = bufd!char .encodeText // Missing This part .outputPipe(output()); } ```Unfortunately this did not write a text file as I expected.
Jan 27 2020
On 1/27/20 1:12 PM, Steven Schveighoffer wrote:void writeln(Pipe)(ref Pipe sink, string text) { enforce(sink.ensureElems(text.size) >= text.length); // make sure there's enough buffer space to hold the text sink[0 .. text.length] = text; // write to the buffer sink.release(text.length); // release to be written }Ugh, should have tested. sink.window[0 .. text.length] = text; -Steve
Jan 27 2020
On Monday, 27 January 2020 at 18:12:40 UTC, Steven Schveighoffer wrote:Before I show you what to do, let me explain what the above actually does. 1. You constructed a buffer of characters. Good, this is the first step. 2. You used encodeText to convert the data to ubyte. Note that for char buffer, this basically is just a cast. Other encodings might do byteswapping. But you have done this step a bit early. At this point now, the window type is ubyte[]. You want to put data into the buffer as char[]. 3. You appended an outputPipe, which is a pass through while writing the data to a file (which is fed a stream of uninitialized data I think, so you probably got a file with garbage in it). Writing to this pipe's buffer is kind of pointless because the writing has already happened (pretty much all effects and actions performed on the buffer happen in the .extend function). What you need is to access the buffer for writing BEFORE encoding and streaming to the file. For this, you need the push [1] mechanism, which wraps up everything you want to happen to data you have written to the buffer into a lambda template: auto outputBuffered = bufd!char .push!(p => p .encodeText .outputPipe(output())); [1] http://schveiguy.github.io/iopipe/iopipe/valve/push.htmlI really feel like this is all very well thought out and clean, I don't appear to have a previous model to help visualize this output approach. Right now something like tee is coming to mind. Thank you for explaining with the answer.
Jan 27 2020
On 1/28/20 1:25 AM, Jesse Phillips wrote:I really feel like this is all very well thought out and clean, I don't appear to have a previous model to help visualize this output approach. Right now something like tee is coming to mind. Thank you for explaining with the answer.Thanks! Tee is actually a pretty close approximation. For example, if you wanted to save to 2 files, you could do: push!(p => p .encodeText .outputPipe(file1) .outputPipe(file2) ) The whole advantage is that you don't need to use the output pipes as buffered output. For example, the convert function doesn't need them, it just pulls the whole thing (using the process function). Everything is pulled with iopipe, even output, so it's just a matter of who is pulling and when. Pushing is a matter of telling the other end to pull. -Steve
Jan 28 2020
On Tuesday, 28 January 2020 at 16:09:55 UTC, Steven Schveighoffer wrote:Everything is pulled with iopipe, even output, so it's just a matter of who is pulling and when. Pushing is a matter of telling the other end to pull. -SteveThat statement I think will be very helpful to me. The push would control the buffer, creating that value concept, where the buffer is flush which creates a pull, specified in the delegate. Pusher(buffer) <- put(content) An output range wrapper would provide a push interface which would fill in the buffer of the Pusher. When the buffer fills the range wrapper would ask to release which Pusher does by calling the delegate. Hopefully I'm following this correctly.
Jan 29 2020
On 1/30/20 12:59 AM, Jesse Phillips wrote:On Tuesday, 28 January 2020 at 16:09:55 UTC, Steven Schveighoffer wrote:The push function divides the buffer into 2 parts, the "writeable" items (i.e. items ready to push), and the buffer that can be accessed for manipulation (this is where you put your data). Eventually, the latter fills up and you need more buffer space. At this point, it performs the push, which is done by running the push chain as if it were an input stream, where the data source is the writeable data.Everything is pulled with iopipe, even output, so it's just a matter of who is pulling and when. Pushing is a matter of telling the other end to pull.That statement I think will be very helpful to me. The push would control the buffer, creating that value concept, where the buffer is flush which creates a pull, specified in the delegate.Pusher(buffer) <- put(content) An output range wrapper would provide a push interface which would fill in the buffer of the Pusher. When the buffer fills the range wrapper would ask to release which Pusher does by calling the delegate.An output range wrapper should fill in the accessible buffer and then immediately release it (same as that writeln example I showed before).Hopefully I'm following this correctly.I was going to write an ascii art concept to show how the pushing works, but I think I'll maybe draw an actual picture. I need some time to accomplish this, though. It's very clear in my head how this works, but very hard to describe :) -Steve
Jan 31 2020