www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - iopipe: Writing output to std.io File

reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
I'd like to start utilizing IOPipe[1] more. Right now I have an 
interest in utilizing it for buffering output (actually I don't 
have a need for buffering, I just want to utilize iopipe)

Looking through some different examples[2][3] I thought I would 
have something with this:

```dlang
/+ dub.sdl:
     name "iobuftofile"
     dependency "iopipe" version="~>0.1.7"
     dependency "io" version="~>0.2.4"
+/

void main() {
     import iopipe.valve;
     import iopipe.textpipe;
     import iopipe.bufpipe;
     import iopipe.buffer;
     import std.io;
     import std.typecons;
     auto output() {
         return std.io.File("somefile.txt", mode!"w").refCounted;
     }
     auto outputBuffered = bufd!char
         .outputPipe(output());
}
```

1. https://code.dlang.org/packages/iopipe
2. 
https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d#L13
3. https://youtu.be/9fzttyj4JCs?t=1210
Jan 26 2020
parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
Just as I'm hitting send the part I'm missing clicked:

I needed to add the text encoding because my buffer is `char` but 
File writes `ubyte`

```dlang
/+ dub.sdl:
     name "iobuftofile"
     dependency "iopipe" version="~>0.1.7"
     dependency "io" version="~>0.2.4"
+/

void main() {
     import iopipe.valve;
     import iopipe.textpipe;
     import iopipe.bufpipe;
     import iopipe.buffer;
     import std.io;
     import std.typecons;
     auto output() {
         return std.io.File("somefile.txt", mode!"w").refCounted;
     }
     auto outputBuffered = bufd!char
         .encodeText // Missing This part
         .outputPipe(output());
}
```
Jan 26 2020
parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 27 January 2020 at 01:50:00 UTC, Jesse Phillips wrote:

 Just as I'm hitting send the part I'm missing clicked:

 I needed to add the text encoding because my buffer is `char` 
 but File writes `ubyte`

 ```dlang
     auto output() {
         return std.io.File("somefile.txt", mode!"w").refCounted;
     }
     auto outputBuffered = bufd!char
         .encodeText // Missing This part
         .outputPipe(output());
 }
 ```


Unfortunately this did not write a text file as I expected.
Jan 26 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/26/20 11:59 PM, Jesse Phillips wrote:

 On Monday, 27 January 2020 at 01:50:00 UTC, Jesse Phillips wrote:

 Just as I'm hitting send the part I'm missing clicked:

 I needed to add the text encoding because my buffer is `char` but File 
 writes `ubyte`

 ```dlang
     auto output() {
         return std.io.File("somefile.txt", mode!"w").refCounted;
     }
     auto outputBuffered = bufd!char
         .encodeText // Missing This part
         .outputPipe(output());
 }
 ```


 
 Unfortunately this did not write a text file as I expected.


This is because output is buffered differently from input. With output 
you are writing to the buffer, then flushing the result as needed. But I 
constructed iopipe to be able to utilize all mechanisms as both input 
and output.

Before I show you what to do, let me explain what the above actually does.

1. You constructed a buffer of characters. Good, this is the first step.
2. You used encodeText to convert the data to ubyte. Note that for char 
buffer, this basically is just a cast. Other encodings might do 
byteswapping. But you have done this step a bit early. At this point 
now, the window type is ubyte[]. You want to put data into the buffer as 
char[].
3. You appended an outputPipe, which is a pass through while writing the 
data to a file (which is fed a stream of uninitialized data I think, so 
you probably got a file with garbage in it). Writing to this pipe's 
buffer is kind of pointless because the writing has already happened 
(pretty much all effects and actions performed on the buffer happen in 
the .extend function).

What you need is to access the buffer for writing BEFORE encoding and 
streaming to the file. For this, you need the push [1] mechanism, which 
wraps up everything you want to happen to data you have written to the 
buffer into a lambda template:

auto outputBuffered = bufd!char
    .push!(p => p
       .encodeText
       .outputPipe(output()));

now, you write to the outputBuffered "buffer" as needed. releasing any 
data that is ready to go. As soon as it needs more buffer space and 
there is pending data, it will write the data, giving you back the 
buffer space. It also auto-flushes on the last reference destructor 
(this behavior is configurable).

A simple writeln on such a pipe would look like:

void writeln(Pipe)(ref Pipe sink, string text)
{
    enforce(sink.ensureElems(text.size) >= text.length); // make sure 
there's enough buffer space to hold the text
    sink[0 .. text.length] = text; // write to the buffer
    sink.release(text.length); // release to be written
}

I really need to do an article on iopipe. I've been neglecting that...

-Steve

[1] http://schveiguy.github.io/iopipe/iopipe/valve/push.html
Jan 27 2020
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/27/20 1:12 PM, Steven Schveighoffer wrote:


 void writeln(Pipe)(ref Pipe sink, string text)
 {
     enforce(sink.ensureElems(text.size) >= text.length); // make sure 
 there's enough buffer space to hold the text
     sink[0 .. text.length] = text; // write to the buffer
     sink.release(text.length); // release to be written
 }
 


Ugh, should have tested.

sink.window[0 .. text.length] = text;

-Steve
Jan 27 2020
prev sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 27 January 2020 at 18:12:40 UTC, Steven Schveighoffer 
wrote:

 Before I show you what to do, let me explain what the above 
 actually does.

 1. You constructed a buffer of characters. Good, this is the 
 first step.
 2. You used encodeText to convert the data to ubyte. Note that 
 for char buffer, this basically is just a cast. Other encodings 
 might do byteswapping. But you have done this step a bit early. 
 At this point now, the window type is ubyte[]. You want to put 
 data into the buffer as char[].
 3. You appended an outputPipe, which is a pass through while 
 writing the data to a file (which is fed a stream of 
 uninitialized data I think, so you probably got a file with 
 garbage in it). Writing to this pipe's buffer is kind of 
 pointless because the writing has already happened (pretty much 
 all effects and actions performed on the buffer happen in the 
 .extend function).

 What you need is to access the buffer for writing BEFORE 
 encoding and streaming to the file. For this, you need the push 
 [1] mechanism, which wraps up everything you want to happen to 
 data you have written to the buffer into a lambda template:

 auto outputBuffered = bufd!char
    .push!(p => p
       .encodeText
       .outputPipe(output()));

 [1] http://schveiguy.github.io/iopipe/iopipe/valve/push.html


I really feel like this is all very well thought out and clean, I 
don't appear to have a previous model to help visualize this 
output approach. Right now something like tee is coming to mind. 
Thank you for explaining with the answer.
Jan 27 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/28/20 1:25 AM, Jesse Phillips wrote:


 I really feel like this is all very well thought out and clean, I don't 
 appear to have a previous model to help visualize this output approach. 
 Right now something like tee is coming to mind. Thank you for explaining 
 with the answer.


Thanks! Tee is actually a pretty close approximation. For example, if 
you wanted to save to 2 files, you could do:

push!(p => p
    .encodeText
    .outputPipe(file1)
    .outputPipe(file2)
)

The whole advantage is that you don't need to use the output pipes as 
buffered output. For example, the convert function doesn't need them, it 
just pulls the whole thing (using the process function).

Everything is pulled with iopipe, even output, so it's just a matter of 
who is pulling and when. Pushing is a matter of telling the other end to 
pull.

-Steve
Jan 28 2020
parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 28 January 2020 at 16:09:55 UTC, Steven Schveighoffer 
wrote:

 Everything is pulled with iopipe, even output, so it's just a 
 matter of who is pulling and when. Pushing is a matter of 
 telling the other end to pull.

 -Steve


That statement I think will be very helpful to me.

The push would control the buffer, creating that value concept, 
where the buffer is flush which creates a pull, specified in the 
delegate.

Pusher(buffer) <- put(content)

An output range wrapper would provide a push interface which 
would fill in the buffer of the Pusher. When the buffer fills the 
range wrapper would ask to release which Pusher does by calling 
the delegate.

Hopefully I'm following this correctly.
Jan 29 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/30/20 12:59 AM, Jesse Phillips wrote:

 On Tuesday, 28 January 2020 at 16:09:55 UTC, Steven Schveighoffer wrote:

 Everything is pulled with iopipe, even output, so it's just a matter 
 of who is pulling and when. Pushing is a matter of telling the other 
 end to pull.


 
 That statement I think will be very helpful to me.
 
 The push would control the buffer, creating that value concept, where 
 the buffer is flush which creates a pull, specified in the delegate.


The push function divides the buffer into 2 parts, the "writeable" items 
(i.e. items ready to push), and the buffer that can be accessed for 
manipulation (this is where you put your data). Eventually, the latter 
fills up and you need more buffer space. At this point, it performs the 
push, which is done by running the push chain as if it were an input 
stream, where the data source is the writeable data.


 
 Pusher(buffer) <- put(content)
 
 An output range wrapper would provide a push interface which would fill 
 in the buffer of the Pusher. When the buffer fills the range wrapper 
 would ask to release which Pusher does by calling the delegate.


An output range wrapper should fill in the accessible buffer and then 
immediately release it (same as that writeln example I showed before).


 Hopefully I'm following this correctly.


I was going to write an ascii art concept to show how the pushing works, 
but I think I'll maybe draw an actual picture. I need some time to 
accomplish this, though.

It's very clear in my head how this works, but very hard to describe :)

-Steve
Jan 31 2020