www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - iopipe v0.1.0 - now with Windows support!

reply Steven Schveighoffer <schveiguy yahoo.com> writes:
iopipe version 0.1.0 has been released.

iopipe is a high-performance pipe processing system that makes it easy 
to string together pipelines to process data with as little buffer 
copying as possible.

Nothing has really been changed, but it now has Windows i/o support. I 
will note at this time, however, that ring buffers are not yet supported 
on Windows.

This version deprecates the IODev type that I had included, in favor of 
a new io library that shows extreme promise.

This version ONLY builds on 2.080.1 or later (the bug fix that I 
submitted at dconf has been merged in that version, and so iopipe will 
now build against Martin Nowak's io library). In fact, iopipe 
development was kind of stalled due to this bug, so I'm super-happy to 
see it fixed and released!

Note that the new io library also supports sockets, which IODev did not 
have support for, AND has a pluggable driver system, so you could 
potentially use fiber-based async io without rebuilding. It just makes a 
lot of sense for D to have a standard low-level io library that 
everything can use without having to kludge together multiple types of 
io libraries.

Near future plans:

1. Utilize a CI to make sure it continues to work on all platforms.
2. Add RingBuffer support on Windows
3. Add more driver support for std.io.
4. Continue development of JSON library that depends on iopipe (not yet 
on code.dlang.org).

git - https://github.com/schveiguy/iopipe
dub - https://code.dlang.org/packages/iopipe
docs - http://schveiguy.github.io/iopipe/

-Steve
Jun 10
next sibling parent reply Dejan Lekic <dejan.lekic gmail.com> writes:
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer 
wrote:
 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes 
 it easy to string together pipelines to process data with as 
 little buffer copying as possible.
All I can say (again, like I repeated on IRC many times) is that iopipe should be in the current form, or another, in Phobos. I just love it!
Jun 11
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/11/18 6:14 AM, Dejan Lekic wrote:
 On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:
 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes it easy 
 to string together pipelines to process data with as little buffer 
 copying as possible.
All I can say (again, like I repeated on IRC many times) is that iopipe should be in the current form, or another, in Phobos. I just love it!
Thanks for the kind words! If Martin's std.io makes it in to Phobos, there is a realistic chance iopipe could go as well. Just curious, do you have any projects based on iopipe? -Steve
Jun 11
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/10/18 4:10 PM, Steven Schveighoffer wrote:

 Nothing has really been changed, but it now has Windows i/o support. I 
 will note at this time, however, that ring buffers are not yet supported 
 on Windows.
 
I just pushed v0.1.1 -- I realized that I never *actually* compiled on windows, and there were a couple things that didn't work. Note: the examples still don't work as they rely on openDev, which is only available on Posix systems now. I need to figure out a good way to open stdin/stdout in a cross platform way with std.io. -Steve
Jun 11
parent reply Jacob Carlborg <doob me.com> writes:
On 2018-06-11 16:45, Steven Schveighoffer wrote:

 I just pushed v0.1.1 -- I realized that I never *actually* compiled on
 windows, and there were a couple things that didn't work.

 Note: the examples still don't work as they rely on openDev, which is
 only available on Posix systems now.

 I need to figure out a good way to open stdin/stdout in a cross platform
 way with std.io.
You should setup AppVeyor [1] to make it works on Windows (when it works). [1] https://www.appveyor.com -- /Jacob Carlborg
Jun 19
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/19/18 7:18 AM, Jacob Carlborg wrote:
 On 2018-06-11 16:45, Steven Schveighoffer wrote:
 
 I just pushed v0.1.1 -- I realized that I never *actually* compiled on
 windows, and there were a couple things that didn't work.

 Note: the examples still don't work as they rely on openDev, which is
 only available on Posix systems now.

 I need to figure out a good way to open stdin/stdout in a cross platform
 way with std.io.
You should setup AppVeyor [1] to make it works on Windows (when it works). [1] https://www.appveyor.com
I just set up travis to do the Linux/mac testing. I need to add appveyor as well, but haven't gotten to it. I'm a complete CI noob, so I'm learning slowly :) -Steve
Jun 19
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2018-06-19 15:04, Steven Schveighoffer wrote:

 I just set up travis to do the Linux/mac testing. I need to add appveyor
 as well, but haven't gotten to it. I'm a complete CI noob, so I'm
 learning slowly :)
To save you some trouble, AppVeyor supports both a YAML, like Travis, and a web UI to configure the CI system. If you're not including some parts of the YAML file, like "build_script", it will use the default, which is preforming some Visual Studio specific task. You also need to download the D compiler manually since AppVeyor doesn't have built-in support for D the same way as Travis. -- /Jacob Carlborg
Jun 21
prev sibling parent Seb <seb wilzba.ch> writes:
On Tuesday, 19 June 2018 at 13:04:24 UTC, Steven Schveighoffer 
wrote:
 On 6/19/18 7:18 AM, Jacob Carlborg wrote:
 On 2018-06-11 16:45, Steven Schveighoffer wrote:
 
 I just pushed v0.1.1 -- I realized that I never *actually* 
 compiled on
 windows, and there were a couple things that didn't work.

 Note: the examples still don't work as they rely on openDev, 
 which is
 only available on Posix systems now.

 I need to figure out a good way to open stdin/stdout in a 
 cross platform
 way with std.io.
You should setup AppVeyor [1] to make it works on Windows (when it works). [1] https://www.appveyor.com
I just set up travis to do the Linux/mac testing. I need to add appveyor as well, but haven't gotten to it. I'm a complete CI noob, so I'm learning slowly :) -Steve
Just copy/paste one of the existing configuration scripts and you should be good to go, e.g. https://github.com/dlang-community/DCD/blob/master/appveyor.yml And there's also: https://github.com/Abscissa/AppVeyor-D
Jun 21
prev sibling next sibling parent reply DigitalDesigns <DigitalDesigns gmail.com> writes:
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer 
wrote:
 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes 
 it easy to string together pipelines to process data with as 
 little buffer copying as possible.

 Nothing has really been changed, but it now has Windows i/o 
 support. I will note at this time, however, that ring buffers 
 are not yet supported on Windows.

 This version deprecates the IODev type that I had included, in 
 favor of a new io library that shows extreme promise.

 This version ONLY builds on 2.080.1 or later (the bug fix that 
 I submitted at dconf has been merged in that version, and so 
 iopipe will now build against Martin Nowak's io library). In 
 fact, iopipe development was kind of stalled due to this bug, 
 so I'm super-happy to see it fixed and released!

 Note that the new io library also supports sockets, which IODev 
 did not have support for, AND has a pluggable driver system, so 
 you could potentially use fiber-based async io without 
 rebuilding. It just makes a lot of sense for D to have a 
 standard low-level io library that everything can use without 
 having to kludge together multiple types of io libraries.

 Near future plans:

 1. Utilize a CI to make sure it continues to work on all 
 platforms.
 2. Add RingBuffer support on Windows
 3. Add more driver support for std.io.
 4. Continue development of JSON library that depends on iopipe 
 (not yet on code.dlang.org).

 git - https://github.com/schveiguy/iopipe
 dub - https://code.dlang.org/packages/iopipe
 docs - http://schveiguy.github.io/iopipe/

 -Steve
Could you explain some benefits specific to this implementation and a bit of the functional aspects for a proper overview of it's capabilities and why I should chose this method over others?
Jun 11
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/12/18 1:51 AM, DigitalDesigns wrote:
 
 Could you explain some benefits specific to this implementation and a 
 bit of the functional aspects for a proper overview of it's capabilities 
 and why I should chose this method over others?
The things that I think make this approach better are: 1. Direct buffer access Direct buffer access I have found is one of those ideas that doesn't seem like it's that impressive until you start using it. Many times, buffering is used in a generic library for optimization (basically amortizing reads) and is an implementation detail hidden from your view. Think of how FILE * keeps a large buffer of data inside itself, but only gives you access to one character at a time. This forces you to create your *own* buffering scheme on top of that. What a waste! Iopipe allows you to use buffering for your purposes on top of the benefits of amortization. It's my belief that this is why iopipe's byline feature is 2x faster than Phobos'. 2. Using templates to their fullest Iopipes are all templated on the buffer or iopipe underneath it. This makes tings easily swappable. It's really cool to be able to take your JSON or XML parser, and hook it onto an in-memory string in one line, and then hook it onto a socket, and everything is optimized for that situation. It takes the fun and flexibility of range programming and brings it to i/o. This is why iopipe's byline handles all forms of UTF, compared to Phobos which only handles UTF8. For example, I handle all forms of UTF with iopipe, with a decent set of utilities. Here is a complete program using iopipe that converts any type of UTF into another type, optimized for the specific situation: https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d 3. Compiler optimization for everything All parts of iopipe, except for the low-level reads and writes (which ironically are not really part of iopipe) are visible to the compiler for inlining and optimization. I'm leveraging the power of the decades of optimization experience that the compiler can provide. This makes it easy to write code that performs well. An anecdote: For my talk on iopipe in 2017 (http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live demo showing the performance power. I literally was still working on it 2 or 3 days before, while at dconf. I had already written a JSON parser, which was part of my presentation, but when I was showing it to another D user (Daniel Murphy), I couldn't really answer the question "how does it perform?". So he gave me a challenge -- do pretty printing on a JSON file. Simple enough, with the parser I had already written, took me about 1 hour to write it. It performed poorly compared to what we would have expected, but tweaking a few things (almost all were due to using some algorithms incorrectly), I got it to go faster than RapidJson in certain use cases, and reasonably close in others. And I did nothing in terms of lookup tables or using any special instructions. All in all, it was probably 2 hours of work, and the code is beautiful IMO! https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d I think anyone who is doing parsing should have a look at iopipe, it not only may make your code much simpler and easier to read and write, but it would help me tune iopipe to cater to parsers, which I think is its wheelhouse. I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things. -Steve
Jun 12
next sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
 I plan to eventually finish the JSON parser for a releasable 
 state, and eventually tackle XML and a few other things.

 -Steve
You should definitely tackle xml by branching dxml. I'm really liking the api.
Jun 12
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/12/18 10:19 AM, Jesse Phillips wrote:
 I plan to eventually finish the JSON parser for a releasable state, 
 and eventually tackle XML and a few other things.
You should definitely tackle xml by branching dxml. I'm really liking the api.
Of course that is my plan! I would never want to have to build an xml parser from scratch :) -Steve
Jun 12
prev sibling parent DigitalDesigns <DigitalDesigns gmail.com> writes:
On Tuesday, 12 June 2018 at 13:51:56 UTC, Steven Schveighoffer 
wrote:
 On 6/12/18 1:51 AM, DigitalDesigns wrote:
 
 Could you explain some benefits specific to this 
 implementation and a bit of the functional aspects for a 
 proper overview of it's capabilities and why I should chose 
 this method over others?
The things that I think make this approach better are: 1. Direct buffer access Direct buffer access I have found is one of those ideas that doesn't seem like it's that impressive until you start using it. Many times, buffering is used in a generic library for optimization (basically amortizing reads) and is an implementation detail hidden from your view. Think of how FILE * keeps a large buffer of data inside itself, but only gives you access to one character at a time. This forces you to create your *own* buffering scheme on top of that. What a waste! Iopipe allows you to use buffering for your purposes on top of the benefits of amortization. It's my belief that this is why iopipe's byline feature is 2x faster than Phobos'. 2. Using templates to their fullest Iopipes are all templated on the buffer or iopipe underneath it. This makes tings easily swappable. It's really cool to be able to take your JSON or XML parser, and hook it onto an in-memory string in one line, and then hook it onto a socket, and everything is optimized for that situation. It takes the fun and flexibility of range programming and brings it to i/o. This is why iopipe's byline handles all forms of UTF, compared to Phobos which only handles UTF8. For example, I handle all forms of UTF with iopipe, with a decent set of utilities. Here is a complete program using iopipe that converts any type of UTF into another type, optimized for the specific situation: https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d 3. Compiler optimization for everything All parts of iopipe, except for the low-level reads and writes (which ironically are not really part of iopipe) are visible to the compiler for inlining and optimization. I'm leveraging the power of the decades of optimization experience that the compiler can provide. This makes it easy to write code that performs well. An anecdote: For my talk on iopipe in 2017 (http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live demo showing the performance power. I literally was still working on it 2 or 3 days before, while at dconf. I had already written a JSON parser, which was part of my presentation, but when I was showing it to another D user (Daniel Murphy), I couldn't really answer the question "how does it perform?". So he gave me a challenge -- do pretty printing on a JSON file. Simple enough, with the parser I had already written, took me about 1 hour to write it. It performed poorly compared to what we would have expected, but tweaking a few things (almost all were due to using some algorithms incorrectly), I got it to go faster than RapidJson in certain use cases, and reasonably close in others. And I did nothing in terms of lookup tables or using any special instructions. All in all, it was probably 2 hours of work, and the code is beautiful IMO! https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d I think anyone who is doing parsing should have a look at iopipe, it not only may make your code much simpler and easier to read and write, but it would help me tune iopipe to cater to parsers, which I think is its wheelhouse. I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things. -Steve
Thanks!
Jun 12
prev sibling next sibling parent reply Anton Fediushin <fediushin.anton yandex.ru> writes:
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer 
wrote:
 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes 
 it easy to string together pipelines to process data with as 
 little buffer copying as possible.
I saw iopipe a while back, but never looked at it closely. Now I did and... it implements its own kind of pattern and not ranges? I guess it is done for the sake of performance, but how easy it is to use iopipe with standard range-based methods from std.algorithm for example? As long as it's easy to use with the rest of the phobos - I'd like to see it in the standard library.
 This version ONLY builds on 2.080.1 or later (the bug fix that 
 I submitted at dconf has been merged in that version, and so 
 iopipe will now build against Martin Nowak's io library). In 
 fact, iopipe development was kind of stalled due to this bug, 
 so I'm super-happy to see it fixed and released!
These are great news, keep it up!
Jun 12
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/12/18 3:08 AM, Anton Fediushin wrote:
 On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:
 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes it easy 
 to string together pipelines to process data with as little buffer 
 copying as possible.
I saw iopipe a while back, but never looked at it closely. Now I did and... it implements its own kind of pattern and not ranges?
Correct, although it's very similar to ranges.
 I guess it 
 is done for the sake of performance, but how easy it is to use iopipe 
 with standard range-based methods from std.algorithm for example?
Very simple. You just have to define what is an "element" of a range that is a sliding window of data. For example: https://github.com/schveiguy/iopipe/blob/master/source/iopipe/textpipe.d#L542 Note that asInputRange simply treats the current window as "front", and "popFront" discards that window and loads the next. It's a crude but effective tool to convert any iopipe into a range. The reason iopipe is not based on ranges exactly (the window must be a random-access range), is because ranges don't handle i/o very performantly. E.g. I would never use lockingTextReader to process text data, it would be slow as hell, and too limiting. File.byLine is fast, but only because of the underlying non-range tricks it uses to achieve performance. And iopipe still is 2x faster.
 As long as it's easy to use with the rest of the phobos - I'd like to 
 see it in the standard library.
It should be easy to use on its own, and with algorithms from phobos. I've done so in some of the toy parsers I've written. I think the one sticking point (which I'm not sure how to reconcile, but we can probably hash it out) is that for iopipes, char and wchar arrays are arrays, not auto-decoding ranges. -Steve
Jun 12
parent Anton Fediushin <fediushin.anton yandex.ru> writes:
On Tuesday, 12 June 2018 at 14:00:32 UTC, Steven Schveighoffer 
wrote:
 File.byLine is fast, but only because of the underlying 
 non-range tricks it uses to achieve performance. And iopipe 
 still is 2x faster.
I wish iopipe was around a little bit earlier so I could use it in my small project. It doesn't use IO much, just reads large file (60GB+) and computes some hashes. Now I'd like to rewrite it using iopipe just to compare the performance.
Jun 13
prev sibling next sibling parent reply bauss <jj_1337 live.dk> writes:
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer 
wrote:
 -Steve
Does iopipe work with CTFE?
Jun 13
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/13/18 8:35 AM, bauss wrote:
 Does iopipe work with CTFE?
It may work in some cases. Some of the things it does are not conducive to CTFE working well -- like using a buffer. But generally at compile time, you don't want to use a buffer. But I would expect, for instance, using algorithms and iopipes on a string would probably work. hm... just thought I'd try it: import std.range : walkLength; static assert("hello\nworld\nthis\nis\na\ntest".byLineRange.walkLength == 6); It didn't work at first, as I'm using memchr for searching for the newlines, but with a nice little if(__ctfe) override, it works just great! I'm going to push this (I'll do some tests for the other widths to make sure it works for all UTF), but if you have any more things you want to work at CTFE, submit some issues on the github project. I don't expect a lot of the array casting stuff to work, but maybe there are ways. I actually never thought much about making it work at compile-time. I suppose though, it would be cool to import a file at compile time, and generate the JSON or XML DOM at compile time too :) I think zip and unzip are never going to work since the underlying C calls are not CTFE-able. -Steve
Jun 13
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/13/18 12:03 PM, Steven Schveighoffer wrote:

 I'm going to push this (I'll do some tests for the other widths to make 
 sure it works for all UTF), but if you have any more things you want to 
 work at CTFE, submit some issues on the github project.
v0.1.2 released -Steve
Jun 13
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 06/10/2018 10:10 PM, Steven Schveighoffer wrote:
 Note that the new io library also supports sockets, which IODev did not
 have support for, AND has a pluggable driver system, so you could
 potentially use fiber-based async io without rebuilding. It just makes a
 lot of sense for D to have a standard low-level io library that
 everything can use without having to kludge together multiple types of
 io libraries.
Note that the WIP std.io library is fully nogc safe, so it's a bit edgy on using latest features. Soon want to move to use DIP10008 instead of preallocated exceptions. With that and nogc in the Driver interface¹ it's still to be seen whether we can adapt this well with vibe.d or need to adjust the low-level design. -Martin ¹: https://martinnowak.github.io/io/std/io/driver/Driver.html
Jun 16
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Sunday, 17 June 2018 at 04:52:07 UTC, Martin Nowak wrote:
 On 06/10/2018 10:10 PM, Steven Schveighoffer wrote:
 Note that the new io library also supports sockets, which 
 IODev did not have support for, AND has a pluggable driver 
 system, so you could potentially use fiber-based async io 
 without rebuilding. It just makes a lot of sense for D to have 
 a standard low-level io library that everything can use 
 without having to kludge together multiple types of io 
 libraries.
Note that the WIP std.io library is fully nogc safe, so it's a bit edgy on using latest features. Soon want to move to use DIP10008 instead of preallocated exceptions.
This is very encouraging. I’d like to see it working well with Photon (though my time is very limited atm). Any thoughts on what set of syscalls I need to support? Maybe I could just provide my own “native” driver that fits your concept of I/O driver in io library.
 With that and  nogc in the Driver interface¹ it's still to be 
 seen
 whether we can adapt this well with vibe.d or need to adjust the
 low-level design.

 -Martin

 ¹: https://martinnowak.github.io/io/std/io/driver/Driver.html
Jun 17