www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - iopipe v0.2.0 - safe update

reply Steven Schveighoffer <schveiguy gmail.com> writes:
Just wanted to post that I finished my update of iopipe to be  safe. I 
still have some work to do with std.io so things are more usable (next 
on my list is to make standard handles accessible).

In this update, I've made it so nearly all of iopipe can be used in 
 safe code. The one exception is the RingBuffer, which can leave 
dangling pointers, hence the unsafety.

Inside iopipe is a RefCounted type that is  safe. It uses the GC for 
memory management, so while destruction is synchronous, the memory 
itself is left to the GC to clean up.

-Steve

https://code.dlang.org/packages/iopipe
Jun 28
next sibling parent reply JN <666total wp.pl> writes:
On Sunday, 28 June 2020 at 18:57:22 UTC, Steven Schveighoffer 
wrote:
 Just wanted to post that I finished my update of iopipe to be 
  safe. I still have some work to do with std.io so things are 
 more usable (next on my list is to make standard handles 
 accessible).

 In this update, I've made it so nearly all of iopipe can be 
 used in  safe code. The one exception is the RingBuffer, which 
 can leave dangling pointers, hence the unsafety.

 Inside iopipe is a RefCounted type that is  safe. It uses the 
 GC for memory management, so while destruction is synchronous, 
 the memory itself is left to the GC to clean up.

 -Steve

 https://code.dlang.org/packages/iopipe
What's iopipe and what does it do? How does it compare with std.process?
Jun 28
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/28/20 4:26 PM, JN wrote:
 
 What's iopipe and what does it do? How does it compare with std.process?
Yeah, I guess I should be more specific in such announcements, sorry! iopipe is a performance-oriented input/output library for D, which uses direct buffer access, and provides a similar experience to unix pipes, but for buffered data instead. For example: import std.io; // https://github.com/MartinNowak/io foreach(line; File("input.txt.gz") .refCounted // std.io files are not copyable .bufd // buffer it .unzip // decompress it .assumeText // assume it's text (char[] instead of ubyte[]) .byLineRange) // convert to a line-based range -Steve
Jun 28
prev sibling next sibling parent Clarice <cl ar.ice> writes:
On Sunday, 28 June 2020 at 20:26:43 UTC, JN wrote:
 On Sunday, 28 June 2020 at 18:57:22 UTC, Steven Schveighoffer 
 wrote:
 Just wanted to post that I finished my update of iopipe to be 
  safe. I still have some work to do with std.io so things are 
 more usable (next on my list is to make standard handles 
 accessible).

 In this update, I've made it so nearly all of iopipe can be 
 used in  safe code. The one exception is the RingBuffer, which 
 can leave dangling pointers, hence the unsafety.

 Inside iopipe is a RefCounted type that is  safe. It uses the 
 GC for memory management, so while destruction is synchronous, 
 the memory itself is left to the GC to clean up.

 -Steve

 https://code.dlang.org/packages/iopipe
What's iopipe and what does it do? How does it compare with std.process?
Here's a pertinent dconf presentation that might help you: https://www.youtube.com/watch?v=un-bZdyumog The abstract: http://dconf.org/2017/talks/schveighoffer.html
Jun 28
prev sibling parent reply aberba <karabutaworld gmail.com> writes:
On Sunday, 28 June 2020 at 20:26:43 UTC, JN wrote:
 On Sunday, 28 June 2020 at 18:57:22 UTC, Steven Schveighoffer 
 wrote:
 Just wanted to post that I finished my update of iopipe to be 
  safe. I still have some work to do with std.io so things are 
 more usable (next on my list is to make standard handles 
 accessible).

 In this update, I've made it so nearly all of iopipe can be 
 used in  safe code. The one exception is the RingBuffer, which 
 can leave dangling pointers, hence the unsafety.

 Inside iopipe is a RefCounted type that is  safe. It uses the 
 GC for memory management, so while destruction is synchronous, 
 the memory itself is left to the GC to clean up.

 -Steve

 https://code.dlang.org/packages/iopipe
What's iopipe and what does it do? How does it compare with std.process?
I my line of words, its what you'd use to stream large files and do processing on it. Like CSV, video??, Json, and the like. Its high performance cus it optimized to haven great performance even for large files. Designed to let you chain the processing pipeline using reusable functions...and you can compose one yourself. I'm curious myself how it differs from the NodeJs Stream API. Would you say iopipe is a Streaming API by definition?
Jun 28
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/28/20 8:37 PM, aberba wrote:
 On Sunday, 28 June 2020 at 20:26:43 UTC, JN wrote:
 What's iopipe and what does it do? How does it compare with std.process?
I my line of words, its what you'd use to stream large files and do processing on it. Like CSV, video??, Json, and the like. Its high performance cus it optimized to haven great performance even for large files. Designed to let you chain the processing pipeline using reusable functions...and  you can compose one yourself.
Yes, the emphasis is on being able to handle large sets of data similarly to how you would handle an array, but not having to load the whole thing into an array. It uses buffering to present an array-like interface. The only difference between it and arrays is you have to tell it how much data is important to you at the moment. There is also a focus on composability -- I want you to be able to build any type of harness to take what you have and turn it into something you can feed into a parser/handler. Take for example, jsoniopipe [1], which is a library that uses iopipe to parse JSON. You can feed it any source -- a string, a zipped file, a network socket, and all you have to do is compose the pieces to get it to be a text iopipe (an iopipe where the window type is a string type). Then the json library will parse it and provide you with a stream of JSON data. Because everything is available to the compiler, it can optimize/inline across all the layers, and provide the best performance possible (see my presentation at dconf 2017 [2] for how this works out, also linked by Clarice).
 
 I'm curious myself how it differs from the NodeJs Stream API. Would you 
 say iopipe is a Streaming API by definition?
I'm not familiar with NodeJs in general so I can't comment on that. I would say iopipe is focused on making streaming more like dealing with arrays, with as high performance as possible. One thing I've always disliked about most streaming libraries is that they hide the buffer as an implementation detail, but it should be the focus. -Steve [1] https://code.dlang.org/packages/jsoniopipe [2] https://www.youtube.com/watch?v=un-bZdyumog
Jun 29
parent reply aberba <karabutaworld gmail.com> writes:
On Monday, 29 June 2020 at 11:58:10 UTC, Steven Schveighoffer 
wrote:
 On 6/28/20 8:37 PM, aberba wrote:
 On Sunday, 28 June 2020 at 20:26:43 UTC, JN wrote:
 What's iopipe and what does it do? How does it compare with 
 std.process?
I my line of words, its what you'd use to stream large files and do processing on it. Like CSV, video??, Json, and the like. Its high performance cus it optimized to haven great performance even for large files. Designed to let you chain the processing pipeline using reusable functions...and  you can compose one yourself.
Yes, the emphasis is on being able to handle large sets of data similarly to how you would handle an array, but not having to load the whole thing into an array. It uses buffering to present an array-like interface. The only difference between it and arrays is you have to tell it how much data is important to you at the moment. There is also a focus on composability -- I want you to be able to build any type of harness to take what you have and turn it into something you can feed into a parser/handler.
Thanks for the expansion. I could really use this for an article. Added to my to-do list.
 Take for example, jsoniopipe [1], which is a library that uses 
 iopipe to parse JSON. You can feed it any source -- a string, a 
 zipped file, a network socket, and all you have to do is 
 compose the pieces to get it to be a text iopipe (an iopipe 
 where the window type is a string type). Then the json library 
 will parse it and provide you with a stream of JSON data. 
 Because everything is available to the compiler, it can 
 optimize/inline across all the layers, and provide the best 
 performance possible (see my presentation at dconf 2017 [2] for 
 how this works out, also linked by Clarice).
Will looking into that library more.
 and all you have to do is compose the pieces to get it to be a 
 text iopipe (an iopipe where the window type is a string type).
No sure I understand this statement.
 
 I'm curious myself how it differs from the NodeJs Stream API. 
 Would you say iopipe is a Streaming API by definition?
I'm not familiar with NodeJs in general so I can't comment on that. I would say iopipe is focused on making streaming more like dealing with arrays, with as high performance as possible. One thing I've always disliked about most streaming libraries is that they hide the buffer as an implementation detail, but it should be the focus.
How well does iopipe work in a multi-threaded environment?
 -Steve

 [1] https://code.dlang.org/packages/jsoniopipe
 [2] https://www.youtube.com/watch?v=un-bZdyumog
I'll probably re-watch it before writing the article.
Jun 30
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/30/20 5:05 AM, aberba wrote:
 On Monday, 29 June 2020 at 11:58:10 UTC, Steven Schveighoffer wrote:
 and all you have to do is compose the pieces to get it to be a text 
 iopipe (an iopipe where the window type is a string type).
No sure I understand this statement.
What I mean is, no matter where your data comes from, you compose iopipes together to get it to be a text iopipe (one that is comprised of UTF code points), and then the json parser can handle it. For instance, it could be zipped data on a network socket, it could be data that comes from a segment of a file that you loaded into a local buffer. It could even be data that is generated. As long as it provides a mechanism to be read into a buffer (or is already in a buffer), jsoniopipe can handle it. This is one thing I think makes iopipe so useful -- you can compose them together like you can compose ranges, and the end result is fully optimized on all layers.
 I'm not familiar with NodeJs in general so I can't comment on that. I 
 would say iopipe is focused on making streaming more like dealing with 
 arrays, with as high performance as possible. One thing I've always 
 disliked about most streaming libraries is that they hide the buffer 
 as an implementation detail, but it should be the focus.
How well does iopipe work in a multi-threaded environment?
I'm assuming you mean multiple threads using the same iopipe? It does not have any implementation to handle something like that. You would have to handle the locking yourself. I plan at some point to write a full replacement for writeln and friends using iopipe, which would necessarily have to deal with multiple threads. I'm not sure whether the multithreading constructs would be put into iopipe, or something else, I haven't thought enough about it. Let me know if there's any other questions you have, either here or on the github issues. -Steve
Jun 30
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/28/20 2:57 PM, Steven Schveighoffer wrote:
 Just wanted to post that I finished my update of iopipe to be  safe. I 
 still have some work to do with std.io so things are more usable (next 
 on my list is to make standard handles accessible).
 
 In this update, I've made it so nearly all of iopipe can be used in 
  safe code. The one exception is the RingBuffer, which can leave 
 dangling pointers, hence the unsafety.
I forgot to mention (and was kindly reminded by Jon Degenhardt), this is the first release where Martin's std.io library (https://github.com/MartinNowak/io) is optional. In the future, iopipe will remove this dependency. The only thing that depends on std.io is the deprecated `openDev` function. So if you were expecting iopipe to bring in that dependency, you will have to add it to your dub.json/dub.sdl file. -Steve
Jun 28
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Sunday, 28 June 2020 at 18:57:22 UTC, Steven Schveighoffer 
wrote:
 Just wanted to post that I finished my update of iopipe to be 
  safe. I still have some work to do with std.io so things are 
 more usable (next on my list is to make standard handles 
 accessible).
Cool, and congratulations :-)
 Inside iopipe is a RefCounted type that is  safe. It uses the 
 GC for memory management, so while destruction is synchronous, 
 the memory itself is left to the GC to clean up.
One concern here -- this means that one cannot control when an app using this might block for GC, right? Any thoughts on how a user could get more deterministic control on when the memory would be released?
Jun 30
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/30/20 6:41 AM, Joseph Rushton Wakeling wrote:
 On Sunday, 28 June 2020 at 18:57:22 UTC, Steven Schveighoffer wrote:
 Just wanted to post that I finished my update of iopipe to be  safe. I 
 still have some work to do with std.io so things are more usable (next 
 on my list is to make standard handles accessible).
Cool, and congratulations :-)
 Inside iopipe is a RefCounted type that is  safe. It uses the GC for 
 memory management, so while destruction is synchronous, the memory 
 itself is left to the GC to clean up.
One concern here -- this means that one cannot control when an app using this might block for GC, right?  Any thoughts on how a user could get more deterministic control on when the memory would be released?
Ref counting is only used internally when I need automatic cleanup. I should be able to make it possible to specify an alternate ref-counting scheme (i.e. allow using std.typecons.refCounted instead). Certainly, you don't need to use it for most things. The current places where it's automatically used are for output (in the case of auto-flush, if you don't want auto-flush, then it does not refcount the pipe), and in the zip/unzip pipes, which depend on zlib, and zlib uses some self-referencing pointers I think inside its struct. It is already possible to use an alternative allocator for buffering. -Steve
Jun 30