www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How do I compose pipes?

reply Anthony <anthoq88 gmail.com> writes:
This post https://dlang.org/library/std/process/pipe.html 
mentions:

Pipes can, for example, be used for interprocess communication 
by spawning a new process and passing one end of the pipe to the 
child, while the parent uses the other end. (See also 
pipeProcess and pipeShell for an easier way of doing this.)
``` auto p = pipe(); auto outFile = File("D downloads.txt", "w"); auto cpid = spawnProcess(["curl", "http://dlang.org/download.html"], stdin, p.writeEnd); scope(exit) wait(cpid); auto gpid = spawnProcess(["grep", "-o", `http://\S*\.zip`], p.readEnd, outFile); scope(exit) wait(gpid); ``` How am I able to compose the pipes from pipeProcess if I can't pass in an existing pipe into the function. Eg. ``` auto p = pipeProcess("ls"); auto q = pipeProcess("cat", stdin = p.stdout); //it would be good to do this or something like it ``` Do I need to manually extract the output from pipes.stdin.readln and place it in pipes.stdout.writeln? Or is there an easier way to do this? Thanks
Jan 28
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 1/28/21 2:16 AM, Anthony wrote:

 auto p = pipeProcess("ls");
 auto q = pipeProcess("cat", stdin = p.stdout); //it would be good to do
That would work if `cat` received the *contents* of the files (and with a "-" command line switch). Since `ls` produces file names, you would have to make the complete `cat` command line from `ls`'s output.
 Do I need to manually extract the output from pipes.stdin.readln
Seems to be so for the `ls | cat` case. But the following `find | grep` example shows how two ends of pipes can be connected: import std.stdio; import std.process; import std.range; // BONUS: Enable one of the following lines to enjoy an issue. // version = bonus_bug; // version = bonus_bug_but_this_works; void main() { // Writes to 'a': auto a = pipe(); auto lsPid = spawnProcess([ "find", "."], stdin, a.writeEnd); scope (exit) wait(lsPid); // Reads from 'a', writes to 'b': auto b = pipe(); auto catPid = spawnProcess([ "grep", "-e", `\.d$` ], a.readEnd, b.writeEnd); scope (exit) wait(catPid); version (bonus_bug) { // Fails with the following error. // // "/usr/include/dmd/phobos/std/typecons.d(6540): Error: // `"Attempted to access an uninitialized payload."`" writefln!"Some of the D source files under the current directory:\n%-( %s\n%)"( b.readEnd.byLine); } else version (bonus_bug_but_this_works) { // Note .take at the end: writefln!"Some of the D source files under the current directory:\n%-( %s\n%)"( b.readEnd.byLine.take(1000)); } else { // The results are read from 'b': writeln(b.readEnd.byLine); } } I've discovered a strange issue, which can be observed by uncommenting the 'version = bonus_bug;' line above. But comment that one back in and uncomment the next line, now it works. (?) Ali
Jan 28
parent reply Anthony <anthoq88 gmail.com> writes:
On Thursday, 28 January 2021 at 17:18:46 UTC, Ali Çehreli wrote:
 On 1/28/21 2:16 AM, Anthony wrote:

 auto p = pipeProcess("ls");
 auto q = pipeProcess("cat", stdin = p.stdout); //it would be
good to do That would work if `cat` received the *contents* of the files (and with a "-" command line switch). Since `ls` produces file names, you would have to make the complete `cat` command line from `ls`'s output.
 Do I need to manually extract the output from
pipes.stdin.readln Seems to be so for the `ls | cat` case. But the following `find | grep` example shows how two ends of pipes can be connected: import std.stdio; import std.process; import std.range; // BONUS: Enable one of the following lines to enjoy an issue. // version = bonus_bug; // version = bonus_bug_but_this_works; void main() { // Writes to 'a': auto a = pipe(); auto lsPid = spawnProcess([ "find", "."], stdin, a.writeEnd); scope (exit) wait(lsPid); // Reads from 'a', writes to 'b': auto b = pipe(); auto catPid = spawnProcess([ "grep", "-e", `\.d$` ], a.readEnd, b.writeEnd); scope (exit) wait(catPid); version (bonus_bug) { // Fails with the following error. // // "/usr/include/dmd/phobos/std/typecons.d(6540): Error: // `"Attempted to access an uninitialized payload."`" writefln!"Some of the D source files under the current directory:\n%-( %s\n%)"( b.readEnd.byLine); } else version (bonus_bug_but_this_works) { // Note .take at the end: writefln!"Some of the D source files under the current directory:\n%-( %s\n%)"( b.readEnd.byLine.take(1000)); } else { // The results are read from 'b': writeln(b.readEnd.byLine); } } I've discovered a strange issue, which can be observed by uncommenting the 'version = bonus_bug;' line above. But comment that one back in and uncomment the next line, now it works. (?) Ali
Thanks Ali. I was messing around and below seems to work well enough for me. ``` struct AccumulatorPipe { Pid[] pids; File stdin; File stdout; } AccumulatorPipe run(string cmd) { AccumulatorPipe acc; auto p = P.pipeShell(cmd, P.Redirect.stdout); acc.pids ~= p.pid; acc.stdout = p.stdout; return acc; } AccumulatorPipe pipe(AccumulatorPipe acc0, string cmd) { AccumulatorPipe acc; Pipe p = P.pipe(); acc.stdin = p.writeEnd; acc.stdout = p.readEnd; auto pid = P.spawnShell(cmd, acc0.stdout, acc.stdin); acc.pids = acc0.pids ~ pid; return acc; } void end(AccumulatorPipe acc) { auto pids = acc.pids ~ P.spawnShell("cat", acc.stdout); foreach (pid; pids) { P.wait(pid); } } ``` So now I can do something like: ``` run("find source -name '*.d'") .pipe("entr ./make.d tests") .end(), ```
 That would work if `cat` received the *contents* of the files 
 (and with a "-" command line switch)
I was actually trying to use cat to just spit out the filenames of the directory as a test. But I see what you mean.
Jan 28
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 1/28/21 3:45 PM, Anthony wrote:

 void end(AccumulatorPipe acc) {
      auto pids = acc.pids ~ P.spawnShell("cat", acc.stdout);

      foreach (pid; pids) {
          P.wait(pid);
      }
 }
 ```


 So now I can do something like:
 ```
 run("find source -name '*.d'")
          .pipe("entr ./make.d tests")
          .end(),
Cool but there should be one improvement because I don't think end() is guaranteed to be executed in that code, which may leave zombie processes around. From 'man waitpid': "A child that terminates, but has not been waited for becomes a "zombie". Which is relayed to std.process documentation as "to avoid child processes becoming "zombies"". Ali
Jan 28
parent Anthony <anthoq88 gmail.com> writes:
On Friday, 29 January 2021 at 03:49:38 UTC, Ali Çehreli wrote:
 On 1/28/21 3:45 PM, Anthony wrote:

 void end(AccumulatorPipe acc) {
      auto pids = acc.pids ~ P.spawnShell("cat", acc.stdout);

      foreach (pid; pids) {
          P.wait(pid);
      }
 }
 ```


 So now I can do something like:
 ```
 run("find source -name '*.d'")
          .pipe("entr ./make.d tests")
          .end(),
Cool but there should be one improvement because I don't think end() is guaranteed to be executed in that code, which may leave zombie processes around. From 'man waitpid': "A child that terminates, but has not been waited for becomes a "zombie". Which is relayed to std.process documentation as "to avoid child processes becoming "zombies"". Ali
I take it you're referring to missing scope guards like in your code `scope (exit) wait(lsPid);` Yeah, that is a tricky one. I can't think of a way to have a nice interface that also closes the pids on exit since scope guards are handled on function exit in this case. Perhaps thats just the nature of the problem though. I'll take a look at what scriptlike does https://github.com/Abscissa/scriptlike#script-style-shell-commands
Jan 28