www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - execute bash?

reply Puming <zhaopuming gmail.com> writes:
Hi,

I'd like to write an interactive commmand line tool for my 
commands, and that also support bash commands.

My first thinking is 'why not just execute those bash commands 
with bash'? But it turns out to have some problem.

When I use executeShell, I found that .bashrc is not loaded so 
that custom aliases like 'll' are not usable, and there is no 
color in the output.

Then I use spawnProcess:

```
import std.process;
void main() {
   for (int i = 0; i < 3; ++i) { // emulate the interactive loop
     string cmd = "ll";
     wait(spwanProcess(["/bin/bash", "-i", "-c", cmd]));
   }
}
```

with the bash option "-i", the .bashrc is loaded and the output 
is colored, but with each command loop, the program is stopped 
(equal to Ctrl-Z).

I thought the subprocess may have returned some special char like 
Ctrl-Z, so I changed to use pipeProcess to try to catch them. But 
it does not work either.

My question is:

1. what can I do to prevent this stop effect?
2. what is the best way to make a proxy to a bash process?
Apr 08 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 8 April 2016 at 10:08:07 UTC, Puming wrote:
 but with each command loop, the program is stopped (equal to 
 Ctrl-Z).
Your program is stopped, right? Odds are it is that there's terminal output for the background process, which sends your program a signal which, by default, stops it. From man 7 stop: SIGSTOP 17,19,23 Stop Stop process SIGTSTP 18,20,24 Stop Stop typed at terminal SIGTTIN 21,21,26 Stop Terminal input for background process SIGTTOU 22,22,27 Stop Terminal output for background process I'd say try catching one of those signals and confirm if it is one of them and then figure out what to do next. Maybe ignoring the signal is what you want to do.
Apr 08 2016
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 8 April 2016 at 13:23:10 UTC, Adam D. Ruppe wrote:
 Odds are it is that there's terminal output for the background 
 process
NOT a character btw, just any output, then the OS puts you on hold so it can do its thing. To catch a signal, it is just like in C http://stackoverflow.com/a/35687472/1457000 simplest case: import core.stdc.signal; signal(SIGTTOU, SIG_IGN); // ignore the output thing and see what happens next
Apr 08 2016
next sibling parent reply Puming <zhaopuming gmail.com> writes:
On Friday, 8 April 2016 at 13:25:37 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 13:23:10 UTC, Adam D. Ruppe wrote:
 Odds are it is that there's terminal output for the background 
 process
NOT a character btw, just any output, then the OS puts you on hold so it can do its thing. To catch a signal, it is just like in C http://stackoverflow.com/a/35687472/1457000 simplest case: import core.stdc.signal; signal(SIGTTOU, SIG_IGN); // ignore the output thing and see what happens next
I tried with signal, but didn't catch SIGTTOU, it seems that spawnProcess with `bash -i -c` will signal with SIGTTIN. When I catch SIGTTIN with: extern(C) void sig_hand(int signal) nothrow nogc system { import core.stdc.stdio: printf; printf("signal %d catched!\n", signal); } I get an infinite loop of signal 21 catched! signal 21 catched! signal 21 catched! Not sure how to handle that. Maybe I can't use `bash -i`.
Apr 08 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 8 April 2016 at 15:20:09 UTC, Puming wrote:
 I tried with signal, but didn't catch SIGTTOU, it seems that 
 spawnProcess with `bash -i -c` will signal with SIGTTIN.
Oh, surely because it wants to be interactive and is thus waiting for user input from the terminal.. You might need to rig up a pseudo terminal that the bash can talk to. That's getting to be a pain though.
Apr 08 2016
next sibling parent reply wobbles <grogan.colin gmail.com> writes:
On Friday, 8 April 2016 at 16:07:13 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 15:20:09 UTC, Puming wrote:
 I tried with signal, but didn't catch SIGTTOU, it seems that 
 spawnProcess with `bash -i -c` will signal with SIGTTIN.
Oh, surely because it wants to be interactive and is thus waiting for user input from the terminal.. You might need to rig up a pseudo terminal that the bash can talk to. That's getting to be a pain though.
You could run it through dexpect to get the effect of a pseudo terminal. https://github.com/grogancolin/dexpect
Apr 08 2016
parent reply Puming <zhaopuming gmail.com> writes:
On Friday, 8 April 2016 at 18:23:32 UTC, wobbles wrote:
 On Friday, 8 April 2016 at 16:07:13 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 15:20:09 UTC, Puming wrote:
 I tried with signal, but didn't catch SIGTTOU, it seems that 
 spawnProcess with `bash -i -c` will signal with SIGTTIN.
Oh, surely because it wants to be interactive and is thus waiting for user input from the terminal.. You might need to rig up a pseudo terminal that the bash can talk to. That's getting to be a pain though.
You could run it through dexpect to get the effect of a pseudo terminal. https://github.com/grogancolin/dexpect
Looked in the code, it is exacly what I need! Thanks. Also it has spawnInPty :-) combined with terminal.d, I think I might get a very handy shell now.
Apr 08 2016
parent reply wobbles <grogan.colin gmail.com> writes:
On Friday, 8 April 2016 at 23:06:06 UTC, Puming wrote:
 On Friday, 8 April 2016 at 18:23:32 UTC, wobbles wrote:
 On Friday, 8 April 2016 at 16:07:13 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 15:20:09 UTC, Puming wrote:
 I tried with signal, but didn't catch SIGTTOU, it seems that 
 spawnProcess with `bash -i -c` will signal with SIGTTIN.
Oh, surely because it wants to be interactive and is thus waiting for user input from the terminal.. You might need to rig up a pseudo terminal that the bash can talk to. That's getting to be a pain though.
You could run it through dexpect to get the effect of a pseudo terminal. https://github.com/grogancolin/dexpect
Looked in the code, it is exacly what I need! Thanks. Also it has spawnInPty
Cool. Any questions on using it let me know. I'm all for a bit of feedback also!
Apr 09 2016
parent reply Puming <zhaopuming gmail.com> writes:
On Saturday, 9 April 2016 at 08:56:17 UTC, wobbles wrote:
 On Friday, 8 April 2016 at 23:06:06 UTC, Puming wrote:
 On Friday, 8 April 2016 at 18:23:32 UTC, wobbles wrote:
 On Friday, 8 April 2016 at 16:07:13 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 15:20:09 UTC, Puming wrote:
 I tried with signal, but didn't catch SIGTTOU, it seems 
 that spawnProcess with `bash -i -c` will signal with 
 SIGTTIN.
Oh, surely because it wants to be interactive and is thus waiting for user input from the terminal.. You might need to rig up a pseudo terminal that the bash can talk to. That's getting to be a pain though.
You could run it through dexpect to get the effect of a pseudo terminal. https://github.com/grogancolin/dexpect
Looked in the code, it is exacly what I need! Thanks. Also it has spawnInPty
Cool. Any questions on using it let me know. I'm all for a bit of feedback also!
I tried dexpect, now it works for the bash emulation! But there are still some issues: 1. Spawn's data is a string, so it stores ALL data, but Expect class does not have a clear() method, so the data piles up and takes memory. 2. There seems to be a visible lag for each send->read cycle. I haven't look in the details to find where, but it feels not as smooth as ssh does. 3. when hiting 'vim a.file' on the command, things go messy. Have you got these interactive commands work in dexpect? My wish list for the Expect class ( or design a separate class, like BashProxy ): 1. Fully proxy for a bash shell. 2. Result data are separated for each command. So I can easily search for hitorical sent commands or results. But for now it works fine for my needs. I'll try to improve it when I get major parts of my repl lib done.
Apr 09 2016
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 10 April 2016 at 00:47:28 UTC, Puming wrote:
 3. when hiting 'vim a.file' on the command, things go messy. 
 Have you got these interactive commands work in dexpect?
It is surely capturing exactly what vim sends to a terminal, which is primarily a series of control sequences to draw the screen, move the cursor, color it, etc. You might want to watch for those and just forward them directly to the user terminal... but this is getting kinda wild....
 1. Fully proxy for a bash shell.
 2. Result data are separated for each command. So I can easily 
 search for hitorical sent commands or results.
What you probably want to do is just send one command at a time to the bash process and handle the rest yourself. Or just forget bash and write your own shell.
Apr 09 2016
parent Puming <zhaopuming gmail.com> writes:
On Sunday, 10 April 2016 at 02:59:41 UTC, Adam D. Ruppe wrote:
 On Sunday, 10 April 2016 at 00:47:28 UTC, Puming wrote:
 3. when hiting 'vim a.file' on the command, things go messy. 
 Have you got these interactive commands work in dexpect?
It is surely capturing exactly what vim sends to a terminal, which is primarily a series of control sequences to draw the screen, move the cursor, color it, etc.
Yes, I guess if I fully forward every output/input, it'll work.
 You might want to watch for those and just forward them 
 directly to the user terminal... but this is getting kinda 
 wild....
Well, I'm not sure whether I can handle the complexity here. I don't even quite understand terminal io/sequence characters yet. Come back to this later when I get my little repl running.
 1. Fully proxy for a bash shell.
 2. Result data are separated for each command. So I can easily 
 search for hitorical sent commands or results.
What you probably want to do is just send one command at a time to the bash process and handle the rest yourself. Or just forget bash and write your own shell.
An indepedent shell would be GREAT! I hope someday we get a REPL in D, and it is also an independent shell, just like FishShell, but implemented in D, and scripts are also in D.
Apr 10 2016
prev sibling parent reply wobbles <grogan.colin gmail.com> writes:
On Sunday, 10 April 2016 at 00:47:28 UTC, Puming wrote:
 On Saturday, 9 April 2016 at 08:56:17 UTC, wobbles wrote:
 On Friday, 8 April 2016 at 23:06:06 UTC, Puming wrote:
 On Friday, 8 April 2016 at 18:23:32 UTC, wobbles wrote:
 On Friday, 8 April 2016 at 16:07:13 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 15:20:09 UTC, Puming wrote:
 I tried with signal, but didn't catch SIGTTOU, it seems 
 that spawnProcess with `bash -i -c` will signal with 
 SIGTTIN.
Oh, surely because it wants to be interactive and is thus waiting for user input from the terminal.. You might need to rig up a pseudo terminal that the bash can talk to. That's getting to be a pain though.
You could run it through dexpect to get the effect of a pseudo terminal. https://github.com/grogancolin/dexpect
Looked in the code, it is exacly what I need! Thanks. Also it has spawnInPty
Cool. Any questions on using it let me know. I'm all for a bit of feedback also!
I tried dexpect, now it works for the bash emulation!
Good - glad it works!
 But there are still some issues:
Bad - but to be expected :)
 1. Spawn's data is a string, so it stores ALL data, but Expect 
 class does not have a clear() method, so the data piles up and 
 takes memory.
Yeah, this was something I came across too but didn't have the time to fix it. My plan is to have the Expect class take an OutputRange, and all output is then sent to that Range. The only data that it keeps track of then is the data just before and since the latest 'expect' call.
 2. There seems to be a visible lag for each send->read cycle. I 
 haven't look in the details to find where, but it feels not as 
 smooth as ssh does.
I have added a good few Thread.sleep function calls in the send / read calls. This is because it seemed to crash out when there was nothing there and I needed something quick to get it working at the time. There must be a better way of handling this though. Also, I suspect I've added way too many sleep calls so I should do a bit of work on this!
 3. when hiting 'vim a.file' on the command, things go messy. 
 Have you got these interactive commands work in dexpect?
Yeah, dexpect won't be handling something like vim. As Adam said, it sends lots of signals / commands to the terminal to tell it how to draw the window. Dexpect just cares about reading the data sent to/from the process, not what any of that data means.
 My wish list for the Expect class ( or design a separate class, 
 like BashProxy ):

 1. Fully proxy for a bash shell.
 2. Result data are separated for each command. So I can easily 
 search for hitorical sent commands or results.
This would be cool. I'll have a think about how to go about it!
 But for now it works fine for my needs.
 I'll try to improve it when I get major parts of my repl lib 
 done.
Also, pull requests are welcome :)
Apr 10 2016
parent Puming <zhaopuming gmail.com> writes:
On Sunday, 10 April 2016 at 10:43:48 UTC, wobbles wrote:

 This would be cool. I'll have a think about how to go about it!
Looking forward to your updates! :P
Apr 10 2016
prev sibling parent Puming <zhaopuming gmail.com> writes:
On Friday, 8 April 2016 at 16:07:13 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 15:20:09 UTC, Puming wrote:
 I tried with signal, but didn't catch SIGTTOU, it seems that 
 spawnProcess with `bash -i -c` will signal with SIGTTIN.
Oh, surely because it wants to be interactive and is thus waiting for user input from the terminal.. You might need to rig up a pseudo terminal that the bash can talk to. That's getting to be a pain though.
Indeed, this makes sense.
Apr 08 2016
prev sibling parent reply Puming <zhaopuming gmail.com> writes:
On Friday, 8 April 2016 at 13:25:37 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 13:23:10 UTC, Adam D. Ruppe wrote:
 Odds are it is that there's terminal output for the background 
 process
NOT a character btw, just any output, then the OS puts you on hold so it can do its thing. To catch a signal, it is just like in C http://stackoverflow.com/a/35687472/1457000 simplest case: import core.stdc.signal; signal(SIGTTOU, SIG_IGN); // ignore the output thing and see what happens next
Found another problem: I can't even catch SIGINT in a simple program: import std.stdio; import std.process; import core.stdc.signal; import core.thread; import std.datetime; extern(C) void sig_hand(int signal) nothrow nogc system { import core.stdc.stdio: printf; printf("signal %d catched!\n", signal); } void main() { if (signal(SIGINT, &sig_hand) == SIG_ERR) { writeln("can't catch SIGINT"); } string line; write("> "); while(1) { Thread.sleep(1.seconds); } /* while ((line = readln()) != null) { writeln("you said:", line); write("> "); } */ } When using while with Thread.sleep, Ctrl+C will show signal 2 catched! then the program is gone, but its process is still found with `ps -ef`, which I have to kill manually. In the c version as shown here <http://www.thegeekstuff.com/2012/03/catch-signals-sample-c-code/>, the loop will go on and you can press multiple Ctrl-C. The D version behavior is strange. When Using while with readln, after hitting Ctrl-C, the next readln will throw exception: std.stdio.StdioException std/stdio.d(3977): Input/output error ---------------- ??:? void std.stdio.StdioException.opCall() [0x498531] ??:? ulong std.stdio.readlnImpl(shared(core.stdc.stdio._IO_FILE)*, ref char[], dchar, std.stdio.File.Orientation) [0x498676] /usr/include/dmd/phobos/std/stdio.d:1565 ulong std.stdio.File.readln!(char).readln(ref char[], dchar) [0x486fac] /usr/include/dmd/phobos/std/stdio.d:1426 immutable(char)[] std.stdio.File.readln!(immutable(char)[]).readln(dchar) [0x486eb1] /usr/include/dmd/phobos/std/stdio.d:3385 immutable(char)[] std.stdio.readln!(immutable(char)[]).readln(dchar) [0x486e3b] source/app.d:25 _Dmain [0x48677e] ??:? _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv [0x48ad2a] ??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x48ac74] ??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll() [0x48ace6] ??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x48ac74] ??:? _d_run_main [0x48abe5] ??:? main [0x487b0b] ??:? __libc_start_main [0xc7628ec4]
Apr 08 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 8 April 2016 at 15:31:13 UTC, Puming wrote:
 The D version behavior is strange.
Are you still calling bash? Cuz that is going to complicate things a lot because bash does its own signal handling too and could be intercepting it.
 When Using while with readln, after hitting Ctrl-C, the next 
 readln will throw exception:
This is normal though. If a signal interrupts a read/write call, it returns an error saying it was interrupted so you can choose to abort or try again. (this way you can catch ctrl+c to cancel input)
Apr 08 2016
parent Puming <zhaopuming gmail.com> writes:
On Friday, 8 April 2016 at 16:16:27 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 15:31:13 UTC, Puming wrote:
 The D version behavior is strange.
Are you still calling bash? Cuz that is going to complicate things a lot because bash does its own signal handling too and could be intercepting it.
No, this happens in a simple demo program: import std.stdio; import std.process; import core.stdc.signal; import core.thread; import std.datetime; extern(C) void sig_hand(int signal) nothrow nogc system { import core.stdc.stdio: printf; printf("signal %d catched!\n", signal); } void main() { if (signal(SIGINT, &sig_hand) == SIG_ERR) { writeln("can't catch SIGINT"); } string line; write("> "); while(1) { Thread.sleep(1.seconds); } } But after trying again on my machine, it behaved the same as in the C code. Maybe I got the wrong behavior last night due to running it on a remote server.
 When Using while with readln, after hitting Ctrl-C, the next 
 readln will throw exception:
This is normal though. If a signal interrupts a read/write call, it returns an error saying it was interrupted so you can choose to abort or try again. (this way you can catch ctrl+c to cancel input)
Thanks, so I'll just catch it.
Apr 08 2016
prev sibling parent reply Puming <zhaopuming gmail.com> writes:
On Friday, 8 April 2016 at 13:23:10 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 10:08:07 UTC, Puming wrote:
 but with each command loop, the program is stopped (equal to 
 Ctrl-Z).
Your program is stopped, right? Odds are it is that there's terminal output for the background process, which sends your program a signal which, by default, stops it. From man 7 stop: SIGSTOP 17,19,23 Stop Stop process SIGTSTP 18,20,24 Stop Stop typed at terminal SIGTTIN 21,21,26 Stop Terminal input for background process SIGTTOU 22,22,27 Stop Terminal output for background process
Ah, that must be the case. Thanks :-) I just found that you have terminal.d in arsd repo, are you writing a repl with it? I'm hoping I might be able to use it. I'm writing a simple repl for my toy language for learning.
 I'd say try catching one of those signals and confirm if it is 
 one of them and then figure out what to do next. Maybe ignoring 
 the signal is what you want to do.
Apr 08 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 8 April 2016 at 14:09:16 UTC, Puming wrote:
 I just found that you have terminal.d in arsd repo, are you
 writing a repl with it? I'm hoping I might be able to use it.
I have done it before. terminal.d has a getline function and writeln functions you could loop over. Its getline has features like tab completion, editing, etc.
Apr 08 2016
parent reply Puming <zhaopuming gmail.com> writes:
On Friday, 8 April 2016 at 16:08:02 UTC, Adam D. Ruppe wrote:
 On Friday, 8 April 2016 at 14:09:16 UTC, Puming wrote:
 I just found that you have terminal.d in arsd repo, are you
 writing a repl with it? I'm hoping I might be able to use it.
I have done it before. terminal.d has a getline function and writeln functions you could loop over. Its getline has features like tab completion, editing, etc.
wow, that is just what I need. I'll look at it, thanks. Do you have this pseudo terminal thing in terminal.d?
Apr 08 2016
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 8 April 2016 at 23:03:15 UTC, Puming wrote:
 Do you have this pseudo terminal thing in terminal.d?
No, terminal.d is for your program to interact with the user's terminal rather than program to program stuff. My terminal emulator <https://github.com/adamdruppe/terminal-emulator> has the pty stuff... but I think the expect lib is a better match for what you need. The full emulator is too heavy for your needs.
Apr 08 2016