[00:00.000 --> 00:17.600]  Okay, hello everybody, welcome. Thank you for coming to my talk. My name is Brian Duggan.
[00:17.600 --> 00:24.720]  I'm going to be talking today about something called TERME, which is Practical and Fun Automation
[00:24.720 --> 00:32.080]  for all your terminal sessions. I'd like to thank my employer Instacart and the Pearl Foundation
[00:32.080 --> 00:42.160]  for helping me to be here. I'm on the logistics team at Instacart. Okay, so here's an outline of
[00:42.160 --> 00:48.160]  the talk. I'm going to give a quick overview of the concept of TERME, what it does, what it's all
[00:48.160 --> 00:55.120]  about, go through some of the features, explain the scripting capabilities, and then a little bit
[00:55.120 --> 01:02.000]  about why it's written in Raku, which is usually the first question I get, but I'm saving it for last.
[01:04.400 --> 01:10.400]  Okay, so here's the basic concept of TERME, works like this. So you have your shell,
[01:10.400 --> 01:18.560]  you type TERME to start the session, it starts a T-Mux session, how many T-Mux users in here?
[01:19.200 --> 01:28.320]  Oh good, okay, screen. Okay, you guys can fight it later. Okay, so it starts a T-Mux,
[01:29.120 --> 01:34.640]  starts a T-Mux session, it puts you in the bottom half, and basically anything that you type into
[01:34.640 --> 01:42.000]  the bottom half goes into both the bottom half and the top half. Okay, so you type, what is TERME?
[01:42.640 --> 01:48.960]  And it sends it to the bash shell session at the top, which doesn't know what what means,
[01:48.960 --> 01:52.560]  depending on your environment, you'll get some strange error messages about,
[01:53.280 --> 01:58.560]  you know, what command are you trying to type. So, you know, I did this and since this is the
[01:58.560 --> 02:03.440]  automation room, I thought, you know, probably, you know, maybe I could just automate this talk
[02:03.440 --> 02:10.720]  completely, so I found a command line version of ChatGPT on the internet that had a command line
[02:10.720 --> 02:14.560]  wrapper, I thought maybe I could just get ChatGPT to write my entire talk for me,
[02:14.560 --> 02:21.200]  automate it away, and then I would be done. Okay, so I did the pip install, which sends
[02:21.200 --> 02:27.040]  a lot of things to the terminal, as everybody probably knows, lots of recommendations about
[02:27.040 --> 02:33.520]  what to upgrade. Finally, I have the ChatGPT executable, so I typed that and I said, what is
[02:33.520 --> 02:40.400]  TERME? But did not get very much information since it didn't know about the talk that I hadn't given
[02:40.400 --> 02:47.200]  yet. So then I said, you know, TERME is being presented at FOSSTEM, which was a little bit more,
[02:48.160 --> 02:52.960]  a little bit better, but still I had to add a little more substance to the talk. And then,
[02:52.960 --> 03:02.160]  then the program hung, so I had to interrupt it with control C. And the way you do that with
[03:02.160 --> 03:07.440]  TERME is you use a backslash, which starts the command, anything that starts with a backslash,
[03:07.440 --> 03:14.400]  you kind of like the Postgres command line interface is a directive to TERME, so stop says
[03:14.400 --> 03:20.880]  send a control C signal to the other pane. Okay, so I got a keyboard interrupt, but that wasn't
[03:20.880 --> 03:26.560]  enough to stop it because that was trapped by the Python interpreter. So then I sent another one,
[03:26.560 --> 03:33.280]  and then that finally gave me a stack trace, which everybody who uses Python sees a lot.
[03:34.160 --> 03:39.600]  And then finally, you know, I was done with this session. Okay, so the basic concept here
[03:40.560 --> 03:44.400]  is simple, you know, you have something on the top, something on the bottom, it's the same,
[03:45.120 --> 03:48.880]  and the things on the bottom go to the top, so you have kind of an interactive session.
[03:48.880 --> 03:57.280]  And then you can also send these additional commands to the top. Okay, so now I'm going to go through
[03:57.280 --> 04:03.360]  some of the features. Okay, so as you saw from the last one, you have everything sort of organized
[04:03.360 --> 04:07.440]  on the bottom, even if you have stack traces and things on the top, you still have a nice little
[04:07.440 --> 04:14.000]  session that shows you what you're doing, and you have, you can set up macros, you can run scripts,
[04:14.000 --> 04:19.360]  you can wait for things, and I'm going to go through a few more of these in detail now in
[04:19.360 --> 04:26.560]  the next few minutes. So it has a new read line built in, it has a few ways of getting history,
[04:26.560 --> 04:34.400]  right, read line, there's a last command, there's also fuzzy find, fzf, anybody use fuzzy find for
[04:34.400 --> 04:40.640]  things, yep, so you can search your history for that. And it searches, right, even if you have
[04:40.640 --> 04:45.920]  several different sessions on the top, maybe you're on different machines or maybe part of it is in
[04:46.560 --> 04:51.680]  some other application, some of it is in a shell, it'll search your local history.
[04:53.840 --> 05:00.400]  So for instance, let's say you're using PSQL, you're connected to a remote database and you have,
[05:00.960 --> 05:04.720]  you know, your local history and maybe you run some sequel that's going to show you the long
[05:04.720 --> 05:11.280]  running queries. So you can write a macro to send all of this, and the way you do that is you say
[05:11.280 --> 05:17.840]  slash edit, then you have a text editor, put your file in an SQL file, and then slash alias
[05:17.840 --> 05:24.960]  will create a macro that says, you set the name of the macro and then slash run says run this,
[05:24.960 --> 05:31.120]  run this little script, which will send it to the other console. And then after that,
[05:31.120 --> 05:38.240]  you can just type slash find queries and the top will get the SQL that you put into the file.
[05:39.600 --> 05:44.720]  Okay, so it can be convenient for things like that, you know, or you could use it with Redis or,
[05:45.360 --> 05:49.120]  you know, building your, doing your kernel testings, they saw in the first talk or,
[05:49.120 --> 05:53.760]  you know, whatever, just any, any sort of session you can just make a macro and send it.
[05:53.760 --> 06:01.760]  So here's, here's another example. In this case, instead of using a macro, we're going to send
[06:01.760 --> 06:08.640]  standard out from a command that we run locally to the other pane. So on the bottom, I say delay
[06:08.640 --> 06:13.920]  three, which means wait three seconds between every line that you send to the top, and then slash
[06:13.920 --> 06:20.160]  shell means just run this, run this command, and then show me the output. So I say slash shell,
[06:20.160 --> 06:24.240]  cat, e.g. simple.bash, and here you can see my bash script, which does an echo,
[06:24.800 --> 06:31.280]  echo docker run, and then echo hostname. So then it runs that command when I say slash do,
[06:31.280 --> 06:37.440]  it runs that command, and the output from that command gets sent to the top, and it's sent,
[06:37.440 --> 06:41.200]  you know, after every line, it waits three seconds. So it sort of throttles the output,
[06:41.200 --> 06:45.920]  you know, which we, which we might need because it might take docker, you know, a few seconds to
[06:45.920 --> 06:51.200]  start before you run the hostname command on, you know, on the shell inside the container.
[06:52.720 --> 06:58.240]  So, so the bottom is what you're typing, the top is what you see. So here's another,
[06:58.240 --> 07:02.640]  here's another feature. So in addition to standard out, you can take, take the output of the top
[07:02.640 --> 07:08.000]  and send it to standard in of anything that you write. So in this example, I'm using the nl
[07:08.000 --> 07:14.240]  command, which the standard in, it basically takes standard in and outputs line numbers for,
[07:14.240 --> 07:19.840]  for the commands that are coming in. So I say sleep three and head user share addict words.
[07:21.120 --> 07:27.520]  The reason I do sleep three, if you think about it for a second is because when I do the exec
[07:27.520 --> 07:33.920]  command, I need time to type it, right? So I type sleep three and head, and that gets sent
[07:33.920 --> 07:39.920]  immediately to the shell, which waits a few seconds, then I type exec nl, and then standard
[07:39.920 --> 07:44.880]  in comes in, and then it prints out what goes out. So, you know, in the real world, you probably
[07:44.880 --> 07:49.040]  won't have to sleep because there will be constantly stuff coming through the top terminal.
[07:51.280 --> 07:56.880]  A few other interesting commands. So await, we'll just wait for either a string or a regular
[07:56.880 --> 08:02.800]  expression to appear in the top. In queue is something, is a way to in queue a command
[08:03.440 --> 08:08.080]  after you're finished awaiting it. Grab, repeat, send the same thing over and over,
[08:08.080 --> 08:13.920]  maybe add an interval, send a file, and we already saw what delay does, setting the delay.
[08:15.200 --> 08:18.560]  And there are actually a lot of commands. There are 43. Whenever I needed to do something,
[08:18.560 --> 08:25.520]  I added a new one. So if you have any ideas, send me a PR or send a request. There are 43
[08:25.520 --> 08:29.760]  different commands right now. Actually, 44. I think I added one this morning.
[08:29.760 --> 08:39.040]  Okay, so script, anybody here use expect? A few people. Okay, so expect is been around for a
[08:39.040 --> 08:46.560]  long time, 1993, but it's still pretty useful if you have to interact with a program that requires
[08:46.560 --> 08:54.080]  a TTY. So here is an example of an expect script on the left. In this, in this case,
[08:54.080 --> 08:58.400]  what we're going to do is we're going to start a Docker container again, and then we're going to run
[08:58.400 --> 09:04.720]  user add to add a user. And then we'd like to set a password for the user. So we're going to run
[09:04.720 --> 09:10.160]  the password command, and we're also going to look for the prompts that are coming back. Okay, so
[09:11.840 --> 09:18.080]  on the left we see the way expect works is you say spawn, and then expect takes a pattern. So
[09:18.080 --> 09:24.480]  root at is what comes back in the prompt. And then you send user add, termy, we're going to add a
[09:24.480 --> 09:30.640]  username to termy. Then it has a regular expression expect dash re that you can then capture with
[09:30.640 --> 09:36.560]  the expect out. And then finally at the bottom, we're going to print out what we caught, like
[09:36.560 --> 09:41.440]  we captured the fact that the host name was something that was in the prompt. So you can do
[09:41.440 --> 09:46.560]  the same thing in termy. You can say a user bin end of termy and then set it to be an executable
[09:46.560 --> 09:52.560]  file. The default is to just send everything. So it's just kind of like you're interacting from
[09:52.560 --> 09:58.800]  the console. All the lines just get sent directly to the other pane. So you just say docker run,
[09:58.800 --> 10:05.760]  and then backslash expect is just kind of like the expect command. There's a little subtlety in
[10:05.760 --> 10:12.000]  there that you usually don't think about because it's sort of intuitive as a human, you know,
[10:12.000 --> 10:15.760]  you type it and you're expecting something. But really there's a race condition there, right?
[10:15.760 --> 10:21.120]  Because between the time that you send your command and the time you send the expect command,
[10:21.120 --> 10:26.160]  the output might have already happened. So the way that expect deals with it is it keeps track of
[10:26.160 --> 10:31.760]  the stream and then it kind of goes back and there's sort of this running, you know, a pointer to the
[10:31.760 --> 10:38.720]  output stream. And the way we do that with termy is we basically run the expect before we run the
[10:38.720 --> 10:44.800]  command. So it'll basically say now I'm starting to watch the output and then it'll send the output
[10:44.800 --> 10:51.520]  and then it'll capture it. So when we run this and it runs interactively in Tmux,
[10:52.800 --> 10:58.080]  so you can see on the top it sends docker run, pulls the image and it's waiting now to see
[10:58.640 --> 11:05.200]  the prompt. So it waits for the root and then after that it sends the user add, waits for new
[11:05.200 --> 11:14.720]  password and then finishes successfully. So the output here is in the test anything protocol.
[11:15.280 --> 11:19.200]  You may or may not be familiar with it, it's just okay and then the number of the test.
[11:19.200 --> 11:23.520]  And then currently if it doesn't get what it expects then it aborts the tests.
[11:24.640 --> 11:30.320]  Okay, so quickly I'm just going to say why it's written in Raku. And the main idea here is that
[11:30.320 --> 11:37.600]  Raku has a lot of very nice ways to do asynchronous programming and also interact with other
[11:37.600 --> 11:46.960]  commands. So you can quickly open a command, open a pipe to Tmux and interact with it using
[11:46.960 --> 11:53.520]  asynchronous processes. You can run things synchronously. It's got good not just inter-process
[11:53.520 --> 12:00.400]  but inter-thread communication. It has built-in constructs that like supplies and channels
[12:00.400 --> 12:07.040]  and promises which you may be familiar with from other languages. So this is like an example of
[12:07.040 --> 12:13.680]  how you could tail a file and create a supply which is a built-in type in Raku. And finally,
[12:14.240 --> 12:18.160]  this is kind of like the implementation of expect. So it's really, it's really pretty
[12:18.160 --> 12:25.280]  straightforward. You set, you can basically set up an event loop in a separate thread using this
[12:26.320 --> 12:32.320]  construct, you know, start, starts a thread, react whenever it says here's an event loop.
[12:32.320 --> 12:37.360]  And then when, if it's a string, we look to see if it contains the target, if it's a regex,
[12:38.160 --> 12:44.480]  then we send it to a channel so that then we can have it available to use locally. So even if
[12:44.480 --> 12:49.120]  you're not interested in using Termi, you might find some value in using Raku for automation.
[12:51.040 --> 12:52.560]  That's the end. Thank you for listening.
[12:52.560 --> 13:22.160]  Questions? I think I have a few minutes. Yeah, that's a good question. So different shells
[13:22.160 --> 13:27.360]  do different things that you kind of are not really aware of, even just the simple like printing
[13:27.360 --> 13:32.000]  of a prompt. They don't always send a new line character. Sometimes they'll send escape sequences
[13:32.000 --> 13:36.800]  that go to the beginning of the line and then go down the line or sometimes they'll even redraw
[13:36.800 --> 13:44.240]  the line above it. So it works fine. But you just have to be aware of the idiosyncrasies of the
[13:44.240 --> 13:50.400]  various shells in terms of what they do to the terminal. It does interfere. You can, one of the,
[13:50.400 --> 13:54.640]  one of the commands is to buffer the lines and it does get tricky to split up the lines when
[13:54.640 --> 14:08.880]  there's a lot of cursor movement. Sorry, with serial consoles. So that, that aspect is basically
[14:08.880 --> 14:15.520]  taken care of by Tmux. So it doesn't do the direct communication with the serial console the way
[14:15.520 --> 14:36.400]  that expect would. Yep. We have one more minute. Last question. So we only have a handful of users.
[14:36.400 --> 14:42.400]  So now is your opportunity to request features. We don't have to worry too much about backwards
[14:42.400 --> 14:52.400]  compatibility. Yep. That's it. Thank you.