[00:00.000 --> 00:19.680] I'm going to talk about a project I've been building on and off for the last two years [00:19.680 --> 00:20.680] also. [00:20.680 --> 00:23.560] So to get started, who am I? [00:23.560 --> 00:29.080] My name is Ellie and I'm the lead infrastructure engineer at a company called Post Hog. [00:29.080 --> 00:33.000] When I'm not writing software for work, I try and maintain a couple of side projects [00:33.000 --> 00:37.480] in my free time and when I don't have the energy for that, I'm normally exploring outdoors, [00:37.480 --> 00:41.040] which as you can probably see is usually on a motorbike for me. [00:41.040 --> 00:46.640] So to dive into a two-in, first of all, I'm going to start with the name. [00:46.640 --> 00:50.680] Originally it was called Shink for like shell and sink, but I couldn't really say that out [00:50.680 --> 00:52.280] loud without cringing. [00:52.280 --> 00:55.040] So I looked at something new. [00:55.040 --> 00:59.920] I've been a fan of Terry Pratchett's disc world books for a really long time and for [00:59.920 --> 01:05.560] those who are unfamiliar, the sort of premise there is that the world is a disc and it rests [01:05.560 --> 01:10.440] on the shoulders of four giant elephants stood on the shell of a space turtle called the [01:10.440 --> 01:13.960] great a two-in, which I'm probably mispronouncing. [01:13.960 --> 01:17.720] I thought it would be a bit pretentious to include the words the great in my project [01:17.720 --> 01:22.040] name and putting an apostrophe in a binary is probably not a good idea. [01:22.040 --> 01:25.480] So I ended up with just the name a two-in. [01:25.480 --> 01:30.960] A little bit more specifically, a two-in was made to synchronize shell history between [01:30.960 --> 01:32.600] multiple computers. [01:32.600 --> 01:37.080] So I had the problem that I would be switching between a whole bunch of laptops. [01:37.080 --> 01:43.040] I'd be remoting into various different boxes and trying to find one command that I ran [01:43.040 --> 01:47.800] a few days previously on whichever computer it was was pretty difficult. [01:47.800 --> 01:50.400] So I wanted it all in the same place. [01:50.400 --> 01:56.120] The first thing I did was replace the normal ZSH history, bash history, or whatever fish [01:56.120 --> 02:00.320] uses, I don't really remember, with a SQLite database. [02:00.320 --> 02:06.080] And we could then have some functions to import your normal text history into the database [02:06.080 --> 02:11.080] and because databases are a little bit more flexible than flat text files, we could also [02:11.080 --> 02:13.120] include some additional context. [02:13.120 --> 02:19.560] So in the case of a two-in, this is context such as how long a command took to run, whether [02:19.560 --> 02:25.400] or not it was successful, which directory it was ran in, as well as the shell session. [02:25.400 --> 02:30.240] So the way we do this is we plug into your shell. [02:30.240 --> 02:34.520] If your shell supports it, it's via the normal shell hooks, like pre-command or pre-exact [02:34.520 --> 02:36.240] and post-command, I think they're called. [02:36.240 --> 02:42.720] But in the case of bash, which I do not have positive feelings towards, we do a really [02:42.720 --> 02:47.800] horrible pack with the prompt. [02:47.800 --> 02:51.240] So hopefully you can see the GIF on the right. [02:51.240 --> 02:55.560] On top of this database, we also built a search TUI. [02:55.560 --> 03:00.920] And this is bound by default to control R and the off arrow, which is a little bit contentious [03:00.920 --> 03:05.520] for some people, so you can remap that too. [03:05.520 --> 03:08.320] Search UI has three different search modes. [03:08.320 --> 03:13.240] By default, one of them is a fuzzy search, kind of inspired by FZF. [03:13.240 --> 03:18.960] The other is a prefix search, which is pretty self-explanatory and a substring search, which [03:18.960 --> 03:19.960] same thing. [03:19.960 --> 03:21.960] You should know what that means. [03:21.960 --> 03:24.440] We also have several different filter modes. [03:24.440 --> 03:30.680] So a TUI allows you to search your shell history for the current session, for the current directory, [03:30.680 --> 03:35.560] for the current machine, or just all of your shell history for every machine ever that [03:35.560 --> 03:36.560] you've connected anyway. [03:36.560 --> 03:38.560] It would be cool if it could have otherwise. [03:38.560 --> 03:41.560] A little bit more on that extra context. [03:41.560 --> 03:46.360] A turn has a stats command, which kind of analyzes all of your history and will show [03:46.360 --> 03:49.280] you things like the most used command, which for me is LS. [03:49.280 --> 03:52.400] I didn't realize I ran that so much. [03:52.400 --> 03:57.560] How many commands you have ran, as well as how many unique commands you've ran. [03:57.560 --> 04:01.440] We're definitely not making the most of all the data available, and there's a lot more [04:01.440 --> 04:04.400] sort of cool analysis we could do. [04:04.400 --> 04:10.880] And you can also get the stats for a specific day or week or month or whatever. [04:10.880 --> 04:12.320] A little bit more on the search. [04:12.320 --> 04:15.440] You don't have to use the search UI. [04:15.440 --> 04:19.240] We also have a command line search interface. [04:19.240 --> 04:23.480] This is kind of useful if you have like a specific command in mind. [04:23.480 --> 04:26.920] Maybe you know roughly when it was or roughly what it looks like. [04:26.920 --> 04:29.120] And it's also useful to integrate with other tools. [04:29.120 --> 04:34.520] So someone on the Discord told me that apparently they've used this to integrate directly with [04:34.520 --> 04:37.480] FZF as their search instead, which is pretty cool. [04:37.480 --> 04:42.960] So you can see here that I'm searching for all successfully ran commands after yesterday [04:42.960 --> 04:45.480] at 3 p.m. that start with git. [04:45.480 --> 04:50.320] Obviously, I did not make these slides today. [04:50.320 --> 04:56.800] The time specifier supports like a human way of expressing time, and the command search [04:56.800 --> 04:58.320] supports regular expressions. [04:58.320 --> 05:02.200] A little bit more about the sync server. [05:02.200 --> 05:07.880] It's a kind of pretty boring HTTP API that shares blobs. [05:07.880 --> 05:10.960] It has no idea what the blobs actually contain. [05:10.960 --> 05:18.000] And it was originally written in with warp, which I found to be very fun, kind of nice [05:18.000 --> 05:20.120] mental exercise, I guess. [05:20.120 --> 05:26.440] And we ended up rewriting with AXM because while warp was fun, it was difficult for contributors [05:26.440 --> 05:32.920] to figure out how to use, and it also contributed pretty massively to a high compile time. [05:32.920 --> 05:36.280] And AXM is just sort of the problems there. [05:36.280 --> 05:39.400] The ato and sync server is completely self-hostable. [05:39.400 --> 05:43.640] Anyone with it installed can just run ato and server and have a running server. [05:43.640 --> 05:47.680] We also have Docker images and Kubernetes manifests for anyone that wants to get a little bit [05:47.680 --> 05:48.680] more fancy. [05:48.680 --> 05:54.120] And a little bit more on the sync is that it's not quite real time yet. [05:54.120 --> 05:59.400] While I would love it if it was, it currently syncs an interval of 15 minutes, and you can [05:59.400 --> 06:06.400] reduce this down to zero, which basically means it will sync after every single command. [06:06.400 --> 06:12.840] If you know fancy running your own infrastructure, there's a public deployment of it in the IRAN. [06:12.840 --> 06:15.760] Currently it's got about 11 million lines of shell history on it. [06:15.760 --> 06:18.280] There's about 300 active users. [06:18.280 --> 06:23.480] And it's all running on just one dedicated Hesner box, and it handles way more requests [06:23.480 --> 06:24.840] than I thought it ever would. [06:24.840 --> 06:29.800] I'd also like to thank the GitHub sponsors I got, which I didn't really expect anyone [06:29.800 --> 06:34.840] to contribute, but they cover the server bills entirely now, which is a really nice feeling. [06:34.840 --> 06:38.440] And a little bit more about privacy. [06:38.440 --> 06:43.440] I imagine people here probably feel more strongly about that than others. [06:43.440 --> 06:47.960] Everything's fully end-to-end encrypted in the sync because I really don't want the responsibility [06:47.960 --> 06:54.200] of people's accidentally pasted into a shell API keys on my machine. [06:54.200 --> 06:59.560] We use LibSodium secret box because I'm not at all a cryptographer, and it's more difficult [06:59.560 --> 07:03.360] to mess up than most other things. [07:03.360 --> 07:06.920] Finding a reliably maintained library for that was a bit tricky. [07:06.920 --> 07:13.040] The original bindings we used were not maintained beyond security patches. [07:13.040 --> 07:16.680] We recently switched to, I think, Rust Crypto, for a member rightly. [07:16.680 --> 07:20.600] All of the encryption keys get automatically generated when you log in, and you have to [07:20.600 --> 07:21.960] keep track of them yourself. [07:21.960 --> 07:24.480] So if you lose your keys, there's nothing I can do. [07:24.480 --> 07:27.120] Your data's gone. [07:27.120 --> 07:28.120] So why Rust? [07:28.120 --> 07:31.600] This is the Rust dev room, after all. [07:31.600 --> 07:34.360] It runs twice for every shell command you run. [07:34.360 --> 07:36.520] So it runs just before and just afterwards. [07:36.520 --> 07:40.120] It lets us get the timing data and everything else. [07:40.120 --> 07:45.240] And if we had latency there for an interpreter to start up or a runtime to do whatever it [07:45.240 --> 07:48.680] does, the experience would not be great. [07:48.680 --> 07:53.760] If you added 50 to 100 milliseconds to every command you ran, people would pretty rightfully [07:53.760 --> 07:54.760] complain. [07:54.760 --> 07:58.080] So Rust fits the bill very nicely there. [07:58.080 --> 08:04.760] It also has to be reliable because if we're dropping shell history randomly, then it's [08:04.760 --> 08:08.520] not at all serving the purpose it was written for. [08:08.520 --> 08:12.000] Having a static binary to deploy is also really nice. [08:12.000 --> 08:17.440] No one has to make sure they have Rust 3.7 not pointing any languages in particular installed [08:17.440 --> 08:23.120] on the system with the right versions of various libraries installed or anything like that. [08:23.120 --> 08:28.000] And it's also safe, so you don't have to worry about any memory issues or anything like that. [08:28.000 --> 08:33.280] The other factor which I think for a side project is especially important is that Rust [08:33.280 --> 08:35.280] is fun. [08:35.280 --> 08:40.960] When I started this project, I was also considering using Go, and I was also writing Go for my [08:40.960 --> 08:41.960] day job. [08:41.960 --> 08:46.920] And I didn't really fancy the idea of getting home after work, writing Go all day, and then [08:46.920 --> 08:49.320] writing some more Go. [08:49.320 --> 08:53.480] So Rust solved that very nicely, and I think the main reason I actually got around to finishing [08:53.480 --> 08:56.000] this is because I was enjoying writing it. [08:56.000 --> 08:58.920] Additionally, the Rust community is fantastic. [08:58.920 --> 09:02.160] Every time I've asked for help, people have been really helpful. [09:02.160 --> 09:05.320] Everything I wanted has been available, and they're just generally very welcoming and [09:05.320 --> 09:10.560] accepting, especially compared to some other tech communities. [09:10.560 --> 09:15.120] So I actually have one other service, and I'm glad most of the previous talks have discussed [09:15.120 --> 09:20.120] Python, because now I don't feel as weird for mentioning it in my presentation too. [09:20.120 --> 09:24.120] I have another service called Rinsewind, a bit of a naming pattern there, if anyone's [09:24.120 --> 09:25.560] familiar with it. [09:25.560 --> 09:30.920] And what this basically does is it peeks into the database and generates graphs like this, [09:30.920 --> 09:36.560] which are heavily inspired by the GitHub commit activity chart, but for your shell history. [09:36.560 --> 09:41.320] And it's currently closed source for no real reason other than that it's a really horrible [09:41.320 --> 09:45.480] hack that I don't want to package nicely for anyone. [09:45.480 --> 09:49.280] It mostly uses NumPy and OpenCV and a few other things. [09:49.280 --> 09:54.360] It's also completely opt-in, so you don't get this by default if you don't want any proprietary [09:54.360 --> 09:55.480] code touching your data. [09:55.480 --> 09:56.480] You don't have to. [09:56.480 --> 09:57.480] It's cool. [09:57.480 --> 10:01.920] Just with one curl command, you enable this. [10:01.920 --> 10:06.200] On the open source side of things, this is the first open source project. [10:06.200 --> 10:09.600] I've released that people have actually been interested in. [10:09.600 --> 10:14.120] I made it just for myself and stuck it on my GitHub, and it ended up being quite well [10:14.120 --> 10:16.320] received by a whole bunch of people. [10:16.320 --> 10:19.320] We ended up in a lot of package repositories. [10:19.320 --> 10:25.360] I think off the top of my head it's the Arch Linux community repo, Homebrew, Alpine Linux, [10:25.360 --> 10:26.360] and some Nix. [10:26.360 --> 10:30.080] I'm not entirely sure how Nix works, but one of the Nix repositories. [10:30.080 --> 10:33.600] And there's probably a whole bunch more that I'm not aware of. [10:33.600 --> 10:38.760] And we've actually got 63 contributors at sort of as of today. [10:38.760 --> 10:42.720] Some of them are sort of returning regular contributors, which is very nice that people [10:42.720 --> 10:45.600] want to regularly give time to my project. [10:45.600 --> 10:47.400] Some of them are just sort of drive by. [10:47.400 --> 10:50.920] They found something that annoyed them or bug they wanted to fix or something like that, [10:50.920 --> 10:53.320] so they contributed, which was lovely. [10:53.320 --> 10:56.000] I'd also like to especially thank Conrad. [10:56.000 --> 10:59.560] He's much more involved in the Rust community than I am and also a very long-term friend [10:59.560 --> 11:00.960] of mine. [11:00.960 --> 11:05.640] He helps me maintain a twin, and when I was first starting and not so good at Rust, he [11:05.640 --> 11:10.640] did a great job of tidying things up a bit. [11:10.640 --> 11:15.600] In terms of the future, right now a twin has a bit of a flaw in that you can't actually [11:15.600 --> 11:18.960] delete history once it's been synced. [11:18.960 --> 11:23.080] This is mostly because the sync's pretty eventually consistent, and every machine you have is a [11:23.080 --> 11:27.520] potential writer, so ensuring that you delete something and it stays deleted is actually [11:27.520 --> 11:28.520] really difficult. [11:28.520 --> 11:32.760] I've currently got a solution to it, which works on my laptop. [11:32.760 --> 11:36.640] I just need to make sure it works on everyone else's too. [11:36.640 --> 11:41.720] I'd also like to sort out bash, because pretty much all the complaints we get about shell [11:41.720 --> 11:46.200] integrations are from people running bash, and it's very frustrating. [11:46.200 --> 11:51.720] I think I don't actually use bash, and I hate having a setup on my machine just for that. [11:51.720 --> 11:56.680] I'd also like to show some more information in the TUI, so I don't know if you saw very [11:56.680 --> 12:01.080] much on the GIF earlier, but it basically just shows what's useful for search results. [12:01.080 --> 12:05.360] I would love it if there was another tab where you could also see sort of statistics about [12:05.360 --> 12:10.360] a command that's run, maybe how often it succeeds versus fails, you could get some nice stats [12:10.360 --> 12:13.760] about make build that way, and that sort of thing. [12:13.760 --> 12:20.240] I'd like to improve the search a little bit too, because right now it's good enough, and [12:20.240 --> 12:21.720] I think it could always be improved. [12:21.720 --> 12:26.440] I've been meaning to explore some of the full-text search modules that SQLite has, or maybe [12:26.440 --> 12:30.640] something like Tantivi or one of the other search libraries in Rust. [12:30.640 --> 12:35.040] Otherwise, I'd really like to improve the sorting. [12:35.040 --> 12:38.960] Right now we sort chronologically, which is a pretty safe default. [12:38.960 --> 12:43.080] I'm not going to turn this into a horrible Twitter timeline type thing, but it would [12:43.080 --> 12:46.680] be nice if we could sort based on the context we have. [12:46.680 --> 12:50.920] Maybe every day at 9 a.m. you CD into your repo and you run GitPool. [12:50.920 --> 12:54.720] By default, it would be nice if you pressed Ctrl R, and GitPool was already there at [12:54.720 --> 12:56.760] the time that you frequently run it. [12:56.760 --> 13:01.120] We've got all the data for that, it just needs to be plugged together. [13:01.120 --> 13:06.880] In the even further future, the number of people that have spoken to me about the fact [13:06.880 --> 13:11.840] that they have development API keys in their shell history, it would be nice if we could [13:11.840 --> 13:17.600] do something to get that out of the shell history and sync that alongside the data. [13:17.600 --> 13:23.560] Being able to bookmark commands is also something I would quite like to be able to do, because [13:23.560 --> 13:28.000] there's some longer commands I run frequently in search for, frequently having some sort [13:28.000 --> 13:30.920] of hockey or alias would be really nice. [13:30.920 --> 13:37.520] Otherwise, I realized that a subset of a two-end history could also be used as a runbook if [13:37.520 --> 13:42.280] you had to begin and an end marker to it, and you could just replay some commands from [13:42.280 --> 13:44.280] your past. [13:44.280 --> 13:46.160] That's actually it. [13:46.160 --> 13:50.400] I went a bit faster than I was expecting, but if there are any questions, I'd be very [13:50.400 --> 14:06.560] happy to answer them. [14:06.560 --> 14:12.520] Can you search for things which have come after your most recent command frequently? [14:12.520 --> 14:14.400] I'm not sure what you mean, sorry. [14:14.400 --> 14:19.440] So to take what you've just typed and see what you typically do next, so actually returning [14:19.440 --> 14:21.880] the command after the one you've searched for. [14:21.880 --> 14:25.960] That's one of the things I'd love to be able to do with the smarter ordering is know that [14:25.960 --> 14:30.560] like a sequence of commands that's commonly run and predict the next one based on history, [14:30.560 --> 14:41.760] if that's, yeah. [14:41.760 --> 14:47.040] So I tried to install your tool, but I'm using Bash, and I was wondering how far are you with [14:47.040 --> 14:48.720] like fixing Bash? [14:48.720 --> 14:50.240] Bash generally works fine. [14:50.240 --> 14:55.400] It's usually the people that have a whole bunch of Bash plugins installed or have a [14:55.400 --> 15:00.520] weird Bash prompt that start to have some issues, but generally, it's okay for most [15:00.520 --> 15:01.520] people. [15:01.520 --> 15:02.520] Yeah. [15:02.520 --> 15:03.520] Sorry. [15:03.520 --> 15:16.960] Does it handle having different cells in different computers? [15:16.960 --> 15:23.640] For example, if I'm using one computer piece and another CS8, does the same work between [15:23.640 --> 15:24.640] those two? [15:24.640 --> 15:25.640] Yes. [15:25.640 --> 15:31.120] So we translate from whatever your shell uses natively into the format we use, so whichever [15:31.120 --> 15:33.120] shell you use on each machine doesn't matter. [15:33.120 --> 15:34.120] Okay. [15:34.120 --> 15:35.120] Thanks. [15:35.120 --> 15:40.320] I have a couple of questions. [15:40.320 --> 15:47.120] First, I didn't quite get how do you authenticate with the server by having a key analysis? [15:47.120 --> 15:51.800] So the sort of user authentication is just a username and password, but then your actual [15:51.800 --> 15:55.160] data is encrypted by a key that's only held locally. [15:55.160 --> 15:56.160] All right. [15:56.160 --> 15:57.160] And second question. [15:57.160 --> 16:02.160] Do you have a ZSH plugin or have you considered one? [16:02.160 --> 16:03.440] So we have a ZSH plugin. [16:03.440 --> 16:07.320] You can use normal ZSH plugin managers to install and use it. [16:07.320 --> 16:08.320] All right. [16:08.320 --> 16:09.320] Thank you. [16:09.320 --> 16:21.400] Getting some exercise in. [16:21.400 --> 16:31.120] Is it possible to disable the history for a few commands and then re-enable it? [16:31.120 --> 16:32.120] Not currently. [16:32.120 --> 16:34.880] We have spoken about the idea of like an incognito mode. [16:34.880 --> 16:38.880] If you prefix a command with a space, it won't be saved, but it's kind of annoying if you [16:38.880 --> 16:41.720] got to run a lot of them in a row. [16:41.720 --> 16:46.080] We have some questions from the matrix. [16:46.080 --> 16:51.960] So Olivier Robert says, how would it interact with something like Starship? [16:51.960 --> 16:57.360] I actually use Starship and it doesn't interact with it at all in that it works completely [16:57.360 --> 16:58.360] fine. [16:58.360 --> 16:59.360] Awesome. [16:59.360 --> 17:04.160] And yep, that was the other question. [17:04.160 --> 17:05.160] Cool. [17:05.160 --> 17:06.160] Thank you. [17:06.160 --> 17:10.680] There's one at the front, too. [17:10.680 --> 17:14.560] Two short questions. [17:14.560 --> 17:20.120] The first one is, since I'm using BESH, what's your favorite shell? [17:20.120 --> 17:25.280] I like ZSH, I think purely because I started using it maybe 10 years ago and have it so [17:25.280 --> 17:27.040] hard to break. [17:27.040 --> 17:30.680] I think if I was going to start again, I'd probably try FISH a bit more. [17:30.680 --> 17:36.320] And a question about the timestamps, are you using the client-side timestamps from the [17:36.320 --> 17:37.320] machines or server-side? [17:37.320 --> 17:42.320] So we actually store client-side, the timestamp will be whatever your client is, but we actually [17:42.320 --> 17:44.720] use two timestamps to sync to work. [17:44.720 --> 17:48.320] So we have the server-local timestamp, which is only really used for syncing, and then [17:48.320 --> 17:52.720] the actual data, it's all encrypted and hidden, so it's whatever your client stores. [17:52.720 --> 17:58.000] Because sometimes the local timestamp is important if you want to sync with a system or whatever, [17:58.000 --> 18:03.680] but sometimes also the real time, if the computers are out of sync, which that should happen. [18:03.680 --> 18:06.800] I had a bunch of issues with timestamps when I was first writing it, but we got it all [18:06.800 --> 18:11.760] sorted out in the end. [18:11.760 --> 18:16.200] Is there a limit to the length of a command, for example, imagine a huge pipeline with [18:16.200 --> 18:19.800] the SQLs and JQL queries in there? [18:19.800 --> 18:25.160] Currently it's eight megabytes of whatever it is once it's been encrypted, it's only [18:25.160 --> 18:28.160] a server-side limit, and it's pretty arbitrary. [18:28.160 --> 18:33.600] So another question, any plans for special handling for similar commands? [18:33.600 --> 18:40.000] We do fix syntax, run similar commands in a row? [18:40.000 --> 18:43.440] I hadn't really thought of that before, but it might be worth considering. [18:43.440 --> 18:49.480] Sorry, I did have a few more questions from Matrix. [18:49.480 --> 18:53.960] I think my device is not synchronizing properly, but Andy sent me a screenshot. [18:53.960 --> 18:58.560] So does it integrate with regular history mechanisms provided by the shell? [18:58.560 --> 19:05.040] For example, excluding certain commands automatically like CDNLS, skipping storing in history by [19:05.040 --> 19:08.760] prefixing with white space for sensitive commands, et cetera. [19:08.760 --> 19:14.880] So the prefixing with white space is included, the default ignoring is not, but it doesn't [19:14.880 --> 19:17.920] actually replace the text file history either. [19:17.920 --> 19:21.720] You will still write to that if you ever decide, do you want to stop using it? [19:21.720 --> 19:25.920] And where would context to where recommendations come from? [19:25.920 --> 19:30.680] So if we have a history of your shell, we know the directories you're in, we know what [19:30.680 --> 19:34.400] commands you've been running at what times, so if we're predicting the next command that [19:34.400 --> 19:36.680] you want to run, we can use your own history. [19:36.680 --> 19:40.000] So, but the question follows up with, it's end-to-end encrypted? [19:40.000 --> 19:42.080] Oh, it would all be from the client. [19:42.080 --> 19:45.640] So there's nothing, the server's just a dumb blob store, it doesn't really know much of [19:45.640 --> 19:46.640] anything. [19:46.640 --> 19:53.640] Any more questions? [19:53.640 --> 19:56.640] I think that's it. [19:56.640 --> 19:57.640] Awesome. [19:57.640 --> 19:58.640] Thank you. [19:58.640 --> 20:18.640] That was really well.