I'm going to talk about a project I've been building on and off for the last two years
also.
So to get started, who am I?
My name is Ellie and I'm the lead infrastructure engineer at a company called Post Hog.
When I'm not writing software for work, I try and maintain a couple of side projects
in my free time and when I don't have the energy for that, I'm normally exploring outdoors,
which as you can probably see is usually on a motorbike for me.
So to dive into a two-in, first of all, I'm going to start with the name.
Originally it was called Shink for like shell and sink, but I couldn't really say that out
loud without cringing.
So I looked at something new.
I've been a fan of Terry Pratchett's disc world books for a really long time and for
those who are unfamiliar, the sort of premise there is that the world is a disc and it rests
on the shoulders of four giant elephants stood on the shell of a space turtle called the
great a two-in, which I'm probably mispronouncing.
I thought it would be a bit pretentious to include the words the great in my project
name and putting an apostrophe in a binary is probably not a good idea.
So I ended up with just the name a two-in.
A little bit more specifically, a two-in was made to synchronize shell history between
multiple computers.
So I had the problem that I would be switching between a whole bunch of laptops.
I'd be remoting into various different boxes and trying to find one command that I ran
a few days previously on whichever computer it was was pretty difficult.
So I wanted it all in the same place.
The first thing I did was replace the normal ZSH history, bash history, or whatever fish
uses, I don't really remember, with a SQLite database.
And we could then have some functions to import your normal text history into the database
and because databases are a little bit more flexible than flat text files, we could also
include some additional context.
So in the case of a two-in, this is context such as how long a command took to run, whether
or not it was successful, which directory it was ran in, as well as the shell session.
So the way we do this is we plug into your shell.
If your shell supports it, it's via the normal shell hooks, like pre-command or pre-exact
and post-command, I think they're called.
But in the case of bash, which I do not have positive feelings towards, we do a really
horrible pack with the prompt.
So hopefully you can see the GIF on the right.
On top of this database, we also built a search TUI.
And this is bound by default to control R and the off arrow, which is a little bit contentious
for some people, so you can remap that too.
Search UI has three different search modes.
By default, one of them is a fuzzy search, kind of inspired by FZF.
The other is a prefix search, which is pretty self-explanatory and a substring search, which
same thing.
You should know what that means.
We also have several different filter modes.
So a TUI allows you to search your shell history for the current session, for the current directory,
for the current machine, or just all of your shell history for every machine ever that
you've connected anyway.
It would be cool if it could have otherwise.
A little bit more on that extra context.
A turn has a stats command, which kind of analyzes all of your history and will show
you things like the most used command, which for me is LS.
I didn't realize I ran that so much.
How many commands you have ran, as well as how many unique commands you've ran.
We're definitely not making the most of all the data available, and there's a lot more
sort of cool analysis we could do.
And you can also get the stats for a specific day or week or month or whatever.
A little bit more on the search.
You don't have to use the search UI.
We also have a command line search interface.
This is kind of useful if you have like a specific command in mind.
Maybe you know roughly when it was or roughly what it looks like.
And it's also useful to integrate with other tools.
So someone on the Discord told me that apparently they've used this to integrate directly with
FZF as their search instead, which is pretty cool.
So you can see here that I'm searching for all successfully ran commands after yesterday
at 3 p.m. that start with git.
Obviously, I did not make these slides today.
The time specifier supports like a human way of expressing time, and the command search
supports regular expressions.
A little bit more about the sync server.
It's a kind of pretty boring HTTP API that shares blobs.
It has no idea what the blobs actually contain.
And it was originally written in with warp, which I found to be very fun, kind of nice
mental exercise, I guess.
And we ended up rewriting with AXM because while warp was fun, it was difficult for contributors
to figure out how to use, and it also contributed pretty massively to a high compile time.
And AXM is just sort of the problems there.
The ato and sync server is completely self-hostable.
Anyone with it installed can just run ato and server and have a running server.
We also have Docker images and Kubernetes manifests for anyone that wants to get a little bit
more fancy.
And a little bit more on the sync is that it's not quite real time yet.
While I would love it if it was, it currently syncs an interval of 15 minutes, and you can
reduce this down to zero, which basically means it will sync after every single command.
If you know fancy running your own infrastructure, there's a public deployment of it in the IRAN.
Currently it's got about 11 million lines of shell history on it.
There's about 300 active users.
And it's all running on just one dedicated Hesner box, and it handles way more requests
than I thought it ever would.
I'd also like to thank the GitHub sponsors I got, which I didn't really expect anyone
to contribute, but they cover the server bills entirely now, which is a really nice feeling.
And a little bit more about privacy.
I imagine people here probably feel more strongly about that than others.
Everything's fully end-to-end encrypted in the sync because I really don't want the responsibility
of people's accidentally pasted into a shell API keys on my machine.
We use LibSodium secret box because I'm not at all a cryptographer, and it's more difficult
to mess up than most other things.
Finding a reliably maintained library for that was a bit tricky.
The original bindings we used were not maintained beyond security patches.
We recently switched to, I think, Rust Crypto, for a member rightly.
All of the encryption keys get automatically generated when you log in, and you have to
keep track of them yourself.
So if you lose your keys, there's nothing I can do.
Your data's gone.
So why Rust?
This is the Rust dev room, after all.
It runs twice for every shell command you run.
So it runs just before and just afterwards.
It lets us get the timing data and everything else.
And if we had latency there for an interpreter to start up or a runtime to do whatever it
does, the experience would not be great.
If you added 50 to 100 milliseconds to every command you ran, people would pretty rightfully
complain.
So Rust fits the bill very nicely there.
It also has to be reliable because if we're dropping shell history randomly, then it's
not at all serving the purpose it was written for.
Having a static binary to deploy is also really nice.
No one has to make sure they have Rust 3.7 not pointing any languages in particular installed
on the system with the right versions of various libraries installed or anything like that.
And it's also safe, so you don't have to worry about any memory issues or anything like that.
The other factor which I think for a side project is especially important is that Rust
is fun.
When I started this project, I was also considering using Go, and I was also writing Go for my
day job.
And I didn't really fancy the idea of getting home after work, writing Go all day, and then
writing some more Go.
So Rust solved that very nicely, and I think the main reason I actually got around to finishing
this is because I was enjoying writing it.
Additionally, the Rust community is fantastic.
Every time I've asked for help, people have been really helpful.
Everything I wanted has been available, and they're just generally very welcoming and
accepting, especially compared to some other tech communities.
So I actually have one other service, and I'm glad most of the previous talks have discussed
Python, because now I don't feel as weird for mentioning it in my presentation too.
I have another service called Rinsewind, a bit of a naming pattern there, if anyone's
familiar with it.
And what this basically does is it peeks into the database and generates graphs like this,
which are heavily inspired by the GitHub commit activity chart, but for your shell history.
And it's currently closed source for no real reason other than that it's a really horrible
hack that I don't want to package nicely for anyone.
It mostly uses NumPy and OpenCV and a few other things.
It's also completely opt-in, so you don't get this by default if you don't want any proprietary
code touching your data.
You don't have to.
It's cool.
Just with one curl command, you enable this.
On the open source side of things, this is the first open source project.
I've released that people have actually been interested in.
I made it just for myself and stuck it on my GitHub, and it ended up being quite well
received by a whole bunch of people.
We ended up in a lot of package repositories.
I think off the top of my head it's the Arch Linux community repo, Homebrew, Alpine Linux,
and some Nix.
I'm not entirely sure how Nix works, but one of the Nix repositories.
And there's probably a whole bunch more that I'm not aware of.
And we've actually got 63 contributors at sort of as of today.
Some of them are sort of returning regular contributors, which is very nice that people
want to regularly give time to my project.
Some of them are just sort of drive by.
They found something that annoyed them or bug they wanted to fix or something like that,
so they contributed, which was lovely.
I'd also like to especially thank Conrad.
He's much more involved in the Rust community than I am and also a very long-term friend
of mine.
He helps me maintain a twin, and when I was first starting and not so good at Rust, he
did a great job of tidying things up a bit.
In terms of the future, right now a twin has a bit of a flaw in that you can't actually
delete history once it's been synced.
This is mostly because the sync's pretty eventually consistent, and every machine you have is a
potential writer, so ensuring that you delete something and it stays deleted is actually
really difficult.
I've currently got a solution to it, which works on my laptop.
I just need to make sure it works on everyone else's too.
I'd also like to sort out bash, because pretty much all the complaints we get about shell
integrations are from people running bash, and it's very frustrating.
I think I don't actually use bash, and I hate having a setup on my machine just for that.
I'd also like to show some more information in the TUI, so I don't know if you saw very
much on the GIF earlier, but it basically just shows what's useful for search results.
I would love it if there was another tab where you could also see sort of statistics about
a command that's run, maybe how often it succeeds versus fails, you could get some nice stats
about make build that way, and that sort of thing.
I'd like to improve the search a little bit too, because right now it's good enough, and
I think it could always be improved.
I've been meaning to explore some of the full-text search modules that SQLite has, or maybe
something like Tantivi or one of the other search libraries in Rust.
Otherwise, I'd really like to improve the sorting.
Right now we sort chronologically, which is a pretty safe default.
I'm not going to turn this into a horrible Twitter timeline type thing, but it would
be nice if we could sort based on the context we have.
Maybe every day at 9 a.m. you CD into your repo and you run GitPool.
By default, it would be nice if you pressed Ctrl R, and GitPool was already there at
the time that you frequently run it.
We've got all the data for that, it just needs to be plugged together.
In the even further future, the number of people that have spoken to me about the fact
that they have development API keys in their shell history, it would be nice if we could
do something to get that out of the shell history and sync that alongside the data.
Being able to bookmark commands is also something I would quite like to be able to do, because
there's some longer commands I run frequently in search for, frequently having some sort
of hockey or alias would be really nice.
Otherwise, I realized that a subset of a two-end history could also be used as a runbook if
you had to begin and an end marker to it, and you could just replay some commands from
your past.
That's actually it.
I went a bit faster than I was expecting, but if there are any questions, I'd be very
happy to answer them.
Can you search for things which have come after your most recent command frequently?
I'm not sure what you mean, sorry.
So to take what you've just typed and see what you typically do next, so actually returning
the command after the one you've searched for.
That's one of the things I'd love to be able to do with the smarter ordering is know that
like a sequence of commands that's commonly run and predict the next one based on history,
if that's, yeah.
So I tried to install your tool, but I'm using Bash, and I was wondering how far are you with
like fixing Bash?
Bash generally works fine.
It's usually the people that have a whole bunch of Bash plugins installed or have a
weird Bash prompt that start to have some issues, but generally, it's okay for most
people.
Yeah.
Sorry.
Does it handle having different cells in different computers?
For example, if I'm using one computer piece and another CS8, does the same work between
those two?
Yes.
So we translate from whatever your shell uses natively into the format we use, so whichever
shell you use on each machine doesn't matter.
Okay.
Thanks.
I have a couple of questions.
First, I didn't quite get how do you authenticate with the server by having a key analysis?
So the sort of user authentication is just a username and password, but then your actual
data is encrypted by a key that's only held locally.
All right.
And second question.
Do you have a ZSH plugin or have you considered one?
So we have a ZSH plugin.
You can use normal ZSH plugin managers to install and use it.
All right.
Thank you.
Getting some exercise in.
Is it possible to disable the history for a few commands and then re-enable it?
Not currently.
We have spoken about the idea of like an incognito mode.
If you prefix a command with a space, it won't be saved, but it's kind of annoying if you
got to run a lot of them in a row.
We have some questions from the matrix.
So Olivier Robert says, how would it interact with something like Starship?
I actually use Starship and it doesn't interact with it at all in that it works completely
fine.
Awesome.
And yep, that was the other question.
Cool.
Thank you.
There's one at the front, too.
Two short questions.
The first one is, since I'm using BESH, what's your favorite shell?
I like ZSH, I think purely because I started using it maybe 10 years ago and have it so
hard to break.
I think if I was going to start again, I'd probably try FISH a bit more.
And a question about the timestamps, are you using the client-side timestamps from the
machines or server-side?
So we actually store client-side, the timestamp will be whatever your client is, but we actually
use two timestamps to sync to work.
So we have the server-local timestamp, which is only really used for syncing, and then
the actual data, it's all encrypted and hidden, so it's whatever your client stores.
Because sometimes the local timestamp is important if you want to sync with a system or whatever,
but sometimes also the real time, if the computers are out of sync, which that should happen.
I had a bunch of issues with timestamps when I was first writing it, but we got it all
sorted out in the end.
Is there a limit to the length of a command, for example, imagine a huge pipeline with
the SQLs and JQL queries in there?
Currently it's eight megabytes of whatever it is once it's been encrypted, it's only
a server-side limit, and it's pretty arbitrary.
So another question, any plans for special handling for similar commands?
We do fix syntax, run similar commands in a row?
I hadn't really thought of that before, but it might be worth considering.
Sorry, I did have a few more questions from Matrix.
I think my device is not synchronizing properly, but Andy sent me a screenshot.
So does it integrate with regular history mechanisms provided by the shell?
For example, excluding certain commands automatically like CDNLS, skipping storing in history by
prefixing with white space for sensitive commands, et cetera.
So the prefixing with white space is included, the default ignoring is not, but it doesn't
actually replace the text file history either.
You will still write to that if you ever decide, do you want to stop using it?
And where would context to where recommendations come from?
So if we have a history of your shell, we know the directories you're in, we know what
commands you've been running at what times, so if we're predicting the next command that
you want to run, we can use your own history.
So, but the question follows up with, it's end-to-end encrypted?
Oh, it would all be from the client.
So there's nothing, the server's just a dumb blob store, it doesn't really know much of
anything.
Any more questions?
I think that's it.
Awesome.
Thank you.
That was really well.