Alright. Hello everyone. Thank you for packing the room. No pressure. As Peter mentioned,
I'm Kevin Fleming. I've actually spoken at FOSDM quite a few times before but never
in the DNS room. So this was a bit of a stretch to think that I might even get a talk accepted
here and I'm thankful both that the team did and that you all decided to show up. So as
he mentioned, I don't know if I called a mistake. I made a decision a few years ago about something
that I wanted to do differently than I had been doing and as a result I am now running
my own public DNS server. This is done for personal basis. I don't do this for pay or
anything like that. It's the zero profit part. Nobody pays me to do this. In fact, it actually
costs money so it's negative profit really if you want to think about it that way. Also,
this is a bit odd but I wrote these slides in a somewhat unusual way where every title
of a slide is actually a question from the audience which I will then answer in the slide.
They get somewhat humorous further into the presentation. So if someone in the audience
wants to play the role of the person asking the questions as we go to each slide that would
be fantastic. In fact, you could all do it as a group. That would be even more fantastic.
So if you decide to do that, that's great. If not, I will be happy to read them.
Yes, exactly. So start off with everybody in the audience who currently is responsible
for running authoritative name servers that are reachable on the public internet. Raise
your hand and keep your hand up. Now, if you have your hand up but you're doing that because
you run an internet service provider, put your hand down. So we're down to less than
a quarter of the ones who had their hands up originally. That's more than I thought actually.
I didn't think as many people were as strange as I am. So you can see what this up, I already
covered most of this anyway. It's relatively easy to do obviously because this is phosom.
I wouldn't be talking about if it required non-free software. It does not require anything
that's non-free software. It can be fun for the subset of humanity that would come to
a room like this. And it's also not expensive to do.
So all right, now we're ready. We're going to the next slide.
Okay, why don't you?
Yeah, all right. Why would we do something like this? So the fun part we already covered.
The second part is, for years in my personal infrastructure, I had been using, you're probably
not familiar with Hurricane Electric. Most of them are bigger in the U.S. than they are
here for hosting things. But the effort of free DNS hosting service, which I'd been
using, was working fine. And then I decided I wanted to sort of enhance my DNS usage and
implement DNSSEC for SSHFP records so I could stop having to deal with SSHP distribution
issues. And a couple of other things, like I wanted to get real TLS certificates for
all of my services. And of course, five years ago, Let's Encrypt was just a new thing.
And DNS challenges were the best way to do that, blah, blah, blah.
So the things I wanted, the DNS hosting service that I was using did not offer. So I thought,
fine, I'll go find a different one. And I can't even remember where it came from.
But years ago, some wonderful person on the Internet used to maintain a Google spreadsheet
of all of the DNS hosting providers, commercial and otherwise, and all these different attributes
of what they supported. And so I went to that spreadsheet and I filtered down to all the
ones that had the things that I wanted. And there were three or four left. And none of
them were ones I would actually have wanted to use. Most of them because the minimum cost
was too high. They were designed for large enterprises that are doing this sort of thing
all the time. So I thought, all right, fine. I will consider another alternative.
All right. What do you need? So for those of you who have been around on the Internet
for a much longer time than a lot of the people who I see in the room who aren't old enough
for this, if you ever had to fill out the actual paperwork to register a domain name
with network solutions and fax it to them, you will know what I'm talking about. For
those of you who went way over your head, don't worry about it. So it used to be a
long time ago that this was a really complicated thing to do. Today, it's not such a complicated
thing to do. So these are the really simple things you need. You need some place to host
the service that's going to be the source of truth for all of your zones. What do they
contain? Obviously, you can choose to put that in lots of different places. I choose
to put it on a private network that's not reachable from the Internet because it's better
and more secure that way. You need a few places on the Internet where you can make your zones
available to the Internet. Preferably, those are not all in the same machine or all on
the same network or all on the same data center or preferably not even all on the same continent,
really. It would be best if they're geographically distributed, if they're not all going to go
down because you put them all in the same hosting provider and their control plane breaks
all the time, Microsoft, and their entire cloud infrastructure goes down all at once
around the world. So keep those things in mind when you decide where to put these things.
It's perfectly fine to put these on Raspberry Pi's and your friends' houses if that's something
you choose to do. If you have friends geographically distributed around the world who are willing
to do that for you, that's fine. But there are plenty, and trust you, but there are plenty
of other ways to do it as well. You will of course need DNS authoritative server software.
There's lots of those out there. There's a bunch of people in this room who are responsible
for Power DNS, so I'm thankful for giving me the chance to do this. And then, and this
is actually trickier than you would seem, your domain register, the place you host your
domains possibly, the also place you bought them from, although they don't have to be
the same, needs to let you have the tools to set this up. They need to be able to let
you specify your own name servers, not all of them will. And if you want to do DNS sec,
they have to let you upload DNS records for your signing keys for your zones. Fewer of
them will do that. Many of them that are especially the free ones do not give you the ability
to do that.
Is that really enough?
Yeah. That's really it. There's not a whole lot required to do this. Now, this is all
written from the point of view of me doing this personally. I would actually be happy
to use the same kind of structure if I was doing this for, let's say, a small on profit
organization or something like that. I would not go in this specific direction if I was
responsible for doing this for 10,000 domains for a bunch of people who are paying me to
do that. Things would have to be much more complicated in that situation. But now, going
back to the way things were decades and decades ago, network solutions time, back then we
did not have Google public DNS, cloud for our public DNS, quad nine public DNS, massive
ISPs with millions and millions or billions of subscribers that ran resolvers on behalf
of their clients. So back then, if you wanted to host your own zones, your own authoritative
servers, pretty much every like small chunk of the internet was going to, if it needed
to know the answer for one of your names, it was going to be reaching out to your servers
to get it because there was no caching layer in between them and you. That's no longer
true. I mean, obviously, lots of us also run our own resolvers. And so we are going to
be part of the noise that's hitting your authoritative servers. But if you have a website that millions
and millions of people on mobile phones are going to touch, those mobile carriers resolvers
are going to go ask your servers for the answers and then hold on to them for however long
you've told them they're allowed to hold on to them a day or two days or a week or whatever
is appropriate. So you don't really need big, beefy servers to do this. Really, really
tiny little machines are perfectly fine for doing this sort of thing. Also, even the smallest
machines you can get in clouds nowadays have multi gigahertz CPUs with very fast RAM. And
as I think Stefan pointed out in his last talk, a normal, non-dynamic authoritative server
is very fast. It's very predictable. It doesn't take a lot of time to come up with an answer.
Probably all of the data it needs is in memory already anyway. So these are not, they don't
have to be really super powerful machines.
Why did you do this?
Yeah. So I covered some of that already. Actually, I kind of did that backwards, didn't I? I
talked about that before I was supposed to. Darn it. So a couple of the things that I
ran into when I was doing the search was there was quite a few of the available ones. If
you just wanted to host one or two zones, they were okay. The prices were reasonable,
actually considered doing it. At the time, I think I had seven. Now, I've probably got
11 or 12 for various different purposes because we all just grab domain names for projects
and then never use them for anything, but you keep them around. So I thought there was
a thread on the Fediverse about that. People ran, it's one round of poll. How many domain
names do you own that you've never used? The number was high.
And so there were some where, you know, if you went above three, like it was $100 US dollars
a month for the service to be able to do this. There were others who, shockingly, charged
by query volume and not by the number of zones. It's like, well, I can't, I don't have to
control over that. I mean, if someone starts hammering the servers, not paying attention
to the TTLs, I don't want my bill to suddenly be $14,000 for this. This is crazy. So fine,
we are not going to do that. All right, ready.
What do you use today?
Well, what do I use today? So, like probably a lot of people at POSDEM, I have at least
one reasonably beefy computer at home, network storage appliance. So there's a container
on that where I run the hidden primary authoritative server. Many people who, there was lots of
people who will choose not to put things in Amazon Web Services. I'm not going to complain
one way or the other, choose whatever you like. I will say that their ARM-based nano
virtual machines are the cheapest machines I could find that were not VPSs, that I could
actually install my own software on and open up my own ports and make them available to
the internet. There are lots and lots and lots of cloud providers out there, obviously,
and if you can do this with a, you know, one CPU, Graviton 2 nano instance, you can certainly
do it with a Raspberry Pi or some other small computer like that. If you weren't trying
to use PowerDNS, you could probably do it with an ESP32, although I don't think I want
to try that, but you could probably do it. So, I have three, I have two of those and
then I have a dedicated server in a data center which I use for off-site backups and running
my mastodon server and matrix server and all that kind of stuff. I put another one there
just because. And it gives me good distribution. One's on the west coast of the US, one's
on a roughly east coast of North America, it's not in the US and Canada, and then one
of them is in Europe, so that works out fairly well. So, and then in our home network, because
my home network is wildly over-engineered, I have two PC engines, APUs, which are our
border devices that connect us to the internet and handle all sorts of things. They're the
ones that run the resolvers, the recursive resolvers my wife learned about when she
saw the slides. And then, of course, they also have a copy, they also have authoritative
servers sitting right next to the resolvers for a reason that I will talk about in a minute.
And now, you don't have to read this because we're on the same topic. So, I use PowerDNS
auth server on all of them. I use SQLite databases because they're really simple and
I want to try to keep all of the machines except the primary as stateless as possible
and simple to manage and deploy. That also means that I use DNS-based distribution of
the data. So, it's not like a shared mass-equal database or any of that sort of much more
complicated thing. But, again, since this is why I went back to before, I would not
do this if I was doing this for 10,000 zones for paying customers. SQLite 3 would probably
not be the best choice. Well, actually, SQLite 3 probably would be okay. AXR would not be
okay in that situation because you'd be doing this constantly pushing out zone updates.
And then, because I use Ansible to manage all of my infrastructure and since there was
no actual good Ansible modules to poke at the PowerDNS API to create zones and manage
them and set up TCIC keys, I wrote some, which there's a link to at the end of the presentation.
What is that cost? Oh, I thought you were asking for the Ansible modules. Yeah, good
job. Good job. Excellent. So, what is all this cost? So, you already got the NAS, putting
another container and it's free, right? Especially if something is lightweight as a primary auth
service server, almost no resources whatsoever. I translated all of the costs I'm paying
AWS into Euro for you. You can see that it cost me less than four Euro a month to host
those machines. Now, that includes paying for them for up front for three years. So, because
of the AWS basically cuts the price in half if you do that as opposed to paying for it
on a monthly basis. But, it's 78 Euro for three years. If you decide two years in that
you don't want those machines located there anymore and you're basically throwing away
20 Euro of up front payment, that's not the end of the world. We're not talking about
a gigantic amount of money here. And then, because I have the rented server and the other
data center, putting a server on that was free as well. The software, of course, is
all free and the modules I wrote are also free. So, cost is very low. The half of the
cost per month is actually the storage cost from AWS for the root volume for the VMs.
The actual cost of the VM is less than the storage cost if you do it this way. So, and
they won't let you create a root VM, a root volume that's smaller than 8 gigabytes even
if you don't need that much. That's the small, well, I'm using the Debian AMI and it has
a hard-coded minimum of at least 8 gigabytes for the root volume. So, I suppose I could
make my own and try to cut that down by another few pennies or something like that. So, what
do I do with that? So, as I mentioned before briefly, all of the network infrastructure
I manage uses real up-to-end group certificates for all a browser accessible and actually
some non-browser accessible endpoints. I had been using self-signed certificates previously.
For those of you who've done that, you know how painful that can be having to make
sure that everything has the right CA certificate to trust them and everything has it. I don't
want to do that anymore. So, much easier to use. Let's encrypt for that. Similar thing
for SSH. All of this 25 different SSHable endpoints across this infrastructure. Managing
root key rotation for them was a pain because then things that had copies of the public
half of the key all had to be updated and everything else. Using SSHFP solves that problem
entirely except that it requires DNSSEC. Well, I mean, open SSH requires that it be a signed
answer for fairly good reasons. So, I'm not going to try to work around that. And then
something I didn't mention before is doing this not only gives you access to be able
to host all of the, you know, standards track RFC DNS records, you probably get to do lots
of cool things that aren't actually standards track RFCs yet. So, for example, HTTPS records
which are still just a draft and we don't even know if they're really going to end up
being approved or not. It probably will. And Firefox and Chrome already know how to use
them. I host all of those already for my HTTPS services because why not? It just took almost
no effort to set it up. I do Ansible based management for all of this stuff. And then
I'm sure that most of the open source auth servers out there can do online signing, but
Power DNS auth server certainly can. So, I don't have to worry about, you know, regularly
re-signing the zones and handling all the key to speech and everything else because it
does all that for me. So, the last thing that's mentioned there, which I'll get to just a
little bit, but I have it on purpose. How many of you know what catalog zones are? Wow.
That's shocking because it's you. You wrote the code. Of course, you know what it is.
So, catalog zones made this whole thing much simpler. In fact, we were joking dinner last
night that there's probably would have had to be three slides in the section about how
to do maintenance of all of this that don't have to exist anymore because catalog zones
take care of the whole thing for you. So, I just led right into this, didn't I? So,
obviously, new software releases have to be deployed. For the moment, although I know
Peter's working on it, there aren't currently ARM64, AR64 packages in the Power DNS repos.
So, I use their builder scripts to just build them myself and make publish my own packages
to my machines to install them. Eventually, I won't have to do that first part and I'll
just run app to get update and it'll install the new packages. And then whenever I need
to add or remove zones, I go to my Ansible Playbooks and I change the list of zones that
I want to maintain, go run them and they poke at the API to do the right things. First of
all, make sure those zones now exist or no longer exist on the hidden primary server.
And in the past, they also had to reach out to all of the other servers to make the corresponding
change. So, every secondary had to know that you've added a new zone so that it could know
a lot of the data. Catalog zones take care of all of that for me. So, now when I stand
up a new secondary server, I only have to tell it which catalog zones it's supposed
to pay attention to and which server those are supposed to come from and then it automatically
populates its secondary zone list all by itself and it's automatically updated every
time I add or remove zones. It takes, I don't know, a minute maybe for all of them to be
updated and everything is happy. It's really fantastic. So, there's going to be more cool
stuff we're going to do with that in the future. So, step zero, one, two, three, four, put
them all together. That first one is the most important thing. Go find out what your domain
registrar supports. If they don't give you the tools you need to be able to do this,
you're going to have to move your domains to a different registrar or give up. Those
are obviously two choices, equally valid. So, but there are lots and lots and lots of really
good ones out there, different parts of the world. So, it's easy enough to find out. There's
a very short list of things you need to be able to do. The biggest things are the DS
record is probably going to be the most, the first thing to check. Will they let you upload
DS records for your own zones? If they won't, you're dead. You can't do DNS sec with your
zones and that registrar. You have to switch to somebody else. The one I currently use
supports both IPv4 and IPv6 glue records, but only IPv4 through the web interface. IPv6
has to be done via support ticket, which is annoying, but it doesn't change very often,
so I'm willing to live with that pain. So, decide which server-server software you want
to use. Obviously, that's going to be an important decision for you. Decide where you want to
put all the stuff, where you want it to live, both the hidden primary and all the secondaries.
And then, how are you going to manage the zone list? And as you can see there, if your
answer has happened to follow that category, you can do exactly what I did and be on your
way. Now, a little bit of bonus here. I did write that as a question. I forgot I changed
it. So, on those network appliances in our home network, I run the recursive resolver
for all of the clients on the home network to use to resolve names. And with our last
ISP, we had more outages than I would have liked to have. And once you start setting
up something like this, you forget the IP addresses for all of your own infrastructure.
You only know the names. And when you've made the zone signed and your resolvers can't
reach the internet, they can't resolve any of the names because they can't, in my case,
my domain is km6g.us, so they have to know what's signed at the root, what's signed
at the US, what's signed and all that stuff, right? So, we would have very bizarre cases
where things inside the network stopped working because our internet link was not working
correctly even though the things we were trying to use didn't use the internet at all. They
just couldn't talk to each other because they couldn't resolve names. So, now I have
the resolvers sitting right with an authoritative server sitting right next to them that hosts
all of our zones that nothing talks to except the resolver so that even if that box can't
talk to the internet, it can still resolve any of our internal names with no problems
at all. An additional thing there, thank you again to the...
What's that? I am, yeah, yeah, so what happens here is a feature in the Power.DS recurser
which I will say unabashedly, I wrote, which is that you can actually tell the auth server
to set notifies to their recursers, which is not something you would think would have
ever, I mean it's not something that's normal. It's really cool though because what we do
is when that happens, the recurser says, aha, that's a notification from the authoritative
server that any content I might have in my cache for that zone is probably stale, throw
it all away. So that means if I've got internal infrastructure changes, I've moved the container
to a different machine and its IP address has changed, within a minute everything that
needs to use that will get the correct address as long as it's not running a local cache
on the box itself. So, and then as I said, over engineering I used anycast and OSPF and
all kinds of other stuff to reach the recurser resolver. So, there's a bunch of links, the
slides all are up on the FOSDM website, so feel free to download them. All of these links
are useful and you can see even the HTTPS thing which is not, not even a stent, what's
that? Oh, you're right, I forgot to put that link in there. Tell me where it is and I'll
add it. Yeah, so, and that's that, I have two and a half minutes left. So, so questions,
yes sir? You said you wanted to be required to add, change DS records. For anyone else
want to do, to do this, please note that some TLDs require you to send up the DNS key record
and they will hash it to DS. Ah, okay. So, he just missed me a comment that depending
on which TLD your zones are under, you may have to send a different thing for them depending
on exactly what they want from you. So, that's fine, thank you. Don't fight, yes, that one.
Just another comment, some TLDs are scanning for CDS, CDNSP and CDNs. Yes. So, you don't
have to play with the registrar at all. Right. Emerging technology that's been around for
a really long time. Yeah, so I just, so I had just a repeat that there are some top level
domains not related to the registrar that will actually notice the correct type of records
in your zone and pick up the keys from there so that you don't even have to manually update
them when you change them or rotate them. Yes. Yes. Okay. Robust infrastructure but
you still use only one DNS software. That's a good point. I have diversity and robust
infrastructure but no diversity in the software. That's absolutely a good point. I'm okay
with that risk though. Yes, sir. So, how do I ensure that I'm not under attack on the
virtual machines and how do I apply CV updates? So, because of I'm a geek every Saturday morning
before my wife gets up I run this gigantic Ansible playbook that goes and touches everything
and applies all package updates and reboots and it needs to be rebooted and all of that
sort of thing. So, I'm reasonably good there. Plus, on the public facing machines I have
the Debian unattended upgrade thing in place so that if a really important thing gets a
package update shut comes in and gets applied it will do it for itself. So, I don't have
to restart that. What was the other question? What was the other part of that one? Oh, the
attack thing. So, this is really cool. Those particular AWS VMs have a hard cap on the
amount of CPU that they're allowed to use. So, if somebody tries to do a DOS attack
on them they just stop getting responses. So, I don't have to do anything. It's just
not granted that resolution of my zones would be harmed by that. But, these are personal
zones. We're not hitting them all that hard from outside. So, that's kind of a neat feature
that it's just a side effect of the way those particular VMs work is you're limited in how
much CPU you can use. So, yep. And, are we out?
Out of time, yeah.
All right. Thank you, Captain.
Thank you. Thank you all.