Today is Scott Bryan.
He's going to talk about improving infrastructure security through access.
So, you're up.
Morning, everyone.
So, I recently joined Red Hat and I work full time on the Adoptium Temerin JDK project.
So, we use a very traditional build model with a large suite of machines.
We support between 12 and 15 different platform and architecture combinations.
So, it's very difficult to do just with dock containers, just single machines.
So, we have a massive, massive suite of infrastructure.
This doesn't work.
So, we're currently undertaking a massive piece of work to secure our supply chain.
So, we are looking at S-bombs, reproducible builds.
But underpinning it all is a good infrastructure security strategy.
And we've implemented centralized keys, rootless access, things of that nature.
But how do you know all of that stuff is working?
Unless you can visually see the results of all your security work.
It's very difficult to prove whether it's working.
So, I came in.
There's no security or no strategy for verifying that any security fixes have worked.
So, it's a very cut down presentation from the full length one.
So, first things first for us was identifying what we wanted to get out of an auditing system.
So, we want to capture, login any access attempts, anything at all where somebody was accessing a system,
particularly in the build sphere.
If you think about the Sol wins attack, which was a compromised Jenkins server, I believe.
If your build system infrastructure is compromised, your builds and source code are potentially compromised.
You build something, it's got a vulnerability in, but it checks and everything else looks valid.
So, any end user sees that.
The other thing we need, we wanted was an automated response and alerting.
So, should somebody try to log in as root on a build system, we need to be one.
That needs to be stopped straight away.
And we need to be alerted that that's a thing that's happened.
Come to it, why in a little while, the scale of the problem when you don't know about it is very different to when you do know the numbers involved.
So, and then we want some analytics and reporting so we can, again, gauge the program and the success thereof it.
Ultimately, for us, our infrastructure is all provided by a dozen different cloud providers.
It's all publicly accessible.
So, all of our, even our build infrastructure is open to the web.
You can request access to it when you join the projects.
So, again, the attack vector is significantly large.
We don't have a single firewall that we can use, sneak and restrict the IP addresses.
It's all publicly available.
So, for us, host-based intrusion detection using Wazoo, not a tool we build, but it's open source.
It's a very good tool for this use case.
I would recommend you do a very similar exercise, analyze your requirements, and then have a look into the tools that are available.
There's quite a few of them.
Wazoo itself is a fork of OSSEC, which kind of stopped development when it became semi-paidware.
Wazoo was an offshoot that is still open source, and they've continued to feature develop it.
So, the scale of the problem.
So, some numbers, which 24 hours across our infrastructure suite,
202, just slightly over 2 million attacks in 24 hours.
It's a bit of an eye-opener.
Of those, 12 are deemed by our, and the standard rule set from Wazoo is really excellent,
of being serious enough to warrant concern.
And you can see in 24 hours, about half a million people, people just brute-forcing the build machines to try and compromise them.
I think a demo is slightly impossible without my laptop, but you can drill down into all these.
You can see there are all the metrics that are available for the attack vectors, the CVEs,
and you see there are also the 79,000 authentication successes.
It's here on the right.
What's the difference between SSH and brute-forcing?
Not all machines are accessed by SSH, so they will be things like Windows,
brute-force, password attacks.
But Wazoo detects, again, remote services, modify registry attempts,
all via RPCs and things like that.
So, again, this is the, the first thing it does is give you a nice kind of visual view
of how big the scope of problems are.
It's why I like this tool quite so much.
So, drilling down a little bit just into the authentication failures,
you'll notice that Windows, by far, is the key attack platform compared to the Linux service.
The numbers are hundreds of thousands times as many.
And you'll see there, the top three machines are all build, Azure, Windows.
It's a very popular thing for attacking and, again,
get a much better kind of breakdown of the attack vectors.
People trying to access restricted accounts.
People trying to get valid accounts.
So, although they're disabled on ours, the standard Windows administrator,
the standard Windows guest accounts, although they're all disabled,
everybody can guess one of the Windows or can find out one of the Windows standard accounts
that, unless you've disabled it, is a very easy attack vector.
And then just brute forcing things.
And then, looking even deeper into just a single host.
You can see down here at the bottom of the screen,
you're getting the login failures, unknown user, a bad password.
In theory, it's somebody just typing an IP addressing wrong.
However, every single one of these attacks has been stopped with an automated response.
You can go even further into blocking IP ranges, geographic ranges,
so you don't even get the alerts.
It's that I like the visibility.
I would say only the really high priority stuff.
And you'll notice once you drill down, there are actually no serious alerts.
That proves it's working.
So, again, you can take some knowledge in that your infrastructure is fairly secure.
And then another really useful feature, and again, is you can then go into the details of each individual attack.
Although you get a geographic region name, IP address, things like the target users,
they've tried to brute force on our SSH-based host.
There isn't a slide for this.
We've extended it because Wazoo is eminently customizable.
So, we also capture the SHA-256 checksum of the SSH key being used to try and attack.
So, we can then determine if it's one of our valid users,
because we have all our keys stored centrally and distributed centrally via Vestillion.
If it's not one of our keys, we can then start blocking SSH keys at that level.
But, again, we've extended it to capture that information.
And Wazoo is basically an Elk stack-based tool system.
It uses the logging part of it, the elastic search,
and it just captures all the logs from all the systems.
Again, you can customize it to capture whatever you like, your Windows system registry,
whatever the Mac equivalent is, audit log, syslog, and it just harvests it all into one.
Really nice, easy to query, work with.
It's got the capability of doing dashboards and searches.
We're still fairly new to rolling out and leveraging it for real serious stuff,
but I think it's worth sharing even at this stage.
And again, more extended audit information.
This is from one of our dock hosts.
Somebody there has logged in as Root.
It's probably me.
But again, you can see the kind of information you capture even on successful logins.
If you're trying to find out who's doing stuff, they shouldn't.
And Wazoo itself goes much further.
It's got a file integrity management tool, which again, you can alert on,
so you can track all the changes to key system files.
It's got a SEA component, so it will check your system against the NIST databases,
look for any vulnerabilities, give you the links to the CVEs,
and then the potential fixes if that information's in the NIST databases.
All of that in one happy place.
Worth a look, and if you want some more information about how we use it,
feel free to connect with me on the adoptium slack after this meeting.
Whatever you need.
I think we've got like a minute left, so time for one question, maybe.
Say we're already using something like HashiCorp Vault.
There's that lagging behind in audit capability.
Let's say audit capability is something we want to elaborate on right and get ahead.
Does that even give us an advantage?
Is it doing everything in Vault or not?
What is the wisdom and that?
Okay, so that's the question is, compared to HashiCorp Vault, what does Wazoo give you?
I can't see any reason why you couldn't use both.
You could still use Vault for everything you're using Vault for,
but what this would give you is the reporting tool on top.
Would that work?
Yeah, yeah.
How much effort would go into it?
I've never used HashiCorp Vault, so I really...
But Wazoo, say you could get it to monitor your Vault,
as long as Vault's putting some logs out for you to monitor,
you could customize Wazoo to look at those logs,
as well as your system logs,
and still use the same visibility features and log harvesting.
I don't see why that wouldn't work, but...
So it's string matching based, right, as long as I have log performance?
At the base level, yes, it's string matching and regex from log files,
but that's just what it ships with by default.
You can extend it to do whatever you like pretty much.
If you're willing to write it.
OK.
Right, I think that's it.
Thank you very much.
APPLAUSE
Thank you.
Thank you.
APPLAUSE
Doctim is an Eclipse Foundation project for Temrin JDK.
Although I ran out of paying my wages,
I worked full-time on the adopting project.
So...
Wazoo is a third party to look into the Eclipse Foundation.
I just think it's...
Yep, sorry.
Sorry.
Well, cheers, George.
I'll catch up with you later, mate.
Wazoo, just a little bit.
I saw it was best for our needs.
OK.
And all good things about being a little bit independent
about working for the foundation.