from RUTWIC about demystifying, demystifying Stackrocks. Welcome RUTWIC.
Thank you. Good evening everyone. Thanks for showing back for the late evening talk.
I appreciate your time. So my name is RUTWIC Shiv Sagar. I work at
that as a senior technical support engineer. I mainly work on solving open shift as well as
Stackrocks issues with the customer and with engineering teams. So with the recent time and
security threats or attacks, we have seen that container and Kubernetes adoption has equally
increased. With that, security has become a biggest concern, right? So we'll see how
Stackrocks is paving the way for Kubernetes native security and helping us to achieve or resolve
the security issues with ease and automation. So this is the brief agenda for today's talk.
So in the first few slides, we'll discuss current state of Kubernetes security,
what are the best practices and how DevSecOps approach benefits into the
security posture, you know, to shift the security for your developers as well as your security
admins. And then we will see how Stackrocks ecosystem is helping end users, developers,
as well as your security teams to overcome the security issues with ease. And then we will
have some demo at then. Yeah. So first of all, let's understand what is zero-dose security,
why we require zero-dose security, right? So zero-dose security is basically a framework
which requires all the users to be authenticated and authorized continuously before they've been
granted an access to your application and data. So if you manage to achieve zero-dose security
model, then I would say that we could resolve or minimize the impact at the very early stage of
your application lifecycle. Then how exactly zero-dose security fits into the software supply
chain. So what exactly is software supply chain? It includes everything, everyone and everything
that touches your application code into entire software development lifecycle, right? It could
be your deployment, it could be your final artifact, it could be a CI CD pipeline. So it's
essential that we build the application in such a way that the assurance at every stage of
application is being taken seriously. That way we could achieve the trust rates of software
supply chain. So yeah, we can see that security dependencies, securing code, securing containers
as well as the infrastructure are all part of the software supply chain.
Let me ask you this question. If you're using Kubernetes or in general any applications,
have you ever delayed or slowed down the application deployment into production due
to container security concerns? Anyone of you? All right, I assume so because that's how we go
through application lifecycle. We deploy the application, then we analyze the application,
behavior and we detect the vulnerabilities. So in the recent trend, we have seen some common factors
or common anti-patterns which were causing delays for an application to get deployed on the production.
So misconfiguration has topped the percentage where following to that we have vulnerabilities
to remediate, right? So for example, we kind of able to detect the vulnerabilities but somehow
we tend to overlook them or we could not assist them accurately. That leads to the vulnerability.
I mean, we get to know that okay, vulnerability exists but there are no proper ways or tooling
to fix that kind of vulnerability. Then we ultimately have security issues at the runtime,
you know, which could be costly or it could affect your entire production.
So how can we make sure that these kind of issues are reduced? Let's say,
so in today's world, we need DevSecOps approach, DevOps, just DevOps isn't enough, right? We need
DevSecOps to shift the security from our traditional security practices. So DevSecOps helps us define
microservices architecture. It provides us declared definition to, you know, harden your
security parameters, network policies and deployment conflicts. It also makes sure that
the infrastructure stays immutable. So at the runtime, nobody else is allowed to, you know,
touch the software or your application deployment. At the same time, it is important that we know
Kubernetes native security is increasingly critical and securing supply chain is also
equally essential. So what are the basic Kubernetes security scan challenges? So we know
that containers are numerous and everywhere. If I have to put analogy, like we say that
everything is a file in Linux in a similar way, everything runs in a container when we
talk about Kubernetes, right? So they may tend to pose compliance challenges. Every container
image is tied up with some of the other container registers, right? So sometimes we even forget to
add TLS with best authentication to our image registry that may, you know, expose
security threats over the internet if at all we expose that. And we are also aware that
containers by default talk to each other without any network policies. So it is important that we
define network policies at early stages. And this one, I think most of you can relate that
when we show Kubernetes, all of the configurations looks pretty easy, but defaults are usually
the less secure, right? So we as an admin or developer have to proactively understand what
configuration or what risk tolerance required for my organization or developer environment.
So in Kubernetes, application lifecycle span across three phases mainly, that is build phase,
deployment phase, and runtime phase. So how we can make sure that we secure each and every stage
of the application, right? So when we talk about build phase, it's important that we isolate the
vulnerability of security issue at the earliest. Otherwise, it would be very costly and risky to
detect the vulnerabilities at the runtime, right? So what we can do, we can use minimal base images
so that we can avoid unnecessary package managers or, you know, any executable programs into your
container images. Then we can always use image scanner to identify known vulnerabilities. I think
identifying vulnerabilities just once is not enough. You need to make sure that whatever
security integration scanner you're using that will continuously validate your container images and
send the real-time alerts to your development team as well as security admins. Then yes, at the
build phase, we need to integrate CI CD pipeline. So that way, most of the things becomes automated
and you don't have to look around each and every build config to understand where the security issue
lies through CI CD pipeline. If the stage gets filled, your production won't be affected and
build would be stopped over there. Then at the deployment phase, as I mentioned, the default
deployment config doesn't come with network policy. We need to understand what services that
deployment is trying to communicate, what are the ports that are defining the deployment config.
And accordingly, we can define our own network policies. Then we also need to make sure that
the deployment doesn't allow root-level privileges or any unknown users, you know,
user IDs to access your application. You should be always aware of what users are
going to access your application. And then yes, we can extend the image scanning to
deployment phase. So it's important that we do not restrict our image scanning at the build phase,
but we continue doing that at the deployment phase as well. Then runtime phase, as I mentioned,
we need to extend our scanning at the runtime as well. So we can easily understand and quickly
understand what issues have appeared and what actions I need to take. It also helps
monitoring network traffic to limit unnecessary or insecure communications. Then if you find any
suspicious activity, and if we, at the same time, if we have multiple replicas of your application,
then we can compare all the replicas and processes in time to understand what anonymous
activity is happening. So to overcome all the challenges, we see Sycrox is helping the end
users and the community as well. So why Sycrox is open source, right? Red Hat believes open model
when it comes to your software or developing the application. And we believe that open source
software can significantly help developers to drive the project with innovation as well as foster
the collaboration within community. So Sycrox is working towards providing the open source solution
which will allow end users to decide how they want to protect their Kubernetes clusters.
So let's understand what Sycrox has to offer us. It enable users to address all significant
secretive cases across entire application lifecycle that we discussed, right? Right from
your build deployment and runtime. It also gives you greater visibility over vulnerability management,
configuration management, network segmentation, compliance, threat detection, incident response,
and risk profiling and tolerance. So Sycrox has a policy engine that allows user to run the policies
out of the box, meaning that let's say if I have severity with CVSS score greater than or equal to
seven, then I could have alert for the same CVSS score and understand what deployments are associated
with it. Then Sycrox API allows user to integrate with the image scanning tools, CICD tools,
container and times of their own choice, secret management, DevOps notification,
to ease and security flow end to end. You can also run it on any cloud or hybrid cloud,
or if you want to choose on prem, you can deploy it over there.
So this is the bird eye view architecture. I would say where you would see a central
in the blue box as a central hub, which gets exposed over load balancer for the
clients to consume the Sycrox API. It is written in the REST API. And then we have sensor,
admission controller collector, which is logically grouped and called as a secure cluster, right?
So you can, once you configure this set of components, you can call a Kubernetes cluster
as a secure cluster. And then you can keep on adding as many Kubernetes cluster as you want
into the secure central. Then central also has scanner, which aggregates the vulnerability
feeds that are fetched from the central. So central basically collect vulnerabilities
feed from upstream sources as well as NVD database. Then on each and every node,
we would have collector agent, which will collect host level data for the container network and
the runtime. So this is the UI where let's say if I have integrated 100 Kubernetes clusters,
then how can I manage or understand how those are behaving? What are the healthy components
and what are unhealthy? So we can have a quick look to see how systems are performing.
So what problem segments Stacrox is going to solve? So these are four problem
segments, which I found very common between developers and security teams to understand
whether my container contains content-compromising infrastructure, are there any known vulnerabilities,
are there any runtime and OS layers, container up-to-date, is my Kubernetes cluster compliant
with industry-certified security benchmarks? So let's see how Stacrox solved these problems.
So Stacrox can identify the vulnerabilities in the base image package that are installed by the
package managers, then programming language-specific dependencies, programming runtime and frameworks.
It supports package formats, which I have mentioned there. And I believe most of you
work with the same package formats. And there are supported operating systems like Alpine,
Amazon, CentOS, Red Hat, Enterprise Linux. So managing compliance is equally important for
our organizations to the security standards. So it supports out-of-the-box compliance standards
like CIS, benchmark for Kubernetes is occur, then HIPAA, NIST, PCI. So you can run scans
through this profile. So Centel or Stacrox specifically collect snapshots of your Kubernetes
cluster, then it aggregates the data and analyzes what checks are being passed and what checks are
getting filled. It will help to evaluate for the regulatory compliance. It will help to harden
your Docker and underlying container runtime. So this is the UI where you can see passing
percentage across your cluster, across namespaces, across the nodes. So you can have a better idea
where the issue or what compliance checks are filling. Accordingly, you can navigate to that.
In the right section, here you will see what controls are filling, what needs to be set.
For example, here I have taken an example of CNF files, which says that the file permission
should be more restricted. You can accordingly take the actions and fix that control.
So what is Collector? Collector overall helps all the Stacrox ecosystem to maintain and manage the
container runtime activities as well as post-level processes information. So it's an agent that
runs on every node, under strict performance limitations and gather data via either kernel
module or a BPF probes. It collects, it analyzes and monitor content activity on cluster nodes.
It collects information about runtime and network activity and sends collected data to the sensor.
Sensor then will help central to display all the data over the UI.
Okay, we'll quickly see. This is the traditional way of how we used to see at kernel when we
when we deployed the application. We have user space where application, user application runs
and for every resource that we need into user application, we need system calls. So
then user request any data, the kernel copies that information from kernel space to user space.
But due to some limitations, it is not possible for user to access
everything that is into the kernel space, right? And this was not a problem when we talk about
a single Linux source, but with container adoption, we know that the number of processes or container
that may run on a Linux source have increased, the density of container have increased, right?
So resource overhead, managing container issues, container runtime issues has become a great
challenge. So all these required activities require kernel support that we know. So how,
how do we overcome that? We can use EPPF rules. What is that? It is an extended Berkeley packet
filter, right? It is not just a packet filter. It is more than that. It helps us in networking,
tracing, profiling, observability and monitoring and security. I will quickly go ahead because
of time constraint. Then we have network policies. In Kubernetes, we know that by default,
network is, network policies are not there. We need to define our network policies by our own.
But considering a production grade environment, it is really difficult to, you know, write
each and every network policy ML because sometimes we do not understand what source,
from what source the traffic is coming. At large scale, it could be a difficult, right?
So it provides network graph, network segment, segmentation to understand or to modify baselines
so that we can define, okay, if traffic is coming from this source, then this should be blocked
or network policy should be created accordingly with this baseline.
So this provide is, so yeah, we, Cyclox provide a network simulator, network policy simulator
through which you can understand what are the active connection from where the connection is
coming, whether it is allowed by the deployment or whether it is anonymous. Accordingly, you can
define your baseline and restrict the traffic. It will help us to create the network policies at
the runtime. So we can just copy that network policy and configure it in our Kubernetes lecture.
Then we have admission controllers. So it basically helps control, to enforce the security
policies before Kubernetes creates workload. For example, deployment, demo sets, it intercepts
the API request when any program runs or application runs into the pod. So in Cyclox,
we use admission controller with security policies so that any policy gets violated,
then it will immediately prevent the deployment from getting into running straight.
Okay, so I will quickly show a demo where I have given an example of log forces,
forces CV and to understand how it can prevent the deployment. Just let me show it quickly.
I hope screen is visible, yeah. So this is the cluster dashboard,
where I can see images at the most risk, what are the policies, current policies violated.
So Cyclox provides some default policies as per the best practices pertaining to the security
posture. So considering the criticality of the log forces, we have included this policy as well.
So you can configure policies into two modes, inform as well as enforce.
So currently, if I look at this policy, it is into inform mode only. So I have edited it and
make it enforced.
Yeah, so it executes on build stage, deployment stage.
I marked inform and enforce and enable it for the deployment phase.
Right, so once the policy created, it will show whether any existing deployments are
violating this policy or not. Then for the demonstration purpose, I have run a vulnerable
deployment which has this log forces CVT. So this container image has the vulnerable app.
So in the parallel terminal, I have keep a watch to trace the events in the run time.
So as soon as I create this deployment, you will see that the parts are getting terminated
because of the policy violation. So it won't allow the part to get into a running state
because of policy violation. And in the events, you will see that
stack rock enforcement has been detected and the deployment has been scaled to zero.
Okay, time is up. I have one more demo, quick demo. If you would like to see, let me know.
Quick demo, yeah, that would be interesting. So in this demo, I have explained how we can
leverage the DevSecOps approach to shift the security. For that, I have used Tekton in the
pipeline operator, which is deployed in an open shift. So this operator is nothing but
using Tekton framework under the hood. Let's see it quickly. So it provides a standard CICD
pipeline definition in a declarative approach. So we can define the task as well as pipeline,
which further than can be portable across all your Kubernetes infrastructure.
So I have defined these three tasks where images, image will be checked and scanned.
And in the task, it's in the background, it is calling a stack rocks API through rock CTL.
It's same as keep CTL. It talks with the stack rocks API and performs the scanning for the image.
So these two tasks I have mentioned in the pipeline definition, image check and image scan.
And there is one more secret where I have provided stack rocks API endpoint and the
credentials. So we'll create a name space called pipeline demo.
Then I have created secret as well as the pipeline definition. Next, we will
execute those tasks. We should develop more and
see that pipeline has been defined. So these two tasks are there. Pipeline run is not initiated
yet. So we'll initiate the pipeline run. We'll pass the container image that we want to scan.
For example, here I have provided MySQL 80. So pipeline has been created. You can check the
logs, real-time logs through Tecton. It's a client for Tecton through which you can perform the
operations. So it also gives you better visibility if in case your tasks are failing. For example,
here my credentials were expired. So I had to refresh the credentials and then I ran the pipeline.
Now we will see the pipeline gets into running set.
The tasks has been passed. Now we will see all the CVs that are associated with this
particular container images. You can get each and every CV ID, its CVS score, and you can
accordingly share those security admin. You can also check policy violation through image check
tasks to understand what policies have been violated, what are their ratings, whether those
are rated as low or moderate or risky. That is it. So I have put some handful resources
for you to go ahead and get started with the StackFox community project. You can also hop
into our Slack channel and that is it from my side. So do we have some questions here?
Thanks for the excellent presentation. I have one question regarding you mentioned a lot about
the agent which is kind of scanning and detecting the vulnerabilities. You briefly touched upon the
object central, which I think if I understand correctly you are pushing that detection of
vulnerabilities into the central. Is that right? Yes. So central fetches the vulnerability
feeds from the upstream sources or let's say you have NVD database. So every five minutes it will
keep on checking what vulnerability are present in the upstream. So accordingly once you download
then the collector or the sensor fetches those data into your respective Kubernetes cluster.
So what if when the container is running, the pod is running and suddenly the agent checks the
vulnerability database and detects possibly that the version running in the pod has having some
critical vulnerability. What actions would it do actually? It actually depends on us what actions
we want the admission controller to perform. Either we can have it in inform mode so that we
understand okay policy is violated but that whether that is really affecting my workload or the
runtime accordingly we can take actions. If you want strictly not to allow any deployment to run
as soon as the policy is violated we can put it into enforced mode and we can decide at what stage
we want to terminate that at the build stage, deploy stage. It's basically based on your policy.
And the central is kind of accessible by is it like a closed environment or it is open where
anywhere anyone can access that. Any containers running in any cloud can access that. It can be
configured in online mode as well as air gap environment. So again it depends on your case
or your organizational requirements how you want to install it. In terms of offline mode you can
always download those vulnerability feeds or kernel probes modules in your secure host and
then you can inject those to center offline way. Okay. That option is also there. Thank you.
Any other questions? Yes. I just have a question. Can you use stack rocks as a
honeypot? I mean can you just let the intruder or the security thing to go to actually get a
like a description of all the things it's doing. The attacker instead. So let's say you not just
cut it because you just right now basically applying a policy you're cutting the thing. But
can you let it just isolate the container and let it run just to have 4 and 6 out of it. See how
things are behaving. Yeah other than policies we can always do the risk analysis. Sometimes it
happens that vulnerability that may found as a critical but in terms of my application I might
not have that vulnerable code at the runtime stage right. So I can always mark that vulnerability
as a false positive or I can defer that vulnerability. Does that answer your question or you have
something else? Yeah I mean as long as you can get I mean sometimes the scenario is that you
have the pod actually in production and something happens to it and you want to actually isolate
it but you still want to have 4 and 6. You don't want to just cut it. You just want to understand
the attack. So in terms of isolation it gives us a rich context from the UI at what layer the
vulnerability is present. For example we can inspect each and every Docker layer. It allows us to see
at what component the vulnerability exists. So you can always you know modify the image. You
can build it again and patch the changes. Thank you for the question and thank you for
the talk. I think we are out of the time. Thank you.