Hello everyone, welcome to this session about Cilium Service Mesh.
My name is Raymond De Jong, I'm Field CTO for ISOFALENT, the originators from Cilium.
Today I'm going to talk a bit about EBPF and Cilium as an introduction, after which
I'm going to talk about how the Service Mesh is evolving, after which we'll talk about
the Cilium Service Mesh features, what we can do today, and what we're planning to
support in the future.
Quick highlight of some upcoming features and some current features, and if we have time
I have a little demo to show how it works.
Can I see some hands from you if you know EBPF?
Quite a lot, good.
How many of you know Cilium?
Cool.
OK, how many of you use Cilium, actually?
Not as much.
OK, cool.
So, for the ones who don't know what EBPF is, I'm going to do an introduction here,
is EBPF is standing for Extended Berkeley Packet Filter, by itself that doesn't mean
a lot, but what we like to compare it with is what JavaScript is to the browser, EBPF
is to the kernel.
What that means is that using EBPF we can attach programs to kernel events, and for
the purpose of this session is that we can attach EBPF programs to kernel events related
to networking, so that's either a socket being opened, a network packet being sent on a network
device, that means that's a kernel event, and that means that we can attach a program
to it and we can get a metrics from that packet, for example, or we can do load balancing
and such.
So Cilium is built on EBPF, you don't need to be a network, or sorry, you don't need
to be an EBPF developer to actually work with Cilium.
Cilium abstracts this complexity on technology under the hood, so based on the configuration
you set, Cilium will mount the required EBPF programs for you to run, and Cilium in short
provides networking and load balancing capabilities, security capabilities, and also a lot of observability
capabilities using EBPF.
So this is the 30,000 feet view where we are today, we started with plain networking,
IPv6, IPv4 years ago, and now we expand that all the networking capabilities with BGP
implementations, Netfor6, 64, extended load balancing out of the box we're working on,
having GOBGP control playing fully supported with Cilium.
On top of that we have an observability layer with our Hubble technology, which is a observability
tool which provides service-to-service communication for your namespaces, so you can see what components,
what services are talking to which services, after which you can make informed decisions,
for example what kind of network policies you want to apply, also exporting metrics
to tools like Rafaana, and service mesh on top of that to provide authentication, layer
7 path-based routing and such.
On the right hand side we also have Kettergon, that's not something we'll talk about today,
but that's runtime security using EBPF, which is also very interesting, and rerun across
clouds, doesn't matter if it's on-prem or hybrid or multi-cluster, so it's agnostic
of the platform, and supported by multiple cloud vendors.
So as you may know, Google and false data plane V2 under the hood is actually Cilium.
Microsoft has recently adopted Cilium as the default CNI for AKS clusters, and all their
clusters will be migrated to Cilium, and AWS, EKS, anywhere by default is Cilium, so we
see huge adoption in the field of Cilium.
So let's talk about service mesh, so obviously if we talk about service mesh we talk about
observing traffic, being able to secure traffic from application to application across clusters,
doing traffic management, building resilience across applications across clouds.
Service mesh originally, if you needed that capabilities, originally you would program
your application either in Python or Go to get that observability.
That wasn't really useful because you have to maintain all those libraries to get the
information you need.
That's where the sidecars came in, right, so that they abstract that complexity from
the application to have a standard sidecar implementation to monitor traffic, to be able
to route traffic, and to be able to extract metrics from that traffic.
However, now with Cilium our goal is to move as close to the kernel as we already run in
kernel with EBPF, so we're moving from a sidecar model to the kernel, and where we
can we will support it using EBPF.
The only part which is not yet there is Layer 7, so all the low balancing capabilities,
routing capabilities in terms of IP to IP metrics are already available with Cilium using EBPF.
Layer 7 routing is not yet in EBPF for multiple reasons, of which one is that EBPF has constraints
in terms of how big a program can be, obviously it runs in kernel space, so it has constraints
for a good reason, but in the future maybe we can even transport complex Layer 7 routing
in EBPF.
However, we already provide Layer 7 visibility and observability in using Cilium and EBPF,
we already have the capabilities to inspect traffic using EBPF.
We can already do the low balancing with the creep process replacement.
The only part is the Layer 7, but the visibility of traffic, so HTTP traffic and such is
already there.
So surface mesh capabilities are extending those capabilities moving forward.
So how does it work?
So some of you may know that Cilium runs as an agent, as a demon set on the nodes, it
programs the nodes to be mounting the EBPF programs for the capabilities you need, and
we have an embedded Envoy running inside the Cilium agent.
This is a narrow down Envoy proxy in the agent for networking capabilities, and we
leverage this Envoy proxy on the node level to do surface mesh capabilities, so all the
things like the HTTP path routing and such.
So for each namespace you would create, and where you create, for example, an ingress resource
or a gateway resource, that means that a listener will be created through the Envoy for that
specific namespace for that specific workload.
And we leverage C groups and stuff to have separation as well for the security reasons
to not be able to have traffic across namespaces as such.
So what is different with Cilium's surface mesh compared to other surface mesh implementations?
First of all, our goal is to reduce operational complexity by removing sidecars, resource
usage, reduced, better performance, and avoid sidecar startup shutdown race conditions.
So obviously if you're not running sidecars at scale, this makes a huge difference.
You don't have all the sidecar pods running alongside your normal pods, that will save
memory, that will save CPU, that will save connection tracking, et cetera, et cetera.
So a lot more efficient.
And also in terms of latency, running a sidecar has a cost.
So in this diagram you see that an application wants to send traffic to another application.
What that means technically is that it goes through the TCP IP stack three times with
the sidecar.
First from the app, then inbound in the sidecar where the sidecar does its processing, and
then external from the sidecar to the physical network device to hit the network to reach
another node.
With EBPF we are able to shortcut that connection and improve the latency because we can detect
that traffic and we can see if it's destined for the physical network or it should be routed
to the proxy.
So once this app opens the socket using EBPF we can shortcut that connection to the physical
network device to be routed on the physical network.
If we need layer seven processing, that means that using EBPF we can shortcut the connection
on the socket layer directly to the envoy proxy where the envoy proxy on the node does
his HTTP routing and then forwards the traffic again on the physical network.
So a lot less hops there.
And it means that latency is much, much improved because we're not going through this TCP IP
stack multiple times.
In terms of throughput there's also a small difference because we can push more packets
and in terms of pod ready performance this is also a consideration at scale because when
you're scaling out your applications you always, with traditional sidecars, you need to wait
for the sidecar to be spun up as well and to be ready to serve connections for that
application.
So without the sidecars with Cilium service mess it's already there.
It's running on the node, it's embedded in the proxy so once you scale out your application
the proxy immediately on that node can serve connections.
So in short Cilium service mess provides traffic management, observability, security
and resilience.
The goal is to bring your own control plane or we are not developing a control plane on
our own.
What it means is that you can already use Ingress resources with Cilium 1.13 will support Gateway
API.
We are working on Spiffy integration so with the 1.13 release actually the ground work
for MTLS and Spiffy integration is already there.
You are not really able to use it yet but the goal is to support both MTLS and Spiffy
using Cilium network policies so you can reference for example a Spiffy ID as a source
and destination using Cilium network policies and then under the hood the Cilium agent part
will connect to spy reserver where that identity is tracked and confirm if that's allowed.
In terms of observability you can leverage the already available observability with Grafana
or Hubble if you need to export events you can use scene platforms such as Splunk and
open telemetry is also supported.
If you are new, if you are running new classes you have an option you can run Cilium and
you can already use Cilium service mesh out of the box this is obviously the preferred
method but if you are running already an Istio based implementation there is still a lot
of benefit to run Cilium under the hood there as well because for example we already encrypt
the connectivity between the sidecar from an Istio based implementation towards the destination
pod.
What I mean by that is that when you run sidecars, when you run MTLS between applications that
connectivity may be secure but the connection between the sidecar and the actual destination
is encrypted on the node so anyone with specific privileges on a node could potentially listen
on that virtual interface and e-drop traffic and that's obviously not secure.
The running Cilium under the hood already gives you the benefit because we can encrypt
on layer 4 directly on the socket layer to the destination pod obviously.
With 1.12 so currently we have 1.12 available since I think 7 months.
We already have a production ready Cilium service mesh, a conformant ingress controller
which you can use for HDD path routing, canary releases and such.
You can use Kubernetes as your service mesh control plane, fromisius metrics, open telemetry
is supported.
For power users we have Cilium Envoy Convict and Cilium cluster wide Envoy Convict CRDs
available.
These are temporarily I would say because the goal is to replace all that capabilities
with Gateway API.
And we're releasing more and more extended Grafana dashboards for layer 7 visibility
so you can actually see between service to service what kind of metrics there are and
what the latencies are and what return codes are, so golden signals.
So the roadmap for 1.13 and we're very close for releasing 1.13, expected somewhere this
month hopefully.
You can already try a release candidate for Cilium 1.13 which includes a Gateway API support
for HTTP routing, TLS termination, HTTP traffic splitting and waiting.
So this allows you to do percentage based routing or canary releases as such without
configuring Cilium Envoy Convict resources.
And also the capability to have multiple ingresses parallel balancer.
What that means is that currently when you create a Cilium ingress we rely on the hood
on a low balancer to attract traffic and forward that to the proxy.
Obviously at scale having a low balancer for each ingress, especially in clouds is expensive.
So this with an annotation we allow multiple ingresses per low balancer so you can save
cost there.
So how am I doing at the time?
Good features.
So today ingress 1.12, also with services we are having support for annotations.
So imagine you have received traffic from your ingress, you forward it to a service.
That means we support annotations on a simple cluster IP to forward traffic for example
to a specific endpoint.
If you know what Cilium cluster mesh is we can connect Cilium across clusters.
With simple annotations you can have even higher availability of services across clusters.
Gateway API which I will show a bit later and the Envoy Convict.
So this is a simple example of ingress and this is also something I will show in a demo.
You have an ingress and from a specific path you want to forward traffic to specific service.
We also support GRPC so you can also have specific GRPC URLs to be forwarded to specific
services.
TLS termination to terminate TLS using secrets, using ingress.
A question I get a lot is what about SSL pass-through, that's on roadmap so keep that in mind.
And obviously new in 1.13 is Gateway API and how it looks like is you will configure a
Gateway resource.
You specify the Gateway class name for Cilium to make sure that the Gateway is created and
maintained through Cilium and then create listeners.
So in this case an HTTP listener on port 80.
Then additionally you create multiple HTTP routes, one or more.
And this specify for example a path prefix for values forward slash details to be forwarded
to a backend reference service called details.
In terms of TLS termination, same constructs.
You can also have for example a host name in there to only accept traffic for this given
host name and you reference a secret in the Gateway resource and then in the HTTP routes
you will specify the host name, you will reference the Gateway you want to use and then again
a path prefix for example to forward to specific service.
And then traffic splitting, very simple, also using HTTP routes.
Again referencing your Gateway, a path prefix and then you have in this case an Echo 1 and
Echo 2 service where you want to introduce slowly Echo 2 and in this case 25% of that
traffic will be forwarded to the Echo 2 service.
And this is the example what I talked about earlier.
Using simple annotations you can extend service miscapabilities by annotating services.
So in this case this service will receive traffic for GRPC and we can attach low balancing
modes for in this case weighted least requests to be forwarded to backend endpoints.
And using multi cluster capabilities you can extend these capabilities across two or more
clusters depending on your cluster mesh configuration.
And canary roll out, so you can even introduce new clusters, have the new version of your
application running on the new cluster, so you're absolutely sure that you have no resource
contingent on your original cluster and then on the service annotate traffic to forward
slowly to remote cluster before you do the flip over.
So this concludes the presentation part, so for example when you want to know more about
Cilium go to the Cilium community, I encourage you to join our Slack channel if you have
any questions, our team is there as well to answer questions for in Slack, any issues
you may have or any roadmap or feature request you may have, we're very interested to hear
from you.
You can also contribute, so obviously if you want to develop on Cilium, join the Cilium
Github and contribute, if you want to know more about EBPF go to EBPF.io and if you
want to know more about Isovalent, the company who originated Cilium and want to for example
work there, have a look there, we are looking for engineers as well, so feel free to have
a look and if you want to know more, ask me after the session as well.
All right, let me do, see how I'm doing with time, so in order to run Ingress and Gateway
API, you need to set a certain amount of flags on your for example your hand value style,
so this is an example, I've run a small demo on GKE, so this is a GKE cluster with Cilium
installed, what you need to do is you need to enable the Ingress controller and in this
case I'm also enabling metrics just because it's interesting to see what's going on.
For Gateway API there's also a value, so Gateway API enabled through, this will trigger Gateway
API to be enabled, for service mesh it's important to configure the cube proxy replacement
to strict or probe, strict is recommended because you have the full cube proxy replacement
capabilities in your cluster, this is also required for service mesh and that's basically
it to get started.
So for this simple demo, I've created a simple gateway with the Gateway class named Cilium,
so this is running Cilium 1.13 Release Candidate 5 which has the Gateway API support and then
a simple HTTP route for the book info example application which has matches for the details
and the default path prefixes, so when I go into my environment, I want to show quickly
the following, so if I do a Qubectl getService, you can see I already for the sake of time
created this gateways, what I wanted to show you is that obviously a low balance is required,
so GKE provisions me a low balancer, low balancer IP I can use to attract traffic, in this case
I'm demoing a default HTTP gateway and a default HTTPS gateway, so I have two low balancers
with each an external IP address assigned, so this configuration is applied, so if I
do a Qubectl get a gateway, good point, obviously in your cluster you also need to install the
CRDs for Gateway API support, here you can see I have my Gateway and our TLS Gateway and
if I do a Qubectl get HTTP routes, I can see I have my book info HTTP route installed and
this relates to this part obviously, so with that I should be able to connect to the details,
so this is running the bookstore example, so I'm using that public IP as you can see
it works and if I go to details I should be forwarded using the Gateway API HTTP routes
to that specific details service and that works as well, for HTTPS again a simple example
I've created that gateway, TLS gateway, I've created two listeners, so a listener for bookinfo.cillium.rocks
and a listener for hipster shop.cillium.rocks, I didn't have installed the hipster shop for
demo purposes, I'm also referencing two secrets, so I've used makesert to create a simple self
signed certificate installed in my certificate store and created a secret which I reference
using this listener, then again HTTP routes for the TLS gateway for bookinfo.cillium.rocks
matches to only the details path prefix on port 9080 and again apart for the hipster
shop, so that's what I'm going to show here, so if I do the default URL that doesn't work
there's no list, there's no HTTP route configured, but for details I can see I can connect it
securely and this certificate is run from the gateway resource as well. Obviously this
is a self signed certificate, but obviously you can create signed certificates as well.
With that, that concludes my presentation and the demo, I'm open for questions. Any
questions?
Hi, thank you very much for your presentation. When you talk about no layer 7 support in
going to come or not? I'm not sure about that. HTTP routing requires quite a lot of memory,
so obviously memory is limited in eBPF programs for good reasons, so it will depend on the
eBPF foundation and the roadmap there, what we can support. Technically there's no reason
why we shouldn't be able to do that, but in terms of memory we have constraints, so if
those are being raised we potentially can have parts of even all parts using eBPF.
Any other questions?
Hi, does it provide or can you provide end to end encryption, especially between the
nodes automatically or not?
Yes, so our vision there is that you should configure, for example, IP stack or wire guard
for node to node encryption in transit, and if you want authentication and authorization
on top of that to configure SPIFI or MTLS between your applications. It's a multi-layered
approach, so we're not doing the encryption on the MTLS part, but on the node level, if
that makes sense. So MTLS again, SPIFI is on roadmap, hopefully for 1.13.
Any other? No, okay, thank you. Thank you.