All right. Good afternoon, everyone. Before I get started, how many of you attended the
session about service mesh this afternoon? Sorry about that. If you're fed up with me,
you have to stay with me for another session. I will repeat some parts of it to introduce
the topic. But otherwise, I'm sorry about that. So welcome to the session, Golden Signals
with Cillium and Grafana. My name is Raymond De Jong. I'm field CTO for EMEA at Isovalent.
Isovalent is the company which originated Cillium. And today I'm going to talk about
Grafana and how Cillium enables you to get golden signals out of the box. Introduction
about technology, a little bit about EBPF and Cillium, then about observability challenges
and how we can tackle those challenges using observability. A bit on monitoring data operations
and the default dashboards we provide. And then hopefully the demo gods are with me for
a small demo to actually see how we get layer seven metrics. And we can see return codes
and monitor application, response times, et cetera. So to start this topic, I want to
introduce Cillium and EBPF. How many of you know about EBPF? Quite a lot, awesome. How
many of you know about Cillium and are using Cillium? Cool. Great. So Isovalent is the
company behind each Cillium and EBPF. We maintain it with the community. And to start
with, EBPF is explaining what EBPF is and how it works. So we like to say what JavaScript
is to the browser, EBPF is to the kernel. What we mean by that is that we make it extensible
in a dynamic way. So that means we're not changing the kernel, but at kernel speed we
can run programs based on kernel events. And for the context of this session, what's important
is that considering metrics, considering getting useful information from your applications,
what we're doing here is whenever a process opens to Sokort or a packet gets sent on the
network device, on the node, we can expose metrics or we can export those metrics using
EBPF. And we can use tools like Rafaana to display it in a good way so you can make,
you can do data operations or you can see how your application or cluster is performing.
Cillium runs on EBPF. The good thing about Cillium is you don't need to be an EBPF engineer
to run an operate Cillium. You just set certain configuration options and EBPF programs will
be mounted on the nodes and will run when they need to run. And Cillium on the high
level provides a number of capabilities, networking capabilities, observability capabilities,
also service mesh, and using catricon solution we can also use runtime visibility and observability
security based on processes such as opening files, process execution, et cetera, et cetera.
Today we'll focus about the observability part. So besides networking we can get rich visibility
of metrics on your cluster. As you may know Google uses on data plane
V2 actually under the hood Cillium, Microsoft has moved Azure default AKS clusters to Cillium
so all the cloud providers see that Cillium is a powerful technology to run clusters at
scale and to get useful metrics running them. So let's start with common challenges we see.
One of the main challenges if we talk about performance of your application or performance
of your clusters at scale is that you run into this issue of the finger pointing problem.
What I mean by that is that network connectivity is layered and when you run into issues you
need to especially at scale you need to be easily aware where a possible problem could
exist and it can be in multiple layers and if you look at the OZ layer obviously it can
be at the data link layer, the network layer, the transport layer or in the application
layer. So the goal of Cillium with Kavana is to give you the tools to quickly inspect
what's going on and to be more efficient in troubleshooting your cluster and or applications.
Another common issue especially at scale is obviously the signal to noise problem. You
may run in the cloud you get data from your notes you see IP addresses communicating with
each other but IP addresses by itself mean nothing in Kubernetes clusters. They come,
they go and at scale it's impossible to track and trace what's going on with service to
service communication in your applications. Also where our existing mechanisms falling
short so first of all network devices think about centralized monitoring or firewall solutions,
think about Splunk. They are great to get alerts, to get dashboards but again at scale
they can be very costly or they can be a bottleneck. Also these devices don't have awareness of
the identities of your applications running on your Kubernetes clusters. Cloud provider
network logs are nice for note to note communication but don't provide identity as well. You can
monitor the host, you can see Linux host statistics but that gives you only visibility on the
note level and again a Linux note doesn't have any awareness of the identities of the
services running in your Kubernetes cluster. You may want to try to modify application
code and this applies a bit to the service mesh session. You may want to install libraries
which gives you visibility of the application but then you don't have visibility on the
networking layer. Service mesh, main goal of service mesh is obviously visibility of
the network and trend communication to get metrics out of that but that with the sidecar
based implementation comes with a footprint and with induced latency plus you have operational
complexity maintaining your service mesh solution on top of your Kubernetes clusters.
So this is where Cilium comes around the corner is that we provide identity aware based observability
and security. What that means is based on the labels you set on your workloads, we create
unique identities and we're able to attach that identity in the data plane using eBPF
and using that identity we can do things with that so we can secure the connectivity in
this example a front end to a back end is allowed to communicate based on the network
policies we set and the identities we are aware of and these identities are cluster
wide property but in terms of observability this also means that we can use this identity
to get rich metrics and data for that identity and you can monitor it effectively. This means
that you're not looking anymore at IP addresses, you're looking at identities so the whole
set of workloads the service to service communication for a front end to a back end.
Hubble is our observability solution built on top of Cilium, how it works is that Cilium
runs as a demon set on your cluster nodes as an agent and Hubble can retrieve data from
those agents through a CLI or UI and we can export metrics based on your workloads. So
there are three parts, first of all the Hubble UI gives a service dependency map so on a
namespace level you can see what's deployed, what is communicating with each other, what
kind of protocols they're using, what's coming between namespaces so you would see for example
if there's inter namespace communication you can identify the source namespace, if you
use cluster mesh you can even identify the source cluster, you can also identify egress
traffic and ingress traffic on a namespace level. The Hubble CLI is more a power user
tool to give you detailed flow, you can export it to JSON, you can do a lot of filtering
based on labels. Hubble metrics is the part where mostly the topic for today is where
you export metrics and you use Grafana for example to observe the performance of your
cluster application. This is all fueled through EBPF so again think about a network device
sending a packet, that's a kernel event, EBPF program gets attached to it, gets the metrics
and it's done. This is a small screenshot of the CLI so this gives you flow visibility
using Hubble observe commands, you can follow for example based on the label in this case
X-Wing so we're following all the workloads labeled with X-Wing so again no IP addresses
just labels. In purple it's highlighted these IDs we use so again each unique set of labels
gets a unique cluster wide ID and based on those IDs we can track based on labels what
communication is going on and there's a lot of metadata you can filter on things like
headers, things like ports, things like protocols, obviously labels in Q&A spot names, services,
worker nodes, DNS, we also have Cilium network policies which allow you to filter and observe
two FQDN rules meaning we can inspect queries to external domains and we can filter based
on that and obviously Cilium related identity such as world, ingress, egress, host and that
kind of stuff. Policy verdict matches, things like dropped, allowed and stuff. This is the
Hubble UI surface map like I said before this gives you a namespace level view in this case
we have a jobs app and I'm using this app as well in the demo I'm showing a bit later
so here you're looking at a namespace level view where you can see all the surface to
surface communication of your application running in that namespace. In this case it's
only intra namespace communication and you can see for example that the recruiter is
talking to core API, the core API is talking to Elasticsearch, we have a zookeeper component,
we identify Kafka, also identifying Kafka protocols so there are a number of protocols
we can inspect and see and we also see layer 7 information so in this case HTTP calls to
a specific URI or URL with specific method and return calls and this is triggered through
just simple construct as a Syllium Network Policy. If you just allow let's say internamespace
traffic and you are accepting HTTP that already triggers this visibility for you to see. Now
using this data we can also export metrics to Grafana so we are working with Grafana
a lot more lately that means that we are building a lot of more useful dashboards and also integrating
with things like Tempo for getting transparent tracing in Grafana, all powered through Syllium
and EBPF. This allows us to not only see on the network level metrics on performance
on the node but also for surface to surface communication to provide golden signals things
like HTTP request rate, latency, response codes and error codes which would as an application
engineer would allow you to quickly see which component of the application is not responding
as it should. But also detecting transient network layers so this will be more network
related we may see retransmissions, we can see bytes sent and received and we can indicate
things like boundary time to indicate a network layer problem. So maybe in a data center you
have a specific rack switch or top of rack switch not performing as it should so nodes
connected to that switch will have improved or will have reduced performance and you would
see latency increasing. Now with the latest dashboards we also able to see programmatic
API request using transparent tracing. This goes to the integration with Grafana. So at
the moment your application need to be able to support it so you need to be able to inject
traces. But we are working out of the box getting also this support and be able to help
by help support HTTP traces as such. And then you get this exemplar so I am pointing
at a small exemplar here after which you can inspect this with Tempo and you can see a
span of a specific request and see where the problem may reside. A bit more on monitoring
so this is more day 2 ops. I want to highlight that if you run Cillian and you are also using
Grafana make sure that you install the agent, Hubble and operator metrics plugins. These
are out of the box plugins we provide through the Grafana marketplace you can download.
This gives you visibility in the performance of your cluster. So first of all agent metrics
everything on the node level how the node is performing, how many throughput they are
processing, how many memory the BPF is using, all this related stuff. Hubble metrics gives
you the visibility across your cluster in terms of application, layer 7 return calls,
policy verdicts so allows versus drops so you can monitor on the cluster level the performance
of your applications. And in some cases you run an operator so you may want to track the
number of identities, how the cluster in general is behaving, API responses and such. And finally
what we released just a few days ago thanks to Raphael who is also here is the Cillian
policy verdict metrics dashboard which gives you the capability to get meaningful graphs
if you have workloads actually hitting network policies you set. What I mean by that is that
when we work with customers with Cillian is they want to go to this micro segmentation
zero trust model and you can use obviously Hubble to monitor service-to-service communication
and to see if traffic is allowed and denied. But this dashboard also is a very useful tool
to confirm if you have either ingress or egress policies which are matching with your traffic.
So in this case we see green graphs which means that on ingress and egress we have matching
traffic. The purple represents DNS matching traffic but if there's some yellow traffic
that's either allow all match traffic which is too broadly which should trigger you to
get even better network policies to make sure that kind of flows are actually related to
a network policy to ensure that both ingress and egress traffic is secured as such. If
you do so all the graphs will turn green and you know and you can confirm for that given
namespace that you have secured it. Alright I've prepared a little demo. This runs this
tenant jobs application I mentioned before. I'm running this on Kynes so just a simple
Kynes cluster on my laptop. I'm showing here the components of my application so it's you
know a number of workloads I've shown before on the screen shot. To help me through this
demo I've created a little script and what this does it only updates a helm chart for
this application so it makes my workflow a lot easier. I don't have to enter commands
but we should see some things changing in a Grafana dashboard. Before I start this let
me highlight the metrics so I need to log in. So I've installed Grafana, I've installed
Tempo, I've installed Prometheus and configured Silium to export those metrics. So this is
currently the performance of my application running on my Kynes cluster on my laptop.
So as you can see we have 100% success rate, we have incoming 100% and we also have good
Grafana information for the performance of the application. Okay so let me start with
starting the script. Yes so I mentioned before that the Hubble metrics are available as soon
as you configure some kind of layer 7 Silium network policy because that triggers the collection
of those metrics for layer 7 and I'm showing this but I will show this a bit better in
a different window. So what I'm going to do now is I want to increase the request volume
so I'm configuring the crawler component to get more requests in my application. As you
can see it's redeploying the crawler component. So this is something we should see in the
Grafana dashboard. While this is redeploying I can show the helm chart I'm using. You
need to be a bit patient with me because it takes one minute before the graphics, the
Grafana dashboards are updated and you can actually see the impact of this new version
of the application. So typically you configure Silium through a helm values file so in this
case on the operator component I've enabled metrics and Prometheus. On the Hubble side
I've configured Hubble relay to gather the metrics and also Prometheus and metrics. So
in this part it's very interesting because if you want to have layer 7 visibility you
need to have specific metrics being enabled. This will be documented in the Silium IO website
once we have the new release ready. So as you can see we are matching HTTP V2, we have
enabled exemplars, we are looking for labels in terms of context, source IP, source namespace
etc. So these are important sets of labels you need to set and on the Prometheus side
we've enabled it to gather the graphs. The Silium network policy I mentioned before this
is just a simple example. We allow everything within a namespace. We have enabled DNS visibility
so we're inspecting all DNS traffic to cube DNS that allows us to get visibility of the
DNS queries. We've enabled ingress and egress for the purpose of the demo so we can also
see that traffic. And what's important is that we have an empty or open rule HTTP which
allows us to see all traffic, it allows all traffic, but that triggers the collection
of metrics. Alright so on the demo side so it has deployed a new version of my application
looking at my metrics. I can see incoming request volume increasing so you see already
an increase of volume. We also see requests by source and response codes increasing so
still 200, always fine, always good, just an increase of request per seconds. And also
on the incoming side. Okay all good. Okay let's now deploy a new configuration of our
app and this app has an error. So let's see what we can see there. I can redeploying the
core API components and now we should be able to see the error rate increase as a result
of core API configuration changing. So this will take one minute. Here I can select the
destination workload so I can switch between core API or the loader component to see how
the traffic for that destination is matching and how it's performing. Let's give it a few
seconds to actually show. What I'm looking for is the incoming request to access rate.
Obviously this application version has an error so the success rate will be lower than
before. It's running. It's always a bit, takes a bit longer than I wanted. There we
have it. In the meantime I'll already start the next step so I don't have you waiting.
So here you see that the success rate is decreasing because of this faulty version of my application
so I can really see there's something wrong with my application and as application developer
or owning this namespace I should now be able to investigate what's going on. This also
means that here on the incoming request to source and response code I would see the resumes
components showing 500 and 503 error return codes which triggers me to check that component
and communication between those components. Also on the destination site. All right. So
now I've introduced a new version and what this does is changing the request duration.
So again a new version of the application and let's see how we can monitor this performance
of the application in Grafana. So let's check the request duration increase as a result
of core API configuration changing. Okay. So let's use here. So I'm monitoring HTTP request
duration by source and destination. So if the demo works well we see an increase there.
Okay. It takes a bit too long. I'm comfortable with but it should be there any minute. It
should appear any moment. Let me just... In the meantime I will deploy a new version of
the application which also introduces tracing. So again for tracing to be supported you at
this moment your application needs to support that. So in this case I'll deploy a new version
of this application to support tracing. And this is using open telemetry. So let's deploy
that in the meantime. That's deploying. In the meantime I can check how the request duration
is doing. Okay. This part is not working yet but we should see a request duration increase.
Oh God yeah thanks. That doesn't help. I clicked on something. Ah yes thank you so much. Yeah
here you can see the request duration increasing. And I just deployed a new version of my application
which supports tracing using open telemetry. And then you already can see that I have these
exemplars appearing. So I now can not only inspect HTTP request duration but I can also
inspect specific traces and exemplars. So if you click on this little box you get this
window. You can get valuable information about this trace point. And then you can query it
with track tempo. Yep let's leave this side. So here you can see a specific trace ID and
you can see a node graph. So this is also nice. You can see between nodes what's going
on and you see highlighted in red here what has a high latency as such. And here we can
see that in this specific API call there is an error. So a post call and it has some events
exception, random error. So something is wrong with my application. So this enables me as
an application owner to troubleshoot my application effectively. So this concludes the demo. Let
me quickly move to here. All right, so if you want to know more how to configure Cilium
to enable metrics, how to configure Cilium with the right values for layer 7 monitoring,
I recommend to read the documentation on Cilium.io. If you're using Cilium or planning to use
Cilium and you have questions go to our Slack channel. We're happy to help you there. The
community is out there and very helpful answering questions. If you want to know more about
eBPF go to eBPF.io. We also have released or close to release a lab with this kind of dashboards
as well. So feel free to check them out at isovenom.com. And if you want to know more
about isovenom.com or you may want to contribute, we also have open positions for engineering
as such. So if you want to know more, please check us out. I'm happy to take questions.
Any questions? Hello. Thank you for your talk. Is it possible in the service graph of the
Hubbell UI to show transitive dependencies of services with the tracing enabled?
So with Hubbell UI, the open source version, you will see the service connectivity only.
So that related information is not integrated in Hubbell as such. So you would switch between
Hubbell and Grafana to get that information. On the enterprise, we have built in dashboards
for getting that specific areas of monitoring. So let's say application or node performance
or cluster-wide performance. We have some dashboards which should quickly highlight performance
issues there. Okay. Thank you.
Any other questions? Hello. Did you measure the impact of metrics,
recollect of metrics on network performance? Yeah. We do have some performance-related
reports on sodium.io. So yes, it comes with a price. Using EBPF, we keep it as low as
possible. It's a very hard question to answer because it will depend on which flags you
configure. So if you have full layer 7 visibility across all workloads in your cluster, of course
it will have some performance impact for sure. Yes. Using EBPF, we keep it as low as possible.
But yeah, it's a multi-dimensional question. It depends on the amount of traffic, the amount
of applications, how big your cluster is, et cetera. So we have some performance reports
you can check. So that's 500 nodes, 1,000 network policies, helpful, enabled, and you
get some feel of how memory consumption and processing is with Selium. So feel free to
check them out on the selium.io website. But in practice, it's a multi-dimensional story.
Welcome. Any other questions in the background?
Hi. Thanks for the talk. A couple of questions about the integration of Selium on AQS and
GKE. Is there anything specific regarding those implementations or are all the tools
that work natively on these kind of clusters? And second questions regarding above UI, is
it possible to see intern namespace traffic flows or is it limited to intranamespace?
OK. Good question. So to answer the second question first, yes, you can see that. So
if there is communication between, from a different namespace in regards to your namespace
and monitoring, you will see that. You will see those labels, and you will see the workloads,
even across clusters if you enable cluster mesh. So yes, that works out of the box.
On the cloud provider side, so if you run AQS with Azure CNI Powered by Selium, you have
a limited set of metrics which are enabled. And that's obviously from support reasons
for Microsoft to support that solution out of the box. However, you can also choose to
bring your own CNI with AQS, and that also applies to GKE and AQS, to manage Selium yourself.
Right? So then you have the freedom to configure the flags I just shown and to configure Selium
as such. Keep in mind that you're responsible obviously of monitoring, managing Selium,
and the cloud provider will manage the cluster. Any other question?
Cool. We have to cut it in.
Oh, yes. Sorry.
OK. Thank you very much.
Thank you.