Alright, so good morning everybody, my name is Joris and I'm a PhD student over at Hasselt
University here in Belgium and I'm doing a PhD on multimedia streaming and network
transport layer protocols or even better the intersection of those two.
Today I'm here to talk a little bit about a project we did called Vagvizir which is
an automated testing framework for orchestrating client and server setups using the quick transport
layer protocols but before we jump into that maybe let's talk a little bit about what quick
actually is because I assume not everybody has heard about it. Well quick is a general
purpose transport layer protocol that was standardized by the ITF in May 2021 and if
you have any updated like applications on your phone or have been using the latest releases
of browsers such as Firefox, Chrome or whatever you're using you've probably been already
using quick as it has been deployed to a lot of many different applications and websites
already. For example Facebook, Instagram they are using it, if you're streaming videos
over YouTube you have probably been already using quick. Quick is a name, it's not an
acronym, it used to stand for quick UDP internet connections but it has actually not been called
that for quite some time already now. Some of its features, the biggest feature is encryption
so the protocol actually encrypts everything by default which is great because that's the
main driver against ossification also it's reason that it was created because TCP is
actually an ossified protocol when we compare the two. It's also great for preventing third
parties from actually interfering with the data you are transmitting over the network.
It's less great for research and development as you have to actually account for that in
your test setups which we are going to talk about a little bit further ahead. It's currently
most implementations implemented on top of UDP in user space. Some implementations are
actually looking at implementing it more towards the kernel but those steps have not been taken
by many of the implementations. At present at least as far as I know 25 implementations
exist most of them also being open source written in multiple programming languages.
They also provide libraries which you can directly use to actually use quick in your
applications. Another benefit of quick or another new thing with quick is HTTP 3 which
you might have heard of. The reason for the introduction of H3 is that H1 and H2 actually
only run over TCP that's why they created a new version of HTTP called HTTP 3. There's
not that big of a difference between H2 and H3 in practice but just for sake of naming
it it's HTTP 3. Right so now that you know what quick is that's at least a requirement
for understanding the stock. Let's talk about how we can actually use this in like experiments
and stuff. Maybe let's try doing something very simple. I just told you that most browsers
implement quick. Maybe let's try connecting to a website that only implements an HTTP
3 server. Should be simple right but as you can see on the screenshot it is not in practice.
And the reason for that is really simple that's because browsers decided early on that HTTP
3 server should be discoverable through the old SVC header provided by an H1 or H2 deployment
which really sucks if you want to do some automated testing because that means you also
have to put up like or spin up an H1 or H2 server and actually account for this. Luckily
we have some options like for example within Chrome and Firefox to enable force quick on
certain domains which we can automate through or by means of for example parameters supplied
in the command line or by configurations in the browser itself. Right so we can connect
to a web server at this point. How do we actually know what's happening under the hood should
also be simple right. Remember everything is encrypted so actually seeing what's happening
is a little bit is not that trivial actually. Luckily most implementations nowadays use
like standard of the shelf TLS libraries and these TLS back ends actually support an environment
variable called SSL key lock file and the idea behind the SSL key lock file is that you can
like point it towards a file which then gets used by these TLS back ends to like output
all the secrets used for encryption during a whatever the application is actually doing.
If you load those SSL key lock files into programs like wire shark you can actually
decrypt the traffic which is nice if you want to see what happened. Unfortunately tools
like wire shark at least as far as I know don't actually have any visualizations about
stuff that's happening at like the congestion or flow control layer. You have that for TCP
but quick those things don't exist yet. But luckily we have other stuff for that. Q lock
is one of them. There has been a nice talk about this by its inventor a couple of years
ago at FOSDOM. I really invite you to look at it. But basically in a nutshell what Q
lock is is like a structured way of logging and a unified way of logging that can be implemented
by any endpoint implementation using quick. In a nutshell this is basically a file for
example a JSON file that just locks everything that's happening and if you have some scripts
or tools that can parse this you can actually do a lot of fun stuff with it. For example
the QVIS visualization tool which is also by the same creator is a tool that allows
you to load these files and like actually visualize similar to what wire shark can do
for TCP but then for quick what's happening on the congestion layers. For example on the
left you see a flow control and congestion flow graph and on the right you see a plot
of the home trip time that was experienced by the application. So we can look at what's
happening under the hood maybe let's try something more advanced setting up like your own quick
client and quick server to do some local experiments maybe change something to the implementations
it doesn't matter really what you want to do. Even that is not that trivial simply because
there are many implementations written in many languages meaning that they have their
own requirements their own installation procedures. Another distinction is that different implementations
actually have different performance characteristics meaning that some are more tuned towards certain
scenarios some only support a certain feature set so you also have to account for that.
An additional requirement is that you also have to set up like self-signed certificates
and for some reason some implementations accept all kinds of certificates and then for some
reason others fail we have never really figured out why we just use a common way that works
for them all now anyway. Another query that you can experience is within the code bases
themselves a fun one I always show to my students is this one this is from the quick
code base which uses the cubic congestion control algorithm if you know something about
congestion control and makes sense like the file is called new cubic sander the function
is called new cubic new cubic sander it even specifies in documentation that it makes a
new cubic sander unless you actually put a Renault ball on true then it behaves like
a totally different beast actually new Renault in that case so some some weird quirks that
you actually have to account for too. So the point I'm trying to make it is is that there
are a lot of different implementations testing the mall takes time it's not that easy to
set it up you experience a lot of these small issues it's cumbersome to like test multiple
implementations which is the reason why I am presenting Vagvizir today. The idea behind
Vagvizir is to actually aid with this kind of development so if you're doing research
or even development within quick the idea is that Vagvizir can automatically set up
these kinds of interactions between clients and servers but also using simulated networks
such that you can have actual repeatable and shareable experiments. The way you do it this
is by defining experiments with configuration files and a single experiment can consist
out of multiple test cases and the idea is that a single test case looks something like
this. So you have the two entities the server and the client which just assume their prototypical
roles as known within the server client model and in between them sits a network component
that we call the shaper and the idea of the shaper is that it actually applies some kind
of scenario on the traffic passing between the server and client for example it can introduce
some latency or it could limit the throughput doesn't really matter what you want to do
the idea is that you can do it in a repeatable way. You also see the docker container stuff
on top the idea of using or actually deploying these test cases within docker containers
is that we can easily share them with other people which is a really nice benefit within
the academic community but also we can free certain implementations like we can actually
save a docker container and reuse it at a later point so say for example something changes
and we want to try an older version that's totally possible with this setup. Additionally
within the quick community there have been some other efforts if you are part of the
quick community you might actually recognize this setup it's pretty much the same as one
used by an interoperability project called the quick interoperability runner. They also
provide containers for their setup that are more tuned towards testing the actual interoperability
between quick implementations but the benefit of using the same architecture is that we
are actually completely compatible with their setups so that means that even though Vegvizir
is relatively new at this point in time we are already compatible with 15 out of the
25 quick implementations right out of the box. You also see on the right side that we
have a client that can be defined as a CLI command that's because early on we realized
that if we want to test applications not everything is not every kind of test is suitable to be
placed in a docker container which is why we also allow to define test by just spinning
up local programs as you are used to from a terminal. A good example of this is a browser
if you're doing some kind of media streaming experiments you actually want hardware acceleration
as such to be enabled which I guess you can do this in docker containers but it's really
cumbersome to actually do this in a good way. Right okay so how are these experiments
actually defined? Well we decided to not use one single configuration file simply because
that would mean we had to be very verbose. We actually split it up in two types of configurations.
On the left you can see the implementation configuration which is actually what defines
what is available within an experiment. So the idea is that an implementation configuration
is similar to like your list of installed software on your computer. You simply have
a list of entities that you can pick from. We also introduced a parametric system to
make it actually really dynamic steerable from within an experiment configuration and
we will see some examples on that in a second. On the right you see the experiment configuration
that's the actual definition of what needs to happen within one experiment so what defines
the test cases. The idea is that you define how the entities from the implementation configuration
should behave and what parameters should actually contain as values through arguments.
Also configure sensors I'm going to talk about that in a second but the biggest benefit
of splitting these two up is actually that the experiment configuration automatically
produces this permutation or rather a total of combinations from all these entities. So
say for example you define two servers, two clients and two shapers. The total amount
of tests within this experiment will actually be eight because it just compiles a complete
combination of all these configurations. Another benefit is loose coupling so you might wonder
yeah I still don't see the reason why you split these two up. Well a big thing with
an academic research is that we actually want to test different versions. So if we have
an implementation configuration that for example defines a client called Chrome which
then refers to a Chrome browser we can actually have one implementation configuration that
refers to version for example 99 and we can have another implementation configuration
that refers to for example version 100. The benefit of that is that if we simply swap
these implementation configurations we don't need to change the experiment configurations
meaning that we can without having to verbose rewrite all these stats really easily test
multiple setups. Some examples this is an example of an implementation configuration.
I do invite you to go to the GitHub repository where everything is really nicely explained
and we provide some more examples unfortunately limited by the screen size. You see that we
always have to define in the implementation configuration three types of entities like
we talked about earlier the clients the servers and the shapers and these three are examples
using Docker system. So in the top two you see that we actually refer to Docker Hub images.
These are actual examples that come from the quick interop runner project which we are
compatible with. The bottom one is a locally built Docker image. The reason I highlight
this difference is because the framework automatically pulls the latest Docker Hub images if these
are available. But if you are using some kind of local implementation that you build as
a Docker image you actually have to build it locally and then refer to it locally. Another
thing you can see here is the parametric system. So for example the top client defines a request
parameter that is then used within an experiment. The idea is that an experiment configuration
then contains a value of it and that you can access this value within a Docker image simply
by using requests then in this case as a environment variable. So all the parameters are passed
as environment variables if you are using Docker images. Or in the case that you are
using CLI commands or even in a more specific case of shapers because shapers are a little
bit more complicated. You can also use them directly in the commands you specify within
the implementation and experiment configurations. These are directly substituted and you can
actually reference other parameters within arguments which is nice. A simple example
of a CLI client, so one that is not using a Docker image in other words, you can see
that the command is rather long. That is because we cannot, well compared to a Docker container
we actually have to specify everything that needs to happen in CLI command. This example
provides three or rather four system parameters which are highlighted here. The reason I did
this is because the framework automatically generates all these details for you such that
the experiments can be even more dynamically steered. This is especially handy for future
use cases where we for example want to expand upon multiple client setups and stuff like
that. On the bottom you see a construct key. We actually have two special mechanisms for
CLI commands. In Docker images, the benefit of Docker images is that they can actually
have scripts on board that actually prime the environment. That is the downside of using
CLI commands unless you want to put everything on one single line which is rather also cumbersome.
Instead we provide two mechanisms called construct and destruct which are run before and after
a command is executed. These can be used to prime an environment and clean it up afterwards.
This example sets like the changes or manipulates actually the Google Chrome preferences to
set like the standard download folder output towards one generated by the framework.
Then we come to the experiment configurations examples. These are the actual configurations
that define how a test should behave. You see here once again we have client shapers
and servers which we picked from the implementation that we just showed. We simply filled them
in with the arguments required for the test to work. A special thing to notice here is
the shapers scenarios. Clients and servers are really simple. You just mentioned which
one you want to use. But for shapers we have a more complicated setup. The idea behind
the shapers is that it actually entails one kind of shaping. For example, you can use
a TC-netum shaper within one container. But this one container does not only do one kind
of shaping. The idea is that you can define multiple scenarios within this container and
by passing through the scenario key you can actually pick which one is used during a test.
In this use case we have one client, two shapers and three servers which means that we will
have a total of six test cases that will get generated and compiled by the framework and
run sequentially one after the other. I mentioned sensors earlier. That is also a configuration
you can do with the experiment configuration files. The idea is normally that the framework
just automates all these tests and that when a client exits this should signal like the
end of the test. However, in certain circumstances it is not possible. For example, if you use
a browser, well, it is obvious that browsers do not have the ability to shut down from
within a web page which would pose some security risks. Which is why we built a sensor system
which can actually govern what happens within a client. For example, we provide two simple
sensor setups, time mode, which simply checks if a certain amount of time has passed and
then closes the client and signals that the test case has ended. Another one that we built
is the browser file watchdog sensor which enables us to check if certain files were downloaded
by a browser context. This enables us to pull metrics from the browser and also signify
the end of a test. If you provide these two configuration files to the framework, the
framework will spin up a nice story. On the bottom you can see which tests are happening,
how much time has passed. You can see a little bit of packet spossing between them, signaling
that some kind of traffic is happening. You can actually increase the verbosity, but this
is not necessarily needed from within the terminal as the framework automatically saves
everything that happens as output in a file within the test case folders that we will
now discuss. The experiment output is always saved under the label that is provided with
an experiment configuration because we can have multiple runs of an experiment. The first
entries that you will find within such a folder are actually time stamped to signify multiple
runs. If you enter that, you will actually find the different folders that contain the
data of the multiple test cases that were compiled by the framework. If you take a dive
into one of these folders, you can see what output we are collecting in these cases. By
default, the framework will always create a server, client and shaper folder which get
automatically mounted on the Docker volumes under the slash logs directory. Anything the
implementations want to save, they can just write files to this directory and the framework
will capture this and save this to in the log files. Additionally, clients also have
a downloads folder mounted simply because we want to differentiate and not come into
a situation where downloads accidentally override output logs generated by a client. You can
also see that we have, especially under the server and client entries, you can see keys.log
and a Qlog folder. The framework is automatically primed to save these encryption details and
what's happening at the quick and HTTP tree layers by setting the SSL Qlog file which we
discussed in the beginning but also by setting a Qlog environment variable which gets recognized
by most quick implementations out there nowadays. Finally, we come to extensibility. At this
point in time, we have a framework that is great at aggregating a lot of data. We did
some tests that ran for two or three days straight containing more than 8,000 test cases
which were great if you want to gather a lot of data. But what makes a testing framework,
a testing framework, is the actual ability to infer something from the output generated
by a test which is why we provide these two programmable interfaces called sensors and
hooks. I explain sensors a little bit. We provide some basic sensors but you actually
also have the ability to program custom sensors. This makes a lot of sense if you want to do
very specific or test for very specific behavior within your experiment. For example, if you
are doing a video stream in the browser, you can actually send the decoding metrics of
the video out of band to an HTTP endpoint, for example, that you set up in a custom sensor.
If the sensor, for example, detects that some frames are being dropped or decoded in a wrong
way, it could prematurely hold a test signaling that something went wrong. If you have lots
of test cases like we do, we actually have test cases like I just said, running 48 hours,
this is really beneficial because it holds the test in an early phase, saving us a lot
of time. On the other hand, we have the hook system. So the framework currently is very
broadly applicable. The downside of that is that we don't really know what's happening
inside the test. But you can actually program some custom behavior through the pre-run hook
and post-run hook system. As the name suggests, the pre-run hook runs before an actual test
is run. So you can prime environments by, for example, generating some dynamic files
that you will need during the experiment. It doesn't really matter what you want to do
there. The post-run hook is really nice because you can use it to analyze whatever happened
during a test. For example, you could, if queue logs are being generated, look at the
queue logs and maybe even generate some nice graphs that you can immediately check after
a test case has ended. Right. Another thing I need to mention with the pre-run hook and
the post-run hook, if you don't like programming in Python, it's not really a problem. Python
has this really great submodule called subprocesses. If you have some existing scripts that are
made to work with the output produced by your experiment, you can simply call them also from
this hook, meaning that you get exactly the same results without having to actually translate
your existing code within these provided hooks. Right. That's in a nutshell what Vekvizir
does. Thank you for your attention. And I think we have a couple of minutes left for
questions.
Yeah. So, a test case can be anything you want. If you have, like, if you're programming
right now, you're developing something locally. The thing you need to do is actually wrap
it within a Docker container. That's one way. Or run it as a CLI command. You simply need
to provide it to the framework, and the framework will just spin it up. So, the framework doesn't
actually check what your test case is doing. If you want to, like, spin up a simple, let's
say, CLI command, like, echo, and you want to print something to the terminal, you simply
put it in the JSON, it will run.
So, more questions, please. We have a couple of minutes. Okay. I have a question. I see
you're from university. What does university have to do with testing, like, what's the
question? Okay. So, good question, actually. I'm not sure if there is a direct relationship
with testing in the university. It's just that, like, during my PhD, and also the PhD
of some of my colleagues here in front of me, we actually encountered that we had a need
of such a framework, right? We had an actual need of spinning up multiple test cases and
like, helping us with setting up these experiments, which is why we designed this. Early on, we
just had a very minimal thing that just worked for us. And then, as time progressed, it actually
became more and more mature, and we decided, well, this is actually a very good idea. So,
we created an open source project for it, and we actually also submitted it to a open source
and data set track for the MMSIS conference, which is happening in June in Vancouver. So,
okay. I think we have time for one more question. The last question. No thankers. So, thank
you very much.