Looking at this from the angle,
how can I manage such a large graph in a good way,
and moving forward to that, so Nikolai.
Thanks for the introduction.
We have to speak up because the audience only for the-
Okay. So my name is Nikolai Kondashov.
I work at Red Hat on the CKI project,
which has built in one of
those Linux kernel testing systems for Red Hat and for Upstream.
I also work in the kernel, louder?
Okay. I also work with the Kernel CI Upstream Community on
the KCI DB project,
which is the source of this presentation,
and I do electronics and embedded as a hobby.
Okay. So I'm going to walk you
quickly through the kernel contribution workflow,
through the testing systems,
then what we are trying to do with KCI DB at Kernel CI,
and then how we want to solve the problem,
and what the actual problem is with
the Kernel CI process in general.
Then I go briefly through the data model,
and what kind of a few questions,
what a few queries that we need,
and how it went with Neo4j,
and what we can do instead.
So the kernel contribution workflow,
I don't know if everybody's familiar with that.
I hope not because it's not very pleasant.
But basically, you do your changes,
you commit your changes,
then you make an email out of that and send it to
a mail list and to a maintainer for them to review,
to give you feedback,
then you repeat that again until everybody's satisfied,
including maintainer, whoever is concerned with that change.
After this, your patches get merged into
a sub-tree for the particular subsystem that you were changing,
and then sometime later, this is getting merged into
the mainline which Linus maintains,
and you're done basically.
But at any point in that process,
you can get some test results for your change.
It could be if you're lucky,
you can get it before it even gets reviewed,
or sometime it gets reviewed,
or after it was merged, any time.
So there's a whole bunch of
kernel testing systems, this is just a sample.
Each of them is trying to solve their own problem.
For example, CKI is a Red Hat system,
they would test particular hardware that our customers use,
particular features that our customers request,
to make sure that they work,
that the distribution works,
Intel tests their hardware,
their graphics cards, and make sure that those work.
Google fuzzer system calls, SysColor and SysBot,
LKFT from Linaro, they test ARM boards,
and finally, kernel CI is aiming to be
the official CI system for the Linux kernel,
it's supported by Linux Foundation,
and they're trying to run tests on
the whatever hardware others can provide, we can have.
You can see everybody has their own interest in that game.
So this is how your various email reports can look from
those systems correspondingly,
and this is their dashboards from different systems.
So kernel CI, as I said,
is striving to be the DCI system,
and we have a testing system and
the hardware management and
the framework and everything to run the tests in various labs,
and these labs can be located in
different premises by people who
have some hardware to run them on the test zone,
and then that gets collected and put into the database,
and then we have various other CI systems
collecting their results and sending them to the KCIW database,
and KCIW was conceived as a system to try to
reduce the effort that all CI systems
have to put into their dashboards,
into their reports, and instead have
one dashboard and one report if possible or close to that,
and as well to save the developer's attention,
which is a precious resource because as you see,
it's not so easy to investigate every report
and from different CI systems
because they are differently formatted emails,
different data, different dashboards,
you have to look at them this way,
that way, and you have to figure it out.
So that's case IDB is the effort to bring
this one into all the wall.
So conceptually, it's very simple,
these are systems and JSON which can consist
like various objects in any combination,
and we have the database we put them in,
we have the dashboard to display that,
and we have a subscription system where you can give
some rules and say like, okay,
I want to see these results from this test and from
this tree or for this architecture or whatever,
and we can generate the reports based on
that whenever you need it as the data comes in.
One important note about this is that
compared to our regular CI system where you control everything,
in this system, the data can come in in any order.
In a regular CI system, you have
the results come in the same order as commits come in.
So if you tested something earlier,
that means for an earlier commit,
if you tested something later,
it's for a later commit,
and you can have a line of history with those results.
But for case IDB,
since various different CI systems,
they get in any order you wish.
So we have about 100,000 test results per day,
a few thousands of builds,
and hundreds of 100 revisions per day tests
that received by the case IDB database.
Well, actually, I think, yeah, that's correct.
That's the correct scale.
So it looks something like this as
Grafana is like a prototype dashboard.
We're thinking about building a new one,
but I don't know how soon that's going to happen.
So graphs, tables, all that jazz.
This is our prototype reports look like this.
So what's the problem with the kernel CI in general,
not with the kernel CI, the project?
So first of all,
kernel is intended to be
an obstruction layer for hardware.
That's this whole purpose,
and to make it easier to write software.
So in theory, to make sure that it works,
you have to test it with every piece
that you're abstract away from.
But that's not possible, of course,
and hardware is expensive,
so it's a natural scarcity in this whole system.
Then the tests, since you cannot get
all the hardware at the same time,
and you cannot possibly run all the tests on
all the hardware for every commit that people post,
it means that sometimes the tests run on this hardware,
sometimes on that hardware,
sometimes they don't run,
and the tests themselves are not so reliable
because there's a lot of
concurrency management in the kernel,
and that's hard to get right,
and in general, things happen at
the same time in the operating system,
so then sometimes they're not so reliable.
So you can get a pass on your change,
even if it's broken or get a fail on your change,
even if it's not broken,
or even if it's somebody else's change that broke it,
basically, hell.
So it's hard to remove noise from those results,
and for developers,
it's hard to investigate even a valid change.
While it's a kernel,
you have to meet all the conditions,
and well, sometimes you have to get the right hardware,
or ask people for the right hardware,
or ask them to actually run the test and send you results,
like you know, over email takes a while.
So if we start sending people emails with
results that are not valid,
false positive, false negatives,
then people kind of get pissed because of that,
because it takes such a long time to reproduce them.
So a lot of CI systems resort to
human review before sending those reports,
like they see the failures,
they say, okay, well, let's send this to
this mail list and then they send them,
and only a few manage without that so far.
So obviously, nobody stops the development to fix CI,
because there's just so many developers,
and if one system breaks something,
like another subsystem doesn't want to care about that,
and the feedback loop is just too long.
So tests keep running,
keep failing, and it takes a while to fix them.
So instead of the ideal case where you can move
past, only move past the tests if they pass,
and then do all the stages,
like a review, and then it's merged, and it's test,
and it's fine, and then you can upstream it,
you get something like this where all tests fail,
okay, it's probably not our problem,
not have time to investigate it,
or we just didn't get any test result with new one.
So what we're trying to do is we got to fix this, right?
So we got to fix the test results.
So we fix the test result.
We look at the test output conditions,
et cetera, and we add a rule to the database saying like,
okay, well, this failed,
but we know about this,
here's the bug that was open,
so don't complain to developers,
don't waste their attention,
and it looks like this,
shiny and sparkly,
but after a while,
we get this fix into the test,
and we repeat the process with another issue.
So these things are already working in
separate CI systems like the CKI.
There's a UI screen for an issue in the kernel,
it says like, okay, look for this output in the test,
for this string in the test output,
if you see it for this test,
then we consider it a kernel bug and don't raise the problem.
Okay, so or bug log CI,
Intel's CI system,
they have like a huge form.
For file in this, you can see another string that
is you're supposed to look in
the error output and the conditions and
what kind of status you want to assign to the test, et cetera.
So here's a dog tags for you to take a breath,
and for me to take a drink.
So I'll dive into the model.
We start with checkouts which basically just specify
what kind of revision you're checking out,
we have taken it from repository branch and which commit,
and if you have patches applied on top,
and the patch log and everything like that,
then we aggregate that to get the revision data,
like from multiple checkouts of the same revision,
they get the same single revision,
and they have builds which link to the checkouts,
to say like, oh, we just tested this check out,
and therefore link to the revision.
The builds describe which architecture,
compiler and configuration,
output files and logs and everything,
and we get the test results finally,
and yeah, builds can fail,
they have failed builds all the time and it stops nobody.
So we have kind of test which we are running
the environment to train on,
what kind of result it was,
the status result, pass, fail, et cetera,
and the output files logs and stuff like that, very typical.
Then we get the issues which describe like which bug it is,
and who is to blame like the kernel,
the test or the framework,
and we will have the pattern there matching the test results,
okay, this test, this output,
what you saw on that screen.
The status that it should have and the issue version,
because we want to change those issues over time,
and finally have the incidents which are linked in
those builds and issues together,
so saying like, oh, this is the issue with this build,
and things like that.
So that's all we keep in the relational database,
but then we got to talk about the revisions.
So revisions could be just a commit to get history,
and here's your graph.
So that's the basic thing that we've tried to do,
but we also need to have revisions of
patches applied on top and somebody
posts the patch on the main list.
We take it, apply it to some commit,
which is pointed to and we test it,
we get the results,
and we know it was applied to this commit.
Then somebody reworks that patch and posts a new version,
they got a link, both the commit we tested
upon and to the previous revision of the patch set.
Then there is this weird thing when
maintainers keep a special branch for
CI for the testing systems to pick up
their work and test and send them results,
and they just keep pushing there like they're
working on something, they push there,
they get results after a while from testing,
then they push a new version,
and then they get new results and they got to say like,
okay, this is the Git commit history,
but we also know that we checked
this branch out previously,
so this is the child of that branch,
of that previous revision.
This basically it.
Well, as you probably all know,
this is a directed acyclic graph,
so test directed edges and it doesn't loop on itself.
So that's about what I know about graphs.
So bear with me.
Finally, I think that there's
just too many build and test results to
put them all into a graph database at least so far.
I might be wrong, but that's my idea.
We obviously need to keep the graph of
the revisions to be able to reason about them,
but we might be able to put
issues there as well in the same database
if it saves us something.
So this is just a short list.
Basically, what we want to know,
okay, as the data commit comes in,
the test results you got to
try them and match them against the issue.
So we can say, okay, we found an issue here,
so don't raise the flag or something like that,
like similar, okay.
There is no issue here on test result,
but we want to raise the flag because there's actually an issue.
We cannot possibly try all the issues
against all test results because there's going to be a lot.
So we have to build a priority for those issues,
and then we have to cut off that priority somehow,
and say like, okay, at this moment,
we can tell the developer that we've
basically tried these results,
you can go take a look,
but we can still continue and
try those issues as the time goes on.
So we have to base that on one of
the criteria that we might need is how far,
for example, that revision is from the current situation,
like if this issue only appeared somewhere,
I don't know, like 1,000 commits ago,
or 1,000 is not that much for the Linux kernel,
okay, 10,000 commits ago,
then we don't need to try it right now.
We can tell the developer, okay, it's fine,
and then we'll go and continue
try it and if we find something,
then we can raise the alarm.
Okay, then we can ask,
like what were the last X-test results,
like for this particular test,
for this number of commits to be able to say,
okay, this test wasn't often failing,
okay, it was failing sometimes, but that's okay,
but if it suddenly starts failing more often,
we got to raise the alarm,
or if it stops failing so often,
we got to also raise the alarm and see what's changed.
Then we need to track the performance trends,
of course, over the history of the development,
and once again, we cannot do this just based on time,
because some systems move at
a different speed and some systems might start to decide to,
okay, we're going to test this old branch because
somebody if some of our clients wants to base their BSP on it,
wants to base the release some software with that kernel,
and we got to start testing it,
and it starts coming in like
the last year's release or something,
and we cannot just take that data into account
for testing the current releases or vice versa.
So, or for stable kernel maintainer,
if Greg wants to release a branch,
he might want to see like,
okay, which issues were discovered starting
from the previous release in this branch,
and finally, yeah,
like just for the dashboard, like,
okay, I want to see issues in this branch,
or which branches contain this issue.
So, that's what we tried to do with Neo4j.
I did basic things,
so I wrote a little script to get
the Git log in a particular format,
and then generate the data for commits and for relations.
It was a little over a million commits look like this,
and it was a little more relations,
because as you probably know,
a commit can have more than one parent in Git,
and it looks like this, very simple.
So, I loaded this into Neo4j with something like this.
This is updated to the latest release.
It was different than created an index for
hashes and then loaded the relations,
and it worked fine,
but not a few days ago when I
tried the Thresh Neo4j release,
it just hung like this forever.
So, I don't know, I could not give you
a fresh data how it works right now,
but I tried it last year,
and I couldn't get answer a simple question
if these two commits are connected.
It was just go on forever,
then run out of RAM.
But with Epoch, I could do that.
I could get the answer.
It was okay, but if I wanted to get
the nodes between those two commits,
it would do the same thing.
But with Git, I complete that in milliseconds.
So, here you go.
I think the problem, well, in my opinion,
is that the graph management databases and
software there aimed at a general graph problem,
and not tuned to DAGs.
How Git does that, Git is tuned to DAG,
they have a lot of optimizations for that,
and there are streaks to make
like repositories like the Linux kernel work.
So, I don't know nothing how you do this.
This is magic to me,
and this would be new to me in this book.
But from a purely engineering perspective,
I would have liked to see something like
a support for databases that are restricted for DAGs only,
and that apparently could be done
with not so much computation.
Then, once you have that,
then you can do some branching and say like,
okay, if we are DAG database,
then we can do the optimizations
and do the fast thing with them.
So, the full back plan is obviously just
put everything in Git, put the commits,
and the patches, and all the branches,
and all the subsystems, it's going to be giant repo.
Maybe we can manage that,
and then query it with libGit2,
which is the library that Git uses to work with the data.
Then, well, shuttle the commits with the relational database.
Okay, we want to see if between those releases,
we have issues and we take
the commit hashes from Git and then query the database with that.
That's all. Thanks.
So, we can help you with the Neo4j things.
It's just like literally this string, this length.
No, it's text index is for full text back for.
Okay. Well, it was just this one thing.
So, do you have the data somewhere to try it out?
Of course. Of course.
There's a link from the slides to
the script that you can use yourself on any Git repo.
Yeah. Any more questions?
Yes?
Did you try any other graph databases?
Well, I looked at the question is,
did I try any other graph databases?
Yeah, I looked at a bunch of them.
Some of them require so much setup that I was just floored,
but I read the documentation.
I couldn't see any indication that it would be any different
because nobody says anything about DAGs,
any optimizations or anything.
I tried memgraph before this talk,
but I had the same problem with loading revisions,
I think for some reason.
Because previously, I could load revisions.
I guess in Neo4j, the syntax for indexes has changed since then.
Maybe I did create indexing correctly as I was just hinted at.
But I could load them in reasonable time before in Neo4j and
everything fine and like in query and except that thing.
In memgraph, I just hit the wall because it's
a different syntax slightly, it was slow.
But yeah, no such luck and it took like four gigabytes of disk space.
So, not too bad, okay.
What version of Neo4j was successful?
I don't remember now.
I think it was, if I take a look now,
I think I-
The version will also be successful, it's just research.
I tried one Neo4j desktop 1.4 before,
1.415 and that worked.
I don't know which one, which version it was included.
Any other questions?
Thank you so much, Nikolaj.
Thank you.
Thank you, everyone.
I'm still looking forward to work with data in the graph database.
Because I think that's actually good for the graph database.
And so we can make it work and then Dexter,
you can come back and do some large scale analysis on the data.
Okay, that would be great.
That's what you can do.
Yes, thank you.
Thank you.