[00:00.000 --> 00:14.000]  Hello everyone and welcome. I will today introduce you to the C-PASS project and the virtual
[00:14.000 --> 00:17.880]  edition for real-time Power Grid Substitution Automation.
[00:17.880 --> 00:24.160]  First of all, I will present myself. I'm a consultant in open source software at Savoie
[00:24.520 --> 00:32.920]  which is a French and Canadian company which is an expert in open source software and we
[00:32.920 --> 00:39.040]  offer technical services to other companies using open source software. The company is a
[00:39.040 --> 00:45.480]  member of the Yoctop project, the Linux Foundation and the Linux Foundation Energy which hosts the
[00:45.480 --> 00:53.280]  C-PASS project. Okay, let's dive into the context. We're experiencing an energy transition
[00:54.240 --> 01:04.080]  mostly because of two points. The first of all, the new power production. For example,
[01:04.080 --> 01:12.320]  the renewable energy. You have many new power sources which are distributed around the country
[01:12.320 --> 01:21.040]  and not constant in power. But also because the request of electricity has changed with electric
[01:21.040 --> 01:30.480]  mobility with smart services and the Internet of Things. We have a new customer, a new services
[01:30.480 --> 01:38.640]  for the power distribution and so we need a new power grid and a power grid control architecture.
[01:38.640 --> 01:47.440]  We need to be much more distributed and to have a power grid that is flexible and adaptive.
[01:48.160 --> 01:56.000]  That goes with the idea of new and much more data management in the substation
[01:56.000 --> 02:01.840]  to adapt quickly to the request of the customer and the production of renewable energy.
[02:02.960 --> 02:11.280]  The C-PASS project comes in response to this energy transition. The vision of the project
[02:11.280 --> 02:19.440]  is to move from a hardware-centric model to a software-centric model. We want to abstract
[02:19.440 --> 02:25.440]  everything we can from hardware to software. So instead of having many pieces of hardware
[02:25.440 --> 02:33.920]  in the substation that dialogues together, we want to control all these hardware pieces,
[02:33.920 --> 02:43.200]  the hardware boxes, with main software to control them. That will offer the flexibility
[02:43.200 --> 02:50.240]  and scalability we want. The C-PASS project also choose to direct into the open source
[02:50.240 --> 02:59.360]  development. The idea behind that is not to depend on any industrial and any other company
[03:00.080 --> 03:07.840]  but also to offer for everyone to come and see what is developed in the C-PASS project
[03:07.840 --> 03:15.760]  and let everyone offer his point of view from every other industry to develop a project that
[03:15.760 --> 03:23.840]  will need to be not only suitable for the power grid. We need many vision for the project to grow.
[03:24.000 --> 03:34.320]  This is inspired by the North American model. For example, AT&T did that to develop the new
[03:34.320 --> 03:42.640]  5G software, the new 5G grid. They needed this flexibility and scalability and they did this
[03:42.640 --> 03:46.960]  transition from hardware-centric model to software-centric model.
[03:47.760 --> 03:54.080]  So here comes the C-PASS project, which is a software-enabled automation platform on
[03:54.080 --> 04:01.600]  Artifacturine. The mission of the C-PASS project is to develop a reference design at an industrial
[04:01.600 --> 04:09.440]  level for all applications to dialogue with. This platform will be open source and will run
[04:09.440 --> 04:17.440]  a virtualized automation for each goal we want to achieve, for each piece of software for each
[04:17.440 --> 04:27.360]  actuator or captor. They will be virtualized and encapsulated with the platform controlling
[04:27.360 --> 04:34.960]  the overall action. Our needs, we have many needs in this project. First of all, we need
[04:35.600 --> 04:42.960]  a very high preference in terms of real-time and latency because we want the grid to be flexible,
[04:42.960 --> 04:50.560]  yes, but also for security because we want to react very quickly if there is a problem.
[04:52.000 --> 04:58.560]  We also need the C-PASS project to be adaptable in terms of security. We want to be able to
[04:58.560 --> 05:07.600]  deploy patches and to close security breaches really quickly. We want it to be hardware-agnostic
[05:08.400 --> 05:14.640]  for everyone to be used, that goes with the open source mindset, and we want it to be really
[05:14.640 --> 05:25.920]  customizable and adaptable. Because we use open source software, we also will follow the state
[05:26.000 --> 05:32.160]  of the art of what already exists. The idea is to not reinvent the wheel every time but use
[05:32.160 --> 05:36.880]  already existing open source software, because we can because we are an open source project,
[05:37.680 --> 05:44.640]  and to configure it and to benefit from the fact that they are already really well done
[05:44.640 --> 05:51.680]  and already certified. We don't want to rewrite an entire program ourselves. That is nonsense
[05:52.160 --> 06:03.040]  in terms of open source software production. To achieve this requirement, we first set up
[06:03.040 --> 06:09.520]  a Yocto project which allows us to create a custom Linux distribution which is entirely
[06:09.520 --> 06:15.920]  hardware-agnostic. We just have to recompile everything for another hardware and that allows
[06:15.920 --> 06:21.680]  us to have a full control of package and versions that are on our Linux distribution.
[06:23.200 --> 06:29.360]  That's also really good for cybersecurity because we can easily track and patch every CVE.
[06:30.080 --> 06:36.720]  The Yocto project informed the user if they found a new CVE in one software that it is used,
[06:36.720 --> 06:40.960]  and we can patch directly using the source code because everything is open source.
[06:41.920 --> 06:50.240]  But by doing that, we got into a problem that many industries don't want to deal with the
[06:50.240 --> 06:56.880]  complexity of the Yocto project, and it's much more suitable for them to use, for example,
[06:56.880 --> 07:06.480]  Debian. We created another way of doing the CVE project, but using Debian and a real-time kernel,
[07:06.480 --> 07:14.160]  of course, and an unseable to configure the already created Debian. That's much more
[07:14.960 --> 07:24.080]  useful and easy to use, and we saw that there is a real need here that many industries want
[07:24.080 --> 07:31.600]  to use Debian instead of Yocto. But the two approaches exist today, and any customers
[07:31.600 --> 07:39.440]  that want to implement C-Pass can choose one of them. Okay, so here is our C-Pass project.
[07:41.360 --> 07:45.520]  At the bottom of it, we have the hardware platform, here Intel, but that can be anything,
[07:45.520 --> 07:54.160]  as I said, a Linux real-time kernel above it, and as I said, all the open source software that we
[07:54.160 --> 08:00.720]  want to use and to configure instead of writing it ourselves. All the parts with pacemaker and
[08:00.720 --> 08:07.520]  self are used for distributed file system and distributed VMs between many hypervisors,
[08:08.080 --> 08:14.800]  because the C-Pass project will use a cluster of hypervisors. We don't want all VMs to shut down
[08:14.800 --> 08:19.680]  if an hypervisor is dead. We want to replicate them and relaunch them immediately, automatically.
[08:21.520 --> 08:28.240]  OpenVswitch is used for controlling switch, Internet switches automatically. Of course,
[08:28.320 --> 08:32.800]  we don't want someone to come in the substation every time we need to do changes,
[08:32.800 --> 08:39.280]  so we use OpenVswitch. This comes with the software-centric approach, as I described before.
[08:39.280 --> 08:47.360]  DPDK is basically hardware acceleration for OpenVswitch, let's say that, and of course QMU and
[08:47.360 --> 08:54.320]  KVM, which is the basic couple for virtualization in Linux. So this is our C-Pass project.
[08:54.320 --> 09:03.600]  Two things are important here. First, that the C-Pass project in itself doesn't have any
[09:03.600 --> 09:09.280]  software itself. It is used to configure all these already existing software and use it
[09:09.840 --> 09:16.320]  to benefit from them. And on top of the C-Pass project, we will have all the VMs, and for
[09:16.320 --> 09:26.240]  example, every industrial we work with, and if one day we want to choose to change a piece of
[09:26.240 --> 09:33.680]  hardware because it's deprecated, we just have to shut down the VM, call someone else, and let it
[09:33.680 --> 09:40.880]  write software that will go in this VM. The changing of the piece of hardware will not
[09:40.880 --> 09:47.760]  interfere with the rest. This is a basic idea, basic innovation, first innovation of the C-Pass
[09:47.760 --> 09:58.160]  project. Okay, now I have described the project. I will go a bit more technical and speak about
[09:58.160 --> 10:04.240]  C-Pass testing project. How do we test an open-source project when everyone can write
[10:04.880 --> 10:12.960]  and can propose progress, and how do we write, how do we test a project that doesn't have any
[10:12.960 --> 10:21.120]  software in it, that use, that propose a mainframe but doesn't have any software, because all the
[10:21.120 --> 10:28.560]  software will be in the VMs, the customer software will be in the VMs. I will speak here for the
[10:28.560 --> 10:36.640]  Debian version, because it's simpler, but I'll explain it later. As I said, C-Pass must meet many
[10:36.640 --> 10:43.760]  requirements and provide many guarantees, and for that, we launch the CI at every pull request.
[10:44.800 --> 10:49.600]  Someone propose a pull request, it is accepted by a member of C-Pass, and the CI will launch
[10:49.600 --> 10:58.160]  automatically from this pull request. That means the pull request must build, of course, but also
[10:58.160 --> 11:04.400]  all the tests must be successful. All the tests that we wrote later must be successful for the
[11:04.400 --> 11:12.240]  pull request to be merged. That allows us to avoid regression on one part, but also to display all
[11:12.240 --> 11:21.440]  future tests for the parts we haven't implemented yet. And of course, that will be visible for
[11:21.440 --> 11:25.920]  everyone on GitHub, and especially the man that made the pull request.
[11:29.920 --> 11:38.960]  To achieve that, we will generate a test report to display all tests that pass or not. This report
[11:38.960 --> 11:46.880]  will organize all the 1,500 tests we have between categories and between machines, for example,
[11:46.880 --> 11:52.400]  cybersecurity category or real-time category, and all the hypervisors we will have.
[11:53.840 --> 12:00.960]  It will link all tests to requirements. This is especially useful in cybersecurity,
[12:00.960 --> 12:05.840]  because we have a bunch of requirements to meet, and many tests will link to a single
[12:05.840 --> 12:10.320]  requirement, so we can patch them and see them in terms of requirements.
[12:10.640 --> 12:16.000]  That will separate non-regression part on future work part, as I said before,
[12:16.640 --> 12:19.040]  and that will be visible for everyone on GitHub.
[12:23.360 --> 12:30.640]  What tests, let's say now, what tests we write? As I said, there is no software
[12:32.000 --> 12:38.400]  itself in CPAS, because it will be in the VMs. So all customer code is in the virtual machine.
[12:38.400 --> 12:44.480]  We can do any functional testing. This is nonsense here. But what we do want to do is to
[12:45.120 --> 12:50.720]  check unit requirements, for example, for cybersecurity, system-level testing, let's say.
[12:51.840 --> 12:59.440]  I put some example here. For example, we want to test that the key, the RSA key has the right
[12:59.440 --> 13:04.160]  permission for the right users. We want to check that the root password is randomized or is encrypted.
[13:04.720 --> 13:12.880]  We want to check a bash timeout that SSH has some permission and doesn't allow some connection,
[13:12.880 --> 13:21.760]  et cetera, et cetera. So this is very basic testing, no functional long testing, but
[13:21.760 --> 13:30.960]  configuration testing, let's say. To achieve that, we use a software called Kukinya that I will
[13:30.960 --> 13:41.120]  introduce here. This is a Linux firmware validation framework, and that allows us to write human-readable
[13:41.120 --> 13:47.120]  tests. I put an example on the right. This is a Kukinya configuration file, and you can test,
[13:47.120 --> 13:54.800]  for example, if a user exists, if a process is running, if a disk is mounted, if a Python package
[13:54.880 --> 14:00.960]  is here, et cetera, et cetera. So all of this is human-readable and written in a simple text file.
[14:02.640 --> 14:09.520]  Kukinya offers the abstraction necessary and allows us to write ourselves the complicated
[14:09.520 --> 14:16.000]  shell command to test something and something, and we will inevitably forget an option or something.
[14:16.960 --> 14:24.640]  Kukinya is used and most of all used in the embedded world, because it is easily portable,
[14:24.640 --> 14:32.720]  it requires only a shell, not even bash, just a shell. It is written itself in shell, so it
[14:32.720 --> 14:38.880]  doesn't have any compilation or an installation. It's really easy, and it can extract the results
[14:38.880 --> 14:46.800]  for us, either in CSV, simple or in more complicated, for example, XML, with a logging on the number
[14:46.800 --> 14:51.760]  of tests that pass and so on, many information. And of course, it is open source.
[14:54.480 --> 14:59.600]  Okay, now I have described everything. This is a complete CI that we have.
[15:00.720 --> 15:07.520]  A poll request is made on GitHub. All sources are downloaded on a self-hosted runner. We need
[15:07.520 --> 15:14.640]  to host a runner because we have a cluster to build. We can do that virtually. The Debian,
[15:16.080 --> 15:24.000]  all hypervisors, have already a Debian version as operating system. This version is,
[15:24.000 --> 15:31.520]  this Debian is configured by Ansible. We then deploy Kukinya, so the testing process, and
[15:31.520 --> 15:37.200]  all the testing files that I described before, the tests are launched,
[15:38.000 --> 15:43.840]  gathered in a PDF report, which is uploaded, and the link is given on the poll request on GitHub.
[15:45.280 --> 15:55.360]  This is the CI we currently have on CTAS. I will now go on two points of implementation
[15:55.360 --> 15:59.920]  that are interesting in our CI. First of all, as I said just before,
[16:01.040 --> 16:09.440]  all hypervisors have already a Debian version, and they use our operating system Debian.
[16:10.080 --> 16:16.880]  It is already deployed and already set up that allow us to avoid two problems. First of all,
[16:16.880 --> 16:23.040]  the compilation. So there is no compilation in this CI yet. And the flashing problem,
[16:23.040 --> 16:29.920]  which is a very big problem because automatically flashing and rebooting a machine is complicated.
[16:30.880 --> 16:36.480]  With Debian, we do not even think about that. This is already the same operating system,
[16:36.480 --> 16:43.440]  and we will just configure it every time with Ansible. That means we have to control the basic
[16:43.440 --> 16:52.000]  state, the default state of the CI, and that this is done through Ansible, first of all,
[16:52.000 --> 16:57.520]  using idempotency. So just mean Ansible will not do the same thing twice.
[16:59.120 --> 17:04.720]  But it is not really, not totally useful in our case, because some,
[17:07.040 --> 17:12.400]  for example, if we move or remove files, Ansible cannot roll back to the last version. It's not
[17:12.400 --> 17:19.920]  possible with Ansible. So to deal with that, we will shortly, not done yet, we will shortly
[17:19.920 --> 17:26.880]  implement an LVM snapshot of the default Debian. So we create a snapshot of the
[17:26.880 --> 17:33.040]  default state of the Debian before the CI, we launch the CI, and we then roll back with LVM
[17:33.040 --> 17:42.640]  to the default version. This is the ID. Another problem that we encounter with this CI,
[17:43.440 --> 17:50.640]  is that this is a complicated CI. We don't have compilation, but we need to
[17:51.680 --> 17:57.600]  recover our sources, launch test, configure, launch test, gather results, generate report,
[17:57.600 --> 18:04.080]  upload report, and so on. So many complicated things that required the runner to be
[18:06.080 --> 18:11.920]  configured to do all these things. And because the CI will evaluate over time,
[18:12.640 --> 18:19.360]  we don't want it to, we don't want to redeploy the runner or to reconfigure the runner every time.
[18:21.120 --> 18:27.840]  To avoid that, we use a Docker container, and we clone, re-download the sources of the CI,
[18:27.840 --> 18:33.760]  the code of the CI, every time, directly in the Docker container, and launch all the commands
[18:33.760 --> 18:38.880]  in this container. So Ansible is the Ansible command for conceiving configuration, the report
[18:38.880 --> 18:46.320]  generation, the upload of the report are each launched in the Docker container. That allows us
[18:46.320 --> 18:53.440]  not to deal with configuring and downloading package for the runner itself every time we
[18:54.000 --> 19:02.080]  made the CI evaluate. To do that, we also use a small tool called CQFD, which is really useful
[19:02.080 --> 19:07.840]  in our case. This is a simple common line Docker wrapper, but it will allow us to launch
[19:07.840 --> 19:14.480]  commands directly inside the Docker container. It maps the current directory in Docker,
[19:14.480 --> 19:20.640]  it creates a user in Docker with the same username as ours in order to deal with permissions,
[19:20.640 --> 19:27.920]  and so we can recall a CQFD run Ansible command, and it will execute Ansible in the
[19:27.920 --> 19:36.160]  Docker container, really useful in our case. Okay. Before the end of this presentation,
[19:36.160 --> 19:43.840]  I will talk about a bit about future works and what we will do later. First, as I said,
[19:43.840 --> 19:50.160]  this is a CI for the Debian version, because it was simpler, and we want to create the same
[19:50.160 --> 19:56.960]  for the Yocto version. Many problems with that. First, Yocto has a very complicated compilation
[19:56.960 --> 20:04.000]  and a really long compilation, so we need to deploy other runners to do that, and maybe handle
[20:04.000 --> 20:11.120]  concurrency problem. The machine needs to be flashed every time. We cannot, as with Debian,
[20:11.120 --> 20:17.680]  roll back to an old snapshot. This is not possible. The flashing problem is really
[20:17.680 --> 20:26.160]  difficult. We already tried to use a PXE, but that doesn't work. That's absolutely not hardware,
[20:26.160 --> 20:32.240]  hardware that's really dependent of hardware, and that doesn't work every time. We have two
[20:32.240 --> 20:38.640]  ideas to deal with that. Maybe configure with an update mechanism and consider the new version
[20:40.560 --> 20:48.560]  that has to be flashed as an update that can do job, or with a USB gadget to present the new
[20:48.560 --> 20:56.800]  version with a virtual USB key, as it was with a laptop and a real USB key that would plug in it.
[20:56.800 --> 21:02.000]  That is possible, but we don't have a configuration yet, and we are not sure which
[21:02.000 --> 21:08.800]  solution we will choose. The other thing to do is to run longer tests. I didn't talk about real-time
[21:08.800 --> 21:16.800]  tests here, or latency tests, because it is a really long test, many days. We cannot launch it
[21:16.800 --> 21:23.200]  at every request, but we have to launch it at every release. That's the thing we will do in the
[21:23.200 --> 21:31.840]  future, and to certify, to demonstrate that we met the requirement. That will have the form of
[21:31.840 --> 21:38.240]  this graph, for example, with all the measures we do, the maximum latency we have, the median latency
[21:38.240 --> 21:47.520]  we have, and so on. That will be launched at every release of CPAS. Thank you for hearing me during
[21:47.520 --> 21:54.160]  this presentation. If you want to experience some more with CPAS, it is open source, so you can go
[21:54.160 --> 22:00.960]  on the GitHub pages of CPAS. There are also some other conferences about CPAS available on YouTube,
[22:01.680 --> 22:09.360]  and for myself, I am already open to answer all your questions already.
[22:18.240 --> 22:28.080]  We already have a question from Markus. Markus says that it looks very similar to him,
[22:28.640 --> 22:36.560]  and because he is a maintainer of OpenHappian, there is extension to OpenHapp, and it chooses
[22:36.560 --> 22:42.560]  essentially lots of advanced batch scripting on top of Debian, and they also roll out a virtual
[22:43.040 --> 22:52.640]  VM in a Docker container in CI on every pull request upload. Can you say something about that?
[22:53.520 --> 23:02.960]  I read that. I don't know the software that you are talking about, but that's interesting,
[23:02.960 --> 23:14.960]  because essentially, batch scripting on top of Debian is a sort of trick we found to make the
[23:14.960 --> 23:20.560]  CPAS project with Debian, because originally, the only CPAS version was with Yocto, which
[23:20.560 --> 23:28.000]  offers much more configuration over the Linux system you have, but as I said, we found out that
[23:28.080 --> 23:35.280]  many constructors and many industrials don't want to dive into the complexity of Yocto, and so
[23:36.160 --> 23:42.240]  here we are creating many batch scripts to configure Debian properly, especially for
[23:42.240 --> 23:49.440]  cybersecurity questions, which are a bit complicated with that. But if you go to the GitHub pages of
[23:49.440 --> 23:56.640]  the CPAS project, you will see that many, many problems we have with Debian are not there with
[23:56.640 --> 24:10.880]  the Yocto version. Okay, so I don't see any other questions for now, but I just said two or three
[24:10.880 --> 24:18.320]  things that are not technically correct, and I just thought of that after I uploaded the video,
[24:18.320 --> 24:28.640]  but I said that there is no software embedded in the CPAS project, which is not technically true.
[24:29.360 --> 24:36.080]  The software we want to reach, so the interesting software will be the software that controls the
[24:36.080 --> 24:41.040]  actuator of the captors, and that will be in the VMs, in the virtual machines, of course,
[24:41.040 --> 24:48.240]  but we do develop some software in the CPAS project. For example, we have a virtual machine
[24:48.320 --> 24:55.520]  management system written in Python, which is just to manage our virtual machine in the cluster
[24:56.560 --> 25:00.240]  in a simpler way, and we do develop that in the CPAS project.
[25:03.440 --> 25:14.560]  I also said that the Yocto project doesn't use unseable, which is also not true, so the
[25:14.720 --> 25:19.360]  Yocto version of the CPAS project doesn't use unseable. Of course, we use it because we want to
[25:19.360 --> 25:25.440]  configure a cluster, so unseable is used just to do the network configuration of the cluster,
[25:26.080 --> 25:32.160]  because in the Yocto project, we already have configured the Linux system, because we have
[25:32.160 --> 25:38.480]  built it the way we want. In the Debian version, this is not possible, we just plug a Debian USB
[25:38.480 --> 25:45.280]  key with a real-time kernel, and that's all we can do, so everything else has to be configured
[25:45.280 --> 25:52.720]  through unseable. It's a much messier, a much bigger unseable repository for the Debian version.
[25:53.680 --> 26:00.960]  Yes, I think Marcus knows what we are talking about.
[26:11.280 --> 26:14.880]  And yes, for the last correction I can add,
[26:15.280 --> 26:26.480]  we have thought about how to implement the CI for the Yocto version and the problem of flashing
[26:26.480 --> 26:34.480]  the machines, and we found that the simpler way to do that is just to use an update mechanism.
[26:34.480 --> 26:40.400]  So we use software updates in the Yocto project, in the Yocto version, with two double banks,
[26:41.280 --> 26:48.160]  so we can just upload a new version of the C pass projects, as if it was a new version,
[26:48.160 --> 26:54.640]  even if it is just another pre-request to test. That works pretty well, and that avoid flashing
[26:54.640 --> 27:09.040]  problems, PXE problems that we have. Okay, so if there isn't any questions anymore, I say goodbye,
[27:09.040 --> 27:15.680]  and you can see me, I will be in the first dem this afternoon, you can solve for the pink
[27:16.960 --> 27:26.880]  sweat and of course the hat with the text on it, I think you can find me there. Thank you.