[00:00.000 --> 00:11.120] Hello, like I said, it will be digested because this topic is a little bit, let's say, complex [00:11.120 --> 00:17.880] at one side, but from the other side, it will help you to save a lot of time and, of course, [00:17.880 --> 00:18.880] money. [00:18.880 --> 00:19.880] My name is Alexander Dubrikov. [00:19.880 --> 00:26.720] I am CTO of QXAP Company with the Gas Resolverance Manganee, we build with QXAP BE. [00:26.720 --> 00:33.000] So I'm an open source enthusiast, I'm involved in many open source projects, Camelia, Frisvich, [00:33.000 --> 00:38.120] OpenZeef, Asterisk, and many, many, many else. [00:38.120 --> 00:45.280] So QXAP BE is a company based in Amsterdam, it's a company which is behind Homer and Hapk [00:45.280 --> 00:51.960] and a lot of other projects which helps you to monitor systems. [00:51.960 --> 00:58.360] Of course, we like to make open source projects and a lot of our projects are open sources [00:58.360 --> 01:00.840] with good license. [01:00.840 --> 01:02.760] So what's the problem we have? [01:02.760 --> 01:08.960] Yeah, somebody of you knows about Homer and Homer is a very good, great tool to make monitoring [01:08.960 --> 01:15.560] system to store all data, use e-messages, also some local information, but sometimes [01:15.560 --> 01:20.280] you also need to store some metrics, some statistics, and therefore you use Prometoids [01:20.280 --> 01:25.640] or you use InfluxDB for time series, you use Elasticsearch to store some logs information, [01:25.640 --> 01:32.280] some CDRs, and at the end it's mess, because it's so many staff which confuse you, you [01:32.280 --> 01:39.480] have to maintain, you have to spend all of time, and in our company we decide what we [01:39.480 --> 01:44.640] can do here, how we can help you guys to make your life easier. [01:44.640 --> 01:52.360] Then we step back, we view this problem on a different angle, and we decide to make a [01:52.360 --> 01:55.000] new application, it's called Quirin. [01:55.000 --> 01:56.000] What is about Quirin? [01:56.000 --> 02:02.440] Quirin is normally its collector, which you have already Grafana, you have already Prometos, [02:02.440 --> 02:08.800] you have already Telegraph, which will send the data in special formats, and we created [02:08.800 --> 02:15.360] this application which can read all these formats and store data for you. [02:15.360 --> 02:22.760] So of course you can ask what about Homoids, Homoids can also send some information, some [02:22.760 --> 02:27.800] SIP information to Quirin, and this can be always stored in the database. [02:27.800 --> 02:32.480] But it's not only about Homoids, it's also about these different statistics, what you [02:32.480 --> 02:39.840] can receive from your agents, from Prometos agents, from InflexDB, etc. [02:39.840 --> 02:47.240] Everything is stored to Quirin, and we created this engine and stored it in the database. [02:47.240 --> 02:53.160] So what we did better than in Homo, we wrote a great documentation. [02:53.160 --> 02:59.600] So if you go to this website, I trust you guys, this was the number one point, once we [02:59.600 --> 03:05.640] started the project, we wrote a great documentation, you can go to our website and you will see [03:05.640 --> 03:12.760] all steps, how you can install, how you configure without headaches. [03:12.760 --> 03:17.400] So at the end, you know what SIP is normally for us, it's just an event. [03:17.400 --> 03:21.920] But what about VEPRTC, it's also an event, because we have different platforms, it can [03:21.920 --> 03:27.320] be genres, it can be even free switch and so on, they generated own events in JSON format [03:27.320 --> 03:29.480] and how we can collect it. [03:29.480 --> 03:34.640] At the end, we decide, so we have only metrics, we have logs, and we have traces. [03:34.640 --> 03:40.120] At the end, it's all our information, what we generated in voice over IP stacks, it's [03:40.120 --> 03:42.560] related only to these three categories. [03:42.560 --> 03:45.240] And of course, it can be generated from different sites. [03:45.240 --> 03:52.040] So if we're talking about working Prometos, Elasticsearch, it's already existing with agents [03:52.040 --> 03:58.680] which you probably guys already use, and you can generate this data and send to querying. [03:58.680 --> 04:04.120] How you can read it, if you use Grafana, probably it's everybody, if you use Grafana, Grafana [04:04.120 --> 04:09.920] has already native plugins for local, from KL, temple, API, and we support it as well. [04:09.920 --> 04:16.800] So you should not install any additional plugins, it works from the box. [04:16.800 --> 04:24.760] We have very cool query stuff which you can extract any data from querying, we'll show [04:24.760 --> 04:25.760] them. [04:25.760 --> 04:29.600] And of course, you can use any agents what you already exist, this can be Grafana agents, [04:29.600 --> 04:34.120] this can be lockstash, vectors, telegraph, and so on, so on. [04:34.120 --> 04:40.480] Also, for you, make your life easy, we develop our data explorer, which is already integrated [04:40.480 --> 04:44.600] inside of querying, you can use similar to Grafana. [04:44.600 --> 04:51.080] And what is very important, we already have a lot of deployment, and it's some big gaming [04:51.080 --> 04:57.200] providers, enterprise solutions, and this query, you can use it also for EoT, it's scalable [04:57.200 --> 04:59.560] very, very, very well. [04:59.560 --> 05:04.200] Now working samples, like I said already, you have these agents, so I don't have too [05:04.200 --> 05:12.120] much time, it's an industry standard which you can use its Prometos API, it can be influx [05:12.120 --> 05:15.160] CDB insertions, Temporal API, etc. [05:15.160 --> 05:25.520] We insert this data to stacks which reads from API points, it can be on open telemetry, [05:25.520 --> 05:28.000] it can be local elastic search, etc. [05:28.000 --> 05:33.960] It goes to different basket and we insert to database. [05:33.960 --> 05:40.040] It's like back end, like database we use Klikals, Klikals is very, you probably heard [05:40.040 --> 05:46.520] about Klikals, Klikals is very, very, very performant database, it can be scalable very [05:46.520 --> 05:51.400] well linear and horizontal. [05:51.400 --> 05:55.600] You can use a lot of some features like UDF functions, etc. [05:55.600 --> 06:04.640] You can use also S3 storage if you would like to save your money and if you use AWS or two. [06:04.640 --> 06:06.240] And how you can read data? [06:06.240 --> 06:12.440] You can read data using this API, it's LOKL, do you know guys what your LOKL is Prometoiskl? [06:12.440 --> 06:13.440] No? [06:13.440 --> 06:14.440] Okay. [06:14.440 --> 06:22.520] LOKL, Prometoiskl is special languages which develop in this company is Prometoisk and [06:22.520 --> 06:29.120] it helps you to make some complex statistics, some complex search for logs and information. [06:29.120 --> 06:33.400] So it's not like before we use all select from blah, blah, blah, but it's very, very, [06:33.400 --> 06:34.400] very limited. [06:34.400 --> 06:39.920] So very for the guys from Influx, from Prometoiskl, they develop this promkl language which is [06:39.920 --> 06:45.760] very, very flexible and you don't have any limits to do any queries and how it works [06:45.760 --> 06:46.760] I will show you. [06:46.760 --> 06:56.560] For example, you store data in query and you set labels, how you insert data, you set labels, [06:56.560 --> 07:01.480] it's free switch, it's generated fingerprints and in that query you just say, ah, show me [07:01.480 --> 07:06.440] everything what is related to free switch and it's very, very fast, lightning fast, [07:06.440 --> 07:08.520] display your data here. [07:08.520 --> 07:16.360] Second one, what to do if you store Zip messages, you can also set, ah, pipe with results and [07:16.360 --> 07:21.320] extract any type of fields from Zip messages, it can be airport, it can be callity, whatever [07:21.320 --> 07:23.000] you want. [07:23.000 --> 07:27.960] And Janus, for example, Janus generated a lot of events which we can store almost in [07:27.960 --> 07:37.760] query and we can extract RTT in labels from this Janus event, but it's not the last. [07:37.760 --> 07:42.760] Now what we can do with RTT events, you can just make, unrape and put this information [07:42.760 --> 07:48.320] to basket for 10 seconds and immediately from this event information you generate charts, [07:48.320 --> 07:50.840] so it's converted automatically. [07:50.840 --> 07:56.120] Now you can also do exactly same for elastic storage input or RTT, roundtrip, so information [07:56.120 --> 08:00.680] what your switch is generated, you can convert any information what you already stored in [08:00.680 --> 08:04.160] database to charts. [08:04.160 --> 08:11.160] What about HEPLIFI, Homer, you can also set, ah, let me check all method invites and put [08:11.160 --> 08:19.960] to basket for one minute and display what's, how it looks in time series. [08:19.960 --> 08:25.680] What about RTTCP, you can also send RTTCP information, you can display most data packet [08:25.680 --> 08:27.160] to us and so on. [08:27.160 --> 08:34.080] You can send any HEP statistics and display it automatically in query. [08:34.080 --> 08:41.440] You can do same with Spromkl, HEPMAP, so about OpenTelemetry, OpenTelemetry it's de facto [08:41.440 --> 08:47.560] standard which guys from next room developed and it's already used in many applications. [08:47.560 --> 08:52.160] OpenTelemetry it's just internal tracing, you can use our special libraries, you connect [08:52.160 --> 08:57.240] it to your application and it will trace all your functions, execution time, these ideas [08:57.240 --> 09:04.320] and you will send this open traces to query and you can display how many seconds, microseconds [09:04.320 --> 09:10.240] your function execution was taken, what's plugin was, how much time it took, plugin [09:10.240 --> 09:11.760] usage, etc. [09:11.760 --> 09:21.320] And this is how you can handle and see in, for example in this Janus, it's offer, offer [09:21.440 --> 09:25.600] how many microseconds it takes, how many ises taken, etc. [09:25.600 --> 09:33.200] This is exact, it's not only about Janus but also you can make, you can also enable [09:33.200 --> 09:42.000] OpenTelemetry stuff and you will see how many microseconds, milliseconds takes your query. [09:42.000 --> 09:45.960] In service graph you can also make automatically display, it's generated automatically you [09:45.960 --> 09:54.200] can check rate and automatically it's generated query and display charts for each node. [09:54.200 --> 09:58.880] If you don't like OpenTelemetry and you don't know how to, let's say, connect it to external [09:58.880 --> 10:05.520] library we can use eBPF, eBPF is special functions in kernel, you can compile special [10:05.520 --> 10:13.080] model which will trace all your functions and generate all traces sent to our collector. [10:13.080 --> 10:18.920] Without eBPF we will display, we will present in Berlin for Camelio how you can use OpenTelemetry [10:18.920 --> 10:23.920] and make performance optimization. [10:23.920 --> 10:30.800] So Janus, we created for Janus, we created special application which is called JAWS. [10:30.800 --> 10:36.800] It's a web socket collector which collect all information from Janus and we converted [10:36.800 --> 10:39.400] all data to OpenTelemetry. [10:39.400 --> 10:44.800] And we can display this data like media, okay, okay, next, next, next. [10:44.800 --> 10:50.400] It's ice failure for Janus, the same information you can exactly display here. [10:50.400 --> 10:57.320] You can do aggregation type in Janus telemetry, see which nodes Janus proceed, etc. [10:57.320 --> 11:02.800] Also very important you can set any alert on any metrics what you send to query using [11:02.800 --> 11:06.320] alert manager, what is very important. [11:06.320 --> 11:10.520] You can even use fraud detection if you want. [11:10.520 --> 11:20.800] So last topic, Open 5G, probably guys you saw yesterday we did some hack. [11:20.800 --> 11:28.480] Open 5G stacks, it's stacks which does EMS and all this stuff, it includes Camelio, some [11:28.480 --> 11:29.640] TPA engines. [11:29.640 --> 11:38.720] In 5 minutes we installed HEPLIFI agents and we sent information to querying, exactly using [11:38.720 --> 11:39.720] same stuff. [11:39.720 --> 11:46.240] This is device how it looks like, so it's bad quality but at the end it's small mini-computer [11:46.240 --> 11:57.200] which has Docker which starts all EMS stacks and we, yeah, using querying, we trace everything. [11:58.200 --> 12:07.160] Giovanni sent in Facebook this post, we did it exactly yesterday and it works very, very [12:07.160 --> 12:14.480] well and when we did some test he connected to Vicentene, go to another room because Vicentene [12:14.480 --> 12:20.040] has only 0.2 watts here I think and goes in the next room and immediately it was displayed [12:20.040 --> 12:23.160] what packet was here because his signal is dropped. [12:23.160 --> 12:28.920] So if you support, if you like open source and if you like this project, star us, so [12:28.920 --> 12:33.640] it's cost nothing but the process like a cookie. [12:33.640 --> 12:39.920] So of course sponsor open source projects because without open source our life will be [12:39.920 --> 12:45.720] more difficult and we have this block querying there, we have a lot, our team wrote a lot [12:45.720 --> 12:52.120] of nice documentation how you use querying, how you can integrate your traces, everything. [12:52.200 --> 12:55.560] So it's all examples, all good stuff inside. [12:57.160 --> 12:58.160] Yeah, that's it. [13:03.920 --> 13:09.000] Like I said, sorry guys, this topic is very, very, very, very complex but it's, I have [13:09.000 --> 13:10.000] only 15 minutes. [13:10.800 --> 13:13.880] So, yeah, so if you have any questions, go ahead. [13:15.960 --> 13:16.480] Yeah. [13:16.480 --> 13:18.800] It's very something like Joss for Facebook. [13:20.480 --> 13:21.480] Yeah, Joss, it's. [13:23.120 --> 13:27.840] Okay, it's Joss for 3C, theoretically, it's because it's open source, you can adapt events [13:27.840 --> 13:34.280] but this is what I said, it's each application generated own format. [13:34.280 --> 13:37.240] At the end it's a JSON format, so it's JSON events. [13:37.240 --> 13:40.040] You can of course adapt and convert it and send to open telemetry. [13:40.040 --> 13:41.040] Yeah, you can. [13:41.440 --> 13:56.320] Okay, it's about here, you know, so just to repeat, so of course you can send all data [13:56.320 --> 14:03.800] to querying and you can create locks and zip information by using open telemetry stuff. [14:03.800 --> 14:09.040] What we work in this, Camelia for example, to send all this information in open telemetry [14:09.040 --> 14:15.200] you send out using HAP encapsulation and you can collect all locks and information with [14:15.200 --> 14:19.040] your dialogs ID with, for example, for zip, you know, and you can store it. [14:19.040 --> 14:20.520] So you can select, for example. [14:20.520 --> 14:24.720] Yeah, you can select it, exactly in querying, you can click and you will immediately jump [14:24.720 --> 14:33.080] to Homer website, you have HomerView where you can display this in chat, in view, zip [14:33.080 --> 14:34.760] code view or in table. [14:36.320 --> 14:37.600] And the other way around? [14:37.600 --> 14:41.360] Yeah, it's awesome, it works also as well, yeah, it's a round-trip integration. [14:43.920 --> 14:45.240] Okay, thank you.