[00:00.000 --> 00:17.720] So, good doing everyone. Before going to my presentation, I just want to thank Jan and [00:17.720 --> 00:27.200] Nils. I guess these folks are organizing the show at the room for last six years or seven. [00:27.200 --> 00:39.200] So, yeah, today I hope I'm audible and yeah, the network is, yeah, sure. So, today I'm going [00:39.200 --> 00:47.840] to talk about autoscaling feature in Kubernetes and how it can be done if using CADA for RGW [00:47.840 --> 00:54.480] specifically. So, most of the presentation covered Sef and Rook and last presentation [00:54.480 --> 01:01.280] from Gaurav and Alexander was about Rook. So, it's bit advanced topic over that. So, [01:01.280 --> 01:08.280] myself, Jifin Tonythotta and I am working as backend engineer in IBM storage and I work [01:08.280 --> 01:14.400] closely with Rook and Sef community. So, as I mentioned before, like most of the talks [01:14.400 --> 01:20.080] already covered Sef and Rook topics. So, I mostly focus on CADA. So, my section covers [01:20.080 --> 01:27.360] what is CADA and basic CADA concepts. Then, a brief thing about Rook operator and finally, [01:27.360 --> 01:34.680] a demo how the autoscaling works. So, what is CADA? So, in the last presentation, one of the [01:34.680 --> 01:38.840] things Alexander mentioned, you can change the configuration, right, if you are in the [01:38.840 --> 01:44.160] deployment. With help of autoscaling, like, you don't want to change it. Like, it basically, [01:45.040 --> 01:52.320] the autoscaler will find and scale those for you. And, Kubernetes inbuilt have the HPA and [01:52.320 --> 01:59.080] VPS scalers. So, CADA is kind of a bit, what you say, advanced version for the HPA. So, [01:59.080 --> 02:06.160] what CADA does? It makes the Kubernetes even driving autoscaling that symbol. I don't know [02:06.160 --> 02:12.520] if you folks use HPA. So, HPA by default support, CPU and memory kind of scale. Like, if your [02:12.520 --> 02:16.920] port is using most, lot of CPUs or something like that, it will scale or if you use memory, [02:16.920 --> 02:24.200] it will scale. But, it also supports custom metrics. And, CADA, and the one of the issues [02:24.200 --> 02:28.120] with the custom metrics, it does not have a standard version for that. Like, even Prometheus [02:28.120 --> 02:33.040] has an implementation, I mean, Kubernetes has an implementation, but nothing is standardized. So, [02:33.040 --> 02:38.720] this is where CADA comes. And, it started as a partnership between Microsoft and other that, [02:39.000 --> 02:45.840] and it was donated to CNCF. And, during the two-point-o version, like, it may be, [02:45.840 --> 02:51.560] go through a major design change and all that stuff. And, last couple of years back, like, [02:51.560 --> 02:58.240] it got become a, like, incubation project. And, currently, like, it's on the 2.9 release, [02:58.240 --> 03:03.320] like, that's the latest version. And, I hope the next major change will happen in the third [03:03.360 --> 03:10.280] version coming years. Now, going a bit about CADA concepts. So, as I mentioned, [03:10.280 --> 03:19.040] it automatically scales the Kubernetes resources, like deployments, or stateful sets, [03:19.040 --> 03:25.280] like replica, or even customer sources. Then, it has, like, inbuilt 50-scalers, [03:25.400 --> 03:32.600] like, like a plug-in which we can attach. Like, for example, you have Prometheus, [03:32.600 --> 03:40.160] or like Kafka, and RabbitMQ, AWS, and Azure, all those big players are there. Then, [03:40.160 --> 03:45.480] other thing is, it just scales the resources. It does not manipulate your data. Then, [03:45.480 --> 03:52.840] the scaling is done on the event basis. So, it doesn't want anything to do with your data. [03:52.880 --> 03:57.560] Like, it won't manipulate your, like, your side unit or something like that. It just scales [03:57.560 --> 04:03.480] based on the events which it's facing. And, you can install CADA via OLM or Helm, whatever. [04:03.480 --> 04:12.640] Now, this is basic architecture with CADA. So, CADA just enhances the HPF feature of the [04:12.640 --> 04:21.440] Kubernetes. So, you need to have the Kubernetes cluster. Then, there's an, on the bottom side, [04:21.440 --> 04:25.960] you can see an external source, like, which you, which you have given the information about, [04:25.960 --> 04:34.800] like, what resources, or, like, what even you need to scale. And, how the, like, how the deployment [04:34.800 --> 04:38.400] will scale, it based on a customer source, not a scaled object or a scaled job. Now, [04:38.400 --> 04:45.200] the scaled object will be created by CADA. And, CADA has, like, the three components, [04:45.200 --> 04:48.960] like, one is a scalar, like, as I mentioned, like, there are a lot of scalers, like, it's a plug-in [04:49.160 --> 04:55.440] for CADA. Then, you have the CADA controller, basically, manage the CADA, I mean, CADA services [04:55.440 --> 05:05.120] and demons. Then, you have the metrics adapter. So, the custom, like, in the case of HPA, the custom [05:05.120 --> 05:09.840] metrics is driven by a metrics adapter, like, you need to provide a metrics adapter for the [05:09.840 --> 05:16.440] custom scaling. So, CADA itself brings up a metrics adapter for that. Now, you have your [05:16.520 --> 05:23.320] workload, and after that, based on the events, it may increase your deployment or it may decrease [05:23.320 --> 05:28.680] your deployment. And, it does it with the help of HPA, like, even though you define a scaled object [05:28.680 --> 05:34.840] or a scaled job, then, in general, it creates an HPA. Now, one of the differences is that, like, [05:34.840 --> 05:42.000] it can scale down to zero, like, normally, with HPA, the scaling starts on one or, like, and it [05:42.040 --> 05:49.240] ends on the maximum which you have defined. Then, yeah, that's all. And, the metrics adapter [05:49.240 --> 05:54.880] is covered. So, yeah, the custom metrics is provided by the auto-scaler, like, provided [05:54.880 --> 06:01.600] to auto-scaler by this metrics adapter. Now, I am just giving an example about scaled objects, [06:01.600 --> 06:08.640] because that's what I'm going to cover into this presentation. So, basically, it has a name, [06:08.760 --> 06:14.840] the metadata information. Then, the, you can see the target of, which will mention what type of [06:14.840 --> 06:20.880] resource you want to scale. Like, by default, it will be deployment. But, you can also add the [06:20.880 --> 06:27.840] replica sets or stateful resources. Then, if you have a customer resource with a scale defined, [06:27.840 --> 06:33.720] you can also define that. So, these, then, you need to mention the type of resource. Like, [06:33.720 --> 06:38.200] by default, if you give a name, it will be deployment. Then, you can mention about the [06:38.200 --> 06:43.880] Plick account. Then, you can give Triggers. So, Triggers is an event which, the based on [06:43.880 --> 06:49.320] scaling will happen and you can define multiple Triggers as well. So, so far, any questions or, [06:49.320 --> 07:01.720] like, okay, I'll move forward. Now, I am just mentioning a few, most of them I already covered, [07:01.720 --> 07:07.960] like, CADAP features. It can scale down to zero if you want. Another part is, like, if there's a [07:07.960 --> 07:13.480] failback, the Plick account, if something happens to the cluster, you can failback to one number. [07:13.480 --> 07:19.880] So, it's not min or max. You can define a failback value. Maybe, say, your min is three and your [07:19.880 --> 07:24.520] five is your max and you can set a failback one, in some of the cases, if you want to failback two. [07:25.320 --> 07:29.960] It can really, say, like, pose for autoscaling. Like, you can start and stop. Like, you don't [07:30.040 --> 07:35.640] delete those sources. You can just, I mean, you don't want to delete the scaled object or something [07:35.640 --> 07:41.240] like that. You can keep those sources and you can just do a pose. Then, CADAP, by default, can [07:41.240 --> 07:49.080] expose the Prometheus metrics in this adapter as well as the Kafka events. And one thing is, [07:49.080 --> 07:53.640] like, you can also use secure connections, like, potentials in things, like, it can be [07:54.200 --> 08:00.760] defined by another subclass or, like, a subsection in the scaled, I mean, [08:02.120 --> 08:07.240] in the scale, like, in the Prometheus, I mean, the type, in the scalar type, you can mention a [08:07.240 --> 08:12.680] subclass about the trigger authentication, which you'll refer, like, how you can authenticate [08:12.680 --> 08:19.480] with the server. Now, today, in the latest version, even, you can have the events or [08:19.480 --> 08:25.480] metrics from the GRPC or, like, from the other JPA, but I have never tried or I have never used [08:25.480 --> 08:34.440] those things. Now, coming to RGW. So, RGBL case is very simple case for CADAP. First of all, [08:34.440 --> 08:40.280] it just proves through the Wootkend RGW stuff, like, so, in the last presentation, they mentioned [08:40.280 --> 08:45.320] Wootkiston orchestrator, which conducts the self storage and it simplifies the deployment and [08:45.320 --> 08:55.160] management services for the self cluster. Now, here, for RGW, like, the access can be given as an [08:55.160 --> 09:01.720] OBCs, like, something similar to PBO PVC for the fly and file and block. Other source is known as [09:01.720 --> 09:08.760] self-object storage. So, OBC, like, you will get a bucket, but in case of self-object storage, like, [09:08.760 --> 09:14.520] you will get the entire user credential, too. So, you can create multiple buckets and [09:14.520 --> 09:19.640] all those features can be done. Then, other part is, like, Wootk also have a service monitor. So, [09:19.640 --> 09:23.560] if you're familiar with Pomerateus and all, like, if Pomerateus want to fetch the metrics from your, [09:24.760 --> 09:30.520] so, I mean, your DMN, like, you need to have a service monitor. So, what the service monitor does, [09:30.520 --> 09:34.520] the self-manager supports the metrics and this metrics will be passed to the Pomerateus service [09:34.520 --> 09:39.640] with the help of the service monitor. Now, for my test case, I am using HS bench. So, it's a [09:39.640 --> 09:45.400] performance-evaluating tool for S3 workloads. So, yeah, that will be tool, like, that will [09:45.400 --> 09:54.280] be S3 client, which I will be using for HW. Now, yeah, so, in the demo, I cover, like, [09:54.280 --> 09:59.960] the Pomerateus scale I will be using and the self-cluster will be already deployed by a hook [09:59.960 --> 10:06.440] and HW is configured. Also, the Pomerateus server is up and have defined the Pomerateus, [10:07.000 --> 10:11.720] like, the requirements for service monitors and all those stuff. Then, I need to define a scaled [10:11.720 --> 10:18.600] object so that my HW can scale. And the scaling is based on the metrics provided by the manager. [10:18.600 --> 10:24.280] So, for HW, most of the metrics are related to performance count. It's based on the how many [10:24.280 --> 10:28.760] requests we are receiving on the HW server. It's nothing depends on the backend or something like [10:28.760 --> 10:33.720] that. So, that's one thing which we may need to change. But currently, it's like a web server, [10:33.720 --> 10:37.720] like, when you are getting the request based on request a lot, like, the scaling will happen. [10:39.080 --> 10:53.480] Now, I will go to the demo. So, okay. I hope it's visible. So, I am running a mini cube cluster [10:55.960 --> 11:00.280] for my demo purpose, like, everything is up and running. And I already installed [11:04.280 --> 11:13.640] the look cluster. So, you can see look operator is running. Then, [11:14.600 --> 11:19.640] HW is also running. Currently, I have only one HW on my cluster. Then, [11:20.600 --> 11:27.720] this is the service monitor, like, the format is, which look deploys. Then, if you check the services, [11:28.680 --> 11:34.360] there are two HW service, like, one is the internal service, which can be accessed for the [11:35.240 --> 11:40.360] humanities workloads. And for my HS1, I need to expose the HW service. Hence, [11:41.080 --> 11:46.120] I am using the channel HW service. It's just, it's an old port, like, it will just expose the [11:46.120 --> 11:54.040] HW service on the remission. Then, yeah, there is also the Udpometheus service as well. [11:54.280 --> 12:01.560] So, I have created SF subject show user, and I need to pass these credentials for my demo, [12:01.560 --> 12:09.720] I mean, for my workload. This is the Pometheus operator running on the default name space. [12:10.680 --> 12:17.400] Now, I will install the CADA via Helm. It's deployed. [12:31.000 --> 12:34.680] Just checking the ports are up and running. It's nothing fancy. [12:35.000 --> 12:42.600] So, everything is up and running. Now, I need to define the scaled object to source for HW. [12:44.600 --> 12:52.920] Sorry. I'm just taking the Pometheus web's console UI. So, I am doing the scaling based on the [12:53.560 --> 12:57.400] request, like, SFHJ request. So, just showing the current value. That's it. [12:58.120 --> 13:04.520] Yeah. Now, I need to define the scaled object. So, [13:19.480 --> 13:23.800] so, this is the, in the GitHub repo, like, this example is the lamified is the, [13:24.360 --> 13:30.120] so, I have given the name for my scaled object, and I am doing the scale, like, on the scaling, [13:30.120 --> 13:36.200] on the deployment, on the SF object show, like, HW show, set the minimum of pick account 1 and [13:36.200 --> 13:41.480] maximum 3. Then, this is the Pometheus endpoint, I mean, the metrics endpoint, which I need to [13:41.480 --> 13:45.560] fetch the metrics, and this is the metrics name which I am giving, like, so, this is based on, [13:46.440 --> 13:52.920] this is a, like, definition for the triggers, nothing else, and the threshold value is 10. So, [13:52.920 --> 13:57.800] basically, it's 10 million requests, not normal 10. I mean, [14:10.360 --> 14:15.400] now, the scaled object is created. If you check for the scaled object and [14:16.120 --> 14:23.800] HPA, you can, we have defined the scaled object, and HPA is automatically created. So, [14:23.800 --> 14:29.960] this is a scaled object, and yeah, it is not active, that's why the state is unknown, [14:30.760 --> 14:36.920] and you have not defined the fallback, that's why, but it's ready for scaling. If you go for the, so, [14:39.560 --> 14:42.600] for the scaled object, and HPA will be triggered internally. [14:46.120 --> 14:53.160] So, this is the current load on the RW deployment, and current status, like, [14:53.160 --> 14:55.480] you can see them in NMAX and the pick accounts. [15:00.120 --> 15:05.480] Now, I am doing a watch on the ports, and the scaled object and HPA. [15:16.360 --> 15:25.800] So, yeah. Now, I am triggering the load. So, for that, I need to fetch the [15:25.800 --> 15:36.200] credential from the subfuser. So, just getting the access key and secret key. [15:36.600 --> 15:56.600] So, I am triggering the HS bench. So, HS bench has the access key, so you can [15:56.600 --> 16:01.160] see the endpoint details. Then, currently, I am running a load of, like, [16:02.120 --> 16:08.920] so, this is, 1 mb is the size of the object, and 1 dollar tree, I mean, 1 hierarchy. [16:08.920 --> 16:11.960] This is 10 clients it will be running, and 1000 objects. [16:16.840 --> 16:22.680] This is triggered. Now, on the left-hand side, like, [16:22.680 --> 16:27.400] still the, like, it may take some time to flatten that side, like, the watch part. [16:36.680 --> 16:41.400] So, you can see the load is increasing. So, it is still not reaching that 10 million, as I [16:41.400 --> 16:46.520] mentioned before. Like, if it costs 10,000 or something like that, then it triggers the scaling. [16:47.320 --> 16:49.240] So, still I have one port. [16:53.640 --> 17:05.160] Yeah. Now, it costs the limit. So, if you look closely, like, so, there is a kind of, like, [17:05.160 --> 17:09.720] what you said. School of period or something like that, it will wait for some time, [17:10.360 --> 17:13.720] 90 seconds or something like that before scaling, then only the scale will happen. [17:14.440 --> 17:20.520] As you can see, like, after costing the limit, it just scaled one. Now, it will scale again, [17:20.520 --> 17:26.280] still increasing the two ports is not satisfying the request. So, it scaled again. Now, the workload [17:26.280 --> 17:33.880] become bit less. Still, it is about 10 million requests. So, now, if I execute here on the [17:33.880 --> 17:40.120] probability server, like, you can see three requests, three instances providing the same request. [17:47.400 --> 17:48.840] Another fourth port is up. [17:53.160 --> 17:58.600] That is a four instances, like, if I go back to the terminal, yeah. [17:59.480 --> 18:06.360] So, you can see a bit decrease in the load, but, yeah, the load is never become less than 10. [18:06.360 --> 18:17.080] That says it was still increasing. So, yeah, I guess that is all I have and I do not have the, [18:17.880 --> 18:22.520] like, the scaling down part, like, before that it was taking a lot of time. So, [18:23.400 --> 18:27.960] that is it. Any questions or, like, [18:36.440 --> 18:39.400] what is the use case of scaling down to zero? [18:43.560 --> 18:48.200] So, I have a question. The question is, what is the use case of scaling down to zero? [18:49.160 --> 18:52.840] Maybe you can save those sources or see if you are not using that service up. [18:52.840 --> 18:57.560] It depends on you, like, if you want to defend zero, like, then it will, if it is ideal, [18:57.560 --> 19:02.920] it will scale to down to zero. That is it. But will it then scale up fast enough? [19:02.920 --> 19:10.200] Yeah, if it causes the threshold is coming up, I have played zero. So, I do not know whether, [19:10.200 --> 19:14.840] whether RGW will have bug because if you scale down, then the server is not the right for RGW. [19:15.720 --> 19:20.680] So, I am not sure whether it will work for RGW. But, yeah, majorly, it will save those sources, [19:20.680 --> 19:25.160] if you have anything else. Yeah, sure. [19:25.160 --> 19:29.400] With something like that, work for scaling OSDs? [19:29.400 --> 19:33.320] OSDs, I am not quite sure because OSD has the dependency of hard disk. [19:33.320 --> 19:39.160] So, I do not know how it will play in case of OSDs. But it can obviously work for NFS or it [19:39.240 --> 19:41.480] can work for MDSs. [19:41.480 --> 19:46.520] Scaling OSDs is very expensive. You need to move better. It does not fit this, [19:46.520 --> 19:52.280] this method of scaling up and down or momentarily, you think. If you already moved it to OSD, [19:52.280 --> 19:54.600] you need a big event to move it down. [19:54.600 --> 19:58.280] Just whether that would be more of an upgrade for the server. [19:58.280 --> 20:06.200] I think the usual argument for OSDs is that you have to have hard drives, like, in storage, [20:06.680 --> 20:10.840] to put OSDs on, and then you might as well deploy them right away, because then the server [20:10.840 --> 20:16.920] will operate better. But in public cloud, it is different. And then it is cost. You do not want to [20:16.920 --> 20:22.920] scale up because it is cost you more and you want to do it in the last minute. So, it makes sense, [20:22.920 --> 20:31.240] but not using this. And the problem scaling down OSDs is also not a simple task. You need [20:31.240 --> 20:40.040] to do it all in a manner, otherwise you will be at risk of errors, trying to take it down. [20:43.800 --> 20:49.480] I try to write once the process how to, in the public cloud, how to scale down OSDs, [20:49.480 --> 20:54.280] scale out OSDs. It will document the speech of something like 45 pages. [20:55.000 --> 20:58.680] If you want to do it in a safe manner, it is not simple. [21:19.000 --> 21:23.240] So, the question is whether we can use CADA for [21:23.320 --> 21:27.480] Cepheidium, something like that. So, I guess Cepheidium mostly works with [21:27.480 --> 21:31.160] Portman. It does not need the Kubernetes, and this is specific to Kubernetes. [21:36.040 --> 21:43.880] No, not with CADA. Maybe if we have defined something for the Cepheidium step, then yeah, [21:44.520 --> 21:50.120] that is like, based on like, we need to, yeah, like, we need to see the scalar like, [21:50.680 --> 21:55.480] based on like, if this is a question that we had, hit this request, then Cepheidium can trigger [21:55.480 --> 21:58.840] or Ingressor, etc. That is possible, but not with CADA. [22:01.320 --> 22:02.280] Kubernetes, yeah. [22:06.280 --> 22:07.560] Okay, yeah, sure. [22:08.520 --> 22:14.120] Is there anything similar for tuning, for suggesting tuning? [22:17.400 --> 22:19.720] Sorry. So, the question is anything? [22:23.000 --> 22:29.000] So, I think he's asking if there is, I mean, if you can scale or, I mean, if I understood [22:29.000 --> 22:33.320] correctly, you can suggest configuration tuning based on the workload. [22:33.320 --> 22:40.440] So, the tuning comes from the, like, kind of a YAML, like, which preferred YAML you need to [22:40.440 --> 22:46.280] use or something like that. It's kind of a data based on the scaling, right? And, like, it's, [22:47.240 --> 22:51.320] I don't know, HPA has some mechanism, like, it's based, the scaling is based on HPA only. So, [22:51.320 --> 22:57.000] HPA does not look up for the configuration. It just look up for the deployment, like, [22:57.000 --> 22:59.320] how the deployment see, just see the counts. [22:59.880 --> 23:05.640] About tuning, right, a little long topic. But I recently read a research paper where Luster [23:06.280 --> 23:11.800] was doing machine learning performance analysis based on ML and AM workloads and suggesting what [23:11.800 --> 23:18.600] configuration is that you are doing. There was some, I think I shared that research paper in [23:18.600 --> 23:25.080] this performance weekly once and shared that research paper. But this is a really nice idea that, [23:25.800 --> 23:31.800] I mean, based on AI, I mean, you use AI as models to list the workload and everything, [23:31.800 --> 23:35.240] and just configuration tuning that I want to, in this F cluster. [23:37.960 --> 23:38.280] Okay. [23:38.280 --> 23:42.280] Discussion was, it's more of a, the discussion more of, like, to which I have a performance [23:42.280 --> 23:48.920] weekly discussion. We have more discussion in the upstream community meetings or in the [23:49.080 --> 23:52.120] network. We could work for them for that. [23:55.240 --> 23:58.920] Sounds fun initially, but I don't know how much [24:05.720 --> 24:09.080] Okay. So, thank you, folks. Thank you.