So, good afternoon everyone and thank you for joining me today.
My name is Bello and I'm a software engineer at Red Hat.
And over the past year I have the opportunity to be part of the forklift team and take it
for a spin.
So, today I'm about to share with you our recent journey and without further ado let's jump in.
So, in today rapidly involving both of IT we can observe an increase in moving away from
traditional virtualization environments towards more hybrid cloud solutions.
And with Red Hat RNOT just observing this trend we're actively participating in it.
So, recently we had the opportunity to go on a journey and migrating from a
virtual established environment to a newer solution.
And today I'm going to share with you some of the inside challenges and benefits of such a transition.
So, let's start by discussing these two very different solutions.
So, picture you on a journey through the IT computing landscape.
Our first step is Ovid.
It's like an older reliable train that's been running for years.
So, Ovid is an open source product based on KVM technologies and it's offering
cost efficient way for enterprise to managing their virtual workloads.
It's an alternative to vSphere.
But our journey does not end there.
We then continue to the world of OKD.
So, picture it as a high speed train whisking us to the future of cloud computing.
So, OKD is also an open source project based on Kubernetes and it's providing us with
cloud computing capabilities alongside enhanced Kubernetes features such as added security,
automation and user friendly interface.
And it supports both containers alongside virtual machines.
So, when considering such a transition it's important to take into account how it can be done.
So, there are several path we could take each with its own set of advantages and challenges.
But today I would like to focus on main three.
So, first we can't reprovisioning all the virtual workloads and start from scratch.
Even though this solution may be sound pretty straight forward it's both costly and time intensive.
And for complex workloads it's not always possible without risking some data integrity
and operational disruptions.
Next, we can migrate all our virtual workloads into containers.
So, with the use of conveyor project we can really reduce the cost here.
But it's still not an easy task.
And again we have the same issues before not all workloads can be containerized.
So, while this may be a good solution for certain types of applications,
it's not suitable for everyone.
And finally, which seems to be the best one is keeping our migration workloads,
our virtual workloads as they are.
And with the use of forklift tool migrate them to the new environment.
So, by that we don't have to worry about any data lozage.
And with the use of this tool we can have simple and smooth transition.
So, what is it forklift?
So, forklift is a tool designed in a system that is designed to be a virtual environment.
Design in assisting migrating from traditional virtualization environments
to Kubernetes-based environments.
And it's taking care of the entire migration process for us.
It's working alongside with another project named CubeVirt.
And CubeVirt providing us the virtualization capabilities
on top of Kubernetes-based environments.
And once forklift migrating the virtual workloads, they will be placed on top of CubeVirt.
So, forklift as a versatile tool supports a variety of source providers,
source environments as you can see here in this list.
So, now I would like to take a deeper look at forklift high-level functionalities.
So, forklift supports two types of environments, KVM-based and VMware-based.
And for both of them, it's taking care for the entire migration process.
That means creating the disk, copying the data, and for VMware-based product,
converting the virtualization stack to match CubeVirt requirements.
And, of course, finally creating the VM itself with its original setup to run on top of CubeVirt.
So, the use of this tool will make easier and smoother transition to the new environment.
So, now that we finish discussing these different solutions and approaches,
let's dive in into the specifics of our own migration from OVirt to OKD,
where forklift used as a crucial tool in facilitating this migration.
So, I would like to start with a little bit background on why we decided to go ahead
and proceed with this transition in the first place.
So, our OVirt environment being in use for more than a decade,
supporting hundreds of virtual machines with diverse usage,
some for production while others for developing and testing.
While the fact that OVirt reaching its end-of-life zone wasn't the main reason
we decided to go on this transition, it certainly matches in this direction.
And, moreover, we wanted to take this opportunity and reallocate some of our resources
and remove underutilized workloads while causing as minimum interference to the users as possible.
So, taking all this into account, the shift to OKD seems to be the most reasonable fitting choice.
So, as any successful story, planning is always essential, and our migration wasn't exception.
So, we started our journey by having in-depth analysis of our current environment
and just understand what the migration requirements and what we need from this transition exactly.
We then continued to having some resource evaluation.
That means we had to make sure that our target environment will have enough resources
to accommodate the incoming workloads in terms of compute, storage and network.
And finally, we had to create a clear timeline to make sure that each step of the way is well known
and everyone involved from users and IT teams are in the loop of this transition.
So, now we would like to zoom in even more into the preparation step and focus on the resource allocation.
So, we had to start by finalizing our VM list for migration.
And when we thought about what going to be the criteria for VM to be eligible for this transition,
we decided to proceed with actively used VMs only and had to have close conversation with their owners to understand their specific needs.
After that, we had to calculate the storage and IP addresses of all the VMs in this list
to make sure that our target environment will have enough resources.
This step was more than just technical preparation.
It was essential to ensure that once the migration is started, we won't have additional downtime to lack of resources.
And last, we had to come up with a way to reflect our original ownership and access mode from the overt environment to OKD.
So, with a well-planned and tool like forklifts at our disposal,
you might think this migration is going to be a walk in the park, right?
Well, not quite. As we started our journey, we discovered that the path ahead of us is going to be quite challenging.
So, now I would like to share with you some of the obstacles we encountered
and how we tackled each of them to keep our migration on track.
So, the first challenge was regarding the VM selection.
So, as I mentioned earlier, we wanted to continue with only actively used VMs.
That required from us to analyze the VM usage patterns
and understand which VMs were actively used during specific time period,
tasks that proven to be quite challenging.
Then we had to gather the information about these VMs, such as disk size, network and ownership.
And that task appeared to be quite demanding as well, both in matters of complexity and in time intensive.
And the first, our two environment had different provisioning models.
Our overt was more admin-driven and our OKD was more user-driven.
And we had to come up with a way to bridge this gap somehow.
So, in order to overcome these challenges, we went ahead and developed Python script
specific for the migration process.
And they can be broken into two categories.
The first one, based on OvitasDK, was mainly used for finalizing the VM list for migration
and two data gathering, such as the disk size, IP allocation and ownership.
The second sort of scripts were based on Kubernetes API
and they were used for creating the namespace on the target environment
and for assigning the appropriate role for the users.
We also uploaded the script to our GitHub region, so they can be used as a blueprint
if anyone wants to take a look, you're more than welcome.
So, now I would like to focus into a specific issue we had
and just walk you through the different stages that we took to solve it.
So, as I mentioned earlier, our two environments had different provisioning models.
So, our Ovitas environment were more centralized models,
where admin had full control of the environment and managed all the resources and created new VMs.
Our OKD environment, on the other hand, is more user-driven,
where the user have freedom to manage and create their own resources within their namespace.
The namespace resources are set by predefined quotas.
So, to bridge this gap, we decided to go ahead and create new namespaces on the target environment
and place in each one of these namespaces all the shared VMs by the same users.
And by giving them an admin access, we made sure that each user will remain
with the original permissions.
So, let's clarify it with an example.
So, let's say after we finished finalizing our VM list, we ended up having four VMs for migration.
So, as you can see, on the new environment, we created three new namespaces,
and in each one of them placed all the shared VMs by the same user.
So, Bob and Alice, who shared two VMs, now will have shared namespaces
with both having admin access to it.
And Bob ended up having three projects assigned to him,
which really reflects the diverse usage on the original setup.
So, now I would like to guide you through the script we used for this mapping process.
So, the first part is based on Ovid SDK, and we did the following.
So, we started by creating a list that mapped between all the VMs and the users from the system.
Then, based on information from another script, we removed all the admin and system users from that list.
Then, we created a dictionary that mapped between sets of VMs and all their corresponding users.
And based on this dictionary and Kubernetes API, we created a YAML file.
So, here we can see a set of actions for one set of VMs.
So, we started by creating the new namespace on the target environment.
Then, we created an admin role that gave full permissions on all the resources under this namespace.
And finally, we created a role binding that bind between a specific user and the role, the admin role.
And by that, we made sure that each user will retain its original access to its resources.
So, now that we finished with the planning and preparation phase,
let's dive into the migration execution.
So, our first step was to deploy Forklift.
Forklift can be installed from the operator hub, and it's managed by an operator lifecycle manager.
In our case, we decided to install it on the same cluster as the target one,
but it also can be deployed on a remote different cluster.
Next, we had to create a new namespace that will hold all the migration resources,
including providers, different mappings, and the plans themselves.
It's important to know that the user used to create the namespace
should have sufficient permissions on the migration resources.
Next, we had to create the target and source provider.
So, each provider represents the environment we're migrating from and to.
Once we deploy Forklift, a new tab named migration will appear in the console,
and from there, we can manage all of our migration resources, including the addition of new providers.
So, we started by creating the source provider, and here we chose Redhead Virtualization,
which is the downstream name for Ovid.
We then had to fill in all the information about this environment, so Forklift will be able to connect it.
Here, it's important to use users that have sufficient permissions on the VMs where about to migrate,
or else the migration will fail.
In our case, since we were dealing with scale migration, we went ahead and used Administrator account.
Next, we created the target provider.
So, here we chose OpenChief Virtualization, which is the downstream name for OKD.
Here, we only need to fill in the name, and all other information is automatically filled in.
Next, we had to create our network and storage mapping.
Once the migration starts, Forklift needs to know how to redirect the incoming workloads in terms of villains and storage class.
This mapping will tell him how to handle the incoming workloads.
So, here we can see our network mapping, and we can see the new villains we created for our migration needs.
Here, we can see the storage mapping and the storage class used for accommodating our incoming workloads.
Finally, with the use of script, we had to create our migration plans.
So, each plan holds inside of it all the VMs that are about to be migrated to do the same namespace.
This means used by the same users.
Once we were ready, we triggered, again, with the use of script, all the migration, and the migration started.
As you can see, it also can be triggered from the console, but since we were handling with scale migration, we automate this process.
Now, I would like to have a quick overview of the steps we had and add some additional information.
So, we started by deploying Forklift and setting up all the costume resources for migration.
Then, with the use of script, we automated all the plans and the migration.
In our case, we decided to go with cold migrations.
That means that during the transition, the VM is going to be shut down, because it best suited our needs.
We're migration, on the other hand, keeping the VM operational during the migration,
but it's leading to longer migration time, because we need constantly backing up the data to keep the VM operational.
So, during this transition, we also monitored and troubleshoot the entire process just to make sure that we're on track.
And once the migration was over, we chose some randomly VMs and tested to see their up and running, and then waited for some user feedback.
So, although eventually we had a successful migration, we did encounter some issues during it.
So, the first two issues related to the fact that we had a lot of simultaneously migration running at once.
That caused both storage and network strain, and eventually led to longer migration times than we originally anticipated.
Another issue we encountered caused some of the migration to fail, and after we had some investigation,
we realized it was related to some bug in our codebase.
After that, we released a fix, and with the use of that fix, we were able to migrate all the VMs, and it was included in the next version of Forklift.
And finally, since the downtime was involved, we had to keep a clear communication and make sure everyone in the loop of what's happening.
So, once we started receiving user feedback from the field, it was clear that we still have some issues to solve in order to make this transition fully successful.
So, the first one was related to boot order issues.
So, VMs with multiple disks were not booting from the right one.
So, we addressed this issue manually, and later we discovered it was caused by another bug in our codebase that was fixed in the next version of Forklift.
The second issue was related to the new VLAN we used.
That caused our FQDN names to change, and the workload inside the VMs were no longer accessible.
So, we had to update our DNS records, and the user had to adjust their FQDN names inside the workload to use the new ones.
And after that, all the workloads were accessible again.
So, as we're reaching the end of today's journey, I think it's a good point to reflect and draw some conclusions.
So, overall, we had a successful migration.
We were able to migrate more than 100 VMs and copy 12 terabytes of data.
We mainly were able to achieve this result through thorough in-depth preparation and planning, and we realized how much it's crucial for a successful migration.
Another thing is that we understand that each migration process can be different and held between different environments,
but we do see some common ground and best practice that can be used to similar journeys.
And finally, and probably the most important, is that even though Forklift is a really powerful tool and it gives us great capabilities,
it cannot facilitate migration on its own, and additional steps are required, such as the use of scripts and thorough preparation.
So, as we're wrapping up today's session, I would like to extend my biggest gratitude for each and every one of you.
I hope that the session today will be valuable for people that want to go on the same journey.
I wasn't able to cover into details all today topics, but we post a blog post about this.
So, whoever wants to get another information, you're more than welcome to take a look.
And that's it. Question and some insights.
Thank you.
Yeah.
How did you handle notifying the VM owners during the process you automated the fanning with the old automated notifications?
Yeah, we had... Can you repeat the question?
No, you should get the question.
Sorry.
For the streaming, so maybe people are watching from the audience.
So, did we automate the process of notifying the VM owners?
So, in our case, we had a VM list that all the owners on this environment were included in, and we said as spreadsheet that all the VMs that were eligible for migration were included.
And then we asked the owners to let us know if they want to migrate their VMs, because there was people that decided to continue to different environments or didn't need the VM at all.
And based on this information, we also built our migration, our final migration list.
So, yeah.
Yes.
Kai, could you please give us some example of what issues you had during the migration steps?
Yeah, so I will give an example about some boat issue we had after the migration.
So, we had a lot of VMs with multiple disks, and when you're trying to boot from the disk that doesn't have the operation system on it, the boot will fail.
So, you see just like a black screen, and the OS is not found.
So, we understood that it's probably not booting from the right disk, because we saw there is another one.
And once we manually changed that, we really saw that it's solving the issue.
So, we adjust this manually for all the VMs in the migration list.
And after that, as I said, we released the fix in our next version.
Yeah.
Hi.
Is this tool also performing some kind of a preferred check over the plane?
I don't know.
It's checking that you have enough space on the target storage class, or it's checking that the VM you selected is not exposing particular devices.
But can make the middle of the end?
We do have set of, so the question was if we did some verification to make sure that we have enough space on our target environment or in devices like in compute.
So, we do have set of validations on our plans, but this one are not included.
We're checking more things like names that match Kubernetes and more security things, not something like that.
Yes.
You transferred, you mentioned 12 terabytes of data.
I was in the presentation yesterday about talking about PCCOP, and then we're talking about validating that all the data was named correction over a large database migration.
Did you do some things?
Because I was saying quite a hard problem.
A lot of days you might get a crack or off.
So, the question is if we're doing some validation on the data, if it's copying correctly.
So, it's depend on the source environment, but we do use some external tools for that, and these external tools supposed to make sure that all the data copied correctly.
So, it's really depending on the source environment you're using because there's different flows between the different environments.
But the tools that we're using, for VMware for example, we're using VIRT V2V, so we're taking care for this check.
For Ovid and OpenStack, we're using ImageIO, so it's taking care of under this tool.
Okay, so if anyone want to ask any specific question, feel free to approach outside. Thank you.