When you didn't know what was going on, I'm looking up at the faces that are blank.
Okay.
So AI.
At one stage, AI was a preserve of data scientists, machine learning folks, and people in universities.
And us developers said, let them have their day.
Okay.
But what I want to look at AI, and I know chart CBT, you know, at the end of November 2022,
throughout and everybody from your parents to your aunts to your uncles,
suddenly know what AI is.
There's kids around the world writing papers and some of the information is right and some of it is wrong.
So there's a lot of teachers out there that are fed up, especially in secondary school or high school,
that are fed up with getting answers from models and from AI.
But on a serious note, what I want to look at today is from the angle of,
I've been a developer for nearly the last 30 years,
and my background is about writing services, applications, mostly back in middle there and earlier in my career in UI.
And because of the growth of AI and because of what chart CBT does,
an awful lot of our leaders in our companies now want us to leverage and consume models in our applications.
So I just want to get a little demographic here.
Hands up, who's the data scientist?
Okay, there's a few of you.
You might be leaving in a couple of minutes.
Any machine learning engineers?
Some more.
Okay, I'm getting a bit nervous now.
Any developers?
Yes!
Sorry for the roar.
I'm safe.
I won't get lynched from here out to the door.
So let's get this show going.
As JJ said, my name is Barton Hickey.
I'm a software engineer working over at IBM.
And I spent around the last eight to ten years in the cloud native space in different communities.
But my key role, I've been very lucky, is contributing to open source communities and trying to drive open source communities forward.
So a little background.
I'm going to do a very small background in AI.
AI folks don't shoot me down.
It's my interpretation.
I'm only dipping my toe in the water over the last couple of months.
So this is how I see it.
But really I want to get into the frameworks and open source in general and how can we use the models using these frameworks?
And then we might have a bit of a demo if I have enough time.
So I'm just going to throw up this definition.
That kind of two sentences on it.
And from what I've looked at and from what I understand, the way I look at the model is I look at it like any other program.
Or library.
Or whatever else that we've often called to use.
So it has an API generally.
We can call it and we can get a result.
We can consume it in our application.
But I suppose the difference here with, for me as a traditional programmer, our traditional programming is, you know, we give it a set of rules and instructions to tell the computer what to do.
And more often than not, we should know how that program is going to work.
With AI models, it's a little bit different here because it's not exactly explicitly programmed for predictor tasks to do or a particular prediction.
It learns with the data.
So there's the big difference from restarting out.
So what is the journey of a model?
So first of all, I suppose it's the building or the prototyping of that model.
And data is the key here.
So as a lot of my North American colleagues would say, garbage in, garbage out.
So the data is really, really important when you're training or creating your model.
And the first part of that is getting that data from reputable sources, reputable domains for the particular operations you want the model to do.
Then it's the preparation of the data because the algorithms that process this data or create the model, they need the data of a certain quality and in certain formats.
So things like errors and omissions need to go out of it.
If there's any duplication, any missing values, and also then maybe converting into a format that the algorithm can use.
There might be some aggregation as well, so that you know what I mean, to normalize values, etc.
Then you choose the algorithm.
The algorithm depends on the operation you want to do.
And it also depends on how you're going to train it, how it's going to function, and then what resources you've available to do it.
So you can see when we started out with the generative models, they were huge, and I'll talk about them in a minute.
But now there seems to be a turn at the start of this year of smaller models and more modular models, etc.
Because of the resources to train them and also to run them.
The key part of the algorithm is to turn that data into a model and produce the model you have that's trained on certain data, which is the next step.
The training data set is very important because you want to see within a certain level of tolerance how exacting is that model.
So what are the results? Are they within a particular tolerance rate that you are going to be willing to accept?
And then at this stage then you say, right, I have a model here, I've trained, what I need to do is the final step, which is validation.
So then, testing the model against data that you didn't use when you were training to see if the model can actually learn when new data comes into it, and our as its predictions or its tasks as you expect them to be.
So a big part of this I think going forward is open source could be the key with models because it's going to come down to trust.
Do we trust the model? Because let's be fair about it.
Even, you know, the people who write the models or the graded data scientists out there are machine learning engineers.
Sometimes they can't even predict what the answers are.
So the key here for transparency and in the open and trust of models is going to be key.
And the last two parts are the parts I like.
Running the model and calling the model, okay?
So, you know, sometimes, you know, back in the day when you used, you know, binary tree library, you just wanted it to give you the answer.
You didn't want to be writing a vanilla binary tree unless you were into that.
So the same with the model. You want the model to do an operation for you, perform a task or make a prediction.
So that's what they call inferencing.
A little bit around genera.vii because that's the buzzword. Everyone's talking about it.
And the key here is that it's a different type of model.
So your tradition in machine learning models were trained on label data.
So data that was specific for specific tasks.
A lot went into knowledge of the domain and knowledge that the data scientists need to know around that particular area.
And it was very intensive in the training.
In this situation with the foundation models, what we're saying here is we're going to train it on a massive data setup on label data.
And then you can use that model and fine tune it.
Or so you can do different tuning like fine tuning where you take the model.
And because these models are deep learning models, there's going to have a lot of layers in them.
You may take one to number of layers off, put your own layers on top, i.e. you're going to train it against your own data.
Or you might use prompt tuning where when you're calling the model, you're going to pass it prompts like examples, what you're looking for, and guide it towards the answer you're looking for.
An example of this is around the large language models which are based on huge language data sets and can generate content from there.
And we can see that in which generative AI. It's about using these models or these generic style models to be able to generate high quality text, video, etc.
So that's the whole idea of the generative where we're going now.
And the idea here that these models, one model can be used for different operations as opposed to be normally trained for one.
Okay, so that's my intro to the AI. And the next part is what I really want to get onto is around the frameworks.
So hands up who's heard of Hoganface. Okay, that's not a bad number.
So Hoganface has built up an AI community which is nice to see based around open source.
It's key things being it is a series of libraries. It has a huge catalog of open source models and data sets.
Now, how is this appealing to you if you want to use open source models or sorry, you want to use AI and models.
The great thing about this is you can pull those models down and you can run and use them locally.
So that if something changes tomorrow morning with Hoganface, you can still use those models. You can use them locally.
Now, they also provide a service where they host the models themselves. So you're directly just calling the models through their API.
But I think it depends on what your setup is and your use cases, etc.
So what I'd like to do with this when I'm looking through this is an example.
And this is an example using the Hoganface Transformer library. And it was an example up on the Ray framework, which I'm going to touch on straight after this.
So the first thing is we're using a model called the T5, which is an encoder-decoder model.
And these types of models are really geared for natural language processing, or NLP as they call it.
So when you're dealing with text of some sort in different languages, etc., it's usually text in, text out.
And the task we want to do here is just a very simple example.
Looking at the code is that what we want to do is just convert a bit of text to French.
And I use French because we're here in Belgium. I couldn't get Flemish or I didn't think of it, so sorry about that.
So when we look at this, we see it's two real calls. The first one is we use the Transformer library, and we call Pipeline.
And in Pipeline, you specify the model and the task you want to do.
And then inside the translate method, we're then going to call the model with the text that we send in.
Very simple Python class. When you run the class, if it's your first time calling that model, it will call out to Hug & Face,
Open Source Cal log, it will pull down the T5 model, and it will put it in cache.
So any call after that for that particular version of the T5, it will get it locally.
Ray. So Ray is a framework for scaling and distributing Python and machine learning applications.
So the capability is it's providing which is batch inferencing on CPUs, GPUs.
So the inferencing being running the models, which we talked about earlier.
But the stuff we're interested in mostly. Obviously the serving of the models, hosting them up, having them ready to go, providing an API in front of them.
Also training of models, large language models. So that's if you have your large language model.
It's not, I suppose it's not giving you results you want for the particular operations you want.
Then you may decide to train that with your own data.
So in that case, you'll create a new version of the model.
And then other operations like reinforced learning, etc.
So if we want to use Ray to now host the model, in this scenario, what's going to happen is Ray has a nice little HTTP server.
And pretty much all you have to do is put an annotation on top.
You can see just below the translator class called serve deployment.
And you can pass args in there if you want.
And the next part that's important is the callback function, which is called underscore underscore call underscore underscore.
So in this situation, once you you have your call, you have your annotation, you can then call this class using.
Using server on. So it then will load it in a HTTP server and it'll provide the interface to it and you can call it as follows.
The next and the final framework I want to look at is the Triton inference server.
And the inference server provides you with support.
For most machine learning applications and frameworks, as well as custom C++ and Python backends.
So you can see the different frameworks that supports there and processes support, etc.
And this time, if you want to wrap it or use it, you need to actually call the class.
I'm going to read it from up here.
It should be either Triton, Python model, which is a bit of a mouthful.
And then you need an execute method. And that's where the HTTP request we're calling to just note here.
It's not using JSON data types. It's using tensor data types.
Okay, so I don't know if you've experienced of using tensors, etc.
PyTorch or anything like that, you'll be used to that.
So you have to convert it in into into Python data types so you can process it.
The other thing you need to do to bootstrap it is you're going to need a configuration file and that config file.
You give it the name of your particular model.
So I'll call it the model you want to host.
So it's going to be the local name that's going to host.
And then you tell it what back end, in this case is Python, and your input output.
And those input outputs are binary tensor flow types.
And the last thing you do then is you need this model directory down here.
So in a minute when we call models, it must be under this with the name of the wrapper model class and your config and your model file.
And finally, to call it then, there's a number of ways you can do it.
One easy way is to use Docker and run its container.
And when you run that then, you then need to copy the artifacts, which is the model directory that we set up into the container.
And then you call the Triton server executable to run it as a HTTP server.
And you can call it in the call request above with that path in the name of your model and in fare.
So before I go on to the demo, I just want to just do a little summary of that.
So I'm after choosing three frameworks and you're probably someone's probably up the back on each and go on.
There's loads more. Yes, there is.
You VLM, you know, which is a nice alternative note to Hogan phase teachers.
And there's just so many of them because this space is just growing phenomenally.
We probably saw it with cloud native over 10 years ago.
And you can see it at the moment that there's more and more frameworks.
And I think the key here will be if we can have these frameworks be open source as well with the models.
And then if companies want to put their value on top, then they can put their sauce on top.
But for most of us here, we would like to be able to choose our framework and see if it'll do what we needed to do.
So load our models and stuff like that.
The other one I'm shown here, I'm just obviously running locally and I'll be running something locally here in a minute.
But the idea here is that you'll deploy this into some system bare metal, a cloud system, whatever, because depending on the model,
it's going to need serious resources to run a lot of the time.
But as I said, there seems to be a shift towards the smaller models and then the plugability in these models where you can have different capabilities in the models.
But that's something for me to learn down the line and for the data scientists to come and tell us about.
Okay, so let's do a quick download.
Yeah, I'll just escape here one second.
You want?
Yeah, if you don't mind.
Thanks very much.
Just before I show this.
So what I'm doing is, I'm just running a framework here called KK.
And why I'm running this is a colleague of mine, Mark Stewart, did a really nice UI using radio, the radio framework, if you know it,
a Python framework for it's very handy for for doing UI, active access stuff and elements and things like that.
And I'm just showing you here that I'm running it.
And what it's running is it's running the AI, sorry, it's running the UI as a HPP server.
And it's also running the back end server with a gRPC and HPP interface.
And the back end server is the host.
So where it's hosting the model and it's and the way to run those models are similar to the other frameworks I've shown by wrapping the model for the particular artifacts that's needed.
Okay, so here's the simple UI.
And what I want to do is just play with models for a minute and then just show, look, it's just code wrapped in the back end.
And as developers, we can just use these models to perform operations that, you know, we might take us ages to write or would be quite difficult to write.
And we can use these off the shelf models, hopefully.
So the first one is putting in a sentence like.
I can.
How's that?
Better?
More.
This is where you were told in class to come down to the front.
Did you hear that?
That's better.
Thank you.
So what I want to do here is I'm using a model that does sentence similarity and I put in the canine is fast is my source.
And then I'm saying the dog is running the cat is asleep.
And if you know your cat from your dogs, a dog is the canine.
A cat is not.
So there we go.
It's telling us 68%.
It thinks the dog is asleep.
So there we go.
It's telling us 68%.
It thinks the dog is running is the best sentence.
But if I change it to say the cat.
Is running.
We now put the cat among the pigeons, so to speak, excuse the pun.
But now we're saying it's a 36% chance that, you know, you're not going to be able to get the cat.
It might be like the canine is running.
So you can see here how you compare on with things.
And because of vector spaces and so forth, where words are aligned, that if you do change some of the words like running, you know, fast, they're going to be near each other.
Dogs and cats are going to be near each other because they're animals.
So yeah, that's what happens there.
So looking again, let's do image classification.
I'm going to choose an image here.
All right.
Yay, puppies.
Okay.
Now, I love dogs, but it's telling me it's a 88% chance that's a golden retriever.
Any person can confirm that?
Okay.
Thank you very much.
I just thought it was a Labrador, but sorry about that.
And the last one I'm going to do is something close to my own heart, which is I've taken an image of a sport we have in Ireland called Hurlin.
Okay.
So Hurlin is players are on the field.
They have a ball.
They have a stick.
Okay.
So when it does the detection here, it detects a sports ball.
That's fantastic.
It detects people, which is fantastic.
But it calls the stick a baseball bat.
Is there anyone in the audience that knows what the name of the stick is?
A hurly.
Yay.
I love to hear.
Okay.
So that's an example of the data the model has been trained with.
What you'd need to do there is either to prompt tune that model or fine tune that model with a data set that tells you more about the game of hurlin.
It's able to detect the field sport, which is which is amazing people and ball, considering it hasn't been trained anything on the sport, but then it lets itself down a little bit.
So why did I show you these things bar showing a really nice UI for my colleague?
The reason is this.
At the end of the day, we're calling libraries like we've always done or we're calling API is like we always done.
And as long as we know the models can be trusted, then that's okay.
All right.
So the first part in here is this is just a great UI code.
And in here, we're just saying there's a series of different classes for the for the different UI tabs.
And we're just calling them here.
And here's an example of one of the UI tabs down here for the image classification.
You can see here the submit button and the other elements that are on the tab.
And in this situation, you can see the submit is going to call into a function called function for that's a great name, by the way.
FN.
I haven't seen that for a long time.
And you can see here in here, then this is where it's going to call the model.
Okay.
Now, why is this a little bit funny because it's a g is the g rpc API is calling and you can see here inside the UI when it starts up, it's going to get it's going to get a handle to the channel of the g rpc server.
And then over on this side, you see here the different tabs classes to handle the UI interaction.
And then finally down here is the wrapper code.
And you can see here it's got similarities to some of the frameworks I showed a while ago.
But in this situation, you use an annotation up here called module, you pass it a task and so forth.
And the key here then is you can see in the init again, we're calling a pipeline, which is the hug and face transformer pipeline up here.
Sorry, I've just jumped.
Yeah, just up here.
If you don't pass a model to it, the default model is going to use is the Google VIT model.
And then down here is the key in the run method.
So when you call the predict, it's going to call the HTTP server and it's going to be redirected in the server to the run method here of this module that's loaded in the server.
And it's going to call the pipe here.
And when it calls the setup pipe and passes the image like it did the last time, then it's going to get the result back.
And that's how you get it.
Okay, so let's jump back to our deck.
All right, to wrap this up, anything's on the screen there.
I'm actually going to talk about that for some reason.
So you can read that if you want.
No.
Why did I do this talk today?
I want to do this talk from the angle of somebody who's been writing code a long, long time.
And I'm afraid that someday someone's going to say you're too old and bald and you've got glasses.
And I'd...
And a beard.
Sorry about that.
No, but when we look at the change, it's probably the biggest change.
If you've been in the industry for maybe 20, 30 years, it's probably the biggest change we've seen the way things are going.
And there's always that phrase, you know, you either adapt or you stand still and everything moves on.
So the ability here to be able to use these models to write applications and improve things is really the key.
And this is something that I suppose to grow out of AI definitely in the last 10 years or plus.
I suppose starting with, or maybe 20 years starting with the Linux Foundation, then into OpenStack, Kubernetes, you know, all the other different great foundations and communities out there.
I think a lot of companies have realized one company or one set of developers cannot keep up with the rate of development, the rate of change, the way technology is going.
And having this ability to get these models eventually out of the grasp of the scientists stuck in the labs that don't want to release anything.
Sorry data scientists, I'm not picking on you.
But you know, to eventually see the light of day and they can go home and tell their families, you know, that thing I was working on for 20 years, they're now using it in fridges, in cars, everywhere.
But the key here is that we need to have it open.
We need to drive forward with our models and be transparent.
We need to have trust in the models because we're asking the model to do something for us or give us a result that we depend on.
And like libraries we use in the old days, you needed to go and do the groundwork and find out can we trust that library?
Is it doing something we don't want it to do? Is it doing something malicious?
So I think there's a big change here from the initial AI maybe in the last 10 years where it was around fraud detection, spam detection, you know, chatbots.
I'd say there's going to be a big proliferation offered the next 5 to 10 years.
So thank you very much.
Questions?