Very much. So our next person is Ahel Boroy from Highland that's going to talk to us about
using generative AI and content service platforms together.
Thanks. I was on. I was on.
I was checking the microphone.
Okay, yep. Welcome to everyone. So this is another view on the same topic. So we are
going on the technical side now. It's not like a final feature for a product, but it's
a framework in order to help you to build all the features that we were seeing before
in the context of a content service platform or a document. Okay, so we are going to review
some in a stack. The next step we are going to use that is including also LLN on premise.
We are going to review all the options. We are going also to describe the features we
can build with this stack. And then we are going to review how to integrate that with
your in our case because I'm working for Highland and we are building an open source product
with the name of Fresco that is related to content management. So we are going to see
how to integrate that with that content management platform and also just looking a few to the
future. And obviously I need to include some AI picture because it is what it is. Anyway,
so this GenAI stack that we are using includes mainly three components. So the first one
is Olama. Olama is a service that is able to provide an API to interact with different
LLMs. We are going to see later all the list but you can download your LLN on premise and
this layer is providing the interaction with the LLM. You can even interact with different
LLMs at the same time. The second one is Neo4j. So Neo4j is the vector database when
you are using rack, we augmented reality and so on. Then you need to increase the information
for the LLM. So you are storing all this information in this database. And finally we are using
land change. So this framework is providing land change that is a framework to communicate
all these different elements. So this framework is in Python but if you are not comfortable
with Python there are many other languages that are including this kind of piece. Okay,
so mainly what we have, if someone doesn't like Docker there is no problem so you still
can deploy that without it but it is oriented to services. So you have Olama that is the
one providing the services for the LLM that can be used in GPU or not. So you can, we
are going to run this without the GPU. Just using my regular CPU on my computer. This
is lower. I recommend you to use a GPU but you can do that. And we are piling all the
models that we need so we can just use more than one model. With that we can increase
the information for the operation with that Neo4j database and we can develop an API with
this string LLM, with this framework. Okay, so these are the pieces. You have the project
Docker.gen.a stack. Mainly these pieces is a sample. This sample is oriented to prompting.
You have to reply questions. So we are going to do something a bit different from that
but the first sample you can try is this one. Okay, so all the LLMs that are able to manage
Olama today are these ones. Likely as this is growing every day there are more. But this
was like last week. Okay, so this is what you need to understand. Obviously the larger
the better but you need to take into account your resources. So these are very small. Your
4 gigabytes of RAM and 2 gigabytes of storage. So you can run that on a computer and then
if you want to use something that is better is also larger in resources. And you can even
use these LLMs that require, I don't know, many different computers once I say, okay.
So today we are going to use this kind of LLMs. Also it's relevant to look at the license.
So just was talking before about the license. So this is also something relevant if you
want to build something commercial or something open source or whatever. You need to take care
of the license. Also you can look that there is some weight license there because you have
this LLM2 community license agreement that some people say that this open source, some
other people say that it is not. So it's something different. So better to check if you don't
see a patchy license or something that you can recognize. Better to check the conditions.
So you have a lot of them to choose. We are going to work today on the demo with Mistral,
CementB, that is a French company that is producing this kind of LLMs that are more or less the
same performance as GPT 3.5. So it's good enough. And so what is open source, the LLM is free
to download and to use, but the training data is not free and likely it has some copyright
material on it. We don't know because it's not free. So on the next law ethical AI writing
we have, sorry, yellow. I thought it was orange but it's yellow. Okay. It's more or less fine.
So we are just only missing one. That was for text and for pictures. We know some LLM
with a visual encoder on it. So for this part we are going to use lava. And lava really
is granting all the different requirements. So we are using a green LLM for this other
sample. Okay. Perfect. So all the demo is running on my computer while I'm there in the presentation.
So I have everything running inside is 32 gigabytes of RAM and is AM64 architecture.
So it's not AMD. It's MacBook Pro two years ago, something like that. Okay. As we were
also reviewing the previous version before this GEN AI momentum, we also had some data
section, test recognition, test classification, content analysis. Anyone is using content
analysis for a real use case? Okay. It was not me. So it's something but you saw. But
we have all the things, right? Some kind of automation. But now with the GEN AI, we have
also a power classification. We could classify in the past. But now we can classify better.
We can also, and when I say translate, we are going to see later the demo. Obviously
we can translate. But we can also interact with the LLM in one language and to get the
response in another language. Right? So that is the difference. We can also summarize
a test. This is the most common use case and we can describe a picture. Prompting. Obviously
we can use prompting. We can read that. So we have some new features that we can use in
our documents. Okay. We are going to see some of them implemented. Okay. So what is this
project about? It's not yet. Okay. The project is at some point of the slides. Okay. If not,
I will give you the link. So in this project, what is created is a API by using this, all
these infrastructure in order to provide different services. What we are using is some LLM embeddings.
So we are just trying to avoid hallucinations. Just giving some additional information to
the database from the document. So we are working with a document. Right? So we are
not going with search. We are not going with some other applications of GNI. So we are
focused on features of a document. So we are adding all that information so we can get
a better response and more suitable to the document we are dealing with. And for that
we are using Mistral. And if we are talking about a picture, then we can use the other
LLM that was Java in order, for instance, to describe or classify the picture. We have
also some, so we can choose the LLM. If you want to choose some other LLM than Mistral,
you can do that for text. You can choose some other LLM with a vision and color enabled,
like Java or some other on the list. And we can also choose the language. So we are going
to see that later. We can just drop a document in Japanese and we are getting the summary
in English or in the other side. Right? And also you can choose some numbers like the
summary size or the number of tasks and so on. So these are parameters. Okay. So this
is the API. Right? Pretty cool invocations. But let's see that leave. As always is better.
Can you see the, better? Okay. Okay. So for instance, I'm going to work, let me find the,
I'm going to work with this document. Right? I could be using an English document, but
it should be easier for the AI. So we are using this one. And I'm also going to use
this picture. So for your reference. Okay. Okay. Perfect. So for this document, we are
going to ask for a summary. So give me a summary of this document that is in Japanese. So with
that, if I'm able to. Okay. So this is running on my computer. So I have this ENAI stack
running in this Docker deployment. And I'm getting the request. Okay. And with that, I'm
getting the answer. Okay. So the test, this is a problem with kindergarten, in Japan,
blah, blah, blah. Okay. That's fine. So I'm giving something in Japanese and I'm getting
the summary in English. The second one, come on, note this one. I did it. Okay. The second
one is just to classify. Classify a document that picking a term of a list of terms. So
I want you to classify this document according to Japanese, Spanish or Vietnamese. Again,
it's an easy example. Right. But you can choose whatever list of values. So if I say just
classify this document into one of these three categories, the term is Japanese because the
document is in Japanese. Okay. This is also a Revan for classification. And finally, we
can also make some prompt on the document. What is the name of the zone or this document
in Japanese document? The name of the zone is Musoku. Okay. So three different features
that we can use on this, on a document. You can build more. Again, it's a Python, Python
program with these three specific features, but you can grow up to include something else.
And if we move to the, to the pictures that was for text, but for the pictures, we can
describe this, this picture. We can also extract some, this is a person, this is, but that
was done before. But describing is the, the, the new thing that GNI is providing for us.
This is a bit slower, but in the end, they made so some man posting for the camera. He's
wearing a green beanie, glasses, a black hoodie. And the land yall says air fraked. Well, no,
it's a fresco, but more or less. Okay. The picture was not big enough, but it's fine.
It's, it's something that is, is useful. And it's not that consuming internal resources,
because it's running in, on my machine. So it's, it's fair enough. Okay. Once that we
have all these features, and we have this, Python, just let me show you a bit. So this
is the project, right? You have the Aeboroi, a fresco GNI, and you have the GNI stack,
and mainly it's a Python program. Okay. With all these endpoints described, classified,
prompt, and somebody. Okay. It's no more than that. Okay. If we go back to the original
goal, is to integrate this kind of operations with our, with our product than in our case
is a fresco. So a fresco, we can deploy that also in Docker or whatever you want. And we
have two different APIs. So the first one is the classical press API. And the second
one is a messages API, synchronous and asynchronous. So if we have existing content in the repository,
you have a folder with 100 pictures, and you want to describe that. So you can use the
recipe. Yes, to get the document, apply the operation, and update the document. And that's
fine, because you can make a batch with that. Okay. You have all the operations available.
And if you want to create that like more dynamically, when the people drops the document, yes, perform
the action, then you have the messages API, the asynchronous API. So you can listen to
the event, okay, there is a new picture, and this picture needs to be summarized. I'm going
to summarize the picture, and that's updated. Okay. So these are the two different patterns
we can, we can apply for it. What we are going to see now, again, live, everything is running
on my laptop, just believe me, is something that allows us to classify a document. So
we are going to upload a document. We are creating this rule. The rule is the same just
for you to make the similarity with what is before. So we have a list of languages, Japanese,
Vietnamese, English, whatever. And we are creating a rule to move the document to the right
folder. So you draw a path document, and the document is moved to the right folder. Okay.
Okay. So let's do that. Okay. Let's open a fresco. So there is a folder at some point.
And this folder has a rule that is classifying the documents that I'm dropping on it. Okay.
So if I, for instance, come for classify, no, for classify things, we are going to try
with a Vietnamese one. It has to be a bit creative. Okay. Okay. So at this point, a fresco
is listening to this new document, and it's classifying the document. So it's just selecting
a term from the list of terms, and the document has been updated. So it has been classified.
So if I refresh, what I find is that the document is on the Vietnamese folder, and you can do
that with invoices, with whatever you want. And we can track that it was mistral, the LLM,
that created this classification. Okay. Pretty easy, right? So you can integrate also all
the other operations in that to get some automation. Okay. So I guess that I was running out. But
no problem. So we have more time for questions. So again, this is a simple framework. You
can deploy that on premise. You can choose your LLM. You have an initial REST API for
operations. Public works are welcome. And then you need to integrate that with your product,
with your organization, or whatever. Right. There is also some interesting hackathon with
more use cases. So I presented you some use cases, but you have more of them on this hackathon.
The slides will be, they are available on the, on Foxen. Okay. And also I'm using Olamma,
but there are many other alternatives. You don't need to choose Olamma. So you have GPT4
all locally. This solution is the one used by, by next cloud, second state, high-end
phase is the most known probably. But again, just, this is an initial framework. Take it
as it and try some things with, with the NAA. Okay. That was all. Thanks.
Thank you very much, Angel. Are there any questions?
I'm going to do it in the order of the rule.
Thank you, Angel. It seems to me all these operations are on one picture or one document.
Are you also considering me asking a question on all my documents?
No. So this, this sample is only for a single document or a single picture. But, but that
is as easy as you have the database, the Neo4j database, then you can include as information
as you want for a single document or for a single query. Right. So what I'm doing in
the source code is to remove the previous information. You have to create something that is only for
a single document. But you can modify that in order to add more than one document to
one query. But on the sample is only for a document or a picture.
While summarizing the Japanese PDF, why did you need to provide for context your picture
Sorry.
You showed the summarization of the Japanese PDF.
Yeah.
And then you provided for context the picture.
No, no, the picture was for the last operation. So the three first operations for summarize,
for classify and for prompting were related with the document in Japanese. I could use
some other document. I know, but I love the document because I'm using this for testing
for 15 years, something like that. So it's like my, my precious document. And, and the
picture was there for the last one. It was the description of that picture that is more
or less like, like yours then.
Thank you.
Similar to the previous question that I had, but for a single document, right. So the summarization
for very large documents.
Yeah.
So, the problem is that again, I'm running on my lap. So I cannot use like a very large
document, but I was just trying to summarize, for instance, books. Do you know the Gutenberg
project? On the Gutenberg project, you have all the classics of Alice in Wonderland and
so on. So I was trying to do that with that kind of documents. And it's able to do that,
it takes a while, like minutes on my machine. Again, if instant adjusin, the regular CPU,
you use a GPU, the tiny slide, I don't know, 100 faster, something like that. So I don't
know. I need to make serious test with that. But having the right infrastructure, I guess
that the, the performance is enough. It's not something like very instantaneous, right?
But you can work with it.
Thank you very much. Any other questions?
Yes.
Hi. A follow up on the previous question. Was the insertion into the vector is database
taking a lot of time or was the actual query to the LLM? Because the insertion into the
vector is database has to be done once, whereas the query can be done multiple times if, if
you already vectorize the document, right?
Yeah. So again, I was not trying to deliver a session on how to develop AI, right? It
was just to create a framework. You have the AI track that can reply to you better than
me in relation to that. But yeah, obviously, you can use the database. I'm not, I'm only
using the database for a context of a single document, right? So you can create categories,
you can add more than one document. You can add also the, the links to the response and,
and so on. So yeah, sorry. Maybe I didn't understand you.
Maybe you misunderstood my question. My question was when you added the Alice in Wonderland
book, was it the vectorization that took time or was it the query to the LLM?
No, no, it was vectorization, vectorization of the chance of the document.
Okay. Sorry.
That was the only one question.
I'm not an expert, but I know a bit.
Any other question? Okay.
Thanks. Okay.
One more question.
Last one.
I'll be around. So if someone just wants to, to catch me.
Can you say a bit more about like the biggest use cases you see and if there's any open source
setups of this that are out there for us to look at?
In my opinion, the, the main use case of that is searching.
So but this is a different world with different beasts.
So but for searching AI, it's really quite relevant.
So again, this is just to create a framework and then it's just to apply your imagination.
Thank you very much, Angel.
Thanks.
Thank you.