We'll be starting our next talk here from Tawana Czelec.
She lives in Amsterdam, but she's from Istanbul.
She loves historical fiction and free diving.
Surprisingly, she's dived 25 meters down to save her GoPro before,
which we got a bunch of asses this year.
This is pretty crazy.
So thank you so much and take it away.
Alright, thank you.
I hope, let me know if I need to eat the mic.
Alright, so this particular talk is a bit of an outlier to the talks I usually give
because it's nearly fully a showcase of a very simple actually project
that I built with some community members of Haystack
and some functionalities of Haystack that made this project possible.
And then I'll end up by showing a few other projects that we built together.
And the way it usually goes with the Haystack community,
so a quick side note, I work for Deepset,
which is the company behind the open source project Haystack.
And we have a Discord server, and from time to time, this is basically what happens.
So this evening I say, I'm a bit bored, I want to do something,
and I go and join a voice channel on our Discord,
and there's one particular community member that I have to give a shout out to
for this particular talk, Rec,
because oftentimes Rec will then come up with a random idea,
and either myself or the two of us will just share screens
and do some pair coding, hack together something.
And this particular project is exactly something that happened like this.
I'm pretty sure a lot of you know this page, it's Hack and Use, there's a lot,
it changes a lot.
So the idea that Rec came up with was, well, why don't we build something
very simple that gives you like a TLDR of the top K Hack and Use articles.
So we built that, and then just recently,
and when I say recently like two days ago,
that became a hugging face space that you can actually get to at this QR code.
And we've kind of vamped it up a bit, and we can now pick between two models.
You can use MixTrail or an OpenAI GPT-4 model,
and then provide a number of the top something,
and I make it go up to five because no one's made of money,
and you're going to be making APR calls to OpenAI possibly,
so it goes up to five, and this is literally when I ran it yesterday,
and the funny thing about this one was that at the time,
the second top article was actually the FOSTA and Livestreams,
but you get like a short summary of what the top three articles are at this point
with a URL to get to the full article itself.
So my whole talk is based on how this was made possible,
how we actually built this project, and we built it with Haystack.
Haystack is a fully open source, large language model framework.
It's all written in Python.
The main ideas behind Haystack is providing tooling for developers,
so nothing is a plug-and-play really, you're building it all yourself,
and the two main structures in Haystack that make it possible
are called pipelines and components,
and a pipeline is made up of a few components attached to each other
where every component is forwarding some data to the next one.
And I'm not going to get into what RAG is,
I'm pretty sure a lot of you know what RAG is at this point,
but a retrieval augmented generative pipeline might look a bit like this.
You have a query, and then an embedder component is creating an embedding for that query,
then a retriever component is retrieving the most relevant context
for your LLM to actually use,
and then it's forwarding that to what in Haystack world we call a prompt builder,
so that context gets embedded into your prompt itself,
and then you use the generator,
and that can be any model, open source model of a hugging phase, or open AI, etc.
And then you get an answer.
This is a pipeline, but what a pipeline does is really dictated by what components it's comprised of.
You might also create a pipeline that indexes documents.
I'm not really going to get into this here,
you're just basically fetching the contents of a URL,
and then writing that into one of the available document stores that we have.
But this is all made possible because of the structure called a component,
and a component in Haystack is something that could have, for example,
in this case it has two inputs and an output,
but you don't necessarily have to have a defined output.
Haystack doesn't really make assumptions as to what a component has to be.
It can also be something that has two inputs, two outputs,
and then the idea is you attach those components to each other,
and you can be very, very specific here.
You can be specific into saying, like, I want output one to be forwarded to input two of the next component.
You can be very, very precise here.
And maybe you can already start to see this starts to look quite like a graph.
So how do we build these components?
There's only a few requirements for something to be a component in Haystack world.
We provide a bunch of ready-made components that, if you go to the Haystack documentation,
you'll see a bunch of sections there, generators, converters, embedders, etc.
These are all basically components that have been built exactly like this
that we just provide in the package itself.
But what you can do is build your own.
And what you need to have is a class.
So here I've just got a very, very well-named MyCustomComponent class.
I need a run function.
And the other thing you need are these decorators.
So basically this is telling Haystack that this class is a component.
And then the second one is around the run function.
And this is actually used for pipeline validation down the line.
But it's basically telling the Haystack pipeline what outputs it should expect from this component.
In this scenario, I've got a MyCustomComponent that's expecting a query,
which is supposed to be a string, and then it's returning documents.
In this case, it's just hard-coded.
It returns high and by.
So we know that this, whatever query this gets,
is going to be returning two documents, high and by.
And this has then led to quite a bunch of components that don't actually,
they're not served through the Haystack framework itself, not all of them are,
but you can just install them as separate packages.
And it's meant community members have just gone ahead and built components
that they need for their very specific custom needs
and made them available to the rest of the Haystack community.
So let's come to our Hack and Use TLDR, if you will, project.
The idea was that we wanted a component that would take top K,
that could be a number, and it would return articles.
And again, this is a colab that you can use, it should be running.
And the way we did that was, this is very much pseudocode.
Later, if we have time, I'm going to show you the actual code.
But we built this component called Hack and Use Fetcher.
It takes top K, it queries the Hack and Use API
and gets the top whatever number we've decided.
And the other thing I wanted to show here is, at the end,
I don't know how well you can see it, but we've also added some meta information,
because down the line, we can use meta information in our prompt,
because you also get titles of the Hack and Use articles,
you also get URLs, which is great for referencing down the line too.
So we return full documents that have the content,
the title, and the URLs of each Hack and Use articles that we fetched.
And at the end of the day, we're going to be building a pipeline
that looks like this, and everything you see in green
is already provided with Haystack.
So that came with pip install Haystack AI anyway.
And the orange is what we've just built for ourselves,
and it just slots into the rest of the Haystack pipeline ecosystem.
And for this co-lab that I've shared with everyone here is,
I've decided I'd just go ahead and use Mixstrahl,
I've tried this with OpenAI models a lot, so why not try something new?
And then the last thing I want to highlight about this particular pipeline
is how the prompt is being built.
So prompt templating happens in Haystack world
by a component called the prompt builder.
And templates use Ginger templating.
And what's really important here is,
okay, we have an instruction, you'll be provided with one or more
hack and use articles, please provide summaries.
But if you look at this close theme, we have actually a for loop.
So this prompt builder automatically knows that it should be expecting
an input called articles, and it can loop through those articles,
and then it can access the contents of that article object
individually in every step of that for loop.
And that's how we're embedding URL here as well.
And this is the final product.
At the end of the day, we were able to build a pipeline where,
given top three, we were able to run it,
and we've got the TLDR summary and the URLs that you can find,
the full articles of hack and use, current hack and use top articles.
So with that, I want to show a few other projects
that this custom component building functionality has enabled.
The next one is slightly questionable.
Please take it with a pinch of salt.
I put it everywhere on that tugging face space too.
And the idea came from, at the time, Twitter was very different.
So the idea was, could we like build a Twitter fetcher that,
given a username, could give you, this is really bad,
could give you like a vibe check of the account,
and we called it like, should I follow?
And it gets like the last, I think, 40 posts of that user.
Obviously after that, Twitter changed,
so I went ahead and built a master on fetcher.
You can also find that on the Haystack integrations page.
And the best way I like showcasing this is actually using my boyfriend's
master on account, because every time this tells me something a bit funny
about his account, once it called him pessimistic,
this time it called him sarcastic when discussing personal opinions.
So that's also open.
I think I linked to it in the notes as well,
so you can go ahead and try that out.
You just need to provide the full master on username.
Without the at at the front, that's a bug I haven't fixed yet.
Another thing that this enabled,
actually not only used the Haystack custom component functionality,
but also, I don't know if you remember when I showed the components earlier
with the two outputs and the two inputs, et cetera,
you can already start to imagine that you can actually have these pipelines loop too.
So the idea was, what if we have some meeting notes,
and we have our own GitHub repository,
and anyone who's used GitHub repositories,
you know that you can create those issue labels
that are very specific to that GitHub repository.
Could we build a system that, given meeting notes,
generates a list of GitHub issues specifically for that repository
that you're discussing in that meeting?
And could we actually then use those generated structured outputs
to query GitHub to actually create those issues?
Now, this is great, and our experience has been that actually a lot of large language models
are great at generating structured output,
but not necessarily in the structure that you need.
So it's going to be JSON, but is it going to abide by
what you need that JSON object to look like?
So the idea here was, okay,
well, why don't we create an output validator component,
and we use Pydantic for that.
This is all based on a tutorial that's up on the Haystack website right now,
and basically what we did for the GitHub demo was modify this tutorial just a bit.
In the tutorial, we said that we provide a Pydantic model,
and we said we need the output to be cities data,
where in cities data you've got cities,
and each city has a name, a country, and a population.
And then we used a GPT model,
and we saw that initially, for the first round,
we did get structured output, but it's not valid JSON,
or it doesn't abide by what we need that object to look like.
So the idea is, what if we provide back to the LLM,
the output that it just gave us,
with an error message from Pydantic as to why it's wrong,
why it doesn't abide by the Pydantic model we just provided.
So the resulting pipeline looks a bit like this
for our GitHub issues demo.
We want to provide meeting notes, and we want to provide a schema.
We give that to the prompt builder, the prompt builder exists in Haystack World.
Then that whole prompt is given to a generator that generates...
There's a one pass, like a first attempt at generating some structured output,
which is then validated by our output validator,
which doesn't exist in Haystack World, so this is a custom component.
And either you're all good, done.
But if it's not good, then we go back to the prompt builder
with the invalid reply that was produced, plus the error messages.
So for our use case, where you were trying to build this for Haystack,
this is not accurate, by the way, our labels are not exactly that,
but just for demonstration purposes,
we went ahead and built a Pydantic model called issues,
and we had to be very specific as to what our labels were,
because you can't make a query with a new label that doesn't belong to that repository.
And then we used our output validator.
And then this is where things start to look a bit complicated,
but the ginger templating is very useful here.
Earlier, for the Hacker News articles, you saw a for loop.
Instead here, we have an if statement.
So if we have an error message and invalid replies coming in from any component in our pipeline,
then this little section here, you already created the following output,
yada yada, is appended to the full prompt.
Again, at the end of the day, we ended up with a pipeline that looks a bit like this.
So do I have time? I do have time, right?
Four minutes. Okay, so the last thing I wanted to show is how these pipeline connections are actually defined in Haystack.
Oh, great. Okay, I have plenty of time.
All right, so ignore the corgis running around.
Can everyone see this, or should I make this bigger?
Okay.
All right, so I told you that before the Hacker News Fetcher component was very much pseudocode,
this is kind of boring.
We're basically making requests to the API and getting the articles.
Here, we're going to be using mixed trial through hugging face TGI.
Hugging face TGI is free, but it is rate limited, and you need to provide an API key to use it.
So you can go ahead and use this collab, but you do have to provide an API key.
And then you see the prompt template you saw, and here's what's going on in the Haystack pipeline itself.
We've got our prompt builder.
We've got our LLM, so mixed trial via hugging face TGI.
We just created the Hacker News Fetcher.
What we do is simply add those to the Haystack pipeline.
And then this is where Haystack can be quite verbose, but it can also mean that you can be very,
you can create very custom pipelines, and it can get a bit crazy.
You can have pipelines that branch out, loop back in, et cetera.
We're being very specific that Hacker News Fetcher's articles output is being provided to prompt builder articles,
which is going in here.
And then finally, the only thing missing to actually run this is Hacker News Fetcher is the only component here
that is missing an input.
All of the rest have been provided inputs through the pipeline itself.
So I can then define what the input of Hacker News Fetcher is when I do pipeline.run or pipe.run.
And then, optionally, you can also give more inputs that are not necessary for the others,
but for example, here, I'm using mixed-rull, and I wanted to up the created max tokens at the end,
so I can also provide that at runtime.
And that's it.
Thank you very much.
And you can also access the GitHub issues pipelines here,
but I'm happy to take questions if there are any.
Thank you very much.
Thank you.
Hi.
So in the Hacker News article summarizer,
to the LLM, the URL, and asking it to both summarize the article and also just print
back the URL. That appears to me a bit risky because it might change the URL. Do you consider
it best practice to pass the URL through some other way, or do you find it fine to always
ask the LLM to do that?
I love this question because try that hugging face space, especially with mixed trial a
few times and sometimes you just won't get the URL. Yes, and there are a few ways to make
this a lot better because actually the Hack and Use Fetcher component itself earlier is
just an API called Hack and Use and you have the URL there. So probably the best practice
here would be to have the LLM only produce summaries and the other component provide
an output of the URLs that was used to produce those summaries because yes, my experiences
a lot of the OpenAI GPT models do a great job of following that instruction, reference
this specific URL, but this is very much LLM based and how that large language model expects
to be prompted. Not every instruction works the same way with every model. Any other questions?
Oh, this one. Thanks for the presentation. I have a question on the prompt. I saw for
an if, is it specific to Haystack or? Not at all. We use Ginger for the templating
language. Actually, I will add a link to the Ginger documentation in the speaker notes
of this slide deck that you'll find on FOSM2, but that's all Ginger syntax, which comes
very handy because you get for loops, if loops, and you can actually start defining
your own custom functions for that Ginger templating as well.
All right. Give her a round of applause, please.