Good morning everybody. I am Eric Borra. I am an assistant professor in journalism and
new media at the University of Amsterdam. I have a background in artificial intelligence
and I have been a tool maker with the Digital Medicine Initiative for about two decades now.
And one of the reasons why I make tools is to understand new technologies. And today
I will talk about large language models and particularly their use in social science and
humanities research. So who of you has used chatGPT? I think most of the Western world actually
has used chatGPT in the last year. ChatGPT is based on a large language model where you ask a
question and get an answer. But social science and humanities researchers have also found that
you can just send it instructions. And you can instruct it to do all kinds of things such as
sentiment analysis. So you prompt. It's the way of interacting with large language models. You
specify a prompt and you say classify this tweet. You input a tweet and it will give you a nice
classification of that tweet. You can also extract entities and actors. You can include topics and
teams. So you have the prompt I just showed. You enter a New York Times article and it will
extract all these things for you. It will extract country names, organization names, people names,
specific teams and topics. And it's actually pretty good at this. At least chatGPT is. And I'll
discuss other models in a second. And it's not just named entity extraction or simple classification.
It also, like, somehow extracts teams and it can abstract from the text. And researchers have been
using this, for example, to extract narratives from posts. So here's a very complex prompt, which you
can see in the slides after the talk. But there they use this prompt to go through many, many Reddit
posts, hundreds of thousands, to find out whether there were any conspiracy theories in there. And
they devised a prompt to draw out these narratives. And they actually found that LLMs worked really
well in extracting conspiracy theories as well. So researchers have been using this and they have
been looking at all the tasks that are typically done in social science and humanities and are
starting to test whether LLMs can help in doing these tasks. And there has been a lot of research
in the last year, especially 2023. This is just a really small snippet of this research.
But this research also comes with problems, which I'll touch upon in a bit. But their use is also
understandable, the use of large language models in social science and humanities research, because
they seem to ease and speed up previously difficult and laborious tasks, such as classification,
extraction summarization, and so forth. And they're actually employed as junior research assistants.
Now, while this may seem useful somehow, a lot of people seem to be using chat GPT. Actually,
all the papers I've just shown, they're based on chat GPT. And chat GPT comes with problems,
because it's a platform service. And platform services, as I guess all of you know, are volatile
black boxes. You don't know what's going on in the back end when they update their model, when they
align something differently or sensor or whatever, whether it's getting dumber or not. You basically
don't know. Chat GPT is also very expensive. If you're using the API, you pay for a request.
This is one research project by Miguel Escobar of Arela, who calculated that to process the one and
a half million news items he had in his corpus. He'd need $150,000. It's just too expensive.
There are, of course, also privacy concerns with chat GPT and other platform models, whereby
with chat GPT we know that whatever you input into chat GPT is also used as training data for the
LLM. Users have also found personal and private information resurfacing from other users, etc.
So if you think in terms of open science, replicable research and ethical
research with privacy concerns, you basically cannot use these models, even though you can go to
privacy.openai.com and state that you don't want your inputs to be used as training data.
Well, so how to deal with this? Can we use LLMs in social science and humanities research?
Fortunately, the answer is yes. Chat GPT is not the only model available. You probably heard of
Google Bart or Gemini. You may have heard of Clodes, which are other platform models, but there are
also a lot of publicly available models. All the yellow ones highlighted here, and this is only,
I think, until the second quarter of 2023. Since then, a lot of new models have appeared,
most notably Mistral, for example, the French model, or Mistral, the 8x7B model of Mistral,
which are really good and are catching up on the performance of chat GPT. Publicly available,
however, doesn't mean that it's open, that it's open source, that it's free, because there is this
whole infrastructure and apparatus to train models, to fine tune models, to use models in your own
work. And you see all the orange and red here. Most of these models aren't open,
or have different licenses, etc. So it's not, yeah, you can't just use another one. You need to think
about these things. Two other considerations before I go to the actual tool. If you use
the same prompt in different models, you'll get different results. And this is actually
the same prompt in a series of image models, but you can visually see how results may differ.
This is something to take into account. And last but not least, there are a few technical
parameters in using LLMs. And one way to control differences or variability in the output of LLMs
is through the so-called temperature parameter. If you set the temperature to zero, you'll always
get the same result. The most probable or most likely outcome was if you increase the temperature,
there is a chance of less likely outputs to be included in the results. But again, all these
papers, none of them mention temperature, whilst it's a very important parameter. Last but not
least, this is work I've been doing with Maichieu, small syntactic differences in a query that's
semantically the same may lead to different results as well. So you need to test your prompt for
robustness or consistency. So summarizing, open AI or platforms like chat GPT are volatile black
boxes that cost a lot of money. There are issues of privacy and security. There are different models
which have different licenses, which have different results. LLMs are not deterministic and small
changes in prompts may lead to different outputs. So we need research interfaces where we can
control for such things. We want to be able to do open science with LLMs. So how do we take into
account the volatility of platforms, the robustness of research and its replicability and explainability?
And this is where I started tinkering with a tool I called prompt compass, or actually I had chat
GPT call it prompt compass. And it's a research interface. It's not a chat interface. It allows
you to take into account all these considerations that I've put up. So you can choose various local
models. It has default parameters for replicability. It contains a library of research prompts,
allows for batch processing user input, and allows you to evaluate prompt model combinations.
Do I still have some time to demo this? Cool. Let's do that. So prompt compass is available
on GitHub. We also run it at one of our servers. The design doesn't really shine on this beamer,
but anyway. Here you can select various models which are loaded from hugging face. You can easily
add a new model and select one of these. You can find out more by clicking on the model card
and then see what the model was made for, how it was trained and so forth. All these models are
loaded from hugging face, which is like the GitHub for language models, but we can also
choose GPT for where you and then or any other model of open AI or platforms and then
enter your API key and go over that. You can go into the settings which are default sets to
replicability. There's a little explanation of it. There are a lot of prompts extracted from the
literature and from actual research. And you can input your own prompts like this. You can or you
can adapt existing prompts. And then you can provide user input either line by line or upload a CSV.
And then if you click submit, the selected model will be loaded. And each of the lines will be run
through the model with the indicated prompt. So in this time we chose sentiment analysis,
which says you're in advanced classifying AI, you're tasked with classifying the sentiment of a
text, which can be either positive, negative or neutral. And this is where we'll input or loop
over our inputs. So in this time the user is happy, it's classified as positive. When user is just
a user, it's classified as neutral and the other user is a liar, it's classified as negative.
And this tool is not the end all go all tool for working with LLMs, but it is a way to test
models, to test parameters, to test prompts, to test the robustness of prompts and to
get all this into easily digestible outputs CSV.
So far for the demo. The technology is used, it's really simple. I'm not like a hardcore coder,
but I'm like more of a tie some stuff together coder. Streamlit is a Python interface for
making easily making web applications of machine learning tasks. Lang chain is a very bloated
way to easily connect LLMs and to work with LLMs and prompts and Huckingface is the place where
all these LLMs are stored. We run this on a 24 gigabyte GPU, which is a bit expensive,
but it's not very expensive. Like each research group should be able to get one.
And I mean, yeah, so to get back to my rent against platforms, making LLMs
locally accessible makes them stable and replicable. But we cannot run the biggest models unless we
have access to bigger infrastructure, which we sometimes have. But this is really meant for
researchers that want easy access to local models. I made a video tutorial,
tutorial which you may want to watch. And maybe there's still room for questions.
Just three minutes.
Atlas TI is a rather big and well-known software package for qualitative coding, right?
I'm not sure why they chose to only use Chatchi PT. But yeah, I mean, we've had experience with
local LLMs that you can actually also do similar things with extraction and coding.
So I would definitely be in favor of actually using local LLMs. On the other hand, if you have
proper validation procedures such as intercoder reliability and F1 scores, etc.,
you can get a long way with Chatchi PT because human coders are also fallible and may also
be different today than they were a few weeks ago. So it's not that it's not possible or not
usable at all, but you should be prudent, I think.
You said it's mostly for testing the models, but how big of an input file do you think?
We've run this on more than 100,000 lines of CSV, I think even more. So in the digital methods
winter school and past summer school, we actually run a lot of prompts through it.
And it seemed to work. Sorry. It was asked how big of CSV files it could handle,
and I answered more than 100,000. So it's actually also used in production for
relatively small-scale qualitative research, but it's not limited to things you could do manually
anyway. So let's switch.