All right, so I'll start a little bit earlier of the, of the, actually the 2 p.m.
Just the thing that, so I'm wearing this t-shirt because I'm one of the day room manager here
in this, this room and I'm taking over actually a talk slots that has been cancelled.
So we were supposed to, to, to hear a talk by Maria Arias de Reina, who couldn't make it today,
unfortunately. She was supposed to talk about data flowing in the right way, which is a talk
about a tool called Cauto, which implemented data workflows with a low-code, no-code approach.
This is, this is what it looked like. So, of course, I can't talk about this tool because I,
I don't know it. It actually looks pretty cool. So we are very sorry for Maria not being here
and we hope we can host her next time. So I will speak about a project, a research project
called Ricardo in the digital mentees, which I've done with, oops, sorry, how are we going? Yes,
I worked with Beatrice de Dinger from Sciences Po, Centre d'Histoire, so she is historian and I am
Paul Girard. I am from a company called Westware, a small agency specialized into developing data,
data-oriented web applications, and we do work a lot with researchers. Today I'm here to talk
about how actually a collaboration between a researcher, Beatrice, and a data engineer myself
can be fostered by using data control systems. By data control systems, I mean making sure we
care about the data we are using in the research, by documentations, version control, and also quality
control. So the research is about the history of trade. So we, together with Beatrice, we build
a database of trade flows between nations, well, between geopolitical entities in the world,
in the 19th century, which means that we have the main data, as we know, how many, how much amount
of money in different currencies has been exchanged between different geopolitical entities in the
19th century. We know important exports, and we know this with a bilateral view, which means that
we know the trade from France to UK, for instance, and the rivers too, from two different sources,
which makes quite a nightmare to deal with, but still. So this is basically the main publication
we already achieved. So we started by releasing in 2016-17 a web exploration tool, I'll show you,
and also a paper about how we build this database. And then in 2021, we released a new database
called Geopol East, which is basically a data set that explained trying to track which geopolitical
entity, so I'm using this weird word, because we have countries, of course, but we also have
entities that are part of countries, but we also have trade with entities that were colonies at
that time, and all those kind of weird political statuses. And because of that, we built this Geopol
East database where we tried to track which geopolitical entities were controlled by which
other one in time. And recently, we released a new version of the database, adding 230,000 new
trade flows. And this releasing of new data, because actually Beatrice discovered new archives,
new archival book about trade, this massive updates needed a tool to make sure we can
actually release the data, which are cleaned the structures the way we want it, without having to
deal with all those kind of issues manually. I will speak about that a little bit later. So this
is what the website, the main website looks like. It's a web application you can go where you can
explore basically the trade of the world in the 19th century. So we have different kind of
civilizations. I will not go through all of them because I don't want to focus too much on this
today. Well, if you have questions about this, we can go back to that later. We also have this
website Geopol East that allows you to actually explore the political evolutions of the links,
sovereignty links between the different entities. I'll show you a little bit what it's like just
afterwards, I think. So just to be totally honest, this slide deck is actually something I already
presented in another conference. So I wanted to speak about the visual data exploration tool we
made, the frictionless data integration. So this is the main point I want to speak about today,
point two. And also the third point was how we can actually analyze heterogeneous data in the
long run, like one century of data. My main, I will try to convince you that actually using data
integration is a very nice and important tool to foster this long-lasting collaboration we had
between Beatrice, historian, and I, data engineer. So about collaboration, I just put a link to a
conference I gave a few years ago about this specific subject. So visual data exploration,
so I will really go quickly on this part to focus more on the second part. Our main objective here
is to propose basically a tool, a set of interactive data visualizations on the web,
that all those researchers are basically people exploring this, to change points of view on
the data, looking at, for instance, the total trade of the world, then focusing on one specific
country, then on one specific currency, and to be able to add all these different ways to look
at the data in the same tool. We also like to offer visual documentation, like visualization is
a very nice important tool to spot issues or surprises or errors in the data, and to unfold
the complexity of the old phenomenon. So this is, for instance, the world view. So we are able
to retrace the world trade in a century, but as you can see, there is more than one curve. So we
have different ways we can calculate that actually from the data. We can, for instance, take some
researchers that really did re-estimations of this total trade by correcting sources and all
this kind of stuff. So that's one way to do it. That's the green curve in this visualization,
but we also can, we can actually sum all the totals we have in our data. This is the yellow one,
and we also have the, so here we are summing all the flows we have, the yellow, and the red is
more, we are summing only the total that were in the archive books, and it's not the same thing.
If you sum what we have, or if you take the sum that's done at the time, you don't have the same
results. Welcome to the nightmare of dealing with archive data. In this visualization, for instance,
we are focusing on one country, Denmark, and then we can actually spot the trade on the
long run of this specific country, and we can also visualize, so here is Germany's on the
Rhine, we can also depict actually not only the total trade, but also the difference trade,
bilateral trade between Germany and his trade partner along time. Okay, so this is an objective.
So geopolitists here, for instance, we see that when we talk about Germany's on the Rhine,
what are we talking about? And we are talking about a geopolitical entity that had a different
set of seasons on time, you can see here, and then you can see on the bottom line, which all the
parts, all other geopolitical entities were actually part of Germany through time. Because
sometimes we have trade with only Saxony or Waldeck, and we want to know eventually if those
entities are part of another one. So frictionless data integration. So we are using data package
from frictionless data here from Open Knowledge Foundation. So actually, there is a talk from
Evergony from frictionless team later today in the online part of our room. We'll talk about the
new tool fostering data package. And actually, I'm very interested into that. But I will talk
about what I've done myself. So about this project, the main thing we do is
versioning the data. So we put data as CSVs into a version control system like Git, simply. Here
it's on GitHub. And you can see that you can track actually just the same way we do with code,
who changed which data when and why. So here, for instance, is Beatrice, who actually corrected
the typo in the flow number, adding the comma at the right place. And we have the commentary here.
This is very important to keep track of what's going on because we have like hundreds and even
thousands of files like that. So it's very important to have that also to know if we have issues,
if that happens, where it comes from. So data package. So data package is a formalism where it's
basically a JSON file where you will actually describe the data you have, adding a documentation.
So the first interest of using data package is actually to document your data set to make it
easier for other people to actually understand what you want it to do. And it's very important for
publication at the end, open science. So here we have the title of the project. We have the
license, the contributors. That's also very important to have. And then we describe resources.
Resources can be seen as a data table. So here, for instance, we have rec entities. And for each
resources, which is a CSV here, we describe the fields we have in the table. So we know that
the rec name table is a string, it's unique, and it is required. So it's very like a relational
database schema. It's very kind of the same spirit, but in a JSON format. The reason why you can do
that, as I said, is documenting. The second reason is actually to control your data. So doing
right, driving data validation. So if you have a data package described like that, you can then
use a Python library, frictionless, it's called frictionless, which will actually check all your
data line, if each data line you have, respect the schema you wrote. And if it's not the case,
it will actually provide you with reports, with errors like, for instance, here, I have a foreign
key error because the modified currency year is not known in the table that is supposed to
have this data. So it's a very nice way to actually, we get new data, and then we check, okay,
where do we stand about what we want to achieve at the end, which is to respect the data package
formalism we wrote. So that's very nice. It's very cool for data engineers. But as I said,
our goal with Beatrice is that we work together. And she, because the thing is like,
when she enters new data in the system, she has has an historian to take decisions on how to
interpret the data that were in the archive. I can't. That's not my job. That's not my
responsibility. I don't have the skills to do it. So we need something to allow her to actually
correct the data, update the data that comes in into the data package format. And she can't use
common line tools in Python script and that kind of stuff. So we need something, we missed a tool
here to let actually humanist researchers, in this case, but people in general, to be able to
interact with the data flow with something else than actually two technical interface.
So we built, we developed a specific web application that actually helps Beatrice to
integrate new data by using the data package as a validation system. So all of this is done in
JavaScript. You also have a JavaScript library for data package. So basically this is the steps.
So the idea is Beatrice will upload a data spreadsheet, so a new data file, transcription
of one new archival book she found. The tool which first checks the spreadsheet format,
saying like, do we have all the columns we want and everything. If it's correct, then it will go
through all the data points, checking all the errors and grouping them to make, to propose a
creation interface for her to correct all those issues through a form. And we tried to do, to
develop something that makes this process, which is tedious and a long process, as easy and as
fast as we could for her Beatrice to actually go through this. At the very end, this tool will
actually commit to and push, commit it into a Git repository and push it into the GitHub repository.
All of that done into a serverless web application, which means that I didn't have, had to
introduce a Git command line to Beatrice neither. The tool does that for her.
So this is what it looks like. So it's a React web application. Here we have
the Schema validation summary where we see for the fields. So the different fields we have,
for which we have errors, the kind of the error we have. And at the end, we have the
error overview, which says how many rows that has an issue. For instance here, in the source
column, we have two different invalid values that impact 169 rows. The idea here is to correct
all this group of 1 and 169 rows with only one edit. So once we have all those errors,
basically the process of workflow using this tool will be to go through the error groups
one by one. The web application will actually generate a form with inputs
to actually help Beatrice to decide. So in this example, we have a value for a partner. So partner
is a trade partner. So it's a geopolitical entity. Here it's in French. It's il de selon. So we use
English-based vocabularies to translate the partner. So we need to decide what is il de
selon in our vocabulary in the rest of the data. And this is where we have a search input
where actually Beatrice can actually search for selon, which is called in our vocabulary
Sri Lanka Selon in parenthesis. And once she chose that, actually the tool we correct
this column and put the data at the right place to make sure we will translate il de selon to the
Sri Lanka Selon. At the end, once she went through all the process, we have somebody here explaining
like all the corrections she made. So in the first here line, for instance, a year was misspelled.
All that kind of thing, we change the source name and everything is fixed. So once all the errors
have been corrected through the form, the data form I just showed you, then she can move on with
the last step, which is actually to publish this new data that is now valid because we
know it's valid because we can control it with the data package into the GitHub repository.
And this is how basically the React web application will really prepare the data.
So I could go into details into what does that mean later and make it possible for Beatrice to
actually take the right decisions to adapt the raw data into the data package we worked with.
So I have a little bit more time. So this is the analysis. Maybe I can try to deem
all a little bit of tool live. Okay. So the very important thing is like it's a
several less web applications. So here it's on my local host on my laptop,
but actually it's hosted on GitHub directly. So what is the media lab? Actually, a lot of
this work has been done at Sciences for Media Lab, my previous employer. So congrats to Zen2
because they contributed to that work too a lot. So this is a tool. It's hosted on
GitHub.io because it doesn't need any server. All the logging process with GitHub is done
through a personal token, which is a very specific key that you have to produce in your
GitHub account for once. Then you use that as a login mechanism. So this is what it looks like.
Once I am logged in, I can fetch the data from GitHub to make sure I have the last version
of the data before adding new things. Then here I can prepare the little file here, which normally
should have some errors. So the first thing here you see like this green message here on the bottom
says that actually this CSE file is valid structure-wise when the errors of the columns are good,
which is a good first step. And then this is all the errors I have in my file. This is a nice
step because you want to overview what kind of mess you are going to on WR before starting the
process because if you have too many, maybe you want to do that later or asking for help.
So once you've seen that, you can start. So this is basically all the thing we have to do.
So this is the first one. I can move to the next error or go back, even though I haven't corrected
it yet. And here I say like, okay, so the value, I don't know, whatever. This character is not
actually a unit because the unit should be an integer. Yeah, it's true. So it's better with the
one. And I can confirm this fix. And now we're good. Unit is one. Now I move to the next one.
You see here I am in two on nine. So all the process is also trying to make that as smooth as
we can. So as soon as I fix it, so here I have, it's written in French, it's 1938. But actually
we want that to be an integer again. I don't want the later version of the year. So we understand
how it tweets as a number. As soon as I confirm the fix, I move to the next page. So that we can
try to make that process as seamless as possible. So here I have a source. So this is the foreign
key. So in the data table, the source is actually, it's a key that is referring, which is referring
to the source table. And say like, so here basically foreign key source violation. So it means that
this sort doesn't exist in our system. So here I have two choice. Or I was supposed to, okay,
normally I should, I should, oh yeah, sorry. So normally, okay, so whatever. So I can search for
it and find it. Or, and in which case it will, the edit will be only replacing the key. Or I can
create a new item. And here you can see that here I'm creating a new source because sometimes the
source doesn't exist yet. So I have to go through all the, you see this form is much, much longer
because here I'm creating a new line into the source table. I will not do that because it's too
long. I will just give me something, please. And that will make it, okay. And so on and so forth.
Again, we have an issue with the, sorry. This example is a little bit up. Okay. Here it's a
Trinidad and Tobago. It's a geographical line time. I don't know because it's a A. Trinidad and
Tobago, not A. Okay. And we're good. Australia with a lot of E at the end is not correct.
It's Australia. Yep. Sorry, it's very long. Yeah, whatever.
A dollar. And let's say it's a scrap. Don't do that, right? But, okay.
Okay. So you see, that's the important point is like, so we are based on the data, in the data
package. So we are using for NKIS specification and so on. But actually, we had to add specific
forms for our case. So the application here is not generic. You can just put a new data
package of yourself with your data and it will not work because we had actually for UX and UI
reasons to make specific cases like that where the forms are not exactly as the data package
described it. It was too complex to make it very generic. But actually, with more work,
that could be achieved maybe at some point. And actually, the talk from Evgeny this afternoon
will talk a little bit about that kind of stuff. So here we are. I'll stop the demo here.
Just to finish, why do we do all that kind of stuff? Because we want to analyze trade in the
long run. We have lots of trade values as you can see. A lot of trading entities, very too much.
And then at the end, we try to, or this is a visual documentation where we depict the kind
of different source we use in the database. And at the end, we try to do something like
that where actually here we have the trade of the world in 1890. So each nodes here, circles,
are geopolitical entities. And the links between them are the trade of that year. So it's total
trade. We could choose import or export here. I just summed it up. The important part here is like
the color here is based on the type of geopolitical entity we have. So in this
orange or kind of yellow thing, it's what we call the GPH entity. It's entities that geopolitical
entity we know, mainly countries but not always. In green, those are colonial areas. So it's not
a colony. It's not a country which is a colony. It's like French Africa. It's like we know it's
colony. We don't know which one exactly. Like here, for instance, we have European Russia,
which is a city part of. It's from Russia, but it's the European part of it. And this is what
we find in the archive books. So we can't really decide what that means exactly. And we're trying
to analyze this kind of, so we have this very, this gap between very heterogeneous data, very
difficult to interpret, but still try to do a quantification and analysis like this network
on top of this very complex and rich data set. I think I'm good with what I wanted to share with
you today. We can move to question if you have. Yes.
There was a slide. The slide had a title which was development of a specific web application
to integrate new data. That one. Yes. You might not have time to tell me, but please tell me
if you have time at some point. What was the conversation with your historian? How did this
happen? What did you actually do to plan? Yeah. So the question is like how Beatrice and I ended
up deciding to do that. So basically, the whole point is very like the collaboration, because
we worked without that for a very long time. And the process was we had to meet in the same room.
I was doing the script, checking the data, editing the data, because editing the data in a spreadsheet
that doesn't mess up your numbers and everything is not easy, actually. And we were working together
on that. It was necessary and actually very nice to do, because we had to exchange. So she was
explaining me why she was taking this decision or not, and she was taking this, and I was just
putting data. But at some point, we ended up with the fact that we had so much more to add
that this process couldn't scale, basically. So we had to find something else to make sure
she could do this process on her own. And I would intervene once the data is in the GitHub
repository, checking myself with quantification and script and everything again, because you
still always need to check everything many times. And then it makes the whole process much more
smoother. Yes?
I have a question about this slide, too. So would there be any benefit in you committing it to GitHub at the very top of this process? Would there be any benefit to you committing the data set into GitHub at the top of this process after you upload it so that you can compare?
Yeah. So the question is like, would it be beneficial or possible to actually commit the data before checking it and put it into GitHub? So yes and no. The reason why we don't do that is because the first one, because I need batteries to take the decision of documenting the raw data to make it compatible with all the nice visualization I showed you.
And she needs to take those decisions. She needs to do it. So that's why we put this data into the GitHub after she has done this work of data creation. We could actually host the data as a raw file first and then do that later on the kind of stuff.
Still, we still need a web interface that lets Beatrice, the historian, take the decision. So no. Any more? Yeah?
Yes. So this tool I'm using here is actually brand new. It's a Gefi, but on the web. We are working on this with my company, Westware, and we are very close to release it. It's basically the same thing as Gefi, but lighter version and a web version.
It's already there, but you shouldn't go because it's not live yet.