All right, so thanks for coming to this talk.
I know there are a lot of interesting talks
for interest talks, so thanks for choosing this one.
So I'm going to talk about such more specifically
about how to deploy one within an organization.
So in this today's talk, I'm going to cover these topics.
So it all started with engineers using multiple content
management tools.
And I'm going to talk about how searching became a problem
and how we use an OSS tool to address their problem
and the name of the tool is FES.
And how we overcame some of the problems with FES.
And I'm going to touch on the contribution side
of the project.
And I'm going to share some insights and observations.
OK, so here's the list of chapters, so let's go.
So just quickly, me and the team.
So I'm an engineer at Toshiba.
I've been recently maintaining the company's cloud
infrastructure and just trying to increase the scope
and range of automation through these activities
because otherwise, without increasing the automation,
we are doomed.
But more importantly, here's the list
of very capable, hard-working engineers that
made this project possible.
So I have a huge respect to them.
So now onto the background.
So in large corporations like ours,
we typically have lots and lots of companies.
And our team's job as a software engineering center
is to deploy and provide tools to other Toshiba companies,
like product development departments and R&D departments.
Now, we have about 200 units of deployment.
And we have these tools that are basically sort of heavily
OSS-based.
And now let me quickly turn on the user point here.
So we have a few more, but these four
are like a core of our tools.
So now as we diversify with the tools,
the searching became increasingly a problem.
And there were two major reasons.
For one thing, there is no easy way
to search laterally.
That is, sometimes we want to search all of these places
exhaustively to make sure that we're not missing anything.
But there's no easy way to do that.
And one more thing is, one more problem
is that as we found out, these tools are not quite cut out
for searching inside certain binary files, like PDF files
and Office document files, which are really something
we use quite often.
So what all of these lead to is that, ideally, what we want
is a single search box that, given a query, searches
all the places, like really no matter where they are
and where the documents are and what the formats are.
So that's what we are going for.
However, this is going to be a daunting task, though,
because such a tool would have to not only
be able to solve the two problems that I talked about,
but it would have to come with all the essential features,
both on the user side and on the admin side.
That is, we have to be able to easily set up crawlers
and all those things and run them and maintain them easily.
But there is a tool specifically designed for this task.
And the name of the tool is FES.
So next, I'm going to quickly talk
about what this tool is and what it's like to use it,
as I don't think this is particularly a well-known tool.
So FES is, as read me on the GitHub repo sets,
a powerful but easily deployable enterprise search server.
So an enterprise search here describes software
for searching information within an enterprise,
as opposed to web search, like Google and Dr. Go.
Now, FES uses elastic search as its search engine,
meaning that indexing certain binary files, like office files
and PD files, are more or less automatic.
And one notable feature of this tool
is that it comes with several types of web crawlers.
There's one for web pages, and there's also
one for file system, like a directory hierarchies.
And there's one for a database as well.
And all of this is to get data from many different kinds
of sources.
And if you look at the screenshots,
there is a search box.
And also, they have admin console.
And the search engine results page
looks only familiar to many, I think.
And this tool is developed by a company named CodeLyps,
which is a company that develops and opens sources tools.
And they have a lot of experience
engaging with OSS community.
And let's take a quick look at how this tool works
by looking at one of its core features, which is a web crawler.
And I think it's a web crawler.
It's basically a backbone of this whole system.
And I think the concept is familiar to everyone.
It basically crawls and indexes web pages, web page contents,
and uploaded files.
So the way you create web crawlers on Fisk
is you go to admin console and then
set these parameters for web crawlers.
Now, there are quite a few parameters here,
but I'm going to focus on a few important ones.
And first, there is, of course, URLs field.
And then you can include and exclude certain URLs.
Fisk respects robots.gxt, but certain robots.gxt
file doesn't disallow certain not so relevant pages.
So in this case, in that kind of case, this comes in handy.
And there are also fields like depth and max access count,
which you probably want to set to a very high large value
so that crawler is not going to stop pretty much early.
And then we came to the permissions parameter.
And I think this one needs a little bit more expansion.
So this parameter is where you can implement per user access
control.
That is, when you, I hope the phone is large enough,
but when you list users like that,
and let's say the crawlers index everything
and search is ready, then, and the users search something,
only the users listed there see the results.
So but notice that this setting is per web crawler basis,
meaning that if you have 100 projects on GitLab,
you're going to need 100 web crawlers, which is a lot.
So clearly, some kind of automation is necessary.
And I get back to this point later.
One more thing to mention here is that user name here
can be either users on FIS, FIS has its own users and groups,
but can also be users authenticated
by LDAP directory service.
There's an option to configure this on FIS.
So I hope that gave you some feel on how things work on FIS.
So let's move on to the customization part of the stock.
So no tools are perfect, and FIS is not an exception.
So we have to customize on patch FIS in a few ways.
So just quickly, here is a list of patches.
So our dev team engineers over time
wrote more than a few patches.
And the general quality improvement patches and bug
fix patches have been merged upstream.
But there are also more experimental patches
that are very specific to our problem.
And those are kept proprietary.
And I'm going to talk about two of those patches.
First one is the authentication for web crawlers.
Now, most of the pages of GitLab and RedMind
are behind logging pages.
And FIS has, so the web crawler has to authenticate itself
as it makes its way.
Now, FIS has a mechanism for this.
That is, you can create web authentication object
and attach it to each web crawler.
This works in some cases out of the box
if the logging form page is fairly standard.
But our GitLab uses SAML.
And then as it turns out, FIS does not support this.
We have to do some patching.
To just go over how the patching works
at the conceptual level, what we did
is we defined extra optional parameters
that Admin can write on the console.
That is, if there were parameters
starting with SAML underscore, the patched parser
of this form is going to pick them up and store them.
And later, web crawler is going to see these parameters
and recognizes that SAML authentication, SAML logging,
needs to be attempted and runs extra SAML specific logic.
So that's the patch one.
And then the second one is about repository contents.
So many of the repositories we have on GitLab
are several gigabytes in size.
And both GitLab and RedMind have pages
to view repository files.
So in theory, if you wait long enough,
web crawler is going to index all these contents
through those pages, in theory.
But this turned out to be a complete non-starter
because it's too slow and quite understandably so.
And the reason is it's just so the web crawler is
going to make HTTP requests.
And GitLab fetches the file, just one file from Ripple,
and then renders it to the web page.
So there's just too many steps to just get
the content of one file.
So what we did is to first clone the repository contents
to a local file system, and then run
this file crawler, which is a crawler for directory hierarchy
and do everything locally.
Now this more or less solved the problem of speed.
But one problem is that since everything is done locally,
when the search indices are stored,
these things are the filesystem paths.
So we had to remap these to the URL
so that later when the user clicks a link,
it takes the user to the repository file page
on GitLab.
So what we did is, again, we defined
custom optional parameters that the admin can
write on the console config parameters field,
specifically this prefix URL and map URL parameters.
And when these parameters are present,
the parser is going to pick them up,
and then later the web crawler is going to see these parameters.
And then if these are present, it will perform remapping.
So this is the conceptual overview
of how this patching works.
And now most of these parameters are
related to the web driver client on this.
And there is information about this web driver
client and Fist 14 issue, as since the web driver client
is discontinued on Fist 14, and Fist 14 is latest.
And Fist 14 has playwright-based crawler,
which is still in development, and the information
in the appendix.
Now I'm going to talk about another important subject,
and which is automation.
And as you might have guessed, our configuration
grew more and more complicated, as it always does.
So for instance, we have way more than quite a few configurations
for each Fist instance.
And so there are lots of manual edits of config files.
But these are taken care of by Ansible and Docker file,
and I think that's standard.
But perhaps a more interesting instance
is we have to create several hundred web
crawlers per Fist instance.
And the reason is, typically on GitLab,
you have projects.
And for each project, you have members.
And what you want to make sure is, when
user searches something, you use it
to see only the resources they have access to.
So to automate the creation of web crawlers in such a case,
Fist has APIs, just like GitLab APIs.
So to explain how this is handled,
to look at the sample script, you
can combine the GitLab APIs and Fist APIs.
First, you can use GitLab APIs to get all the projects.
And then for each project, you can get the list of members.
And then for using that list of members,
you can create web crawler.
And this is where Fist API comes in to create web crawler.
And then you can also create web authentication object
and attach it to that web crawler.
So the Fist APIs are mostly like GitLab APIs.
And for those who have used them,
I think they'll be fairly intuitive.
So that's the quick intro to the Fist APIs.
And I'm going to share some insights and observations
that I can make.
So the first is did Fist solve our problems?
And the answer would be definitely yes.
So the users can now search across tools
and inside binary files.
And this turned out to be quite powerful as, for instance,
even if the file is like a DocEx file,
or even if it's a legacy doc format,
and even if it's in a very obscure location
in a very old repository, deeply nested places,
users can actually find texts inside the file.
So this turned out to be quite powerful.
But it's not without a problem, though.
So one problem is the performance.
If you want to index contents of GitLab, which
have, say, several hundred projects and several thousand
issues using Fist Instance running
on this level of computing resource,
it takes us about a couple days to index everything.
And this is something we are trying to improve on,
like how to incrementally index contents
and using some other techniques.
So for now, that's it for this talk.
So thank you so much for listening.
And I want to open this up to the Q&A session.
So you have a different type of resources, right?
So every resource has different properties, I guess.
So I do index the same properties for every source.
So you have a set of properties to index.
Are you only indexing content?
So I'm trying to understand the question.
When you're indexing documents, you
are indexing the content of the document, the text of the document?
Or are you indexing also some of the properties,
like the name, the title, the author, the?
I would say yes.
So let me repeat the question.
The question is, whether this first index, only the contents,
like the text inside the file, or does it index things
such as metadata, like title field, or PowerPoint,
or they have several metadata.
Yes, it does.
It indexes those metadata as well.
And those are handled by, most likely, the elastic search.
But yes, those metadata are indexed.
And actually, it shows the metadata title
if they are present on the search results page.
So yeah, I guess that's fine.
The second one is a quick one, so are you accepting contributions
to the project?
It's an open project, and you are accepting
contributions and new plugins?
I'm sorry.
So your question is, whether we are accepting contributions
to the?
So on our side of it, I don't quite think so.
So you have a catalogue of connectors to all of them,
and so on, to elastic search, and so on.
So for instance, I was thinking on CMIS,
that is a standard for content management.
So I was thinking on trying to contribute this new connector
to the best project.
Is that an option?
I'm sorry.
I'm trying to understand the question.
You said something about connectors, right?
I'm naming that connector, but I don't know if connectors
is the right word for you.
I mean, so you have a specific browser
for every different system, right?
You have a browser for elastic search,
a browser for office, a browser for set options, right?
So we can add new browsers for different systems.
Yes, yes.
I mean, FES has seven different crawlers
for different types of resources.
And so the question is that are we
going to be accepting new types of crawlers as contribution?
Is that the question?
Yes.
It's not right.
We're kind of sort of like a corporation
and then working on our side, working
as a project inside a corporation.
So we buy ourselves, not at the moment,
accepting a contribution to the project, but yeah.
So yeah.
Perfect thing.
Another question?
Yes.
Are you going to publish the slides that you presented?
OK.
Are the slides that you presented,
are you going to publish them somewhere?
Then they are on their own page.
Yes.
Yeah, they are on Fesda website.
So you should be able to download them.
I'm sure I'll be able to.
That's the short one.
You said indexing takes about several days
for this project you have.
Yes.
It's about re-indexing.
So if content changes, re-index all the time a new index,
making the whole index new, or is it fast?
Just re-index certain documents.
So the question is, let's say after index everything,
and then in the subsequent run of web crawlers
and all kind of crawlers, are they updated incrementally,
faster, or do we have to re-index everything?
So the Fesda tries to crawl efficiently.
It tries to ignore the contents of web pages
that haven't been updated.
It checks the last modified field.
So the mechanism of incremental crawling
is not very ideal.
For instance, the last modified field is not quite
well-inforced, for instance.
It's only the certain types of static web pages use it.
And so there are a lot of unnecessary re-indexing
happening.
But there are some mechanisms to only index things
that have been updated.
But I've got to say that that mechanism is not very well run.
And there are a lot of re-indexing.
And then the subsequent crawling is not as efficient
as we want.
And then it is.
Yeah.
Yes.
OK.
Thank you very much.