Thank you.
We start with a small introduction to have a bit of context about Djubaco.
I'm Mathieu Goetje, I'm a freelance developer and I'm working, my main client is the Qwix
project and for there I'm the lead developer of Flimzim.
What is Qwix?
Qwix is a project to provide content where internet is not there and the question we
try to answer and we have answered is how to distribute static websites.
And for example, if you don't know all Wikipedia in English, it's 95 gigabytes and it's 6.5
billion articles and media.
And to do that we use the Zim format. It's an archive format for web content and content
is partially compressed so you can compress textual content or not compress images or
videos and you can do a random access without initial decompression so you can access the
content inside the archive directly.
It works well and pretty efficient but there is a few flaws within the design and the archive
is really tied to web contents and to Qwix and you cannot add another metadata but the
question I tried to answer is could we reuse the Zim format, the good idea of the Zim format
and do better and more generic.
So here is Djubaco.
Djubaco is a Japanese name for the bento boxes and it's more boxes you can compose the way
you want depending on your needs.
And Djubaco is a new format independent of Qwix project and this is a good idea of the
Zim format but generic.
And Djubaco is a meta container.
It tells you how to store things but it's up to you to decide what do you want to store
and how do you want to organize them and there is a reference library written in Rust.
The feature of Djubaco, it's mainly read only, archive are mainly read only, this is selective
compression so you can compress the content or not.
No initial decompression needed and you can do random access on the archive.
It's configurable so you can decide which property you want on the entries.
There is an extension system so your user can download an archive and they can download
extra content to add content to the archive you provide.
It's embeddable in another file and it's composable so you can compose different kind of entry
together in the same container.
So it checks them and a few features to do, signature and encryption, direct access to
uncompressed content, content deduplication, modification, different patch between archive
and overlay.
Let's have a quick tour on the internal structure.
The Djubaco containers are organized around packs.
There is three kinds of packs, manifest packs, content and the directory.
Each pack can be stored individually in a file in the file system or they can be put
together in one file and then you distribute this file to your user.
The manifest pack is the main pack, this is a pack you will try to open when you want
to open a Djubaco container and it's mainly a list of all the other packs of the container.
The content pack is a pack which contains the raw content, compressed or not, and without
any metadata.
The directory pack is where you store the entries and the entries can print to contents
in the content pack.
This is a configurable part of Djubaco and inside the directory pack there is entries
with a specific schema so you have to define the schema and the schema is the series of
properties and their types.
The content is just a property, it's a link to the content in the content pack so you
can have entries at that point to several contents or no contents at all and each entry
schema can contain violence, it's kind of union or enum in Proclamation EC or REST and
you can have different kind of entries inside one directory pack.
Each use case, why you would like to use Djubaco?
The first use case is file archive, there is two arcs which is an equivalent of tar
and here we have one kind of entry with three variants, file, symlink and directory, all
three variants share two common property and for example the file variants add the pointer
to a contents, symlink and the directory just store the first and pointer to the first entry
and the number of entries in the directory.
So it's kind of an organization and three structure as a file system.
There is no index property for now but just mainly because arcs is pretty young and I
don't want to bother with them while testing arcs and Djubaco but it's hard.
It's a file archive so we can compare a bit arcs with tar to see how Djubaco and arcs
perform.
If we take the Linux source code, the full Linux source code is more than one gigabyte
and both our tar and arcs are compressing the source code is about 130 or 14 megabytes.
Crescent time arcs is a bit faster than tar and expression time we are pretty close arcs
is a bit slower but we have someone pretty close, both tools are pretty close.
What is interesting is when we try to list the contents of the archive, tar took almost
the same time that expression because to list the contents in the tar archive you need to
uncompress all the contents and arcs is very faster because the list of the entries are
separated from the contents itself.
If you want to extract only one content from the archive and we try to, what's that called
dumping and when you try to dump a third of all the entries, you can see that arcs is
really really really faster and the same way extracting one entry from the tar is pretty
close from the time of listing the contents the same way as you need to uncompress all
the contents of the tar archive and arcs you can locate the content and do a direct access
to the content without uncompressing other contents.
Once you think that we can do that is mount the archive, directly mount the archive on
the file system and if you mount the archive and you do a diff of the content between the
original source and what is mounted, if you do a diff between two plain directories it's
a bit less than a second with arcs it's four seconds and half and tar is an estimation
it will take something like ten hours to do the comparison.
You can do something even more interesting with a mounted file system or with a mounting
Linux source is compiling the kernel so if you compile the kernel on the plain file system
it's a bit more than half an hour and if you compile the kernel using the mounted arcs
archive it's a bit less than an hour.
What is interesting here is that the compilation is made with G8 so there is eight processes
and arcs a fuel file system is monostated so there is a huge bottleneck for now but
if we move to a multi-threaded fuel file system it should be even better.
The use case is the GIM it's an equivalent of kind of equivalent of ZIM format there
is two variants only and here we are storing the entries as a plain list and there is no
tree structure and the GIM binary just integrates a small HTTP server looking for the entries.
What we can do also is we could replace for example RPM and DEB with arcs or things based
on jubacca so you could download your package and not extract it from the file system just
open it directly and even a GVL or debugging fault that could be put in specific content
pack of the same archive so it could simplify the management and you will not need to have
different package to different sub-type of contents of your packages.
OCI containers are based on Tor you need to extract them on the file system before running
a container so you could just use arcs among them or you can even use directly put different
layer in different content pack and so the wall images will be one jubacca container.
File format almost all file formats are in fact container for other content so you could
use jubacca to just organize the content you want to store what you want for your own project
and your own file format.
Websites jubacca is written in rest you could run it in wasm and so jubacca could run you
could load your jubacca archive in the browser once and just open it directly in the browser.
Backups backup jubacca is almost incremental by design if you reuse the content pack of
the backups previous backup this is incremental and you can decide which property you want
to have so for example you can add a checksum on each sentries to do a comparison between
the content store in the backup and what you have on the file system.
Embedding resources jubacca can be embedded in executor programs or even more this presentation
you can download this presentation at this address and you will have a file and this
file is an arched archive so you can just use the arch tool to list the content extract
or month archive and you will have access to all the file of this presentation it is
revealed yes and it's HTML content but the same file is also a gym archive so you can
just use the gym tool to just set the content and open a browser to the local host and the
same files is also a program so if you make it executable you can run the program itself
to month extract or set the content what is interesting is that between our the content
is not shared it is an arch and gym archive but it's just a view to the same content there
is no duplication it's not two archive put together it's really one archive with two
kind of view of the same content and the last line is the exact command used to serve this
actual presentation conclusion this is a new way of thinking
we could extract you could use archive directly instead of extracting it so we can reinvent
the wall without thinking about using directly the archive it's a new way of thinking it's
generic it's a command based tool that can add that can add that to different usage
but it's pretty new maybe some crash and you can expect maybe some change in the specification
thank you are there any questions can you repeat the question okay I don't know
I know about squash fs but the thing is that jubaco is not a file system arcs is an archive
to store files but jubaco is not so jubaco is more generic than crime fs or squash fs
probably and arcs compared to squash fs is is is arcs slower than squash fs on size arcs
is better but on the performance is slower we could implement that in other languages
yeah could we re-implement this in other languages you could there is the specification
is language and mystic but just I just implement reference library in rest battle but the specification
is is public sorry that zip is pretty small but zip is is a slower the arcs in almost
any any kind of operation and is bigger than arcs also thank you