So, yes, it is everywhere.
And what is sometimes I think people tend to forget or don't realize is that we keep
releasing curly releases like crazy.
So these are single blobs for every release we have done since 1996 until now.
That's 254 releases.
We did one this Wednesday.
So anyway, that's just, you know, yeah, we keep releasing stuff and we keep adding
source code and changing source code.
This is the, so the blue line, the top line is all code in curl and live curl taken together.
The red one is just the command line tool.
But since the command line tool uses the library, so it's, there's a lot of development, a
lot of changes, and we're adding a lot of command line options.
That, I'm not sure that's so good, but we do that anyway.
And so, yeah, back in, when I renamed the project curl in 1998, we had 24 options, I
believe, and now we have 258.
So that's roughly 10 a year on average.
So yeah, and chances are we're going to have even more soon, right?
Well, and even if we just look at the last few years, you can see that all of these graphs
show a significant growth there as well, meaning that we're still, we keep doing things and
we keep changing things and we actually make curly better product every year.
So I just wanted to mention that it's a good idea sometimes to upgrade this, you know.
I don't know exactly how it is in all projects, but in curl we have this, I don't know, there's
a trend, people keep using very, very old curved versions.
Sometimes it's a little bit boring, you know, and we actually fix stuff.
Yeah, so curl is this Swiss Army knife of internet transfers and one of the cooler things
we did within the last few years is that we had a parallel transfer site.
I hope you all know that.
So nowadays then, traditionally, of course, if you just write curl in a lot of URLs, it
will get them seriously one by one by one by one, but now you can use this option and
it'll get them all in parallel.
Well, actually it will get up to 50 by default in parallel, you can change that.
I guess there's some, usually by default there's a 10, 24 open file descriptors, right?
So there's some kind of maximum that will cause you problems, but otherwise it's awesome
for when you want to get a lot of files down faster, possibly from just lingering on your
disk like curl has always done, or it does by default, right?
So you can ask curl to remove it, remove.
For example, you have a timeout, you really want to get this download done in two seconds,
and if it's not done in two seconds, it's going to fail, right?
But what happens with that leftover file?
If you have this flag, it won't be there.
If you don't use the flag, you will have a partial one, possibly, right?
So, and we also have added a lot of fun other ways to control multiple transfers, really,
or transfers in general.
We've had this option for a long time, which is, this has, this is a sort of a reverse
speed limit.
If it's slower than this, kill it.
If it's slower than this many bytes per second for this amount of time, we don't want it
anymore, basically, detecting catching, stall transfers, too slow for me, stop.
But we also have this other one.
So sure, only, only do this transfer at the maximum speed of 100K, that's bytes per second,
or you can also do, if you want to do those, you know, download a million files, maybe you
don't want, you don't want those million files to happen immediately, maybe you want to slow
down the rate a little bit, and then you can do it like this.
I just want to get those files at maximum two per second, or that means, that means
started requests, or maybe three per hour, or 14 per month, or week, and blah, blah,
blah.
Basically, a way to, if you really want to get a lot of files, maybe from the same server,
and you, maybe it's your server, you don't want to overload it immediately, or maybe
you know that the files are just updated every once in a while, anyway, so why not slow
it down.
And of course, you can also ask, so ignore the file if it's too big, and this will also
nowadays actually stop the transfer if it, if it downloads that much, which in some cases
you don't know how big the file is, right, and you might not want to fill up your disk
space when the server surprises you Monday morning.
And of course, one of the bigger things I've done recently is, yeah, KERL has had this
concept that I call config file, it's really not a config file, it's passing the user names
or whatever in those config files, and you couldn't do that, but now you can, because
we have introduced a way, so we have introduced this new option to KERL called variable, so
it's basically a way to set the variables in the command line and in these kind of config
files for KERL, and why do you want that?
I'll show you just some examples, so basically you can set, it's just a name, it has some
content and you can get that content from the command line like that, or read it from,
you can also, you can set it in the config file with different syntax, but that's the
same as the old, all right.
So you can basically set it as pure content like this, or you can read it from the file,
you can get it from the environment variable, which means that you can do stuff like this,
or you can set it from the environment variable like this, you import it like with a percent
sign.
I'll show you some examples and why it's cool.
Blah, blah, blah, blah.
So basically, so you can set, why would you, you set a command line and you can then expand
them of course, so if you want to for example set a command, a name, and you want to use
it in an option, you can ask it explicitly to expand it, and it sounds a little bit weird,
but it works like this basically, so you can expand, if you want to, the data option as
someone might know, it's short version is dash D is for sending a post for example, so
in this case you can send the content, the variable content in here would be sent as
a post, and the content of the variables host user here would be in the URL that it uses
to the command line.
Basically just gives you more freedom to create weird config files and use that to do more
curl, and you can also, what's even more fun than this, you can then for example read those
contents of files for example, so if you wanted to post from a file or you wanted to read
credentials from a file, you can do that as well, and then if you want to make it even
more cool, you can apply different functions when you expand these contents, so you can
JSON encode it, you can URL encode it, and you can be 64 encoded if you want to in the
config file, so really to help you do those weird things that people want to do.
For example, you get the, and are complicated more so than we want to sometimes.
So basically with this tool, it's a really simple one, it basically just gets sets parts
of a URL for you, like this, if you want to, you can give it a URL and you set a host name
and it'll just replace the host name in the URL and output that, or you can just ask for
the host name part from a URL, or you can redirect a URL, so if that is the first URL
and you want to redirect another, how would the end result become, right, typically then
of course a relative redirect, dot dot slash dot dot slash blah blah blah, what would happen,
or change parts of it, like update the port number or you could append query strings,
or you can do more complicated things, you can output everything as JSON, so we would split it
up in different components and output everything as JSON and you could j que it or whatever you
want to do, and you can also do things like extracting parts of the query string, so maybe I
want to just a component from that weird URL, a little thing just to help you work with URLs
better, we also worked quite a lot on adding JSON stuff in curl recently, so for example you can,
we have this option called write out, you should know about it, it's called, it's actually
dash w in the short form, it helps you just output stuff from the previous transfer, it has a lot
of variables, you can download speed or headers or things, and we also, it works like that,
we recently added support for getting all that data output as JSON, so now you can do like this
dash w and the variable called JSON, and that variable then will spew out the huge
JSON object with all of those variables that I call them as JSON object, pretty much it's a fancy
way, if you want to get everything in JSON, you work with JSON, you want to do j que it, curl
helps you go there, and in the, oh right, it also, we have this other variable called
header JSON, it's called header JSON, right, it then outputs all the response headers as a JSON
object, also to help you work with the headers maybe scripted, j que it, whatever you want to do,
so basically you do more JSON, so you can, and of course we ship hb3 enabled by default, if you
happen to be, have that config enabled, no, this too has that, but you know, in theory you could,
no, but it's a tls setup problem for all of us, but we enable hb3 by default, and if it's possible
and it's a really cool way, and now you can do it with curl, and you can just, you know,
ask for curl to use hb3, it's called dash dash hb3, and hb3 as you know uses quick, quick is a
different transport protocol, it's done over udp, so it's not the same connection, right, so we can't
upgrade from a hb1 or hb2 to an hb3 connection, so they actually have to be different connections,
and since hb3 is still being blocked, problematic in expersentable attempts,
so when you ask curl to do hb3, it will actually try hb3 and hb1 and 2 at the same time,
and go with the one that works best, that works, it's actually, it's raising them together, so it
starts with hb3, a little bit before it tries the other ones, it works really well, so it's
in the happy eyeballs style, so we do happy eyeballs on pv4, pv6, and then happy eyeballs
between hb3 and hb2, and hb2, one will of course downgrade to hb1 if it can't speak hb2, kind of fun,
you should try it out, and of course if you're using then the parallel option thing with
uppercase z, it will do them multiplex if possible if it's on the same host,
really fun, and of course I have written about most of this in everything curl, you should
read it, it's a lot of pages, thank you.
Thank you.
Questions? There's a question, I have stickers.
Yeah.
Thanks for the presentation, are those all nice features accessible also for
library users in C or in Python bindings, or is it available only from command line?
Most of this is powered by the library, so almost everything that is that I mentioned here,
that is network related, is available to the library, really problem, and that's also why I
wanted it to be used the same parser, so true rel and curl uses the exact same parser, so they
will work exactly the same way on those URLs. One more question?
Nothing more, thank you. Thank you.
Thank you.