The curl command-line instrument is without doubt one of the most helpful and versatile packages you may study. Its versatility and complete HTTP implementation imply that, if there’s a URL for it, curl can do it.
8
Fetch a Net Web page
In its most simple kind, curl fetches a URL and shows its contents in your terminal:
curl information.cern.ch
What you see will rely on the content material kind of the URL. In case you request an online web page, you’ll see its HTML supply:
You possibly can redirect this output to a file to avoid wasting an online web page:
curl information.cern.ch > information.cern.ch.html
Discover how curl detects that you just’re redirecting the traditional output and sends timing data to stderr:
A URL might comprise characters with particular that means to the shell, like “?” or “#.” You’ll have to be cautious to keep away from issues this will trigger, so be sure you encompass URLs with single quotes in the event that they comprise particular characters. In case you’re undecided, be cautious and all the time quote them:
curl 'information.cern.ch'
Associated
7
Obtain a File
In case you attempt to use curl to obtain a binary file—like a picture or a Microsoft Phrase doc—you’ll see an error telling you that “Binary output can mess up your terminal.” It’s because curl will print the response physique that it receives to output. Within the case of a picture, that is virtually definitely not what you need to be doing.
The error message mentions the –output choice, which helps you to specify a file to avoid wasting the output to somewhat than print it. You can too use -o as a shorter equal:
curl -o neo-the-cat.jpg https://placecats.com/neo/300/200
This can save the file at that URL within the present listing. As earlier than, since curl is now not outputting the response, it offers lots of element concerning the time your obtain takes. If you’d like a easy progress bar as a substitute, use the -# or –progress-bar choice:
curl --progress-bar -o neo-the-cat.jpg https://placecats.com/neo/300/200
6
Set up Software program
You might typically come throughout set up directions that ask you to make use of curl. For instance, atuin—a alternative for the history command—has the next set up instruction:
curl --proto '=https' --tlsv1.2 -LsSf https://setup.atuin.sh | sh
In case you open that URL in an online browser, you’ll see a shell script that performs set up and setup. You possibly can obtain it and run it manually if you need, the shell one-liner is only a faster means. Word that the output—the contents of this script—is piped to sh which can run it as a shell script.
The urged curl utilization within the atuin instance makes use of these choices:
- –proto ‘=https’ to make sure curl makes use of the HTTPS protocol.
- –tlsv1.2 to drive curl to make use of TLS version 1.2 or better.
- -L to mechanically comply with redirects.
- -sS to suppress the progress meter however nonetheless present error messages.
- -f to deal with an HTTP error as a command error. That is to make sure that no error message will get piped to sh, avoiding a doubtlessly harmful motion.
Operating a shell script downloaded from the web is dangerous since you don’t know precisely what it’s going to do. Not solely do that you must belief the supply, however you additionally have to be assured you’re safe from any Man-in-the-Middle attacks. HTTPS will help to mitigate this threat, however if you wish to be additional cautious, take into account dropping the pipe, downloading the file, and checking that it’s protected to run earlier than you accomplish that.
5
Verify Your IP Handle
This sensible use for curl is a fast and straightforward means of discovering your public IP address, due to the service at ifconfig.co:
curl ifconfig.co
The positioning responds together with your public IP handle:
This default output is apparent textual content which makes it a superb useful resource for automation. You may discover that you just get a really completely different consequence if you happen to go to the identical URL in your internet browser:
It’s because the location behaves in a different way based mostly in your consumer agent. By default, curl sends one thing like “curl/8.7.1” within the Person-Agent header. In case you ever must set this to one thing else, e.g. for testing functions, use the -A/–user-agent choice:
curl -s --user-agent "my faux browser" ifconfig.co | head
Now the ifconfig web site will reply with HTML, simply because it does in an online browser:
Typically, you need to fetch simply the response headers from a URL. You is likely to be debugging an error or checking cache headers to see if that you must redownload a file. To view simply the response headers, use -I/–head:
curl -I instance.com
This can ship a HEAD request, which not all internet servers help. In case you get an error utilizing this technique, you may inform curl to ship a GET request and solely output the headers:
curl -I -X GET instance.com
If you wish to see the request headers that curl sends, use the –verbose/-v choice:
3
Entry an API
In case you’re working with an API, an excellent consumer like Postman or Posting will enable you to automate and debug the duty. However curl could be simply as helpful for exploring how an API works.
The DummyJSON service is an efficient place to begin. This web site offers an instance API with dummy information that you should use to check consumer apps and your personal packages. You should use completely different endpoints to verify redirects, take a look at authorization, and simulate time delays or random responses.
Making an attempt out the service with curl is so simple as passing the suitable endpoint:
curl https://dummyjson.com/take a look at
You need to see a easy JSON doc in response, with two properties. The strategy property will mirror the HTTP technique of your request:
When you’ve fetched information from an API, you may simply course of it utilizing customary command-line instruments. For instance, you may process JSON data using jq. This pipeline fetches product information utilizing curl, then passes it to jq to extract the primary product:
curl https://dummyjson.com/merchandise | jq '.merchandise[0]' | extra
Associated
How to Parse JSON Files on the Linux Command Line with jq
You should use jq to interrogate JSON information on the Linux command line. We’ll get you began!
One among my present initiatives is a shell-based RSS consumer. At its core lies curl as a result of the instrument is so handy and helps each low-level HTTP operation I want. For instance, you may fetch an RSS feed and look at its contents utilizing curl:
curl -s https://www.w3.org/weblog/feed/ | extra
When you’ve got libxml put in, you may course of the XML output utilizing xmllint:
curl -s https://www.w3.org/weblog/feed/ | xmllint --xpath '//channel/title/textual content()' -
This pipeline makes use of xmllint to judge an XPath expression that extracts information from XML, on this case, the title of the channel:
1
Use curl Programmatically
The cURL mission is about extra than simply curl, the command-line instrument. It additionally features a library, libcurl, that curl makes use of itself. This library is extremely moveable and out there for all method of working programs together with Linux, Home windows, macOS, and Android. libcurl comes with a C API, however there are bindings for a lot of different languages like C++, Python, Rust, and Go.
The library mirrors a lot of what the command-line instrument does, significantly its choices and their conduct. Right here’s a primary instance of PHP utilization:
$ch = curl_init("http://www.instance.com/");
$fp = fopen("www-example-com.html", "w");curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
if (curl_error($ch))
fwrite($fp, curl_error($ch));
curl_close($ch);
fclose($fp);
?>
The curl_init() perform begins a session and returns a deal with that you should use to entry it. You possibly can management the library’s conduct utilizing the choices that curl helps through curl_setopt(). curl_exec() truly sends the request whereas curl_error() and curl_close() are helpful admin features.
This instance makes use of the CURLOPT_FILE setting to put in writing the response physique to an open file deal with. Word that that is the equal of curl’s –output choice.
In case you’re searching for different terminal-based instruments to grasp, study some basic git commands or take a look at some bash script examples.