When needing to perform HTTP calls in shell scripts, Curl is the go-to tool: it's flexible and installed almost everywhere. But Curl is not primarily designed for use in shell scripts. This article explores common caveats when using Curl in shell scripts, and showcases best practices to adhere to.

Treating non-2xx/3xx responses as errors: -f

All shell scripts should have proper error handling. At minimum, you'll want your shell script to abort with an error if any command fails.

What constitutes an error? By default, Curl only considers connection problems to be errors. But it's quite common to consider non-2xx/3xx HTTP responses to be errors too. By default, Curl does not take the HTTP response code into consideration. The following example shows that Curl returns with exit code 0 (indicating success) even though the HTTP response is 500:

$ curl https://httpstat.us/500; echo $'\n' $?
500 Internal Server Error
 0

By passing the -f flag, Curl recognizes non-2xx/3xx responses as errors:

$ curl -f https://httpstat.us/500; echo $'\n' $?
curl: (22) The requested URL returned error: 500 Internal Server Error
 22

Treating some non-2xx/3xx responses as success

There's only one reason when you may not want to pass -f, and that's when you want to treat some non-2xx/non-3xx responses as success rather than failure. In this case, we'll want to extract the HTTP response code from Curl, and check in the shell script whether the response code should be considered success or not.

Here's an example in which we try to install some software on a server. The script tries to download a pre-compiled binary from an HTTP server, but if the binary does not exist (response code 404) then we'll fallback to compiling instead of aborting the script altogether.

CODE=$(curl -sSL -w '%{http_code}' -o binary.tar.gz https://myserver.com/binary.tar.gz)
if [[ "$CODE" =~ ^2 ]]; then
    # Server returned 2xx response
    do_something_with binary.tar.gz
elif [[ "$CODE" = 404 ]]; then
    # Server returned 404, so compiling from source
    compile_from_source
else
    echo "ERROR: server returned HTTP code $CODE"
    exit 1
fi

Let's examine the above code:

  • The most important flag is -w '%{http_code}'. This tells Curl to write the HTTP response code to standard output.
  • We wrap Curl in $(...) in order to capture its standard output into a variable named CODE.
  • -o <filename> tells Curl to write the HTTP response body to the given file, not to standard output. This is important because we want the standard output to only contain the HTTP code.
  • [[ "$CODE" =~ ^2 ]] is a Bash regular expression check. We check whether the response code is 2xx.
  • -sSL is also important. This will be explained below.

Disabling the progress meter: -sS

When Curl is not writing to a terminal, then Curl displays a progress meter by default:

$ curl http://slashdot.org/ | cat
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   194  100   194    0     0    336      0 --:--:-- --:--:-- --:--:--   336
<html>
<head><title>301 Moved Permanently</title></head>

In shell scripts we usually don't want this. Disable the progress meter with -sS.

The -s disables the progress meter, as well as error messages. But we do want Curl to report errors, which we reenable with -S.

Handling redirects: -L

HTTP servers are allowed to respond with redirects at any time. Especially when you're using Curl to download a file, it is increasingly common for web servers to redirect to Amazon S3. Unfortunately, Curl doesn't follow redirects by default, and instead returns the raw redirect response.

Pass -L to make Curl follow redirects.

Conclusion

When using Curl in shell scripts, always pass -fsSL, which:

  1. Treats non-2xx/3xx responses as errors (-f).
  2. Disables the progress meter (-sS).
  3. Handles HTTP redirects (-L).

Occassionally, if you consider some non-2xx/3xx responses to be non-fatal, then omit -f and use the trick described in "Treating some non-2xx/3xx responses as success".

Happy Curling.