Polishing Your Curl Expertise
Previous post covers bare minimum of curl you need to know for testing RESTful microservices. Read it first if you need basics. This writing focuses on corner cases and advanced options, making curl experience more enjoyable.
Microservice for experiments
For demonstrations, I’ve created a simple RESTful microservice in Golang. Use it if you have nothing to experiment with.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
)
type RequestSummary struct {
URL string
Method string
Headers http.Header
Params url.Values
Auth string
Body string
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
bytes, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
auth := ""
if user, pass, ok := r.BasicAuth(); ok {
auth = user + ":" + pass
}
rs := RequestSummary{
URL: r.URL.RequestURI(),
Method: r.Method,
Headers: r.Header,
Params: r.URL.Query(),
Auth: auth,
Body: string(bytes),
}
resp, err := json.MarshalIndent(&rs, "", "\t")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Write(resp)
w.Write([]byte("\n"))
})
http.ListenAndServe(":8080", nil)
fmt.Println("Exiting...")
}
Suppressing progress meter
curl displays progress meter while fetching data and hides it when data transfer is over. Seeing progress is nice, but it causes an issue when you pipe curl with less for reading a long response. When you do it, progress meter mixes with response, making output clumsy:
$ curl -X GET http://localhost:8080/something | less
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 171 100 171 0 0 4228 0 --:--:-- --:--:-- --:--:-- 4275
{
"URL": "/something",
"Method": "GET",
"Headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.43.0"
]
},
"Params": {},
"Auth": null,
"Body": ""
}
To suppress progress meter, curl can be switched to silent mode with -s (or --silent) option:
$ curl -X GET -s http://localhost:8080/something | less
{
"URL": "/something",
"Method": "GET",
"Headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.43.0"
]
},
"Params": {},
"Auth": null,
"Body": ""
}
This solves issue with progress meter, but introduce another one - silent mode also suppresses error output:
$ curl -X GET -s http://unknownhost:8080/something | less
(There should have been an error that host cannot be resolved)
A full solution is to use -s together with -S (or --show-error), which displays errors in silent mode:
$ curl -X GET -s -S http://unknownhost:8080/something | less
curl: (6) Could not resolve host: unknownhost
Authentication
To authenticate a user with a password, use -u <user>:<password> (or --user) user:password option.
Don’t pass password in command line though, unless you really know what you do. This is insecure, because executed command line (including your password) will be stored in history and log.
The right way is to pass user only. When password is not provided, curl will ask for it and let you type it securely:
$ curl -X GET -u login http://localhost:8080/something
Enter host password for user 'login':
{
"URL": "/something",
"Method": "GET",
"Headers": {
"Accept": [
"*/*"
],
"Authorization": [
"Basic bG9naW46cGFzc3dvcmQ="
],
"User-Agent": [
"curl/7.43.0"
]
},
"Params": {},
"Auth": "login:password",
"Body": ""
}
Outputting metadata
Sometimes you need to output information about response, instead of response itself. Option -w <format> (or --write-out <format>) lets you fully control the output and is often used to display response metadata, e.g. HTTP status code:
$ curl -X GET -w "HTTP code is %{http_code}\n" -o /dev/null -Ss http://localhost:8080/something
HTTP code is 200
Note variable http_code is specified as %{http_code}. See curl documentation for list of available variables.
When you control output with -w option, response is often not needed and suppressed, e.g. with -o /dev/null.
Debugging
When something goes wrong, the first step is usually to look at what is sent to endpoint and what returns back. You can do it with -v (or --verbose) option:
curl -X GET -v http://localhost:8080/something
* Trying ::1...
* Connected to localhost (::1) port 8080 (#0)
> GET /something HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 23 Aug 2016 22:49:21 GMT
< Content-Length: 169
< Content-Type: text/plain; charset=utf-8
<
{
"URL": "/something",
"Method": "GET",
"Headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.43.0"
]
},
"Params": {},
"Auth": "",
"Body": ""
}
* Connection #0 to host localhost left intact
Tracing
When debugging doesn’t help and you need to dig deeper, try --trace <file> instead:
$ curl -X GET --trace t.txt http://localhost:8080/something
{
"URL": "/something",
"Method": "GET",
"Headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.43.0"
]
},
"Params": {},
"Auth": "",
"Body": ""
}
$ cat t.txt
== Info: Trying ::1...
== Info: Connected to localhost (::1) port 8080 (#0)
=> Send header, 87 bytes (0x57)
0000: 47 45 54 20 2f 73 6f 6d 65 74 68 69 6e 67 20 48 GET /something H
0010: 54 54 50 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 6c TTP/1.1..Host: l
0020: 6f 63 61 6c 68 6f 73 74 3a 38 30 38 30 0d 0a 55 ocalhost:8080..U
0030: 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f ser-Agent: curl/
0040: 37 2e 34 33 2e 30 0d 0a 41 63 63 65 70 74 3a 20 7.43.0..Accept:
0050: 2a 2f 2a 0d 0a 0d 0a */*....
<= Recv header, 17 bytes (0x11)
0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a .
<= Recv header, 37 bytes (0x25)
0000: 44 61 74 65 3a 20 54 75 65 2c 20 32 33 20 41 75 Date: Tue, 23 Au
0010: 67 20 32 30 31 36 20 32 32 3a 34 33 3a 30 35 20 g 2016 22:43:05
0020: 47 4d 54 0d 0a GMT..
<= Recv header, 21 bytes (0x15)
0000: 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 Content-Length:
0010: 31 36 39 0d 0a 169..
<= Recv header, 41 bytes (0x29)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te
0010: 78 74 2f 70 6c 61 69 6e 3b 20 63 68 61 72 73 65 xt/plain; charse
0020: 74 3d 75 74 66 2d 38 0d 0a t=utf-8..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a ..
<= Recv data, 169 bytes (0xa9)
0000: 7b 0a 09 22 55 52 4c 22 3a 20 22 2f 73 6f 6d 65 {.."URL": "/some
0010: 74 68 69 6e 67 22 2c 0a 09 22 4d 65 74 68 6f 64 thing",.."Method
0020: 22 3a 20 22 47 45 54 22 2c 0a 09 22 48 65 61 64 ": "GET",.."Head
0030: 65 72 73 22 3a 20 7b 0a 09 09 22 41 63 63 65 70 ers": {..."Accep
0040: 74 22 3a 20 5b 0a 09 09 09 22 2a 2f 2a 22 0a 09 t": [...."*/*"..
0050: 09 5d 2c 0a 09 09 22 55 73 65 72 2d 41 67 65 6e .],..."User-Agen
0060: 74 22 3a 20 5b 0a 09 09 09 22 63 75 72 6c 2f 37 t": [...."curl/7
0070: 2e 34 33 2e 30 22 0a 09 09 5d 0a 09 7d 2c 0a 09 .43.0"...]..},..
0080: 22 50 61 72 61 6d 73 22 3a 20 7b 7d 2c 0a 09 22 "Params": {},.."
0090: 41 75 74 68 22 3a 20 22 22 2c 0a 09 22 42 6f 64 Auth": "",.."Bod
00a0: 79 22 3a 20 22 22 0a 7d 0a y": "".}.
== Info: Connection #0 to host localhost left intact