service-status.sh already tells process, port and application apart, but it stops at the door: it knows /salud returns a 200, not what it says. The veloz-api answers JSON —database state, queued shipments, deployed version, metrics—, and that is where the toolkit stops reading files somebody left on disk and starts integrating with the rest of the system. This lesson covers curl in depth for talking to APIs and jq for handling JSON properly: extracting it, filtering it, turning it into columns and also building it safely, because in the end daily-report.sh will publish its daily summary as JSON.

1. curl in scripts: the -sSf trio
2. Methods, headers and bodies
3. Files, redirects and timeouts
4. Credentials without leaving a trace
5. HTTP codes: what a script must tell apart
6. Body and status code in a single call
7. Why you do NOT parse JSON with grep or sed
8. jq: the filter model
9. Selecting, filtering and aggregating
10. Controlling the output: -r, -c, -e and @tsv
11. Building JSON safely
12. Reading JSON in Bash, and JSON as configuration
13. Application: the toolkit talks to the API and publishes JSON

By default curl is designed for a person at a terminal: it writes a progress bar and, if the server answers an error, it prints the error page and returns 0. In a script that is exactly the opposite of what you want. Three options fix it:

Memorize -sSf as a block: silent, but complaining, and genuinely failing. It is the most widespread mistake in scripts that talk to APIs, and it produces the worst possible kind of failure: a silent one that makes you believe everything is fine.

For REST APIs you need four more options:

-X sets the method (GET, POST, PUT, DELETE); it is not needed with -d, which already implies POST. -H adds a header and is repeated as many times as necessary. -d sends the body —if it starts with @, it reads it from a file: -d @body.json—. For parameters with special characters there is --data-urlencode, which encodes the value: curl -sSfG --data-urlencode "ciudad=Palma de Mallorca" http://localhost:8080/envios. That -G turns the data into URL parameters instead of a body, so the result is /envios?ciudad=Palma%20de%20Mallorca. Writing that URL by hand with a Bash variable is a bug waiting to happen: one space, one & or one accent in the value and the request changes meaning.

The two timeouts are different and you have to set both. --connect-timeout 5 cuts off quickly when the machine is not there; --max-time 30 protects you from the server that accepts the connection and then thinks about it indefinitely. A curl without --max-time in an automated script is a time bomb: the day the service gets stuck, your cron job will hang forever and block the following ones (07-01).

-u user:password does basic authentication, but written like that the password shows up in ps, in the history (02-06) and in any log of the command. The two correct ways:

--netrc-file reads machine, user and password from a file with 600 permissions, just like veloz-ops.conf (05-06). The second option uses an environment variable loaded from that configuration file: it does not stay in the history, although it is visible in /proc/<pid>/environ to the user themselves and to root. What must never be done is to embed the credential in the URL, as already seen in 06-04.

That distinction between 4xx and 5xx is what makes an integration script useful: retrying a 404 is a waste of time, and not retrying a 503 means writing off a service that was restarting. And remember the starting point: without -f, curl returns 0 with any code, so without it you do not even find out.

-f tells you there was an error, but it discards the body, which usually contains the explanation. To have both without making two requests, you ask for the code with -w at the end and separate it afterward:

Three things from previous modules come together here: -w '\n%{http_code}' appends the code on a new line at the end, the expansions ${var##*} and ${var%*} from 04-04 separate it from the body without spawning processes, and the case with globs from 04-05 classifies by family with 2??. Notice that -f is not used here: we want the error body, so the case makes the decision.

The temptation is enormous and the result always ends badly. To pull the status out of /salud, what everybody writes the first time is curl -s $API/salud | grep -o '"status":"[^"]*"' | cut -d'"' -f4. It works until it stops working, and the reasons are not far-fetched: the JSON may come compact or with line breaks (your regex depends on the formatting, which the server can change without warning); the key order is not guaranteed; a nested object with another status key may show up and your grep will take the first one it finds; a value may contain escaped quotes ("message":"error \"severe\"") that destroy [^"]*; and null, numbers and booleans carry no quotes, so the pattern does not even see them. In short: JSON is not a line-based format, and line-based tools cannot understand it reliably. What you need is a parser, and that is jq.

jq is a language of filters: it takes a JSON document, applies an expression to it and emits JSON. The central idea is that every filter transforms an input into zero, one or several outputs, and they are chained with | like shell pipes.

Suppose /salud returns {"status":"ok","version":"2.4.1","db":{"connected":true,"latency_ms":12},"queue":[3,7,2]}:

The last two rows are the ones that prevent half the scares: // gives a default value when the API omits a field, and ? keeps a type error from aborting the whole filter. The difference between .queue (an array) and .queue[] (three outputs) is the concept to internalize: most interesting filters work on streams of values, not on a single one.

Over a response from /envios?ciudad=Madrid, which returns an array of objects with id, status and amount:

The brackets in [.[] | .amount] matter: .[] | .amount emits three loose values, and add needs an array, so they have to be collected. Confusing "stream of values" with "array" is mistake number one with jq, and it is always fixed with [ ] or with map(). You will notice that group_by + map is the same aggregation you were doing with associative arrays in awk (06-01), only over JSON.

By default jq emits JSON, so a string comes out quoted: "ok", not ok. When you assign it to a Bash variable, those quotes travel with it and ruin any later comparison.

-e turns a JSON query into a shell condition, with no intermediate variables. And to get back into awk territory, @tsv and @csv turn arrays into columns:

That chaining is the bridge between the two worlds: jq understands the structure and flattens it into columns; awk aggregates. @csv quotes the fields according to CSV rules, and both require -r so that everything does not come out as an escaped JSON string.

Generating JSON by hand —printf '{"city":"%s","note":"%s"}\n' "$city" "$note"— works until the first value with a quote, a backslash or a line break, and then it produces invalid JSON, or worse, valid JSON with altered content. The correct way is jq -n, which builds from scratch, with --arg for strings and --argjson for values that are already JSON (numbers, booleans, objects):

jq takes care of the escaping: a quote inside $note comes out as \", a line break as \n and accented characters are encoded correctly. The distinction between --arg and --argjson is the usual trap: --arg total 1001 produces "1001" (a string) and --argjson total 1001 produces 1001 (a number). If the value can come in empty, --argjson will fail —jq does not accept invalid JSON—, so a ${total:-0} is a good idea. And as with awk -v in 06-01, the principle is the same: Bash values go in as parameters, never interpolated inside the filter, because interpolating them is code injection.

To walk a response in Bash, the combination is jq -r '.[] | @tsv' with the loop from 04-01:

@tsv is preferable to splitting on spaces because the values may contain them, and it also escapes any tabs and line breaks that were inside a field. The process substitution < <( ) from 05-05 keeps the loop in the current shell. When what you need is a Bash array with a single column, mapfile -t ids < <(jq -r '.[].id' <<< "$json") (04-03) is more direct.

JSON also works as a configuration format, and jq reads it with the same rules: jq -r '.thresholds.disk // 85' etc/veloz.json returns the value or the default if it is missing. Compared to the key=value of veloz-ops.conf (05-06), it wins on nested structure and types, and loses in that it cannot be loaded with source and takes no comments. For YAML —the format of CI and Kubernetes files— there is yq, which replicates jq's syntax over YAML and even converts between the two with yq -o=json. If you already know jq, you know yq.

First, service-status.sh queries the API for real and tells an outage apart from an application error, which is the difference between restarting the service and alerting the development team:

The three outcomes are deliberately different: no response (the curl fails), a response with an HTTP error (.message is extracted from the body for the alert) and a correct response but with a degraded status. jq -e uses the JSON as a condition and jq's if ... then ... else ... end formats the boolean for the message. Note the <<< "$body", the here-string from 05-05: it avoids rereading the network response every time.

And daily-report.sh closes the circle by publishing its summary as JSON, built with jq -n:

The mktemp with trap from 05-01 and the final mv guarantee that nobody ever reads a half-written file: either the complete summary exists or the previous day's does. now | todate generates the timestamp in UTC ISO-8601 without calling date. And now the report is no longer just text to read: it is a piece of data that another program —a dashboard, an alert, project 09-05— can consume without parsing anything again.

• curl without -f (or without checking the code). It returns 0 with a 500 and the script carries on as if nothing happened. Use -sSf, or -w '%{http_code}' and a case.
• curl without --max-time. A stuck service hangs the cron job indefinitely.
• Parsing JSON with grep/sed. It fails with compact formatting, key order, nesting and escaped quotes. Use jq.
• Forgetting -r. status=$(jq '.status') stores "ok" with quotes and [[ $status == ok ]] fails.
• Confusing a stream with an array. .[] | .amount emits loose values; add needs [.[] | .amount].
• --arg for numbers. It produces "1001" instead of 1001; for numbers and booleans, --argjson.
• Interpolating Bash variables inside the jq filter. Same injection risk as in awk: use --arg.
• Retrying a 4xx. It is not going to fix itself. Retry the 5xx and the network failures, with backoff (06-04).
• Tip: build your filters step by step (jq '.', then jq '.field'…) against a sample response saved in a file: it is faster than calling the API, it works offline and jq is much easier to debug in increments than in one go.

Exercise 1. Write a function veloz_api_get for lib/common.sh that does a GET to an API path, returns the body on standard output, tells network / 4xx / 5xx apart with different messages and different exit codes, and cannot hang.

Exercise 2. With /metricas returning {"requests":48210,"errors":37,"latency_p95_ms":210,"queued_shipments":14}, write a check that warns if the error rate exceeds 0.1% or if the p95 latency goes over 500 ms, treating missing fields as 0.

Exercise 3. Turn the output of /envios?ciudad=Madrid (an array of objects with id, courier, status, amount) into a summary per courier with the number of shipments and the total amount, sorted from highest to lowest, using only jq.

Solution 1.

The three codes are deliberate and follow the table in 05-03: 69 (EX_UNAVAILABLE) when there is no response, 64 (EX_USAGE) for a 4xx —the mistake is ours— and 75 (EX_TEMPFAIL) for a 5xx, which is precisely the code that tells the caller "this is temporary, you can retry". The body comes out clean on stdout and the diagnostics on stderr (02-04), so that data=$(veloz_api_get /salud) captures only what matters.

Solution 2.

The // 0 on each field keeps an absent metric from turning into null and breaking the arithmetic. The three values are extracted in a single invocation of jq with @tsv and a read, instead of calling it three times. The error rate is decimal, so the comparison goes in awk with the exit !(...) idiom from 06-03; the latency is an integer and (( )) is enough for it.

Solution 3.

group_by produces an array of arrays —one per courier—, and map turns it into objects: .[0].courier takes the name from any element of the group, length counts and map(.amount) | add sums. sort_by(-.total) sorts descending by negating the value, cleaner than sorting and then reverse. At the end, @tsv flattens it into columns and column -t (02-02) aligns them. It is exactly the aggregation from solution 1 of 06-01, but starting from JSON instead of CSV.

Talking to an API from a script is two tools and a handful of rules. From curl: -sSf as an indivisible block —silent, complaining about network failures and genuinely failing on an HTTP error, because without -f it returns 0 with a 500—; -X, -H, -d and --data-urlencode to build the request; -L, -o/-O for the body; --connect-timeout and --max-time always, or the day the service gets stuck you will hang the cron job; credentials via --netrc or a header from a 600 file, never in the URL; and -w '\n%{http_code}' to keep body and code in a single call and classify with a case that tells 4xx (do not retry) from 5xx (retry with backoff). From jq: a language of filters chained with | where .field, .a.b, .[], select, map, group_by, add, sort_by, length, to_entries, // and ? cover almost everything; -r to get strings without quotes, -e to use JSON as a condition, @tsv/@csv to get back into awk territory; and jq -n --arg/--argjson to build JSON with the escaping already solved, passing Bash values as parameters and never interpolated. And the rule that covers it all: JSON is not a line-based format, so grep and sed are no use for reading it.

With this Module 6 closes and the Veloz Envíos toolkit changes in nature. daily-report.sh aggregates in a single pass with awk and publishes its summary in JSON; service-status.sh gathers the system context, checks network and port, interrogates the veloz-api and tells an outage apart from an application error. It is no longer a file reader: it is a piece integrated with the rest of the system. But notice what every run in this module has in common: you are the one who typed the command. The daily report only exists if somebody remembers to launch it, and a status check that runs when you already suspect something is wrong arrives late by definition. In Module 7 that ends: cron so tasks run by themselves (07-01), the design of truly unattended jobs (07-02), automatic backups (07-03), continuous monitoring and logging (07-04), systemd services and timers (07-05) and remote automation with ssh (07-06). The toolkit stops waiting for your orders.