lessence — extract the essence of your logs
Your pod is crash-looping. kubectl logs dumps 70,000 lines. What's actually broken?
$ lessence kubelet.log
--- lessence briefing: kubelet_2k.log (2,000 lines)
span: E0909 13:07:09.181236 → E0909 13:21:46.847407 (14m 37s, 2.281 lines/s)
shape: █▇▇▆▆▆▆▅▇▆█▅▇▅▇▆▆▇▅▇▆▇▅▅ busiest 09-09 13:07 +37s holds 70 (5.8%)
format: plain 2,000 (100%)
levels: error 925 (78.9%), warn 68 (5.8%), info 179 (15.3%) — on 1,172 lines (59%)
top templates (10 of 56, 68% of all lines):
13.1% 261 Is the agent running?
13.1% 261 rpc error: code = Unknown desc = failed to setup network for sandbox "<HASH>": plugin type=<QUOTED_…
8.7% 174 > pod="<NAMESPACE>/<POD_NAME>"
...[73x] id=0 E0909 13:07:09.181236 → E0909 13:21:02.461198 <TIMESTAMP <PID nestedpendingoperations.go:348] Operation for "{volumeName:<K8SNAME podName:<UUID nodeName:}" failed. No retries permitted until <TIMESTAMP UTC m=+<DECIMAL (durationBeforeRetry <DURATION). Error: MountVolume.SetUp failed for volume <VARIES (UniqueName: "<K8SNAME") pod <VARIES (UID: "<UUID") : failed to fetch token: Post "<PATH": read tcp <IP:<PORT-<IP:<PORT: read: connection reset by peer variation: DURATION=64 …[report kept 0] IPV4=1 [127.0.0.1] K8SNAMESPACE=16 [kubernetes.io/projected/<UUID-kube-api-access-52r58|kubernetes.io/projected/<UUID-kube-api-access-b5ws4|kubernetes.io/projected/<UUID-kube-api-access-gm4xp] …[showing 3 of 7] …[report kept 7] K8SVOLUME=1 [oidc-token] NAME=29 [kube-api-access-b5ws4|kube-api-access-gm4xp|kube-api-access-l97vx] …[showing 3 of 7] …[report kept 7] PATH=15 [https://127.0.0.1:6443/api/v1/namespaces/gpu-operator/serviceaccounts/nvidia-con|https://127.0.0.1:6443/api/v1/namespaces/gpu-operator/serviceaccounts/nvidia-dcg|https://127.0.0.1:6443/api/v1/namespaces/gpu-operator/serviceaccounts/nvidia-ope] …[showing 3 of 7] …[report kept 7] PID=1 …[report kept 0] PORT=64 …[report kept 0] QUOTEDSTRING=11 ["<COMPONENT-<SUFFIX"|"<K8SNAME"|"<UUID"] …[showing 3 of 7] …[report kept 7] TIMESTAMP=64 …[report kept 0] UUID=15 [01af48d9-3471-4acf-93aa-689c01b31dff|1f0c6b7f-a1f8-4128-be41-448fb016a65a|1f4fdc9d-12d9-451b-9456-110b32706d57] …[showing 3 of 7] …[report kept 7] VARIES=7 ["kube-api-access-<SUFFIX"|"<COMPONENT-<SUFFIX"|"csi-rbdplugin-<SUFFIX"] …[showing 3 of 7] (report-sampled: DURATION, K8SNAMESPACE, NAME, PATH, PID, PORT, QUOTEDSTRING, TIMESTAMP, UUID — the report itself holds fewer values than the group had) previewed here: samples [39x] id=1 W0909 13:07:12.237366 → W0909 13:21:42.989676 <TIMESTAMP <PID transport.go:356] Unable to cancel request for otelhttp.Transport variation: PID=1 …[report kept 0] TIMESTAMP=39 …[report kept 0] (report-sampled: PID, TIMESTAMP — the report itself holds fewer values than the group had) ... report: ~/.local/state/lessence/reports/run-<date-<id/report.jsonl file: complete input: complete run: run-<date-<id size: <n bytes groups: 56 total, 40 selected, 17 printed, 39 omitted recipes (the report is JSONL; none of these prints the whole file): top 40 by count: jq -r 'select(.type=="group")|"\(.count)\t\(.id)\t\(.normalized[0:120])"' -- '~/.local/state/lessence/reports/run-<date-<id/report.jsonl' | sort -rn | head -40 ...
<!-- gen:example:end -->
That is the committed 2,000-line slice of a production kubelet log, run on
every CI build so the example is always real output.
## What a run gives you
`lessence app.log` does two things. It writes the **report**, the complete
folded JSON of the run (one record per distinct event, with its count, its
first and last raw lines and the values that varied), to
`~/.local/state/lessence/reports/<run>/report.jsonl`. And it prints an
**overview** on stdout, bounded to 16 KiB: the **briefing** (the summary blockgroups it holds, shows and left out), a selection of the rarest and most frequent groups (one distinct event with its count), and four jq recipes that query the report by group id. The report is complete; stdout is a selection, and the locator says how many groups it left out.
--overview all prints every group with no budget. --report-dir moves the reports, --report-max-bytes caps one run (default 1G). lessence never deletes a report; rm -r ~/.local/state/lessence/reports when you want the space.
lessence --no-report app.log skips the file and streams the folded text instead. Use --no-report for tail -f or any source that never ends: the report is written at end of input.
For Coding Agents & LLMs
70,000 log lines burn context and bury the signal. Pipe through lessence first: the agent sees each distinct event once, with its count, and can query the report for anything the overview left out.
kubectl logs pod/api | lessence | claude -p "what's wrong?"
kubectl logs pod/api | lessence --preflight | claude -p "analyze this log report"Which output for which need:
- the default run for orientation plus a queryable report file;
- --format json for the whole fold as JSONL on stdout, nothing saved;
- --preflight for a one-object health summary of the log;
- --explain is a developer mode that says why lines folded or split.
Structured output for agents: --format json
--format json emits a JSONL stream — one JSON object per folded group plus a terminating summary record. Each group record carries per-token-type rollup metadata: distinct counts, deterministic samples, a capped flag, a raw time range, and exact per-file locations for its first and last representatives. Stdin records use source: null because no original filename is known. The terminal summary includes a completeness contract with exact, lower-bound, or unknown counts for input skipped by safety limits, groups omitted by --top/--summary/--fit, and variation values hidden by sampling or rollup caps. Agents can answer "which pods?", "how many distinct UUIDs?", "when did this start?" from a single invocation without re-reading the log.
kubectl logs pod/api | lessence --format json \
| jq -r 'select(.type == "group" and .count >= 100)
| "\(.normalized): \(.variation.UUID.distinct_count) distinct UUIDs"'Full schema: docs/format-json-schema.md. Same input gives the same output, except the elapsed-time field; the rollup parameters are corpus-calibrated, see docs/rollup-calibration.md.