Skip to content

An unparseable audit ledger silently halts all audit recording and forces every nono run to exit 1 #1583

Description

@iler

Summary

A single malformed line in ~/.local/state/nono/audit/ledger.ndjson — two records
concatenated with no newline between them — puts nono into a state where it
prints one line of diagnostic and otherwise appears to work, while in fact:

  1. all audit recording stops permanently, and
  2. every nono run exits 1 regardless of the child's exit status.

Both persist across runs and reboots. Neither is stated by the message, and
neither is obvious from the outside. In my case it went unnoticed for a day and
29 sessions.

The message is:

nono: Snapshot error: Failed to parse audit ledger line 60: trailing characters at line 1 column 342

I'd argue (2) is the more serious defect independent of how the ledger got
corrupted: an audit-subsystem parse failure should never be able to change the
observable exit status of the sandboxed command.

Environment

  • nono 0.71.0 (Homebrew, bottled)
  • macOS 26.5.2 (build 25F84), arm64
  • Not enrolled with a control plane (nono audit statusEnrolled: no)

Impact

1. Audit recording stops silently

nono cannot extend a hash chain it cannot parse, so it skips the append — but
it does not say so, and it does not fail. The ledger simply stops growing.
nono audit list keeps working (it reads per-session files), so the audit trail
looks healthy from the CLI while the tamper-evident chain has flatlined.

For a tamper-evident audit ledger this is the worst failure direction: the
guarantee is silently absent rather than loudly broken.

2. Exit codes are clobbered

Measured with a corrupt ledger:

command expected actual
nono run -- /usr/bin/true 0 1
nono run -- /usr/bin/false 1 1
nono run -- /bin/sh -c 'exit 7' 7 1

Every sandboxed success becomes a failure and every distinct failure code
collapses to 1. Anything of the form nono run ... && next-step, or a CI step,
or a script branching on $?, is silently wrong. This is also independent of a
tty — same result with and without a pty.

Reproduction

Deterministic. Starting from a healthy ledger of N lines, join the last two
lines (i.e. delete one newline) and run anything:

L=~/.local/state/nono/audit/ledger.ndjson
cp "$L" "$L.bak"
n=$(wc -l < "$L")
awk -v last="$n" 'NR==last-1{printf "%s", $0; next} {print}' "$L.bak" > "$L"

nono run -- /bin/sh -c 'exit 7'; echo "exit=$?"
# nono: Snapshot error: Failed to parse audit ledger line <n-1>: trailing characters at line 1 column ...
# exit=1        <- expected 7
# ledger record count is unchanged, and stays unchanged for all later runs

cp "$L.bak" "$L"   # restore; exit codes return to normal immediately

I confirmed the restore returns exit=7 and recording resumes, so the corrupt
ledger is both necessary and sufficient for both symptoms.

Expected behaviour

  • A ledger that cannot be parsed should not affect the child's exit status. The
    exit code of nono run should be the child's, always.
  • Failure to append to the audit ledger should be loud and repeated — ideally
    refusing to run in an audited configuration, rather than degrading silently
    to "no audit" while continuing to sandbox.
  • Ideally nono should be able to recover: the defect is one missing byte, and
    the surrounding records are intact and chain-valid.

How the file got corrupted — not reproduced

Reporting honestly: I could not reproduce the original corruption.

The two colliding records had completed_at 0.64s apart. Both of these
failed to reproduce it on 0.71.0:

  • 10 concurrent nono run invocations — all 10 recorded, chain intact, no gaps
  • 20 runs killed mid-flight with SIGINT/SIGKILL at staggered delays across the
    window where the record is written — no malformed lines

So the trigger is still unknown, and the write path may be fine in the common
cases I could exercise. The defects above stand on their own regardless, since
they are reproducible from the corrupt state.

Possibly relevant: ~/.local/state/nono/audit/ledger.lock exists and is
0 bytes.

Repair, for anyone who hits this

The chain and every record were intact — only a newline was missing. Insert it
byte-exactly and do not reserialize (jq -c . splits the records correctly
but rewrites the JSON, which is unsafe for a hash-chained ledger):

# line 60 held two records; the second began at column 342
awk 'NR<60{print} NR==60{print substr($0,1,341); print substr($0,342)}' \
  ledger.ndjson.bak > ledger.ndjson

Verify losslessness — identical once newlines are stripped, one byte larger:

cmp <(tr -d '\n' < ledger.ndjson.bak) <(tr -d '\n' < ledger.ndjson)

After repair, nono audit verify <session> reports Ledger: verified (entries=85, head=...), the error is gone, exit codes propagate correctly, and
recording resumes.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions