task/ansible: mark nodes down on hardware failures

[djgalloway]

Fixes: https://tracker.ceph.com/issues/75669
Related: https://tracker.ceph.com/issues/75208

What

We already mark a node down after it fails to reimage 10 times in a row. When ceph-cm-ansible's Ensure we found enough OSD disks task fails, though, we know right away that the node is missing a disk — and every job that lands on it until someone notices will fail the same way.

This scans the ansible failure log for messages that mean the hardware itself is broken and marks the affected nodes down immediately.

• FailureAnalyzer.find_hardware_failures() parses the failure log (keyed by hostname) and returns {hostname: msg} for failures matching a known hardware-failure pattern. The one pattern for now is Wanted N disks of ... but only matched M. Whitespace is collapsed before matching because ansible wraps that message across lines.
• Ansible._handle_failure() feeds the hits to a new _mark_hardware_failures_down() hook — a no-op in the base class, implemented in CephLab as lock_ops.update_lock(hostname, status='down'). This mirrors the existing _set_status() split. It's wrapped in its own try/except so a lock server problem can't disturb the existing failure-reporting path.

Notes for reviewers

Two deliberate choices:

The lock description is left untouched. The dispatcher's 10-strikes path sets description='reimage failed 10 times', but it runs after unlocking. Here the job still holds the lock, and unlock_targets() refuses to unlock a node whose description no longer matches archive_path — so setting it would leave the dead node locked forever. Only up is flipped; the reason goes to the log.

We match on the message, not the task name. ceph-cm-ansible's callback_plugins/failure_log.py only records the result dict (msg, changed, ...), never the task name. Verified against a real ansible_failures.yaml from the tracker and against roles/testnode/tasks/configure_lvm.yml. Keying off the task name instead would require a callback plugin change in ceph-cm-ansible first.

Testing

8 new cases in tests/task/test_ansible.py: analyzer coverage (the verbatim message from job 116107, a line-wrapped variant, a results list, a non-matching failure, an empty log) plus end-to-end checks that CephLab marks the node down, Ansible does not, and a non-hardware failure does not.

tests/task/test_ansible.py: 105 passed, 6 skipped. Full suite: 1222 passed, 3 failed — the 3 are in tests/orchestra/test_connection.py and fail identically on a clean checkout of main.

[kshtsk]

The feature looks interesting, though I haven't read deeply what the criteria of marking host down.
But we should not do any calls to paddles from any task, otherwise it makes the teuthology run depended on not necessary infrastructure. The actual node manipulation are supposed to be done on supervisor side. Also, would be probably a good idea to have an option to disable this behavior on config level.

[kshtsk]

Also if we want this mechanism incorporated it is better to put the documentation into teuthology how all the stuff supposed to work, and that it is relied on ansible task or/and on ceph-cm-ansible role task and what format is expected to trigger the node marking down, etc.