[bug] Ingest processor: infinite retry loop on processed/timed-out files causes UI unresponsiveness and database corruption

[crocodilestick]

Summary

The ingest processor fails to properly track which files it has already handled, causing infinite retry loops. Processed or timed-out files are picked up again and again, with each retry spawning duplicate processing pipelines. This eventually overwhelms the container, making the web UI completely unresponsive and causing database corruption. The only recovery is a container restart.

Observed patterns

Duplicate processing (#1276):
Files that time out waiting to become ready get picked up again immediately instead of being skipped. Each retry locks the DB again and the logs show the same file being processed twice simultaneously:

[ingest-processor] WARN: File did not become ready in time (after 15 minutes)
[ingest-processor] Skipping delete for ignored file
[ingest-processor] Lock acquired successfully (PID: 59207)  # ← immediately retries same file

Infinite retry after successful import (#1326):
Books that have already been successfully imported and appear in the UI are still being retried from the ingest folder. Each new import batch compounds the problem — retries pile up and the UI becomes unresponsive after ~10 books.

Environment

• CWA v4.0.6
• Common trigger: NETWORK_SHARE_MODE=true, NFS/bind-mounted storage, >3 books in ingest
• Recovery: container restart only

Related (consolidated from)

• Ingest loop, webpage becomes unresponsive #1276 — Ingest loop, webpage becomes unresponsive
• [bug] Infinite ingestion loop #1326 — Infinite ingestion loop