Flaky async tests due to missing explicit condition awaits in heka-identity-service

[deepshikhatutorials]

Description

What's going on

There's a recurring comment scattered across heka-identity-service/test/*: // Find a way to explicitly await the required condition. This isn't a random note left in a single test — it shows up in multiple test files across the suite, which points to a systemic pattern in how async tests were written in this codebase, rather than a one-off mistake.

Where exactly the problem is

The comment appears right before (or near) an assertion that depends on some asynchronous operation having completed — for example, a database write finishing, a record's status changing, an event being processed, or a background job wrapping up. Instead of the test actually confirming that the operation has finished, it's almost certainly relying on something like a fixed setTimeout() delay, or simply proceeding immediately and hoping the timing happens to work out.

Why this is a real problem, not just messy code

1. The tests are flaky by design.
A fixed delay is a guess, not a guarantee. If the async operation happens to finish faster than the delay, the test passes. If the system is under load, CI runs slower that day, or the operation genuinely takes a bit longer than usual, the exact same test — with zero code changes — fails. That's the textbook definition of a flaky test.

2. Flaky tests quietly destroy trust in CI.
Once developers start seeing "test X failed again, just rerun it," they stop treating failures as meaningful signals. Over time, real bugs can slip through because a failure gets dismissed as "probably just flaky" instead of being investigated properly.

3. It's possibly hiding actual race conditions in the application code.
These tests were never able to deterministically prove that "operation A completes before B happens." So while the test suite gives the impression that things work correctly, we don't actually have confirmation of that — we've just been getting lucky often enough not to notice a real timing bug underneath.

4. This isn't isolated to one test.
Because the same comment shows up repeatedly across different files, whatever async-handling approach was used when this test suite was written is systemically missing a proper "wait for real completion" mechanism. This wasn't a mistake made once and left unfixed — it looks like a known gap that got repeated across the suite as more tests were added.

How to see it yourself

Search the test directory for the comment, and you'll get every file and line where this shows up:

Running this will show the actual scope of the problem — how many tests are affected, and which parts of the codebase (issuance flow, verification flow, etc.) are involved.

Impact

• Unreliable CI signal — intermittent failures reduce confidence in the pipeline and slow down debugging, since a red build doesn't reliably mean something actually broke.
• Masked race conditions — if there's a genuine timing bug in the application code, these tests are not currently capable of catching it consistently.
• Maintenance overhead — every flaky failure costs a developer's time to investigate, re-run, or dismiss, which adds up across a team over time.

What "fixed" looks like

A corrected version of these tests wouldn't depend on timing at all. Instead, each test would explicitly poll or wait for the actual condition it cares about (for example, "keep checking until this record's status becomes issued") and only proceed once that condition is confirmed true — with a sensible timeout so the test still fails clearly and quickly if the condition is never met, rather than hanging indefinitely or passing/failing based on luck.

Proposed Solution

Introduce a shared polling/wait utility that tests can use instead of relying on fixed delays — something like a waitUntil(conditionFn, { timeout, interval }) helper that keeps checking a condition on a short interval and resolves as soon as it's true, or throws a clear error if it times out.

Example:

async function waitUntil(
  conditionFn: () => Promise<boolean> | boolean,
  { timeout = 5000, interval = 100 } = {},
): Promise<void> {
  const start = Date.now()
  while (Date.now() - start < timeout) {
    if (await conditionFn()) return
    await new Promise((resolve) => setTimeout(resolve, interval))
  }
  throw new Error('Condition not met within timeout')
}

Each flagged test would then swap its current wait logic for a call to this helper, passing in whatever specific condition it's actually waiting on (e.g. a record reaching a certain status, a webhook firing, a queue draining).

This is scoped to test infrastructure only — no application code changes expected, unless a specific test investigation uncovers an actual race condition, which would then be filed separately as its own bug.

Steps to reproduce

Steps to Reproduce

1. Clone the repository and navigate to the heka-identity-service directory.

2. Search the test suite for the marker comment to identify the affected tests:

This will list every file and line number where a test is relying on an implicit or fixed-delay wait instead of an explicit condition check.

3. Open one of the flagged test files and inspect the code around the flagged line. Confirm that the test either:

   • Uses a fixed-duration delay (e.g. setTimeout() / sleep(ms)) before making its assertion, or
   • Proceeds immediately after triggering an async operation, without confirming that operation has actually completed.

4. Run the affected test file in isolation several times in a row to observe inconsistent results:
   (Adjust the exact test command/flag based on the test runner configured in this repo — e.g. Mocha, Jest.)

5. Optionally, simulate a slower environment to increase the failure rate and make the flakiness more visible — for example, by artificially throttling CPU/network, or by adding a small artificial delay inside the code path the test depends on, then re-running the test.

6. Observe that the test passes on some runs and fails on others, without any changes to the underlying application logic — confirming the test's outcome depends on timing rather than on a deterministic, awaited condition.

Expected Behavior

The test should reliably pass or fail based on the actual correctness of the code being tested, regardless of system timing or load — by explicitly waiting for the specific condition it depends on (e.g. a status change, an event, a completed write) rather than an arbitrary delay.

Actual Behavior

The test's outcome is inconsistent across runs due to reliance on implicit/fixed timing instead of an explicit wait for the required condition, as indicated by the // Find a way to explicitly await the required condition comment left in the code.

Additional context

No response

Hedera network

No response

Version

v0.3.1

Operating system

None