Decoder deadlocks when `DECODER_OPTION_NUM_OF_THREADS` is set after `Initialize`

[adamroach]

Summary

Setting DECODER_OPTION_NUM_OF_THREADS (to any value ≥ 1) on an
already-initialized decoder reports success (SetOption returns
cmResultSuccess, and GetOption reads the value back), but leaves the
decoder in a state where:

1. the first DecodeFrameNoDelay call never returns — it blocks forever
   in ThreadDecodeFrameInternal waiting on a semaphore, and
2. if no decode call is made, WelsDestroyDecoder never returns — it
   blocks forever in CloseDecoderThreads.

Setting the same option before Initialize works correctly (decode and
teardown both complete). The failure mode is silent: because SetOption
succeeds and GetOption confirms the value, the API gives no indication
that the decoder has been rendered unusable.

The process sample of both hangs shows only a single thread: the decoder is
executing its threaded decode/teardown paths, but no worker threads exist —
consistent with the thread pool being created only during Initialize,
while the post-init SetOption still switches the decoder onto the
threaded code paths that wait on pool semaphores no worker will ever post.

Environment

• OpenH264 2.6.0 (WelsGetCodecVersion reports 2.6.0.2502), Homebrew
  build from the v2.6.0 release source
• macOS 26.5 (Apple Silicon, arm64), Apple clang 17.0.0
• Reproduction is plain C against wels/codec_api.h with no other
  dependencies

Steps to reproduce

cc -o repro repro.c $(pkg-config --cflags --libs openh264)

./repro 0 post decode    # control: no threading           -> clean
./repro 1 post decode    # threads=1 AFTER Initialize      -> DEADLOCK in DecodeFrameNoDelay
./repro 2 post decode    # threads=2 AFTER Initialize      -> DEADLOCK in DecodeFrameNoDelay
./repro 1 pre  decode    # threads=1 BEFORE Initialize     -> clean
./repro 1 post destroy   # threads=1 AFTER Initialize,
                         #   destroy without decoding      -> DEADLOCK in WelsDestroyDecoder
./repro 1 pre  destroy   # threads=1 BEFORE Initialize     -> clean

The embedded bitstream is a single access unit (SPS+PPS+IDR,
Annex-B) produced by this same OpenH264 build, and decodes to a frame in
the non-threaded control run.

Minimal reproduction (repro.c)

// repro.c — OpenH264: DECODER_OPTION_NUM_OF_THREADS set after Initialize
// deadlocks the decoder (first decode call and/or teardown never return).
//
// Build:  cc -o repro repro.c $(pkg-config --cflags --libs openh264)
// Usage:  ./repro <threads> <pre|post> <decode|destroy>
//   threads:  value for DECODER_OPTION_NUM_OF_THREADS (0 = never set it)
//   pre|post: set the option before or after ISVCDecoder::Initialize
//   decode|destroy: decode one access unit, or skip straight to teardown
//
// A watchdog aborts with "DEADLOCK" if any step takes longer than 8 s;
// every step otherwise completes in well under a millisecond.

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#include <wels/codec_api.h>

// One 64x48 H.264 access unit (SPS+PPS+IDR, Annex-B), 44 bytes,
// produced by this same OpenH264 build.
static const unsigned char kAccessUnit[44] = {
    0x00, 0x00, 0x00, 0x01, 0x67, 0x42, 0xc0, 0x0a, 0x8c, 0x68, 0x47, 0x20,
    0x1e, 0x11, 0x08, 0xd4, 0x00, 0x00, 0x00, 0x01, 0x68, 0xce, 0x3c, 0x80,
    0x00, 0x00, 0x00, 0x01, 0x65, 0xb8, 0x00, 0x04, 0x13, 0x93, 0x93, 0x93,
    0x93, 0xab, 0xae, 0xba, 0xea, 0x4a, 0xeb, 0xaf,
};

static const char *g_step = "startup";

static void on_timeout(int sig) {
    (void)sig;
    fprintf(stderr, "DEADLOCK: no return from step \"%s\" after 8s\n", g_step);
    _exit(2);
}

static void step(const char *name) {
    g_step = name;
    fprintf(stderr, "-> %s\n", name);
    alarm(8);
}

int main(int argc, char **argv) {
    if (argc != 4) {
        fprintf(stderr, "usage: %s <threads> <pre|post> <decode|destroy>\n", argv[0]);
        return 1;
    }
    int threads = atoi(argv[1]);
    int pre = strcmp(argv[2], "pre") == 0;
    int decode = strcmp(argv[3], "decode") == 0;

    signal(SIGALRM, on_timeout);

    OpenH264Version v = WelsGetCodecVersion();
    fprintf(stderr, "OpenH264 %u.%u.%u.%u\n", v.uMajor, v.uMinor, v.uRevision, v.uReserved);

    ISVCDecoder *dec = NULL;
    step("WelsCreateDecoder");
    if (WelsCreateDecoder(&dec) != 0 || !dec) {
        fprintf(stderr, "create failed\n");
        return 1;
    }

    if (threads > 0 && pre) {
        step("SetOption(NUM_OF_THREADS) before Initialize");
        long r = (*dec)->SetOption(dec, DECODER_OPTION_NUM_OF_THREADS, &threads);
        fprintf(stderr, "   SetOption returned %ld\n", r);
    }

    SDecodingParam param;
    memset(&param, 0, sizeof(param));
    param.sVideoProperty.size = sizeof(param.sVideoProperty);
    param.sVideoProperty.eVideoBsType = VIDEO_BITSTREAM_AVC;
    param.eEcActiveIdc = ERROR_CON_DISABLE;

    step("Initialize");
    if ((*dec)->Initialize(dec, &param) != 0) {
        fprintf(stderr, "Initialize failed\n");
        return 1;
    }

    if (threads > 0 && !pre) {
        step("SetOption(NUM_OF_THREADS) after Initialize");
        long r = (*dec)->SetOption(dec, DECODER_OPTION_NUM_OF_THREADS, &threads);
        fprintf(stderr, "   SetOption returned %ld\n", r);
    }

    if (threads > 0) {
        int got = -1;
        step("GetOption(NUM_OF_THREADS)");
        (*dec)->GetOption(dec, DECODER_OPTION_NUM_OF_THREADS, &got);
        fprintf(stderr, "   NUM_OF_THREADS reads back %d\n", got);
    }

    if (decode) {
        unsigned char *dst[3] = {0};
        SBufferInfo info;
        memset(&info, 0, sizeof(info));
        step("DecodeFrameNoDelay");
        DECODING_STATE st = (*dec)->DecodeFrameNoDelay(
            dec, kAccessUnit, (int)sizeof(kAccessUnit), dst, &info);
        fprintf(stderr, "   state=0x%x iBufferStatus=%d\n", st, info.iBufferStatus);
    }

    step("Uninitialize");
    (*dec)->Uninitialize(dec);
    step("WelsDestroyDecoder");
    WelsDestroyDecoder(dec);

    alarm(0);
    fprintf(stderr, "clean exit\n");
    return 0;
}

Observed output

$ ./repro 0 post decode
OpenH264 2.6.0.2502
-> WelsCreateDecoder
-> Initialize
-> DecodeFrameNoDelay
   state=0x0 iBufferStatus=1
-> Uninitialize
-> WelsDestroyDecoder
clean exit

$ ./repro 1 post decode
OpenH264 2.6.0.2502
-> WelsCreateDecoder
-> Initialize
-> SetOption(NUM_OF_THREADS) after Initialize
   SetOption returned 0
-> GetOption(NUM_OF_THREADS)
   NUM_OF_THREADS reads back 1
-> DecodeFrameNoDelay
DEADLOCK: no return from step "DecodeFrameNoDelay" after 8s

$ ./repro 2 post decode
OpenH264 2.6.0.2502
-> WelsCreateDecoder
-> Initialize
-> SetOption(NUM_OF_THREADS) after Initialize
   SetOption returned 0
-> GetOption(NUM_OF_THREADS)
   NUM_OF_THREADS reads back 2
-> DecodeFrameNoDelay
DEADLOCK: no return from step "DecodeFrameNoDelay" after 8s

$ ./repro 1 pre decode
OpenH264 2.6.0.2502
-> WelsCreateDecoder
-> SetOption(NUM_OF_THREADS) before Initialize
   SetOption returned 0
-> Initialize
-> GetOption(NUM_OF_THREADS)
   NUM_OF_THREADS reads back 1
-> DecodeFrameNoDelay
   state=0x0 iBufferStatus=0
-> Uninitialize
-> WelsDestroyDecoder
clean exit

$ ./repro 1 post destroy
OpenH264 2.6.0.2502
-> WelsCreateDecoder
-> Initialize
-> SetOption(NUM_OF_THREADS) after Initialize
   SetOption returned 0
-> GetOption(NUM_OF_THREADS)
   NUM_OF_THREADS reads back 1
-> Uninitialize
-> WelsDestroyDecoder
DEADLOCK: no return from step "WelsDestroyDecoder" after 8s

$ ./repro 1 pre destroy
OpenH264 2.6.0.2502
-> WelsCreateDecoder
-> SetOption(NUM_OF_THREADS) before Initialize
   SetOption returned 0
-> Initialize
-> GetOption(NUM_OF_THREADS)
   NUM_OF_THREADS reads back 1
-> Uninitialize
-> WelsDestroyDecoder
clean exit

(The DEADLOCK lines are printed by the reproduction's own 8-second
watchdog; without it, the affected calls block indefinitely.)

Stack traces at the point of deadlock

Captured with macOS sample while the process was hung; in both cases the
process contains only the main thread — no pool workers were ever
created.

./repro 1 post decode, hung in the first decode call:

1710 main
  1710 WelsDec::CWelsDecoder::DecodeFrameNoDelay(unsigned char const*, int, unsigned char**, TagBufferInfo*)
    1710 WelsDec::CWelsDecoder::ThreadDecodeFrameInternal(unsigned char const*, int, unsigned char**, TagBufferInfo*)
      1041 SemWait
      + 629 _pthread_cond_wait
      ...

./repro 1 post destroy, hung in teardown:

1719 main
  1719 WelsDec::CWelsDecoder::~CWelsDecoder()
    1719 WelsDec::CWelsDecoder::~CWelsDecoder()
      1719 WelsDec::CWelsDecoder::CloseDecoderThreads()
        1073 SemWait
        + 641 _pthread_cond_wait
        ...

Expected behavior

One of:

• SetOption(DECODER_OPTION_NUM_OF_THREADS) on an initialized decoder is
  rejected with an error, leaving the decoder in its working
  single-threaded state; or
• the option takes effect safely post-initialization (the thread pool is
  created on demand), so the subsequent decode/teardown paths have workers
  to wait on.

Either way, a SetOption that returns cmResultSuccess (and whose value
GetOption reads back) should not leave the decoder unable to complete any
subsequent decode or to be destroyed.