Voice Interaction Service

Dịch vụ tương tác giọng nói thời gian thực với kiến trúc streaming pipeline cho tiếng Việt.

Tổng quan

Voice Interaction Service là hệ thống xử lý giọng nói real-time kết hợp VAD → ASR → LLM → TTS sử dụng 100% Cloud API. Kiến trúc 3 tầng tách biệt:

Tầng	Công nghệ	Provider
VAD	Voice Activity Detection	WebRTC VAD (CPU, no model)
ASR	Speech-to-Text	OpenAI Whisper API
LLM	Chatbot	OpenRouter API (Gemini, GPT-4o, Claude, Llama...)
TTS	Text-to-Speech	OpenAI TTS API

Điểm đặc biệt: Không cần GPU, không cần download local models, chạy trên bất kỳ máy nào.

Mục tiêu: Latency P95 < 900ms, hỗ trợ barge-in để user chen ngang khi AI đang nói.

Kiến trúc

Client (Web/App)
        │
        ▼ WebSocket (opus 48kHz)
┌──────────────────┐
│    Gateway       │ ◄── WebSocket management, session
│   (FastAPI)      │ ◄── Audio decode, resample
└────────┬─────────┘
          │ Audio (16kHz PCM)
          ▼
┌──────────────────┐
│  Orchestrator    │ ◄── Pipeline coordinator
│   (FastAPI)      │ ◄── ASR → LLM → TTS
└────────┬─────────┘
          │
          ▼ OpenAI API
┌──────────────────┐
│  Whisper API     │ ◄── Audio → Text
│  (openai.com)    │
└──────────────────┘
          │
          ▼ OpenRouter API
┌──────────────────┐
│  OpenRouter      │ ◄── Gemini, GPT-4o, Claude...
│  (openrouter.ai) │
└──────────────────┘
          │
          ▼ OpenAI API
┌──────────────────┐
│  TTS API         │ ◄── Text → Audio
│  (openai.com)    │
└──────────────────┘

So sánh Local vs Cloud API

Tiêu chí	Local (cũ)	Cloud API (mới)
GPU	Cần GPU T4/L4	Không cần
Model download	5-10GB models	Không cần
Docker image	~5GB (cuda)	~200MB
Latency	~200ms	~500ms
Scale	Cố định GPU	Vô hạn
Chi phí vận hành	~$200/tháng	~$100/tháng

Luồng hoạt động

sequenceDiagram
    participant Client
    participant Gateway
    participant VAD
    participant ASR
    participant LLM
    participant TTS

    Note over Client,Gateway: 1. Client connect WS
    Client->>Gateway: WS connect /stream/{session_id}
    Gateway->>Gateway: Create session

    Note over Client,Gateway: 2. Audio streaming loop
    loop
        Client->>Gateway: Audio (opus 48kHz)
        Note over Gateway: VAD.detect (WebRTC, local)
        Gateway->>TTS: TTS.cancel (if barge-in)
        Gateway->>TTS: TTS.cancel (if interrupt)
        Gateway->>ASR: ASR.recognize (OpenAI Whisper API)
        ASR-->>Gateway: ASRResult
        Gateway->>Client: transcript result
        Gateway->>LLM: chat (OpenRouter API)
        LLM-->>Gateway: response text
        Gateway->>TTS: TTS.speak (OpenAI TTS API)
        loop
            TTS-->>Gateway: TTSResult(audio)
            Gateway->>Client: Audio (PCM)
        end
        Gateway->>Client: done
    end

Luồng chi tiết theo Component

flowchart TD
    subgraph Client
        WS[WebSocket Client]
        HTTP[HTTP Client]
    end

    subgraph Gateway
        WSE[WebSocket Endpoint]
        HTTPE[HTTP Endpoint]
        SM[Session Manager]
        AP[AudioProcessor]
        VAD[VAD - WebRTC]
    end

    subgraph Orchestrator
        PE[Pipeline Endpoint]
        OE[Orchestrator Engine]
        CA[ChatbotAdapter]
        ASR[ASR - Whisper]
        TTS[TTS - OpenAI]
    end

    subgraph External
        OAI[OpenAI API]
        OR[OpenRouter API]
    end

    WS -->|1. Connect| WSE
    WSE --> SM
    SM --> AP
    
    WS -->|2. Audio| AP
    AP --> VAD
    VAD -->|2a. Barge-in| AP
    
    AP --> ASR
    ASR -->|3. Transcript| AP
    
    HTTP -->|4. POST| PE
    PE --> OE
    OE --> ASR
    ASR -->|API| OAI
    OAI --> ASR
    ASR --> OE
    OE --> CA
    CA -->|API| OR
    OR --> CA
    CA --> OE
    OE --> TTS
    TTS -->|API| OAI
    OAI --> TTS
    TTS --> OE
    OE -->|5. Response| HTTP

    AP --> TTS
    TTS -->|6. Audio| AP
    AP --> WS

Giải thích các trạng thái session

Trạng thái	Mô tả
`IDLE`	Session đang chờ, không hoạt động
`LISTENING`	AI đang nghe input từ user
`PROCESSING`	Đang xử lý (ASR → LLM → TTS)
`SPEAKING`	AI đang nói, output TTS đang streaming
`INTERRUPTED`	User đã chen ngang (barge-in)

Barge-in Flow

sequenceDiagram
    participant User
    participant Gateway
    participant TTS

    Note over User,Gateway: User speaks while AI is speaking
    User->>Gateway: Audio (barge-in)
    
    Note over Gateway: VAD.detect (WebRTC, local)
    
    alt confidence > 0.7
        Gateway->>TTS: cancel(stream_id)
        Gateway->>User: {type: interrupted}
    else confidence <= 0.7
        Note over Gateway: Ignore audio
    end

Cấu trúc project

voice-interaction/
├── apps/
│   ├── gateway/               # FastAPI WebSocket server
│   │   ├── src/
│   │   │   ├── routes/        # websocket.py, http.py
│   │   │   ├── session/       # manager.py
│   │   │   ├── pipeline/      # audio_processor.py
│   │   │   └── dependencies.py
│   │   ├── tests/
│   │   └── pyproject.toml
│   │
│   └── orchestrator/          # Pipeline coordinator
│       ├── src/
│       │   ├── routes/        # pipeline.py, session.py
│       │   ├── orchestrator/  # engine.py, context.py
│       │   └── adapters/      # chatbot.py (OpenRouter)
│       ├── tests/
│       └── pyproject.toml
│
├── services/
│   ├── asr/                   # ASR service (OpenAI Whisper API)
│   │   ├── src/
│   │   │   ├── interface.py
│   │   │   ├── openai_asr.py   # OpenAI Whisper API wrapper
│   │   │   └── factory.py
│   │   ├── tests/
│   │   └── pyproject.toml
│   │
│   ├── tts/                   # TTS service (OpenAI TTS API)
│   │   ├── src/
│   │   │   ├── interface.py
│   │   │   ├── openai_tts.py   # OpenAI TTS API wrapper
│   │   │   └── factory.py
│   │   ├── tests/
│   │   └── pyproject.toml
│   │
│   └── vad/                   # VAD service (WebRTC VAD)
│       ├── src/
│       │   ├── interface.py
│       │   ├── webrtc_vad.py  # WebRTC VAD (CPU, no model)
│       │   └── factory.py
│       ├── tests/
│       └── pyproject.toml
│
├── packages/
│   ├── common/                # Shared utilities
│   │   └── src/voice_common/
│   │       ├── audio/         # utils.py, resampler.py
│   │       ├── config/        # loader.py (load models.yaml)
│   │       ├── logging/       # setup.py
│   │       └── metrics/       # prometheus.py
│   │
│   ├── proto/                 # gRPC definitions
│   │   └── voice.proto
│   │
│   └── sdk-web/               # JavaScript client SDK
│       └── voice-client.js
│
├── infra/
│   ├── docker/                # Dockerfiles
│   ├── k8s/                   # Helm charts
│   ├── prometheus/            # prometheus.yml
│   └── docker-compose*.yml
│
├── configs/
│   └── models.yaml            # Model configuration
│
├── scripts/                   # Utility scripts
├── tests/                     # Integration tests
├── .github/workflows/         # CI/CD
├── pyproject.toml             # Root workspace
├── Makefile                   # Task runner
└── README.md                  # This file

Yêu cầu

Python 3.11+
GPU (T4/L4) cho ASR/TTS inference
Docker & Docker Compose
[Optional] Kubernetes cho production

Cài đặt

1. Install dependencies

# Clone và cài
git clone https://github.com/108-PJIOrthoGen/108POG_Voice_interaction.git
cd voice-interaction

# Tạo virtual environment (bắt buộc với Python 3.12+)
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies trước
pip install editables hatchling aiohttp structlog numpy pydantic pydantic-settings pyyaml scipy prometheus-client webrtcvad-wheels fastapi uvicorn openai python-multipart

# Install all packages (editable mode)
pip install --no-build-isolation -e packages/common/
pip install --no-build-isolation -e services/vad/
pip install --no-build-isolation -e services/asr/
pip install --no-build-isolation -e services/tts/
pip install --no-build-isolation -e apps/gateway/
pip install --no-build-isolation -e apps/orchestrator/

# Tạo symlinks để Python tìm thấy modules
cd services/asr && ln -sf src voice_asr && cd ../..
cd services/vad && ln -sf src voice_vad && cd ../..
cd services/tts && ln -sf src voice_tts && cd ../..
cd packages/common/src && ln -sf voice_common voice_common && cd ../../../..

# Install ffmpeg cho audio recording (Linux)
sudo apt install ffmpeg

# Hoặc macOS
brew install ffmpeg

# Hoặc dùng Makefile (sau khi activate venv)
make install

2. Cấu hình API Keys

Lưu ý: Dự án dùng 100% Cloud API - không cần download local models.

# Copy .env.example thành .env và điền API keys
cp .env.example .env
# Hoặc set env vars:
#   export ASR_API_KEY=sk-...
#   export TTS_API_KEY=sk-...
#   export CHATBOT_API_KEY=sk-or-v1-...

3. Chạy local

# Terminal 1: Gateway
cd /home/sotatek/Projects/voice-interaction
source .venv/bin/activate
make run-gateway

# Terminal 2: Orchestrator (cần cài python-multipart trước)
pip install python-multipart
make run-orchestrator

Hoặc chạy trực tiếp:

# Terminal 1
source .venv/bin/activate
uvicorn voice_gateway:app --host 0.0.0.0 --port 8000

# Terminal 2
source .venv/bin/activate
uvicorn voice_orchestrator:app --host 0.0.0.0 --port 8001

Test:

curl http://localhost:8000/health
curl http://localhost:8001/health

4. Docker Compose

# Production
docker compose -f infra/docker-compose.yml up --build

# Development (hot reload)
docker compose -f infra/docker-compose.yml -f infra/docker-compose.dev.yml up

5. Run tests

# Tất cả tests
make test

# Chỉ unit tests
make test-unit

# Integration tests
make test-integration

# Lint
make lint

6. Test voice với microphone

# Chạy script test voice (cần ffmpeg đã cài ở bước 1)
python scripts/test_voice.py

Yêu cầu:

Đang chạy Gateway (port 8000)
Đang chạy Orchestrator (port 8001)
Microphone được cấp quyền

Script sẽ:

Kết nối WebSocket tới /stream/{client_id}
Ghi âm từ microphone (16kHz mono)
Gửi audio lên Gateway
Nhận transcript và phản hồi từ AI

Test thủ công qua cURL:

# Test HTTP endpoint với file audio
curl -X POST http://localhost:8000/v1/process \
  -F "audio=@test.wav" \
  -F "language=vi"

Sử dụng

JavaScript SDK

<script src="packages/sdk-web/voice-client.js"></script>
<script>
const client = new VoiceClient({
  url: 'wss://your-server.com',
  debug: true
});

client.onTranscript((result) => {
  console.log(result.isFinal ? 'Final:' : 'Partial:', result.text);
});

client.onAudio((event) => {
  if (event.type === 'done') console.log('AI done speaking');
});

await client.connect();
await client.startRecording();

// Khi muốn ngắt AI đang nói (barge-in)
client.interrupt();

// Dừng recording
client.stopRecording();

// Disconnect
client.disconnect();
</script>

WebSocket API

const ws = new WebSocket('ws://localhost:8000/stream/session-123');

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  
  if (data.type === 'transcript') {
    console.log(data.isFinal ? data.text : `Partial: ${data.text}`);
  } else if (data.type === 'done') {
    console.log('TTS done');
  }
};

// Gửi audio (base64 encoded PCM 16-bit)
ws.send(audioBase64);

// Gửi text (fallback)
ws.send(JSON.stringify({ type: 'text', text: 'Xin chào' }));

// Interrupt
ws.send(JSON.stringify({ type: 'interrupt' }));

HTTP API

# Health check
curl http://localhost:8000/health

# TTS fallback
curl -X POST http://localhost:8000/tts/speak \
  -H "Content-Type: application/json" \
  -d '{"text": "Xin chào bạn", "session_id": "abc"}'

Cấu hình

Environment variables

# .env
GATEWAY_HOST=0.0.0.0
GATEWAY_PORT=8000

ORCHESTRATOR_HOST=0.0.0.0
ORCHESTRATOR_PORT=8001

# VAD (WebRTC - không cần API key)
VAD_AGGRESSIVENESS=3

# ASR - OpenAI Whisper API
ASR_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxx
ASR_BASE_URL=https://api.openai.com/v1
ASR_MODEL=whisper-1

# TTS - OpenAI TTS API
TTS_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxx
TTS_BASE_URL=https://api.openai.com/v1
TTS_MODEL=tts-1
TTS_VOICE=alloy

# Chatbot - OpenRouter API
CHATBOT_API_KEY=sk-or-v1-xxxxxxxxxxxxxxxxxxxx
CHATBOT_BASE_URL=https://openrouter.ai/api/v1
CHATBOT_MODEL=google/gemini-2.0-flash-001

# Config
CONFIG_PATH=configs/models.yaml

LOG_LEVEL=INFO

Lưu ý: KHÔNG bao giờ commit API keys thật. File .env đã được thêm vào .gitignore.

models.yaml

models:
  vad:
    name: "webrtc_vad"
    config:
      sample_rate: 16000
      frame_duration_ms: 30
      aggressiveness: 3

  asr:
    name: "openai_whisper"
    config:
      base_url: "https://api.openai.com/v1"
      model: "whisper-1"
      language: "vi"
      response_format: "json"
      timeout: 30

  tts:
    name: "openai_tts"
    config:
      base_url: "https://api.openai.com/v1"
      model: "tts-1"
      voice: "alloy"
      speed: 1.0
      response_format: "pcm"
      timeout: 30

# OpenRouter LLM configuration
chatbot:
  base_url: "https://openrouter.ai/api/v1"
  model: "google/gemini-2.0-flash-001"
  timeout: 30
  max_retries: 3
  max_tokens: 256
  temperature: 0.7
  streaming: true
  system_prompt: "Bạn là trợ lý ảo thân thiện, trả lời ngắn gọn bằng tiếng Việt."

OpenRouter Models

Model	Provider	Latency	Quality	Khuyến nghị
`google/gemini-2.0-flash-001`	Google	Rất thấp (~200ms)	Tốt	Recommended cho voice
`google/gemini-2.5-flash-preview`	Google	Thấp	Rất tốt	Reasoning tốt hơn
`anthropic/claude-3.5-sonnet`	Anthropic	Trung bình	Xuất sắc	Chất lượng cao nhất
`meta-llama/llama-3.1-8b-instruct`	Meta	Thấp	Khá	Budget option
`openai/gpt-4o-mini`	OpenAI	Thấp	Tốt	Alternative tốt

Deployment

Kubernetes

# Tạo secret cho tất cả API keys
kubectl create secret generic api-keys \
  --from-literal=asr-api-key='sk-xxxxxxxxxxxxxxxxxxxxxxxx' \
  --from-literal=tts-api-key='sk-xxxxxxxxxxxxxxxxxxxxxxxx' \
  --from-literal=chatbot-api-key='sk-or-v1-xxxxxxxxxxxxxxxxxxxx'

# Thêm Helm chart
helm repo add voice-interaction https://yourorg.github.io/voice-interaction

# Cài đặt
helm install voice-release voice-interaction/voice-interaction

Secret api-keys sẽ được inject vào các pods qua environment variables:

ASR_API_KEY → ASR service
TTS_API_KEY → TTS service
CHATBOT_API_KEY → Orchestrator (LLM)

Prometheus

Metrics available tại:

Gateway: http://localhost:8000/metrics
Orchestrator: http://localhost:8001/metrics

Metrics chính:

voice_latency_asr_seconds: ASR processing time
voice_latency_tts_seconds: TTS processing time
voice_latency_e2e_seconds: End-to-end latency
voice_sessions_active: Active sessions
voice_sessions_total: Total sessions
voice_vad_detections_total: VAD detections
voice_barge_in_total: Barge-in events

Latency targets

Stage	Target P50	Target P95
VAD → ASR	100ms	200ms
ASR → LLM	200ms	500ms
LLM → TTS	100ms	200ms
Total E2E	500ms	900ms

Contributing

Fork repository
Tạo branch: git checkout -b feature/your-feature
Commit changes: git commit -m 'Add something'
Push: git push origin feature/your-feature
Tạo Pull Request

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 110 Commits
.github/workflows		.github/workflows
apps		apps
configs		configs
infra		infra
packages		packages
scripts		scripts
services		services
tests/integration		tests/integration
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml

Uh oh!

Folders and files

Latest commit

History

Repository files navigation