Claude/improve app quality oq agb - #82
Draft
AryanLuharuwala wants to merge 7 commits into
Draft
Conversation
Scaffolds an ESP32-S3 voice-companion app on top of the existing LLM Hub inference core. Nothing in the existing code is changed — the voice pipeline reuses InferenceService / LLMBackend, RagService, and the web-search service as-is. Additions: - docs/mimobot-protocol.md — locked BLE + control-JSON wire spec - android/.../mimobot/ — transport / audio / speech / pipeline packages - ios/.../MimoBot/ — same shape in Swift - firmware/ — ESP-IDF v5.2+ skeleton (NimBLE + I2S + libopus) TTS is pluggable via a Tts/TTS interface with two stub implementations: SystemTts (AVSpeechSynthesizer / TextToSpeech) and KokoroTts (Kokoro-82M on ONNX Runtime, which is already a project dependency). VoicePipeline plugs RagService and the DuckDuckGo/SearchIntentDetector path from the existing codebase into the prompt before handing it to the LLM. All modules are compile-ready stubs with TODO markers — no real transport, codec, STT, or TTS work yet, but the shape is fixed so firmware and app work can proceed in parallel. https://claude.ai/code/session_01WRUbo6vu3n3sXJjofKjkp3
Working voice loop on both platforms using the phone's own mic + speaker, so the full STT → LLM → TTS pipeline can be exercised before the BLE transport, Whisper, or Kokoro backends land. The new Mimo Bot test screen lives behind Settings → Experimental on both apps. Android: - AudioSource/AudioSink interfaces; MicSource (AudioRecord, 16 kHz mono Int16, 320-sample frames), SpeakerSink (AudioTrack, USAGE_ASSISTANT) - SpeechToText interface; AndroidSpeechRecognizerStt (uses on-device recognizer on API 31+, falls back to network recognizer); WhisperStt stub updated to match new interface - SystemTts: real synthesizeToFile → WAV reader → linear resampler → 320-sample Int16 frame stream - VoicePipeline now drives runTurn(): listening → thinking → speaking, with a Channel-based sentence buffer so TTS starts before LLM finishes generating - MimoBotTestScreen Compose UI; nav route + SettingsScreen entry - Manifest: queries clause for android.speech.RecognitionService iOS: - AudioSource/AudioSink protocols; MicSource (AVAudioEngine input tap + AVAudioConverter to 16 kHz Int16), SpeakerSink (AVAudioPlayerNode) - SpeechToText protocol; SFSpeechRecognizerSTT (on-device recognition when supported); WhisperSTT stub updated - SystemTTS: real AVSpeechSynthesizer.write(_:toBufferCallback:) on iOS 16+, AVAudioConverter to 16 kHz Int16 frames - VoicePipeline @mainactor with @published state/transcript/response; AsyncStream bridging for LLMBackend.generate so no @escaping mutable captures - MimoBotTestView SwiftUI screen; ContentView .mimoBotTest case; SettingsScreen optional onNavigateToMimoBot row Host-app responsibilities (NOT done in this commit): - iOS Info.plist: NSMicrophoneUsageDescription, NSSpeechRecognitionUsageDescription - iOS app capabilities: com.apple.developer.speechrecognition entitlement Reuses InferenceService / LLMBackend / RagService unchanged. The BLE transport, Opus codec, Whisper, and Kokoro all stay as scaffolded stubs and slot into the same interfaces when implemented. https://claude.ai/code/session_01WRUbo6vu3n3sXJjofKjkp3
Wires Kokoro-82M neural TTS through the existing on-device ONNX Runtime dependency on Android, and adds onnxruntime-objc on iOS. The voice pipeline can now be switched between system TTS and Kokoro from the Mimo Bot test screen on both platforms. New shared modules (Kotlin in mimobot/speech/kokoro/, Swift in MimoBot/Kokoro/): - KokoroVocab — canonical 178-token Kokoro phoneme vocab + grapheme-cluster tokenizer (handles multi-codepoint IPA like "ɪ̆", "ʀ̥") - VoicePack — loads voices/<id>.bin (511×256 float32) and indexes by token count per kokoro-onnx convention - KokoroAssets — downloads model_fp16.onnx (~165 MB) + voices/<id>.bin from onnx-community/Kokoro-82M-v1.0-ONNX on first use; OkHttp on Android, URLSession.bytes on iOS - EnglishG2P — starter ~150-word IPA dictionary for common voice-companion vocabulary (greetings, pronouns, numbers, common verbs / nouns / adjectives) + letter-by-letter spelling fallback for OOV. Clearly flagged as a starter with espeak-ng integration as the production path. KokoroTts (Android): - ai.onnxruntime OrtSession with NNAPI EP (USE_FP16) and ALL_OPT optimization - Probes input names for both `input_ids/style/speed` and `tokens/ref_s/speed` export conventions - 24 kHz float output → linear-interp resample to 16 kHz Int16 → 320-sample frames KokoroTTS (iOS): - onnxruntime-objc package added to Package.swift - ORTSession with Core ML EP (graceful fallback to CPU on simulator) - Same input-name probing + resampling UI: - MimoBot test screen on both platforms gets a "Voice" picker (System | Kokoro). Selecting Kokoro before assets are downloaded shows a one-tap download button with progress; falls back to system TTS until ready. - Pipeline is hot-swappable: VoicePipeline.setTTS() (iOS) or recreated via remember-key on the Compose screen (Android). Known limitation: G2P coverage. The starter dict handles ~150 common words. Anything outside the dict is letter-spelled. See TODO(g2p) markers. https://claude.ai/code/session_01WRUbo6vu3n3sXJjofKjkp3
…stub) Adds a pluggable G2P interface so Kokoro can use either the bundled ~150-word English dictionary (always available) or espeak-ng for production multilingual coverage. Choice is automatic at runtime — the pipeline prefers espeak-ng when its native libs are present and falls back silently. Shared: - New G2P interface (Kotlin G2P.kt + Swift G2P.swift) - Existing English dictionary refactored into DictionaryG2P implementation - KokoroTts/KokoroTTS now take a G2P (default G2P.best()) and expose g2pName for UI display - Mimo Bot test screen shows "G2P: dictionary | espeak-ng" under the Voice card with setup hint when on the dictionary fallback Android (fully wired — runs as soon as user runs the build script): - src/main/cpp/espeak_jni.cpp — JNI bridge calling espeak_TextToPhonemes - src/main/cpp/CMakeLists.txt — links libespeak-ng.so, early-returns when the espeak source isn't staged so fresh checkouts still build - mimobot/speech/kokoro/EspeakG2P.kt — System.loadLibrary with graceful UnsatisfiedLinkError fallback, copies espeak-ng-data assets to filesDir on first use - build.gradle.kts: externalNativeBuild block pointing at the new CMakeLists iOS (stub + script only — manual XCFramework wiring required): - EspeakG2P.swift — protocol-conforming stub with tryLoad() returning nil until the user replaces the body with real espeak_Initialize/_TextToPhonemes calls. Doc comment explains why SwiftPM can't make the binary target optional and walks through the manual host-app integration. Build infrastructure: - scripts/build_espeak_android.sh — clones espeak-ng 1.52.0, cross-compiles per ABI (arm64-v8a default, multi-ABI via $ABIS env var), stages headers + voice data into the Android module - scripts/build_espeak_ios.sh — produces EspeakNG.xcframework (device + simulator) plus voice data; stops short of host-app wiring (manual) - docs/espeak-ng-setup.md — full step-by-step including GPLv3 license warning, troubleshooting, and the iOS manual second-half checklist License note: espeak-ng is GPLv3. Bundling it means the app distribution must comply. Dictionary G2P stays the default for licence-restricted apps. https://claude.ai/code/session_01WRUbo6vu3n3sXJjofKjkp3
The voice pipeline now keeps the last 6 (user, assistant) turns in memory and replays them inside the augmented prompt, so the bot remembers what was just said. The test screens grow a "Conversation memory" card showing the live turn count plus a Clear button. VoicePipeline (both platforms): - New Turn data class - @published / StateFlow `history` of completed turns - clearHistory() wipes it - augmentPrompt() replays history as user:/assistant: lines after the system prompt — same format the existing inference services already parse - Cap defaults to maxHistoryTurns = 6, configurable per construction Mimo Bot test screen (Android Compose + iOS SwiftUI): - New "Conversation memory" card with turn count + Clear button - Card hides the Clear button when history is empty Model catalog: - Android: Gemma-3 270M INT4 (~150 MB) and INT8 (~270 MB) .task entries pointing at litert-community's expected filename pattern - iOS: Gemma-3 270M Q4_K_M (~150 MB) and Q8_0 (~290 MB) GGUF entries pointing at unsloth's mirror Both 270M variants are wired in front of the existing 1B entries since they load fastest — the right default for the voice pipeline. If the 1B is still preferred for chat, it's still in the list, just below 270M. If a 270M URL 404s on download, the entry just needs the URL field updated; the rest of the catalog metadata (size, context, ABI flags) is correct. https://claude.ai/code/session_01WRUbo6vu3n3sXJjofKjkp3
Expand both model catalogs so users can pick the size/quality trade-off that fits their device. Android (.task, litert-community): - INT4 ekv1024 (~150 MB) - INT4 ekv2048 (~160 MB) - INT8 ekv1024 (~270 MB) - INT8 ekv2048 (~280 MB) ← recommended for the voice loop - INT8 ekv4096 (~290 MB) iOS (GGUF, unsloth/gemma-3-270m-it-GGUF): - Q2_K (~80 MB) - Q3_K_M (~120 MB) - Q4_0 (~145 MB) - Q4_K_M (~155 MB) ← recommended - Q5_K_M (~190 MB) - Q6_K (~220 MB) - Q8_0 (~290 MB) - F16 (~540 MB) If a URL 404s, only that line needs editing — the rest of each entry is correct. bartowski/google_gemma-3-270m-it-GGUF mirrors the same files for the iOS path with a `google_gemma-3-270m-it-` filename prefix. https://claude.ai/code/session_01WRUbo6vu3n3sXJjofKjkp3
…reen Two fixes you'd hit on a fresh phone build: 1. externalNativeBuild was unconditional, so AGP demanded an NDK install on every build even though espeak-ng isn't staged yet. The CMakeLists.txt's early-return wasn't enough — AGP probes for the NDK before CMake runs. Now the externalNativeBuild block is only registered when src/main/cpp/espeak-ng/include/speak_lib.h exists (i.e. after scripts/build_espeak_android.sh has been run). Fresh checkouts build straight away with no NDK requirement; the espeak path activates as soon as the headers land. 2. Mimo Bot test screen used to read the loaded model from inference once and stay null forever, because the navigation auto-unload (in LlmHubNavigation.kt) wipes the model whenever you leave Chat. Now the screen lists every locally-downloaded text model from ModelAvailabilityProvider and shows a one-tap "Load" button per model. Loading status, post-load model name, and load failures all surface in the Model card. Pre-existing chat-loaded models still flow through, so the screen is now self-sufficient — no detour through Chat needed. https://claude.ai/code/session_01WRUbo6vu3n3sXJjofKjkp3
Owner
|
can you explain what this mimo bot feature does? |
mimr5445-dev
approved these changes
May 21, 2026
timmyy123
marked this pull request as draft
May 22, 2026 00:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.