feat(embedding): add jina and mistral providers, fix stale install hints (#154)

Compared MFS's embedding provider set against upstream Memsearch (the
project this code was originally adapted from) and found two gaps:

- jina and mistral are supported upstream but missing here. Both are
  ported over (jina talks to Jina's embeddings REST API directly over
  the core httpx dep, no extra needed; mistral uses the official SDK
  behind a new `mistral` extra), wired into the provider registry, the
  setup wizard, and docs/providers.md.
- openai.py, voyage.py, ollama.py, and local.py still told users to
  `pip install memsearch` in their module docstrings — leftover from
  when this code was copied over. Fixed to say mfs-server.

Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>

Author: zc277584121

README.md

@@ -638,7 +638,7 @@ What each section of `server.toml` configures:
 
 | Section | Configures | Default | Swap in |
 |---|---|---|---|
-| `[embedding]` | embedding provider · model · dim | local ONNX (BGE-M3 int8, no key) | `openai` · `gemini` · `voyage` · `ollama` · `local` — needs that provider's key/extra |
+| `[embedding]` | embedding provider · model · dim | local ONNX (BGE-M3 int8, no key) | `openai` · `gemini` · `voyage` · `jina` · `mistral` · `ollama` · `local` — needs that provider's key/extra |
 | `[milvus]` | vector store | Milvus Lite (file under `~/.mfs/`) | self-hosted Milvus or [Zilliz Cloud](https://cloud.zilliz.com/signup?utm_source=github&utm_medium=referral&utm_campaign=mfs-readme) — set `uri` + `token` |
 | `[database]` | metadata + transformation-cache DB | SQLite (file) | Postgres — set `dsn` |
 | `[artifact_cache]` | converted-blob cache | local fs under `~/.mfs/cache` | size / eviction knobs |

docs/providers.md

@@ -16,16 +16,20 @@ topologies, use [Deployment](deployment.md).
 | Hosted OpenAI embeddings | `openai` embeddings | `uv tool install mfs-server`, then `mfs-server setup --section embedding` | `OPENAI_API_KEY` |
 | Google Gemini embeddings | `gemini` embeddings | `uv tool install "mfs-server[gemini]"`, then `mfs-server setup --section embedding` | `GOOGLE_API_KEY`, or Vertex AI auth for the embedding SDK |
 | Voyage embeddings | `voyage` embeddings | `uv tool install "mfs-server[voyage]"`, then `mfs-server setup --section embedding` | `VOYAGE_API_KEY` |
+| Jina embeddings | `jina` embeddings | `uv tool install mfs-server`, then `mfs-server setup --section embedding` | `JINA_API_KEY` |
+| Mistral embeddings | `mistral` embeddings | `uv tool install "mfs-server[mistral]"`, then `mfs-server setup --section embedding` | `MISTRAL_API_KEY` |
 | Local Ollama embeddings | `ollama` embeddings | `uv tool install "mfs-server[ollama]"`, then `mfs-server setup --section embedding` | Running Ollama server; `OLLAMA_HOST` is optional |
 | Local sentence-transformers embeddings | `local` embeddings | `uv tool install "mfs-server[local]"`, then `mfs-server setup --section embedding` | None; this extra pulls the sentence-transformers stack |
 | Image-description and summary setup | `openai`, `anthropic`, or `gemini` LLM/VLM | `mfs-server setup --section description` (and `--section summary`) | Provider key for the selected LLM/VLM provider |
 
 Provider names are exact. The supported embedding names are `openai`, `onnx`,
-`gemini`, `voyage`, `ollama`, and `local`.
+`gemini`, `voyage`, `jina`, `mistral`, `ollama`, and `local`.
 
 ## Install Paths
 
-The base install includes the OpenAI SDK and the default ONNX embedding stack:
+The base install includes the OpenAI SDK and the default ONNX embedding stack.
+`jina` also needs no extra — it talks to Jina's REST API over the core `httpx`
+dependency:
 
 ```bash
 uv tool install mfs-server
@@ -36,14 +40,16 @@ Alternate provider extras are separate:
 ```bash
 uv tool install "mfs-server[gemini]"
 uv tool install "mfs-server[voyage]"
+uv tool install "mfs-server[mistral]"
 uv tool install "mfs-server[ollama]"
 uv tool install "mfs-server[local]"
 uv tool install "mfs-server[anthropic]"
 uv tool install "mfs-server[all-providers]"
 ```
 
-`all-providers` installs Gemini, Voyage, Ollama, and Anthropic provider
-dependencies. It does not include `local`, because `local` pulls the larger
+`all-providers` installs Gemini, Voyage, Mistral, Ollama, and Anthropic
+provider dependencies (`jina` needs no extra, so it's already covered). It
+does not include `local`, because `local` pulls the larger
 sentence-transformers dependency stack.
 
 ## Embedding Providers
@@ -57,6 +63,8 @@ models.
 | `openai` | `text-embedding-3-small` | 1536 for the default model | Core | `OPENAI_API_KEY`; `OPENAI_BASE_URL` is optional | No local model download. Unknown model dimensions may require a trial embedding call. |
 | `gemini` | `gemini-embedding-001` | 768 for the default model | `uv tool install "mfs-server[gemini]"` or `all-providers` | `GOOGLE_API_KEY`, or Vertex AI auth with `GOOGLE_GENAI_USE_VERTEXAI=true` | Known model dimensions use a local table; unknown models require a trial embedding call. |
 | `voyage` | `voyage-3-lite` | 512 for the default model | `uv tool install "mfs-server[voyage]"` or `all-providers` | `VOYAGE_API_KEY` | Known model dimensions use a local table; unknown models require a trial embedding call. |
+| `jina` | `jina-embeddings-v4` | 2048 for the default model (Matryoshka-truncatable 256–2048) | Core (talks to the Jina REST API over `httpx`) | `JINA_API_KEY` | No local model download. Known model dimensions use a local table; unknown models fall back to 2048. |
+| `mistral` | `mistral-embed` | 1024 for the default model | `uv tool install "mfs-server[mistral]"` or `all-providers` | `MISTRAL_API_KEY` | Known model dimensions use a local table; unknown models require a trial embedding call. |
 | `ollama` | `nomic-embed-text` | Detected by a trial embed against the selected Ollama model | `uv tool install "mfs-server[ollama]"` or `all-providers` | Running Ollama server; `OLLAMA_HOST` can point at a non-default host | The selected model must be available to the Ollama server before dimension probing and embedding can succeed. |
 | `local` | `all-MiniLM-L6-v2` | Detected from sentence-transformers model metadata | `uv tool install "mfs-server[local]"` | No API key | `mfs-server run` and `mfs-server worker` preload the sentence-transformers model on the detected device: CUDA, MPS, then CPU. |
 

server/python/pyproject.toml

@@ -85,17 +85,21 @@ embedding-onnx = []
 # mfs_server.common.embeddings).
 gemini = ["google-genai>=0.3"]          # gemini embedding + LLM (vision-capable)
 voyage = ["voyageai>=0.3"]              # voyage-3-lite etc.
+mistral = ["mistralai>=1.0"]            # mistral-embed / codestral-embed
+# jina: no extra — talks to the REST API directly over the core httpx dep
 ollama = ["ollama>=0.4"]                # local LLM/embedding via Ollama server
 local = ["sentence-transformers>=3.0", "einops>=0.8"]   # CPU/GPU sentence-transformers
 
 # Alternate LLM providers for summary / VLM (mfs_server.common.llm).
 anthropic = ["anthropic>=0.40"]         # claude-sonnet-* (text + vision)
 
 # Convenience bundle: every provider (text + vision LLMs and remote embeddings).
-# Excludes 'local' (sentence-transformers pulls torch ~2 GB).
+# Excludes 'local' (sentence-transformers pulls torch ~2 GB). jina needs no
+# extra (core httpx), so it's already covered without appearing here.
 all-providers = [
     "google-genai>=0.3",
     "voyageai>=0.3",
+    "mistralai>=1.0",
     "ollama>=0.4",
     "anthropic>=0.40",
 ]

server/python/src/mfs_server/common/embedding.py

@@ -5,7 +5,8 @@
 api_calls / cache_hits for observability/tests.
 
 Provider selection lives in `mfs_server.common.embeddings` (lazy imports).
-Supported names today: onnx (default), openai, google, voyage, ollama, local.
+Supported names today: onnx (default), openai, gemini, voyage, jina, mistral,
+ollama, local.
 """
 
 from __future__ import annotations

server/python/src/mfs_server/common/embeddings/__init__.py

@@ -24,6 +24,8 @@ async def embed(self, texts: list[str]) -> list[list[float]]: ...
     "onnx": ("mfs_server.common.embeddings.onnx", "OnnxEmbedding"),
     "gemini": ("mfs_server.common.embeddings.gemini", "GeminiEmbedding"),
     "voyage": ("mfs_server.common.embeddings.voyage", "VoyageEmbedding"),
+    "jina": ("mfs_server.common.embeddings.jina", "JinaEmbedding"),
+    "mistral": ("mfs_server.common.embeddings.mistral", "MistralEmbedding"),
     "ollama": ("mfs_server.common.embeddings.ollama", "OllamaEmbedding"),
     "local": ("mfs_server.common.embeddings.local", "LocalEmbedding"),
 }
@@ -35,6 +37,8 @@ async def embed(self, texts: list[str]) -> list[list[float]]: ...
     "onnx": "gpahal/bge-m3-onnx-int8",
     "gemini": "gemini-embedding-001",
     "voyage": "voyage-3-lite",
+    "jina": "jina-embeddings-v4",
+    "mistral": "mistral-embed",
     "ollama": "nomic-embed-text",
     "local": "all-MiniLM-L6-v2",
 }
@@ -46,6 +50,8 @@ async def embed(self, texts: list[str]) -> list[list[float]]: ...
     "onnx": "uv sync  (core dep)",
     "gemini": "uv sync --extra gemini",
     "voyage": "uv sync --extra voyage",
+    "jina": "uv sync  (core dep, uses httpx)",
+    "mistral": "uv sync --extra mistral",
     "ollama": "uv sync --extra ollama",
     "local": "uv sync --extra local",
 }

server/python/src/mfs_server/common/embeddings/jina.py

@@ -0,0 +1,97 @@
+"""Jina AI embedding provider.
+
+Requires: no extra — uses the core ``httpx`` dependency
+Environment variables:
+    JINA_API_KEY — required
+
+Jina does not publish a dedicated Python SDK for its embedding REST API, so
+this provider talks to ``https://api.jina.ai/v1/embeddings`` directly via
+httpx. The default model is the latest ``jina-embeddings-v4`` (2048-dim,
+Matryoshka-truncatable between 256 and 2048).
+
+The ``task`` parameter activates a task-specific LoRA adapter. MFS embeds
+indexed chunks and search queries through the same ``embed()`` call, so we
+default to ``retrieval.passage`` — the common case for indexing a corpus.
+Override via the constructor if you need query- or code-specific behavior.
+"""
+
+from __future__ import annotations
+
+import os
+
+_API_URL = "https://api.jina.ai/v1/embeddings"
+
+# Native output dimensions for well-known Jina models. v4 additionally
+# supports Matryoshka truncation between 256 and 2048; callers can override
+# by passing ``dimensions=`` to the constructor.
+_KNOWN_DIMENSIONS: dict[str, int] = {
+    "jina-embeddings-v4": 2048,
+    "jina-embeddings-v3": 1024,
+    "jina-embeddings-v2-base-en": 768,
+    "jina-embeddings-v2-base-code": 768,
+}
+
+
+class JinaEmbedding:
+    """Jina AI embedding provider (REST)."""
+
+    _DEFAULT_BATCH_SIZE = 128
+    _TIMEOUT_SECONDS = 60.0
+
+    def __init__(
+        self,
+        model: str = "jina-embeddings-v4",
+        *,
+        batch_size: int = 0,
+        task: str = "retrieval.passage",
+        dimensions: int | None = None,
+        api_key: str | None = None,
+    ) -> None:
+        import httpx
+
+        self._api_key = api_key or os.environ.get("JINA_API_KEY")
+        if not self._api_key:
+            raise RuntimeError("JINA_API_KEY is required for the Jina embedding provider")
+
+        self._model = model
+        self._task = task
+        self._dimensions = (
+            dimensions if dimensions is not None else _KNOWN_DIMENSIONS.get(model, 2048)
+        )
+        self._batch_size = batch_size if batch_size > 0 else self._DEFAULT_BATCH_SIZE
+        self._client = httpx.AsyncClient(timeout=self._TIMEOUT_SECONDS)
+
+    @property
+    def model_name(self) -> str:
+        return self._model
+
+    @property
+    def dimension(self) -> int:
+        return self._dimensions
+
+    async def embed(self, texts: list[str]) -> list[list[float]]:
+        from .utils import batched_embed
+
+        return await batched_embed(texts, self._embed_batch, self._batch_size)
+
+    async def _embed_batch(self, texts: list[str]) -> list[list[float]]:
+        body: dict = {
+            "model": self._model,
+            "input": texts,
+        }
+        if self._task:
+            body["task"] = self._task
+        if self._dimensions:
+            body["dimensions"] = self._dimensions
+
+        resp = await self._client.post(
+            _API_URL,
+            json=body,
+            headers={
+                "Authorization": f"Bearer {self._api_key}",
+                "Content-Type": "application/json",
+            },
+        )
+        resp.raise_for_status()
+        payload = resp.json()
+        return [item["embedding"] for item in payload["data"]]

server/python/src/mfs_server/common/embeddings/local.py

@@ -1,6 +1,6 @@
 """Local embedding via sentence-transformers (runs on CPU/GPU).
 
-Requires: ``pip install 'memsearch[local]'`` or ``uv add 'memsearch[local]'``
+Requires: ``pip install 'mfs-server[local]'`` or ``uv add 'mfs-server[local]'``
 No API key needed.
 """
 

server/python/src/mfs_server/common/embeddings/mistral.py

@@ -0,0 +1,74 @@
+"""Mistral AI embedding provider.
+
+Requires: ``pip install 'mfs-server[mistral]'`` or ``uv add 'mfs-server[mistral]'``
+Environment variables:
+    MISTRAL_API_KEY — required
+
+Uses the official ``mistralai`` Python SDK. The default model is
+``mistral-embed`` — Mistral's general-purpose text embedding model
+(1024-dim, 8K context). Code-heavy corpora may prefer ``codestral-embed``.
+"""
+
+from __future__ import annotations
+
+import os
+
+# Known output dimensions for Mistral embedding models.
+_KNOWN_DIMENSIONS: dict[str, int] = {
+    "mistral-embed": 1024,
+    "codestral-embed": 1536,
+    "codestral-embed-2505": 1536,
+}
+
+
+class MistralEmbedding:
+    """Mistral AI embedding provider."""
+
+    _DEFAULT_BATCH_SIZE = 64
+
+    def __init__(
+        self,
+        model: str = "mistral-embed",
+        *,
+        batch_size: int = 0,
+        api_key: str | None = None,
+    ) -> None:
+        from mistralai.client import Mistral
+
+        self._api_key = api_key or os.environ.get("MISTRAL_API_KEY")
+        if not self._api_key:
+            raise RuntimeError("MISTRAL_API_KEY is required for the Mistral embedding provider")
+
+        self._client = Mistral(api_key=self._api_key)
+        self._model = model
+        self._dimension = _detect_dimension(self._client, model)
+        self._batch_size = batch_size if batch_size > 0 else self._DEFAULT_BATCH_SIZE
+
+    @property
+    def model_name(self) -> str:
+        return self._model
+
+    @property
+    def dimension(self) -> int:
+        return self._dimension
+
+    async def embed(self, texts: list[str]) -> list[list[float]]:
+        from .utils import batched_embed
+
+        return await batched_embed(texts, self._embed_batch, self._batch_size)
+
+    async def _embed_batch(self, texts: list[str]) -> list[list[float]]:
+        resp = await self._client.embeddings.create_async(model=self._model, inputs=texts)
+        return [item.embedding for item in resp.data]
+
+
+def _detect_dimension(client, model: str) -> int:
+    """Return the embedding dimension for *model*.
+
+    Uses a lookup table for well-known Mistral models. For unknown models,
+    a sync trial embed is performed to discover the dimension.
+    """
+    if model in _KNOWN_DIMENSIONS:
+        return _KNOWN_DIMENSIONS[model]
+    trial = client.embeddings.create(model=model, inputs=["dim"])
+    return len(trial.data[0].embedding)

server/python/src/mfs_server/common/embeddings/ollama.py

@@ -1,6 +1,6 @@
 """Ollama embedding provider (local models via Ollama server).
 
-Requires: ``pip install 'memsearch[ollama]'`` or ``uv add 'memsearch[ollama]'``
+Requires: ``pip install 'mfs-server[ollama]'`` or ``uv add 'mfs-server[ollama]'``
 Environment variables:
     OLLAMA_HOST — optional, default http://localhost:11434
 """

server/python/src/mfs_server/common/embeddings/openai.py

@@ -1,6 +1,6 @@
 """OpenAI embedding provider.
 
-Requires: ``pip install memsearch`` (openai is included by default)
+Requires: ``pip install mfs-server`` (openai is included by default)
 Environment variables:
     OPENAI_API_KEY   — required
     OPENAI_BASE_URL  — optional, override API base URL