Refactoring source code to be fully a bot

.dockerignore

@@ -12,3 +12,7 @@ bot/docs
 
 config.yaml
 .env
+
+accounts/
+backups/
+data-dir/

Dockerfile

@@ -5,9 +5,10 @@ COPY requirements.txt .
 COPY bot/requirements.txt bot/requirements.txt
 
 RUN pip install --no-cache-dir -r requirements.txt \
-    && if [ -f bot/requirements.txt ]; then pip install --no-cache-dir -r bot/requirements.txt; fi
+    && if [ -f bot/requirements.txt ]; then pip install --no-cache-dir -r bot/requirements.txt; fi \
+    && mkdir -p modules
 
 COPY . .
 
-ENTRYPOINT ["python", "monitor.py"]
+ENTRYPOINT ["python", "main.py"]
 CMD []

README.md

@@ -1,95 +1,33 @@
-# [Status App Community Monitoring](https://status.app/)
-
-Monitoring tool for Status App communities. **No personal data is collected from users.**
-
-
-| Field                 | Hashed   | Description                                                 |
-|:----------------------|:---------|:------------------------------------------------------------|
-| **id**                | **Yes**  | The message's ID                                            |
-| **whisper_timestamp** | No       | The whisper timestamp of the message                        |
-| **from**              | **Yes**  | The public key of the user                                  |
-| **message_type**      | No       | The message type                                            |
-| **seen**              | No       | True if the message has been seen otherwise False           |
-| **chat_id**           | No       | The chat ID is a combination of community ID and channel ID |
-| **community_id**      | No       | The ID of the community                                     |
-| **response_to**       | **Yes**  | Ithe public key of the user who the response is for         |
-| **timestamp**         | No       | The timestamp of the message                                |
-| **deleted**           | No       | True if the message was deleted otherwise False             |
-
-Status Bot account information can be found in [`config.yaml`](./config.yaml).
-
-## How it works
-
-```mermaid
-graph LR
-    subgraph Communities[Status App]
-        subgraph Status[Status Community]
-            StatusMessages[Messages]
-            StatusInfo[Information]
-        end
-        subgraph Logos[Logos Community]
-            LogosMessages[Messages]
-            LogosInfo[Information]
-        end
-
-    end
-
-    subgraph Bot[Docker Container]
-        RawDataLocal[(Raw Data)]
-        Script[monitor.py]
-    end
-
-    subgraph IFT[IFT Infrastructure]
-        RawDataIFT[(Raw Data)]
-        ProcessedDataIFT[(Processed Data)]
-
-    end
-
-    Communities <--> |class Account| Script
-    Script --> |SHA256| RawDataLocal
-    RawDataLocal --> |Airbyte| RawDataIFT
-    RawDataIFT --> |dbt| ProcessedDataIFT
-```
+# Status Bot
 
-# Setup
+## Description
 
-## Environment Variables
+This repository allow to run a Bot for the [Status App](https://status.app). The bot can be used for multiple reason:
+- [Interracting with Status Chats](./docs/usage/messaging.md)
+- [Monitoring Community](./docs/usage/monitoring.md)
 
-- `POSTGRES_USERNAME` - Postgres username.
-- `POSTGRES_PASSWORD` - Postgres password.
-- `POSTGRES_DATABASE` - The database name in the Postgres connection.
-- `POSTGRES_HOST` - The Postgres host name that will be remotely connected to.
-- `POSTGRES_PORT` - The Postgres port that will be remotely connected to.
-- `STATUS_DISPLAY_NAME` - The Status display name that will be used to create an account.
-- `STATUS_PASSWORD` - The Status password that will be used to create an account.
-- `STATUS_MNEMONIC` - The mnemonic used to recover the account. If passed a `.bkp` file will be loaded as well. Use this when you want to login to a bot account via Status App, join a community / leave community and export the `.bkp` file.
-- `STATUS_INFURA_TOKEN` - [Infura token](https://www.infura.io/) is required for **token gated communities**
-- `STATUS_COINGECKO_API_KEY` - [Coingecko API Key](https://www.coingecko.com/) is required for **token gated communities**
 
-## Docker deployement
+## Setup
 
-You can use the `docker-compose.yaml` to run the project.
+The recommanded deployement option is with docker compose.
 
-Example of `.env` file to use
-```
-# Status Backend connection
-STATUS_DISPLAY_NAME = "bot-status"
-STATUS_PASSWORD = "ChangeThisPassword"
-STATUS_MNEMONIC= "test test test test test test test test test test test test"
-
-# Necessary for communities that have tokens
-STATUS_INFURA_TOKEN = "Token from https://www.infura.io/"
-STATUS_COINGECKO_API_KEY = "Token from https://www.coingecko.com/"
-
-# Database config
-POSTGRES_HOST=database
-POSTGRES_PORT=5432
-POSTGRES_DATABASE=status-bot
-POSTGRES_USERNAME=status
-POSTGRES_PASSWORD=ChangeThisOneAlso
+### Docker deployment
+
+The [docker compose](./docker-compose.yaml) deployement will spin 3 containers:
+- Status Backend: to interract with the Logos Delivery network used by Status App.
+- Status Bot: the instance of the bot
+- A Postgres database
+
+You can run it with:
+
+```bash
+docker-compose up -d --build
 ```
+it will require a `.env` file for secrets and `config.yaml`. The configuration is detailed in [the documentation](./docs/usage//configuration.md).
 
-## Python
+### Python
+
+You can also run the bot with python, it will require a Status Backend instance available.
 
 1. Setup environment. [Conda](https://www.anaconda.com/) example:
 ```bash
@@ -98,14 +36,10 @@ conda create -n status-monitoring python=3.12
 
 **Note**: Code has been tested with **Python 3.12**.
 
-2. Install `monitor.py` and `bot` requirements
+2. Install requirements
 
 ```bash
-# To run Status bot
-pip install -r ./bot/requirements.txt
-
-# To run monitor.py
-pip install -r ./requirements.txt
+pip install -r requirements.txt
 ```
 
 **Note**: If you are on Windows, you will have to install `psycopg2` instead of `psycopg2-binary`.
@@ -114,13 +48,11 @@ pip install -r ./requirements.txt
 
 If you have already created a Status account and want to use it with it's current data, please make sure you export the `.bkp` file and put it in folder **backups** and have the following `.env` variables:
 
-- `STATUS_DISPLAY_NAME`
-- `STATUS_PASSWORD`
-- `STATUS_MNEMONIC`
+- `BOT_DISPLAY_NAME`
+- `BOT_PASSWORD`
+- `BOT_MNEMONIC_PHRASE`
 
-## Files
 
-- `monitor.py` - Status community message monitoring. It will download and upload messages in parallel.
 
 # Guidelines
 

bot/__init__.py

@@ -1,2 +1,4 @@
 from .account import Account
 from .logger import Logger
+from .postgres import Postgres
+from .config import Config

bot/account.py

@@ -158,9 +158,14 @@ def login(self, password: str, key_uid: Optional[str] = None, display_name: Opti
         # Wallet usage
         if infura_token:
             params["infuraToken"] = infura_token
+        else:
+            self.logger.info("No infura token")
 
         if coingecko_api_key:
             params["coingeckoApiKey"] = coingecko_api_key
+        else:
+            self.logger.info("No coingecko token")
+
 
         if infura_token and coingecko_api_key:
             self.__is_wallet_set = True
@@ -181,7 +186,6 @@ def login(self, password: str, key_uid: Optional[str] = None, display_name: Opti
         self.__info = {
             "public_key": event["public-key"],
             "url": None,
-            "emojis": event["emojiHash"],
             "key_uid": event["key-uid"],
             "compressed_key": event["compressedKey"],
             "mnemonic": event.get("mnemonic", mnemonic),
@@ -388,7 +392,6 @@ def contacts(self) -> dict[str, dict]:
                 "url": self.__call_rpc("urls", "shareUserURLWithData", [contact["id"]]).get("result"),
                 "chat_id": contact["id"],
                 "key_uid": contact["compressedKey"],
-                "emojis": contact["emojiHash"],
                 "contact_state": self.__mappings["contact_request"][contact["contactRequestState"]],
                 "external_contact_state": self.__mappings["contact_request"][contact["contactRequestRemoteState"]],
                 "has_added_us": contact["hasAddedUs"],
@@ -523,6 +526,7 @@ def balance(self) -> pd.DataFrame:
         params = [[self.info["wallet_address"]], True]
         results = self.__call_rpc("wallets", "fetchOrGetCachedWalletBalances", params).get("result", {}).get(self.info["wallet_address"].lower(), [])
         if not results:
+            self.logger.warning(f"No result in RPC response")
             return empty.copy()
 
         balance = pd.DataFrame(results)
@@ -990,8 +994,8 @@ def __load_backup(self):
                 self.__signal.get("messages.new")
                 self.logger.info(f"Successfully loaded file!")
                 break
-
-            self.logger.warning(error)
+            else:
+                self.logger.warning(f"Error while loading the backup: {error}")
 
     def __call_rpc(self, prefix: str, method_name: str, params: Optional[Union[list, dict]] = None) -> dict:
         """

bot/config.py

@@ -0,0 +1,84 @@
+from pydantic import BaseModel
+from pydantic_settings import BaseSettings, SettingsConfigDict
+from pydantic_settings.sources import YamlConfigSettingsSource
+
+
+class BackendConfig(BaseModel):
+    domain: str = "localhost"
+    port: int = 8080
+    is_secure: bool = False
+
+
+class BotConfig(BaseModel):
+    display_name: str = ""
+    public_key: str = ""
+    password: str = ""
+    mnemonic_phrase: str = ""
+    init_account: bool = False
+    compressed_key: str = ""
+    infura_token: str = ""
+    coingecko_api_key: str = ""
+
+class DatabaseConfig(BaseModel):
+    host: str = "database"
+    port: int = 5432
+    user: str = ""
+    password: str = ""
+    name: str = ""
+    schema: str = "public"
+    tables: dict = {}
+
+
+class ApiConfig(BaseModel):
+    enable: bool = True
+    host: str = "0.0.0.0"
+    port: int = 8081
+
+
+class MetricsConfig(BaseModel):
+    enabled: bool = False
+    host: str = "0.0.0.0"
+    port: int = 8000
+
+
+class FilesConfig(BaseModel):
+    current_state: str = "dates.pkl"
+
+
+class ModulesConfig(BaseModel):
+    directories: list[str] = ["./modules", "bot/modules"]
+    enabled: list[str] = []
+    settings: dict = {}
+
+
+class Config(BaseSettings):
+    model_config = SettingsConfigDict(
+        env_nested_delimiter="__",
+        extra="ignore",
+    )
+
+    sleep: int = 10
+    files: FilesConfig = FilesConfig()
+    bot: BotConfig = BotConfig()
+    backend: BackendConfig = BackendConfig()
+    api: ApiConfig = ApiConfig()
+    modules: ModulesConfig = ModulesConfig()
+    metrics: MetricsConfig = MetricsConfig()
+    database: DatabaseConfig = DatabaseConfig()
+
+    _yaml_file = "./config.yaml"
+
+    @classmethod
+    def settings_customise_sources(
+        cls,
+        settings_cls,
+        init_settings,
+        env_settings,
+        dotenv_settings,
+        file_secret_settings,
+    ):
+        return (
+            YamlConfigSettingsSource(settings_cls, yaml_file=cls._yaml_file),
+            env_settings,
+            dotenv_settings,
+        )

bot/metrics.py

@@ -0,0 +1,34 @@
+import logging
+
+from bot.config import MetricsConfig
+from bot.modules.manager import ModuleManager
+from prometheus_client import start_http_server, Gauge, Counter
+
+
+def start_prometheus(metrics_config: MetricsConfig, manager: ModuleManager, logger: logging.Logger):
+    if not metrics_config.enabled:
+        logger.info("Prometheus metrics exporter disabled")
+        return
+
+    health = Gauge("status_bot_health", "Bot health status")
+    version = Gauge("status_bot_version", "Bot version", ["version"])
+    module_loaded = Gauge(
+        "status_bot_module_loaded", "Module loaded", ["module"]
+    )
+    module_errors = Counter(
+        "status_bot_module_errors_total", "Module errors", ["module"]
+    )
+    module_restarts = Counter(
+        "status_bot_module_restarts_total", "Module restarts", ["module"]
+    )
+
+    health.set(1)
+    version.labels(version="0.1.0").set(1)
+
+    for module_name in manager.module_names:
+        module_loaded.labels(module=module_name).set(1)
+
+    host = metrics_config.host
+    port = metrics_config.port
+    start_http_server(port, host)
+    logger.info(f"Prometheus metrics exporter server started on {host}:{port}")

bot/modules/__init__.py

@@ -0,0 +1,4 @@
+from .base import BaseModule, ModuleType, ModuleConfig, ModuleContext
+from .manager import ModuleManager
+
+__all__ = ["BaseModule", "ModuleType", "ModuleConfig", "ModuleContext", "ModuleManager"]

bot/modules/api_server.py

@@ -0,0 +1,43 @@
+import threading
+
+import uvicorn
+from fastapi import FastAPI
+
+from bot.modules.base import BaseModule, ModuleType
+
+
+class APIServerModule(BaseModule):
+
+    @property
+    def module_type(self) -> ModuleType:
+        return ModuleType.SERVICE
+
+    def on_start(self):
+        api_config = self.ctx.shared_state["config"].api
+        self._host = api_config.host
+        self._port = api_config.port
+        self._app: FastAPI = self.ctx.shared_state["fastapi_app"]
+        self._server = None
+
+    def execute(self):
+        if not self.ctx.shared_state["config"].api.enable:
+            self.ctx.logger.info("API server is disabled, skipping startup")
+            return
+
+        config = uvicorn.Config(
+            self._app,
+            host=self._host,
+            port=self._port,
+            log_level="info",
+        )
+        server = uvicorn.Server(config)
+        self._server = server
+        thread = threading.Thread(target=server.run, daemon=True)
+        thread.start()
+        self.ctx.stop_event.wait()
+        server.should_exit = True
+        thread.join(timeout=10)
+
+    def on_stop(self):
+        if self._server:
+            self._server.should_exit = True