Allow custom mapper configs to be edited via CLI

[jarhodes314]

Background

In thin-edge.io v2.0.0, we added support for custom mappers, and a very basic shell of a tedge mapper CLI. This included tedge mapper list to see what mappers are defined, and tedge mapper config get to read config keys for a custom mapper.

At the moment, tedge config supports much more than just reading configuration: it has a set command for writing values, add/remove for per-item updates of list types, listing config keys and their documentation. Ultimately, there should be an equivalent for the mapper config. This could take the form of tedge mapper config {action} {name} {key} (like we have at the moment for mapper config get), or something like tedge config {action} mappers.{name}.{key} (e.g. tedge config set mappers.tb.url example.com)

Any solution we come up with should be compatible with the thin-edge v2 config and the existing CLI structure as that is familiar to users. This means a root tedge.toml with non-mapper configs and a tedge.toml for each configured mapper (plus the ability to upgrade the config from tedge1 to tedge2, but that is a relatively simple process that needn't depend on the tedge_config APIs). Ideally, the change should result in no outward-facing differences in what the CLI looks like to use, even once we've simplified the main tedge config implementation.

Current implementation notes

The existing tedge config CLI is largely powered by a bunch of custom logic in the define_tedge_config! macro, allowing us to generate structs from the basic config shape alongside the tools needed to manipulate this configuration. This works well from a user perspective, but does limit us when it comes to adding new features -- at the moment I am the only team member comfortable making changes to the macro implementation. As a result, I have thus far avoided reusing the macro in the custom mappers configuration

Proposed solution

facet is a new crate designed to solve problems very much like this. We have an established schema for our config file, and we want to manipulate the data within that schema programmatically. This is a fairly significant subset of what the macro currently does, and certainly accounts for a fair amount of complexity.

I believe based on some PoC implementation I have been working on, facet can support at the very least most of what the macro currently does. I would still suggest we have some sort of macro implementation so we can keep the elegant nested struct syntax, e.g.

define_tedge_config! {
    c8y: {
         proxy: {
              bind: {
                  port: u16,
              }
         }
    }
}

instead of

struct Config {
   c8y: ConfigC8y
}
struct ConfigC8y {
    proxy: ConfigC8yProxy,
}
struct ConfigC8yProxy {
    bind: ConfigC8yProxyBind,
}
struct ConfigC8yProxyBind {
   port: u16,
}

However, we should make near-enough every attempt to move logic out of the macro - it's difficult to understand and otherwise generally quite hard to maintain.

Implementation plan

The implementation of this needs to happen in stages. A (rough) suggestion for what this could be is:

1. Create a PoC demonstrating the features that will need to exits in order to create a viable tedge mapper config CLI
2. Add this logic to the real tedge-mapper config, so (for custom mapper configs only) we can establish set and list commands for tedge mapper config
3. Add add and remove commands to the tedge mapper config CLI
4. Add #[derive(Facet)] to the various config structs generated by the macro and roll out tedge mapper config set to the c8y/az/aws configs (as these depend directly on the generated structs from tedge_config).
5. Use facet as the backend for the existing tedge config CLI
6. Clean up the unneeded parts of derive_tedge_config!

Until the full tedge mapper config CLI works with all mappers, the new subcommands should remain hidden.

Required features

Here's a (probably not exhaustive) list of features that a macro replacement needs to support in order to maintain feature parity with tedge_config today:

• Listing config keys with the associated doc comment and examples
• Populating config values from environment variables when reading them
• Warnings for unrecognised environment variables which start with our TEDGE_ prefix
• add/remove config commands which add/remove items from TemplatesSets
• Querying the config for an arbitrary key
• Writing to the config with a key and a value supplied as a string, using FromStr to parse the value
• Default configuration values: these can come from a number of sources: literal values, constants, other keys (optional or their default) or functions that look at adjacent configurations
• Default values based in the config dir (e.g. the default for device.key_path is <config_dir>/device-certs/tedge-private-key.pem)
• Read-only configurations: not currently used as device.id can now be set manually, but this is a feature currently supported by the macro and it would be good to assess whether it can be supported in future if needed
• Deprecated keys: we have some keys that are renamed and should warn the user with the new key name
• Errors when accessing an invalid key: if a key needs to be set, we need to tell the user what key it is that's not set
• Profiled mapper configurations
• Easy access for code consumers: it should be something akin to a struct access as we read a lot of configuration