DC-324(Feat): AES-256-GCM at-rest encryption for PII fields

docs/adr/0015-pii-encryption-at-rest.md

@@ -0,0 +1,45 @@
+# 15. PII encryption at rest (AES-256-GCM + key rotation)
+
+Date: 2026-05-04
+
+## Status
+
+Accepted
+
+## Context
+
+The portal persists personally identifiable information (PII) that must be shown again to the user or used in flows after it is stored (for example, email, phone, program identifiers, and date of birth from ID proofing). One-way hashing is appropriate for matching-only fields (for example, SSN via `IIdentifierHasher`), but reversible protection is required for fields that are read back in clear form at the application layer.
+
+We also need email lookup by equality in SQL without storing a searchable plaintext email. A deterministic lookup fingerprint (HMAC over the normalized address) supports indexed lookup while ciphertext remains non-deterministic under AES-GCM.
+
+## Decision
+
+1. **Algorithm and envelope.** Use AES-256-GCM with a random nonce per encryption, prefixed by a stable ASCII sentinel (`sep-pii:v1:`). The ciphertext column stores UTF-8 text that embeds **key id**, nonce, tag, and payload so decryption can resolve the correct key from a configured ring.
+
+2. **Key management.** Configure `PiiEncryption:ActiveKeyId` and `PiiEncryption:Keys[]` (`KeyId` + Base64-encoded 256-bit material). Implementations resolve keys via `PiiEncryptionSettings.ResolveKeyRing()`. New columns or environments add keys without schema changes.
+
+3. **Rotation vs. backfill.** **Key rotation** (moving ciphertext from an older key id to the current `ActiveKeyId`) is done with `IPiiSymmetricEncryption.ReSealWithActiveEncryptor(...)` in an operational or batch job — decrypt using the key id embedded in the envelope, then encrypt with the active key. **Startup backfill** (`PiiPlaintextEncryptionBackfill`) is separate: it runs after migrations to encrypt legacy plaintext at rest and to attach `EmailHash` where the email column is already an envelope but `EmailHash` was missing — it does not, by itself, re-wrap every row when only `ActiveKeyId` changes.
+
+4. **Email lookup.** Persist `Email` as ciphertext and `EmailHash` as `IEmailLookupHasher.HashNormalized(...)`. Queries use equality on `EmailHash`, with transitional fallbacks where legacy plaintext rows remain. The unique index applies to non-null `EmailHash` values. Bulk adds via `IDataSeeder.AddUsers` / `AddUsersAsync` skip users whose normalized email is already recognized by `GetExistingUserEmails*` / `GetExistingUserEmailsAsync`, so a legacy plaintext row cannot sit beside a duplicate insert for the same address.
+
+5. **Profile update vs. legacy plaintext.** User updates encrypt email and populate `EmailHash`. Before saving, `DatabaseUserRepository` rejects updates when another user (different Id) still stores the target address as plaintext (EmailHash null and Email column equals the normalized address without the envelope prefix), avoiding silent duplicates alongside the filtered unique index on `EmailHash`.
+
+6. **Failure semantics.** Authenticated decryption failures surface as `PiiDecryptException` (wrapping the underlying crypto error) so corruption or tampering is explicit rather than returning empty strings.
+
+7. **Operational guardrails.** Logging for encryption backfill and seed paths must not include decrypted or plaintext PII; messages use counts and generic conflict text only.
+
+## Consequences
+
+- **Pros:** Strong confidentiality at rest for reversible PII, forward-compatible key rotation, indexed email lookup without plaintext email in the database, and a single encryption path for future columns that need the same pattern.
+- **Cons:** Key material must be managed like other secrets (rotation runbook, secure distribution). Decryption is required on every read path touched by repositories; performance impact is small relative to I/O but must be kept in hot paths. Production startup rejects placeholder `PiiEncryption` keys (`PiiEncryptionGuard`), mirroring `IdentifierHasherGuard`.
+- **Migration:** SQL Server `date`-typed `Users.DateOfBirth` is converted to `nvarchar` via a migration batch that uses dynamic SQL so column renames and copies parse correctly. Down-migration is intentionally unsupported once ciphertext is written.
+- **Email lookup keying:** `EmailLookupHasher` derives its HMAC key from the same `IdentifierHasher:SecretKey` used for other deterministic hashes, with a distinct domain prefix so message formats do not collide. Rotating that secret affects **both** identifier hashing and email lookup hashes — plan coordinated rotation or introduce a dedicated secret later if isolation is required.
+- **Backfill failures:** If `PiiPlaintextEncryptionBackfill` throws during startup, the failure is logged at **error** severity (structured logs / Datadog). The API process still starts so transient DB issues do not brick the service; **operations should alert on that log** and re-run or fix the underlying issue until backfill completes, since plaintext-at-rest rows may remain until then.
+
+## References
+
+- `SEBT.Portal.Core.Services.IPiiSymmetricEncryption`, `IEmailLookupHasher`
+- `SEBT.Portal.Infrastructure.Services.PiiAesGcmSymmetricEncryption`, `PiiPlaintextEncryptionBackfill`
+- `SEBT.Portal.Infrastructure.Repositories.DatabaseUserRepository`, `DatabaseDocVerificationChallengeRepository`
+- Unit tests: `PiiAesGcmSymmetricEncryptionTests`, `PiiPlaintextEncryptionBackfillTests`, `PiiEncryptionGuardTests`, `PiiEncryptionSettingsValidatorTests`
+- Production guard: `SEBT.Portal.Api.Startup.PiiEncryptionGuard`; options coherence: `SEBT.Portal.Infrastructure.Configuration.PiiEncryptionSettingsValidator`

src/SEBT.Portal.Api/Program.cs

@@ -1,5 +1,6 @@
 using System.Text;
 using System.Threading.RateLimiting;
+using System.Data.Common;
 using Microsoft.AspNetCore.Authentication.JwtBearer;
 using Microsoft.AspNetCore.Diagnostics.HealthChecks;
 using Microsoft.AspNetCore.HttpOverrides;
@@ -16,6 +17,7 @@
 using SEBT.Portal.Api.Options;
 using SEBT.Portal.Api.Services;
 using SEBT.Portal.Core.AppSettings;
+using SEBT.Portal.Core.Exceptions;
 using SEBT.Portal.Core.Services;
 using SEBT.Portal.Infrastructure.Configuration;
 using SEBT.Portal.Infrastructure.Services;
@@ -413,6 +415,10 @@ await context.HttpContext.Response.WriteAsJsonAsync(
 if (app.Environment.IsProduction())
 {
     IdentifierHasherGuard.ValidateForProduction(app.Configuration["IdentifierHasher:SecretKey"]);
+
+    var piiEncryptionSettings = app.Configuration.GetSection(PiiEncryptionSettings.SectionName)
+        .Get<PiiEncryptionSettings>();
+    PiiEncryptionGuard.ValidateForProduction(piiEncryptionSettings);
 }
 
 // HMAC-SHA256 requires ≥256-bit (32-byte) key. Fail fast if configured but too short.
@@ -431,6 +437,32 @@ await context.HttpContext.Response.WriteAsJsonAsync(
     var databaseMigrator = scope.ServiceProvider.GetRequiredService<IDatabaseMigrator>();
     await databaseMigrator.MigrateAsync();
 
+    try
+    {
+        var piiBackfill = scope.ServiceProvider.GetRequiredService<PiiPlaintextEncryptionBackfill>();
+        await piiBackfill.ApplyAsync(CancellationToken.None);
+    }
+    catch (PiiDecryptException backfillEx)
+    {
+        Log.Error(
+            backfillEx,
+            "PII ciphertext backfill failed due to decryption/authentication error. " +
+            "Startup continues, but legacy plaintext may remain until this is resolved.");
+    }
+    catch (DbException backfillEx)
+    {
+        Log.Warning(
+            backfillEx,
+            "PII ciphertext backfill hit a database error (likely transient). " +
+            "Startup continues; backfill should be retried.");
+    }
+    catch (Exception backfillEx)
+    {
+        // Error severity so deploy logs / Datadog can alert; app still starts so a transient DB
+        // blip does not take down the service — see docs/adr/0015-pii-encryption-at-rest.md.
+        Log.Error(backfillEx, "PII ciphertext backfill step failed.");
+    }
+
     var seedingSettings = app.Configuration.GetSection(SeedingSettings.SectionName).Get<SeedingSettings>();
     if (app.Environment.IsDevelopment() || seedingSettings?.Enabled == true)
     {

src/SEBT.Portal.Api/Startup/PiiEncryptionGuard.cs

@@ -0,0 +1,75 @@
+using SEBT.Portal.Core.AppSettings;
+
+namespace SEBT.Portal.Api.Startup;
+
+/// <summary>
+/// Validates that PII encryption keys are not default or placeholder values in production.
+/// </summary>
+public static class PiiEncryptionGuard
+{
+    /// <summary>Matches <c>appsettings.json</c> sample ActiveKeyId — not safe for production.</summary>
+    public const string ForbiddenDevelopmentActiveKeyId = "local-dev-primary";
+
+    /// <summary>32× ASCII 'a' (256-bit) — sample key in repo; must not be used in production.</summary>
+    public const string ForbiddenPlaceholderKeyMaterialBase64 = "YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWE=";
+
+    /// <summary>
+    /// Validates configured PII encryption for production. Throws if missing, empty, or known placeholders.
+    /// </summary>
+    public static void ValidateForProduction(PiiEncryptionSettings? settings)
+    {
+        if (settings == null)
+        {
+            throw new InvalidOperationException(
+                "PiiEncryption configuration section is missing. Configure PiiEncryption:ActiveKeyId and PiiEncryption:Keys.");
+        }
+
+        if (string.IsNullOrWhiteSpace(settings.ActiveKeyId))
+        {
+            throw new InvalidOperationException(
+                "PiiEncryption:ActiveKeyId must be set in production (e.g. PIIENCRYPTION__ACTIVEKEYID).");
+        }
+
+        if (settings.Keys == null || settings.Keys.Count == 0)
+        {
+            throw new InvalidOperationException(
+                "PiiEncryption:Keys must contain at least one key entry in production.");
+        }
+
+        if (string.Equals(settings.ActiveKeyId.Trim(), ForbiddenDevelopmentActiveKeyId, StringComparison.OrdinalIgnoreCase))
+        {
+            throw new InvalidOperationException(
+                $"PiiEncryption:ActiveKeyId must not be '{ForbiddenDevelopmentActiveKeyId}' in production. " +
+                "Use a deployment-specific key id and secrets management.");
+        }
+
+        foreach (var entry in settings.Keys)
+        {
+            if (entry == null)
+            {
+                throw new InvalidOperationException(
+                    "PiiEncryption:Keys must not contain null entries in production.");
+            }
+
+            if (string.IsNullOrWhiteSpace(entry.KeyId))
+            {
+                throw new InvalidOperationException(
+                    "Each PiiEncryption:Keys entry must have a non-empty KeyId in production.");
+            }
+
+            if (string.IsNullOrWhiteSpace(entry.KeyMaterialBase64))
+            {
+                throw new InvalidOperationException(
+                    "Each PiiEncryption:Keys entry must have KeyMaterialBase64 set in production.");
+            }
+
+            var material = entry.KeyMaterialBase64.Trim();
+            if (string.Equals(material, ForbiddenPlaceholderKeyMaterialBase64, StringComparison.Ordinal))
+            {
+                throw new InvalidOperationException(
+                    "PiiEncryption key material must not use the repository placeholder Base64 value in production. " +
+                    "Generate random 256-bit keys and store them in secrets (e.g. PIIENCRYPTION__KEYS__0__KEYMATERIALBASE64).");
+            }
+        }
+    }
+}

src/SEBT.Portal.Api/appsettings.Development.example.json

@@ -31,6 +31,15 @@
   "IdentifierHasher": {
     "SecretKey": "YOUR_IDENTIFIER_HASHER_KEY_AT_LEAST_32_CHARS"
   },
+  "PiiEncryption": {
+    "ActiveKeyId": "local-dev-primary",
+    "Keys": [
+      {
+        "KeyId": "local-dev-primary",
+        "KeyMaterialBase64": "YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWE="
+      }
+    ]
+  },
   "StateHouseholdId": {
     "PreferredHouseholdIdTypes": ["Phone"]
   },